Balance streaming parents (#18945)
* recreate the circular buffer from time to time * do not update cloud url if the node id is not updated * remove deadlock and optimize pipe size * removed const * finer control on randomized delays * restore children re-connecting to parents * handle partial pipe reads; sender_commit() now checks if the sender is still connected to avoid bombarding it with data that cannot be sent * added commented code about optimizing the array of pollfds * improve interactivity of sender; code cleanup * do not use the pipe for sending messages, instead use a queue in memory (that can never be full) * fix dictionaries families * do not destroy aral on replication exit - it crashes the senders * support multiple dispatchers and connectors; code cleanup * more cleanup * Add serde support for KMeans models. - Serialization/Deserialization support of KMeans models. - Send/receive ML models between a child/parent. - Fix some rare and old crash reports. - Reduce allocations by a couple thousand per second when training. - Enable ML statistics temporarily which might increase CPU consumption. * fix ml models streaming * up to 10 dispatchers and 2 connectors * experiment: limit the number of receivers to the number of cores - 2 * reworked compression at the receiver to minimize read operations * multi-core receivers * use slot 0 on receivers * use slot 0 on receivers * use half the cores for receivers with a minimum of 4 * cancel receiver threads * use offsets instead of pointers in the compressed buffer; track last reads * fix crash on using freed decompressor; core re-org * fix incorrect job registration * fix send_to_plugin() for SSL * add reason to disconnect message * fix signaling receivers to stop * added --dev option to netdata-installer.sh to prevent it from removing the build directory * Fix serde of double values. NaNs and +/- infinities are encoded as strings. * unused param * reset max cbuffer size when it is recreated * struct receiver_state is now private * 1 dispatcher, 1 connector, 2/3 cores for receivers * all replication requests are served by replication threads - never the dispatcher threads * optimize partitions and cache lines for dbengine cache * fix crash on receiver shutdown * rw spinlock now prioritizes writers * backfill all higher tiers * extent cache to 10% * automatic sizing of replication threads * add more replication threads * configure cache eviction parameters to avoid running in aggressive mode all the time * run evictions and flushes every 100ms * add missing initialization * add missing initialization - again * add evictors for all caches * add dedicated evict thread per cache * destroy the completion * avoid sending too many signals to eviction threads * alternative way to make sure there are data to evict * measure inline cache events * disable inline evictions and flushing for open and extent cache * use a spinlock to avoid sending too many signals * batch evictions are not in steps of pages * fix wanted cache size when there are no clean entries in it * fix wanted cache size when there are no clean entries in it * fix wanted cache size again * adaptive batch evictions; batch evictions first try all partitions * move waste events to waste chart * added evict_traversed * evict is smaller steps * removed obsolete code * disabled inlining of evictions and flushing; added timings for evictions * more detailed timings for evictions * use inline evictors * use aral for gorilla pages of 512 bytes, when they are loaded from disk * use aral for all gorilla page sizes loaded from disk * disable inlining again to test it after the memory optimization * timings for dbengine evictions * added timing names * detailed timings * detailed timings - again * removed timings and restored inline evictions * eviction on release only under critical pressure * cleanup and replication tuning * tune cache size calculation * tune replication threads calculation * make streaming receiver exit * Do not allocate/copy extent data twice. * Build/link mimalloc Just for testing, it will be reverted. * lower memory requirements * Link mimalloc statically * run replication with synchronous queries * added missing worker jobs in sender dispatcher * enable batch evictions in pgc * fix sender-dispatcher workers * set max dispatchers to 2 * increase the default replication threads * log stream_info errors * increase replication threads * log the json text when we fail to parse json response of stream_info * stream info response may come back in multiple steps * print the socket error of stream info * added debug to stream info socket error * loop while content-length is smaller than the payload received * Revert "Link mimalloc statically" This reverts commit c98e482d47b8dac39f96e54981ee622cf6d6c6de. * Revert "Build/link mimalloc" This reverts commit 8aae22a28a41da7ac2688bc898846cc6a10a0892. * Remove NEED_PROTOBUF * Use mimalloc * Revert "Use mimalloc" This reverts commit 9a6803478616b3efa8e2671216bfb03bd2040d21. * Use mimalloc * support 256 bytes gorilla pages, when they are loaded from disk * added os_mem_available() * test memory protection * use protection only on one cache * use the free memory of the main cache in the other caches too * use the free memory of the main cache in the open cache too * Batch gorilla writes by tracking the last written number. In a setup with 200 children, `perf` shows that the worst offender is the gorilla write operation, reporting ~17% overhead. With this change `perf` reports ~4% overhead and netdata's CPU consumption decreased by ~16%. * make buffered_reader_next_line() a couple times faster * flushing open cache * Use re2c for the line splitting pluginsd. Function get's optimized around 3x. We should delete old code and use the re2c for the rest of the functions, but we need to keep the PR size as minimal as possible. Will do in follow up PRs. * use cores - 1 for receivers, use only 1 sender * move sender processing to a separate function * Revert "Batch gorilla writes by tracking the last written number." This reverts commit 2e72a5c56daa145d9ee8fed48c39268486e2a12e. * Batch gorilla writes only from writers This reapplies df79be2f01145bd79091a8934d7c80b4b3eb915b and introduces a couple changes to remomove writes from readers. * log information for buffer overflow * fix heap use after free * added comments to the main stream receiver loop * 3 dispatchers * single threaded receiver and sender * code cleanup * de-associate hosts from streaming threads when both the receiver and sender stop, so that each time the threads are re-balanced * fix heap use after free * properly get the slot number of pollfd * fixes * fixes * revert worker changes * reuse streaming threads * backfilling should be synchronous * remove the node last * do not keep a pointer to rellocatable buffer * give to pgc the right page size, not less * restore spreading metrics size across time * use the calculated slots for gorilla pages * accurately track gorilla page size changes * check the sth pointer for validity * code cleanup, files re-org and renames to reflect the new structure of streaming * updated referenced size when the size of a page changes; removed flush spins - fluhses cancelled is a waste event * improve families in netdata statistics * page size histogram per cache * page size histogram per cache queue (hot, dirty, clean) * fix heap after use in pdc.c * rw_spinlocks: when preferring a writer yield so that the writer has the chance to get the lock * do not balloon open and extent caches more than needed (it fragments memory and there is not enough memory for the main cache) * fixed typo * enable trace allocations to work * Skip adding kmeans model when ML dimension has not been created. * PGD is now entirely on ARAL for all types of pages * 2 partitions for PGD * Check for ML queue prior to pushing as well. * merge multiple arals, to avoid wasting memory * significantly less arals; proper calculation of gorilla efficiency * report pgd buffers separately from pgc * aral only for sizes less than 512 bytes * tune aral caches * log the functions using the streaming buffer when concurrent use is detected * aral supporting different pages for collected pages and clean pages - an attempt to minimize fragmentation at high performance * fix misuse of sender thread buffers * select the right buffer, based on the receiver tid * no more rrdpush, renamed to stream * lower aral max page size to 16KiB - in an attempt to lower fragmentation under memory pressure * update opcode handling * automatic sizing of aral limiting its size to 200 items per page or 4 x system pages * tune cache eviction strategy * renamed global statistics to telemetry and split it into multiple files * left over renames of global statistics to telemetry * added heatmap to chart types * note about re-balancing a parents cluster * fix formating * added aral telemetry to find the fragmentation per aral * experiment with a different strategy when making clean pages: always append so that the cache is being constantly rotated; aral telemetry reports utilization instead of fragmentation * aral now takes into account waiting deallocators when it creates new pages * split netdata-conf functions into multiple files; added dbengine use all caches and dbengine out of memory protection settings * tune cache eviction strategy * cache parameters cleanup * rename mem_available to system_memory * Fix variable type. * Add fuzzer for pluginsd line splitter. * use cgroup v1 and v2 to detect memory protection; log on start the detection of memory * fixed typo * added logs about system memory detection * remove debug logs from system memory detection * move the rest of dbengine config to netdata-conf * respect streaming buffer size configured * add workers to pgc eviction threads * renamed worker * fixed flip-flop in size and entries conversions * use aral_by_size when we actually agreegate stats to aral by size * use keyword defintions * move opcode definitions to stream-thread.h * swap struct pollfd slots to make sure all the sockets have an equal chance of being processed * Revert "Add fuzzer for pluginsd line splitter." This reverts commit 454cbcf6e107811a5d7b0265888dfd74abac052c. * Revert "Use re2c for the line splitting pluginsd." This reverts commit 2b2f9d38877780f3058727c343e67b902c5b615d. * stream thread use judy arrays instead of linked lists and pre-allocated arrays * added comment about pfd structure on sender and receiver * fixed logs and made the defaut sender timeout 5 seconds * Spawn ML worker threads based on number of CPUs. * Add statistics for ML allocations/deallocations. * Add host flag to check for pending alert transitions to save Remove precompiled statements Offload processing of alerts in the event loop Queue alert transitions to the metadata event loop to be saved Run metadata checks every 5 seconds * do not block doing socket retries when errno indicates EWOULDBLOCK; insist sending data in send_to_plugin() * Revert "Add host flag to check for pending alert transitions to save" This reverts commit 86ade0e87e596625cf21235b71b897ba093a083d. * fix error reasons * Disable ML memory statistics when using mimalloc * add reason when ml cannot acquire the dimension * added ML memory and depending on the DICT_WITH_STATS define, add aral by size too * do not stream ML when the parent does not have ML enabled * nd_poll() to overcome the starvation of poll() and use epoll() under Linux * nd_poll() optimization to minimize the number of system calls * nd_poll() fix * nd_poll() fix again * make glibc release memory to the system when the system is critical in memory * try bigger aral pages, to enable releasing memory back to the system * Queue alert transitions to the metadata event loop (global list not per host) Add host count to check for pending alert transitions to save Remove precompiled statements Offload processing of alerts in the event loop Run metadata checks every 5 seconds * round robin aral allocations * fix aral round robin * ask glibc to release memory when the allocations are aggressive * tinysleep yields the processor instead of waiting * run malloc_trim() more frequently * Add reference count on alarm_entry * selective tinysleep and processor yielding * revert gorilla batch writes * codacy fixes --------- Co-authored-by: vkalintiris <vasilis@netdata.cloud> Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
Costa Tsaousis committed
Dec 5, 2024 at 18:17 UTC
6b8c6baac28060edbcf01e2b83eb3967736538ec
281 files changed
+21049
-14613
CMakeLists.txt
+122
-26
@@ -438,6 +438,7 @@ check_function_exists(backtrace HAVE_BACKTRACE)
438
check_function_exists(arc4random_buf HAVE_ARC4RANDOM_BUF)
439
check_function_exists(arc4random_uniform HAVE_ARC4RANDOM_UNIFORM)
440
check_function_exists(getrandom HAVE_GETRANDOM)
441
+check_function_exists(sysinfo HAVE_SYSINFO)
442
443
#
444
# check source compilation
@@ -475,6 +476,14 @@ int main() {
476
}
477
" HAVE_C_MALLOPT)
478
479
+check_c_source_compiles("
480
+#include <malloc.h>
481
+int main() {
482
+ malloc_trim(0);
483
+ return 0;
484
+}
485
+" HAVE_C_MALLOC_TRIM)
486
+
487
check_c_source_compiles("
488
#define _GNU_SOURCE
489
#include <stdio.h>
@@ -920,6 +929,21 @@ set(LIBNETDATA_FILES
929
src/libnetdata/xxHash/xxhash.h
930
src/libnetdata/os/random.c
931
src/libnetdata/os/random.h
932
+ src/libnetdata/socket/nd-sock.c
933
+ src/libnetdata/socket/nd-sock.h
934
+ src/libnetdata/socket/listen-sockets.c
935
+ src/libnetdata/socket/listen-sockets.h
936
+ src/libnetdata/socket/poll-events.c
937
+ src/libnetdata/socket/poll-events.h
938
+ src/libnetdata/socket/connect-to.c
939
+ src/libnetdata/socket/connect-to.h
940
+ src/libnetdata/socket/socket-peers.c
941
+ src/libnetdata/socket/socket-peers.h
942
+ src/libnetdata/libjudy/judyl-typed.h
943
+ src/libnetdata/os/system_memory.c
944
+ src/libnetdata/os/system_memory.h
945
+ src/libnetdata/socket/nd-poll.c
946
+ src/libnetdata/socket/nd-poll.h
947
)
948
949
set(LIBH2O_FILES
@@ -1013,8 +1037,8 @@ set(DAEMON_FILES
1037
src/daemon/daemon.h
1038
src/daemon/libuv_workers.c
1039
src/daemon/libuv_workers.h
1016
- src/daemon/global_statistics.c
1017
- src/daemon/global_statistics.h
1040
+ src/daemon/telemetry/telemetry.c
1041
+ src/daemon/telemetry/telemetry.h
1042
src/daemon/analytics.c
1043
src/daemon/analytics.h
1044
src/daemon/main.c
@@ -1035,15 +1059,59 @@ set(DAEMON_FILES
1059
src/daemon/pipename.h
1060
src/daemon/unit_test.c
1061
src/daemon/unit_test.h
1038
- src/daemon/config/dyncfg.c
1039
- src/daemon/config/dyncfg.h
1040
- src/daemon/config/dyncfg-files.c
1041
- src/daemon/config/dyncfg-unittest.c
1042
- src/daemon/config/dyncfg-inline.c
1043
- src/daemon/config/dyncfg-echo.c
1044
- src/daemon/config/dyncfg-internals.h
1045
- src/daemon/config/dyncfg-intercept.c
1046
- src/daemon/config/dyncfg-tree.c
1062
+ src/daemon/dyncfg/dyncfg.c
1063
+ src/daemon/dyncfg/dyncfg.h
1064
+ src/daemon/dyncfg/dyncfg-files.c
1065
+ src/daemon/dyncfg/dyncfg-unittest.c
1066
+ src/daemon/dyncfg/dyncfg-inline.c
1067
+ src/daemon/dyncfg/dyncfg-echo.c
1068
+ src/daemon/dyncfg/dyncfg-internals.h
1069
+ src/daemon/dyncfg/dyncfg-intercept.c
1070
+ src/daemon/dyncfg/dyncfg-tree.c
1071
+ src/daemon/telemetry/telemetry-http-api.c
1072
+ src/daemon/telemetry/telemetry-http-api.h
1073
+ src/daemon/telemetry/telemetry-queries.c
1074
+ src/daemon/telemetry/telemetry-queries.h
1075
+ src/daemon/telemetry/telemetry-ingestion.c
1076
+ src/daemon/telemetry/telemetry-ingestion.h
1077
+ src/daemon/telemetry/telemetry-ml.c
1078
+ src/daemon/telemetry/telemetry-ml.h
1079
+ src/daemon/telemetry/telemetry-gorilla.c
1080
+ src/daemon/telemetry/telemetry-gorilla.h
1081
+ src/daemon/telemetry/telemetry-daemon.c
1082
+ src/daemon/telemetry/telemetry-daemon.h
1083
+ src/daemon/telemetry/telemetry-daemon-memory.c
1084
+ src/daemon/telemetry/telemetry-daemon-memory.h
1085
+ src/daemon/telemetry/telemetry-sqlite3.c
1086
+ src/daemon/telemetry/telemetry-sqlite3.h
1087
+ src/daemon/telemetry/telemetry-dbengine.c
1088
+ src/daemon/telemetry/telemetry-dbengine.h
1089
+ src/daemon/telemetry/telemetry-string.c
1090
+ src/daemon/telemetry/telemetry-string.h
1091
+ src/daemon/telemetry/telemetry-heartbeat.c
1092
+ src/daemon/telemetry/telemetry-heartbeat.h
1093
+ src/daemon/telemetry/telemetry-dictionary.c
1094
+ src/daemon/telemetry/telemetry-dictionary.h
1095
+ src/daemon/telemetry/telemetry-workers.c
1096
+ src/daemon/telemetry/telemetry-workers.h
1097
+ src/daemon/telemetry/telemetry-trace-allocations.c
1098
+ src/daemon/telemetry/telemetry-trace-allocations.h
1099
+ src/daemon/telemetry/telemetry-aral.c
1100
+ src/daemon/telemetry/telemetry-aral.h
1101
+ src/daemon/config/netdata-conf-db.c
1102
+ src/daemon/config/netdata-conf-db.h
1103
+ src/daemon/config/netdata-conf.h
1104
+ src/daemon/config/netdata-conf-backwards-compatibility.c
1105
+ src/daemon/config/netdata-conf-backwards-compatibility.h
1106
+ src/daemon/config/netdata-conf-web.c
1107
+ src/daemon/config/netdata-conf-web.h
1108
+ src/daemon/config/netdata-conf-directories.c
1109
+ src/daemon/config/netdata-conf-directories.h
1110
+ src/daemon/config/netdata-conf-logs.c
1111
+ src/daemon/config/netdata-conf-logs.h
1112
+ src/daemon/config/netdata-conf-global.c
1113
+ src/daemon/config/netdata-conf-global.h
1114
+ src/daemon/config/netdata-conf.c
1115
)
1116
1117
set(H2O_FILES
@@ -1227,15 +1295,34 @@ if(ENABLE_ML)
1295
set(ML_FILES
1296
src/ml/ad_charts.h
1297
src/ml/ad_charts.cc
1230
- src/ml/Config.cc
1298
src/ml/dlib/dlib/all/source.cpp
1232
- src/ml/ml.h
1299
src/ml/ml.cc
1234
- src/ml/ml-private.h
1300
+ src/ml/ml_calculated_number.h
1301
+ src/ml/ml_host.h
1302
+ src/ml/ml_config.h
1303
+ src/ml/ml_config.cc
1304
+ src/ml/ml_dimension.h
1305
+ src/ml/ml_enums.h
1306
+ src/ml/ml_enums.cc
1307
+ src/ml/ml_features.h
1308
+ src/ml/ml_features.cc
1309
+ src/ml/ml_kmeans.h
1310
+ src/ml/ml_kmeans.cc
1311
+ src/ml/ml_queue.h
1312
+ src/ml/ml_worker.h
1313
+ src/ml/ml_string_wrapper.h
1314
+ src/ml/ml_queue.cc
1315
+ src/ml/ml_private.h
1316
+ src/ml/ml_public.h
1317
+ src/ml/ml_public.cc
1318
)
1319
+
1320
+ if(NOT ENABLE_MIMALLOC)
1321
+ list(APPEND ML_FILES src/ml/ml_memory.cc)
1322
+ endif()
1323
else()
1324
set(ML_FILES
1238
- src/ml/ml.h
1325
+ src/ml/ml_public.h
1326
src/ml/ml-dummy.c
1327
)
1328
endif()
@@ -1338,6 +1425,8 @@ set(RRD_PLUGIN_FILES
1425
src/database/rrdfunctions-exporters.h
1426
src/database/rrdfunctions-internals.h
1427
src/database/rrdcollector-internals.h
1428
+ src/database/rrd-database-mode.h
1429
+ src/database/rrd-database-mode.c
1430
)
1431
1432
if(ENABLE_DBENGINE)
@@ -1405,7 +1494,7 @@ set(SYSTEMD_JOURNAL_PLUGIN_FILES
1494
)
1495
1496
set(STREAMING_PLUGIN_FILES
1408
- src/streaming/rrdpush.h
1497
+ src/streaming/stream.h
1498
src/streaming/stream-compression/compression.c
1499
src/streaming/stream-compression/compression.h
1500
src/streaming/stream-compression/brotli.c
@@ -1416,8 +1505,8 @@ set(STREAMING_PLUGIN_FILES
1505
src/streaming/stream-compression/lz4.h
1506
src/streaming/stream-compression/zstd.c
1507
src/streaming/stream-compression/zstd.h
1419
- src/streaming/receiver.c
1420
- src/streaming/sender.c
1508
+ src/streaming/stream-receiver.c
1509
+ src/streaming/stream-sender.c
1510
src/streaming/replication.c
1511
src/streaming/replication.h
1512
src/streaming/h2o-common.h
@@ -1429,11 +1518,11 @@ set(STREAMING_PLUGIN_FILES
1518
src/streaming/stream-path.h
1519
src/streaming/stream-capabilities.c
1520
src/streaming/stream-capabilities.h
1432
- src/streaming/sender-connect.c
1433
- src/streaming/sender-internals.h
1434
- src/streaming/sender-execute.c
1435
- src/streaming/sender-commit.c
1436
- src/streaming/sender-destinations.c
1521
+ src/streaming/stream-connector.c
1522
+ src/streaming/stream-sender-internals.h
1523
+ src/streaming/stream-sender-execute.c
1524
+ src/streaming/stream-sender-commit.c
1525
+ src/streaming/stream-parents.c
1526
src/streaming/stream-handshake.c
1527
src/streaming/protocol/command-function.c
1528
src/streaming/protocol/command-host-labels.c
@@ -1443,11 +1532,17 @@ set(STREAMING_PLUGIN_FILES
1532
src/streaming/stream-conf.c
1533
src/streaming/stream-conf.h
1534
src/streaming/stream-handshake.h
1446
- src/streaming/sender.h
1447
- src/streaming/sender-destinations.h
1535
+ src/streaming/stream-parents.h
1536
src/streaming/rrdhost-status.c
1537
src/streaming/rrdhost-status.h
1450
- src/streaming/receiver.h
1538
+ src/streaming/stream-sender-api.c
1539
+ src/streaming/stream-receiver-internals.h
1540
+ src/streaming/stream-receiver-api.c
1541
+ src/streaming/stream-thread.c
1542
+ src/streaming/stream-thread.h
1543
+ src/streaming/stream-receiver-connection.c
1544
+ src/streaming/stream-sender-commit.h
1545
+ src/streaming/stream-traffic-types.h
1546
)
1547
1548
set(WEB_PLUGIN_FILES
@@ -1459,6 +1554,7 @@ set(WEB_PLUGIN_FILES
1554
src/web/server/static/static-threaded.h
1555
src/web/server/web_client_cache.c
1556
src/web/server/web_client_cache.h
1557
+ src/web/api/v3/api_v3_stream_info.c
1558
src/web/api/v3/api_v3_stream_path.c
1559
)
1560
docs/developer-and-contributor-corner/python-collector.txt
+1
-1
@@ -115,7 +115,7 @@ context, charttype]`, where:
115
- `family`: An identifier used to group charts together (can be null).
116
- `context`: An identifier used to group contextually similar charts together. The best practice is to provide a context
117
that is `A.B`, with `A` being the name of the collector, and `B` being the name of the specific metric.
118
-- `charttype`: Either `line`, `area`, or `stacked`. If null line is the default value.
118
+- `charttype`: Either `line`, `area`, `stacked` or `heatmap`. If null line is the default value.
119
120
You can read more about `family` and `context` in the [Netdata Charts](/docs/dashboards-and-charts/netdata-charts.md) doc.
121
docs/diagrams/data_structures/web.svg
+1
-1
@@ -1,2 +1,2 @@
1
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
2
-<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1805px" height="765px" viewBox="-0.5 -0.5 1805 765" style="background-color: rgb(255, 255, 255);"><defs><clipPath id="mx-clip-7-61-322-26-0"><rect x="7" y="61" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-87-322-26-0"><rect x="7" y="87" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-113-322-26-0"><rect x="7" y="113" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-139-322-26-0"><rect x="7" y="139" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-165-322-26-0"><rect x="7" y="165" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-191-322-26-0"><rect x="7" y="191" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-217-322-26-0"><rect x="7" y="217" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-243-322-26-0"><rect x="7" y="243" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-269-322-26-0"><rect x="7" y="269" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-295-322-26-0"><rect x="7" y="295" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-321-322-26-0"><rect x="7" y="321" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-347-322-26-0"><rect x="7" y="347" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-373-322-26-0"><rect x="7" y="373" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-399-322-26-0"><rect x="7" y="399" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-425-322-26-0"><rect x="7" y="425" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-451-322-26-0"><rect x="7" y="451" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-477-322-26-0"><rect x="7" y="477" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-503-322-26-0"><rect x="7" y="503" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-529-322-26-0"><rect x="7" y="529" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-555-322-26-0"><rect x="7" y="555" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-581-322-26-0"><rect x="7" y="581" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-607-322-26-0"><rect x="7" y="607" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-633-322-26-0"><rect x="7" y="633" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-659-322-26-0"><rect x="7" y="659" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-685-322-26-0"><rect x="7" y="685" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-711-322-26-0"><rect x="7" y="711" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-737-322-26-0"><rect x="7" y="737" width="322" height="26"/></clipPath><clipPath id="mx-clip-477-165-252-26-0"><rect x="477" y="165" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-191-252-26-0"><rect x="477" y="191" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-217-252-26-0"><rect x="477" y="217" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-243-252-26-0"><rect x="477" y="243" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-269-252-26-0"><rect x="477" y="269" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-295-252-26-0"><rect x="477" y="295" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-321-252-26-0"><rect x="477" y="321" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-347-252-26-0"><rect x="477" y="347" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-373-252-26-0"><rect x="477" y="373" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-399-252-26-0"><rect x="477" y="399" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-425-252-26-0"><rect x="477" y="425" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-451-252-26-0"><rect x="477" y="451" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-577-252-26-0"><rect x="477" y="577" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-603-252-26-0"><rect x="477" y="603" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-629-252-26-0"><rect x="477" y="629" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-655-252-26-0"><rect x="477" y="655" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-681-252-26-0"><rect x="477" y="681" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-707-252-26-0"><rect x="477" y="707" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-733-252-26-0"><rect x="477" y="733" width="252" height="26"/></clipPath><clipPath id="mx-clip-837-113-232-26-0"><rect x="837" y="113" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-139-232-26-0"><rect x="837" y="139" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-165-232-26-0"><rect x="837" y="165" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-191-232-26-0"><rect x="837" y="191" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-217-232-26-0"><rect x="837" y="217" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-243-232-26-0"><rect x="837" y="243" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-269-232-26-0"><rect x="837" y="269" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-295-232-26-0"><rect x="837" y="295" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-321-232-26-0"><rect x="837" y="321" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-347-232-26-0"><rect x="837" y="347" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-373-232-26-0"><rect x="837" y="373" width="232" height="26"/></clipPath><clipPath id="mx-clip-1177-61-222-26-0"><rect x="1177" y="61" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-87-222-26-0"><rect x="1177" y="87" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-113-222-26-0"><rect x="1177" y="113" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-139-222-26-0"><rect x="1177" y="139" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-165-222-26-0"><rect x="1177" y="165" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-191-222-26-0"><rect x="1177" y="191" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-217-222-26-0"><rect x="1177" y="217" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-243-222-26-0"><rect x="1177" y="243" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-269-222-26-0"><rect x="1177" y="269" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-295-222-26-0"><rect x="1177" y="295" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-321-222-26-0"><rect x="1177" y="321" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-347-222-26-0"><rect x="1177" y="347" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-373-222-26-0"><rect x="1177" y="373" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-399-222-26-0"><rect x="1177" y="399" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-425-222-26-0"><rect x="1177" y="425" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-451-222-26-0"><rect x="1177" y="451" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-477-222-26-0"><rect x="1177" y="477" width="222" height="26"/></clipPath><clipPath id="mx-clip-1527-126-242-26-0"><rect x="1527" y="126" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-152-242-26-0"><rect x="1527" y="152" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-178-242-26-0"><rect x="1527" y="178" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-204-242-26-0"><rect x="1527" y="204" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-230-242-26-0"><rect x="1527" y="230" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-256-242-26-0"><rect x="1527" y="256" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-282-242-26-0"><rect x="1527" y="282" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-308-242-26-0"><rect x="1527" y="308" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-334-242-26-0"><rect x="1527" y="334" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-360-242-26-0"><rect x="1527" y="360" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-386-242-26-0"><rect x="1527" y="386" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-412-242-26-0"><rect x="1527" y="412" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-438-242-26-0"><rect x="1527" y="438" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-464-242-26-0"><rect x="1527" y="464" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-490-242-26-0"><rect x="1527" y="490" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-516-242-26-0"><rect x="1527" y="516" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-542-242-26-0"><rect x="1527" y="542" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-568-242-26-0"><rect x="1527" y="568" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-594-242-26-0"><rect x="1527" y="594" width="242" height="26"/></clipPath></defs><path d="M 3 56 L 3 30 L 333 30 L 333 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 3 56 L 3 758 L 333 758 L 333 56" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 3 56 L 333 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="167.5" y="47.5">web_client</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-61-322-26-0)" font-size="12px"><text x="8.5" y="73.5">unsigned long long id</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-87-322-26-0)" font-size="12px"><text x="8.5" y="99.5">WEB_CLIENT_FLAGS flags</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-113-322-26-0)" font-size="12px"><text x="8.5" y="125.5">WEB_CLIENT_MODE mode</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-139-322-26-0)" font-size="12px"><text x="8.5" y="151.5">WEB_CLIENT_ACL acl</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-165-322-26-0)" font-size="12px"><text x="8.5" y="177.5">size_t header_parse_tries</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-191-322-26-0)" font-size="12px"><text x="8.5" y="203.5">size_t header_parse_last_size</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-217-322-26-0)" font-size="12px"><text x="8.5" y="229.5">int tcp_cork</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-243-322-26-0)" font-size="12px"><text x="8.5" y="255.5">int ifd</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-269-322-26-0)" font-size="12px"><text x="8.5" y="281.5">int ofd</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-295-322-26-0)" font-size="12px"><text x="8.5" y="307.5">char client_ip[NI_MAXHOST+1}</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-321-322-26-0)" font-size="12px"><text x="8.5" y="333.5">char client_port[NI_MAXSERV+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-347-322-26-0)" font-size="12px"><text x="8.5" y="359.5">char decoded_url[NETDATA_WEB_REQUEST_URL_SIZE+1</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-373-322-26-0)" font-size="12px"><text x="8.5" y="385.5">char last_url[NETDATA_WEB_REQUEST_URL_SIZE+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-399-322-26-0)" font-size="12px"><text x="8.5" y="411.5">struct timeval tv_in, tv_ready</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-425-322-26-0)" font-size="12px"><text x="8.5" y="437.5">char cookie1[NETDATA_WEB_REQUEST_COOKIE_SIZE+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-451-322-26-0)" font-size="12px"><text x="8.5" y="463.5">char cookie2[NETDATA_WEB_REQUEST_COOKIE_SIZE+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-477-322-26-0)" font-size="12px"><text x="8.5" y="489.5">char origin[NETDATA_WEB_REQUEST_ORIGIN_HEADER_SIZE+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-503-322-26-0)" font-size="12px"><text x="8.5" y="515.5">char *user_agent</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-529-322-26-0)" font-size="12px"><text x="8.5" y="541.5">struct response response</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-555-322-26-0)" font-size="12px"><text x="8.5" y="567.5">size_t stats_received_bytes</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-581-322-26-0)" font-size="12px"><text x="8.5" y="593.5">size_t stats_sent_bytes</text></g><path d="M 333 615 L 353 615 L 353 10 L 168 10 L 168 23.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 168 28.88 L 164.5 21.88 L 168 23.63 L 171.5 21.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-607-322-26-0)" font-size="12px"><text x="8.5" y="619.5">struct web_client *prev</text></g><path d="M 333 641 L 353 641 L 353 10 L 168 10 L 168 23.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 168 28.88 L 164.5 21.88 L 168 23.63 L 171.5 21.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(304.5,299.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="102" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;background-color:#ffffff;">double linked list of</div></div></foreignObject><text x="51" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">double linked list of</text></switch></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-633-322-26-0)" font-size="12px"><text x="8.5" y="645.5">struct web_client *next</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-659-322-26-0)" font-size="12px"><text x="8.5" y="671.5">netdata_thread_t thread</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-685-322-26-0)" font-size="12px"><text x="8.5" y="697.5">volatile int running</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-711-322-26-0)" font-size="12px"><text x="8.5" y="723.5">size_t pollinfo_slot</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-737-322-26-0)" font-size="12px"><text x="8.5" y="749.5">size_t pollinfo_filecopy_slot</text></g><path d="M 473 160 L 473 134 L 733 134 L 733 160" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 160 L 473 472 L 733 472 L 733 160" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 160 L 733 160" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="602.5" y="151.5">response</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-165-252-26-0)" font-size="12px"><text x="478.5" y="177.5">BUFFER *header</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-191-252-26-0)" font-size="12px"><text x="478.5" y="203.5">BUFFER *header_output</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-217-252-26-0)" font-size="12px"><text x="478.5" y="229.5">BUFFER *data</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-243-252-26-0)" font-size="12px"><text x="478.5" y="255.5">int code</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-269-252-26-0)" font-size="12px"><text x="478.5" y="281.5">size_t rlen</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-295-252-26-0)" font-size="12px"><text x="478.5" y="307.5">size_t sent</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-321-252-26-0)" font-size="12px"><text x="478.5" y="333.5">int zoutput</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-347-252-26-0)" font-size="12px"><text x="478.5" y="359.5">z_stream zstream</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-373-252-26-0)" font-size="12px"><text x="478.5" y="385.5">Bytef zbuffer[NETDATA_WEB_RESPONSE_ZLIB_CHUNK_SIZE]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-399-252-26-0)" font-size="12px"><text x="478.5" y="411.5">size_t zsent</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-425-252-26-0)" font-size="12px"><text x="478.5" y="437.5">size_t zhave</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-451-252-26-0)" font-size="12px"><text x="478.5" y="463.5">unsigned int zinitialized</text></g><path d="M 333 537 L 403 537 L 403 114 L 603 114 L 603 127.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 603 132.88 L 599.5 125.88 L 603 127.63 L 606.5 125.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 572 L 473 546 L 733 546 L 733 572" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 572 L 473 754 L 733 754 L 733 572" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 572 L 733 572" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="602.5" y="563.5">clients_cache</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-577-252-26-0)" font-size="12px"><text x="478.5" y="589.5">pid_t pid</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-603-252-26-0)" font-size="12px"><text x="478.5" y="615.5">struct web_client *used</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-629-252-26-0)" font-size="12px"><text x="478.5" y="641.5">size_t used_count</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-655-252-26-0)" font-size="12px"><text x="478.5" y="667.5">struct web_client *avail</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-681-252-26-0)" font-size="12px"><text x="478.5" y="693.5">size_t avail_count</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-707-252-26-0)" font-size="12px"><text x="478.5" y="719.5">size_t reused</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-733-252-26-0)" font-size="12px"><text x="478.5" y="745.5">size_t allocated</text></g><path d="M 733 611 L 753 611 L 753 10 L 168 10 L 168 23.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 168 28.88 L 164.5 21.88 L 168 23.63 L 171.5 21.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(724.5,12.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="57" height="11" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 11px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;background-color:#ffffff;">linked list of</div></div></foreignObject><text x="29" y="11" fill="#000000" text-anchor="middle" font-size="11px" font-family="Helvetica">linked list of</text></switch></g><path d="M 733 663 L 753 663 L 753 10 L 168 10 L 168 23.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 168 28.88 L 164.5 21.88 L 168 23.63 L 171.5 21.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(724.5,38.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="57" height="11" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 11px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;background-color:#ffffff;">linked list of</div></div></foreignObject><text x="29" y="11" fill="#000000" text-anchor="middle" font-size="11px" font-family="Helvetica">linked list of</text></switch></g><path d="M 833 108 L 833 82 L 1073 82 L 1073 108" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 833 108 L 833 394 L 1073 394 L 1073 108" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 833 108 L 1073 108" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="952.5" y="99.5">listen_sockets</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-113-232-26-0)" font-size="12px"><text x="838.5" y="125.5">struct config *config</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-139-232-26-0)" font-size="12px"><text x="838.5" y="151.5">const char *config_section</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-165-232-26-0)" font-size="12px"><text x="838.5" y="177.5">const char *default_bind_to</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-191-232-26-0)" font-size="12px"><text x="838.5" y="203.5">uint16_t default_port</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-217-232-26-0)" font-size="12px"><text x="838.5" y="229.5">int backlog</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-243-232-26-0)" font-size="12px"><text x="838.5" y="255.5">size_t opened</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-269-232-26-0)" font-size="12px"><text x="838.5" y="281.5">size_t failed</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-295-232-26-0)" font-size="12px"><text x="838.5" y="307.5">int fds[MAX_LISTEN_FDS]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-321-232-26-0)" font-size="12px"><text x="838.5" y="333.5">int *fds_names[MAX_LISTEN_FDS]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-347-232-26-0)" font-size="12px"><text x="838.5" y="359.5">int fds_types[MAX_LISTEN_FDS]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-373-232-26-0)" font-size="12px"><text x="838.5" y="385.5">int fds_families[MAX_LISTEN_FDS]</text></g><path d="M 1173 56 L 1173 30 L 1403 30 L 1403 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1173 56 L 1173 498 L 1403 498 L 1403 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1173 56 L 1403 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="1287.5" y="47.5">POLLINFO</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-61-222-26-0)" font-size="12px"><text x="1178.5" y="73.5">POLLJOB *p</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-87-222-26-0)" font-size="12px"><text x="1178.5" y="99.5">size_t slot</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-113-222-26-0)" font-size="12px"><text x="1178.5" y="125.5">int fd</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-139-222-26-0)" font-size="12px"><text x="1178.5" y="151.5">int socktype</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-165-222-26-0)" font-size="12px"><text x="1178.5" y="177.5">char *client_ip</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-191-222-26-0)" font-size="12px"><text x="1178.5" y="203.5">char *client_port</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-217-222-26-0)" font-size="12px"><text x="1178.5" y="229.5">time_t connected_t</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-243-222-26-0)" font-size="12px"><text x="1178.5" y="255.5">time_t last_received_t</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-269-222-26-0)" font-size="12px"><text x="1178.5" y="281.5">time_t last_sent_t</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-295-222-26-0)" font-size="12px"><text x="1178.5" y="307.5">size_t recv_count</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-321-222-26-0)" font-size="12px"><text x="1178.5" y="333.5">size_t send_count</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-347-222-26-0)" font-size="12px"><text x="1178.5" y="359.5">uint32_t flags</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-373-222-26-0)" font-size="12px"><text x="1178.5" y="385.5">void (*del_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-399-222-26-0)" font-size="12px"><text x="1178.5" y="411.5">int (*rcv_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-425-222-26-0)" font-size="12px"><text x="1178.5" y="437.5">int (*snd_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-451-222-26-0)" font-size="12px"><text x="1178.5" y="463.5">void *data</text></g><path d="M 1403 485 L 1423 485 L 1423 10 L 1279 10 L 1279 22.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1279 27.88 L 1275.5 20.88 L 1279 22.63 L 1282.5 20.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-477-222-26-0)" font-size="12px"><text x="1178.5" y="489.5">struct pollinfo *next</text></g><path d="M 1523 121 L 1523 95 L 1773 95 L 1773 121" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1523 121 L 1523 615 L 1773 615 L 1773 121" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1523 121 L 1773 121" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="1647.5" y="112.5">POLLJOB</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-126-242-26-0)" font-size="12px"><text x="1528.5" y="138.5">size_t slots</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-152-242-26-0)" font-size="12px"><text x="1528.5" y="164.5">size_t used</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-178-242-26-0)" font-size="12px"><text x="1528.5" y="190.5">size_t min</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-204-242-26-0)" font-size="12px"><text x="1528.5" y="216.5">size_t max</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-230-242-26-0)" font-size="12px"><text x="1528.5" y="242.5">size_t limit</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-256-242-26-0)" font-size="12px"><text x="1528.5" y="268.5">time_t complete_request_timeout</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-282-242-26-0)" font-size="12px"><text x="1528.5" y="294.5">time_t idle_timeout</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-308-242-26-0)" font-size="12px"><text x="1528.5" y="320.5">time_t check_every</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-334-242-26-0)" font-size="12px"><text x="1528.5" y="346.5">time_t timer_milliseconds</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-360-242-26-0)" font-size="12px"><text x="1528.5" y="372.5">void *timer_data</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-386-242-26-0)" font-size="12px"><text x="1528.5" y="398.5">struct pollfd *fds</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-412-242-26-0)" font-size="12px"><text x="1528.5" y="424.5">struct pollinfo *inf</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-438-242-26-0)" font-size="12px"><text x="1528.5" y="450.5">struct pollinfo *first_free</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-464-242-26-0)" font-size="12px"><text x="1528.5" y="476.5">SIMPLE_PATTERN *access_list</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-490-242-26-0)" font-size="12px"><text x="1528.5" y="502.5">void *(*add_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-516-242-26-0)" font-size="12px"><text x="1528.5" y="528.5">void (*dell_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-542-242-26-0)" font-size="12px"><text x="1528.5" y="554.5">int (*rcv_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-568-242-26-0)" font-size="12px"><text x="1528.5" y="580.5">int (*snd_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-594-242-26-0)" font-size="12px"><text x="1528.5" y="606.5">void (*tmr_callback)</text></g><path d="M 1403 69 L 1648 69 L 1648 88.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1648 93.88 L 1644.5 86.88 L 1648 88.63 L 1651.5 86.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1773 420 L 1793 420 L 1793 10 L 1279 10 L 1279 22.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1279 27.88 L 1275.5 20.88 L 1279 22.63 L 1282.5 20.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1773 446 L 1793 446 L 1793 10 L 1278 10 L 1278 22.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1278 27.88 L 1274.5 20.88 L 1278 22.63 L 1281.5 20.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/></svg>
\ No newline at end of file
2
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1805px" height="765px" viewBox="-0.5 -0.5 1805 765" style="background-color: rgb(255, 255, 255);"><defs><clipPath id="mx-clip-7-61-322-26-0"><rect x="7" y="61" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-87-322-26-0"><rect x="7" y="87" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-113-322-26-0"><rect x="7" y="113" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-139-322-26-0"><rect x="7" y="139" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-165-322-26-0"><rect x="7" y="165" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-191-322-26-0"><rect x="7" y="191" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-217-322-26-0"><rect x="7" y="217" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-243-322-26-0"><rect x="7" y="243" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-269-322-26-0"><rect x="7" y="269" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-295-322-26-0"><rect x="7" y="295" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-321-322-26-0"><rect x="7" y="321" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-347-322-26-0"><rect x="7" y="347" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-373-322-26-0"><rect x="7" y="373" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-399-322-26-0"><rect x="7" y="399" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-425-322-26-0"><rect x="7" y="425" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-451-322-26-0"><rect x="7" y="451" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-477-322-26-0"><rect x="7" y="477" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-503-322-26-0"><rect x="7" y="503" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-529-322-26-0"><rect x="7" y="529" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-555-322-26-0"><rect x="7" y="555" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-581-322-26-0"><rect x="7" y="581" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-607-322-26-0"><rect x="7" y="607" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-633-322-26-0"><rect x="7" y="633" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-659-322-26-0"><rect x="7" y="659" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-685-322-26-0"><rect x="7" y="685" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-711-322-26-0"><rect x="7" y="711" width="322" height="26"/></clipPath><clipPath id="mx-clip-7-737-322-26-0"><rect x="7" y="737" width="322" height="26"/></clipPath><clipPath id="mx-clip-477-165-252-26-0"><rect x="477" y="165" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-191-252-26-0"><rect x="477" y="191" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-217-252-26-0"><rect x="477" y="217" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-243-252-26-0"><rect x="477" y="243" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-269-252-26-0"><rect x="477" y="269" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-295-252-26-0"><rect x="477" y="295" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-321-252-26-0"><rect x="477" y="321" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-347-252-26-0"><rect x="477" y="347" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-373-252-26-0"><rect x="477" y="373" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-399-252-26-0"><rect x="477" y="399" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-425-252-26-0"><rect x="477" y="425" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-451-252-26-0"><rect x="477" y="451" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-577-252-26-0"><rect x="477" y="577" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-603-252-26-0"><rect x="477" y="603" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-629-252-26-0"><rect x="477" y="629" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-655-252-26-0"><rect x="477" y="655" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-681-252-26-0"><rect x="477" y="681" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-707-252-26-0"><rect x="477" y="707" width="252" height="26"/></clipPath><clipPath id="mx-clip-477-733-252-26-0"><rect x="477" y="733" width="252" height="26"/></clipPath><clipPath id="mx-clip-837-113-232-26-0"><rect x="837" y="113" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-139-232-26-0"><rect x="837" y="139" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-165-232-26-0"><rect x="837" y="165" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-191-232-26-0"><rect x="837" y="191" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-217-232-26-0"><rect x="837" y="217" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-243-232-26-0"><rect x="837" y="243" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-269-232-26-0"><rect x="837" y="269" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-295-232-26-0"><rect x="837" y="295" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-321-232-26-0"><rect x="837" y="321" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-347-232-26-0"><rect x="837" y="347" width="232" height="26"/></clipPath><clipPath id="mx-clip-837-373-232-26-0"><rect x="837" y="373" width="232" height="26"/></clipPath><clipPath id="mx-clip-1177-61-222-26-0"><rect x="1177" y="61" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-87-222-26-0"><rect x="1177" y="87" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-113-222-26-0"><rect x="1177" y="113" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-139-222-26-0"><rect x="1177" y="139" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-165-222-26-0"><rect x="1177" y="165" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-191-222-26-0"><rect x="1177" y="191" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-217-222-26-0"><rect x="1177" y="217" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-243-222-26-0"><rect x="1177" y="243" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-269-222-26-0"><rect x="1177" y="269" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-295-222-26-0"><rect x="1177" y="295" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-321-222-26-0"><rect x="1177" y="321" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-347-222-26-0"><rect x="1177" y="347" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-373-222-26-0"><rect x="1177" y="373" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-399-222-26-0"><rect x="1177" y="399" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-425-222-26-0"><rect x="1177" y="425" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-451-222-26-0"><rect x="1177" y="451" width="222" height="26"/></clipPath><clipPath id="mx-clip-1177-477-222-26-0"><rect x="1177" y="477" width="222" height="26"/></clipPath><clipPath id="mx-clip-1527-126-242-26-0"><rect x="1527" y="126" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-152-242-26-0"><rect x="1527" y="152" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-178-242-26-0"><rect x="1527" y="178" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-204-242-26-0"><rect x="1527" y="204" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-230-242-26-0"><rect x="1527" y="230" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-256-242-26-0"><rect x="1527" y="256" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-282-242-26-0"><rect x="1527" y="282" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-308-242-26-0"><rect x="1527" y="308" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-334-242-26-0"><rect x="1527" y="334" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-360-242-26-0"><rect x="1527" y="360" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-386-242-26-0"><rect x="1527" y="386" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-412-242-26-0"><rect x="1527" y="412" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-438-242-26-0"><rect x="1527" y="438" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-464-242-26-0"><rect x="1527" y="464" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-490-242-26-0"><rect x="1527" y="490" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-516-242-26-0"><rect x="1527" y="516" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-542-242-26-0"><rect x="1527" y="542" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-568-242-26-0"><rect x="1527" y="568" width="242" height="26"/></clipPath><clipPath id="mx-clip-1527-594-242-26-0"><rect x="1527" y="594" width="242" height="26"/></clipPath></defs><path d="M 3 56 L 3 30 L 333 30 L 333 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 3 56 L 3 758 L 333 758 L 333 56" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 3 56 L 333 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="167.5" y="47.5">web_client</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-61-322-26-0)" font-size="12px"><text x="8.5" y="73.5">unsigned long long id</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-87-322-26-0)" font-size="12px"><text x="8.5" y="99.5">WEB_CLIENT_FLAGS flags</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-113-322-26-0)" font-size="12px"><text x="8.5" y="125.5">WEB_CLIENT_MODE mode</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-139-322-26-0)" font-size="12px"><text x="8.5" y="151.5">WEB_CLIENT_ACL acl</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-165-322-26-0)" font-size="12px"><text x="8.5" y="177.5">size_t header_parse_tries</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-191-322-26-0)" font-size="12px"><text x="8.5" y="203.5">size_t header_parse_last_size</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-217-322-26-0)" font-size="12px"><text x="8.5" y="229.5">int tcp_cork</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-243-322-26-0)" font-size="12px"><text x="8.5" y="255.5">int ifd</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-269-322-26-0)" font-size="12px"><text x="8.5" y="281.5">int ofd</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-295-322-26-0)" font-size="12px"><text x="8.5" y="307.5">char client_ip[NI_MAXHOST+1}</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-321-322-26-0)" font-size="12px"><text x="8.5" y="333.5">char client_port[NI_MAXSERV+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-347-322-26-0)" font-size="12px"><text x="8.5" y="359.5">char decoded_url[NETDATA_WEB_REQUEST_URL_SIZE+1</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-373-322-26-0)" font-size="12px"><text x="8.5" y="385.5">char last_url[NETDATA_WEB_REQUEST_URL_SIZE+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-399-322-26-0)" font-size="12px"><text x="8.5" y="411.5">struct timeval tv_in, tv_ready</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-425-322-26-0)" font-size="12px"><text x="8.5" y="437.5">char cookie1[NETDATA_WEB_REQUEST_COOKIE_SIZE+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-451-322-26-0)" font-size="12px"><text x="8.5" y="463.5">char cookie2[NETDATA_WEB_REQUEST_COOKIE_SIZE+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-477-322-26-0)" font-size="12px"><text x="8.5" y="489.5">char origin[NETDATA_WEB_REQUEST_ORIGIN_HEADER_SIZE+1]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-503-322-26-0)" font-size="12px"><text x="8.5" y="515.5">char *user_agent</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-529-322-26-0)" font-size="12px"><text x="8.5" y="541.5">struct response response</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-555-322-26-0)" font-size="12px"><text x="8.5" y="567.5">size_t stats_received_bytes</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-581-322-26-0)" font-size="12px"><text x="8.5" y="593.5">size_t stats_sent_bytes</text></g><path d="M 333 615 L 353 615 L 353 10 L 168 10 L 168 23.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 168 28.88 L 164.5 21.88 L 168 23.63 L 171.5 21.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-607-322-26-0)" font-size="12px"><text x="8.5" y="619.5">struct web_client *prev</text></g><path d="M 333 641 L 353 641 L 353 10 L 168 10 L 168 23.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 168 28.88 L 164.5 21.88 L 168 23.63 L 171.5 21.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(304.5,299.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="102" height="12" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;background-color:#ffffff;">double linked list of</div></div></foreignObject><text x="51" y="12" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">double linked list of</text></switch></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-633-322-26-0)" font-size="12px"><text x="8.5" y="645.5">struct web_client *next</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-659-322-26-0)" font-size="12px"><text x="8.5" y="671.5">netdata_thread_t thread</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-685-322-26-0)" font-size="12px"><text x="8.5" y="697.5">volatile int running</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-711-322-26-0)" font-size="12px"><text x="8.5" y="723.5">size_t pollinfo_slot</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-7-737-322-26-0)" font-size="12px"><text x="8.5" y="749.5">size_t pollinfo_filecopy_slot</text></g><path d="M 473 160 L 473 134 L 733 134 L 733 160" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 160 L 473 472 L 733 472 L 733 160" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 160 L 733 160" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="602.5" y="151.5">response</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-165-252-26-0)" font-size="12px"><text x="478.5" y="177.5">BUFFER *header</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-191-252-26-0)" font-size="12px"><text x="478.5" y="203.5">BUFFER *header_output</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-217-252-26-0)" font-size="12px"><text x="478.5" y="229.5">BUFFER *data</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-243-252-26-0)" font-size="12px"><text x="478.5" y="255.5">int code</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-269-252-26-0)" font-size="12px"><text x="478.5" y="281.5">size_t rlen</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-295-252-26-0)" font-size="12px"><text x="478.5" y="307.5">size_t sent</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-321-252-26-0)" font-size="12px"><text x="478.5" y="333.5">int zoutput</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-347-252-26-0)" font-size="12px"><text x="478.5" y="359.5">z_stream zstream</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-373-252-26-0)" font-size="12px"><text x="478.5" y="385.5">Bytef zbuffer[NETDATA_WEB_RESPONSE_ZLIB_CHUNK_SIZE]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-399-252-26-0)" font-size="12px"><text x="478.5" y="411.5">size_t zsent</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-425-252-26-0)" font-size="12px"><text x="478.5" y="437.5">size_t zhave</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-451-252-26-0)" font-size="12px"><text x="478.5" y="463.5">unsigned int zinitialized</text></g><path d="M 333 537 L 403 537 L 403 114 L 603 114 L 603 127.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 603 132.88 L 599.5 125.88 L 603 127.63 L 606.5 125.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 572 L 473 546 L 733 546 L 733 572" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 572 L 473 754 L 733 754 L 733 572" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 473 572 L 733 572" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="602.5" y="563.5">clients_cache</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-577-252-26-0)" font-size="12px"><text x="478.5" y="589.5">pid_t pid</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-603-252-26-0)" font-size="12px"><text x="478.5" y="615.5">struct web_client *used</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-629-252-26-0)" font-size="12px"><text x="478.5" y="641.5">size_t used_count</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-655-252-26-0)" font-size="12px"><text x="478.5" y="667.5">struct web_client *avail</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-681-252-26-0)" font-size="12px"><text x="478.5" y="693.5">size_t avail_count</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-707-252-26-0)" font-size="12px"><text x="478.5" y="719.5">size_t reused</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-477-733-252-26-0)" font-size="12px"><text x="478.5" y="745.5">size_t allocated</text></g><path d="M 733 611 L 753 611 L 753 10 L 168 10 L 168 23.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 168 28.88 L 164.5 21.88 L 168 23.63 L 171.5 21.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(724.5,12.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="57" height="11" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 11px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;background-color:#ffffff;">linked list of</div></div></foreignObject><text x="29" y="11" fill="#000000" text-anchor="middle" font-size="11px" font-family="Helvetica">linked list of</text></switch></g><path d="M 733 663 L 753 663 L 753 10 L 168 10 L 168 23.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 168 28.88 L 164.5 21.88 L 168 23.63 L 171.5 21.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(724.5,38.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="57" height="11" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 11px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;background-color:#ffffff;">linked list of</div></div></foreignObject><text x="29" y="11" fill="#000000" text-anchor="middle" font-size="11px" font-family="Helvetica">linked list of</text></switch></g><path d="M 833 108 L 833 82 L 1073 82 L 1073 108" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 833 108 L 833 394 L 1073 394 L 1073 108" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 833 108 L 1073 108" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="952.5" y="99.5">listen_sockets</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-113-232-26-0)" font-size="12px"><text x="838.5" y="125.5">struct config *config</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-139-232-26-0)" font-size="12px"><text x="838.5" y="151.5">const char *config_section</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-165-232-26-0)" font-size="12px"><text x="838.5" y="177.5">const char *default_bind_to</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-191-232-26-0)" font-size="12px"><text x="838.5" y="203.5">uint16_t default_port</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-217-232-26-0)" font-size="12px"><text x="838.5" y="229.5">int backlog</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-243-232-26-0)" font-size="12px"><text x="838.5" y="255.5">size_t opened</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-269-232-26-0)" font-size="12px"><text x="838.5" y="281.5">size_t failed</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-295-232-26-0)" font-size="12px"><text x="838.5" y="307.5">int fds[MAX_LISTEN_FDS]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-321-232-26-0)" font-size="12px"><text x="838.5" y="333.5">int *fds_names[MAX_LISTEN_FDS]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-347-232-26-0)" font-size="12px"><text x="838.5" y="359.5">int fds_types[MAX_LISTEN_FDS]</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-837-373-232-26-0)" font-size="12px"><text x="838.5" y="385.5">int fds_families[MAX_LISTEN_FDS]</text></g><path d="M 1173 56 L 1173 30 L 1403 30 L 1403 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1173 56 L 1173 498 L 1403 498 L 1403 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1173 56 L 1403 56" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="1287.5" y="47.5">POLLINFO</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-61-222-26-0)" font-size="12px"><text x="1178.5" y="73.5">POLLJOB *p</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-87-222-26-0)" font-size="12px"><text x="1178.5" y="99.5">size_t slot</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-113-222-26-0)" font-size="12px"><text x="1178.5" y="125.5">int fd</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-139-222-26-0)" font-size="12px"><text x="1178.5" y="151.5">int socktype</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-165-222-26-0)" font-size="12px"><text x="1178.5" y="177.5">char *client_ip</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-191-222-26-0)" font-size="12px"><text x="1178.5" y="203.5">char *client_port</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-217-222-26-0)" font-size="12px"><text x="1178.5" y="229.5">time_t connected_t</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-243-222-26-0)" font-size="12px"><text x="1178.5" y="255.5">time_t last_received_t</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-269-222-26-0)" font-size="12px"><text x="1178.5" y="281.5">time_t last_sent_t</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-295-222-26-0)" font-size="12px"><text x="1178.5" y="307.5">size_t recv_count</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-321-222-26-0)" font-size="12px"><text x="1178.5" y="333.5">size_t send_count</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-347-222-26-0)" font-size="12px"><text x="1178.5" y="359.5">uint32_t flags</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-373-222-26-0)" font-size="12px"><text x="1178.5" y="385.5">void (*del_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-399-222-26-0)" font-size="12px"><text x="1178.5" y="411.5">int (*rcv_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-425-222-26-0)" font-size="12px"><text x="1178.5" y="437.5">int (*snd_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-451-222-26-0)" font-size="12px"><text x="1178.5" y="463.5">void *data</text></g><path d="M 1403 485 L 1423 485 L 1423 10 L 1279 10 L 1279 22.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1279 27.88 L 1275.5 20.88 L 1279 22.63 L 1282.5 20.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1177-477-222-26-0)" font-size="12px"><text x="1178.5" y="489.5">struct pollinfo *next</text></g><path d="M 1523 121 L 1523 95 L 1773 95 L 1773 121" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1523 121 L 1523 615 L 1773 615 L 1773 121" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1523 121 L 1773 121" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="12px"><text x="1647.5" y="112.5">POLLJOB</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-126-242-26-0)" font-size="12px"><text x="1528.5" y="138.5">size_t slots</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-152-242-26-0)" font-size="12px"><text x="1528.5" y="164.5">size_t used</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-178-242-26-0)" font-size="12px"><text x="1528.5" y="190.5">size_t min</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-204-242-26-0)" font-size="12px"><text x="1528.5" y="216.5">size_t max</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-230-242-26-0)" font-size="12px"><text x="1528.5" y="242.5">size_t limit</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-256-242-26-0)" font-size="12px"><text x="1528.5" y="268.5">time_t complete_request_timeout</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-282-242-26-0)" font-size="12px"><text x="1528.5" y="294.5">time_t idle_timeout</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-308-242-26-0)" font-size="12px"><text x="1528.5" y="320.5">time_t check_every</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-334-242-26-0)" font-size="12px"><text x="1528.5" y="346.5">time_t timer_milliseconds</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-360-242-26-0)" font-size="12px"><text x="1528.5" y="372.5">void *timer_data</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-386-242-26-0)" font-size="12px"><text x="1528.5" y="398.5">struct run *fds</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-412-242-26-0)" font-size="12px"><text x="1528.5" y="424.5">struct pollinfo *inf</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-438-242-26-0)" font-size="12px"><text x="1528.5" y="450.5">struct pollinfo *first_free</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-464-242-26-0)" font-size="12px"><text x="1528.5" y="476.5">SIMPLE_PATTERN *access_list</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-490-242-26-0)" font-size="12px"><text x="1528.5" y="502.5">void *(*add_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-516-242-26-0)" font-size="12px"><text x="1528.5" y="528.5">void (*dell_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-542-242-26-0)" font-size="12px"><text x="1528.5" y="554.5">int (*rcv_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-568-242-26-0)" font-size="12px"><text x="1528.5" y="580.5">int (*snd_callback)</text></g><g fill="#000000" font-family="Helvetica" clip-path="url(#mx-clip-1527-594-242-26-0)" font-size="12px"><text x="1528.5" y="606.5">void (*tmr_callback)</text></g><path d="M 1403 69 L 1648 69 L 1648 88.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1648 93.88 L 1644.5 86.88 L 1648 88.63 L 1651.5 86.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1773 420 L 1793 420 L 1793 10 L 1279 10 L 1279 22.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1279 27.88 L 1275.5 20.88 L 1279 22.63 L 1282.5 20.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1773 446 L 1793 446 L 1793 10 L 1278 10 L 1278 22.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 1278 27.88 L 1274.5 20.88 L 1278 22.63 L 1281.5 20.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/></svg>
\ No newline at end of file
packaging/cmake/config.cmake.h.in
+2
@@ -73,6 +73,7 @@
73
#cmakedefine HAVE_ARC4RANDOM_UNIFORM
74
#cmakedefine HAVE_RAND_S
75
#cmakedefine HAVE_GETRANDOM
76
+#cmakedefine HAVE_SYSINFO
77
78
#cmakedefine HAVE_BACKTRACE
79
#cmakedefine HAVE_CLOSE_RANGE
@@ -95,6 +96,7 @@
96
#cmakedefine STRERROR_R_CHAR_P
97
#cmakedefine HAVE_C__GENERIC
98
#cmakedefine HAVE_C_MALLOPT
99
+#cmakedefine HAVE_C_MALLOC_TRIM
100
#cmakedefine HAVE_SETNS
101
#cmakedefine HAVE_STRNDUP
102
#cmakedefine SSL_HAS_PENDING
src/aclk/aclk.c
+4
-4
@@ -201,7 +201,7 @@ static int wait_till_agent_claim_ready()
201
// We trap the impossible NULL here to keep the linter happy without using a fatal() in the code.
202
const char *cloud_base_url = cloud_config_url_get();
203
if (cloud_base_url == NULL) {
204
- netdata_log_error("Do not move the \"url\" out of post_conf_load!!");
204
+ netdata_log_error("Do not move the \"url\" out of netdata_conf_section_global_run_as_user!!");
205
return 1;
206
}
207
@@ -559,7 +559,7 @@ static int aclk_attempt_to_connect(mqtt_wss_client client)
559
while (service_running(SERVICE_ACLK)) {
560
aclk_cloud_base_url = cloud_config_url_get();
561
if (aclk_cloud_base_url == NULL) {
562
- error_report("Do not move the \"url\" out of post_conf_load!!");
562
+ error_report("Do not move the \"url\" out of netdata_conf_section_global_run_as_user!!");
563
aclk_status = ACLK_STATUS_NO_CLOUD_URL;
564
return -1;
565
}
@@ -868,7 +868,7 @@ void aclk_host_state_update(RRDHOST *host, int cmd, int queryable)
868
create_query->data.bin_payload.topic = ACLK_TOPICID_CREATE_NODE;
869
create_query->data.bin_payload.msg_name = "CreateNodeInstance";
870
nd_log(NDLS_DAEMON, NDLP_DEBUG,
871
- "Registering host=%s, hops=%u", host->machine_guid, host->system_info->hops);
871
+ "Registering host=%s, hops=%d", host->machine_guid, host->system_info->hops);
872
873
aclk_execute_query(create_query);
874
return;
@@ -892,7 +892,7 @@ void aclk_host_state_update(RRDHOST *host, int cmd, int queryable)
892
query->data.bin_payload.payload = generate_node_instance_connection(&query->data.bin_payload.size, &node_state_update);
893
894
nd_log(NDLS_DAEMON, NDLP_DEBUG,
895
- "Queuing status update for node=%s, live=%d, hops=%u, queryable=%d",
895
+ "Queuing status update for node=%s, live=%d, hops=%d, queryable=%d",
896
(char*)node_state_update.node_id, cmd, host->system_info->hops, queryable);
897
freez((void*)node_state_update.node_id);
898
query->data.bin_payload.msg_name = "UpdateNodeInstanceConnection";
src/aclk/aclk_capas.c
+4
-4
@@ -2,7 +2,7 @@
2
3
#include "aclk_capas.h"
4
5
-#include "ml/ml.h"
5
+#include "ml/ml_public.h"
6
7
#define HTTP_API_V2_VERSION 7
8
@@ -31,14 +31,14 @@ const struct capability *aclk_get_agent_capas()
31
agent_capabilities[3].version = metric_correlations_version;
32
agent_capabilities[3].enabled = 1;
33
34
- agent_capabilities[7].enabled = localhost->health.health_enabled;
34
+ agent_capabilities[7].enabled = localhost->health.enabled;
35
36
return agent_capabilities;
37
}
38
39
struct capability *aclk_get_node_instance_capas(RRDHOST *host)
40
{
41
- bool functions = (host == localhost || (host->receiver && stream_has_capability(host->receiver, STREAM_CAP_FUNCTIONS)));
41
+ bool functions = (host == localhost || receiver_has_capability(host, STREAM_CAP_FUNCTIONS));
42
bool dyncfg = (host == localhost || dyncfg_available_for_rrdhost(host));
43
44
struct capability ni_caps[] = {
@@ -48,7 +48,7 @@ struct capability *aclk_get_node_instance_capas(RRDHOST *host)
48
{ .name = "ctx", .version = 1, .enabled = 1 },
49
{ .name = "funcs", .version = functions ? 1 : 0, .enabled = functions ? 1 : 0 },
50
{ .name = "http_api_v2", .version = HTTP_API_V2_VERSION, .enabled = 1 },
51
- { .name = "health", .version = 2, .enabled = host->health.health_enabled },
51
+ { .name = "health", .version = 2, .enabled = host->health.enabled},
52
{ .name = "req_cancel", .version = 1, .enabled = 1 },
53
{ .name = "dyncfg", .version = 2, .enabled = dyncfg },
54
{ .name = NULL, .version = 0, .enabled = 0 }
src/aclk/https_client.c
+1
-1
@@ -6,7 +6,7 @@
6
7
#include "aclk_util.h"
8
9
-#include "daemon/global_statistics.h"
9
+#include "daemon/telemetry/telemetry.h"
10
11
static const char *http_req_type_to_str(http_req_type_t req) {
12
switch (req) {
src/claim/claim.c
+1
-1
@@ -189,7 +189,7 @@ CLOUD_STATUS claim_reload_and_wait_online(void) {
189
cloud_conf_load(0);
190
bool claimed = load_claiming_state();
191
registry_update_cloud_base_url();
192
- rrdpush_sender_send_claimed_id(localhost);
192
+ stream_sender_send_claimed_id(localhost);
193
nd_log_limits_reset();
194
195
CLOUD_STATUS status = cloud_status();
src/claim/cloud-status.c
+2
-2
@@ -30,8 +30,8 @@ CLOUD_STATUS cloud_status(void) {
30
return CLOUD_STATUS_ONLINE;
31
32
if(localhost->sender &&
33
- rrdhost_flag_check(localhost, RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS) &&
34
- stream_has_capability(localhost->sender, STREAM_CAP_NODE_ID) &&
33
+ rrdhost_flag_check(localhost, RRDHOST_FLAG_STREAM_SENDER_READY_4_METRICS) &&
34
+ stream_sender_has_capabilities(localhost, STREAM_CAP_NODE_ID) &&
35
!UUIDiszero(localhost->node_id) &&
36
!UUIDiszero(localhost->aclk.claim_id_of_parent))
37
return CLOUD_STATUS_INDIRECT;
src/collectors/apps.plugin/apps_pid.c
+1
-1
@@ -53,7 +53,7 @@ size_t all_pids_count(void) {
53
}
54
55
void apps_pids_init(void) {
56
- pids.all_pids.aral = aral_create("pid_stat", sizeof(struct pid_stat), 1, 65536, NULL, NULL, NULL, false, true);
56
+ pids.all_pids.aral = aral_create("pid_stat", sizeof(struct pid_stat), 1, 0, NULL, NULL, NULL, false, true);
57
simple_hashtable_init_PID(&pids.all_pids.ht, 1024);
58
}
59
src/collectors/ebpf.plugin/ebpf.c
+1
-1
@@ -739,7 +739,7 @@ ARAL *ebpf_allocate_pid_aral(char *name, size_t size)
739
}
740
741
return aral_create(name, size,
742
- 0, max_elements,
742
+ 0, 0,
743
NULL, NULL, NULL, false, false);
744
}
745
src/collectors/statsd.plugin/statsd.c
+2
-2
@@ -2654,7 +2654,7 @@ void *statsd_main(void *ptr) {
2654
RRDSET *st_pcharts = NULL;
2655
RRDDIM *rd_pcharts = NULL;
2656
2657
- if(global_statistics_enabled) {
2657
+ if(telemetry_enabled) {
2658
st_metrics = rrdset_create_localhost(
2659
"netdata",
2660
"statsd_metrics",
@@ -2851,7 +2851,7 @@ void *statsd_main(void *ptr) {
2851
if(unlikely(!service_running(SERVICE_COLLECTORS)))
2852
break;
2853
2854
- if(global_statistics_enabled) {
2854
+ if(telemetry_enabled) {
2855
rrddim_set_by_pointer(st_metrics, rd_metrics_gauge, (collected_number)statsd.gauges.metrics);
2856
rrddim_set_by_pointer(st_metrics, rd_metrics_counter, (collected_number)statsd.counters.metrics);
2857
rrddim_set_by_pointer(st_metrics, rd_metrics_timer, (collected_number)statsd.timers.metrics);
src/daemon/analytics.c
+2
-5
@@ -377,10 +377,7 @@ void analytics_https(void)
377
BUFFER *b = buffer_create(30, NULL);
378
analytics_exporting_connectors_ssl(b);
379
380
- buffer_strcat(b, netdata_ssl_streaming_sender_ctx &&
381
- rrdhost_flag_check(localhost, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED) &&
382
- SSL_connection(&localhost->sender->ssl) ? "streaming|" : "|");
383
-
380
+ buffer_strcat(b, stream_sender_is_connected_with_ssl(localhost) ? "streaming|" : "|");
381
buffer_strcat(b, netdata_ssl_web_server_ctx ? "web" : "");
382
383
analytics_set_data_str(&analytics_data.netdata_config_https_available, (char *)buffer_tostring(b));
@@ -619,7 +616,7 @@ cleanup:
616
*/
617
void set_late_analytics_variables(struct rrdhost_system_info *system_info)
618
{
622
- analytics_set_data(&analytics_data.netdata_config_stream_enabled, stream_conf_send_enabled ? "true" : "false");
619
+ analytics_set_data(&analytics_data.netdata_config_stream_enabled, stream_send.enabled ? "true" : "false");
620
analytics_set_data_str(&analytics_data.netdata_config_memory_mode, (char *)rrd_memory_mode_name(default_rrd_memory_mode));
621
analytics_set_data(&analytics_data.netdata_host_cloud_enabled, "true");
622
src/daemon/common.h
+14
-4
@@ -3,7 +3,12 @@
3
#ifndef NETDATA_COMMON_H
4
#define NETDATA_COMMON_H 1
5
6
+#ifdef __cplusplus
7
+extern "C" {
8
+#endif
9
+
10
#include "libnetdata/libnetdata.h"
11
+#include "config/netdata-conf.h"
12
#include "libuv_workers.h"
13
14
// ----------------------------------------------------------------------------
@@ -11,9 +16,10 @@
16
17
#include "web/api/maps/maps.h"
18
14
-#include "daemon/config/dyncfg.h"
19
+#include "daemon/config/netdata-conf.h"
20
+#include "daemon/dyncfg/dyncfg.h"
21
16
-#include "global_statistics.h"
22
+#include "daemon/telemetry/telemetry.h"
23
24
// health monitoring and alarm notifications
25
#include "health/health.h"
@@ -30,10 +36,10 @@
36
#endif
37
38
// streaming metrics between netdata servers
33
-#include "streaming/rrdpush.h"
39
+#include "streaming/stream.h"
40
41
// anomaly detection
36
-#include "ml/ml.h"
42
+#include "ml/ml_public.h"
43
44
// the netdata registry
45
// the registry is actually an API feature
@@ -94,4 +100,8 @@ long get_netdata_cpus(void);
100
101
void set_environment_for_plugins_and_scripts(void);
102
103
+#ifdef __cplusplus
104
+}
105
+#endif
106
+
107
#endif /* NETDATA_COMMON_H */
src/daemon/config/README.md
+1
-1
@@ -18,7 +18,7 @@ The configuration file uses an INI-style format with `[SECTION]` headers:
18
| [[health]](#health-section-options) | [Health monitoring](/src/health/README.md) |
19
| `[web]` | [Web Server](/src/web/server/README.md) |
20
| `[registry]` | [Registry](/src/registry/README.md) |
21
-| `[global statistics]` | Internal monitoring |
21
+| `[telemetry]` | Internal monitoring |
22
| `[statsd]` | [StatsD plugin](/src/collectors/statsd.plugin/README.md) |
23
| [`[plugins]`](#plugins-section-options) | Data collection Plugins (Collectors) |
24
| [[plugin:NAME]](#per-plugin-configuration) | Individual [Plugins](#per-plugin-configuration) |
src/daemon/config/netdata-conf-backwards-compatibility.c
new
+285
@@ -0,0 +1,285 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "netdata-conf-backwards-compatibility.h"
4
+#include "database/engine/rrdengineapi.h"
5
+
6
+void netdata_conf_backwards_compatibility(void) {
7
+ static bool run = false;
8
+ if(run) return;
9
+ run = true;
10
+
11
+ // move [global] options to the [web] section
12
+
13
+ config_move(CONFIG_SECTION_GLOBAL, "http port listen backlog",
14
+ CONFIG_SECTION_WEB, "listen backlog");
15
+
16
+ config_move(CONFIG_SECTION_GLOBAL, "bind socket to IP",
17
+ CONFIG_SECTION_WEB, "bind to");
18
+
19
+ config_move(CONFIG_SECTION_GLOBAL, "bind to",
20
+ CONFIG_SECTION_WEB, "bind to");
21
+
22
+ config_move(CONFIG_SECTION_GLOBAL, "port",
23
+ CONFIG_SECTION_WEB, "default port");
24
+
25
+ config_move(CONFIG_SECTION_GLOBAL, "default port",
26
+ CONFIG_SECTION_WEB, "default port");
27
+
28
+ config_move(CONFIG_SECTION_GLOBAL, "disconnect idle web clients after seconds",
29
+ CONFIG_SECTION_WEB, "disconnect idle clients after seconds");
30
+
31
+ config_move(CONFIG_SECTION_GLOBAL, "respect web browser do not track policy",
32
+ CONFIG_SECTION_WEB, "respect do not track policy");
33
+
34
+ config_move(CONFIG_SECTION_GLOBAL, "web x-frame-options header",
35
+ CONFIG_SECTION_WEB, "x-frame-options response header");
36
+
37
+ config_move(CONFIG_SECTION_GLOBAL, "enable web responses gzip compression",
38
+ CONFIG_SECTION_WEB, "enable gzip compression");
39
+
40
+ config_move(CONFIG_SECTION_GLOBAL, "web compression strategy",
41
+ CONFIG_SECTION_WEB, "gzip compression strategy");
42
+
43
+ config_move(CONFIG_SECTION_GLOBAL, "web compression level",
44
+ CONFIG_SECTION_WEB, "gzip compression level");
45
+
46
+ config_move(CONFIG_SECTION_GLOBAL, "config directory",
47
+ CONFIG_SECTION_DIRECTORIES, "config");
48
+
49
+ config_move(CONFIG_SECTION_GLOBAL, "stock config directory",
50
+ CONFIG_SECTION_DIRECTORIES, "stock config");
51
+
52
+ config_move(CONFIG_SECTION_GLOBAL, "log directory",
53
+ CONFIG_SECTION_DIRECTORIES, "log");
54
+
55
+ config_move(CONFIG_SECTION_GLOBAL, "web files directory",
56
+ CONFIG_SECTION_DIRECTORIES, "web");
57
+
58
+ config_move(CONFIG_SECTION_GLOBAL, "cache directory",
59
+ CONFIG_SECTION_DIRECTORIES, "cache");
60
+
61
+ config_move(CONFIG_SECTION_GLOBAL, "lib directory",
62
+ CONFIG_SECTION_DIRECTORIES, "lib");
63
+
64
+ config_move(CONFIG_SECTION_GLOBAL, "home directory",
65
+ CONFIG_SECTION_DIRECTORIES, "home");
66
+
67
+ config_move(CONFIG_SECTION_GLOBAL, "lock directory",
68
+ CONFIG_SECTION_DIRECTORIES, "lock");
69
+
70
+ config_move(CONFIG_SECTION_GLOBAL, "plugins directory",
71
+ CONFIG_SECTION_DIRECTORIES, "plugins");
72
+
73
+ config_move(CONFIG_SECTION_HEALTH, "health configuration directory",
74
+ CONFIG_SECTION_DIRECTORIES, "health config");
75
+
76
+ config_move(CONFIG_SECTION_HEALTH, "stock health configuration directory",
77
+ CONFIG_SECTION_DIRECTORIES, "stock health config");
78
+
79
+ config_move(CONFIG_SECTION_REGISTRY, "registry db directory",
80
+ CONFIG_SECTION_DIRECTORIES, "registry");
81
+
82
+ config_move(CONFIG_SECTION_GLOBAL, "debug log",
83
+ CONFIG_SECTION_LOGS, "debug");
84
+
85
+ config_move(CONFIG_SECTION_GLOBAL, "error log",
86
+ CONFIG_SECTION_LOGS, "error");
87
+
88
+ config_move(CONFIG_SECTION_GLOBAL, "access log",
89
+ CONFIG_SECTION_LOGS, "access");
90
+
91
+ config_move(CONFIG_SECTION_GLOBAL, "facility log",
92
+ CONFIG_SECTION_LOGS, "facility");
93
+
94
+ config_move(CONFIG_SECTION_GLOBAL, "errors flood protection period",
95
+ CONFIG_SECTION_LOGS, "errors flood protection period");
96
+
97
+ config_move(CONFIG_SECTION_GLOBAL, "errors to trigger flood protection",
98
+ CONFIG_SECTION_LOGS, "errors to trigger flood protection");
99
+
100
+ config_move(CONFIG_SECTION_GLOBAL, "debug flags",
101
+ CONFIG_SECTION_LOGS, "debug flags");
102
+
103
+ config_move(CONFIG_SECTION_GLOBAL, "TZ environment variable",
104
+ CONFIG_SECTION_ENV_VARS, "TZ");
105
+
106
+ config_move(CONFIG_SECTION_PLUGINS, "PATH environment variable",
107
+ CONFIG_SECTION_ENV_VARS, "PATH");
108
+
109
+ config_move(CONFIG_SECTION_PLUGINS, "PYTHONPATH environment variable",
110
+ CONFIG_SECTION_ENV_VARS, "PYTHONPATH");
111
+
112
+ config_move(CONFIG_SECTION_STATSD, "enabled",
113
+ CONFIG_SECTION_PLUGINS, "statsd");
114
+
115
+ config_move(CONFIG_SECTION_GLOBAL, "memory mode",
116
+ CONFIG_SECTION_DB, "db");
117
+
118
+ config_move(CONFIG_SECTION_DB, "mode",
119
+ CONFIG_SECTION_DB, "db");
120
+
121
+ config_move(CONFIG_SECTION_GLOBAL, "history",
122
+ CONFIG_SECTION_DB, "retention");
123
+
124
+ config_move(CONFIG_SECTION_GLOBAL, "update every",
125
+ CONFIG_SECTION_DB, "update every");
126
+
127
+ config_move(CONFIG_SECTION_GLOBAL, "page cache size",
128
+ CONFIG_SECTION_DB, "dbengine page cache size");
129
+
130
+ config_move(CONFIG_SECTION_DB, "dbengine page cache size MB",
131
+ CONFIG_SECTION_DB, "dbengine page cache size");
132
+
133
+ config_move(CONFIG_SECTION_DB, "dbengine extent cache size MB",
134
+ CONFIG_SECTION_DB, "dbengine extent cache size");
135
+
136
+ config_move(CONFIG_SECTION_DB, "page cache size",
137
+ CONFIG_SECTION_DB, "dbengine page cache size MB");
138
+
139
+ config_move(CONFIG_SECTION_GLOBAL, "page cache uses malloc",
140
+ CONFIG_SECTION_DB, "dbengine page cache with malloc");
141
+
142
+ config_move(CONFIG_SECTION_DB, "page cache with malloc",
143
+ CONFIG_SECTION_DB, "dbengine page cache with malloc");
144
+
145
+ config_move(CONFIG_SECTION_GLOBAL, "memory deduplication (ksm)",
146
+ CONFIG_SECTION_DB, "memory deduplication (ksm)");
147
+
148
+ config_move(CONFIG_SECTION_GLOBAL, "dbengine page fetch timeout",
149
+ CONFIG_SECTION_DB, "dbengine page fetch timeout secs");
150
+
151
+ config_move(CONFIG_SECTION_GLOBAL, "dbengine page fetch retries",
152
+ CONFIG_SECTION_DB, "dbengine page fetch retries");
153
+
154
+ config_move(CONFIG_SECTION_GLOBAL, "dbengine extent pages",
155
+ CONFIG_SECTION_DB, "dbengine pages per extent");
156
+
157
+ config_move(CONFIG_SECTION_GLOBAL, "cleanup obsolete charts after seconds",
158
+ CONFIG_SECTION_DB, "cleanup obsolete charts after");
159
+
160
+ config_move(CONFIG_SECTION_DB, "cleanup obsolete charts after secs",
161
+ CONFIG_SECTION_DB, "cleanup obsolete charts after");
162
+
163
+ config_move(CONFIG_SECTION_GLOBAL, "gap when lost iterations above",
164
+ CONFIG_SECTION_DB, "gap when lost iterations above");
165
+
166
+ config_move(CONFIG_SECTION_GLOBAL, "cleanup orphan hosts after seconds",
167
+ CONFIG_SECTION_DB, "cleanup orphan hosts after");
168
+
169
+ config_move(CONFIG_SECTION_DB, "cleanup orphan hosts after secs",
170
+ CONFIG_SECTION_DB, "cleanup orphan hosts after");
171
+
172
+ config_move(CONFIG_SECTION_DB, "cleanup ephemeral hosts after secs",
173
+ CONFIG_SECTION_DB, "cleanup ephemeral hosts after");
174
+
175
+ config_move(CONFIG_SECTION_DB, "seconds to replicate",
176
+ CONFIG_SECTION_DB, "replication period");
177
+
178
+ config_move(CONFIG_SECTION_DB, "seconds per replication step",
179
+ CONFIG_SECTION_DB, "replication step");
180
+
181
+ config_move(CONFIG_SECTION_GLOBAL, "enable zero metrics",
182
+ CONFIG_SECTION_DB, "enable zero metrics");
183
+
184
+ config_move("global statistics", "update every",
185
+ CONFIG_SECTION_TELEMETRY, "update every");
186
+
187
+ config_move(CONFIG_SECTION_PLUGINS, "netdata monitoring",
188
+ CONFIG_SECTION_PLUGINS, "netdata telemetry");
189
+
190
+ config_move(CONFIG_SECTION_PLUGINS, "netdata monitoring extended",
191
+ CONFIG_SECTION_TELEMETRY, "extended telemetry");
192
+
193
+
194
+ // ----------------------------------------------------------------------------------------------------------------
195
+
196
+ bool found_old_config = false;
197
+
198
+ if(config_move(CONFIG_SECTION_GLOBAL, "dbengine disk space",
199
+ CONFIG_SECTION_DB, "dbengine tier 0 retention size") != -1)
200
+ found_old_config = true;
201
+
202
+ if(config_move(CONFIG_SECTION_GLOBAL, "dbengine multihost disk space",
203
+ CONFIG_SECTION_DB, "dbengine tier 0 retention size") != -1)
204
+ found_old_config = true;
205
+
206
+ if(config_move(CONFIG_SECTION_DB, "dbengine disk space MB",
207
+ CONFIG_SECTION_DB, "dbengine tier 0 retention size") != -1)
208
+ found_old_config = true;
209
+
210
+ for(size_t tier = 0; tier < RRD_STORAGE_TIERS ;tier++) {
211
+ char old_config[128], new_config[128];
212
+
213
+ snprintfz(old_config, sizeof(old_config), "dbengine tier %zu retention days", tier);
214
+ snprintfz(new_config, sizeof(new_config), "dbengine tier %zu retention time", tier);
215
+ config_move(CONFIG_SECTION_DB, old_config,
216
+ CONFIG_SECTION_DB, new_config);
217
+
218
+ if(tier == 0)
219
+ snprintfz(old_config, sizeof(old_config), "dbengine multihost disk space MB");
220
+ else
221
+ snprintfz(old_config, sizeof(old_config), "dbengine tier %zu multihost disk space MB", tier);
222
+ snprintfz(new_config, sizeof(new_config), "dbengine tier %zu retention size", tier);
223
+ if(config_move(CONFIG_SECTION_DB, old_config,
224
+ CONFIG_SECTION_DB, new_config) != -1 && tier == 0)
225
+ found_old_config = true;
226
+
227
+ snprintfz(old_config, sizeof(old_config), "dbengine tier %zu disk space MB", tier);
228
+ snprintfz(new_config, sizeof(new_config), "dbengine tier %zu retention size", tier);
229
+ if(config_move(CONFIG_SECTION_DB, old_config,
230
+ CONFIG_SECTION_DB, new_config) != -1 && tier == 0)
231
+ found_old_config = true;
232
+ }
233
+
234
+ legacy_multihost_db_space = found_old_config;
235
+
236
+ // ----------------------------------------------------------------------------------------------------------------
237
+
238
+ config_move(CONFIG_SECTION_LOGS, "error",
239
+ CONFIG_SECTION_LOGS, "daemon");
240
+
241
+ config_move(CONFIG_SECTION_LOGS, "severity level",
242
+ CONFIG_SECTION_LOGS, "level");
243
+
244
+ config_move(CONFIG_SECTION_LOGS, "errors to trigger flood protection",
245
+ CONFIG_SECTION_LOGS, "logs to trigger flood protection");
246
+
247
+ config_move(CONFIG_SECTION_LOGS, "errors flood protection period",
248
+ CONFIG_SECTION_LOGS, "logs flood protection period");
249
+
250
+ config_move(CONFIG_SECTION_HEALTH, "is ephemeral",
251
+ CONFIG_SECTION_GLOBAL, "is ephemeral node");
252
+
253
+ config_move(CONFIG_SECTION_HEALTH, "has unstable connection",
254
+ CONFIG_SECTION_GLOBAL, "has unstable connection");
255
+
256
+ config_move(CONFIG_SECTION_HEALTH, "run at least every seconds",
257
+ CONFIG_SECTION_HEALTH, "run at least every");
258
+
259
+ config_move(CONFIG_SECTION_HEALTH, "postpone alarms during hibernation for seconds",
260
+ CONFIG_SECTION_HEALTH, "postpone alarms during hibernation for");
261
+
262
+ config_move(CONFIG_SECTION_HEALTH, "health log history",
263
+ CONFIG_SECTION_HEALTH, "health log retention");
264
+
265
+ config_move(CONFIG_SECTION_REGISTRY, "registry expire idle persons days",
266
+ CONFIG_SECTION_REGISTRY, "registry expire idle persons");
267
+
268
+ config_move(CONFIG_SECTION_WEB, "disconnect idle clients after seconds",
269
+ CONFIG_SECTION_WEB, "disconnect idle clients after");
270
+
271
+ config_move(CONFIG_SECTION_WEB, "accept a streaming request every seconds",
272
+ CONFIG_SECTION_WEB, "accept a streaming request every");
273
+
274
+ config_move(CONFIG_SECTION_STATSD, "set charts as obsolete after secs",
275
+ CONFIG_SECTION_STATSD, "set charts as obsolete after");
276
+
277
+ config_move(CONFIG_SECTION_STATSD, "disconnect idle tcp clients after seconds",
278
+ CONFIG_SECTION_STATSD, "disconnect idle tcp clients after");
279
+
280
+ config_move("plugin:idlejitter", "loop time in ms",
281
+ "plugin:idlejitter", "loop time");
282
+
283
+ config_move("plugin:proc:/sys/class/infiniband", "refresh ports state every seconds",
284
+ "plugin:proc:/sys/class/infiniband", "refresh ports state every");
285
+}
src/daemon/config/netdata-conf-backwards-compatibility.h
new
+10
@@ -0,0 +1,10 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_DAEMON_NETDATA_CONF_BACKWARDS_COMPATIBILITY_H
4
+#define NETDATA_DAEMON_NETDATA_CONF_BACKWARDS_COMPATIBILITY_H
5
+
6
+#include "config.h"
7
+
8
+void netdata_conf_backwards_compatibility(void);
9
+
10
+#endif //NETDATA_DAEMON_NETDATA_CONF_BACKWARDS_COMPATIBILITY_H
src/daemon/config/netdata-conf-db.c
new
+418
@@ -0,0 +1,418 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "netdata-conf-db.h"
4
+
5
+int default_rrd_update_every = UPDATE_EVERY;
6
+int default_rrd_history_entries = RRD_DEFAULT_HISTORY_ENTRIES;
7
+
8
+bool dbengine_enabled = false; // will become true if and when dbengine is initialized
9
+size_t storage_tiers = 3;
10
+bool dbengine_use_direct_io = true;
11
+static size_t storage_tiers_grouping_iterations[RRD_STORAGE_TIERS] = {1, 60, 60, 60, 60};
12
+static double storage_tiers_retention_days[RRD_STORAGE_TIERS] = {14, 90, 2 * 365, 2 * 365, 2 * 365};
13
+
14
+time_t rrdset_free_obsolete_time_s = 3600;
15
+time_t rrdhost_free_orphan_time_s = 3600;
16
+time_t rrdhost_free_ephemeral_time_s = 86400;
17
+
18
+size_t get_tier_grouping(size_t tier) {
19
+ if(unlikely(tier >= storage_tiers)) tier = storage_tiers - 1;
20
+
21
+ size_t grouping = 1;
22
+ // first tier is always 1 iteration of whatever update every the chart has
23
+ for(size_t i = 1; i <= tier ;i++)
24
+ grouping *= storage_tiers_grouping_iterations[i];
25
+
26
+ return grouping;
27
+}
28
+
29
+static void netdata_conf_dbengine_pre_logs(void) {
30
+ static bool run = false;
31
+ if(run) return;
32
+ run = true;
33
+
34
+ errno_clear();
35
+
36
+#ifdef ENABLE_DBENGINE
37
+ // this is required for dbegnine to work, so call it here (it is ok, it won't run twice)
38
+ netdata_conf_section_directories();
39
+
40
+ // ------------------------------------------------------------------------
41
+ // get default Database Engine page type
42
+
43
+ const char *page_type = config_get(CONFIG_SECTION_DB, "dbengine page type", "gorilla");
44
+ if (strcmp(page_type, "gorilla") == 0)
45
+ tier_page_type[0] = RRDENG_PAGE_TYPE_GORILLA_32BIT;
46
+ else if (strcmp(page_type, "raw") == 0)
47
+ tier_page_type[0] = RRDENG_PAGE_TYPE_ARRAY_32BIT;
48
+ else {
49
+ tier_page_type[0] = RRDENG_PAGE_TYPE_ARRAY_32BIT;
50
+ netdata_log_error("Invalid dbengine page type ''%s' given. Defaulting to 'raw'.", page_type);
51
+ }
52
+
53
+ // ------------------------------------------------------------------------
54
+ // get default Database Engine page cache size in MiB
55
+
56
+ default_rrdeng_page_cache_mb = (int) config_get_size_mb(CONFIG_SECTION_DB, "dbengine page cache size", default_rrdeng_page_cache_mb);
57
+ default_rrdeng_extent_cache_mb = (int) config_get_size_mb(CONFIG_SECTION_DB, "dbengine extent cache size", default_rrdeng_extent_cache_mb);
58
+ db_engine_journal_check = config_get_boolean(CONFIG_SECTION_DB, "dbengine enable journal integrity check", CONFIG_BOOLEAN_NO);
59
+
60
+ if(default_rrdeng_extent_cache_mb < 0) {
61
+ default_rrdeng_extent_cache_mb = 0;
62
+ config_set_size_mb(CONFIG_SECTION_DB, "dbengine extent cache size", default_rrdeng_extent_cache_mb);
63
+ }
64
+
65
+ if(default_rrdeng_page_cache_mb < RRDENG_MIN_PAGE_CACHE_SIZE_MB) {
66
+ netdata_log_error("Invalid page cache size %d given. Defaulting to %d.", default_rrdeng_page_cache_mb, RRDENG_MIN_PAGE_CACHE_SIZE_MB);
67
+ default_rrdeng_page_cache_mb = RRDENG_MIN_PAGE_CACHE_SIZE_MB;
68
+ config_set_size_mb(CONFIG_SECTION_DB, "dbengine page cache size", default_rrdeng_page_cache_mb);
69
+ }
70
+
71
+ // ------------------------------------------------------------------------
72
+ // get default Database Engine disk space quota in MiB
73
+ //
74
+ // // if (!config_exists(CONFIG_SECTION_DB, "dbengine disk space MB") && !config_exists(CONFIG_SECTION_DB, "dbengine multihost disk space MB"))
75
+ //
76
+ // default_rrdeng_disk_quota_mb = (int) config_get_number(CONFIG_SECTION_DB, "dbengine disk space MB", default_rrdeng_disk_quota_mb);
77
+ // if(default_rrdeng_disk_quota_mb < RRDENG_MIN_DISK_SPACE_MB) {
78
+ // netdata_log_error("Invalid dbengine disk space %d given. Defaulting to %d.", default_rrdeng_disk_quota_mb, RRDENG_MIN_DISK_SPACE_MB);
79
+ // default_rrdeng_disk_quota_mb = RRDENG_MIN_DISK_SPACE_MB;
80
+ // config_set_number(CONFIG_SECTION_DB, "dbengine disk space MB", default_rrdeng_disk_quota_mb);
81
+ // }
82
+ //
83
+ // default_multidb_disk_quota_mb = (int) config_get_number(CONFIG_SECTION_DB, "dbengine multihost disk space MB", compute_multidb_diskspace());
84
+ // if(default_multidb_disk_quota_mb < RRDENG_MIN_DISK_SPACE_MB) {
85
+ // netdata_log_error("Invalid multidb disk space %d given. Defaulting to %d.", default_multidb_disk_quota_mb, default_rrdeng_disk_quota_mb);
86
+ // default_multidb_disk_quota_mb = default_rrdeng_disk_quota_mb;
87
+ // config_set_number(CONFIG_SECTION_DB, "dbengine multihost disk space MB", default_multidb_disk_quota_mb);
88
+ // }
89
+
90
+#else
91
+ if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
92
+ error_report("RRD_MEMORY_MODE_DBENGINE is not supported in this platform. The agent will use db mode 'save' instead.");
93
+ default_rrd_memory_mode = RRD_MEMORY_MODE_RAM;
94
+ }
95
+#endif
96
+}
97
+
98
+#ifdef ENABLE_DBENGINE
99
+struct dbengine_initialization {
100
+ ND_THREAD *thread;
101
+ char path[FILENAME_MAX + 1];
102
+ int disk_space_mb;
103
+ size_t retention_seconds;
104
+ size_t tier;
105
+ int ret;
106
+};
107
+
108
+void *dbengine_tier_init(void *ptr) {
109
+ struct dbengine_initialization *dbi = ptr;
110
+ dbi->ret = rrdeng_init(NULL, dbi->path, dbi->disk_space_mb, dbi->tier, dbi->retention_seconds);
111
+ return ptr;
112
+}
113
+
114
+RRD_BACKFILL get_dbengine_backfill(RRD_BACKFILL backfill)
115
+{
116
+ const char *bf = config_get(
117
+ CONFIG_SECTION_DB,
118
+ "dbengine tier backfill",
119
+ backfill == RRD_BACKFILL_NEW ? "new" :
120
+ backfill == RRD_BACKFILL_FULL ? "full" :
121
+ "none");
122
+
123
+ if (strcmp(bf, "new") == 0)
124
+ backfill = RRD_BACKFILL_NEW;
125
+ else if (strcmp(bf, "full") == 0)
126
+ backfill = RRD_BACKFILL_FULL;
127
+ else if (strcmp(bf, "none") == 0)
128
+ backfill = RRD_BACKFILL_NONE;
129
+ else {
130
+ nd_log(NDLS_DAEMON, NDLP_WARNING, "DBENGINE: unknown backfill value '%s', assuming 'new'", bf);
131
+ config_set(CONFIG_SECTION_DB, "dbengine tier backfill", "new");
132
+ backfill = RRD_BACKFILL_NEW;
133
+ }
134
+ return backfill;
135
+}
136
+#endif
137
+
138
+void netdata_conf_dbengine_init(const char *hostname) {
139
+#ifdef ENABLE_DBENGINE
140
+
141
+ // ----------------------------------------------------------------------------------------------------------------
142
+ // out of memory protection and use all ram for caches
143
+
144
+ dbengine_out_of_memory_protection = 0; // will be calculated below
145
+ OS_SYSTEM_MEMORY sm = os_system_memory(true);
146
+ if(sm.ram_total_bytes && sm.ram_available_bytes && sm.ram_total_bytes > sm.ram_available_bytes) {
147
+ // calculate the default out of memory protection size
148
+ char buf[64];
149
+ size_snprintf(buf, sizeof(buf), sm.ram_total_bytes / 10, "B", false);
150
+ size_parse(buf, &dbengine_out_of_memory_protection, "B");
151
+ }
152
+
153
+ if(dbengine_out_of_memory_protection) {
154
+ dbengine_use_all_ram_for_caches = config_get_boolean(CONFIG_SECTION_DB, "dbengine use all ram for caches", dbengine_use_all_ram_for_caches);
155
+ dbengine_out_of_memory_protection = config_get_size_bytes(CONFIG_SECTION_DB, "dbengine out of memory protection", dbengine_out_of_memory_protection);
156
+
157
+ char buf_total[64], buf_avail[64], buf_oom[64];
158
+ size_snprintf(buf_total, sizeof(buf_total), sm.ram_total_bytes, "B", false);
159
+ size_snprintf(buf_avail, sizeof(buf_avail), sm.ram_available_bytes, "B", false);
160
+ size_snprintf(buf_oom, sizeof(buf_oom), dbengine_out_of_memory_protection, "B", false);
161
+
162
+ nd_log(NDLS_DAEMON, NDLP_NOTICE,
163
+ "DBENGINE Out of Memory Protection. "
164
+ "System Memory Total: %s, Currently Available: %s, Out of Memory Protection: %s, Use All RAM: %s",
165
+ buf_total, buf_avail, buf_oom, dbengine_use_all_ram_for_caches ? "enabled" : "disabled");
166
+ }
167
+ else {
168
+ dbengine_out_of_memory_protection = 0;
169
+ dbengine_use_all_ram_for_caches = false;
170
+
171
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
172
+ "DBENGINE Out of Memory Protection and Use All Ram cannot be enabled. "
173
+ "Failed to detect memory size on this system.");
174
+ }
175
+
176
+ // ----------------------------------------------------------------------------------------------------------------
177
+
178
+ dbengine_use_direct_io = config_get_boolean(CONFIG_SECTION_DB, "dbengine use direct io", dbengine_use_direct_io);
179
+
180
+ unsigned read_num = (unsigned)config_get_number(CONFIG_SECTION_DB, "dbengine pages per extent", DEFAULT_PAGES_PER_EXTENT);
181
+ if (read_num > 0 && read_num <= DEFAULT_PAGES_PER_EXTENT)
182
+ rrdeng_pages_per_extent = read_num;
183
+ else {
184
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
185
+ "Invalid dbengine pages per extent %u given. Using %u.",
186
+ read_num, rrdeng_pages_per_extent);
187
+
188
+ config_set_number(CONFIG_SECTION_DB, "dbengine pages per extent", rrdeng_pages_per_extent);
189
+ }
190
+
191
+ storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
192
+ if(storage_tiers < 1) {
193
+ nd_log(NDLS_DAEMON, NDLP_WARNING, "At least 1 storage tier is required. Assuming 1.");
194
+
195
+ storage_tiers = 1;
196
+ config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
197
+ }
198
+ if(storage_tiers > RRD_STORAGE_TIERS) {
199
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
200
+ "Up to %d storage tier are supported. Assuming %d.",
201
+ RRD_STORAGE_TIERS, RRD_STORAGE_TIERS);
202
+
203
+ storage_tiers = RRD_STORAGE_TIERS;
204
+ config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
205
+ }
206
+
207
+ new_dbengine_defaults =
208
+ (!legacy_multihost_db_space &&
209
+ !config_exists(CONFIG_SECTION_DB, "dbengine tier 1 update every iterations") &&
210
+ !config_exists(CONFIG_SECTION_DB, "dbengine tier 2 update every iterations") &&
211
+ !config_exists(CONFIG_SECTION_DB, "dbengine tier 3 update every iterations") &&
212
+ !config_exists(CONFIG_SECTION_DB, "dbengine tier 4 update every iterations") &&
213
+ !config_exists(CONFIG_SECTION_DB, "dbengine tier 1 retention size") &&
214
+ !config_exists(CONFIG_SECTION_DB, "dbengine tier 2 retention size") &&
215
+ !config_exists(CONFIG_SECTION_DB, "dbengine tier 3 retention size") &&
216
+ !config_exists(CONFIG_SECTION_DB, "dbengine tier 4 retention size"));
217
+
218
+ default_backfill = get_dbengine_backfill(RRD_BACKFILL_NEW);
219
+ char dbengineconfig[200 + 1];
220
+
221
+ size_t grouping_iterations = default_rrd_update_every;
222
+ storage_tiers_grouping_iterations[0] = default_rrd_update_every;
223
+
224
+ for (size_t tier = 1; tier < storage_tiers; tier++) {
225
+ grouping_iterations = storage_tiers_grouping_iterations[tier];
226
+ snprintfz(dbengineconfig, sizeof(dbengineconfig) - 1, "dbengine tier %zu update every iterations", tier);
227
+ grouping_iterations = config_get_number(CONFIG_SECTION_DB, dbengineconfig, grouping_iterations);
228
+ if(grouping_iterations < 2) {
229
+ grouping_iterations = 2;
230
+ config_set_number(CONFIG_SECTION_DB, dbengineconfig, grouping_iterations);
231
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
232
+ "DBENGINE on '%s': 'dbegnine tier %zu update every iterations' cannot be less than 2. Assuming 2.",
233
+ hostname, tier);
234
+ }
235
+ storage_tiers_grouping_iterations[tier] = grouping_iterations;
236
+ }
237
+
238
+ default_multidb_disk_quota_mb = (int) config_get_size_mb(CONFIG_SECTION_DB, "dbengine tier 0 retention size", RRDENG_DEFAULT_TIER_DISK_SPACE_MB);
239
+ if(default_multidb_disk_quota_mb && default_multidb_disk_quota_mb < RRDENG_MIN_DISK_SPACE_MB) {
240
+ netdata_log_error("Invalid disk space %d for tier 0 given. Defaulting to %d.", default_multidb_disk_quota_mb, RRDENG_MIN_DISK_SPACE_MB);
241
+ default_multidb_disk_quota_mb = RRDENG_MIN_DISK_SPACE_MB;
242
+ config_set_size_mb(CONFIG_SECTION_DB, "dbengine tier 0 retention size", default_multidb_disk_quota_mb);
243
+ }
244
+
245
+#ifdef OS_WINDOWS
246
+ // FIXME: for whatever reason joining the initialization threads
247
+ // fails on Windows.
248
+ bool parallel_initialization = false;
249
+#else
250
+ bool parallel_initialization = (storage_tiers <= (size_t)get_netdata_cpus()) ? true : false;
251
+#endif
252
+
253
+ struct dbengine_initialization tiers_init[RRD_STORAGE_TIERS] = {};
254
+
255
+ size_t created_tiers = 0;
256
+ char dbenginepath[FILENAME_MAX + 1];
257
+
258
+ for (size_t tier = 0; tier < storage_tiers; tier++) {
259
+
260
+ if (tier == 0)
261
+ snprintfz(dbenginepath, FILENAME_MAX, "%s/dbengine", netdata_configured_cache_dir);
262
+ else
263
+ snprintfz(dbenginepath, FILENAME_MAX, "%s/dbengine-tier%zu", netdata_configured_cache_dir, tier);
264
+
265
+ int ret = mkdir(dbenginepath, 0775);
266
+ if (ret != 0 && errno != EEXIST) {
267
+ nd_log(NDLS_DAEMON, NDLP_CRIT, "DBENGINE on '%s': cannot create directory '%s'", hostname, dbenginepath);
268
+ continue;
269
+ }
270
+
271
+ int disk_space_mb = tier ? RRDENG_DEFAULT_TIER_DISK_SPACE_MB : default_multidb_disk_quota_mb;
272
+ snprintfz(dbengineconfig, sizeof(dbengineconfig) - 1, "dbengine tier %zu retention size", tier);
273
+ disk_space_mb = config_get_size_mb(CONFIG_SECTION_DB, dbengineconfig, disk_space_mb);
274
+
275
+ snprintfz(dbengineconfig, sizeof(dbengineconfig) - 1, "dbengine tier %zu retention time", tier);
276
+ storage_tiers_retention_days[tier] = config_get_duration_days(
277
+ CONFIG_SECTION_DB, dbengineconfig, new_dbengine_defaults ? storage_tiers_retention_days[tier] : 0);
278
+
279
+ tiers_init[tier].disk_space_mb = (int) disk_space_mb;
280
+ tiers_init[tier].tier = tier;
281
+ tiers_init[tier].retention_seconds = (size_t) (86400.0 * storage_tiers_retention_days[tier]);
282
+ strncpyz(tiers_init[tier].path, dbenginepath, FILENAME_MAX);
283
+ tiers_init[tier].ret = 0;
284
+
285
+ if(parallel_initialization) {
286
+ char tag[NETDATA_THREAD_TAG_MAX + 1];
287
+ snprintfz(tag, NETDATA_THREAD_TAG_MAX, "DBENGINIT[%zu]", tier);
288
+ tiers_init[tier].thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_JOINABLE, dbengine_tier_init, &tiers_init[tier]);
289
+ }
290
+ else
291
+ dbengine_tier_init(&tiers_init[tier]);
292
+ }
293
+
294
+ for(size_t tier = 0; tier < storage_tiers ;tier++) {
295
+ if(parallel_initialization)
296
+ nd_thread_join(tiers_init[tier].thread);
297
+
298
+ if(tiers_init[tier].ret != 0) {
299
+ nd_log(NDLS_DAEMON, NDLP_ERR,
300
+ "DBENGINE on '%s': Failed to initialize multi-host database tier %zu on path '%s'",
301
+ hostname, tiers_init[tier].tier, tiers_init[tier].path);
302
+ }
303
+ else if(created_tiers == tier)
304
+ created_tiers++;
305
+ }
306
+
307
+ if(created_tiers && created_tiers < storage_tiers) {
308
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
309
+ "DBENGINE on '%s': Managed to create %zu tiers instead of %zu. Continuing with %zu available.",
310
+ hostname, created_tiers, storage_tiers, created_tiers);
311
+
312
+ storage_tiers = created_tiers;
313
+ }
314
+ else if(!created_tiers)
315
+ fatal("DBENGINE on '%s', failed to initialize databases at '%s'.", hostname, netdata_configured_cache_dir);
316
+
317
+ for(size_t tier = 0; tier < storage_tiers ;tier++)
318
+ rrdeng_readiness_wait(multidb_ctx[tier]);
319
+
320
+ calculate_tier_disk_space_percentage();
321
+
322
+ dbengine_enabled = true;
323
+#else
324
+ storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", 1);
325
+ if(storage_tiers != 1) {
326
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
327
+ "DBENGINE is not available on '%s', so only 1 database tier can be supported.",
328
+ hostname);
329
+
330
+ storage_tiers = 1;
331
+ config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
332
+ }
333
+ dbengine_enabled = false;
334
+#endif
335
+}
336
+
337
+void netdata_conf_section_db(void) {
338
+ static bool run = false;
339
+ if(run) return;
340
+ run = true;
341
+
342
+ // ------------------------------------------------------------------------
343
+
344
+ rrdhost_free_orphan_time_s =
345
+ config_get_duration_seconds(CONFIG_SECTION_DB, "cleanup orphan hosts after", rrdhost_free_orphan_time_s);
346
+
347
+ // ------------------------------------------------------------------------
348
+ // get default database update frequency
349
+
350
+ default_rrd_update_every = (int) config_get_duration_seconds(CONFIG_SECTION_DB, "update every", UPDATE_EVERY);
351
+ if(default_rrd_update_every < 1 || default_rrd_update_every > 600) {
352
+ netdata_log_error("Invalid data collection frequency (update every) %d given. Defaulting to %d.", default_rrd_update_every, UPDATE_EVERY);
353
+ default_rrd_update_every = UPDATE_EVERY;
354
+ config_set_duration_seconds(CONFIG_SECTION_DB, "update every", default_rrd_update_every);
355
+ }
356
+
357
+ // ------------------------------------------------------------------------
358
+ // get the database selection
359
+
360
+ {
361
+ const char *mode = config_get(CONFIG_SECTION_DB, "db", rrd_memory_mode_name(default_rrd_memory_mode));
362
+ default_rrd_memory_mode = rrd_memory_mode_id(mode);
363
+ if(strcmp(mode, rrd_memory_mode_name(default_rrd_memory_mode)) != 0) {
364
+ netdata_log_error("Invalid memory mode '%s' given. Using '%s'", mode, rrd_memory_mode_name(default_rrd_memory_mode));
365
+ config_set(CONFIG_SECTION_DB, "db", rrd_memory_mode_name(default_rrd_memory_mode));
366
+ }
367
+ }
368
+
369
+ // ------------------------------------------------------------------------
370
+ // get default database size
371
+
372
+ if(default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE && default_rrd_memory_mode != RRD_MEMORY_MODE_NONE) {
373
+ default_rrd_history_entries = (int)config_get_number(
374
+ CONFIG_SECTION_DB, "retention",
375
+ align_entries_to_pagesize(default_rrd_memory_mode, RRD_DEFAULT_HISTORY_ENTRIES));
376
+
377
+ long h = align_entries_to_pagesize(default_rrd_memory_mode, default_rrd_history_entries);
378
+ if (h != default_rrd_history_entries) {
379
+ config_set_number(CONFIG_SECTION_DB, "retention", h);
380
+ default_rrd_history_entries = (int)h;
381
+ }
382
+ }
383
+
384
+ // --------------------------------------------------------------------
385
+ // get KSM settings
386
+
387
+#ifdef MADV_MERGEABLE
388
+ enable_ksm = config_get_boolean_ondemand(CONFIG_SECTION_DB, "memory deduplication (ksm)", enable_ksm);
389
+#endif
390
+
391
+ // --------------------------------------------------------------------
392
+
393
+ rrdhost_free_ephemeral_time_s =
394
+ config_get_duration_seconds(CONFIG_SECTION_DB, "cleanup ephemeral hosts after", rrdhost_free_ephemeral_time_s);
395
+
396
+ rrdset_free_obsolete_time_s =
397
+ config_get_duration_seconds(CONFIG_SECTION_DB, "cleanup obsolete charts after", rrdset_free_obsolete_time_s);
398
+
399
+ // Current chart locking and invalidation scheme doesn't prevent Netdata from segmentation faults if a short
400
+ // cleanup delay is set. Extensive stress tests showed that 10 seconds is quite a safe delay. Look at
401
+ // https://github.com/netdata/netdata/pull/11222#issuecomment-868367920 for more information.
402
+ if (rrdset_free_obsolete_time_s < 10) {
403
+ rrdset_free_obsolete_time_s = 10;
404
+ netdata_log_info("The \"cleanup obsolete charts after\" option was set to 10 seconds.");
405
+ config_set_duration_seconds(CONFIG_SECTION_DB, "cleanup obsolete charts after", rrdset_free_obsolete_time_s);
406
+ }
407
+
408
+ gap_when_lost_iterations_above = (int)config_get_number(CONFIG_SECTION_DB, "gap when lost iterations above", gap_when_lost_iterations_above);
409
+ if (gap_when_lost_iterations_above < 1) {
410
+ gap_when_lost_iterations_above = 1;
411
+ config_set_number(CONFIG_SECTION_DB, "gap when lost iterations above", gap_when_lost_iterations_above);
412
+ }
413
+ gap_when_lost_iterations_above += 2;
414
+
415
+ // ------------------------------------------------------------------------
416
+
417
+ netdata_conf_dbengine_pre_logs();
418
+}
src/daemon/config/netdata-conf-db.h
new
+24
@@ -0,0 +1,24 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_DAEMON_NETDATA_CONF_DBENGINE_H
4
+#define NETDATA_DAEMON_NETDATA_CONF_DBENGINE_H
5
+
6
+#include "libnetdata/libnetdata.h"
7
+
8
+extern bool dbengine_enabled;
9
+extern size_t storage_tiers;
10
+extern bool dbengine_use_direct_io;
11
+
12
+extern int default_rrd_update_every;
13
+extern int default_rrd_history_entries;
14
+extern int gap_when_lost_iterations_above;
15
+extern time_t rrdset_free_obsolete_time_s;
16
+
17
+size_t get_tier_grouping(size_t tier);
18
+
19
+void netdata_conf_section_db(void);
20
+void netdata_conf_dbengine_init(const char *hostname);
21
+
22
+#include "netdata-conf.h"
23
+
24
+#endif //NETDATA_DAEMON_NETDATA_CONF_DBENGINE_H
src/daemon/config/netdata-conf-directories.c
new
+31
@@ -0,0 +1,31 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "netdata-conf-directories.h"
4
+
5
+static const char *get_varlib_subdir_from_config(const char *prefix, const char *dir) {
6
+ char filename[FILENAME_MAX + 1];
7
+ snprintfz(filename, FILENAME_MAX, "%s/%s", prefix, dir);
8
+ return config_get(CONFIG_SECTION_DIRECTORIES, dir, filename);
9
+}
10
+
11
+void netdata_conf_section_directories(void) {
12
+ static bool run = false;
13
+ if(run) return;
14
+ run = true;
15
+
16
+ // ------------------------------------------------------------------------
17
+ // get system paths
18
+
19
+ netdata_configured_user_config_dir = config_get(CONFIG_SECTION_DIRECTORIES, "config", netdata_configured_user_config_dir);
20
+ netdata_configured_stock_config_dir = config_get(CONFIG_SECTION_DIRECTORIES, "stock config", netdata_configured_stock_config_dir);
21
+ netdata_configured_log_dir = config_get(CONFIG_SECTION_DIRECTORIES, "log", netdata_configured_log_dir);
22
+ netdata_configured_web_dir = config_get(CONFIG_SECTION_DIRECTORIES, "web", netdata_configured_web_dir);
23
+ netdata_configured_cache_dir = config_get(CONFIG_SECTION_DIRECTORIES, "cache", netdata_configured_cache_dir);
24
+ netdata_configured_varlib_dir = config_get(CONFIG_SECTION_DIRECTORIES, "lib", netdata_configured_varlib_dir);
25
+
26
+ netdata_configured_lock_dir = get_varlib_subdir_from_config(netdata_configured_varlib_dir, "lock");
27
+ netdata_configured_cloud_dir = get_varlib_subdir_from_config(netdata_configured_varlib_dir, "cloud.d");
28
+
29
+ pluginsd_initialize_plugin_directories();
30
+ netdata_configured_primary_plugins_dir = plugin_directories[PLUGINSD_STOCK_PLUGINS_DIRECTORY_PATH];
31
+}
src/daemon/config/netdata-conf-directories.h
new
+10
@@ -0,0 +1,10 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_NETDATA_CONF_DIRECTORIES_H
4
+#define NETDATA_NETDATA_CONF_DIRECTORIES_H
5
+
6
+#include "netdata-conf.h"
7
+
8
+void netdata_conf_section_directories(void);
9
+
10
+#endif //NETDATA_NETDATA_CONF_DIRECTORIES_H
src/daemon/config/netdata-conf-global.c
new
+57
@@ -0,0 +1,57 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "netdata-conf-global.h"
4
+
5
+static int get_hostname(char *buf, size_t buf_size) {
6
+ if (netdata_configured_host_prefix && *netdata_configured_host_prefix) {
7
+ char filename[FILENAME_MAX + 1];
8
+ snprintfz(filename, FILENAME_MAX, "%s/etc/hostname", netdata_configured_host_prefix);
9
+
10
+ if (!read_txt_file(filename, buf, buf_size)) {
11
+ trim(buf);
12
+ return 0;
13
+ }
14
+ }
15
+
16
+ return gethostname(buf, buf_size);
17
+}
18
+
19
+void netdata_conf_section_global(void) {
20
+ netdata_conf_backwards_compatibility();
21
+
22
+ // ------------------------------------------------------------------------
23
+ // get the hostname
24
+
25
+ netdata_configured_host_prefix = config_get(CONFIG_SECTION_GLOBAL, "host access prefix", "");
26
+ (void) verify_netdata_host_prefix(true);
27
+
28
+ char buf[HOSTNAME_MAX + 1];
29
+ if (get_hostname(buf, HOSTNAME_MAX))
30
+ netdata_log_error("Cannot get machine hostname.");
31
+
32
+ netdata_configured_hostname = config_get(CONFIG_SECTION_GLOBAL, "hostname", buf);
33
+ netdata_log_debug(D_OPTIONS, "hostname set to '%s'", netdata_configured_hostname);
34
+
35
+ netdata_conf_section_directories();
36
+ netdata_conf_section_db();
37
+
38
+ // --------------------------------------------------------------------
39
+ // get various system parameters
40
+
41
+ os_get_system_cpus_uncached();
42
+ os_get_system_pid_max();
43
+}
44
+
45
+void netdata_conf_section_global_run_as_user(const char **user) {
46
+ // --------------------------------------------------------------------
47
+ // get the user we should run
48
+
49
+ // IMPORTANT: this is required before web_files_uid()
50
+ if(getuid() == 0) {
51
+ *user = config_get(CONFIG_SECTION_GLOBAL, "run as user", NETDATA_USER);
52
+ }
53
+ else {
54
+ struct passwd *passwd = getpwuid(getuid());
55
+ *user = config_get(CONFIG_SECTION_GLOBAL, "run as user", (passwd && passwd->pw_name)?passwd->pw_name:"");
56
+ }
57
+}
src/daemon/config/netdata-conf-global.h
new
+13
@@ -0,0 +1,13 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_NETDATA_CONF_GLOBAL_H
4
+#define NETDATA_NETDATA_CONF_GLOBAL_H
5
+
6
+#include "libnetdata/libnetdata.h"
7
+
8
+void netdata_conf_section_global(void);
9
+void netdata_conf_section_global_run_as_user(const char **user);
10
+
11
+#include "netdata-conf.h"
12
+
13
+#endif //NETDATA_NETDATA_CONF_GLOBAL_H
src/daemon/config/netdata-conf-logs.c
new
+82
@@ -0,0 +1,82 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "netdata-conf-logs.h"
4
+
5
+void netdata_conf_section_logs(void) {
6
+ static bool run = false;
7
+ if(run) return;
8
+ run = true;
9
+
10
+ nd_log_set_facility(config_get(CONFIG_SECTION_LOGS, "facility", "daemon"));
11
+
12
+ time_t period = ND_LOG_DEFAULT_THROTTLE_PERIOD;
13
+ size_t logs = ND_LOG_DEFAULT_THROTTLE_LOGS;
14
+ period = config_get_duration_seconds(CONFIG_SECTION_LOGS, "logs flood protection period", period);
15
+ logs = (unsigned long)config_get_number(CONFIG_SECTION_LOGS, "logs to trigger flood protection", (long long int)logs);
16
+ nd_log_set_flood_protection(logs, period);
17
+
18
+ const char *netdata_log_level = getenv("NETDATA_LOG_LEVEL");
19
+ netdata_log_level = netdata_log_level ? nd_log_id2priority(nd_log_priority2id(netdata_log_level)) : NDLP_INFO_STR;
20
+
21
+ nd_log_set_priority_level(config_get(CONFIG_SECTION_LOGS, "level", netdata_log_level));
22
+
23
+ char filename[FILENAME_MAX + 1];
24
+ char* os_default_method = NULL;
25
+#if defined(OS_LINUX)
26
+ os_default_method = is_stderr_connected_to_journal() /* || nd_log_journal_socket_available() */ ? "journal" : NULL;
27
+#elif defined(OS_WINDOWS)
28
+#if defined(HAVE_ETW)
29
+ os_default_method = "etw";
30
+#elif defined(HAVE_WEL)
31
+ os_default_method = "wel";
32
+#endif
33
+#endif
34
+
35
+#if defined(OS_WINDOWS)
36
+ // on windows, debug log goes to windows events
37
+ snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
38
+#else
39
+ snprintfz(filename, FILENAME_MAX, "%s/debug.log", netdata_configured_log_dir);
40
+#endif
41
+
42
+ nd_log_set_user_settings(NDLS_DEBUG, config_get(CONFIG_SECTION_LOGS, "debug", filename));
43
+
44
+ if(os_default_method)
45
+ snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
46
+ else
47
+ snprintfz(filename, FILENAME_MAX, "%s/daemon.log", netdata_configured_log_dir);
48
+ nd_log_set_user_settings(NDLS_DAEMON, config_get(CONFIG_SECTION_LOGS, "daemon", filename));
49
+
50
+ if(os_default_method)
51
+ snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
52
+ else
53
+ snprintfz(filename, FILENAME_MAX, "%s/collector.log", netdata_configured_log_dir);
54
+ nd_log_set_user_settings(NDLS_COLLECTORS, config_get(CONFIG_SECTION_LOGS, "collector", filename));
55
+
56
+#if defined(OS_WINDOWS)
57
+ // on windows, access log goes to windows events
58
+ snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
59
+#else
60
+ snprintfz(filename, FILENAME_MAX, "%s/access.log", netdata_configured_log_dir);
61
+#endif
62
+ nd_log_set_user_settings(NDLS_ACCESS, config_get(CONFIG_SECTION_LOGS, "access", filename));
63
+
64
+ if(os_default_method)
65
+ snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
66
+ else
67
+ snprintfz(filename, FILENAME_MAX, "%s/health.log", netdata_configured_log_dir);
68
+ nd_log_set_user_settings(NDLS_HEALTH, config_get(CONFIG_SECTION_LOGS, "health", filename));
69
+
70
+ aclklog_enabled = config_get_boolean(CONFIG_SECTION_CLOUD, "conversation log", CONFIG_BOOLEAN_NO);
71
+ if (aclklog_enabled) {
72
+#if defined(OS_WINDOWS)
73
+ // on windows, aclk log goes to windows events
74
+ snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
75
+#else
76
+ snprintfz(filename, FILENAME_MAX, "%s/aclk.log", netdata_configured_log_dir);
77
+#endif
78
+ nd_log_set_user_settings(NDLS_ACLK, config_get(CONFIG_SECTION_CLOUD, "conversation log file", filename));
79
+ }
80
+
81
+ aclk_config_get_query_scope();
82
+}
src/daemon/config/netdata-conf-logs.h
new
+10
@@ -0,0 +1,10 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_NETDATA_CONF_LOGS_H
4
+#define NETDATA_NETDATA_CONF_LOGS_H
5
+
6
+#include "netdata-conf.h"
7
+
8
+void netdata_conf_section_logs(void);
9
+
10
+#endif //NETDATA_NETDATA_CONF_LOGS_H
src/daemon/config/netdata-conf-web.c
new
+143
@@ -0,0 +1,143 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "netdata-conf-web.h"
4
+#include "daemon/static_threads.h"
5
+
6
+static int make_dns_decision(const char *section_name, const char *config_name, const char *default_value, SIMPLE_PATTERN *p) {
7
+ const char *value = config_get(section_name,config_name,default_value);
8
+
9
+ if(!strcmp("yes",value))
10
+ return 1;
11
+
12
+ if(!strcmp("no",value))
13
+ return 0;
14
+
15
+ if(strcmp("heuristic",value) != 0)
16
+ netdata_log_error("Invalid configuration option '%s' for '%s'/'%s'. Valid options are 'yes', 'no' and 'heuristic'. Proceeding with 'heuristic'",
17
+ value, section_name, config_name);
18
+
19
+ return simple_pattern_is_potential_name(p);
20
+}
21
+
22
+extern struct netdata_static_thread *static_threads;
23
+void web_server_threading_selection(void) {
24
+ static bool run = false;
25
+ if(run) return;
26
+ run = true;
27
+
28
+ web_server_mode = web_server_mode_id(config_get(CONFIG_SECTION_WEB, "mode", web_server_mode_name(web_server_mode)));
29
+
30
+ int static_threaded = (web_server_mode == WEB_SERVER_MODE_STATIC_THREADED);
31
+
32
+ int i;
33
+ for (i = 0; static_threads[i].name; i++) {
34
+ if (static_threads[i].start_routine == socket_listen_main_static_threaded)
35
+ static_threads[i].enabled = static_threaded;
36
+ }
37
+}
38
+
39
+void netdata_conf_section_web(void) {
40
+ static bool run = false;
41
+ if(run) return;
42
+ run = true;
43
+
44
+ web_client_timeout =
45
+ (int)config_get_duration_seconds(CONFIG_SECTION_WEB, "disconnect idle clients after", web_client_timeout);
46
+
47
+ web_client_first_request_timeout =
48
+ (int)config_get_duration_seconds(CONFIG_SECTION_WEB, "timeout for first request", web_client_first_request_timeout);
49
+
50
+ web_client_streaming_rate_t =
51
+ config_get_duration_seconds(CONFIG_SECTION_WEB, "accept a streaming request every", web_client_streaming_rate_t);
52
+
53
+ respect_web_browser_do_not_track_policy =
54
+ config_get_boolean(CONFIG_SECTION_WEB, "respect do not track policy", respect_web_browser_do_not_track_policy);
55
+ web_x_frame_options = config_get(CONFIG_SECTION_WEB, "x-frame-options response header", "");
56
+ if(!*web_x_frame_options)
57
+ web_x_frame_options = NULL;
58
+
59
+ web_allow_connections_from =
60
+ simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow connections from", "localhost *"),
61
+ NULL, SIMPLE_PATTERN_EXACT, true);
62
+ web_allow_connections_dns =
63
+ make_dns_decision(CONFIG_SECTION_WEB, "allow connections by dns", "heuristic", web_allow_connections_from);
64
+ web_allow_dashboard_from =
65
+ simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow dashboard from", "localhost *"),
66
+ NULL, SIMPLE_PATTERN_EXACT, true);
67
+ web_allow_dashboard_dns =
68
+ make_dns_decision(CONFIG_SECTION_WEB, "allow dashboard by dns", "heuristic", web_allow_dashboard_from);
69
+ web_allow_badges_from =
70
+ simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow badges from", "*"), NULL, SIMPLE_PATTERN_EXACT,
71
+ true);
72
+ web_allow_badges_dns =
73
+ make_dns_decision(CONFIG_SECTION_WEB, "allow badges by dns", "heuristic", web_allow_badges_from);
74
+ web_allow_registry_from =
75
+ simple_pattern_create(config_get(CONFIG_SECTION_REGISTRY, "allow from", "*"), NULL, SIMPLE_PATTERN_EXACT,
76
+ true);
77
+ web_allow_registry_dns = make_dns_decision(CONFIG_SECTION_REGISTRY, "allow by dns", "heuristic",
78
+ web_allow_registry_from);
79
+ web_allow_streaming_from = simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow streaming from", "*"),
80
+ NULL, SIMPLE_PATTERN_EXACT, true);
81
+ web_allow_streaming_dns = make_dns_decision(CONFIG_SECTION_WEB, "allow streaming by dns", "heuristic",
82
+ web_allow_streaming_from);
83
+ // Note the default is not heuristic, the wildcards could match DNS but the intent is ip-addresses.
84
+ web_allow_netdataconf_from = simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow netdata.conf from",
85
+ "localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.*"
86
+ " 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.*"
87
+ " 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.*"
88
+ " 172.31.* UNKNOWN"), NULL, SIMPLE_PATTERN_EXACT,
89
+ true);
90
+ web_allow_netdataconf_dns =
91
+ make_dns_decision(CONFIG_SECTION_WEB, "allow netdata.conf by dns", "no", web_allow_netdataconf_from);
92
+ web_allow_mgmt_from =
93
+ simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow management from", "localhost"),
94
+ NULL, SIMPLE_PATTERN_EXACT, true);
95
+ web_allow_mgmt_dns =
96
+ make_dns_decision(CONFIG_SECTION_WEB, "allow management by dns","heuristic",web_allow_mgmt_from);
97
+
98
+ web_enable_gzip = config_get_boolean(CONFIG_SECTION_WEB, "enable gzip compression", web_enable_gzip);
99
+
100
+ const char *s = config_get(CONFIG_SECTION_WEB, "gzip compression strategy", "default");
101
+ if(!strcmp(s, "default"))
102
+ web_gzip_strategy = Z_DEFAULT_STRATEGY;
103
+ else if(!strcmp(s, "filtered"))
104
+ web_gzip_strategy = Z_FILTERED;
105
+ else if(!strcmp(s, "huffman only"))
106
+ web_gzip_strategy = Z_HUFFMAN_ONLY;
107
+ else if(!strcmp(s, "rle"))
108
+ web_gzip_strategy = Z_RLE;
109
+ else if(!strcmp(s, "fixed"))
110
+ web_gzip_strategy = Z_FIXED;
111
+ else {
112
+ netdata_log_error("Invalid compression strategy '%s'. Valid strategies are 'default', 'filtered', 'huffman only', 'rle' and 'fixed'. Proceeding with 'default'.", s);
113
+ web_gzip_strategy = Z_DEFAULT_STRATEGY;
114
+ }
115
+
116
+ web_gzip_level = (int)config_get_number(CONFIG_SECTION_WEB, "gzip compression level", 3);
117
+ if(web_gzip_level < 1) {
118
+ netdata_log_error("Invalid compression level %d. Valid levels are 1 (fastest) to 9 (best ratio). Proceeding with level 1 (fastest compression).", web_gzip_level);
119
+ web_gzip_level = 1;
120
+ }
121
+ else if(web_gzip_level > 9) {
122
+ netdata_log_error("Invalid compression level %d. Valid levels are 1 (fastest) to 9 (best ratio). Proceeding with level 9 (best compression).", web_gzip_level);
123
+ web_gzip_level = 9;
124
+ }
125
+}
126
+
127
+void netdata_conf_web_security_init(void) {
128
+ static bool run = false;
129
+ if(run) return;
130
+ run = true;
131
+
132
+ char filename[FILENAME_MAX + 1];
133
+ snprintfz(filename, FILENAME_MAX, "%s/ssl/key.pem",netdata_configured_user_config_dir);
134
+ netdata_ssl_security_key = config_get(CONFIG_SECTION_WEB, "ssl key", filename);
135
+
136
+ snprintfz(filename, FILENAME_MAX, "%s/ssl/cert.pem",netdata_configured_user_config_dir);
137
+ netdata_ssl_security_cert = config_get(CONFIG_SECTION_WEB, "ssl certificate", filename);
138
+
139
+ tls_version = config_get(CONFIG_SECTION_WEB, "tls version", "1.3");
140
+ tls_ciphers = config_get(CONFIG_SECTION_WEB, "tls ciphers", "none");
141
+
142
+ netdata_ssl_initialize_openssl();
143
+}
src/daemon/config/netdata-conf-web.h
new
+12
@@ -0,0 +1,12 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_NETDATA_CONF_WEB_H
4
+#define NETDATA_NETDATA_CONF_WEB_H
5
+
6
+#include "netdata-conf.h"
7
+
8
+void netdata_conf_section_web(void);
9
+void web_server_threading_selection(void);
10
+void netdata_conf_web_security_init(void);
11
+
12
+#endif //NETDATA_NETDATA_CONF_WEB_H
src/daemon/config/netdata-conf.c
new
+34
@@ -0,0 +1,34 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "netdata-conf.h"
4
+
5
+bool netdata_conf_load(char *filename, char overwrite_used, const char **user) {
6
+ errno_clear();
7
+
8
+ int ret = 0;
9
+
10
+ if(filename && *filename) {
11
+ ret = config_load(filename, overwrite_used, NULL);
12
+ if(!ret)
13
+ netdata_log_error("CONFIG: cannot load config file '%s'.", filename);
14
+ }
15
+ else {
16
+ filename = filename_from_path_entry_strdupz(netdata_configured_user_config_dir, "netdata.conf");
17
+
18
+ ret = config_load(filename, overwrite_used, NULL);
19
+ if(!ret) {
20
+ netdata_log_info("CONFIG: cannot load user config '%s'. Will try the stock version.", filename);
21
+ freez(filename);
22
+
23
+ filename = filename_from_path_entry_strdupz(netdata_configured_stock_config_dir, "netdata.conf");
24
+ ret = config_load(filename, overwrite_used, NULL);
25
+ if(!ret)
26
+ netdata_log_info("CONFIG: cannot load stock config '%s'. Running with internal defaults.", filename);
27
+ }
28
+
29
+ freez(filename);
30
+ }
31
+
32
+ netdata_conf_section_global_run_as_user(user);
33
+ return ret;
34
+}
src/daemon/config/netdata-conf.h
new
+19
@@ -0,0 +1,19 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_DAEMON_NETDATA_CONF_H
4
+#define NETDATA_DAEMON_NETDATA_CONF_H
5
+
6
+#include "libnetdata/libnetdata.h"
7
+
8
+bool netdata_conf_load(char *filename, char overwrite_used, const char **user);
9
+
10
+#include "netdata-conf-backwards-compatibility.h"
11
+#include "netdata-conf-db.h"
12
+#include "netdata-conf-directories.h"
13
+#include "netdata-conf-global.h"
14
+#include "netdata-conf-logs.h"
15
+#include "netdata-conf-web.h"
16
+
17
+#include "daemon/common.h"
18
+
19
+#endif //NETDATA_DAEMON_NETDATA_CONF_H
src/daemon/daemon.c
+39
@@ -6,6 +6,45 @@
6
char *pidfile = NULL;
7
char *netdata_exe_path = NULL;
8
9
+long get_netdata_cpus(void) {
10
+ static long processors = 0;
11
+
12
+ if(processors)
13
+ return processors;
14
+
15
+ long cores_proc_stat = os_get_system_cpus_cached(false, true);
16
+ long cores_cpuset_v1 = (long)os_read_cpuset_cpus("/sys/fs/cgroup/cpuset/cpuset.cpus", cores_proc_stat);
17
+ long cores_cpuset_v2 = (long)os_read_cpuset_cpus("/sys/fs/cgroup/cpuset.cpus", cores_proc_stat);
18
+
19
+ if(cores_cpuset_v2)
20
+ processors = cores_cpuset_v2;
21
+ else if(cores_cpuset_v1)
22
+ processors = cores_cpuset_v1;
23
+ else
24
+ processors = cores_proc_stat;
25
+
26
+ long cores_user_configured = config_get_number(CONFIG_SECTION_GLOBAL, "cpu cores", processors);
27
+
28
+ errno_clear();
29
+ internal_error(true,
30
+ "System CPUs: %ld, ("
31
+ "system: %ld, cgroups cpuset v1: %ld, cgroups cpuset v2: %ld, netdata.conf: %ld"
32
+ ")"
33
+ , processors
34
+ , cores_proc_stat
35
+ , cores_cpuset_v1
36
+ , cores_cpuset_v2
37
+ , cores_user_configured
38
+ );
39
+
40
+ processors = cores_user_configured;
41
+
42
+ if(processors < 1)
43
+ processors = 1;
44
+
45
+ return processors;
46
+}
47
+
48
void get_netdata_execution_path(void) {
49
struct passwd *passwd = getpwuid(getuid());
50
char *user = (passwd && passwd->pw_name) ? passwd->pw_name : "";
src/daemon/dyncfg/dyncfg-echo.c
renamed
src/daemon/dyncfg/dyncfg-files.c
renamed
src/daemon/dyncfg/dyncfg-inline.c
renamed
src/daemon/dyncfg/dyncfg-intercept.c
renamed
src/daemon/dyncfg/dyncfg-internals.h
renamed
src/daemon/dyncfg/dyncfg-tree.c
renamed
src/daemon/dyncfg/dyncfg-unittest.c
renamed
src/daemon/dyncfg/dyncfg.c
renamed
src/daemon/dyncfg/dyncfg.h
renamed
src/daemon/global_statistics.c
deleted
-4339
@@ -1,4339 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#include "common.h"
4
-
5
-#define GLOBAL_STATS_RESET_WEB_USEC_MAX 0x01
6
-
7
-#define WORKER_JOB_GLOBAL 0
8
-#define WORKER_JOB_GLOBAL_EXT 1
9
-#define WORKER_JOB_REGISTRY 2
10
-#define WORKER_JOB_WORKERS 3
11
-#define WORKER_JOB_DBENGINE 4
12
-#define WORKER_JOB_HEARTBEAT 5
13
-#define WORKER_JOB_STRINGS 6
14
-#define WORKER_JOB_DICTIONARIES 7
15
-#define WORKER_JOB_MALLOC_TRACE 8
16
-#define WORKER_JOB_SQLITE3 9
17
-
18
-#if WORKER_UTILIZATION_MAX_JOB_TYPES < 10
19
-#error WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 10
20
-#endif
21
-
22
-bool global_statistics_enabled = true;
23
-
24
-struct netdata_buffers_statistics netdata_buffers_statistics = {};
25
-
26
-static size_t dbengine_total_memory = 0;
27
-size_t rrddim_db_memory_size = 0;
28
-
29
-static struct global_statistics {
30
- uint16_t connected_clients;
31
-
32
- uint64_t web_requests;
33
- uint64_t web_usec;
34
- uint64_t web_usec_max;
35
- uint64_t bytes_received;
36
- uint64_t bytes_sent;
37
- uint64_t content_size;
38
- uint64_t compressed_content_size;
39
-
40
- uint64_t web_client_count;
41
-
42
- uint64_t api_data_queries_made;
43
- uint64_t api_data_db_points_read;
44
- uint64_t api_data_result_points_generated;
45
-
46
- uint64_t api_weights_queries_made;
47
- uint64_t api_weights_db_points_read;
48
- uint64_t api_weights_result_points_generated;
49
-
50
- uint64_t api_badges_queries_made;
51
- uint64_t api_badges_db_points_read;
52
- uint64_t api_badges_result_points_generated;
53
-
54
- uint64_t health_queries_made;
55
- uint64_t health_db_points_read;
56
- uint64_t health_result_points_generated;
57
-
58
- uint64_t ml_queries_made;
59
- uint64_t ml_db_points_read;
60
- uint64_t ml_result_points_generated;
61
- uint64_t ml_models_consulted;
62
-
63
- uint64_t exporters_queries_made;
64
- uint64_t exporters_db_points_read;
65
-
66
- uint64_t backfill_queries_made;
67
- uint64_t backfill_db_points_read;
68
-
69
- uint64_t tier0_hot_gorilla_buffers;
70
-
71
- uint64_t tier0_disk_compressed_bytes;
72
- uint64_t tier0_disk_uncompressed_bytes;
73
-
74
- uint64_t db_points_stored_per_tier[RRD_STORAGE_TIERS];
75
-
76
-} global_statistics = {
77
- .connected_clients = 0,
78
- .web_requests = 0,
79
- .web_usec = 0,
80
- .bytes_received = 0,
81
- .bytes_sent = 0,
82
- .content_size = 0,
83
- .compressed_content_size = 0,
84
- .web_client_count = 1,
85
-
86
- .api_data_queries_made = 0,
87
- .api_data_db_points_read = 0,
88
- .api_data_result_points_generated = 0,
89
-
90
- .tier0_hot_gorilla_buffers = 0,
91
- .tier0_disk_compressed_bytes = 0,
92
- .tier0_disk_uncompressed_bytes = 0,
93
-};
94
-
95
-void global_statistics_rrdset_done_chart_collection_completed(size_t *points_read_per_tier_array) {
96
- for(size_t tier = 0; tier < storage_tiers ;tier++) {
97
- __atomic_fetch_add(&global_statistics.db_points_stored_per_tier[tier], points_read_per_tier_array[tier], __ATOMIC_RELAXED);
98
- points_read_per_tier_array[tier] = 0;
99
- }
100
-}
101
-
102
-void global_statistics_ml_query_completed(size_t points_read) {
103
- __atomic_fetch_add(&global_statistics.ml_queries_made, 1, __ATOMIC_RELAXED);
104
- __atomic_fetch_add(&global_statistics.ml_db_points_read, points_read, __ATOMIC_RELAXED);
105
-}
106
-
107
-void global_statistics_ml_models_consulted(size_t models_consulted) {
108
- __atomic_fetch_add(&global_statistics.ml_models_consulted, models_consulted, __ATOMIC_RELAXED);
109
-}
110
-
111
-void global_statistics_exporters_query_completed(size_t points_read) {
112
- __atomic_fetch_add(&global_statistics.exporters_queries_made, 1, __ATOMIC_RELAXED);
113
- __atomic_fetch_add(&global_statistics.exporters_db_points_read, points_read, __ATOMIC_RELAXED);
114
-}
115
-
116
-void global_statistics_backfill_query_completed(size_t points_read) {
117
- __atomic_fetch_add(&global_statistics.backfill_queries_made, 1, __ATOMIC_RELAXED);
118
- __atomic_fetch_add(&global_statistics.backfill_db_points_read, points_read, __ATOMIC_RELAXED);
119
-}
120
-
121
-void global_statistics_gorilla_buffer_add_hot() {
122
- __atomic_fetch_add(&global_statistics.tier0_hot_gorilla_buffers, 1, __ATOMIC_RELAXED);
123
-}
124
-
125
-void global_statistics_tier0_disk_compressed_bytes(uint32_t size) {
126
- __atomic_fetch_add(&global_statistics.tier0_disk_compressed_bytes, size, __ATOMIC_RELAXED);
127
-}
128
-
129
-void global_statistics_tier0_disk_uncompressed_bytes(uint32_t size) {
130
- __atomic_fetch_add(&global_statistics.tier0_disk_uncompressed_bytes, size, __ATOMIC_RELAXED);
131
-}
132
-
133
-void global_statistics_rrdr_query_completed(size_t queries, uint64_t db_points_read, uint64_t result_points_generated, QUERY_SOURCE query_source) {
134
- switch(query_source) {
135
- case QUERY_SOURCE_API_DATA:
136
- __atomic_fetch_add(&global_statistics.api_data_queries_made, queries, __ATOMIC_RELAXED);
137
- __atomic_fetch_add(&global_statistics.api_data_db_points_read, db_points_read, __ATOMIC_RELAXED);
138
- __atomic_fetch_add(&global_statistics.api_data_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
139
- break;
140
-
141
- case QUERY_SOURCE_ML:
142
- __atomic_fetch_add(&global_statistics.ml_queries_made, queries, __ATOMIC_RELAXED);
143
- __atomic_fetch_add(&global_statistics.ml_db_points_read, db_points_read, __ATOMIC_RELAXED);
144
- __atomic_fetch_add(&global_statistics.ml_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
145
- break;
146
-
147
- case QUERY_SOURCE_API_WEIGHTS:
148
- __atomic_fetch_add(&global_statistics.api_weights_queries_made, queries, __ATOMIC_RELAXED);
149
- __atomic_fetch_add(&global_statistics.api_weights_db_points_read, db_points_read, __ATOMIC_RELAXED);
150
- __atomic_fetch_add(&global_statistics.api_weights_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
151
- break;
152
-
153
- case QUERY_SOURCE_API_BADGE:
154
- __atomic_fetch_add(&global_statistics.api_badges_queries_made, queries, __ATOMIC_RELAXED);
155
- __atomic_fetch_add(&global_statistics.api_badges_db_points_read, db_points_read, __ATOMIC_RELAXED);
156
- __atomic_fetch_add(&global_statistics.api_badges_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
157
- break;
158
-
159
- case QUERY_SOURCE_HEALTH:
160
- __atomic_fetch_add(&global_statistics.health_queries_made, queries, __ATOMIC_RELAXED);
161
- __atomic_fetch_add(&global_statistics.health_db_points_read, db_points_read, __ATOMIC_RELAXED);
162
- __atomic_fetch_add(&global_statistics.health_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
163
- break;
164
-
165
- default:
166
- case QUERY_SOURCE_UNITTEST:
167
- case QUERY_SOURCE_UNKNOWN:
168
- break;
169
- }
170
-}
171
-
172
-void global_statistics_web_request_completed(uint64_t dt,
173
- uint64_t bytes_received,
174
- uint64_t bytes_sent,
175
- uint64_t content_size,
176
- uint64_t compressed_content_size) {
177
- uint64_t old_web_usec_max = global_statistics.web_usec_max;
178
- while(dt > old_web_usec_max)
179
- __atomic_compare_exchange(&global_statistics.web_usec_max, &old_web_usec_max, &dt, 1, __ATOMIC_RELAXED, __ATOMIC_RELAXED);
180
-
181
- __atomic_fetch_add(&global_statistics.web_requests, 1, __ATOMIC_RELAXED);
182
- __atomic_fetch_add(&global_statistics.web_usec, dt, __ATOMIC_RELAXED);
183
- __atomic_fetch_add(&global_statistics.bytes_received, bytes_received, __ATOMIC_RELAXED);
184
- __atomic_fetch_add(&global_statistics.bytes_sent, bytes_sent, __ATOMIC_RELAXED);
185
- __atomic_fetch_add(&global_statistics.content_size, content_size, __ATOMIC_RELAXED);
186
- __atomic_fetch_add(&global_statistics.compressed_content_size, compressed_content_size, __ATOMIC_RELAXED);
187
-}
188
-
189
-uint64_t global_statistics_web_client_connected(void) {
190
- __atomic_fetch_add(&global_statistics.connected_clients, 1, __ATOMIC_RELAXED);
191
- return __atomic_fetch_add(&global_statistics.web_client_count, 1, __ATOMIC_RELAXED);
192
-}
193
-
194
-void global_statistics_web_client_disconnected(void) {
195
- __atomic_fetch_sub(&global_statistics.connected_clients, 1, __ATOMIC_RELAXED);
196
-}
197
-
198
-static inline void global_statistics_copy(struct global_statistics *gs, uint8_t options) {
199
- gs->connected_clients = __atomic_load_n(&global_statistics.connected_clients, __ATOMIC_RELAXED);
200
- gs->web_requests = __atomic_load_n(&global_statistics.web_requests, __ATOMIC_RELAXED);
201
- gs->web_usec = __atomic_load_n(&global_statistics.web_usec, __ATOMIC_RELAXED);
202
- gs->web_usec_max = __atomic_load_n(&global_statistics.web_usec_max, __ATOMIC_RELAXED);
203
- gs->bytes_received = __atomic_load_n(&global_statistics.bytes_received, __ATOMIC_RELAXED);
204
- gs->bytes_sent = __atomic_load_n(&global_statistics.bytes_sent, __ATOMIC_RELAXED);
205
- gs->content_size = __atomic_load_n(&global_statistics.content_size, __ATOMIC_RELAXED);
206
- gs->compressed_content_size = __atomic_load_n(&global_statistics.compressed_content_size, __ATOMIC_RELAXED);
207
- gs->web_client_count = __atomic_load_n(&global_statistics.web_client_count, __ATOMIC_RELAXED);
208
-
209
- gs->api_data_queries_made = __atomic_load_n(&global_statistics.api_data_queries_made, __ATOMIC_RELAXED);
210
- gs->api_data_db_points_read = __atomic_load_n(&global_statistics.api_data_db_points_read, __ATOMIC_RELAXED);
211
- gs->api_data_result_points_generated = __atomic_load_n(&global_statistics.api_data_result_points_generated, __ATOMIC_RELAXED);
212
-
213
- gs->api_weights_queries_made = __atomic_load_n(&global_statistics.api_weights_queries_made, __ATOMIC_RELAXED);
214
- gs->api_weights_db_points_read = __atomic_load_n(&global_statistics.api_weights_db_points_read, __ATOMIC_RELAXED);
215
- gs->api_weights_result_points_generated = __atomic_load_n(&global_statistics.api_weights_result_points_generated, __ATOMIC_RELAXED);
216
-
217
- gs->api_badges_queries_made = __atomic_load_n(&global_statistics.api_badges_queries_made, __ATOMIC_RELAXED);
218
- gs->api_badges_db_points_read = __atomic_load_n(&global_statistics.api_badges_db_points_read, __ATOMIC_RELAXED);
219
- gs->api_badges_result_points_generated = __atomic_load_n(&global_statistics.api_badges_result_points_generated, __ATOMIC_RELAXED);
220
-
221
- gs->health_queries_made = __atomic_load_n(&global_statistics.health_queries_made, __ATOMIC_RELAXED);
222
- gs->health_db_points_read = __atomic_load_n(&global_statistics.health_db_points_read, __ATOMIC_RELAXED);
223
- gs->health_result_points_generated = __atomic_load_n(&global_statistics.health_result_points_generated, __ATOMIC_RELAXED);
224
-
225
- gs->ml_queries_made = __atomic_load_n(&global_statistics.ml_queries_made, __ATOMIC_RELAXED);
226
- gs->ml_db_points_read = __atomic_load_n(&global_statistics.ml_db_points_read, __ATOMIC_RELAXED);
227
- gs->ml_result_points_generated = __atomic_load_n(&global_statistics.ml_result_points_generated, __ATOMIC_RELAXED);
228
- gs->ml_models_consulted = __atomic_load_n(&global_statistics.ml_models_consulted, __ATOMIC_RELAXED);
229
-
230
- gs->exporters_queries_made = __atomic_load_n(&global_statistics.exporters_queries_made, __ATOMIC_RELAXED);
231
- gs->exporters_db_points_read = __atomic_load_n(&global_statistics.exporters_db_points_read, __ATOMIC_RELAXED);
232
- gs->backfill_queries_made = __atomic_load_n(&global_statistics.backfill_queries_made, __ATOMIC_RELAXED);
233
- gs->backfill_db_points_read = __atomic_load_n(&global_statistics.backfill_db_points_read, __ATOMIC_RELAXED);
234
-
235
- gs->tier0_hot_gorilla_buffers = __atomic_load_n(&global_statistics.tier0_hot_gorilla_buffers, __ATOMIC_RELAXED);
236
-
237
- gs->tier0_disk_compressed_bytes = __atomic_load_n(&global_statistics.tier0_disk_compressed_bytes, __ATOMIC_RELAXED);
238
- gs->tier0_disk_uncompressed_bytes = __atomic_load_n(&global_statistics.tier0_disk_uncompressed_bytes, __ATOMIC_RELAXED);
239
-
240
- for(size_t tier = 0; tier < storage_tiers ;tier++)
241
- gs->db_points_stored_per_tier[tier] = __atomic_load_n(&global_statistics.db_points_stored_per_tier[tier], __ATOMIC_RELAXED);
242
-
243
- if(options & GLOBAL_STATS_RESET_WEB_USEC_MAX) {
244
- uint64_t n = 0;
245
- __atomic_compare_exchange(&global_statistics.web_usec_max, (uint64_t *) &gs->web_usec_max, &n, 1, __ATOMIC_RELAXED, __ATOMIC_RELAXED);
246
- }
247
-}
248
-
249
-#define dictionary_stats_memory_total(stats) \
250
- ((stats).memory.dict + (stats).memory.values + (stats).memory.index)
251
-
252
-static void global_statistics_charts(void) {
253
- static unsigned long long old_web_requests = 0, old_web_usec = 0;
254
-
255
- static collected_number average_response_time = -1;
256
-
257
- static time_t netdata_boottime_time = 0;
258
- if (!netdata_boottime_time)
259
- netdata_boottime_time = now_boottime_sec();
260
- time_t netdata_uptime = now_boottime_sec() - netdata_boottime_time;
261
-
262
- struct global_statistics gs;
263
- struct rusage me;
264
-
265
- global_statistics_copy(&gs, GLOBAL_STATS_RESET_WEB_USEC_MAX);
266
- getrusage(RUSAGE_SELF, &me);
267
-
268
- // ----------------------------------------------------------------
269
-
270
- {
271
- static RRDSET *st_cpu = NULL;
272
- static RRDDIM *rd_cpu_user = NULL,
273
- *rd_cpu_system = NULL;
274
-
275
- if (unlikely(!st_cpu)) {
276
- st_cpu = rrdset_create_localhost(
277
- "netdata"
278
- , "server_cpu"
279
- , NULL
280
- , "netdata"
281
- , NULL
282
- , "Netdata CPU usage"
283
- , "milliseconds/s"
284
- , "netdata"
285
- , "stats"
286
- , 130000
287
- , localhost->rrd_update_every
288
- , RRDSET_TYPE_STACKED
289
- );
290
-
291
- rd_cpu_user = rrddim_add(st_cpu, "user", NULL, 1, 1000, RRD_ALGORITHM_INCREMENTAL);
292
- rd_cpu_system = rrddim_add(st_cpu, "system", NULL, 1, 1000, RRD_ALGORITHM_INCREMENTAL);
293
- }
294
-
295
- rrddim_set_by_pointer(st_cpu, rd_cpu_user, me.ru_utime.tv_sec * 1000000ULL + me.ru_utime.tv_usec);
296
- rrddim_set_by_pointer(st_cpu, rd_cpu_system, me.ru_stime.tv_sec * 1000000ULL + me.ru_stime.tv_usec);
297
- rrdset_done(st_cpu);
298
- }
299
-
300
- {
301
- static RRDSET *st_uptime = NULL;
302
- static RRDDIM *rd_uptime = NULL;
303
-
304
- if (unlikely(!st_uptime)) {
305
- st_uptime = rrdset_create_localhost(
306
- "netdata",
307
- "uptime",
308
- NULL,
309
- "netdata",
310
- NULL,
311
- "Netdata uptime",
312
- "seconds",
313
- "netdata",
314
- "stats",
315
- 130150,
316
- localhost->rrd_update_every,
317
- RRDSET_TYPE_LINE);
318
-
319
- rd_uptime = rrddim_add(st_uptime, "uptime", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
320
- }
321
-
322
- rrddim_set_by_pointer(st_uptime, rd_uptime, netdata_uptime);
323
- rrdset_done(st_uptime);
324
- }
325
-
326
- // ----------------------------------------------------------------
327
-
328
- {
329
- static RRDSET *st_clients = NULL;
330
- static RRDDIM *rd_clients = NULL;
331
-
332
- if (unlikely(!st_clients)) {
333
- st_clients = rrdset_create_localhost(
334
- "netdata"
335
- , "clients"
336
- , NULL
337
- , "api"
338
- , NULL
339
- , "Netdata Web Clients"
340
- , "connected clients"
341
- , "netdata"
342
- , "stats"
343
- , 130200
344
- , localhost->rrd_update_every
345
- , RRDSET_TYPE_LINE
346
- );
347
-
348
- rd_clients = rrddim_add(st_clients, "clients", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
349
- }
350
-
351
- rrddim_set_by_pointer(st_clients, rd_clients, gs.connected_clients);
352
- rrdset_done(st_clients);
353
- }
354
-
355
- // ----------------------------------------------------------------
356
-
357
- {
358
- static RRDSET *st_reqs = NULL;
359
- static RRDDIM *rd_requests = NULL;
360
-
361
- if (unlikely(!st_reqs)) {
362
- st_reqs = rrdset_create_localhost(
363
- "netdata"
364
- , "requests"
365
- , NULL
366
- , "api"
367
- , NULL
368
- , "Netdata Web Requests"
369
- , "requests/s"
370
- , "netdata"
371
- , "stats"
372
- , 130300
373
- , localhost->rrd_update_every
374
- , RRDSET_TYPE_LINE
375
- );
376
-
377
- rd_requests = rrddim_add(st_reqs, "requests", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
378
- }
379
-
380
- rrddim_set_by_pointer(st_reqs, rd_requests, (collected_number) gs.web_requests);
381
- rrdset_done(st_reqs);
382
- }
383
-
384
- // ----------------------------------------------------------------
385
-
386
- {
387
- static RRDSET *st_bytes = NULL;
388
- static RRDDIM *rd_in = NULL,
389
- *rd_out = NULL;
390
-
391
- if (unlikely(!st_bytes)) {
392
- st_bytes = rrdset_create_localhost(
393
- "netdata"
394
- , "net"
395
- , NULL
396
- , "api"
397
- , NULL
398
- , "Netdata Network Traffic"
399
- , "kilobits/s"
400
- , "netdata"
401
- , "stats"
402
- , 130400
403
- , localhost->rrd_update_every
404
- , RRDSET_TYPE_AREA
405
- );
406
-
407
- rd_in = rrddim_add(st_bytes, "in", NULL, 8, BITS_IN_A_KILOBIT, RRD_ALGORITHM_INCREMENTAL);
408
- rd_out = rrddim_add(st_bytes, "out", NULL, -8, BITS_IN_A_KILOBIT, RRD_ALGORITHM_INCREMENTAL);
409
- }
410
-
411
- rrddim_set_by_pointer(st_bytes, rd_in, (collected_number) gs.bytes_received);
412
- rrddim_set_by_pointer(st_bytes, rd_out, (collected_number) gs.bytes_sent);
413
- rrdset_done(st_bytes);
414
- }
415
-
416
- // ----------------------------------------------------------------
417
-
418
- {
419
- static RRDSET *st_duration = NULL;
420
- static RRDDIM *rd_average = NULL,
421
- *rd_max = NULL;
422
-
423
- if (unlikely(!st_duration)) {
424
- st_duration = rrdset_create_localhost(
425
- "netdata"
426
- , "response_time"
427
- , NULL
428
- , "api"
429
- , NULL
430
- , "Netdata API Response Time"
431
- , "milliseconds/request"
432
- , "netdata"
433
- , "stats"
434
- , 130500
435
- , localhost->rrd_update_every
436
- , RRDSET_TYPE_LINE
437
- );
438
-
439
- rd_average = rrddim_add(st_duration, "average", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
440
- rd_max = rrddim_add(st_duration, "max", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
441
- }
442
-
443
- uint64_t gweb_usec = gs.web_usec;
444
- uint64_t gweb_requests = gs.web_requests;
445
-
446
- uint64_t web_usec = (gweb_usec >= old_web_usec) ? gweb_usec - old_web_usec : 0;
447
- uint64_t web_requests = (gweb_requests >= old_web_requests) ? gweb_requests - old_web_requests : 0;
448
-
449
- old_web_usec = gweb_usec;
450
- old_web_requests = gweb_requests;
451
-
452
- if (web_requests)
453
- average_response_time = (collected_number) (web_usec / web_requests);
454
-
455
- if (unlikely(average_response_time != -1))
456
- rrddim_set_by_pointer(st_duration, rd_average, average_response_time);
457
- else
458
- rrddim_set_by_pointer(st_duration, rd_average, 0);
459
-
460
- rrddim_set_by_pointer(st_duration, rd_max, ((gs.web_usec_max)?(collected_number)gs.web_usec_max:average_response_time));
461
- rrdset_done(st_duration);
462
- }
463
-}
464
-
465
-static void global_statistics_extended_charts(void) {
466
- static unsigned long long old_content_size = 0, old_compressed_content_size = 0;
467
- static collected_number compression_ratio = -1;
468
-
469
- struct global_statistics gs;
470
- struct replication_query_statistics replication = replication_get_query_statistics();
471
- global_statistics_copy(&gs, GLOBAL_STATS_RESET_WEB_USEC_MAX);
472
-
473
- {
474
- static RRDSET *st_memory = NULL;
475
- static RRDDIM *rd_database = NULL;
476
- static RRDDIM *rd_collectors = NULL;
477
- static RRDDIM *rd_hosts = NULL;
478
- static RRDDIM *rd_rrd = NULL;
479
- static RRDDIM *rd_contexts = NULL;
480
- static RRDDIM *rd_health = NULL;
481
- static RRDDIM *rd_functions = NULL;
482
- static RRDDIM *rd_labels = NULL;
483
- static RRDDIM *rd_strings = NULL;
484
- static RRDDIM *rd_streaming = NULL;
485
- static RRDDIM *rd_replication = NULL;
486
- static RRDDIM *rd_buffers = NULL;
487
- static RRDDIM *rd_workers = NULL;
488
- static RRDDIM *rd_aral = NULL;
489
- static RRDDIM *rd_judy = NULL;
490
- static RRDDIM *rd_other = NULL;
491
-
492
- if (unlikely(!st_memory)) {
493
- st_memory = rrdset_create_localhost(
494
- "netdata",
495
- "memory",
496
- NULL,
497
- "netdata",
498
- NULL,
499
- "Netdata Memory",
500
- "bytes",
501
- "netdata",
502
- "stats",
503
- 130100,
504
- localhost->rrd_update_every,
505
- RRDSET_TYPE_STACKED);
506
-
507
- rd_database = rrddim_add(st_memory, "db", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
508
- rd_collectors = rrddim_add(st_memory, "collectors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
509
- rd_hosts = rrddim_add(st_memory, "hosts", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
510
- rd_rrd = rrddim_add(st_memory, "rrdset rrddim", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
511
- rd_contexts = rrddim_add(st_memory, "contexts", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
512
- rd_health = rrddim_add(st_memory, "health", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
513
- rd_functions = rrddim_add(st_memory, "functions", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
514
- rd_labels = rrddim_add(st_memory, "labels", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
515
- rd_strings = rrddim_add(st_memory, "strings", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
516
- rd_streaming = rrddim_add(st_memory, "streaming", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
517
- rd_replication = rrddim_add(st_memory, "replication", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
518
- rd_buffers = rrddim_add(st_memory, "buffers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
519
- rd_workers = rrddim_add(st_memory, "workers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
520
- rd_aral = rrddim_add(st_memory, "aral", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
521
- rd_judy = rrddim_add(st_memory, "judy", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
522
- rd_other = rrddim_add(st_memory, "other", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
523
- }
524
-
525
- size_t buffers =
526
- netdata_buffers_statistics.query_targets_size +
527
- netdata_buffers_statistics.rrdset_done_rda_size +
528
- netdata_buffers_statistics.buffers_aclk +
529
- netdata_buffers_statistics.buffers_api +
530
- netdata_buffers_statistics.buffers_functions +
531
- netdata_buffers_statistics.buffers_sqlite +
532
- netdata_buffers_statistics.buffers_exporters +
533
- netdata_buffers_statistics.buffers_health +
534
- netdata_buffers_statistics.buffers_streaming +
535
- netdata_buffers_statistics.cbuffers_streaming +
536
- netdata_buffers_statistics.buffers_web +
537
- replication_allocated_buffers() +
538
- aral_by_size_overhead() +
539
- judy_aral_overhead();
540
-
541
- size_t strings = 0;
542
- string_statistics(NULL, NULL, NULL, NULL, NULL, &strings, NULL, NULL);
543
-
544
- rrddim_set_by_pointer(st_memory, rd_database, (collected_number)dbengine_total_memory + (collected_number)rrddim_db_memory_size);
545
- rrddim_set_by_pointer(st_memory, rd_collectors, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_collectors));
546
- rrddim_set_by_pointer(st_memory, rd_hosts, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdhost) + (collected_number)netdata_buffers_statistics.rrdhost_allocations_size);
547
- rrddim_set_by_pointer(st_memory, rd_rrd, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdset_rrddim));
548
- rrddim_set_by_pointer(st_memory, rd_contexts, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdcontext));
549
- rrddim_set_by_pointer(st_memory, rd_health, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdhealth));
550
- rrddim_set_by_pointer(st_memory, rd_functions, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_functions));
551
- rrddim_set_by_pointer(st_memory, rd_labels, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdlabels));
552
- rrddim_set_by_pointer(st_memory, rd_strings, (collected_number)strings);
553
- rrddim_set_by_pointer(st_memory, rd_streaming, (collected_number)netdata_buffers_statistics.rrdhost_senders + (collected_number)netdata_buffers_statistics.rrdhost_receivers);
554
- rrddim_set_by_pointer(st_memory, rd_replication, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_replication) + (collected_number)replication_allocated_memory());
555
- rrddim_set_by_pointer(st_memory, rd_buffers, (collected_number)buffers);
556
- rrddim_set_by_pointer(st_memory, rd_workers, (collected_number) workers_allocated_memory());
557
- rrddim_set_by_pointer(st_memory, rd_aral, (collected_number) aral_by_size_structures());
558
- rrddim_set_by_pointer(st_memory, rd_judy, (collected_number) judy_aral_structures());
559
- rrddim_set_by_pointer(st_memory, rd_other, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_other));
560
-
561
- rrdset_done(st_memory);
562
- }
563
-
564
- {
565
- static RRDSET *st_memory_buffers = NULL;
566
- static RRDDIM *rd_queries = NULL;
567
- static RRDDIM *rd_collectors = NULL;
568
- static RRDDIM *rd_buffers_aclk = NULL;
569
- static RRDDIM *rd_buffers_api = NULL;
570
- static RRDDIM *rd_buffers_functions = NULL;
571
- static RRDDIM *rd_buffers_sqlite = NULL;
572
- static RRDDIM *rd_buffers_exporters = NULL;
573
- static RRDDIM *rd_buffers_health = NULL;
574
- static RRDDIM *rd_buffers_streaming = NULL;
575
- static RRDDIM *rd_cbuffers_streaming = NULL;
576
- static RRDDIM *rd_buffers_replication = NULL;
577
- static RRDDIM *rd_buffers_web = NULL;
578
- static RRDDIM *rd_buffers_aral = NULL;
579
- static RRDDIM *rd_buffers_judy = NULL;
580
-
581
- if (unlikely(!st_memory_buffers)) {
582
- st_memory_buffers = rrdset_create_localhost(
583
- "netdata",
584
- "memory_buffers",
585
- NULL,
586
- "netdata",
587
- NULL,
588
- "Netdata Memory Buffers",
589
- "bytes",
590
- "netdata",
591
- "stats",
592
- 130101,
593
- localhost->rrd_update_every,
594
- RRDSET_TYPE_STACKED);
595
-
596
- rd_queries = rrddim_add(st_memory_buffers, "queries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
597
- rd_collectors = rrddim_add(st_memory_buffers, "collection", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
598
- rd_buffers_aclk = rrddim_add(st_memory_buffers, "aclk", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
599
- rd_buffers_api = rrddim_add(st_memory_buffers, "api", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
600
- rd_buffers_functions = rrddim_add(st_memory_buffers, "functions", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
601
- rd_buffers_sqlite = rrddim_add(st_memory_buffers, "sqlite", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
602
- rd_buffers_exporters = rrddim_add(st_memory_buffers, "exporters", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
603
- rd_buffers_health = rrddim_add(st_memory_buffers, "health", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
604
- rd_buffers_streaming = rrddim_add(st_memory_buffers, "streaming", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
605
- rd_cbuffers_streaming = rrddim_add(st_memory_buffers, "streaming cbuf", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
606
- rd_buffers_replication = rrddim_add(st_memory_buffers, "replication", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
607
- rd_buffers_web = rrddim_add(st_memory_buffers, "web", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
608
- rd_buffers_aral = rrddim_add(st_memory_buffers, "aral", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
609
- rd_buffers_judy = rrddim_add(st_memory_buffers, "judy", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
610
- }
611
-
612
- rrddim_set_by_pointer(st_memory_buffers, rd_queries, (collected_number)netdata_buffers_statistics.query_targets_size + (collected_number) onewayalloc_allocated_memory());
613
- rrddim_set_by_pointer(st_memory_buffers, rd_collectors, (collected_number)netdata_buffers_statistics.rrdset_done_rda_size);
614
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_aclk, (collected_number)netdata_buffers_statistics.buffers_aclk);
615
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_api, (collected_number)netdata_buffers_statistics.buffers_api);
616
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_functions, (collected_number)netdata_buffers_statistics.buffers_functions);
617
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_sqlite, (collected_number)netdata_buffers_statistics.buffers_sqlite);
618
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_exporters, (collected_number)netdata_buffers_statistics.buffers_exporters);
619
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_health, (collected_number)netdata_buffers_statistics.buffers_health);
620
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_streaming, (collected_number)netdata_buffers_statistics.buffers_streaming);
621
- rrddim_set_by_pointer(st_memory_buffers, rd_cbuffers_streaming, (collected_number)netdata_buffers_statistics.cbuffers_streaming);
622
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_replication, (collected_number)replication_allocated_buffers());
623
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_web, (collected_number)netdata_buffers_statistics.buffers_web);
624
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_aral, (collected_number)aral_by_size_overhead());
625
- rrddim_set_by_pointer(st_memory_buffers, rd_buffers_judy, (collected_number)judy_aral_overhead());
626
-
627
- rrdset_done(st_memory_buffers);
628
- }
629
-
630
-
631
- {
632
- static RRDSET *st_compression = NULL;
633
- static RRDDIM *rd_savings = NULL;
634
-
635
- if (unlikely(!st_compression)) {
636
- st_compression = rrdset_create_localhost(
637
- "netdata"
638
- , "compression_ratio"
639
- , NULL
640
- , "api"
641
- , NULL
642
- , "Netdata API Responses Compression Savings Ratio"
643
- , "percentage"
644
- , "netdata"
645
- , "stats"
646
- , 130600
647
- , localhost->rrd_update_every
648
- , RRDSET_TYPE_LINE
649
- );
650
-
651
- rd_savings = rrddim_add(st_compression, "savings", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
652
- }
653
-
654
- // since we don't lock here to read the global statistics
655
- // read the smaller value first
656
- unsigned long long gcompressed_content_size = gs.compressed_content_size;
657
- unsigned long long gcontent_size = gs.content_size;
658
-
659
- unsigned long long compressed_content_size = gcompressed_content_size - old_compressed_content_size;
660
- unsigned long long content_size = gcontent_size - old_content_size;
661
-
662
- old_compressed_content_size = gcompressed_content_size;
663
- old_content_size = gcontent_size;
664
-
665
- if (content_size && content_size >= compressed_content_size)
666
- compression_ratio = ((content_size - compressed_content_size) * 100 * 1000) / content_size;
667
-
668
- if (compression_ratio != -1)
669
- rrddim_set_by_pointer(st_compression, rd_savings, compression_ratio);
670
-
671
- rrdset_done(st_compression);
672
- }
673
-
674
- {
675
- static RRDSET *st_queries = NULL;
676
- static RRDDIM *rd_api_data_queries = NULL;
677
- static RRDDIM *rd_api_weights_queries = NULL;
678
- static RRDDIM *rd_api_badges_queries = NULL;
679
- static RRDDIM *rd_health_queries = NULL;
680
- static RRDDIM *rd_ml_queries = NULL;
681
- static RRDDIM *rd_exporters_queries = NULL;
682
- static RRDDIM *rd_backfill_queries = NULL;
683
- static RRDDIM *rd_replication_queries = NULL;
684
-
685
- if (unlikely(!st_queries)) {
686
- st_queries = rrdset_create_localhost(
687
- "netdata"
688
- , "queries"
689
- , NULL
690
- , "queries"
691
- , NULL
692
- , "Netdata DB Queries"
693
- , "queries/s"
694
- , "netdata"
695
- , "stats"
696
- , 131000
697
- , localhost->rrd_update_every
698
- , RRDSET_TYPE_STACKED
699
- );
700
-
701
- rd_api_data_queries = rrddim_add(st_queries, "/api/v1/data", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
702
- rd_api_weights_queries = rrddim_add(st_queries, "/api/v1/weights", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
703
- rd_api_badges_queries = rrddim_add(st_queries, "/api/v1/badge", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
704
- rd_health_queries = rrddim_add(st_queries, "health", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
705
- rd_ml_queries = rrddim_add(st_queries, "ml", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
706
- rd_exporters_queries = rrddim_add(st_queries, "exporters", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
707
- rd_backfill_queries = rrddim_add(st_queries, "backfill", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
708
- rd_replication_queries = rrddim_add(st_queries, "replication", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
709
- }
710
-
711
- rrddim_set_by_pointer(st_queries, rd_api_data_queries, (collected_number)gs.api_data_queries_made);
712
- rrddim_set_by_pointer(st_queries, rd_api_weights_queries, (collected_number)gs.api_weights_queries_made);
713
- rrddim_set_by_pointer(st_queries, rd_api_badges_queries, (collected_number)gs.api_badges_queries_made);
714
- rrddim_set_by_pointer(st_queries, rd_health_queries, (collected_number)gs.health_queries_made);
715
- rrddim_set_by_pointer(st_queries, rd_ml_queries, (collected_number)gs.ml_queries_made);
716
- rrddim_set_by_pointer(st_queries, rd_exporters_queries, (collected_number)gs.exporters_queries_made);
717
- rrddim_set_by_pointer(st_queries, rd_backfill_queries, (collected_number)gs.backfill_queries_made);
718
- rrddim_set_by_pointer(st_queries, rd_replication_queries, (collected_number)replication.queries_finished);
719
-
720
- rrdset_done(st_queries);
721
- }
722
-
723
- {
724
- static RRDSET *st_points_read = NULL;
725
- static RRDDIM *rd_api_data_points_read = NULL;
726
- static RRDDIM *rd_api_weights_points_read = NULL;
727
- static RRDDIM *rd_api_badges_points_read = NULL;
728
- static RRDDIM *rd_health_points_read = NULL;
729
- static RRDDIM *rd_ml_points_read = NULL;
730
- static RRDDIM *rd_exporters_points_read = NULL;
731
- static RRDDIM *rd_backfill_points_read = NULL;
732
- static RRDDIM *rd_replication_points_read = NULL;
733
-
734
- if (unlikely(!st_points_read)) {
735
- st_points_read = rrdset_create_localhost(
736
- "netdata"
737
- , "db_points_read"
738
- , NULL
739
- , "queries"
740
- , NULL
741
- , "Netdata DB Points Query Read"
742
- , "points/s"
743
- , "netdata"
744
- , "stats"
745
- , 131001
746
- , localhost->rrd_update_every
747
- , RRDSET_TYPE_STACKED
748
- );
749
-
750
- rd_api_data_points_read = rrddim_add(st_points_read, "/api/v1/data", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
751
- rd_api_weights_points_read = rrddim_add(st_points_read, "/api/v1/weights", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
752
- rd_api_badges_points_read = rrddim_add(st_points_read, "/api/v1/badge", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
753
- rd_health_points_read = rrddim_add(st_points_read, "health", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
754
- rd_ml_points_read = rrddim_add(st_points_read, "ml", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
755
- rd_exporters_points_read = rrddim_add(st_points_read, "exporters", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
756
- rd_backfill_points_read = rrddim_add(st_points_read, "backfill", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
757
- rd_replication_points_read = rrddim_add(st_points_read, "replication", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
758
- }
759
-
760
- rrddim_set_by_pointer(st_points_read, rd_api_data_points_read, (collected_number)gs.api_data_db_points_read);
761
- rrddim_set_by_pointer(st_points_read, rd_api_weights_points_read, (collected_number)gs.api_weights_db_points_read);
762
- rrddim_set_by_pointer(st_points_read, rd_api_badges_points_read, (collected_number)gs.api_badges_db_points_read);
763
- rrddim_set_by_pointer(st_points_read, rd_health_points_read, (collected_number)gs.health_db_points_read);
764
- rrddim_set_by_pointer(st_points_read, rd_ml_points_read, (collected_number)gs.ml_db_points_read);
765
- rrddim_set_by_pointer(st_points_read, rd_exporters_points_read, (collected_number)gs.exporters_db_points_read);
766
- rrddim_set_by_pointer(st_points_read, rd_backfill_points_read, (collected_number)gs.backfill_db_points_read);
767
- rrddim_set_by_pointer(st_points_read, rd_replication_points_read, (collected_number)replication.points_read);
768
-
769
- rrdset_done(st_points_read);
770
- }
771
-
772
- if(gs.api_data_result_points_generated || replication.points_generated) {
773
- static RRDSET *st_points_generated = NULL;
774
- static RRDDIM *rd_api_data_points_generated = NULL;
775
- static RRDDIM *rd_api_weights_points_generated = NULL;
776
- static RRDDIM *rd_api_badges_points_generated = NULL;
777
- static RRDDIM *rd_health_points_generated = NULL;
778
- static RRDDIM *rd_ml_points_generated = NULL;
779
- static RRDDIM *rd_replication_points_generated = NULL;
780
-
781
- if (unlikely(!st_points_generated)) {
782
- st_points_generated = rrdset_create_localhost(
783
- "netdata"
784
- , "db_points_results"
785
- , NULL
786
- , "queries"
787
- , NULL
788
- , "Netdata Points in Query Results"
789
- , "points/s"
790
- , "netdata"
791
- , "stats"
792
- , 131002
793
- , localhost->rrd_update_every
794
- , RRDSET_TYPE_STACKED
795
- );
796
-
797
- rd_api_data_points_generated = rrddim_add(st_points_generated, "/api/v1/data", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
798
- rd_api_weights_points_generated = rrddim_add(st_points_generated, "/api/v1/weights", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
799
- rd_api_badges_points_generated = rrddim_add(st_points_generated, "/api/v1/badge", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
800
- rd_health_points_generated = rrddim_add(st_points_generated, "health", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
801
- rd_ml_points_generated = rrddim_add(st_points_generated, "ml", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
802
- rd_replication_points_generated = rrddim_add(st_points_generated, "replication", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
803
- }
804
-
805
- rrddim_set_by_pointer(st_points_generated, rd_api_data_points_generated, (collected_number)gs.api_data_result_points_generated);
806
- rrddim_set_by_pointer(st_points_generated, rd_api_weights_points_generated, (collected_number)gs.api_weights_result_points_generated);
807
- rrddim_set_by_pointer(st_points_generated, rd_api_badges_points_generated, (collected_number)gs.api_badges_result_points_generated);
808
- rrddim_set_by_pointer(st_points_generated, rd_health_points_generated, (collected_number)gs.health_result_points_generated);
809
- rrddim_set_by_pointer(st_points_generated, rd_ml_points_generated, (collected_number)gs.ml_result_points_generated);
810
- rrddim_set_by_pointer(st_points_generated, rd_replication_points_generated, (collected_number)replication.points_generated);
811
-
812
- rrdset_done(st_points_generated);
813
- }
814
-
815
- ml_update_global_statistics_charts(gs.ml_models_consulted);
816
-
817
- {
818
- static RRDSET *st_points_stored = NULL;
819
- static RRDDIM *rds[RRD_STORAGE_TIERS] = {};
820
-
821
- if (unlikely(!st_points_stored)) {
822
- st_points_stored = rrdset_create_localhost(
823
- "netdata"
824
- , "db_points_stored"
825
- , NULL
826
- , "queries"
827
- , NULL
828
- , "Netdata DB Points Stored"
829
- , "points/s"
830
- , "netdata"
831
- , "stats"
832
- , 131003
833
- , localhost->rrd_update_every
834
- , RRDSET_TYPE_STACKED
835
- );
836
-
837
- for(size_t tier = 0; tier < storage_tiers ;tier++) {
838
- char buf[30 + 1];
839
- snprintfz(buf, sizeof(buf) - 1, "tier%zu", tier);
840
- rds[tier] = rrddim_add(st_points_stored, buf, NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
841
- }
842
- }
843
-
844
- for(size_t tier = 0; tier < storage_tiers ;tier++)
845
- rrddim_set_by_pointer(st_points_stored, rds[tier], (collected_number)gs.db_points_stored_per_tier[tier]);
846
-
847
- rrdset_done(st_points_stored);
848
- }
849
-
850
-#ifdef ENABLE_DBENGINE
851
- if (tier_page_type[0] == RRDENG_PAGE_TYPE_GORILLA_32BIT)
852
- {
853
- static RRDSET *st_tier0_gorilla_pages = NULL;
854
- static RRDDIM *rd_num_gorilla_pages = NULL;
855
-
856
- if (unlikely(!st_tier0_gorilla_pages)) {
857
- st_tier0_gorilla_pages = rrdset_create_localhost(
858
- "netdata"
859
- , "tier0_gorilla_pages"
860
- , NULL
861
- , "dbengine gorilla"
862
- , NULL
863
- , "Number of gorilla_pages"
864
- , "count"
865
- , "netdata"
866
- , "stats"
867
- , 131004
868
- , localhost->rrd_update_every
869
- , RRDSET_TYPE_LINE
870
- );
871
-
872
- rd_num_gorilla_pages = rrddim_add(st_tier0_gorilla_pages, "count", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
873
- }
874
-
875
- rrddim_set_by_pointer(st_tier0_gorilla_pages, rd_num_gorilla_pages, (collected_number)gs.tier0_hot_gorilla_buffers);
876
-
877
- rrdset_done(st_tier0_gorilla_pages);
878
- }
879
-
880
- if (tier_page_type[0] == RRDENG_PAGE_TYPE_GORILLA_32BIT)
881
- {
882
- static RRDSET *st_tier0_compression_info = NULL;
883
-
884
- static RRDDIM *rd_compressed_bytes = NULL;
885
- static RRDDIM *rd_uncompressed_bytes = NULL;
886
-
887
- if (unlikely(!st_tier0_compression_info)) {
888
- st_tier0_compression_info = rrdset_create_localhost(
889
- "netdata"
890
- , "tier0_compression_info"
891
- , NULL
892
- , "dbengine gorilla"
893
- , NULL
894
- , "Tier 0 compression info"
895
- , "bytes"
896
- , "netdata"
897
- , "stats"
898
- , 131005
899
- , localhost->rrd_update_every
900
- , RRDSET_TYPE_LINE
901
- );
902
-
903
- rd_compressed_bytes = rrddim_add(st_tier0_compression_info, "compressed", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
904
- rd_uncompressed_bytes = rrddim_add(st_tier0_compression_info, "uncompressed", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
905
- }
906
-
907
- rrddim_set_by_pointer(st_tier0_compression_info, rd_compressed_bytes, (collected_number)gs.tier0_disk_compressed_bytes);
908
- rrddim_set_by_pointer(st_tier0_compression_info, rd_uncompressed_bytes, (collected_number)gs.tier0_disk_uncompressed_bytes);
909
-
910
- rrdset_done(st_tier0_compression_info);
911
- }
912
-#endif
913
-}
914
-// ----------------------------------------------------------------------------
915
-// sqlite3 statistics
916
-
917
-struct sqlite3_statistics {
918
- uint64_t sqlite3_queries_made;
919
- uint64_t sqlite3_queries_ok;
920
- uint64_t sqlite3_queries_failed;
921
- uint64_t sqlite3_queries_failed_busy;
922
- uint64_t sqlite3_queries_failed_locked;
923
- uint64_t sqlite3_rows;
924
- uint64_t sqlite3_metadata_cache_hit;
925
- uint64_t sqlite3_context_cache_hit;
926
- uint64_t sqlite3_metadata_cache_miss;
927
- uint64_t sqlite3_context_cache_miss;
928
- uint64_t sqlite3_metadata_cache_spill;
929
- uint64_t sqlite3_context_cache_spill;
930
- uint64_t sqlite3_metadata_cache_write;
931
- uint64_t sqlite3_context_cache_write;
932
-
933
-} sqlite3_statistics = { };
934
-
935
-void global_statistics_sqlite3_query_completed(bool success, bool busy, bool locked) {
936
- __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_made, 1, __ATOMIC_RELAXED);
937
-
938
- if(success) {
939
- __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_ok, 1, __ATOMIC_RELAXED);
940
- }
941
- else {
942
- __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_failed, 1, __ATOMIC_RELAXED);
943
-
944
- if(busy)
945
- __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_failed_busy, 1, __ATOMIC_RELAXED);
946
-
947
- if(locked)
948
- __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_failed_locked, 1, __ATOMIC_RELAXED);
949
- }
950
-}
951
-
952
-void global_statistics_sqlite3_row_completed(void) {
953
- __atomic_fetch_add(&sqlite3_statistics.sqlite3_rows, 1, __ATOMIC_RELAXED);
954
-}
955
-
956
-static inline void sqlite3_statistics_copy(struct sqlite3_statistics *gs) {
957
- static usec_t last_run = 0;
958
-
959
- gs->sqlite3_queries_made = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_made, __ATOMIC_RELAXED);
960
- gs->sqlite3_queries_ok = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_ok, __ATOMIC_RELAXED);
961
- gs->sqlite3_queries_failed = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_failed, __ATOMIC_RELAXED);
962
- gs->sqlite3_queries_failed_busy = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_failed_busy, __ATOMIC_RELAXED);
963
- gs->sqlite3_queries_failed_locked = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_failed_locked, __ATOMIC_RELAXED);
964
- gs->sqlite3_rows = __atomic_load_n(&sqlite3_statistics.sqlite3_rows, __ATOMIC_RELAXED);
965
-
966
- usec_t timeout = default_rrd_update_every * USEC_PER_SEC + default_rrd_update_every * USEC_PER_SEC / 3;
967
- usec_t now = now_monotonic_usec();
968
- if(!last_run)
969
- last_run = now;
970
- usec_t delta = now - last_run;
971
- bool query_sqlite3 = delta < timeout;
972
-
973
- if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
974
- gs->sqlite3_metadata_cache_hit = (uint64_t) sql_metadata_cache_stats(SQLITE_DBSTATUS_CACHE_HIT);
975
- else {
976
- gs->sqlite3_metadata_cache_hit = UINT64_MAX;
977
- query_sqlite3 = false;
978
- }
979
-
980
- if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
981
- gs->sqlite3_context_cache_hit = (uint64_t) sql_context_cache_stats(SQLITE_DBSTATUS_CACHE_HIT);
982
- else {
983
- gs->sqlite3_context_cache_hit = UINT64_MAX;
984
- query_sqlite3 = false;
985
- }
986
-
987
- if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
988
- gs->sqlite3_metadata_cache_miss = (uint64_t) sql_metadata_cache_stats(SQLITE_DBSTATUS_CACHE_MISS);
989
- else {
990
- gs->sqlite3_metadata_cache_miss = UINT64_MAX;
991
- query_sqlite3 = false;
992
- }
993
-
994
- if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
995
- gs->sqlite3_context_cache_miss = (uint64_t) sql_context_cache_stats(SQLITE_DBSTATUS_CACHE_MISS);
996
- else {
997
- gs->sqlite3_context_cache_miss = UINT64_MAX;
998
- query_sqlite3 = false;
999
- }
1000
-
1001
- if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
1002
- gs->sqlite3_metadata_cache_spill = (uint64_t) sql_metadata_cache_stats(SQLITE_DBSTATUS_CACHE_SPILL);
1003
- else {
1004
- gs->sqlite3_metadata_cache_spill = UINT64_MAX;
1005
- query_sqlite3 = false;
1006
- }
1007
-
1008
- if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
1009
- gs->sqlite3_context_cache_spill = (uint64_t) sql_context_cache_stats(SQLITE_DBSTATUS_CACHE_SPILL);
1010
- else {
1011
- gs->sqlite3_context_cache_spill = UINT64_MAX;
1012
- query_sqlite3 = false;
1013
- }
1014
-
1015
- if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
1016
- gs->sqlite3_metadata_cache_write = (uint64_t) sql_metadata_cache_stats(SQLITE_DBSTATUS_CACHE_WRITE);
1017
- else {
1018
- gs->sqlite3_metadata_cache_write = UINT64_MAX;
1019
- query_sqlite3 = false;
1020
- }
1021
-
1022
- if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
1023
- gs->sqlite3_context_cache_write = (uint64_t) sql_context_cache_stats(SQLITE_DBSTATUS_CACHE_WRITE);
1024
- else {
1025
- gs->sqlite3_context_cache_write = UINT64_MAX;
1026
- query_sqlite3 = false;
1027
- }
1028
-
1029
- last_run = now_monotonic_usec();
1030
-}
1031
-
1032
-static void sqlite3_statistics_charts(void) {
1033
- struct sqlite3_statistics gs;
1034
- sqlite3_statistics_copy(&gs);
1035
-
1036
- if(gs.sqlite3_queries_made) {
1037
- static RRDSET *st_sqlite3_queries = NULL;
1038
- static RRDDIM *rd_queries = NULL;
1039
-
1040
- if (unlikely(!st_sqlite3_queries)) {
1041
- st_sqlite3_queries = rrdset_create_localhost(
1042
- "netdata"
1043
- , "sqlite3_queries"
1044
- , NULL
1045
- , "sqlite3"
1046
- , NULL
1047
- , "Netdata SQLite3 Queries"
1048
- , "queries/s"
1049
- , "netdata"
1050
- , "stats"
1051
- , 131100
1052
- , localhost->rrd_update_every
1053
- , RRDSET_TYPE_LINE
1054
- );
1055
-
1056
- rd_queries = rrddim_add(st_sqlite3_queries, "queries", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1057
- }
1058
-
1059
- rrddim_set_by_pointer(st_sqlite3_queries, rd_queries, (collected_number)gs.sqlite3_queries_made);
1060
-
1061
- rrdset_done(st_sqlite3_queries);
1062
- }
1063
-
1064
- // ----------------------------------------------------------------
1065
-
1066
- if(gs.sqlite3_queries_ok || gs.sqlite3_queries_failed) {
1067
- static RRDSET *st_sqlite3_queries_by_status = NULL;
1068
- static RRDDIM *rd_ok = NULL, *rd_failed = NULL, *rd_busy = NULL, *rd_locked = NULL;
1069
-
1070
- if (unlikely(!st_sqlite3_queries_by_status)) {
1071
- st_sqlite3_queries_by_status = rrdset_create_localhost(
1072
- "netdata"
1073
- , "sqlite3_queries_by_status"
1074
- , NULL
1075
- , "sqlite3"
1076
- , NULL
1077
- , "Netdata SQLite3 Queries by status"
1078
- , "queries/s"
1079
- , "netdata"
1080
- , "stats"
1081
- , 131101
1082
- , localhost->rrd_update_every
1083
- , RRDSET_TYPE_LINE
1084
- );
1085
-
1086
- rd_ok = rrddim_add(st_sqlite3_queries_by_status, "ok", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1087
- rd_failed = rrddim_add(st_sqlite3_queries_by_status, "failed", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1088
- rd_busy = rrddim_add(st_sqlite3_queries_by_status, "busy", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1089
- rd_locked = rrddim_add(st_sqlite3_queries_by_status, "locked", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1090
- }
1091
-
1092
- rrddim_set_by_pointer(st_sqlite3_queries_by_status, rd_ok, (collected_number)gs.sqlite3_queries_made);
1093
- rrddim_set_by_pointer(st_sqlite3_queries_by_status, rd_failed, (collected_number)gs.sqlite3_queries_failed);
1094
- rrddim_set_by_pointer(st_sqlite3_queries_by_status, rd_busy, (collected_number)gs.sqlite3_queries_failed_busy);
1095
- rrddim_set_by_pointer(st_sqlite3_queries_by_status, rd_locked, (collected_number)gs.sqlite3_queries_failed_locked);
1096
-
1097
- rrdset_done(st_sqlite3_queries_by_status);
1098
- }
1099
-
1100
- // ----------------------------------------------------------------
1101
-
1102
- if(gs.sqlite3_rows) {
1103
- static RRDSET *st_sqlite3_rows = NULL;
1104
- static RRDDIM *rd_rows = NULL;
1105
-
1106
- if (unlikely(!st_sqlite3_rows)) {
1107
- st_sqlite3_rows = rrdset_create_localhost(
1108
- "netdata"
1109
- , "sqlite3_rows"
1110
- , NULL
1111
- , "sqlite3"
1112
- , NULL
1113
- , "Netdata SQLite3 Rows"
1114
- , "rows/s"
1115
- , "netdata"
1116
- , "stats"
1117
- , 131102
1118
- , localhost->rrd_update_every
1119
- , RRDSET_TYPE_LINE
1120
- );
1121
-
1122
- rd_rows = rrddim_add(st_sqlite3_rows, "ok", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1123
- }
1124
-
1125
- rrddim_set_by_pointer(st_sqlite3_rows, rd_rows, (collected_number)gs.sqlite3_rows);
1126
-
1127
- rrdset_done(st_sqlite3_rows);
1128
- }
1129
-
1130
- if(gs.sqlite3_metadata_cache_hit) {
1131
- static RRDSET *st_sqlite3_cache = NULL;
1132
- static RRDDIM *rd_cache_hit = NULL;
1133
- static RRDDIM *rd_cache_miss= NULL;
1134
- static RRDDIM *rd_cache_spill= NULL;
1135
- static RRDDIM *rd_cache_write= NULL;
1136
-
1137
- if (unlikely(!st_sqlite3_cache)) {
1138
- st_sqlite3_cache = rrdset_create_localhost(
1139
- "netdata"
1140
- , "sqlite3_metatada_cache"
1141
- , NULL
1142
- , "sqlite3"
1143
- , NULL
1144
- , "Netdata SQLite3 metadata cache"
1145
- , "ops/s"
1146
- , "netdata"
1147
- , "stats"
1148
- , 131103
1149
- , localhost->rrd_update_every
1150
- , RRDSET_TYPE_LINE
1151
- );
1152
-
1153
- rd_cache_hit = rrddim_add(st_sqlite3_cache, "cache_hit", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1154
- rd_cache_miss = rrddim_add(st_sqlite3_cache, "cache_miss", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1155
- rd_cache_spill = rrddim_add(st_sqlite3_cache, "cache_spill", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1156
- rd_cache_write = rrddim_add(st_sqlite3_cache, "cache_write", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1157
- }
1158
-
1159
- if(gs.sqlite3_metadata_cache_hit != UINT64_MAX)
1160
- rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_hit, (collected_number)gs.sqlite3_metadata_cache_hit);
1161
-
1162
- if(gs.sqlite3_metadata_cache_miss != UINT64_MAX)
1163
- rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_miss, (collected_number)gs.sqlite3_metadata_cache_miss);
1164
-
1165
- if(gs.sqlite3_metadata_cache_spill != UINT64_MAX)
1166
- rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_spill, (collected_number)gs.sqlite3_metadata_cache_spill);
1167
-
1168
- if(gs.sqlite3_metadata_cache_write != UINT64_MAX)
1169
- rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_write, (collected_number)gs.sqlite3_metadata_cache_write);
1170
-
1171
- rrdset_done(st_sqlite3_cache);
1172
- }
1173
-
1174
- if(gs.sqlite3_context_cache_hit) {
1175
- static RRDSET *st_sqlite3_cache = NULL;
1176
- static RRDDIM *rd_cache_hit = NULL;
1177
- static RRDDIM *rd_cache_miss= NULL;
1178
- static RRDDIM *rd_cache_spill= NULL;
1179
- static RRDDIM *rd_cache_write= NULL;
1180
-
1181
- if (unlikely(!st_sqlite3_cache)) {
1182
- st_sqlite3_cache = rrdset_create_localhost(
1183
- "netdata"
1184
- , "sqlite3_context_cache"
1185
- , NULL
1186
- , "sqlite3"
1187
- , NULL
1188
- , "Netdata SQLite3 context cache"
1189
- , "ops/s"
1190
- , "netdata"
1191
- , "stats"
1192
- , 131104
1193
- , localhost->rrd_update_every
1194
- , RRDSET_TYPE_LINE
1195
- );
1196
-
1197
- rd_cache_hit = rrddim_add(st_sqlite3_cache, "cache_hit", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1198
- rd_cache_miss = rrddim_add(st_sqlite3_cache, "cache_miss", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1199
- rd_cache_spill = rrddim_add(st_sqlite3_cache, "cache_spill", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1200
- rd_cache_write = rrddim_add(st_sqlite3_cache, "cache_write", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1201
- }
1202
-
1203
- if(gs.sqlite3_context_cache_hit != UINT64_MAX)
1204
- rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_hit, (collected_number)gs.sqlite3_context_cache_hit);
1205
-
1206
- if(gs.sqlite3_context_cache_miss != UINT64_MAX)
1207
- rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_miss, (collected_number)gs.sqlite3_context_cache_miss);
1208
-
1209
- if(gs.sqlite3_context_cache_spill != UINT64_MAX)
1210
- rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_spill, (collected_number)gs.sqlite3_context_cache_spill);
1211
-
1212
- if(gs.sqlite3_context_cache_write != UINT64_MAX)
1213
- rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_write, (collected_number)gs.sqlite3_context_cache_write);
1214
-
1215
- rrdset_done(st_sqlite3_cache);
1216
- }
1217
-
1218
- // ----------------------------------------------------------------
1219
-}
1220
-
1221
-#ifdef ENABLE_DBENGINE
1222
-
1223
-struct dbengine2_cache_pointers {
1224
- RRDSET *st_cache_hit_ratio;
1225
- RRDDIM *rd_hit_ratio_closest;
1226
- RRDDIM *rd_hit_ratio_exact;
1227
-
1228
- RRDSET *st_operations;
1229
- RRDDIM *rd_searches_closest;
1230
- RRDDIM *rd_searches_exact;
1231
- RRDDIM *rd_add_hot;
1232
- RRDDIM *rd_add_clean;
1233
- RRDDIM *rd_evictions;
1234
- RRDDIM *rd_flushes;
1235
- RRDDIM *rd_acquires;
1236
- RRDDIM *rd_releases;
1237
- RRDDIM *rd_acquires_for_deletion;
1238
-
1239
- RRDSET *st_pgc_memory;
1240
- RRDDIM *rd_pgc_memory_free;
1241
- RRDDIM *rd_pgc_memory_clean;
1242
- RRDDIM *rd_pgc_memory_hot;
1243
- RRDDIM *rd_pgc_memory_dirty;
1244
- RRDDIM *rd_pgc_memory_index;
1245
- RRDDIM *rd_pgc_memory_evicting;
1246
- RRDDIM *rd_pgc_memory_flushing;
1247
-
1248
- RRDSET *st_pgc_tm;
1249
- RRDDIM *rd_pgc_tm_current;
1250
- RRDDIM *rd_pgc_tm_wanted;
1251
- RRDDIM *rd_pgc_tm_hot_max;
1252
- RRDDIM *rd_pgc_tm_dirty_max;
1253
- RRDDIM *rd_pgc_tm_hot;
1254
- RRDDIM *rd_pgc_tm_dirty;
1255
- RRDDIM *rd_pgc_tm_referenced;
1256
-
1257
- RRDSET *st_pgc_pages;
1258
- RRDDIM *rd_pgc_pages_clean;
1259
- RRDDIM *rd_pgc_pages_hot;
1260
- RRDDIM *rd_pgc_pages_dirty;
1261
- RRDDIM *rd_pgc_pages_referenced;
1262
-
1263
- RRDSET *st_pgc_memory_changes;
1264
- RRDDIM *rd_pgc_memory_new_hot;
1265
- RRDDIM *rd_pgc_memory_new_clean;
1266
- RRDDIM *rd_pgc_memory_clean_evictions;
1267
-
1268
- RRDSET *st_pgc_memory_migrations;
1269
- RRDDIM *rd_pgc_memory_hot_to_dirty;
1270
- RRDDIM *rd_pgc_memory_dirty_to_clean;
1271
-
1272
- RRDSET *st_pgc_workers;
1273
- RRDDIM *rd_pgc_workers_evictors;
1274
- RRDDIM *rd_pgc_workers_flushers;
1275
- RRDDIM *rd_pgc_workers_adders;
1276
- RRDDIM *rd_pgc_workers_searchers;
1277
- RRDDIM *rd_pgc_workers_jv2_flushers;
1278
- RRDDIM *rd_pgc_workers_hot2dirty;
1279
-
1280
- RRDSET *st_pgc_memory_events;
1281
- RRDDIM *rd_pgc_memory_evictions_critical;
1282
- RRDDIM *rd_pgc_memory_evictions_aggressive;
1283
- RRDDIM *rd_pgc_memory_flushes_critical;
1284
-
1285
- RRDSET *st_pgc_waste;
1286
- RRDDIM *rd_pgc_waste_evictions_skipped;
1287
- RRDDIM *rd_pgc_waste_flushes_cancelled;
1288
- RRDDIM *rd_pgc_waste_insert_spins;
1289
- RRDDIM *rd_pgc_waste_evict_spins;
1290
- RRDDIM *rd_pgc_waste_release_spins;
1291
- RRDDIM *rd_pgc_waste_acquire_spins;
1292
- RRDDIM *rd_pgc_waste_delete_spins;
1293
- RRDDIM *rd_pgc_waste_flush_spins;
1294
-
1295
-};
1296
-
1297
-static void dbengine2_cache_statistics_charts(struct dbengine2_cache_pointers *ptrs, struct pgc_statistics *pgc_stats, struct pgc_statistics *pgc_stats_old __maybe_unused, const char *name, int priority) {
1298
-
1299
- {
1300
- if (unlikely(!ptrs->st_cache_hit_ratio)) {
1301
- BUFFER *id = buffer_create(100, NULL);
1302
- buffer_sprintf(id, "dbengine_%s_cache_hit_ratio", name);
1303
-
1304
- BUFFER *family = buffer_create(100, NULL);
1305
- buffer_sprintf(family, "dbengine %s cache", name);
1306
-
1307
- BUFFER *title = buffer_create(100, NULL);
1308
- buffer_sprintf(title, "Netdata %s Cache Hit Ratio", name);
1309
-
1310
- ptrs->st_cache_hit_ratio = rrdset_create_localhost(
1311
- "netdata",
1312
- buffer_tostring(id),
1313
- NULL,
1314
- buffer_tostring(family),
1315
- NULL,
1316
- buffer_tostring(title),
1317
- "%",
1318
- "netdata",
1319
- "stats",
1320
- priority,
1321
- localhost->rrd_update_every,
1322
- RRDSET_TYPE_LINE);
1323
-
1324
- ptrs->rd_hit_ratio_closest = rrddim_add(ptrs->st_cache_hit_ratio, "closest", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1325
- ptrs->rd_hit_ratio_exact = rrddim_add(ptrs->st_cache_hit_ratio, "exact", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1326
-
1327
- buffer_free(id);
1328
- buffer_free(family);
1329
- buffer_free(title);
1330
- priority++;
1331
- }
1332
-
1333
- size_t closest_percent = 100 * 10000;
1334
- if(pgc_stats->searches_closest > pgc_stats_old->searches_closest)
1335
- closest_percent = (pgc_stats->searches_closest_hits - pgc_stats_old->searches_closest_hits) * 100 * 10000 / (pgc_stats->searches_closest - pgc_stats_old->searches_closest);
1336
-
1337
- size_t exact_percent = 100 * 10000;
1338
- if(pgc_stats->searches_exact > pgc_stats_old->searches_exact)
1339
- exact_percent = (pgc_stats->searches_exact_hits - pgc_stats_old->searches_exact_hits) * 100 * 10000 / (pgc_stats->searches_exact - pgc_stats_old->searches_exact);
1340
-
1341
- rrddim_set_by_pointer(ptrs->st_cache_hit_ratio, ptrs->rd_hit_ratio_closest, (collected_number)closest_percent);
1342
- rrddim_set_by_pointer(ptrs->st_cache_hit_ratio, ptrs->rd_hit_ratio_exact, (collected_number)exact_percent);
1343
-
1344
- rrdset_done(ptrs->st_cache_hit_ratio);
1345
- }
1346
-
1347
- {
1348
- if (unlikely(!ptrs->st_operations)) {
1349
- BUFFER *id = buffer_create(100, NULL);
1350
- buffer_sprintf(id, "dbengine_%s_cache_operations", name);
1351
-
1352
- BUFFER *family = buffer_create(100, NULL);
1353
- buffer_sprintf(family, "dbengine %s cache", name);
1354
-
1355
- BUFFER *title = buffer_create(100, NULL);
1356
- buffer_sprintf(title, "Netdata %s Cache Operations", name);
1357
-
1358
- ptrs->st_operations = rrdset_create_localhost(
1359
- "netdata",
1360
- buffer_tostring(id),
1361
- NULL,
1362
- buffer_tostring(family),
1363
- NULL,
1364
- buffer_tostring(title),
1365
- "ops/s",
1366
- "netdata",
1367
- "stats",
1368
- priority,
1369
- localhost->rrd_update_every,
1370
- RRDSET_TYPE_LINE);
1371
-
1372
- ptrs->rd_searches_closest = rrddim_add(ptrs->st_operations, "search closest", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1373
- ptrs->rd_searches_exact = rrddim_add(ptrs->st_operations, "search exact", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1374
- ptrs->rd_add_hot = rrddim_add(ptrs->st_operations, "add hot", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1375
- ptrs->rd_add_clean = rrddim_add(ptrs->st_operations, "add clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1376
- ptrs->rd_evictions = rrddim_add(ptrs->st_operations, "evictions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1377
- ptrs->rd_flushes = rrddim_add(ptrs->st_operations, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1378
- ptrs->rd_acquires = rrddim_add(ptrs->st_operations, "acquires", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1379
- ptrs->rd_releases = rrddim_add(ptrs->st_operations, "releases", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1380
- ptrs->rd_acquires_for_deletion = rrddim_add(ptrs->st_operations, "del acquires", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1381
-
1382
- buffer_free(id);
1383
- buffer_free(family);
1384
- buffer_free(title);
1385
- priority++;
1386
- }
1387
-
1388
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_searches_closest, (collected_number)pgc_stats->searches_closest);
1389
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_searches_exact, (collected_number)pgc_stats->searches_exact);
1390
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_add_hot, (collected_number)pgc_stats->queues.hot.added_entries);
1391
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_add_clean, (collected_number)(pgc_stats->added_entries - pgc_stats->queues.hot.added_entries));
1392
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_evictions, (collected_number)pgc_stats->queues.clean.removed_entries);
1393
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_flushes, (collected_number)pgc_stats->flushes_completed);
1394
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_acquires, (collected_number)pgc_stats->acquires);
1395
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_releases, (collected_number)pgc_stats->releases);
1396
- rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_acquires_for_deletion, (collected_number)pgc_stats->acquires_for_deletion);
1397
-
1398
- rrdset_done(ptrs->st_operations);
1399
- }
1400
-
1401
- {
1402
- if (unlikely(!ptrs->st_pgc_memory)) {
1403
- BUFFER *id = buffer_create(100, NULL);
1404
- buffer_sprintf(id, "dbengine_%s_cache_memory", name);
1405
-
1406
- BUFFER *family = buffer_create(100, NULL);
1407
- buffer_sprintf(family, "dbengine %s cache", name);
1408
-
1409
- BUFFER *title = buffer_create(100, NULL);
1410
- buffer_sprintf(title, "Netdata %s Cache Memory", name);
1411
-
1412
- ptrs->st_pgc_memory = rrdset_create_localhost(
1413
- "netdata",
1414
- buffer_tostring(id),
1415
- NULL,
1416
- buffer_tostring(family),
1417
- NULL,
1418
- buffer_tostring(title),
1419
- "bytes",
1420
- "netdata",
1421
- "stats",
1422
- priority,
1423
- localhost->rrd_update_every,
1424
- RRDSET_TYPE_STACKED);
1425
-
1426
- ptrs->rd_pgc_memory_free = rrddim_add(ptrs->st_pgc_memory, "free", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1427
- ptrs->rd_pgc_memory_hot = rrddim_add(ptrs->st_pgc_memory, "hot", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1428
- ptrs->rd_pgc_memory_dirty = rrddim_add(ptrs->st_pgc_memory, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1429
- ptrs->rd_pgc_memory_clean = rrddim_add(ptrs->st_pgc_memory, "clean", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1430
- ptrs->rd_pgc_memory_index = rrddim_add(ptrs->st_pgc_memory, "index", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1431
- ptrs->rd_pgc_memory_evicting = rrddim_add(ptrs->st_pgc_memory, "evicting", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1432
- ptrs->rd_pgc_memory_flushing = rrddim_add(ptrs->st_pgc_memory, "flushing", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1433
-
1434
- buffer_free(id);
1435
- buffer_free(family);
1436
- buffer_free(title);
1437
- priority++;
1438
- }
1439
-
1440
- collected_number free = (pgc_stats->current_cache_size > pgc_stats->wanted_cache_size) ? 0 :
1441
- (collected_number)(pgc_stats->wanted_cache_size - pgc_stats->current_cache_size);
1442
-
1443
- rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_free, free);
1444
- rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_hot, (collected_number)pgc_stats->queues.hot.size);
1445
- rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_dirty, (collected_number)pgc_stats->queues.dirty.size);
1446
- rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_clean, (collected_number)pgc_stats->queues.clean.size);
1447
- rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_evicting, (collected_number)pgc_stats->evicting_size);
1448
- rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_flushing, (collected_number)pgc_stats->flushing_size);
1449
- rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_index,
1450
- (collected_number)(pgc_stats->size - pgc_stats->queues.clean.size - pgc_stats->queues.hot.size - pgc_stats->queues.dirty.size - pgc_stats->evicting_size - pgc_stats->flushing_size));
1451
-
1452
- rrdset_done(ptrs->st_pgc_memory);
1453
- }
1454
-
1455
- {
1456
- if (unlikely(!ptrs->st_pgc_tm)) {
1457
- BUFFER *id = buffer_create(100, NULL);
1458
- buffer_sprintf(id, "dbengine_%s_target_memory", name);
1459
-
1460
- BUFFER *family = buffer_create(100, NULL);
1461
- buffer_sprintf(family, "dbengine %s cache", name);
1462
-
1463
- BUFFER *title = buffer_create(100, NULL);
1464
- buffer_sprintf(title, "Netdata %s Target Cache Memory", name);
1465
-
1466
- ptrs->st_pgc_tm = rrdset_create_localhost(
1467
- "netdata",
1468
- buffer_tostring(id),
1469
- NULL,
1470
- buffer_tostring(family),
1471
- NULL,
1472
- buffer_tostring(title),
1473
- "bytes",
1474
- "netdata",
1475
- "stats",
1476
- priority,
1477
- localhost->rrd_update_every,
1478
- RRDSET_TYPE_LINE);
1479
-
1480
- ptrs->rd_pgc_tm_current = rrddim_add(ptrs->st_pgc_tm, "current", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1481
- ptrs->rd_pgc_tm_wanted = rrddim_add(ptrs->st_pgc_tm, "wanted", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1482
- ptrs->rd_pgc_tm_referenced = rrddim_add(ptrs->st_pgc_tm, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1483
- ptrs->rd_pgc_tm_hot_max = rrddim_add(ptrs->st_pgc_tm, "hot max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1484
- ptrs->rd_pgc_tm_dirty_max = rrddim_add(ptrs->st_pgc_tm, "dirty max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1485
- ptrs->rd_pgc_tm_hot = rrddim_add(ptrs->st_pgc_tm, "hot", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1486
- ptrs->rd_pgc_tm_dirty = rrddim_add(ptrs->st_pgc_tm, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1487
-
1488
- buffer_free(id);
1489
- buffer_free(family);
1490
- buffer_free(title);
1491
- priority++;
1492
- }
1493
-
1494
- rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_current, (collected_number)pgc_stats->current_cache_size);
1495
- rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_wanted, (collected_number)pgc_stats->wanted_cache_size);
1496
- rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_referenced, (collected_number)pgc_stats->referenced_size);
1497
- rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_hot_max, (collected_number)pgc_stats->queues.hot.max_size);
1498
- rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_dirty_max, (collected_number)pgc_stats->queues.dirty.max_size);
1499
- rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_hot, (collected_number)pgc_stats->queues.hot.size);
1500
- rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_dirty, (collected_number)pgc_stats->queues.dirty.size);
1501
-
1502
- rrdset_done(ptrs->st_pgc_tm);
1503
- }
1504
-
1505
- {
1506
- if (unlikely(!ptrs->st_pgc_pages)) {
1507
- BUFFER *id = buffer_create(100, NULL);
1508
- buffer_sprintf(id, "dbengine_%s_cache_pages", name);
1509
-
1510
- BUFFER *family = buffer_create(100, NULL);
1511
- buffer_sprintf(family, "dbengine %s cache", name);
1512
-
1513
- BUFFER *title = buffer_create(100, NULL);
1514
- buffer_sprintf(title, "Netdata %s Cache Pages", name);
1515
-
1516
- ptrs->st_pgc_pages = rrdset_create_localhost(
1517
- "netdata",
1518
- buffer_tostring(id),
1519
- NULL,
1520
- buffer_tostring(family),
1521
- NULL,
1522
- buffer_tostring(title),
1523
- "pages",
1524
- "netdata",
1525
- "stats",
1526
- priority,
1527
- localhost->rrd_update_every,
1528
- RRDSET_TYPE_LINE);
1529
-
1530
- ptrs->rd_pgc_pages_clean = rrddim_add(ptrs->st_pgc_pages, "clean", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1531
- ptrs->rd_pgc_pages_hot = rrddim_add(ptrs->st_pgc_pages, "hot", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1532
- ptrs->rd_pgc_pages_dirty = rrddim_add(ptrs->st_pgc_pages, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1533
- ptrs->rd_pgc_pages_referenced = rrddim_add(ptrs->st_pgc_pages, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1534
-
1535
- buffer_free(id);
1536
- buffer_free(family);
1537
- buffer_free(title);
1538
- priority++;
1539
- }
1540
-
1541
- rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_clean, (collected_number)pgc_stats->queues.clean.entries);
1542
- rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_hot, (collected_number)pgc_stats->queues.hot.entries);
1543
- rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_dirty, (collected_number)pgc_stats->queues.dirty.entries);
1544
- rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_referenced, (collected_number)pgc_stats->referenced_entries);
1545
-
1546
- rrdset_done(ptrs->st_pgc_pages);
1547
- }
1548
-
1549
- {
1550
- if (unlikely(!ptrs->st_pgc_memory_changes)) {
1551
- BUFFER *id = buffer_create(100, NULL);
1552
- buffer_sprintf(id, "dbengine_%s_cache_memory_changes", name);
1553
-
1554
- BUFFER *family = buffer_create(100, NULL);
1555
- buffer_sprintf(family, "dbengine %s cache", name);
1556
-
1557
- BUFFER *title = buffer_create(100, NULL);
1558
- buffer_sprintf(title, "Netdata %s Cache Memory Changes", name);
1559
-
1560
- ptrs->st_pgc_memory_changes = rrdset_create_localhost(
1561
- "netdata",
1562
- buffer_tostring(id),
1563
- NULL,
1564
- buffer_tostring(family),
1565
- NULL,
1566
- buffer_tostring(title),
1567
- "bytes/s",
1568
- "netdata",
1569
- "stats",
1570
- priority,
1571
- localhost->rrd_update_every,
1572
- RRDSET_TYPE_AREA);
1573
-
1574
- ptrs->rd_pgc_memory_new_clean = rrddim_add(ptrs->st_pgc_memory_changes, "new clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1575
- ptrs->rd_pgc_memory_clean_evictions = rrddim_add(ptrs->st_pgc_memory_changes, "evictions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1576
- ptrs->rd_pgc_memory_new_hot = rrddim_add(ptrs->st_pgc_memory_changes, "new hot", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1577
-
1578
- buffer_free(id);
1579
- buffer_free(family);
1580
- buffer_free(title);
1581
- priority++;
1582
- }
1583
-
1584
- rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_new_clean, (collected_number)(pgc_stats->added_size - pgc_stats->queues.hot.added_size));
1585
- rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_clean_evictions, (collected_number)pgc_stats->queues.clean.removed_size);
1586
- rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_new_hot, (collected_number)pgc_stats->queues.hot.added_size);
1587
-
1588
- rrdset_done(ptrs->st_pgc_memory_changes);
1589
- }
1590
-
1591
- {
1592
- if (unlikely(!ptrs->st_pgc_memory_migrations)) {
1593
- BUFFER *id = buffer_create(100, NULL);
1594
- buffer_sprintf(id, "dbengine_%s_cache_memory_migrations", name);
1595
-
1596
- BUFFER *family = buffer_create(100, NULL);
1597
- buffer_sprintf(family, "dbengine %s cache", name);
1598
-
1599
- BUFFER *title = buffer_create(100, NULL);
1600
- buffer_sprintf(title, "Netdata %s Cache Memory Migrations", name);
1601
-
1602
- ptrs->st_pgc_memory_migrations = rrdset_create_localhost(
1603
- "netdata",
1604
- buffer_tostring(id),
1605
- NULL,
1606
- buffer_tostring(family),
1607
- NULL,
1608
- buffer_tostring(title),
1609
- "bytes/s",
1610
- "netdata",
1611
- "stats",
1612
- priority,
1613
- localhost->rrd_update_every,
1614
- RRDSET_TYPE_AREA);
1615
-
1616
- ptrs->rd_pgc_memory_dirty_to_clean = rrddim_add(ptrs->st_pgc_memory_migrations, "dirty to clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1617
- ptrs->rd_pgc_memory_hot_to_dirty = rrddim_add(ptrs->st_pgc_memory_migrations, "hot to dirty", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1618
-
1619
- buffer_free(id);
1620
- buffer_free(family);
1621
- buffer_free(title);
1622
- priority++;
1623
- }
1624
-
1625
- rrddim_set_by_pointer(ptrs->st_pgc_memory_migrations, ptrs->rd_pgc_memory_dirty_to_clean, (collected_number)pgc_stats->queues.dirty.removed_size);
1626
- rrddim_set_by_pointer(ptrs->st_pgc_memory_migrations, ptrs->rd_pgc_memory_hot_to_dirty, (collected_number)pgc_stats->queues.dirty.added_size);
1627
-
1628
- rrdset_done(ptrs->st_pgc_memory_migrations);
1629
- }
1630
-
1631
- {
1632
- if (unlikely(!ptrs->st_pgc_memory_events)) {
1633
- BUFFER *id = buffer_create(100, NULL);
1634
- buffer_sprintf(id, "dbengine_%s_cache_events", name);
1635
-
1636
- BUFFER *family = buffer_create(100, NULL);
1637
- buffer_sprintf(family, "dbengine %s cache", name);
1638
-
1639
- BUFFER *title = buffer_create(100, NULL);
1640
- buffer_sprintf(title, "Netdata %s Cache Events", name);
1641
-
1642
- ptrs->st_pgc_memory_events = rrdset_create_localhost(
1643
- "netdata",
1644
- buffer_tostring(id),
1645
- NULL,
1646
- buffer_tostring(family),
1647
- NULL,
1648
- buffer_tostring(title),
1649
- "events/s",
1650
- "netdata",
1651
- "stats",
1652
- priority,
1653
- localhost->rrd_update_every,
1654
- RRDSET_TYPE_AREA);
1655
-
1656
- ptrs->rd_pgc_memory_evictions_aggressive = rrddim_add(ptrs->st_pgc_memory_events, "evictions aggressive", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1657
- ptrs->rd_pgc_memory_evictions_critical = rrddim_add(ptrs->st_pgc_memory_events, "evictions critical", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1658
- ptrs->rd_pgc_memory_flushes_critical = rrddim_add(ptrs->st_pgc_memory_events, "flushes critical", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1659
-
1660
- buffer_free(id);
1661
- buffer_free(family);
1662
- buffer_free(title);
1663
- priority++;
1664
- }
1665
-
1666
- rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_evictions_aggressive, (collected_number)pgc_stats->events_cache_needs_space_aggressively);
1667
- rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_evictions_critical, (collected_number)pgc_stats->events_cache_under_severe_pressure);
1668
- rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_flushes_critical, (collected_number)pgc_stats->events_flush_critical);
1669
-
1670
- rrdset_done(ptrs->st_pgc_memory_events);
1671
- }
1672
-
1673
- {
1674
- if (unlikely(!ptrs->st_pgc_waste)) {
1675
- BUFFER *id = buffer_create(100, NULL);
1676
- buffer_sprintf(id, "dbengine_%s_waste_events", name);
1677
-
1678
- BUFFER *family = buffer_create(100, NULL);
1679
- buffer_sprintf(family, "dbengine %s cache", name);
1680
-
1681
- BUFFER *title = buffer_create(100, NULL);
1682
- buffer_sprintf(title, "Netdata %s Waste Events", name);
1683
-
1684
- ptrs->st_pgc_waste = rrdset_create_localhost(
1685
- "netdata",
1686
- buffer_tostring(id),
1687
- NULL,
1688
- buffer_tostring(family),
1689
- NULL,
1690
- buffer_tostring(title),
1691
- "events/s",
1692
- "netdata",
1693
- "stats",
1694
- priority,
1695
- localhost->rrd_update_every,
1696
- RRDSET_TYPE_LINE);
1697
-
1698
- ptrs->rd_pgc_waste_evictions_skipped = rrddim_add(ptrs->st_pgc_waste, "evictions skipped", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1699
- ptrs->rd_pgc_waste_flushes_cancelled = rrddim_add(ptrs->st_pgc_waste, "flushes cancelled", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1700
- ptrs->rd_pgc_waste_acquire_spins = rrddim_add(ptrs->st_pgc_waste, "acquire spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1701
- ptrs->rd_pgc_waste_release_spins = rrddim_add(ptrs->st_pgc_waste, "release spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1702
- ptrs->rd_pgc_waste_insert_spins = rrddim_add(ptrs->st_pgc_waste, "insert spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1703
- ptrs->rd_pgc_waste_delete_spins = rrddim_add(ptrs->st_pgc_waste, "delete spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1704
- ptrs->rd_pgc_waste_evict_spins = rrddim_add(ptrs->st_pgc_waste, "evict spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1705
- ptrs->rd_pgc_waste_flush_spins = rrddim_add(ptrs->st_pgc_waste, "flush spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1706
-
1707
- buffer_free(id);
1708
- buffer_free(family);
1709
- buffer_free(title);
1710
- priority++;
1711
- }
1712
-
1713
- rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evictions_skipped, (collected_number)pgc_stats->evict_skipped);
1714
- rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_flushes_cancelled, (collected_number)pgc_stats->flushes_cancelled);
1715
- rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_acquire_spins, (collected_number)pgc_stats->acquire_spins);
1716
- rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_release_spins, (collected_number)pgc_stats->release_spins);
1717
- rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_insert_spins, (collected_number)pgc_stats->insert_spins);
1718
- rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_delete_spins, (collected_number)pgc_stats->delete_spins);
1719
- rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evict_spins, (collected_number)pgc_stats->evict_spins);
1720
- rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_flush_spins, (collected_number)pgc_stats->flush_spins);
1721
-
1722
- rrdset_done(ptrs->st_pgc_waste);
1723
- }
1724
-
1725
- {
1726
- if (unlikely(!ptrs->st_pgc_workers)) {
1727
- BUFFER *id = buffer_create(100, NULL);
1728
- buffer_sprintf(id, "dbengine_%s_cache_workers", name);
1729
-
1730
- BUFFER *family = buffer_create(100, NULL);
1731
- buffer_sprintf(family, "dbengine %s cache", name);
1732
-
1733
- BUFFER *title = buffer_create(100, NULL);
1734
- buffer_sprintf(title, "Netdata %s Cache Workers", name);
1735
-
1736
- ptrs->st_pgc_workers = rrdset_create_localhost(
1737
- "netdata",
1738
- buffer_tostring(id),
1739
- NULL,
1740
- buffer_tostring(family),
1741
- NULL,
1742
- buffer_tostring(title),
1743
- "workers",
1744
- "netdata",
1745
- "stats",
1746
- priority,
1747
- localhost->rrd_update_every,
1748
- RRDSET_TYPE_LINE);
1749
-
1750
- ptrs->rd_pgc_workers_searchers = rrddim_add(ptrs->st_pgc_workers, "searchers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1751
- ptrs->rd_pgc_workers_adders = rrddim_add(ptrs->st_pgc_workers, "adders", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1752
- ptrs->rd_pgc_workers_evictors = rrddim_add(ptrs->st_pgc_workers, "evictors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1753
- ptrs->rd_pgc_workers_flushers = rrddim_add(ptrs->st_pgc_workers, "flushers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1754
- ptrs->rd_pgc_workers_hot2dirty = rrddim_add(ptrs->st_pgc_workers, "hot2dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1755
- ptrs->rd_pgc_workers_jv2_flushers = rrddim_add(ptrs->st_pgc_workers, "jv2 flushers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1756
-
1757
- buffer_free(id);
1758
- buffer_free(family);
1759
- buffer_free(title);
1760
- priority++;
1761
- }
1762
-
1763
- rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_searchers, (collected_number)pgc_stats->workers_search);
1764
- rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_adders, (collected_number)pgc_stats->workers_add);
1765
- rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_evictors, (collected_number)pgc_stats->workers_evict);
1766
- rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_flushers, (collected_number)pgc_stats->workers_flush);
1767
- rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_hot2dirty, (collected_number)pgc_stats->workers_hot2dirty);
1768
- rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_jv2_flushers, (collected_number)pgc_stats->workers_jv2_flush);
1769
-
1770
- rrdset_done(ptrs->st_pgc_workers);
1771
- }
1772
-}
1773
-
1774
-
1775
-static void dbengine2_statistics_charts(void) {
1776
- if(!main_cache || !main_mrg)
1777
- return;
1778
-
1779
- static struct dbengine2_cache_pointers main_cache_ptrs = {}, open_cache_ptrs = {}, extent_cache_ptrs = {};
1780
- static struct rrdeng_cache_efficiency_stats cache_efficiency_stats = {}, cache_efficiency_stats_old = {};
1781
- static struct pgc_statistics pgc_main_stats = {}, pgc_main_stats_old = {}; (void)pgc_main_stats_old;
1782
- static struct pgc_statistics pgc_open_stats = {}, pgc_open_stats_old = {}; (void)pgc_open_stats_old;
1783
- static struct pgc_statistics pgc_extent_stats = {}, pgc_extent_stats_old = {}; (void)pgc_extent_stats_old;
1784
- static struct mrg_statistics mrg_stats = {}, mrg_stats_old = {}; (void)mrg_stats_old;
1785
-
1786
- pgc_main_stats_old = pgc_main_stats;
1787
- pgc_main_stats = pgc_get_statistics(main_cache);
1788
- dbengine2_cache_statistics_charts(&main_cache_ptrs, &pgc_main_stats, &pgc_main_stats_old, "main", 135100);
1789
-
1790
- pgc_open_stats_old = pgc_open_stats;
1791
- pgc_open_stats = pgc_get_statistics(open_cache);
1792
- dbengine2_cache_statistics_charts(&open_cache_ptrs, &pgc_open_stats, &pgc_open_stats_old, "open", 135200);
1793
-
1794
- pgc_extent_stats_old = pgc_extent_stats;
1795
- pgc_extent_stats = pgc_get_statistics(extent_cache);
1796
- dbengine2_cache_statistics_charts(&extent_cache_ptrs, &pgc_extent_stats, &pgc_extent_stats_old, "extent", 135300);
1797
-
1798
- cache_efficiency_stats_old = cache_efficiency_stats;
1799
- cache_efficiency_stats = rrdeng_get_cache_efficiency_stats();
1800
-
1801
- mrg_stats_old = mrg_stats;
1802
- mrg_get_statistics(main_mrg, &mrg_stats);
1803
-
1804
- struct rrdeng_buffer_sizes buffers = rrdeng_get_buffer_sizes();
1805
- size_t buffers_total_size = buffers.handles + buffers.xt_buf + buffers.xt_io + buffers.pdc + buffers.descriptors +
1806
- buffers.opcodes + buffers.wal + buffers.workers + buffers.epdl + buffers.deol + buffers.pd + buffers.pgc + buffers.mrg;
1807
-
1808
-#ifdef PDC_USE_JULYL
1809
- buffers_total_size += buffers.julyl;
1810
-#endif
1811
-
1812
- dbengine_total_memory = pgc_main_stats.size + pgc_open_stats.size + pgc_extent_stats.size + mrg_stats.size + buffers_total_size;
1813
-
1814
- size_t priority = 135000;
1815
-
1816
- {
1817
- static RRDSET *st_pgc_memory = NULL;
1818
- static RRDDIM *rd_pgc_memory_main = NULL;
1819
- static RRDDIM *rd_pgc_memory_open = NULL; // open journal memory
1820
- static RRDDIM *rd_pgc_memory_extent = NULL; // extent compresses cache memory
1821
- static RRDDIM *rd_pgc_memory_metrics = NULL; // metric registry memory
1822
- static RRDDIM *rd_pgc_memory_buffers = NULL;
1823
-
1824
- if (unlikely(!st_pgc_memory)) {
1825
- st_pgc_memory = rrdset_create_localhost(
1826
- "netdata",
1827
- "dbengine_memory",
1828
- NULL,
1829
- "dbengine memory",
1830
- NULL,
1831
- "Netdata DB Memory",
1832
- "bytes",
1833
- "netdata",
1834
- "stats",
1835
- priority,
1836
- localhost->rrd_update_every,
1837
- RRDSET_TYPE_STACKED);
1838
-
1839
- rd_pgc_memory_main = rrddim_add(st_pgc_memory, "main cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1840
- rd_pgc_memory_open = rrddim_add(st_pgc_memory, "open cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1841
- rd_pgc_memory_extent = rrddim_add(st_pgc_memory, "extent cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1842
- rd_pgc_memory_metrics = rrddim_add(st_pgc_memory, "metrics registry", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1843
- rd_pgc_memory_buffers = rrddim_add(st_pgc_memory, "buffers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1844
- }
1845
- priority++;
1846
-
1847
-
1848
- rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_main, (collected_number)pgc_main_stats.size);
1849
- rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_open, (collected_number)pgc_open_stats.size);
1850
- rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_extent, (collected_number)pgc_extent_stats.size);
1851
- rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_metrics, (collected_number)mrg_stats.size);
1852
- rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_buffers, (collected_number)buffers_total_size);
1853
-
1854
- rrdset_done(st_pgc_memory);
1855
- }
1856
-
1857
- {
1858
- static RRDSET *st_pgc_buffers = NULL;
1859
- static RRDDIM *rd_pgc_buffers_pgc = NULL;
1860
- static RRDDIM *rd_pgc_buffers_mrg = NULL;
1861
- static RRDDIM *rd_pgc_buffers_opcodes = NULL;
1862
- static RRDDIM *rd_pgc_buffers_handles = NULL;
1863
- static RRDDIM *rd_pgc_buffers_descriptors = NULL;
1864
- static RRDDIM *rd_pgc_buffers_wal = NULL;
1865
- static RRDDIM *rd_pgc_buffers_workers = NULL;
1866
- static RRDDIM *rd_pgc_buffers_pdc = NULL;
1867
- static RRDDIM *rd_pgc_buffers_xt_io = NULL;
1868
- static RRDDIM *rd_pgc_buffers_xt_buf = NULL;
1869
- static RRDDIM *rd_pgc_buffers_epdl = NULL;
1870
- static RRDDIM *rd_pgc_buffers_deol = NULL;
1871
- static RRDDIM *rd_pgc_buffers_pd = NULL;
1872
-#ifdef PDC_USE_JULYL
1873
- static RRDDIM *rd_pgc_buffers_julyl = NULL;
1874
-#endif
1875
-
1876
- if (unlikely(!st_pgc_buffers)) {
1877
- st_pgc_buffers = rrdset_create_localhost(
1878
- "netdata",
1879
- "dbengine_buffers",
1880
- NULL,
1881
- "dbengine memory",
1882
- NULL,
1883
- "Netdata DB Buffers",
1884
- "bytes",
1885
- "netdata",
1886
- "stats",
1887
- priority,
1888
- localhost->rrd_update_every,
1889
- RRDSET_TYPE_STACKED);
1890
-
1891
- rd_pgc_buffers_pgc = rrddim_add(st_pgc_buffers, "pgc", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1892
- rd_pgc_buffers_mrg = rrddim_add(st_pgc_buffers, "mrg", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1893
- rd_pgc_buffers_opcodes = rrddim_add(st_pgc_buffers, "opcodes", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1894
- rd_pgc_buffers_handles = rrddim_add(st_pgc_buffers, "query handles", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1895
- rd_pgc_buffers_descriptors = rrddim_add(st_pgc_buffers, "descriptors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1896
- rd_pgc_buffers_wal = rrddim_add(st_pgc_buffers, "wal", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1897
- rd_pgc_buffers_workers = rrddim_add(st_pgc_buffers, "workers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1898
- rd_pgc_buffers_pdc = rrddim_add(st_pgc_buffers, "pdc", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1899
- rd_pgc_buffers_pd = rrddim_add(st_pgc_buffers, "pd", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1900
- rd_pgc_buffers_xt_io = rrddim_add(st_pgc_buffers, "extent io", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1901
- rd_pgc_buffers_xt_buf = rrddim_add(st_pgc_buffers, "extent buffers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1902
- rd_pgc_buffers_epdl = rrddim_add(st_pgc_buffers, "epdl", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1903
- rd_pgc_buffers_deol = rrddim_add(st_pgc_buffers, "deol", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1904
-#ifdef PDC_USE_JULYL
1905
- rd_pgc_buffers_julyl = rrddim_add(st_pgc_buffers, "julyl", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1906
-#endif
1907
- }
1908
- priority++;
1909
-
1910
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pgc, (collected_number)buffers.pgc);
1911
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_mrg, (collected_number)buffers.mrg);
1912
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_opcodes, (collected_number)buffers.opcodes);
1913
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_handles, (collected_number)buffers.handles);
1914
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_descriptors, (collected_number)buffers.descriptors);
1915
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_wal, (collected_number)buffers.wal);
1916
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_workers, (collected_number)buffers.workers);
1917
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pdc, (collected_number)buffers.pdc);
1918
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pd, (collected_number)buffers.pd);
1919
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_xt_io, (collected_number)buffers.xt_io);
1920
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_xt_buf, (collected_number)buffers.xt_buf);
1921
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_epdl, (collected_number)buffers.epdl);
1922
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_deol, (collected_number)buffers.deol);
1923
-#ifdef PDC_USE_JULYL
1924
- rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_julyl, (collected_number)buffers.julyl);
1925
-#endif
1926
-
1927
- rrdset_done(st_pgc_buffers);
1928
- }
1929
-
1930
-#ifdef PDC_USE_JULYL
1931
- {
1932
- static RRDSET *st_julyl_moved = NULL;
1933
- static RRDDIM *rd_julyl_moved = NULL;
1934
-
1935
- if (unlikely(!st_julyl_moved)) {
1936
- st_julyl_moved = rrdset_create_localhost(
1937
- "netdata",
1938
- "dbengine_julyl_moved",
1939
- NULL,
1940
- "dbengine memory",
1941
- NULL,
1942
- "Netdata JulyL Memory Moved",
1943
- "bytes/s",
1944
- "netdata",
1945
- "stats",
1946
- priority,
1947
- localhost->rrd_update_every,
1948
- RRDSET_TYPE_AREA);
1949
-
1950
- rd_julyl_moved = rrddim_add(st_julyl_moved, "moved", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1951
- }
1952
- priority++;
1953
-
1954
- rrddim_set_by_pointer(st_julyl_moved, rd_julyl_moved, (collected_number)julyl_bytes_moved());
1955
-
1956
- rrdset_done(st_julyl_moved);
1957
- }
1958
-#endif
1959
-
1960
- {
1961
- static RRDSET *st_mrg_metrics = NULL;
1962
- static RRDDIM *rd_mrg_metrics = NULL;
1963
- static RRDDIM *rd_mrg_acquired = NULL;
1964
- static RRDDIM *rd_mrg_collected = NULL;
1965
- static RRDDIM *rd_mrg_multiple_writers = NULL;
1966
-
1967
- if (unlikely(!st_mrg_metrics)) {
1968
- st_mrg_metrics = rrdset_create_localhost(
1969
- "netdata",
1970
- "dbengine_metrics",
1971
- NULL,
1972
- "dbengine metrics",
1973
- NULL,
1974
- "Netdata Metrics in Metrics Registry",
1975
- "metrics",
1976
- "netdata",
1977
- "stats",
1978
- priority,
1979
- localhost->rrd_update_every,
1980
- RRDSET_TYPE_LINE);
1981
-
1982
- rd_mrg_metrics = rrddim_add(st_mrg_metrics, "all", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1983
- rd_mrg_acquired = rrddim_add(st_mrg_metrics, "acquired", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1984
- rd_mrg_collected = rrddim_add(st_mrg_metrics, "collected", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1985
- rd_mrg_multiple_writers = rrddim_add(st_mrg_metrics, "multi-collected", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1986
- }
1987
- priority++;
1988
-
1989
- rrddim_set_by_pointer(st_mrg_metrics, rd_mrg_metrics, (collected_number)mrg_stats.entries);
1990
- rrddim_set_by_pointer(st_mrg_metrics, rd_mrg_acquired, (collected_number)mrg_stats.entries_referenced);
1991
- rrddim_set_by_pointer(st_mrg_metrics, rd_mrg_collected, (collected_number)mrg_stats.writers);
1992
- rrddim_set_by_pointer(st_mrg_metrics, rd_mrg_multiple_writers, (collected_number)mrg_stats.writers_conflicts);
1993
-
1994
- rrdset_done(st_mrg_metrics);
1995
- }
1996
-
1997
- {
1998
- static RRDSET *st_mrg_ops = NULL;
1999
- static RRDDIM *rd_mrg_add = NULL;
2000
- static RRDDIM *rd_mrg_del = NULL;
2001
- static RRDDIM *rd_mrg_search = NULL;
2002
-
2003
- if (unlikely(!st_mrg_ops)) {
2004
- st_mrg_ops = rrdset_create_localhost(
2005
- "netdata",
2006
- "dbengine_metrics_registry_operations",
2007
- NULL,
2008
- "dbengine metrics",
2009
- NULL,
2010
- "Netdata Metrics Registry Operations",
2011
- "metrics",
2012
- "netdata",
2013
- "stats",
2014
- priority,
2015
- localhost->rrd_update_every,
2016
- RRDSET_TYPE_LINE);
2017
-
2018
- rd_mrg_add = rrddim_add(st_mrg_ops, "add", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2019
- rd_mrg_del = rrddim_add(st_mrg_ops, "delete", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2020
- rd_mrg_search = rrddim_add(st_mrg_ops, "search", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2021
- }
2022
- priority++;
2023
-
2024
- rrddim_set_by_pointer(st_mrg_ops, rd_mrg_add, (collected_number)mrg_stats.additions);
2025
- rrddim_set_by_pointer(st_mrg_ops, rd_mrg_del, (collected_number)mrg_stats.deletions);
2026
- rrddim_set_by_pointer(st_mrg_ops, rd_mrg_search, (collected_number)mrg_stats.search_hits + (collected_number)mrg_stats.search_misses);
2027
-
2028
- rrdset_done(st_mrg_ops);
2029
- }
2030
-
2031
- {
2032
- static RRDSET *st_mrg_references = NULL;
2033
- static RRDDIM *rd_mrg_references = NULL;
2034
-
2035
- if (unlikely(!st_mrg_references)) {
2036
- st_mrg_references = rrdset_create_localhost(
2037
- "netdata",
2038
- "dbengine_metrics_registry_references",
2039
- NULL,
2040
- "dbengine metrics",
2041
- NULL,
2042
- "Netdata Metrics Registry References",
2043
- "references",
2044
- "netdata",
2045
- "stats",
2046
- priority,
2047
- localhost->rrd_update_every,
2048
- RRDSET_TYPE_LINE);
2049
-
2050
- rd_mrg_references = rrddim_add(st_mrg_references, "references", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2051
- }
2052
- priority++;
2053
-
2054
- rrddim_set_by_pointer(st_mrg_references, rd_mrg_references, (collected_number)mrg_stats.current_references);
2055
-
2056
- rrdset_done(st_mrg_references);
2057
- }
2058
-
2059
- {
2060
- static RRDSET *st_cache_hit_ratio = NULL;
2061
- static RRDDIM *rd_hit_ratio = NULL;
2062
- static RRDDIM *rd_main_cache_hit_ratio = NULL;
2063
- static RRDDIM *rd_extent_cache_hit_ratio = NULL;
2064
- static RRDDIM *rd_extent_merge_hit_ratio = NULL;
2065
-
2066
- if (unlikely(!st_cache_hit_ratio)) {
2067
- st_cache_hit_ratio = rrdset_create_localhost(
2068
- "netdata",
2069
- "dbengine_cache_hit_ratio",
2070
- NULL,
2071
- "dbengine query router",
2072
- NULL,
2073
- "Netdata Queries Cache Hit Ratio",
2074
- "%",
2075
- "netdata",
2076
- "stats",
2077
- priority,
2078
- localhost->rrd_update_every,
2079
- RRDSET_TYPE_LINE);
2080
-
2081
- rd_hit_ratio = rrddim_add(st_cache_hit_ratio, "overall", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
2082
- rd_main_cache_hit_ratio = rrddim_add(st_cache_hit_ratio, "main cache", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
2083
- rd_extent_cache_hit_ratio = rrddim_add(st_cache_hit_ratio, "extent cache", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
2084
- rd_extent_merge_hit_ratio = rrddim_add(st_cache_hit_ratio, "extent merge", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
2085
- }
2086
- priority++;
2087
-
2088
- size_t delta_pages_total = cache_efficiency_stats.pages_total - cache_efficiency_stats_old.pages_total;
2089
- size_t delta_pages_to_load_from_disk = cache_efficiency_stats.pages_to_load_from_disk - cache_efficiency_stats_old.pages_to_load_from_disk;
2090
- size_t delta_extents_loaded_from_disk = cache_efficiency_stats.extents_loaded_from_disk - cache_efficiency_stats_old.extents_loaded_from_disk;
2091
-
2092
- size_t delta_pages_data_source_main_cache = cache_efficiency_stats.pages_data_source_main_cache - cache_efficiency_stats_old.pages_data_source_main_cache;
2093
- size_t delta_pages_pending_found_in_cache_at_pass4 = cache_efficiency_stats.pages_data_source_main_cache_at_pass4 - cache_efficiency_stats_old.pages_data_source_main_cache_at_pass4;
2094
-
2095
- size_t delta_pages_data_source_extent_cache = cache_efficiency_stats.pages_data_source_extent_cache - cache_efficiency_stats_old.pages_data_source_extent_cache;
2096
- size_t delta_pages_load_extent_merged = cache_efficiency_stats.pages_load_extent_merged - cache_efficiency_stats_old.pages_load_extent_merged;
2097
-
2098
- size_t pages_total_hit = delta_pages_total - delta_extents_loaded_from_disk;
2099
-
2100
- static size_t overall_hit_ratio = 100;
2101
- size_t main_cache_hit_ratio = 0, extent_cache_hit_ratio = 0, extent_merge_hit_ratio = 0;
2102
- if(delta_pages_total) {
2103
- if(pages_total_hit > delta_pages_total)
2104
- pages_total_hit = delta_pages_total;
2105
-
2106
- overall_hit_ratio = pages_total_hit * 100 * 10000 / delta_pages_total;
2107
-
2108
- size_t delta_pages_main_cache = delta_pages_data_source_main_cache + delta_pages_pending_found_in_cache_at_pass4;
2109
- if(delta_pages_main_cache > delta_pages_total)
2110
- delta_pages_main_cache = delta_pages_total;
2111
-
2112
- main_cache_hit_ratio = delta_pages_main_cache * 100 * 10000 / delta_pages_total;
2113
- }
2114
-
2115
- if(delta_pages_to_load_from_disk) {
2116
- if(delta_pages_data_source_extent_cache > delta_pages_to_load_from_disk)
2117
- delta_pages_data_source_extent_cache = delta_pages_to_load_from_disk;
2118
-
2119
- extent_cache_hit_ratio = delta_pages_data_source_extent_cache * 100 * 10000 / delta_pages_to_load_from_disk;
2120
-
2121
- if(delta_pages_load_extent_merged > delta_pages_to_load_from_disk)
2122
- delta_pages_load_extent_merged = delta_pages_to_load_from_disk;
2123
-
2124
- extent_merge_hit_ratio = delta_pages_load_extent_merged * 100 * 10000 / delta_pages_to_load_from_disk;
2125
- }
2126
-
2127
- rrddim_set_by_pointer(st_cache_hit_ratio, rd_hit_ratio, (collected_number)overall_hit_ratio);
2128
- rrddim_set_by_pointer(st_cache_hit_ratio, rd_main_cache_hit_ratio, (collected_number)main_cache_hit_ratio);
2129
- rrddim_set_by_pointer(st_cache_hit_ratio, rd_extent_cache_hit_ratio, (collected_number)extent_cache_hit_ratio);
2130
- rrddim_set_by_pointer(st_cache_hit_ratio, rd_extent_merge_hit_ratio, (collected_number)extent_merge_hit_ratio);
2131
-
2132
- rrdset_done(st_cache_hit_ratio);
2133
- }
2134
-
2135
- {
2136
- static RRDSET *st_queries = NULL;
2137
- static RRDDIM *rd_total = NULL;
2138
- static RRDDIM *rd_open = NULL;
2139
- static RRDDIM *rd_jv2 = NULL;
2140
- static RRDDIM *rd_planned_with_gaps = NULL;
2141
- static RRDDIM *rd_executed_with_gaps = NULL;
2142
-
2143
- if (unlikely(!st_queries)) {
2144
- st_queries = rrdset_create_localhost(
2145
- "netdata",
2146
- "dbengine_queries",
2147
- NULL,
2148
- "dbengine query router",
2149
- NULL,
2150
- "Netdata Queries",
2151
- "queries/s",
2152
- "netdata",
2153
- "stats",
2154
- priority,
2155
- localhost->rrd_update_every,
2156
- RRDSET_TYPE_LINE);
2157
-
2158
- rd_total = rrddim_add(st_queries, "total", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2159
- rd_open = rrddim_add(st_queries, "open cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2160
- rd_jv2 = rrddim_add(st_queries, "journal v2", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2161
- rd_planned_with_gaps = rrddim_add(st_queries, "planned with gaps", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2162
- rd_executed_with_gaps = rrddim_add(st_queries, "executed with gaps", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2163
- }
2164
- priority++;
2165
-
2166
- rrddim_set_by_pointer(st_queries, rd_total, (collected_number)cache_efficiency_stats.queries);
2167
- rrddim_set_by_pointer(st_queries, rd_open, (collected_number)cache_efficiency_stats.queries_open);
2168
- rrddim_set_by_pointer(st_queries, rd_jv2, (collected_number)cache_efficiency_stats.queries_journal_v2);
2169
- rrddim_set_by_pointer(st_queries, rd_planned_with_gaps, (collected_number)cache_efficiency_stats.queries_planned_with_gaps);
2170
- rrddim_set_by_pointer(st_queries, rd_executed_with_gaps, (collected_number)cache_efficiency_stats.queries_executed_with_gaps);
2171
-
2172
- rrdset_done(st_queries);
2173
- }
2174
-
2175
- {
2176
- static RRDSET *st_queries_running = NULL;
2177
- static RRDDIM *rd_queries = NULL;
2178
-
2179
- if (unlikely(!st_queries_running)) {
2180
- st_queries_running = rrdset_create_localhost(
2181
- "netdata",
2182
- "dbengine_queries_running",
2183
- NULL,
2184
- "dbengine query router",
2185
- NULL,
2186
- "Netdata Queries Running",
2187
- "queries",
2188
- "netdata",
2189
- "stats",
2190
- priority,
2191
- localhost->rrd_update_every,
2192
- RRDSET_TYPE_LINE);
2193
-
2194
- rd_queries = rrddim_add(st_queries_running, "queries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2195
- }
2196
- priority++;
2197
-
2198
- rrddim_set_by_pointer(st_queries_running, rd_queries, (collected_number)cache_efficiency_stats.currently_running_queries);
2199
-
2200
- rrdset_done(st_queries_running);
2201
- }
2202
-
2203
- {
2204
- static RRDSET *st_query_pages_metadata_source = NULL;
2205
- static RRDDIM *rd_cache = NULL;
2206
- static RRDDIM *rd_open = NULL;
2207
- static RRDDIM *rd_jv2 = NULL;
2208
-
2209
- if (unlikely(!st_query_pages_metadata_source)) {
2210
- st_query_pages_metadata_source = rrdset_create_localhost(
2211
- "netdata",
2212
- "dbengine_query_pages_metadata_source",
2213
- NULL,
2214
- "dbengine query router",
2215
- NULL,
2216
- "Netdata Query Pages Metadata Source",
2217
- "pages/s",
2218
- "netdata",
2219
- "stats",
2220
- priority,
2221
- localhost->rrd_update_every,
2222
- RRDSET_TYPE_STACKED);
2223
-
2224
- rd_cache = rrddim_add(st_query_pages_metadata_source, "cache hit", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2225
- rd_jv2 = rrddim_add(st_query_pages_metadata_source, "journal v2 scan", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2226
- rd_open = rrddim_add(st_query_pages_metadata_source, "open journal", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2227
- }
2228
- priority++;
2229
-
2230
- rrddim_set_by_pointer(st_query_pages_metadata_source, rd_cache, (collected_number)cache_efficiency_stats.pages_meta_source_main_cache);
2231
- rrddim_set_by_pointer(st_query_pages_metadata_source, rd_jv2, (collected_number)cache_efficiency_stats.pages_meta_source_journal_v2);
2232
- rrddim_set_by_pointer(st_query_pages_metadata_source, rd_open, (collected_number)cache_efficiency_stats.pages_meta_source_open_cache);
2233
-
2234
- rrdset_done(st_query_pages_metadata_source);
2235
- }
2236
-
2237
- {
2238
- static RRDSET *st_query_pages_data_source = NULL;
2239
- static RRDDIM *rd_pages_main_cache = NULL;
2240
- static RRDDIM *rd_pages_disk = NULL;
2241
- static RRDDIM *rd_pages_extent_cache = NULL;
2242
-
2243
- if (unlikely(!st_query_pages_data_source)) {
2244
- st_query_pages_data_source = rrdset_create_localhost(
2245
- "netdata",
2246
- "dbengine_query_pages_data_source",
2247
- NULL,
2248
- "dbengine query router",
2249
- NULL,
2250
- "Netdata Query Pages to Data Source",
2251
- "pages/s",
2252
- "netdata",
2253
- "stats",
2254
- priority,
2255
- localhost->rrd_update_every,
2256
- RRDSET_TYPE_STACKED);
2257
-
2258
- rd_pages_main_cache = rrddim_add(st_query_pages_data_source, "main cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2259
- rd_pages_disk = rrddim_add(st_query_pages_data_source, "disk", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2260
- rd_pages_extent_cache = rrddim_add(st_query_pages_data_source, "extent cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2261
- }
2262
- priority++;
2263
-
2264
- rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_main_cache, (collected_number)cache_efficiency_stats.pages_data_source_main_cache + (collected_number)cache_efficiency_stats.pages_data_source_main_cache_at_pass4);
2265
- rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_disk, (collected_number)cache_efficiency_stats.pages_to_load_from_disk);
2266
- rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_extent_cache, (collected_number)cache_efficiency_stats.pages_data_source_extent_cache);
2267
-
2268
- rrdset_done(st_query_pages_data_source);
2269
- }
2270
-
2271
- {
2272
- static RRDSET *st_query_next_page = NULL;
2273
- static RRDDIM *rd_pass4 = NULL;
2274
- static RRDDIM *rd_nowait_failed = NULL;
2275
- static RRDDIM *rd_wait_failed = NULL;
2276
- static RRDDIM *rd_wait_loaded = NULL;
2277
- static RRDDIM *rd_nowait_loaded = NULL;
2278
-
2279
- if (unlikely(!st_query_next_page)) {
2280
- st_query_next_page = rrdset_create_localhost(
2281
- "netdata",
2282
- "dbengine_query_next_page",
2283
- NULL,
2284
- "dbengine query router",
2285
- NULL,
2286
- "Netdata Query Next Page",
2287
- "pages/s",
2288
- "netdata",
2289
- "stats",
2290
- priority,
2291
- localhost->rrd_update_every,
2292
- RRDSET_TYPE_STACKED);
2293
-
2294
- rd_pass4 = rrddim_add(st_query_next_page, "pass4", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2295
- rd_wait_failed = rrddim_add(st_query_next_page, "failed slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2296
- rd_nowait_failed = rrddim_add(st_query_next_page, "failed fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2297
- rd_wait_loaded = rrddim_add(st_query_next_page, "loaded slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2298
- rd_nowait_loaded = rrddim_add(st_query_next_page, "loaded fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2299
- }
2300
- priority++;
2301
-
2302
- rrddim_set_by_pointer(st_query_next_page, rd_pass4, (collected_number)cache_efficiency_stats.pages_data_source_main_cache_at_pass4);
2303
- rrddim_set_by_pointer(st_query_next_page, rd_wait_failed, (collected_number)cache_efficiency_stats.page_next_wait_failed);
2304
- rrddim_set_by_pointer(st_query_next_page, rd_nowait_failed, (collected_number)cache_efficiency_stats.page_next_nowait_failed);
2305
- rrddim_set_by_pointer(st_query_next_page, rd_wait_loaded, (collected_number)cache_efficiency_stats.page_next_wait_loaded);
2306
- rrddim_set_by_pointer(st_query_next_page, rd_nowait_loaded, (collected_number)cache_efficiency_stats.page_next_nowait_loaded);
2307
-
2308
- rrdset_done(st_query_next_page);
2309
- }
2310
-
2311
- {
2312
- static RRDSET *st_query_page_issues = NULL;
2313
- static RRDDIM *rd_pages_zero_time = NULL;
2314
- static RRDDIM *rd_pages_past_time = NULL;
2315
- static RRDDIM *rd_pages_invalid_size = NULL;
2316
- static RRDDIM *rd_pages_fixed_update_every = NULL;
2317
- static RRDDIM *rd_pages_fixed_entries = NULL;
2318
- static RRDDIM *rd_pages_overlapping = NULL;
2319
-
2320
- if (unlikely(!st_query_page_issues)) {
2321
- st_query_page_issues = rrdset_create_localhost(
2322
- "netdata",
2323
- "dbengine_query_next_page_issues",
2324
- NULL,
2325
- "dbengine query router",
2326
- NULL,
2327
- "Netdata Query Next Page Issues",
2328
- "pages/s",
2329
- "netdata",
2330
- "stats",
2331
- priority,
2332
- localhost->rrd_update_every,
2333
- RRDSET_TYPE_STACKED);
2334
-
2335
- rd_pages_zero_time = rrddim_add(st_query_page_issues, "zero timestamp", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2336
- rd_pages_invalid_size = rrddim_add(st_query_page_issues, "invalid size", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2337
- rd_pages_past_time = rrddim_add(st_query_page_issues, "past time", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2338
- rd_pages_overlapping = rrddim_add(st_query_page_issues, "overlapping", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2339
- rd_pages_fixed_update_every = rrddim_add(st_query_page_issues, "update every fixed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2340
- rd_pages_fixed_entries = rrddim_add(st_query_page_issues, "entries fixed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2341
- }
2342
- priority++;
2343
-
2344
- rrddim_set_by_pointer(st_query_page_issues, rd_pages_zero_time, (collected_number)cache_efficiency_stats.pages_zero_time_skipped);
2345
- rrddim_set_by_pointer(st_query_page_issues, rd_pages_invalid_size, (collected_number)cache_efficiency_stats.pages_invalid_size_skipped);
2346
- rrddim_set_by_pointer(st_query_page_issues, rd_pages_past_time, (collected_number)cache_efficiency_stats.pages_past_time_skipped);
2347
- rrddim_set_by_pointer(st_query_page_issues, rd_pages_overlapping, (collected_number)cache_efficiency_stats.pages_overlapping_skipped);
2348
- rrddim_set_by_pointer(st_query_page_issues, rd_pages_fixed_update_every, (collected_number)cache_efficiency_stats.pages_invalid_update_every_fixed);
2349
- rrddim_set_by_pointer(st_query_page_issues, rd_pages_fixed_entries, (collected_number)cache_efficiency_stats.pages_invalid_entries_fixed);
2350
-
2351
- rrdset_done(st_query_page_issues);
2352
- }
2353
-
2354
- {
2355
- static RRDSET *st_query_pages_from_disk = NULL;
2356
- static RRDDIM *rd_compressed = NULL;
2357
- static RRDDIM *rd_invalid = NULL;
2358
- static RRDDIM *rd_uncompressed = NULL;
2359
- static RRDDIM *rd_mmap_failed = NULL;
2360
- static RRDDIM *rd_unavailable = NULL;
2361
- static RRDDIM *rd_unroutable = NULL;
2362
- static RRDDIM *rd_not_found = NULL;
2363
- static RRDDIM *rd_cancelled = NULL;
2364
- static RRDDIM *rd_invalid_extent = NULL;
2365
- static RRDDIM *rd_extent_merged = NULL;
2366
-
2367
- if (unlikely(!st_query_pages_from_disk)) {
2368
- st_query_pages_from_disk = rrdset_create_localhost(
2369
- "netdata",
2370
- "dbengine_query_pages_disk_load",
2371
- NULL,
2372
- "dbengine query router",
2373
- NULL,
2374
- "Netdata Query Pages Loaded from Disk",
2375
- "pages/s",
2376
- "netdata",
2377
- "stats",
2378
- priority,
2379
- localhost->rrd_update_every,
2380
- RRDSET_TYPE_LINE);
2381
-
2382
- rd_compressed = rrddim_add(st_query_pages_from_disk, "ok compressed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2383
- rd_invalid = rrddim_add(st_query_pages_from_disk, "fail invalid page", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2384
- rd_uncompressed = rrddim_add(st_query_pages_from_disk, "ok uncompressed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2385
- rd_mmap_failed = rrddim_add(st_query_pages_from_disk, "fail cant mmap", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2386
- rd_unavailable = rrddim_add(st_query_pages_from_disk, "fail unavailable", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2387
- rd_unroutable = rrddim_add(st_query_pages_from_disk, "fail unroutable", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2388
- rd_not_found = rrddim_add(st_query_pages_from_disk, "fail not found", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2389
- rd_invalid_extent = rrddim_add(st_query_pages_from_disk, "fail invalid extent", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2390
- rd_extent_merged = rrddim_add(st_query_pages_from_disk, "extent merged", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2391
- rd_cancelled = rrddim_add(st_query_pages_from_disk, "cancelled", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2392
- }
2393
- priority++;
2394
-
2395
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_compressed, (collected_number)cache_efficiency_stats.pages_load_ok_compressed);
2396
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_invalid, (collected_number)cache_efficiency_stats.pages_load_fail_invalid_page_in_extent);
2397
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_uncompressed, (collected_number)cache_efficiency_stats.pages_load_ok_uncompressed);
2398
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_mmap_failed, (collected_number)cache_efficiency_stats.pages_load_fail_cant_mmap_extent);
2399
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_unavailable, (collected_number)cache_efficiency_stats.pages_load_fail_datafile_not_available);
2400
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_unroutable, (collected_number)cache_efficiency_stats.pages_load_fail_unroutable);
2401
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_not_found, (collected_number)cache_efficiency_stats.pages_load_fail_not_found);
2402
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_cancelled, (collected_number)cache_efficiency_stats.pages_load_fail_cancelled);
2403
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_invalid_extent, (collected_number)cache_efficiency_stats.pages_load_fail_invalid_extent);
2404
- rrddim_set_by_pointer(st_query_pages_from_disk, rd_extent_merged, (collected_number)cache_efficiency_stats.pages_load_extent_merged);
2405
-
2406
- rrdset_done(st_query_pages_from_disk);
2407
- }
2408
-
2409
- {
2410
- static RRDSET *st_events = NULL;
2411
- static RRDDIM *rd_journal_v2_mapped = NULL;
2412
- static RRDDIM *rd_journal_v2_unmapped = NULL;
2413
- static RRDDIM *rd_datafile_creation = NULL;
2414
- static RRDDIM *rd_datafile_deletion = NULL;
2415
- static RRDDIM *rd_datafile_deletion_spin = NULL;
2416
- static RRDDIM *rd_jv2_indexing = NULL;
2417
- static RRDDIM *rd_retention = NULL;
2418
-
2419
- if (unlikely(!st_events)) {
2420
- st_events = rrdset_create_localhost(
2421
- "netdata",
2422
- "dbengine_events",
2423
- NULL,
2424
- "dbengine query router",
2425
- NULL,
2426
- "Netdata Database Events",
2427
- "events/s",
2428
- "netdata",
2429
- "stats",
2430
- priority,
2431
- localhost->rrd_update_every,
2432
- RRDSET_TYPE_LINE);
2433
-
2434
- rd_journal_v2_mapped = rrddim_add(st_events, "journal v2 mapped", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2435
- rd_journal_v2_unmapped = rrddim_add(st_events, "journal v2 unmapped", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2436
- rd_datafile_creation = rrddim_add(st_events, "datafile creation", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2437
- rd_datafile_deletion = rrddim_add(st_events, "datafile deletion", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2438
- rd_datafile_deletion_spin = rrddim_add(st_events, "datafile deletion spin", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2439
- rd_jv2_indexing = rrddim_add(st_events, "journal v2 indexing", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2440
- rd_retention = rrddim_add(st_events, "retention", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2441
- }
2442
- priority++;
2443
-
2444
- rrddim_set_by_pointer(st_events, rd_journal_v2_mapped, (collected_number)cache_efficiency_stats.journal_v2_mapped);
2445
- rrddim_set_by_pointer(st_events, rd_journal_v2_unmapped, (collected_number)cache_efficiency_stats.journal_v2_unmapped);
2446
- rrddim_set_by_pointer(st_events, rd_datafile_creation, (collected_number)cache_efficiency_stats.datafile_creation_started);
2447
- rrddim_set_by_pointer(st_events, rd_datafile_deletion, (collected_number)cache_efficiency_stats.datafile_deletion_started);
2448
- rrddim_set_by_pointer(st_events, rd_datafile_deletion_spin, (collected_number)cache_efficiency_stats.datafile_deletion_spin);
2449
- rrddim_set_by_pointer(st_events, rd_jv2_indexing, (collected_number)cache_efficiency_stats.journal_v2_indexing_started);
2450
- rrddim_set_by_pointer(st_events, rd_retention, (collected_number)cache_efficiency_stats.metrics_retention_started);
2451
-
2452
- rrdset_done(st_events);
2453
- }
2454
-
2455
- {
2456
- static RRDSET *st_prep_timings = NULL;
2457
- static RRDDIM *rd_routing = NULL;
2458
- static RRDDIM *rd_main_cache = NULL;
2459
- static RRDDIM *rd_open_cache = NULL;
2460
- static RRDDIM *rd_journal_v2 = NULL;
2461
- static RRDDIM *rd_pass4 = NULL;
2462
-
2463
- if (unlikely(!st_prep_timings)) {
2464
- st_prep_timings = rrdset_create_localhost(
2465
- "netdata",
2466
- "dbengine_prep_timings",
2467
- NULL,
2468
- "dbengine query router",
2469
- NULL,
2470
- "Netdata Query Preparation Timings",
2471
- "usec/s",
2472
- "netdata",
2473
- "stats",
2474
- priority,
2475
- localhost->rrd_update_every,
2476
- RRDSET_TYPE_STACKED);
2477
-
2478
- rd_routing = rrddim_add(st_prep_timings, "routing", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2479
- rd_main_cache = rrddim_add(st_prep_timings, "main cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2480
- rd_open_cache = rrddim_add(st_prep_timings, "open cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2481
- rd_journal_v2 = rrddim_add(st_prep_timings, "journal v2", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2482
- rd_pass4 = rrddim_add(st_prep_timings, "pass4", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2483
- }
2484
- priority++;
2485
-
2486
- rrddim_set_by_pointer(st_prep_timings, rd_routing, (collected_number)cache_efficiency_stats.prep_time_to_route);
2487
- rrddim_set_by_pointer(st_prep_timings, rd_main_cache, (collected_number)cache_efficiency_stats.prep_time_in_main_cache_lookup);
2488
- rrddim_set_by_pointer(st_prep_timings, rd_open_cache, (collected_number)cache_efficiency_stats.prep_time_in_open_cache_lookup);
2489
- rrddim_set_by_pointer(st_prep_timings, rd_journal_v2, (collected_number)cache_efficiency_stats.prep_time_in_journal_v2_lookup);
2490
- rrddim_set_by_pointer(st_prep_timings, rd_pass4, (collected_number)cache_efficiency_stats.prep_time_in_pass4_lookup);
2491
-
2492
- rrdset_done(st_prep_timings);
2493
- }
2494
-
2495
- {
2496
- static RRDSET *st_query_timings = NULL;
2497
- static RRDDIM *rd_init = NULL;
2498
- static RRDDIM *rd_prep_wait = NULL;
2499
- static RRDDIM *rd_next_page_disk_fast = NULL;
2500
- static RRDDIM *rd_next_page_disk_slow = NULL;
2501
- static RRDDIM *rd_next_page_preload_fast = NULL;
2502
- static RRDDIM *rd_next_page_preload_slow = NULL;
2503
-
2504
- if (unlikely(!st_query_timings)) {
2505
- st_query_timings = rrdset_create_localhost(
2506
- "netdata",
2507
- "dbengine_query_timings",
2508
- NULL,
2509
- "dbengine query router",
2510
- NULL,
2511
- "Netdata Query Timings",
2512
- "usec/s",
2513
- "netdata",
2514
- "stats",
2515
- priority,
2516
- localhost->rrd_update_every,
2517
- RRDSET_TYPE_STACKED);
2518
-
2519
- rd_init = rrddim_add(st_query_timings, "init", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2520
- rd_prep_wait = rrddim_add(st_query_timings, "prep wait", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2521
- rd_next_page_disk_fast = rrddim_add(st_query_timings, "next page disk fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2522
- rd_next_page_disk_slow = rrddim_add(st_query_timings, "next page disk slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2523
- rd_next_page_preload_fast = rrddim_add(st_query_timings, "next page preload fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2524
- rd_next_page_preload_slow = rrddim_add(st_query_timings, "next page preload slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2525
- }
2526
- priority++;
2527
-
2528
- rrddim_set_by_pointer(st_query_timings, rd_init, (collected_number)cache_efficiency_stats.query_time_init);
2529
- rrddim_set_by_pointer(st_query_timings, rd_prep_wait, (collected_number)cache_efficiency_stats.query_time_wait_for_prep);
2530
- rrddim_set_by_pointer(st_query_timings, rd_next_page_disk_fast, (collected_number)cache_efficiency_stats.query_time_to_fast_disk_next_page);
2531
- rrddim_set_by_pointer(st_query_timings, rd_next_page_disk_slow, (collected_number)cache_efficiency_stats.query_time_to_slow_disk_next_page);
2532
- rrddim_set_by_pointer(st_query_timings, rd_next_page_preload_fast, (collected_number)cache_efficiency_stats.query_time_to_fast_preload_next_page);
2533
- rrddim_set_by_pointer(st_query_timings, rd_next_page_preload_slow, (collected_number)cache_efficiency_stats.query_time_to_slow_preload_next_page);
2534
-
2535
- rrdset_done(st_query_timings);
2536
- }
2537
-
2538
- if(netdata_rwlock_tryrdlock(&rrd_rwlock) == 0) {
2539
- priority = 135400;
2540
-
2541
- RRDHOST *host;
2542
- unsigned long long stats_array[RRDENG_NR_STATS] = {0};
2543
- unsigned long long local_stats_array[RRDENG_NR_STATS];
2544
- unsigned dbengine_contexts = 0, counted_multihost_db[RRD_STORAGE_TIERS] = { 0 }, i;
2545
-
2546
- rrdhost_foreach_read(host) {
2547
- if (!rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) {
2548
-
2549
- /* get localhost's DB engine's statistics for each tier */
2550
- for(size_t tier = 0; tier < storage_tiers ;tier++) {
2551
- if(host->db[tier].mode != RRD_MEMORY_MODE_DBENGINE) continue;
2552
- if(!host->db[tier].si) continue;
2553
-
2554
- if(counted_multihost_db[tier])
2555
- continue;
2556
- else
2557
- counted_multihost_db[tier] = 1;
2558
-
2559
- ++dbengine_contexts;
2560
- rrdeng_get_37_statistics((struct rrdengine_instance *)host->db[tier].si, local_stats_array);
2561
- for (i = 0; i < RRDENG_NR_STATS; ++i) {
2562
- /* aggregate statistics across hosts */
2563
- stats_array[i] += local_stats_array[i];
2564
- }
2565
- }
2566
- }
2567
- }
2568
- rrd_rdunlock();
2569
-
2570
- if (dbengine_contexts) {
2571
- /* deduplicate global statistics by getting the ones from the last context */
2572
- stats_array[30] = local_stats_array[30];
2573
- stats_array[31] = local_stats_array[31];
2574
- stats_array[32] = local_stats_array[32];
2575
- stats_array[34] = local_stats_array[34];
2576
- stats_array[36] = local_stats_array[36];
2577
-
2578
- // ----------------------------------------------------------------
2579
-
2580
- {
2581
- static RRDSET *st_compression = NULL;
2582
- static RRDDIM *rd_savings = NULL;
2583
-
2584
- if (unlikely(!st_compression)) {
2585
- st_compression = rrdset_create_localhost(
2586
- "netdata",
2587
- "dbengine_compression_ratio",
2588
- NULL,
2589
- "dbengine io",
2590
- NULL,
2591
- "Netdata DB engine data extents' compression savings ratio",
2592
- "percentage",
2593
- "netdata",
2594
- "stats",
2595
- priority,
2596
- localhost->rrd_update_every,
2597
- RRDSET_TYPE_LINE);
2598
-
2599
- rd_savings = rrddim_add(st_compression, "savings", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
2600
- }
2601
- priority++;
2602
-
2603
- unsigned long long ratio;
2604
- unsigned long long compressed_content_size = stats_array[12];
2605
- unsigned long long content_size = stats_array[11];
2606
-
2607
- if (content_size) {
2608
- // allow negative savings
2609
- ratio = ((content_size - compressed_content_size) * 100 * 1000) / content_size;
2610
- } else {
2611
- ratio = 0;
2612
- }
2613
- rrddim_set_by_pointer(st_compression, rd_savings, ratio);
2614
-
2615
- rrdset_done(st_compression);
2616
- }
2617
-
2618
- // ----------------------------------------------------------------
2619
-
2620
- {
2621
- static RRDSET *st_io_stats = NULL;
2622
- static RRDDIM *rd_reads = NULL;
2623
- static RRDDIM *rd_writes = NULL;
2624
-
2625
- if (unlikely(!st_io_stats)) {
2626
- st_io_stats = rrdset_create_localhost(
2627
- "netdata",
2628
- "dbengine_io_throughput",
2629
- NULL,
2630
- "dbengine io",
2631
- NULL,
2632
- "Netdata DB engine I/O throughput",
2633
- "MiB/s",
2634
- "netdata",
2635
- "stats",
2636
- priority,
2637
- localhost->rrd_update_every,
2638
- RRDSET_TYPE_LINE);
2639
-
2640
- rd_reads = rrddim_add(st_io_stats, "reads", NULL, 1, 1024 * 1024, RRD_ALGORITHM_INCREMENTAL);
2641
- rd_writes = rrddim_add(st_io_stats, "writes", NULL, -1, 1024 * 1024, RRD_ALGORITHM_INCREMENTAL);
2642
- }
2643
- priority++;
2644
-
2645
- rrddim_set_by_pointer(st_io_stats, rd_reads, (collected_number)stats_array[17]);
2646
- rrddim_set_by_pointer(st_io_stats, rd_writes, (collected_number)stats_array[15]);
2647
- rrdset_done(st_io_stats);
2648
- }
2649
-
2650
- // ----------------------------------------------------------------
2651
-
2652
- {
2653
- static RRDSET *st_io_stats = NULL;
2654
- static RRDDIM *rd_reads = NULL;
2655
- static RRDDIM *rd_writes = NULL;
2656
-
2657
- if (unlikely(!st_io_stats)) {
2658
- st_io_stats = rrdset_create_localhost(
2659
- "netdata",
2660
- "dbengine_io_operations",
2661
- NULL,
2662
- "dbengine io",
2663
- NULL,
2664
- "Netdata DB engine I/O operations",
2665
- "operations/s",
2666
- "netdata",
2667
- "stats",
2668
- priority,
2669
- localhost->rrd_update_every,
2670
- RRDSET_TYPE_LINE);
2671
-
2672
- rd_reads = rrddim_add(st_io_stats, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2673
- rd_writes = rrddim_add(st_io_stats, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2674
- }
2675
- priority++;
2676
-
2677
- rrddim_set_by_pointer(st_io_stats, rd_reads, (collected_number)stats_array[18]);
2678
- rrddim_set_by_pointer(st_io_stats, rd_writes, (collected_number)stats_array[16]);
2679
- rrdset_done(st_io_stats);
2680
- }
2681
-
2682
- // ----------------------------------------------------------------
2683
-
2684
- {
2685
- static RRDSET *st_errors = NULL;
2686
- static RRDDIM *rd_fs_errors = NULL;
2687
- static RRDDIM *rd_io_errors = NULL;
2688
- static RRDDIM *pg_cache_over_half_dirty_events = NULL;
2689
-
2690
- if (unlikely(!st_errors)) {
2691
- st_errors = rrdset_create_localhost(
2692
- "netdata",
2693
- "dbengine_global_errors",
2694
- NULL,
2695
- "dbengine io",
2696
- NULL,
2697
- "Netdata DB engine errors",
2698
- "errors/s",
2699
- "netdata",
2700
- "stats",
2701
- priority,
2702
- localhost->rrd_update_every,
2703
- RRDSET_TYPE_LINE);
2704
-
2705
- rd_io_errors = rrddim_add(st_errors, "io_errors", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2706
- rd_fs_errors = rrddim_add(st_errors, "fs_errors", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2707
- pg_cache_over_half_dirty_events =
2708
- rrddim_add(st_errors, "pg_cache_over_half_dirty_events", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2709
- }
2710
- priority++;
2711
-
2712
- rrddim_set_by_pointer(st_errors, rd_io_errors, (collected_number)stats_array[30]);
2713
- rrddim_set_by_pointer(st_errors, rd_fs_errors, (collected_number)stats_array[31]);
2714
- rrddim_set_by_pointer(st_errors, pg_cache_over_half_dirty_events, (collected_number)stats_array[34]);
2715
- rrdset_done(st_errors);
2716
- }
2717
-
2718
- // ----------------------------------------------------------------
2719
-
2720
- {
2721
- static RRDSET *st_fd = NULL;
2722
- static RRDDIM *rd_fd_current = NULL;
2723
- static RRDDIM *rd_fd_max = NULL;
2724
-
2725
- if (unlikely(!st_fd)) {
2726
- st_fd = rrdset_create_localhost(
2727
- "netdata",
2728
- "dbengine_global_file_descriptors",
2729
- NULL,
2730
- "dbengine io",
2731
- NULL,
2732
- "Netdata DB engine File Descriptors",
2733
- "descriptors",
2734
- "netdata",
2735
- "stats",
2736
- priority,
2737
- localhost->rrd_update_every,
2738
- RRDSET_TYPE_LINE);
2739
-
2740
- rd_fd_current = rrddim_add(st_fd, "current", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2741
- rd_fd_max = rrddim_add(st_fd, "max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2742
- }
2743
- priority++;
2744
-
2745
- rrddim_set_by_pointer(st_fd, rd_fd_current, (collected_number)stats_array[32]);
2746
- /* Careful here, modify this accordingly if the File-Descriptor budget ever changes */
2747
- rrddim_set_by_pointer(st_fd, rd_fd_max, (collected_number)rlimit_nofile.rlim_cur / 4);
2748
- rrdset_done(st_fd);
2749
- }
2750
- }
2751
- }
2752
-}
2753
-#endif // ENABLE_DBENGINE
2754
-
2755
-static void update_strings_charts() {
2756
- static RRDSET *st_ops = NULL, *st_entries = NULL, *st_mem = NULL;
2757
- static RRDDIM *rd_ops_inserts = NULL, *rd_ops_deletes = NULL;
2758
- static RRDDIM *rd_entries_entries = NULL;
2759
- static RRDDIM *rd_mem = NULL;
2760
-#ifdef NETDATA_INTERNAL_CHECKS
2761
- static RRDDIM *rd_entries_refs = NULL, *rd_ops_releases = NULL, *rd_ops_duplications = NULL, *rd_ops_searches = NULL;
2762
-#endif
2763
-
2764
- size_t inserts, deletes, searches, entries, references, memory, duplications, releases;
2765
-
2766
- string_statistics(&inserts, &deletes, &searches, &entries, &references, &memory, &duplications, &releases);
2767
-
2768
- if (unlikely(!st_ops)) {
2769
- st_ops = rrdset_create_localhost(
2770
- "netdata"
2771
- , "strings_ops"
2772
- , NULL
2773
- , "strings"
2774
- , NULL
2775
- , "Strings operations"
2776
- , "ops/s"
2777
- , "netdata"
2778
- , "stats"
2779
- , 910000
2780
- , localhost->rrd_update_every
2781
- , RRDSET_TYPE_LINE);
2782
-
2783
- rd_ops_inserts = rrddim_add(st_ops, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2784
- rd_ops_deletes = rrddim_add(st_ops, "deletes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2785
-#ifdef NETDATA_INTERNAL_CHECKS
2786
- rd_ops_searches = rrddim_add(st_ops, "searches", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2787
- rd_ops_duplications = rrddim_add(st_ops, "duplications", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2788
- rd_ops_releases = rrddim_add(st_ops, "releases", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2789
-#endif
2790
- }
2791
-
2792
- rrddim_set_by_pointer(st_ops, rd_ops_inserts, (collected_number)inserts);
2793
- rrddim_set_by_pointer(st_ops, rd_ops_deletes, (collected_number)deletes);
2794
-#ifdef NETDATA_INTERNAL_CHECKS
2795
- rrddim_set_by_pointer(st_ops, rd_ops_searches, (collected_number)searches);
2796
- rrddim_set_by_pointer(st_ops, rd_ops_duplications, (collected_number)duplications);
2797
- rrddim_set_by_pointer(st_ops, rd_ops_releases, (collected_number)releases);
2798
-#endif
2799
- rrdset_done(st_ops);
2800
-
2801
- if (unlikely(!st_entries)) {
2802
- st_entries = rrdset_create_localhost(
2803
- "netdata"
2804
- , "strings_entries"
2805
- , NULL
2806
- , "strings"
2807
- , NULL
2808
- , "Strings entries"
2809
- , "entries"
2810
- , "netdata"
2811
- , "stats"
2812
- , 910001
2813
- , localhost->rrd_update_every
2814
- , RRDSET_TYPE_AREA);
2815
-
2816
- rd_entries_entries = rrddim_add(st_entries, "entries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2817
-#ifdef NETDATA_INTERNAL_CHECKS
2818
- rd_entries_refs = rrddim_add(st_entries, "references", NULL, 1, -1, RRD_ALGORITHM_ABSOLUTE);
2819
-#endif
2820
- }
2821
-
2822
- rrddim_set_by_pointer(st_entries, rd_entries_entries, (collected_number)entries);
2823
-#ifdef NETDATA_INTERNAL_CHECKS
2824
- rrddim_set_by_pointer(st_entries, rd_entries_refs, (collected_number)references);
2825
-#endif
2826
- rrdset_done(st_entries);
2827
-
2828
- if (unlikely(!st_mem)) {
2829
- st_mem = rrdset_create_localhost(
2830
- "netdata"
2831
- , "strings_memory"
2832
- , NULL
2833
- , "strings"
2834
- , NULL
2835
- , "Strings memory"
2836
- , "bytes"
2837
- , "netdata"
2838
- , "stats"
2839
- , 910001
2840
- , localhost->rrd_update_every
2841
- , RRDSET_TYPE_AREA);
2842
-
2843
- rd_mem = rrddim_add(st_mem, "memory", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2844
- }
2845
-
2846
- rrddim_set_by_pointer(st_mem, rd_mem, (collected_number)memory);
2847
- rrdset_done(st_mem);
2848
-}
2849
-
2850
-static void update_heartbeat_charts() {
2851
- static RRDSET *st_heartbeat = NULL;
2852
- static RRDDIM *rd_heartbeat_min = NULL;
2853
- static RRDDIM *rd_heartbeat_max = NULL;
2854
- static RRDDIM *rd_heartbeat_avg = NULL;
2855
-
2856
- if (unlikely(!st_heartbeat)) {
2857
- st_heartbeat = rrdset_create_localhost(
2858
- "netdata"
2859
- , "heartbeat"
2860
- , NULL
2861
- , "heartbeat"
2862
- , NULL
2863
- , "System clock jitter"
2864
- , "microseconds"
2865
- , "netdata"
2866
- , "stats"
2867
- , 900000
2868
- , localhost->rrd_update_every
2869
- , RRDSET_TYPE_AREA);
2870
-
2871
- rd_heartbeat_min = rrddim_add(st_heartbeat, "min", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2872
- rd_heartbeat_max = rrddim_add(st_heartbeat, "max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2873
- rd_heartbeat_avg = rrddim_add(st_heartbeat, "average", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2874
- }
2875
-
2876
- usec_t min, max, average;
2877
- size_t count;
2878
-
2879
- heartbeat_statistics(&min, &max, &average, &count);
2880
-
2881
- rrddim_set_by_pointer(st_heartbeat, rd_heartbeat_min, (collected_number)min);
2882
- rrddim_set_by_pointer(st_heartbeat, rd_heartbeat_max, (collected_number)max);
2883
- rrddim_set_by_pointer(st_heartbeat, rd_heartbeat_avg, (collected_number)average);
2884
-
2885
- rrdset_done(st_heartbeat);
2886
-}
2887
-
2888
-// ---------------------------------------------------------------------------------------------------------------------
2889
-// dictionary statistics
2890
-
2891
-struct dictionary_stats dictionary_stats_category_collectors = { .name = "collectors" };
2892
-struct dictionary_stats dictionary_stats_category_rrdhost = { .name = "rrdhost" };
2893
-struct dictionary_stats dictionary_stats_category_rrdset_rrddim = { .name = "rrdset_rrddim" };
2894
-struct dictionary_stats dictionary_stats_category_rrdcontext = { .name = "context" };
2895
-struct dictionary_stats dictionary_stats_category_rrdlabels = { .name = "labels" };
2896
-struct dictionary_stats dictionary_stats_category_rrdhealth = { .name = "health" };
2897
-struct dictionary_stats dictionary_stats_category_functions = { .name = "functions" };
2898
-struct dictionary_stats dictionary_stats_category_replication = { .name = "replication" };
2899
-
2900
-#ifdef DICT_WITH_STATS
2901
-struct dictionary_categories {
2902
- struct dictionary_stats *stats;
2903
- const char *family;
2904
- const char *context_prefix;
2905
- int priority;
2906
-
2907
- RRDSET *st_dicts;
2908
- RRDDIM *rd_dicts_active;
2909
- RRDDIM *rd_dicts_deleted;
2910
-
2911
- RRDSET *st_items;
2912
- RRDDIM *rd_items_entries;
2913
- RRDDIM *rd_items_referenced;
2914
- RRDDIM *rd_items_pending_deletion;
2915
-
2916
- RRDSET *st_ops;
2917
- RRDDIM *rd_ops_creations;
2918
- RRDDIM *rd_ops_destructions;
2919
- RRDDIM *rd_ops_flushes;
2920
- RRDDIM *rd_ops_traversals;
2921
- RRDDIM *rd_ops_walkthroughs;
2922
- RRDDIM *rd_ops_garbage_collections;
2923
- RRDDIM *rd_ops_searches;
2924
- RRDDIM *rd_ops_inserts;
2925
- RRDDIM *rd_ops_resets;
2926
- RRDDIM *rd_ops_deletes;
2927
-
2928
- RRDSET *st_callbacks;
2929
- RRDDIM *rd_callbacks_inserts;
2930
- RRDDIM *rd_callbacks_conflicts;
2931
- RRDDIM *rd_callbacks_reacts;
2932
- RRDDIM *rd_callbacks_deletes;
2933
-
2934
- RRDSET *st_memory;
2935
- RRDDIM *rd_memory_indexed;
2936
- RRDDIM *rd_memory_values;
2937
- RRDDIM *rd_memory_dict;
2938
-
2939
- RRDSET *st_spins;
2940
- RRDDIM *rd_spins_use;
2941
- RRDDIM *rd_spins_search;
2942
- RRDDIM *rd_spins_insert;
2943
- RRDDIM *rd_spins_delete;
2944
-
2945
-} dictionary_categories[] = {
2946
- { .stats = &dictionary_stats_category_collectors, "dictionaries collectors", "dictionaries", 900000 },
2947
- { .stats = &dictionary_stats_category_rrdhost, "dictionaries hosts", "dictionaries", 900000 },
2948
- { .stats = &dictionary_stats_category_rrdset_rrddim, "dictionaries rrd", "dictionaries", 900000 },
2949
- { .stats = &dictionary_stats_category_rrdcontext, "dictionaries contexts", "dictionaries", 900000 },
2950
- { .stats = &dictionary_stats_category_rrdlabels, "dictionaries labels", "dictionaries", 900000 },
2951
- { .stats = &dictionary_stats_category_rrdhealth, "dictionaries health", "dictionaries", 900000 },
2952
- { .stats = &dictionary_stats_category_functions, "dictionaries functions", "dictionaries", 900000 },
2953
- { .stats = &dictionary_stats_category_replication, "dictionaries replication", "dictionaries", 900000 },
2954
- { .stats = &dictionary_stats_category_other, "dictionaries other", "dictionaries", 900000 },
2955
-
2956
- // terminator
2957
- { .stats = NULL, NULL, NULL, 0 },
2958
-};
2959
-
2960
-#define load_dictionary_stats_entry(x) total += (size_t)(stats.x = __atomic_load_n(&c->stats->x, __ATOMIC_RELAXED))
2961
-
2962
-static void update_dictionary_category_charts(struct dictionary_categories *c) {
2963
- struct dictionary_stats stats;
2964
- stats.name = c->stats->name;
2965
-
2966
- // ------------------------------------------------------------------------
2967
-
2968
- size_t total = 0;
2969
- load_dictionary_stats_entry(dictionaries.active);
2970
- load_dictionary_stats_entry(dictionaries.deleted);
2971
-
2972
- if(c->st_dicts || total != 0) {
2973
- if (unlikely(!c->st_dicts)) {
2974
- char id[RRD_ID_LENGTH_MAX + 1];
2975
- snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.dictionaries", c->context_prefix, stats.name);
2976
-
2977
- char context[RRD_ID_LENGTH_MAX + 1];
2978
- snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.dictionaries", c->context_prefix);
2979
-
2980
- c->st_dicts = rrdset_create_localhost(
2981
- "netdata"
2982
- , id
2983
- , NULL
2984
- , c->family
2985
- , context
2986
- , "Dictionaries"
2987
- , "dictionaries"
2988
- , "netdata"
2989
- , "stats"
2990
- , c->priority + 0
2991
- , localhost->rrd_update_every
2992
- , RRDSET_TYPE_LINE
2993
- );
2994
-
2995
- c->rd_dicts_active = rrddim_add(c->st_dicts, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2996
- c->rd_dicts_deleted = rrddim_add(c->st_dicts, "deleted", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
2997
-
2998
- rrdlabels_add(c->st_dicts->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
2999
- }
3000
-
3001
- rrddim_set_by_pointer(c->st_dicts, c->rd_dicts_active, (collected_number)stats.dictionaries.active);
3002
- rrddim_set_by_pointer(c->st_dicts, c->rd_dicts_deleted, (collected_number)stats.dictionaries.deleted);
3003
- rrdset_done(c->st_dicts);
3004
- }
3005
-
3006
- // ------------------------------------------------------------------------
3007
-
3008
- total = 0;
3009
- load_dictionary_stats_entry(items.entries);
3010
- load_dictionary_stats_entry(items.referenced);
3011
- load_dictionary_stats_entry(items.pending_deletion);
3012
-
3013
- if(c->st_items || total != 0) {
3014
- if (unlikely(!c->st_items)) {
3015
- char id[RRD_ID_LENGTH_MAX + 1];
3016
- snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.items", c->context_prefix, stats.name);
3017
-
3018
- char context[RRD_ID_LENGTH_MAX + 1];
3019
- snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.items", c->context_prefix);
3020
-
3021
- c->st_items = rrdset_create_localhost(
3022
- "netdata"
3023
- , id
3024
- , NULL
3025
- , c->family
3026
- , context
3027
- , "Dictionary Items"
3028
- , "items"
3029
- , "netdata"
3030
- , "stats"
3031
- , c->priority + 1
3032
- , localhost->rrd_update_every
3033
- , RRDSET_TYPE_LINE
3034
- );
3035
-
3036
- c->rd_items_entries = rrddim_add(c->st_items, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3037
- c->rd_items_pending_deletion = rrddim_add(c->st_items, "deleted", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
3038
- c->rd_items_referenced = rrddim_add(c->st_items, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3039
-
3040
- rrdlabels_add(c->st_items->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
3041
- }
3042
-
3043
- rrddim_set_by_pointer(c->st_items, c->rd_items_entries, stats.items.entries);
3044
- rrddim_set_by_pointer(c->st_items, c->rd_items_pending_deletion, stats.items.pending_deletion);
3045
- rrddim_set_by_pointer(c->st_items, c->rd_items_referenced, stats.items.referenced);
3046
- rrdset_done(c->st_items);
3047
- }
3048
-
3049
- // ------------------------------------------------------------------------
3050
-
3051
- total = 0;
3052
- load_dictionary_stats_entry(ops.creations);
3053
- load_dictionary_stats_entry(ops.destructions);
3054
- load_dictionary_stats_entry(ops.flushes);
3055
- load_dictionary_stats_entry(ops.traversals);
3056
- load_dictionary_stats_entry(ops.walkthroughs);
3057
- load_dictionary_stats_entry(ops.garbage_collections);
3058
- load_dictionary_stats_entry(ops.searches);
3059
- load_dictionary_stats_entry(ops.inserts);
3060
- load_dictionary_stats_entry(ops.resets);
3061
- load_dictionary_stats_entry(ops.deletes);
3062
-
3063
- if(c->st_ops || total != 0) {
3064
- if (unlikely(!c->st_ops)) {
3065
- char id[RRD_ID_LENGTH_MAX + 1];
3066
- snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.ops", c->context_prefix, stats.name);
3067
-
3068
- char context[RRD_ID_LENGTH_MAX + 1];
3069
- snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.ops", c->context_prefix);
3070
-
3071
- c->st_ops = rrdset_create_localhost(
3072
- "netdata"
3073
- , id
3074
- , NULL
3075
- , c->family
3076
- , context
3077
- , "Dictionary Operations"
3078
- , "ops/s"
3079
- , "netdata"
3080
- , "stats"
3081
- , c->priority + 2
3082
- , localhost->rrd_update_every
3083
- , RRDSET_TYPE_LINE
3084
- );
3085
-
3086
- c->rd_ops_creations = rrddim_add(c->st_ops, "creations", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3087
- c->rd_ops_destructions = rrddim_add(c->st_ops, "destructions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3088
- c->rd_ops_flushes = rrddim_add(c->st_ops, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3089
- c->rd_ops_traversals = rrddim_add(c->st_ops, "traversals", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3090
- c->rd_ops_walkthroughs = rrddim_add(c->st_ops, "walkthroughs", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3091
- c->rd_ops_garbage_collections = rrddim_add(c->st_ops, "garbage_collections", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3092
- c->rd_ops_searches = rrddim_add(c->st_ops, "searches", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3093
- c->rd_ops_inserts = rrddim_add(c->st_ops, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3094
- c->rd_ops_resets = rrddim_add(c->st_ops, "resets", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3095
- c->rd_ops_deletes = rrddim_add(c->st_ops, "deletes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3096
-
3097
- rrdlabels_add(c->st_ops->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
3098
- }
3099
-
3100
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_creations, (collected_number)stats.ops.creations);
3101
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_destructions, (collected_number)stats.ops.destructions);
3102
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_flushes, (collected_number)stats.ops.flushes);
3103
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_traversals, (collected_number)stats.ops.traversals);
3104
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_walkthroughs, (collected_number)stats.ops.walkthroughs);
3105
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_garbage_collections, (collected_number)stats.ops.garbage_collections);
3106
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_searches, (collected_number)stats.ops.searches);
3107
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_inserts, (collected_number)stats.ops.inserts);
3108
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_resets, (collected_number)stats.ops.resets);
3109
- rrddim_set_by_pointer(c->st_ops, c->rd_ops_deletes, (collected_number)stats.ops.deletes);
3110
-
3111
- rrdset_done(c->st_ops);
3112
- }
3113
-
3114
- // ------------------------------------------------------------------------
3115
-
3116
- total = 0;
3117
- load_dictionary_stats_entry(callbacks.inserts);
3118
- load_dictionary_stats_entry(callbacks.conflicts);
3119
- load_dictionary_stats_entry(callbacks.reacts);
3120
- load_dictionary_stats_entry(callbacks.deletes);
3121
-
3122
- if(c->st_callbacks || total != 0) {
3123
- if (unlikely(!c->st_callbacks)) {
3124
- char id[RRD_ID_LENGTH_MAX + 1];
3125
- snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.callbacks", c->context_prefix, stats.name);
3126
-
3127
- char context[RRD_ID_LENGTH_MAX + 1];
3128
- snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.callbacks", c->context_prefix);
3129
-
3130
- c->st_callbacks = rrdset_create_localhost(
3131
- "netdata"
3132
- , id
3133
- , NULL
3134
- , c->family
3135
- , context
3136
- , "Dictionary Callbacks"
3137
- , "callbacks/s"
3138
- , "netdata"
3139
- , "stats"
3140
- , c->priority + 3
3141
- , localhost->rrd_update_every
3142
- , RRDSET_TYPE_LINE
3143
- );
3144
-
3145
- c->rd_callbacks_inserts = rrddim_add(c->st_callbacks, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3146
- c->rd_callbacks_deletes = rrddim_add(c->st_callbacks, "deletes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3147
- c->rd_callbacks_conflicts = rrddim_add(c->st_callbacks, "conflicts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3148
- c->rd_callbacks_reacts = rrddim_add(c->st_callbacks, "reacts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3149
-
3150
- rrdlabels_add(c->st_callbacks->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
3151
- }
3152
-
3153
- rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_inserts, (collected_number)stats.callbacks.inserts);
3154
- rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_conflicts, (collected_number)stats.callbacks.conflicts);
3155
- rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_reacts, (collected_number)stats.callbacks.reacts);
3156
- rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_deletes, (collected_number)stats.callbacks.deletes);
3157
-
3158
- rrdset_done(c->st_callbacks);
3159
- }
3160
-
3161
- // ------------------------------------------------------------------------
3162
-
3163
- total = 0;
3164
- load_dictionary_stats_entry(memory.index);
3165
- load_dictionary_stats_entry(memory.values);
3166
- load_dictionary_stats_entry(memory.dict);
3167
-
3168
- if(c->st_memory || total != 0) {
3169
- if (unlikely(!c->st_memory)) {
3170
- char id[RRD_ID_LENGTH_MAX + 1];
3171
- snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.memory", c->context_prefix, stats.name);
3172
-
3173
- char context[RRD_ID_LENGTH_MAX + 1];
3174
- snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.memory", c->context_prefix);
3175
-
3176
- c->st_memory = rrdset_create_localhost(
3177
- "netdata"
3178
- , id
3179
- , NULL
3180
- , c->family
3181
- , context
3182
- , "Dictionary Memory"
3183
- , "bytes"
3184
- , "netdata"
3185
- , "stats"
3186
- , c->priority + 4
3187
- , localhost->rrd_update_every
3188
- , RRDSET_TYPE_STACKED
3189
- );
3190
-
3191
- c->rd_memory_indexed = rrddim_add(c->st_memory, "index", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3192
- c->rd_memory_values = rrddim_add(c->st_memory, "data", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3193
- c->rd_memory_dict = rrddim_add(c->st_memory, "structures", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3194
-
3195
- rrdlabels_add(c->st_memory->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
3196
- }
3197
-
3198
- rrddim_set_by_pointer(c->st_memory, c->rd_memory_indexed, (collected_number)stats.memory.index);
3199
- rrddim_set_by_pointer(c->st_memory, c->rd_memory_values, (collected_number)stats.memory.values);
3200
- rrddim_set_by_pointer(c->st_memory, c->rd_memory_dict, (collected_number)stats.memory.dict);
3201
-
3202
- rrdset_done(c->st_memory);
3203
- }
3204
-
3205
- // ------------------------------------------------------------------------
3206
-
3207
- total = 0;
3208
- load_dictionary_stats_entry(spin_locks.use_spins);
3209
- load_dictionary_stats_entry(spin_locks.search_spins);
3210
- load_dictionary_stats_entry(spin_locks.insert_spins);
3211
- load_dictionary_stats_entry(spin_locks.delete_spins);
3212
-
3213
- if(c->st_spins || total != 0) {
3214
- if (unlikely(!c->st_spins)) {
3215
- char id[RRD_ID_LENGTH_MAX + 1];
3216
- snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.spins", c->context_prefix, stats.name);
3217
-
3218
- char context[RRD_ID_LENGTH_MAX + 1];
3219
- snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.spins", c->context_prefix);
3220
-
3221
- c->st_spins = rrdset_create_localhost(
3222
- "netdata"
3223
- , id
3224
- , NULL
3225
- , c->family
3226
- , context
3227
- , "Dictionary Spins"
3228
- , "count"
3229
- , "netdata"
3230
- , "stats"
3231
- , c->priority + 5
3232
- , localhost->rrd_update_every
3233
- , RRDSET_TYPE_LINE
3234
- );
3235
-
3236
- c->rd_spins_use = rrddim_add(c->st_spins, "use", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3237
- c->rd_spins_search = rrddim_add(c->st_spins, "search", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3238
- c->rd_spins_insert = rrddim_add(c->st_spins, "insert", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3239
- c->rd_spins_delete = rrddim_add(c->st_spins, "delete", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3240
-
3241
- rrdlabels_add(c->st_spins->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
3242
- }
3243
-
3244
- rrddim_set_by_pointer(c->st_spins, c->rd_spins_use, (collected_number)stats.spin_locks.use_spins);
3245
- rrddim_set_by_pointer(c->st_spins, c->rd_spins_search, (collected_number)stats.spin_locks.search_spins);
3246
- rrddim_set_by_pointer(c->st_spins, c->rd_spins_insert, (collected_number)stats.spin_locks.insert_spins);
3247
- rrddim_set_by_pointer(c->st_spins, c->rd_spins_delete, (collected_number)stats.spin_locks.delete_spins);
3248
-
3249
- rrdset_done(c->st_spins);
3250
- }
3251
-}
3252
-
3253
-static void dictionary_statistics(void) {
3254
- for(int i = 0; dictionary_categories[i].stats ;i++) {
3255
- update_dictionary_category_charts(&dictionary_categories[i]);
3256
- }
3257
-}
3258
-#endif // DICT_WITH_STATS
3259
-
3260
-#ifdef NETDATA_TRACE_ALLOCATIONS
3261
-
3262
-struct memory_trace_data {
3263
- RRDSET *st_memory;
3264
- RRDSET *st_allocations;
3265
- RRDSET *st_avg_alloc;
3266
- RRDSET *st_ops;
3267
-};
3268
-
3269
-static int do_memory_trace_item(void *item, void *data) {
3270
- struct memory_trace_data *tmp = data;
3271
- struct malloc_trace *p = item;
3272
-
3273
- // ------------------------------------------------------------------------
3274
-
3275
- if(!p->rd_bytes)
3276
- p->rd_bytes = rrddim_add(tmp->st_memory, p->function, NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3277
-
3278
- collected_number bytes = (collected_number)__atomic_load_n(&p->bytes, __ATOMIC_RELAXED);
3279
- rrddim_set_by_pointer(tmp->st_memory, p->rd_bytes, bytes);
3280
-
3281
- // ------------------------------------------------------------------------
3282
-
3283
- if(!p->rd_allocations)
3284
- p->rd_allocations = rrddim_add(tmp->st_allocations, p->function, NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3285
-
3286
- collected_number allocs = (collected_number)__atomic_load_n(&p->allocations, __ATOMIC_RELAXED);
3287
- rrddim_set_by_pointer(tmp->st_allocations, p->rd_allocations, allocs);
3288
-
3289
- // ------------------------------------------------------------------------
3290
-
3291
- if(!p->rd_avg_alloc)
3292
- p->rd_avg_alloc = rrddim_add(tmp->st_avg_alloc, p->function, NULL, 1, 100, RRD_ALGORITHM_ABSOLUTE);
3293
-
3294
- collected_number avg_alloc = (allocs)?(bytes * 100 / allocs):0;
3295
- rrddim_set_by_pointer(tmp->st_avg_alloc, p->rd_avg_alloc, avg_alloc);
3296
-
3297
- // ------------------------------------------------------------------------
3298
-
3299
- if(!p->rd_ops)
3300
- p->rd_ops = rrddim_add(tmp->st_ops, p->function, NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
3301
-
3302
- collected_number ops = 0;
3303
- ops += (collected_number)__atomic_load_n(&p->malloc_calls, __ATOMIC_RELAXED);
3304
- ops += (collected_number)__atomic_load_n(&p->calloc_calls, __ATOMIC_RELAXED);
3305
- ops += (collected_number)__atomic_load_n(&p->realloc_calls, __ATOMIC_RELAXED);
3306
- ops += (collected_number)__atomic_load_n(&p->strdup_calls, __ATOMIC_RELAXED);
3307
- ops += (collected_number)__atomic_load_n(&p->free_calls, __ATOMIC_RELAXED);
3308
- rrddim_set_by_pointer(tmp->st_ops, p->rd_ops, ops);
3309
-
3310
- // ------------------------------------------------------------------------
3311
-
3312
- return 1;
3313
-}
3314
-static void malloc_trace_statistics(void) {
3315
- static struct memory_trace_data tmp = {
3316
- .st_memory = NULL,
3317
- .st_allocations = NULL,
3318
- .st_avg_alloc = NULL,
3319
- .st_ops = NULL,
3320
- };
3321
-
3322
- if(!tmp.st_memory) {
3323
- tmp.st_memory = rrdset_create_localhost(
3324
- "netdata"
3325
- , "memory_size"
3326
- , NULL
3327
- , "memory"
3328
- , "netdata.memory.size"
3329
- , "Netdata Memory Used by Function"
3330
- , "bytes"
3331
- , "netdata"
3332
- , "stats"
3333
- , 900000
3334
- , localhost->rrd_update_every
3335
- , RRDSET_TYPE_STACKED
3336
- );
3337
- }
3338
-
3339
- if(!tmp.st_ops) {
3340
- tmp.st_ops = rrdset_create_localhost(
3341
- "netdata"
3342
- , "memory_operations"
3343
- , NULL
3344
- , "memory"
3345
- , "netdata.memory.operations"
3346
- , "Netdata Memory Operations by Function"
3347
- , "ops/s"
3348
- , "netdata"
3349
- , "stats"
3350
- , 900001
3351
- , localhost->rrd_update_every
3352
- , RRDSET_TYPE_LINE
3353
- );
3354
- }
3355
-
3356
- if(!tmp.st_allocations) {
3357
- tmp.st_allocations = rrdset_create_localhost(
3358
- "netdata"
3359
- , "memory_allocations"
3360
- , NULL
3361
- , "memory"
3362
- , "netdata.memory.allocations"
3363
- , "Netdata Memory Allocations by Function"
3364
- , "allocations"
3365
- , "netdata"
3366
- , "stats"
3367
- , 900002
3368
- , localhost->rrd_update_every
3369
- , RRDSET_TYPE_STACKED
3370
- );
3371
- }
3372
-
3373
- if(!tmp.st_avg_alloc) {
3374
- tmp.st_avg_alloc = rrdset_create_localhost(
3375
- "netdata"
3376
- , "memory_avg_alloc"
3377
- , NULL
3378
- , "memory"
3379
- , "netdata.memory.avg_alloc"
3380
- , "Netdata Average Allocation Size by Function"
3381
- , "bytes"
3382
- , "netdata"
3383
- , "stats"
3384
- , 900003
3385
- , localhost->rrd_update_every
3386
- , RRDSET_TYPE_LINE
3387
- );
3388
- }
3389
-
3390
- malloc_trace_walkthrough(do_memory_trace_item, &tmp);
3391
-
3392
- rrdset_done(tmp.st_memory);
3393
- rrdset_done(tmp.st_ops);
3394
- rrdset_done(tmp.st_allocations);
3395
- rrdset_done(tmp.st_avg_alloc);
3396
-}
3397
-#endif
3398
-
3399
-// ---------------------------------------------------------------------------------------------------------------------
3400
-// worker utilization
3401
-
3402
-#define WORKERS_MIN_PERCENT_DEFAULT 10000.0
3403
-
3404
-struct worker_job_type_gs {
3405
- STRING *name;
3406
- STRING *units;
3407
-
3408
- size_t jobs_started;
3409
- usec_t busy_time;
3410
-
3411
- RRDDIM *rd_jobs_started;
3412
- RRDDIM *rd_busy_time;
3413
-
3414
- WORKER_METRIC_TYPE type;
3415
- NETDATA_DOUBLE min_value;
3416
- NETDATA_DOUBLE max_value;
3417
- NETDATA_DOUBLE sum_value;
3418
- size_t count_value;
3419
-
3420
- RRDSET *st;
3421
- RRDDIM *rd_min;
3422
- RRDDIM *rd_max;
3423
- RRDDIM *rd_avg;
3424
-};
3425
-
3426
-struct worker_thread {
3427
- pid_t pid;
3428
- bool enabled;
3429
-
3430
- bool cpu_enabled;
3431
- double cpu;
3432
-
3433
- kernel_uint_t utime;
3434
- kernel_uint_t stime;
3435
-
3436
- kernel_uint_t utime_old;
3437
- kernel_uint_t stime_old;
3438
-
3439
- usec_t collected_time;
3440
- usec_t collected_time_old;
3441
-
3442
- size_t jobs_started;
3443
- usec_t busy_time;
3444
-
3445
- struct worker_thread *next;
3446
- struct worker_thread *prev;
3447
-};
3448
-
3449
-struct worker_utilization {
3450
- const char *name;
3451
- const char *family;
3452
- size_t priority;
3453
- uint32_t flags;
3454
-
3455
- char *name_lowercase;
3456
-
3457
- struct worker_job_type_gs per_job_type[WORKER_UTILIZATION_MAX_JOB_TYPES];
3458
-
3459
- size_t workers_max_job_id;
3460
- size_t workers_registered;
3461
- size_t workers_busy;
3462
- usec_t workers_total_busy_time;
3463
- usec_t workers_total_duration;
3464
- size_t workers_total_jobs_started;
3465
- double workers_min_busy_time;
3466
- double workers_max_busy_time;
3467
-
3468
- size_t workers_cpu_registered;
3469
- double workers_cpu_min;
3470
- double workers_cpu_max;
3471
- double workers_cpu_total;
3472
-
3473
- struct worker_thread *threads;
3474
-
3475
- RRDSET *st_workers_time;
3476
- RRDDIM *rd_workers_time_avg;
3477
- RRDDIM *rd_workers_time_min;
3478
- RRDDIM *rd_workers_time_max;
3479
-
3480
- RRDSET *st_workers_cpu;
3481
- RRDDIM *rd_workers_cpu_avg;
3482
- RRDDIM *rd_workers_cpu_min;
3483
- RRDDIM *rd_workers_cpu_max;
3484
-
3485
- RRDSET *st_workers_threads;
3486
- RRDDIM *rd_workers_threads_free;
3487
- RRDDIM *rd_workers_threads_busy;
3488
-
3489
- RRDSET *st_workers_jobs_per_job_type;
3490
- RRDSET *st_workers_busy_per_job_type;
3491
-
3492
- RRDDIM *rd_total_cpu_utilizaton;
3493
-};
3494
-
3495
-static struct worker_utilization all_workers_utilization[] = {
3496
- { .name = "STATS", .family = "workers global statistics", .priority = 1000000 },
3497
- { .name = "HEALTH", .family = "workers health alarms", .priority = 1000000 },
3498
- { .name = "MLTRAIN", .family = "workers ML training", .priority = 1000000 },
3499
- { .name = "MLDETECT", .family = "workers ML detection", .priority = 1000000 },
3500
- { .name = "STREAMRCV", .family = "workers streaming receive", .priority = 1000000 },
3501
- { .name = "STREAMSND", .family = "workers streaming send", .priority = 1000000 },
3502
- { .name = "DBENGINE", .family = "workers dbengine instances", .priority = 1000000 },
3503
- { .name = "LIBUV", .family = "workers libuv threadpool", .priority = 1000000 },
3504
- { .name = "WEB", .family = "workers web server", .priority = 1000000 },
3505
- { .name = "ACLKSYNC", .family = "workers aclk sync", .priority = 1000000 },
3506
- { .name = "METASYNC", .family = "workers metadata sync", .priority = 1000000 },
3507
- { .name = "PLUGINSD", .family = "workers plugins.d", .priority = 1000000 },
3508
- { .name = "STATSD", .family = "workers plugin statsd", .priority = 1000000 },
3509
- { .name = "STATSDFLUSH", .family = "workers plugin statsd flush", .priority = 1000000 },
3510
- { .name = "PROC", .family = "workers plugin proc", .priority = 1000000 },
3511
- { .name = "WIN", .family = "workers plugin windows", .priority = 1000000 },
3512
- { .name = "NETDEV", .family = "workers plugin proc netdev", .priority = 1000000 },
3513
- { .name = "FREEBSD", .family = "workers plugin freebsd", .priority = 1000000 },
3514
- { .name = "MACOS", .family = "workers plugin macos", .priority = 1000000 },
3515
- { .name = "CGROUPS", .family = "workers plugin cgroups", .priority = 1000000 },
3516
- { .name = "CGROUPSDISC", .family = "workers plugin cgroups find", .priority = 1000000 },
3517
- { .name = "DISKSPACE", .family = "workers plugin diskspace", .priority = 1000000 },
3518
- { .name = "TC", .family = "workers plugin tc", .priority = 1000000 },
3519
- { .name = "TIMEX", .family = "workers plugin timex", .priority = 1000000 },
3520
- { .name = "IDLEJITTER", .family = "workers plugin idlejitter", .priority = 1000000 },
3521
- { .name = "LOGSMANAGPLG",.family = "workers plugin logs management", .priority = 1000000 },
3522
- { .name = "RRDCONTEXT", .family = "workers contexts", .priority = 1000000 },
3523
- { .name = "REPLICATION", .family = "workers replication sender", .priority = 1000000 },
3524
- { .name = "SERVICE", .family = "workers service", .priority = 1000000 },
3525
- { .name = "PROFILER", .family = "workers profile", .priority = 1000000 },
3526
-
3527
- // has to be terminated with a NULL
3528
- { .name = NULL, .family = NULL }
3529
-};
3530
-
3531
-static void workers_total_cpu_utilization_chart(void) {
3532
- size_t i, cpu_enabled = 0;
3533
- for(i = 0; all_workers_utilization[i].name ;i++)
3534
- if(all_workers_utilization[i].workers_cpu_registered) cpu_enabled++;
3535
-
3536
- if(!cpu_enabled) return;
3537
-
3538
- static RRDSET *st = NULL;
3539
-
3540
- if(!st) {
3541
- st = rrdset_create_localhost(
3542
- "netdata",
3543
- "workers_cpu",
3544
- NULL,
3545
- "workers",
3546
- "netdata.workers.cpu_total",
3547
- "Netdata Workers CPU Utilization (100% = 1 core)",
3548
- "%",
3549
- "netdata",
3550
- "stats",
3551
- 999000,
3552
- localhost->rrd_update_every,
3553
- RRDSET_TYPE_STACKED);
3554
- }
3555
-
3556
- for(i = 0; all_workers_utilization[i].name ;i++) {
3557
- struct worker_utilization *wu = &all_workers_utilization[i];
3558
- if(!wu->workers_cpu_registered) continue;
3559
-
3560
- if(!wu->rd_total_cpu_utilizaton)
3561
- wu->rd_total_cpu_utilizaton = rrddim_add(st, wu->name_lowercase, NULL, 1, 100, RRD_ALGORITHM_ABSOLUTE);
3562
-
3563
- rrddim_set_by_pointer(st, wu->rd_total_cpu_utilizaton, (collected_number)((double)wu->workers_cpu_total * 100.0));
3564
- }
3565
-
3566
- rrdset_done(st);
3567
-}
3568
-
3569
-#define WORKER_CHART_DECIMAL_PRECISION 100
3570
-
3571
-static void workers_utilization_update_chart(struct worker_utilization *wu) {
3572
- if(!wu->workers_registered) return;
3573
-
3574
- //fprintf(stderr, "%-12s WORKER UTILIZATION: %-3.2f%%, %zu jobs done, %zu running, on %zu workers, min %-3.02f%%, max %-3.02f%%.\n",
3575
- // wu->name,
3576
- // (double)wu->workers_total_busy_time * 100.0 / (double)wu->workers_total_duration,
3577
- // wu->workers_total_jobs_started, wu->workers_busy, wu->workers_registered,
3578
- // wu->workers_min_busy_time, wu->workers_max_busy_time);
3579
-
3580
- // ----------------------------------------------------------------------
3581
-
3582
- if(unlikely(!wu->st_workers_time)) {
3583
- char name[RRD_ID_LENGTH_MAX + 1];
3584
- snprintfz(name, RRD_ID_LENGTH_MAX, "workers_time_%s", wu->name_lowercase);
3585
-
3586
- char context[RRD_ID_LENGTH_MAX + 1];
3587
- snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.time", wu->name_lowercase);
3588
-
3589
- wu->st_workers_time = rrdset_create_localhost(
3590
- "netdata"
3591
- , name
3592
- , NULL
3593
- , wu->family
3594
- , context
3595
- , "Netdata Workers Busy Time (100% = all workers busy)"
3596
- , "%"
3597
- , "netdata"
3598
- , "stats"
3599
- , wu->priority
3600
- , localhost->rrd_update_every
3601
- , RRDSET_TYPE_AREA
3602
- );
3603
- }
3604
-
3605
- // we add the min and max dimensions only when we have multiple workers
3606
-
3607
- if(unlikely(!wu->rd_workers_time_min && wu->workers_registered > 1))
3608
- wu->rd_workers_time_min = rrddim_add(wu->st_workers_time, "min", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3609
-
3610
- if(unlikely(!wu->rd_workers_time_max && wu->workers_registered > 1))
3611
- wu->rd_workers_time_max = rrddim_add(wu->st_workers_time, "max", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3612
-
3613
- if(unlikely(!wu->rd_workers_time_avg))
3614
- wu->rd_workers_time_avg = rrddim_add(wu->st_workers_time, "average", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3615
-
3616
- if(unlikely(wu->workers_min_busy_time == WORKERS_MIN_PERCENT_DEFAULT)) wu->workers_min_busy_time = 0.0;
3617
-
3618
- if(wu->rd_workers_time_min)
3619
- rrddim_set_by_pointer(wu->st_workers_time, wu->rd_workers_time_min, (collected_number)((double)wu->workers_min_busy_time * WORKER_CHART_DECIMAL_PRECISION));
3620
-
3621
- if(wu->rd_workers_time_max)
3622
- rrddim_set_by_pointer(wu->st_workers_time, wu->rd_workers_time_max, (collected_number)((double)wu->workers_max_busy_time * WORKER_CHART_DECIMAL_PRECISION));
3623
-
3624
- if(wu->workers_total_duration == 0)
3625
- rrddim_set_by_pointer(wu->st_workers_time, wu->rd_workers_time_avg, 0);
3626
- else
3627
- rrddim_set_by_pointer(wu->st_workers_time, wu->rd_workers_time_avg, (collected_number)((double)wu->workers_total_busy_time * 100.0 * WORKER_CHART_DECIMAL_PRECISION / (double)wu->workers_total_duration));
3628
-
3629
- rrdset_done(wu->st_workers_time);
3630
-
3631
- // ----------------------------------------------------------------------
3632
-
3633
-#ifdef __linux__
3634
- if(wu->workers_cpu_registered || wu->st_workers_cpu) {
3635
- if(unlikely(!wu->st_workers_cpu)) {
3636
- char name[RRD_ID_LENGTH_MAX + 1];
3637
- snprintfz(name, RRD_ID_LENGTH_MAX, "workers_cpu_%s", wu->name_lowercase);
3638
-
3639
- char context[RRD_ID_LENGTH_MAX + 1];
3640
- snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.cpu", wu->name_lowercase);
3641
-
3642
- wu->st_workers_cpu = rrdset_create_localhost(
3643
- "netdata"
3644
- , name
3645
- , NULL
3646
- , wu->family
3647
- , context
3648
- , "Netdata Workers CPU Utilization (100% = all workers busy)"
3649
- , "%"
3650
- , "netdata"
3651
- , "stats"
3652
- , wu->priority + 1
3653
- , localhost->rrd_update_every
3654
- , RRDSET_TYPE_AREA
3655
- );
3656
- }
3657
-
3658
- if (unlikely(!wu->rd_workers_cpu_min && wu->workers_registered > 1))
3659
- wu->rd_workers_cpu_min = rrddim_add(wu->st_workers_cpu, "min", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3660
-
3661
- if (unlikely(!wu->rd_workers_cpu_max && wu->workers_registered > 1))
3662
- wu->rd_workers_cpu_max = rrddim_add(wu->st_workers_cpu, "max", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3663
-
3664
- if(unlikely(!wu->rd_workers_cpu_avg))
3665
- wu->rd_workers_cpu_avg = rrddim_add(wu->st_workers_cpu, "average", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3666
-
3667
- if(unlikely(wu->workers_cpu_min == WORKERS_MIN_PERCENT_DEFAULT)) wu->workers_cpu_min = 0.0;
3668
-
3669
- if(wu->rd_workers_cpu_min)
3670
- rrddim_set_by_pointer(wu->st_workers_cpu, wu->rd_workers_cpu_min, (collected_number)(wu->workers_cpu_min * WORKER_CHART_DECIMAL_PRECISION));
3671
-
3672
- if(wu->rd_workers_cpu_max)
3673
- rrddim_set_by_pointer(wu->st_workers_cpu, wu->rd_workers_cpu_max, (collected_number)(wu->workers_cpu_max * WORKER_CHART_DECIMAL_PRECISION));
3674
-
3675
- if(wu->workers_cpu_registered == 0)
3676
- rrddim_set_by_pointer(wu->st_workers_cpu, wu->rd_workers_cpu_avg, 0);
3677
- else
3678
- rrddim_set_by_pointer(wu->st_workers_cpu, wu->rd_workers_cpu_avg, (collected_number)( wu->workers_cpu_total * WORKER_CHART_DECIMAL_PRECISION / (NETDATA_DOUBLE)wu->workers_cpu_registered ));
3679
-
3680
- rrdset_done(wu->st_workers_cpu);
3681
- }
3682
-#endif
3683
-
3684
- // ----------------------------------------------------------------------
3685
-
3686
- if(unlikely(!wu->st_workers_jobs_per_job_type)) {
3687
- char name[RRD_ID_LENGTH_MAX + 1];
3688
- snprintfz(name, RRD_ID_LENGTH_MAX, "workers_jobs_by_type_%s", wu->name_lowercase);
3689
-
3690
- char context[RRD_ID_LENGTH_MAX + 1];
3691
- snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.jobs_started_by_type", wu->name_lowercase);
3692
-
3693
- wu->st_workers_jobs_per_job_type = rrdset_create_localhost(
3694
- "netdata"
3695
- , name
3696
- , NULL
3697
- , wu->family
3698
- , context
3699
- , "Netdata Workers Jobs Started by Type"
3700
- , "jobs"
3701
- , "netdata"
3702
- , "stats"
3703
- , wu->priority + 2
3704
- , localhost->rrd_update_every
3705
- , RRDSET_TYPE_STACKED
3706
- );
3707
- }
3708
-
3709
- {
3710
- size_t i;
3711
- for(i = 0; i <= wu->workers_max_job_id ;i++) {
3712
- if(unlikely(wu->per_job_type[i].type != WORKER_METRIC_IDLE_BUSY))
3713
- continue;
3714
-
3715
- if (wu->per_job_type[i].name) {
3716
-
3717
- if(unlikely(!wu->per_job_type[i].rd_jobs_started))
3718
- wu->per_job_type[i].rd_jobs_started = rrddim_add(wu->st_workers_jobs_per_job_type, string2str(wu->per_job_type[i].name), NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3719
-
3720
- rrddim_set_by_pointer(wu->st_workers_jobs_per_job_type, wu->per_job_type[i].rd_jobs_started, (collected_number)(wu->per_job_type[i].jobs_started));
3721
- }
3722
- }
3723
- }
3724
-
3725
- rrdset_done(wu->st_workers_jobs_per_job_type);
3726
-
3727
- // ----------------------------------------------------------------------
3728
-
3729
- if(unlikely(!wu->st_workers_busy_per_job_type)) {
3730
- char name[RRD_ID_LENGTH_MAX + 1];
3731
- snprintfz(name, RRD_ID_LENGTH_MAX, "workers_busy_time_by_type_%s", wu->name_lowercase);
3732
-
3733
- char context[RRD_ID_LENGTH_MAX + 1];
3734
- snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.time_by_type", wu->name_lowercase);
3735
-
3736
- wu->st_workers_busy_per_job_type = rrdset_create_localhost(
3737
- "netdata"
3738
- , name
3739
- , NULL
3740
- , wu->family
3741
- , context
3742
- , "Netdata Workers Busy Time by Type"
3743
- , "ms"
3744
- , "netdata"
3745
- , "stats"
3746
- , wu->priority + 3
3747
- , localhost->rrd_update_every
3748
- , RRDSET_TYPE_STACKED
3749
- );
3750
- }
3751
-
3752
- {
3753
- size_t i;
3754
- for(i = 0; i <= wu->workers_max_job_id ;i++) {
3755
- if(unlikely(wu->per_job_type[i].type != WORKER_METRIC_IDLE_BUSY))
3756
- continue;
3757
-
3758
- if (wu->per_job_type[i].name) {
3759
-
3760
- if(unlikely(!wu->per_job_type[i].rd_busy_time))
3761
- wu->per_job_type[i].rd_busy_time = rrddim_add(wu->st_workers_busy_per_job_type, string2str(wu->per_job_type[i].name), NULL, 1, USEC_PER_MS, RRD_ALGORITHM_ABSOLUTE);
3762
-
3763
- rrddim_set_by_pointer(wu->st_workers_busy_per_job_type, wu->per_job_type[i].rd_busy_time, (collected_number)(wu->per_job_type[i].busy_time));
3764
- }
3765
- }
3766
- }
3767
-
3768
- rrdset_done(wu->st_workers_busy_per_job_type);
3769
-
3770
- // ----------------------------------------------------------------------
3771
-
3772
- if(wu->st_workers_threads || wu->workers_registered > 1) {
3773
- if(unlikely(!wu->st_workers_threads)) {
3774
- char name[RRD_ID_LENGTH_MAX + 1];
3775
- snprintfz(name, RRD_ID_LENGTH_MAX, "workers_threads_%s", wu->name_lowercase);
3776
-
3777
- char context[RRD_ID_LENGTH_MAX + 1];
3778
- snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.threads", wu->name_lowercase);
3779
-
3780
- wu->st_workers_threads = rrdset_create_localhost(
3781
- "netdata"
3782
- , name
3783
- , NULL
3784
- , wu->family
3785
- , context
3786
- , "Netdata Workers Threads"
3787
- , "threads"
3788
- , "netdata"
3789
- , "stats"
3790
- , wu->priority + 4
3791
- , localhost->rrd_update_every
3792
- , RRDSET_TYPE_STACKED
3793
- );
3794
-
3795
- wu->rd_workers_threads_free = rrddim_add(wu->st_workers_threads, "free", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3796
- wu->rd_workers_threads_busy = rrddim_add(wu->st_workers_threads, "busy", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
3797
- }
3798
-
3799
- rrddim_set_by_pointer(wu->st_workers_threads, wu->rd_workers_threads_free, (collected_number)(wu->workers_registered - wu->workers_busy));
3800
- rrddim_set_by_pointer(wu->st_workers_threads, wu->rd_workers_threads_busy, (collected_number)(wu->workers_busy));
3801
- rrdset_done(wu->st_workers_threads);
3802
- }
3803
-
3804
- // ----------------------------------------------------------------------
3805
- // custom metric types WORKER_METRIC_ABSOLUTE
3806
-
3807
- {
3808
- size_t i;
3809
- for (i = 0; i <= wu->workers_max_job_id; i++) {
3810
- if(wu->per_job_type[i].type != WORKER_METRIC_ABSOLUTE)
3811
- continue;
3812
-
3813
- if(!wu->per_job_type[i].count_value)
3814
- continue;
3815
-
3816
- if(!wu->per_job_type[i].st) {
3817
- size_t job_name_len = string_strlen(wu->per_job_type[i].name);
3818
- if(job_name_len > RRD_ID_LENGTH_MAX) job_name_len = RRD_ID_LENGTH_MAX;
3819
-
3820
- char job_name_sanitized[job_name_len + 1];
3821
- rrdset_strncpyz_name(job_name_sanitized, string2str(wu->per_job_type[i].name), job_name_len);
3822
-
3823
- char name[RRD_ID_LENGTH_MAX + 1];
3824
- snprintfz(name, RRD_ID_LENGTH_MAX, "workers_%s_value_%s", wu->name_lowercase, job_name_sanitized);
3825
-
3826
- char context[RRD_ID_LENGTH_MAX + 1];
3827
- snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.value.%s", wu->name_lowercase, job_name_sanitized);
3828
-
3829
- char title[1000 + 1];
3830
- snprintf(title, 1000, "Netdata Workers %s value of %s", wu->name_lowercase, string2str(wu->per_job_type[i].name));
3831
-
3832
- wu->per_job_type[i].st = rrdset_create_localhost(
3833
- "netdata"
3834
- , name
3835
- , NULL
3836
- , wu->family
3837
- , context
3838
- , title
3839
- , (wu->per_job_type[i].units)?string2str(wu->per_job_type[i].units):"value"
3840
- , "netdata"
3841
- , "stats"
3842
- , wu->priority + 5 + i
3843
- , localhost->rrd_update_every
3844
- , RRDSET_TYPE_LINE
3845
- );
3846
-
3847
- wu->per_job_type[i].rd_min = rrddim_add(wu->per_job_type[i].st, "min", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3848
- wu->per_job_type[i].rd_max = rrddim_add(wu->per_job_type[i].st, "max", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3849
- wu->per_job_type[i].rd_avg = rrddim_add(wu->per_job_type[i].st, "average", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3850
- }
3851
-
3852
- rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_min, (collected_number)(wu->per_job_type[i].min_value * WORKER_CHART_DECIMAL_PRECISION));
3853
- rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_max, (collected_number)(wu->per_job_type[i].max_value * WORKER_CHART_DECIMAL_PRECISION));
3854
- rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_avg, (collected_number)(wu->per_job_type[i].sum_value / wu->per_job_type[i].count_value * WORKER_CHART_DECIMAL_PRECISION));
3855
-
3856
- rrdset_done(wu->per_job_type[i].st);
3857
- }
3858
- }
3859
-
3860
- // ----------------------------------------------------------------------
3861
- // custom metric types WORKER_METRIC_INCREMENTAL
3862
-
3863
- {
3864
- size_t i;
3865
- for (i = 0; i <= wu->workers_max_job_id ; i++) {
3866
- if(wu->per_job_type[i].type != WORKER_METRIC_INCREMENT && wu->per_job_type[i].type != WORKER_METRIC_INCREMENTAL_TOTAL)
3867
- continue;
3868
-
3869
- if(!wu->per_job_type[i].count_value)
3870
- continue;
3871
-
3872
- if(!wu->per_job_type[i].st) {
3873
- size_t job_name_len = string_strlen(wu->per_job_type[i].name);
3874
- if(job_name_len > RRD_ID_LENGTH_MAX) job_name_len = RRD_ID_LENGTH_MAX;
3875
-
3876
- char job_name_sanitized[job_name_len + 1];
3877
- rrdset_strncpyz_name(job_name_sanitized, string2str(wu->per_job_type[i].name), job_name_len);
3878
-
3879
- char name[RRD_ID_LENGTH_MAX + 1];
3880
- snprintfz(name, RRD_ID_LENGTH_MAX, "workers_%s_rate_%s", wu->name_lowercase, job_name_sanitized);
3881
-
3882
- char context[RRD_ID_LENGTH_MAX + 1];
3883
- snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.rate.%s", wu->name_lowercase, job_name_sanitized);
3884
-
3885
- char title[1000 + 1];
3886
- snprintf(title, 1000, "Netdata Workers %s rate of %s", wu->name_lowercase, string2str(wu->per_job_type[i].name));
3887
-
3888
- wu->per_job_type[i].st = rrdset_create_localhost(
3889
- "netdata"
3890
- , name
3891
- , NULL
3892
- , wu->family
3893
- , context
3894
- , title
3895
- , (wu->per_job_type[i].units)?string2str(wu->per_job_type[i].units):"rate"
3896
- , "netdata"
3897
- , "stats"
3898
- , wu->priority + 5 + i
3899
- , localhost->rrd_update_every
3900
- , RRDSET_TYPE_LINE
3901
- );
3902
-
3903
- wu->per_job_type[i].rd_min = rrddim_add(wu->per_job_type[i].st, "min", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3904
- wu->per_job_type[i].rd_max = rrddim_add(wu->per_job_type[i].st, "max", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3905
- wu->per_job_type[i].rd_avg = rrddim_add(wu->per_job_type[i].st, "average", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
3906
- }
3907
-
3908
- rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_min, (collected_number)(wu->per_job_type[i].min_value * WORKER_CHART_DECIMAL_PRECISION));
3909
- rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_max, (collected_number)(wu->per_job_type[i].max_value * WORKER_CHART_DECIMAL_PRECISION));
3910
- rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_avg, (collected_number)(wu->per_job_type[i].sum_value / wu->per_job_type[i].count_value * WORKER_CHART_DECIMAL_PRECISION));
3911
-
3912
- rrdset_done(wu->per_job_type[i].st);
3913
- }
3914
- }
3915
-}
3916
-
3917
-static void workers_utilization_reset_statistics(struct worker_utilization *wu) {
3918
- wu->workers_registered = 0;
3919
- wu->workers_busy = 0;
3920
- wu->workers_total_busy_time = 0;
3921
- wu->workers_total_duration = 0;
3922
- wu->workers_total_jobs_started = 0;
3923
- wu->workers_min_busy_time = WORKERS_MIN_PERCENT_DEFAULT;
3924
- wu->workers_max_busy_time = 0;
3925
-
3926
- wu->workers_cpu_registered = 0;
3927
- wu->workers_cpu_min = WORKERS_MIN_PERCENT_DEFAULT;
3928
- wu->workers_cpu_max = 0;
3929
- wu->workers_cpu_total = 0;
3930
-
3931
- size_t i;
3932
- for(i = 0; i < WORKER_UTILIZATION_MAX_JOB_TYPES ;i++) {
3933
- if(unlikely(!wu->name_lowercase)) {
3934
- wu->name_lowercase = strdupz(wu->name);
3935
- char *s = wu->name_lowercase;
3936
- for( ; *s ; s++) *s = tolower(*s);
3937
- }
3938
-
3939
- wu->per_job_type[i].jobs_started = 0;
3940
- wu->per_job_type[i].busy_time = 0;
3941
-
3942
- wu->per_job_type[i].min_value = NAN;
3943
- wu->per_job_type[i].max_value = NAN;
3944
- wu->per_job_type[i].sum_value = NAN;
3945
- wu->per_job_type[i].count_value = 0;
3946
- }
3947
-
3948
- struct worker_thread *wt;
3949
- for(wt = wu->threads; wt ; wt = wt->next) {
3950
- wt->enabled = false;
3951
- wt->cpu_enabled = false;
3952
- }
3953
-}
3954
-
3955
-#define TASK_STAT_PREFIX "/proc/self/task/"
3956
-#define TASK_STAT_SUFFIX "/stat"
3957
-
3958
-static int read_thread_cpu_time_from_proc_stat(pid_t pid __maybe_unused, kernel_uint_t *utime __maybe_unused, kernel_uint_t *stime __maybe_unused) {
3959
-#ifdef __linux__
3960
- static char filename[sizeof(TASK_STAT_PREFIX) + sizeof(TASK_STAT_SUFFIX) + 20] = TASK_STAT_PREFIX;
3961
- static size_t start_pos = sizeof(TASK_STAT_PREFIX) - 1;
3962
- static procfile *ff = NULL;
3963
-
3964
- // construct the filename
3965
- size_t end_pos = snprintfz(&filename[start_pos], 20, "%d", pid);
3966
- strcpy(&filename[start_pos + end_pos], TASK_STAT_SUFFIX);
3967
-
3968
- // (re)open the procfile to the new filename
3969
- bool set_quotes = (ff == NULL) ? true : false;
3970
- ff = procfile_reopen(ff, filename, NULL, PROCFILE_FLAG_ERROR_ON_ERROR_LOG);
3971
- if(unlikely(!ff)) return -1;
3972
-
3973
- if(set_quotes)
3974
- procfile_set_open_close(ff, "(", ")");
3975
-
3976
- // read the entire file and split it to lines and words
3977
- ff = procfile_readall(ff);
3978
- if(unlikely(!ff)) return -1;
3979
-
3980
- // parse the numbers we are interested
3981
- *utime = str2kernel_uint_t(procfile_lineword(ff, 0, 13));
3982
- *stime = str2kernel_uint_t(procfile_lineword(ff, 0, 14));
3983
-
3984
- // leave the file open for the next iteration
3985
-
3986
- return 0;
3987
-#else
3988
- // TODO: add here cpu time detection per thread, for FreeBSD and MacOS
3989
- *utime = 0;
3990
- *stime = 0;
3991
- return 1;
3992
-#endif
3993
-}
3994
-
3995
-static Pvoid_t workers_by_pid_JudyL_array = NULL;
3996
-
3997
-static void workers_threads_cleanup(struct worker_utilization *wu) {
3998
- struct worker_thread *t = wu->threads;
3999
- while(t) {
4000
- struct worker_thread *next = t->next;
4001
-
4002
- if(!t->enabled) {
4003
- JudyLDel(&workers_by_pid_JudyL_array, t->pid, PJE0);
4004
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(wu->threads, t, prev, next);
4005
- freez(t);
4006
- }
4007
- t = next;
4008
- }
4009
- }
4010
-
4011
-static struct worker_thread *worker_thread_find(struct worker_utilization *wu __maybe_unused, pid_t pid) {
4012
- struct worker_thread *wt = NULL;
4013
-
4014
- Pvoid_t *PValue = JudyLGet(workers_by_pid_JudyL_array, pid, PJE0);
4015
- if(PValue)
4016
- wt = *PValue;
4017
-
4018
- return wt;
4019
-}
4020
-
4021
-static struct worker_thread *worker_thread_create(struct worker_utilization *wu, pid_t pid) {
4022
- struct worker_thread *wt;
4023
-
4024
- wt = (struct worker_thread *)callocz(1, sizeof(struct worker_thread));
4025
- wt->pid = pid;
4026
-
4027
- Pvoid_t *PValue = JudyLIns(&workers_by_pid_JudyL_array, pid, PJE0);
4028
- *PValue = wt;
4029
-
4030
- // link it
4031
- DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(wu->threads, wt, prev, next);
4032
-
4033
- return wt;
4034
-}
4035
-
4036
-static struct worker_thread *worker_thread_find_or_create(struct worker_utilization *wu, pid_t pid) {
4037
- struct worker_thread *wt;
4038
- wt = worker_thread_find(wu, pid);
4039
- if(!wt) wt = worker_thread_create(wu, pid);
4040
-
4041
- return wt;
4042
-}
4043
-
4044
-static void worker_utilization_charts_callback(void *ptr
4045
- , pid_t pid __maybe_unused
4046
- , const char *thread_tag __maybe_unused
4047
- , size_t max_job_id __maybe_unused
4048
- , size_t utilization_usec __maybe_unused
4049
- , size_t duration_usec __maybe_unused
4050
- , size_t jobs_started __maybe_unused
4051
- , size_t is_running __maybe_unused
4052
- , STRING **job_types_names __maybe_unused
4053
- , STRING **job_types_units __maybe_unused
4054
- , WORKER_METRIC_TYPE *job_types_metric_types __maybe_unused
4055
- , size_t *job_types_jobs_started __maybe_unused
4056
- , usec_t *job_types_busy_time __maybe_unused
4057
- , NETDATA_DOUBLE *job_types_custom_metrics __maybe_unused
4058
- ) {
4059
- struct worker_utilization *wu = (struct worker_utilization *)ptr;
4060
-
4061
- // find the worker_thread in the list
4062
- struct worker_thread *wt = worker_thread_find_or_create(wu, pid);
4063
-
4064
- if(utilization_usec > duration_usec)
4065
- utilization_usec = duration_usec;
4066
-
4067
- wt->enabled = true;
4068
- wt->busy_time = utilization_usec;
4069
- wt->jobs_started = jobs_started;
4070
-
4071
- wt->utime_old = wt->utime;
4072
- wt->stime_old = wt->stime;
4073
- wt->collected_time_old = wt->collected_time;
4074
-
4075
- if(max_job_id > wu->workers_max_job_id)
4076
- wu->workers_max_job_id = max_job_id;
4077
-
4078
- wu->workers_total_busy_time += utilization_usec;
4079
- wu->workers_total_duration += duration_usec;
4080
- wu->workers_total_jobs_started += jobs_started;
4081
- wu->workers_busy += is_running;
4082
- wu->workers_registered++;
4083
-
4084
- double util = (double)utilization_usec * 100.0 / (double)duration_usec;
4085
- if(util > wu->workers_max_busy_time)
4086
- wu->workers_max_busy_time = util;
4087
-
4088
- if(util < wu->workers_min_busy_time)
4089
- wu->workers_min_busy_time = util;
4090
-
4091
- // accumulate per job type statistics
4092
- size_t i;
4093
- for(i = 0; i <= max_job_id ;i++) {
4094
- if(!wu->per_job_type[i].name && job_types_names[i])
4095
- wu->per_job_type[i].name = string_dup(job_types_names[i]);
4096
-
4097
- if(!wu->per_job_type[i].units && job_types_units[i])
4098
- wu->per_job_type[i].units = string_dup(job_types_units[i]);
4099
-
4100
- wu->per_job_type[i].type = job_types_metric_types[i];
4101
-
4102
- wu->per_job_type[i].jobs_started += job_types_jobs_started[i];
4103
- wu->per_job_type[i].busy_time += job_types_busy_time[i];
4104
-
4105
- NETDATA_DOUBLE value = job_types_custom_metrics[i];
4106
- if(netdata_double_isnumber(value)) {
4107
- if(!wu->per_job_type[i].count_value) {
4108
- wu->per_job_type[i].count_value = 1;
4109
- wu->per_job_type[i].min_value = value;
4110
- wu->per_job_type[i].max_value = value;
4111
- wu->per_job_type[i].sum_value = value;
4112
- }
4113
- else {
4114
- wu->per_job_type[i].count_value++;
4115
- wu->per_job_type[i].sum_value += value;
4116
- if(value < wu->per_job_type[i].min_value) wu->per_job_type[i].min_value = value;
4117
- if(value > wu->per_job_type[i].max_value) wu->per_job_type[i].max_value = value;
4118
- }
4119
- }
4120
- }
4121
-
4122
- // find its CPU utilization
4123
- if((!read_thread_cpu_time_from_proc_stat(pid, &wt->utime, &wt->stime))) {
4124
- wt->collected_time = now_realtime_usec();
4125
- usec_t delta = wt->collected_time - wt->collected_time_old;
4126
-
4127
- double utime = (double)(wt->utime - wt->utime_old) / (double)system_hz * 100.0 * (double)USEC_PER_SEC / (double)delta;
4128
- double stime = (double)(wt->stime - wt->stime_old) / (double)system_hz * 100.0 * (double)USEC_PER_SEC / (double)delta;
4129
- double cpu = utime + stime;
4130
- wt->cpu = cpu;
4131
- wt->cpu_enabled = true;
4132
-
4133
- wu->workers_cpu_total += cpu;
4134
- if(cpu < wu->workers_cpu_min) wu->workers_cpu_min = cpu;
4135
- if(cpu > wu->workers_cpu_max) wu->workers_cpu_max = cpu;
4136
- }
4137
- wu->workers_cpu_registered += (wt->cpu_enabled) ? 1 : 0;
4138
-}
4139
-
4140
-static void worker_utilization_charts(void) {
4141
- static size_t iterations = 0;
4142
- iterations++;
4143
-
4144
- for(int i = 0; all_workers_utilization[i].name ;i++) {
4145
- workers_utilization_reset_statistics(&all_workers_utilization[i]);
4146
-
4147
- workers_foreach(all_workers_utilization[i].name, worker_utilization_charts_callback, &all_workers_utilization[i]);
4148
-
4149
- // skip the first iteration, so that we don't accumulate startup utilization to our charts
4150
- if(likely(iterations > 1))
4151
- workers_utilization_update_chart(&all_workers_utilization[i]);
4152
-
4153
- workers_threads_cleanup(&all_workers_utilization[i]);
4154
- }
4155
-
4156
- workers_total_cpu_utilization_chart();
4157
-}
4158
-
4159
-static void worker_utilization_finish(void) {
4160
- int i, j;
4161
- for(i = 0; all_workers_utilization[i].name ;i++) {
4162
- struct worker_utilization *wu = &all_workers_utilization[i];
4163
-
4164
- if(wu->name_lowercase) {
4165
- freez(wu->name_lowercase);
4166
- wu->name_lowercase = NULL;
4167
- }
4168
-
4169
- for(j = 0; j < WORKER_UTILIZATION_MAX_JOB_TYPES ;j++) {
4170
- string_freez(wu->per_job_type[j].name);
4171
- wu->per_job_type[j].name = NULL;
4172
-
4173
- string_freez(wu->per_job_type[j].units);
4174
- wu->per_job_type[j].units = NULL;
4175
- }
4176
-
4177
- // mark all threads as not enabled
4178
- struct worker_thread *t;
4179
- for(t = wu->threads; t ; t = t->next)
4180
- t->enabled = false;
4181
-
4182
- // let the cleanup job free them
4183
- workers_threads_cleanup(wu);
4184
- }
4185
-}
4186
-
4187
-// ---------------------------------------------------------------------------------------------------------------------
4188
-// global statistics thread
4189
-
4190
-
4191
-static void global_statistics_register_workers(void) {
4192
- worker_register("STATS");
4193
- worker_register_job_name(WORKER_JOB_GLOBAL, "global");
4194
- worker_register_job_name(WORKER_JOB_GLOBAL_EXT, "global_ext");
4195
- worker_register_job_name(WORKER_JOB_REGISTRY, "registry");
4196
- worker_register_job_name(WORKER_JOB_DBENGINE, "dbengine");
4197
- worker_register_job_name(WORKER_JOB_STRINGS, "strings");
4198
- worker_register_job_name(WORKER_JOB_DICTIONARIES, "dictionaries");
4199
- worker_register_job_name(WORKER_JOB_MALLOC_TRACE, "malloc_trace");
4200
- worker_register_job_name(WORKER_JOB_WORKERS, "workers");
4201
- worker_register_job_name(WORKER_JOB_SQLITE3, "sqlite3");
4202
-}
4203
-
4204
-static void global_statistics_cleanup(void *pptr)
4205
-{
4206
- struct netdata_static_thread *static_thread = CLEANUP_FUNCTION_GET_PTR(pptr);
4207
- if(!static_thread) return;
4208
-
4209
- static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
4210
-
4211
- worker_unregister();
4212
- netdata_log_info("cleaning up...");
4213
-
4214
- static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
4215
-}
4216
-
4217
-void *global_statistics_main(void *ptr)
4218
-{
4219
- CLEANUP_FUNCTION_REGISTER(global_statistics_cleanup) cleanup_ptr = ptr;
4220
-
4221
- global_statistics_register_workers();
4222
-
4223
- int update_every =
4224
- (int)config_get_duration_seconds(CONFIG_SECTION_GLOBAL_STATISTICS, "update every", localhost->rrd_update_every);
4225
- if (update_every < localhost->rrd_update_every) {
4226
- update_every = localhost->rrd_update_every;
4227
- config_set_duration_seconds(CONFIG_SECTION_GLOBAL_STATISTICS, "update every", update_every);
4228
- }
4229
-
4230
- usec_t step = update_every * USEC_PER_SEC;
4231
- heartbeat_t hb;
4232
- heartbeat_init(&hb, USEC_PER_SEC);
4233
- usec_t real_step = USEC_PER_SEC;
4234
-
4235
- // keep the randomness at zero
4236
- // to make sure we are not close to any other thread
4237
- hb.randomness = 0;
4238
-
4239
- while (service_running(SERVICE_COLLECTORS)) {
4240
- worker_is_idle();
4241
- heartbeat_next(&hb);
4242
- if (real_step < step) {
4243
- real_step += USEC_PER_SEC;
4244
- continue;
4245
- }
4246
- real_step = USEC_PER_SEC;
4247
-
4248
- worker_is_busy(WORKER_JOB_GLOBAL);
4249
- global_statistics_charts();
4250
- }
4251
-
4252
- return NULL;
4253
-}
4254
-
4255
-
4256
-// ---------------------------------------------------------------------------------------------------------------------
4257
-// global statistics extended thread
4258
-
4259
-static void global_statistics_extended_cleanup(void *pptr)
4260
-{
4261
- struct netdata_static_thread *static_thread = CLEANUP_FUNCTION_GET_PTR(pptr);
4262
- if (!static_thread)
4263
- return;
4264
-
4265
- static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
4266
-
4267
- netdata_log_info("cleaning up...");
4268
-
4269
- worker_unregister();
4270
- worker_utilization_finish();
4271
-
4272
- static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
4273
-}
4274
-
4275
-void *global_statistics_extended_main(void *ptr)
4276
-{
4277
- CLEANUP_FUNCTION_REGISTER(global_statistics_extended_cleanup) cleanup_ptr = ptr;
4278
-
4279
- global_statistics_register_workers();
4280
-
4281
- int update_every =
4282
- (int)config_get_duration_seconds(CONFIG_SECTION_GLOBAL_STATISTICS, "update every", localhost->rrd_update_every);
4283
- if (update_every < localhost->rrd_update_every) {
4284
- update_every = localhost->rrd_update_every;
4285
- config_set_duration_seconds(CONFIG_SECTION_GLOBAL_STATISTICS, "update every", update_every);
4286
- }
4287
-
4288
- usec_t step = update_every * USEC_PER_SEC;
4289
- heartbeat_t hb;
4290
- heartbeat_init(&hb, USEC_PER_SEC);
4291
- usec_t real_step = USEC_PER_SEC;
4292
-
4293
- while (service_running(SERVICE_COLLECTORS)) {
4294
- worker_is_idle();
4295
- heartbeat_next(&hb);
4296
- if (real_step < step) {
4297
- real_step += USEC_PER_SEC;
4298
- continue;
4299
- }
4300
- real_step = USEC_PER_SEC;
4301
-
4302
- worker_is_busy(WORKER_JOB_HEARTBEAT);
4303
- update_heartbeat_charts();
4304
-
4305
- worker_is_busy(WORKER_JOB_GLOBAL_EXT);
4306
- global_statistics_extended_charts();
4307
-
4308
-#ifdef ENABLE_DBENGINE
4309
- if(dbengine_enabled) {
4310
- worker_is_busy(WORKER_JOB_DBENGINE);
4311
- dbengine2_statistics_charts();
4312
- }
4313
-#endif
4314
-
4315
- worker_is_busy(WORKER_JOB_REGISTRY);
4316
- registry_statistics();
4317
-
4318
- worker_is_busy(WORKER_JOB_STRINGS);
4319
- update_strings_charts();
4320
-
4321
-#ifdef DICT_WITH_STATS
4322
- worker_is_busy(WORKER_JOB_DICTIONARIES);
4323
- dictionary_statistics();
4324
-#endif
4325
-
4326
-#ifdef NETDATA_TRACE_ALLOCATIONS
4327
- worker_is_busy(WORKER_JOB_MALLOC_TRACE);
4328
- malloc_trace_statistics();
4329
-#endif
4330
-
4331
- worker_is_busy(WORKER_JOB_WORKERS);
4332
- worker_utilization_charts();
4333
-
4334
- worker_is_busy(WORKER_JOB_SQLITE3);
4335
- sqlite3_statistics_charts();
4336
- }
4337
-
4338
- return NULL;
4339
-}
src/daemon/global_statistics.h
deleted
-64
@@ -1,64 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#ifndef NETDATA_GLOBAL_STATISTICS_H
4
-#define NETDATA_GLOBAL_STATISTICS_H 1
5
-
6
-#include "database/rrd.h"
7
-
8
-extern struct netdata_buffers_statistics {
9
- size_t rrdhost_allocations_size;
10
- size_t rrdhost_senders;
11
- size_t rrdhost_receivers;
12
- size_t query_targets_size;
13
- size_t rrdset_done_rda_size;
14
- size_t buffers_aclk;
15
- size_t buffers_api;
16
- size_t buffers_functions;
17
- size_t buffers_sqlite;
18
- size_t buffers_exporters;
19
- size_t buffers_health;
20
- size_t buffers_streaming;
21
- size_t cbuffers_streaming;
22
- size_t buffers_web;
23
-} netdata_buffers_statistics;
24
-
25
-extern struct dictionary_stats dictionary_stats_category_collectors;
26
-extern struct dictionary_stats dictionary_stats_category_rrdhost;
27
-extern struct dictionary_stats dictionary_stats_category_rrdset_rrddim;
28
-extern struct dictionary_stats dictionary_stats_category_rrdcontext;
29
-extern struct dictionary_stats dictionary_stats_category_rrdlabels;
30
-extern struct dictionary_stats dictionary_stats_category_rrdhealth;
31
-extern struct dictionary_stats dictionary_stats_category_functions;
32
-extern struct dictionary_stats dictionary_stats_category_replication;
33
-
34
-extern size_t rrddim_db_memory_size;
35
-
36
-// ----------------------------------------------------------------------------
37
-// global statistics
38
-
39
-void global_statistics_ml_query_completed(size_t points_read);
40
-void global_statistics_ml_models_consulted(size_t models_consulted);
41
-void global_statistics_exporters_query_completed(size_t points_read);
42
-void global_statistics_backfill_query_completed(size_t points_read);
43
-void global_statistics_rrdr_query_completed(size_t queries, uint64_t db_points_read, uint64_t result_points_generated, QUERY_SOURCE query_source);
44
-void global_statistics_sqlite3_query_completed(bool success, bool busy, bool locked);
45
-void global_statistics_sqlite3_row_completed(void);
46
-void global_statistics_rrdset_done_chart_collection_completed(size_t *points_read_per_tier_array);
47
-
48
-void global_statistics_gorilla_buffer_add_hot();
49
-
50
-void global_statistics_tier0_disk_compressed_bytes(uint32_t size);
51
-void global_statistics_tier0_disk_uncompressed_bytes(uint32_t size);
52
-
53
-void global_statistics_web_request_completed(uint64_t dt,
54
- uint64_t bytes_received,
55
- uint64_t bytes_sent,
56
- uint64_t content_size,
57
- uint64_t compressed_content_size);
58
-
59
-uint64_t global_statistics_web_client_connected(void);
60
-void global_statistics_web_client_disconnected(void);
61
-
62
-extern bool global_statistics_enabled;
63
-
64
-#endif /* NETDATA_GLOBAL_STATISTICS_H */
src/daemon/h2o-common.c
-39
@@ -19,42 +19,3 @@ const char *netdata_configured_abbrev_timezone = NULL;
19
int32_t netdata_configured_utc_offset = 0;
20
21
bool netdata_ready = false;
22
-
23
-long get_netdata_cpus(void) {
24
- static long processors = 0;
25
-
26
- if(processors)
27
- return processors;
28
-
29
- long cores_proc_stat = os_get_system_cpus_cached(false, true);
30
- long cores_cpuset_v1 = (long)os_read_cpuset_cpus("/sys/fs/cgroup/cpuset/cpuset.cpus", cores_proc_stat);
31
- long cores_cpuset_v2 = (long)os_read_cpuset_cpus("/sys/fs/cgroup/cpuset.cpus", cores_proc_stat);
32
-
33
- if(cores_cpuset_v2)
34
- processors = cores_cpuset_v2;
35
- else if(cores_cpuset_v1)
36
- processors = cores_cpuset_v1;
37
- else
38
- processors = cores_proc_stat;
39
-
40
- long cores_user_configured = config_get_number(CONFIG_SECTION_GLOBAL, "cpu cores", processors);
41
-
42
- errno_clear();
43
- internal_error(true,
44
- "System CPUs: %ld, ("
45
- "system: %ld, cgroups cpuset v1: %ld, cgroups cpuset v2: %ld, netdata.conf: %ld"
46
- ")"
47
- , processors
48
- , cores_proc_stat
49
- , cores_cpuset_v1
50
- , cores_cpuset_v2
51
- , cores_user_configured
52
- );
53
-
54
- processors = cores_user_configured;
55
-
56
- if(processors < 1)
57
- processors = 1;
58
-
59
- return processors;
60
-}
src/daemon/libuv_workers.c
+2
@@ -44,6 +44,8 @@ void register_libuv_worker_jobs() {
44
45
// other dbengine events
46
worker_register_job_name(UV_EVENT_DBENGINE_EVICT_MAIN_CACHE, "evict main");
47
+ worker_register_job_name(UV_EVENT_DBENGINE_EVICT_OPEN_CACHE, "evict open");
48
+ worker_register_job_name(UV_EVENT_DBENGINE_EVICT_EXTENT_CACHE, "evict extent");
49
worker_register_job_name(UV_EVENT_DBENGINE_BUFFERS_CLEANUP, "dbengine buffers cleanup");
50
worker_register_job_name(UV_EVENT_DBENGINE_QUIESCE, "dbengine quiesce");
51
worker_register_job_name(UV_EVENT_DBENGINE_SHUTDOWN, "dbengine shutdown");
src/daemon/libuv_workers.h
+2
@@ -36,6 +36,8 @@ enum event_loop_job {
36
37
// other dbengine events
38
UV_EVENT_DBENGINE_EVICT_MAIN_CACHE,
39
+ UV_EVENT_DBENGINE_EVICT_OPEN_CACHE,
40
+ UV_EVENT_DBENGINE_EVICT_EXTENT_CACHE,
41
UV_EVENT_DBENGINE_BUFFERS_CLEANUP,
42
UV_EVENT_DBENGINE_QUIESCE,
43
UV_EVENT_DBENGINE_SHUTDOWN,
src/daemon/main.c
+27
-724
@@ -361,6 +361,7 @@ void netdata_cleanup_and_exit(int ret, const char *action, const char *action_re
361
service_wait_exit(SERVICE_EXPORTERS | SERVICE_HEALTH | SERVICE_WEB_SERVER | SERVICE_HTTPD, 3 * USEC_PER_SEC);
362
watcher_step_complete(WATCHER_STEP_ID_STOP_EXPORTERS_HEALTH_AND_WEB_SERVERS_THREADS);
363
364
+ stream_threads_cancel();
365
service_wait_exit(SERVICE_COLLECTORS | SERVICE_STREAMING, 3 * USEC_PER_SEC);
366
watcher_step_complete(WATCHER_STEP_ID_STOP_COLLECTORS_AND_STREAMING_THREADS);
367
@@ -501,117 +502,6 @@ void netdata_cleanup_and_exit(int ret, const char *action, const char *action_re
502
exit(ret);
503
}
504
504
-void web_server_threading_selection(void) {
505
- web_server_mode = web_server_mode_id(config_get(CONFIG_SECTION_WEB, "mode", web_server_mode_name(web_server_mode)));
506
-
507
- int static_threaded = (web_server_mode == WEB_SERVER_MODE_STATIC_THREADED);
508
-
509
- int i;
510
- for (i = 0; static_threads[i].name; i++) {
511
- if (static_threads[i].start_routine == socket_listen_main_static_threaded)
512
- static_threads[i].enabled = static_threaded;
513
- }
514
-}
515
-
516
-int make_dns_decision(const char *section_name, const char *config_name, const char *default_value, SIMPLE_PATTERN *p)
517
-{
518
- const char *value = config_get(section_name,config_name,default_value);
519
- if(!strcmp("yes",value))
520
- return 1;
521
- if(!strcmp("no",value))
522
- return 0;
523
- if(strcmp("heuristic",value) != 0)
524
- netdata_log_error("Invalid configuration option '%s' for '%s'/'%s'. Valid options are 'yes', 'no' and 'heuristic'. Proceeding with 'heuristic'",
525
- value, section_name, config_name);
526
-
527
- return simple_pattern_is_potential_name(p);
528
-}
529
-
530
-void web_server_config_options(void)
531
-{
532
- web_client_timeout =
533
- (int)config_get_duration_seconds(CONFIG_SECTION_WEB, "disconnect idle clients after", web_client_timeout);
534
-
535
- web_client_first_request_timeout =
536
- (int)config_get_duration_seconds(CONFIG_SECTION_WEB, "timeout for first request", web_client_first_request_timeout);
537
-
538
- web_client_streaming_rate_t =
539
- config_get_duration_seconds(CONFIG_SECTION_WEB, "accept a streaming request every", web_client_streaming_rate_t);
540
-
541
- respect_web_browser_do_not_track_policy =
542
- config_get_boolean(CONFIG_SECTION_WEB, "respect do not track policy", respect_web_browser_do_not_track_policy);
543
- web_x_frame_options = config_get(CONFIG_SECTION_WEB, "x-frame-options response header", "");
544
- if(!*web_x_frame_options)
545
- web_x_frame_options = NULL;
546
-
547
- web_allow_connections_from =
548
- simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow connections from", "localhost *"),
549
- NULL, SIMPLE_PATTERN_EXACT, true);
550
- web_allow_connections_dns =
551
- make_dns_decision(CONFIG_SECTION_WEB, "allow connections by dns", "heuristic", web_allow_connections_from);
552
- web_allow_dashboard_from =
553
- simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow dashboard from", "localhost *"),
554
- NULL, SIMPLE_PATTERN_EXACT, true);
555
- web_allow_dashboard_dns =
556
- make_dns_decision(CONFIG_SECTION_WEB, "allow dashboard by dns", "heuristic", web_allow_dashboard_from);
557
- web_allow_badges_from =
558
- simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow badges from", "*"), NULL, SIMPLE_PATTERN_EXACT,
559
- true);
560
- web_allow_badges_dns =
561
- make_dns_decision(CONFIG_SECTION_WEB, "allow badges by dns", "heuristic", web_allow_badges_from);
562
- web_allow_registry_from =
563
- simple_pattern_create(config_get(CONFIG_SECTION_REGISTRY, "allow from", "*"), NULL, SIMPLE_PATTERN_EXACT,
564
- true);
565
- web_allow_registry_dns = make_dns_decision(CONFIG_SECTION_REGISTRY, "allow by dns", "heuristic",
566
- web_allow_registry_from);
567
- web_allow_streaming_from = simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow streaming from", "*"),
568
- NULL, SIMPLE_PATTERN_EXACT, true);
569
- web_allow_streaming_dns = make_dns_decision(CONFIG_SECTION_WEB, "allow streaming by dns", "heuristic",
570
- web_allow_streaming_from);
571
- // Note the default is not heuristic, the wildcards could match DNS but the intent is ip-addresses.
572
- web_allow_netdataconf_from = simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow netdata.conf from",
573
- "localhost fd* 10.* 192.168.* 172.16.* 172.17.* 172.18.*"
574
- " 172.19.* 172.20.* 172.21.* 172.22.* 172.23.* 172.24.*"
575
- " 172.25.* 172.26.* 172.27.* 172.28.* 172.29.* 172.30.*"
576
- " 172.31.* UNKNOWN"), NULL, SIMPLE_PATTERN_EXACT,
577
- true);
578
- web_allow_netdataconf_dns =
579
- make_dns_decision(CONFIG_SECTION_WEB, "allow netdata.conf by dns", "no", web_allow_netdataconf_from);
580
- web_allow_mgmt_from =
581
- simple_pattern_create(config_get(CONFIG_SECTION_WEB, "allow management from", "localhost"),
582
- NULL, SIMPLE_PATTERN_EXACT, true);
583
- web_allow_mgmt_dns =
584
- make_dns_decision(CONFIG_SECTION_WEB, "allow management by dns","heuristic",web_allow_mgmt_from);
585
-
586
- web_enable_gzip = config_get_boolean(CONFIG_SECTION_WEB, "enable gzip compression", web_enable_gzip);
587
-
588
- const char *s = config_get(CONFIG_SECTION_WEB, "gzip compression strategy", "default");
589
- if(!strcmp(s, "default"))
590
- web_gzip_strategy = Z_DEFAULT_STRATEGY;
591
- else if(!strcmp(s, "filtered"))
592
- web_gzip_strategy = Z_FILTERED;
593
- else if(!strcmp(s, "huffman only"))
594
- web_gzip_strategy = Z_HUFFMAN_ONLY;
595
- else if(!strcmp(s, "rle"))
596
- web_gzip_strategy = Z_RLE;
597
- else if(!strcmp(s, "fixed"))
598
- web_gzip_strategy = Z_FIXED;
599
- else {
600
- netdata_log_error("Invalid compression strategy '%s'. Valid strategies are 'default', 'filtered', 'huffman only', 'rle' and 'fixed'. Proceeding with 'default'.", s);
601
- web_gzip_strategy = Z_DEFAULT_STRATEGY;
602
- }
603
-
604
- web_gzip_level = (int)config_get_number(CONFIG_SECTION_WEB, "gzip compression level", 3);
605
- if(web_gzip_level < 1) {
606
- netdata_log_error("Invalid compression level %d. Valid levels are 1 (fastest) to 9 (best ratio). Proceeding with level 1 (fastest compression).", web_gzip_level);
607
- web_gzip_level = 1;
608
- }
609
- else if(web_gzip_level > 9) {
610
- netdata_log_error("Invalid compression level %d. Valid levels are 1 (fastest) to 9 (best ratio). Proceeding with level 9 (best compression).", web_gzip_level);
611
- web_gzip_level = 9;
612
- }
613
-}
614
-
505
static void set_nofile_limit(struct rlimit *rl) {
506
// get the num files allowed
507
if(getrlimit(RLIMIT_NOFILE, rl) != 0) {
@@ -813,597 +703,6 @@ int help(int exitcode) {
703
return exitcode;
704
}
705
816
-static void security_init(){
817
- char filename[FILENAME_MAX + 1];
818
- snprintfz(filename, FILENAME_MAX, "%s/ssl/key.pem",netdata_configured_user_config_dir);
819
- netdata_ssl_security_key = config_get(CONFIG_SECTION_WEB, "ssl key", filename);
820
-
821
- snprintfz(filename, FILENAME_MAX, "%s/ssl/cert.pem",netdata_configured_user_config_dir);
822
- netdata_ssl_security_cert = config_get(CONFIG_SECTION_WEB, "ssl certificate", filename);
823
-
824
- tls_version = config_get(CONFIG_SECTION_WEB, "tls version", "1.3");
825
- tls_ciphers = config_get(CONFIG_SECTION_WEB, "tls ciphers", "none");
826
-
827
- netdata_ssl_initialize_openssl();
828
-}
829
-
830
-static void log_init(void) {
831
- nd_log_set_facility(config_get(CONFIG_SECTION_LOGS, "facility", "daemon"));
832
-
833
- time_t period = ND_LOG_DEFAULT_THROTTLE_PERIOD;
834
- size_t logs = ND_LOG_DEFAULT_THROTTLE_LOGS;
835
- period = config_get_duration_seconds(CONFIG_SECTION_LOGS, "logs flood protection period", period);
836
- logs = (unsigned long)config_get_number(CONFIG_SECTION_LOGS, "logs to trigger flood protection", (long long int)logs);
837
- nd_log_set_flood_protection(logs, period);
838
-
839
- const char *netdata_log_level = getenv("NETDATA_LOG_LEVEL");
840
- netdata_log_level = netdata_log_level ? nd_log_id2priority(nd_log_priority2id(netdata_log_level)) : NDLP_INFO_STR;
841
-
842
- nd_log_set_priority_level(config_get(CONFIG_SECTION_LOGS, "level", netdata_log_level));
843
-
844
- char filename[FILENAME_MAX + 1];
845
- char* os_default_method = NULL;
846
-#if defined(OS_LINUX)
847
- os_default_method = is_stderr_connected_to_journal() /* || nd_log_journal_socket_available() */ ? "journal" : NULL;
848
-#elif defined(OS_WINDOWS)
849
-#if defined(HAVE_ETW)
850
- os_default_method = "etw";
851
-#elif defined(HAVE_WEL)
852
- os_default_method = "wel";
853
-#endif
854
-#endif
855
-
856
-#if defined(OS_WINDOWS)
857
- // on windows, debug log goes to windows events
858
- snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
859
-#else
860
- snprintfz(filename, FILENAME_MAX, "%s/debug.log", netdata_configured_log_dir);
861
-#endif
862
-
863
- nd_log_set_user_settings(NDLS_DEBUG, config_get(CONFIG_SECTION_LOGS, "debug", filename));
864
-
865
- if(os_default_method)
866
- snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
867
- else
868
- snprintfz(filename, FILENAME_MAX, "%s/daemon.log", netdata_configured_log_dir);
869
- nd_log_set_user_settings(NDLS_DAEMON, config_get(CONFIG_SECTION_LOGS, "daemon", filename));
870
-
871
- if(os_default_method)
872
- snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
873
- else
874
- snprintfz(filename, FILENAME_MAX, "%s/collector.log", netdata_configured_log_dir);
875
- nd_log_set_user_settings(NDLS_COLLECTORS, config_get(CONFIG_SECTION_LOGS, "collector", filename));
876
-
877
-#if defined(OS_WINDOWS)
878
- // on windows, access log goes to windows events
879
- snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
880
-#else
881
- snprintfz(filename, FILENAME_MAX, "%s/access.log", netdata_configured_log_dir);
882
-#endif
883
- nd_log_set_user_settings(NDLS_ACCESS, config_get(CONFIG_SECTION_LOGS, "access", filename));
884
-
885
- if(os_default_method)
886
- snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
887
- else
888
- snprintfz(filename, FILENAME_MAX, "%s/health.log", netdata_configured_log_dir);
889
- nd_log_set_user_settings(NDLS_HEALTH, config_get(CONFIG_SECTION_LOGS, "health", filename));
890
-
891
- aclklog_enabled = config_get_boolean(CONFIG_SECTION_CLOUD, "conversation log", CONFIG_BOOLEAN_NO);
892
- if (aclklog_enabled) {
893
-#if defined(OS_WINDOWS)
894
- // on windows, aclk log goes to windows events
895
- snprintfz(filename, FILENAME_MAX, "%s", os_default_method);
896
-#else
897
- snprintfz(filename, FILENAME_MAX, "%s/aclk.log", netdata_configured_log_dir);
898
-#endif
899
- nd_log_set_user_settings(NDLS_ACLK, config_get(CONFIG_SECTION_CLOUD, "conversation log file", filename));
900
- }
901
-
902
- aclk_config_get_query_scope();
903
-}
904
-
905
-static const char *get_varlib_subdir_from_config(const char *prefix, const char *dir) {
906
- char filename[FILENAME_MAX + 1];
907
- snprintfz(filename, FILENAME_MAX, "%s/%s", prefix, dir);
908
- return config_get(CONFIG_SECTION_DIRECTORIES, dir, filename);
909
-}
910
-
911
-static void backwards_compatible_config() {
912
- // move [global] options to the [web] section
913
-
914
- config_move(CONFIG_SECTION_GLOBAL, "http port listen backlog",
915
- CONFIG_SECTION_WEB, "listen backlog");
916
-
917
- config_move(CONFIG_SECTION_GLOBAL, "bind socket to IP",
918
- CONFIG_SECTION_WEB, "bind to");
919
-
920
- config_move(CONFIG_SECTION_GLOBAL, "bind to",
921
- CONFIG_SECTION_WEB, "bind to");
922
-
923
- config_move(CONFIG_SECTION_GLOBAL, "port",
924
- CONFIG_SECTION_WEB, "default port");
925
-
926
- config_move(CONFIG_SECTION_GLOBAL, "default port",
927
- CONFIG_SECTION_WEB, "default port");
928
-
929
- config_move(CONFIG_SECTION_GLOBAL, "disconnect idle web clients after seconds",
930
- CONFIG_SECTION_WEB, "disconnect idle clients after seconds");
931
-
932
- config_move(CONFIG_SECTION_GLOBAL, "respect web browser do not track policy",
933
- CONFIG_SECTION_WEB, "respect do not track policy");
934
-
935
- config_move(CONFIG_SECTION_GLOBAL, "web x-frame-options header",
936
- CONFIG_SECTION_WEB, "x-frame-options response header");
937
-
938
- config_move(CONFIG_SECTION_GLOBAL, "enable web responses gzip compression",
939
- CONFIG_SECTION_WEB, "enable gzip compression");
940
-
941
- config_move(CONFIG_SECTION_GLOBAL, "web compression strategy",
942
- CONFIG_SECTION_WEB, "gzip compression strategy");
943
-
944
- config_move(CONFIG_SECTION_GLOBAL, "web compression level",
945
- CONFIG_SECTION_WEB, "gzip compression level");
946
-
947
- config_move(CONFIG_SECTION_GLOBAL, "config directory",
948
- CONFIG_SECTION_DIRECTORIES, "config");
949
-
950
- config_move(CONFIG_SECTION_GLOBAL, "stock config directory",
951
- CONFIG_SECTION_DIRECTORIES, "stock config");
952
-
953
- config_move(CONFIG_SECTION_GLOBAL, "log directory",
954
- CONFIG_SECTION_DIRECTORIES, "log");
955
-
956
- config_move(CONFIG_SECTION_GLOBAL, "web files directory",
957
- CONFIG_SECTION_DIRECTORIES, "web");
958
-
959
- config_move(CONFIG_SECTION_GLOBAL, "cache directory",
960
- CONFIG_SECTION_DIRECTORIES, "cache");
961
-
962
- config_move(CONFIG_SECTION_GLOBAL, "lib directory",
963
- CONFIG_SECTION_DIRECTORIES, "lib");
964
-
965
- config_move(CONFIG_SECTION_GLOBAL, "home directory",
966
- CONFIG_SECTION_DIRECTORIES, "home");
967
-
968
- config_move(CONFIG_SECTION_GLOBAL, "lock directory",
969
- CONFIG_SECTION_DIRECTORIES, "lock");
970
-
971
- config_move(CONFIG_SECTION_GLOBAL, "plugins directory",
972
- CONFIG_SECTION_DIRECTORIES, "plugins");
973
-
974
- config_move(CONFIG_SECTION_HEALTH, "health configuration directory",
975
- CONFIG_SECTION_DIRECTORIES, "health config");
976
-
977
- config_move(CONFIG_SECTION_HEALTH, "stock health configuration directory",
978
- CONFIG_SECTION_DIRECTORIES, "stock health config");
979
-
980
- config_move(CONFIG_SECTION_REGISTRY, "registry db directory",
981
- CONFIG_SECTION_DIRECTORIES, "registry");
982
-
983
- config_move(CONFIG_SECTION_GLOBAL, "debug log",
984
- CONFIG_SECTION_LOGS, "debug");
985
-
986
- config_move(CONFIG_SECTION_GLOBAL, "error log",
987
- CONFIG_SECTION_LOGS, "error");
988
-
989
- config_move(CONFIG_SECTION_GLOBAL, "access log",
990
- CONFIG_SECTION_LOGS, "access");
991
-
992
- config_move(CONFIG_SECTION_GLOBAL, "facility log",
993
- CONFIG_SECTION_LOGS, "facility");
994
-
995
- config_move(CONFIG_SECTION_GLOBAL, "errors flood protection period",
996
- CONFIG_SECTION_LOGS, "errors flood protection period");
997
-
998
- config_move(CONFIG_SECTION_GLOBAL, "errors to trigger flood protection",
999
- CONFIG_SECTION_LOGS, "errors to trigger flood protection");
1000
-
1001
- config_move(CONFIG_SECTION_GLOBAL, "debug flags",
1002
- CONFIG_SECTION_LOGS, "debug flags");
1003
-
1004
- config_move(CONFIG_SECTION_GLOBAL, "TZ environment variable",
1005
- CONFIG_SECTION_ENV_VARS, "TZ");
1006
-
1007
- config_move(CONFIG_SECTION_PLUGINS, "PATH environment variable",
1008
- CONFIG_SECTION_ENV_VARS, "PATH");
1009
-
1010
- config_move(CONFIG_SECTION_PLUGINS, "PYTHONPATH environment variable",
1011
- CONFIG_SECTION_ENV_VARS, "PYTHONPATH");
1012
-
1013
- config_move(CONFIG_SECTION_STATSD, "enabled",
1014
- CONFIG_SECTION_PLUGINS, "statsd");
1015
-
1016
- config_move(CONFIG_SECTION_GLOBAL, "memory mode",
1017
- CONFIG_SECTION_DB, "db");
1018
-
1019
- config_move(CONFIG_SECTION_DB, "mode",
1020
- CONFIG_SECTION_DB, "db");
1021
-
1022
- config_move(CONFIG_SECTION_GLOBAL, "history",
1023
- CONFIG_SECTION_DB, "retention");
1024
-
1025
- config_move(CONFIG_SECTION_GLOBAL, "update every",
1026
- CONFIG_SECTION_DB, "update every");
1027
-
1028
- config_move(CONFIG_SECTION_GLOBAL, "page cache size",
1029
- CONFIG_SECTION_DB, "dbengine page cache size");
1030
-
1031
- config_move(CONFIG_SECTION_DB, "dbengine page cache size MB",
1032
- CONFIG_SECTION_DB, "dbengine page cache size");
1033
-
1034
- config_move(CONFIG_SECTION_DB, "dbengine extent cache size MB",
1035
- CONFIG_SECTION_DB, "dbengine extent cache size");
1036
-
1037
- config_move(CONFIG_SECTION_DB, "page cache size",
1038
- CONFIG_SECTION_DB, "dbengine page cache size MB");
1039
-
1040
- config_move(CONFIG_SECTION_GLOBAL, "page cache uses malloc",
1041
- CONFIG_SECTION_DB, "dbengine page cache with malloc");
1042
-
1043
- config_move(CONFIG_SECTION_DB, "page cache with malloc",
1044
- CONFIG_SECTION_DB, "dbengine page cache with malloc");
1045
-
1046
- config_move(CONFIG_SECTION_GLOBAL, "memory deduplication (ksm)",
1047
- CONFIG_SECTION_DB, "memory deduplication (ksm)");
1048
-
1049
- config_move(CONFIG_SECTION_GLOBAL, "dbengine page fetch timeout",
1050
- CONFIG_SECTION_DB, "dbengine page fetch timeout secs");
1051
-
1052
- config_move(CONFIG_SECTION_GLOBAL, "dbengine page fetch retries",
1053
- CONFIG_SECTION_DB, "dbengine page fetch retries");
1054
-
1055
- config_move(CONFIG_SECTION_GLOBAL, "dbengine extent pages",
1056
- CONFIG_SECTION_DB, "dbengine pages per extent");
1057
-
1058
- config_move(CONFIG_SECTION_GLOBAL, "cleanup obsolete charts after seconds",
1059
- CONFIG_SECTION_DB, "cleanup obsolete charts after");
1060
-
1061
- config_move(CONFIG_SECTION_DB, "cleanup obsolete charts after secs",
1062
- CONFIG_SECTION_DB, "cleanup obsolete charts after");
1063
-
1064
- config_move(CONFIG_SECTION_GLOBAL, "gap when lost iterations above",
1065
- CONFIG_SECTION_DB, "gap when lost iterations above");
1066
-
1067
- config_move(CONFIG_SECTION_GLOBAL, "cleanup orphan hosts after seconds",
1068
- CONFIG_SECTION_DB, "cleanup orphan hosts after");
1069
-
1070
- config_move(CONFIG_SECTION_DB, "cleanup orphan hosts after secs",
1071
- CONFIG_SECTION_DB, "cleanup orphan hosts after");
1072
-
1073
- config_move(CONFIG_SECTION_DB, "cleanup ephemeral hosts after secs",
1074
- CONFIG_SECTION_DB, "cleanup ephemeral hosts after");
1075
-
1076
- config_move(CONFIG_SECTION_DB, "seconds to replicate",
1077
- CONFIG_SECTION_DB, "replication period");
1078
-
1079
- config_move(CONFIG_SECTION_DB, "seconds per replication step",
1080
- CONFIG_SECTION_DB, "replication step");
1081
-
1082
- config_move(CONFIG_SECTION_GLOBAL, "enable zero metrics",
1083
- CONFIG_SECTION_DB, "enable zero metrics");
1084
-
1085
- // ----------------------------------------------------------------------------------------------------------------
1086
-
1087
- config_move(CONFIG_SECTION_GLOBAL, "dbengine disk space",
1088
- CONFIG_SECTION_DB, "dbengine tier 0 retention size");
1089
-
1090
- config_move(CONFIG_SECTION_GLOBAL, "dbengine multihost disk space",
1091
- CONFIG_SECTION_DB, "dbengine tier 0 retention size");
1092
-
1093
- config_move(CONFIG_SECTION_DB, "dbengine disk space MB",
1094
- CONFIG_SECTION_DB, "dbengine tier 0 retention size");
1095
-
1096
- for(size_t tier = 0; tier < RRD_STORAGE_TIERS ;tier++) {
1097
- char old_config[128], new_config[128];
1098
-
1099
- snprintfz(old_config, sizeof(old_config), "dbengine tier %zu retention days", tier);
1100
- snprintfz(new_config, sizeof(new_config), "dbengine tier %zu retention time", tier);
1101
- config_move(CONFIG_SECTION_DB, old_config,
1102
- CONFIG_SECTION_DB, new_config);
1103
-
1104
- if(tier == 0)
1105
- snprintfz(old_config, sizeof(old_config), "dbengine multihost disk space MB");
1106
- else
1107
- snprintfz(old_config, sizeof(old_config), "dbengine tier %zu multihost disk space MB", tier);
1108
- snprintfz(new_config, sizeof(new_config), "dbengine tier %zu retention size", tier);
1109
- config_move(CONFIG_SECTION_DB, old_config,
1110
- CONFIG_SECTION_DB, new_config);
1111
-
1112
- snprintfz(old_config, sizeof(old_config), "dbengine tier %zu disk space MB", tier);
1113
- snprintfz(new_config, sizeof(new_config), "dbengine tier %zu retention size", tier);
1114
- config_move(CONFIG_SECTION_DB, old_config,
1115
- CONFIG_SECTION_DB, new_config);
1116
- }
1117
-
1118
- // ----------------------------------------------------------------------------------------------------------------
1119
-
1120
- config_move(CONFIG_SECTION_LOGS, "error",
1121
- CONFIG_SECTION_LOGS, "daemon");
1122
-
1123
- config_move(CONFIG_SECTION_LOGS, "severity level",
1124
- CONFIG_SECTION_LOGS, "level");
1125
-
1126
- config_move(CONFIG_SECTION_LOGS, "errors to trigger flood protection",
1127
- CONFIG_SECTION_LOGS, "logs to trigger flood protection");
1128
-
1129
- config_move(CONFIG_SECTION_LOGS, "errors flood protection period",
1130
- CONFIG_SECTION_LOGS, "logs flood protection period");
1131
-
1132
- config_move(CONFIG_SECTION_HEALTH, "is ephemeral",
1133
- CONFIG_SECTION_GLOBAL, "is ephemeral node");
1134
-
1135
- config_move(CONFIG_SECTION_HEALTH, "has unstable connection",
1136
- CONFIG_SECTION_GLOBAL, "has unstable connection");
1137
-
1138
- config_move(CONFIG_SECTION_HEALTH, "run at least every seconds",
1139
- CONFIG_SECTION_HEALTH, "run at least every");
1140
-
1141
- config_move(CONFIG_SECTION_HEALTH, "postpone alarms during hibernation for seconds",
1142
- CONFIG_SECTION_HEALTH, "postpone alarms during hibernation for");
1143
-
1144
- config_move(CONFIG_SECTION_HEALTH, "health log history",
1145
- CONFIG_SECTION_HEALTH, "health log retention");
1146
-
1147
- config_move(CONFIG_SECTION_REGISTRY, "registry expire idle persons days",
1148
- CONFIG_SECTION_REGISTRY, "registry expire idle persons");
1149
-
1150
- config_move(CONFIG_SECTION_WEB, "disconnect idle clients after seconds",
1151
- CONFIG_SECTION_WEB, "disconnect idle clients after");
1152
-
1153
- config_move(CONFIG_SECTION_WEB, "accept a streaming request every seconds",
1154
- CONFIG_SECTION_WEB, "accept a streaming request every");
1155
-
1156
- config_move(CONFIG_SECTION_STATSD, "set charts as obsolete after secs",
1157
- CONFIG_SECTION_STATSD, "set charts as obsolete after");
1158
-
1159
- config_move(CONFIG_SECTION_STATSD, "disconnect idle tcp clients after seconds",
1160
- CONFIG_SECTION_STATSD, "disconnect idle tcp clients after");
1161
-
1162
- config_move("plugin:idlejitter", "loop time in ms",
1163
- "plugin:idlejitter", "loop time");
1164
-
1165
- config_move("plugin:proc:/sys/class/infiniband", "refresh ports state every seconds",
1166
- "plugin:proc:/sys/class/infiniband", "refresh ports state every");
1167
-}
1168
-
1169
-static int get_hostname(char *buf, size_t buf_size) {
1170
- if (netdata_configured_host_prefix && *netdata_configured_host_prefix) {
1171
- char filename[FILENAME_MAX + 1];
1172
- snprintfz(filename, FILENAME_MAX, "%s/etc/hostname", netdata_configured_host_prefix);
1173
-
1174
- if (!read_txt_file(filename, buf, buf_size)) {
1175
- trim(buf);
1176
- return 0;
1177
- }
1178
- }
1179
-
1180
- return gethostname(buf, buf_size);
1181
-}
1182
-
1183
-static void get_netdata_configured_variables()
1184
-{
1185
-#ifdef ENABLE_DBENGINE
1186
- legacy_multihost_db_space = config_exists(CONFIG_SECTION_DB, "dbengine multihost disk space MB");
1187
- if (!legacy_multihost_db_space)
1188
- legacy_multihost_db_space = config_exists(CONFIG_SECTION_GLOBAL, "dbengine multihost disk space");
1189
- if (!legacy_multihost_db_space)
1190
- legacy_multihost_db_space = config_exists(CONFIG_SECTION_GLOBAL, "dbengine disk space");
1191
-#endif
1192
-
1193
- backwards_compatible_config();
1194
-
1195
- // ------------------------------------------------------------------------
1196
- // get the hostname
1197
-
1198
- netdata_configured_host_prefix = config_get(CONFIG_SECTION_GLOBAL, "host access prefix", "");
1199
- (void) verify_netdata_host_prefix(true);
1200
-
1201
- char buf[HOSTNAME_MAX + 1];
1202
- if (get_hostname(buf, HOSTNAME_MAX))
1203
- netdata_log_error("Cannot get machine hostname.");
1204
-
1205
- netdata_configured_hostname = config_get(CONFIG_SECTION_GLOBAL, "hostname", buf);
1206
- netdata_log_debug(D_OPTIONS, "hostname set to '%s'", netdata_configured_hostname);
1207
-
1208
- // ------------------------------------------------------------------------
1209
- // get default database update frequency
1210
-
1211
- default_rrd_update_every = (int) config_get_duration_seconds(CONFIG_SECTION_DB, "update every", UPDATE_EVERY);
1212
- if(default_rrd_update_every < 1 || default_rrd_update_every > 600) {
1213
- netdata_log_error("Invalid data collection frequency (update every) %d given. Defaulting to %d.", default_rrd_update_every, UPDATE_EVERY);
1214
- default_rrd_update_every = UPDATE_EVERY;
1215
- config_set_duration_seconds(CONFIG_SECTION_DB, "update every", default_rrd_update_every);
1216
- }
1217
-
1218
- // ------------------------------------------------------------------------
1219
- // get the database selection
1220
-
1221
- {
1222
- const char *mode = config_get(CONFIG_SECTION_DB, "db", rrd_memory_mode_name(default_rrd_memory_mode));
1223
- default_rrd_memory_mode = rrd_memory_mode_id(mode);
1224
- if(strcmp(mode, rrd_memory_mode_name(default_rrd_memory_mode)) != 0) {
1225
- netdata_log_error("Invalid memory mode '%s' given. Using '%s'", mode, rrd_memory_mode_name(default_rrd_memory_mode));
1226
- config_set(CONFIG_SECTION_DB, "db", rrd_memory_mode_name(default_rrd_memory_mode));
1227
- }
1228
- }
1229
-
1230
- // ------------------------------------------------------------------------
1231
- // get default database size
1232
-
1233
- if(default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE && default_rrd_memory_mode != RRD_MEMORY_MODE_NONE) {
1234
- default_rrd_history_entries = (int)config_get_number(
1235
- CONFIG_SECTION_DB, "retention",
1236
- align_entries_to_pagesize(default_rrd_memory_mode, RRD_DEFAULT_HISTORY_ENTRIES));
1237
-
1238
- long h = align_entries_to_pagesize(default_rrd_memory_mode, default_rrd_history_entries);
1239
- if (h != default_rrd_history_entries) {
1240
- config_set_number(CONFIG_SECTION_DB, "retention", h);
1241
- default_rrd_history_entries = (int)h;
1242
- }
1243
- }
1244
-
1245
- // ------------------------------------------------------------------------
1246
- // get system paths
1247
-
1248
- netdata_configured_user_config_dir = config_get(CONFIG_SECTION_DIRECTORIES, "config", netdata_configured_user_config_dir);
1249
- netdata_configured_stock_config_dir = config_get(CONFIG_SECTION_DIRECTORIES, "stock config", netdata_configured_stock_config_dir);
1250
- netdata_configured_log_dir = config_get(CONFIG_SECTION_DIRECTORIES, "log", netdata_configured_log_dir);
1251
- netdata_configured_web_dir = config_get(CONFIG_SECTION_DIRECTORIES, "web", netdata_configured_web_dir);
1252
- netdata_configured_cache_dir = config_get(CONFIG_SECTION_DIRECTORIES, "cache", netdata_configured_cache_dir);
1253
- netdata_configured_varlib_dir = config_get(CONFIG_SECTION_DIRECTORIES, "lib", netdata_configured_varlib_dir);
1254
-
1255
- netdata_configured_lock_dir = get_varlib_subdir_from_config(netdata_configured_varlib_dir, "lock");
1256
- netdata_configured_cloud_dir = get_varlib_subdir_from_config(netdata_configured_varlib_dir, "cloud.d");
1257
-
1258
- {
1259
- pluginsd_initialize_plugin_directories();
1260
- netdata_configured_primary_plugins_dir = plugin_directories[PLUGINSD_STOCK_PLUGINS_DIRECTORY_PATH];
1261
- }
1262
-
1263
-#ifdef ENABLE_DBENGINE
1264
- // ------------------------------------------------------------------------
1265
- // get default Database Engine page type
1266
-
1267
- const char *page_type = config_get(CONFIG_SECTION_DB, "dbengine page type", "gorilla");
1268
- if (strcmp(page_type, "gorilla") == 0)
1269
- tier_page_type[0] = RRDENG_PAGE_TYPE_GORILLA_32BIT;
1270
- else if (strcmp(page_type, "raw") == 0)
1271
- tier_page_type[0] = RRDENG_PAGE_TYPE_ARRAY_32BIT;
1272
- else {
1273
- tier_page_type[0] = RRDENG_PAGE_TYPE_ARRAY_32BIT;
1274
- netdata_log_error("Invalid dbengine page type ''%s' given. Defaulting to 'raw'.", page_type);
1275
- }
1276
-
1277
- // ------------------------------------------------------------------------
1278
- // get default Database Engine page cache size in MiB
1279
-
1280
- default_rrdeng_page_cache_mb = (int) config_get_size_mb(CONFIG_SECTION_DB, "dbengine page cache size", default_rrdeng_page_cache_mb);
1281
- default_rrdeng_extent_cache_mb = (int) config_get_size_mb(CONFIG_SECTION_DB, "dbengine extent cache size", default_rrdeng_extent_cache_mb);
1282
- db_engine_journal_check = config_get_boolean(CONFIG_SECTION_DB, "dbengine enable journal integrity check", CONFIG_BOOLEAN_NO);
1283
-
1284
- if(default_rrdeng_extent_cache_mb < 0) {
1285
- default_rrdeng_extent_cache_mb = 0;
1286
- config_set_size_mb(CONFIG_SECTION_DB, "dbengine extent cache size", default_rrdeng_extent_cache_mb);
1287
- }
1288
-
1289
- if(default_rrdeng_page_cache_mb < RRDENG_MIN_PAGE_CACHE_SIZE_MB) {
1290
- netdata_log_error("Invalid page cache size %d given. Defaulting to %d.", default_rrdeng_page_cache_mb, RRDENG_MIN_PAGE_CACHE_SIZE_MB);
1291
- default_rrdeng_page_cache_mb = RRDENG_MIN_PAGE_CACHE_SIZE_MB;
1292
- config_set_size_mb(CONFIG_SECTION_DB, "dbengine page cache size", default_rrdeng_page_cache_mb);
1293
- }
1294
-
1295
- // ------------------------------------------------------------------------
1296
- // get default Database Engine disk space quota in MiB
1297
-//
1298
-// // if (!config_exists(CONFIG_SECTION_DB, "dbengine disk space MB") && !config_exists(CONFIG_SECTION_DB, "dbengine multihost disk space MB"))
1299
-//
1300
-// default_rrdeng_disk_quota_mb = (int) config_get_number(CONFIG_SECTION_DB, "dbengine disk space MB", default_rrdeng_disk_quota_mb);
1301
-// if(default_rrdeng_disk_quota_mb < RRDENG_MIN_DISK_SPACE_MB) {
1302
-// netdata_log_error("Invalid dbengine disk space %d given. Defaulting to %d.", default_rrdeng_disk_quota_mb, RRDENG_MIN_DISK_SPACE_MB);
1303
-// default_rrdeng_disk_quota_mb = RRDENG_MIN_DISK_SPACE_MB;
1304
-// config_set_number(CONFIG_SECTION_DB, "dbengine disk space MB", default_rrdeng_disk_quota_mb);
1305
-// }
1306
-//
1307
-// default_multidb_disk_quota_mb = (int) config_get_number(CONFIG_SECTION_DB, "dbengine multihost disk space MB", compute_multidb_diskspace());
1308
-// if(default_multidb_disk_quota_mb < RRDENG_MIN_DISK_SPACE_MB) {
1309
-// netdata_log_error("Invalid multidb disk space %d given. Defaulting to %d.", default_multidb_disk_quota_mb, default_rrdeng_disk_quota_mb);
1310
-// default_multidb_disk_quota_mb = default_rrdeng_disk_quota_mb;
1311
-// config_set_number(CONFIG_SECTION_DB, "dbengine multihost disk space MB", default_multidb_disk_quota_mb);
1312
-// }
1313
-
1314
-#else
1315
- if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
1316
- error_report("RRD_MEMORY_MODE_DBENGINE is not supported in this platform. The agent will use db mode 'save' instead.");
1317
- default_rrd_memory_mode = RRD_MEMORY_MODE_RAM;
1318
- }
1319
-#endif
1320
-
1321
- // --------------------------------------------------------------------
1322
- // get KSM settings
1323
-
1324
-#ifdef MADV_MERGEABLE
1325
- enable_ksm = config_get_boolean_ondemand(CONFIG_SECTION_DB, "memory deduplication (ksm)", enable_ksm);
1326
-#endif
1327
-
1328
- // --------------------------------------------------------------------
1329
-
1330
- rrdhost_free_ephemeral_time_s =
1331
- config_get_duration_seconds(CONFIG_SECTION_DB, "cleanup ephemeral hosts after", rrdhost_free_ephemeral_time_s);
1332
-
1333
- rrdset_free_obsolete_time_s =
1334
- config_get_duration_seconds(CONFIG_SECTION_DB, "cleanup obsolete charts after", rrdset_free_obsolete_time_s);
1335
-
1336
- // Current chart locking and invalidation scheme doesn't prevent Netdata from segmentation faults if a short
1337
- // cleanup delay is set. Extensive stress tests showed that 10 seconds is quite a safe delay. Look at
1338
- // https://github.com/netdata/netdata/pull/11222#issuecomment-868367920 for more information.
1339
- if (rrdset_free_obsolete_time_s < 10) {
1340
- rrdset_free_obsolete_time_s = 10;
1341
- netdata_log_info("The \"cleanup obsolete charts after\" option was set to 10 seconds.");
1342
- config_set_duration_seconds(CONFIG_SECTION_DB, "cleanup obsolete charts after", rrdset_free_obsolete_time_s);
1343
- }
1344
-
1345
- gap_when_lost_iterations_above = (int)config_get_number(CONFIG_SECTION_DB, "gap when lost iterations above", gap_when_lost_iterations_above);
1346
- if (gap_when_lost_iterations_above < 1) {
1347
- gap_when_lost_iterations_above = 1;
1348
- config_set_number(CONFIG_SECTION_DB, "gap when lost iterations above", gap_when_lost_iterations_above);
1349
- }
1350
- gap_when_lost_iterations_above += 2;
1351
-
1352
- // --------------------------------------------------------------------
1353
- // get various system parameters
1354
-
1355
- os_get_system_cpus_uncached();
1356
- os_get_system_pid_max();
1357
-
1358
-
1359
-}
1360
-
1361
-static void post_conf_load(const char **user)
1362
-{
1363
- // --------------------------------------------------------------------
1364
- // get the user we should run
1365
-
1366
- // IMPORTANT: this is required before web_files_uid()
1367
- if(getuid() == 0) {
1368
- *user = config_get(CONFIG_SECTION_GLOBAL, "run as user", NETDATA_USER);
1369
- }
1370
- else {
1371
- struct passwd *passwd = getpwuid(getuid());
1372
- *user = config_get(CONFIG_SECTION_GLOBAL, "run as user", (passwd && passwd->pw_name)?passwd->pw_name:"");
1373
- }
1374
-}
1375
-
1376
-static bool load_netdata_conf(char *filename, char overwrite_used, const char **user) {
1377
- errno_clear();
1378
-
1379
- int ret = 0;
1380
-
1381
- if(filename && *filename) {
1382
- ret = config_load(filename, overwrite_used, NULL);
1383
- if(!ret)
1384
- netdata_log_error("CONFIG: cannot load config file '%s'.", filename);
1385
- }
1386
- else {
1387
- filename = filename_from_path_entry_strdupz(netdata_configured_user_config_dir, "netdata.conf");
1388
-
1389
- ret = config_load(filename, overwrite_used, NULL);
1390
- if(!ret) {
1391
- netdata_log_info("CONFIG: cannot load user config '%s'. Will try the stock version.", filename);
1392
- freez(filename);
1393
-
1394
- filename = filename_from_path_entry_strdupz(netdata_configured_stock_config_dir, "netdata.conf");
1395
- ret = config_load(filename, overwrite_used, NULL);
1396
- if(!ret)
1397
- netdata_log_info("CONFIG: cannot load stock config '%s'. Running with internal defaults.", filename);
1398
- }
1399
-
1400
- freez(filename);
1401
- }
1402
-
1403
- post_conf_load(user);
1404
- return ret;
1405
-}
1406
-
706
// coverity[ +tainted_string_sanitize_content : arg-0 ]
707
static inline void coverity_remove_taint(char *s)
708
{
@@ -1476,7 +775,7 @@ int julytest(void);
775
int pluginsd_parser_unittest(void);
776
void replication_initialize(void);
777
void bearer_tokens_init(void);
1479
-int unittest_rrdpush_compressions(void);
778
+int unittest_stream_compressions(void);
779
int uuid_unittest(void);
780
int progress_unittest(void);
781
int dyncfg_unittest(void);
@@ -1487,8 +786,8 @@ int windows_perflib_dump(const char *key);
786
#endif
787
788
int unittest_prepare_rrd(const char **user) {
1490
- post_conf_load(user);
1491
- get_netdata_configured_variables();
789
+ netdata_conf_section_global_run_as_user(user);
790
+ netdata_conf_section_global();
791
default_rrd_update_every = 1;
792
default_rrd_memory_mode = RRD_MEMORY_MODE_RAM;
793
health_plugin_disable();
@@ -1498,7 +797,7 @@ int unittest_prepare_rrd(const char **user) {
797
fprintf(stderr, "rrd_init failed for unittest\n");
798
return 1;
799
}
1501
- stream_conf_send_enabled = 0;
800
+ stream_send.enabled = false;
801
802
return 0;
803
}
@@ -1554,7 +853,7 @@ int netdata_main(int argc, char **argv) {
853
while( (opt = getopt(argc, argv, optstring)) != -1 ) {
854
switch(opt) {
855
case 'c':
1557
- if(!load_netdata_conf(optarg, 1, &user)) {
856
+ if(!netdata_conf_load(optarg, 1, &user)) {
857
netdata_log_error("Cannot load configuration file %s.", optarg);
858
return 1;
859
}
@@ -1725,9 +1024,9 @@ int netdata_main(int argc, char **argv) {
1024
unittest_running = true;
1025
return pluginsd_parser_unittest();
1026
}
1728
- else if(strcmp(optarg, "rrdpush_compressions_test") == 0) {
1027
+ else if(strcmp(optarg, "stream_compressions_test") == 0) {
1028
unittest_running = true;
1730
- return unittest_rrdpush_compressions();
1029
+ return unittest_stream_compressions();
1030
}
1031
else if(strcmp(optarg, "progresstest") == 0) {
1032
unittest_running = true;
@@ -1742,8 +1041,8 @@ int netdata_main(int argc, char **argv) {
1041
else if(strncmp(optarg, createdataset_string, strlen(createdataset_string)) == 0) {
1042
optarg += strlen(createdataset_string);
1043
unsigned history_seconds = strtoul(optarg, NULL, 0);
1745
- post_conf_load(&user);
1746
- get_netdata_configured_variables();
1044
+ netdata_conf_section_global_run_as_user(&user);
1045
+ netdata_conf_section_global();
1046
default_rrd_update_every = 1;
1047
registry_init();
1048
if(rrd_init("dbengine-dataset", NULL, true)) {
@@ -1917,10 +1216,10 @@ int netdata_main(int argc, char **argv) {
1216
1217
if(!config_loaded) {
1218
fprintf(stderr, "warning: no configuration file has been loaded. Use -c CONFIG_FILE, before -W get. Using default config.\n");
1920
- load_netdata_conf(NULL, 0, &user);
1219
+ netdata_conf_load(NULL, 0, &user);
1220
}
1221
1923
- get_netdata_configured_variables();
1222
+ netdata_conf_section_global();
1223
1224
const char *section = argv[optind];
1225
const char *key = argv[optind + 1];
@@ -1944,11 +1243,11 @@ int netdata_main(int argc, char **argv) {
1243
1244
if(!config_loaded) {
1245
fprintf(stderr, "warning: no configuration file has been loaded. Use -c CONFIG_FILE, before -W get. Using default config.\n");
1947
- load_netdata_conf(NULL, 0, &user);
1246
+ netdata_conf_load(NULL, 0, &user);
1247
cloud_conf_load(1);
1248
}
1249
1951
- get_netdata_configured_variables();
1250
+ netdata_conf_section_global();
1251
1252
const char *conf_file = argv[optind]; /* "cloud" is cloud.conf, otherwise netdata.conf */
1253
struct config *tmp_config = strcmp(conf_file, "cloud") ? &netdata_config : &cloud_config;
@@ -1994,7 +1293,7 @@ int netdata_main(int argc, char **argv) {
1293
}
1294
1295
if(!config_loaded) {
1997
- load_netdata_conf(NULL, 0, &user);
1296
+ netdata_conf_load(NULL, 0, &user);
1297
cloud_conf_load(0);
1298
}
1299
@@ -2040,7 +1339,7 @@ int netdata_main(int argc, char **argv) {
1339
}
1340
1341
// prepare configuration environment variables for the plugins
2043
- get_netdata_configured_variables();
1342
+ netdata_conf_section_global();
1343
set_environment_for_plugins_and_scripts();
1344
analytics_reset();
1345
@@ -2078,7 +1377,7 @@ int netdata_main(int argc, char **argv) {
1377
// --------------------------------------------------------------------
1378
// get log filenames and settings
1379
2081
- log_init();
1380
+ netdata_conf_section_logs();
1381
nd_log_limits_unlimited();
1382
1383
// initialize the log files
@@ -2096,7 +1395,7 @@ int netdata_main(int argc, char **argv) {
1395
// --------------------------------------------------------------------
1396
// get the certificate and start security
1397
2099
- security_init();
1398
+ netdata_conf_web_security_init();
1399
1400
// --------------------------------------------------------------------
1401
// This is the safest place to start the SILENCERS structure
@@ -2125,11 +1424,14 @@ int netdata_main(int argc, char **argv) {
1424
default_stacksize = 1 * 1024 * 1024;
1425
1426
#ifdef NETDATA_INTERNAL_CHECKS
2128
- config_set_boolean(CONFIG_SECTION_PLUGINS, "netdata monitoring", true);
2129
- config_set_boolean(CONFIG_SECTION_PLUGINS, "netdata monitoring extended", true);
1427
+ telemetry_enabled = true;
1428
+ telemetry_extended_enabled = true;
1429
#endif
1430
2132
- if(config_get_boolean(CONFIG_SECTION_PLUGINS, "netdata monitoring extended", false))
1431
+ telemetry_extended_enabled =
1432
+ config_get_boolean(CONFIG_SECTION_TELEMETRY, "extended telemetry", telemetry_extended_enabled);
1433
+
1434
+ if(telemetry_extended_enabled)
1435
// this has to run before starting any other threads that use workers
1436
workers_utilization_enable();
1437
@@ -2239,7 +1541,7 @@ int netdata_main(int argc, char **argv) {
1541
netdata_random_session_id_generate();
1542
1543
// ------------------------------------------------------------------------
2242
- // initialize rrd, registry, health, rrdpush, etc.
1544
+ // initialize rrd, registry, health, streaming, etc.
1545
1546
delta_startup_time("collecting system info");
1547
@@ -2295,11 +1597,12 @@ int netdata_main(int argc, char **argv) {
1597
// ------------------------------------------------------------------------
1598
// spawn the threads
1599
1600
+ get_agent_event_time_median_init();
1601
bearer_tokens_init();
1602
1603
delta_startup_time("start the static threads");
1604
2302
- web_server_config_options();
1605
+ netdata_conf_section_web();
1606
1607
set_late_analytics_variables(system_info);
1608
for (i = 0; static_threads[i].name != NULL ; i++) {
src/daemon/service.c
+10
-9
@@ -166,7 +166,7 @@ static void svc_rrdhost_detect_obsolete_charts(RRDHOST *host) {
166
time_t last_entry_t;
167
RRDSET *st;
168
169
- time_t child_connect_time = host->child_connect_time;
169
+ time_t child_connect_time = host->stream.rcv.status.last_connected;
170
171
rrdset_foreach_read(st, host) {
172
if(rrdset_is_replicating(st))
@@ -203,19 +203,19 @@ static void svc_rrd_cleanup_obsolete_charts_from_all_hosts() {
203
if (host == localhost)
204
continue;
205
206
- spinlock_lock(&host->receiver_lock);
206
+ rrdhost_receiver_lock(host);
207
208
time_t now = now_realtime_sec();
209
210
- if (host->trigger_chart_obsoletion_check &&
211
- ((host->child_last_chart_command &&
212
- host->child_last_chart_command + host->health.health_delay_up_to < now) ||
213
- (host->child_connect_time + TIME_TO_RUN_OBSOLETIONS_ON_CHILD_CONNECT < now))) {
210
+ if (host->stream.rcv.status.check_obsolete &&
211
+ ((host->stream.rcv.status.last_chart &&
212
+ host->stream.rcv.status.last_chart + host->health.delay_up_to < now) ||
213
+ (host->stream.rcv.status.last_connected + TIME_TO_RUN_OBSOLETIONS_ON_CHILD_CONNECT < now))) {
214
svc_rrdhost_detect_obsolete_charts(host);
215
- host->trigger_chart_obsoletion_check = 0;
215
+ host->stream.rcv.status.check_obsolete = false;
216
}
217
218
- spinlock_unlock(&host->receiver_lock);
218
+ rrdhost_receiver_unlock(host);
219
}
220
221
rrd_rdunlock();
@@ -235,7 +235,8 @@ restart_after_removal:
235
continue;
236
237
bool force = false;
238
- if (rrdhost_option_check(host, RRDHOST_OPTION_EPHEMERAL_HOST) && now - host->last_connected > rrdhost_free_ephemeral_time_s)
238
+ if (rrdhost_option_check(host, RRDHOST_OPTION_EPHEMERAL_HOST) &&
239
+ now - host->stream.snd.status.last_connected > rrdhost_free_ephemeral_time_s)
240
force = true;
241
242
bool is_archived = rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED);
src/daemon/static_threads.c
+13
-16
@@ -5,8 +5,6 @@
5
void *aclk_main(void *ptr);
6
void *analytics_main(void *ptr);
7
void *cpuidlejitter_main(void *ptr);
8
-void *global_statistics_main(void *ptr);
9
-void *global_statistics_extended_main(void *ptr);
8
void *health_main(void *ptr);
9
void *pluginsd_main(void *ptr);
10
void *service_main(void *ptr);
@@ -14,7 +12,7 @@ void *statsd_main(void *ptr);
12
void *profile_main(void *ptr);
13
void *replication_thread_main(void *ptr);
14
17
-extern bool global_statistics_enabled;
15
+extern bool telemetry_enabled;
16
17
const struct netdata_static_thread static_threads_common[] = {
18
{
@@ -45,26 +43,26 @@ const struct netdata_static_thread static_threads_common[] = {
43
.start_routine = analytics_main
44
},
45
{
48
- .name = "STATS_GLOBAL",
46
+ .name = "TELEMETRY",
47
.config_section = CONFIG_SECTION_PLUGINS,
50
- .config_name = "netdata monitoring",
48
+ .config_name = "netdata telemetry",
49
.env_name = "NETDATA_INTERNALS_MONITORING",
52
- .global_variable = &global_statistics_enabled,
50
+ .global_variable = &telemetry_enabled,
51
.enabled = 1,
52
.thread = NULL,
53
.init_routine = NULL,
56
- .start_routine = global_statistics_main
54
+ .start_routine = telemetry_thread_main
55
},
56
{
59
- .name = "STATS_GLOBAL_EXT",
60
- .config_section = CONFIG_SECTION_PLUGINS,
61
- .config_name = "netdata monitoring extended",
62
- .env_name = "NETDATA_INTERNALS_EXTENDED_MONITORING",
63
- .global_variable = &global_statistics_enabled,
64
- .enabled = 0, // this is ignored - check main() for "netdata monitoring extended"
57
+ .name = "TLMTRY-SQLITE3",
58
+ .config_section = CONFIG_SECTION_TELEMETRY,
59
+ .config_name = "extended telemetry",
60
+ .env_name = NULL,
61
+ .global_variable = &telemetry_extended_enabled,
62
+ .enabled = 0, // the default value - it uses netdata.conf for users to enable it
63
.thread = NULL,
64
.init_routine = NULL,
67
- .start_routine = global_statistics_extended_main
65
+ .start_routine = telemetry_thread_sqlite3_main
66
},
67
{
68
.name = "PLUGINSD",
@@ -109,8 +107,7 @@ const struct netdata_static_thread static_threads_common[] = {
107
.enabled = 0,
108
.thread = NULL,
109
.init_routine = NULL,
112
- .start_routine = rrdpush_sender_thread
113
- },
110
+ .start_routine = stream_sender_start_localhost},
111
{
112
.name = "WEB[1]",
113
.config_section = NULL,
src/daemon/telemetry/telemetry-aral.c
new
+164
@@ -0,0 +1,164 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-aral.h"
5
+
6
+struct aral_info {
7
+ const char *name;
8
+ RRDSET *st_memory;
9
+ RRDDIM *rd_used, *rd_free, *rd_structures;
10
+
11
+ RRDSET *st_utilization;
12
+ RRDDIM *rd_utilization;
13
+};
14
+
15
+DEFINE_JUDYL_TYPED(ARAL_STATS, struct aral_info *);
16
+
17
+static struct {
18
+ SPINLOCK spinlock;
19
+ ARAL_STATS_JudyLSet idx;
20
+} globals = { 0 };
21
+
22
+static void telemetry_aral_register_statistics(struct aral_statistics *stats, const char *name) {
23
+ if(!name || !stats)
24
+ return;
25
+
26
+ spinlock_lock(&globals.spinlock);
27
+ struct aral_info *ai = ARAL_STATS_GET(&globals.idx, (Word_t)stats);
28
+ if(!ai) {
29
+ ai = callocz(1, sizeof(struct aral_info));
30
+ ai->name = strdupz(name);
31
+ ARAL_STATS_SET(&globals.idx, (Word_t)stats, ai);
32
+ }
33
+ spinlock_unlock(&globals.spinlock);
34
+}
35
+
36
+void telemetry_aral_register(ARAL *ar, const char *name) {
37
+ if(!ar) return;
38
+
39
+ if(!name)
40
+ name = aral_name(ar);
41
+
42
+ struct aral_statistics *stats = aral_get_statistics(ar);
43
+
44
+ telemetry_aral_register_statistics(stats, name);
45
+}
46
+
47
+void telemetry_aral_unregister(ARAL *ar) {
48
+ if(!ar) return;
49
+ struct aral_statistics *stats = aral_get_statistics(ar);
50
+
51
+ spinlock_lock(&globals.spinlock);
52
+ struct aral_info *ai = ARAL_STATS_GET(&globals.idx, (Word_t)stats);
53
+ if(ai) {
54
+ ARAL_STATS_DEL(&globals.idx, (Word_t)stats);
55
+ freez((void *)ai->name);
56
+ freez(ai);
57
+ }
58
+ spinlock_unlock(&globals.spinlock);
59
+}
60
+
61
+void telemerty_aral_init(void) {
62
+ telemetry_aral_register_statistics(aral_by_size_statistics(), "by-size");
63
+}
64
+
65
+void telemetry_aral_do(bool extended) {
66
+ if(!extended) return;
67
+
68
+ spinlock_lock(&globals.spinlock);
69
+ Word_t s = 0;
70
+ for(struct aral_info *ai = ARAL_STATS_FIRST(&globals.idx, &s);
71
+ ai;
72
+ ai = ARAL_STATS_NEXT(&globals.idx, &s)) {
73
+ struct aral_statistics *stats = (void *)(uintptr_t)s;
74
+ if (!stats)
75
+ continue;
76
+
77
+ size_t allocated_bytes = __atomic_load_n(&stats->malloc.allocated_bytes, __ATOMIC_RELAXED) +
78
+ __atomic_load_n(&stats->mmap.allocated_bytes, __ATOMIC_RELAXED);
79
+
80
+ size_t used_bytes = __atomic_load_n(&stats->malloc.used_bytes, __ATOMIC_RELAXED) +
81
+ __atomic_load_n(&stats->mmap.used_bytes, __ATOMIC_RELAXED);
82
+
83
+ // slight difference may exist, due to the time needed to get these values
84
+ // fix the obvious discrepancies
85
+ if(used_bytes > allocated_bytes)
86
+ used_bytes = allocated_bytes;
87
+
88
+ size_t structures_bytes = __atomic_load_n(&stats->structures.allocated_bytes, __ATOMIC_RELAXED);
89
+
90
+ size_t free_bytes = allocated_bytes - used_bytes;
91
+
92
+ NETDATA_DOUBLE utilization;
93
+ if(used_bytes && allocated_bytes)
94
+ utilization = 100.0 * (NETDATA_DOUBLE)used_bytes / (NETDATA_DOUBLE)allocated_bytes;
95
+ else
96
+ utilization = 100.0;
97
+
98
+ {
99
+ if (unlikely(!ai->st_memory)) {
100
+ char id[256];
101
+
102
+ snprintfz(id, sizeof(id), "aral_%s_memory", ai->name);
103
+ netdata_fix_chart_id(id);
104
+
105
+ ai->st_memory = rrdset_create_localhost(
106
+ "netdata",
107
+ id,
108
+ NULL,
109
+ "ARAL",
110
+ "netdata.aral_memory",
111
+ "Array Allocator Memory Utilization",
112
+ "bytes",
113
+ "netdata",
114
+ "telemetry",
115
+ 910000,
116
+ localhost->rrd_update_every,
117
+ RRDSET_TYPE_STACKED);
118
+
119
+ rrdlabels_add(ai->st_memory->rrdlabels, "ARAL", ai->name, RRDLABEL_SRC_AUTO);
120
+
121
+ ai->rd_free = rrddim_add(ai->st_memory, "free", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
122
+ ai->rd_used = rrddim_add(ai->st_memory, "used", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
123
+ ai->rd_structures = rrddim_add(ai->st_memory, "structures", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
124
+ }
125
+
126
+ rrddim_set_by_pointer(ai->st_memory, ai->rd_used, (collected_number)allocated_bytes);
127
+ rrddim_set_by_pointer(ai->st_memory, ai->rd_free, (collected_number)free_bytes);
128
+ rrddim_set_by_pointer(ai->st_memory, ai->rd_structures, (collected_number)structures_bytes);
129
+ rrdset_done(ai->st_memory);
130
+ }
131
+
132
+ {
133
+ if (unlikely(!ai->st_utilization)) {
134
+ char id[256];
135
+
136
+ snprintfz(id, sizeof(id), "aral_%s_utilization", ai->name);
137
+ netdata_fix_chart_id(id);
138
+
139
+ ai->st_utilization = rrdset_create_localhost(
140
+ "netdata",
141
+ id,
142
+ NULL,
143
+ "ARAL",
144
+ "netdata.aral_utilization",
145
+ "Array Allocator Memory Utilization",
146
+ "%",
147
+ "netdata",
148
+ "telemetry",
149
+ 910001,
150
+ localhost->rrd_update_every,
151
+ RRDSET_TYPE_LINE);
152
+
153
+ rrdlabels_add(ai->st_utilization->rrdlabels, "ARAL", ai->name, RRDLABEL_SRC_AUTO);
154
+
155
+ ai->rd_utilization = rrddim_add(ai->st_utilization, "utilization", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
156
+ }
157
+
158
+ rrddim_set_by_pointer(ai->st_utilization, ai->rd_utilization, (collected_number)(utilization * 10000.0));
159
+ rrdset_done(ai->st_utilization);
160
+ }
161
+ }
162
+
163
+ spinlock_unlock(&globals.spinlock);
164
+}
src/daemon/telemetry/telemetry-aral.h
new
+16
@@ -0,0 +1,16 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_ARAL_H
4
+#define NETDATA_TELEMETRY_ARAL_H
5
+
6
+#include "daemon/common.h"
7
+
8
+void telemetry_aral_register(ARAL *ar, const char *name);
9
+void telemetry_aral_unregister(ARAL *ar);
10
+
11
+#if defined(TELEMETRY_INTERNALS)
12
+void telemerty_aral_init(void);
13
+void telemetry_aral_do(bool extended);
14
+#endif
15
+
16
+#endif //NETDATA_TELEMETRY_ARAL_H
src/daemon/telemetry/telemetry-daemon-memory.c
new
+243
@@ -0,0 +1,243 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-daemon-memory.h"
5
+
6
+#define dictionary_stats_memory_total(stats) \
7
+ ((stats).memory.dict + (stats).memory.values + (stats).memory.index)
8
+
9
+struct netdata_buffers_statistics netdata_buffers_statistics = {};
10
+
11
+void telemetry_daemon_memory_do(bool extended) {
12
+ {
13
+ static RRDSET *st_memory = NULL;
14
+ static RRDDIM *rd_database = NULL;
15
+#ifdef DICT_WITH_STATS
16
+ static RRDDIM *rd_collectors = NULL;
17
+ static RRDDIM *rd_rrdhosts = NULL;
18
+ static RRDDIM *rd_rrdsets = NULL;
19
+ static RRDDIM *rd_rrddims = NULL;
20
+ static RRDDIM *rd_contexts = NULL;
21
+ static RRDDIM *rd_health = NULL;
22
+ static RRDDIM *rd_functions = NULL;
23
+ static RRDDIM *rd_replication = NULL;
24
+#else
25
+ static RRDDIM *rd_metadata = NULL;
26
+#endif
27
+ static RRDDIM *rd_labels = NULL; // labels use dictionary like statistics, but it is not ARAL based dictionary
28
+ static RRDDIM *rd_ml = NULL;
29
+ static RRDDIM *rd_strings = NULL;
30
+ static RRDDIM *rd_streaming = NULL;
31
+ static RRDDIM *rd_buffers = NULL;
32
+ static RRDDIM *rd_workers = NULL;
33
+ static RRDDIM *rd_aral = NULL;
34
+ static RRDDIM *rd_judy = NULL;
35
+ static RRDDIM *rd_other = NULL;
36
+
37
+ if (unlikely(!st_memory)) {
38
+ st_memory = rrdset_create_localhost(
39
+ "netdata",
40
+ "memory",
41
+ NULL,
42
+ "Memory Usage",
43
+ NULL,
44
+ "Netdata Memory",
45
+ "bytes",
46
+ "netdata",
47
+ "stats",
48
+ 130100,
49
+ localhost->rrd_update_every,
50
+ RRDSET_TYPE_STACKED);
51
+
52
+ rd_database = rrddim_add(st_memory, "db", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
53
+
54
+#ifdef DICT_WITH_STATS
55
+ rd_collectors = rrddim_add(st_memory, "collectors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
56
+ rd_rrdhosts = rrddim_add(st_memory, "hosts", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
57
+ rd_rrdsets = rrddim_add(st_memory, "rrdset", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
58
+ rd_rrddims = rrddim_add(st_memory, "rrddim", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
59
+ rd_contexts = rrddim_add(st_memory, "contexts", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
60
+ rd_health = rrddim_add(st_memory, "health", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
61
+ rd_functions = rrddim_add(st_memory, "functions", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
62
+ rd_replication = rrddim_add(st_memory, "replication", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
63
+#else
64
+ rd_metadata = rrddim_add(st_memory, "metadata", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
65
+#endif
66
+ rd_labels = rrddim_add(st_memory, "labels", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
67
+ rd_ml = rrddim_add(st_memory, "ML", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
68
+ rd_strings = rrddim_add(st_memory, "strings", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
69
+ rd_streaming = rrddim_add(st_memory, "streaming", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
70
+ rd_buffers = rrddim_add(st_memory, "buffers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
71
+ rd_workers = rrddim_add(st_memory, "workers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
72
+ rd_aral = rrddim_add(st_memory, "aral", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
73
+ rd_judy = rrddim_add(st_memory, "judy", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
74
+ rd_other = rrddim_add(st_memory, "other", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
75
+ }
76
+
77
+ size_t buffers =
78
+ netdata_buffers_statistics.query_targets_size +
79
+ netdata_buffers_statistics.rrdset_done_rda_size +
80
+ netdata_buffers_statistics.buffers_aclk +
81
+ netdata_buffers_statistics.buffers_api +
82
+ netdata_buffers_statistics.buffers_functions +
83
+ netdata_buffers_statistics.buffers_sqlite +
84
+ netdata_buffers_statistics.buffers_exporters +
85
+ netdata_buffers_statistics.buffers_health +
86
+ netdata_buffers_statistics.buffers_streaming +
87
+ netdata_buffers_statistics.cbuffers_streaming +
88
+ netdata_buffers_statistics.buffers_web +
89
+ replication_allocated_buffers() +
90
+ aral_by_size_overhead() +
91
+ judy_aral_overhead();
92
+
93
+ size_t strings = 0;
94
+ string_statistics(NULL, NULL, NULL, NULL, NULL, &strings, NULL, NULL);
95
+
96
+ rrddim_set_by_pointer(st_memory, rd_database,
97
+ (collected_number)telemetry_dbengine_total_memory + (collected_number)rrddim_db_memory_size);
98
+
99
+#ifdef DICT_WITH_STATS
100
+ rrddim_set_by_pointer(st_memory, rd_collectors,
101
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_collectors));
102
+
103
+ rrddim_set_by_pointer(st_memory,
104
+ rd_rrdhosts,
105
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdhost) + (collected_number)netdata_buffers_statistics.rrdhost_allocations_size);
106
+
107
+ rrddim_set_by_pointer(st_memory, rd_rrdsets,
108
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdset));
109
+
110
+ rrddim_set_by_pointer(st_memory, rd_rrddims,
111
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrddim));
112
+
113
+ rrddim_set_by_pointer(st_memory, rd_contexts,
114
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdcontext));
115
+
116
+ rrddim_set_by_pointer(st_memory, rd_health,
117
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdhealth));
118
+
119
+ rrddim_set_by_pointer(st_memory, rd_functions,
120
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_functions));
121
+
122
+ rrddim_set_by_pointer(st_memory, rd_replication,
123
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_replication) + (collected_number)replication_allocated_memory());
124
+#else
125
+ uint64_t metadata =
126
+ aral_by_size_used_bytes() +
127
+ dictionary_stats_category_rrdhost.memory.dict +
128
+ dictionary_stats_category_rrdset.memory.dict +
129
+ dictionary_stats_category_rrddim.memory.dict +
130
+ dictionary_stats_category_rrdcontext.memory.dict +
131
+ dictionary_stats_category_rrdhealth.memory.dict +
132
+ dictionary_stats_category_functions.memory.dict +
133
+ dictionary_stats_category_replication.memory.dict +
134
+ replication_allocated_memory();
135
+
136
+ rrddim_set_by_pointer(st_memory, rd_metadata, (collected_number)metadata);
137
+#endif
138
+
139
+ // labels use dictionary like statistics, but it is not ARAL based dictionary
140
+ rrddim_set_by_pointer(st_memory, rd_labels,
141
+ (collected_number)dictionary_stats_memory_total(dictionary_stats_category_rrdlabels));
142
+
143
+ rrddim_set_by_pointer(st_memory, rd_ml,
144
+ (collected_number)telemetry_ml_get_current_memory_usage());
145
+
146
+ rrddim_set_by_pointer(st_memory, rd_strings,
147
+ (collected_number)strings);
148
+
149
+ rrddim_set_by_pointer(st_memory, rd_streaming,
150
+ (collected_number)netdata_buffers_statistics.rrdhost_senders + (collected_number)netdata_buffers_statistics.rrdhost_receivers);
151
+
152
+ rrddim_set_by_pointer(st_memory, rd_buffers,
153
+ (collected_number)buffers);
154
+
155
+ rrddim_set_by_pointer(st_memory, rd_workers,
156
+ (collected_number) workers_allocated_memory());
157
+
158
+ rrddim_set_by_pointer(st_memory, rd_aral,
159
+ (collected_number) aral_by_size_structures());
160
+
161
+ rrddim_set_by_pointer(st_memory,
162
+ rd_judy, (collected_number) judy_aral_structures());
163
+
164
+ rrddim_set_by_pointer(st_memory,
165
+ rd_other, (collected_number)dictionary_stats_memory_total(dictionary_stats_category_other));
166
+
167
+ rrdset_done(st_memory);
168
+ }
169
+
170
+ {
171
+ static RRDSET *st_memory_buffers = NULL;
172
+ static RRDDIM *rd_queries = NULL;
173
+ static RRDDIM *rd_collectors = NULL;
174
+ static RRDDIM *rd_buffers_aclk = NULL;
175
+ static RRDDIM *rd_buffers_api = NULL;
176
+ static RRDDIM *rd_buffers_functions = NULL;
177
+ static RRDDIM *rd_buffers_sqlite = NULL;
178
+ static RRDDIM *rd_buffers_exporters = NULL;
179
+ static RRDDIM *rd_buffers_health = NULL;
180
+ static RRDDIM *rd_buffers_streaming = NULL;
181
+ static RRDDIM *rd_cbuffers_streaming = NULL;
182
+ static RRDDIM *rd_buffers_replication = NULL;
183
+ static RRDDIM *rd_buffers_web = NULL;
184
+ static RRDDIM *rd_buffers_aral = NULL;
185
+ static RRDDIM *rd_buffers_judy = NULL;
186
+
187
+ if (unlikely(!st_memory_buffers)) {
188
+ st_memory_buffers = rrdset_create_localhost(
189
+ "netdata",
190
+ "memory_buffers",
191
+ NULL,
192
+ "Memory Usage",
193
+ NULL,
194
+ "Netdata Memory Buffers",
195
+ "bytes",
196
+ "netdata",
197
+ "stats",
198
+ 130101,
199
+ localhost->rrd_update_every,
200
+ RRDSET_TYPE_STACKED);
201
+
202
+ rd_queries = rrddim_add(st_memory_buffers, "queries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
203
+ rd_collectors = rrddim_add(st_memory_buffers, "collection", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
204
+ rd_buffers_aclk = rrddim_add(st_memory_buffers, "aclk", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
205
+ rd_buffers_api = rrddim_add(st_memory_buffers, "api", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
206
+ rd_buffers_functions = rrddim_add(st_memory_buffers, "functions", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
207
+ rd_buffers_sqlite = rrddim_add(st_memory_buffers, "sqlite", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
208
+ rd_buffers_exporters = rrddim_add(st_memory_buffers, "exporters", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
209
+ rd_buffers_health = rrddim_add(st_memory_buffers, "health", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
210
+ rd_buffers_streaming = rrddim_add(st_memory_buffers, "streaming", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
211
+ rd_cbuffers_streaming = rrddim_add(st_memory_buffers, "streaming cbuf", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
212
+ rd_buffers_replication = rrddim_add(st_memory_buffers, "replication", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
213
+ rd_buffers_web = rrddim_add(st_memory_buffers, "web", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
214
+ rd_buffers_aral = rrddim_add(st_memory_buffers, "aral", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
215
+ rd_buffers_judy = rrddim_add(st_memory_buffers, "judy", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
216
+ }
217
+
218
+ rrddim_set_by_pointer(st_memory_buffers, rd_queries, (collected_number)netdata_buffers_statistics.query_targets_size + (collected_number) onewayalloc_allocated_memory());
219
+ rrddim_set_by_pointer(st_memory_buffers, rd_collectors, (collected_number)netdata_buffers_statistics.rrdset_done_rda_size);
220
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_aclk, (collected_number)netdata_buffers_statistics.buffers_aclk);
221
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_api, (collected_number)netdata_buffers_statistics.buffers_api);
222
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_functions, (collected_number)netdata_buffers_statistics.buffers_functions);
223
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_sqlite, (collected_number)netdata_buffers_statistics.buffers_sqlite);
224
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_exporters, (collected_number)netdata_buffers_statistics.buffers_exporters);
225
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_health, (collected_number)netdata_buffers_statistics.buffers_health);
226
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_streaming, (collected_number)netdata_buffers_statistics.buffers_streaming);
227
+ rrddim_set_by_pointer(st_memory_buffers, rd_cbuffers_streaming, (collected_number)netdata_buffers_statistics.cbuffers_streaming);
228
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_replication, (collected_number)replication_allocated_buffers());
229
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_web, (collected_number)netdata_buffers_statistics.buffers_web);
230
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_aral, (collected_number)aral_by_size_overhead());
231
+ rrddim_set_by_pointer(st_memory_buffers, rd_buffers_judy, (collected_number)judy_aral_overhead());
232
+
233
+ rrdset_done(st_memory_buffers);
234
+ }
235
+
236
+ // ----------------------------------------------------------------------------------------------------------------
237
+
238
+ if(!extended)
239
+ return;
240
+
241
+ // ----------------------------------------------------------------------------------------------------------------
242
+
243
+}
src/daemon/telemetry/telemetry-daemon-memory.h
new
+29
@@ -0,0 +1,29 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_DAEMON_MEMORY_H
4
+#define NETDATA_TELEMETRY_DAEMON_MEMORY_H
5
+
6
+#include "daemon/common.h"
7
+
8
+extern struct netdata_buffers_statistics {
9
+ size_t rrdhost_allocations_size;
10
+ size_t rrdhost_senders;
11
+ size_t rrdhost_receivers;
12
+ size_t query_targets_size;
13
+ size_t rrdset_done_rda_size;
14
+ size_t buffers_aclk;
15
+ size_t buffers_api;
16
+ size_t buffers_functions;
17
+ size_t buffers_sqlite;
18
+ size_t buffers_exporters;
19
+ size_t buffers_health;
20
+ size_t buffers_streaming;
21
+ size_t cbuffers_streaming;
22
+ size_t buffers_web;
23
+} netdata_buffers_statistics;
24
+
25
+#if defined(TELEMETRY_INTERNALS)
26
+void telemetry_daemon_memory_do(bool extended);
27
+#endif
28
+
29
+#endif //NETDATA_TELEMETRY_DAEMON_MEMORY_H
src/daemon/telemetry/telemetry-daemon.c
new
+79
@@ -0,0 +1,79 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-daemon.h"
5
+
6
+static void telemetry_daemon_cpu_usage_do(bool extended __maybe_unused) {
7
+ struct rusage me;
8
+ getrusage(RUSAGE_SELF, &me);
9
+
10
+ {
11
+ static RRDSET *st_cpu = NULL;
12
+ static RRDDIM *rd_cpu_user = NULL,
13
+ *rd_cpu_system = NULL;
14
+
15
+ if (unlikely(!st_cpu)) {
16
+ st_cpu = rrdset_create_localhost(
17
+ "netdata"
18
+ , "server_cpu"
19
+ , NULL
20
+ , "CPU usage"
21
+ , NULL
22
+ , "Netdata CPU usage"
23
+ , "milliseconds/s"
24
+ , "netdata"
25
+ , "stats"
26
+ , 130000
27
+ , localhost->rrd_update_every
28
+ , RRDSET_TYPE_STACKED
29
+ );
30
+
31
+ rd_cpu_user = rrddim_add(st_cpu, "user", NULL, 1, 1000, RRD_ALGORITHM_INCREMENTAL);
32
+ rd_cpu_system = rrddim_add(st_cpu, "system", NULL, 1, 1000, RRD_ALGORITHM_INCREMENTAL);
33
+ }
34
+
35
+ rrddim_set_by_pointer(st_cpu, rd_cpu_user, (collected_number )(me.ru_utime.tv_sec * 1000000ULL + me.ru_utime.tv_usec));
36
+ rrddim_set_by_pointer(st_cpu, rd_cpu_system, (collected_number )(me.ru_stime.tv_sec * 1000000ULL + me.ru_stime.tv_usec));
37
+ rrdset_done(st_cpu);
38
+ }
39
+}
40
+
41
+static void telemetry_daemon_uptime_do(bool extended __maybe_unused) {
42
+ {
43
+ static time_t netdata_boottime_time = 0;
44
+ if (!netdata_boottime_time)
45
+ netdata_boottime_time = now_boottime_sec();
46
+
47
+ time_t netdata_uptime = now_boottime_sec() - netdata_boottime_time;
48
+
49
+ static RRDSET *st_uptime = NULL;
50
+ static RRDDIM *rd_uptime = NULL;
51
+
52
+ if (unlikely(!st_uptime)) {
53
+ st_uptime = rrdset_create_localhost(
54
+ "netdata",
55
+ "uptime",
56
+ NULL,
57
+ "Uptime",
58
+ NULL,
59
+ "Netdata uptime",
60
+ "seconds",
61
+ "netdata",
62
+ "stats",
63
+ 130150,
64
+ localhost->rrd_update_every,
65
+ RRDSET_TYPE_LINE);
66
+
67
+ rd_uptime = rrddim_add(st_uptime, "uptime", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
68
+ }
69
+
70
+ rrddim_set_by_pointer(st_uptime, rd_uptime, netdata_uptime);
71
+ rrdset_done(st_uptime);
72
+ }
73
+}
74
+
75
+void telemetry_daemon_do(bool extended) {
76
+ telemetry_daemon_cpu_usage_do(extended);
77
+ telemetry_daemon_uptime_do(extended);
78
+ telemetry_daemon_memory_do(extended);
79
+}
src/daemon/telemetry/telemetry-daemon.h
new
+12
@@ -0,0 +1,12 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_DAEMON_H
4
+#define NETDATA_TELEMETRY_DAEMON_H
5
+
6
+#include "daemon/common.h"
7
+
8
+#if defined(TELEMETRY_INTERNALS)
9
+void telemetry_daemon_do(bool extended);
10
+#endif
11
+
12
+#endif //NETDATA_TELEMETRY_DAEMON_H
src/daemon/telemetry/telemetry-dbengine.c
new
+1624
@@ -0,0 +1,1624 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-dbengine.h"
5
+
6
+size_t telemetry_dbengine_total_memory = 0;
7
+
8
+#if defined(ENABLE_DBENGINE)
9
+
10
+struct dbengine2_cache_pointers {
11
+ RRDSET *st_cache_hit_ratio;
12
+ RRDDIM *rd_hit_ratio_closest;
13
+ RRDDIM *rd_hit_ratio_exact;
14
+
15
+ RRDSET *st_operations;
16
+ RRDDIM *rd_searches_closest;
17
+ RRDDIM *rd_searches_exact;
18
+ RRDDIM *rd_add_hot;
19
+ RRDDIM *rd_add_clean;
20
+ RRDDIM *rd_evictions;
21
+ RRDDIM *rd_flushes;
22
+ RRDDIM *rd_acquires;
23
+ RRDDIM *rd_releases;
24
+ RRDDIM *rd_acquires_for_deletion;
25
+
26
+ RRDSET *st_pgc_memory;
27
+ RRDDIM *rd_pgc_memory_free;
28
+ RRDDIM *rd_pgc_memory_clean;
29
+ RRDDIM *rd_pgc_memory_hot;
30
+ RRDDIM *rd_pgc_memory_dirty;
31
+ RRDDIM *rd_pgc_memory_index;
32
+ RRDDIM *rd_pgc_memory_evicting;
33
+ RRDDIM *rd_pgc_memory_flushing;
34
+
35
+ struct {
36
+ RRDSET *st_pgc_page_size_heatmap;
37
+ RRDDIM *rd_pgc_page_size_x[PGC_SIZE_HISTOGRAM_ENTRIES];
38
+ } queues[3];
39
+
40
+ RRDSET *st_pgc_tm;
41
+ RRDDIM *rd_pgc_tm_current;
42
+ RRDDIM *rd_pgc_tm_wanted;
43
+ RRDDIM *rd_pgc_tm_hot_max;
44
+ RRDDIM *rd_pgc_tm_dirty_max;
45
+ RRDDIM *rd_pgc_tm_hot;
46
+ RRDDIM *rd_pgc_tm_dirty;
47
+ RRDDIM *rd_pgc_tm_referenced;
48
+
49
+ RRDSET *st_pgc_pages;
50
+ RRDDIM *rd_pgc_pages_clean;
51
+ RRDDIM *rd_pgc_pages_hot;
52
+ RRDDIM *rd_pgc_pages_dirty;
53
+ RRDDIM *rd_pgc_pages_referenced;
54
+
55
+ RRDSET *st_pgc_memory_changes;
56
+ RRDDIM *rd_pgc_memory_new_hot;
57
+ RRDDIM *rd_pgc_memory_new_clean;
58
+ RRDDIM *rd_pgc_memory_clean_evictions;
59
+
60
+ RRDSET *st_pgc_memory_migrations;
61
+ RRDDIM *rd_pgc_memory_hot_to_dirty;
62
+ RRDDIM *rd_pgc_memory_dirty_to_clean;
63
+
64
+ RRDSET *st_pgc_workers;
65
+ RRDDIM *rd_pgc_workers_evictors;
66
+ RRDDIM *rd_pgc_workers_flushers;
67
+ RRDDIM *rd_pgc_workers_adders;
68
+ RRDDIM *rd_pgc_workers_searchers;
69
+ RRDDIM *rd_pgc_workers_jv2_flushers;
70
+ RRDDIM *rd_pgc_workers_hot2dirty;
71
+
72
+ RRDSET *st_pgc_memory_events;
73
+ RRDDIM *rd_pgc_memory_evictions_critical;
74
+ RRDDIM *rd_pgc_memory_evictions_aggressive;
75
+ RRDDIM *rd_pgc_memory_flushes_critical;
76
+ RRDDIM *rd_pgc_waste_evict_thread_signals;
77
+ RRDDIM *rd_pgc_waste_evict_inline_on_add;
78
+ RRDDIM *rd_pgc_waste_evict_inline_on_release;
79
+ RRDDIM *rd_pgc_waste_flush_inline_on_add;
80
+ RRDDIM *rd_pgc_waste_flush_inline_on_release;
81
+
82
+ RRDSET *st_pgc_waste;
83
+ RRDDIM *rd_pgc_waste_evict_relocated;
84
+ RRDDIM *rd_pgc_waste_flushes_cancelled;
85
+ RRDDIM *rd_pgc_waste_insert_spins;
86
+ RRDDIM *rd_pgc_waste_evict_spins;
87
+ RRDDIM *rd_pgc_waste_release_spins;
88
+ RRDDIM *rd_pgc_waste_acquire_spins;
89
+ RRDDIM *rd_pgc_waste_delete_spins;
90
+};
91
+
92
+static void dbengine2_cache_statistics_charts(struct dbengine2_cache_pointers *ptrs, struct pgc_statistics *pgc_stats, struct pgc_statistics *pgc_stats_old __maybe_unused, const char *name, int priority) {
93
+
94
+ {
95
+ if (unlikely(!ptrs->st_cache_hit_ratio)) {
96
+ BUFFER *id = buffer_create(100, NULL);
97
+ buffer_sprintf(id, "dbengine_%s_cache_hit_ratio", name);
98
+
99
+ BUFFER *family = buffer_create(100, NULL);
100
+ buffer_sprintf(family, "dbengine %s cache", name);
101
+
102
+ BUFFER *title = buffer_create(100, NULL);
103
+ buffer_sprintf(title, "Netdata %s Cache Hit Ratio", name);
104
+
105
+ ptrs->st_cache_hit_ratio = rrdset_create_localhost(
106
+ "netdata",
107
+ buffer_tostring(id),
108
+ NULL,
109
+ buffer_tostring(family),
110
+ NULL,
111
+ buffer_tostring(title),
112
+ "%",
113
+ "netdata",
114
+ "stats",
115
+ priority,
116
+ localhost->rrd_update_every,
117
+ RRDSET_TYPE_LINE);
118
+
119
+ ptrs->rd_hit_ratio_closest = rrddim_add(ptrs->st_cache_hit_ratio, "closest", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
120
+ ptrs->rd_hit_ratio_exact = rrddim_add(ptrs->st_cache_hit_ratio, "exact", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
121
+
122
+ buffer_free(id);
123
+ buffer_free(family);
124
+ buffer_free(title);
125
+ priority++;
126
+ }
127
+
128
+ size_t closest_percent = 100 * 10000;
129
+ if(pgc_stats->searches_closest > pgc_stats_old->searches_closest)
130
+ closest_percent = (pgc_stats->searches_closest_hits - pgc_stats_old->searches_closest_hits) * 100 * 10000 / (pgc_stats->searches_closest - pgc_stats_old->searches_closest);
131
+
132
+ size_t exact_percent = 100 * 10000;
133
+ if(pgc_stats->searches_exact > pgc_stats_old->searches_exact)
134
+ exact_percent = (pgc_stats->searches_exact_hits - pgc_stats_old->searches_exact_hits) * 100 * 10000 / (pgc_stats->searches_exact - pgc_stats_old->searches_exact);
135
+
136
+ rrddim_set_by_pointer(ptrs->st_cache_hit_ratio, ptrs->rd_hit_ratio_closest, (collected_number)closest_percent);
137
+ rrddim_set_by_pointer(ptrs->st_cache_hit_ratio, ptrs->rd_hit_ratio_exact, (collected_number)exact_percent);
138
+
139
+ rrdset_done(ptrs->st_cache_hit_ratio);
140
+ }
141
+
142
+ {
143
+ if (unlikely(!ptrs->st_operations)) {
144
+ BUFFER *id = buffer_create(100, NULL);
145
+ buffer_sprintf(id, "dbengine_%s_cache_operations", name);
146
+
147
+ BUFFER *family = buffer_create(100, NULL);
148
+ buffer_sprintf(family, "dbengine %s cache", name);
149
+
150
+ BUFFER *title = buffer_create(100, NULL);
151
+ buffer_sprintf(title, "Netdata %s Cache Operations", name);
152
+
153
+ ptrs->st_operations = rrdset_create_localhost(
154
+ "netdata",
155
+ buffer_tostring(id),
156
+ NULL,
157
+ buffer_tostring(family),
158
+ NULL,
159
+ buffer_tostring(title),
160
+ "ops/s",
161
+ "netdata",
162
+ "stats",
163
+ priority,
164
+ localhost->rrd_update_every,
165
+ RRDSET_TYPE_LINE);
166
+
167
+ ptrs->rd_searches_closest = rrddim_add(ptrs->st_operations, "search closest", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
168
+ ptrs->rd_searches_exact = rrddim_add(ptrs->st_operations, "search exact", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
169
+ ptrs->rd_add_hot = rrddim_add(ptrs->st_operations, "add hot", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
170
+ ptrs->rd_add_clean = rrddim_add(ptrs->st_operations, "add clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
171
+ ptrs->rd_evictions = rrddim_add(ptrs->st_operations, "evictions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
172
+ ptrs->rd_flushes = rrddim_add(ptrs->st_operations, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
173
+ ptrs->rd_acquires = rrddim_add(ptrs->st_operations, "acquires", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
174
+ ptrs->rd_releases = rrddim_add(ptrs->st_operations, "releases", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
175
+ ptrs->rd_acquires_for_deletion = rrddim_add(ptrs->st_operations, "del acquires", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
176
+
177
+ buffer_free(id);
178
+ buffer_free(family);
179
+ buffer_free(title);
180
+ priority++;
181
+ }
182
+
183
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_searches_closest, (collected_number)pgc_stats->searches_closest);
184
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_searches_exact, (collected_number)pgc_stats->searches_exact);
185
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_add_hot, (collected_number)pgc_stats->queues[PGC_QUEUE_HOT].added_entries);
186
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_add_clean, (collected_number)(pgc_stats->added_entries - pgc_stats->queues[PGC_QUEUE_HOT].added_entries));
187
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_evictions, (collected_number)pgc_stats->queues[PGC_QUEUE_CLEAN].removed_entries);
188
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_flushes, (collected_number)pgc_stats->flushes_completed);
189
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_acquires, (collected_number)pgc_stats->acquires);
190
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_releases, (collected_number)pgc_stats->releases);
191
+ rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_acquires_for_deletion, (collected_number)pgc_stats->acquires_for_deletion);
192
+
193
+ rrdset_done(ptrs->st_operations);
194
+ }
195
+
196
+ {
197
+ if (unlikely(!ptrs->st_pgc_memory)) {
198
+ BUFFER *id = buffer_create(100, NULL);
199
+ buffer_sprintf(id, "dbengine_%s_cache_memory", name);
200
+
201
+ BUFFER *family = buffer_create(100, NULL);
202
+ buffer_sprintf(family, "dbengine %s cache", name);
203
+
204
+ BUFFER *title = buffer_create(100, NULL);
205
+ buffer_sprintf(title, "Netdata %s Cache Memory", name);
206
+
207
+ ptrs->st_pgc_memory = rrdset_create_localhost(
208
+ "netdata",
209
+ buffer_tostring(id),
210
+ NULL,
211
+ buffer_tostring(family),
212
+ NULL,
213
+ buffer_tostring(title),
214
+ "bytes",
215
+ "netdata",
216
+ "stats",
217
+ priority,
218
+ localhost->rrd_update_every,
219
+ RRDSET_TYPE_STACKED);
220
+
221
+ ptrs->rd_pgc_memory_free = rrddim_add(ptrs->st_pgc_memory, "free", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
222
+ ptrs->rd_pgc_memory_hot = rrddim_add(ptrs->st_pgc_memory, "hot", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
223
+ ptrs->rd_pgc_memory_dirty = rrddim_add(ptrs->st_pgc_memory, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
224
+ ptrs->rd_pgc_memory_clean = rrddim_add(ptrs->st_pgc_memory, "clean", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
225
+ ptrs->rd_pgc_memory_index = rrddim_add(ptrs->st_pgc_memory, "index", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
226
+ ptrs->rd_pgc_memory_evicting = rrddim_add(ptrs->st_pgc_memory, "evicting", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
227
+ ptrs->rd_pgc_memory_flushing = rrddim_add(ptrs->st_pgc_memory, "flushing", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
228
+
229
+ buffer_free(id);
230
+ buffer_free(family);
231
+ buffer_free(title);
232
+ priority++;
233
+ }
234
+
235
+ collected_number free = (pgc_stats->current_cache_size > pgc_stats->wanted_cache_size) ? 0 :
236
+ (collected_number)(pgc_stats->wanted_cache_size - pgc_stats->current_cache_size);
237
+
238
+ rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_free, free);
239
+ rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_hot, (collected_number)pgc_stats->queues[PGC_QUEUE_HOT].size);
240
+ rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_dirty, (collected_number)pgc_stats->queues[PGC_QUEUE_DIRTY].size);
241
+ rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_clean, (collected_number)pgc_stats->queues[PGC_QUEUE_CLEAN].size);
242
+ rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_evicting, (collected_number)pgc_stats->evicting_size);
243
+ rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_flushing, (collected_number)pgc_stats->flushing_size);
244
+ rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_index,(collected_number)(pgc_stats->size - pgc_stats->queues[PGC_QUEUE_CLEAN].size - pgc_stats->queues[PGC_QUEUE_HOT].size - pgc_stats->queues[PGC_QUEUE_DIRTY].size - pgc_stats->evicting_size - pgc_stats->flushing_size));
245
+
246
+ rrdset_done(ptrs->st_pgc_memory);
247
+ }
248
+
249
+ for(size_t q = 0; q < 3 ;q++) {
250
+ const char *queue;
251
+ switch(q) {
252
+ case PGC_QUEUE_HOT:
253
+ queue = "hot";
254
+ break;
255
+
256
+ case PGC_QUEUE_DIRTY:
257
+ queue = "dirty";
258
+ break;
259
+
260
+ default:
261
+ case PGC_QUEUE_CLEAN:
262
+ queue = "clean";
263
+ break;
264
+ }
265
+
266
+ if (unlikely(!ptrs->queues[q].st_pgc_page_size_heatmap)) {
267
+ CLEAN_BUFFER *ctx = buffer_create(100, NULL);
268
+ buffer_sprintf(ctx, "netdata.dbengine_%s_page_sizes", name);
269
+
270
+ CLEAN_BUFFER *id = buffer_create(100, NULL);
271
+ buffer_sprintf(id, "dbengine_%s_%s_page_sizes", name, queue);
272
+
273
+ CLEAN_BUFFER *family = buffer_create(100, NULL);
274
+ buffer_sprintf(family, "dbengine %s cache", name);
275
+
276
+ CLEAN_BUFFER *title = buffer_create(100, NULL);
277
+ buffer_sprintf(title, "Netdata %s Nominal Page Sizes (without overheads)", name);
278
+
279
+ ptrs->queues[q].st_pgc_page_size_heatmap = rrdset_create_localhost(
280
+ "netdata",
281
+ buffer_tostring(id),
282
+ NULL,
283
+ buffer_tostring(family),
284
+ buffer_tostring(ctx),
285
+ buffer_tostring(title),
286
+ "pages",
287
+ "netdata",
288
+ "stats",
289
+ priority,
290
+ localhost->rrd_update_every,
291
+ RRDSET_TYPE_HEATMAP);
292
+
293
+ ptrs->queues[q].rd_pgc_page_size_x[0] = rrddim_add(ptrs->queues[q].st_pgc_page_size_heatmap, "empty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
294
+ for(size_t i = 1; i < _countof(ptrs->queues[q].rd_pgc_page_size_x) - 1 ;i++) {
295
+ char buf[64];
296
+ snprintfz(buf, sizeof(buf), "%zu", pgc_stats->queues[q].size_histogram.array[i].upto);
297
+ // size_snprintf(&buf[1], sizeof(buf) - 1, pgc_stats->size_histogram.array[i].upto, "B", true);
298
+ ptrs->queues[q].rd_pgc_page_size_x[i] = rrddim_add(ptrs->queues[q].st_pgc_page_size_heatmap, buf, NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
299
+ }
300
+ ptrs->queues[q].rd_pgc_page_size_x[_countof(ptrs->queues[q].rd_pgc_page_size_x) - 1] = rrddim_add(ptrs->queues[q].st_pgc_page_size_heatmap, "+inf", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
301
+
302
+ rrdlabels_add(ptrs->queues[q].st_pgc_page_size_heatmap->rrdlabels, "Cache", name, RRDLABEL_SRC_AUTO);
303
+ rrdlabels_add(ptrs->queues[q].st_pgc_page_size_heatmap->rrdlabels, "Queue", queue, RRDLABEL_SRC_AUTO);
304
+
305
+ priority++;
306
+ }
307
+
308
+ for(size_t i = 0; i < _countof(ptrs->queues[q].rd_pgc_page_size_x) - 1 ;i++)
309
+ rrddim_set_by_pointer(ptrs->queues[q].st_pgc_page_size_heatmap, ptrs->queues[q].rd_pgc_page_size_x[i], (collected_number)pgc_stats->queues[q].size_histogram.array[i].count);
310
+
311
+ rrdset_done(ptrs->queues[q].st_pgc_page_size_heatmap);
312
+ }
313
+
314
+ {
315
+ if (unlikely(!ptrs->st_pgc_tm)) {
316
+ BUFFER *id = buffer_create(100, NULL);
317
+ buffer_sprintf(id, "dbengine_%s_target_memory", name);
318
+
319
+ BUFFER *family = buffer_create(100, NULL);
320
+ buffer_sprintf(family, "dbengine %s cache", name);
321
+
322
+ BUFFER *title = buffer_create(100, NULL);
323
+ buffer_sprintf(title, "Netdata %s Target Cache Memory", name);
324
+
325
+ ptrs->st_pgc_tm = rrdset_create_localhost(
326
+ "netdata",
327
+ buffer_tostring(id),
328
+ NULL,
329
+ buffer_tostring(family),
330
+ NULL,
331
+ buffer_tostring(title),
332
+ "bytes",
333
+ "netdata",
334
+ "stats",
335
+ priority,
336
+ localhost->rrd_update_every,
337
+ RRDSET_TYPE_LINE);
338
+
339
+ ptrs->rd_pgc_tm_current = rrddim_add(ptrs->st_pgc_tm, "current", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
340
+ ptrs->rd_pgc_tm_wanted = rrddim_add(ptrs->st_pgc_tm, "wanted", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
341
+ ptrs->rd_pgc_tm_referenced = rrddim_add(ptrs->st_pgc_tm, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
342
+ ptrs->rd_pgc_tm_hot_max = rrddim_add(ptrs->st_pgc_tm, "hot max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
343
+ ptrs->rd_pgc_tm_dirty_max = rrddim_add(ptrs->st_pgc_tm, "dirty max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
344
+ ptrs->rd_pgc_tm_hot = rrddim_add(ptrs->st_pgc_tm, "hot", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
345
+ ptrs->rd_pgc_tm_dirty = rrddim_add(ptrs->st_pgc_tm, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
346
+
347
+ buffer_free(id);
348
+ buffer_free(family);
349
+ buffer_free(title);
350
+ priority++;
351
+ }
352
+
353
+ rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_current, (collected_number)pgc_stats->current_cache_size);
354
+ rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_wanted, (collected_number)pgc_stats->wanted_cache_size);
355
+ rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_referenced, (collected_number)pgc_stats->referenced_size);
356
+ rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_hot_max, (collected_number)pgc_stats->queues[PGC_QUEUE_HOT].max_size);
357
+ rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_dirty_max, (collected_number)pgc_stats->queues[PGC_QUEUE_DIRTY].max_size);
358
+ rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_hot, (collected_number)pgc_stats->queues[PGC_QUEUE_HOT].size);
359
+ rrddim_set_by_pointer(ptrs->st_pgc_tm, ptrs->rd_pgc_tm_dirty, (collected_number)pgc_stats->queues[PGC_QUEUE_DIRTY].size);
360
+
361
+ rrdset_done(ptrs->st_pgc_tm);
362
+ }
363
+
364
+ {
365
+ if (unlikely(!ptrs->st_pgc_pages)) {
366
+ BUFFER *id = buffer_create(100, NULL);
367
+ buffer_sprintf(id, "dbengine_%s_cache_pages", name);
368
+
369
+ BUFFER *family = buffer_create(100, NULL);
370
+ buffer_sprintf(family, "dbengine %s cache", name);
371
+
372
+ BUFFER *title = buffer_create(100, NULL);
373
+ buffer_sprintf(title, "Netdata %s Cache Pages", name);
374
+
375
+ ptrs->st_pgc_pages = rrdset_create_localhost(
376
+ "netdata",
377
+ buffer_tostring(id),
378
+ NULL,
379
+ buffer_tostring(family),
380
+ NULL,
381
+ buffer_tostring(title),
382
+ "pages",
383
+ "netdata",
384
+ "stats",
385
+ priority,
386
+ localhost->rrd_update_every,
387
+ RRDSET_TYPE_LINE);
388
+
389
+ ptrs->rd_pgc_pages_clean = rrddim_add(ptrs->st_pgc_pages, "clean", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
390
+ ptrs->rd_pgc_pages_hot = rrddim_add(ptrs->st_pgc_pages, "hot", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
391
+ ptrs->rd_pgc_pages_dirty = rrddim_add(ptrs->st_pgc_pages, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
392
+ ptrs->rd_pgc_pages_referenced = rrddim_add(ptrs->st_pgc_pages, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
393
+
394
+ buffer_free(id);
395
+ buffer_free(family);
396
+ buffer_free(title);
397
+ priority++;
398
+ }
399
+
400
+ rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_clean, (collected_number)pgc_stats->queues[PGC_QUEUE_CLEAN].entries);
401
+ rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_hot, (collected_number)pgc_stats->queues[PGC_QUEUE_HOT].entries);
402
+ rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_dirty, (collected_number)pgc_stats->queues[PGC_QUEUE_DIRTY].entries);
403
+ rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_referenced, (collected_number)pgc_stats->referenced_entries);
404
+
405
+ rrdset_done(ptrs->st_pgc_pages);
406
+ }
407
+
408
+ {
409
+ if (unlikely(!ptrs->st_pgc_memory_changes)) {
410
+ BUFFER *id = buffer_create(100, NULL);
411
+ buffer_sprintf(id, "dbengine_%s_cache_memory_changes", name);
412
+
413
+ BUFFER *family = buffer_create(100, NULL);
414
+ buffer_sprintf(family, "dbengine %s cache", name);
415
+
416
+ BUFFER *title = buffer_create(100, NULL);
417
+ buffer_sprintf(title, "Netdata %s Cache Memory Changes", name);
418
+
419
+ ptrs->st_pgc_memory_changes = rrdset_create_localhost(
420
+ "netdata",
421
+ buffer_tostring(id),
422
+ NULL,
423
+ buffer_tostring(family),
424
+ NULL,
425
+ buffer_tostring(title),
426
+ "bytes/s",
427
+ "netdata",
428
+ "stats",
429
+ priority,
430
+ localhost->rrd_update_every,
431
+ RRDSET_TYPE_AREA);
432
+
433
+ ptrs->rd_pgc_memory_new_clean = rrddim_add(ptrs->st_pgc_memory_changes, "new clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
434
+ ptrs->rd_pgc_memory_clean_evictions = rrddim_add(ptrs->st_pgc_memory_changes, "evictions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
435
+ ptrs->rd_pgc_memory_new_hot = rrddim_add(ptrs->st_pgc_memory_changes, "new hot", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
436
+
437
+ buffer_free(id);
438
+ buffer_free(family);
439
+ buffer_free(title);
440
+ priority++;
441
+ }
442
+
443
+ rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_new_clean, (collected_number)(pgc_stats->added_size - pgc_stats->queues[PGC_QUEUE_HOT].added_size));
444
+ rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_clean_evictions, (collected_number)pgc_stats->queues[PGC_QUEUE_CLEAN].removed_size);
445
+ rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_new_hot, (collected_number)pgc_stats->queues[PGC_QUEUE_HOT].added_size);
446
+
447
+ rrdset_done(ptrs->st_pgc_memory_changes);
448
+ }
449
+
450
+ {
451
+ if (unlikely(!ptrs->st_pgc_memory_migrations)) {
452
+ BUFFER *id = buffer_create(100, NULL);
453
+ buffer_sprintf(id, "dbengine_%s_cache_memory_migrations", name);
454
+
455
+ BUFFER *family = buffer_create(100, NULL);
456
+ buffer_sprintf(family, "dbengine %s cache", name);
457
+
458
+ BUFFER *title = buffer_create(100, NULL);
459
+ buffer_sprintf(title, "Netdata %s Cache Memory Migrations", name);
460
+
461
+ ptrs->st_pgc_memory_migrations = rrdset_create_localhost(
462
+ "netdata",
463
+ buffer_tostring(id),
464
+ NULL,
465
+ buffer_tostring(family),
466
+ NULL,
467
+ buffer_tostring(title),
468
+ "bytes/s",
469
+ "netdata",
470
+ "stats",
471
+ priority,
472
+ localhost->rrd_update_every,
473
+ RRDSET_TYPE_AREA);
474
+
475
+ ptrs->rd_pgc_memory_dirty_to_clean = rrddim_add(ptrs->st_pgc_memory_migrations, "dirty to clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
476
+ ptrs->rd_pgc_memory_hot_to_dirty = rrddim_add(ptrs->st_pgc_memory_migrations, "hot to dirty", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
477
+
478
+ buffer_free(id);
479
+ buffer_free(family);
480
+ buffer_free(title);
481
+ priority++;
482
+ }
483
+
484
+ rrddim_set_by_pointer(ptrs->st_pgc_memory_migrations, ptrs->rd_pgc_memory_dirty_to_clean, (collected_number)pgc_stats->queues[PGC_QUEUE_DIRTY].removed_size);
485
+ rrddim_set_by_pointer(ptrs->st_pgc_memory_migrations, ptrs->rd_pgc_memory_hot_to_dirty, (collected_number)pgc_stats->queues[PGC_QUEUE_DIRTY].added_size);
486
+
487
+ rrdset_done(ptrs->st_pgc_memory_migrations);
488
+ }
489
+
490
+ {
491
+ if (unlikely(!ptrs->st_pgc_memory_events)) {
492
+ BUFFER *id = buffer_create(100, NULL);
493
+ buffer_sprintf(id, "dbengine_%s_cache_events", name);
494
+
495
+ BUFFER *family = buffer_create(100, NULL);
496
+ buffer_sprintf(family, "dbengine %s cache", name);
497
+
498
+ BUFFER *title = buffer_create(100, NULL);
499
+ buffer_sprintf(title, "Netdata %s Cache Events", name);
500
+
501
+ ptrs->st_pgc_memory_events = rrdset_create_localhost(
502
+ "netdata",
503
+ buffer_tostring(id),
504
+ NULL,
505
+ buffer_tostring(family),
506
+ NULL,
507
+ buffer_tostring(title),
508
+ "events/s",
509
+ "netdata",
510
+ "stats",
511
+ priority,
512
+ localhost->rrd_update_every,
513
+ RRDSET_TYPE_AREA);
514
+
515
+ ptrs->rd_pgc_memory_evictions_aggressive = rrddim_add(ptrs->st_pgc_memory_events, "evictions aggressive", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
516
+ ptrs->rd_pgc_memory_evictions_critical = rrddim_add(ptrs->st_pgc_memory_events, "evictions critical", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
517
+ ptrs->rd_pgc_memory_flushes_critical = rrddim_add(ptrs->st_pgc_memory_events, "flushes critical", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
518
+
519
+ buffer_free(id);
520
+ buffer_free(family);
521
+ buffer_free(title);
522
+ priority++;
523
+ }
524
+
525
+ rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_evictions_aggressive, (collected_number)pgc_stats->events_cache_needs_space_aggressively);
526
+ rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_evictions_critical, (collected_number)pgc_stats->events_cache_under_severe_pressure);
527
+ rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_flushes_critical, (collected_number)pgc_stats->events_flush_critical);
528
+
529
+ rrdset_done(ptrs->st_pgc_memory_events);
530
+ }
531
+
532
+ {
533
+ if (unlikely(!ptrs->st_pgc_waste)) {
534
+ BUFFER *id = buffer_create(100, NULL);
535
+ buffer_sprintf(id, "dbengine_%s_waste_events", name);
536
+
537
+ BUFFER *family = buffer_create(100, NULL);
538
+ buffer_sprintf(family, "dbengine %s cache", name);
539
+
540
+ BUFFER *title = buffer_create(100, NULL);
541
+ buffer_sprintf(title, "Netdata %s Waste Events", name);
542
+
543
+ ptrs->st_pgc_waste = rrdset_create_localhost(
544
+ "netdata",
545
+ buffer_tostring(id),
546
+ NULL,
547
+ buffer_tostring(family),
548
+ NULL,
549
+ buffer_tostring(title),
550
+ "events/s",
551
+ "netdata",
552
+ "stats",
553
+ priority,
554
+ localhost->rrd_update_every,
555
+ RRDSET_TYPE_LINE);
556
+
557
+ ptrs->rd_pgc_waste_evict_relocated = rrddim_add(ptrs->st_pgc_waste, "evict relocated", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
558
+ ptrs->rd_pgc_waste_flushes_cancelled = rrddim_add(ptrs->st_pgc_waste, "flushes cancelled", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
559
+ ptrs->rd_pgc_waste_acquire_spins = rrddim_add(ptrs->st_pgc_waste, "acquire spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
560
+ ptrs->rd_pgc_waste_release_spins = rrddim_add(ptrs->st_pgc_waste, "release spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
561
+ ptrs->rd_pgc_waste_insert_spins = rrddim_add(ptrs->st_pgc_waste, "insert spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
562
+ ptrs->rd_pgc_waste_delete_spins = rrddim_add(ptrs->st_pgc_waste, "delete spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
563
+ ptrs->rd_pgc_waste_evict_spins = rrddim_add(ptrs->st_pgc_waste, "evict useless spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
564
+ ptrs->rd_pgc_waste_evict_thread_signals = rrddim_add(ptrs->st_pgc_waste, "evict thread signals", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
565
+ ptrs->rd_pgc_waste_evict_inline_on_add = rrddim_add(ptrs->st_pgc_waste, "evict inline on add", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
566
+ ptrs->rd_pgc_waste_evict_inline_on_release = rrddim_add(ptrs->st_pgc_waste, "evict inline on rel", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
567
+ ptrs->rd_pgc_waste_flush_inline_on_add = rrddim_add(ptrs->st_pgc_waste, "flush inline on add", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
568
+ ptrs->rd_pgc_waste_flush_inline_on_release = rrddim_add(ptrs->st_pgc_waste, "flush inline on rel", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
569
+
570
+ buffer_free(id);
571
+ buffer_free(family);
572
+ buffer_free(title);
573
+ priority++;
574
+ }
575
+
576
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evict_relocated, (collected_number)pgc_stats->waste_evict_relocated);
577
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_flushes_cancelled, (collected_number)pgc_stats->waste_flushes_cancelled);
578
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_acquire_spins, (collected_number)pgc_stats->waste_acquire_spins);
579
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_release_spins, (collected_number)pgc_stats->waste_release_spins);
580
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_insert_spins, (collected_number)pgc_stats->waste_insert_spins);
581
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_delete_spins, (collected_number)pgc_stats->waste_delete_spins);
582
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evict_spins, (collected_number)pgc_stats->waste_evict_useless_spins);
583
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evict_thread_signals, (collected_number)pgc_stats->waste_evict_thread_signals);
584
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evict_inline_on_add, (collected_number)pgc_stats->waste_evictions_inline_on_add);
585
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evict_inline_on_release, (collected_number)pgc_stats->waste_evictions_inline_on_release);
586
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_flush_inline_on_add, (collected_number)pgc_stats->waste_flush_on_add);
587
+ rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_flush_inline_on_release, (collected_number)pgc_stats->waste_flush_on_release);
588
+
589
+ rrdset_done(ptrs->st_pgc_waste);
590
+ }
591
+
592
+ {
593
+ if (unlikely(!ptrs->st_pgc_workers)) {
594
+ BUFFER *id = buffer_create(100, NULL);
595
+ buffer_sprintf(id, "dbengine_%s_cache_workers", name);
596
+
597
+ BUFFER *family = buffer_create(100, NULL);
598
+ buffer_sprintf(family, "dbengine %s cache", name);
599
+
600
+ BUFFER *title = buffer_create(100, NULL);
601
+ buffer_sprintf(title, "Netdata %s Cache Workers", name);
602
+
603
+ ptrs->st_pgc_workers = rrdset_create_localhost(
604
+ "netdata",
605
+ buffer_tostring(id),
606
+ NULL,
607
+ buffer_tostring(family),
608
+ NULL,
609
+ buffer_tostring(title),
610
+ "workers",
611
+ "netdata",
612
+ "stats",
613
+ priority,
614
+ localhost->rrd_update_every,
615
+ RRDSET_TYPE_LINE);
616
+
617
+ ptrs->rd_pgc_workers_searchers = rrddim_add(ptrs->st_pgc_workers, "searchers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
618
+ ptrs->rd_pgc_workers_adders = rrddim_add(ptrs->st_pgc_workers, "adders", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
619
+ ptrs->rd_pgc_workers_evictors = rrddim_add(ptrs->st_pgc_workers, "evictors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
620
+ ptrs->rd_pgc_workers_flushers = rrddim_add(ptrs->st_pgc_workers, "flushers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
621
+ ptrs->rd_pgc_workers_hot2dirty = rrddim_add(ptrs->st_pgc_workers, "hot2dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
622
+ ptrs->rd_pgc_workers_jv2_flushers = rrddim_add(ptrs->st_pgc_workers, "jv2 flushers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
623
+
624
+ buffer_free(id);
625
+ buffer_free(family);
626
+ buffer_free(title);
627
+ priority++;
628
+ }
629
+
630
+ rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_searchers, (collected_number)pgc_stats->workers_search);
631
+ rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_adders, (collected_number)pgc_stats->workers_add);
632
+ rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_evictors, (collected_number)pgc_stats->workers_evict);
633
+ rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_flushers, (collected_number)pgc_stats->workers_flush);
634
+ rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_hot2dirty, (collected_number)pgc_stats->workers_hot2dirty);
635
+ rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_jv2_flushers, (collected_number)pgc_stats->workers_jv2_flush);
636
+
637
+ rrdset_done(ptrs->st_pgc_workers);
638
+ }
639
+}
640
+
641
+
642
+void telemetry_dbengine_do(bool extended) {
643
+ if(!main_cache || !main_mrg || !extended)
644
+ return;
645
+
646
+ static struct dbengine2_cache_pointers main_cache_ptrs = {}, open_cache_ptrs = {}, extent_cache_ptrs = {};
647
+ static struct rrdeng_cache_efficiency_stats cache_efficiency_stats = {}, cache_efficiency_stats_old = {};
648
+ static struct pgc_statistics pgc_main_stats = {}, pgc_main_stats_old = {}; (void)pgc_main_stats_old;
649
+ static struct pgc_statistics pgc_open_stats = {}, pgc_open_stats_old = {}; (void)pgc_open_stats_old;
650
+ static struct pgc_statistics pgc_extent_stats = {}, pgc_extent_stats_old = {}; (void)pgc_extent_stats_old;
651
+ static struct mrg_statistics mrg_stats = {}, mrg_stats_old = {}; (void)mrg_stats_old;
652
+
653
+ pgc_main_stats_old = pgc_main_stats;
654
+ pgc_main_stats = pgc_get_statistics(main_cache);
655
+ dbengine2_cache_statistics_charts(&main_cache_ptrs, &pgc_main_stats, &pgc_main_stats_old, "main", 135100);
656
+
657
+ pgc_open_stats_old = pgc_open_stats;
658
+ pgc_open_stats = pgc_get_statistics(open_cache);
659
+ dbengine2_cache_statistics_charts(&open_cache_ptrs, &pgc_open_stats, &pgc_open_stats_old, "open", 135200);
660
+
661
+ pgc_extent_stats_old = pgc_extent_stats;
662
+ pgc_extent_stats = pgc_get_statistics(extent_cache);
663
+ dbengine2_cache_statistics_charts(&extent_cache_ptrs, &pgc_extent_stats, &pgc_extent_stats_old, "extent", 135300);
664
+
665
+ cache_efficiency_stats_old = cache_efficiency_stats;
666
+ cache_efficiency_stats = rrdeng_get_cache_efficiency_stats();
667
+
668
+ mrg_stats_old = mrg_stats;
669
+ mrg_get_statistics(main_mrg, &mrg_stats);
670
+
671
+ struct rrdeng_buffer_sizes buffers = rrdeng_get_buffer_sizes();
672
+ size_t buffers_total_size = buffers.handles + buffers.xt_buf + buffers.xt_io + buffers.pdc + buffers.descriptors +
673
+ buffers.opcodes + buffers.wal + buffers.workers + buffers.epdl + buffers.deol + buffers.pd + buffers.pgc + buffers.pgd + buffers.mrg;
674
+
675
+#ifdef PDC_USE_JULYL
676
+ buffers_total_size += buffers.julyl;
677
+#endif
678
+
679
+ telemetry_dbengine_total_memory = pgc_main_stats.size + pgc_open_stats.size + pgc_extent_stats.size + mrg_stats.size + buffers_total_size;
680
+
681
+ size_t priority = 135000;
682
+
683
+ {
684
+ static RRDSET *st_pgc_memory = NULL;
685
+ static RRDDIM *rd_pgc_memory_main = NULL;
686
+ static RRDDIM *rd_pgc_memory_open = NULL; // open journal memory
687
+ static RRDDIM *rd_pgc_memory_extent = NULL; // extent compresses cache memory
688
+ static RRDDIM *rd_pgc_memory_metrics = NULL; // metric registry memory
689
+ static RRDDIM *rd_pgc_memory_buffers = NULL;
690
+
691
+ if (unlikely(!st_pgc_memory)) {
692
+ st_pgc_memory = rrdset_create_localhost(
693
+ "netdata",
694
+ "dbengine_memory",
695
+ NULL,
696
+ "dbengine memory",
697
+ NULL,
698
+ "Netdata DB Memory",
699
+ "bytes",
700
+ "netdata",
701
+ "stats",
702
+ priority,
703
+ localhost->rrd_update_every,
704
+ RRDSET_TYPE_STACKED);
705
+
706
+ rd_pgc_memory_main = rrddim_add(st_pgc_memory, "main cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
707
+ rd_pgc_memory_open = rrddim_add(st_pgc_memory, "open cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
708
+ rd_pgc_memory_extent = rrddim_add(st_pgc_memory, "extent cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
709
+ rd_pgc_memory_metrics = rrddim_add(st_pgc_memory, "metrics registry", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
710
+ rd_pgc_memory_buffers = rrddim_add(st_pgc_memory, "buffers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
711
+ }
712
+ priority++;
713
+
714
+
715
+ rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_main, (collected_number)pgc_main_stats.size);
716
+ rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_open, (collected_number)pgc_open_stats.size);
717
+ rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_extent, (collected_number)pgc_extent_stats.size);
718
+ rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_metrics, (collected_number)mrg_stats.size);
719
+ rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_buffers, (collected_number)buffers_total_size);
720
+
721
+ rrdset_done(st_pgc_memory);
722
+ }
723
+
724
+ {
725
+ static RRDSET *st_pgc_buffers = NULL;
726
+ static RRDDIM *rd_pgc_buffers_pgc = NULL;
727
+ static RRDDIM *rd_pgc_buffers_pgd = NULL;
728
+ static RRDDIM *rd_pgc_buffers_mrg = NULL;
729
+ static RRDDIM *rd_pgc_buffers_opcodes = NULL;
730
+ static RRDDIM *rd_pgc_buffers_handles = NULL;
731
+ static RRDDIM *rd_pgc_buffers_descriptors = NULL;
732
+ static RRDDIM *rd_pgc_buffers_wal = NULL;
733
+ static RRDDIM *rd_pgc_buffers_workers = NULL;
734
+ static RRDDIM *rd_pgc_buffers_pdc = NULL;
735
+ static RRDDIM *rd_pgc_buffers_xt_io = NULL;
736
+ static RRDDIM *rd_pgc_buffers_xt_buf = NULL;
737
+ static RRDDIM *rd_pgc_buffers_epdl = NULL;
738
+ static RRDDIM *rd_pgc_buffers_deol = NULL;
739
+ static RRDDIM *rd_pgc_buffers_pd = NULL;
740
+#ifdef PDC_USE_JULYL
741
+ static RRDDIM *rd_pgc_buffers_julyl = NULL;
742
+#endif
743
+
744
+ if (unlikely(!st_pgc_buffers)) {
745
+ st_pgc_buffers = rrdset_create_localhost(
746
+ "netdata",
747
+ "dbengine_buffers",
748
+ NULL,
749
+ "dbengine memory",
750
+ NULL,
751
+ "Netdata DB Buffers",
752
+ "bytes",
753
+ "netdata",
754
+ "stats",
755
+ priority,
756
+ localhost->rrd_update_every,
757
+ RRDSET_TYPE_STACKED);
758
+
759
+ rd_pgc_buffers_pgc = rrddim_add(st_pgc_buffers, "pgc", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
760
+ rd_pgc_buffers_pgd = rrddim_add(st_pgc_buffers, "pgd", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
761
+ rd_pgc_buffers_mrg = rrddim_add(st_pgc_buffers, "mrg", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
762
+ rd_pgc_buffers_opcodes = rrddim_add(st_pgc_buffers, "opcodes", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
763
+ rd_pgc_buffers_handles = rrddim_add(st_pgc_buffers, "query handles", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
764
+ rd_pgc_buffers_descriptors = rrddim_add(st_pgc_buffers, "descriptors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
765
+ rd_pgc_buffers_wal = rrddim_add(st_pgc_buffers, "wal", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
766
+ rd_pgc_buffers_workers = rrddim_add(st_pgc_buffers, "workers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
767
+ rd_pgc_buffers_pdc = rrddim_add(st_pgc_buffers, "pdc", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
768
+ rd_pgc_buffers_pd = rrddim_add(st_pgc_buffers, "pd", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
769
+ rd_pgc_buffers_xt_io = rrddim_add(st_pgc_buffers, "extent io", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
770
+ rd_pgc_buffers_xt_buf = rrddim_add(st_pgc_buffers, "extent buffers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
771
+ rd_pgc_buffers_epdl = rrddim_add(st_pgc_buffers, "epdl", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
772
+ rd_pgc_buffers_deol = rrddim_add(st_pgc_buffers, "deol", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
773
+#ifdef PDC_USE_JULYL
774
+ rd_pgc_buffers_julyl = rrddim_add(st_pgc_buffers, "julyl", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
775
+#endif
776
+ }
777
+ priority++;
778
+
779
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pgc, (collected_number)buffers.pgc);
780
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pgd, (collected_number)buffers.pgd);
781
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_mrg, (collected_number)buffers.mrg);
782
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_opcodes, (collected_number)buffers.opcodes);
783
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_handles, (collected_number)buffers.handles);
784
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_descriptors, (collected_number)buffers.descriptors);
785
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_wal, (collected_number)buffers.wal);
786
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_workers, (collected_number)buffers.workers);
787
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pdc, (collected_number)buffers.pdc);
788
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pd, (collected_number)buffers.pd);
789
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_xt_io, (collected_number)buffers.xt_io);
790
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_xt_buf, (collected_number)buffers.xt_buf);
791
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_epdl, (collected_number)buffers.epdl);
792
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_deol, (collected_number)buffers.deol);
793
+#ifdef PDC_USE_JULYL
794
+ rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_julyl, (collected_number)buffers.julyl);
795
+#endif
796
+
797
+ rrdset_done(st_pgc_buffers);
798
+ }
799
+
800
+#ifdef PDC_USE_JULYL
801
+ {
802
+ static RRDSET *st_julyl_moved = NULL;
803
+ static RRDDIM *rd_julyl_moved = NULL;
804
+
805
+ if (unlikely(!st_julyl_moved)) {
806
+ st_julyl_moved = rrdset_create_localhost(
807
+ "netdata",
808
+ "dbengine_julyl_moved",
809
+ NULL,
810
+ "dbengine memory",
811
+ NULL,
812
+ "Netdata JulyL Memory Moved",
813
+ "bytes/s",
814
+ "netdata",
815
+ "stats",
816
+ priority,
817
+ localhost->rrd_update_every,
818
+ RRDSET_TYPE_AREA);
819
+
820
+ rd_julyl_moved = rrddim_add(st_julyl_moved, "moved", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
821
+ }
822
+ priority++;
823
+
824
+ rrddim_set_by_pointer(st_julyl_moved, rd_julyl_moved, (collected_number)julyl_bytes_moved());
825
+
826
+ rrdset_done(st_julyl_moved);
827
+ }
828
+#endif
829
+
830
+ {
831
+ static RRDSET *st_mrg_metrics = NULL;
832
+ static RRDDIM *rd_mrg_metrics = NULL;
833
+ static RRDDIM *rd_mrg_acquired = NULL;
834
+ static RRDDIM *rd_mrg_collected = NULL;
835
+ static RRDDIM *rd_mrg_multiple_writers = NULL;
836
+
837
+ if (unlikely(!st_mrg_metrics)) {
838
+ st_mrg_metrics = rrdset_create_localhost(
839
+ "netdata",
840
+ "dbengine_metrics",
841
+ NULL,
842
+ "dbengine metrics",
843
+ NULL,
844
+ "Netdata Metrics in Metrics Registry",
845
+ "metrics",
846
+ "netdata",
847
+ "stats",
848
+ priority,
849
+ localhost->rrd_update_every,
850
+ RRDSET_TYPE_LINE);
851
+
852
+ rd_mrg_metrics = rrddim_add(st_mrg_metrics, "all", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
853
+ rd_mrg_acquired = rrddim_add(st_mrg_metrics, "acquired", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
854
+ rd_mrg_collected = rrddim_add(st_mrg_metrics, "collected", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
855
+ rd_mrg_multiple_writers = rrddim_add(st_mrg_metrics, "multi-collected", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
856
+ }
857
+ priority++;
858
+
859
+ rrddim_set_by_pointer(st_mrg_metrics, rd_mrg_metrics, (collected_number)mrg_stats.entries);
860
+ rrddim_set_by_pointer(st_mrg_metrics, rd_mrg_acquired, (collected_number)mrg_stats.entries_referenced);
861
+ rrddim_set_by_pointer(st_mrg_metrics, rd_mrg_collected, (collected_number)mrg_stats.writers);
862
+ rrddim_set_by_pointer(st_mrg_metrics, rd_mrg_multiple_writers, (collected_number)mrg_stats.writers_conflicts);
863
+
864
+ rrdset_done(st_mrg_metrics);
865
+ }
866
+
867
+ {
868
+ static RRDSET *st_mrg_ops = NULL;
869
+ static RRDDIM *rd_mrg_add = NULL;
870
+ static RRDDIM *rd_mrg_del = NULL;
871
+ static RRDDIM *rd_mrg_search = NULL;
872
+
873
+ if (unlikely(!st_mrg_ops)) {
874
+ st_mrg_ops = rrdset_create_localhost(
875
+ "netdata",
876
+ "dbengine_metrics_registry_operations",
877
+ NULL,
878
+ "dbengine metrics",
879
+ NULL,
880
+ "Netdata Metrics Registry Operations",
881
+ "metrics",
882
+ "netdata",
883
+ "stats",
884
+ priority,
885
+ localhost->rrd_update_every,
886
+ RRDSET_TYPE_LINE);
887
+
888
+ rd_mrg_add = rrddim_add(st_mrg_ops, "add", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
889
+ rd_mrg_del = rrddim_add(st_mrg_ops, "delete", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
890
+ rd_mrg_search = rrddim_add(st_mrg_ops, "search", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
891
+ }
892
+ priority++;
893
+
894
+ rrddim_set_by_pointer(st_mrg_ops, rd_mrg_add, (collected_number)mrg_stats.additions);
895
+ rrddim_set_by_pointer(st_mrg_ops, rd_mrg_del, (collected_number)mrg_stats.deletions);
896
+ rrddim_set_by_pointer(st_mrg_ops, rd_mrg_search, (collected_number)mrg_stats.search_hits + (collected_number)mrg_stats.search_misses);
897
+
898
+ rrdset_done(st_mrg_ops);
899
+ }
900
+
901
+ {
902
+ static RRDSET *st_mrg_references = NULL;
903
+ static RRDDIM *rd_mrg_references = NULL;
904
+
905
+ if (unlikely(!st_mrg_references)) {
906
+ st_mrg_references = rrdset_create_localhost(
907
+ "netdata",
908
+ "dbengine_metrics_registry_references",
909
+ NULL,
910
+ "dbengine metrics",
911
+ NULL,
912
+ "Netdata Metrics Registry References",
913
+ "references",
914
+ "netdata",
915
+ "stats",
916
+ priority,
917
+ localhost->rrd_update_every,
918
+ RRDSET_TYPE_LINE);
919
+
920
+ rd_mrg_references = rrddim_add(st_mrg_references, "references", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
921
+ }
922
+ priority++;
923
+
924
+ rrddim_set_by_pointer(st_mrg_references, rd_mrg_references, (collected_number)mrg_stats.current_references);
925
+
926
+ rrdset_done(st_mrg_references);
927
+ }
928
+
929
+ {
930
+ static RRDSET *st_cache_hit_ratio = NULL;
931
+ static RRDDIM *rd_hit_ratio = NULL;
932
+ static RRDDIM *rd_main_cache_hit_ratio = NULL;
933
+ static RRDDIM *rd_extent_cache_hit_ratio = NULL;
934
+ static RRDDIM *rd_extent_merge_hit_ratio = NULL;
935
+
936
+ if (unlikely(!st_cache_hit_ratio)) {
937
+ st_cache_hit_ratio = rrdset_create_localhost(
938
+ "netdata",
939
+ "dbengine_cache_hit_ratio",
940
+ NULL,
941
+ "dbengine query router",
942
+ NULL,
943
+ "Netdata Queries Cache Hit Ratio",
944
+ "%",
945
+ "netdata",
946
+ "stats",
947
+ priority,
948
+ localhost->rrd_update_every,
949
+ RRDSET_TYPE_LINE);
950
+
951
+ rd_hit_ratio = rrddim_add(st_cache_hit_ratio, "overall", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
952
+ rd_main_cache_hit_ratio = rrddim_add(st_cache_hit_ratio, "main cache", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
953
+ rd_extent_cache_hit_ratio = rrddim_add(st_cache_hit_ratio, "extent cache", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
954
+ rd_extent_merge_hit_ratio = rrddim_add(st_cache_hit_ratio, "extent merge", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
955
+ }
956
+ priority++;
957
+
958
+ size_t delta_pages_total = cache_efficiency_stats.pages_total - cache_efficiency_stats_old.pages_total;
959
+ size_t delta_pages_to_load_from_disk = cache_efficiency_stats.pages_to_load_from_disk - cache_efficiency_stats_old.pages_to_load_from_disk;
960
+ size_t delta_extents_loaded_from_disk = cache_efficiency_stats.extents_loaded_from_disk - cache_efficiency_stats_old.extents_loaded_from_disk;
961
+
962
+ size_t delta_pages_data_source_main_cache = cache_efficiency_stats.pages_data_source_main_cache - cache_efficiency_stats_old.pages_data_source_main_cache;
963
+ size_t delta_pages_pending_found_in_cache_at_pass4 = cache_efficiency_stats.pages_data_source_main_cache_at_pass4 - cache_efficiency_stats_old.pages_data_source_main_cache_at_pass4;
964
+
965
+ size_t delta_pages_data_source_extent_cache = cache_efficiency_stats.pages_data_source_extent_cache - cache_efficiency_stats_old.pages_data_source_extent_cache;
966
+ size_t delta_pages_load_extent_merged = cache_efficiency_stats.pages_load_extent_merged - cache_efficiency_stats_old.pages_load_extent_merged;
967
+
968
+ size_t pages_total_hit = delta_pages_total - delta_extents_loaded_from_disk;
969
+
970
+ static size_t overall_hit_ratio = 100;
971
+ size_t main_cache_hit_ratio = 0, extent_cache_hit_ratio = 0, extent_merge_hit_ratio = 0;
972
+ if(delta_pages_total) {
973
+ if(pages_total_hit > delta_pages_total)
974
+ pages_total_hit = delta_pages_total;
975
+
976
+ overall_hit_ratio = pages_total_hit * 100 * 10000 / delta_pages_total;
977
+
978
+ size_t delta_pages_main_cache = delta_pages_data_source_main_cache + delta_pages_pending_found_in_cache_at_pass4;
979
+ if(delta_pages_main_cache > delta_pages_total)
980
+ delta_pages_main_cache = delta_pages_total;
981
+
982
+ main_cache_hit_ratio = delta_pages_main_cache * 100 * 10000 / delta_pages_total;
983
+ }
984
+
985
+ if(delta_pages_to_load_from_disk) {
986
+ if(delta_pages_data_source_extent_cache > delta_pages_to_load_from_disk)
987
+ delta_pages_data_source_extent_cache = delta_pages_to_load_from_disk;
988
+
989
+ extent_cache_hit_ratio = delta_pages_data_source_extent_cache * 100 * 10000 / delta_pages_to_load_from_disk;
990
+
991
+ if(delta_pages_load_extent_merged > delta_pages_to_load_from_disk)
992
+ delta_pages_load_extent_merged = delta_pages_to_load_from_disk;
993
+
994
+ extent_merge_hit_ratio = delta_pages_load_extent_merged * 100 * 10000 / delta_pages_to_load_from_disk;
995
+ }
996
+
997
+ rrddim_set_by_pointer(st_cache_hit_ratio, rd_hit_ratio, (collected_number)overall_hit_ratio);
998
+ rrddim_set_by_pointer(st_cache_hit_ratio, rd_main_cache_hit_ratio, (collected_number)main_cache_hit_ratio);
999
+ rrddim_set_by_pointer(st_cache_hit_ratio, rd_extent_cache_hit_ratio, (collected_number)extent_cache_hit_ratio);
1000
+ rrddim_set_by_pointer(st_cache_hit_ratio, rd_extent_merge_hit_ratio, (collected_number)extent_merge_hit_ratio);
1001
+
1002
+ rrdset_done(st_cache_hit_ratio);
1003
+ }
1004
+
1005
+ {
1006
+ static RRDSET *st_queries = NULL;
1007
+ static RRDDIM *rd_total = NULL;
1008
+ static RRDDIM *rd_open = NULL;
1009
+ static RRDDIM *rd_jv2 = NULL;
1010
+ static RRDDIM *rd_planned_with_gaps = NULL;
1011
+ static RRDDIM *rd_executed_with_gaps = NULL;
1012
+
1013
+ if (unlikely(!st_queries)) {
1014
+ st_queries = rrdset_create_localhost(
1015
+ "netdata",
1016
+ "dbengine_queries",
1017
+ NULL,
1018
+ "dbengine query router",
1019
+ NULL,
1020
+ "Netdata Queries",
1021
+ "queries/s",
1022
+ "netdata",
1023
+ "stats",
1024
+ priority,
1025
+ localhost->rrd_update_every,
1026
+ RRDSET_TYPE_LINE);
1027
+
1028
+ rd_total = rrddim_add(st_queries, "total", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1029
+ rd_open = rrddim_add(st_queries, "open cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1030
+ rd_jv2 = rrddim_add(st_queries, "journal v2", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1031
+ rd_planned_with_gaps = rrddim_add(st_queries, "planned with gaps", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1032
+ rd_executed_with_gaps = rrddim_add(st_queries, "executed with gaps", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1033
+ }
1034
+ priority++;
1035
+
1036
+ rrddim_set_by_pointer(st_queries, rd_total, (collected_number)cache_efficiency_stats.queries);
1037
+ rrddim_set_by_pointer(st_queries, rd_open, (collected_number)cache_efficiency_stats.queries_open);
1038
+ rrddim_set_by_pointer(st_queries, rd_jv2, (collected_number)cache_efficiency_stats.queries_journal_v2);
1039
+ rrddim_set_by_pointer(st_queries, rd_planned_with_gaps, (collected_number)cache_efficiency_stats.queries_planned_with_gaps);
1040
+ rrddim_set_by_pointer(st_queries, rd_executed_with_gaps, (collected_number)cache_efficiency_stats.queries_executed_with_gaps);
1041
+
1042
+ rrdset_done(st_queries);
1043
+ }
1044
+
1045
+ {
1046
+ static RRDSET *st_queries_running = NULL;
1047
+ static RRDDIM *rd_queries = NULL;
1048
+
1049
+ if (unlikely(!st_queries_running)) {
1050
+ st_queries_running = rrdset_create_localhost(
1051
+ "netdata",
1052
+ "dbengine_queries_running",
1053
+ NULL,
1054
+ "dbengine query router",
1055
+ NULL,
1056
+ "Netdata Queries Running",
1057
+ "queries",
1058
+ "netdata",
1059
+ "stats",
1060
+ priority,
1061
+ localhost->rrd_update_every,
1062
+ RRDSET_TYPE_LINE);
1063
+
1064
+ rd_queries = rrddim_add(st_queries_running, "queries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1065
+ }
1066
+ priority++;
1067
+
1068
+ rrddim_set_by_pointer(st_queries_running, rd_queries, (collected_number)cache_efficiency_stats.currently_running_queries);
1069
+
1070
+ rrdset_done(st_queries_running);
1071
+ }
1072
+
1073
+ {
1074
+ static RRDSET *st_query_pages_metadata_source = NULL;
1075
+ static RRDDIM *rd_cache = NULL;
1076
+ static RRDDIM *rd_open = NULL;
1077
+ static RRDDIM *rd_jv2 = NULL;
1078
+
1079
+ if (unlikely(!st_query_pages_metadata_source)) {
1080
+ st_query_pages_metadata_source = rrdset_create_localhost(
1081
+ "netdata",
1082
+ "dbengine_query_pages_metadata_source",
1083
+ NULL,
1084
+ "dbengine query router",
1085
+ NULL,
1086
+ "Netdata Query Pages Metadata Source",
1087
+ "pages/s",
1088
+ "netdata",
1089
+ "stats",
1090
+ priority,
1091
+ localhost->rrd_update_every,
1092
+ RRDSET_TYPE_STACKED);
1093
+
1094
+ rd_cache = rrddim_add(st_query_pages_metadata_source, "cache hit", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1095
+ rd_jv2 = rrddim_add(st_query_pages_metadata_source, "journal v2 scan", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1096
+ rd_open = rrddim_add(st_query_pages_metadata_source, "open journal", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1097
+ }
1098
+ priority++;
1099
+
1100
+ rrddim_set_by_pointer(st_query_pages_metadata_source, rd_cache, (collected_number)cache_efficiency_stats.pages_meta_source_main_cache);
1101
+ rrddim_set_by_pointer(st_query_pages_metadata_source, rd_jv2, (collected_number)cache_efficiency_stats.pages_meta_source_journal_v2);
1102
+ rrddim_set_by_pointer(st_query_pages_metadata_source, rd_open, (collected_number)cache_efficiency_stats.pages_meta_source_open_cache);
1103
+
1104
+ rrdset_done(st_query_pages_metadata_source);
1105
+ }
1106
+
1107
+ {
1108
+ static RRDSET *st_query_pages_data_source = NULL;
1109
+ static RRDDIM *rd_pages_main_cache = NULL;
1110
+ static RRDDIM *rd_pages_disk = NULL;
1111
+ static RRDDIM *rd_pages_extent_cache = NULL;
1112
+
1113
+ if (unlikely(!st_query_pages_data_source)) {
1114
+ st_query_pages_data_source = rrdset_create_localhost(
1115
+ "netdata",
1116
+ "dbengine_query_pages_data_source",
1117
+ NULL,
1118
+ "dbengine query router",
1119
+ NULL,
1120
+ "Netdata Query Pages to Data Source",
1121
+ "pages/s",
1122
+ "netdata",
1123
+ "stats",
1124
+ priority,
1125
+ localhost->rrd_update_every,
1126
+ RRDSET_TYPE_STACKED);
1127
+
1128
+ rd_pages_main_cache = rrddim_add(st_query_pages_data_source, "main cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1129
+ rd_pages_disk = rrddim_add(st_query_pages_data_source, "disk", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1130
+ rd_pages_extent_cache = rrddim_add(st_query_pages_data_source, "extent cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1131
+ }
1132
+ priority++;
1133
+
1134
+ rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_main_cache, (collected_number)cache_efficiency_stats.pages_data_source_main_cache + (collected_number)cache_efficiency_stats.pages_data_source_main_cache_at_pass4);
1135
+ rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_disk, (collected_number)cache_efficiency_stats.pages_to_load_from_disk);
1136
+ rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_extent_cache, (collected_number)cache_efficiency_stats.pages_data_source_extent_cache);
1137
+
1138
+ rrdset_done(st_query_pages_data_source);
1139
+ }
1140
+
1141
+ {
1142
+ static RRDSET *st_query_next_page = NULL;
1143
+ static RRDDIM *rd_pass4 = NULL;
1144
+ static RRDDIM *rd_nowait_failed = NULL;
1145
+ static RRDDIM *rd_wait_failed = NULL;
1146
+ static RRDDIM *rd_wait_loaded = NULL;
1147
+ static RRDDIM *rd_nowait_loaded = NULL;
1148
+
1149
+ if (unlikely(!st_query_next_page)) {
1150
+ st_query_next_page = rrdset_create_localhost(
1151
+ "netdata",
1152
+ "dbengine_query_next_page",
1153
+ NULL,
1154
+ "dbengine query router",
1155
+ NULL,
1156
+ "Netdata Query Next Page",
1157
+ "pages/s",
1158
+ "netdata",
1159
+ "stats",
1160
+ priority,
1161
+ localhost->rrd_update_every,
1162
+ RRDSET_TYPE_STACKED);
1163
+
1164
+ rd_pass4 = rrddim_add(st_query_next_page, "pass4", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1165
+ rd_wait_failed = rrddim_add(st_query_next_page, "failed slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1166
+ rd_nowait_failed = rrddim_add(st_query_next_page, "failed fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1167
+ rd_wait_loaded = rrddim_add(st_query_next_page, "loaded slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1168
+ rd_nowait_loaded = rrddim_add(st_query_next_page, "loaded fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1169
+ }
1170
+ priority++;
1171
+
1172
+ rrddim_set_by_pointer(st_query_next_page, rd_pass4, (collected_number)cache_efficiency_stats.pages_data_source_main_cache_at_pass4);
1173
+ rrddim_set_by_pointer(st_query_next_page, rd_wait_failed, (collected_number)cache_efficiency_stats.page_next_wait_failed);
1174
+ rrddim_set_by_pointer(st_query_next_page, rd_nowait_failed, (collected_number)cache_efficiency_stats.page_next_nowait_failed);
1175
+ rrddim_set_by_pointer(st_query_next_page, rd_wait_loaded, (collected_number)cache_efficiency_stats.page_next_wait_loaded);
1176
+ rrddim_set_by_pointer(st_query_next_page, rd_nowait_loaded, (collected_number)cache_efficiency_stats.page_next_nowait_loaded);
1177
+
1178
+ rrdset_done(st_query_next_page);
1179
+ }
1180
+
1181
+ {
1182
+ static RRDSET *st_query_page_issues = NULL;
1183
+ static RRDDIM *rd_pages_zero_time = NULL;
1184
+ static RRDDIM *rd_pages_past_time = NULL;
1185
+ static RRDDIM *rd_pages_invalid_size = NULL;
1186
+ static RRDDIM *rd_pages_fixed_update_every = NULL;
1187
+ static RRDDIM *rd_pages_fixed_entries = NULL;
1188
+ static RRDDIM *rd_pages_overlapping = NULL;
1189
+
1190
+ if (unlikely(!st_query_page_issues)) {
1191
+ st_query_page_issues = rrdset_create_localhost(
1192
+ "netdata",
1193
+ "dbengine_query_next_page_issues",
1194
+ NULL,
1195
+ "dbengine query router",
1196
+ NULL,
1197
+ "Netdata Query Next Page Issues",
1198
+ "pages/s",
1199
+ "netdata",
1200
+ "stats",
1201
+ priority,
1202
+ localhost->rrd_update_every,
1203
+ RRDSET_TYPE_STACKED);
1204
+
1205
+ rd_pages_zero_time = rrddim_add(st_query_page_issues, "zero timestamp", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1206
+ rd_pages_invalid_size = rrddim_add(st_query_page_issues, "invalid size", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1207
+ rd_pages_past_time = rrddim_add(st_query_page_issues, "past time", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1208
+ rd_pages_overlapping = rrddim_add(st_query_page_issues, "overlapping", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1209
+ rd_pages_fixed_update_every = rrddim_add(st_query_page_issues, "update every fixed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1210
+ rd_pages_fixed_entries = rrddim_add(st_query_page_issues, "entries fixed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1211
+ }
1212
+ priority++;
1213
+
1214
+ rrddim_set_by_pointer(st_query_page_issues, rd_pages_zero_time, (collected_number)cache_efficiency_stats.pages_zero_time_skipped);
1215
+ rrddim_set_by_pointer(st_query_page_issues, rd_pages_invalid_size, (collected_number)cache_efficiency_stats.pages_invalid_size_skipped);
1216
+ rrddim_set_by_pointer(st_query_page_issues, rd_pages_past_time, (collected_number)cache_efficiency_stats.pages_past_time_skipped);
1217
+ rrddim_set_by_pointer(st_query_page_issues, rd_pages_overlapping, (collected_number)cache_efficiency_stats.pages_overlapping_skipped);
1218
+ rrddim_set_by_pointer(st_query_page_issues, rd_pages_fixed_update_every, (collected_number)cache_efficiency_stats.pages_invalid_update_every_fixed);
1219
+ rrddim_set_by_pointer(st_query_page_issues, rd_pages_fixed_entries, (collected_number)cache_efficiency_stats.pages_invalid_entries_fixed);
1220
+
1221
+ rrdset_done(st_query_page_issues);
1222
+ }
1223
+
1224
+ {
1225
+ static RRDSET *st_query_pages_from_disk = NULL;
1226
+ static RRDDIM *rd_compressed = NULL;
1227
+ static RRDDIM *rd_invalid = NULL;
1228
+ static RRDDIM *rd_uncompressed = NULL;
1229
+ static RRDDIM *rd_mmap_failed = NULL;
1230
+ static RRDDIM *rd_unavailable = NULL;
1231
+ static RRDDIM *rd_unroutable = NULL;
1232
+ static RRDDIM *rd_not_found = NULL;
1233
+ static RRDDIM *rd_cancelled = NULL;
1234
+ static RRDDIM *rd_invalid_extent = NULL;
1235
+ static RRDDIM *rd_extent_merged = NULL;
1236
+
1237
+ if (unlikely(!st_query_pages_from_disk)) {
1238
+ st_query_pages_from_disk = rrdset_create_localhost(
1239
+ "netdata",
1240
+ "dbengine_query_pages_disk_load",
1241
+ NULL,
1242
+ "dbengine query router",
1243
+ NULL,
1244
+ "Netdata Query Pages Loaded from Disk",
1245
+ "pages/s",
1246
+ "netdata",
1247
+ "stats",
1248
+ priority,
1249
+ localhost->rrd_update_every,
1250
+ RRDSET_TYPE_LINE);
1251
+
1252
+ rd_compressed = rrddim_add(st_query_pages_from_disk, "ok compressed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1253
+ rd_invalid = rrddim_add(st_query_pages_from_disk, "fail invalid page", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1254
+ rd_uncompressed = rrddim_add(st_query_pages_from_disk, "ok uncompressed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1255
+ rd_mmap_failed = rrddim_add(st_query_pages_from_disk, "fail cant mmap", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1256
+ rd_unavailable = rrddim_add(st_query_pages_from_disk, "fail unavailable", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1257
+ rd_unroutable = rrddim_add(st_query_pages_from_disk, "fail unroutable", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1258
+ rd_not_found = rrddim_add(st_query_pages_from_disk, "fail not found", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1259
+ rd_invalid_extent = rrddim_add(st_query_pages_from_disk, "fail invalid extent", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1260
+ rd_extent_merged = rrddim_add(st_query_pages_from_disk, "extent merged", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1261
+ rd_cancelled = rrddim_add(st_query_pages_from_disk, "cancelled", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1262
+ }
1263
+ priority++;
1264
+
1265
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_compressed, (collected_number)cache_efficiency_stats.pages_load_ok_compressed);
1266
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_invalid, (collected_number)cache_efficiency_stats.pages_load_fail_invalid_page_in_extent);
1267
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_uncompressed, (collected_number)cache_efficiency_stats.pages_load_ok_uncompressed);
1268
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_mmap_failed, (collected_number)cache_efficiency_stats.pages_load_fail_cant_mmap_extent);
1269
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_unavailable, (collected_number)cache_efficiency_stats.pages_load_fail_datafile_not_available);
1270
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_unroutable, (collected_number)cache_efficiency_stats.pages_load_fail_unroutable);
1271
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_not_found, (collected_number)cache_efficiency_stats.pages_load_fail_not_found);
1272
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_cancelled, (collected_number)cache_efficiency_stats.pages_load_fail_cancelled);
1273
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_invalid_extent, (collected_number)cache_efficiency_stats.pages_load_fail_invalid_extent);
1274
+ rrddim_set_by_pointer(st_query_pages_from_disk, rd_extent_merged, (collected_number)cache_efficiency_stats.pages_load_extent_merged);
1275
+
1276
+ rrdset_done(st_query_pages_from_disk);
1277
+ }
1278
+
1279
+ {
1280
+ static RRDSET *st_events = NULL;
1281
+ static RRDDIM *rd_journal_v2_mapped = NULL;
1282
+ static RRDDIM *rd_journal_v2_unmapped = NULL;
1283
+ static RRDDIM *rd_datafile_creation = NULL;
1284
+ static RRDDIM *rd_datafile_deletion = NULL;
1285
+ static RRDDIM *rd_datafile_deletion_spin = NULL;
1286
+ static RRDDIM *rd_jv2_indexing = NULL;
1287
+ static RRDDIM *rd_retention = NULL;
1288
+
1289
+ if (unlikely(!st_events)) {
1290
+ st_events = rrdset_create_localhost(
1291
+ "netdata",
1292
+ "dbengine_events",
1293
+ NULL,
1294
+ "dbengine query router",
1295
+ NULL,
1296
+ "Netdata Database Events",
1297
+ "events/s",
1298
+ "netdata",
1299
+ "stats",
1300
+ priority,
1301
+ localhost->rrd_update_every,
1302
+ RRDSET_TYPE_LINE);
1303
+
1304
+ rd_journal_v2_mapped = rrddim_add(st_events, "journal v2 mapped", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1305
+ rd_journal_v2_unmapped = rrddim_add(st_events, "journal v2 unmapped", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1306
+ rd_datafile_creation = rrddim_add(st_events, "datafile creation", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1307
+ rd_datafile_deletion = rrddim_add(st_events, "datafile deletion", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1308
+ rd_datafile_deletion_spin = rrddim_add(st_events, "datafile deletion spin", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1309
+ rd_jv2_indexing = rrddim_add(st_events, "journal v2 indexing", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1310
+ rd_retention = rrddim_add(st_events, "retention", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1311
+ }
1312
+ priority++;
1313
+
1314
+ rrddim_set_by_pointer(st_events, rd_journal_v2_mapped, (collected_number)cache_efficiency_stats.journal_v2_mapped);
1315
+ rrddim_set_by_pointer(st_events, rd_journal_v2_unmapped, (collected_number)cache_efficiency_stats.journal_v2_unmapped);
1316
+ rrddim_set_by_pointer(st_events, rd_datafile_creation, (collected_number)cache_efficiency_stats.datafile_creation_started);
1317
+ rrddim_set_by_pointer(st_events, rd_datafile_deletion, (collected_number)cache_efficiency_stats.datafile_deletion_started);
1318
+ rrddim_set_by_pointer(st_events, rd_datafile_deletion_spin, (collected_number)cache_efficiency_stats.datafile_deletion_spin);
1319
+ rrddim_set_by_pointer(st_events, rd_jv2_indexing, (collected_number)cache_efficiency_stats.journal_v2_indexing_started);
1320
+ rrddim_set_by_pointer(st_events, rd_retention, (collected_number)cache_efficiency_stats.metrics_retention_started);
1321
+
1322
+ rrdset_done(st_events);
1323
+ }
1324
+
1325
+ {
1326
+ static RRDSET *st_prep_timings = NULL;
1327
+ static RRDDIM *rd_routing = NULL;
1328
+ static RRDDIM *rd_main_cache = NULL;
1329
+ static RRDDIM *rd_open_cache = NULL;
1330
+ static RRDDIM *rd_journal_v2 = NULL;
1331
+ static RRDDIM *rd_pass4 = NULL;
1332
+
1333
+ if (unlikely(!st_prep_timings)) {
1334
+ st_prep_timings = rrdset_create_localhost(
1335
+ "netdata",
1336
+ "dbengine_prep_timings",
1337
+ NULL,
1338
+ "dbengine query router",
1339
+ NULL,
1340
+ "Netdata Query Preparation Timings",
1341
+ "usec/s",
1342
+ "netdata",
1343
+ "stats",
1344
+ priority,
1345
+ localhost->rrd_update_every,
1346
+ RRDSET_TYPE_STACKED);
1347
+
1348
+ rd_routing = rrddim_add(st_prep_timings, "routing", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1349
+ rd_main_cache = rrddim_add(st_prep_timings, "main cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1350
+ rd_open_cache = rrddim_add(st_prep_timings, "open cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1351
+ rd_journal_v2 = rrddim_add(st_prep_timings, "journal v2", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1352
+ rd_pass4 = rrddim_add(st_prep_timings, "pass4", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1353
+ }
1354
+ priority++;
1355
+
1356
+ rrddim_set_by_pointer(st_prep_timings, rd_routing, (collected_number)cache_efficiency_stats.prep_time_to_route);
1357
+ rrddim_set_by_pointer(st_prep_timings, rd_main_cache, (collected_number)cache_efficiency_stats.prep_time_in_main_cache_lookup);
1358
+ rrddim_set_by_pointer(st_prep_timings, rd_open_cache, (collected_number)cache_efficiency_stats.prep_time_in_open_cache_lookup);
1359
+ rrddim_set_by_pointer(st_prep_timings, rd_journal_v2, (collected_number)cache_efficiency_stats.prep_time_in_journal_v2_lookup);
1360
+ rrddim_set_by_pointer(st_prep_timings, rd_pass4, (collected_number)cache_efficiency_stats.prep_time_in_pass4_lookup);
1361
+
1362
+ rrdset_done(st_prep_timings);
1363
+ }
1364
+
1365
+ {
1366
+ static RRDSET *st_query_timings = NULL;
1367
+ static RRDDIM *rd_init = NULL;
1368
+ static RRDDIM *rd_prep_wait = NULL;
1369
+ static RRDDIM *rd_next_page_disk_fast = NULL;
1370
+ static RRDDIM *rd_next_page_disk_slow = NULL;
1371
+ static RRDDIM *rd_next_page_preload_fast = NULL;
1372
+ static RRDDIM *rd_next_page_preload_slow = NULL;
1373
+
1374
+ if (unlikely(!st_query_timings)) {
1375
+ st_query_timings = rrdset_create_localhost(
1376
+ "netdata",
1377
+ "dbengine_query_timings",
1378
+ NULL,
1379
+ "dbengine query router",
1380
+ NULL,
1381
+ "Netdata Query Timings",
1382
+ "usec/s",
1383
+ "netdata",
1384
+ "stats",
1385
+ priority,
1386
+ localhost->rrd_update_every,
1387
+ RRDSET_TYPE_STACKED);
1388
+
1389
+ rd_init = rrddim_add(st_query_timings, "init", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1390
+ rd_prep_wait = rrddim_add(st_query_timings, "prep wait", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1391
+ rd_next_page_disk_fast = rrddim_add(st_query_timings, "next page disk fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1392
+ rd_next_page_disk_slow = rrddim_add(st_query_timings, "next page disk slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1393
+ rd_next_page_preload_fast = rrddim_add(st_query_timings, "next page preload fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1394
+ rd_next_page_preload_slow = rrddim_add(st_query_timings, "next page preload slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1395
+ }
1396
+ priority++;
1397
+
1398
+ rrddim_set_by_pointer(st_query_timings, rd_init, (collected_number)cache_efficiency_stats.query_time_init);
1399
+ rrddim_set_by_pointer(st_query_timings, rd_prep_wait, (collected_number)cache_efficiency_stats.query_time_wait_for_prep);
1400
+ rrddim_set_by_pointer(st_query_timings, rd_next_page_disk_fast, (collected_number)cache_efficiency_stats.query_time_to_fast_disk_next_page);
1401
+ rrddim_set_by_pointer(st_query_timings, rd_next_page_disk_slow, (collected_number)cache_efficiency_stats.query_time_to_slow_disk_next_page);
1402
+ rrddim_set_by_pointer(st_query_timings, rd_next_page_preload_fast, (collected_number)cache_efficiency_stats.query_time_to_fast_preload_next_page);
1403
+ rrddim_set_by_pointer(st_query_timings, rd_next_page_preload_slow, (collected_number)cache_efficiency_stats.query_time_to_slow_preload_next_page);
1404
+
1405
+ rrdset_done(st_query_timings);
1406
+ }
1407
+
1408
+ if(netdata_rwlock_tryrdlock(&rrd_rwlock) == 0) {
1409
+ priority = 135400;
1410
+
1411
+ RRDHOST *host;
1412
+ unsigned long long stats_array[RRDENG_NR_STATS] = {0};
1413
+ unsigned long long local_stats_array[RRDENG_NR_STATS];
1414
+ unsigned dbengine_contexts = 0, counted_multihost_db[RRD_STORAGE_TIERS] = { 0 }, i;
1415
+
1416
+ rrdhost_foreach_read(host) {
1417
+ if (!rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) {
1418
+
1419
+ /* get localhost's DB engine's statistics for each tier */
1420
+ for(size_t tier = 0; tier < storage_tiers ;tier++) {
1421
+ if(host->db[tier].mode != RRD_MEMORY_MODE_DBENGINE) continue;
1422
+ if(!host->db[tier].si) continue;
1423
+
1424
+ if(counted_multihost_db[tier])
1425
+ continue;
1426
+ else
1427
+ counted_multihost_db[tier] = 1;
1428
+
1429
+ ++dbengine_contexts;
1430
+ rrdeng_get_37_statistics((struct rrdengine_instance *)host->db[tier].si, local_stats_array);
1431
+ for (i = 0; i < RRDENG_NR_STATS; ++i) {
1432
+ /* aggregate statistics across hosts */
1433
+ stats_array[i] += local_stats_array[i];
1434
+ }
1435
+ }
1436
+ }
1437
+ }
1438
+ rrd_rdunlock();
1439
+
1440
+ if (dbengine_contexts) {
1441
+ /* deduplicate telemetry by getting the ones from the last context */
1442
+ stats_array[30] = local_stats_array[30];
1443
+ stats_array[31] = local_stats_array[31];
1444
+ stats_array[32] = local_stats_array[32];
1445
+ stats_array[34] = local_stats_array[34];
1446
+ stats_array[36] = local_stats_array[36];
1447
+
1448
+ // ----------------------------------------------------------------
1449
+
1450
+ {
1451
+ static RRDSET *st_compression = NULL;
1452
+ static RRDDIM *rd_savings = NULL;
1453
+
1454
+ if (unlikely(!st_compression)) {
1455
+ st_compression = rrdset_create_localhost(
1456
+ "netdata",
1457
+ "dbengine_compression_ratio",
1458
+ NULL,
1459
+ "dbengine io",
1460
+ NULL,
1461
+ "Netdata DB engine data extents' compression savings ratio",
1462
+ "percentage",
1463
+ "netdata",
1464
+ "stats",
1465
+ priority,
1466
+ localhost->rrd_update_every,
1467
+ RRDSET_TYPE_LINE);
1468
+
1469
+ rd_savings = rrddim_add(st_compression, "savings", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
1470
+ }
1471
+ priority++;
1472
+
1473
+ unsigned long long ratio;
1474
+ unsigned long long compressed_content_size = stats_array[12];
1475
+ unsigned long long content_size = stats_array[11];
1476
+
1477
+ if (content_size) {
1478
+ // allow negative savings
1479
+ ratio = ((content_size - compressed_content_size) * 100 * 1000) / content_size;
1480
+ } else {
1481
+ ratio = 0;
1482
+ }
1483
+ rrddim_set_by_pointer(st_compression, rd_savings, ratio);
1484
+
1485
+ rrdset_done(st_compression);
1486
+ }
1487
+
1488
+ // ----------------------------------------------------------------
1489
+
1490
+ {
1491
+ static RRDSET *st_io_stats = NULL;
1492
+ static RRDDIM *rd_reads = NULL;
1493
+ static RRDDIM *rd_writes = NULL;
1494
+
1495
+ if (unlikely(!st_io_stats)) {
1496
+ st_io_stats = rrdset_create_localhost(
1497
+ "netdata",
1498
+ "dbengine_io_throughput",
1499
+ NULL,
1500
+ "dbengine io",
1501
+ NULL,
1502
+ "Netdata DB engine I/O throughput",
1503
+ "MiB/s",
1504
+ "netdata",
1505
+ "stats",
1506
+ priority,
1507
+ localhost->rrd_update_every,
1508
+ RRDSET_TYPE_LINE);
1509
+
1510
+ rd_reads = rrddim_add(st_io_stats, "reads", NULL, 1, 1024 * 1024, RRD_ALGORITHM_INCREMENTAL);
1511
+ rd_writes = rrddim_add(st_io_stats, "writes", NULL, -1, 1024 * 1024, RRD_ALGORITHM_INCREMENTAL);
1512
+ }
1513
+ priority++;
1514
+
1515
+ rrddim_set_by_pointer(st_io_stats, rd_reads, (collected_number)stats_array[17]);
1516
+ rrddim_set_by_pointer(st_io_stats, rd_writes, (collected_number)stats_array[15]);
1517
+ rrdset_done(st_io_stats);
1518
+ }
1519
+
1520
+ // ----------------------------------------------------------------
1521
+
1522
+ {
1523
+ static RRDSET *st_io_stats = NULL;
1524
+ static RRDDIM *rd_reads = NULL;
1525
+ static RRDDIM *rd_writes = NULL;
1526
+
1527
+ if (unlikely(!st_io_stats)) {
1528
+ st_io_stats = rrdset_create_localhost(
1529
+ "netdata",
1530
+ "dbengine_io_operations",
1531
+ NULL,
1532
+ "dbengine io",
1533
+ NULL,
1534
+ "Netdata DB engine I/O operations",
1535
+ "operations/s",
1536
+ "netdata",
1537
+ "stats",
1538
+ priority,
1539
+ localhost->rrd_update_every,
1540
+ RRDSET_TYPE_LINE);
1541
+
1542
+ rd_reads = rrddim_add(st_io_stats, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1543
+ rd_writes = rrddim_add(st_io_stats, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1544
+ }
1545
+ priority++;
1546
+
1547
+ rrddim_set_by_pointer(st_io_stats, rd_reads, (collected_number)stats_array[18]);
1548
+ rrddim_set_by_pointer(st_io_stats, rd_writes, (collected_number)stats_array[16]);
1549
+ rrdset_done(st_io_stats);
1550
+ }
1551
+
1552
+ // ----------------------------------------------------------------
1553
+
1554
+ {
1555
+ static RRDSET *st_errors = NULL;
1556
+ static RRDDIM *rd_fs_errors = NULL;
1557
+ static RRDDIM *rd_io_errors = NULL;
1558
+ static RRDDIM *pg_cache_over_half_dirty_events = NULL;
1559
+
1560
+ if (unlikely(!st_errors)) {
1561
+ st_errors = rrdset_create_localhost(
1562
+ "netdata",
1563
+ "dbengine_global_errors",
1564
+ NULL,
1565
+ "dbengine io",
1566
+ NULL,
1567
+ "Netdata DB engine errors",
1568
+ "errors/s",
1569
+ "netdata",
1570
+ "stats",
1571
+ priority,
1572
+ localhost->rrd_update_every,
1573
+ RRDSET_TYPE_LINE);
1574
+
1575
+ rd_io_errors = rrddim_add(st_errors, "io_errors", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1576
+ rd_fs_errors = rrddim_add(st_errors, "fs_errors", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1577
+ pg_cache_over_half_dirty_events =
1578
+ rrddim_add(st_errors, "pg_cache_over_half_dirty_events", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1579
+ }
1580
+ priority++;
1581
+
1582
+ rrddim_set_by_pointer(st_errors, rd_io_errors, (collected_number)stats_array[30]);
1583
+ rrddim_set_by_pointer(st_errors, rd_fs_errors, (collected_number)stats_array[31]);
1584
+ rrddim_set_by_pointer(st_errors, pg_cache_over_half_dirty_events, (collected_number)stats_array[34]);
1585
+ rrdset_done(st_errors);
1586
+ }
1587
+
1588
+ // ----------------------------------------------------------------
1589
+
1590
+ {
1591
+ static RRDSET *st_fd = NULL;
1592
+ static RRDDIM *rd_fd_current = NULL;
1593
+ static RRDDIM *rd_fd_max = NULL;
1594
+
1595
+ if (unlikely(!st_fd)) {
1596
+ st_fd = rrdset_create_localhost(
1597
+ "netdata",
1598
+ "dbengine_global_file_descriptors",
1599
+ NULL,
1600
+ "dbengine io",
1601
+ NULL,
1602
+ "Netdata DB engine File Descriptors",
1603
+ "descriptors",
1604
+ "netdata",
1605
+ "stats",
1606
+ priority,
1607
+ localhost->rrd_update_every,
1608
+ RRDSET_TYPE_LINE);
1609
+
1610
+ rd_fd_current = rrddim_add(st_fd, "current", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1611
+ rd_fd_max = rrddim_add(st_fd, "max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1612
+ }
1613
+ priority++;
1614
+
1615
+ rrddim_set_by_pointer(st_fd, rd_fd_current, (collected_number)stats_array[32]);
1616
+ /* Careful here, modify this accordingly if the File-Descriptor budget ever changes */
1617
+ rrddim_set_by_pointer(st_fd, rd_fd_max, (collected_number)rlimit_nofile.rlim_cur / 4);
1618
+ rrdset_done(st_fd);
1619
+ }
1620
+ }
1621
+ }
1622
+}
1623
+
1624
+#endif
src/daemon/telemetry/telemetry-dbengine.h
new
+17
@@ -0,0 +1,17 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_DBENGINE_H
4
+#define NETDATA_TELEMETRY_DBENGINE_H
5
+
6
+#include "daemon/common.h"
7
+
8
+#if defined(TELEMETRY_INTERNALS)
9
+extern size_t telemetry_dbengine_total_memory;
10
+
11
+#if defined(ENABLE_DBENGINE)
12
+void telemetry_dbengine_do(bool extended);
13
+#endif
14
+
15
+#endif
16
+
17
+#endif //NETDATA_TELEMETRY_DBENGINE_H
src/daemon/telemetry/telemetry-dictionary.c
new
+378
@@ -0,0 +1,378 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-dictionary.h"
5
+
6
+struct dictionary_stats dictionary_stats_category_collectors = { .name = "collectors" };
7
+struct dictionary_stats dictionary_stats_category_rrdhost = { .name = "rrdhost" };
8
+struct dictionary_stats dictionary_stats_category_rrdset = { .name = "rrdset" };
9
+struct dictionary_stats dictionary_stats_category_rrddim = { .name = "rrddim" };
10
+struct dictionary_stats dictionary_stats_category_rrdcontext = { .name = "context" };
11
+struct dictionary_stats dictionary_stats_category_rrdlabels = { .name = "labels" };
12
+struct dictionary_stats dictionary_stats_category_rrdhealth = { .name = "health" };
13
+struct dictionary_stats dictionary_stats_category_functions = { .name = "functions" };
14
+struct dictionary_stats dictionary_stats_category_replication = { .name = "replication" };
15
+
16
+size_t rrddim_db_memory_size = 0;
17
+
18
+#ifdef DICT_WITH_STATS
19
+struct dictionary_categories {
20
+ struct dictionary_stats *stats;
21
+
22
+ RRDSET *st_dicts;
23
+ RRDDIM *rd_dicts_active;
24
+ RRDDIM *rd_dicts_deleted;
25
+
26
+ RRDSET *st_items;
27
+ RRDDIM *rd_items_entries;
28
+ RRDDIM *rd_items_referenced;
29
+ RRDDIM *rd_items_pending_deletion;
30
+
31
+ RRDSET *st_ops;
32
+ RRDDIM *rd_ops_creations;
33
+ RRDDIM *rd_ops_destructions;
34
+ RRDDIM *rd_ops_flushes;
35
+ RRDDIM *rd_ops_traversals;
36
+ RRDDIM *rd_ops_walkthroughs;
37
+ RRDDIM *rd_ops_garbage_collections;
38
+ RRDDIM *rd_ops_searches;
39
+ RRDDIM *rd_ops_inserts;
40
+ RRDDIM *rd_ops_resets;
41
+ RRDDIM *rd_ops_deletes;
42
+
43
+ RRDSET *st_callbacks;
44
+ RRDDIM *rd_callbacks_inserts;
45
+ RRDDIM *rd_callbacks_conflicts;
46
+ RRDDIM *rd_callbacks_reacts;
47
+ RRDDIM *rd_callbacks_deletes;
48
+
49
+ RRDSET *st_memory;
50
+ RRDDIM *rd_memory_indexed;
51
+ RRDDIM *rd_memory_values;
52
+ RRDDIM *rd_memory_dict;
53
+
54
+ RRDSET *st_spins;
55
+ RRDDIM *rd_spins_use;
56
+ RRDDIM *rd_spins_search;
57
+ RRDDIM *rd_spins_insert;
58
+ RRDDIM *rd_spins_delete;
59
+
60
+} dictionary_categories[] = {
61
+ { .stats = &dictionary_stats_category_collectors, },
62
+ { .stats = &dictionary_stats_category_rrdhost, },
63
+ { .stats = &dictionary_stats_category_rrdset, },
64
+ { .stats = &dictionary_stats_category_rrdcontext, },
65
+ { .stats = &dictionary_stats_category_rrdlabels, },
66
+ { .stats = &dictionary_stats_category_rrdhealth, },
67
+ { .stats = &dictionary_stats_category_functions, },
68
+ { .stats = &dictionary_stats_category_replication, },
69
+ { .stats = &dictionary_stats_category_other, },
70
+
71
+ // terminator
72
+ { .stats = NULL, NULL, NULL, 0 },
73
+};
74
+
75
+#define load_dictionary_stats_entry(x) total += (size_t)(stats.x = __atomic_load_n(&c->stats->x, __ATOMIC_RELAXED))
76
+
77
+static void update_dictionary_category_charts(struct dictionary_categories *c) {
78
+ struct dictionary_stats stats;
79
+ stats.name = c->stats->name;
80
+ int priority = 900000;
81
+ const char *family = "dictionaries";
82
+ const char *context_prefix = "dictionaries";
83
+
84
+ // ------------------------------------------------------------------------
85
+
86
+ size_t total = 0;
87
+ load_dictionary_stats_entry(dictionaries.active);
88
+ load_dictionary_stats_entry(dictionaries.deleted);
89
+
90
+ if(c->st_dicts || total != 0) {
91
+ if (unlikely(!c->st_dicts)) {
92
+ char id[RRD_ID_LENGTH_MAX + 1];
93
+ snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.dictionaries", context_prefix, stats.name);
94
+
95
+ char context[RRD_ID_LENGTH_MAX + 1];
96
+ snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.dictionaries", context_prefix);
97
+
98
+ c->st_dicts = rrdset_create_localhost(
99
+ "netdata"
100
+ , id
101
+ , NULL
102
+ , family
103
+ , context
104
+ , "Dictionaries"
105
+ , "dictionaries"
106
+ , "netdata"
107
+ , "stats"
108
+ , priority + 0
109
+ , localhost->rrd_update_every
110
+ , RRDSET_TYPE_LINE
111
+ );
112
+
113
+ c->rd_dicts_active = rrddim_add(c->st_dicts, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
114
+ c->rd_dicts_deleted = rrddim_add(c->st_dicts, "deleted", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
115
+
116
+ rrdlabels_add(c->st_dicts->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
117
+ }
118
+
119
+ rrddim_set_by_pointer(c->st_dicts, c->rd_dicts_active, (collected_number)stats.dictionaries.active);
120
+ rrddim_set_by_pointer(c->st_dicts, c->rd_dicts_deleted, (collected_number)stats.dictionaries.deleted);
121
+ rrdset_done(c->st_dicts);
122
+ }
123
+
124
+ // ------------------------------------------------------------------------
125
+
126
+ total = 0;
127
+ load_dictionary_stats_entry(items.entries);
128
+ load_dictionary_stats_entry(items.referenced);
129
+ load_dictionary_stats_entry(items.pending_deletion);
130
+
131
+ if(c->st_items || total != 0) {
132
+ if (unlikely(!c->st_items)) {
133
+ char id[RRD_ID_LENGTH_MAX + 1];
134
+ snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.items", context_prefix, stats.name);
135
+
136
+ char context[RRD_ID_LENGTH_MAX + 1];
137
+ snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.items", context_prefix);
138
+
139
+ c->st_items = rrdset_create_localhost(
140
+ "netdata"
141
+ , id
142
+ , NULL
143
+ , family
144
+ , context
145
+ , "Dictionary Items"
146
+ , "items"
147
+ , "netdata"
148
+ , "stats"
149
+ , priority + 1
150
+ , localhost->rrd_update_every
151
+ , RRDSET_TYPE_LINE
152
+ );
153
+
154
+ c->rd_items_entries = rrddim_add(c->st_items, "active", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
155
+ c->rd_items_pending_deletion = rrddim_add(c->st_items, "deleted", NULL, -1, 1, RRD_ALGORITHM_ABSOLUTE);
156
+ c->rd_items_referenced = rrddim_add(c->st_items, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
157
+
158
+ rrdlabels_add(c->st_items->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
159
+ }
160
+
161
+ rrddim_set_by_pointer(c->st_items, c->rd_items_entries, stats.items.entries);
162
+ rrddim_set_by_pointer(c->st_items, c->rd_items_pending_deletion, stats.items.pending_deletion);
163
+ rrddim_set_by_pointer(c->st_items, c->rd_items_referenced, stats.items.referenced);
164
+ rrdset_done(c->st_items);
165
+ }
166
+
167
+ // ------------------------------------------------------------------------
168
+
169
+ total = 0;
170
+ load_dictionary_stats_entry(ops.creations);
171
+ load_dictionary_stats_entry(ops.destructions);
172
+ load_dictionary_stats_entry(ops.flushes);
173
+ load_dictionary_stats_entry(ops.traversals);
174
+ load_dictionary_stats_entry(ops.walkthroughs);
175
+ load_dictionary_stats_entry(ops.garbage_collections);
176
+ load_dictionary_stats_entry(ops.searches);
177
+ load_dictionary_stats_entry(ops.inserts);
178
+ load_dictionary_stats_entry(ops.resets);
179
+ load_dictionary_stats_entry(ops.deletes);
180
+
181
+ if(c->st_ops || total != 0) {
182
+ if (unlikely(!c->st_ops)) {
183
+ char id[RRD_ID_LENGTH_MAX + 1];
184
+ snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.ops", context_prefix, stats.name);
185
+
186
+ char context[RRD_ID_LENGTH_MAX + 1];
187
+ snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.ops", context_prefix);
188
+
189
+ c->st_ops = rrdset_create_localhost(
190
+ "netdata"
191
+ , id
192
+ , NULL
193
+ , family
194
+ , context
195
+ , "Dictionary Operations"
196
+ , "ops/s"
197
+ , "netdata"
198
+ , "stats"
199
+ , priority + 2
200
+ , localhost->rrd_update_every
201
+ , RRDSET_TYPE_LINE
202
+ );
203
+
204
+ c->rd_ops_creations = rrddim_add(c->st_ops, "creations", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
205
+ c->rd_ops_destructions = rrddim_add(c->st_ops, "destructions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
206
+ c->rd_ops_flushes = rrddim_add(c->st_ops, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
207
+ c->rd_ops_traversals = rrddim_add(c->st_ops, "traversals", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
208
+ c->rd_ops_walkthroughs = rrddim_add(c->st_ops, "walkthroughs", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
209
+ c->rd_ops_garbage_collections = rrddim_add(c->st_ops, "garbage_collections", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
210
+ c->rd_ops_searches = rrddim_add(c->st_ops, "searches", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
211
+ c->rd_ops_inserts = rrddim_add(c->st_ops, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
212
+ c->rd_ops_resets = rrddim_add(c->st_ops, "resets", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
213
+ c->rd_ops_deletes = rrddim_add(c->st_ops, "deletes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
214
+
215
+ rrdlabels_add(c->st_ops->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
216
+ }
217
+
218
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_creations, (collected_number)stats.ops.creations);
219
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_destructions, (collected_number)stats.ops.destructions);
220
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_flushes, (collected_number)stats.ops.flushes);
221
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_traversals, (collected_number)stats.ops.traversals);
222
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_walkthroughs, (collected_number)stats.ops.walkthroughs);
223
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_garbage_collections, (collected_number)stats.ops.garbage_collections);
224
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_searches, (collected_number)stats.ops.searches);
225
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_inserts, (collected_number)stats.ops.inserts);
226
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_resets, (collected_number)stats.ops.resets);
227
+ rrddim_set_by_pointer(c->st_ops, c->rd_ops_deletes, (collected_number)stats.ops.deletes);
228
+
229
+ rrdset_done(c->st_ops);
230
+ }
231
+
232
+ // ------------------------------------------------------------------------
233
+
234
+ total = 0;
235
+ load_dictionary_stats_entry(callbacks.inserts);
236
+ load_dictionary_stats_entry(callbacks.conflicts);
237
+ load_dictionary_stats_entry(callbacks.reacts);
238
+ load_dictionary_stats_entry(callbacks.deletes);
239
+
240
+ if(c->st_callbacks || total != 0) {
241
+ if (unlikely(!c->st_callbacks)) {
242
+ char id[RRD_ID_LENGTH_MAX + 1];
243
+ snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.callbacks", context_prefix, stats.name);
244
+
245
+ char context[RRD_ID_LENGTH_MAX + 1];
246
+ snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.callbacks", context_prefix);
247
+
248
+ c->st_callbacks = rrdset_create_localhost(
249
+ "netdata"
250
+ , id
251
+ , NULL
252
+ , family
253
+ , context
254
+ , "Dictionary Callbacks"
255
+ , "callbacks/s"
256
+ , "netdata"
257
+ , "stats"
258
+ , priority + 3
259
+ , localhost->rrd_update_every
260
+ , RRDSET_TYPE_LINE
261
+ );
262
+
263
+ c->rd_callbacks_inserts = rrddim_add(c->st_callbacks, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
264
+ c->rd_callbacks_deletes = rrddim_add(c->st_callbacks, "deletes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
265
+ c->rd_callbacks_conflicts = rrddim_add(c->st_callbacks, "conflicts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
266
+ c->rd_callbacks_reacts = rrddim_add(c->st_callbacks, "reacts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
267
+
268
+ rrdlabels_add(c->st_callbacks->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
269
+ }
270
+
271
+ rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_inserts, (collected_number)stats.callbacks.inserts);
272
+ rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_conflicts, (collected_number)stats.callbacks.conflicts);
273
+ rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_reacts, (collected_number)stats.callbacks.reacts);
274
+ rrddim_set_by_pointer(c->st_callbacks, c->rd_callbacks_deletes, (collected_number)stats.callbacks.deletes);
275
+
276
+ rrdset_done(c->st_callbacks);
277
+ }
278
+
279
+ // ------------------------------------------------------------------------
280
+
281
+ total = 0;
282
+ load_dictionary_stats_entry(memory.index);
283
+ load_dictionary_stats_entry(memory.values);
284
+ load_dictionary_stats_entry(memory.dict);
285
+
286
+ if(c->st_memory || total != 0) {
287
+ if (unlikely(!c->st_memory)) {
288
+ char id[RRD_ID_LENGTH_MAX + 1];
289
+ snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.memory", context_prefix, stats.name);
290
+
291
+ char context[RRD_ID_LENGTH_MAX + 1];
292
+ snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.memory", context_prefix);
293
+
294
+ c->st_memory = rrdset_create_localhost(
295
+ "netdata"
296
+ , id
297
+ , NULL
298
+ , family
299
+ , context
300
+ , "Dictionary Memory"
301
+ , "bytes"
302
+ , "netdata"
303
+ , "stats"
304
+ , priority + 4
305
+ , localhost->rrd_update_every
306
+ , RRDSET_TYPE_STACKED
307
+ );
308
+
309
+ c->rd_memory_indexed = rrddim_add(c->st_memory, "index", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
310
+ c->rd_memory_values = rrddim_add(c->st_memory, "data", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
311
+ c->rd_memory_dict = rrddim_add(c->st_memory, "structures", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
312
+
313
+ rrdlabels_add(c->st_memory->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
314
+ }
315
+
316
+ rrddim_set_by_pointer(c->st_memory, c->rd_memory_indexed, (collected_number)stats.memory.index);
317
+ rrddim_set_by_pointer(c->st_memory, c->rd_memory_values, (collected_number)stats.memory.values);
318
+ rrddim_set_by_pointer(c->st_memory, c->rd_memory_dict, (collected_number)stats.memory.dict);
319
+
320
+ rrdset_done(c->st_memory);
321
+ }
322
+
323
+ // ------------------------------------------------------------------------
324
+
325
+ total = 0;
326
+ load_dictionary_stats_entry(spin_locks.use_spins);
327
+ load_dictionary_stats_entry(spin_locks.search_spins);
328
+ load_dictionary_stats_entry(spin_locks.insert_spins);
329
+ load_dictionary_stats_entry(spin_locks.delete_spins);
330
+
331
+ if(c->st_spins || total != 0) {
332
+ if (unlikely(!c->st_spins)) {
333
+ char id[RRD_ID_LENGTH_MAX + 1];
334
+ snprintfz(id, RRD_ID_LENGTH_MAX, "%s.%s.spins", context_prefix, stats.name);
335
+
336
+ char context[RRD_ID_LENGTH_MAX + 1];
337
+ snprintfz(context, RRD_ID_LENGTH_MAX, "netdata.%s.category.spins", context_prefix);
338
+
339
+ c->st_spins = rrdset_create_localhost(
340
+ "netdata"
341
+ , id
342
+ , NULL
343
+ , family
344
+ , context
345
+ , "Dictionary Spins"
346
+ , "count"
347
+ , "netdata"
348
+ , "stats"
349
+ , priority + 5
350
+ , localhost->rrd_update_every
351
+ , RRDSET_TYPE_LINE
352
+ );
353
+
354
+ c->rd_spins_use = rrddim_add(c->st_spins, "use", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
355
+ c->rd_spins_search = rrddim_add(c->st_spins, "search", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
356
+ c->rd_spins_insert = rrddim_add(c->st_spins, "insert", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
357
+ c->rd_spins_delete = rrddim_add(c->st_spins, "delete", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
358
+
359
+ rrdlabels_add(c->st_spins->rrdlabels, "category", stats.name, RRDLABEL_SRC_AUTO);
360
+ }
361
+
362
+ rrddim_set_by_pointer(c->st_spins, c->rd_spins_use, (collected_number)stats.spin_locks.use_spins);
363
+ rrddim_set_by_pointer(c->st_spins, c->rd_spins_search, (collected_number)stats.spin_locks.search_spins);
364
+ rrddim_set_by_pointer(c->st_spins, c->rd_spins_insert, (collected_number)stats.spin_locks.insert_spins);
365
+ rrddim_set_by_pointer(c->st_spins, c->rd_spins_delete, (collected_number)stats.spin_locks.delete_spins);
366
+
367
+ rrdset_done(c->st_spins);
368
+ }
369
+}
370
+
371
+void telemetry_dictionary_do(bool extended) {
372
+ if(!extended) return;
373
+
374
+ for(int i = 0; dictionary_categories[i].stats ;i++) {
375
+ update_dictionary_category_charts(&dictionary_categories[i]);
376
+ }
377
+}
378
+#endif // DICT_WITH_STATS
src/daemon/telemetry/telemetry-dictionary.h
new
+24
@@ -0,0 +1,24 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_DICTIONARY_H
4
+#define NETDATA_TELEMETRY_DICTIONARY_H
5
+
6
+#include "daemon/common.h"
7
+
8
+extern struct dictionary_stats dictionary_stats_category_collectors;
9
+extern struct dictionary_stats dictionary_stats_category_rrdhost;
10
+extern struct dictionary_stats dictionary_stats_category_rrdset;
11
+extern struct dictionary_stats dictionary_stats_category_rrddim;
12
+extern struct dictionary_stats dictionary_stats_category_rrdcontext;
13
+extern struct dictionary_stats dictionary_stats_category_rrdlabels;
14
+extern struct dictionary_stats dictionary_stats_category_rrdhealth;
15
+extern struct dictionary_stats dictionary_stats_category_functions;
16
+extern struct dictionary_stats dictionary_stats_category_replication;
17
+
18
+extern size_t rrddim_db_memory_size;
19
+
20
+#if defined(TELEMETRY_INTERNALS)
21
+void telemetry_dictionary_do(bool extended);
22
+#endif
23
+
24
+#endif //NETDATA_TELEMETRY_DICTIONARY_H
src/daemon/telemetry/telemetry-gorilla.c
new
+110
@@ -0,0 +1,110 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-gorilla.h"
5
+
6
+static struct gorilla_statistics {
7
+ bool enabled;
8
+
9
+ alignas(64) uint64_t tier0_hot_gorilla_buffers;
10
+
11
+ alignas(64) uint64_t gorilla_tier0_disk_actual_bytes;
12
+ alignas(64) uint64_t gorilla_tier0_disk_optimal_bytes;
13
+ alignas(64) uint64_t gorilla_tier0_disk_original_bytes;
14
+} gorilla_statistics = { 0 };
15
+
16
+void telemetry_gorilla_hot_buffer_added() {
17
+ if(!gorilla_statistics.enabled) return;
18
+
19
+ __atomic_fetch_add(&gorilla_statistics.tier0_hot_gorilla_buffers, 1, __ATOMIC_RELAXED);
20
+}
21
+
22
+void telemetry_gorilla_tier0_page_flush(uint32_t actual, uint32_t optimal, uint32_t original) {
23
+ if(!gorilla_statistics.enabled) return;
24
+
25
+ __atomic_fetch_add(&gorilla_statistics.gorilla_tier0_disk_actual_bytes, actual, __ATOMIC_RELAXED);
26
+ __atomic_fetch_add(&gorilla_statistics.gorilla_tier0_disk_optimal_bytes, optimal, __ATOMIC_RELAXED);
27
+ __atomic_fetch_add(&gorilla_statistics.gorilla_tier0_disk_original_bytes, original, __ATOMIC_RELAXED);
28
+}
29
+
30
+static inline void global_statistics_copy(struct gorilla_statistics *gs) {
31
+ gs->tier0_hot_gorilla_buffers = __atomic_load_n(&gorilla_statistics.tier0_hot_gorilla_buffers, __ATOMIC_RELAXED);
32
+ gs->gorilla_tier0_disk_actual_bytes = __atomic_load_n(&gorilla_statistics.gorilla_tier0_disk_actual_bytes, __ATOMIC_RELAXED);
33
+ gs->gorilla_tier0_disk_optimal_bytes = __atomic_load_n(&gorilla_statistics.gorilla_tier0_disk_optimal_bytes, __ATOMIC_RELAXED);
34
+ gs->gorilla_tier0_disk_original_bytes = __atomic_load_n(&gorilla_statistics.gorilla_tier0_disk_original_bytes, __ATOMIC_RELAXED);
35
+}
36
+
37
+void telemetry_gorilla_do(bool extended __maybe_unused) {
38
+#ifdef ENABLE_DBENGINE
39
+ if(!extended) return;
40
+ gorilla_statistics.enabled = true;
41
+
42
+ struct gorilla_statistics gs;
43
+ global_statistics_copy(&gs);
44
+
45
+ if (tier_page_type[0] == RRDENG_PAGE_TYPE_GORILLA_32BIT)
46
+ {
47
+ static RRDSET *st_tier0_gorilla_pages = NULL;
48
+ static RRDDIM *rd_num_gorilla_pages = NULL;
49
+
50
+ if (unlikely(!st_tier0_gorilla_pages)) {
51
+ st_tier0_gorilla_pages = rrdset_create_localhost(
52
+ "netdata"
53
+ , "tier0_gorilla_pages"
54
+ , NULL
55
+ , "dbengine gorilla"
56
+ , NULL
57
+ , "Number of gorilla_pages"
58
+ , "count"
59
+ , "netdata"
60
+ , "stats"
61
+ , 131004
62
+ , localhost->rrd_update_every
63
+ , RRDSET_TYPE_LINE
64
+ );
65
+
66
+ rd_num_gorilla_pages = rrddim_add(st_tier0_gorilla_pages, "count", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
67
+ }
68
+
69
+ rrddim_set_by_pointer(st_tier0_gorilla_pages, rd_num_gorilla_pages, (collected_number)gs.tier0_hot_gorilla_buffers);
70
+
71
+ rrdset_done(st_tier0_gorilla_pages);
72
+ }
73
+
74
+ if (tier_page_type[0] == RRDENG_PAGE_TYPE_GORILLA_32BIT)
75
+ {
76
+ static RRDSET *st_tier0_compression_info = NULL;
77
+
78
+ static RRDDIM *rd_actual_bytes = NULL;
79
+ static RRDDIM *rd_optimal_bytes = NULL;
80
+ static RRDDIM *rd_uncompressed_bytes = NULL;
81
+
82
+ if (unlikely(!st_tier0_compression_info)) {
83
+ st_tier0_compression_info = rrdset_create_localhost(
84
+ "netdata"
85
+ , "tier0_gorilla_efficiency"
86
+ , NULL
87
+ , "dbengine gorilla"
88
+ , NULL
89
+ , "DBENGINE Gorilla Compression Efficiency on Tier 0"
90
+ , "bytes"
91
+ , "netdata"
92
+ , "stats"
93
+ , 131005
94
+ , localhost->rrd_update_every
95
+ , RRDSET_TYPE_LINE
96
+ );
97
+
98
+ rd_actual_bytes = rrddim_add(st_tier0_compression_info, "actual", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
99
+ rd_optimal_bytes = rrddim_add(st_tier0_compression_info, "optimal", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
100
+ rd_uncompressed_bytes = rrddim_add(st_tier0_compression_info, "uncompressed", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
101
+ }
102
+
103
+ rrddim_set_by_pointer(st_tier0_compression_info, rd_actual_bytes, (collected_number)gs.gorilla_tier0_disk_actual_bytes);
104
+ rrddim_set_by_pointer(st_tier0_compression_info, rd_optimal_bytes, (collected_number)gs.gorilla_tier0_disk_optimal_bytes);
105
+ rrddim_set_by_pointer(st_tier0_compression_info, rd_uncompressed_bytes, (collected_number)gs.gorilla_tier0_disk_original_bytes);
106
+
107
+ rrdset_done(st_tier0_compression_info);
108
+ }
109
+#endif
110
+}
src/daemon/telemetry/telemetry-gorilla.h
new
+15
@@ -0,0 +1,15 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_GORILLA_H
4
+#define NETDATA_TELEMETRY_GORILLA_H
5
+
6
+#include "daemon/common.h"
7
+
8
+void telemetry_gorilla_hot_buffer_added();
9
+void telemetry_gorilla_tier0_page_flush(uint32_t actual, uint32_t optimal, uint32_t original);
10
+
11
+#if defined(TELEMETRY_INTERNALS)
12
+void telemetry_gorilla_do(bool extended);
13
+#endif
14
+
15
+#endif //NETDATA_TELEMETRY_GORILLA_H
src/daemon/telemetry/telemetry-heartbeat.c
new
+44
@@ -0,0 +1,44 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-heartbeat.h"
5
+
6
+void telemetry_heartbeat_do(bool extended) {
7
+ if(!extended) return;
8
+
9
+ static RRDSET *st_heartbeat = NULL;
10
+ static RRDDIM *rd_heartbeat_min = NULL;
11
+ static RRDDIM *rd_heartbeat_max = NULL;
12
+ static RRDDIM *rd_heartbeat_avg = NULL;
13
+
14
+ if (unlikely(!st_heartbeat)) {
15
+ st_heartbeat = rrdset_create_localhost(
16
+ "netdata"
17
+ , "heartbeat"
18
+ , NULL
19
+ , "heartbeat"
20
+ , NULL
21
+ , "System clock jitter"
22
+ , "microseconds"
23
+ , "netdata"
24
+ , "stats"
25
+ , 900000
26
+ , localhost->rrd_update_every
27
+ , RRDSET_TYPE_AREA);
28
+
29
+ rd_heartbeat_min = rrddim_add(st_heartbeat, "min", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
30
+ rd_heartbeat_max = rrddim_add(st_heartbeat, "max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
31
+ rd_heartbeat_avg = rrddim_add(st_heartbeat, "average", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
32
+ }
33
+
34
+ usec_t min, max, average;
35
+ size_t count;
36
+
37
+ heartbeat_statistics(&min, &max, &average, &count);
38
+
39
+ rrddim_set_by_pointer(st_heartbeat, rd_heartbeat_min, (collected_number)min);
40
+ rrddim_set_by_pointer(st_heartbeat, rd_heartbeat_max, (collected_number)max);
41
+ rrddim_set_by_pointer(st_heartbeat, rd_heartbeat_avg, (collected_number)average);
42
+
43
+ rrdset_done(st_heartbeat);
44
+}
src/daemon/telemetry/telemetry-heartbeat.h
new
+12
@@ -0,0 +1,12 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_HEARTBEAT_H
4
+#define NETDATA_TELEMETRY_HEARTBEAT_H
5
+
6
+#include "daemon/common.h"
7
+
8
+#if defined(TELEMETRY_INTERNALS)
9
+void telemetry_heartbeat_do(bool extended);
10
+#endif
11
+
12
+#endif //NETDATA_TELEMETRY_HEARTBEAT_H
src/daemon/telemetry/telemetry-http-api.c
new
+263
@@ -0,0 +1,263 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-http-api.h"
5
+
6
+#define GLOBAL_STATS_RESET_WEB_USEC_MAX 0x01
7
+
8
+static struct web_statistics {
9
+ bool extended;
10
+
11
+ uint16_t connected_clients;
12
+ uint64_t web_client_count; // oops! this is used for giving unique IDs to web_clients!
13
+
14
+ uint64_t web_requests;
15
+ uint64_t web_usec;
16
+ uint64_t web_usec_max;
17
+ uint64_t bytes_received;
18
+ uint64_t bytes_sent;
19
+
20
+ uint64_t content_size_uncompressed;
21
+ uint64_t content_size_compressed;
22
+} web_statistics;
23
+
24
+uint64_t telemetry_web_client_connected(void) {
25
+ __atomic_fetch_add(&web_statistics.connected_clients, 1, __ATOMIC_RELAXED);
26
+ return __atomic_fetch_add(&web_statistics.web_client_count, 1, __ATOMIC_RELAXED);
27
+}
28
+
29
+void telemetry_web_client_disconnected(void) {
30
+ __atomic_fetch_sub(&web_statistics.connected_clients, 1, __ATOMIC_RELAXED);
31
+}
32
+
33
+void telemetry_web_request_completed(uint64_t dt,
34
+ uint64_t bytes_received,
35
+ uint64_t bytes_sent,
36
+ uint64_t content_size,
37
+ uint64_t compressed_content_size) {
38
+ uint64_t old_web_usec_max = web_statistics.web_usec_max;
39
+ while(dt > old_web_usec_max)
40
+ __atomic_compare_exchange(&web_statistics.web_usec_max, &old_web_usec_max, &dt, 1, __ATOMIC_RELAXED, __ATOMIC_RELAXED);
41
+
42
+ __atomic_fetch_add(&web_statistics.web_requests, 1, __ATOMIC_RELAXED);
43
+ __atomic_fetch_add(&web_statistics.web_usec, dt, __ATOMIC_RELAXED);
44
+ __atomic_fetch_add(&web_statistics.bytes_received, bytes_received, __ATOMIC_RELAXED);
45
+ __atomic_fetch_add(&web_statistics.bytes_sent, bytes_sent, __ATOMIC_RELAXED);
46
+ __atomic_fetch_add(&web_statistics.content_size_uncompressed, content_size, __ATOMIC_RELAXED);
47
+ __atomic_fetch_add(&web_statistics.content_size_compressed, compressed_content_size, __ATOMIC_RELAXED);
48
+}
49
+
50
+static inline void telemetry_web_copy(struct web_statistics *gs, uint8_t options) {
51
+ gs->connected_clients = __atomic_load_n(&web_statistics.connected_clients, __ATOMIC_RELAXED);
52
+ gs->web_requests = __atomic_load_n(&web_statistics.web_requests, __ATOMIC_RELAXED);
53
+ gs->web_usec = __atomic_load_n(&web_statistics.web_usec, __ATOMIC_RELAXED);
54
+ gs->web_usec_max = __atomic_load_n(&web_statistics.web_usec_max, __ATOMIC_RELAXED);
55
+ gs->bytes_received = __atomic_load_n(&web_statistics.bytes_received, __ATOMIC_RELAXED);
56
+ gs->bytes_sent = __atomic_load_n(&web_statistics.bytes_sent, __ATOMIC_RELAXED);
57
+ gs->content_size_uncompressed = __atomic_load_n(&web_statistics.content_size_uncompressed, __ATOMIC_RELAXED);
58
+ gs->content_size_compressed = __atomic_load_n(&web_statistics.content_size_compressed, __ATOMIC_RELAXED);
59
+ gs->web_client_count = __atomic_load_n(&web_statistics.web_client_count, __ATOMIC_RELAXED);
60
+
61
+ if(options & GLOBAL_STATS_RESET_WEB_USEC_MAX) {
62
+ uint64_t n = 0;
63
+ __atomic_compare_exchange(&web_statistics.web_usec_max, (uint64_t *) &gs->web_usec_max, &n, 1, __ATOMIC_RELAXED, __ATOMIC_RELAXED);
64
+ }
65
+}
66
+
67
+void telemetry_web_do(bool extended) {
68
+ static struct web_statistics gs;
69
+ telemetry_web_copy(&gs, GLOBAL_STATS_RESET_WEB_USEC_MAX);
70
+
71
+ // ----------------------------------------------------------------
72
+
73
+ {
74
+ static RRDSET *st_clients = NULL;
75
+ static RRDDIM *rd_clients = NULL;
76
+
77
+ if (unlikely(!st_clients)) {
78
+ st_clients = rrdset_create_localhost(
79
+ "netdata"
80
+ , "clients"
81
+ , NULL
82
+ , "HTTP API"
83
+ , NULL
84
+ , "Netdata Web API Clients"
85
+ , "connected clients"
86
+ , "netdata"
87
+ , "stats"
88
+ , 130200
89
+ , localhost->rrd_update_every
90
+ , RRDSET_TYPE_LINE
91
+ );
92
+
93
+ rd_clients = rrddim_add(st_clients, "clients", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
94
+ }
95
+
96
+ rrddim_set_by_pointer(st_clients, rd_clients, gs.connected_clients);
97
+ rrdset_done(st_clients);
98
+ }
99
+
100
+ // ----------------------------------------------------------------
101
+
102
+ {
103
+ static RRDSET *st_reqs = NULL;
104
+ static RRDDIM *rd_requests = NULL;
105
+
106
+ if (unlikely(!st_reqs)) {
107
+ st_reqs = rrdset_create_localhost(
108
+ "netdata"
109
+ , "requests"
110
+ , NULL
111
+ , "HTTP API"
112
+ , NULL
113
+ , "Netdata Web API Requests Received"
114
+ , "requests/s"
115
+ , "netdata"
116
+ , "stats"
117
+ , 130300
118
+ , localhost->rrd_update_every
119
+ , RRDSET_TYPE_LINE
120
+ );
121
+
122
+ rd_requests = rrddim_add(st_reqs, "requests", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
123
+ }
124
+
125
+ rrddim_set_by_pointer(st_reqs, rd_requests, (collected_number) gs.web_requests);
126
+ rrdset_done(st_reqs);
127
+ }
128
+
129
+ // ----------------------------------------------------------------
130
+
131
+ {
132
+ static RRDSET *st_bytes = NULL;
133
+ static RRDDIM *rd_in = NULL,
134
+ *rd_out = NULL;
135
+
136
+ if (unlikely(!st_bytes)) {
137
+ st_bytes = rrdset_create_localhost(
138
+ "netdata"
139
+ , "net"
140
+ , NULL
141
+ , "HTTP API"
142
+ , NULL
143
+ , "Netdata Web API Network Traffic"
144
+ , "kilobits/s"
145
+ , "netdata"
146
+ , "stats"
147
+ , 130400
148
+ , localhost->rrd_update_every
149
+ , RRDSET_TYPE_AREA
150
+ );
151
+
152
+ rd_in = rrddim_add(st_bytes, "in", NULL, 8, BITS_IN_A_KILOBIT, RRD_ALGORITHM_INCREMENTAL);
153
+ rd_out = rrddim_add(st_bytes, "out", NULL, -8, BITS_IN_A_KILOBIT, RRD_ALGORITHM_INCREMENTAL);
154
+ }
155
+
156
+ rrddim_set_by_pointer(st_bytes, rd_in, (collected_number) gs.bytes_received);
157
+ rrddim_set_by_pointer(st_bytes, rd_out, (collected_number) gs.bytes_sent);
158
+ rrdset_done(st_bytes);
159
+ }
160
+
161
+ // ----------------------------------------------------------------
162
+
163
+ {
164
+ static unsigned long long old_web_requests = 0, old_web_usec = 0;
165
+ static collected_number average_response_time = -1;
166
+
167
+ static RRDSET *st_duration = NULL;
168
+ static RRDDIM *rd_average = NULL,
169
+ *rd_max = NULL;
170
+
171
+ if (unlikely(!st_duration)) {
172
+ st_duration = rrdset_create_localhost(
173
+ "netdata"
174
+ , "response_time"
175
+ , NULL
176
+ , "HTTP API"
177
+ , NULL
178
+ , "Netdata Web API Response Time"
179
+ , "milliseconds/request"
180
+ , "netdata"
181
+ , "stats"
182
+ , 130500
183
+ , localhost->rrd_update_every
184
+ , RRDSET_TYPE_LINE
185
+ );
186
+
187
+ rd_average = rrddim_add(st_duration, "average", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
188
+ rd_max = rrddim_add(st_duration, "max", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
189
+ }
190
+
191
+ uint64_t gweb_usec = gs.web_usec;
192
+ uint64_t gweb_requests = gs.web_requests;
193
+
194
+ uint64_t web_usec = (gweb_usec >= old_web_usec) ? gweb_usec - old_web_usec : 0;
195
+ uint64_t web_requests = (gweb_requests >= old_web_requests) ? gweb_requests - old_web_requests : 0;
196
+
197
+ old_web_usec = gweb_usec;
198
+ old_web_requests = gweb_requests;
199
+
200
+ if (web_requests)
201
+ average_response_time = (collected_number) (web_usec / web_requests);
202
+
203
+ if (unlikely(average_response_time != -1))
204
+ rrddim_set_by_pointer(st_duration, rd_average, average_response_time);
205
+ else
206
+ rrddim_set_by_pointer(st_duration, rd_average, 0);
207
+
208
+ rrddim_set_by_pointer(st_duration, rd_max, ((gs.web_usec_max)?(collected_number)gs.web_usec_max:average_response_time));
209
+ rrdset_done(st_duration);
210
+ }
211
+
212
+ // ----------------------------------------------------------------
213
+
214
+ if(!extended) return;
215
+
216
+ // ----------------------------------------------------------------
217
+
218
+ {
219
+ static unsigned long long old_content_size = 0, old_compressed_content_size = 0;
220
+ static collected_number compression_ratio = -1;
221
+
222
+ static RRDSET *st_compression = NULL;
223
+ static RRDDIM *rd_savings = NULL;
224
+
225
+ if (unlikely(!st_compression)) {
226
+ st_compression = rrdset_create_localhost(
227
+ "netdata"
228
+ , "compression_ratio"
229
+ , NULL
230
+ , "HTTP API"
231
+ , NULL
232
+ , "Netdata Web API Responses Compression Savings Ratio"
233
+ , "percentage"
234
+ , "netdata"
235
+ , "stats"
236
+ , 130600
237
+ , localhost->rrd_update_every
238
+ , RRDSET_TYPE_LINE
239
+ );
240
+
241
+ rd_savings = rrddim_add(st_compression, "savings", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
242
+ }
243
+
244
+ // since we don't lock here to read the telemetry
245
+ // read the smaller value first
246
+ unsigned long long gcompressed_content_size = gs.content_size_compressed;
247
+ unsigned long long gcontent_size = gs.content_size_uncompressed;
248
+
249
+ unsigned long long compressed_content_size = gcompressed_content_size - old_compressed_content_size;
250
+ unsigned long long content_size = gcontent_size - old_content_size;
251
+
252
+ old_compressed_content_size = gcompressed_content_size;
253
+ old_content_size = gcontent_size;
254
+
255
+ if (content_size && content_size >= compressed_content_size)
256
+ compression_ratio = ((content_size - compressed_content_size) * 100 * 1000) / content_size;
257
+
258
+ if (compression_ratio != -1)
259
+ rrddim_set_by_pointer(st_compression, rd_savings, compression_ratio);
260
+
261
+ rrdset_done(st_compression);
262
+ }
263
+}
\ No newline at end of file
src/daemon/telemetry/telemetry-http-api.h
new
+21
@@ -0,0 +1,21 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_HTTP_API_H
4
+#define NETDATA_TELEMETRY_HTTP_API_H
5
+
6
+#include "daemon/common.h"
7
+
8
+uint64_t telemetry_web_client_connected(void);
9
+void telemetry_web_client_disconnected(void);
10
+
11
+void telemetry_web_request_completed(uint64_t dt,
12
+ uint64_t bytes_received,
13
+ uint64_t bytes_sent,
14
+ uint64_t content_size,
15
+ uint64_t compressed_content_size);
16
+
17
+#if defined(TELEMETRY_INTERNALS)
18
+void telemetry_web_do(bool extended);
19
+#endif
20
+
21
+#endif //NETDATA_TELEMETRY_HTTP_API_H
src/daemon/telemetry/telemetry-ingestion.c
new
+58
@@ -0,0 +1,58 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-ingestion.h"
5
+
6
+static struct ingest_statistics {
7
+ uint64_t db_points_stored_per_tier[RRD_STORAGE_TIERS];
8
+} ingest_statistics;
9
+
10
+void telemetry_queries_rrdset_collection_completed(size_t *points_read_per_tier_array) {
11
+ for(size_t tier = 0; tier < storage_tiers ;tier++) {
12
+ __atomic_fetch_add(&ingest_statistics.db_points_stored_per_tier[tier], points_read_per_tier_array[tier], __ATOMIC_RELAXED);
13
+ points_read_per_tier_array[tier] = 0;
14
+ }
15
+}
16
+
17
+static inline void telemetry_ingestion_copy(struct ingest_statistics *gs) {
18
+ for(size_t tier = 0; tier < storage_tiers ;tier++)
19
+ gs->db_points_stored_per_tier[tier] = __atomic_load_n(&ingest_statistics.db_points_stored_per_tier[tier], __ATOMIC_RELAXED);
20
+}
21
+
22
+void telemetry_ingestion_do(bool extended __maybe_unused) {
23
+ static struct ingest_statistics gs;
24
+ telemetry_ingestion_copy(&gs);
25
+
26
+ {
27
+ static RRDSET *st_points_stored = NULL;
28
+ static RRDDIM *rds[RRD_STORAGE_TIERS] = {};
29
+
30
+ if (unlikely(!st_points_stored)) {
31
+ st_points_stored = rrdset_create_localhost(
32
+ "netdata"
33
+ , "db_samples_collected"
34
+ , NULL
35
+ , "Data Collection Samples"
36
+ , NULL
37
+ , "Netdata Time-Series Collected Samples"
38
+ , "samples/s"
39
+ , "netdata"
40
+ , "stats"
41
+ , 131003
42
+ , localhost->rrd_update_every
43
+ , RRDSET_TYPE_STACKED
44
+ );
45
+
46
+ for(size_t tier = 0; tier < storage_tiers ;tier++) {
47
+ char buf[30 + 1];
48
+ snprintfz(buf, sizeof(buf) - 1, "tier%zu", tier);
49
+ rds[tier] = rrddim_add(st_points_stored, buf, NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
50
+ }
51
+ }
52
+
53
+ for(size_t tier = 0; tier < storage_tiers ;tier++)
54
+ rrddim_set_by_pointer(st_points_stored, rds[tier], (collected_number)gs.db_points_stored_per_tier[tier]);
55
+
56
+ rrdset_done(st_points_stored);
57
+ }
58
+}
src/daemon/telemetry/telemetry-ingestion.h
new
+14
@@ -0,0 +1,14 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_INGESTION_H
4
+#define NETDATA_TELEMETRY_INGESTION_H
5
+
6
+#include "daemon/common.h"
7
+
8
+void telemetry_queries_rrdset_collection_completed(size_t *points_read_per_tier_array);
9
+
10
+#if defined(TELEMETRY_INTERNALS)
11
+void telemetry_ingestion_do(bool extended);
12
+#endif
13
+
14
+#endif //NETDATA_TELEMETRY_INGESTION_H
src/daemon/telemetry/telemetry-ml.c
new
+89
@@ -0,0 +1,89 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-ml.h"
5
+
6
+static struct ml_statistics {
7
+ alignas(64) uint64_t ml_models_consulted;
8
+ alignas(64) uint64_t ml_models_received;
9
+ alignas(64) uint64_t ml_models_ignored;
10
+ alignas(64) uint64_t ml_models_sent;
11
+ alignas(64) uint64_t ml_models_deserialization_failures;
12
+ alignas(64) uint64_t ml_memory_consumption;
13
+ alignas(64) uint64_t ml_memory_new;
14
+ alignas(64) uint64_t ml_memory_delete;
15
+} ml_statistics = {0};
16
+
17
+void telemetry_ml_models_received()
18
+{
19
+ __atomic_fetch_add(&ml_statistics.ml_models_received, 1, __ATOMIC_RELAXED);
20
+}
21
+
22
+void telemetry_ml_models_ignored()
23
+{
24
+ __atomic_fetch_add(&ml_statistics.ml_models_ignored, 1, __ATOMIC_RELAXED);
25
+}
26
+
27
+void telemetry_ml_models_sent()
28
+{
29
+ __atomic_fetch_add(&ml_statistics.ml_models_sent, 1, __ATOMIC_RELAXED);
30
+}
31
+
32
+void global_statistics_ml_models_deserialization_failures()
33
+{
34
+ __atomic_fetch_add(&ml_statistics.ml_models_deserialization_failures, 1, __ATOMIC_RELAXED);
35
+}
36
+
37
+void telemetry_ml_models_consulted(size_t models_consulted)
38
+{
39
+ __atomic_fetch_add(&ml_statistics.ml_models_consulted, models_consulted, __ATOMIC_RELAXED);
40
+}
41
+
42
+void telemetry_ml_memory_allocated(size_t n)
43
+{
44
+ __atomic_fetch_add(&ml_statistics.ml_memory_consumption, n, __ATOMIC_RELAXED);
45
+ __atomic_fetch_add(&ml_statistics.ml_memory_new, 1, __ATOMIC_RELAXED);
46
+}
47
+
48
+void telemetry_ml_memory_freed(size_t n)
49
+{
50
+ __atomic_fetch_sub(&ml_statistics.ml_memory_consumption, n, __ATOMIC_RELAXED);
51
+ __atomic_fetch_add(&ml_statistics.ml_memory_delete, 1, __ATOMIC_RELAXED);
52
+}
53
+
54
+uint64_t telemetry_ml_get_current_memory_usage(void) {
55
+ return __atomic_load_n(&ml_statistics.ml_memory_consumption, __ATOMIC_RELAXED);
56
+}
57
+
58
+static inline void ml_statistics_copy(struct ml_statistics *gs)
59
+{
60
+ gs->ml_models_consulted = __atomic_load_n(&ml_statistics.ml_models_consulted, __ATOMIC_RELAXED);
61
+ gs->ml_models_received = __atomic_load_n(&ml_statistics.ml_models_received, __ATOMIC_RELAXED);
62
+ gs->ml_models_sent = __atomic_load_n(&ml_statistics.ml_models_sent, __ATOMIC_RELAXED);
63
+ gs->ml_models_ignored = __atomic_load_n(&ml_statistics.ml_models_ignored, __ATOMIC_RELAXED);
64
+ gs->ml_models_deserialization_failures =
65
+ __atomic_load_n(&ml_statistics.ml_models_deserialization_failures, __ATOMIC_RELAXED);
66
+
67
+ gs->ml_memory_consumption = __atomic_load_n(&ml_statistics.ml_memory_consumption, __ATOMIC_RELAXED);
68
+ gs->ml_memory_new = __atomic_load_n(&ml_statistics.ml_memory_new, __ATOMIC_RELAXED);
69
+ gs->ml_memory_delete = __atomic_load_n(&ml_statistics.ml_memory_delete, __ATOMIC_RELAXED);
70
+}
71
+
72
+void telemetry_ml_do(bool extended)
73
+{
74
+ if (!extended)
75
+ return;
76
+
77
+ struct ml_statistics gs;
78
+ ml_statistics_copy(&gs);
79
+
80
+ ml_update_global_statistics_charts(
81
+ gs.ml_models_consulted,
82
+ gs.ml_models_received,
83
+ gs.ml_models_sent,
84
+ gs.ml_models_ignored,
85
+ gs.ml_models_deserialization_failures,
86
+ gs.ml_memory_consumption,
87
+ gs.ml_memory_new,
88
+ gs.ml_memory_delete);
89
+}
src/daemon/telemetry/telemetry-ml.h
new
+33
@@ -0,0 +1,33 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_ML_H
4
+#define NETDATA_TELEMETRY_ML_H
5
+
6
+#include "daemon/common.h"
7
+
8
+#ifdef __cplusplus
9
+extern "C" {
10
+#endif
11
+
12
+void telemetry_ml_models_consulted(size_t models_consulted);
13
+void telemetry_ml_models_received();
14
+void telemetry_ml_models_ignored();
15
+void telemetry_ml_models_sent();
16
+
17
+void telemetry_ml_memory_allocated(size_t n);
18
+void telemetry_ml_memory_freed(size_t n);
19
+
20
+void global_statistics_ml_models_deserialization_failures();
21
+
22
+uint64_t telemetry_ml_get_current_memory_usage(void);
23
+
24
+#if defined(TELEMETRY_INTERNALS)
25
+void telemetry_ml_do(bool extended);
26
+#endif
27
+
28
+#ifdef __cplusplus
29
+}
30
+#endif
31
+
32
+
33
+#endif //NETDATA_TELEMETRY_ML_H
src/daemon/telemetry/telemetry-queries.c
new
+261
@@ -0,0 +1,261 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-queries.h"
5
+
6
+static struct query_statistics {
7
+ uint64_t api_data_queries_made;
8
+ uint64_t api_data_db_points_read;
9
+ uint64_t api_data_result_points_generated;
10
+
11
+ uint64_t api_weights_queries_made;
12
+ uint64_t api_weights_db_points_read;
13
+ uint64_t api_weights_result_points_generated;
14
+
15
+ uint64_t api_badges_queries_made;
16
+ uint64_t api_badges_db_points_read;
17
+ uint64_t api_badges_result_points_generated;
18
+
19
+ uint64_t health_queries_made;
20
+ uint64_t health_db_points_read;
21
+ uint64_t health_result_points_generated;
22
+
23
+ uint64_t ml_queries_made;
24
+ uint64_t ml_db_points_read;
25
+ uint64_t ml_result_points_generated;
26
+
27
+ uint64_t backfill_queries_made;
28
+ uint64_t backfill_db_points_read;
29
+
30
+ uint64_t exporters_queries_made;
31
+ uint64_t exporters_db_points_read;
32
+} query_statistics;
33
+
34
+void telemetry_queries_ml_query_completed(size_t points_read) {
35
+ __atomic_fetch_add(&query_statistics.ml_queries_made, 1, __ATOMIC_RELAXED);
36
+ __atomic_fetch_add(&query_statistics.ml_db_points_read, points_read, __ATOMIC_RELAXED);
37
+}
38
+
39
+void telemetry_queries_exporters_query_completed(size_t points_read) {
40
+ __atomic_fetch_add(&query_statistics.exporters_queries_made, 1, __ATOMIC_RELAXED);
41
+ __atomic_fetch_add(&query_statistics.exporters_db_points_read, points_read, __ATOMIC_RELAXED);
42
+}
43
+
44
+void telemetry_queries_backfill_query_completed(size_t points_read) {
45
+ __atomic_fetch_add(&query_statistics.backfill_queries_made, 1, __ATOMIC_RELAXED);
46
+ __atomic_fetch_add(&query_statistics.backfill_db_points_read, points_read, __ATOMIC_RELAXED);
47
+}
48
+
49
+void telemetry_queries_rrdr_query_completed(size_t queries, uint64_t db_points_read, uint64_t result_points_generated, QUERY_SOURCE query_source) {
50
+ switch(query_source) {
51
+ case QUERY_SOURCE_API_DATA:
52
+ __atomic_fetch_add(&query_statistics.api_data_queries_made, queries, __ATOMIC_RELAXED);
53
+ __atomic_fetch_add(&query_statistics.api_data_db_points_read, db_points_read, __ATOMIC_RELAXED);
54
+ __atomic_fetch_add(&query_statistics.api_data_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
55
+ break;
56
+
57
+ case QUERY_SOURCE_ML:
58
+ __atomic_fetch_add(&query_statistics.ml_queries_made, queries, __ATOMIC_RELAXED);
59
+ __atomic_fetch_add(&query_statistics.ml_db_points_read, db_points_read, __ATOMIC_RELAXED);
60
+ __atomic_fetch_add(&query_statistics.ml_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
61
+ break;
62
+
63
+ case QUERY_SOURCE_API_WEIGHTS:
64
+ __atomic_fetch_add(&query_statistics.api_weights_queries_made, queries, __ATOMIC_RELAXED);
65
+ __atomic_fetch_add(&query_statistics.api_weights_db_points_read, db_points_read, __ATOMIC_RELAXED);
66
+ __atomic_fetch_add(&query_statistics.api_weights_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
67
+ break;
68
+
69
+ case QUERY_SOURCE_API_BADGE:
70
+ __atomic_fetch_add(&query_statistics.api_badges_queries_made, queries, __ATOMIC_RELAXED);
71
+ __atomic_fetch_add(&query_statistics.api_badges_db_points_read, db_points_read, __ATOMIC_RELAXED);
72
+ __atomic_fetch_add(&query_statistics.api_badges_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
73
+ break;
74
+
75
+ case QUERY_SOURCE_HEALTH:
76
+ __atomic_fetch_add(&query_statistics.health_queries_made, queries, __ATOMIC_RELAXED);
77
+ __atomic_fetch_add(&query_statistics.health_db_points_read, db_points_read, __ATOMIC_RELAXED);
78
+ __atomic_fetch_add(&query_statistics.health_result_points_generated, result_points_generated, __ATOMIC_RELAXED);
79
+ break;
80
+
81
+ default:
82
+ case QUERY_SOURCE_UNITTEST:
83
+ case QUERY_SOURCE_UNKNOWN:
84
+ break;
85
+ }
86
+}
87
+
88
+static inline void telemetry_queries_copy(struct query_statistics *gs) {
89
+ gs->api_data_queries_made = __atomic_load_n(&query_statistics.api_data_queries_made, __ATOMIC_RELAXED);
90
+ gs->api_data_db_points_read = __atomic_load_n(&query_statistics.api_data_db_points_read, __ATOMIC_RELAXED);
91
+ gs->api_data_result_points_generated = __atomic_load_n(&query_statistics.api_data_result_points_generated, __ATOMIC_RELAXED);
92
+
93
+ gs->api_weights_queries_made = __atomic_load_n(&query_statistics.api_weights_queries_made, __ATOMIC_RELAXED);
94
+ gs->api_weights_db_points_read = __atomic_load_n(&query_statistics.api_weights_db_points_read, __ATOMIC_RELAXED);
95
+ gs->api_weights_result_points_generated = __atomic_load_n(&query_statistics.api_weights_result_points_generated, __ATOMIC_RELAXED);
96
+
97
+ gs->api_badges_queries_made = __atomic_load_n(&query_statistics.api_badges_queries_made, __ATOMIC_RELAXED);
98
+ gs->api_badges_db_points_read = __atomic_load_n(&query_statistics.api_badges_db_points_read, __ATOMIC_RELAXED);
99
+ gs->api_badges_result_points_generated = __atomic_load_n(&query_statistics.api_badges_result_points_generated, __ATOMIC_RELAXED);
100
+
101
+ gs->health_queries_made = __atomic_load_n(&query_statistics.health_queries_made, __ATOMIC_RELAXED);
102
+ gs->health_db_points_read = __atomic_load_n(&query_statistics.health_db_points_read, __ATOMIC_RELAXED);
103
+ gs->health_result_points_generated = __atomic_load_n(&query_statistics.health_result_points_generated, __ATOMIC_RELAXED);
104
+
105
+ gs->ml_queries_made = __atomic_load_n(&query_statistics.ml_queries_made, __ATOMIC_RELAXED);
106
+ gs->ml_db_points_read = __atomic_load_n(&query_statistics.ml_db_points_read, __ATOMIC_RELAXED);
107
+ gs->ml_result_points_generated = __atomic_load_n(&query_statistics.ml_result_points_generated, __ATOMIC_RELAXED);
108
+
109
+ gs->exporters_queries_made = __atomic_load_n(&query_statistics.exporters_queries_made, __ATOMIC_RELAXED);
110
+ gs->exporters_db_points_read = __atomic_load_n(&query_statistics.exporters_db_points_read, __ATOMIC_RELAXED);
111
+ gs->backfill_queries_made = __atomic_load_n(&query_statistics.backfill_queries_made, __ATOMIC_RELAXED);
112
+ gs->backfill_db_points_read = __atomic_load_n(&query_statistics.backfill_db_points_read, __ATOMIC_RELAXED);
113
+}
114
+
115
+void telemetry_queries_do(bool extended __maybe_unused) {
116
+ static struct query_statistics gs;
117
+ telemetry_queries_copy(&gs);
118
+
119
+ struct replication_query_statistics replication = replication_get_query_statistics();
120
+
121
+ {
122
+ static RRDSET *st_queries = NULL;
123
+ static RRDDIM *rd_api_data_queries = NULL;
124
+ static RRDDIM *rd_api_weights_queries = NULL;
125
+ static RRDDIM *rd_api_badges_queries = NULL;
126
+ static RRDDIM *rd_health_queries = NULL;
127
+ static RRDDIM *rd_ml_queries = NULL;
128
+ static RRDDIM *rd_exporters_queries = NULL;
129
+ static RRDDIM *rd_backfill_queries = NULL;
130
+ static RRDDIM *rd_replication_queries = NULL;
131
+
132
+ if (unlikely(!st_queries)) {
133
+ st_queries = rrdset_create_localhost(
134
+ "netdata"
135
+ , "queries"
136
+ , NULL
137
+ , "Time-Series Queries"
138
+ , NULL
139
+ , "Netdata Time-Series DB Queries"
140
+ , "queries/s"
141
+ , "netdata"
142
+ , "stats"
143
+ , 131000
144
+ , localhost->rrd_update_every
145
+ , RRDSET_TYPE_STACKED
146
+ );
147
+
148
+ rd_api_data_queries = rrddim_add(st_queries, "/api/vX/data", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
149
+ rd_api_weights_queries = rrddim_add(st_queries, "/api/vX/weights", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
150
+ rd_api_badges_queries = rrddim_add(st_queries, "/api/vX/badge", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
151
+ rd_health_queries = rrddim_add(st_queries, "health", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
152
+ rd_ml_queries = rrddim_add(st_queries, "ml", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
153
+ rd_exporters_queries = rrddim_add(st_queries, "exporters", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
154
+ rd_backfill_queries = rrddim_add(st_queries, "backfill", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
155
+ rd_replication_queries = rrddim_add(st_queries, "replication", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
156
+ }
157
+
158
+ rrddim_set_by_pointer(st_queries, rd_api_data_queries, (collected_number)gs.api_data_queries_made);
159
+ rrddim_set_by_pointer(st_queries, rd_api_weights_queries, (collected_number)gs.api_weights_queries_made);
160
+ rrddim_set_by_pointer(st_queries, rd_api_badges_queries, (collected_number)gs.api_badges_queries_made);
161
+ rrddim_set_by_pointer(st_queries, rd_health_queries, (collected_number)gs.health_queries_made);
162
+ rrddim_set_by_pointer(st_queries, rd_ml_queries, (collected_number)gs.ml_queries_made);
163
+ rrddim_set_by_pointer(st_queries, rd_exporters_queries, (collected_number)gs.exporters_queries_made);
164
+ rrddim_set_by_pointer(st_queries, rd_backfill_queries, (collected_number)gs.backfill_queries_made);
165
+ rrddim_set_by_pointer(st_queries, rd_replication_queries, (collected_number)replication.queries_finished);
166
+
167
+ rrdset_done(st_queries);
168
+ }
169
+
170
+ {
171
+ static RRDSET *st_points_read = NULL;
172
+ static RRDDIM *rd_api_data_points_read = NULL;
173
+ static RRDDIM *rd_api_weights_points_read = NULL;
174
+ static RRDDIM *rd_api_badges_points_read = NULL;
175
+ static RRDDIM *rd_health_points_read = NULL;
176
+ static RRDDIM *rd_ml_points_read = NULL;
177
+ static RRDDIM *rd_exporters_points_read = NULL;
178
+ static RRDDIM *rd_backfill_points_read = NULL;
179
+ static RRDDIM *rd_replication_points_read = NULL;
180
+
181
+ if (unlikely(!st_points_read)) {
182
+ st_points_read = rrdset_create_localhost(
183
+ "netdata"
184
+ , "db_points_read"
185
+ , NULL
186
+ , "Time-Series Queries"
187
+ , NULL
188
+ , "Netdata Time-Series DB Samples Read"
189
+ , "points/s"
190
+ , "netdata"
191
+ , "stats"
192
+ , 131001
193
+ , localhost->rrd_update_every
194
+ , RRDSET_TYPE_STACKED
195
+ );
196
+
197
+ rd_api_data_points_read = rrddim_add(st_points_read, "/api/vX/data", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
198
+ rd_api_weights_points_read = rrddim_add(st_points_read, "/api/vX/weights", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
199
+ rd_api_badges_points_read = rrddim_add(st_points_read, "/api/vX/badge", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
200
+ rd_health_points_read = rrddim_add(st_points_read, "health", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
201
+ rd_ml_points_read = rrddim_add(st_points_read, "ml", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
202
+ rd_exporters_points_read = rrddim_add(st_points_read, "exporters", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
203
+ rd_backfill_points_read = rrddim_add(st_points_read, "backfill", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
204
+ rd_replication_points_read = rrddim_add(st_points_read, "replication", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
205
+ }
206
+
207
+ rrddim_set_by_pointer(st_points_read, rd_api_data_points_read, (collected_number)gs.api_data_db_points_read);
208
+ rrddim_set_by_pointer(st_points_read, rd_api_weights_points_read, (collected_number)gs.api_weights_db_points_read);
209
+ rrddim_set_by_pointer(st_points_read, rd_api_badges_points_read, (collected_number)gs.api_badges_db_points_read);
210
+ rrddim_set_by_pointer(st_points_read, rd_health_points_read, (collected_number)gs.health_db_points_read);
211
+ rrddim_set_by_pointer(st_points_read, rd_ml_points_read, (collected_number)gs.ml_db_points_read);
212
+ rrddim_set_by_pointer(st_points_read, rd_exporters_points_read, (collected_number)gs.exporters_db_points_read);
213
+ rrddim_set_by_pointer(st_points_read, rd_backfill_points_read, (collected_number)gs.backfill_db_points_read);
214
+ rrddim_set_by_pointer(st_points_read, rd_replication_points_read, (collected_number)replication.points_read);
215
+
216
+ rrdset_done(st_points_read);
217
+ }
218
+
219
+ if(gs.api_data_result_points_generated || replication.points_generated) {
220
+ static RRDSET *st_points_generated = NULL;
221
+ static RRDDIM *rd_api_data_points_generated = NULL;
222
+ static RRDDIM *rd_api_weights_points_generated = NULL;
223
+ static RRDDIM *rd_api_badges_points_generated = NULL;
224
+ static RRDDIM *rd_health_points_generated = NULL;
225
+ static RRDDIM *rd_ml_points_generated = NULL;
226
+ static RRDDIM *rd_replication_points_generated = NULL;
227
+
228
+ if (unlikely(!st_points_generated)) {
229
+ st_points_generated = rrdset_create_localhost(
230
+ "netdata"
231
+ , "db_points_results"
232
+ , NULL
233
+ , "Time-Series Queries"
234
+ , NULL
235
+ , "Netdata Time-Series Samples Generated"
236
+ , "points/s"
237
+ , "netdata"
238
+ , "stats"
239
+ , 131002
240
+ , localhost->rrd_update_every
241
+ , RRDSET_TYPE_STACKED
242
+ );
243
+
244
+ rd_api_data_points_generated = rrddim_add(st_points_generated, "/api/vX/data", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
245
+ rd_api_weights_points_generated = rrddim_add(st_points_generated, "/api/vX/weights", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
246
+ rd_api_badges_points_generated = rrddim_add(st_points_generated, "/api/vX/badge", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
247
+ rd_health_points_generated = rrddim_add(st_points_generated, "health", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
248
+ rd_ml_points_generated = rrddim_add(st_points_generated, "ml", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
249
+ rd_replication_points_generated = rrddim_add(st_points_generated, "replication", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
250
+ }
251
+
252
+ rrddim_set_by_pointer(st_points_generated, rd_api_data_points_generated, (collected_number)gs.api_data_result_points_generated);
253
+ rrddim_set_by_pointer(st_points_generated, rd_api_weights_points_generated, (collected_number)gs.api_weights_result_points_generated);
254
+ rrddim_set_by_pointer(st_points_generated, rd_api_badges_points_generated, (collected_number)gs.api_badges_result_points_generated);
255
+ rrddim_set_by_pointer(st_points_generated, rd_health_points_generated, (collected_number)gs.health_result_points_generated);
256
+ rrddim_set_by_pointer(st_points_generated, rd_ml_points_generated, (collected_number)gs.ml_result_points_generated);
257
+ rrddim_set_by_pointer(st_points_generated, rd_replication_points_generated, (collected_number)replication.points_generated);
258
+
259
+ rrdset_done(st_points_generated);
260
+ }
261
+}
src/daemon/telemetry/telemetry-queries.h
new
+17
@@ -0,0 +1,17 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_QUERIES_H
4
+#define NETDATA_TELEMETRY_QUERIES_H
5
+
6
+#include "daemon/common.h"
7
+
8
+void telemetry_queries_ml_query_completed(size_t points_read);
9
+void telemetry_queries_exporters_query_completed(size_t points_read);
10
+void telemetry_queries_backfill_query_completed(size_t points_read);
11
+void telemetry_queries_rrdr_query_completed(size_t queries, uint64_t db_points_read, uint64_t result_points_generated, QUERY_SOURCE query_source);
12
+
13
+#if defined(TELEMETRY_INTERNALS)
14
+void telemetry_queries_do(bool extended);
15
+#endif
16
+
17
+#endif //NETDATA_TELEMETRY_QUERIES_H
src/daemon/telemetry/telemetry-sqlite3.c
new
+314
@@ -0,0 +1,314 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-sqlite3.h"
5
+
6
+static struct sqlite3_statistics {
7
+ bool enabled;
8
+
9
+ alignas(64) uint64_t sqlite3_queries_made;
10
+ alignas(64) uint64_t sqlite3_queries_ok;
11
+ alignas(64) uint64_t sqlite3_queries_failed;
12
+ alignas(64) uint64_t sqlite3_queries_failed_busy;
13
+ alignas(64) uint64_t sqlite3_queries_failed_locked;
14
+ alignas(64) uint64_t sqlite3_rows;
15
+ alignas(64) uint64_t sqlite3_metadata_cache_hit;
16
+ alignas(64) uint64_t sqlite3_context_cache_hit;
17
+ alignas(64) uint64_t sqlite3_metadata_cache_miss;
18
+ alignas(64) uint64_t sqlite3_context_cache_miss;
19
+ alignas(64) uint64_t sqlite3_metadata_cache_spill;
20
+ alignas(64) uint64_t sqlite3_context_cache_spill;
21
+ alignas(64) uint64_t sqlite3_metadata_cache_write;
22
+ alignas(64) uint64_t sqlite3_context_cache_write;
23
+} sqlite3_statistics = { };
24
+
25
+void telemetry_sqlite3_query_completed(bool success, bool busy, bool locked) {
26
+ if(!sqlite3_statistics.enabled) return;
27
+
28
+ __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_made, 1, __ATOMIC_RELAXED);
29
+
30
+ if(success) {
31
+ __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_ok, 1, __ATOMIC_RELAXED);
32
+ }
33
+ else {
34
+ __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_failed, 1, __ATOMIC_RELAXED);
35
+
36
+ if(busy)
37
+ __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_failed_busy, 1, __ATOMIC_RELAXED);
38
+
39
+ if(locked)
40
+ __atomic_fetch_add(&sqlite3_statistics.sqlite3_queries_failed_locked, 1, __ATOMIC_RELAXED);
41
+ }
42
+}
43
+
44
+void telemetry_sqlite3_row_completed(void) {
45
+ if(!sqlite3_statistics.enabled) return;
46
+
47
+ __atomic_fetch_add(&sqlite3_statistics.sqlite3_rows, 1, __ATOMIC_RELAXED);
48
+}
49
+
50
+static inline void sqlite3_statistics_copy(struct sqlite3_statistics *gs) {
51
+ static usec_t last_run = 0;
52
+
53
+ gs->sqlite3_queries_made = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_made, __ATOMIC_RELAXED);
54
+ gs->sqlite3_queries_ok = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_ok, __ATOMIC_RELAXED);
55
+ gs->sqlite3_queries_failed = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_failed, __ATOMIC_RELAXED);
56
+ gs->sqlite3_queries_failed_busy = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_failed_busy, __ATOMIC_RELAXED);
57
+ gs->sqlite3_queries_failed_locked = __atomic_load_n(&sqlite3_statistics.sqlite3_queries_failed_locked, __ATOMIC_RELAXED);
58
+ gs->sqlite3_rows = __atomic_load_n(&sqlite3_statistics.sqlite3_rows, __ATOMIC_RELAXED);
59
+
60
+ usec_t timeout = default_rrd_update_every * USEC_PER_SEC + default_rrd_update_every * USEC_PER_SEC / 3;
61
+ usec_t now = now_monotonic_usec();
62
+ if(!last_run)
63
+ last_run = now;
64
+ usec_t delta = now - last_run;
65
+ bool query_sqlite3 = delta < timeout;
66
+
67
+ if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
68
+ gs->sqlite3_metadata_cache_hit = (uint64_t) sql_metadata_cache_stats(SQLITE_DBSTATUS_CACHE_HIT);
69
+ else {
70
+ gs->sqlite3_metadata_cache_hit = UINT64_MAX;
71
+ query_sqlite3 = false;
72
+ }
73
+
74
+ if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
75
+ gs->sqlite3_context_cache_hit = (uint64_t) sql_context_cache_stats(SQLITE_DBSTATUS_CACHE_HIT);
76
+ else {
77
+ gs->sqlite3_context_cache_hit = UINT64_MAX;
78
+ query_sqlite3 = false;
79
+ }
80
+
81
+ if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
82
+ gs->sqlite3_metadata_cache_miss = (uint64_t) sql_metadata_cache_stats(SQLITE_DBSTATUS_CACHE_MISS);
83
+ else {
84
+ gs->sqlite3_metadata_cache_miss = UINT64_MAX;
85
+ query_sqlite3 = false;
86
+ }
87
+
88
+ if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
89
+ gs->sqlite3_context_cache_miss = (uint64_t) sql_context_cache_stats(SQLITE_DBSTATUS_CACHE_MISS);
90
+ else {
91
+ gs->sqlite3_context_cache_miss = UINT64_MAX;
92
+ query_sqlite3 = false;
93
+ }
94
+
95
+ if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
96
+ gs->sqlite3_metadata_cache_spill = (uint64_t) sql_metadata_cache_stats(SQLITE_DBSTATUS_CACHE_SPILL);
97
+ else {
98
+ gs->sqlite3_metadata_cache_spill = UINT64_MAX;
99
+ query_sqlite3 = false;
100
+ }
101
+
102
+ if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
103
+ gs->sqlite3_context_cache_spill = (uint64_t) sql_context_cache_stats(SQLITE_DBSTATUS_CACHE_SPILL);
104
+ else {
105
+ gs->sqlite3_context_cache_spill = UINT64_MAX;
106
+ query_sqlite3 = false;
107
+ }
108
+
109
+ if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
110
+ gs->sqlite3_metadata_cache_write = (uint64_t) sql_metadata_cache_stats(SQLITE_DBSTATUS_CACHE_WRITE);
111
+ else {
112
+ gs->sqlite3_metadata_cache_write = UINT64_MAX;
113
+ query_sqlite3 = false;
114
+ }
115
+
116
+ if(query_sqlite3 && now_monotonic_usec() - last_run < timeout)
117
+ gs->sqlite3_context_cache_write = (uint64_t) sql_context_cache_stats(SQLITE_DBSTATUS_CACHE_WRITE);
118
+ else {
119
+ gs->sqlite3_context_cache_write = UINT64_MAX;
120
+ query_sqlite3 = false;
121
+ }
122
+
123
+ last_run = now_monotonic_usec();
124
+}
125
+
126
+void telemetry_sqlite3_do(bool extended) {
127
+ if(!extended) return;
128
+ sqlite3_statistics.enabled = true;
129
+
130
+ struct sqlite3_statistics gs;
131
+ sqlite3_statistics_copy(&gs);
132
+
133
+ if(gs.sqlite3_queries_made) {
134
+ static RRDSET *st_sqlite3_queries = NULL;
135
+ static RRDDIM *rd_queries = NULL;
136
+
137
+ if (unlikely(!st_sqlite3_queries)) {
138
+ st_sqlite3_queries = rrdset_create_localhost(
139
+ "netdata"
140
+ , "sqlite3_queries"
141
+ , NULL
142
+ , "sqlite3"
143
+ , NULL
144
+ , "Netdata SQLite3 Queries"
145
+ , "queries/s"
146
+ , "netdata"
147
+ , "stats"
148
+ , 131100
149
+ , localhost->rrd_update_every
150
+ , RRDSET_TYPE_LINE
151
+ );
152
+
153
+ rd_queries = rrddim_add(st_sqlite3_queries, "queries", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
154
+ }
155
+
156
+ rrddim_set_by_pointer(st_sqlite3_queries, rd_queries, (collected_number)gs.sqlite3_queries_made);
157
+
158
+ rrdset_done(st_sqlite3_queries);
159
+ }
160
+
161
+ // ----------------------------------------------------------------
162
+
163
+ if(gs.sqlite3_queries_ok || gs.sqlite3_queries_failed) {
164
+ static RRDSET *st_sqlite3_queries_by_status = NULL;
165
+ static RRDDIM *rd_ok = NULL, *rd_failed = NULL, *rd_busy = NULL, *rd_locked = NULL;
166
+
167
+ if (unlikely(!st_sqlite3_queries_by_status)) {
168
+ st_sqlite3_queries_by_status = rrdset_create_localhost(
169
+ "netdata"
170
+ , "sqlite3_queries_by_status"
171
+ , NULL
172
+ , "sqlite3"
173
+ , NULL
174
+ , "Netdata SQLite3 Queries by status"
175
+ , "queries/s"
176
+ , "netdata"
177
+ , "stats"
178
+ , 131101
179
+ , localhost->rrd_update_every
180
+ , RRDSET_TYPE_LINE
181
+ );
182
+
183
+ rd_ok = rrddim_add(st_sqlite3_queries_by_status, "ok", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
184
+ rd_failed = rrddim_add(st_sqlite3_queries_by_status, "failed", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
185
+ rd_busy = rrddim_add(st_sqlite3_queries_by_status, "busy", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
186
+ rd_locked = rrddim_add(st_sqlite3_queries_by_status, "locked", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
187
+ }
188
+
189
+ rrddim_set_by_pointer(st_sqlite3_queries_by_status, rd_ok, (collected_number)gs.sqlite3_queries_made);
190
+ rrddim_set_by_pointer(st_sqlite3_queries_by_status, rd_failed, (collected_number)gs.sqlite3_queries_failed);
191
+ rrddim_set_by_pointer(st_sqlite3_queries_by_status, rd_busy, (collected_number)gs.sqlite3_queries_failed_busy);
192
+ rrddim_set_by_pointer(st_sqlite3_queries_by_status, rd_locked, (collected_number)gs.sqlite3_queries_failed_locked);
193
+
194
+ rrdset_done(st_sqlite3_queries_by_status);
195
+ }
196
+
197
+ // ----------------------------------------------------------------
198
+
199
+ if(gs.sqlite3_rows) {
200
+ static RRDSET *st_sqlite3_rows = NULL;
201
+ static RRDDIM *rd_rows = NULL;
202
+
203
+ if (unlikely(!st_sqlite3_rows)) {
204
+ st_sqlite3_rows = rrdset_create_localhost(
205
+ "netdata"
206
+ , "sqlite3_rows"
207
+ , NULL
208
+ , "sqlite3"
209
+ , NULL
210
+ , "Netdata SQLite3 Rows"
211
+ , "rows/s"
212
+ , "netdata"
213
+ , "stats"
214
+ , 131102
215
+ , localhost->rrd_update_every
216
+ , RRDSET_TYPE_LINE
217
+ );
218
+
219
+ rd_rows = rrddim_add(st_sqlite3_rows, "ok", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
220
+ }
221
+
222
+ rrddim_set_by_pointer(st_sqlite3_rows, rd_rows, (collected_number)gs.sqlite3_rows);
223
+
224
+ rrdset_done(st_sqlite3_rows);
225
+ }
226
+
227
+ if(gs.sqlite3_metadata_cache_hit) {
228
+ static RRDSET *st_sqlite3_cache = NULL;
229
+ static RRDDIM *rd_cache_hit = NULL;
230
+ static RRDDIM *rd_cache_miss= NULL;
231
+ static RRDDIM *rd_cache_spill= NULL;
232
+ static RRDDIM *rd_cache_write= NULL;
233
+
234
+ if (unlikely(!st_sqlite3_cache)) {
235
+ st_sqlite3_cache = rrdset_create_localhost(
236
+ "netdata"
237
+ , "sqlite3_metatada_cache"
238
+ , NULL
239
+ , "sqlite3"
240
+ , NULL
241
+ , "Netdata SQLite3 metadata cache"
242
+ , "ops/s"
243
+ , "netdata"
244
+ , "stats"
245
+ , 131103
246
+ , localhost->rrd_update_every
247
+ , RRDSET_TYPE_LINE
248
+ );
249
+
250
+ rd_cache_hit = rrddim_add(st_sqlite3_cache, "cache_hit", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
251
+ rd_cache_miss = rrddim_add(st_sqlite3_cache, "cache_miss", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
252
+ rd_cache_spill = rrddim_add(st_sqlite3_cache, "cache_spill", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
253
+ rd_cache_write = rrddim_add(st_sqlite3_cache, "cache_write", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
254
+ }
255
+
256
+ if(gs.sqlite3_metadata_cache_hit != UINT64_MAX)
257
+ rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_hit, (collected_number)gs.sqlite3_metadata_cache_hit);
258
+
259
+ if(gs.sqlite3_metadata_cache_miss != UINT64_MAX)
260
+ rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_miss, (collected_number)gs.sqlite3_metadata_cache_miss);
261
+
262
+ if(gs.sqlite3_metadata_cache_spill != UINT64_MAX)
263
+ rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_spill, (collected_number)gs.sqlite3_metadata_cache_spill);
264
+
265
+ if(gs.sqlite3_metadata_cache_write != UINT64_MAX)
266
+ rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_write, (collected_number)gs.sqlite3_metadata_cache_write);
267
+
268
+ rrdset_done(st_sqlite3_cache);
269
+ }
270
+
271
+ if(gs.sqlite3_context_cache_hit) {
272
+ static RRDSET *st_sqlite3_cache = NULL;
273
+ static RRDDIM *rd_cache_hit = NULL;
274
+ static RRDDIM *rd_cache_miss= NULL;
275
+ static RRDDIM *rd_cache_spill= NULL;
276
+ static RRDDIM *rd_cache_write= NULL;
277
+
278
+ if (unlikely(!st_sqlite3_cache)) {
279
+ st_sqlite3_cache = rrdset_create_localhost(
280
+ "netdata"
281
+ , "sqlite3_context_cache"
282
+ , NULL
283
+ , "sqlite3"
284
+ , NULL
285
+ , "Netdata SQLite3 context cache"
286
+ , "ops/s"
287
+ , "netdata"
288
+ , "stats"
289
+ , 131104
290
+ , localhost->rrd_update_every
291
+ , RRDSET_TYPE_LINE
292
+ );
293
+
294
+ rd_cache_hit = rrddim_add(st_sqlite3_cache, "cache_hit", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
295
+ rd_cache_miss = rrddim_add(st_sqlite3_cache, "cache_miss", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
296
+ rd_cache_spill = rrddim_add(st_sqlite3_cache, "cache_spill", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
297
+ rd_cache_write = rrddim_add(st_sqlite3_cache, "cache_write", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
298
+ }
299
+
300
+ if(gs.sqlite3_context_cache_hit != UINT64_MAX)
301
+ rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_hit, (collected_number)gs.sqlite3_context_cache_hit);
302
+
303
+ if(gs.sqlite3_context_cache_miss != UINT64_MAX)
304
+ rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_miss, (collected_number)gs.sqlite3_context_cache_miss);
305
+
306
+ if(gs.sqlite3_context_cache_spill != UINT64_MAX)
307
+ rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_spill, (collected_number)gs.sqlite3_context_cache_spill);
308
+
309
+ if(gs.sqlite3_context_cache_write != UINT64_MAX)
310
+ rrddim_set_by_pointer(st_sqlite3_cache, rd_cache_write, (collected_number)gs.sqlite3_context_cache_write);
311
+
312
+ rrdset_done(st_sqlite3_cache);
313
+ }
314
+}
src/daemon/telemetry/telemetry-sqlite3.h
new
+15
@@ -0,0 +1,15 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_SQLITE3_H
4
+#define NETDATA_TELEMETRY_SQLITE3_H
5
+
6
+#include "daemon/common.h"
7
+
8
+void telemetry_sqlite3_query_completed(bool success, bool busy, bool locked);
9
+void telemetry_sqlite3_row_completed(void);
10
+
11
+#if defined(TELEMETRY_INTERNALS)
12
+void telemetry_sqlite3_do(bool extended);
13
+#endif
14
+
15
+#endif //NETDATA_TELEMETRY_SQLITE3_H
src/daemon/telemetry/telemetry-string.c
new
+101
@@ -0,0 +1,101 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-string.h"
5
+
6
+void telemetry_string_do(bool extended) {
7
+ if(!extended) return;
8
+
9
+ static RRDSET *st_ops = NULL, *st_entries = NULL, *st_mem = NULL;
10
+ static RRDDIM *rd_ops_inserts = NULL, *rd_ops_deletes = NULL;
11
+ static RRDDIM *rd_entries_entries = NULL;
12
+ static RRDDIM *rd_mem = NULL;
13
+#ifdef NETDATA_INTERNAL_CHECKS
14
+ static RRDDIM *rd_entries_refs = NULL, *rd_ops_releases = NULL, *rd_ops_duplications = NULL, *rd_ops_searches = NULL;
15
+#endif
16
+
17
+ size_t inserts, deletes, searches, entries, references, memory, duplications, releases;
18
+
19
+ string_statistics(&inserts, &deletes, &searches, &entries, &references, &memory, &duplications, &releases);
20
+
21
+ if (unlikely(!st_ops)) {
22
+ st_ops = rrdset_create_localhost(
23
+ "netdata"
24
+ , "strings_ops"
25
+ , NULL
26
+ , "strings"
27
+ , NULL
28
+ , "Strings operations"
29
+ , "ops/s"
30
+ , "netdata"
31
+ , "stats"
32
+ , 910000
33
+ , localhost->rrd_update_every
34
+ , RRDSET_TYPE_LINE);
35
+
36
+ rd_ops_inserts = rrddim_add(st_ops, "inserts", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
37
+ rd_ops_deletes = rrddim_add(st_ops, "deletes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
38
+#ifdef NETDATA_INTERNAL_CHECKS
39
+ rd_ops_searches = rrddim_add(st_ops, "searches", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
40
+ rd_ops_duplications = rrddim_add(st_ops, "duplications", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
41
+ rd_ops_releases = rrddim_add(st_ops, "releases", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
42
+#endif
43
+ }
44
+
45
+ rrddim_set_by_pointer(st_ops, rd_ops_inserts, (collected_number)inserts);
46
+ rrddim_set_by_pointer(st_ops, rd_ops_deletes, (collected_number)deletes);
47
+#ifdef NETDATA_INTERNAL_CHECKS
48
+ rrddim_set_by_pointer(st_ops, rd_ops_searches, (collected_number)searches);
49
+ rrddim_set_by_pointer(st_ops, rd_ops_duplications, (collected_number)duplications);
50
+ rrddim_set_by_pointer(st_ops, rd_ops_releases, (collected_number)releases);
51
+#endif
52
+ rrdset_done(st_ops);
53
+
54
+ if (unlikely(!st_entries)) {
55
+ st_entries = rrdset_create_localhost(
56
+ "netdata"
57
+ , "strings_entries"
58
+ , NULL
59
+ , "strings"
60
+ , NULL
61
+ , "Strings entries"
62
+ , "entries"
63
+ , "netdata"
64
+ , "stats"
65
+ , 910001
66
+ , localhost->rrd_update_every
67
+ , RRDSET_TYPE_AREA);
68
+
69
+ rd_entries_entries = rrddim_add(st_entries, "entries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
70
+#ifdef NETDATA_INTERNAL_CHECKS
71
+ rd_entries_refs = rrddim_add(st_entries, "references", NULL, 1, -1, RRD_ALGORITHM_ABSOLUTE);
72
+#endif
73
+ }
74
+
75
+ rrddim_set_by_pointer(st_entries, rd_entries_entries, (collected_number)entries);
76
+#ifdef NETDATA_INTERNAL_CHECKS
77
+ rrddim_set_by_pointer(st_entries, rd_entries_refs, (collected_number)references);
78
+#endif
79
+ rrdset_done(st_entries);
80
+
81
+ if (unlikely(!st_mem)) {
82
+ st_mem = rrdset_create_localhost(
83
+ "netdata"
84
+ , "strings_memory"
85
+ , NULL
86
+ , "strings"
87
+ , NULL
88
+ , "Strings memory"
89
+ , "bytes"
90
+ , "netdata"
91
+ , "stats"
92
+ , 910001
93
+ , localhost->rrd_update_every
94
+ , RRDSET_TYPE_AREA);
95
+
96
+ rd_mem = rrddim_add(st_mem, "memory", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
97
+ }
98
+
99
+ rrddim_set_by_pointer(st_mem, rd_mem, (collected_number)memory);
100
+ rrdset_done(st_mem);
101
+}
src/daemon/telemetry/telemetry-string.h
new
+12
@@ -0,0 +1,12 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_STRING_H
4
+#define NETDATA_TELEMETRY_STRING_H
5
+
6
+#include "daemon/common.h"
7
+
8
+#if defined(TELEMETRY_INTERNALS)
9
+void telemetry_string_do(bool extended);
10
+#endif
11
+
12
+#endif //NETDATA_TELEMETRY_STRING_H
src/daemon/telemetry/telemetry-trace-allocations.c
new
+147
@@ -0,0 +1,147 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-trace-allocations.h"
5
+
6
+#ifdef NETDATA_TRACE_ALLOCATIONS
7
+
8
+struct memory_trace_data {
9
+ RRDSET *st_memory;
10
+ RRDSET *st_allocations;
11
+ RRDSET *st_avg_alloc;
12
+ RRDSET *st_ops;
13
+};
14
+
15
+static int do_memory_trace_item(void *item, void *data) {
16
+ struct memory_trace_data *tmp = data;
17
+ struct malloc_trace *p = item;
18
+
19
+ // ------------------------------------------------------------------------
20
+
21
+ if(!p->rd_bytes)
22
+ p->rd_bytes = rrddim_add(tmp->st_memory, p->function, NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
23
+
24
+ collected_number bytes = (collected_number)__atomic_load_n(&p->bytes, __ATOMIC_RELAXED);
25
+ rrddim_set_by_pointer(tmp->st_memory, p->rd_bytes, bytes);
26
+
27
+ // ------------------------------------------------------------------------
28
+
29
+ if(!p->rd_allocations)
30
+ p->rd_allocations = rrddim_add(tmp->st_allocations, p->function, NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
31
+
32
+ collected_number allocs = (collected_number)__atomic_load_n(&p->allocations, __ATOMIC_RELAXED);
33
+ rrddim_set_by_pointer(tmp->st_allocations, p->rd_allocations, allocs);
34
+
35
+ // ------------------------------------------------------------------------
36
+
37
+ if(!p->rd_avg_alloc)
38
+ p->rd_avg_alloc = rrddim_add(tmp->st_avg_alloc, p->function, NULL, 1, 100, RRD_ALGORITHM_ABSOLUTE);
39
+
40
+ collected_number avg_alloc = (allocs)?(bytes * 100 / allocs):0;
41
+ rrddim_set_by_pointer(tmp->st_avg_alloc, p->rd_avg_alloc, avg_alloc);
42
+
43
+ // ------------------------------------------------------------------------
44
+
45
+ if(!p->rd_ops)
46
+ p->rd_ops = rrddim_add(tmp->st_ops, p->function, NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
47
+
48
+ collected_number ops = 0;
49
+ ops += (collected_number)__atomic_load_n(&p->malloc_calls, __ATOMIC_RELAXED);
50
+ ops += (collected_number)__atomic_load_n(&p->calloc_calls, __ATOMIC_RELAXED);
51
+ ops += (collected_number)__atomic_load_n(&p->realloc_calls, __ATOMIC_RELAXED);
52
+ ops += (collected_number)__atomic_load_n(&p->strdup_calls, __ATOMIC_RELAXED);
53
+ ops += (collected_number)__atomic_load_n(&p->free_calls, __ATOMIC_RELAXED);
54
+ rrddim_set_by_pointer(tmp->st_ops, p->rd_ops, ops);
55
+
56
+ // ------------------------------------------------------------------------
57
+
58
+ return 1;
59
+}
60
+
61
+void telemetry_trace_allocations_do(bool extended) {
62
+ if(!extended) return;
63
+
64
+ static struct memory_trace_data tmp = {
65
+ .st_memory = NULL,
66
+ .st_allocations = NULL,
67
+ .st_avg_alloc = NULL,
68
+ .st_ops = NULL,
69
+ };
70
+
71
+ if(!tmp.st_memory) {
72
+ tmp.st_memory = rrdset_create_localhost(
73
+ "netdata"
74
+ , "memory_size"
75
+ , NULL
76
+ , "memory"
77
+ , "netdata.memory.size"
78
+ , "Netdata Memory Used by Function"
79
+ , "bytes"
80
+ , "netdata"
81
+ , "stats"
82
+ , 900000
83
+ , localhost->rrd_update_every
84
+ , RRDSET_TYPE_STACKED
85
+ );
86
+ }
87
+
88
+ if(!tmp.st_ops) {
89
+ tmp.st_ops = rrdset_create_localhost(
90
+ "netdata"
91
+ , "memory_operations"
92
+ , NULL
93
+ , "memory"
94
+ , "netdata.memory.operations"
95
+ , "Netdata Memory Operations by Function"
96
+ , "ops/s"
97
+ , "netdata"
98
+ , "stats"
99
+ , 900001
100
+ , localhost->rrd_update_every
101
+ , RRDSET_TYPE_LINE
102
+ );
103
+ }
104
+
105
+ if(!tmp.st_allocations) {
106
+ tmp.st_allocations = rrdset_create_localhost(
107
+ "netdata"
108
+ , "memory_allocations"
109
+ , NULL
110
+ , "memory"
111
+ , "netdata.memory.allocations"
112
+ , "Netdata Memory Allocations by Function"
113
+ , "allocations"
114
+ , "netdata"
115
+ , "stats"
116
+ , 900002
117
+ , localhost->rrd_update_every
118
+ , RRDSET_TYPE_STACKED
119
+ );
120
+ }
121
+
122
+ if(!tmp.st_avg_alloc) {
123
+ tmp.st_avg_alloc = rrdset_create_localhost(
124
+ "netdata"
125
+ , "memory_avg_alloc"
126
+ , NULL
127
+ , "memory"
128
+ , "netdata.memory.avg_alloc"
129
+ , "Netdata Average Allocation Size by Function"
130
+ , "bytes"
131
+ , "netdata"
132
+ , "stats"
133
+ , 900003
134
+ , localhost->rrd_update_every
135
+ , RRDSET_TYPE_LINE
136
+ );
137
+ }
138
+
139
+ malloc_trace_walkthrough(do_memory_trace_item, &tmp);
140
+
141
+ rrdset_done(tmp.st_memory);
142
+ rrdset_done(tmp.st_ops);
143
+ rrdset_done(tmp.st_allocations);
144
+ rrdset_done(tmp.st_avg_alloc);
145
+}
146
+
147
+#endif
src/daemon/telemetry/telemetry-trace-allocations.h
new
+14
@@ -0,0 +1,14 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_TRACE_ALLOCATIONS_H
4
+#define NETDATA_TELEMETRY_TRACE_ALLOCATIONS_H
5
+
6
+#include "daemon/common.h"
7
+
8
+#if defined(TELEMETRY_INTERNALS)
9
+#ifdef NETDATA_TRACE_ALLOCATIONS
10
+void telemetry_trace_allocations_do(bool extended);
11
+#endif
12
+#endif
13
+
14
+#endif //NETDATA_TELEMETRY_TRACE_ALLOCATIONS_H
src/daemon/telemetry/telemetry-workers.c
new
+791
@@ -0,0 +1,791 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "telemetry-workers.h"
5
+
6
+#define WORKERS_MIN_PERCENT_DEFAULT 10000.0
7
+
8
+struct worker_job_type_gs {
9
+ STRING *name;
10
+ STRING *units;
11
+
12
+ size_t jobs_started;
13
+ usec_t busy_time;
14
+
15
+ RRDDIM *rd_jobs_started;
16
+ RRDDIM *rd_busy_time;
17
+
18
+ WORKER_METRIC_TYPE type;
19
+ NETDATA_DOUBLE min_value;
20
+ NETDATA_DOUBLE max_value;
21
+ NETDATA_DOUBLE sum_value;
22
+ size_t count_value;
23
+
24
+ RRDSET *st;
25
+ RRDDIM *rd_min;
26
+ RRDDIM *rd_max;
27
+ RRDDIM *rd_avg;
28
+};
29
+
30
+struct worker_thread {
31
+ pid_t pid;
32
+ bool enabled;
33
+
34
+ bool cpu_enabled;
35
+ double cpu;
36
+
37
+ kernel_uint_t utime;
38
+ kernel_uint_t stime;
39
+
40
+ kernel_uint_t utime_old;
41
+ kernel_uint_t stime_old;
42
+
43
+ usec_t collected_time;
44
+ usec_t collected_time_old;
45
+
46
+ size_t jobs_started;
47
+ usec_t busy_time;
48
+
49
+ struct worker_thread *next;
50
+ struct worker_thread *prev;
51
+};
52
+
53
+struct worker_utilization {
54
+ const char *name;
55
+ const char *family;
56
+ size_t priority;
57
+ uint32_t flags;
58
+
59
+ char *name_lowercase;
60
+
61
+ struct worker_job_type_gs per_job_type[WORKER_UTILIZATION_MAX_JOB_TYPES];
62
+
63
+ size_t workers_max_job_id;
64
+ size_t workers_registered;
65
+ size_t workers_busy;
66
+ usec_t workers_total_busy_time;
67
+ usec_t workers_total_duration;
68
+ size_t workers_total_jobs_started;
69
+ double workers_min_busy_time;
70
+ double workers_max_busy_time;
71
+
72
+ size_t workers_cpu_registered;
73
+ double workers_cpu_min;
74
+ double workers_cpu_max;
75
+ double workers_cpu_total;
76
+
77
+ struct worker_thread *threads;
78
+
79
+ RRDSET *st_workers_time;
80
+ RRDDIM *rd_workers_time_avg;
81
+ RRDDIM *rd_workers_time_min;
82
+ RRDDIM *rd_workers_time_max;
83
+
84
+ RRDSET *st_workers_cpu;
85
+ RRDDIM *rd_workers_cpu_avg;
86
+ RRDDIM *rd_workers_cpu_min;
87
+ RRDDIM *rd_workers_cpu_max;
88
+
89
+ RRDSET *st_workers_threads;
90
+ RRDDIM *rd_workers_threads_free;
91
+ RRDDIM *rd_workers_threads_busy;
92
+
93
+ RRDSET *st_workers_jobs_per_job_type;
94
+ RRDSET *st_workers_busy_per_job_type;
95
+
96
+ RRDDIM *rd_total_cpu_utilizaton;
97
+};
98
+
99
+static struct worker_utilization all_workers_utilization[] = {
100
+ { .name = "STATS", .family = "workers telemetry", .priority = 1000000 },
101
+ { .name = "HEALTH", .family = "workers health alarms", .priority = 1000000 },
102
+ { .name = "MLTRAIN", .family = "workers ML training", .priority = 1000000 },
103
+ { .name = "MLDETECT", .family = "workers ML detection", .priority = 1000000 },
104
+ { .name = "STREAM", .family = "workers streaming", .priority = 1000000 },
105
+ { .name = "STREAMCNT", .family = "workers streaming connect", .priority = 1000000 },
106
+ { .name = "DBENGINE", .family = "workers dbengine instances", .priority = 1000000 },
107
+ { .name = "LIBUV", .family = "workers libuv threadpool", .priority = 1000000 },
108
+ { .name = "WEB", .family = "workers web server", .priority = 1000000 },
109
+ { .name = "ACLKSYNC", .family = "workers aclk sync", .priority = 1000000 },
110
+ { .name = "METASYNC", .family = "workers metadata sync", .priority = 1000000 },
111
+ { .name = "PLUGINSD", .family = "workers plugins.d", .priority = 1000000 },
112
+ { .name = "STATSD", .family = "workers plugin statsd", .priority = 1000000 },
113
+ { .name = "STATSDFLUSH", .family = "workers plugin statsd flush", .priority = 1000000 },
114
+ { .name = "PROC", .family = "workers plugin proc", .priority = 1000000 },
115
+ { .name = "WIN", .family = "workers plugin windows", .priority = 1000000 },
116
+ { .name = "NETDEV", .family = "workers plugin proc netdev", .priority = 1000000 },
117
+ { .name = "FREEBSD", .family = "workers plugin freebsd", .priority = 1000000 },
118
+ { .name = "MACOS", .family = "workers plugin macos", .priority = 1000000 },
119
+ { .name = "CGROUPS", .family = "workers plugin cgroups", .priority = 1000000 },
120
+ { .name = "CGROUPSDISC", .family = "workers plugin cgroups find", .priority = 1000000 },
121
+ { .name = "DISKSPACE", .family = "workers plugin diskspace", .priority = 1000000 },
122
+ { .name = "TC", .family = "workers plugin tc", .priority = 1000000 },
123
+ { .name = "TIMEX", .family = "workers plugin timex", .priority = 1000000 },
124
+ { .name = "IDLEJITTER", .family = "workers plugin idlejitter", .priority = 1000000 },
125
+ { .name = "RRDCONTEXT", .family = "workers contexts", .priority = 1000000 },
126
+ { .name = "REPLICATION", .family = "workers replication sender", .priority = 1000000 },
127
+ { .name = "SERVICE", .family = "workers service", .priority = 1000000 },
128
+ { .name = "PROFILER", .family = "workers profile", .priority = 1000000 },
129
+ { .name = "PGCEVICT", .family = "workers dbengine eviction", .priority = 1000000 },
130
+
131
+ // has to be terminated with a NULL
132
+ { .name = NULL, .family = NULL }
133
+};
134
+
135
+static void workers_total_cpu_utilization_chart(void) {
136
+ size_t i, cpu_enabled = 0;
137
+ for(i = 0; all_workers_utilization[i].name ;i++)
138
+ if(all_workers_utilization[i].workers_cpu_registered) cpu_enabled++;
139
+
140
+ if(!cpu_enabled) return;
141
+
142
+ static RRDSET *st = NULL;
143
+
144
+ if(!st) {
145
+ st = rrdset_create_localhost(
146
+ "netdata",
147
+ "workers_cpu",
148
+ NULL,
149
+ "workers",
150
+ "netdata.workers.cpu_total",
151
+ "Netdata Workers CPU Utilization (100% = 1 core)",
152
+ "%",
153
+ "netdata",
154
+ "stats",
155
+ 999000,
156
+ localhost->rrd_update_every,
157
+ RRDSET_TYPE_STACKED);
158
+ }
159
+
160
+ for(i = 0; all_workers_utilization[i].name ;i++) {
161
+ struct worker_utilization *wu = &all_workers_utilization[i];
162
+ if(!wu->workers_cpu_registered) continue;
163
+
164
+ if(!wu->rd_total_cpu_utilizaton)
165
+ wu->rd_total_cpu_utilizaton = rrddim_add(st, wu->name_lowercase, NULL, 1, 100, RRD_ALGORITHM_ABSOLUTE);
166
+
167
+ rrddim_set_by_pointer(st, wu->rd_total_cpu_utilizaton, (collected_number)((double)wu->workers_cpu_total * 100.0));
168
+ }
169
+
170
+ rrdset_done(st);
171
+}
172
+
173
+#define WORKER_CHART_DECIMAL_PRECISION 100
174
+
175
+static void workers_utilization_update_chart(struct worker_utilization *wu) {
176
+ if(!wu->workers_registered) return;
177
+
178
+ //fprintf(stderr, "%-12s WORKER UTILIZATION: %-3.2f%%, %zu jobs done, %zu running, on %zu workers, min %-3.02f%%, max %-3.02f%%.\n",
179
+ // wu->name,
180
+ // (double)wu->workers_total_busy_time * 100.0 / (double)wu->workers_total_duration,
181
+ // wu->workers_total_jobs_started, wu->workers_busy, wu->workers_registered,
182
+ // wu->workers_min_busy_time, wu->workers_max_busy_time);
183
+
184
+ // ----------------------------------------------------------------------
185
+
186
+ if(unlikely(!wu->st_workers_time)) {
187
+ char name[RRD_ID_LENGTH_MAX + 1];
188
+ snprintfz(name, RRD_ID_LENGTH_MAX, "workers_time_%s", wu->name_lowercase);
189
+
190
+ char context[RRD_ID_LENGTH_MAX + 1];
191
+ snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.time", wu->name_lowercase);
192
+
193
+ wu->st_workers_time = rrdset_create_localhost(
194
+ "netdata"
195
+ , name
196
+ , NULL
197
+ , wu->family
198
+ , context
199
+ , "Netdata Workers Busy Time (100% = all workers busy)"
200
+ , "%"
201
+ , "netdata"
202
+ , "stats"
203
+ , wu->priority
204
+ , localhost->rrd_update_every
205
+ , RRDSET_TYPE_AREA
206
+ );
207
+ }
208
+
209
+ // we add the min and max dimensions only when we have multiple workers
210
+
211
+ if(unlikely(!wu->rd_workers_time_min && wu->workers_registered > 1))
212
+ wu->rd_workers_time_min = rrddim_add(wu->st_workers_time, "min", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
213
+
214
+ if(unlikely(!wu->rd_workers_time_max && wu->workers_registered > 1))
215
+ wu->rd_workers_time_max = rrddim_add(wu->st_workers_time, "max", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
216
+
217
+ if(unlikely(!wu->rd_workers_time_avg))
218
+ wu->rd_workers_time_avg = rrddim_add(wu->st_workers_time, "average", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
219
+
220
+ if(unlikely(wu->workers_min_busy_time == WORKERS_MIN_PERCENT_DEFAULT)) wu->workers_min_busy_time = 0.0;
221
+
222
+ if(wu->rd_workers_time_min)
223
+ rrddim_set_by_pointer(wu->st_workers_time, wu->rd_workers_time_min, (collected_number)((double)wu->workers_min_busy_time * WORKER_CHART_DECIMAL_PRECISION));
224
+
225
+ if(wu->rd_workers_time_max)
226
+ rrddim_set_by_pointer(wu->st_workers_time, wu->rd_workers_time_max, (collected_number)((double)wu->workers_max_busy_time * WORKER_CHART_DECIMAL_PRECISION));
227
+
228
+ if(wu->workers_total_duration == 0)
229
+ rrddim_set_by_pointer(wu->st_workers_time, wu->rd_workers_time_avg, 0);
230
+ else
231
+ rrddim_set_by_pointer(wu->st_workers_time, wu->rd_workers_time_avg, (collected_number)((double)wu->workers_total_busy_time * 100.0 * WORKER_CHART_DECIMAL_PRECISION / (double)wu->workers_total_duration));
232
+
233
+ rrdset_done(wu->st_workers_time);
234
+
235
+ // ----------------------------------------------------------------------
236
+
237
+#ifdef __linux__
238
+ if(wu->workers_cpu_registered || wu->st_workers_cpu) {
239
+ if(unlikely(!wu->st_workers_cpu)) {
240
+ char name[RRD_ID_LENGTH_MAX + 1];
241
+ snprintfz(name, RRD_ID_LENGTH_MAX, "workers_cpu_%s", wu->name_lowercase);
242
+
243
+ char context[RRD_ID_LENGTH_MAX + 1];
244
+ snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.cpu", wu->name_lowercase);
245
+
246
+ wu->st_workers_cpu = rrdset_create_localhost(
247
+ "netdata"
248
+ , name
249
+ , NULL
250
+ , wu->family
251
+ , context
252
+ , "Netdata Workers CPU Utilization (100% = all workers busy)"
253
+ , "%"
254
+ , "netdata"
255
+ , "stats"
256
+ , wu->priority + 1
257
+ , localhost->rrd_update_every
258
+ , RRDSET_TYPE_AREA
259
+ );
260
+ }
261
+
262
+ if (unlikely(!wu->rd_workers_cpu_min && wu->workers_registered > 1))
263
+ wu->rd_workers_cpu_min = rrddim_add(wu->st_workers_cpu, "min", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
264
+
265
+ if (unlikely(!wu->rd_workers_cpu_max && wu->workers_registered > 1))
266
+ wu->rd_workers_cpu_max = rrddim_add(wu->st_workers_cpu, "max", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
267
+
268
+ if(unlikely(!wu->rd_workers_cpu_avg))
269
+ wu->rd_workers_cpu_avg = rrddim_add(wu->st_workers_cpu, "average", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
270
+
271
+ if(unlikely(wu->workers_cpu_min == WORKERS_MIN_PERCENT_DEFAULT)) wu->workers_cpu_min = 0.0;
272
+
273
+ if(wu->rd_workers_cpu_min)
274
+ rrddim_set_by_pointer(wu->st_workers_cpu, wu->rd_workers_cpu_min, (collected_number)(wu->workers_cpu_min * WORKER_CHART_DECIMAL_PRECISION));
275
+
276
+ if(wu->rd_workers_cpu_max)
277
+ rrddim_set_by_pointer(wu->st_workers_cpu, wu->rd_workers_cpu_max, (collected_number)(wu->workers_cpu_max * WORKER_CHART_DECIMAL_PRECISION));
278
+
279
+ if(wu->workers_cpu_registered == 0)
280
+ rrddim_set_by_pointer(wu->st_workers_cpu, wu->rd_workers_cpu_avg, 0);
281
+ else
282
+ rrddim_set_by_pointer(wu->st_workers_cpu, wu->rd_workers_cpu_avg, (collected_number)( wu->workers_cpu_total * WORKER_CHART_DECIMAL_PRECISION / (NETDATA_DOUBLE)wu->workers_cpu_registered ));
283
+
284
+ rrdset_done(wu->st_workers_cpu);
285
+ }
286
+#endif
287
+
288
+ // ----------------------------------------------------------------------
289
+
290
+ if(unlikely(!wu->st_workers_jobs_per_job_type)) {
291
+ char name[RRD_ID_LENGTH_MAX + 1];
292
+ snprintfz(name, RRD_ID_LENGTH_MAX, "workers_jobs_by_type_%s", wu->name_lowercase);
293
+
294
+ char context[RRD_ID_LENGTH_MAX + 1];
295
+ snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.jobs_started_by_type", wu->name_lowercase);
296
+
297
+ wu->st_workers_jobs_per_job_type = rrdset_create_localhost(
298
+ "netdata"
299
+ , name
300
+ , NULL
301
+ , wu->family
302
+ , context
303
+ , "Netdata Workers Jobs Started by Type"
304
+ , "jobs"
305
+ , "netdata"
306
+ , "stats"
307
+ , wu->priority + 2
308
+ , localhost->rrd_update_every
309
+ , RRDSET_TYPE_STACKED
310
+ );
311
+ }
312
+
313
+ {
314
+ size_t i;
315
+ for(i = 0; i <= wu->workers_max_job_id ;i++) {
316
+ if(unlikely(wu->per_job_type[i].type != WORKER_METRIC_IDLE_BUSY))
317
+ continue;
318
+
319
+ if (wu->per_job_type[i].name) {
320
+
321
+ if(unlikely(!wu->per_job_type[i].rd_jobs_started))
322
+ wu->per_job_type[i].rd_jobs_started = rrddim_add(wu->st_workers_jobs_per_job_type, string2str(wu->per_job_type[i].name), NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
323
+
324
+ rrddim_set_by_pointer(wu->st_workers_jobs_per_job_type, wu->per_job_type[i].rd_jobs_started, (collected_number)(wu->per_job_type[i].jobs_started));
325
+ }
326
+ }
327
+ }
328
+
329
+ rrdset_done(wu->st_workers_jobs_per_job_type);
330
+
331
+ // ----------------------------------------------------------------------
332
+
333
+ if(unlikely(!wu->st_workers_busy_per_job_type)) {
334
+ char name[RRD_ID_LENGTH_MAX + 1];
335
+ snprintfz(name, RRD_ID_LENGTH_MAX, "workers_busy_time_by_type_%s", wu->name_lowercase);
336
+
337
+ char context[RRD_ID_LENGTH_MAX + 1];
338
+ snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.time_by_type", wu->name_lowercase);
339
+
340
+ wu->st_workers_busy_per_job_type = rrdset_create_localhost(
341
+ "netdata"
342
+ , name
343
+ , NULL
344
+ , wu->family
345
+ , context
346
+ , "Netdata Workers Busy Time by Type"
347
+ , "ms"
348
+ , "netdata"
349
+ , "stats"
350
+ , wu->priority + 3
351
+ , localhost->rrd_update_every
352
+ , RRDSET_TYPE_STACKED
353
+ );
354
+ }
355
+
356
+ {
357
+ size_t i;
358
+ for(i = 0; i <= wu->workers_max_job_id ;i++) {
359
+ if(unlikely(wu->per_job_type[i].type != WORKER_METRIC_IDLE_BUSY))
360
+ continue;
361
+
362
+ if (wu->per_job_type[i].name) {
363
+
364
+ if(unlikely(!wu->per_job_type[i].rd_busy_time))
365
+ wu->per_job_type[i].rd_busy_time = rrddim_add(wu->st_workers_busy_per_job_type, string2str(wu->per_job_type[i].name), NULL, 1, USEC_PER_MS, RRD_ALGORITHM_ABSOLUTE);
366
+
367
+ rrddim_set_by_pointer(wu->st_workers_busy_per_job_type, wu->per_job_type[i].rd_busy_time, (collected_number)(wu->per_job_type[i].busy_time));
368
+ }
369
+ }
370
+ }
371
+
372
+ rrdset_done(wu->st_workers_busy_per_job_type);
373
+
374
+ // ----------------------------------------------------------------------
375
+
376
+ if(wu->st_workers_threads || wu->workers_registered > 1) {
377
+ if(unlikely(!wu->st_workers_threads)) {
378
+ char name[RRD_ID_LENGTH_MAX + 1];
379
+ snprintfz(name, RRD_ID_LENGTH_MAX, "workers_threads_%s", wu->name_lowercase);
380
+
381
+ char context[RRD_ID_LENGTH_MAX + 1];
382
+ snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.threads", wu->name_lowercase);
383
+
384
+ wu->st_workers_threads = rrdset_create_localhost(
385
+ "netdata"
386
+ , name
387
+ , NULL
388
+ , wu->family
389
+ , context
390
+ , "Netdata Workers Threads"
391
+ , "threads"
392
+ , "netdata"
393
+ , "stats"
394
+ , wu->priority + 4
395
+ , localhost->rrd_update_every
396
+ , RRDSET_TYPE_STACKED
397
+ );
398
+
399
+ wu->rd_workers_threads_free = rrddim_add(wu->st_workers_threads, "free", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
400
+ wu->rd_workers_threads_busy = rrddim_add(wu->st_workers_threads, "busy", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
401
+ }
402
+
403
+ rrddim_set_by_pointer(wu->st_workers_threads, wu->rd_workers_threads_free, (collected_number)(wu->workers_registered - wu->workers_busy));
404
+ rrddim_set_by_pointer(wu->st_workers_threads, wu->rd_workers_threads_busy, (collected_number)(wu->workers_busy));
405
+ rrdset_done(wu->st_workers_threads);
406
+ }
407
+
408
+ // ----------------------------------------------------------------------
409
+ // custom metric types WORKER_METRIC_ABSOLUTE
410
+
411
+ {
412
+ size_t i;
413
+ for (i = 0; i <= wu->workers_max_job_id; i++) {
414
+ if(wu->per_job_type[i].type != WORKER_METRIC_ABSOLUTE)
415
+ continue;
416
+
417
+ if(!wu->per_job_type[i].count_value)
418
+ continue;
419
+
420
+ if(!wu->per_job_type[i].st) {
421
+ size_t job_name_len = string_strlen(wu->per_job_type[i].name);
422
+ if(job_name_len > RRD_ID_LENGTH_MAX) job_name_len = RRD_ID_LENGTH_MAX;
423
+
424
+ char job_name_sanitized[job_name_len + 1];
425
+ rrdset_strncpyz_name(job_name_sanitized, string2str(wu->per_job_type[i].name), job_name_len);
426
+
427
+ char name[RRD_ID_LENGTH_MAX + 1];
428
+ snprintfz(name, RRD_ID_LENGTH_MAX, "workers_%s_value_%s", wu->name_lowercase, job_name_sanitized);
429
+
430
+ char context[RRD_ID_LENGTH_MAX + 1];
431
+ snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.value.%s", wu->name_lowercase, job_name_sanitized);
432
+
433
+ char title[1000 + 1];
434
+ snprintf(title, 1000, "Netdata Workers %s value of %s", wu->name_lowercase, string2str(wu->per_job_type[i].name));
435
+
436
+ wu->per_job_type[i].st = rrdset_create_localhost(
437
+ "netdata"
438
+ , name
439
+ , NULL
440
+ , wu->family
441
+ , context
442
+ , title
443
+ , (wu->per_job_type[i].units)?string2str(wu->per_job_type[i].units):"value"
444
+ , "netdata"
445
+ , "stats"
446
+ , wu->priority + 5 + i
447
+ , localhost->rrd_update_every
448
+ , RRDSET_TYPE_LINE
449
+ );
450
+
451
+ wu->per_job_type[i].rd_min = rrddim_add(wu->per_job_type[i].st, "min", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
452
+ wu->per_job_type[i].rd_max = rrddim_add(wu->per_job_type[i].st, "max", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
453
+ wu->per_job_type[i].rd_avg = rrddim_add(wu->per_job_type[i].st, "average", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
454
+ }
455
+
456
+ rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_min, (collected_number)(wu->per_job_type[i].min_value * WORKER_CHART_DECIMAL_PRECISION));
457
+ rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_max, (collected_number)(wu->per_job_type[i].max_value * WORKER_CHART_DECIMAL_PRECISION));
458
+ rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_avg, (collected_number)(wu->per_job_type[i].sum_value / wu->per_job_type[i].count_value * WORKER_CHART_DECIMAL_PRECISION));
459
+
460
+ rrdset_done(wu->per_job_type[i].st);
461
+ }
462
+ }
463
+
464
+ // ----------------------------------------------------------------------
465
+ // custom metric types WORKER_METRIC_INCREMENTAL
466
+
467
+ {
468
+ size_t i;
469
+ for (i = 0; i <= wu->workers_max_job_id ; i++) {
470
+ if(wu->per_job_type[i].type != WORKER_METRIC_INCREMENT && wu->per_job_type[i].type != WORKER_METRIC_INCREMENTAL_TOTAL)
471
+ continue;
472
+
473
+ if(!wu->per_job_type[i].count_value)
474
+ continue;
475
+
476
+ if(!wu->per_job_type[i].st) {
477
+ size_t job_name_len = string_strlen(wu->per_job_type[i].name);
478
+ if(job_name_len > RRD_ID_LENGTH_MAX) job_name_len = RRD_ID_LENGTH_MAX;
479
+
480
+ char job_name_sanitized[job_name_len + 1];
481
+ rrdset_strncpyz_name(job_name_sanitized, string2str(wu->per_job_type[i].name), job_name_len);
482
+
483
+ char name[RRD_ID_LENGTH_MAX + 1];
484
+ snprintfz(name, RRD_ID_LENGTH_MAX, "workers_%s_rate_%s", wu->name_lowercase, job_name_sanitized);
485
+
486
+ char context[RRD_ID_LENGTH_MAX + 1];
487
+ snprintf(context, RRD_ID_LENGTH_MAX, "netdata.workers.%s.rate.%s", wu->name_lowercase, job_name_sanitized);
488
+
489
+ char title[1000 + 1];
490
+ snprintf(title, 1000, "Netdata Workers %s rate of %s", wu->name_lowercase, string2str(wu->per_job_type[i].name));
491
+
492
+ wu->per_job_type[i].st = rrdset_create_localhost(
493
+ "netdata"
494
+ , name
495
+ , NULL
496
+ , wu->family
497
+ , context
498
+ , title
499
+ , (wu->per_job_type[i].units)?string2str(wu->per_job_type[i].units):"rate"
500
+ , "netdata"
501
+ , "stats"
502
+ , wu->priority + 5 + i
503
+ , localhost->rrd_update_every
504
+ , RRDSET_TYPE_LINE
505
+ );
506
+
507
+ wu->per_job_type[i].rd_min = rrddim_add(wu->per_job_type[i].st, "min", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
508
+ wu->per_job_type[i].rd_max = rrddim_add(wu->per_job_type[i].st, "max", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
509
+ wu->per_job_type[i].rd_avg = rrddim_add(wu->per_job_type[i].st, "average", NULL, 1, WORKER_CHART_DECIMAL_PRECISION, RRD_ALGORITHM_ABSOLUTE);
510
+ }
511
+
512
+ rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_min, (collected_number)(wu->per_job_type[i].min_value * WORKER_CHART_DECIMAL_PRECISION));
513
+ rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_max, (collected_number)(wu->per_job_type[i].max_value * WORKER_CHART_DECIMAL_PRECISION));
514
+ rrddim_set_by_pointer(wu->per_job_type[i].st, wu->per_job_type[i].rd_avg, (collected_number)(wu->per_job_type[i].sum_value / wu->per_job_type[i].count_value * WORKER_CHART_DECIMAL_PRECISION));
515
+
516
+ rrdset_done(wu->per_job_type[i].st);
517
+ }
518
+ }
519
+}
520
+
521
+static void workers_utilization_reset_statistics(struct worker_utilization *wu) {
522
+ wu->workers_registered = 0;
523
+ wu->workers_busy = 0;
524
+ wu->workers_total_busy_time = 0;
525
+ wu->workers_total_duration = 0;
526
+ wu->workers_total_jobs_started = 0;
527
+ wu->workers_min_busy_time = WORKERS_MIN_PERCENT_DEFAULT;
528
+ wu->workers_max_busy_time = 0;
529
+
530
+ wu->workers_cpu_registered = 0;
531
+ wu->workers_cpu_min = WORKERS_MIN_PERCENT_DEFAULT;
532
+ wu->workers_cpu_max = 0;
533
+ wu->workers_cpu_total = 0;
534
+
535
+ size_t i;
536
+ for(i = 0; i < WORKER_UTILIZATION_MAX_JOB_TYPES ;i++) {
537
+ if(unlikely(!wu->name_lowercase)) {
538
+ wu->name_lowercase = strdupz(wu->name);
539
+ char *s = wu->name_lowercase;
540
+ for( ; *s ; s++) *s = tolower(*s);
541
+ }
542
+
543
+ wu->per_job_type[i].jobs_started = 0;
544
+ wu->per_job_type[i].busy_time = 0;
545
+
546
+ wu->per_job_type[i].min_value = NAN;
547
+ wu->per_job_type[i].max_value = NAN;
548
+ wu->per_job_type[i].sum_value = NAN;
549
+ wu->per_job_type[i].count_value = 0;
550
+ }
551
+
552
+ struct worker_thread *wt;
553
+ for(wt = wu->threads; wt ; wt = wt->next) {
554
+ wt->enabled = false;
555
+ wt->cpu_enabled = false;
556
+ }
557
+}
558
+
559
+#define TASK_STAT_PREFIX "/proc/self/task/"
560
+#define TASK_STAT_SUFFIX "/stat"
561
+
562
+static int read_thread_cpu_time_from_proc_stat(pid_t pid __maybe_unused, kernel_uint_t *utime __maybe_unused, kernel_uint_t *stime __maybe_unused) {
563
+#ifdef __linux__
564
+ static char filename[sizeof(TASK_STAT_PREFIX) + sizeof(TASK_STAT_SUFFIX) + 20] = TASK_STAT_PREFIX;
565
+ static size_t start_pos = sizeof(TASK_STAT_PREFIX) - 1;
566
+ static procfile *ff = NULL;
567
+
568
+ // construct the filename
569
+ size_t end_pos = snprintfz(&filename[start_pos], 20, "%d", pid);
570
+ strcpy(&filename[start_pos + end_pos], TASK_STAT_SUFFIX);
571
+
572
+ // (re)open the procfile to the new filename
573
+ bool set_quotes = (ff == NULL) ? true : false;
574
+ ff = procfile_reopen(ff, filename, NULL, PROCFILE_FLAG_ERROR_ON_ERROR_LOG);
575
+ if(unlikely(!ff)) return -1;
576
+
577
+ if(set_quotes)
578
+ procfile_set_open_close(ff, "(", ")");
579
+
580
+ // read the entire file and split it to lines and words
581
+ ff = procfile_readall(ff);
582
+ if(unlikely(!ff)) return -1;
583
+
584
+ // parse the numbers we are interested
585
+ *utime = str2kernel_uint_t(procfile_lineword(ff, 0, 13));
586
+ *stime = str2kernel_uint_t(procfile_lineword(ff, 0, 14));
587
+
588
+ // leave the file open for the next iteration
589
+
590
+ return 0;
591
+#else
592
+ // TODO: add here cpu time detection per thread, for FreeBSD and MacOS
593
+ *utime = 0;
594
+ *stime = 0;
595
+ return 1;
596
+#endif
597
+}
598
+
599
+static Pvoid_t workers_by_pid_JudyL_array = NULL;
600
+
601
+static void workers_threads_cleanup(struct worker_utilization *wu) {
602
+ struct worker_thread *t = wu->threads;
603
+ while(t) {
604
+ struct worker_thread *next = t->next;
605
+
606
+ if(!t->enabled) {
607
+ JudyLDel(&workers_by_pid_JudyL_array, t->pid, PJE0);
608
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(wu->threads, t, prev, next);
609
+ freez(t);
610
+ }
611
+ t = next;
612
+ }
613
+}
614
+
615
+static struct worker_thread *worker_thread_find(struct worker_utilization *wu __maybe_unused, pid_t pid) {
616
+ struct worker_thread *wt = NULL;
617
+
618
+ Pvoid_t *PValue = JudyLGet(workers_by_pid_JudyL_array, pid, PJE0);
619
+ if(PValue)
620
+ wt = *PValue;
621
+
622
+ return wt;
623
+}
624
+
625
+static struct worker_thread *worker_thread_create(struct worker_utilization *wu, pid_t pid) {
626
+ struct worker_thread *wt;
627
+
628
+ wt = (struct worker_thread *)callocz(1, sizeof(struct worker_thread));
629
+ wt->pid = pid;
630
+
631
+ Pvoid_t *PValue = JudyLIns(&workers_by_pid_JudyL_array, pid, PJE0);
632
+ *PValue = wt;
633
+
634
+ // link it
635
+ DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(wu->threads, wt, prev, next);
636
+
637
+ return wt;
638
+}
639
+
640
+static struct worker_thread *worker_thread_find_or_create(struct worker_utilization *wu, pid_t pid) {
641
+ struct worker_thread *wt;
642
+ wt = worker_thread_find(wu, pid);
643
+ if(!wt) wt = worker_thread_create(wu, pid);
644
+
645
+ return wt;
646
+}
647
+
648
+static void worker_utilization_charts_callback(void *ptr
649
+ , pid_t pid __maybe_unused
650
+ , const char *thread_tag __maybe_unused
651
+ , size_t max_job_id __maybe_unused
652
+ , size_t utilization_usec __maybe_unused
653
+ , size_t duration_usec __maybe_unused
654
+ , size_t jobs_started __maybe_unused
655
+ , size_t is_running __maybe_unused
656
+ , STRING **job_types_names __maybe_unused
657
+ , STRING **job_types_units __maybe_unused
658
+ , WORKER_METRIC_TYPE *job_types_metric_types __maybe_unused
659
+ , size_t *job_types_jobs_started __maybe_unused
660
+ , usec_t *job_types_busy_time __maybe_unused
661
+ , NETDATA_DOUBLE *job_types_custom_metrics __maybe_unused
662
+) {
663
+ struct worker_utilization *wu = (struct worker_utilization *)ptr;
664
+
665
+ // find the worker_thread in the list
666
+ struct worker_thread *wt = worker_thread_find_or_create(wu, pid);
667
+
668
+ if(utilization_usec > duration_usec)
669
+ utilization_usec = duration_usec;
670
+
671
+ wt->enabled = true;
672
+ wt->busy_time = utilization_usec;
673
+ wt->jobs_started = jobs_started;
674
+
675
+ wt->utime_old = wt->utime;
676
+ wt->stime_old = wt->stime;
677
+ wt->collected_time_old = wt->collected_time;
678
+
679
+ if(max_job_id > wu->workers_max_job_id)
680
+ wu->workers_max_job_id = max_job_id;
681
+
682
+ wu->workers_total_busy_time += utilization_usec;
683
+ wu->workers_total_duration += duration_usec;
684
+ wu->workers_total_jobs_started += jobs_started;
685
+ wu->workers_busy += is_running;
686
+ wu->workers_registered++;
687
+
688
+ double util = (double)utilization_usec * 100.0 / (double)duration_usec;
689
+ if(util > wu->workers_max_busy_time)
690
+ wu->workers_max_busy_time = util;
691
+
692
+ if(util < wu->workers_min_busy_time)
693
+ wu->workers_min_busy_time = util;
694
+
695
+ // accumulate per job type statistics
696
+ size_t i;
697
+ for(i = 0; i <= max_job_id ;i++) {
698
+ if(!wu->per_job_type[i].name && job_types_names[i])
699
+ wu->per_job_type[i].name = string_dup(job_types_names[i]);
700
+
701
+ if(!wu->per_job_type[i].units && job_types_units[i])
702
+ wu->per_job_type[i].units = string_dup(job_types_units[i]);
703
+
704
+ wu->per_job_type[i].type = job_types_metric_types[i];
705
+
706
+ wu->per_job_type[i].jobs_started += job_types_jobs_started[i];
707
+ wu->per_job_type[i].busy_time += job_types_busy_time[i];
708
+
709
+ NETDATA_DOUBLE value = job_types_custom_metrics[i];
710
+ if(netdata_double_isnumber(value)) {
711
+ if(!wu->per_job_type[i].count_value) {
712
+ wu->per_job_type[i].count_value = 1;
713
+ wu->per_job_type[i].min_value = value;
714
+ wu->per_job_type[i].max_value = value;
715
+ wu->per_job_type[i].sum_value = value;
716
+ }
717
+ else {
718
+ wu->per_job_type[i].count_value++;
719
+ wu->per_job_type[i].sum_value += value;
720
+ if(value < wu->per_job_type[i].min_value) wu->per_job_type[i].min_value = value;
721
+ if(value > wu->per_job_type[i].max_value) wu->per_job_type[i].max_value = value;
722
+ }
723
+ }
724
+ }
725
+
726
+ // find its CPU utilization
727
+ if((!read_thread_cpu_time_from_proc_stat(pid, &wt->utime, &wt->stime))) {
728
+ wt->collected_time = now_realtime_usec();
729
+ usec_t delta = wt->collected_time - wt->collected_time_old;
730
+
731
+ double utime = (double)(wt->utime - wt->utime_old) / (double)system_hz * 100.0 * (double)USEC_PER_SEC / (double)delta;
732
+ double stime = (double)(wt->stime - wt->stime_old) / (double)system_hz * 100.0 * (double)USEC_PER_SEC / (double)delta;
733
+ double cpu = utime + stime;
734
+ wt->cpu = cpu;
735
+ wt->cpu_enabled = true;
736
+
737
+ wu->workers_cpu_total += cpu;
738
+ if(cpu < wu->workers_cpu_min) wu->workers_cpu_min = cpu;
739
+ if(cpu > wu->workers_cpu_max) wu->workers_cpu_max = cpu;
740
+ }
741
+ wu->workers_cpu_registered += (wt->cpu_enabled) ? 1 : 0;
742
+}
743
+
744
+void telemetry_workers_cleanup(void) {
745
+ int i, j;
746
+ for(i = 0; all_workers_utilization[i].name ;i++) {
747
+ struct worker_utilization *wu = &all_workers_utilization[i];
748
+
749
+ if(wu->name_lowercase) {
750
+ freez(wu->name_lowercase);
751
+ wu->name_lowercase = NULL;
752
+ }
753
+
754
+ for(j = 0; j < WORKER_UTILIZATION_MAX_JOB_TYPES ;j++) {
755
+ string_freez(wu->per_job_type[j].name);
756
+ wu->per_job_type[j].name = NULL;
757
+
758
+ string_freez(wu->per_job_type[j].units);
759
+ wu->per_job_type[j].units = NULL;
760
+ }
761
+
762
+ // mark all threads as not enabled
763
+ struct worker_thread *t;
764
+ for(t = wu->threads; t ; t = t->next)
765
+ t->enabled = false;
766
+
767
+ // let the cleanup job free them
768
+ workers_threads_cleanup(wu);
769
+ }
770
+}
771
+
772
+void telemetry_workers_do(bool extended) {
773
+ if(!extended) return;
774
+
775
+ static size_t iterations = 0;
776
+ iterations++;
777
+
778
+ for(int i = 0; all_workers_utilization[i].name ;i++) {
779
+ workers_utilization_reset_statistics(&all_workers_utilization[i]);
780
+
781
+ workers_foreach(all_workers_utilization[i].name, worker_utilization_charts_callback, &all_workers_utilization[i]);
782
+
783
+ // skip the first iteration, so that we don't accumulate startup utilization to our charts
784
+ if(likely(iterations > 1))
785
+ workers_utilization_update_chart(&all_workers_utilization[i]);
786
+
787
+ workers_threads_cleanup(&all_workers_utilization[i]);
788
+ }
789
+
790
+ workers_total_cpu_utilization_chart();
791
+}
src/daemon/telemetry/telemetry-workers.h
new
+13
@@ -0,0 +1,13 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_WORKERS_H
4
+#define NETDATA_TELEMETRY_WORKERS_H
5
+
6
+#include "daemon/common.h"
7
+
8
+#if defined(TELEMETRY_INTERNALS)
9
+void telemetry_workers_do(bool extended);
10
+void telemetry_workers_cleanup(void);
11
+#endif
12
+
13
+#endif //NETDATA_TELEMETRY_WORKERS_H
src/daemon/telemetry/telemetry.c
new
+198
@@ -0,0 +1,198 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#define TELEMETRY_INTERNALS 1
4
+#include "daemon/common.h"
5
+
6
+#define WORKER_JOB_TELEMETRY_DAEMON 0
7
+#define WORKER_JOB_SQLITE3 1
8
+#define WORKER_JOB_TELEMETRY_HTTP_API 2
9
+#define WORKER_JOB_TELEMETRY_QUERIES 3
10
+#define WORKER_JOB_TELEMETRY_INGESTION 4
11
+#define WORKER_JOB_DBENGINE 5
12
+#define WORKER_JOB_STRINGS 6
13
+#define WORKER_JOB_DICTIONARIES 7
14
+#define WORKER_JOB_TELEMETRY_ML 8
15
+#define WORKER_JOB_TELEMETRY_GORILLA 9
16
+#define WORKER_JOB_HEARTBEAT 10
17
+#define WORKER_JOB_WORKERS 11
18
+#define WORKER_JOB_MALLOC_TRACE 12
19
+#define WORKER_JOB_REGISTRY 13
20
+#define WORKER_JOB_ARAL 14
21
+
22
+#if WORKER_UTILIZATION_MAX_JOB_TYPES < 15
23
+#error "WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 14"
24
+#endif
25
+
26
+bool telemetry_enabled = true;
27
+bool telemetry_extended_enabled = false;
28
+
29
+static void telemetry_register_workers(void) {
30
+ worker_register("STATS");
31
+
32
+ worker_register_job_name(WORKER_JOB_TELEMETRY_DAEMON, "daemon");
33
+ worker_register_job_name(WORKER_JOB_SQLITE3, "sqlite3");
34
+ worker_register_job_name(WORKER_JOB_TELEMETRY_HTTP_API, "http-api");
35
+ worker_register_job_name(WORKER_JOB_TELEMETRY_QUERIES, "queries");
36
+ worker_register_job_name(WORKER_JOB_TELEMETRY_INGESTION, "ingestion");
37
+ worker_register_job_name(WORKER_JOB_DBENGINE, "dbengine");
38
+ worker_register_job_name(WORKER_JOB_STRINGS, "strings");
39
+ worker_register_job_name(WORKER_JOB_DICTIONARIES, "dictionaries");
40
+ worker_register_job_name(WORKER_JOB_TELEMETRY_ML, "ML");
41
+ worker_register_job_name(WORKER_JOB_TELEMETRY_GORILLA, "gorilla");
42
+ worker_register_job_name(WORKER_JOB_HEARTBEAT, "heartbeat");
43
+ worker_register_job_name(WORKER_JOB_WORKERS, "workers");
44
+ worker_register_job_name(WORKER_JOB_MALLOC_TRACE, "malloc_trace");
45
+ worker_register_job_name(WORKER_JOB_REGISTRY, "registry");
46
+ worker_register_job_name(WORKER_JOB_ARAL, "aral");
47
+}
48
+
49
+static void telementry_cleanup(void *pptr)
50
+{
51
+ struct netdata_static_thread *static_thread = CLEANUP_FUNCTION_GET_PTR(pptr);
52
+ if(!static_thread) return;
53
+
54
+ static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
55
+
56
+ telemetry_workers_cleanup();
57
+ worker_unregister();
58
+ netdata_log_info("cleaning up...");
59
+
60
+ static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
61
+}
62
+
63
+void *telemetry_thread_main(void *ptr) {
64
+ CLEANUP_FUNCTION_REGISTER(telementry_cleanup) cleanup_ptr = ptr;
65
+ telemetry_register_workers();
66
+
67
+ int update_every =
68
+ (int)config_get_duration_seconds(CONFIG_SECTION_TELEMETRY, "update every", localhost->rrd_update_every);
69
+ if (update_every < localhost->rrd_update_every) {
70
+ update_every = localhost->rrd_update_every;
71
+ config_set_duration_seconds(CONFIG_SECTION_TELEMETRY, "update every", update_every);
72
+ }
73
+
74
+ telemerty_aral_init();
75
+
76
+ usec_t step = update_every * USEC_PER_SEC;
77
+ heartbeat_t hb;
78
+ heartbeat_init(&hb, USEC_PER_SEC);
79
+ usec_t real_step = USEC_PER_SEC;
80
+
81
+ // keep the randomness at zero
82
+ // to make sure we are not close to any other thread
83
+ hb.randomness = 0;
84
+
85
+ while (service_running(SERVICE_COLLECTORS)) {
86
+ worker_is_idle();
87
+ heartbeat_next(&hb);
88
+ if (real_step < step) {
89
+ real_step += USEC_PER_SEC;
90
+ continue;
91
+ }
92
+ real_step = USEC_PER_SEC;
93
+
94
+ worker_is_busy(WORKER_JOB_TELEMETRY_INGESTION);
95
+ telemetry_ingestion_do(telemetry_extended_enabled);
96
+
97
+ worker_is_busy(WORKER_JOB_TELEMETRY_HTTP_API);
98
+ telemetry_web_do(telemetry_extended_enabled);
99
+
100
+ worker_is_busy(WORKER_JOB_TELEMETRY_QUERIES);
101
+ telemetry_queries_do(telemetry_extended_enabled);
102
+
103
+ worker_is_busy(WORKER_JOB_TELEMETRY_ML);
104
+ telemetry_ml_do(telemetry_extended_enabled);
105
+
106
+ worker_is_busy(WORKER_JOB_TELEMETRY_GORILLA);
107
+ telemetry_gorilla_do(telemetry_extended_enabled);
108
+
109
+ worker_is_busy(WORKER_JOB_HEARTBEAT);
110
+ telemetry_heartbeat_do(telemetry_extended_enabled);
111
+
112
+#ifdef ENABLE_DBENGINE
113
+ if(dbengine_enabled) {
114
+ worker_is_busy(WORKER_JOB_DBENGINE);
115
+ telemetry_dbengine_do(telemetry_extended_enabled);
116
+ }
117
+#endif
118
+
119
+ worker_is_busy(WORKER_JOB_REGISTRY);
120
+ registry_statistics();
121
+
122
+ worker_is_busy(WORKER_JOB_STRINGS);
123
+ telemetry_string_do(telemetry_extended_enabled);
124
+
125
+#ifdef DICT_WITH_STATS
126
+ worker_is_busy(WORKER_JOB_DICTIONARIES);
127
+ telemetry_dictionary_do(telemetry_extended_enabled);
128
+#endif
129
+
130
+#ifdef NETDATA_TRACE_ALLOCATIONS
131
+ worker_is_busy(WORKER_JOB_MALLOC_TRACE);
132
+ telemetry_trace_allocations_do(telemetry_extended_enabled);
133
+#endif
134
+
135
+ worker_is_busy(WORKER_JOB_WORKERS);
136
+ telemetry_workers_do(telemetry_extended_enabled);
137
+
138
+ worker_is_busy(WORKER_JOB_ARAL);
139
+ telemetry_aral_do(telemetry_extended_enabled);
140
+
141
+ // keep this last to have access to the memory counters
142
+ // exposed by everyone else
143
+ worker_is_busy(WORKER_JOB_TELEMETRY_DAEMON);
144
+ telemetry_daemon_do(telemetry_extended_enabled);
145
+ }
146
+
147
+ return NULL;
148
+}
149
+
150
+// ---------------------------------------------------------------------------------------------------------------------
151
+// telemetry extended thread
152
+
153
+static void telemetry_thread_sqlite3_cleanup(void *pptr)
154
+{
155
+ struct netdata_static_thread *static_thread = CLEANUP_FUNCTION_GET_PTR(pptr);
156
+ if (!static_thread)
157
+ return;
158
+
159
+ static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
160
+
161
+ netdata_log_info("cleaning up...");
162
+
163
+ worker_unregister();
164
+
165
+ static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
166
+}
167
+
168
+void *telemetry_thread_sqlite3_main(void *ptr) {
169
+ CLEANUP_FUNCTION_REGISTER(telemetry_thread_sqlite3_cleanup) cleanup_ptr = ptr;
170
+ telemetry_register_workers();
171
+
172
+ int update_every =
173
+ (int)config_get_duration_seconds(CONFIG_SECTION_TELEMETRY, "update every", localhost->rrd_update_every);
174
+ if (update_every < localhost->rrd_update_every) {
175
+ update_every = localhost->rrd_update_every;
176
+ config_set_duration_seconds(CONFIG_SECTION_TELEMETRY, "update every", update_every);
177
+ }
178
+
179
+ usec_t step = update_every * USEC_PER_SEC;
180
+ heartbeat_t hb;
181
+ heartbeat_init(&hb, USEC_PER_SEC);
182
+ usec_t real_step = USEC_PER_SEC;
183
+
184
+ while (service_running(SERVICE_COLLECTORS)) {
185
+ worker_is_idle();
186
+ heartbeat_next(&hb);
187
+ if (real_step < step) {
188
+ real_step += USEC_PER_SEC;
189
+ continue;
190
+ }
191
+ real_step = USEC_PER_SEC;
192
+
193
+ worker_is_busy(WORKER_JOB_SQLITE3);
194
+ telemetry_sqlite3_do(telemetry_extended_enabled);
195
+ }
196
+
197
+ return NULL;
198
+}
src/daemon/telemetry/telemetry.h
new
+30
@@ -0,0 +1,30 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_TELEMETRY_H
4
+#define NETDATA_TELEMETRY_H 1
5
+
6
+#include "database/rrd.h"
7
+
8
+extern bool telemetry_enabled;
9
+extern bool telemetry_extended_enabled;
10
+
11
+#include "telemetry-http-api.h"
12
+#include "telemetry-queries.h"
13
+#include "telemetry-ingestion.h"
14
+#include "telemetry-ml.h"
15
+#include "telemetry-gorilla.h"
16
+#include "telemetry-daemon.h"
17
+#include "telemetry-daemon-memory.h"
18
+#include "telemetry-sqlite3.h"
19
+#include "telemetry-dbengine.h"
20
+#include "telemetry-string.h"
21
+#include "telemetry-heartbeat.h"
22
+#include "telemetry-dictionary.h"
23
+#include "telemetry-workers.h"
24
+#include "telemetry-trace-allocations.h"
25
+#include "telemetry-aral.h"
26
+
27
+void *telemetry_thread_main(void *ptr);
28
+void *telemetry_thread_sqlite3_main(void *ptr);
29
+
30
+#endif /* NETDATA_TELEMETRY_H */
src/database/contexts/api_v2_contexts.c
+12
-35
@@ -215,7 +215,7 @@ static void rrdhost_receiver_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *
215
buffer_json_member_add_object(wb, key);
216
{
217
buffer_json_member_add_uint64(wb, "id", s->ingest.id);
218
- buffer_json_member_add_uint64(wb, "hops", s->ingest.hops);
218
+ buffer_json_member_add_int64(wb, "hops", s->ingest.hops);
219
buffer_json_member_add_string(wb, "type", rrdhost_ingest_type_to_string(s->ingest.type));
220
buffer_json_member_add_string(wb, "status", rrdhost_ingest_status_to_string(s->ingest.status));
221
buffer_json_member_add_time_t(wb, "since", s->ingest.since);
@@ -272,15 +272,13 @@ static void rrdhost_sender_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *ke
272
if (s->stream.status == RRDHOST_STREAM_STATUS_OFFLINE)
273
buffer_json_member_add_string(wb, "reason", stream_handshake_error_to_string(s->stream.reason));
274
275
- if (s->stream.status == RRDHOST_STREAM_STATUS_REPLICATING) {
276
- buffer_json_member_add_object(wb, "replication");
277
- {
278
- buffer_json_member_add_boolean(wb, "in_progress", s->stream.replication.in_progress);
279
- buffer_json_member_add_double(wb, "completion", s->stream.replication.completion);
280
- buffer_json_member_add_uint64(wb, "instances", s->stream.replication.instances);
281
- }
282
- buffer_json_object_close(wb);
275
+ buffer_json_member_add_object(wb, "replication");
276
+ {
277
+ buffer_json_member_add_boolean(wb, "in_progress", s->stream.replication.in_progress);
278
+ buffer_json_member_add_double(wb, "completion", s->stream.replication.completion);
279
+ buffer_json_member_add_uint64(wb, "instances", s->stream.replication.instances);
280
}
281
+ buffer_json_object_close(wb); // replication
282
283
buffer_json_member_add_object(wb, "destination");
284
{
@@ -300,35 +298,14 @@ static void rrdhost_sender_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *ke
298
buffer_json_member_add_uint64(wb, "metadata", s->stream.sent_bytes_on_this_connection_per_type[STREAM_TRAFFIC_TYPE_METADATA]);
299
buffer_json_member_add_uint64(wb, "functions", s->stream.sent_bytes_on_this_connection_per_type[STREAM_TRAFFIC_TYPE_FUNCTIONS]);
300
buffer_json_member_add_uint64(wb, "replication", s->stream.sent_bytes_on_this_connection_per_type[STREAM_TRAFFIC_TYPE_REPLICATION]);
303
- buffer_json_member_add_uint64(wb, "dyncfg", s->stream.sent_bytes_on_this_connection_per_type[STREAM_TRAFFIC_TYPE_DYNCFG]);
301
}
302
buffer_json_object_close(wb); // traffic
303
307
- buffer_json_member_add_array(wb, "candidates");
308
- struct rrdpush_destinations *d;
309
- for (d = s->host->destinations; d; d = d->next) {
310
- buffer_json_add_array_item_object(wb);
311
- buffer_json_member_add_uint64(wb, "attempts", d->attempts);
312
- {
304
+ buffer_json_member_add_array(wb, "parents");
305
+ rrdhost_stream_parents_to_json(wb, s);
306
+ buffer_json_array_close(wb); // parents
307
314
- if (d->ssl) {
315
- snprintfz(buf, sizeof(buf) - 1, "%s:SSL", string2str(d->destination));
316
- buffer_json_member_add_string(wb, "destination", buf);
317
- }
318
- else
319
- buffer_json_member_add_string(wb, "destination", string2str(d->destination));
320
-
321
- buffer_json_member_add_time_t(wb, "since", d->since);
322
- buffer_json_member_add_time_t(wb, "age", s->now - d->since);
323
- buffer_json_member_add_string(wb, "last_handshake", stream_handshake_error_to_string(d->reason));
324
- if(d->postpone_reconnection_until > s->now) {
325
- buffer_json_member_add_time_t(wb, "next_check", d->postpone_reconnection_until);
326
- buffer_json_member_add_time_t(wb, "next_in", d->postpone_reconnection_until - s->now);
327
- }
328
- }
329
- buffer_json_object_close(wb); // each candidate
330
- }
331
- buffer_json_array_close(wb); // candidates
308
+ rrdhost_stream_path_to_json(wb, s->host, STREAM_PATH_JSON_MEMBER, false);
309
}
310
buffer_json_object_close(wb); // destination
311
}
@@ -365,7 +342,7 @@ static inline void rrdhost_health_to_json_v2(BUFFER *wb, const char *key, RRDHOS
342
buffer_json_member_add_object(wb, key);
343
{
344
buffer_json_member_add_string(wb, "status", rrdhost_health_status_to_string(s->health.status));
368
- if (s->health.status == RRDHOST_HEALTH_STATUS_RUNNING) {
345
+ if (s->health.status == RRDHOST_HEALTH_STATUS_RUNNING || s->health.status == RRDHOST_HEALTH_STATUS_INITIALIZING) {
346
buffer_json_member_add_object(wb, "alerts");
347
{
348
buffer_json_member_add_uint64(wb, "critical", s->health.alerts.critical);
src/database/contexts/api_v2_contexts_agents.c
+3
-3
@@ -49,7 +49,7 @@ void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now
49
available_instances += __atomic_load_n(&host->rrdctx.instances_count, __ATOMIC_RELAXED);
50
available_contexts += __atomic_load_n(&host->rrdctx.contexts_count, __ATOMIC_RELAXED);
51
52
- if(rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED))
52
+ if(rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_SENDER_CONNECTED))
53
sending++;
54
55
if (rrdhost_is_online(host)) {
@@ -103,12 +103,12 @@ void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now
103
buffer_json_object_close(wb); // api
104
105
buffer_json_member_add_array(wb, "db_size");
106
- size_t group_seconds = localhost->rrd_update_every;
106
+ size_t group_seconds;
107
for (size_t tier = 0; tier < storage_tiers; tier++) {
108
STORAGE_ENGINE *eng = localhost->db[tier].eng;
109
if (!eng) continue;
110
111
- group_seconds *= storage_tiers_grouping_iterations[tier];
111
+ group_seconds = get_tier_grouping(tier) * localhost->rrd_update_every;
112
uint64_t max = storage_engine_disk_space_max(eng->seb, localhost->db[tier].si);
113
uint64_t used = storage_engine_disk_space_used(eng->seb, localhost->db[tier].si);
114
#ifdef ENABLE_DBENGINE
src/database/contexts/api_v2_contexts_alert_config.c
+1
-1
@@ -81,7 +81,7 @@ void contexts_v2_alert_config_to_json_from_sql_alert_config_data(struct sql_aler
81
{
82
buffer_json_member_add_string(wb, "type", "agent");
83
buffer_json_member_add_string(wb, "exec", t->notification.exec ? t->notification.exec : NULL);
84
- buffer_json_member_add_string(wb, "to", t->notification.to_key ? t->notification.to_key : string2str(localhost->health.health_default_recipient));
84
+ buffer_json_member_add_string(wb, "to", t->notification.to_key ? t->notification.to_key : string2str(localhost->health.default_recipient));
85
buffer_json_member_add_string(wb, "delay", t->notification.delay);
86
buffer_json_member_add_string(wb, "repeat", t->notification.repeat);
87
buffer_json_member_add_string(wb, "options", t->notification.options);
src/database/contexts/api_v2_contexts_alert_transitions.c
+3
-3
@@ -240,7 +240,7 @@ static void contexts_v2_alert_transition_callback(struct sql_alert_transition_da
240
[ATF_CLASS] = t->classification,
241
[ATF_TYPE] = t->type,
242
[ATF_COMPONENT] = t->component,
243
- [ATF_ROLE] = t->recipient && *t->recipient ? t->recipient : string2str(localhost->health.health_default_recipient),
243
+ [ATF_ROLE] = t->recipient && *t->recipient ? t->recipient : string2str(localhost->health.default_recipient),
244
[ATF_NODE] = machine_guid,
245
[ATF_ALERT_NAME] = t->alert_name,
246
[ATF_CHART_NAME] = t->chart_name,
@@ -411,9 +411,9 @@ void contexts_v2_alert_transitions_to_json(BUFFER *wb, struct rrdcontext_to_json
411
buffer_json_member_add_time_t(wb, "delay", t->delay);
412
buffer_json_member_add_time_t(wb, "delay_up_to_time", t->delay_up_to_timestamp);
413
health_entry_flags_to_json_array(wb, "flags", t->flags);
414
- buffer_json_member_add_string(wb, "exec", *t->exec ? t->exec : string2str(localhost->health.health_default_exec));
414
+ buffer_json_member_add_string(wb, "exec", *t->exec ? t->exec : string2str(localhost->health.default_exec));
415
buffer_json_member_add_uint64(wb, "exec_code", t->exec_code);
416
- buffer_json_member_add_string(wb, "to", *t->recipient ? t->recipient : string2str(localhost->health.health_default_recipient));
416
+ buffer_json_member_add_string(wb, "to", *t->recipient ? t->recipient : string2str(localhost->health.default_recipient));
417
}
418
buffer_json_object_close(wb); // notification
419
}
src/database/engine/README.md
+1
-1
@@ -159,7 +159,7 @@ Then `x 2` is the worst case estimate for the dirty queue. If all collected metr
159
160
The memory we saved with the above is used to improve the LRU cache. So, although we reserved 32MiB for the LRU, in bigger setups (Netdata Parents) the LRU grows a lot more, within the limits of the equation.
161
162
-In practice, the main cache sizes itself with `hot x 1.5` instead of `host x 2`. The reason is that 5% of the main cache is reserved for expanding open cache, 5% for expanding extent cache, and we need Room for the extensive buffers that are allocated in these setups. When the main cache exceeds `hot x 1.5` it enters a mode of critical evictions, and aggressively frees pages from the LRU to maintain a healthy memory footprint within its design limits.
162
+In practice, the main cache sizes itself with `hot x 1.5` instead of `hot x 2`. The reason is that 5% of the main cache is reserved for expanding open cache, 5% for expanding extent cache, and we need Room for the extensive buffers that are allocated in these setups. When the main cache exceeds `hot x 1.5` it enters a mode of critical evictions, and aggressively frees pages from the LRU to maintain a healthy memory footprint within its design limits.
163
164
#### Open Cache
165
src/database/engine/cache.c
+624
-277
@@ -18,7 +18,9 @@ typedef int32_t REFCOUNT;
18
#define REFCOUNT_DELETING (-100)
19
20
// to use ARAL uncomment the following line:
21
+#if !defined(FSANITIZE_ADDRESS)
22
#define PGC_WITH_ARAL 1
23
+#endif
24
25
typedef enum __attribute__ ((__packed__)) {
26
// mutually exclusive flags
@@ -68,8 +70,8 @@ struct pgc_page {
70
// THIS STRUCTURE NEEDS TO BE INITIALIZED BY HAND!
71
};
72
71
-struct pgc_linked_list {
72
- SPINLOCK spinlock;
73
+struct pgc_queue {
74
+ alignas(64) SPINLOCK spinlock;
75
union {
76
PGC_PAGE *base;
77
Pvoid_t sections_judy;
@@ -84,6 +86,8 @@ struct pgc_linked_list {
86
struct pgc {
87
struct {
88
char name[PGC_NAME_MAX + 1];
89
+ bool stats; // enable extended statistics
90
+ bool use_all_ram;
91
92
size_t partitions;
93
size_t clean_size;
@@ -93,6 +97,7 @@ struct pgc {
97
size_t max_flushes_inline;
98
size_t max_workers_evict_inline;
99
size_t additional_bytes_per_page;
100
+ size_t out_of_memory_protection_bytes;
101
free_clean_page_callback pgc_free_clean_cb;
102
save_dirty_page_callback pgc_save_dirty_cb;
103
save_dirty_init_callback pgc_save_init_cb;
@@ -104,52 +109,40 @@ struct pgc {
109
size_t evict_low_threshold_per1000;
110
111
dynamic_target_cache_size_callback dynamic_target_size_cb;
112
+ nominal_page_size_callback nominal_page_size_cb;
113
} config;
114
109
-#ifdef PGC_WITH_ARAL
110
- ARAL **aral;
111
-#endif
112
-
113
- PGC_CACHE_LINE_PADDING(0);
115
+ struct {
116
+ SPINLOCK spinlock; // when locked, the evict_thread is currently evicting pages
117
+ ND_THREAD *thread; // the thread
118
+ struct completion completion; // signal the thread to wake up
119
+ } evictor;
120
121
struct pgc_index {
116
- RW_SPINLOCK rw_spinlock;
122
+ alignas(64) RW_SPINLOCK rw_spinlock;
123
Pvoid_t sections_judy;
118
- PGC_CACHE_LINE_PADDING(0);
124
} *index;
125
121
- PGC_CACHE_LINE_PADDING(1);
126
+#ifdef PGC_WITH_ARAL
127
+ ARAL *aral;
128
+#endif
129
130
struct {
124
- SPINLOCK spinlock;
131
+ alignas(64) SPINLOCK spinlock;
132
size_t per1000;
133
} usage;
134
128
- PGC_CACHE_LINE_PADDING(2);
129
-
130
- struct pgc_linked_list clean; // LRU is applied here to free memory from the cache
131
-
132
- PGC_CACHE_LINE_PADDING(3);
133
-
134
- struct pgc_linked_list dirty; // in the dirty list, pages are ordered the way they were marked dirty
135
-
136
- PGC_CACHE_LINE_PADDING(4);
137
-
138
- struct pgc_linked_list hot; // in the hot list, pages are order the way they were marked hot
139
-
140
- PGC_CACHE_LINE_PADDING(5);
141
-
135
+ struct pgc_queue clean; // LRU is applied here to free memory from the cache
136
+ struct pgc_queue dirty; // in the dirty list, pages are ordered the way they were marked dirty
137
+ struct pgc_queue hot; // in the hot list, pages are order the way they were marked hot
138
struct pgc_statistics stats; // statistics
139
140
#ifdef NETDATA_PGC_POINTER_CHECK
145
- PGC_CACHE_LINE_PADDING(6);
146
- netdata_mutex_t global_pointer_registry_mutex;
141
+ alignas(64) netdata_mutex_t global_pointer_registry_mutex;
142
Pvoid_t global_pointer_registry;
143
#endif
144
};
145
151
-
152
-
146
// ----------------------------------------------------------------------------
147
// validate each pointer is indexed once - internal checks only
148
@@ -207,6 +200,17 @@ static inline void pointer_del(PGC *cache __maybe_unused, PGC_PAGE *page __maybe
200
#endif
201
}
202
203
+// ----------------------------------------------------------------------------
204
+// helpers
205
+
206
+static inline size_t page_assumed_size(PGC *cache, size_t size) {
207
+ return size + (sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page + sizeof(Word_t) * 3);
208
+}
209
+
210
+static inline size_t page_size_from_assumed_size(PGC *cache, size_t assumed_size) {
211
+ return assumed_size - (sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page + sizeof(Word_t) * 3);
212
+}
213
+
214
// ----------------------------------------------------------------------------
215
// locking
216
@@ -235,16 +239,23 @@ static inline void pgc_index_write_lock(PGC *cache, size_t partition) {
239
static inline void pgc_index_write_unlock(PGC *cache, size_t partition) {
240
rw_spinlock_write_unlock(&cache->index[partition].rw_spinlock);
241
}
242
+static inline bool pgc_index_trywrite_lock(PGC *cache, size_t partition, bool force) {
243
+ if(force) {
244
+ rw_spinlock_write_lock(&cache->index[partition].rw_spinlock);
245
+ return true;
246
+ }
247
+ return rw_spinlock_trywrite_lock(&cache->index[partition].rw_spinlock);
248
+}
249
239
-static inline bool pgc_ll_trylock(PGC *cache __maybe_unused, struct pgc_linked_list *ll) {
250
+static inline bool pgc_queue_trylock(PGC *cache __maybe_unused, struct pgc_queue *ll) {
251
return spinlock_trylock(&ll->spinlock);
252
}
253
243
-static inline void pgc_ll_lock(PGC *cache __maybe_unused, struct pgc_linked_list *ll) {
254
+static inline void pgc_queue_lock(PGC *cache __maybe_unused, struct pgc_queue *ll) {
255
spinlock_lock(&ll->spinlock);
256
}
257
247
-static inline void pgc_ll_unlock(PGC *cache __maybe_unused, struct pgc_linked_list *ll) {
258
+static inline void pgc_queue_unlock(PGC *cache __maybe_unused, struct pgc_queue *ll) {
259
spinlock_unlock(&ll->spinlock);
260
}
261
@@ -260,6 +271,76 @@ static inline void page_transition_unlock(PGC *cache __maybe_unused, PGC_PAGE *p
271
spinlock_unlock(&page->transition_spinlock);
272
}
273
274
+// ----------------------------------------------------------------------------
275
+// size histogram
276
+
277
+static void pgc_size_histogram_init(struct pgc_size_histogram *h) {
278
+ // the histogram needs to be all-inclusive for the possible sizes
279
+ // so, we start from 0, and the last value is SIZE_MAX.
280
+
281
+ size_t values[PGC_SIZE_HISTOGRAM_ENTRIES] = {
282
+ 0, 32, 64, 128, 256, 512, 1024, 2048,
283
+ 4096, 8192, 16384, 32768, 65536, 128 * 1024, SIZE_MAX
284
+ };
285
+
286
+ size_t last_value = 0;
287
+ for(size_t i = 0; i < PGC_SIZE_HISTOGRAM_ENTRIES; i++) {
288
+ if(i > 0 && values[i] == 0)
289
+ fatal("only the first value in the array can be zero");
290
+
291
+ if(i > 0 && values[i] <= last_value)
292
+ fatal("the values need to be sorted");
293
+
294
+ h->array[i].upto = values[i];
295
+ last_value = values[i];
296
+ }
297
+}
298
+
299
+static inline size_t pgc_size_histogram_slot(struct pgc_size_histogram *h, size_t size) {
300
+ if(size <= h->array[0].upto)
301
+ return 0;
302
+
303
+ if(size >= h->array[_countof(h->array) - 1].upto)
304
+ return _countof(h->array) - 1;
305
+
306
+ // binary search for the right size
307
+ size_t low = 0, high = _countof(h->array) - 1;
308
+ while (low < high) {
309
+ size_t mid = low + (high - low) / 2;
310
+ if (size < h->array[mid].upto)
311
+ high = mid;
312
+ else
313
+ low = mid + 1;
314
+ }
315
+ return low - 1;
316
+}
317
+
318
+static inline void pgc_size_histogram_add(PGC *cache, struct pgc_size_histogram *h, PGC_PAGE *page) {
319
+ size_t size;
320
+ if(cache->config.nominal_page_size_cb)
321
+ size = cache->config.nominal_page_size_cb(page->data);
322
+ else
323
+ size = page_size_from_assumed_size(cache, page->assumed_size);
324
+
325
+ size_t slot = pgc_size_histogram_slot(h, size);
326
+ internal_fatal(slot >= _countof(h->array), "hey!");
327
+
328
+ __atomic_add_fetch(&h->array[slot].count, 1, __ATOMIC_RELAXED);
329
+}
330
+
331
+static inline void pgc_size_histogram_del(PGC *cache, struct pgc_size_histogram *h, PGC_PAGE *page) {
332
+ size_t size;
333
+ if(cache->config.nominal_page_size_cb)
334
+ size = cache->config.nominal_page_size_cb(page->data);
335
+ else
336
+ size = page_size_from_assumed_size(cache, page->assumed_size);
337
+
338
+ size_t slot = pgc_size_histogram_slot(h, size);
339
+ internal_fatal(slot >= _countof(h->array), "hey!");
340
+
341
+ __atomic_sub_fetch(&h->array[slot].count, 1, __ATOMIC_RELAXED);
342
+}
343
+
344
// ----------------------------------------------------------------------------
345
// evictions control
346
@@ -271,48 +352,72 @@ static inline size_t cache_usage_per1000(PGC *cache, size_t *size_to_evict) {
352
else if(!spinlock_trylock(&cache->usage.spinlock))
353
return __atomic_load_n(&cache->usage.per1000, __ATOMIC_RELAXED);
354
274
- size_t current_cache_size;
355
size_t wanted_cache_size;
276
- size_t per1000;
356
278
- size_t dirty = __atomic_load_n(&cache->dirty.stats->size, __ATOMIC_RELAXED);
279
- size_t hot = __atomic_load_n(&cache->hot.stats->size, __ATOMIC_RELAXED);
357
+ const size_t dirty = __atomic_load_n(&cache->dirty.stats->size, __ATOMIC_RELAXED);
358
+ const size_t hot = __atomic_load_n(&cache->hot.stats->size, __ATOMIC_RELAXED);
359
+ const size_t clean = __atomic_load_n(&cache->clean.stats->size, __ATOMIC_RELAXED);
360
+ const size_t evicting = __atomic_load_n(&cache->stats.evicting_size, __ATOMIC_RELAXED);
361
+ const size_t flushing = __atomic_load_n(&cache->stats.flushing_size, __ATOMIC_RELAXED);
362
+ const size_t current_cache_size = __atomic_load_n(&cache->stats.size, __ATOMIC_RELAXED);
363
+ const size_t all_pages_size = hot + dirty + clean + evicting + flushing;
364
+ const size_t index = current_cache_size > all_pages_size ? current_cache_size - all_pages_size : 0;
365
+ const size_t referenced_size = __atomic_load_n(&cache->stats.referenced_size, __ATOMIC_RELAXED);
366
367
if(cache->config.options & PGC_OPTIONS_AUTOSCALE) {
282
- size_t dirty_max = __atomic_load_n(&cache->dirty.stats->max_size, __ATOMIC_RELAXED);
283
- size_t hot_max = __atomic_load_n(&cache->hot.stats->max_size, __ATOMIC_RELAXED);
368
+ const size_t dirty_max = __atomic_load_n(&cache->dirty.stats->max_size, __ATOMIC_RELAXED);
369
+ const size_t hot_max = __atomic_load_n(&cache->hot.stats->max_size, __ATOMIC_RELAXED);
370
371
// our promise to users
286
- size_t max_size1 = MAX(hot_max, hot) * 2;
372
+ const size_t max_size1 = MAX(hot_max, hot) * 2;
373
374
// protection against slow flushing
289
- size_t max_size2 = hot_max + ((dirty_max < hot_max / 2) ? hot_max / 2 : dirty_max * 2);
375
+ const size_t max_size2 = hot_max + ((dirty_max * 2 < hot_max * 2 / 3) ? hot_max * 2 / 3 : dirty_max * 2) + index;
376
377
// the final wanted cache size
378
wanted_cache_size = MIN(max_size1, max_size2);
379
380
if(cache->config.dynamic_target_size_cb) {
295
- size_t wanted_cache_size_cb = cache->config.dynamic_target_size_cb();
381
+ const size_t wanted_cache_size_cb = cache->config.dynamic_target_size_cb();
382
if(wanted_cache_size_cb > wanted_cache_size)
383
wanted_cache_size = wanted_cache_size_cb;
384
}
385
300
- if (wanted_cache_size < hot + dirty + cache->config.clean_size)
301
- wanted_cache_size = hot + dirty + cache->config.clean_size;
386
+ if (wanted_cache_size < hot + dirty + index + cache->config.clean_size)
387
+ wanted_cache_size = hot + dirty + index + cache->config.clean_size;
388
}
389
else
304
- wanted_cache_size = hot + dirty + cache->config.clean_size;
390
+ wanted_cache_size = hot + dirty + index + cache->config.clean_size;
391
306
- // protection again huge queries
392
+ // protection against huge queries
393
// if huge queries are running, or huge amounts need to be saved
308
- // allow the cache to grow more (hot pages in main cache are also referenced)
309
- size_t referenced_size = __atomic_load_n(&cache->stats.referenced_size, __ATOMIC_RELAXED);
310
- if(unlikely(wanted_cache_size < referenced_size * 2 / 3))
311
- wanted_cache_size = referenced_size * 2 / 3;
312
-
313
- current_cache_size = __atomic_load_n(&cache->stats.size, __ATOMIC_RELAXED); // + pgc_aral_overhead();
394
+ // allow the cache to grow more (hot pages in the main cache are also referenced)
395
+ if(unlikely(wanted_cache_size < referenced_size + dirty))
396
+ wanted_cache_size = referenced_size + dirty;
397
+
398
+ // if we don't have enough clean pages, there is no reason to be aggressive or critical
399
+ if(current_cache_size > wanted_cache_size && wanted_cache_size < current_cache_size - clean)
400
+ wanted_cache_size = current_cache_size - clean;
401
+
402
+ if(cache->config.out_of_memory_protection_bytes) {
403
+ // out of memory protection
404
+ OS_SYSTEM_MEMORY sm = os_system_memory(false);
405
+ if(sm.ram_total_bytes) {
406
+ // when the total exists, ram_available_bytes is also right
407
+
408
+ const size_t min_available = cache->config.out_of_memory_protection_bytes;
409
+ if (sm.ram_available_bytes < min_available) {
410
+ // we must shrink
411
+ wanted_cache_size = current_cache_size - (min_available - sm.ram_available_bytes);
412
+ }
413
+ else if(cache->config.use_all_ram) {
414
+ // we can grow
415
+ wanted_cache_size = current_cache_size + (sm.ram_available_bytes - min_available);
416
+ }
417
+ }
418
+ }
419
315
- per1000 = (size_t)((unsigned long long)current_cache_size * 1000ULL / (unsigned long long)wanted_cache_size);
420
+ const size_t per1000 = (size_t)((unsigned long long)current_cache_size * 1000ULL / (unsigned long long)wanted_cache_size);
421
422
__atomic_store_n(&cache->usage.per1000, per1000, __ATOMIC_RELAXED);
423
__atomic_store_n(&cache->stats.wanted_cache_size, wanted_cache_size, __ATOMIC_RELAXED);
@@ -321,7 +426,11 @@ static inline size_t cache_usage_per1000(PGC *cache, size_t *size_to_evict) {
426
spinlock_unlock(&cache->usage.spinlock);
427
428
if(size_to_evict) {
324
- size_t target = (size_t)((unsigned long long)wanted_cache_size * (unsigned long long)cache->config.evict_low_threshold_per1000 / 1000ULL);
429
+ size_t target = (size_t)((uint64_t)wanted_cache_size * (uint64_t)cache->config.evict_low_threshold_per1000 / 1000ULL);
430
+
431
+ if(target < wanted_cache_size - clean)
432
+ target = wanted_cache_size - clean;
433
+
434
if(current_cache_size > target)
435
*size_to_evict = current_cache_size - target;
436
else
@@ -349,24 +458,67 @@ typedef bool (*evict_filter)(PGC_PAGE *page, void *data);
458
static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evict, bool wait, bool all_of_them, evict_filter filter, void *data);
459
#define evict_pages(cache, max_skip, max_evict, wait, all_of_them) evict_pages_with_filter(cache, max_skip, max_evict, wait, all_of_them, NULL, NULL)
460
352
-static inline void evict_on_clean_page_added(PGC *cache __maybe_unused) {
353
- if((cache->config.options & PGC_OPTIONS_EVICT_PAGES_INLINE) || cache_needs_space_aggressively(cache)) {
354
- evict_pages(cache,
355
- cache->config.max_skip_pages_per_inline_eviction,
356
- cache->config.max_pages_per_inline_eviction,
357
- false, false);
461
+static inline bool flushing_critical(PGC *cache);
462
+static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wait, bool all_of_them);
463
+
464
+static void signal_evict_thread_or_evict_inline(PGC *cache, bool on_release) {
465
+ const size_t per1000 = cache_usage_per1000(cache, NULL);
466
+
467
+ if (per1000 >= cache->config.healthy_size_per1000 && spinlock_trylock(&cache->evictor.spinlock)) {
468
+ __atomic_add_fetch(&cache->stats.waste_evict_thread_signals, 1, __ATOMIC_RELAXED);
469
+ completion_mark_complete_a_job(&cache->evictor.completion);
470
+ spinlock_unlock(&cache->evictor.spinlock);
471
+ }
472
+
473
+ if(!(cache->config.options & PGC_OPTIONS_EVICT_PAGES_NO_INLINE)) {
474
+ if (per1000 > cache->config.aggressive_evict_per1000 && !on_release) {
475
+ // the threads that add pages, turn into evictors when the cache needs evictions aggressively
476
+ __atomic_add_fetch(&cache->stats.waste_evictions_inline_on_add, 1, __ATOMIC_RELAXED);
477
+ evict_pages(cache,
478
+ cache->config.max_skip_pages_per_inline_eviction,
479
+ cache->config.max_pages_per_inline_eviction,
480
+ false, false);
481
+ }
482
+ else if (per1000 > cache->config.severe_pressure_per1000 && on_release) {
483
+ // the threads that are releasing pages, turn into evictors when the cache is critical
484
+ __atomic_add_fetch(&cache->stats.waste_evictions_inline_on_release, 1, __ATOMIC_RELAXED);
485
+
486
+ evict_pages(cache,
487
+ cache->config.max_skip_pages_per_inline_eviction,
488
+ cache->config.max_pages_per_inline_eviction,
489
+ false, false);
490
+ }
491
}
492
}
493
361
-static inline void evict_on_page_release_when_permitted(PGC *cache __maybe_unused) {
362
- if ((cache->config.options & PGC_OPTIONS_EVICT_PAGES_INLINE) || cache_under_severe_pressure(cache)) {
363
- evict_pages(cache,
364
- cache->config.max_skip_pages_per_inline_eviction,
365
- cache->config.max_pages_per_inline_eviction,
366
- false, false);
494
+static inline void evict_on_clean_page_added(PGC *cache) {
495
+ signal_evict_thread_or_evict_inline(cache, false);
496
+}
497
+
498
+static inline void evict_on_page_release_when_permitted(PGC *cache) {
499
+ signal_evict_thread_or_evict_inline(cache, true);
500
+}
501
+
502
+static inline void flush_inline(PGC *cache, bool on_release) {
503
+ if(!(cache->config.options & PGC_OPTIONS_FLUSH_PAGES_NO_INLINE) && flushing_critical(cache)) {
504
+ if (on_release)
505
+ __atomic_add_fetch(&cache->stats.waste_flush_on_release, 1, __ATOMIC_RELAXED);
506
+ else
507
+ __atomic_add_fetch(&cache->stats.waste_flush_on_add, 1, __ATOMIC_RELAXED);
508
+
509
+ flush_pages(cache, cache->config.max_flushes_inline, PGC_SECTION_ALL, false, false);
510
}
511
}
512
513
+static inline void flush_on_page_add(PGC *cache) {
514
+ flush_inline(cache, false);
515
+}
516
+
517
+static inline void flush_on_page_hot_release(PGC *cache) {
518
+ flush_inline(cache, true);
519
+}
520
+
521
+
522
// ----------------------------------------------------------------------------
523
// flushing control
524
@@ -381,17 +533,6 @@ static inline bool flushing_critical(PGC *cache) {
533
return false;
534
}
535
384
-// ----------------------------------------------------------------------------
385
-// helpers
386
-
387
-static inline size_t page_assumed_size(PGC *cache, size_t size) {
388
- return size + (sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page + sizeof(Word_t) * 3);
389
-}
390
-
391
-static inline size_t page_size_from_assumed_size(PGC *cache, size_t assumed_size) {
392
- return assumed_size - (sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page + sizeof(Word_t) * 3);
393
-}
394
-
536
// ----------------------------------------------------------------------------
537
// Linked list management
538
@@ -416,27 +557,34 @@ struct section_pages {
557
PGC_PAGE *base;
558
};
559
419
-static ARAL *pgc_section_pages_aral = NULL;
560
+static struct aral_statistics aral_statistics_for_pgc = { 0 };
561
+
562
+static ARAL *pgc_sections_aral = NULL;
563
+static ARAL *pgc_pages_aral = NULL;
564
+
565
static void pgc_section_pages_static_aral_init(void) {
566
static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER;
567
423
- if(unlikely(!pgc_section_pages_aral)) {
424
- spinlock_lock(&spinlock);
568
+ spinlock_lock(&spinlock);
569
426
- // we have to check again
427
- if(!pgc_section_pages_aral)
428
- pgc_section_pages_aral = aral_create(
429
- "pgc_section",
430
- sizeof(struct section_pages),
431
- 0,
432
- 65536, NULL,
433
- NULL, NULL, false, false);
570
+ if(!pgc_sections_aral)
571
+ pgc_sections_aral = aral_by_size_acquire(sizeof(struct section_pages));
572
435
- spinlock_unlock(&spinlock);
573
+ if(!pgc_pages_aral) {
574
+ pgc_pages_aral = aral_create(
575
+ "pgc_pages",
576
+ sizeof(PGC_PAGE),
577
+ 0,
578
+ 0,
579
+ &aral_statistics_for_pgc,
580
+ NULL, NULL, false, false);
581
}
582
+
583
+ spinlock_unlock(&spinlock);
584
}
585
439
-static inline void pgc_stats_ll_judy_change(PGC *cache, struct pgc_linked_list *ll, size_t mem_before_judyl, size_t mem_after_judyl) {
586
+static inline void
587
+pgc_stats_queue_judy_change(PGC *cache, struct pgc_queue *ll, size_t mem_before_judyl, size_t mem_after_judyl) {
588
if(mem_after_judyl > mem_before_judyl) {
589
__atomic_add_fetch(&ll->stats->size, mem_after_judyl - mem_before_judyl, __ATOMIC_RELAXED);
590
__atomic_add_fetch(&cache->stats.size, mem_after_judyl - mem_before_judyl, __ATOMIC_RELAXED);
@@ -456,88 +604,98 @@ static inline void pgc_stats_index_judy_change(PGC *cache, size_t mem_before_jud
604
}
605
}
606
459
-static void pgc_ll_add(PGC *cache __maybe_unused, struct pgc_linked_list *ll, PGC_PAGE *page, bool having_lock) {
607
+static void pgc_queue_add(PGC *cache __maybe_unused, struct pgc_queue *q, PGC_PAGE *page, bool having_lock) {
608
if(!having_lock)
461
- pgc_ll_lock(cache, ll);
609
+ pgc_queue_lock(cache, q);
610
611
internal_fatal(page_get_status_flags(page) != 0,
612
"DBENGINE CACHE: invalid page flags, the page has %d, but it is should be %d",
613
page_get_status_flags(page),
614
0);
615
468
- if(ll->linked_list_in_sections_judy) {
616
+ if(q->linked_list_in_sections_judy) {
617
+ // HOT and DIRTY pages end up here.
618
+
619
size_t mem_before_judyl, mem_after_judyl;
620
471
- mem_before_judyl = JudyLMemUsed(ll->sections_judy);
472
- Pvoid_t *section_pages_pptr = JudyLIns(&ll->sections_judy, page->section, PJE0);
473
- mem_after_judyl = JudyLMemUsed(ll->sections_judy);
621
+ mem_before_judyl = JudyLMemUsed(q->sections_judy);
622
+ Pvoid_t *section_pages_pptr = JudyLIns(&q->sections_judy, page->section, PJE0);
623
+ mem_after_judyl = JudyLMemUsed(q->sections_judy);
624
625
struct section_pages *sp = *section_pages_pptr;
626
if(!sp) {
627
// sp = callocz(1, sizeof(struct section_pages));
478
- sp = aral_mallocz(pgc_section_pages_aral);
628
+ sp = aral_mallocz(pgc_sections_aral);
629
memset(sp, 0, sizeof(struct section_pages));
630
631
*section_pages_pptr = sp;
632
633
mem_after_judyl += sizeof(struct section_pages);
634
}
485
- pgc_stats_ll_judy_change(cache, ll, mem_before_judyl, mem_after_judyl);
635
+ pgc_stats_queue_judy_change(cache, q, mem_before_judyl, mem_after_judyl);
636
637
sp->entries++;
638
sp->size += page->assumed_size;
639
DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(sp->base, page, link.prev, link.next);
640
641
if((sp->entries % cache->config.max_dirty_pages_per_call) == 0)
492
- ll->version++;
642
+ q->version++;
643
}
644
else {
645
// CLEAN pages end up here.
646
// - New pages created as CLEAN, always have 1 access.
647
// - DIRTY pages made CLEAN, depending on their accesses may be appended (accesses > 0) or prepended (accesses = 0).
648
499
- if(page->accesses || page_flag_check(page, PGC_PAGE_HAS_BEEN_ACCESSED | PGC_PAGE_HAS_NO_DATA_IGNORE_ACCESSES) == PGC_PAGE_HAS_BEEN_ACCESSED) {
500
- DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(ll->base, page, link.prev, link.next);
649
+ // FIXME - is it better for fragmentation to always append?
650
+
651
+// if(page->accesses || page_flag_check(page, PGC_PAGE_HAS_BEEN_ACCESSED | PGC_PAGE_HAS_NO_DATA_IGNORE_ACCESSES) == PGC_PAGE_HAS_BEEN_ACCESSED) {
652
+ DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(q->base, page, link.prev, link.next);
653
page_flag_clear(page, PGC_PAGE_HAS_BEEN_ACCESSED);
502
- }
503
- else
504
- DOUBLE_LINKED_LIST_PREPEND_ITEM_UNSAFE(ll->base, page, link.prev, link.next);
654
+// }
655
+// else
656
+// DOUBLE_LINKED_LIST_PREPEND_ITEM_UNSAFE(q->base, page, link.prev, link.next);
657
506
- ll->version++;
658
+ q->version++;
659
}
660
509
- page_flag_set(page, ll->flags);
661
+ page_flag_set(page, q->flags);
662
663
if(!having_lock)
512
- pgc_ll_unlock(cache, ll);
664
+ pgc_queue_unlock(cache, q);
665
+
666
+ size_t entries = __atomic_add_fetch(&q->stats->entries, 1, __ATOMIC_RELAXED);
667
+ size_t size = __atomic_add_fetch(&q->stats->size, page->assumed_size, __ATOMIC_RELAXED);
668
+ __atomic_add_fetch(&q->stats->added_entries, 1, __ATOMIC_RELAXED);
669
+ __atomic_add_fetch(&q->stats->added_size, page->assumed_size, __ATOMIC_RELAXED);
670
514
- size_t entries = __atomic_add_fetch(&ll->stats->entries, 1, __ATOMIC_RELAXED);
515
- size_t size = __atomic_add_fetch(&ll->stats->size, page->assumed_size, __ATOMIC_RELAXED);
516
- __atomic_add_fetch(&ll->stats->added_entries, 1, __ATOMIC_RELAXED);
517
- __atomic_add_fetch(&ll->stats->added_size, page->assumed_size, __ATOMIC_RELAXED);
671
+ atomic_set_max(&q->stats->max_entries, entries);
672
+ atomic_set_max(&q->stats->max_size, size);
673
519
- atomic_set_max(&ll->stats->max_entries, entries);
520
- atomic_set_max(&ll->stats->max_size, size);
674
+ if(cache->config.stats)
675
+ pgc_size_histogram_add(cache, &q->stats->size_histogram, page);
676
}
677
523
-static void pgc_ll_del(PGC *cache __maybe_unused, struct pgc_linked_list *ll, PGC_PAGE *page, bool having_lock) {
524
- __atomic_sub_fetch(&ll->stats->entries, 1, __ATOMIC_RELAXED);
525
- __atomic_sub_fetch(&ll->stats->size, page->assumed_size, __ATOMIC_RELAXED);
526
- __atomic_add_fetch(&ll->stats->removed_entries, 1, __ATOMIC_RELAXED);
527
- __atomic_add_fetch(&ll->stats->removed_size, page->assumed_size, __ATOMIC_RELAXED);
678
+static void pgc_queue_del(PGC *cache __maybe_unused, struct pgc_queue *q, PGC_PAGE *page, bool having_lock) {
679
+ if(cache->config.stats)
680
+ pgc_size_histogram_del(cache, &q->stats->size_histogram, page);
681
+
682
+ __atomic_sub_fetch(&q->stats->entries, 1, __ATOMIC_RELAXED);
683
+ __atomic_sub_fetch(&q->stats->size, page->assumed_size, __ATOMIC_RELAXED);
684
+ __atomic_add_fetch(&q->stats->removed_entries, 1, __ATOMIC_RELAXED);
685
+ __atomic_add_fetch(&q->stats->removed_size, page->assumed_size, __ATOMIC_RELAXED);
686
687
if(!having_lock)
530
- pgc_ll_lock(cache, ll);
688
+ pgc_queue_lock(cache, q);
689
532
- internal_fatal(page_get_status_flags(page) != ll->flags,
690
+ internal_fatal(page_get_status_flags(page) != q->flags,
691
"DBENGINE CACHE: invalid page flags, the page has %d, but it is should be %d",
692
page_get_status_flags(page),
535
- ll->flags);
693
+ q->flags);
694
537
- page_flag_clear(page, ll->flags);
695
+ page_flag_clear(page, q->flags);
696
539
- if(ll->linked_list_in_sections_judy) {
540
- Pvoid_t *section_pages_pptr = JudyLGet(ll->sections_judy, page->section, PJE0);
697
+ if(q->linked_list_in_sections_judy) {
698
+ Pvoid_t *section_pages_pptr = JudyLGet(q->sections_judy, page->section, PJE0);
699
internal_fatal(!section_pages_pptr, "DBENGINE CACHE: page should be in Judy LL, but it is not");
700
701
struct section_pages *sp = *section_pages_pptr;
@@ -548,26 +706,26 @@ static void pgc_ll_del(PGC *cache __maybe_unused, struct pgc_linked_list *ll, PG
706
if(!sp->base) {
707
size_t mem_before_judyl, mem_after_judyl;
708
551
- mem_before_judyl = JudyLMemUsed(ll->sections_judy);
552
- int rc = JudyLDel(&ll->sections_judy, page->section, PJE0);
553
- mem_after_judyl = JudyLMemUsed(ll->sections_judy);
709
+ mem_before_judyl = JudyLMemUsed(q->sections_judy);
710
+ int rc = JudyLDel(&q->sections_judy, page->section, PJE0);
711
+ mem_after_judyl = JudyLMemUsed(q->sections_judy);
712
713
if(!rc)
714
fatal("DBENGINE CACHE: cannot delete section from Judy LL");
715
716
// freez(sp);
559
- aral_freez(pgc_section_pages_aral, sp);
717
+ aral_freez(pgc_sections_aral, sp);
718
mem_after_judyl -= sizeof(struct section_pages);
561
- pgc_stats_ll_judy_change(cache, ll, mem_before_judyl, mem_after_judyl);
719
+ pgc_stats_queue_judy_change(cache, q, mem_before_judyl, mem_after_judyl);
720
}
721
}
722
else {
565
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ll->base, page, link.prev, link.next);
566
- ll->version++;
723
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(q->base, page, link.prev, link.next);
724
+ q->version++;
725
}
726
727
if(!having_lock)
570
- pgc_ll_unlock(cache, ll);
728
+ pgc_queue_unlock(cache, q);
729
}
730
731
static inline void page_has_been_accessed(PGC *cache, PGC_PAGE *page) {
@@ -577,10 +735,10 @@ static inline void page_has_been_accessed(PGC *cache, PGC_PAGE *page) {
735
__atomic_add_fetch(&page->accesses, 1, __ATOMIC_RELAXED);
736
737
if (flags & PGC_PAGE_CLEAN) {
580
- if(pgc_ll_trylock(cache, &cache->clean)) {
738
+ if(pgc_queue_trylock(cache, &cache->clean)) {
739
DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(cache->clean.base, page, link.prev, link.next);
740
DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(cache->clean.base, page, link.prev, link.next);
583
- pgc_ll_unlock(cache, &cache->clean);
741
+ pgc_queue_unlock(cache, &cache->clean);
742
page_flag_clear(page, PGC_PAGE_HAS_BEEN_ACCESSED);
743
}
744
else
@@ -606,13 +764,13 @@ static inline void page_set_clean(PGC *cache, PGC_PAGE *page, bool having_transi
764
}
765
766
if(flags & PGC_PAGE_HOT)
609
- pgc_ll_del(cache, &cache->hot, page, false);
767
+ pgc_queue_del(cache, &cache->hot, page, false);
768
769
if(flags & PGC_PAGE_DIRTY)
612
- pgc_ll_del(cache, &cache->dirty, page, false);
770
+ pgc_queue_del(cache, &cache->dirty, page, false);
771
772
// first add to linked list, the set the flag (required for move_page_last())
615
- pgc_ll_add(cache, &cache->clean, page, having_clean_lock);
773
+ pgc_queue_add(cache, &cache->clean, page, having_clean_lock);
774
775
if(!having_transition_lock)
776
page_transition_unlock(cache, page);
@@ -622,7 +780,7 @@ static inline void page_set_dirty(PGC *cache, PGC_PAGE *page, bool having_hot_lo
780
if(!having_hot_lock)
781
// to avoid deadlocks, we have to get the hot lock before the page transition
782
// since this is what all_hot_to_dirty() does
625
- pgc_ll_lock(cache, &cache->hot);
783
+ pgc_queue_lock(cache, &cache->hot);
784
785
page_transition_lock(cache, page);
786
@@ -633,7 +791,7 @@ static inline void page_set_dirty(PGC *cache, PGC_PAGE *page, bool having_hot_lo
791
792
if(!having_hot_lock)
793
// we don't need the hot lock anymore
636
- pgc_ll_unlock(cache, &cache->hot);
794
+ pgc_queue_unlock(cache, &cache->hot);
795
796
return;
797
}
@@ -642,17 +800,17 @@ static inline void page_set_dirty(PGC *cache, PGC_PAGE *page, bool having_hot_lo
800
__atomic_add_fetch(&cache->stats.hot2dirty_size, page->assumed_size, __ATOMIC_RELAXED);
801
802
if(likely(flags & PGC_PAGE_HOT))
645
- pgc_ll_del(cache, &cache->hot, page, true);
803
+ pgc_queue_del(cache, &cache->hot, page, true);
804
805
if(!having_hot_lock)
806
// we don't need the hot lock anymore
649
- pgc_ll_unlock(cache, &cache->hot);
807
+ pgc_queue_unlock(cache, &cache->hot);
808
809
if(unlikely(flags & PGC_PAGE_CLEAN))
652
- pgc_ll_del(cache, &cache->clean, page, false);
810
+ pgc_queue_del(cache, &cache->clean, page, false);
811
812
// first add to linked list, the set the flag (required for move_page_last())
655
- pgc_ll_add(cache, &cache->dirty, page, false);
813
+ pgc_queue_add(cache, &cache->dirty, page, false);
814
815
__atomic_sub_fetch(&cache->stats.hot2dirty_entries, 1, __ATOMIC_RELAXED);
816
__atomic_sub_fetch(&cache->stats.hot2dirty_size, page->assumed_size, __ATOMIC_RELAXED);
@@ -671,13 +829,13 @@ static inline void page_set_hot(PGC *cache, PGC_PAGE *page) {
829
}
830
831
if(flags & PGC_PAGE_DIRTY)
674
- pgc_ll_del(cache, &cache->dirty, page, false);
832
+ pgc_queue_del(cache, &cache->dirty, page, false);
833
834
if(flags & PGC_PAGE_CLEAN)
677
- pgc_ll_del(cache, &cache->clean, page, false);
835
+ pgc_queue_del(cache, &cache->clean, page, false);
836
837
// first add to linked list, the set the flag (required for move_page_last())
680
- pgc_ll_add(cache, &cache->hot, page, false);
838
+ pgc_queue_add(cache, &cache->hot, page, false);
839
840
page_transition_unlock(cache, page);
841
}
@@ -722,7 +880,7 @@ static inline bool page_acquire(PGC *cache, PGC_PAGE *page) {
880
} while(!__atomic_compare_exchange_n(&page->refcount, &expected, desired, false, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED));
881
882
if(unlikely(spins > 1))
725
- __atomic_add_fetch(&cache->stats.acquire_spins, spins - 1, __ATOMIC_RELAXED);
883
+ __atomic_add_fetch(&cache->stats.waste_acquire_spins, spins - 1, __ATOMIC_RELAXED);
884
885
if(desired == 1)
886
PGC_REFERENCED_PAGES_PLUS1(cache, page);
@@ -750,7 +908,7 @@ static inline void page_release(PGC *cache, PGC_PAGE *page, bool evict_if_necess
908
} while(!__atomic_compare_exchange_n(&page->refcount, &expected, desired, false, __ATOMIC_RELEASE, __ATOMIC_RELAXED));
909
910
if(unlikely(spins > 1))
753
- __atomic_add_fetch(&cache->stats.release_spins, spins - 1, __ATOMIC_RELAXED);
911
+ __atomic_add_fetch(&cache->stats.waste_release_spins, spins - 1, __ATOMIC_RELAXED);
912
913
if(desired == 0) {
914
PGC_REFERENCED_PAGES_MINUS1(cache, assumed_size);
@@ -795,7 +953,7 @@ static inline bool non_acquired_page_get_for_deletion___while_having_clean_locke
953
}
954
955
if(unlikely(spins > 1))
798
- __atomic_add_fetch(&cache->stats.delete_spins, spins - 1, __ATOMIC_RELAXED);
956
+ __atomic_add_fetch(&cache->stats.waste_delete_spins, spins - 1, __ATOMIC_RELAXED);
957
958
return delete_it;
959
}
@@ -841,7 +999,7 @@ static inline bool acquired_page_get_for_deletion_or_release_it(PGC *cache __may
999
}
1000
1001
if(unlikely(spins > 1))
844
- __atomic_add_fetch(&cache->stats.delete_spins, spins - 1, __ATOMIC_RELAXED);
1002
+ __atomic_add_fetch(&cache->stats.waste_delete_spins, spins - 1, __ATOMIC_RELAXED);
1003
1004
return delete_it;
1005
}
@@ -851,6 +1009,8 @@ static inline bool acquired_page_get_for_deletion_or_release_it(PGC *cache __may
1009
// Indexing
1010
1011
static inline void free_this_page(PGC *cache, PGC_PAGE *page, size_t partition __maybe_unused) {
1012
+ size_t size = page_size_from_assumed_size(cache, page->assumed_size);
1013
+
1014
// call the callback to free the user supplied memory
1015
cache->config.pgc_free_clean_cb(cache, (PGC_ENTRY){
1016
.section = page->section,
@@ -858,12 +1018,14 @@ static inline void free_this_page(PGC *cache, PGC_PAGE *page, size_t partition _
1018
.start_time_s = page->start_time_s,
1019
.end_time_s = __atomic_load_n(&page->end_time_s, __ATOMIC_RELAXED),
1020
.update_every_s = page->update_every_s,
861
- .size = page_size_from_assumed_size(cache, page->assumed_size),
1021
+ .size = size,
1022
.hot = (is_page_hot(page)) ? true : false,
1023
.data = page->data,
1024
.custom_data = (cache->config.additional_bytes_per_page) ? page->custom_data : NULL,
1025
});
1026
1027
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_CB);
1028
+
1029
// update statistics
1030
__atomic_add_fetch(&cache->stats.removed_entries, 1, __ATOMIC_RELAXED);
1031
__atomic_add_fetch(&cache->stats.removed_size, page->assumed_size, __ATOMIC_RELAXED);
@@ -871,12 +1033,16 @@ static inline void free_this_page(PGC *cache, PGC_PAGE *page, size_t partition _
1033
__atomic_sub_fetch(&cache->stats.entries, 1, __ATOMIC_RELAXED);
1034
__atomic_sub_fetch(&cache->stats.size, page->assumed_size, __ATOMIC_RELAXED);
1035
1036
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_ATOMICS2);
1037
+
1038
// free our memory
1039
#ifdef PGC_WITH_ARAL
876
- aral_freez(cache->aral[partition], page);
1040
+ aral_freez(cache->aral, page);
1041
#else
1042
freez(page);
1043
#endif
1044
+
1045
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_ARAL);
1046
}
1047
1048
static void remove_this_page_from_index_unsafe(PGC *cache, PGC_PAGE *page, size_t partition) {
@@ -948,20 +1114,20 @@ static inline bool make_acquired_page_clean_and_evict_or_page_release(PGC *cache
1114
pointer_check(cache, page);
1115
1116
page_transition_lock(cache, page);
951
- pgc_ll_lock(cache, &cache->clean);
1117
+ pgc_queue_lock(cache, &cache->clean);
1118
1119
// make it clean - it does not have any accesses, so it will be prepended
1120
page_set_clean(cache, page, true, true);
1121
1122
if(!acquired_page_get_for_deletion_or_release_it(cache, page)) {
957
- pgc_ll_unlock(cache, &cache->clean);
1123
+ pgc_queue_unlock(cache, &cache->clean);
1124
page_transition_unlock(cache, page);
1125
return false;
1126
}
1127
1128
// remove it from the linked list
963
- pgc_ll_del(cache, &cache->clean, page, true);
964
- pgc_ll_unlock(cache, &cache->clean);
1129
+ pgc_queue_del(cache, &cache->clean, page, true);
1130
+ pgc_queue_unlock(cache, &cache->clean);
1131
page_transition_unlock(cache, page);
1132
1133
remove_and_free_page_not_in_any_queue_and_acquired_for_deletion(cache, page);
@@ -969,7 +1135,7 @@ static inline bool make_acquired_page_clean_and_evict_or_page_release(PGC *cache
1135
return true;
1136
}
1137
972
-// returns true, when there is more work to do
1138
+// returns true, when there is potentially more work to do
1139
static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evict, bool wait, bool all_of_them, evict_filter filter, void *data) {
1140
size_t per1000 = cache_usage_per1000(cache, NULL);
1141
@@ -977,8 +1143,9 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1143
// don't bother - not enough to do anything
1144
return false;
1145
1146
+ bool under_sever_pressure = per1000 >= cache->config.severe_pressure_per1000;
1147
size_t workers_running = __atomic_add_fetch(&cache->stats.workers_evict, 1, __ATOMIC_RELAXED);
981
- if(!wait && !all_of_them && workers_running > cache->config.max_workers_evict_inline && per1000 < cache->config.severe_pressure_per1000) {
1148
+ if(!wait && !all_of_them && workers_running > cache->config.max_workers_evict_inline && !under_sever_pressure) {
1149
__atomic_sub_fetch(&cache->stats.workers_evict, 1, __ATOMIC_RELAXED);
1150
return false;
1151
}
@@ -996,31 +1163,46 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1163
else if(unlikely(max_evict < 2))
1164
max_evict = 2;
1165
1166
+ size_t this_loop_evicted = 0;
1167
size_t total_pages_evicted = 0;
1000
- size_t total_pages_skipped = 0;
1168
+ size_t total_pages_relocated = 0;
1169
bool stopped_before_finishing = false;
1170
size_t spins = 0;
1171
+ size_t max_pages_to_evict = 0;
1172
1173
do {
1005
- if(++spins > 1)
1006
- __atomic_add_fetch(&cache->stats.evict_spins, 1, __ATOMIC_RELAXED);
1007
-
1008
- bool batch;
1174
size_t max_size_to_evict = 0;
1175
if (unlikely(all_of_them)) {
1176
+ // evict them all
1177
max_size_to_evict = SIZE_MAX;
1012
- batch = true;
1178
+ max_pages_to_evict = SIZE_MAX;
1179
+ under_sever_pressure = true;
1180
}
1181
else if(unlikely(wait)) {
1182
+ // evict as many as necessary for the cache to go at the predefined threshold
1183
per1000 = cache_usage_per1000(cache, &max_size_to_evict);
1016
- batch = (wait && per1000 > cache->config.severe_pressure_per1000) ? true : false;
1184
+ if(per1000 >= cache->config.severe_pressure_per1000) {
1185
+ under_sever_pressure = true;
1186
+ max_pages_to_evict = max_pages_to_evict ? max_pages_to_evict * 2 : 4096;
1187
+ // max_pages_to_evict = 1;
1188
+ }
1189
+ else if(per1000 >= cache->config.aggressive_evict_per1000) {
1190
+ under_sever_pressure = false;
1191
+ max_pages_to_evict = max_pages_to_evict ? max_pages_to_evict * 2 : 128;
1192
+ // max_pages_to_evict = 1;
1193
+ }
1194
+ else {
1195
+ under_sever_pressure = false;
1196
+ max_pages_to_evict = 1;
1197
+ }
1198
}
1199
else {
1019
- batch = false;
1200
+ // this is an adder, so evict just 1 page
1201
max_size_to_evict = (cache_above_healthy_limit(cache)) ? 1 : 0;
1202
+ max_pages_to_evict = 1;
1203
}
1204
1023
- if (!max_size_to_evict)
1205
+ if (!max_size_to_evict || !max_pages_to_evict)
1206
break;
1207
1208
// check if we have to stop
@@ -1029,8 +1211,15 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1211
break;
1212
}
1213
1214
+ if(++spins > 1 && !this_loop_evicted)
1215
+ __atomic_add_fetch(&cache->stats.waste_evict_useless_spins, 1, __ATOMIC_RELAXED);
1216
+
1217
+ this_loop_evicted = 0;
1218
+
1219
+ timing_dbengine_evict_init();
1220
+
1221
if(!all_of_them && !wait) {
1033
- if(!pgc_ll_trylock(cache, &cache->clean)) {
1222
+ if(!pgc_queue_trylock(cache, &cache->clean)) {
1223
stopped_before_finishing = true;
1224
goto premature_exit;
1225
}
@@ -1038,11 +1227,14 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1227
// at this point we have the clean lock
1228
}
1229
else
1041
- pgc_ll_lock(cache, &cache->clean);
1230
+ pgc_queue_lock(cache, &cache->clean);
1231
+
1232
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_LOCK);
1233
1234
// find a page to evict
1235
PGC_PAGE *pages_to_evict = NULL;
1236
size_t pages_to_evict_size = 0;
1237
+ size_t pages_to_evict_count = 0;
1238
for(PGC_PAGE *page = cache->clean.base, *next = NULL, *first_page_we_relocated = NULL; page ; page = next) {
1239
next = page->link.next;
1240
@@ -1064,7 +1256,7 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1256
// we can delete this page
1257
1258
// remove it from the clean list
1067
- pgc_ll_del(cache, &cache->clean, page, true);
1259
+ pgc_queue_del(cache, &cache->clean, page, true);
1260
1261
__atomic_add_fetch(&cache->stats.evicting_entries, 1, __ATOMIC_RELAXED);
1262
__atomic_add_fetch(&cache->stats.evicting_size, page->assumed_size, __ATOMIC_RELAXED);
@@ -1072,8 +1264,11 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1264
DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(pages_to_evict, page, link.prev, link.next);
1265
1266
pages_to_evict_size += page->assumed_size;
1267
+ pages_to_evict_count++;
1268
1076
- if(unlikely(all_of_them || (batch && pages_to_evict_size < max_size_to_evict)))
1269
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_SELECT_PAGE);
1270
+
1271
+ if((pages_to_evict_count < max_pages_to_evict && pages_to_evict_size < max_size_to_evict) || all_of_them)
1272
// get more pages
1273
;
1274
else
@@ -1089,14 +1284,20 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1284
DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(cache->clean.base, page, link.prev, link.next);
1285
DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(cache->clean.base, page, link.prev, link.next);
1286
1287
+ total_pages_relocated++;
1288
+
1289
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_RELOCATE_PAGE);
1290
+
1291
// check if we have to stop
1093
- if(unlikely(++total_pages_skipped >= max_skip && !all_of_them)) {
1292
+ if(unlikely(total_pages_relocated >= max_skip && !all_of_them)) {
1293
stopped_before_finishing = true;
1294
break;
1295
}
1296
}
1297
}
1099
- pgc_ll_unlock(cache, &cache->clean);
1298
+ pgc_queue_unlock(cache, &cache->clean);
1299
+
1300
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_SELECT);
1301
1302
if(likely(pages_to_evict)) {
1303
// remove them from the index
@@ -1107,6 +1308,9 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1308
PGC_PAGE *pages_per_partition[cache->config.partitions];
1309
memset(pages_per_partition, 0, sizeof(PGC_PAGE *) * cache->config.partitions);
1310
1311
+ bool partitions_done[cache->config.partitions];
1312
+ memset(partitions_done, 0, sizeof(bool) * cache->config.partitions);
1313
+
1314
// sort them by partition
1315
for (PGC_PAGE *page = pages_to_evict, *next = NULL; page; page = next) {
1316
next = page->link.next;
@@ -1116,18 +1320,37 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1320
DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(pages_per_partition[partition], page, link.prev, link.next);
1321
}
1322
1323
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_SORT);
1324
+
1325
// remove them from the index
1120
- for (size_t partition = 0; partition < cache->config.partitions; partition++) {
1121
- if (!pages_per_partition[partition]) continue;
1326
+ size_t remaining_partitions = cache->config.partitions;
1327
+ size_t last_remaining_partitions = remaining_partitions + 1;
1328
+ while(remaining_partitions) {
1329
+ bool force = remaining_partitions == last_remaining_partitions;
1330
+ last_remaining_partitions = remaining_partitions;
1331
+ remaining_partitions = 0;
1332
+
1333
+ for (size_t partition = 0; partition < cache->config.partitions; partition++) {
1334
+ if (!pages_per_partition[partition] || partitions_done[partition])
1335
+ continue;
1336
1123
- pgc_index_write_lock(cache, partition);
1337
+ if(pgc_index_trywrite_lock(cache, partition, force)) {
1338
+ partitions_done[partition] = true;
1339
1125
- for (PGC_PAGE *page = pages_per_partition[partition]; page; page = page->link.next)
1126
- remove_this_page_from_index_unsafe(cache, page, partition);
1340
+ for (PGC_PAGE *page = pages_per_partition[partition]; page; page = page->link.next)
1341
+ remove_this_page_from_index_unsafe(cache, page, partition);
1342
1128
- pgc_index_write_unlock(cache, partition);
1343
+ pgc_index_write_unlock(cache, partition);
1344
+
1345
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_DEINDEX_PAGE);
1346
+ }
1347
+ else
1348
+ remaining_partitions++;
1349
+ }
1350
}
1351
1352
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_DEINDEX);
1353
+
1354
// free them
1355
for (size_t partition = 0; partition < cache->config.partitions; partition++) {
1356
if (!pages_per_partition[partition]) continue;
@@ -1135,15 +1358,24 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1358
for (PGC_PAGE *page = pages_per_partition[partition], *next = NULL; page; page = next) {
1359
next = page->link.next;
1360
1361
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_LOOP);
1362
+
1363
size_t page_size = page->assumed_size;
1364
free_this_page(cache, page, partition);
1365
1366
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_PAGE);
1367
+
1368
__atomic_sub_fetch(&cache->stats.evicting_entries, 1, __ATOMIC_RELAXED);
1369
__atomic_sub_fetch(&cache->stats.evicting_size, page_size, __ATOMIC_RELAXED);
1370
1371
total_pages_evicted++;
1372
+ this_loop_evicted++;
1373
+
1374
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_ATOMICS);
1375
}
1376
}
1377
+
1378
+ timing_dbengine_evict_report();
1379
}
1380
else {
1381
// just one page to be evicted
@@ -1161,27 +1393,29 @@ static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evic
1393
__atomic_sub_fetch(&cache->stats.evicting_size, page_size, __ATOMIC_RELAXED);
1394
1395
total_pages_evicted++;
1396
+ this_loop_evicted++;
1397
}
1398
}
1399
else
1400
break;
1401
1169
- } while(all_of_them || (total_pages_evicted < max_evict && total_pages_skipped < max_skip));
1402
+ } while(all_of_them || (total_pages_evicted < max_evict && total_pages_relocated < max_skip));
1403
1404
if(all_of_them && !filter) {
1172
- pgc_ll_lock(cache, &cache->clean);
1173
- if(cache->clean.stats->entries) {
1405
+ pgc_queue_lock(cache, &cache->clean);
1406
+ size_t entries = __atomic_load_n(&cache->clean.stats->entries, __ATOMIC_RELAXED);
1407
+ if(entries) {
1408
nd_log_limit_static_global_var(erl, 1, 0);
1409
nd_log_limit(&erl, NDLS_DAEMON, NDLP_NOTICE,
1410
"DBENGINE CACHE: cannot free all clean pages, %zu are still in the clean queue",
1177
- cache->clean.stats->entries);
1411
+ entries);
1412
}
1179
- pgc_ll_unlock(cache, &cache->clean);
1413
+ pgc_queue_unlock(cache, &cache->clean);
1414
}
1415
1416
premature_exit:
1183
- if(unlikely(total_pages_skipped))
1184
- __atomic_add_fetch(&cache->stats.evict_skipped, total_pages_skipped, __ATOMIC_RELAXED);
1417
+ if(unlikely(total_pages_relocated))
1418
+ __atomic_add_fetch(&cache->stats.waste_evict_relocated, total_pages_relocated, __ATOMIC_RELAXED);
1419
1420
__atomic_sub_fetch(&cache->stats.workers_evict, 1, __ATOMIC_RELAXED);
1421
@@ -1197,7 +1431,7 @@ static PGC_PAGE *page_add(PGC *cache, PGC_ENTRY *entry, bool *added) {
1431
size_t partition = pgc_indexing_partition(cache, entry->metric_id);
1432
1433
#ifdef PGC_WITH_ARAL
1200
- PGC_PAGE *allocation = aral_mallocz(cache->aral[partition]);
1434
+ PGC_PAGE *allocation = aral_mallocz(cache->aral);
1435
#endif
1436
PGC_PAGE *page;
1437
size_t spins = 0;
@@ -1210,7 +1444,7 @@ static PGC_PAGE *page_add(PGC *cache, PGC_ENTRY *entry, bool *added) {
1444
1445
do {
1446
if(++spins > 1)
1213
- __atomic_add_fetch(&cache->stats.insert_spins, 1, __ATOMIC_RELAXED);
1447
+ __atomic_add_fetch(&cache->stats.waste_insert_spins, 1, __ATOMIC_RELAXED);
1448
1449
pgc_index_write_lock(cache, partition);
1450
@@ -1300,7 +1534,7 @@ static PGC_PAGE *page_add(PGC *cache, PGC_ENTRY *entry, bool *added) {
1534
if(unlikely(!page)) {
1535
// now that we don't have the lock,
1536
// give it some time for the old page to go away
1303
- tinysleep();
1537
+ yield_the_processor();
1538
}
1539
}
1540
@@ -1308,7 +1542,7 @@ static PGC_PAGE *page_add(PGC *cache, PGC_ENTRY *entry, bool *added) {
1542
1543
#ifdef PGC_WITH_ARAL
1544
if(allocation)
1311
- aral_freez(cache->aral[partition], allocation);
1545
+ aral_freez(cache->aral, allocation);
1546
#endif
1547
1548
__atomic_sub_fetch(&cache->stats.workers_add, 1, __ATOMIC_RELAXED);
@@ -1316,10 +1550,7 @@ static PGC_PAGE *page_add(PGC *cache, PGC_ENTRY *entry, bool *added) {
1550
if(!entry->hot)
1551
evict_on_clean_page_added(cache);
1552
1319
- if((cache->config.options & PGC_OPTIONS_FLUSH_PAGES_INLINE) || flushing_critical(cache)) {
1320
- flush_pages(cache, cache->config.max_flushes_inline, PGC_SECTION_ALL,
1321
- false, false);
1322
- }
1553
+ flush_on_page_add(cache);
1554
1555
return page;
1556
}
@@ -1459,7 +1690,7 @@ cleanup:
1690
}
1691
1692
static void all_hot_pages_to_dirty(PGC *cache, Word_t section) {
1462
- pgc_ll_lock(cache, &cache->hot);
1693
+ pgc_queue_lock(cache, &cache->hot);
1694
1695
bool first = true;
1696
Word_t last_section = (section == PGC_SECTION_ALL) ? 0 : section;
@@ -1483,7 +1714,7 @@ static void all_hot_pages_to_dirty(PGC *cache, Word_t section) {
1714
page = next;
1715
}
1716
}
1486
- pgc_ll_unlock(cache, &cache->hot);
1717
+ pgc_queue_unlock(cache, &cache->hot);
1718
}
1719
1720
// returns true when there is more work to do
@@ -1495,20 +1726,21 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1726
// we have been called from a data collection thread
1727
// let's not waste its time...
1728
1498
- if(!pgc_ll_trylock(cache, &cache->dirty)) {
1729
+ if(!pgc_queue_trylock(cache, &cache->dirty)) {
1730
// we would block, so give up...
1500
- return true;
1731
+ return false;
1732
}
1733
1734
// we got the lock at this point
1735
}
1736
else
1506
- pgc_ll_lock(cache, &cache->dirty);
1737
+ pgc_queue_lock(cache, &cache->dirty);
1738
1739
size_t optimal_flush_size = cache->config.max_dirty_pages_per_call;
1740
size_t dirty_version_at_entry = cache->dirty.version;
1510
- if(!all_of_them && (cache->dirty.stats->entries < optimal_flush_size || cache->dirty.last_version_checked == dirty_version_at_entry)) {
1511
- pgc_ll_unlock(cache, &cache->dirty);
1741
+ size_t entries = __atomic_load_n(&cache->dirty.stats->entries, __ATOMIC_RELAXED);
1742
+ if(!all_of_them && (entries < optimal_flush_size || cache->dirty.last_version_checked == dirty_version_at_entry)) {
1743
+ pgc_queue_unlock(cache, &cache->dirty);
1744
return false;
1745
}
1746
@@ -1523,7 +1755,6 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1755
size_t flushes_so_far = 0;
1756
Pvoid_t *section_pages_pptr;
1757
bool stopped_before_finishing = false;
1526
- size_t spins = 0;
1758
bool first = true;
1759
1760
while (have_dirty_lock && (section_pages_pptr = JudyLFirstThenNext(cache->dirty.sections_judy, &last_section, &first))) {
@@ -1539,9 +1770,6 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1770
break;
1771
}
1772
1542
- if(++spins > 1)
1543
- __atomic_add_fetch(&cache->stats.flush_spins, 1, __ATOMIC_RELAXED);
1544
-
1773
PGC_ENTRY array[optimal_flush_size];
1774
PGC_PAGE *pages[optimal_flush_size];
1775
size_t pages_added = 0, pages_added_size = 0;
@@ -1603,7 +1831,7 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1831
__atomic_add_fetch(&cache->stats.flushing_size, tpg->assumed_size, __ATOMIC_RELAXED);
1832
1833
// remove it from the dirty list
1606
- pgc_ll_del(cache, &cache->dirty, tpg, true);
1834
+ pgc_queue_del(cache, &cache->dirty, tpg, true);
1835
1836
pages_removed_dirty_size += tpg->assumed_size;
1837
pages_removed_dirty++;
@@ -1629,7 +1857,7 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1857
// page ptr may be invalid now
1858
}
1859
1632
- __atomic_add_fetch(&cache->stats.flushes_cancelled, pages_cancelled, __ATOMIC_RELAXED);
1860
+ __atomic_add_fetch(&cache->stats.waste_flushes_cancelled, pages_cancelled, __ATOMIC_RELAXED);
1861
__atomic_add_fetch(&cache->stats.flushes_cancelled_size, pages_cancelled_size, __ATOMIC_RELAXED);
1862
1863
internal_fatal(pages_added != pages_cancelled || pages_added_size != pages_cancelled_size,
@@ -1643,7 +1871,7 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1871
if(cache->config.pgc_save_init_cb)
1872
cache->config.pgc_save_init_cb(cache, last_section);
1873
1646
- pgc_ll_unlock(cache, &cache->dirty);
1874
+ pgc_queue_unlock(cache, &cache->dirty);
1875
have_dirty_lock = false;
1876
1877
// call the callback to save them
@@ -1681,7 +1909,7 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1909
, "DBENGINE CACHE: flushing pages mismatch");
1910
1911
if(!all_of_them && !wait) {
1684
- if(pgc_ll_trylock(cache, &cache->dirty))
1912
+ if(pgc_queue_trylock(cache, &cache->dirty))
1913
have_dirty_lock = true;
1914
1915
else {
@@ -1690,7 +1918,7 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1918
}
1919
}
1920
else {
1693
- pgc_ll_lock(cache, &cache->dirty);
1921
+ pgc_queue_lock(cache, &cache->dirty);
1922
have_dirty_lock = true;
1923
}
1924
}
@@ -1699,7 +1927,7 @@ static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wai
1927
if(!stopped_before_finishing && dirty_version_at_entry > cache->dirty.last_version_checked)
1928
cache->dirty.last_version_checked = dirty_version_at_entry;
1929
1702
- pgc_ll_unlock(cache, &cache->dirty);
1930
+ pgc_queue_unlock(cache, &cache->dirty);
1931
}
1932
1933
__atomic_sub_fetch(&cache->stats.workers_flush, 1, __ATOMIC_RELAXED);
@@ -1711,21 +1939,75 @@ void free_all_unreferenced_clean_pages(PGC *cache) {
1939
evict_pages(cache, 0, 0, true, true);
1940
}
1941
1942
+static void *pgc_evict_thread(void *ptr) {
1943
+ PGC *cache = ptr;
1944
+
1945
+ worker_register("PGCEVICT");
1946
+ worker_register_job_name(0, "signaled");
1947
+ worker_register_job_name(1, "scheduled");
1948
+
1949
+ unsigned job_id = 0;
1950
+
1951
+ while (true) {
1952
+ worker_is_idle();
1953
+ unsigned new_job_id = completion_wait_for_a_job_with_timeout(&cache->evictor.completion, job_id, 50);
1954
+ bool was_signaled = new_job_id > job_id;
1955
+ worker_is_busy(was_signaled ? 1 : 0);
1956
+ job_id = new_job_id;
1957
+
1958
+ if (nd_thread_signaled_to_cancel())
1959
+ return NULL;
1960
+
1961
+ spinlock_lock(&cache->evictor.spinlock);
1962
+
1963
+ size_t at_once = 10;
1964
+ size_t size_to_evict = 0;
1965
+ size_t per1000 = cache_usage_per1000(cache, &size_to_evict);
1966
+ bool was_aggressive = per1000 > cache->config.aggressive_evict_per1000;
1967
+
1968
+ while (size_to_evict && ((--at_once && size_to_evict && per1000 > cache->config.healthy_size_per1000) || (per1000 > cache->config.aggressive_evict_per1000))) {
1969
+ if (nd_thread_signaled_to_cancel()) {
1970
+ spinlock_unlock(&cache->evictor.spinlock);
1971
+ return NULL;
1972
+ }
1973
+
1974
+ evict_pages(cache, 0, 0, true, false);
1975
+
1976
+ if(was_signaled || was_aggressive)
1977
+ mallocz_release_as_much_memory_to_the_system();
1978
+
1979
+ yield_the_processor();
1980
+
1981
+ size_to_evict = 0;
1982
+ per1000 = cache_usage_per1000(cache, &size_to_evict);
1983
+ }
1984
+
1985
+ spinlock_unlock(&cache->evictor.spinlock);
1986
+ }
1987
+
1988
+ worker_unregister();
1989
+ return NULL;
1990
+}
1991
+
1992
// ----------------------------------------------------------------------------
1993
// public API
1994
1995
PGC *pgc_create(const char *name,
1718
- size_t clean_size_bytes, free_clean_page_callback pgc_free_cb,
1996
+ size_t clean_size_bytes,
1997
+ free_clean_page_callback pgc_free_cb,
1998
size_t max_dirty_pages_per_flush,
1999
save_dirty_init_callback pgc_save_init_cb,
2000
save_dirty_page_callback pgc_save_dirty_cb,
1722
- size_t max_pages_per_inline_eviction, size_t max_inline_evictors,
2001
+ size_t max_pages_per_inline_eviction,
2002
+ size_t max_inline_evictors,
2003
size_t max_skip_pages_per_inline_eviction,
2004
size_t max_flushes_inline,
1725
- PGC_OPTIONS options, size_t partitions, size_t additional_bytes_per_page) {
2005
+ PGC_OPTIONS options,
2006
+ size_t partitions,
2007
+ size_t additional_bytes_per_page) {
2008
1727
- if(max_pages_per_inline_eviction < 2)
1728
- max_pages_per_inline_eviction = 2;
2009
+ if(max_pages_per_inline_eviction < 1)
2010
+ max_pages_per_inline_eviction = 1;
2011
2012
if(max_dirty_pages_per_flush < 1)
2013
max_dirty_pages_per_flush = 1;
@@ -1735,77 +2017,97 @@ PGC *pgc_create(const char *name,
2017
2018
PGC *cache = callocz(1, sizeof(PGC));
2019
strncpyz(cache->config.name, name, PGC_NAME_MAX);
2020
+
2021
cache->config.options = options;
1739
- cache->config.clean_size = (clean_size_bytes < 1 * 1024 * 1024) ? 1 * 1024 * 1024 : clean_size_bytes;
1740
- cache->config.pgc_free_clean_cb = pgc_free_cb;
1741
- cache->config.max_dirty_pages_per_call = max_dirty_pages_per_flush;
1742
- cache->config.pgc_save_init_cb = pgc_save_init_cb;
1743
- cache->config.pgc_save_dirty_cb = pgc_save_dirty_cb;
2022
+ cache->config.additional_bytes_per_page = additional_bytes_per_page;
2023
+ cache->config.stats = telemetry_enabled;
2024
+
2025
+ // flushing
2026
+ cache->config.max_flushes_inline = (max_flushes_inline == 0) ? 2 : max_flushes_inline;
2027
+ cache->config.max_dirty_pages_per_call = max_dirty_pages_per_flush;
2028
+ cache->config.pgc_save_init_cb = pgc_save_init_cb;
2029
+ cache->config.pgc_save_dirty_cb = pgc_save_dirty_cb;
2030
+
2031
+ // eviction strategy
2032
+ cache->config.clean_size = (clean_size_bytes < 1 * 1024 * 1024) ? 1 * 1024 * 1024 : clean_size_bytes;
2033
+ cache->config.pgc_free_clean_cb = pgc_free_cb;
2034
+ cache->config.max_workers_evict_inline = max_inline_evictors;
2035
cache->config.max_pages_per_inline_eviction = max_pages_per_inline_eviction;
2036
cache->config.max_skip_pages_per_inline_eviction = (max_skip_pages_per_inline_eviction < 2) ? 2 : max_skip_pages_per_inline_eviction;
1746
- cache->config.max_flushes_inline = (max_flushes_inline < 1) ? 1 : max_flushes_inline;
1747
- cache->config.partitions = partitions < 1 ? (size_t)get_netdata_cpus() : partitions;
1748
- cache->config.additional_bytes_per_page = additional_bytes_per_page;
2037
+ cache->config.severe_pressure_per1000 = 1010; // INLINE: use releasers to evict pages (up to max_pages_per_inline_eviction)
2038
+ cache->config.aggressive_evict_per1000 = 990; // INLINE: use adders to evict page (up to max_pages_per_inline_eviction)
2039
+ cache->config.healthy_size_per1000 = 980; // signal the eviction thread to evict immediately
2040
+ cache->config.evict_low_threshold_per1000 = 970; // when evicting, bring the size down to this threshold
2041
1750
- cache->config.max_workers_evict_inline = max_inline_evictors;
1751
- cache->config.severe_pressure_per1000 = 1010;
1752
- cache->config.aggressive_evict_per1000 = 990;
1753
- cache->config.healthy_size_per1000 = 980;
1754
- cache->config.evict_low_threshold_per1000 = 970;
2042
+ // use all ram and protection from out of memory
2043
+ cache->config.use_all_ram = dbengine_use_all_ram_for_caches;
2044
+ cache->config.out_of_memory_protection_bytes = dbengine_out_of_memory_protection;
2045
1756
- cache->index = callocz(cache->config.partitions, sizeof(struct pgc_index));
2046
+ // partitions
2047
+ cache->config.partitions = partitions == 0 ? 1ULL + get_netdata_cpus() / 2 : partitions;
2048
+ cache->index = callocz(cache->config.partitions, sizeof(struct pgc_index));
2049
+
2050
+ pgc_section_pages_static_aral_init();
2051
2052
for(size_t part = 0; part < cache->config.partitions ; part++)
2053
rw_spinlock_init(&cache->index[part].rw_spinlock);
2054
2055
+#ifdef PGC_WITH_ARAL
2056
+ if(cache->config.additional_bytes_per_page) {
2057
+ char buf[100];
2058
+ snprintfz(buf, sizeof(buf), "%s", name);
2059
+ cache->aral = aral_create(
2060
+ buf,
2061
+ sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page,
2062
+ 0,
2063
+ 16364,
2064
+ &aral_statistics_for_pgc,
2065
+ NULL, NULL, false, false);
2066
+ }
2067
+ else
2068
+ cache->aral = pgc_pages_aral;
2069
+
2070
+ telemetry_aral_register(cache->aral, "pgc");
2071
+#endif
2072
+
2073
+
2074
spinlock_init(&cache->hot.spinlock);
2075
spinlock_init(&cache->dirty.spinlock);
2076
spinlock_init(&cache->clean.spinlock);
2077
2078
cache->hot.flags = PGC_PAGE_HOT;
2079
cache->hot.linked_list_in_sections_judy = true;
1767
- cache->hot.stats = &cache->stats.queues.hot;
2080
+ cache->hot.stats = &cache->stats.queues[PGC_QUEUE_HOT];
2081
2082
cache->dirty.flags = PGC_PAGE_DIRTY;
2083
cache->dirty.linked_list_in_sections_judy = true;
1771
- cache->dirty.stats = &cache->stats.queues.dirty;
2084
+ cache->dirty.stats = &cache->stats.queues[PGC_QUEUE_DIRTY];
2085
2086
cache->clean.flags = PGC_PAGE_CLEAN;
2087
cache->clean.linked_list_in_sections_judy = false;
1775
- cache->clean.stats = &cache->stats.queues.clean;
2088
+ cache->clean.stats = &cache->stats.queues[PGC_QUEUE_CLEAN];
2089
1777
- pgc_section_pages_static_aral_init();
2090
+ pointer_index_init(cache);
2091
+ pgc_size_histogram_init(&cache->hot.stats->size_histogram);
2092
+ pgc_size_histogram_init(&cache->dirty.stats->size_histogram);
2093
+ pgc_size_histogram_init(&cache->clean.stats->size_histogram);
2094
1779
-#ifdef PGC_WITH_ARAL
1780
- cache->aral = callocz(cache->config.partitions, sizeof(ARAL *));
1781
- for(size_t part = 0; part < cache->config.partitions ; part++) {
1782
- char buf[100 +1];
1783
- snprintfz(buf, sizeof(buf) - 1, "%s[%zu]", name, part);
1784
- cache->aral[part] = aral_create(
1785
- buf,
1786
- sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page,
1787
- 0,
1788
- 16384,
1789
- aral_get_statistics(pgc_section_pages_aral),
1790
- NULL, NULL, false, false);
2095
+ // last create the eviction thread
2096
+ {
2097
+ spinlock_init(&cache->evictor.spinlock);
2098
+ completion_init(&cache->evictor.completion);
2099
+ cache->evictor.thread = nd_thread_create(name, NETDATA_THREAD_OPTION_JOINABLE, pgc_evict_thread, cache);
2100
}
1792
-#endif
1793
-
1794
- pointer_index_init(cache);
2101
2102
return cache;
2103
}
2104
1799
-struct aral_statistics *pgc_aral_statistics(void) {
1800
- return aral_get_statistics(pgc_section_pages_aral);
1801
-}
1802
-
2105
size_t pgc_aral_structures(void) {
1804
- return aral_structures(pgc_section_pages_aral);
2106
+ return aral_structures(pgc_pages_aral);
2107
}
2108
2109
size_t pgc_aral_overhead(void) {
1808
- return aral_overhead(pgc_section_pages_aral);
2110
+ return aral_overhead(pgc_pages_aral);
2111
}
2112
2113
void pgc_flush_all_hot_and_dirty_pages(PGC *cache, Word_t section) {
@@ -1825,6 +2127,12 @@ void pgc_destroy(PGC *cache) {
2127
// free all unreferenced clean pages
2128
free_all_unreferenced_clean_pages(cache);
2129
2130
+ // stop the eviction thread
2131
+ nd_thread_signal_cancel(cache->evictor.thread);
2132
+ completion_mark_complete_a_job(&cache->evictor.completion);
2133
+ nd_thread_join(cache->evictor.thread);
2134
+ completion_destroy(&cache->evictor.completion);
2135
+
2136
if(PGC_REFERENCED_PAGES(cache))
2137
netdata_log_error("DBENGINE CACHE: there are %zu referenced cache pages - leaving the cache allocated", PGC_REFERENCED_PAGES(cache));
2138
else {
@@ -1834,10 +2142,8 @@ void pgc_destroy(PGC *cache) {
2142
// netdata_rwlock_destroy(&cache->index[part].rw_spinlock);
2143
2144
#ifdef PGC_WITH_ARAL
1837
- for(size_t part = 0; part < cache->config.partitions ; part++)
1838
- aral_destroy(cache->aral[part]);
1839
-
1840
- freez(cache->aral);
2145
+ if(cache->config.additional_bytes_per_page)
2146
+ aral_destroy(cache->aral);
2147
#endif
2148
freez(cache->index);
2149
freez(cache);
@@ -1878,8 +2184,8 @@ void pgc_page_hot_to_dirty_and_release(PGC *cache, PGC_PAGE *page, bool never_fl
2184
__atomic_sub_fetch(&cache->stats.workers_hot2dirty, 1, __ATOMIC_RELAXED);
2185
2186
// flush, if we have to
1881
- if(!never_flush && ((cache->config.options & PGC_OPTIONS_FLUSH_PAGES_INLINE) || flushing_critical(cache)))
1882
- flush_pages(cache, cache->config.max_flushes_inline, PGC_SECTION_ALL, false, false);
2187
+ if(!never_flush)
2188
+ flush_on_page_hot_release(cache);
2189
}
2190
2191
bool pgc_page_to_clean_evict_or_release(PGC *cache, PGC_PAGE *page) {
@@ -1981,12 +2287,18 @@ void pgc_reset_hot_max(PGC *cache) {
2287
2288
void pgc_set_dynamic_target_cache_size_callback(PGC *cache, dynamic_target_cache_size_callback callback) {
2289
cache->config.dynamic_target_size_cb = callback;
2290
+ cache->config.out_of_memory_protection_bytes = 0;
2291
+ cache->config.use_all_ram = false;
2292
2293
size_t size_to_evict = 0;
2294
cache_usage_per1000(cache, &size_to_evict);
2295
evict_pages(cache, 0, 0, true, false);
2296
}
2297
2298
+void pgc_set_nominal_page_size_callback(PGC *cache, nominal_page_size_callback callback) {
2299
+ cache->config.nominal_page_size_cb = callback;
2300
+}
2301
+
2302
size_t pgc_get_current_cache_size(PGC *cache) {
2303
cache_usage_per1000(cache, NULL);
2304
return __atomic_load_n(&cache->stats.current_cache_size, __ATOMIC_RELAXED);
@@ -2005,12 +2317,11 @@ bool pgc_evict_pages(PGC *cache, size_t max_skip, size_t max_evict) {
2317
true, false);
2318
}
2319
2008
-bool pgc_flush_pages(PGC *cache, size_t max_flushes) {
2009
- bool under_pressure = flushing_critical(cache);
2010
- return flush_pages(cache, under_pressure ? 0 : max_flushes, PGC_SECTION_ALL, true, false);
2320
+bool pgc_flush_pages(PGC *cache) {
2321
+ return flush_pages(cache, 0, PGC_SECTION_ALL, true, false);
2322
}
2323
2013
-void pgc_page_hot_set_end_time_s(PGC *cache __maybe_unused, PGC_PAGE *page, time_t end_time_s) {
2324
+void pgc_page_hot_set_end_time_s(PGC *cache __maybe_unused, PGC_PAGE *page, time_t end_time_s, size_t additional_bytes) {
2325
internal_fatal(!is_page_hot(page),
2326
"DBENGINE CACHE: end_time_s update on non-hot page");
2327
@@ -2019,6 +2330,42 @@ void pgc_page_hot_set_end_time_s(PGC *cache __maybe_unused, PGC_PAGE *page, time
2330
2331
__atomic_store_n(&page->end_time_s, end_time_s, __ATOMIC_RELAXED);
2332
2333
+ if(additional_bytes) {
2334
+ page_transition_lock(cache, page);
2335
+
2336
+ struct pgc_queue_statistics *queue_stats = NULL;
2337
+ if(page->flags & PGC_PAGE_HOT)
2338
+ queue_stats = cache->hot.stats;
2339
+ else if(page->flags & PGC_PAGE_DIRTY)
2340
+ queue_stats = cache->dirty.stats;
2341
+ else if(page->flags & PGC_PAGE_CLEAN)
2342
+ queue_stats = cache->clean.stats;
2343
+
2344
+ if(queue_stats && cache->config.stats)
2345
+ pgc_size_histogram_del(cache, &queue_stats->size_histogram, page);
2346
+
2347
+ size_t old_assumed_size = page->assumed_size;
2348
+
2349
+ size_t old_size = page_size_from_assumed_size(cache, old_assumed_size);
2350
+ size_t size = old_size + additional_bytes;
2351
+ page->assumed_size = page_assumed_size(cache, size);
2352
+
2353
+ size_t delta = page->assumed_size - old_assumed_size;
2354
+ __atomic_add_fetch(&cache->stats.size, delta, __ATOMIC_RELAXED);
2355
+ __atomic_add_fetch(&cache->stats.added_size, delta, __ATOMIC_RELAXED);
2356
+ __atomic_add_fetch(&cache->stats.referenced_size, delta, __ATOMIC_RELAXED);
2357
+
2358
+ if(queue_stats) {
2359
+ __atomic_add_fetch(&queue_stats->size, delta, __ATOMIC_RELAXED);
2360
+ __atomic_add_fetch(&queue_stats->added_size, delta, __ATOMIC_RELAXED);
2361
+
2362
+ if(cache->config.stats)
2363
+ pgc_size_histogram_add(cache, &queue_stats->size_histogram, page);
2364
+ }
2365
+
2366
+ page_transition_unlock(cache, page);
2367
+ }
2368
+
2369
#ifdef PGC_COUNT_POINTS_COLLECTED
2370
__atomic_add_fetch(&cache->stats.points_collected, 1, __ATOMIC_RELAXED);
2371
#endif
@@ -2050,7 +2397,7 @@ PGC_PAGE *pgc_page_get_and_acquire(PGC *cache, Word_t section, Word_t metric_id,
2397
if(page || !retry)
2398
break;
2399
2053
- tinysleep();
2400
+ yield_the_processor();
2401
}
2402
2403
if(page) {
@@ -2085,7 +2432,7 @@ void pgc_open_cache_to_journal_v2(PGC *cache, Word_t section, unsigned datafile_
2432
__atomic_add_fetch(&rrdeng_cache_efficiency_stats.journal_v2_indexing_started, 1, __ATOMIC_RELAXED);
2433
__atomic_add_fetch(&cache->stats.workers_jv2_flush, 1, __ATOMIC_RELAXED);
2434
2088
- pgc_ll_lock(cache, &cache->hot);
2435
+ pgc_queue_lock(cache, &cache->hot);
2436
2437
Pvoid_t JudyL_metrics = NULL;
2438
Pvoid_t JudyL_extents_pos = NULL;
@@ -2098,14 +2445,14 @@ void pgc_open_cache_to_journal_v2(PGC *cache, Word_t section, unsigned datafile_
2445
2446
Pvoid_t *section_pages_pptr = JudyLGet(cache->hot.sections_judy, section, PJE0);
2447
if(!section_pages_pptr) {
2101
- pgc_ll_unlock(cache, &cache->hot);
2448
+ pgc_queue_unlock(cache, &cache->hot);
2449
return;
2450
}
2451
2452
struct section_pages *sp = *section_pages_pptr;
2453
if(!spinlock_trylock(&sp->migration_to_v2_spinlock)) {
2454
netdata_log_info("DBENGINE: migration to journal v2 for datafile %u is postponed, another jv2 indexer is already running for this section", datafile_fileno);
2108
- pgc_ll_unlock(cache, &cache->hot);
2455
+ pgc_queue_unlock(cache, &cache->hot);
2456
return;
2457
}
2458
@@ -2134,7 +2481,7 @@ void pgc_open_cache_to_journal_v2(PGC *cache, Word_t section, unsigned datafile_
2481
2482
page_flag_set(page, PGC_PAGE_IS_BEING_MIGRATED_TO_V2);
2483
2137
- pgc_ll_unlock(cache, &cache->hot);
2484
+ pgc_queue_unlock(cache, &cache->hot);
2485
2486
// update the extents JudyL
2487
@@ -2214,11 +2561,11 @@ void pgc_open_cache_to_journal_v2(PGC *cache, Word_t section, unsigned datafile_
2561
page_release(cache, page, false);
2562
}
2563
2217
- pgc_ll_lock(cache, &cache->hot);
2564
+ pgc_queue_lock(cache, &cache->hot);
2565
}
2566
2567
spinlock_unlock(&sp->migration_to_v2_spinlock);
2221
- pgc_ll_unlock(cache, &cache->hot);
2568
+ pgc_queue_unlock(cache, &cache->hot);
2569
2570
// callback
2571
cb(section, datafile_fileno, type, JudyL_metrics, JudyL_extents_pos, count_of_unique_extents, count_of_unique_metrics, count_of_unique_pages, data);
@@ -2278,10 +2625,10 @@ void pgc_open_evict_clean_pages_of_datafile(PGC *cache, struct rrdengine_datafil
2625
size_t pgc_count_clean_pages_having_data_ptr(PGC *cache, Word_t section, void *ptr) {
2626
size_t found = 0;
2627
2281
- pgc_ll_lock(cache, &cache->clean);
2628
+ pgc_queue_lock(cache, &cache->clean);
2629
for(PGC_PAGE *page = cache->clean.base; page ;page = page->link.next)
2630
found += (page->data == ptr && page->section == section) ? 1 : 0;
2284
- pgc_ll_unlock(cache, &cache->clean);
2631
+ pgc_queue_unlock(cache, &cache->clean);
2632
2633
return found;
2634
}
@@ -2289,14 +2636,14 @@ size_t pgc_count_clean_pages_having_data_ptr(PGC *cache, Word_t section, void *p
2636
size_t pgc_count_hot_pages_having_data_ptr(PGC *cache, Word_t section, void *ptr) {
2637
size_t found = 0;
2638
2292
- pgc_ll_lock(cache, &cache->hot);
2639
+ pgc_queue_lock(cache, &cache->hot);
2640
Pvoid_t *section_pages_pptr = JudyLGet(cache->hot.sections_judy, section, PJE0);
2641
if(section_pages_pptr) {
2642
struct section_pages *sp = *section_pages_pptr;
2643
for(PGC_PAGE *page = sp->base; page ;page = page->link.next)
2644
found += (page->data == ptr) ? 1 : 0;
2645
}
2299
- pgc_ll_unlock(cache, &cache->hot);
2646
+ pgc_queue_unlock(cache, &cache->hot);
2647
2648
return found;
2649
}
@@ -2726,7 +3073,7 @@ int pgc_unittest(void) {
3073
.hot = true,
3074
}, NULL);
3075
2729
- pgc_page_hot_set_end_time_s(cache, page2, 2001);
3076
+ pgc_page_hot_set_end_time_s(cache, page2, 2001, 0);
3077
pgc_page_hot_to_dirty_and_release(cache, page2, false);
3078
3079
PGC_PAGE *page3 = pgc_page_add_and_acquire(cache, (PGC_ENTRY){
@@ -2739,7 +3086,7 @@ int pgc_unittest(void) {
3086
.hot = true,
3087
}, NULL);
3088
2742
- pgc_page_hot_set_end_time_s(cache, page3, 2001);
3089
+ pgc_page_hot_set_end_time_s(cache, page3, 2001, 0);
3090
pgc_page_hot_to_dirty_and_release(cache, page3, false);
3091
3092
pgc_destroy(cache);
src/database/engine/cache.h
+120
-112
@@ -14,12 +14,12 @@ typedef struct pgc_page PGC_PAGE;
14
15
typedef enum __attribute__ ((__packed__)) {
16
PGC_OPTIONS_NONE = 0,
17
- PGC_OPTIONS_EVICT_PAGES_INLINE = (1 << 0),
18
- PGC_OPTIONS_FLUSH_PAGES_INLINE = (1 << 1),
19
- PGC_OPTIONS_AUTOSCALE = (1 << 2),
17
+ PGC_OPTIONS_EVICT_PAGES_NO_INLINE = (1 << 0),
18
+ PGC_OPTIONS_FLUSH_PAGES_NO_INLINE = (1 << 1),
19
+ PGC_OPTIONS_AUTOSCALE = (1 << 2),
20
} PGC_OPTIONS;
21
22
-#define PGC_OPTIONS_DEFAULT (PGC_OPTIONS_EVICT_PAGES_INLINE | PGC_OPTIONS_FLUSH_PAGES_INLINE | PGC_OPTIONS_AUTOSCALE)
22
+#define PGC_OPTIONS_DEFAULT (PGC_OPTIONS_EVICT_PAGES_NO_INLINE | PGC_OPTIONS_AUTOSCALE)
23
24
typedef struct pgc_entry {
25
Word_t section; // the section this belongs to
@@ -33,136 +33,137 @@ typedef struct pgc_entry {
33
uint8_t *custom_data;
34
} PGC_ENTRY;
35
36
-#define PGC_CACHE_LINE_PADDING(x) uint8_t padding##x[64]
37
-
38
-struct pgc_queue_statistics {
39
- size_t entries;
40
- size_t size;
36
+struct pgc_size_histogram_entry {
37
+ size_t upto;
38
+ size_t count;
39
+};
40
42
- PGC_CACHE_LINE_PADDING(1);
41
+#define PGC_SIZE_HISTOGRAM_ENTRIES 15
42
+#define PGC_QUEUE_HOT 0
43
+#define PGC_QUEUE_DIRTY 1
44
+#define PGC_QUEUE_CLEAN 2
45
44
- size_t max_entries;
45
- size_t max_size;
46
+struct pgc_size_histogram {
47
+ struct pgc_size_histogram_entry array[PGC_SIZE_HISTOGRAM_ENTRIES];
48
+};
49
47
- PGC_CACHE_LINE_PADDING(2);
50
+struct pgc_queue_statistics {
51
+ struct pgc_size_histogram size_histogram;
52
49
- size_t added_entries;
50
- size_t added_size;
53
+ alignas(64) size_t entries;
54
+ alignas(64) size_t size;
55
52
- PGC_CACHE_LINE_PADDING(3);
56
+ alignas(64) size_t max_entries;
57
+ alignas(64) size_t max_size;
58
54
- size_t removed_entries;
55
- size_t removed_size;
59
+ alignas(64) size_t added_entries;
60
+ alignas(64) size_t added_size;
61
57
- PGC_CACHE_LINE_PADDING(4);
62
+ alignas(64) size_t removed_entries;
63
+ alignas(64) size_t removed_size;
64
};
65
66
struct pgc_statistics {
61
- size_t wanted_cache_size;
62
- size_t current_cache_size;
67
+ alignas(64) size_t wanted_cache_size;
68
+ alignas(64) size_t current_cache_size;
69
64
- PGC_CACHE_LINE_PADDING(1);
70
+ // ----------------------------------------------------------------------------------------------------------------
71
+ // volume
72
66
- size_t added_entries;
67
- size_t added_size;
73
+ alignas(64) size_t entries; // all the entries (includes clean, dirty, hot)
74
+ alignas(64) size_t size; // all the entries (includes clean, dirty, hot)
75
69
- PGC_CACHE_LINE_PADDING(2);
76
+ alignas(64) size_t referenced_entries; // all the entries currently referenced
77
+ alignas(64) size_t referenced_size; // all the entries currently referenced
78
71
- size_t removed_entries;
72
- size_t removed_size;
79
+ alignas(64) size_t added_entries;
80
+ alignas(64) size_t added_size;
81
74
- PGC_CACHE_LINE_PADDING(3);
82
+ alignas(64) size_t removed_entries;
83
+ alignas(64) size_t removed_size;
84
76
- size_t entries; // all the entries (includes clean, dirty, hot)
77
- size_t size; // all the entries (includes clean, dirty, hot)
85
+#ifdef PGC_COUNT_POINTS_COLLECTED
86
+ alignas(64) size_t points_collected;
87
+#endif
88
79
- size_t evicting_entries;
80
- size_t evicting_size;
89
+ // ----------------------------------------------------------------------------------------------------------------
90
+ // migrations
91
82
- size_t flushing_entries;
83
- size_t flushing_size;
92
+ alignas(64) size_t evicting_entries;
93
+ alignas(64) size_t evicting_size;
94
85
- size_t hot2dirty_entries;
86
- size_t hot2dirty_size;
95
+ alignas(64) size_t flushing_entries;
96
+ alignas(64) size_t flushing_size;
97
88
- PGC_CACHE_LINE_PADDING(4);
98
+ alignas(64) size_t hot2dirty_entries;
99
+ alignas(64) size_t hot2dirty_size;
100
90
- size_t acquires;
91
- PGC_CACHE_LINE_PADDING(4a);
92
- size_t releases;
93
- PGC_CACHE_LINE_PADDING(4b);
94
- size_t acquires_for_deletion;
95
- PGC_CACHE_LINE_PADDING(4c);
101
+ alignas(64) size_t hot_empty_pages_evicted_immediately;
102
+ alignas(64) size_t hot_empty_pages_evicted_later;
103
97
- size_t referenced_entries; // all the entries currently referenced
98
- size_t referenced_size; // all the entries currently referenced
104
+ // ----------------------------------------------------------------------------------------------------------------
105
+ // workload
106
100
- PGC_CACHE_LINE_PADDING(5);
107
+ alignas(64) size_t acquires;
108
+ alignas(64) size_t releases;
109
102
- size_t searches_exact;
103
- size_t searches_exact_hits;
104
- size_t searches_exact_misses;
110
+ alignas(64) size_t acquires_for_deletion;
111
106
- PGC_CACHE_LINE_PADDING(6);
112
+ alignas(64) size_t searches_exact;
113
+ alignas(64) size_t searches_exact_hits;
114
+ alignas(64) size_t searches_exact_misses;
115
108
- size_t searches_closest;
109
- size_t searches_closest_hits;
110
- size_t searches_closest_misses;
116
+ alignas(64) size_t searches_closest;
117
+ alignas(64) size_t searches_closest_hits;
118
+ alignas(64) size_t searches_closest_misses;
119
112
- PGC_CACHE_LINE_PADDING(7);
120
+ alignas(64) size_t flushes_completed;
121
+ alignas(64) size_t flushes_completed_size;
122
+ alignas(64) size_t flushes_cancelled_size;
123
114
- size_t flushes_completed;
115
- size_t flushes_completed_size;
116
- size_t flushes_cancelled;
117
- size_t flushes_cancelled_size;
124
+ // ----------------------------------------------------------------------------------------------------------------
125
+ // critical events
126
119
-#ifdef PGC_COUNT_POINTS_COLLECTED
120
- PGC_CACHE_LINE_PADDING(8);
121
- size_t points_collected;
122
-#endif
127
+ alignas(64) size_t events_cache_under_severe_pressure;
128
+ alignas(64) size_t events_cache_needs_space_aggressively;
129
+ alignas(64) size_t events_flush_critical;
130
124
- PGC_CACHE_LINE_PADDING(9);
125
-
126
- size_t insert_spins;
127
- size_t evict_spins;
128
- size_t release_spins;
129
- size_t acquire_spins;
130
- size_t delete_spins;
131
- size_t flush_spins;
132
-
133
- PGC_CACHE_LINE_PADDING(10);
134
-
135
- size_t workers_search;
136
- size_t workers_add;
137
- size_t workers_evict;
138
- size_t workers_flush;
139
- size_t workers_jv2_flush;
140
- size_t workers_hot2dirty;
141
-
142
- size_t evict_skipped;
143
- size_t hot_empty_pages_evicted_immediately;
144
- size_t hot_empty_pages_evicted_later;
145
-
146
- PGC_CACHE_LINE_PADDING(11);
147
-
148
- // events
149
- size_t events_cache_under_severe_pressure;
150
- size_t events_cache_needs_space_aggressively;
151
- size_t events_flush_critical;
152
-
153
- PGC_CACHE_LINE_PADDING(12);
154
-
155
- struct {
156
- PGC_CACHE_LINE_PADDING(0);
157
- struct pgc_queue_statistics hot;
158
- PGC_CACHE_LINE_PADDING(1);
159
- struct pgc_queue_statistics dirty;
160
- PGC_CACHE_LINE_PADDING(2);
161
- struct pgc_queue_statistics clean;
162
- PGC_CACHE_LINE_PADDING(3);
163
- } queues;
164
-};
131
+ // ----------------------------------------------------------------------------------------------------------------
132
+ // worker threads
133
+
134
+ alignas(64) size_t workers_search;
135
+ alignas(64) size_t workers_add;
136
+ alignas(64) size_t workers_evict;
137
+ alignas(64) size_t workers_flush;
138
+ alignas(64) size_t workers_jv2_flush;
139
+ alignas(64) size_t workers_hot2dirty;
140
+
141
+ // ----------------------------------------------------------------------------------------------------------------
142
+ // waste events
143
+
144
+ // waste events - spins
145
+ alignas(64) size_t waste_insert_spins;
146
+ alignas(64) size_t waste_evict_useless_spins;
147
+ alignas(64) size_t waste_release_spins;
148
+ alignas(64) size_t waste_acquire_spins;
149
+ alignas(64) size_t waste_delete_spins;
150
151
+ // waste events - eviction
152
+ alignas(64) size_t waste_evict_relocated;
153
+ alignas(64) size_t waste_evict_thread_signals;
154
+ alignas(64) size_t waste_evictions_inline_on_add;
155
+ alignas(64) size_t waste_evictions_inline_on_release;
156
+
157
+ // waste events - flushing
158
+ alignas(64) size_t waste_flush_on_add;
159
+ alignas(64) size_t waste_flush_on_release;
160
+ alignas(64) size_t waste_flushes_cancelled;
161
+
162
+ // ----------------------------------------------------------------------------------------------------------------
163
+ // per queue statistics
164
+
165
+ struct pgc_queue_statistics queues[3];
166
+};
167
168
typedef void (*free_clean_page_callback)(PGC *cache, PGC_ENTRY entry);
169
typedef void (*save_dirty_page_callback)(PGC *cache, PGC_ENTRY *entries_array, PGC_PAGE **pages_array, size_t entries);
@@ -225,7 +226,7 @@ size_t pgc_get_current_cache_size(PGC *cache);
226
size_t pgc_get_wanted_cache_size(PGC *cache);
227
228
// resetting the end time of a hot page
228
-void pgc_page_hot_set_end_time_s(PGC *cache, PGC_PAGE *page, time_t end_time_s);
229
+void pgc_page_hot_set_end_time_s(PGC *cache, PGC_PAGE *page, time_t end_time_s, size_t additional_bytes);
230
bool pgc_page_to_clean_evict_or_release(PGC *cache, PGC_PAGE *page);
231
232
typedef void (*migrate_to_v2_callback)(Word_t section, unsigned datafile_fileno, uint8_t type, Pvoid_t JudyL_metrics, Pvoid_t JudyL_extents_pos, size_t count_of_unique_extents, size_t count_of_unique_metrics, size_t count_of_unique_pages, void *data);
@@ -237,26 +238,33 @@ size_t pgc_count_hot_pages_having_data_ptr(PGC *cache, Word_t section, void *ptr
238
typedef size_t (*dynamic_target_cache_size_callback)(void);
239
void pgc_set_dynamic_target_cache_size_callback(PGC *cache, dynamic_target_cache_size_callback callback);
240
241
+typedef size_t (*nominal_page_size_callback)(void *);
242
+void pgc_set_nominal_page_size_callback(PGC *cache, nominal_page_size_callback callback);
243
+
244
// return true when there is more work to do
245
bool pgc_evict_pages(PGC *cache, size_t max_skip, size_t max_evict);
242
-bool pgc_flush_pages(PGC *cache, size_t max_flushes);
246
+bool pgc_flush_pages(PGC *cache);
247
248
struct pgc_statistics pgc_get_statistics(PGC *cache);
249
size_t pgc_hot_and_dirty_entries(PGC *cache);
250
247
-struct aral_statistics *pgc_aral_statistics(void);
251
size_t pgc_aral_structures(void);
252
size_t pgc_aral_overhead(void);
253
254
static inline size_t indexing_partition(Word_t ptr, Word_t modulo) __attribute__((const));
255
static inline size_t indexing_partition(Word_t ptr, Word_t modulo) {
253
-#ifdef ENV64BIT
254
- uint64_t hash = murmur64(ptr);
255
- return hash % modulo;
256
-#else
257
- uint32_t hash = murmur32(ptr);
256
+ XXH64_hash_t hash = XXH3_64bits(&ptr, sizeof(ptr));
257
return hash % modulo;
259
-#endif
258
+}
259
+
260
+long get_netdata_cpus(void);
261
+
262
+static inline size_t pgc_max_evictors(void) {
263
+ return 1 + get_netdata_cpus() / 2;
264
+}
265
+
266
+static inline size_t pgc_max_flushers(void) {
267
+ return get_netdata_cpus();
268
}
269
270
#endif // DBENGINE_CACHE_H
src/database/engine/datafile.c
+2
-2
@@ -251,7 +251,7 @@ int create_data_file(struct rrdengine_datafile *datafile)
251
char path[RRDENG_PATH_MAX];
252
253
generate_datafilepath(datafile, path, sizeof(path));
254
- fd = open_file_for_io(path, O_CREAT | O_RDWR | O_TRUNC, &file, use_direct_io);
254
+ fd = open_file_for_io(path, O_CREAT | O_RDWR | O_TRUNC, &file, dbengine_use_direct_io);
255
if (fd < 0) {
256
ctx_fs_error(ctx);
257
return fd;
@@ -334,7 +334,7 @@ static int load_data_file(struct rrdengine_datafile *datafile)
334
char path[RRDENG_PATH_MAX];
335
336
generate_datafilepath(datafile, path, sizeof(path));
337
- fd = open_file_for_io(path, O_RDWR, &file, use_direct_io);
337
+ fd = open_file_for_io(path, O_RDWR, &file, dbengine_use_direct_io);
338
if (fd < 0) {
339
ctx_fs_error(ctx);
340
return fd;
src/database/engine/dbengine-stresstest.c
+7
-7
@@ -22,13 +22,13 @@ static RRDHOST *dbengine_rrdhost_find_or_create(char *name) {
22
default_rrd_history_entries,
23
RRD_MEMORY_MODE_DBENGINE,
24
health_plugin_enabled(),
25
- stream_conf_send_enabled,
26
- stream_conf_send_destination,
27
- stream_conf_send_api_key,
28
- stream_conf_send_charts_matching,
29
- stream_conf_replication_enabled,
30
- stream_conf_replication_period,
31
- stream_conf_replication_step,
25
+ stream_send.enabled,
26
+ stream_send.parents.destination,
27
+ stream_send.api_key,
28
+ stream_send.send_charts_matching,
29
+ stream_receive.replication.enabled,
30
+ stream_receive.replication.period,
31
+ stream_receive.replication.step,
32
NULL,
33
0
34
);
src/database/engine/dbengine-unittest.c
+7
-7
@@ -108,13 +108,13 @@ static RRDHOST *dbengine_rrdhost_find_or_create(char *name) {
108
default_rrd_history_entries,
109
RRD_MEMORY_MODE_DBENGINE,
110
health_plugin_enabled(),
111
- stream_conf_send_enabled,
112
- stream_conf_send_destination,
113
- stream_conf_send_api_key,
114
- stream_conf_send_charts_matching,
115
- stream_conf_replication_enabled,
116
- stream_conf_replication_period,
117
- stream_conf_replication_step,
111
+ stream_send.enabled,
112
+ stream_send.parents.destination,
113
+ stream_send.api_key,
114
+ stream_send.send_charts_matching,
115
+ stream_receive.replication.enabled,
116
+ stream_receive.replication.period,
117
+ stream_receive.replication.step,
118
NULL,
119
0
120
);
src/database/engine/journalfile.c
+2
-2
@@ -577,7 +577,7 @@ int journalfile_create(struct rrdengine_journalfile *journalfile, struct rrdengi
577
char path[RRDENG_PATH_MAX];
578
579
journalfile_v1_generate_path(datafile, path, sizeof(path));
580
- fd = open_file_for_io(path, O_CREAT | O_RDWR | O_TRUNC, &file, use_direct_io);
580
+ fd = open_file_for_io(path, O_CREAT | O_RDWR | O_TRUNC, &file, dbengine_use_direct_io);
581
if (fd < 0) {
582
ctx_fs_error(ctx);
583
return fd;
@@ -1522,7 +1522,7 @@ int journalfile_load(struct rrdengine_instance *ctx, struct rrdengine_journalfil
1522
1523
journalfile_v1_generate_path(datafile, path, sizeof(path));
1524
1525
- fd = open_file_for_io(path, O_RDWR, &file, use_direct_io);
1525
+ fd = open_file_for_io(path, O_RDWR, &file, dbengine_use_direct_io);
1526
if (fd < 0) {
1527
ctx_fs_error(ctx);
1528
src/database/engine/metric.c
+2
-1
@@ -375,6 +375,7 @@ inline MRG *mrg_create(ssize_t partitions) {
375
376
mrg->index[i].aral = aral_create(buf, sizeof(METRIC), 0, 16384, &mrg_aral_statistics, NULL, NULL, false, false);
377
}
378
+ telemetry_aral_register(mrg->index[0].aral, "mrg");
379
380
return mrg;
381
}
@@ -394,7 +395,7 @@ inline void mrg_destroy(MRG *mrg __maybe_unused) {
395
// to delete entries, the caller needs to keep pointers to them
396
// and delete them one by one
397
397
- ;
398
+ telemetry_aral_unregister(mrg->index[0].aral);
399
}
400
401
inline METRIC *mrg_metric_add_and_acquire(MRG *mrg, MRG_ENTRY entry, bool *ret) {
src/database/engine/page.c
+381
-138
@@ -6,6 +6,8 @@
6
7
typedef enum __attribute__((packed)) {
8
PAGE_OPTION_ALL_VALUES_EMPTY = (1 << 0),
9
+ PAGE_OPTION_ARAL_MARKED = (1 << 1),
10
+ PAGE_OPTION_ARAL_UNMARKED = (1 << 2),
11
} PAGE_OPTIONS;
12
13
typedef enum __attribute__((packed)) {
@@ -17,31 +19,32 @@ typedef enum __attribute__((packed)) {
19
20
typedef struct {
21
uint8_t *data;
20
- uint32_t size;
22
+ uint16_t size;
23
} page_raw_t;
24
23
-
25
typedef struct {
25
- size_t num_buffers;
26
gorilla_writer_t *writer;
27
- int aral_index;
27
+ uint16_t num_buffers;
28
} page_gorilla_t;
29
30
struct pgd {
31
+ // the used number of slots in the page
32
+ uint16_t used;
33
+
34
+ // the total number of slots available in the page
35
+ uint16_t slots;
36
+
37
// the page type
38
uint8_t type;
39
34
- // options related to the page
40
+ // the partition this pgd was allocated from
41
+ uint8_t partition;
42
+
43
+ // options related to the page
44
PAGE_OPTIONS options;
45
46
PGD_STATES states;
47
39
- // the uses number of slots in the page
40
- uint32_t used;
41
-
42
- // the total number of slots available in the page
43
- uint32_t slots;
44
-
48
union {
49
page_raw_t raw;
50
page_gorilla_t gorilla;
@@ -51,118 +54,241 @@ struct pgd {
54
// ----------------------------------------------------------------------------
55
// memory management
56
57
+#define ARAL_TOLERANCE_TO_DEDUP 7 // deduplicate aral sizes, if the delta is below this number of bytes
58
+#define PGD_ARAL_PARTITIONS 4
59
+
60
struct {
55
- ARAL *aral_pgd;
56
- ARAL *aral_data[RRD_STORAGE_TIERS];
57
- ARAL *aral_gorilla_buffer[4];
58
- ARAL *aral_gorilla_writer[4];
61
+ size_t sizeof_pgd;
62
+ size_t sizeof_gorilla_writer_t;
63
+ size_t sizeof_gorilla_buffer_32bit;
64
+
65
+ ARAL *aral_pgd[PGD_ARAL_PARTITIONS];
66
+ ARAL *aral_gorilla_buffer[PGD_ARAL_PARTITIONS];
67
+ ARAL *aral_gorilla_writer[PGD_ARAL_PARTITIONS];
68
} pgd_alloc_globals = {};
69
61
-static ARAL *pgd_aral_data_lookup(size_t size)
62
-{
63
- for (size_t tier = 0; tier < storage_tiers; tier++)
64
- if (size == tier_page_size[tier])
65
- return pgd_alloc_globals.aral_data[tier];
70
+#if RRD_STORAGE_TIERS != 5
71
+#error "You need to update the slots reserved for storage tiers"
72
+#endif
73
+
74
+static struct aral_statistics aral_statistics_for_pgd = { 0 };
75
67
- return NULL;
76
+static size_t aral_sizes_delta;
77
+static size_t aral_sizes_count;
78
+static size_t aral_sizes[] = {
79
+// // leave space for the storage tier page sizes
80
+ [RRD_STORAGE_TIERS - 5] = 0,
81
+ [RRD_STORAGE_TIERS - 4] = 0,
82
+ [RRD_STORAGE_TIERS - 3] = 0,
83
+ [RRD_STORAGE_TIERS - 2] = 0,
84
+ [RRD_STORAGE_TIERS - 1] = 0,
85
+
86
+ // gorilla buffer size
87
+ RRDENG_GORILLA_32BIT_BUFFER_SIZE,
88
+
89
+ // our structures
90
+ sizeof(gorilla_writer_t),
91
+ sizeof(PGD),
92
+};
93
+static ARAL **arals = NULL;
94
+
95
+#define arals_slot(slot, partition) ((partition) * aral_sizes_count + (slot))
96
+static ARAL *pgd_get_aral_by_size_and_partition(size_t size, size_t partition);
97
+
98
+size_t pgd_aral_structures(void) {
99
+ return aral_structures(pgd_alloc_globals.aral_pgd[0]);
100
}
101
70
-void pgd_init_arals(void)
71
-{
72
- // pgd aral
73
- {
74
- char buf[20 + 1];
75
- snprintfz(buf, sizeof(buf) - 1, "pgd");
102
+size_t pgd_aral_overhead(void) {
103
+ return aral_overhead(pgd_alloc_globals.aral_pgd[0]);
104
+}
105
77
- // FIXME: add stats
78
- pgd_alloc_globals.aral_pgd = aral_create(
79
- buf,
80
- sizeof(struct pgd),
81
- 64,
82
- 512 * (sizeof(struct pgd)),
83
- pgc_aral_statistics(),
84
- NULL, NULL, false, false);
85
- }
106
+int aral_size_sort_compare(const void *a, const void *b) {
107
+ size_t size_a = *(const size_t *)a;
108
+ size_t size_b = *(const size_t *)b;
109
+ return (size_a > size_b) - (size_a < size_b);
110
+}
111
87
- // tier page aral
88
- {
89
- for (size_t i = storage_tiers; i > 0 ;i--)
90
- {
91
- size_t tier = storage_tiers - i;
92
-
93
- char buf[20 + 1];
94
- snprintfz(buf, sizeof(buf) - 1, "tier%zu-pages", tier);
95
-
96
- pgd_alloc_globals.aral_data[tier] = aral_create(
97
- buf,
98
- tier_page_size[tier],
99
- 64,
100
- 512 * (tier_page_size[tier]),
101
- pgc_aral_statistics(),
102
- NULL, NULL, false, false);
103
- }
112
+void pgd_init_arals(void) {
113
+ aral_sizes_count = _countof(aral_sizes);
114
+
115
+ for(size_t i = 0; i < RRD_STORAGE_TIERS ;i++)
116
+ aral_sizes[i] = tier_page_size[i];
117
+
118
+ size_t max_delta = 0;
119
+ for(size_t i = 0; i < aral_sizes_count ;i++) {
120
+ size_t wanted = aral_sizes[i];
121
+ size_t usable = aral_sizes[i]; /* aral_allocation_slot_size(wanted, true);*/
122
+ internal_fatal(usable < wanted, "usable cannot be less than wanted");
123
+ if(usable > wanted && usable - wanted > max_delta)
124
+ max_delta = usable - wanted;
125
+
126
+ aral_sizes[i] = usable;
127
}
128
+ aral_sizes_delta = max_delta + ARAL_TOLERANCE_TO_DEDUP;
129
+
130
+ // sort the array
131
+ qsort(aral_sizes, aral_sizes_count, sizeof(size_t), aral_size_sort_compare);
132
+
133
+ // deduplicate (with some tolerance)
134
+ size_t unique_count = 1;
135
+ for (size_t i = 1; i < aral_sizes_count; ++i) {
136
+ if (aral_sizes[i] > aral_sizes[unique_count - 1] + aral_sizes_delta)
137
+ aral_sizes[unique_count++] = aral_sizes[i];
138
+ else
139
+ aral_sizes[unique_count - 1] = aral_sizes[i];
140
+ }
141
+ aral_sizes_count = unique_count;
142
+
143
+ // clear the rest
144
+ for(size_t i = unique_count; i < _countof(aral_sizes) ;i++)
145
+ aral_sizes[i] = 0;
146
+
147
+ // allocate all the arals
148
+ arals = callocz(aral_sizes_count * PGD_ARAL_PARTITIONS, sizeof(ARAL *));
149
+ for(size_t slot = 0; slot < aral_sizes_count ; slot++) {
150
+ for(size_t partition = 0; partition < PGD_ARAL_PARTITIONS; partition++) {
151
+
152
+ if(partition > 0 && aral_sizes[slot] > 128) {
153
+ // do not create partitions for sizes above 128 bytes
154
+ // use the first partition for all of them
155
+ arals[arals_slot(slot, partition)] = arals[arals_slot(slot, 0)];
156
+ continue;
157
+ }
158
106
- // gorilla buffers aral
107
- for (size_t i = 0; i != 4; i++) {
108
- char buf[20 + 1];
109
- snprintfz(buf, sizeof(buf) - 1, "gbuffer-%zu", i);
159
+ char buf[32];
160
+ snprintfz(buf, sizeof(buf), "pgd-%zu-%zu", aral_sizes[slot], partition);
161
111
- // FIXME: add stats
112
- pgd_alloc_globals.aral_gorilla_buffer[i] = aral_create(
162
+ arals[arals_slot(slot, partition)] = aral_create(
163
buf,
114
- RRDENG_GORILLA_32BIT_BUFFER_SIZE,
115
- 64,
116
- 512 * RRDENG_GORILLA_32BIT_BUFFER_SIZE,
117
- pgc_aral_statistics(),
164
+ aral_sizes[slot],
165
+ 0,
166
+ 0,
167
+ &aral_statistics_for_pgd,
168
NULL, NULL, false, false);
169
+ }
170
}
171
121
- // gorilla writers aral
122
- for (size_t i = 0; i != 4; i++) {
123
- char buf[20 + 1];
124
- snprintfz(buf, sizeof(buf) - 1, "gwriter-%zu", i);
172
+ for(size_t p = 0; p < PGD_ARAL_PARTITIONS ;p++) {
173
+ pgd_alloc_globals.aral_pgd[p] = pgd_get_aral_by_size_and_partition(sizeof(PGD), p);
174
+ pgd_alloc_globals.aral_gorilla_writer[p] = pgd_get_aral_by_size_and_partition(sizeof(gorilla_writer_t), p);
175
+ pgd_alloc_globals.aral_gorilla_buffer[p] = pgd_get_aral_by_size_and_partition(RRDENG_GORILLA_32BIT_BUFFER_SIZE, p);
176
126
- // FIXME: add stats
127
- pgd_alloc_globals.aral_gorilla_writer[i] = aral_create(
128
- buf,
129
- sizeof(gorilla_writer_t),
130
- 64,
131
- 512 * sizeof(gorilla_writer_t),
132
- pgc_aral_statistics(),
133
- NULL, NULL, false, false);
177
+ internal_fatal(!pgd_alloc_globals.aral_pgd[p] ||
178
+ !pgd_alloc_globals.aral_gorilla_writer[p] ||
179
+ !pgd_alloc_globals.aral_gorilla_buffer[p]
180
+ , "required PGD aral sizes not found");
181
}
182
+
183
+ pgd_alloc_globals.sizeof_pgd = aral_actual_element_size(pgd_alloc_globals.aral_pgd[0]);
184
+ pgd_alloc_globals.sizeof_gorilla_writer_t = aral_actual_element_size(pgd_alloc_globals.aral_gorilla_writer[0]);
185
+ pgd_alloc_globals.sizeof_gorilla_buffer_32bit = aral_actual_element_size(pgd_alloc_globals.aral_gorilla_buffer[0]);
186
+
187
+ telemetry_aral_register(pgd_alloc_globals.aral_pgd[0], "pgd");
188
}
189
137
-static void *pgd_data_aral_alloc(size_t size)
138
-{
139
- ARAL *ar = pgd_aral_data_lookup(size);
140
- if (!ar)
141
- return mallocz(size);
190
+static ARAL *pgd_get_aral_by_size_and_partition(size_t size, size_t partition) {
191
+ internal_fatal(partition >= PGD_ARAL_PARTITIONS, "Wrong partition %zu", partition);
192
+
193
+ size_t slot;
194
+
195
+ if (size <= aral_sizes[0])
196
+ slot = 0;
197
+
198
+ else if (size > aral_sizes[aral_sizes_count - 1])
199
+ return NULL;
200
+
201
+ else {
202
+ // binary search for the smallest size >= requested size
203
+ size_t low = 0, high = aral_sizes_count - 1;
204
+ while (low < high) {
205
+ size_t mid = low + (high - low) / 2;
206
+ if (aral_sizes[mid] >= size)
207
+ high = mid;
208
+ else
209
+ low = mid + 1;
210
+ }
211
+ slot = low; // This is the smallest index where aral_sizes[slot] >= size
212
+ }
213
+ internal_fatal(slot >= aral_sizes_count || aral_sizes[slot] < size, "Invalid PGD size binary search");
214
+
215
+ ARAL *ar = arals[arals_slot(slot, partition)];
216
+ internal_fatal(!ar || aral_requested_element_size(ar) < size, "Invalid PGD aral lookup");
217
+ return ar;
218
+}
219
+
220
+static inline gorilla_writer_t *pgd_gorilla_writer_alloc(size_t partition) {
221
+ internal_fatal(partition >= PGD_ARAL_PARTITIONS, "invalid gorilla writer partition %zu", partition);
222
+ return aral_mallocz_marked(pgd_alloc_globals.aral_gorilla_writer[partition]);
223
+}
224
+
225
+static inline gorilla_buffer_t *pgd_gorilla_buffer_alloc(size_t partition) {
226
+ internal_fatal(partition >= PGD_ARAL_PARTITIONS, "invalid gorilla buffer partition %zu", partition);
227
+ return aral_mallocz_marked(pgd_alloc_globals.aral_gorilla_buffer[partition]);
228
+}
229
+
230
+static inline PGD *pgd_alloc(bool for_collector) {
231
+ size_t partition = gettid_cached() % PGD_ARAL_PARTITIONS;
232
+ PGD *pgd;
233
+
234
+ if(for_collector)
235
+ pgd = aral_mallocz_marked(pgd_alloc_globals.aral_pgd[partition]);
236
else
143
- return aral_mallocz(ar);
237
+ pgd = aral_mallocz(pgd_alloc_globals.aral_pgd[partition]);
238
+
239
+ pgd->partition = partition;
240
+ return pgd;
241
}
242
146
-static void pgd_data_aral_free(void *page, size_t size)
147
-{
148
- ARAL *ar = pgd_aral_data_lookup(size);
149
- if (!ar)
150
- freez(page);
243
+static inline void *pgd_data_alloc(size_t size, size_t partition, bool for_collector) {
244
+ ARAL *ar = pgd_get_aral_by_size_and_partition(size, partition);
245
+ if(ar) {
246
+ if(for_collector)
247
+ return aral_mallocz_marked(ar);
248
+ else
249
+ return aral_mallocz(ar);
250
+ }
251
else
252
+ return mallocz(size);
253
+}
254
+
255
+static void pgd_data_free(void *page, size_t size, size_t partition) {
256
+ ARAL *ar = pgd_get_aral_by_size_and_partition(size, partition);
257
+ if(ar)
258
aral_freez(ar, page);
259
+ else
260
+ freez(page);
261
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_TIER1_ARAL);
262
+}
263
+
264
+static void pgd_data_unmark(void *page, size_t size, size_t partition) {
265
+ if(!page) return;
266
+
267
+ ARAL *ar = pgd_get_aral_by_size_and_partition(size, partition);
268
+ if(ar)
269
+ aral_unmark_allocation(ar, page);
270
+}
271
+
272
+static size_t pgd_data_footprint(size_t size, size_t partition) {
273
+ ARAL *ar = pgd_get_aral_by_size_and_partition(size, partition);
274
+ if(ar)
275
+ return aral_actual_element_size(ar);
276
+ else
277
+ return size;
278
}
279
280
// ----------------------------------------------------------------------------
281
// management api
282
158
-PGD *pgd_create(uint8_t type, uint32_t slots)
159
-{
160
- PGD *pg = aral_mallocz(pgd_alloc_globals.aral_pgd);
283
+PGD *pgd_create(uint8_t type, uint32_t slots) {
284
+
285
+ PGD *pg = pgd_alloc(true); // this is malloc'd !
286
pg->type = type;
287
+ pg->states = PGD_STATE_CREATED_FROM_COLLECTOR;
288
+ pg->options = PAGE_OPTION_ALL_VALUES_EMPTY | PAGE_OPTION_ARAL_MARKED;
289
+
290
pg->used = 0;
291
pg->slots = slots;
164
- pg->options = PAGE_OPTION_ALL_VALUES_EMPTY;
165
- pg->states = PGD_STATE_CREATED_FROM_COLLECTOR;
292
293
switch (type) {
294
case RRDENG_PAGE_TYPE_ARRAY_32BIT:
@@ -173,23 +299,20 @@ PGD *pgd_create(uint8_t type, uint32_t slots)
299
"DBENGINE: invalid number of slots (%u) or page type (%u)", slots, type);
300
301
pg->raw.size = size;
176
- pg->raw.data = pgd_data_aral_alloc(size);
302
+ pg->raw.data = pgd_data_alloc(size, pg->partition, true);
303
break;
304
}
305
case RRDENG_PAGE_TYPE_GORILLA_32BIT: {
306
internal_fatal(slots == 1,
307
"DBENGINE: invalid number of slots (%u) or page type (%u)", slots, type);
308
183
- pg->slots = 8 * RRDENG_GORILLA_32BIT_BUFFER_SLOTS;
184
-
309
// allocate new gorilla writer
186
- pg->gorilla.aral_index = gettid_cached() % 4;
187
- pg->gorilla.writer = aral_mallocz(pgd_alloc_globals.aral_gorilla_writer[pg->gorilla.aral_index]);
310
+ pg->gorilla.writer = pgd_gorilla_writer_alloc(pg->partition);
311
312
// allocate new gorilla buffer
190
- gorilla_buffer_t *gbuf = aral_mallocz(pgd_alloc_globals.aral_gorilla_buffer[pg->gorilla.aral_index]);
313
+ gorilla_buffer_t *gbuf = pgd_gorilla_buffer_alloc(pg->partition);
314
memset(gbuf, 0, RRDENG_GORILLA_32BIT_BUFFER_SIZE);
192
- global_statistics_gorilla_buffer_add_hot();
315
+ telemetry_gorilla_hot_buffer_added();
316
317
*pg->gorilla.writer = gorilla_writer_init(gbuf, RRDENG_GORILLA_32BIT_BUFFER_SLOTS);
318
pg->gorilla.num_buffers = 1;
@@ -198,7 +321,7 @@ PGD *pgd_create(uint8_t type, uint32_t slots)
321
}
322
default:
323
netdata_log_error("%s() - Unknown page type: %uc", __FUNCTION__, type);
201
- aral_freez(pgd_alloc_globals.aral_pgd, pg);
324
+ aral_freez(pgd_alloc_globals.aral_pgd[pg->partition], pg);
325
pg = PGD_EMPTY;
326
break;
327
}
@@ -206,52 +329,47 @@ PGD *pgd_create(uint8_t type, uint32_t slots)
329
return pg;
330
}
331
209
-PGD *pgd_create_from_disk_data(uint8_t type, void *base, uint32_t size)
210
-{
211
- if (!size)
212
- return PGD_EMPTY;
332
+PGD *pgd_create_from_disk_data(uint8_t type, void *base, uint32_t size) {
333
214
- if (size < page_type_size[type])
334
+ if (!size || size < page_type_size[type])
335
return PGD_EMPTY;
336
217
- PGD *pg = aral_mallocz(pgd_alloc_globals.aral_pgd);
218
-
337
+ PGD *pg = pgd_alloc(false); // this is malloc'd !
338
pg->type = type;
339
pg->states = PGD_STATE_CREATED_FROM_DISK;
221
- pg->options = ~PAGE_OPTION_ALL_VALUES_EMPTY;
340
+ pg->options = PAGE_OPTION_ARAL_UNMARKED;
341
342
switch (type)
343
{
344
case RRDENG_PAGE_TYPE_ARRAY_32BIT:
345
case RRDENG_PAGE_TYPE_ARRAY_TIER1:
227
- pg->raw.size = size;
346
pg->used = size / page_type_size[type];
347
pg->slots = pg->used;
348
231
- pg->raw.data = pgd_data_aral_alloc(size);
349
+ pg->raw.size = size;
350
+ pg->raw.data = pgd_data_alloc(size, pg->partition, false);
351
memcpy(pg->raw.data, base, size);
352
break;
353
+
354
case RRDENG_PAGE_TYPE_GORILLA_32BIT:
355
internal_fatal(size == 0, "Asked to create page with 0 data!!!");
356
internal_fatal(size % sizeof(uint32_t), "Unaligned gorilla buffer size");
357
internal_fatal(size % RRDENG_GORILLA_32BIT_BUFFER_SIZE, "Expected size to be a multiple of %zu-bytes",
358
RRDENG_GORILLA_32BIT_BUFFER_SIZE);
359
240
- pg->raw.data = mallocz(size);
360
+ pg->raw.data = (void *)pgd_data_alloc(size, pg->partition, false);
361
pg->raw.size = size;
362
243
- // TODO: rm this
244
- memset(pg->raw.data, 0, size);
245
- memcpy(pg->raw.data, base, size);
363
+ memcpy(pg->raw.data, base, pg->raw.size);
364
365
uint32_t total_entries = gorilla_buffer_patch((void *) pg->raw.data);
248
-
366
pg->used = total_entries;
367
pg->slots = pg->used;
368
break;
369
+
370
default:
371
netdata_log_error("%s() - Unknown page type: %uc", __FUNCTION__, type);
254
- aral_freez(pgd_alloc_globals.aral_pgd, pg);
372
+ aral_freez(pgd_alloc_globals.aral_pgd[pg->partition], pg);
373
pg = PGD_EMPTY;
374
break;
375
}
@@ -259,26 +377,30 @@ PGD *pgd_create_from_disk_data(uint8_t type, void *base, uint32_t size)
377
return pg;
378
}
379
262
-void pgd_free(PGD *pg)
263
-{
264
- if (!pg)
380
+void pgd_free(PGD *pg) {
381
+ if (!pg || pg == PGD_EMPTY)
382
return;
383
267
- if (pg == PGD_EMPTY)
268
- return;
384
+ internal_fatal(pg->partition >= PGD_ARAL_PARTITIONS,
385
+ "PGD partition is invalid %u", pg->partition);
386
387
switch (pg->type)
388
{
389
case RRDENG_PAGE_TYPE_ARRAY_32BIT:
390
case RRDENG_PAGE_TYPE_ARRAY_TIER1:
274
- pgd_data_aral_free(pg->raw.data, pg->raw.size);
391
+ pgd_data_free(pg->raw.data, pg->raw.size, pg->partition);
392
break;
393
+
394
case RRDENG_PAGE_TYPE_GORILLA_32BIT: {
395
if (pg->states & PGD_STATE_CREATED_FROM_DISK)
396
{
397
internal_fatal(pg->raw.data == NULL, "Tried to free gorilla PGD loaded from disk with NULL data");
280
- freez(pg->raw.data);
398
+
399
+ pgd_data_free(pg->raw.data, pg->raw.size, pg->partition);
400
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_ARAL);
401
+
402
pg->raw.data = NULL;
403
+ pg->raw.size = 0;
404
}
405
else if ((pg->states & PGD_STATE_CREATED_FROM_COLLECTOR) ||
406
(pg->states & PGD_STATE_SCHEDULED_FOR_FLUSHING) ||
@@ -294,15 +416,19 @@ void pgd_free(PGD *pg)
416
gorilla_buffer_t *gbuf = gorilla_writer_drop_head_buffer(pg->gorilla.writer);
417
if (!gbuf)
418
break;
297
- aral_freez(pgd_alloc_globals.aral_gorilla_buffer[pg->gorilla.aral_index], gbuf);
419
+ aral_freez(pgd_alloc_globals.aral_gorilla_buffer[pg->partition], gbuf);
420
pg->gorilla.num_buffers -= 1;
421
}
422
423
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_GLIVE);
424
+
425
internal_fatal(pg->gorilla.num_buffers != 0,
426
"Could not free all gorilla writer buffers");
427
304
- aral_freez(pgd_alloc_globals.aral_gorilla_writer[pg->gorilla.aral_index], pg->gorilla.writer);
428
+ aral_freez(pgd_alloc_globals.aral_gorilla_writer[pg->partition], pg->gorilla.writer);
429
pg->gorilla.writer = NULL;
430
+
431
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_GWORKER);
432
} else {
433
fatal("pgd_free() called on gorilla page with unsupported state");
434
// TODO: should we support any other states?
@@ -317,7 +443,62 @@ void pgd_free(PGD *pg)
443
break;
444
}
445
320
- aral_freez(pgd_alloc_globals.aral_pgd, pg);
446
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_DATA);
447
+
448
+ aral_freez(pgd_alloc_globals.aral_pgd[pg->partition], pg);
449
+
450
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_ARAL);
451
+}
452
+
453
+static void pgd_aral_unmark(PGD *pg) {
454
+ if (!pg ||
455
+ pg == PGD_EMPTY ||
456
+ (pg->options & PAGE_OPTION_ARAL_UNMARKED) ||
457
+ !(pg->options & PAGE_OPTION_ARAL_MARKED))
458
+ return;
459
+
460
+ internal_fatal(pg->partition >= PGD_ARAL_PARTITIONS,
461
+ "PGD partition is invalid %u", pg->partition);
462
+
463
+ switch (pg->type)
464
+ {
465
+ case RRDENG_PAGE_TYPE_ARRAY_32BIT:
466
+ case RRDENG_PAGE_TYPE_ARRAY_TIER1:
467
+ pgd_data_unmark(pg->raw.data, pg->raw.size, pg->partition);
468
+ break;
469
+
470
+ case RRDENG_PAGE_TYPE_GORILLA_32BIT: {
471
+ if (pg->states & PGD_STATE_CREATED_FROM_DISK)
472
+ pgd_data_unmark(pg->raw.data, pg->raw.size, pg->partition);
473
+
474
+ else if ((pg->states & PGD_STATE_CREATED_FROM_COLLECTOR) ||
475
+ (pg->states & PGD_STATE_SCHEDULED_FOR_FLUSHING) ||
476
+ (pg->states & PGD_STATE_FLUSHED_TO_DISK))
477
+ {
478
+ internal_fatal(pg->gorilla.writer == NULL, "PGD does not have an active gorilla writer");
479
+ internal_fatal(pg->gorilla.num_buffers == 0, "PGD does not have any gorilla buffers allocated");
480
+
481
+ gorilla_writer_aral_unmark(pg->gorilla.writer, pgd_alloc_globals.aral_gorilla_buffer[pg->partition]);
482
+ aral_unmark_allocation(pgd_alloc_globals.aral_gorilla_writer[pg->partition], pg->gorilla.writer);
483
+ }
484
+ else {
485
+ fatal("pgd_free() called on gorilla page with unsupported state");
486
+ // TODO: should we support any other states?
487
+ // if (!(pg->states & PGD_STATE_FLUSHED_TO_DISK))
488
+ // fatal("pgd_free() is not supported yet for pages flushed to disk");
489
+ }
490
+
491
+ break;
492
+ }
493
+ default:
494
+ netdata_log_error("%s() - Unknown page type: %uc", __FUNCTION__, pg->type);
495
+ break;
496
+ }
497
+
498
+ aral_unmark_allocation(pgd_alloc_globals.aral_pgd[pg->partition], pg);
499
+
500
+ // make sure we will not do this again
501
+ pg->options |= PAGE_OPTION_ARAL_UNMARKED;
502
}
503
504
// ----------------------------------------------------------------------------
@@ -356,7 +537,54 @@ uint32_t pgd_slots_used(PGD *pg)
537
return pg->used;
538
}
539
540
+uint32_t pgd_capacity(PGD *pg) {
541
+ if (!pg)
542
+ return 0;
543
+
544
+ if (pg == PGD_EMPTY)
545
+ return 0;
546
+
547
+ return pg->slots;
548
+}
549
+
550
+// return the overall memory footprint of the page, including all its structures and overheads
551
uint32_t pgd_memory_footprint(PGD *pg)
552
+{
553
+ if (!pg)
554
+ return 0;
555
+
556
+ if (pg == PGD_EMPTY)
557
+ return 0;
558
+
559
+ size_t footprint = pgd_alloc_globals.sizeof_pgd;
560
+
561
+ switch (pg->type) {
562
+ case RRDENG_PAGE_TYPE_ARRAY_32BIT:
563
+ case RRDENG_PAGE_TYPE_ARRAY_TIER1:
564
+ footprint += pgd_data_footprint(pg->raw.size, pg->partition);
565
+ break;
566
+
567
+ case RRDENG_PAGE_TYPE_GORILLA_32BIT: {
568
+ if (pg->states & PGD_STATE_CREATED_FROM_DISK)
569
+ footprint += pgd_data_footprint(pg->raw.size, pg->partition);
570
+
571
+ else {
572
+ footprint += pgd_alloc_globals.sizeof_gorilla_writer_t;
573
+ footprint += pg->gorilla.num_buffers * pgd_alloc_globals.sizeof_gorilla_buffer_32bit;
574
+ }
575
+ break;
576
+ }
577
+
578
+ default:
579
+ netdata_log_error("%s() - Unknown page type: %uc", __FUNCTION__, pg->type);
580
+ break;
581
+ }
582
+
583
+ return footprint;
584
+}
585
+
586
+// return the nominal buffer size depending on the page type - used by the PGC histogram
587
+uint32_t pgd_buffer_memory_footprint(PGD *pg)
588
{
589
if (!pg)
590
return 0;
@@ -365,19 +593,22 @@ uint32_t pgd_memory_footprint(PGD *pg)
593
return 0;
594
595
size_t footprint = 0;
596
+
597
switch (pg->type) {
598
case RRDENG_PAGE_TYPE_ARRAY_32BIT:
599
case RRDENG_PAGE_TYPE_ARRAY_TIER1:
371
- footprint = sizeof(PGD) + pg->raw.size;
600
+ footprint = pg->raw.size;
601
break;
602
+
603
case RRDENG_PAGE_TYPE_GORILLA_32BIT: {
604
if (pg->states & PGD_STATE_CREATED_FROM_DISK)
375
- footprint = sizeof(PGD) + pg->raw.size;
376
- else
377
- footprint = sizeof(PGD) + sizeof(gorilla_writer_t) + (pg->gorilla.num_buffers * RRDENG_GORILLA_32BIT_BUFFER_SIZE);
605
+ footprint = pg->raw.size;
606
607
+ else
608
+ footprint = pg->gorilla.num_buffers * RRDENG_GORILLA_32BIT_BUFFER_SIZE;
609
break;
610
}
611
+
612
default:
613
netdata_log_error("%s() - Unknown page type: %uc", __FUNCTION__, pg->type);
614
break;
@@ -393,6 +624,9 @@ uint32_t pgd_disk_footprint(PGD *pg)
624
625
size_t size = 0;
626
627
+ // since the page is ready for flushing, let's unmark its pages to ARAL
628
+ pgd_aral_unmark(pg);
629
+
630
switch (pg->type) {
631
case RRDENG_PAGE_TYPE_ARRAY_32BIT:
632
case RRDENG_PAGE_TYPE_ARRAY_TIER1: {
@@ -415,10 +649,12 @@ uint32_t pgd_disk_footprint(PGD *pg)
649
650
size = pg->gorilla.num_buffers * RRDENG_GORILLA_32BIT_BUFFER_SIZE;
651
418
- if (pg->states & PGD_STATE_CREATED_FROM_COLLECTOR) {
419
- global_statistics_tier0_disk_compressed_bytes(gorilla_writer_nbytes(pg->gorilla.writer));
420
- global_statistics_tier0_disk_uncompressed_bytes(gorilla_writer_entries(pg->gorilla.writer) * sizeof(storage_number));
421
- }
652
+ if (pg->states & PGD_STATE_CREATED_FROM_COLLECTOR)
653
+ telemetry_gorilla_tier0_page_flush(
654
+ gorilla_writer_actual_nbytes(pg->gorilla.writer),
655
+ gorilla_writer_optimal_nbytes(pg->gorilla.writer),
656
+ tier_page_size[0]);
657
+
658
} else if (pg->states & PGD_STATE_CREATED_FROM_DISK) {
659
size = pg->raw.size;
660
} else {
@@ -434,6 +670,7 @@ uint32_t pgd_disk_footprint(PGD *pg)
670
671
internal_fatal(pg->states & PGD_STATE_CREATED_FROM_DISK,
672
"Disk footprint asked for page created from disk.");
673
+
674
pg->states = PGD_STATE_SCHEDULED_FOR_FLUSHING;
675
return size;
676
}
@@ -461,7 +698,7 @@ void pgd_copy_to_extent(PGD *pg, uint8_t *dst, uint32_t dst_size)
698
bool ok = gorilla_writer_serialize(pg->gorilla.writer, dst, dst_size);
699
UNUSED(ok);
700
internal_fatal(!ok,
464
- "pgd_copy_to_extent() tried to serialize pg=%p, gw=%p (with dst_size=%u bytes, num_buffers=%zu)",
701
+ "pgd_copy_to_extent() tried to serialize pg=%p, gw=%p (with dst_size=%u bytes, num_buffers=%u)",
702
pg, pg->gorilla.writer, dst_size, pg->gorilla.num_buffers);
703
break;
704
}
@@ -476,7 +713,8 @@ void pgd_copy_to_extent(PGD *pg, uint8_t *dst, uint32_t dst_size)
713
// ----------------------------------------------------------------------------
714
// data collection
715
479
-void pgd_append_point(PGD *pg,
716
+// returns additional memory that may have been allocated to store this point
717
+size_t pgd_append_point(PGD *pg,
718
usec_t point_in_time_ut __maybe_unused,
719
NETDATA_DOUBLE n,
720
NETDATA_DOUBLE min_value,
@@ -535,22 +773,27 @@ void pgd_append_point(PGD *pg,
773
774
bool ok = gorilla_writer_write(pg->gorilla.writer, t);
775
if (!ok) {
538
- gorilla_buffer_t *new_buffer = aral_mallocz(pgd_alloc_globals.aral_gorilla_buffer[pg->gorilla.aral_index]);
776
+ gorilla_buffer_t *new_buffer = pgd_gorilla_buffer_alloc(pg->partition);
777
memset(new_buffer, 0, RRDENG_GORILLA_32BIT_BUFFER_SIZE);
778
779
gorilla_writer_add_buffer(pg->gorilla.writer, new_buffer, RRDENG_GORILLA_32BIT_BUFFER_SLOTS);
780
pg->gorilla.num_buffers += 1;
543
- global_statistics_gorilla_buffer_add_hot();
781
+ telemetry_gorilla_hot_buffer_added();
782
783
ok = gorilla_writer_write(pg->gorilla.writer, t);
784
internal_fatal(ok == false, "Failed to writer value in newly allocated gorilla buffer.");
785
+
786
+ return RRDENG_GORILLA_32BIT_BUFFER_SIZE;
787
}
788
+
789
break;
790
}
791
default:
792
netdata_log_error("%s() - Unknown page type: %uc", __FUNCTION__, pg->type);
793
break;
794
}
795
+
796
+ return 0;
797
}
798
799
// ----------------------------------------------------------------------------
@@ -589,7 +832,6 @@ static void pgdc_seek(PGDC *pgdc, uint32_t position)
832
uint32_t value;
833
834
bool ok = gorilla_reader_read(&pgdc->gr, &value);
592
-
835
if (!ok) {
836
// this is fine, the reader will return empty points
837
break;
@@ -664,6 +906,7 @@ bool pgdc_get_next_point(PGDC *pgdc, uint32_t expected_position __maybe_unused,
906
907
uint32_t n = 666666666;
908
bool ok = gorilla_reader_read(&pgdc->gr, &n);
909
+
910
if (ok) {
911
sp->min = sp->max = sp->sum = unpack_storage_number(n);
912
sp->flags = (SN_FLAGS)(n & SN_USER_FLAGS);
src/database/engine/page.h
+6
-1
@@ -33,12 +33,17 @@ uint32_t pgd_type(PGD *pg);
33
bool pgd_is_empty(PGD *pg);
34
uint32_t pgd_slots_used(PGD *pg);
35
36
+uint32_t pgd_buffer_memory_footprint(PGD *pg);
37
uint32_t pgd_memory_footprint(PGD *pg);
38
+uint32_t pgd_capacity(PGD *pg);
39
uint32_t pgd_disk_footprint(PGD *pg);
40
41
+size_t pgd_aral_structures(void);
42
+size_t pgd_aral_overhead(void);
43
+
44
void pgd_copy_to_extent(PGD *pg, uint8_t *dst, uint32_t dst_size);
45
41
-void pgd_append_point(PGD *pg,
46
+size_t pgd_append_point(PGD *pg,
47
usec_t point_in_time_ut,
48
NETDATA_DOUBLE n,
49
NETDATA_DOUBLE min_value,
src/database/engine/pagecache.c
+65
-31
@@ -63,6 +63,7 @@ static void open_cache_free_clean_page_callback(PGC *cache __maybe_unused, PGC_E
63
{
64
struct rrdengine_datafile *datafile = entry.data;
65
datafile_release(datafile, DATAFILE_ACQUIRE_OPEN_CACHE);
66
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_OPEN);
67
}
68
69
static void open_cache_flush_dirty_page_callback(PGC *cache __maybe_unused, PGC_ENTRY *entries_array __maybe_unused, PGC_PAGE **pages_array __maybe_unused, size_t entries __maybe_unused)
@@ -73,6 +74,7 @@ static void open_cache_flush_dirty_page_callback(PGC *cache __maybe_unused, PGC_
74
static void extent_cache_free_clean_page_callback(PGC *cache __maybe_unused, PGC_ENTRY entry __maybe_unused)
75
{
76
dbengine_extent_free(entry.data, entry.size);
77
+ timing_dbengine_evict_step(TIMING_STEP_DBENGINE_EVICT_FREE_EXTENT);
78
}
79
80
static void extent_cache_flush_dirty_page_callback(PGC *cache __maybe_unused, PGC_ENTRY *entries_array __maybe_unused, PGC_PAGE **pages_array __maybe_unused, size_t entries __maybe_unused)
@@ -1030,8 +1032,21 @@ void pgc_open_add_hot_page(Word_t section, Word_t metric_id, time_t start_time_s
1032
}
1033
1034
size_t dynamic_open_cache_size(void) {
1033
- size_t main_cache_size = pgc_get_wanted_cache_size(main_cache);
1034
- size_t target_size = main_cache_size / 100 * 5;
1035
+ size_t main_wanted_cache_size = pgc_get_wanted_cache_size(main_cache);
1036
+ size_t target_size = main_wanted_cache_size / 100 * 5; // 5%
1037
+
1038
+// static bool query_current_size = true;
1039
+// if(query_current_size) {
1040
+// size_t main_current_cache_size = pgc_get_current_cache_size(main_cache);
1041
+//
1042
+// size_t main_free_cache_size = (main_wanted_cache_size > main_current_cache_size) ?
1043
+// main_wanted_cache_size - main_current_cache_size : 0;
1044
+//
1045
+// if(main_free_cache_size > target_size)
1046
+// target_size = main_free_cache_size;
1047
+// else
1048
+// query_current_size = false;
1049
+// }
1050
1051
if(target_size < 2 * 1024 * 1024)
1052
target_size = 2 * 1024 * 1024;
@@ -1040,15 +1055,33 @@ size_t dynamic_open_cache_size(void) {
1055
}
1056
1057
size_t dynamic_extent_cache_size(void) {
1043
- size_t main_cache_size = pgc_get_wanted_cache_size(main_cache);
1044
- size_t target_size = main_cache_size / 100 * 5;
1045
-
1046
- if(target_size < 3 * 1024 * 1024)
1047
- target_size = 3 * 1024 * 1024;
1058
+ size_t main_wanted_cache_size = pgc_get_wanted_cache_size(main_cache);
1059
+
1060
+ size_t target_size = main_wanted_cache_size / 100 * 10; // 10%
1061
+
1062
+// static bool query_current_size = true;
1063
+// if(query_current_size) {
1064
+// size_t main_current_cache_size = pgc_get_current_cache_size(main_cache);
1065
+//
1066
+// size_t main_free_cache_size = (main_wanted_cache_size > main_current_cache_size) ?
1067
+// main_wanted_cache_size - main_current_cache_size : 0;
1068
+//
1069
+// if(main_free_cache_size > target_size)
1070
+// target_size = main_free_cache_size;
1071
+// else
1072
+// query_current_size = false;
1073
+// }
1074
+
1075
+ if(target_size < 5 * 1024 * 1024)
1076
+ target_size = 5 * 1024 * 1024;
1077
1078
return target_size;
1079
}
1080
1081
+size_t pgc_main_nominal_page_size(void *data) {
1082
+ return pgd_buffer_memory_footprint(data);
1083
+}
1084
+
1085
void pgc_and_mrg_initialize(void)
1086
{
1087
main_mrg = mrg_create(0);
@@ -1066,51 +1099,52 @@ void pgc_and_mrg_initialize(void)
1099
extent_cache_size += (size_t)(default_rrdeng_extent_cache_mb * 1024ULL * 1024ULL);
1100
1101
main_cache = pgc_create(
1069
- "main_cache",
1102
+ "MAIN_PGC",
1103
main_cache_size,
1104
main_cache_free_clean_page_callback,
1105
(size_t) rrdeng_pages_per_extent,
1106
main_cache_flush_dirty_page_init_callback,
1107
main_cache_flush_dirty_page_callback,
1075
- 10,
1076
- 10240, // if there are that many threads, evict so many at once!
1077
- 1000, //
1078
- 5, // don't delay too much other threads
1079
- PGC_OPTIONS_AUTOSCALE, // AUTOSCALE = 2x max hot pages
1080
- 0, // 0 = as many as the system cpus
1108
+ 2,
1109
+ pgc_max_evictors(),
1110
+ 1000,
1111
+ 1,
1112
+ PGC_OPTIONS_AUTOSCALE,
1113
+ 0,
1114
0
1115
);
1116
+ pgc_set_nominal_page_size_callback(main_cache, pgc_main_nominal_page_size);
1117
1118
open_cache = pgc_create(
1085
- "open_cache",
1086
- open_cache_size, // the default is 1MB
1119
+ "OPEN_PGC",
1120
+ open_cache_size,
1121
open_cache_free_clean_page_callback,
1088
- 1,
1122
+ 2,
1123
NULL,
1124
open_cache_flush_dirty_page_callback,
1091
- 10,
1092
- 10240, // if there are that many threads, evict that many at once!
1093
- 1000, //
1094
- 3, // don't delay too much other threads
1095
- PGC_OPTIONS_AUTOSCALE | PGC_OPTIONS_EVICT_PAGES_INLINE | PGC_OPTIONS_FLUSH_PAGES_INLINE,
1096
- 0, // 0 = as many as the system cpus
1125
+ 1,
1126
+ pgc_max_evictors(),
1127
+ 1000,
1128
+ 1,
1129
+ PGC_OPTIONS_AUTOSCALE, // flushing inline: all dirty pages are just converted to clean
1130
+ 0,
1131
sizeof(struct extent_io_data)
1132
);
1133
pgc_set_dynamic_target_cache_size_callback(open_cache, dynamic_open_cache_size);
1134
1135
extent_cache = pgc_create(
1102
- "extent_cache",
1136
+ "EXTENT_PGC",
1137
extent_cache_size,
1138
extent_cache_free_clean_page_callback,
1105
- 1,
1139
+ 2,
1140
NULL,
1141
extent_cache_flush_dirty_page_callback,
1108
- 5,
1109
- 10, // it will lose up to that extents at once!
1110
- 100, //
1111
- 2, // don't delay too much other threads
1112
- PGC_OPTIONS_AUTOSCALE | PGC_OPTIONS_EVICT_PAGES_INLINE | PGC_OPTIONS_FLUSH_PAGES_INLINE,
1113
- 0, // 0 = as many as the system cpus
1142
+ 1,
1143
+ pgc_max_evictors(),
1144
+ 1000,
1145
+ 1,
1146
+ PGC_OPTIONS_AUTOSCALE | PGC_OPTIONS_FLUSH_PAGES_NO_INLINE, // no flushing needed
1147
+ 0,
1148
0
1149
);
1150
pgc_set_dynamic_target_cache_size_callback(extent_cache, dynamic_extent_cache_size);
src/database/engine/pdc.c
+18
-8
@@ -53,10 +53,12 @@ void pdc_init(void) {
53
"dbengine-pdc",
54
sizeof(PDC),
55
0,
56
- 65536,
56
+ 0,
57
NULL,
58
NULL, NULL, false, false
59
);
60
+
61
+ telemetry_aral_register(pdc_globals.pdc.ar, "pdc");
62
}
63
64
PDC *pdc_get(void) {
@@ -81,10 +83,11 @@ void page_details_init(void) {
83
"dbengine-pd",
84
sizeof(struct page_details),
85
0,
84
- 65536,
86
+ 0,
87
NULL,
88
NULL, NULL, false, false
89
);
90
+ telemetry_aral_register(pdc_globals.pd.ar, "pd");
91
}
92
93
struct page_details *page_details_get(void) {
@@ -109,10 +112,11 @@ void epdl_init(void) {
112
"dbengine-epdl",
113
sizeof(EPDL),
114
0,
112
- 65536,
115
+ 0,
116
NULL,
117
NULL, NULL, false, false
118
);
119
+ telemetry_aral_register(pdc_globals.epdl.ar, "epdl");
120
}
121
122
static EPDL *epdl_get(void) {
@@ -137,10 +141,12 @@ void deol_init(void) {
141
"dbengine-deol",
142
sizeof(DEOL),
143
0,
140
- 65536,
144
+ 0,
145
NULL,
146
NULL, NULL, false, false
147
);
148
+
149
+ telemetry_aral_register(pdc_globals.deol.ar, "deol");
150
}
151
152
static DEOL *deol_get(void) {
@@ -1126,6 +1132,7 @@ static bool epdl_populate_pages_from_extent_data(
1132
PGC_PAGE *page = pgc_page_add_and_acquire(main_cache, page_entry, &added);
1133
if (false == added) {
1134
pgd_free(pgd);
1135
+ pgd = pgc_page_data(page);
1136
stats_cache_hit_while_inserting++;
1137
stats_data_from_main_cache++;
1138
}
@@ -1256,9 +1263,12 @@ void epdl_find_extent_and_populate_pages(struct rrdengine_instance *ctx, EPDL *e
1263
void *extent_data = datafile_extent_read(ctx, epdl->file, epdl->extent_offset, epdl->extent_size);
1264
if(extent_data != NULL) {
1265
1259
- void *copied_extent_compressed_data = dbengine_extent_alloc(epdl->extent_size);
1260
- memcpy(copied_extent_compressed_data, extent_data, epdl->extent_size);
1266
+#if defined(NETDATA_TRACE_ALLOCATIONS)
1267
+ void *tmp = dbengine_extent_alloc(epdl->extent_size);
1268
+ memcpy(tmp, extent_data, epdl->extent_size);
1269
datafile_extent_read_free(extent_data);
1270
+ extent_data = tmp;
1271
+#endif
1272
1273
if(worker)
1274
worker_is_busy(UV_EVENT_DBENGINE_EXTENT_CACHE_LOOKUP);
@@ -1272,11 +1282,11 @@ void epdl_find_extent_and_populate_pages(struct rrdengine_instance *ctx, EPDL *e
1282
.size = epdl->extent_size,
1283
.end_time_s = 0,
1284
.update_every_s = 0,
1275
- .data = copied_extent_compressed_data,
1285
+ .data = extent_data,
1286
}, &added);
1287
1288
if (!added) {
1279
- dbengine_extent_free(copied_extent_compressed_data, epdl->extent_size);
1289
+ dbengine_extent_free(extent_data, epdl->extent_size);
1290
internal_fatal(epdl->extent_size != pgc_page_data_size(extent_cache, extent_cache_page),
1291
"DBENGINE: cache size does not match the expected size");
1292
}
src/database/engine/rrdengine.c
+98
-33
@@ -45,7 +45,9 @@ struct rrdeng_main {
45
bool shutdown;
46
47
size_t flushes_running;
48
- size_t evictions_running;
48
+ size_t evict_main_running;
49
+ size_t evict_open_running;
50
+ size_t evict_extent_running;
51
size_t cleanup_running;
52
53
struct {
@@ -86,8 +88,9 @@ struct rrdeng_main {
88
.loop = {},
89
.async = {},
90
.timer = {},
91
+ .retention_timer = {},
92
.flushes_running = 0,
90
- .evictions_running = 0,
93
+ .evict_main_running = 0,
94
.cleanup_running = 0,
95
96
.cmd_queue = {
@@ -138,12 +141,15 @@ struct rrdeng_work {
141
142
static void work_request_init(void) {
143
rrdeng_main.work_cmd.ar = aral_create(
141
- "dbengine-work-cmd",
142
- sizeof(struct rrdeng_work),
143
- 0,
144
- 65536, NULL,
145
- NULL, NULL, false, false
144
+ "dbengine-work-cmd",
145
+ sizeof(struct rrdeng_work),
146
+ 0,
147
+ 0,
148
+ NULL,
149
+ NULL, NULL, false, false
150
);
151
+
152
+ telemetry_aral_register(rrdeng_main.work_cmd.ar, "workers");
153
}
154
155
enum LIBUV_WORKERS_STATUS {
@@ -259,9 +265,11 @@ void page_descriptors_init(void) {
265
"dbengine-descriptors",
266
sizeof(struct page_descr_with_data),
267
0,
262
- 65536 * 4,
268
+ 0,
269
NULL,
270
NULL, NULL, false, false);
271
+
272
+ telemetry_aral_register(rrdeng_main.xt_io_descr.ar, "descriptors");
273
}
274
275
struct page_descr_with_data *page_descriptor_get(void) {
@@ -282,10 +290,12 @@ static void extent_io_descriptor_init(void) {
290
"dbengine-extent-io",
291
sizeof(struct extent_io_descriptor),
292
0,
285
- 65536,
293
+ 0,
294
NULL,
295
NULL, NULL, false, false
296
);
297
+
298
+ telemetry_aral_register(rrdeng_main.xt_io_descr.ar, "extent io");
299
}
300
301
static struct extent_io_descriptor *extent_io_descriptor_get(void) {
@@ -306,9 +316,11 @@ void rrdeng_query_handle_init(void) {
316
"dbengine-query-handles",
317
sizeof(struct rrdeng_query_handle),
318
0,
309
- 65536,
319
+ 0,
320
NULL,
321
NULL, NULL, false, false);
322
+
323
+ telemetry_aral_register(rrdeng_main.handles.ar, "query handles");
324
}
325
326
struct rrdeng_query_handle *rrdeng_query_handle_get(void) {
@@ -426,9 +438,11 @@ static void rrdeng_cmd_queue_init(void) {
438
rrdeng_main.cmd_queue.ar = aral_create("dbengine-opcodes",
439
sizeof(struct rrdeng_cmd),
440
0,
429
- 65536,
441
+ 0,
442
NULL,
443
NULL, NULL, false, false);
444
+
445
+ telemetry_aral_register(rrdeng_main.cmd_queue.ar, "opcodes");
446
}
447
448
static inline STORAGE_PRIORITY rrdeng_enq_cmd_map_opcode_to_priority(enum rrdeng_opcode opcode, STORAGE_PRIORITY priority) {
@@ -1376,17 +1390,41 @@ static void *cache_flush_tp_worker(struct rrdengine_instance *ctx __maybe_unused
1390
return data;
1391
1392
worker_is_busy(UV_EVENT_DBENGINE_FLUSH_MAIN_CACHE);
1379
- pgc_flush_pages(main_cache, 0);
1393
+ while (pgc_flush_pages(main_cache))
1394
+ yield_the_processor();
1395
1396
return data;
1397
}
1398
1384
-static void *cache_evict_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *req __maybe_unused) {
1399
+static void *cache_evict_main_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *req __maybe_unused) {
1400
if (!main_cache)
1401
return data;
1402
1403
worker_is_busy(UV_EVENT_DBENGINE_EVICT_MAIN_CACHE);
1389
- pgc_evict_pages(main_cache, 0, 0);
1404
+ while (pgc_evict_pages(main_cache, 0, 0))
1405
+ yield_the_processor();
1406
+
1407
+ return data;
1408
+}
1409
+
1410
+static void *cache_evict_open_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *req __maybe_unused) {
1411
+ if (!open_cache)
1412
+ return data;
1413
+
1414
+ worker_is_busy(UV_EVENT_DBENGINE_EVICT_OPEN_CACHE);
1415
+ while (pgc_evict_pages(open_cache, 0, 0))
1416
+ yield_the_processor();
1417
+
1418
+ return data;
1419
+}
1420
+
1421
+static void *cache_evict_extent_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *req __maybe_unused) {
1422
+ if (!extent_cache)
1423
+ return data;
1424
+
1425
+ worker_is_busy(UV_EVENT_DBENGINE_EVICT_EXTENT_CACHE);
1426
+ while (pgc_evict_pages(extent_cache, 0, 0))
1427
+ yield_the_processor();
1428
1429
return data;
1430
}
@@ -1532,8 +1570,16 @@ static void after_do_cache_flush(struct rrdengine_instance *ctx __maybe_unused,
1570
rrdeng_main.flushes_running--;
1571
}
1572
1535
-static void after_do_cache_evict(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1536
- rrdeng_main.evictions_running--;
1573
+static void after_do_main_cache_evict(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1574
+ rrdeng_main.evict_main_running--;
1575
+}
1576
+
1577
+static void after_do_open_cache_evict(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1578
+ rrdeng_main.evict_open_running--;
1579
+}
1580
+
1581
+static void after_do_extent_cache_evict(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1582
+ rrdeng_main.evict_extent_running--;
1583
}
1584
1585
static void after_journal_v2_indexing(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
@@ -1544,6 +1590,7 @@ static void after_journal_v2_indexing(struct rrdengine_instance *ctx __maybe_unu
1590
struct rrdeng_buffer_sizes rrdeng_get_buffer_sizes(void) {
1591
return (struct rrdeng_buffer_sizes) {
1592
.pgc = pgc_aral_overhead() + pgc_aral_structures(),
1593
+ .pgd = pgd_aral_overhead() + pgd_aral_structures(),
1594
.mrg = mrg_aral_overhead() + mrg_aral_structures(),
1595
.opcodes = aral_overhead(rrdeng_main.cmd_queue.ar) + aral_structures(rrdeng_main.cmd_queue.ar),
1596
.handles = aral_overhead(rrdeng_main.handles.ar) + aral_structures(rrdeng_main.handles.ar),
@@ -1642,8 +1689,7 @@ bool rrdeng_ctx_tier_cap_exceeded(struct rrdengine_instance *ctx)
1689
return false;
1690
}
1691
1645
-void retention_timer_cb(uv_timer_t *handle)
1646
-{
1692
+static void retention_timer_cb(uv_timer_t *handle) {
1693
if (!localhost)
1694
return;
1695
@@ -1663,7 +1709,7 @@ void retention_timer_cb(uv_timer_t *handle)
1709
worker_is_idle();
1710
}
1711
1666
-void timer_cb(uv_timer_t* handle) {
1712
+static void timer_per_sec_cb(uv_timer_t* handle) {
1713
worker_is_busy(RRDENG_TIMER_CB);
1714
uv_stop(handle->loop);
1715
uv_update_time(handle->loop);
@@ -1672,14 +1718,17 @@ void timer_cb(uv_timer_t* handle) {
1718
worker_set_metric(RRDENG_WORKS_DISPATCHED, (NETDATA_DOUBLE)__atomic_load_n(&rrdeng_main.work_cmd.atomics.dispatched, __ATOMIC_RELAXED));
1719
worker_set_metric(RRDENG_WORKS_EXECUTING, (NETDATA_DOUBLE)__atomic_load_n(&rrdeng_main.work_cmd.atomics.executing, __ATOMIC_RELAXED));
1720
1675
- rrdeng_enq_cmd(NULL, RRDENG_OPCODE_FLUSH_INIT, NULL, NULL, STORAGE_PRIORITY_INTERNAL_DBENGINE, NULL, NULL);
1676
- rrdeng_enq_cmd(NULL, RRDENG_OPCODE_EVICT_INIT, NULL, NULL, STORAGE_PRIORITY_INTERNAL_DBENGINE, NULL, NULL);
1721
+ // rrdeng_enq_cmd(NULL, RRDENG_OPCODE_EVICT_MAIN, NULL, NULL, STORAGE_PRIORITY_INTERNAL_DBENGINE, NULL, NULL);
1722
+ // rrdeng_enq_cmd(NULL, RRDENG_OPCODE_EVICT_OPEN, NULL, NULL, STORAGE_PRIORITY_INTERNAL_DBENGINE, NULL, NULL);
1723
+ // rrdeng_enq_cmd(NULL, RRDENG_OPCODE_EVICT_EXTENT, NULL, NULL, STORAGE_PRIORITY_INTERNAL_DBENGINE, NULL, NULL);
1724
+ rrdeng_enq_cmd(NULL, RRDENG_OPCODE_FLUSH_MAIN, NULL, NULL, STORAGE_PRIORITY_INTERNAL_DBENGINE, NULL, NULL);
1725
rrdeng_enq_cmd(NULL, RRDENG_OPCODE_CLEANUP, NULL, NULL, STORAGE_PRIORITY_INTERNAL_DBENGINE, NULL, NULL);
1726
1727
worker_is_idle();
1728
}
1729
1730
static void dbengine_initialize_structures(void) {
1731
+ pgd_init_arals();
1732
pgc_and_mrg_initialize();
1733
1734
pdc_init();
@@ -1691,7 +1740,6 @@ static void dbengine_initialize_structures(void) {
1740
rrdeng_query_handle_init();
1741
page_descriptors_init();
1742
extent_buffer_init();
1694
- pgd_init_arals();
1743
extent_io_descriptor_init();
1744
}
1745
@@ -1900,8 +1948,8 @@ void dbengine_event_loop(void* arg) {
1948
worker_register_job_name(RRDENG_OPCODE_FLUSHED_TO_OPEN, "flushed to open");
1949
worker_register_job_name(RRDENG_OPCODE_DATABASE_ROTATE, "db rotate");
1950
worker_register_job_name(RRDENG_OPCODE_JOURNAL_INDEX, "journal index");
1903
- worker_register_job_name(RRDENG_OPCODE_FLUSH_INIT, "flush init");
1904
- worker_register_job_name(RRDENG_OPCODE_EVICT_INIT, "evict init");
1951
+ worker_register_job_name(RRDENG_OPCODE_FLUSH_MAIN, "flush init");
1952
+ worker_register_job_name(RRDENG_OPCODE_EVICT_MAIN, "evict init");
1953
worker_register_job_name(RRDENG_OPCODE_CTX_SHUTDOWN, "ctx shutdown");
1954
worker_register_job_name(RRDENG_OPCODE_CTX_QUIESCE, "ctx quiesce");
1955
worker_register_job_name(RRDENG_OPCODE_SHUTDOWN_EVLOOP, "dbengine shutdown");
@@ -1914,8 +1962,8 @@ void dbengine_event_loop(void* arg) {
1962
worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_FLUSHED_TO_OPEN, "flushed to open cb");
1963
worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_DATABASE_ROTATE, "db rotate cb");
1964
worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_JOURNAL_INDEX, "journal index cb");
1917
- worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_FLUSH_INIT, "flush init cb");
1918
- worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_EVICT_INIT, "evict init cb");
1965
+ worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_FLUSH_MAIN, "flush init cb");
1966
+ worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_EVICT_MAIN, "evict init cb");
1967
worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_CTX_SHUTDOWN, "ctx shutdown cb");
1968
worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_CTX_QUIESCE, "ctx quiesce cb");
1969
@@ -1932,7 +1980,7 @@ void dbengine_event_loop(void* arg) {
1980
struct rrdeng_cmd cmd;
1981
main->tid = gettid_cached();
1982
1935
- fatal_assert(0 == uv_timer_start(&main->timer, timer_cb, TIMER_PERIOD_MS, TIMER_PERIOD_MS));
1983
+ fatal_assert(0 == uv_timer_start(&main->timer, timer_per_sec_cb, TIMER_PERIOD_MS, TIMER_PERIOD_MS));
1984
fatal_assert(0 == uv_timer_start(&main->retention_timer, retention_timer_cb, TIMER_PERIOD_MS * 60, TIMER_PERIOD_MS * 60));
1985
1986
bool shutdown = false;
@@ -1974,18 +2022,34 @@ void dbengine_event_loop(void* arg) {
2022
break;
2023
}
2024
1977
- case RRDENG_OPCODE_FLUSH_INIT: {
1978
- if(rrdeng_main.flushes_running < (size_t)(libuv_worker_threads / 4)) {
2025
+ case RRDENG_OPCODE_FLUSH_MAIN: {
2026
+ if(rrdeng_main.flushes_running < pgc_max_flushers()) {
2027
rrdeng_main.flushes_running++;
2028
work_dispatch(NULL, NULL, NULL, opcode, cache_flush_tp_worker, after_do_cache_flush);
2029
}
2030
break;
2031
}
2032
1985
- case RRDENG_OPCODE_EVICT_INIT: {
1986
- if(!rrdeng_main.evictions_running) {
1987
- rrdeng_main.evictions_running++;
1988
- work_dispatch(NULL, NULL, NULL, opcode, cache_evict_tp_worker, after_do_cache_evict);
2033
+ case RRDENG_OPCODE_EVICT_MAIN: {
2034
+ if(rrdeng_main.evict_main_running < pgc_max_evictors()) {
2035
+ rrdeng_main.evict_main_running++;
2036
+ work_dispatch(NULL, NULL, NULL, opcode, cache_evict_main_tp_worker, after_do_main_cache_evict);
2037
+ }
2038
+ break;
2039
+ }
2040
+
2041
+ case RRDENG_OPCODE_EVICT_OPEN: {
2042
+ if(rrdeng_main.evict_open_running < pgc_max_evictors()) {
2043
+ rrdeng_main.evict_open_running++;
2044
+ work_dispatch(NULL, NULL, NULL, opcode, cache_evict_open_tp_worker, after_do_open_cache_evict);
2045
+ }
2046
+ break;
2047
+ }
2048
+
2049
+ case RRDENG_OPCODE_EVICT_EXTENT: {
2050
+ if(rrdeng_main.evict_extent_running < pgc_max_evictors()) {
2051
+ rrdeng_main.evict_extent_running++;
2052
+ work_dispatch(NULL, NULL, NULL, opcode, cache_evict_extent_tp_worker, after_do_extent_cache_evict);
2053
}
2054
break;
2055
}
@@ -2049,6 +2113,7 @@ void dbengine_event_loop(void* arg) {
2113
2114
case RRDENG_OPCODE_SHUTDOWN_EVLOOP: {
2115
uv_close((uv_handle_t *)&main->async, NULL);
2116
+
2117
(void) uv_timer_stop(&main->timer);
2118
uv_close((uv_handle_t *)&main->timer, NULL);
2119
src/database/engine/rrdengine.h
+4
-3
@@ -201,7 +201,6 @@ struct rrdeng_collect_handle {
201
struct metric *metric;
202
struct pgc_page *pgc_page;
203
struct pgd *page_data;
204
- size_t page_data_size;
204
struct pg_alignment *alignment;
205
uint32_t page_entries_max;
206
uint32_t page_position; // keep track of the current page size, to make sure we don't exceed it
@@ -249,8 +248,10 @@ enum rrdeng_opcode {
248
RRDENG_OPCODE_FLUSHED_TO_OPEN,
249
RRDENG_OPCODE_DATABASE_ROTATE,
250
RRDENG_OPCODE_JOURNAL_INDEX,
252
- RRDENG_OPCODE_FLUSH_INIT,
253
- RRDENG_OPCODE_EVICT_INIT,
251
+ RRDENG_OPCODE_FLUSH_MAIN,
252
+ RRDENG_OPCODE_EVICT_MAIN,
253
+ RRDENG_OPCODE_EVICT_OPEN,
254
+ RRDENG_OPCODE_EVICT_EXTENT,
255
RRDENG_OPCODE_CTX_SHUTDOWN,
256
RRDENG_OPCODE_CTX_QUIESCE,
257
RRDENG_OPCODE_CTX_POPULATE_MRG,
src/database/engine/rrdengineapi.c
+16
-22
@@ -59,6 +59,8 @@ __attribute__((constructor)) void initialize_multidb_ctx(void) {
59
initialize_single_ctx(multidb_ctx[i]);
60
}
61
62
+uint64_t dbengine_out_of_memory_protection = 0;
63
+bool dbengine_use_all_ram_for_caches = false;
64
int db_engine_journal_check = 0;
65
bool new_dbengine_defaults = false;
66
bool legacy_multihost_db_space = false;
@@ -279,8 +281,7 @@ STORAGE_COLLECT_HANDLE *rrdeng_store_metric_init(STORAGE_METRIC_HANDLE *smh, uin
281
282
handle->pgc_page = NULL;
283
handle->page_data = NULL;
282
- handle->page_data_size = 0;
283
-
284
+
285
handle->page_position = 0;
286
handle->page_entries_max = 0;
287
handle->update_every_ut = (usec_t)update_every * USEC_PER_SEC;
@@ -339,7 +340,6 @@ void rrdeng_store_metric_flush_current_page(STORAGE_COLLECT_HANDLE *sch) {
340
handle->page_position = 0;
341
handle->page_entries_max = 0;
342
handle->page_data = NULL;
342
- handle->page_data_size = 0;
343
344
// important!
345
// we should never zero page end time ut, because this will allow
@@ -355,8 +355,7 @@ void rrdeng_store_metric_flush_current_page(STORAGE_COLLECT_HANDLE *sch) {
355
static void rrdeng_store_metric_create_new_page(struct rrdeng_collect_handle *handle,
356
struct rrdengine_instance *ctx,
357
usec_t point_in_time_ut,
358
- PGD *data,
359
- size_t data_size) {
358
+ PGD *data) {
359
time_t point_in_time_s = (time_t)(point_in_time_ut / USEC_PER_SEC);
360
const uint32_t update_every_s = (uint32_t)(handle->update_every_ut / USEC_PER_SEC);
361
@@ -365,7 +364,7 @@ static void rrdeng_store_metric_create_new_page(struct rrdeng_collect_handle *ha
364
.metric_id = mrg_metric_id(main_mrg, handle->metric),
365
.start_time_s = point_in_time_s,
366
.end_time_s = point_in_time_s,
368
- .size = data_size,
367
+ .size = pgd_memory_footprint(data),
368
.data = data,
369
.update_every_s = update_every_s,
370
.hot = true
@@ -405,7 +404,7 @@ static void rrdeng_store_metric_create_new_page(struct rrdeng_collect_handle *ha
404
pgc_page = pgc_page_add_and_acquire(main_cache, page_entry, &added);
405
}
406
408
- handle->page_entries_max = data_size / CTX_POINT_SIZE_BYTES(ctx);
407
+ handle->page_entries_max = pgd_capacity(data);
408
handle->page_start_time_ut = point_in_time_ut;
409
handle->page_end_time_ut = point_in_time_ut;
410
handle->page_position = 1; // zero is already in our data
@@ -436,7 +435,7 @@ static size_t aligned_allocation_entries(size_t max_slots, size_t target_slot, t
435
return slots;
436
}
437
439
-static PGD *rrdeng_alloc_new_page_data(struct rrdeng_collect_handle *handle, size_t *data_size, usec_t point_in_time_ut) {
438
+static PGD *rrdeng_alloc_new_page_data(struct rrdeng_collect_handle *handle, usec_t point_in_time_ut) {
439
struct rrdengine_instance *ctx = mrg_metric_ctx(handle->metric);
440
441
PGD *d = NULL;
@@ -463,17 +462,11 @@ static PGD *rrdeng_alloc_new_page_data(struct rrdeng_collect_handle *handle, siz
462
internal_fatal(slots < 3 || slots > max_slots, "ooops! wrong distribution of metrics across time");
463
internal_fatal(size > tier_page_size[ctx->config.tier] || size < CTX_POINT_SIZE_BYTES(ctx) * 2, "ooops! wrong page size");
464
466
- *data_size = size;
467
-
465
switch (ctx->config.page_type) {
466
case RRDENG_PAGE_TYPE_ARRAY_32BIT:
467
case RRDENG_PAGE_TYPE_ARRAY_TIER1:
471
- d = pgd_create(ctx->config.page_type, slots);
472
- break;
468
case RRDENG_PAGE_TYPE_GORILLA_32BIT:
474
- // ignore slots, and use the fixed number of slots per gorilla buffer.
475
- // gorilla will automatically add more buffers if needed.
476
- d = pgd_create(ctx->config.page_type, RRDENG_GORILLA_32BIT_BUFFER_SLOTS);
469
+ d = pgd_create(ctx->config.page_type, slots);
470
break;
471
default:
472
fatal("Unknown page type: %uc\n", ctx->config.page_type);
@@ -496,24 +489,25 @@ static void rrdeng_store_metric_append_point(STORAGE_COLLECT_HANDLE *sch,
489
struct rrdengine_instance *ctx = mrg_metric_ctx(handle->metric);
490
491
if(unlikely(!handle->page_data))
499
- handle->page_data = rrdeng_alloc_new_page_data(handle, &handle->page_data_size, point_in_time_ut);
492
+ handle->page_data = rrdeng_alloc_new_page_data(handle, point_in_time_ut);
493
494
timing_step(TIMING_STEP_DBENGINE_CHECK_DATA);
495
503
- pgd_append_point(handle->page_data,
504
- point_in_time_ut,
505
- n, min_value, max_value, count, anomaly_count, flags,
506
- handle->page_position);
496
+ size_t additional_bytes = pgd_append_point(handle->page_data,
497
+ point_in_time_ut,
498
+ n, min_value, max_value, count, anomaly_count, flags,
499
+ handle->page_position);
500
501
timing_step(TIMING_STEP_DBENGINE_PACK);
502
503
if(unlikely(!handle->pgc_page)) {
511
- rrdeng_store_metric_create_new_page(handle, ctx, point_in_time_ut, handle->page_data, handle->page_data_size);
504
+ rrdeng_store_metric_create_new_page(handle, ctx, point_in_time_ut, handle->page_data);
505
// handle->position is set to 1 already
506
}
507
else {
508
// update an existing page
516
- pgc_page_hot_set_end_time_s(main_cache, handle->pgc_page, (time_t) (point_in_time_ut / USEC_PER_SEC));
509
+ pgc_page_hot_set_end_time_s(main_cache, handle->pgc_page,
510
+ (time_t) (point_in_time_ut / USEC_PER_SEC), additional_bytes);
511
handle->page_end_time_ut = point_in_time_ut;
512
513
if(unlikely(++handle->page_position >= handle->page_entries_max)) {
src/database/engine/rrdengineapi.h
+4
@@ -13,6 +13,9 @@
13
14
#define RRDENG_FD_BUDGET_PER_INSTANCE (50)
15
16
+extern uint64_t dbengine_out_of_memory_protection;
17
+extern bool dbengine_use_all_ram_for_caches;
18
+
19
extern int default_rrdeng_page_cache_mb;
20
extern int default_rrdeng_extent_cache_mb;
21
extern int db_engine_journal_check;
@@ -218,6 +221,7 @@ struct rrdeng_buffer_sizes {
221
size_t deol;
222
size_t pd;
223
size_t pgc;
224
+ size_t pgd;
225
size_t mrg;
226
#ifdef PDC_USE_JULYL
227
size_t julyl;
src/database/rrd-database-mode.c
new
+34
@@ -0,0 +1,34 @@
1
+
2
+#include "rrd.h"
3
+
4
+inline const char *rrd_memory_mode_name(RRD_MEMORY_MODE id) {
5
+ switch(id) {
6
+ case RRD_MEMORY_MODE_RAM:
7
+ return RRD_MEMORY_MODE_RAM_NAME;
8
+
9
+ case RRD_MEMORY_MODE_NONE:
10
+ return RRD_MEMORY_MODE_NONE_NAME;
11
+
12
+ case RRD_MEMORY_MODE_ALLOC:
13
+ return RRD_MEMORY_MODE_ALLOC_NAME;
14
+
15
+ case RRD_MEMORY_MODE_DBENGINE:
16
+ return RRD_MEMORY_MODE_DBENGINE_NAME;
17
+ }
18
+
19
+ STORAGE_ENGINE* eng = storage_engine_get(id);
20
+ if (eng) {
21
+ return eng->name;
22
+ }
23
+
24
+ return RRD_MEMORY_MODE_RAM_NAME;
25
+}
26
+
27
+RRD_MEMORY_MODE rrd_memory_mode_id(const char *name) {
28
+ STORAGE_ENGINE* eng = storage_engine_find(name);
29
+ if (eng) {
30
+ return eng->id;
31
+ }
32
+
33
+ return RRD_MEMORY_MODE_RAM;
34
+}
src/database/rrd-database-mode.h
new
+25
@@ -0,0 +1,25 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_RRD_DATABASE_MODE_H
4
+#define NETDATA_RRD_DATABASE_MODE_H
5
+
6
+typedef enum __attribute__ ((__packed__)) rrd_memory_mode {
7
+ RRD_MEMORY_MODE_NONE = 0,
8
+ RRD_MEMORY_MODE_RAM = 1,
9
+ RRD_MEMORY_MODE_ALLOC = 4,
10
+ RRD_MEMORY_MODE_DBENGINE = 5,
11
+
12
+ // this is 8-bit
13
+} RRD_MEMORY_MODE;
14
+
15
+#define RRD_MEMORY_MODE_NONE_NAME "none"
16
+#define RRD_MEMORY_MODE_RAM_NAME "ram"
17
+#define RRD_MEMORY_MODE_ALLOC_NAME "alloc"
18
+#define RRD_MEMORY_MODE_DBENGINE_NAME "dbengine"
19
+
20
+extern RRD_MEMORY_MODE default_rrd_memory_mode;
21
+
22
+const char *rrd_memory_mode_name(RRD_MEMORY_MODE id);
23
+RRD_MEMORY_MODE rrd_memory_mode_id(const char *name);
24
+
25
+#endif //NETDATA_RRD_DATABASE_MODE_H
src/database/rrd.c
+6
-38
@@ -14,8 +14,6 @@
14
int rrd_delete_unupdated_dimensions = 0;
15
*/
16
17
-int default_rrd_update_every = UPDATE_EVERY;
18
-int default_rrd_history_entries = RRD_DEFAULT_HISTORY_ENTRIES;
17
#ifdef ENABLE_DBENGINE
18
RRD_MEMORY_MODE default_rrd_memory_mode = RRD_MEMORY_MODE_DBENGINE;
19
#else
@@ -24,42 +22,6 @@ RRD_MEMORY_MODE default_rrd_memory_mode = RRD_MEMORY_MODE_RAM;
22
int gap_when_lost_iterations_above = 1;
23
24
27
-// ----------------------------------------------------------------------------
28
-// RRD - memory modes
29
-
30
-inline const char *rrd_memory_mode_name(RRD_MEMORY_MODE id) {
31
- switch(id) {
32
- case RRD_MEMORY_MODE_RAM:
33
- return RRD_MEMORY_MODE_RAM_NAME;
34
-
35
- case RRD_MEMORY_MODE_NONE:
36
- return RRD_MEMORY_MODE_NONE_NAME;
37
-
38
- case RRD_MEMORY_MODE_ALLOC:
39
- return RRD_MEMORY_MODE_ALLOC_NAME;
40
-
41
- case RRD_MEMORY_MODE_DBENGINE:
42
- return RRD_MEMORY_MODE_DBENGINE_NAME;
43
- }
44
-
45
- STORAGE_ENGINE* eng = storage_engine_get(id);
46
- if (eng) {
47
- return eng->name;
48
- }
49
-
50
- return RRD_MEMORY_MODE_RAM_NAME;
51
-}
52
-
53
-RRD_MEMORY_MODE rrd_memory_mode_id(const char *name) {
54
- STORAGE_ENGINE* eng = storage_engine_find(name);
55
- if (eng) {
56
- return eng->id;
57
- }
58
-
59
- return RRD_MEMORY_MODE_RAM;
60
-}
61
-
62
-
25
// ----------------------------------------------------------------------------
26
// RRD - algorithms types
27
@@ -108,6 +70,9 @@ inline RRDSET_TYPE rrdset_type_id(const char *name) {
70
else if(unlikely(strcmp(name, RRDSET_TYPE_STACKED_NAME) == 0))
71
return RRDSET_TYPE_STACKED;
72
73
+ else if(unlikely(strcmp(name, RRDSET_TYPE_HEATMAP_NAME) == 0))
74
+ return RRDSET_TYPE_HEATMAP;
75
+
76
else // if(unlikely(strcmp(name, RRDSET_TYPE_LINE_NAME) == 0))
77
return RRDSET_TYPE_LINE;
78
}
@@ -123,6 +88,9 @@ const char *rrdset_type_name(RRDSET_TYPE chart_type) {
88
89
case RRDSET_TYPE_STACKED:
90
return RRDSET_TYPE_STACKED_NAME;
91
+
92
+ case RRDSET_TYPE_HEATMAP:
93
+ return RRDSET_TYPE_HEATMAP_NAME;
94
}
95
}
96
src/database/rrd.h
+147
-108
@@ -8,6 +8,9 @@ extern "C" {
8
#endif
9
10
#include "libnetdata/libnetdata.h"
11
+#include "rrd-database-mode.h"
12
+#include "streaming/stream-traffic-types.h"
13
+#include "streaming/stream-sender-commit.h"
14
15
// non-existing structs instead of voids
16
// to enable type checking at compile time
@@ -69,26 +72,6 @@ struct rrdengine_instance;
72
#endif
73
74
// ----------------------------------------------------------------------------
72
-// memory mode
73
-
74
-typedef enum __attribute__ ((__packed__)) rrd_memory_mode {
75
- RRD_MEMORY_MODE_NONE = 0,
76
- RRD_MEMORY_MODE_RAM = 1,
77
- RRD_MEMORY_MODE_ALLOC = 4,
78
- RRD_MEMORY_MODE_DBENGINE = 5,
79
-
80
- // this is 8-bit
81
-} RRD_MEMORY_MODE;
82
-
83
-#define RRD_MEMORY_MODE_NONE_NAME "none"
84
-#define RRD_MEMORY_MODE_RAM_NAME "ram"
85
-#define RRD_MEMORY_MODE_ALLOC_NAME "alloc"
86
-#define RRD_MEMORY_MODE_DBENGINE_NAME "dbengine"
87
-
88
-extern RRD_MEMORY_MODE default_rrd_memory_mode;
89
-
90
-const char *rrd_memory_mode_name(RRD_MEMORY_MODE id);
91
-RRD_MEMORY_MODE rrd_memory_mode_id(const char *name);
75
76
struct ml_metrics_statistics {
77
size_t anomalous;
@@ -145,7 +128,7 @@ typedef enum __attribute__ ((__packed__)) rrdset_flags {
128
#include "rrdlabels.h"
129
#include "streaming/stream-capabilities.h"
130
#include "streaming/stream-path.h"
148
-#include "streaming/rrdpush.h"
131
+#include "streaming/stream.h"
132
//#include "aclk/aclk_rrdhost_state.h"
133
#include "sqlite/sqlite_health.h"
134
@@ -174,22 +157,19 @@ typedef enum __attribute__ ((__packed__)) rrdset_type {
157
RRDSET_TYPE_LINE = 0,
158
RRDSET_TYPE_AREA = 1,
159
RRDSET_TYPE_STACKED = 2,
160
+ RRDSET_TYPE_HEATMAP = 3,
161
} RRDSET_TYPE;
162
163
#define RRDSET_TYPE_LINE_NAME "line"
164
#define RRDSET_TYPE_AREA_NAME "area"
165
#define RRDSET_TYPE_STACKED_NAME "stacked"
166
+#define RRDSET_TYPE_HEATMAP_NAME "heatmap"
167
168
RRDSET_TYPE rrdset_type_id(const char *name);
169
const char *rrdset_type_name(RRDSET_TYPE chart_type);
170
171
#include "contexts/rrdcontext.h"
172
188
-extern bool dbengine_enabled;
189
-extern size_t storage_tiers;
190
-extern bool use_direct_io;
191
-extern size_t storage_tiers_grouping_iterations[RRD_STORAGE_TIERS];
192
-
173
typedef enum __attribute__ ((__packed__)) {
174
RRD_BACKFILL_NONE = 0,
175
RRD_BACKFILL_FULL,
@@ -202,11 +182,6 @@ typedef enum __attribute__ ((__packed__)) {
182
#define RRD_DEFAULT_HISTORY_ENTRIES 3600
183
#define RRD_HISTORY_ENTRIES_MAX (86400*365)
184
205
-extern int default_rrd_update_every;
206
-extern int default_rrd_history_entries;
207
-extern int gap_when_lost_iterations_above;
208
-extern time_t rrdset_free_obsolete_time_s;
209
-
185
#if defined(ENV32BIT)
186
#define MIN_LIBUV_WORKER_THREADS 8
187
#define MAX_LIBUV_WORKER_THREADS 128
@@ -358,8 +333,8 @@ struct rrddim {
333
struct {
334
uint32_t sent_version;
335
uint32_t dim_slot;
361
- } sender;
362
- } rrdpush;
336
+ } snd;
337
+ } stream;
338
339
// ------------------------------------------------------------------------
340
// data collection members
@@ -797,8 +772,8 @@ struct rrdset {
772
uint32_t dim_last_slot_used;
773
774
time_t resync_time_s; // the timestamp up to which we should resync clock upstream
800
- } sender;
801
- } rrdpush;
775
+ } snd;
776
+ } stream;
777
778
// ------------------------------------------------------------------------
779
// db mode SAVE, MAP specifics
@@ -869,13 +844,13 @@ static inline uint32_t rrdset_metadata_version(RRDSET *st) {
844
}
845
846
static inline uint32_t rrdset_metadata_upstream_version(RRDSET *st) {
872
- return __atomic_load_n(&st->rrdpush.sender.sent_version, __ATOMIC_RELAXED);
847
+ return __atomic_load_n(&st->stream.snd.sent_version, __ATOMIC_RELAXED);
848
}
849
850
void rrdset_metadata_updated(RRDSET *st);
851
852
static inline void rrdset_metadata_exposed_upstream(RRDSET *st, uint32_t version) {
878
- __atomic_store_n(&st->rrdpush.sender.sent_version, version, __ATOMIC_RELAXED);
853
+ __atomic_store_n(&st->stream.snd.sent_version, version, __ATOMIC_RELAXED);
854
}
855
856
static inline bool rrdset_check_upstream_exposed(RRDSET *st) {
@@ -888,17 +863,17 @@ static inline uint32_t rrddim_metadata_version(RRDDIM *rd) {
863
}
864
865
static inline uint32_t rrddim_metadata_upstream_version(RRDDIM *rd) {
891
- return __atomic_load_n(&rd->rrdpush.sender.sent_version, __ATOMIC_RELAXED);
866
+ return __atomic_load_n(&rd->stream.snd.sent_version, __ATOMIC_RELAXED);
867
}
868
869
void rrddim_metadata_updated(RRDDIM *rd);
870
871
static inline void rrddim_metadata_exposed_upstream(RRDDIM *rd, uint32_t version) {
897
- __atomic_store_n(&rd->rrdpush.sender.sent_version, version, __ATOMIC_RELAXED);
872
+ __atomic_store_n(&rd->stream.snd.sent_version, version, __ATOMIC_RELAXED);
873
}
874
875
static inline void rrddim_metadata_exposed_upstream_clear(RRDDIM *rd) {
901
- __atomic_store_n(&rd->rrdpush.sender.sent_version, 0, __ATOMIC_RELAXED);
876
+ __atomic_store_n(&rd->stream.snd.sent_version, 0, __ATOMIC_RELAXED);
877
}
878
879
static inline bool rrddim_check_upstream_exposed(RRDDIM *rd) {
@@ -909,7 +884,7 @@ static inline bool rrddim_check_upstream_exposed(RRDDIM *rd) {
884
// still, it can be removed, after the collector has finished
885
// so, it is safe to check it without atomics
886
static inline bool rrddim_check_upstream_exposed_collector(RRDDIM *rd) {
912
- return rd->rrdset->version == rd->rrdpush.sender.sent_version;
887
+ return rd->rrdset->version == rd->stream.snd.sent_version;
888
}
889
890
STRING *rrd_string_strdupz(const char *s);
@@ -952,16 +927,15 @@ typedef enum __attribute__ ((__packed__)) rrdhost_flags {
927
RRDHOST_FLAG_PENDING_OBSOLETE_DIMENSIONS = (1 << 11), // the host has pending dimension obsoletions
928
929
// Streaming sender
955
- RRDHOST_FLAG_RRDPUSH_SENDER_INITIALIZED = (1 << 12), // the host has initialized rrdpush structures
956
- RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN = (1 << 13), // When set, the sender thread is running
957
- RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED = (1 << 14), // When set, the host is connected to a parent
958
- RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS = (1 << 15), // when set, rrdset_done() should push metrics to parent
959
- RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS = (1 << 16), // when set, we have logged the status of metrics streaming
930
+ RRDHOST_FLAG_STREAM_SENDER_INITIALIZED = (1 << 12), // the host has initialized streaming sender structures
931
+ RRDHOST_FLAG_STREAM_SENDER_ADDED = (1 << 13), // When set, the sender thread is running
932
+ RRDHOST_FLAG_STREAM_SENDER_CONNECTED = (1 << 14), // When set, the host is connected to a parent
933
+ RRDHOST_FLAG_STREAM_SENDER_READY_4_METRICS = (1 << 15), // when set, rrdset_done() should push metrics to parent
934
+ RRDHOST_FLAG_STREAM_SENDER_LOGGED_STATUS = (1 << 16), // when set, we have logged the status of metrics streaming
935
936
// Health
937
RRDHOST_FLAG_PENDING_HEALTH_INITIALIZATION = (1 << 17), // contains charts and dims with uninitialized variables
938
RRDHOST_FLAG_INITIALIZED_HEALTH = (1 << 18), // the host has initialized health structures
964
-
939
// Exporting
940
RRDHOST_FLAG_EXPORTING_SEND = (1 << 19), // send it to external databases
941
RRDHOST_FLAG_EXPORTING_DONT_SEND = (1 << 20), // don't send it to external databases
@@ -977,7 +951,7 @@ typedef enum __attribute__ ((__packed__)) rrdhost_flags {
951
RRDHOST_FLAG_PENDING_CONTEXT_LOAD = (1 << 26), // Context needs to be loaded
952
953
RRDHOST_FLAG_METADATA_CLAIMID = (1 << 27), // metadata needs to be stored in the database
980
- RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED = (1 << 28), // set when the receiver part is disconnected
954
+ RRDHOST_FLAG_STREAM_RECEIVER_DISCONNECTED = (1 << 28), // set when the receiver part is disconnected
955
956
RRDHOST_FLAG_GLOBAL_FUNCTIONS_UPDATED = (1 << 29), // set when the host has updated global functions
957
} RRDHOST_FLAGS;
@@ -1011,9 +985,13 @@ typedef enum __attribute__ ((__packed__)) {
985
#define rrdhost_option_set(host, flag) (host)->options |= flag
986
#define rrdhost_option_clear(host, flag) (host)->options &= ~(flag)
987
1014
-#define rrdhost_has_rrdpush_sender_enabled(host) (rrdhost_option_check(host, RRDHOST_OPTION_SENDER_ENABLED) && (host)->sender)
988
+#define rrdhost_has_stream_sender_enabled(host) (rrdhost_option_check(host, RRDHOST_OPTION_SENDER_ENABLED) && (host)->sender)
989
1016
-#define rrdhost_can_send_definitions_to_parent(host) (rrdhost_has_rrdpush_sender_enabled(host) && rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED))
990
+#define rrdhost_can_stream_metadata_to_parent(host) \
991
+ (rrdhost_has_stream_sender_enabled(host) && \
992
+ rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_SENDER_READY_4_METRICS) && \
993
+ !rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_RECEIVER_DISCONNECTED) \
994
+ )
995
996
// ----------------------------------------------------------------------------
997
// Health data
@@ -1059,6 +1037,7 @@ struct alarm_entry {
1037
RRDCALC_STATUS new_status;
1038
1039
uint32_t flags;
1040
+ int32_t pending_save_count;
1041
1042
int delay;
1043
time_t delay_up_to_timestamp;
@@ -1100,11 +1079,12 @@ typedef struct alarm_log {
1079
} ALARM_LOG;
1080
1081
typedef struct health {
1103
- time_t health_delay_up_to; // a timestamp to delay alarms processing up to
1104
- STRING *health_default_exec; // the full path of the alarms notifications program
1105
- STRING *health_default_recipient; // the default recipient for all alarms
1106
- bool health_enabled; // 1 when this host has health enabled
1107
- bool use_summary_for_notifications; // whether to use the summary field as a subject for notifications
1082
+ time_t delay_up_to; // a timestamp to delay alarms processing up to
1083
+ STRING *default_exec; // the full path of the alarms notifications program
1084
+ STRING *default_recipient; // the default recipient for all alarms
1085
+ bool enabled; // 1 when this host has health enabled
1086
+ bool use_summary_for_notifications; // whether to use the summary field as a subject for notifications
1087
+ int32_t pending_transitions; // pending alert transitions to store
1088
} HEALTH;
1089
1090
// ----------------------------------------------------------------------------
@@ -1140,7 +1120,7 @@ struct rrdhost_system_info {
1120
char *container;
1121
char *container_detection;
1122
char *is_k8s_node;
1143
- uint16_t hops;
1123
+ int16_t hops;
1124
bool ml_capable;
1125
bool ml_enabled;
1126
char *install_type;
@@ -1151,6 +1131,8 @@ struct rrdhost_system_info {
1131
1132
struct rrdhost_system_info *rrdhost_labels_to_system_info(RRDLABELS *labels);
1133
1134
+struct stream_thread;
1135
+
1136
struct rrdhost {
1137
char machine_guid[GUID_LEN + 1]; // the unique ID of this host
1138
@@ -1189,9 +1171,14 @@ struct rrdhost {
1171
struct rrdhost_system_info *system_info; // information collected from the host environment
1172
1173
// ------------------------------------------------------------------------
1192
- // streaming of data to remote hosts - rrdpush
1174
+ // streaming and replication, configuration and status
1175
1176
struct {
1177
+ struct stream_thread *thread;
1178
+ uint8_t refcount;
1179
+
1180
+ // --- sender ---
1181
+
1182
struct {
1183
struct {
1184
struct {
@@ -1206,10 +1193,27 @@ struct rrdhost {
1193
uint32_t last_used; // the last slot we used for a chart (increments only)
1194
} pluginsd_chart_slots;
1195
1209
- char *destination; // where to send metrics to
1210
- char *api_key; // the api key at the receiving netdata
1196
+ struct {
1197
+ pid_t tid;
1198
+
1199
+ time_t last_connected; // last time child connected (stored in db)
1200
+ uint32_t connections; // the number of times this sender has connected
1201
+
1202
+ struct {
1203
+ size_t charts; // the number of charts currently being replicated to a parent
1204
+ } replication;
1205
+ } status;
1206
+
1207
+ // reserved for the receiver/sender thread - do not use for other purposes
1208
+ struct sender_buffer commit;
1209
+
1210
+ STRING *destination; // where to send metrics to
1211
+ STRING *api_key; // the api key at the receiving netdata
1212
SIMPLE_PATTERN *charts_matching; // pattern to match the charts to be sent
1212
- } send;
1213
+ RRDHOST_STREAM_PARENTS parents; // the list of parents (extracted from destination)
1214
+ } snd;
1215
+
1216
+ // --- receiver ---
1217
1218
struct {
1219
struct {
@@ -1217,43 +1221,46 @@ struct rrdhost {
1221
uint32_t size;
1222
RRDSET **array;
1223
} pluginsd_chart_slots;
1220
- } receive;
1224
1222
- RRDHOST_STREAM_PATH path;
1223
- } rrdpush;
1225
+ struct {
1226
+ pid_t tid;
1227
+
1228
+ time_t last_connected; // the time the last sender was connected
1229
+ time_t last_disconnected; // the time the last sender was disconnected
1230
+ time_t last_chart; // the time of the last CHART streaming command
1231
+ bool check_obsolete; // set when child connects, will instruct parent to
1232
+ // trigger a check for obsoleted charts since previous connect
1233
1225
- struct rrdpush_destinations *destinations; // a linked list of possible destinations
1226
- struct rrdpush_destinations *destination; // the current destination from the above list
1234
+ uint32_t connections; // the number of times this receiver has connected
1235
+ STREAM_HANDSHAKE exit_reason; // the last receiver exit reason
1236
1228
- int32_t rrdpush_last_receiver_exit_reason;
1229
- time_t rrdpush_seconds_to_replicate; // max time we want to replicate from the child
1230
- time_t rrdpush_replication_step; // seconds per replication step
1231
- size_t rrdpush_receiver_replicating_charts; // the number of charts currently being replicated from a child
1232
- NETDATA_DOUBLE rrdpush_receiver_replication_percent; // the % of replication completion
1237
+ struct {
1238
+ size_t charts; // the number of charts currently being replicated from a child
1239
+ NETDATA_DOUBLE percent; // the % of replication completion
1240
+ } replication;
1241
+ } status;
1242
+ } rcv;
1243
+
1244
+ // --- configuration ---
1245
+
1246
+ struct {
1247
+ time_t period; // max time we want to replicate from the child
1248
+ time_t step; // seconds per replication step
1249
+ } replication;
1250
+
1251
+ RRDHOST_STREAM_PATH path;
1252
+ } stream;
1253
1254
// the following are state information for the threading
1255
// streaming metrics from this netdata to an upstream netdata
1256
struct sender_state *sender;
1237
- ND_THREAD *rrdpush_sender_thread; // the sender thread
1238
- size_t rrdpush_sender_replicating_charts; // the number of charts currently being replicated to a parent
1239
- struct aclk_sync_cfg_t *aclk_config;
1257
1241
- uint32_t rrdpush_receiver_connection_counter; // the number of times this receiver has connected
1242
- uint32_t rrdpush_sender_connection_counter; // the number of times this sender has connected
1258
+ struct receiver_state *receiver;
1259
+ SPINLOCK receiver_lock;
1260
1261
// ------------------------------------------------------------------------
1245
- // streaming of data from remote hosts - rrdpush receiver
1246
-
1247
- time_t last_connected; // last time child connected (stored in db)
1248
- time_t child_connect_time; // the time the last sender was connected
1249
- time_t child_last_chart_command; // the time of the last CHART streaming command
1250
- time_t child_disconnected_time; // the time the last sender was disconnected
1251
- int connected_children_count; // number of senders currently streaming
1262
1253
- struct receiver_state *receiver;
1254
- SPINLOCK receiver_lock;
1255
- int trigger_chart_obsoletion_check; // set when child connects, will instruct parent to
1256
- // trigger a check for obsoleted charts since previous connect
1263
+ struct aclk_sync_cfg_t *aclk_config;
1264
1265
// ------------------------------------------------------------------------
1266
// health monitoring options
@@ -1330,6 +1337,9 @@ struct rrdhost {
1337
};
1338
extern RRDHOST *localhost;
1339
1340
+#define rrdhost_receiver_lock(host) spinlock_lock(&(host)->receiver_lock)
1341
+#define rrdhost_receiver_unlock(host) spinlock_unlock(&(host)->receiver_lock)
1342
+
1343
#define rrdhost_hostname(host) string2str((host)->hostname)
1344
#define rrdhost_registry_hostname(host) string2str((host)->registry_hostname)
1345
#define rrdhost_os(host) string2str((host)->os)
@@ -1338,17 +1348,17 @@ extern RRDHOST *localhost;
1348
#define rrdhost_program_name(host) string2str((host)->program_name)
1349
#define rrdhost_program_version(host) string2str((host)->program_version)
1350
1341
-#define rrdhost_receiver_replicating_charts(host) (__atomic_load_n(&((host)->rrdpush_receiver_replicating_charts), __ATOMIC_RELAXED))
1342
-#define rrdhost_receiver_replicating_charts_plus_one(host) (__atomic_add_fetch(&((host)->rrdpush_receiver_replicating_charts), 1, __ATOMIC_RELAXED))
1343
-#define rrdhost_receiver_replicating_charts_minus_one(host) (__atomic_sub_fetch(&((host)->rrdpush_receiver_replicating_charts), 1, __ATOMIC_RELAXED))
1344
-#define rrdhost_receiver_replicating_charts_zero(host) (__atomic_store_n(&((host)->rrdpush_receiver_replicating_charts), 0, __ATOMIC_RELAXED))
1351
+#define rrdhost_receiver_replicating_charts(host) (__atomic_load_n(&((host)->stream.rcv.status.replication.charts), __ATOMIC_RELAXED))
1352
+#define rrdhost_receiver_replicating_charts_plus_one(host) (__atomic_add_fetch(&((host)->stream.rcv.status.replication.charts), 1, __ATOMIC_RELAXED))
1353
+#define rrdhost_receiver_replicating_charts_minus_one(host) (__atomic_sub_fetch(&((host)->stream.rcv.status.replication.charts), 1, __ATOMIC_RELAXED))
1354
+#define rrdhost_receiver_replicating_charts_zero(host) (__atomic_store_n(&((host)->stream.rcv.status.replication.charts), 0, __ATOMIC_RELAXED))
1355
1346
-#define rrdhost_sender_replicating_charts(host) (__atomic_load_n(&((host)->rrdpush_sender_replicating_charts), __ATOMIC_RELAXED))
1347
-#define rrdhost_sender_replicating_charts_plus_one(host) (__atomic_add_fetch(&((host)->rrdpush_sender_replicating_charts), 1, __ATOMIC_RELAXED))
1348
-#define rrdhost_sender_replicating_charts_minus_one(host) (__atomic_sub_fetch(&((host)->rrdpush_sender_replicating_charts), 1, __ATOMIC_RELAXED))
1349
-#define rrdhost_sender_replicating_charts_zero(host) (__atomic_store_n(&((host)->rrdpush_sender_replicating_charts), 0, __ATOMIC_RELAXED))
1356
+#define rrdhost_sender_replicating_charts(host) (__atomic_load_n(&((host)->stream.snd.status.replication.charts), __ATOMIC_RELAXED))
1357
+#define rrdhost_sender_replicating_charts_plus_one(host) (__atomic_add_fetch(&((host)->stream.snd.status.replication.charts), 1, __ATOMIC_RELAXED))
1358
+#define rrdhost_sender_replicating_charts_minus_one(host) (__atomic_sub_fetch(&((host)->stream.snd.status.replication.charts), 1, __ATOMIC_RELAXED))
1359
+#define rrdhost_sender_replicating_charts_zero(host) (__atomic_store_n(&((host)->stream.snd.status.replication.charts), 0, __ATOMIC_RELAXED))
1360
1351
-#define rrdhost_is_online(host) ((host) == localhost || rrdhost_option_check(host, RRDHOST_OPTION_VIRTUAL_HOST) || !rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN | RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED))
1361
+#define rrdhost_is_online(host) ((host) == localhost || rrdhost_option_check(host, RRDHOST_OPTION_VIRTUAL_HOST) || !rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN | RRDHOST_FLAG_STREAM_RECEIVER_DISCONNECTED))
1362
bool rrdhost_matches_window(RRDHOST *host, time_t after, time_t before, time_t now);
1363
1364
extern DICTIONARY *rrdhost_root_index;
@@ -1409,14 +1419,14 @@ RRDHOST *rrdhost_find_or_create(
1419
int update_every,
1420
long history,
1421
RRD_MEMORY_MODE mode,
1412
- unsigned int health_enabled,
1413
- unsigned int rrdpush_enabled,
1414
- const char *rrdpush_destination,
1415
- const char *rrdpush_api_key,
1416
- const char *rrdpush_send_charts_matching,
1417
- bool rrdpush_enable_replication,
1418
- time_t rrdpush_seconds_to_replicate,
1419
- time_t rrdpush_replication_step,
1422
+ bool health,
1423
+ bool stream,
1424
+ STRING *parents,
1425
+ STRING *api_key,
1426
+ STRING *send_charts_matching,
1427
+ bool replication,
1428
+ time_t replication_period,
1429
+ time_t replication_step,
1430
struct rrdhost_system_info *system_info,
1431
bool is_archived);
1432
@@ -1443,11 +1453,41 @@ RRDSET *rrdset_create_custom(RRDHOST *host
1453
, RRD_MEMORY_MODE memory_mode
1454
, long history_entries);
1455
1446
-#define rrdset_create(host, type, id, name, family, context, title, units, plugin, module, priority, update_every, chart_type) \
1447
- rrdset_create_custom(host, type, id, name, family, context, title, units, plugin, module, priority, update_every, chart_type, (host)->rrd_memory_mode, (host)->rrd_history_entries)
1456
+static inline
1457
+RRDSET *rrdset_create(RRDHOST *host
1458
+ , const char *type
1459
+ , const char *id
1460
+ , const char *name
1461
+ , const char *family
1462
+ , const char *context
1463
+ , const char *title
1464
+ , const char *units
1465
+ , const char *plugin
1466
+ , const char *module
1467
+ , long priority
1468
+ , int update_every
1469
+ , RRDSET_TYPE chart_type) {
1470
+ return rrdset_create_custom(
1471
+ host, type, id, name, family, context, title, units, plugin, module, priority, update_every, chart_type, (host)->rrd_memory_mode, (host)->rrd_history_entries);
1472
+}
1473
1449
-#define rrdset_create_localhost(type, id, name, family, context, title, units, plugin, module, priority, update_every, chart_type) \
1450
- rrdset_create(localhost, type, id, name, family, context, title, units, plugin, module, priority, update_every, chart_type)
1474
+static inline
1475
+RRDSET *rrdset_create_localhost(
1476
+ const char *type
1477
+ , const char *id
1478
+ , const char *name
1479
+ , const char *family
1480
+ , const char *context
1481
+ , const char *title
1482
+ , const char *units
1483
+ , const char *plugin
1484
+ , const char *module
1485
+ , long priority
1486
+ , int update_every
1487
+ , RRDSET_TYPE chart_type) {
1488
+ return rrdset_create(
1489
+ localhost, type, id, name, family, context, title, units, plugin, module, priority, update_every, chart_type);
1490
+}
1491
1492
void rrdhost_free_all(void);
1493
@@ -1597,7 +1637,6 @@ void set_host_properties(
1637
const char *os, const char *tzone, const char *abbrev_tzone, int32_t utc_offset,
1638
const char *prog_name, const char *prog_version);
1639
1600
-size_t get_tier_grouping(size_t tier);
1640
void store_metric_collection_completed(void);
1641
1642
static inline void rrdhost_retention(RRDHOST *host, time_t now, bool online, time_t *from, time_t *to) {
src/database/rrddim.c
+2
-2
@@ -53,7 +53,7 @@ static void rrddim_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, v
53
54
rd->rrdset = st;
55
56
- rd->rrdpush.sender.dim_slot = __atomic_add_fetch(&st->rrdpush.sender.dim_last_slot_used, 1, __ATOMIC_RELAXED);
56
+ rd->stream.snd.dim_slot = __atomic_add_fetch(&st->stream.snd.dim_last_slot_used, 1, __ATOMIC_RELAXED);
57
58
if(rrdset_flag_check(st, RRDSET_FLAG_STORE_FIRST))
59
rd->collector.counter = 1;
@@ -289,7 +289,7 @@ size_t rrddim_size(void) {
289
void rrddim_index_init(RRDSET *st) {
290
if(!st->rrddim_root_index) {
291
st->rrddim_root_index = dictionary_create_advanced(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
292
- &dictionary_stats_category_rrdset_rrddim, rrddim_size());
292
+ &dictionary_stats_category_rrddim, rrddim_size());
293
294
dictionary_register_insert_callback(st->rrddim_root_index, rrddim_insert_callback, NULL);
295
dictionary_register_conflict_callback(st->rrddim_root_index, rrddim_conflict_callback, NULL);
src/database/rrdfunctions-exporters.c
+2
-2
@@ -5,7 +5,7 @@
5
#include "rrdfunctions-internals.h"
6
#include "rrdfunctions-exporters.h"
7
8
-void rrd_chart_functions_expose_rrdpush(RRDSET *st, BUFFER *wb) {
8
+void stream_sender_send_rrdset_functions(RRDSET *st, BUFFER *wb) {
9
if(!st->functions_view)
10
return;
11
@@ -27,7 +27,7 @@ void rrd_chart_functions_expose_rrdpush(RRDSET *st, BUFFER *wb) {
27
dfe_done(t);
28
}
29
30
-void rrd_global_functions_expose_rrdpush(RRDHOST *host, BUFFER *wb, bool dyncfg) {
30
+void stream_sender_send_global_rrdhost_functions(RRDHOST *host, BUFFER *wb, bool dyncfg) {
31
rrdhost_flag_clear(host, RRDHOST_FLAG_GLOBAL_FUNCTIONS_UPDATED);
32
33
size_t configs = 0;
src/database/rrdfunctions-exporters.h
+2
-2
@@ -7,8 +7,8 @@
7
8
#define RRDFUNCTIONS_VERSION_SEPARATOR "|"
9
10
-void rrd_chart_functions_expose_rrdpush(RRDSET *st, BUFFER *wb);
11
-void rrd_global_functions_expose_rrdpush(RRDHOST *host, BUFFER *wb, bool dyncfg);
10
+void stream_sender_send_rrdset_functions(RRDSET *st, BUFFER *wb);
11
+void stream_sender_send_global_rrdhost_functions(RRDHOST *host, BUFFER *wb, bool dyncfg);
12
13
void chart_functions2json(RRDSET *st, BUFFER *wb);
14
void chart_functions_to_dict(DICTIONARY *rrdset_functions_view, DICTIONARY *dst, void *value, size_t value_size);
src/database/rrdhost.c
+119
-432
@@ -7,33 +7,9 @@
7
#error RRD_STORAGE_TIERS is not 5 - you need to update the grouping iterations per tier
8
#endif
9
10
-static void rrdhost_streaming_sender_structures_init(RRDHOST *host);
11
-
12
-bool dbengine_enabled = false; // will become true if and when dbengine is initialized
13
-size_t storage_tiers = 3;
14
-bool use_direct_io = true;
15
-size_t storage_tiers_grouping_iterations[RRD_STORAGE_TIERS] = {1, 60, 60, 60, 60};
16
-size_t storage_tiers_collection_per_sec[RRD_STORAGE_TIERS] = {1, 60, 3600, 8 * 3600, 24 * 3600};
17
-double storage_tiers_retention_days[RRD_STORAGE_TIERS] = {14, 90, 2 * 365, 2 * 365, 2 * 365};
18
-
19
-size_t get_tier_grouping(size_t tier) {
20
- if(unlikely(tier >= storage_tiers)) tier = storage_tiers - 1;
21
-
22
- size_t grouping = 1;
23
- // first tier is always 1 iteration of whatever update every the chart has
24
- for(size_t i = 1; i <= tier ;i++)
25
- grouping *= storage_tiers_grouping_iterations[i];
26
-
27
- return grouping;
28
-}
29
-
10
RRDHOST *localhost = NULL;
11
netdata_rwlock_t rrd_rwlock = NETDATA_RWLOCK_INITIALIZER;
12
33
-time_t rrdset_free_obsolete_time_s = 3600;
34
-time_t rrdhost_free_orphan_time_s = 3600;
35
-time_t rrdhost_free_ephemeral_time_s = 86400;
36
-
13
RRDHOST *find_host_by_node_id(char *node_id) {
14
15
ND_UUID node_uuid;
@@ -133,7 +109,8 @@ inline RRDHOST *rrdhost_find_by_hostname(const char *hostname) {
109
if(unlikely(!strcmp(hostname, "localhost")))
110
return localhost;
111
136
- return dictionary_get(rrdhost_root_index_hostname, hostname);
112
+ RRDHOST *host = dictionary_get(rrdhost_root_index_hostname, hostname);
113
+ return host;
114
}
115
116
static inline void rrdhost_index_del_hostname(RRDHOST *host) {
@@ -231,34 +208,6 @@ void set_host_properties(RRDHOST *host, int update_every, RRD_MEMORY_MODE memory
208
// ----------------------------------------------------------------------------
209
// RRDHOST - add a host
210
234
-static void rrdhost_initialize_rrdpush_sender(RRDHOST *host,
235
- unsigned int rrdpush_enabled,
236
- const char *rrdpush_destination,
237
- const char *rrdpush_api_key,
238
- const char *rrdpush_send_charts_matching
239
-) {
240
- if(rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_SENDER_INITIALIZED)) return;
241
-
242
- if(rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) {
243
- rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_INITIALIZED);
244
-
245
- rrdhost_streaming_sender_structures_init(host);
246
-
247
- host->sender->ssl = NETDATA_SSL_UNSET_CONNECTION;
248
-
249
- host->rrdpush.send.destination = strdupz(rrdpush_destination);
250
- rrdpush_destinations_init(host);
251
-
252
- host->rrdpush.send.api_key = strdupz(rrdpush_api_key);
253
- host->rrdpush.send.charts_matching = simple_pattern_create(rrdpush_send_charts_matching, NULL,
254
- SIMPLE_PATTERN_EXACT, true);
255
-
256
- rrdhost_option_set(host, RRDHOST_OPTION_SENDER_ENABLED);
257
- }
258
- else
259
- rrdhost_option_clear(host, RRDHOST_OPTION_SENDER_ENABLED);
260
-}
261
-
211
#ifdef ENABLE_DBENGINE
212
//
213
// true on success
@@ -326,6 +275,24 @@ static RRDHOST *prepare_host_for_unittest(RRDHOST *host)
275
}
276
#endif
277
278
+static void rrdhost_set_replication_parameters(RRDHOST *host, RRD_MEMORY_MODE memory_mode, time_t period, time_t step) {
279
+ host->stream.replication.period = period;
280
+ host->stream.replication.step = step;
281
+ host->stream.rcv.status.replication.percent = 100.0;
282
+
283
+ switch(memory_mode) {
284
+ default:
285
+ case RRD_MEMORY_MODE_ALLOC:
286
+ case RRD_MEMORY_MODE_RAM:
287
+ if(host->stream.replication.period > (time_t) host->rrd_history_entries * (time_t) host->rrd_update_every)
288
+ host->stream.replication.period = (time_t) host->rrd_history_entries * (time_t) host->rrd_update_every;
289
+ break;
290
+
291
+ case RRD_MEMORY_MODE_DBENGINE:
292
+ break;
293
+ }
294
+}
295
+
296
static RRDHOST *rrdhost_create(
297
const char *hostname,
298
const char *registry_hostname,
@@ -339,14 +306,14 @@ static RRDHOST *rrdhost_create(
306
int update_every,
307
long entries,
308
RRD_MEMORY_MODE memory_mode,
342
- unsigned int health_enabled,
343
- unsigned int rrdpush_enabled,
344
- const char *rrdpush_destination,
345
- const char *rrdpush_api_key,
346
- const char *rrdpush_send_charts_matching,
347
- bool rrdpush_enable_replication,
348
- time_t rrdpush_seconds_to_replicate,
349
- time_t rrdpush_replication_step,
309
+ bool health,
310
+ bool stream,
311
+ STRING *parents,
312
+ STRING *api_key,
313
+ STRING *send_charts_matching,
314
+ bool replication,
315
+ time_t replication_period,
316
+ time_t replication_step,
317
struct rrdhost_system_info *system_info,
318
int is_localhost,
319
bool archived
@@ -363,6 +330,8 @@ static RRDHOST *rrdhost_create(
330
__atomic_add_fetch(&netdata_buffers_statistics.rrdhost_allocations_size, sizeof(RRDHOST), __ATOMIC_RELAXED);
331
332
strncpyz(host->machine_guid, guid, GUID_LEN + 1);
333
+ rrdhost_stream_path_init(host);
334
+ rrdhost_stream_parents_init(host);
335
336
set_host_properties(
337
host,
@@ -379,38 +348,23 @@ static RRDHOST *rrdhost_create(
348
rrdhost_init_hostname(host, hostname, false);
349
350
host->rrd_history_entries = align_entries_to_pagesize(memory_mode, entries);
382
- host->health.health_enabled = ((memory_mode == RRD_MEMORY_MODE_NONE)) ? 0 : health_enabled;
351
+ host->health.enabled = ((memory_mode == RRD_MEMORY_MODE_NONE)) ? 0 : health;
352
353
spinlock_init(&host->receiver_lock);
354
355
if (likely(!archived)) {
356
rrd_functions_host_init(host);
388
- host->last_connected = now_realtime_sec();
357
+ host->stream.snd.status.last_connected = now_realtime_sec();
358
host->rrdlabels = rrdlabels_create();
390
- rrdhost_initialize_rrdpush_sender(
391
- host, rrdpush_enabled, rrdpush_destination, rrdpush_api_key, rrdpush_send_charts_matching);
359
+ stream_sender_structures_init(host, stream, parents, api_key, send_charts_matching);
360
}
361
394
- if(rrdpush_enable_replication)
362
+ if(replication)
363
rrdhost_option_set(host, RRDHOST_OPTION_REPLICATION);
364
else
365
rrdhost_option_clear(host, RRDHOST_OPTION_REPLICATION);
366
399
- host->rrdpush_seconds_to_replicate = rrdpush_seconds_to_replicate;
400
- host->rrdpush_replication_step = rrdpush_replication_step;
401
- host->rrdpush_receiver_replication_percent = 100.0;
402
-
403
- switch(memory_mode) {
404
- default:
405
- case RRD_MEMORY_MODE_ALLOC:
406
- case RRD_MEMORY_MODE_RAM:
407
- if(host->rrdpush_seconds_to_replicate > (time_t) host->rrd_history_entries * (time_t) host->rrd_update_every)
408
- host->rrdpush_seconds_to_replicate = (time_t) host->rrd_history_entries * (time_t) host->rrd_update_every;
409
- break;
410
-
411
- case RRD_MEMORY_MODE_DBENGINE:
412
- break;
413
- }
367
+ rrdhost_set_replication_parameters(host, memory_mode, replication_period, replication_step);
368
369
host->system_info = system_info;
370
@@ -531,13 +485,14 @@ static RRDHOST *rrdhost_create(
485
, host->rrd_update_every
486
, rrd_memory_mode_name(host->rrd_memory_mode)
487
, host->rrd_history_entries
534
- , rrdhost_has_rrdpush_sender_enabled(host)?"enabled":"disabled"
535
- , host->rrdpush.send.destination?host->rrdpush.send.destination:""
536
- , host->rrdpush.send.api_key?host->rrdpush.send.api_key:""
537
- , host->health.health_enabled?"enabled":"disabled"
488
+ ,
489
+ rrdhost_has_stream_sender_enabled(host)?"enabled":"disabled"
490
+ , string2str(host->stream.snd.destination)
491
+ , string2str(host->stream.snd.api_key)
492
+ , host->health.enabled ?"enabled":"disabled"
493
, host->cache_dir
539
- , string2str(host->health.health_default_exec)
540
- , string2str(host->health.health_default_recipient)
494
+ , string2str(host->health.default_exec)
495
+ , string2str(host->health.default_recipient)
496
);
497
498
if(!archived) {
@@ -564,14 +519,14 @@ static void rrdhost_update(RRDHOST *host
519
, int update_every
520
, long history
521
, RRD_MEMORY_MODE mode
567
- , unsigned int health_enabled
568
- , unsigned int rrdpush_enabled
569
- , const char *rrdpush_destination
570
- , const char *rrdpush_api_key
571
- , const char *rrdpush_send_charts_matching
572
- , bool rrdpush_enable_replication
573
- , time_t rrdpush_seconds_to_replicate
574
- , time_t rrdpush_replication_step
522
+ , bool health
523
+ , bool stream
524
+ , STRING *parents
525
+ , STRING *api_key
526
+ , STRING *send_charts_matching
527
+ , bool replication
528
+ , time_t replication_period
529
+ , time_t replication_step
530
, struct rrdhost_system_info *system_info
531
)
532
{
@@ -579,7 +534,7 @@ static void rrdhost_update(RRDHOST *host
534
535
spinlock_lock(&host->rrdhost_update_lock);
536
582
- host->health.health_enabled = (mode == RRD_MEMORY_MODE_NONE) ? 0 : health_enabled;
537
+ host->health.enabled = (mode == RRD_MEMORY_MODE_NONE) ? 0 : health;
538
539
{
540
struct rrdhost_system_info *old = host->system_info;
@@ -651,7 +606,7 @@ static void rrdhost_update(RRDHOST *host
606
if(!host->rrdvars)
607
host->rrdvars = rrdvariables_create();
608
654
- host->last_connected = now_realtime_sec();
609
+ host->stream.snd.status.last_connected = now_realtime_sec();
610
611
if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)) {
612
rrdhost_flag_clear(host, RRDHOST_FLAG_ARCHIVED);
@@ -664,21 +619,16 @@ static void rrdhost_update(RRDHOST *host
619
if (!host->rrdset_root_index)
620
rrdset_index_init(host);
621
667
- rrdhost_initialize_rrdpush_sender(host,
668
- rrdpush_enabled,
669
- rrdpush_destination,
670
- rrdpush_api_key,
671
- rrdpush_send_charts_matching);
622
+ stream_sender_structures_init(host, stream, parents, api_key, send_charts_matching);
623
624
rrdcalc_rrdhost_index_init(host);
625
675
- if(rrdpush_enable_replication)
626
+ if(replication)
627
rrdhost_option_set(host, RRDHOST_OPTION_REPLICATION);
628
else
629
rrdhost_option_clear(host, RRDHOST_OPTION_REPLICATION);
630
680
- host->rrdpush_seconds_to_replicate = rrdpush_seconds_to_replicate;
681
- host->rrdpush_replication_step = rrdpush_replication_step;
631
+ rrdhost_set_replication_parameters(host, host->rrd_memory_mode, replication_period, replication_step);
632
633
ml_host_new(host);
634
@@ -704,14 +654,14 @@ RRDHOST *rrdhost_find_or_create(
654
, int update_every
655
, long history
656
, RRD_MEMORY_MODE mode
707
- , unsigned int health_enabled
708
- , unsigned int rrdpush_enabled
709
- , const char *rrdpush_destination
710
- , const char *rrdpush_api_key
711
- , const char *rrdpush_send_charts_matching
712
- , bool rrdpush_enable_replication
713
- , time_t rrdpush_seconds_to_replicate
714
- , time_t rrdpush_replication_step
657
+ , bool health
658
+ , bool stream
659
+ , STRING *parents
660
+ , STRING *api_key
661
+ , STRING *send_charts_matching
662
+ , bool replication
663
+ , time_t replication_period
664
+ , time_t replication_step
665
, struct rrdhost_system_info *system_info
666
, bool archived
667
) {
@@ -748,14 +698,14 @@ RRDHOST *rrdhost_find_or_create(
698
, update_every
699
, history
700
, mode
751
- , health_enabled
752
- , rrdpush_enabled
753
- , rrdpush_destination
754
- , rrdpush_api_key
755
- , rrdpush_send_charts_matching
756
- , rrdpush_enable_replication
757
- , rrdpush_seconds_to_replicate
758
- , rrdpush_replication_step
701
+ , health
702
+ , stream
703
+ , parents
704
+ , api_key
705
+ , send_charts_matching
706
+ , replication
707
+ , replication_period
708
+ , replication_step
709
, system_info
710
, 0
711
, archived
@@ -763,28 +713,29 @@ RRDHOST *rrdhost_find_or_create(
713
}
714
else {
715
if (likely(!rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_CONTEXT_LOAD)))
766
- rrdhost_update(host
767
- , hostname
768
- , registry_hostname
769
- , guid
770
- , os
771
- , timezone
772
- , abbrev_timezone
773
- , utc_offset
774
- , prog_name
775
- , prog_version
776
- , update_every
777
- , history
778
- , mode
779
- , health_enabled
780
- , rrdpush_enabled
781
- , rrdpush_destination
782
- , rrdpush_api_key
783
- , rrdpush_send_charts_matching
784
- , rrdpush_enable_replication
785
- , rrdpush_seconds_to_replicate
786
- , rrdpush_replication_step
787
- , system_info);
716
+ rrdhost_update(
717
+ host
718
+ , hostname
719
+ , registry_hostname
720
+ , guid
721
+ , os
722
+ , timezone
723
+ , abbrev_timezone
724
+ , utc_offset
725
+ , prog_name
726
+ , prog_version
727
+ , update_every
728
+ , history
729
+ , mode
730
+ , health
731
+ , stream
732
+ , parents
733
+ , api_key
734
+ , send_charts_matching
735
+ , replication
736
+ , replication_period
737
+ , replication_step
738
+ , system_info);
739
}
740
741
return host;
@@ -798,232 +749,13 @@ inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, tim
749
&& rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN)
750
&& !rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_CONTEXT_LOAD)
751
&& !host->receiver
801
- && host->child_disconnected_time
802
- && host->child_disconnected_time + rrdhost_free_orphan_time_s < now_s)
752
+ && host->stream.rcv.status.last_disconnected
753
+ && host->stream.rcv.status.last_disconnected + rrdhost_free_orphan_time_s < now_s)
754
return 1;
755
756
return 0;
757
}
758
808
-// ----------------------------------------------------------------------------
809
-// RRDHOST global / startup initialization
810
-
811
-#ifdef ENABLE_DBENGINE
812
-struct dbengine_initialization {
813
- ND_THREAD *thread;
814
- char path[FILENAME_MAX + 1];
815
- int disk_space_mb;
816
- size_t retention_seconds;
817
- size_t tier;
818
- int ret;
819
-};
820
-
821
-typedef struct rrd_alert_prototype {
822
- struct rrd_alert_match match;
823
- struct rrd_alert_config config;
824
-
825
- struct {
826
- uint32_t uses;
827
- bool enabled;
828
- bool is_on_disk;
829
- SPINLOCK spinlock;
830
- struct rrd_alert_prototype *prev, *next;
831
- } _internal;
832
-} RRD_ALERT_PROTOTYPE;
833
-
834
-void *dbengine_tier_init(void *ptr) {
835
- struct dbengine_initialization *dbi = ptr;
836
- dbi->ret = rrdeng_init(NULL, dbi->path, dbi->disk_space_mb, dbi->tier, dbi->retention_seconds);
837
- return ptr;
838
-}
839
-
840
-RRD_BACKFILL get_dbengine_backfill(RRD_BACKFILL backfill)
841
-{
842
- const char *bf = config_get(
843
- CONFIG_SECTION_DB,
844
- "dbengine tier backfill",
845
- backfill == RRD_BACKFILL_NEW ? "new" :
846
- backfill == RRD_BACKFILL_FULL ? "full" :
847
- "none");
848
-
849
- if (strcmp(bf, "new") == 0)
850
- backfill = RRD_BACKFILL_NEW;
851
- else if (strcmp(bf, "full") == 0)
852
- backfill = RRD_BACKFILL_FULL;
853
- else if (strcmp(bf, "none") == 0)
854
- backfill = RRD_BACKFILL_NONE;
855
- else {
856
- nd_log(NDLS_DAEMON, NDLP_WARNING, "DBENGINE: unknown backfill value '%s', assuming 'new'", bf);
857
- config_set(CONFIG_SECTION_DB, "dbengine tier backfill", "new");
858
- backfill = RRD_BACKFILL_NEW;
859
- }
860
- return backfill;
861
-}
862
-
863
-#endif
864
-
865
-static void dbengine_init(const char *hostname) {
866
-#ifdef ENABLE_DBENGINE
867
-
868
- use_direct_io = config_get_boolean(CONFIG_SECTION_DB, "dbengine use direct io", use_direct_io);
869
-
870
- unsigned read_num = (unsigned)config_get_number(CONFIG_SECTION_DB, "dbengine pages per extent", DEFAULT_PAGES_PER_EXTENT);
871
- if (read_num > 0 && read_num <= DEFAULT_PAGES_PER_EXTENT)
872
- rrdeng_pages_per_extent = read_num;
873
- else {
874
- nd_log(NDLS_DAEMON, NDLP_WARNING,
875
- "Invalid dbengine pages per extent %u given. Using %u.",
876
- read_num, rrdeng_pages_per_extent);
877
-
878
- config_set_number(CONFIG_SECTION_DB, "dbengine pages per extent", rrdeng_pages_per_extent);
879
- }
880
-
881
- storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
882
- if(storage_tiers < 1) {
883
- nd_log(NDLS_DAEMON, NDLP_WARNING, "At least 1 storage tier is required. Assuming 1.");
884
-
885
- storage_tiers = 1;
886
- config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
887
- }
888
- if(storage_tiers > RRD_STORAGE_TIERS) {
889
- nd_log(NDLS_DAEMON, NDLP_WARNING,
890
- "Up to %d storage tier are supported. Assuming %d.",
891
- RRD_STORAGE_TIERS, RRD_STORAGE_TIERS);
892
-
893
- storage_tiers = RRD_STORAGE_TIERS;
894
- config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
895
- }
896
-
897
- new_dbengine_defaults =
898
- (!legacy_multihost_db_space &&
899
- !config_exists(CONFIG_SECTION_DB, "dbengine tier 1 update every iterations") &&
900
- !config_exists(CONFIG_SECTION_DB, "dbengine tier 2 update every iterations") &&
901
- !config_exists(CONFIG_SECTION_DB, "dbengine tier 3 update every iterations") &&
902
- !config_exists(CONFIG_SECTION_DB, "dbengine tier 4 update every iterations") &&
903
- !config_exists(CONFIG_SECTION_DB, "dbengine tier 1 retention size") &&
904
- !config_exists(CONFIG_SECTION_DB, "dbengine tier 2 retention size") &&
905
- !config_exists(CONFIG_SECTION_DB, "dbengine tier 3 retention size") &&
906
- !config_exists(CONFIG_SECTION_DB, "dbengine tier 4 retention size"));
907
-
908
- default_backfill = get_dbengine_backfill(RRD_BACKFILL_NEW);
909
- char dbengineconfig[200 + 1];
910
-
911
- size_t grouping_iterations = default_rrd_update_every;
912
- storage_tiers_grouping_iterations[0] = default_rrd_update_every;
913
-
914
- for (size_t tier = 1; tier < storage_tiers; tier++) {
915
- grouping_iterations = storage_tiers_grouping_iterations[tier];
916
- snprintfz(dbengineconfig, sizeof(dbengineconfig) - 1, "dbengine tier %zu update every iterations", tier);
917
- grouping_iterations = config_get_number(CONFIG_SECTION_DB, dbengineconfig, grouping_iterations);
918
- if(grouping_iterations < 2) {
919
- grouping_iterations = 2;
920
- config_set_number(CONFIG_SECTION_DB, dbengineconfig, grouping_iterations);
921
- nd_log(NDLS_DAEMON, NDLP_WARNING,
922
- "DBENGINE on '%s': 'dbegnine tier %zu update every iterations' cannot be less than 2. Assuming 2.",
923
- hostname, tier);
924
- }
925
- storage_tiers_grouping_iterations[tier] = grouping_iterations;
926
- }
927
-
928
- default_multidb_disk_quota_mb = (int) config_get_size_mb(CONFIG_SECTION_DB, "dbengine tier 0 retention size", RRDENG_DEFAULT_TIER_DISK_SPACE_MB);
929
- if(default_multidb_disk_quota_mb && default_multidb_disk_quota_mb < RRDENG_MIN_DISK_SPACE_MB) {
930
- netdata_log_error("Invalid disk space %d for tier 0 given. Defaulting to %d.", default_multidb_disk_quota_mb, RRDENG_MIN_DISK_SPACE_MB);
931
- default_multidb_disk_quota_mb = RRDENG_MIN_DISK_SPACE_MB;
932
- config_set_size_mb(CONFIG_SECTION_DB, "dbengine tier 0 retention size", default_multidb_disk_quota_mb);
933
- }
934
-
935
-#ifdef OS_WINDOWS
936
- // FIXME: for whatever reason joining the initialization threads
937
- // fails on Windows.
938
- bool parallel_initialization = false;
939
-#else
940
- bool parallel_initialization = (storage_tiers <= (size_t)get_netdata_cpus()) ? true : false;
941
-#endif
942
-
943
- struct dbengine_initialization tiers_init[RRD_STORAGE_TIERS] = {};
944
-
945
- size_t created_tiers = 0;
946
- char dbenginepath[FILENAME_MAX + 1];
947
-
948
- for (size_t tier = 0; tier < storage_tiers; tier++) {
949
-
950
- if (tier == 0)
951
- snprintfz(dbenginepath, FILENAME_MAX, "%s/dbengine", netdata_configured_cache_dir);
952
- else
953
- snprintfz(dbenginepath, FILENAME_MAX, "%s/dbengine-tier%zu", netdata_configured_cache_dir, tier);
954
-
955
- int ret = mkdir(dbenginepath, 0775);
956
- if (ret != 0 && errno != EEXIST) {
957
- nd_log(NDLS_DAEMON, NDLP_CRIT, "DBENGINE on '%s': cannot create directory '%s'", hostname, dbenginepath);
958
- continue;
959
- }
960
-
961
- int disk_space_mb = tier ? RRDENG_DEFAULT_TIER_DISK_SPACE_MB : default_multidb_disk_quota_mb;
962
- snprintfz(dbengineconfig, sizeof(dbengineconfig) - 1, "dbengine tier %zu retention size", tier);
963
- disk_space_mb = config_get_size_mb(CONFIG_SECTION_DB, dbengineconfig, disk_space_mb);
964
-
965
- snprintfz(dbengineconfig, sizeof(dbengineconfig) - 1, "dbengine tier %zu retention time", tier);
966
- storage_tiers_retention_days[tier] = config_get_duration_days(
967
- CONFIG_SECTION_DB, dbengineconfig, new_dbengine_defaults ? storage_tiers_retention_days[tier] : 0);
968
-
969
- tiers_init[tier].disk_space_mb = (int) disk_space_mb;
970
- tiers_init[tier].tier = tier;
971
- tiers_init[tier].retention_seconds = (size_t) (86400.0 * storage_tiers_retention_days[tier]);
972
- strncpyz(tiers_init[tier].path, dbenginepath, FILENAME_MAX);
973
- tiers_init[tier].ret = 0;
974
-
975
- if(parallel_initialization) {
976
- char tag[NETDATA_THREAD_TAG_MAX + 1];
977
- snprintfz(tag, NETDATA_THREAD_TAG_MAX, "DBENGINIT[%zu]", tier);
978
- tiers_init[tier].thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_JOINABLE, dbengine_tier_init, &tiers_init[tier]);
979
- }
980
- else
981
- dbengine_tier_init(&tiers_init[tier]);
982
- }
983
-
984
- for(size_t tier = 0; tier < storage_tiers ;tier++) {
985
- if(parallel_initialization)
986
- nd_thread_join(tiers_init[tier].thread);
987
-
988
- if(tiers_init[tier].ret != 0) {
989
- nd_log(NDLS_DAEMON, NDLP_ERR,
990
- "DBENGINE on '%s': Failed to initialize multi-host database tier %zu on path '%s'",
991
- hostname, tiers_init[tier].tier, tiers_init[tier].path);
992
- }
993
- else if(created_tiers == tier)
994
- created_tiers++;
995
- }
996
-
997
- if(created_tiers && created_tiers < storage_tiers) {
998
- nd_log(NDLS_DAEMON, NDLP_WARNING,
999
- "DBENGINE on '%s': Managed to create %zu tiers instead of %zu. Continuing with %zu available.",
1000
- hostname, created_tiers, storage_tiers, created_tiers);
1001
-
1002
- storage_tiers = created_tiers;
1003
- }
1004
- else if(!created_tiers)
1005
- fatal("DBENGINE on '%s', failed to initialize databases at '%s'.", hostname, netdata_configured_cache_dir);
1006
-
1007
- for(size_t tier = 0; tier < storage_tiers ;tier++)
1008
- rrdeng_readiness_wait(multidb_ctx[tier]);
1009
-
1010
- calculate_tier_disk_space_percentage();
1011
-
1012
- dbengine_enabled = true;
1013
-#else
1014
- storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", 1);
1015
- if(storage_tiers != 1) {
1016
- nd_log(NDLS_DAEMON, NDLP_WARNING,
1017
- "DBENGINE is not available on '%s', so only 1 database tier can be supported.",
1018
- hostname);
1019
-
1020
- storage_tiers = 1;
1021
- config_set_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
1022
- }
1023
- dbengine_enabled = false;
1024
-#endif
1025
-}
1026
-
759
void api_v1_management_init(void);
760
761
int rrd_init(const char *hostname, struct rrdhost_system_info *system_info, bool unittest) {
@@ -1053,7 +785,7 @@ int rrd_init(const char *hostname, struct rrdhost_system_info *system_info, bool
785
nd_log(NDLS_DAEMON, NDLP_DEBUG,
786
"DBENGINE: Initializing ...");
787
1056
- dbengine_init(hostname);
788
+ netdata_conf_dbengine_init(hostname);
789
}
790
else
791
storage_tiers = 1;
@@ -1094,14 +826,14 @@ int rrd_init(const char *hostname, struct rrdhost_system_info *system_info, bool
826
, default_rrd_history_entries
827
, default_rrd_memory_mode
828
, health_plugin_enabled()
1097
- ,
1098
- stream_conf_send_enabled,
1099
- stream_conf_send_destination,
1100
- stream_conf_send_api_key,
1101
- stream_conf_send_charts_matching,
1102
- stream_conf_replication_enabled,
1103
- stream_conf_replication_period,
1104
- stream_conf_replication_step, system_info
829
+ , stream_send.enabled
830
+ , stream_send.parents.destination
831
+ , stream_send.api_key
832
+ , stream_send.send_charts_matching
833
+ , stream_receive.replication.enabled
834
+ , stream_receive.replication.period
835
+ , stream_receive.replication.step
836
+ , system_info
837
, 1
838
, 0
839
);
@@ -1167,51 +899,6 @@ void rrdhost_system_info_free(struct rrdhost_system_info *system_info) {
899
}
900
}
901
1170
-static void rrdhost_streaming_sender_structures_init(RRDHOST *host)
1171
-{
1172
- if (host->sender)
1173
- return;
1174
-
1175
- host->sender = callocz(1, sizeof(*host->sender));
1176
- __atomic_add_fetch(&netdata_buffers_statistics.rrdhost_senders, sizeof(*host->sender), __ATOMIC_RELAXED);
1177
-
1178
- host->sender->host = host;
1179
- host->sender->buffer = cbuffer_new(CBUFFER_INITIAL_SIZE, 1024 * 1024, &netdata_buffers_statistics.cbuffers_streaming);
1180
- host->sender->capabilities = stream_our_capabilities(host, true);
1181
-
1182
- host->sender->rrdpush_sender_pipe[PIPE_READ] = -1;
1183
- host->sender->rrdpush_sender_pipe[PIPE_WRITE] = -1;
1184
- host->sender->rrdpush_sender_socket = -1;
1185
- host->sender->disabled_capabilities = STREAM_CAP_NONE;
1186
-
1187
- if(!stream_conf_compression_enabled)
1188
- host->sender->disabled_capabilities |= STREAM_CAP_COMPRESSIONS_AVAILABLE;
1189
-
1190
- spinlock_init(&host->sender->spinlock);
1191
- replication_init_sender(host->sender);
1192
-}
1193
-
1194
-static void rrdhost_streaming_sender_structures_free(RRDHOST *host)
1195
-{
1196
- rrdhost_option_clear(host, RRDHOST_OPTION_SENDER_ENABLED);
1197
-
1198
- if (unlikely(!host->sender))
1199
- return;
1200
-
1201
- rrdpush_sender_thread_stop(host, STREAM_HANDSHAKE_DISCONNECT_HOST_CLEANUP, true); // stop a possibly running thread
1202
- cbuffer_free(host->sender->buffer);
1203
-
1204
- rrdpush_compressor_destroy(&host->sender->compressor);
1205
-
1206
- replication_cleanup_sender(host->sender);
1207
-
1208
- __atomic_sub_fetch(&netdata_buffers_statistics.rrdhost_senders, sizeof(*host->sender), __ATOMIC_RELAXED);
1209
-
1210
- freez(host->sender);
1211
- host->sender = NULL;
1212
- rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_INITIALIZED);
1213
-}
1214
-
902
void rrdhost_free___while_having_rrd_wrlock(RRDHOST *host, bool force) {
903
if(!host) return;
904
@@ -1243,10 +930,10 @@ void rrdhost_free___while_having_rrd_wrlock(RRDHOST *host, bool force) {
930
// ------------------------------------------------------------------------
931
// clean up streaming
932
1246
- rrdhost_streaming_sender_structures_free(host);
933
+ stream_sender_structures_free(host);
934
935
if (netdata_exit || force)
1249
- stop_streaming_receiver(host, STREAM_HANDSHAKE_DISCONNECT_HOST_CLEANUP);
936
+ stream_receiver_signal_to_stop_and_wait(host, STREAM_HANDSHAKE_DISCONNECT_HOST_CLEANUP);
937
938
939
// ------------------------------------------------------------------------
@@ -1285,13 +972,13 @@ void rrdhost_free___while_having_rrd_wrlock(RRDHOST *host, bool force) {
972
string_freez(host->program_version);
973
rrdhost_system_info_free(host->system_info);
974
freez(host->cache_dir);
1288
- freez(host->rrdpush.send.api_key);
1289
- freez(host->rrdpush.send.destination);
1290
- rrdpush_destinations_free(host);
1291
- string_freez(host->health.health_default_exec);
1292
- string_freez(host->health.health_default_recipient);
975
+ string_freez(host->stream.snd.api_key);
976
+ string_freez(host->stream.snd.destination);
977
+ rrdhost_stream_parents_free(host, false);
978
+ string_freez(host->health.default_exec);
979
+ string_freez(host->health.default_recipient);
980
string_freez(host->registry_hostname);
1294
- simple_pattern_free(host->rrdpush.send.charts_matching);
981
+ simple_pattern_free(host->stream.snd.charts_matching);
982
983
rrd_functions_host_destroy(host);
984
rrdvariables_destroy(host->rrdvars);
@@ -1429,23 +1116,23 @@ static void rrdhost_load_auto_labels(void) {
1116
int has_unstable_connection = appconfig_get_boolean(&netdata_config, CONFIG_SECTION_GLOBAL, "has unstable connection", CONFIG_BOOLEAN_NO);
1117
rrdlabels_add(labels, "_has_unstable_connection", has_unstable_connection ? "true" : "false", RRDLABEL_SRC_AUTO);
1118
1432
- rrdlabels_add(labels, "_is_parent", (localhost->connected_children_count > 0) ? "true" : "false", RRDLABEL_SRC_AUTO);
1119
+ rrdlabels_add(labels, "_is_parent", (stream_receivers_currently_connected() > 0) ? "true" : "false", RRDLABEL_SRC_AUTO);
1120
1121
rrdlabels_add(labels, "_hostname", string2str(localhost->hostname), RRDLABEL_SRC_AUTO);
1122
rrdlabels_add(labels, "_os", string2str(localhost->os), RRDLABEL_SRC_AUTO);
1123
1437
- if (localhost->rrdpush.send.destination)
1438
- rrdlabels_add(labels, "_streams_to", localhost->rrdpush.send.destination, RRDLABEL_SRC_AUTO);
1124
+ if (localhost->stream.snd.destination)
1125
+ rrdlabels_add(labels, "_streams_to", string2str(localhost->stream.snd.destination), RRDLABEL_SRC_AUTO);
1126
}
1127
1128
void rrdhost_set_is_parent_label(void) {
1442
- int count = __atomic_load_n(&localhost->connected_children_count, __ATOMIC_RELAXED);
1129
+ uint32_t count = stream_receivers_currently_connected();
1130
1131
if (count == 0 || count == 1) {
1132
RRDLABELS *labels = localhost->rrdlabels;
1133
rrdlabels_add(labels, "_is_parent", (count) ? "true" : "false", RRDLABEL_SRC_AUTO);
1134
1448
- //queue a node info
1135
+ // queue a node info
1136
aclk_queue_node_info(localhost, false);
1137
}
1138
}
@@ -1508,7 +1195,7 @@ void reload_host_labels(void) {
1195
1196
rrdhost_flag_set(localhost,RRDHOST_FLAG_METADATA_LABELS | RRDHOST_FLAG_METADATA_UPDATE);
1197
1511
- rrdpush_send_host_labels(localhost);
1198
+ stream_send_host_labels(localhost);
1199
}
1200
1201
void rrdhost_finalize_collection(RRDHOST *host) {
src/database/rrdset.c
+54
-54
@@ -11,58 +11,58 @@ void rrdset_metadata_updated(RRDSET *st) {
11
}
12
13
// ----------------------------------------------------------------------------
14
-// RRDSET rrdpush send chart_slots
14
+// RRDSET streaming send chart_slots
15
16
-static void rrdset_rrdpush_send_chart_slot_assign(RRDSET *st) {
16
+static void rrdset_stream_send_chart_slot_assign(RRDSET *st) {
17
RRDHOST *host = st->rrdhost;
18
- spinlock_lock(&host->rrdpush.send.pluginsd_chart_slots.available.spinlock);
18
+ spinlock_lock(&host->stream.snd.pluginsd_chart_slots.available.spinlock);
19
20
- if(host->rrdpush.send.pluginsd_chart_slots.available.used > 0)
21
- st->rrdpush.sender.chart_slot =
22
- host->rrdpush.send.pluginsd_chart_slots.available.array[--host->rrdpush.send.pluginsd_chart_slots.available.used];
20
+ if(host->stream.snd.pluginsd_chart_slots.available.used > 0)
21
+ st->stream.snd.chart_slot =
22
+ host->stream.snd.pluginsd_chart_slots.available.array[--host->stream.snd.pluginsd_chart_slots.available.used];
23
else
24
- st->rrdpush.sender.chart_slot = ++host->rrdpush.send.pluginsd_chart_slots.last_used;
24
+ st->stream.snd.chart_slot = ++host->stream.snd.pluginsd_chart_slots.last_used;
25
26
- spinlock_unlock(&host->rrdpush.send.pluginsd_chart_slots.available.spinlock);
26
+ spinlock_unlock(&host->stream.snd.pluginsd_chart_slots.available.spinlock);
27
}
28
29
-static void rrdset_rrdpush_send_chart_slot_release(RRDSET *st) {
30
- if(!st->rrdpush.sender.chart_slot || st->rrdhost->rrdpush.send.pluginsd_chart_slots.available.ignore)
29
+static void rrdset_stream_send_chart_slot_release(RRDSET *st) {
30
+ if(!st->stream.snd.chart_slot || st->rrdhost->stream.snd.pluginsd_chart_slots.available.ignore)
31
return;
32
33
RRDHOST *host = st->rrdhost;
34
- spinlock_lock(&host->rrdpush.send.pluginsd_chart_slots.available.spinlock);
34
+ spinlock_lock(&host->stream.snd.pluginsd_chart_slots.available.spinlock);
35
36
- if(host->rrdpush.send.pluginsd_chart_slots.available.used >= host->rrdpush.send.pluginsd_chart_slots.available.size) {
37
- uint32_t old_size = host->rrdpush.send.pluginsd_chart_slots.available.size;
36
+ if(host->stream.snd.pluginsd_chart_slots.available.used >= host->stream.snd.pluginsd_chart_slots.available.size) {
37
+ uint32_t old_size = host->stream.snd.pluginsd_chart_slots.available.size;
38
uint32_t new_size = (old_size > 0) ? (old_size * 2) : 1024;
39
40
- host->rrdpush.send.pluginsd_chart_slots.available.array =
41
- reallocz(host->rrdpush.send.pluginsd_chart_slots.available.array, new_size * sizeof(uint32_t));
40
+ host->stream.snd.pluginsd_chart_slots.available.array =
41
+ reallocz(host->stream.snd.pluginsd_chart_slots.available.array, new_size * sizeof(uint32_t));
42
43
- host->rrdpush.send.pluginsd_chart_slots.available.size = new_size;
43
+ host->stream.snd.pluginsd_chart_slots.available.size = new_size;
44
}
45
46
- host->rrdpush.send.pluginsd_chart_slots.available.array[host->rrdpush.send.pluginsd_chart_slots.available.used++] =
47
- st->rrdpush.sender.chart_slot;
46
+ host->stream.snd.pluginsd_chart_slots.available.array[host->stream.snd.pluginsd_chart_slots.available.used++] =
47
+ st->stream.snd.chart_slot;
48
49
- st->rrdpush.sender.chart_slot = 0;
50
- spinlock_unlock(&host->rrdpush.send.pluginsd_chart_slots.available.spinlock);
49
+ st->stream.snd.chart_slot = 0;
50
+ spinlock_unlock(&host->stream.snd.pluginsd_chart_slots.available.spinlock);
51
}
52
53
void rrdhost_pluginsd_send_chart_slots_free(RRDHOST *host) {
54
- spinlock_lock(&host->rrdpush.send.pluginsd_chart_slots.available.spinlock);
55
- host->rrdpush.send.pluginsd_chart_slots.available.ignore = true;
56
- freez(host->rrdpush.send.pluginsd_chart_slots.available.array);
57
- host->rrdpush.send.pluginsd_chart_slots.available.array = NULL;
58
- host->rrdpush.send.pluginsd_chart_slots.available.used = 0;
59
- host->rrdpush.send.pluginsd_chart_slots.available.size = 0;
60
- spinlock_unlock(&host->rrdpush.send.pluginsd_chart_slots.available.spinlock);
54
+ spinlock_lock(&host->stream.snd.pluginsd_chart_slots.available.spinlock);
55
+ host->stream.snd.pluginsd_chart_slots.available.ignore = true;
56
+ freez(host->stream.snd.pluginsd_chart_slots.available.array);
57
+ host->stream.snd.pluginsd_chart_slots.available.array = NULL;
58
+ host->stream.snd.pluginsd_chart_slots.available.used = 0;
59
+ host->stream.snd.pluginsd_chart_slots.available.size = 0;
60
+ spinlock_unlock(&host->stream.snd.pluginsd_chart_slots.available.spinlock);
61
62
// zero all the slots on all charts, so that they will not attempt to access the array
63
RRDSET *st;
64
rrdset_foreach_read(st, host) {
65
- st->rrdpush.sender.chart_slot = 0;
65
+ st->stream.snd.chart_slot = 0;
66
}
67
rrdset_foreach_done(st);
68
}
@@ -78,9 +78,9 @@ void rrdset_pluginsd_receive_unslot(RRDSET *st) {
78
RRDHOST *host = st->rrdhost;
79
80
if(st->pluginsd.last_slot >= 0 &&
81
- (uint32_t)st->pluginsd.last_slot < host->rrdpush.receive.pluginsd_chart_slots.size &&
82
- host->rrdpush.receive.pluginsd_chart_slots.array[st->pluginsd.last_slot] == st) {
83
- host->rrdpush.receive.pluginsd_chart_slots.array[st->pluginsd.last_slot] = NULL;
81
+ (uint32_t)st->pluginsd.last_slot < host->stream.rcv.pluginsd_chart_slots.size &&
82
+ host->stream.rcv.pluginsd_chart_slots.array[st->pluginsd.last_slot] == st) {
83
+ host->stream.rcv.pluginsd_chart_slots.array[st->pluginsd.last_slot] = NULL;
84
}
85
86
st->pluginsd.last_slot = -1;
@@ -113,18 +113,18 @@ static void rrdset_pluginsd_receive_slots_initialize(RRDSET *st) {
113
}
114
115
void rrdhost_pluginsd_receive_chart_slots_free(RRDHOST *host) {
116
- spinlock_lock(&host->rrdpush.receive.pluginsd_chart_slots.spinlock);
116
+ spinlock_lock(&host->stream.rcv.pluginsd_chart_slots.spinlock);
117
118
- if(host->rrdpush.receive.pluginsd_chart_slots.array) {
119
- for (size_t s = 0; s < host->rrdpush.receive.pluginsd_chart_slots.size; s++)
120
- rrdset_pluginsd_receive_unslot_and_cleanup(host->rrdpush.receive.pluginsd_chart_slots.array[s]);
118
+ if(host->stream.rcv.pluginsd_chart_slots.array) {
119
+ for (size_t s = 0; s < host->stream.rcv.pluginsd_chart_slots.size; s++)
120
+ rrdset_pluginsd_receive_unslot_and_cleanup(host->stream.rcv.pluginsd_chart_slots.array[s]);
121
122
- freez(host->rrdpush.receive.pluginsd_chart_slots.array);
123
- host->rrdpush.receive.pluginsd_chart_slots.array = NULL;
124
- host->rrdpush.receive.pluginsd_chart_slots.size = 0;
122
+ freez(host->stream.rcv.pluginsd_chart_slots.array);
123
+ host->stream.rcv.pluginsd_chart_slots.array = NULL;
124
+ host->stream.rcv.pluginsd_chart_slots.size = 0;
125
}
126
127
- spinlock_unlock(&host->rrdpush.receive.pluginsd_chart_slots.spinlock);
127
+ spinlock_unlock(&host->stream.rcv.pluginsd_chart_slots.spinlock);
128
}
129
130
// ----------------------------------------------------------------------------
@@ -257,7 +257,7 @@ static void rrdset_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, v
257
st->chart_type = ctr->chart_type;
258
st->rrdhost = host;
259
260
- rrdset_rrdpush_send_chart_slot_assign(st);
260
+ rrdset_stream_send_chart_slot_assign(st);
261
262
spinlock_init(&st->data_collection_lock);
263
@@ -340,7 +340,7 @@ static void rrdset_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, v
340
341
rrdset_finalize_collection(st, false);
342
343
- rrdset_rrdpush_send_chart_slot_release(st);
343
+ rrdset_stream_send_chart_slot_release(st);
344
345
// remove it from the name index
346
rrdset_index_del_name(host, st);
@@ -498,7 +498,7 @@ static void rrdset_react_callback(const DICTIONARY_ITEM *item __maybe_unused, vo
498
void rrdset_index_init(RRDHOST *host) {
499
if(!host->rrdset_root_index) {
500
host->rrdset_root_index = dictionary_create_advanced(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
501
- &dictionary_stats_category_rrdset_rrddim, sizeof(RRDSET));
501
+ &dictionary_stats_category_rrdset, sizeof(RRDSET));
502
503
dictionary_register_insert_callback(host->rrdset_root_index, rrdset_insert_callback, NULL);
504
dictionary_register_conflict_callback(host->rrdset_root_index, rrdset_conflict_callback, NULL);
@@ -509,7 +509,7 @@ void rrdset_index_init(RRDHOST *host) {
509
if(!host->rrdset_root_index_name) {
510
host->rrdset_root_index_name = dictionary_create_advanced(
511
DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_VALUE_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE,
512
- &dictionary_stats_category_rrdset_rrddim, 0);
512
+ &dictionary_stats_category_rrdset, 0);
513
514
dictionary_register_insert_callback(host->rrdset_root_index_name, rrdset_name_insert_callback, host);
515
dictionary_register_delete_callback(host->rrdset_root_index_name, rrdset_name_delete_callback, host);
@@ -749,7 +749,7 @@ inline void rrdset_is_obsolete___safe_from_collector_thread(RRDSET *st) {
749
750
// the chart will not get more updates (data collection)
751
// so, we have to push its definition now
752
- rrdset_push_chart_definition_now(st);
752
+ stream_sender_send_rrdset_definition_now(st);
753
rrdcontext_updated_rrdset_flags(st);
754
}
755
}
@@ -920,7 +920,7 @@ RRDSET *rrdset_create_custom(
920
, long history_entries
921
) {
922
if (host != localhost)
923
- host->child_last_chart_command = now_realtime_sec();
923
+ host->stream.rcv.status.last_chart = now_realtime_sec();
924
925
if(!type || !type[0])
926
fatal("Cannot create rrd stats without a type: id '%s', name '%s', family '%s', context '%s', title '%s', units '%s', plugin '%s', module '%s'."
@@ -1282,18 +1282,18 @@ void rrddim_store_metric(RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n,
1282
// we have not collected this tier before
1283
// let's fill any gap that may exist
1284
rrdr_fill_tier_gap_from_smaller_tiers(rd, tier, now_s);
1285
- rrddim_option_set(rd, RRDDIM_OPTION_BACKFILLED_HIGH_TIERS);
1285
}
1286
1287
store_metric_at_tier(rd, tier, t, sp, point_end_time_ut);
1288
}
1289
+ rrddim_option_set(rd, RRDDIM_OPTION_BACKFILLED_HIGH_TIERS);
1290
1291
rrdcontext_collected_rrddim(rd);
1292
log_stack_pop(&lgs);
1293
}
1294
1295
void store_metric_collection_completed() {
1296
- global_statistics_rrdset_done_chart_collection_completed(rrdset_done_statistics_points_stored_per_tier);
1296
+ telemetry_queries_rrdset_collection_completed(rrdset_done_statistics_points_stored_per_tier);
1297
}
1298
1299
// caching of dimensions rrdset_done() and rrdset_done_interpolate() loop through
@@ -1464,7 +1464,7 @@ static inline size_t rrdset_done_interpolate(
1464
(void) ml_dimension_is_anomalous(rd, current_time_s, 0, false);
1465
1466
if(rsb->wb && rsb->v2)
1467
- rrddim_push_metrics_v2(rsb, rd, next_store_ut, NAN, SN_FLAG_NONE);
1467
+ stream_send_rrddim_metrics_v2(rsb, rd, next_store_ut, NAN, SN_FLAG_NONE);
1468
1469
rrddim_store_metric(rd, next_store_ut, NAN, SN_FLAG_NONE);
1470
continue;
@@ -1479,7 +1479,7 @@ static inline size_t rrdset_done_interpolate(
1479
}
1480
1481
if(rsb->wb && rsb->v2)
1482
- rrddim_push_metrics_v2(rsb, rd, next_store_ut, new_value, dim_storage_flags);
1482
+ stream_send_rrddim_metrics_v2(rsb, rd, next_store_ut, new_value, dim_storage_flags);
1483
1484
rrddim_store_metric(rd, next_store_ut, new_value, dim_storage_flags);
1485
rd->collector.last_stored_value = new_value;
@@ -1490,7 +1490,7 @@ static inline size_t rrdset_done_interpolate(
1490
rrdset_debug(st, "%s: STORE[%ld] = NON EXISTING ", rrddim_name(rd), current_entry);
1491
1492
if(rsb->wb && rsb->v2)
1493
- rrddim_push_metrics_v2(rsb, rd, next_store_ut, NAN, SN_FLAG_NONE);
1493
+ stream_send_rrddim_metrics_v2(rsb, rd, next_store_ut, NAN, SN_FLAG_NONE);
1494
1495
rrddim_store_metric(rd, next_store_ut, NAN, SN_FLAG_NONE);
1496
rd->collector.last_stored_value = NAN;
@@ -1534,8 +1534,8 @@ void rrdset_timed_done(RRDSET *st, struct timeval now, bool pending_rrdset_next)
1534
if(unlikely(!service_running(SERVICE_COLLECTORS))) return;
1535
1536
RRDSET_STREAM_BUFFER stream_buffer = { .wb = NULL, };
1537
- if(unlikely(rrdhost_has_rrdpush_sender_enabled(st->rrdhost)))
1538
- stream_buffer = rrdset_push_metric_initialize(st, now.tv_sec);
1537
+ if(unlikely(rrdhost_has_stream_sender_enabled(st->rrdhost)))
1538
+ stream_buffer = stream_send_metrics_init(st, now.tv_sec);
1539
1540
spinlock_lock(&st->data_collection_lock);
1541
@@ -1660,7 +1660,7 @@ void rrdset_timed_done(RRDSET *st, struct timeval now, bool pending_rrdset_next)
1660
st->counter_done++;
1661
1662
if(stream_buffer.wb && !stream_buffer.v2)
1663
- rrdset_push_metrics_v1(&stream_buffer, st);
1663
+ stream_send_rrdset_metrics_v1(&stream_buffer, st);
1664
1665
size_t rda_slots = dictionary_entries(st->rrddim_root_index);
1666
struct rda_item *rda_base = rrdset_thread_rda_get(&rda_slots);
@@ -1988,7 +1988,7 @@ void rrdset_timed_done(RRDSET *st, struct timeval now, bool pending_rrdset_next)
1988
}
1989
1990
spinlock_unlock(&st->data_collection_lock);
1991
- rrdset_push_metrics_finished(&stream_buffer, st);
1991
+ stream_send_rrdset_metrics_finished(&stream_buffer, st);
1992
1993
// ALL DONE ABOUT THE DATA UPDATE
1994
// --------------------------------------------------------------------
src/database/sqlite/sqlite_aclk.c
+15
-23
@@ -116,7 +116,7 @@ static int create_host_callback(void *data, int argc, char **argv, char **column
116
struct rrdhost_system_info *system_info = callocz(1, sizeof(struct rrdhost_system_info));
117
__atomic_sub_fetch(&netdata_buffers_statistics.rrdhost_allocations_size, sizeof(struct rrdhost_system_info), __ATOMIC_RELAXED);
118
119
- system_info->hops = str2i((const char *) argv[IDX_HOPS]);
119
+ system_info->hops = (int16_t)str2i((const char *) argv[IDX_HOPS]);
120
121
sql_build_host_system_info((nd_uuid_t *)argv[IDX_HOST_ID], system_info);
122
@@ -133,22 +133,14 @@ static int create_host_callback(void *data, int argc, char **argv, char **column
133
argv[IDX_UPDATE_EVERY] ? str2i(argv[IDX_UPDATE_EVERY]) : 1,
134
argv[IDX_ENTRIES] ? str2i(argv[IDX_ENTRIES]) : 0,
135
default_rrd_memory_mode,
136
- 0 // health
137
- ,
138
- 0 // rrdpush enabled
139
- ,
140
- NULL //destination
141
- ,
142
- NULL // api key
143
- ,
144
- NULL // send charts matching
145
- ,
146
- false // rrdpush_enable_replication
147
- ,
148
- 0 // rrdpush_seconds_to_replicate
149
- ,
150
- 0 // rrdpush_replication_step
151
- ,
136
+ 0, // health
137
+ 0, // rrdpush enabled
138
+ NULL, // destination
139
+ NULL, // api key
140
+ NULL, // send charts matching
141
+ false, // rrdpush_enable_replication
142
+ 0, // rrdpush_seconds_to_replicate
143
+ 0, // rrdpush_replication_step
144
system_info,
145
1);
146
@@ -157,10 +149,10 @@ static int create_host_callback(void *data, int argc, char **argv, char **column
149
rrdhost_option_set(host, RRDHOST_OPTION_EPHEMERAL_HOST);
150
151
if (is_ephemeral)
160
- host->child_disconnected_time = now_realtime_sec();
152
+ host->stream.rcv.status.last_disconnected = now_realtime_sec();
153
154
host->rrdlabels = sql_load_host_labels((nd_uuid_t *)argv[IDX_HOST_ID]);
163
- host->last_connected = last_connected;
155
+ host->stream.snd.status.last_connected = last_connected;
156
}
157
158
(*number_of_chidren)++;
@@ -361,9 +353,9 @@ static void node_update_timer_cb(uv_timer_t *handle)
353
struct aclk_sync_cfg_t *ahc = handle->data;
354
RRDHOST *host = ahc->host;
355
364
- spinlock_lock(&host->receiver_lock);
356
+ rrdhost_receiver_lock(host);
357
int live = (host == localhost || host->receiver || !(rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN))) ? 1 : 0;
366
- spinlock_unlock(&host->receiver_lock);
358
+ rrdhost_receiver_unlock(host);
359
nd_log(NDLS_ACLK, NDLP_DEBUG,"Timer: Sending node update info for %s, LIVE = %d", rrdhost_hostname(host), live);
360
aclk_host_state_update(host, live, 1);
361
}
@@ -461,9 +453,9 @@ static void aclk_synchronization(void *arg)
453
}
454
455
// This is fallback if timer fails
464
- spinlock_lock(&host->receiver_lock);
456
+ rrdhost_receiver_lock(host);
457
int live = (host == localhost || host->receiver || !(rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN))) ? 1 : 0;
466
- spinlock_unlock(&host->receiver_lock);
458
+ rrdhost_receiver_unlock(host);
459
aclk_host_state_update(host, live, 1);
460
nd_log(NDLS_ACLK, NDLP_DEBUG,"Sending node update info for %s, LIVE = %d", rrdhost_hostname(host), live);
461
break;
src/database/sqlite/sqlite_aclk_alert.c
+30
-31
@@ -18,9 +18,9 @@
18
19
static inline bool is_event_from_alert_variable_config(int64_t unique_id, nd_uuid_t *host_id)
20
{
21
- static __thread sqlite3_stmt *res = NULL;
21
+ sqlite3_stmt *res = NULL;
22
23
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_SELECT_VARIABLE_ALERT_BY_UNIQUE_ID, &res))
23
+ if (!PREPARE_STATEMENT(db_meta, SQL_SELECT_VARIABLE_ALERT_BY_UNIQUE_ID, &res))
24
return false;
25
26
bool ret = false;
@@ -34,20 +34,18 @@ static inline bool is_event_from_alert_variable_config(int64_t unique_id, nd_uui
34
35
done:
36
REPORT_BIND_FAIL(res, param);
37
- SQLITE_RESET(res);
37
+ SQLITE_FINALIZE(res);
38
return ret;
39
}
40
41
-#define MAX_REMOVED_PERIOD 604800 //a week
42
-
41
#define SQL_UPDATE_ALERT_VERSION_TRANSITION \
42
"UPDATE alert_version SET unique_id = @unique_id WHERE health_log_id = @health_log_id"
43
44
static void update_alert_version_transition(int64_t health_log_id, int64_t unique_id)
45
{
48
- static __thread sqlite3_stmt *res = NULL;
46
+ sqlite3_stmt *res = NULL;
47
50
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_UPDATE_ALERT_VERSION_TRANSITION, &res))
48
+ if (!PREPARE_STATEMENT(db_meta, SQL_UPDATE_ALERT_VERSION_TRANSITION, &res))
49
return;
50
51
int param = 0;
@@ -61,7 +59,7 @@ static void update_alert_version_transition(int64_t health_log_id, int64_t uniqu
59
60
done:
61
REPORT_BIND_FAIL(res, param);
64
- SQLITE_RESET(res);
62
+ SQLITE_FINALIZE(res);
63
}
64
65
//decide if some events should be sent or not
@@ -70,9 +68,9 @@ done:
68
69
static bool cloud_status_matches(int64_t health_log_id, RRDCALC_STATUS status)
70
{
73
- static __thread sqlite3_stmt *res = NULL;
71
+ sqlite3_stmt *res = NULL;
72
75
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_SELECT_LAST_ALERT_STATUS, &res))
73
+ if (!PREPARE_STATEMENT(db_meta, SQL_SELECT_LAST_ALERT_STATUS, &res))
74
return true;
75
76
bool send = false;
@@ -89,7 +87,7 @@ static bool cloud_status_matches(int64_t health_log_id, RRDCALC_STATUS status)
87
88
done:
89
REPORT_BIND_FAIL(res, param);
92
- SQLITE_RESET(res);
90
+ SQLITE_FINALIZE(res);
91
return send;
92
}
93
@@ -108,7 +106,7 @@ done:
106
//
107
static int insert_alert_to_submit_queue(RRDHOST *host, int64_t health_log_id, uint32_t unique_id, RRDCALC_STATUS status)
108
{
111
- static __thread sqlite3_stmt *res = NULL;
109
+ sqlite3_stmt *res = NULL;
110
111
if (cloud_status_matches(health_log_id, status)) {
112
update_alert_version_transition(health_log_id, unique_id);
@@ -118,7 +116,7 @@ static int insert_alert_to_submit_queue(RRDHOST *host, int64_t health_log_id, ui
116
if (is_event_from_alert_variable_config(unique_id, &host->host_id.uuid))
117
return 2;
118
121
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_QUEUE_ALERT_TO_CLOUD, &res))
119
+ if (!PREPARE_STATEMENT(db_meta, SQL_QUEUE_ALERT_TO_CLOUD, &res))
120
return -1;
121
122
int param = 0;
@@ -133,7 +131,7 @@ static int insert_alert_to_submit_queue(RRDHOST *host, int64_t health_log_id, ui
131
132
done:
133
REPORT_BIND_FAIL(res, param);
136
- SQLITE_RESET(res);
134
+ SQLITE_FINALIZE(res);
135
return 0;
136
}
137
@@ -145,9 +143,9 @@ done:
143
//
144
static int delete_alert_from_submit_queue(RRDHOST *host, int64_t first_seq_id, int64_t last_seq_id)
145
{
148
- static __thread sqlite3_stmt *res = NULL;
146
+ sqlite3_stmt *res = NULL;
147
150
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_DELETE_QUEUE_ALERT_TO_CLOUD, &res))
148
+ if (!PREPARE_STATEMENT(db_meta, SQL_DELETE_QUEUE_ALERT_TO_CLOUD, &res))
149
return -1;
150
151
int param = 0;
@@ -162,7 +160,7 @@ static int delete_alert_from_submit_queue(RRDHOST *host, int64_t first_seq_id, i
160
161
done:
162
REPORT_BIND_FAIL(res, param);
165
- SQLITE_RESET(res);
163
+ SQLITE_FINALIZE(res);
164
return 0;
165
}
166
@@ -224,9 +222,9 @@ static inline char *sqlite3_text_strdupz_empty(sqlite3_stmt *res, int iCol) {
222
//
223
static void sql_update_alert_version(int64_t health_log_id, int64_t unique_id, RRDCALC_STATUS status, uint64_t version)
224
{
227
- static __thread sqlite3_stmt *res = NULL;
225
+ sqlite3_stmt *res = NULL;
226
229
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_UPDATE_ALERT_VERSION, &res))
227
+ if (!PREPARE_STATEMENT(db_meta, SQL_UPDATE_ALERT_VERSION, &res))
228
return;
229
230
int param = 0;
@@ -242,7 +240,7 @@ static void sql_update_alert_version(int64_t health_log_id, int64_t unique_id, R
240
241
done:
242
REPORT_BIND_FAIL(res, param);
245
- SQLITE_RESET(res);
243
+ SQLITE_FINALIZE(res);
244
}
245
246
#define SQL_SELECT_ALERT_TO_DUMMY \
@@ -355,7 +353,7 @@ void health_alarm_log_populate(
353
alarm_log->timezone = strdupz(rrdhost_abbrev_timezone(host));
354
alarm_log->exec_path = sqlite3_column_bytes(res, EXEC) ?
355
strdupz((char *)sqlite3_column_text(res, EXEC)) :
358
- strdupz((char *)string2str(host->health.health_default_exec));
356
+ strdupz((char *)string2str(host->health.default_exec));
357
358
alarm_log->conf_source = source ? strdupz(source) : strdupz("");
359
@@ -422,6 +420,7 @@ void health_alarm_log_populate(
420
" ORDER BY aq.sequence_id ASC LIMIT "ACLK_MAX_ALERT_UPDATES
421
422
static void aclk_push_alert_event(RRDHOST *host __maybe_unused)
423
+
424
{
425
CLAIM_ID claim_id = claim_id_get();
426
@@ -487,9 +486,9 @@ done:
486
487
static void delete_alert_from_pending_queue(RRDHOST *host, int64_t row)
488
{
490
- static __thread sqlite3_stmt *res = NULL;
489
+ sqlite3_stmt *res = NULL;
490
492
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_DELETE_PROCESSED_ROWS, &res))
491
+ if (!PREPARE_STATEMENT(db_meta, SQL_DELETE_PROCESSED_ROWS, &res))
492
return;
493
494
int param = 0;
@@ -503,7 +502,7 @@ static void delete_alert_from_pending_queue(RRDHOST *host, int64_t row)
502
503
done:
504
REPORT_BIND_FAIL(res, param);
506
- SQLITE_RESET(res);
505
+ SQLITE_FINALIZE(res);
506
}
507
508
#define SQL_REBUILD_HOST_ALERT_VERSION_TABLE \
@@ -555,9 +554,9 @@ done:
554
555
bool process_alert_pending_queue(RRDHOST *host)
556
{
558
- static __thread sqlite3_stmt *res = NULL;
557
+ sqlite3_stmt *res = NULL;
558
560
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_PROCESS_ALERT_PENDING_QUEUE, &res))
559
+ if (!PREPARE_STATEMENT(db_meta, SQL_PROCESS_ALERT_PENDING_QUEUE, &res))
560
return false;
561
562
int param = 0;
@@ -587,7 +586,7 @@ bool process_alert_pending_queue(RRDHOST *host)
586
nd_log(NDLS_ACCESS, NDLP_NOTICE, "ACLK STA [%s (N/A)]: Processed %d entries, queued %d", rrdhost_hostname(host), count, added);
587
done:
588
REPORT_BIND_FAIL(res, param);
590
- SQLITE_RESET(res);
589
+ SQLITE_FINALIZE(res);
590
return added > 0;
591
}
592
@@ -766,9 +765,9 @@ done:
765
766
static uint64_t calculate_node_alert_version(RRDHOST *host)
767
{
769
- static __thread sqlite3_stmt *res = NULL;
768
+ sqlite3_stmt *res = NULL;
769
771
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_ALERT_VERSION_CALC, &res))
770
+ if (!PREPARE_STATEMENT(db_meta, SQL_ALERT_VERSION_CALC, &res))
771
return 0;
772
773
uint64_t version = 0;
@@ -782,7 +781,7 @@ static uint64_t calculate_node_alert_version(RRDHOST *host)
781
782
done:
783
REPORT_BIND_FAIL(res, param);
785
- SQLITE_RESET(res);
784
+ SQLITE_FINALIZE(res);
785
return version;
786
}
787
@@ -1008,7 +1007,7 @@ void aclk_start_alert_streaming(char *node_id, uint64_t cloud_version)
1007
return;
1008
}
1009
1011
- if (unlikely(!host->health.health_enabled)) {
1010
+ if (unlikely(!host->health.enabled)) {
1011
nd_log(NDLS_ACCESS, NDLP_NOTICE, "ACLK STA [%s (N/A)]: Ignoring request to stream alert state changes, health is disabled.", node_id);
1012
return;
1013
}
src/database/sqlite/sqlite_aclk_node.c
+2
-7
@@ -65,13 +65,8 @@ static void build_node_info(RRDHOST *host)
65
now_realtime_timeval(&node_info.updated_at);
66
67
char *host_version = NULL;
68
- if (host != localhost) {
69
- spinlock_lock(&host->receiver_lock);
70
- host_version = strdupz(
71
- host->receiver && host->receiver->program_version ? host->receiver->program_version :
72
- rrdhost_program_version(host));
73
- spinlock_unlock(&host->receiver_lock);
74
- }
68
+ if (host != localhost)
69
+ host_version = stream_receiver_program_version_strdupz(host);
70
71
node_info.data.name = rrdhost_hostname(host);
72
node_info.data.os = rrdhost_os(host);
src/database/sqlite/sqlite_functions.c
+4
-4
@@ -13,7 +13,7 @@ SQLITE_API int sqlite3_exec_monitored(
13
char **errmsg /* Error msg written here */
14
) {
15
int rc = sqlite3_exec(db, sql, callback, data, errmsg);
16
- global_statistics_sqlite3_query_completed(rc == SQLITE_OK, rc == SQLITE_BUSY, rc == SQLITE_LOCKED);
16
+ telemetry_sqlite3_query_completed(rc == SQLITE_OK, rc == SQLITE_BUSY, rc == SQLITE_LOCKED);
17
return rc;
18
}
19
@@ -25,14 +25,14 @@ SQLITE_API int sqlite3_step_monitored(sqlite3_stmt *stmt) {
25
rc = sqlite3_step(stmt);
26
switch (rc) {
27
case SQLITE_DONE:
28
- global_statistics_sqlite3_query_completed(1, 0, 0);
28
+ telemetry_sqlite3_query_completed(1, 0, 0);
29
break;
30
case SQLITE_ROW:
31
- global_statistics_sqlite3_row_completed();
31
+ telemetry_sqlite3_row_completed();
32
break;
33
case SQLITE_BUSY:
34
case SQLITE_LOCKED:
35
- global_statistics_sqlite3_query_completed(false, rc == SQLITE_BUSY, rc == SQLITE_LOCKED);
35
+ telemetry_sqlite3_query_completed(false, rc == SQLITE_BUSY, rc == SQLITE_LOCKED);
36
usleep(SQLITE_INSERT_DELAY * USEC_PER_MS);
37
continue;
38
default:
src/database/sqlite/sqlite_health.c
+15
-18
@@ -26,12 +26,11 @@
26
27
static void sql_health_alarm_log_update(RRDHOST *host, ALARM_ENTRY *ae)
28
{
29
- static __thread sqlite3_stmt *res = NULL;
30
- int rc;
29
32
- REQUIRE_DB(db_meta);
30
+ sqlite3_stmt *res = NULL;
31
+ int rc;
32
34
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_UPDATE_HEALTH_LOG, &res))
33
+ if (!PREPARE_STATEMENT(db_meta, SQL_UPDATE_HEALTH_LOG, &res))
34
return;
35
36
int param = 0;
@@ -51,7 +50,7 @@ static void sql_health_alarm_log_update(RRDHOST *host, ALARM_ENTRY *ae)
50
51
done:
52
REPORT_BIND_FAIL(res, param);
54
- SQLITE_RESET(res);
53
+ SQLITE_FINALIZE(res);
54
}
55
56
/* Health related SQL queries
@@ -149,13 +148,13 @@ static void insert_alert_queue(
148
RRDCALC_STATUS old_status,
149
RRDCALC_STATUS new_status)
150
{
152
- static __thread sqlite3_stmt *res = NULL;
151
+ sqlite3_stmt *res = NULL;
152
int rc;
153
154
if (!host->aclk_config)
155
return;
156
158
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_INSERT_ALERT_PENDING_QUEUE, &res))
157
+ if (!PREPARE_STATEMENT(db_meta, SQL_INSERT_ALERT_PENDING_QUEUE, &res))
158
return;
159
160
int submit_delay = calculate_delay(old_status, new_status);
@@ -176,7 +175,7 @@ static void insert_alert_queue(
175
176
done:
177
REPORT_BIND_FAIL(res, param);
179
- SQLITE_RESET(res);
178
+ SQLITE_FINALIZE(res);
179
}
180
181
#define SQL_INSERT_HEALTH_LOG_DETAIL \
@@ -189,10 +188,10 @@ done:
188
189
static void sql_health_alarm_log_insert_detail(RRDHOST *host, uint64_t health_log_id, ALARM_ENTRY *ae)
190
{
192
- static __thread sqlite3_stmt *res = NULL;
191
+ sqlite3_stmt *res = NULL;
192
int rc;
193
195
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_INSERT_HEALTH_LOG_DETAIL, &res))
194
+ if (!PREPARE_STATEMENT(db_meta, SQL_INSERT_HEALTH_LOG_DETAIL, &res))
195
return;
196
197
int param = 0;
@@ -230,7 +229,7 @@ static void sql_health_alarm_log_insert_detail(RRDHOST *host, uint64_t health_lo
229
230
done:
231
REPORT_BIND_FAIL(res, param);
233
- SQLITE_RESET(res);
232
+ SQLITE_FINALIZE(res);
233
}
234
235
#define SQL_INSERT_HEALTH_LOG \
@@ -243,13 +242,11 @@ done:
242
243
static void sql_health_alarm_log_insert(RRDHOST *host, ALARM_ENTRY *ae)
244
{
246
- static __thread sqlite3_stmt *res = NULL;
245
+ sqlite3_stmt *res = NULL;
246
int rc;
247
uint64_t health_log_id;
248
250
- REQUIRE_DB(db_meta);
251
-
252
- if (!PREPARE_COMPILED_STATEMENT(db_meta, SQL_INSERT_HEALTH_LOG, &res))
249
+ if (!PREPARE_STATEMENT(db_meta, SQL_INSERT_HEALTH_LOG, &res))
250
return;
251
252
int param = 0;
@@ -277,7 +274,7 @@ static void sql_health_alarm_log_insert(RRDHOST *host, ALARM_ENTRY *ae)
274
275
done:
276
REPORT_BIND_FAIL(res, param);
280
- SQLITE_RESET(res);
277
+ SQLITE_FINALIZE(res);
278
}
279
280
void sql_health_alarm_log_save(RRDHOST *host, ALARM_ENTRY *ae)
@@ -1020,8 +1017,8 @@ void sql_health_alarm_log2json(RRDHOST *host, BUFFER *wb, time_t after, const ch
1017
buffer_json_member_add_boolean(wb, "updated", (sqlite3_column_int64(stmt_query, 9) & HEALTH_ENTRY_FLAG_UPDATED));
1018
buffer_json_member_add_int64(wb, "exec_run", (int64_t)sqlite3_column_int64(stmt_query, 10));
1019
buffer_json_member_add_boolean(wb, "exec_failed", (sqlite3_column_int64(stmt_query, 9) & HEALTH_ENTRY_FLAG_EXEC_FAILED));
1023
- buffer_json_member_add_string_or_empty(wb, "exec", sqlite3_column_text(stmt_query, 14) ? (const char *) sqlite3_column_text(stmt_query, 14) : string2str(host->health.health_default_exec));
1024
- buffer_json_member_add_string_or_empty(wb, "recipient", sqlite3_column_text(stmt_query, 15) ? (const char *) sqlite3_column_text(stmt_query, 15) : string2str(host->health.health_default_recipient));
1020
+ buffer_json_member_add_string_or_empty(wb, "exec", sqlite3_column_text(stmt_query, 14) ? (const char *) sqlite3_column_text(stmt_query, 14) : string2str(host->health.default_exec));
1021
+ buffer_json_member_add_string_or_empty(wb, "recipient", sqlite3_column_text(stmt_query, 15) ? (const char *) sqlite3_column_text(stmt_query, 15) : string2str(host->health.default_recipient));
1022
buffer_json_member_add_int64(wb, "exec_code", sqlite3_column_int(stmt_query, 19));
1023
buffer_json_member_add_string_or_empty(wb, "source", sqlite3_column_text(stmt_query, 16) ? (const char *) sqlite3_column_text(stmt_query, 16) : (char *) "Unknown");
1024
buffer_json_member_add_string_or_empty(wb, "command", edit_command);
src/database/sqlite/sqlite_metadata.c
+137
-25
@@ -173,8 +173,7 @@ sqlite3 *db_meta = NULL;
173
#define METADATA_RUNTIME_THRESHOLD (5) // Run time threshold for cleanup task
174
175
#define METADATA_HOST_CHECK_FIRST_CHECK (5) // First check for pending metadata
176
-#define METADATA_HOST_CHECK_INTERVAL (30) // Repeat check for pending metadata
177
-#define METADATA_HOST_CHECK_IMMEDIATE (5) // Repeat immediate run because we have more metadata to write
176
+#define METADATA_HOST_CHECK_INTERVAL (5) // Repeat check for pending metadata
177
#define MAX_METADATA_CLEANUP (500) // Maximum metadata write operations (e.g deletes before retrying)
178
#define METADATA_MAX_BATCH_SIZE (512) // Maximum commands to execute before running the event loop
179
@@ -190,6 +189,8 @@ enum metadata_opcode {
189
METADATA_SCAN_HOSTS,
190
METADATA_LOAD_HOST_CONTEXT,
191
METADATA_DELETE_HOST_CHART_LABELS,
192
+ METADATA_ADD_HOST_AE,
193
+ METADATA_DEL_HOST_AE,
194
METADATA_MAINTENANCE,
195
METADATA_SYNC_SHUTDOWN,
196
METADATA_UNITTEST,
@@ -217,6 +218,7 @@ struct metadata_wc {
218
uv_async_t async;
219
uv_timer_t timer_req;
220
time_t metadata_check_after;
221
+ Pvoid_t ae_DelJudyL;
222
METADATA_FLAG flags;
223
struct completion start_stop_complete;
224
struct completion *scan_complete;
@@ -271,7 +273,7 @@ static inline void set_host_node_id(RRDHOST *host, nd_uuid_t *node_id)
273
else
274
uuid_unparse_lower(*node_id, wc->node_id);
275
274
- rrdpush_receiver_send_node_and_claim_id_to_child(host);
276
+ stream_receiver_send_node_and_claim_id_to_child(host);
277
stream_path_node_id_updated(host);
278
}
279
@@ -958,8 +960,8 @@ static int store_host_metadata(RRDHOST *host)
960
SQLITE_BIND_FAIL(bind_fail, bind_text_null(res, ++param, rrdhost_program_name(host), 1));
961
SQLITE_BIND_FAIL(bind_fail, bind_text_null(res, ++param, rrdhost_program_version(host), 1));
962
SQLITE_BIND_FAIL(bind_fail, sqlite3_bind_int64(res, ++param, host->rrd_history_entries));
961
- SQLITE_BIND_FAIL(bind_fail, sqlite3_bind_int(res, ++param, (int)host->health.health_enabled));
962
- SQLITE_BIND_FAIL(bind_fail, sqlite3_bind_int64(res, ++param, (sqlite3_int64) host->last_connected));
963
+ SQLITE_BIND_FAIL(bind_fail, sqlite3_bind_int(res, ++param, (int)host->health.enabled));
964
+ SQLITE_BIND_FAIL(bind_fail, sqlite3_bind_int64(res, ++param, (sqlite3_int64) host->stream.snd.status.last_connected));
965
966
int store_rc = sqlite3_step_monitored(res);
967
@@ -1602,7 +1604,8 @@ void run_metadata_cleanup(struct metadata_wc *wc)
1604
struct scan_metadata_payload {
1605
uv_work_t request;
1606
struct metadata_wc *wc;
1605
- void *data;
1607
+ void *chart_label_cleanup;
1608
+ void *pending_alert_list;
1609
BUFFER *work_buffer;
1610
uint32_t max_count;
1611
};
@@ -1749,6 +1752,19 @@ static void after_metadata_hosts(uv_work_t *req, int status __maybe_unused)
1752
struct scan_metadata_payload *data = req->data;
1753
struct metadata_wc *wc = data->wc;
1754
1755
+ bool first = false;
1756
+ Word_t Index = 0;
1757
+ Pvoid_t *Pvalue;
1758
+ while ((Pvalue = JudyLFirstThenNext(wc->ae_DelJudyL, &Index, &first))) {
1759
+ ALARM_ENTRY *ae = (ALARM_ENTRY *) Index;
1760
+ if(!__atomic_load_n(&ae->pending_save_count, __ATOMIC_RELAXED)) {
1761
+ health_alarm_log_free_one_nochecks_nounlink(ae);
1762
+ (void) JudyLDel(&wc->ae_DelJudyL, Index, PJE0);
1763
+ first = false;
1764
+ Index = 0;
1765
+ }
1766
+ }
1767
+
1768
metadata_flag_clear(wc, METADATA_FLAG_PROCESSING);
1769
1770
if (unlikely(wc->scan_complete))
@@ -1855,12 +1871,46 @@ static void store_host_and_system_info(RRDHOST *host, size_t *query_counter)
1871
}
1872
}
1873
1858
-struct host_chart_label_cleanup {
1874
+struct judy_list_t {
1875
Pvoid_t JudyL;
1876
Word_t count;
1877
};
1878
1863
-static void do_chart_label_cleanup(struct host_chart_label_cleanup *cl_cleanup_data)
1879
+static void store_alert_transitions(struct judy_list_t *pending_alert_list)
1880
+{
1881
+ if (!pending_alert_list)
1882
+ return;
1883
+
1884
+ usec_t started_ut = now_monotonic_usec(); (void)started_ut;
1885
+
1886
+ size_t entries = pending_alert_list->count;
1887
+ Word_t Index = 0;
1888
+ bool first = true;
1889
+ Pvoid_t *PValue;
1890
+ while ((PValue = JudyLFirstThenNext(pending_alert_list->JudyL, &Index, &first))) {
1891
+ RRDHOST *host = *PValue;
1892
+
1893
+ PValue = JudyLGet(pending_alert_list->JudyL, ++Index, PJE0);
1894
+ ALARM_ENTRY *ae = *PValue;
1895
+
1896
+ sql_health_alarm_log_save(host, ae);
1897
+
1898
+ __atomic_add_fetch(&ae->pending_save_count, -1, __ATOMIC_RELAXED);
1899
+ __atomic_add_fetch(&host->health.pending_transitions, -1, __ATOMIC_RELAXED);
1900
+ }
1901
+ (void) JudyLFreeArray(&pending_alert_list->JudyL, PJE0);
1902
+ freez(pending_alert_list);
1903
+
1904
+ usec_t ended_ut = now_monotonic_usec(); (void)ended_ut;
1905
+ nd_log(
1906
+ NDLS_DAEMON,
1907
+ NDLP_DEBUG,
1908
+ "Stored and processed %zu alert transitions in %0.2f ms",
1909
+ entries,
1910
+ (double)(ended_ut - started_ut) / USEC_PER_MS);
1911
+}
1912
+
1913
+static void do_chart_label_cleanup(struct judy_list_t *cl_cleanup_data)
1914
{
1915
if (!cl_cleanup_data)
1916
return;
@@ -1900,7 +1950,8 @@ static void start_metadata_hosts(uv_work_t *req __maybe_unused)
1950
nd_log(NDLS_DAEMON, NDLP_DEBUG, "Checking all hosts started");
1951
usec_t started_ut = now_monotonic_usec(); (void)started_ut;
1952
1903
- do_chart_label_cleanup((struct host_chart_label_cleanup *) data->data);
1953
+ store_alert_transitions((struct judy_list_t *)data->pending_alert_list);
1954
+ do_chart_label_cleanup((struct judy_list_t *)data->chart_label_cleanup);
1955
1956
bool run_again = false;
1957
worker_is_busy(UV_EVENT_METADATA_STORE);
@@ -1909,6 +1960,7 @@ static void start_metadata_hosts(uv_work_t *req __maybe_unused)
1960
transaction_started = !db_execute(db_meta, "BEGIN TRANSACTION");
1961
1962
dfe_start_reentrant(rrdhost_root_index, host) {
1963
+
1964
if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED) || !rrdhost_flag_check(host, RRDHOST_FLAG_METADATA_UPDATE))
1965
continue;
1966
@@ -1988,12 +2040,10 @@ static void start_metadata_hosts(uv_work_t *req __maybe_unused)
2040
"Checking all hosts completed in %0.2f ms",
2041
(double)(all_ended_ut - all_started_ut) / USEC_PER_MS);
2042
1991
- if (unlikely(run_again))
1992
- wc->metadata_check_after = now_realtime_sec() + METADATA_HOST_CHECK_IMMEDIATE;
1993
- else {
1994
- wc->metadata_check_after = now_realtime_sec() + METADATA_HOST_CHECK_INTERVAL;
2043
+ if (likely(!run_again))
2044
run_metadata_cleanup(wc);
1996
- }
2045
+
2046
+ wc->metadata_check_after = now_realtime_sec() + METADATA_HOST_CHECK_INTERVAL;
2047
worker_is_idle();
2048
}
2049
@@ -2049,11 +2099,15 @@ static void metadata_event_loop(void *arg)
2099
completion_mark_complete(&wc->start_stop_complete);
2100
BUFFER *work_buffer = buffer_create(1024, &netdata_buffers_statistics.buffers_sqlite);
2101
struct scan_metadata_payload *data;
2052
- struct host_chart_label_cleanup *cl_cleanup_data = NULL;
2102
+ struct judy_list_t *cl_cleanup_data = NULL;
2103
+ Pvoid_t *PValue;
2104
+ struct judy_list_t *pending_ae_list = NULL;
2105
2106
while (shutdown == 0 || (wc->flags & METADATA_FLAG_PROCESSING)) {
2107
nd_uuid_t *uuid;
2108
RRDHOST *host = NULL;
2109
+ ALARM_ENTRY *ae = NULL;
2110
+// struct aclk_sync_cfg_t *host_aclk_sync;
2111
2112
worker_is_idle();
2113
uv_run(loop, UV_RUN_DEFAULT);
@@ -2106,9 +2160,11 @@ static void metadata_event_loop(void *arg)
2160
data = mallocz(sizeof(*data));
2161
data->request.data = data;
2162
data->wc = wc;
2109
- data->data = cl_cleanup_data;
2163
+ data->chart_label_cleanup = cl_cleanup_data;
2164
+ data->pending_alert_list = pending_ae_list;
2165
data->work_buffer = work_buffer;
2166
cl_cleanup_data = NULL;
2167
+ pending_ae_list = NULL;
2168
2169
if (unlikely(cmd.completion)) {
2170
data->max_count = 0; // 0 will process all pending updates
@@ -2118,13 +2174,11 @@ static void metadata_event_loop(void *arg)
2174
data->max_count = 5000;
2175
2176
metadata_flag_set(wc, METADATA_FLAG_PROCESSING);
2121
- if (unlikely(
2122
- uv_queue_work(loop,&data->request,
2123
- start_metadata_hosts,
2124
- after_metadata_hosts))) {
2177
+ if (uv_queue_work(loop, &data->request, start_metadata_hosts, after_metadata_hosts)) {
2178
// Failed to launch worker -- let the event loop handle completion
2179
cmd.completion = wc->scan_complete;
2127
- cl_cleanup_data = data->data;
2180
+ cl_cleanup_data = data->chart_label_cleanup;
2181
+ pending_ae_list = data->pending_alert_list;
2182
freez(data);
2183
metadata_flag_clear(wc, METADATA_FLAG_PROCESSING);
2184
}
@@ -2136,9 +2190,7 @@ static void metadata_event_loop(void *arg)
2190
data = callocz(1,sizeof(*data));
2191
data->request.data = data;
2192
data->wc = wc;
2139
- if (unlikely(
2140
- uv_queue_work(loop,&data->request, start_all_host_load_context,
2141
- after_start_host_load_context))) {
2193
+ if (uv_queue_work(loop, &data->request, start_all_host_load_context, after_start_host_load_context)) {
2194
freez(data);
2195
}
2196
break;
@@ -2146,10 +2198,28 @@ static void metadata_event_loop(void *arg)
2198
if (!cl_cleanup_data)
2199
cl_cleanup_data = callocz(1,sizeof(*cl_cleanup_data));
2200
2149
- Pvoid_t *PValue = JudyLIns(&cl_cleanup_data->JudyL, (Word_t) ++cl_cleanup_data->count, PJE0);
2201
+ PValue = JudyLIns(&cl_cleanup_data->JudyL, (Word_t) ++cl_cleanup_data->count, PJE0);
2202
if (PValue)
2203
*PValue = (void *) cmd.param[0];
2204
2205
+ break;
2206
+ case METADATA_ADD_HOST_AE:
2207
+ host = (RRDHOST *) cmd.param[0];
2208
+ ae = (ALARM_ENTRY *) cmd.param[1];
2209
+
2210
+ if (!pending_ae_list)
2211
+ pending_ae_list = callocz(1, sizeof(*pending_ae_list));
2212
+
2213
+ PValue = JudyLIns(&pending_ae_list->JudyL, ++pending_ae_list->count, PJE0);
2214
+ if (PValue)
2215
+ *PValue = (void *)host;
2216
+
2217
+ PValue = JudyLIns(&pending_ae_list->JudyL, ++pending_ae_list->count, PJE0);
2218
+ if (PValue)
2219
+ *PValue = (void *)ae;
2220
+ break;
2221
+ case METADATA_DEL_HOST_AE:;
2222
+ (void) JudyLIns(&wc->ae_DelJudyL, (Word_t) (void *) cmd.param[0], PJE0);
2223
break;
2224
case METADATA_UNITTEST:;
2225
struct thread_unittest *tu = (struct thread_unittest *) cmd.param[0];
@@ -2335,6 +2405,31 @@ void metadata_delete_host_chart_labels(char *machine_guid)
2405
nd_log(NDLS_DAEMON, NDLP_DEBUG, "Queued command delete chart labels for host %s", machine_guid);
2406
}
2407
2408
+void metadata_queue_ae_save(RRDHOST *host, ALARM_ENTRY *ae)
2409
+{
2410
+ if (unlikely(!metasync_worker.loop))
2411
+ return;
2412
+ __atomic_add_fetch(&host->health.pending_transitions, 1, __ATOMIC_RELAXED);
2413
+ __atomic_add_fetch(&ae->pending_save_count, 1, __ATOMIC_RELAXED);
2414
+ queue_metadata_cmd(METADATA_ADD_HOST_AE, host, ae);
2415
+}
2416
+
2417
+void metadata_queue_ae_deletion(ALARM_ENTRY *ae)
2418
+{
2419
+ if (unlikely(!metasync_worker.loop))
2420
+ return;
2421
+
2422
+ queue_metadata_cmd(METADATA_DEL_HOST_AE, ae, NULL);
2423
+}
2424
+
2425
+void commit_alert_transitions(RRDHOST *host __maybe_unused)
2426
+{
2427
+ if (unlikely(!metasync_worker.loop))
2428
+ return;
2429
+
2430
+ queue_metadata_cmd(METADATA_SCAN_HOSTS, NULL, NULL);
2431
+}
2432
+
2433
uint64_t sqlite_get_meta_space(void)
2434
{
2435
return sqlite_get_db_space(db_meta);
@@ -2378,6 +2473,15 @@ void cleanup_agent_event_log(void)
2473
2474
usec_t get_agent_event_time_median(event_log_type_t event_id)
2475
{
2476
+ static bool initialized[EVENT_AGENT_MAX] = { 0 };
2477
+ static usec_t median[EVENT_AGENT_MAX] = { 0 };
2478
+
2479
+ if(event_id >= EVENT_AGENT_MAX)
2480
+ return 0;
2481
+
2482
+ if(initialized[event_id])
2483
+ return median[event_id];
2484
+
2485
sqlite3_stmt *res = NULL;
2486
if (!PREPARE_STATEMENT(db_meta, SQL_GET_AGENT_EVENT_TYPE_MEDIAN, &res))
2487
return 0;
@@ -2393,9 +2497,17 @@ usec_t get_agent_event_time_median(event_log_type_t event_id)
2497
done:
2498
REPORT_BIND_FAIL(res, param);
2499
SQLITE_FINALIZE(res);
2500
+
2501
+ median[event_id] = avg_time;
2502
+ initialized[event_id] = true;
2503
return avg_time;
2504
}
2505
2506
+void get_agent_event_time_median_init(void) {
2507
+ for(event_log_type_t event_id = 1; event_id < EVENT_AGENT_MAX; event_id++)
2508
+ get_agent_event_time_median(event_id);
2509
+}
2510
+
2511
//
2512
// unitests
2513
//
src/database/sqlite/sqlite_metadata.h
+7
@@ -9,7 +9,11 @@
9
typedef enum event_log_type {
10
EVENT_AGENT_START_TIME = 1,
11
EVENT_AGENT_SHUTDOWN_TIME,
12
+
13
+ // terminator
14
+ EVENT_AGENT_MAX,
15
} event_log_type_t;
16
+void get_agent_event_time_median_init(void);
17
18
// return a node list
19
struct node_instance_list {
@@ -62,6 +66,9 @@ int sql_init_meta_database(db_check_action_type_t rebuild, int memory);
66
void cleanup_agent_event_log(void);
67
void add_agent_event(event_log_type_t event_id, int64_t value);
68
usec_t get_agent_event_time_median(event_log_type_t event_id);
69
+void metadata_queue_ae_save(RRDHOST *host, ALARM_ENTRY *ae);
70
+void metadata_queue_ae_deletion(ALARM_ENTRY *ae);
71
+void commit_alert_transitions(RRDHOST *host);
72
73
// UNIT TEST
74
int metadata_unittest(void);
src/exporting/process_data.c
+1
-1
@@ -139,7 +139,7 @@ NETDATA_DOUBLE exporting_calculate_value_from_stored_data(
139
counter += sp.count;
140
}
141
storage_engine_query_finalize(&handle);
142
- global_statistics_exporters_query_completed(points_read);
142
+ telemetry_queries_exporters_query_completed(points_read);
143
144
if (unlikely(!counter)) {
145
netdata_log_debug(
src/exporting/send_internal_metrics.c
+3
-3
@@ -11,7 +11,7 @@
11
*/
12
void create_main_rusage_chart(RRDSET **st_rusage, RRDDIM **rd_user, RRDDIM **rd_system)
13
{
14
- if (!global_statistics_enabled)
14
+ if (!telemetry_enabled)
15
return;
16
17
if (*st_rusage && *rd_user && *rd_system)
@@ -44,7 +44,7 @@ void create_main_rusage_chart(RRDSET **st_rusage, RRDDIM **rd_user, RRDDIM **rd_
44
*/
45
void send_main_rusage(RRDSET *st_rusage, RRDDIM *rd_user, RRDDIM *rd_system)
46
{
47
- if (!global_statistics_enabled)
47
+ if (!telemetry_enabled)
48
return;
49
50
struct rusage thread;
@@ -65,7 +65,7 @@ void send_main_rusage(RRDSET *st_rusage, RRDDIM *rd_user, RRDDIM *rd_system)
65
*/
66
void send_internal_metrics(struct instance *instance)
67
{
68
- if (!global_statistics_enabled)
68
+ if (!telemetry_enabled)
69
return;
70
71
struct stats *stats = &instance->stats;
src/health/health_dyncfg.c
+1
-1
@@ -509,7 +509,7 @@ int dyncfg_health_prototype_to_conf(BUFFER *wb, RRD_ALERT_PROTOTYPE *ap, const c
509
if(nap->config.info)
510
buffer_sprintf(wb, "%13s: %s\n", "info", string2str(nap->config.info));
511
512
- if(nap->config.exec && nap->config.exec != localhost->health.health_default_exec)
512
+ if(nap->config.exec && nap->config.exec != localhost->health.default_exec)
513
buffer_sprintf(wb, "%13s: %s\n", "exec", string2str(nap->config.exec));
514
515
if(nap->config.recipient)
src/health/health_event_loop.c
+34
-16
@@ -125,7 +125,7 @@ static void health_execute_delayed_initializations(RRDHOST *host) {
125
static void health_initialize_rrdhost(RRDHOST *host) {
126
health_plugin_init();
127
128
- if(!host->health.health_enabled ||
128
+ if(!host->health.enabled ||
129
rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH) ||
130
!service_running(SERVICE_HEALTH))
131
return;
@@ -134,8 +134,8 @@ static void health_initialize_rrdhost(RRDHOST *host) {
134
135
host->health_log.max = health_globals.config.health_log_entries_max;
136
host->health_log.health_log_retention_s = health_globals.config.health_log_retention_s;
137
- host->health.health_default_exec = string_dup(health_globals.config.default_exec);
138
- host->health.health_default_recipient = string_dup(health_globals.config.default_recipient);
137
+ host->health.default_exec = string_dup(health_globals.config.default_exec);
138
+ host->health.default_recipient = string_dup(health_globals.config.default_recipient);
139
host->health.use_summary_for_notifications = health_globals.config.use_summary_for_notifications;
140
141
host->health_log.next_log_id = get_uint32_id();
@@ -247,9 +247,20 @@ static void health_event_loop(void) {
247
if(unlikely(!service_running(SERVICE_HEALTH)))
248
break;
249
250
- if (unlikely(!host->health.health_enabled))
250
+ if (unlikely(!host->health.enabled))
251
continue;
252
253
+//#define rrdhost_pending_alert_transitions(host) (__atomic_load_n(&((host)->aclk_config.alert_transition.pending), __ATOMIC_RELAXED))
254
+
255
+ if (unlikely(__atomic_load_n(&host->health.pending_transitions, __ATOMIC_RELAXED))) {
256
+ nd_log(
257
+ NDLS_DAEMON,
258
+ NDLP_DEBUG,
259
+ "Host \"%s\" has pending alert transitions to save, postponing health checks",
260
+ rrdhost_hostname(host));
261
+ continue;
262
+ }
263
+
264
if (unlikely(!rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH)))
265
health_initialize_rrdhost(host);
266
@@ -261,12 +272,12 @@ static void health_event_loop(void) {
272
rrdhost_hostname(host),
273
health_globals.config.postpone_alarms_during_hibernation_for_seconds);
274
264
- host->health.health_delay_up_to =
275
+ host->health.delay_up_to =
276
now + health_globals.config.postpone_alarms_during_hibernation_for_seconds;
277
}
278
268
- if (unlikely(host->health.health_delay_up_to)) {
269
- if (unlikely(now < host->health.health_delay_up_to)) {
279
+ if (unlikely(host->health.delay_up_to)) {
280
+ if (unlikely(now < host->health.delay_up_to)) {
281
continue;
282
}
283
@@ -274,12 +285,12 @@ static void health_event_loop(void) {
285
"[%s]: Resuming health checks after delay.",
286
rrdhost_hostname(host));
287
277
- host->health.health_delay_up_to = 0;
288
+ host->health.delay_up_to = 0;
289
}
290
291
// wait until cleanup of obsolete charts on children is complete
292
if (host != localhost) {
282
- if (unlikely(host->trigger_chart_obsoletion_check == 1)) {
293
+ if (unlikely(host->stream.rcv.status.check_obsolete)) {
294
295
nd_log(NDLS_DAEMON, NDLP_DEBUG,
296
"[%s]: Waiting for chart obsoletion check.",
@@ -649,14 +660,21 @@ static void health_event_loop(void) {
660
break;
661
}
662
}
652
- struct aclk_sync_cfg_t *wc = host->aclk_config;
653
- if (wc && wc->send_snapshot == 1) {
654
- wc->send_snapshot = 2;
655
- rrdhost_flag_set(host, RRDHOST_FLAG_ACLK_STREAM_ALERTS);
656
- }
657
- else
658
- if (process_alert_pending_queue(host))
663
+
664
+ int32_t pending = __atomic_load_n(&host->health.pending_transitions, __ATOMIC_RELAXED);
665
+ if (pending)
666
+ commit_alert_transitions(host);
667
+
668
+ if (!__atomic_load_n(&host->health.pending_transitions, __ATOMIC_RELAXED)) {
669
+ struct aclk_sync_cfg_t *wc = host->aclk_config;
670
+ if (wc && wc->send_snapshot == 1) {
671
+ wc->send_snapshot = 2;
672
rrdhost_flag_set(host, RRDHOST_FLAG_ACLK_STREAM_ALERTS);
673
+ } else {
674
+ if (process_alert_pending_queue(host))
675
+ rrdhost_flag_set(host, RRDHOST_FLAG_ACLK_STREAM_ALERTS);
676
+ }
677
+ }
678
679
dfe_done(host);
680
src/health/health_json.c
+3
-3
@@ -88,8 +88,8 @@ static inline void health_rrdcalc2json_nolock(RRDHOST *host, BUFFER *wb, RRDCALC
88
, (rc->rrdset)?"true":"false"
89
, (rc->run_flags & RRDCALC_FLAG_DISABLED)?"true":"false"
90
, (rc->run_flags & RRDCALC_FLAG_SILENCED)?"true":"false"
91
- , rc->config.exec?rrdcalc_exec(rc):string2str(host->health.health_default_exec)
92
- , rc->config.recipient?rrdcalc_recipient(rc):string2str(host->health.health_default_recipient)
91
+ , rc->config.exec?rrdcalc_exec(rc):string2str(host->health.default_exec)
92
+ , rc->config.recipient?rrdcalc_recipient(rc):string2str(host->health.default_recipient)
93
, rrdcalc_source(rc)
94
, rrdcalc_units(rc)
95
, string2str(rc->summary)
@@ -238,7 +238,7 @@ void health_alarms2json(RRDHOST *host, BUFFER *wb, int all) {
238
"\n\t\"alarms\": {\n",
239
rrdhost_hostname(host),
240
(host->health_log.next_log_id > 0)?(host->health_log.next_log_id - 1):0,
241
- host->health.health_enabled?"true":"false",
241
+ host->health.enabled ?"true":"false",
242
(unsigned long)now_realtime_sec());
243
244
health_alarms2json_fill_alarms(host, wb, all, health_rrdcalc2json_nolock);
src/health/health_log.c
+22
-20
@@ -6,10 +6,9 @@
6
7
inline void health_alarm_log_save(RRDHOST *host, ALARM_ENTRY *ae)
8
{
9
- sql_health_alarm_log_save(host, ae);
9
+ metadata_queue_ae_save(host, ae);
10
}
11
12
-
12
void health_log_alert_transition_with_trace(RRDHOST *host, ALARM_ENTRY *ae, int line, const char *file, const char *function) {
13
if(!host || !ae) return;
14
@@ -160,6 +159,7 @@ inline ALARM_ENTRY* health_create_alarm_entry(
159
ae->flags |= flags;
160
161
ae->last_repeat = 0;
162
+ ae->pending_save_count = 0;
163
164
if(ae->old_status == RRDCALC_STATUS_WARNING || ae->old_status == RRDCALC_STATUS_CRITICAL)
165
ae->non_clear_duration += ae->duration;
@@ -167,10 +167,8 @@ inline ALARM_ENTRY* health_create_alarm_entry(
167
return ae;
168
}
169
170
-inline void health_alarm_log_add_entry(
171
- RRDHOST *host,
172
- ALARM_ENTRY *ae
173
-) {
170
+inline void health_alarm_log_add_entry(RRDHOST *host, ALARM_ENTRY *ae)
171
+{
172
netdata_log_debug(D_HEALTH, "Health adding alarm log entry with id: %u", ae->unique_id);
173
174
__atomic_add_fetch(&host->health_transitions, 1, __ATOMIC_RELAXED);
@@ -209,20 +207,24 @@ inline void health_alarm_log_add_entry(
207
}
208
209
inline void health_alarm_log_free_one_nochecks_nounlink(ALARM_ENTRY *ae) {
212
- string_freez(ae->name);
213
- string_freez(ae->chart);
214
- string_freez(ae->chart_context);
215
- string_freez(ae->classification);
216
- string_freez(ae->component);
217
- string_freez(ae->type);
218
- string_freez(ae->exec);
219
- string_freez(ae->recipient);
220
- string_freez(ae->source);
221
- string_freez(ae->units);
222
- string_freez(ae->info);
223
- string_freez(ae->old_value_string);
224
- string_freez(ae->new_value_string);
225
- freez(ae);
210
+ if(__atomic_load_n(&ae->pending_save_count, __ATOMIC_RELAXED))
211
+ metadata_queue_ae_deletion(ae);
212
+ else {
213
+ string_freez(ae->name);
214
+ string_freez(ae->chart);
215
+ string_freez(ae->chart_context);
216
+ string_freez(ae->classification);
217
+ string_freez(ae->component);
218
+ string_freez(ae->type);
219
+ string_freez(ae->exec);
220
+ string_freez(ae->recipient);
221
+ string_freez(ae->source);
222
+ string_freez(ae->units);
223
+ string_freez(ae->info);
224
+ string_freez(ae->old_value_string);
225
+ string_freez(ae->new_value_string);
226
+ freez(ae);
227
+ }
228
}
229
230
inline void health_alarm_log_free(RRDHOST *host) {
src/health/health_notifications.c
+2
-2
@@ -421,8 +421,8 @@ void health_send_notification(RRDHOST *host, ALARM_ENTRY *ae, struct health_rais
421
"[%s]: Sending notification for alarm '%s.%s' status %s.",
422
rrdhost_hostname(host), ae_chart_id(ae), ae_name(ae), rrdcalc_status2string(ae->new_status));
423
424
- const char *exec = (ae->exec) ? ae_exec(ae) : string2str(host->health.health_default_exec);
425
- const char *recipient = (ae->recipient) ? ae_recipient(ae) : string2str(host->health.health_default_recipient);
424
+ const char *exec = (ae->exec) ? ae_exec(ae) : string2str(host->health.default_exec);
425
+ const char *recipient = (ae->recipient) ? ae_recipient(ae) : string2str(host->health.default_recipient);
426
427
char *edit_command = ae->source ? health_edit_command_from_source(ae_source(ae)) : strdupz("UNKNOWN=0=UNKNOWN");
428
src/health/health_prototypes.c
+2
-2
@@ -642,7 +642,7 @@ void health_apply_prototype_to_host(RRDHOST *host, RRD_ALERT_PROTOTYPE *ap) {
642
if(!ap->_internal.enabled)
643
return;
644
645
- if(unlikely(!host->health.health_enabled) && !rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH))
645
+ if(unlikely(!host->health.enabled) && !rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH))
646
return;
647
648
RRDSET *st;
@@ -666,7 +666,7 @@ void health_prototype_apply_to_all_hosts(RRD_ALERT_PROTOTYPE *ap) {
666
// ---------------------------------------------------------------------------------------------------------------------
667
668
void health_apply_prototypes_to_host(RRDHOST *host) {
669
- if(unlikely(!host->health.health_enabled) && !rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH))
669
+ if(unlikely(!host->health.enabled) && !rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH))
670
return;
671
672
// free all running alarms
src/health/rrdvar.c
+1
-1
@@ -85,7 +85,7 @@ void rrdvar_host_variable_set(RRDHOST *host, const RRDVAR_ACQUIRED *rva, NETDATA
85
rv->value = value;
86
87
// if the host is streaming, send this variable upstream immediately
88
- rrdpush_sender_send_this_host_variable_now(host, rva);
88
+ stream_sender_send_this_host_variable_now(host, rva);
89
}
90
}
91
src/libnetdata/aral/aral.c
+494
-288
@@ -1,6 +1,8 @@
1
#include "../libnetdata.h"
2
#include "aral.h"
3
4
+// #define NETDATA_ARAL_INTERNAL_CHECKS 1
5
+
6
#ifdef NETDATA_TRACE_ALLOCATIONS
7
#define TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS , const char *file, const char *function, size_t line
8
#define TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS , file, function, line
@@ -9,15 +11,18 @@
11
#define TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS
12
#endif
13
12
-#define ARAL_FREE_PAGES_DELTA_TO_REARRANGE_LIST 5
13
-
14
// max file size
15
-#define ARAL_MAX_PAGE_SIZE_MMAP (1*1024*1024*1024)
15
+#define ARAL_MAX_PAGE_SIZE_MMAP (1ULL * 1024 * 1024 * 1024)
16
17
// max malloc size
18
// optimal at current versions of libc is up to 256k
19
// ideal to have the same overhead as libc is 4k
20
-#define ARAL_MAX_PAGE_SIZE_MALLOC (65*1024)
20
+#define ARAL_MAX_PAGE_SIZE_MALLOC (256ULL * 1024)
21
+
22
+// we don't need alignof(max_align_t) for normal C structures
23
+// alignof(uintptr_r) is sufficient for our use cases
24
+// #define SYSTEM_REQUIRED_ALIGNMENT (alignof(max_align_t))
25
+#define SYSTEM_REQUIRED_ALIGNMENT (alignof(uintptr_t))
26
27
typedef struct aral_free {
28
size_t size;
@@ -25,19 +30,20 @@ typedef struct aral_free {
30
} ARAL_FREE;
31
32
typedef struct aral_page {
28
- size_t size; // the allocation size of the page
33
+ bool marked;
34
+ uint32_t size; // the allocation size of the page
35
const char *filename;
36
uint8_t *data;
37
32
- uint32_t free_elements_to_move_first;
33
- uint32_t max_elements; // the number of elements that can fit on this page
38
+ uint32_t max_elements; // the number of elements that can fit on this page
39
40
struct {
41
uint32_t used_elements; // the number of used elements on this page
42
uint32_t free_elements; // the number of free elements on this page
43
+ uint32_t marked_elements;
44
39
- struct aral_page *prev; // the prev page on the list
40
- struct aral_page *next; // the next page on the list
45
+ struct aral_page *prev; // the prev page on the list
46
+ struct aral_page *next; // the next page on the list
47
} aral_lock;
48
49
struct {
@@ -48,12 +54,25 @@ typedef struct aral_page {
54
} ARAL_PAGE;
55
56
typedef enum {
51
- ARAL_LOCKLESS = (1 << 0),
52
- ARAL_DEFRAGMENT = (1 << 1),
53
- ARAL_ALLOCATED_STATS = (1 << 2),
57
+ ARAL_LOCKLESS = (1 << 0),
58
+ ARAL_ALLOCATED_STATS = (1 << 1),
59
} ARAL_OPTIONS;
60
61
+struct aral_ops {
62
+ struct {
63
+ alignas(64) size_t allocators; // the number of threads currently trying to allocate memory
64
+ alignas(64) size_t deallocators; // the number of threads currently trying to deallocate memory
65
+ } atomic;
66
+
67
+ struct {
68
+ alignas(64) SPINLOCK spinlock;
69
+ size_t allocating_elements; // currently allocating elements
70
+ size_t allocation_size; // current / next allocation size
71
+ } adders;
72
+};
73
+
74
struct aral {
75
+
76
struct {
77
char name[ARAL_MAX_NAME + 1];
78
@@ -63,7 +82,7 @@ struct aral {
82
size_t max_allocation_size; // calculated in bytes
83
size_t max_page_elements; // calculated
84
size_t page_ptr_offset; // calculated
66
- size_t natural_page_size; // calculated
85
+ size_t system_page_size; // calculated
86
87
size_t initial_page_elements;
88
size_t requested_element_size;
@@ -77,9 +96,14 @@ struct aral {
96
} config;
97
98
struct {
80
- SPINLOCK spinlock;
99
+ alignas(64) SPINLOCK spinlock;
100
size_t file_number; // for mmap
82
- struct aral_page *pages; // linked list of pages
101
+
102
+ ARAL_PAGE *pages_free; // pages with free items
103
+ ARAL_PAGE *pages_full; // pages that are completely full
104
+
105
+ ARAL_PAGE *pages_marked_free; // pages with marked items and free slots
106
+ ARAL_PAGE *pages_marked_full; // pages with marked items completely full
107
108
size_t user_malloc_operations;
109
size_t user_free_operations;
@@ -87,26 +111,42 @@ struct aral {
111
size_t defragment_linked_list_traversals;
112
} aral_lock;
113
90
- struct {
91
- SPINLOCK spinlock;
92
- size_t allocating_elements; // currently allocating elements
93
- size_t allocation_size; // current / next allocation size
94
- } adders;
95
-
96
- struct {
97
- size_t allocators; // the number of threads currently trying to allocate memory
98
- } atomic;
114
+ struct aral_ops ops[2];
115
116
struct aral_statistics *stats;
117
};
118
119
+#define mark_to_idx(marked) (marked ? 1 : 0)
120
+#define aral_pages_head_free(ar, marked) (marked ? &ar->aral_lock.pages_marked_free : &ar->aral_lock.pages_free)
121
+#define aral_pages_head_full(ar, marked) (marked ? &ar->aral_lock.pages_marked_full : &ar->aral_lock.pages_full)
122
+
123
+const char *aral_name(ARAL *ar) {
124
+ return ar->config.name;
125
+}
126
+
127
size_t aral_structures_from_stats(struct aral_statistics *stats) {
128
+ if(!stats) return 0;
129
return __atomic_load_n(&stats->structures.allocated_bytes, __ATOMIC_RELAXED);
130
}
131
132
size_t aral_overhead_from_stats(struct aral_statistics *stats) {
108
- return __atomic_load_n(&stats->malloc.allocated_bytes, __ATOMIC_RELAXED) -
109
- __atomic_load_n(&stats->malloc.used_bytes, __ATOMIC_RELAXED);
133
+ if(!stats) return 0;
134
+
135
+ size_t allocated = __atomic_load_n(&stats->malloc.allocated_bytes, __ATOMIC_RELAXED) +
136
+ __atomic_load_n(&stats->mmap.allocated_bytes, __ATOMIC_RELAXED);
137
+
138
+ size_t used = __atomic_load_n(&stats->malloc.used_bytes, __ATOMIC_RELAXED) +
139
+ __atomic_load_n(&stats->mmap.used_bytes, __ATOMIC_RELAXED);
140
+
141
+ if(allocated > used) return allocated - used;
142
+ return allocated;
143
+}
144
+
145
+size_t aral_used_bytes_from_stats(struct aral_statistics *stats) {
146
+ size_t used = __atomic_load_n(&stats->malloc.used_bytes, __ATOMIC_RELAXED) +
147
+ __atomic_load_n(&stats->mmap.used_bytes, __ATOMIC_RELAXED);
148
+
149
+ return used;
150
}
151
152
size_t aral_overhead(ARAL *ar) {
@@ -121,17 +161,13 @@ struct aral_statistics *aral_get_statistics(ARAL *ar) {
161
return ar->stats;
162
}
163
124
-#define ARAL_NATURAL_ALIGNMENT (sizeof(uintptr_t) * 2)
125
-static inline size_t natural_alignment(size_t size, size_t alignment) {
126
- if(unlikely(size % alignment))
127
- size = size + alignment - (size % alignment);
128
-
129
- return size;
164
+static inline size_t memory_alignment(size_t size, size_t alignment) {
165
+ // return (size + alignment - 1) & ~(alignment - 1); // assumees alignment is power of 2
166
+ return ((size + alignment - 1) / alignment) * alignment;
167
}
168
169
static size_t aral_align_alloc_size(ARAL *ar, uint64_t size) {
133
- if(size % ar->config.natural_page_size)
134
- size += ar->config.natural_page_size - (size % ar->config.natural_page_size) ;
170
+ size = memory_alignment(size, ar->config.system_page_size);
171
172
if(size % ar->config.element_size)
173
size -= size % ar->config.element_size;
@@ -159,21 +195,27 @@ static inline void aral_page_free_unlock(ARAL *ar, ARAL_PAGE *page) {
195
spinlock_unlock(&page->free.spinlock);
196
}
197
162
-static inline bool aral_adders_trylock(ARAL *ar) {
163
- if(likely(!(ar->config.options & ARAL_LOCKLESS)))
164
- return spinlock_trylock(&ar->adders.spinlock);
198
+static inline bool aral_adders_trylock(ARAL *ar, bool marked) {
199
+ if(likely(!(ar->config.options & ARAL_LOCKLESS))) {
200
+ size_t idx = mark_to_idx(marked);
201
+ return spinlock_trylock(&ar->ops[idx].adders.spinlock);
202
+ }
203
204
return true;
205
}
206
169
-static inline void aral_adders_lock(ARAL *ar) {
170
- if(likely(!(ar->config.options & ARAL_LOCKLESS)))
171
- spinlock_lock(&ar->adders.spinlock);
207
+static inline void aral_adders_lock(ARAL *ar, bool marked) {
208
+ if(likely(!(ar->config.options & ARAL_LOCKLESS))) {
209
+ size_t idx = mark_to_idx(marked);
210
+ spinlock_lock(&ar->ops[idx].adders.spinlock);
211
+ }
212
}
213
174
-static inline void aral_adders_unlock(ARAL *ar) {
175
- if(likely(!(ar->config.options & ARAL_LOCKLESS)))
176
- spinlock_unlock(&ar->adders.spinlock);
214
+static inline void aral_adders_unlock(ARAL *ar, bool marked) {
215
+ if(likely(!(ar->config.options & ARAL_LOCKLESS))) {
216
+ size_t idx = mark_to_idx(marked);
217
+ spinlock_unlock(&ar->ops[idx].adders.spinlock);
218
+ }
219
}
220
221
static void aral_delete_leftover_files(const char *name, const char *path, const char *required_prefix) {
@@ -200,38 +242,76 @@ static void aral_delete_leftover_files(const char *name, const char *path, const
242
closedir(dir);
243
}
244
203
-// ----------------------------------------------------------------------------
204
-// check a free slot
245
+// --------------------------------------------------------------------------------------------------------------------
246
206
-#ifdef NETDATA_INTERNAL_CHECKS
207
-static inline void aral_free_validate_internal_check(ARAL *ar, ARAL_FREE *fr) {
208
- if(unlikely(fr->size < ar->config.element_size))
209
- fatal("ARAL: '%s' free item of size %zu, less than the expected element size %zu",
210
- ar->config.name, fr->size, ar->config.element_size);
247
+#ifdef NETDATA_ARAL_INTERNAL_CHECKS
248
+struct free_space {
249
+ size_t pages;
250
+ size_t pages_with_free_elements;
251
+ size_t max_free_elements_on_a_page;
252
+ size_t free_elements;
253
+ size_t max_page_elements;
254
+ ARAL_PAGE *p, *lp;
255
+};
256
212
- if(unlikely(fr->size % ar->config.element_size))
213
- fatal("ARAL: '%s' free item of size %zu is not multiple to element size %zu",
214
- ar->config.name, fr->size, ar->config.element_size);
257
+static inline struct free_space check_free_space___aral_lock_needed(ARAL *ar, ARAL_PAGE *my_page, bool marked) {
258
+ struct free_space f = { 0 };
259
+
260
+ f.max_page_elements = ar->config.max_allocation_size / ar->config.element_size;
261
+ for(f.p = *aral_pages_head_free(ar, marked); f.p ; f.lp = f.p, f.p = f.p->aral_lock.next) {
262
+ f.pages++;
263
+ internal_fatal(!f.p->aral_lock.free_elements, "page is in the free list, but does not have any elements free");
264
+ internal_fatal(f.p->marked != marked, "page is in the wrong mark list");
265
+
266
+ if(f.p != my_page && f.max_free_elements_on_a_page < f.p->aral_lock.free_elements)
267
+ f.max_free_elements_on_a_page = f.p->aral_lock.free_elements;
268
+
269
+ f.free_elements += f.p->aral_lock.free_elements;
270
+ f.pages_with_free_elements++;
271
+ }
272
+
273
+ for(f.p = *aral_pages_head_full(ar, marked); f.p ; f.lp = f.p, f.p = f.p->aral_lock.next) {
274
+ f.pages++;
275
+ internal_fatal(f.p->aral_lock.free_elements, "found page with free items in a full page");
276
+ internal_fatal(f.p->marked != marked, "page is in the wrong mark list");
277
+ }
278
+
279
+ return f;
280
+}
281
+static inline bool is_page_in_list(ARAL_PAGE *head, ARAL_PAGE *page) {
282
+ for(ARAL_PAGE *p = head; p ; p = p->aral_lock.next)
283
+ if(p == page) return true;
284
+ return false;
285
}
286
#else
217
-#define aral_free_validate_internal_check(ar, fr) debug_dummy()
287
+
288
+#define is_page_in_list(head, page) true
289
+
290
#endif
291
220
-// ----------------------------------------------------------------------------
292
+
293
+// --------------------------------------------------------------------------------------------------------------------
294
// find the page a pointer belongs to
295
223
-#ifdef NETDATA_INTERNAL_CHECKS
224
-static inline ARAL_PAGE *find_page_with_allocation_internal_check(ARAL *ar, void *ptr) {
296
+#ifdef NETDATA_ARAL_INTERNAL_CHECKS
297
+static inline ARAL_PAGE *find_page_with_allocation_internal_check(ARAL *ar, void *ptr, bool marked) {
298
aral_lock(ar);
299
300
uintptr_t seeking = (uintptr_t)ptr;
301
ARAL_PAGE *page;
302
230
- for(page = ar->aral_lock.pages; page ; page = page->aral_lock.next) {
231
- if(unlikely(seeking >= (uintptr_t)page->data && seeking < (uintptr_t)page->data + page->size))
303
+ for (page = *aral_pages_head_full(ar, marked); page; page = page->aral_lock.next) {
304
+ if (unlikely(seeking >= (uintptr_t)page->data && seeking < (uintptr_t)page->data + page->size))
305
break;
306
}
307
308
+ if(!page) {
309
+ for(page = *aral_pages_head_free(ar, marked); page ; page = page->aral_lock.next) {
310
+ if(unlikely(seeking >= (uintptr_t)page->data && seeking < (uintptr_t)page->data + page->size))
311
+ break;
312
+ }
313
+ }
314
+
315
aral_unlock(ar);
316
317
return page;
@@ -239,34 +319,78 @@ static inline ARAL_PAGE *find_page_with_allocation_internal_check(ARAL *ar, void
319
#endif
320
321
// ----------------------------------------------------------------------------
242
-// find a page with a free slot (there shouldn't be any)
322
+// Tagging the pointer with the 'marked' flag
323
244
-#ifdef NETDATA_ARAL_INTERNAL_CHECKS
245
-static inline ARAL_PAGE *find_page_with_free_slots_internal_check___with_aral_lock(ARAL *ar) {
246
- ARAL_PAGE *page;
324
+// Retrieving the pointer and the 'marked' flag
325
+static ARAL_PAGE *aral_get_page_pointer_after_element___do_NOT_have_aral_lock(ARAL *ar, void *ptr, bool *marked) {
326
+ uint8_t *data = ptr;
327
+ uintptr_t *page_ptr = (uintptr_t *)&data[ar->config.page_ptr_offset];
328
+ uintptr_t tagged_page = __atomic_load_n(page_ptr, __ATOMIC_ACQUIRE); // Atomically load the tagged pointer
329
+ *marked = (tagged_page & 1) != 0; // Extract the LSB as the 'marked' flag
330
+ ARAL_PAGE *page = (ARAL_PAGE *)(tagged_page & ~1); // Mask out the LSB to get the original pointer
331
248
- for(page = ar->aral_lock.pages; page ; page = page->next) {
249
- if(page->aral_lock.free_elements)
250
- break;
332
+ internal_fatal(!page,
333
+ "ARAL: '%s' possible corruption or double free of pointer %p",
334
+ ar->config.name, ptr);
335
252
- internal_fatal(page->size - page->aral_lock.used_elements * ar->config.element_size >= ar->config.element_size,
253
- "ARAL: '%s' a page is marked full, but it is not!", ar->config.name);
336
+#ifdef NETDATA_ARAL_INTERNAL_CHECKS
337
+ {
338
+ // find the page ptr belongs
339
+ ARAL_PAGE *page2 = find_page_with_allocation_internal_check(ar, ptr, *marked);
340
+ if(!page2) {
341
+ page2 = find_page_with_allocation_internal_check(ar, ptr, !(*marked));
342
+ internal_fatal(page2 && (*marked) && !page2->marked, "ARAL: '%s' page pointer is in different mark index",
343
+ ar->config.name);
344
+ }
345
255
- internal_fatal(page->size < page->aral_lock.used_elements * ar->config.element_size,
256
- "ARAL: '%s' a page has been overflown!", ar->config.name);
346
+ internal_fatal(page != page2,
347
+ "ARAL: '%s' page pointers do not match!",
348
+ ar->config.name);
349
+
350
+ internal_fatal(!page2,
351
+ "ARAL: '%s' free of pointer %p is not in ARAL address space.",
352
+ ar->config.name, ptr);
353
}
354
+#endif
355
356
return page;
357
}
358
+
359
+static void aral_set_page_pointer_after_element___do_NOT_have_aral_lock(ARAL *ar, void *page, void *ptr, bool marked) {
360
+ uint8_t *data = ptr;
361
+ uintptr_t *page_ptr = (uintptr_t *)&data[ar->config.page_ptr_offset];
362
+ uintptr_t tagged_page = (uintptr_t)page; // Cast the pointer to an integer
363
+ if (marked) tagged_page |= 1; // Set the LSB to 1 if 'marked' is true
364
+ __atomic_store_n(page_ptr, tagged_page, __ATOMIC_RELEASE); // Atomically store the tagged pointer
365
+}
366
+
367
+// ----------------------------------------------------------------------------
368
+// check a free slot
369
+
370
+#ifdef NETDATA_INTERNAL_CHECKS
371
+static inline void aral_free_validate_internal_check(ARAL *ar, ARAL_FREE *fr) {
372
+ if(unlikely(fr->size < ar->config.element_size))
373
+ fatal("ARAL: '%s' free item of size %zu, less than the expected element size %zu",
374
+ ar->config.name, fr->size, ar->config.element_size);
375
+
376
+ if(unlikely(fr->size % ar->config.element_size))
377
+ fatal("ARAL: '%s' free item of size %zu is not multiple to element size %zu",
378
+ ar->config.name, fr->size, ar->config.element_size);
379
+}
380
+#else
381
+#define aral_free_validate_internal_check(ar, fr) debug_dummy()
382
#endif
383
263
-size_t aral_next_allocation_size___adders_lock_needed(ARAL *ar) {
264
- size_t size = ar->adders.allocation_size;
384
+// ----------------------------------------------------------------------------
385
+
386
+size_t aral_next_allocation_size___adders_lock_needed(ARAL *ar, bool marked) {
387
+ size_t idx = mark_to_idx(marked);
388
+ size_t size = ar->ops[idx].adders.allocation_size;
389
390
if(size > ar->config.max_allocation_size)
391
size = ar->config.max_allocation_size;
392
else
269
- ar->adders.allocation_size = aral_align_alloc_size(ar, (uint64_t)ar->adders.allocation_size * 2);
393
+ ar->ops[idx].adders.allocation_size = aral_align_alloc_size(ar, (uint64_t)ar->ops[idx].adders.allocation_size * 2);
394
395
return size;
396
}
@@ -277,9 +401,6 @@ static ARAL_PAGE *aral_create_page___no_lock_needed(ARAL *ar, size_t size TRACE_
401
page->size = size;
402
page->max_elements = page->size / ar->config.element_size;
403
page->aral_lock.free_elements = page->max_elements;
280
- page->free_elements_to_move_first = page->max_elements / 4;
281
- if(unlikely(page->free_elements_to_move_first < 1))
282
- page->free_elements_to_move_first = 1;
404
405
__atomic_add_fetch(&ar->stats->structures.allocations, 1, __ATOMIC_RELAXED);
406
__atomic_add_fetch(&ar->stats->structures.allocated_bytes, sizeof(ARAL_PAGE), __ATOMIC_RELAXED);
@@ -291,7 +412,7 @@ static ARAL_PAGE *aral_create_page___no_lock_needed(ARAL *ar, size_t size TRACE_
412
page->filename = strdupz(filename);
413
page->data = netdata_mmap(page->filename, page->size, MAP_SHARED, 0, false, NULL);
414
if (unlikely(!page->data))
294
- fatal("ARAL: '%s' cannot allocate aral buffer of size %zu on filename '%s'",
415
+ fatal("ARAL: '%s' cannot allocate aral buffer of size %u on filename '%s'",
416
ar->config.name, page->size, page->filename);
417
__atomic_add_fetch(&ar->stats->mmap.allocations, 1, __ATOMIC_RELAXED);
418
__atomic_add_fetch(&ar->stats->mmap.allocated_bytes, page->size, __ATOMIC_RELAXED);
@@ -302,12 +423,14 @@ static ARAL_PAGE *aral_create_page___no_lock_needed(ARAL *ar, size_t size TRACE_
423
#else
424
page->data = mallocz(page->size);
425
#endif
426
+
427
__atomic_add_fetch(&ar->stats->malloc.allocations, 1, __ATOMIC_RELAXED);
428
__atomic_add_fetch(&ar->stats->malloc.allocated_bytes, page->size, __ATOMIC_RELAXED);
429
}
430
431
// link the free space to its page
432
ARAL_FREE *fr = (ARAL_FREE *)page->data;
433
+
434
fr->size = page->size;
435
fr->next = NULL;
436
page->free.list = fr;
@@ -347,90 +470,86 @@ void aral_del_page___no_lock_needed(ARAL *ar, ARAL_PAGE *page TRACE_ALLOCATIONS_
470
__atomic_sub_fetch(&ar->stats->structures.allocated_bytes, sizeof(ARAL_PAGE), __ATOMIC_RELAXED);
471
}
472
350
-static inline void aral_insert_not_linked_page_with_free_items_to_proper_position___aral_lock_needed(ARAL *ar, ARAL_PAGE *page) {
351
- ARAL_PAGE *first = ar->aral_lock.pages;
352
-
353
- if (page->aral_lock.free_elements <= page->free_elements_to_move_first ||
354
- !first ||
355
- !first->aral_lock.free_elements ||
356
- page->aral_lock.free_elements <= first->aral_lock.free_elements + ARAL_FREE_PAGES_DELTA_TO_REARRANGE_LIST) {
357
- // first position
358
- DOUBLE_LINKED_LIST_PREPEND_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
359
- }
360
- else {
361
- ARAL_PAGE *second = first->aral_lock.next;
362
-
363
- if (!second ||
364
- !second->aral_lock.free_elements ||
365
- page->aral_lock.free_elements <= second->aral_lock.free_elements)
366
- // second position
367
- DOUBLE_LINKED_LIST_INSERT_ITEM_AFTER_UNSAFE(ar->aral_lock.pages, first, page, aral_lock.prev, aral_lock.next);
368
- else
369
- // third position
370
- DOUBLE_LINKED_LIST_INSERT_ITEM_AFTER_UNSAFE(ar->aral_lock.pages, second, page, aral_lock.prev, aral_lock.next);
371
- }
372
-}
373
-
374
-static inline ARAL_PAGE *aral_acquire_a_free_slot(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS) {
375
- __atomic_add_fetch(&ar->atomic.allocators, 1, __ATOMIC_RELAXED);
473
+static inline ARAL_PAGE *aral_get_first_page_with_a_free_slot(ARAL *ar, bool marked TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS) {
474
+ size_t idx = mark_to_idx(marked);
475
+ __atomic_add_fetch(&ar->ops[idx].atomic.allocators, 1, __ATOMIC_RELAXED);
476
aral_lock(ar);
477
378
- ARAL_PAGE *page = ar->aral_lock.pages;
478
+ ARAL_PAGE **head_ptr_free = aral_pages_head_free(ar, marked);
479
+ ARAL_PAGE *page = *head_ptr_free;
480
+
481
+#ifdef NETDATA_ARAL_INTERNAL_CHECKS
482
+ // bool added = false;
483
+ struct free_space f1, f2;
484
+#endif
485
486
while(!page || !page->aral_lock.free_elements) {
487
+ internal_fatal(page && page->aral_lock.next && page->aral_lock.next->aral_lock.free_elements, "hey!");
488
+
489
#ifdef NETDATA_ARAL_INTERNAL_CHECKS
382
- internal_fatal(find_page_with_free_slots_internal_check___with_aral_lock(ar), "ARAL: '%s' found page with free slot!", ar->config.name);
490
+ f1 = check_free_space___aral_lock_needed(ar, NULL, marked);
491
#endif
384
- aral_unlock(ar);
492
386
- if(aral_adders_trylock(ar)) {
387
- if(ar->adders.allocating_elements < __atomic_load_n(&ar->atomic.allocators, __ATOMIC_RELAXED)) {
493
+ size_t page_allocation_size = 0;
494
+ bool can_add = false;
495
+ if(aral_adders_trylock(ar, marked)) {
496
+ // we can add a page - let's see it is really needed
497
+ size_t threads_currently_allocating = __atomic_load_n(&ar->ops[idx].atomic.allocators, __ATOMIC_RELAXED);
498
+ size_t threads_currently_deallocating = __atomic_load_n(&ar->ops[idx].atomic.deallocators, __ATOMIC_RELAXED);
499
+
500
+ // we will allocate a page, only if the number of elements required is more than the
501
+ // sum of all new allocations under their way plus the pages currently being deallocated
502
+ if(ar->ops[idx].adders.allocating_elements + threads_currently_deallocating < threads_currently_allocating) {
503
+ can_add = true;
504
+ page_allocation_size = aral_next_allocation_size___adders_lock_needed(ar, marked);
505
+ ar->ops[idx].adders.allocating_elements += page_allocation_size / ar->config.element_size;
506
+ }
507
+ aral_adders_unlock(ar, marked);
508
+ }
509
+ aral_unlock(ar);
510
389
- size_t size = aral_next_allocation_size___adders_lock_needed(ar);
390
- ar->adders.allocating_elements += size / ar->config.element_size;
391
- aral_adders_unlock(ar);
511
+ if(can_add) {
512
+ page = aral_create_page___no_lock_needed(ar, page_allocation_size TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
513
+ page->marked = marked;
514
393
- page = aral_create_page___no_lock_needed(ar, size TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
515
+ aral_lock(ar);
516
395
- aral_lock(ar);
396
- aral_insert_not_linked_page_with_free_items_to_proper_position___aral_lock_needed(ar, page);
517
+ DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(*head_ptr_free, page, aral_lock.prev, aral_lock.next);
518
398
- aral_adders_lock(ar);
399
- ar->adders.allocating_elements -= size / ar->config.element_size;
400
- aral_adders_unlock(ar);
519
+//#ifdef NETDATA_ARAL_INTERNAL_CHECKS
520
+// added = true;
521
+//#endif
522
402
- // we have a page that is all empty
403
- // and only aral_lock() is held, so
404
- // break the loop
405
- break;
406
- }
523
+ aral_adders_lock(ar, marked);
524
+ ar->ops[idx].adders.allocating_elements -= page_allocation_size / ar->config.element_size;
525
+ aral_adders_unlock(ar, marked);
526
408
- aral_adders_unlock(ar);
527
+ // we have a page that is all empty
528
+ // and only aral_lock() is held, so
529
+ // break the loop
530
+ break;
531
}
532
+ else {
533
+ // let the adders/deallocators do it
534
+ // tinysleep();
535
+ sched_yield();
536
411
- aral_lock(ar);
412
- page = ar->aral_lock.pages;
537
+ aral_lock(ar);
538
+ page = *head_ptr_free;
539
+ }
540
}
541
415
- __atomic_sub_fetch(&ar->atomic.allocators, 1, __ATOMIC_RELAXED);
416
-
542
// we have a page
543
// and aral locked
544
420
- {
421
- ARAL_PAGE *first = ar->aral_lock.pages;
422
- ARAL_PAGE *second = first->aral_lock.next;
545
+ internal_fatal(marked && !page->marked, "ARAL: requested a marked page, but the page found is not marked");
546
424
- if (!second ||
425
- !second->aral_lock.free_elements ||
426
- first->aral_lock.free_elements <= second->aral_lock.free_elements + ARAL_FREE_PAGES_DELTA_TO_REARRANGE_LIST)
427
- page = first;
428
- else {
429
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, second, aral_lock.prev, aral_lock.next);
430
- DOUBLE_LINKED_LIST_PREPEND_ITEM_UNSAFE(ar->aral_lock.pages, second, aral_lock.prev, aral_lock.next);
431
- page = second;
432
- }
433
- }
547
+//#ifdef NETDATA_ARAL_INTERNAL_CHECKS
548
+// if(added) {
549
+// f2 = check_free_space___aral_lock_needed(ar, page, marked);
550
+// internal_fatal(f2.failed, "hey!");
551
+// }
552
+//#endif
553
554
internal_fatal(!page || !page->aral_lock.free_elements,
555
"ARAL: '%s' selected page does not have a free slot in it",
@@ -451,31 +570,41 @@ static inline ARAL_PAGE *aral_acquire_a_free_slot(ARAL *ar TRACE_ALLOCATIONS_FUN
570
571
// acquire a slot for the caller
572
page->aral_lock.used_elements++;
454
- if(--page->aral_lock.free_elements == 0) {
455
- // we are done with this page
456
- // move the full page last
457
- // so that pages with free items remain first in the list
458
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
459
- DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
573
+ page->aral_lock.free_elements--;
574
+
575
+ if(marked)
576
+ page->aral_lock.marked_elements++;
577
+
578
+ internal_fatal(page->aral_lock.marked_elements > page->aral_lock.used_elements,
579
+ "page has more marked elements than the used ones");
580
+
581
+ if(page->aral_lock.free_elements == 0) {
582
+ ARAL_PAGE **head_ptr_full = aral_pages_head_full(ar, marked);
583
+ internal_fatal(!is_page_in_list(*head_ptr_free, page), "Page is not in this list");
584
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(*head_ptr_free, page, aral_lock.prev, aral_lock.next);
585
+ DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(*head_ptr_full, page, aral_lock.prev, aral_lock.next);
586
}
587
588
+ __atomic_sub_fetch(&ar->ops[idx].atomic.allocators, 1, __ATOMIC_RELAXED);
589
aral_unlock(ar);
590
591
return page;
592
}
593
467
-void *aral_callocz_internal(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS) {
468
- void *r = aral_mallocz_internal(ar TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
594
+void *aral_callocz_internal(ARAL *ar, bool marked TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS) {
595
+ void *r = aral_mallocz_internal(ar, marked TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
596
memset(r, 0, ar->config.requested_element_size);
597
return r;
598
}
599
473
-void *aral_mallocz_internal(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS) {
474
-#ifdef FSANITIZE_ADDRESS
600
+void *aral_mallocz_internal(ARAL *ar, bool marked TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS) {
601
+#if defined(FSANITIZE_ADDRESS)
602
return mallocz(ar->config.requested_element_size);
603
#endif
604
478
- ARAL_PAGE *page = aral_acquire_a_free_slot(ar TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
605
+ // reserve a slot on a free page
606
+ ARAL_PAGE *page = aral_get_first_page_with_a_free_slot(ar, marked TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
607
+ // the page returned has reserved a slot for us
608
609
aral_page_free_lock(ar, page);
610
@@ -499,6 +628,7 @@ void *aral_mallocz_internal(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAM
628
629
uint8_t *data = (uint8_t *)found_fr;
630
ARAL_FREE *fr = (ARAL_FREE *)&data[ar->config.element_size];
631
+
632
fr->size = found_fr->size - ar->config.element_size;
633
634
// link the free slot first in the page
@@ -511,9 +641,7 @@ void *aral_mallocz_internal(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAM
641
aral_page_free_unlock(ar, page);
642
643
// put the page pointer after the element
514
- uint8_t *data = (uint8_t *)found_fr;
515
- ARAL_PAGE **page_ptr = (ARAL_PAGE **)&data[ar->config.page_ptr_offset];
516
- *page_ptr = page;
644
+ aral_set_page_pointer_after_element___do_NOT_have_aral_lock(ar, page, found_fr, marked);
645
646
if(unlikely(ar->config.mmap.enabled))
647
__atomic_add_fetch(&ar->stats->mmap.used_bytes, ar->config.element_size, __ATOMIC_RELAXED);
@@ -523,121 +651,71 @@ void *aral_mallocz_internal(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAM
651
return (void *)found_fr;
652
}
653
526
-static inline ARAL_PAGE *aral_ptr_to_page___must_NOT_have_aral_lock(ARAL *ar, void *ptr) {
527
- // given a data pointer we returned before,
528
- // find the ARAL_PAGE it belongs to
529
-
530
- uint8_t *data = (uint8_t *)ptr;
531
- ARAL_PAGE **page_ptr = (ARAL_PAGE **)&data[ar->config.page_ptr_offset];
532
- ARAL_PAGE *page = *page_ptr;
654
+// returns true if it moved the page to the unmarked list
655
+static ARAL_PAGE **aral_remove_marked_allocation___aral_lock_needed(ARAL *ar, ARAL_PAGE **head_ptr, ARAL_PAGE *page) {
656
+ internal_fatal(!page->aral_lock.marked_elements, "marked elements refcount found zero");
657
+ internal_fatal(!is_page_in_list(*head_ptr, page), "Page is not in this list");
658
534
-#ifdef NETDATA_INTERNAL_CHECKS
535
- // make it NULL so that we will fail on double free
536
- // do not enable this on production, because the MMAP file
537
- // will need to be saved again!
538
- *page_ptr = NULL;
539
-#endif
659
+ page->aral_lock.marked_elements--;
660
+ if (!page->aral_lock.marked_elements && page->aral_lock.used_elements) {
661
+ internal_fatal(!page->marked, "The page should be marked at this point");
662
541
-#ifdef NETDATA_ARAL_INTERNAL_CHECKS
542
- {
543
- // find the page ptr belongs
544
- ARAL_PAGE *page2 = find_page_with_allocation_internal_check(ar, ptr);
663
+ ARAL_PAGE **head_ptr_to = (page->aral_lock.free_elements) ? aral_pages_head_free(ar, false) : aral_pages_head_full(ar, false);
664
+ internal_fatal(!is_page_in_list(*head_ptr, page), "Page is not in this list");
665
546
- internal_fatal(page != page2,
547
- "ARAL: '%s' page pointers do not match!",
548
- ar->name);
549
-
550
- internal_fatal(!page2,
551
- "ARAL: '%s' free of pointer %p is not in ARAL address space.",
552
- ar->name, ptr);
666
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(*head_ptr, page, aral_lock.prev, aral_lock.next);
667
+ DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(*head_ptr_to, page, aral_lock.prev, aral_lock.next);
668
+ page->marked = false;
669
+ return head_ptr_to;
670
}
554
-#endif
671
556
- internal_fatal(!page,
557
- "ARAL: '%s' possible corruption or double free of pointer %p",
558
- ar->config.name, ptr);
672
+ internal_fatal(page->aral_lock.marked_elements > page->aral_lock.used_elements,
673
+ "page has more marked elements than the used ones");
674
560
- return page;
675
+ return head_ptr;
676
}
677
563
-static void aral_defrag_sorted_page_position___aral_lock_needed(ARAL *ar, ARAL_PAGE *page) {
564
- ARAL_PAGE *tmp;
565
-
566
- int action = 0; (void)action;
567
- size_t move_later = 0, move_earlier = 0;
568
-
569
- for(tmp = page->aral_lock.next ;
570
- tmp && tmp->aral_lock.free_elements && tmp->aral_lock.free_elements < page->aral_lock.free_elements ;
571
- tmp = tmp->aral_lock.next)
572
- move_later++;
573
-
574
- if(!tmp && page->aral_lock.next) {
575
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
576
- DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
577
- action = 1;
578
- }
579
- else if(tmp != page->aral_lock.next) {
580
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
581
- DOUBLE_LINKED_LIST_INSERT_ITEM_BEFORE_UNSAFE(ar->aral_lock.pages, tmp, page, aral_lock.prev, aral_lock.next);
582
- action = 2;
583
- }
584
- else {
585
- for(tmp = (page == ar->aral_lock.pages) ? NULL : page->aral_lock.prev ;
586
- tmp && (!tmp->aral_lock.free_elements || tmp->aral_lock.free_elements > page->aral_lock.free_elements);
587
- tmp = (tmp == ar->aral_lock.pages) ? NULL : tmp->aral_lock.prev)
588
- move_earlier++;
589
-
590
- if(!tmp) {
591
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
592
- DOUBLE_LINKED_LIST_PREPEND_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
593
- action = 3;
594
- }
595
- else if(tmp != page->aral_lock.prev){
596
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
597
- DOUBLE_LINKED_LIST_INSERT_ITEM_AFTER_UNSAFE(ar->aral_lock.pages, tmp, page, aral_lock.prev, aral_lock.next);
598
- action = 4;
599
- }
600
- }
601
-
602
- ar->aral_lock.defragment_operations++;
603
- ar->aral_lock.defragment_linked_list_traversals += move_earlier + move_later;
678
+void aral_unmark_allocation(ARAL *ar, void *ptr) {
679
+ if(unlikely(!ptr)) return;
680
605
- internal_fatal(page->aral_lock.next && page->aral_lock.next->aral_lock.free_elements && page->aral_lock.next->aral_lock.free_elements < page->aral_lock.free_elements,
606
- "ARAL: '%s' item should be later in the list", ar->config.name);
681
+ // get the page pointer
682
+ bool marked;
683
+ ARAL_PAGE *page = aral_get_page_pointer_after_element___do_NOT_have_aral_lock(ar, ptr, &marked);
684
608
- internal_fatal(page != ar->aral_lock.pages && (!page->aral_lock.prev->aral_lock.free_elements || page->aral_lock.prev->aral_lock.free_elements > page->aral_lock.free_elements),
609
- "ARAL: '%s' item should be earlier in the list", ar->config.name);
610
-}
685
+ internal_fatal(!page->marked, "This allocation does not belong to a marked page");
686
+ internal_fatal(!marked, "This allocation does is not marked");
687
612
-static inline void aral_move_page_with_free_list___aral_lock_needed(ARAL *ar, ARAL_PAGE *page) {
613
- if(unlikely(page == ar->aral_lock.pages))
614
- // we are the first already
615
- return;
688
+ if(marked)
689
+ aral_set_page_pointer_after_element___do_NOT_have_aral_lock(ar, page, ptr, false);
690
617
- if(likely(!(ar->config.options & ARAL_DEFRAGMENT))) {
618
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
619
- aral_insert_not_linked_page_with_free_items_to_proper_position___aral_lock_needed(ar, page);
691
+ if(marked && page->marked) {
692
+ aral_lock(ar);
693
+ ARAL_PAGE **head_ptr = page->aral_lock.free_elements ? aral_pages_head_free(ar, page->marked) : aral_pages_head_full(ar, page->marked);
694
+ aral_remove_marked_allocation___aral_lock_needed(ar, head_ptr, page);
695
+ aral_unlock(ar);
696
}
621
- else
622
- aral_defrag_sorted_page_position___aral_lock_needed(ar, page);
697
}
698
699
void aral_freez_internal(ARAL *ar, void *ptr TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS) {
626
-#ifdef FSANITIZE_ADDRESS
700
+#if defined(FSANITIZE_ADDRESS)
701
freez(ptr);
702
return;
703
#endif
704
705
if(unlikely(!ptr)) return;
706
633
- // get the page pointer
634
- ARAL_PAGE *page = aral_ptr_to_page___must_NOT_have_aral_lock(ar, ptr);
635
-
707
if(unlikely(ar->config.mmap.enabled))
708
__atomic_sub_fetch(&ar->stats->mmap.used_bytes, ar->config.element_size, __ATOMIC_RELAXED);
709
else
710
__atomic_sub_fetch(&ar->stats->malloc.used_bytes, ar->config.element_size, __ATOMIC_RELAXED);
711
712
+ // get the page pointer
713
+ bool marked;
714
+ ARAL_PAGE *page = aral_get_page_pointer_after_element___do_NOT_have_aral_lock(ar, ptr, &marked);
715
+
716
+ size_t idx = mark_to_idx(marked);
717
+ __atomic_add_fetch(&ar->ops[idx].atomic.deallocators, 1, __ATOMIC_RELAXED);
718
+
719
// make this element available
720
ARAL_FREE *fr = (ARAL_FREE *)ptr;
721
fr->size = ar->config.element_size;
@@ -662,37 +740,83 @@ void aral_freez_internal(ARAL *ar, void *ptr TRACE_ALLOCATIONS_FUNCTION_DEFINITI
740
(size_t)page->max_elements,
741
(size_t)page->aral_lock.used_elements, (size_t)page->aral_lock.free_elements,
742
(size_t)page->aral_lock.used_elements + (size_t)page->aral_lock.free_elements
665
- );
743
+ );
744
+
745
+ ARAL_PAGE **head_ptr = page->aral_lock.free_elements ? aral_pages_head_free(ar, page->marked) : aral_pages_head_full(ar, page->marked);
746
+ internal_fatal(!is_page_in_list(*head_ptr, page), "Page is not in this list");
747
748
page->aral_lock.used_elements--;
749
page->aral_lock.free_elements++;
750
751
ar->aral_lock.user_free_operations++;
752
753
+ internal_fatal(marked && !page->marked, "ARAL: found a marked element on a non-marked page");
754
+
755
+ if(marked && page->marked) {
756
+ head_ptr = aral_remove_marked_allocation___aral_lock_needed(ar, head_ptr, page);
757
+ internal_fatal(!is_page_in_list(*head_ptr, page), "Page is not in this list");
758
+ }
759
+
760
+ internal_fatal(page->aral_lock.marked_elements > page->aral_lock.used_elements,
761
+ "page has more marked elements than the used ones");
762
+
763
// if the page is empty, release it
764
if(unlikely(!page->aral_lock.used_elements)) {
674
- bool is_this_page_the_last_one = ar->aral_lock.pages == page && !page->aral_lock.next;
765
+ internal_fatal(page->aral_lock.marked_elements, "page has marked elements but not used ones");
766
676
- if(!is_this_page_the_last_one)
677
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
767
+ bool is_this_page_the_last_one = *head_ptr == page && !page->aral_lock.next;
768
769
+ if(!is_this_page_the_last_one) {
770
+ internal_fatal(!is_page_in_list(*head_ptr, page), "Page is not in this list");
771
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(*head_ptr, page, aral_lock.prev, aral_lock.next);
772
+ }
773
+
774
+ __atomic_sub_fetch(&ar->ops[idx].atomic.deallocators, 1, __ATOMIC_RELAXED);
775
aral_unlock(ar);
776
777
if(!is_this_page_the_last_one)
778
aral_del_page___no_lock_needed(ar, page TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
779
+
780
+ return;
781
}
684
- else {
685
- aral_move_page_with_free_list___aral_lock_needed(ar, page);
686
- aral_unlock(ar);
782
+ else if(page->aral_lock.free_elements) {
783
+ ARAL_PAGE **head_ptr_to = aral_pages_head_free(ar, page->marked);
784
+ if(head_ptr != head_ptr_to) {
785
+ internal_fatal(!is_page_in_list(*head_ptr, page), "Page is not in this list");
786
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(*head_ptr, page, aral_lock.prev, aral_lock.next);
787
+ DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(*head_ptr_to, page, aral_lock.prev, aral_lock.next);
788
+ }
789
}
790
+
791
+ __atomic_sub_fetch(&ar->ops[idx].atomic.deallocators, 1, __ATOMIC_RELAXED);
792
+ aral_unlock(ar);
793
}
794
795
void aral_destroy_internal(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS) {
796
aral_lock(ar);
797
798
+ ARAL_PAGE **head_ptr = aral_pages_head_free(ar, false);
799
ARAL_PAGE *page;
694
- while((page = ar->aral_lock.pages)) {
695
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(ar->aral_lock.pages, page, aral_lock.prev, aral_lock.next);
800
+ while((page = *head_ptr)) {
801
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(*head_ptr, page, aral_lock.prev, aral_lock.next);
802
+ aral_del_page___no_lock_needed(ar, page TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
803
+ }
804
+
805
+ head_ptr = aral_pages_head_free(ar, true);
806
+ while((page = *head_ptr)) {
807
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(*head_ptr, page, aral_lock.prev, aral_lock.next);
808
+ aral_del_page___no_lock_needed(ar, page TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
809
+ }
810
+
811
+ head_ptr = aral_pages_head_full(ar, false);
812
+ while((page = *head_ptr)) {
813
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(*head_ptr, page, aral_lock.prev, aral_lock.next);
814
+ aral_del_page___no_lock_needed(ar, page TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
815
+ }
816
+
817
+ head_ptr = aral_pages_head_full(ar, true);
818
+ while((page = *head_ptr)) {
819
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(*head_ptr, page, aral_lock.prev, aral_lock.next);
820
aral_del_page___no_lock_needed(ar, page TRACE_ALLOCATIONS_FUNCTION_CALL_PARAMS);
821
}
822
@@ -704,14 +828,50 @@ void aral_destroy_internal(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_PARAMS
828
freez(ar);
829
}
830
707
-size_t aral_element_size(ARAL *ar) {
831
+size_t aral_requested_element_size(ARAL *ar) {
832
return ar->config.requested_element_size;
833
}
834
835
+size_t aral_actual_element_size(ARAL *ar) {
836
+ return ar->config.element_size;
837
+}
838
+
839
+static size_t aral_allocation_slot_size(size_t requested_element_size, bool usable) {
840
+ // we need to add a page pointer after the element
841
+ // so, first align the element size to the pointer size
842
+ size_t element_size = memory_alignment(requested_element_size, sizeof(uintptr_t));
843
+
844
+ // then add the size of a pointer to it
845
+ element_size += sizeof(uintptr_t);
846
+
847
+ // make sure it is at least what we need for an ARAL_FREE slot
848
+ if (element_size < sizeof(ARAL_FREE))
849
+ element_size = sizeof(ARAL_FREE);
850
+
851
+ // and finally align it to the natural alignment
852
+ element_size = memory_alignment(element_size, SYSTEM_REQUIRED_ALIGNMENT);
853
+
854
+ if(usable)
855
+ return element_size - sizeof(uintptr_t);
856
+
857
+ return element_size;
858
+}
859
+
860
+size_t aral_optimal_page_size(void) {
861
+ return ARAL_MAX_PAGE_SIZE_MALLOC;
862
+}
863
+
864
+static void optimal_max_page_size(ARAL *ar) {
865
+ if(ar->config.requested_max_page_size)
866
+ return;
867
+
868
+ ar->config.requested_max_page_size = aral_optimal_page_size();
869
+}
870
+
871
ARAL *aral_create(const char *name, size_t element_size, size_t initial_page_elements, size_t max_page_size,
872
struct aral_statistics *stats, const char *filename, const char **cache_dir, bool mmap, bool lockless) {
873
ARAL *ar = callocz(1, sizeof(ARAL));
714
- ar->config.options = (lockless) ? ARAL_LOCKLESS : 0;
874
+ ar->config.options = ((lockless) ? ARAL_LOCKLESS : 0);
875
ar->config.requested_element_size = element_size;
876
ar->config.initial_page_elements = initial_page_elements;
877
ar->config.requested_max_page_size = max_page_size;
@@ -720,7 +880,8 @@ ARAL *aral_create(const char *name, size_t element_size, size_t initial_page_ele
880
ar->config.mmap.enabled = mmap;
881
strncpyz(ar->config.name, name, ARAL_MAX_NAME);
882
spinlock_init(&ar->aral_lock.spinlock);
723
- spinlock_init(&ar->adders.spinlock);
883
+ spinlock_init(&ar->ops[0].adders.spinlock);
884
+ spinlock_init(&ar->ops[1].adders.spinlock);
885
886
if(stats) {
887
ar->stats = stats;
@@ -733,23 +894,12 @@ ARAL *aral_create(const char *name, size_t element_size, size_t initial_page_ele
894
895
long int page_size = sysconf(_SC_PAGE_SIZE);
896
if (unlikely(page_size == -1))
736
- ar->config.natural_page_size = 4096;
897
+ ar->config.system_page_size = 4096;
898
else
738
- ar->config.natural_page_size = page_size;
899
+ ar->config.system_page_size = page_size;
900
740
- // we need to add a page pointer after the element
741
- // so, first align the element size to the pointer size
742
- ar->config.element_size = natural_alignment(ar->config.requested_element_size, sizeof(uintptr_t));
743
-
744
- // then add the size of a pointer to it
745
- ar->config.element_size += sizeof(uintptr_t);
746
-
747
- // make sure it is at least what we need for an ARAL_FREE slot
748
- if (ar->config.element_size < sizeof(ARAL_FREE))
749
- ar->config.element_size = sizeof(ARAL_FREE);
750
-
751
- // and finally align it to the natural alignment
752
- ar->config.element_size = natural_alignment(ar->config.element_size, ARAL_NATURAL_ALIGNMENT);
901
+ ar->config.element_size = aral_allocation_slot_size(ar->config.requested_element_size, false);
902
+ optimal_max_page_size(ar);
903
904
ar->config.max_page_elements = ar->config.requested_max_page_size / ar->config.element_size;
905
@@ -760,7 +910,8 @@ ARAL *aral_create(const char *name, size_t element_size, size_t initial_page_ele
910
fatal("ARAL: '%s' failed to calculate properly page_ptr_offset: "
911
"element size %zu, sizeof(uintptr_t) %zu, natural alignment %zu, "
912
"final element size %zu, page_ptr_offset %zu",
763
- ar->config.name, ar->config.requested_element_size, sizeof(uintptr_t), ARAL_NATURAL_ALIGNMENT,
913
+ ar->config.name, ar->config.requested_element_size, sizeof(uintptr_t),
914
+ SYSTEM_REQUIRED_ALIGNMENT,
915
ar->config.element_size, ar->config.page_ptr_offset);
916
917
//netdata_log_info("ARAL: element size %zu, sizeof(uintptr_t) %zu, natural alignment %zu, final element size %zu, page_ptr_offset %zu",
@@ -783,8 +934,11 @@ ARAL *aral_create(const char *name, size_t element_size, size_t initial_page_ele
934
max_alloc_size = ar->config.max_page_elements * ar->config.element_size;
935
936
ar->config.max_allocation_size = aral_align_alloc_size(ar, max_alloc_size);
786
- ar->adders.allocation_size = aral_align_alloc_size(ar, (uint64_t)ar->config.element_size * ar->config.initial_page_elements);
787
- ar->aral_lock.pages = NULL;
937
+ ar->ops[0].adders.allocation_size =
938
+ ar->ops[1].adders.allocation_size =
939
+ aral_align_alloc_size(ar, (uint64_t)ar->config.element_size * ar->config.initial_page_elements);
940
+ ar->aral_lock.pages_free = NULL;
941
+ ar->aral_lock.pages_marked_free = NULL;
942
ar->aral_lock.file_number = 0;
943
944
if(ar->config.mmap.enabled) {
@@ -808,7 +962,7 @@ ARAL *aral_create(const char *name, size_t element_size, size_t initial_page_ele
962
"max page size %zu bytes (requested %zu) "
963
, ar->config.name
964
, ar->config.element_size, ar->config.requested_element_size
811
- , ar->adders.allocation_size / ar->config.element_size, ar->config.initial_page_elements
965
+ , ar->ops[0].adders.allocation_size / ar->config.element_size, ar->config.initial_page_elements
966
, ar->config.max_allocation_size / ar->config.element_size
967
, ar->config.max_allocation_size, ar->config.requested_max_page_size
968
);
@@ -846,6 +1000,10 @@ size_t aral_by_size_overhead(void) {
1000
return aral_overhead_from_stats(&aral_by_size_globals.shared_statistics);
1001
}
1002
1003
+size_t aral_by_size_used_bytes(void) {
1004
+ return aral_used_bytes_from_stats(&aral_by_size_globals.shared_statistics);
1005
+}
1006
+
1007
ARAL *aral_by_size_acquire(size_t size) {
1008
spinlock_lock(&aral_by_size_globals.spinlock);
1009
@@ -855,8 +1013,9 @@ ARAL *aral_by_size_acquire(size_t size) {
1013
ar = aral_by_size_globals.array[size].ar;
1014
aral_by_size_globals.array[size].refcount++;
1015
858
- internal_fatal(aral_element_size(ar) != size, "DICTIONARY: aral has size %zu but we want %zu",
859
- aral_element_size(ar), size);
1016
+ internal_fatal(
1017
+ aral_requested_element_size(ar) != size, "DICTIONARY: aral has size %zu but we want %zu",
1018
+ aral_requested_element_size(ar), size);
1019
}
1020
1021
if(!ar) {
@@ -865,7 +1024,7 @@ ARAL *aral_by_size_acquire(size_t size) {
1024
ar = aral_create(buf,
1025
size,
1026
0,
868
- 65536 * ((size / 150) + 1),
1027
+ 0,
1028
&aral_by_size_globals.shared_statistics,
1029
NULL, NULL, false, false);
1030
@@ -881,7 +1040,7 @@ ARAL *aral_by_size_acquire(size_t size) {
1040
}
1041
1042
void aral_by_size_release(ARAL *ar) {
884
- size_t size = aral_element_size(ar);
1043
+ size_t size = aral_requested_element_size(ar);
1044
1045
if(size <= ARAL_BY_SIZE_MAX_SIZE) {
1046
spinlock_lock(&aral_by_size_globals.spinlock);
@@ -916,16 +1075,49 @@ struct aral_unittest_config {
1075
int errors;
1076
};
1077
1078
+struct aral_unittest_entry {
1079
+ char TXT[27];
1080
+ char txt[27];
1081
+ char nnn[10];
1082
+};
1083
+
1084
+#define UNITTEST_ITEM (struct aral_unittest_entry){ \
1085
+ .TXT = "ABCDEFGHIJKLMNOPQRSTUVWXYZ", \
1086
+ .txt = "abcdefghijklmnopqrstuvwxyz", \
1087
+ .nnn = "123456789", \
1088
+}
1089
+
1090
+static inline struct aral_unittest_entry *unittest_aral_malloc(ARAL *ar, bool marked) {
1091
+ struct aral_unittest_entry *t;
1092
+ if(marked)
1093
+ t = aral_mallocz_marked(ar);
1094
+ else
1095
+ t = aral_mallocz(ar);
1096
+
1097
+ *t = UNITTEST_ITEM;
1098
+ return t;
1099
+}
1100
+
1101
static void *aral_test_thread(void *ptr) {
1102
struct aral_unittest_config *auc = ptr;
1103
ARAL *ar = auc->ar;
1104
size_t elements = auc->elements;
1105
924
- void **pointers = callocz(elements, sizeof(void *));
1106
+ bool marked = os_random(2);
1107
+ struct aral_unittest_entry **pointers = callocz(elements, sizeof(struct aral_unittest_entry *));
1108
1109
+ size_t iterations = 0;
1110
do {
1111
+ iterations++;
1112
+
1113
for (size_t i = 0; i < elements; i++) {
928
- pointers[i] = aral_mallocz(ar);
1114
+ pointers[i] = unittest_aral_malloc(ar, marked);
1115
+ }
1116
+
1117
+ if(marked) {
1118
+ for (size_t i = 0; i < elements; i++) {
1119
+ aral_unmark_allocation(ar, pointers[i]);
1120
+ }
1121
}
1122
1123
for (size_t div = 5; div >= 2; div--) {
@@ -935,7 +1127,7 @@ static void *aral_test_thread(void *ptr) {
1127
}
1128
1129
for (size_t i = 0; i < elements / div; i++) {
938
- pointers[i] = aral_mallocz(ar);
1130
+ pointers[i] = unittest_aral_malloc(ar, marked);
1131
}
1132
}
1133
@@ -946,7 +1138,7 @@ static void *aral_test_thread(void *ptr) {
1138
}
1139
1140
for (size_t i = 0; i < elements; i += step) {
949
- pointers[i] = aral_mallocz(ar);
1141
+ pointers[i] = unittest_aral_malloc(ar, marked);
1142
}
1143
}
1144
@@ -955,7 +1147,7 @@ static void *aral_test_thread(void *ptr) {
1147
pointers[i] = NULL;
1148
}
1149
958
- if (auc->single_threaded && ar->aral_lock.pages && ar->aral_lock.pages->aral_lock.used_elements) {
1150
+ if (auc->single_threaded && ar->aral_lock.pages_free && ar->aral_lock.pages_free->aral_lock.used_elements) {
1151
fprintf(stderr, "\n\nARAL leftovers detected (1)\n\n");
1152
__atomic_add_fetch(&auc->errors, 1, __ATOMIC_RELAXED);
1153
}
@@ -964,7 +1156,7 @@ static void *aral_test_thread(void *ptr) {
1156
break;
1157
1158
for (size_t i = 0; i < elements; i++) {
967
- pointers[i] = aral_mallocz(ar);
1159
+ pointers[i] = unittest_aral_malloc(ar, marked);
1160
}
1161
1162
size_t increment = elements / ar->config.max_page_elements;
@@ -986,7 +1178,7 @@ static void *aral_test_thread(void *ptr) {
1178
1179
for (size_t i = 0; i < to_free; i++) {
1180
size_t pos = free_list[i];
989
- pointers[pos] = aral_mallocz(ar);
1181
+ pointers[pos] = unittest_aral_malloc(ar, marked);
1182
}
1183
}
1184
@@ -995,7 +1187,7 @@ static void *aral_test_thread(void *ptr) {
1187
pointers[i] = NULL;
1188
}
1189
998
- if (auc->single_threaded && ar->aral_lock.pages && ar->aral_lock.pages->aral_lock.used_elements) {
1190
+ if (auc->single_threaded && ar->aral_lock.pages_free && ar->aral_lock.pages_free->aral_lock.used_elements) {
1191
fprintf(stderr, "\n\nARAL leftovers detected (2)\n\n");
1192
__atomic_add_fetch(&auc->errors, 1, __ATOMIC_RELAXED);
1193
}
@@ -1014,7 +1206,13 @@ int aral_stress_test(size_t threads, size_t elements, size_t seconds) {
1206
struct aral_unittest_config auc = {
1207
.single_threaded = false,
1208
.threads = threads,
1017
- .ar = aral_create("aral-stress-test", 20, 0, 8192, NULL, "aral-stress-test", NULL, false, false),
1209
+ .ar = aral_create("aral-stress-test",
1210
+ sizeof(struct aral_unittest_entry),
1211
+ 0,
1212
+ 16384,
1213
+ NULL,
1214
+ "aral-stress-test",
1215
+ NULL, false, false),
1216
.elements = elements,
1217
.errors = 0,
1218
};
@@ -1061,7 +1259,7 @@ int aral_stress_test(size_t threads, size_t elements, size_t seconds) {
1259
1260
usec_t ended_ut = now_monotonic_usec();
1261
1064
- if (auc.ar->aral_lock.pages && auc.ar->aral_lock.pages->aral_lock.used_elements) {
1262
+ if (auc.ar->aral_lock.pages_free && auc.ar->aral_lock.pages_free->aral_lock.used_elements) {
1263
fprintf(stderr, "\n\nARAL leftovers detected (3)\n\n");
1264
__atomic_add_fetch(&auc.errors, 1, __ATOMIC_RELAXED);
1265
}
@@ -1084,7 +1282,15 @@ int aral_unittest(size_t elements) {
1282
struct aral_unittest_config auc = {
1283
.single_threaded = true,
1284
.threads = 1,
1087
- .ar = aral_create("aral-test", 20, 0, 8192, NULL, "aral-test", &cache_dir, false, false),
1285
+ .ar = aral_create("aral-test",
1286
+ sizeof(struct aral_unittest_entry),
1287
+ 0,
1288
+ 65536,
1289
+ NULL,
1290
+ "aral-test",
1291
+ &cache_dir,
1292
+ false,
1293
+ false),
1294
.elements = elements,
1295
.errors = 0,
1296
};
src/libnetdata/aral/aral.h
+36
-17
@@ -10,26 +10,33 @@ typedef struct aral ARAL;
10
11
struct aral_statistics {
12
struct {
13
- size_t allocations;
14
- size_t allocated_bytes;
13
+ alignas(64) size_t allocations;
14
+ alignas(64) size_t allocated_bytes;
15
} structures;
16
17
struct {
18
- size_t allocations;
19
- size_t allocated_bytes;
20
- size_t used_bytes;
18
+ alignas(64) size_t allocations;
19
+ alignas(64) size_t allocated_bytes;
20
+ alignas(64) size_t used_bytes;
21
} malloc;
22
23
struct {
24
- size_t allocations;
25
- size_t allocated_bytes;
26
- size_t used_bytes;
24
+ alignas(64) size_t allocations;
25
+ alignas(64) size_t allocated_bytes;
26
+ alignas(64) size_t used_bytes;
27
} mmap;
28
};
29
30
ARAL *aral_create(const char *name, size_t element_size, size_t initial_page_elements, size_t max_page_size,
31
struct aral_statistics *stats, const char *filename, const char **cache_dir, bool mmap, bool lockless);
32
-size_t aral_element_size(ARAL *ar);
32
+
33
+// return the size of the element, as requested
34
+size_t aral_requested_element_size(ARAL *ar);
35
+
36
+// return the exact memory footprint of the elements
37
+size_t aral_actual_element_size(ARAL *ar);
38
+
39
+const char *aral_name(ARAL *ar);
40
size_t aral_overhead(ARAL *ar);
41
size_t aral_structures(ARAL *ar);
42
struct aral_statistics *aral_get_statistics(ARAL *ar);
@@ -42,32 +49,44 @@ size_t aral_by_size_structures(void);
49
size_t aral_by_size_overhead(void);
50
struct aral_statistics *aral_by_size_statistics(void);
51
52
+size_t aral_by_size_used_bytes(void);
53
+size_t aral_used_bytes_from_stats(struct aral_statistics *stats);
54
+
55
+size_t aral_optimal_page_size(void);
56
+
57
int aral_unittest(size_t elements);
58
59
#ifdef NETDATA_TRACE_ALLOCATIONS
60
49
-#define aral_callocz(ar) aral_callocz_internal(ar, __FILE__, __FUNCTION__, __LINE__)
50
-#define aral_mallocz(ar) aral_mallocz_internal(ar, __FILE__, __FUNCTION__, __LINE__)
61
+#define aral_callocz(ar) aral_callocz_internal(ar, false, __FILE__, __FUNCTION__, __LINE__)
62
+#define aral_callocz_marked(ar) aral_callocz_internal(ar, true, __FILE__, __FUNCTION__, __LINE__)
63
+#define aral_mallocz(ar) aral_mallocz_internal(ar, false, __FILE__, __FUNCTION__, __LINE__)
64
+#define aral_mallocz_marked(ar) aral_mallocz_internal(ar, true, __FILE__, __FUNCTION__, __LINE__)
65
#define aral_freez(ar, ptr) aral_freez_internal(ar, ptr, __FILE__, __FUNCTION__, __LINE__)
66
#define aral_destroy(ar) aral_destroy_internal(ar, __FILE__, __FUNCTION__, __LINE__)
67
54
-void *aral_callocz_internal(ARAL *ar, const char *file, const char *function, size_t line);
55
-void *aral_mallocz_internal(ARAL *ar, const char *file, const char *function, size_t line);
68
+void *aral_callocz_internal(ARAL *ar, bool marked, const char *file, const char *function, size_t line);
69
+void *aral_mallocz_internal(ARAL *ar, bool marked, const char *file, const char *function, size_t line);
70
void aral_freez_internal(ARAL *ar, void *ptr, const char *file, const char *function, size_t line);
71
void aral_destroy_internal(ARAL *ar, const char *file, const char *function, size_t line);
72
73
#else // NETDATA_TRACE_ALLOCATIONS
74
61
-#define aral_mallocz(ar) aral_mallocz_internal(ar)
62
-#define aral_callocz(ar) aral_callocz_internal(ar)
75
+#define aral_mallocz(ar) aral_mallocz_internal(ar, false)
76
+#define aral_mallocz_marked(ar) aral_mallocz_internal(ar, true)
77
+#define aral_callocz(ar) aral_callocz_internal(ar, false)
78
+#define aral_callocz_marked(ar) aral_callocz_internal(ar, true)
79
#define aral_freez(ar, ptr) aral_freez_internal(ar, ptr)
80
#define aral_destroy(ar) aral_destroy_internal(ar)
81
66
-void *aral_callocz_internal(ARAL *ar);
67
-void *aral_mallocz_internal(ARAL *ar);
82
+
83
+void *aral_callocz_internal(ARAL *ar, bool marked);
84
+void *aral_mallocz_internal(ARAL *ar, bool marked);
85
void aral_freez_internal(ARAL *ar, void *ptr);
86
void aral_destroy_internal(ARAL *ar);
87
88
+void aral_unmark_allocation(ARAL *ar, void *ptr);
89
+
90
#endif // NETDATA_TRACE_ALLOCATIONS
91
92
#endif // ARAL_H
src/libnetdata/buffer/buffer.c
+14
@@ -350,6 +350,20 @@ __attribute__((constructor)) void initialize_ascii_maps(void) {
350
base64_value_from_ascii[(int)base64_digits[i]] = i;
351
}
352
353
+// ----------------------------------------------------------------------------
354
+
355
+void buffer_json_member_add_datetime_rfc3339(BUFFER *wb, const char *key, uint64_t datetime_ut, bool utc) {
356
+ char buf[RFC3339_MAX_LENGTH];
357
+ rfc3339_datetime_ut(buf, sizeof(buf), datetime_ut, 2, utc);
358
+ buffer_json_member_add_string(wb, key, buf);
359
+}
360
+
361
+void buffer_json_member_add_duration_ut(BUFFER *wb, const char *key, int64_t duration_ut) {
362
+ char buf[64];
363
+ duration_snprintf(buf, sizeof(buf), duration_ut, "us", true);
364
+ buffer_json_member_add_string(wb, key, buf);
365
+}
366
+
367
// ----------------------------------------------------------------------------
368
// unit test
369
src/libnetdata/buffer/buffer.h
+3
@@ -800,6 +800,9 @@ static inline void buffer_json_member_add_string_or_empty(BUFFER *wb, const char
800
buffer_json_member_add_string(wb, key, value);
801
}
802
803
+void buffer_json_member_add_datetime_rfc3339(BUFFER *wb, const char *key, uint64_t datetime_ut, bool utc);
804
+void buffer_json_member_add_duration_ut(BUFFER *wb, const char *key, int64_t duration_ut);
805
+
806
static inline void buffer_json_member_add_quoted_string(BUFFER *wb, const char *key, const char *value) {
807
buffer_print_json_comma_newline_spacing(wb);
808
buffer_print_json_key(wb, key);
src/libnetdata/buffered_reader/buffered_reader.h
+21
-12
@@ -115,22 +115,31 @@ static inline bool buffered_reader_next_line(struct buffered_reader *reader, BUF
115
return false;
116
}
117
118
- // copy all bytes to buffer
119
- while(ss < se && ds < de && *ss != '\n') {
120
- *ds++ = *ss++;
121
- dst->len++;
118
+ // Find out how many bytes we want to copy and whether we found a newline
119
+ size_t bytes_to_copy;
120
+ bool found_newline = false;
121
+ {
122
+ char *next_newline = (char *) memchr(ss, '\n', se - ss);
123
+ if (!next_newline) {
124
+ bytes_to_copy = se - ss;
125
+ } else {
126
+ bytes_to_copy = (next_newline - ss) + 1;
127
+ found_newline = true;
128
+ }
129
}
130
124
- // if we have a newline, return the buffer
125
- if(ss < se && ds < de && *ss == '\n') {
126
- // newline found in the r->read_buffer
127
-
128
- *ds++ = *ss++; // copy the newline too
129
- dst->len++;
131
+ // Check we don't overflow the destination buffer
132
+ if (bytes_to_copy > (size_t)(de - ds)) {
133
+ bytes_to_copy = de - ds;
134
+ found_newline = false;
135
+ }
136
131
- *ds = '\0';
137
+ memcpy(ds, ss, bytes_to_copy);
138
+ ds[bytes_to_copy] = '\0';
139
+ dst->len += bytes_to_copy;
140
133
- reader->pos = ss - reader->read_buffer;
141
+ if (found_newline) {
142
+ reader->pos = start + bytes_to_copy;
143
return true;
144
}
145
src/libnetdata/completion/completion.c
+28
-5
@@ -26,16 +26,16 @@ void completion_wait_for(struct completion *p)
26
uv_mutex_unlock(&p->mutex);
27
}
28
29
-bool completion_timedwait_for(struct completion *p, uint64_t timeout)
29
+bool completion_timedwait_for(struct completion *p, uint64_t timeout_s)
30
{
31
- timeout *= NSEC_PER_SEC;
31
+ timeout_s *= NSEC_PER_SEC;
32
33
uint64_t start_time = uv_hrtime();
34
bool result = true;
35
36
uv_mutex_lock(&p->mutex);
37
while (!p->completed) {
38
- int rc = uv_cond_timedwait(&p->cond, &p->mutex, timeout);
38
+ int rc = uv_cond_timedwait(&p->cond, &p->mutex, timeout_s);
39
40
if (rc == 0) {
41
result = true;
@@ -50,11 +50,11 @@ bool completion_timedwait_for(struct completion *p, uint64_t timeout)
50
*/
51
52
uint64_t elapsed = uv_hrtime() - start_time;
53
- if (elapsed >= timeout) {
53
+ if (elapsed >= timeout_s) {
54
result = false;
55
break;
56
}
57
- timeout -= elapsed;
57
+ timeout_s -= elapsed;
58
}
59
uv_mutex_unlock(&p->mutex);
60
@@ -81,6 +81,29 @@ unsigned completion_wait_for_a_job(struct completion *p, unsigned completed_jobs
81
return completed_jobs;
82
}
83
84
+unsigned completion_wait_for_a_job_with_timeout(struct completion *p, unsigned completed_jobs, uint64_t timeout_ms)
85
+{
86
+ uint64_t timeout_ns = timeout_ms * NSEC_PER_MSEC;
87
+ if(!timeout_ns) timeout_ns = 1;
88
+
89
+ uint64_t start_time_ns = uv_hrtime();
90
+
91
+ uv_mutex_lock(&p->mutex);
92
+ while (0 == p->completed && p->completed_jobs <= completed_jobs) {
93
+ int rc = uv_cond_timedwait(&p->cond, &p->mutex, timeout_ns);
94
+ if(rc == UV_ETIMEDOUT)
95
+ break;
96
+
97
+ uint64_t elapsed = uv_hrtime() - start_time_ns;
98
+ if (elapsed >= timeout_ns) break;
99
+ timeout_ns -= elapsed;
100
+ }
101
+ completed_jobs = p->completed_jobs;
102
+ uv_mutex_unlock(&p->mutex);
103
+
104
+ return completed_jobs;
105
+}
106
+
107
void completion_mark_complete_a_job(struct completion *p)
108
{
109
uv_mutex_lock(&p->mutex);
src/libnetdata/completion/completion.h
+2
-1
@@ -20,11 +20,12 @@ void completion_wait_for(struct completion *p);
20
21
// Wait for at most `timeout` seconds. Return true on success, false on
22
// error or timeout.
23
-bool completion_timedwait_for(struct completion *p, uint64_t timeout);
23
+bool completion_timedwait_for(struct completion *p, uint64_t timeout_s);
24
25
void completion_mark_complete(struct completion *p);
26
27
unsigned completion_wait_for_a_job(struct completion *p, unsigned completed_jobs);
28
+unsigned completion_wait_for_a_job_with_timeout(struct completion *p, unsigned completed_jobs, uint64_t timeout_ms);
29
void completion_mark_complete_a_job(struct completion *p);
30
bool completion_is_done(struct completion *p);
31
src/libnetdata/config/appconfig.c
+10
@@ -80,3 +80,13 @@ bool stream_conf_has_uuid_section(struct config *root) {
80
81
return is_parent;
82
}
83
+
84
+void appconfig_foreach_section(struct config *root, void (*cb)(struct config *root, const char *name, void *data), void *data) {
85
+ struct config_section *sect = NULL;
86
+
87
+ APPCONFIG_LOCK(root);
88
+ for (sect = root->sections; sect; sect = sect->next) {
89
+ cb(root, string2str(sect->name), data);
90
+ }
91
+ APPCONFIG_UNLOCK(root);
92
+}
src/libnetdata/config/appconfig.h
+3
-1
@@ -100,7 +100,7 @@
100
#define CONFIG_SECTION_PROMETHEUS "prometheus:exporter"
101
#define CONFIG_SECTION_HOST_LABEL "host labels"
102
#define EXPORTING_CONF "exporting.conf"
103
-#define CONFIG_SECTION_GLOBAL_STATISTICS "global statistics"
103
+#define CONFIG_SECTION_TELEMETRY "telemetry"
104
#define CONFIG_SECTION_DB "db"
105
106
// these are used to limit the configuration names and values lengths
@@ -183,6 +183,8 @@ _CONNECTOR_INSTANCE *add_connector_instance(struct config_section *connector, st
183
bool stream_conf_needs_dbengine(struct config *root);
184
bool stream_conf_has_uuid_section(struct config *root);
185
186
+void appconfig_foreach_section(struct config *root, void (*cb)(struct config *root, const char *name, void *data), void *data);
187
+
188
#include "appconfig_api_text.h"
189
#include "appconfig_api_numbers.h"
190
#include "appconfig_api_boolean.h"
src/libnetdata/config/appconfig_api_sizes.c
+2
-2
@@ -19,7 +19,7 @@ static STRING *reformat_size_bytes(STRING *value) {
19
20
uint64_t appconfig_get_size_bytes(struct config *root, const char *section, const char *name, uint64_t default_value) {
21
char default_str[128];
22
- size_snprintf_bytes(default_str, sizeof(default_str), (int)default_value);
22
+ size_snprintf_bytes(default_str, sizeof(default_str), default_value);
23
24
struct config_option *opt =
25
appconfig_get_raw_value(root, section, name, default_str, CONFIG_VALUE_TYPE_SIZE_IN_BYTES, reformat_size_bytes);
@@ -60,7 +60,7 @@ static STRING *reformat_size_mb(STRING *value) {
60
61
uint64_t appconfig_get_size_mb(struct config *root, const char *section, const char *name, uint64_t default_value) {
62
char default_str[128];
63
- size_snprintf_mb(default_str, sizeof(default_str), (int)default_value);
63
+ size_snprintf_mb(default_str, sizeof(default_str), default_value);
64
65
struct config_option *opt =
66
appconfig_get_raw_value(root, section, name, default_str, CONFIG_VALUE_TYPE_SIZE_IN_MB, reformat_size_mb);
src/libnetdata/config/appconfig_conf_file.c
+1
-1
@@ -227,7 +227,7 @@ void appconfig_generate(struct config *root, BUFFER *wb, int only_changed, bool
227
else if(!string_strcmp(sect->name, CONFIG_SECTION_WEBRTC)) pri = 11;
228
// by default, new sections will get pri = 12 (set at the end, below)
229
else if(!string_strcmp(sect->name, CONFIG_SECTION_REGISTRY)) pri = 13;
230
- else if(!string_strcmp(sect->name, CONFIG_SECTION_GLOBAL_STATISTICS)) pri = 14;
230
+ else if(!string_strcmp(sect->name, CONFIG_SECTION_TELEMETRY)) pri = 14;
231
else if(!string_strcmp(sect->name, CONFIG_SECTION_PLUGINS)) pri = 15;
232
else if(!string_strcmp(sect->name, CONFIG_SECTION_STATSD)) pri = 16;
233
else if(!string_strncmp(sect->name, "plugin:", 7)) pri = 17; // << change the loop too if you change this
src/libnetdata/dictionary/dictionary-item.h
+2
-2
@@ -84,9 +84,9 @@ static inline DICTIONARY_ITEM *dict_item_create(DICTIONARY *dict __maybe_unused,
84
85
static inline void *dict_item_value_mallocz(DICTIONARY *dict, size_t value_len) {
86
if(dict->value_aral) {
87
- internal_fatal(aral_element_size(dict->value_aral) != value_len,
87
+ internal_fatal(aral_requested_element_size(dict->value_aral) != value_len,
88
"DICTIONARY: item value size %zu does not match the configured fixed one %zu",
89
- value_len, aral_element_size(dict->value_aral));
89
+ value_len, aral_requested_element_size(dict->value_aral));
90
return aral_mallocz(dict->value_aral);
91
}
92
else
src/libnetdata/dictionary/dictionary.c
+4
-18
@@ -204,25 +204,11 @@ void dictionary_static_items_aral_init(void) {
204
if(unlikely(!dict_items_aral || !dict_shared_items_aral)) {
205
spinlock_lock(&spinlock);
206
207
- // we have to check again
207
if(!dict_items_aral)
209
- dict_items_aral = aral_create(
210
- "dict-items",
211
- sizeof(DICTIONARY_ITEM),
212
- 0,
213
- 65536,
214
- aral_by_size_statistics(),
215
- NULL, NULL, false, false);
216
-
217
- // we have to check again
208
+ dict_items_aral = aral_by_size_acquire(sizeof(DICTIONARY_ITEM));
209
+
210
if(!dict_shared_items_aral)
219
- dict_shared_items_aral = aral_create(
220
- "dict-shared-items",
221
- sizeof(DICTIONARY_ITEM_SHARED),
222
- 0,
223
- 65536,
224
- aral_by_size_statistics(),
225
- NULL, NULL, false, false);
211
+ dict_shared_items_aral = aral_by_size_acquire(sizeof(DICTIONARY_ITEM_SHARED));
212
213
spinlock_unlock(&spinlock);
214
}
@@ -540,7 +526,7 @@ DICTIONARY *dictionary_create_view(DICTIONARY *master) {
526
#endif
527
528
DICTIONARY *dict = dictionary_create_internal(master->options, master->stats,
543
- master->value_aral ? aral_element_size(master->value_aral) : 0);
529
+ master->value_aral ? aral_requested_element_size(master->value_aral) : 0);
530
531
dict->master = master;
532
src/libnetdata/dictionary/dictionary.h
+3
-3
@@ -98,9 +98,9 @@ struct dictionary_stats {
98
99
// memory
100
struct {
101
- long index; // bytes of keys indexed (indication of the index size)
102
- long values; // bytes of caller structures
103
- long dict; // bytes of the structures dictionary needs
101
+ ssize_t index; // bytes of keys indexed (indication of the index size)
102
+ ssize_t values; // bytes of caller structures
103
+ ssize_t dict; // bytes of the structures dictionary needs
104
} memory;
105
106
// spin locks
src/libnetdata/functions_evloop/functions_evloop.h
+2
-1
@@ -77,7 +77,8 @@
77
78
#define PLUGINSD_KEYWORD_JSON "JSON"
79
#define PLUGINSD_KEYWORD_JSON_END "JSON_PAYLOAD_END"
80
-#define PLUGINSD_KEYWORD_STREAM_PATH "STREAM_PATH"
80
+#define PLUGINSD_KEYWORD_JSON_CMD_STREAM_PATH "STREAM_PATH"
81
+#define PLUGINSD_KEYWORD_JSON_CMD_ML_MODEL "ML_MODEL"
82
83
typedef void (*functions_evloop_worker_execute_t)(const char *transaction, char *function, usec_t *stop_monotonic_ut,
84
bool *cancelled, BUFFER *payload, HTTP_ACCESS access,
src/libnetdata/gorilla/gorilla.cc
+85
-4
@@ -17,6 +17,17 @@ static constexpr size_t bit_size() noexcept
17
return (sizeof(T) * CHAR_BIT);
18
}
19
20
+static uint32_t gorilla_buffer_nbytes(uint32_t nbits) {
21
+ uint32_t slots = (nbits + RRDENG_GORILLA_32BIT_SLOT_BITS - 1) / RRDENG_GORILLA_32BIT_SLOT_BITS;
22
+ assert(slots > 0 && slots <= RRDENG_GORILLA_32BIT_BUFFER_SLOTS);
23
+
24
+ // this is needed to avoid heap buffer overflow in bit_buffer_read()
25
+ if(slots < RRDENG_GORILLA_32BIT_BUFFER_SLOTS)
26
+ slots++;
27
+
28
+ return slots * RRDENG_GORILLA_32BIT_SLOT_BYTES;
29
+}
30
+
31
static void bit_buffer_write(uint32_t *buf, size_t pos, uint32_t v, size_t nbits)
32
{
33
assert(nbits > 0 && nbits <= bit_size<uint32_t>());
@@ -191,20 +202,39 @@ gorilla_buffer_t *gorilla_writer_drop_head_buffer(gorilla_writer_t *gw) {
202
return curr_head;
203
}
204
194
-uint32_t gorilla_writer_nbytes(const gorilla_writer_t *gw)
205
+uint32_t gorilla_writer_actual_nbytes(const gorilla_writer_t *gw)
206
+{
207
+ uint32_t nbytes = 0;
208
+
209
+ const gorilla_buffer_t *curr_gbuf = __atomic_load_n(&gw->head_buffer, __ATOMIC_SEQ_CST);
210
+ do {
211
+ const gorilla_buffer_t *next_gbuf = __atomic_load_n(&curr_gbuf->header.next, __ATOMIC_SEQ_CST);
212
+
213
+ nbytes += RRDENG_GORILLA_32BIT_BUFFER_SIZE;
214
+
215
+ curr_gbuf = next_gbuf;
216
+ } while (curr_gbuf);
217
+
218
+ return nbytes;
219
+}
220
+
221
+uint32_t gorilla_writer_optimal_nbytes(const gorilla_writer_t *gw)
222
{
196
- uint32_t nbits = 0;
223
+ uint32_t nbytes = 0;
224
225
const gorilla_buffer_t *curr_gbuf = __atomic_load_n(&gw->head_buffer, __ATOMIC_SEQ_CST);
226
do {
227
const gorilla_buffer_t *next_gbuf = __atomic_load_n(&curr_gbuf->header.next, __ATOMIC_SEQ_CST);
228
202
- nbits += __atomic_load_n(&curr_gbuf->header.nbits, __ATOMIC_SEQ_CST);
229
+ if(next_gbuf)
230
+ nbytes += RRDENG_GORILLA_32BIT_BUFFER_SIZE;
231
+ else
232
+ nbytes += gorilla_buffer_nbytes(__atomic_load_n(&curr_gbuf->header.nbits, __ATOMIC_SEQ_CST));
233
234
curr_gbuf = next_gbuf;
235
} while (curr_gbuf);
236
207
- return (nbits + (CHAR_BIT - 1)) / CHAR_BIT;
237
+ return nbytes;
238
}
239
240
bool gorilla_writer_serialize(const gorilla_writer_t *gw, uint8_t *dst, uint32_t dst_size) {
@@ -247,6 +277,39 @@ uint32_t gorilla_buffer_patch(gorilla_buffer_t *gbuf) {
277
return n;
278
}
279
280
+size_t gorilla_buffer_unpatched_nbuffers(const gorilla_buffer_t *gbuf) {
281
+ size_t nbuffers = 0;
282
+ while(gbuf) {
283
+ nbuffers++;
284
+
285
+ if(gbuf->header.next) {
286
+ const auto *buf = reinterpret_cast<const uint32_t *>(gbuf);
287
+ gbuf = reinterpret_cast<const gorilla_buffer_t *>(&buf[RRDENG_GORILLA_32BIT_BUFFER_SLOTS]);
288
+ }
289
+ else
290
+ break;
291
+ }
292
+
293
+ return nbuffers;
294
+}
295
+
296
+size_t gorilla_buffer_unpatched_nbytes(const gorilla_buffer_t *gbuf) {
297
+ size_t nbytes = sizeof(gorilla_buffer_t);
298
+ while(gbuf) {
299
+ if(gbuf->header.next) {
300
+ nbytes += RRDENG_GORILLA_32BIT_BUFFER_SIZE;
301
+ const auto *buf = reinterpret_cast<const uint32_t *>(gbuf);
302
+ gbuf = reinterpret_cast<const gorilla_buffer_t *>(&buf[RRDENG_GORILLA_32BIT_BUFFER_SLOTS]);
303
+ }
304
+ else {
305
+ nbytes += gorilla_buffer_nbytes(gbuf->header.nbits);
306
+ break;
307
+ }
308
+ }
309
+
310
+ return nbytes;
311
+}
312
+
313
gorilla_reader_t gorilla_writer_get_reader(const gorilla_writer_t *gw)
314
{
315
const gorilla_buffer_t *buffer = __atomic_load_n(&gw->head_buffer, __ATOMIC_SEQ_CST);
@@ -358,6 +421,24 @@ bool gorilla_reader_read(gorilla_reader_t *gr, uint32_t *number)
421
return true;
422
}
423
424
+extern "C" {
425
+struct aral;
426
+void aral_unmark_allocation(struct aral *ar, void *ptr);
427
+}
428
+
429
+void gorilla_writer_aral_unmark(const gorilla_writer_t *gw, struct aral *ar)
430
+{
431
+ const gorilla_buffer_t *curr_gbuf = __atomic_load_n(&gw->head_buffer, __ATOMIC_SEQ_CST);
432
+ do {
433
+ const gorilla_buffer_t *next_gbuf = __atomic_load_n(&curr_gbuf->header.next, __ATOMIC_SEQ_CST);
434
+
435
+ // Call the C function here
436
+ aral_unmark_allocation(ar, const_cast<void*>(static_cast<const void*>(curr_gbuf)));
437
+
438
+ curr_gbuf = next_gbuf;
439
+ } while (curr_gbuf);
440
+}
441
+
442
/*
443
* Internal code used for fuzzing the library
444
*/
src/libnetdata/gorilla/gorilla.h
+10
-2
@@ -56,19 +56,27 @@ void gorilla_writer_add_buffer(gorilla_writer_t *gw, gorilla_buffer_t *gbuf, siz
56
bool gorilla_writer_write(gorilla_writer_t *gw, uint32_t number);
57
uint32_t gorilla_writer_entries(const gorilla_writer_t *gw);
58
59
+struct aral;
60
+void gorilla_writer_aral_unmark(const gorilla_writer_t *gw, struct aral *ar);
61
+
62
gorilla_reader_t gorilla_writer_get_reader(const gorilla_writer_t *gw);
63
64
gorilla_buffer_t *gorilla_writer_drop_head_buffer(gorilla_writer_t *gw);
65
63
-uint32_t gorilla_writer_nbytes(const gorilla_writer_t *gw);
66
+uint32_t gorilla_writer_actual_nbytes(const gorilla_writer_t *gw);
67
+uint32_t gorilla_writer_optimal_nbytes(const gorilla_writer_t *gw);
68
bool gorilla_writer_serialize(const gorilla_writer_t *gw, uint8_t *dst, uint32_t dst_size);
69
70
uint32_t gorilla_buffer_patch(gorilla_buffer_t *buf);
71
+size_t gorilla_buffer_unpatched_nbuffers(const gorilla_buffer_t *gbuf);
72
+size_t gorilla_buffer_unpatched_nbytes(const gorilla_buffer_t *gbuf);
73
gorilla_reader_t gorilla_reader_init(gorilla_buffer_t *buf);
74
bool gorilla_reader_read(gorilla_reader_t *gr, uint32_t *number);
75
76
+#define RRDENG_GORILLA_32BIT_SLOT_BYTES sizeof(uint32_t)
77
+#define RRDENG_GORILLA_32BIT_SLOT_BITS (RRDENG_GORILLA_32BIT_SLOT_BYTES * CHAR_BIT)
78
#define RRDENG_GORILLA_32BIT_BUFFER_SLOTS 128
71
-#define RRDENG_GORILLA_32BIT_BUFFER_SIZE (RRDENG_GORILLA_32BIT_BUFFER_SLOTS * sizeof(uint32_t))
79
+#define RRDENG_GORILLA_32BIT_BUFFER_SIZE (RRDENG_GORILLA_32BIT_BUFFER_SLOTS * RRDENG_GORILLA_32BIT_SLOT_BYTES)
80
81
#ifdef __cplusplus
82
}
src/libnetdata/json/json.c
+6
-5
@@ -1,8 +1,9 @@
1
-#include "jsmn.h"
2
-#include "../libnetdata.h"
3
-#include "json.h"
1
#include "libnetdata/libnetdata.h"
5
-#include "health/health.h"
2
+#include "json.h"
3
+
4
+#ifndef ENABLE_JSONC
5
+#include "jsmn.h"
6
+#endif
7
8
#define JSON_TOKENS 1024
9
@@ -450,7 +451,7 @@ size_t json_walk(json_object *t, void *callback_data, int (*callback_function)(s
451
type = json_object_get_type(val);
452
if (type == json_type_array) {
453
e.type = JSON_ARRAY;
453
- json_jsonc_parse_array(val,NULL,health_silencers_json_read_callback);
454
+ json_jsonc_parse_array(val,NULL,callback_function);
455
} else if (type == json_type_object) {
456
e.type = JSON_OBJECT;
457
} else if (type == json_type_string) {
src/libnetdata/libjudy/judy-malloc.c
+17
-1
@@ -28,7 +28,7 @@ __attribute__((constructor)) void aral_judy_init(void) {
28
buf,
29
Words * sizeof(Word_t),
30
0,
31
- 65536,
31
+ 0,
32
&judy_sizes_aral_statistics,
33
NULL, NULL, false, false);
34
}
@@ -49,6 +49,18 @@ static ARAL *judy_size_aral(Word_t Words) {
49
return NULL;
50
}
51
52
+static __thread int64_t judy_allocated = 0;
53
+
54
+void JudyAllocThreadTelemetryReset(void) {
55
+ judy_allocated = 0;
56
+}
57
+
58
+int64_t JudyAllocThreadTelemetryGetAndReset(void) {
59
+ int64_t rc = judy_allocated;
60
+ judy_allocated = 0;
61
+ return rc;
62
+}
63
+
64
inline Word_t JudyMalloc(Word_t Words) {
65
Word_t Addr;
66
@@ -58,6 +70,8 @@ inline Word_t JudyMalloc(Word_t Words) {
70
else
71
Addr = (Word_t) mallocz(Words * sizeof(Word_t));
72
73
+ judy_allocated += Words * sizeof(Word_t);
74
+
75
return(Addr);
76
}
77
@@ -67,6 +81,8 @@ inline void JudyFree(void * PWord, Word_t Words) {
81
aral_freez(ar, PWord);
82
else
83
freez(PWord);
84
+
85
+ judy_allocated -= Words * sizeof(Word_t);
86
}
87
88
Word_t JudyMallocVirtual(Word_t Words) {
src/libnetdata/libjudy/judy-malloc.h
+3
@@ -8,4 +8,7 @@
8
size_t judy_aral_overhead(void);
9
size_t judy_aral_structures(void);
10
11
+void JudyAllocThreadTelemetryReset(void);
12
+int64_t JudyAllocThreadTelemetryGetAndReset(void);
13
+
14
#endif //NETDATA_JUDY_MALLOC_H
src/libnetdata/libjudy/judyl-typed.h
new
+78
@@ -0,0 +1,78 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_JUDYL_TYPED_H
4
+#define NETDATA_JUDYL_TYPED_H
5
+
6
+#include <Judy.h>
7
+
8
+#define DEFINE_JUDYL_TYPED(NAME, TYPE) \
9
+ _Static_assert(sizeof(TYPE) == sizeof(Word_t), \
10
+ #NAME "_type_must_have_same_size_as_Word_t"); \
11
+ typedef struct { \
12
+ Pvoid_t judyl; \
13
+ } NAME##_JudyLSet; \
14
+ \
15
+ static inline void NAME##_INIT(NAME##_JudyLSet *set) { \
16
+ set->judyl = NULL; \
17
+ } \
18
+ \
19
+ static inline bool NAME##_SET(NAME##_JudyLSet *set, Word_t index, TYPE value) { \
20
+ Pvoid_t *pValue = JudyLIns(&set->judyl, index, PJE0); \
21
+ if (pValue == PJERR) return false; \
22
+ *pValue = (void *)(uintptr_t)value; \
23
+ return true; \
24
+ } \
25
+ \
26
+ static inline TYPE NAME##_GET(NAME##_JudyLSet *set, Word_t index) { \
27
+ Pvoid_t *pValue = JudyLGet(set->judyl, index, PJE0); \
28
+ return (pValue != NULL) ? (TYPE)(uintptr_t)(*pValue) : (TYPE)0; \
29
+ } \
30
+ \
31
+ static inline TYPE *NAME##_GETPTR(NAME##_JudyLSet *set, Word_t index) { \
32
+ Pvoid_t *pValue = JudyLGet(set->judyl, index, PJE0); \
33
+ return (TYPE *)pValue; \
34
+ } \
35
+ \
36
+ static inline bool NAME##_DEL(NAME##_JudyLSet *set, Word_t index) { \
37
+ int Rc; \
38
+ PPvoid_t ppJudy = &set->judyl; \
39
+ Rc = JudyLDel(ppJudy, index, PJE0); \
40
+ return Rc == 1; \
41
+ } \
42
+ \
43
+ static inline TYPE NAME##_FIRST(NAME##_JudyLSet *set, Word_t *index) { \
44
+ Pvoid_t *pValue = JudyLFirst(set->judyl, index, PJE0); \
45
+ return (pValue != NULL) ? (TYPE)(uintptr_t)(*pValue) : (TYPE)0; \
46
+ } \
47
+ \
48
+ static inline TYPE NAME##_NEXT(NAME##_JudyLSet *set, Word_t *index) { \
49
+ Pvoid_t *pValue = JudyLNext(set->judyl, index, PJE0); \
50
+ return (pValue != NULL) ? (TYPE)(uintptr_t)(*pValue) : (TYPE)0; \
51
+ } \
52
+ \
53
+ static inline TYPE NAME##_LAST(NAME##_JudyLSet *set, Word_t *index) { \
54
+ Pvoid_t *pValue = JudyLLast(set->judyl, index, PJE0); \
55
+ return (pValue != NULL) ? (TYPE)(uintptr_t)(*pValue) : (TYPE)0; \
56
+ } \
57
+ \
58
+ static inline TYPE NAME##_PREV(NAME##_JudyLSet *set, Word_t *index) { \
59
+ Pvoid_t *pValue = JudyLPrev(set->judyl, index, PJE0); \
60
+ return (pValue != NULL) ? (TYPE)(uintptr_t)(*pValue) : (TYPE)0; \
61
+ } \
62
+ \
63
+ static inline void NAME##_FREE(NAME##_JudyLSet *set, void (*callback)(TYPE)) { \
64
+ Word_t index = 0; \
65
+ Pvoid_t *pValue; \
66
+ if (callback) { \
67
+ for (pValue = JudyLFirst(set->judyl, &index, PJE0); \
68
+ pValue != NULL; \
69
+ pValue = JudyLNext(set->judyl, &index, PJE0)) { \
70
+ callback((TYPE)(uintptr_t)(*pValue)); \
71
+ } \
72
+ } \
73
+ JudyLFreeArray(&set->judyl, PJE0); \
74
+ }
75
+
76
+
77
+
78
+#endif //NETDATA_JUDYL_TYPED_H
src/libnetdata/libnetdata.c
+39
@@ -446,9 +446,26 @@ void *reallocz(void *ptr, size_t size) {
446
void posix_memfree(void *ptr) {
447
free(ptr);
448
}
449
+#endif
450
+
451
+void mallocz_release_as_much_memory_to_the_system(void) {
452
+#if defined(HAVE_C_MALLOPT) || defined(HAVE_C_MALLOC_TRIM)
453
+ static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER;
454
+ spinlock_lock(&spinlock);
455
+
456
+#ifdef HAVE_C_MALLOPT
457
+ size_t trim_threshold = aral_optimal_page_size();
458
+ mallopt(M_TRIM_THRESHOLD, (int)trim_threshold);
459
+#endif
460
461
+#ifdef HAVE_C_MALLOC_TRIM
462
+ malloc_trim(0);
463
#endif
464
465
+ spinlock_unlock(&spinlock);
466
+#endif
467
+}
468
+
469
// --------------------------------------------------------------------------------------------------------------------
470
471
void json_escape_string(char *dst, const char *src, size_t size) {
@@ -848,6 +865,28 @@ struct timing_steps {
865
[TIMING_STEP_END2_PROPAGATE] = { .name = "END2 propagate", .time = 0, },
866
[TIMING_STEP_END2_STORE] = { .name = "END2 store", .time = 0, },
867
868
+ [TIMING_STEP_DBENGINE_EVICT_LOCK] = { .name = "EVC_LOCK", .time = 0, },
869
+ [TIMING_STEP_DBENGINE_EVICT_SELECT] = { .name = "EVC_SELECT", .time = 0, },
870
+ [TIMING_STEP_DBENGINE_EVICT_SELECT_PAGE ] = { .name = "EVT_SELECT_PAGE", .time = 0, },
871
+ [TIMING_STEP_DBENGINE_EVICT_RELOCATE_PAGE ] = { .name = "EVT_RELOCATE_PAGE", .time = 0, },
872
+ [TIMING_STEP_DBENGINE_EVICT_SORT] = { .name = "EVC_SORT", .time = 0, },
873
+ [TIMING_STEP_DBENGINE_EVICT_DEINDEX] = { .name = "EVC_DEINDEX", .time = 0, },
874
+ [TIMING_STEP_DBENGINE_EVICT_DEINDEX_PAGE] = { .name = "EVC_DEINDEX_PAGE", .time = 0, },
875
+ [TIMING_STEP_DBENGINE_EVICT_FINISHED] = { .name = "EVC_FINISHED", .time = 0, },
876
+ [TIMING_STEP_DBENGINE_EVICT_FREE_LOOP] = { .name = "EVC_FREE_LOOP", .time = 0, },
877
+ [TIMING_STEP_DBENGINE_EVICT_FREE_PAGE] = { .name = "EVC_FREE_PAGE", .time = 0, },
878
+ [TIMING_STEP_DBENGINE_EVICT_FREE_ATOMICS] = { .name = "EVC_FREE_ATOMICS", .time = 0, },
879
+ [TIMING_STEP_DBENGINE_EVICT_FREE_CB] = { .name = "EVC_FREE_CB", .time = 0, },
880
+ [TIMING_STEP_DBENGINE_EVICT_FREE_ATOMICS2] = { .name = "EVC_FREE_ATOMICS2", .time = 0, },
881
+ [TIMING_STEP_DBENGINE_EVICT_FREE_ARAL] = { .name = "EVC_FREE_ARAL", .time = 0, },
882
+ [TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_DATA] = { .name = "EVC_FREE_PGD_DATA", .time = 0, },
883
+ [TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_ARAL] = { .name = "EVC_FREE_PGD_ARAL", .time = 0, },
884
+ [TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_TIER1_ARAL] = { .name = "EVC_FREE_MAIN_T1ARL", .time = 0, },
885
+ [TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_GLIVE] = { .name = "EVC_FREE_MAIN_GLIVE", .time = 0, },
886
+ [TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_GWORKER] = { .name = "EVC_FREE_MAIN_GWORK", .time = 0, },
887
+ [TIMING_STEP_DBENGINE_EVICT_FREE_OPEN] = { .name = "EVC_FREE_OPEN", .time = 0, },
888
+ [TIMING_STEP_DBENGINE_EVICT_FREE_EXTENT] = { .name = "EVC_FREE_EXTENT", .time = 0, },
889
+
890
// terminator
891
[TIMING_STEP_MAX] = { .name = NULL, .time = 0, },
892
};
src/libnetdata/libnetdata.h
+40
-2
@@ -61,6 +61,7 @@ void *reallocz(void *ptr, size_t size) MALLOCLIKE NEVERNULL;
61
void freez(void *ptr);
62
#endif // NETDATA_TRACE_ALLOCATIONS
63
64
+void mallocz_release_as_much_memory_to_the_system(void);
65
void posix_memfree(void *ptr);
66
67
void json_escape_string(char *dst, const char *src, size_t size);
@@ -86,8 +87,6 @@ char *find_and_replace(const char *src, const char *find, const char *replace, c
87
#define BITS_IN_A_KILOBIT 1000
88
#define KILOBITS_IN_A_MEGABIT 1000
89
89
-#define error_report(x, args...) do { errno_clear(); netdata_log_error(x, ##args); } while(0)
90
-
90
#include "bitmap/bitmap64.h"
91
92
#define COMPRESSION_MAX_CHUNK 0x4000
@@ -109,6 +108,7 @@ extern const char *netdata_configured_host_prefix;
108
// safe includes before O/S specific functions
109
#include "template-enum.h"
110
#include "libjudy/src/Judy.h"
111
+#include "libjudy/judyl-typed.h"
112
#include "july/july.h"
113
114
#include "string/string.h"
@@ -134,6 +134,12 @@ extern const char *netdata_configured_host_prefix;
134
#include "os/os.h"
135
136
#include "socket/socket.h"
137
+#include "socket/nd-sock.h"
138
+#include "socket/nd-poll.h"
139
+#include "socket/listen-sockets.h"
140
+#include "socket/poll-events.h"
141
+#include "socket/connect-to.h"
142
+#include "socket/socket-peers.h"
143
#include "avl/avl.h"
144
145
#include "line_splitter/line_splitter.h"
@@ -290,6 +296,28 @@ typedef enum {
296
TIMING_STEP_FREEIPMI_READ_event_offset_string,
297
TIMING_STEP_FREEIPMI_READ_manufacturer_id,
298
299
+ TIMING_STEP_DBENGINE_EVICT_LOCK,
300
+ TIMING_STEP_DBENGINE_EVICT_SELECT,
301
+ TIMING_STEP_DBENGINE_EVICT_SELECT_PAGE,
302
+ TIMING_STEP_DBENGINE_EVICT_RELOCATE_PAGE,
303
+ TIMING_STEP_DBENGINE_EVICT_SORT,
304
+ TIMING_STEP_DBENGINE_EVICT_DEINDEX,
305
+ TIMING_STEP_DBENGINE_EVICT_DEINDEX_PAGE,
306
+ TIMING_STEP_DBENGINE_EVICT_FINISHED,
307
+ TIMING_STEP_DBENGINE_EVICT_FREE_LOOP,
308
+ TIMING_STEP_DBENGINE_EVICT_FREE_PAGE,
309
+ TIMING_STEP_DBENGINE_EVICT_FREE_ATOMICS,
310
+ TIMING_STEP_DBENGINE_EVICT_FREE_CB,
311
+ TIMING_STEP_DBENGINE_EVICT_FREE_ATOMICS2,
312
+ TIMING_STEP_DBENGINE_EVICT_FREE_ARAL,
313
+ TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_DATA,
314
+ TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_ARAL,
315
+ TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_TIER1_ARAL,
316
+ TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_GLIVE,
317
+ TIMING_STEP_DBENGINE_EVICT_FREE_MAIN_PGD_GWORKER,
318
+ TIMING_STEP_DBENGINE_EVICT_FREE_OPEN,
319
+ TIMING_STEP_DBENGINE_EVICT_FREE_EXTENT,
320
+
321
// terminator
322
TIMING_STEP_MAX,
323
} TIMING_STEP;
@@ -304,11 +332,21 @@ typedef enum {
332
#define timing_init() timing_action(TIMING_ACTION_INIT, TIMING_STEP_INTERNAL)
333
#define timing_step(step) timing_action(TIMING_ACTION_STEP, step)
334
#define timing_report() timing_action(TIMING_ACTION_FINISH, TIMING_STEP_INTERNAL)
335
+
336
+#define timing_dbengine_evict_init() timing_action(TIMING_ACTION_INIT, TIMING_STEP_INTERNAL)
337
+#define timing_dbengine_evict_step(step) timing_action(TIMING_ACTION_STEP, step)
338
+#define timing_dbengine_evict_report() timing_action(TIMING_ACTION_FINISH, TIMING_STEP_INTERNAL)
339
#else
340
#define timing_init() debug_dummy()
341
#define timing_step(step) debug_dummy()
342
#define timing_report() debug_dummy()
343
+
344
+#define timing_dbengine_evict_init() debug_dummy()
345
+#define timing_dbengine_evict_step(step) debug_dummy()
346
+#define timing_dbengine_evict_report() debug_dummy()
347
#endif
348
+
349
+
350
void timing_action(TIMING_ACTION action, TIMING_STEP step);
351
352
int hash256_string(const unsigned char *string, size_t size, char *hash);
src/libnetdata/local-sockets/local-sockets.h
+2
-2
@@ -1188,14 +1188,14 @@ static inline void local_sockets_init(LS_STATE *ls) {
1188
ls->local_socket_aral = aral_create(
1189
"local-sockets",
1190
sizeof(LOCAL_SOCKET),
1191
- 65536,
1191
+ 65536 / sizeof(LOCAL_SOCKET),
1192
65536,
1193
NULL, NULL, NULL, false, true);
1194
1195
ls->pid_socket_aral = aral_create(
1196
"pid-sockets",
1197
sizeof(struct pid_socket),
1198
- 65536,
1198
+ 65536 / sizeof(struct pid_socket),
1199
65536,
1200
NULL, NULL, NULL, false, true);
1201
src/libnetdata/locks/locks.c
+32
-10
@@ -253,9 +253,13 @@ static inline void spinlock_lock_internal(SPINLOCK *spinlock)
253
spins++;
254
#endif
255
256
- if(unlikely(i == 8)) {
257
- i = 0;
258
- tinysleep();
256
+ if(unlikely(i % 8 == 0)) {
257
+ if(i == 8 * 4) {
258
+ i = 0;
259
+ yield_the_processor();
260
+ }
261
+ else
262
+ tinysleep();
263
}
264
}
265
@@ -373,13 +377,22 @@ bool spinlock_trylock_cancelable(SPINLOCK *spinlock)
377
378
void rw_spinlock_init(RW_SPINLOCK *rw_spinlock) {
379
rw_spinlock->readers = 0;
380
+ rw_spinlock->writers_waiting = 0;
381
spinlock_init(&rw_spinlock->spinlock);
382
}
383
384
void rw_spinlock_read_lock(RW_SPINLOCK *rw_spinlock) {
380
- spinlock_lock(&rw_spinlock->spinlock);
381
- __atomic_add_fetch(&rw_spinlock->readers, 1, __ATOMIC_RELAXED);
382
- spinlock_unlock(&rw_spinlock->spinlock);
385
+ while(1) {
386
+ spinlock_lock(&rw_spinlock->spinlock);
387
+ if (!rw_spinlock->writers_waiting) {
388
+ __atomic_add_fetch(&rw_spinlock->readers, 1, __ATOMIC_RELAXED);
389
+ spinlock_unlock(&rw_spinlock->spinlock);
390
+ break;
391
+ }
392
+
393
+ spinlock_unlock(&rw_spinlock->spinlock);
394
+ yield_the_processor(); // let the writer run
395
+ }
396
397
nd_thread_rwspinlock_read_locked();
398
}
@@ -398,16 +411,25 @@ void rw_spinlock_read_unlock(RW_SPINLOCK *rw_spinlock) {
411
412
void rw_spinlock_write_lock(RW_SPINLOCK *rw_spinlock) {
413
size_t spins = 0;
401
- while(1) {
402
- spins++;
414
+ for(size_t i = 1; true ;i++) {
415
spinlock_lock(&rw_spinlock->spinlock);
416
405
- if(__atomic_load_n(&rw_spinlock->readers, __ATOMIC_RELAXED) == 0)
417
+ if(__atomic_load_n(&rw_spinlock->readers, __ATOMIC_RELAXED) == 0) {
418
+ if(spins != 0)
419
+ rw_spinlock->writers_waiting--;
420
break;
421
+ }
422
+
423
+ if(spins == 0)
424
+ rw_spinlock->writers_waiting++;
425
426
// Busy wait until all readers have released their locks.
427
spinlock_unlock(&rw_spinlock->spinlock);
410
- tinysleep();
428
+ if(i == 8 * 2) {
429
+ i = 0;
430
+ tinysleep();
431
+ }
432
+ spins++;
433
}
434
435
(void)spins;
src/libnetdata/locks/locks.h
+1
@@ -46,6 +46,7 @@ bool spinlock_trylock_cancelable(SPINLOCK *spinlock);
46
47
typedef struct netdata_rw_spinlock {
48
int32_t readers;
49
+ int32_t writers_waiting;
50
SPINLOCK spinlock;
51
} RW_SPINLOCK;
52
src/libnetdata/log/nd_log.h
+2
@@ -172,6 +172,8 @@ void netdata_logger_with_limit(ERROR_LIMIT *erl, ND_LOG_SOURCES source, ND_LOG_F
172
173
void netdata_logger_fatal( const char *file, const char *function, unsigned long line, const char *fmt, ... ) NORETURN PRINTFLIKE(4, 5);
174
175
+#define error_report(x, args...) do { errno_clear(); netdata_log_error(x, ##args); } while(0)
176
+
177
# ifdef __cplusplus
178
}
179
# endif
src/libnetdata/os/os.h
+1
@@ -7,6 +7,7 @@
7
#include <sys/syscall.h>
8
#endif
9
10
+#include "system_memory.h"
11
#include "random.h"
12
#include "timestamps.h"
13
#include "setproctitle.h"
src/libnetdata/os/sleep.c
+12
-1
@@ -4,7 +4,8 @@
4
5
#ifdef OS_WINDOWS
6
void tinysleep(void) {
7
- Sleep(1);
7
+ Sleep(0);
8
+ // SwitchToThread();
9
}
10
#else
11
void tinysleep(void) {
@@ -13,6 +14,16 @@ void tinysleep(void) {
14
}
15
#endif
16
17
+#ifdef OS_WINDOWS
18
+void yield_the_processor(void) {
19
+ Sleep(0);
20
+}
21
+#else
22
+void yield_the_processor(void) {
23
+ sched_yield();
24
+}
25
+#endif
26
+
27
#ifdef OS_WINDOWS
28
void microsleep(usec_t ut) {
29
size_t ms = ut / USEC_PER_MS + ((ut == 0 || (ut % USEC_PER_MS)) ? 1 : 0);
src/libnetdata/os/sleep.h
+1
@@ -3,6 +3,7 @@
3
#ifndef NETDATA_SLEEP_H
4
#define NETDATA_SLEEP_H
5
6
+void yield_the_processor(void);
7
void tinysleep(void);
8
void microsleep(usec_t ut);
9
src/libnetdata/os/system_memory.c
new
+308
@@ -0,0 +1,308 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "libnetdata/libnetdata.h"
4
+
5
+// Windows
6
+#if defined(OS_WINDOWS)
7
+#include <windows.h>
8
+
9
+OS_SYSTEM_MEMORY os_system_memory(bool query_total_ram __maybe_unused) {
10
+ OS_SYSTEM_MEMORY sm = {0, 0};
11
+
12
+ MEMORYSTATUSEX statex;
13
+ statex.dwLength = sizeof(statex);
14
+ if (GlobalMemoryStatusEx(&statex)) {
15
+ sm.ram_total_bytes = statex.ullTotalPhys;
16
+ sm.ram_available_bytes = statex.ullAvailPhys;
17
+ }
18
+
19
+ return sm;
20
+}
21
+#endif
22
+
23
+// macOS
24
+#if defined(OS_MACOS)
25
+#include <mach/mach.h>
26
+#include <sys/sysctl.h>
27
+
28
+OS_SYSTEM_MEMORY os_system_memory(bool query_total_ram) {
29
+ static uint64_t total_ram = 0;
30
+ static uint64_t page_size = 0;
31
+
32
+ if (page_size == 0) {
33
+ size_t len = sizeof(page_size);
34
+ if (sysctlbyname("hw.pagesize", &page_size, &len, NULL, 0) != 0)
35
+ return (OS_SYSTEM_MEMORY){ 0, 0 };
36
+ }
37
+
38
+ if (query_total_ram || total_ram == 0) {
39
+ size_t len = sizeof(total_ram);
40
+ if (sysctlbyname("hw.memsize", &total_ram, &len, NULL, 0) != 0)
41
+ return (OS_SYSTEM_MEMORY){ 0, 0 };
42
+ }
43
+
44
+ uint64_t ram_available = 0;
45
+ if (page_size > 0) {
46
+ vm_statistics64_data_t vm_info;
47
+ mach_msg_type_number_t count = HOST_VM_INFO64_COUNT;
48
+ mach_port_t mach_port = mach_host_self();
49
+
50
+ if (host_statistics64(mach_port, HOST_VM_INFO64, (host_info_t)&vm_info, &count) != KERN_SUCCESS) {
51
+ mach_port_deallocate(mach_task_self(), mach_port);
52
+ return (OS_SYSTEM_MEMORY){0, 0};
53
+ }
54
+
55
+ ram_available = (vm_info.free_count + vm_info.inactive_count + vm_info.purgeable_count) * page_size;
56
+ mach_port_deallocate(mach_task_self(), mach_port);
57
+ }
58
+
59
+ return (OS_SYSTEM_MEMORY){
60
+ .ram_total_bytes = total_ram,
61
+ .ram_available_bytes = ram_available,
62
+ };
63
+}
64
+#endif
65
+
66
+// Linux
67
+#if defined(OS_LINUX)
68
+
69
+static OS_SYSTEM_MEMORY os_system_memory_cgroup_v1(bool query_total_ram __maybe_unused) {
70
+ static OS_SYSTEM_MEMORY sm = {0, 0};
71
+ char buf[64];
72
+
73
+ if(query_total_ram || sm.ram_total_bytes == 0) {
74
+ if (read_txt_file("/sys/fs/cgroup/memory/memory.limit_in_bytes", buf, sizeof(buf)) != 0) {
75
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroups v1: cannot read /sys/fs/cgroup/memory/memory.limit_in_bytes");
76
+ goto failed;
77
+ }
78
+
79
+ sm.ram_total_bytes = strtoull(buf, NULL, 10);
80
+ if(!sm.ram_total_bytes) {
81
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroups v1: /sys/fs/cgroup/memory/memory.limit_in_bytes is zero");
82
+ goto failed;
83
+ }
84
+ }
85
+
86
+ buf[0] = '\0';
87
+ if (read_txt_file("/sys/fs/cgroup/memory/memory.usage_in_bytes", buf, sizeof(buf)) != 0) {
88
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroups v1: cannot read /sys/fs/cgroup/memory/memory.usage_in_bytes");
89
+ goto failed;
90
+ }
91
+
92
+ uint64_t used = strtoull(buf, NULL, 10);
93
+ if(!used || used > sm.ram_total_bytes) {
94
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroups v1: used is %llu, total is %llu: used is invalid",
95
+// used, sm.ram_total_bytes);
96
+ goto failed;
97
+ }
98
+
99
+ sm.ram_available_bytes = sm.ram_total_bytes - used;
100
+ return sm;
101
+
102
+failed:
103
+ sm.ram_total_bytes = 0;
104
+ sm.ram_available_bytes = 0;
105
+ return sm;
106
+}
107
+
108
+static OS_SYSTEM_MEMORY os_system_memory_cgroup_v2(bool query_total_ram __maybe_unused) {
109
+ static OS_SYSTEM_MEMORY sm = {0, 0};
110
+ char buf[64];
111
+
112
+ if(query_total_ram || sm.ram_total_bytes == 0) {
113
+ if (read_txt_file("/sys/fs/cgroup/memory.max", buf, sizeof(buf)) != 0) {
114
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroups v2: cannot read /sys/fs/cgroup/memory.max");
115
+ goto failed;
116
+ }
117
+
118
+ if(strcmp(buf, "max") == 0)
119
+ sm.ram_total_bytes = UINT64_MAX;
120
+ else
121
+ sm.ram_total_bytes = strtoull(buf, NULL, 0);
122
+
123
+ if(!sm.ram_total_bytes) {
124
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroups v2: /sys/fs/cgroup/memory.max is zero");
125
+ goto failed;
126
+ }
127
+ }
128
+
129
+ buf[0] = '\0';
130
+ if (read_txt_file("/sys/fs/cgroup/memory.current", buf, sizeof(buf)) != 0) {
131
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroups v2: cannot read /sys/fs/cgroup/memory.current");
132
+ goto failed;
133
+ }
134
+
135
+ uint64_t used = strtoull(buf, NULL, 0);
136
+ if(!used || used > sm.ram_total_bytes) {
137
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroups v2: used is %llu, total is %llu: used is invalid",
138
+// used, sm.ram_total_bytes);
139
+ goto failed;
140
+ }
141
+
142
+ sm.ram_available_bytes = sm.ram_total_bytes - used;
143
+ return sm;
144
+
145
+failed:
146
+ sm.ram_total_bytes = 0;
147
+ sm.ram_available_bytes = 0;
148
+ return sm;
149
+}
150
+
151
+OS_SYSTEM_MEMORY os_system_memory_meminfo(bool query_total_ram __maybe_unused) {
152
+ static OS_SYSTEM_MEMORY sm = {0, 0};
153
+ static procfile *ff = NULL;
154
+
155
+ if(unlikely(!ff)) {
156
+ ff = procfile_open("/proc/meminfo", ": \t", PROCFILE_FLAG_DEFAULT);
157
+ if(unlikely(!ff))
158
+ goto failed;
159
+ }
160
+
161
+ ff = procfile_readall(ff);
162
+ if(unlikely(!ff))
163
+ goto failed;
164
+
165
+ bool matched_total = false, matched_available = false;
166
+ size_t lines = procfile_lines(ff);
167
+ for(size_t line = 0; line < lines ;line++) {
168
+ if(!matched_total && strcmp(procfile_lineword(ff, line, 0), "MemTotal") == 0) {
169
+ sm.ram_total_bytes = str2ull(procfile_lineword(ff, line, 1), NULL) * 1024;
170
+ matched_total = true;
171
+ }
172
+
173
+ if(!matched_available && strcmp(procfile_lineword(ff, line, 0), "MemAvailable") == 0) {
174
+ sm.ram_available_bytes = str2ull(procfile_lineword(ff, line, 1), NULL) * 1024;
175
+ matched_available = true;
176
+ }
177
+
178
+ if(matched_total && matched_available)
179
+ break;
180
+ }
181
+
182
+ // we keep ff open to speed up the next calls
183
+ return sm;
184
+
185
+failed:
186
+ sm.ram_total_bytes = 0;
187
+ sm.ram_available_bytes = 0;
188
+ return sm;
189
+}
190
+
191
+typedef enum {
192
+ OS_MEM_SRC_UNKNOWN,
193
+ OS_MEM_SRC_CGROUP_V1,
194
+ OS_MEM_SRC_CGROUP_V2,
195
+ OS_MEM_SRC_MEMINFO,
196
+} OS_MEM_SRC;
197
+
198
+OS_SYSTEM_MEMORY os_system_memory(bool query_total_ram __maybe_unused) {
199
+ static OS_SYSTEM_MEMORY sm = {0, 0};
200
+ static usec_t last_ut = 0, last_total_ut = 0;
201
+ static OS_MEM_SRC src = OS_MEM_SRC_UNKNOWN;
202
+
203
+ usec_t now_ut = now_monotonic_usec();
204
+ if(sm.ram_total_bytes && sm.ram_available_bytes && last_ut + USEC_PER_MS > now_ut)
205
+ return sm;
206
+
207
+ last_ut = now_ut;
208
+
209
+ if(query_total_ram)
210
+ // let it auto-detect
211
+ src = OS_MEM_SRC_UNKNOWN;
212
+
213
+ if(last_total_ut + USEC_PER_SEC > now_ut)
214
+ // query also the total ram
215
+ query_total_ram = true;
216
+
217
+ switch(src) {
218
+ case OS_MEM_SRC_MEMINFO:
219
+ sm = os_system_memory_meminfo(query_total_ram);
220
+ break;
221
+
222
+ case OS_MEM_SRC_CGROUP_V2:
223
+ sm = os_system_memory_cgroup_v2(query_total_ram);
224
+ break;
225
+
226
+ case OS_MEM_SRC_CGROUP_V1:
227
+ sm = os_system_memory_cgroup_v1(query_total_ram);
228
+ break;
229
+
230
+ default:
231
+ case OS_MEM_SRC_UNKNOWN: {
232
+ OS_SYSTEM_MEMORY mi = os_system_memory_meminfo(true);
233
+ OS_SYSTEM_MEMORY v1 = os_system_memory_cgroup_v1(true);
234
+ OS_SYSTEM_MEMORY v2 = os_system_memory_cgroup_v2(true);
235
+
236
+ if(v2.ram_total_bytes && v2.ram_available_bytes && v2.ram_total_bytes <= mi.ram_total_bytes && v2.ram_available_bytes < mi.ram_available_bytes) {
237
+ sm = v2;
238
+ src = OS_MEM_SRC_CGROUP_V2;
239
+ }
240
+ else {
241
+// if(v2.ram_total_bytes || v2.ram_available_bytes)
242
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroup v2 reports more memory than meminfo. Ignoring cgroup v2.");
243
+
244
+ if (v1.ram_total_bytes && v1.ram_available_bytes && v1.ram_total_bytes <= mi.ram_total_bytes &&
245
+ v1.ram_available_bytes < mi.ram_available_bytes) {
246
+ sm = v1;
247
+ src = OS_MEM_SRC_CGROUP_V1;
248
+ }
249
+ else {
250
+// if(v1.ram_total_bytes || v1.ram_available_bytes)
251
+// nd_log(NDLS_DAEMON, NDLP_ERR, "SYSTEM_MEMORY: cgroup v1 reports more memory than meminfo. Ignoring cgroup v1.");
252
+
253
+ sm = mi;
254
+ src = OS_MEM_SRC_MEMINFO;
255
+ }
256
+ }
257
+ }
258
+ }
259
+
260
+ return sm;
261
+}
262
+#endif
263
+
264
+// FreeBSD
265
+#if defined(OS_FREEBSD)
266
+#include <sys/types.h>
267
+#include <sys/sysctl.h>
268
+
269
+OS_SYSTEM_MEMORY os_system_memory(bool query_total_ram) {
270
+ static OS_SYSTEM_MEMORY sm = {0, 0};
271
+
272
+ // Query the total RAM only if needed or if it hasn't been cached
273
+ if (query_total_ram || sm.ram_total_bytes == 0) {
274
+ uint64_t total_pages = 0;
275
+ size_t size = sizeof(total_pages);
276
+ if (sysctlbyname("vm.stats.vm.v_page_count", &total_pages, &size, NULL, 0) != 0)
277
+ goto failed;
278
+
279
+ unsigned long page_size = 0;
280
+ size = sizeof(page_size);
281
+ if (sysctlbyname("hw.pagesize", &page_size, &size, NULL, 0) != 0)
282
+ goto failed;
283
+
284
+ sm.ram_total_bytes = total_pages * page_size;
285
+ }
286
+
287
+ // Query the available RAM (free + inactive pages)
288
+ uint64_t free_pages = 0, inactive_pages = 0;
289
+ size_t size = sizeof(free_pages);
290
+ if (sysctlbyname("vm.stats.vm.v_free_count", &free_pages, &size, NULL, 0) != 0 ||
291
+ sysctlbyname("vm.stats.vm.v_inactive_count", &inactive_pages, &size, NULL, 0) != 0)
292
+ goto failed;
293
+
294
+ unsigned long page_size = 0;
295
+ size = sizeof(page_size);
296
+ if (sysctlbyname("hw.pagesize", &page_size, &size, NULL, 0) != 0)
297
+ goto failed;
298
+
299
+ sm.ram_available_bytes = (free_pages + inactive_pages) * page_size;
300
+
301
+ return sm;
302
+
303
+failed:
304
+ sm.ram_total_bytes = 0;
305
+ sm.ram_available_bytes = 0;
306
+ return sm;
307
+}
308
+#endif
src/libnetdata/os/system_memory.h
new
+26
@@ -0,0 +1,26 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_OS_MEM_AVAILABLE_H
4
+#define NETDATA_OS_MEM_AVAILABLE_H
5
+
6
+#include "libnetdata/libnetdata.h"
7
+
8
+typedef struct {
9
+ // ram_total_bytes is the total memory available in the system
10
+ // and it includes all the physical RAM the system may have.
11
+ // It does not include non-RAM memory (i.e. SWAP in not included).
12
+ // ram_total_bytes may be cached between calls to these functions
13
+ // (i.e. it may not be as up to date as ram_available_bytes).
14
+ uint64_t ram_total_bytes;
15
+
16
+ // ram_available_bytes is the total RAM memory available for
17
+ // applications, before the system and its applications run
18
+ // out-of-memory. This is always provided live by querying the system.
19
+ // It does not include non-RAM memory (i.e. SWAP in not included).
20
+ uint64_t ram_available_bytes;
21
+} OS_SYSTEM_MEMORY;
22
+
23
+// The function to get current system memory:
24
+OS_SYSTEM_MEMORY os_system_memory(bool query_total_ram);
25
+
26
+#endif //NETDATA_OS_MEM_AVAILABLE_H
src/libnetdata/parsers/entries.c
+2
-2
@@ -147,11 +147,11 @@ ssize_t entries_snprintf(char *dst, size_t dst_size, uint64_t value, const char
147
148
double converted = entries_round_to_resolution_dbl2(bytes, su->multiplier);
149
150
- uint64_t reversed_bytes = (uint64_t)(converted * (double)su->multiplier);
150
+ uint64_t reversed_bytes = (uint64_t)round((converted * (double)su->multiplier));
151
152
if(accurate) {
153
// no precision loss is required
154
- if (reversed_bytes == bytes)
154
+ if (reversed_bytes == bytes && converted > 1.0)
155
// no precision loss, this is good to use
156
su_best = su;
157
}
src/libnetdata/parsers/size.c
+2
-3
@@ -175,12 +175,11 @@ ssize_t size_snprintf(char *dst, size_t dst_size, uint64_t value, const char *un
175
continue;
176
177
double converted = size_round_to_resolution_dbl2(bytes, su->multiplier);
178
-
179
- uint64_t reversed_bytes = (uint64_t)(converted * (double)su->multiplier);
178
+ uint64_t reversed_bytes = (uint64_t)round((converted * (double)su->multiplier));
179
180
if(accurate) {
181
// no precision loss is required
183
- if (reversed_bytes == bytes)
182
+ if (reversed_bytes == bytes && converted > 1.0)
183
// no precision loss, this is good to use
184
su_best = su;
185
}
src/libnetdata/required_dummies.h
+1
-1
@@ -14,7 +14,7 @@ void netdata_cleanup_and_exit(int ret, const char *action, const char *action_re
14
}
15
16
void rrdset_thread_rda_free(void){}
17
-void sender_thread_buffer_free(void){}
17
+void sender_commit_thread_buffer_free(void){}
18
void query_target_free(void){}
19
void service_exits(void){}
20
void rrd_collector_finished(void){}
src/libnetdata/socket/connect-to.c
new
+396
@@ -0,0 +1,396 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "libnetdata/libnetdata.h"
4
+
5
+// --------------------------------------------------------------------------------------------------------------------
6
+// connect to another host/port
7
+
8
+// connect_to_this_unix()
9
+// path the path of the unix socket
10
+// timeout the timeout for establishing a connection
11
+
12
+static inline int connect_to_unix(const char *path, struct timeval *timeout) {
13
+ int fd = socket(AF_UNIX, SOCK_STREAM | DEFAULT_SOCKET_FLAGS, 0);
14
+ if(fd == -1) {
15
+ nd_log(NDLS_DAEMON, NDLP_ERR,
16
+ "Failed to create UNIX socket() for '%s'",
17
+ path);
18
+
19
+ return -1;
20
+ }
21
+
22
+ if(timeout) {
23
+ if(setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, (char *) timeout, sizeof(struct timeval)) < 0)
24
+ nd_log(NDLS_DAEMON, NDLP_ERR,
25
+ "Failed to set timeout on UNIX socket '%s'",
26
+ path);
27
+ }
28
+
29
+ sock_setcloexec(fd);
30
+
31
+ struct sockaddr_un addr;
32
+ memset(&addr, 0, sizeof(addr));
33
+ addr.sun_family = AF_UNIX;
34
+ strncpyz(addr.sun_path, path, sizeof(addr.sun_path) - 1);
35
+
36
+ if (connect(fd, (struct sockaddr*)&addr, sizeof(addr)) == -1) {
37
+ nd_log(NDLS_DAEMON, NDLP_ERR,
38
+ "Cannot connect to UNIX socket on path '%s'.",
39
+ path);
40
+
41
+ close(fd);
42
+ return -1;
43
+ }
44
+
45
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
46
+ "Connected to UNIX socket on path '%s'.",
47
+ path);
48
+
49
+ return fd;
50
+}
51
+
52
+// connect_to_this_ip46()
53
+// protocol IPPROTO_TCP, IPPROTO_UDP
54
+// socktype SOCK_STREAM, SOCK_DGRAM
55
+// host the destination hostname or IP address (IPv4 or IPv6) to connect to
56
+// if it resolves to many IPs, all are tried (IPv4 and IPv6)
57
+// scope_id the if_index id of the interface to use for connecting (0 = any)
58
+// (used only under IPv6)
59
+// service the service name or port to connect to
60
+// timeout the timeout for establishing a connection
61
+
62
+int connect_to_this_ip46(
63
+ int protocol,
64
+ int socktype,
65
+ const char *host,
66
+ uint32_t scope_id,
67
+ const char *service,
68
+ struct timeval *timeout,
69
+ bool *fallback_ipv4)
70
+{
71
+ struct addrinfo hints;
72
+ struct addrinfo *ai_head = NULL, *ai = NULL;
73
+
74
+ memset(&hints, 0, sizeof(hints));
75
+ hints.ai_family = PF_UNSPEC; /* Allow IPv4 or IPv6 */
76
+ hints.ai_socktype = socktype;
77
+ hints.ai_protocol = protocol;
78
+
79
+ int ai_err = getaddrinfo(host, service, &hints, &ai_head);
80
+ if (ai_err != 0) {
81
+
82
+ nd_log(NDLS_DAEMON, NDLP_ERR,
83
+ "Cannot resolve host '%s', port '%s': %s",
84
+ host, service, gai_strerror(ai_err));
85
+
86
+ return -ND_SOCK_ERR_CANNOT_RESOLVE_HOSTNAME;
87
+ }
88
+
89
+ char hostBfr[NI_MAXHOST + 1];
90
+ char servBfr[NI_MAXSERV + 1];
91
+
92
+ ND_LOG_STACK lgs[] = {
93
+ ND_LOG_FIELD_TXT(NDF_DST_IP, hostBfr),
94
+ ND_LOG_FIELD_TXT(NDF_DST_PORT, servBfr),
95
+ ND_LOG_FIELD_END(),
96
+ };
97
+ ND_LOG_STACK_PUSH(lgs);
98
+
99
+ int fd = -1;
100
+ for (ai = ai_head; ai != NULL && fd == -1; ai = ai->ai_next) {
101
+ if(nd_thread_signaled_to_cancel()) break;
102
+
103
+ if (fallback_ipv4 && *fallback_ipv4 && ai->ai_family == PF_INET6)
104
+ continue;
105
+
106
+ if (ai->ai_family == PF_INET6) {
107
+ struct sockaddr_in6 *pSadrIn6 = (struct sockaddr_in6 *) ai->ai_addr;
108
+ if(pSadrIn6->sin6_scope_id == 0) {
109
+ pSadrIn6->sin6_scope_id = scope_id;
110
+ }
111
+ }
112
+
113
+ getnameinfo(ai->ai_addr,
114
+ ai->ai_addrlen,
115
+ hostBfr,
116
+ sizeof(hostBfr),
117
+ servBfr,
118
+ sizeof(servBfr),
119
+ NI_NUMERICHOST | NI_NUMERICSERV);
120
+
121
+ switch (ai->ai_addr->sa_family) {
122
+ case PF_INET: {
123
+ struct sockaddr_in *pSadrIn = (struct sockaddr_in *)ai->ai_addr;
124
+ (void)pSadrIn;
125
+ break;
126
+ }
127
+
128
+ case PF_INET6: {
129
+ struct sockaddr_in6 *pSadrIn6 = (struct sockaddr_in6 *) ai->ai_addr;
130
+ (void)pSadrIn6;
131
+ break;
132
+ }
133
+
134
+ default: {
135
+ // Unknown protocol family
136
+ continue;
137
+ }
138
+ }
139
+
140
+ fd = socket(ai->ai_family, ai->ai_socktype | DEFAULT_SOCKET_FLAGS, ai->ai_protocol);
141
+ if(fd != -1) {
142
+ if(timeout) {
143
+ if(setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, (char *) timeout, sizeof(struct timeval)) < 0)
144
+ nd_log(NDLS_DAEMON, NDLP_ERR,
145
+ "Failed to set timeout on the socket to ip '%s' port '%s'",
146
+ hostBfr, servBfr);
147
+ }
148
+ sock_setcloexec(fd);
149
+
150
+ errno_clear();
151
+ if(connect(fd, ai->ai_addr, ai->ai_addrlen) < 0) {
152
+ if(errno == EALREADY || errno == EINPROGRESS) {
153
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
154
+ "Waiting for connection to ip %s port %s to be established",
155
+ hostBfr, servBfr);
156
+
157
+ // Convert 'struct timeval' to milliseconds for poll():
158
+ int timeout_ms = timeout ? (timeout->tv_sec * 1000 + timeout->tv_usec / 1000) : 1000;
159
+
160
+ switch(wait_on_socket_or_cancel_with_timeout(NULL, fd, timeout_ms, POLLOUT, NULL)) {
161
+ case 0: // proceed
162
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
163
+ "connect() to ip %s port %s completed successfully",
164
+ hostBfr, servBfr);
165
+ break;
166
+
167
+ case -1: // thread cancelled
168
+ nd_log(NDLS_DAEMON, NDLP_ERR,
169
+ "Thread is cancelled while connecting to '%s', port '%s'.",
170
+ hostBfr, servBfr);
171
+
172
+ close(fd);
173
+ fd = -ND_SOCK_ERR_THREAD_CANCELLED;
174
+ break;
175
+
176
+ case 1: // timeout
177
+ nd_log(NDLS_DAEMON, NDLP_ERR,
178
+ "Timed out while connecting to '%s', port '%s'.",
179
+ hostBfr, servBfr);
180
+
181
+ close(fd);
182
+ fd = -ND_SOCK_ERR_TIMEOUT;
183
+
184
+ if (fallback_ipv4 && ai->ai_family == PF_INET6)
185
+ *fallback_ipv4 = true;
186
+ break;
187
+
188
+ default:
189
+ case 2: // poll error
190
+ nd_log(NDLS_DAEMON, NDLP_ERR,
191
+ "Failed to connect to '%s', port '%s'.",
192
+ hostBfr, servBfr);
193
+
194
+ close(fd);
195
+ fd = -ND_SOCK_ERR_POLL_ERROR;
196
+ break;
197
+ }
198
+ }
199
+ else {
200
+ nd_log(NDLS_DAEMON, NDLP_ERR,
201
+ "Failed to connect to '%s', port '%s'",
202
+ hostBfr, servBfr);
203
+
204
+ close(fd);
205
+ fd = -ND_SOCK_ERR_CONNECTION_REFUSED;
206
+ }
207
+ }
208
+ }
209
+ else {
210
+ nd_log(NDLS_DAEMON, NDLP_ERR, "Failed to socket() to '%s', port '%s'", hostBfr, servBfr);
211
+ fd = -ND_SOCK_ERR_FAILED_TO_CREATE_SOCKET;
212
+ }
213
+ }
214
+
215
+ freeaddrinfo(ai_head);
216
+
217
+ return fd;
218
+}
219
+
220
+// connect_to_this()
221
+//
222
+// definition format:
223
+//
224
+// [PROTOCOL:]IP[%INTERFACE][:PORT]
225
+//
226
+// PROTOCOL = tcp or udp
227
+// IP = IPv4 or IPv6 IP or hostname, optionally enclosed in [] (required for IPv6)
228
+// INTERFACE = for IPv6 only, the network interface to use
229
+// PORT = port number or service name
230
+
231
+int connect_to_this(const char *definition, int default_port, struct timeval *timeout) {
232
+ char buffer[strlen(definition) + 1];
233
+ strcpy(buffer, definition);
234
+
235
+ char default_service[10 + 1];
236
+ snprintfz(default_service, 10, "%d", default_port);
237
+
238
+ char *host = buffer, *service = default_service, *iface = "";
239
+ int protocol = IPPROTO_TCP, socktype = SOCK_STREAM;
240
+ uint32_t scope_id = 0;
241
+
242
+ if(strncmp(host, "tcp:", 4) == 0) {
243
+ host += 4;
244
+ protocol = IPPROTO_TCP;
245
+ socktype = SOCK_STREAM;
246
+ }
247
+ else if(strncmp(host, "udp:", 4) == 0) {
248
+ host += 4;
249
+ protocol = IPPROTO_UDP;
250
+ socktype = SOCK_DGRAM;
251
+ }
252
+ else if(strncmp(host, "unix:", 5) == 0) {
253
+ char *path = host + 5;
254
+ return connect_to_unix(path, timeout);
255
+ }
256
+ else if(*host == '/') {
257
+ char *path = host;
258
+ return connect_to_unix(path, timeout);
259
+ }
260
+
261
+ char *e = host;
262
+ if(*e == '[') {
263
+ e = ++host;
264
+ while(*e && *e != ']') e++;
265
+ if(*e == ']') {
266
+ *e = '\0';
267
+ e++;
268
+ }
269
+ }
270
+ else {
271
+ while(*e && *e != ':' && *e != '%') e++;
272
+ }
273
+
274
+ if(*e == '%') {
275
+ *e = '\0';
276
+ e++;
277
+ iface = e;
278
+ while(*e && *e != ':') e++;
279
+ }
280
+
281
+ if(*e == ':') {
282
+ *e = '\0';
283
+ e++;
284
+ service = e;
285
+ }
286
+
287
+ if(!*host) {
288
+ nd_log(NDLS_DAEMON, NDLP_ERR,
289
+ "Definition '%s' does not specify a host.",
290
+ definition);
291
+
292
+ return -ND_SOCK_ERR_NO_HOST_IN_DEFINITION;
293
+ }
294
+
295
+ if(*iface) {
296
+ scope_id = if_nametoindex(iface);
297
+ if(!scope_id)
298
+ nd_log(NDLS_DAEMON, NDLP_ERR,
299
+ "Cannot find a network interface named '%s'. Continuing with limiting the network interface",
300
+ iface);
301
+ }
302
+
303
+ if(!*service)
304
+ service = default_service;
305
+
306
+
307
+ return connect_to_this_ip46(protocol, socktype, host, scope_id, service, timeout, NULL);
308
+}
309
+
310
+void foreach_entry_in_connection_string(const char *destination, bool (*callback)(char *entry, void *data), void *data) {
311
+ const char *s = destination;
312
+ while(*s) {
313
+ const char *e = s;
314
+
315
+ // skip separators, moving both s(tart) and e(nd)
316
+ while(isspace((uint8_t)*e) || *e == ',') s = ++e;
317
+
318
+ // move e(nd) to the first separator
319
+ while(*e && !isspace((uint8_t)*e) && *e != ',') e++;
320
+
321
+ // is there anything?
322
+ if(!*s || s == e) break;
323
+
324
+ char buf[e - s + 1];
325
+ strncpyz(buf, s, e - s);
326
+
327
+ if(callback(buf, data)) break;
328
+
329
+ s = e;
330
+ }
331
+}
332
+
333
+struct connect_to_one_of_data {
334
+ int default_port;
335
+ struct timeval *timeout;
336
+ size_t *reconnects_counter;
337
+ char *connected_to;
338
+ size_t connected_to_size;
339
+ int sock;
340
+};
341
+
342
+static bool connect_to_one_of_callback(char *entry, void *data) {
343
+ struct connect_to_one_of_data *t = data;
344
+
345
+ if(t->reconnects_counter)
346
+ t->reconnects_counter++;
347
+
348
+ t->sock = connect_to_this(entry, t->default_port, t->timeout);
349
+ if(t->sock != -1) {
350
+ if(t->connected_to && t->connected_to_size) {
351
+ strncpyz(t->connected_to, entry, t->connected_to_size);
352
+ t->connected_to[t->connected_to_size - 1] = '\0';
353
+ }
354
+
355
+ return true;
356
+ }
357
+
358
+ return false;
359
+}
360
+
361
+int connect_to_one_of(const char *destination, int default_port, struct timeval *timeout, size_t *reconnects_counter, char *connected_to, size_t connected_to_size) {
362
+ struct connect_to_one_of_data t = {
363
+ .default_port = default_port,
364
+ .timeout = timeout,
365
+ .reconnects_counter = reconnects_counter,
366
+ .connected_to = connected_to,
367
+ .connected_to_size = connected_to_size,
368
+ .sock = -1,
369
+ };
370
+
371
+ foreach_entry_in_connection_string(destination, connect_to_one_of_callback, &t);
372
+
373
+ return t.sock;
374
+}
375
+
376
+static bool connect_to_one_of_urls_callback(char *entry, void *data) {
377
+ char *s = strchr(entry, '/');
378
+ if(s) *s = '\0';
379
+
380
+ return connect_to_one_of_callback(entry, data);
381
+}
382
+
383
+int connect_to_one_of_urls(const char *destination, int default_port, struct timeval *timeout, size_t *reconnects_counter, char *connected_to, size_t connected_to_size) {
384
+ struct connect_to_one_of_data t = {
385
+ .default_port = default_port,
386
+ .timeout = timeout,
387
+ .reconnects_counter = reconnects_counter,
388
+ .connected_to = connected_to,
389
+ .connected_to_size = connected_to_size,
390
+ .sock = -1,
391
+ };
392
+
393
+ foreach_entry_in_connection_string(destination, connect_to_one_of_urls_callback, &t);
394
+
395
+ return t.sock;
396
+}
src/libnetdata/socket/connect-to.h
new
+19
@@ -0,0 +1,19 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_CONNECT_TO_H
4
+#define NETDATA_CONNECT_TO_H
5
+
6
+void foreach_entry_in_connection_string(const char *destination, bool (*callback)(char *entry, void *data), void *data);
7
+int connect_to_this_ip46(
8
+ int protocol,
9
+ int socktype,
10
+ const char *host,
11
+ uint32_t scope_id,
12
+ const char *service,
13
+ struct timeval *timeout,
14
+ bool *fallback_ipv4);
15
+int connect_to_this(const char *definition, int default_port, struct timeval *timeout);
16
+int connect_to_one_of(const char *destination, int default_port, struct timeval *timeout, size_t *reconnects_counter, char *connected_to, size_t connected_to_size);
17
+int connect_to_one_of_urls(const char *destination, int default_port, struct timeval *timeout, size_t *reconnects_counter, char *connected_to, size_t connected_to_size);
18
+
19
+#endif //NETDATA_CONNECT_TO_H
src/libnetdata/socket/listen-sockets.c
new
+530
@@ -0,0 +1,530 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "libnetdata/libnetdata.h"
4
+#include "listen-sockets.h"
5
+
6
+static HTTP_ACL socket_ssl_acl(char *acl) {
7
+ char *ssl = strchr(acl,'^');
8
+ if(ssl) {
9
+ //Due the format of the SSL command it is always the last command,
10
+ //we finish it here to avoid problems with the ACLs
11
+ *ssl = '\0';
12
+ ssl++;
13
+ if (!strncmp("SSL=",ssl,4)) {
14
+ ssl += 4;
15
+ if (!strcmp(ssl,"optional")) {
16
+ return HTTP_ACL_SSL_OPTIONAL;
17
+ }
18
+ else if (!strcmp(ssl,"force")) {
19
+ return HTTP_ACL_SSL_FORCE;
20
+ }
21
+ }
22
+ }
23
+
24
+ return HTTP_ACL_NONE;
25
+}
26
+
27
+static HTTP_ACL read_acl(char *st) {
28
+ HTTP_ACL ret = socket_ssl_acl(st);
29
+
30
+ if (!strcmp(st,"dashboard")) ret |= HTTP_ACL_DASHBOARD;
31
+ if (!strcmp(st,"registry")) ret |= HTTP_ACL_REGISTRY;
32
+ if (!strcmp(st,"badges")) ret |= HTTP_ACL_BADGES;
33
+ if (!strcmp(st,"management")) ret |= HTTP_ACL_MANAGEMENT;
34
+ if (!strcmp(st,"streaming")) ret |= HTTP_ACL_STREAMING;
35
+ if (!strcmp(st,"netdata.conf")) ret |= HTTP_ACL_NETDATACONF;
36
+
37
+ return ret;
38
+}
39
+
40
+static char *strdup_client_description(int family, const char *protocol, const char *ip, uint16_t port) {
41
+ char buffer[100 + 1];
42
+
43
+ switch(family) {
44
+ case AF_INET:
45
+ snprintfz(buffer, sizeof(buffer) - 1, "%s:%s:%d", protocol, ip, port);
46
+ break;
47
+
48
+ case AF_INET6:
49
+ default:
50
+ snprintfz(buffer, sizeof(buffer) - 1, "%s:[%s]:%d", protocol, ip, port);
51
+ break;
52
+
53
+ case AF_UNIX:
54
+ snprintfz(buffer, sizeof(buffer) - 1, "%s:%s", protocol, ip);
55
+ break;
56
+ }
57
+
58
+ return strdupz(buffer);
59
+}
60
+
61
+static int create_listen_socket_unix(const char *path, int listen_backlog) {
62
+ int sock;
63
+
64
+ sock = socket(AF_UNIX, SOCK_STREAM | DEFAULT_SOCKET_FLAGS, 0);
65
+ if(sock < 0) {
66
+ nd_log(NDLS_DAEMON, NDLP_ERR,
67
+ "LISTENER: UNIX socket() on path '%s' failed.",
68
+ path);
69
+
70
+ return -1;
71
+ }
72
+
73
+ sock_setnonblock(sock);
74
+ sock_setcloexec(sock);
75
+ sock_enlarge_in(sock);
76
+
77
+ struct sockaddr_un name;
78
+ memset(&name, 0, sizeof(struct sockaddr_un));
79
+ name.sun_family = AF_UNIX;
80
+ strncpyz(name.sun_path, path, sizeof(name.sun_path) - 1);
81
+
82
+ errno_clear();
83
+ if (unlink(path) == -1 && errno != ENOENT)
84
+ nd_log(NDLS_DAEMON, NDLP_ERR,
85
+ "LISTENER: failed to remove existing (probably obsolete or left-over) file on UNIX socket path '%s'.",
86
+ path);
87
+
88
+ if(bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
89
+ close(sock);
90
+ nd_log(NDLS_DAEMON, NDLP_ERR,
91
+ "LISTENER: UNIX bind() on path '%s' failed.",
92
+ path);
93
+
94
+ return -1;
95
+ }
96
+
97
+ // we have to chmod this to 0777 so that the client will be able
98
+ // to read from and write to this socket.
99
+ if(chmod(path, 0777) == -1)
100
+ nd_log(NDLS_DAEMON, NDLP_ERR,
101
+ "LISTENER: failed to chmod() socket file '%s'.",
102
+ path);
103
+
104
+ if(listen(sock, listen_backlog) < 0) {
105
+ close(sock);
106
+ nd_log(NDLS_DAEMON, NDLP_ERR,
107
+ "LISTENER: UNIX listen() on path '%s' failed.",
108
+ path);
109
+
110
+ return -1;
111
+ }
112
+
113
+ return sock;
114
+}
115
+
116
+static int create_listen_socket4(int socktype, const char *ip, uint16_t port, int listen_backlog) {
117
+ int sock;
118
+
119
+ sock = socket(AF_INET, socktype | DEFAULT_SOCKET_FLAGS, 0);
120
+ if(sock < 0) {
121
+ nd_log(NDLS_DAEMON, NDLP_ERR,
122
+ "LISTENER: IPv4 socket() on ip '%s' port %d, socktype %d failed.",
123
+ ip, port, socktype);
124
+
125
+ return -1;
126
+ }
127
+ sock_setreuse(sock, 1);
128
+ sock_setreuse_port(sock, 0);
129
+ sock_setnonblock(sock);
130
+ sock_setcloexec(sock);
131
+ sock_enlarge_in(sock);
132
+
133
+ struct sockaddr_in name;
134
+ memset(&name, 0, sizeof(struct sockaddr_in));
135
+ name.sin_family = AF_INET;
136
+ name.sin_port = htons (port);
137
+
138
+ int ret = inet_pton(AF_INET, ip, (void *)&name.sin_addr.s_addr);
139
+ if(ret != 1) {
140
+ nd_log(NDLS_DAEMON, NDLP_ERR,
141
+ "LISTENER: Failed to convert IP '%s' to a valid IPv4 address.",
142
+ ip);
143
+
144
+ close(sock);
145
+ return -1;
146
+ }
147
+
148
+ if(bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
149
+ close(sock);
150
+ nd_log(NDLS_DAEMON, NDLP_ERR,
151
+ "LISTENER: IPv4 bind() on ip '%s' port %d, socktype %d failed.",
152
+ ip, port, socktype);
153
+
154
+ return -1;
155
+ }
156
+
157
+ if(socktype == SOCK_STREAM && listen(sock, listen_backlog) < 0) {
158
+ close(sock);
159
+ nd_log(NDLS_DAEMON, NDLP_ERR,
160
+ "LISTENER: IPv4 listen() on ip '%s' port %d, socktype %d failed.",
161
+ ip, port, socktype);
162
+
163
+ return -1;
164
+ }
165
+
166
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
167
+ "LISTENER: Listening on IPv4 ip '%s' port %d, socktype %d",
168
+ ip, port, socktype);
169
+
170
+ return sock;
171
+}
172
+
173
+static int create_listen_socket6(int socktype, uint32_t scope_id, const char *ip, int port, int listen_backlog) {
174
+ int sock;
175
+ int ipv6only = 1;
176
+
177
+ sock = socket(AF_INET6, socktype | DEFAULT_SOCKET_FLAGS, 0);
178
+ if (sock < 0) {
179
+ nd_log(NDLS_DAEMON, NDLP_ERR,
180
+ "LISTENER: IPv6 socket() on ip '%s' port %d, socktype %d, failed.",
181
+ ip, port, socktype);
182
+
183
+ return -1;
184
+ }
185
+ sock_setreuse(sock, 1);
186
+ sock_setreuse_port(sock, 0);
187
+ sock_setnonblock(sock);
188
+ sock_setcloexec(sock);
189
+ sock_enlarge_in(sock);
190
+
191
+ /* IPv6 only */
192
+ if(setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (void*)&ipv6only, sizeof(ipv6only)) != 0)
193
+ nd_log(NDLS_DAEMON, NDLP_ERR,
194
+ "LISTENER: Cannot set IPV6_V6ONLY on ip '%s' port %d, socktype %d.",
195
+ ip, port, socktype);
196
+
197
+ struct sockaddr_in6 name;
198
+ memset(&name, 0, sizeof(struct sockaddr_in6));
199
+ name.sin6_family = AF_INET6;
200
+ name.sin6_port = htons ((uint16_t) port);
201
+ name.sin6_scope_id = scope_id;
202
+
203
+ int ret = inet_pton(AF_INET6, ip, (void *)&name.sin6_addr.s6_addr);
204
+ if(ret != 1) {
205
+ nd_log(NDLS_DAEMON, NDLP_ERR,
206
+ "LISTENER: Failed to convert IP '%s' to a valid IPv6 address.",
207
+ ip);
208
+
209
+ close(sock);
210
+ return -1;
211
+ }
212
+
213
+ name.sin6_scope_id = scope_id;
214
+
215
+ if (bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
216
+ close(sock);
217
+ nd_log(NDLS_DAEMON, NDLP_ERR,
218
+ "LISTENER: IPv6 bind() on ip '%s' port %d, socktype %d failed.",
219
+ ip, port, socktype);
220
+
221
+ return -1;
222
+ }
223
+
224
+ if (socktype == SOCK_STREAM && listen(sock, listen_backlog) < 0) {
225
+ close(sock);
226
+ nd_log(NDLS_DAEMON, NDLP_ERR,
227
+ "LISTENER: IPv6 listen() on ip '%s' port %d, socktype %d failed.",
228
+ ip, port, socktype);
229
+
230
+ return -1;
231
+ }
232
+
233
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
234
+ "LISTENER: Listening on IPv6 ip '%s' port %d, socktype %d",
235
+ ip, port, socktype);
236
+
237
+ return sock;
238
+}
239
+
240
+static inline int listen_sockets_add(LISTEN_SOCKETS *sockets, int fd, int family, int socktype, const char *protocol, const char *ip, uint16_t port, int acl_flags) {
241
+ if(sockets->opened >= MAX_LISTEN_FDS) {
242
+ nd_log(NDLS_DAEMON, NDLP_ERR,
243
+ "LISTENER: Too many listening sockets. Failed to add listening %s socket at ip '%s' port %d, protocol %s, socktype %d",
244
+ protocol, ip, port, protocol, socktype);
245
+
246
+ close(fd);
247
+ return -1;
248
+ }
249
+
250
+ sockets->fds[sockets->opened] = fd;
251
+ sockets->fds_types[sockets->opened] = socktype;
252
+ sockets->fds_families[sockets->opened] = family;
253
+ sockets->fds_names[sockets->opened] = strdup_client_description(family, protocol, ip, port);
254
+ sockets->fds_acl_flags[sockets->opened] = acl_flags;
255
+
256
+ sockets->opened++;
257
+ return 0;
258
+}
259
+
260
+static inline int listen_sockets_check_is_member(LISTEN_SOCKETS *sockets, int fd) {
261
+ size_t i;
262
+ for(i = 0; i < sockets->opened ;i++)
263
+ if(sockets->fds[i] == fd) return 1;
264
+
265
+ return 0;
266
+}
267
+
268
+static inline void listen_sockets_init(LISTEN_SOCKETS *sockets) {
269
+ size_t i;
270
+ for(i = 0; i < MAX_LISTEN_FDS ;i++) {
271
+ sockets->fds[i] = -1;
272
+ sockets->fds_names[i] = NULL;
273
+ sockets->fds_types[i] = -1;
274
+ }
275
+
276
+ sockets->opened = 0;
277
+ sockets->failed = 0;
278
+}
279
+
280
+void listen_sockets_close(LISTEN_SOCKETS *sockets) {
281
+ size_t i;
282
+ for(i = 0; i < sockets->opened ;i++) {
283
+ close(sockets->fds[i]);
284
+ sockets->fds[i] = -1;
285
+
286
+ freez(sockets->fds_names[i]);
287
+ sockets->fds_names[i] = NULL;
288
+
289
+ sockets->fds_types[i] = -1;
290
+ }
291
+
292
+ sockets->opened = 0;
293
+ sockets->failed = 0;
294
+}
295
+
296
+static inline int bind_to_this(LISTEN_SOCKETS *sockets, const char *definition, uint16_t default_port, int listen_backlog) {
297
+ int added = 0;
298
+ HTTP_ACL acl_flags = HTTP_ACL_NONE;
299
+
300
+ struct addrinfo hints;
301
+ struct addrinfo *result = NULL, *rp = NULL;
302
+
303
+ char buffer[strlen(definition) + 1];
304
+ strcpy(buffer, definition);
305
+
306
+ char buffer2[10 + 1];
307
+ snprintfz(buffer2, 10, "%d", default_port);
308
+
309
+ char *ip = buffer, *port = buffer2, *iface = "", *portconfig;
310
+
311
+ int protocol = IPPROTO_TCP, socktype = SOCK_STREAM;
312
+ const char *protocol_str = "tcp";
313
+
314
+ if(strncmp(ip, "tcp:", 4) == 0) {
315
+ ip += 4;
316
+ protocol = IPPROTO_TCP;
317
+ socktype = SOCK_STREAM;
318
+ protocol_str = "tcp";
319
+ acl_flags |= HTTP_ACL_API;
320
+ }
321
+ else if(strncmp(ip, "udp:", 4) == 0) {
322
+ ip += 4;
323
+ protocol = IPPROTO_UDP;
324
+ socktype = SOCK_DGRAM;
325
+ protocol_str = "udp";
326
+ acl_flags |= HTTP_ACL_API_UDP;
327
+ }
328
+ else if(strncmp(ip, "unix:", 5) == 0) {
329
+ char *path = ip + 5;
330
+ socktype = SOCK_STREAM;
331
+ protocol_str = "unix";
332
+ int fd = create_listen_socket_unix(path, listen_backlog);
333
+ if (fd == -1) {
334
+ nd_log(NDLS_DAEMON, NDLP_ERR,
335
+ "LISTENER: Cannot create unix socket '%s'",
336
+ path);
337
+
338
+ sockets->failed++;
339
+ } else {
340
+ acl_flags = HTTP_ACL_API_UNIX | HTTP_ACL_DASHBOARD | HTTP_ACL_REGISTRY | HTTP_ACL_BADGES |
341
+ HTTP_ACL_MANAGEMENT | HTTP_ACL_NETDATACONF | HTTP_ACL_STREAMING | HTTP_ACL_SSL_DEFAULT;
342
+ listen_sockets_add(sockets, fd, AF_UNIX, socktype, protocol_str, path, 0, acl_flags);
343
+ added++;
344
+ }
345
+ return added;
346
+ }
347
+
348
+ char *e = ip;
349
+ if(*e == '[') {
350
+ e = ++ip;
351
+ while(*e && *e != ']') e++;
352
+ if(*e == ']') {
353
+ *e = '\0';
354
+ e++;
355
+ }
356
+ }
357
+ else {
358
+ while(*e && *e != ':' && *e != '%' && *e != '=') e++;
359
+ }
360
+
361
+ if(*e == '%') {
362
+ *e = '\0';
363
+ e++;
364
+ iface = e;
365
+ while(*e && *e != ':' && *e != '=') e++;
366
+ }
367
+
368
+ if(*e == ':') {
369
+ port = e + 1;
370
+ *e = '\0';
371
+ e++;
372
+ while(*e && *e != '=') e++;
373
+ }
374
+
375
+ if(*e == '=') {
376
+ *e='\0';
377
+ e++;
378
+ portconfig = e;
379
+ while (*e != '\0') {
380
+ if (*e == '|') {
381
+ *e = '\0';
382
+ acl_flags |= read_acl(portconfig);
383
+ e++;
384
+ portconfig = e;
385
+ continue;
386
+ }
387
+ e++;
388
+ }
389
+ acl_flags |= read_acl(portconfig);
390
+ } else {
391
+ acl_flags |= HTTP_ACL_DASHBOARD | HTTP_ACL_REGISTRY | HTTP_ACL_BADGES | HTTP_ACL_MANAGEMENT | HTTP_ACL_NETDATACONF | HTTP_ACL_STREAMING | HTTP_ACL_SSL_DEFAULT;
392
+ }
393
+
394
+ //Case the user does not set the option SSL in the "bind to", but he has
395
+ //the certificates, I must redirect, so I am assuming here the default option
396
+ if(!(acl_flags & HTTP_ACL_SSL_OPTIONAL) && !(acl_flags & HTTP_ACL_SSL_FORCE)) {
397
+ acl_flags |= HTTP_ACL_SSL_DEFAULT;
398
+ }
399
+
400
+ uint32_t scope_id = 0;
401
+ if(*iface) {
402
+ scope_id = if_nametoindex(iface);
403
+ if(!scope_id)
404
+ nd_log(NDLS_DAEMON, NDLP_ERR,
405
+ "LISTENER: Cannot find a network interface named '%s'. "
406
+ "Continuing with limiting the network interface",
407
+ iface);
408
+ }
409
+
410
+ if(!*ip || *ip == '*' || !strcmp(ip, "any") || !strcmp(ip, "all"))
411
+ ip = NULL;
412
+
413
+ if(!*port)
414
+ port = buffer2;
415
+
416
+ memset(&hints, 0, sizeof(hints));
417
+ hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */
418
+ hints.ai_socktype = socktype;
419
+ hints.ai_flags = AI_PASSIVE; /* For wildcard IP address */
420
+ hints.ai_protocol = protocol;
421
+ hints.ai_canonname = NULL;
422
+ hints.ai_addr = NULL;
423
+ hints.ai_next = NULL;
424
+
425
+ int r = getaddrinfo(ip, port, &hints, &result);
426
+ if (r != 0) {
427
+ nd_log(NDLS_DAEMON, NDLP_ERR,
428
+ "LISTENER: getaddrinfo('%s', '%s'): %s\n",
429
+ ip, port, gai_strerror(r));
430
+
431
+ return -1;
432
+ }
433
+
434
+ for (rp = result; rp != NULL; rp = rp->ai_next) {
435
+ int fd = -1;
436
+ int family;
437
+
438
+ char rip[INET_ADDRSTRLEN + INET6_ADDRSTRLEN] = "INVALID";
439
+ uint16_t rport = default_port;
440
+
441
+ family = rp->ai_addr->sa_family;
442
+ switch (family) {
443
+ case AF_INET: {
444
+ struct sockaddr_in *sin = (struct sockaddr_in *) rp->ai_addr;
445
+ inet_ntop(AF_INET, &sin->sin_addr, rip, INET_ADDRSTRLEN);
446
+ rport = ntohs(sin->sin_port);
447
+ fd = create_listen_socket4(socktype, rip, rport, listen_backlog);
448
+ break;
449
+ }
450
+
451
+ case AF_INET6: {
452
+ struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *) rp->ai_addr;
453
+ inet_ntop(AF_INET6, &sin6->sin6_addr, rip, INET6_ADDRSTRLEN);
454
+ rport = ntohs(sin6->sin6_port);
455
+ fd = create_listen_socket6(socktype, scope_id, rip, rport, listen_backlog);
456
+ break;
457
+ }
458
+
459
+ default:
460
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
461
+ "LISTENER: Unknown socket family %d",
462
+ family);
463
+
464
+ break;
465
+ }
466
+
467
+ if (fd == -1) {
468
+ nd_log(NDLS_DAEMON, NDLP_ERR,
469
+ "LISTENER: Cannot bind to ip '%s', port %d",
470
+ rip, rport);
471
+
472
+ sockets->failed++;
473
+ }
474
+ else {
475
+ listen_sockets_add(sockets, fd, family, socktype, protocol_str, rip, rport, acl_flags);
476
+ added++;
477
+ }
478
+ }
479
+
480
+ freeaddrinfo(result);
481
+
482
+ return added;
483
+}
484
+
485
+int listen_sockets_setup(LISTEN_SOCKETS *sockets) {
486
+ listen_sockets_init(sockets);
487
+
488
+ sockets->backlog = (int) appconfig_get_number(sockets->config, sockets->config_section, "listen backlog", sockets->backlog);
489
+
490
+ long long int old_port = sockets->default_port;
491
+ long long int new_port = appconfig_get_number(sockets->config, sockets->config_section, "default port", sockets->default_port);
492
+ if(new_port < 1 || new_port > 65535) {
493
+ nd_log(NDLS_DAEMON, NDLP_ERR,
494
+ "LISTENER: Invalid listen port %lld given. Defaulting to %lld.",
495
+ new_port, old_port);
496
+
497
+ sockets->default_port = (uint16_t) appconfig_set_number(sockets->config, sockets->config_section, "default port", old_port);
498
+ }
499
+ else sockets->default_port = (uint16_t)new_port;
500
+
501
+ const char *s = appconfig_get(sockets->config, sockets->config_section, "bind to", sockets->default_bind_to);
502
+ while(*s) {
503
+ const char *e = s;
504
+
505
+ // skip separators, moving both s(tart) and e(nd)
506
+ while(isspace((uint8_t)*e) || *e == ',') s = ++e;
507
+
508
+ // move e(nd) to the first separator
509
+ while(*e && !isspace((uint8_t)*e) && *e != ',') e++;
510
+
511
+ // is there anything?
512
+ if(!*s || s == e) break;
513
+
514
+ char buf[e - s + 1];
515
+ strncpyz(buf, s, e - s);
516
+ bind_to_this(sockets, buf, sockets->default_port, sockets->backlog);
517
+
518
+ s = e;
519
+ }
520
+
521
+ if(sockets->failed) {
522
+ size_t i;
523
+ for(i = 0; i < sockets->opened ;i++)
524
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
525
+ "LISTENER: Listen socket %s opened successfully.",
526
+ sockets->fds_names[i]);
527
+ }
528
+
529
+ return (int)sockets->opened;
530
+}
src/libnetdata/socket/listen-sockets.h
new
+31
@@ -0,0 +1,31 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_LISTEN_SOCKETS_H
4
+#define NETDATA_LISTEN_SOCKETS_H
5
+
6
+#include "libnetdata/common.h"
7
+
8
+#ifndef MAX_LISTEN_FDS
9
+#define MAX_LISTEN_FDS 50
10
+#endif
11
+
12
+typedef struct listen_sockets {
13
+ struct config *config; // the config file to use
14
+ const char *config_section; // the netdata configuration section to read settings from
15
+ const char *default_bind_to; // the default bind to configuration string
16
+ uint16_t default_port; // the default port to use
17
+ int backlog; // the default listen backlog to use
18
+
19
+ size_t opened; // the number of sockets opened
20
+ size_t failed; // the number of sockets attempted to open, but failed
21
+ int fds[MAX_LISTEN_FDS]; // the open sockets
22
+ char *fds_names[MAX_LISTEN_FDS]; // descriptions for the open sockets
23
+ int fds_types[MAX_LISTEN_FDS]; // the socktype for the open sockets (SOCK_STREAM, SOCK_DGRAM)
24
+ int fds_families[MAX_LISTEN_FDS]; // the family of the open sockets (AF_UNIX, AF_INET, AF_INET6)
25
+ HTTP_ACL fds_acl_flags[MAX_LISTEN_FDS]; // the acl to apply to the open sockets (dashboard, badges, streaming, netdata.conf, management)
26
+} LISTEN_SOCKETS;
27
+
28
+int listen_sockets_setup(LISTEN_SOCKETS *sockets);
29
+void listen_sockets_close(LISTEN_SOCKETS *sockets);
30
+
31
+#endif //NETDATA_LISTEN_SOCKETS_H
src/libnetdata/socket/nd-poll.c
new
+291
@@ -0,0 +1,291 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "nd-poll.h"
4
+
5
+#if defined(OS_LINUX)
6
+#include <sys/epoll.h>
7
+
8
+#define MAX_EVENTS_PER_CALL 100
9
+
10
+// Event poll context
11
+struct nd_poll_t {
12
+ int epoll_fd;
13
+ struct epoll_event ev[MAX_EVENTS_PER_CALL];
14
+ size_t last_pos;
15
+ size_t used;
16
+};
17
+
18
+// Initialize the event poll context
19
+nd_poll_t *nd_poll_create() {
20
+ nd_poll_t *ndpl = callocz(1, sizeof(nd_poll_t));
21
+
22
+ ndpl->epoll_fd = epoll_create1(0);
23
+ if (ndpl->epoll_fd < 0) {
24
+ freez(ndpl);
25
+ return NULL;
26
+ }
27
+
28
+ return ndpl;
29
+}
30
+
31
+// Add a file descriptor to the event poll
32
+bool nd_poll_add(nd_poll_t *ndpl, int fd, nd_poll_event_t events, void *data) {
33
+ struct epoll_event ev = {
34
+ .events = (events & ND_POLL_READ ? EPOLLIN : 0) | (events & ND_POLL_WRITE ? EPOLLOUT : 0),
35
+ .data.ptr = data,
36
+ };
37
+ return epoll_ctl(ndpl->epoll_fd, EPOLL_CTL_ADD, fd, &ev) == 0;
38
+}
39
+
40
+// Remove a file descriptor from the event poll
41
+bool nd_poll_del(nd_poll_t *ndpl, int fd) {
42
+ return epoll_ctl(ndpl->epoll_fd, EPOLL_CTL_DEL, fd, NULL) == 0;
43
+}
44
+
45
+// Update an existing file descriptor in the event poll
46
+bool nd_poll_upd(nd_poll_t *ndpl, int fd, nd_poll_event_t events, void *data) {
47
+ struct epoll_event ev = {
48
+ .events = (events & ND_POLL_READ ? EPOLLIN : 0) | (events & ND_POLL_WRITE ? EPOLLOUT : 0),
49
+ .data.ptr = data,
50
+ };
51
+ return epoll_ctl(ndpl->epoll_fd, EPOLL_CTL_MOD, fd, &ev) == 0;
52
+}
53
+
54
+static inline bool nd_poll_get_next_event(nd_poll_t *ndpl, nd_poll_result_t *result) {
55
+ for(size_t i = ndpl->last_pos; i < ndpl->used ;i++) {
56
+ *result = (nd_poll_result_t){
57
+ .events = 0,
58
+ .data = ndpl->ev[i].data.ptr,
59
+ };
60
+
61
+ if (ndpl->ev[i].events & EPOLLIN)
62
+ result->events |= ND_POLL_READ;
63
+
64
+ if (ndpl->ev[i].events & EPOLLOUT)
65
+ result->events |= ND_POLL_WRITE;
66
+
67
+ if (ndpl->ev[i].events & EPOLLERR)
68
+ result->events |= ND_POLL_ERROR;
69
+
70
+ if (ndpl->ev[i].events & EPOLLHUP)
71
+ result->events |= ND_POLL_HUP;
72
+
73
+ ndpl->last_pos = i + 1;
74
+ return true;
75
+ }
76
+
77
+ ndpl->last_pos = _countof(ndpl->ev);
78
+ return false;
79
+}
80
+
81
+// Wait for events
82
+int nd_poll_wait(nd_poll_t *ndpl, int timeout_ms, nd_poll_result_t *result) {
83
+ if(nd_poll_get_next_event(ndpl, result))
84
+ return 1;
85
+
86
+ do {
87
+ errno_clear();
88
+ ndpl->last_pos = 0;
89
+ ndpl->used = 0;
90
+ int n = epoll_wait(ndpl->epoll_fd, &ndpl->ev[0], _countof(ndpl->ev), timeout_ms);
91
+
92
+ if(unlikely(n <= 0)) {
93
+ if (n < 0) {
94
+ result->events = ND_POLL_OTHER_ERROR;
95
+ result->data = NULL;
96
+ return -1;
97
+ }
98
+ else {
99
+ result->events = ND_POLL_TIMEOUT;
100
+ result->data = NULL;
101
+ return 0;
102
+ }
103
+ }
104
+
105
+ ndpl->used = n;
106
+ if (nd_poll_get_next_event(ndpl, result))
107
+ return 1;
108
+
109
+ } while(true);
110
+}
111
+
112
+// Destroy the event poll context
113
+void nd_poll_destroy(nd_poll_t *ndpl) {
114
+ if (ndpl) {
115
+ close(ndpl->epoll_fd);
116
+ freez(ndpl);
117
+ }
118
+}
119
+#else
120
+
121
+DEFINE_JUDYL_TYPED(POINTERS, void *);
122
+
123
+struct nd_poll_t {
124
+ struct pollfd *fds; // Array of file descriptors
125
+ nfds_t nfds; // Number of active file descriptors
126
+ nfds_t capacity; // Allocated capacity for `fds` array
127
+ nfds_t last_pos;
128
+ POINTERS_JudyLSet pointers; // Judy array to store user data
129
+};
130
+
131
+#define INITIAL_CAPACITY 4
132
+
133
+// Initialize the event poll context
134
+nd_poll_t *nd_poll_create() {
135
+ nd_poll_t *ndpl = callocz(1, sizeof(nd_poll_t));
136
+ ndpl->fds = mallocz(INITIAL_CAPACITY * sizeof(struct pollfd));
137
+ ndpl->nfds = 0;
138
+ ndpl->capacity = INITIAL_CAPACITY;
139
+
140
+ POINTERS_INIT(&ndpl->pointers);
141
+
142
+ return ndpl;
143
+}
144
+
145
+// Ensure capacity for adding new file descriptors
146
+static void ensure_capacity(nd_poll_t *ndpl) {
147
+ if (ndpl->nfds < ndpl->capacity) return;
148
+
149
+ nfds_t new_capacity = ndpl->capacity * 2;
150
+ struct pollfd *new_fds = reallocz(ndpl->fds, new_capacity * sizeof(struct pollfd));
151
+
152
+ ndpl->fds = new_fds;
153
+ ndpl->capacity = new_capacity;
154
+}
155
+
156
+bool nd_poll_add(nd_poll_t *ndpl, int fd, nd_poll_event_t events, void *data) {
157
+ internal_fatal(POINTERS_GET(&ndpl->pointers, fd) != NULL, "File descriptor %d is already served - cannot add", fd);
158
+
159
+ ensure_capacity(ndpl);
160
+ struct pollfd *pfd = &ndpl->fds[ndpl->nfds++];
161
+ pfd->fd = fd;
162
+ pfd->events = 0;
163
+ if (events & ND_POLL_READ) pfd->events |= POLLIN;
164
+ if (events & ND_POLL_WRITE) pfd->events |= POLLOUT;
165
+ pfd->revents = 0;
166
+
167
+ POINTERS_SET(&ndpl->pointers, fd, data);
168
+
169
+ return true;
170
+}
171
+
172
+// Remove a file descriptor from the event poll
173
+bool nd_poll_del(nd_poll_t *ndpl, int fd) {
174
+ for (nfds_t i = 0; i < ndpl->nfds; i++) {
175
+ if (ndpl->fds[i].fd == fd) {
176
+ // Remove the file descriptor by shifting the array
177
+ memmove(&ndpl->fds[i], &ndpl->fds[i + 1], (ndpl->nfds - i - 1) * sizeof(struct pollfd));
178
+ ndpl->nfds--;
179
+ POINTERS_DEL(&ndpl->pointers, fd);
180
+ return true;
181
+ }
182
+ }
183
+
184
+ return false; // File descriptor not found
185
+}
186
+
187
+// Update an existing file descriptor in the event poll
188
+bool nd_poll_upd(nd_poll_t *ndpl, int fd, nd_poll_event_t events, void *data) {
189
+ internal_fatal(POINTERS_GET(&ndpl->pointers, fd) == NULL, "File descriptor %d is not found - cannot modify", fd);
190
+
191
+ for (nfds_t i = 0; i < ndpl->nfds; i++) {
192
+ if (ndpl->fds[i].fd == fd) {
193
+ struct pollfd *pfd = &ndpl->fds[i];
194
+ pfd->events = 0;
195
+ if (events & ND_POLL_READ) pfd->events |= POLLIN;
196
+ if (events & ND_POLL_WRITE) pfd->events |= POLLOUT;
197
+ POINTERS_SET(&ndpl->pointers, fd, data);
198
+ return true;
199
+ }
200
+ }
201
+
202
+ // File descriptor not found
203
+ return false;
204
+}
205
+
206
+static inline bool nd_poll_get_next_event(nd_poll_t *ndpl, nd_poll_result_t *result) {
207
+ for (nfds_t i = ndpl->last_pos; i < ndpl->nfds; i++) {
208
+ if (ndpl->fds[i].revents != 0) {
209
+
210
+ result->data = POINTERS_GET(&ndpl->pointers, ndpl->fds[i].fd);
211
+
212
+ result->events = 0;
213
+ if (ndpl->fds[i].revents & (POLLIN|POLLPRI))
214
+ result->events |= ND_POLL_READ;
215
+
216
+ if (ndpl->fds[i].revents & POLLOUT)
217
+ result->events |= ND_POLL_WRITE;
218
+
219
+ if (ndpl->fds[i].revents & POLLERR)
220
+ result->events |= ND_POLL_ERROR;
221
+
222
+ if (ndpl->fds[i].revents & POLLHUP)
223
+ result->events |= ND_POLL_HUP;
224
+
225
+ if (ndpl->fds[i].revents & POLLNVAL)
226
+ result->events |= ND_POLL_INVALID;
227
+
228
+ ndpl->fds[i].revents = 0; // Clear the event after handling
229
+ ndpl->last_pos = i + 1;
230
+ return true; // Return only the first triggered event
231
+ }
232
+ }
233
+
234
+ ndpl->last_pos = ndpl->nfds;
235
+ return false;
236
+}
237
+
238
+// Rotate the fds array to prevent starvation
239
+static inline void rotate_fds(nd_poll_t *ndpl) {
240
+ if (ndpl->nfds == 0 || ndpl->nfds == 1)
241
+ return; // No rotation needed for empty or single-entry arrays
242
+
243
+ struct pollfd first = ndpl->fds[0];
244
+ memmove(&ndpl->fds[0], &ndpl->fds[1], (ndpl->nfds - 1) * sizeof(struct pollfd));
245
+ ndpl->fds[ndpl->nfds - 1] = first;
246
+}
247
+
248
+// Wait for events
249
+int nd_poll_wait(nd_poll_t *ndpl, int timeout_ms, nd_poll_result_t *result) {
250
+ if (nd_poll_get_next_event(ndpl, result))
251
+ return 1; // Return immediately if there's a pending event
252
+
253
+ do {
254
+ errno_clear();
255
+ ndpl->last_pos = 0;
256
+ rotate_fds(ndpl); // Rotate the array on every wait
257
+ int ret = poll(ndpl->fds, ndpl->nfds, timeout_ms);
258
+
259
+ if(unlikely(ret <= 0)) {
260
+ if (ret < 0) {
261
+ if(errno == EAGAIN || errno == EINTR)
262
+ continue;
263
+
264
+ result->events = ND_POLL_OTHER_ERROR;
265
+ result->data = NULL;
266
+ return -1;
267
+ }
268
+ else {
269
+ result->events = ND_POLL_TIMEOUT;
270
+ result->data = NULL;
271
+ return 0;
272
+ }
273
+ }
274
+
275
+ // Process the next event
276
+ if (nd_poll_get_next_event(ndpl, result))
277
+ return 1;
278
+
279
+ } while (true);
280
+}
281
+
282
+// Destroy the event poll context
283
+void nd_poll_destroy(nd_poll_t *ndpl) {
284
+ if (ndpl) {
285
+ free(ndpl->fds);
286
+ POINTERS_FREE(&ndpl->pointers, NULL);
287
+ freez(ndpl);
288
+ }
289
+}
290
+
291
+#endif
src/libnetdata/socket/nd-poll.h
new
+35
@@ -0,0 +1,35 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_ND_POLL_H
4
+#define NETDATA_ND_POLL_H
5
+
6
+#include "libnetdata/libnetdata.h"
7
+
8
+typedef enum {
9
+ ND_POLL_READ = 1 << 0,
10
+ ND_POLL_WRITE = 1 << 1,
11
+ ND_POLL_ERROR = 1 << 2,
12
+ ND_POLL_HUP = 1 << 3,
13
+ ND_POLL_INVALID = 1 << 4,
14
+ ND_POLL_TIMEOUT = 1 << 5,
15
+ ND_POLL_OTHER_ERROR = 1 << 6,
16
+} nd_poll_event_t;
17
+
18
+typedef struct {
19
+ nd_poll_event_t events;
20
+ void *data;
21
+} nd_poll_result_t;
22
+
23
+typedef struct nd_poll_t nd_poll_t;
24
+
25
+nd_poll_t *nd_poll_create();
26
+bool nd_poll_add(nd_poll_t *ndpl, int fd, nd_poll_event_t events, void *data);
27
+bool nd_poll_del(nd_poll_t *ndpl, int fd);
28
+bool nd_poll_upd(nd_poll_t *ndpl, int fd, nd_poll_event_t events, void *data);
29
+
30
+// returns -1 = error, 0 = timeout, 1 = event in result
31
+int nd_poll_wait(nd_poll_t *ndpl, int timeout_ms, nd_poll_result_t *result);
32
+
33
+void nd_poll_destroy(nd_poll_t *ndpl);
34
+
35
+#endif //NETDATA_ND_POLL_H
src/libnetdata/socket/nd-sock.c
new
+141
@@ -0,0 +1,141 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "libnetdata/libnetdata.h"
4
+
5
+ENUM_STR_MAP_DEFINE(ND_SOCK_ERROR) = {
6
+ { .id = ND_SOCK_ERR_NONE, .name = "NONE", },
7
+ { .id = ND_SOCK_ERR_CONNECTION_REFUSED, .name = "CONNECTION REFUSED", },
8
+ { .id = ND_SOCK_ERR_CANNOT_RESOLVE_HOSTNAME, .name = "CANNOT RESOLVE HOSTNAME", },
9
+ { .id = ND_SOCK_ERR_FAILED_TO_CREATE_SOCKET, .name = "FAILED TO CREATE SOCKET", },
10
+ { .id = ND_SOCK_ERR_NO_HOST_IN_DEFINITION, .name = "NO HOST IN DEFINITION", },
11
+ { .id = ND_SOCK_ERR_POLL_ERROR, .name = "POLL ERROR", },
12
+ { .id = ND_SOCK_ERR_TIMEOUT, .name = "TIMEOUT", },
13
+ { .id = ND_SOCK_ERR_SSL_CANT_ESTABLISH_SSL_CONNECTION, .name = "CANT ESTABLISH SSL CONNECTION", },
14
+ { .id = ND_SOCK_ERR_SSL_INVALID_CERTIFICATE, .name = "INVALID SSL CERTIFICATE", },
15
+ { .id = ND_SOCK_ERR_SSL_FAILED_TO_OPEN, .name = "FAILED TO OPEN SSL", },
16
+ { .id = ND_SOCK_ERR_THREAD_CANCELLED, .name = "THREAD CANCELLED", },
17
+ { .id = ND_SOCK_ERR_NO_DESTINATION_AVAILABLE, .name = "NO PARENT AVAILABLE", },
18
+ { .id = ND_SOCK_ERR_UNKNOWN_ERROR, .name = "UNKNOWN ERROR", },
19
+
20
+ // terminator
21
+ { .name = NULL, .id = 0 }
22
+};
23
+
24
+ENUM_STR_DEFINE_FUNCTIONS(ND_SOCK_ERROR, ND_SOCK_ERR_NONE, "");
25
+
26
+// --------------------------------------------------------------------------------------------------------------------
27
+
28
+static const unsigned char alpn_proto_list[] = {
29
+ 18, 'n', 'e', 't', 'd', 'a', 't', 'a', '_', 's', 't', 'r', 'e', 'a', 'm', '/', '2', '.', '0',
30
+ 8, 'h', 't', 't', 'p', '/', '1', '.', '1'
31
+};
32
+
33
+static bool nd_sock_open_ssl(ND_SOCK *s) {
34
+ if(!s) return false;
35
+
36
+ if (netdata_ssl_open_ext(&s->ssl, s->ctx, s->fd, alpn_proto_list, sizeof(alpn_proto_list))) {
37
+ if(!netdata_ssl_connect(&s->ssl)) {
38
+ // couldn't connect
39
+ s->error = ND_SOCK_ERR_SSL_CANT_ESTABLISH_SSL_CONNECTION;
40
+ return false;
41
+ }
42
+
43
+ if (s->verify_certificate && security_test_certificate(s->ssl.conn)) {
44
+ // certificate is not valid
45
+ s->error = ND_SOCK_ERR_SSL_INVALID_CERTIFICATE;
46
+ return false;
47
+ }
48
+
49
+ return true;
50
+ }
51
+
52
+ s->error = ND_SOCK_ERR_SSL_FAILED_TO_OPEN;
53
+ return false;
54
+}
55
+
56
+bool nd_sock_connect_to_this(ND_SOCK *s, const char *definition, int default_port, time_t timeout, bool ssl) {
57
+ nd_sock_close(s);
58
+
59
+ struct timeval tv = {
60
+ .tv_sec = timeout,
61
+ .tv_usec = 0
62
+ };
63
+
64
+ s->fd = connect_to_this(definition, default_port, &tv);
65
+ if(s->fd < 0) {
66
+ s->error = -s->fd;
67
+ return false;
68
+ }
69
+
70
+ if(ssl && s->ctx) {
71
+ if (!nd_sock_open_ssl(s)) {
72
+ close(s->fd);
73
+ s->fd = -1;
74
+ return false;
75
+ }
76
+ }
77
+ else
78
+ s->ssl = NETDATA_SSL_UNSET_CONNECTION;
79
+
80
+ return true;
81
+}
82
+
83
+ssize_t nd_sock_send_timeout(ND_SOCK *s, void *buf, size_t len, int flags, time_t timeout) {
84
+ switch(wait_on_socket_or_cancel_with_timeout(&s->ssl, s->fd, (int)(timeout * 1000), POLLOUT, NULL)) {
85
+ case 0: // data are waiting
86
+ break;
87
+
88
+ case 1: // timeout
89
+ s->error = ND_SOCK_ERR_TIMEOUT;
90
+ return 0;
91
+
92
+ case -1: // thread cancelled
93
+ s->error = ND_SOCK_ERR_THREAD_CANCELLED;
94
+ return -1;
95
+
96
+ case 2: // poll() error
97
+ s->error = ND_SOCK_ERR_POLL_ERROR;
98
+ return -1;
99
+
100
+ default:
101
+ s->error = ND_SOCK_ERR_UNKNOWN_ERROR;
102
+ return -1;
103
+ }
104
+
105
+ if(s->ssl.conn) {
106
+ if (nd_sock_is_ssl(s))
107
+ return netdata_ssl_write(&s->ssl, buf, len);
108
+ else
109
+ return -1;
110
+ }
111
+
112
+ return send(s->fd, buf, len, flags);
113
+}
114
+
115
+ssize_t nd_sock_recv_timeout(ND_SOCK *s, void *buf, size_t len, int flags, time_t timeout) {
116
+ switch(wait_on_socket_or_cancel_with_timeout(&s->ssl, s->fd, (int)(timeout * 1000), POLLIN, NULL)) {
117
+ case 0: // data are waiting
118
+ break;
119
+
120
+ case 1: // timeout
121
+ s->error = ND_SOCK_ERR_TIMEOUT;
122
+ return 0;
123
+
124
+ case -1: // thread cancelled
125
+ s->error = ND_SOCK_ERR_THREAD_CANCELLED;
126
+ return -1;
127
+
128
+ case 2: // poll() error
129
+ s->error = ND_SOCK_ERR_POLL_ERROR;
130
+ return -1;
131
+
132
+ default:
133
+ s->error = ND_SOCK_ERR_UNKNOWN_ERROR;
134
+ return -1;
135
+ }
136
+
137
+ if (nd_sock_is_ssl(s))
138
+ return netdata_ssl_read(&s->ssl, buf, len);
139
+
140
+ return recv(s->fd, buf, len, flags);
141
+}
src/libnetdata/socket/nd-sock.h
new
+135
@@ -0,0 +1,135 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_ND_SOCK_H
4
+#define NETDATA_ND_SOCK_H
5
+
6
+#include "socket-peers.h"
7
+
8
+typedef enum __attribute__((packed)) {
9
+ ND_SOCK_ERR_NONE = 0,
10
+ ND_SOCK_ERR_CONNECTION_REFUSED,
11
+ ND_SOCK_ERR_CANNOT_RESOLVE_HOSTNAME,
12
+ ND_SOCK_ERR_FAILED_TO_CREATE_SOCKET,
13
+ ND_SOCK_ERR_NO_HOST_IN_DEFINITION,
14
+ ND_SOCK_ERR_POLL_ERROR,
15
+ ND_SOCK_ERR_TIMEOUT,
16
+ ND_SOCK_ERR_SSL_CANT_ESTABLISH_SSL_CONNECTION,
17
+ ND_SOCK_ERR_SSL_INVALID_CERTIFICATE,
18
+ ND_SOCK_ERR_SSL_FAILED_TO_OPEN,
19
+ ND_SOCK_ERR_THREAD_CANCELLED,
20
+ ND_SOCK_ERR_NO_DESTINATION_AVAILABLE,
21
+ ND_SOCK_ERR_UNKNOWN_ERROR,
22
+} ND_SOCK_ERROR;
23
+
24
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(ND_SOCK_ERROR);
25
+
26
+typedef struct nd_sock {
27
+ bool verify_certificate;
28
+ ND_SOCK_ERROR error;
29
+ int fd;
30
+ NETDATA_SSL ssl;
31
+ SSL_CTX *ctx;
32
+} ND_SOCK;
33
+
34
+#define ND_SOCK_INIT(ssl_ctx, ssl_verify) (ND_SOCK){ \
35
+ .verify_certificate = ssl_verify, \
36
+ .error = ND_SOCK_ERR_NONE, \
37
+ .fd = -1, \
38
+ .ssl = NETDATA_SSL_UNSET_CONNECTION, \
39
+ .ctx = ssl_ctx, \
40
+}
41
+
42
+static inline void nd_sock_init(ND_SOCK *s, SSL_CTX *ctx, bool verify_certificate) {
43
+ s->verify_certificate = verify_certificate;
44
+ s->error = ND_SOCK_ERR_NONE;
45
+ s->fd = -1;
46
+ s->ssl = NETDATA_SSL_UNSET_CONNECTION;
47
+ s->ctx = ctx;
48
+}
49
+
50
+static inline bool nd_sock_is_ssl(ND_SOCK *s) {
51
+ return SSL_connection(&s->ssl);
52
+}
53
+
54
+static inline SOCKET_PEERS nd_sock_socket_peers(ND_SOCK *s) {
55
+ return socket_peers(s->fd);
56
+}
57
+
58
+static inline void nd_sock_close(ND_SOCK *s) {
59
+ netdata_ssl_close(&s->ssl);
60
+
61
+ if(s->fd != -1) {
62
+ close(s->fd);
63
+ s->fd = -1;
64
+ }
65
+
66
+ s->error = ND_SOCK_ERR_NONE;
67
+}
68
+
69
+static inline ssize_t nd_sock_read(ND_SOCK *s, void *buf, size_t num, size_t retries) {
70
+ ssize_t rc;
71
+ do {
72
+ if (nd_sock_is_ssl(s))
73
+ rc = netdata_ssl_read(&s->ssl, buf, num);
74
+ else
75
+ rc = read(s->fd, buf, num);
76
+ }
77
+ while(rc <= 0 && (errno == EWOULDBLOCK || errno == EAGAIN || errno == EINTR) && retries--);
78
+
79
+ return rc;
80
+}
81
+
82
+static inline ssize_t nd_sock_write(ND_SOCK *s, const void *buf, size_t num, size_t retries) {
83
+ ssize_t rc;
84
+
85
+ do {
86
+ errno_clear();
87
+ if (nd_sock_is_ssl(s))
88
+ rc = netdata_ssl_write(&s->ssl, buf, num);
89
+ else
90
+ rc = write(s->fd, buf, num);
91
+ }
92
+ while(rc <= 0 && (errno == EWOULDBLOCK || errno == EAGAIN || errno == EINTR) && retries--);
93
+
94
+ return rc;
95
+}
96
+
97
+static inline ssize_t nd_sock_write_persist(ND_SOCK *s, const void *buf, const size_t num, size_t retries) {
98
+ const uint8_t *src = (const uint8_t *)buf;
99
+ ssize_t bytes = 0;
100
+
101
+ do {
102
+ ssize_t sent = nd_sock_write(s, &src[bytes], (ssize_t)num - bytes, retries);
103
+ if(sent <= 0) return sent;
104
+ bytes += sent;
105
+ }
106
+ while(bytes < (ssize_t)num && retries--);
107
+
108
+ return bytes;
109
+}
110
+
111
+static inline ssize_t nd_sock_revc_nowait(ND_SOCK *s, void *buf, size_t num) {
112
+ if (nd_sock_is_ssl(s))
113
+ return netdata_ssl_read(&s->ssl, buf, num);
114
+ else
115
+ return recv(s->fd, buf, num, MSG_DONTWAIT);
116
+}
117
+
118
+static inline ssize_t nd_sock_send_nowait(ND_SOCK *s, void *buf, size_t num) {
119
+ if (nd_sock_is_ssl(s))
120
+ return netdata_ssl_write(&s->ssl, buf, num);
121
+ else
122
+ return send(s->fd, buf, num, MSG_DONTWAIT);
123
+}
124
+
125
+ssize_t nd_sock_send_timeout(ND_SOCK *s, void *buf, size_t len, int flags, time_t timeout);
126
+ssize_t nd_sock_recv_timeout(ND_SOCK *s, void *buf, size_t len, int flags, time_t timeout);
127
+
128
+bool nd_sock_connect_to_this(ND_SOCK *s, const char *definition, int default_port, time_t timeout, bool ssl);
129
+
130
+static inline void cleanup_nd_sock_p(ND_SOCK *s) {
131
+ if(s) nd_sock_close(s);
132
+}
133
+#define CLEAN_ND_SOCK _cleanup_(cleanup_nd_sock_p) ND_SOCK
134
+
135
+#endif //NETDATA_ND_SOCK_H
src/libnetdata/socket/poll-events.c
new
+677
@@ -0,0 +1,677 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "libnetdata/libnetdata.h"
4
+
5
+// poll() based listener
6
+// this should be the fastest possible listener for up to 100 sockets
7
+// above 100, an epoll() interface is needed on Linux
8
+
9
+#define POLL_FDS_INCREASE_STEP 10
10
+
11
+inline POLLINFO *poll_add_fd(POLLJOB *p
12
+ , int fd
13
+ , int socktype
14
+ , HTTP_ACL port_acl
15
+ , uint32_t flags
16
+ , const char *client_ip
17
+ , const char *client_port
18
+ , const char *client_host
19
+ , void *(*add_callback)(POLLINFO * /*pi*/, short int * /*events*/, void * /*data*/)
20
+ , void (*del_callback)(POLLINFO * /*pi*/)
21
+ , int (*rcv_callback)(POLLINFO * /*pi*/, short int * /*events*/)
22
+ , int (*snd_callback)(POLLINFO * /*pi*/, short int * /*events*/)
23
+ , void *data
24
+) {
25
+ if(unlikely(fd < 0)) return NULL;
26
+
27
+ //if(p->limit && p->used >= p->limit) {
28
+ // nd_log(NDLS_DAEMON, NDLP_WARNING, "Max sockets limit reached (%zu sockets), dropping connection", p->used);
29
+ // close(fd);
30
+ // return NULL;
31
+ //}
32
+
33
+ if(unlikely(!p->first_free)) {
34
+ size_t new_slots = p->slots + POLL_FDS_INCREASE_STEP;
35
+
36
+ p->fds = reallocz(p->fds, sizeof(struct pollfd) * new_slots);
37
+ p->inf = reallocz(p->inf, sizeof(POLLINFO) * new_slots);
38
+
39
+ // reset all the newly added slots
40
+ ssize_t i;
41
+ for(i = new_slots - 1; i >= (ssize_t)p->slots ; i--) {
42
+ p->fds[i].fd = -1;
43
+ p->fds[i].events = 0;
44
+ p->fds[i].revents = 0;
45
+
46
+ p->inf[i].p = p;
47
+ p->inf[i].slot = (size_t)i;
48
+ p->inf[i].flags = 0;
49
+ p->inf[i].socktype = -1;
50
+ p->inf[i].port_acl = -1;
51
+
52
+ p->inf[i].client_ip = NULL;
53
+ p->inf[i].client_port = NULL;
54
+ p->inf[i].client_host = NULL;
55
+ p->inf[i].del_callback = p->del_callback;
56
+ p->inf[i].rcv_callback = p->rcv_callback;
57
+ p->inf[i].snd_callback = p->snd_callback;
58
+ p->inf[i].data = NULL;
59
+
60
+ // link them so that the first free will be earlier in the array
61
+ // (we loop decrementing i)
62
+ p->inf[i].next = p->first_free;
63
+ p->first_free = &p->inf[i];
64
+ }
65
+
66
+ p->slots = new_slots;
67
+ }
68
+
69
+ POLLINFO *pi = p->first_free;
70
+ p->first_free = p->first_free->next;
71
+
72
+ struct pollfd *pf = &p->fds[pi->slot];
73
+ pf->fd = fd;
74
+ pf->events = POLLIN;
75
+ pf->revents = 0;
76
+
77
+ pi->fd = fd;
78
+ pi->p = p;
79
+ pi->socktype = socktype;
80
+ pi->port_acl = port_acl;
81
+ pi->flags = flags;
82
+ pi->next = NULL;
83
+ pi->client_ip = strdupz(client_ip);
84
+ pi->client_port = strdupz(client_port);
85
+ pi->client_host = strdupz(client_host);
86
+
87
+ pi->del_callback = del_callback;
88
+ pi->rcv_callback = rcv_callback;
89
+ pi->snd_callback = snd_callback;
90
+
91
+ pi->connected_t = now_boottime_sec();
92
+ pi->last_received_t = 0;
93
+ pi->last_sent_t = 0;
94
+ pi->last_sent_t = 0;
95
+ pi->recv_count = 0;
96
+ pi->send_count = 0;
97
+
98
+ p->used++;
99
+ if(unlikely(pi->slot > p->max))
100
+ p->max = pi->slot;
101
+
102
+ if(pi->flags & POLLINFO_FLAG_CLIENT_SOCKET) {
103
+ pi->data = add_callback(pi, &pf->events, data);
104
+ }
105
+
106
+ if(pi->flags & POLLINFO_FLAG_SERVER_SOCKET) {
107
+ p->min = pi->slot;
108
+ }
109
+
110
+ return pi;
111
+}
112
+
113
+inline void poll_close_fd(POLLINFO *pi) {
114
+ POLLJOB *p = pi->p;
115
+
116
+ struct pollfd *pf = &p->fds[pi->slot];
117
+
118
+ if(unlikely(pf->fd == -1)) return;
119
+
120
+ if(pi->flags & POLLINFO_FLAG_CLIENT_SOCKET) {
121
+ pi->del_callback(pi);
122
+
123
+ if(likely(!(pi->flags & POLLINFO_FLAG_DONT_CLOSE))) {
124
+ if(close(pf->fd) == -1)
125
+ nd_log(NDLS_DAEMON, NDLP_ERR,
126
+ "Failed to close() poll_events() socket %d",
127
+ pf->fd);
128
+ }
129
+ }
130
+
131
+ pf->fd = -1;
132
+ pf->events = 0;
133
+ pf->revents = 0;
134
+
135
+ pi->fd = -1;
136
+ pi->socktype = -1;
137
+ pi->flags = 0;
138
+ pi->data = NULL;
139
+
140
+ pi->del_callback = NULL;
141
+ pi->rcv_callback = NULL;
142
+ pi->snd_callback = NULL;
143
+
144
+ freez(pi->client_ip);
145
+ pi->client_ip = NULL;
146
+
147
+ freez(pi->client_port);
148
+ pi->client_port = NULL;
149
+
150
+ freez(pi->client_host);
151
+ pi->client_host = NULL;
152
+
153
+ pi->next = p->first_free;
154
+ p->first_free = pi;
155
+
156
+ p->used--;
157
+ if(unlikely(p->max == pi->slot)) {
158
+ p->max = p->min;
159
+ ssize_t i;
160
+ for(i = (ssize_t)pi->slot; i > (ssize_t)p->min ;i--) {
161
+ if (unlikely(p->fds[i].fd != -1)) {
162
+ p->max = (size_t)i;
163
+ break;
164
+ }
165
+ }
166
+ }
167
+}
168
+
169
+void *poll_default_add_callback(POLLINFO *pi, short int *events, void *data) {
170
+ (void)pi;
171
+ (void)events;
172
+ (void)data;
173
+
174
+ return NULL;
175
+}
176
+
177
+void poll_default_del_callback(POLLINFO *pi) {
178
+ if(pi->data)
179
+ nd_log(NDLS_DAEMON, NDLP_ERR,
180
+ "POLLFD: internal error: del_callback_default() called with data pointer - possible memory leak");
181
+}
182
+
183
+int poll_default_rcv_callback(POLLINFO *pi, short int *events) {
184
+ *events |= POLLIN;
185
+
186
+ char buffer[1024 + 1];
187
+
188
+ ssize_t rc;
189
+ do {
190
+ rc = recv(pi->fd, buffer, 1024, MSG_DONTWAIT);
191
+ if (rc < 0) {
192
+ // read failed
193
+ if (errno != EWOULDBLOCK && errno != EAGAIN) {
194
+ nd_log(NDLS_DAEMON, NDLP_ERR,
195
+ "POLLFD: poll_default_rcv_callback(): recv() failed with %zd.",
196
+ rc);
197
+
198
+ return -1;
199
+ }
200
+ } else if (rc) {
201
+ // data received
202
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
203
+ "POLLFD: internal error: poll_default_rcv_callback() is discarding %zd bytes received on socket %d",
204
+ rc, pi->fd);
205
+ }
206
+ } while (rc != -1);
207
+
208
+ return 0;
209
+}
210
+
211
+int poll_default_snd_callback(POLLINFO *pi, short int *events) {
212
+ *events &= ~POLLOUT;
213
+
214
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
215
+ "POLLFD: internal error: poll_default_snd_callback(): nothing to send on socket %d",
216
+ pi->fd);
217
+
218
+ return 0;
219
+}
220
+
221
+void poll_default_tmr_callback(void *timer_data) {
222
+ (void)timer_data;
223
+}
224
+
225
+static void poll_events_cleanup(void *pptr) {
226
+ POLLJOB *p = CLEANUP_FUNCTION_GET_PTR(pptr);
227
+ if(!p) return;
228
+
229
+ for(size_t i = 0 ; i <= p->max ; i++) {
230
+ POLLINFO *pi = &p->inf[i];
231
+ poll_close_fd(pi);
232
+ }
233
+
234
+ freez(p->fds);
235
+ freez(p->inf);
236
+}
237
+
238
+static int poll_process_error(POLLINFO *pi, struct pollfd *pf, short int revents) {
239
+ ND_LOG_STACK lgs[] = {
240
+ ND_LOG_FIELD_TXT(NDF_SRC_IP, pi->client_ip),
241
+ ND_LOG_FIELD_TXT(NDF_SRC_PORT, pi->client_port),
242
+ ND_LOG_FIELD_END(),
243
+ };
244
+ ND_LOG_STACK_PUSH(lgs);
245
+
246
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
247
+ "POLLFD: LISTENER: received %s %s %s on socket at slot %zu (fd %d) client '%s' port '%s' expecting %s %s %s, having %s %s %s"
248
+ , revents & POLLERR ? "POLLERR" : ""
249
+ , revents & POLLHUP ? "POLLHUP" : ""
250
+ , revents & POLLNVAL ? "POLLNVAL" : ""
251
+ , pi->slot
252
+ , pi->fd
253
+ , pi->client_ip ? pi->client_ip : "<undefined-ip>"
254
+ , pi->client_port ? pi->client_port : "<undefined-port>"
255
+ , pf->events & POLLIN ? "POLLIN" : "", pf->events & POLLOUT ? "POLLOUT" : "", pf->events & POLLPRI ? "POLLPRI" : ""
256
+ , revents & POLLIN ? "POLLIN" : "", revents & POLLOUT ? "POLLOUT" : "", revents & POLLPRI ? "POLLPRI" : ""
257
+ );
258
+
259
+ pf->events = 0;
260
+ poll_close_fd(pi);
261
+ return 1;
262
+}
263
+
264
+static inline int poll_process_send(POLLJOB *p, POLLINFO *pi, struct pollfd *pf, time_t now) {
265
+ pi->last_sent_t = now;
266
+ pi->send_count++;
267
+
268
+ pf->events = 0;
269
+
270
+ // remember the slot, in case we need to close it later
271
+ // the callback may manipulate the socket list and our pf and pi pointers may be invalid after that call
272
+ size_t slot = pi->slot;
273
+
274
+ if (unlikely(pi->snd_callback(pi, &pf->events) == -1))
275
+ poll_close_fd(&p->inf[slot]);
276
+
277
+ // IMPORTANT:
278
+ // pf and pi may be invalid below this point, they may have been reallocated.
279
+
280
+ return 1;
281
+}
282
+
283
+static inline int poll_process_tcp_read(POLLJOB *p, POLLINFO *pi, struct pollfd *pf, time_t now) {
284
+ pi->last_received_t = now;
285
+ pi->recv_count++;
286
+
287
+ pf->events = 0;
288
+
289
+ // remember the slot, in case we need to close it later
290
+ // the callback may manipulate the socket list and our pf and pi pointers may be invalid after that call
291
+ size_t slot = pi->slot;
292
+
293
+ if (pi->rcv_callback(pi, &pf->events) == -1)
294
+ poll_close_fd(&p->inf[slot]);
295
+
296
+ // IMPORTANT:
297
+ // pf and pi may be invalid below this point, they may have been reallocated.
298
+
299
+ return 1;
300
+}
301
+
302
+static inline int poll_process_udp_read(POLLINFO *pi, struct pollfd *pf, time_t now __maybe_unused) {
303
+ pi->last_received_t = now;
304
+ pi->recv_count++;
305
+
306
+ // TODO: access_list is not applied to UDP
307
+ // but checking the access list on every UDP packet will destroy
308
+ // performance, especially for statsd.
309
+
310
+ pf->events = 0;
311
+ if(pi->rcv_callback(pi, &pf->events) == -1)
312
+ return 0;
313
+
314
+ // IMPORTANT:
315
+ // pf and pi may be invalid below this point, they may have been reallocated.
316
+
317
+ return 1;
318
+}
319
+
320
+static int poll_process_new_tcp_connection(POLLJOB *p, POLLINFO *pi, struct pollfd *pf, time_t now) {
321
+ pi->last_received_t = now;
322
+ pi->recv_count++;
323
+
324
+ char client_ip[INET6_ADDRSTRLEN] = "";
325
+ char client_port[NI_MAXSERV] = "";
326
+ char client_host[NI_MAXHOST] = "";
327
+
328
+#ifdef SOCK_NONBLOCK
329
+ int flags = SOCK_NONBLOCK;
330
+#else
331
+ int flags = 0;
332
+#endif
333
+
334
+ int nfd = accept_socket(
335
+ pf->fd, flags,
336
+ client_ip, INET6_ADDRSTRLEN, client_port,NI_MAXSERV, client_host, NI_MAXHOST,
337
+ p->access_list, p->allow_dns
338
+ );
339
+
340
+#ifndef SOCK_NONBLOCK
341
+ if (nfd > 0) {
342
+ int flags = fcntl(nfd, F_GETFL);
343
+ (void)fcntl(nfd, F_SETFL, flags| O_NONBLOCK);
344
+ }
345
+#endif
346
+
347
+ if (unlikely(nfd < 0)) {
348
+ // accept failed
349
+
350
+ if(unlikely(errno == EMFILE)) {
351
+ nd_log_limit_static_global_var(erl, 10, 1000);
352
+ nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
353
+ "POLLFD: LISTENER: too many open files - used by this thread %zu, max for this thread %zu",
354
+ p->used, p->limit);
355
+ }
356
+ else if(unlikely(errno != EWOULDBLOCK && errno != EAGAIN))
357
+ nd_log(NDLS_DAEMON, NDLP_ERR,
358
+ "POLLFD: LISTENER: accept() failed.");
359
+
360
+ }
361
+ else {
362
+ // accept ok
363
+
364
+ poll_add_fd(p
365
+ , nfd
366
+ , SOCK_STREAM
367
+ , pi->port_acl
368
+ , POLLINFO_FLAG_CLIENT_SOCKET
369
+ , client_ip
370
+ , client_port
371
+ , client_host
372
+ , p->add_callback
373
+ , p->del_callback
374
+ , p->rcv_callback
375
+ , p->snd_callback
376
+ , NULL
377
+ );
378
+
379
+ // IMPORTANT:
380
+ // pf and pi may be invalid below this point, they may have been reallocated.
381
+
382
+ return 1;
383
+ }
384
+
385
+ return 0;
386
+}
387
+
388
+void poll_events(LISTEN_SOCKETS *sockets
389
+ , void *(*add_callback)(POLLINFO * /*pi*/, short int * /*events*/, void * /*data*/)
390
+ , void (*del_callback)(POLLINFO * /*pi*/)
391
+ , int (*rcv_callback)(POLLINFO * /*pi*/, short int * /*events*/)
392
+ , int (*snd_callback)(POLLINFO * /*pi*/, short int * /*events*/)
393
+ , void (*tmr_callback)(void * /*timer_data*/)
394
+ , bool (*check_to_stop_callback)(void)
395
+ , SIMPLE_PATTERN *access_list
396
+ , int allow_dns
397
+ , void *data
398
+ , time_t tcp_request_timeout_seconds
399
+ , time_t tcp_idle_timeout_seconds
400
+ , time_t timer_milliseconds
401
+ , void *timer_data
402
+ , size_t max_tcp_sockets
403
+) {
404
+ if(!sockets || !sockets->opened) {
405
+ nd_log(NDLS_DAEMON, NDLP_ERR,
406
+ "POLLFD: internal error: no listening sockets are opened");
407
+ return;
408
+ }
409
+
410
+ if(timer_milliseconds <= 0) timer_milliseconds = 0;
411
+
412
+ int retval;
413
+
414
+ POLLJOB p = {
415
+ .slots = 0,
416
+ .used = 0,
417
+ .max = 0,
418
+ .limit = max_tcp_sockets,
419
+ .fds = NULL,
420
+ .inf = NULL,
421
+ .first_free = NULL,
422
+
423
+ .complete_request_timeout = tcp_request_timeout_seconds,
424
+ .idle_timeout = tcp_idle_timeout_seconds,
425
+ .checks_every = (tcp_idle_timeout_seconds / 3) + 1,
426
+
427
+ .access_list = access_list,
428
+ .allow_dns = allow_dns,
429
+
430
+ .timer_milliseconds = timer_milliseconds,
431
+ .timer_data = timer_data,
432
+
433
+ .add_callback = add_callback?add_callback:poll_default_add_callback,
434
+ .del_callback = del_callback?del_callback:poll_default_del_callback,
435
+ .rcv_callback = rcv_callback?rcv_callback:poll_default_rcv_callback,
436
+ .snd_callback = snd_callback?snd_callback:poll_default_snd_callback,
437
+ .tmr_callback = tmr_callback?tmr_callback:poll_default_tmr_callback
438
+ };
439
+
440
+ size_t i;
441
+ for(i = 0; i < sockets->opened ;i++) {
442
+
443
+ POLLINFO *pi = poll_add_fd(&p
444
+ , sockets->fds[i]
445
+ , sockets->fds_types[i]
446
+ , sockets->fds_acl_flags[i]
447
+ , POLLINFO_FLAG_SERVER_SOCKET
448
+ , (sockets->fds_names[i])?sockets->fds_names[i]:"UNKNOWN"
449
+ , ""
450
+ , ""
451
+ , p.add_callback
452
+ , p.del_callback
453
+ , p.rcv_callback
454
+ , p.snd_callback
455
+ , NULL
456
+ );
457
+
458
+ pi->data = data;
459
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
460
+ "POLLFD: LISTENER: listening on '%s'",
461
+ (sockets->fds_names[i])?sockets->fds_names[i]:"UNKNOWN");
462
+ }
463
+
464
+ int listen_sockets_active = 1;
465
+
466
+ time_t last_check = now_boottime_sec();
467
+
468
+ usec_t timer_usec = timer_milliseconds * USEC_PER_MS;
469
+ usec_t now_usec = 0, next_timer_usec = 0, last_timer_usec = 0;
470
+ (void)last_timer_usec;
471
+
472
+ if(unlikely(timer_usec)) {
473
+ now_usec = now_boottime_usec();
474
+ next_timer_usec = now_usec - (now_usec % timer_usec) + timer_usec;
475
+ }
476
+
477
+ CLEANUP_FUNCTION_REGISTER(poll_events_cleanup) cleanup_ptr = &p;
478
+
479
+ while(!check_to_stop_callback() && !nd_thread_signaled_to_cancel()) {
480
+ if(unlikely(timer_usec)) {
481
+ now_usec = now_boottime_usec();
482
+
483
+ if(unlikely(timer_usec && now_usec >= next_timer_usec)) {
484
+ last_timer_usec = now_usec;
485
+ p.tmr_callback(p.timer_data);
486
+ now_usec = now_boottime_usec();
487
+ next_timer_usec = now_usec - (now_usec % timer_usec) + timer_usec;
488
+ }
489
+ }
490
+
491
+ // enable or disable the TCP listening sockets, based on the current number of sockets used and the limit set
492
+ if((listen_sockets_active && (p.limit && p.used >= p.limit)) || (!listen_sockets_active && (!p.limit || p.used < p.limit))) {
493
+ listen_sockets_active = !listen_sockets_active;
494
+
495
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
496
+ "%s listening sockets (used TCP sockets %zu, max allowed for this worker %zu)",
497
+ (listen_sockets_active)?"ENABLING":"DISABLING", p.used, p.limit);
498
+
499
+ for (i = 0; i <= p.max; i++) {
500
+ if(p.inf[i].flags & POLLINFO_FLAG_SERVER_SOCKET && p.inf[i].socktype == SOCK_STREAM) {
501
+ p.fds[i].events = (short int) ((listen_sockets_active) ? POLLIN : 0);
502
+ }
503
+ }
504
+ }
505
+
506
+ retval = poll(p.fds, p.max + 1, ND_CHECK_CANCELLABILITY_WHILE_WAITING_EVERY_MS);
507
+ time_t now = now_boottime_sec();
508
+
509
+ if(unlikely(retval == -1)) {
510
+ nd_log(NDLS_DAEMON, NDLP_ERR,
511
+ "POLLFD: LISTENER: poll() failed while waiting on %zu sockets.",
512
+ p.max + 1);
513
+
514
+ break;
515
+ }
516
+ else if(unlikely(!retval)) {
517
+ // timeout
518
+ ;
519
+ }
520
+ else {
521
+ POLLINFO *pi;
522
+ struct pollfd *pf;
523
+ size_t idx, processed = 0;
524
+ short int revents;
525
+
526
+ // keep fast lookup arrays per function
527
+ // to avoid looping through the entire list every time
528
+ size_t sends[p.max + 1], sends_max = 0;
529
+ size_t reads[p.max + 1], reads_max = 0;
530
+ size_t conns[p.max + 1], conns_max = 0;
531
+ size_t udprd[p.max + 1], udprd_max = 0;
532
+
533
+ for (i = 0; i <= p.max; i++) {
534
+ pi = &p.inf[i];
535
+ pf = &p.fds[i];
536
+ revents = pf->revents;
537
+
538
+ if(unlikely(revents == 0 || pf->fd == -1))
539
+ continue;
540
+
541
+ if (unlikely(revents & (POLLERR|POLLHUP|POLLNVAL))) {
542
+ // something is wrong to one of our sockets
543
+
544
+ pf->revents = 0;
545
+ processed += poll_process_error(pi, pf, revents);
546
+ }
547
+ else if (likely(revents & POLLOUT)) {
548
+ // a client is ready to receive data
549
+
550
+ sends[sends_max++] = i;
551
+ }
552
+ else if (likely(revents & (POLLIN|POLLPRI))) {
553
+ if (pi->flags & POLLINFO_FLAG_CLIENT_SOCKET) {
554
+ // a client sent data to us
555
+
556
+ reads[reads_max++] = i;
557
+ }
558
+ else if (pi->flags & POLLINFO_FLAG_SERVER_SOCKET) {
559
+ // something is coming to our server sockets
560
+
561
+ if(pi->socktype == SOCK_DGRAM) {
562
+ // UDP receive, directly on our listening socket
563
+
564
+ udprd[udprd_max++] = i;
565
+ }
566
+ else if(pi->socktype == SOCK_STREAM) {
567
+ // new TCP connection
568
+
569
+ conns[conns_max++] = i;
570
+ }
571
+ else
572
+ nd_log(NDLS_DAEMON, NDLP_ERR,
573
+ "POLLFD: LISTENER: server slot %zu (fd %d) connection from %s port %s using unhandled socket type %d."
574
+ , i
575
+ , pi->fd
576
+ , pi->client_ip ? pi->client_ip : "<undefined-ip>"
577
+ , pi->client_port ? pi->client_port : "<undefined-port>"
578
+ , pi->socktype
579
+ );
580
+ }
581
+ else
582
+ nd_log(NDLS_DAEMON, NDLP_ERR,
583
+ "POLLFD: LISTENER: client slot %zu (fd %d) data from %s port %s using flags %08X is neither client nor server."
584
+ , i
585
+ , pi->fd
586
+ , pi->client_ip ? pi->client_ip : "<undefined-ip>"
587
+ , pi->client_port ? pi->client_port : "<undefined-port>"
588
+ , pi->flags
589
+ );
590
+ }
591
+ else
592
+ nd_log(NDLS_DAEMON, NDLP_ERR,
593
+ "POLLFD: LISTENER: socket slot %zu (fd %d) client %s port %s unhandled event id %d."
594
+ , i
595
+ , pi->fd
596
+ , pi->client_ip ? pi->client_ip : "<undefined-ip>"
597
+ , pi->client_port ? pi->client_port : "<undefined-port>"
598
+ , revents
599
+ );
600
+ }
601
+
602
+ // process sends
603
+ for (idx = 0; idx < sends_max; idx++) {
604
+ i = sends[idx];
605
+ pi = &p.inf[i];
606
+ pf = &p.fds[i];
607
+ pf->revents = 0;
608
+ processed += poll_process_send(&p, pi, pf, now);
609
+ }
610
+
611
+ // process UDP reads
612
+ for (idx = 0; idx < udprd_max; idx++) {
613
+ i = udprd[idx];
614
+ pi = &p.inf[i];
615
+ pf = &p.fds[i];
616
+ pf->revents = 0;
617
+ processed += poll_process_udp_read(pi, pf, now);
618
+ }
619
+
620
+ // process TCP reads
621
+ for (idx = 0; idx < reads_max; idx++) {
622
+ i = reads[idx];
623
+ pi = &p.inf[i];
624
+ pf = &p.fds[i];
625
+ pf->revents = 0;
626
+ processed += poll_process_tcp_read(&p, pi, pf, now);
627
+ }
628
+
629
+ if(!processed && (!p.limit || p.used < p.limit)) {
630
+ // nothing processed above (rcv, snd) and we have room for another TCP connection
631
+ // so, accept one TCP connection
632
+ for (idx = 0; idx < conns_max; idx++) {
633
+ i = conns[idx];
634
+ pi = &p.inf[i];
635
+ pf = &p.fds[i];
636
+ pf->revents = 0;
637
+ if (poll_process_new_tcp_connection(&p, pi, pf, now))
638
+ break;
639
+ }
640
+ }
641
+ }
642
+
643
+ if(unlikely(p.checks_every > 0 && now - last_check > p.checks_every)) {
644
+ last_check = now;
645
+
646
+ // cleanup old sockets
647
+ for(i = 0; i <= p.max; i++) {
648
+ POLLINFO *pi = &p.inf[i];
649
+
650
+ if(likely(pi->flags & POLLINFO_FLAG_CLIENT_SOCKET)) {
651
+ if (unlikely(pi->send_count == 0 && p.complete_request_timeout > 0 && (now - pi->connected_t) >= p.complete_request_timeout)) {
652
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
653
+ "POLLFD: LISTENER: client slot %zu (fd %d) from %s port %s has not sent a complete request in %zu seconds - closing it. "
654
+ , i
655
+ , pi->fd
656
+ , pi->client_ip ? pi->client_ip : "<undefined-ip>"
657
+ , pi->client_port ? pi->client_port : "<undefined-port>"
658
+ , (size_t) p.complete_request_timeout
659
+ );
660
+ poll_close_fd(pi);
661
+ }
662
+ else if(unlikely(pi->recv_count && p.idle_timeout > 0 && now - ((pi->last_received_t > pi->last_sent_t) ? pi->last_received_t : pi->last_sent_t) >= p.idle_timeout )) {
663
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
664
+ "POLLFD: LISTENER: client slot %zu (fd %d) from %s port %s is idle for more than %zu seconds - closing it. "
665
+ , i
666
+ , pi->fd
667
+ , pi->client_ip ? pi->client_ip : "<undefined-ip>"
668
+ , pi->client_port ? pi->client_port : "<undefined-port>"
669
+ , (size_t) p.idle_timeout
670
+ );
671
+ poll_close_fd(pi);
672
+ }
673
+ }
674
+ }
675
+ }
676
+ }
677
+}
src/libnetdata/socket/poll-events.h
new
+116
@@ -0,0 +1,116 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_POLL_EVENTS_H
4
+#define NETDATA_POLL_EVENTS_H
5
+
6
+#define POLLINFO_FLAG_SERVER_SOCKET 0x00000001
7
+#define POLLINFO_FLAG_CLIENT_SOCKET 0x00000002
8
+#define POLLINFO_FLAG_DONT_CLOSE 0x00000004
9
+
10
+typedef struct poll POLLJOB;
11
+
12
+typedef struct pollinfo {
13
+ POLLJOB *p; // the parent
14
+ size_t slot; // the slot id
15
+
16
+ int fd; // the file descriptor
17
+ int socktype; // the client socket type
18
+ HTTP_ACL port_acl; // the access lists permitted on this web server port (it's -1 for client sockets)
19
+ char *client_ip; // Max INET6_ADDRSTRLEN bytes
20
+ char *client_port; // Max NI_MAXSERV bytes
21
+ char *client_host; // Max NI_MAXHOST bytes
22
+
23
+ time_t connected_t; // the time the socket connected
24
+ time_t last_received_t; // the time the socket last received data
25
+ time_t last_sent_t; // the time the socket last sent data
26
+
27
+ size_t recv_count; // the number of times the socket was ready for inbound traffic
28
+ size_t send_count; // the number of times the socket was ready for outbound traffic
29
+
30
+ uint32_t flags; // internal flags
31
+
32
+ // callbacks for this socket
33
+ void (*del_callback)(struct pollinfo *pi);
34
+ int (*rcv_callback)(struct pollinfo *pi, short int *events);
35
+ int (*snd_callback)(struct pollinfo *pi, short int *events);
36
+
37
+ // the user data
38
+ void *data;
39
+
40
+ // linking of free pollinfo structures
41
+ // for quickly finding the next available
42
+ // this is like a stack, it grows and shrinks
43
+ // (with gaps - lower empty slots are preferred)
44
+ struct pollinfo *next;
45
+} POLLINFO;
46
+
47
+struct poll {
48
+ size_t slots;
49
+ size_t used;
50
+ size_t min;
51
+ size_t max;
52
+
53
+ size_t limit;
54
+
55
+ time_t complete_request_timeout;
56
+ time_t idle_timeout;
57
+ time_t checks_every;
58
+
59
+ time_t timer_milliseconds;
60
+ void *timer_data;
61
+
62
+ struct pollfd *fds;
63
+ struct pollinfo *inf;
64
+ struct pollinfo *first_free;
65
+
66
+ SIMPLE_PATTERN *access_list;
67
+ int allow_dns;
68
+
69
+ void *(*add_callback)(POLLINFO *pi, short int *events, void *data);
70
+ void (*del_callback)(POLLINFO *pi);
71
+ int (*rcv_callback)(POLLINFO *pi, short int *events);
72
+ int (*snd_callback)(POLLINFO *pi, short int *events);
73
+ void (*tmr_callback)(void *timer_data);
74
+};
75
+
76
+#define pollinfo_from_slot(p, slot) (&((p)->inf[(slot)]))
77
+
78
+int poll_default_snd_callback(POLLINFO *pi, short int *events);
79
+int poll_default_rcv_callback(POLLINFO *pi, short int *events);
80
+void poll_default_del_callback(POLLINFO *pi);
81
+void *poll_default_add_callback(POLLINFO *pi, short int *events, void *data);
82
+
83
+POLLINFO *poll_add_fd(POLLJOB *p
84
+ , int fd
85
+ , int socktype
86
+ , HTTP_ACL port_acl
87
+ , uint32_t flags
88
+ , const char *client_ip
89
+ , const char *client_port
90
+ , const char *client_host
91
+ , void *(*add_callback)(POLLINFO *pi, short int *events, void *data)
92
+ , void (*del_callback)(POLLINFO *pi)
93
+ , int (*rcv_callback)(POLLINFO *pi, short int *events)
94
+ , int (*snd_callback)(POLLINFO *pi, short int *events)
95
+ , void *data
96
+);
97
+void poll_close_fd(POLLINFO *pi);
98
+
99
+void poll_events(LISTEN_SOCKETS *sockets
100
+ , void *(*add_callback)(POLLINFO *pi, short int *events, void *data)
101
+ , void (*del_callback)(POLLINFO *pi)
102
+ , int (*rcv_callback)(POLLINFO *pi, short int *events)
103
+ , int (*snd_callback)(POLLINFO *pi, short int *events)
104
+ , void (*tmr_callback)(void *timer_data)
105
+ , bool (*check_to_stop_callback)(void)
106
+ , SIMPLE_PATTERN *access_list
107
+ , int allow_dns
108
+ , void *data
109
+ , time_t tcp_request_timeout_seconds
110
+ , time_t tcp_idle_timeout_seconds
111
+ , time_t timer_milliseconds
112
+ , void *timer_data
113
+ , size_t max_tcp_sockets
114
+);
115
+
116
+#endif //NETDATA_POLL_EVENTS_H
src/libnetdata/socket/security.c
+3
@@ -731,6 +731,9 @@ int security_test_certificate(SSL *ssl) {
731
* @return It returns 0 on success and -1 otherwise.
732
*/
733
int ssl_security_location_for_context(SSL_CTX *ctx, const char *file, const char *path) {
734
+ if(file && !*file) file = NULL;
735
+ if(path && !*path) path = NULL;
736
+
737
int load_custom = 1, load_default = 1;
738
if (file || path) {
739
if(!SSL_CTX_load_verify_locations(ctx, file, path)) {
src/libnetdata/socket/socket-peers.c
new
+58
@@ -0,0 +1,58 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "libnetdata/libnetdata.h"
4
+
5
+SOCKET_PEERS socket_peers(int sock_fd) {
6
+ SOCKET_PEERS peers;
7
+
8
+ if(sock_fd < 0) {
9
+ strncpyz(peers.peer.ip, "not connected", sizeof(peers.peer.ip) - 1);
10
+ peers.peer.port = 0;
11
+
12
+ strncpyz(peers.local.ip, "not connected", sizeof(peers.local.ip) - 1);
13
+ peers.local.port = 0;
14
+
15
+ return peers;
16
+ }
17
+
18
+ struct sockaddr_storage addr;
19
+ socklen_t addr_len = sizeof(addr);
20
+
21
+ // Get peer info
22
+ if (getpeername(sock_fd, (struct sockaddr *)&addr, &addr_len) == 0) {
23
+ if (addr.ss_family == AF_INET) { // IPv4
24
+ struct sockaddr_in *s = (struct sockaddr_in *)&addr;
25
+ inet_ntop(AF_INET, &s->sin_addr, peers.peer.ip, sizeof(peers.peer.ip));
26
+ peers.peer.port = ntohs(s->sin_port);
27
+ }
28
+ else { // IPv6
29
+ struct sockaddr_in6 *s = (struct sockaddr_in6 *)&addr;
30
+ inet_ntop(AF_INET6, &s->sin6_addr, peers.peer.ip, sizeof(peers.peer.ip));
31
+ peers.peer.port = ntohs(s->sin6_port);
32
+ }
33
+ }
34
+ else {
35
+ strncpyz(peers.peer.ip, "unknown", sizeof(peers.peer.ip) - 1);
36
+ peers.peer.port = 0;
37
+ }
38
+
39
+ // Get local info
40
+ addr_len = sizeof(addr);
41
+ if (getsockname(sock_fd, (struct sockaddr *)&addr, &addr_len) == 0) {
42
+ if (addr.ss_family == AF_INET) { // IPv4
43
+ struct sockaddr_in *s = (struct sockaddr_in *) &addr;
44
+ inet_ntop(AF_INET, &s->sin_addr, peers.local.ip, sizeof(peers.local.ip));
45
+ peers.local.port = ntohs(s->sin_port);
46
+ } else { // IPv6
47
+ struct sockaddr_in6 *s = (struct sockaddr_in6 *) &addr;
48
+ inet_ntop(AF_INET6, &s->sin6_addr, peers.local.ip, sizeof(peers.local.ip));
49
+ peers.local.port = ntohs(s->sin6_port);
50
+ }
51
+ }
52
+ else {
53
+ strncpyz(peers.local.ip, "unknown", sizeof(peers.local.ip) - 1);
54
+ peers.local.port = 0;
55
+ }
56
+
57
+ return peers;
58
+}
src/libnetdata/socket/socket-peers.h
new
+24
@@ -0,0 +1,24 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_SOCKET_PEERS_H
4
+#define NETDATA_SOCKET_PEERS_H
5
+
6
+#ifndef INET6_ADDRSTRLEN
7
+#define INET6_ADDRSTRLEN 46
8
+#endif
9
+
10
+typedef struct {
11
+ struct {
12
+ char ip[INET6_ADDRSTRLEN];
13
+ int port;
14
+ } local;
15
+
16
+ struct {
17
+ char ip[INET6_ADDRSTRLEN];
18
+ int port;
19
+ } peer;
20
+} SOCKET_PEERS;
21
+
22
+SOCKET_PEERS socket_peers(int sock_fd);
23
+
24
+#endif //NETDATA_SOCKET_PEERS_H
src/libnetdata/socket/socket.c
+65
-1759
@@ -45,62 +45,6 @@ bool ip_to_hostname(const char *ip, char *dst, size_t dst_len) {
45
return true;
46
}
47
48
-SOCKET_PEERS socket_peers(int sock_fd) {
49
- SOCKET_PEERS peers;
50
-
51
- if(sock_fd < 0) {
52
- strncpyz(peers.peer.ip, "not connected", sizeof(peers.peer.ip) - 1);
53
- peers.peer.port = 0;
54
-
55
- strncpyz(peers.local.ip, "not connected", sizeof(peers.local.ip) - 1);
56
- peers.local.port = 0;
57
-
58
- return peers;
59
- }
60
-
61
- struct sockaddr_storage addr;
62
- socklen_t addr_len = sizeof(addr);
63
-
64
- // Get peer info
65
- if (getpeername(sock_fd, (struct sockaddr *)&addr, &addr_len) == 0) {
66
- if (addr.ss_family == AF_INET) { // IPv4
67
- struct sockaddr_in *s = (struct sockaddr_in *)&addr;
68
- inet_ntop(AF_INET, &s->sin_addr, peers.peer.ip, sizeof(peers.peer.ip));
69
- peers.peer.port = ntohs(s->sin_port);
70
- }
71
- else { // IPv6
72
- struct sockaddr_in6 *s = (struct sockaddr_in6 *)&addr;
73
- inet_ntop(AF_INET6, &s->sin6_addr, peers.peer.ip, sizeof(peers.peer.ip));
74
- peers.peer.port = ntohs(s->sin6_port);
75
- }
76
- }
77
- else {
78
- strncpyz(peers.peer.ip, "unknown", sizeof(peers.peer.ip) - 1);
79
- peers.peer.port = 0;
80
- }
81
-
82
- // Get local info
83
- addr_len = sizeof(addr);
84
- if (getsockname(sock_fd, (struct sockaddr *)&addr, &addr_len) == 0) {
85
- if (addr.ss_family == AF_INET) { // IPv4
86
- struct sockaddr_in *s = (struct sockaddr_in *) &addr;
87
- inet_ntop(AF_INET, &s->sin_addr, peers.local.ip, sizeof(peers.local.ip));
88
- peers.local.port = ntohs(s->sin_port);
89
- } else { // IPv6
90
- struct sockaddr_in6 *s = (struct sockaddr_in6 *) &addr;
91
- inet_ntop(AF_INET6, &s->sin6_addr, peers.local.ip, sizeof(peers.local.ip));
92
- peers.local.port = ntohs(s->sin6_port);
93
- }
94
- }
95
- else {
96
- strncpyz(peers.local.ip, "unknown", sizeof(peers.local.ip) - 1);
97
- peers.local.port = 0;
98
- }
99
-
100
- return peers;
101
-}
102
-
103
-
48
// --------------------------------------------------------------------------------------------------------------------
49
// various library calls
50
@@ -174,1028 +118,92 @@ int sock_setnonblock(int fd) {
118
int flags;
119
120
flags = fcntl(fd, F_GETFL);
177
- flags |= O_NONBLOCK;
178
-
179
- int ret = fcntl(fd, F_SETFL, flags);
180
- if(ret < 0)
181
- nd_log(NDLS_DAEMON, NDLP_ERR,
182
- "Failed to set O_NONBLOCK on socket %d",
183
- fd);
184
-
185
- return ret;
186
-}
187
-
188
-int sock_delnonblock(int fd) {
189
- int flags;
190
-
191
- flags = fcntl(fd, F_GETFL);
192
- flags &= ~O_NONBLOCK;
193
-
194
- int ret = fcntl(fd, F_SETFL, flags);
195
- if(ret < 0)
196
- nd_log(NDLS_DAEMON, NDLP_ERR,
197
- "Failed to remove O_NONBLOCK on socket %d",
198
- fd);
199
-
200
- return ret;
201
-}
202
-
203
-int sock_setreuse(int fd, int reuse) {
204
- int ret = setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
205
-
206
- if(ret == -1)
207
- nd_log(NDLS_DAEMON, NDLP_ERR,
208
- "Failed to set SO_REUSEADDR on socket %d",
209
- fd);
210
-
211
- return ret;
212
-}
213
-
214
-void sock_setcloexec(int fd)
215
-{
216
- UNUSED(fd);
217
- int flags = fcntl(fd, F_GETFD);
218
- if (flags != -1)
219
- (void) fcntl(fd, F_SETFD, flags | FD_CLOEXEC);
220
-}
221
-
222
-int sock_setreuse_port(int fd __maybe_unused, int reuse __maybe_unused) {
223
- int ret;
224
-
225
-#ifdef SO_REUSEPORT
226
- ret = setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &reuse, sizeof(reuse));
227
- if(ret == -1 && errno != ENOPROTOOPT)
228
- nd_log(NDLS_DAEMON, NDLP_ERR,
229
- "failed to set SO_REUSEPORT on socket %d",
230
- fd);
231
-#else
232
- ret = -1;
233
-#endif
234
-
235
- return ret;
236
-}
237
-
238
-int sock_enlarge_in(int fd) {
239
- int ret, bs = LARGE_SOCK_SIZE;
240
-
241
- ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &bs, sizeof(bs));
242
-
243
- if(ret == -1)
244
- nd_log(NDLS_DAEMON, NDLP_ERR,
245
- "Failed to set SO_RCVBUF on socket %d",
246
- fd);
247
-
248
- return ret;
249
-}
250
-
251
-int sock_enlarge_out(int fd) {
252
- int ret, bs = LARGE_SOCK_SIZE;
253
- ret = setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &bs, sizeof(bs));
254
-
255
- if(ret == -1)
256
- nd_log(NDLS_DAEMON, NDLP_ERR,
257
- "Failed to set SO_SNDBUF on socket %d",
258
- fd);
259
-
260
- return ret;
261
-}
262
-
263
-
264
-// --------------------------------------------------------------------------------------------------------------------
265
-
266
-char *strdup_client_description(int family, const char *protocol, const char *ip, uint16_t port) {
267
- char buffer[100 + 1];
268
-
269
- switch(family) {
270
- case AF_INET:
271
- snprintfz(buffer, sizeof(buffer) - 1, "%s:%s:%d", protocol, ip, port);
272
- break;
273
-
274
- case AF_INET6:
275
- default:
276
- snprintfz(buffer, sizeof(buffer) - 1, "%s:[%s]:%d", protocol, ip, port);
277
- break;
278
-
279
- case AF_UNIX:
280
- snprintfz(buffer, sizeof(buffer) - 1, "%s:%s", protocol, ip);
281
- break;
282
- }
283
-
284
- return strdupz(buffer);
285
-}
286
-
287
-// --------------------------------------------------------------------------------------------------------------------
288
-// listening sockets
289
-
290
-int create_listen_socket_unix(const char *path, int listen_backlog) {
291
- int sock;
292
-
293
- sock = socket(AF_UNIX, SOCK_STREAM | DEFAULT_SOCKET_FLAGS, 0);
294
- if(sock < 0) {
295
- nd_log(NDLS_DAEMON, NDLP_ERR,
296
- "LISTENER: UNIX socket() on path '%s' failed.",
297
- path);
298
-
299
- return -1;
300
- }
301
-
302
- sock_setnonblock(sock);
303
- sock_setcloexec(sock);
304
- sock_enlarge_in(sock);
305
-
306
- struct sockaddr_un name;
307
- memset(&name, 0, sizeof(struct sockaddr_un));
308
- name.sun_family = AF_UNIX;
309
- strncpy(name.sun_path, path, sizeof(name.sun_path)-1);
310
-
311
- errno_clear();
312
- if (unlink(path) == -1 && errno != ENOENT)
313
- nd_log(NDLS_DAEMON, NDLP_ERR,
314
- "LISTENER: failed to remove existing (probably obsolete or left-over) file on UNIX socket path '%s'.",
315
- path);
316
-
317
- if(bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
318
- close(sock);
319
- nd_log(NDLS_DAEMON, NDLP_ERR,
320
- "LISTENER: UNIX bind() on path '%s' failed.",
321
- path);
322
-
323
- return -1;
324
- }
325
-
326
- // we have to chmod this to 0777 so that the client will be able
327
- // to read from and write to this socket.
328
- if(chmod(path, 0777) == -1)
329
- nd_log(NDLS_DAEMON, NDLP_ERR,
330
- "LISTENER: failed to chmod() socket file '%s'.",
331
- path);
332
-
333
- if(listen(sock, listen_backlog) < 0) {
334
- close(sock);
335
- nd_log(NDLS_DAEMON, NDLP_ERR,
336
- "LISTENER: UNIX listen() on path '%s' failed.",
337
- path);
338
-
339
- return -1;
340
- }
341
-
342
- return sock;
343
-}
344
-
345
-int create_listen_socket4(int socktype, const char *ip, uint16_t port, int listen_backlog) {
346
- int sock;
347
-
348
- sock = socket(AF_INET, socktype | DEFAULT_SOCKET_FLAGS, 0);
349
- if(sock < 0) {
350
- nd_log(NDLS_DAEMON, NDLP_ERR,
351
- "LISTENER: IPv4 socket() on ip '%s' port %d, socktype %d failed.",
352
- ip, port, socktype);
353
-
354
- return -1;
355
- }
356
- sock_setreuse(sock, 1);
357
- sock_setreuse_port(sock, 0);
358
- sock_setnonblock(sock);
359
- sock_setcloexec(sock);
360
- sock_enlarge_in(sock);
361
-
362
- struct sockaddr_in name;
363
- memset(&name, 0, sizeof(struct sockaddr_in));
364
- name.sin_family = AF_INET;
365
- name.sin_port = htons (port);
366
-
367
- int ret = inet_pton(AF_INET, ip, (void *)&name.sin_addr.s_addr);
368
- if(ret != 1) {
369
- nd_log(NDLS_DAEMON, NDLP_ERR,
370
- "LISTENER: Failed to convert IP '%s' to a valid IPv4 address.",
371
- ip);
372
-
373
- close(sock);
374
- return -1;
375
- }
376
-
377
- if(bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
378
- close(sock);
379
- nd_log(NDLS_DAEMON, NDLP_ERR,
380
- "LISTENER: IPv4 bind() on ip '%s' port %d, socktype %d failed.",
381
- ip, port, socktype);
382
-
383
- return -1;
384
- }
385
-
386
- if(socktype == SOCK_STREAM && listen(sock, listen_backlog) < 0) {
387
- close(sock);
388
- nd_log(NDLS_DAEMON, NDLP_ERR,
389
- "LISTENER: IPv4 listen() on ip '%s' port %d, socktype %d failed.",
390
- ip, port, socktype);
391
-
392
- return -1;
393
- }
394
-
395
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
396
- "LISTENER: Listening on IPv4 ip '%s' port %d, socktype %d",
397
- ip, port, socktype);
398
-
399
- return sock;
400
-}
401
-
402
-int create_listen_socket6(int socktype, uint32_t scope_id, const char *ip, int port, int listen_backlog) {
403
- int sock;
404
- int ipv6only = 1;
405
-
406
- sock = socket(AF_INET6, socktype | DEFAULT_SOCKET_FLAGS, 0);
407
- if (sock < 0) {
408
- nd_log(NDLS_DAEMON, NDLP_ERR,
409
- "LISTENER: IPv6 socket() on ip '%s' port %d, socktype %d, failed.",
410
- ip, port, socktype);
411
-
412
- return -1;
413
- }
414
- sock_setreuse(sock, 1);
415
- sock_setreuse_port(sock, 0);
416
- sock_setnonblock(sock);
417
- sock_setcloexec(sock);
418
- sock_enlarge_in(sock);
419
-
420
- /* IPv6 only */
421
- if(setsockopt(sock, IPPROTO_IPV6, IPV6_V6ONLY, (void*)&ipv6only, sizeof(ipv6only)) != 0)
422
- nd_log(NDLS_DAEMON, NDLP_ERR,
423
- "LISTENER: Cannot set IPV6_V6ONLY on ip '%s' port %d, socktype %d.",
424
- ip, port, socktype);
425
-
426
- struct sockaddr_in6 name;
427
- memset(&name, 0, sizeof(struct sockaddr_in6));
428
- name.sin6_family = AF_INET6;
429
- name.sin6_port = htons ((uint16_t) port);
430
- name.sin6_scope_id = scope_id;
431
-
432
- int ret = inet_pton(AF_INET6, ip, (void *)&name.sin6_addr.s6_addr);
433
- if(ret != 1) {
434
- nd_log(NDLS_DAEMON, NDLP_ERR,
435
- "LISTENER: Failed to convert IP '%s' to a valid IPv6 address.",
436
- ip);
437
-
438
- close(sock);
439
- return -1;
440
- }
441
-
442
- name.sin6_scope_id = scope_id;
443
-
444
- if (bind (sock, (struct sockaddr *) &name, sizeof (name)) < 0) {
445
- close(sock);
446
- nd_log(NDLS_DAEMON, NDLP_ERR,
447
- "LISTENER: IPv6 bind() on ip '%s' port %d, socktype %d failed.",
448
- ip, port, socktype);
449
-
450
- return -1;
451
- }
452
-
453
- if (socktype == SOCK_STREAM && listen(sock, listen_backlog) < 0) {
454
- close(sock);
455
- nd_log(NDLS_DAEMON, NDLP_ERR,
456
- "LISTENER: IPv6 listen() on ip '%s' port %d, socktype %d failed.",
457
- ip, port, socktype);
458
-
459
- return -1;
460
- }
461
-
462
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
463
- "LISTENER: Listening on IPv6 ip '%s' port %d, socktype %d",
464
- ip, port, socktype);
465
-
466
- return sock;
467
-}
468
-
469
-static inline int listen_sockets_add(LISTEN_SOCKETS *sockets, int fd, int family, int socktype, const char *protocol, const char *ip, uint16_t port, int acl_flags) {
470
- if(sockets->opened >= MAX_LISTEN_FDS) {
471
- nd_log(NDLS_DAEMON, NDLP_ERR,
472
- "LISTENER: Too many listening sockets. Failed to add listening %s socket at ip '%s' port %d, protocol %s, socktype %d",
473
- protocol, ip, port, protocol, socktype);
474
-
475
- close(fd);
476
- return -1;
477
- }
478
-
479
- sockets->fds[sockets->opened] = fd;
480
- sockets->fds_types[sockets->opened] = socktype;
481
- sockets->fds_families[sockets->opened] = family;
482
- sockets->fds_names[sockets->opened] = strdup_client_description(family, protocol, ip, port);
483
- sockets->fds_acl_flags[sockets->opened] = acl_flags;
484
-
485
- sockets->opened++;
486
- return 0;
487
-}
488
-
489
-int listen_sockets_check_is_member(LISTEN_SOCKETS *sockets, int fd) {
490
- size_t i;
491
- for(i = 0; i < sockets->opened ;i++)
492
- if(sockets->fds[i] == fd) return 1;
493
-
494
- return 0;
495
-}
496
-
497
-static inline void listen_sockets_init(LISTEN_SOCKETS *sockets) {
498
- size_t i;
499
- for(i = 0; i < MAX_LISTEN_FDS ;i++) {
500
- sockets->fds[i] = -1;
501
- sockets->fds_names[i] = NULL;
502
- sockets->fds_types[i] = -1;
503
- }
504
-
505
- sockets->opened = 0;
506
- sockets->failed = 0;
507
-}
508
-
509
-void listen_sockets_close(LISTEN_SOCKETS *sockets) {
510
- size_t i;
511
- for(i = 0; i < sockets->opened ;i++) {
512
- close(sockets->fds[i]);
513
- sockets->fds[i] = -1;
514
-
515
- freez(sockets->fds_names[i]);
516
- sockets->fds_names[i] = NULL;
517
-
518
- sockets->fds_types[i] = -1;
519
- }
520
-
521
- sockets->opened = 0;
522
- sockets->failed = 0;
523
-}
524
-
525
-/*
526
- * SSL ACL
527
- *
528
- * Search the SSL acl and apply it case it is set.
529
- *
530
- * @param acl is the acl given by the user.
531
- */
532
-HTTP_ACL socket_ssl_acl(char *acl) {
533
- char *ssl = strchr(acl,'^');
534
- if(ssl) {
535
- //Due the format of the SSL command it is always the last command,
536
- //we finish it here to avoid problems with the ACLs
537
- *ssl = '\0';
538
- ssl++;
539
- if (!strncmp("SSL=",ssl,4)) {
540
- ssl += 4;
541
- if (!strcmp(ssl,"optional")) {
542
- return HTTP_ACL_SSL_OPTIONAL;
543
- }
544
- else if (!strcmp(ssl,"force")) {
545
- return HTTP_ACL_SSL_FORCE;
546
- }
547
- }
548
- }
549
-
550
- return HTTP_ACL_NONE;
551
-}
552
-
553
-HTTP_ACL read_acl(char *st) {
554
- HTTP_ACL ret = socket_ssl_acl(st);
555
-
556
- if (!strcmp(st,"dashboard")) ret |= HTTP_ACL_DASHBOARD;
557
- if (!strcmp(st,"registry")) ret |= HTTP_ACL_REGISTRY;
558
- if (!strcmp(st,"badges")) ret |= HTTP_ACL_BADGES;
559
- if (!strcmp(st,"management")) ret |= HTTP_ACL_MANAGEMENT;
560
- if (!strcmp(st,"streaming")) ret |= HTTP_ACL_STREAMING;
561
- if (!strcmp(st,"netdata.conf")) ret |= HTTP_ACL_NETDATACONF;
562
-
563
- return ret;
564
-}
565
-
566
-static inline int bind_to_this(LISTEN_SOCKETS *sockets, const char *definition, uint16_t default_port, int listen_backlog) {
567
- int added = 0;
568
- HTTP_ACL acl_flags = HTTP_ACL_NONE;
569
-
570
- struct addrinfo hints;
571
- struct addrinfo *result = NULL, *rp = NULL;
572
-
573
- char buffer[strlen(definition) + 1];
574
- strcpy(buffer, definition);
575
-
576
- char buffer2[10 + 1];
577
- snprintfz(buffer2, 10, "%d", default_port);
578
-
579
- char *ip = buffer, *port = buffer2, *iface = "", *portconfig;
580
-
581
- int protocol = IPPROTO_TCP, socktype = SOCK_STREAM;
582
- const char *protocol_str = "tcp";
583
-
584
- if(strncmp(ip, "tcp:", 4) == 0) {
585
- ip += 4;
586
- protocol = IPPROTO_TCP;
587
- socktype = SOCK_STREAM;
588
- protocol_str = "tcp";
589
- acl_flags |= HTTP_ACL_API;
590
- }
591
- else if(strncmp(ip, "udp:", 4) == 0) {
592
- ip += 4;
593
- protocol = IPPROTO_UDP;
594
- socktype = SOCK_DGRAM;
595
- protocol_str = "udp";
596
- acl_flags |= HTTP_ACL_API_UDP;
597
- }
598
- else if(strncmp(ip, "unix:", 5) == 0) {
599
- char *path = ip + 5;
600
- socktype = SOCK_STREAM;
601
- protocol_str = "unix";
602
- int fd = create_listen_socket_unix(path, listen_backlog);
603
- if (fd == -1) {
604
- nd_log(NDLS_DAEMON, NDLP_ERR,
605
- "LISTENER: Cannot create unix socket '%s'",
606
- path);
607
-
608
- sockets->failed++;
609
- } else {
610
- acl_flags = HTTP_ACL_API_UNIX | HTTP_ACL_DASHBOARD | HTTP_ACL_REGISTRY | HTTP_ACL_BADGES |
611
- HTTP_ACL_MANAGEMENT | HTTP_ACL_NETDATACONF | HTTP_ACL_STREAMING | HTTP_ACL_SSL_DEFAULT;
612
- listen_sockets_add(sockets, fd, AF_UNIX, socktype, protocol_str, path, 0, acl_flags);
613
- added++;
614
- }
615
- return added;
616
- }
617
-
618
- char *e = ip;
619
- if(*e == '[') {
620
- e = ++ip;
621
- while(*e && *e != ']') e++;
622
- if(*e == ']') {
623
- *e = '\0';
624
- e++;
625
- }
626
- }
627
- else {
628
- while(*e && *e != ':' && *e != '%' && *e != '=') e++;
629
- }
630
-
631
- if(*e == '%') {
632
- *e = '\0';
633
- e++;
634
- iface = e;
635
- while(*e && *e != ':' && *e != '=') e++;
636
- }
637
-
638
- if(*e == ':') {
639
- port = e + 1;
640
- *e = '\0';
641
- e++;
642
- while(*e && *e != '=') e++;
643
- }
644
-
645
- if(*e == '=') {
646
- *e='\0';
647
- e++;
648
- portconfig = e;
649
- while (*e != '\0') {
650
- if (*e == '|') {
651
- *e = '\0';
652
- acl_flags |= read_acl(portconfig);
653
- e++;
654
- portconfig = e;
655
- continue;
656
- }
657
- e++;
658
- }
659
- acl_flags |= read_acl(portconfig);
660
- } else {
661
- acl_flags |= HTTP_ACL_DASHBOARD | HTTP_ACL_REGISTRY | HTTP_ACL_BADGES | HTTP_ACL_MANAGEMENT | HTTP_ACL_NETDATACONF | HTTP_ACL_STREAMING | HTTP_ACL_SSL_DEFAULT;
662
- }
663
-
664
- //Case the user does not set the option SSL in the "bind to", but he has
665
- //the certificates, I must redirect, so I am assuming here the default option
666
- if(!(acl_flags & HTTP_ACL_SSL_OPTIONAL) && !(acl_flags & HTTP_ACL_SSL_FORCE)) {
667
- acl_flags |= HTTP_ACL_SSL_DEFAULT;
668
- }
669
-
670
- uint32_t scope_id = 0;
671
- if(*iface) {
672
- scope_id = if_nametoindex(iface);
673
- if(!scope_id)
674
- nd_log(NDLS_DAEMON, NDLP_ERR,
675
- "LISTENER: Cannot find a network interface named '%s'. "
676
- "Continuing with limiting the network interface",
677
- iface);
678
- }
679
-
680
- if(!*ip || *ip == '*' || !strcmp(ip, "any") || !strcmp(ip, "all"))
681
- ip = NULL;
682
-
683
- if(!*port)
684
- port = buffer2;
685
-
686
- memset(&hints, 0, sizeof(hints));
687
- hints.ai_family = AF_UNSPEC; /* Allow IPv4 or IPv6 */
688
- hints.ai_socktype = socktype;
689
- hints.ai_flags = AI_PASSIVE; /* For wildcard IP address */
690
- hints.ai_protocol = protocol;
691
- hints.ai_canonname = NULL;
692
- hints.ai_addr = NULL;
693
- hints.ai_next = NULL;
694
-
695
- int r = getaddrinfo(ip, port, &hints, &result);
696
- if (r != 0) {
697
- nd_log(NDLS_DAEMON, NDLP_ERR,
698
- "LISTENER: getaddrinfo('%s', '%s'): %s\n",
699
- ip, port, gai_strerror(r));
700
-
701
- return -1;
702
- }
703
-
704
- for (rp = result; rp != NULL; rp = rp->ai_next) {
705
- int fd = -1;
706
- int family;
707
-
708
- char rip[INET_ADDRSTRLEN + INET6_ADDRSTRLEN] = "INVALID";
709
- uint16_t rport = default_port;
710
-
711
- family = rp->ai_addr->sa_family;
712
- switch (family) {
713
- case AF_INET: {
714
- struct sockaddr_in *sin = (struct sockaddr_in *) rp->ai_addr;
715
- inet_ntop(AF_INET, &sin->sin_addr, rip, INET_ADDRSTRLEN);
716
- rport = ntohs(sin->sin_port);
717
- fd = create_listen_socket4(socktype, rip, rport, listen_backlog);
718
- break;
719
- }
720
-
721
- case AF_INET6: {
722
- struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *) rp->ai_addr;
723
- inet_ntop(AF_INET6, &sin6->sin6_addr, rip, INET6_ADDRSTRLEN);
724
- rport = ntohs(sin6->sin6_port);
725
- fd = create_listen_socket6(socktype, scope_id, rip, rport, listen_backlog);
726
- break;
727
- }
728
-
729
- default:
730
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
731
- "LISTENER: Unknown socket family %d",
732
- family);
733
-
734
- break;
735
- }
736
-
737
- if (fd == -1) {
738
- nd_log(NDLS_DAEMON, NDLP_ERR,
739
- "LISTENER: Cannot bind to ip '%s', port %d",
740
- rip, rport);
741
-
742
- sockets->failed++;
743
- }
744
- else {
745
- listen_sockets_add(sockets, fd, family, socktype, protocol_str, rip, rport, acl_flags);
746
- added++;
747
- }
748
- }
749
-
750
- freeaddrinfo(result);
751
-
752
- return added;
753
-}
754
-
755
-int listen_sockets_setup(LISTEN_SOCKETS *sockets) {
756
- listen_sockets_init(sockets);
757
-
758
- sockets->backlog = (int) appconfig_get_number(sockets->config, sockets->config_section, "listen backlog", sockets->backlog);
759
-
760
- long long int old_port = sockets->default_port;
761
- long long int new_port = appconfig_get_number(sockets->config, sockets->config_section, "default port", sockets->default_port);
762
- if(new_port < 1 || new_port > 65535) {
763
- nd_log(NDLS_DAEMON, NDLP_ERR,
764
- "LISTENER: Invalid listen port %lld given. Defaulting to %lld.",
765
- new_port, old_port);
766
-
767
- sockets->default_port = (uint16_t) appconfig_set_number(sockets->config, sockets->config_section, "default port", old_port);
768
- }
769
- else sockets->default_port = (uint16_t)new_port;
770
-
771
- const char *s = appconfig_get(sockets->config, sockets->config_section, "bind to", sockets->default_bind_to);
772
- while(*s) {
773
- const char *e = s;
774
-
775
- // skip separators, moving both s(tart) and e(nd)
776
- while(isspace((uint8_t)*e) || *e == ',') s = ++e;
777
-
778
- // move e(nd) to the first separator
779
- while(*e && !isspace((uint8_t)*e) && *e != ',') e++;
780
-
781
- // is there anything?
782
- if(!*s || s == e) break;
783
-
784
- char buf[e - s + 1];
785
- strncpyz(buf, s, e - s);
786
- bind_to_this(sockets, buf, sockets->default_port, sockets->backlog);
787
-
788
- s = e;
789
- }
790
-
791
- if(sockets->failed) {
792
- size_t i;
793
- for(i = 0; i < sockets->opened ;i++)
794
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
795
- "LISTENER: Listen socket %s opened successfully.",
796
- sockets->fds_names[i]);
797
- }
798
-
799
- return (int)sockets->opened;
800
-}
801
-
802
-
803
-// --------------------------------------------------------------------------------------------------------------------
804
-// connect to another host/port
805
-
806
-// connect_to_this_unix()
807
-// path the path of the unix socket
808
-// timeout the timeout for establishing a connection
809
-
810
-static inline int connect_to_unix(const char *path, struct timeval *timeout) {
811
- int fd = socket(AF_UNIX, SOCK_STREAM | DEFAULT_SOCKET_FLAGS, 0);
812
- if(fd == -1) {
813
- nd_log(NDLS_DAEMON, NDLP_ERR,
814
- "Failed to create UNIX socket() for '%s'",
815
- path);
816
-
817
- return -1;
818
- }
819
-
820
- if(timeout) {
821
- if(setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, (char *) timeout, sizeof(struct timeval)) < 0)
822
- nd_log(NDLS_DAEMON, NDLP_ERR,
823
- "Failed to set timeout on UNIX socket '%s'",
824
- path);
825
- }
826
-
827
- sock_setcloexec(fd);
828
-
829
- struct sockaddr_un addr;
830
- memset(&addr, 0, sizeof(addr));
831
- addr.sun_family = AF_UNIX;
832
- strncpy(addr.sun_path, path, sizeof(addr.sun_path)-1);
833
-
834
- if (connect(fd, (struct sockaddr*)&addr, sizeof(addr)) == -1) {
835
- nd_log(NDLS_DAEMON, NDLP_ERR,
836
- "Cannot connect to UNIX socket on path '%s'.",
837
- path);
838
-
839
- close(fd);
840
- return -1;
841
- }
842
-
843
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
844
- "Connected to UNIX socket on path '%s'.",
845
- path);
846
-
847
- return fd;
848
-}
849
-
850
-// connect_to_this_ip46()
851
-// protocol IPPROTO_TCP, IPPROTO_UDP
852
-// socktype SOCK_STREAM, SOCK_DGRAM
853
-// host the destination hostname or IP address (IPv4 or IPv6) to connect to
854
-// if it resolves to many IPs, all are tried (IPv4 and IPv6)
855
-// scope_id the if_index id of the interface to use for connecting (0 = any)
856
-// (used only under IPv6)
857
-// service the service name or port to connect to
858
-// timeout the timeout for establishing a connection
859
-
860
-int connect_to_this_ip46(
861
- int protocol,
862
- int socktype,
863
- const char *host,
864
- uint32_t scope_id,
865
- const char *service,
866
- struct timeval *timeout,
867
- bool *fallback_ipv4)
868
-{
869
- struct addrinfo hints;
870
- struct addrinfo *ai_head = NULL, *ai = NULL;
871
-
872
- memset(&hints, 0, sizeof(hints));
873
- hints.ai_family = PF_UNSPEC; /* Allow IPv4 or IPv6 */
874
- hints.ai_socktype = socktype;
875
- hints.ai_protocol = protocol;
876
-
877
- int ai_err = getaddrinfo(host, service, &hints, &ai_head);
878
- if (ai_err != 0) {
879
-
880
- nd_log(NDLS_DAEMON, NDLP_ERR,
881
- "Cannot resolve host '%s', port '%s': %s",
882
- host, service, gai_strerror(ai_err));
883
-
884
- return -1;
885
- }
886
-
887
- char hostBfr[NI_MAXHOST + 1];
888
- char servBfr[NI_MAXSERV + 1];
889
-
890
- ND_LOG_STACK lgs[] = {
891
- ND_LOG_FIELD_TXT(NDF_DST_IP, hostBfr),
892
- ND_LOG_FIELD_TXT(NDF_DST_PORT, servBfr),
893
- ND_LOG_FIELD_END(),
894
- };
895
- ND_LOG_STACK_PUSH(lgs);
896
-
897
- int fd = -1;
898
- for (ai = ai_head; ai != NULL && fd == -1; ai = ai->ai_next) {
899
- if(nd_thread_signaled_to_cancel()) break;
900
-
901
- if (fallback_ipv4 && *fallback_ipv4 && ai->ai_family == PF_INET6)
902
- continue;
903
-
904
- if (ai->ai_family == PF_INET6) {
905
- struct sockaddr_in6 *pSadrIn6 = (struct sockaddr_in6 *) ai->ai_addr;
906
- if(pSadrIn6->sin6_scope_id == 0) {
907
- pSadrIn6->sin6_scope_id = scope_id;
908
- }
909
- }
910
-
911
- getnameinfo(ai->ai_addr,
912
- ai->ai_addrlen,
913
- hostBfr,
914
- sizeof(hostBfr),
915
- servBfr,
916
- sizeof(servBfr),
917
- NI_NUMERICHOST | NI_NUMERICSERV);
918
-
919
- switch (ai->ai_addr->sa_family) {
920
- case PF_INET: {
921
- struct sockaddr_in *pSadrIn = (struct sockaddr_in *)ai->ai_addr;
922
- (void)pSadrIn;
923
- break;
924
- }
925
-
926
- case PF_INET6: {
927
- struct sockaddr_in6 *pSadrIn6 = (struct sockaddr_in6 *) ai->ai_addr;
928
- (void)pSadrIn6;
929
- break;
930
- }
931
-
932
- default: {
933
- // Unknown protocol family
934
- continue;
935
- }
936
- }
937
-
938
- fd = socket(ai->ai_family, ai->ai_socktype | DEFAULT_SOCKET_FLAGS, ai->ai_protocol);
939
- if(fd != -1) {
940
- if(timeout) {
941
- if(setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, (char *) timeout, sizeof(struct timeval)) < 0)
942
- nd_log(NDLS_DAEMON, NDLP_ERR,
943
- "Failed to set timeout on the socket to ip '%s' port '%s'",
944
- hostBfr, servBfr);
945
- }
946
- sock_setcloexec(fd);
947
-
948
- errno_clear();
949
- if(connect(fd, ai->ai_addr, ai->ai_addrlen) < 0) {
950
- if(errno == EALREADY || errno == EINPROGRESS) {
951
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
952
- "Waiting for connection to ip %s port %s to be established",
953
- hostBfr, servBfr);
954
-
955
- // Convert 'struct timeval' to milliseconds for poll():
956
- int timeout_ms = timeout ? (timeout->tv_sec * 1000 + timeout->tv_usec / 1000) : 1000;
957
-
958
- switch(wait_on_socket_or_cancel_with_timeout(
959
- NULL,
960
- fd, timeout_ms, POLLOUT, NULL)) {
961
- case 0: // proceed
962
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
963
- "connect() to ip %s port %s completed successfully",
964
- hostBfr, servBfr);
965
- break;
966
-
967
- case -1: // thread cancelled
968
- nd_log(NDLS_DAEMON, NDLP_ERR,
969
- "Thread is cancelled while connecting to '%s', port '%s'.",
970
- hostBfr, servBfr);
971
-
972
- close(fd);
973
- fd = -1;
974
- break;
975
-
976
- case 1: // timeout
977
- nd_log(NDLS_DAEMON, NDLP_ERR,
978
- "Timed out while connecting to '%s', port '%s'.",
979
- hostBfr, servBfr);
980
-
981
- close(fd);
982
- fd = -1;
983
-
984
- if (fallback_ipv4 && ai->ai_family == PF_INET6)
985
- *fallback_ipv4 = true;
986
- break;
987
-
988
- default:
989
- case 2: // error
990
- nd_log(NDLS_DAEMON, NDLP_ERR,
991
- "Failed to connect to '%s', port '%s'.",
992
- hostBfr, servBfr);
993
-
994
- close(fd);
995
- fd = -1;
996
- break;
997
- }
998
- }
999
- else {
1000
- nd_log(NDLS_DAEMON, NDLP_ERR,
1001
- "Failed to connect to '%s', port '%s'",
1002
- hostBfr, servBfr);
1003
-
1004
- close(fd);
1005
- fd = -1;
1006
- }
1007
- }
1008
- }
1009
- else
1010
- nd_log(NDLS_DAEMON, NDLP_ERR,
1011
- "Failed to socket() to '%s', port '%s'",
1012
- hostBfr, servBfr);
1013
- }
1014
-
1015
- freeaddrinfo(ai_head);
1016
-
1017
- return fd;
1018
-}
1019
-
1020
-// connect_to_this()
1021
-//
1022
-// definition format:
1023
-//
1024
-// [PROTOCOL:]IP[%INTERFACE][:PORT]
1025
-//
1026
-// PROTOCOL = tcp or udp
1027
-// IP = IPv4 or IPv6 IP or hostname, optionally enclosed in [] (required for IPv6)
1028
-// INTERFACE = for IPv6 only, the network interface to use
1029
-// PORT = port number or service name
1030
-
1031
-int connect_to_this(const char *definition, int default_port, struct timeval *timeout) {
1032
- char buffer[strlen(definition) + 1];
1033
- strcpy(buffer, definition);
1034
-
1035
- char default_service[10 + 1];
1036
- snprintfz(default_service, 10, "%d", default_port);
1037
-
1038
- char *host = buffer, *service = default_service, *iface = "";
1039
- int protocol = IPPROTO_TCP, socktype = SOCK_STREAM;
1040
- uint32_t scope_id = 0;
1041
-
1042
- if(strncmp(host, "tcp:", 4) == 0) {
1043
- host += 4;
1044
- protocol = IPPROTO_TCP;
1045
- socktype = SOCK_STREAM;
1046
- }
1047
- else if(strncmp(host, "udp:", 4) == 0) {
1048
- host += 4;
1049
- protocol = IPPROTO_UDP;
1050
- socktype = SOCK_DGRAM;
1051
- }
1052
- else if(strncmp(host, "unix:", 5) == 0) {
1053
- char *path = host + 5;
1054
- return connect_to_unix(path, timeout);
1055
- }
1056
- else if(*host == '/') {
1057
- char *path = host;
1058
- return connect_to_unix(path, timeout);
1059
- }
1060
-
1061
- char *e = host;
1062
- if(*e == '[') {
1063
- e = ++host;
1064
- while(*e && *e != ']') e++;
1065
- if(*e == ']') {
1066
- *e = '\0';
1067
- e++;
1068
- }
1069
- }
1070
- else {
1071
- while(*e && *e != ':' && *e != '%') e++;
1072
- }
1073
-
1074
- if(*e == '%') {
1075
- *e = '\0';
1076
- e++;
1077
- iface = e;
1078
- while(*e && *e != ':') e++;
1079
- }
1080
-
1081
- if(*e == ':') {
1082
- *e = '\0';
1083
- e++;
1084
- service = e;
1085
- }
1086
-
1087
- if(!*host) {
1088
- nd_log(NDLS_DAEMON, NDLP_ERR,
1089
- "Definition '%s' does not specify a host.",
1090
- definition);
1091
-
1092
- return -1;
1093
- }
1094
-
1095
- if(*iface) {
1096
- scope_id = if_nametoindex(iface);
1097
- if(!scope_id)
1098
- nd_log(NDLS_DAEMON, NDLP_ERR,
1099
- "Cannot find a network interface named '%s'. Continuing with limiting the network interface",
1100
- iface);
1101
- }
1102
-
1103
- if(!*service)
1104
- service = default_service;
121
+ flags |= O_NONBLOCK;
122
123
+ int ret = fcntl(fd, F_SETFL, flags);
124
+ if(ret < 0)
125
+ nd_log(NDLS_DAEMON, NDLP_ERR,
126
+ "Failed to set O_NONBLOCK on socket %d",
127
+ fd);
128
1107
- return connect_to_this_ip46(protocol, socktype, host, scope_id, service, timeout,NULL);
129
+ return ret;
130
}
131
1110
-void foreach_entry_in_connection_string(const char *destination, bool (*callback)(char *entry, void *data), void *data) {
1111
- const char *s = destination;
1112
- while(*s) {
1113
- const char *e = s;
132
+int sock_delnonblock(int fd) {
133
+ int flags;
134
1115
- // skip separators, moving both s(tart) and e(nd)
1116
- while(isspace((uint8_t)*e) || *e == ',') s = ++e;
135
+ flags = fcntl(fd, F_GETFL);
136
+ flags &= ~O_NONBLOCK;
137
1118
- // move e(nd) to the first separator
1119
- while(*e && !isspace((uint8_t)*e) && *e != ',') e++;
138
+ int ret = fcntl(fd, F_SETFL, flags);
139
+ if(ret < 0)
140
+ nd_log(NDLS_DAEMON, NDLP_ERR,
141
+ "Failed to remove O_NONBLOCK on socket %d",
142
+ fd);
143
1121
- // is there anything?
1122
- if(!*s || s == e) break;
144
+ return ret;
145
+}
146
1124
- char buf[e - s + 1];
1125
- strncpyz(buf, s, e - s);
147
+int sock_setreuse(int fd, int reuse) {
148
+ int ret = setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
149
1127
- if(callback(buf, data)) break;
150
+ if(ret == -1)
151
+ nd_log(NDLS_DAEMON, NDLP_ERR,
152
+ "Failed to set SO_REUSEADDR on socket %d",
153
+ fd);
154
1129
- s = e;
1130
- }
155
+ return ret;
156
}
157
1133
-struct connect_to_one_of_data {
1134
- int default_port;
1135
- struct timeval *timeout;
1136
- size_t *reconnects_counter;
1137
- char *connected_to;
1138
- size_t connected_to_size;
1139
- int sock;
1140
-};
1141
-
1142
-static bool connect_to_one_of_callback(char *entry, void *data) {
1143
- struct connect_to_one_of_data *t = data;
1144
-
1145
- if(t->reconnects_counter)
1146
- t->reconnects_counter++;
1147
-
1148
- t->sock = connect_to_this(entry, t->default_port, t->timeout);
1149
- if(t->sock != -1) {
1150
- if(t->connected_to && t->connected_to_size) {
1151
- strncpyz(t->connected_to, entry, t->connected_to_size);
1152
- t->connected_to[t->connected_to_size - 1] = '\0';
1153
- }
1154
-
1155
- return true;
1156
- }
1157
-
1158
- return false;
158
+void sock_setcloexec(int fd)
159
+{
160
+ UNUSED(fd);
161
+ int flags = fcntl(fd, F_GETFD);
162
+ if (flags != -1)
163
+ (void) fcntl(fd, F_SETFD, flags | FD_CLOEXEC);
164
}
165
1161
-int connect_to_one_of(const char *destination, int default_port, struct timeval *timeout, size_t *reconnects_counter, char *connected_to, size_t connected_to_size) {
1162
- struct connect_to_one_of_data t = {
1163
- .default_port = default_port,
1164
- .timeout = timeout,
1165
- .reconnects_counter = reconnects_counter,
1166
- .connected_to = connected_to,
1167
- .connected_to_size = connected_to_size,
1168
- .sock = -1,
1169
- };
166
+int sock_setreuse_port(int fd __maybe_unused, int reuse __maybe_unused) {
167
+ int ret;
168
1171
- foreach_entry_in_connection_string(destination, connect_to_one_of_callback, &t);
169
+#ifdef SO_REUSEPORT
170
+ ret = setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &reuse, sizeof(reuse));
171
+ if(ret == -1 && errno != ENOPROTOOPT)
172
+ nd_log(NDLS_DAEMON, NDLP_ERR,
173
+ "failed to set SO_REUSEPORT on socket %d",
174
+ fd);
175
+#else
176
+ ret = -1;
177
+#endif
178
1173
- return t.sock;
179
+ return ret;
180
}
181
1176
-static bool connect_to_one_of_urls_callback(char *entry, void *data) {
1177
- char *s = strchr(entry, '/');
1178
- if(s) *s = '\0';
182
+int sock_enlarge_in(int fd) {
183
+ int ret, bs = LARGE_SOCK_SIZE;
184
+
185
+ ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &bs, sizeof(bs));
186
+
187
+ if(ret == -1)
188
+ nd_log(NDLS_DAEMON, NDLP_ERR,
189
+ "Failed to set SO_RCVBUF on socket %d",
190
+ fd);
191
1180
- return connect_to_one_of_callback(entry, data);
192
+ return ret;
193
}
194
1183
-int connect_to_one_of_urls(const char *destination, int default_port, struct timeval *timeout, size_t *reconnects_counter, char *connected_to, size_t connected_to_size) {
1184
- struct connect_to_one_of_data t = {
1185
- .default_port = default_port,
1186
- .timeout = timeout,
1187
- .reconnects_counter = reconnects_counter,
1188
- .connected_to = connected_to,
1189
- .connected_to_size = connected_to_size,
1190
- .sock = -1,
1191
- };
195
+int sock_enlarge_out(int fd) {
196
+ int ret, bs = LARGE_SOCK_SIZE;
197
+ ret = setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &bs, sizeof(bs));
198
1193
- foreach_entry_in_connection_string(destination, connect_to_one_of_urls_callback, &t);
199
+ if(ret == -1)
200
+ nd_log(NDLS_DAEMON, NDLP_ERR,
201
+ "Failed to set SO_SNDBUF on socket %d",
202
+ fd);
203
1195
- return t.sock;
204
+ return ret;
205
}
206
1198
-
207
// --------------------------------------------------------------------------------------------------------------------
208
// helpers to send/receive data in one call, in blocking mode, with a timeout
209
@@ -1259,34 +267,9 @@ inline int wait_on_socket_or_cancel_with_timeout(
267
return 1;
268
}
269
1262
-ssize_t recv_timeout(NETDATA_SSL *ssl, int sockfd, void *buf, size_t len, int flags, int timeout) {
1263
-
1264
- switch(wait_on_socket_or_cancel_with_timeout(
1265
- ssl,
1266
- sockfd, timeout * 1000, POLLIN, NULL)) {
1267
- case 0: // data are waiting
1268
- break;
1269
-
1270
- case 1: // timeout
1271
- return 0;
1272
-
1273
- default:
1274
- case -1: // thread cancelled
1275
- case 2: // error on socket
1276
- return -1;
1277
- }
1278
-
1279
- if (SSL_connection(ssl))
1280
- return netdata_ssl_read(ssl, buf, len);
1281
-
1282
- return recv(sockfd, buf, len, flags);
1283
-}
1284
-
1285
-ssize_t send_timeout(NETDATA_SSL *ssl, int sockfd, void *buf, size_t len, int flags, int timeout) {
270
+ssize_t send_timeout(NETDATA_SSL *ssl, int sockfd, void *buf, size_t len, int flags, time_t timeout) {
271
1287
- switch(wait_on_socket_or_cancel_with_timeout(
1288
- ssl,
1289
- sockfd, timeout * 1000, POLLOUT, NULL)) {
272
+ switch(wait_on_socket_or_cancel_with_timeout(ssl, sockfd, timeout * 1000, POLLOUT, NULL)) {
273
case 0: // data are waiting
274
break;
275
@@ -1300,9 +283,9 @@ ssize_t send_timeout(NETDATA_SSL *ssl, int sockfd, void *buf, size_t len, int fl
283
}
284
285
if(ssl->conn) {
1303
- if (SSL_connection(ssl)) {
286
+ if (SSL_connection(ssl))
287
return netdata_ssl_write(ssl, buf, len);
1305
- }
288
+
289
else {
290
nd_log(NDLS_DAEMON, NDLP_ERR,
291
"cannot write to SSL connection - connection is not ready.");
@@ -1314,7 +297,6 @@ ssize_t send_timeout(NETDATA_SSL *ssl, int sockfd, void *buf, size_t len, int fl
297
return send(sockfd, buf, len, flags);
298
}
299
1317
-
300
// --------------------------------------------------------------------------------------------------------------------
301
// accept4() replacement for systems that do not have one
302
@@ -1509,679 +491,3 @@ int accept_socket(int fd, int flags, char *client_ip, size_t ipsize, char *clien
491
492
return nfd;
493
}
1512
-
1513
-
1514
-// --------------------------------------------------------------------------------------------------------------------
1515
-// poll() based listener
1516
-// this should be the fastest possible listener for up to 100 sockets
1517
-// above 100, an epoll() interface is needed on Linux
1518
-
1519
-#define POLL_FDS_INCREASE_STEP 10
1520
-
1521
-inline POLLINFO *poll_add_fd(POLLJOB *p
1522
- , int fd
1523
- , int socktype
1524
- , HTTP_ACL port_acl
1525
- , uint32_t flags
1526
- , const char *client_ip
1527
- , const char *client_port
1528
- , const char *client_host
1529
- , void *(*add_callback)(POLLINFO * /*pi*/, short int * /*events*/, void * /*data*/)
1530
- , void (*del_callback)(POLLINFO * /*pi*/)
1531
- , int (*rcv_callback)(POLLINFO * /*pi*/, short int * /*events*/)
1532
- , int (*snd_callback)(POLLINFO * /*pi*/, short int * /*events*/)
1533
- , void *data
1534
-) {
1535
- if(unlikely(fd < 0)) return NULL;
1536
-
1537
- //if(p->limit && p->used >= p->limit) {
1538
- // nd_log(NDLS_DAEMON, NDLP_WARNING, "Max sockets limit reached (%zu sockets), dropping connection", p->used);
1539
- // close(fd);
1540
- // return NULL;
1541
- //}
1542
-
1543
- if(unlikely(!p->first_free)) {
1544
- size_t new_slots = p->slots + POLL_FDS_INCREASE_STEP;
1545
-
1546
- p->fds = reallocz(p->fds, sizeof(struct pollfd) * new_slots);
1547
- p->inf = reallocz(p->inf, sizeof(POLLINFO) * new_slots);
1548
-
1549
- // reset all the newly added slots
1550
- ssize_t i;
1551
- for(i = new_slots - 1; i >= (ssize_t)p->slots ; i--) {
1552
- p->fds[i].fd = -1;
1553
- p->fds[i].events = 0;
1554
- p->fds[i].revents = 0;
1555
-
1556
- p->inf[i].p = p;
1557
- p->inf[i].slot = (size_t)i;
1558
- p->inf[i].flags = 0;
1559
- p->inf[i].socktype = -1;
1560
- p->inf[i].port_acl = -1;
1561
-
1562
- p->inf[i].client_ip = NULL;
1563
- p->inf[i].client_port = NULL;
1564
- p->inf[i].client_host = NULL;
1565
- p->inf[i].del_callback = p->del_callback;
1566
- p->inf[i].rcv_callback = p->rcv_callback;
1567
- p->inf[i].snd_callback = p->snd_callback;
1568
- p->inf[i].data = NULL;
1569
-
1570
- // link them so that the first free will be earlier in the array
1571
- // (we loop decrementing i)
1572
- p->inf[i].next = p->first_free;
1573
- p->first_free = &p->inf[i];
1574
- }
1575
-
1576
- p->slots = new_slots;
1577
- }
1578
-
1579
- POLLINFO *pi = p->first_free;
1580
- p->first_free = p->first_free->next;
1581
-
1582
- struct pollfd *pf = &p->fds[pi->slot];
1583
- pf->fd = fd;
1584
- pf->events = POLLIN;
1585
- pf->revents = 0;
1586
-
1587
- pi->fd = fd;
1588
- pi->p = p;
1589
- pi->socktype = socktype;
1590
- pi->port_acl = port_acl;
1591
- pi->flags = flags;
1592
- pi->next = NULL;
1593
- pi->client_ip = strdupz(client_ip);
1594
- pi->client_port = strdupz(client_port);
1595
- pi->client_host = strdupz(client_host);
1596
-
1597
- pi->del_callback = del_callback;
1598
- pi->rcv_callback = rcv_callback;
1599
- pi->snd_callback = snd_callback;
1600
-
1601
- pi->connected_t = now_boottime_sec();
1602
- pi->last_received_t = 0;
1603
- pi->last_sent_t = 0;
1604
- pi->last_sent_t = 0;
1605
- pi->recv_count = 0;
1606
- pi->send_count = 0;
1607
-
1608
- p->used++;
1609
- if(unlikely(pi->slot > p->max))
1610
- p->max = pi->slot;
1611
-
1612
- if(pi->flags & POLLINFO_FLAG_CLIENT_SOCKET) {
1613
- pi->data = add_callback(pi, &pf->events, data);
1614
- }
1615
-
1616
- if(pi->flags & POLLINFO_FLAG_SERVER_SOCKET) {
1617
- p->min = pi->slot;
1618
- }
1619
-
1620
- return pi;
1621
-}
1622
-
1623
-inline void poll_close_fd(POLLINFO *pi) {
1624
- POLLJOB *p = pi->p;
1625
-
1626
- struct pollfd *pf = &p->fds[pi->slot];
1627
-
1628
- if(unlikely(pf->fd == -1)) return;
1629
-
1630
- if(pi->flags & POLLINFO_FLAG_CLIENT_SOCKET) {
1631
- pi->del_callback(pi);
1632
-
1633
- if(likely(!(pi->flags & POLLINFO_FLAG_DONT_CLOSE))) {
1634
- if(close(pf->fd) == -1)
1635
- nd_log(NDLS_DAEMON, NDLP_ERR,
1636
- "Failed to close() poll_events() socket %d",
1637
- pf->fd);
1638
- }
1639
- }
1640
-
1641
- pf->fd = -1;
1642
- pf->events = 0;
1643
- pf->revents = 0;
1644
-
1645
- pi->fd = -1;
1646
- pi->socktype = -1;
1647
- pi->flags = 0;
1648
- pi->data = NULL;
1649
-
1650
- pi->del_callback = NULL;
1651
- pi->rcv_callback = NULL;
1652
- pi->snd_callback = NULL;
1653
-
1654
- freez(pi->client_ip);
1655
- pi->client_ip = NULL;
1656
-
1657
- freez(pi->client_port);
1658
- pi->client_port = NULL;
1659
-
1660
- freez(pi->client_host);
1661
- pi->client_host = NULL;
1662
-
1663
- pi->next = p->first_free;
1664
- p->first_free = pi;
1665
-
1666
- p->used--;
1667
- if(unlikely(p->max == pi->slot)) {
1668
- p->max = p->min;
1669
- ssize_t i;
1670
- for(i = (ssize_t)pi->slot; i > (ssize_t)p->min ;i--) {
1671
- if (unlikely(p->fds[i].fd != -1)) {
1672
- p->max = (size_t)i;
1673
- break;
1674
- }
1675
- }
1676
- }
1677
-}
1678
-
1679
-void *poll_default_add_callback(POLLINFO *pi, short int *events, void *data) {
1680
- (void)pi;
1681
- (void)events;
1682
- (void)data;
1683
-
1684
- return NULL;
1685
-}
1686
-
1687
-void poll_default_del_callback(POLLINFO *pi) {
1688
- if(pi->data)
1689
- nd_log(NDLS_DAEMON, NDLP_ERR,
1690
- "POLLFD: internal error: del_callback_default() called with data pointer - possible memory leak");
1691
-}
1692
-
1693
-int poll_default_rcv_callback(POLLINFO *pi, short int *events) {
1694
- *events |= POLLIN;
1695
-
1696
- char buffer[1024 + 1];
1697
-
1698
- ssize_t rc;
1699
- do {
1700
- rc = recv(pi->fd, buffer, 1024, MSG_DONTWAIT);
1701
- if (rc < 0) {
1702
- // read failed
1703
- if (errno != EWOULDBLOCK && errno != EAGAIN) {
1704
- nd_log(NDLS_DAEMON, NDLP_ERR,
1705
- "POLLFD: poll_default_rcv_callback(): recv() failed with %zd.",
1706
- rc);
1707
-
1708
- return -1;
1709
- }
1710
- } else if (rc) {
1711
- // data received
1712
- nd_log(NDLS_DAEMON, NDLP_WARNING,
1713
- "POLLFD: internal error: poll_default_rcv_callback() is discarding %zd bytes received on socket %d",
1714
- rc, pi->fd);
1715
- }
1716
- } while (rc != -1);
1717
-
1718
- return 0;
1719
-}
1720
-
1721
-int poll_default_snd_callback(POLLINFO *pi, short int *events) {
1722
- *events &= ~POLLOUT;
1723
-
1724
- nd_log(NDLS_DAEMON, NDLP_WARNING,
1725
- "POLLFD: internal error: poll_default_snd_callback(): nothing to send on socket %d",
1726
- pi->fd);
1727
-
1728
- return 0;
1729
-}
1730
-
1731
-void poll_default_tmr_callback(void *timer_data) {
1732
- (void)timer_data;
1733
-}
1734
-
1735
-static void poll_events_cleanup(void *pptr) {
1736
- POLLJOB *p = CLEANUP_FUNCTION_GET_PTR(pptr);
1737
- if(!p) return;
1738
-
1739
- for(size_t i = 0 ; i <= p->max ; i++) {
1740
- POLLINFO *pi = &p->inf[i];
1741
- poll_close_fd(pi);
1742
- }
1743
-
1744
- freez(p->fds);
1745
- freez(p->inf);
1746
-}
1747
-
1748
-static int poll_process_error(POLLINFO *pi, struct pollfd *pf, short int revents) {
1749
- ND_LOG_STACK lgs[] = {
1750
- ND_LOG_FIELD_TXT(NDF_SRC_IP, pi->client_ip),
1751
- ND_LOG_FIELD_TXT(NDF_SRC_PORT, pi->client_port),
1752
- ND_LOG_FIELD_END(),
1753
- };
1754
- ND_LOG_STACK_PUSH(lgs);
1755
-
1756
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
1757
- "POLLFD: LISTENER: received %s %s %s on socket at slot %zu (fd %d) client '%s' port '%s' expecting %s %s %s, having %s %s %s"
1758
- , revents & POLLERR ? "POLLERR" : ""
1759
- , revents & POLLHUP ? "POLLHUP" : ""
1760
- , revents & POLLNVAL ? "POLLNVAL" : ""
1761
- , pi->slot
1762
- , pi->fd
1763
- , pi->client_ip ? pi->client_ip : "<undefined-ip>"
1764
- , pi->client_port ? pi->client_port : "<undefined-port>"
1765
- , pf->events & POLLIN ? "POLLIN" : "", pf->events & POLLOUT ? "POLLOUT" : "", pf->events & POLLPRI ? "POLLPRI" : ""
1766
- , revents & POLLIN ? "POLLIN" : "", revents & POLLOUT ? "POLLOUT" : "", revents & POLLPRI ? "POLLPRI" : ""
1767
- );
1768
-
1769
- pf->events = 0;
1770
- poll_close_fd(pi);
1771
- return 1;
1772
-}
1773
-
1774
-static inline int poll_process_send(POLLJOB *p, POLLINFO *pi, struct pollfd *pf, time_t now) {
1775
- pi->last_sent_t = now;
1776
- pi->send_count++;
1777
-
1778
- pf->events = 0;
1779
-
1780
- // remember the slot, in case we need to close it later
1781
- // the callback may manipulate the socket list and our pf and pi pointers may be invalid after that call
1782
- size_t slot = pi->slot;
1783
-
1784
- if (unlikely(pi->snd_callback(pi, &pf->events) == -1))
1785
- poll_close_fd(&p->inf[slot]);
1786
-
1787
- // IMPORTANT:
1788
- // pf and pi may be invalid below this point, they may have been reallocated.
1789
-
1790
- return 1;
1791
-}
1792
-
1793
-static inline int poll_process_tcp_read(POLLJOB *p, POLLINFO *pi, struct pollfd *pf, time_t now) {
1794
- pi->last_received_t = now;
1795
- pi->recv_count++;
1796
-
1797
- pf->events = 0;
1798
-
1799
- // remember the slot, in case we need to close it later
1800
- // the callback may manipulate the socket list and our pf and pi pointers may be invalid after that call
1801
- size_t slot = pi->slot;
1802
-
1803
- if (pi->rcv_callback(pi, &pf->events) == -1)
1804
- poll_close_fd(&p->inf[slot]);
1805
-
1806
- // IMPORTANT:
1807
- // pf and pi may be invalid below this point, they may have been reallocated.
1808
-
1809
- return 1;
1810
-}
1811
-
1812
-static inline int poll_process_udp_read(POLLINFO *pi, struct pollfd *pf, time_t now __maybe_unused) {
1813
- pi->last_received_t = now;
1814
- pi->recv_count++;
1815
-
1816
- // TODO: access_list is not applied to UDP
1817
- // but checking the access list on every UDP packet will destroy
1818
- // performance, especially for statsd.
1819
-
1820
- pf->events = 0;
1821
- if(pi->rcv_callback(pi, &pf->events) == -1)
1822
- return 0;
1823
-
1824
- // IMPORTANT:
1825
- // pf and pi may be invalid below this point, they may have been reallocated.
1826
-
1827
- return 1;
1828
-}
1829
-
1830
-static int poll_process_new_tcp_connection(POLLJOB *p, POLLINFO *pi, struct pollfd *pf, time_t now) {
1831
- pi->last_received_t = now;
1832
- pi->recv_count++;
1833
-
1834
- char client_ip[INET6_ADDRSTRLEN] = "";
1835
- char client_port[NI_MAXSERV] = "";
1836
- char client_host[NI_MAXHOST] = "";
1837
-
1838
-#ifdef SOCK_NONBLOCK
1839
- int flags = SOCK_NONBLOCK;
1840
-#else
1841
- int flags = 0;
1842
-#endif
1843
-
1844
- int nfd = accept_socket(
1845
- pf->fd, flags,
1846
- client_ip, INET6_ADDRSTRLEN, client_port,NI_MAXSERV, client_host, NI_MAXHOST,
1847
- p->access_list, p->allow_dns
1848
- );
1849
-
1850
-#ifndef SOCK_NONBLOCK
1851
- if (nfd > 0) {
1852
- int flags = fcntl(nfd, F_GETFL);
1853
- (void)fcntl(nfd, F_SETFL, flags| O_NONBLOCK);
1854
- }
1855
-#endif
1856
-
1857
- if (unlikely(nfd < 0)) {
1858
- // accept failed
1859
-
1860
- if(unlikely(errno == EMFILE)) {
1861
- nd_log_limit_static_global_var(erl, 10, 1000);
1862
- nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
1863
- "POLLFD: LISTENER: too many open files - used by this thread %zu, max for this thread %zu",
1864
- p->used, p->limit);
1865
- }
1866
- else if(unlikely(errno != EWOULDBLOCK && errno != EAGAIN))
1867
- nd_log(NDLS_DAEMON, NDLP_ERR,
1868
- "POLLFD: LISTENER: accept() failed.");
1869
-
1870
- }
1871
- else {
1872
- // accept ok
1873
-
1874
- poll_add_fd(p
1875
- , nfd
1876
- , SOCK_STREAM
1877
- , pi->port_acl
1878
- , POLLINFO_FLAG_CLIENT_SOCKET
1879
- , client_ip
1880
- , client_port
1881
- , client_host
1882
- , p->add_callback
1883
- , p->del_callback
1884
- , p->rcv_callback
1885
- , p->snd_callback
1886
- , NULL
1887
- );
1888
-
1889
- // IMPORTANT:
1890
- // pf and pi may be invalid below this point, they may have been reallocated.
1891
-
1892
- return 1;
1893
- }
1894
-
1895
- return 0;
1896
-}
1897
-
1898
-void poll_events(LISTEN_SOCKETS *sockets
1899
- , void *(*add_callback)(POLLINFO * /*pi*/, short int * /*events*/, void * /*data*/)
1900
- , void (*del_callback)(POLLINFO * /*pi*/)
1901
- , int (*rcv_callback)(POLLINFO * /*pi*/, short int * /*events*/)
1902
- , int (*snd_callback)(POLLINFO * /*pi*/, short int * /*events*/)
1903
- , void (*tmr_callback)(void * /*timer_data*/)
1904
- , bool (*check_to_stop_callback)(void)
1905
- , SIMPLE_PATTERN *access_list
1906
- , int allow_dns
1907
- , void *data
1908
- , time_t tcp_request_timeout_seconds
1909
- , time_t tcp_idle_timeout_seconds
1910
- , time_t timer_milliseconds
1911
- , void *timer_data
1912
- , size_t max_tcp_sockets
1913
-) {
1914
- if(!sockets || !sockets->opened) {
1915
- nd_log(NDLS_DAEMON, NDLP_ERR,
1916
- "POLLFD: internal error: no listening sockets are opened");
1917
- return;
1918
- }
1919
-
1920
- if(timer_milliseconds <= 0) timer_milliseconds = 0;
1921
-
1922
- int retval;
1923
-
1924
- POLLJOB p = {
1925
- .slots = 0,
1926
- .used = 0,
1927
- .max = 0,
1928
- .limit = max_tcp_sockets,
1929
- .fds = NULL,
1930
- .inf = NULL,
1931
- .first_free = NULL,
1932
-
1933
- .complete_request_timeout = tcp_request_timeout_seconds,
1934
- .idle_timeout = tcp_idle_timeout_seconds,
1935
- .checks_every = (tcp_idle_timeout_seconds / 3) + 1,
1936
-
1937
- .access_list = access_list,
1938
- .allow_dns = allow_dns,
1939
-
1940
- .timer_milliseconds = timer_milliseconds,
1941
- .timer_data = timer_data,
1942
-
1943
- .add_callback = add_callback?add_callback:poll_default_add_callback,
1944
- .del_callback = del_callback?del_callback:poll_default_del_callback,
1945
- .rcv_callback = rcv_callback?rcv_callback:poll_default_rcv_callback,
1946
- .snd_callback = snd_callback?snd_callback:poll_default_snd_callback,
1947
- .tmr_callback = tmr_callback?tmr_callback:poll_default_tmr_callback
1948
- };
1949
-
1950
- size_t i;
1951
- for(i = 0; i < sockets->opened ;i++) {
1952
-
1953
- POLLINFO *pi = poll_add_fd(&p
1954
- , sockets->fds[i]
1955
- , sockets->fds_types[i]
1956
- , sockets->fds_acl_flags[i]
1957
- , POLLINFO_FLAG_SERVER_SOCKET
1958
- , (sockets->fds_names[i])?sockets->fds_names[i]:"UNKNOWN"
1959
- , ""
1960
- , ""
1961
- , p.add_callback
1962
- , p.del_callback
1963
- , p.rcv_callback
1964
- , p.snd_callback
1965
- , NULL
1966
- );
1967
-
1968
- pi->data = data;
1969
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
1970
- "POLLFD: LISTENER: listening on '%s'",
1971
- (sockets->fds_names[i])?sockets->fds_names[i]:"UNKNOWN");
1972
- }
1973
-
1974
- int listen_sockets_active = 1;
1975
-
1976
- time_t last_check = now_boottime_sec();
1977
-
1978
- usec_t timer_usec = timer_milliseconds * USEC_PER_MS;
1979
- usec_t now_usec = 0, next_timer_usec = 0, last_timer_usec = 0;
1980
- (void)last_timer_usec;
1981
-
1982
- if(unlikely(timer_usec)) {
1983
- now_usec = now_boottime_usec();
1984
- next_timer_usec = now_usec - (now_usec % timer_usec) + timer_usec;
1985
- }
1986
-
1987
- CLEANUP_FUNCTION_REGISTER(poll_events_cleanup) cleanup_ptr = &p;
1988
-
1989
- while(!check_to_stop_callback() && !nd_thread_signaled_to_cancel()) {
1990
- if(unlikely(timer_usec)) {
1991
- now_usec = now_boottime_usec();
1992
-
1993
- if(unlikely(timer_usec && now_usec >= next_timer_usec)) {
1994
- last_timer_usec = now_usec;
1995
- p.tmr_callback(p.timer_data);
1996
- now_usec = now_boottime_usec();
1997
- next_timer_usec = now_usec - (now_usec % timer_usec) + timer_usec;
1998
- }
1999
- }
2000
-
2001
- // enable or disable the TCP listening sockets, based on the current number of sockets used and the limit set
2002
- if((listen_sockets_active && (p.limit && p.used >= p.limit)) || (!listen_sockets_active && (!p.limit || p.used < p.limit))) {
2003
- listen_sockets_active = !listen_sockets_active;
2004
-
2005
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
2006
- "%s listening sockets (used TCP sockets %zu, max allowed for this worker %zu)",
2007
- (listen_sockets_active)?"ENABLING":"DISABLING", p.used, p.limit);
2008
-
2009
- for (i = 0; i <= p.max; i++) {
2010
- if(p.inf[i].flags & POLLINFO_FLAG_SERVER_SOCKET && p.inf[i].socktype == SOCK_STREAM) {
2011
- p.fds[i].events = (short int) ((listen_sockets_active) ? POLLIN : 0);
2012
- }
2013
- }
2014
- }
2015
-
2016
- retval = poll(p.fds, p.max + 1, ND_CHECK_CANCELLABILITY_WHILE_WAITING_EVERY_MS);
2017
- time_t now = now_boottime_sec();
2018
-
2019
- if(unlikely(retval == -1)) {
2020
- nd_log(NDLS_DAEMON, NDLP_ERR,
2021
- "POLLFD: LISTENER: poll() failed while waiting on %zu sockets.",
2022
- p.max + 1);
2023
-
2024
- break;
2025
- }
2026
- else if(unlikely(!retval)) {
2027
- // timeout
2028
- ;
2029
- }
2030
- else {
2031
- POLLINFO *pi;
2032
- struct pollfd *pf;
2033
- size_t idx, processed = 0;
2034
- short int revents;
2035
-
2036
- // keep fast lookup arrays per function
2037
- // to avoid looping through the entire list every time
2038
- size_t sends[p.max + 1], sends_max = 0;
2039
- size_t reads[p.max + 1], reads_max = 0;
2040
- size_t conns[p.max + 1], conns_max = 0;
2041
- size_t udprd[p.max + 1], udprd_max = 0;
2042
-
2043
- for (i = 0; i <= p.max; i++) {
2044
- pi = &p.inf[i];
2045
- pf = &p.fds[i];
2046
- revents = pf->revents;
2047
-
2048
- if(unlikely(revents == 0 || pf->fd == -1))
2049
- continue;
2050
-
2051
- if (unlikely(revents & (POLLERR|POLLHUP|POLLNVAL))) {
2052
- // something is wrong to one of our sockets
2053
-
2054
- pf->revents = 0;
2055
- processed += poll_process_error(pi, pf, revents);
2056
- }
2057
- else if (likely(revents & POLLOUT)) {
2058
- // a client is ready to receive data
2059
-
2060
- sends[sends_max++] = i;
2061
- }
2062
- else if (likely(revents & (POLLIN|POLLPRI))) {
2063
- if (pi->flags & POLLINFO_FLAG_CLIENT_SOCKET) {
2064
- // a client sent data to us
2065
-
2066
- reads[reads_max++] = i;
2067
- }
2068
- else if (pi->flags & POLLINFO_FLAG_SERVER_SOCKET) {
2069
- // something is coming to our server sockets
2070
-
2071
- if(pi->socktype == SOCK_DGRAM) {
2072
- // UDP receive, directly on our listening socket
2073
-
2074
- udprd[udprd_max++] = i;
2075
- }
2076
- else if(pi->socktype == SOCK_STREAM) {
2077
- // new TCP connection
2078
-
2079
- conns[conns_max++] = i;
2080
- }
2081
- else
2082
- nd_log(NDLS_DAEMON, NDLP_ERR,
2083
- "POLLFD: LISTENER: server slot %zu (fd %d) connection from %s port %s using unhandled socket type %d."
2084
- , i
2085
- , pi->fd
2086
- , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2087
- , pi->client_port ? pi->client_port : "<undefined-port>"
2088
- , pi->socktype
2089
- );
2090
- }
2091
- else
2092
- nd_log(NDLS_DAEMON, NDLP_ERR,
2093
- "POLLFD: LISTENER: client slot %zu (fd %d) data from %s port %s using flags %08X is neither client nor server."
2094
- , i
2095
- , pi->fd
2096
- , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2097
- , pi->client_port ? pi->client_port : "<undefined-port>"
2098
- , pi->flags
2099
- );
2100
- }
2101
- else
2102
- nd_log(NDLS_DAEMON, NDLP_ERR,
2103
- "POLLFD: LISTENER: socket slot %zu (fd %d) client %s port %s unhandled event id %d."
2104
- , i
2105
- , pi->fd
2106
- , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2107
- , pi->client_port ? pi->client_port : "<undefined-port>"
2108
- , revents
2109
- );
2110
- }
2111
-
2112
- // process sends
2113
- for (idx = 0; idx < sends_max; idx++) {
2114
- i = sends[idx];
2115
- pi = &p.inf[i];
2116
- pf = &p.fds[i];
2117
- pf->revents = 0;
2118
- processed += poll_process_send(&p, pi, pf, now);
2119
- }
2120
-
2121
- // process UDP reads
2122
- for (idx = 0; idx < udprd_max; idx++) {
2123
- i = udprd[idx];
2124
- pi = &p.inf[i];
2125
- pf = &p.fds[i];
2126
- pf->revents = 0;
2127
- processed += poll_process_udp_read(pi, pf, now);
2128
- }
2129
-
2130
- // process TCP reads
2131
- for (idx = 0; idx < reads_max; idx++) {
2132
- i = reads[idx];
2133
- pi = &p.inf[i];
2134
- pf = &p.fds[i];
2135
- pf->revents = 0;
2136
- processed += poll_process_tcp_read(&p, pi, pf, now);
2137
- }
2138
-
2139
- if(!processed && (!p.limit || p.used < p.limit)) {
2140
- // nothing processed above (rcv, snd) and we have room for another TCP connection
2141
- // so, accept one TCP connection
2142
- for (idx = 0; idx < conns_max; idx++) {
2143
- i = conns[idx];
2144
- pi = &p.inf[i];
2145
- pf = &p.fds[i];
2146
- pf->revents = 0;
2147
- if (poll_process_new_tcp_connection(&p, pi, pf, now))
2148
- break;
2149
- }
2150
- }
2151
- }
2152
-
2153
- if(unlikely(p.checks_every > 0 && now - last_check > p.checks_every)) {
2154
- last_check = now;
2155
-
2156
- // cleanup old sockets
2157
- for(i = 0; i <= p.max; i++) {
2158
- POLLINFO *pi = &p.inf[i];
2159
-
2160
- if(likely(pi->flags & POLLINFO_FLAG_CLIENT_SOCKET)) {
2161
- if (unlikely(pi->send_count == 0 && p.complete_request_timeout > 0 && (now - pi->connected_t) >= p.complete_request_timeout)) {
2162
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
2163
- "POLLFD: LISTENER: client slot %zu (fd %d) from %s port %s has not sent a complete request in %zu seconds - closing it. "
2164
- , i
2165
- , pi->fd
2166
- , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2167
- , pi->client_port ? pi->client_port : "<undefined-port>"
2168
- , (size_t) p.complete_request_timeout
2169
- );
2170
- poll_close_fd(pi);
2171
- }
2172
- else if(unlikely(pi->recv_count && p.idle_timeout > 0 && now - ((pi->last_received_t > pi->last_sent_t) ? pi->last_received_t : pi->last_sent_t) >= p.idle_timeout )) {
2173
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
2174
- "POLLFD: LISTENER: client slot %zu (fd %d) from %s port %s is idle for more than %zu seconds - closing it. "
2175
- , i
2176
- , pi->fd
2177
- , pi->client_ip ? pi->client_ip : "<undefined-ip>"
2178
- , pi->client_port ? pi->client_port : "<undefined-port>"
2179
- , (size_t) p.idle_timeout
2180
- );
2181
- poll_close_fd(pi);
2182
- }
2183
- }
2184
- }
2185
- }
2186
- }
2187
-}
src/libnetdata/socket/socket.h
+1
-171
@@ -5,49 +5,9 @@
5
6
#include "../libnetdata.h"
7
8
-#ifndef MAX_LISTEN_FDS
9
-#define MAX_LISTEN_FDS 50
10
-#endif
11
-
8
#define ND_CHECK_CANCELLABILITY_WHILE_WAITING_EVERY_MS 100
9
14
-typedef struct listen_sockets {
15
- struct config *config; // the config file to use
16
- const char *config_section; // the netdata configuration section to read settings from
17
- const char *default_bind_to; // the default bind to configuration string
18
- uint16_t default_port; // the default port to use
19
- int backlog; // the default listen backlog to use
20
-
21
- size_t opened; // the number of sockets opened
22
- size_t failed; // the number of sockets attempted to open, but failed
23
- int fds[MAX_LISTEN_FDS]; // the open sockets
24
- char *fds_names[MAX_LISTEN_FDS]; // descriptions for the open sockets
25
- int fds_types[MAX_LISTEN_FDS]; // the socktype for the open sockets (SOCK_STREAM, SOCK_DGRAM)
26
- int fds_families[MAX_LISTEN_FDS]; // the family of the open sockets (AF_UNIX, AF_INET, AF_INET6)
27
- HTTP_ACL fds_acl_flags[MAX_LISTEN_FDS]; // the acl to apply to the open sockets (dashboard, badges, streaming, netdata.conf, management)
28
-} LISTEN_SOCKETS;
29
-
30
-char *strdup_client_description(int family, const char *protocol, const char *ip, uint16_t port);
31
-
32
-int listen_sockets_setup(LISTEN_SOCKETS *sockets);
33
-void listen_sockets_close(LISTEN_SOCKETS *sockets);
34
-
35
-void foreach_entry_in_connection_string(const char *destination, bool (*callback)(char *entry, void *data), void *data);
36
-int connect_to_this_ip46(
37
- int protocol,
38
- int socktype,
39
- const char *host,
40
- uint32_t scope_id,
41
- const char *service,
42
- struct timeval *timeout,
43
- bool *fallback_ipv4);
44
-int connect_to_this(const char *definition, int default_port, struct timeval *timeout);
45
-int connect_to_one_of(const char *destination, int default_port, struct timeval *timeout, size_t *reconnects_counter, char *connected_to, size_t connected_to_size);
46
-int connect_to_one_of_urls(const char *destination, int default_port, struct timeval *timeout, size_t *reconnects_counter, char *connected_to, size_t connected_to_size);
47
-
48
-
49
-ssize_t recv_timeout(NETDATA_SSL *ssl,int sockfd, void *buf, size_t len, int flags, int timeout);
50
-ssize_t send_timeout(NETDATA_SSL *ssl,int sockfd, void *buf, size_t len, int flags, int timeout);
10
+ssize_t send_timeout(NETDATA_SSL *ssl,int sockfd, void *buf, size_t len, int flags, time_t timeout);
11
int wait_on_socket_or_cancel_with_timeout(NETDATA_SSL *ssl, int fd, int timeout_ms, short int poll_events, short int *revents);
12
13
bool fd_is_socket(int fd);
@@ -77,136 +37,6 @@ int accept4(int sock, struct sockaddr *addr, socklen_t *addrlen, int flags);
37
#endif
38
39
80
-// ----------------------------------------------------------------------------
81
-// poll() based listener
82
-
83
-#define POLLINFO_FLAG_SERVER_SOCKET 0x00000001
84
-#define POLLINFO_FLAG_CLIENT_SOCKET 0x00000002
85
-#define POLLINFO_FLAG_DONT_CLOSE 0x00000004
86
-
87
-typedef struct poll POLLJOB;
88
-
89
-typedef struct pollinfo {
90
- POLLJOB *p; // the parent
91
- size_t slot; // the slot id
92
-
93
- int fd; // the file descriptor
94
- int socktype; // the client socket type
95
- HTTP_ACL port_acl; // the access lists permitted on this web server port (it's -1 for client sockets)
96
- char *client_ip; // Max INET6_ADDRSTRLEN bytes
97
- char *client_port; // Max NI_MAXSERV bytes
98
- char *client_host; // Max NI_MAXHOST bytes
99
-
100
- time_t connected_t; // the time the socket connected
101
- time_t last_received_t; // the time the socket last received data
102
- time_t last_sent_t; // the time the socket last sent data
103
-
104
- size_t recv_count; // the number of times the socket was ready for inbound traffic
105
- size_t send_count; // the number of times the socket was ready for outbound traffic
106
-
107
- uint32_t flags; // internal flags
108
-
109
- // callbacks for this socket
110
- void (*del_callback)(struct pollinfo *pi);
111
- int (*rcv_callback)(struct pollinfo *pi, short int *events);
112
- int (*snd_callback)(struct pollinfo *pi, short int *events);
113
-
114
- // the user data
115
- void *data;
116
-
117
- // linking of free pollinfo structures
118
- // for quickly finding the next available
119
- // this is like a stack, it grows and shrinks
120
- // (with gaps - lower empty slots are preferred)
121
- struct pollinfo *next;
122
-} POLLINFO;
123
-
124
-struct poll {
125
- size_t slots;
126
- size_t used;
127
- size_t min;
128
- size_t max;
129
-
130
- size_t limit;
131
-
132
- time_t complete_request_timeout;
133
- time_t idle_timeout;
134
- time_t checks_every;
135
-
136
- time_t timer_milliseconds;
137
- void *timer_data;
138
-
139
- struct pollfd *fds;
140
- struct pollinfo *inf;
141
- struct pollinfo *first_free;
142
-
143
- SIMPLE_PATTERN *access_list;
144
- int allow_dns;
145
-
146
- void *(*add_callback)(POLLINFO *pi, short int *events, void *data);
147
- void (*del_callback)(POLLINFO *pi);
148
- int (*rcv_callback)(POLLINFO *pi, short int *events);
149
- int (*snd_callback)(POLLINFO *pi, short int *events);
150
- void (*tmr_callback)(void *timer_data);
151
-};
152
-
153
-#define pollinfo_from_slot(p, slot) (&((p)->inf[(slot)]))
154
-
155
-int poll_default_snd_callback(POLLINFO *pi, short int *events);
156
-int poll_default_rcv_callback(POLLINFO *pi, short int *events);
157
-void poll_default_del_callback(POLLINFO *pi);
158
-void *poll_default_add_callback(POLLINFO *pi, short int *events, void *data);
159
-
160
-POLLINFO *poll_add_fd(POLLJOB *p
161
- , int fd
162
- , int socktype
163
- , HTTP_ACL port_acl
164
- , uint32_t flags
165
- , const char *client_ip
166
- , const char *client_port
167
- , const char *client_host
168
- , void *(*add_callback)(POLLINFO *pi, short int *events, void *data)
169
- , void (*del_callback)(POLLINFO *pi)
170
- , int (*rcv_callback)(POLLINFO *pi, short int *events)
171
- , int (*snd_callback)(POLLINFO *pi, short int *events)
172
- , void *data
173
-);
174
-void poll_close_fd(POLLINFO *pi);
175
-
176
-void poll_events(LISTEN_SOCKETS *sockets
177
- , void *(*add_callback)(POLLINFO *pi, short int *events, void *data)
178
- , void (*del_callback)(POLLINFO *pi)
179
- , int (*rcv_callback)(POLLINFO *pi, short int *events)
180
- , int (*snd_callback)(POLLINFO *pi, short int *events)
181
- , void (*tmr_callback)(void *timer_data)
182
- , bool (*check_to_stop_callback)(void)
183
- , SIMPLE_PATTERN *access_list
184
- , int allow_dns
185
- , void *data
186
- , time_t tcp_request_timeout_seconds
187
- , time_t tcp_idle_timeout_seconds
188
- , time_t timer_milliseconds
189
- , void *timer_data
190
- , size_t max_tcp_sockets
191
-);
192
-
193
-#ifndef INET6_ADDRSTRLEN
194
-#define INET6_ADDRSTRLEN 46
195
-#endif
196
-
197
-typedef struct {
198
- struct {
199
- char ip[INET6_ADDRSTRLEN];
200
- int port;
201
- } local;
202
-
203
- struct {
204
- char ip[INET6_ADDRSTRLEN];
205
- int port;
206
- } peer;
207
-} SOCKET_PEERS;
208
-
209
-SOCKET_PEERS socket_peers(int sock_fd);
40
bool ip_to_hostname(const char *ip, char *dst, size_t dst_len);
41
42
#endif //NETDATA_SOCKET_H
src/libnetdata/spawn_server/log-forwarder.c
+1
-1
@@ -257,7 +257,7 @@ static void *log_forwarder_thread_func(void *arg) {
257
ssize_t bytes_read = read(lf->pipe_fds[PIPE_READ], buf, sizeof(buf));
258
// Ignore the data; proceed regardless of the result
259
if (bytes_read == -1) {
260
- if (errno != EAGAIN && errno != EWOULDBLOCK) {
260
+ if (errno != EAGAIN && errno != EWOULDBLOCK && errno != EINTR) {
261
// Handle read error if necessary
262
nd_log(NDLS_COLLECTORS, NDLP_ERR, "Failed to read from notification pipe");
263
return NULL;
src/libnetdata/threads/threads.c
+2
-2
@@ -236,7 +236,7 @@ void netdata_threads_init_for_external_plugins(size_t stacksize) {
236
// ----------------------------------------------------------------------------
237
238
void rrdset_thread_rda_free(void);
239
-void sender_thread_buffer_free(void);
239
+void sender_commit_thread_buffer_free(void);
240
void query_target_free(void);
241
void service_exits(void);
242
void rrd_collector_finished(void);
@@ -305,7 +305,7 @@ static void nd_thread_exit(void *pptr) {
305
nd_log(NDLS_DAEMON, NDLP_DEBUG, "thread with task id %d finished", nti->tid);
306
307
rrd_collector_finished();
308
- sender_thread_buffer_free();
308
+ sender_commit_thread_buffer_free();
309
rrdset_thread_rda_free();
310
query_target_free();
311
thread_cache_destroy();
src/libnetdata/worker_utilization/README.md
+1
-1
@@ -80,6 +80,6 @@ busy or idle all the time or part of the time. Works well for both thousands of
80
per second and unlimited working time (being totally busy with a single request for
81
ages).
82
83
-The statistics collector is called by the global statistics thread of netdata. So,
83
+The statistics collector is called by the telemetry thread of netdata. So,
84
even if the workers are extremely busy with their jobs, netdata will be able to know
85
how busy they are.
src/libnetdata/worker_utilization/worker_utilization.c
+1
-3
@@ -223,9 +223,7 @@ void worker_is_busy(size_t job_id) {
223
}
224
225
void worker_set_metric(size_t job_id, NETDATA_DOUBLE value) {
226
- if(unlikely(!worker)) return;
227
-
228
- if(unlikely(job_id >= WORKER_UTILIZATION_MAX_JOB_TYPES))
226
+ if(unlikely(!worker || job_id >= WORKER_UTILIZATION_MAX_JOB_TYPES))
227
return;
228
229
switch(worker->per_job_type[job_id].type) {
src/libnetdata/worker_utilization/worker_utilization.h
+1
-1
@@ -5,7 +5,7 @@
5
6
// workers interfaces
7
8
-#define WORKER_UTILIZATION_MAX_JOB_TYPES 50
8
+#define WORKER_UTILIZATION_MAX_JOB_TYPES 80
9
10
typedef enum __attribute__((packed)) {
11
WORKER_METRIC_EMPTY = 0,
src/ml/ad_charts.cc
+170
-64
@@ -1,6 +1,7 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "ad_charts.h"
4
+#include "ml_config.h"
5
6
void ml_update_dimensions_chart(ml_host_t *host, const ml_machine_learning_stats_t &mls) {
7
/*
@@ -399,19 +400,19 @@ void ml_update_host_and_detection_rate_charts(ml_host_t *host, collected_number
400
}
401
}
402
402
-void ml_update_training_statistics_chart(ml_training_thread_t *training_thread, const ml_training_stats_t &ts) {
403
+void ml_update_training_statistics_chart(ml_worker_t *worker, const ml_queue_stats_t &stats) {
404
/*
405
* queue stats
406
*/
407
{
407
- if (!training_thread->queue_stats_rs) {
408
+ if (!worker->queue_stats_rs) {
409
char id_buf[1024];
410
char name_buf[1024];
411
411
- snprintfz(id_buf, 1024, "training_queue_%zu_stats", training_thread->id);
412
- snprintfz(name_buf, 1024, "training_queue_%zu_stats", training_thread->id);
412
+ snprintfz(id_buf, 1024, "training_queue_%zu_stats", worker->id);
413
+ snprintfz(name_buf, 1024, "training_queue_%zu_stats", worker->id);
414
414
- training_thread->queue_stats_rs = rrdset_create(
415
+ worker->queue_stats_rs = rrdset_create(
416
localhost,
417
"netdata", // type
418
id_buf, // id
@@ -426,34 +427,34 @@ void ml_update_training_statistics_chart(ml_training_thread_t *training_thread,
427
localhost->rrd_update_every, // update_every
428
RRDSET_TYPE_LINE// chart_type
429
);
429
- rrdset_flag_set(training_thread->queue_stats_rs, RRDSET_FLAG_ANOMALY_DETECTION);
430
+ rrdset_flag_set(worker->queue_stats_rs, RRDSET_FLAG_ANOMALY_DETECTION);
431
431
- training_thread->queue_stats_queue_size_rd =
432
- rrddim_add(training_thread->queue_stats_rs, "queue_size", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
433
- training_thread->queue_stats_popped_items_rd =
434
- rrddim_add(training_thread->queue_stats_rs, "popped_items", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
432
+ worker->queue_stats_queue_size_rd =
433
+ rrddim_add(worker->queue_stats_rs, "queue_size", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
434
+ worker->queue_stats_popped_items_rd =
435
+ rrddim_add(worker->queue_stats_rs, "popped_items", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
436
}
437
437
- rrddim_set_by_pointer(training_thread->queue_stats_rs,
438
- training_thread->queue_stats_queue_size_rd, ts.queue_size);
439
- rrddim_set_by_pointer(training_thread->queue_stats_rs,
440
- training_thread->queue_stats_popped_items_rd, ts.num_popped_items);
438
+ rrddim_set_by_pointer(worker->queue_stats_rs,
439
+ worker->queue_stats_queue_size_rd, stats.queue_size);
440
+ rrddim_set_by_pointer(worker->queue_stats_rs,
441
+ worker->queue_stats_popped_items_rd, stats.num_popped_items);
442
442
- rrdset_done(training_thread->queue_stats_rs);
443
+ rrdset_done(worker->queue_stats_rs);
444
}
445
446
/*
447
* training stats
448
*/
449
{
449
- if (!training_thread->training_time_stats_rs) {
450
+ if (!worker->training_time_stats_rs) {
451
char id_buf[1024];
452
char name_buf[1024];
453
453
- snprintfz(id_buf, 1024, "training_queue_%zu_time_stats", training_thread->id);
454
- snprintfz(name_buf, 1024, "training_queue_%zu_time_stats", training_thread->id);
454
+ snprintfz(id_buf, 1024, "training_queue_%zu_time_stats", worker->id);
455
+ snprintfz(name_buf, 1024, "training_queue_%zu_time_stats", worker->id);
456
456
- training_thread->training_time_stats_rs = rrdset_create(
457
+ worker->training_time_stats_rs = rrdset_create(
458
localhost,
459
"netdata", // type
460
id_buf, // id
@@ -468,38 +469,38 @@ void ml_update_training_statistics_chart(ml_training_thread_t *training_thread,
469
localhost->rrd_update_every, // update_every
470
RRDSET_TYPE_LINE// chart_type
471
);
471
- rrdset_flag_set(training_thread->training_time_stats_rs, RRDSET_FLAG_ANOMALY_DETECTION);
472
-
473
- training_thread->training_time_stats_allotted_rd =
474
- rrddim_add(training_thread->training_time_stats_rs, "allotted", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
475
- training_thread->training_time_stats_consumed_rd =
476
- rrddim_add(training_thread->training_time_stats_rs, "consumed", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
477
- training_thread->training_time_stats_remaining_rd =
478
- rrddim_add(training_thread->training_time_stats_rs, "remaining", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
472
+ rrdset_flag_set(worker->training_time_stats_rs, RRDSET_FLAG_ANOMALY_DETECTION);
473
+
474
+ worker->training_time_stats_allotted_rd =
475
+ rrddim_add(worker->training_time_stats_rs, "allotted", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
476
+ worker->training_time_stats_consumed_rd =
477
+ rrddim_add(worker->training_time_stats_rs, "consumed", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
478
+ worker->training_time_stats_remaining_rd =
479
+ rrddim_add(worker->training_time_stats_rs, "remaining", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
480
}
481
481
- rrddim_set_by_pointer(training_thread->training_time_stats_rs,
482
- training_thread->training_time_stats_allotted_rd, ts.allotted_ut);
483
- rrddim_set_by_pointer(training_thread->training_time_stats_rs,
484
- training_thread->training_time_stats_consumed_rd, ts.consumed_ut);
485
- rrddim_set_by_pointer(training_thread->training_time_stats_rs,
486
- training_thread->training_time_stats_remaining_rd, ts.remaining_ut);
482
+ rrddim_set_by_pointer(worker->training_time_stats_rs,
483
+ worker->training_time_stats_allotted_rd, stats.allotted_ut);
484
+ rrddim_set_by_pointer(worker->training_time_stats_rs,
485
+ worker->training_time_stats_consumed_rd, stats.consumed_ut);
486
+ rrddim_set_by_pointer(worker->training_time_stats_rs,
487
+ worker->training_time_stats_remaining_rd, stats.remaining_ut);
488
488
- rrdset_done(training_thread->training_time_stats_rs);
489
+ rrdset_done(worker->training_time_stats_rs);
490
}
491
492
/*
493
* training result stats
494
*/
495
{
495
- if (!training_thread->training_results_rs) {
496
+ if (!worker->training_results_rs) {
497
char id_buf[1024];
498
char name_buf[1024];
499
499
- snprintfz(id_buf, 1024, "training_queue_%zu_results", training_thread->id);
500
- snprintfz(name_buf, 1024, "training_queue_%zu_results", training_thread->id);
500
+ snprintfz(id_buf, 1024, "training_queue_%zu_results", worker->id);
501
+ snprintfz(name_buf, 1024, "training_queue_%zu_results", worker->id);
502
502
- training_thread->training_results_rs = rrdset_create(
503
+ worker->training_results_rs = rrdset_create(
504
localhost,
505
"netdata", // type
506
id_buf, // id
@@ -514,37 +515,48 @@ void ml_update_training_statistics_chart(ml_training_thread_t *training_thread,
515
localhost->rrd_update_every, // update_every
516
RRDSET_TYPE_LINE// chart_type
517
);
517
- rrdset_flag_set(training_thread->training_results_rs, RRDSET_FLAG_ANOMALY_DETECTION);
518
-
519
- training_thread->training_results_ok_rd =
520
- rrddim_add(training_thread->training_results_rs, "ok", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
521
- training_thread->training_results_invalid_query_time_range_rd =
522
- rrddim_add(training_thread->training_results_rs, "invalid-queries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
523
- training_thread->training_results_not_enough_collected_values_rd =
524
- rrddim_add(training_thread->training_results_rs, "not-enough-values", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
525
- training_thread->training_results_null_acquired_dimension_rd =
526
- rrddim_add(training_thread->training_results_rs, "null-acquired-dimensions", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
527
- training_thread->training_results_chart_under_replication_rd =
528
- rrddim_add(training_thread->training_results_rs, "chart-under-replication", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
518
+ rrdset_flag_set(worker->training_results_rs, RRDSET_FLAG_ANOMALY_DETECTION);
519
+
520
+ worker->training_results_ok_rd =
521
+ rrddim_add(worker->training_results_rs, "ok", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
522
+ worker->training_results_invalid_query_time_range_rd =
523
+ rrddim_add(worker->training_results_rs, "invalid-queries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
524
+ worker->training_results_not_enough_collected_values_rd =
525
+ rrddim_add(worker->training_results_rs, "not-enough-values", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
526
+ worker->training_results_null_acquired_dimension_rd =
527
+ rrddim_add(worker->training_results_rs, "null-acquired-dimensions", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
528
+ worker->training_results_chart_under_replication_rd =
529
+ rrddim_add(worker->training_results_rs, "chart-under-replication", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
530
}
531
531
- rrddim_set_by_pointer(training_thread->training_results_rs,
532
- training_thread->training_results_ok_rd, ts.training_result_ok);
533
- rrddim_set_by_pointer(training_thread->training_results_rs,
534
- training_thread->training_results_invalid_query_time_range_rd, ts.training_result_invalid_query_time_range);
535
- rrddim_set_by_pointer(training_thread->training_results_rs,
536
- training_thread->training_results_not_enough_collected_values_rd, ts.training_result_not_enough_collected_values);
537
- rrddim_set_by_pointer(training_thread->training_results_rs,
538
- training_thread->training_results_null_acquired_dimension_rd, ts.training_result_null_acquired_dimension);
539
- rrddim_set_by_pointer(training_thread->training_results_rs,
540
- training_thread->training_results_chart_under_replication_rd, ts.training_result_chart_under_replication);
541
-
542
- rrdset_done(training_thread->training_results_rs);
532
+ rrddim_set_by_pointer(worker->training_results_rs,
533
+ worker->training_results_ok_rd, stats.item_result_ok);
534
+ rrddim_set_by_pointer(worker->training_results_rs,
535
+ worker->training_results_invalid_query_time_range_rd, stats.item_result_invalid_query_time_range);
536
+ rrddim_set_by_pointer(worker->training_results_rs,
537
+ worker->training_results_not_enough_collected_values_rd, stats.item_result_not_enough_collected_values);
538
+ rrddim_set_by_pointer(worker->training_results_rs,
539
+ worker->training_results_null_acquired_dimension_rd, stats.item_result_null_acquired_dimension);
540
+ rrddim_set_by_pointer(worker->training_results_rs,
541
+ worker->training_results_chart_under_replication_rd, stats.item_result_chart_under_replication);
542
+
543
+ rrdset_done(worker->training_results_rs);
544
}
545
}
546
546
-void ml_update_global_statistics_charts(uint64_t models_consulted) {
547
- if (Cfg.enable_statistics_charts) {
547
+void ml_update_global_statistics_charts(uint64_t models_consulted,
548
+ uint64_t models_received,
549
+ uint64_t models_sent,
550
+ uint64_t models_ignored,
551
+ uint64_t models_deserialization_failures,
552
+ uint64_t memory_consumption,
553
+ uint64_t memory_new,
554
+ uint64_t memory_delete)
555
+{
556
+ if (!Cfg.enable_statistics_charts)
557
+ return;
558
+
559
+ {
560
static RRDSET *st = NULL;
561
static RRDDIM *rd = NULL;
562
@@ -571,4 +583,98 @@ void ml_update_global_statistics_charts(uint64_t models_consulted) {
583
584
rrdset_done(st);
585
}
586
+
587
+ {
588
+ static RRDSET *st = NULL;
589
+ static RRDDIM *rd_received = NULL;
590
+ static RRDDIM *rd_sent = NULL;
591
+ static RRDDIM *rd_ignored = NULL;
592
+ static RRDDIM *rd_deserialization_failures = NULL;
593
+
594
+ if (unlikely(!st)) {
595
+ st = rrdset_create_localhost(
596
+ "netdata" // type
597
+ , "ml_models_streamed" // id
598
+ , NULL // name
599
+ , NETDATA_ML_CHART_FAMILY // family
600
+ , NULL // context
601
+ , "KMeans models streamed" // title
602
+ , "models" // units
603
+ , NETDATA_ML_PLUGIN // plugin
604
+ , NETDATA_ML_MODULE_DETECTION // module
605
+ , NETDATA_ML_CHART_PRIO_MACHINE_LEARNING_STATUS // priority
606
+ , localhost->rrd_update_every // update_every
607
+ , RRDSET_TYPE_LINE // chart_type
608
+ );
609
+
610
+ rd_received = rrddim_add(st, "received", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
611
+ rd_sent = rrddim_add(st, "sent", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
612
+ rd_ignored = rrddim_add(st, "ignored", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
613
+ rd_deserialization_failures = rrddim_add(st, "deserialization failures", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
614
+ }
615
+
616
+ rrddim_set_by_pointer(st, rd_received, (collected_number) models_received);
617
+ rrddim_set_by_pointer(st, rd_sent, (collected_number) models_sent);
618
+ rrddim_set_by_pointer(st, rd_ignored, (collected_number) models_ignored);
619
+ rrddim_set_by_pointer(st, rd_deserialization_failures, (collected_number) models_deserialization_failures);
620
+
621
+ rrdset_done(st);
622
+ }
623
+
624
+ {
625
+ static RRDSET *st = NULL;
626
+ static RRDDIM *rd_memory_consumption = NULL;
627
+
628
+ if (unlikely(!st)) {
629
+ st = rrdset_create_localhost(
630
+ "netdata" // type
631
+ , "ml_memory_used" // id
632
+ , NULL // name
633
+ , NETDATA_ML_CHART_FAMILY // family
634
+ , NULL // context
635
+ , "ML memory usage" // title
636
+ , "bytes" // units
637
+ , NETDATA_ML_PLUGIN // plugin
638
+ , NETDATA_ML_MODULE_DETECTION // module
639
+ , NETDATA_ML_CHART_PRIO_MACHINE_LEARNING_STATUS // priority
640
+ , localhost->rrd_update_every // update_every
641
+ , RRDSET_TYPE_LINE // chart_type
642
+ );
643
+
644
+ rd_memory_consumption = rrddim_add(st, "used", NULL, 1024, 1, RRD_ALGORITHM_ABSOLUTE);
645
+ }
646
+
647
+ rrddim_set_by_pointer(st, rd_memory_consumption, (collected_number) memory_consumption / (1024));
648
+ rrdset_done(st);
649
+ }
650
+
651
+ {
652
+ static RRDSET *st = NULL;
653
+ static RRDDIM *rd_memory_new = NULL;
654
+ static RRDDIM *rd_memory_delete = NULL;
655
+
656
+ if (unlikely(!st)) {
657
+ st = rrdset_create_localhost(
658
+ "netdata" // type
659
+ , "ml_memory_ops" // id
660
+ , NULL // name
661
+ , NETDATA_ML_CHART_FAMILY // family
662
+ , NULL // context
663
+ , "ML memory operations" // title
664
+ , "count" // units
665
+ , NETDATA_ML_PLUGIN // plugin
666
+ , NETDATA_ML_MODULE_DETECTION // module
667
+ , NETDATA_ML_CHART_PRIO_MACHINE_LEARNING_STATUS // priority
668
+ , localhost->rrd_update_every // update_every
669
+ , RRDSET_TYPE_LINE // chart_type
670
+ );
671
+
672
+ rd_memory_new = rrddim_add(st, "new", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
673
+ rd_memory_delete = rrddim_add(st, "delete", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
674
+ }
675
+
676
+ rrddim_set_by_pointer(st, rd_memory_new, (collected_number) memory_new);
677
+ rrddim_set_by_pointer(st, rd_memory_delete, (collected_number) memory_delete);
678
+ rrdset_done(st);
679
+ }
680
}
src/ml/ad_charts.h
+2
-2
@@ -3,12 +3,12 @@
3
#ifndef ML_ADCHARTS_H
4
#define ML_ADCHARTS_H
5
6
-#include "ml-private.h"
6
+#include "ml_private.h"
7
8
void ml_update_dimensions_chart(ml_host_t *host, const ml_machine_learning_stats_t &mls);
9
10
void ml_update_host_and_detection_rate_charts(ml_host_t *host, collected_number anomaly_rate);
11
12
-void ml_update_training_statistics_chart(ml_training_thread_t *training_thread, const ml_training_stats_t &ts);
12
+void ml_update_training_statistics_chart(ml_worker_t *worker, const ml_queue_stats_t &ts);
13
14
#endif /* ML_ADCHARTS_H */
src/ml/ml-dummy.c
+27
-14
@@ -1,6 +1,6 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#include "ml.h"
3
+#include "ml_public.h"
4
5
#if !defined(ENABLE_ML)
6
@@ -41,18 +41,6 @@ void ml_host_stop(RRDHOST *rh) {
41
UNUSED(rh);
42
}
43
44
-void ml_host_start_training_thread(RRDHOST *rh) {
45
- UNUSED(rh);
46
-}
47
-
48
-void ml_host_stop_training_thread(RRDHOST *rh) {
49
- UNUSED(rh);
50
-}
51
-
52
-void ml_host_cancel_training_thread(RRDHOST *rh) {
53
- UNUSED(rh);
54
-}
55
-
44
void ml_host_get_info(RRDHOST *rh, BUFFER *wb) {
45
UNUSED(rh);
46
UNUSED(wb);
@@ -105,8 +93,27 @@ int ml_dimension_load_models(RRDDIM *rd, sqlite3_stmt **stmp __maybe_unused) {
93
return 0;
94
}
95
108
-void ml_update_global_statistics_charts(uint64_t models_consulted) {
96
+void ml_dimension_received_anomaly(RRDDIM *rd, bool is_anomalous) {
97
+ UNUSED(rd);
98
+ UNUSED(is_anomalous);
99
+}
100
+
101
+void ml_update_global_statistics_charts(uint64_t models_consulted,
102
+ uint64_t models_received,
103
+ uint64_t models_sent,
104
+ uint64_t models_ignored,
105
+ uint64_t models_deserialization_failures,
106
+ uint64_t memory_consumption,
107
+ uint64_t memory_new,
108
+ uint64_t memory_delete) {
109
UNUSED(models_consulted);
110
+ UNUSED(models_received);
111
+ UNUSED(models_sent);
112
+ UNUSED(models_ignored);
113
+ UNUSED(models_deserialization_failures);
114
+ UNUSED(memory_consumption);
115
+ UNUSED(memory_new);
116
+ UNUSED(memory_delete);
117
}
118
119
bool ml_host_get_host_status(RRDHOST *rh __maybe_unused, struct ml_metrics_statistics *mlm) {
@@ -118,4 +125,10 @@ bool ml_host_running(RRDHOST *rh __maybe_unused) {
125
return false;
126
}
127
128
+bool ml_model_received_from_child(RRDHOST *host, const char *json) {
129
+ UNUSED(host);
130
+ UNUSED(json);
131
+ return false;
132
+}
133
+
134
#endif
src/ml/ml-private.h
deleted
-369
@@ -1,369 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#ifndef NETDATA_ML_PRIVATE_H
4
-#define NETDATA_ML_PRIVATE_H
5
-
6
-#include "dlib/dlib/matrix.h"
7
-
8
-// CentOS 7 shenanigans
9
-#include <cmath>
10
-using std::isfinite;
11
-
12
-#include "ml/ml.h"
13
-
14
-#include <vector>
15
-#include <queue>
16
-#include <unordered_map>
17
-
18
-typedef double calculated_number_t;
19
-typedef dlib::matrix<calculated_number_t, 6, 1> DSample;
20
-
21
-/*
22
- * Features
23
- */
24
-
25
-typedef struct {
26
- size_t diff_n;
27
- size_t smooth_n;
28
- size_t lag_n;
29
-
30
- calculated_number_t *dst;
31
- size_t dst_n;
32
-
33
- calculated_number_t *src;
34
- size_t src_n;
35
-
36
- std::vector<DSample> &preprocessed_features;
37
-} ml_features_t;
38
-
39
-/*
40
- * KMeans
41
- */
42
-
43
-typedef struct {
44
- std::vector<DSample> cluster_centers;
45
-
46
- calculated_number_t min_dist;
47
- calculated_number_t max_dist;
48
-
49
- uint32_t after;
50
- uint32_t before;
51
-} ml_kmeans_t;
52
-
53
-typedef struct machine_learning_stats_t {
54
- size_t num_machine_learning_status_enabled;
55
- size_t num_machine_learning_status_disabled_sp;
56
-
57
- size_t num_metric_type_constant;
58
- size_t num_metric_type_variable;
59
-
60
- size_t num_training_status_untrained;
61
- size_t num_training_status_pending_without_model;
62
- size_t num_training_status_trained;
63
- size_t num_training_status_pending_with_model;
64
- size_t num_training_status_silenced;
65
-
66
- size_t num_anomalous_dimensions;
67
- size_t num_normal_dimensions;
68
-} ml_machine_learning_stats_t;
69
-
70
-typedef struct training_stats_t {
71
- size_t queue_size;
72
- size_t num_popped_items;
73
-
74
- usec_t allotted_ut;
75
- usec_t consumed_ut;
76
- usec_t remaining_ut;
77
-
78
- size_t training_result_ok;
79
- size_t training_result_invalid_query_time_range;
80
- size_t training_result_not_enough_collected_values;
81
- size_t training_result_null_acquired_dimension;
82
- size_t training_result_chart_under_replication;
83
-} ml_training_stats_t;
84
-
85
-enum ml_metric_type {
86
- // The dimension has constant values, no need to train
87
- METRIC_TYPE_CONSTANT,
88
-
89
- // The dimension's values fluctuate, we need to generate a model
90
- METRIC_TYPE_VARIABLE,
91
-};
92
-
93
-enum ml_machine_learning_status {
94
- // Enable training/prediction
95
- MACHINE_LEARNING_STATUS_ENABLED,
96
-
97
- // Disable because configuration pattern matches the chart's id
98
- MACHINE_LEARNING_STATUS_DISABLED_DUE_TO_EXCLUDED_CHART,
99
-};
100
-
101
-enum ml_training_status {
102
- // We don't have a model for this dimension
103
- TRAINING_STATUS_UNTRAINED,
104
-
105
- // Request for training sent, but we don't have any models yet
106
- TRAINING_STATUS_PENDING_WITHOUT_MODEL,
107
-
108
- // Request to update existing models sent
109
- TRAINING_STATUS_PENDING_WITH_MODEL,
110
-
111
- // Have a valid, up-to-date model
112
- TRAINING_STATUS_TRAINED,
113
-
114
- // Have a valid, up-to-date model that is silenced because its too noisy
115
- TRAINING_STATUS_SILENCED,
116
-};
117
-
118
-enum ml_training_result {
119
- // We managed to create a KMeans model
120
- TRAINING_RESULT_OK,
121
-
122
- // Could not query DB with a correct time range
123
- TRAINING_RESULT_INVALID_QUERY_TIME_RANGE,
124
-
125
- // Did not gather enough data from DB to run KMeans
126
- TRAINING_RESULT_NOT_ENOUGH_COLLECTED_VALUES,
127
-
128
- // Acquired a null dimension
129
- TRAINING_RESULT_NULL_ACQUIRED_DIMENSION,
130
-
131
- // Chart is under replication
132
- TRAINING_RESULT_CHART_UNDER_REPLICATION,
133
-};
134
-
135
-typedef struct {
136
- // Chart/dimension we want to train
137
- char machine_guid[GUID_LEN + 1];
138
- STRING *chart_id;
139
- STRING *dimension_id;
140
-
141
- // Creation time of request
142
- time_t request_time;
143
-
144
- // First/last entry of this dimension in DB
145
- // at the point the request was made
146
- time_t first_entry_on_request;
147
- time_t last_entry_on_request;
148
-} ml_training_request_t;
149
-
150
-typedef struct {
151
- // Time when the request for this response was made
152
- time_t request_time;
153
-
154
- // First/last entry of the dimension in DB when generating the request
155
- time_t first_entry_on_request;
156
- time_t last_entry_on_request;
157
-
158
- // First/last entry of the dimension in DB when generating the response
159
- time_t first_entry_on_response;
160
- time_t last_entry_on_response;
161
-
162
- // After/Before timestamps of our DB query
163
- time_t query_after_t;
164
- time_t query_before_t;
165
-
166
- // Actual after/before returned by the DB query ops
167
- time_t db_after_t;
168
- time_t db_before_t;
169
-
170
- // Number of doubles returned by the DB query
171
- size_t collected_values;
172
-
173
- // Number of values we return to the caller
174
- size_t total_values;
175
-
176
- // Result of training response
177
- enum ml_training_result result;
178
-} ml_training_response_t;
179
-
180
-/*
181
- * Queue
182
-*/
183
-
184
-typedef struct {
185
- std::queue<ml_training_request_t> internal;
186
- netdata_mutex_t mutex;
187
- pthread_cond_t cond_var;
188
- std::atomic<bool> exit;
189
-} ml_queue_t;
190
-
191
-typedef struct {
192
- RRDDIM *rd;
193
-
194
- enum ml_metric_type mt;
195
- enum ml_training_status ts;
196
- enum ml_machine_learning_status mls;
197
-
198
- ml_training_response_t tr;
199
- time_t last_training_time;
200
-
201
- std::vector<calculated_number_t> cns;
202
-
203
- std::vector<ml_kmeans_t> km_contexts;
204
- SPINLOCK slock;
205
- ml_kmeans_t kmeans;
206
- std::vector<DSample> feature;
207
-
208
- uint32_t suppression_window_counter;
209
- uint32_t suppression_anomaly_counter;
210
-} ml_dimension_t;
211
-
212
-typedef struct {
213
- RRDSET *rs;
214
- ml_machine_learning_stats_t mls;
215
-} ml_chart_t;
216
-
217
-void ml_chart_update_dimension(ml_chart_t *chart, ml_dimension_t *dim, bool is_anomalous);
218
-
219
-typedef struct {
220
- RRDDIM *rd;
221
- size_t normal_dimensions;
222
- size_t anomalous_dimensions;
223
-} ml_type_anomaly_rate_t;
224
-
225
-typedef struct {
226
- RRDHOST *rh;
227
-
228
- std::atomic<bool> ml_running;
229
-
230
- ml_machine_learning_stats_t mls;
231
-
232
- calculated_number_t host_anomaly_rate;
233
-
234
- netdata_mutex_t mutex;
235
-
236
- ml_queue_t *training_queue;
237
-
238
- /*
239
- * bookkeeping for anomaly detection charts
240
- */
241
-
242
- RRDSET *ml_running_rs;
243
- RRDDIM *ml_running_rd;
244
-
245
- RRDSET *machine_learning_status_rs;
246
- RRDDIM *machine_learning_status_enabled_rd;
247
- RRDDIM *machine_learning_status_disabled_sp_rd;
248
-
249
- RRDSET *metric_type_rs;
250
- RRDDIM *metric_type_constant_rd;
251
- RRDDIM *metric_type_variable_rd;
252
-
253
- RRDSET *training_status_rs;
254
- RRDDIM *training_status_untrained_rd;
255
- RRDDIM *training_status_pending_without_model_rd;
256
- RRDDIM *training_status_trained_rd;
257
- RRDDIM *training_status_pending_with_model_rd;
258
- RRDDIM *training_status_silenced_rd;
259
-
260
- RRDSET *dimensions_rs;
261
- RRDDIM *dimensions_anomalous_rd;
262
- RRDDIM *dimensions_normal_rd;
263
-
264
- RRDSET *anomaly_rate_rs;
265
- RRDDIM *anomaly_rate_rd;
266
-
267
- RRDSET *detector_events_rs;
268
- RRDDIM *detector_events_above_threshold_rd;
269
- RRDDIM *detector_events_new_anomaly_event_rd;
270
-
271
- RRDSET *type_anomaly_rate_rs;
272
- SPINLOCK type_anomaly_rate_spinlock;
273
- std::unordered_map<STRING *, ml_type_anomaly_rate_t> type_anomaly_rate;
274
-} ml_host_t;
275
-
276
-typedef struct {
277
- nd_uuid_t metric_uuid;
278
- ml_kmeans_t kmeans;
279
-} ml_model_info_t;
280
-
281
-typedef struct {
282
- size_t id;
283
- ND_THREAD *nd_thread;
284
- netdata_mutex_t nd_mutex;
285
-
286
- ml_queue_t *training_queue;
287
- ml_training_stats_t training_stats;
288
-
289
- calculated_number_t *training_cns;
290
- calculated_number_t *scratch_training_cns;
291
- std::vector<DSample> training_samples;
292
-
293
- std::vector<ml_model_info_t> pending_model_info;
294
-
295
- RRDSET *queue_stats_rs;
296
- RRDDIM *queue_stats_queue_size_rd;
297
- RRDDIM *queue_stats_popped_items_rd;
298
-
299
- RRDSET *training_time_stats_rs;
300
- RRDDIM *training_time_stats_allotted_rd;
301
- RRDDIM *training_time_stats_consumed_rd;
302
- RRDDIM *training_time_stats_remaining_rd;
303
-
304
- RRDSET *training_results_rs;
305
- RRDDIM *training_results_ok_rd;
306
- RRDDIM *training_results_invalid_query_time_range_rd;
307
- RRDDIM *training_results_not_enough_collected_values_rd;
308
- RRDDIM *training_results_null_acquired_dimension_rd;
309
- RRDDIM *training_results_chart_under_replication_rd;
310
-
311
- size_t num_db_transactions;
312
- size_t num_models_to_prune;
313
-} ml_training_thread_t;
314
-
315
-typedef struct {
316
- int enable_anomaly_detection;
317
-
318
- unsigned max_train_samples;
319
- unsigned min_train_samples;
320
- unsigned train_every;
321
-
322
- unsigned num_models_to_use;
323
- unsigned delete_models_older_than;
324
-
325
- unsigned db_engine_anomaly_rate_every;
326
-
327
- unsigned diff_n;
328
- unsigned smooth_n;
329
- unsigned lag_n;
330
-
331
- double random_sampling_ratio;
332
- unsigned max_kmeans_iters;
333
-
334
- double dimension_anomaly_score_threshold;
335
-
336
- double host_anomaly_rate_threshold;
337
- RRDR_TIME_GROUPING anomaly_detection_grouping_method;
338
- time_t anomaly_detection_query_duration;
339
-
340
- bool stream_anomaly_detection_charts;
341
-
342
- std::string hosts_to_skip;
343
- SIMPLE_PATTERN *sp_host_to_skip;
344
-
345
- std::string charts_to_skip;
346
- SIMPLE_PATTERN *sp_charts_to_skip;
347
-
348
- std::vector<uint32_t> random_nums;
349
-
350
- ND_THREAD *detection_thread;
351
- std::atomic<bool> detection_stop;
352
-
353
- size_t num_training_threads;
354
- size_t flush_models_batch_size;
355
-
356
- std::vector<ml_training_thread_t> training_threads;
357
- std::atomic<bool> training_stop;
358
-
359
- size_t suppression_window;
360
- size_t suppression_threshold;
361
-
362
- bool enable_statistics_charts;
363
-} ml_config_t;
364
-
365
-void ml_config_load(ml_config_t *cfg);
366
-
367
-extern ml_config_t Cfg;
368
-
369
-#endif /* NETDATA_ML_PRIVATE_H */
src/ml/ml.cc
+389
-1001
@@ -1,16 +1,12 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#include "dlib/dlib/clustering.h"
3
+#include "ml_private.h"
4
5
-#include "ml-private.h"
6
-
7
-#include <random>
5
+#include <array>
6
7
#include "ad_charts.h"
8
#include "database/sqlite/sqlite3.h"
9
12
-#define ML_METADATA_VERSION 2
13
-
10
#define WORKER_TRAIN_QUEUE_POP 0
11
#define WORKER_TRAIN_ACQUIRE_DIMENSION 1
12
#define WORKER_TRAIN_QUERY 2
@@ -20,315 +16,44 @@
16
#define WORKER_TRAIN_UPDATE_HOST 6
17
#define WORKER_TRAIN_FLUSH_MODELS 7
18
23
-static sqlite3 *db = NULL;
19
+sqlite3 *ml_db = NULL;
20
static netdata_mutex_t db_mutex = NETDATA_MUTEX_INITIALIZER;
21
26
-/*
27
- * Functions to convert enums to strings
28
-*/
29
-
30
-__attribute__((unused)) static const char *
31
-ml_machine_learning_status_to_string(enum ml_machine_learning_status mls)
32
-{
33
- switch (mls) {
34
- case MACHINE_LEARNING_STATUS_ENABLED:
35
- return "enabled";
36
- case MACHINE_LEARNING_STATUS_DISABLED_DUE_TO_EXCLUDED_CHART:
37
- return "disabled-sp";
38
- default:
39
- return "unknown";
40
- }
41
-}
42
-
43
-__attribute__((unused)) static const char *
44
-ml_metric_type_to_string(enum ml_metric_type mt)
45
-{
46
- switch (mt) {
47
- case METRIC_TYPE_CONSTANT:
48
- return "constant";
49
- case METRIC_TYPE_VARIABLE:
50
- return "variable";
51
- default:
52
- return "unknown";
53
- }
54
-}
55
-
56
-__attribute__((unused)) static const char *
57
-ml_training_status_to_string(enum ml_training_status ts)
58
-{
59
- switch (ts) {
60
- case TRAINING_STATUS_PENDING_WITH_MODEL:
61
- return "pending-with-model";
62
- case TRAINING_STATUS_PENDING_WITHOUT_MODEL:
63
- return "pending-without-model";
64
- case TRAINING_STATUS_TRAINED:
65
- return "trained";
66
- case TRAINING_STATUS_UNTRAINED:
67
- return "untrained";
68
- case TRAINING_STATUS_SILENCED:
69
- return "silenced";
70
- default:
71
- return "unknown";
72
- }
73
-}
74
-
75
-__attribute__((unused)) static const char *
76
-ml_training_result_to_string(enum ml_training_result tr)
77
-{
78
- switch (tr) {
79
- case TRAINING_RESULT_OK:
80
- return "ok";
81
- case TRAINING_RESULT_INVALID_QUERY_TIME_RANGE:
82
- return "invalid-query";
83
- case TRAINING_RESULT_NOT_ENOUGH_COLLECTED_VALUES:
84
- return "missing-values";
85
- case TRAINING_RESULT_NULL_ACQUIRED_DIMENSION:
86
- return "null-acquired-dim";
87
- case TRAINING_RESULT_CHART_UNDER_REPLICATION:
88
- return "chart-under-replication";
89
- default:
90
- return "unknown";
91
- }
92
-}
93
-
94
-/*
95
- * Features
96
-*/
97
-
98
-// subtract elements that are `diff_n` positions apart
99
-static void
100
-ml_features_diff(ml_features_t *features)
101
-{
102
- if (features->diff_n == 0)
103
- return;
104
-
105
- for (size_t idx = 0; idx != (features->src_n - features->diff_n); idx++) {
106
- size_t high = (features->src_n - 1) - idx;
107
- size_t low = high - features->diff_n;
108
-
109
- features->dst[low] = features->src[high] - features->src[low];
110
- }
111
-
112
- size_t n = features->src_n - features->diff_n;
113
- memcpy(features->src, features->dst, n * sizeof(calculated_number_t));
114
-
115
- for (size_t idx = features->src_n - features->diff_n; idx != features->src_n; idx++)
116
- features->src[idx] = 0.0;
117
-}
118
-
119
-// a function that computes the window average of an array inplace
120
-static void
121
-ml_features_smooth(ml_features_t *features)
122
-{
123
- calculated_number_t sum = 0.0;
124
-
125
- size_t idx = 0;
126
- for (; idx != features->smooth_n - 1; idx++)
127
- sum += features->src[idx];
128
-
129
- for (; idx != (features->src_n - features->diff_n); idx++) {
130
- sum += features->src[idx];
131
- calculated_number_t prev_cn = features->src[idx - (features->smooth_n - 1)];
132
- features->src[idx - (features->smooth_n - 1)] = sum / features->smooth_n;
133
- sum -= prev_cn;
134
- }
135
-
136
- for (idx = 0; idx != features->smooth_n; idx++)
137
- features->src[(features->src_n - 1) - idx] = 0.0;
138
-}
139
-
140
-// create lag'd vectors out of the preprocessed buffer
141
-static void
142
-ml_features_lag(ml_features_t *features)
143
-{
144
- size_t n = features->src_n - features->diff_n - features->smooth_n + 1 - features->lag_n;
145
- features->preprocessed_features.resize(n);
146
-
147
- unsigned target_num_samples = Cfg.max_train_samples * Cfg.random_sampling_ratio;
148
- double sampling_ratio = std::min(static_cast<double>(target_num_samples) / n, 1.0);
149
-
150
- uint32_t max_mt = std::numeric_limits<uint32_t>::max();
151
- uint32_t cutoff = static_cast<double>(max_mt) * sampling_ratio;
152
-
153
- size_t sample_idx = 0;
154
-
155
- for (size_t idx = 0; idx != n; idx++) {
156
- DSample &DS = features->preprocessed_features[sample_idx++];
157
- DS.set_size(features->lag_n);
158
-
159
- if (Cfg.random_nums[idx] > cutoff) {
160
- sample_idx--;
161
- continue;
162
- }
163
-
164
- for (size_t feature_idx = 0; feature_idx != features->lag_n + 1; feature_idx++)
165
- DS(feature_idx) = features->src[idx + feature_idx];
166
- }
167
-
168
- features->preprocessed_features.resize(sample_idx);
169
-}
170
-
171
-static void
172
-ml_features_preprocess(ml_features_t *features)
173
-{
174
- ml_features_diff(features);
175
- ml_features_smooth(features);
176
- ml_features_lag(features);
177
-}
178
-
179
-/*
180
- * KMeans
181
-*/
182
-
183
-static void
184
-ml_kmeans_init(ml_kmeans_t *kmeans)
185
-{
186
- kmeans->cluster_centers.reserve(2);
187
- kmeans->min_dist = std::numeric_limits<calculated_number_t>::max();
188
- kmeans->max_dist = std::numeric_limits<calculated_number_t>::min();
189
-}
190
-
191
-static void
192
-ml_kmeans_train(ml_kmeans_t *kmeans, const ml_features_t *features, time_t after, time_t before)
193
-{
194
- kmeans->after = (uint32_t) after;
195
- kmeans->before = (uint32_t) before;
196
-
197
- kmeans->min_dist = std::numeric_limits<calculated_number_t>::max();
198
- kmeans->max_dist = std::numeric_limits<calculated_number_t>::min();
199
-
200
- kmeans->cluster_centers.clear();
201
-
202
- dlib::pick_initial_centers(2, kmeans->cluster_centers, features->preprocessed_features);
203
- dlib::find_clusters_using_kmeans(features->preprocessed_features, kmeans->cluster_centers, Cfg.max_kmeans_iters);
204
-
205
- for (const auto &preprocessed_feature : features->preprocessed_features) {
206
- calculated_number_t mean_dist = 0.0;
207
-
208
- for (const auto &cluster_center : kmeans->cluster_centers) {
209
- mean_dist += dlib::length(cluster_center - preprocessed_feature);
210
- }
211
-
212
- mean_dist /= kmeans->cluster_centers.size();
213
-
214
- if (mean_dist < kmeans->min_dist)
215
- kmeans->min_dist = mean_dist;
216
-
217
- if (mean_dist > kmeans->max_dist)
218
- kmeans->max_dist = mean_dist;
219
- }
220
-}
221
-
222
-static calculated_number_t
223
-ml_kmeans_anomaly_score(const ml_kmeans_t *kmeans, const DSample &DS)
224
-{
225
- calculated_number_t mean_dist = 0.0;
226
- for (const auto &CC: kmeans->cluster_centers)
227
- mean_dist += dlib::length(CC - DS);
228
-
229
- mean_dist /= kmeans->cluster_centers.size();
230
-
231
- if (kmeans->max_dist == kmeans->min_dist)
232
- return 0.0;
233
-
234
- calculated_number_t anomaly_score = 100.0 * std::abs((mean_dist - kmeans->min_dist) / (kmeans->max_dist - kmeans->min_dist));
235
- return (anomaly_score > 100.0) ? 100.0 : anomaly_score;
236
-}
237
-
238
-/*
239
- * Queue
240
-*/
241
-
242
-static ml_queue_t *
243
-ml_queue_init()
244
-{
245
- ml_queue_t *q = new ml_queue_t();
246
-
247
- netdata_mutex_init(&q->mutex);
248
- pthread_cond_init(&q->cond_var, NULL);
249
- q->exit = false;
250
- return q;
251
-}
252
-
253
-static void
254
-ml_queue_destroy(ml_queue_t *q)
255
-{
256
- netdata_mutex_destroy(&q->mutex);
257
- pthread_cond_destroy(&q->cond_var);
258
- delete q;
259
-}
260
-
261
-static void
262
-ml_queue_push(ml_queue_t *q, const ml_training_request_t req)
263
-{
264
- netdata_mutex_lock(&q->mutex);
265
- q->internal.push(req);
266
- pthread_cond_signal(&q->cond_var);
267
- netdata_mutex_unlock(&q->mutex);
268
-}
269
-
270
-static ml_training_request_t
271
-ml_queue_pop(ml_queue_t *q)
272
-{
273
- netdata_mutex_lock(&q->mutex);
274
-
275
- ml_training_request_t req = {
276
- {'\0'}, // machine_guid
277
- NULL, // chart id
278
- NULL, // dimension id
279
- 0, // current time
280
- 0, // first entry
281
- 0 // last entry
282
- };
22
+typedef struct {
23
+ // Time when the request for this response was made
24
+ time_t request_time;
25
284
- while (q->internal.empty()) {
285
- pthread_cond_wait(&q->cond_var, &q->mutex);
26
+ // First/last entry of the dimension in DB when generating the request
27
+ time_t first_entry_on_request;
28
+ time_t last_entry_on_request;
29
287
- if (q->exit) {
288
- netdata_mutex_unlock(&q->mutex);
30
+ // First/last entry of the dimension in DB when generating the response
31
+ time_t first_entry_on_response;
32
+ time_t last_entry_on_response;
33
290
- // We return a dummy request because the queue has been signaled
291
- return req;
292
- }
293
- }
34
+ // After/Before timestamps of our DB query
35
+ time_t query_after_t;
36
+ time_t query_before_t;
37
295
- req = q->internal.front();
296
- q->internal.pop();
38
+ // Actual after/before returned by the DB query ops
39
+ time_t db_after_t;
40
+ time_t db_before_t;
41
298
- netdata_mutex_unlock(&q->mutex);
299
- return req;
300
-}
301
-
302
-static size_t
303
-ml_queue_size(ml_queue_t *q)
304
-{
305
- netdata_mutex_lock(&q->mutex);
306
- size_t size = q->internal.size();
307
- netdata_mutex_unlock(&q->mutex);
308
- return size;
309
-}
310
-
311
-static void
312
-ml_queue_signal(ml_queue_t *q)
313
-{
314
- netdata_mutex_lock(&q->mutex);
315
- q->exit = true;
316
- pthread_cond_signal(&q->cond_var);
317
- netdata_mutex_unlock(&q->mutex);
318
-}
42
+ // Number of doubles returned by the DB query
43
+ size_t collected_values;
44
320
-/*
321
- * Dimension
322
-*/
45
+ // Number of values we return to the caller
46
+ size_t total_values;
47
+} ml_training_response_t;
48
324
-static std::pair<calculated_number_t *, ml_training_response_t>
325
-ml_dimension_calculated_numbers(ml_training_thread_t *training_thread, ml_dimension_t *dim, const ml_training_request_t &training_request)
49
+static std::pair<enum ml_worker_result, ml_training_response_t>
50
+ml_dimension_calculated_numbers(ml_worker_t *worker, ml_dimension_t *dim, const ml_request_create_new_model_t &req)
51
{
52
ml_training_response_t training_response = {};
53
329
- training_response.request_time = training_request.request_time;
330
- training_response.first_entry_on_request = training_request.first_entry_on_request;
331
- training_response.last_entry_on_request = training_request.last_entry_on_request;
54
+ training_response.request_time = req.request_time;
55
+ training_response.first_entry_on_request = req.first_entry_on_request;
56
+ training_response.last_entry_on_request = req.last_entry_on_request;
57
58
training_response.first_entry_on_response = rrddim_first_entry_s_of_tier(dim->rd, 0);
59
training_response.last_entry_on_response = rrddim_last_entry_s_of_tier(dim->rd, 0);
@@ -344,13 +69,11 @@ ml_dimension_calculated_numbers(ml_training_thread_t *training_thread, ml_dimens
69
);
70
71
if (training_response.query_after_t >= training_response.query_before_t) {
347
- training_response.result = TRAINING_RESULT_INVALID_QUERY_TIME_RANGE;
348
- return { NULL, training_response };
72
+ return { ML_WORKER_RESULT_INVALID_QUERY_TIME_RANGE, training_response };
73
}
74
75
if (rrdset_is_replicating(dim->rd->rrdset)) {
352
- training_response.result = TRAINING_RESULT_CHART_UNDER_REPLICATION;
353
- return { NULL, training_response };
76
+ return { ML_WORKER_RESULT_CHART_UNDER_REPLICATION, training_response };
77
}
78
79
/*
@@ -363,7 +86,7 @@ ml_dimension_calculated_numbers(ml_training_thread_t *training_thread, ml_dimens
86
STORAGE_PRIORITY_BEST_EFFORT);
87
88
size_t idx = 0;
366
- memset(training_thread->training_cns, 0, sizeof(calculated_number_t) * max_n * (Cfg.lag_n + 1));
89
+ memset(worker->training_cns, 0, sizeof(calculated_number_t) * max_n * (Cfg.lag_n + 1));
90
calculated_number_t last_value = std::numeric_limits<calculated_number_t>::quiet_NaN();
91
92
while (!storage_engine_query_is_finished(&handle)) {
@@ -380,33 +103,31 @@ ml_dimension_calculated_numbers(ml_training_thread_t *training_thread, ml_dimens
103
training_response.db_after_t = timestamp;
104
training_response.db_before_t = timestamp;
105
383
- training_thread->training_cns[idx] = value;
384
- last_value = training_thread->training_cns[idx];
106
+ worker->training_cns[idx] = value;
107
+ last_value = worker->training_cns[idx];
108
training_response.collected_values++;
109
} else
387
- training_thread->training_cns[idx] = last_value;
110
+ worker->training_cns[idx] = last_value;
111
112
idx++;
113
}
114
storage_engine_query_finalize(&handle);
115
393
- global_statistics_ml_query_completed(/* points_read */ idx);
116
+ telemetry_queries_ml_query_completed(/* points_read */ idx);
117
118
training_response.total_values = idx;
119
if (training_response.collected_values < min_n) {
397
- training_response.result = TRAINING_RESULT_NOT_ENOUGH_COLLECTED_VALUES;
398
- return { NULL, training_response };
120
+ return { ML_WORKER_RESULT_NOT_ENOUGH_COLLECTED_VALUES, training_response };
121
}
122
123
// Find first non-NaN value.
402
- for (idx = 0; std::isnan(training_thread->training_cns[idx]); idx++, training_response.total_values--) { }
124
+ for (idx = 0; std::isnan(worker->training_cns[idx]); idx++, training_response.total_values--) { }
125
126
// Overwrite NaN values.
127
if (idx != 0)
406
- memmove(training_thread->training_cns, &training_thread->training_cns[idx], sizeof(calculated_number_t) * training_response.total_values);
128
+ memmove(worker->training_cns, &worker->training_cns[idx], sizeof(calculated_number_t) * training_response.total_values);
129
408
- training_response.result = TRAINING_RESULT_OK;
409
- return { training_thread->training_cns, training_response };
130
+ return { ML_WORKER_RESULT_OK, training_response };
131
}
132
133
const char *db_models_create_table =
@@ -443,19 +164,19 @@ const char *db_models_prune =
164
"WHERE after < @after LIMIT @n;";
165
166
static int
446
-ml_dimension_add_model(const nd_uuid_t *metric_uuid, const ml_kmeans_t *km)
167
+ml_dimension_add_model(const nd_uuid_t *metric_uuid, const ml_kmeans_inlined_t *inlined_km)
168
{
169
static __thread sqlite3_stmt *res = NULL;
170
int param = 0;
171
int rc = 0;
172
452
- if (unlikely(!db)) {
173
+ if (unlikely(!ml_db)) {
174
error_report("Database has not been initialized");
175
return 1;
176
}
177
178
if (unlikely(!res)) {
458
- rc = prepare_statement(db, db_models_add_model, &res);
179
+ rc = prepare_statement(ml_db, db_models_add_model, &res);
180
if (unlikely(rc != SQLITE_OK)) {
181
error_report("Failed to prepare statement to store model, rc = %d", rc);
182
return 1;
@@ -466,26 +187,23 @@ ml_dimension_add_model(const nd_uuid_t *metric_uuid, const ml_kmeans_t *km)
187
if (unlikely(rc != SQLITE_OK))
188
goto bind_fail;
189
469
- rc = sqlite3_bind_int(res, ++param, (int) km->after);
190
+ rc = sqlite3_bind_int(res, ++param, (int) inlined_km->after);
191
if (unlikely(rc != SQLITE_OK))
192
goto bind_fail;
193
473
- rc = sqlite3_bind_int(res, ++param, (int) km->before);
194
+ rc = sqlite3_bind_int(res, ++param, (int) inlined_km->before);
195
if (unlikely(rc != SQLITE_OK))
196
goto bind_fail;
197
477
- rc = sqlite3_bind_double(res, ++param, km->min_dist);
198
+ rc = sqlite3_bind_double(res, ++param, inlined_km->min_dist);
199
if (unlikely(rc != SQLITE_OK))
200
goto bind_fail;
201
481
- rc = sqlite3_bind_double(res, ++param, km->max_dist);
202
+ rc = sqlite3_bind_double(res, ++param, inlined_km->max_dist);
203
if (unlikely(rc != SQLITE_OK))
204
goto bind_fail;
205
485
- if (km->cluster_centers.size() != 2)
486
- fatal("Expected 2 cluster centers, got %zu", km->cluster_centers.size());
487
-
488
- for (const DSample &ds : km->cluster_centers) {
206
+ for (const DSample &ds : inlined_km->cluster_centers) {
207
if (ds.size() != 6)
208
fatal("Expected dsample with 6 dimensions, got %ld", ds.size());
209
@@ -526,13 +244,13 @@ ml_dimension_delete_models(const nd_uuid_t *metric_uuid, time_t before)
244
int rc = 0;
245
int param = 0;
246
529
- if (unlikely(!db)) {
247
+ if (unlikely(!ml_db)) {
248
error_report("Database has not been initialized");
249
return 1;
250
}
251
252
if (unlikely(!res)) {
535
- rc = prepare_statement(db, db_models_delete, &res);
253
+ rc = prepare_statement(ml_db, db_models_delete, &res);
254
if (unlikely(rc != SQLITE_OK)) {
255
error_report("Failed to prepare statement to delete models, rc = %d", rc);
256
return rc;
@@ -576,13 +294,13 @@ ml_prune_old_models(size_t num_models_to_prune)
294
int rc = 0;
295
int param = 0;
296
579
- if (unlikely(!db)) {
297
+ if (unlikely(!ml_db)) {
298
error_report("Database has not been initialized");
299
return 1;
300
}
301
302
if (unlikely(!res)) {
585
- rc = prepare_statement(db, db_models_prune, &res);
303
+ rc = prepare_statement(ml_db, db_models_prune, &res);
304
if (unlikely(rc != SQLITE_OK)) {
305
error_report("Failed to prepare statement to prune models, rc = %d", rc);
306
return rc;
@@ -639,13 +357,13 @@ int ml_dimension_load_models(RRDDIM *rd, sqlite3_stmt **active_stmt) {
357
int rc = 0;
358
int param = 0;
359
642
- if (unlikely(!db)) {
360
+ if (unlikely(!ml_db)) {
361
error_report("Database has not been initialized");
362
return 1;
363
}
364
365
if (unlikely(!res)) {
648
- rc = sqlite3_prepare_v2(db, db_models_load, -1, &res, NULL);
366
+ rc = sqlite3_prepare_v2(ml_db, db_models_load, -1, &res, NULL);
367
if (unlikely(rc != SQLITE_OK)) {
368
error_report("Failed to prepare statement to load models, rc = %d", rc);
369
return 1;
@@ -692,7 +410,7 @@ int ml_dimension_load_models(RRDDIM *rd, sqlite3_stmt **active_stmt) {
410
km.cluster_centers[1](4) = sqlite3_column_double(res, 16);
411
km.cluster_centers[1](5) = sqlite3_column_double(res, 17);
412
695
- dim->km_contexts.push_back(km);
413
+ dim->km_contexts.emplace_back(km);
414
}
415
416
if (!dim->km_contexts.empty()) {
@@ -721,14 +439,210 @@ bind_fail:
439
return 1;
440
}
441
724
-static enum ml_training_result
725
-ml_dimension_train_model(ml_training_thread_t *training_thread, ml_dimension_t *dim, const ml_training_request_t &training_request)
442
+static void ml_dimension_serialize_kmeans(const ml_dimension_t *dim, BUFFER *wb)
443
+{
444
+ RRDDIM *rd = dim->rd;
445
+
446
+ buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_DEFAULT);
447
+ buffer_json_member_add_string(wb, "version", "1");
448
+ buffer_json_member_add_string(wb, "machine-guid", rd->rrdset->rrdhost->machine_guid);
449
+ buffer_json_member_add_string(wb, "chart", rrdset_id(rd->rrdset));
450
+ buffer_json_member_add_string(wb, "dimension", rrddim_id(rd));
451
+
452
+ buffer_json_member_add_object(wb, "model");
453
+ ml_kmeans_serialize(&dim->km_contexts.back(), wb);
454
+ buffer_json_object_close(wb);
455
+
456
+ buffer_json_finalize(wb);
457
+}
458
+
459
+bool
460
+ml_dimension_deserialize_kmeans(const char *json_str)
461
+{
462
+ if (!json_str) {
463
+ netdata_log_error("Failed to deserialize kmeans: json string is null");
464
+ return false;
465
+ }
466
+
467
+ struct json_object *root = json_tokener_parse(json_str);
468
+ if (!root) {
469
+ netdata_log_error("Failed to deserialize kmeans: json parsing failed");
470
+ return false;
471
+ }
472
+
473
+ // Check the version
474
+ {
475
+ struct json_object *tmp_obj;
476
+ if (!json_object_object_get_ex(root, "version", &tmp_obj)) {
477
+ netdata_log_error("Failed to deserialize kmeans: missing key 'version'");
478
+ json_object_put(root);
479
+ return false;
480
+ }
481
+ if (!json_object_is_type(tmp_obj, json_type_string)) {
482
+ netdata_log_error("Failed to deserialize kmeans: failed to parse string for 'version'");
483
+ json_object_put(root);
484
+ return false;
485
+ }
486
+ const char *version = json_object_get_string(tmp_obj);
487
+
488
+ if (strcmp(version, "1")) {
489
+ netdata_log_error("Failed to deserialize kmeans: expected version 1");
490
+ json_object_put(root);
491
+ return false;
492
+ }
493
+ }
494
+
495
+ // Get the value of each key
496
+ std::array<const char *, 3> values;
497
+ {
498
+ std::array<const char *, 3> keys = {
499
+ "machine-guid",
500
+ "chart",
501
+ "dimension",
502
+ };
503
+
504
+ struct json_object *tmp_obj;
505
+ for (size_t i = 0; i != keys.size(); i++) {
506
+ if (!json_object_object_get_ex(root, keys[i], &tmp_obj)) {
507
+ netdata_log_error("Failed to deserialize kmeans: missing key '%s'", keys[i]);
508
+ json_object_put(root);
509
+ return false;
510
+ }
511
+ if (!json_object_is_type(tmp_obj, json_type_string)) {
512
+ netdata_log_error("Failed to deserialize kmeans: missing string value for key '%s'", keys[i]);
513
+ json_object_put(root);
514
+ return false;
515
+ }
516
+ values[i] = json_object_get_string(tmp_obj);
517
+ }
518
+ }
519
+
520
+ DimensionLookupInfo DLI(values[0], values[1], values[2]);
521
+
522
+ // Parse the kmeans model
523
+ ml_kmeans_inlined_t inlined_km;
524
+ {
525
+ struct json_object *kmeans_obj;
526
+ if (!json_object_object_get_ex(root, "model", &kmeans_obj)) {
527
+ netdata_log_error("Failed to deserialize kmeans: missing key 'model'");
528
+ json_object_put(root);
529
+ return false;
530
+ }
531
+ if (!json_object_is_type(kmeans_obj, json_type_object)) {
532
+ netdata_log_error("Failed to deserialize kmeans: failed to parse object for 'model'");
533
+ json_object_put(root);
534
+ return false;
535
+ }
536
+
537
+ if (!ml_kmeans_deserialize(&inlined_km, kmeans_obj)) {
538
+ json_object_put(root);
539
+ return false;
540
+ }
541
+ }
542
+
543
+ AcquiredDimension AcqDim(DLI);
544
+ if (!AcqDim.acquired()) {
545
+ netdata_log_error("Failed to deserialize kmeans: could not acquire dimension (machine-guid: %s, dimension: '%s.%s', reason: %s)",
546
+ DLI.machineGuid(), DLI.chartId(), DLI.dimensionId(), AcqDim.acquire_failure());
547
+ json_object_put(root);
548
+ return false;
549
+ }
550
+
551
+ ml_dimension_t *Dim = reinterpret_cast<ml_dimension_t *>(AcqDim.dimension());
552
+ if (!Dim) {
553
+ telemetry_ml_models_ignored();
554
+ return true;
555
+ }
556
+
557
+ ml_queue_item_t item;
558
+ item.type = ML_QUEUE_ITEM_TYPE_ADD_EXISTING_MODEL;
559
+ item.add_existing_model = {
560
+ DLI, inlined_km
561
+ };
562
+ ml_queue_push(AcqDim.queue(), item);
563
+
564
+ json_object_put(root);
565
+ return true;
566
+}
567
+
568
+static void ml_dimension_stream_kmeans(const ml_dimension_t *dim)
569
+{
570
+ struct sender_state *s = dim->rd->rrdset->rrdhost->sender;
571
+ if (!s)
572
+ return;
573
+
574
+ if(!stream_sender_has_capabilities(dim->rd->rrdset->rrdhost, STREAM_CAP_ML_MODELS))
575
+ return;
576
+
577
+ CLEAN_BUFFER *payload = buffer_create(0, NULL);
578
+ ml_dimension_serialize_kmeans(dim, payload);
579
+
580
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
581
+
582
+ buffer_sprintf(
583
+ wb, PLUGINSD_KEYWORD_JSON " " PLUGINSD_KEYWORD_JSON_CMD_ML_MODEL "\n%s\n" PLUGINSD_KEYWORD_JSON_END "\n",
584
+ buffer_tostring(payload));
585
+
586
+ sender_commit_clean_buffer(s, wb, STREAM_TRAFFIC_TYPE_METADATA);
587
+ telemetry_ml_models_sent();
588
+}
589
+
590
+static void ml_dimension_update_models(ml_worker_t *worker, ml_dimension_t *dim)
591
+{
592
+ worker_is_busy(WORKER_TRAIN_UPDATE_MODELS);
593
+
594
+ spinlock_lock(&dim->slock);
595
+
596
+ if (dim->km_contexts.size() < Cfg.num_models_to_use) {
597
+ dim->km_contexts.emplace_back(dim->kmeans);
598
+ } else {
599
+ bool can_drop_middle_km = false;
600
+
601
+ if (Cfg.num_models_to_use > 2) {
602
+ const ml_kmeans_inlined_t *old_km = &dim->km_contexts[dim->km_contexts.size() - 1];
603
+ const ml_kmeans_inlined_t *middle_km = &dim->km_contexts[dim->km_contexts.size() - 2];
604
+ const ml_kmeans_t *new_km = &dim->kmeans;
605
+
606
+ can_drop_middle_km = (middle_km->after < old_km->before) &&
607
+ (middle_km->before > new_km->after);
608
+ }
609
+
610
+ if (can_drop_middle_km) {
611
+ dim->km_contexts.back() = dim->kmeans;
612
+ } else {
613
+ std::rotate(std::begin(dim->km_contexts), std::begin(dim->km_contexts) + 1, std::end(dim->km_contexts));
614
+ dim->km_contexts[dim->km_contexts.size() - 1] = dim->kmeans;
615
+ }
616
+ }
617
+
618
+ dim->mt = METRIC_TYPE_CONSTANT;
619
+ dim->ts = TRAINING_STATUS_TRAINED;
620
+
621
+ dim->suppression_anomaly_counter = 0;
622
+ dim->suppression_window_counter = 0;
623
+
624
+ dim->last_training_time = rrddim_last_entry_s(dim->rd);
625
+
626
+ // Add the newly generated model to the list of pending models to flush
627
+ ml_model_info_t model_info;
628
+ uuid_copy(model_info.metric_uuid, dim->rd->metric_uuid);
629
+ model_info.inlined_kmeans = dim->km_contexts.back();
630
+ worker->pending_model_info.push_back(model_info);
631
+
632
+ ml_dimension_stream_kmeans(dim);
633
+
634
+ spinlock_unlock(&dim->slock);
635
+}
636
+
637
+static enum ml_worker_result
638
+ml_dimension_train_model(ml_worker_t *worker, ml_dimension_t *dim, const ml_request_create_new_model_t &req)
639
{
640
worker_is_busy(WORKER_TRAIN_QUERY);
728
- auto P = ml_dimension_calculated_numbers(training_thread, dim, training_request);
641
+ auto P = ml_dimension_calculated_numbers(worker, dim, req);
642
+ ml_worker_result worker_result = P.first;
643
ml_training_response_t training_response = P.second;
644
731
- if (training_response.result != TRAINING_RESULT_OK) {
645
+ if (worker_result != ML_WORKER_RESULT_OK) {
646
spinlock_lock(&dim->slock);
647
648
dim->mt = METRIC_TYPE_CONSTANT;
@@ -746,80 +660,36 @@ ml_dimension_train_model(ml_training_thread_t *training_thread, ml_dimension_t *
660
661
dim->suppression_anomaly_counter = 0;
662
dim->suppression_window_counter = 0;
749
- dim->tr = training_response;
663
664
dim->last_training_time = training_response.last_entry_on_response;
752
- enum ml_training_result result = training_response.result;
665
666
spinlock_unlock(&dim->slock);
667
756
- return result;
668
+ return worker_result;
669
}
670
671
// compute kmeans
672
worker_is_busy(WORKER_TRAIN_KMEANS);
673
{
762
- memcpy(training_thread->scratch_training_cns, training_thread->training_cns,
674
+ memcpy(worker->scratch_training_cns, worker->training_cns,
675
training_response.total_values * sizeof(calculated_number_t));
676
677
ml_features_t features = {
678
Cfg.diff_n, Cfg.smooth_n, Cfg.lag_n,
767
- training_thread->scratch_training_cns, training_response.total_values,
768
- training_thread->training_cns, training_response.total_values,
769
- training_thread->training_samples
679
+ worker->scratch_training_cns, training_response.total_values,
680
+ worker->training_cns, training_response.total_values,
681
+ worker->training_samples
682
};
683
ml_features_preprocess(&features);
684
685
ml_kmeans_init(&dim->kmeans);
774
- ml_kmeans_train(&dim->kmeans, &features, training_response.query_after_t, training_response.query_before_t);
686
+ ml_kmeans_train(&dim->kmeans, &features, Cfg.max_kmeans_iters, training_response.query_after_t, training_response.query_before_t);
687
}
688
689
// update models
778
- worker_is_busy(WORKER_TRAIN_UPDATE_MODELS);
779
- {
780
- spinlock_lock(&dim->slock);
781
-
782
- if (dim->km_contexts.size() < Cfg.num_models_to_use) {
783
- dim->km_contexts.push_back(std::move(dim->kmeans));
784
- } else {
785
- bool can_drop_middle_km = false;
786
-
787
- if (Cfg.num_models_to_use > 2) {
788
- const ml_kmeans_t *old_km = &dim->km_contexts[dim->km_contexts.size() - 1];
789
- const ml_kmeans_t *middle_km = &dim->km_contexts[dim->km_contexts.size() - 2];
790
- const ml_kmeans_t *new_km = &dim->kmeans;
791
-
792
- can_drop_middle_km = (middle_km->after < old_km->before) &&
793
- (middle_km->before > new_km->after);
794
- }
795
-
796
- if (can_drop_middle_km) {
797
- dim->km_contexts.back() = dim->kmeans;
798
- } else {
799
- std::rotate(std::begin(dim->km_contexts), std::begin(dim->km_contexts) + 1, std::end(dim->km_contexts));
800
- dim->km_contexts[dim->km_contexts.size() - 1] = std::move(dim->kmeans);
801
- }
802
- }
803
-
804
- dim->mt = METRIC_TYPE_CONSTANT;
805
- dim->ts = TRAINING_STATUS_TRAINED;
806
-
807
- dim->suppression_anomaly_counter = 0;
808
- dim->suppression_window_counter = 0;
690
+ ml_dimension_update_models(worker, dim);
691
810
- dim->tr = training_response;
811
- dim->last_training_time = rrddim_last_entry_s(dim->rd);
812
-
813
- // Add the newly generated model to the list of pending models to flush
814
- ml_model_info_t model_info;
815
- uuid_copy(model_info.metric_uuid, dim->rd->metric_uuid);
816
- model_info.kmeans = dim->km_contexts.back();
817
- training_thread->pending_model_info.push_back(model_info);
818
-
819
- spinlock_unlock(&dim->slock);
820
- }
821
-
822
- return training_response.result;
692
+ return worker_result;
693
}
694
695
static void
@@ -853,21 +723,27 @@ ml_dimension_schedule_for_training(ml_dimension_t *dim, time_t curr_time)
723
}
724
725
if (schedule_for_training) {
856
- ml_training_request_t req;
726
+ ml_request_create_new_model_t req;
727
858
- memcpy(req.machine_guid, dim->rd->rrdset->rrdhost->machine_guid, GUID_LEN + 1);
859
- req.chart_id = string_dup(dim->rd->rrdset->id);
860
- req.dimension_id = string_dup(dim->rd->id);
728
+ req.DLI = DimensionLookupInfo(
729
+ &dim->rd->rrdset->rrdhost->machine_guid[0],
730
+ dim->rd->rrdset->id,
731
+ dim->rd->id
732
+ );
733
req.request_time = curr_time;
734
req.first_entry_on_request = rrddim_first_entry_s(dim->rd);
735
req.last_entry_on_request = rrddim_last_entry_s(dim->rd);
736
737
ml_host_t *host = (ml_host_t *) dim->rd->rrdset->rrdhost->ml_host;
866
- ml_queue_push(host->training_queue, req);
738
+
739
+ ml_queue_item_t item;
740
+ item.type = ML_QUEUE_ITEM_TYPE_CREATE_NEW_MODEL;
741
+ item.create_new_model = req;
742
+ ml_queue_push(host->queue, item);
743
}
744
}
745
870
-static bool
746
+bool
747
ml_dimension_predict(ml_dimension_t *dim, time_t curr_time, calculated_number_t value, bool exists)
748
{
749
// Nothing to do if ML is disabled for this dimension
@@ -954,7 +830,7 @@ ml_dimension_predict(ml_dimension_t *dim, time_t curr_time, calculated_number_t
830
continue;
831
832
if (anomaly_score < (100 * Cfg.dimension_anomaly_score_threshold)) {
957
- global_statistics_ml_models_consulted(models_consulted);
833
+ telemetry_ml_models_consulted(models_consulted);
834
spinlock_unlock(&dim->slock);
835
return false;
836
}
@@ -971,7 +847,7 @@ ml_dimension_predict(ml_dimension_t *dim, time_t curr_time, calculated_number_t
847
848
spinlock_unlock(&dim->slock);
849
974
- global_statistics_ml_models_consulted(models_consulted);
850
+ telemetry_ml_models_consulted(models_consulted);
851
return sum;
852
}
853
@@ -1139,74 +1015,7 @@ ml_host_detect_once(ml_host_t *host)
1015
ml_update_host_and_detection_rate_charts(host, host->host_anomaly_rate * 10000.0);
1016
}
1017
1142
-typedef struct {
1143
- RRDHOST_ACQUIRED *acq_rh;
1144
- RRDSET_ACQUIRED *acq_rs;
1145
- RRDDIM_ACQUIRED *acq_rd;
1146
- ml_dimension_t *dim;
1147
-} ml_acquired_dimension_t;
1148
-
1149
-static ml_acquired_dimension_t
1150
-ml_acquired_dimension_get(char *machine_guid, STRING *chart_id, STRING *dimension_id)
1151
-{
1152
- RRDHOST_ACQUIRED *acq_rh = NULL;
1153
- RRDSET_ACQUIRED *acq_rs = NULL;
1154
- RRDDIM_ACQUIRED *acq_rd = NULL;
1155
- ml_dimension_t *dim = NULL;
1156
-
1157
- rrd_rdlock();
1158
-
1159
- acq_rh = rrdhost_find_and_acquire(machine_guid);
1160
- if (acq_rh) {
1161
- RRDHOST *rh = rrdhost_acquired_to_rrdhost(acq_rh);
1162
- if (rh && !rrdhost_flag_check(rh, RRDHOST_FLAG_ORPHAN | RRDHOST_FLAG_ARCHIVED)) {
1163
- acq_rs = rrdset_find_and_acquire(rh, string2str(chart_id));
1164
- if (acq_rs) {
1165
- RRDSET *rs = rrdset_acquired_to_rrdset(acq_rs);
1166
- if (rs && !rrdset_flag_check(rs, RRDSET_FLAG_OBSOLETE)) {
1167
- acq_rd = rrddim_find_and_acquire(rs, string2str(dimension_id));
1168
- if (acq_rd) {
1169
- RRDDIM *rd = rrddim_acquired_to_rrddim(acq_rd);
1170
- if (rd)
1171
- dim = (ml_dimension_t *) rd->ml_dimension;
1172
- }
1173
- }
1174
- }
1175
- }
1176
- }
1177
-
1178
- rrd_rdunlock();
1179
-
1180
- ml_acquired_dimension_t acq_dim = {
1181
- acq_rh, acq_rs, acq_rd, dim
1182
- };
1183
-
1184
- return acq_dim;
1185
-}
1186
-
1187
-static void
1188
-ml_acquired_dimension_release(ml_acquired_dimension_t acq_dim)
1189
-{
1190
- if (acq_dim.acq_rd)
1191
- rrddim_acquired_release(acq_dim.acq_rd);
1192
-
1193
- if (acq_dim.acq_rs)
1194
- rrdset_acquired_release(acq_dim.acq_rs);
1195
-
1196
- if (acq_dim.acq_rh)
1197
- rrdhost_acquired_release(acq_dim.acq_rh);
1198
-}
1199
-
1200
-static enum ml_training_result
1201
-ml_acquired_dimension_train(ml_training_thread_t *training_thread, ml_acquired_dimension_t acq_dim, const ml_training_request_t &tr)
1202
-{
1203
- if (!acq_dim.dim)
1204
- return TRAINING_RESULT_NULL_ACQUIRED_DIMENSION;
1205
-
1206
- return ml_dimension_train_model(training_thread, acq_dim.dim, tr);
1207
-}
1208
-
1209
-static void *
1018
+void *
1019
ml_detect_main(void *arg)
1020
{
1021
UNUSED(arg);
@@ -1239,33 +1048,33 @@ ml_detect_main(void *arg)
1048
1049
if (Cfg.enable_statistics_charts) {
1050
// collect and update training thread stats
1242
- for (size_t idx = 0; idx != Cfg.num_training_threads; idx++) {
1243
- ml_training_thread_t *training_thread = &Cfg.training_threads[idx];
1051
+ for (size_t idx = 0; idx != Cfg.num_worker_threads; idx++) {
1052
+ ml_worker_t *worker = &Cfg.workers[idx];
1053
1245
- netdata_mutex_lock(&training_thread->nd_mutex);
1246
- ml_training_stats_t training_stats = training_thread->training_stats;
1247
- training_thread->training_stats = {};
1248
- netdata_mutex_unlock(&training_thread->nd_mutex);
1054
+ netdata_mutex_lock(&worker->nd_mutex);
1055
+ ml_queue_stats_t queue_stats = worker->queue_stats;
1056
+ worker->queue_stats = {};
1057
+ netdata_mutex_unlock(&worker->nd_mutex);
1058
1059
// calc the avg values
1251
- if (training_stats.num_popped_items) {
1252
- training_stats.queue_size /= training_stats.num_popped_items;
1253
- training_stats.allotted_ut /= training_stats.num_popped_items;
1254
- training_stats.consumed_ut /= training_stats.num_popped_items;
1255
- training_stats.remaining_ut /= training_stats.num_popped_items;
1060
+ if (queue_stats.num_popped_items) {
1061
+ queue_stats.queue_size /= queue_stats.num_popped_items;
1062
+ queue_stats.allotted_ut /= queue_stats.num_popped_items;
1063
+ queue_stats.consumed_ut /= queue_stats.num_popped_items;
1064
+ queue_stats.remaining_ut /= queue_stats.num_popped_items;
1065
} else {
1257
- training_stats.queue_size = ml_queue_size(training_thread->training_queue);
1258
- training_stats.consumed_ut = 0;
1259
- training_stats.remaining_ut = training_stats.allotted_ut;
1260
-
1261
- training_stats.training_result_ok = 0;
1262
- training_stats.training_result_invalid_query_time_range = 0;
1263
- training_stats.training_result_not_enough_collected_values = 0;
1264
- training_stats.training_result_null_acquired_dimension = 0;
1265
- training_stats.training_result_chart_under_replication = 0;
1066
+ queue_stats.queue_size = ml_queue_size(worker->queue);
1067
+ queue_stats.consumed_ut = 0;
1068
+ queue_stats.remaining_ut = queue_stats.allotted_ut;
1069
+
1070
+ queue_stats.item_result_ok = 0;
1071
+ queue_stats.item_result_invalid_query_time_range = 0;
1072
+ queue_stats.item_result_not_enough_collected_values = 0;
1073
+ queue_stats.item_result_null_acquired_dimension = 0;
1074
+ queue_stats.item_result_chart_under_replication = 0;
1075
}
1076
1268
- ml_update_training_statistics_chart(training_thread, training_stats);
1077
+ ml_update_training_statistics_chart(worker, queue_stats);
1078
}
1079
}
1080
}
@@ -1274,382 +1083,101 @@ ml_detect_main(void *arg)
1083
return NULL;
1084
}
1085
1277
-/*
1278
- * Public API
1279
-*/
1280
-
1281
-bool ml_capable()
1282
-{
1283
- return true;
1284
-}
1285
-
1286
-bool ml_enabled(RRDHOST *rh)
1287
-{
1288
- if (!rh)
1289
- return false;
1290
-
1291
- if (!Cfg.enable_anomaly_detection)
1292
- return false;
1293
-
1294
- if (simple_pattern_matches(Cfg.sp_host_to_skip, rrdhost_hostname(rh)))
1295
- return false;
1296
-
1297
- return true;
1298
-}
1299
-
1300
-bool ml_streaming_enabled()
1301
-{
1302
- return Cfg.stream_anomaly_detection_charts;
1303
-}
1304
-
1305
-void ml_host_new(RRDHOST *rh)
1306
-{
1307
- if (!ml_enabled(rh))
1308
- return;
1309
-
1310
- ml_host_t *host = new ml_host_t();
1311
-
1312
- host->rh = rh;
1313
- host->mls = ml_machine_learning_stats_t();
1314
- host->host_anomaly_rate = 0.0;
1315
- host->anomaly_rate_rs = NULL;
1316
-
1317
- static std::atomic<size_t> times_called(0);
1318
- host->training_queue = Cfg.training_threads[times_called++ % Cfg.num_training_threads].training_queue;
1319
-
1320
- netdata_mutex_init(&host->mutex);
1321
- spinlock_init(&host->type_anomaly_rate_spinlock);
1322
-
1323
- host->ml_running = true;
1324
- rh->ml_host = (rrd_ml_host_t *) host;
1325
-}
1326
-
1327
-void ml_host_delete(RRDHOST *rh)
1328
-{
1329
- ml_host_t *host = (ml_host_t *) rh->ml_host;
1330
- if (!host)
1331
- return;
1332
-
1333
- netdata_mutex_destroy(&host->mutex);
1334
-
1335
- delete host;
1336
- rh->ml_host = NULL;
1337
-}
1338
-
1339
-void ml_host_start(RRDHOST *rh) {
1340
- ml_host_t *host = (ml_host_t *) rh->ml_host;
1341
- if (!host)
1342
- return;
1343
-
1344
- host->ml_running = true;
1345
-}
1346
-
1347
-void ml_host_stop(RRDHOST *rh) {
1348
- ml_host_t *host = (ml_host_t *) rh->ml_host;
1349
- if (!host || !host->ml_running)
1350
- return;
1351
-
1352
- netdata_mutex_lock(&host->mutex);
1353
-
1354
- // reset host stats
1355
- host->mls = ml_machine_learning_stats_t();
1356
-
1357
- // reset charts/dims
1358
- void *rsp = NULL;
1359
- rrdset_foreach_read(rsp, host->rh) {
1360
- RRDSET *rs = static_cast<RRDSET *>(rsp);
1361
-
1362
- ml_chart_t *chart = (ml_chart_t *) rs->ml_chart;
1363
- if (!chart)
1364
- continue;
1365
-
1366
- // reset chart
1367
- chart->mls = ml_machine_learning_stats_t();
1368
-
1369
- void *rdp = NULL;
1370
- rrddim_foreach_read(rdp, rs) {
1371
- RRDDIM *rd = static_cast<RRDDIM *>(rdp);
1372
-
1373
- ml_dimension_t *dim = (ml_dimension_t *) rd->ml_dimension;
1374
- if (!dim)
1375
- continue;
1376
-
1377
- spinlock_lock(&dim->slock);
1378
-
1379
- // reset dim
1380
- // TODO: should we drop in-mem models, or mark them as stale? Is it
1381
- // okay to resume training straight away?
1382
-
1383
- dim->mt = METRIC_TYPE_CONSTANT;
1384
- dim->ts = TRAINING_STATUS_UNTRAINED;
1385
- dim->last_training_time = 0;
1386
- dim->suppression_anomaly_counter = 0;
1387
- dim->suppression_window_counter = 0;
1388
- dim->cns.clear();
1389
-
1390
- ml_kmeans_init(&dim->kmeans);
1391
-
1392
- spinlock_unlock(&dim->slock);
1393
- }
1394
- rrddim_foreach_done(rdp);
1395
- }
1396
- rrdset_foreach_done(rsp);
1397
-
1398
- netdata_mutex_unlock(&host->mutex);
1399
-
1400
- host->ml_running = false;
1401
-}
1402
-
1403
-void ml_host_get_info(RRDHOST *rh, BUFFER *wb)
1404
-{
1405
- ml_host_t *host = (ml_host_t *) rh->ml_host;
1406
- if (!host) {
1407
- buffer_json_member_add_boolean(wb, "enabled", false);
1408
- return;
1409
- }
1410
-
1411
- buffer_json_member_add_uint64(wb, "version", 1);
1412
-
1413
- buffer_json_member_add_boolean(wb, "enabled", Cfg.enable_anomaly_detection);
1414
-
1415
- buffer_json_member_add_uint64(wb, "min-train-samples", Cfg.min_train_samples);
1416
- buffer_json_member_add_uint64(wb, "max-train-samples", Cfg.max_train_samples);
1417
- buffer_json_member_add_uint64(wb, "train-every", Cfg.train_every);
1418
-
1419
- buffer_json_member_add_uint64(wb, "diff-n", Cfg.diff_n);
1420
- buffer_json_member_add_uint64(wb, "smooth-n", Cfg.smooth_n);
1421
- buffer_json_member_add_uint64(wb, "lag-n", Cfg.lag_n);
1422
-
1423
- buffer_json_member_add_double(wb, "random-sampling-ratio", Cfg.random_sampling_ratio);
1424
- buffer_json_member_add_uint64(wb, "max-kmeans-iters", Cfg.random_sampling_ratio);
1425
-
1426
- buffer_json_member_add_double(wb, "dimension-anomaly-score-threshold", Cfg.dimension_anomaly_score_threshold);
1427
-
1428
- buffer_json_member_add_string(wb, "anomaly-detection-grouping-method", time_grouping_id2txt(Cfg.anomaly_detection_grouping_method));
1429
-
1430
- buffer_json_member_add_int64(wb, "anomaly-detection-query-duration", Cfg.anomaly_detection_query_duration);
1431
-
1432
- buffer_json_member_add_string(wb, "hosts-to-skip", Cfg.hosts_to_skip.c_str());
1433
- buffer_json_member_add_string(wb, "charts-to-skip", Cfg.charts_to_skip.c_str());
1434
-}
1435
-
1436
-void ml_host_get_detection_info(RRDHOST *rh, BUFFER *wb)
1437
-{
1438
- ml_host_t *host = (ml_host_t *) rh->ml_host;
1439
- if (!host)
1440
- return;
1441
-
1442
- netdata_mutex_lock(&host->mutex);
1443
-
1444
- buffer_json_member_add_uint64(wb, "version", 2);
1445
- buffer_json_member_add_uint64(wb, "ml-running", host->ml_running);
1446
- buffer_json_member_add_uint64(wb, "anomalous-dimensions", host->mls.num_anomalous_dimensions);
1447
- buffer_json_member_add_uint64(wb, "normal-dimensions", host->mls.num_normal_dimensions);
1448
- buffer_json_member_add_uint64(wb, "total-dimensions", host->mls.num_anomalous_dimensions +
1449
- host->mls.num_normal_dimensions);
1450
- buffer_json_member_add_uint64(wb, "trained-dimensions", host->mls.num_training_status_trained +
1451
- host->mls.num_training_status_pending_with_model);
1452
- netdata_mutex_unlock(&host->mutex);
1453
-}
1454
-
1455
-bool ml_host_get_host_status(RRDHOST *rh, struct ml_metrics_statistics *mlm) {
1456
- ml_host_t *host = (ml_host_t *) rh->ml_host;
1457
- if (!host) {
1458
- memset(mlm, 0, sizeof(*mlm));
1459
- return false;
1460
- }
1461
-
1462
- netdata_mutex_lock(&host->mutex);
1463
-
1464
- mlm->anomalous = host->mls.num_anomalous_dimensions;
1465
- mlm->normal = host->mls.num_normal_dimensions;
1466
- mlm->trained = host->mls.num_training_status_trained + host->mls.num_training_status_pending_with_model;
1467
- mlm->pending = host->mls.num_training_status_untrained + host->mls.num_training_status_pending_without_model;
1468
- mlm->silenced = host->mls.num_training_status_silenced;
1469
-
1470
- netdata_mutex_unlock(&host->mutex);
1471
-
1472
- return true;
1473
-}
1474
-
1475
-bool ml_host_running(RRDHOST *rh) {
1476
- ml_host_t *host = (ml_host_t *) rh->ml_host;
1477
- if(!host)
1478
- return false;
1479
-
1480
- return true;
1481
-}
1482
-
1483
-void ml_host_get_models(RRDHOST *rh, BUFFER *wb)
1484
-{
1485
- UNUSED(rh);
1486
- UNUSED(wb);
1487
-
1488
- // TODO: To be implemented
1489
- netdata_log_error("Fetching KMeans models is not supported yet");
1490
-}
1491
-
1492
-void ml_chart_new(RRDSET *rs)
1493
-{
1494
- ml_host_t *host = (ml_host_t *) rs->rrdhost->ml_host;
1495
- if (!host)
1496
- return;
1497
-
1498
- ml_chart_t *chart = new ml_chart_t();
1499
-
1500
- chart->rs = rs;
1501
- chart->mls = ml_machine_learning_stats_t();
1502
-
1503
- rs->ml_chart = (rrd_ml_chart_t *) chart;
1504
-}
1505
-
1506
-void ml_chart_delete(RRDSET *rs)
1507
-{
1508
- ml_host_t *host = (ml_host_t *) rs->rrdhost->ml_host;
1509
- if (!host)
1510
- return;
1511
-
1512
- ml_chart_t *chart = (ml_chart_t *) rs->ml_chart;
1513
-
1514
- delete chart;
1515
- rs->ml_chart = NULL;
1516
-}
1517
-
1518
-bool ml_chart_update_begin(RRDSET *rs)
1519
-{
1520
- ml_chart_t *chart = (ml_chart_t *) rs->ml_chart;
1521
- if (!chart)
1522
- return false;
1523
-
1524
- chart->mls = {};
1525
- return true;
1526
-}
1527
-
1528
-void ml_chart_update_end(RRDSET *rs)
1529
-{
1530
- ml_chart_t *chart = (ml_chart_t *) rs->ml_chart;
1531
- if (!chart)
1532
- return;
1533
-}
1534
-
1535
-void ml_dimension_new(RRDDIM *rd)
1536
-{
1537
- ml_chart_t *chart = (ml_chart_t *) rd->rrdset->ml_chart;
1538
- if (!chart)
1539
- return;
1540
-
1541
- ml_dimension_t *dim = new ml_dimension_t();
1542
-
1543
- dim->rd = rd;
1544
-
1545
- dim->mt = METRIC_TYPE_CONSTANT;
1546
- dim->ts = TRAINING_STATUS_UNTRAINED;
1547
- dim->last_training_time = 0;
1548
- dim->suppression_anomaly_counter = 0;
1549
- dim->suppression_window_counter = 0;
1550
-
1551
- ml_kmeans_init(&dim->kmeans);
1552
-
1553
- if (simple_pattern_matches(Cfg.sp_charts_to_skip, rrdset_name(rd->rrdset)))
1554
- dim->mls = MACHINE_LEARNING_STATUS_DISABLED_DUE_TO_EXCLUDED_CHART;
1555
- else
1556
- dim->mls = MACHINE_LEARNING_STATUS_ENABLED;
1557
-
1558
- spinlock_init(&dim->slock);
1559
-
1560
- dim->km_contexts.reserve(Cfg.num_models_to_use);
1561
-
1562
- rd->ml_dimension = (rrd_ml_dimension_t *) dim;
1563
-
1564
- metaqueue_ml_load_models(rd);
1565
-}
1566
-
1567
-void ml_dimension_delete(RRDDIM *rd)
1568
-{
1569
- ml_dimension_t *dim = (ml_dimension_t *) rd->ml_dimension;
1570
- if (!dim)
1571
- return;
1572
-
1573
- delete dim;
1574
- rd->ml_dimension = NULL;
1575
-}
1576
-
1577
-bool ml_dimension_is_anomalous(RRDDIM *rd, time_t curr_time, double value, bool exists)
1578
-{
1579
- ml_dimension_t *dim = (ml_dimension_t *) rd->ml_dimension;
1580
- if (!dim)
1581
- return false;
1582
-
1583
- ml_host_t *host = (ml_host_t *) rd->rrdset->rrdhost->ml_host;
1584
- if (!host->ml_running)
1585
- return false;
1586
-
1587
- ml_chart_t *chart = (ml_chart_t *) rd->rrdset->ml_chart;
1588
-
1589
- bool is_anomalous = ml_dimension_predict(dim, curr_time, value, exists);
1590
- ml_chart_update_dimension(chart, dim, is_anomalous);
1591
-
1592
- return is_anomalous;
1593
-}
1594
-
1595
-static void ml_flush_pending_models(ml_training_thread_t *training_thread) {
1086
+static void ml_flush_pending_models(ml_worker_t *worker) {
1087
int op_no = 1;
1088
1089
// begin transaction
1599
- int rc = db_execute(db, "BEGIN TRANSACTION;");
1090
+ int rc = db_execute(ml_db, "BEGIN TRANSACTION;");
1091
1092
// add/delete models
1093
if (!rc) {
1094
op_no++;
1095
1605
- for (const auto &pending_model: training_thread->pending_model_info) {
1096
+ for (const auto &pending_model: worker->pending_model_info) {
1097
if (!rc)
1607
- rc = ml_dimension_add_model(&pending_model.metric_uuid, &pending_model.kmeans);
1098
+ rc = ml_dimension_add_model(&pending_model.metric_uuid, &pending_model.inlined_kmeans);
1099
1100
if (!rc)
1610
- rc = ml_dimension_delete_models(&pending_model.metric_uuid, pending_model.kmeans.before - (Cfg.num_models_to_use * Cfg.train_every));
1101
+ rc = ml_dimension_delete_models(&pending_model.metric_uuid, pending_model.inlined_kmeans.before - (Cfg.num_models_to_use * Cfg.train_every));
1102
}
1103
}
1104
1105
// prune old models
1106
if (!rc) {
1616
- if ((training_thread->num_db_transactions % 64) == 0) {
1617
- rc = ml_prune_old_models(training_thread->num_models_to_prune);
1107
+ if ((worker->num_db_transactions % 64) == 0) {
1108
+ rc = ml_prune_old_models(worker->num_models_to_prune);
1109
if (!rc)
1619
- training_thread->num_models_to_prune = 0;
1110
+ worker->num_models_to_prune = 0;
1111
}
1112
}
1113
1114
// commit transaction
1115
if (!rc) {
1116
op_no++;
1626
- rc = db_execute(db, "COMMIT TRANSACTION;");
1117
+ rc = db_execute(ml_db, "COMMIT TRANSACTION;");
1118
}
1119
1120
// rollback transaction on failure
1121
if (rc) {
1122
netdata_log_error("Trying to rollback ML transaction because it failed with rc=%d, op_no=%d", rc, op_no);
1123
op_no++;
1633
- rc = db_execute(db, "ROLLBACK;");
1124
+ rc = db_execute(ml_db, "ROLLBACK;");
1125
if (rc)
1126
netdata_log_error("ML transaction rollback failed with rc=%d", rc);
1127
}
1128
1129
if (!rc) {
1639
- training_thread->num_db_transactions++;
1640
- training_thread->num_models_to_prune += training_thread->pending_model_info.size();
1130
+ worker->num_db_transactions++;
1131
+ worker->num_models_to_prune += worker->pending_model_info.size();
1132
+ }
1133
+
1134
+ vacuum_database(ml_db, "ML", 0, 0);
1135
+
1136
+ worker->pending_model_info.clear();
1137
+}
1138
+
1139
+static enum ml_worker_result ml_worker_create_new_model(ml_worker_t *worker, ml_request_create_new_model_t req) {
1140
+ AcquiredDimension AcqDim(req.DLI);
1141
+
1142
+ if (!AcqDim.acquired()) {
1143
+ netdata_log_error("Failed to create new model: could not acquire dimension (machine-guid: %s, dimension: '%s.%s')",
1144
+ req.DLI.machineGuid(), req.DLI.chartId(), req.DLI.dimensionId());
1145
+ return ML_WORKER_RESULT_NULL_ACQUIRED_DIMENSION;
1146
}
1147
1643
- vacuum_database(db, "ML", 0, 0);
1148
+ ml_dimension_t *Dim = reinterpret_cast<ml_dimension_t *>(AcqDim.dimension());
1149
+ return ml_dimension_train_model(worker, Dim, req);
1150
+}
1151
+
1152
+static enum ml_worker_result ml_worker_add_existing_model(ml_worker_t *worker, ml_request_add_existing_model_t req) {
1153
+ UNUSED(worker);
1154
+ UNUSED(req);
1155
+
1156
+ AcquiredDimension AcqDim(req.DLI);
1157
+
1158
+ if (!AcqDim.acquired()) {
1159
+ netdata_log_error("Failed to add existing model: could not acquire dimension (machine-guid: %s, dimension: '%s.%s')",
1160
+ req.DLI.machineGuid(), req.DLI.chartId(), req.DLI.dimensionId());
1161
+ return ML_WORKER_RESULT_NULL_ACQUIRED_DIMENSION;
1162
+ }
1163
+
1164
+ ml_dimension_t *Dim = reinterpret_cast<ml_dimension_t *>(AcqDim.dimension());
1165
+ if (!Dim) {
1166
+ telemetry_ml_models_ignored();
1167
+ return ML_WORKER_RESULT_OK;
1168
+ }
1169
1645
- training_thread->pending_model_info.clear();
1170
+ Dim->kmeans = req.inlined_km;
1171
+ ml_dimension_update_models(worker, Dim);
1172
+ telemetry_ml_models_received();
1173
+ return ML_WORKER_RESULT_OK;
1174
}
1175
1648
-static void *ml_train_main(void *arg) {
1649
- ml_training_thread_t *training_thread = (ml_training_thread_t *) arg;
1176
+void *ml_train_main(void *arg) {
1177
+ ml_worker_t *worker = (ml_worker_t *) arg;
1178
1179
char worker_name[1024];
1652
- snprintfz(worker_name, 1024, "training_thread_%zu", training_thread->id);
1180
+ snprintfz(worker_name, 1024, "ml_worker_%zu", worker->id);
1181
worker_register("MLTRAIN");
1182
1183
worker_register_job_name(WORKER_TRAIN_QUEUE_POP, "pop queue");
@@ -1664,14 +1192,12 @@ static void *ml_train_main(void *arg) {
1192
while (!Cfg.training_stop) {
1193
worker_is_busy(WORKER_TRAIN_QUEUE_POP);
1194
1667
- ml_training_request_t training_req = ml_queue_pop(training_thread->training_queue);
1668
-
1669
- // we know this thread has been cancelled, when the queue starts
1670
- // returning "null" requests without blocking on queue's pop().
1671
- if (training_req.chart_id == NULL)
1195
+ ml_queue_item_t item = ml_queue_pop(worker->queue);
1196
+ if (item.type == ML_QUEUE_ITEM_STOP_REQUEST) {
1197
break;
1198
+ }
1199
1674
- size_t queue_size = ml_queue_size(training_thread->training_queue) + 1;
1200
+ size_t queue_size = ml_queue_size(worker->queue) + 1;
1201
1202
usec_t allotted_ut = (Cfg.train_every * USEC_PER_SEC) / queue_size;
1203
if (allotted_ut > USEC_PER_SEC)
@@ -1679,21 +1205,20 @@ static void *ml_train_main(void *arg) {
1205
1206
usec_t start_ut = now_monotonic_usec();
1207
1682
- enum ml_training_result training_res;
1683
- {
1684
- worker_is_busy(WORKER_TRAIN_ACQUIRE_DIMENSION);
1685
- ml_acquired_dimension_t acq_dim = ml_acquired_dimension_get(
1686
- training_req.machine_guid,
1687
- training_req.chart_id,
1688
- training_req.dimension_id);
1689
-
1690
- training_res = ml_acquired_dimension_train(training_thread, acq_dim, training_req);
1691
-
1692
- string_freez(training_req.chart_id);
1693
- string_freez(training_req.dimension_id);
1208
+ enum ml_worker_result worker_res;
1209
1695
- worker_is_busy(WORKER_TRAIN_RELEASE_DIMENSION);
1696
- ml_acquired_dimension_release(acq_dim);
1210
+ switch (item.type) {
1211
+ case ML_QUEUE_ITEM_TYPE_CREATE_NEW_MODEL: {
1212
+ worker_res = ml_worker_create_new_model(worker, item.create_new_model);
1213
+ break;
1214
+ }
1215
+ case ML_QUEUE_ITEM_TYPE_ADD_EXISTING_MODEL: {
1216
+ worker_res = ml_worker_add_existing_model(worker, item.add_existing_model);
1217
+ break;
1218
+ }
1219
+ default: {
1220
+ fatal("Unknown queue item type");
1221
+ }
1222
}
1223
1224
usec_t consumed_ut = now_monotonic_usec() - start_ut;
@@ -1705,40 +1230,40 @@ static void *ml_train_main(void *arg) {
1230
if (Cfg.enable_statistics_charts) {
1231
worker_is_busy(WORKER_TRAIN_UPDATE_HOST);
1232
1708
- netdata_mutex_lock(&training_thread->nd_mutex);
1233
+ netdata_mutex_lock(&worker->nd_mutex);
1234
1710
- training_thread->training_stats.queue_size += queue_size;
1711
- training_thread->training_stats.num_popped_items += 1;
1235
+ worker->queue_stats.queue_size += queue_size;
1236
+ worker->queue_stats.num_popped_items += 1;
1237
1713
- training_thread->training_stats.allotted_ut += allotted_ut;
1714
- training_thread->training_stats.consumed_ut += consumed_ut;
1715
- training_thread->training_stats.remaining_ut += remaining_ut;
1238
+ worker->queue_stats.allotted_ut += allotted_ut;
1239
+ worker->queue_stats.consumed_ut += consumed_ut;
1240
+ worker->queue_stats.remaining_ut += remaining_ut;
1241
1717
- switch (training_res) {
1718
- case TRAINING_RESULT_OK:
1719
- training_thread->training_stats.training_result_ok += 1;
1242
+ switch (worker_res) {
1243
+ case ML_WORKER_RESULT_OK:
1244
+ worker->queue_stats.item_result_ok += 1;
1245
break;
1721
- case TRAINING_RESULT_INVALID_QUERY_TIME_RANGE:
1722
- training_thread->training_stats.training_result_invalid_query_time_range += 1;
1246
+ case ML_WORKER_RESULT_INVALID_QUERY_TIME_RANGE:
1247
+ worker->queue_stats.item_result_invalid_query_time_range += 1;
1248
break;
1724
- case TRAINING_RESULT_NOT_ENOUGH_COLLECTED_VALUES:
1725
- training_thread->training_stats.training_result_not_enough_collected_values += 1;
1249
+ case ML_WORKER_RESULT_NOT_ENOUGH_COLLECTED_VALUES:
1250
+ worker->queue_stats.item_result_not_enough_collected_values += 1;
1251
break;
1727
- case TRAINING_RESULT_NULL_ACQUIRED_DIMENSION:
1728
- training_thread->training_stats.training_result_null_acquired_dimension += 1;
1252
+ case ML_WORKER_RESULT_NULL_ACQUIRED_DIMENSION:
1253
+ worker->queue_stats.item_result_null_acquired_dimension += 1;
1254
break;
1730
- case TRAINING_RESULT_CHART_UNDER_REPLICATION:
1731
- training_thread->training_stats.training_result_chart_under_replication += 1;
1255
+ case ML_WORKER_RESULT_CHART_UNDER_REPLICATION:
1256
+ worker->queue_stats.item_result_chart_under_replication += 1;
1257
break;
1258
}
1259
1735
- netdata_mutex_unlock(&training_thread->nd_mutex);
1260
+ netdata_mutex_unlock(&worker->nd_mutex);
1261
}
1262
1738
- if (training_thread->pending_model_info.size() >= Cfg.flush_models_batch_size) {
1263
+ if (worker->pending_model_info.size() >= Cfg.flush_models_batch_size) {
1264
worker_is_busy(WORKER_TRAIN_FLUSH_MODELS);
1265
netdata_mutex_lock(&db_mutex);
1741
- ml_flush_pending_models(training_thread);
1266
+ ml_flush_pending_models(worker);
1267
netdata_mutex_unlock(&db_mutex);
1268
continue;
1269
}
@@ -1749,140 +1274,3 @@ static void *ml_train_main(void *arg) {
1274
1275
return NULL;
1276
}
1752
-
1753
-void ml_init()
1754
-{
1755
- // Read config values
1756
- ml_config_load(&Cfg);
1757
-
1758
- if (!Cfg.enable_anomaly_detection)
1759
- return;
1760
-
1761
- // Generate random numbers to efficiently sample the features we need
1762
- // for KMeans clustering.
1763
- std::random_device RD;
1764
- std::mt19937 Gen(RD());
1765
-
1766
- Cfg.random_nums.reserve(Cfg.max_train_samples);
1767
- for (size_t Idx = 0; Idx != Cfg.max_train_samples; Idx++)
1768
- Cfg.random_nums.push_back(Gen());
1769
-
1770
- // init training thread-specific data
1771
- Cfg.training_threads.resize(Cfg.num_training_threads);
1772
- for (size_t idx = 0; idx != Cfg.num_training_threads; idx++) {
1773
- ml_training_thread_t *training_thread = &Cfg.training_threads[idx];
1774
-
1775
- size_t max_elements_needed_for_training = (size_t) Cfg.max_train_samples * (size_t) (Cfg.lag_n + 1);
1776
- training_thread->training_cns = new calculated_number_t[max_elements_needed_for_training]();
1777
- training_thread->scratch_training_cns = new calculated_number_t[max_elements_needed_for_training]();
1778
-
1779
- training_thread->id = idx;
1780
- training_thread->training_queue = ml_queue_init();
1781
- training_thread->pending_model_info.reserve(Cfg.flush_models_batch_size);
1782
- netdata_mutex_init(&training_thread->nd_mutex);
1783
- }
1784
-
1785
- // open sqlite db
1786
- char path[FILENAME_MAX];
1787
- snprintfz(path, FILENAME_MAX - 1, "%s/%s", netdata_configured_cache_dir, "ml.db");
1788
- int rc = sqlite3_open(path, &db);
1789
- if (rc != SQLITE_OK) {
1790
- error_report("Failed to initialize database at %s, due to \"%s\"", path, sqlite3_errstr(rc));
1791
- sqlite3_close(db);
1792
- db = NULL;
1793
- }
1794
-
1795
- // create table
1796
- if (db) {
1797
- int target_version = perform_ml_database_migration(db, ML_METADATA_VERSION);
1798
- if (configure_sqlite_database(db, target_version, "ml_config")) {
1799
- error_report("Failed to setup ML database");
1800
- sqlite3_close(db);
1801
- db = NULL;
1802
- }
1803
- else {
1804
- char *err = NULL;
1805
- int rc = sqlite3_exec(db, db_models_create_table, NULL, NULL, &err);
1806
- if (rc != SQLITE_OK) {
1807
- error_report("Failed to create models table (%s, %s)", sqlite3_errstr(rc), err ? err : "");
1808
- sqlite3_close(db);
1809
- sqlite3_free(err);
1810
- db = NULL;
1811
- }
1812
- }
1813
- }
1814
-}
1815
-
1816
-uint64_t sqlite_get_ml_space(void)
1817
-{
1818
- return sqlite_get_db_space(db);
1819
-}
1820
-
1821
-void ml_fini() {
1822
- if (!Cfg.enable_anomaly_detection || !db)
1823
- return;
1824
-
1825
- sql_close_database(db, "ML");
1826
- db = NULL;
1827
-}
1828
-
1829
-void ml_start_threads() {
1830
- if (!Cfg.enable_anomaly_detection)
1831
- return;
1832
-
1833
- // start detection & training threads
1834
- Cfg.detection_stop = false;
1835
- Cfg.training_stop = false;
1836
-
1837
- char tag[NETDATA_THREAD_TAG_MAX + 1];
1838
-
1839
- snprintfz(tag, NETDATA_THREAD_TAG_MAX, "%s", "PREDICT");
1840
- Cfg.detection_thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_JOINABLE,
1841
- ml_detect_main, NULL);
1842
-
1843
- for (size_t idx = 0; idx != Cfg.num_training_threads; idx++) {
1844
- ml_training_thread_t *training_thread = &Cfg.training_threads[idx];
1845
- snprintfz(tag, NETDATA_THREAD_TAG_MAX, "TRAIN[%zu]", training_thread->id);
1846
- training_thread->nd_thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_JOINABLE,
1847
- ml_train_main, training_thread);
1848
- }
1849
-}
1850
-
1851
-void ml_stop_threads()
1852
-{
1853
- if (!Cfg.enable_anomaly_detection)
1854
- return;
1855
-
1856
- Cfg.detection_stop = true;
1857
- Cfg.training_stop = true;
1858
-
1859
- if (!Cfg.detection_thread)
1860
- return;
1861
-
1862
- nd_thread_join(Cfg.detection_thread);
1863
- Cfg.detection_thread = 0;
1864
-
1865
- // signal the training queue of each thread
1866
- for (size_t idx = 0; idx != Cfg.num_training_threads; idx++) {
1867
- ml_training_thread_t *training_thread = &Cfg.training_threads[idx];
1868
-
1869
- ml_queue_signal(training_thread->training_queue);
1870
- }
1871
-
1872
- // join training threads
1873
- for (size_t idx = 0; idx != Cfg.num_training_threads; idx++) {
1874
- ml_training_thread_t *training_thread = &Cfg.training_threads[idx];
1875
-
1876
- nd_thread_join(training_thread->nd_thread);
1877
- }
1878
-
1879
- // clear training thread data
1880
- for (size_t idx = 0; idx != Cfg.num_training_threads; idx++) {
1881
- ml_training_thread_t *training_thread = &Cfg.training_threads[idx];
1882
-
1883
- delete[] training_thread->training_cns;
1884
- delete[] training_thread->scratch_training_cns;
1885
- ml_queue_destroy(training_thread->training_queue);
1886
- netdata_mutex_destroy(&training_thread->nd_mutex);
1887
- }
1888
-}
src/ml/ml_calculated_number.h
new
+15
@@ -0,0 +1,15 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_ML_CALCULATED_NUMBER_H
4
+#define NETDATA_ML_CALCULATED_NUMBER_H
5
+
6
+#include "dlib/dlib/matrix.h"
7
+
8
+// CentOS 7 shenanigans
9
+#include <cmath>
10
+using std::isfinite;
11
+
12
+typedef double calculated_number_t;
13
+typedef dlib::matrix<calculated_number_t, 6, 1> DSample;
14
+
15
+#endif /* NETDATA_ML_CALCULATED_NUMBER_H */
src/ml/ml_chart.h
new
+17
@@ -0,0 +1,17 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_ML_CHART_H
4
+#define NETDATA_ML_CHART_H
5
+
6
+#include "ml_host.h"
7
+
8
+struct ml_dimension_t;
9
+
10
+struct ml_chart_t {
11
+ RRDSET *rs;
12
+ ml_machine_learning_stats_t mls;
13
+};
14
+
15
+void ml_chart_update_dimension(ml_chart_t *chart, ml_dimension_t *dim, bool is_anomalous);
16
+
17
+#endif /* NETDATA_ML_CHART_H */
src/ml/ml_config.cc
renamed
+5
-5
@@ -1,6 +1,6 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#include "ml-private.h"
3
+#include "ml_config.h"
4
5
/*
6
* Global configuration instance to be shared between training and
@@ -45,7 +45,7 @@ void ml_config_load(ml_config_t *cfg) {
45
std::string anomaly_detection_grouping_method = config_get(config_section_ml, "anomaly detection grouping method", "average");
46
time_t anomaly_detection_query_duration = config_get_duration_seconds(config_section_ml, "anomaly detection grouping duration", 5 * 60);
47
48
- size_t num_training_threads = config_get_number(config_section_ml, "num training threads", 4);
48
+ size_t num_worker_threads = config_get_number(config_section_ml, "num training threads", os_get_system_cpus() / 4);
49
size_t flush_models_batch_size = config_get_number(config_section_ml, "flush models batch size", 128);
50
51
size_t suppression_window =
@@ -54,7 +54,7 @@ void ml_config_load(ml_config_t *cfg) {
54
size_t suppression_threshold =
55
config_get_number(config_section_ml, "dimension anomaly rate suppression threshold", suppression_window / 2);
56
57
- bool enable_statistics_charts = config_get_boolean(config_section_ml, "enable statistics charts", false);
57
+ bool enable_statistics_charts = config_get_boolean(config_section_ml, "enable statistics charts", true);
58
59
/*
60
* Clamp
@@ -79,7 +79,7 @@ void ml_config_load(ml_config_t *cfg) {
79
host_anomaly_rate_threshold = clamp(host_anomaly_rate_threshold, 0.1, 10.0);
80
anomaly_detection_query_duration = clamp<time_t>(anomaly_detection_query_duration, 60, 15 * 60);
81
82
- num_training_threads = clamp<size_t>(num_training_threads, 1, 128);
82
+ num_worker_threads = clamp<size_t>(num_worker_threads, 4, os_get_system_cpus());
83
flush_models_batch_size = clamp<size_t>(flush_models_batch_size, 8, 512);
84
85
suppression_window = clamp<size_t>(suppression_window, 1, max_train_samples);
@@ -132,7 +132,7 @@ void ml_config_load(ml_config_t *cfg) {
132
133
cfg->stream_anomaly_detection_charts = config_get_boolean(config_section_ml, "stream anomaly detection charts", true);
134
135
- cfg->num_training_threads = num_training_threads;
135
+ cfg->num_worker_threads = num_worker_threads;
136
cfg->flush_models_batch_size = flush_models_batch_size;
137
138
cfg->suppression_window = suppression_window;
src/ml/ml_config.h
new
+62
@@ -0,0 +1,62 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef ML_CONFIG_H
4
+#define ML_CONFIG_H
5
+
6
+#include "ml_worker.h"
7
+
8
+typedef struct {
9
+ int enable_anomaly_detection;
10
+
11
+ unsigned max_train_samples;
12
+ unsigned min_train_samples;
13
+ unsigned train_every;
14
+
15
+ unsigned num_models_to_use;
16
+ unsigned delete_models_older_than;
17
+
18
+ unsigned db_engine_anomaly_rate_every;
19
+
20
+ unsigned diff_n;
21
+ unsigned smooth_n;
22
+ unsigned lag_n;
23
+
24
+ double random_sampling_ratio;
25
+ unsigned max_kmeans_iters;
26
+
27
+ double dimension_anomaly_score_threshold;
28
+
29
+ double host_anomaly_rate_threshold;
30
+ RRDR_TIME_GROUPING anomaly_detection_grouping_method;
31
+ time_t anomaly_detection_query_duration;
32
+
33
+ bool stream_anomaly_detection_charts;
34
+
35
+ std::string hosts_to_skip;
36
+ SIMPLE_PATTERN *sp_host_to_skip;
37
+
38
+ std::string charts_to_skip;
39
+ SIMPLE_PATTERN *sp_charts_to_skip;
40
+
41
+ std::vector<uint32_t> random_nums;
42
+
43
+ ND_THREAD *detection_thread;
44
+ std::atomic<bool> detection_stop;
45
+
46
+ size_t num_worker_threads;
47
+ size_t flush_models_batch_size;
48
+
49
+ std::vector<ml_worker_t> workers;
50
+ std::atomic<bool> training_stop;
51
+
52
+ size_t suppression_window;
53
+ size_t suppression_threshold;
54
+
55
+ bool enable_statistics_charts;
56
+} ml_config_t;
57
+
58
+void ml_config_load(ml_config_t *cfg);
59
+
60
+extern ml_config_t Cfg;
61
+
62
+#endif /* ML_CONFIG_H */
src/ml/ml_dimension.h
new
+168
@@ -0,0 +1,168 @@
1
+#ifndef ML_LOOKUP_H
2
+#define ML_LOOKUP_H
3
+
4
+#include "ml_string_wrapper.h"
5
+#include "ml_enums.h"
6
+#include "ml_kmeans.h"
7
+#include "ml_chart.h"
8
+
9
+#include <array>
10
+
11
+struct ml_dimension_t {
12
+ RRDDIM *rd;
13
+
14
+ enum ml_metric_type mt;
15
+ enum ml_training_status ts;
16
+ enum ml_machine_learning_status mls;
17
+
18
+ time_t last_training_time;
19
+
20
+ std::vector<calculated_number_t> cns;
21
+
22
+ std::vector<ml_kmeans_inlined_t> km_contexts;
23
+ SPINLOCK slock;
24
+ ml_kmeans_t kmeans;
25
+ std::vector<DSample> feature;
26
+
27
+ uint32_t suppression_window_counter;
28
+ uint32_t suppression_anomaly_counter;
29
+};
30
+
31
+bool
32
+ml_dimension_predict(ml_dimension_t *dim, time_t curr_time, calculated_number_t value, bool exists);
33
+
34
+bool ml_dimension_deserialize_kmeans(const char *json_str);
35
+
36
+class DimensionLookupInfo {
37
+public:
38
+ DimensionLookupInfo()
39
+ {
40
+ memset(MachineGuid.data(), 0, MachineGuid.size());
41
+ }
42
+
43
+ DimensionLookupInfo(const char *MachineGuid, STRING *ChartId, STRING *DimensionId)
44
+ : ChartId(ChartId), DimensionId(DimensionId)
45
+ {
46
+ memcpy(this->MachineGuid.data(), MachineGuid, this->MachineGuid.size());
47
+ }
48
+
49
+ DimensionLookupInfo(const char *MachineGuid, const char *ChartId, const char *DimensionId)
50
+ : ChartId(ChartId), DimensionId(DimensionId)
51
+ {
52
+ memcpy(this->MachineGuid.data(), MachineGuid, this->MachineGuid.size());
53
+ }
54
+
55
+ const char *machineGuid() const
56
+ {
57
+ return MachineGuid.data();
58
+ }
59
+
60
+ const char *chartId() const
61
+ {
62
+ return ChartId;
63
+ }
64
+
65
+ const char *dimensionId() const
66
+ {
67
+ return DimensionId;
68
+ }
69
+
70
+private:
71
+ std::array<char, GUID_LEN + 1> MachineGuid;
72
+ StringWrapper ChartId;
73
+ StringWrapper DimensionId;
74
+};
75
+
76
+class AcquiredDimension {
77
+public:
78
+ AcquiredDimension(const DimensionLookupInfo &DLI) : AcqRH(nullptr), AcqRS(nullptr), AcqRD(nullptr), Dim(nullptr)
79
+ {
80
+ rrd_rdlock();
81
+
82
+ AcqRH = rrdhost_find_and_acquire(DLI.machineGuid());
83
+ if (AcqRH) {
84
+ RRDHOST *RH = rrdhost_acquired_to_rrdhost(AcqRH);
85
+ if (RH && !rrdhost_flag_check(RH, RRDHOST_FLAG_ORPHAN | RRDHOST_FLAG_ARCHIVED)) {
86
+ AcqRS = rrdset_find_and_acquire(RH, DLI.chartId());
87
+ if (AcqRS) {
88
+ RRDSET *RS = rrdset_acquired_to_rrdset(AcqRS);
89
+ if (RS && !rrdset_flag_check(RS, RRDSET_FLAG_OBSOLETE)) {
90
+ AcqRD = rrddim_find_and_acquire(RS, DLI.dimensionId());
91
+ if (AcqRD) {
92
+ RRDDIM *RD = rrddim_acquired_to_rrddim(AcqRD);
93
+ if (RD) {
94
+ Dim = reinterpret_cast<ml_dimension_t *>(RD->ml_dimension);
95
+ acquire_failure_reason = "ok";
96
+ }
97
+ else
98
+ acquire_failure_reason = "no dimension";
99
+ }
100
+ else
101
+ acquire_failure_reason = "can't find dimension";
102
+ }
103
+ else
104
+ acquire_failure_reason = "chart is obsolete";
105
+ }
106
+ else
107
+ acquire_failure_reason = "can't find chart";
108
+ }
109
+ else
110
+ acquire_failure_reason = "host is orphan or archived";
111
+ }
112
+ else
113
+ acquire_failure_reason = "can't find host";
114
+
115
+ rrd_rdunlock();
116
+ }
117
+
118
+ AcquiredDimension(const AcquiredDimension &) = delete;
119
+ AcquiredDimension operator=(const AcquiredDimension &) = delete;
120
+
121
+ AcquiredDimension(AcquiredDimension &&) = default;
122
+ AcquiredDimension &operator=(AcquiredDimension &&) = default;
123
+
124
+ bool acquired() const {
125
+ return AcqRD != nullptr;
126
+ }
127
+
128
+ const char *acquire_failure() const {
129
+ return acquire_failure_reason;
130
+ }
131
+
132
+ ml_host_t *host() const {
133
+ assert(acquired());
134
+ RRDHOST *RH = rrdhost_acquired_to_rrdhost(AcqRH);
135
+ return reinterpret_cast<ml_host_t *>(RH->ml_host);
136
+ }
137
+
138
+ ml_queue_t *queue() const {
139
+ assert(acquired());
140
+ return host()->queue;
141
+ }
142
+
143
+ ml_dimension_t *dimension() const {
144
+ assert(acquired());
145
+ return Dim;
146
+ }
147
+
148
+ ~AcquiredDimension()
149
+ {
150
+ if (AcqRD)
151
+ rrddim_acquired_release(AcqRD);
152
+
153
+ if (AcqRS)
154
+ rrdset_acquired_release(AcqRS);
155
+
156
+ if (AcqRD)
157
+ rrdhost_acquired_release(AcqRH);
158
+ }
159
+
160
+private:
161
+ const char *acquire_failure_reason;
162
+ RRDHOST_ACQUIRED *AcqRH;
163
+ RRDSET_ACQUIRED *AcqRS;
164
+ RRDDIM_ACQUIRED *AcqRD;
165
+ ml_dimension_t *Dim;
166
+};
167
+
168
+#endif /* ML_LOOKUP_H */
src/ml/ml_enums.cc
new
+82
@@ -0,0 +1,82 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "ml_enums.h"
4
+
5
+const char *
6
+ml_machine_learning_status_to_string(enum ml_machine_learning_status mls)
7
+{
8
+ switch (mls) {
9
+ case MACHINE_LEARNING_STATUS_ENABLED:
10
+ return "enabled";
11
+ case MACHINE_LEARNING_STATUS_DISABLED_DUE_TO_EXCLUDED_CHART:
12
+ return "disabled-sp";
13
+ default:
14
+ return "unknown";
15
+ }
16
+}
17
+
18
+const char *
19
+ml_metric_type_to_string(enum ml_metric_type mt)
20
+{
21
+ switch (mt) {
22
+ case METRIC_TYPE_CONSTANT:
23
+ return "constant";
24
+ case METRIC_TYPE_VARIABLE:
25
+ return "variable";
26
+ default:
27
+ return "unknown";
28
+ }
29
+}
30
+
31
+const char *
32
+ml_training_status_to_string(enum ml_training_status ts)
33
+{
34
+ switch (ts) {
35
+ case TRAINING_STATUS_PENDING_WITH_MODEL:
36
+ return "pending-with-model";
37
+ case TRAINING_STATUS_PENDING_WITHOUT_MODEL:
38
+ return "pending-without-model";
39
+ case TRAINING_STATUS_TRAINED:
40
+ return "trained";
41
+ case TRAINING_STATUS_UNTRAINED:
42
+ return "untrained";
43
+ case TRAINING_STATUS_SILENCED:
44
+ return "silenced";
45
+ default:
46
+ return "unknown";
47
+ }
48
+}
49
+
50
+const char *
51
+ml_worker_result_to_string(enum ml_worker_result tr)
52
+{
53
+ switch (tr) {
54
+ case ML_WORKER_RESULT_OK:
55
+ return "ok";
56
+ case ML_WORKER_RESULT_INVALID_QUERY_TIME_RANGE:
57
+ return "invalid-query";
58
+ case ML_WORKER_RESULT_NOT_ENOUGH_COLLECTED_VALUES:
59
+ return "missing-values";
60
+ case ML_WORKER_RESULT_NULL_ACQUIRED_DIMENSION:
61
+ return "null-acquired-dim";
62
+ case ML_WORKER_RESULT_CHART_UNDER_REPLICATION:
63
+ return "chart-under-replication";
64
+ default:
65
+ return "unknown";
66
+ }
67
+}
68
+
69
+const char *
70
+ml_queue_item_type_to_string(enum ml_queue_item_type qit)
71
+{
72
+ switch (qit) {
73
+ case ML_QUEUE_ITEM_TYPE_CREATE_NEW_MODEL:
74
+ return "create-new-model";
75
+ case ML_QUEUE_ITEM_TYPE_ADD_EXISTING_MODEL:
76
+ return "add-existing-model";
77
+ case ML_QUEUE_ITEM_STOP_REQUEST:
78
+ return "stop-request";
79
+ default:
80
+ return "unknown";
81
+ }
82
+}
src/ml/ml_enums.h
new
+72
@@ -0,0 +1,72 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_ML_ENUMS_H
4
+#define NETDATA_ML_ENUMS_H
5
+
6
+enum ml_metric_type {
7
+ // The dimension has constant values, no need to train
8
+ METRIC_TYPE_CONSTANT,
9
+
10
+ // The dimension's values fluctuate, we need to generate a model
11
+ METRIC_TYPE_VARIABLE,
12
+};
13
+
14
+const char *ml_metric_type_to_string(enum ml_metric_type mt);
15
+
16
+enum ml_machine_learning_status {
17
+ // Enable training/prediction
18
+ MACHINE_LEARNING_STATUS_ENABLED,
19
+
20
+ // Disable because configuration pattern matches the chart's id
21
+ MACHINE_LEARNING_STATUS_DISABLED_DUE_TO_EXCLUDED_CHART,
22
+};
23
+
24
+const char *ml_machine_learning_status_to_string(enum ml_machine_learning_status mls);
25
+
26
+enum ml_training_status {
27
+ // We don't have a model for this dimension
28
+ TRAINING_STATUS_UNTRAINED,
29
+
30
+ // Request for training sent, but we don't have any models yet
31
+ TRAINING_STATUS_PENDING_WITHOUT_MODEL,
32
+
33
+ // Request to update existing models sent
34
+ TRAINING_STATUS_PENDING_WITH_MODEL,
35
+
36
+ // Have a valid, up-to-date model
37
+ TRAINING_STATUS_TRAINED,
38
+
39
+ // Have a valid, up-to-date model that is silenced because its too noisy
40
+ TRAINING_STATUS_SILENCED,
41
+};
42
+
43
+const char *ml_training_status_to_string(enum ml_training_status ts);
44
+
45
+enum ml_worker_result {
46
+ // We managed to create a KMeans model
47
+ ML_WORKER_RESULT_OK,
48
+
49
+ // Could not query DB with a correct time range
50
+ ML_WORKER_RESULT_INVALID_QUERY_TIME_RANGE,
51
+
52
+ // Did not gather enough data from DB to run KMeans
53
+ ML_WORKER_RESULT_NOT_ENOUGH_COLLECTED_VALUES,
54
+
55
+ // Acquired a null dimension
56
+ ML_WORKER_RESULT_NULL_ACQUIRED_DIMENSION,
57
+
58
+ // Chart is under replication
59
+ ML_WORKER_RESULT_CHART_UNDER_REPLICATION,
60
+};
61
+
62
+const char *ml_worker_result_to_string(enum ml_worker_result tr);
63
+
64
+enum ml_queue_item_type {
65
+ ML_QUEUE_ITEM_TYPE_CREATE_NEW_MODEL,
66
+ ML_QUEUE_ITEM_TYPE_ADD_EXISTING_MODEL,
67
+ ML_QUEUE_ITEM_STOP_REQUEST,
68
+};
69
+
70
+const char *ml_queue_item_type_to_string(enum ml_queue_item_type qit);
71
+
72
+#endif /* NETDATA_ML_ENUMS_H */
src/ml/ml_features.cc
new
+78
@@ -0,0 +1,78 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "ml_config.h"
4
+#include "ml_features.h"
5
+
6
+static void ml_features_diff(ml_features_t *features)
7
+{
8
+ if (features->diff_n == 0)
9
+ return;
10
+
11
+ for (size_t idx = 0; idx != (features->src_n - features->diff_n); idx++) {
12
+ size_t high = (features->src_n - 1) - idx;
13
+ size_t low = high - features->diff_n;
14
+
15
+ features->dst[low] = features->src[high] - features->src[low];
16
+ }
17
+
18
+ size_t n = features->src_n - features->diff_n;
19
+ memcpy(features->src, features->dst, n * sizeof(calculated_number_t));
20
+
21
+ for (size_t idx = features->src_n - features->diff_n; idx != features->src_n; idx++)
22
+ features->src[idx] = 0.0;
23
+}
24
+
25
+static void ml_features_smooth(ml_features_t *features)
26
+{
27
+ calculated_number_t sum = 0.0;
28
+
29
+ size_t idx = 0;
30
+ for (; idx != features->smooth_n - 1; idx++)
31
+ sum += features->src[idx];
32
+
33
+ for (; idx != (features->src_n - features->diff_n); idx++) {
34
+ sum += features->src[idx];
35
+ calculated_number_t prev_cn = features->src[idx - (features->smooth_n - 1)];
36
+ features->src[idx - (features->smooth_n - 1)] = sum / features->smooth_n;
37
+ sum -= prev_cn;
38
+ }
39
+
40
+ for (idx = 0; idx != features->smooth_n; idx++)
41
+ features->src[(features->src_n - 1) - idx] = 0.0;
42
+}
43
+
44
+static void ml_features_lag(ml_features_t *features)
45
+{
46
+ size_t n = features->src_n - features->diff_n - features->smooth_n + 1 - features->lag_n;
47
+ features->preprocessed_features.resize(n);
48
+
49
+ unsigned target_num_samples = Cfg.max_train_samples * Cfg.random_sampling_ratio;
50
+ double sampling_ratio = std::min(static_cast<double>(target_num_samples) / n, 1.0);
51
+
52
+ uint32_t max_mt = std::numeric_limits<uint32_t>::max();
53
+ uint32_t cutoff = static_cast<double>(max_mt) * sampling_ratio;
54
+
55
+ size_t sample_idx = 0;
56
+
57
+ for (size_t idx = 0; idx != n; idx++) {
58
+ DSample &DS = features->preprocessed_features[sample_idx++];
59
+ DS.set_size(features->lag_n);
60
+
61
+ if (Cfg.random_nums[idx] > cutoff) {
62
+ sample_idx--;
63
+ continue;
64
+ }
65
+
66
+ for (size_t feature_idx = 0; feature_idx != features->lag_n + 1; feature_idx++)
67
+ DS(feature_idx) = features->src[idx + feature_idx];
68
+ }
69
+
70
+ features->preprocessed_features.resize(sample_idx);
71
+}
72
+
73
+void ml_features_preprocess(ml_features_t *features)
74
+{
75
+ ml_features_diff(features);
76
+ ml_features_smooth(features);
77
+ ml_features_lag(features);
78
+}
src/ml/ml_features.h
new
+26
@@ -0,0 +1,26 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef ML_FEATURES_H
4
+#define ML_FEATURES_H
5
+
6
+#include "ml_calculated_number.h"
7
+
8
+#include <vector>
9
+
10
+typedef struct {
11
+ size_t diff_n;
12
+ size_t smooth_n;
13
+ size_t lag_n;
14
+
15
+ calculated_number_t *dst;
16
+ size_t dst_n;
17
+
18
+ calculated_number_t *src;
19
+ size_t src_n;
20
+
21
+ std::vector<DSample> &preprocessed_features;
22
+} ml_features_t;
23
+
24
+void ml_features_preprocess(ml_features_t *features);
25
+
26
+#endif /* ML_FEATURES_H */
src/ml/ml_host.h
new
+89
@@ -0,0 +1,89 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_ML_HOST_H
4
+#define NETDATA_ML_HOST_H
5
+
6
+#include "ml_calculated_number.h"
7
+
8
+#include "database/rrd.h"
9
+
10
+#include <atomic>
11
+#include <unordered_map>
12
+
13
+struct ml_queue_t;
14
+
15
+typedef struct machine_learning_stats_t {
16
+ size_t num_machine_learning_status_enabled;
17
+ size_t num_machine_learning_status_disabled_sp;
18
+
19
+ size_t num_metric_type_constant;
20
+ size_t num_metric_type_variable;
21
+
22
+ size_t num_training_status_untrained;
23
+ size_t num_training_status_pending_without_model;
24
+ size_t num_training_status_trained;
25
+ size_t num_training_status_pending_with_model;
26
+ size_t num_training_status_silenced;
27
+
28
+ size_t num_anomalous_dimensions;
29
+ size_t num_normal_dimensions;
30
+} ml_machine_learning_stats_t;
31
+
32
+typedef struct {
33
+ RRDDIM *rd;
34
+ size_t normal_dimensions;
35
+ size_t anomalous_dimensions;
36
+} ml_type_anomaly_rate_t;
37
+
38
+typedef struct {
39
+ RRDHOST *rh;
40
+
41
+ std::atomic<bool> ml_running;
42
+
43
+ ml_machine_learning_stats_t mls;
44
+
45
+ calculated_number_t host_anomaly_rate;
46
+
47
+ netdata_mutex_t mutex;
48
+
49
+ ml_queue_t *queue;
50
+
51
+ /*
52
+ * bookkeeping for anomaly detection charts
53
+ */
54
+
55
+ RRDSET *ml_running_rs;
56
+ RRDDIM *ml_running_rd;
57
+
58
+ RRDSET *machine_learning_status_rs;
59
+ RRDDIM *machine_learning_status_enabled_rd;
60
+ RRDDIM *machine_learning_status_disabled_sp_rd;
61
+
62
+ RRDSET *metric_type_rs;
63
+ RRDDIM *metric_type_constant_rd;
64
+ RRDDIM *metric_type_variable_rd;
65
+
66
+ RRDSET *training_status_rs;
67
+ RRDDIM *training_status_untrained_rd;
68
+ RRDDIM *training_status_pending_without_model_rd;
69
+ RRDDIM *training_status_trained_rd;
70
+ RRDDIM *training_status_pending_with_model_rd;
71
+ RRDDIM *training_status_silenced_rd;
72
+
73
+ RRDSET *dimensions_rs;
74
+ RRDDIM *dimensions_anomalous_rd;
75
+ RRDDIM *dimensions_normal_rd;
76
+
77
+ RRDSET *anomaly_rate_rs;
78
+ RRDDIM *anomaly_rate_rd;
79
+
80
+ RRDSET *detector_events_rs;
81
+ RRDDIM *detector_events_above_threshold_rd;
82
+ RRDDIM *detector_events_new_anomaly_event_rd;
83
+
84
+ RRDSET *type_anomaly_rate_rs;
85
+ SPINLOCK type_anomaly_rate_spinlock;
86
+ std::unordered_map<STRING *, ml_type_anomaly_rate_t> type_anomaly_rate;
87
+} ml_host_t;
88
+
89
+#endif /* NETDATA_ML_HOST_H */
src/ml/ml_kmeans.cc
new
+243
@@ -0,0 +1,243 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "ml_kmeans.h"
4
+#include "libnetdata/libnetdata.h"
5
+#include "dlib/dlib/clustering.h"
6
+
7
+void
8
+ml_kmeans_init(ml_kmeans_t *kmeans)
9
+{
10
+ kmeans->cluster_centers.reserve(2);
11
+ kmeans->cluster_centers.clear();
12
+ kmeans->min_dist = std::numeric_limits<calculated_number_t>::max();
13
+ kmeans->max_dist = std::numeric_limits<calculated_number_t>::min();
14
+}
15
+
16
+void
17
+ml_kmeans_train(ml_kmeans_t *kmeans, const ml_features_t *features, unsigned max_iters, time_t after, time_t before)
18
+{
19
+ kmeans->after = (uint32_t) after;
20
+ kmeans->before = (uint32_t) before;
21
+
22
+ kmeans->min_dist = std::numeric_limits<calculated_number_t>::max();
23
+ kmeans->max_dist = std::numeric_limits<calculated_number_t>::min();
24
+
25
+ kmeans->cluster_centers.clear();
26
+
27
+ dlib::pick_initial_centers(2, kmeans->cluster_centers, features->preprocessed_features);
28
+ dlib::find_clusters_using_kmeans(features->preprocessed_features, kmeans->cluster_centers, max_iters);
29
+
30
+ for (const auto &preprocessed_feature : features->preprocessed_features) {
31
+ calculated_number_t mean_dist = 0.0;
32
+
33
+ for (const auto &cluster_center : kmeans->cluster_centers) {
34
+ mean_dist += dlib::length(cluster_center - preprocessed_feature);
35
+ }
36
+
37
+ mean_dist /= kmeans->cluster_centers.size();
38
+
39
+ if (mean_dist < kmeans->min_dist)
40
+ kmeans->min_dist = mean_dist;
41
+
42
+ if (mean_dist > kmeans->max_dist)
43
+ kmeans->max_dist = mean_dist;
44
+ }
45
+}
46
+
47
+calculated_number_t
48
+ml_kmeans_anomaly_score(const ml_kmeans_inlined_t *inlined_km, const DSample &DS)
49
+{
50
+ calculated_number_t mean_dist = 0.0;
51
+ for (const auto &CC: inlined_km->cluster_centers)
52
+ mean_dist += dlib::length(CC - DS);
53
+
54
+ mean_dist /= inlined_km->cluster_centers.size();
55
+
56
+ if (inlined_km->max_dist == inlined_km->min_dist)
57
+ return 0.0;
58
+
59
+ calculated_number_t anomaly_score = 100.0 * std::abs((mean_dist - inlined_km->min_dist) / (inlined_km->max_dist - inlined_km->min_dist));
60
+ return (anomaly_score > 100.0) ? 100.0 : anomaly_score;
61
+}
62
+
63
+static void ml_buffer_json_member_add_double(BUFFER *wb, const char *key, calculated_number_t cn) {
64
+ if (!isnan(cn) && !isinf(cn)) {
65
+ buffer_json_member_add_double(wb, key, cn);
66
+ return;
67
+ }
68
+
69
+ const char *classification = nullptr;
70
+ if (isnan(cn)) {
71
+ classification = "nan";
72
+ } else if (isinf(cn)) {
73
+ if (cn > 0) {
74
+ classification = "+inf";
75
+ } else {
76
+ classification = "-inf";
77
+ }
78
+ }
79
+
80
+ buffer_json_member_add_string(wb, key, classification);
81
+}
82
+
83
+static void ml_buffer_json_add_array_item_double(BUFFER *wb, calculated_number_t cn) {
84
+ if (!isnan(cn) && !isinf(cn)) {
85
+ buffer_json_add_array_item_double(wb, cn);
86
+ return;
87
+ }
88
+
89
+ const char *classification = nullptr;
90
+ if (isnan(cn)) {
91
+ classification = "nan";
92
+ } else if (isinf(cn)) {
93
+ if (cn > 0) {
94
+ classification = "+inf";
95
+ } else if (cn < 0) {
96
+ classification = "-inf";
97
+ }
98
+ }
99
+
100
+ buffer_json_add_array_item_string(wb, classification);
101
+}
102
+
103
+bool ml_json_parse_double(struct json_object *jo, calculated_number_t *cn) {
104
+ switch(json_object_get_type(jo)) {
105
+ case json_type_string: {
106
+ const char *s = json_object_get_string(jo);
107
+ if (strcmp(s, "nan") == 0) {
108
+ *cn = NAN;
109
+ return true;
110
+ }
111
+ else if (strcmp(s, "+inf") == 0) {
112
+ *cn = INFINITY;
113
+ return true;
114
+ }
115
+ else if (strcmp(s, "-inf") == 0) {
116
+ *cn = -INFINITY;
117
+ return true;
118
+ }
119
+
120
+ return false;
121
+ }
122
+ case json_type_double: {
123
+ *cn = json_object_get_double(jo);
124
+ return true;
125
+ }
126
+ case json_type_int: {
127
+ *cn = json_object_get_int64(jo);
128
+ return true;
129
+ }
130
+ default:
131
+ return false;
132
+ }
133
+}
134
+
135
+void
136
+ml_kmeans_serialize(const ml_kmeans_inlined_t *inlined_km, BUFFER *wb)
137
+{
138
+ buffer_json_member_add_uint64(wb, "after", inlined_km->after);
139
+ buffer_json_member_add_uint64(wb, "before", inlined_km->before);
140
+
141
+ ml_buffer_json_member_add_double(wb, "min_dist", inlined_km->min_dist);
142
+ ml_buffer_json_member_add_double(wb, "max_dist", inlined_km->max_dist);
143
+
144
+ buffer_json_member_add_array(wb, "cluster_centers");
145
+ for (const auto &cc: inlined_km->cluster_centers) {
146
+ buffer_json_add_array_item_array(wb);
147
+
148
+ for (const auto &d: cc) {
149
+ ml_buffer_json_add_array_item_double(wb, d);
150
+ }
151
+
152
+ buffer_json_array_close(wb);
153
+ }
154
+ buffer_json_array_close(wb);
155
+}
156
+
157
+bool ml_kmeans_deserialize(ml_kmeans_inlined_t *inlined_km, struct json_object *root)
158
+{
159
+ struct json_object *value;
160
+
161
+ if (!json_object_object_get_ex(root, "after", &value)) {
162
+ netdata_log_error("Failed to deserialize kmeans: missing key 'after'");
163
+ return false;
164
+ }
165
+ if (!json_object_is_type(value, json_type_int)) {
166
+ netdata_log_error("Failed to deserialize kmeans: failed to parse int for 'after'");
167
+ return false;
168
+ }
169
+ inlined_km->after = json_object_get_int(value);
170
+
171
+ if (!json_object_object_get_ex(root, "before", &value)) {
172
+ netdata_log_error("Failed to deserialize kmeans: missing key 'before'");
173
+ return false;
174
+ }
175
+ if (!json_object_is_type(value, json_type_int)) {
176
+ netdata_log_error("Failed to deserialize kmeans: failed to parse int for 'before'");
177
+ return false;
178
+ }
179
+ inlined_km->before = json_object_get_int(value);
180
+
181
+ if (!json_object_object_get_ex(root, "min_dist", &value)) {
182
+ netdata_log_error("Failed to deserialize kmeans: missing key 'min_dist'");
183
+ return false;
184
+ }
185
+ if (!ml_json_parse_double(value, &inlined_km->min_dist)) {
186
+ netdata_log_error("Failed to deserialize kmeans: failed to parse double for 'min_dist'");
187
+ return false;
188
+ }
189
+
190
+ if (!json_object_object_get_ex(root, "max_dist", &value)) {
191
+ netdata_log_error("Failed to deserialize kmeans: missing key 'max_dist'");
192
+ return false;
193
+ }
194
+ if (!ml_json_parse_double(value, &inlined_km->max_dist)) {
195
+ netdata_log_error("Failed to deserialize kmeans: failed to parse double for 'max_dist'");
196
+ return false;
197
+ }
198
+
199
+ struct json_object *cc_root;
200
+ if (!json_object_object_get_ex(root, "cluster_centers", &cc_root)) {
201
+ netdata_log_error("Failed to deserialize kmeans: missing key 'cluster_centers'");
202
+ return false;
203
+ }
204
+ if (!json_object_is_type(cc_root, json_type_array)) {
205
+ netdata_log_error("Failed to deserialize kmeans: failed to parse array for 'cluster_centers'");
206
+ return false;
207
+ }
208
+
209
+ size_t num_centers = json_object_array_length(cc_root);
210
+ if (num_centers != 2) {
211
+ netdata_log_error("Failed to deserialize kmeans: expected cluster centers array of size 2");
212
+ return false;
213
+ }
214
+
215
+ for (size_t i = 0; i < num_centers; i++) {
216
+ struct json_object *cc_obj = json_object_array_get_idx(cc_root, i);
217
+ if (!cc_obj || !json_object_is_type(cc_obj, json_type_array)) {
218
+ netdata_log_error("Failed to deserialize kmeans: expected cluster center array");
219
+ return false;
220
+ }
221
+
222
+ size_t size = json_object_array_length(cc_obj);
223
+ if (size != 6) {
224
+ netdata_log_error("Failed to deserialize kmeans: expected cluster center array of size 6");
225
+ return false;
226
+ }
227
+
228
+ inlined_km->cluster_centers[i].set_size(size);
229
+ for (size_t j = 0; j < size; j++) {
230
+ struct json_object *value = json_object_array_get_idx(cc_obj, j);
231
+ calculated_number_t cn;
232
+
233
+ if (!ml_json_parse_double(value, &cn)) {
234
+ netdata_log_error("Failed to deserialize kmeans: failed to parse double %zu for cluster center %zu", j, i);
235
+ return false;
236
+ }
237
+
238
+ inlined_km->cluster_centers[i](j) = cn;
239
+ }
240
+ }
241
+
242
+ return true;
243
+}
src/ml/ml_kmeans.h
new
+105
@@ -0,0 +1,105 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef ML_KMEANS_H
4
+#define ML_KMEANS_H
5
+
6
+#include "ml_features.h"
7
+
8
+typedef struct web_buffer BUFFER;
9
+
10
+struct ml_kmeans_inlined_t;
11
+
12
+struct ml_kmeans_t {
13
+ std::vector<DSample> cluster_centers;
14
+ calculated_number_t min_dist;
15
+ calculated_number_t max_dist;
16
+ uint32_t after;
17
+ uint32_t before;
18
+
19
+ ml_kmeans_t() : min_dist(0), max_dist(0), after(0), before(0)
20
+ {
21
+ }
22
+
23
+ explicit ml_kmeans_t(const ml_kmeans_inlined_t &inlined);
24
+ ml_kmeans_t &operator=(const ml_kmeans_inlined_t &inlined);
25
+};
26
+
27
+struct ml_kmeans_inlined_t {
28
+ std::array<DSample, 2> cluster_centers;
29
+ calculated_number_t min_dist;
30
+ calculated_number_t max_dist;
31
+ uint32_t after;
32
+ uint32_t before;
33
+
34
+ ml_kmeans_inlined_t() : min_dist(0), max_dist(0), after(0), before(0)
35
+ {
36
+ }
37
+
38
+ explicit ml_kmeans_inlined_t(const ml_kmeans_t &km)
39
+ {
40
+ if (km.cluster_centers.size() != 2) {
41
+ throw std::runtime_error("ml_kmeans_t must have exactly 2 cluster centers");
42
+ }
43
+
44
+ cluster_centers[0] = km.cluster_centers[0];
45
+ cluster_centers[1] = km.cluster_centers[1];
46
+ min_dist = km.min_dist;
47
+ max_dist = km.max_dist;
48
+ after = km.after;
49
+ before = km.before;
50
+ }
51
+
52
+ ml_kmeans_inlined_t &operator=(const ml_kmeans_t &km)
53
+ {
54
+ if (km.cluster_centers.size() != 2) {
55
+ throw std::runtime_error("ml_kmeans_t must have exactly 2 cluster centers");
56
+ }
57
+ cluster_centers[0] = km.cluster_centers[0];
58
+ cluster_centers[1] = km.cluster_centers[1];
59
+ min_dist = km.min_dist;
60
+ max_dist = km.max_dist;
61
+ after = km.after;
62
+ before = km.before;
63
+ return *this;
64
+ }
65
+};
66
+
67
+inline ml_kmeans_t::ml_kmeans_t(const ml_kmeans_inlined_t &inlined_km)
68
+{
69
+ cluster_centers.reserve(2);
70
+ cluster_centers.push_back(inlined_km.cluster_centers[0]);
71
+ cluster_centers.push_back(inlined_km.cluster_centers[1]);
72
+
73
+ min_dist = inlined_km.min_dist;
74
+ max_dist = inlined_km.max_dist;
75
+
76
+ after = inlined_km.after;
77
+ before = inlined_km.before;
78
+}
79
+
80
+inline ml_kmeans_t &ml_kmeans_t::operator=(const ml_kmeans_inlined_t &inlined_km)
81
+{
82
+ cluster_centers.clear();
83
+ cluster_centers.reserve(2);
84
+ cluster_centers.push_back(inlined_km.cluster_centers[0]);
85
+ cluster_centers.push_back(inlined_km.cluster_centers[1]);
86
+
87
+ min_dist = inlined_km.min_dist;
88
+ max_dist = inlined_km.max_dist;
89
+
90
+ after = inlined_km.after;
91
+ before = inlined_km.before;
92
+ return *this;
93
+}
94
+
95
+void ml_kmeans_init(ml_kmeans_t *kmeans);
96
+
97
+void ml_kmeans_train(ml_kmeans_t *kmeans, const ml_features_t *features, unsigned max_iters, time_t after, time_t before);
98
+
99
+calculated_number_t ml_kmeans_anomaly_score(const ml_kmeans_inlined_t *kmeans, const DSample &DS);
100
+
101
+void ml_kmeans_serialize(const ml_kmeans_inlined_t *inlined_km, BUFFER *wb);
102
+
103
+bool ml_kmeans_deserialize(ml_kmeans_inlined_t *inlined_km, struct json_object *root);
104
+
105
+#endif /* ML_KMEANS_H */
src/ml/ml_memory.cc
new
+54
@@ -0,0 +1,54 @@
1
+#include <cstdlib>
2
+#include <memory>
3
+
4
+#include "daemon/telemetry/telemetry-ml.h"
5
+
6
+void *operator new(size_t size)
7
+{
8
+ void *ptr = malloc(size);
9
+ if (!ptr)
10
+ throw std::bad_alloc();
11
+
12
+ telemetry_ml_memory_allocated(size);
13
+ return ptr;
14
+}
15
+
16
+void *operator new[](size_t size)
17
+{
18
+ void *ptr = malloc(size);
19
+ if (!ptr)
20
+ throw std::bad_alloc();
21
+
22
+ telemetry_ml_memory_allocated(size);
23
+ return ptr;
24
+}
25
+
26
+void operator delete(void *ptr, size_t size) noexcept
27
+{
28
+ if (ptr) {
29
+ telemetry_ml_memory_freed(size);
30
+ free(ptr);
31
+ }
32
+}
33
+
34
+void operator delete[](void *ptr, size_t size) noexcept
35
+{
36
+ if (ptr) {
37
+ telemetry_ml_memory_freed(size);
38
+ free(ptr);
39
+ }
40
+}
41
+
42
+void operator delete(void *ptr) noexcept
43
+{
44
+ if (ptr) {
45
+ free(ptr);
46
+ }
47
+}
48
+
49
+void operator delete[](void *ptr) noexcept
50
+{
51
+ if (ptr) {
52
+ free(ptr);
53
+ }
54
+}
src/ml/ml_private.h
new
+18
@@ -0,0 +1,18 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_ML_PRIVATE_H
4
+#define NETDATA_ML_PRIVATE_H
5
+
6
+#include <vector>
7
+#include <unordered_map>
8
+
9
+#include "ml_config.h"
10
+
11
+void *ml_train_main(void *arg);
12
+void *ml_detect_main(void *arg);
13
+
14
+extern sqlite3 *ml_db;
15
+extern const char *db_models_create_table;
16
+
17
+
18
+#endif /* NETDATA_ML_PRIVATE_H */
src/ml/ml_public.cc
new
+483
@@ -0,0 +1,483 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "ml_private.h"
4
+
5
+#include <random>
6
+
7
+#define ML_METADATA_VERSION 2
8
+
9
+bool ml_capable()
10
+{
11
+ return true;
12
+}
13
+
14
+bool ml_enabled(RRDHOST *rh)
15
+{
16
+ if (!rh)
17
+ return false;
18
+
19
+ if (!Cfg.enable_anomaly_detection)
20
+ return false;
21
+
22
+ if (simple_pattern_matches(Cfg.sp_host_to_skip, rrdhost_hostname(rh)))
23
+ return false;
24
+
25
+ return true;
26
+}
27
+
28
+bool ml_streaming_enabled()
29
+{
30
+ return Cfg.stream_anomaly_detection_charts;
31
+}
32
+
33
+void ml_host_new(RRDHOST *rh)
34
+{
35
+ if (!ml_enabled(rh))
36
+ return;
37
+
38
+ ml_host_t *host = new ml_host_t();
39
+
40
+ host->rh = rh;
41
+ host->mls = ml_machine_learning_stats_t();
42
+ host->host_anomaly_rate = 0.0;
43
+ host->anomaly_rate_rs = NULL;
44
+
45
+ static std::atomic<size_t> times_called(0);
46
+ host->queue = Cfg.workers[times_called++ % Cfg.num_worker_threads].queue;
47
+
48
+ netdata_mutex_init(&host->mutex);
49
+ spinlock_init(&host->type_anomaly_rate_spinlock);
50
+
51
+ host->ml_running = true;
52
+ rh->ml_host = (rrd_ml_host_t *) host;
53
+}
54
+
55
+void ml_host_delete(RRDHOST *rh)
56
+{
57
+ ml_host_t *host = (ml_host_t *) rh->ml_host;
58
+ if (!host)
59
+ return;
60
+
61
+ netdata_mutex_destroy(&host->mutex);
62
+
63
+ delete host;
64
+ rh->ml_host = NULL;
65
+}
66
+
67
+void ml_host_start(RRDHOST *rh) {
68
+ ml_host_t *host = (ml_host_t *) rh->ml_host;
69
+ if (!host)
70
+ return;
71
+
72
+ host->ml_running = true;
73
+}
74
+
75
+void ml_host_stop(RRDHOST *rh) {
76
+ ml_host_t *host = (ml_host_t *) rh->ml_host;
77
+ if (!host || !host->ml_running)
78
+ return;
79
+
80
+ netdata_mutex_lock(&host->mutex);
81
+
82
+ // reset host stats
83
+ host->mls = ml_machine_learning_stats_t();
84
+
85
+ // reset charts/dims
86
+ void *rsp = NULL;
87
+ rrdset_foreach_read(rsp, host->rh) {
88
+ RRDSET *rs = static_cast<RRDSET *>(rsp);
89
+
90
+ ml_chart_t *chart = (ml_chart_t *) rs->ml_chart;
91
+ if (!chart)
92
+ continue;
93
+
94
+ // reset chart
95
+ chart->mls = ml_machine_learning_stats_t();
96
+
97
+ void *rdp = NULL;
98
+ rrddim_foreach_read(rdp, rs) {
99
+ RRDDIM *rd = static_cast<RRDDIM *>(rdp);
100
+
101
+ ml_dimension_t *dim = (ml_dimension_t *) rd->ml_dimension;
102
+ if (!dim)
103
+ continue;
104
+
105
+ spinlock_lock(&dim->slock);
106
+
107
+ // reset dim
108
+ // TODO: should we drop in-mem models, or mark them as stale? Is it
109
+ // okay to resume training straight away?
110
+
111
+ dim->mt = METRIC_TYPE_CONSTANT;
112
+ dim->ts = TRAINING_STATUS_UNTRAINED;
113
+ dim->last_training_time = 0;
114
+ dim->suppression_anomaly_counter = 0;
115
+ dim->suppression_window_counter = 0;
116
+ dim->cns.clear();
117
+
118
+ ml_kmeans_init(&dim->kmeans);
119
+
120
+ spinlock_unlock(&dim->slock);
121
+ }
122
+ rrddim_foreach_done(rdp);
123
+ }
124
+ rrdset_foreach_done(rsp);
125
+
126
+ netdata_mutex_unlock(&host->mutex);
127
+
128
+ host->ml_running = false;
129
+}
130
+
131
+void ml_host_get_info(RRDHOST *rh, BUFFER *wb)
132
+{
133
+ ml_host_t *host = (ml_host_t *) rh->ml_host;
134
+ if (!host) {
135
+ buffer_json_member_add_boolean(wb, "enabled", false);
136
+ return;
137
+ }
138
+
139
+ buffer_json_member_add_uint64(wb, "version", 1);
140
+
141
+ buffer_json_member_add_boolean(wb, "enabled", Cfg.enable_anomaly_detection);
142
+
143
+ buffer_json_member_add_uint64(wb, "min-train-samples", Cfg.min_train_samples);
144
+ buffer_json_member_add_uint64(wb, "max-train-samples", Cfg.max_train_samples);
145
+ buffer_json_member_add_uint64(wb, "train-every", Cfg.train_every);
146
+
147
+ buffer_json_member_add_uint64(wb, "diff-n", Cfg.diff_n);
148
+ buffer_json_member_add_uint64(wb, "smooth-n", Cfg.smooth_n);
149
+ buffer_json_member_add_uint64(wb, "lag-n", Cfg.lag_n);
150
+
151
+ buffer_json_member_add_double(wb, "random-sampling-ratio", Cfg.random_sampling_ratio);
152
+ buffer_json_member_add_uint64(wb, "max-kmeans-iters", Cfg.random_sampling_ratio);
153
+
154
+ buffer_json_member_add_double(wb, "dimension-anomaly-score-threshold", Cfg.dimension_anomaly_score_threshold);
155
+
156
+ buffer_json_member_add_string(wb, "anomaly-detection-grouping-method", time_grouping_id2txt(Cfg.anomaly_detection_grouping_method));
157
+
158
+ buffer_json_member_add_int64(wb, "anomaly-detection-query-duration", Cfg.anomaly_detection_query_duration);
159
+
160
+ buffer_json_member_add_string(wb, "hosts-to-skip", Cfg.hosts_to_skip.c_str());
161
+ buffer_json_member_add_string(wb, "charts-to-skip", Cfg.charts_to_skip.c_str());
162
+}
163
+
164
+void ml_host_get_detection_info(RRDHOST *rh, BUFFER *wb)
165
+{
166
+ ml_host_t *host = (ml_host_t *) rh->ml_host;
167
+ if (!host)
168
+ return;
169
+
170
+ netdata_mutex_lock(&host->mutex);
171
+
172
+ buffer_json_member_add_uint64(wb, "version", 2);
173
+ buffer_json_member_add_uint64(wb, "ml-running", host->ml_running);
174
+ buffer_json_member_add_uint64(wb, "anomalous-dimensions", host->mls.num_anomalous_dimensions);
175
+ buffer_json_member_add_uint64(wb, "normal-dimensions", host->mls.num_normal_dimensions);
176
+ buffer_json_member_add_uint64(wb, "total-dimensions", host->mls.num_anomalous_dimensions +
177
+ host->mls.num_normal_dimensions);
178
+ buffer_json_member_add_uint64(wb, "trained-dimensions", host->mls.num_training_status_trained +
179
+ host->mls.num_training_status_pending_with_model);
180
+ netdata_mutex_unlock(&host->mutex);
181
+}
182
+
183
+bool ml_host_get_host_status(RRDHOST *rh, struct ml_metrics_statistics *mlm) {
184
+ ml_host_t *host = (ml_host_t *) rh->ml_host;
185
+ if (!host) {
186
+ memset(mlm, 0, sizeof(*mlm));
187
+ return false;
188
+ }
189
+
190
+ netdata_mutex_lock(&host->mutex);
191
+
192
+ mlm->anomalous = host->mls.num_anomalous_dimensions;
193
+ mlm->normal = host->mls.num_normal_dimensions;
194
+ mlm->trained = host->mls.num_training_status_trained + host->mls.num_training_status_pending_with_model;
195
+ mlm->pending = host->mls.num_training_status_untrained + host->mls.num_training_status_pending_without_model;
196
+ mlm->silenced = host->mls.num_training_status_silenced;
197
+
198
+ netdata_mutex_unlock(&host->mutex);
199
+
200
+ return true;
201
+}
202
+
203
+bool ml_host_running(RRDHOST *rh) {
204
+ ml_host_t *host = (ml_host_t *) rh->ml_host;
205
+ if(!host)
206
+ return false;
207
+
208
+ return true;
209
+}
210
+
211
+void ml_host_get_models(RRDHOST *rh, BUFFER *wb)
212
+{
213
+ UNUSED(rh);
214
+ UNUSED(wb);
215
+
216
+ // TODO: To be implemented
217
+ netdata_log_error("Fetching KMeans models is not supported yet");
218
+}
219
+
220
+void ml_chart_new(RRDSET *rs)
221
+{
222
+ ml_host_t *host = (ml_host_t *) rs->rrdhost->ml_host;
223
+ if (!host)
224
+ return;
225
+
226
+ ml_chart_t *chart = new ml_chart_t();
227
+
228
+ chart->rs = rs;
229
+ chart->mls = ml_machine_learning_stats_t();
230
+
231
+ rs->ml_chart = (rrd_ml_chart_t *) chart;
232
+}
233
+
234
+void ml_chart_delete(RRDSET *rs)
235
+{
236
+ ml_host_t *host = (ml_host_t *) rs->rrdhost->ml_host;
237
+ if (!host)
238
+ return;
239
+
240
+ ml_chart_t *chart = (ml_chart_t *) rs->ml_chart;
241
+
242
+ delete chart;
243
+ rs->ml_chart = NULL;
244
+}
245
+
246
+bool ml_chart_update_begin(RRDSET *rs)
247
+{
248
+ ml_chart_t *chart = (ml_chart_t *) rs->ml_chart;
249
+ if (!chart)
250
+ return false;
251
+
252
+ chart->mls = {};
253
+ return true;
254
+}
255
+
256
+void ml_chart_update_end(RRDSET *rs)
257
+{
258
+ ml_chart_t *chart = (ml_chart_t *) rs->ml_chart;
259
+ if (!chart)
260
+ return;
261
+}
262
+
263
+void ml_dimension_new(RRDDIM *rd)
264
+{
265
+ ml_chart_t *chart = (ml_chart_t *) rd->rrdset->ml_chart;
266
+ if (!chart)
267
+ return;
268
+
269
+ ml_dimension_t *dim = new ml_dimension_t();
270
+
271
+ dim->rd = rd;
272
+
273
+ dim->mt = METRIC_TYPE_CONSTANT;
274
+ dim->ts = TRAINING_STATUS_UNTRAINED;
275
+ dim->last_training_time = 0;
276
+ dim->suppression_anomaly_counter = 0;
277
+ dim->suppression_window_counter = 0;
278
+
279
+ ml_kmeans_init(&dim->kmeans);
280
+
281
+ if (simple_pattern_matches(Cfg.sp_charts_to_skip, rrdset_name(rd->rrdset)))
282
+ dim->mls = MACHINE_LEARNING_STATUS_DISABLED_DUE_TO_EXCLUDED_CHART;
283
+ else
284
+ dim->mls = MACHINE_LEARNING_STATUS_ENABLED;
285
+
286
+ spinlock_init(&dim->slock);
287
+
288
+ dim->km_contexts.reserve(Cfg.num_models_to_use);
289
+
290
+ rd->ml_dimension = (rrd_ml_dimension_t *) dim;
291
+
292
+ metaqueue_ml_load_models(rd);
293
+}
294
+
295
+void ml_dimension_delete(RRDDIM *rd)
296
+{
297
+ ml_dimension_t *dim = (ml_dimension_t *) rd->ml_dimension;
298
+ if (!dim)
299
+ return;
300
+
301
+ delete dim;
302
+ rd->ml_dimension = NULL;
303
+}
304
+
305
+void ml_dimension_received_anomaly(RRDDIM *rd, bool is_anomalous) {
306
+ ml_dimension_t *dim = (ml_dimension_t *) rd->ml_dimension;
307
+ if (!dim)
308
+ return;
309
+
310
+ ml_host_t *host = (ml_host_t *) rd->rrdset->rrdhost->ml_host;
311
+ if (!host->ml_running)
312
+ return;
313
+
314
+ ml_chart_t *chart = (ml_chart_t *) rd->rrdset->ml_chart;
315
+
316
+ ml_chart_update_dimension(chart, dim, is_anomalous);
317
+}
318
+
319
+bool ml_dimension_is_anomalous(RRDDIM *rd, time_t curr_time, double value, bool exists)
320
+{
321
+ ml_dimension_t *dim = (ml_dimension_t *) rd->ml_dimension;
322
+ if (!dim)
323
+ return false;
324
+
325
+ ml_host_t *host = (ml_host_t *) rd->rrdset->rrdhost->ml_host;
326
+ if (!host->ml_running)
327
+ return false;
328
+
329
+ ml_chart_t *chart = (ml_chart_t *) rd->rrdset->ml_chart;
330
+
331
+ bool is_anomalous = ml_dimension_predict(dim, curr_time, value, exists);
332
+ ml_chart_update_dimension(chart, dim, is_anomalous);
333
+
334
+ return is_anomalous;
335
+}
336
+
337
+void ml_init()
338
+{
339
+ // Read config values
340
+ ml_config_load(&Cfg);
341
+
342
+ if (!Cfg.enable_anomaly_detection)
343
+ return;
344
+
345
+ // Generate random numbers to efficiently sample the features we need
346
+ // for KMeans clustering.
347
+ std::random_device RD;
348
+ std::mt19937 Gen(RD());
349
+
350
+ Cfg.random_nums.reserve(Cfg.max_train_samples);
351
+ for (size_t Idx = 0; Idx != Cfg.max_train_samples; Idx++)
352
+ Cfg.random_nums.push_back(Gen());
353
+
354
+ // init training thread-specific data
355
+ Cfg.workers.resize(Cfg.num_worker_threads);
356
+ for (size_t idx = 0; idx != Cfg.num_worker_threads; idx++) {
357
+ ml_worker_t *worker = &Cfg.workers[idx];
358
+
359
+ size_t max_elements_needed_for_training = (size_t) Cfg.max_train_samples * (size_t) (Cfg.lag_n + 1);
360
+ worker->training_cns = new calculated_number_t[max_elements_needed_for_training]();
361
+ worker->scratch_training_cns = new calculated_number_t[max_elements_needed_for_training]();
362
+
363
+ worker->id = idx;
364
+ worker->queue = ml_queue_init();
365
+ worker->pending_model_info.reserve(Cfg.flush_models_batch_size);
366
+ netdata_mutex_init(&worker->nd_mutex);
367
+ }
368
+
369
+ // open sqlite db
370
+ char path[FILENAME_MAX];
371
+ snprintfz(path, FILENAME_MAX - 1, "%s/%s", netdata_configured_cache_dir, "ml.db");
372
+ int rc = sqlite3_open(path, &ml_db);
373
+ if (rc != SQLITE_OK) {
374
+ error_report("Failed to initialize database at %s, due to \"%s\"", path, sqlite3_errstr(rc));
375
+ sqlite3_close(ml_db);
376
+ ml_db = NULL;
377
+ }
378
+
379
+ // create table
380
+ if (ml_db) {
381
+ int target_version = perform_ml_database_migration(ml_db, ML_METADATA_VERSION);
382
+ if (configure_sqlite_database(ml_db, target_version, "ml_config")) {
383
+ error_report("Failed to setup ML database");
384
+ sqlite3_close(ml_db);
385
+ ml_db = NULL;
386
+ }
387
+ else {
388
+ char *err = NULL;
389
+ int rc = sqlite3_exec(ml_db, db_models_create_table, NULL, NULL, &err);
390
+ if (rc != SQLITE_OK) {
391
+ error_report("Failed to create models table (%s, %s)", sqlite3_errstr(rc), err ? err : "");
392
+ sqlite3_close(ml_db);
393
+ sqlite3_free(err);
394
+ ml_db = NULL;
395
+ }
396
+ }
397
+ }
398
+}
399
+
400
+uint64_t sqlite_get_ml_space(void)
401
+{
402
+ return sqlite_get_db_space(ml_db);
403
+}
404
+
405
+void ml_fini() {
406
+ if (!Cfg.enable_anomaly_detection || !ml_db)
407
+ return;
408
+
409
+ sql_close_database(ml_db, "ML");
410
+ ml_db = NULL;
411
+}
412
+
413
+void ml_start_threads() {
414
+ if (!Cfg.enable_anomaly_detection)
415
+ return;
416
+
417
+ // start detection & training threads
418
+ Cfg.detection_stop = false;
419
+ Cfg.training_stop = false;
420
+
421
+ char tag[NETDATA_THREAD_TAG_MAX + 1];
422
+
423
+ snprintfz(tag, NETDATA_THREAD_TAG_MAX, "%s", "PREDICT");
424
+ Cfg.detection_thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_JOINABLE,
425
+ ml_detect_main, NULL);
426
+
427
+ for (size_t idx = 0; idx != Cfg.num_worker_threads; idx++) {
428
+ ml_worker_t *worker = &Cfg.workers[idx];
429
+ snprintfz(tag, NETDATA_THREAD_TAG_MAX, "TRAIN[%zu]", worker->id);
430
+ worker->nd_thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_JOINABLE,
431
+ ml_train_main, worker);
432
+ }
433
+}
434
+
435
+void ml_stop_threads()
436
+{
437
+ if (!Cfg.enable_anomaly_detection)
438
+ return;
439
+
440
+ Cfg.detection_stop = true;
441
+ Cfg.training_stop = true;
442
+
443
+ if (!Cfg.detection_thread)
444
+ return;
445
+
446
+ nd_thread_join(Cfg.detection_thread);
447
+ Cfg.detection_thread = 0;
448
+
449
+ // signal the worker queue of each thread
450
+ for (size_t idx = 0; idx != Cfg.num_worker_threads; idx++) {
451
+ ml_worker_t *worker = &Cfg.workers[idx];
452
+ ml_queue_signal(worker->queue);
453
+ }
454
+
455
+ // join worker threads
456
+ for (size_t idx = 0; idx != Cfg.num_worker_threads; idx++) {
457
+ ml_worker_t *worker = &Cfg.workers[idx];
458
+
459
+ nd_thread_join(worker->nd_thread);
460
+ }
461
+
462
+ // clear worker thread data
463
+ for (size_t idx = 0; idx != Cfg.num_worker_threads; idx++) {
464
+ ml_worker_t *worker = &Cfg.workers[idx];
465
+
466
+ delete[] worker->training_cns;
467
+ delete[] worker->scratch_training_cns;
468
+ ml_queue_destroy(worker->queue);
469
+ netdata_mutex_destroy(&worker->nd_mutex);
470
+ }
471
+}
472
+
473
+bool ml_model_received_from_child(RRDHOST *host, const char *json)
474
+{
475
+ UNUSED(host);
476
+
477
+ bool ok = ml_dimension_deserialize_kmeans(json);
478
+ if (!ok) {
479
+ global_statistics_ml_models_deserialization_failures();
480
+ }
481
+
482
+ return ok;
483
+}
src/ml/ml_public.h
renamed
+14
-4
@@ -1,7 +1,7 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#ifndef NETDATA_ML_H
4
-#define NETDATA_ML_H
3
+#ifndef NETDATA_ML_PUBLIC_H
4
+#define NETDATA_ML_PUBLIC_H
5
6
#ifdef __cplusplus
7
extern "C" {
@@ -39,17 +39,27 @@ void ml_chart_update_end(RRDSET *rs);
39
void ml_dimension_new(RRDDIM *rd);
40
void ml_dimension_delete(RRDDIM *rd);
41
bool ml_dimension_is_anomalous(RRDDIM *rd, time_t curr_time, double value, bool exists);
42
+void ml_dimension_received_anomaly(RRDDIM *rd, bool is_anomalous);
43
44
int ml_dimension_load_models(RRDDIM *rd, sqlite3_stmt **stmt);
45
45
-void ml_update_global_statistics_charts(uint64_t models_consulted);
46
+void ml_update_global_statistics_charts(uint64_t models_consulted,
47
+ uint64_t models_received,
48
+ uint64_t models_sent,
49
+ uint64_t models_ignored,
50
+ uint64_t models_deserialization_failures,
51
+ uint64_t memory_consumption,
52
+ uint64_t memory_new,
53
+ uint64_t memory_delete);
54
55
bool ml_host_get_host_status(RRDHOST *rh, struct ml_metrics_statistics *mlm);
56
bool ml_host_running(RRDHOST *rh);
57
uint64_t sqlite_get_ml_space(void);
58
59
+bool ml_model_received_from_child(RRDHOST *host, const char *json);
60
+
61
#ifdef __cplusplus
62
};
63
#endif
64
55
-#endif /* NETDATA_ML_H */
65
+#endif /* NETDATA_ML_PUBLIC_H */
src/ml/ml_queue.cc
new
+67
@@ -0,0 +1,67 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "ml_private.h"
4
+
5
+ml_queue_t * ml_queue_init()
6
+{
7
+ ml_queue_t *q = new ml_queue_t();
8
+
9
+ netdata_mutex_init(&q->mutex);
10
+ pthread_cond_init(&q->cond_var, NULL);
11
+ q->exit = false;
12
+ return q;
13
+}
14
+
15
+void ml_queue_destroy(ml_queue_t *q)
16
+{
17
+ netdata_mutex_destroy(&q->mutex);
18
+ pthread_cond_destroy(&q->cond_var);
19
+ delete q;
20
+}
21
+
22
+void ml_queue_push(ml_queue_t *q, const ml_queue_item_t req)
23
+{
24
+ netdata_mutex_lock(&q->mutex);
25
+ q->internal.push(req);
26
+ pthread_cond_signal(&q->cond_var);
27
+ netdata_mutex_unlock(&q->mutex);
28
+}
29
+
30
+ml_queue_item_t ml_queue_pop(ml_queue_t *q)
31
+{
32
+ netdata_mutex_lock(&q->mutex);
33
+
34
+ ml_queue_item_t req;
35
+ req.type = ML_QUEUE_ITEM_STOP_REQUEST;
36
+
37
+ while (q->internal.empty()) {
38
+ pthread_cond_wait(&q->cond_var, &q->mutex);
39
+
40
+ if (q->exit) {
41
+ netdata_mutex_unlock(&q->mutex);
42
+ return req;
43
+ }
44
+ }
45
+
46
+ req = q->internal.front();
47
+ q->internal.pop();
48
+
49
+ netdata_mutex_unlock(&q->mutex);
50
+ return req;
51
+}
52
+
53
+size_t ml_queue_size(ml_queue_t *q)
54
+{
55
+ netdata_mutex_lock(&q->mutex);
56
+ size_t size = q->internal.size();
57
+ netdata_mutex_unlock(&q->mutex);
58
+ return size;
59
+}
60
+
61
+void ml_queue_signal(ml_queue_t *q)
62
+{
63
+ netdata_mutex_lock(&q->mutex);
64
+ q->exit = true;
65
+ pthread_cond_signal(&q->cond_var);
66
+ netdata_mutex_unlock(&q->mutex);
67
+}
src/ml/ml_queue.h
new
+69
@@ -0,0 +1,69 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef ML_QUEUE_H
4
+#define ML_QUEUE_H
5
+
6
+#include "ml_dimension.h"
7
+
8
+#include <atomic>
9
+#include <queue>
10
+
11
+typedef struct ml_request_create_new_model {
12
+ DimensionLookupInfo DLI;
13
+
14
+ // Creation time of request
15
+ time_t request_time;
16
+
17
+ // First/last entry of this dimension in DB
18
+ // at the point the request was made
19
+ time_t first_entry_on_request;
20
+ time_t last_entry_on_request;
21
+} ml_request_create_new_model_t;
22
+
23
+typedef struct ml_request_add_existing_model {
24
+ DimensionLookupInfo DLI;
25
+
26
+ ml_kmeans_inlined_t inlined_km;
27
+} ml_request_add_existing_model_t;
28
+
29
+typedef struct ml_queue_item {
30
+ ml_queue_item_type type;
31
+ ml_request_create_new_model_t create_new_model;
32
+ ml_request_add_existing_model add_existing_model;
33
+} ml_queue_item_t;
34
+
35
+struct ml_queue_t {
36
+ std::queue<ml_queue_item_t> internal;
37
+ netdata_mutex_t mutex;
38
+ pthread_cond_t cond_var;
39
+ std::atomic<bool> exit;
40
+};
41
+
42
+typedef struct {
43
+ size_t queue_size;
44
+ size_t num_popped_items;
45
+
46
+ usec_t allotted_ut;
47
+ usec_t consumed_ut;
48
+ usec_t remaining_ut;
49
+
50
+ size_t item_result_ok;
51
+ size_t item_result_invalid_query_time_range;
52
+ size_t item_result_not_enough_collected_values;
53
+ size_t item_result_null_acquired_dimension;
54
+ size_t item_result_chart_under_replication;
55
+} ml_queue_stats_t;
56
+
57
+ml_queue_t *ml_queue_init();
58
+
59
+void ml_queue_destroy(ml_queue_t *q);
60
+
61
+void ml_queue_push(ml_queue_t *q, const ml_queue_item_t req);
62
+
63
+ml_queue_item_t ml_queue_pop(ml_queue_t *q);
64
+
65
+size_t ml_queue_size(ml_queue_t *q);
66
+
67
+void ml_queue_signal(ml_queue_t *q);
68
+
69
+#endif /* ML_QUEUE_H */
src/ml/ml_string_wrapper.h
new
+81
@@ -0,0 +1,81 @@
1
+#ifndef ML_STRING_WRAPPER_H
2
+#define ML_STRING_WRAPPER_H
3
+
4
+#include "libnetdata/libnetdata.h"
5
+
6
+#include <algorithm>
7
+
8
+class StringWrapper {
9
+public:
10
+ StringWrapper() noexcept : Inner(nullptr)
11
+ {
12
+ }
13
+
14
+ explicit StringWrapper(const char *S) noexcept : Inner(string_strdupz(S))
15
+ {
16
+ }
17
+
18
+ explicit StringWrapper(STRING *S) noexcept : Inner(string_dup(S))
19
+ {
20
+ }
21
+
22
+ StringWrapper(const StringWrapper &Other) noexcept : Inner(string_dup(Other.Inner))
23
+ {
24
+ }
25
+
26
+ StringWrapper &operator=(const StringWrapper &Other) noexcept
27
+ {
28
+ if (this != &Other) {
29
+ STRING *Tmp = string_dup(Other.Inner);
30
+ string_freez(Inner);
31
+ Inner = Tmp;
32
+ }
33
+ return *this;
34
+ }
35
+
36
+ StringWrapper(StringWrapper &&Other) noexcept : Inner(Other.Inner)
37
+ {
38
+ Other.Inner = nullptr;
39
+ }
40
+
41
+ StringWrapper &operator=(StringWrapper &&Other) noexcept
42
+ {
43
+ if (this != &Other) {
44
+ string_freez(Inner);
45
+ Inner = Other.Inner;
46
+ Other.Inner = nullptr;
47
+ }
48
+ return *this;
49
+ }
50
+
51
+ ~StringWrapper()
52
+ {
53
+ string_freez(Inner);
54
+ }
55
+
56
+ STRING *inner() const noexcept
57
+ {
58
+ return Inner;
59
+ }
60
+
61
+ operator const char *() const noexcept
62
+ {
63
+ return string2str(Inner);
64
+ }
65
+
66
+ void swap(StringWrapper &Other) noexcept
67
+ {
68
+ std::swap(Inner, Other.Inner);
69
+ }
70
+
71
+private:
72
+ STRING *Inner;
73
+};
74
+
75
+// Free swap function
76
+inline void swap(StringWrapper &LHS, StringWrapper &RHS) noexcept
77
+{
78
+ LHS.swap(RHS);
79
+}
80
+
81
+#endif /* ML_STRING_WRAPPER_H */
src/ml/ml_worker.h
new
+47
@@ -0,0 +1,47 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef ML_WORKER_H
4
+#define ML_WORKER_H
5
+
6
+#include "ml_queue.h"
7
+
8
+typedef struct {
9
+ nd_uuid_t metric_uuid;
10
+ ml_kmeans_inlined_t inlined_kmeans;
11
+} ml_model_info_t;
12
+
13
+typedef struct {
14
+ size_t id;
15
+ ND_THREAD *nd_thread;
16
+ netdata_mutex_t nd_mutex;
17
+
18
+ ml_queue_t *queue;
19
+ ml_queue_stats_t queue_stats;
20
+
21
+ calculated_number_t *training_cns;
22
+ calculated_number_t *scratch_training_cns;
23
+ std::vector<DSample> training_samples;
24
+
25
+ std::vector<ml_model_info_t> pending_model_info;
26
+
27
+ RRDSET *queue_stats_rs;
28
+ RRDDIM *queue_stats_queue_size_rd;
29
+ RRDDIM *queue_stats_popped_items_rd;
30
+
31
+ RRDSET *training_time_stats_rs;
32
+ RRDDIM *training_time_stats_allotted_rd;
33
+ RRDDIM *training_time_stats_consumed_rd;
34
+ RRDDIM *training_time_stats_remaining_rd;
35
+
36
+ RRDSET *training_results_rs;
37
+ RRDDIM *training_results_ok_rd;
38
+ RRDDIM *training_results_invalid_query_time_range_rd;
39
+ RRDDIM *training_results_not_enough_collected_values_rd;
40
+ RRDDIM *training_results_null_acquired_dimension_rd;
41
+ RRDDIM *training_results_chart_under_replication_rd;
42
+
43
+ size_t num_db_transactions;
44
+ size_t num_models_to_prune;
45
+} ml_worker_t;
46
+
47
+#endif /* ML_WORKER_H */
src/plugins.d/README.md
+1
-1
@@ -314,7 +314,7 @@ the template is:
314
315
- `charttype`
316
317
- one of `line`, `area` or `stacked`,
317
+ one of `line`, `area`, `stacked` or `heatmap`,
318
if empty or missing, the `line` will be used
319
320
- `priority`
src/plugins.d/plugins_d.h
+5
-3
@@ -3,7 +3,9 @@
3
#ifndef NETDATA_PLUGINS_D_H
4
#define NETDATA_PLUGINS_D_H 1
5
6
-#include "daemon/common.h"
6
+#include "libnetdata/libnetdata.h"
7
+
8
+struct rrdhost;
9
10
#define PLUGINSD_CMD_MAX (FILENAME_MAX*2)
11
#define PLUGINSD_STOCK_PLUGINS_DIRECTORY_PATH 0
@@ -24,7 +26,7 @@ struct plugind {
26
size_t serial_failures; // the number of times the plugin started
27
// without collecting values
28
27
- RRDHOST *host; // the host the plugin collects data for
29
+ struct rrdhost *host; // the host the plugin collects data for
30
int update_every; // the plugin default data collection frequency
31
32
struct {
@@ -44,7 +46,7 @@ struct plugind {
46
47
extern struct plugind *pluginsd_root;
48
47
-size_t pluginsd_process(RRDHOST *host, struct plugind *cd, int fd_input, int fd_output, int trust_durations);
49
+size_t pluginsd_process(struct rrdhost *host, struct plugind *cd, int fd_input, int fd_output, int trust_durations);
50
51
struct parser;
52
void pluginsd_process_cleanup(struct parser *parser);
src/plugins.d/pluginsd_internals.c
+24
-31
@@ -11,38 +11,22 @@ ssize_t send_to_plugin(const char *txt, PARSER *parser) {
11
return h2o_stream_write(parser->h2o_ctx, txt, strlen(txt));
12
#endif
13
14
- errno_clear();
14
spinlock_lock(&parser->writer.spinlock);
16
- ssize_t bytes = -1;
15
18
- NETDATA_SSL *ssl = parser->ssl_output;
19
- if(ssl) {
16
+ ND_SOCK tmp = { .fd = parser->fd_output, };
17
+ ND_SOCK *s = parser->sock; // try the socket
18
+ if(!s) s = &tmp; // socket is not there, use the pipe
19
21
- if(SSL_connection(ssl))
22
- bytes = netdata_ssl_write(ssl, (void *) txt, strlen(txt));
20
+ if(s->fd != -1) {
21
+ // plugins pipe or socket (with or without SSL)
22
24
- else
25
- netdata_log_error("PLUGINSD: cannot send command (SSL)");
26
-
27
- spinlock_unlock(&parser->writer.spinlock);
28
- return bytes;
29
- }
30
-
31
- if(parser->fd_output != -1) {
32
- bytes = 0;
33
- ssize_t total = (ssize_t)strlen(txt);
34
- ssize_t sent;
35
-
36
- do {
37
- sent = write(parser->fd_output, &txt[bytes], total - bytes);
38
- if(sent <= 0) {
39
- netdata_log_error("PLUGINSD: cannot send command (fd)");
40
- spinlock_unlock(&parser->writer.spinlock);
41
- return -3;
42
- }
43
- bytes += sent;
23
+ size_t total = strlen(txt);
24
+ ssize_t bytes = nd_sock_write_persist(s, txt, total, 100);
25
+ if(bytes < (ssize_t)total) {
26
+ netdata_log_error("PLUGINSD: cannot send command (fd = %d, bytes = %zd out of %zu)", s->fd, bytes, total);
27
+ spinlock_unlock(&parser->writer.spinlock);
28
+ return -3;
29
}
45
- while(bytes < total);
30
31
spinlock_unlock(&parser->writer.spinlock);
32
return (int)bytes;
@@ -83,15 +67,24 @@ void parser_destroy(PARSER *parser) {
67
68
69
PARSER *parser_init(struct parser_user_object *user, int fd_input, int fd_output,
86
- PARSER_INPUT_TYPE flags, void *ssl __maybe_unused) {
70
+ PARSER_INPUT_TYPE flags, ND_SOCK *sock) {
71
PARSER *parser;
72
73
parser = callocz(1, sizeof(*parser));
74
+
75
if(user)
76
parser->user = *user;
92
- parser->fd_input = fd_input;
93
- parser->fd_output = fd_output;
94
- parser->ssl_output = ssl;
77
+
78
+ if(sock) {
79
+ parser->fd_input = sock->fd;
80
+ parser->fd_output = sock->fd;
81
+ parser->sock = sock;
82
+ }
83
+ else {
84
+ parser->fd_input = fd_input;
85
+ parser->fd_output = fd_output;
86
+ }
87
+
88
parser->flags = flags;
89
90
spinlock_init(&parser->writer.spinlock);
src/plugins.d/pluginsd_internals.h
+11
-11
@@ -282,34 +282,34 @@ static inline void pluginsd_rrdset_cache_put_to_slot(PARSER *parser, RRDSET *st,
282
283
RRDHOST *host = st->rrdhost;
284
285
- if(unlikely((size_t)slot > host->rrdpush.receive.pluginsd_chart_slots.size)) {
286
- spinlock_lock(&host->rrdpush.receive.pluginsd_chart_slots.spinlock);
287
- size_t old_slots = host->rrdpush.receive.pluginsd_chart_slots.size;
285
+ if(unlikely((size_t)slot > host->stream.rcv.pluginsd_chart_slots.size)) {
286
+ spinlock_lock(&host->stream.rcv.pluginsd_chart_slots.spinlock);
287
+ size_t old_slots = host->stream.rcv.pluginsd_chart_slots.size;
288
size_t new_slots = (old_slots < PLUGINSD_MIN_RRDSET_POINTERS_CACHE) ? PLUGINSD_MIN_RRDSET_POINTERS_CACHE : old_slots * 2;
289
290
if(new_slots < (size_t)slot)
291
new_slots = slot;
292
293
- host->rrdpush.receive.pluginsd_chart_slots.array =
294
- reallocz(host->rrdpush.receive.pluginsd_chart_slots.array, new_slots * sizeof(RRDSET *));
293
+ host->stream.rcv.pluginsd_chart_slots.array =
294
+ reallocz(host->stream.rcv.pluginsd_chart_slots.array, new_slots * sizeof(RRDSET *));
295
296
for(size_t i = old_slots; i < new_slots ;i++)
297
- host->rrdpush.receive.pluginsd_chart_slots.array[i] = NULL;
297
+ host->stream.rcv.pluginsd_chart_slots.array[i] = NULL;
298
299
- host->rrdpush.receive.pluginsd_chart_slots.size = new_slots;
300
- spinlock_unlock(&host->rrdpush.receive.pluginsd_chart_slots.spinlock);
299
+ host->stream.rcv.pluginsd_chart_slots.size = new_slots;
300
+ spinlock_unlock(&host->stream.rcv.pluginsd_chart_slots.spinlock);
301
}
302
303
- host->rrdpush.receive.pluginsd_chart_slots.array[slot - 1] = st;
303
+ host->stream.rcv.pluginsd_chart_slots.array[slot - 1] = st;
304
st->pluginsd.last_slot = (int32_t)slot - 1;
305
parser->user.cleanup_slots = obsolete;
306
}
307
308
static inline RRDSET *pluginsd_rrdset_cache_get_from_slot(PARSER *parser, RRDHOST *host, const char *id, ssize_t slot, const char *keyword) {
309
- if(unlikely(slot < 1 || (size_t)slot > host->rrdpush.receive.pluginsd_chart_slots.size))
309
+ if(unlikely(slot < 1 || (size_t)slot > host->stream.rcv.pluginsd_chart_slots.size))
310
return pluginsd_find_chart(host, id, keyword);
311
312
- RRDSET *st = host->rrdpush.receive.pluginsd_chart_slots.array[slot - 1];
312
+ RRDSET *st = host->stream.rcv.pluginsd_chart_slots.array[slot - 1];
313
314
if(!st) {
315
st = pluginsd_find_chart(host, id, keyword);
src/plugins.d/pluginsd_parser.c
+46
-26
@@ -190,13 +190,13 @@ static inline PARSER_RC pluginsd_host_define_end(char **words __maybe_unused, si
190
default_rrd_history_entries,
191
default_rrd_memory_mode,
192
health_plugin_enabled(),
193
- stream_conf_send_enabled,
194
- stream_conf_send_destination,
195
- stream_conf_send_api_key,
196
- stream_conf_send_charts_matching,
197
- stream_conf_replication_enabled,
198
- stream_conf_replication_period,
199
- stream_conf_replication_step,
193
+ stream_send.enabled,
194
+ stream_send.parents.destination,
195
+ stream_send.api_key,
196
+ stream_send.send_charts_matching,
197
+ stream_receive.replication.enabled,
198
+ stream_receive.replication.period,
199
+ stream_receive.replication.step,
200
rrdhost_labels_to_system_info(parser->user.host_define.rrdlabels),
201
false);
202
@@ -795,8 +795,8 @@ static inline PARSER_RC pluginsd_begin_v2(char **words, size_t num_words, PARSER
795
// ------------------------------------------------------------------------
796
// propagate it forward in v2
797
798
- if(!parser->user.v2.stream_buffer.wb && rrdhost_has_rrdpush_sender_enabled(st->rrdhost))
799
- parser->user.v2.stream_buffer = rrdset_push_metric_initialize(parser->user.st, wall_clock_time);
798
+ if(!parser->user.v2.stream_buffer.wb && rrdhost_has_stream_sender_enabled(st->rrdhost))
799
+ parser->user.v2.stream_buffer = stream_send_metrics_init(parser->user.st, wall_clock_time);
800
801
if(parser->user.v2.stream_buffer.v2 && parser->user.v2.stream_buffer.wb) {
802
// check receiver capabilities
@@ -817,7 +817,7 @@ static inline PARSER_RC pluginsd_begin_v2(char **words, size_t num_words, PARSER
817
818
if(with_slots) {
819
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
820
- buffer_print_uint64_encoded(wb, integer_encoding, st->rrdpush.sender.chart_slot);
820
+ buffer_print_uint64_encoded(wb, integer_encoding, st->stream.snd.chart_slot);
821
}
822
823
buffer_fast_strcat(wb, " '", 2);
@@ -922,20 +922,33 @@ static inline PARSER_RC pluginsd_set_v2(char **words, size_t num_words, PARSER *
922
// ------------------------------------------------------------------------
923
// check value and ML
924
925
- if (unlikely(!netdata_double_isnumber(value) || (flags == SN_EMPTY_SLOT))) {
926
- value = NAN;
927
- flags = SN_EMPTY_SLOT;
925
+ if(stream_has_capability(&parser->user, STREAM_CAP_ML_MODELS)) {
926
+ // we receive anomaly information, no need for prediction on this node
927
+ if (unlikely(!netdata_double_isnumber(value) || (flags == SN_EMPTY_SLOT))) {
928
+ value = NAN;
929
+ flags = SN_EMPTY_SLOT;
930
+ }
931
932
if(parser->user.v2.ml_locked)
930
- ml_dimension_is_anomalous(rd, parser->user.v2.end_time, 0, false);
933
+ ml_dimension_received_anomaly(rd, !(flags & SN_FLAG_NOT_ANOMALOUS));
934
}
932
- else if(parser->user.v2.ml_locked) {
933
- if (ml_dimension_is_anomalous(rd, parser->user.v2.end_time, value, true)) {
934
- // clear anomaly bit: 0 -> is anomalous, 1 -> not anomalous
935
- flags &= ~((storage_number) SN_FLAG_NOT_ANOMALOUS);
935
+ else {
936
+ // we don't receive anomaly information, we need to run prediction on this node
937
+ if (unlikely(!netdata_double_isnumber(value) || (flags == SN_EMPTY_SLOT))) {
938
+ value = NAN;
939
+ flags = SN_EMPTY_SLOT;
940
+
941
+ if(parser->user.v2.ml_locked)
942
+ ml_dimension_is_anomalous(rd, parser->user.v2.end_time, 0, false);
943
+ }
944
+ else if(parser->user.v2.ml_locked) {
945
+ if (ml_dimension_is_anomalous(rd, parser->user.v2.end_time, value, true)) {
946
+ // clear anomaly bit: 0 -> is anomalous, 1 -> not anomalous
947
+ flags &= ~((storage_number) SN_FLAG_NOT_ANOMALOUS);
948
+ }
949
+ else
950
+ flags |= SN_FLAG_NOT_ANOMALOUS;
951
}
937
- else
938
- flags |= SN_FLAG_NOT_ANOMALOUS;
952
}
953
954
timing_step(TIMING_STEP_SET2_ML);
@@ -958,7 +971,7 @@ static inline PARSER_RC pluginsd_set_v2(char **words, size_t num_words, PARSER *
971
972
if(with_slots) {
973
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
961
- buffer_print_uint64_encoded(wb, integer_encoding, rd->rrdpush.sender.dim_slot);
974
+ buffer_print_uint64_encoded(wb, integer_encoding, rd->stream.snd.dim_slot);
975
}
976
977
buffer_fast_strcat(wb, " '", 2);
@@ -1014,7 +1027,7 @@ static inline PARSER_RC pluginsd_end_v2(char **words __maybe_unused, size_t num_
1027
// propagate the whole chart update in v1
1028
1029
if(unlikely(!parser->user.v2.stream_buffer.v2 && !parser->user.v2.stream_buffer.begin_v2_added && parser->user.v2.stream_buffer.wb))
1017
- rrdset_push_metrics_v1(&parser->user.v2.stream_buffer, st);
1030
+ stream_send_rrdset_metrics_v1(&parser->user.v2.stream_buffer, st);
1031
1032
timing_step(TIMING_STEP_END2_PUSH_V1);
1033
@@ -1030,7 +1043,7 @@ static inline PARSER_RC pluginsd_end_v2(char **words __maybe_unused, size_t num_
1043
// ------------------------------------------------------------------------
1044
// propagate it forward
1045
1033
- rrdset_push_metrics_finished(&parser->user.v2.stream_buffer, st);
1046
+ stream_send_rrdset_metrics_finished(&parser->user.v2.stream_buffer, st);
1047
1048
timing_step(TIMING_STEP_END2_PROPAGATE);
1049
@@ -1080,6 +1093,11 @@ static void pluginsd_json_stream_paths(PARSER *parser, void *action_data __maybe
1093
buffer_free(parser->defer.response);
1094
}
1095
1096
+static void pluginsd_json_ml_model(PARSER *parser, void *action_data __maybe_unused) {
1097
+ ml_model_received_from_child(parser->user.host, buffer_tostring(parser->defer.response));
1098
+ buffer_free(parser->defer.response);
1099
+}
1100
+
1101
static void pluginsd_json_dev_null(PARSER *parser, void *action_data __maybe_unused) {
1102
buffer_free(parser->defer.response);
1103
}
@@ -1096,15 +1114,17 @@ static PARSER_RC pluginsd_json(char **words __maybe_unused, size_t num_words __m
1114
parser->defer.action_data = NULL;
1115
parser->flags |= PARSER_DEFER_UNTIL_KEYWORD;
1116
1099
- if(strcmp(keyword, PLUGINSD_KEYWORD_STREAM_PATH) == 0)
1117
+ if(strcmp(keyword, PLUGINSD_KEYWORD_JSON_CMD_STREAM_PATH) == 0)
1118
parser->defer.action = pluginsd_json_stream_paths;
1119
+ else if(strcmp(keyword, PLUGINSD_KEYWORD_JSON_CMD_ML_MODEL) == 0)
1120
+ parser->defer.action = pluginsd_json_ml_model;
1121
else
1122
netdata_log_error("PLUGINSD: invalid JSON payload keyword '%s'", keyword);
1123
1124
return PARSER_RC_OK;
1125
}
1126
1107
-PARSER_RC rrdpush_receiver_pluginsd_claimed_id(char **words, size_t num_words, PARSER *parser);
1127
+PARSER_RC stream_receiver_pluginsd_claimed_id(char **words, size_t num_words, PARSER *parser);
1128
1129
// ----------------------------------------------------------------------------
1130
@@ -1290,7 +1310,7 @@ PARSER_RC parser_execute(PARSER *parser, const PARSER_KEYWORD *keyword, char **w
1310
case PLUGINSD_KEYWORD_ID_VARIABLE:
1311
return pluginsd_variable(words, num_words, parser);
1312
case PLUGINSD_KEYWORD_ID_CLAIMED_ID:
1293
- return rrdpush_receiver_pluginsd_claimed_id(words, num_words, parser);
1313
+ return stream_receiver_pluginsd_claimed_id(words, num_words, parser);
1314
case PLUGINSD_KEYWORD_ID_HOST:
1315
return pluginsd_host(words, num_words, parser);
1316
case PLUGINSD_KEYWORD_ID_HOST_DEFINE:
src/plugins.d/pluginsd_parser.h
+5
-6
@@ -5,10 +5,10 @@
5
6
#include "daemon/common.h"
7
8
-#define WORKER_PARSER_FIRST_JOB 3
8
+#define WORKER_PARSER_FIRST_JOB 34
9
10
-// this has to be in-sync with the same at receiver.c
11
-#define WORKER_RECEIVER_JOB_REPLICATION_COMPLETION (WORKER_PARSER_FIRST_JOB - 3)
10
+// this has to be in-sync with the same at stream-thread.c
11
+#define WORKER_RECEIVER_JOB_REPLICATION_COMPLETION (WORKER_PARSER_FIRST_JOB - 9)
12
13
// this controls the max response size of a function
14
#define PLUGINSD_MAX_DEFERRED_SIZE (100 * 1024 * 1024)
@@ -101,8 +101,7 @@ struct parser {
101
uint32_t flags;
102
int fd_input;
103
int fd_output;
104
-
105
- NETDATA_SSL *ssl_output;
104
+ ND_SOCK *sock;
105
106
#ifdef ENABLE_H2O
107
void *h2o_ctx; // if set we use h2o_stream functions to send data
@@ -133,7 +132,7 @@ struct parser {
132
133
typedef struct parser PARSER;
134
136
-PARSER *parser_init(struct parser_user_object *user, int fd_input, int fd_output, PARSER_INPUT_TYPE flags, void *ssl);
135
+PARSER *parser_init(struct parser_user_object *user, int fd_input, int fd_output, PARSER_INPUT_TYPE flags, ND_SOCK *sock);
136
void parser_init_repertoire(PARSER *parser, PARSER_REPERTOIRE repertoire);
137
void parser_destroy(PARSER *working_parser);
138
void pluginsd_cleanup_v2(PARSER *parser);
src/plugins.d/pluginsd_replication.c
+5
-4
@@ -1,6 +1,7 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "pluginsd_replication.h"
4
+#include "streaming/stream-receiver-internals.h"
5
6
PARSER_RC pluginsd_replay_begin(char **words, size_t num_words, PARSER *parser) {
7
int idx = 1;
@@ -311,9 +312,9 @@ PARSER_RC pluginsd_replay_end(char **words, size_t num_words, PARSER *parser) {
312
time_t current = parser->user.replay.end_time;
313
314
if(started && current > started) {
314
- host->rrdpush_receiver_replication_percent = (NETDATA_DOUBLE) (current - started) * 100.0 / (NETDATA_DOUBLE) (now - started);
315
+ host->stream.rcv.status.replication.percent = (NETDATA_DOUBLE) (current - started) * 100.0 / (NETDATA_DOUBLE) (now - started);
316
worker_set_metric(WORKER_RECEIVER_JOB_REPLICATION_COMPLETION,
316
- host->rrdpush_receiver_replication_percent);
317
+ host->stream.rcv.status.replication.percent);
318
}
319
}
320
@@ -354,8 +355,8 @@ PARSER_RC pluginsd_replay_end(char **words, size_t num_words, PARSER *parser) {
355
356
pluginsd_clear_scope_chart(parser, PLUGINSD_KEYWORD_REPLAY_END);
357
357
- host->rrdpush_receiver_replication_percent = 100.0;
358
- worker_set_metric(WORKER_RECEIVER_JOB_REPLICATION_COMPLETION, host->rrdpush_receiver_replication_percent);
358
+ host->stream.rcv.status.replication.percent = 100.0;
359
+ worker_set_metric(WORKER_RECEIVER_JOB_REPLICATION_COMPLETION, host->stream.rcv.status.replication.percent);
360
361
return PARSER_RC_OK;
362
}
src/streaming/PARENT-CLUSTERS.md
new
+143
@@ -0,0 +1,143 @@
1
+# Notes on Netdata Active-Active Parent Clusters
2
+
3
+#### **Streaming Connection Overview**
4
+
5
+Each Netdata child node specifies its parent nodes through the
6
+`[stream].destination` configuration in `stream.conf`. While a child can list
7
+multiple parents, it will connect to only one at a time. If the connection to
8
+the first parent fails, the child will try the next parent in the list,
9
+continuing this process until a successful connection is established. If no
10
+connection can be made, the child will retry the list in order.
11
+
12
+Once a Netdata parent receives data from its child nodes, it can also act as a
13
+child to another parent (or "grandparent") to propagate the data further up the
14
+hierarchy.
15
+
16
+#### **Active-Active Parent Clusters Overview**
17
+
18
+Active-active parent clusters involve circular data propagation among parent
19
+nodes. For example, parent A streams its data to parent B (its grandparent),
20
+while parent B streams back to parent A, creating redundancy. This configuration
21
+ensures that each parent node has the same data, allowing child nodes to connect
22
+to any available parent.
23
+
24
+This setup can be expanded to more than two parents by configuring all parents
25
+as grandparents of each other.
26
+
27
+---
28
+
29
+### **Data Replication**
30
+
31
+When a child node connects to a parent, it enters a negotiation phase to
32
+announce the metrics it will stream, including their retention period. The
33
+parent checks its database for missing data. If data gaps exist, the parent
34
+requests replication of the missing metrics from the child before transitioning
35
+to streaming fresh data.
36
+
37
+Replication occurs at the instance (metric group) level, meaning some metrics
38
+may replicate historical data while others stream in real time. Only high-
39
+resolution (`tier0`) data is replicated since higher-tier data can be derived
40
+from `tier0`. Therefore, maintaining sufficient `tier0` retention on the child
41
+is crucial to prevent gaps in the parent’s database.
42
+
43
+---
44
+
45
+### **Challenges in Active-Active Clusters**
46
+
47
+#### **Adding a New Parent**
48
+
49
+Introducing a new parent to an active-active cluster involves two major
50
+challenges:
51
+
52
+1. **Replicating Existing Data**
53
+ Since Netdata replication only propagates currently collected metrics,
54
+ archived data such as metrics from stopped containers or disconnected devices
55
+ will not be replicated. To ensure the new parent has complete historical
56
+ data:
57
+ - Copy the existing database from another parent (`/var/cache/netdata`) using
58
+ tools like `rsync` for dbengine files (safe for hot-copy) and `sqlite3` for
59
+ SQLite databases.
60
+ - Perform multiple copies and start the new parent promptly to minimize the
61
+ data gap.
62
+
63
+2. **Preventing Premature Connections**
64
+ Child nodes should not connect to the new parent until it has completed data
65
+ replication. Premature connections could lead to data gaps, as the child may
66
+ lack the necessary historical data.
67
+
68
+ In Netdata v2.1+, a balancing feature allows children to query parent
69
+ retention and prioritize connections to parents with the most recent data.
70
+ However, children will still connect to the first available parent,
71
+ potentially introducing gaps to the new parent's database.
72
+ **Solution:** Keep the new parent isolated from children until its
73
+ replication process is complete. Only then should children be configured to
74
+ include the new parent.
75
+
76
+---
77
+
78
+### **Resource Management in Clusters**
79
+
80
+Resource usage on parent nodes depends on three key factors:
81
+
82
+1. **Ingestion Rate**
83
+ All parent nodes ingest all data of all children (not just their own). The
84
+ resource load is the same across all parents.
85
+
86
+2. **Machine Learning**
87
+ Machine learning is CPU-intensive and affects memory usage.
88
+ - **Before Netdata 2.1:** Every node in a cluster independently trained
89
+ machine learning models for all children, increasing resource consumption
90
+ exponentially.
91
+ - **Netdata 2.1+:** The first node (child or parent) to train ML models
92
+ propagates the trained data to other nodes, significantly reducing resource
93
+ requirements. This allows flexibility: machine learning can either run at
94
+ the edge (child nodes) or on the first parent receiving the data.
95
+
96
+3. **Re-Streaming Rate**
97
+ Propagating data to other parents consumes CPU and bandwidth for formatting,
98
+ compressing, and transmitting data. Each parent except the last grandparent
99
+ in the chain contributes to this workload.
100
+
101
+---
102
+
103
+### **Parent Balancing**
104
+
105
+Netdata v2.1 introduces a balancing algorithm for child nodes to optimize parent
106
+connections. This feature is designed to ensure that child nodes connect to the
107
+most suitable parent, balancing the load across the cluster and reducing the
108
+likelihood of data gaps or resource bottlenecks.
109
+
110
+#### **Initial Balancing**
111
+
112
+Before establishing a connection, each child node queries its candidate parents
113
+to retrieve their retention details. Based on this information, the child
114
+evaluates the parents and prioritizes those with the most recent data.
115
+
116
+- **Retention Difference Threshold**:
117
+ Parents are considered equivalent if their retention times differ by less than
118
+ two minutes. In such cases, the child selects a parent randomly to avoid
119
+ overloading a single node. This randomness ensures an even distribution of
120
+ connections when all candidate parents are equally suitable.
121
+
122
+- **Disconnection Handling**:
123
+ To prevent overloading a parent during network disruptions, children
124
+ temporarily block a recently disconnected parent for a randomized duration
125
+ before attempting to reconnect. This cooldown period reduces the risk of
126
+ repeated disconnections and ensures smoother reconnections.
127
+
128
+#### **Re-balancing After Cluster Changes**
129
+
130
+Currently, the only way to re-balance the cluster (i.e. to break the existing
131
+connections so that the children nodes will connect to both parents), is to
132
+restart the children.
133
+
134
+---
135
+
136
+### **Summary**
137
+
138
+Active-active parent clusters in Netdata provide robust data redundancy and
139
+flexibility. Properly configuring replication, balancing resources, and managing
140
+parent-child connections ensures optimal performance and data integrity. With
141
+improvements in v2.1, including machine learning propagation and connection
142
+balancing, Netdata clusters are more efficient and scalable, catering to complex
143
+and dynamic environments.
src/streaming/README.md
+2
-2
@@ -59,7 +59,7 @@ This section defines an API key for other Agents to connect to this Netdata.
59
| [`allow from`](#allow-from) | `*` | A space-separated list of [Netdata simple patterns](/src/libnetdata/simple_pattern/README.md) matching the IPs of nodes that will stream metrics using this API key. [Read more →](#allow-from) |
60
| `retention` | `1h` | The default amount of child metrics history to retain when using the `ram` db. |
61
| [`db`](#default-memory-mode) | `dbengine` | The [database](/src/database/README.md) to use for all nodes using this `API_KEY`. Valid settings are `dbengine`, `ram`, or `none`. [Read more →](#default-memory-mode) |
62
-| `health enabled by default` | `auto` | Whether alerts and notifications should be enabled for nodes using this `API_KEY`. `auto` enables alerts when the child is connected. `yes` enables alerts always, and `no` disables alerts. |
62
+| `health enabled` | `auto` | Whether alerts and notifications should be enabled for nodes using this `API_KEY`. `auto` enables alerts when the child is connected. `yes` enables alerts always, and `no` disables alerts. |
63
| `postpone alerts on connect` | `1m` | Postpone alerts and notifications for a period of time after the child connects. |
64
| `health log retention` | `5d` | History of health log events (in seconds) kept in the database. |
65
| `proxy enabled` | | Route metrics through a proxy. |
@@ -461,7 +461,7 @@ On the parent, set the following in `stream.conf`:
461
default memory = ram
462
463
# alerts checks, only while the child is connected
464
- health enabled by default = auto
464
+ health enabled = auto
465
```
466
467
On the child nodes, set the following in `stream.conf`:
src/streaming/protocol/command-begin-set-end.c
+13
-11
@@ -1,14 +1,16 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "commands.h"
4
+#include "../stream-sender-internals.h"
5
#include "plugins.d/pluginsd_internals.h"
6
6
-static void rrdpush_send_chart_metrics(BUFFER *wb, RRDSET *st, struct sender_state *s __maybe_unused, RRDSET_FLAGS flags) {
7
- buffer_fast_strcat(wb, "BEGIN \"", 7);
7
+static void
8
+stream_send_rrdset_metrics_v1_internal(BUFFER *wb, RRDSET *st, struct sender_state *s __maybe_unused, RRDSET_FLAGS flags) {
9
+ buffer_fast_strcat(wb, PLUGINSD_KEYWORD_BEGIN " \"", 7);
10
buffer_fast_strcat(wb, rrdset_id(st), string_strlen(st->id));
11
buffer_fast_strcat(wb, "\" ", 2);
12
11
- if(st->last_collected_time.tv_sec > st->rrdpush.sender.resync_time_s)
13
+ if(st->last_collected_time.tv_sec > st->stream.snd.resync_time_s)
14
buffer_print_uint64(wb, st->usec_since_last_update);
15
else
16
buffer_fast_strcat(wb, "0", 1);
@@ -21,7 +23,7 @@ static void rrdpush_send_chart_metrics(BUFFER *wb, RRDSET *st, struct sender_sta
23
continue;
24
25
if(likely(rrddim_check_upstream_exposed_collector(rd))) {
24
- buffer_fast_strcat(wb, "SET \"", 5);
26
+ buffer_fast_strcat(wb, PLUGINSD_KEYWORD_SET " \"", 5);
27
buffer_fast_strcat(wb, rrddim_id(rd), string_strlen(rd->id));
28
buffer_fast_strcat(wb, "\" = ", 4);
29
buffer_print_int64(wb, rd->collector.collected_value);
@@ -39,15 +41,15 @@ static void rrdpush_send_chart_metrics(BUFFER *wb, RRDSET *st, struct sender_sta
41
if(unlikely(flags & RRDSET_FLAG_UPSTREAM_SEND_VARIABLES))
42
rrdvar_print_to_streaming_custom_chart_variables(st, wb);
43
42
- buffer_fast_strcat(wb, "END\n", 4);
44
+ buffer_fast_strcat(wb, PLUGINSD_KEYWORD_END "\n", 4);
45
}
46
45
-void rrdset_push_metrics_v1(RRDSET_STREAM_BUFFER *rsb, RRDSET *st) {
47
+void stream_send_rrdset_metrics_v1(RRDSET_STREAM_BUFFER *rsb, RRDSET *st) {
48
RRDHOST *host = st->rrdhost;
47
- rrdpush_send_chart_metrics(rsb->wb, st, host->sender, rsb->rrdset_flags);
49
+ stream_send_rrdset_metrics_v1_internal(rsb->wb, st, host->sender, rsb->rrdset_flags);
50
}
51
50
-void rrddim_push_metrics_v2(RRDSET_STREAM_BUFFER *rsb, RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n, SN_FLAGS flags) {
52
+void stream_send_rrddim_metrics_v2(RRDSET_STREAM_BUFFER *rsb, RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n, SN_FLAGS flags) {
53
if(!rsb->wb || !rsb->v2 || !netdata_double_isnumber(n) || !does_storage_number_exist(flags))
54
return;
55
@@ -65,7 +67,7 @@ void rrddim_push_metrics_v2(RRDSET_STREAM_BUFFER *rsb, RRDDIM *rd, usec_t point_
67
68
if(with_slots) {
69
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
68
- buffer_print_uint64_encoded(wb, integer_encoding, rd->rrdset->rrdpush.sender.chart_slot);
70
+ buffer_print_uint64_encoded(wb, integer_encoding, rd->rrdset->stream.snd.chart_slot);
71
}
72
73
buffer_fast_strcat(wb, " '", 2);
@@ -89,7 +91,7 @@ void rrddim_push_metrics_v2(RRDSET_STREAM_BUFFER *rsb, RRDDIM *rd, usec_t point_
91
92
if(with_slots) {
93
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
92
- buffer_print_uint64_encoded(wb, integer_encoding, rd->rrdpush.sender.dim_slot);
94
+ buffer_print_uint64_encoded(wb, integer_encoding, rd->stream.snd.dim_slot);
95
}
96
97
buffer_fast_strcat(wb, " '", 2);
@@ -108,7 +110,7 @@ void rrddim_push_metrics_v2(RRDSET_STREAM_BUFFER *rsb, RRDDIM *rd, usec_t point_
110
buffer_fast_strcat(wb, "\n", 1);
111
}
112
111
-void rrdset_push_metrics_finished(RRDSET_STREAM_BUFFER *rsb, RRDSET *st) {
113
+void stream_send_rrdset_metrics_finished(RRDSET_STREAM_BUFFER *rsb, RRDSET *st) {
114
if(!rsb->wb)
115
return;
116
src/streaming/protocol/command-chart-definition.c
+26
-30
@@ -1,25 +1,26 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "commands.h"
4
+#include "../stream-sender-internals.h"
5
#include "plugins.d/pluginsd_internals.h"
6
7
// chart labels
7
-static int send_clabels_callback(const char *name, const char *value, RRDLABEL_SRC ls, void *data) {
8
+static int stream_send_clabels_callback(const char *name, const char *value, RRDLABEL_SRC ls, void *data) {
9
BUFFER *wb = (BUFFER *)data;
10
buffer_sprintf(wb, PLUGINSD_KEYWORD_CLABEL " \"%s\" \"%s\" %d\n", name, value, ls & ~(RRDLABEL_FLAG_INTERNAL));
11
return 1;
12
}
13
13
-static void rrdpush_send_clabels(BUFFER *wb, RRDSET *st) {
14
+static void stream_send_clabels(BUFFER *wb, RRDSET *st) {
15
if (st->rrdlabels) {
15
- if(rrdlabels_walkthrough_read(st->rrdlabels, send_clabels_callback, wb) > 0)
16
+ if(rrdlabels_walkthrough_read(st->rrdlabels, stream_send_clabels_callback, wb) > 0)
17
buffer_sprintf(wb, PLUGINSD_KEYWORD_CLABEL_COMMIT "\n");
18
}
19
}
20
21
// Send the current chart definition.
22
// Assumes that collector thread has already called sender_start for mutex / buffer state.
22
-bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
23
+bool stream_sender_send_rrdset_definition(BUFFER *wb, RRDSET *st) {
24
uint32_t version = rrdset_metadata_version(st);
25
26
RRDHOST *host = st->rrdhost;
@@ -45,7 +46,7 @@ bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
46
47
if(with_slots) {
48
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
48
- buffer_print_uint64_encoded(wb, integer_encoding, st->rrdpush.sender.chart_slot);
49
+ buffer_print_uint64_encoded(wb, integer_encoding, st->stream.snd.chart_slot);
50
}
51
52
// send the chart
@@ -53,24 +54,24 @@ bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
54
wb
55
, " \"%s\" \"%s\" \"%s\" \"%s\" \"%s\" \"%s\" \"%s\" %d %d \"%s %s %s\" \"%s\" \"%s\"\n"
56
, rrdset_id(st)
56
- , name
57
+ , name
58
, rrdset_title(st)
58
- , rrdset_units(st)
59
- , rrdset_family(st)
60
- , rrdset_context(st)
61
- , rrdset_type_name(st->chart_type)
62
- , st->priority
59
+ , rrdset_units(st)
60
+ , rrdset_family(st)
61
+ , rrdset_context(st)
62
+ , rrdset_type_name(st->chart_type)
63
+ , st->priority
64
, st->update_every
65
, rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)?"obsolete":""
66
, rrdset_flag_check(st, RRDSET_FLAG_STORE_FIRST)?"store_first":""
67
, rrdset_flag_check(st, RRDSET_FLAG_HIDDEN)?"hidden":""
68
, rrdset_plugin_name(st)
68
- , rrdset_module_name(st)
69
+ , rrdset_module_name(st)
70
);
71
72
// send the chart labels
73
if (stream_has_capability(host->sender, STREAM_CAP_CLABELS))
73
- rrdpush_send_clabels(wb, st);
74
+ stream_send_clabels(wb, st);
75
76
// send the dimensions
77
RRDDIM *rd;
@@ -79,7 +80,7 @@ bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
80
81
if(with_slots) {
82
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
82
- buffer_print_uint64_encoded(wb, integer_encoding, rd->rrdpush.sender.dim_slot);
83
+ buffer_print_uint64_encoded(wb, integer_encoding, rd->stream.snd.dim_slot);
84
}
85
86
buffer_sprintf(
@@ -99,7 +100,7 @@ bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
100
101
// send the chart functions
102
if(stream_has_capability(host->sender, STREAM_CAP_FUNCTIONS))
102
- rrd_chart_functions_expose_rrdpush(st, wb);
103
+ stream_sender_send_rrdset_functions(st, wb);
104
105
// send the chart local custom variables
106
rrdvar_print_to_streaming_custom_chart_variables(st, wb);
@@ -128,8 +129,6 @@ bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
129
#endif
130
}
131
131
- sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
132
-
132
// we can set the exposed flag, after we commit the buffer
133
// because replication may pick it up prematurely
134
rrddim_foreach_read(rd, st) {
@@ -138,11 +137,11 @@ bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
137
rrddim_foreach_done(rd);
138
rrdset_metadata_exposed_upstream(st, version);
139
141
- st->rrdpush.sender.resync_time_s = st->last_collected_time.tv_sec + (stream_conf_initial_clock_resync_iterations * st->update_every);
140
+ st->stream.snd.resync_time_s = st->last_collected_time.tv_sec + (stream_send.initial_clock_resync_iterations * st->update_every);
141
return replication_progress;
142
}
143
145
-bool should_send_chart_matching(RRDSET *st, RRDSET_FLAGS flags) {
144
+bool should_send_rrdset_matching(RRDSET *st, RRDSET_FLAGS flags) {
145
if(!(flags & RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED))
146
return false;
147
@@ -159,18 +158,18 @@ bool should_send_chart_matching(RRDSET *st, RRDSET_FLAGS flags) {
158
int negative = 0, positive = 0;
159
SIMPLE_PATTERN_RESULT r;
160
162
- r = simple_pattern_matches_string_extract(host->rrdpush.send.charts_matching, st->context, NULL, 0);
161
+ r = simple_pattern_matches_string_extract(host->stream.snd.charts_matching, st->context, NULL, 0);
162
if(r == SP_MATCHED_POSITIVE) positive++;
163
else if(r == SP_MATCHED_NEGATIVE) negative++;
164
165
if(!negative) {
167
- r = simple_pattern_matches_string_extract(host->rrdpush.send.charts_matching, st->name, NULL, 0);
166
+ r = simple_pattern_matches_string_extract(host->stream.snd.charts_matching, st->name, NULL, 0);
167
if (r == SP_MATCHED_POSITIVE) positive++;
168
else if (r == SP_MATCHED_NEGATIVE) negative++;
169
}
170
171
if(!negative) {
173
- r = simple_pattern_matches_string_extract(host->rrdpush.send.charts_matching, st->id, NULL, 0);
172
+ r = simple_pattern_matches_string_extract(host->stream.snd.charts_matching, st->id, NULL, 0);
173
if (r == SP_MATCHED_POSITIVE) positive++;
174
else if (r == SP_MATCHED_NEGATIVE) negative++;
175
}
@@ -189,18 +188,15 @@ bool should_send_chart_matching(RRDSET *st, RRDSET_FLAGS flags) {
188
}
189
190
// Called from the internal collectors to mark a chart obsolete.
192
-bool rrdset_push_chart_definition_now(RRDSET *st) {
191
+bool stream_sender_send_rrdset_definition_now(RRDSET *st) {
192
RRDHOST *host = st->rrdhost;
193
195
- if(unlikely(!rrdhost_can_send_definitions_to_parent(host)
196
- || !should_send_chart_matching(st, rrdset_flag_get(st)))) {
194
+ if(unlikely(!rrdhost_can_stream_metadata_to_parent(host) || !should_send_rrdset_matching(st, rrdset_flag_get(st))))
195
return false;
198
- }
196
200
- BUFFER *wb = sender_start(host->sender);
201
- rrdpush_send_chart_definition(wb, st);
202
- sender_thread_buffer_free();
197
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
198
+ stream_sender_send_rrdset_definition(wb, st);
199
+ sender_commit_clean_buffer(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
200
201
return true;
202
}
206
-
src/streaming/protocol/command-claimed_id.c
+7
-9
@@ -3,7 +3,7 @@
3
#include "commands.h"
4
#include "plugins.d/pluginsd_internals.h"
5
6
-PARSER_RC rrdpush_receiver_pluginsd_claimed_id(char **words, size_t num_words, PARSER *parser) {
6
+PARSER_RC stream_receiver_pluginsd_claimed_id(char **words, size_t num_words, PARSER *parser) {
7
const char *machine_guid_str = get_word(words, num_words, 1);
8
const char *claim_id_str = get_word(words, num_words, 2);
9
@@ -47,20 +47,20 @@ PARSER_RC rrdpush_receiver_pluginsd_claimed_id(char **words, size_t num_words, P
47
48
if(!uuid_is_null(claim_uuid)) {
49
uuid_copy(host->aclk.claim_id_of_origin.uuid, claim_uuid);
50
- rrdpush_sender_send_claimed_id(host);
50
+ stream_sender_send_claimed_id(host);
51
}
52
53
return PARSER_RC_OK;
54
}
55
56
-void rrdpush_sender_send_claimed_id(RRDHOST *host) {
57
- if(!stream_has_capability(host->sender, STREAM_CAP_CLAIM))
56
+void stream_sender_send_claimed_id(RRDHOST *host) {
57
+ if(!stream_sender_has_capabilities(host, STREAM_CAP_CLAIM))
58
return;
59
60
- if(unlikely(!rrdhost_can_send_definitions_to_parent(host)))
60
+ if(unlikely(!rrdhost_can_stream_metadata_to_parent(host)))
61
return;
62
63
- BUFFER *wb = sender_start(host->sender);
63
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
64
65
char str[UUID_STR_LEN] = "";
66
ND_UUID uuid = host->aclk.claim_id_of_origin;
@@ -72,7 +72,5 @@ void rrdpush_sender_send_claimed_id(RRDHOST *host) {
72
buffer_sprintf(wb, PLUGINSD_KEYWORD_CLAIMED_ID " '%s' '%s'\n",
73
host->machine_guid, str);
74
75
- sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
76
-
77
- sender_thread_buffer_free();
75
+ sender_commit_clean_buffer(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
76
}
src/streaming/protocol/command-function.c
+8
-7
@@ -1,20 +1,21 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "commands.h"
4
+#include "../stream-sender-internals.h"
5
#include "plugins.d/pluginsd_internals.h"
6
6
-void rrdpush_send_global_functions(RRDHOST *host) {
7
+void stream_send_global_functions(RRDHOST *host) {
8
if(!stream_has_capability(host->sender, STREAM_CAP_FUNCTIONS))
9
return;
10
10
- if(unlikely(!rrdhost_can_send_definitions_to_parent(host)))
11
+ if(unlikely(!rrdhost_can_stream_metadata_to_parent(host)))
12
return;
13
13
- BUFFER *wb = sender_start(host->sender);
14
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
15
15
- rrd_global_functions_expose_rrdpush(host, wb, stream_has_capability(host->sender, STREAM_CAP_DYNCFG));
16
+ stream_sender_send_global_rrdhost_functions(host, wb, stream_has_capability(host->sender, STREAM_CAP_DYNCFG));
17
17
- sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_FUNCTIONS);
18
-
19
- sender_thread_buffer_free();
18
+ // send it as STREAM_TRAFFIC_TYPE_METADATA, not STREAM_TRAFFIC_TYPE_FUNCTIONS
19
+ // this is just metadata not an interactive function call
20
+ sender_commit_clean_buffer(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
21
}
src/streaming/protocol/command-host-labels.c
+9
-10
@@ -1,25 +1,24 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "commands.h"
4
+#include "../stream-sender-internals.h"
5
#include "plugins.d/pluginsd_internals.h"
6
6
-static int send_labels_callback(const char *name, const char *value, RRDLABEL_SRC ls, void *data) {
7
+static int send_host_labels_callback(const char *name, const char *value, RRDLABEL_SRC ls, void *data) {
8
BUFFER *wb = (BUFFER *)data;
8
- buffer_sprintf(wb, "LABEL \"%s\" = %d \"%s\"\n", name, ls, value);
9
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_LABEL " \"%s\" = %u \"%s\"\n", name, (unsigned)ls, value);
10
return 1;
11
}
12
12
-void rrdpush_send_host_labels(RRDHOST *host) {
13
- if(unlikely(!rrdhost_can_send_definitions_to_parent(host)
13
+void stream_send_host_labels(RRDHOST *host) {
14
+ if(unlikely(!rrdhost_can_stream_metadata_to_parent(host)
15
|| !stream_has_capability(host->sender, STREAM_CAP_HLABELS)))
16
return;
17
17
- BUFFER *wb = sender_start(host->sender);
18
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
19
19
- rrdlabels_walkthrough_read(host->rrdlabels, send_labels_callback, wb);
20
- buffer_sprintf(wb, "OVERWRITE %s\n", "labels");
20
+ rrdlabels_walkthrough_read(host->rrdlabels, send_host_labels_callback, wb);
21
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_OVERWRITE " %s\n", "labels");
22
22
- sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
23
-
24
- sender_thread_buffer_free();
23
+ sender_commit_clean_buffer(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
24
}
src/streaming/protocol/command-host-variables.c
+15
-17
@@ -3,23 +3,22 @@
3
#include "commands.h"
4
#include "plugins.d/pluginsd_internals.h"
5
6
-static inline void rrdpush_sender_add_host_variable_to_buffer(BUFFER *wb, const RRDVAR_ACQUIRED *rva) {
6
+static inline void stream_sender_add_host_variable_to_buffer(BUFFER *wb, const RRDVAR_ACQUIRED *rva) {
7
buffer_sprintf(
8
wb
9
- , "VARIABLE HOST %s = " NETDATA_DOUBLE_FORMAT "\n"
9
+ , PLUGINSD_KEYWORD_VARIABLE " HOST %s = " NETDATA_DOUBLE_FORMAT "\n"
10
, rrdvar_name(rva)
11
- , rrdvar2number(rva)
11
+ , rrdvar2number(rva)
12
);
13
14
netdata_log_debug(D_STREAM, "RRDVAR pushed HOST VARIABLE %s = " NETDATA_DOUBLE_FORMAT, rrdvar_name(rva), rrdvar2number(rva));
15
}
16
17
-void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva) {
18
- if(rrdhost_can_send_definitions_to_parent(host)) {
19
- BUFFER *wb = sender_start(host->sender);
20
- rrdpush_sender_add_host_variable_to_buffer(wb, rva);
21
- sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
22
- sender_thread_buffer_free();
17
+void stream_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva) {
18
+ if(rrdhost_can_stream_metadata_to_parent(host)) {
19
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
20
+ stream_sender_add_host_variable_to_buffer(wb, rva);
21
+ sender_commit_clean_buffer(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
22
}
23
}
24
@@ -27,25 +26,24 @@ struct custom_host_variables_callback {
26
BUFFER *wb;
27
};
28
30
-static int rrdpush_sender_thread_custom_host_variables_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdvar_ptr __maybe_unused, void *struct_ptr) {
29
+static int stream_sender_thread_custom_host_variables_callback(const DICTIONARY_ITEM *item __maybe_unused, void *rrdvar_ptr __maybe_unused, void *struct_ptr) {
30
const RRDVAR_ACQUIRED *rv = (const RRDVAR_ACQUIRED *)item;
31
struct custom_host_variables_callback *tmp = struct_ptr;
32
BUFFER *wb = tmp->wb;
33
35
- rrdpush_sender_add_host_variable_to_buffer(wb, rv);
34
+ stream_sender_add_host_variable_to_buffer(wb, rv);
35
return 1;
36
}
37
39
-void rrdpush_sender_thread_send_custom_host_variables(RRDHOST *host) {
40
- if(rrdhost_can_send_definitions_to_parent(host)) {
41
- BUFFER *wb = sender_start(host->sender);
38
+void stream_sender_send_custom_host_variables(RRDHOST *host) {
39
+ if(rrdhost_can_stream_metadata_to_parent(host)) {
40
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
41
struct custom_host_variables_callback tmp = {
42
.wb = wb
43
};
45
- int ret = rrdvar_walkthrough_read(host->rrdvars, rrdpush_sender_thread_custom_host_variables_callback, &tmp);
44
+ int ret = rrdvar_walkthrough_read(host->rrdvars, stream_sender_thread_custom_host_variables_callback, &tmp);
45
(void)ret;
47
- sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
48
- sender_thread_buffer_free();
46
+ sender_commit_clean_buffer(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
47
48
netdata_log_debug(D_STREAM, "RRDVAR sent %d VARIABLES", ret);
49
}
src/streaming/protocol/command-nodeid.c
+15
-12
@@ -1,19 +1,21 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "commands.h"
4
+#include "../stream-receiver-internals.h"
5
+#include "../stream-sender-internals.h"
6
#include "plugins.d/pluginsd_internals.h"
7
8
// the child disconnected from the parent, and it has to clear the parent's claim id
7
-void rrdpush_sender_clear_parent_claim_id(RRDHOST *host) {
9
+void stream_sender_clear_parent_claim_id(RRDHOST *host) {
10
host->aclk.claim_id_of_parent = UUID_ZERO;
11
}
12
13
// the parent sends to the child its claim id, node id and cloud url
12
-void rrdpush_receiver_send_node_and_claim_id_to_child(RRDHOST *host) {
14
+void stream_receiver_send_node_and_claim_id_to_child(RRDHOST *host) {
15
if(host == localhost || UUIDiszero(host->node_id)) return;
16
15
- spinlock_lock(&host->receiver_lock);
16
- if(host->receiver && stream_has_capability(host->receiver, STREAM_CAP_NODE_ID)) {
17
+ rrdhost_receiver_lock(host);
18
+ if(stream_has_capability(host->receiver, STREAM_CAP_NODE_ID)) {
19
char node_id_str[UUID_STR_LEN] = "";
20
uuid_unparse_lower(host->node_id.uuid, node_id_str);
21
@@ -32,16 +34,16 @@ void rrdpush_receiver_send_node_and_claim_id_to_child(RRDHOST *host) {
34
PLUGINSD_KEYWORD_NODE_ID " '%s' '%s' '%s'\n",
35
claim_id.str, node_id_str, cloud_config_url_get());
36
35
- send_to_plugin(buf, __atomic_load_n(&host->receiver->parser, __ATOMIC_RELAXED));
37
+ send_to_plugin(buf, __atomic_load_n(&host->receiver->thread.parser, __ATOMIC_RELAXED));
38
}
37
- spinlock_unlock(&host->receiver_lock);
39
+ rrdhost_receiver_unlock(host);
40
}
41
42
// the sender of the child receives node id, claim id and cloud url from the receiver of the parent
41
-void rrdpush_sender_get_node_and_claim_id_from_parent(struct sender_state *s) {
42
- char *claim_id_str = get_word(s->line.words, s->line.num_words, 1);
43
- char *node_id_str = get_word(s->line.words, s->line.num_words, 2);
44
- char *url = get_word(s->line.words, s->line.num_words, 3);
43
+void stream_sender_get_node_and_claim_id_from_parent(struct sender_state *s) {
44
+ char *claim_id_str = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 1);
45
+ char *node_id_str = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 2);
46
+ char *url = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 3);
47
48
bool claimed = is_agent_claimed();
49
bool update_node_id = false;
@@ -118,10 +120,11 @@ void rrdpush_sender_get_node_and_claim_id_from_parent(struct sender_state *s) {
120
}
121
122
// we change the URL, to allow the agent dashboard to work with Netdata Cloud on-prem, if any.
121
- cloud_config_url_set(url);
123
+ if(node_id_updated)
124
+ cloud_config_url_set(url);
125
126
// send it down the line (to children)
124
- rrdpush_receiver_send_node_and_claim_id_to_child(s->host);
127
+ stream_receiver_send_node_and_claim_id_to_child(s->host);
128
129
if(node_id_updated)
130
stream_path_node_id_updated(s->host);
src/streaming/protocol/commands.c
+24
-15
@@ -1,35 +1,43 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "commands.h"
4
+#include "../stream-sender-internals.h"
5
5
-RRDSET_STREAM_BUFFER rrdset_push_metric_initialize(RRDSET *st, time_t wall_clock_time) {
6
+static BUFFER *preferred_sender_buffer(RRDHOST *host) {
7
+ if(host->stream.snd.commit.receiver_tid == gettid_cached())
8
+ return sender_host_buffer(host);
9
+ else
10
+ return sender_thread_buffer(localhost->sender);
11
+}
12
+
13
+RRDSET_STREAM_BUFFER stream_send_metrics_init(RRDSET *st, time_t wall_clock_time) {
14
RRDHOST *host = st->rrdhost;
15
16
// fetch the flags we need to check with one atomic operation
17
RRDHOST_FLAGS host_flags = __atomic_load_n(&host->flags, __ATOMIC_SEQ_CST);
18
19
// check if we are not connected
12
- if(unlikely(!(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS))) {
20
+ if(unlikely(!(host_flags & RRDHOST_FLAG_STREAM_SENDER_READY_4_METRICS))) {
21
14
- if(unlikely(!(host_flags & (RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN | RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED))))
15
- rrdpush_sender_thread_spawn(host);
22
+ if(unlikely(!(host_flags & (RRDHOST_FLAG_STREAM_SENDER_ADDED | RRDHOST_FLAG_STREAM_RECEIVER_DISCONNECTED))))
23
+ stream_sender_start_host(host);
24
17
- if(unlikely(!(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS))) {
18
- rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS);
25
+ if(unlikely(!(host_flags & RRDHOST_FLAG_STREAM_SENDER_LOGGED_STATUS))) {
26
+ rrdhost_flag_set(host, RRDHOST_FLAG_STREAM_SENDER_LOGGED_STATUS);
27
nd_log_daemon(NDLP_NOTICE, "STREAM %s [send]: not ready - collected metrics are not sent to parent.", rrdhost_hostname(host));
28
}
29
30
return (RRDSET_STREAM_BUFFER) { .wb = NULL, };
31
}
24
- else if(unlikely(host_flags & RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS)) {
32
+ else if(unlikely(host_flags & RRDHOST_FLAG_STREAM_SENDER_LOGGED_STATUS)) {
33
nd_log_daemon(NDLP_INFO, "STREAM %s [send]: sending metrics to parent...", rrdhost_hostname(host));
26
- rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS);
34
+ rrdhost_flag_clear(host, RRDHOST_FLAG_STREAM_SENDER_LOGGED_STATUS);
35
}
36
37
if(unlikely(host_flags & RRDHOST_FLAG_GLOBAL_FUNCTIONS_UPDATED)) {
30
- BUFFER *wb = sender_start(host->sender);
31
- rrd_global_functions_expose_rrdpush(host, wb, stream_has_capability(host->sender, STREAM_CAP_DYNCFG));
32
- sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_FUNCTIONS);
38
+ BUFFER *wb = preferred_sender_buffer(host);
39
+ stream_sender_send_global_rrdhost_functions(host, wb, stream_has_capability(host->sender, STREAM_CAP_DYNCFG));
40
+ sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
41
}
42
43
bool exposed_upstream = rrdset_check_upstream_exposed(st);
@@ -37,12 +45,13 @@ RRDSET_STREAM_BUFFER rrdset_push_metric_initialize(RRDSET *st, time_t wall_clock
45
bool replication_in_progress = !(rrdset_flags & RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
46
47
if(unlikely((exposed_upstream && replication_in_progress) ||
40
- !should_send_chart_matching(st, rrdset_flags)))
48
+ !should_send_rrdset_matching(st, rrdset_flags)))
49
return (RRDSET_STREAM_BUFFER) { .wb = NULL, };
50
51
if(unlikely(!exposed_upstream)) {
44
- BUFFER *wb = sender_start(host->sender);
45
- replication_in_progress = rrdpush_send_chart_definition(wb, st);
52
+ BUFFER *wb = preferred_sender_buffer(host);
53
+ replication_in_progress = stream_sender_send_rrdset_definition(wb, st);
54
+ sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_METADATA);
55
}
56
57
if(replication_in_progress)
@@ -52,7 +61,7 @@ RRDSET_STREAM_BUFFER rrdset_push_metric_initialize(RRDSET *st, time_t wall_clock
61
.capabilities = host->sender->capabilities,
62
.v2 = stream_has_capability(host->sender, STREAM_CAP_INTERPOLATED),
63
.rrdset_flags = rrdset_flags,
55
- .wb = sender_start(host->sender),
64
+ .wb = preferred_sender_buffer(host),
65
.wall_clock_time = wall_clock_time,
66
};
67
}
src/streaming/protocol/commands.h
+16
-16
@@ -4,7 +4,7 @@
4
#define NETDATA_STREAMING_PROTCOL_COMMANDS_H
5
6
#include "database/rrd.h"
7
-#include "../rrdpush.h"
7
+#include "../stream.h"
8
9
typedef struct rrdset_stream_buffer {
10
STREAM_CAPABILITIES capabilities;
@@ -16,26 +16,26 @@ typedef struct rrdset_stream_buffer {
16
BUFFER *wb;
17
} RRDSET_STREAM_BUFFER;
18
19
-RRDSET_STREAM_BUFFER rrdset_push_metric_initialize(RRDSET *st, time_t wall_clock_time);
19
+RRDSET_STREAM_BUFFER stream_send_metrics_init(RRDSET *st, time_t wall_clock_time);
20
21
-void rrdpush_sender_get_node_and_claim_id_from_parent(struct sender_state *s);
22
-void rrdpush_receiver_send_node_and_claim_id_to_child(RRDHOST *host);
23
-void rrdpush_sender_clear_parent_claim_id(RRDHOST *host);
21
+void stream_sender_get_node_and_claim_id_from_parent(struct sender_state *s);
22
+void stream_receiver_send_node_and_claim_id_to_child(RRDHOST *host);
23
+void stream_sender_clear_parent_claim_id(RRDHOST *host);
24
25
-void rrdpush_sender_send_claimed_id(RRDHOST *host);
25
+void stream_sender_send_claimed_id(RRDHOST *host);
26
27
-void rrdpush_send_global_functions(RRDHOST *host);
28
-void rrdpush_send_host_labels(RRDHOST *host);
27
+void stream_send_global_functions(RRDHOST *host);
28
+void stream_send_host_labels(RRDHOST *host);
29
30
-void rrdpush_sender_thread_send_custom_host_variables(RRDHOST *host);
31
-void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva);
30
+void stream_sender_send_custom_host_variables(RRDHOST *host);
31
+void stream_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva);
32
33
-bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st);
34
-bool rrdset_push_chart_definition_now(RRDSET *st);
35
-bool should_send_chart_matching(RRDSET *st, RRDSET_FLAGS flags);
33
+bool stream_sender_send_rrdset_definition(BUFFER *wb, RRDSET *st);
34
+bool stream_sender_send_rrdset_definition_now(RRDSET *st);
35
+bool should_send_rrdset_matching(RRDSET *st, RRDSET_FLAGS flags);
36
37
-void rrdset_push_metrics_v1(RRDSET_STREAM_BUFFER *rsb, RRDSET *st);
38
-void rrddim_push_metrics_v2(RRDSET_STREAM_BUFFER *rsb, RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n, SN_FLAGS flags);
39
-void rrdset_push_metrics_finished(RRDSET_STREAM_BUFFER *rsb, RRDSET *st);
37
+void stream_send_rrdset_metrics_v1(RRDSET_STREAM_BUFFER *rsb, RRDSET *st);
38
+void stream_send_rrddim_metrics_v2(RRDSET_STREAM_BUFFER *rsb, RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n, SN_FLAGS flags);
39
+void stream_send_rrdset_metrics_finished(RRDSET_STREAM_BUFFER *rsb, RRDSET *st);
40
41
#endif //NETDATA_STREAMING_PROTCOL_COMMANDS_H
src/streaming/receiver.c
deleted
-1385
@@ -1,1385 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#include "rrdpush.h"
4
-#include "web/server/h2o/http_server.h"
5
-
6
-// When a child disconnects this is the maximum we will wait
7
-// before we update the cloud that the child is offline
8
-#define MAX_CHILD_DISC_DELAY (30000)
9
-#define MAX_CHILD_DISC_TOLERANCE (125 / 100)
10
-
11
-void receiver_state_free(struct receiver_state *rpt) {
12
- netdata_ssl_close(&rpt->ssl);
13
-
14
- if(rpt->fd != -1) {
15
- internal_error(true, "closing socket...");
16
- close(rpt->fd);
17
- }
18
-
19
- rrdpush_decompressor_destroy(&rpt->decompressor);
20
-
21
- if(rpt->system_info)
22
- rrdhost_system_info_free(rpt->system_info);
23
-
24
- __atomic_sub_fetch(&netdata_buffers_statistics.rrdhost_receivers, sizeof(*rpt), __ATOMIC_RELAXED);
25
-
26
- freez(rpt->key);
27
- freez(rpt->hostname);
28
- freez(rpt->registry_hostname);
29
- freez(rpt->machine_guid);
30
- freez(rpt->os);
31
- freez(rpt->timezone);
32
- freez(rpt->abbrev_timezone);
33
- freez(rpt->client_ip);
34
- freez(rpt->client_port);
35
- freez(rpt->program_name);
36
- freez(rpt->program_version);
37
- freez(rpt);
38
-}
39
-
40
-#include "plugins.d/pluginsd_parser.h"
41
-
42
-// IMPORTANT: to add workers, you have to edit WORKER_PARSER_FIRST_JOB accordingly
43
-#define WORKER_RECEIVER_JOB_BYTES_READ (WORKER_PARSER_FIRST_JOB - 1)
44
-#define WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED (WORKER_PARSER_FIRST_JOB - 2)
45
-
46
-// this has to be the same at parser.h
47
-#define WORKER_RECEIVER_JOB_REPLICATION_COMPLETION (WORKER_PARSER_FIRST_JOB - 3)
48
-
49
-#if WORKER_PARSER_FIRST_JOB < 1
50
-#error The define WORKER_PARSER_FIRST_JOB needs to be at least 1
51
-#endif
52
-
53
-static inline int read_stream(struct receiver_state *r, char* buffer, size_t size) {
54
- if(unlikely(!size)) {
55
- internal_error(true, "%s() asked to read zero bytes", __FUNCTION__);
56
- return 0;
57
- }
58
-
59
-#ifdef ENABLE_H2O
60
- if (is_h2o_rrdpush(r)) {
61
- if(nd_thread_signaled_to_cancel())
62
- return -4;
63
-
64
- return (int)h2o_stream_read(r->h2o_ctx, buffer, size);
65
- }
66
-#endif
67
-
68
- int tries = 100;
69
- ssize_t bytes_read;
70
-
71
- do {
72
- errno_clear();
73
-
74
- switch(wait_on_socket_or_cancel_with_timeout(
75
- &r->ssl,
76
- r->fd, 0, POLLIN, NULL))
77
- {
78
- case 0: // data are waiting
79
- break;
80
-
81
- case 1: // timeout reached
82
- netdata_log_error("STREAM: %s(): timeout while waiting for data on socket!", __FUNCTION__);
83
- return -3;
84
-
85
- case -1: // thread cancelled
86
- netdata_log_error("STREAM: %s(): thread has been cancelled timeout while waiting for data on socket!", __FUNCTION__);
87
- return -4;
88
-
89
- default:
90
- case 2: // error on socket
91
- netdata_log_error("STREAM: %s() socket error!", __FUNCTION__);
92
- return -2;
93
- }
94
-
95
- if (SSL_connection(&r->ssl))
96
- bytes_read = netdata_ssl_read(&r->ssl, buffer, size);
97
- else
98
- bytes_read = read(r->fd, buffer, size);
99
-
100
- } while(bytes_read < 0 && errno == EINTR && tries--);
101
-
102
- if((bytes_read == 0 || bytes_read == -1) && (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINPROGRESS)) {
103
- netdata_log_error("STREAM: %s(): timeout while waiting for data on socket!", __FUNCTION__);
104
- bytes_read = -3;
105
- }
106
- else if (bytes_read == 0) {
107
- netdata_log_error("STREAM: %s(): EOF while reading data from socket!", __FUNCTION__);
108
- bytes_read = -1;
109
- }
110
- else if (bytes_read < 0) {
111
- netdata_log_error("STREAM: %s() failed to read from socket!", __FUNCTION__);
112
- bytes_read = -2;
113
- }
114
-
115
- return (int)bytes_read;
116
-}
117
-
118
-static inline STREAM_HANDSHAKE read_stream_error_to_reason(int code) {
119
- if(code > 0)
120
- return 0;
121
-
122
- switch(code) {
123
- case 0:
124
- // asked to read zero bytes
125
- return STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_READ_BUFFER;
126
-
127
- case -1:
128
- // EOF
129
- return STREAM_HANDSHAKE_DISCONNECT_SOCKET_EOF;
130
-
131
- case -2:
132
- // failed to read
133
- return STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_FAILED;
134
-
135
- case -3:
136
- // timeout
137
- return STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_TIMEOUT;
138
-
139
- case -4:
140
- // the thread is cancelled
141
- return STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN;
142
-
143
- default:
144
- // anything else
145
- return STREAM_HANDSHAKE_DISCONNECT_UNKNOWN_SOCKET_READ_ERROR;
146
- }
147
-}
148
-
149
-static inline bool receiver_read_uncompressed(struct receiver_state *r, STREAM_HANDSHAKE *reason) {
150
-#ifdef NETDATA_INTERNAL_CHECKS
151
- if(r->reader.read_buffer[r->reader.read_len] != '\0')
152
- fatal("%s(): read_buffer does not start with zero", __FUNCTION__ );
153
-#endif
154
-
155
- int bytes_read = read_stream(r, r->reader.read_buffer + r->reader.read_len, sizeof(r->reader.read_buffer) - r->reader.read_len - 1);
156
- if(unlikely(bytes_read <= 0)) {
157
- *reason = read_stream_error_to_reason(bytes_read);
158
- return false;
159
- }
160
-
161
- worker_set_metric(WORKER_RECEIVER_JOB_BYTES_READ, (NETDATA_DOUBLE)bytes_read);
162
- worker_set_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED, (NETDATA_DOUBLE)bytes_read);
163
-
164
- r->reader.read_len += bytes_read;
165
- r->reader.read_buffer[r->reader.read_len] = '\0';
166
-
167
- return true;
168
-}
169
-
170
-static inline bool receiver_read_compressed(struct receiver_state *r, STREAM_HANDSHAKE *reason) {
171
-
172
- internal_fatal(r->reader.read_buffer[r->reader.read_len] != '\0',
173
- "%s: read_buffer does not start with zero #2", __FUNCTION__ );
174
-
175
- // first use any available uncompressed data
176
- if (likely(rrdpush_decompressed_bytes_in_buffer(&r->decompressor))) {
177
- size_t available = sizeof(r->reader.read_buffer) - r->reader.read_len - 1;
178
- if (likely(available)) {
179
- size_t len = rrdpush_decompressor_get(&r->decompressor, r->reader.read_buffer + r->reader.read_len, available);
180
- if (unlikely(!len)) {
181
- internal_error(true, "decompressor returned zero length #1");
182
- return false;
183
- }
184
-
185
- r->reader.read_len += (int)len;
186
- r->reader.read_buffer[r->reader.read_len] = '\0';
187
- }
188
- else
189
- internal_fatal(true, "The line to read is too big! Already have %zd bytes in read_buffer.", r->reader.read_len);
190
-
191
- return true;
192
- }
193
-
194
- // no decompressed data available
195
- // read the compression signature of the next block
196
-
197
- if(unlikely(r->reader.read_len + r->decompressor.signature_size > sizeof(r->reader.read_buffer) - 1)) {
198
- internal_error(true, "The last incomplete line does not leave enough room for the next compression header! "
199
- "Already have %zd bytes in read_buffer.", r->reader.read_len);
200
- return false;
201
- }
202
-
203
- // read the compression signature from the stream
204
- // we have to do a loop here, because read_stream() may return less than the data we need
205
- int bytes_read = 0;
206
- do {
207
- int ret = read_stream(r, r->reader.read_buffer + r->reader.read_len + bytes_read, r->decompressor.signature_size - bytes_read);
208
- if (unlikely(ret <= 0)) {
209
- *reason = read_stream_error_to_reason(ret);
210
- return false;
211
- }
212
-
213
- bytes_read += ret;
214
- } while(unlikely(bytes_read < (int)r->decompressor.signature_size));
215
-
216
- worker_set_metric(WORKER_RECEIVER_JOB_BYTES_READ, (NETDATA_DOUBLE)bytes_read);
217
-
218
- if(unlikely(bytes_read != (int)r->decompressor.signature_size))
219
- fatal("read %d bytes, but expected compression signature of size %zu", bytes_read, r->decompressor.signature_size);
220
-
221
- size_t compressed_message_size = rrdpush_decompressor_start(&r->decompressor, r->reader.read_buffer + r->reader.read_len, bytes_read);
222
- if (unlikely(!compressed_message_size)) {
223
- internal_error(true, "multiplexed uncompressed data in compressed stream!");
224
- r->reader.read_len += bytes_read;
225
- r->reader.read_buffer[r->reader.read_len] = '\0';
226
- return true;
227
- }
228
-
229
- if(unlikely(compressed_message_size > COMPRESSION_MAX_MSG_SIZE)) {
230
- netdata_log_error("received a compressed message of %zu bytes, which is bigger than the max compressed message size supported of %zu. Ignoring message.",
231
- compressed_message_size, (size_t)COMPRESSION_MAX_MSG_SIZE);
232
- return false;
233
- }
234
-
235
- // delete compression header from our read buffer
236
- r->reader.read_buffer[r->reader.read_len] = '\0';
237
-
238
- // Read the entire compressed block of compressed data
239
- char compressed[compressed_message_size];
240
- size_t compressed_bytes_read = 0;
241
- do {
242
- size_t start = compressed_bytes_read;
243
- size_t remaining = compressed_message_size - start;
244
-
245
- int last_read_bytes = read_stream(r, &compressed[start], remaining);
246
- if (unlikely(last_read_bytes <= 0)) {
247
- *reason = read_stream_error_to_reason(last_read_bytes);
248
- return false;
249
- }
250
-
251
- compressed_bytes_read += last_read_bytes;
252
-
253
- } while(unlikely(compressed_message_size > compressed_bytes_read));
254
-
255
- worker_set_metric(WORKER_RECEIVER_JOB_BYTES_READ, (NETDATA_DOUBLE)compressed_bytes_read);
256
-
257
- // decompress the compressed block
258
- size_t bytes_to_parse = rrdpush_decompress(&r->decompressor, compressed, compressed_bytes_read);
259
- if (unlikely(!bytes_to_parse)) {
260
- internal_error(true, "no bytes to parse.");
261
- return false;
262
- }
263
-
264
- worker_set_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED, (NETDATA_DOUBLE)bytes_to_parse);
265
-
266
- // fill read buffer with decompressed data
267
- size_t len = (int) rrdpush_decompressor_get(&r->decompressor, r->reader.read_buffer + r->reader.read_len, sizeof(r->reader.read_buffer) - r->reader.read_len - 1);
268
- if (unlikely(!len)) {
269
- internal_error(true, "decompressor returned zero length #2");
270
- return false;
271
- }
272
- r->reader.read_len += (int)len;
273
- r->reader.read_buffer[r->reader.read_len] = '\0';
274
-
275
- return true;
276
-}
277
-
278
-bool plugin_is_enabled(struct plugind *cd);
279
-
280
-static void receiver_set_exit_reason(struct receiver_state *rpt, STREAM_HANDSHAKE reason, bool force) {
281
- if(force || !rpt->exit.reason)
282
- rpt->exit.reason = reason;
283
-}
284
-
285
-static inline bool receiver_should_stop(struct receiver_state *rpt) {
286
- static __thread size_t counter = 0;
287
-
288
- if(nd_thread_signaled_to_cancel()) {
289
- receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN, false);
290
- return true;
291
- }
292
-
293
- if(unlikely(rpt->exit.shutdown)) {
294
- receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN, false);
295
- return true;
296
- }
297
-
298
- if(unlikely(!service_running(SERVICE_STREAMING))) {
299
- receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_NETDATA_EXIT, false);
300
- return true;
301
- }
302
-
303
- if(unlikely((counter++ % 1000) == 0))
304
- rpt->last_msg_t = now_monotonic_sec();
305
-
306
- return false;
307
-}
308
-
309
-static size_t streaming_parser(struct receiver_state *rpt, struct plugind *cd, int fd, void *ssl) {
310
- size_t result = 0;
311
-
312
- PARSER *parser = NULL;
313
- {
314
- PARSER_USER_OBJECT user = {
315
- .enabled = plugin_is_enabled(cd),
316
- .host = rpt->host,
317
- .opaque = rpt,
318
- .cd = cd,
319
- .trust_durations = 1,
320
- .capabilities = rpt->capabilities,
321
- };
322
-
323
- parser = parser_init(&user, fd, fd, PARSER_INPUT_SPLIT, ssl);
324
- }
325
-
326
-#ifdef ENABLE_H2O
327
- parser->h2o_ctx = rpt->h2o_ctx;
328
-#endif
329
-
330
- pluginsd_keywords_init(parser, PARSER_INIT_STREAMING);
331
-
332
- rrd_collector_started();
333
-
334
- bool compressed_connection = rrdpush_decompression_initialize(rpt);
335
- buffered_reader_init(&rpt->reader);
336
-
337
-#ifdef NETDATA_LOG_STREAM_RECEIVE
338
- {
339
- char filename[FILENAME_MAX + 1];
340
- snprintfz(filename, FILENAME_MAX, "/tmp/stream-receiver-%s.txt", rpt->host ? rrdhost_hostname(
341
- rpt->host) : "unknown"
342
- );
343
- parser->user.stream_log_fp = fopen(filename, "w");
344
- parser->user.stream_log_repertoire = PARSER_REP_METADATA;
345
- }
346
-#endif
347
-
348
- CLEAN_BUFFER *buffer = buffer_create(sizeof(rpt->reader.read_buffer), NULL);
349
-
350
- ND_LOG_STACK lgs[] = {
351
- ND_LOG_FIELD_CB(NDF_REQUEST, line_splitter_reconstruct_line, &parser->line),
352
- ND_LOG_FIELD_CB(NDF_NIDL_NODE, parser_reconstruct_node, parser),
353
- ND_LOG_FIELD_CB(NDF_NIDL_INSTANCE, parser_reconstruct_instance, parser),
354
- ND_LOG_FIELD_CB(NDF_NIDL_CONTEXT, parser_reconstruct_context, parser),
355
- ND_LOG_FIELD_END(),
356
- };
357
- ND_LOG_STACK_PUSH(lgs);
358
-
359
- __atomic_store_n(&rpt->parser, parser, __ATOMIC_RELAXED);
360
- rrdpush_receiver_send_node_and_claim_id_to_child(rpt->host);
361
-
362
- while(!receiver_should_stop(rpt)) {
363
-
364
- if(!buffered_reader_next_line(&rpt->reader, buffer)) {
365
- STREAM_HANDSHAKE reason = STREAM_HANDSHAKE_DISCONNECT_UNKNOWN_SOCKET_READ_ERROR;
366
-
367
- bool have_new_data = compressed_connection ? receiver_read_compressed(rpt, &reason)
368
- : receiver_read_uncompressed(rpt, &reason);
369
-
370
- if(unlikely(!have_new_data)) {
371
- receiver_set_exit_reason(rpt, reason, false);
372
- break;
373
- }
374
-
375
- continue;
376
- }
377
-
378
- if(unlikely(parser_action(parser, buffer->buffer))) {
379
- receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, false);
380
- break;
381
- }
382
-
383
- buffer->len = 0;
384
- buffer->buffer[0] = '\0';
385
- }
386
-
387
- // cleanup the sender buffer, because we may end-up reusing an incomplete buffer
388
- sender_thread_buffer_free();
389
- parser->user.v2.stream_buffer.wb = NULL;
390
-
391
- // make sure send_to_plugin() will not write any data to the socket
392
- spinlock_lock(&parser->writer.spinlock);
393
- parser->fd_output = -1;
394
- parser->ssl_output = NULL;
395
- spinlock_unlock(&parser->writer.spinlock);
396
-
397
- result = parser->user.data_collections_count;
398
- return result;
399
-}
400
-
401
-static void rrdpush_receiver_replication_reset(RRDHOST *host) {
402
- RRDSET *st;
403
- rrdset_foreach_read(st, host) {
404
- rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS);
405
- rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
406
- }
407
- rrdset_foreach_done(st);
408
- rrdhost_receiver_replicating_charts_zero(host);
409
-}
410
-
411
-static bool rrdhost_set_receiver(RRDHOST *host, struct receiver_state *rpt) {
412
- bool signal_rrdcontext = false;
413
- bool set_this = false;
414
-
415
- spinlock_lock(&host->receiver_lock);
416
-
417
- if (!host->receiver) {
418
- rrdhost_flag_clear(host, RRDHOST_FLAG_ORPHAN);
419
-
420
- host->rrdpush_receiver_connection_counter++;
421
- __atomic_add_fetch(&localhost->connected_children_count, 1, __ATOMIC_RELAXED);
422
-
423
- host->receiver = rpt;
424
- rpt->host = host;
425
-
426
- host->child_connect_time = now_realtime_sec();
427
- host->child_disconnected_time = 0;
428
- host->child_last_chart_command = 0;
429
- host->trigger_chart_obsoletion_check = 1;
430
-
431
- if (rpt->config.health_enabled != CONFIG_BOOLEAN_NO) {
432
- if (rpt->config.alarms_delay > 0) {
433
- host->health.health_delay_up_to = now_realtime_sec() + rpt->config.alarms_delay;
434
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
435
- "[%s]: Postponing health checks for %" PRId64 " seconds, because it was just connected.",
436
- rrdhost_hostname(host),
437
- (int64_t) rpt->config.alarms_delay);
438
- }
439
- }
440
-
441
- host->health_log.health_log_retention_s = rpt->config.alarms_history;
442
-
443
-// this is a test
444
-// if(rpt->hops <= host->sender->hops)
445
-// rrdpush_sender_thread_stop(host, "HOPS MISMATCH", false);
446
-
447
- signal_rrdcontext = true;
448
- rrdpush_receiver_replication_reset(host);
449
-
450
- rrdhost_flag_clear(rpt->host, RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED);
451
- aclk_queue_node_info(rpt->host, true);
452
-
453
- rrdpush_reset_destinations_postpone_time(host);
454
-
455
- set_this = true;
456
- }
457
-
458
- spinlock_unlock(&host->receiver_lock);
459
-
460
- if(signal_rrdcontext)
461
- rrdcontext_host_child_connected(host);
462
-
463
- return set_this;
464
-}
465
-
466
-static void rrdhost_clear_receiver(struct receiver_state *rpt) {
467
- RRDHOST *host = rpt->host;
468
- if(!host) return;
469
-
470
- spinlock_lock(&host->receiver_lock);
471
- {
472
- // Make sure that we detach this thread and don't kill a freshly arriving receiver
473
-
474
- if (host->receiver == rpt) {
475
- spinlock_unlock(&host->receiver_lock);
476
- {
477
- // run all these without having the receiver lock
478
-
479
- stream_path_child_disconnected(host);
480
- rrdpush_sender_thread_stop(host, STREAM_HANDSHAKE_DISCONNECT_RECEIVER_LEFT, false);
481
- rrdpush_receiver_replication_reset(host);
482
- rrdcontext_host_child_disconnected(host);
483
-
484
- if (rpt->config.health_enabled)
485
- rrdcalc_child_disconnected(host);
486
-
487
- rrdpush_reset_destinations_postpone_time(host);
488
- }
489
- spinlock_lock(&host->receiver_lock);
490
-
491
- // now we have the lock again
492
-
493
- __atomic_sub_fetch(&localhost->connected_children_count, 1, __ATOMIC_RELAXED);
494
- rrdhost_flag_set(rpt->host, RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED);
495
-
496
- host->trigger_chart_obsoletion_check = 0;
497
- host->child_connect_time = 0;
498
- host->child_disconnected_time = now_realtime_sec();
499
- host->health.health_enabled = 0;
500
-
501
- host->rrdpush_last_receiver_exit_reason = rpt->exit.reason;
502
- rrdhost_flag_set(host, RRDHOST_FLAG_ORPHAN);
503
- host->receiver = NULL;
504
- }
505
- }
506
-
507
- // this must be cleared with the receiver lock
508
- pluginsd_process_cleanup(rpt->parser);
509
- __atomic_store_n(&rpt->parser, NULL, __ATOMIC_RELAXED);
510
-
511
- spinlock_unlock(&host->receiver_lock);
512
-}
513
-
514
-bool stop_streaming_receiver(RRDHOST *host, STREAM_HANDSHAKE reason) {
515
- bool ret = false;
516
-
517
- spinlock_lock(&host->receiver_lock);
518
-
519
- if(host->receiver) {
520
- if(!host->receiver->exit.shutdown) {
521
- host->receiver->exit.shutdown = true;
522
- receiver_set_exit_reason(host->receiver, reason, true);
523
- shutdown(host->receiver->fd, SHUT_RDWR);
524
- }
525
-
526
- nd_thread_signal_cancel(host->receiver->thread);
527
- }
528
-
529
- int count = 2000;
530
- while (host->receiver && count-- > 0) {
531
- spinlock_unlock(&host->receiver_lock);
532
-
533
- // let the lock for the receiver thread to exit
534
- sleep_usec(1 * USEC_PER_MS);
535
-
536
- spinlock_lock(&host->receiver_lock);
537
- }
538
-
539
- if(host->receiver)
540
- netdata_log_error("STREAM '%s' [receive from [%s]:%s]: "
541
- "thread %d takes too long to stop, giving up..."
542
- , rrdhost_hostname(host)
543
- , host->receiver->client_ip, host->receiver->client_port
544
- , host->receiver->tid);
545
- else
546
- ret = true;
547
-
548
- spinlock_unlock(&host->receiver_lock);
549
-
550
- return ret;
551
-}
552
-
553
-static void rrdpush_send_error_on_taken_over_connection(struct receiver_state *rpt, const char *msg) {
554
- (void) send_timeout(
555
- &rpt->ssl,
556
- rpt->fd,
557
- (char *)msg,
558
- strlen(msg),
559
- 0,
560
- 5);
561
-}
562
-
563
-static void rrdpush_receive_log_status(struct receiver_state *rpt, const char *msg, const char *status, ND_LOG_FIELD_PRIORITY priority) {
564
- // this function may be called BEFORE we spawn the receiver thread
565
- // so, we need to add the fields again (it does not harm)
566
- ND_LOG_STACK lgs[] = {
567
- ND_LOG_FIELD_TXT(NDF_SRC_IP, rpt->client_ip),
568
- ND_LOG_FIELD_TXT(NDF_SRC_PORT, rpt->client_port),
569
- ND_LOG_FIELD_TXT(NDF_NIDL_NODE, (rpt->hostname && *rpt->hostname) ? rpt->hostname : ""),
570
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, status),
571
- ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_from_child_msgid),
572
- ND_LOG_FIELD_END(),
573
- };
574
- ND_LOG_STACK_PUSH(lgs);
575
-
576
- nd_log(NDLS_ACCESS, priority, "api_key:'%s' machine_guid:'%s' msg:'%s'"
577
- , (rpt->key && *rpt->key)? rpt->key : ""
578
- , (rpt->machine_guid && *rpt->machine_guid) ? rpt->machine_guid : ""
579
- , msg);
580
-
581
- nd_log(NDLS_DAEMON, priority, "STREAM_RECEIVER for '%s': %s %s%s%s"
582
- , (rpt->hostname && *rpt->hostname) ? rpt->hostname : ""
583
- , msg
584
- , rpt->exit.reason != STREAM_HANDSHAKE_NEVER?" (":""
585
- , stream_handshake_error_to_string(rpt->exit.reason)
586
- , rpt->exit.reason != STREAM_HANDSHAKE_NEVER?")":""
587
- );
588
-}
589
-
590
-static void rrdpush_receive(struct receiver_state *rpt)
591
-{
592
- rpt->config.mode = default_rrd_memory_mode;
593
- rpt->config.history = default_rrd_history_entries;
594
-
595
- rpt->config.health_enabled = health_plugin_enabled();
596
- rpt->config.alarms_delay = 60;
597
- rpt->config.alarms_history = HEALTH_LOG_RETENTION_DEFAULT;
598
-
599
- rpt->config.rrdpush_enabled = (int)stream_conf_send_enabled;
600
- rpt->config.rrdpush_destination = stream_conf_send_destination;
601
- rpt->config.rrdpush_api_key = stream_conf_send_api_key;
602
- rpt->config.rrdpush_send_charts_matching = stream_conf_send_charts_matching;
603
-
604
- rpt->config.rrdpush_enable_replication = stream_conf_replication_enabled;
605
- rpt->config.rrdpush_seconds_to_replicate = stream_conf_replication_period;
606
- rpt->config.rrdpush_replication_step = stream_conf_replication_step;
607
-
608
- rpt->config.update_every = (int)appconfig_get_duration_seconds(&stream_config, rpt->machine_guid, "update every", rpt->config.update_every);
609
- if(rpt->config.update_every < 0) rpt->config.update_every = 1;
610
-
611
- rpt->config.history = (int)appconfig_get_number(&stream_config, rpt->key, "retention", rpt->config.history);
612
- rpt->config.history = (int)appconfig_get_number(&stream_config, rpt->machine_guid, "retention", rpt->config.history);
613
- if(rpt->config.history < 5) rpt->config.history = 5;
614
-
615
- rpt->config.mode = rrd_memory_mode_id(appconfig_get(&stream_config, rpt->key, "db", rrd_memory_mode_name(rpt->config.mode)));
616
- rpt->config.mode = rrd_memory_mode_id(appconfig_get(&stream_config, rpt->machine_guid, "db", rrd_memory_mode_name(rpt->config.mode)));
617
-
618
- if (unlikely(rpt->config.mode == RRD_MEMORY_MODE_DBENGINE && !dbengine_enabled)) {
619
- netdata_log_error("STREAM '%s' [receive from %s:%s]: "
620
- "dbengine is not enabled, falling back to default."
621
- , rpt->hostname
622
- , rpt->client_ip, rpt->client_port
623
- );
624
-
625
- rpt->config.mode = default_rrd_memory_mode;
626
- }
627
-
628
- rpt->config.health_enabled = appconfig_get_boolean_ondemand(&stream_config, rpt->key, "health enabled by default", rpt->config.health_enabled);
629
- rpt->config.health_enabled = appconfig_get_boolean_ondemand(&stream_config, rpt->machine_guid, "health enabled", rpt->config.health_enabled);
630
-
631
- rpt->config.alarms_delay = appconfig_get_duration_seconds(&stream_config, rpt->key, "postpone alerts on connect", rpt->config.alarms_delay);
632
- rpt->config.alarms_delay = appconfig_get_duration_seconds(&stream_config, rpt->machine_guid, "postpone alerts on connect", rpt->config.alarms_delay);
633
-
634
- rpt->config.alarms_history = appconfig_get_duration_seconds(&stream_config, rpt->key, "health log retention", rpt->config.alarms_history);
635
- rpt->config.alarms_history = appconfig_get_duration_seconds(&stream_config, rpt->machine_guid, "health log retention", rpt->config.alarms_history);
636
-
637
- rpt->config.rrdpush_enabled = appconfig_get_boolean(&stream_config, rpt->key, "proxy enabled", rpt->config.rrdpush_enabled);
638
- rpt->config.rrdpush_enabled = appconfig_get_boolean(&stream_config, rpt->machine_guid, "proxy enabled", rpt->config.rrdpush_enabled);
639
-
640
- rpt->config.rrdpush_destination = appconfig_get(&stream_config, rpt->key, "proxy destination", rpt->config.rrdpush_destination);
641
- rpt->config.rrdpush_destination = appconfig_get(&stream_config, rpt->machine_guid, "proxy destination", rpt->config.rrdpush_destination);
642
-
643
- rpt->config.rrdpush_api_key = appconfig_get(&stream_config, rpt->key, "proxy api key", rpt->config.rrdpush_api_key);
644
- rpt->config.rrdpush_api_key = appconfig_get(&stream_config, rpt->machine_guid, "proxy api key", rpt->config.rrdpush_api_key);
645
-
646
- rpt->config.rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->key, "proxy send charts matching", rpt->config.rrdpush_send_charts_matching);
647
- rpt->config.rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->machine_guid, "proxy send charts matching", rpt->config.rrdpush_send_charts_matching);
648
-
649
- rpt->config.rrdpush_enable_replication = appconfig_get_boolean(&stream_config, rpt->key, "enable replication", rpt->config.rrdpush_enable_replication);
650
- rpt->config.rrdpush_enable_replication = appconfig_get_boolean(&stream_config, rpt->machine_guid, "enable replication", rpt->config.rrdpush_enable_replication);
651
-
652
- rpt->config.rrdpush_seconds_to_replicate = appconfig_get_duration_seconds(&stream_config, rpt->key, "replication period", rpt->config.rrdpush_seconds_to_replicate);
653
- rpt->config.rrdpush_seconds_to_replicate = appconfig_get_duration_seconds(&stream_config, rpt->machine_guid, "replication period", rpt->config.rrdpush_seconds_to_replicate);
654
-
655
- rpt->config.rrdpush_replication_step = appconfig_get_number(&stream_config, rpt->key, "replication step", rpt->config.rrdpush_replication_step);
656
- rpt->config.rrdpush_replication_step = appconfig_get_number(&stream_config, rpt->machine_guid, "replication step", rpt->config.rrdpush_replication_step);
657
-
658
- rpt->config.rrdpush_compression = stream_conf_compression_enabled;
659
- rpt->config.rrdpush_compression = appconfig_get_boolean(&stream_config, rpt->key, "enable compression", rpt->config.rrdpush_compression);
660
- rpt->config.rrdpush_compression = appconfig_get_boolean(&stream_config, rpt->machine_guid, "enable compression", rpt->config.rrdpush_compression);
661
-
662
- bool is_ephemeral = false;
663
- is_ephemeral = appconfig_get_boolean(&stream_config, rpt->key, "is ephemeral node", CONFIG_BOOLEAN_NO);
664
- is_ephemeral = appconfig_get_boolean(&stream_config, rpt->machine_guid, "is ephemeral node", is_ephemeral);
665
-
666
- if(rpt->config.rrdpush_compression) {
667
- const char *order = appconfig_get(&stream_config, rpt->key, "compression algorithms order", RRDPUSH_COMPRESSION_ALGORITHMS_ORDER);
668
- order = appconfig_get(&stream_config, rpt->machine_guid, "compression algorithms order", order);
669
- rrdpush_parse_compression_order(rpt, order);
670
- }
671
-
672
- // find the host for this receiver
673
- {
674
- // this will also update the host with our system_info
675
- RRDHOST *host = rrdhost_find_or_create(
676
- rpt->hostname,
677
- rpt->registry_hostname,
678
- rpt->machine_guid,
679
- rpt->os,
680
- rpt->timezone,
681
- rpt->abbrev_timezone,
682
- rpt->utc_offset,
683
- rpt->program_name,
684
- rpt->program_version,
685
- rpt->config.update_every,
686
- rpt->config.history,
687
- rpt->config.mode,
688
- (unsigned int)(rpt->config.health_enabled != CONFIG_BOOLEAN_NO),
689
- (unsigned int)(rpt->config.rrdpush_enabled && rpt->config.rrdpush_destination &&
690
- *rpt->config.rrdpush_destination && rpt->config.rrdpush_api_key &&
691
- *rpt->config.rrdpush_api_key),
692
- rpt->config.rrdpush_destination,
693
- rpt->config.rrdpush_api_key,
694
- rpt->config.rrdpush_send_charts_matching,
695
- rpt->config.rrdpush_enable_replication,
696
- rpt->config.rrdpush_seconds_to_replicate,
697
- rpt->config.rrdpush_replication_step,
698
- rpt->system_info,
699
- 0);
700
-
701
- if(!host) {
702
- rrdpush_receive_log_status(
703
- rpt,"failed to find/create host structure, rejecting connection",
704
- RRDPUSH_STATUS_INTERNAL_SERVER_ERROR, NDLP_ERR);
705
-
706
- rrdpush_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_INTERNAL_ERROR);
707
- goto cleanup;
708
- }
709
-
710
- if (unlikely(rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_CONTEXT_LOAD))) {
711
- rrdpush_receive_log_status(
712
- rpt, "host is initializing, retry later",
713
- RRDPUSH_STATUS_INITIALIZATION_IN_PROGRESS, NDLP_NOTICE);
714
-
715
- rrdpush_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_INITIALIZATION);
716
- goto cleanup;
717
- }
718
-
719
- // system_info has been consumed by the host structure
720
- rpt->system_info = NULL;
721
-
722
- if(!rrdhost_set_receiver(host, rpt)) {
723
- rrdpush_receive_log_status(
724
- rpt, "host is already served by another receiver",
725
- RRDPUSH_STATUS_DUPLICATE_RECEIVER, NDLP_INFO);
726
-
727
- rrdpush_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_ALREADY_STREAMING);
728
- goto cleanup;
729
- }
730
- }
731
-
732
-#ifdef NETDATA_INTERNAL_CHECKS
733
- netdata_log_info("STREAM '%s' [receive from [%s]:%s]: "
734
- "client willing to stream metrics for host '%s' with machine_guid '%s': "
735
- "update every = %d, history = %d, memory mode = %s, health %s,%s"
736
- , rpt->hostname
737
- , rpt->client_ip
738
- , rpt->client_port
739
- , rrdhost_hostname(rpt->host)
740
- , rpt->host->machine_guid
741
- , rpt->host->rrd_update_every
742
- , rpt->host->rrd_history_entries
743
- , rrd_memory_mode_name(rpt->host->rrd_memory_mode)
744
- , (rpt->config.health_enabled == CONFIG_BOOLEAN_NO)?"disabled":((rpt->config.health_enabled == CONFIG_BOOLEAN_YES)?"enabled":"auto")
745
- , (rpt->ssl.conn != NULL) ? " SSL," : ""
746
- );
747
-#endif // NETDATA_INTERNAL_CHECKS
748
-
749
-
750
- struct plugind cd = {
751
- .update_every = default_rrd_update_every,
752
- .unsafe = {
753
- .spinlock = NETDATA_SPINLOCK_INITIALIZER,
754
- .running = true,
755
- .enabled = true,
756
- },
757
- .started_t = now_realtime_sec(),
758
- };
759
-
760
- // put the client IP and port into the buffers used by plugins.d
761
- snprintfz(cd.id, CONFIG_MAX_NAME, "%s:%s", rpt->client_ip, rpt->client_port);
762
- snprintfz(cd.filename, FILENAME_MAX, "%s:%s", rpt->client_ip, rpt->client_port);
763
- snprintfz(cd.fullfilename, FILENAME_MAX, "%s:%s", rpt->client_ip, rpt->client_port);
764
- snprintfz(cd.cmd, PLUGINSD_CMD_MAX, "%s:%s", rpt->client_ip, rpt->client_port);
765
-
766
- rrdpush_select_receiver_compression_algorithm(rpt);
767
-
768
- {
769
- // netdata_log_info("STREAM %s [receive from [%s]:%s]: initializing communication...", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
770
- char initial_response[HTTP_HEADER_SIZE];
771
- if (stream_has_capability(rpt, STREAM_CAP_VCAPS)) {
772
- log_receiver_capabilities(rpt);
773
- sprintf(initial_response, "%s%u", START_STREAMING_PROMPT_VN, rpt->capabilities);
774
- }
775
- else if (stream_has_capability(rpt, STREAM_CAP_VN)) {
776
- log_receiver_capabilities(rpt);
777
- sprintf(initial_response, "%s%d", START_STREAMING_PROMPT_VN, stream_capabilities_to_vn(rpt->capabilities));
778
- }
779
- else if (stream_has_capability(rpt, STREAM_CAP_V2)) {
780
- log_receiver_capabilities(rpt);
781
- sprintf(initial_response, "%s", START_STREAMING_PROMPT_V2);
782
- }
783
- else { // stream_has_capability(rpt, STREAM_CAP_V1)
784
- log_receiver_capabilities(rpt);
785
- sprintf(initial_response, "%s", START_STREAMING_PROMPT_V1);
786
- }
787
-
788
- netdata_log_debug(D_STREAM, "Initial response to %s: %s", rpt->client_ip, initial_response);
789
-#ifdef ENABLE_H2O
790
- if (is_h2o_rrdpush(rpt)) {
791
- h2o_stream_write(rpt->h2o_ctx, initial_response, strlen(initial_response));
792
- } else {
793
-#endif
794
- ssize_t bytes_sent = send_timeout(
795
- &rpt->ssl,
796
- rpt->fd, initial_response, strlen(initial_response), 0, 60);
797
-
798
- if(bytes_sent != (ssize_t)strlen(initial_response)) {
799
- internal_error(true, "Cannot send response, got %zd bytes, expecting %zu bytes", bytes_sent, strlen(initial_response));
800
- rrdpush_receive_log_status(
801
- rpt, "cannot reply back, dropping connection",
802
- RRDPUSH_STATUS_CANT_REPLY, NDLP_ERR);
803
- goto cleanup;
804
- }
805
-#ifdef ENABLE_H2O
806
- }
807
-#endif
808
- }
809
-
810
-#ifdef ENABLE_H2O
811
- unless_h2o_rrdpush(rpt)
812
-#endif
813
- {
814
- // remove the non-blocking flag from the socket
815
- if(sock_delnonblock(rpt->fd) < 0)
816
- netdata_log_error("STREAM '%s' [receive from [%s]:%s]: "
817
- "cannot remove the non-blocking flag from socket %d"
818
- , rrdhost_hostname(rpt->host)
819
- , rpt->client_ip, rpt->client_port
820
- , rpt->fd);
821
-
822
- struct timeval timeout;
823
- timeout.tv_sec = 600;
824
- timeout.tv_usec = 0;
825
- if (unlikely(setsockopt(rpt->fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof timeout) != 0))
826
- netdata_log_error("STREAM '%s' [receive from [%s]:%s]: "
827
- "cannot set timeout for socket %d"
828
- , rrdhost_hostname(rpt->host)
829
- , rpt->client_ip, rpt->client_port
830
- , rpt->fd);
831
- }
832
-
833
- rrdpush_receive_log_status(
834
- rpt, "connected and ready to receive data",
835
- RRDPUSH_STATUS_CONNECTED, NDLP_INFO);
836
-
837
- // in case we have cloud connection we inform cloud
838
- // new child connected
839
- schedule_node_state_update(rpt->host, 300);
840
- rrdhost_set_is_parent_label();
841
-
842
- if (is_ephemeral)
843
- rrdhost_option_set(rpt->host, RRDHOST_OPTION_EPHEMERAL_HOST);
844
-
845
- // let it reconnect to parent immediately
846
- rrdpush_reset_destinations_postpone_time(rpt->host);
847
-
848
- // receive data
849
- size_t count = streaming_parser(rpt, &cd, rpt->fd, (rpt->ssl.conn) ? &rpt->ssl : NULL);
850
-
851
- // the parser stopped
852
- receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_EXIT, false);
853
-
854
- {
855
- char msg[100 + 1];
856
- snprintfz(msg, sizeof(msg) - 1, "disconnected (completed %zu updates)", count);
857
- rrdpush_receive_log_status(rpt, msg, RRDPUSH_STATUS_DISCONNECTED, NDLP_WARNING);
858
- }
859
-
860
- // in case we have cloud connection we inform cloud
861
- // a child disconnected
862
- STREAM_PATH tmp = rrdhost_stream_path_fetch(rpt->host);
863
- uint64_t total_reboot = (tmp.start_time + tmp.shutdown_time);
864
- schedule_node_state_update(rpt->host, MIN((total_reboot * MAX_CHILD_DISC_TOLERANCE), MAX_CHILD_DISC_DELAY));
865
-
866
-cleanup:
867
- ;
868
-}
869
-
870
-static bool stream_receiver_log_capabilities(BUFFER *wb, void *ptr) {
871
- struct receiver_state *rpt = ptr;
872
- if(!rpt)
873
- return false;
874
-
875
- stream_capabilities_to_string(wb, rpt->capabilities);
876
- return true;
877
-}
878
-
879
-static bool stream_receiver_log_transport(BUFFER *wb, void *ptr) {
880
- struct receiver_state *rpt = ptr;
881
- if(!rpt)
882
- return false;
883
-
884
- buffer_strcat(wb, SSL_connection(&rpt->ssl) ? "https" : "http");
885
- return true;
886
-}
887
-
888
-void *rrdpush_receiver_thread(void *ptr) {
889
- worker_register("STREAMRCV");
890
-
891
- worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_READ,
892
- "received bytes", "bytes/s",
893
- WORKER_METRIC_INCREMENT);
894
-
895
- worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED,
896
- "uncompressed bytes", "bytes/s",
897
- WORKER_METRIC_INCREMENT);
898
-
899
- worker_register_job_custom_metric(WORKER_RECEIVER_JOB_REPLICATION_COMPLETION,
900
- "replication completion", "%",
901
- WORKER_METRIC_ABSOLUTE);
902
-
903
- struct receiver_state *rpt = (struct receiver_state *) ptr;
904
- rpt->tid = gettid_cached();
905
-
906
- ND_LOG_STACK lgs[] = {
907
- ND_LOG_FIELD_TXT(NDF_SRC_IP, rpt->client_ip),
908
- ND_LOG_FIELD_TXT(NDF_SRC_PORT, rpt->client_port),
909
- ND_LOG_FIELD_TXT(NDF_NIDL_NODE, rpt->hostname),
910
- ND_LOG_FIELD_CB(NDF_SRC_TRANSPORT, stream_receiver_log_transport, rpt),
911
- ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_receiver_log_capabilities, rpt),
912
- ND_LOG_FIELD_END(),
913
- };
914
- ND_LOG_STACK_PUSH(lgs);
915
-
916
- netdata_log_info("STREAM %s [%s]:%s: receive thread started", rpt->hostname, rpt->client_ip
917
- , rpt->client_port);
918
-
919
- rrdpush_receive(rpt);
920
-
921
- netdata_log_info("STREAM '%s' [receive from [%s]:%s]: "
922
- "receive thread ended (task id %d)"
923
- , rpt->hostname ? rpt->hostname : "-"
924
- , rpt->client_ip ? rpt->client_ip : "-", rpt->client_port ? rpt->client_port : "-", gettid_cached());
925
-
926
- worker_unregister();
927
- rrdhost_clear_receiver(rpt);
928
- rrdhost_set_is_parent_label();
929
- receiver_state_free(rpt);
930
- return NULL;
931
-}
932
-
933
-int rrdpush_receiver_permission_denied(struct web_client *w) {
934
- // we always respond with the same message and error code
935
- // to prevent an attacker from gaining info about the error
936
- buffer_flush(w->response.data);
937
- buffer_strcat(w->response.data, START_STREAMING_ERROR_NOT_PERMITTED);
938
- return HTTP_RESP_UNAUTHORIZED;
939
-}
940
-
941
-int rrdpush_receiver_too_busy_now(struct web_client *w) {
942
- // we always respond with the same message and error code
943
- // to prevent an attacker from gaining info about the error
944
- buffer_flush(w->response.data);
945
- buffer_strcat(w->response.data, START_STREAMING_ERROR_BUSY_TRY_LATER);
946
- return HTTP_RESP_SERVICE_UNAVAILABLE;
947
-}
948
-
949
-static void rrdpush_receiver_takeover_web_connection(struct web_client *w, struct receiver_state *rpt) {
950
- rpt->fd = w->ifd;
951
-
952
- rpt->ssl.conn = w->ssl.conn;
953
- rpt->ssl.state = w->ssl.state;
954
-
955
- w->ssl = NETDATA_SSL_UNSET_CONNECTION;
956
-
957
- WEB_CLIENT_IS_DEAD(w);
958
-
959
- if(web_server_mode == WEB_SERVER_MODE_STATIC_THREADED) {
960
- web_client_flag_set(w, WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET);
961
- }
962
- else {
963
- if(w->ifd == w->ofd)
964
- w->ifd = w->ofd = -1;
965
- else
966
- w->ifd = -1;
967
- }
968
-
969
- buffer_flush(w->response.data);
970
-}
971
-
972
-int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_string, void *h2o_ctx __maybe_unused) {
973
-
974
- if(!service_running(ABILITY_STREAMING_CONNECTIONS))
975
- return rrdpush_receiver_too_busy_now(w);
976
-
977
- struct receiver_state *rpt = callocz(1, sizeof(*rpt));
978
- rpt->connected_since_s = now_realtime_sec();
979
- rpt->last_msg_t = now_monotonic_sec();
980
- rpt->hops = 1;
981
-
982
- rpt->capabilities = STREAM_CAP_INVALID;
983
-
984
-#ifdef ENABLE_H2O
985
- rpt->h2o_ctx = h2o_ctx;
986
-#endif
987
-
988
- __atomic_add_fetch(&netdata_buffers_statistics.rrdhost_receivers, sizeof(*rpt), __ATOMIC_RELAXED);
989
- __atomic_add_fetch(&netdata_buffers_statistics.rrdhost_allocations_size, sizeof(struct rrdhost_system_info), __ATOMIC_RELAXED);
990
-
991
- rpt->system_info = callocz(1, sizeof(struct rrdhost_system_info));
992
- rpt->system_info->hops = rpt->hops;
993
-
994
- rpt->fd = -1;
995
- rpt->client_ip = strdupz(w->client_ip);
996
- rpt->client_port = strdupz(w->client_port);
997
-
998
- rpt->ssl = NETDATA_SSL_UNSET_CONNECTION;
999
-
1000
- rpt->config.update_every = default_rrd_update_every;
1001
-
1002
- // parse the parameters and fill rpt and rpt->system_info
1003
-
1004
- while(decoded_query_string) {
1005
- char *value = strsep_skip_consecutive_separators(&decoded_query_string, "&");
1006
- if(!value || !*value) continue;
1007
-
1008
- char *name = strsep_skip_consecutive_separators(&value, "=");
1009
- if(!name || !*name) continue;
1010
- if(!value || !*value) continue;
1011
-
1012
- if(!strcmp(name, "key") && !rpt->key)
1013
- rpt->key = strdupz(value);
1014
-
1015
- else if(!strcmp(name, "hostname") && !rpt->hostname)
1016
- rpt->hostname = strdupz(value);
1017
-
1018
- else if(!strcmp(name, "registry_hostname") && !rpt->registry_hostname)
1019
- rpt->registry_hostname = strdupz(value);
1020
-
1021
- else if(!strcmp(name, "machine_guid") && !rpt->machine_guid)
1022
- rpt->machine_guid = strdupz(value);
1023
-
1024
- else if(!strcmp(name, "update_every"))
1025
- rpt->config.update_every = (int)strtoul(value, NULL, 0);
1026
-
1027
- else if(!strcmp(name, "os") && !rpt->os)
1028
- rpt->os = strdupz(value);
1029
-
1030
- else if(!strcmp(name, "timezone") && !rpt->timezone)
1031
- rpt->timezone = strdupz(value);
1032
-
1033
- else if(!strcmp(name, "abbrev_timezone") && !rpt->abbrev_timezone)
1034
- rpt->abbrev_timezone = strdupz(value);
1035
-
1036
- else if(!strcmp(name, "utc_offset"))
1037
- rpt->utc_offset = (int32_t)strtol(value, NULL, 0);
1038
-
1039
- else if(!strcmp(name, "hops"))
1040
- rpt->hops = rpt->system_info->hops = (uint16_t) strtoul(value, NULL, 0);
1041
-
1042
- else if(!strcmp(name, "ml_capable"))
1043
- rpt->system_info->ml_capable = strtoul(value, NULL, 0);
1044
-
1045
- else if(!strcmp(name, "ml_enabled"))
1046
- rpt->system_info->ml_enabled = strtoul(value, NULL, 0);
1047
-
1048
- else if(!strcmp(name, "mc_version"))
1049
- rpt->system_info->mc_version = strtoul(value, NULL, 0);
1050
-
1051
- else if(!strcmp(name, "ver") && (rpt->capabilities & STREAM_CAP_INVALID))
1052
- rpt->capabilities = convert_stream_version_to_capabilities(strtoul(value, NULL, 0), NULL, false);
1053
-
1054
- else {
1055
- // An old Netdata child does not have a compatible streaming protocol, map to something sane.
1056
- if (!strcmp(name, "NETDATA_SYSTEM_OS_NAME"))
1057
- name = "NETDATA_HOST_OS_NAME";
1058
-
1059
- else if (!strcmp(name, "NETDATA_SYSTEM_OS_ID"))
1060
- name = "NETDATA_HOST_OS_ID";
1061
-
1062
- else if (!strcmp(name, "NETDATA_SYSTEM_OS_ID_LIKE"))
1063
- name = "NETDATA_HOST_OS_ID_LIKE";
1064
-
1065
- else if (!strcmp(name, "NETDATA_SYSTEM_OS_VERSION"))
1066
- name = "NETDATA_HOST_OS_VERSION";
1067
-
1068
- else if (!strcmp(name, "NETDATA_SYSTEM_OS_VERSION_ID"))
1069
- name = "NETDATA_HOST_OS_VERSION_ID";
1070
-
1071
- else if (!strcmp(name, "NETDATA_SYSTEM_OS_DETECTION"))
1072
- name = "NETDATA_HOST_OS_DETECTION";
1073
-
1074
- else if(!strcmp(name, "NETDATA_PROTOCOL_VERSION") && (rpt->capabilities & STREAM_CAP_INVALID))
1075
- rpt->capabilities = convert_stream_version_to_capabilities(1, NULL, false);
1076
-
1077
- if (unlikely(rrdhost_set_system_info_variable(rpt->system_info, name, value))) {
1078
- nd_log_daemon(NDLP_NOTICE, "STREAM '%s' [receive from [%s]:%s]: "
1079
- "request has parameter '%s' = '%s', which is not used."
1080
- , (rpt->hostname && *rpt->hostname) ? rpt->hostname : "-"
1081
- , rpt->client_ip, rpt->client_port
1082
- , name, value);
1083
- }
1084
- }
1085
- }
1086
-
1087
- if (rpt->capabilities & STREAM_CAP_INVALID)
1088
- // no version is supplied, assume version 0;
1089
- rpt->capabilities = convert_stream_version_to_capabilities(0, NULL, false);
1090
-
1091
- // find the program name and version
1092
- if(w->user_agent && w->user_agent[0]) {
1093
- char *t = strchr(w->user_agent, '/');
1094
- if(t && *t) {
1095
- *t = '\0';
1096
- t++;
1097
- }
1098
-
1099
- rpt->program_name = strdupz(w->user_agent);
1100
- if(t && *t) rpt->program_version = strdupz(t);
1101
- }
1102
-
1103
- // check if we should accept this connection
1104
-
1105
- if(!rpt->key || !*rpt->key) {
1106
- rrdpush_receive_log_status(
1107
- rpt, "request without an API key, rejecting connection",
1108
- RRDPUSH_STATUS_NO_API_KEY, NDLP_WARNING);
1109
-
1110
- receiver_state_free(rpt);
1111
- return rrdpush_receiver_permission_denied(w);
1112
- }
1113
-
1114
- if(!rpt->hostname || !*rpt->hostname) {
1115
- rrdpush_receive_log_status(
1116
- rpt, "request without a hostname, rejecting connection",
1117
- RRDPUSH_STATUS_NO_HOSTNAME, NDLP_WARNING);
1118
-
1119
- receiver_state_free(rpt);
1120
- return rrdpush_receiver_permission_denied(w);
1121
- }
1122
-
1123
- if(!rpt->registry_hostname)
1124
- rpt->registry_hostname = strdupz(rpt->hostname);
1125
-
1126
- if(!rpt->machine_guid || !*rpt->machine_guid) {
1127
- rrdpush_receive_log_status(
1128
- rpt, "request without a machine GUID, rejecting connection",
1129
- RRDPUSH_STATUS_NO_MACHINE_GUID, NDLP_WARNING);
1130
-
1131
- receiver_state_free(rpt);
1132
- return rrdpush_receiver_permission_denied(w);
1133
- }
1134
-
1135
- {
1136
- char buf[GUID_LEN + 1];
1137
-
1138
- if (regenerate_guid(rpt->key, buf) == -1) {
1139
- rrdpush_receive_log_status(
1140
- rpt, "API key is not a valid UUID (use the command uuidgen to generate one)",
1141
- RRDPUSH_STATUS_INVALID_API_KEY, NDLP_WARNING);
1142
-
1143
- receiver_state_free(rpt);
1144
- return rrdpush_receiver_permission_denied(w);
1145
- }
1146
-
1147
- if (regenerate_guid(rpt->machine_guid, buf) == -1) {
1148
- rrdpush_receive_log_status(
1149
- rpt, "machine GUID is not a valid UUID",
1150
- RRDPUSH_STATUS_INVALID_MACHINE_GUID, NDLP_WARNING);
1151
-
1152
- receiver_state_free(rpt);
1153
- return rrdpush_receiver_permission_denied(w);
1154
- }
1155
- }
1156
-
1157
- const char *api_key_type = appconfig_get(&stream_config, rpt->key, "type", "api");
1158
- if(!api_key_type || !*api_key_type) api_key_type = "unknown";
1159
- if(strcmp(api_key_type, "api") != 0) {
1160
- rrdpush_receive_log_status(
1161
- rpt, "API key is a machine GUID",
1162
- RRDPUSH_STATUS_INVALID_API_KEY, NDLP_WARNING);
1163
-
1164
- receiver_state_free(rpt);
1165
- return rrdpush_receiver_permission_denied(w);
1166
- }
1167
-
1168
- if(!appconfig_get_boolean(&stream_config, rpt->key, "enabled", 0)) {
1169
- rrdpush_receive_log_status(
1170
- rpt, "API key is not enabled",
1171
- RRDPUSH_STATUS_API_KEY_DISABLED, NDLP_WARNING);
1172
-
1173
- receiver_state_free(rpt);
1174
- return rrdpush_receiver_permission_denied(w);
1175
- }
1176
-
1177
- {
1178
- SIMPLE_PATTERN *key_allow_from = simple_pattern_create(
1179
- appconfig_get(&stream_config, rpt->key, "allow from", "*"),
1180
- NULL, SIMPLE_PATTERN_EXACT, true);
1181
-
1182
- if(key_allow_from) {
1183
- if(!simple_pattern_matches(key_allow_from, w->client_ip)) {
1184
- simple_pattern_free(key_allow_from);
1185
-
1186
- rrdpush_receive_log_status(
1187
- rpt, "API key is not allowed from this IP",
1188
- RRDPUSH_STATUS_NOT_ALLOWED_IP, NDLP_WARNING);
1189
-
1190
- receiver_state_free(rpt);
1191
- return rrdpush_receiver_permission_denied(w);
1192
- }
1193
-
1194
- simple_pattern_free(key_allow_from);
1195
- }
1196
- }
1197
-
1198
- {
1199
- const char *machine_guid_type = appconfig_get(&stream_config, rpt->machine_guid, "type", "machine");
1200
- if (!machine_guid_type || !*machine_guid_type) machine_guid_type = "unknown";
1201
-
1202
- if (strcmp(machine_guid_type, "machine") != 0) {
1203
- rrdpush_receive_log_status(
1204
- rpt, "machine GUID is an API key",
1205
- RRDPUSH_STATUS_INVALID_MACHINE_GUID, NDLP_WARNING);
1206
-
1207
- receiver_state_free(rpt);
1208
- return rrdpush_receiver_permission_denied(w);
1209
- }
1210
- }
1211
-
1212
- if(!appconfig_get_boolean(&stream_config, rpt->machine_guid, "enabled", 1)) {
1213
- rrdpush_receive_log_status(
1214
- rpt, "machine GUID is not enabled",
1215
- RRDPUSH_STATUS_MACHINE_GUID_DISABLED, NDLP_WARNING);
1216
-
1217
- receiver_state_free(rpt);
1218
- return rrdpush_receiver_permission_denied(w);
1219
- }
1220
-
1221
- {
1222
- SIMPLE_PATTERN *machine_allow_from = simple_pattern_create(
1223
- appconfig_get(&stream_config, rpt->machine_guid, "allow from", "*"),
1224
- NULL, SIMPLE_PATTERN_EXACT, true);
1225
-
1226
- if(machine_allow_from) {
1227
- if(!simple_pattern_matches(machine_allow_from, w->client_ip)) {
1228
- simple_pattern_free(machine_allow_from);
1229
-
1230
- rrdpush_receive_log_status(
1231
- rpt, "machine GUID is not allowed from this IP",
1232
- RRDPUSH_STATUS_NOT_ALLOWED_IP, NDLP_WARNING);
1233
-
1234
- receiver_state_free(rpt);
1235
- return rrdpush_receiver_permission_denied(w);
1236
- }
1237
-
1238
- simple_pattern_free(machine_allow_from);
1239
- }
1240
- }
1241
-
1242
- if (strcmp(rpt->machine_guid, localhost->machine_guid) == 0) {
1243
-
1244
- rrdpush_receiver_takeover_web_connection(w, rpt);
1245
-
1246
- rrdpush_receive_log_status(
1247
- rpt, "machine GUID is my own",
1248
- RRDPUSH_STATUS_LOCALHOST, NDLP_DEBUG);
1249
-
1250
- char initial_response[HTTP_HEADER_SIZE + 1];
1251
- snprintfz(initial_response, HTTP_HEADER_SIZE, "%s", START_STREAMING_ERROR_SAME_LOCALHOST);
1252
-
1253
- if(send_timeout(
1254
- &rpt->ssl,
1255
- rpt->fd, initial_response, strlen(initial_response), 0, 60) != (ssize_t)strlen(initial_response)) {
1256
-
1257
- nd_log_daemon(NDLP_ERR, "STREAM '%s' [receive from [%s]:%s]: "
1258
- "failed to reply."
1259
- , rpt->hostname
1260
- , rpt->client_ip, rpt->client_port
1261
- );
1262
- }
1263
-
1264
- receiver_state_free(rpt);
1265
- return HTTP_RESP_OK;
1266
- }
1267
-
1268
- if(unlikely(web_client_streaming_rate_t > 0)) {
1269
- static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER;
1270
- static time_t last_stream_accepted_t = 0;
1271
-
1272
- time_t now = now_realtime_sec();
1273
- spinlock_lock(&spinlock);
1274
-
1275
- if(unlikely(last_stream_accepted_t == 0))
1276
- last_stream_accepted_t = now;
1277
-
1278
- if(now - last_stream_accepted_t < web_client_streaming_rate_t) {
1279
- spinlock_unlock(&spinlock);
1280
-
1281
- char msg[100 + 1];
1282
- snprintfz(msg, sizeof(msg) - 1,
1283
- "rate limit, will accept new connection in %ld secs",
1284
- (long)(web_client_streaming_rate_t - (now - last_stream_accepted_t)));
1285
-
1286
- rrdpush_receive_log_status(
1287
- rpt, msg,
1288
- RRDPUSH_STATUS_RATE_LIMIT, NDLP_NOTICE);
1289
-
1290
- receiver_state_free(rpt);
1291
- return rrdpush_receiver_too_busy_now(w);
1292
- }
1293
-
1294
- last_stream_accepted_t = now;
1295
- spinlock_unlock(&spinlock);
1296
- }
1297
-
1298
- /*
1299
- * Quick path for rejecting multiple connections. The lock taken is fine-grained - it only protects the receiver
1300
- * pointer within the host (if a host exists). This protects against multiple concurrent web requests hitting
1301
- * separate threads within the web-server and landing here. The lock guards the thread-shutdown sequence that
1302
- * detaches the receiver from the host. If the host is being created (first time-access) then we also use the
1303
- * lock to prevent race-hazard (two threads try to create the host concurrently, one wins and the other does a
1304
- * lookup to the now-attached structure).
1305
- */
1306
-
1307
- {
1308
- time_t age = 0;
1309
- bool receiver_stale = false;
1310
- bool receiver_working = false;
1311
-
1312
- rrd_rdlock();
1313
- RRDHOST *host = rrdhost_find_by_guid(rpt->machine_guid);
1314
- if (unlikely(host && rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))) /* Ignore archived hosts. */
1315
- host = NULL;
1316
-
1317
- if (host) {
1318
- spinlock_lock(&host->receiver_lock);
1319
- if (host->receiver) {
1320
- age = now_monotonic_sec() - host->receiver->last_msg_t;
1321
-
1322
- if (age < 30)
1323
- receiver_working = true;
1324
- else
1325
- receiver_stale = true;
1326
- }
1327
- spinlock_unlock(&host->receiver_lock);
1328
- }
1329
- rrd_rdunlock();
1330
-
1331
- if (receiver_stale && stop_streaming_receiver(host, STREAM_HANDSHAKE_DISCONNECT_STALE_RECEIVER)) {
1332
- // we stopped the receiver
1333
- // we can proceed with this connection
1334
- receiver_stale = false;
1335
-
1336
- nd_log_daemon(NDLP_NOTICE, "STREAM '%s' [receive from [%s]:%s]: "
1337
- "stopped previous stale receiver to accept this one."
1338
- , rpt->hostname
1339
- , rpt->client_ip, rpt->client_port
1340
- );
1341
- }
1342
-
1343
- if (receiver_working || receiver_stale) {
1344
- // another receiver is already connected
1345
- // try again later
1346
-
1347
- char msg[200 + 1];
1348
- snprintfz(msg, sizeof(msg) - 1,
1349
- "multiple connections for same host, "
1350
- "old connection was last used %ld secs ago%s",
1351
- age, receiver_stale ? " (signaled old receiver to stop)" : " (new connection not accepted)");
1352
-
1353
- rrdpush_receive_log_status(
1354
- rpt, msg,
1355
- RRDPUSH_STATUS_ALREADY_CONNECTED, NDLP_DEBUG);
1356
-
1357
- // Have not set WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET - caller should clean up
1358
- buffer_flush(w->response.data);
1359
- buffer_strcat(w->response.data, START_STREAMING_ERROR_ALREADY_STREAMING);
1360
- receiver_state_free(rpt);
1361
- return HTTP_RESP_CONFLICT;
1362
- }
1363
- }
1364
-
1365
- rrdpush_receiver_takeover_web_connection(w, rpt);
1366
-
1367
- char tag[NETDATA_THREAD_TAG_MAX + 1];
1368
- snprintfz(tag, NETDATA_THREAD_TAG_MAX, THREAD_TAG_STREAM_RECEIVER "[%s]", rpt->hostname);
1369
- tag[NETDATA_THREAD_TAG_MAX] = '\0';
1370
-
1371
- rpt->thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_DEFAULT, rrdpush_receiver_thread, (void *)rpt);
1372
- if(!rpt->thread) {
1373
- rrdpush_receive_log_status(
1374
- rpt, "can't create receiver thread",
1375
- RRDPUSH_STATUS_INTERNAL_SERVER_ERROR, NDLP_ERR);
1376
-
1377
- buffer_flush(w->response.data);
1378
- buffer_strcat(w->response.data, "Can't handle this request");
1379
- receiver_state_free(rpt);
1380
- return HTTP_RESP_INTERNAL_SERVER_ERROR;
1381
- }
1382
-
1383
- // prevent the caller from closing the streaming socket
1384
- return HTTP_RESP_OK;
1385
-}
src/streaming/receiver.h
deleted
-93
@@ -1,93 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#ifndef NETDATA_RECEIVER_H
4
-#define NETDATA_RECEIVER_H
5
-
6
-#include "libnetdata/libnetdata.h"
7
-#include "database/rrd.h"
8
-
9
-struct parser;
10
-
11
-struct receiver_state {
12
- RRDHOST *host;
13
- pid_t tid;
14
- ND_THREAD *thread;
15
- int fd;
16
- char *key;
17
- char *hostname;
18
- char *registry_hostname;
19
- char *machine_guid;
20
- char *os;
21
- char *timezone; // Unused?
22
- char *abbrev_timezone;
23
- int32_t utc_offset;
24
- char *client_ip; // Duplicated in pluginsd
25
- char *client_port; // Duplicated in pluginsd
26
- char *program_name; // Duplicated in pluginsd
27
- char *program_version;
28
- struct rrdhost_system_info *system_info;
29
- STREAM_CAPABILITIES capabilities;
30
- time_t last_msg_t;
31
- time_t connected_since_s;
32
-
33
- struct buffered_reader reader;
34
-
35
- uint16_t hops;
36
-
37
- struct {
38
- bool shutdown; // signal the streaming parser to exit
39
- STREAM_HANDSHAKE reason;
40
- } exit;
41
-
42
- struct {
43
- RRD_MEMORY_MODE mode;
44
- int history;
45
- int update_every;
46
- int health_enabled; // CONFIG_BOOLEAN_YES, CONFIG_BOOLEAN_NO, CONFIG_BOOLEAN_AUTO
47
- time_t alarms_delay;
48
- uint32_t alarms_history;
49
- int rrdpush_enabled;
50
- const char *rrdpush_api_key; // DONT FREE - it is allocated in appconfig
51
- const char *rrdpush_send_charts_matching; // DONT FREE - it is allocated in appconfig
52
- bool rrdpush_enable_replication;
53
- time_t rrdpush_seconds_to_replicate;
54
- time_t rrdpush_replication_step;
55
- const char *rrdpush_destination; // DONT FREE - it is allocated in appconfig
56
- unsigned int rrdpush_compression;
57
- STREAM_CAPABILITIES compression_priorities[COMPRESSION_ALGORITHM_MAX];
58
- } config;
59
-
60
- NETDATA_SSL ssl;
61
-
62
- time_t replication_first_time_t;
63
-
64
- struct decompressor_state decompressor;
65
- /*
66
- struct {
67
- uint32_t count;
68
- STREAM_NODE_INSTANCE *array;
69
- } instances;
70
-*/
71
-
72
- // The parser pointer is safe to read and use, only when having the host receiver lock.
73
- // Without this lock, the data pointed by the pointer may vanish randomly.
74
- // Also, since the receiver sets it when it starts, it should be read with
75
- // an atomic read.
76
- struct parser *parser;
77
-
78
-#ifdef ENABLE_H2O
79
- void *h2o_ctx;
80
-#endif
81
-};
82
-
83
-#ifdef ENABLE_H2O
84
-#define is_h2o_rrdpush(x) ((x)->h2o_ctx != NULL)
85
-#define unless_h2o_rrdpush(x) if(!is_h2o_rrdpush(x))
86
-#endif
87
-
88
-int rrdpush_receiver_thread_spawn(struct web_client *w, char *decoded_query_string, void *h2o_ctx);
89
-
90
-void receiver_state_free(struct receiver_state *rpt);
91
-bool stop_streaming_receiver(RRDHOST *host, STREAM_HANDSHAKE reason);
92
-
93
-#endif //NETDATA_RECEIVER_H
src/streaming/replication.c
+91
-80
@@ -1,5 +1,7 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
+#include "stream-receiver-internals.h"
4
+#include "stream-sender-internals.h"
5
#include "replication.h"
6
#include "Judy.h"
7
@@ -31,6 +33,9 @@
33
#define ITERATIONS_IDLE_WITHOUT_PENDING_TO_RUN_SENDER_VERIFICATION 30
34
#define SECONDS_TO_RESET_POINT_IN_TIME 10
35
36
+#define MAX_REPLICATION_THREADS 32
37
+#define REQUESTS_AHEAD_PER_THREAD 1 // 1 = enable synchronous queries
38
+
39
static struct replication_query_statistics replication_queries = {
40
.spinlock = NETDATA_SPINLOCK_INITIALIZER,
41
.queries_started = 0,
@@ -114,7 +119,8 @@ static struct replication_query *replication_query_prepare(
119
time_t query_before,
120
bool query_enable_streaming,
121
time_t wall_clock_time,
117
- STREAM_CAPABILITIES capabilities
122
+ STREAM_CAPABILITIES capabilities,
123
+ bool synchronous
124
) {
125
size_t dimensions = rrdset_number_of_dimensions(st);
126
struct replication_query *q = callocz(1, sizeof(struct replication_query) + dimensions * sizeof(struct replication_dimension));
@@ -184,8 +190,11 @@ static struct replication_query *replication_query_prepare(
190
d->rda = dictionary_acquired_item_dup(rd_dfe.dict, rd_dfe.item);
191
d->rd = rd;
192
187
- storage_engine_query_init(q->backend, rd->tiers[0].smh, &d->handle, q->query.after, q->query.before,
188
- q->query.locked_data_collection ? STORAGE_PRIORITY_HIGH : STORAGE_PRIORITY_LOW);
193
+ STORAGE_PRIORITY priority = q->query.locked_data_collection ? STORAGE_PRIORITY_HIGH : STORAGE_PRIORITY_LOW;
194
+ if(synchronous) priority = STORAGE_PRIORITY_SYNCHRONOUS;
195
+
196
+ storage_engine_query_init(q->backend, rd->tiers[0].smh, &d->handle,
197
+ q->query.after, q->query.before, priority);
198
d->enabled = true;
199
d->skip = false;
200
count++;
@@ -223,7 +232,7 @@ static void replication_send_chart_collection_state(BUFFER *wb, RRDSET *st, STRE
232
233
if(with_slots) {
234
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
226
- buffer_print_uint64_encoded(wb, integer_encoding, rd->rrdpush.sender.dim_slot);
235
+ buffer_print_uint64_encoded(wb, integer_encoding, rd->stream.snd.dim_slot);
236
}
237
238
buffer_fast_strcat(wb, " '", 2);
@@ -457,7 +466,7 @@ static bool replication_query_execute(BUFFER *wb, struct replication_query *q, s
466
467
if(with_slots) {
468
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
460
- buffer_print_uint64_encoded(wb, integer_encoding, q->st->rrdpush.sender.chart_slot);
469
+ buffer_print_uint64_encoded(wb, integer_encoding, q->st->stream.snd.chart_slot);
470
}
471
472
buffer_fast_strcat(wb, " '' ", 4);
@@ -482,7 +491,7 @@ static bool replication_query_execute(BUFFER *wb, struct replication_query *q, s
491
492
if(with_slots) {
493
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
485
- buffer_print_uint64_encoded(wb, integer_encoding, d->rd->rrdpush.sender.dim_slot);
494
+ buffer_print_uint64_encoded(wb, integer_encoding, d->rd->stream.snd.dim_slot);
495
}
496
497
buffer_fast_strcat(wb, " \"", 2);
@@ -546,7 +555,8 @@ static struct replication_query *replication_response_prepare(
555
bool requested_enable_streaming,
556
time_t requested_after,
557
time_t requested_before,
549
- STREAM_CAPABILITIES capabilities
558
+ STREAM_CAPABILITIES capabilities,
559
+ bool synchronous
560
) {
561
time_t wall_clock_time = now_realtime_sec();
562
@@ -608,7 +618,7 @@ static struct replication_query *replication_response_prepare(
618
db_first_entry, db_last_entry,
619
requested_after, requested_before, requested_enable_streaming,
620
query_after, query_before, query_enable_streaming,
611
- wall_clock_time, capabilities);
621
+ wall_clock_time, capabilities, synchronous);
622
}
623
624
void replication_response_cancel_and_finalize(struct replication_query *q) {
@@ -618,7 +628,7 @@ void replication_response_cancel_and_finalize(struct replication_query *q) {
628
629
static bool sender_is_still_connected_for_this_request(struct replication_request *rq);
630
621
-bool replication_response_execute_and_finalize(struct replication_query *q, size_t max_msg_size) {
631
+bool replication_response_execute_and_finalize(struct replication_query *q, size_t max_msg_size, bool workers) {
632
bool with_slots = (q->query.capabilities & STREAM_CAP_SLOTS) ? true : false;
633
NUMBER_ENCODING integer_encoding = (q->query.capabilities & STREAM_CAP_IEEE754) ? NUMBER_ENCODING_BASE64 : NUMBER_ENCODING_DECIMAL;
634
struct replication_request *rq = q->rq;
@@ -628,13 +638,13 @@ bool replication_response_execute_and_finalize(struct replication_query *q, size
638
// we might want to optimize this by filling a temporary buffer
639
// and copying the result to the host's buffer in order to avoid
640
// holding the host's buffer lock for too long
631
- BUFFER *wb = sender_start(host->sender);
641
+ BUFFER *wb = sender_thread_buffer(host->sender);
642
643
buffer_fast_strcat(wb, PLUGINSD_KEYWORD_REPLAY_BEGIN, sizeof(PLUGINSD_KEYWORD_REPLAY_BEGIN) - 1);
644
645
if(with_slots) {
646
buffer_fast_strcat(wb, " "PLUGINSD_KEYWORD_SLOT":", sizeof(PLUGINSD_KEYWORD_SLOT) - 1 + 2);
637
- buffer_print_uint64_encoded(wb, integer_encoding, q->st->rrdpush.sender.chart_slot);
647
+ buffer_print_uint64_encoded(wb, integer_encoding, q->st->stream.snd.chart_slot);
648
}
649
650
buffer_fast_strcat(wb, " '", 2);
@@ -679,9 +689,9 @@ bool replication_response_execute_and_finalize(struct replication_query *q, size
689
buffer_print_uint64_encoded(wb, integer_encoding, wall_clock_time);
690
buffer_fast_strcat(wb, "\n", 1);
691
682
- worker_is_busy(WORKER_JOB_BUFFER_COMMIT);
692
+ if(workers) worker_is_busy(WORKER_JOB_BUFFER_COMMIT);
693
sender_commit(host->sender, wb, STREAM_TRAFFIC_TYPE_REPLICATION);
684
- worker_is_busy(WORKER_JOB_CLEANUP);
694
+ if(workers) worker_is_busy(WORKER_JOB_CLEANUP);
695
696
if(enable_streaming) {
697
if(sender_is_still_connected_for_this_request(rq)) {
@@ -694,7 +704,7 @@ bool replication_response_execute_and_finalize(struct replication_query *q, size
704
rrdhost_sender_replicating_charts_minus_one(st->rrdhost);
705
706
if(!finished_with_gap)
697
- st->rrdpush.sender.resync_time_s = 0;
707
+ st->stream.snd.resync_time_s = 0;
708
709
#ifdef NETDATA_LOG_REPLICATION_REQUESTS
710
internal_error(true, "STREAM_SENDER REPLAY: 'host:%s/chart:%s' streaming starts",
@@ -892,7 +902,7 @@ bool replicate_chart_request(send_command callback, struct parser *parser, RRDHO
902
r.gap.from = r.local_db.last_entry_t;
903
else
904
// we don't have any data, the gap is the max timeframe we are allowed to replicate
895
- r.gap.from = r.local_db.wall_clock_time - r.host->rrdpush_seconds_to_replicate;
905
+ r.gap.from = r.local_db.wall_clock_time - r.host->stream.replication.period;
906
907
}
908
else {
@@ -937,9 +947,9 @@ bool replicate_chart_request(send_command callback, struct parser *parser, RRDHO
947
// ok, the child can fill the entire gap we have
948
r.wanted.after = r.gap.from;
949
940
- if(r.gap.to - r.wanted.after > host->rrdpush_replication_step)
950
+ if(r.gap.to - r.wanted.after > host->stream.replication.step)
951
// the duration is too big for one request - let's take the first step
942
- r.wanted.before = r.wanted.after + host->rrdpush_replication_step;
952
+ r.wanted.before = r.wanted.after + host->stream.replication.step;
953
else
954
// wow, we can do it in one request
955
r.wanted.before = r.gap.to;
@@ -956,7 +966,7 @@ bool replicate_chart_request(send_command callback, struct parser *parser, RRDHO
966
}
967
968
// the child should start streaming immediately if the wanted duration is small, or we reached the last entry of the child
959
- r.wanted.start_streaming = (r.local_db.wall_clock_time - r.wanted.after <= host->rrdpush_replication_step ||
969
+ r.wanted.start_streaming = (r.local_db.wall_clock_time - r.wanted.after <= host->stream.replication.step ||
970
r.wanted.before >= r.child_db.last_entry_t ||
971
r.wanted.before >= r.child_db.wall_clock_time ||
972
r.wanted.before >= r.local_db.wall_clock_time);
@@ -1000,8 +1010,6 @@ struct replication_sort_entry {
1010
size_t unique_id; // used as a key to identify the sort entry - we never access its contents
1011
};
1012
1003
-#define MAX_REPLICATION_THREADS 20 // + 1 for the main thread
1004
-
1013
// the global variables for the replication thread
1014
static struct replication_thread {
1015
ARAL *aral_rse;
@@ -1130,7 +1138,7 @@ static inline struct replication_sort_entry *replication_sort_entry_create(struc
1138
struct replication_sort_entry *rse = aral_mallocz(replication_globals.aral_rse);
1139
__atomic_add_fetch(&replication_globals.atomic.memory, sizeof(struct replication_sort_entry), __ATOMIC_RELAXED);
1140
1133
- rrdpush_sender_pending_replication_requests_plus_one(rq->sender);
1141
+ stream_sender_pending_replication_requests_plus_one(rq->sender);
1142
1143
// copy the request
1144
rse->rq = rq;
@@ -1150,7 +1158,7 @@ static void replication_sort_entry_destroy(struct replication_sort_entry *rse) {
1158
}
1159
1160
static void replication_sort_entry_add(struct replication_request *rq) {
1153
- if(unlikely(rrdpush_sender_replication_buffer_full_get(rq->sender))) {
1161
+ if(unlikely(stream_sender_replication_buffer_full_get(rq->sender))) {
1162
rq->indexed_in_judy = false;
1163
rq->not_indexed_buffer_full = true;
1164
rq->not_indexed_preprocessing = false;
@@ -1218,7 +1226,7 @@ static bool replication_sort_entry_unlink_and_free_unsafe(struct replication_sor
1226
replication_globals.unsafe.removed++;
1227
replication_globals.unsafe.pending--;
1228
1221
- rrdpush_sender_pending_replication_requests_minus_one(rse->rq->sender);
1229
+ stream_sender_pending_replication_requests_minus_one(rse->rq->sender);
1230
1231
rse->rq->indexed_in_judy = false;
1232
rse->rq->not_indexed_preprocessing = preprocessing;
@@ -1359,7 +1367,7 @@ static void replication_request_react_callback(const DICTIONARY_ITEM *item __may
1367
replication_sort_entry_add(rq);
1368
1369
// this request is about a unique chart for this sender
1362
- rrdpush_sender_replicating_charts_plus_one(s);
1370
+ stream_sender_replicating_charts_plus_one(s);
1371
}
1372
1373
static bool replication_request_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *old_value, void *new_value, void *sender_state) {
@@ -1411,7 +1419,7 @@ static void replication_request_delete_callback(const DICTIONARY_ITEM *item __ma
1419
struct replication_request *rq = value;
1420
1421
// this request is about a unique chart for this sender
1414
- rrdpush_sender_replicating_charts_minus_one(rq->sender);
1422
+ stream_sender_replicating_charts_minus_one(rq->sender);
1423
1424
if(rq->indexed_in_judy)
1425
replication_sort_entry_del(rq, false);
@@ -1426,16 +1434,14 @@ static void replication_request_delete_callback(const DICTIONARY_ITEM *item __ma
1434
}
1435
1436
static bool sender_is_still_connected_for_this_request(struct replication_request *rq) {
1429
- return rq->sender_last_flush_ut == rrdpush_sender_get_flush_time(rq->sender);
1437
+ return rq->sender_last_flush_ut == stream_sender_get_flush_time(rq->sender);
1438
}
1439
1440
static bool replication_execute_request(struct replication_request *rq, bool workers) {
1441
bool ret = false;
1442
1443
if(!rq->st) {
1436
- if(likely(workers))
1437
- worker_is_busy(WORKER_JOB_FIND_CHART);
1438
-
1444
+ if(likely(workers)) worker_is_busy(WORKER_JOB_FIND_CHART);
1445
rq->st = rrdset_find(rq->sender->host, string2str(rq->chart_id));
1446
}
1447
@@ -1447,24 +1453,21 @@ static bool replication_execute_request(struct replication_request *rq, bool wor
1453
}
1454
1455
if(!rq->q) {
1450
- if(likely(workers))
1451
- worker_is_busy(WORKER_JOB_PREPARE_QUERY);
1452
-
1456
+ if(likely(workers)) worker_is_busy(WORKER_JOB_PREPARE_QUERY);
1457
rq->q = replication_response_prepare(
1458
rq->st,
1459
rq->start_streaming,
1460
rq->after,
1461
rq->before,
1458
- rq->sender->capabilities);
1462
+ rq->sender->capabilities, true);
1463
}
1464
1461
- if(likely(workers))
1462
- worker_is_busy(WORKER_JOB_QUERYING);
1465
+ if(likely(workers)) worker_is_busy(WORKER_JOB_QUERYING);
1466
1467
// send the replication data
1468
rq->q->rq = rq;
1469
replication_response_execute_and_finalize(
1467
- rq->q, (size_t)((unsigned long long)rq->sender->host->sender->buffer->max_size * MAX_REPLICATION_MESSAGE_PERCENT_SENDER_BUFFER / 100ULL));
1470
+ rq->q, (size_t)((unsigned long long)rq->sender->host->sender->sbuf.cb->max_size * MAX_REPLICATION_MESSAGE_PERCENT_SENDER_BUFFER / 100ULL), workers);
1471
1472
rq->q = NULL;
1473
@@ -1493,7 +1496,7 @@ void replication_add_request(struct sender_state *sender, const char *chart_id,
1496
.after = after,
1497
.before = before,
1498
.start_streaming = start_streaming,
1496
- .sender_last_flush_ut = rrdpush_sender_get_flush_time(sender),
1499
+ .sender_last_flush_ut = stream_sender_get_flush_time(sender),
1500
.indexed_in_judy = false,
1501
.not_indexed_buffer_full = false,
1502
.not_indexed_preprocessing = false,
@@ -1502,19 +1505,20 @@ void replication_add_request(struct sender_state *sender, const char *chart_id,
1505
if(!sender->replication.oldest_request_after_t || rq.after < sender->replication.oldest_request_after_t)
1506
sender->replication.oldest_request_after_t = rq.after;
1507
1505
- if(start_streaming && rrdpush_sender_get_buffer_used_percent(sender) <= STREAMING_START_MAX_SENDER_BUFFER_PERCENTAGE_ALLOWED)
1506
- replication_execute_request(&rq, false);
1507
-
1508
- else
1509
- dictionary_set(sender->replication.requests, chart_id, &rq, sizeof(struct replication_request));
1508
+// if(start_streaming && rrdpush_sender_get_buffer_used_percent(sender) <= STREAMING_START_MAX_SENDER_BUFFER_PERCENTAGE_ALLOWED)
1509
+// replication_execute_request(&rq, false);
1510
+//
1511
+// else
1512
+ dictionary_set(sender->replication.requests, chart_id, &rq, sizeof(struct replication_request));
1513
}
1514
1515
void replication_sender_delete_pending_requests(struct sender_state *sender) {
1516
// allow the dictionary destructor to go faster on locks
1517
dictionary_flush(sender->replication.requests);
1518
+ sender->replication.oldest_request_after_t = 0;
1519
}
1520
1517
-void replication_init_sender(struct sender_state *sender) {
1521
+void replication_sender_init(struct sender_state *sender) {
1522
sender->replication.requests = dictionary_create_advanced(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
1523
NULL, sizeof(struct replication_request));
1524
@@ -1531,11 +1535,11 @@ void replication_cleanup_sender(struct sender_state *sender) {
1535
}
1536
1537
void replication_recalculate_buffer_used_ratio_unsafe(struct sender_state *s) {
1534
- size_t available = cbuffer_available_size_unsafe(s->host->sender->buffer);
1535
- size_t percentage = (s->buffer->max_size - available) * 100 / s->buffer->max_size;
1538
+ size_t available = cbuffer_available_size_unsafe(s->host->sender->sbuf.cb);
1539
+ size_t percentage = (s->sbuf.cb->max_size - available) * 100 / s->sbuf.cb->max_size;
1540
1537
- if(unlikely(percentage > MAX_SENDER_BUFFER_PERCENTAGE_ALLOWED && !rrdpush_sender_replication_buffer_full_get(s))) {
1538
- rrdpush_sender_replication_buffer_full_set(s, true);
1541
+ if(unlikely(percentage > MAX_SENDER_BUFFER_PERCENTAGE_ALLOWED && !stream_sender_replication_buffer_full_get(s))) {
1542
+ stream_sender_replication_buffer_full_set(s, true);
1543
1544
struct replication_request *rq;
1545
dfe_start_read(s->replication.requests, rq) {
@@ -1548,8 +1552,8 @@ void replication_recalculate_buffer_used_ratio_unsafe(struct sender_state *s) {
1552
replication_globals.unsafe.senders_full++;
1553
replication_recursive_unlock();
1554
}
1551
- else if(unlikely(percentage < MIN_SENDER_BUFFER_PERCENTAGE_ALLOWED && rrdpush_sender_replication_buffer_full_get(s))) {
1552
- rrdpush_sender_replication_buffer_full_set(s, false);
1555
+ else if(unlikely(percentage < MIN_SENDER_BUFFER_PERCENTAGE_ALLOWED && stream_sender_replication_buffer_full_get(s))) {
1556
+ stream_sender_replication_buffer_full_set(s, false);
1557
1558
struct replication_request *rq;
1559
dfe_start_read(s->replication.requests, rq) {
@@ -1565,7 +1569,7 @@ void replication_recalculate_buffer_used_ratio_unsafe(struct sender_state *s) {
1569
replication_recursive_unlock();
1570
}
1571
1568
- rrdpush_sender_set_buffer_used_percent(s, percentage);
1572
+ stream_sender_set_buffer_used_percent(s, percentage);
1573
}
1574
1575
// ----------------------------------------------------------------------------
@@ -1574,11 +1578,11 @@ void replication_recalculate_buffer_used_ratio_unsafe(struct sender_state *s) {
1578
static size_t verify_host_charts_are_streaming_now(RRDHOST *host) {
1579
internal_error(
1580
host->sender &&
1577
- !rrdpush_sender_pending_replication_requests(host->sender) &&
1581
+ !stream_sender_pending_replication_requests(host->sender) &&
1582
dictionary_entries(host->sender->replication.requests) != 0,
1583
"REPLICATION SUMMARY: 'host:%s' reports %zu pending replication requests, but its chart replication index says there are %zu charts pending replication",
1584
rrdhost_hostname(host),
1581
- rrdpush_sender_pending_replication_requests(host->sender),
1585
+ stream_sender_pending_replication_requests(host->sender),
1586
dictionary_entries(host->sender->replication.requests)
1587
);
1588
@@ -1720,14 +1724,7 @@ static int replication_pipeline_execute_next(void) {
1724
struct replication_request *rq;
1725
1726
if(unlikely(!rtp.rqs)) {
1723
- rtp.max_requests_ahead = (int)get_netdata_cpus() / 2;
1724
-
1725
- if(rtp.max_requests_ahead > libuv_worker_threads * 2)
1726
- rtp.max_requests_ahead = libuv_worker_threads * 2;
1727
-
1728
- if(rtp.max_requests_ahead < 2)
1729
- rtp.max_requests_ahead = 2;
1730
-
1727
+ rtp.max_requests_ahead = REQUESTS_AHEAD_PER_THREAD;
1728
rtp.rqs = callocz(rtp.max_requests_ahead, sizeof(struct replication_request));
1729
__atomic_add_fetch(&replication_buffers_allocated, rtp.max_requests_ahead * sizeof(struct replication_request), __ATOMIC_RELAXED);
1730
}
@@ -1755,11 +1752,12 @@ static int replication_pipeline_execute_next(void) {
1752
if (rq->st && !rq->q) {
1753
worker_is_busy(WORKER_JOB_PREPARE_QUERY);
1754
rq->q = replication_response_prepare(
1758
- rq->st,
1759
- rq->start_streaming,
1760
- rq->after,
1761
- rq->before,
1762
- rq->sender->capabilities);
1755
+ rq->st,
1756
+ rq->start_streaming,
1757
+ rq->after,
1758
+ rq->before,
1759
+ rq->sender->capabilities,
1760
+ rtp.max_requests_ahead == 1);
1761
}
1762
1763
rq->executed = false;
@@ -1777,7 +1775,7 @@ static int replication_pipeline_execute_next(void) {
1775
if(rq->found) {
1776
internal_fatal(rq->executed, "REPLAY FATAL: query has already been executed!");
1777
1780
- if (rq->sender_last_flush_ut != rrdpush_sender_get_flush_time(rq->sender)) {
1778
+ if (rq->sender_last_flush_ut != stream_sender_get_flush_time(rq->sender)) {
1779
// the sender has reconnected since this request was queued,
1780
// we can safely throw it away, since the parent will resend it
1781
replication_response_cancel_and_finalize(rq->q);
@@ -1785,7 +1783,7 @@ static int replication_pipeline_execute_next(void) {
1783
rq->found = false;
1784
rq->q = NULL;
1785
}
1788
- else if (rrdpush_sender_replication_buffer_full_get(rq->sender)) {
1786
+ else if (stream_sender_replication_buffer_full_get(rq->sender)) {
1787
// the sender buffer is full, so we can ignore this request,
1788
// it has already been marked as 'preprocessed' in the dictionary,
1789
// and the sender will put it back in when there is
@@ -1840,7 +1838,7 @@ static void *replication_worker_thread(void *ptr __maybe_unused) {
1838
1839
while (service_running(SERVICE_REPLICATION)) {
1840
if (unlikely(replication_pipeline_execute_next() == REQUEST_QUEUE_EMPTY)) {
1843
- sender_thread_buffer_free();
1841
+ sender_commit_thread_buffer_free();
1842
worker_is_busy(WORKER_JOB_WAIT);
1843
worker_is_idle();
1844
sleep_usec(1 * USEC_PER_SEC);
@@ -1867,8 +1865,10 @@ static void replication_main_cleanup(void *pptr) {
1865
replication_globals.main_thread.threads_ptrs = NULL;
1866
__atomic_sub_fetch(&replication_buffers_allocated, threads * sizeof(ND_THREAD *), __ATOMIC_RELAXED);
1867
1870
- aral_destroy(replication_globals.aral_rse);
1871
- replication_globals.aral_rse = NULL;
1868
+ // we should not destroy aral on exit
1869
+ // the sender threads may still be working on flushing senders replication requests
1870
+ //aral_destroy(replication_globals.aral_rse);
1871
+ //replication_globals.aral_rse = NULL;
1872
1873
// custom code
1874
worker_unregister();
@@ -1877,20 +1877,34 @@ static void replication_main_cleanup(void *pptr) {
1877
}
1878
1879
void replication_initialize(void) {
1880
- replication_globals.aral_rse = aral_create("rse", sizeof(struct replication_sort_entry),
1881
- 0, 65536, aral_by_size_statistics(),
1882
- NULL, NULL, false, false);
1880
+ replication_globals.aral_rse = aral_by_size_acquire(sizeof(struct replication_sort_entry));
1881
}
1882
1885
-void *replication_thread_main(void *ptr __maybe_unused) {
1883
+void *replication_thread_main(void *ptr) {
1884
+ CLEANUP_FUNCTION_REGISTER(replication_main_cleanup) cleanup_ptr = ptr;
1885
+
1886
replication_initialize_workers(true);
1887
1888
- int threads = config_get_number(CONFIG_SECTION_DB, "replication threads", 1);
1889
- if(threads < 1 || threads > MAX_REPLICATION_THREADS) {
1888
+ int nodes = (int)dictionary_entries(rrdhost_root_index);
1889
+ int cpus = (int)get_netdata_cpus();
1890
+ int threads = MIN(cpus * 2 / 3, nodes / 5);
1891
+ if (threads < 1) threads = 1;
1892
+ else if (threads > MAX_REPLICATION_THREADS) threads = MAX_REPLICATION_THREADS;
1893
+
1894
+ threads = config_get_number(CONFIG_SECTION_DB, "replication threads", threads);
1895
+ if(threads < 1) {
1896
netdata_log_error("replication threads given %d is invalid, resetting to 1", threads);
1897
threads = 1;
1898
+ config_set_number(CONFIG_SECTION_DB, "replication threads", threads);
1899
+ }
1900
+ else if(threads > MAX_REPLICATION_THREADS) {
1901
+ netdata_log_error("replication threads given %d is invalid, resetting to %d", threads, (int)MAX_REPLICATION_THREADS);
1902
+ threads = MAX_REPLICATION_THREADS;
1903
+ config_set_number(CONFIG_SECTION_DB, "replication threads", threads);
1904
}
1905
1906
+ netdata_log_info("replication threads set to %d (cpu cores = %d, nodes = %d)", threads, cpus, nodes);
1907
+
1908
if(--threads) {
1909
replication_globals.main_thread.threads = threads;
1910
replication_globals.main_thread.threads_ptrs = mallocz(threads * sizeof(ND_THREAD *));
@@ -1906,8 +1920,6 @@ void *replication_thread_main(void *ptr __maybe_unused) {
1920
}
1921
}
1922
1909
- CLEANUP_FUNCTION_REGISTER(replication_main_cleanup) cleanup_ptr = ptr;
1910
-
1923
// start from 100% completed
1924
worker_set_metric(WORKER_JOB_CUSTOM_METRIC_COMPLETION, 100.0);
1925
@@ -1982,7 +1994,6 @@ void *replication_thread_main(void *ptr __maybe_unused) {
1994
}
1995
1996
if(unlikely(replication_pipeline_execute_next() == REQUEST_QUEUE_EMPTY)) {
1985
-
1997
worker_is_busy(WORKER_JOB_WAIT);
1998
replication_recursive_lock();
1999
@@ -1992,7 +2003,7 @@ void *replication_thread_main(void *ptr __maybe_unused) {
2003
if(slow) {
2004
// no work to be done, wait for a request to come in
2005
timeout = 1000 * USEC_PER_MS;
1995
- sender_thread_buffer_free();
2006
+ sender_commit_thread_buffer_free();
2007
}
2008
2009
else if(replication_globals.unsafe.pending > 0) {
src/streaming/replication.h
+1
-1
@@ -26,7 +26,7 @@ bool replicate_chart_request(send_command callback, struct parser *parser,
26
time_t child_first_entry, time_t child_last_entry, time_t child_wall_clock_time,
27
time_t response_first_start_time, time_t response_last_end_time);
28
29
-void replication_init_sender(struct sender_state *sender);
29
+void replication_sender_init(struct sender_state *sender);
30
void replication_cleanup_sender(struct sender_state *sender);
31
void replication_sender_delete_pending_requests(struct sender_state *sender);
32
void replication_add_request(struct sender_state *sender, const char *chart_id, time_t after, time_t before, bool start_streaming);
src/streaming/rrdhost-status.c
+126
-181
@@ -1,135 +1,82 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "rrdhost-status.h"
4
-
5
-const char *rrdhost_db_status_to_string(RRDHOST_DB_STATUS status) {
6
- switch(status) {
7
- default:
8
- case RRDHOST_DB_STATUS_INITIALIZING:
9
- return "initializing";
10
-
11
- case RRDHOST_DB_STATUS_QUERYABLE:
12
- return "online";
13
- }
14
-}
15
-
16
-const char *rrdhost_db_liveness_to_string(RRDHOST_DB_LIVENESS status) {
17
- switch(status) {
18
- default:
19
- case RRDHOST_DB_LIVENESS_STALE:
20
- return "stale";
21
-
22
- case RRDHOST_DB_LIVENESS_LIVE:
23
- return "live";
24
- }
25
-}
26
-
27
-const char *rrdhost_ingest_status_to_string(RRDHOST_INGEST_STATUS status) {
28
- switch(status) {
29
- case RRDHOST_INGEST_STATUS_ARCHIVED:
30
- return "archived";
31
-
32
- case RRDHOST_INGEST_STATUS_INITIALIZING:
33
- return "initializing";
34
-
35
- case RRDHOST_INGEST_STATUS_REPLICATING:
36
- return "replicating";
37
-
38
- case RRDHOST_INGEST_STATUS_ONLINE:
39
- return "online";
40
-
41
- default:
42
- case RRDHOST_INGEST_STATUS_OFFLINE:
43
- return "offline";
44
- }
45
-}
46
-
47
-const char *rrdhost_ingest_type_to_string(RRDHOST_INGEST_TYPE type) {
48
- switch(type) {
49
- case RRDHOST_INGEST_TYPE_LOCALHOST:
50
- return "localhost";
51
-
52
- case RRDHOST_INGEST_TYPE_VIRTUAL:
53
- return "virtual";
54
-
55
- case RRDHOST_INGEST_TYPE_CHILD:
56
- return "child";
57
-
58
- default:
59
- case RRDHOST_INGEST_TYPE_ARCHIVED:
60
- return "archived";
61
- }
62
-}
63
-
64
-const char *rrdhost_streaming_status_to_string(RRDHOST_STREAMING_STATUS status) {
65
- switch(status) {
66
- case RRDHOST_STREAM_STATUS_DISABLED:
67
- return "disabled";
68
-
69
- case RRDHOST_STREAM_STATUS_REPLICATING:
70
- return "replicating";
71
-
72
- case RRDHOST_STREAM_STATUS_ONLINE:
73
- return "online";
74
-
75
- default:
76
- case RRDHOST_STREAM_STATUS_OFFLINE:
77
- return "offline";
78
- }
79
-}
80
-
81
-const char *rrdhost_ml_status_to_string(RRDHOST_ML_STATUS status) {
82
- switch(status) {
83
- case RRDHOST_ML_STATUS_RUNNING:
84
- return "online";
85
-
86
- case RRDHOST_ML_STATUS_OFFLINE:
87
- return "offline";
88
-
89
- default:
90
- case RRDHOST_ML_STATUS_DISABLED:
91
- return "disabled";
92
- }
93
-}
94
-
95
-const char *rrdhost_ml_type_to_string(RRDHOST_ML_TYPE type) {
96
- switch(type) {
97
- case RRDHOST_ML_TYPE_SELF:
98
- return "self";
99
-
100
- case RRDHOST_ML_TYPE_RECEIVED:
101
- return "received";
102
-
103
- default:
104
- case RRDHOST_ML_TYPE_DISABLED:
105
- return "disabled";
106
- }
107
-}
108
-
109
-const char *rrdhost_health_status_to_string(RRDHOST_HEALTH_STATUS status) {
110
- switch(status) {
111
- default:
112
- case RRDHOST_HEALTH_STATUS_DISABLED:
113
- return "disabled";
114
-
115
- case RRDHOST_HEALTH_STATUS_INITIALIZING:
116
- return "initializing";
117
-
118
- case RRDHOST_HEALTH_STATUS_RUNNING:
119
- return "online";
120
- }
121
-}
122
-
123
-const char *rrdhost_dyncfg_status_to_string(RRDHOST_DYNCFG_STATUS status) {
124
- switch(status) {
125
- default:
126
- case RRDHOST_DYNCFG_STATUS_UNAVAILABLE:
127
- return "unavailable";
128
-
129
- case RRDHOST_DYNCFG_STATUS_AVAILABLE:
130
- return "online";
131
- }
132
-}
4
+#include "stream-receiver-internals.h"
5
+#include "stream-sender-internals.h"
6
+
7
+ENUM_STR_MAP_DEFINE(RRDHOST_DB_STATUS) = {
8
+ { RRDHOST_DB_STATUS_INITIALIZING, "initializing" },
9
+ { RRDHOST_DB_STATUS_QUERYABLE, "online" },
10
+ { 0, NULL } // Sentinel
11
+};
12
+
13
+ENUM_STR_MAP_DEFINE(RRDHOST_DB_LIVENESS) = {
14
+ { RRDHOST_DB_LIVENESS_STALE, "stale" },
15
+ { RRDHOST_DB_LIVENESS_LIVE, "live" },
16
+ { 0, NULL } // Sentinel
17
+};
18
+
19
+ENUM_STR_MAP_DEFINE(RRDHOST_INGEST_STATUS) = {
20
+ { RRDHOST_INGEST_STATUS_ARCHIVED, "archived" },
21
+ { RRDHOST_INGEST_STATUS_INITIALIZING, "initializing" },
22
+ { RRDHOST_INGEST_STATUS_REPLICATING, "replicating" },
23
+ { RRDHOST_INGEST_STATUS_ONLINE, "online" },
24
+ { RRDHOST_INGEST_STATUS_OFFLINE, "offline" },
25
+ { 0, NULL } // Sentinel
26
+};
27
+
28
+ENUM_STR_MAP_DEFINE(RRDHOST_INGEST_TYPE) = {
29
+ { RRDHOST_INGEST_TYPE_LOCALHOST, "localhost" },
30
+ { RRDHOST_INGEST_TYPE_VIRTUAL, "virtual" },
31
+ { RRDHOST_INGEST_TYPE_CHILD, "child" },
32
+ { RRDHOST_INGEST_TYPE_ARCHIVED, "archived" },
33
+ { 0, NULL } // Sentinel
34
+};
35
+
36
+ENUM_STR_MAP_DEFINE(RRDHOST_STREAMING_STATUS) = {
37
+ { RRDHOST_STREAM_STATUS_DISABLED, "disabled" },
38
+ { RRDHOST_STREAM_STATUS_REPLICATING, "replicating" },
39
+ { RRDHOST_STREAM_STATUS_ONLINE, "online" },
40
+ { RRDHOST_STREAM_STATUS_OFFLINE, "offline" },
41
+ { 0, NULL } // Sentinel
42
+};
43
+
44
+ENUM_STR_MAP_DEFINE(RRDHOST_ML_STATUS) = {
45
+ { RRDHOST_ML_STATUS_DISABLED, "disabled" },
46
+ { RRDHOST_ML_STATUS_OFFLINE, "offline" },
47
+ { RRDHOST_ML_STATUS_RUNNING, "online" },
48
+ { 0, NULL } // Sentinel
49
+};
50
+
51
+ENUM_STR_MAP_DEFINE(RRDHOST_ML_TYPE) = {
52
+ { RRDHOST_ML_TYPE_DISABLED, "disabled" },
53
+ { RRDHOST_ML_TYPE_SELF, "self" },
54
+ { RRDHOST_ML_TYPE_RECEIVED, "received" },
55
+ { 0, NULL } // Sentinel
56
+};
57
+
58
+ENUM_STR_MAP_DEFINE(RRDHOST_HEALTH_STATUS) = {
59
+ { RRDHOST_HEALTH_STATUS_DISABLED, "disabled" },
60
+ { RRDHOST_HEALTH_STATUS_INITIALIZING, "initializing" },
61
+ { RRDHOST_HEALTH_STATUS_RUNNING, "online" },
62
+ { 0, NULL } // Sentinel
63
+};
64
+
65
+ENUM_STR_MAP_DEFINE(RRDHOST_DYNCFG_STATUS) = {
66
+ { RRDHOST_DYNCFG_STATUS_UNAVAILABLE, "unavailable" },
67
+ { RRDHOST_DYNCFG_STATUS_AVAILABLE, "online" },
68
+ { 0, NULL } // Sentinel
69
+};
70
+
71
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_DB_STATUS, RRDHOST_DB_STATUS_INITIALIZING, "initializing");
72
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_DB_LIVENESS, RRDHOST_DB_LIVENESS_STALE, "stale");
73
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_INGEST_STATUS, RRDHOST_INGEST_STATUS_OFFLINE, "offline");
74
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_INGEST_TYPE, RRDHOST_INGEST_TYPE_ARCHIVED, "archived");
75
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_STREAMING_STATUS, RRDHOST_STREAM_STATUS_OFFLINE, "offline");
76
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_ML_STATUS, RRDHOST_ML_STATUS_DISABLED, "disabled");
77
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_ML_TYPE, RRDHOST_ML_TYPE_DISABLED, "disabled");
78
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_HEALTH_STATUS, RRDHOST_HEALTH_STATUS_DISABLED, "disabled");
79
+ENUM_STR_DEFINE_FUNCTIONS(RRDHOST_DYNCFG_STATUS, RRDHOST_DYNCFG_STATUS_UNAVAILABLE, "unavailable");
80
81
static NETDATA_DOUBLE rrdhost_sender_replication_completion_unsafe(RRDHOST *host, time_t now, size_t *instances) {
82
size_t charts = rrdhost_sender_replicating_charts(host);
@@ -179,23 +126,23 @@ void rrdhost_status(RRDHOST *host, time_t now, RRDHOST_STATUS *s) {
126
127
// --- ingest ---
128
182
- s->ingest.since = MAX(host->child_connect_time, host->child_disconnected_time);
183
- s->ingest.reason = (online) ? STREAM_HANDSHAKE_NEVER : host->rrdpush_last_receiver_exit_reason;
129
+ s->ingest.since = MAX(host->stream.rcv.status.last_connected, host->stream.rcv.status.last_disconnected);
130
+ s->ingest.reason = (online) ? STREAM_HANDSHAKE_NEVER : host->stream.rcv.status.exit_reason;
131
185
- spinlock_lock(&host->receiver_lock);
186
- s->ingest.hops = (host->system_info ? host->system_info->hops : (host == localhost) ? 0 : 1);
132
+ rrdhost_receiver_lock(host);
133
+ s->ingest.hops = (int16_t)(host->system_info ? host->system_info->hops : (host == localhost) ? 0 : 1);
134
bool has_receiver = false;
188
- if (host->receiver && !rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED)) {
135
+ if (host->receiver && !rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_RECEIVER_DISCONNECTED)) {
136
has_receiver = true;
137
s->ingest.replication.instances = rrdhost_receiver_replicating_charts(host);
191
- s->ingest.replication.completion = host->rrdpush_receiver_replication_percent;
138
+ s->ingest.replication.completion = host->stream.rcv.status.replication.percent;
139
s->ingest.replication.in_progress = s->ingest.replication.instances > 0;
140
141
s->ingest.capabilities = host->receiver->capabilities;
195
- s->ingest.peers = socket_peers(host->receiver->fd);
196
- s->ingest.ssl = SSL_connection(&host->receiver->ssl);
142
+ s->ingest.peers = nd_sock_socket_peers(&host->receiver->sock);
143
+ s->ingest.ssl = nd_sock_is_ssl(&host->receiver->sock);
144
}
198
- spinlock_unlock(&host->receiver_lock);
145
+ rrdhost_receiver_unlock(host);
146
147
if (online) {
148
if(s->db.status == RRDHOST_DB_STATUS_INITIALIZING)
@@ -235,7 +182,7 @@ void rrdhost_status(RRDHOST *host, time_t now, RRDHOST_STATUS *s) {
182
else
183
s->ingest.type = RRDHOST_INGEST_TYPE_ARCHIVED;
184
238
- s->ingest.id = host->rrdpush_receiver_connection_counter;
185
+ s->ingest.id = host->stream.rcv.status.connections;
186
187
if(!s->ingest.since)
188
s->ingest.since = netdata_start_time;
@@ -249,21 +196,21 @@ void rrdhost_status(RRDHOST *host, time_t now, RRDHOST_STATUS *s) {
196
197
if (!host->sender) {
198
s->stream.status = RRDHOST_STREAM_STATUS_DISABLED;
252
- s->stream.hops = s->ingest.hops + 1;
199
+ s->stream.hops = (int16_t)(s->ingest.hops + 1);
200
}
201
else {
255
- sender_lock(host->sender);
202
+ stream_sender_lock(host->sender);
203
204
s->stream.since = host->sender->last_state_since_t;
258
- s->stream.peers = socket_peers(host->sender->rrdpush_sender_socket);
259
- s->stream.ssl = SSL_connection(&host->sender->ssl);
205
+ s->stream.peers = nd_sock_socket_peers(&host->sender->sock);
206
+ s->stream.ssl = nd_sock_is_ssl(&host->sender->sock);
207
208
memcpy(s->stream.sent_bytes_on_this_connection_per_type,
262
- host->sender->sent_bytes_on_this_connection_per_type,
209
+ host->sender->thread.bytes_sent_by_type,
210
MIN(sizeof(s->stream.sent_bytes_on_this_connection_per_type),
264
- sizeof(host->sender->sent_bytes_on_this_connection_per_type)));
211
+ sizeof(host->sender->thread.bytes_sent_by_type)));
212
266
- if (rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED)) {
213
+ if (rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_SENDER_CONNECTED)) {
214
s->stream.hops = host->sender->hops;
215
s->stream.reason = STREAM_HANDSHAKE_NEVER;
216
s->stream.capabilities = host->sender->capabilities;
@@ -280,14 +227,14 @@ void rrdhost_status(RRDHOST *host, time_t now, RRDHOST_STATUS *s) {
227
}
228
else {
229
s->stream.status = RRDHOST_STREAM_STATUS_OFFLINE;
283
- s->stream.hops = s->ingest.hops + 1;
230
+ s->stream.hops = (int16_t)(s->ingest.hops + 1);
231
s->stream.reason = host->sender->exit.reason;
232
}
233
287
- sender_unlock(host->sender);
234
+ stream_sender_unlock(host->sender);
235
}
236
290
- s->stream.id = host->rrdpush_sender_connection_counter;
237
+ s->stream.id = host->stream.snd.status.connections;
238
239
if(!s->stream.since)
240
s->stream.since = netdata_start_time;
@@ -295,17 +242,16 @@ void rrdhost_status(RRDHOST *host, time_t now, RRDHOST_STATUS *s) {
242
// --- ml ---
243
244
if(ml_host_get_host_status(host, &s->ml.metrics)) {
298
- s->ml.type = RRDHOST_ML_TYPE_SELF;
245
+ if(stream_has_capability(&s->ingest, STREAM_CAP_ML_MODELS))
246
+ s->ml.type = RRDHOST_ML_TYPE_RECEIVED;
247
+ else
248
+ s->ml.type = RRDHOST_ML_TYPE_SELF;
249
250
if(s->ingest.status == RRDHOST_INGEST_STATUS_OFFLINE || s->ingest.status == RRDHOST_INGEST_STATUS_ARCHIVED)
251
s->ml.status = RRDHOST_ML_STATUS_OFFLINE;
252
else
253
s->ml.status = RRDHOST_ML_STATUS_RUNNING;
254
}
305
- else if(stream_has_capability(&s->ingest, STREAM_CAP_DATA_WITH_ML)) {
306
- s->ml.type = RRDHOST_ML_TYPE_RECEIVED;
307
- s->ml.status = RRDHOST_ML_STATUS_RUNNING;
308
- }
255
else {
256
// does not receive ML, does not run ML
257
s->ml.type = RRDHOST_ML_TYPE_DISABLED;
@@ -314,45 +260,44 @@ void rrdhost_status(RRDHOST *host, time_t now, RRDHOST_STATUS *s) {
260
261
// --- health ---
262
317
- if(host->health.health_enabled) {
263
+ if(host->health.enabled) {
264
if(flags & RRDHOST_FLAG_PENDING_HEALTH_INITIALIZATION)
265
s->health.status = RRDHOST_HEALTH_STATUS_INITIALIZING;
320
- else {
266
+ else
267
s->health.status = RRDHOST_HEALTH_STATUS_RUNNING;
268
323
- RRDCALC *rc;
324
- foreach_rrdcalc_in_rrdhost_read(host, rc) {
325
- if (unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
326
- continue;
269
+ RRDCALC *rc;
270
+ foreach_rrdcalc_in_rrdhost_read(host, rc) {
271
+ if (unlikely(!rc->rrdset || !rc->rrdset->last_collected_time.tv_sec))
272
+ continue;
273
328
- switch (rc->status) {
329
- default:
330
- case RRDCALC_STATUS_REMOVED:
331
- break;
274
+ switch (rc->status) {
275
+ default:
276
+ case RRDCALC_STATUS_REMOVED:
277
+ break;
278
333
- case RRDCALC_STATUS_CLEAR:
334
- s->health.alerts.clear++;
335
- break;
279
+ case RRDCALC_STATUS_CLEAR:
280
+ s->health.alerts.clear++;
281
+ break;
282
337
- case RRDCALC_STATUS_WARNING:
338
- s->health.alerts.warning++;
339
- break;
283
+ case RRDCALC_STATUS_WARNING:
284
+ s->health.alerts.warning++;
285
+ break;
286
341
- case RRDCALC_STATUS_CRITICAL:
342
- s->health.alerts.critical++;
343
- break;
287
+ case RRDCALC_STATUS_CRITICAL:
288
+ s->health.alerts.critical++;
289
+ break;
290
345
- case RRDCALC_STATUS_UNDEFINED:
346
- s->health.alerts.undefined++;
347
- break;
291
+ case RRDCALC_STATUS_UNDEFINED:
292
+ s->health.alerts.undefined++;
293
+ break;
294
349
- case RRDCALC_STATUS_UNINITIALIZED:
350
- s->health.alerts.uninitialized++;
351
- break;
352
- }
295
+ case RRDCALC_STATUS_UNINITIALIZED:
296
+ s->health.alerts.uninitialized++;
297
+ break;
298
}
354
- foreach_rrdcalc_in_rrdhost_done(rc);
299
}
300
+ foreach_rrdcalc_in_rrdhost_done(rc);
301
}
302
else
303
s->health.status = RRDHOST_HEALTH_STATUS_DISABLED;
src/streaming/rrdhost-status.h
+26
-23
@@ -4,24 +4,17 @@
4
#define NETDATA_RRDHOST_STATUS_H
5
6
#include "libnetdata/libnetdata.h"
7
-#include "stream-handshake.h"
8
-#include "stream-capabilities.h"
9
-#include "database/rrd.h"
7
8
typedef enum __attribute__((packed)) {
9
RRDHOST_DB_STATUS_INITIALIZING = 0,
10
RRDHOST_DB_STATUS_QUERYABLE,
11
} RRDHOST_DB_STATUS;
12
16
-const char *rrdhost_db_status_to_string(RRDHOST_DB_STATUS status);
17
-
13
typedef enum __attribute__((packed)) {
14
RRDHOST_DB_LIVENESS_STALE = 0,
15
RRDHOST_DB_LIVENESS_LIVE,
16
} RRDHOST_DB_LIVENESS;
17
23
-const char *rrdhost_db_liveness_to_string(RRDHOST_DB_LIVENESS status);
24
-
18
typedef enum __attribute__((packed)) {
19
RRDHOST_INGEST_STATUS_ARCHIVED = 0,
20
RRDHOST_INGEST_STATUS_INITIALIZING,
@@ -30,8 +23,6 @@ typedef enum __attribute__((packed)) {
23
RRDHOST_INGEST_STATUS_OFFLINE,
24
} RRDHOST_INGEST_STATUS;
25
33
-const char *rrdhost_ingest_status_to_string(RRDHOST_INGEST_STATUS status);
34
-
26
typedef enum __attribute__((packed)) {
27
RRDHOST_INGEST_TYPE_LOCALHOST = 0,
28
RRDHOST_INGEST_TYPE_VIRTUAL,
@@ -39,8 +30,6 @@ typedef enum __attribute__((packed)) {
30
RRDHOST_INGEST_TYPE_ARCHIVED,
31
} RRDHOST_INGEST_TYPE;
32
42
-const char *rrdhost_ingest_type_to_string(RRDHOST_INGEST_TYPE type);
43
-
33
typedef enum __attribute__((packed)) {
34
RRDHOST_STREAM_STATUS_DISABLED = 0,
35
RRDHOST_STREAM_STATUS_REPLICATING,
@@ -48,40 +37,54 @@ typedef enum __attribute__((packed)) {
37
RRDHOST_STREAM_STATUS_OFFLINE,
38
} RRDHOST_STREAMING_STATUS;
39
51
-const char *rrdhost_streaming_status_to_string(RRDHOST_STREAMING_STATUS status);
52
-
40
typedef enum __attribute__((packed)) {
41
RRDHOST_ML_STATUS_DISABLED = 0,
42
RRDHOST_ML_STATUS_OFFLINE,
43
RRDHOST_ML_STATUS_RUNNING,
44
} RRDHOST_ML_STATUS;
45
59
-const char *rrdhost_ml_status_to_string(RRDHOST_ML_STATUS status);
60
-
46
typedef enum __attribute__((packed)) {
47
RRDHOST_ML_TYPE_DISABLED = 0,
48
RRDHOST_ML_TYPE_SELF,
49
RRDHOST_ML_TYPE_RECEIVED,
50
} RRDHOST_ML_TYPE;
51
67
-const char *rrdhost_ml_type_to_string(RRDHOST_ML_TYPE type);
68
-
52
typedef enum __attribute__((packed)) {
53
RRDHOST_HEALTH_STATUS_DISABLED = 0,
54
RRDHOST_HEALTH_STATUS_INITIALIZING,
55
RRDHOST_HEALTH_STATUS_RUNNING,
56
} RRDHOST_HEALTH_STATUS;
57
75
-const char *rrdhost_health_status_to_string(RRDHOST_HEALTH_STATUS status);
76
-
58
typedef enum __attribute__((packed)) {
59
RRDHOST_DYNCFG_STATUS_UNAVAILABLE = 0,
60
RRDHOST_DYNCFG_STATUS_AVAILABLE,
61
} RRDHOST_DYNCFG_STATUS;
62
82
-const char *rrdhost_dyncfg_status_to_string(RRDHOST_DYNCFG_STATUS status);
63
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_DB_STATUS);
64
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_DB_LIVENESS);
65
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_INGEST_STATUS);
66
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_INGEST_TYPE);
67
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_STREAMING_STATUS);
68
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_ML_STATUS);
69
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_ML_TYPE);
70
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_HEALTH_STATUS);
71
+ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDHOST_DYNCFG_STATUS);
72
+
73
+#define rrdhost_db_status_to_string(status) RRDHOST_DB_STATUS_2str(status)
74
+#define rrdhost_db_liveness_to_string(status) RRDHOST_DB_LIVENESS_2str(status)
75
+#define rrdhost_ingest_status_to_string(status) RRDHOST_INGEST_STATUS_2str(status)
76
+#define rrdhost_ingest_type_to_string(type) RRDHOST_INGEST_TYPE_2str(type)
77
+#define rrdhost_streaming_status_to_string(status) RRDHOST_STREAMING_STATUS_2str(status)
78
+#define rrdhost_ml_status_to_string(status) RRDHOST_ML_STATUS_2str(status)
79
+#define rrdhost_ml_type_to_string(type) RRDHOST_ML_TYPE_2str(type)
80
+#define rrdhost_health_status_to_string(status) RRDHOST_HEALTH_STATUS_2str(status)
81
+#define rrdhost_dyncfg_status_to_string(status) RRDHOST_DYNCFG_STATUS_2str(status)
82
+
83
+#include "stream-handshake.h"
84
+#include "stream-capabilities.h"
85
+#include "database/rrd.h"
86
84
-typedef struct {
87
+typedef struct rrdhost_status {
88
RRDHOST *host;
89
time_t now;
90
@@ -107,7 +110,7 @@ typedef struct {
110
} ml;
111
112
struct {
110
- size_t hops;
113
+ int16_t hops;
114
RRDHOST_INGEST_TYPE type;
115
RRDHOST_INGEST_STATUS status;
116
SOCKET_PEERS peers;
@@ -131,7 +134,7 @@ typedef struct {
134
} ingest;
135
136
struct {
134
- size_t hops;
137
+ int16_t hops;
138
RRDHOST_STREAMING_STATUS status;
139
SOCKET_PEERS peers;
140
bool ssl;
src/streaming/rrdpush.h
deleted
-18
@@ -1,18 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#ifndef NETDATA_RRDPUSH_H
4
-#define NETDATA_RRDPUSH_H 1
5
-
6
-#include "stream-handshake.h"
7
-#include "stream-capabilities.h"
8
-#include "stream-conf.h"
9
-#include "stream-compression/compression.h"
10
-
11
-#include "sender.h"
12
-#include "receiver.h"
13
-
14
-#include "rrdhost-status.h"
15
-#include "protocol/commands.h"
16
-#include "stream-path.h"
17
-
18
-#endif //NETDATA_RRDPUSH_H
src/streaming/sender-commit.c
deleted
-168
@@ -1,168 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#include "sender-internals.h"
4
-
5
-static __thread BUFFER *sender_thread_buffer = NULL;
6
-static __thread bool sender_thread_buffer_used = false;
7
-static __thread time_t sender_thread_buffer_last_reset_s = 0;
8
-
9
-void sender_thread_buffer_free(void) {
10
- buffer_free(sender_thread_buffer);
11
- sender_thread_buffer = NULL;
12
- sender_thread_buffer_used = false;
13
-}
14
-
15
-// Collector thread starting a transmission
16
-BUFFER *sender_start(struct sender_state *s) {
17
- if(unlikely(sender_thread_buffer_used))
18
- fatal("STREAMING: thread buffer is used multiple times concurrently.");
19
-
20
- if(unlikely(rrdpush_sender_last_buffer_recreate_get(s) > sender_thread_buffer_last_reset_s)) {
21
- if(unlikely(sender_thread_buffer && sender_thread_buffer->size > THREAD_BUFFER_INITIAL_SIZE)) {
22
- buffer_free(sender_thread_buffer);
23
- sender_thread_buffer = NULL;
24
- }
25
- }
26
-
27
- if(unlikely(!sender_thread_buffer)) {
28
- sender_thread_buffer = buffer_create(THREAD_BUFFER_INITIAL_SIZE, &netdata_buffers_statistics.buffers_streaming);
29
- sender_thread_buffer_last_reset_s = rrdpush_sender_last_buffer_recreate_get(s);
30
- }
31
-
32
- sender_thread_buffer_used = true;
33
- buffer_flush(sender_thread_buffer);
34
- return sender_thread_buffer;
35
-}
36
-
37
-#define SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE 3
38
-
39
-// Collector thread finishing a transmission
40
-void sender_commit(struct sender_state *s, BUFFER *wb, STREAM_TRAFFIC_TYPE type) {
41
-
42
- if(unlikely(wb != sender_thread_buffer))
43
- fatal("STREAMING: sender is trying to commit a buffer that is not this thread's buffer.");
44
-
45
- if(unlikely(!sender_thread_buffer_used))
46
- fatal("STREAMING: sender is committing a buffer twice.");
47
-
48
- sender_thread_buffer_used = false;
49
-
50
- char *src = (char *)buffer_tostring(wb);
51
- size_t src_len = buffer_strlen(wb);
52
-
53
- if(unlikely(!src || !src_len))
54
- return;
55
-
56
- sender_lock(s);
57
-
58
-#ifdef NETDATA_LOG_STREAM_SENDER
59
- if(type == STREAM_TRAFFIC_TYPE_METADATA) {
60
- if(!s->stream_log_fp) {
61
- char filename[FILENAME_MAX + 1];
62
- snprintfz(filename, FILENAME_MAX, "/tmp/stream-sender-%s.txt", s->host ? rrdhost_hostname(s->host) : "unknown");
63
-
64
- s->stream_log_fp = fopen(filename, "w");
65
- }
66
-
67
- fprintf(s->stream_log_fp, "\n--- SEND MESSAGE START: %s ----\n"
68
- "%s"
69
- "--- SEND MESSAGE END ----------------------------------------\n"
70
- , rrdhost_hostname(s->host), src
71
- );
72
- }
73
-#endif
74
-
75
- if(unlikely(s->buffer->max_size < (src_len + 1) * SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE)) {
76
- netdata_log_info("STREAM %s [send to %s]: max buffer size of %zu is too small for a data message of size %zu. Increasing the max buffer size to %d times the max data message size.",
77
- rrdhost_hostname(s->host), s->connected_to, s->buffer->max_size, buffer_strlen(wb) + 1, SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE);
78
-
79
- s->buffer->max_size = (src_len + 1) * SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE;
80
- }
81
-
82
- if (s->compressor.initialized) {
83
- while(src_len) {
84
- size_t size_to_compress = src_len;
85
-
86
- if(unlikely(size_to_compress > COMPRESSION_MAX_MSG_SIZE)) {
87
- if (stream_has_capability(s, STREAM_CAP_BINARY))
88
- size_to_compress = COMPRESSION_MAX_MSG_SIZE;
89
- else {
90
- if (size_to_compress > COMPRESSION_MAX_MSG_SIZE) {
91
- // we need to find the last newline
92
- // so that the decompressor will have a whole line to work with
93
-
94
- const char *t = &src[COMPRESSION_MAX_MSG_SIZE];
95
- while (--t >= src)
96
- if (unlikely(*t == '\n'))
97
- break;
98
-
99
- if (t <= src) {
100
- size_to_compress = COMPRESSION_MAX_MSG_SIZE;
101
- } else
102
- size_to_compress = t - src + 1;
103
- }
104
- }
105
- }
106
-
107
- const char *dst;
108
- size_t dst_len = rrdpush_compress(&s->compressor, src, size_to_compress, &dst);
109
- if (!dst_len) {
110
- netdata_log_error("STREAM %s [send to %s]: COMPRESSION failed. Resetting compressor and re-trying",
111
- rrdhost_hostname(s->host), s->connected_to);
112
-
113
- rrdpush_compression_initialize(s);
114
- dst_len = rrdpush_compress(&s->compressor, src, size_to_compress, &dst);
115
- if(!dst_len) {
116
- netdata_log_error("STREAM %s [send to %s]: COMPRESSION failed again. Deactivating compression",
117
- rrdhost_hostname(s->host), s->connected_to);
118
-
119
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_NO_COMPRESSION);
120
- rrdpush_compression_deactivate(s);
121
- rrdpush_sender_thread_close_socket(s);
122
- sender_unlock(s);
123
- return;
124
- }
125
- }
126
-
127
- rrdpush_signature_t signature = rrdpush_compress_encode_signature(dst_len);
128
-
129
-#ifdef NETDATA_INTERNAL_CHECKS
130
- // check if reversing the signature provides the same length
131
- size_t decoded_dst_len = rrdpush_decompress_decode_signature((const char *)&signature, sizeof(signature));
132
- if(decoded_dst_len != dst_len)
133
- fatal("RRDPUSH COMPRESSION: invalid signature, original payload %zu bytes, "
134
- "compressed payload length %zu bytes, but signature says payload is %zu bytes",
135
- size_to_compress, dst_len, decoded_dst_len);
136
-#endif
137
-
138
- if(cbuffer_add_unsafe(s->buffer, (const char *)&signature, sizeof(signature)))
139
- s->flags |= SENDER_FLAG_OVERFLOW;
140
- else {
141
- if(cbuffer_add_unsafe(s->buffer, dst, dst_len))
142
- s->flags |= SENDER_FLAG_OVERFLOW;
143
- else
144
- s->sent_bytes_on_this_connection_per_type[type] += dst_len + sizeof(signature);
145
- }
146
-
147
- src = src + size_to_compress;
148
- src_len -= size_to_compress;
149
- }
150
- }
151
- else if(cbuffer_add_unsafe(s->buffer, src, src_len))
152
- s->flags |= SENDER_FLAG_OVERFLOW;
153
- else
154
- s->sent_bytes_on_this_connection_per_type[type] += src_len;
155
-
156
- replication_recalculate_buffer_used_ratio_unsafe(s);
157
-
158
- bool signal_sender = false;
159
- if(!rrdpush_sender_pipe_has_pending_data(s)) {
160
- rrdpush_sender_pipe_set_pending_data(s);
161
- signal_sender = true;
162
- }
163
-
164
- sender_unlock(s);
165
-
166
- if(signal_sender && (!stream_has_capability(s, STREAM_CAP_INTERPOLATED) || type != STREAM_TRAFFIC_TYPE_DATA))
167
- rrdpush_signal_sender_to_wake_up(s);
168
-}
src/streaming/sender-destinations.c
deleted
-143
@@ -1,143 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#include "sender-internals.h"
4
-
5
-void rrdpush_reset_destinations_postpone_time(RRDHOST *host) {
6
- uint32_t wait = (host->sender) ? host->sender->reconnect_delay : 5;
7
- time_t now = now_realtime_sec();
8
- for (struct rrdpush_destinations *d = host->destinations; d; d = d->next)
9
- d->postpone_reconnection_until = now + wait;
10
-}
11
-
12
-void rrdpush_sender_ssl_init(RRDHOST *host) {
13
- static SPINLOCK sp = NETDATA_SPINLOCK_INITIALIZER;
14
- spinlock_lock(&sp);
15
-
16
- if(netdata_ssl_streaming_sender_ctx || !host) {
17
- spinlock_unlock(&sp);
18
- return;
19
- }
20
-
21
- for(struct rrdpush_destinations *d = host->destinations; d ; d = d->next) {
22
- if (d->ssl) {
23
- // we need to initialize SSL
24
-
25
- netdata_ssl_initialize_ctx(NETDATA_SSL_STREAMING_SENDER_CTX);
26
- ssl_security_location_for_context(netdata_ssl_streaming_sender_ctx, stream_conf_ssl_ca_file, stream_conf_ssl_ca_path);
27
-
28
- // stop the loop
29
- break;
30
- }
31
- }
32
-
33
- spinlock_unlock(&sp);
34
-}
35
-
36
-int connect_to_one_of_destinations(
37
- RRDHOST *host,
38
- int default_port,
39
- struct timeval *timeout,
40
- size_t *reconnects_counter,
41
- char *connected_to,
42
- size_t connected_to_size,
43
- struct rrdpush_destinations **destination)
44
-{
45
- int sock = -1;
46
-
47
- for (struct rrdpush_destinations *d = host->destinations; d; d = d->next) {
48
- time_t now = now_realtime_sec();
49
-
50
- if(nd_thread_signaled_to_cancel())
51
- return -1;
52
-
53
- if(d->postpone_reconnection_until > now)
54
- continue;
55
-
56
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
57
- "STREAM %s: connecting to '%s' (default port: %d)...",
58
- rrdhost_hostname(host), string2str(d->destination), default_port);
59
-
60
- if (reconnects_counter)
61
- *reconnects_counter += 1;
62
-
63
- d->since = now;
64
- d->attempts++;
65
- sock = connect_to_this(string2str(d->destination), default_port, timeout);
66
-
67
- if (sock != -1) {
68
- if (connected_to && connected_to_size)
69
- strncpyz(connected_to, string2str(d->destination), connected_to_size);
70
-
71
- *destination = d;
72
-
73
- // move the current item to the end of the list
74
- // without this, this destination will break the loop again and again
75
- // not advancing the destinations to find one that may work
76
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(host->destinations, d, prev, next);
77
- DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(host->destinations, d, prev, next);
78
-
79
- break;
80
- }
81
- }
82
-
83
- return sock;
84
-}
85
-
86
-struct destinations_init_tmp {
87
- RRDHOST *host;
88
- struct rrdpush_destinations *list;
89
- int count;
90
-};
91
-
92
-static bool destinations_init_add_one(char *entry, void *data) {
93
- struct destinations_init_tmp *t = data;
94
-
95
- struct rrdpush_destinations *d = callocz(1, sizeof(struct rrdpush_destinations));
96
- char *colon_ssl = strstr(entry, ":SSL");
97
- if(colon_ssl) {
98
- *colon_ssl = '\0';
99
- d->ssl = true;
100
- }
101
- else
102
- d->ssl = false;
103
-
104
- d->destination = string_strdupz(entry);
105
-
106
- __atomic_add_fetch(&netdata_buffers_statistics.rrdhost_senders, sizeof(struct rrdpush_destinations), __ATOMIC_RELAXED);
107
-
108
- DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(t->list, d, prev, next);
109
-
110
- t->count++;
111
- nd_log_daemon(NDLP_INFO, "STREAM: added streaming destination No %d: '%s' to host '%s'", t->count, string2str(d->destination), rrdhost_hostname(t->host));
112
-
113
- return false; // we return false, so that we will get all defined destinations
114
-}
115
-
116
-void rrdpush_destinations_init(RRDHOST *host) {
117
- if(!host->rrdpush.send.destination) return;
118
-
119
- rrdpush_destinations_free(host);
120
-
121
- struct destinations_init_tmp t = {
122
- .host = host,
123
- .list = NULL,
124
- .count = 0,
125
- };
126
-
127
- foreach_entry_in_connection_string(host->rrdpush.send.destination, destinations_init_add_one, &t);
128
-
129
- host->destinations = t.list;
130
-}
131
-
132
-void rrdpush_destinations_free(RRDHOST *host) {
133
- while (host->destinations) {
134
- struct rrdpush_destinations *tmp = host->destinations;
135
- DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(host->destinations, tmp, prev, next);
136
- string_freez(tmp->destination);
137
- freez(tmp);
138
- __atomic_sub_fetch(&netdata_buffers_statistics.rrdhost_senders, sizeof(struct rrdpush_destinations), __ATOMIC_RELAXED);
139
- }
140
-
141
- host->destinations = NULL;
142
-}
143
-
src/streaming/sender-destinations.h
deleted
-38
@@ -1,38 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#ifndef NETDATA_SENDER_DESTINATIONS_H
4
-#define NETDATA_SENDER_DESTINATIONS_H
5
-
6
-#include "libnetdata/libnetdata.h"
7
-#include "stream-handshake.h"
8
-#include "database/rrd.h"
9
-
10
-struct rrdpush_destinations {
11
- STRING *destination;
12
- bool ssl;
13
- uint32_t attempts;
14
- time_t since;
15
- time_t postpone_reconnection_until;
16
- STREAM_HANDSHAKE reason;
17
-
18
- struct rrdpush_destinations *prev;
19
- struct rrdpush_destinations *next;
20
-};
21
-
22
-void rrdpush_sender_ssl_init(RRDHOST *host);
23
-
24
-void rrdpush_reset_destinations_postpone_time(RRDHOST *host);
25
-
26
-void rrdpush_destinations_init(RRDHOST *host);
27
-void rrdpush_destinations_free(RRDHOST *host);
28
-
29
-int connect_to_one_of_destinations(
30
- RRDHOST *host,
31
- int default_port,
32
- struct timeval *timeout,
33
- size_t *reconnects_counter,
34
- char *connected_to,
35
- size_t connected_to_size,
36
- struct rrdpush_destinations **destination);
37
-
38
-#endif //NETDATA_SENDER_DESTINATIONS_H
src/streaming/sender-internals.h
deleted
-48
@@ -1,48 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#ifndef NETDATA_SENDER_INTERNALS_H
4
-#define NETDATA_SENDER_INTERNALS_H
5
-
6
-#include "rrdpush.h"
7
-#include "h2o-common.h"
8
-#include "aclk/https_client.h"
9
-
10
-#define WORKER_SENDER_JOB_CONNECT 0
11
-#define WORKER_SENDER_JOB_PIPE_READ 1
12
-#define WORKER_SENDER_JOB_SOCKET_RECEIVE 2
13
-#define WORKER_SENDER_JOB_EXECUTE 3
14
-#define WORKER_SENDER_JOB_SOCKET_SEND 4
15
-#define WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE 5
16
-#define WORKER_SENDER_JOB_DISCONNECT_OVERFLOW 6
17
-#define WORKER_SENDER_JOB_DISCONNECT_TIMEOUT 7
18
-#define WORKER_SENDER_JOB_DISCONNECT_POLL_ERROR 8
19
-#define WORKER_SENDER_JOB_DISCONNECT_SOCKET_ERROR 9
20
-#define WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR 10
21
-#define WORKER_SENDER_JOB_DISCONNECT_PARENT_CLOSED 11
22
-#define WORKER_SENDER_JOB_DISCONNECT_RECEIVE_ERROR 12
23
-#define WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR 13
24
-#define WORKER_SENDER_JOB_DISCONNECT_NO_COMPRESSION 14
25
-#define WORKER_SENDER_JOB_BUFFER_RATIO 15
26
-#define WORKER_SENDER_JOB_BYTES_RECEIVED 16
27
-#define WORKER_SENDER_JOB_BYTES_SENT 17
28
-#define WORKER_SENDER_JOB_BYTES_COMPRESSED 18
29
-#define WORKER_SENDER_JOB_BYTES_UNCOMPRESSED 19
30
-#define WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO 20
31
-#define WORKER_SENDER_JOB_REPLAY_REQUEST 21
32
-#define WORKER_SENDER_JOB_FUNCTION_REQUEST 22
33
-#define WORKER_SENDER_JOB_REPLAY_DICT_SIZE 23
34
-#define WORKER_SENDER_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION 24
35
-
36
-#if WORKER_UTILIZATION_MAX_JOB_TYPES < 25
37
-#error WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 25
38
-#endif
39
-
40
-bool attempt_to_connect(struct sender_state *state);
41
-void rrdpush_sender_on_connect(RRDHOST *host);
42
-void rrdpush_sender_after_connect(RRDHOST *host);
43
-void rrdpush_sender_thread_close_socket(struct sender_state *s);
44
-
45
-void rrdpush_sender_execute_commands_cleanup(struct sender_state *s);
46
-void rrdpush_sender_execute_commands(struct sender_state *s);
47
-
48
-#endif //NETDATA_SENDER_INTERNALS_H
src/streaming/sender.c
deleted
-671
@@ -1,671 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#include "sender-internals.h"
4
-
5
-// resets all the chart, so that their definitions
6
-// will be resent to the central netdata
7
-static void rrdpush_sender_thread_reset_all_charts(RRDHOST *host) {
8
- RRDSET *st;
9
- rrdset_foreach_read(st, host) {
10
- rrdset_flag_clear(st, RRDSET_FLAG_SENDER_REPLICATION_IN_PROGRESS);
11
- rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
12
-
13
- st->rrdpush.sender.resync_time_s = 0;
14
-
15
- RRDDIM *rd;
16
- rrddim_foreach_read(rd, st)
17
- rrddim_metadata_exposed_upstream_clear(rd);
18
- rrddim_foreach_done(rd);
19
-
20
- rrdset_metadata_updated(st);
21
- }
22
- rrdset_foreach_done(st);
23
-
24
- rrdhost_sender_replicating_charts_zero(host);
25
-}
26
-
27
-void rrdpush_sender_cbuffer_recreate_timed(struct sender_state *s, time_t now_s, bool have_mutex, bool force) {
28
- static __thread time_t last_reset_time_s = 0;
29
-
30
- if(!force && now_s - last_reset_time_s < 300)
31
- return;
32
-
33
- if(!have_mutex)
34
- sender_lock(s);
35
-
36
- rrdpush_sender_last_buffer_recreate_set(s, now_s);
37
- last_reset_time_s = now_s;
38
-
39
- if(s->buffer && s->buffer->size > CBUFFER_INITIAL_SIZE) {
40
- size_t max = s->buffer->max_size;
41
- cbuffer_free(s->buffer);
42
- s->buffer = cbuffer_new(CBUFFER_INITIAL_SIZE, max, &netdata_buffers_statistics.cbuffers_streaming);
43
- }
44
-
45
- sender_thread_buffer_free();
46
-
47
- if(!have_mutex)
48
- sender_unlock(s);
49
-}
50
-
51
-static void rrdpush_sender_cbuffer_flush(RRDHOST *host) {
52
- rrdpush_sender_set_flush_time(host->sender);
53
-
54
- sender_lock(host->sender);
55
-
56
- // flush the output buffer from any data it may have
57
- cbuffer_flush(host->sender->buffer);
58
- rrdpush_sender_cbuffer_recreate_timed(host->sender, now_monotonic_sec(), true, true);
59
- replication_recalculate_buffer_used_ratio_unsafe(host->sender);
60
-
61
- sender_unlock(host->sender);
62
-}
63
-
64
-static void rrdpush_sender_charts_and_replication_reset(RRDHOST *host) {
65
- rrdpush_sender_set_flush_time(host->sender);
66
-
67
- // stop all replication commands inflight
68
- replication_sender_delete_pending_requests(host->sender);
69
-
70
- // reset the state of all charts
71
- rrdpush_sender_thread_reset_all_charts(host);
72
-
73
- rrdpush_sender_replicating_charts_zero(host->sender);
74
-}
75
-
76
-void rrdpush_sender_on_connect(RRDHOST *host) {
77
- rrdpush_sender_cbuffer_flush(host);
78
- rrdpush_sender_charts_and_replication_reset(host);
79
-}
80
-
81
-void rrdpush_sender_after_connect(RRDHOST *host) {
82
- rrdpush_sender_thread_send_custom_host_variables(host);
83
-}
84
-
85
-static void rrdpush_sender_on_disconnect(RRDHOST *host) {
86
- // we have been connected to this parent - let's cleanup
87
-
88
- rrdpush_sender_charts_and_replication_reset(host);
89
-
90
- // clear the parent's claim id
91
- rrdpush_sender_clear_parent_claim_id(host);
92
- rrdpush_receiver_send_node_and_claim_id_to_child(host);
93
- stream_path_parent_disconnected(host);
94
-}
95
-
96
-// TCP window is open, and we have data to transmit.
97
-static ssize_t attempt_to_send(struct sender_state *s) {
98
- ssize_t ret;
99
-
100
-#ifdef NETDATA_INTERNAL_CHECKS
101
- struct circular_buffer *cb = s->buffer;
102
-#endif
103
-
104
- sender_lock(s);
105
- char *chunk;
106
- size_t outstanding = cbuffer_next_unsafe(s->buffer, &chunk);
107
- netdata_log_debug(D_STREAM, "STREAM: Sending data. Buffer r=%zu w=%zu s=%zu, next chunk=%zu", cb->read, cb->write, cb->size, outstanding);
108
-
109
- if(SSL_connection(&s->ssl))
110
- ret = netdata_ssl_write(&s->ssl, chunk, outstanding);
111
- else
112
- ret = send(s->rrdpush_sender_socket, chunk, outstanding, MSG_DONTWAIT);
113
-
114
- if (likely(ret > 0)) {
115
- cbuffer_remove_unsafe(s->buffer, ret);
116
- s->sent_bytes_on_this_connection += ret;
117
- s->sent_bytes += ret;
118
- netdata_log_debug(D_STREAM, "STREAM %s [send to %s]: Sent %zd bytes", rrdhost_hostname(s->host), s->connected_to, ret);
119
- }
120
- else if (ret == -1 && (errno == EAGAIN || errno == EINTR || errno == EWOULDBLOCK))
121
- netdata_log_debug(D_STREAM, "STREAM %s [send to %s]: unavailable after polling POLLOUT", rrdhost_hostname(s->host), s->connected_to);
122
- else if (ret == -1) {
123
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR);
124
- netdata_log_debug(D_STREAM, "STREAM: Send failed - closing socket...");
125
- netdata_log_error("STREAM %s [send to %s]: failed to send metrics - closing connection - we have sent %zu bytes on this connection.", rrdhost_hostname(s->host), s->connected_to, s->sent_bytes_on_this_connection);
126
- rrdpush_sender_thread_close_socket(s);
127
- }
128
- else
129
- netdata_log_debug(D_STREAM, "STREAM: send() returned 0 -> no error but no transmission");
130
-
131
- replication_recalculate_buffer_used_ratio_unsafe(s);
132
- sender_unlock(s);
133
-
134
- return ret;
135
-}
136
-
137
-static ssize_t attempt_read(struct sender_state *s) {
138
- ssize_t ret;
139
-
140
- if (SSL_connection(&s->ssl))
141
- ret = netdata_ssl_read(&s->ssl, s->read_buffer + s->read_len, sizeof(s->read_buffer) - s->read_len - 1);
142
- else
143
- ret = recv(s->rrdpush_sender_socket, s->read_buffer + s->read_len, sizeof(s->read_buffer) - s->read_len - 1,MSG_DONTWAIT);
144
-
145
- if (ret > 0) {
146
- s->read_len += ret;
147
- return ret;
148
- }
149
-
150
- if (ret < 0 && (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR))
151
- return ret;
152
-
153
- if (SSL_connection(&s->ssl))
154
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR);
155
- else if (ret == 0 || errno == ECONNRESET) {
156
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_PARENT_CLOSED);
157
- netdata_log_error("STREAM %s [send to %s]: connection closed by far end.", rrdhost_hostname(s->host), s->connected_to);
158
- }
159
- else {
160
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_RECEIVE_ERROR);
161
- netdata_log_error("STREAM %s [send to %s]: error during receive (%zd) - closing connection.", rrdhost_hostname(s->host), s->connected_to, ret);
162
- }
163
-
164
- rrdpush_sender_thread_close_socket(s);
165
-
166
- return ret;
167
-}
168
-
169
-static bool rrdpush_sender_pipe_close(RRDHOST *host, int *pipe_fds, bool reopen) {
170
- static netdata_mutex_t mutex = NETDATA_MUTEX_INITIALIZER;
171
-
172
- bool ret = true;
173
-
174
- netdata_mutex_lock(&mutex);
175
-
176
- int new_pipe_fds[2];
177
- if(reopen) {
178
- if(pipe(new_pipe_fds) != 0) {
179
- netdata_log_error("STREAM %s [send]: cannot create required pipe.", rrdhost_hostname(host));
180
- new_pipe_fds[PIPE_READ] = -1;
181
- new_pipe_fds[PIPE_WRITE] = -1;
182
- ret = false;
183
- }
184
- }
185
-
186
- int old_pipe_fds[2];
187
- old_pipe_fds[PIPE_READ] = pipe_fds[PIPE_READ];
188
- old_pipe_fds[PIPE_WRITE] = pipe_fds[PIPE_WRITE];
189
-
190
- if(reopen) {
191
- pipe_fds[PIPE_READ] = new_pipe_fds[PIPE_READ];
192
- pipe_fds[PIPE_WRITE] = new_pipe_fds[PIPE_WRITE];
193
- }
194
- else {
195
- pipe_fds[PIPE_READ] = -1;
196
- pipe_fds[PIPE_WRITE] = -1;
197
- }
198
-
199
- if(old_pipe_fds[PIPE_READ] > 2)
200
- close(old_pipe_fds[PIPE_READ]);
201
-
202
- if(old_pipe_fds[PIPE_WRITE] > 2)
203
- close(old_pipe_fds[PIPE_WRITE]);
204
-
205
- netdata_mutex_unlock(&mutex);
206
- return ret;
207
-}
208
-
209
-void rrdpush_signal_sender_to_wake_up(struct sender_state *s) {
210
- if(unlikely(s->tid == gettid_cached()))
211
- return;
212
-
213
- RRDHOST *host = s->host;
214
-
215
- int pipe_fd = s->rrdpush_sender_pipe[PIPE_WRITE];
216
-
217
- // signal the sender there are more data
218
- if (pipe_fd != -1 && write(pipe_fd, " ", 1) == -1) {
219
- netdata_log_error("STREAM %s [send]: cannot write to internal pipe.", rrdhost_hostname(host));
220
- rrdpush_sender_pipe_close(host, s->rrdpush_sender_pipe, true);
221
- }
222
-}
223
-
224
-static bool rrdhost_set_sender(RRDHOST *host) {
225
- if(unlikely(!host->sender)) return false;
226
-
227
- bool ret = false;
228
- sender_lock(host->sender);
229
- if(!host->sender->tid) {
230
- rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED | RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
231
- rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN);
232
- host->rrdpush_sender_connection_counter++;
233
- host->sender->tid = gettid_cached();
234
- host->sender->last_state_since_t = now_realtime_sec();
235
- host->sender->exit.reason = STREAM_HANDSHAKE_NEVER;
236
- ret = true;
237
- }
238
- sender_unlock(host->sender);
239
-
240
- rrdpush_reset_destinations_postpone_time(host);
241
-
242
- return ret;
243
-}
244
-
245
-static void rrdhost_clear_sender___while_having_sender_mutex(RRDHOST *host) {
246
- if(unlikely(!host->sender)) return;
247
-
248
- if(host->sender->tid == gettid_cached()) {
249
- host->sender->tid = 0;
250
- host->sender->exit.shutdown = false;
251
- rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN | RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED | RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
252
- host->sender->last_state_since_t = now_realtime_sec();
253
- if(host->destination) {
254
- host->destination->since = host->sender->last_state_since_t;
255
- host->destination->reason = host->sender->exit.reason;
256
- }
257
- }
258
-
259
- rrdpush_reset_destinations_postpone_time(host);
260
-}
261
-
262
-bool rrdhost_sender_should_exit(struct sender_state *s) {
263
- if(unlikely(nd_thread_signaled_to_cancel())) {
264
- if(!s->exit.reason)
265
- s->exit.reason = STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN;
266
- return true;
267
- }
268
-
269
- if(unlikely(!service_running(SERVICE_STREAMING))) {
270
- if(!s->exit.reason)
271
- s->exit.reason = STREAM_HANDSHAKE_DISCONNECT_NETDATA_EXIT;
272
- return true;
273
- }
274
-
275
- if(unlikely(!rrdhost_has_rrdpush_sender_enabled(s->host))) {
276
- if(!s->exit.reason)
277
- s->exit.reason = STREAM_HANDSHAKE_NON_STREAMABLE_HOST;
278
- return true;
279
- }
280
-
281
- if(unlikely(s->exit.shutdown)) {
282
- if(!s->exit.reason)
283
- s->exit.reason = STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN;
284
- return true;
285
- }
286
-
287
- if(unlikely(rrdhost_flag_check(s->host, RRDHOST_FLAG_ORPHAN))) {
288
- if(!s->exit.reason)
289
- s->exit.reason = STREAM_HANDSHAKE_DISCONNECT_ORPHAN_HOST;
290
- return true;
291
- }
292
-
293
- return false;
294
-}
295
-
296
-static bool stream_sender_log_capabilities(BUFFER *wb, void *ptr) {
297
- struct sender_state *state = ptr;
298
- if(!state)
299
- return false;
300
-
301
- stream_capabilities_to_string(wb, state->capabilities);
302
- return true;
303
-}
304
-
305
-static bool stream_sender_log_transport(BUFFER *wb, void *ptr) {
306
- struct sender_state *state = ptr;
307
- if(!state)
308
- return false;
309
-
310
- buffer_strcat(wb, SSL_connection(&state->ssl) ? "https" : "http");
311
- return true;
312
-}
313
-
314
-static bool stream_sender_log_dst_ip(BUFFER *wb, void *ptr) {
315
- struct sender_state *state = ptr;
316
- if(!state || state->rrdpush_sender_socket == -1)
317
- return false;
318
-
319
- SOCKET_PEERS peers = socket_peers(state->rrdpush_sender_socket);
320
- buffer_strcat(wb, peers.peer.ip);
321
- return true;
322
-}
323
-
324
-static bool stream_sender_log_dst_port(BUFFER *wb, void *ptr) {
325
- struct sender_state *state = ptr;
326
- if(!state || state->rrdpush_sender_socket == -1)
327
- return false;
328
-
329
- SOCKET_PEERS peers = socket_peers(state->rrdpush_sender_socket);
330
- buffer_print_uint64(wb, peers.peer.port);
331
- return true;
332
-}
333
-
334
-void *rrdpush_sender_thread(void *ptr) {
335
- struct sender_state *s = ptr;
336
-
337
- ND_LOG_STACK lgs[] = {
338
- ND_LOG_FIELD_STR(NDF_NIDL_NODE, s->host->hostname),
339
- ND_LOG_FIELD_CB(NDF_DST_IP, stream_sender_log_dst_ip, s),
340
- ND_LOG_FIELD_CB(NDF_DST_PORT, stream_sender_log_dst_port, s),
341
- ND_LOG_FIELD_CB(NDF_DST_TRANSPORT, stream_sender_log_transport, s),
342
- ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_sender_log_capabilities, s),
343
- ND_LOG_FIELD_END(),
344
- };
345
- ND_LOG_STACK_PUSH(lgs);
346
-
347
- worker_register("STREAMSND");
348
- worker_register_job_name(WORKER_SENDER_JOB_CONNECT, "connect");
349
- worker_register_job_name(WORKER_SENDER_JOB_PIPE_READ, "pipe read");
350
- worker_register_job_name(WORKER_SENDER_JOB_SOCKET_RECEIVE, "receive");
351
- worker_register_job_name(WORKER_SENDER_JOB_EXECUTE, "execute");
352
- worker_register_job_name(WORKER_SENDER_JOB_SOCKET_SEND, "send");
353
-
354
- // disconnection reasons
355
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_TIMEOUT, "disconnect timeout");
356
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_POLL_ERROR, "disconnect poll error");
357
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_SOCKET_ERROR, "disconnect socket error");
358
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_OVERFLOW, "disconnect overflow");
359
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR, "disconnect ssl error");
360
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_PARENT_CLOSED, "disconnect parent closed");
361
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_RECEIVE_ERROR, "disconnect receive error");
362
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR, "disconnect send error");
363
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_NO_COMPRESSION, "disconnect no compression");
364
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE, "disconnect bad handshake");
365
- worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION, "disconnect cant upgrade");
366
-
367
- worker_register_job_name(WORKER_SENDER_JOB_REPLAY_REQUEST, "replay request");
368
- worker_register_job_name(WORKER_SENDER_JOB_FUNCTION_REQUEST, "function");
369
-
370
- worker_register_job_custom_metric(WORKER_SENDER_JOB_BUFFER_RATIO, "used buffer ratio", "%", WORKER_METRIC_ABSOLUTE);
371
- worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_RECEIVED, "bytes received", "bytes/s", WORKER_METRIC_INCREMENT);
372
- worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_SENT, "bytes sent", "bytes/s", WORKER_METRIC_INCREMENT);
373
- worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_COMPRESSED, "bytes compressed", "bytes/s", WORKER_METRIC_INCREMENTAL_TOTAL);
374
- worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_UNCOMPRESSED, "bytes uncompressed", "bytes/s", WORKER_METRIC_INCREMENTAL_TOTAL);
375
- worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO, "cumulative compression savings ratio", "%", WORKER_METRIC_ABSOLUTE);
376
- worker_register_job_custom_metric(WORKER_SENDER_JOB_REPLAY_DICT_SIZE, "replication dict entries", "entries", WORKER_METRIC_ABSOLUTE);
377
-
378
- if(!rrdhost_has_rrdpush_sender_enabled(s->host) || !s->host->rrdpush.send.destination ||
379
- !*s->host->rrdpush.send.destination || !s->host->rrdpush.send.api_key ||
380
- !*s->host->rrdpush.send.api_key) {
381
- netdata_log_error("STREAM %s [send]: thread created (task id %d), but host has streaming disabled.",
382
- rrdhost_hostname(s->host), gettid_cached());
383
- return NULL;
384
- }
385
-
386
- if(!rrdhost_set_sender(s->host)) {
387
- netdata_log_error("STREAM %s [send]: thread created (task id %d), but there is another sender running for this host.",
388
- rrdhost_hostname(s->host), gettid_cached());
389
- return NULL;
390
- }
391
-
392
- rrdpush_sender_ssl_init(s->host);
393
-
394
- netdata_log_info("STREAM %s [send]: thread created (task id %d)", rrdhost_hostname(s->host), gettid_cached());
395
-
396
- s->timeout = (int)appconfig_get_duration_seconds(
397
- &stream_config, CONFIG_SECTION_STREAM, "timeout", 600);
398
-
399
- s->default_port = (int)appconfig_get_number(
400
- &stream_config, CONFIG_SECTION_STREAM, "default port", 19999);
401
-
402
- s->buffer->max_size = (size_t)appconfig_get_number(
403
- &stream_config, CONFIG_SECTION_STREAM, "buffer size bytes", 1024 * 1024 * 10);
404
-
405
- s->reconnect_delay = (unsigned int)appconfig_get_duration_seconds(
406
- &stream_config, CONFIG_SECTION_STREAM, "reconnect delay", 5);
407
-
408
- stream_conf_initial_clock_resync_iterations = (unsigned int)appconfig_get_number(
409
- &stream_config, CONFIG_SECTION_STREAM,
410
- "initial clock resync iterations",
411
- stream_conf_initial_clock_resync_iterations); // TODO: REMOVE FOR SLEW / GAPFILLING
412
-
413
- s->parent_using_h2o = appconfig_get_boolean(
414
- &stream_config, CONFIG_SECTION_STREAM, "parent using h2o", false);
415
-
416
- // initialize rrdpush globals
417
- rrdhost_flag_clear(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED | RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
418
-
419
- int pipe_buffer_size = 10 * 1024;
420
-#ifdef F_GETPIPE_SZ
421
- pipe_buffer_size = fcntl(s->rrdpush_sender_pipe[PIPE_READ], F_GETPIPE_SZ);
422
-#endif
423
- if(pipe_buffer_size < 10 * 1024)
424
- pipe_buffer_size = 10 * 1024;
425
-
426
- if(!rrdpush_sender_pipe_close(s->host, s->rrdpush_sender_pipe, true)) {
427
- netdata_log_error("STREAM %s [send]: cannot create inter-thread communication pipe. Disabling streaming.",
428
- rrdhost_hostname(s->host));
429
- return NULL;
430
- }
431
-
432
- char *pipe_buffer = mallocz(pipe_buffer_size);
433
-
434
- bool was_connected = false;
435
- size_t iterations = 0;
436
- time_t now_s = now_monotonic_sec();
437
- while(!rrdhost_sender_should_exit(s)) {
438
- iterations++;
439
-
440
- // The connection attempt blocks (after which we use the socket in nonblocking)
441
- if(unlikely(s->rrdpush_sender_socket == -1)) {
442
- if(was_connected)
443
- rrdpush_sender_on_disconnect(s->host);
444
-
445
- was_connected = rrdpush_sender_connect(s);
446
- now_s = s->last_traffic_seen_t;
447
- continue;
448
- }
449
-
450
- if(iterations % 1000 == 0)
451
- now_s = now_monotonic_sec();
452
-
453
- // If the TCP window never opened then something is wrong, restart connection
454
- if(unlikely(now_s - s->last_traffic_seen_t > s->timeout &&
455
- !rrdpush_sender_pending_replication_requests(s) &&
456
- !rrdpush_sender_replicating_charts(s)
457
- )) {
458
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_TIMEOUT);
459
- netdata_log_error("STREAM %s [send to %s]: could not send metrics for %d seconds - closing connection - we have sent %zu bytes on this connection via %zu send attempts.", rrdhost_hostname(s->host), s->connected_to, s->timeout, s->sent_bytes_on_this_connection, s->send_attempts);
460
- rrdpush_sender_thread_close_socket(s);
461
- continue;
462
- }
463
-
464
- sender_lock(s);
465
- size_t outstanding = cbuffer_next_unsafe(s->buffer, NULL);
466
- size_t available = cbuffer_available_size_unsafe(s->buffer);
467
- if (unlikely(!outstanding)) {
468
- rrdpush_sender_pipe_clear_pending_data(s);
469
- rrdpush_sender_cbuffer_recreate_timed(s, now_s, true, false);
470
- }
471
-
472
- if(s->compressor.initialized) {
473
- size_t bytes_uncompressed = s->compressor.sender_locked.total_uncompressed;
474
- size_t bytes_compressed = s->compressor.sender_locked.total_compressed + s->compressor.sender_locked.total_compressions * sizeof(rrdpush_signature_t);
475
- NETDATA_DOUBLE ratio = 100.0 - ((NETDATA_DOUBLE)bytes_compressed * 100.0 / (NETDATA_DOUBLE)bytes_uncompressed);
476
- worker_set_metric(WORKER_SENDER_JOB_BYTES_UNCOMPRESSED, (NETDATA_DOUBLE)bytes_uncompressed);
477
- worker_set_metric(WORKER_SENDER_JOB_BYTES_COMPRESSED, (NETDATA_DOUBLE)bytes_compressed);
478
- worker_set_metric(WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO, ratio);
479
- }
480
- sender_unlock(s);
481
-
482
- worker_set_metric(WORKER_SENDER_JOB_BUFFER_RATIO, (NETDATA_DOUBLE)(s->buffer->max_size - available) * 100.0 / (NETDATA_DOUBLE)s->buffer->max_size);
483
-
484
- if(outstanding)
485
- s->send_attempts++;
486
-
487
- if(unlikely(s->rrdpush_sender_pipe[PIPE_READ] == -1)) {
488
- if(!rrdpush_sender_pipe_close(s->host, s->rrdpush_sender_pipe, true)) {
489
- netdata_log_error("STREAM %s [send]: cannot create inter-thread communication pipe. "
490
- "Disabling streaming.", rrdhost_hostname(s->host));
491
- rrdpush_sender_thread_close_socket(s);
492
- break;
493
- }
494
- }
495
-
496
- worker_is_idle();
497
-
498
- // Wait until buffer opens in the socket or a rrdset_done_push wakes us
499
- enum {
500
- Collector = 0,
501
- Socket = 1,
502
- };
503
- struct pollfd fds[2] = {
504
- [Collector] = {
505
- .fd = s->rrdpush_sender_pipe[PIPE_READ],
506
- .events = POLLIN,
507
- .revents = 0,
508
- },
509
- [Socket] = {
510
- .fd = s->rrdpush_sender_socket,
511
- .events = POLLIN | (outstanding ? POLLOUT : 0 ),
512
- .revents = 0,
513
- }
514
- };
515
-
516
- int poll_rc = poll(fds, 2, 50); // timeout in milliseconds
517
-
518
- netdata_log_debug(D_STREAM, "STREAM: poll() finished collector=%d socket=%d (current chunk %zu bytes)...",
519
- fds[Collector].revents, fds[Socket].revents, outstanding);
520
-
521
- if(unlikely(rrdhost_sender_should_exit(s)))
522
- break;
523
-
524
- internal_error(fds[Collector].fd != s->rrdpush_sender_pipe[PIPE_READ],
525
- "STREAM %s [send to %s]: pipe changed after poll().", rrdhost_hostname(s->host), s->connected_to);
526
-
527
- internal_error(fds[Socket].fd != s->rrdpush_sender_socket,
528
- "STREAM %s [send to %s]: socket changed after poll().", rrdhost_hostname(s->host), s->connected_to);
529
-
530
- // Spurious wake-ups without error - loop again
531
- if (poll_rc == 0 || ((poll_rc == -1) && (errno == EAGAIN || errno == EINTR))) {
532
- netdata_log_debug(D_STREAM, "Spurious wakeup");
533
- now_s = now_monotonic_sec();
534
- continue;
535
- }
536
-
537
- // Only errors from poll() are internal, but try restarting the connection
538
- if(unlikely(poll_rc == -1)) {
539
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_POLL_ERROR);
540
- netdata_log_error("STREAM %s [send to %s]: failed to poll(). Closing socket.", rrdhost_hostname(s->host), s->connected_to);
541
- rrdpush_sender_pipe_close(s->host, s->rrdpush_sender_pipe, true);
542
- rrdpush_sender_thread_close_socket(s);
543
- continue;
544
- }
545
-
546
- // If we have data and have seen the TCP window open then try to close it by a transmission.
547
- if(likely(outstanding && (fds[Socket].revents & POLLOUT))) {
548
- worker_is_busy(WORKER_SENDER_JOB_SOCKET_SEND);
549
- ssize_t bytes = attempt_to_send(s);
550
- if(bytes > 0) {
551
- s->last_traffic_seen_t = now_monotonic_sec();
552
- worker_set_metric(WORKER_SENDER_JOB_BYTES_SENT, (NETDATA_DOUBLE)bytes);
553
- }
554
- }
555
-
556
- // If the collector woke us up then empty the pipe to remove the signal
557
- if (fds[Collector].revents & (POLLIN|POLLPRI)) {
558
- worker_is_busy(WORKER_SENDER_JOB_PIPE_READ);
559
- netdata_log_debug(D_STREAM, "STREAM: Data added to send buffer (current buffer chunk %zu bytes)...", outstanding);
560
-
561
- if (read(fds[Collector].fd, pipe_buffer, pipe_buffer_size) == -1)
562
- netdata_log_error("STREAM %s [send to %s]: cannot read from internal pipe.", rrdhost_hostname(s->host), s->connected_to);
563
- }
564
-
565
- // Read as much as possible to fill the buffer, split into full lines for execution.
566
- if (fds[Socket].revents & POLLIN) {
567
- worker_is_busy(WORKER_SENDER_JOB_SOCKET_RECEIVE);
568
- ssize_t bytes = attempt_read(s);
569
- if(bytes > 0) {
570
- s->last_traffic_seen_t = now_monotonic_sec();
571
- worker_set_metric(WORKER_SENDER_JOB_BYTES_RECEIVED, (NETDATA_DOUBLE)bytes);
572
- }
573
- }
574
-
575
- if(unlikely(s->read_len))
576
- rrdpush_sender_execute_commands(s);
577
-
578
- if(unlikely(fds[Collector].revents & (POLLERR|POLLHUP|POLLNVAL))) {
579
- char *error = NULL;
580
-
581
- if (unlikely(fds[Collector].revents & POLLERR))
582
- error = "pipe reports errors (POLLERR)";
583
- else if (unlikely(fds[Collector].revents & POLLHUP))
584
- error = "pipe closed (POLLHUP)";
585
- else if (unlikely(fds[Collector].revents & POLLNVAL))
586
- error = "pipe is invalid (POLLNVAL)";
587
-
588
- if(error) {
589
- rrdpush_sender_pipe_close(s->host, s->rrdpush_sender_pipe, true);
590
- netdata_log_error("STREAM %s [send to %s]: restarting internal pipe: %s.",
591
- rrdhost_hostname(s->host), s->connected_to, error);
592
- }
593
- }
594
-
595
- if(unlikely(fds[Socket].revents & (POLLERR|POLLHUP|POLLNVAL))) {
596
- char *error = NULL;
597
-
598
- if (unlikely(fds[Socket].revents & POLLERR))
599
- error = "socket reports errors (POLLERR)";
600
- else if (unlikely(fds[Socket].revents & POLLHUP))
601
- error = "connection closed by remote end (POLLHUP)";
602
- else if (unlikely(fds[Socket].revents & POLLNVAL))
603
- error = "connection is invalid (POLLNVAL)";
604
-
605
- if(unlikely(error)) {
606
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SOCKET_ERROR);
607
- netdata_log_error("STREAM %s [send to %s]: restarting connection: %s - %zu bytes transmitted.",
608
- rrdhost_hostname(s->host), s->connected_to, error, s->sent_bytes_on_this_connection);
609
- rrdpush_sender_thread_close_socket(s);
610
- }
611
- }
612
-
613
- // protection from overflow
614
- if(unlikely(s->flags & SENDER_FLAG_OVERFLOW)) {
615
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_OVERFLOW);
616
- errno_clear();
617
- netdata_log_error("STREAM %s [send to %s]: buffer full (allocated %zu bytes) after sending %zu bytes. Restarting connection",
618
- rrdhost_hostname(s->host), s->connected_to, s->buffer->size, s->sent_bytes_on_this_connection);
619
- rrdpush_sender_thread_close_socket(s);
620
- }
621
-
622
- worker_set_metric(WORKER_SENDER_JOB_REPLAY_DICT_SIZE, (NETDATA_DOUBLE) dictionary_entries(s->replication.requests));
623
- }
624
-
625
- if(was_connected)
626
- rrdpush_sender_on_disconnect(s->host);
627
-
628
- netdata_log_info("STREAM %s [send]: sending thread exits %s",
629
- rrdhost_hostname(s->host),
630
- s->exit.reason != STREAM_HANDSHAKE_NEVER ? stream_handshake_error_to_string(s->exit.reason) : "");
631
-
632
- sender_lock(s);
633
- {
634
- rrdpush_sender_thread_close_socket(s);
635
- rrdpush_sender_pipe_close(s->host, s->rrdpush_sender_pipe, false);
636
- rrdpush_sender_execute_commands_cleanup(s);
637
-
638
- rrdhost_clear_sender___while_having_sender_mutex(s->host);
639
-
640
-#ifdef NETDATA_LOG_STREAM_SENDER
641
- if (s->stream_log_fp) {
642
- fclose(s->stream_log_fp);
643
- s->stream_log_fp = NULL;
644
- }
645
-#endif
646
- }
647
- sender_unlock(s);
648
-
649
- freez(pipe_buffer);
650
- worker_unregister();
651
-
652
- return NULL;
653
-}
654
-
655
-void rrdpush_sender_thread_spawn(RRDHOST *host) {
656
- sender_lock(host->sender);
657
-
658
- if(!rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN)) {
659
- char tag[NETDATA_THREAD_TAG_MAX + 1];
660
- snprintfz(tag, NETDATA_THREAD_TAG_MAX, THREAD_TAG_STREAM_SENDER "[%s]", rrdhost_hostname(host));
661
-
662
- host->rrdpush_sender_thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_DEFAULT,
663
- rrdpush_sender_thread, (void *)host->sender);
664
- if(!host->rrdpush_sender_thread)
665
- nd_log_daemon(NDLP_ERR, "STREAM %s [send]: failed to create new thread for client.", rrdhost_hostname(host));
666
- else
667
- rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN);
668
- }
669
-
670
- sender_unlock(host->sender);
671
-}
src/streaming/sender.h
deleted
-169
@@ -1,169 +0,0 @@
1
-// SPDX-License-Identifier: GPL-3.0-or-later
2
-
3
-#ifndef NETDATA_SENDER_H
4
-#define NETDATA_SENDER_H
5
-
6
-#include "libnetdata/libnetdata.h"
7
-
8
-#define CONNECTED_TO_SIZE 100
9
-
10
-#define CBUFFER_INITIAL_SIZE (16 * 1024)
11
-#define THREAD_BUFFER_INITIAL_SIZE (CBUFFER_INITIAL_SIZE / 2)
12
-
13
-typedef enum __attribute__((packed)) {
14
- STREAM_TRAFFIC_TYPE_REPLICATION = 0,
15
- STREAM_TRAFFIC_TYPE_FUNCTIONS,
16
- STREAM_TRAFFIC_TYPE_METADATA,
17
- STREAM_TRAFFIC_TYPE_DATA,
18
- STREAM_TRAFFIC_TYPE_DYNCFG,
19
-
20
- // terminator
21
- STREAM_TRAFFIC_TYPE_MAX,
22
-} STREAM_TRAFFIC_TYPE;
23
-
24
-typedef enum __attribute__((packed)) {
25
- SENDER_FLAG_OVERFLOW = (1 << 0), // The buffer has been overflown
26
-} SENDER_FLAGS;
27
-
28
-typedef struct {
29
- char *os_name;
30
- char *os_id;
31
- char *os_version;
32
- char *kernel_name;
33
- char *kernel_version;
34
-} stream_encoded_t;
35
-
36
-#include "stream-handshake.h"
37
-#include "stream-capabilities.h"
38
-#include "stream-conf.h"
39
-#include "stream-compression/compression.h"
40
-
41
-#include "sender-destinations.h"
42
-
43
-typedef void (*rrdpush_defer_action_t)(struct sender_state *s, void *data);
44
-typedef void (*rrdpush_defer_cleanup_t)(struct sender_state *s, void *data);
45
-
46
-struct sender_state {
47
- RRDHOST *host;
48
- pid_t tid; // the thread id of the sender, from gettid_cached()
49
- SENDER_FLAGS flags;
50
- int timeout;
51
- int default_port;
52
- uint32_t reconnect_delay;
53
- char connected_to[CONNECTED_TO_SIZE + 1]; // We don't know which proxy we connect to, passed back from socket.c
54
- size_t begin;
55
- size_t reconnects_counter;
56
- size_t sent_bytes;
57
- size_t sent_bytes_on_this_connection;
58
- size_t send_attempts;
59
- time_t last_traffic_seen_t;
60
- time_t last_state_since_t; // the timestamp of the last state (online/offline) change
61
- size_t not_connected_loops;
62
- // Metrics are collected asynchronously by collector threads calling rrdset_done_push(). This can also trigger
63
- // the lazy creation of the sender thread - both cases (buffer access and thread creation) are guarded here.
64
- SPINLOCK spinlock;
65
- struct circular_buffer *buffer;
66
- char read_buffer[PLUGINSD_LINE_MAX + 1];
67
- ssize_t read_len;
68
- STREAM_CAPABILITIES capabilities;
69
- STREAM_CAPABILITIES disabled_capabilities;
70
-
71
- size_t sent_bytes_on_this_connection_per_type[STREAM_TRAFFIC_TYPE_MAX];
72
-
73
- int rrdpush_sender_pipe[2]; // collector to sender thread signaling
74
- int rrdpush_sender_socket;
75
-
76
- uint16_t hops;
77
-
78
- struct line_splitter line;
79
- struct compressor_state compressor;
80
-
81
-#ifdef NETDATA_LOG_STREAM_SENDER
82
- FILE *stream_log_fp;
83
-#endif
84
-
85
- NETDATA_SSL ssl; // structure used to encrypt the connection
86
-
87
- struct {
88
- bool shutdown;
89
- STREAM_HANDSHAKE reason;
90
- } exit;
91
-
92
- struct {
93
- DICTIONARY *requests; // de-duplication of replication requests, per chart
94
- time_t oldest_request_after_t; // the timestamp of the oldest replication request
95
- time_t latest_completed_before_t; // the timestamp of the latest replication request
96
-
97
- struct {
98
- size_t pending_requests; // the currently outstanding replication requests
99
- size_t charts_replicating; // the number of unique charts having pending replication requests (on every request one is added and is removed when we finish it - it does not track completion of the replication for this chart)
100
- bool reached_max; // true when the sender buffer should not get more replication responses
101
- } atomic;
102
-
103
- } replication;
104
-
105
- struct {
106
- bool pending_data;
107
- size_t buffer_used_percentage; // the current utilization of the sending buffer
108
- usec_t last_flush_time_ut; // the last time the sender flushed the sending buffer in USEC
109
- time_t last_buffer_recreate_s; // true when the sender buffer should be re-created
110
- } atomic;
111
-
112
- struct {
113
- const char *end_keyword;
114
- BUFFER *payload;
115
- rrdpush_defer_action_t action;
116
- rrdpush_defer_cleanup_t cleanup;
117
- void *action_data;
118
- } defer;
119
-
120
- bool parent_using_h2o;
121
-};
122
-
123
-#define sender_lock(sender) spinlock_lock(&(sender)->spinlock)
124
-#define sender_unlock(sender) spinlock_unlock(&(sender)->spinlock)
125
-
126
-#define rrdpush_sender_pipe_has_pending_data(sender) __atomic_load_n(&(sender)->atomic.pending_data, __ATOMIC_RELAXED)
127
-#define rrdpush_sender_pipe_set_pending_data(sender) __atomic_store_n(&(sender)->atomic.pending_data, true, __ATOMIC_RELAXED)
128
-#define rrdpush_sender_pipe_clear_pending_data(sender) __atomic_store_n(&(sender)->atomic.pending_data, false, __ATOMIC_RELAXED)
129
-
130
-#define rrdpush_sender_last_buffer_recreate_get(sender) __atomic_load_n(&(sender)->atomic.last_buffer_recreate_s, __ATOMIC_RELAXED)
131
-#define rrdpush_sender_last_buffer_recreate_set(sender, value) __atomic_store_n(&(sender)->atomic.last_buffer_recreate_s, value, __ATOMIC_RELAXED)
132
-
133
-#define rrdpush_sender_replication_buffer_full_set(sender, value) __atomic_store_n(&((sender)->replication.atomic.reached_max), value, __ATOMIC_SEQ_CST)
134
-#define rrdpush_sender_replication_buffer_full_get(sender) __atomic_load_n(&((sender)->replication.atomic.reached_max), __ATOMIC_SEQ_CST)
135
-
136
-#define rrdpush_sender_set_buffer_used_percent(sender, value) __atomic_store_n(&((sender)->atomic.buffer_used_percentage), value, __ATOMIC_RELAXED)
137
-#define rrdpush_sender_get_buffer_used_percent(sender) __atomic_load_n(&((sender)->atomic.buffer_used_percentage), __ATOMIC_RELAXED)
138
-
139
-#define rrdpush_sender_set_flush_time(sender) __atomic_store_n(&((sender)->atomic.last_flush_time_ut), now_realtime_usec(), __ATOMIC_RELAXED)
140
-#define rrdpush_sender_get_flush_time(sender) __atomic_load_n(&((sender)->atomic.last_flush_time_ut), __ATOMIC_RELAXED)
141
-
142
-#define rrdpush_sender_replicating_charts(sender) __atomic_load_n(&((sender)->replication.atomic.charts_replicating), __ATOMIC_RELAXED)
143
-#define rrdpush_sender_replicating_charts_plus_one(sender) __atomic_add_fetch(&((sender)->replication.atomic.charts_replicating), 1, __ATOMIC_RELAXED)
144
-#define rrdpush_sender_replicating_charts_minus_one(sender) __atomic_sub_fetch(&((sender)->replication.atomic.charts_replicating), 1, __ATOMIC_RELAXED)
145
-#define rrdpush_sender_replicating_charts_zero(sender) __atomic_store_n(&((sender)->replication.atomic.charts_replicating), 0, __ATOMIC_RELAXED)
146
-
147
-#define rrdpush_sender_pending_replication_requests(sender) __atomic_load_n(&((sender)->replication.atomic.pending_requests), __ATOMIC_RELAXED)
148
-#define rrdpush_sender_pending_replication_requests_plus_one(sender) __atomic_add_fetch(&((sender)->replication.atomic.pending_requests), 1, __ATOMIC_RELAXED)
149
-#define rrdpush_sender_pending_replication_requests_minus_one(sender) __atomic_sub_fetch(&((sender)->replication.atomic.pending_requests), 1, __ATOMIC_RELAXED)
150
-#define rrdpush_sender_pending_replication_requests_zero(sender) __atomic_store_n(&((sender)->replication.atomic.pending_requests), 0, __ATOMIC_RELAXED)
151
-
152
-BUFFER *sender_start(struct sender_state *s);
153
-void sender_commit(struct sender_state *s, BUFFER *wb, STREAM_TRAFFIC_TYPE type);
154
-
155
-void *rrdpush_sender_thread(void *ptr);
156
-void rrdpush_sender_thread_stop(RRDHOST *host, STREAM_HANDSHAKE reason, bool wait);
157
-
158
-void sender_thread_buffer_free(void);
159
-
160
-void rrdpush_signal_sender_to_wake_up(struct sender_state *s);
161
-
162
-bool rrdpush_sender_connect(struct sender_state *s);
163
-void rrdpush_sender_cbuffer_recreate_timed(struct sender_state *s, time_t now_s, bool have_mutex, bool force);
164
-bool rrdhost_sender_should_exit(struct sender_state *s);
165
-void rrdpush_sender_thread_spawn(RRDHOST *host);
166
-
167
-#include "replication.h"
168
-
169
-#endif //NETDATA_SENDER_H
src/streaming/stream-capabilities.c
+17
-9
@@ -1,13 +1,18 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#include "rrdpush.h"
3
+#include "stream.h"
4
+#include "stream-receiver-internals.h"
5
+#include "stream-sender-internals.h"
6
5
-static STREAM_CAPABILITIES globally_disabled_capabilities = STREAM_CAP_NONE;
7
+static STREAM_CAPABILITIES globally_disabled_capabilities = STREAM_CAP_ALWAYS_DISABLED;
8
9
static struct {
10
STREAM_CAPABILITIES cap;
11
const char *str;
12
} capability_names[] = {
13
+ // DO NOT CHANGE NAMES
14
+ // THEY ARE USED BY STREAM_PATH, SO CONNECTING OF DIFFERENT NODES WILL BREAK
15
+
16
{STREAM_CAP_V1, "V1" },
17
{STREAM_CAP_V2, "V2" },
18
{STREAM_CAP_VN, "VN" },
@@ -21,7 +26,8 @@ static struct {
26
{STREAM_CAP_BINARY, "BINARY" },
27
{STREAM_CAP_INTERPOLATED, "INTERPOLATED" },
28
{STREAM_CAP_IEEE754, "IEEE754" },
24
- {STREAM_CAP_DATA_WITH_ML, "ML" },
29
+ {STREAM_CAP_DATA_WITH_ML, "ML"}, // do not remove this - stream_path fails to parse old nodes
30
+ {STREAM_CAP_ML_MODELS, "MLMODELS" },
31
{STREAM_CAP_DYNCFG, "DYNCFG" },
32
{STREAM_CAP_SLOTS, "SLOTS" },
33
{STREAM_CAP_ZSTD, "ZSTD" },
@@ -30,6 +36,8 @@ static struct {
36
{STREAM_CAP_PROGRESS, "PROGRESS" },
37
{STREAM_CAP_NODE_ID, "NODEID" },
38
{STREAM_CAP_PATHS, "PATHS" },
39
+
40
+ // terminator
41
{0 , NULL },
42
};
43
@@ -95,12 +103,12 @@ STREAM_CAPABILITIES stream_our_capabilities(RRDHOST *host, bool sender) {
103
// we have DATA_WITH_ML capability
104
// we should remove the DATA_WITH_ML capability if our database does not have anomaly info
105
// this can happen under these conditions: 1. we don't run ML, and 2. we don't receive ML
98
- spinlock_lock(&host->receiver_lock);
106
+ rrdhost_receiver_lock(host);
107
100
- if(!ml_host_running(host) && !stream_has_capability(host->receiver, STREAM_CAP_DATA_WITH_ML))
101
- disabled_capabilities |= STREAM_CAP_DATA_WITH_ML;
108
+ if (!ml_host_running(host) && !stream_has_capability(host->receiver, STREAM_CAP_ML_MODELS))
109
+ disabled_capabilities |= STREAM_CAP_ML_MODELS;
110
103
- spinlock_unlock(&host->receiver_lock);
111
+ rrdhost_receiver_unlock(host);
112
113
if(host->sender)
114
disabled_capabilities |= host->sender->disabled_capabilities;
@@ -124,7 +132,7 @@ STREAM_CAPABILITIES stream_our_capabilities(RRDHOST *host, bool sender) {
132
STREAM_CAP_NODE_ID |
133
STREAM_CAP_PATHS |
134
STREAM_CAP_IEEE754 |
127
- STREAM_CAP_DATA_WITH_ML |
135
+ STREAM_CAP_ML_MODELS |
136
0) & ~disabled_capabilities;
137
}
138
@@ -151,7 +159,7 @@ STREAM_CAPABILITIES convert_stream_version_to_capabilities(int32_t version, RRDH
159
160
if(!(common_caps & STREAM_CAP_INTERPOLATED))
161
// DATA WITH ML requires INTERPOLATED
154
- common_caps &= ~STREAM_CAP_DATA_WITH_ML;
162
+ common_caps &= ~(STREAM_CAP_ML_MODELS);
163
164
return common_caps;
165
}
src/streaming/stream-capabilities.h
+7
-4
@@ -25,9 +25,9 @@ typedef enum {
25
STREAM_CAP_V1 = (1 << 3), // v1 = the oldest protocol
26
STREAM_CAP_V2 = (1 << 4), // v2 = the second version of the protocol (with host labels)
27
STREAM_CAP_VN = (1 << 5), // version negotiation supported (for versions 3, 4, 5 of the protocol)
28
- // v3 = claiming supported
29
- // v4 = chart labels supported
30
- // v5 = lz4 compression supported
28
+ // v3 = claiming supported
29
+ // v4 = chart labels supported
30
+ // v5 = lz4 compression supported
31
STREAM_CAP_VCAPS = (1 << 6), // capabilities negotiation supported
32
STREAM_CAP_HLABELS = (1 << 7), // host labels supported
33
STREAM_CAP_CLAIM = (1 << 8), // claiming supported
@@ -38,7 +38,7 @@ typedef enum {
38
STREAM_CAP_BINARY = (1 << 13), // streaming supports binary data
39
STREAM_CAP_INTERPOLATED = (1 << 14), // streaming supports interpolated streaming of values
40
STREAM_CAP_IEEE754 = (1 << 15), // streaming supports binary/hex transfer of double values
41
- STREAM_CAP_DATA_WITH_ML = (1 << 16), // streaming supports transferring anomaly bit
41
+ STREAM_CAP_DATA_WITH_ML = (1 << 16), // leave this unused for as long as possible - NOT USED, BUT KEEP IT
42
// STREAM_CAP_DYNCFG = (1 << 17), // leave this unused for as long as possible
43
STREAM_CAP_SLOTS = (1 << 18), // the sender can appoint a unique slot for each chart
44
STREAM_CAP_ZSTD = (1 << 19), // ZSTD compression supported
@@ -48,12 +48,15 @@ typedef enum {
48
STREAM_CAP_DYNCFG = (1 << 23), // support for DYNCFG
49
STREAM_CAP_NODE_ID = (1 << 24), // support for sending NODE_ID back to the child
50
STREAM_CAP_PATHS = (1 << 25), // support for sending PATHS upstream and downstream
51
+ STREAM_CAP_ML_MODELS = (1 << 26), // support for sending MODELS upstream
52
53
STREAM_CAP_INVALID = (1 << 30), // used as an invalid value for capabilities when this is set
54
// this must be signed int, so don't use the last bit
55
// needed for negotiating errors between parent and child
56
} STREAM_CAPABILITIES;
57
58
+#define STREAM_CAP_ALWAYS_DISABLED (STREAM_CAP_DATA_WITH_ML)
59
+
60
#ifdef ENABLE_LZ4
61
#define STREAM_CAP_LZ4_AVAILABLE STREAM_CAP_LZ4
62
#else
src/streaming/stream-compression/brotli.c
+14
-14
@@ -6,7 +6,7 @@
6
#include <brotli/encode.h>
7
#include <brotli/decode.h>
8
9
-void rrdpush_compressor_init_brotli(struct compressor_state *state) {
9
+void stream_compressor_init_brotli(struct compressor_state *state) {
10
if (!state->initialized) {
11
state->initialized = true;
12
state->stream = BrotliEncoderCreateInstance(NULL, NULL, NULL);
@@ -21,14 +21,14 @@ void rrdpush_compressor_init_brotli(struct compressor_state *state) {
21
}
22
}
23
24
-void rrdpush_compressor_destroy_brotli(struct compressor_state *state) {
24
+void stream_compressor_destroy_brotli(struct compressor_state *state) {
25
if (state->stream) {
26
BrotliEncoderDestroyInstance(state->stream);
27
state->stream = NULL;
28
}
29
}
30
31
-size_t rrdpush_compress_brotli(struct compressor_state *state, const char *data, size_t size, const char **out) {
31
+size_t stream_compress_brotli(struct compressor_state *state, const char *data, size_t size, const char **out) {
32
if (unlikely(!state || !size || !out))
33
return 0;
34
@@ -41,26 +41,26 @@ size_t rrdpush_compress_brotli(struct compressor_state *state, const char *data,
41
uint8_t *next_out = (uint8_t *)state->output.data;
42
43
if (!BrotliEncoderCompressStream(state->stream, BROTLI_OPERATION_FLUSH, &available_in, &next_in, &available_out, &next_out, NULL)) {
44
- netdata_log_error("STREAM: Brotli compression failed.");
44
+ netdata_log_error("STREAM_COMPRESS: Brotli compression failed.");
45
return 0;
46
}
47
48
if(available_in != 0) {
49
- netdata_log_error("STREAM: BrotliEncoderCompressStream() did not use all the input buffer, %zu bytes out of %zu remain",
49
+ netdata_log_error("STREAM_COMPRESS: BrotliEncoderCompressStream() did not use all the input buffer, %zu bytes out of %zu remain",
50
available_in, size);
51
return 0;
52
}
53
54
size_t compressed_size = state->output.size - available_out;
55
if(available_out == 0) {
56
- netdata_log_error("STREAM: BrotliEncoderCompressStream() needs a bigger output buffer than the one we provided "
56
+ netdata_log_error("STREAM_COMPRESS: BrotliEncoderCompressStream() needs a bigger output buffer than the one we provided "
57
"(output buffer %zu bytes, compressed payload %zu bytes)",
58
state->output.size, size);
59
return 0;
60
}
61
62
if(compressed_size == 0) {
63
- netdata_log_error("STREAM: BrotliEncoderCompressStream() did not produce any output from the input provided "
63
+ netdata_log_error("STREAM_COMPRESS: BrotliEncoderCompressStream() did not produce any output from the input provided "
64
"(input buffer %zu bytes)",
65
size);
66
return 0;
@@ -74,7 +74,7 @@ size_t rrdpush_compress_brotli(struct compressor_state *state, const char *data,
74
return compressed_size;
75
}
76
77
-void rrdpush_decompressor_init_brotli(struct decompressor_state *state) {
77
+void stream_decompressor_init_brotli(struct decompressor_state *state) {
78
if (!state->initialized) {
79
state->initialized = true;
80
state->stream = BrotliDecoderCreateInstance(NULL, NULL, NULL);
@@ -83,14 +83,14 @@ void rrdpush_decompressor_init_brotli(struct decompressor_state *state) {
83
}
84
}
85
86
-void rrdpush_decompressor_destroy_brotli(struct decompressor_state *state) {
86
+void stream_decompressor_destroy_brotli(struct decompressor_state *state) {
87
if (state->stream) {
88
BrotliDecoderDestroyInstance(state->stream);
89
state->stream = NULL;
90
}
91
}
92
93
-size_t rrdpush_decompress_brotli(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
93
+size_t stream_decompress_brotli(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
94
if (unlikely(!state || !compressed_data || !compressed_size))
95
return 0;
96
@@ -104,26 +104,26 @@ size_t rrdpush_decompress_brotli(struct decompressor_state *state, const char *c
104
uint8_t *next_out = (uint8_t *)state->output.data;
105
106
if (BrotliDecoderDecompressStream(state->stream, &available_in, &next_in, &available_out, &next_out, NULL) == BROTLI_DECODER_RESULT_ERROR) {
107
- netdata_log_error("STREAM: Brotli decompression failed.");
107
+ netdata_log_error("STREAM_DECOMPRESS: Brotli decompression failed.");
108
return 0;
109
}
110
111
if(available_in != 0) {
112
- netdata_log_error("STREAM: BrotliDecoderDecompressStream() did not use all the input buffer, %zu bytes out of %zu remain",
112
+ netdata_log_error("STREAM_DECOMPRESS: BrotliDecoderDecompressStream() did not use all the input buffer, %zu bytes out of %zu remain",
113
available_in, compressed_size);
114
return 0;
115
}
116
117
size_t decompressed_size = state->output.size - available_out;
118
if(available_out == 0) {
119
- netdata_log_error("STREAM: BrotliDecoderDecompressStream() needs a bigger output buffer than the one we provided "
119
+ netdata_log_error("STREAM_DECOMPRESS: BrotliDecoderDecompressStream() needs a bigger output buffer than the one we provided "
120
"(output buffer %zu bytes, compressed payload %zu bytes)",
121
state->output.size, compressed_size);
122
return 0;
123
}
124
125
if(decompressed_size == 0) {
126
- netdata_log_error("STREAM: BrotliDecoderDecompressStream() did not produce any output from the input provided "
126
+ netdata_log_error("STREAM_DECOMPRESS: BrotliDecoderDecompressStream() did not produce any output from the input provided "
127
"(input buffer %zu bytes)",
128
compressed_size);
129
return 0;
src/streaming/stream-compression/brotli.h
+6
-6
@@ -5,11 +5,11 @@
5
#ifndef NETDATA_STREAMING_COMPRESSION_BROTLI_H
6
#define NETDATA_STREAMING_COMPRESSION_BROTLI_H
7
8
-void rrdpush_compressor_init_brotli(struct compressor_state *state);
9
-void rrdpush_compressor_destroy_brotli(struct compressor_state *state);
10
-size_t rrdpush_compress_brotli(struct compressor_state *state, const char *data, size_t size, const char **out);
11
-size_t rrdpush_decompress_brotli(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
12
-void rrdpush_decompressor_init_brotli(struct decompressor_state *state);
13
-void rrdpush_decompressor_destroy_brotli(struct decompressor_state *state);
8
+void stream_compressor_init_brotli(struct compressor_state *state);
9
+void stream_compressor_destroy_brotli(struct compressor_state *state);
10
+size_t stream_compress_brotli(struct compressor_state *state, const char *data, size_t size, const char **out);
11
+size_t stream_decompress_brotli(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
12
+void stream_decompressor_init_brotli(struct decompressor_state *state);
13
+void stream_decompressor_destroy_brotli(struct decompressor_state *state);
14
15
#endif //NETDATA_STREAMING_COMPRESSION_BROTLI_H
src/streaming/stream-compression/compression.c
+96
-101
@@ -1,6 +1,9 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "compression.h"
4
+#include "../stream-conf.h"
5
+#include "../stream-receiver-internals.h"
6
+#include "../stream-sender-internals.h"
7
8
#include "gzip.h"
9
@@ -16,18 +19,10 @@
19
#include "brotli.h"
20
#endif
21
19
-int rrdpush_compression_levels[COMPRESSION_ALGORITHM_MAX] = {
20
- [COMPRESSION_ALGORITHM_NONE] = 0,
21
- [COMPRESSION_ALGORITHM_ZSTD] = 3, // 1 (faster) - 22 (smaller)
22
- [COMPRESSION_ALGORITHM_LZ4] = 1, // 1 (smaller) - 9 (faster)
23
- [COMPRESSION_ALGORITHM_BROTLI] = 3, // 0 (faster) - 11 (smaller)
24
- [COMPRESSION_ALGORITHM_GZIP] = 1, // 1 (faster) - 9 (smaller)
25
-};
26
-
27
-void rrdpush_parse_compression_order(struct receiver_state *rpt, const char *order) {
22
+void stream_parse_compression_order(struct stream_receiver_config *config, const char *order) {
23
// empty all slots
24
for(size_t i = 0; i < COMPRESSION_ALGORITHM_MAX ;i++)
30
- rpt->config.compression_priorities[i] = STREAM_CAP_NONE;
25
+ config->compression.priorities[i] = STREAM_CAP_NONE;
26
27
char *s = strdupz(order);
28
@@ -37,19 +32,19 @@ void rrdpush_parse_compression_order(struct receiver_state *rpt, const char *ord
32
STREAM_CAPABILITIES added = STREAM_CAP_NONE;
33
for(size_t i = 0; i < num_words && slot < COMPRESSION_ALGORITHM_MAX ;i++) {
34
if((STREAM_CAP_ZSTD_AVAILABLE) && strcasecmp(words[i], "zstd") == 0 && !(added & STREAM_CAP_ZSTD)) {
40
- rpt->config.compression_priorities[slot++] = STREAM_CAP_ZSTD;
35
+ config->compression.priorities[slot++] = STREAM_CAP_ZSTD;
36
added |= STREAM_CAP_ZSTD;
37
}
38
else if((STREAM_CAP_LZ4_AVAILABLE) && strcasecmp(words[i], "lz4") == 0 && !(added & STREAM_CAP_LZ4)) {
44
- rpt->config.compression_priorities[slot++] = STREAM_CAP_LZ4;
39
+ config->compression.priorities[slot++] = STREAM_CAP_LZ4;
40
added |= STREAM_CAP_LZ4;
41
}
42
else if((STREAM_CAP_BROTLI_AVAILABLE) && strcasecmp(words[i], "brotli") == 0 && !(added & STREAM_CAP_BROTLI)) {
48
- rpt->config.compression_priorities[slot++] = STREAM_CAP_BROTLI;
43
+ config->compression.priorities[slot++] = STREAM_CAP_BROTLI;
44
added |= STREAM_CAP_BROTLI;
45
}
46
else if(strcasecmp(words[i], "gzip") == 0 && !(added & STREAM_CAP_GZIP)) {
52
- rpt->config.compression_priorities[slot++] = STREAM_CAP_GZIP;
47
+ config->compression.priorities[slot++] = STREAM_CAP_GZIP;
48
added |= STREAM_CAP_GZIP;
49
}
50
}
@@ -58,24 +53,24 @@ void rrdpush_parse_compression_order(struct receiver_state *rpt, const char *ord
53
54
// make sure all participate
55
if((STREAM_CAP_ZSTD_AVAILABLE) && slot < COMPRESSION_ALGORITHM_MAX && !(added & STREAM_CAP_ZSTD))
61
- rpt->config.compression_priorities[slot++] = STREAM_CAP_ZSTD;
56
+ config->compression.priorities[slot++] = STREAM_CAP_ZSTD;
57
if((STREAM_CAP_LZ4_AVAILABLE) && slot < COMPRESSION_ALGORITHM_MAX && !(added & STREAM_CAP_LZ4))
63
- rpt->config.compression_priorities[slot++] = STREAM_CAP_LZ4;
58
+ config->compression.priorities[slot++] = STREAM_CAP_LZ4;
59
if((STREAM_CAP_BROTLI_AVAILABLE) && slot < COMPRESSION_ALGORITHM_MAX && !(added & STREAM_CAP_BROTLI))
65
- rpt->config.compression_priorities[slot++] = STREAM_CAP_BROTLI;
60
+ config->compression.priorities[slot++] = STREAM_CAP_BROTLI;
61
if(slot < COMPRESSION_ALGORITHM_MAX && !(added & STREAM_CAP_GZIP))
67
- rpt->config.compression_priorities[slot++] = STREAM_CAP_GZIP;
62
+ config->compression.priorities[slot++] = STREAM_CAP_GZIP;
63
}
64
70
-void rrdpush_select_receiver_compression_algorithm(struct receiver_state *rpt) {
71
- if (!rpt->config.rrdpush_compression)
65
+void stream_select_receiver_compression_algorithm(struct receiver_state *rpt) {
66
+ if (!rpt->config.compression.enabled)
67
rpt->capabilities &= ~STREAM_CAP_COMPRESSIONS_AVAILABLE;
68
69
// select the right compression before sending our capabilities to the child
70
if(stream_has_more_than_one_capability_of(rpt->capabilities, STREAM_CAP_COMPRESSIONS_AVAILABLE)) {
71
STREAM_CAPABILITIES compressions = rpt->capabilities & STREAM_CAP_COMPRESSIONS_AVAILABLE;
72
for(int i = 0; i < COMPRESSION_ALGORITHM_MAX; i++) {
78
- STREAM_CAPABILITIES c = rpt->config.compression_priorities[i];
73
+ STREAM_CAPABILITIES c = rpt->config.compression.priorities[i];
74
75
if(!(c & STREAM_CAP_COMPRESSIONS_AVAILABLE))
76
continue;
@@ -91,8 +86,8 @@ void rrdpush_select_receiver_compression_algorithm(struct receiver_state *rpt) {
86
}
87
}
88
94
-bool rrdpush_compression_initialize(struct sender_state *s) {
95
- rrdpush_compressor_destroy(&s->compressor);
89
+bool stream_compression_initialize(struct sender_state *s) {
90
+ stream_compressor_destroy(&s->compressor);
91
92
// IMPORTANT
93
// KEEP THE SAME ORDER IN DECOMPRESSION
@@ -109,33 +104,33 @@ bool rrdpush_compression_initialize(struct sender_state *s) {
104
s->compressor.algorithm = COMPRESSION_ALGORITHM_NONE;
105
106
if(s->compressor.algorithm != COMPRESSION_ALGORITHM_NONE) {
112
- s->compressor.level = rrdpush_compression_levels[s->compressor.algorithm];
113
- rrdpush_compressor_init(&s->compressor);
107
+ s->compressor.level = stream_send.compression.levels[s->compressor.algorithm];
108
+ stream_compressor_init(&s->compressor);
109
return true;
110
}
111
112
return false;
113
}
114
120
-bool rrdpush_decompression_initialize(struct receiver_state *rpt) {
121
- rrdpush_decompressor_destroy(&rpt->decompressor);
115
+bool stream_decompression_initialize(struct receiver_state *rpt) {
116
+ stream_decompressor_destroy(&rpt->thread.compressed.decompressor);
117
118
// IMPORTANT
119
// KEEP THE SAME ORDER IN COMPRESSION
120
121
if(stream_has_capability(rpt, STREAM_CAP_ZSTD))
127
- rpt->decompressor.algorithm = COMPRESSION_ALGORITHM_ZSTD;
122
+ rpt->thread.compressed.decompressor.algorithm = COMPRESSION_ALGORITHM_ZSTD;
123
else if(stream_has_capability(rpt, STREAM_CAP_LZ4))
129
- rpt->decompressor.algorithm = COMPRESSION_ALGORITHM_LZ4;
124
+ rpt->thread.compressed.decompressor.algorithm = COMPRESSION_ALGORITHM_LZ4;
125
else if(stream_has_capability(rpt, STREAM_CAP_BROTLI))
131
- rpt->decompressor.algorithm = COMPRESSION_ALGORITHM_BROTLI;
126
+ rpt->thread.compressed.decompressor.algorithm = COMPRESSION_ALGORITHM_BROTLI;
127
else if(stream_has_capability(rpt, STREAM_CAP_GZIP))
133
- rpt->decompressor.algorithm = COMPRESSION_ALGORITHM_GZIP;
128
+ rpt->thread.compressed.decompressor.algorithm = COMPRESSION_ALGORITHM_GZIP;
129
else
135
- rpt->decompressor.algorithm = COMPRESSION_ALGORITHM_NONE;
130
+ rpt->thread.compressed.decompressor.algorithm = COMPRESSION_ALGORITHM_NONE;
131
137
- if(rpt->decompressor.algorithm != COMPRESSION_ALGORITHM_NONE) {
138
- rrdpush_decompressor_init(&rpt->decompressor);
132
+ if(rpt->thread.compressed.decompressor.algorithm != COMPRESSION_ALGORITHM_NONE) {
133
+ stream_decompressor_init(&rpt->thread.compressed.decompressor);
134
return true;
135
}
136
@@ -147,7 +142,7 @@ bool rrdpush_decompression_initialize(struct receiver_state *rpt) {
142
* Inform the user through the error log file and
143
* deactivate compression by downgrading the stream protocol.
144
*/
150
-void rrdpush_compression_deactivate(struct sender_state *s) {
145
+void stream_compression_deactivate(struct sender_state *s) {
146
switch(s->compressor.algorithm) {
147
case COMPRESSION_ALGORITHM_MAX:
148
case COMPRESSION_ALGORITHM_NONE:
@@ -184,29 +179,29 @@ void rrdpush_compression_deactivate(struct sender_state *s) {
179
// ----------------------------------------------------------------------------
180
// compressor public API
181
187
-void rrdpush_compressor_init(struct compressor_state *state) {
182
+void stream_compressor_init(struct compressor_state *state) {
183
switch(state->algorithm) {
184
#ifdef ENABLE_ZSTD
185
case COMPRESSION_ALGORITHM_ZSTD:
191
- rrdpush_compressor_init_zstd(state);
186
+ stream_compressor_init_zstd(state);
187
break;
188
#endif
189
190
#ifdef ENABLE_LZ4
191
case COMPRESSION_ALGORITHM_LZ4:
197
- rrdpush_compressor_init_lz4(state);
192
+ stream_compressor_init_lz4(state);
193
break;
194
#endif
195
196
#ifdef ENABLE_BROTLI
197
case COMPRESSION_ALGORITHM_BROTLI:
203
- rrdpush_compressor_init_brotli(state);
198
+ stream_compressor_init_brotli(state);
199
break;
200
#endif
201
202
default:
203
case COMPRESSION_ALGORITHM_GZIP:
209
- rrdpush_compressor_init_gzip(state);
204
+ stream_compressor_init_gzip(state);
205
break;
206
}
207
@@ -214,29 +209,29 @@ void rrdpush_compressor_init(struct compressor_state *state) {
209
simple_ring_buffer_reset(&state->output);
210
}
211
217
-void rrdpush_compressor_destroy(struct compressor_state *state) {
212
+void stream_compressor_destroy(struct compressor_state *state) {
213
switch(state->algorithm) {
214
#ifdef ENABLE_ZSTD
215
case COMPRESSION_ALGORITHM_ZSTD:
221
- rrdpush_compressor_destroy_zstd(state);
216
+ stream_compressor_destroy_zstd(state);
217
break;
218
#endif
219
220
#ifdef ENABLE_LZ4
221
case COMPRESSION_ALGORITHM_LZ4:
227
- rrdpush_compressor_destroy_lz4(state);
222
+ stream_compressor_destroy_lz4(state);
223
break;
224
#endif
225
226
#ifdef ENABLE_BROTLI
227
case COMPRESSION_ALGORITHM_BROTLI:
233
- rrdpush_compressor_destroy_brotli(state);
228
+ stream_compressor_destroy_brotli(state);
229
break;
230
#endif
231
232
default:
233
case COMPRESSION_ALGORITHM_GZIP:
239
- rrdpush_compressor_destroy_gzip(state);
234
+ stream_compressor_destroy_gzip(state);
235
break;
236
}
237
@@ -246,36 +241,36 @@ void rrdpush_compressor_destroy(struct compressor_state *state) {
241
simple_ring_buffer_destroy(&state->output);
242
}
243
249
-size_t rrdpush_compress(struct compressor_state *state, const char *data, size_t size, const char **out) {
244
+size_t stream_compress(struct compressor_state *state, const char *data, size_t size, const char **out) {
245
size_t ret = 0;
246
247
switch(state->algorithm) {
248
#ifdef ENABLE_ZSTD
249
case COMPRESSION_ALGORITHM_ZSTD:
255
- ret = rrdpush_compress_zstd(state, data, size, out);
250
+ ret = stream_compress_zstd(state, data, size, out);
251
break;
252
#endif
253
254
#ifdef ENABLE_LZ4
255
case COMPRESSION_ALGORITHM_LZ4:
261
- ret = rrdpush_compress_lz4(state, data, size, out);
256
+ ret = stream_compress_lz4(state, data, size, out);
257
break;
258
#endif
259
260
#ifdef ENABLE_BROTLI
261
case COMPRESSION_ALGORITHM_BROTLI:
267
- ret = rrdpush_compress_brotli(state, data, size, out);
262
+ ret = stream_compress_brotli(state, data, size, out);
263
break;
264
#endif
265
266
default:
267
case COMPRESSION_ALGORITHM_GZIP:
273
- ret = rrdpush_compress_gzip(state, data, size, out);
268
+ ret = stream_compress_gzip(state, data, size, out);
269
break;
270
}
271
272
if(unlikely(ret >= COMPRESSION_MAX_CHUNK)) {
278
- netdata_log_error("RRDPUSH_COMPRESS: compressed data is %zu bytes, which is >= than the max chunk size %d",
273
+ netdata_log_error("STREAM_COMPRESS: compressed data is %zu bytes, which is >= than the max chunk size %d",
274
ret, COMPRESSION_MAX_CHUNK);
275
return 0;
276
}
@@ -286,32 +281,32 @@ size_t rrdpush_compress(struct compressor_state *state, const char *data, size_t
281
// ----------------------------------------------------------------------------
282
// decompressor public API
283
289
-void rrdpush_decompressor_destroy(struct decompressor_state *state) {
284
+void stream_decompressor_destroy(struct decompressor_state *state) {
285
if(unlikely(!state->initialized))
286
return;
287
288
switch(state->algorithm) {
289
#ifdef ENABLE_ZSTD
290
case COMPRESSION_ALGORITHM_ZSTD:
296
- rrdpush_decompressor_destroy_zstd(state);
291
+ stream_decompressor_destroy_zstd(state);
292
break;
293
#endif
294
295
#ifdef ENABLE_LZ4
296
case COMPRESSION_ALGORITHM_LZ4:
302
- rrdpush_decompressor_destroy_lz4(state);
297
+ stream_decompressor_destroy_lz4(state);
298
break;
299
#endif
300
301
#ifdef ENABLE_BROTLI
302
case COMPRESSION_ALGORITHM_BROTLI:
308
- rrdpush_decompressor_destroy_brotli(state);
303
+ stream_decompressor_destroy_brotli(state);
304
break;
305
#endif
306
307
default:
308
case COMPRESSION_ALGORITHM_GZIP:
314
- rrdpush_decompressor_destroy_gzip(state);
309
+ stream_decompressor_destroy_gzip(state);
310
break;
311
}
312
@@ -320,71 +315,71 @@ void rrdpush_decompressor_destroy(struct decompressor_state *state) {
315
state->initialized = false;
316
}
317
323
-void rrdpush_decompressor_init(struct decompressor_state *state) {
318
+void stream_decompressor_init(struct decompressor_state *state) {
319
switch(state->algorithm) {
320
#ifdef ENABLE_ZSTD
321
case COMPRESSION_ALGORITHM_ZSTD:
327
- rrdpush_decompressor_init_zstd(state);
322
+ stream_decompressor_init_zstd(state);
323
break;
324
#endif
325
326
#ifdef ENABLE_LZ4
327
case COMPRESSION_ALGORITHM_LZ4:
333
- rrdpush_decompressor_init_lz4(state);
328
+ stream_decompressor_init_lz4(state);
329
break;
330
#endif
331
332
#ifdef ENABLE_BROTLI
333
case COMPRESSION_ALGORITHM_BROTLI:
339
- rrdpush_decompressor_init_brotli(state);
334
+ stream_decompressor_init_brotli(state);
335
break;
336
#endif
337
338
default:
339
case COMPRESSION_ALGORITHM_GZIP:
345
- rrdpush_decompressor_init_gzip(state);
340
+ stream_decompressor_init_gzip(state);
341
break;
342
}
343
349
- state->signature_size = RRDPUSH_COMPRESSION_SIGNATURE_SIZE;
344
+ state->signature_size = STREAM_COMPRESSION_SIGNATURE_SIZE;
345
simple_ring_buffer_reset(&state->output);
346
}
347
353
-size_t rrdpush_decompress(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
348
+size_t stream_decompress(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
349
if (unlikely(state->output.read_pos != state->output.write_pos))
355
- fatal("RRDPUSH_DECOMPRESS: asked to decompress new data, while there are unread data in the decompression buffer!");
350
+ fatal("STREAM_DECOMPRESS: asked to decompress new data, while there are unread data in the decompression buffer!");
351
352
size_t ret = 0;
353
354
switch(state->algorithm) {
355
#ifdef ENABLE_ZSTD
356
case COMPRESSION_ALGORITHM_ZSTD:
362
- ret = rrdpush_decompress_zstd(state, compressed_data, compressed_size);
357
+ ret = stream_decompress_zstd(state, compressed_data, compressed_size);
358
break;
359
#endif
360
361
#ifdef ENABLE_LZ4
362
case COMPRESSION_ALGORITHM_LZ4:
368
- ret = rrdpush_decompress_lz4(state, compressed_data, compressed_size);
363
+ ret = stream_decompress_lz4(state, compressed_data, compressed_size);
364
break;
365
#endif
366
367
#ifdef ENABLE_BROTLI
368
case COMPRESSION_ALGORITHM_BROTLI:
374
- ret = rrdpush_decompress_brotli(state, compressed_data, compressed_size);
369
+ ret = stream_decompress_brotli(state, compressed_data, compressed_size);
370
break;
371
#endif
372
373
default:
374
case COMPRESSION_ALGORITHM_GZIP:
380
- ret = rrdpush_decompress_gzip(state, compressed_data, compressed_size);
375
+ ret = stream_decompress_gzip(state, compressed_data, compressed_size);
376
break;
377
}
378
379
// for backwards compatibility we cannot check for COMPRESSION_MAX_MSG_SIZE,
380
// because old children may send this big payloads.
381
if(unlikely(ret > COMPRESSION_MAX_CHUNK)) {
387
- netdata_log_error("RRDPUSH_DECOMPRESS: decompressed data is %zu bytes, which is bigger than the max msg size %d",
382
+ netdata_log_error("STREAM_DECOMPRESS: decompressed data is %zu bytes, which is bigger than the max msg size %d",
383
ret, COMPRESSION_MAX_CHUNK);
384
return 0;
385
}
@@ -478,7 +473,7 @@ void unittest_generate_message(BUFFER *wb, time_t now_s, size_t counter) {
473
buffer_fast_strcat(wb, PLUGINSD_KEYWORD_END_V2 "\n", sizeof(PLUGINSD_KEYWORD_END_V2) - 1 + 1);
474
}
475
481
-int unittest_rrdpush_compression_speed(compression_algorithm_t algorithm, const char *name) {
476
+int unittest_stream_compression_speed(compression_algorithm_t algorithm, const char *name) {
477
fprintf(stderr, "\nTesting streaming compression speed with %s\n", name);
478
479
struct compressor_state cctx = {
@@ -490,8 +485,8 @@ int unittest_rrdpush_compression_speed(compression_algorithm_t algorithm, const
485
.algorithm = algorithm,
486
};
487
493
- rrdpush_compressor_init(&cctx);
494
- rrdpush_decompressor_init(&dctx);
488
+ stream_compressor_init(&cctx);
489
+ stream_decompressor_init(&dctx);
490
491
int errors = 0;
492
@@ -518,7 +513,7 @@ int unittest_rrdpush_compression_speed(compression_algorithm_t algorithm, const
513
bytes_uncompressed += txt_len;
514
515
const char *out;
521
- size_t size = rrdpush_compress(&cctx, txt, txt_len, &out);
516
+ size_t size = stream_compress(&cctx, txt, txt_len, &out);
517
518
bytes_compressed += size;
519
decompression_started_ut = now_monotonic_usec();
@@ -537,13 +532,12 @@ int unittest_rrdpush_compression_speed(compression_algorithm_t algorithm, const
532
goto cleanup;
533
}
534
else {
540
- size_t dtxt_len = rrdpush_decompress(&dctx, out, size);
535
+ size_t dtxt_len = stream_decompress(&dctx, out, size);
536
char *dtxt = (char *) &dctx.output.data[dctx.output.read_pos];
537
543
- if(rrdpush_decompressed_bytes_in_buffer(&dctx) != dtxt_len) {
544
- fprintf(stderr, "iteration %d: decompressed size %zu does not rrdpush_decompressed_bytes_in_buffer() %zu\n",
545
- i, dtxt_len, rrdpush_decompressed_bytes_in_buffer(&dctx)
546
- );
538
+ if(stream_decompressed_bytes_in_buffer(&dctx) != dtxt_len) {
539
+ fprintf(stderr, "iteration %d: decompressed size %zu does not stream_decompressed_bytes_in_buffer() %zu\n",
540
+ i, dtxt_len, stream_decompressed_bytes_in_buffer(&dctx));
541
errors++;
542
goto cleanup;
543
}
@@ -571,12 +565,12 @@ int unittest_rrdpush_compression_speed(compression_algorithm_t algorithm, const
565
}
566
567
// here we are supposed to copy the data and advance the position
574
- dctx.output.read_pos += rrdpush_decompressed_bytes_in_buffer(&dctx);
568
+ dctx.output.read_pos += stream_decompressed_bytes_in_buffer(&dctx);
569
}
570
571
cleanup:
578
- rrdpush_compressor_destroy(&cctx);
579
- rrdpush_decompressor_destroy(&dctx);
572
+ stream_compressor_destroy(&cctx);
573
+ stream_decompressor_destroy(&dctx);
574
575
if(errors)
576
fprintf(stderr, "Compression with %s: FAILED (%d errors)\n", name, errors);
@@ -590,7 +584,7 @@ cleanup:
584
return errors;
585
}
586
593
-int unittest_rrdpush_compression(compression_algorithm_t algorithm, const char *name) {
587
+int unittest_stream_compression(compression_algorithm_t algorithm, const char *name) {
588
fprintf(stderr, "\nTesting streaming compression with %s\n", name);
589
590
struct compressor_state cctx = {
@@ -604,8 +598,8 @@ int unittest_rrdpush_compression(compression_algorithm_t algorithm, const char *
598
599
char txt[COMPRESSION_MAX_MSG_SIZE];
600
607
- rrdpush_compressor_init(&cctx);
608
- rrdpush_decompressor_init(&dctx);
601
+ stream_compressor_init(&cctx);
602
+ stream_decompressor_init(&dctx);
603
604
int errors = 0;
605
@@ -616,7 +610,7 @@ int unittest_rrdpush_compression(compression_algorithm_t algorithm, const char *
610
size_t txt_len = i + 1;
611
612
const char *out;
619
- size_t size = rrdpush_compress(&cctx, txt, txt_len, &out);
613
+ size_t size = stream_compress(&cctx, txt, txt_len, &out);
614
615
if(size == 0) {
616
fprintf(stderr, "iteration %d: compressed size %zu is zero\n",
@@ -631,12 +625,13 @@ int unittest_rrdpush_compression(compression_algorithm_t algorithm, const char *
625
goto cleanup;
626
}
627
else {
634
- size_t dtxt_len = rrdpush_decompress(&dctx, out, size);
628
+ size_t dtxt_len = stream_decompress(&dctx, out, size);
629
char *dtxt = (char *) &dctx.output.data[dctx.output.read_pos];
630
637
- if(rrdpush_decompressed_bytes_in_buffer(&dctx) != dtxt_len) {
638
- fprintf(stderr, "iteration %d: decompressed size %zu does not rrdpush_decompressed_bytes_in_buffer() %zu\n",
639
- i, dtxt_len, rrdpush_decompressed_bytes_in_buffer(&dctx)
631
+ if(stream_decompressed_bytes_in_buffer(&dctx) != dtxt_len) {
632
+ fprintf(stderr, "iteration %d: decompressed size %zu does not stream_decompressed_bytes_in_buffer() %zu\n",
633
+ i, dtxt_len,
634
+ stream_decompressed_bytes_in_buffer(&dctx)
635
);
636
errors++;
637
goto cleanup;
@@ -671,12 +666,12 @@ int unittest_rrdpush_compression(compression_algorithm_t algorithm, const char *
666
memset((void *)out, 'x', size);
667
668
// here we are supposed to copy the data and advance the position
674
- dctx.output.read_pos += rrdpush_decompressed_bytes_in_buffer(&dctx);
669
+ dctx.output.read_pos += stream_decompressed_bytes_in_buffer(&dctx);
670
}
671
672
cleanup:
678
- rrdpush_compressor_destroy(&cctx);
679
- rrdpush_decompressor_destroy(&dctx);
673
+ stream_compressor_destroy(&cctx);
674
+ stream_decompressor_destroy(&dctx);
675
676
if(errors)
677
fprintf(stderr, "Compression with %s: FAILED (%d errors)\n", name, errors);
@@ -686,18 +681,18 @@ cleanup:
681
return errors;
682
}
683
689
-int unittest_rrdpush_compressions(void) {
684
+int unittest_stream_compressions(void) {
685
int ret = 0;
686
692
- ret += unittest_rrdpush_compression(COMPRESSION_ALGORITHM_ZSTD, "ZSTD");
693
- ret += unittest_rrdpush_compression(COMPRESSION_ALGORITHM_LZ4, "LZ4");
694
- ret += unittest_rrdpush_compression(COMPRESSION_ALGORITHM_BROTLI, "BROTLI");
695
- ret += unittest_rrdpush_compression(COMPRESSION_ALGORITHM_GZIP, "GZIP");
687
+ ret += unittest_stream_compression(COMPRESSION_ALGORITHM_ZSTD, "ZSTD");
688
+ ret += unittest_stream_compression(COMPRESSION_ALGORITHM_LZ4, "LZ4");
689
+ ret += unittest_stream_compression(COMPRESSION_ALGORITHM_BROTLI, "BROTLI");
690
+ ret += unittest_stream_compression(COMPRESSION_ALGORITHM_GZIP, "GZIP");
691
697
- ret += unittest_rrdpush_compression_speed(COMPRESSION_ALGORITHM_ZSTD, "ZSTD");
698
- ret += unittest_rrdpush_compression_speed(COMPRESSION_ALGORITHM_LZ4, "LZ4");
699
- ret += unittest_rrdpush_compression_speed(COMPRESSION_ALGORITHM_BROTLI, "BROTLI");
700
- ret += unittest_rrdpush_compression_speed(COMPRESSION_ALGORITHM_GZIP, "GZIP");
692
+ ret += unittest_stream_compression_speed(COMPRESSION_ALGORITHM_ZSTD, "ZSTD");
693
+ ret += unittest_stream_compression_speed(COMPRESSION_ALGORITHM_LZ4, "LZ4");
694
+ ret += unittest_stream_compression_speed(COMPRESSION_ALGORITHM_BROTLI, "BROTLI");
695
+ ret += unittest_stream_compression_speed(COMPRESSION_ALGORITHM_GZIP, "GZIP");
696
697
return ret;
698
}
src/streaming/stream-compression/compression.h
+37
-37
@@ -1,7 +1,7 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#ifndef NETDATA_RRDPUSH_COMPRESSION_H
4
-#define NETDATA_RRDPUSH_COMPRESSION_H 1
3
+#ifndef NETDATA_STREAM_COMPRESSION_H
4
+#define NETDATA_STREAM_COMPRESSION_H 1
5
6
#include "libnetdata/libnetdata.h"
7
@@ -11,14 +11,14 @@
11
#error "COMPRESSION_MAX_MSG_SIZE >= (COMPRESSION_MAX_CHUNK - COMPRESSION_MAX_OVERHEAD)"
12
#endif
13
14
-typedef uint32_t rrdpush_signature_t;
15
-#define RRDPUSH_COMPRESSION_SIGNATURE ((rrdpush_signature_t)('z' | 0x80) | (0x80 << 8) | (0x80 << 16) | ('\n' << 24))
16
-#define RRDPUSH_COMPRESSION_SIGNATURE_MASK ((rrdpush_signature_t) 0xffU | (0x80U << 8) | (0x80U << 16) | (0xffU << 24))
17
-#define RRDPUSH_COMPRESSION_SIGNATURE_SIZE sizeof(rrdpush_signature_t)
14
+typedef uint32_t stream_compression_signature_t;
15
+#define STREAM_COMPRESSION_SIGNATURE ((stream_compression_signature_t)('z' | 0x80) | (0x80 << 8) | (0x80 << 16) | ('\n' << 24))
16
+#define STREAM_COMPRESSION_SIGNATURE_MASK ((stream_compression_signature_t) 0xffU | (0x80U << 8) | (0x80U << 16) | (0xffU << 24))
17
+#define STREAM_COMPRESSION_SIGNATURE_SIZE sizeof(stream_compression_signature_t)
18
19
-static inline rrdpush_signature_t rrdpush_compress_encode_signature(size_t compressed_data_size) {
20
- rrdpush_signature_t len = ((compressed_data_size & 0x7f) | 0x80 | (((compressed_data_size & (0x7f << 7)) << 1) | 0x8000)) << 8;
21
- return len | RRDPUSH_COMPRESSION_SIGNATURE;
19
+static inline stream_compression_signature_t stream_compress_encode_signature(size_t compressed_data_size) {
20
+ stream_compression_signature_t len = ((compressed_data_size & 0x7f) | 0x80 | (((compressed_data_size & (0x7f << 7)) << 1) | 0x8000)) << 8;
21
+ return len | STREAM_COMPRESSION_SIGNATURE;
22
}
23
24
typedef enum {
@@ -32,10 +32,8 @@ typedef enum {
32
COMPRESSION_ALGORITHM_MAX,
33
} compression_algorithm_t;
34
35
-extern int rrdpush_compression_levels[COMPRESSION_ALGORITHM_MAX];
36
-
35
// this defines the order the algorithms will be selected by the receiver (parent)
38
-#define RRDPUSH_COMPRESSION_ALGORITHMS_ORDER "zstd lz4 brotli gzip"
36
+#define STREAM_COMPRESSION_ALGORITHMS_ORDER "zstd lz4 brotli gzip"
37
38
// ----------------------------------------------------------------------------
39
@@ -95,9 +93,9 @@ struct compressor_state {
93
} sender_locked;
94
};
95
98
-void rrdpush_compressor_init(struct compressor_state *state);
99
-void rrdpush_compressor_destroy(struct compressor_state *state);
100
-size_t rrdpush_compress(struct compressor_state *state, const char *data, size_t size, const char **out);
96
+void stream_compressor_init(struct compressor_state *state);
97
+void stream_compressor_destroy(struct compressor_state *state);
98
+size_t stream_compress(struct compressor_state *state, const char *data, size_t size, const char **out);
99
100
// ----------------------------------------------------------------------------
101
@@ -115,44 +113,44 @@ struct decompressor_state {
113
void *stream;
114
};
115
118
-void rrdpush_decompressor_destroy(struct decompressor_state *state);
119
-void rrdpush_decompressor_init(struct decompressor_state *state);
120
-size_t rrdpush_decompress(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
116
+void stream_decompressor_destroy(struct decompressor_state *state);
117
+void stream_decompressor_init(struct decompressor_state *state);
118
+size_t stream_decompress(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
119
122
-static inline size_t rrdpush_decompress_decode_signature(const char *data, size_t data_size) {
120
+static inline size_t stream_decompress_decode_signature(const char *data, size_t data_size) {
121
if (unlikely(!data || !data_size))
122
return 0;
123
126
- if (unlikely(data_size != RRDPUSH_COMPRESSION_SIGNATURE_SIZE))
124
+ if (unlikely(data_size != STREAM_COMPRESSION_SIGNATURE_SIZE))
125
return 0;
126
129
- rrdpush_signature_t sign = *(rrdpush_signature_t *)data;
130
- if (unlikely((sign & RRDPUSH_COMPRESSION_SIGNATURE_MASK) != RRDPUSH_COMPRESSION_SIGNATURE))
127
+ stream_compression_signature_t sign = *(stream_compression_signature_t *)data;
128
+ if (unlikely((sign & STREAM_COMPRESSION_SIGNATURE_MASK) != STREAM_COMPRESSION_SIGNATURE))
129
return 0;
130
131
size_t length = ((sign >> 8) & 0x7f) | ((sign >> 9) & (0x7f << 7));
132
return length;
133
}
134
137
-static inline size_t rrdpush_decompressor_start(struct decompressor_state *state, const char *header, size_t header_size) {
135
+static inline size_t stream_decompressor_start(struct decompressor_state *state, const char *header, size_t header_size) {
136
if(unlikely(state->output.read_pos != state->output.write_pos))
139
- fatal("RRDPUSH DECOMPRESS: asked to decompress new data, while there are unread data in the decompression buffer!");
137
+ fatal("STREAM_DECOMPRESS: asked to decompress new data, while there are unread data in the decompression buffer!");
138
141
- return rrdpush_decompress_decode_signature(header, header_size);
139
+ return stream_decompress_decode_signature(header, header_size);
140
}
141
144
-static inline size_t rrdpush_decompressed_bytes_in_buffer(struct decompressor_state *state) {
142
+static inline size_t stream_decompressed_bytes_in_buffer(struct decompressor_state *state) {
143
if(unlikely(state->output.read_pos > state->output.write_pos))
146
- fatal("RRDPUSH DECOMPRESS: invalid read/write stream positions");
144
+ fatal("STREAM_DECOMPRESS: invalid read/write stream positions");
145
146
return state->output.write_pos - state->output.read_pos;
147
}
148
151
-static inline size_t rrdpush_decompressor_get(struct decompressor_state *state, char *dst, size_t size) {
149
+static inline size_t stream_decompressor_get(struct decompressor_state *state, char *dst, size_t size) {
150
if (unlikely(!state || !size || !dst))
151
return 0;
152
155
- size_t remaining = rrdpush_decompressed_bytes_in_buffer(state);
153
+ size_t remaining = stream_decompressed_bytes_in_buffer(state);
154
155
if(unlikely(!remaining))
156
return 0;
@@ -165,19 +163,21 @@ static inline size_t rrdpush_decompressor_get(struct decompressor_state *state,
163
state->output.read_pos += bytes_to_return;
164
165
if(unlikely(state->output.read_pos > state->output.write_pos))
168
- fatal("RRDPUSH DECOMPRESS: invalid read/write stream positions");
166
+ fatal("STREAM_DECOMPRESS: invalid read/write stream positions");
167
168
return bytes_to_return;
169
}
170
171
// ----------------------------------------------------------------------------
172
175
-#include "../rrdpush.h"
173
+struct sender_state;
174
+struct receiver_state;
175
+struct stream_receiver_config;
176
177
-bool rrdpush_compression_initialize(struct sender_state *s);
178
-bool rrdpush_decompression_initialize(struct receiver_state *rpt);
179
-void rrdpush_parse_compression_order(struct receiver_state *rpt, const char *order);
180
-void rrdpush_select_receiver_compression_algorithm(struct receiver_state *rpt);
181
-void rrdpush_compression_deactivate(struct sender_state *s);
177
+bool stream_compression_initialize(struct sender_state *s);
178
+bool stream_decompression_initialize(struct receiver_state *rpt);
179
+void stream_parse_compression_order(struct stream_receiver_config *config, const char *order);
180
+void stream_select_receiver_compression_algorithm(struct receiver_state *rpt);
181
+void stream_compression_deactivate(struct sender_state *s);
182
183
-#endif // NETDATA_RRDPUSH_COMPRESSION_H 1
183
+#endif // NETDATA_STREAM_COMPRESSION_H 1
src/streaming/stream-compression/gzip.c
+15
-15
@@ -3,7 +3,7 @@
3
#include "gzip.h"
4
#include <zlib.h>
5
6
-void rrdpush_compressor_init_gzip(struct compressor_state *state) {
6
+void stream_compressor_init_gzip(struct compressor_state *state) {
7
if (!state->initialized) {
8
state->initialized = true;
9
@@ -22,7 +22,7 @@ void rrdpush_compressor_init_gzip(struct compressor_state *state) {
22
// int r = deflateInit2(strm, Z_BEST_COMPRESSION, Z_DEFLATED, 15 + 16, 8, Z_DEFAULT_STRATEGY);
23
int r = deflateInit2(strm, state->level, Z_DEFLATED, 15 + 16, 8, Z_DEFAULT_STRATEGY);
24
if (r != Z_OK) {
25
- netdata_log_error("Failed to initialize deflate with error: %d", r);
25
+ netdata_log_error("STREAM_COMPRESS: Failed to initialize deflate with error: %d", r);
26
freez(state->stream);
27
state->initialized = false;
28
return;
@@ -31,7 +31,7 @@ void rrdpush_compressor_init_gzip(struct compressor_state *state) {
31
}
32
}
33
34
-void rrdpush_compressor_destroy_gzip(struct compressor_state *state) {
34
+void stream_compressor_destroy_gzip(struct compressor_state *state) {
35
if (state->stream) {
36
deflateEnd(state->stream);
37
freez(state->stream);
@@ -39,7 +39,7 @@ void rrdpush_compressor_destroy_gzip(struct compressor_state *state) {
39
}
40
}
41
42
-size_t rrdpush_compress_gzip(struct compressor_state *state, const char *data, size_t size, const char **out) {
42
+size_t stream_compress_gzip(struct compressor_state *state, const char *data, size_t size, const char **out) {
43
if (unlikely(!state || !size || !out))
44
return 0;
45
@@ -53,18 +53,18 @@ size_t rrdpush_compress_gzip(struct compressor_state *state, const char *data, s
53
54
int ret = deflate(strm, Z_SYNC_FLUSH);
55
if (ret != Z_OK && ret != Z_STREAM_END) {
56
- netdata_log_error("STREAM: deflate() failed with error %d", ret);
56
+ netdata_log_error("STREAM_COMPRESS: deflate() failed with error %d", ret);
57
return 0;
58
}
59
60
if(strm->avail_in != 0) {
61
- netdata_log_error("STREAM: deflate() did not use all the input buffer, %u bytes out of %zu remain",
61
+ netdata_log_error("STREAM_COMPRESS: deflate() did not use all the input buffer, %u bytes out of %zu remain",
62
strm->avail_in, size);
63
return 0;
64
}
65
66
if(strm->avail_out == 0) {
67
- netdata_log_error("STREAM: deflate() needs a bigger output buffer than the one we provided "
67
+ netdata_log_error("STREAM_COMPRESS: deflate() needs a bigger output buffer than the one we provided "
68
"(output buffer %zu bytes, compressed payload %zu bytes)",
69
state->output.size, size);
70
return 0;
@@ -73,7 +73,7 @@ size_t rrdpush_compress_gzip(struct compressor_state *state, const char *data, s
73
size_t compressed_data_size = state->output.size - strm->avail_out;
74
75
if(compressed_data_size == 0) {
76
- netdata_log_error("STREAM: deflate() did not produce any output "
76
+ netdata_log_error("STREAM_COMPRESS: deflate() did not produce any output "
77
"(output buffer %zu bytes, compressed payload %zu bytes)",
78
state->output.size, size);
79
return 0;
@@ -87,7 +87,7 @@ size_t rrdpush_compress_gzip(struct compressor_state *state, const char *data, s
87
return compressed_data_size;
88
}
89
90
-void rrdpush_decompressor_init_gzip(struct decompressor_state *state) {
90
+void stream_decompressor_init_gzip(struct decompressor_state *state) {
91
if (!state->initialized) {
92
state->initialized = true;
93
@@ -99,7 +99,7 @@ void rrdpush_decompressor_init_gzip(struct decompressor_state *state) {
99
100
int r = inflateInit2(strm, 15 + 16);
101
if (r != Z_OK) {
102
- netdata_log_error("Failed to initialize inflateInit2() with error: %d", r);
102
+ netdata_log_error("STREAM_DECOMPRESS: Failed to initialize inflateInit2() with error: %d", r);
103
freez(state->stream);
104
state->initialized = false;
105
return;
@@ -109,7 +109,7 @@ void rrdpush_decompressor_init_gzip(struct decompressor_state *state) {
109
}
110
}
111
112
-void rrdpush_decompressor_destroy_gzip(struct decompressor_state *state) {
112
+void stream_decompressor_destroy_gzip(struct decompressor_state *state) {
113
if (state->stream) {
114
inflateEnd(state->stream);
115
freez(state->stream);
@@ -117,7 +117,7 @@ void rrdpush_decompressor_destroy_gzip(struct decompressor_state *state) {
117
}
118
}
119
120
-size_t rrdpush_decompress_gzip(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
120
+size_t stream_decompress_gzip(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
121
if (unlikely(!state || !compressed_data || !compressed_size))
122
return 0;
123
@@ -133,19 +133,19 @@ size_t rrdpush_decompress_gzip(struct decompressor_state *state, const char *com
133
134
int ret = inflate(strm, Z_SYNC_FLUSH);
135
if (ret != Z_STREAM_END && ret != Z_OK) {
136
- netdata_log_error("RRDPUSH DECOMPRESS: inflate() failed with error %d", ret);
136
+ netdata_log_error("STREAM_DECOMPRESS: inflate() failed with error %d", ret);
137
return 0;
138
}
139
140
if(strm->avail_in != 0) {
141
- netdata_log_error("RRDPUSH DECOMPRESS: inflate() did not use all compressed data we provided "
141
+ netdata_log_error("STREAM_DECOMPRESS: inflate() did not use all compressed data we provided "
142
"(compressed payload %zu bytes, remaining to be uncompressed %u)"
143
, compressed_size, strm->avail_in);
144
return 0;
145
}
146
147
if(strm->avail_out == 0) {
148
- netdata_log_error("RRDPUSH DECOMPRESS: inflate() needs a bigger output buffer than the one we provided "
148
+ netdata_log_error("STREAM_DECOMPRESS: inflate() needs a bigger output buffer than the one we provided "
149
"(compressed payload %zu bytes, output buffer size %zu bytes)"
150
, compressed_size, state->output.size);
151
return 0;
src/streaming/stream-compression/gzip.h
+6
-6
@@ -5,11 +5,11 @@
5
#ifndef NETDATA_STREAMING_COMPRESSION_GZIP_H
6
#define NETDATA_STREAMING_COMPRESSION_GZIP_H
7
8
-void rrdpush_compressor_init_gzip(struct compressor_state *state);
9
-void rrdpush_compressor_destroy_gzip(struct compressor_state *state);
10
-size_t rrdpush_compress_gzip(struct compressor_state *state, const char *data, size_t size, const char **out);
11
-size_t rrdpush_decompress_gzip(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
12
-void rrdpush_decompressor_init_gzip(struct decompressor_state *state);
13
-void rrdpush_decompressor_destroy_gzip(struct decompressor_state *state);
8
+void stream_compressor_init_gzip(struct compressor_state *state);
9
+void stream_compressor_destroy_gzip(struct compressor_state *state);
10
+size_t stream_compress_gzip(struct compressor_state *state, const char *data, size_t size, const char **out);
11
+size_t stream_decompress_gzip(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
12
+void stream_decompressor_init_gzip(struct decompressor_state *state);
13
+void stream_decompressor_destroy_gzip(struct decompressor_state *state);
14
15
#endif //NETDATA_STREAMING_COMPRESSION_GZIP_H
src/streaming/stream-compression/lz4.c
+11
-10
@@ -1,14 +1,15 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
+#include "libnetdata/libnetdata.h"
4
#include "lz4.h"
5
6
#ifdef ENABLE_LZ4
6
-#include "lz4.h"
7
+#include <lz4.h>
8
9
// ----------------------------------------------------------------------------
10
// compress
11
11
-void rrdpush_compressor_init_lz4(struct compressor_state *state) {
12
+void stream_compressor_init_lz4(struct compressor_state *state) {
13
if(!state->initialized) {
14
state->initialized = true;
15
state->stream = LZ4_createStream();
@@ -19,7 +20,7 @@ void rrdpush_compressor_init_lz4(struct compressor_state *state) {
20
}
21
}
22
22
-void rrdpush_compressor_destroy_lz4(struct compressor_state *state) {
23
+void stream_compressor_destroy_lz4(struct compressor_state *state) {
24
if (state->stream) {
25
LZ4_freeStream(state->stream);
26
state->stream = NULL;
@@ -32,7 +33,7 @@ void rrdpush_compressor_destroy_lz4(struct compressor_state *state) {
33
* Return the size of compressed data block as result and the pointer to internal buffer using the last argument
34
* or 0 in case of error
35
*/
35
-size_t rrdpush_compress_lz4(struct compressor_state *state, const char *data, size_t size, const char **out) {
36
+size_t stream_compress_lz4(struct compressor_state *state, const char *data, size_t size, const char **out) {
37
if(unlikely(!state || !size || !out))
38
return 0;
39
@@ -56,7 +57,7 @@ size_t rrdpush_compress_lz4(struct compressor_state *state, const char *data, si
57
state->level);
58
59
if (compressed_data_size <= 0) {
59
- netdata_log_error("STREAM: LZ4_compress_fast_continue() returned %ld "
60
+ netdata_log_error("STREAM_COMPRESS: LZ4_compress_fast_continue() returned %ld "
61
"(source is %zu bytes, output buffer can fit %zu bytes)",
62
compressed_data_size, size, state->output.size);
63
return 0;
@@ -75,7 +76,7 @@ size_t rrdpush_compress_lz4(struct compressor_state *state, const char *data, si
76
// ----------------------------------------------------------------------------
77
// decompress
78
78
-void rrdpush_decompressor_init_lz4(struct decompressor_state *state) {
79
+void stream_decompressor_init_lz4(struct decompressor_state *state) {
80
if(!state->initialized) {
81
state->initialized = true;
82
state->stream = LZ4_createStreamDecode();
@@ -83,7 +84,7 @@ void rrdpush_decompressor_init_lz4(struct decompressor_state *state) {
84
}
85
}
86
86
-void rrdpush_decompressor_destroy_lz4(struct decompressor_state *state) {
87
+void stream_decompressor_destroy_lz4(struct decompressor_state *state) {
88
if (state->stream) {
89
LZ4_freeStreamDecode(state->stream);
90
state->stream = NULL;
@@ -94,7 +95,7 @@ void rrdpush_decompressor_destroy_lz4(struct decompressor_state *state) {
95
* Decompress the compressed data in the internal buffer
96
* Return the size of uncompressed data or 0 for error
97
*/
97
-size_t rrdpush_decompress_lz4(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
98
+size_t stream_decompress_lz4(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
99
if (unlikely(!state || !compressed_data || !compressed_size))
100
return 0;
101
@@ -115,14 +116,14 @@ size_t rrdpush_decompress_lz4(struct decompressor_state *state, const char *comp
116
);
117
118
if (unlikely(decompressed_size < 0)) {
118
- netdata_log_error("RRDPUSH DECOMPRESS: LZ4_decompress_safe_continue() returned negative value: %ld "
119
+ netdata_log_error("STREAM_DECOMPRESS: LZ4_decompress_safe_continue() returned negative value: %ld "
120
"(compressed chunk is %zu bytes)"
121
, decompressed_size, compressed_size);
122
return 0;
123
}
124
125
if(unlikely(decompressed_size + state->output.write_pos > state->output.size))
125
- fatal("RRDPUSH DECOMPRESS: LZ4_decompress_safe_continue() overflown the stream_buffer "
126
+ fatal("STREAM_DECOMPRESS: LZ4_decompress_safe_continue() overflown the stream_buffer "
127
"(size: %zu, pos: %zu, added: %ld, exceeding the buffer by %zu)"
128
, state->output.size
129
, state->output.write_pos
src/streaming/stream-compression/lz4.h
+6
-6
@@ -7,12 +7,12 @@
7
8
#ifdef ENABLE_LZ4
9
10
-void rrdpush_compressor_init_lz4(struct compressor_state *state);
11
-void rrdpush_compressor_destroy_lz4(struct compressor_state *state);
12
-size_t rrdpush_compress_lz4(struct compressor_state *state, const char *data, size_t size, const char **out);
13
-size_t rrdpush_decompress_lz4(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
14
-void rrdpush_decompressor_init_lz4(struct decompressor_state *state);
15
-void rrdpush_decompressor_destroy_lz4(struct decompressor_state *state);
10
+void stream_compressor_init_lz4(struct compressor_state *state);
11
+void stream_compressor_destroy_lz4(struct compressor_state *state);
12
+size_t stream_compress_lz4(struct compressor_state *state, const char *data, size_t size, const char **out);
13
+size_t stream_decompress_lz4(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
14
+void stream_decompressor_init_lz4(struct decompressor_state *state);
15
+void stream_decompressor_destroy_lz4(struct decompressor_state *state);
16
17
#endif // ENABLE_LZ4
18
src/streaming/stream-compression/zstd.c
+14
-14
@@ -5,7 +5,7 @@
5
#ifdef ENABLE_ZSTD
6
#include <zstd.h>
7
8
-void rrdpush_compressor_init_zstd(struct compressor_state *state) {
8
+void stream_compressor_init_zstd(struct compressor_state *state) {
9
if(!state->initialized) {
10
state->initialized = true;
11
state->stream = ZSTD_createCStream();
@@ -18,21 +18,21 @@ void rrdpush_compressor_init_zstd(struct compressor_state *state) {
18
19
size_t ret = ZSTD_initCStream(state->stream, state->level);
20
if(ZSTD_isError(ret))
21
- netdata_log_error("STREAM: ZSTD_initCStream() returned error: %s", ZSTD_getErrorName(ret));
21
+ netdata_log_error("STREAM_COMPRESS: ZSTD_initCStream() returned error: %s", ZSTD_getErrorName(ret));
22
23
// ZSTD_CCtx_setParameter(state->stream, ZSTD_c_compressionLevel, 1);
24
// ZSTD_CCtx_setParameter(state->stream, ZSTD_c_strategy, ZSTD_fast);
25
}
26
}
27
28
-void rrdpush_compressor_destroy_zstd(struct compressor_state *state) {
28
+void stream_compressor_destroy_zstd(struct compressor_state *state) {
29
if(state->stream) {
30
ZSTD_freeCStream(state->stream);
31
state->stream = NULL;
32
}
33
}
34
35
-size_t rrdpush_compress_zstd(struct compressor_state *state, const char *data, size_t size, const char **out) {
35
+size_t stream_compress_zstd(struct compressor_state *state, const char *data, size_t size, const char **out) {
36
if(unlikely(!state || !size || !out))
37
return 0;
38
@@ -56,12 +56,12 @@ size_t rrdpush_compress_zstd(struct compressor_state *state, const char *data, s
56
57
// error handling
58
if(ZSTD_isError(ret)) {
59
- netdata_log_error("STREAM: ZSTD_compressStream() return error: %s", ZSTD_getErrorName(ret));
59
+ netdata_log_error("STREAM_COMPRESS: ZSTD_compressStream() return error: %s", ZSTD_getErrorName(ret));
60
return 0;
61
}
62
63
if(inBuffer.pos < inBuffer.size) {
64
- netdata_log_error("STREAM: ZSTD_compressStream() left unprocessed input (source payload %zu bytes, consumed %zu bytes)",
64
+ netdata_log_error("STREAM_COMPRESS: ZSTD_compressStream() left unprocessed input (source payload %zu bytes, consumed %zu bytes)",
65
inBuffer.size, inBuffer.pos);
66
return 0;
67
}
@@ -71,12 +71,12 @@ size_t rrdpush_compress_zstd(struct compressor_state *state, const char *data, s
71
ret = ZSTD_flushStream(state->stream, &outBuffer);
72
73
if(ZSTD_isError(ret)) {
74
- netdata_log_error("STREAM: ZSTD_flushStream() return error: %s", ZSTD_getErrorName(ret));
74
+ netdata_log_error("STREAM_COMPRESS: ZSTD_flushStream() return error: %s", ZSTD_getErrorName(ret));
75
return 0;
76
}
77
78
if(outBuffer.pos == 0) {
79
- netdata_log_error("STREAM: ZSTD_compressStream() returned zero compressed bytes "
79
+ netdata_log_error("STREAM_COMPRESS: ZSTD_compressStream() returned zero compressed bytes "
80
"(source is %zu bytes, output buffer can fit %zu bytes) "
81
, size, outBuffer.size);
82
return 0;
@@ -92,27 +92,27 @@ size_t rrdpush_compress_zstd(struct compressor_state *state, const char *data, s
92
return outBuffer.pos;
93
}
94
95
-void rrdpush_decompressor_init_zstd(struct decompressor_state *state) {
95
+void stream_decompressor_init_zstd(struct decompressor_state *state) {
96
if(!state->initialized) {
97
state->initialized = true;
98
state->stream = ZSTD_createDStream();
99
100
size_t ret = ZSTD_initDStream(state->stream);
101
if(ZSTD_isError(ret))
102
- netdata_log_error("STREAM: ZSTD_initDStream() returned error: %s", ZSTD_getErrorName(ret));
102
+ netdata_log_error("STREAM_DECOMPRESS: ZSTD_initDStream() returned error: %s", ZSTD_getErrorName(ret));
103
104
simple_ring_buffer_make_room(&state->output, MAX(COMPRESSION_MAX_CHUNK, ZSTD_DStreamOutSize()));
105
}
106
}
107
108
-void rrdpush_decompressor_destroy_zstd(struct decompressor_state *state) {
108
+void stream_decompressor_destroy_zstd(struct decompressor_state *state) {
109
if (state->stream) {
110
ZSTD_freeDStream(state->stream);
111
state->stream = NULL;
112
}
113
}
114
115
-size_t rrdpush_decompress_zstd(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
115
+size_t stream_decompress_zstd(struct decompressor_state *state, const char *compressed_data, size_t compressed_size) {
116
if (unlikely(!state || !compressed_data || !compressed_size))
117
return 0;
118
@@ -138,12 +138,12 @@ size_t rrdpush_decompress_zstd(struct decompressor_state *state, const char *com
138
, &inBuffer);
139
140
if(ZSTD_isError(ret)) {
141
- netdata_log_error("STREAM: ZSTD_decompressStream() return error: %s", ZSTD_getErrorName(ret));
141
+ netdata_log_error("STREAM_DECOMPRESS: ZSTD_decompressStream() return error: %s", ZSTD_getErrorName(ret));
142
return 0;
143
}
144
145
if(inBuffer.pos < inBuffer.size)
146
- fatal("RRDPUSH DECOMPRESS: ZSTD ZSTD_decompressStream() decompressed %zu bytes, "
146
+ fatal("STREAM_DECOMPRESS: ZSTD ZSTD_decompressStream() decompressed %zu bytes, "
147
"but %zu bytes of compressed data remain",
148
inBuffer.pos, inBuffer.size);
149
src/streaming/stream-compression/zstd.h
+6
-6
@@ -7,12 +7,12 @@
7
8
#ifdef ENABLE_ZSTD
9
10
-void rrdpush_compressor_init_zstd(struct compressor_state *state);
11
-void rrdpush_compressor_destroy_zstd(struct compressor_state *state);
12
-size_t rrdpush_compress_zstd(struct compressor_state *state, const char *data, size_t size, const char **out);
13
-size_t rrdpush_decompress_zstd(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
14
-void rrdpush_decompressor_init_zstd(struct decompressor_state *state);
15
-void rrdpush_decompressor_destroy_zstd(struct decompressor_state *state);
10
+void stream_compressor_init_zstd(struct compressor_state *state);
11
+void stream_compressor_destroy_zstd(struct compressor_state *state);
12
+size_t stream_compress_zstd(struct compressor_state *state, const char *data, size_t size, const char **out);
13
+size_t stream_decompress_zstd(struct decompressor_state *state, const char *compressed_data, size_t compressed_size);
14
+void stream_decompressor_init_zstd(struct decompressor_state *state);
15
+void stream_decompressor_destroy_zstd(struct decompressor_state *state);
16
17
#endif // ENABLE_ZSTD
18
src/streaming/stream-conf.c
+232
-60
@@ -1,28 +1,48 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#include "stream-conf.h"
4
-
5
-struct config stream_config = APPCONFIG_INITIALIZER;
6
-
7
-bool stream_conf_send_enabled = false;
8
-bool stream_conf_compression_enabled = true;
9
-bool stream_conf_replication_enabled = true;
10
-
11
-const char *stream_conf_send_destination = NULL;
12
-const char *stream_conf_send_api_key = NULL;
13
-const char *stream_conf_send_charts_matching = "*";
14
-
15
-time_t stream_conf_replication_period = 86400;
16
-time_t stream_conf_replication_step = 600;
17
-
18
-const char *stream_conf_ssl_ca_path = NULL;
19
-const char *stream_conf_ssl_ca_file = NULL;
20
-
21
-// to have the remote netdata re-sync the charts
22
-// to its current clock, we send for this many
23
-// iterations a BEGIN line without microseconds
24
-// this is for the first iterations of each chart
25
-unsigned int stream_conf_initial_clock_resync_iterations = 60;
3
+#include "daemon/common.h"
4
+#include "stream-receiver-internals.h"
5
+#include "stream-sender-internals.h"
6
+
7
+static struct config stream_config = APPCONFIG_INITIALIZER;
8
+
9
+struct _stream_send stream_send = {
10
+ .enabled = false,
11
+ .api_key = NULL,
12
+ .send_charts_matching = NULL,
13
+ .initial_clock_resync_iterations = 60,
14
+
15
+ .buffer_max_size = CBUFFER_INITIAL_MAX_SIZE,
16
+
17
+ .parents = {
18
+ .destination = NULL,
19
+ .default_port = 19999,
20
+ .h2o = false,
21
+ .timeout_s = 300,
22
+ .reconnect_delay_s = 15,
23
+ .ssl_ca_path = NULL,
24
+ .ssl_ca_file = NULL,
25
+ },
26
+
27
+ .compression = {
28
+ .enabled = true,
29
+ .levels = {
30
+ [COMPRESSION_ALGORITHM_NONE] = 0,
31
+ [COMPRESSION_ALGORITHM_ZSTD] = 3, // 1 (faster) - 22 (smaller)
32
+ [COMPRESSION_ALGORITHM_LZ4] = 1, // 1 (smaller) - 9 (faster)
33
+ [COMPRESSION_ALGORITHM_BROTLI] = 3, // 0 (faster) - 11 (smaller)
34
+ [COMPRESSION_ALGORITHM_GZIP] = 1, // 1 (faster) - 9 (smaller)
35
+ }
36
+ },
37
+};
38
+
39
+struct _stream_receive stream_receive = {
40
+ .replication = {
41
+ .enabled = true,
42
+ .period = 86400,
43
+ .step = 600,
44
+ }
45
+};
46
47
static void stream_conf_load() {
48
errno_clear();
@@ -61,6 +81,7 @@ static void stream_conf_load() {
81
appconfig_move_everywhere(&stream_config, "seconds per replication step", "replication step");
82
appconfig_move_everywhere(&stream_config, "default postpone alarms on connect seconds", "postpone alerts on connect");
83
appconfig_move_everywhere(&stream_config, "postpone alarms on connect seconds", "postpone alerts on connect");
84
+ appconfig_move_everywhere(&stream_config, "health enabled by default", "health enabled");
85
}
86
87
bool stream_conf_receiver_needs_dbengine(void) {
@@ -68,70 +89,221 @@ bool stream_conf_receiver_needs_dbengine(void) {
89
}
90
91
bool stream_conf_init() {
71
- // --------------------------------------------------------------------
72
- // load stream.conf
92
stream_conf_load();
93
75
- stream_conf_send_enabled =
76
- appconfig_get_boolean(&stream_config, CONFIG_SECTION_STREAM, "enabled", stream_conf_send_enabled);
94
+ stream_send.enabled =
95
+ appconfig_get_boolean(&stream_config, CONFIG_SECTION_STREAM, "enabled", stream_send.enabled);
96
+
97
+ stream_send.parents.destination =
98
+ string_strdupz(appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "destination", ""));
99
78
- stream_conf_send_destination =
79
- appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "destination", "");
100
+ stream_send.api_key =
101
+ string_strdupz(appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "api key", ""));
102
81
- stream_conf_send_api_key =
82
- appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "api key", "");
103
+ stream_send.send_charts_matching =
104
+ string_strdupz(appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "send charts matching", "*"));
105
84
- stream_conf_send_charts_matching =
85
- appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "send charts matching", stream_conf_send_charts_matching);
106
+ stream_receive.replication.enabled =
107
+ config_get_boolean(CONFIG_SECTION_DB, "enable replication",
108
+ stream_receive.replication.enabled);
109
87
- stream_conf_replication_enabled =
88
- config_get_boolean(CONFIG_SECTION_DB, "enable replication", stream_conf_replication_enabled);
110
+ stream_receive.replication.period =
111
+ config_get_duration_seconds(CONFIG_SECTION_DB, "replication period",
112
+ stream_receive.replication.period);
113
90
- stream_conf_replication_period =
91
- config_get_duration_seconds(CONFIG_SECTION_DB, "replication period", stream_conf_replication_period);
114
+ stream_receive.replication.step =
115
+ config_get_duration_seconds(CONFIG_SECTION_DB, "replication step",
116
+ stream_receive.replication.step);
117
93
- stream_conf_replication_step =
94
- config_get_duration_seconds(CONFIG_SECTION_DB, "replication step", stream_conf_replication_step);
118
+ stream_send.buffer_max_size = (size_t)appconfig_get_number(
119
+ &stream_config, CONFIG_SECTION_STREAM, "buffer size bytes",
120
+ stream_send.buffer_max_size);
121
96
- rrdhost_free_orphan_time_s =
97
- config_get_duration_seconds(CONFIG_SECTION_DB, "cleanup orphan hosts after", rrdhost_free_orphan_time_s);
122
+ stream_send.parents.reconnect_delay_s = (unsigned int)appconfig_get_duration_seconds(
123
+ &stream_config, CONFIG_SECTION_STREAM, "reconnect delay",
124
+ stream_send.parents.reconnect_delay_s);
125
+ if(stream_send.parents.reconnect_delay_s < SENDER_MIN_RECONNECT_DELAY)
126
+ stream_send.parents.reconnect_delay_s = SENDER_MIN_RECONNECT_DELAY;
127
99
- stream_conf_compression_enabled =
100
- appconfig_get_boolean(&stream_config, CONFIG_SECTION_STREAM,
101
- "enable compression", stream_conf_compression_enabled);
128
+ stream_send.compression.enabled =
129
+ appconfig_get_boolean(&stream_config, CONFIG_SECTION_STREAM, "enable compression",
130
+ stream_send.compression.enabled);
131
103
- rrdpush_compression_levels[COMPRESSION_ALGORITHM_BROTLI] = (int)appconfig_get_number(
132
+ stream_send.compression.levels[COMPRESSION_ALGORITHM_BROTLI] = (int)appconfig_get_number(
133
&stream_config, CONFIG_SECTION_STREAM, "brotli compression level",
105
- rrdpush_compression_levels[COMPRESSION_ALGORITHM_BROTLI]);
134
+ stream_send.compression.levels[COMPRESSION_ALGORITHM_BROTLI]);
135
107
- rrdpush_compression_levels[COMPRESSION_ALGORITHM_ZSTD] = (int)appconfig_get_number(
136
+ stream_send.compression.levels[COMPRESSION_ALGORITHM_ZSTD] = (int)appconfig_get_number(
137
&stream_config, CONFIG_SECTION_STREAM, "zstd compression level",
109
- rrdpush_compression_levels[COMPRESSION_ALGORITHM_ZSTD]);
138
+ stream_send.compression.levels[COMPRESSION_ALGORITHM_ZSTD]);
139
111
- rrdpush_compression_levels[COMPRESSION_ALGORITHM_LZ4] = (int)appconfig_get_number(
140
+ stream_send.compression.levels[COMPRESSION_ALGORITHM_LZ4] = (int)appconfig_get_number(
141
&stream_config, CONFIG_SECTION_STREAM, "lz4 compression acceleration",
113
- rrdpush_compression_levels[COMPRESSION_ALGORITHM_LZ4]);
142
+ stream_send.compression.levels[COMPRESSION_ALGORITHM_LZ4]);
143
115
- rrdpush_compression_levels[COMPRESSION_ALGORITHM_GZIP] = (int)appconfig_get_number(
144
+ stream_send.compression.levels[COMPRESSION_ALGORITHM_GZIP] = (int)appconfig_get_number(
145
&stream_config, CONFIG_SECTION_STREAM, "gzip compression level",
117
- rrdpush_compression_levels[COMPRESSION_ALGORITHM_GZIP]);
146
+ stream_send.compression.levels[COMPRESSION_ALGORITHM_GZIP]);
147
119
- if(stream_conf_send_enabled && (!stream_conf_send_destination || !*stream_conf_send_destination || !stream_conf_send_api_key || !*stream_conf_send_api_key)) {
120
- nd_log_daemon(NDLP_WARNING, "STREAM [send]: cannot enable sending thread - information is missing.");
121
- stream_conf_send_enabled = false;
122
- }
148
+ stream_send.parents.h2o = appconfig_get_boolean(
149
+ &stream_config, CONFIG_SECTION_STREAM, "parent using h2o",
150
+ stream_send.parents.h2o);
151
+
152
+ stream_send.parents.timeout_s = (int)appconfig_get_duration_seconds(
153
+ &stream_config, CONFIG_SECTION_STREAM, "timeout",
154
+ stream_send.parents.timeout_s);
155
+
156
+ stream_send.buffer_max_size = (size_t)appconfig_get_number(
157
+ &stream_config, CONFIG_SECTION_STREAM, "buffer size bytes",
158
+ stream_send.buffer_max_size);
159
+
160
+ stream_send.parents.default_port = (int)appconfig_get_number(
161
+ &stream_config, CONFIG_SECTION_STREAM, "default port",
162
+ stream_send.parents.default_port);
163
124
- netdata_ssl_validate_certificate_sender = !appconfig_get_boolean(&stream_config, CONFIG_SECTION_STREAM, "ssl skip certificate verification", !netdata_ssl_validate_certificate);
164
+ stream_send.initial_clock_resync_iterations = (unsigned int)appconfig_get_number(
165
+ &stream_config, CONFIG_SECTION_STREAM, "initial clock resync iterations",
166
+ stream_send.initial_clock_resync_iterations); // TODO: REMOVE FOR SLEW / GAPFILLING
167
+
168
+ netdata_ssl_validate_certificate_sender = !appconfig_get_boolean(
169
+ &stream_config, CONFIG_SECTION_STREAM, "ssl skip certificate verification",
170
+ !netdata_ssl_validate_certificate);
171
172
if(!netdata_ssl_validate_certificate_sender)
173
nd_log_daemon(NDLP_NOTICE, "SSL: streaming senders will skip SSL certificates verification.");
174
129
- stream_conf_ssl_ca_path = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "CApath", NULL);
130
- stream_conf_ssl_ca_file = appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "CAfile", NULL);
175
+ stream_send.parents.ssl_ca_path = string_strdupz(appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "CApath", NULL));
176
+ stream_send.parents.ssl_ca_file = string_strdupz(appconfig_get(&stream_config, CONFIG_SECTION_STREAM, "CAfile", NULL));
177
+
178
+ if(stream_send.enabled && (!stream_send.parents.destination || !stream_send.api_key)) {
179
+ nd_log_daemon(NDLP_ERR, "STREAM [send]: cannot enable sending thread - information is missing.");
180
+ stream_send.enabled = false;
181
+ }
182
132
- return stream_conf_send_enabled;
183
+ return stream_send.enabled;
184
}
185
186
bool stream_conf_configured_as_parent() {
187
return stream_conf_has_uuid_section(&stream_config);
188
}
189
+
190
+void stream_conf_receiver_config(struct receiver_state *rpt, struct stream_receiver_config *config, const char *api_key, const char *machine_guid) {
191
+ config->mode = rrd_memory_mode_id(
192
+ appconfig_get(&stream_config, machine_guid, "db",
193
+ appconfig_get(&stream_config, api_key, "db",
194
+ rrd_memory_mode_name(default_rrd_memory_mode))));
195
+
196
+ if (unlikely(config->mode == RRD_MEMORY_MODE_DBENGINE && !dbengine_enabled)) {
197
+ netdata_log_error("STREAM '%s' [receive from %s:%s]: "
198
+ "dbengine is not enabled, falling back to default."
199
+ , rpt->hostname
200
+ , rpt->client_ip, rpt->client_port
201
+ );
202
+ config->mode = default_rrd_memory_mode;
203
+ }
204
+
205
+ config->history = (int)
206
+ appconfig_get_number(&stream_config, machine_guid, "retention",
207
+ appconfig_get_number(&stream_config, api_key, "retention",
208
+ default_rrd_history_entries));
209
+ if(config->history < 5) config->history = 5;
210
+
211
+ config->health.enabled =
212
+ appconfig_get_boolean_ondemand(&stream_config, machine_guid, "health enabled",
213
+ appconfig_get_boolean_ondemand(&stream_config, api_key, "health enabled",
214
+ health_plugin_enabled()));
215
+
216
+ config->health.delay =
217
+ appconfig_get_duration_seconds(&stream_config, machine_guid, "postpone alerts on connect",
218
+ appconfig_get_duration_seconds(&stream_config, api_key, "postpone alerts on connect",
219
+ 60));
220
+
221
+ config->update_every = (int)appconfig_get_duration_seconds(&stream_config, machine_guid, "update every", config->update_every);
222
+ if(config->update_every < 0) config->update_every = 1;
223
+
224
+ config->health.history =
225
+ appconfig_get_duration_seconds(&stream_config, machine_guid, "health log retention",
226
+ appconfig_get_duration_seconds(&stream_config, api_key, "health log retention",
227
+ HEALTH_LOG_RETENTION_DEFAULT));
228
+
229
+ config->send.enabled =
230
+ appconfig_get_boolean(&stream_config, machine_guid, "proxy enabled",
231
+ appconfig_get_boolean(&stream_config, api_key, "proxy enabled",
232
+ stream_send.enabled));
233
+
234
+ config->send.parents = string_strdupz(
235
+ appconfig_get(&stream_config, machine_guid, "proxy destination",
236
+ appconfig_get(&stream_config, api_key, "proxy destination",
237
+ string2str(stream_send.parents.destination))));
238
+
239
+ config->send.api_key = string_strdupz(
240
+ appconfig_get(&stream_config, machine_guid, "proxy api key",
241
+ appconfig_get(&stream_config, api_key, "proxy api key",
242
+ string2str(stream_send.api_key))));
243
+
244
+ config->send.charts_matching = string_strdupz(
245
+ appconfig_get(&stream_config, machine_guid, "proxy send charts matching",
246
+ appconfig_get(&stream_config, api_key, "proxy send charts matching",
247
+ string2str(stream_send.send_charts_matching))));
248
+
249
+ config->replication.enabled =
250
+ appconfig_get_boolean(&stream_config, machine_guid, "enable replication",
251
+ appconfig_get_boolean(&stream_config, api_key, "enable replication",
252
+ stream_receive.replication.enabled));
253
+
254
+ config->replication.period =
255
+ appconfig_get_duration_seconds(&stream_config, machine_guid, "replication period",
256
+ appconfig_get_duration_seconds(&stream_config, api_key, "replication period",
257
+ stream_receive.replication.period));
258
+
259
+ config->replication.step =
260
+ appconfig_get_number(&stream_config, machine_guid, "replication step",
261
+ appconfig_get_number(&stream_config, api_key, "replication step",
262
+ stream_receive.replication.step));
263
+
264
+ config->compression.enabled =
265
+ appconfig_get_boolean(&stream_config, machine_guid, "enable compression",
266
+ appconfig_get_boolean(&stream_config, api_key, "enable compression",
267
+ stream_send.compression.enabled));
268
+
269
+ if(config->compression.enabled) {
270
+ stream_parse_compression_order(
271
+ config,
272
+ appconfig_get(
273
+ &stream_config,
274
+ machine_guid,
275
+ "compression algorithms order",
276
+ appconfig_get(
277
+ &stream_config, api_key, "compression algorithms order", STREAM_COMPRESSION_ALGORITHMS_ORDER)));
278
+ }
279
+
280
+ config->ephemeral =
281
+ appconfig_get_boolean(&stream_config, machine_guid, "is ephemeral node",
282
+ appconfig_get_boolean(&stream_config, api_key, "is ephemeral node",
283
+ CONFIG_BOOLEAN_NO));
284
+}
285
+
286
+bool stream_conf_is_key_type(const char *api_key, const char *type) {
287
+ const char *api_key_type = appconfig_get(&stream_config, api_key, "type", type);
288
+ if(!api_key_type || !*api_key_type) api_key_type = "unknown";
289
+ return strcmp(api_key_type, type) == 0;
290
+}
291
+
292
+bool stream_conf_api_key_is_enabled(const char *api_key, bool enabled) {
293
+ return appconfig_get_boolean(&stream_config, api_key, "enabled", enabled);
294
+}
295
+
296
+bool stream_conf_api_key_allows_client(const char *api_key, const char *client_ip) {
297
+ SIMPLE_PATTERN *key_allow_from = simple_pattern_create(
298
+ appconfig_get(&stream_config, api_key, "allow from", "*"),
299
+ NULL, SIMPLE_PATTERN_EXACT, true);
300
+
301
+ bool rc = true;
302
+
303
+ if(key_allow_from) {
304
+ rc = simple_pattern_matches(key_allow_from, client_ip);
305
+ simple_pattern_free(key_allow_from);
306
+ }
307
+
308
+ return rc;
309
+}
src/streaming/stream-conf.h
+79
-13
@@ -4,25 +4,91 @@
4
#define NETDATA_STREAM_CONF_H
5
6
#include "libnetdata/libnetdata.h"
7
-#include "daemon/common.h"
7
+#include "stream-compression/compression.h"
8
+#include "stream-capabilities.h"
9
+#include "database/rrd-database-mode.h"
10
9
-extern bool stream_conf_send_enabled;
10
-extern bool stream_conf_compression_enabled;
11
-extern bool stream_conf_replication_enabled;
11
+#define SENDER_MIN_RECONNECT_DELAY 5
12
13
-extern const char *stream_conf_send_destination;
14
-extern const char *stream_conf_send_api_key;
15
-extern const char *stream_conf_send_charts_matching;
16
-extern time_t stream_conf_replication_period;
17
-extern time_t stream_conf_replication_step;
18
-extern unsigned int stream_conf_initial_clock_resync_iterations;
13
+struct _stream_send {
14
+ bool enabled;
15
20
-extern struct config stream_config;
21
-extern const char *stream_conf_ssl_ca_path;
22
-extern const char *stream_conf_ssl_ca_file;
16
+ STRING *api_key;
17
+ STRING *send_charts_matching;
18
+
19
+ // to have the remote netdata re-sync the charts
20
+ // to its current clock, we send for this many
21
+ // iterations a BEGIN line without microseconds
22
+ // this is for the first iterations of each chart
23
+ uint16_t initial_clock_resync_iterations;
24
+
25
+ uint32_t buffer_max_size;
26
+
27
+ struct {
28
+ STRING *destination;
29
+ STRING *ssl_ca_path;
30
+ STRING *ssl_ca_file;
31
+ bool h2o;
32
+ uint16_t default_port;
33
+ time_t timeout_s;
34
+ time_t reconnect_delay_s;
35
+ } parents;
36
+
37
+ struct {
38
+ bool enabled;
39
+ int levels[COMPRESSION_ALGORITHM_MAX];
40
+ } compression;
41
+};
42
+extern struct _stream_send stream_send;
43
+
44
+struct _stream_receive {
45
+ struct {
46
+ bool enabled;
47
+ time_t period;
48
+ time_t step;
49
+ } replication;
50
+};
51
+extern struct _stream_receive stream_receive;
52
+
53
+struct stream_receiver_config {
54
+ RRD_MEMORY_MODE mode;
55
+ bool ephemeral;
56
+ int history;
57
+ int update_every;
58
+
59
+ struct {
60
+ bool enabled; // enable replication on this child
61
+ time_t period;
62
+ time_t step;
63
+ } replication;
64
+
65
+ struct {
66
+ int enabled; // CONFIG_BOOLEAN_YES, CONFIG_BOOLEAN_NO, CONFIG_BOOLEAN_AUTO
67
+ time_t delay;
68
+ uint32_t history;
69
+ } health;
70
+
71
+ struct {
72
+ bool enabled;
73
+ STRING *api_key;
74
+ STRING *parents;
75
+ STRING *charts_matching;
76
+ } send;
77
+
78
+ struct {
79
+ bool enabled;
80
+ STREAM_CAPABILITIES priorities[COMPRESSION_ALGORITHM_MAX];
81
+ } compression;
82
+};
83
+
84
+void stream_conf_receiver_config(struct receiver_state *rpt, struct stream_receiver_config *config, const char *api_key, const char *machine_guid);
85
86
bool stream_conf_init();
87
bool stream_conf_receiver_needs_dbengine();
88
bool stream_conf_configured_as_parent();
89
90
+bool stream_conf_is_key_type(const char *api_key, const char *type);
91
+bool stream_conf_api_key_is_enabled(const char *api_key, bool enabled);
92
+bool stream_conf_api_key_allows_client(const char *api_key, const char *client_ip);
93
+
94
#endif //NETDATA_STREAM_CONF_H
src/streaming/stream-connector.c
renamed
+335
-320
@@ -1,22 +1,16 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#include "sender-internals.h"
3
+#include "stream-sender-internals.h"
4
5
-void rrdpush_sender_thread_close_socket(struct sender_state *s) {
6
- rrdhost_flag_clear(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED | RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
5
+typedef struct {
6
+ char *os_name;
7
+ char *os_id;
8
+ char *os_version;
9
+ char *kernel_name;
10
+ char *kernel_version;
11
+} stream_encoded_t;
12
8
- netdata_ssl_close(&s->ssl);
9
-
10
- if(s->rrdpush_sender_socket != -1) {
11
- close(s->rrdpush_sender_socket);
12
- s->rrdpush_sender_socket = -1;
13
- }
14
-
15
- // do not flush the circular buffer here
16
- // this function is called sometimes with the sender lock, sometimes without the lock
17
-}
18
-
19
-void rrdpush_encode_variable(stream_encoded_t *se, RRDHOST *host) {
13
+static void rrdpush_encode_variable(stream_encoded_t *se, RRDHOST *host) {
14
se->os_name = (host->system_info->host_os_name)?url_encode(host->system_info->host_os_name):strdupz("");
15
se->os_id = (host->system_info->host_os_id)?url_encode(host->system_info->host_os_id):strdupz("");
16
se->os_version = (host->system_info->host_os_version)?url_encode(host->system_info->host_os_version):strdupz("");
@@ -24,7 +18,7 @@ void rrdpush_encode_variable(stream_encoded_t *se, RRDHOST *host) {
18
se->kernel_version = (host->system_info->kernel_version)?url_encode(host->system_info->kernel_version):strdupz("");
19
}
20
27
-void rrdpush_clean_encoded(stream_encoded_t *se) {
21
+static void rrdpush_clean_encoded(stream_encoded_t *se) {
22
if (se->os_name) {
23
freez(se->os_name);
24
se->os_name = NULL;
@@ -51,7 +45,7 @@ void rrdpush_clean_encoded(stream_encoded_t *se) {
45
}
46
}
47
54
-struct {
48
+static struct {
49
const char *response;
50
const char *status;
51
size_t length;
@@ -65,7 +59,7 @@ struct {
59
{
60
.response = START_STREAMING_PROMPT_VN,
61
.length = sizeof(START_STREAMING_PROMPT_VN) - 1,
68
- .status = RRDPUSH_STATUS_CONNECTED,
62
+ .status = STREAM_STATUS_CONNECTED,
63
.version = STREAM_HANDSHAKE_OK_V3, // and above
64
.dynamic = true, // dynamic = we will parse the version / capabilities
65
.error = NULL,
@@ -76,7 +70,7 @@ struct {
70
{
71
.response = START_STREAMING_PROMPT_V2,
72
.length = sizeof(START_STREAMING_PROMPT_V2) - 1,
79
- .status = RRDPUSH_STATUS_CONNECTED,
73
+ .status = STREAM_STATUS_CONNECTED,
74
.version = STREAM_HANDSHAKE_OK_V2,
75
.dynamic = false,
76
.error = NULL,
@@ -87,7 +81,7 @@ struct {
81
{
82
.response = START_STREAMING_PROMPT_V1,
83
.length = sizeof(START_STREAMING_PROMPT_V1) - 1,
90
- .status = RRDPUSH_STATUS_CONNECTED,
84
+ .status = STREAM_STATUS_CONNECTED,
85
.version = STREAM_HANDSHAKE_OK_V1,
86
.dynamic = false,
87
.error = NULL,
@@ -98,66 +92,66 @@ struct {
92
{
93
.response = START_STREAMING_ERROR_SAME_LOCALHOST,
94
.length = sizeof(START_STREAMING_ERROR_SAME_LOCALHOST) - 1,
101
- .status = RRDPUSH_STATUS_LOCALHOST,
95
+ .status = STREAM_STATUS_LOCALHOST,
96
.version = STREAM_HANDSHAKE_ERROR_LOCALHOST,
97
.dynamic = false,
98
.error = "remote server rejected this stream, the host we are trying to stream is its localhost",
105
- .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
99
+ .worker_job_id = WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE,
100
.postpone_reconnect_seconds = 60 * 60, // the IP may change, try it every hour
101
.priority = NDLP_DEBUG,
102
},
103
{
104
.response = START_STREAMING_ERROR_ALREADY_STREAMING,
105
.length = sizeof(START_STREAMING_ERROR_ALREADY_STREAMING) - 1,
112
- .status = RRDPUSH_STATUS_ALREADY_CONNECTED,
106
+ .status = STREAM_STATUS_ALREADY_CONNECTED,
107
.version = STREAM_HANDSHAKE_ERROR_ALREADY_CONNECTED,
108
.dynamic = false,
109
.error = "remote server rejected this stream, the host we are trying to stream is already streamed to it",
116
- .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
110
+ .worker_job_id = WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE,
111
.postpone_reconnect_seconds = 2 * 60, // 2 minutes
112
.priority = NDLP_DEBUG,
113
},
114
{
115
.response = START_STREAMING_ERROR_NOT_PERMITTED,
116
.length = sizeof(START_STREAMING_ERROR_NOT_PERMITTED) - 1,
123
- .status = RRDPUSH_STATUS_PERMISSION_DENIED,
117
+ .status = STREAM_STATUS_PERMISSION_DENIED,
118
.version = STREAM_HANDSHAKE_ERROR_DENIED,
119
.dynamic = false,
120
.error = "remote server denied access, probably we don't have the right API key?",
127
- .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
121
+ .worker_job_id = WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE,
122
.postpone_reconnect_seconds = 1 * 60, // 1 minute
123
.priority = NDLP_ERR,
124
},
125
{
126
.response = START_STREAMING_ERROR_BUSY_TRY_LATER,
127
.length = sizeof(START_STREAMING_ERROR_BUSY_TRY_LATER) - 1,
134
- .status = RRDPUSH_STATUS_RATE_LIMIT,
128
+ .status = STREAM_STATUS_RATE_LIMIT,
129
.version = STREAM_HANDSHAKE_BUSY_TRY_LATER,
130
.dynamic = false,
131
.error = "remote server is currently busy, we should try later",
138
- .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
132
+ .worker_job_id = WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE,
133
.postpone_reconnect_seconds = 2 * 60, // 2 minutes
134
.priority = NDLP_NOTICE,
135
},
136
{
137
.response = START_STREAMING_ERROR_INTERNAL_ERROR,
138
.length = sizeof(START_STREAMING_ERROR_INTERNAL_ERROR) - 1,
145
- .status = RRDPUSH_STATUS_INTERNAL_SERVER_ERROR,
139
+ .status = STREAM_STATUS_INTERNAL_SERVER_ERROR,
140
.version = STREAM_HANDSHAKE_INTERNAL_ERROR,
141
.dynamic = false,
142
.error = "remote server is encountered an internal error, we should try later",
149
- .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
143
+ .worker_job_id = WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE,
144
.postpone_reconnect_seconds = 5 * 60, // 5 minutes
145
.priority = NDLP_CRIT,
146
},
147
{
148
.response = START_STREAMING_ERROR_INITIALIZATION,
149
.length = sizeof(START_STREAMING_ERROR_INITIALIZATION) - 1,
156
- .status = RRDPUSH_STATUS_INITIALIZATION_IN_PROGRESS,
150
+ .status = STREAM_STATUS_INITIALIZATION_IN_PROGRESS,
151
.version = STREAM_HANDSHAKE_INITIALIZATION,
152
.dynamic = false,
153
.error = "remote server is initializing, we should try later",
160
- .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
154
+ .worker_job_id = WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE,
155
.postpone_reconnect_seconds = 2 * 60, // 2 minute
156
.priority = NDLP_NOTICE,
157
},
@@ -166,134 +160,18 @@ struct {
160
{
161
.response = NULL,
162
.length = 0,
169
- .status = RRDPUSH_STATUS_BAD_HANDSHAKE,
163
+ .status = STREAM_STATUS_BAD_HANDSHAKE,
164
.version = STREAM_HANDSHAKE_ERROR_BAD_HANDSHAKE,
165
.dynamic = false,
166
.error = "remote node response is not understood, is it Netdata?",
173
- .worker_job_id = WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE,
167
+ .worker_job_id = WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE,
168
.postpone_reconnect_seconds = 1 * 60, // 1 minute
169
.priority = NDLP_ERR,
170
}
171
};
172
179
-static inline bool rrdpush_sender_validate_response(RRDHOST *host, struct sender_state *s, char *http, size_t http_length) {
180
- int32_t version = STREAM_HANDSHAKE_ERROR_BAD_HANDSHAKE;
181
-
182
- int i;
183
- for(i = 0; stream_responses[i].response ; i++) {
184
- if(stream_responses[i].dynamic &&
185
- http_length > stream_responses[i].length && http_length < (stream_responses[i].length + 30) &&
186
- strncmp(http, stream_responses[i].response, stream_responses[i].length) == 0) {
187
-
188
- version = str2i(&http[stream_responses[i].length]);
189
- break;
190
- }
191
- else if(http_length == stream_responses[i].length && strcmp(http, stream_responses[i].response) == 0) {
192
- version = stream_responses[i].version;
193
-
194
- break;
195
- }
196
- }
197
-
198
- if(version >= STREAM_HANDSHAKE_OK_V1) {
199
- host->destination->reason = version;
200
- host->destination->postpone_reconnection_until = now_realtime_sec() + s->reconnect_delay;
201
- s->capabilities = convert_stream_version_to_capabilities(version, host, true);
202
- return true;
203
- }
204
-
205
- ND_LOG_FIELD_PRIORITY priority = stream_responses[i].priority;
206
- const char *error = stream_responses[i].error;
207
- const char *status = stream_responses[i].status;
208
- int worker_job_id = stream_responses[i].worker_job_id;
209
- int delay = stream_responses[i].postpone_reconnect_seconds;
210
-
211
- worker_is_busy(worker_job_id);
212
- rrdpush_sender_thread_close_socket(s);
213
- host->destination->reason = version;
214
- host->destination->postpone_reconnection_until = now_realtime_sec() + delay;
215
-
216
- ND_LOG_STACK lgs[] = {
217
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, status),
218
- ND_LOG_FIELD_END(),
219
- };
220
- ND_LOG_STACK_PUSH(lgs);
221
-
222
- char buf[RFC3339_MAX_LENGTH];
223
- rfc3339_datetime_ut(buf, sizeof(buf), host->destination->postpone_reconnection_until * USEC_PER_SEC, 0, false);
224
-
225
- nd_log(NDLS_DAEMON, priority,
226
- "STREAM %s [send to %s]: %s - will retry in %d secs, at %s",
227
- rrdhost_hostname(host), s->connected_to, error, delay, buf);
228
-
229
- return false;
230
-}
231
-
232
-unsigned char alpn_proto_list[] = {
233
- 18, 'n', 'e', 't', 'd', 'a', 't', 'a', '_', 's', 't', 'r', 'e', 'a', 'm', '/', '2', '.', '0',
234
- 8, 'h', 't', 't', 'p', '/', '1', '.', '1'
235
-};
236
-
173
#define CONN_UPGRADE_VAL "upgrade"
238
-
239
-static bool rrdpush_sender_connect_ssl(struct sender_state *s) {
240
- RRDHOST *host = s->host;
241
- bool ssl_required = host && host->destination && host->destination->ssl;
242
-
243
- netdata_ssl_close(&host->sender->ssl);
244
-
245
- if(!ssl_required)
246
- return true;
247
-
248
- if (netdata_ssl_open_ext(&host->sender->ssl, netdata_ssl_streaming_sender_ctx, s->rrdpush_sender_socket, alpn_proto_list, sizeof(alpn_proto_list))) {
249
- if(!netdata_ssl_connect(&host->sender->ssl)) {
250
- // couldn't connect
251
-
252
- ND_LOG_STACK lgs[] = {
253
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_SSL_ERROR),
254
- ND_LOG_FIELD_END(),
255
- };
256
- ND_LOG_STACK_PUSH(lgs);
257
-
258
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR);
259
- rrdpush_sender_thread_close_socket(s);
260
- host->destination->reason = STREAM_HANDSHAKE_ERROR_SSL_ERROR;
261
- host->destination->postpone_reconnection_until = now_realtime_sec() + 5 * 60;
262
- return false;
263
- }
264
-
265
- if (netdata_ssl_validate_certificate_sender &&
266
- security_test_certificate(host->sender->ssl.conn)) {
267
- // certificate is not valid
268
-
269
- ND_LOG_STACK lgs[] = {
270
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_INVALID_SSL_CERTIFICATE),
271
- ND_LOG_FIELD_END(),
272
- };
273
- ND_LOG_STACK_PUSH(lgs);
274
-
275
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SSL_ERROR);
276
- netdata_log_error("SSL: closing the stream connection, because the server SSL certificate is not valid.");
277
- rrdpush_sender_thread_close_socket(s);
278
- host->destination->reason = STREAM_HANDSHAKE_ERROR_INVALID_CERTIFICATE;
279
- host->destination->postpone_reconnection_until = now_realtime_sec() + 5 * 60;
280
- return false;
281
- }
282
-
283
- return true;
284
- }
285
-
286
- ND_LOG_STACK lgs[] = {
287
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_CANT_ESTABLISH_SSL_CONNECTION),
288
- ND_LOG_FIELD_END(),
289
- };
290
- ND_LOG_STACK_PUSH(lgs);
291
-
292
- netdata_log_error("SSL: failed to establish connection.");
293
- return false;
294
-}
295
-
296
-static int rrdpush_http_upgrade_prelude(RRDHOST *host, struct sender_state *s) {
174
+static int stream_connect_upgrade_prelude(RRDHOST *host __maybe_unused, struct sender_state *s) {
175
176
char http[HTTP_HEADER_SIZE + 1];
177
snprintfz(http, HTTP_HEADER_SIZE,
@@ -302,22 +180,14 @@ static int rrdpush_http_upgrade_prelude(RRDHOST *host, struct sender_state *s) {
180
"Connection: Upgrade"
181
HTTP_HDR_END);
182
305
- ssize_t bytes = send_timeout(
306
- &host->sender->ssl,
307
- s->rrdpush_sender_socket,
308
- http,
309
- strlen(http),
310
- 0,
311
- 1000);
312
-
313
- bytes = recv_timeout(
314
- &host->sender->ssl,
315
- s->rrdpush_sender_socket,
316
- http,
317
- HTTP_HEADER_SIZE,
318
- 0,
319
- 1000);
183
+ ssize_t bytes;
184
+ bytes = nd_sock_send_timeout(&s->sock, http, strlen(http), 0, 1000);
185
+ if (bytes <= 0) {
186
+ error_report("Error writing to remote");
187
+ return 1;
188
+ }
189
190
+ bytes = nd_sock_recv_timeout(&s->sock, http, HTTP_HEADER_SIZE, 0, 1000);
191
if (bytes <= 0) {
192
error_report("Error reading from remote");
193
return 1;
@@ -387,33 +257,85 @@ err_cleanup:
257
return 1;
258
}
259
390
-static bool sender_send_connection_request(RRDHOST *host, int default_port, int timeout, struct sender_state *s) {
260
+static bool
261
+stream_connect_validate_first_response(RRDHOST *host, struct sender_state *s, char *http, size_t http_length) {
262
+ int32_t version = STREAM_HANDSHAKE_ERROR_BAD_HANDSHAKE;
263
+
264
+ int i;
265
+ for(i = 0; stream_responses[i].response ; i++) {
266
+ if(stream_responses[i].dynamic &&
267
+ http_length > stream_responses[i].length && http_length < (stream_responses[i].length + 30) &&
268
+ strncmp(http, stream_responses[i].response, stream_responses[i].length) == 0) {
269
392
- struct timeval tv = {
393
- .tv_sec = timeout,
394
- .tv_usec = 0
270
+ version = str2i(&http[stream_responses[i].length]);
271
+ break;
272
+ }
273
+ else if(http_length == stream_responses[i].length && strcmp(http, stream_responses[i].response) == 0) {
274
+ version = stream_responses[i].version;
275
+
276
+ break;
277
+ }
278
+ }
279
+
280
+ if(version >= STREAM_HANDSHAKE_OK_V1) {
281
+ stream_parent_set_reconnect_delay(host->stream.snd.parents.current, STREAM_HANDSHAKE_CONNECTED,
282
+ stream_send.parents.reconnect_delay_s);
283
+ s->capabilities = convert_stream_version_to_capabilities(version, host, true);
284
+ return true;
285
+ }
286
+
287
+ ND_LOG_FIELD_PRIORITY priority = stream_responses[i].priority;
288
+ const char *error = stream_responses[i].error;
289
+ const char *status = stream_responses[i].status;
290
+ int worker_job_id = stream_responses[i].worker_job_id;
291
+ int delay = stream_responses[i].postpone_reconnect_seconds;
292
+
293
+ worker_is_busy(worker_job_id);
294
+ stream_parent_set_reconnect_delay(host->stream.snd.parents.current, version, delay);
295
+
296
+ ND_LOG_STACK lgs[] = {
297
+ ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, status),
298
+ ND_LOG_FIELD_END(),
299
};
300
+ ND_LOG_STACK_PUSH(lgs);
301
+
302
+ char buf[RFC3339_MAX_LENGTH];
303
+ rfc3339_datetime_ut(buf, sizeof(buf), stream_parent_get_reconnection_ut(host->stream.snd.parents.current), 0, false);
304
+
305
+ nd_log(NDLS_DAEMON, priority,
306
+ "STREAM %s [send to %s]: %s - will retry in %d secs, at %s",
307
+ rrdhost_hostname(host), s->connected_to, error, delay, buf);
308
+
309
+ return false;
310
+}
311
+
312
+bool stream_connect(struct sender_state *s, uint16_t default_port, time_t timeout) {
313
+ worker_is_busy(WORKER_SENDER_CONNECTOR_JOB_CONNECTING);
314
+
315
+ RRDHOST *host = s->host;
316
317
// make sure the socket is closed
398
- rrdpush_sender_thread_close_socket(s);
399
-
400
- s->rrdpush_sender_socket = connect_to_one_of_destinations(
401
- host
402
- , default_port
403
- , &tv
404
- , &s->reconnects_counter
405
- , s->connected_to
406
- , sizeof(s->connected_to)-1
407
- , &host->destination
408
- );
318
+ nd_sock_close(&s->sock);
319
+
320
+ s->hops = (int16_t)(host->system_info->hops + 1);
321
+
322
+ // reset this to make sure we have its current value
323
+ s->sock.verify_certificate = netdata_ssl_validate_certificate_sender;
324
+ s->sock.ctx = netdata_ssl_streaming_sender_ctx;
325
410
- if(unlikely(s->rrdpush_sender_socket == -1)) {
411
- // netdata_log_error("STREAM %s [send to %s]: could not connect to parent node at this time.", rrdhost_hostname(host), host->rrdpush_send_destination);
326
+ if(!stream_parent_connect_to_one(
327
+ &s->sock, host, default_port, timeout,
328
+ s->connected_to, sizeof(s->connected_to) - 1,
329
+ &host->stream.snd.parents.current)) {
330
+
331
+ if(s->sock.error != ND_SOCK_ERR_NO_DESTINATION_AVAILABLE)
332
+ nd_log(NDLS_DAEMON, NDLP_WARNING, "can't connect to a parent, last error: %s",
333
+ ND_SOCK_ERROR_2str(s->sock.error));
334
+
335
+ nd_sock_close(&s->sock);
336
return false;
337
}
338
415
- // netdata_log_info("STREAM %s [send to %s]: initializing communication...", rrdhost_hostname(host), s->connected_to);
416
-
339
// reset our capabilities to default
340
s->capabilities = stream_our_capabilities(host, true);
341
@@ -423,8 +345,6 @@ static bool sender_send_connection_request(RRDHOST *host, int default_port, int
345
stream_encoded_t se;
346
rrdpush_encode_variable(&se, host);
347
426
- host->sender->hops = host->system_info->hops + 1;
427
-
348
char http[HTTP_HEADER_SIZE + 1];
349
int eol = snprintfz(http, HTTP_HEADER_SIZE,
350
"STREAM "
@@ -471,18 +391,18 @@ static bool sender_send_connection_request(RRDHOST *host, int default_port, int
391
"&NETDATA_SYSTEM_TOTAL_DISK_SIZE=%s"
392
"&NETDATA_PROTOCOL_VERSION=%s"
393
HTTP_1_1 HTTP_ENDL
474
- "User-Agent: %s/%s\r\n"
475
- "Accept: */*\r\n\r\n"
476
- , host->rrdpush.send.api_key
394
+ "User-Agent: %s/%s" HTTP_ENDL
395
+ "Accept: */*" HTTP_HDR_END
396
+ , string2str(host->stream.snd.api_key)
397
, rrdhost_hostname(host)
478
- , rrdhost_registry_hostname(host)
479
- , host->machine_guid
398
+ , rrdhost_registry_hostname(host)
399
+ , host->machine_guid
400
, default_rrd_update_every
401
, rrdhost_os(host)
482
- , rrdhost_timezone(host)
483
- , rrdhost_abbrev_timezone(host)
484
- , host->utc_offset
485
- , host->sender->hops
402
+ , rrdhost_timezone(host)
403
+ , rrdhost_abbrev_timezone(host)
404
+ , host->utc_offset
405
+ , s->hops
406
, host->system_info->ml_capable
407
, host->system_info->ml_enabled
408
, host->system_info->mc_version
@@ -516,226 +436,321 @@ static bool sender_send_connection_request(RRDHOST *host, int default_port, int
436
, (host->system_info->host_disk_space) ? host->system_info->host_disk_space : ""
437
, STREAMING_PROTOCOL_VERSION
438
, rrdhost_program_name(host)
519
- , rrdhost_program_version(host)
439
+ , rrdhost_program_version(host)
440
);
441
http[eol] = 0x00;
442
rrdpush_clean_encoded(&se);
443
524
- if(!rrdpush_sender_connect_ssl(s))
525
- return false;
526
-
527
- if (s->parent_using_h2o && rrdpush_http_upgrade_prelude(host, s)) {
444
+ if (s->parent_using_h2o && stream_connect_upgrade_prelude(host, s)) {
445
ND_LOG_STACK lgs[] = {
529
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_CANT_UPGRADE_CONNECTION),
446
+ ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, STREAM_STATUS_CANT_UPGRADE_CONNECTION),
447
ND_LOG_FIELD_END(),
448
};
449
ND_LOG_STACK_PUSH(lgs);
450
534
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION);
535
- rrdpush_sender_thread_close_socket(s);
536
- host->destination->reason = STREAM_HANDSHAKE_ERROR_HTTP_UPGRADE;
537
- host->destination->postpone_reconnection_until = now_realtime_sec() + 1 * 60;
451
+ worker_is_busy(WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION);
452
+ nd_sock_close(&s->sock);
453
+ stream_parent_set_reconnect_delay(
454
+ host->stream.snd.parents.current, STREAM_HANDSHAKE_ERROR_HTTP_UPGRADE, 60);
455
return false;
456
}
457
458
ssize_t len = (ssize_t)strlen(http);
542
- ssize_t bytes = send_timeout(
543
- &host->sender->ssl,
544
- s->rrdpush_sender_socket,
545
- http,
546
- len,
547
- 0,
548
- timeout);
549
-
459
+ ssize_t bytes = nd_sock_send_timeout(&s->sock, http, len, 0, timeout);
460
if(bytes <= 0) { // timeout is 0
461
ND_LOG_STACK lgs[] = {
552
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_TIMEOUT),
462
+ ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, STREAM_STATUS_TIMEOUT),
463
ND_LOG_FIELD_END(),
464
};
465
ND_LOG_STACK_PUSH(lgs);
466
557
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_TIMEOUT);
558
- rrdpush_sender_thread_close_socket(s);
467
+ worker_is_busy(WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_TIMEOUT);
468
+ nd_sock_close(&s->sock);
469
470
nd_log(NDLS_DAEMON, NDLP_ERR,
471
"STREAM %s [send to %s]: failed to send HTTP header to remote netdata.",
472
rrdhost_hostname(host), s->connected_to);
473
564
- host->destination->reason = STREAM_HANDSHAKE_ERROR_SEND_TIMEOUT;
565
- host->destination->postpone_reconnection_until = now_realtime_sec() + 1 * 60;
474
+ stream_parent_set_reconnect_delay(
475
+ host->stream.snd.parents.current, STREAM_HANDSHAKE_ERROR_SEND_TIMEOUT, 60);
476
return false;
477
}
478
569
- bytes = recv_timeout(
570
- &host->sender->ssl,
571
- s->rrdpush_sender_socket,
572
- http,
573
- HTTP_HEADER_SIZE,
574
- 0,
575
- timeout);
576
-
479
+ bytes = nd_sock_recv_timeout(&s->sock, http, HTTP_HEADER_SIZE, 0, timeout);
480
if(bytes <= 0) { // timeout is 0
481
+ nd_sock_close(&s->sock);
482
+
483
ND_LOG_STACK lgs[] = {
579
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_TIMEOUT),
484
+ ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, STREAM_STATUS_TIMEOUT),
485
ND_LOG_FIELD_END(),
486
};
487
ND_LOG_STACK_PUSH(lgs);
488
584
- worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_TIMEOUT);
585
- rrdpush_sender_thread_close_socket(s);
489
+ worker_is_busy(WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_TIMEOUT);
490
491
nd_log(NDLS_DAEMON, NDLP_ERR,
492
"STREAM %s [send to %s]: remote netdata does not respond.",
493
rrdhost_hostname(host), s->connected_to);
494
591
- host->destination->reason = STREAM_HANDSHAKE_ERROR_RECEIVE_TIMEOUT;
592
- host->destination->postpone_reconnection_until = now_realtime_sec() + 30;
495
+ stream_parent_set_reconnect_delay(
496
+ host->stream.snd.parents.current, STREAM_HANDSHAKE_ERROR_RECEIVE_TIMEOUT, 30);
497
+
498
return false;
499
}
500
+ http[bytes] = '\0';
501
596
- if(sock_setnonblock(s->rrdpush_sender_socket) < 0)
502
+ if(sock_setnonblock(s->sock.fd) < 0)
503
nd_log(NDLS_DAEMON, NDLP_WARNING,
504
"STREAM %s [send to %s]: cannot set non-blocking mode for socket.",
505
rrdhost_hostname(host), s->connected_to);
600
- sock_setcloexec(s->rrdpush_sender_socket);
506
602
- if(sock_enlarge_out(s->rrdpush_sender_socket) < 0)
507
+ sock_setcloexec(s->sock.fd);
508
+
509
+ if(sock_enlarge_out(s->sock.fd) < 0)
510
nd_log(NDLS_DAEMON, NDLP_WARNING,
511
"STREAM %s [send to %s]: cannot enlarge the socket buffer.",
512
rrdhost_hostname(host), s->connected_to);
513
607
- http[bytes] = '\0';
608
- if(!rrdpush_sender_validate_response(host, s, http, bytes))
514
+ if(!stream_connect_validate_first_response(host, s, http, bytes)) {
515
+ nd_sock_close(&s->sock);
516
return false;
517
+ }
518
611
- rrdpush_compression_initialize(s);
519
+ stream_compression_initialize(s);
520
521
log_sender_capabilities(s);
522
523
ND_LOG_STACK lgs[] = {
616
- ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, RRDPUSH_STATUS_CONNECTED),
524
+ ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, STREAM_STATUS_CONNECTED),
525
ND_LOG_FIELD_END(),
526
};
527
ND_LOG_STACK_PUSH(lgs);
528
529
nd_log(NDLS_DAEMON, NDLP_DEBUG,
622
- "STREAM %s: connected to %s...",
530
+ "STREAM [connector] %s: connected to %s...",
531
rrdhost_hostname(host), s->connected_to);
532
533
return true;
534
}
535
628
-bool attempt_to_connect(struct sender_state *state) {
629
- ND_LOG_STACK lgs[] = {
630
- ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
631
- ND_LOG_FIELD_END(),
632
- };
633
- ND_LOG_STACK_PUSH(lgs);
536
+#define MAX_CONNECTORS 1
537
635
- state->send_attempts = 0;
538
+struct connector {
539
+ int8_t id;
540
+ pid_t tid;
541
+ ND_THREAD *thread;
542
+ struct completion completion;
543
637
- // reset the bytes we have sent for this session
638
- state->sent_bytes_on_this_connection = 0;
639
- memset(state->sent_bytes_on_this_connection_per_type, 0, sizeof(state->sent_bytes_on_this_connection_per_type));
544
+ size_t nodes;
545
641
- if(sender_send_connection_request(state->host, state->default_port, state->timeout, state)) {
642
- // reset the buffer, to properly send charts and metrics
643
- rrdpush_sender_on_connect(state->host);
546
+ struct {
547
+ // the incoming queue of the connector thread
548
+ // all other threads leave new senders here, to be connected to their parents
549
+ SPINLOCK spinlock;
550
+ SENDERS_JudyLSet senders;
551
+ } queue;
552
+};
553
645
- // send from the beginning
646
- state->begin = 0;
554
+static struct {
555
+ int id;
556
+ struct connector connectors[MAX_CONNECTORS];
557
+} connector_globals = { 0 };
558
648
- // make sure the next reconnection will be immediate
649
- state->not_connected_loops = 0;
559
+bool stream_connector_is_signaled_to_stop(struct sender_state *s) {
560
+ return __atomic_load_n(&s->exit.shutdown, __ATOMIC_RELAXED);
561
+}
562
651
- // let the data collection threads know we are ready
652
- rrdhost_flag_set(state->host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED);
563
+struct connector *stream_connector_get(struct sender_state *s) {
564
+ stream_sender_lock(s);
565
+
566
+ if(s->connector.id < 0 || s->connector.id >= MAX_CONNECTORS) {
567
+ // assign this to the dispatcher with fewer nodes
568
+
569
+ static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER;
570
+ spinlock_lock(&spinlock);
571
+ int min_slot = 0;
572
+ size_t min_nodes = __atomic_load_n(&connector_globals.connectors[0].nodes, __ATOMIC_RELAXED);
573
+ for(int i = 1; i < MAX_CONNECTORS ;i++) {
574
+ size_t nodes = __atomic_load_n(&connector_globals.connectors[i].nodes, __ATOMIC_RELAXED);
575
+ if(nodes < min_nodes) {
576
+ min_nodes = nodes;
577
+ min_slot = i;
578
+ }
579
+ }
580
+ __atomic_add_fetch(&connector_globals.connectors[min_slot].nodes, 1, __ATOMIC_RELAXED);
581
+ s->connector.id = min_slot;
582
+ spinlock_unlock(&spinlock);
583
+ }
584
654
- rrdpush_sender_after_connect(state->host);
585
+ struct connector *sc = &connector_globals.connectors[s->connector.id];
586
+ stream_sender_unlock(s);
587
656
- return true;
657
- }
588
+ return sc;
589
+}
590
659
- // we couldn't connect
591
+void stream_connector_requeue(struct sender_state *s) {
592
+ struct connector *sc = stream_connector_get(s);
593
661
- // increase the failed connections counter
662
- state->not_connected_loops++;
594
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
595
+ "STREAM [connector] [%s]: adding host in connector queue...",
596
+ rrdhost_hostname(s->host));
597
+
598
+ spinlock_lock(&sc->queue.spinlock);
599
+ internal_fatal(SENDERS_GET(&sc->queue.senders, (Word_t)s) != NULL, "Sender is already in the connector queue");
600
+ SENDERS_SET(&sc->queue.senders, (Word_t)s, s);
601
+ spinlock_unlock(&sc->queue.spinlock);
602
664
- // slow re-connection on repeating errors
665
- usec_t now_ut = now_monotonic_usec();
666
- usec_t end_ut = now_ut + USEC_PER_SEC * state->reconnect_delay;
667
- while(now_ut < end_ut) {
668
- if(nd_thread_signaled_to_cancel())
669
- return false;
603
+ // signal the connector to catch the job
604
+ completion_mark_complete_a_job(&sc->completion);
605
+}
606
671
- sleep_usec(100 * USEC_PER_MS); // seconds
672
- now_ut = now_monotonic_usec();
607
+void stream_connector_add(struct sender_state *s) {
608
+ // multiple threads may come here - only one should be able to pass through
609
+ stream_sender_lock(s);
610
+ if(!rrdhost_has_stream_sender_enabled(s->host) || !s->host->stream.snd.destination || !s->host->stream.snd.api_key) {
611
+ nd_log(NDLS_DAEMON, NDLP_ERR, "STREAM %s [send]: host has streaming disabled - not sending data to a parent.",
612
+ rrdhost_hostname(s->host));
613
+ stream_sender_unlock(s);
614
+ return;
615
+ }
616
+ if(rrdhost_flag_check(s->host, RRDHOST_FLAG_STREAM_SENDER_ADDED)) {
617
+ nd_log(NDLS_DAEMON, NDLP_DEBUG, "STREAM %s [send]: host has already added to sender - ignoring request",
618
+ rrdhost_hostname(s->host));
619
+ stream_sender_unlock(s);
620
+ return;
621
}
622
+ rrdhost_flag_set(s->host, RRDHOST_FLAG_STREAM_SENDER_ADDED);
623
+ rrdhost_flag_clear(s->host, RRDHOST_FLAG_STREAM_SENDER_CONNECTED | RRDHOST_FLAG_STREAM_SENDER_READY_4_METRICS);
624
+ stream_sender_unlock(s);
625
675
- return false;
626
+ nd_sock_close(&s->sock);
627
+ s->sbuf.cb->max_size = stream_send.buffer_max_size;
628
+ s->parent_using_h2o = stream_send.parents.h2o;
629
+
630
+ // do not call this with any locks held
631
+ stream_connector_requeue(s);
632
}
633
678
-bool rrdpush_sender_connect(struct sender_state *s) {
679
- worker_is_busy(WORKER_SENDER_JOB_CONNECT);
634
+static void stream_connector_remove(struct sender_state *s) {
635
+ nd_log(NDLS_DAEMON, NDLP_NOTICE,
636
+ "STREAM [connector] [%s]: stopped streaming connector for host: %s",
637
+ rrdhost_hostname(s->host), stream_handshake_error_to_string(s->exit.reason));
638
681
- time_t now_s = now_monotonic_sec();
682
- rrdpush_sender_cbuffer_recreate_timed(s, now_s, false, true);
683
- rrdpush_sender_execute_commands_cleanup(s);
639
+ struct connector *sc = stream_connector_get(s);
640
+ __atomic_sub_fetch(&sc->nodes, 1, __ATOMIC_RELAXED);
641
685
- rrdhost_flag_clear(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
686
- s->flags &= ~SENDER_FLAG_OVERFLOW;
687
- s->read_len = 0;
688
- s->buffer->read = 0;
689
- s->buffer->write = 0;
642
+ stream_sender_remove(s);
643
+}
644
691
- if(!attempt_to_connect(s))
692
- return false;
645
+static void *stream_connector_thread(void *ptr) {
646
+ struct connector *sc = ptr;
647
+ sc->tid = gettid_cached();
648
694
- if(rrdhost_sender_should_exit(s))
695
- return false;
649
+ worker_register("STREAMCNT");
650
+ worker_register_job_name(WORKER_SENDER_CONNECTOR_JOB_CONNECTING, "connect");
651
+ worker_register_job_name(WORKER_SENDER_CONNECTOR_JOB_CONNECTED, "connected");
652
+ worker_register_job_name(WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE, "bad handshake");
653
+ worker_register_job_name(WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_TIMEOUT, "timeout");
654
+ worker_register_job_name(WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION, "cant upgrade");
655
697
- s->last_traffic_seen_t = now_monotonic_sec();
698
- stream_path_send_to_parent(s->host);
699
- rrdpush_sender_send_claimed_id(s->host);
700
- rrdpush_send_host_labels(s->host);
701
- rrdpush_send_global_functions(s->host);
702
- s->replication.oldest_request_after_t = 0;
656
+ worker_register_job_custom_metric(WORKER_SENDER_CONNECTOR_JOB_QUEUED_NODES, "queued nodes", "nodes", WORKER_METRIC_ABSOLUTE);
657
+ worker_register_job_custom_metric(WORKER_SENDER_CONNECTOR_JOB_CONNECTED_NODES, "connected nodes", "nodes", WORKER_METRIC_ABSOLUTE);
658
+ worker_register_job_custom_metric(WORKER_SENDER_CONNECTOR_JOB_FAILED_NODES, "failed nodes", "nodes", WORKER_METRIC_ABSOLUTE);
659
+ worker_register_job_custom_metric(WORKER_SENDER_CONNECTOR_JOB_CANCELLED_NODES, "cancelled nodes", "nodes", WORKER_METRIC_ABSOLUTE);
660
704
- rrdhost_flag_set(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
661
+ unsigned job_id = 0;
662
706
- nd_log(NDLS_DAEMON, NDLP_DEBUG,
707
- "STREAM %s [send to %s]: enabling metrics streaming...",
708
- rrdhost_hostname(s->host), s->connected_to);
663
+ while(!nd_thread_signaled_to_cancel() && service_running(SERVICE_STREAMING)) {
664
+ worker_is_idle();
665
+ job_id = completion_wait_for_a_job_with_timeout(&sc->completion, job_id, 1000);
666
+ size_t nodes = 0, connected_nodes = 0, failed_nodes = 0, cancelled_nodes = 0;
667
710
- return true;
711
-}
668
+ spinlock_lock(&sc->queue.spinlock);
669
+ Word_t idx = 0;
670
+ for(struct sender_state *s = SENDERS_FIRST(&sc->queue.senders, &idx);
671
+ s;
672
+ s = SENDERS_NEXT(&sc->queue.senders, &idx)) {
673
+ nodes++;
674
713
-// Either the receiver lost the connection or the host is being destroyed.
714
-// The sender mutex guards thread creation, any spurious data is wiped on reconnection.
715
-void rrdpush_sender_thread_stop(RRDHOST *host, STREAM_HANDSHAKE reason, bool wait) {
716
- if (!host->sender)
717
- return;
675
+ ND_LOG_STACK lgs[] = {
676
+ ND_LOG_FIELD_STR(NDF_NIDL_NODE, s->host->hostname),
677
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
678
+ ND_LOG_FIELD_END(),
679
+ };
680
+ ND_LOG_STACK_PUSH(lgs);
681
719
- sender_lock(host->sender);
682
+ if(stream_connector_is_signaled_to_stop(s)) {
683
+ cancelled_nodes++;
684
+ SENDERS_DEL(&sc->queue.senders, (Word_t)s);
685
+ stream_connector_remove(s);
686
+ continue;
687
+ }
688
721
- if(rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN)) {
689
+ spinlock_unlock(&sc->queue.spinlock);
690
+ worker_is_busy(WORKER_SENDER_CONNECTOR_JOB_CONNECTING);
691
+ bool move_to_sender = stream_connect(s, stream_send.parents.default_port, stream_send.parents.timeout_s);
692
+ spinlock_lock(&sc->queue.spinlock);
693
723
- host->sender->exit.shutdown = true;
724
- host->sender->exit.reason = reason;
694
+ if(move_to_sender) {
695
+ connected_nodes++;
696
+ stream_sender_on_connect(s);
697
726
- // signal it to cancel
727
- nd_thread_signal_cancel(host->rrdpush_sender_thread);
728
- }
698
+ worker_is_busy(WORKER_SENDER_CONNECTOR_JOB_CONNECTED);
699
+ SENDERS_DEL(&sc->queue.senders, (Word_t)s);
700
+ spinlock_unlock(&sc->queue.spinlock);
701
+
702
+ // do not have the connector lock when calling this
703
+ stream_sender_add_to_queue(s);
704
730
- sender_unlock(host->sender);
705
+ spinlock_lock(&sc->queue.spinlock);
706
+ }
707
+ else
708
+ failed_nodes++;
709
732
- if(wait) {
733
- sender_lock(host->sender);
734
- while(host->sender->tid) {
735
- sender_unlock(host->sender);
736
- sleep_usec(10 * USEC_PER_MS);
737
- sender_lock(host->sender);
710
+ worker_is_idle();
711
}
739
- sender_unlock(host->sender);
712
+ spinlock_unlock(&sc->queue.spinlock);
713
+
714
+ worker_set_metric(WORKER_SENDER_CONNECTOR_JOB_QUEUED_NODES, (NETDATA_DOUBLE)nodes);
715
+ worker_set_metric(WORKER_SENDER_CONNECTOR_JOB_CONNECTED_NODES, (NETDATA_DOUBLE)connected_nodes);
716
+ worker_set_metric(WORKER_SENDER_CONNECTOR_JOB_FAILED_NODES, (NETDATA_DOUBLE)failed_nodes);
717
+ worker_set_metric(WORKER_SENDER_CONNECTOR_JOB_CANCELLED_NODES, (NETDATA_DOUBLE)cancelled_nodes);
718
+ }
719
+
720
+ return NULL;
721
+}
722
+
723
+bool stream_connector_init(struct sender_state *s) {
724
+ static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER;
725
+ if(!s) return false;
726
+
727
+ spinlock_lock(&spinlock);
728
+
729
+ struct connector *sc = stream_connector_get(s);
730
+
731
+ if(!sc->thread) {
732
+ sc->id = (int8_t)(sc - connector_globals.connectors); // find the slot number
733
+ if(&connector_globals.connectors[sc->id] != sc)
734
+ fatal("Connector ID and slot do not match!");
735
+
736
+ spinlock_init(&sc->queue.spinlock);
737
+ completion_init(&sc->completion);
738
+
739
+ char tag[NETDATA_THREAD_TAG_MAX + 1];
740
+ snprintfz(tag, NETDATA_THREAD_TAG_MAX, THREAD_TAG_STREAM_SENDER "-CN" "[%d]",
741
+ sc->id);
742
+
743
+ sc->thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_DEFAULT, stream_connector_thread, sc);
744
+ if (!sc->thread)
745
+ nd_log_daemon(NDLP_ERR, "STREAM connector: failed to create new thread for client.");
746
}
747
+
748
+ spinlock_unlock(&spinlock);
749
+
750
+ return sc->thread != NULL;
751
+}
752
+
753
+void stream_connector_cancel_threads(void) {
754
+ for(int id = 0; id < MAX_CONNECTORS ; id++)
755
+ nd_thread_signal_cancel(connector_globals.connectors[id].thread);
756
}
src/streaming/stream-handshake.c
+49
-34
@@ -1,51 +1,66 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#include "rrdpush.h"
3
+#include "stream.h"
4
5
static struct {
6
STREAM_HANDSHAKE err;
7
const char *str;
8
} handshake_errors[] = {
9
- { STREAM_HANDSHAKE_OK_V3, "CONNECTED" },
10
- { STREAM_HANDSHAKE_OK_V2, "CONNECTED" },
11
- { STREAM_HANDSHAKE_OK_V1, "CONNECTED" },
12
- { STREAM_HANDSHAKE_NEVER, "" },
13
- { STREAM_HANDSHAKE_ERROR_BAD_HANDSHAKE, "BAD HANDSHAKE" },
14
- { STREAM_HANDSHAKE_ERROR_LOCALHOST, "LOCALHOST" },
15
- { STREAM_HANDSHAKE_ERROR_ALREADY_CONNECTED, "ALREADY CONNECTED" },
16
- { STREAM_HANDSHAKE_ERROR_DENIED, "DENIED" },
17
- { STREAM_HANDSHAKE_ERROR_SEND_TIMEOUT, "SEND TIMEOUT" },
18
- { STREAM_HANDSHAKE_ERROR_RECEIVE_TIMEOUT, "RECEIVE TIMEOUT" },
19
- { STREAM_HANDSHAKE_ERROR_INVALID_CERTIFICATE, "INVALID CERTIFICATE" },
20
- { STREAM_HANDSHAKE_ERROR_SSL_ERROR, "SSL ERROR" },
21
- { STREAM_HANDSHAKE_ERROR_CANT_CONNECT, "CANT CONNECT" },
22
- { STREAM_HANDSHAKE_BUSY_TRY_LATER, "BUSY TRY LATER" },
23
- { STREAM_HANDSHAKE_INTERNAL_ERROR, "INTERNAL ERROR" },
24
- { STREAM_HANDSHAKE_INITIALIZATION, "REMOTE IS INITIALIZING" },
25
- { STREAM_HANDSHAKE_DISCONNECT_HOST_CLEANUP, "DISCONNECTED HOST CLEANUP" },
26
- { STREAM_HANDSHAKE_DISCONNECT_STALE_RECEIVER, "DISCONNECTED STALE RECEIVER" },
27
- { STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN, "DISCONNECTED SHUTDOWN REQUESTED" },
28
- { STREAM_HANDSHAKE_DISCONNECT_NETDATA_EXIT, "DISCONNECTED NETDATA EXIT" },
29
- { STREAM_HANDSHAKE_DISCONNECT_PARSER_EXIT, "DISCONNECTED PARSE ENDED" },
30
- {STREAM_HANDSHAKE_DISCONNECT_UNKNOWN_SOCKET_READ_ERROR, "DISCONNECTED UNKNOWN SOCKET READ ERROR" },
31
- { STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, "DISCONNECTED PARSE ERROR" },
32
- { STREAM_HANDSHAKE_DISCONNECT_RECEIVER_LEFT, "DISCONNECTED RECEIVER LEFT" },
33
- { STREAM_HANDSHAKE_DISCONNECT_ORPHAN_HOST, "DISCONNECTED ORPHAN HOST" },
34
- { STREAM_HANDSHAKE_NON_STREAMABLE_HOST, "NON STREAMABLE HOST" },
35
- { STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_READ_BUFFER, "DISCONNECTED NOT SUFFICIENT READ BUFFER" },
36
- {STREAM_HANDSHAKE_DISCONNECT_SOCKET_EOF, "DISCONNECTED SOCKET EOF" },
37
- {STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_FAILED, "DISCONNECTED SOCKET READ FAILED" },
38
- {STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_TIMEOUT, "DISCONNECTED SOCKET READ TIMEOUT" },
9
+ {STREAM_HANDSHAKE_OK_V3, "CONNECTED"},
10
+ {STREAM_HANDSHAKE_OK_V2, "CONNECTED"},
11
+ {STREAM_HANDSHAKE_OK_V1, "CONNECTED"},
12
+ {STREAM_HANDSHAKE_NEVER, ""},
13
+ {STREAM_HANDSHAKE_ERROR_BAD_HANDSHAKE, "BAD HANDSHAKE"},
14
+ {STREAM_HANDSHAKE_ERROR_LOCALHOST, "LOCALHOST"},
15
+ {STREAM_HANDSHAKE_ERROR_ALREADY_CONNECTED, "ALREADY CONNECTED"},
16
+ {STREAM_HANDSHAKE_ERROR_DENIED, "DENIED"},
17
+ {STREAM_HANDSHAKE_ERROR_SEND_TIMEOUT, "SEND TIMEOUT"},
18
+ {STREAM_HANDSHAKE_ERROR_RECEIVE_TIMEOUT, "RECEIVE TIMEOUT"},
19
+ {STREAM_HANDSHAKE_ERROR_INVALID_CERTIFICATE, "INVALID CERTIFICATE"},
20
+ {STREAM_HANDSHAKE_ERROR_SSL_ERROR, "SSL ERROR"},
21
+ {STREAM_HANDSHAKE_ERROR_CANT_CONNECT, "CANT CONNECT"},
22
+ {STREAM_HANDSHAKE_BUSY_TRY_LATER, "BUSY TRY LATER"},
23
+ {STREAM_HANDSHAKE_INTERNAL_ERROR, "INTERNAL ERROR"},
24
+ {STREAM_HANDSHAKE_INITIALIZATION, "REMOTE IS INITIALIZING"},
25
+ {STREAM_HANDSHAKE_DISCONNECT_HOST_CLEANUP, "DISCONNECTED HOST CLEANUP"},
26
+ {STREAM_HANDSHAKE_DISCONNECT_STALE_RECEIVER, "DISCONNECTED STALE RECEIVER"},
27
+ {STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN, "DISCONNECTED SHUTDOWN REQUESTED"},
28
+ {STREAM_HANDSHAKE_DISCONNECT_NETDATA_EXIT, "DISCONNECTED NETDATA EXIT"},
29
+ {STREAM_HANDSHAKE_DISCONNECT_PARSER_EXIT, "DISCONNECTED PARSE ENDED"},
30
+ {STREAM_HANDSHAKE_DISCONNECT_UNKNOWN_SOCKET_READ_ERROR, "DISCONNECTED UNKNOWN SOCKET READ ERROR"},
31
+ {STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, "DISCONNECTED PARSE ERROR"},
32
+ {STREAM_HANDSHAKE_DISCONNECT_RECEIVER_LEFT, "DISCONNECTED RECEIVER LEFT"},
33
+ {STREAM_HANDSHAKE_DISCONNECT_ORPHAN_HOST, "DISCONNECTED ORPHAN HOST"},
34
+ {STREAM_HANDSHAKE_NON_STREAMABLE_HOST, "NON STREAMABLE HOST"},
35
+ {STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_RECEIVER_READ_BUFFER, "DISCONNECTED NOT SUFFICIENT RCV READ BUFFER"},
36
+ {STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_SENDER_COMPRESSION_FAILED, "DISCONNECTED SND COMPRESSION FAILED"},
37
+ {STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_SENDER_SEND_BUFFER, "DISCONNECTED NOT SUFFICIENT SEND BUFFER"},
38
+ {STREAM_HANDSHAKE_DISCONNECT_SOCKET_EOF, "DISCONNECTED SOCKET EOF"},
39
+ {STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_FAILED, "DISCONNECTED SOCKET READ FAILED"},
40
+ {STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_TIMEOUT, "DISCONNECTED SOCKET READ TIMEOUT"},
41
+ {STREAM_HANDSHAKE_DISCONNECT_SOCKET_ERROR, "DISCONNECT SOCKET ERROR"},
42
+ {STREAM_HANDSHAKE_DISCONNECT_SOCKET_WRITE_FAILED, "DISCONNECTED SOCKET WRITE FAILED"},
43
+ {STREAM_HANDSHAKE_DISCONNECT_SOCKET_CLOSED_BY_PARENT, "DISCONNECTED SOCKET CLOSED BY PARENT"},
44
+ {STREAM_HANDSHAKE_ERROR_HTTP_UPGRADE, "HTTP UPGRADE ERROR"},
45
+ {STREAM_HANDSHAKE_NO_HOST_IN_DESTINATION, "NO HOST IN DESTINATION - CONFIG ERROR"},
46
+ {STREAM_HANDSHAKE_CONNECT_TIMEOUT, "CONNECT TIMEOUT"},
47
+ {STREAM_HANDSHAKE_CONNECTION_REFUSED, "CONNECTION REFUSED"},
48
+ {STREAM_HANDSHAKE_CANT_RESOLVE_HOSTNAME, "CANT RESOLVE HOSTNAME"},
49
+ {STREAM_HANDSHAKE_PREPARING, "PREPARING"},
50
+ {STREAM_HANDSHAKE_CONNECTING, "CONNECTING"},
51
+ {STREAM_HANDSHAKE_CONNECTED, "CONNECTED"},
52
+ {STREAM_HANDSHAKE_EXITING, "EXITING"},
53
+ {STREAM_HANDSHAKE_NO_STREAM_INFO, "NO STREAM INFO"},
54
{ 0, NULL },
55
};
56
42
-const char *stream_handshake_error_to_string(STREAM_HANDSHAKE handshake_error) {
43
- if(handshake_error >= STREAM_HANDSHAKE_OK_V1)
57
+const char *stream_handshake_error_to_string(STREAM_HANDSHAKE reason) {
58
+ if(reason >= STREAM_HANDSHAKE_OK_V1)
59
// handshake_error is the whole version / capabilities number
60
return "CONNECTED";
61
62
for(size_t i = 0; handshake_errors[i].str ; i++) {
48
- if(handshake_error == handshake_errors[i].err)
63
+ if(reason == handshake_errors[i].err)
64
return handshake_errors[i].str;
65
}
66
src/streaming/stream-handshake.h
+41
-30
@@ -17,30 +17,27 @@
17
#define START_STREAMING_ERROR_INTERNAL_ERROR "The server encountered an internal error. Try later."
18
#define START_STREAMING_ERROR_INITIALIZATION "The server is initializing. Try later."
19
20
-#define RRDPUSH_STATUS_CONNECTED "CONNECTED"
21
-#define RRDPUSH_STATUS_ALREADY_CONNECTED "ALREADY CONNECTED"
22
-#define RRDPUSH_STATUS_DISCONNECTED "DISCONNECTED"
23
-#define RRDPUSH_STATUS_RATE_LIMIT "RATE LIMIT TRY LATER"
24
-#define RRDPUSH_STATUS_INITIALIZATION_IN_PROGRESS "INITIALIZATION IN PROGRESS RETRY LATER"
25
-#define RRDPUSH_STATUS_INTERNAL_SERVER_ERROR "INTERNAL SERVER ERROR DROPPING CONNECTION"
26
-#define RRDPUSH_STATUS_DUPLICATE_RECEIVER "DUPLICATE RECEIVER DROPPING CONNECTION"
27
-#define RRDPUSH_STATUS_CANT_REPLY "CANT REPLY DROPPING CONNECTION"
28
-#define RRDPUSH_STATUS_NO_HOSTNAME "NO HOSTNAME PERMISSION DENIED"
29
-#define RRDPUSH_STATUS_NO_API_KEY "NO API KEY PERMISSION DENIED"
30
-#define RRDPUSH_STATUS_INVALID_API_KEY "INVALID API KEY PERMISSION DENIED"
31
-#define RRDPUSH_STATUS_NO_MACHINE_GUID "NO MACHINE GUID PERMISSION DENIED"
32
-#define RRDPUSH_STATUS_MACHINE_GUID_DISABLED "MACHINE GUID DISABLED PERMISSION DENIED"
33
-#define RRDPUSH_STATUS_INVALID_MACHINE_GUID "INVALID MACHINE GUID PERMISSION DENIED"
34
-#define RRDPUSH_STATUS_API_KEY_DISABLED "API KEY DISABLED PERMISSION DENIED"
35
-#define RRDPUSH_STATUS_NOT_ALLOWED_IP "NOT ALLOWED IP PERMISSION DENIED"
36
-#define RRDPUSH_STATUS_LOCALHOST "LOCALHOST PERMISSION DENIED"
37
-#define RRDPUSH_STATUS_PERMISSION_DENIED "PERMISSION DENIED"
38
-#define RRDPUSH_STATUS_BAD_HANDSHAKE "BAD HANDSHAKE"
39
-#define RRDPUSH_STATUS_TIMEOUT "TIMEOUT"
40
-#define RRDPUSH_STATUS_CANT_UPGRADE_CONNECTION "CANT UPGRADE CONNECTION"
41
-#define RRDPUSH_STATUS_SSL_ERROR "SSL ERROR"
42
-#define RRDPUSH_STATUS_INVALID_SSL_CERTIFICATE "INVALID SSL CERTIFICATE"
43
-#define RRDPUSH_STATUS_CANT_ESTABLISH_SSL_CONNECTION "CANT ESTABLISH SSL CONNECTION"
20
+#define STREAM_STATUS_CONNECTED "CONNECTED"
21
+#define STREAM_STATUS_ALREADY_CONNECTED "ALREADY CONNECTED"
22
+#define STREAM_STATUS_DISCONNECTED "DISCONNECTED"
23
+#define STREAM_STATUS_RATE_LIMIT "RATE LIMIT TRY LATER"
24
+#define STREAM_STATUS_INITIALIZATION_IN_PROGRESS "INITIALIZATION IN PROGRESS RETRY LATER"
25
+#define STREAM_STATUS_INTERNAL_SERVER_ERROR "INTERNAL SERVER ERROR DROPPING CONNECTION"
26
+#define STREAM_STATUS_DUPLICATE_RECEIVER "DUPLICATE RECEIVER DROPPING CONNECTION"
27
+#define STREAM_STATUS_CANT_REPLY "CANT REPLY DROPPING CONNECTION"
28
+#define STREAM_STATUS_NO_HOSTNAME "NO HOSTNAME PERMISSION DENIED"
29
+#define STREAM_STATUS_NO_API_KEY "NO API KEY PERMISSION DENIED"
30
+#define STREAM_STATUS_INVALID_API_KEY "INVALID API KEY PERMISSION DENIED"
31
+#define STREAM_STATUS_NO_MACHINE_GUID "NO MACHINE GUID PERMISSION DENIED"
32
+#define STREAM_STATUS_MACHINE_GUID_DISABLED "MACHINE GUID DISABLED PERMISSION DENIED"
33
+#define STREAM_STATUS_INVALID_MACHINE_GUID "INVALID MACHINE GUID PERMISSION DENIED"
34
+#define STREAM_STATUS_API_KEY_DISABLED "API KEY DISABLED PERMISSION DENIED"
35
+#define STREAM_STATUS_NOT_ALLOWED_IP "NOT ALLOWED IP PERMISSION DENIED"
36
+#define STREAM_STATUS_LOCALHOST "LOCALHOST PERMISSION DENIED"
37
+#define STREAM_STATUS_PERMISSION_DENIED "PERMISSION DENIED"
38
+#define STREAM_STATUS_BAD_HANDSHAKE "BAD HANDSHAKE"
39
+#define STREAM_STATUS_TIMEOUT "TIMEOUT"
40
+#define STREAM_STATUS_CANT_UPGRADE_CONNECTION "CANT UPGRADE CONNECTION"
41
42
typedef enum {
43
STREAM_HANDSHAKE_OK_V3 = 3, // v3+
@@ -69,14 +66,28 @@ typedef enum {
66
STREAM_HANDSHAKE_DISCONNECT_RECEIVER_LEFT = -20,
67
STREAM_HANDSHAKE_DISCONNECT_ORPHAN_HOST = -21,
68
STREAM_HANDSHAKE_NON_STREAMABLE_HOST = -22,
72
- STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_READ_BUFFER = -23,
73
- STREAM_HANDSHAKE_DISCONNECT_SOCKET_EOF = -24,
74
- STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_FAILED = -25,
75
- STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_TIMEOUT = -26,
76
- STREAM_HANDSHAKE_ERROR_HTTP_UPGRADE = -27,
69
+ STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_RECEIVER_READ_BUFFER = -23,
70
+ STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_SENDER_COMPRESSION_FAILED = -24,
71
+ STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_SENDER_SEND_BUFFER = -25,
72
+ STREAM_HANDSHAKE_DISCONNECT_SOCKET_EOF = -26,
73
+ STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_FAILED = -27,
74
+ STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_TIMEOUT = -28,
75
+ STREAM_HANDSHAKE_DISCONNECT_SOCKET_ERROR = -29,
76
+ STREAM_HANDSHAKE_DISCONNECT_SOCKET_WRITE_FAILED = -30,
77
+ STREAM_HANDSHAKE_DISCONNECT_SOCKET_CLOSED_BY_PARENT = -31,
78
+ STREAM_HANDSHAKE_ERROR_HTTP_UPGRADE = -32,
79
+ STREAM_HANDSHAKE_NO_HOST_IN_DESTINATION = -33,
80
+ STREAM_HANDSHAKE_CONNECT_TIMEOUT = -34,
81
+ STREAM_HANDSHAKE_CONNECTION_REFUSED = -35,
82
+ STREAM_HANDSHAKE_CANT_RESOLVE_HOSTNAME = -36,
83
+ STREAM_HANDSHAKE_PREPARING = -37,
84
+ STREAM_HANDSHAKE_CONNECTING = -38,
85
+ STREAM_HANDSHAKE_CONNECTED = -39,
86
+ STREAM_HANDSHAKE_EXITING = -40,
87
+ STREAM_HANDSHAKE_NO_STREAM_INFO = -41,
88
89
} STREAM_HANDSHAKE;
90
80
-const char *stream_handshake_error_to_string(STREAM_HANDSHAKE handshake_error);
91
+const char *stream_handshake_error_to_string(STREAM_HANDSHAKE reason);
92
93
#endif //NETDATA_STREAM_HANDSHAKE_H
src/streaming/stream-parents.c
new
+900
@@ -0,0 +1,900 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "stream-sender-internals.h"
4
+
5
+#define TIME_TO_CONSIDER_PARENTS_SIMILAR 120
6
+
7
+struct stream_parent {
8
+ STRING *destination; // the parent destination
9
+ bool ssl; // the parent uses SSL
10
+
11
+ bool banned_permanently; // when the parent is the origin of this host
12
+ bool banned_for_this_session; // when the parent is before us in the streaming path
13
+ bool banned_temporarily_erroneous; // when the parent is blocked by another node we host
14
+ STREAM_HANDSHAKE reason;
15
+ uint32_t attempts; // how many times we have tried to connect to this parent
16
+ usec_t since_ut; // the last time we tried to connect to it
17
+ usec_t postpone_until_ut; // based on the reason, a randomized time to wait for reconnection
18
+
19
+ struct {
20
+ ND_UUID host_id; // the machine_guid of the agent
21
+ int status; // the response code of the stream_info call
22
+ uint32_t nonce; // a random 32-bit number
23
+ size_t nodes; // how many nodes the parent has
24
+ size_t receivers; // how many receivers the parent has
25
+
26
+ // these are from RRDHOST_STATUS and can only be used when status == 200
27
+ RRDHOST_DB_STATUS db_status;
28
+ RRDHOST_DB_LIVENESS db_liveness;
29
+ RRDHOST_INGEST_TYPE ingest_type;
30
+ RRDHOST_INGEST_STATUS ingest_status;
31
+ time_t db_first_time_s; // the oldest timestamp for us in the parent's database
32
+ time_t db_last_time_s; // the latest timestamp for us in the parent's database
33
+ } remote;
34
+
35
+ struct {
36
+ size_t batch; // the batch priority (>= 1, 0 == excluded)
37
+ size_t order; // the final order of the parent (>= 1, 0 == excluded)
38
+ bool random; // this batch has more than 1 parents, so we flipped coins to select order
39
+ bool info; // we go stream info from the parent
40
+ bool skipped; // we skipped this parent for some reason
41
+ } selection;
42
+
43
+ STREAM_PARENT *prev;
44
+ STREAM_PARENT *next;
45
+};
46
+
47
+// --------------------------------------------------------------------------------------------------------------------
48
+// block unresponsive parents for some time, to allow speeding up the connection of the rest
49
+
50
+struct blocked_parent {
51
+ STRING *destination;
52
+ usec_t until;
53
+};
54
+
55
+DEFINE_JUDYL_TYPED(BLOCKED_PARENTS, struct blocked_parent *);
56
+static BLOCKED_PARENTS_JudyLSet blocked_parents_set = { 0 };
57
+static RW_SPINLOCK blocked_parents_spinlock = NETDATA_RW_SPINLOCK_INITIALIZER;
58
+
59
+static void block_parent_for_all_nodes(STREAM_PARENT *d, time_t duration_s) {
60
+ rw_spinlock_write_lock(&blocked_parents_spinlock);
61
+
62
+ struct blocked_parent *p = BLOCKED_PARENTS_GET(&blocked_parents_set, (Word_t)d->destination);
63
+ if(!p) {
64
+ p = callocz(1, sizeof(*p));
65
+ p->destination = string_dup(d->destination);
66
+ BLOCKED_PARENTS_SET(&blocked_parents_set, (Word_t)p->destination, p);
67
+ }
68
+ p->until = now_monotonic_usec() + duration_s * USEC_PER_SEC;
69
+
70
+ rw_spinlock_write_unlock(&blocked_parents_spinlock);
71
+}
72
+
73
+static bool is_a_blocked_parent(STREAM_PARENT *d) {
74
+ rw_spinlock_read_lock(&blocked_parents_spinlock);
75
+
76
+ struct blocked_parent *p = BLOCKED_PARENTS_GET(&blocked_parents_set, (Word_t)d->destination);
77
+ bool ret = p && p->until > now_monotonic_usec();
78
+
79
+ rw_spinlock_read_unlock(&blocked_parents_spinlock);
80
+ return ret;
81
+}
82
+
83
+// --------------------------------------------------------------------------------------------------------------------
84
+
85
+STREAM_HANDSHAKE stream_parent_get_disconnect_reason(STREAM_PARENT *d) {
86
+ if(!d) return STREAM_HANDSHAKE_INTERNAL_ERROR;
87
+ return d->reason;
88
+}
89
+
90
+void stream_parent_set_disconnect_reason(STREAM_PARENT *d, STREAM_HANDSHAKE reason, time_t since) {
91
+ if(!d) return;
92
+ d->since_ut = since * USEC_PER_SEC;
93
+ d->reason = reason;
94
+}
95
+
96
+static inline usec_t randomize_wait_ut(time_t min, time_t max) {
97
+ min = (min < SENDER_MIN_RECONNECT_DELAY ? SENDER_MIN_RECONNECT_DELAY : min);
98
+ if(max < min) max = min;
99
+
100
+ usec_t min_ut = min * USEC_PER_SEC;
101
+ usec_t max_ut = max * USEC_PER_SEC;
102
+ usec_t wait_ut = min_ut + os_random(max_ut - min_ut);
103
+ return now_realtime_usec() + wait_ut;
104
+}
105
+
106
+void rrdhost_stream_parents_reset(RRDHOST *host, STREAM_HANDSHAKE reason) {
107
+ usec_t until_ut = randomize_wait_ut(5, stream_send.parents.reconnect_delay_s);
108
+ rw_spinlock_write_lock(&host->stream.snd.parents.spinlock);
109
+ for (STREAM_PARENT *d = host->stream.snd.parents.all; d; d = d->next) {
110
+ d->postpone_until_ut = until_ut;
111
+ d->banned_for_this_session = false;
112
+ d->reason = reason;
113
+ }
114
+ rw_spinlock_write_unlock(&host->stream.snd.parents.spinlock);
115
+}
116
+
117
+void stream_parent_set_reconnect_delay(STREAM_PARENT *d, STREAM_HANDSHAKE reason, time_t secs) {
118
+ if(!d) return;
119
+ d->reason = reason;
120
+ d->postpone_until_ut = randomize_wait_ut(5, secs);
121
+}
122
+
123
+usec_t stream_parent_get_reconnection_ut(STREAM_PARENT *d) {
124
+ return d ? d->postpone_until_ut : 0;
125
+}
126
+
127
+bool stream_parent_is_ssl(STREAM_PARENT *d) {
128
+ return d ? d->ssl : false;
129
+}
130
+
131
+usec_t stream_parent_handshake_error_to_json(BUFFER *wb, RRDHOST *host) {
132
+ usec_t last_attempt = 0;
133
+ rw_spinlock_read_lock(&host->stream.snd.parents.spinlock);
134
+ for(STREAM_PARENT *d = host->stream.snd.parents.all; d ; d = d->next) {
135
+ if(d->since_ut > last_attempt)
136
+ last_attempt = d->since_ut;
137
+
138
+ buffer_json_add_array_item_string(wb, stream_handshake_error_to_string(d->reason));
139
+ }
140
+ rw_spinlock_read_unlock(&host->stream.snd.parents.spinlock);
141
+ return last_attempt;
142
+}
143
+
144
+void rrdhost_stream_parents_to_json(BUFFER *wb, RRDHOST_STATUS *s) {
145
+ char buf[1024];
146
+
147
+ rw_spinlock_read_lock(&s->host->stream.snd.parents.spinlock);
148
+
149
+ usec_t now_ut = now_realtime_usec();
150
+ STREAM_PARENT *d;
151
+ for (d = s->host->stream.snd.parents.all; d; d = d->next) {
152
+ buffer_json_add_array_item_object(wb);
153
+ buffer_json_member_add_uint64(wb, "attempts", d->attempts);
154
+ {
155
+ if (d->ssl) {
156
+ snprintfz(buf, sizeof(buf) - 1, "%s:SSL", string2str(d->destination));
157
+ buffer_json_member_add_string(wb, "destination", buf);
158
+ }
159
+ else
160
+ buffer_json_member_add_string(wb, "destination", string2str(d->destination));
161
+
162
+ buffer_json_member_add_datetime_rfc3339(wb, "since", d->since_ut, false);
163
+ buffer_json_member_add_duration_ut(wb, "age", d->since_ut < now_ut ? (int64_t)(now_ut - d->since_ut) : 0);
164
+
165
+ if(!d->banned_for_this_session && !d->banned_permanently && !d->banned_temporarily_erroneous) {
166
+ buffer_json_member_add_string(wb, "last_handshake", stream_handshake_error_to_string(d->reason));
167
+
168
+ if (d->postpone_until_ut > now_ut) {
169
+ buffer_json_member_add_datetime_rfc3339(wb, "next_check", d->postpone_until_ut, false);
170
+ buffer_json_member_add_duration_ut(wb, "next_in", (int64_t)(d->postpone_until_ut - now_ut));
171
+ }
172
+
173
+ if(d->selection.batch) {
174
+ buffer_json_member_add_uint64(wb, "batch", d->selection.batch);
175
+ buffer_json_member_add_uint64(wb, "order", d->selection.order);
176
+ buffer_json_member_add_boolean(wb, "random", d->selection.random);
177
+ }
178
+
179
+ buffer_json_member_add_boolean(wb, "info", d->selection.info);
180
+ buffer_json_member_add_boolean(wb, "skipped", d->selection.skipped);
181
+ }
182
+ else {
183
+ if(d->banned_permanently)
184
+ buffer_json_member_add_string(wb, "ban", "it is the localhost");
185
+ else if(d->banned_for_this_session)
186
+ buffer_json_member_add_string(wb, "ban", "it is our parent");
187
+ else if(d->banned_temporarily_erroneous)
188
+ buffer_json_member_add_string(wb, "ban", "it is erroneous");
189
+ }
190
+ }
191
+ buffer_json_object_close(wb); // each candidate
192
+ }
193
+
194
+ rw_spinlock_read_unlock(&s->host->stream.snd.parents.spinlock);
195
+}
196
+
197
+void rrdhost_stream_parent_ssl_init(struct sender_state *s) {
198
+ static SPINLOCK sp = NETDATA_SPINLOCK_INITIALIZER;
199
+ spinlock_lock(&sp);
200
+
201
+ if(netdata_ssl_streaming_sender_ctx || !s->host) {
202
+ spinlock_unlock(&sp);
203
+ goto cleanup;
204
+ }
205
+
206
+ rw_spinlock_read_lock(&s->host->stream.snd.parents.spinlock);
207
+
208
+ for(STREAM_PARENT *d = s->host->stream.snd.parents.all; d ; d = d->next) {
209
+ if (d->ssl) {
210
+ // we need to initialize SSL
211
+
212
+ netdata_ssl_initialize_ctx(NETDATA_SSL_STREAMING_SENDER_CTX);
213
+
214
+ ssl_security_location_for_context(
215
+ netdata_ssl_streaming_sender_ctx,
216
+ string2str(stream_send.parents.ssl_ca_file),
217
+ string2str(stream_send.parents.ssl_ca_path));
218
+
219
+ // stop the loop
220
+ break;
221
+ }
222
+ }
223
+
224
+ rw_spinlock_read_unlock(&s->host->stream.snd.parents.spinlock);
225
+ spinlock_unlock(&sp);
226
+
227
+cleanup:
228
+ s->sock.ctx = netdata_ssl_streaming_sender_ctx;
229
+ s->sock.verify_certificate = netdata_ssl_validate_certificate_sender;
230
+}
231
+
232
+static void stream_parent_nd_sock_error_to_reason(STREAM_PARENT *d, ND_SOCK *sock) {
233
+ switch (sock->error) {
234
+ case ND_SOCK_ERR_CONNECTION_REFUSED:
235
+ d->reason = STREAM_HANDSHAKE_CONNECTION_REFUSED;
236
+ d->postpone_until_ut = randomize_wait_ut(30, 60);
237
+ block_parent_for_all_nodes(d, 30);
238
+ break;
239
+
240
+ case ND_SOCK_ERR_CANNOT_RESOLVE_HOSTNAME:
241
+ d->reason = STREAM_HANDSHAKE_CANT_RESOLVE_HOSTNAME;
242
+ d->postpone_until_ut = randomize_wait_ut(30, 60);
243
+ block_parent_for_all_nodes(d, 30);
244
+ break;
245
+
246
+ case ND_SOCK_ERR_NO_HOST_IN_DEFINITION:
247
+ d->reason = STREAM_HANDSHAKE_NO_HOST_IN_DESTINATION;
248
+ d->banned_for_this_session = true;
249
+ d->postpone_until_ut = randomize_wait_ut(30, 60);
250
+ block_parent_for_all_nodes(d, 30);
251
+ break;
252
+
253
+ case ND_SOCK_ERR_TIMEOUT:
254
+ d->reason = STREAM_HANDSHAKE_CONNECT_TIMEOUT;
255
+ d->postpone_until_ut = randomize_wait_ut(300, d->remote.nodes < 10 ? 600 : 900);
256
+ block_parent_for_all_nodes(d, 300);
257
+ break;
258
+
259
+ case ND_SOCK_ERR_SSL_INVALID_CERTIFICATE:
260
+ d->reason = STREAM_HANDSHAKE_ERROR_INVALID_CERTIFICATE;
261
+ d->postpone_until_ut = randomize_wait_ut(300, 600);
262
+ block_parent_for_all_nodes(d, 300);
263
+ break;
264
+
265
+ case ND_SOCK_ERR_SSL_CANT_ESTABLISH_SSL_CONNECTION:
266
+ case ND_SOCK_ERR_SSL_FAILED_TO_OPEN:
267
+ d->reason = STREAM_HANDSHAKE_ERROR_SSL_ERROR;
268
+ d->postpone_until_ut = randomize_wait_ut(60, 180);
269
+ block_parent_for_all_nodes(d, 60);
270
+ break;
271
+
272
+ default:
273
+ case ND_SOCK_ERR_POLL_ERROR:
274
+ case ND_SOCK_ERR_FAILED_TO_CREATE_SOCKET:
275
+ case ND_SOCK_ERR_UNKNOWN_ERROR:
276
+ d->reason = STREAM_HANDSHAKE_INTERNAL_ERROR;
277
+ d->postpone_until_ut = randomize_wait_ut(30, 60);
278
+ break;
279
+
280
+ case ND_SOCK_ERR_THREAD_CANCELLED:
281
+ case ND_SOCK_ERR_NO_DESTINATION_AVAILABLE:
282
+ d->reason = STREAM_HANDSHAKE_INTERNAL_ERROR;
283
+ d->postpone_until_ut = randomize_wait_ut(30, 60);
284
+ break;
285
+ }
286
+}
287
+
288
+int stream_info_to_json_v1(BUFFER *wb, const char *machine_guid) {
289
+ buffer_reset(wb);
290
+ buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_DEFAULT);
291
+
292
+ RRDHOST_STATUS status = { 0 };
293
+ int ret = HTTP_RESP_OK;
294
+ RRDHOST *host;
295
+ if(!machine_guid || !*machine_guid || !(host = rrdhost_find_by_guid(machine_guid)))
296
+ ret = HTTP_RESP_NOT_FOUND;
297
+ else
298
+ rrdhost_status(host, now_realtime_sec(), &status);
299
+
300
+ buffer_json_member_add_uint64(wb, "version", 1);
301
+ buffer_json_member_add_uint64(wb, "status", ret);
302
+ buffer_json_member_add_uuid(wb, "host_id", localhost->host_id.uuid);
303
+ buffer_json_member_add_uint64(wb, "nodes", dictionary_entries(rrdhost_root_index));
304
+ buffer_json_member_add_uint64(wb, "receivers", stream_receivers_currently_connected());
305
+ buffer_json_member_add_uint64(wb, "nonce", os_random32());
306
+
307
+ if(ret == HTTP_RESP_OK) {
308
+ buffer_json_member_add_string(wb, "db_status", rrdhost_db_status_to_string(status.db.status));
309
+ buffer_json_member_add_string(wb, "db_liveness", rrdhost_db_liveness_to_string(status.db.liveness));
310
+ buffer_json_member_add_string(wb, "ingest_type", rrdhost_ingest_type_to_string(status.ingest.type));
311
+ buffer_json_member_add_string(wb, "ingest_status", rrdhost_ingest_status_to_string(status.ingest.status));
312
+ buffer_json_member_add_uint64(wb, "first_time_s", status.db.first_time_s);
313
+ buffer_json_member_add_uint64(wb, "last_time_s", status.db.last_time_s);
314
+ }
315
+
316
+ buffer_json_finalize(wb);
317
+ return ret;
318
+}
319
+
320
+static bool stream_info_json_parse_v1(struct json_object *jobj, const char *path, STREAM_PARENT *d, BUFFER *error) {
321
+ uint32_t version = 0; (void)version;
322
+ JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "version", version, error, true);
323
+
324
+ JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "status", d->remote.status, error, true);
325
+ JSONC_PARSE_TXT2UUID_OR_ERROR_AND_RETURN(jobj, path, "host_id", d->remote.host_id.uuid, error, true);
326
+ JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "nodes", d->remote.nodes, error, true);
327
+ JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "receivers", d->remote.receivers, error, true);
328
+ JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "nonce", d->remote.nonce, error, true);
329
+
330
+ if(d->remote.status == HTTP_RESP_OK) {
331
+ JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "first_time_s", d->remote.db_first_time_s, error, true);
332
+ JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "last_time_s", d->remote.db_last_time_s, error, true);
333
+ JSONC_PARSE_TXT2ENUM_OR_ERROR_AND_RETURN(jobj, path, "db_status", RRDHOST_DB_STATUS_2id, d->remote.db_status, error, true);
334
+ JSONC_PARSE_TXT2ENUM_OR_ERROR_AND_RETURN(jobj, path, "db_liveness", RRDHOST_DB_LIVENESS_2id, d->remote.db_liveness, error, true);
335
+ JSONC_PARSE_TXT2ENUM_OR_ERROR_AND_RETURN(jobj, path, "ingest_type", RRDHOST_INGEST_TYPE_2id, d->remote.ingest_type, error, true);
336
+ JSONC_PARSE_TXT2ENUM_OR_ERROR_AND_RETURN(jobj, path, "ingest_status", RRDHOST_INGEST_STATUS_2id, d->remote.ingest_status, error, true);
337
+ return true;
338
+ }
339
+
340
+ d->remote.db_first_time_s = 0;
341
+ d->remote.db_last_time_s = 0;
342
+ d->remote.db_status = 0;
343
+ d->remote.db_liveness = 0;
344
+ d->remote.ingest_type = 0;
345
+ d->remote.ingest_status = 0;
346
+ return false;
347
+}
348
+
349
+static bool stream_info_fetch(STREAM_PARENT *d, const char *uuid, int default_port, ND_SOCK *sender_sock, bool ssl, const char *hostname) {
350
+ ND_LOG_STACK lgs[] = {
351
+ ND_LOG_FIELD_STR(NDF_DST_IP, d->destination),
352
+ ND_LOG_FIELD_I64(NDF_DST_PORT, default_port),
353
+ ND_LOG_FIELD_TXT(NDF_REQUEST_METHOD, "GET"),
354
+ ND_LOG_FIELD_END(),
355
+ };
356
+ ND_LOG_STACK_PUSH(lgs);
357
+
358
+ char buf[HTTP_HEADER_SIZE];
359
+ CLEAN_ND_SOCK sock = ND_SOCK_INIT(sender_sock->ctx, sender_sock->verify_certificate);
360
+
361
+ // Build HTTP request
362
+ snprintf(buf, sizeof(buf),
363
+ "GET /api/v3/stream_info?machine_guid=%s" HTTP_1_1 HTTP_ENDL
364
+ "Host: %s" HTTP_ENDL
365
+ "User-Agent: %s/%s" HTTP_ENDL
366
+ "Accept: */*" HTTP_ENDL
367
+ "Accept-Encoding: identity" HTTP_ENDL // disable chunked encoding
368
+ "TE: identity" HTTP_ENDL // disable chunked encoding
369
+ "Pragma: no-cache" HTTP_ENDL
370
+ "Cache-Control: no-cache" HTTP_ENDL
371
+ "Connection: close" HTTP_HDR_END,
372
+ uuid,
373
+ string2str(d->destination),
374
+ rrdhost_program_name(localhost),
375
+ rrdhost_program_version(localhost));
376
+
377
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
378
+ "STREAM PARENTS of %s: fetching stream info from '%s'...",
379
+ hostname, string2str(d->destination));
380
+
381
+ // Establish connection
382
+ d->reason = STREAM_HANDSHAKE_CONNECTING;
383
+ if (!nd_sock_connect_to_this(&sock, string2str(d->destination), default_port, 5, ssl)) {
384
+ d->selection.info = false;
385
+ stream_parent_nd_sock_error_to_reason(d, &sock);
386
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
387
+ "STREAM PARENTS of %s: failed to connect for stream info to '%s': %s",
388
+ hostname, string2str(d->destination),
389
+ ND_SOCK_ERROR_2str(sock.error));
390
+ return false;
391
+ }
392
+
393
+ // Send HTTP request
394
+ ssize_t sent = nd_sock_send_timeout(&sock, buf, strlen(buf), 0, 5);
395
+ if (sent <= 0) {
396
+ d->selection.info = false;
397
+ stream_parent_nd_sock_error_to_reason(d, &sock);
398
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
399
+ "STREAM PARENTS of %s: failed to send stream info request to '%s': %s",
400
+ hostname, string2str(d->destination),
401
+ ND_SOCK_ERROR_2str(sock.error));
402
+ return false;
403
+ }
404
+
405
+ // Receive HTTP response
406
+ size_t total_received = 0;
407
+ size_t payload_received = 0;
408
+ size_t content_length = 0;
409
+ char *payload_start = NULL;
410
+
411
+ while (!payload_received || content_length < payload_received) {
412
+ size_t remaining = sizeof(buf) - total_received;
413
+
414
+ if (remaining <= 1) {
415
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
416
+ "STREAM PARENTS of %s: stream info receive buffer is full while receiving response from '%s'",
417
+ hostname, string2str(d->destination));
418
+ d->selection.info = false;
419
+ d->reason = STREAM_HANDSHAKE_INTERNAL_ERROR;
420
+ return false;
421
+ }
422
+
423
+ ssize_t received = nd_sock_recv_timeout(&sock, buf + total_received, remaining - 1, 0, 5);
424
+ if (received <= 0) {
425
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
426
+ "STREAM PARENTS of %s: socket receive error while querying stream info on '%s' "
427
+ "(total received %zu, payload received %zu, content length %zu): %s",
428
+ hostname, string2str(d->destination),
429
+ total_received, payload_received, content_length,
430
+ ND_SOCK_ERROR_2str(sock.error));
431
+
432
+ d->selection.info = false;
433
+ stream_parent_nd_sock_error_to_reason(d, &sock);
434
+ return false;
435
+ }
436
+
437
+ total_received += received;
438
+ buf[total_received] = '\0';
439
+
440
+ if(!payload_start) {
441
+ char *headers_end = strstr(buf, HTTP_HDR_END);
442
+ if (!headers_end)
443
+ // we have not received the whole header yet
444
+ continue;
445
+
446
+ payload_start = headers_end + sizeof(HTTP_HDR_END) - 1;
447
+ }
448
+
449
+ // the payload size so far
450
+ payload_received = total_received - (payload_start - buf);
451
+
452
+ if(!content_length) {
453
+ char *content_length_ptr = strstr(buf, "Content-Length: ");
454
+ if (!content_length_ptr) {
455
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
456
+ "STREAM PARENTS of %s: stream info response from '%s' does not have a Content-Length",
457
+ hostname, string2str(d->destination));
458
+
459
+ d->selection.info = false;
460
+ d->reason = STREAM_HANDSHAKE_INTERNAL_ERROR;
461
+ return false;
462
+ }
463
+ content_length = strtoul(content_length_ptr + strlen("Content-Length: "), NULL, 10);
464
+ if (!content_length) {
465
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
466
+ "STREAM PARENTS of %s: stream info response from '%s' has invalid Content-Length",
467
+ hostname, string2str(d->destination));
468
+
469
+ d->selection.info = false;
470
+ d->reason = STREAM_HANDSHAKE_INTERNAL_ERROR;
471
+ return false;
472
+ }
473
+ }
474
+ }
475
+
476
+ // Parse HTTP response and extract JSON
477
+ CLEAN_JSON_OBJECT *jobj = json_tokener_parse(payload_start);
478
+ if (!jobj) {
479
+ d->selection.info = false;
480
+ d->reason = STREAM_HANDSHAKE_NO_STREAM_INFO;
481
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
482
+ "STREAM PARENTS of %s: failed to parse stream info response from '%s', JSON data: %s",
483
+ hostname, string2str(d->destination), payload_start);
484
+ return false;
485
+ }
486
+
487
+ CLEAN_BUFFER *error = buffer_create(0, NULL);
488
+
489
+ if(!stream_info_json_parse_v1(jobj, "", d, error)) {
490
+ d->selection.info = false;
491
+ d->reason = STREAM_HANDSHAKE_NO_STREAM_INFO;
492
+ nd_log(NDLS_DAEMON, NDLP_WARNING,
493
+ "STREAM PARENTS of %s: failed to extract fields from JSON stream info response from '%s': %s",
494
+ hostname, string2str(d->destination),
495
+ buffer_tostring(error));
496
+ return false;
497
+ }
498
+
499
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
500
+ "STREAM PARENTS of %s: received stream_info data from '%s': "
501
+ "status: %d, nodes: %zu, receivers: %zu, first_time_s: %ld, last_time_s: %ld, "
502
+ "db status: %s, db liveness: %s, ingest type: %s, ingest status: %s",
503
+ hostname, string2str(d->destination),
504
+ d->remote.status, d->remote.nodes, d->remote.receivers,
505
+ d->remote.db_first_time_s, d->remote.db_last_time_s,
506
+ RRDHOST_DB_STATUS_2str(d->remote.db_status),
507
+ RRDHOST_DB_LIVENESS_2str(d->remote.db_liveness),
508
+ RRDHOST_INGEST_TYPE_2str(d->remote.ingest_type),
509
+ RRDHOST_INGEST_STATUS_2str(d->remote.ingest_status));
510
+
511
+ d->selection.info = true;
512
+ d->reason = STREAM_HANDSHAKE_NEVER;
513
+ return true;
514
+}
515
+
516
+static int compare_last_time(const void *a, const void *b) {
517
+ STREAM_PARENT *parent_a = *(STREAM_PARENT **)a;
518
+ STREAM_PARENT *parent_b = *(STREAM_PARENT **)b;
519
+
520
+ if (parent_a->remote.db_last_time_s < parent_b->remote.db_last_time_s) return 1;
521
+ else if (parent_a->remote.db_last_time_s > parent_b->remote.db_last_time_s) return -1;
522
+ else {
523
+ if(parent_a->since_ut < parent_b->since_ut) return -1;
524
+ else if(parent_a->since_ut > parent_b->since_ut) return 1;
525
+ else {
526
+ if(parent_a->attempts < parent_b->attempts) return -1;
527
+ else if(parent_a->attempts > parent_b->attempts) return 1;
528
+ else return 0;
529
+ }
530
+ }
531
+}
532
+
533
+bool stream_parent_connect_to_one_unsafe(
534
+ ND_SOCK *sender_sock,
535
+ RRDHOST *host,
536
+ int default_port,
537
+ time_t timeout,
538
+ char *connected_to,
539
+ size_t connected_to_size,
540
+ STREAM_PARENT **destination)
541
+{
542
+ sender_sock->error = ND_SOCK_ERR_NO_DESTINATION_AVAILABLE;
543
+
544
+ // count the parents
545
+ size_t size = 0;
546
+ for (STREAM_PARENT *d = host->stream.snd.parents.all; d; d = d->next) {
547
+ d->selection.order = 0;
548
+ d->selection.batch = 0;
549
+ d->selection.random = false;
550
+ d->selection.info = false;
551
+ d->selection.skipped = true;
552
+ size++;
553
+ }
554
+
555
+ // do we have any parents?
556
+ if(!size) {
557
+ nd_log(NDLS_DAEMON, NDLP_DEBUG, "STREAM PARENTS of %s: no parents configured", rrdhost_hostname(host));
558
+ return false;
559
+ }
560
+
561
+ STREAM_PARENT *array[size];
562
+ usec_t now_ut = now_realtime_usec();
563
+
564
+ // fetch stream info for all of them and put them in the array
565
+ size_t count = 0, skipped_but_useful = 0, skipped_not_useful = 0;
566
+ for (STREAM_PARENT *d = host->stream.snd.parents.all; d && count < size ; d = d->next) {
567
+ if (nd_thread_signaled_to_cancel()) {
568
+ sender_sock->error = ND_SOCK_ERR_THREAD_CANCELLED;
569
+ return false;
570
+ }
571
+
572
+ // make sure they all have a random number
573
+ // this is taken from the parent, but if the stream_info call fails
574
+ // we generate a random number for every parent here
575
+ d->remote.nonce = os_random32();
576
+ d->banned_temporarily_erroneous = is_a_blocked_parent(d);
577
+
578
+ if (d->banned_permanently || d->banned_for_this_session || d->banned_temporarily_erroneous)
579
+ continue;
580
+
581
+ if (d->postpone_until_ut > now_ut) {
582
+ skipped_but_useful++;
583
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
584
+ "STREAM PARENTS of %s: skipping useful parent '%s': POSTPONED FOR %ld SECS MORE: %s",
585
+ rrdhost_hostname(host),
586
+ string2str(d->destination),
587
+ (time_t)((d->postpone_until_ut - now_ut) / USEC_PER_SEC),
588
+ stream_handshake_error_to_string(d->reason));
589
+ continue;
590
+ }
591
+
592
+ bool skip = false;
593
+ if(stream_info_fetch(d, host->machine_guid, default_port,
594
+ sender_sock, stream_parent_is_ssl(d), rrdhost_hostname(host))) {
595
+ switch(d->remote.ingest_type) {
596
+ case RRDHOST_INGEST_TYPE_VIRTUAL:
597
+ case RRDHOST_INGEST_TYPE_LOCALHOST:
598
+ d->reason = STREAM_HANDSHAKE_ERROR_LOCALHOST;
599
+ if(rrdhost_is_host_in_stream_path_before_us(host, d->remote.host_id, 1)) {
600
+ // we passed hops == 1, to make sure this succeeds only when the parent
601
+ // is the origin child of this node
602
+ d->since_ut = now_ut;
603
+ d->banned_permanently = true;
604
+ skipped_not_useful++;
605
+ nd_log(NDLS_DAEMON, NDLP_NOTICE,
606
+ "STREAM PARENTS of %s: destination '%s' is banned permanently because it is the origin server",
607
+ rrdhost_hostname(host), string2str(d->destination));
608
+ continue;
609
+ }
610
+ else
611
+ skip = true;
612
+ break;
613
+
614
+ default:
615
+ case RRDHOST_INGEST_TYPE_CHILD:
616
+ case RRDHOST_INGEST_TYPE_ARCHIVED:
617
+ break;
618
+ }
619
+
620
+ switch(d->remote.ingest_status) {
621
+ case RRDHOST_INGEST_STATUS_INITIALIZING:
622
+ d->reason = STREAM_HANDSHAKE_INITIALIZATION;
623
+ skip = true;
624
+ break;
625
+
626
+ case RRDHOST_INGEST_STATUS_REPLICATING:
627
+ case RRDHOST_INGEST_STATUS_ONLINE:
628
+ d->reason = STREAM_HANDSHAKE_ERROR_ALREADY_CONNECTED;
629
+ if(rrdhost_is_host_in_stream_path_before_us(host, d->remote.host_id, host->sender->hops)) {
630
+ d->since_ut = now_ut;
631
+ d->banned_for_this_session = true;
632
+ skipped_not_useful++;
633
+ nd_log(NDLS_DAEMON, NDLP_NOTICE,
634
+ "STREAM PARENTS of %s: destination '%s' is banned for this session, because it is in our path before us.",
635
+ rrdhost_hostname(host), string2str(d->destination));
636
+ continue;
637
+ }
638
+ else
639
+ skip = true;
640
+ break;
641
+
642
+ default:
643
+ case RRDHOST_INGEST_STATUS_OFFLINE:
644
+ break;
645
+ }
646
+ }
647
+
648
+ if(skip) {
649
+ skipped_but_useful++;
650
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
651
+ "STREAM PARENTS of %s: skipping useful parent '%s': %s",
652
+ rrdhost_hostname(host),
653
+ string2str(d->destination),
654
+ stream_handshake_error_to_string(d->reason));
655
+ }
656
+ else {
657
+ d->selection.skipped = false;
658
+ d->selection.batch = count + 1;
659
+ d->selection.order = count + 1;
660
+ array[count++] = d;
661
+ }
662
+ }
663
+
664
+ // can we use any parent?
665
+ if(!count) {
666
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
667
+ "STREAM PARENTS of %s: no parents available (%zu skipped but useful, %zu skipped not useful)",
668
+ rrdhost_hostname(host),
669
+ skipped_but_useful, skipped_not_useful);
670
+ return false;
671
+ }
672
+
673
+ // order the parents in the array the way we want to connect
674
+ if(count > 1) {
675
+ qsort(array, count, sizeof(STREAM_PARENT *), compare_last_time);
676
+
677
+ size_t base = 0, batch = 0;
678
+ while (base < count) {
679
+ // find how many have similar db_last_time_s;
680
+ size_t similar = 1;
681
+ if(!array[base]->remote.nonce) array[base]->remote.nonce = os_random32();
682
+ time_t tB = array[base]->remote.db_last_time_s;
683
+ for (size_t i = base + 1; i < count; i++) {
684
+ time_t tN = array[i]->remote.db_last_time_s;
685
+ if ((tN > tB && tN - tB <= TIME_TO_CONSIDER_PARENTS_SIMILAR) ||
686
+ (tB - tN <= TIME_TO_CONSIDER_PARENTS_SIMILAR))
687
+ similar++;
688
+ else
689
+ break;
690
+ }
691
+
692
+ // if we have only 1 similar, move on
693
+ if (similar == 1) {
694
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
695
+ "STREAM PARENTS of %s: reordering keeps parent No %zu, '%s'",
696
+ rrdhost_hostname(host), base, string2str(array[base]->destination));
697
+ array[base]->selection.order = base + 1;
698
+ array[base]->selection.batch = batch + 1;
699
+ array[base]->selection.random = false;
700
+ base++;
701
+ batch++;
702
+ continue;
703
+ }
704
+ else {
705
+ // reorder the parents who have similar db_last_time
706
+
707
+ while (similar > 1) {
708
+ size_t chosen = base;
709
+ for(size_t i = base + 1 ; i < base + similar ;i++) {
710
+ uint32_t i_nonce = array[i]->remote.nonce | os_random32();
711
+ uint32_t chosen_nonce = array[chosen]->remote.nonce | os_random32();
712
+ if(i_nonce > chosen_nonce) chosen = i;
713
+ }
714
+
715
+ if (chosen != base)
716
+ SWAP(array[base], array[chosen]);
717
+
718
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
719
+ "STREAM PARENTS of %s: random reordering of %zu similar parents (slots %zu to %zu), No %zu is '%s'",
720
+ rrdhost_hostname(host),
721
+ similar, base, base + similar,
722
+ base, string2str(array[base]->destination));
723
+
724
+ array[base]->selection.order = base + 1;
725
+ array[base]->selection.batch = batch + 1;
726
+ array[base]->selection.random = true;
727
+ base++;
728
+ similar--;
729
+ }
730
+
731
+ // the last one of the similar
732
+ array[base]->selection.order = base + 1;
733
+ array[base]->selection.batch = batch + 1;
734
+ array[base]->selection.random = true;
735
+ base++;
736
+ batch++;
737
+ }
738
+ }
739
+ }
740
+ else {
741
+ array[0]->selection.order = 1;
742
+ array[0]->selection.batch = 1;
743
+ array[0]->selection.random = false;
744
+
745
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
746
+ "STREAM PARENTS of %s: only 1 parent is available: '%s'",
747
+ rrdhost_hostname(host), string2str(array[0]->destination));
748
+ }
749
+
750
+ // now the parents are sorted based on preference of connection
751
+ for(size_t i = 0; i < count ;i++) {
752
+ STREAM_PARENT *d = array[i];
753
+
754
+ if(d->postpone_until_ut > now_ut)
755
+ continue;
756
+
757
+ if(nd_thread_signaled_to_cancel()) {
758
+ sender_sock->error = ND_SOCK_ERR_THREAD_CANCELLED;
759
+ return false;
760
+ }
761
+
762
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
763
+ "STREAM PARENTS of %s: connecting to '%s' (default port: %d, parent %zu of %zu)...",
764
+ rrdhost_hostname(host), string2str(d->destination), default_port,
765
+ i + 1, count);
766
+
767
+ ND_LOG_STACK lgs[] = {
768
+ ND_LOG_FIELD_STR(NDF_DST_IP, d->destination),
769
+ ND_LOG_FIELD_I64(NDF_DST_PORT, default_port),
770
+ ND_LOG_FIELD_END(),
771
+ };
772
+ ND_LOG_STACK_PUSH(lgs);
773
+
774
+ d->since_ut = now_ut;
775
+ d->attempts++;
776
+ if (nd_sock_connect_to_this(sender_sock, string2str(d->destination),
777
+ default_port, timeout, stream_parent_is_ssl(d))) {
778
+
779
+ if (connected_to && connected_to_size)
780
+ strncpyz(connected_to, string2str(d->destination), connected_to_size);
781
+
782
+ *destination = d;
783
+
784
+ // move the current item to the end of the list
785
+ // without this, this destination will break the loop again and again
786
+ // not advancing the destinations to find one that may work
787
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(host->stream.snd.parents.all, d, prev, next);
788
+ DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(host->stream.snd.parents.all, d, prev, next);
789
+
790
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
791
+ "STREAM PARENTS of %s: connected to '%s' (default port: %d, fd %d)...",
792
+ rrdhost_hostname(host), string2str(d->destination), default_port,
793
+ sender_sock->fd);
794
+
795
+ sender_sock->error = ND_SOCK_ERR_NONE;
796
+ return true;
797
+ }
798
+ else {
799
+ stream_parent_nd_sock_error_to_reason(d, sender_sock);
800
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
801
+ "STREAM PARENTS of %s: stream connection to '%s' failed (default port: %d): %s",
802
+ rrdhost_hostname(host),
803
+ string2str(d->destination), default_port,
804
+ ND_SOCK_ERROR_2str(sender_sock->error));
805
+ }
806
+ }
807
+
808
+ return false;
809
+}
810
+
811
+bool stream_parent_connect_to_one(
812
+ ND_SOCK *sender_sock,
813
+ RRDHOST *host,
814
+ int default_port,
815
+ time_t timeout,
816
+ char *connected_to,
817
+ size_t connected_to_size,
818
+ STREAM_PARENT **destination) {
819
+
820
+ rw_spinlock_read_lock(&host->stream.snd.parents.spinlock);
821
+ bool rc = stream_parent_connect_to_one_unsafe(
822
+ sender_sock, host, default_port, timeout, connected_to, connected_to_size, destination);
823
+ rw_spinlock_read_unlock(&host->stream.snd.parents.spinlock);
824
+ return rc;
825
+}
826
+
827
+// --------------------------------------------------------------------------------------------------------------------
828
+// create stream parents linked list
829
+
830
+struct stream_parent_init_tmp {
831
+ RRDHOST *host;
832
+ STREAM_PARENT *list;
833
+ int count;
834
+};
835
+
836
+static bool stream_parent_add_one_unsafe(char *entry, void *data) {
837
+ struct stream_parent_init_tmp *t = data;
838
+
839
+ STREAM_PARENT *d = callocz(1, sizeof(STREAM_PARENT));
840
+ char *colon_ssl = strstr(entry, ":SSL");
841
+ if(colon_ssl) {
842
+ *colon_ssl = '\0';
843
+ d->ssl = true;
844
+ }
845
+ else
846
+ d->ssl = false;
847
+
848
+ d->destination = string_strdupz(entry);
849
+ d->since_ut = now_realtime_usec();
850
+
851
+ __atomic_add_fetch(&netdata_buffers_statistics.rrdhost_senders, sizeof(STREAM_PARENT), __ATOMIC_RELAXED);
852
+
853
+ DOUBLE_LINKED_LIST_APPEND_ITEM_UNSAFE(t->list, d, prev, next);
854
+
855
+ t->count++;
856
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
857
+ "STREAM PARENTS of %s: added streaming destination No %d: '%s'",
858
+ rrdhost_hostname(t->host), t->count, string2str(d->destination));
859
+
860
+ return false; // we return false, so that we will get all defined destinations
861
+}
862
+
863
+void rrdhost_stream_parents_update_from_destination(RRDHOST *host) {
864
+ rw_spinlock_write_lock(&host->stream.snd.parents.spinlock);
865
+ rrdhost_stream_parents_free(host, true);
866
+
867
+ if(host->stream.snd.destination) {
868
+ struct stream_parent_init_tmp t = {
869
+ .host = host,
870
+ .list = NULL,
871
+ .count = 0,
872
+ };
873
+ foreach_entry_in_connection_string(string2str(host->stream.snd.destination), stream_parent_add_one_unsafe, &t);
874
+ host->stream.snd.parents.all = t.list;
875
+ }
876
+
877
+ rw_spinlock_write_unlock(&host->stream.snd.parents.spinlock);
878
+}
879
+
880
+void rrdhost_stream_parents_free(RRDHOST *host, bool having_write_lock) {
881
+ if(!having_write_lock)
882
+ rw_spinlock_write_lock(&host->stream.snd.parents.spinlock);
883
+
884
+ while (host->stream.snd.parents.all) {
885
+ STREAM_PARENT *tmp = host->stream.snd.parents.all;
886
+ DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(host->stream.snd.parents.all, tmp, prev, next);
887
+ string_freez(tmp->destination);
888
+ freez(tmp);
889
+ __atomic_sub_fetch(&netdata_buffers_statistics.rrdhost_senders, sizeof(STREAM_PARENT), __ATOMIC_RELAXED);
890
+ }
891
+
892
+ host->stream.snd.parents.all = NULL;
893
+
894
+ if(!having_write_lock)
895
+ rw_spinlock_write_unlock(&host->stream.snd.parents.spinlock);
896
+}
897
+
898
+void rrdhost_stream_parents_init(RRDHOST *host) {
899
+ rw_spinlock_init(&host->stream.snd.parents.spinlock);
900
+}
src/streaming/stream-parents.h
new
+51
@@ -0,0 +1,51 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_STREAM_PARENTS_H
4
+#define NETDATA_STREAM_PARENTS_H
5
+
6
+#include "libnetdata/libnetdata.h"
7
+
8
+struct rrdhost;
9
+struct rrdhost_status;
10
+struct stream_parent;
11
+typedef struct stream_parent STREAM_PARENT;
12
+
13
+typedef struct rrdhost_stream_parents {
14
+ RW_SPINLOCK spinlock;
15
+ STREAM_PARENT *all; // a linked list of possible destinations
16
+ STREAM_PARENT *current; // the current destination from the above list
17
+} RRDHOST_STREAM_PARENTS;
18
+
19
+#include "stream-handshake.h"
20
+#include "rrdhost-status.h"
21
+
22
+void rrdhost_stream_parent_ssl_init(struct sender_state *s);
23
+
24
+int stream_info_to_json_v1(BUFFER *wb, const char *machine_guid);
25
+
26
+void rrdhost_stream_parents_reset(RRDHOST *host, STREAM_HANDSHAKE reason);
27
+
28
+void rrdhost_stream_parents_update_from_destination(RRDHOST *host);
29
+void rrdhost_stream_parents_free(struct rrdhost *host, bool having_write_lock);
30
+
31
+bool stream_parent_connect_to_one(
32
+ ND_SOCK *sender_sock,
33
+ struct rrdhost *host,
34
+ int default_port,
35
+ time_t timeout,
36
+ char *connected_to,
37
+ size_t connected_to_size,
38
+ STREAM_PARENT **destination);
39
+
40
+void rrdhost_stream_parents_to_json(BUFFER *wb, struct rrdhost_status *s);
41
+STREAM_HANDSHAKE stream_parent_get_disconnect_reason(STREAM_PARENT *d);
42
+void stream_parent_set_disconnect_reason(STREAM_PARENT *d, STREAM_HANDSHAKE reason, time_t since);
43
+void stream_parent_set_reconnect_delay(STREAM_PARENT *d, STREAM_HANDSHAKE reason, time_t secs);
44
+usec_t stream_parent_get_reconnection_ut(STREAM_PARENT *d);
45
+bool stream_parent_is_ssl(STREAM_PARENT *d);
46
+
47
+usec_t stream_parent_handshake_error_to_json(BUFFER *wb, struct rrdhost *host);
48
+
49
+void rrdhost_stream_parents_init(RRDHOST *host);
50
+
51
+#endif //NETDATA_STREAM_PARENTS_H
src/streaming/stream-path.c
+130
-75
@@ -1,9 +1,34 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
#include "stream-path.h"
4
-#include "rrdpush.h"
4
+#include "stream.h"
5
+#include "stream-receiver-internals.h"
6
+#include "stream-sender-internals.h"
7
#include "plugins.d/pluginsd_internals.h"
8
9
+typedef enum __attribute__((packed)) {
10
+ STREAM_PATH_FLAG_NONE = 0,
11
+ STREAM_PATH_FLAG_ACLK = (1 << 0),
12
+ STREAM_PATH_FLAG_HEALTH = (1 << 1),
13
+ STREAM_PATH_FLAG_ML = (1 << 2),
14
+ STREAM_PATH_FLAG_EPHEMERAL = (1 << 3),
15
+ STREAM_PATH_FLAG_VIRTUAL = (1 << 4),
16
+} STREAM_PATH_FLAGS;
17
+
18
+typedef struct stream_path {
19
+ STRING *hostname; // the hostname of the agent
20
+ ND_UUID host_id; // the machine guid of the agent
21
+ ND_UUID node_id; // the cloud node id of the agent
22
+ ND_UUID claim_id; // the cloud claim id of the agent
23
+ time_t since; // the timestamp of the last update
24
+ time_t first_time_t; // the oldest timestamp in the db
25
+ int16_t hops; // -1 = stale node, 0 = localhost, >0 the hops count
26
+ STREAM_PATH_FLAGS flags; // ACLK or NONE for the moment
27
+ STREAM_CAPABILITIES capabilities; // streaming connection capabilities
28
+ uint32_t start_time_ms; // median time in ms the agent needs to start
29
+ uint32_t shutdown_time_ms; // median time in ms the agent needs to shutdown
30
+} STREAM_PATH;
31
+
32
ENUM_STR_MAP_DEFINE(STREAM_PATH_FLAGS) = {
33
{ .id = STREAM_PATH_FLAG_ACLK, .name = "aclk" },
34
{ .id = STREAM_PATH_FLAG_HEALTH, .name = "health" },
@@ -17,7 +42,7 @@ ENUM_STR_MAP_DEFINE(STREAM_PATH_FLAGS) = {
42
43
BITMAP_STR_DEFINE_FUNCTIONS(STREAM_PATH_FLAGS, STREAM_PATH_FLAG_NONE, "");
44
20
-static void stream_path_clear(STREAM_PATH *p) {
45
+static void stream_path_cleanup(STREAM_PATH *p) {
46
string_freez(p->hostname);
47
p->hostname = NULL;
48
p->host_id = UUID_ZERO;
@@ -28,42 +53,45 @@ static void stream_path_clear(STREAM_PATH *p) {
53
p->first_time_t = 0;
54
p->capabilities = 0;
55
p->flags = STREAM_PATH_FLAG_NONE;
31
- p->start_time = 0;
32
- p->shutdown_time = 0;
56
+ p->start_time_ms = 0;
57
+ p->shutdown_time_ms = 0;
58
}
59
60
static void rrdhost_stream_path_clear_unsafe(RRDHOST *host, bool destroy) {
36
- for(size_t i = 0; i < host->rrdpush.path.used ; i++)
37
- stream_path_clear(&host->rrdpush.path.array[i]);
61
+ for(size_t i = 0; i < host->stream.path.used ; i++)
62
+ stream_path_cleanup(&host->stream.path.array[i]);
63
39
- host->rrdpush.path.used = 0;
64
+ host->stream.path.used = 0;
65
66
if(destroy) {
42
- freez(host->rrdpush.path.array);
43
- host->rrdpush.path.array = NULL;
44
- host->rrdpush.path.size = 0;
67
+ freez(host->stream.path.array);
68
+ host->stream.path.array = NULL;
69
+ host->stream.path.size = 0;
70
}
71
}
72
73
void rrdhost_stream_path_clear(RRDHOST *host, bool destroy) {
49
- spinlock_lock(&host->rrdpush.path.spinlock);
74
+ rw_spinlock_write_lock(&host->stream.path.spinlock);
75
rrdhost_stream_path_clear_unsafe(host, destroy);
51
- spinlock_unlock(&host->rrdpush.path.spinlock);
76
+ rw_spinlock_write_unlock(&host->stream.path.spinlock);
77
}
78
79
static void stream_path_to_json_object(BUFFER *wb, STREAM_PATH *p) {
80
buffer_json_add_array_item_object(wb);
56
- buffer_json_member_add_string(wb, "hostname", string2str(p->hostname));
57
- buffer_json_member_add_uuid(wb, "host_id", p->host_id.uuid);
58
- buffer_json_member_add_uuid(wb, "node_id", p->node_id.uuid);
59
- buffer_json_member_add_uuid(wb, "claim_id", p->claim_id.uuid);
60
- buffer_json_member_add_int64(wb, "hops", p->hops);
61
- buffer_json_member_add_uint64(wb, "since", p->since);
62
- buffer_json_member_add_uint64(wb, "first_time_t", p->first_time_t);
63
- buffer_json_member_add_uint64(wb, "start_time", p->start_time);
64
- buffer_json_member_add_uint64(wb, "shutdown_time", p->shutdown_time);
65
- stream_capabilities_to_json_array(wb, p->capabilities, "capabilities");
66
- STREAM_PATH_FLAGS_2json(wb, "flags", p->flags);
81
+ {
82
+ buffer_json_member_add_uint64(wb, "version", 1);
83
+ buffer_json_member_add_string(wb, "hostname", string2str(p->hostname));
84
+ buffer_json_member_add_uuid(wb, "host_id", p->host_id.uuid);
85
+ buffer_json_member_add_uuid(wb, "node_id", p->node_id.uuid);
86
+ buffer_json_member_add_uuid(wb, "claim_id", p->claim_id.uuid);
87
+ buffer_json_member_add_int64(wb, "hops", p->hops);
88
+ buffer_json_member_add_uint64(wb, "since", p->since);
89
+ buffer_json_member_add_uint64(wb, "first_time_t", p->first_time_t);
90
+ buffer_json_member_add_uint64(wb, "start_time", p->start_time_ms);
91
+ buffer_json_member_add_uint64(wb, "shutdown_time", p->shutdown_time_ms);
92
+ stream_capabilities_to_json_array(wb, p->capabilities, "capabilities");
93
+ STREAM_PATH_FLAGS_2json(wb, "flags", p->flags);
94
+ }
95
buffer_json_object_close(wb);
96
}
97
@@ -76,8 +104,8 @@ static STREAM_PATH rrdhost_stream_path_self(RRDHOST *host) {
104
p.host_id = localhost->host_id;
105
p.node_id = localhost->node_id;
106
p.claim_id = claim_id_get_uuid();
79
- p.start_time = get_agent_event_time_median(EVENT_AGENT_START_TIME) / USEC_PER_MS;
80
- p.shutdown_time = get_agent_event_time_median(EVENT_AGENT_SHUTDOWN_TIME) / USEC_PER_MS;
107
+ p.start_time_ms = get_agent_event_time_median(EVENT_AGENT_START_TIME) / USEC_PER_MS;
108
+ p.shutdown_time_ms = get_agent_event_time_median(EVENT_AGENT_SHUTDOWN_TIME) / USEC_PER_MS;
109
110
p.flags = STREAM_PATH_FLAG_NONE;
111
if(!UUIDiszero(p.claim_id))
@@ -89,13 +117,13 @@ static STREAM_PATH rrdhost_stream_path_self(RRDHOST *host) {
117
if(rrdhost_option_check(host, RRDHOST_OPTION_VIRTUAL_HOST))
118
p.flags |= STREAM_PATH_FLAG_VIRTUAL;
119
92
- if(host->health.health_enabled)
120
+ if(host->health.enabled)
121
p.flags |= STREAM_PATH_FLAG_HEALTH;
122
123
if(ml_enabled(host))
124
p.flags |= STREAM_PATH_FLAG_ML;
125
98
- spinlock_lock(&host->receiver_lock);
126
+ rrdhost_receiver_lock(host);
127
if(host->receiver) {
128
p.hops = (int16_t)host->receiver->hops;
129
p.since = host->receiver->connected_since_s;
@@ -104,7 +132,7 @@ static STREAM_PATH rrdhost_stream_path_self(RRDHOST *host) {
132
p.hops = (is_localhost) ? 0 : -1; // -1 for stale nodes
133
p.since = netdata_start_time;
134
}
107
- spinlock_unlock(&host->receiver_lock);
135
+ rrdhost_receiver_unlock(host);
136
137
// the following may get the receiver lock again!
138
p.capabilities = stream_our_capabilities(host, true);
@@ -114,34 +142,51 @@ static STREAM_PATH rrdhost_stream_path_self(RRDHOST *host) {
142
return p;
143
}
144
117
-STREAM_PATH rrdhost_stream_path_fetch(RRDHOST *host) {
118
- STREAM_PATH p = { 0 };
145
+uint64_t rrdhost_stream_path_total_reboot_time_ms(RRDHOST *host) {
146
+ uint64_t total_ms = 0;
147
120
- spinlock_lock(&host->rrdpush.path.spinlock);
121
- for (size_t i = 0; i < host->rrdpush.path.used; i++) {
122
- STREAM_PATH *tmp_path = &host->rrdpush.path.array[i];
148
+ rw_spinlock_read_lock(&host->stream.path.spinlock);
149
+ for (size_t i = 0; i < host->stream.path.used; i++) {
150
+ STREAM_PATH *tmp_path = &host->stream.path.array[i];
151
if(UUIDeq(host->host_id, tmp_path->host_id)) {
124
- p = *tmp_path;
152
+ total_ms = tmp_path->start_time_ms + tmp_path->shutdown_time_ms;
153
break;
154
}
155
}
128
- spinlock_unlock(&host->rrdpush.path.spinlock);
129
- return p;
156
+ rw_spinlock_read_unlock(&host->stream.path.spinlock);
157
+ return total_ms;
158
+}
159
+
160
+bool rrdhost_is_host_in_stream_path_before_us(struct rrdhost *host, ND_UUID remote_agent_host_id, int16_t our_hops) {
161
+ if(UUIDiszero(remote_agent_host_id)) return false;
162
+ if(UUIDeq(localhost->host_id, remote_agent_host_id)) return true;
163
+
164
+ bool rc = false;
165
+ rw_spinlock_read_lock(&host->stream.path.spinlock);
166
+ for (size_t i = 0; i < host->stream.path.used; i++) {
167
+ STREAM_PATH *p = &host->stream.path.array[i];
168
+ if(UUIDeq(remote_agent_host_id, p->host_id) && p->hops < our_hops) {
169
+ rc = true;
170
+ break;
171
+ }
172
+ }
173
+ rw_spinlock_read_unlock(&host->stream.path.spinlock);
174
+ return rc;
175
}
176
177
void rrdhost_stream_path_to_json(BUFFER *wb, struct rrdhost *host, const char *key, bool add_version) {
178
if(add_version)
179
buffer_json_member_add_uint64(wb, "version", 1);
180
136
- spinlock_lock(&host->rrdpush.path.spinlock);
181
+ STREAM_PATH tmp = rrdhost_stream_path_self(host);
182
+
183
+ rw_spinlock_read_lock(&host->stream.path.spinlock);
184
buffer_json_member_add_array(wb, key);
185
{
186
{
140
- STREAM_PATH tmp = rrdhost_stream_path_self(host);
141
-
187
bool found_self = false;
143
- for (size_t i = 0; i < host->rrdpush.path.used; i++) {
144
- STREAM_PATH *p = &host->rrdpush.path.array[i];
188
+ for (size_t i = 0; i < host->stream.path.used; i++) {
189
+ STREAM_PATH *p = &host->stream.path.array[i];
190
if(UUIDeq(localhost->host_id, p->host_id)) {
191
// this is us, use the current data
192
p = &tmp;
@@ -155,12 +200,12 @@ void rrdhost_stream_path_to_json(BUFFER *wb, struct rrdhost *host, const char *k
200
// append us.
201
stream_path_to_json_object(wb, &tmp);
202
}
158
-
159
- stream_path_clear(&tmp);
203
}
204
}
205
buffer_json_array_close(wb); // key
163
- spinlock_unlock(&host->rrdpush.path.spinlock);
206
+ rw_spinlock_read_unlock(&host->stream.path.spinlock);
207
+
208
+ stream_path_cleanup(&tmp);
209
}
210
211
static BUFFER *stream_path_payload(RRDHOST *host) {
@@ -173,13 +218,15 @@ static BUFFER *stream_path_payload(RRDHOST *host) {
218
219
void stream_path_send_to_parent(RRDHOST *host) {
220
struct sender_state *s = host->sender;
176
- if(!s || !stream_has_capability(s, STREAM_CAP_PATHS)) return;
221
+ if(!s ||
222
+ !stream_has_capability(s, STREAM_CAP_PATHS) ||
223
+ !rrdhost_can_stream_metadata_to_parent(host))
224
+ return;
225
226
CLEAN_BUFFER *payload = stream_path_payload(host);
179
-
180
- BUFFER *wb = sender_start(s);
181
- buffer_sprintf(wb, PLUGINSD_KEYWORD_JSON " " PLUGINSD_KEYWORD_STREAM_PATH "\n%s\n" PLUGINSD_KEYWORD_JSON_END "\n", buffer_tostring(payload));
182
- sender_commit(s, wb, STREAM_TRAFFIC_TYPE_METADATA);
227
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
228
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_JSON " " PLUGINSD_KEYWORD_JSON_CMD_STREAM_PATH "\n%s\n" PLUGINSD_KEYWORD_JSON_END "\n", buffer_tostring(payload));
229
+ sender_commit_clean_buffer(s, wb, STREAM_TRAFFIC_TYPE_METADATA);
230
}
231
232
void stream_path_send_to_child(RRDHOST *host) {
@@ -188,14 +235,15 @@ void stream_path_send_to_child(RRDHOST *host) {
235
236
CLEAN_BUFFER *payload = stream_path_payload(host);
237
191
- spinlock_lock(&host->receiver_lock);
192
- if(host->receiver && stream_has_capability(host->receiver, STREAM_CAP_PATHS)) {
238
+ rrdhost_receiver_lock(host);
239
+ if(stream_has_capability(host->receiver, STREAM_CAP_PATHS) &&
240
+ !rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_RECEIVER_DISCONNECTED)) {
241
242
CLEAN_BUFFER *wb = buffer_create(0, NULL);
195
- buffer_sprintf(wb, PLUGINSD_KEYWORD_JSON " " PLUGINSD_KEYWORD_STREAM_PATH "\n%s\n" PLUGINSD_KEYWORD_JSON_END "\n", buffer_tostring(payload));
196
- send_to_plugin(buffer_tostring(wb), __atomic_load_n(&host->receiver->parser, __ATOMIC_RELAXED));
243
+ buffer_sprintf(wb, PLUGINSD_KEYWORD_JSON " " PLUGINSD_KEYWORD_JSON_CMD_STREAM_PATH "\n%s\n" PLUGINSD_KEYWORD_JSON_END "\n", buffer_tostring(payload));
244
+ send_to_plugin(buffer_tostring(wb), __atomic_load_n(&host->receiver->thread.parser, __ATOMIC_RELAXED));
245
}
198
- spinlock_unlock(&host->receiver_lock);
246
+ rrdhost_receiver_unlock(host);
247
}
248
249
void stream_path_child_disconnected(RRDHOST *host) {
@@ -203,17 +251,17 @@ void stream_path_child_disconnected(RRDHOST *host) {
251
}
252
253
void stream_path_parent_disconnected(RRDHOST *host) {
206
- spinlock_lock(&host->rrdpush.path.spinlock);
254
+ rw_spinlock_write_lock(&host->stream.path.spinlock);
255
256
size_t cleared = 0;
209
- size_t used = host->rrdpush.path.used;
257
+ size_t used = host->stream.path.used;
258
for (size_t i = 0; i < used; i++) {
211
- STREAM_PATH *p = &host->rrdpush.path.array[i];
259
+ STREAM_PATH *p = &host->stream.path.array[i];
260
if(UUIDeq(localhost->host_id, p->host_id)) {
213
- host->rrdpush.path.used = i + 1;
261
+ host->stream.path.used = i + 1;
262
263
for(size_t j = i + 1; j < used ;j++) {
216
- stream_path_clear(&host->rrdpush.path.array[j]);
264
+ stream_path_cleanup(&host->stream.path.array[j]);
265
cleared++;
266
}
267
@@ -221,7 +269,7 @@ void stream_path_parent_disconnected(RRDHOST *host) {
269
}
270
}
271
224
- spinlock_unlock(&host->rrdpush.path.spinlock);
272
+ rw_spinlock_write_unlock(&host->stream.path.spinlock);
273
274
if(cleared)
275
stream_path_send_to_child(host);
@@ -243,6 +291,9 @@ void stream_path_node_id_updated(RRDHOST *host) {
291
292
293
static bool parse_single_path(json_object *jobj, const char *path, STREAM_PATH *p, BUFFER *error) {
294
+ uint32_t version = 0;
295
+ JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "version", version, error, false);
296
+
297
JSONC_PARSE_TXT2STRING_OR_ERROR_AND_RETURN(jobj, path, "hostname", p->hostname, error, true);
298
JSONC_PARSE_TXT2UUID_OR_ERROR_AND_RETURN(jobj, path, "host_id", p->host_id.uuid, error, true);
299
JSONC_PARSE_TXT2UUID_OR_ERROR_AND_RETURN(jobj, path, "node_id", p->node_id.uuid, error, true);
@@ -250,8 +301,8 @@ static bool parse_single_path(json_object *jobj, const char *path, STREAM_PATH *
301
JSONC_PARSE_INT64_OR_ERROR_AND_RETURN(jobj, path, "hops", p->hops, error, true);
302
JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "since", p->since, error, true);
303
JSONC_PARSE_UINT64_OR_ERROR_AND_RETURN(jobj, path, "first_time_t", p->first_time_t, error, true);
253
- JSONC_PARSE_INT64_OR_ERROR_AND_RETURN(jobj, path, "start_time", p->start_time, error, true);
254
- JSONC_PARSE_INT64_OR_ERROR_AND_RETURN(jobj, path, "shutdown_time", p->shutdown_time, error, true);
304
+ JSONC_PARSE_INT64_OR_ERROR_AND_RETURN(jobj, path, "start_time", p->start_time_ms, error, true);
305
+ JSONC_PARSE_INT64_OR_ERROR_AND_RETURN(jobj, path, "shutdown_time", p->shutdown_time_ms, error, true);
306
JSONC_PARSE_ARRAY_OF_TXT2BITMAP_OR_ERROR_AND_RETURN(jobj, path, "flags", STREAM_PATH_FLAGS_2id_one, p->flags, error, true);
307
JSONC_PARSE_ARRAY_OF_TXT2BITMAP_OR_ERROR_AND_RETURN(jobj, path, "capabilities", stream_capabilities_parse_one, p->capabilities, error, true);
308
@@ -284,10 +335,10 @@ static bool parse_single_path(json_object *jobj, const char *path, STREAM_PATH *
335
}
336
337
static XXH128_hash_t stream_path_hash_unsafe(RRDHOST *host) {
287
- if(!host->rrdpush.path.used)
338
+ if(!host->stream.path.used)
339
return (XXH128_hash_t){ 0 };
340
290
- return XXH3_128bits(host->rrdpush.path.array, sizeof(*host->rrdpush.path.array) * host->rrdpush.path.used);
341
+ return XXH3_128bits(host->stream.path.array, sizeof(*host->stream.path.array) * host->stream.path.used);
342
}
343
344
static int compare_by_hops(const void *a, const void *b) {
@@ -313,7 +364,7 @@ bool stream_path_set_from_json(RRDHOST *host, const char *json, bool from_parent
364
return false;
365
}
366
316
- spinlock_lock(&host->rrdpush.path.spinlock);
367
+ rw_spinlock_write_lock(&host->stream.path.spinlock);
368
XXH128_hash_t old_hash = stream_path_hash_unsafe(host);
369
rrdhost_stream_path_clear_unsafe(host, true);
370
@@ -323,8 +374,8 @@ bool stream_path_set_from_json(RRDHOST *host, const char *json, bool from_parent
374
if (json_object_object_get_ex(jobj, STREAM_PATH_JSON_MEMBER, &_jarray) &&
375
json_object_is_type(_jarray, json_type_array)) {
376
size_t items = json_object_array_length(_jarray);
326
- host->rrdpush.path.array = callocz(items, sizeof(*host->rrdpush.path.array));
327
- host->rrdpush.path.size = items;
377
+ host->stream.path.array = callocz(items, sizeof(*host->stream.path.array));
378
+ host->stream.path.size = items;
379
380
for (size_t i = 0; i < items; ++i) {
381
json_object *joption = json_object_array_get_idx(_jarray, i);
@@ -334,24 +385,24 @@ bool stream_path_set_from_json(RRDHOST *host, const char *json, bool from_parent
385
continue;
386
}
387
337
- if(!parse_single_path(joption, "", &host->rrdpush.path.array[host->rrdpush.path.used], error)) {
338
- stream_path_clear(&host->rrdpush.path.array[host->rrdpush.path.used]);
388
+ if(!parse_single_path(joption, "", &host->stream.path.array[host->stream.path.used], error)) {
389
+ stream_path_cleanup(&host->stream.path.array[host->stream.path.used]);
390
nd_log(NDLS_DAEMON, NDLP_ERR,
391
"STREAM PATH: Array item No %zu cannot be parsed: %s: %s", i, buffer_tostring(error), json);
392
}
393
else
343
- host->rrdpush.path.used++;
394
+ host->stream.path.used++;
395
}
396
}
397
347
- if(host->rrdpush.path.used > 1) {
398
+ if(host->stream.path.used > 1) {
399
// sorting is required in order to support stream_path_parent_disconnected()
349
- qsort(host->rrdpush.path.array, host->rrdpush.path.used,
350
- sizeof(*host->rrdpush.path.array), compare_by_hops);
400
+ qsort(host->stream.path.array, host->stream.path.used,
401
+ sizeof(*host->stream.path.array), compare_by_hops);
402
}
403
404
XXH128_hash_t new_hash = stream_path_hash_unsafe(host);
354
- spinlock_unlock(&host->rrdpush.path.spinlock);
405
+ rw_spinlock_write_unlock(&host->stream.path.spinlock);
406
407
if(!XXH128_isEqual(old_hash, new_hash)) {
408
if(!from_parent)
@@ -362,5 +413,9 @@ bool stream_path_set_from_json(RRDHOST *host, const char *json, bool from_parent
413
stream_path_send_to_child(host);
414
}
415
365
- return host->rrdpush.path.used > 0;
416
+ return host->stream.path.used > 0;
417
+}
418
+
419
+void rrdhost_stream_path_init(RRDHOST *host) {
420
+ rw_spinlock_init(&host->stream.path.spinlock);
421
}
src/streaming/stream-path.h
+8
-25
@@ -7,39 +7,19 @@
7
8
#define STREAM_PATH_JSON_MEMBER "streaming_path"
9
10
-typedef enum __attribute__((packed)) {
11
- STREAM_PATH_FLAG_NONE = 0,
12
- STREAM_PATH_FLAG_ACLK = (1 << 0),
13
- STREAM_PATH_FLAG_HEALTH = (1 << 1),
14
- STREAM_PATH_FLAG_ML = (1 << 2),
15
- STREAM_PATH_FLAG_EPHEMERAL = (1 << 3),
16
- STREAM_PATH_FLAG_VIRTUAL = (1 << 4),
17
-} STREAM_PATH_FLAGS;
18
-
19
-typedef struct stream_path {
20
- STRING *hostname; // the hostname of the agent
21
- ND_UUID host_id; // the machine guid of the agent
22
- ND_UUID node_id; // the cloud node id of the agent
23
- ND_UUID claim_id; // the cloud claim id of the agent
24
- time_t since; // the timestamp of the last update
25
- time_t first_time_t; // the oldest timestamp in the db
26
- int16_t hops; // -1 = stale node, 0 = localhost, >0 the hops count
27
- STREAM_PATH_FLAGS flags; // ACLK or NONE for the moment
28
- STREAM_CAPABILITIES capabilities; // streaming connection capabilities
29
- uint32_t start_time; // median time in ms the agent needs to start
30
- uint32_t shutdown_time; // median time in ms the agent needs to shutdown
31
-} STREAM_PATH;
10
+typedef struct stream_path STREAM_PATH;
11
12
typedef struct rrdhost_stream_path {
34
- SPINLOCK spinlock;
13
+ RW_SPINLOCK spinlock;
14
uint16_t size;
15
uint16_t used;
16
STREAM_PATH *array;
17
} RRDHOST_STREAM_PATH;
18
40
-
19
struct rrdhost;
20
21
+void rrdhost_stream_path_init(struct rrdhost *host);
22
+
23
void stream_path_send_to_parent(struct rrdhost *host);
24
void stream_path_send_to_child(struct rrdhost *host);
25
@@ -51,8 +31,11 @@ void stream_path_node_id_updated(struct rrdhost *host);
31
32
void stream_path_child_disconnected(struct rrdhost *host);
33
void stream_path_parent_disconnected(struct rrdhost *host);
54
-STREAM_PATH rrdhost_stream_path_fetch(struct rrdhost *host);
34
+
35
+uint64_t rrdhost_stream_path_total_reboot_time_ms(struct rrdhost *host);
36
37
bool stream_path_set_from_json(struct rrdhost *host, const char *json, bool from_parent);
38
39
+bool rrdhost_is_host_in_stream_path_before_us(struct rrdhost *host, ND_UUID remote_agent_host_id, int16_t our_hops);
40
+
41
#endif //NETDATA_STREAM_PATH_H
src/streaming/stream-receiver-api.c
new
+20
@@ -0,0 +1,20 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "stream-receiver-internals.h"
4
+
5
+char *stream_receiver_program_version_strdupz(RRDHOST *host) {
6
+ rrdhost_receiver_lock(host);
7
+ char *host_version = strdupz(
8
+ host->receiver && host->receiver->program_version ? host->receiver->program_version :
9
+ rrdhost_program_version(host));
10
+ rrdhost_receiver_unlock(host);
11
+
12
+ return host_version;
13
+}
14
+
15
+bool receiver_has_capability(RRDHOST *host, STREAM_CAPABILITIES caps) {
16
+ rrdhost_receiver_lock(host);
17
+ bool rc = stream_has_capability(host->receiver, caps);
18
+ rrdhost_receiver_unlock(host);
19
+ return rc;
20
+}
src/streaming/stream-receiver-connection.c
new
+633
@@ -0,0 +1,633 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "stream.h"
4
+#include "stream-thread.h"
5
+#include "stream-receiver-internals.h"
6
+#include "web/server/h2o/http_server.h"
7
+
8
+// --------------------------------------------------------------------------------------------------------------------
9
+
10
+void stream_receiver_log_status(struct receiver_state *rpt, const char *msg, const char *status, ND_LOG_FIELD_PRIORITY priority) {
11
+ // this function may be called BEFORE we spawn the receiver thread
12
+ // so, we need to add the fields again (it does not harm)
13
+ ND_LOG_STACK lgs[] = {
14
+ ND_LOG_FIELD_TXT(NDF_SRC_IP, rpt->client_ip),
15
+ ND_LOG_FIELD_TXT(NDF_SRC_PORT, rpt->client_port),
16
+ ND_LOG_FIELD_TXT(NDF_NIDL_NODE, (rpt->hostname && *rpt->hostname) ? rpt->hostname : ""),
17
+ ND_LOG_FIELD_TXT(NDF_RESPONSE_CODE, status),
18
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_from_child_msgid),
19
+ ND_LOG_FIELD_END(),
20
+ };
21
+ ND_LOG_STACK_PUSH(lgs);
22
+
23
+ nd_log(NDLS_ACCESS, priority, "api_key:'%s' machine_guid:'%s' msg:'%s'"
24
+ , (rpt->key && *rpt->key)? rpt->key : ""
25
+ , (rpt->machine_guid && *rpt->machine_guid) ? rpt->machine_guid : ""
26
+ , msg);
27
+
28
+ nd_log(NDLS_DAEMON, priority, "STREAM_RECEIVER for '%s': %s %s%s%s"
29
+ , (rpt->hostname && *rpt->hostname) ? rpt->hostname : ""
30
+ , msg
31
+ , rpt->exit.reason != STREAM_HANDSHAKE_NEVER?" (":""
32
+ , stream_handshake_error_to_string(rpt->exit.reason)
33
+ , rpt->exit.reason != STREAM_HANDSHAKE_NEVER?")":""
34
+ );
35
+}
36
+
37
+// --------------------------------------------------------------------------------------------------------------------
38
+
39
+void stream_receiver_free(struct receiver_state *rpt) {
40
+ nd_sock_close(&rpt->sock);
41
+ stream_decompressor_destroy(&rpt->thread.compressed.decompressor);
42
+
43
+ if(rpt->system_info)
44
+ rrdhost_system_info_free(rpt->system_info);
45
+
46
+ __atomic_sub_fetch(&netdata_buffers_statistics.rrdhost_receivers, sizeof(*rpt), __ATOMIC_RELAXED);
47
+
48
+ freez(rpt->key);
49
+ freez(rpt->hostname);
50
+ freez(rpt->registry_hostname);
51
+ freez(rpt->machine_guid);
52
+ freez(rpt->os);
53
+ freez(rpt->timezone);
54
+ freez(rpt->abbrev_timezone);
55
+ freez(rpt->client_ip);
56
+ freez(rpt->client_port);
57
+ freez(rpt->program_name);
58
+ freez(rpt->program_version);
59
+
60
+ string_freez(rpt->config.send.api_key);
61
+ string_freez(rpt->config.send.parents);
62
+ string_freez(rpt->config.send.charts_matching);
63
+
64
+ freez(rpt);
65
+}
66
+
67
+// --------------------------------------------------------------------------------------------------------------------
68
+
69
+static int stream_receiver_response_permission_denied(struct web_client *w) {
70
+ // we always respond with the same message and error code
71
+ // to prevent an attacker from gaining info about the error
72
+ buffer_flush(w->response.data);
73
+ buffer_strcat(w->response.data, START_STREAMING_ERROR_NOT_PERMITTED);
74
+ return HTTP_RESP_UNAUTHORIZED;
75
+}
76
+
77
+static int stream_receiver_response_too_busy_now(struct web_client *w) {
78
+ // we always respond with the same message and error code
79
+ // to prevent an attacker from gaining info about the error
80
+ buffer_flush(w->response.data);
81
+ buffer_strcat(w->response.data, START_STREAMING_ERROR_BUSY_TRY_LATER);
82
+ return HTTP_RESP_SERVICE_UNAVAILABLE;
83
+}
84
+
85
+static void stream_receiver_takeover_web_connection(struct web_client *w, struct receiver_state *rpt) {
86
+ rpt->sock.fd = w->ifd;
87
+ rpt->sock.ssl = w->ssl;
88
+
89
+ w->ssl = NETDATA_SSL_UNSET_CONNECTION;
90
+
91
+ WEB_CLIENT_IS_DEAD(w);
92
+
93
+ if(web_server_mode == WEB_SERVER_MODE_STATIC_THREADED) {
94
+ web_client_flag_set(w, WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET);
95
+ }
96
+ else {
97
+ if(w->ifd == w->ofd)
98
+ w->ifd = w->ofd = -1;
99
+ else
100
+ w->ifd = -1;
101
+ }
102
+
103
+ buffer_flush(w->response.data);
104
+}
105
+
106
+static void stream_send_error_on_taken_over_connection(struct receiver_state *rpt, const char *msg) {
107
+ nd_sock_send_timeout(&rpt->sock, (char *)msg, strlen(msg), 0, 5);
108
+}
109
+
110
+static bool stream_receiver_send_first_response(struct receiver_state *rpt) {
111
+ // find the host for this receiver
112
+ {
113
+ // this will also update the host with our system_info
114
+ RRDHOST *host = rrdhost_find_or_create(
115
+ rpt->hostname,
116
+ rpt->registry_hostname,
117
+ rpt->machine_guid,
118
+ rpt->os,
119
+ rpt->timezone,
120
+ rpt->abbrev_timezone,
121
+ rpt->utc_offset,
122
+ rpt->program_name,
123
+ rpt->program_version,
124
+ rpt->config.update_every,
125
+ rpt->config.history,
126
+ rpt->config.mode,
127
+ rpt->config.health.enabled != CONFIG_BOOLEAN_NO,
128
+ rpt->config.send.enabled && rpt->config.send.parents && rpt->config.send.api_key,
129
+ rpt->config.send.parents,
130
+ rpt->config.send.api_key,
131
+ rpt->config.send.charts_matching,
132
+ rpt->config.replication.enabled,
133
+ rpt->config.replication.period,
134
+ rpt->config.replication.step,
135
+ rpt->system_info,
136
+ 0);
137
+
138
+ if(!host) {
139
+ stream_receiver_log_status(
140
+ rpt,
141
+ "failed to find/create host structure, rejecting connection",
142
+ STREAM_STATUS_INTERNAL_SERVER_ERROR,
143
+ NDLP_ERR);
144
+
145
+ stream_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_INTERNAL_ERROR);
146
+ return false;
147
+ }
148
+
149
+ if (unlikely(rrdhost_flag_check(host, RRDHOST_FLAG_PENDING_CONTEXT_LOAD))) {
150
+ stream_receiver_log_status(
151
+ rpt, "host is initializing, retry later", STREAM_STATUS_INITIALIZATION_IN_PROGRESS, NDLP_NOTICE);
152
+
153
+ stream_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_INITIALIZATION);
154
+ return false;
155
+ }
156
+
157
+ // system_info has been consumed by the host structure
158
+ rpt->system_info = NULL;
159
+
160
+ if(!rrdhost_set_receiver(host, rpt)) {
161
+ stream_receiver_log_status(
162
+ rpt, "host is already served by another receiver", STREAM_STATUS_DUPLICATE_RECEIVER, NDLP_INFO);
163
+
164
+ stream_send_error_on_taken_over_connection(rpt, START_STREAMING_ERROR_ALREADY_STREAMING);
165
+ return false;
166
+ }
167
+ }
168
+
169
+#ifdef NETDATA_INTERNAL_CHECKS
170
+ netdata_log_info("STREAM '%s' [receive from [%s]:%s]: "
171
+ "client willing to stream metrics for host '%s' with machine_guid '%s': "
172
+ "update every = %d, history = %d, memory mode = %s, health %s,%s"
173
+ , rpt->hostname
174
+ , rpt->client_ip
175
+ , rpt->client_port
176
+ , rrdhost_hostname(rpt->host)
177
+ , rpt->host->machine_guid
178
+ , rpt->host->rrd_update_every
179
+ , rpt->host->rrd_history_entries
180
+ , rrd_memory_mode_name(rpt->host->rrd_memory_mode)
181
+ , (rpt->config.health.enabled == CONFIG_BOOLEAN_NO)?"disabled":((rpt->config.health.enabled == CONFIG_BOOLEAN_YES)?"enabled":"auto")
182
+ , (rpt->sock.ssl.conn != NULL) ? " SSL," : ""
183
+ );
184
+#endif // NETDATA_INTERNAL_CHECKS
185
+
186
+ stream_select_receiver_compression_algorithm(rpt);
187
+
188
+ {
189
+ // netdata_log_info("STREAM %s [receive from [%s]:%s]: initializing communication...", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
190
+ char initial_response[HTTP_HEADER_SIZE];
191
+ if (stream_has_capability(rpt, STREAM_CAP_VCAPS)) {
192
+ log_receiver_capabilities(rpt);
193
+ sprintf(initial_response, "%s%u", START_STREAMING_PROMPT_VN, rpt->capabilities);
194
+ }
195
+ else if (stream_has_capability(rpt, STREAM_CAP_VN)) {
196
+ log_receiver_capabilities(rpt);
197
+ sprintf(initial_response, "%s%d", START_STREAMING_PROMPT_VN, stream_capabilities_to_vn(rpt->capabilities));
198
+ }
199
+ else if (stream_has_capability(rpt, STREAM_CAP_V2)) {
200
+ log_receiver_capabilities(rpt);
201
+ sprintf(initial_response, "%s", START_STREAMING_PROMPT_V2);
202
+ }
203
+ else { // stream_has_capability(rpt, STREAM_CAP_V1)
204
+ log_receiver_capabilities(rpt);
205
+ sprintf(initial_response, "%s", START_STREAMING_PROMPT_V1);
206
+ }
207
+
208
+ // OUR FIRST RESPONSE IS READY!
209
+
210
+ // web server sockets are non-blocking - set them to blocking mode
211
+#ifdef ENABLE_H2O
212
+ unless_h2o_rrdpush(rpt)
213
+#endif
214
+ {
215
+ // remove the non-blocking flag from the socket
216
+ if(sock_delnonblock(rpt->sock.fd) < 0)
217
+ nd_log(NDLS_DAEMON, NDLP_ERR,
218
+ "STREAM '%s' [receive from [%s]:%s]: cannot remove the non-blocking flag from socket %d",
219
+ rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port, rpt->sock.fd);
220
+
221
+ struct timeval timeout;
222
+ timeout.tv_sec = 600;
223
+ timeout.tv_usec = 0;
224
+ if (unlikely(setsockopt(rpt->sock.fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof timeout) != 0))
225
+ nd_log(NDLS_DAEMON, NDLP_ERR,
226
+ "STREAM '%s' [receive from [%s]:%s]: cannot set timeout for socket %d",
227
+ rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port, rpt->sock.fd);
228
+ }
229
+
230
+ netdata_log_debug(D_STREAM, "Initial response to %s: %s", rpt->client_ip, initial_response);
231
+#ifdef ENABLE_H2O
232
+ if (is_h2o_rrdpush(rpt)) {
233
+ h2o_stream_write(rpt->h2o_ctx, initial_response, strlen(initial_response));
234
+ } else {
235
+#endif
236
+ ssize_t bytes_sent = nd_sock_send_timeout(&rpt->sock, initial_response, strlen(initial_response), 0, 60);
237
+
238
+ if(bytes_sent != (ssize_t)strlen(initial_response)) {
239
+ internal_error(true, "Cannot send response, got %zd bytes, expecting %zu bytes", bytes_sent, strlen(initial_response));
240
+ stream_receiver_log_status(
241
+ rpt, "cannot reply back, dropping connection", STREAM_STATUS_CANT_REPLY, NDLP_ERR);
242
+ rrdhost_clear_receiver(rpt);
243
+ return false;
244
+ }
245
+#ifdef ENABLE_H2O
246
+ }
247
+#endif
248
+ }
249
+
250
+ return true;
251
+}
252
+
253
+int stream_receiver_accept_connection(struct web_client *w, char *decoded_query_string, void *h2o_ctx __maybe_unused) {
254
+
255
+ if(!service_running(ABILITY_STREAMING_CONNECTIONS))
256
+ return stream_receiver_response_too_busy_now(w);
257
+
258
+ struct receiver_state *rpt = callocz(1, sizeof(*rpt));
259
+ rpt->connected_since_s = now_realtime_sec();
260
+ rpt->last_msg_t = now_monotonic_sec();
261
+ rpt->hops = 1;
262
+
263
+ rpt->capabilities = STREAM_CAP_INVALID;
264
+
265
+#ifdef ENABLE_H2O
266
+ rpt->h2o_ctx = h2o_ctx;
267
+#endif
268
+
269
+ __atomic_add_fetch(&netdata_buffers_statistics.rrdhost_receivers, sizeof(*rpt), __ATOMIC_RELAXED);
270
+ __atomic_add_fetch(&netdata_buffers_statistics.rrdhost_allocations_size, sizeof(struct rrdhost_system_info), __ATOMIC_RELAXED);
271
+
272
+ rpt->system_info = callocz(1, sizeof(struct rrdhost_system_info));
273
+ rpt->system_info->hops = rpt->hops;
274
+
275
+ nd_sock_init(&rpt->sock, netdata_ssl_web_server_ctx, false);
276
+ rpt->client_ip = strdupz(w->client_ip);
277
+ rpt->client_port = strdupz(w->client_port);
278
+
279
+ rpt->config.update_every = default_rrd_update_every;
280
+
281
+ // parse the parameters and fill rpt and rpt->system_info
282
+
283
+ while(decoded_query_string) {
284
+ char *value = strsep_skip_consecutive_separators(&decoded_query_string, "&");
285
+ if(!value || !*value) continue;
286
+
287
+ char *name = strsep_skip_consecutive_separators(&value, "=");
288
+ if(!name || !*name) continue;
289
+ if(!value || !*value) continue;
290
+
291
+ if(!strcmp(name, "key") && !rpt->key)
292
+ rpt->key = strdupz(value);
293
+
294
+ else if(!strcmp(name, "hostname") && !rpt->hostname)
295
+ rpt->hostname = strdupz(value);
296
+
297
+ else if(!strcmp(name, "registry_hostname") && !rpt->registry_hostname)
298
+ rpt->registry_hostname = strdupz(value);
299
+
300
+ else if(!strcmp(name, "machine_guid") && !rpt->machine_guid)
301
+ rpt->machine_guid = strdupz(value);
302
+
303
+ else if(!strcmp(name, "update_every"))
304
+ rpt->config.update_every = (int)strtoul(value, NULL, 0);
305
+
306
+ else if(!strcmp(name, "os") && !rpt->os)
307
+ rpt->os = strdupz(value);
308
+
309
+ else if(!strcmp(name, "timezone") && !rpt->timezone)
310
+ rpt->timezone = strdupz(value);
311
+
312
+ else if(!strcmp(name, "abbrev_timezone") && !rpt->abbrev_timezone)
313
+ rpt->abbrev_timezone = strdupz(value);
314
+
315
+ else if(!strcmp(name, "utc_offset"))
316
+ rpt->utc_offset = (int32_t)strtol(value, NULL, 0);
317
+
318
+ else if(!strcmp(name, "hops"))
319
+ rpt->hops = rpt->system_info->hops = (int16_t)strtol(value, NULL, 0);
320
+
321
+ else if(!strcmp(name, "ml_capable"))
322
+ rpt->system_info->ml_capable = strtoul(value, NULL, 0);
323
+
324
+ else if(!strcmp(name, "ml_enabled"))
325
+ rpt->system_info->ml_enabled = strtoul(value, NULL, 0);
326
+
327
+ else if(!strcmp(name, "mc_version"))
328
+ rpt->system_info->mc_version = strtoul(value, NULL, 0);
329
+
330
+ else if(!strcmp(name, "ver") && (rpt->capabilities & STREAM_CAP_INVALID))
331
+ rpt->capabilities = convert_stream_version_to_capabilities(strtoul(value, NULL, 0), NULL, false);
332
+
333
+ else {
334
+ // An old Netdata child does not have a compatible streaming protocol, map to something sane.
335
+ if (!strcmp(name, "NETDATA_SYSTEM_OS_NAME"))
336
+ name = "NETDATA_HOST_OS_NAME";
337
+
338
+ else if (!strcmp(name, "NETDATA_SYSTEM_OS_ID"))
339
+ name = "NETDATA_HOST_OS_ID";
340
+
341
+ else if (!strcmp(name, "NETDATA_SYSTEM_OS_ID_LIKE"))
342
+ name = "NETDATA_HOST_OS_ID_LIKE";
343
+
344
+ else if (!strcmp(name, "NETDATA_SYSTEM_OS_VERSION"))
345
+ name = "NETDATA_HOST_OS_VERSION";
346
+
347
+ else if (!strcmp(name, "NETDATA_SYSTEM_OS_VERSION_ID"))
348
+ name = "NETDATA_HOST_OS_VERSION_ID";
349
+
350
+ else if (!strcmp(name, "NETDATA_SYSTEM_OS_DETECTION"))
351
+ name = "NETDATA_HOST_OS_DETECTION";
352
+
353
+ else if(!strcmp(name, "NETDATA_PROTOCOL_VERSION") && (rpt->capabilities & STREAM_CAP_INVALID))
354
+ rpt->capabilities = convert_stream_version_to_capabilities(1, NULL, false);
355
+
356
+ if (unlikely(rrdhost_set_system_info_variable(rpt->system_info, name, value))) {
357
+ nd_log_daemon(NDLP_NOTICE, "STREAM '%s' [receive from [%s]:%s]: "
358
+ "request has parameter '%s' = '%s', which is not used."
359
+ , (rpt->hostname && *rpt->hostname) ? rpt->hostname : "-"
360
+ , rpt->client_ip, rpt->client_port
361
+ , name, value);
362
+ }
363
+ }
364
+ }
365
+
366
+ if (rpt->capabilities & STREAM_CAP_INVALID)
367
+ // no version is supplied, assume version 0;
368
+ rpt->capabilities = convert_stream_version_to_capabilities(0, NULL, false);
369
+
370
+ // find the program name and version
371
+ if(w->user_agent && w->user_agent[0]) {
372
+ char *t = strchr(w->user_agent, '/');
373
+ if(t && *t) {
374
+ *t = '\0';
375
+ t++;
376
+ }
377
+
378
+ rpt->program_name = strdupz(w->user_agent);
379
+ if(t && *t) rpt->program_version = strdupz(t);
380
+ }
381
+
382
+ // check if we should accept this connection
383
+
384
+ if(!rpt->key || !*rpt->key) {
385
+ stream_receiver_log_status(
386
+ rpt, "request without an API key, rejecting connection", STREAM_STATUS_NO_API_KEY, NDLP_WARNING);
387
+
388
+ stream_receiver_free(rpt);
389
+ return stream_receiver_response_permission_denied(w);
390
+ }
391
+
392
+ if(!rpt->hostname || !*rpt->hostname) {
393
+ stream_receiver_log_status(
394
+ rpt, "request without a hostname, rejecting connection", STREAM_STATUS_NO_HOSTNAME, NDLP_WARNING);
395
+
396
+ stream_receiver_free(rpt);
397
+ return stream_receiver_response_permission_denied(w);
398
+ }
399
+
400
+ if(!rpt->registry_hostname)
401
+ rpt->registry_hostname = strdupz(rpt->hostname);
402
+
403
+ if(!rpt->machine_guid || !*rpt->machine_guid) {
404
+ stream_receiver_log_status(
405
+ rpt, "request without a machine GUID, rejecting connection", STREAM_STATUS_NO_MACHINE_GUID, NDLP_WARNING);
406
+
407
+ stream_receiver_free(rpt);
408
+ return stream_receiver_response_permission_denied(w);
409
+ }
410
+
411
+ {
412
+ char buf[GUID_LEN + 1];
413
+
414
+ if (regenerate_guid(rpt->key, buf) == -1) {
415
+ stream_receiver_log_status(
416
+ rpt,
417
+ "API key is not a valid UUID (use the command uuidgen to generate one)",
418
+ STREAM_STATUS_INVALID_API_KEY,
419
+ NDLP_WARNING);
420
+
421
+ stream_receiver_free(rpt);
422
+ return stream_receiver_response_permission_denied(w);
423
+ }
424
+
425
+ if (regenerate_guid(rpt->machine_guid, buf) == -1) {
426
+ stream_receiver_log_status(
427
+ rpt, "machine GUID is not a valid UUID", STREAM_STATUS_INVALID_MACHINE_GUID, NDLP_WARNING);
428
+
429
+ stream_receiver_free(rpt);
430
+ return stream_receiver_response_permission_denied(w);
431
+ }
432
+ }
433
+
434
+ if(!stream_conf_is_key_type(rpt->key, "api")) {
435
+ stream_receiver_log_status(rpt, "API key is a machine GUID", STREAM_STATUS_INVALID_API_KEY, NDLP_WARNING);
436
+
437
+ stream_receiver_free(rpt);
438
+ return stream_receiver_response_permission_denied(w);
439
+ }
440
+
441
+ // the default for api keys is false, so that users
442
+ // have to enable them manually
443
+ if(!stream_conf_api_key_is_enabled(rpt->key, false)) {
444
+ stream_receiver_log_status(rpt, "API key is not enabled", STREAM_STATUS_API_KEY_DISABLED, NDLP_WARNING);
445
+
446
+ stream_receiver_free(rpt);
447
+ return stream_receiver_response_permission_denied(w);
448
+ }
449
+
450
+ if(!stream_conf_api_key_allows_client(rpt->key, w->client_ip)) {
451
+ stream_receiver_log_status(
452
+ rpt, "API key is not allowed from this IP", STREAM_STATUS_NOT_ALLOWED_IP, NDLP_WARNING);
453
+
454
+ stream_receiver_free(rpt);
455
+ return stream_receiver_response_permission_denied(w);
456
+ }
457
+
458
+ if (!stream_conf_is_key_type(rpt->machine_guid, "machine")) {
459
+ stream_receiver_log_status(
460
+ rpt, "machine GUID is an API key", STREAM_STATUS_INVALID_MACHINE_GUID, NDLP_WARNING);
461
+
462
+ stream_receiver_free(rpt);
463
+ return stream_receiver_response_permission_denied(w);
464
+ }
465
+
466
+ // the default for machine guids is true, so that users do not
467
+ // have to enable them manually
468
+ if(!stream_conf_api_key_is_enabled(rpt->machine_guid, true)) {
469
+ stream_receiver_log_status(
470
+ rpt, "machine GUID is not enabled", STREAM_STATUS_MACHINE_GUID_DISABLED, NDLP_WARNING);
471
+
472
+ stream_receiver_free(rpt);
473
+ return stream_receiver_response_permission_denied(w);
474
+ }
475
+
476
+ if(!stream_conf_api_key_allows_client(rpt->machine_guid, w->client_ip)) {
477
+ stream_receiver_log_status(
478
+ rpt, "machine GUID is not allowed from this IP", STREAM_STATUS_NOT_ALLOWED_IP, NDLP_WARNING);
479
+
480
+ stream_receiver_free(rpt);
481
+ return stream_receiver_response_permission_denied(w);
482
+ }
483
+
484
+ if (strcmp(rpt->machine_guid, localhost->machine_guid) == 0) {
485
+ stream_receiver_takeover_web_connection(w, rpt);
486
+
487
+ stream_receiver_log_status(rpt, "machine GUID is my own", STREAM_STATUS_LOCALHOST, NDLP_DEBUG);
488
+
489
+ char initial_response[HTTP_HEADER_SIZE + 1];
490
+ snprintfz(initial_response, HTTP_HEADER_SIZE, "%s", START_STREAMING_ERROR_SAME_LOCALHOST);
491
+
492
+ if(nd_sock_send_timeout(&rpt->sock, initial_response, strlen(initial_response), 0, 60) !=
493
+ (ssize_t)strlen(initial_response)) {
494
+
495
+ nd_log_daemon(NDLP_ERR, "STREAM '%s' [receive from [%s]:%s]: "
496
+ "failed to reply."
497
+ , rpt->hostname
498
+ , rpt->client_ip, rpt->client_port
499
+ );
500
+ }
501
+
502
+ stream_receiver_free(rpt);
503
+ return HTTP_RESP_OK;
504
+ }
505
+
506
+ if(unlikely(web_client_streaming_rate_t > 0)) {
507
+ static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER;
508
+ static time_t last_stream_accepted_t = 0;
509
+
510
+ time_t now = now_realtime_sec();
511
+ spinlock_lock(&spinlock);
512
+
513
+ if(unlikely(last_stream_accepted_t == 0))
514
+ last_stream_accepted_t = now;
515
+
516
+ if(now - last_stream_accepted_t < web_client_streaming_rate_t) {
517
+ spinlock_unlock(&spinlock);
518
+
519
+ char msg[100 + 1];
520
+ snprintfz(msg, sizeof(msg) - 1,
521
+ "rate limit, will accept new connection in %ld secs",
522
+ (long)(web_client_streaming_rate_t - (now - last_stream_accepted_t)));
523
+
524
+ stream_receiver_log_status(rpt, msg, STREAM_STATUS_RATE_LIMIT, NDLP_NOTICE);
525
+
526
+ stream_receiver_free(rpt);
527
+ return stream_receiver_response_too_busy_now(w);
528
+ }
529
+
530
+ last_stream_accepted_t = now;
531
+ spinlock_unlock(&spinlock);
532
+ }
533
+
534
+ /*
535
+ * Quick path for rejecting multiple connections. The lock taken is fine-grained - it only protects the receiver
536
+ * pointer within the host (if a host exists). This protects against multiple concurrent web requests hitting
537
+ * separate threads within the web-server and landing here. The lock guards the thread-shutdown sequence that
538
+ * detaches the receiver from the host. If the host is being created (first time-access) then we also use the
539
+ * lock to prevent race-hazard (two threads try to create the host concurrently, one wins and the other does a
540
+ * lookup to the now-attached structure).
541
+ */
542
+
543
+ {
544
+ time_t age = 0;
545
+ bool receiver_stale = false;
546
+ bool receiver_working = false;
547
+
548
+ rrd_rdlock();
549
+ RRDHOST *host = rrdhost_find_by_guid(rpt->machine_guid);
550
+ if (unlikely(host && rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))) /* Ignore archived hosts. */
551
+ host = NULL;
552
+
553
+ if (host) {
554
+ rrdhost_receiver_lock(host);
555
+ if (host->receiver) {
556
+ age = now_monotonic_sec() - host->receiver->last_msg_t;
557
+
558
+ if (age < 30)
559
+ receiver_working = true;
560
+ else
561
+ receiver_stale = true;
562
+ }
563
+ rrdhost_receiver_unlock(host);
564
+ }
565
+ rrd_rdunlock();
566
+
567
+ if (receiver_stale &&
568
+ stream_receiver_signal_to_stop_and_wait(host, STREAM_HANDSHAKE_DISCONNECT_STALE_RECEIVER)) {
569
+ // we stopped the receiver
570
+ // we can proceed with this connection
571
+ receiver_stale = false;
572
+
573
+ nd_log_daemon(NDLP_NOTICE, "STREAM '%s' [receive from [%s]:%s]: "
574
+ "stopped previous stale receiver to accept this one."
575
+ , rpt->hostname
576
+ , rpt->client_ip, rpt->client_port
577
+ );
578
+ }
579
+
580
+ if (receiver_working || receiver_stale) {
581
+ // another receiver is already connected
582
+ // try again later
583
+
584
+ char msg[200 + 1];
585
+ snprintfz(msg, sizeof(msg) - 1,
586
+ "multiple connections for same host, "
587
+ "old connection was last used %ld secs ago%s",
588
+ age, receiver_stale ? " (signaled old receiver to stop)" : " (new connection not accepted)");
589
+
590
+ stream_receiver_log_status(rpt, msg, STREAM_STATUS_ALREADY_CONNECTED, NDLP_DEBUG);
591
+
592
+ // Have not set WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET - caller should clean up
593
+ buffer_flush(w->response.data);
594
+ buffer_strcat(w->response.data, START_STREAMING_ERROR_ALREADY_STREAMING);
595
+ stream_receiver_free(rpt);
596
+ return HTTP_RESP_CONFLICT;
597
+ }
598
+ }
599
+
600
+ stream_receiver_takeover_web_connection(w, rpt);
601
+
602
+ // after this point, our response code is irrelevant
603
+ // the socket is now ours...
604
+
605
+ // read the configuration for this receiver
606
+ stream_conf_receiver_config(rpt, &rpt->config, rpt->key, rpt->machine_guid);
607
+
608
+ if(stream_receiver_send_first_response(rpt)) {
609
+ // we are the receiver of the node
610
+
611
+ stream_receiver_log_status(rpt, "connected and ready to receive data", STREAM_STATUS_CONNECTED, NDLP_INFO);
612
+
613
+ // in case we have cloud connection we inform cloud a new child connected
614
+ schedule_node_state_update(rpt->host, 300);
615
+ rrdhost_set_is_parent_label();
616
+
617
+ if (rpt->config.ephemeral)
618
+ rrdhost_option_set(rpt->host, RRDHOST_OPTION_EPHEMERAL_HOST);
619
+
620
+ // let it reconnect to parents asap
621
+ rrdhost_stream_parents_reset(rpt->host, STREAM_HANDSHAKE_PREPARING);
622
+
623
+ // add it to a stream thread queue
624
+ stream_receiver_add_to_queue(rpt);
625
+ }
626
+ else {
627
+ // we are not the receiver of the node
628
+ // the child has been notified (or we couldn't send a message to it)
629
+ stream_receiver_free(rpt);
630
+ }
631
+
632
+ return HTTP_RESP_OK;
633
+}
src/streaming/stream-receiver-internals.h
new
+84
@@ -0,0 +1,84 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_STREAM_RECEIVER_INTERNALS_H
4
+#define NETDATA_STREAM_RECEIVER_INTERNALS_H
5
+
6
+#include "stream.h"
7
+#include "stream-thread.h"
8
+#include "stream-conf.h"
9
+#include "database/rrd.h"
10
+#include "plugins.d/plugins_d.h"
11
+
12
+struct parser;
13
+
14
+struct receiver_state {
15
+ RRDHOST *host;
16
+ ND_SOCK sock;
17
+ int16_t hops;
18
+ int32_t utc_offset;
19
+ STREAM_CAPABILITIES capabilities;
20
+ char *key;
21
+ char *hostname;
22
+ char *registry_hostname;
23
+ char *machine_guid;
24
+ char *os;
25
+ char *timezone; // Unused?
26
+ char *abbrev_timezone;
27
+ char *client_ip; // Duplicated in pluginsd
28
+ char *client_port; // Duplicated in pluginsd
29
+ char *program_name; // Duplicated in pluginsd
30
+ char *program_version;
31
+ struct rrdhost_system_info *system_info;
32
+ time_t last_msg_t;
33
+ time_t connected_since_s;
34
+
35
+ struct buffered_reader reader;
36
+
37
+ struct {
38
+ // The parser pointer is safe to read and use, only when having the host receiver lock.
39
+ // Without this lock, the data pointed by the pointer may vanish randomly.
40
+ // Also, since the receiver sets it when it starts, it should be read with
41
+ // an atomic read.
42
+ struct parser *parser;
43
+ struct plugind cd;
44
+ BUFFER *buffer;
45
+
46
+ struct {
47
+ bool enabled;
48
+ size_t start;
49
+ size_t used;
50
+ char buf[COMPRESSION_MAX_CHUNK * 2];
51
+ struct decompressor_state decompressor;
52
+ } compressed;
53
+
54
+ struct pollfd_meta meta;
55
+ } thread;
56
+
57
+ struct {
58
+ bool shutdown; // signal the streaming parser to exit
59
+ STREAM_HANDSHAKE reason;
60
+ } exit;
61
+
62
+ struct stream_receiver_config config;
63
+
64
+ time_t replication_first_time_t;
65
+
66
+#ifdef ENABLE_H2O
67
+ void *h2o_ctx;
68
+#endif
69
+};
70
+
71
+#ifdef ENABLE_H2O
72
+#define is_h2o_rrdpush(x) ((x)->h2o_ctx != NULL)
73
+#define unless_h2o_rrdpush(x) if(!is_h2o_rrdpush(x))
74
+#endif
75
+
76
+bool rrdhost_set_receiver(RRDHOST *host, struct receiver_state *rpt);
77
+void rrdhost_clear_receiver(struct receiver_state *rpt);
78
+void stream_receiver_log_status(struct receiver_state *rpt, const char *msg, const char *status, ND_LOG_FIELD_PRIORITY priority);
79
+
80
+void stream_receiver_free(struct receiver_state *rpt);
81
+bool stream_receiver_signal_to_stop_and_wait(RRDHOST *host, STREAM_HANDSHAKE reason);
82
+
83
+
84
+#endif //NETDATA_STREAM_RECEIVER_INTERNALS_H
src/streaming/stream-receiver.c
new
+698
@@ -0,0 +1,698 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "stream.h"
4
+#include "stream-thread.h"
5
+#include "stream-receiver-internals.h"
6
+#include "web/server/h2o/http_server.h"
7
+
8
+// When a child disconnects this is the maximum we will wait
9
+// before we update the cloud that the child is offline
10
+#define MAX_CHILD_DISC_DELAY (30000)
11
+#define MAX_CHILD_DISC_TOLERANCE (125 / 100)
12
+
13
+static uint32_t streaming_connected_receivers = 0;
14
+
15
+bool plugin_is_enabled(struct plugind *cd);
16
+
17
+uint32_t stream_receivers_currently_connected(void) {
18
+ return __atomic_load_n(&streaming_connected_receivers, __ATOMIC_RELAXED);
19
+}
20
+
21
+static void streaming_receiver_connected(void) {
22
+ __atomic_add_fetch(&streaming_connected_receivers, 1, __ATOMIC_RELAXED);
23
+}
24
+
25
+static void streaming_receiver_disconnected(void) {
26
+ __atomic_sub_fetch(&streaming_connected_receivers, 1, __ATOMIC_RELAXED);
27
+}
28
+
29
+// --------------------------------------------------------------------------------------------------------------------
30
+
31
+static inline ssize_t read_stream(struct receiver_state *r, char* buffer, size_t size) {
32
+ if(unlikely(!size)) {
33
+ internal_error(true, "%s() asked to read zero bytes", __FUNCTION__);
34
+ return -2;
35
+ }
36
+
37
+#ifdef ENABLE_H2O
38
+ if (is_h2o_rrdpush(r)) {
39
+ if(nd_thread_signaled_to_cancel())
40
+ return -3;
41
+
42
+ return (ssize_t)h2o_stream_read(r->h2o_ctx, buffer, size);
43
+ }
44
+#endif
45
+
46
+ ssize_t bytes_read = nd_sock_read(&r->sock, buffer, size, 0);
47
+ if(bytes_read <= 0) {
48
+ if (bytes_read == 0)
49
+ netdata_log_error("STREAM: %s(): EOF while reading data from socket!", __FUNCTION__);
50
+ else {
51
+ netdata_log_error("STREAM: %s() failed to read from socket!", __FUNCTION__);
52
+ bytes_read = -1;
53
+ }
54
+ }
55
+
56
+ return bytes_read;
57
+}
58
+
59
+static inline STREAM_HANDSHAKE read_stream_error_to_reason(ssize_t code) {
60
+ if(code > 0)
61
+ return 0;
62
+
63
+ switch(code) {
64
+ case 0:
65
+ // EOF
66
+ return STREAM_HANDSHAKE_DISCONNECT_SOCKET_EOF;
67
+
68
+ case -1:
69
+ // failed to read
70
+ return STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_FAILED;
71
+
72
+ case -2:
73
+ // asked to read zero bytes
74
+ return STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_RECEIVER_READ_BUFFER;
75
+
76
+ case -3:
77
+ // the thread is cancelled
78
+ return STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN;
79
+
80
+ default:
81
+ // anything else
82
+ return STREAM_HANDSHAKE_DISCONNECT_UNKNOWN_SOCKET_READ_ERROR;
83
+ }
84
+}
85
+
86
+// --------------------------------------------------------------------------------------------------------------------
87
+
88
+static inline ssize_t receiver_read_uncompressed(struct receiver_state *r) {
89
+ internal_fatal(r->reader.read_buffer[r->reader.read_len] != '\0',
90
+ "%s: read_buffer does not start with zero #2", __FUNCTION__ );
91
+
92
+ ssize_t bytes = read_stream(r, r->reader.read_buffer + r->reader.read_len, sizeof(r->reader.read_buffer) - r->reader.read_len - 1);
93
+ if(bytes > 0) {
94
+ worker_set_metric(WORKER_RECEIVER_JOB_BYTES_READ, (NETDATA_DOUBLE)bytes);
95
+ worker_set_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED, (NETDATA_DOUBLE)bytes);
96
+
97
+ r->reader.read_len += bytes;
98
+ r->reader.read_buffer[r->reader.read_len] = '\0';
99
+ }
100
+
101
+ return bytes;
102
+}
103
+
104
+typedef enum {
105
+ DECOMPRESS_NEED_MORE_DATA,
106
+ DECOMPRESS_FAILED,
107
+ DECOMPRESS_OK,
108
+} decompressor_status_t;
109
+
110
+static inline void receiver_move_compressed(struct receiver_state *r) {
111
+ size_t remaining = r->thread.compressed.used - r->thread.compressed.start;
112
+ if(remaining > 0) {
113
+ memmove(r->thread.compressed.buf, r->thread.compressed.buf + r->thread.compressed.start, remaining);
114
+ r->thread.compressed.start = 0;
115
+ r->thread.compressed.used = remaining;
116
+ }
117
+ else {
118
+ r->thread.compressed.start = 0;
119
+ r->thread.compressed.used = 0;
120
+ }
121
+}
122
+
123
+static inline decompressor_status_t receiver_feed_decompressor(struct receiver_state *r) {
124
+ char *buf = r->thread.compressed.buf;
125
+ size_t start = r->thread.compressed.start;
126
+ size_t signature_size = r->thread.compressed.decompressor.signature_size;
127
+ size_t used = r->thread.compressed.used;
128
+
129
+ if(start + signature_size > used) {
130
+ // incomplete header, we need to wait for more data
131
+ receiver_move_compressed(r);
132
+ return DECOMPRESS_NEED_MORE_DATA;
133
+ }
134
+
135
+ size_t compressed_message_size =
136
+ stream_decompressor_start(&r->thread.compressed.decompressor, buf + start, signature_size);
137
+
138
+ if (unlikely(!compressed_message_size)) {
139
+ nd_log(NDLS_DAEMON, NDLP_ERR, "multiplexed uncompressed data in compressed stream!");
140
+ return DECOMPRESS_FAILED;
141
+ }
142
+
143
+ if(unlikely(compressed_message_size > COMPRESSION_MAX_MSG_SIZE)) {
144
+ nd_log(NDLS_DAEMON, NDLP_ERR,
145
+ "received a compressed message of %zu bytes, which is bigger than the max compressed message "
146
+ "size supported of %zu. Ignoring message.",
147
+ compressed_message_size, (size_t)COMPRESSION_MAX_MSG_SIZE);
148
+ return DECOMPRESS_FAILED;
149
+ }
150
+
151
+ if(start + signature_size + compressed_message_size > used) {
152
+ // incomplete compressed message, we need to wait for more data
153
+ receiver_move_compressed(r);
154
+ return DECOMPRESS_NEED_MORE_DATA;
155
+ }
156
+
157
+ size_t bytes_to_parse =
158
+ stream_decompress(&r->thread.compressed.decompressor, buf + start + signature_size, compressed_message_size);
159
+
160
+ if (unlikely(!bytes_to_parse)) {
161
+ nd_log(NDLS_DAEMON, NDLP_ERR, "no bytes to parse.");
162
+ return DECOMPRESS_FAILED;
163
+ }
164
+
165
+ worker_set_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED, (NETDATA_DOUBLE)bytes_to_parse);
166
+
167
+ // move the header to the next message
168
+ r->thread.compressed.start += signature_size + compressed_message_size;
169
+
170
+ return DECOMPRESS_OK;
171
+}
172
+
173
+static inline decompressor_status_t receiver_get_decompressed(struct receiver_state *r) {
174
+ if (unlikely(!stream_decompressed_bytes_in_buffer(&r->thread.compressed.decompressor)))
175
+ return DECOMPRESS_NEED_MORE_DATA;
176
+
177
+ size_t available = sizeof(r->reader.read_buffer) - r->reader.read_len - 1;
178
+ if (likely(available)) {
179
+ size_t len = stream_decompressor_get(
180
+ &r->thread.compressed.decompressor, r->reader.read_buffer + r->reader.read_len, available);
181
+ if (unlikely(!len)) {
182
+ internal_error(true, "decompressor returned zero length #1");
183
+ return DECOMPRESS_FAILED;
184
+ }
185
+
186
+ r->reader.read_len += (int)len;
187
+ r->reader.read_buffer[r->reader.read_len] = '\0';
188
+ }
189
+ else {
190
+ internal_fatal(true, "The line to read is too big! Already have %zd bytes in read_buffer.", r->reader.read_len);
191
+ return DECOMPRESS_FAILED;
192
+ }
193
+
194
+ return DECOMPRESS_OK;
195
+}
196
+
197
+static inline ssize_t receiver_read_compressed(struct receiver_state *r) {
198
+
199
+ internal_fatal(r->reader.read_buffer[r->reader.read_len] != '\0',
200
+ "%s: read_buffer does not start with zero #2", __FUNCTION__ );
201
+
202
+ ssize_t bytes_read = read_stream(r, r->thread.compressed.buf + r->thread.compressed.used,
203
+ sizeof(r->thread.compressed.buf) - r->thread.compressed.used);
204
+
205
+ if(bytes_read > 0) {
206
+ r->thread.compressed.used += bytes_read;
207
+ worker_set_metric(WORKER_RECEIVER_JOB_BYTES_READ, (NETDATA_DOUBLE)bytes_read);
208
+ }
209
+
210
+ return bytes_read;
211
+}
212
+
213
+// --------------------------------------------------------------------------------------------------------------------
214
+
215
+static void receiver_set_exit_reason(struct receiver_state *rpt, STREAM_HANDSHAKE reason, bool force) {
216
+ if(force || !rpt->exit.reason)
217
+ rpt->exit.reason = reason;
218
+}
219
+
220
+static inline bool receiver_should_stop(struct receiver_state *rpt) {
221
+ if(unlikely(__atomic_load_n(&rpt->exit.shutdown, __ATOMIC_RELAXED))) {
222
+ receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN, false);
223
+ return true;
224
+ }
225
+
226
+ return false;
227
+}
228
+
229
+// --------------------------------------------------------------------------------------------------------------------
230
+
231
+static void streaming_parser_init(struct receiver_state *rpt) {
232
+ rpt->thread.cd = (struct plugind){
233
+ .update_every = default_rrd_update_every,
234
+ .unsafe = {
235
+ .spinlock = NETDATA_SPINLOCK_INITIALIZER,
236
+ .running = true,
237
+ .enabled = true,
238
+ },
239
+ .started_t = now_realtime_sec(),
240
+ };
241
+
242
+ // put the client IP and port into the buffers used by plugins.d
243
+ snprintfz(rpt->thread.cd.id, CONFIG_MAX_NAME, "%s:%s", rpt->client_ip, rpt->client_port);
244
+ snprintfz(rpt->thread.cd.filename, FILENAME_MAX, "%s:%s", rpt->client_ip, rpt->client_port);
245
+ snprintfz(rpt->thread.cd.fullfilename, FILENAME_MAX, "%s:%s", rpt->client_ip, rpt->client_port);
246
+ snprintfz(rpt->thread.cd.cmd, PLUGINSD_CMD_MAX, "%s:%s", rpt->client_ip, rpt->client_port);
247
+
248
+ PARSER *parser = NULL;
249
+ {
250
+ PARSER_USER_OBJECT user = {
251
+ .enabled = plugin_is_enabled(&rpt->thread.cd),
252
+ .host = rpt->host,
253
+ .opaque = rpt,
254
+ .cd = &rpt->thread.cd,
255
+ .trust_durations = 1,
256
+ .capabilities = rpt->capabilities,
257
+ };
258
+
259
+ parser = parser_init(&user, -1, -1, PARSER_INPUT_SPLIT, &rpt->sock);
260
+ }
261
+
262
+#ifdef ENABLE_H2O
263
+ parser->h2o_ctx = rpt->h2o_ctx;
264
+#endif
265
+
266
+ pluginsd_keywords_init(parser, PARSER_INIT_STREAMING);
267
+
268
+ rrd_collector_started();
269
+
270
+ rpt->thread.compressed.start = 0;
271
+ rpt->thread.compressed.used = 0;
272
+ rpt->thread.compressed.enabled = stream_decompression_initialize(rpt);
273
+ buffered_reader_init(&rpt->reader);
274
+
275
+#ifdef NETDATA_LOG_STREAM_RECEIVE
276
+ {
277
+ char filename[FILENAME_MAX + 1];
278
+ snprintfz(filename, FILENAME_MAX, "/tmp/stream-receiver-%s.txt", rpt->host ? rrdhost_hostname(
279
+ rpt->host) : "unknown"
280
+ );
281
+ parser->user.stream_log_fp = fopen(filename, "w");
282
+ parser->user.stream_log_repertoire = PARSER_REP_METADATA;
283
+ }
284
+#endif
285
+
286
+ __atomic_store_n(&rpt->thread.parser, parser, __ATOMIC_RELAXED);
287
+ stream_receiver_send_node_and_claim_id_to_child(rpt->host);
288
+
289
+ rpt->thread.buffer = buffer_create(sizeof(rpt->reader.read_buffer), NULL);
290
+
291
+ // help rrdset_push_metric_initialize() select the right buffer
292
+ rpt->host->stream.snd.commit.receiver_tid = gettid_cached();
293
+}
294
+
295
+// --------------------------------------------------------------------------------------------------------------------
296
+
297
+static bool stream_receiver_log_capabilities(BUFFER *wb, void *ptr) {
298
+ struct receiver_state *rpt = ptr;
299
+ if(!rpt)
300
+ return false;
301
+
302
+ stream_capabilities_to_string(wb, rpt->capabilities);
303
+ return true;
304
+}
305
+
306
+static bool stream_receiver_log_transport(BUFFER *wb, void *ptr) {
307
+ struct receiver_state *rpt = ptr;
308
+ if(!rpt)
309
+ return false;
310
+
311
+ buffer_strcat(wb, nd_sock_is_ssl(&rpt->sock) ? "https" : "http");
312
+ return true;
313
+}
314
+
315
+// --------------------------------------------------------------------------------------------------------------------
316
+
317
+void stream_receiver_move_queue_to_running_unsafe(struct stream_thread *sth) {
318
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
319
+
320
+ // process the queue
321
+ Word_t idx = 0;
322
+ for(struct receiver_state *rpt = RECEIVERS_FIRST(&sth->queue.receivers, &idx);
323
+ rpt;
324
+ rpt = RECEIVERS_NEXT(&sth->queue.receivers, &idx)) {
325
+ worker_is_busy(WORKER_STREAM_JOB_DEQUEUE);
326
+
327
+ RECEIVERS_DEL(&sth->queue.receivers, (Word_t)rpt);
328
+
329
+ ND_LOG_STACK lgs[] = {
330
+ ND_LOG_FIELD_STR(NDF_NIDL_NODE, rpt->host->hostname),
331
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
332
+ ND_LOG_FIELD_END(),
333
+ };
334
+ ND_LOG_STACK_PUSH(lgs);
335
+
336
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
337
+ "STREAM RECEIVE[%zu] [%s]: moving host from receiver queue to receiver running...",
338
+ sth->id, rrdhost_hostname(rpt->host));
339
+
340
+ internal_fatal(RECEIVERS_GET(&sth->rcv.receivers, (Word_t)rpt) != NULL, "Receiver to be added is already in the list of receivers");
341
+ RECEIVERS_SET(&sth->rcv.receivers, (Word_t)rpt, rpt);
342
+
343
+ streaming_parser_init(rpt);
344
+
345
+ rpt->host->stream.rcv.status.tid = gettid_cached();
346
+ rpt->thread.meta.type = POLLFD_TYPE_RECEIVER;
347
+ rpt->thread.meta.rpt = rpt;
348
+ if(!nd_poll_add(sth->run.ndpl, rpt->sock.fd, ND_POLL_READ, &rpt->thread.meta))
349
+ internal_fatal(true, "Failed to add receiver socket to nd_poll()");
350
+ }
351
+}
352
+
353
+static void stream_receiver_on_disconnect(struct stream_thread *sth __maybe_unused, struct receiver_state *rpt) {
354
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
355
+ if(!rpt) return;
356
+
357
+ buffer_free(rpt->thread.buffer);
358
+ rpt->thread.buffer = NULL;
359
+
360
+ size_t count = 0;
361
+ PARSER *parser = __atomic_load_n(&rpt->thread.parser, __ATOMIC_RELAXED);
362
+ if(parser) {
363
+ parser->user.v2.stream_buffer.wb = NULL;
364
+
365
+ // make sure send_to_plugin() will not write any data to the socket
366
+ spinlock_lock(&parser->writer.spinlock);
367
+ parser->fd_input = -1;
368
+ parser->fd_output = -1;
369
+ parser->sock = NULL;
370
+ spinlock_unlock(&parser->writer.spinlock);
371
+
372
+ count = parser->user.data_collections_count;
373
+ }
374
+
375
+ // the parser stopped
376
+ receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_EXIT, false);
377
+
378
+ {
379
+ char msg[100 + 1];
380
+ snprintfz(msg, sizeof(msg) - 1, "disconnected (completed %zu updates)", count);
381
+ stream_receiver_log_status(rpt, msg, STREAM_STATUS_DISCONNECTED, NDLP_WARNING);
382
+ }
383
+
384
+ // in case we have cloud connection we inform cloud
385
+ // a child disconnected
386
+ uint64_t total_reboot = rrdhost_stream_path_total_reboot_time_ms(rpt->host);
387
+ schedule_node_state_update(rpt->host, MIN((total_reboot * MAX_CHILD_DISC_TOLERANCE), MAX_CHILD_DISC_DELAY));
388
+
389
+ rrdhost_clear_receiver(rpt);
390
+ rrdhost_set_is_parent_label();
391
+ stream_receiver_free(rpt);
392
+}
393
+
394
+static void stream_receiver_remove(struct stream_thread *sth, struct receiver_state *rpt, const char *why) {
395
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
396
+
397
+ nd_log(NDLS_DAEMON, NDLP_ERR,
398
+ "STREAM RECEIVE[%zu] '%s' [from [%s]:%s]: "
399
+ "receiver disconnected: %s"
400
+ , sth->id
401
+ , rpt->hostname ? rpt->hostname : "-"
402
+ , rpt->client_ip ? rpt->client_ip : "-"
403
+ , rpt->client_port ? rpt->client_port : "-"
404
+ , why ? why : "");
405
+
406
+ internal_fatal(RECEIVERS_GET(&sth->rcv.receivers, (Word_t)rpt) == NULL, "Receiver to be removed is not found in the list of receivers");
407
+ RECEIVERS_DEL(&sth->rcv.receivers, (Word_t)rpt);
408
+ if(!nd_poll_del(sth->run.ndpl, rpt->sock.fd))
409
+ internal_fatal(true, "Failed to remove receiver socket from nd_poll()");
410
+
411
+ rpt->host->stream.rcv.status.tid = 0;
412
+
413
+ stream_thread_node_removed(rpt->host);
414
+
415
+ stream_receiver_on_disconnect(sth, rpt);
416
+ // DO NOT USE rpt after this point
417
+}
418
+
419
+// process poll() events for streaming receivers
420
+void stream_receive_process_poll_events(struct stream_thread *sth, struct receiver_state *rpt, nd_poll_event_t events __maybe_unused, time_t now_s) {
421
+ PARSER *parser = __atomic_load_n(&rpt->thread.parser, __ATOMIC_RELAXED);
422
+ ND_LOG_STACK lgs[] = {
423
+ ND_LOG_FIELD_TXT(NDF_SRC_IP, rpt->client_ip),
424
+ ND_LOG_FIELD_TXT(NDF_SRC_PORT, rpt->client_port),
425
+ ND_LOG_FIELD_TXT(NDF_NIDL_NODE, rpt->hostname),
426
+ ND_LOG_FIELD_CB(NDF_SRC_TRANSPORT, stream_receiver_log_transport, rpt),
427
+ ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_receiver_log_capabilities, rpt),
428
+ ND_LOG_FIELD_CB(NDF_REQUEST, line_splitter_reconstruct_line, &parser->line),
429
+ ND_LOG_FIELD_CB(NDF_NIDL_NODE, parser_reconstruct_node, parser),
430
+ ND_LOG_FIELD_CB(NDF_NIDL_INSTANCE, parser_reconstruct_instance, parser),
431
+ ND_LOG_FIELD_CB(NDF_NIDL_CONTEXT, parser_reconstruct_context, parser),
432
+ ND_LOG_FIELD_END(),
433
+ };
434
+ ND_LOG_STACK_PUSH(lgs);
435
+
436
+ if(receiver_should_stop(rpt)) {
437
+ receiver_set_exit_reason(rpt, rpt->exit.reason, false);
438
+ stream_receiver_remove(sth, rpt, "received stop signal");
439
+ return;
440
+ }
441
+
442
+ rpt->last_msg_t = now_s;
443
+
444
+ if(rpt->thread.compressed.enabled) {
445
+ worker_is_busy(WORKER_STREAM_JOB_SOCKET_RECEIVE);
446
+
447
+ ssize_t bytes = receiver_read_compressed(rpt);
448
+ if(unlikely(bytes <= 0)) {
449
+ if(bytes < 0 && (errno == EWOULDBLOCK || errno == EAGAIN || errno == EINTR))
450
+ return;
451
+
452
+ worker_is_busy(WORKER_STREAM_JOB_SOCKET_ERROR);
453
+ receiver_set_exit_reason(rpt, read_stream_error_to_reason(bytes), false);
454
+ stream_receiver_remove(sth, rpt, "socket read error");
455
+ return;
456
+ }
457
+
458
+ bool node_removed = false;
459
+ while(!node_removed && !nd_thread_signaled_to_cancel() && service_running(SERVICE_STREAMING) && !receiver_should_stop(rpt)) {
460
+ worker_is_busy(WORKER_STREAM_JOB_DECOMPRESS);
461
+
462
+ // feed the decompressor with the new data we just read
463
+ decompressor_status_t feed = receiver_feed_decompressor(rpt);
464
+
465
+ if(likely(feed == DECOMPRESS_OK)) {
466
+ while (!node_removed) {
467
+ // feed our uncompressed data buffer with new data
468
+ decompressor_status_t rc = receiver_get_decompressed(rpt);
469
+
470
+ if (likely(rc == DECOMPRESS_OK)) {
471
+ // loop through all the complete lines found in the uncompressed buffer
472
+
473
+ while (buffered_reader_next_line(&rpt->reader, rpt->thread.buffer)) {
474
+ if (unlikely(parser_action(parser, rpt->thread.buffer->buffer))) {
475
+ receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, false);
476
+ stream_receiver_remove(sth, rpt, "parser failed");
477
+ node_removed = true;
478
+ break;
479
+ }
480
+
481
+ rpt->thread.buffer->len = 0;
482
+ rpt->thread.buffer->buffer[0] = '\0';
483
+ }
484
+ }
485
+ else if (rc == DECOMPRESS_NEED_MORE_DATA)
486
+ break;
487
+
488
+ else {
489
+ receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, false);
490
+ stream_receiver_remove(sth, rpt, "decompressor failed");
491
+ node_removed = true;
492
+ break;
493
+ }
494
+ }
495
+ }
496
+ else if (feed == DECOMPRESS_NEED_MORE_DATA)
497
+ break;
498
+ else {
499
+ receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, false);
500
+ stream_receiver_remove(sth, rpt, "compressed data invalid");
501
+ node_removed = true;
502
+ break;
503
+ }
504
+ }
505
+
506
+ if(!node_removed && receiver_should_stop(rpt)) {
507
+ receiver_set_exit_reason(rpt, rpt->exit.reason, false);
508
+ stream_receiver_remove(sth, rpt, "received stop signal");
509
+ return;
510
+ }
511
+ }
512
+ else {
513
+ worker_is_busy(WORKER_STREAM_JOB_SOCKET_RECEIVE);
514
+
515
+ ssize_t bytes = receiver_read_uncompressed(rpt);
516
+ if(unlikely(bytes <= 0)) {
517
+ if(bytes < 0 && (errno == EWOULDBLOCK || errno == EAGAIN || errno == EINTR))
518
+ return;
519
+
520
+ worker_is_busy(WORKER_STREAM_JOB_SOCKET_ERROR);
521
+ receiver_set_exit_reason(rpt, read_stream_error_to_reason(bytes), false);
522
+ stream_receiver_remove(sth, rpt, "socker read error");
523
+ return;
524
+ }
525
+
526
+ while(buffered_reader_next_line(&rpt->reader, rpt->thread.buffer)) {
527
+ if(unlikely(parser_action(parser, rpt->thread.buffer->buffer))) {
528
+ receiver_set_exit_reason(rpt, STREAM_HANDSHAKE_DISCONNECT_PARSER_FAILED, false);
529
+ stream_receiver_remove(sth, rpt, "parser failed");
530
+ break;
531
+ }
532
+
533
+ rpt->thread.buffer->len = 0;
534
+ rpt->thread.buffer->buffer[0] = '\0';
535
+ }
536
+ }
537
+}
538
+
539
+void stream_receiver_cleanup(struct stream_thread *sth) {
540
+ Word_t idx = 0;
541
+ for(struct receiver_state *rpt = RECEIVERS_FIRST(&sth->rcv.receivers, &idx);
542
+ rpt;
543
+ rpt = RECEIVERS_NEXT(&sth->rcv.receivers, &idx))
544
+ stream_receiver_remove(sth, rpt, "shutdown");
545
+
546
+ RECEIVERS_FREE(&sth->rcv.receivers, NULL);
547
+}
548
+
549
+static void stream_receiver_replication_reset(RRDHOST *host) {
550
+ RRDSET *st;
551
+ rrdset_foreach_read(st, host) {
552
+ rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS);
553
+ rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
554
+ }
555
+ rrdset_foreach_done(st);
556
+ rrdhost_receiver_replicating_charts_zero(host);
557
+}
558
+
559
+bool rrdhost_set_receiver(RRDHOST *host, struct receiver_state *rpt) {
560
+ bool signal_rrdcontext = false;
561
+ bool set_this = false;
562
+
563
+ rrdhost_receiver_lock(host);
564
+
565
+ if (!host->receiver) {
566
+ rrdhost_flag_clear(host, RRDHOST_FLAG_ORPHAN);
567
+
568
+ host->stream.rcv.status.connections++;
569
+ streaming_receiver_connected();
570
+
571
+ host->receiver = rpt;
572
+ rpt->host = host;
573
+
574
+ __atomic_store_n(&rpt->exit.shutdown, false, __ATOMIC_RELAXED);
575
+ host->stream.rcv.status.last_connected = now_realtime_sec();
576
+ host->stream.rcv.status.last_disconnected = 0;
577
+ host->stream.rcv.status.last_chart = 0;
578
+ host->stream.rcv.status.check_obsolete = true;
579
+
580
+ if (rpt->config.health.enabled != CONFIG_BOOLEAN_NO) {
581
+ if (rpt->config.health.delay > 0) {
582
+ host->health.delay_up_to = now_realtime_sec() + rpt->config.health.delay;
583
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
584
+ "[%s]: Postponing health checks for %" PRId64 " seconds, because it was just connected.",
585
+ rrdhost_hostname(host),
586
+ (int64_t) rpt->config.health.delay);
587
+ }
588
+ }
589
+
590
+ host->health_log.health_log_retention_s = rpt->config.health.history;
591
+
592
+// this is a test
593
+// if(rpt->hops <= host->sender->hops)
594
+// stream_sender_thread_stop(host, "HOPS MISMATCH", false);
595
+
596
+ signal_rrdcontext = true;
597
+ stream_receiver_replication_reset(host);
598
+
599
+ rrdhost_flag_clear(rpt->host, RRDHOST_FLAG_STREAM_RECEIVER_DISCONNECTED);
600
+ aclk_queue_node_info(rpt->host, true);
601
+
602
+ rrdhost_stream_parents_reset(host, STREAM_HANDSHAKE_PREPARING);
603
+
604
+ set_this = true;
605
+ }
606
+
607
+ rrdhost_receiver_unlock(host);
608
+
609
+ if(signal_rrdcontext)
610
+ rrdcontext_host_child_connected(host);
611
+
612
+ return set_this;
613
+}
614
+
615
+void rrdhost_clear_receiver(struct receiver_state *rpt) {
616
+ RRDHOST *host = rpt->host;
617
+ if(!host) return;
618
+
619
+ rrdhost_receiver_lock(host);
620
+ {
621
+ // Make sure that we detach this thread and don't kill a freshly arriving receiver
622
+
623
+ if (host->receiver == rpt) {
624
+ rrdhost_flag_set(host, RRDHOST_FLAG_STREAM_RECEIVER_DISCONNECTED);
625
+ rrdhost_receiver_unlock(host);
626
+ {
627
+ // run all these without having the receiver lock
628
+
629
+ stream_path_child_disconnected(host);
630
+ stream_sender_signal_to_stop_and_wait(host, STREAM_HANDSHAKE_DISCONNECT_RECEIVER_LEFT, false);
631
+ stream_receiver_replication_reset(host);
632
+ rrdcontext_host_child_disconnected(host);
633
+
634
+ if (rpt->config.health.enabled)
635
+ rrdcalc_child_disconnected(host);
636
+
637
+ rrdhost_stream_parents_reset(host, STREAM_HANDSHAKE_DISCONNECT_RECEIVER_LEFT);
638
+ }
639
+ rrdhost_receiver_lock(host);
640
+
641
+ // now we have the lock again
642
+
643
+ streaming_receiver_disconnected();
644
+
645
+ __atomic_store_n(&host->receiver->exit.shutdown, false, __ATOMIC_RELAXED);
646
+ host->stream.rcv.status.check_obsolete = false;
647
+ host->stream.rcv.status.last_connected = 0;
648
+ host->stream.rcv.status.last_disconnected = now_realtime_sec();
649
+ host->health.enabled = false;
650
+
651
+ host->stream.rcv.status.exit_reason = rpt->exit.reason;
652
+ rrdhost_flag_set(host, RRDHOST_FLAG_ORPHAN);
653
+ host->receiver = NULL;
654
+ }
655
+ }
656
+
657
+ // this must be cleared with the receiver lock
658
+ pluginsd_process_cleanup(rpt->thread.parser);
659
+ __atomic_store_n(&rpt->thread.parser, NULL, __ATOMIC_RELAXED);
660
+
661
+ rrdhost_receiver_unlock(host);
662
+}
663
+
664
+bool stream_receiver_signal_to_stop_and_wait(RRDHOST *host, STREAM_HANDSHAKE reason) {
665
+ bool ret = false;
666
+
667
+ rrdhost_receiver_lock(host);
668
+
669
+ if(host->receiver) {
670
+ if(!__atomic_load_n(&host->receiver->exit.shutdown, __ATOMIC_RELAXED)) {
671
+ __atomic_store_n(&host->receiver->exit.shutdown, true, __ATOMIC_RELAXED);
672
+ receiver_set_exit_reason(host->receiver, reason, true);
673
+ shutdown(host->receiver->sock.fd, SHUT_RDWR);
674
+ }
675
+ }
676
+
677
+ int count = 2000;
678
+ while (host->receiver && count-- > 0) {
679
+ rrdhost_receiver_unlock(host);
680
+
681
+ // let the lock for the receiver thread to exit
682
+ sleep_usec(1 * USEC_PER_MS);
683
+
684
+ rrdhost_receiver_lock(host);
685
+ }
686
+
687
+ if(host->receiver)
688
+ netdata_log_error("STREAM RECEIVE[x] '%s' [from [%s]:%s]: "
689
+ "streaming thread takes too long to stop, giving up..."
690
+ , rrdhost_hostname(host)
691
+ , host->receiver->client_ip, host->receiver->client_port);
692
+ else
693
+ ret = true;
694
+
695
+ rrdhost_receiver_unlock(host);
696
+
697
+ return ret;
698
+}
src/streaming/stream-sender-api.c
new
+118
@@ -0,0 +1,118 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "stream-sender-internals.h"
4
+
5
+bool stream_sender_has_capabilities(struct rrdhost *host, STREAM_CAPABILITIES capabilities) {
6
+ return host && stream_has_capability(host->sender, capabilities);
7
+}
8
+
9
+bool stream_sender_is_connected_with_ssl(struct rrdhost *host) {
10
+ return host && rrdhost_can_stream_metadata_to_parent(host) && nd_sock_is_ssl(&host->sender->sock);
11
+}
12
+
13
+bool stream_sender_has_compression(struct rrdhost *host) {
14
+ return host && host->sender && host->sender->compressor.initialized;
15
+}
16
+
17
+void stream_sender_structures_init(RRDHOST *host, bool stream, STRING *parents, STRING *api_key, STRING *send_charts_matching) {
18
+ if(rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_SENDER_INITIALIZED))
19
+ return;
20
+
21
+ if(!stream || !parents || !api_key) {
22
+ rrdhost_option_clear(host, RRDHOST_OPTION_SENDER_ENABLED);
23
+ return;
24
+ }
25
+
26
+ rrdhost_flag_set(host, RRDHOST_FLAG_STREAM_SENDER_INITIALIZED);
27
+
28
+ if (host->sender) return;
29
+
30
+ host->sender = callocz(1, sizeof(*host->sender));
31
+ __atomic_add_fetch(&netdata_buffers_statistics.rrdhost_senders, sizeof(*host->sender), __ATOMIC_RELAXED);
32
+
33
+ host->sender->connector.id = -1;
34
+ host->sender->host = host;
35
+ host->sender->sbuf.cb = cbuffer_new(CBUFFER_INITIAL_SIZE, CBUFFER_INITIAL_MAX_SIZE, &netdata_buffers_statistics.cbuffers_streaming);
36
+ host->sender->capabilities = stream_our_capabilities(host, true);
37
+
38
+ nd_sock_init(&host->sender->sock, netdata_ssl_streaming_sender_ctx, netdata_ssl_validate_certificate_sender);
39
+ host->sender->disabled_capabilities = STREAM_CAP_NONE;
40
+
41
+ if(!stream_send.compression.enabled)
42
+ host->sender->disabled_capabilities |= STREAM_CAP_COMPRESSIONS_AVAILABLE;
43
+
44
+ spinlock_init(&host->sender->spinlock);
45
+ replication_sender_init(host->sender);
46
+
47
+ host->stream.snd.destination = string_dup(parents);
48
+ rrdhost_stream_parents_update_from_destination(host);
49
+
50
+ host->stream.snd.api_key = string_dup(api_key);
51
+ host->stream.snd.charts_matching = simple_pattern_create(
52
+ string2str(send_charts_matching), NULL, SIMPLE_PATTERN_EXACT, true);
53
+
54
+ rrdhost_option_set(host, RRDHOST_OPTION_SENDER_ENABLED);
55
+}
56
+
57
+void stream_sender_structures_free(struct rrdhost *host) {
58
+ rrdhost_option_clear(host, RRDHOST_OPTION_SENDER_ENABLED);
59
+
60
+ if (unlikely(!host->sender)) return;
61
+
62
+ // stop a possibly running thread
63
+ stream_sender_signal_to_stop_and_wait(host, STREAM_HANDSHAKE_DISCONNECT_HOST_CLEANUP, true);
64
+ cbuffer_free(host->sender->sbuf.cb);
65
+
66
+ stream_compressor_destroy(&host->sender->compressor);
67
+
68
+ replication_cleanup_sender(host->sender);
69
+
70
+ __atomic_sub_fetch(&netdata_buffers_statistics.rrdhost_senders, sizeof(*host->sender), __ATOMIC_RELAXED);
71
+
72
+ freez(host->sender);
73
+ host->sender = NULL;
74
+
75
+ sender_buffer_destroy(&host->stream.snd.commit);
76
+
77
+ rrdhost_flag_clear(host, RRDHOST_FLAG_STREAM_SENDER_INITIALIZED);
78
+}
79
+
80
+void stream_sender_start_host(struct rrdhost *host) {
81
+ internal_fatal(!rrdhost_has_stream_sender_enabled(host),
82
+ "Host '%s' does not have streaming enabled, but %s() was called",
83
+ rrdhost_hostname(host), __FUNCTION__);
84
+
85
+ stream_sender_add_to_connector_queue(host);
86
+}
87
+
88
+void *stream_sender_start_localhost(void *ptr __maybe_unused) {
89
+ if(!localhost) return NULL;
90
+ stream_sender_start_host(localhost);
91
+ return NULL;
92
+}
93
+
94
+// Either the receiver lost the connection or the host is being destroyed.
95
+// The sender mutex guards thread creation, any spurious data is wiped on reconnection.
96
+void stream_sender_signal_to_stop_and_wait(struct rrdhost *host, STREAM_HANDSHAKE reason, bool wait) {
97
+ if (!host->sender)
98
+ return;
99
+
100
+ stream_sender_lock(host->sender);
101
+
102
+ if(rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_SENDER_ADDED)) {
103
+ __atomic_store_n(&host->sender->exit.shutdown, true, __ATOMIC_RELAXED);
104
+ host->sender->exit.reason = reason;
105
+ }
106
+
107
+ struct stream_opcode msg = host->sender->thread.msg;
108
+ stream_sender_unlock(host->sender);
109
+
110
+ if(reason == STREAM_HANDSHAKE_DISCONNECT_RECEIVER_LEFT)
111
+ msg.opcode = STREAM_OPCODE_SENDER_STOP_RECEIVER_LEFT;
112
+ else
113
+ msg.opcode = STREAM_OPCODE_SENDER_STOP_HOST_CLEANUP;
114
+ stream_sender_send_msg_to_dispatcher(host->sender, msg);
115
+
116
+ while(wait && rrdhost_flag_check(host, RRDHOST_FLAG_STREAM_SENDER_ADDED))
117
+ sleep_usec(10 * USEC_PER_MS);
118
+}
src/streaming/stream-sender-commit.c
new
+255
@@ -0,0 +1,255 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "stream-thread.h"
4
+
5
+#define SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE 3
6
+
7
+static __thread struct sender_buffer commit___thread = { 0 };
8
+
9
+void sender_buffer_destroy(struct sender_buffer *commit) {
10
+ buffer_free(commit->wb);
11
+ commit->wb = NULL;
12
+ commit->used = false;
13
+ commit->our_recreates = 0;
14
+ commit->sender_recreates = 0;
15
+ commit->last_function = NULL;
16
+}
17
+
18
+void sender_commit_thread_buffer_free(void) {
19
+ sender_buffer_destroy(&commit___thread);
20
+}
21
+
22
+// Collector thread starting a transmission
23
+BUFFER *sender_commit_start_with_trace(struct sender_state *s __maybe_unused, struct sender_buffer *commit, const char *func) {
24
+ if(unlikely(commit->used))
25
+ fatal("STREAMING: thread buffer is used multiple times concurrently (%u). "
26
+ "It is already being used by '%s()', and now is called by '%s()'",
27
+ (unsigned)commit->used,
28
+ commit->last_function ? commit->last_function : "(null)",
29
+ func ? func : "(null)");
30
+
31
+ if(unlikely(commit->receiver_tid && commit->receiver_tid != gettid_cached()))
32
+ fatal("STREAMING: thread buffer is reserved for tid %d, but it used by thread %d function '%s()'.",
33
+ commit->receiver_tid, gettid_cached(), func ? func : "(null)");
34
+
35
+ if(unlikely(commit->wb &&
36
+ commit->wb->size > THREAD_BUFFER_INITIAL_SIZE &&
37
+ commit->our_recreates != commit->sender_recreates)) {
38
+ buffer_free(commit->wb);
39
+ commit->wb = NULL;
40
+ }
41
+
42
+ if(unlikely(!commit->wb)) {
43
+ commit->wb = buffer_create(THREAD_BUFFER_INITIAL_SIZE, &netdata_buffers_statistics.buffers_streaming);
44
+ commit->our_recreates = commit->sender_recreates;
45
+ }
46
+
47
+ commit->used = true;
48
+ buffer_flush(commit->wb);
49
+ return commit->wb;
50
+}
51
+
52
+BUFFER *sender_thread_buffer_with_trace(struct sender_state *s __maybe_unused, const char *func) {
53
+ return sender_commit_start_with_trace(s, &commit___thread, func);
54
+}
55
+
56
+BUFFER *sender_host_buffer_with_trace(struct rrdhost *host, const char *func) {
57
+ return sender_commit_start_with_trace(host->sender, &host->stream.snd.commit, func);
58
+}
59
+
60
+// Collector thread finishing a transmission
61
+void sender_buffer_commit(struct sender_state *s, BUFFER *wb, struct sender_buffer *commit, STREAM_TRAFFIC_TYPE type) {
62
+ struct stream_opcode msg;
63
+
64
+ char *src = (char *)buffer_tostring(wb);
65
+ size_t src_len = buffer_strlen(wb);
66
+
67
+ if (unlikely(!src || !src_len))
68
+ return;
69
+
70
+ size_t total_uncompressed_len = src_len;
71
+ size_t total_compressed_len = 0;
72
+
73
+ stream_sender_lock(s);
74
+
75
+ // copy the sequence number of sender buffer recreates, while having our lock
76
+ if(commit)
77
+ commit->sender_recreates = s->sbuf.recreates;
78
+
79
+ if (!s->thread.msg.session) {
80
+ // the dispatcher is not there anymore - ignore these data
81
+ stream_sender_unlock(s);
82
+ if(commit)
83
+ sender_buffer_destroy(commit);
84
+ return;
85
+ }
86
+
87
+ if (unlikely(s->sbuf.cb->max_size < (src_len + 1) * SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE)) {
88
+ // adaptive sizing of the circular buffer is needed to get this.
89
+
90
+ nd_log(
91
+ NDLS_DAEMON,
92
+ NDLP_NOTICE,
93
+ "STREAM %s [send to %s]: max buffer size of %zu is too small "
94
+ "for a data message of size %zu. Increasing the max buffer size "
95
+ "to %d times the max data message size.",
96
+ rrdhost_hostname(s->host),
97
+ s->connected_to,
98
+ s->sbuf.cb->max_size,
99
+ buffer_strlen(wb) + 1,
100
+ SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE);
101
+
102
+ s->sbuf.cb->max_size = (src_len + 1) * SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE;
103
+ }
104
+
105
+#ifdef NETDATA_LOG_STREAM_SENDER
106
+ if (type == STREAM_TRAFFIC_TYPE_METADATA) {
107
+ if (!s->stream_log_fp) {
108
+ char filename[FILENAME_MAX + 1];
109
+ snprintfz(
110
+ filename, FILENAME_MAX, "/tmp/stream-sender-%s.txt", s->host ? rrdhost_hostname(s->host) : "unknown");
111
+
112
+ s->stream_log_fp = fopen(filename, "w");
113
+ }
114
+
115
+ fprintf(
116
+ s->stream_log_fp,
117
+ "\n--- SEND MESSAGE START: %s ----\n"
118
+ "%s"
119
+ "--- SEND MESSAGE END ----------------------------------------\n",
120
+ rrdhost_hostname(s->host),
121
+ src);
122
+ }
123
+#endif
124
+
125
+ if (s->compressor.initialized) {
126
+ // compressed traffic
127
+ if(rrdhost_is_this_a_stream_thread(s->host))
128
+ worker_is_busy(WORKER_STREAM_JOB_COMPRESS);
129
+
130
+ while (src_len) {
131
+ size_t size_to_compress = src_len;
132
+
133
+ if (unlikely(size_to_compress > COMPRESSION_MAX_MSG_SIZE)) {
134
+ if (stream_has_capability(s, STREAM_CAP_BINARY))
135
+ size_to_compress = COMPRESSION_MAX_MSG_SIZE;
136
+ else {
137
+ if (size_to_compress > COMPRESSION_MAX_MSG_SIZE) {
138
+ // we need to find the last newline
139
+ // so that the decompressor will have a whole line to work with
140
+
141
+ const char *t = &src[COMPRESSION_MAX_MSG_SIZE];
142
+ while (--t >= src)
143
+ if (unlikely(*t == '\n'))
144
+ break;
145
+
146
+ if (t <= src)
147
+ size_to_compress = COMPRESSION_MAX_MSG_SIZE;
148
+ else
149
+ size_to_compress = t - src + 1;
150
+ }
151
+ }
152
+ }
153
+
154
+ const char *dst;
155
+ size_t dst_len = stream_compress(&s->compressor, src, size_to_compress, &dst);
156
+ if (!dst_len) {
157
+ nd_log(NDLS_DAEMON, NDLP_ERR,
158
+ "STREAM %s [send to %s]: COMPRESSION failed. Resetting compressor and re-trying",
159
+ rrdhost_hostname(s->host), s->connected_to);
160
+
161
+ stream_compression_initialize(s);
162
+ dst_len = stream_compress(&s->compressor, src, size_to_compress, &dst);
163
+ if (!dst_len)
164
+ goto compression_failed_with_lock;
165
+ }
166
+
167
+ stream_compression_signature_t signature = stream_compress_encode_signature(dst_len);
168
+
169
+#ifdef NETDATA_INTERNAL_CHECKS
170
+ // check if reversing the signature provides the same length
171
+ size_t decoded_dst_len = stream_decompress_decode_signature((const char *)&signature, sizeof(signature));
172
+ if (decoded_dst_len != dst_len)
173
+ fatal(
174
+ "RRDPUSH COMPRESSION: invalid signature, original payload %zu bytes, "
175
+ "compressed payload length %zu bytes, but signature says payload is %zu bytes",
176
+ size_to_compress, dst_len, decoded_dst_len);
177
+#endif
178
+
179
+ total_compressed_len += dst_len + sizeof(signature);
180
+
181
+ if (cbuffer_add_unsafe(s->sbuf.cb, (const char *)&signature, sizeof(signature)) ||
182
+ cbuffer_add_unsafe(s->sbuf.cb, dst, dst_len))
183
+ goto overflow_with_lock;
184
+
185
+ src = src + size_to_compress;
186
+ src_len -= size_to_compress;
187
+ }
188
+ }
189
+ else {
190
+ // uncompressed traffic
191
+
192
+ total_compressed_len = src_len;
193
+
194
+ if (cbuffer_add_unsafe(s->sbuf.cb, src, src_len))
195
+ goto overflow_with_lock;
196
+ }
197
+
198
+ // update s->dispatcher entries
199
+ bool enable_sending = s->thread.bytes_outstanding == 0;
200
+ stream_sender_thread_data_added_data_unsafe(s, type, total_compressed_len, total_uncompressed_len);
201
+
202
+ if (enable_sending)
203
+ msg = s->thread.msg;
204
+
205
+ stream_sender_unlock(s);
206
+
207
+ if (enable_sending) {
208
+ msg.opcode = STREAM_OPCODE_SENDER_POLLOUT;
209
+ stream_sender_send_msg_to_dispatcher(s, msg);
210
+ }
211
+
212
+ return;
213
+
214
+overflow_with_lock: {
215
+ size_t buffer_size = s->sbuf.cb->size;
216
+ size_t buffer_max_size = s->sbuf.cb->max_size;
217
+ size_t buffer_available = cbuffer_available_size_unsafe(s->sbuf.cb);
218
+ msg = s->thread.msg;
219
+ stream_sender_unlock(s);
220
+ msg.opcode = STREAM_OPCODE_SENDER_BUFFER_OVERFLOW;
221
+ stream_sender_send_msg_to_dispatcher(s, msg);
222
+ nd_log(NDLS_DAEMON, NDLP_ERR,
223
+ "STREAM %s [send to %s]: buffer overflow while adding %zu bytes (buffer size %zu, max size %zu, available %zu). "
224
+ "Restarting connection.",
225
+ rrdhost_hostname(s->host), s->connected_to,
226
+ total_compressed_len, buffer_size, buffer_max_size, buffer_available);
227
+ return;
228
+ }
229
+
230
+compression_failed_with_lock: {
231
+ stream_compression_deactivate(s);
232
+ msg = s->thread.msg;
233
+ stream_sender_unlock(s);
234
+ msg.opcode = STREAM_OPCODE_SENDER_RECONNECT_WITHOUT_COMPRESSION;
235
+ stream_sender_send_msg_to_dispatcher(s, msg);
236
+ nd_log(NDLS_DAEMON, NDLP_ERR,
237
+ "STREAM %s [send to %s]: COMPRESSION failed (twice). Deactivating compression and restarting connection.",
238
+ rrdhost_hostname(s->host), s->connected_to);
239
+ }
240
+}
241
+
242
+void sender_thread_commit(struct sender_state *s, BUFFER *wb, STREAM_TRAFFIC_TYPE type, const char *func) {
243
+ struct sender_buffer *commit = (wb == commit___thread.wb) ? & commit___thread : &s->host->stream.snd.commit;
244
+
245
+ if (unlikely(wb != commit->wb))
246
+ fatal("STREAMING: function '%s()' is trying to commit an unknown commit buffer.", func);
247
+
248
+ if (unlikely(!commit->used))
249
+ fatal("STREAMING: function '%s()' is committing a sender buffer twice.", func);
250
+
251
+ commit->used = false;
252
+ commit->last_function = NULL;
253
+
254
+ sender_buffer_commit(s, wb, commit, type);
255
+}
src/streaming/stream-sender-commit.h
new
+39
@@ -0,0 +1,39 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_STREAM_SENDER_COMMIT_H
4
+#define NETDATA_STREAM_SENDER_COMMIT_H
5
+
6
+#include "libnetdata/libnetdata.h"
7
+#include "stream-traffic-types.h"
8
+
9
+struct rrdhost;
10
+struct sender_state;
11
+struct receiver_state;
12
+
13
+struct sender_buffer {
14
+ pid_t receiver_tid;
15
+ BUFFER *wb;
16
+ bool used;
17
+ size_t our_recreates;
18
+ size_t sender_recreates;
19
+ const char *last_function;
20
+};
21
+void sender_buffer_destroy(struct sender_buffer *commit);
22
+
23
+// thread buffer for sending data upstream (to a parent)
24
+
25
+BUFFER *sender_thread_buffer_with_trace(struct sender_state *s, const char *func);
26
+#define sender_thread_buffer(s) sender_thread_buffer_with_trace(s, __FUNCTION__)
27
+
28
+BUFFER *sender_host_buffer_with_trace(struct rrdhost *host, const char *func);
29
+#define sender_host_buffer(host) sender_host_buffer_with_trace(host, __FUNCTION__)
30
+
31
+void sender_thread_commit(struct sender_state *s, BUFFER *wb, STREAM_TRAFFIC_TYPE type, const char *func);
32
+#define sender_commit(s, wb, type) sender_thread_commit(s, wb, type, __FUNCTION__)
33
+
34
+void sender_buffer_commit(struct sender_state *s, BUFFER *wb, struct sender_buffer *commit, STREAM_TRAFFIC_TYPE type);
35
+#define sender_commit_clean_buffer(s, wb, type) sender_buffer_commit(s, wb, NULL, type)
36
+
37
+void sender_commit_thread_buffer_free(void);
38
+
39
+#endif //NETDATA_STREAM_SENDER_COMMIT_H
src/streaming/stream-sender-execute.c
renamed
+54
-54
@@ -1,6 +1,6 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
3
-#include "sender-internals.h"
3
+#include "stream-thread.h"
4
5
struct inflight_stream_function {
6
struct sender_state *sender;
@@ -12,20 +12,18 @@ static void stream_execute_function_callback(BUFFER *func_wb, int code, void *da
12
struct inflight_stream_function *tmp = data;
13
struct sender_state *s = tmp->sender;
14
15
- if(rrdhost_can_send_definitions_to_parent(s->host)) {
16
- BUFFER *wb = sender_start(s);
15
+ if(rrdhost_can_stream_metadata_to_parent(s->host)) {
16
+ // for functions we use a new buffer, to avoid keeping a big buffer in memory
17
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
18
18
- pluginsd_function_result_begin_to_buffer(wb
19
- , string2str(tmp->transaction)
20
- , code
21
- , content_type_id2string(func_wb->content_type)
22
- , func_wb->expires);
19
+ pluginsd_function_result_begin_to_buffer(
20
+ wb, string2str(tmp->transaction), code,
21
+ content_type_id2string(func_wb->content_type), func_wb->expires);
22
23
buffer_fast_strcat(wb, buffer_tostring(func_wb), buffer_strlen(func_wb));
24
pluginsd_function_result_end_to_buffer(wb);
25
27
- sender_commit(s, wb, STREAM_TRAFFIC_TYPE_FUNCTIONS);
28
- sender_thread_buffer_free();
26
+ sender_commit_clean_buffer(s, wb, STREAM_TRAFFIC_TYPE_FUNCTIONS);
27
28
internal_error(true, "STREAM %s [send to %s] FUNCTION transaction %s sending back response (%zu bytes, %"PRIu64" usec).",
29
rrdhost_hostname(s->host), s->connected_to,
@@ -43,18 +41,18 @@ static void stream_execute_function_progress_callback(void *data, size_t done, s
41
struct inflight_stream_function *tmp = data;
42
struct sender_state *s = tmp->sender;
43
46
- if(rrdhost_can_send_definitions_to_parent(s->host)) {
47
- BUFFER *wb = sender_start(s);
44
+ if(rrdhost_can_stream_metadata_to_parent(s->host)) {
45
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
46
47
buffer_sprintf(wb, PLUGINSD_KEYWORD_FUNCTION_PROGRESS " '%s' %zu %zu\n",
48
string2str(tmp->transaction), done, all);
49
52
- sender_commit(s, wb, STREAM_TRAFFIC_TYPE_FUNCTIONS);
50
+ sender_commit_clean_buffer(s, wb, STREAM_TRAFFIC_TYPE_FUNCTIONS);
51
}
52
}
53
54
static void execute_commands_function(struct sender_state *s, const char *command, const char *transaction, const char *timeout_s, const char *function, BUFFER *payload, const char *access, const char *source) {
57
- worker_is_busy(WORKER_SENDER_JOB_FUNCTION_REQUEST);
55
+ worker_is_busy(WORKER_SENDER_JOB_EXECUTE_FUNCTION);
56
nd_log(NDLS_ACCESS, NDLP_INFO, NULL);
57
58
if(!transaction || !*transaction || !timeout_s || !*timeout_s || !function || !*function) {
@@ -108,7 +106,7 @@ static void execute_deferred_function(struct sender_state *s, void *data) {
106
static void execute_deferred_json(struct sender_state *s, void *data) {
107
const char *keyword = data;
108
111
- if(strcmp(keyword, PLUGINSD_KEYWORD_STREAM_PATH) == 0)
109
+ if(strcmp(keyword, PLUGINSD_KEYWORD_JSON_CMD_STREAM_PATH) == 0)
110
stream_path_set_from_json(s->host, buffer_tostring(s->defer.payload), true);
111
else
112
nd_log(NDLS_DAEMON, NDLP_ERR, "STREAM: unknown JSON keyword '%s' with payload: %s", keyword, buffer_tostring(s->defer.payload));
@@ -141,21 +139,19 @@ static void cleanup_deferred_data(struct sender_state *s) {
139
s->defer.action_data = NULL;
140
}
141
144
-void rrdpush_sender_execute_commands_cleanup(struct sender_state *s) {
142
+void stream_sender_execute_commands_cleanup(struct sender_state *s) {
143
cleanup_deferred_data(s);
144
}
145
146
// This is just a placeholder until the gap filling state machine is inserted
149
-void rrdpush_sender_execute_commands(struct sender_state *s) {
150
- worker_is_busy(WORKER_SENDER_JOB_EXECUTE);
151
-
147
+void stream_sender_execute_commands(struct sender_state *s) {
148
ND_LOG_STACK lgs[] = {
153
- ND_LOG_FIELD_CB(NDF_REQUEST, line_splitter_reconstruct_line, &s->line),
149
+ ND_LOG_FIELD_CB(NDF_REQUEST, line_splitter_reconstruct_line, &s->rbuf.line),
150
ND_LOG_FIELD_END(),
151
};
152
ND_LOG_STACK_PUSH(lgs);
153
158
- char *start = s->read_buffer, *end = &s->read_buffer[s->read_len], *newline;
154
+ char *start = s->rbuf.b, *end = &s->rbuf.b[s->rbuf.read_len], *newline;
155
*end = '\0';
156
for( ; start < end ; start = newline + 1) {
157
newline = strchr(start, '\n');
@@ -169,7 +165,7 @@ void rrdpush_sender_execute_commands(struct sender_state *s) {
165
}
166
167
*newline = '\0';
172
- s->line.count++;
168
+ s->rbuf.line.count++;
169
170
if(s->defer.end_keyword) {
171
if(strcmp(start, s->defer.end_keyword) == 0) {
@@ -184,25 +180,25 @@ void rrdpush_sender_execute_commands(struct sender_state *s) {
180
continue;
181
}
182
187
- s->line.num_words = quoted_strings_splitter_whitespace(start, s->line.words, PLUGINSD_MAX_WORDS);
188
- const char *command = get_word(s->line.words, s->line.num_words, 0);
183
+ s->rbuf.line.num_words = quoted_strings_splitter_whitespace(start, s->rbuf.line.words, PLUGINSD_MAX_WORDS);
184
+ const char *command = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 0);
185
186
if(command && strcmp(command, PLUGINSD_CALL_FUNCTION) == 0) {
191
- char *transaction = get_word(s->line.words, s->line.num_words, 1);
192
- char *timeout_s = get_word(s->line.words, s->line.num_words, 2);
193
- char *function = get_word(s->line.words, s->line.num_words, 3);
194
- char *access = get_word(s->line.words, s->line.num_words, 4);
195
- char *source = get_word(s->line.words, s->line.num_words, 5);
187
+ char *transaction = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 1);
188
+ char *timeout_s = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 2);
189
+ char *function = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 3);
190
+ char *access = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 4);
191
+ char *source = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 5);
192
193
execute_commands_function(s, command, transaction, timeout_s, function, NULL, access, source);
194
}
195
else if(command && strcmp(command, PLUGINSD_CALL_FUNCTION_PAYLOAD_BEGIN) == 0) {
200
- char *transaction = get_word(s->line.words, s->line.num_words, 1);
201
- char *timeout_s = get_word(s->line.words, s->line.num_words, 2);
202
- char *function = get_word(s->line.words, s->line.num_words, 3);
203
- char *access = get_word(s->line.words, s->line.num_words, 4);
204
- char *source = get_word(s->line.words, s->line.num_words, 5);
205
- char *content_type = get_word(s->line.words, s->line.num_words, 6);
196
+ char *transaction = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 1);
197
+ char *timeout_s = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 2);
198
+ char *function = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 3);
199
+ char *access = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 4);
200
+ char *source = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 5);
201
+ char *content_type = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 6);
202
203
s->defer.end_keyword = PLUGINSD_CALL_FUNCTION_PAYLOAD_END;
204
s->defer.payload = buffer_create(0, NULL);
@@ -220,29 +216,31 @@ void rrdpush_sender_execute_commands(struct sender_state *s) {
216
s->defer.action_data = dfd;
217
}
218
else if(command && strcmp(command, PLUGINSD_CALL_FUNCTION_CANCEL) == 0) {
223
- worker_is_busy(WORKER_SENDER_JOB_FUNCTION_REQUEST);
219
+ worker_is_busy(WORKER_SENDER_JOB_EXECUTE_FUNCTION);
220
nd_log(NDLS_ACCESS, NDLP_DEBUG, NULL);
221
226
- char *transaction = get_word(s->line.words, s->line.num_words, 1);
222
+ char *transaction = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 1);
223
if(transaction && *transaction)
224
rrd_function_cancel(transaction);
225
}
226
else if(command && strcmp(command, PLUGINSD_CALL_FUNCTION_PROGRESS) == 0) {
231
- worker_is_busy(WORKER_SENDER_JOB_FUNCTION_REQUEST);
227
+ worker_is_busy(WORKER_SENDER_JOB_EXECUTE_FUNCTION);
228
nd_log(NDLS_ACCESS, NDLP_DEBUG, NULL);
229
234
- char *transaction = get_word(s->line.words, s->line.num_words, 1);
230
+ char *transaction = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 1);
231
if(transaction && *transaction)
232
rrd_function_progress(transaction);
233
}
234
else if (command && strcmp(command, PLUGINSD_KEYWORD_REPLAY_CHART) == 0) {
239
- worker_is_busy(WORKER_SENDER_JOB_REPLAY_REQUEST);
240
- nd_log(NDLS_ACCESS, NDLP_DEBUG, NULL);
235
+ worker_is_busy(WORKER_SENDER_JOB_EXECUTE_REPLAY);
236
242
- const char *chart_id = get_word(s->line.words, s->line.num_words, 1);
243
- const char *start_streaming = get_word(s->line.words, s->line.num_words, 2);
244
- const char *after = get_word(s->line.words, s->line.num_words, 3);
245
- const char *before = get_word(s->line.words, s->line.num_words, 4);
237
+ // do not log replication commands received - way too many!
238
+ // nd_log(NDLS_ACCESS, NDLP_DEBUG, NULL);
239
+
240
+ const char *chart_id = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 1);
241
+ const char *start_streaming = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 2);
242
+ const char *after = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 3);
243
+ const char *before = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 4);
244
245
if (!chart_id || !start_streaming || !after || !before) {
246
netdata_log_error("STREAM %s [send to %s] %s command is incomplete"
@@ -263,10 +261,13 @@ void rrdpush_sender_execute_commands(struct sender_state *s) {
261
}
262
}
263
else if(command && strcmp(command, PLUGINSD_KEYWORD_NODE_ID) == 0) {
266
- rrdpush_sender_get_node_and_claim_id_from_parent(s);
264
+ worker_is_busy(WORKER_SENDER_JOB_EXECUTE_META);
265
+ stream_sender_get_node_and_claim_id_from_parent(s);
266
}
267
else if(command && strcmp(command, PLUGINSD_KEYWORD_JSON) == 0) {
269
- char *keyword = get_word(s->line.words, s->line.num_words, 1);
268
+ worker_is_busy(WORKER_SENDER_JOB_EXECUTE_META);
269
+
270
+ char *keyword = get_word(s->rbuf.line.words, s->rbuf.line.num_words, 1);
271
272
s->defer.end_keyword = PLUGINSD_KEYWORD_JSON_END;
273
s->defer.payload = buffer_create(0, NULL);
@@ -276,19 +277,18 @@ void rrdpush_sender_execute_commands(struct sender_state *s) {
277
}
278
else {
279
netdata_log_error("STREAM %s [send to %s] received unknown command over connection: %s",
279
- rrdhost_hostname(s->host), s->connected_to, s->line.words[0]?s->line.words[0]:"(unset)");
280
+ rrdhost_hostname(s->host), s->connected_to, s->rbuf.line.words[0]?s->rbuf.line.words[0]:"(unset)");
281
}
282
282
- line_splitter_reset(&s->line);
283
- worker_is_busy(WORKER_SENDER_JOB_EXECUTE);
283
+ line_splitter_reset(&s->rbuf.line);
284
}
285
286
if (start < end) {
287
- memmove(s->read_buffer, start, end-start);
288
- s->read_len = end - start;
287
+ memmove(s->rbuf.b, start, end-start);
288
+ s->rbuf.read_len = end - start;
289
}
290
else {
291
- s->read_buffer[0] = '\0';
292
- s->read_len = 0;
291
+ s->rbuf.b[0] = '\0';
292
+ s->rbuf.read_len = 0;
293
}
294
}
src/streaming/stream-sender-internals.h
new
+178
@@ -0,0 +1,178 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_STREAM_SENDER_INTERNALS_H
4
+#define NETDATA_STREAM_SENDER_INTERNALS_H
5
+
6
+#include "stream.h"
7
+#include "stream-thread.h"
8
+#include "h2o-common.h"
9
+#include "aclk/https_client.h"
10
+#include "stream-parents.h"
11
+
12
+// connector thread
13
+#define WORKER_SENDER_CONNECTOR_JOB_CONNECTING 0
14
+#define WORKER_SENDER_CONNECTOR_JOB_CONNECTED 1
15
+#define WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_BAD_HANDSHAKE 2
16
+#define WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_TIMEOUT 3
17
+#define WORKER_SENDER_CONNECTOR_JOB_DISCONNECT_CANT_UPGRADE_CONNECTION 4
18
+#define WORKER_SENDER_CONNECTOR_JOB_QUEUED_NODES 5
19
+#define WORKER_SENDER_CONNECTOR_JOB_CONNECTED_NODES 6
20
+#define WORKER_SENDER_CONNECTOR_JOB_FAILED_NODES 7
21
+#define WORKER_SENDER_CONNECTOR_JOB_CANCELLED_NODES 8
22
+
23
+#define CONNECTED_TO_SIZE 100
24
+
25
+#define CBUFFER_INITIAL_SIZE (16 * 1024)
26
+#define CBUFFER_INITIAL_MAX_SIZE (10 * 1024 * 1024)
27
+#define THREAD_BUFFER_INITIAL_SIZE (CBUFFER_INITIAL_SIZE / 2)
28
+
29
+#include "stream-compression/compression.h"
30
+#include "stream-conf.h"
31
+
32
+typedef void (*stream_defer_action_t)(struct sender_state *s, void *data);
33
+typedef void (*stream_defer_cleanup_t)(struct sender_state *s, void *data);
34
+
35
+struct sender_state {
36
+ SPINLOCK spinlock;
37
+
38
+ RRDHOST *host;
39
+ STREAM_CAPABILITIES capabilities;
40
+ STREAM_CAPABILITIES disabled_capabilities;
41
+ int16_t hops;
42
+
43
+ ND_SOCK sock;
44
+
45
+ struct {
46
+ struct stream_opcode msg; // the template for sending a message to the dispatcher - protected by sender_lock()
47
+
48
+ // this is a property of stream_sender_send_msg_to_dispatcher()
49
+ // protected by dispatcher->messages.spinlock
50
+ // DO NOT READ OR WRITE ANYWHERE
51
+ uint32_t msg_slot; // ensures a dispatcher queue that can never get full
52
+
53
+ // statistics about our compression efficiency
54
+ size_t bytes_compressed;
55
+ size_t bytes_uncompressed;
56
+
57
+ // the current buffer statistics
58
+ // these SHOULD ALWAYS BE CALCULATED ON EVERY sender_unlock() IF THE BUFFER WAS MODIFIED
59
+ size_t bytes_outstanding;
60
+ size_t bytes_available;
61
+ NETDATA_DOUBLE buffer_ratio;
62
+
63
+ // statistics about successful sends
64
+ size_t sends;
65
+ size_t bytes_sent;
66
+ size_t bytes_sent_by_type[STREAM_TRAFFIC_TYPE_MAX];
67
+
68
+ struct pollfd_meta meta;
69
+ } thread;
70
+
71
+ struct {
72
+ int8_t id; // the connector id - protected by sender_lock()
73
+ } connector;
74
+
75
+ char connected_to[CONNECTED_TO_SIZE + 1]; // We don't know which proxy we connect to, passed back from socket.c
76
+ time_t last_traffic_seen_t;
77
+ time_t last_state_since_t; // the timestamp of the last state (online/offline) change
78
+
79
+ struct {
80
+ struct circular_buffer *cb;
81
+ size_t recreates;
82
+ } sbuf;
83
+
84
+ struct {
85
+ char b[PLUGINSD_LINE_MAX + 1];
86
+ ssize_t read_len;
87
+ struct line_splitter line;
88
+ } rbuf;
89
+
90
+ struct compressor_state compressor;
91
+
92
+#ifdef NETDATA_LOG_STREAM_SENDER
93
+ FILE *stream_log_fp;
94
+#endif
95
+
96
+ struct {
97
+ bool shutdown; // when set, the sender should stop sending this host
98
+ STREAM_HANDSHAKE reason; // the reason we decided to stop this sender
99
+ } exit;
100
+
101
+ struct {
102
+ DICTIONARY *requests; // de-duplication of replication requests, per chart
103
+ time_t oldest_request_after_t; // the timestamp of the oldest replication request
104
+ time_t latest_completed_before_t; // the timestamp of the latest replication request
105
+
106
+ struct {
107
+ size_t pending_requests; // the currently outstanding replication requests
108
+ size_t charts_replicating; // the number of unique charts having pending replication requests (on every request one is added and is removed when we finish it - it does not track completion of the replication for this chart)
109
+ bool reached_max; // true when the sender buffer should not get more replication responses
110
+ } atomic;
111
+
112
+ } replication;
113
+
114
+ struct {
115
+ size_t buffer_used_percentage; // the current utilization of the sending buffer
116
+ usec_t last_flush_time_ut; // the last time the sender flushed the sending buffer in USEC
117
+ } atomic;
118
+
119
+ struct {
120
+ const char *end_keyword;
121
+ BUFFER *payload;
122
+ stream_defer_action_t action;
123
+ stream_defer_cleanup_t cleanup;
124
+ void *action_data;
125
+ } defer;
126
+
127
+ bool parent_using_h2o;
128
+};
129
+
130
+#define stream_sender_lock(sender) spinlock_lock(&(sender)->spinlock)
131
+#define stream_sender_unlock(sender) spinlock_unlock(&(sender)->spinlock)
132
+
133
+#define stream_sender_replication_buffer_full_set(sender, value) __atomic_store_n(&((sender)->replication.atomic.reached_max), value, __ATOMIC_SEQ_CST)
134
+#define stream_sender_replication_buffer_full_get(sender) __atomic_load_n(&((sender)->replication.atomic.reached_max), __ATOMIC_SEQ_CST)
135
+
136
+#define stream_sender_set_buffer_used_percent(sender, value) __atomic_store_n(&((sender)->atomic.buffer_used_percentage), value, __ATOMIC_RELAXED)
137
+#define stream_sender_get_buffer_used_percent(sender) __atomic_load_n(&((sender)->atomic.buffer_used_percentage), __ATOMIC_RELAXED)
138
+
139
+#define stream_sender_set_flush_time(sender) __atomic_store_n(&((sender)->atomic.last_flush_time_ut), now_realtime_usec(), __ATOMIC_RELAXED)
140
+#define stream_sender_get_flush_time(sender) __atomic_load_n(&((sender)->atomic.last_flush_time_ut), __ATOMIC_RELAXED)
141
+
142
+#define stream_sender_replicating_charts(sender) __atomic_load_n(&((sender)->replication.atomic.charts_replicating), __ATOMIC_RELAXED)
143
+#define stream_sender_replicating_charts_plus_one(sender) __atomic_add_fetch(&((sender)->replication.atomic.charts_replicating), 1, __ATOMIC_RELAXED)
144
+#define stream_sender_replicating_charts_minus_one(sender) __atomic_sub_fetch(&((sender)->replication.atomic.charts_replicating), 1, __ATOMIC_RELAXED)
145
+#define stream_sender_replicating_charts_zero(sender) __atomic_store_n(&((sender)->replication.atomic.charts_replicating), 0, __ATOMIC_RELAXED)
146
+
147
+#define stream_sender_pending_replication_requests(sender) __atomic_load_n(&((sender)->replication.atomic.pending_requests), __ATOMIC_RELAXED)
148
+#define stream_sender_pending_replication_requests_plus_one(sender) __atomic_add_fetch(&((sender)->replication.atomic.pending_requests), 1, __ATOMIC_RELAXED)
149
+#define stream_sender_pending_replication_requests_minus_one(sender) __atomic_sub_fetch(&((sender)->replication.atomic.pending_requests), 1, __ATOMIC_RELAXED)
150
+#define stream_sender_pending_replication_requests_zero(sender) __atomic_store_n(&((sender)->replication.atomic.pending_requests), 0, __ATOMIC_RELAXED)
151
+
152
+void stream_sender_add_to_connector_queue(RRDHOST *host);
153
+
154
+void stream_sender_execute_commands_cleanup(struct sender_state *s);
155
+void stream_sender_execute_commands(struct sender_state *s);
156
+
157
+bool stream_connect(struct sender_state *s, uint16_t default_port, time_t timeout);
158
+
159
+bool stream_sender_is_host_stopped(struct sender_state *s);
160
+
161
+void stream_sender_send_msg_to_dispatcher(struct sender_state *s, struct stream_opcode msg);
162
+
163
+void stream_sender_thread_data_added_data_unsafe(struct sender_state *s, STREAM_TRAFFIC_TYPE type, uint64_t bytes_compressed, uint64_t bytes_uncompressed);
164
+
165
+void stream_sender_add_to_queue(struct sender_state *s);
166
+
167
+// stream connector
168
+bool stream_connector_init(struct sender_state *s);
169
+void stream_connector_cancel_threads(void);
170
+void stream_connector_add(struct sender_state *s);
171
+void stream_connector_requeue(struct sender_state *s);
172
+bool stream_connector_is_signaled_to_stop(struct sender_state *s);
173
+
174
+void stream_sender_on_connect(struct sender_state *s);
175
+
176
+void stream_sender_remove(struct sender_state *s);
177
+
178
+#endif //NETDATA_STREAM_SENDER_INTERNALS_H
src/streaming/stream-sender.c
new
+582
@@ -0,0 +1,582 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "stream-thread.h"
4
+#include "stream-sender-internals.h"
5
+
6
+static void stream_sender_move_running_to_connector_or_remove(struct stream_thread *sth, struct sender_state *s, STREAM_HANDSHAKE reason, bool reconnect);
7
+
8
+// --------------------------------------------------------------------------------------------------------------------
9
+
10
+static void stream_sender_cbuffer_recreate_timed_unsafe(struct sender_state *s, time_t now_s, bool force) {
11
+ static __thread time_t last_reset_time_s = 0;
12
+
13
+ if(!force && now_s - last_reset_time_s < 300)
14
+ return;
15
+
16
+ last_reset_time_s = now_s;
17
+
18
+ s->sbuf.recreates++; // we increase even if we don't do it, to have sender_start() recreate its buffers
19
+
20
+ if(s->sbuf.cb && s->sbuf.cb->size > CBUFFER_INITIAL_SIZE) {
21
+ cbuffer_free(s->sbuf.cb);
22
+ s->sbuf.cb = cbuffer_new(CBUFFER_INITIAL_SIZE, stream_send.buffer_max_size, &netdata_buffers_statistics.cbuffers_streaming);
23
+ }
24
+}
25
+
26
+static void rrdpush_sender_cbuffer_flush(RRDHOST *host) {
27
+ stream_sender_set_flush_time(host->sender);
28
+
29
+ stream_sender_lock(host->sender);
30
+
31
+ // flush the output buffer from any data it may have
32
+ cbuffer_flush(host->sender->sbuf.cb);
33
+ stream_sender_cbuffer_recreate_timed_unsafe(host->sender, now_monotonic_sec(), true);
34
+
35
+ stream_sender_unlock(host->sender);
36
+}
37
+
38
+// --------------------------------------------------------------------------------------------------------------------
39
+
40
+static void rrdpush_sender_charts_and_replication_reset(struct sender_state *s) {
41
+ stream_sender_set_flush_time(s);
42
+
43
+ // stop all replication commands inflight
44
+ replication_sender_delete_pending_requests(s);
45
+
46
+ // reset the state of all charts
47
+ RRDSET *st;
48
+ rrdset_foreach_read(st, s->host) {
49
+ rrdset_flag_clear(st, RRDSET_FLAG_SENDER_REPLICATION_IN_PROGRESS);
50
+ rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
51
+
52
+ st->stream.snd.resync_time_s = 0;
53
+
54
+ RRDDIM *rd;
55
+ rrddim_foreach_read(rd, st)
56
+ rrddim_metadata_exposed_upstream_clear(rd);
57
+ rrddim_foreach_done(rd);
58
+
59
+ rrdset_metadata_updated(st);
60
+ }
61
+ rrdset_foreach_done(st);
62
+
63
+ rrdhost_sender_replicating_charts_zero(s->host);
64
+ stream_sender_replicating_charts_zero(s);
65
+}
66
+
67
+// --------------------------------------------------------------------------------------------------------------------
68
+
69
+void stream_sender_on_connect(struct sender_state *s) {
70
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
71
+ "STREAM SEND [%s]: running on-connect hooks...",
72
+ rrdhost_hostname(s->host));
73
+
74
+ rrdhost_flag_set(s->host, RRDHOST_FLAG_STREAM_SENDER_CONNECTED);
75
+
76
+ rrdpush_sender_charts_and_replication_reset(s);
77
+ rrdpush_sender_cbuffer_flush(s->host);
78
+
79
+ s->last_traffic_seen_t = now_monotonic_sec();
80
+ s->rbuf.read_len = 0;
81
+ s->sbuf.cb->read = 0;
82
+ s->sbuf.cb->write = 0;
83
+}
84
+
85
+static void stream_sender_on_ready_to_dispatch(struct sender_state *s) {
86
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
87
+ "STREAM SEND [%s]: running ready-to-dispatch hooks...",
88
+ rrdhost_hostname(s->host));
89
+
90
+ // set this flag before sending any data, or the data will not be sent
91
+ rrdhost_flag_set(s->host, RRDHOST_FLAG_STREAM_SENDER_READY_4_METRICS);
92
+
93
+ stream_sender_execute_commands_cleanup(s);
94
+ stream_sender_send_custom_host_variables(s->host);
95
+ stream_path_send_to_parent(s->host);
96
+ stream_sender_send_claimed_id(s->host);
97
+ stream_send_host_labels(s->host);
98
+ stream_send_global_functions(s->host);
99
+}
100
+
101
+static void stream_sender_on_disconnect(struct sender_state *s) {
102
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
103
+ "STREAM SEND [%s]: running on-disconnect hooks...",
104
+ rrdhost_hostname(s->host));
105
+
106
+ stream_sender_execute_commands_cleanup(s);
107
+ rrdpush_sender_charts_and_replication_reset(s);
108
+ stream_sender_clear_parent_claim_id(s->host);
109
+ stream_receiver_send_node_and_claim_id_to_child(s->host);
110
+ stream_path_parent_disconnected(s->host);
111
+}
112
+
113
+// --------------------------------------------------------------------------------------------------------------------
114
+
115
+static bool stream_sender_log_capabilities(BUFFER *wb, void *ptr) {
116
+ struct sender_state *state = ptr;
117
+ if(!state)
118
+ return false;
119
+
120
+ stream_capabilities_to_string(wb, state->capabilities);
121
+ return true;
122
+}
123
+
124
+static bool stream_sender_log_transport(BUFFER *wb, void *ptr) {
125
+ struct sender_state *state = ptr;
126
+ if(!state)
127
+ return false;
128
+
129
+ buffer_strcat(wb, nd_sock_is_ssl(&state->sock) ? "https" : "http");
130
+ return true;
131
+}
132
+
133
+static bool stream_sender_log_dst_ip(BUFFER *wb, void *ptr) {
134
+ struct sender_state *state = ptr;
135
+ if(!state || state->sock.fd == -1)
136
+ return false;
137
+
138
+ SOCKET_PEERS peers = nd_sock_socket_peers(&state->sock);
139
+ buffer_strcat(wb, peers.peer.ip);
140
+ return true;
141
+}
142
+
143
+static bool stream_sender_log_dst_port(BUFFER *wb, void *ptr) {
144
+ struct sender_state *state = ptr;
145
+ if(!state || state->sock.fd == -1)
146
+ return false;
147
+
148
+ SOCKET_PEERS peers = nd_sock_socket_peers(&state->sock);
149
+ buffer_print_uint64(wb, peers.peer.port);
150
+ return true;
151
+}
152
+
153
+// --------------------------------------------------------------------------------------------------------------------
154
+
155
+static void stream_sender_thread_data_reset_unsafe(struct sender_state *s) {
156
+ memset(s->thread.bytes_sent_by_type, 0, sizeof(s->thread.bytes_sent_by_type));
157
+
158
+ s->thread.bytes_uncompressed = 0;
159
+ s->thread.bytes_compressed = 0;
160
+ s->thread.bytes_outstanding = 0;
161
+ s->thread.bytes_available = 0;
162
+ s->thread.buffer_ratio = 0.0;
163
+ s->thread.sends = 0;
164
+ s->thread.bytes_sent = 0;
165
+ replication_recalculate_buffer_used_ratio_unsafe(s);
166
+}
167
+
168
+static void stream_sender_thread_data_sent_data_unsafe(struct sender_state *s, uint64_t bytes_sent) {
169
+ s->thread.sends++;
170
+ s->thread.bytes_sent += bytes_sent;
171
+ s->thread.bytes_outstanding = cbuffer_next_unsafe(s->sbuf.cb, NULL);
172
+ s->thread.bytes_available = cbuffer_available_size_unsafe(s->sbuf.cb);
173
+ s->thread.buffer_ratio = (NETDATA_DOUBLE)(s->sbuf.cb->max_size - s->thread.bytes_available) * 100.0 / (NETDATA_DOUBLE)s->sbuf.cb->max_size;
174
+ replication_recalculate_buffer_used_ratio_unsafe(s);
175
+}
176
+
177
+void stream_sender_thread_data_added_data_unsafe(struct sender_state *s, STREAM_TRAFFIC_TYPE type, uint64_t bytes_compressed, uint64_t bytes_uncompressed) {
178
+ // calculate the statistics for our dispatcher
179
+ s->thread.bytes_sent_by_type[type] += bytes_compressed;
180
+
181
+ s->thread.bytes_uncompressed += bytes_uncompressed;
182
+ s->thread.bytes_compressed += bytes_compressed;
183
+ s->thread.bytes_outstanding = cbuffer_next_unsafe(s->sbuf.cb, NULL);
184
+ s->thread.bytes_available = cbuffer_available_size_unsafe(s->sbuf.cb);
185
+ s->thread.buffer_ratio = (NETDATA_DOUBLE)(s->sbuf.cb->max_size - s->thread.bytes_available) * 100.0 / (NETDATA_DOUBLE)s->sbuf.cb->max_size;
186
+ replication_recalculate_buffer_used_ratio_unsafe(s);
187
+}
188
+
189
+// --------------------------------------------------------------------------------------------------------------------
190
+// opcodes
191
+
192
+void stream_sender_handle_op(struct stream_thread *sth, struct sender_state *s, struct stream_opcode *msg) {
193
+ ND_LOG_STACK lgs[] = {
194
+ ND_LOG_FIELD_STR(NDF_NIDL_NODE, s->host->hostname),
195
+ ND_LOG_FIELD_CB(NDF_DST_IP, stream_sender_log_dst_ip, s),
196
+ ND_LOG_FIELD_CB(NDF_DST_PORT, stream_sender_log_dst_port, s),
197
+ ND_LOG_FIELD_CB(NDF_DST_TRANSPORT, stream_sender_log_transport, s),
198
+ ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_sender_log_capabilities, s),
199
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
200
+ ND_LOG_FIELD_END(),
201
+ };
202
+ ND_LOG_STACK_PUSH(lgs);
203
+
204
+ if(msg->opcode & STREAM_OPCODE_SENDER_BUFFER_OVERFLOW) {
205
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_OVERFLOW);
206
+ errno_clear();
207
+ stream_sender_lock(s);
208
+ size_t buffer_size = s->sbuf.cb->size;
209
+ size_t buffer_max_size = s->sbuf.cb->max_size;
210
+ size_t buffer_available = cbuffer_available_size_unsafe(s->sbuf.cb);
211
+ stream_sender_unlock(s);
212
+ nd_log(NDLS_DAEMON, NDLP_ERR,
213
+ "STREAM SEND[%zu] %s [to %s]: send buffer is full (buffer size %zu, max %zu, available %zu). "
214
+ "Restarting connection.",
215
+ sth->id, rrdhost_hostname(s->host), s->connected_to,
216
+ buffer_size, buffer_max_size, buffer_available);
217
+
218
+ stream_sender_move_running_to_connector_or_remove(
219
+ sth, s, STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_SENDER_SEND_BUFFER, true);
220
+ return;
221
+ }
222
+
223
+ if(msg->opcode & STREAM_OPCODE_SENDER_STOP_RECEIVER_LEFT) {
224
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_RECEIVER_LEFT);
225
+ stream_sender_move_running_to_connector_or_remove(
226
+ sth, s, STREAM_HANDSHAKE_DISCONNECT_RECEIVER_LEFT, false);
227
+ return;
228
+ }
229
+
230
+ if(msg->opcode & STREAM_OPCODE_SENDER_RECONNECT_WITHOUT_COMPRESSION) {
231
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_COMPRESSION_ERROR);
232
+ errno_clear();
233
+ nd_log(NDLS_DAEMON, NDLP_ERR,
234
+ "STREAM SEND[%zu] %s [send to %s]: restarting connection without compression.",
235
+ sth->id, rrdhost_hostname(s->host), s->connected_to);
236
+
237
+ stream_sender_move_running_to_connector_or_remove(
238
+ sth, s, STREAM_HANDSHAKE_DISCONNECT_NOT_SUFFICIENT_SENDER_COMPRESSION_FAILED, true);
239
+ return;
240
+ }
241
+
242
+ if(msg->opcode & STREAM_OPCODE_SENDER_STOP_HOST_CLEANUP) {
243
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_HOST_CLEANUP);
244
+ stream_sender_move_running_to_connector_or_remove(
245
+ sth, s, STREAM_HANDSHAKE_DISCONNECT_HOST_CLEANUP, false);
246
+ return;
247
+ }
248
+
249
+ nd_log(NDLS_DAEMON, NDLP_ERR,
250
+ "STREAM SEND[%zu]: invalid msg id %u", sth->id, (unsigned)msg->opcode);
251
+}
252
+
253
+
254
+// --------------------------------------------------------------------------------------------------------------------
255
+
256
+void stream_sender_move_queue_to_running_unsafe(struct stream_thread *sth) {
257
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
258
+
259
+ // process the queue
260
+ Word_t idx = 0;
261
+ for(struct sender_state *s = SENDERS_FIRST(&sth->queue.senders, &idx);
262
+ s;
263
+ s = SENDERS_NEXT(&sth->queue.senders, &idx)) {
264
+ worker_is_busy(WORKER_STREAM_JOB_DEQUEUE);
265
+
266
+ SENDERS_DEL(&sth->queue.senders, (Word_t)s);
267
+
268
+ ND_LOG_STACK lgs[] = {
269
+ ND_LOG_FIELD_STR(NDF_NIDL_NODE, s->host->hostname),
270
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
271
+ ND_LOG_FIELD_END(),
272
+ };
273
+ ND_LOG_STACK_PUSH(lgs);
274
+
275
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
276
+ "STREAM SEND[%zu] [%s]: moving host from dispatcher queue to dispatcher running...",
277
+ sth->id, rrdhost_hostname(s->host));
278
+
279
+ internal_fatal(SENDERS_GET(&sth->snd.senders, (Word_t)s) != NULL, "Sender already exists in senders list");
280
+ SENDERS_SET(&sth->snd.senders, (Word_t)s, s);
281
+
282
+ stream_sender_lock(s);
283
+ s->thread.meta.type = POLLFD_TYPE_SENDER;
284
+ s->thread.meta.s = s;
285
+ if(!nd_poll_add(sth->run.ndpl, s->sock.fd, ND_POLL_READ, &s->thread.meta))
286
+ internal_fatal(true, "Failed to add sender socket to nd_poll()");
287
+
288
+ s->thread.msg.thread_slot = (int32_t)sth->id;
289
+ s->thread.msg.session = os_random32();
290
+ s->thread.msg.sender = s;
291
+
292
+ s->host->stream.snd.status.tid = gettid_cached();
293
+ s->host->stream.snd.status.connections++;
294
+ s->last_state_since_t = now_realtime_sec();
295
+
296
+ stream_sender_thread_data_reset_unsafe(s);
297
+ stream_sender_unlock(s);
298
+
299
+ stream_sender_on_ready_to_dispatch(s);
300
+ }
301
+}
302
+
303
+void stream_sender_remove(struct sender_state *s) {
304
+ // THIS FUNCTION IS USED BY THE CONNECTOR TOO
305
+ // when it gives up on a certain node
306
+
307
+ nd_log(NDLS_DAEMON, NDLP_NOTICE,
308
+ "STREAM SEND [%s]: streaming sender removed host: %s",
309
+ rrdhost_hostname(s->host), stream_handshake_error_to_string(s->exit.reason));
310
+
311
+ stream_sender_lock(s);
312
+
313
+ __atomic_store_n(&s->exit.shutdown, false, __ATOMIC_RELAXED);
314
+ rrdhost_flag_clear(s->host,
315
+ RRDHOST_FLAG_STREAM_SENDER_ADDED | RRDHOST_FLAG_STREAM_SENDER_CONNECTED |
316
+ RRDHOST_FLAG_STREAM_SENDER_READY_4_METRICS);
317
+
318
+ s->last_state_since_t = now_realtime_sec();
319
+ stream_parent_set_disconnect_reason(s->host->stream.snd.parents.current, s->exit.reason, s->last_state_since_t);
320
+ s->connector.id = -1;
321
+
322
+ stream_sender_unlock(s);
323
+
324
+ rrdhost_stream_parents_reset(s->host, STREAM_HANDSHAKE_EXITING);
325
+
326
+#ifdef NETDATA_LOG_STREAM_SENDER
327
+ if (s->stream_log_fp) {
328
+ fclose(s->stream_log_fp);
329
+ s->stream_log_fp = NULL;
330
+ }
331
+#endif
332
+}
333
+
334
+static void stream_sender_move_running_to_connector_or_remove(struct stream_thread *sth, struct sender_state *s, STREAM_HANDSHAKE reason, bool reconnect) {
335
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
336
+
337
+ internal_fatal(SENDERS_GET(&sth->snd.senders, (Word_t)s) == NULL, "Sender to be removed is not in the list of senders");
338
+ SENDERS_DEL(&sth->snd.senders, (Word_t)s);
339
+ if(!nd_poll_del(sth->run.ndpl, s->sock.fd))
340
+ internal_fatal(true, "Failed to remove sender socket from nd_poll()");
341
+
342
+ // clear this flag asap, to stop other threads from pushing metrics for this node
343
+ rrdhost_flag_clear(s->host, RRDHOST_FLAG_STREAM_SENDER_CONNECTED | RRDHOST_FLAG_STREAM_SENDER_READY_4_METRICS);
344
+
345
+ // clear these asap, to make sender_commit() stop processing data for this host
346
+ stream_sender_lock(s);
347
+
348
+ s->thread.msg.session = 0;
349
+ s->thread.msg.sender = NULL;
350
+
351
+ s->host->stream.snd.status.tid = 0;
352
+ stream_sender_unlock(s);
353
+
354
+ nd_log(NDLS_DAEMON, NDLP_NOTICE,
355
+ "STREAM SEND [%s]: disconnected from parent, reason: %s",
356
+ rrdhost_hostname(s->host), stream_handshake_error_to_string(reason));
357
+
358
+ nd_sock_close(&s->sock);
359
+
360
+ stream_parent_set_disconnect_reason(s->host->stream.snd.parents.current, reason, now_realtime_sec());
361
+ stream_sender_on_disconnect(s);
362
+
363
+ bool should_remove = !reconnect || stream_connector_is_signaled_to_stop(s);
364
+
365
+ stream_thread_node_removed(s->host);
366
+
367
+ if (should_remove)
368
+ stream_sender_remove(s);
369
+ else
370
+ stream_connector_requeue(s);
371
+}
372
+
373
+void stream_sender_check_all_nodes_from_poll(struct stream_thread *sth) {
374
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
375
+
376
+ usec_t now_ut = now_monotonic_usec();
377
+ time_t now_s = (time_t)(now_ut / USEC_PER_SEC);
378
+
379
+ size_t bytes_uncompressed = 0;
380
+ size_t bytes_compressed = 0;
381
+ NETDATA_DOUBLE buffer_ratio = 0.0;
382
+
383
+ Word_t idx = 0;
384
+ for(struct sender_state *s = SENDERS_FIRST(&sth->snd.senders, &idx);
385
+ s;
386
+ s = SENDERS_NEXT(&sth->snd.senders, &idx)) {
387
+
388
+ // If the TCP window never opened, then something is wrong, restart connection
389
+ if(unlikely(now_s - s->last_traffic_seen_t > stream_send.parents.timeout_s &&
390
+ !stream_sender_pending_replication_requests(s) &&
391
+ !stream_sender_replicating_charts(s)
392
+ )) {
393
+
394
+ ND_LOG_STACK lgs[] = {
395
+ ND_LOG_FIELD_STR(NDF_NIDL_NODE, s->host->hostname),
396
+ ND_LOG_FIELD_CB(NDF_DST_IP, stream_sender_log_dst_ip, s),
397
+ ND_LOG_FIELD_CB(NDF_DST_PORT, stream_sender_log_dst_port, s),
398
+ ND_LOG_FIELD_CB(NDF_DST_TRANSPORT, stream_sender_log_transport, s),
399
+ ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_sender_log_capabilities, s),
400
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
401
+ ND_LOG_FIELD_END(),
402
+ };
403
+ ND_LOG_STACK_PUSH(lgs);
404
+
405
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_TIMEOUT);
406
+
407
+ nd_log(NDLS_DAEMON, NDLP_ERR,
408
+ "STREAM SEND[%zu] %s [send to %s]: could not send metrics for %ld seconds - closing connection - "
409
+ "we have sent %zu bytes on this connection via %zu send attempts.",
410
+ sth->id, rrdhost_hostname(s->host), s->connected_to, stream_send.parents.timeout_s,
411
+ s->thread.bytes_sent, s->thread.sends);
412
+
413
+ stream_sender_move_running_to_connector_or_remove(sth, s, STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_TIMEOUT, true);
414
+ continue;
415
+ }
416
+
417
+ stream_sender_lock(s);
418
+ {
419
+ bytes_compressed += s->thread.bytes_compressed;
420
+ bytes_uncompressed += s->thread.bytes_uncompressed;
421
+ uint64_t outstanding = s->thread.bytes_outstanding;
422
+ if (s->thread.buffer_ratio > buffer_ratio)
423
+ buffer_ratio = s->thread.buffer_ratio;
424
+
425
+ if(!nd_poll_upd(sth->run.ndpl, s->sock.fd, ND_POLL_READ | (outstanding ? ND_POLL_WRITE : 0), &s->thread.meta))
426
+ internal_fatal(true, "Failed to update sender socket in nd_poll()");
427
+ }
428
+ stream_sender_unlock(s);
429
+ }
430
+
431
+ if (bytes_compressed && bytes_uncompressed) {
432
+ NETDATA_DOUBLE compression_ratio = 100.0 - ((NETDATA_DOUBLE)bytes_compressed * 100.0 / (NETDATA_DOUBLE)bytes_uncompressed);
433
+ worker_set_metric(WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO, compression_ratio);
434
+ }
435
+
436
+ worker_set_metric(WORKER_SENDER_JOB_BYTES_UNCOMPRESSED, (NETDATA_DOUBLE)bytes_uncompressed);
437
+ worker_set_metric(WORKER_SENDER_JOB_BYTES_COMPRESSED, (NETDATA_DOUBLE)bytes_compressed);
438
+ worker_set_metric(WORKER_SENDER_JOB_BUFFER_RATIO, buffer_ratio);
439
+}
440
+
441
+void stream_sender_process_poll_events(struct stream_thread *sth, struct sender_state *s, nd_poll_event_t events, time_t now_s) {
442
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
443
+
444
+ ND_LOG_STACK lgs[] = {
445
+ ND_LOG_FIELD_STR(NDF_NIDL_NODE, s->host->hostname),
446
+ ND_LOG_FIELD_CB(NDF_DST_IP, stream_sender_log_dst_ip, s),
447
+ ND_LOG_FIELD_CB(NDF_DST_PORT, stream_sender_log_dst_port, s),
448
+ ND_LOG_FIELD_CB(NDF_DST_TRANSPORT, stream_sender_log_transport, s),
449
+ ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_sender_log_capabilities, s),
450
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
451
+ ND_LOG_FIELD_END(),
452
+ };
453
+ ND_LOG_STACK_PUSH(lgs);
454
+
455
+ if(unlikely(events & ND_POLL_ERROR)) {
456
+ // we have errors on this socket
457
+
458
+ worker_is_busy(WORKER_STREAM_JOB_SOCKET_ERROR);
459
+
460
+ char *error = "unknown error";
461
+
462
+ if (events & ND_POLL_ERROR)
463
+ error = "socket reports errors";
464
+ else if (events & ND_POLL_HUP)
465
+ error = "connection closed by remote end (HUP)";
466
+ else if (events & ND_POLL_INVALID)
467
+ error = "connection is invalid";
468
+
469
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SOCKET_ERROR);
470
+
471
+ nd_log(NDLS_DAEMON, NDLP_ERR,
472
+ "STREAM SEND[%zu] %s [send to %s]: %s restarting connection - %zu bytes transmitted.",
473
+ sth->id, rrdhost_hostname(s->host), s->connected_to, error, s->thread.bytes_sent);
474
+
475
+ stream_sender_move_running_to_connector_or_remove(sth, s, STREAM_HANDSHAKE_DISCONNECT_SOCKET_ERROR, true);
476
+ return;
477
+ }
478
+
479
+ if(events & ND_POLL_WRITE) {
480
+ // we can send data on this socket
481
+
482
+ worker_is_busy(WORKER_STREAM_JOB_SOCKET_SEND);
483
+
484
+ bool disconnect = false;
485
+ stream_sender_lock(s);
486
+ {
487
+ char *chunk;
488
+ size_t outstanding = cbuffer_next_unsafe(s->sbuf.cb, &chunk);
489
+ ssize_t bytes = nd_sock_send_nowait(&s->sock, chunk, outstanding);
490
+ if (likely(bytes > 0)) {
491
+ cbuffer_remove_unsafe(s->sbuf.cb, bytes);
492
+ stream_sender_thread_data_sent_data_unsafe(s, bytes);
493
+ s->last_traffic_seen_t = now_s;
494
+ sth->snd.bytes_sent += bytes;
495
+
496
+ if(!s->thread.bytes_outstanding) {
497
+ // we sent them all - remove POLLOUT
498
+ if(!nd_poll_upd(sth->run.ndpl, s->sock.fd, ND_POLL_READ, &s->thread.meta))
499
+ internal_fatal(true, "Failed to update sender socket in nd_poll()");
500
+
501
+ // recreate the circular buffer if we have to
502
+ stream_sender_cbuffer_recreate_timed_unsafe(s, now_s, false);
503
+ }
504
+ }
505
+ else if (bytes < 0 && errno != EWOULDBLOCK && errno != EAGAIN && errno != EINTR)
506
+ disconnect = true;
507
+ }
508
+ stream_sender_unlock(s);
509
+
510
+ if(disconnect) {
511
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR);
512
+ nd_log(NDLS_DAEMON, NDLP_ERR,
513
+ "STREAM SEND[%zu] %s [send to %s]: failed to send metrics - restarting connection - "
514
+ "we have sent %zu bytes on this connection.",
515
+ sth->id, rrdhost_hostname(s->host), s->connected_to, s->thread.bytes_sent);
516
+ stream_sender_move_running_to_connector_or_remove(
517
+ sth, s, STREAM_HANDSHAKE_DISCONNECT_SOCKET_WRITE_FAILED, true);
518
+ return;
519
+ }
520
+ }
521
+
522
+ if(events & POLLIN) {
523
+ // we can receive data from this socket
524
+
525
+ worker_is_busy(WORKER_STREAM_JOB_SOCKET_RECEIVE);
526
+ ssize_t bytes = nd_sock_revc_nowait(&s->sock, s->rbuf.b + s->rbuf.read_len, sizeof(s->rbuf.b) - s->rbuf.read_len - 1);
527
+ if (bytes > 0) {
528
+ s->rbuf.read_len += bytes;
529
+ s->last_traffic_seen_t = now_s;
530
+ sth->snd.bytes_received += bytes;
531
+ }
532
+ else if (bytes == 0 || errno == ECONNRESET) {
533
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_PARENT_CLOSED);
534
+ nd_log(NDLS_DAEMON, NDLP_ERR,
535
+ "STREAM SEND[%zu] %s [send to %s]: connection (fd %d) closed by far end.",
536
+ sth->id, rrdhost_hostname(s->host), s->connected_to, s->sock.fd);
537
+ stream_sender_move_running_to_connector_or_remove(
538
+ sth, s, STREAM_HANDSHAKE_DISCONNECT_SOCKET_CLOSED_BY_PARENT, true);
539
+ return;
540
+ }
541
+ else if (bytes < 0 && errno != EWOULDBLOCK && errno != EAGAIN && errno != EINTR) {
542
+ worker_is_busy(WORKER_SENDER_JOB_DISCONNECT_RECEIVE_ERROR);
543
+ nd_log(NDLS_DAEMON, NDLP_ERR,
544
+ "STREAM SEND[%zu] %s [send to %s]: error during receive (%zd, on fd %d) - restarting connection.",
545
+ sth->id, rrdhost_hostname(s->host), s->connected_to, bytes, s->sock.fd);
546
+ stream_sender_move_running_to_connector_or_remove(
547
+ sth, s, STREAM_HANDSHAKE_DISCONNECT_SOCKET_READ_FAILED, true);
548
+ return;
549
+ }
550
+ }
551
+
552
+ if(unlikely(s->rbuf.read_len)) {
553
+ worker_is_busy(WORKER_SENDER_JOB_EXECUTE);
554
+ stream_sender_execute_commands(s);
555
+ }
556
+}
557
+
558
+void stream_sender_cleanup(struct stream_thread *sth) {
559
+ // stop all hosts
560
+ Word_t idx = 0;
561
+ for(struct sender_state *s = SENDERS_FIRST(&sth->snd.senders, &idx);
562
+ s;
563
+ s = SENDERS_NEXT(&sth->snd.senders, &idx)) {
564
+
565
+ ND_LOG_STACK lgs[] = {
566
+ ND_LOG_FIELD_STR(NDF_NIDL_NODE, s->host->hostname),
567
+ ND_LOG_FIELD_CB(NDF_DST_IP, stream_sender_log_dst_ip, s),
568
+ ND_LOG_FIELD_CB(NDF_DST_PORT, stream_sender_log_dst_port, s),
569
+ ND_LOG_FIELD_CB(NDF_DST_TRANSPORT, stream_sender_log_transport, s),
570
+ ND_LOG_FIELD_CB(NDF_SRC_CAPABILITIES, stream_sender_log_capabilities, s),
571
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
572
+ ND_LOG_FIELD_END(),
573
+ };
574
+ ND_LOG_STACK_PUSH(lgs);
575
+
576
+ stream_sender_move_running_to_connector_or_remove(sth, s, STREAM_HANDSHAKE_DISCONNECT_SHUTDOWN, false);
577
+ }
578
+
579
+ // cleanup
580
+ SENDERS_FREE(&sth->snd.senders, NULL);
581
+}
582
+
src/streaming/stream-thread.c
new
+573
@@ -0,0 +1,573 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "stream-thread.h"
4
+
5
+struct stream_thread_globals stream_thread_globals = {
6
+ .assign = {
7
+ .spinlock = NETDATA_SPINLOCK_INITIALIZER,
8
+ }
9
+};
10
+
11
+// --------------------------------------------------------------------------------------------------------------------
12
+// pipe messages
13
+
14
+static void stream_thread_handle_op(struct stream_thread *sth, struct stream_opcode *msg) {
15
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
16
+
17
+ sth->messages.processed++;
18
+
19
+ struct sender_state *s = msg->sender ? SENDERS_GET(&sth->snd.senders, (Word_t)msg->sender) : NULL;
20
+
21
+ if (msg->session && // there is a session
22
+ s && // there is a sender
23
+ (size_t)msg->thread_slot == sth->id) // same thread
24
+ {
25
+ if(msg->opcode & STREAM_OPCODE_SENDER_POLLOUT) {
26
+ if(!nd_poll_upd(sth->run.ndpl, s->sock.fd, ND_POLL_READ|ND_POLL_WRITE, &s->thread.meta))
27
+ internal_fatal(true, "Failed to update sender socket in nd_poll()");
28
+ msg->opcode &= ~(STREAM_OPCODE_SENDER_POLLOUT);
29
+ }
30
+
31
+ if(msg->opcode)
32
+ stream_sender_handle_op(sth, s, msg);
33
+ }
34
+ else {
35
+ // this may happen if we receive a POLLOUT opcode, but the sender has been disconnected
36
+ nd_log(NDLS_DAEMON, NDLP_DEBUG, "STREAM THREAD[%zu]: OPCODE %u ignored.", sth->id, (unsigned)msg->opcode);
37
+ }
38
+}
39
+
40
+void stream_sender_send_msg_to_dispatcher(struct sender_state *s, struct stream_opcode msg) {
41
+ if (!msg.session || !msg.sender || !s)
42
+ return;
43
+
44
+ internal_fatal(msg.sender != s, "the sender pointer in the message does not match this sender");
45
+
46
+ struct stream_thread *sth = stream_thread_by_slot_id(msg.thread_slot);
47
+ if(!sth) {
48
+ internal_fatal(true,
49
+ "STREAM SEND[x] [%s] thread pointer in the opcode message does not match the expected",
50
+ rrdhost_hostname(s->host));
51
+ return;
52
+ }
53
+
54
+ bool send_pipe_msg = false;
55
+
56
+ // check if we can execute the message now
57
+ if(sth->tid == gettid_cached()) {
58
+ // we are running at the dispatcher thread
59
+ // no need for locks or queuing
60
+ sth->messages.bypassed++;
61
+ stream_thread_handle_op(sth, &msg);
62
+ return;
63
+ }
64
+
65
+ // add it to the message queue of the thread
66
+ spinlock_lock(&sth->messages.spinlock);
67
+ {
68
+ sth->messages.added++;
69
+ if (s->thread.msg_slot >= sth->messages.used || sth->messages.array[s->thread.msg_slot].sender != s) {
70
+ if (unlikely(sth->messages.used >= sth->messages.size)) {
71
+ // this should never happen, but let's find the root cause
72
+
73
+ if (!sth->messages.size) {
74
+ // we are exiting
75
+ spinlock_unlock(&sth->messages.spinlock);
76
+ return;
77
+ }
78
+
79
+ // try to find us in the list
80
+ for (size_t i = 0; i < sth->messages.size; i++) {
81
+ if (sth->messages.array[i].sender == s) {
82
+ s->thread.msg_slot = i;
83
+ sth->messages.array[s->thread.msg_slot].opcode |= msg.opcode;
84
+ spinlock_unlock(&sth->messages.spinlock);
85
+ internal_fatal(true, "the dispatcher message queue is full, but this sender is already on slot %zu", i);
86
+ return;
87
+ }
88
+ }
89
+
90
+ fatal("the dispatcher message queue is full, but this should never happen");
91
+ }
92
+
93
+ // let's use a new slot
94
+ send_pipe_msg = !sth->messages.used; // write to the pipe, only when the queue was empty before this msg
95
+ s->thread.msg_slot = sth->messages.used++;
96
+ sth->messages.array[s->thread.msg_slot] = msg;
97
+ }
98
+ else
99
+ // the existing slot is good
100
+ sth->messages.array[s->thread.msg_slot].opcode |= msg.opcode;
101
+ }
102
+ spinlock_unlock(&sth->messages.spinlock);
103
+
104
+ // signal the streaming thread to wake up and process messages
105
+ if(send_pipe_msg &&
106
+ sth->pipe.fds[PIPE_WRITE] != -1 &&
107
+ write(sth->pipe.fds[PIPE_WRITE], " ", 1) != 1) {
108
+ nd_log_limit_static_global_var(erl, 1, 1 * USEC_PER_MS);
109
+ nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR,
110
+ "STREAM SEND [%s]: cannot write to signal pipe",
111
+ rrdhost_hostname(s->host));
112
+ }
113
+}
114
+
115
+static void stream_thread_read_pipe_messages(struct stream_thread *sth) {
116
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
117
+
118
+ if(read(sth->pipe.fds[PIPE_READ], sth->pipe.buffer, sth->pipe.size * sizeof(*sth->pipe.buffer)) <= 0)
119
+ nd_log(NDLS_DAEMON, NDLP_ERR, "STREAM THREAD[%zu]: signal pipe read error", sth->id);
120
+
121
+ size_t used = 0;
122
+ spinlock_lock(&sth->messages.spinlock);
123
+ if(sth->messages.used) {
124
+ used = sth->messages.used;
125
+ memcpy(sth->messages.copy, sth->messages.array, used * sizeof(*sth->messages.copy));
126
+ sth->messages.used = 0;
127
+ }
128
+ spinlock_unlock(&sth->messages.spinlock);
129
+
130
+ for(size_t i = 0; i < used ;i++) {
131
+ struct stream_opcode *msg = &sth->messages.copy[i];
132
+ stream_thread_handle_op(sth, msg);
133
+ }
134
+}
135
+
136
+// --------------------------------------------------------------------------------------------------------------------
137
+
138
+static int set_pipe_size(int pipe_fd, int new_size) {
139
+ int default_size = new_size;
140
+ int result = new_size;
141
+
142
+#ifdef F_GETPIPE_SZ
143
+ // get the current size of the pipe
144
+ result = fcntl(pipe_fd, F_GETPIPE_SZ);
145
+ if(result > 0)
146
+ default_size = result;
147
+#endif
148
+
149
+#ifdef F_SETPIPE_SZ
150
+ // set the new size to the pipe
151
+ if(result <= new_size) {
152
+ result = fcntl(pipe_fd, F_SETPIPE_SZ, new_size);
153
+ if (result <= 0)
154
+ return default_size;
155
+ }
156
+#endif
157
+
158
+ // we return either:
159
+ // 1. the new_size (after setting it)
160
+ // 2. the current size (if we can't set it, but we can read it)
161
+ // 3. the new_size (without setting it when we can't read the current size)
162
+ return result; // Returns the new pipe size
163
+}
164
+
165
+// --------------------------------------------------------------------------------------------------------------------
166
+
167
+static void stream_thread_messages_resize_unsafe(struct stream_thread *sth) {
168
+ internal_fatal(sth->tid != gettid_cached(), "Function %s() should only be used by the dispatcher thread", __FUNCTION__ );
169
+
170
+ if(sth->nodes_count >= sth->messages.size) {
171
+ size_t new_size = sth->messages.size ? sth->messages.size * 2 : 2;
172
+ sth->messages.array = reallocz(sth->messages.array, new_size * sizeof(*sth->messages.array));
173
+ sth->messages.copy = reallocz(sth->messages.copy, new_size * sizeof(*sth->messages.copy));
174
+ sth->messages.size = new_size;
175
+ }
176
+}
177
+
178
+// --------------------------------------------------------------------------------------------------------------------
179
+
180
+static bool stream_thread_process_poll_slot(struct stream_thread *sth, nd_poll_result_t *ev, time_t now_s, size_t *replay_entries) {
181
+ struct pollfd_meta *m = ev->data;
182
+ internal_fatal(!m, "Failed to get meta from event");
183
+
184
+ switch(m->type) {
185
+ case POLLFD_TYPE_SENDER: {
186
+ struct sender_state *s = m->s;
187
+ internal_fatal(SENDERS_GET(&sth->snd.senders, (Word_t)s) == NULL, "Sender is not found in the senders list");
188
+ stream_sender_process_poll_events(sth, s, ev->events, now_s);
189
+ *replay_entries += dictionary_entries(s->replication.requests);
190
+ break;
191
+ }
192
+
193
+ case POLLFD_TYPE_RECEIVER: {
194
+ struct receiver_state *rpt = m->rpt;
195
+ internal_fatal(RECEIVERS_GET(&sth->rcv.receivers, (Word_t)rpt) == NULL, "Receiver is not found in the receiver list");
196
+ stream_receive_process_poll_events(sth, rpt, ev->events, now_s);
197
+ break;
198
+ }
199
+
200
+ case POLLFD_TYPE_PIPE:
201
+ if (likely(ev->events & ND_POLL_READ)) {
202
+ worker_is_busy(WORKER_SENDER_JOB_PIPE_READ);
203
+ stream_thread_read_pipe_messages(sth);
204
+ }
205
+ else if(unlikely(ev->events & ND_POLL_ERROR)) {
206
+ // we have errors on this pipe
207
+ nd_log(NDLS_DAEMON, NDLP_ERR,
208
+ "STREAM THREAD[%zu]: got errors on pipe - exiting to be restarted.", sth->id);
209
+ return true;
210
+ }
211
+ break;
212
+
213
+ case POLLFD_TYPE_EMPTY:
214
+ // should never happen - but make sure it never happens again
215
+ internal_fatal(true, "What is this?");
216
+ break;
217
+ }
218
+
219
+ return false;
220
+}
221
+
222
+void *stream_thread(void *ptr) {
223
+ struct stream_thread *sth = ptr;
224
+
225
+ worker_register("STREAM");
226
+
227
+ // stream thread main event loop
228
+ worker_register_job_name(WORKER_STREAM_JOB_LIST, "list");
229
+ worker_register_job_name(WORKER_STREAM_JOB_DEQUEUE, "dequeue");
230
+ worker_register_job_name(WORKER_STREAM_JOB_PREP, "prep");
231
+ worker_register_job_name(WORKER_STREAM_JOB_POLL_ERROR, "poll error");
232
+ worker_register_job_name(WORKER_SENDER_JOB_PIPE_READ, "pipe read");
233
+
234
+ // both sender and receiver
235
+ worker_register_job_name(WORKER_STREAM_JOB_SOCKET_RECEIVE, "receive");
236
+ worker_register_job_name(WORKER_STREAM_JOB_SOCKET_SEND, "send");
237
+ worker_register_job_name(WORKER_STREAM_JOB_SOCKET_ERROR, "sock error");
238
+
239
+ // receiver
240
+ worker_register_job_name(WORKER_STREAM_JOB_COMPRESS, "compress");
241
+ worker_register_job_name(WORKER_STREAM_JOB_DECOMPRESS, "decompress");
242
+
243
+ // sender
244
+ worker_register_job_name(WORKER_SENDER_JOB_EXECUTE, "execute");
245
+ worker_register_job_name(WORKER_SENDER_JOB_EXECUTE_REPLAY, "replay");
246
+ worker_register_job_name(WORKER_SENDER_JOB_EXECUTE_FUNCTION, "function");
247
+ worker_register_job_name(WORKER_SENDER_JOB_EXECUTE_META, "meta");
248
+
249
+ // disconnection reasons
250
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_OVERFLOW, "disconnect overflow");
251
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_TIMEOUT, "disconnect timeout");
252
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_SOCKET_ERROR, "disconnect socket error");
253
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_PARENT_CLOSED, "disconnect parent closed");
254
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_RECEIVE_ERROR, "disconnect receive error");
255
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR, "disconnect send error");
256
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_COMPRESSION_ERROR, "disconnect compression error");
257
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_RECEIVER_LEFT, "disconnect receiver left");
258
+ worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_HOST_CLEANUP, "disconnect host cleanup");
259
+
260
+ // metrics
261
+ worker_register_job_custom_metric(WORKER_STREAM_METRIC_NODES,
262
+ "nodes", "nodes",
263
+ WORKER_METRIC_ABSOLUTE);
264
+
265
+ worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_READ,
266
+ "receiver received bytes", "bytes/s",
267
+ WORKER_METRIC_INCREMENT);
268
+
269
+ worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED,
270
+ "receiver received uncompressed bytes", "bytes/s",
271
+ WORKER_METRIC_INCREMENT);
272
+
273
+ worker_register_job_custom_metric(WORKER_RECEIVER_JOB_REPLICATION_COMPLETION,
274
+ "receiver replication completion", "%",
275
+ WORKER_METRIC_ABSOLUTE);
276
+
277
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_BUFFER_RATIO,
278
+ "sender used buffer ratio", "%",
279
+ WORKER_METRIC_ABSOLUTE);
280
+
281
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_RECEIVED,
282
+ "sender bytes received", "bytes/s",
283
+ WORKER_METRIC_INCREMENT);
284
+
285
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_SENT,
286
+ "sender bytes sent", "bytes/s",
287
+ WORKER_METRIC_INCREMENT);
288
+
289
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_COMPRESSED,
290
+ "sender bytes compressed", "bytes/s",
291
+ WORKER_METRIC_INCREMENTAL_TOTAL);
292
+
293
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_UNCOMPRESSED,
294
+ "sender bytes uncompressed", "bytes/s",
295
+ WORKER_METRIC_INCREMENTAL_TOTAL);
296
+
297
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO,
298
+ "sender cumulative compression savings ratio", "%",
299
+ WORKER_METRIC_ABSOLUTE);
300
+
301
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_REPLAY_DICT_SIZE,
302
+ "sender replication dict entries", "entries",
303
+ WORKER_METRIC_ABSOLUTE);
304
+
305
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_MESSAGES,
306
+ "ops processed", "messages",
307
+ WORKER_METRIC_INCREMENTAL_TOTAL);
308
+
309
+ if(pipe(sth->pipe.fds) != 0) {
310
+ nd_log(NDLS_DAEMON, NDLP_ERR, "STREAM THREAD[%zu]: cannot create required pipe.", sth->id);
311
+ sth->pipe.fds[PIPE_READ] = -1;
312
+ sth->pipe.fds[PIPE_WRITE] = -1;
313
+ return NULL;
314
+ }
315
+
316
+ sth->tid = gettid_cached();
317
+
318
+ sth->pipe.size = set_pipe_size(sth->pipe.fds[PIPE_READ], 65536 * sizeof(*sth->pipe.buffer)) / sizeof(*sth->pipe.buffer);
319
+ sth->pipe.buffer = mallocz(sth->pipe.size * sizeof(*sth->pipe.buffer));
320
+
321
+ usec_t last_check_all_nodes_ut = 0;
322
+ usec_t last_dequeue_ut = 0;
323
+
324
+ sth->run.pipe = (struct pollfd_meta){
325
+ .type = POLLFD_TYPE_PIPE,
326
+ };
327
+ sth->run.ndpl = nd_poll_create();
328
+ if(!sth->run.ndpl)
329
+ fatal("Cannot create nd_poll()");
330
+
331
+ if(!nd_poll_add(sth->run.ndpl, sth->pipe.fds[PIPE_READ], ND_POLL_READ, &sth->run.pipe))
332
+ internal_fatal(true, "Failed to add pipe to nd_poll()");
333
+
334
+ bool exit_thread = false;
335
+ size_t replay_entries = 0;
336
+ sth->snd.bytes_received = 0;
337
+ sth->snd.bytes_sent = 0;
338
+
339
+ while(!exit_thread && !nd_thread_signaled_to_cancel() && service_running(SERVICE_STREAMING)) {
340
+ usec_t now_ut = now_monotonic_usec();
341
+
342
+ if(now_ut - last_dequeue_ut >= 100 * USEC_PER_MS) {
343
+ worker_is_busy(WORKER_STREAM_JOB_DEQUEUE);
344
+
345
+ // move any pending hosts in the inbound queue, to the running list
346
+ spinlock_lock(&sth->queue.spinlock);
347
+ stream_thread_messages_resize_unsafe(sth);
348
+ stream_receiver_move_queue_to_running_unsafe(sth);
349
+ stream_sender_move_queue_to_running_unsafe(sth);
350
+ spinlock_unlock(&sth->queue.spinlock);
351
+ last_dequeue_ut = now_ut;
352
+ }
353
+
354
+ if(now_ut - last_check_all_nodes_ut >= USEC_PER_SEC) {
355
+ worker_is_busy(WORKER_STREAM_JOB_LIST);
356
+
357
+ // periodically check the entire list of nodes
358
+ // this detects unresponsive parents too (timeout)
359
+ stream_sender_check_all_nodes_from_poll(sth);
360
+ worker_set_metric(WORKER_SENDER_JOB_MESSAGES, (NETDATA_DOUBLE)(sth->messages.processed));
361
+ worker_set_metric(WORKER_STREAM_METRIC_NODES, (NETDATA_DOUBLE)sth->nodes_count);
362
+
363
+ worker_set_metric(WORKER_SENDER_JOB_BYTES_RECEIVED, (NETDATA_DOUBLE)sth->snd.bytes_received);
364
+ worker_set_metric(WORKER_SENDER_JOB_BYTES_SENT, (NETDATA_DOUBLE)sth->snd.bytes_sent);
365
+ worker_set_metric(WORKER_SENDER_JOB_REPLAY_DICT_SIZE, (NETDATA_DOUBLE)replay_entries);
366
+ replay_entries = 0;
367
+ sth->snd.bytes_received = 0;
368
+ sth->snd.bytes_sent = 0;
369
+
370
+ last_check_all_nodes_ut = now_ut;
371
+ }
372
+
373
+ worker_is_idle();
374
+
375
+ nd_poll_result_t ev;
376
+ int poll_rc = nd_poll_wait(sth->run.ndpl, 100, &ev);
377
+
378
+ worker_is_busy(WORKER_STREAM_JOB_PREP);
379
+
380
+ if (poll_rc == 0)
381
+ // nd_poll() timed out - just loop again
382
+ continue;
383
+
384
+ if(unlikely(poll_rc == -1)) {
385
+ // nd_poll() returned an error
386
+ internal_fatal(true, "nd_poll() failed");
387
+ worker_is_busy(WORKER_STREAM_JOB_POLL_ERROR);
388
+ nd_log_limit_static_thread_var(erl, 1, 1 * USEC_PER_MS);
389
+ nd_log_limit(&erl, NDLS_DAEMON, NDLP_ERR, "STREAM THREAD[%zu] poll() returned error", sth->id);
390
+ continue;
391
+ }
392
+
393
+ time_t now_s = now_monotonic_sec();
394
+
395
+ if(nd_thread_signaled_to_cancel() || !service_running(SERVICE_STREAMING))
396
+ break;
397
+
398
+ exit_thread = stream_thread_process_poll_slot(sth, &ev, now_s, &replay_entries);
399
+ }
400
+
401
+ // dequeue
402
+ spinlock_lock(&sth->queue.spinlock);
403
+ stream_sender_move_queue_to_running_unsafe(sth);
404
+ stream_receiver_move_queue_to_running_unsafe(sth);
405
+ spinlock_unlock(&sth->queue.spinlock);
406
+
407
+ // cleanup receiver and dispatcher
408
+ stream_sender_cleanup(sth);
409
+ stream_receiver_cleanup(sth);
410
+
411
+ // cleanup the thread structures
412
+ spinlock_lock(&sth->messages.spinlock);
413
+ freez(sth->messages.array);
414
+ sth->messages.array = NULL;
415
+ sth->messages.size = 0;
416
+ sth->messages.used = 0;
417
+ spinlock_unlock(&sth->messages.spinlock);
418
+
419
+ freez(sth->pipe.buffer);
420
+ sth->pipe.buffer = NULL;
421
+ sth->pipe.size = 0;
422
+
423
+ nd_poll_destroy(sth->run.ndpl);
424
+ sth->run.ndpl = NULL;
425
+
426
+ close(sth->pipe.fds[PIPE_READ]);
427
+ close(sth->pipe.fds[PIPE_WRITE]);
428
+ sth->pipe.fds[PIPE_READ] = -1;
429
+ sth->pipe.fds[PIPE_WRITE] = -1;
430
+
431
+ sth->thread = NULL;
432
+ sth->tid = 0;
433
+
434
+ worker_unregister();
435
+
436
+ return NULL;
437
+}
438
+
439
+// --------------------------------------------------------------------------------------------------------------------
440
+
441
+void stream_thread_node_queued(RRDHOST *host) {
442
+ spinlock_lock(&stream_thread_globals.assign.spinlock);
443
+ host->stream.refcount++;
444
+ internal_fatal(host->stream.refcount > 2, "invalid stream refcount %u (while adding node)", host->stream.refcount);
445
+ spinlock_unlock(&stream_thread_globals.assign.spinlock);
446
+}
447
+
448
+void stream_thread_node_removed(RRDHOST *host) {
449
+ spinlock_lock(&stream_thread_globals.assign.spinlock);
450
+ internal_fatal(!host->stream.refcount, "invalid stream refcount %u (while stopping node)", host->stream.refcount);
451
+
452
+ if(--host->stream.refcount == 0) {
453
+ struct stream_thread *sth = host->stream.thread;
454
+ sth->nodes_count--;
455
+ host->stream.thread = NULL;
456
+ }
457
+
458
+ spinlock_unlock(&stream_thread_globals.assign.spinlock);
459
+}
460
+
461
+static struct stream_thread *stream_thread_get_unsafe(RRDHOST *host) {
462
+ if(host->stream.thread)
463
+ return host->stream.thread;
464
+
465
+ if(!stream_thread_globals.assign.cores) {
466
+ stream_thread_globals.assign.cores = get_netdata_cpus() - 1;
467
+ if(stream_thread_globals.assign.cores < 4)
468
+ stream_thread_globals.assign.cores = 4;
469
+ else if(stream_thread_globals.assign.cores > STREAM_MAX_THREADS)
470
+ stream_thread_globals.assign.cores = STREAM_MAX_THREADS;
471
+ }
472
+
473
+ size_t selected_thread_slot = 0;
474
+ size_t min_nodes = stream_thread_globals.threads[0].nodes_count;
475
+ for(size_t i = 1; i < stream_thread_globals.assign.cores ; i++) {
476
+ if(stream_thread_globals.threads[i].nodes_count < min_nodes) {
477
+ selected_thread_slot = i;
478
+ min_nodes = stream_thread_globals.threads[i].nodes_count;
479
+ }
480
+ }
481
+
482
+ struct stream_thread *sth = host->stream.thread = &stream_thread_globals.threads[selected_thread_slot];
483
+ host->stream.refcount = 0;
484
+ sth->nodes_count++;
485
+
486
+ return host->stream.thread;
487
+}
488
+
489
+static struct stream_thread * stream_thread_assign_and_start(RRDHOST *host) {
490
+ spinlock_lock(&stream_thread_globals.assign.spinlock);
491
+
492
+ struct stream_thread *sth = stream_thread_get_unsafe(host);
493
+
494
+ if(!sth->thread) {
495
+ sth->id = (sth - stream_thread_globals.threads); // find the slot number
496
+ if(&stream_thread_globals.threads[sth->id] != sth)
497
+ fatal("STREAM THREAD[x] [%s]: thread id and slot do not match!", rrdhost_hostname(host));
498
+
499
+ sth->pipe.fds[PIPE_READ] = -1;
500
+ sth->pipe.fds[PIPE_WRITE] = -1;
501
+ spinlock_init(&sth->pipe.spinlock);
502
+ spinlock_init(&sth->queue.spinlock);
503
+ spinlock_init(&sth->messages.spinlock);
504
+ sth->messages.used = 0;
505
+
506
+ char tag[NETDATA_THREAD_TAG_MAX + 1];
507
+ snprintfz(tag, NETDATA_THREAD_TAG_MAX, THREAD_TAG_STREAM "[%zu]", sth->id);
508
+
509
+ sth->thread = nd_thread_create(tag, NETDATA_THREAD_OPTION_DEFAULT, stream_thread, sth);
510
+ if (!sth->thread)
511
+ nd_log_daemon(NDLP_ERR, "STREAM THREAD[%zu]: failed to create new thread for client.", sth->id);
512
+ }
513
+
514
+ spinlock_unlock(&stream_thread_globals.assign.spinlock);
515
+
516
+ return sth;
517
+}
518
+
519
+void stream_sender_add_to_connector_queue(RRDHOST *host) {
520
+ ND_LOG_STACK lgs[] = {
521
+ ND_LOG_FIELD_STR(NDF_NIDL_NODE, host->hostname),
522
+ ND_LOG_FIELD_UUID(NDF_MESSAGE_ID, &streaming_to_parent_msgid),
523
+ ND_LOG_FIELD_END(),
524
+ };
525
+ ND_LOG_STACK_PUSH(lgs);
526
+
527
+ stream_connector_init(host->sender);
528
+ rrdhost_stream_parent_ssl_init(host->sender);
529
+ stream_connector_add(host->sender);
530
+}
531
+
532
+void stream_receiver_add_to_queue(struct receiver_state *rpt) {
533
+ struct stream_thread *sth = stream_thread_assign_and_start(rpt->host);
534
+
535
+ stream_thread_node_queued(rpt->host);
536
+
537
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
538
+ "STREAM RECEIVE[%zu] [%s]: moving host to receiver queue...",
539
+ sth->id, rrdhost_hostname(rpt->host));
540
+
541
+ spinlock_lock(&sth->queue.spinlock);
542
+ internal_fatal(RECEIVERS_GET(&sth->queue.receivers, (Word_t)rpt) != NULL, "Receiver is already in the receivers queue");
543
+ RECEIVERS_SET(&sth->queue.receivers, (Word_t)rpt, rpt);
544
+ spinlock_unlock(&sth->queue.spinlock);
545
+}
546
+
547
+void stream_sender_add_to_queue(struct sender_state *s) {
548
+ struct stream_thread *sth = stream_thread_assign_and_start(s->host);
549
+
550
+ stream_thread_node_queued(s->host);
551
+
552
+ nd_log(NDLS_DAEMON, NDLP_DEBUG,
553
+ "STREAM THREAD[%zu] [%s]: moving host to dispatcher queue...",
554
+ sth->id, rrdhost_hostname(s->host));
555
+
556
+ spinlock_lock(&sth->queue.spinlock);
557
+ internal_fatal(SENDERS_GET(&sth->queue.senders, (Word_t)s) != NULL, "Sender is already in the senders queue");
558
+ SENDERS_SET(&sth->queue.senders, (Word_t)s, s);
559
+ spinlock_unlock(&sth->queue.spinlock);
560
+}
561
+
562
+void stream_threads_cancel(void) {
563
+ stream_connector_cancel_threads();
564
+ for(size_t i = 0; i < STREAM_MAX_THREADS ;i++)
565
+ nd_thread_signal_cancel(stream_thread_globals.threads[i].thread);
566
+}
567
+
568
+struct stream_thread *stream_thread_by_slot_id(size_t thread_slot) {
569
+ if(thread_slot < STREAM_MAX_THREADS && stream_thread_globals.threads[thread_slot].thread)
570
+ return &stream_thread_globals.threads[thread_slot];
571
+
572
+ return NULL;
573
+}
src/streaming/stream-thread.h
new
+202
@@ -0,0 +1,202 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_STREAM_THREAD_H
4
+#define NETDATA_STREAM_THREAD_H
5
+
6
+#include "libnetdata/libnetdata.h"
7
+
8
+struct stream_thread;
9
+struct pollfd_slotted {
10
+ struct stream_thread *sth;
11
+ int32_t slot;
12
+ int fd;
13
+};
14
+
15
+#define PFD_EMPTY (struct pollfd_slotted){ .sth = NULL, .fd = -1, .slot = -1, }
16
+
17
+typedef enum __attribute__((packed)) {
18
+ STREAM_OPCODE_NONE = 0,
19
+ STREAM_OPCODE_SENDER_POLLOUT = (1 << 0), // move traffic around as soon as possible
20
+ STREAM_OPCODE_SENDER_BUFFER_OVERFLOW = (1 << 1), // reconnect the node, it has buffer overflow
21
+ STREAM_OPCODE_SENDER_RECONNECT_WITHOUT_COMPRESSION = (1 << 2), // reconnect the node, but disable compression
22
+ STREAM_OPCODE_SENDER_STOP_RECEIVER_LEFT = (1 << 3), // disconnect the node, the receiver left
23
+ STREAM_OPCODE_SENDER_STOP_HOST_CLEANUP = (1 << 4), // disconnect the node, it is being de-allocated
24
+} STREAM_OPCODE;
25
+
26
+struct stream_opcode {
27
+ int32_t thread_slot; // the dispatcher id this message refers to
28
+ uint32_t session; // random number used to verify that the message the dispatcher receives is for this sender
29
+ STREAM_OPCODE opcode; // the actual message to be delivered
30
+ struct sender_state *sender;
31
+};
32
+
33
+// IMPORTANT: to add workers, you have to edit WORKER_PARSER_FIRST_JOB accordingly
34
+
35
+// stream thread events
36
+#define WORKER_STREAM_JOB_LIST (WORKER_PARSER_FIRST_JOB - 34)
37
+#define WORKER_STREAM_JOB_DEQUEUE (WORKER_PARSER_FIRST_JOB - 33)
38
+#define WORKER_STREAM_JOB_PREP (WORKER_PARSER_FIRST_JOB - 32)
39
+#define WORKER_STREAM_JOB_POLL_ERROR (WORKER_PARSER_FIRST_JOB - 31)
40
+#define WORKER_SENDER_JOB_PIPE_READ (WORKER_PARSER_FIRST_JOB - 30)
41
+
42
+// socket operations
43
+#define WORKER_STREAM_JOB_SOCKET_RECEIVE (WORKER_PARSER_FIRST_JOB - 29)
44
+#define WORKER_STREAM_JOB_SOCKET_SEND (WORKER_PARSER_FIRST_JOB - 28)
45
+#define WORKER_STREAM_JOB_SOCKET_ERROR (WORKER_PARSER_FIRST_JOB - 27)
46
+
47
+// compression
48
+#define WORKER_STREAM_JOB_COMPRESS (WORKER_PARSER_FIRST_JOB - 26)
49
+#define WORKER_STREAM_JOB_DECOMPRESS (WORKER_PARSER_FIRST_JOB - 25)
50
+
51
+// receiver events
52
+#define WORKER_RECEIVER_JOB_BYTES_READ (WORKER_PARSER_FIRST_JOB - 24)
53
+#define WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED (WORKER_PARSER_FIRST_JOB - 23)
54
+
55
+// sender received commands
56
+#define WORKER_SENDER_JOB_EXECUTE (WORKER_PARSER_FIRST_JOB - 22)
57
+#define WORKER_SENDER_JOB_EXECUTE_REPLAY (WORKER_PARSER_FIRST_JOB - 21)
58
+#define WORKER_SENDER_JOB_EXECUTE_FUNCTION (WORKER_PARSER_FIRST_JOB - 20)
59
+#define WORKER_SENDER_JOB_EXECUTE_META (WORKER_PARSER_FIRST_JOB - 19)
60
+
61
+#define WORKER_SENDER_JOB_DISCONNECT_OVERFLOW (WORKER_PARSER_FIRST_JOB - 18)
62
+#define WORKER_SENDER_JOB_DISCONNECT_TIMEOUT (WORKER_PARSER_FIRST_JOB - 17)
63
+#define WORKER_SENDER_JOB_DISCONNECT_SOCKET_ERROR (WORKER_PARSER_FIRST_JOB - 16)
64
+#define WORKER_SENDER_JOB_DISCONNECT_PARENT_CLOSED (WORKER_PARSER_FIRST_JOB - 15)
65
+#define WORKER_SENDER_JOB_DISCONNECT_RECEIVE_ERROR (WORKER_PARSER_FIRST_JOB - 14)
66
+#define WORKER_SENDER_JOB_DISCONNECT_SEND_ERROR (WORKER_PARSER_FIRST_JOB - 13)
67
+#define WORKER_SENDER_JOB_DISCONNECT_COMPRESSION_ERROR (WORKER_PARSER_FIRST_JOB - 12)
68
+#define WORKER_SENDER_JOB_DISCONNECT_RECEIVER_LEFT (WORKER_PARSER_FIRST_JOB - 11)
69
+#define WORKER_SENDER_JOB_DISCONNECT_HOST_CLEANUP (WORKER_PARSER_FIRST_JOB - 10)
70
+
71
+// dispatcher metrics
72
+// this has to be the same at pluginsd_parser.h
73
+#define WORKER_RECEIVER_JOB_REPLICATION_COMPLETION (WORKER_PARSER_FIRST_JOB - 9)
74
+#define WORKER_STREAM_METRIC_NODES (WORKER_PARSER_FIRST_JOB - 8)
75
+#define WORKER_SENDER_JOB_BUFFER_RATIO (WORKER_PARSER_FIRST_JOB - 7)
76
+#define WORKER_SENDER_JOB_BYTES_RECEIVED (WORKER_PARSER_FIRST_JOB - 6)
77
+#define WORKER_SENDER_JOB_BYTES_SENT (WORKER_PARSER_FIRST_JOB - 5)
78
+#define WORKER_SENDER_JOB_BYTES_COMPRESSED (WORKER_PARSER_FIRST_JOB - 4)
79
+#define WORKER_SENDER_JOB_BYTES_UNCOMPRESSED (WORKER_PARSER_FIRST_JOB - 3)
80
+#define WORKER_SENDER_JOB_BYTES_COMPRESSION_RATIO (WORKER_PARSER_FIRST_JOB - 2)
81
+#define WORKER_SENDER_JOB_REPLAY_DICT_SIZE (WORKER_PARSER_FIRST_JOB - 1)
82
+#define WORKER_SENDER_JOB_MESSAGES (WORKER_PARSER_FIRST_JOB - 0)
83
+
84
+#if WORKER_UTILIZATION_MAX_JOB_TYPES < 35
85
+#error WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 34
86
+#endif
87
+
88
+#define STREAM_MAX_THREADS 2048
89
+#define THREAD_TAG_STREAM "STREAM"
90
+
91
+typedef enum {
92
+ POLLFD_TYPE_EMPTY,
93
+ POLLFD_TYPE_SENDER,
94
+ POLLFD_TYPE_RECEIVER,
95
+ POLLFD_TYPE_PIPE,
96
+} POLLFD_TYPE;
97
+
98
+struct pollfd_meta {
99
+ POLLFD_TYPE type;
100
+ union {
101
+ struct receiver_state *rpt;
102
+ struct sender_state *s;
103
+ };
104
+};
105
+
106
+DEFINE_JUDYL_TYPED(SENDERS, struct sender_state *);
107
+DEFINE_JUDYL_TYPED(RECEIVERS, struct receiver_state *);
108
+
109
+struct stream_thread {
110
+ ND_THREAD *thread;
111
+
112
+ pid_t tid;
113
+ size_t id;
114
+ size_t nodes_count;
115
+
116
+ struct {
117
+ SENDERS_JudyLSet senders;
118
+ size_t bytes_received;
119
+ size_t bytes_sent;
120
+ } snd;
121
+
122
+ struct {
123
+ RECEIVERS_JudyLSet receivers;
124
+ size_t bytes_received;
125
+ size_t bytes_received_uncompressed;
126
+ NETDATA_DOUBLE replication_completion;
127
+ } rcv;
128
+
129
+ struct {
130
+ SPINLOCK spinlock; // ensure a single writer at a time
131
+ int fds[2];
132
+ size_t size;
133
+ char *buffer;
134
+ } pipe;
135
+
136
+ struct {
137
+ // the incoming queue of the dispatcher thread
138
+ // the connector thread leaves the connected senders in this list, for the dispatcher to pick them up
139
+ SPINLOCK spinlock;
140
+ SENDERS_JudyLSet senders;
141
+ RECEIVERS_JudyLSet receivers;
142
+ } queue;
143
+
144
+ struct {
145
+ SPINLOCK spinlock;
146
+ size_t added;
147
+ size_t processed;
148
+ size_t bypassed;
149
+ size_t size;
150
+ size_t used;
151
+ struct stream_opcode *array; // the array of messages from the senders
152
+ struct stream_opcode *copy; // a copy of the array of messages from the senders, to work on
153
+ } messages;
154
+
155
+ struct {
156
+ nd_poll_t *ndpl;
157
+ struct pollfd_meta pipe;
158
+ } run;
159
+};
160
+
161
+struct stream_thread_globals {
162
+ struct {
163
+ SPINLOCK spinlock;
164
+ size_t id;
165
+ size_t cores;
166
+ } assign;
167
+
168
+ struct stream_thread threads[STREAM_MAX_THREADS];
169
+};
170
+
171
+struct rrdhost;
172
+extern struct stream_thread_globals stream_thread_globals;
173
+
174
+void stream_sender_move_queue_to_running_unsafe(struct stream_thread *sth);
175
+void stream_receiver_move_queue_to_running_unsafe(struct stream_thread *sth);
176
+void stream_sender_check_all_nodes_from_poll(struct stream_thread *sth);
177
+
178
+void stream_receiver_add_to_queue(struct receiver_state *rpt);
179
+void stream_sender_add_to_connector_queue(struct rrdhost *host);
180
+
181
+void stream_sender_process_poll_events(struct stream_thread *sth, struct sender_state *s, nd_poll_event_t events, time_t now_s);
182
+void stream_receive_process_poll_events(struct stream_thread *sth, struct receiver_state *rpt, nd_poll_event_t events, time_t now_s);
183
+
184
+void stream_sender_cleanup(struct stream_thread *sth);
185
+void stream_receiver_cleanup(struct stream_thread *sth);
186
+void stream_sender_handle_op(struct stream_thread *sth, struct sender_state *s, struct stream_opcode *msg);
187
+
188
+struct stream_thread *stream_thread_by_slot_id(size_t thread_slot);
189
+
190
+void stream_thread_node_queued(struct rrdhost *host);
191
+void stream_thread_node_removed(struct rrdhost *host);
192
+
193
+#include "stream-sender-internals.h"
194
+#include "stream-receiver-internals.h"
195
+#include "plugins.d/pluginsd_parser.h"
196
+
197
+static inline bool rrdhost_is_this_a_stream_thread(RRDHOST *host) {
198
+ pid_t tid = gettid_cached();
199
+ return host->stream.rcv.status.tid == tid || host->stream.snd.status.tid == tid;
200
+}
201
+
202
+#endif //NETDATA_STREAM_THREAD_H
src/streaming/stream-traffic-types.h
new
+16
@@ -0,0 +1,16 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_STREAM_TRAFFIC_TYPES_H
4
+#define NETDATA_STREAM_TRAFFIC_TYPES_H
5
+
6
+typedef enum __attribute__((packed)) {
7
+ STREAM_TRAFFIC_TYPE_REPLICATION = 0,
8
+ STREAM_TRAFFIC_TYPE_FUNCTIONS,
9
+ STREAM_TRAFFIC_TYPE_METADATA,
10
+ STREAM_TRAFFIC_TYPE_DATA,
11
+
12
+ // terminator
13
+ STREAM_TRAFFIC_TYPE_MAX,
14
+} STREAM_TRAFFIC_TYPE;
15
+
16
+#endif //NETDATA_STREAM_TRAFFIC_TYPES_H
src/streaming/stream.conf
+7
-7
@@ -62,7 +62,7 @@
62
#enable compression = yes
63
64
# The timeout to connect and send metrics
65
- #timeout = 1m
65
+ #timeout = 5m
66
67
# If the destination line above does not specify a port, use this
68
#default port = 19999
@@ -83,8 +83,8 @@
83
#buffer size bytes = 10485760
84
85
# If the connection fails, or it disconnects,
86
- # retry after that many seconds.
87
- #reconnect delay = 5s
86
+ # retry after that many seconds (randomized from 5s to whatever is here).
87
+ #reconnect delay = 15s
88
89
# Sync the clock of the charts for that many iterations, when starting.
90
# It is ignored when replication is enabled
@@ -149,7 +149,7 @@
149
# Health monitoring will be disabled as soon as the connection is closed.
150
# You can also set it per host, below.
151
# The default is taken from [health].enabled of netdata.conf
152
- #health enabled by default = auto
152
+ #health enabled = auto
153
154
# postpone alerts for a short period after the sender is connected
155
#postpone alerts on connect = 1m
@@ -176,7 +176,7 @@
176
# Enable replication for all hosts using this api key. Default: enabled
177
#enable replication = yes
178
179
- # How many seconds to replicate from each child. Default: a day
179
+ # How many seconds to replicate from each child. Default: configured in netdata.conf (1d)
180
#replication period = 1d
181
182
# The duration we want to replicate per each step.
@@ -249,10 +249,10 @@
249
#enable compression = yes
250
251
# Replication
252
- # Enable replication for all hosts using this api key.
252
+ # Enable replication for this child.
253
#enable replication = yes
254
255
- # How many seconds to replicate from each child.
255
+ # How many seconds to replicate from this child.
256
#replication period = 1d
257
258
# The duration we want to replicate per each step.
src/streaming/stream.h
new
+48
@@ -0,0 +1,48 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_STREAM_H
4
+#define NETDATA_STREAM_H 1
5
+
6
+#include "libnetdata/libnetdata.h"
7
+#include "stream-traffic-types.h"
8
+
9
+struct rrdhost;
10
+struct sender_state;
11
+struct receiver_state;
12
+
13
+#include "stream-conf.h"
14
+#include "stream-handshake.h"
15
+#include "stream-capabilities.h"
16
+#include "stream-parents.h"
17
+
18
+// starting and stopping senders
19
+void *stream_sender_start_localhost(void *ptr);
20
+void stream_sender_start_host(struct rrdhost *host);
21
+void stream_sender_signal_to_stop_and_wait(struct rrdhost *host, STREAM_HANDSHAKE reason, bool wait);
22
+
23
+// managing host sender structures
24
+void stream_sender_structures_init(RRDHOST *host, bool stream, STRING *parents, STRING *api_key, STRING *send_charts_matching);
25
+void stream_sender_structures_free(struct rrdhost *host);
26
+
27
+// querying host sender information
28
+bool stream_sender_is_connected_with_ssl(struct rrdhost *host);
29
+bool stream_sender_has_compression(struct rrdhost *host);
30
+bool stream_sender_has_capabilities(struct rrdhost *host, STREAM_CAPABILITIES capabilities);
31
+
32
+// receiver API
33
+uint32_t stream_receivers_currently_connected(void);
34
+struct web_client;
35
+int stream_receiver_accept_connection(struct web_client *w, char *decoded_query_string, void *h2o_ctx);
36
+bool receiver_has_capability(struct rrdhost *host, STREAM_CAPABILITIES caps);
37
+void stream_receiver_free(struct receiver_state *rpt);
38
+bool stream_receiver_signal_to_stop_and_wait(struct rrdhost *host, STREAM_HANDSHAKE reason);
39
+char *stream_receiver_program_version_strdupz(struct rrdhost *host);
40
+
41
+#include "replication.h"
42
+#include "rrdhost-status.h"
43
+#include "protocol/commands.h"
44
+#include "stream-path.h"
45
+
46
+void stream_threads_cancel(void);
47
+
48
+#endif //NETDATA_STREAM_H
src/web/api/functions/function-streaming.c
+4
-10
@@ -80,7 +80,7 @@ int function_streaming(BUFFER *wb, const char *function __maybe_unused, BUFFER *
80
buffer_json_add_array_item_string(wb, NULL); // InAge
81
}
82
buffer_json_add_array_item_string(wb, stream_handshake_error_to_string(s.ingest.reason)); // InReason
83
- buffer_json_add_array_item_uint64(wb, s.ingest.hops); // InHops
83
+ buffer_json_add_array_item_int64(wb, s.ingest.hops); // InHops
84
buffer_json_add_array_item_double(wb, s.ingest.replication.completion); // InReplCompletion
85
buffer_json_add_array_item_uint64(wb, s.ingest.replication.instances); // InReplInstances
86
buffer_json_add_array_item_string(wb, s.ingest.peers.local.ip); // InLocalIP
@@ -120,13 +120,7 @@ int function_streaming(BUFFER *wb, const char *function __maybe_unused, BUFFER *
120
buffer_json_add_array_item_uint64(wb, s.stream.sent_bytes_on_this_connection_per_type[STREAM_TRAFFIC_TYPE_FUNCTIONS]);
121
122
buffer_json_add_array_item_array(wb); // OutAttemptHandshake
123
- time_t last_attempt = 0;
124
- for(struct rrdpush_destinations *d = host->destinations; d ; d = d->next) {
125
- if(d->since > last_attempt)
126
- last_attempt = d->since;
127
-
128
- buffer_json_add_array_item_string(wb, stream_handshake_error_to_string(d->reason));
129
- }
123
+ usec_t last_attempt = stream_parent_handshake_error_to_json(wb, host);
124
buffer_json_array_close(wb); // // OutAttemptHandshake
125
126
if(!last_attempt) {
@@ -134,8 +128,8 @@ int function_streaming(BUFFER *wb, const char *function __maybe_unused, BUFFER *
128
buffer_json_add_array_item_string(wb, NULL); // OutAttemptAge
129
}
130
else {
137
- buffer_json_add_array_item_uint64(wb, last_attempt * 1000); // OutAttemptSince
138
- buffer_json_add_array_item_time_t(wb, s.now - last_attempt); // OutAttemptAge
131
+ buffer_json_add_array_item_uint64(wb, last_attempt / USEC_PER_MS); // OutAttemptSince
132
+ buffer_json_add_array_item_time_t(wb, s.now - (time_t)(last_attempt / USEC_PER_SEC)); // OutAttemptAge
133
}
134
135
// ML
src/web/api/queries/query.c
+7
-7
@@ -2001,7 +2001,7 @@ void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, size_t tier, time_t now_s
2001
long before_wanted = smaller_tier_last_time;
2002
2003
struct rrddim_tier *tmp = &rd->tiers[read_tier];
2004
- storage_engine_query_init(tmp->seb, tmp->smh, &seqh, after_wanted, before_wanted, STORAGE_PRIORITY_HIGH);
2004
+ storage_engine_query_init(tmp->seb, tmp->smh, &seqh, after_wanted, before_wanted, STORAGE_PRIORITY_SYNCHRONOUS);
2005
2006
size_t points_read = 0;
2007
@@ -2018,7 +2018,7 @@ void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, size_t tier, time_t now_s
2018
2019
storage_engine_query_finalize(&seqh);
2020
store_metric_collection_completed();
2021
- global_statistics_backfill_query_completed(points_read);
2021
+ telemetry_queries_backfill_query_completed(points_read);
2022
2023
//internal_error(true, "DBENGINE: backfilled chart '%s', dimension '%s', tier %d, from %ld to %ld, with %zu points from tier %d",
2024
// rd->rrdset->name, rd->name, tier, after_wanted, before_wanted, points, tr);
@@ -3592,11 +3592,11 @@ RRDR *rrd2rrdr(ONEWAYALLOC *owa, QUERY_TARGET *qt) {
3592
continue;
3593
}
3594
3595
- global_statistics_rrdr_query_completed(
3596
- 1,
3597
- r_tmp->stats.db_points_read - last_db_points_read,
3598
- r_tmp->stats.result_points_generated - last_result_points_generated,
3599
- qt->request.query_source);
3595
+ telemetry_queries_rrdr_query_completed(
3596
+ 1,
3597
+ r_tmp->stats.db_points_read - last_db_points_read,
3598
+ r_tmp->stats.result_points_generated - last_result_points_generated,
3599
+ qt->request.query_source);
3600
3601
last_db_points_read = r_tmp->stats.db_points_read;
3602
last_result_points_generated = r_tmp->stats.result_points_generated;
src/web/api/v1/api_v1_info.c
+2
-3
@@ -161,10 +161,9 @@ static int web_client_api_request_v1_info_fill_buffer(RRDHOST *host, BUFFER *wb)
161
buffer_json_member_add_uint64(wb, "page-cache-size", default_rrdeng_page_cache_mb);
162
#endif // ENABLE_DBENGINE
163
buffer_json_member_add_boolean(wb, "web-enabled", web_server_mode != WEB_SERVER_MODE_NONE);
164
- buffer_json_member_add_boolean(wb, "stream-enabled", stream_conf_send_enabled);
164
+ buffer_json_member_add_boolean(wb, "stream-enabled", stream_send.enabled);
165
166
- buffer_json_member_add_boolean(wb, "stream-compression",
167
- host->sender && host->sender->compressor.initialized);
166
+ buffer_json_member_add_boolean(wb, "stream-compression", stream_sender_has_compression(host));
167
168
buffer_json_member_add_boolean(wb, "https-enabled", true);
169
src/web/api/v3/api_v3_calls.h
+1
@@ -10,6 +10,7 @@ int api_v2_contexts_internal(RRDHOST *host, struct web_client *w, char *url, CON
10
11
int api_v3_settings(RRDHOST *host, struct web_client *w, char *url);
12
int api_v3_me(RRDHOST *host, struct web_client *w, char *url);
13
+int api_v3_stream_info(RRDHOST *host __maybe_unused, struct web_client *w, char *url);
14
int api_v3_stream_path(RRDHOST *host __maybe_unused, struct web_client *w, char *url);
15
16
#endif //NETDATA_API_V3_CALLS_H
src/web/api/v3/api_v3_stream_info.c
new
+24
@@ -0,0 +1,24 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "api_v3_calls.h"
4
+
5
+int api_v3_stream_info(RRDHOST *host __maybe_unused, struct web_client *w, char *url __maybe_unused) {
6
+ const char *machine_guid = NULL;
7
+
8
+ while(url) {
9
+ char *value = strsep_skip_consecutive_separators(&url, "&");
10
+ if(!value || !*value) continue;
11
+
12
+ char *name = strsep_skip_consecutive_separators(&value, "=");
13
+ if(!name || !*name) continue;
14
+ if(!value || !*value) continue;
15
+
16
+ // name and value are now the parameters
17
+ // they are not null and not empty
18
+
19
+ if(!strcmp(name, "machine_guid"))
20
+ machine_guid = value;
21
+ }
22
+
23
+ return stream_info_to_json_v1(w->response.data, machine_guid);
24
+}
src/web/api/web_api_v3.c
+9
@@ -193,6 +193,15 @@ static struct web_api_command api_commands_v3[] = {
193
.allow_subpaths = 0
194
},
195
196
+ {
197
+ .api = "stream_info",
198
+ .hash = 0,
199
+ .acl = HTTP_ACL_NOCHECK,
200
+ .access = HTTP_ACCESS_NONE,
201
+ .callback = api_v3_stream_info,
202
+ .allow_subpaths = 0
203
+ },
204
+
205
// WebRTC APIs
206
{
207
.api = "rtc_offer",
src/web/server/h2o/rrdpush.c
+1
-1
@@ -301,7 +301,7 @@ void stream_process(h2o_stream_conn_t *conn, int initial)
301
w.client_ip[cpy_len - 1] = 0;
302
w.user_agent = conn->user_agent;
303
304
- rc = rrdpush_receiver_thread_spawn(&w, conn->url, conn);
304
+ rc = stream_receiver_accept_connection(&w, conn->url, conn);
305
if (rc != HTTP_RESP_OK) {
306
error_report("HTTPD Failed to spawn the receiver thread %d", rc);
307
conn->state = STREAM_CLOSE;
src/web/server/static/static-threaded.c
+2
-2
@@ -123,7 +123,7 @@ static void web_server_file_del_callback(POLLINFO *pi) {
123
web_server_log_connection(w, "DISCONNECTED");
124
web_client_request_done(w);
125
web_client_release_to_cache(w);
126
- global_statistics_web_client_disconnected();
126
+ telemetry_web_client_disconnected();
127
}
128
129
worker_is_idle();
@@ -269,7 +269,7 @@ static void web_server_del_callback(POLLINFO *pi) {
269
web_server_log_connection(w, "DISCONNECTED");
270
web_client_request_done(w);
271
web_client_release_to_cache(w);
272
- global_statistics_web_client_disconnected();
272
+ telemetry_web_client_disconnected();
273
}
274
275
worker_is_idle();
src/web/server/web_client.c
+4
-6
@@ -225,11 +225,8 @@ void web_client_log_completed_request(struct web_client *w, bool update_web_stat
225
size_t sent = w->response.zoutput ? (size_t)w->response.zstream.total_out : size;
226
227
if(update_web_stats)
228
- global_statistics_web_request_completed(dt_usec(&tv, &w->timings.tv_in),
229
- w->statistics.received_bytes,
230
- w->statistics.sent_bytes,
231
- size,
232
- sent);
228
+ telemetry_web_request_completed(
229
+ dt_usec(&tv, &w->timings.tv_in), w->statistics.received_bytes, w->statistics.sent_bytes, size, sent);
230
231
usec_t prep_ut = w->timings.tv_ready.tv_sec ? dt_usec(&w->timings.tv_ready, &w->timings.tv_in) : 0;
232
usec_t sent_ut = w->timings.tv_ready.tv_sec ? dt_usec(&tv, &w->timings.tv_ready) : 0;
@@ -1346,7 +1343,8 @@ void web_client_process_request_from_web_server(struct web_client *w) {
1343
return;
1344
}
1345
1349
- w->response.code = rrdpush_receiver_thread_spawn(w, (char *)buffer_tostring(w->url_query_string_decoded), NULL);
1346
+ w->response.code = stream_receiver_accept_connection(
1347
+ w, (char *)buffer_tostring(w->url_query_string_decoded), NULL);
1348
return;
1349
1350
case HTTP_REQUEST_MODE_OPTIONS:
src/web/server/web_client_cache.c
+1
-1
@@ -103,7 +103,7 @@ struct web_client *web_client_get_from_cache(void) {
103
w = web_client_create(&netdata_buffers_statistics.buffers_web);
104
spinlock_lock(&web_clients_cache.used.spinlock);
105
106
- w->id = global_statistics_web_client_connected();
106
+ w->id = telemetry_web_client_connected();
107
web_clients_cache.used.allocated++;
108
}
109