@cryptotaxi247 / netdata-1 / commits / 368a26cfe

DBENGINE v2 (#14125)

* count open cache pages refering to datafile * eliminate waste flush attempts * remove eliminated variable * journal v2 scanning split functions * avoid locking open cache for a long time while migrating to journal v2 * dont acquire datafile for the loop; disable thread cancelability while a query is running * work on datafile acquiring * work on datafile deletion * work on datafile deletion again * logs of dbengine should start with DBENGINE * thread specific key for queries to check if a query finishes without a finalize * page_uuid is not used anymore * Cleanup judy traversal when building new v2 Remove not needed calls to metric registry * metric is 8 bytes smaller; timestamps are protected with a spinlock; timestamps in metric are now always coherent * disable checks for invalid time-ranges * Remove type from page details * report scanning time * remove infinite loop from datafile acquire for deletion * remove infinite loop from datafile acquire for deletion again * trace query handles * properly allocate array of dimensions in replication * metrics cleanup * metrics registry uses arrayalloc * arrayalloc free should be protected by lock * use array alloc in page cache * journal v2 scanning fix * datafile reference leaking hunding * do not load metrics of future timestamps * initialize reasons * fix datafile reference leak * do not load pages that are entirely overlapped by others * expand metric retention atomically * split replication logic in initialization and execution * replication prepare ahead queries * replication prepare ahead queries fixed * fix replication workers accounting * add router active queries chart * restore accounting of pages metadata sources; cleanup replication * dont count skipped pages as unroutable * notes on services shutdown * do not migrate to journal v2 too early, while it has pending dirty pages in the main cache for the specific journal file * do not add pages we dont need to pdc * time in range re-work to provide info about past and future matches * finner control on the pages selected for processing; accounting of page related issues * fix invalid reference to handle->page * eliminate data collection handle of pg_lookup_next * accounting for queries with gaps * query preprocessing the same way the processing is done; cache now supports all operations on Judy * dynamic libuv workers based on number of processors; minimum libuv workers 8; replication query init ahead uses libuv workers - reserved ones (3) * get into pdc all matching pages from main cache and open cache; do not do v2 scan if main cache and open cache can satisfy the query * finner gaps calculation; accounting of overlapping pages in queries * fix gaps accounting * move datafile deletion to worker thread * tune libuv workers and thread stack size * stop netdata threads gradually * run indexing together with cache flush/evict * more work on clean shutdown * limit the number of pages to evict per run * do not lock the clean queue for accesses if it is not possible at that time - the page will be moved to the back of the list during eviction * economies on flags for smaller page footprint; cleanup and renames * eviction moves referenced pages to the end of the queue * use murmur hash for indexing partition * murmur should be static * use more indexing partitions * revert number of partitions to number of cpus * cancel threads first, then stop services * revert default thread stack size * dont execute replication requests of disconnected senders * wait more time for services that are exiting gradually * fixed last commit * finer control on page selection algorithm * default stacksize of 1MB * fix formatting * fix worker utilization going crazy when the number is rotating * avoid buffer full due to replication preprocessing of requests * support query priorities * add count of spins in spinlock when compiled with netdata internal checks * remove prioritization from dbengine queries; cache now uses mutexes for the queues * hot pages are now in sections judy arrays, like dirty * align replication queries to optimal page size * during flushing add to clean and evict in batches * Revert "during flushing add to clean and evict in batches" This reverts commit 8fb2b69d068499eacea6de8291c336e5e9f197c7. * dont lock clean while evicting pages during flushing * Revert "dont lock clean while evicting pages during flushing" This reverts commit d6c82b5f40aeba86fc7aead062fab1b819ba58b3. * Revert "Revert "during flushing add to clean and evict in batches"" This reverts commit ca7a187537fb8f743992700427e13042561211ec. * dont cross locks during flushing, for the fastest flushes possible * low-priority queries load pages synchronously * Revert "low-priority queries load pages synchronously" This reverts commit 1ef2662ddcd20fe5842b856c716df134c42d1dc7. * cache uses spinlock again * during flushing, dont lock the clean queue at all; each item is added atomically * do smaller eviction runs * evict one page at a time to minimize lock contention on the clean queue * fix eviction statistics * fix last commit * plain should be main cache * event loop cleanup; evictions and flushes can now happen concurrently * run flush and evictions from tier0 only * remove not needed variables * flushing open cache is not needed; flushing protection is irrelevant since flushing is global for all tiers; added protection to datafiles so that only one flusher can run per datafile at any given time * added worker jobs in timer to find the slow part of it * support fast eviction of pages when all_of_them is set * revert default thread stack size * bypass event loop for dispatching read extent commands to workers - send them directly * Revert "bypass event loop for dispatching read extent commands to workers - send them directly" This reverts commit 2c08bc5bab12881ae33bc73ce5dea03dfc4e1fce. * cache work requests * minimize memory operations during flushing; caching of extent_io_descriptors and page_descriptors * publish flushed pages to open cache in the thread pool * prevent eventloop requests from getting stacked in the event loop * single threaded dbengine controller; support priorities for all queries; major cleanup and restructuring of rrdengine.c * more rrdengine.c cleanup * enable db rotation * do not log when there is a filter * do not run multiple migration to journal v2 * load all extents async * fix wrong paste * report opcodes waiting, works dispatched, works executing * cleanup event loop memory every 10 minutes * dont dispatch more work requests than the number of threads available * use the dispatched counter instead of the executing counter to check if the worker thread pool is full * remove UV_RUN_NOWAIT * replication to fill the queues * caching of extent buffers; code cleanup * caching of pdc and pd; rework on journal v2 indexing, datafile creation, database rotation * single transaction wal * synchronous flushing * first cancel the threads, then signal them to exit * caching of rrdeng query handles; added priority to query target; health is now low prio * add priority to the missing points; do not allow critical priority in queries * offload query preparation and routing to libuv thread pool * updated timing charts for the offloaded query preparation * caching of WALs * accounting for struct caches (buffers); do not load extents with invalid sizes * protection against memory booming during replication due to the optimal alignment of pages; sender thread buffer is now also reset when the circular buffer is reset * also check if the expanded before is not the chart later updated time * also check if the expanded before is not after the wall clock time of when the query started * Remove unused variable * replication to queue less queries; cleanup of internal fatals * Mark dimension to be updated async * caching of extent_page_details_list (epdl) and datafile_extent_offset_list (deol) * disable pgc stress test, under an ifdef * disable mrg stress test under an ifdef * Mark chart and host labels, host info for async check and store in the database * dictionary items use arrayalloc * cache section pages structure is allocated with arrayalloc * Add function to wakeup the aclk query threads and check for exit Register function to be called during shutdown after signaling the service to exit * parallel preparation of all dimensions of queries * be more sensitive to enable streaming after replication * atomically finish chart replication * fix last commit * fix last commit again * fix last commit again again * fix last commit again again again * unify the normalization of retention calculation for collected charts; do not enable streaming if more than 60 points are to be transferred; eliminate an allocation during replication * do not cancel start streaming; use high priority queries when we have locked chart data collection * prevent starvation on opcodes execution, by allowing 2% of the requests to be re-ordered * opcode now uses 2 spinlocks one for the caching of allocations and one for the waiting queue * Remove check locks and NETDATA_VERIFY_LOCKS as it is not needed anymore * Fix bad memory allocation / cleanup * Cleanup ACLK sync initialization (part 1) * Don't update metric registry during shutdown (part 1) * Prevent crash when dashboard is refreshed and host goes away * Mark ctx that is shutting down. Test not adding flushed pages to open cache as hot if we are shutting down * make ML work * Fix compile without NETDATA_INTERNAL_CHECKS * shutdown each ctx independently * fix completion of quiesce * do not update shared ML charts * Create ML charts on child hosts. When a parent runs a ML for a child, the relevant-ML charts should be created on the child host. These charts should use the parent's hostname to differentiate multiple parents that might run ML for a child. The only exception to this rule is the training/prediction resource usage charts. These are created on the localhost of the parent host, because they provide information specific to said host. * check new ml code * first save the database, then free all memory * dbengine prep exit before freeing all memory; fixed deadlock in cache hot to dirty; added missing check to query engine about metrics without any data in the db * Cleanup metadata thread (part 2) * increase refcount before dispatching prep command * Do not try to stop anomaly detection threads twice. A separate function call has been added to stop anomaly detection threads. This commit removes the left over function calls that were made internally when a host was being created/destroyed. * Remove allocations when smoothing samples buffer The number of dims per sample is always 1, ie. we are training and predicting only individual dimensions. * set the orphan flag when loading archived hosts * track worker dispatch callbacks and threadpool worker init * make ML threads joinable; mark ctx having flushing in progress as early as possible * fix allocation counter * Cleanup metadata thread (part 3) * Cleanup metadata thread (part 4) * Skip metadata host scan when running unittest * unittest support during init * dont use all the libuv threads for queries * break an infinite loop when sleep_usec() is interrupted * ml prediction is a collector for several charts * sleep_usec() now makes sure it will never loop if it passes the time expected; sleep_usec() now uses nanosleep() because clock_nanosleep() misses signals on netdata exit * worker_unregister() in netdata threads cleanup * moved pdc/epdl/deol/extent_buffer related code to pdc.c and pdc.h * fixed ML issues * removed engine2 directory * added dbengine2 files in CMakeLists.txt * move query plan data to query target, so that they can be exposed by in jsonwrap * uniform definition of query plan according to the other query target members * event_loop should be in daemon, not libnetdata * metric_retention_by_uuid() is now part of the storage engine abstraction * unify time_t variables to have the suffix _s (meaning: seconds) * old dbengine statistics become "dbengine io" * do not enable ML resource usage charts by default * unify ml chart families, plugins and modules * cleanup query plans from query target * cleanup all extent buffers * added debug info for rrddim slot to time * rrddim now does proper gap management * full rewrite of the mem modes * use library functions for madvise * use CHECKSUM_SZ for the checksum size * fix coverity warning about the impossible case of returning a page that is entirely in the past of the query * fix dbengine shutdown * keep the old datafile lock until a new datafile has been created, to avoid creating multiple datafiles concurrently * fine tune cache evictions * dont initialize health if the health service is not running - prevent crash on shutdown while children get connected * rename AS threads to ACLK[hostname] * prevent re-use of uninitialized memory in queries * use JulyL instead of JudyL for PDC operations - to test it first * add also JulyL files * fix July memory accounting * disable July for PDC (use Judy) * use the function to remove datafiles from linked list * fix july and event_loop * add july to libnetdata subdirs * rename time_t variables that end in _t to end in _s * replicate when there is a gap at the beginning of the replication period * reset postponing of sender connections when a receiver is connected * Adjust update every properly * fix replication infinite loop due to last change * packed enums in rrd.h and cleanup of obsolete rrd structure members * prevent deadlock in replication: replication_recalculate_buffer_used_ratio_unsafe() deadlocking with replication_sender_delete_pending_requests() * void unused variable * void unused variables * fix indentation * entries_by_time calculation in VD was wrong; restored internal checks for checking future timestamps * macros to caclulate page entries by time and size * prevent statsd cleanup crash on exit * cleanup health thread related variables Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com> Co-authored-by: vkalintiris <vasilis@netdata.cloud>

Costa Tsaousis committed Jan 10, 2023 at 19:59 UTC 368a26cfee6887ca0cb2301d93138f63b75e353a
123 files changed +14967 -7480
CMakeLists.txt
+10 -2
@@ -471,6 +471,8 @@ set(LIBNETDATA_FILES
471 libnetdata/eval/eval.h
472 libnetdata/health/health.c
473 libnetdata/health/health.h
474 + libnetdata/july/july.c
475 + libnetdata/july/july.h
476 libnetdata/inlined.h
477 libnetdata/json/json.c
478 libnetdata/json/json.h
@@ -775,8 +777,12 @@ set(RRD_PLUGIN_FILES
777 database/engine/rrdengineapi.h
778 database/engine/pagecache.c
779 database/engine/pagecache.h
778 - database/engine/rrdenglocking.c
779 - database/engine/rrdenglocking.h
780 + database/engine/cache.c
781 + database/engine/cache.h
782 + database/engine/metric.c
783 + database/engine/metric.h
784 + database/engine/pdc.c
785 + database/engine/pdc.h
786 database/KolmogorovSmirnovDist.c
787 database/KolmogorovSmirnovDist.h
788 )
@@ -1002,6 +1008,8 @@ set(DAEMON_FILES
1008 daemon/common.h
1009 daemon/daemon.c
1010 daemon/daemon.h
1011 + daemon/event_loop.c
1012 + daemon/event_loop.h
1013 daemon/global_statistics.c
1014 daemon/global_statistics.h
1015 daemon/analytics.c
Makefile.am
+10 -2
@@ -149,6 +149,8 @@ LIBNETDATA_FILES = \
149 libnetdata/eval/eval.c \
150 libnetdata/eval/eval.h \
151 libnetdata/inlined.h \
152 + libnetdata/july/july.c \
153 + libnetdata/july/july.h \
154 libnetdata/libnetdata.c \
155 libnetdata/libnetdata.h \
156 libnetdata/required_dummies.h \
@@ -547,8 +549,12 @@ if ENABLE_DBENGINE
549 database/engine/rrdengineapi.h \
550 database/engine/pagecache.c \
551 database/engine/pagecache.h \
550 - database/engine/rrdenglocking.c \
551 - database/engine/rrdenglocking.h \
552 + database/engine/cache.c \
553 + database/engine/cache.h \
554 + database/engine/metric.c \
555 + database/engine/metric.h \
556 + database/engine/pdc.c \
557 + database/engine/pdc.h \
558 $(NULL)
559 endif
560
@@ -896,6 +902,8 @@ DAEMON_FILES = \
902 daemon/common.h \
903 daemon/daemon.c \
904 daemon/daemon.h \
905 + daemon/event_loop.c \
906 + daemon/event_loop.h \
907 daemon/global_statistics.c \
908 daemon/global_statistics.h \
909 daemon/analytics.c \
aclk/aclk.c
+38 -33
@@ -157,7 +157,7 @@ static int wait_till_cloud_enabled()
157 info("Waiting for Cloud to be enabled");
158 while (!netdata_cloud_setting) {
159 sleep_usec(USEC_PER_SEC * 1);
160 - if (netdata_exit)
160 + if (!service_running(SERVICE_ACLK))
161 return 1;
162 }
163 return 0;
@@ -176,7 +176,7 @@ static int wait_till_agent_claimed(void)
176 char *agent_id = get_agent_claimid();
177 while (likely(!agent_id)) {
178 sleep_usec(USEC_PER_SEC * 1);
179 - if (netdata_exit)
179 + if (!service_running(SERVICE_ACLK))
180 return 1;
181 agent_id = get_agent_claimid();
182 }
@@ -196,7 +196,7 @@ static int wait_till_agent_claimed(void)
196 static int wait_till_agent_claim_ready()
197 {
198 url_t url;
199 - while (!netdata_exit) {
199 + while (service_running(SERVICE_ACLK)) {
200 if (wait_till_agent_claimed())
201 return 1;
202
@@ -330,7 +330,7 @@ void aclk_graceful_disconnect(mqtt_wss_client client);
330 static int handle_connection(mqtt_wss_client client)
331 {
332 time_t last_periodic_query_wakeup = now_monotonic_sec();
333 - while (!netdata_exit) {
333 + while (service_running(SERVICE_ACLK)) {
334 // timeout 1000 to check at least once a second
335 // for netdata_exit
336 if (mqtt_wss_service(client, 1000) < 0){
@@ -463,7 +463,7 @@ static int aclk_block_till_recon_allowed() {
463 // we want to wake up from time to time to check netdata_exit
464 while (recon_delay)
465 {
466 - if (netdata_exit)
466 + if (!service_running(SERVICE_ACLK))
467 return 1;
468 if (recon_delay > NETDATA_EXIT_POLL_MS) {
469 sleep_usec(NETDATA_EXIT_POLL_MS * USEC_PER_MS);
@@ -473,7 +473,7 @@ static int aclk_block_till_recon_allowed() {
473 sleep_usec(recon_delay * USEC_PER_MS);
474 recon_delay = 0;
475 }
476 - return netdata_exit;
476 + return !service_running(SERVICE_ACLK);
477 }
478
479 #ifndef ACLK_DISABLE_CHALLENGE
@@ -516,7 +516,7 @@ static int aclk_attempt_to_connect(mqtt_wss_client client)
516 url_t mqtt_url;
517 #endif
518
519 - while (!netdata_exit) {
519 + while (service_running(SERVICE_ACLK)) {
520 char *cloud_base_url = appconfig_get(&cloud_config, CONFIG_SECTION_GLOBAL, "cloud base url", NULL);
521 if (cloud_base_url == NULL) {
522 error_report("Do not move the cloud base url out of post_conf_load!!");
@@ -564,7 +564,7 @@ static int aclk_attempt_to_connect(mqtt_wss_client client)
564 continue;
565 }
566
567 - if (netdata_exit)
567 + if (!service_running(SERVICE_ACLK))
568 return 1;
569
570 if (aclk_env->encoding != ACLK_ENC_PROTO) {
@@ -748,7 +748,7 @@ void *aclk_main(void *ptr)
748 aclk_connected = 0;
749 log_access("ACLK DISCONNECTED");
750 }
751 - } while (!netdata_exit);
751 + } while (service_running(SERVICE_ACLK));
752
753 aclk_graceful_disconnect(mqttwss_client);
754
@@ -783,35 +783,40 @@ exit:
783 void aclk_host_state_update(RRDHOST *host, int cmd)
784 {
785 uuid_t node_id;
786 - int ret;
786 + int ret = 0;
787
788 if (!aclk_connected)
789 return;
790
791 - ret = get_node_id(&host->host_uuid, &node_id);
792 - if (ret > 0) {
793 - // this means we were not able to check if node_id already present
794 - error("Unable to check for node_id. Ignoring the host state update.");
795 - return;
791 + if (host->node_id && !uuid_is_null(*host->node_id)) {
792 + uuid_copy(node_id, *host->node_id);
793 }
797 - if (ret < 0) {
798 - // node_id not found
799 - aclk_query_t create_query;
800 - create_query = aclk_query_new(REGISTER_NODE);
801 - rrdhost_aclk_state_lock(localhost);
802 - node_instance_creation_t node_instance_creation = {
803 - .claim_id = localhost->aclk_state.claimed_id,
804 - .hops = host->system_info->hops,
805 - .hostname = rrdhost_hostname(host),
806 - .machine_guid = host->machine_guid
807 - };
808 - create_query->data.bin_payload.payload = generate_node_instance_creation(&create_query->data.bin_payload.size, &node_instance_creation);
809 - rrdhost_aclk_state_unlock(localhost);
810 - create_query->data.bin_payload.topic = ACLK_TOPICID_CREATE_NODE;
811 - create_query->data.bin_payload.msg_name = "CreateNodeInstance";
812 - info("Registering host=%s, hops=%u",host->machine_guid, host->system_info->hops);
813 - aclk_queue_query(create_query);
814 - return;
794 + else {
795 + ret = get_node_id(&host->host_uuid, &node_id);
796 + if (ret > 0) {
797 + // this means we were not able to check if node_id already present
798 + error("Unable to check for node_id. Ignoring the host state update.");
799 + return;
800 + }
801 + if (ret < 0) {
802 + // node_id not found
803 + aclk_query_t create_query;
804 + create_query = aclk_query_new(REGISTER_NODE);
805 + rrdhost_aclk_state_lock(localhost);
806 + node_instance_creation_t node_instance_creation = {
807 + .claim_id = localhost->aclk_state.claimed_id,
808 + .hops = host->system_info->hops,
809 + .hostname = rrdhost_hostname(host),
810 + .machine_guid = host->machine_guid};
811 + create_query->data.bin_payload.payload =
812 + generate_node_instance_creation(&create_query->data.bin_payload.size, &node_instance_creation);
813 + rrdhost_aclk_state_unlock(localhost);
814 + create_query->data.bin_payload.topic = ACLK_TOPICID_CREATE_NODE;
815 + create_query->data.bin_payload.msg_name = "CreateNodeInstance";
816 + info("Registering host=%s, hops=%u", host->machine_guid, host->system_info->hops);
817 + aclk_queue_query(create_query);
818 + return;
819 + }
820 }
821
822 aclk_query_t query = aclk_query_new(NODE_STATE_UPDATE);
aclk/aclk_query.c
+8 -1
@@ -327,6 +327,11 @@ static void worker_aclk_register(void) {
327 }
328 }
329
330 +static void aclk_query_request_cancel(void *data)
331 +{
332 + pthread_cond_broadcast((pthread_cond_t *) data);
333 +}
334 +
335 /**
336 * Main query processing thread
337 */
@@ -336,7 +341,9 @@ void *aclk_query_main_thread(void *ptr)
341
342 struct aclk_query_thread *query_thr = ptr;
343
339 - while (!netdata_exit) {
344 + service_register(SERVICE_THREAD_TYPE_NETDATA, aclk_query_request_cancel, NULL, &query_cond_wait, false);
345 +
346 + while (service_running(SERVICE_ACLK | ABILITY_DATA_QUERIES)) {
347 aclk_query_process_msgs(query_thr);
348
349 worker_is_idle();
aclk/aclk_query_queue.c
+2 -2
@@ -26,7 +26,7 @@ static inline int _aclk_queue_query(aclk_query_t query)
26 ACLK_QUEUE_LOCK;
27 if (aclk_query_queue.block_push) {
28 ACLK_QUEUE_UNLOCK;
29 - if(!netdata_exit)
29 + if(service_running(SERVICE_ACLK | ABILITY_DATA_QUERIES))
30 error("Query Queue is blocked from accepting new requests. This is normally the case when ACLK prepares to shutdown.");
31 aclk_query_free(query);
32 return 1;
@@ -66,7 +66,7 @@ aclk_query_t aclk_queue_pop(void)
66 ACLK_QUEUE_LOCK;
67 if (aclk_query_queue.block_push) {
68 ACLK_QUEUE_UNLOCK;
69 - if(!netdata_exit)
69 + if(service_running(SERVICE_ACLK | ABILITY_DATA_QUERIES))
70 error("POP Query Queue is blocked from accepting new requests. This is normally the case when ACLK prepares to shutdown.");
71 return NULL;
72 }
aclk/aclk_rx_msgs.c
+1 -3
@@ -283,9 +283,7 @@ int create_node_instance_result(const char *msg, size_t msg_len)
283 node_state_update.live = 1;
284 node_state_update.hops = 0;
285 } else {
286 - netdata_mutex_lock(&host->receiver_lock);
287 - node_state_update.live = (host->receiver != NULL);
288 - netdata_mutex_unlock(&host->receiver_lock);
286 + node_state_update.live = (!rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN));
287 node_state_update.hops = host->system_info->hops;
288 }
289 }
aclk/aclk_stats.c
+2 -2
@@ -314,13 +314,13 @@ void *aclk_stats_main_thread(void *ptr)
314 struct aclk_metrics_per_sample per_sample;
315 struct aclk_metrics permanent;
316
317 - while (!netdata_exit) {
317 + while (service_running(SERVICE_ACLK | SERVICE_COLLECTORS)) {
318 netdata_thread_testcancel();
319 // ------------------------------------------------------------------------
320 // Wait for the next iteration point.
321
322 heartbeat_next(&hb, step_ut);
323 - if (netdata_exit) break;
323 + if (!service_running(SERVICE_ACLK | SERVICE_COLLECTORS)) break;
324
325 ACLK_STATS_LOCK;
326 // to not hold lock longer than necessary, especially not to hold it
claim/claim.c
+3 -2
@@ -170,9 +170,10 @@ void load_claiming_state(void)
170 }
171
172 invalidate_node_instances(&localhost->host_uuid, claimed_id ? &uuid : NULL);
173 - metaqueue_store_claim_id(&localhost->host_uuid, claimed_id ? &uuid : NULL);
174 -
173 rrdhost_aclk_state_unlock(localhost);
174 +
175 + rrdhost_flag_set(localhost, RRDHOST_FLAG_METADATA_CLAIMID | RRDHOST_FLAG_METADATA_UPDATE);
176 +
177 if (!claimed_id) {
178 info("Unable to load '%s', setting state to AGENT_UNCLAIMED", filename);
179 return;
collectors/all.h
+7
@@ -382,4 +382,11 @@
382 #define NETDATA_ML_CHART_PRIO_TRAINING_TIME_STATS 890007
383 #define NETDATA_ML_CHART_PRIO_TRAINING_RESULTS 890008
384
385 +#define NETDATA_ML_CHART_FAMILY "ml - machine learning"
386 +#define NETDATA_ML_PLUGIN "ml.plugin"
387 +#define NETDATA_ML_MODULE_TRAINING "training"
388 +#define NETDATA_ML_MODULE_DETECTION "detection"
389 +#define NETDATA_ML_MODULE_PREDICTION "prediction"
390 +
391 +
392 #endif //NETDATA_ALL_H
collectors/cgroups.plugin/sys_fs_cgroup.c
+19 -8
@@ -2754,10 +2754,20 @@ static inline void discovery_find_all_cgroups() {
2754 debug(D_CGROUP, "done searching for cgroups");
2755 }
2756
2757 +static void cgroup_discovery_cleanup(void *ptr) {
2758 + UNUSED(ptr);
2759 +
2760 + discovery_thread.exited = 1;
2761 + worker_unregister();
2762 + service_exits();
2763 +}
2764 +
2765 void cgroup_discovery_worker(void *ptr)
2766 {
2767 UNUSED(ptr);
2768
2769 + netdata_thread_cleanup_push(cgroup_discovery_cleanup, ptr);
2770 +
2771 worker_register("CGROUPSDISC");
2772 worker_register_job_name(WORKER_DISCOVERY_INIT, "init");
2773 worker_register_job_name(WORKER_DISCOVERY_FIND, "find");
@@ -2777,24 +2787,23 @@ void cgroup_discovery_worker(void *ptr)
2787 NULL,
2788 SIMPLE_PATTERN_EXACT);
2789
2780 - while (!netdata_exit) {
2790 + while (service_running(SERVICE_COLLECTORS)) {
2791 worker_is_idle();
2792
2793 uv_mutex_lock(&discovery_thread.mutex);
2784 - while (!discovery_thread.start_discovery)
2794 + while (!discovery_thread.start_discovery && service_running(SERVICE_COLLECTORS))
2795 uv_cond_wait(&discovery_thread.cond_var, &discovery_thread.mutex);
2796 discovery_thread.start_discovery = 0;
2797 uv_mutex_unlock(&discovery_thread.mutex);
2798
2789 - if (unlikely(netdata_exit))
2799 + if (unlikely(!service_running(SERVICE_COLLECTORS)))
2800 break;
2801
2802 discovery_find_all_cgroups();
2803 }
2804
2795 - discovery_thread.exited = 1;
2796 - worker_unregister();
2797 -}
2805 + netdata_thread_cleanup_pop(1);
2806 +}
2807
2808 // ----------------------------------------------------------------------------
2809 // generate charts
@@ -4853,11 +4862,11 @@ void *cgroups_main(void *ptr) {
4862 usec_t step = cgroup_update_every * USEC_PER_SEC;
4863 usec_t find_every = cgroup_check_for_new_every * USEC_PER_SEC, find_dt = 0;
4864
4856 - while(!netdata_exit) {
4865 + while(service_running(SERVICE_COLLECTORS)) {
4866 worker_is_idle();
4867
4868 usec_t hb_dt = heartbeat_next(&hb, step);
4860 - if(unlikely(netdata_exit)) break;
4869 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
4870
4871 find_dt += hb_dt;
4872 if (unlikely(find_dt >= find_every || (!is_inside_k8s && cgroups_check))) {
@@ -4872,9 +4881,11 @@ void *cgroups_main(void *ptr) {
4881
4882 worker_is_busy(WORKER_CGROUPS_READ);
4883 read_all_discovered_cgroups(cgroup_root);
4884 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
4885
4886 worker_is_busy(WORKER_CGROUPS_CHART);
4887 update_cgroup_charts(cgroup_update_every);
4888 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
4889
4890 worker_is_idle();
4891 uv_mutex_unlock(&cgroup_root_mutex);
collectors/diskspace.plugin/plugin_diskspace.c
+8 -8
@@ -515,7 +515,7 @@ void *diskspace_slow_worker(void *ptr)
515 heartbeat_t hb;
516 heartbeat_init(&hb);
517
518 - while(!netdata_exit) {
518 + while(service_running(SERVICE_COLLECTORS)) {
519 worker_is_idle();
520 heartbeat_next(&hb, USEC_PER_SEC);
521
@@ -530,7 +530,7 @@ void *diskspace_slow_worker(void *ptr)
530 if (!dict_mountpoints)
531 continue;
532
533 - if(unlikely(netdata_exit)) break;
533 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
534
535 // --------------------------------------------------------------------------
536 // disk space metrics
@@ -547,10 +547,10 @@ void *diskspace_slow_worker(void *ptr)
547 for(bmi = slow_mountinfo_root; bmi; bmi = bmi->next) {
548 do_slow_disk_space_stats(bmi, slow_update_every);
549
550 - if(unlikely(netdata_exit)) break;
550 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
551 }
552
553 - if(unlikely(netdata_exit)) break;
553 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
554
555 worker_is_busy(WORKER_JOB_SLOW_CLEANUP);
556
@@ -640,11 +640,11 @@ void *diskspace_main(void *ptr) {
640 usec_t step = update_every * USEC_PER_SEC;
641 heartbeat_t hb;
642 heartbeat_init(&hb);
643 - while(!netdata_exit) {
643 + while(service_running(SERVICE_COLLECTORS)) {
644 worker_is_idle();
645 /* usec_t hb_dt = */ heartbeat_next(&hb, step);
646
647 - if(unlikely(netdata_exit)) break;
647 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
648
649 // --------------------------------------------------------------------------
650 // this is smart enough not to reload it every time
@@ -671,11 +671,11 @@ void *diskspace_main(void *ptr) {
671
672 worker_is_busy(WORKER_JOB_MOUNTPOINT);
673 do_disk_space_stats(mi, update_every);
674 - if(unlikely(netdata_exit)) break;
674 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
675 }
676 netdata_mutex_unlock(&slow_mountinfo_mutex);
677
678 - if(unlikely(netdata_exit)) break;
678 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
679
680 if(dict_mountpoints) {
681 worker_is_busy(WORKER_JOB_CLEANUP);
collectors/idlejitter.plugin/plugin_idlejitter.c
+1 -1
@@ -48,7 +48,7 @@ void *cpuidlejitter_main(void *ptr) {
48 usec_t update_every_ut = localhost->rrd_update_every * USEC_PER_SEC;
49 struct timeval before, after;
50
51 - while (!netdata_exit) {
51 + while (service_running(SERVICE_COLLECTORS)) {
52 int iterations = 0;
53 usec_t error_total = 0,
54 error_min = 0,
collectors/plugins.d/plugins_d.c
+4 -4
@@ -120,7 +120,7 @@ void *pluginsd_worker_thread(void *arg)
120 cd->obsolete = 0;
121 size_t count = 0;
122
123 - while (!netdata_exit) {
123 + while (service_running(SERVICE_COLLECTORS)) {
124 FILE *fp_child_input = NULL;
125 FILE *fp_child_output = netdata_popen(cd->cmd, &cd->pid, &fp_child_input);
126 if (unlikely(!fp_child_input || !fp_child_output)) {
@@ -186,12 +186,12 @@ void *pluginsd_main(void *ptr)
186 // so that we don't log broken directories on each loop
187 int directory_errors[PLUGINSD_MAX_DIRECTORIES] = { 0 };
188
189 - while (!netdata_exit) {
189 + while (service_running(SERVICE_COLLECTORS)) {
190 int idx;
191 const char *directory_name;
192
193 for (idx = 0; idx < PLUGINSD_MAX_DIRECTORIES && (directory_name = plugin_directories[idx]); idx++) {
194 - if (unlikely(netdata_exit))
194 + if (unlikely(!service_running(SERVICE_COLLECTORS)))
195 break;
196
197 errno = 0;
@@ -206,7 +206,7 @@ void *pluginsd_main(void *ptr)
206
207 struct dirent *file = NULL;
208 while (likely((file = readdir(dir)))) {
209 - if (unlikely(netdata_exit))
209 + if (unlikely(!service_running(SERVICE_COLLECTORS)))
210 break;
211
212 debug(D_PLUGINSD, "examining file '%s'", file->d_name);
collectors/plugins.d/pluginsd_parser.c
+14 -12
@@ -441,19 +441,20 @@ PARSER_RC pluginsd_dimension(char **words, size_t num_words, void *user)
441 } else
442 rrddim_isnot_obsolete(st, rd);
443
444 + bool should_update_dimension = false;
445 +
446 if (likely(unhide_dimension)) {
447 rrddim_option_clear(rd, RRDDIM_OPTION_HIDDEN);
446 - if (rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN)) {
447 - rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
448 - metaqueue_dimension_update_flags(rd);
449 - }
448 + should_update_dimension = rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN);
449 }
450 else {
451 rrddim_option_set(rd, RRDDIM_OPTION_HIDDEN);
453 - if (!rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN)) {
454 - rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);
455 - metaqueue_dimension_update_flags(rd);
456 - }
452 + should_update_dimension = !rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN);
453 + }
454 +
455 + if (should_update_dimension) {
456 + rrddim_flag_set(rd, RRDDIM_FLAG_METADATA_UPDATE);
457 + rrdhost_flag_set(rd->rrdset->rrdhost, RRDHOST_FLAG_METADATA_UPDATE);
458 }
459
460 return PARSER_RC_OK;
@@ -883,7 +884,7 @@ PARSER_RC pluginsd_overwrite(char **words __maybe_unused, size_t num_words __may
884 host->rrdlabels = rrdlabels_create();
885
886 rrdlabels_migrate_to_these(host->rrdlabels, (DICTIONARY *) (((PARSER_USER_OBJECT *)user)->new_host_labels));
886 - metaqueue_store_host_labels(host->machine_guid);
887 + rrdhost_flag_set(host, RRDHOST_FLAG_METADATA_LABELS | RRDHOST_FLAG_METADATA_UPDATE);
888
889 rrdlabels_destroy(((PARSER_USER_OBJECT *)user)->new_host_labels);
890 ((PARSER_USER_OBJECT *)user)->new_host_labels = NULL;
@@ -991,7 +992,7 @@ PARSER_RC pluginsd_replay_rrdset_begin(char **words, size_t num_words, void *use
992
993 if(start_time && end_time && start_time < wall_clock_time + tolerance && end_time < wall_clock_time + tolerance && start_time < end_time) {
994 if (unlikely(end_time - start_time != st->update_every))
994 - rrdset_set_update_every(st, end_time - start_time);
995 + rrdset_set_update_every_s(st, end_time - start_time);
996
997 st->last_collected_time.tv_sec = end_time;
998 st->last_collected_time.tv_usec = 0;
@@ -1251,6 +1252,7 @@ PARSER_RC pluginsd_replay_end(char **words, size_t num_words, void *user)
1252
1253 st->counter++;
1254 st->counter_done++;
1255 + store_metric_collection_completed();
1256
1257 #ifdef NETDATA_LOG_REPLICATION_REQUESTS
1258 st->replay.start_streaming = false;
@@ -1262,7 +1264,7 @@ PARSER_RC pluginsd_replay_end(char **words, size_t num_words, void *user)
1264
1265 if (start_streaming) {
1266 if (st->update_every != update_every_child)
1265 - rrdset_set_update_every(st, update_every_child);
1267 + rrdset_set_update_every_s(st, update_every_child);
1268
1269 if(rrdset_flag_check(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS)) {
1270 rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
@@ -1339,7 +1341,7 @@ inline size_t pluginsd_process(RRDHOST *host, struct plugind *cd, FILE *fp_plugi
1341 user.parser = parser;
1342
1343 while (likely(!parser_next(parser))) {
1342 - if (unlikely(netdata_exit || parser_action(parser, NULL)))
1344 + if (unlikely(!service_running(SERVICE_COLLECTORS) || parser_action(parser, NULL)))
1345 break;
1346 }
1347
collectors/proc.plugin/plugin_proc.c
+3 -3
@@ -128,15 +128,15 @@ void *proc_main(void *ptr)
128 heartbeat_t hb;
129 heartbeat_init(&hb);
130
131 - while (!netdata_exit) {
131 + while (service_running(SERVICE_COLLECTORS)) {
132 worker_is_idle();
133 usec_t hb_dt = heartbeat_next(&hb, step);
134
135 - if (unlikely(netdata_exit))
135 + if (unlikely(!service_running(SERVICE_COLLECTORS)))
136 break;
137
138 for (i = 0; proc_modules[i].name; i++) {
139 - if (unlikely(netdata_exit))
139 + if (unlikely(!service_running(SERVICE_COLLECTORS)))
140 break;
141
142 struct proc_module *pm = &proc_modules[i];
collectors/proc.plugin/proc_net_dev.c
+2 -2
@@ -1504,11 +1504,11 @@ void *netdev_main(void *ptr)
1504 heartbeat_t hb;
1505 heartbeat_init(&hb);
1506
1507 - while (!netdata_exit) {
1507 + while (service_running(SERVICE_COLLECTORS)) {
1508 worker_is_idle();
1509 usec_t hb_dt = heartbeat_next(&hb, step);
1510
1511 - if (unlikely(netdata_exit))
1511 + if (unlikely(!service_running(SERVICE_COLLECTORS)))
1512 break;
1513
1514 worker_is_busy(0);
collectors/statsd.plugin/statsd.c
+21 -8
@@ -234,7 +234,8 @@ typedef struct statsd_app {
234 // global statsd data
235
236 struct collection_thread_status {
237 - int status;
237 + SPINLOCK spinlock;
238 + bool running;
239 size_t max_sockets;
240
241 netdata_thread_t thread;
@@ -875,7 +876,7 @@ struct statsd_tcp {
876
877 #ifdef HAVE_RECVMMSG
878 struct statsd_udp {
878 - int *running;
879 + struct collection_thread_status *status;
880 STATSD_SOCKET_DATA_TYPE type;
881 size_t size;
882 struct iovec *iovecs;
@@ -1097,7 +1098,9 @@ static int statsd_snd_callback(POLLINFO *pi, short int *events) {
1098
1099 void statsd_collector_thread_cleanup(void *data) {
1100 struct statsd_udp *d = data;
1100 - *d->running = 0;
1101 + netdata_spinlock_lock(&d->status->spinlock);
1102 + d->status->running = false;
1103 + netdata_spinlock_unlock(&d->status->spinlock);
1104
1105 info("cleaning up...");
1106
@@ -1114,9 +1117,15 @@ void statsd_collector_thread_cleanup(void *data) {
1117 worker_unregister();
1118 }
1119
1120 +static bool statsd_should_stop(void) {
1121 + return !service_running(SERVICE_COLLECTORS);
1122 +}
1123 +
1124 void *statsd_collector_thread(void *ptr) {
1125 struct collection_thread_status *status = ptr;
1119 - status->status = 1;
1126 + netdata_spinlock_lock(&status->spinlock);
1127 + status->running = true;
1128 + netdata_spinlock_unlock(&status->spinlock);
1129
1130 worker_register("STATSD");
1131 worker_register_job_name(WORKER_JOB_TYPE_TCP_CONNECTED, "tcp connect");
@@ -1127,7 +1136,7 @@ void *statsd_collector_thread(void *ptr) {
1136 info("STATSD collector thread started with taskid %d", gettid());
1137
1138 struct statsd_udp *d = callocz(sizeof(struct statsd_udp), 1);
1130 - d->running = &status->status;
1139 + d->status = status;
1140
1141 netdata_thread_cleanup_push(statsd_collector_thread_cleanup, d);
1142
@@ -1152,6 +1161,7 @@ void *statsd_collector_thread(void *ptr) {
1161 , statsd_rcv_callback
1162 , statsd_snd_callback
1163 , NULL
1164 + , statsd_should_stop
1165 , NULL // No access control pattern
1166 , 0 // No dns lookups for access control pattern
1167 , (void *)d
@@ -2358,13 +2368,15 @@ static void statsd_main_cleanup(void *data) {
2368 if (statsd.collection_threads_status) {
2369 int i;
2370 for (i = 0; i < statsd.threads; i++) {
2361 - if(statsd.collection_threads_status[i].status) {
2371 + netdata_spinlock_lock(&statsd.collection_threads_status[i].spinlock);
2372 + if(statsd.collection_threads_status[i].running) {
2373 info("STATSD: stopping data collection thread %d...", i + 1);
2374 netdata_thread_cancel(statsd.collection_threads_status[i].thread);
2375 }
2376 else {
2377 info("STATSD: data collection thread %d found stopped.", i + 1);
2378 }
2379 + netdata_spinlock_unlock(&statsd.collection_threads_status[i].spinlock);
2380 }
2381 }
2382
@@ -2537,6 +2549,7 @@ void *statsd_main(void *ptr) {
2549 statsd.collection_threads_status[i].max_sockets = max_sockets / statsd.threads;
2550 char tag[NETDATA_THREAD_TAG_MAX + 1];
2551 snprintfz(tag, NETDATA_THREAD_TAG_MAX, "STATSD_COLLECTOR[%d]", i + 1);
2552 + netdata_spinlock_init(&statsd.collection_threads_status[i].spinlock);
2553 netdata_thread_create(&statsd.collection_threads_status[i].thread, tag, NETDATA_THREAD_OPTION_DEFAULT, statsd_collector_thread, &statsd.collection_threads_status[i]);
2554 }
2555
@@ -2753,7 +2766,7 @@ void *statsd_main(void *ptr) {
2766 usec_t step = statsd.update_every * USEC_PER_SEC;
2767 heartbeat_t hb;
2768 heartbeat_init(&hb);
2756 - while(!netdata_exit) {
2769 + while(service_running(SERVICE_COLLECTORS)) {
2770 worker_is_idle();
2771 heartbeat_next(&hb, step);
2772
@@ -2781,7 +2794,7 @@ void *statsd_main(void *ptr) {
2794 worker_is_busy(WORKER_STATSD_FLUSH_STATS);
2795 statsd_update_all_app_charts();
2796
2784 - if(unlikely(netdata_exit))
2797 + if(unlikely(!service_running(SERVICE_COLLECTORS)))
2798 break;
2799
2800 if(global_statistics_enabled) {
collectors/tc.plugin/plugin_tc.c
+3 -3
@@ -929,7 +929,7 @@ void *tc_main(void *ptr) {
929 snprintfz(command, TC_LINE_MAX, "%s/tc-qos-helper.sh", netdata_configured_primary_plugins_dir);
930 char *tc_script = config_get("plugin:tc", "script to run to get tc values", command);
931
932 - while(!netdata_exit) {
932 + while(service_running(SERVICE_COLLECTORS)) {
933 FILE *fp_child_input, *fp_child_output;
934 struct tc_device *device = NULL;
935 struct tc_class *class = NULL;
@@ -945,7 +945,7 @@ void *tc_main(void *ptr) {
945
946 char buffer[TC_LINE_MAX+1] = "";
947 while(fgets(buffer, TC_LINE_MAX, fp_child_output) != NULL) {
948 - if(unlikely(netdata_exit)) break;
948 + if(unlikely(!service_running(SERVICE_COLLECTORS))) break;
949
950 buffer[TC_LINE_MAX] = '\0';
951 // debug(D_TC_LOOP, "TC: read '%s'", buffer);
@@ -1162,7 +1162,7 @@ void *tc_main(void *ptr) {
1162 class = NULL;
1163 }
1164
1165 - if(unlikely(netdata_exit))
1165 + if(unlikely(!service_running(SERVICE_COLLECTORS)))
1166 goto cleanup;
1167
1168 if(code == 1 || code == 127) {
collectors/timex.plugin/plugin_timex.c
+1 -1
@@ -64,7 +64,7 @@ void *timex_main(void *ptr)
64 usec_t step = update_every * USEC_PER_SEC;
65 heartbeat_t hb;
66 heartbeat_init(&hb);
67 - while (!netdata_exit) {
67 + while (service_running(SERVICE_COLLECTORS)) {
68 worker_is_idle();
69 heartbeat_next(&hb, step);
70 worker_is_busy(0);
configure.ac
+1
@@ -1738,6 +1738,7 @@ AC_CONFIG_FILES([
1738 libnetdata/dictionary/Makefile
1739 libnetdata/ebpf/Makefile
1740 libnetdata/eval/Makefile
1741 + libnetdata/july/Makefile
1742 libnetdata/locks/Makefile
1743 libnetdata/log/Makefile
1744 libnetdata/onewayalloc/Makefile
daemon/analytics.c
+5 -7
@@ -223,9 +223,7 @@ void analytics_mirrored_hosts(void)
223 if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))
224 continue;
225
226 - netdata_mutex_lock(&host->receiver_lock);
227 - ((host->receiver || host == localhost) ? reachable++ : unreachable++);
228 - netdata_mutex_unlock(&host->receiver_lock);
226 + ((host == localhost || !rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN)) ? reachable++ : unreachable++);
227
228 count++;
229 }
@@ -554,7 +552,7 @@ void analytics_gather_mutable_meta_data(void)
552 snprintfz(b, 6, "%d", analytics_data.dashboard_hits);
553 analytics_set_data(&analytics_data.netdata_dashboard_used, b);
554
557 - snprintfz(b, 6, "%zu", rrd_hosts_available);
555 + snprintfz(b, 6, "%zu", rrdhost_hosts_available());
556 analytics_set_data(&analytics_data.netdata_config_hosts_available, b);
557 }
558 }
@@ -587,12 +585,12 @@ void *analytics_main(void *ptr)
585 debug(D_ANALYTICS, "Analytics thread starts");
586
587 //first delay after agent start
590 - while (!netdata_exit && likely(sec <= ANALYTICS_INIT_SLEEP_SEC)) {
588 + while (service_running(SERVICE_ANALYTICS) && likely(sec <= ANALYTICS_INIT_SLEEP_SEC)) {
589 heartbeat_next(&hb, step_ut);
590 sec++;
591 }
592
595 - if (unlikely(netdata_exit))
593 + if (unlikely(!service_running(SERVICE_ANALYTICS)))
594 goto cleanup;
595
596 analytics_gather_immutable_meta_data();
@@ -605,7 +603,7 @@ void *analytics_main(void *ptr)
603 heartbeat_next(&hb, step_ut * 2);
604 sec += 2;
605
608 - if (unlikely(netdata_exit))
606 + if (unlikely(!service_running(SERVICE_ANALYTICS)))
607 break;
608
609 if (likely(sec < ANALYTICS_HEARTBEAT))
daemon/commands.c
+5 -1
@@ -470,9 +470,13 @@ static void after_schedule_command(uv_work_t *req, int status)
470
471 static void schedule_command(uv_work_t *req)
472 {
473 - struct command_context *cmd_ctx = req->data;
473 + register_libuv_worker_jobs();
474 + worker_is_busy(UV_EVENT_SCHEDULE_CMD);
475
476 + struct command_context *cmd_ctx = req->data;
477 cmd_ctx->status = execute_command(cmd_ctx->idx, cmd_ctx->args, &cmd_ctx->message);
478 +
479 + worker_is_idle();
480 }
481
482 /* This will alter the state of the command_info_array.cmd_str
daemon/common.h
+1
@@ -4,6 +4,7 @@
4 #define NETDATA_COMMON_H 1
5
6 #include "libnetdata/libnetdata.h"
7 +#include "event_loop.h"
8
9 // ----------------------------------------------------------------------------
10 // shortcuts for the default netdata configuration
daemon/event_loop.c new
+49
@@ -0,0 +1,49 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include <daemon/main.h>
4 +#include "event_loop.h"
5 +
6 +// Register workers
7 +void register_libuv_worker_jobs() {
8 + static __thread bool registered = false;
9 +
10 + if(likely(registered))
11 + return;
12 +
13 + registered = true;
14 +
15 + worker_register("LIBUV");
16 + worker_register_job_name(UV_EVENT_READ_PAGE_CB, "read page cb");
17 + worker_register_job_name(UV_EVENT_READ_EXTENT_CB, "read extent cb");
18 + worker_register_job_name(UV_EVENT_COMMIT_PAGE_CB, "commit cb");
19 + worker_register_job_name(UV_EVENT_FLUSH_PAGES_CB, "flush cb");
20 + worker_register_job_name(UV_EVENT_PAGE_LOOKUP, "page lookup");
21 + worker_register_job_name(UV_EVENT_METRIC_LOOKUP, "metric lookup");
22 + worker_register_job_name(UV_EVENT_PAGE_POPULATION, "populate page");
23 + worker_register_job_name(UV_EVENT_EXT_DECOMPRESSION, "extent decompression");
24 + worker_register_job_name(UV_EVENT_READ_MMAP_EXTENT, "read extent (mmap)");
25 + worker_register_job_name(UV_EVENT_EXTENT_PROCESSING, "extent processing");
26 + worker_register_job_name(UV_EVENT_METADATA_STORE, "store host metadata");
27 + worker_register_job_name(UV_EVENT_JOURNAL_INDEX_WAIT, "journal v2 wait");
28 + worker_register_job_name(UV_EVENT_JOURNAL_INDEX, "journal v2 indexing");
29 + worker_register_job_name(UV_EVENT_SCHEDULE_CMD, "schedule command");
30 + worker_register_job_name(UV_EVENT_METADATA_CLEANUP, "metadata cleanup");
31 + worker_register_job_name(UV_EVENT_EXTENT_CACHE, "extent cache");
32 + worker_register_job_name(UV_EVENT_EXTENT_MMAP, "extent mmap");
33 + worker_register_job_name(UV_EVENT_PAGE_DISPATCH, "dispatch page list");
34 + worker_register_job_name(UV_EVENT_FLUSH_CALLBACK, "flush callback");
35 + worker_register_job_name(UV_EVENT_FLUSH_MAIN, "flush main");
36 + worker_register_job_name(UV_EVENT_FLUSH_OPEN, "flush open");
37 + worker_register_job_name(UV_EVENT_EVICT_MAIN, "evict main");
38 + worker_register_job_name(UV_EVENT_DELETING_FILE, "delete datafiles");
39 + worker_register_job_name(UV_EVENT_ANALYZE_V2, "analyze journalfile");
40 + worker_register_job_name(UV_EVENT_RETENTION_V2, "calculate retention");
41 + worker_register_job_name(UV_EVENT_RETENTION_UPDATE, "update retention");
42 + worker_register_job_name(UV_EVENT_DATAFILE_ACQUIRE, "datafile acquire");
43 + worker_register_job_name(UV_EVENT_DATAFILE_DELETE, "datafile deletion");
44 + worker_register_job_name(UV_EVENT_FLUSHED_TO_OPEN, "flushed to open");
45 + worker_register_job_name(UV_EVENT_PREP_QUERY, "prep query");
46 + worker_register_job_name(UV_EVENT_WORKER_INIT, "worker init");
47 +
48 + uv_thread_set_name_np(pthread_self(), "LIBUV_WORKER");
49 +}
daemon/event_loop.h new
+44
@@ -0,0 +1,44 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_EVENT_LOOP_H
4 +#define NETDATA_EVENT_LOOP_H
5 +
6 +enum event_loop_job {
7 + UV_EVENT_JOB_NONE = 0,
8 + UV_EVENT_READ_PAGE_CB,
9 + UV_EVENT_READ_EXTENT_CB,
10 + UV_EVENT_COMMIT_PAGE_CB,
11 + UV_EVENT_FLUSH_PAGES_CB,
12 + UV_EVENT_EXT_DECOMPRESSION,
13 + UV_EVENT_PAGE_LOOKUP,
14 + UV_EVENT_METRIC_LOOKUP,
15 + UV_EVENT_PAGE_POPULATION,
16 + UV_EVENT_READ_MMAP_EXTENT,
17 + UV_EVENT_EXTENT_PROCESSING,
18 + UV_EVENT_METADATA_STORE,
19 + UV_EVENT_JOURNAL_INDEX_WAIT,
20 + UV_EVENT_JOURNAL_INDEX,
21 + UV_EVENT_SCHEDULE_CMD,
22 + UV_EVENT_METADATA_CLEANUP,
23 + UV_EVENT_EXTENT_CACHE,
24 + UV_EVENT_EXTENT_MMAP,
25 + UV_EVENT_FLUSH_CALLBACK,
26 + UV_EVENT_EXTEXT_DISPATCH,
27 + UV_EVENT_FLUSH_MAIN,
28 + UV_EVENT_FLUSH_OPEN,
29 + UV_EVENT_EVICT_MAIN,
30 + UV_EVENT_PAGE_DISPATCH,
31 + UV_EVENT_DELETING_FILE,
32 + UV_EVENT_ANALYZE_V2,
33 + UV_EVENT_RETENTION_V2,
34 + UV_EVENT_RETENTION_UPDATE,
35 + UV_EVENT_DATAFILE_ACQUIRE,
36 + UV_EVENT_DATAFILE_DELETE,
37 + UV_EVENT_FLUSHED_TO_OPEN,
38 + UV_EVENT_PREP_QUERY,
39 + UV_EVENT_WORKER_INIT,
40 +};
41 +
42 +void register_libuv_worker_jobs();
43 +
44 +#endif //NETDATA_EVENT_LOOP_H
daemon/global_statistics.c
+1232 -260
@@ -669,15 +669,15 @@ static void global_statistics_charts(void) {
669 "netdata" // type
670 , "ml_models_consulted" // id
671 , NULL // name
672 - , "ml" // family
672 + , NETDATA_ML_CHART_FAMILY // family
673 , NULL // context
674 , "KMeans models used for prediction" // title
675 , "models" // units
676 - , "netdata" // plugin
677 - , "ml" // module
678 - , 131004 // priority
676 + , NETDATA_ML_PLUGIN // plugin
677 + , NETDATA_ML_MODULE_DETECTION // module
678 + , NETDATA_ML_CHART_PRIO_MACHINE_LEARNING_STATUS // priority
679 , localhost->rrd_update_every // update_every
680 - , RRDSET_TYPE_STACKED // chart_type
680 + , RRDSET_TYPE_AREA // chart_type
681 );
682
683 rd = rrddim_add(st, "num_models_consulted", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
@@ -996,9 +996,1119 @@ static void sqlite3_statistics_charts(void) {
996 // ----------------------------------------------------------------
997 }
998
999 -static void dbengine_statistics_charts(void) {
1000 -#ifdef ENABLE_DBENGINE
999 +struct dbengine2_cache_pointers {
1000 + RRDSET *st_cache_hit_ratio;
1001 + RRDDIM *rd_hit_ratio_closest;
1002 + RRDDIM *rd_hit_ratio_exact;
1003 +
1004 + RRDSET *st_operations;
1005 + RRDDIM *rd_searches_closest;
1006 + RRDDIM *rd_searches_exact;
1007 + RRDDIM *rd_add_hot;
1008 + RRDDIM *rd_add_clean;
1009 + RRDDIM *rd_evictions;
1010 + RRDDIM *rd_flushes;
1011 + RRDDIM *rd_acquires;
1012 + RRDDIM *rd_releases;
1013 + RRDDIM *rd_acquires_for_deletion;
1014 +
1015 + RRDSET *st_pgc_memory;
1016 + RRDDIM *rd_pgc_memory_free;
1017 + RRDDIM *rd_pgc_memory_clean;
1018 + RRDDIM *rd_pgc_memory_hot;
1019 + RRDDIM *rd_pgc_memory_dirty;
1020 + RRDDIM *rd_pgc_memory_index;
1021 + RRDDIM *rd_pgc_memory_evicting;
1022 + RRDDIM *rd_pgc_memory_flushing;
1023 +
1024 + RRDSET *st_pgc_pages;
1025 + RRDDIM *rd_pgc_pages_clean;
1026 + RRDDIM *rd_pgc_pages_hot;
1027 + RRDDIM *rd_pgc_pages_dirty;
1028 + RRDDIM *rd_pgc_pages_referenced;
1029 +
1030 + RRDSET *st_pgc_memory_changes;
1031 + RRDDIM *rd_pgc_memory_new_hot;
1032 + RRDDIM *rd_pgc_memory_new_clean;
1033 + RRDDIM *rd_pgc_memory_clean_evictions;
1034 +
1035 + RRDSET *st_pgc_memory_migrations;
1036 + RRDDIM *rd_pgc_memory_hot_to_dirty;
1037 + RRDDIM *rd_pgc_memory_dirty_to_clean;
1038 +
1039 + RRDSET *st_pgc_workers;
1040 + RRDDIM *rd_pgc_workers_evictors;
1041 + RRDDIM *rd_pgc_workers_flushers;
1042 + RRDDIM *rd_pgc_workers_adders;
1043 + RRDDIM *rd_pgc_workers_searchers;
1044 + RRDDIM *rd_pgc_workers_jv2_flushers;
1045 + RRDDIM *rd_pgc_workers_hot2dirty;
1046 +
1047 + RRDSET *st_pgc_memory_events;
1048 + RRDDIM *rd_pgc_memory_evictions_critical;
1049 + RRDDIM *rd_pgc_memory_evictions_aggressive;
1050 + RRDDIM *rd_pgc_memory_flushes_critical;
1051 +
1052 + RRDSET *st_pgc_waste;
1053 + RRDDIM *rd_pgc_waste_evictions_skipped;
1054 + RRDDIM *rd_pgc_waste_flushes_cancelled;
1055 + RRDDIM *rd_pgc_waste_insert_spins;
1056 + RRDDIM *rd_pgc_waste_evict_spins;
1057 + RRDDIM *rd_pgc_waste_release_spins;
1058 + RRDDIM *rd_pgc_waste_acquire_spins;
1059 + RRDDIM *rd_pgc_waste_delete_spins;
1060 + RRDDIM *rd_pgc_waste_flush_spins;
1061 +
1062 +};
1063 +
1064 +static void dbengine2_cache_statistics_charts(struct dbengine2_cache_pointers *ptrs, struct pgc_statistics *pgc_stats, struct pgc_statistics *pgc_stats_old __maybe_unused, const char *name, int priority) {
1065 +
1066 + {
1067 + if (unlikely(!ptrs->st_cache_hit_ratio)) {
1068 + BUFFER *id = buffer_create(100);
1069 + buffer_sprintf(id, "dbengine_%s_cache_hit_ratio", name);
1070 +
1071 + BUFFER *family = buffer_create(100);
1072 + buffer_sprintf(family, "dbengine %s cache", name);
1073 +
1074 + BUFFER *title = buffer_create(100);
1075 + buffer_sprintf(title, "Netdata %s Cache Hit Ratio", name);
1076 +
1077 + ptrs->st_cache_hit_ratio = rrdset_create_localhost(
1078 + "netdata",
1079 + buffer_tostring(id),
1080 + NULL,
1081 + buffer_tostring(family),
1082 + NULL,
1083 + buffer_tostring(title),
1084 + "%",
1085 + "netdata",
1086 + "stats",
1087 + priority,
1088 + localhost->rrd_update_every,
1089 + RRDSET_TYPE_LINE);
1090 +
1091 + ptrs->rd_hit_ratio_closest = rrddim_add(ptrs->st_cache_hit_ratio, "closest", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1092 + ptrs->rd_hit_ratio_exact = rrddim_add(ptrs->st_cache_hit_ratio, "exact", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1093 +
1094 + buffer_free(id);
1095 + buffer_free(family);
1096 + buffer_free(title);
1097 + priority++;
1098 + }
1099 +
1100 + size_t closest_percent = 100 * 10000;
1101 + if(pgc_stats->searches_closest > pgc_stats_old->searches_closest)
1102 + closest_percent = (pgc_stats->searches_closest_hits - pgc_stats_old->searches_closest_hits) * 100 * 10000 / (pgc_stats->searches_closest - pgc_stats_old->searches_closest);
1103 +
1104 + size_t exact_percent = 100 * 10000;
1105 + if(pgc_stats->searches_exact > pgc_stats_old->searches_exact)
1106 + exact_percent = (pgc_stats->searches_exact_hits - pgc_stats_old->searches_exact_hits) * 100 * 10000 / (pgc_stats->searches_exact - pgc_stats_old->searches_exact);
1107 +
1108 + rrddim_set_by_pointer(ptrs->st_cache_hit_ratio, ptrs->rd_hit_ratio_closest, (collected_number)closest_percent);
1109 + rrddim_set_by_pointer(ptrs->st_cache_hit_ratio, ptrs->rd_hit_ratio_exact, (collected_number)exact_percent);
1110 +
1111 + rrdset_done(ptrs->st_cache_hit_ratio);
1112 + }
1113 +
1114 + {
1115 + if (unlikely(!ptrs->st_operations)) {
1116 + BUFFER *id = buffer_create(100);
1117 + buffer_sprintf(id, "dbengine_%s_cache_operations", name);
1118 +
1119 + BUFFER *family = buffer_create(100);
1120 + buffer_sprintf(family, "dbengine %s cache", name);
1121 +
1122 + BUFFER *title = buffer_create(100);
1123 + buffer_sprintf(title, "Netdata %s Cache Operations", name);
1124 +
1125 + ptrs->st_operations = rrdset_create_localhost(
1126 + "netdata",
1127 + buffer_tostring(id),
1128 + NULL,
1129 + buffer_tostring(family),
1130 + NULL,
1131 + buffer_tostring(title),
1132 + "ops/s",
1133 + "netdata",
1134 + "stats",
1135 + priority,
1136 + localhost->rrd_update_every,
1137 + RRDSET_TYPE_LINE);
1138 +
1139 + ptrs->rd_searches_closest = rrddim_add(ptrs->st_operations, "search closest", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1140 + ptrs->rd_searches_exact = rrddim_add(ptrs->st_operations, "search exact", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1141 + ptrs->rd_add_hot = rrddim_add(ptrs->st_operations, "add hot", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1142 + ptrs->rd_add_clean = rrddim_add(ptrs->st_operations, "add clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1143 + ptrs->rd_evictions = rrddim_add(ptrs->st_operations, "evictions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1144 + ptrs->rd_flushes = rrddim_add(ptrs->st_operations, "flushes", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1145 + ptrs->rd_acquires = rrddim_add(ptrs->st_operations, "acquires", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1146 + ptrs->rd_releases = rrddim_add(ptrs->st_operations, "releases", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1147 + ptrs->rd_acquires_for_deletion = rrddim_add(ptrs->st_operations, "del acquires", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1148 +
1149 + buffer_free(id);
1150 + buffer_free(family);
1151 + buffer_free(title);
1152 + priority++;
1153 + }
1154 +
1155 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_searches_closest, (collected_number)pgc_stats->searches_closest);
1156 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_searches_exact, (collected_number)pgc_stats->searches_exact);
1157 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_add_hot, (collected_number)pgc_stats->queues.hot.added_entries);
1158 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_add_clean, (collected_number)(pgc_stats->added_entries - pgc_stats->queues.hot.added_entries));
1159 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_evictions, (collected_number)pgc_stats->queues.clean.removed_entries);
1160 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_flushes, (collected_number)pgc_stats->flushes_completed);
1161 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_acquires, (collected_number)pgc_stats->acquires);
1162 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_releases, (collected_number)pgc_stats->releases);
1163 + rrddim_set_by_pointer(ptrs->st_operations, ptrs->rd_acquires_for_deletion, (collected_number)pgc_stats->acquires_for_deletion);
1164 +
1165 + rrdset_done(ptrs->st_operations);
1166 + }
1167 +
1168 + {
1169 +
1170 + if (unlikely(!ptrs->st_pgc_memory)) {
1171 + BUFFER *id = buffer_create(100);
1172 + buffer_sprintf(id, "dbengine_%s_cache_memory", name);
1173 +
1174 + BUFFER *family = buffer_create(100);
1175 + buffer_sprintf(family, "dbengine %s cache", name);
1176 +
1177 + BUFFER *title = buffer_create(100);
1178 + buffer_sprintf(title, "Netdata %s Cache Memory", name);
1179 +
1180 + ptrs->st_pgc_memory = rrdset_create_localhost(
1181 + "netdata",
1182 + buffer_tostring(id),
1183 + NULL,
1184 + buffer_tostring(family),
1185 + NULL,
1186 + buffer_tostring(title),
1187 + "bytes",
1188 + "netdata",
1189 + "stats",
1190 + priority,
1191 + localhost->rrd_update_every,
1192 + RRDSET_TYPE_STACKED);
1193 +
1194 + ptrs->rd_pgc_memory_free = rrddim_add(ptrs->st_pgc_memory, "free", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1195 + ptrs->rd_pgc_memory_hot = rrddim_add(ptrs->st_pgc_memory, "hot", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1196 + ptrs->rd_pgc_memory_dirty = rrddim_add(ptrs->st_pgc_memory, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1197 + ptrs->rd_pgc_memory_clean = rrddim_add(ptrs->st_pgc_memory, "clean", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1198 + ptrs->rd_pgc_memory_index = rrddim_add(ptrs->st_pgc_memory, "index", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1199 + ptrs->rd_pgc_memory_evicting = rrddim_add(ptrs->st_pgc_memory, "evicting", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1200 + ptrs->rd_pgc_memory_flushing = rrddim_add(ptrs->st_pgc_memory, "flushing", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1201 +
1202 + buffer_free(id);
1203 + buffer_free(family);
1204 + buffer_free(title);
1205 + priority++;
1206 + }
1207 +
1208 + collected_number free = (pgc_stats->current_cache_size > pgc_stats->wanted_cache_size) ? 0 :
1209 + (collected_number)(pgc_stats->wanted_cache_size - pgc_stats->current_cache_size);
1210 +
1211 + rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_free, free);
1212 + rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_hot, (collected_number)pgc_stats->queues.hot.size);
1213 + rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_dirty, (collected_number)pgc_stats->queues.dirty.size);
1214 + rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_clean, (collected_number)pgc_stats->queues.clean.size);
1215 + rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_evicting, (collected_number)pgc_stats->evicting_size);
1216 + rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_flushing, (collected_number)pgc_stats->flushing_size);
1217 + rrddim_set_by_pointer(ptrs->st_pgc_memory, ptrs->rd_pgc_memory_index,
1218 + (collected_number)(pgc_stats->size - pgc_stats->queues.clean.size - pgc_stats->queues.hot.size - pgc_stats->queues.dirty.size - pgc_stats->evicting_size - pgc_stats->flushing_size));
1219 +
1220 + rrdset_done(ptrs->st_pgc_memory);
1221 + }
1222 +
1223 + {
1224 + if (unlikely(!ptrs->st_pgc_pages)) {
1225 + BUFFER *id = buffer_create(100);
1226 + buffer_sprintf(id, "dbengine_%s_cache_pages", name);
1227 +
1228 + BUFFER *family = buffer_create(100);
1229 + buffer_sprintf(family, "dbengine %s cache", name);
1230 +
1231 + BUFFER *title = buffer_create(100);
1232 + buffer_sprintf(title, "Netdata %s Cache Pages", name);
1233 +
1234 + ptrs->st_pgc_pages = rrdset_create_localhost(
1235 + "netdata",
1236 + buffer_tostring(id),
1237 + NULL,
1238 + buffer_tostring(family),
1239 + NULL,
1240 + buffer_tostring(title),
1241 + "pages",
1242 + "netdata",
1243 + "stats",
1244 + priority,
1245 + localhost->rrd_update_every,
1246 + RRDSET_TYPE_LINE);
1247 +
1248 + ptrs->rd_pgc_pages_clean = rrddim_add(ptrs->st_pgc_pages, "clean", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1249 + ptrs->rd_pgc_pages_hot = rrddim_add(ptrs->st_pgc_pages, "hot", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1250 + ptrs->rd_pgc_pages_dirty = rrddim_add(ptrs->st_pgc_pages, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1251 + ptrs->rd_pgc_pages_referenced = rrddim_add(ptrs->st_pgc_pages, "referenced", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1252 +
1253 + buffer_free(id);
1254 + buffer_free(family);
1255 + buffer_free(title);
1256 + priority++;
1257 + }
1258 +
1259 + rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_clean, (collected_number)pgc_stats->queues.clean.entries);
1260 + rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_hot, (collected_number)pgc_stats->queues.hot.entries);
1261 + rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_dirty, (collected_number)pgc_stats->queues.dirty.entries);
1262 + rrddim_set_by_pointer(ptrs->st_pgc_pages, ptrs->rd_pgc_pages_referenced, (collected_number)pgc_stats->referenced_entries);
1263 +
1264 + rrdset_done(ptrs->st_pgc_pages);
1265 + }
1266 +
1267 + {
1268 + if (unlikely(!ptrs->st_pgc_memory_changes)) {
1269 + BUFFER *id = buffer_create(100);
1270 + buffer_sprintf(id, "dbengine_%s_cache_memory_changes", name);
1271 +
1272 + BUFFER *family = buffer_create(100);
1273 + buffer_sprintf(family, "dbengine %s cache", name);
1274 +
1275 + BUFFER *title = buffer_create(100);
1276 + buffer_sprintf(title, "Netdata %s Cache Memory Changes", name);
1277 +
1278 + ptrs->st_pgc_memory_changes = rrdset_create_localhost(
1279 + "netdata",
1280 + buffer_tostring(id),
1281 + NULL,
1282 + buffer_tostring(family),
1283 + NULL,
1284 + buffer_tostring(title),
1285 + "bytes/s",
1286 + "netdata",
1287 + "stats",
1288 + priority,
1289 + localhost->rrd_update_every,
1290 + RRDSET_TYPE_AREA);
1291 +
1292 + ptrs->rd_pgc_memory_new_clean = rrddim_add(ptrs->st_pgc_memory_changes, "new clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1293 + ptrs->rd_pgc_memory_clean_evictions = rrddim_add(ptrs->st_pgc_memory_changes, "evictions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1294 + ptrs->rd_pgc_memory_new_hot = rrddim_add(ptrs->st_pgc_memory_changes, "new hot", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1295 +
1296 + buffer_free(id);
1297 + buffer_free(family);
1298 + buffer_free(title);
1299 + priority++;
1300 + }
1301 +
1302 + rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_new_clean, (collected_number)(pgc_stats->added_size - pgc_stats->queues.hot.added_size));
1303 + rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_clean_evictions, (collected_number)pgc_stats->queues.clean.removed_size);
1304 + rrddim_set_by_pointer(ptrs->st_pgc_memory_changes, ptrs->rd_pgc_memory_new_hot, (collected_number)pgc_stats->queues.hot.added_size);
1305 +
1306 + rrdset_done(ptrs->st_pgc_memory_changes);
1307 + }
1308 +
1309 + {
1310 + if (unlikely(!ptrs->st_pgc_memory_migrations)) {
1311 + BUFFER *id = buffer_create(100);
1312 + buffer_sprintf(id, "dbengine_%s_cache_memory_migrations", name);
1313 +
1314 + BUFFER *family = buffer_create(100);
1315 + buffer_sprintf(family, "dbengine %s cache", name);
1316 +
1317 + BUFFER *title = buffer_create(100);
1318 + buffer_sprintf(title, "Netdata %s Cache Memory Migrations", name);
1319 +
1320 + ptrs->st_pgc_memory_migrations = rrdset_create_localhost(
1321 + "netdata",
1322 + buffer_tostring(id),
1323 + NULL,
1324 + buffer_tostring(family),
1325 + NULL,
1326 + buffer_tostring(title),
1327 + "bytes/s",
1328 + "netdata",
1329 + "stats",
1330 + priority,
1331 + localhost->rrd_update_every,
1332 + RRDSET_TYPE_AREA);
1333 +
1334 + ptrs->rd_pgc_memory_dirty_to_clean = rrddim_add(ptrs->st_pgc_memory_migrations, "dirty to clean", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1335 + ptrs->rd_pgc_memory_hot_to_dirty = rrddim_add(ptrs->st_pgc_memory_migrations, "hot to dirty", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1336 +
1337 + buffer_free(id);
1338 + buffer_free(family);
1339 + buffer_free(title);
1340 + priority++;
1341 + }
1342 +
1343 + rrddim_set_by_pointer(ptrs->st_pgc_memory_migrations, ptrs->rd_pgc_memory_dirty_to_clean, (collected_number)pgc_stats->queues.dirty.removed_size);
1344 + rrddim_set_by_pointer(ptrs->st_pgc_memory_migrations, ptrs->rd_pgc_memory_hot_to_dirty, (collected_number)pgc_stats->queues.dirty.added_size);
1345 +
1346 + rrdset_done(ptrs->st_pgc_memory_migrations);
1347 + }
1348 +
1349 + {
1350 + if (unlikely(!ptrs->st_pgc_memory_events)) {
1351 + BUFFER *id = buffer_create(100);
1352 + buffer_sprintf(id, "dbengine_%s_cache_events", name);
1353 +
1354 + BUFFER *family = buffer_create(100);
1355 + buffer_sprintf(family, "dbengine %s cache", name);
1356 +
1357 + BUFFER *title = buffer_create(100);
1358 + buffer_sprintf(title, "Netdata %s Cache Events", name);
1359 +
1360 + ptrs->st_pgc_memory_events = rrdset_create_localhost(
1361 + "netdata",
1362 + buffer_tostring(id),
1363 + NULL,
1364 + buffer_tostring(family),
1365 + NULL,
1366 + buffer_tostring(title),
1367 + "events/s",
1368 + "netdata",
1369 + "stats",
1370 + priority,
1371 + localhost->rrd_update_every,
1372 + RRDSET_TYPE_AREA);
1373 +
1374 + ptrs->rd_pgc_memory_evictions_aggressive = rrddim_add(ptrs->st_pgc_memory_events, "evictions critical", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1375 + ptrs->rd_pgc_memory_evictions_critical = rrddim_add(ptrs->st_pgc_memory_events, "evictions aggressive", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1376 + ptrs->rd_pgc_memory_flushes_critical = rrddim_add(ptrs->st_pgc_memory_events, "flushes critical", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1377 +
1378 + buffer_free(id);
1379 + buffer_free(family);
1380 + buffer_free(title);
1381 + priority++;
1382 + }
1383 +
1384 + rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_evictions_aggressive, (collected_number)pgc_stats->events_cache_needs_space_aggressively);
1385 + rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_evictions_critical, (collected_number)pgc_stats->events_cache_under_severe_pressure);
1386 + rrddim_set_by_pointer(ptrs->st_pgc_memory_events, ptrs->rd_pgc_memory_flushes_critical, (collected_number)pgc_stats->events_flush_critical);
1387 +
1388 + rrdset_done(ptrs->st_pgc_memory_events);
1389 + }
1390 +
1391 + {
1392 + if (unlikely(!ptrs->st_pgc_waste)) {
1393 + BUFFER *id = buffer_create(100);
1394 + buffer_sprintf(id, "dbengine_%s_waste_events", name);
1395 +
1396 + BUFFER *family = buffer_create(100);
1397 + buffer_sprintf(family, "dbengine %s cache", name);
1398 +
1399 + BUFFER *title = buffer_create(100);
1400 + buffer_sprintf(title, "Netdata %s Waste Events", name);
1401 +
1402 + ptrs->st_pgc_waste = rrdset_create_localhost(
1403 + "netdata",
1404 + buffer_tostring(id),
1405 + NULL,
1406 + buffer_tostring(family),
1407 + NULL,
1408 + buffer_tostring(title),
1409 + "events/s",
1410 + "netdata",
1411 + "stats",
1412 + priority,
1413 + localhost->rrd_update_every,
1414 + RRDSET_TYPE_LINE);
1415 +
1416 + ptrs->rd_pgc_waste_evictions_skipped = rrddim_add(ptrs->st_pgc_waste, "evictions skipped", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1417 + ptrs->rd_pgc_waste_flushes_cancelled = rrddim_add(ptrs->st_pgc_waste, "flushes cancelled", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1418 + ptrs->rd_pgc_waste_acquire_spins = rrddim_add(ptrs->st_pgc_waste, "acquire spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1419 + ptrs->rd_pgc_waste_release_spins = rrddim_add(ptrs->st_pgc_waste, "release spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1420 + ptrs->rd_pgc_waste_insert_spins = rrddim_add(ptrs->st_pgc_waste, "insert spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1421 + ptrs->rd_pgc_waste_delete_spins = rrddim_add(ptrs->st_pgc_waste, "delete spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1422 + ptrs->rd_pgc_waste_evict_spins = rrddim_add(ptrs->st_pgc_waste, "evict spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1423 + ptrs->rd_pgc_waste_flush_spins = rrddim_add(ptrs->st_pgc_waste, "flush spins", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1424 +
1425 + buffer_free(id);
1426 + buffer_free(family);
1427 + buffer_free(title);
1428 + priority++;
1429 + }
1430 +
1431 + rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evictions_skipped, (collected_number)pgc_stats->evict_skipped);
1432 + rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_flushes_cancelled, (collected_number)pgc_stats->flushes_cancelled);
1433 + rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_acquire_spins, (collected_number)pgc_stats->acquire_spins);
1434 + rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_release_spins, (collected_number)pgc_stats->release_spins);
1435 + rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_insert_spins, (collected_number)pgc_stats->insert_spins);
1436 + rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_delete_spins, (collected_number)pgc_stats->delete_spins);
1437 + rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_evict_spins, (collected_number)pgc_stats->evict_spins);
1438 + rrddim_set_by_pointer(ptrs->st_pgc_waste, ptrs->rd_pgc_waste_flush_spins, (collected_number)pgc_stats->flush_spins);
1439 +
1440 + rrdset_done(ptrs->st_pgc_waste);
1441 + }
1442 +
1443 + {
1444 + if (unlikely(!ptrs->st_pgc_workers)) {
1445 + BUFFER *id = buffer_create(100);
1446 + buffer_sprintf(id, "dbengine_%s_cache_workers", name);
1447 +
1448 + BUFFER *family = buffer_create(100);
1449 + buffer_sprintf(family, "dbengine %s cache", name);
1450 +
1451 + BUFFER *title = buffer_create(100);
1452 + buffer_sprintf(title, "Netdata %s Cache Workers", name);
1453 +
1454 + ptrs->st_pgc_workers = rrdset_create_localhost(
1455 + "netdata",
1456 + buffer_tostring(id),
1457 + NULL,
1458 + buffer_tostring(family),
1459 + NULL,
1460 + buffer_tostring(title),
1461 + "workers",
1462 + "netdata",
1463 + "stats",
1464 + priority,
1465 + localhost->rrd_update_every,
1466 + RRDSET_TYPE_LINE);
1467 +
1468 + ptrs->rd_pgc_workers_searchers = rrddim_add(ptrs->st_pgc_workers, "searchers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1469 + ptrs->rd_pgc_workers_adders = rrddim_add(ptrs->st_pgc_workers, "adders", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1470 + ptrs->rd_pgc_workers_evictors = rrddim_add(ptrs->st_pgc_workers, "evictors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1471 + ptrs->rd_pgc_workers_flushers = rrddim_add(ptrs->st_pgc_workers, "flushers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1472 + ptrs->rd_pgc_workers_hot2dirty = rrddim_add(ptrs->st_pgc_workers, "hot2dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1473 + ptrs->rd_pgc_workers_jv2_flushers = rrddim_add(ptrs->st_pgc_workers, "jv2 flushers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1474 +
1475 + buffer_free(id);
1476 + buffer_free(family);
1477 + buffer_free(title);
1478 + priority++;
1479 + }
1480 +
1481 + rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_searchers, (collected_number)pgc_stats->workers_search);
1482 + rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_adders, (collected_number)pgc_stats->workers_add);
1483 + rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_evictors, (collected_number)pgc_stats->workers_evict);
1484 + rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_flushers, (collected_number)pgc_stats->workers_flush);
1485 + rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_hot2dirty, (collected_number)pgc_stats->workers_hot2dirty);
1486 + rrddim_set_by_pointer(ptrs->st_pgc_workers, ptrs->rd_pgc_workers_jv2_flushers, (collected_number)pgc_stats->workers_jv2_flush);
1487 +
1488 + rrdset_done(ptrs->st_pgc_workers);
1489 + }
1490 +}
1491 +
1492 +
1493 +static void dbengine2_statistics_charts(void) {
1494 + if(!main_cache || !main_mrg)
1495 + return;
1496 +
1497 + static struct dbengine2_cache_pointers main_cache_ptrs = {}, open_cache_ptrs = {}, extent_cache_ptrs = {};
1498 + static struct rrdeng_cache_efficiency_stats cache_efficiency_stats = {}, cache_efficiency_stats_old = {};
1499 + static struct pgc_statistics pgc_main_stats = {}, pgc_main_stats_old = {}; (void)pgc_main_stats_old;
1500 + static struct pgc_statistics pgc_open_stats = {}, pgc_open_stats_old = {}; (void)pgc_open_stats_old;
1501 + static struct pgc_statistics pgc_extent_stats = {}, pgc_extent_stats_old = {}; (void)pgc_extent_stats_old;
1502 + static struct mrg_statistics mrg_stats = {}, mrg_stats_old = {}; (void)mrg_stats_old;
1503 +
1504 + pgc_main_stats_old = pgc_main_stats;
1505 + pgc_main_stats = pgc_get_statistics(main_cache);
1506 + dbengine2_cache_statistics_charts(&main_cache_ptrs, &pgc_main_stats, &pgc_main_stats_old, "main", 135100);
1507 +
1508 + pgc_open_stats_old = pgc_open_stats;
1509 + pgc_open_stats = pgc_get_statistics(open_cache);
1510 + dbengine2_cache_statistics_charts(&open_cache_ptrs, &pgc_open_stats, &pgc_open_stats_old, "open", 135200);
1511 +
1512 + pgc_extent_stats_old = pgc_extent_stats;
1513 + pgc_extent_stats = pgc_get_statistics(extent_cache);
1514 + dbengine2_cache_statistics_charts(&extent_cache_ptrs, &pgc_extent_stats, &pgc_extent_stats_old, "extent", 135300);
1515 +
1516 + cache_efficiency_stats_old = cache_efficiency_stats;
1517 + cache_efficiency_stats = rrdeng_get_cache_efficiency_stats();
1518 +
1519 + mrg_stats_old = mrg_stats;
1520 + mrg_stats = mrg_get_statistics(main_mrg);
1521 +
1522 + struct rrdeng_buffer_sizes buffers = rrdeng_get_buffer_sizes();
1523 + size_t buffers_total_size = buffers.handles + buffers.xt_buf + buffers.xt_io + buffers.pdc + buffers.descriptors +
1524 + buffers.opcodes + buffers.wal + buffers.workers + buffers.epdl + buffers.deol + buffers.pd;
1525 +
1526 +#ifdef PDC_USE_JULYL
1527 + buffers_total_size += buffers.julyl;
1528 +#endif
1529 +
1530 + size_t priority = 135000;
1531 +
1532 + {
1533 + static RRDSET *st_pgc_memory = NULL;
1534 + static RRDDIM *rd_pgc_memory_main = NULL;
1535 + static RRDDIM *rd_pgc_memory_open = NULL; // open journal memory
1536 + static RRDDIM *rd_pgc_memory_extent = NULL; // extent compresses cache memory
1537 + static RRDDIM *rd_pgc_memory_metrics = NULL; // metric registry memory
1538 + static RRDDIM *rd_pgc_memory_buffers = NULL;
1539 +
1540 + if (unlikely(!st_pgc_memory)) {
1541 + st_pgc_memory = rrdset_create_localhost(
1542 + "netdata",
1543 + "dbengine_memory",
1544 + NULL,
1545 + "dbengine memory",
1546 + NULL,
1547 + "Netdata DB Memory",
1548 + "bytes",
1549 + "netdata",
1550 + "stats",
1551 + priority,
1552 + localhost->rrd_update_every,
1553 + RRDSET_TYPE_STACKED);
1554 +
1555 + rd_pgc_memory_main = rrddim_add(st_pgc_memory, "main cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1556 + rd_pgc_memory_open = rrddim_add(st_pgc_memory, "open cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1557 + rd_pgc_memory_extent = rrddim_add(st_pgc_memory, "extent cache", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1558 + rd_pgc_memory_metrics = rrddim_add(st_pgc_memory, "metrics registry", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1559 + rd_pgc_memory_buffers = rrddim_add(st_pgc_memory, "buffers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1560 + }
1561 + priority++;
1562 +
1563 + rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_main, (collected_number)pgc_main_stats.size);
1564 + rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_open, (collected_number)pgc_open_stats.size);
1565 + rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_extent, (collected_number)pgc_extent_stats.size);
1566 + rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_metrics, (collected_number)mrg_stats.size);
1567 + rrddim_set_by_pointer(st_pgc_memory, rd_pgc_memory_buffers, (collected_number)buffers_total_size);
1568 +
1569 + rrdset_done(st_pgc_memory);
1570 + }
1571 +
1572 + {
1573 + static RRDSET *st_pgc_buffers = NULL;
1574 + static RRDDIM *rd_pgc_buffers_opcodes = NULL;
1575 + static RRDDIM *rd_pgc_buffers_handles = NULL;
1576 + static RRDDIM *rd_pgc_buffers_descriptors = NULL;
1577 + static RRDDIM *rd_pgc_buffers_wal = NULL;
1578 + static RRDDIM *rd_pgc_buffers_workers = NULL;
1579 + static RRDDIM *rd_pgc_buffers_pdc = NULL;
1580 + static RRDDIM *rd_pgc_buffers_xt_io = NULL;
1581 + static RRDDIM *rd_pgc_buffers_xt_buf = NULL;
1582 + static RRDDIM *rd_pgc_buffers_epdl = NULL;
1583 + static RRDDIM *rd_pgc_buffers_deol = NULL;
1584 + static RRDDIM *rd_pgc_buffers_pd = NULL;
1585 +#ifdef PDC_USE_JULYL
1586 + static RRDDIM *rd_pgc_buffers_julyl = NULL;
1587 +#endif
1588 +
1589 + if (unlikely(!st_pgc_buffers)) {
1590 + st_pgc_buffers = rrdset_create_localhost(
1591 + "netdata",
1592 + "dbengine_buffers",
1593 + NULL,
1594 + "dbengine memory",
1595 + NULL,
1596 + "Netdata DB Buffers",
1597 + "bytes",
1598 + "netdata",
1599 + "stats",
1600 + priority,
1601 + localhost->rrd_update_every,
1602 + RRDSET_TYPE_STACKED);
1603 +
1604 + rd_pgc_buffers_opcodes = rrddim_add(st_pgc_buffers, "opcodes", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1605 + rd_pgc_buffers_handles = rrddim_add(st_pgc_buffers, "query handles", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1606 + rd_pgc_buffers_descriptors = rrddim_add(st_pgc_buffers, "descriptors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1607 + rd_pgc_buffers_wal = rrddim_add(st_pgc_buffers, "wal", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1608 + rd_pgc_buffers_workers = rrddim_add(st_pgc_buffers, "workers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1609 + rd_pgc_buffers_pdc = rrddim_add(st_pgc_buffers, "pdc", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1610 + rd_pgc_buffers_pd = rrddim_add(st_pgc_buffers, "pd", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1611 + rd_pgc_buffers_xt_io = rrddim_add(st_pgc_buffers, "extent io", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1612 + rd_pgc_buffers_xt_buf = rrddim_add(st_pgc_buffers, "extent buffers", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1613 + rd_pgc_buffers_epdl = rrddim_add(st_pgc_buffers, "epdl", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1614 + rd_pgc_buffers_deol = rrddim_add(st_pgc_buffers, "deol", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1615 +#ifdef PDC_USE_JULYL
1616 + rd_pgc_buffers_julyl = rrddim_add(st_pgc_buffers, "julyl", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1617 +#endif
1618 + }
1619 + priority++;
1620 +
1621 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_opcodes, (collected_number)buffers.opcodes);
1622 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_handles, (collected_number)buffers.handles);
1623 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_descriptors, (collected_number)buffers.descriptors);
1624 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_wal, (collected_number)buffers.wal);
1625 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_workers, (collected_number)buffers.workers);
1626 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pdc, (collected_number)buffers.pdc);
1627 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_pd, (collected_number)buffers.pd);
1628 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_xt_io, (collected_number)buffers.xt_io);
1629 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_xt_buf, (collected_number)buffers.xt_buf);
1630 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_epdl, (collected_number)buffers.epdl);
1631 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_deol, (collected_number)buffers.deol);
1632 +#ifdef PDC_USE_JULYL
1633 + rrddim_set_by_pointer(st_pgc_buffers, rd_pgc_buffers_julyl, (collected_number)buffers.julyl);
1634 +#endif
1635 +
1636 + rrdset_done(st_pgc_buffers);
1637 + }
1638 +
1639 +#ifdef PDC_USE_JULYL
1640 + {
1641 + static RRDSET *st_julyl_moved = NULL;
1642 + static RRDDIM *rd_julyl_moved = NULL;
1643 +
1644 + if (unlikely(!st_julyl_moved)) {
1645 + st_julyl_moved = rrdset_create_localhost(
1646 + "netdata",
1647 + "dbengine_julyl_moved",
1648 + NULL,
1649 + "dbengine memory",
1650 + NULL,
1651 + "Netdata JulyL Memory Moved",
1652 + "bytes/s",
1653 + "netdata",
1654 + "stats",
1655 + priority,
1656 + localhost->rrd_update_every,
1657 + RRDSET_TYPE_AREA);
1658 +
1659 + rd_julyl_moved = rrddim_add(st_julyl_moved, "moved", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1660 + }
1661 + priority++;
1662 +
1663 + rrddim_set_by_pointer(st_julyl_moved, rd_julyl_moved, (collected_number)julyl_bytes_moved());
1664 +
1665 + rrdset_done(st_julyl_moved);
1666 + }
1667 +#endif
1668 +
1669 + {
1670 + static RRDSET *st_cache_hit_ratio = NULL;
1671 + static RRDDIM *rd_hit_ratio = NULL;
1672 + static RRDDIM *rd_preloaded_ratio = NULL;
1673 + static RRDDIM *rd_extent_ratio = NULL;
1674 + static RRDDIM *rd_parallel_load_ratio = NULL;
1675 + static RRDDIM *rd_before_allocation_ratio = NULL;
1676 + static RRDDIM *rd_insert_conflict_ratio = NULL;
1677 +
1678 + if (unlikely(!st_cache_hit_ratio)) {
1679 + st_cache_hit_ratio = rrdset_create_localhost(
1680 + "netdata",
1681 + "dbengine_cache_hit_ratio",
1682 + NULL,
1683 + "dbengine query router",
1684 + NULL,
1685 + "Netdata Queries Cache Hit Ratio",
1686 + "%",
1687 + "netdata",
1688 + "stats",
1689 + priority,
1690 + localhost->rrd_update_every,
1691 + RRDSET_TYPE_LINE);
1692 +
1693 + rd_hit_ratio = rrddim_add(st_cache_hit_ratio, "overall", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1694 + rd_preloaded_ratio = rrddim_add(st_cache_hit_ratio, "main cache", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1695 + rd_extent_ratio = rrddim_add(st_cache_hit_ratio, "extent cache", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1696 + rd_parallel_load_ratio = rrddim_add(st_cache_hit_ratio, "parallel load", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1697 + rd_before_allocation_ratio = rrddim_add(st_cache_hit_ratio, "before allocation", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1698 + rd_insert_conflict_ratio = rrddim_add(st_cache_hit_ratio, "insert conflict", NULL, 1, 10000, RRD_ALGORITHM_ABSOLUTE);
1699 + }
1700 + priority++;
1701 +
1702 + size_t pages_to_load = cache_efficiency_stats.pages_to_load_from_disk - cache_efficiency_stats_old.pages_to_load_from_disk;
1703 + size_t pages_hit_parallel_load = cache_efficiency_stats.pages_load_ok_preloaded - cache_efficiency_stats_old.pages_load_ok_preloaded;
1704 + size_t pages_hit_before_allocation = cache_efficiency_stats.pages_load_ok_loaded_but_cache_hit_before_allocation - cache_efficiency_stats_old.pages_load_ok_loaded_but_cache_hit_before_allocation;
1705 + size_t pages_hit_insert_conflict = cache_efficiency_stats.pages_load_ok_loaded_but_cache_hit_while_inserting - cache_efficiency_stats_old.pages_load_ok_loaded_but_cache_hit_while_inserting;
1706 + size_t pages_hit_cached_extent = cache_efficiency_stats.pages_data_source_extent_cache - cache_efficiency_stats_old.pages_data_source_extent_cache;
1707 + size_t pages_hit_at_query_plan = cache_efficiency_stats.pages_meta_source_main_cache - cache_efficiency_stats_old.pages_meta_source_main_cache;
1708 +
1709 + size_t pages_total_hit = pages_hit_at_query_plan + pages_hit_before_allocation + pages_hit_insert_conflict + pages_hit_cached_extent;
1710 + size_t pages_total = cache_efficiency_stats.pages_total - cache_efficiency_stats_old.pages_total;
1711 +
1712 + static size_t overall_hit_ratio = 100;
1713 + size_t preloaded_hit_ratio = 0, extent_hit_ratio = 0;
1714 + size_t before_allocation_hit_ratio = 0, insert_conflict_hit_ratio = 0, parallel_load_hit_ratio = 0;
1715 + if(pages_total) {
1716 + if(pages_total_hit > pages_total)
1717 + pages_total_hit = pages_total;
1718 +
1719 + overall_hit_ratio = pages_total_hit * 100 * 10000 / pages_total;
1720 +
1721 + preloaded_hit_ratio = pages_hit_at_query_plan * 100 * 10000 / pages_total;
1722 + }
1723 +
1724 + if(pages_to_load) {
1725 + extent_hit_ratio = pages_hit_cached_extent * 100 * 10000 / pages_to_load;
1726 + if(extent_hit_ratio > 100 * 10000)
1727 + extent_hit_ratio = 100 * 10000;
1728 +
1729 + parallel_load_hit_ratio = pages_hit_parallel_load * 100 * 10000 / pages_to_load;
1730 + if(parallel_load_hit_ratio > 100 * 10000)
1731 + parallel_load_hit_ratio = 100 * 10000;
1732 +
1733 + before_allocation_hit_ratio = pages_hit_before_allocation * 100 * 10000 / pages_to_load;
1734 + if(before_allocation_hit_ratio > 100 * 10000)
1735 + before_allocation_hit_ratio = 100 * 10000;
1736 +
1737 + insert_conflict_hit_ratio = pages_hit_insert_conflict * 100 * 10000 / pages_to_load;
1738 + if(insert_conflict_hit_ratio > 100 * 10000)
1739 + insert_conflict_hit_ratio = 100 * 10000;
1740 + }
1741 +
1742 + rrddim_set_by_pointer(st_cache_hit_ratio, rd_hit_ratio, (collected_number)overall_hit_ratio);
1743 + rrddim_set_by_pointer(st_cache_hit_ratio, rd_preloaded_ratio, (collected_number)preloaded_hit_ratio);
1744 + rrddim_set_by_pointer(st_cache_hit_ratio, rd_extent_ratio, (collected_number)extent_hit_ratio);
1745 + rrddim_set_by_pointer(st_cache_hit_ratio, rd_parallel_load_ratio, (collected_number)parallel_load_hit_ratio);
1746 + rrddim_set_by_pointer(st_cache_hit_ratio, rd_before_allocation_ratio, (collected_number)before_allocation_hit_ratio);
1747 + rrddim_set_by_pointer(st_cache_hit_ratio, rd_insert_conflict_ratio, (collected_number)insert_conflict_hit_ratio);
1748 +
1749 + rrdset_done(st_cache_hit_ratio);
1750 + }
1751 +
1752 + {
1753 + static RRDSET *st_queries = NULL;
1754 + static RRDDIM *rd_total = NULL;
1755 + static RRDDIM *rd_open = NULL;
1756 + static RRDDIM *rd_jv2 = NULL;
1757 + static RRDDIM *rd_planned_with_gaps = NULL;
1758 + static RRDDIM *rd_executed_with_gaps = NULL;
1759 +
1760 + if (unlikely(!st_queries)) {
1761 + st_queries = rrdset_create_localhost(
1762 + "netdata",
1763 + "dbengine_queries",
1764 + NULL,
1765 + "dbengine query router",
1766 + NULL,
1767 + "Netdata Queries",
1768 + "queries/s",
1769 + "netdata",
1770 + "stats",
1771 + priority,
1772 + localhost->rrd_update_every,
1773 + RRDSET_TYPE_LINE);
1774 +
1775 + rd_total = rrddim_add(st_queries, "total", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1776 + rd_open = rrddim_add(st_queries, "open cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1777 + rd_jv2 = rrddim_add(st_queries, "journal v2", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1778 + rd_planned_with_gaps = rrddim_add(st_queries, "planned with gaps", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1779 + rd_executed_with_gaps = rrddim_add(st_queries, "executed with gaps", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1780 + }
1781 + priority++;
1782 +
1783 + rrddim_set_by_pointer(st_queries, rd_total, (collected_number)cache_efficiency_stats.queries);
1784 + rrddim_set_by_pointer(st_queries, rd_open, (collected_number)cache_efficiency_stats.queries_open);
1785 + rrddim_set_by_pointer(st_queries, rd_jv2, (collected_number)cache_efficiency_stats.queries_journal_v2);
1786 + rrddim_set_by_pointer(st_queries, rd_planned_with_gaps, (collected_number)cache_efficiency_stats.queries_planned_with_gaps);
1787 + rrddim_set_by_pointer(st_queries, rd_executed_with_gaps, (collected_number)cache_efficiency_stats.queries_executed_with_gaps);
1788 +
1789 + rrdset_done(st_queries);
1790 + }
1791 +
1792 + {
1793 + static RRDSET *st_queries_running = NULL;
1794 + static RRDDIM *rd_queries = NULL;
1795 +
1796 + if (unlikely(!st_queries_running)) {
1797 + st_queries_running = rrdset_create_localhost(
1798 + "netdata",
1799 + "dbengine_queries_running",
1800 + NULL,
1801 + "dbengine query router",
1802 + NULL,
1803 + "Netdata Queries Running",
1804 + "queries",
1805 + "netdata",
1806 + "stats",
1807 + priority,
1808 + localhost->rrd_update_every,
1809 + RRDSET_TYPE_LINE);
1810 +
1811 + rd_queries = rrddim_add(st_queries_running, "queries", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1812 + }
1813 + priority++;
1814 +
1815 + rrddim_set_by_pointer(st_queries_running, rd_queries, (collected_number)cache_efficiency_stats.currently_running_queries);
1816 +
1817 + rrdset_done(st_queries_running);
1818 + }
1819 +
1820 + {
1821 + static RRDSET *st_query_pages_metadata_source = NULL;
1822 + static RRDDIM *rd_cache = NULL;
1823 + static RRDDIM *rd_open = NULL;
1824 + static RRDDIM *rd_jv2 = NULL;
1825 +
1826 + if (unlikely(!st_query_pages_metadata_source)) {
1827 + st_query_pages_metadata_source = rrdset_create_localhost(
1828 + "netdata",
1829 + "dbengine_query_pages_metadata_source",
1830 + NULL,
1831 + "dbengine query router",
1832 + NULL,
1833 + "Netdata Query Pages Metadata Source",
1834 + "pages/s",
1835 + "netdata",
1836 + "stats",
1837 + priority,
1838 + localhost->rrd_update_every,
1839 + RRDSET_TYPE_STACKED);
1840 +
1841 + rd_cache = rrddim_add(st_query_pages_metadata_source, "cache hit", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1842 + rd_jv2 = rrddim_add(st_query_pages_metadata_source, "journal v2 scan", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1843 + rd_open = rrddim_add(st_query_pages_metadata_source, "open journal", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1844 + }
1845 + priority++;
1846 +
1847 + rrddim_set_by_pointer(st_query_pages_metadata_source, rd_cache, (collected_number)cache_efficiency_stats.pages_meta_source_main_cache);
1848 + rrddim_set_by_pointer(st_query_pages_metadata_source, rd_jv2, (collected_number)cache_efficiency_stats.pages_meta_source_journal_v2);
1849 + rrddim_set_by_pointer(st_query_pages_metadata_source, rd_open, (collected_number)cache_efficiency_stats.pages_meta_source_open_cache);
1850 +
1851 + rrdset_done(st_query_pages_metadata_source);
1852 + }
1853 +
1854 + {
1855 + static RRDSET *st_query_pages_data_source = NULL;
1856 + static RRDDIM *rd_pages_main_cache = NULL;
1857 + static RRDDIM *rd_pages_disk = NULL;
1858 + static RRDDIM *rd_pages_extent_cache = NULL;
1859 +
1860 + if (unlikely(!st_query_pages_data_source)) {
1861 + st_query_pages_data_source = rrdset_create_localhost(
1862 + "netdata",
1863 + "dbengine_query_pages_data_source",
1864 + NULL,
1865 + "dbengine query router",
1866 + NULL,
1867 + "Netdata Query Pages to Data Source",
1868 + "pages/s",
1869 + "netdata",
1870 + "stats",
1871 + priority,
1872 + localhost->rrd_update_every,
1873 + RRDSET_TYPE_STACKED);
1874 +
1875 + rd_pages_main_cache = rrddim_add(st_query_pages_data_source, "main cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1876 + rd_pages_disk = rrddim_add(st_query_pages_data_source, "disk", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1877 + rd_pages_extent_cache = rrddim_add(st_query_pages_data_source, "extent cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1878 + }
1879 + priority++;
1880 +
1881 + rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_main_cache, (collected_number)cache_efficiency_stats.pages_data_source_main_cache);
1882 + rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_disk, (collected_number)cache_efficiency_stats.pages_to_load_from_disk);
1883 + rrddim_set_by_pointer(st_query_pages_data_source, rd_pages_extent_cache, (collected_number)cache_efficiency_stats.pages_data_source_extent_cache);
1884 +
1885 + rrdset_done(st_query_pages_data_source);
1886 + }
1887 +
1888 + {
1889 + static RRDSET *st_query_next_page = NULL;
1890 + static RRDDIM *rd_pass4 = NULL;
1891 + static RRDDIM *rd_nowait_failed = NULL;
1892 + static RRDDIM *rd_wait_failed = NULL;
1893 + static RRDDIM *rd_wait_loaded = NULL;
1894 + static RRDDIM *rd_nowait_loaded = NULL;
1895 +
1896 + if (unlikely(!st_query_next_page)) {
1897 + st_query_next_page = rrdset_create_localhost(
1898 + "netdata",
1899 + "dbengine_query_next_page",
1900 + NULL,
1901 + "dbengine query router",
1902 + NULL,
1903 + "Netdata Query Next Page",
1904 + "pages/s",
1905 + "netdata",
1906 + "stats",
1907 + priority,
1908 + localhost->rrd_update_every,
1909 + RRDSET_TYPE_STACKED);
1910 +
1911 + rd_pass4 = rrddim_add(st_query_next_page, "pass4", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1912 + rd_wait_failed = rrddim_add(st_query_next_page, "failed slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1913 + rd_nowait_failed = rrddim_add(st_query_next_page, "failed fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1914 + rd_wait_loaded = rrddim_add(st_query_next_page, "loaded slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1915 + rd_nowait_loaded = rrddim_add(st_query_next_page, "loaded fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1916 + }
1917 + priority++;
1918 +
1919 + rrddim_set_by_pointer(st_query_next_page, rd_pass4, (collected_number)cache_efficiency_stats.pages_pending_found_in_cache_at_pass4);
1920 + rrddim_set_by_pointer(st_query_next_page, rd_wait_failed, (collected_number)cache_efficiency_stats.page_next_wait_failed);
1921 + rrddim_set_by_pointer(st_query_next_page, rd_nowait_failed, (collected_number)cache_efficiency_stats.page_next_nowait_failed);
1922 + rrddim_set_by_pointer(st_query_next_page, rd_wait_loaded, (collected_number)cache_efficiency_stats.page_next_wait_loaded);
1923 + rrddim_set_by_pointer(st_query_next_page, rd_nowait_loaded, (collected_number)cache_efficiency_stats.page_next_nowait_loaded);
1924 +
1925 + rrdset_done(st_query_next_page);
1926 + }
1927 +
1928 + {
1929 + static RRDSET *st_query_page_issues = NULL;
1930 + static RRDDIM *rd_pages_zero_time = NULL;
1931 + static RRDDIM *rd_pages_past_time = NULL;
1932 + static RRDDIM *rd_pages_invalid_size = NULL;
1933 + static RRDDIM *rd_pages_fixed_update_every = NULL;
1934 + static RRDDIM *rd_pages_fixed_entries = NULL;
1935 + static RRDDIM *rd_pages_overlapping = NULL;
1936 +
1937 + if (unlikely(!st_query_page_issues)) {
1938 + st_query_page_issues = rrdset_create_localhost(
1939 + "netdata",
1940 + "dbengine_query_next_page_issues",
1941 + NULL,
1942 + "dbengine query router",
1943 + NULL,
1944 + "Netdata Query Next Page Issues",
1945 + "pages/s",
1946 + "netdata",
1947 + "stats",
1948 + priority,
1949 + localhost->rrd_update_every,
1950 + RRDSET_TYPE_STACKED);
1951 +
1952 + rd_pages_zero_time = rrddim_add(st_query_page_issues, "zero timestamp", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1953 + rd_pages_invalid_size = rrddim_add(st_query_page_issues, "invalid size", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1954 + rd_pages_past_time = rrddim_add(st_query_page_issues, "past time", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1955 + rd_pages_overlapping = rrddim_add(st_query_page_issues, "overlapping", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1956 + rd_pages_fixed_update_every = rrddim_add(st_query_page_issues, "update every fixed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1957 + rd_pages_fixed_entries = rrddim_add(st_query_page_issues, "entries fixed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1958 + }
1959 + priority++;
1960 +
1961 + rrddim_set_by_pointer(st_query_page_issues, rd_pages_zero_time, (collected_number)cache_efficiency_stats.pages_zero_time_skipped);
1962 + rrddim_set_by_pointer(st_query_page_issues, rd_pages_invalid_size, (collected_number)cache_efficiency_stats.pages_invalid_size_skipped);
1963 + rrddim_set_by_pointer(st_query_page_issues, rd_pages_past_time, (collected_number)cache_efficiency_stats.pages_past_time_skipped);
1964 + rrddim_set_by_pointer(st_query_page_issues, rd_pages_overlapping, (collected_number)cache_efficiency_stats.pages_overlapping_skipped);
1965 + rrddim_set_by_pointer(st_query_page_issues, rd_pages_fixed_update_every, (collected_number)cache_efficiency_stats.pages_invalid_update_every_fixed);
1966 + rrddim_set_by_pointer(st_query_page_issues, rd_pages_fixed_entries, (collected_number)cache_efficiency_stats.pages_invalid_entries_fixed);
1967 +
1968 + rrdset_done(st_query_page_issues);
1969 + }
1970 +
1971 + {
1972 + static RRDSET *st_query_pages_from_disk = NULL;
1973 + static RRDDIM *rd_compressed = NULL;
1974 + static RRDDIM *rd_invalid = NULL;
1975 + static RRDDIM *rd_uncompressed = NULL;
1976 + static RRDDIM *rd_mmap_failed = NULL;
1977 + static RRDDIM *rd_unavailable = NULL;
1978 + static RRDDIM *rd_already_loaded = NULL;
1979 + static RRDDIM *rd_preloaded = NULL;
1980 + static RRDDIM *rd_unroutable = NULL;
1981 + static RRDDIM *rd_not_found = NULL;
1982 + static RRDDIM *rd_invalid_extent = NULL;
1983 +
1984 + if (unlikely(!st_query_pages_from_disk)) {
1985 + st_query_pages_from_disk = rrdset_create_localhost(
1986 + "netdata",
1987 + "dbengine_query_pages_disk_load",
1988 + NULL,
1989 + "dbengine query router",
1990 + NULL,
1991 + "Netdata Query Pages Loaded from Disk",
1992 + "pages/s",
1993 + "netdata",
1994 + "stats",
1995 + priority,
1996 + localhost->rrd_update_every,
1997 + RRDSET_TYPE_LINE);
1998 +
1999 + rd_compressed = rrddim_add(st_query_pages_from_disk, "ok compressed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2000 + rd_invalid = rrddim_add(st_query_pages_from_disk, "fail invalid page", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2001 + rd_uncompressed = rrddim_add(st_query_pages_from_disk, "ok uncompressed", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2002 + rd_mmap_failed = rrddim_add(st_query_pages_from_disk, "fail cant mmap", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2003 + rd_already_loaded = rrddim_add(st_query_pages_from_disk, "ok but preloaded", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2004 + rd_unavailable = rrddim_add(st_query_pages_from_disk, "fail unavailable", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2005 + rd_preloaded = rrddim_add(st_query_pages_from_disk, "ok preloaded", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2006 + rd_unroutable = rrddim_add(st_query_pages_from_disk, "fail unroutable", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2007 + rd_not_found = rrddim_add(st_query_pages_from_disk, "fail uuid not found", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2008 + rd_invalid_extent = rrddim_add(st_query_pages_from_disk, "fail invalid extent", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2009 + }
2010 + priority++;
2011 +
2012 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_compressed, (collected_number)cache_efficiency_stats.pages_load_ok_compressed);
2013 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_invalid, (collected_number)cache_efficiency_stats.pages_load_fail_invalid_page_in_extent);
2014 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_uncompressed, (collected_number)cache_efficiency_stats.pages_load_ok_uncompressed);
2015 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_mmap_failed, (collected_number)cache_efficiency_stats.pages_load_fail_cant_mmap_extent);
2016 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_already_loaded, (collected_number)cache_efficiency_stats.pages_load_ok_loaded_but_cache_hit_before_allocation);
2017 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_unavailable, (collected_number)cache_efficiency_stats.pages_load_fail_datafile_not_available);
2018 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_preloaded, (collected_number)cache_efficiency_stats.pages_load_ok_preloaded);
2019 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_unroutable, (collected_number)cache_efficiency_stats.pages_load_fail_unroutable);
2020 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_not_found, (collected_number)cache_efficiency_stats.pages_load_fail_uuid_not_found);
2021 + rrddim_set_by_pointer(st_query_pages_from_disk, rd_invalid_extent, (collected_number)cache_efficiency_stats.pages_load_fail_invalid_extent);
2022 +
2023 + rrdset_done(st_query_pages_from_disk);
2024 + }
2025 +
2026 + {
2027 + static RRDSET *st_prep_timings = NULL;
2028 + static RRDDIM *rd_routing = NULL;
2029 + static RRDDIM *rd_main_cache = NULL;
2030 + static RRDDIM *rd_open_cache = NULL;
2031 + static RRDDIM *rd_journal_v2 = NULL;
2032 + static RRDDIM *rd_pass4 = NULL;
2033 +
2034 + if (unlikely(!st_prep_timings)) {
2035 + st_prep_timings = rrdset_create_localhost(
2036 + "netdata",
2037 + "dbengine_prep_timings",
2038 + NULL,
2039 + "dbengine query router",
2040 + NULL,
2041 + "Netdata Query Preparation Timings",
2042 + "usec/s",
2043 + "netdata",
2044 + "stats",
2045 + priority,
2046 + localhost->rrd_update_every,
2047 + RRDSET_TYPE_STACKED);
2048 +
2049 + rd_routing = rrddim_add(st_prep_timings, "routing", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2050 + rd_main_cache = rrddim_add(st_prep_timings, "main cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2051 + rd_open_cache = rrddim_add(st_prep_timings, "open cache", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2052 + rd_journal_v2 = rrddim_add(st_prep_timings, "journal v2", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2053 + rd_pass4 = rrddim_add(st_prep_timings, "pass4", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2054 + }
2055 + priority++;
2056 +
2057 + rrddim_set_by_pointer(st_prep_timings, rd_routing, (collected_number)cache_efficiency_stats.prep_time_to_route);
2058 + rrddim_set_by_pointer(st_prep_timings, rd_main_cache, (collected_number)cache_efficiency_stats.prep_time_in_main_cache_lookup);
2059 + rrddim_set_by_pointer(st_prep_timings, rd_open_cache, (collected_number)cache_efficiency_stats.prep_time_in_open_cache_lookup);
2060 + rrddim_set_by_pointer(st_prep_timings, rd_journal_v2, (collected_number)cache_efficiency_stats.prep_time_in_journal_v2_lookup);
2061 + rrddim_set_by_pointer(st_prep_timings, rd_pass4, (collected_number)cache_efficiency_stats.prep_time_in_pass4_lookup);
2062 +
2063 + rrdset_done(st_prep_timings);
2064 + }
2065 +
2066 + {
2067 + static RRDSET *st_query_timings = NULL;
2068 + static RRDDIM *rd_init = NULL;
2069 + static RRDDIM *rd_prep_wait = NULL;
2070 + static RRDDIM *rd_next_page_disk_fast = NULL;
2071 + static RRDDIM *rd_next_page_disk_slow = NULL;
2072 + static RRDDIM *rd_next_page_preload_fast = NULL;
2073 + static RRDDIM *rd_next_page_preload_slow = NULL;
2074 +
2075 + if (unlikely(!st_query_timings)) {
2076 + st_query_timings = rrdset_create_localhost(
2077 + "netdata",
2078 + "dbengine_query_timings",
2079 + NULL,
2080 + "dbengine query router",
2081 + NULL,
2082 + "Netdata Query Timings",
2083 + "usec/s",
2084 + "netdata",
2085 + "stats",
2086 + priority,
2087 + localhost->rrd_update_every,
2088 + RRDSET_TYPE_STACKED);
2089 +
2090 + rd_init = rrddim_add(st_query_timings, "init", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2091 + rd_prep_wait = rrddim_add(st_query_timings, "prep wait", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2092 + rd_next_page_disk_fast = rrddim_add(st_query_timings, "next page disk fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2093 + rd_next_page_disk_slow = rrddim_add(st_query_timings, "next page disk slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2094 + rd_next_page_preload_fast = rrddim_add(st_query_timings, "next page preload fast", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2095 + rd_next_page_preload_slow = rrddim_add(st_query_timings, "next page preload slow", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2096 + }
2097 + priority++;
2098 +
2099 + rrddim_set_by_pointer(st_query_timings, rd_init, (collected_number)cache_efficiency_stats.query_time_init);
2100 + rrddim_set_by_pointer(st_query_timings, rd_prep_wait, (collected_number)cache_efficiency_stats.query_time_wait_for_prep);
2101 + rrddim_set_by_pointer(st_query_timings, rd_next_page_disk_fast, (collected_number)cache_efficiency_stats.query_time_to_fast_disk_next_page);
2102 + rrddim_set_by_pointer(st_query_timings, rd_next_page_disk_slow, (collected_number)cache_efficiency_stats.query_time_to_slow_disk_next_page);
2103 + rrddim_set_by_pointer(st_query_timings, rd_next_page_preload_fast, (collected_number)cache_efficiency_stats.query_time_to_fast_preload_next_page);
2104 + rrddim_set_by_pointer(st_query_timings, rd_next_page_preload_slow, (collected_number)cache_efficiency_stats.query_time_to_slow_preload_next_page);
2105 +
2106 + rrdset_done(st_query_timings);
2107 + }
2108 +
2109 if(netdata_rwlock_tryrdlock(&rrd_rwlock) == 0) {
2110 + priority = 135400;
2111 +
2112 RRDHOST *host;
2113 unsigned long long stats_array[RRDENG_NR_STATS] = {0};
2114 unsigned long long local_stats_array[RRDENG_NR_STATS];
@@ -1046,21 +2156,22 @@ static void dbengine_statistics_charts(void) {
2156
2157 if (unlikely(!st_compression)) {
2158 st_compression = rrdset_create_localhost(
1049 - "netdata",
1050 - "dbengine_compression_ratio",
1051 - NULL,
1052 - "dbengine",
1053 - NULL,
1054 - "Netdata DB engine data extents' compression savings ratio",
1055 - "percentage",
1056 - "netdata",
1057 - "stats",
1058 - 132000,
1059 - localhost->rrd_update_every,
1060 - RRDSET_TYPE_LINE);
2159 + "netdata",
2160 + "dbengine_compression_ratio",
2161 + NULL,
2162 + "dbengine io",
2163 + NULL,
2164 + "Netdata DB engine data extents' compression savings ratio",
2165 + "percentage",
2166 + "netdata",
2167 + "stats",
2168 + priority,
2169 + localhost->rrd_update_every,
2170 + RRDSET_TYPE_LINE);
2171
2172 rd_savings = rrddim_add(st_compression, "savings", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
2173 }
2174 + priority++;
2175
2176 unsigned long long ratio;
2177 unsigned long long compressed_content_size = stats_array[12];
@@ -1079,136 +2190,6 @@ static void dbengine_statistics_charts(void) {
2190
2191 // ----------------------------------------------------------------
2192
1082 - {
1083 - static RRDSET *st_pg_cache_hit_ratio = NULL;
1084 - static RRDDIM *rd_hit_ratio = NULL;
1085 -
1086 - if (unlikely(!st_pg_cache_hit_ratio)) {
1087 - st_pg_cache_hit_ratio = rrdset_create_localhost(
1088 - "netdata",
1089 - "page_cache_hit_ratio",
1090 - NULL,
1091 - "dbengine",
1092 - NULL,
1093 - "Netdata DB engine page cache hit ratio",
1094 - "percentage",
1095 - "netdata",
1096 - "stats",
1097 - 132003,
1098 - localhost->rrd_update_every,
1099 - RRDSET_TYPE_LINE);
1100 -
1101 - rd_hit_ratio = rrddim_add(st_pg_cache_hit_ratio, "ratio", NULL, 1, 1000, RRD_ALGORITHM_ABSOLUTE);
1102 - }
1103 -
1104 - static unsigned long long old_hits = 0;
1105 - static unsigned long long old_misses = 0;
1106 - unsigned long long hits = stats_array[7];
1107 - unsigned long long misses = stats_array[8];
1108 - unsigned long long hits_delta;
1109 - unsigned long long misses_delta;
1110 - unsigned long long ratio;
1111 -
1112 - hits_delta = hits - old_hits;
1113 - misses_delta = misses - old_misses;
1114 - old_hits = hits;
1115 - old_misses = misses;
1116 -
1117 - if (hits_delta + misses_delta) {
1118 - ratio = (hits_delta * 100 * 1000) / (hits_delta + misses_delta);
1119 - } else {
1120 - ratio = 0;
1121 - }
1122 - rrddim_set_by_pointer(st_pg_cache_hit_ratio, rd_hit_ratio, ratio);
1123 -
1124 - rrdset_done(st_pg_cache_hit_ratio);
1125 - }
1126 -
1127 - // ----------------------------------------------------------------
1128 -
1129 - {
1130 - static RRDSET *st_pg_cache_pages = NULL;
1131 - static RRDDIM *rd_descriptors = NULL;
1132 - static RRDDIM *rd_populated = NULL;
1133 - static RRDDIM *rd_dirty = NULL;
1134 - static RRDDIM *rd_backfills = NULL;
1135 - static RRDDIM *rd_evictions = NULL;
1136 - static RRDDIM *rd_used_by_collectors = NULL;
1137 -
1138 - if (unlikely(!st_pg_cache_pages)) {
1139 - st_pg_cache_pages = rrdset_create_localhost(
1140 - "netdata",
1141 - "page_cache_stats",
1142 - NULL,
1143 - "dbengine",
1144 - NULL,
1145 - "Netdata dbengine page cache statistics",
1146 - "pages",
1147 - "netdata",
1148 - "stats",
1149 - 132004,
1150 - localhost->rrd_update_every,
1151 - RRDSET_TYPE_LINE);
1152 -
1153 - rd_descriptors = rrddim_add(st_pg_cache_pages, "descriptors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1154 - rd_populated = rrddim_add(st_pg_cache_pages, "populated", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1155 - rd_dirty = rrddim_add(st_pg_cache_pages, "dirty", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1156 - rd_backfills = rrddim_add(st_pg_cache_pages, "backfills", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1157 - rd_evictions = rrddim_add(st_pg_cache_pages, "evictions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1158 - rd_used_by_collectors =
1159 - rrddim_add(st_pg_cache_pages, "used_by_collectors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1160 - }
1161 -
1162 - rrddim_set_by_pointer(st_pg_cache_pages, rd_descriptors, (collected_number)stats_array[27]);
1163 - rrddim_set_by_pointer(st_pg_cache_pages, rd_populated, (collected_number)stats_array[3]);
1164 - rrddim_set_by_pointer(st_pg_cache_pages, rd_dirty, (collected_number)stats_array[0] + stats_array[4]);
1165 - rrddim_set_by_pointer(st_pg_cache_pages, rd_backfills, (collected_number)stats_array[9]);
1166 - rrddim_set_by_pointer(st_pg_cache_pages, rd_evictions, (collected_number)stats_array[10]);
1167 - rrddim_set_by_pointer(st_pg_cache_pages, rd_used_by_collectors, (collected_number)stats_array[0]);
1168 - rrdset_done(st_pg_cache_pages);
1169 - }
1170 -
1171 - // ----------------------------------------------------------------
1172 -
1173 - {
1174 - static RRDSET *st_long_term_pages = NULL;
1175 - static RRDDIM *rd_total = NULL;
1176 - static RRDDIM *rd_insertions = NULL;
1177 - static RRDDIM *rd_deletions = NULL;
1178 - static RRDDIM *rd_flushing_pressure_deletions = NULL;
1179 -
1180 - if (unlikely(!st_long_term_pages)) {
1181 - st_long_term_pages = rrdset_create_localhost(
1182 - "netdata",
1183 - "dbengine_long_term_page_stats",
1184 - NULL,
1185 - "dbengine",
1186 - NULL,
1187 - "Netdata dbengine long-term page statistics",
1188 - "pages",
1189 - "netdata",
1190 - "stats",
1191 - 132005,
1192 - localhost->rrd_update_every,
1193 - RRDSET_TYPE_LINE);
1194 -
1195 - rd_total = rrddim_add(st_long_term_pages, "total", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
1196 - rd_insertions = rrddim_add(st_long_term_pages, "insertions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
1197 - rd_deletions = rrddim_add(st_long_term_pages, "deletions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1198 - rd_flushing_pressure_deletions = rrddim_add(
1199 - st_long_term_pages, "flushing_pressure_deletions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
1200 - }
1201 -
1202 - rrddim_set_by_pointer(st_long_term_pages, rd_total, (collected_number)stats_array[2]);
1203 - rrddim_set_by_pointer(st_long_term_pages, rd_insertions, (collected_number)stats_array[5]);
1204 - rrddim_set_by_pointer(st_long_term_pages, rd_deletions, (collected_number)stats_array[6]);
1205 - rrddim_set_by_pointer(
1206 - st_long_term_pages, rd_flushing_pressure_deletions, (collected_number)stats_array[36]);
1207 - rrdset_done(st_long_term_pages);
1208 - }
1209 -
1210 - // ----------------------------------------------------------------
1211 -
2193 {
2194 static RRDSET *st_io_stats = NULL;
2195 static RRDDIM *rd_reads = NULL;
@@ -1216,22 +2197,23 @@ static void dbengine_statistics_charts(void) {
2197
2198 if (unlikely(!st_io_stats)) {
2199 st_io_stats = rrdset_create_localhost(
1219 - "netdata",
1220 - "dbengine_io_throughput",
1221 - NULL,
1222 - "dbengine",
1223 - NULL,
1224 - "Netdata DB engine I/O throughput",
1225 - "MiB/s",
1226 - "netdata",
1227 - "stats",
1228 - 132006,
1229 - localhost->rrd_update_every,
1230 - RRDSET_TYPE_LINE);
2200 + "netdata",
2201 + "dbengine_io_throughput",
2202 + NULL,
2203 + "dbengine io",
2204 + NULL,
2205 + "Netdata DB engine I/O throughput",
2206 + "MiB/s",
2207 + "netdata",
2208 + "stats",
2209 + priority,
2210 + localhost->rrd_update_every,
2211 + RRDSET_TYPE_LINE);
2212
2213 rd_reads = rrddim_add(st_io_stats, "reads", NULL, 1, 1024 * 1024, RRD_ALGORITHM_INCREMENTAL);
2214 rd_writes = rrddim_add(st_io_stats, "writes", NULL, -1, 1024 * 1024, RRD_ALGORITHM_INCREMENTAL);
2215 }
2216 + priority++;
2217
2218 rrddim_set_by_pointer(st_io_stats, rd_reads, (collected_number)stats_array[17]);
2219 rrddim_set_by_pointer(st_io_stats, rd_writes, (collected_number)stats_array[15]);
@@ -1247,22 +2229,23 @@ static void dbengine_statistics_charts(void) {
2229
2230 if (unlikely(!st_io_stats)) {
2231 st_io_stats = rrdset_create_localhost(
1250 - "netdata",
1251 - "dbengine_io_operations",
1252 - NULL,
1253 - "dbengine",
1254 - NULL,
1255 - "Netdata DB engine I/O operations",
1256 - "operations/s",
1257 - "netdata",
1258 - "stats",
1259 - 132007,
1260 - localhost->rrd_update_every,
1261 - RRDSET_TYPE_LINE);
2232 + "netdata",
2233 + "dbengine_io_operations",
2234 + NULL,
2235 + "dbengine io",
2236 + NULL,
2237 + "Netdata DB engine I/O operations",
2238 + "operations/s",
2239 + "netdata",
2240 + "stats",
2241 + priority,
2242 + localhost->rrd_update_every,
2243 + RRDSET_TYPE_LINE);
2244
2245 rd_reads = rrddim_add(st_io_stats, "reads", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2246 rd_writes = rrddim_add(st_io_stats, "writes", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
2247 }
2248 + priority++;
2249
2250 rrddim_set_by_pointer(st_io_stats, rd_reads, (collected_number)stats_array[18]);
2251 rrddim_set_by_pointer(st_io_stats, rd_writes, (collected_number)stats_array[16]);
@@ -1279,24 +2262,25 @@ static void dbengine_statistics_charts(void) {
2262
2263 if (unlikely(!st_errors)) {
2264 st_errors = rrdset_create_localhost(
1282 - "netdata",
1283 - "dbengine_global_errors",
1284 - NULL,
1285 - "dbengine",
1286 - NULL,
1287 - "Netdata DB engine errors",
1288 - "errors/s",
1289 - "netdata",
1290 - "stats",
1291 - 132008,
1292 - localhost->rrd_update_every,
1293 - RRDSET_TYPE_LINE);
2265 + "netdata",
2266 + "dbengine_global_errors",
2267 + NULL,
2268 + "dbengine io",
2269 + NULL,
2270 + "Netdata DB engine errors",
2271 + "errors/s",
2272 + "netdata",
2273 + "stats",
2274 + priority,
2275 + localhost->rrd_update_every,
2276 + RRDSET_TYPE_LINE);
2277
2278 rd_io_errors = rrddim_add(st_errors, "io_errors", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2279 rd_fs_errors = rrddim_add(st_errors, "fs_errors", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2280 pg_cache_over_half_dirty_events =
1298 - rrddim_add(st_errors, "pg_cache_over_half_dirty_events", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2281 + rrddim_add(st_errors, "pg_cache_over_half_dirty_events", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
2282 }
2283 + priority++;
2284
2285 rrddim_set_by_pointer(st_errors, rd_io_errors, (collected_number)stats_array[30]);
2286 rrddim_set_by_pointer(st_errors, rd_fs_errors, (collected_number)stats_array[31]);
@@ -1313,86 +2297,31 @@ static void dbengine_statistics_charts(void) {
2297
2298 if (unlikely(!st_fd)) {
2299 st_fd = rrdset_create_localhost(
1316 - "netdata",
1317 - "dbengine_global_file_descriptors",
1318 - NULL,
1319 - "dbengine",
1320 - NULL,
1321 - "Netdata DB engine File Descriptors",
1322 - "descriptors",
1323 - "netdata",
1324 - "stats",
1325 - 132009,
1326 - localhost->rrd_update_every,
1327 - RRDSET_TYPE_LINE);
2300 + "netdata",
2301 + "dbengine_global_file_descriptors",
2302 + NULL,
2303 + "dbengine io",
2304 + NULL,
2305 + "Netdata DB engine File Descriptors",
2306 + "descriptors",
2307 + "netdata",
2308 + "stats",
2309 + priority,
2310 + localhost->rrd_update_every,
2311 + RRDSET_TYPE_LINE);
2312
2313 rd_fd_current = rrddim_add(st_fd, "current", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2314 rd_fd_max = rrddim_add(st_fd, "max", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
2315 }
2316 + priority++;
2317
2318 rrddim_set_by_pointer(st_fd, rd_fd_current, (collected_number)stats_array[32]);
2319 /* Careful here, modify this accordingly if the File-Descriptor budget ever changes */
2320 rrddim_set_by_pointer(st_fd, rd_fd_max, (collected_number)rlimit_nofile.rlim_cur / 4);
2321 rrdset_done(st_fd);
2322 }
1338 -
1339 - // ----------------------------------------------------------------
1340 -
1341 - {
1342 - static RRDSET *st_ram_usage = NULL;
1343 - static RRDDIM *rd_cached = NULL;
1344 - static RRDDIM *rd_pinned = NULL;
1345 - static RRDDIM *rd_cache_metadata = NULL;
1346 - static RRDDIM *rd_index_metadata = NULL;
1347 - static RRDDIM *rd_pages_metadata = NULL;
1348 -
1349 - collected_number API_producers, populated_pages, cache_metadata, pages_on_disk,
1350 - page_cache_descriptors, index_metadata, pages_metadata;
1351 -
1352 - if (unlikely(!st_ram_usage)) {
1353 - st_ram_usage = rrdset_create_localhost(
1354 - "netdata",
1355 - "dbengine_ram",
1356 - NULL,
1357 - "dbengine",
1358 - NULL,
1359 - "Netdata DB engine RAM usage",
1360 - "MiB",
1361 - "netdata",
1362 - "stats",
1363 - 132010,
1364 - localhost->rrd_update_every,
1365 - RRDSET_TYPE_STACKED);
1366 -
1367 - rd_cached = rrddim_add(st_ram_usage, "cache", NULL, RRDENG_BLOCK_SIZE, 1024*1024, RRD_ALGORITHM_ABSOLUTE);
1368 - rd_pinned = rrddim_add(st_ram_usage, "collectors", NULL, RRDENG_BLOCK_SIZE, 1024*1024, RRD_ALGORITHM_ABSOLUTE);
1369 - rd_cache_metadata = rrddim_add(st_ram_usage, "cache metadata", NULL, 1, 1024*1024, RRD_ALGORITHM_ABSOLUTE);
1370 - rd_pages_metadata = rrddim_add(st_ram_usage, "pages metadata", NULL, 1, 1024*1024, RRD_ALGORITHM_ABSOLUTE);
1371 - rd_index_metadata = rrddim_add(st_ram_usage, "index metadata", NULL, 1, 1024*1024, RRD_ALGORITHM_ABSOLUTE);
1372 - }
1373 -
1374 - API_producers = (collected_number)stats_array[0];
1375 - pages_on_disk = (collected_number)stats_array[2];
1376 - populated_pages = (collected_number)stats_array[3];
1377 - page_cache_descriptors = (collected_number)stats_array[27];
1378 -
1379 - cache_metadata = page_cache_descriptors * sizeof(struct page_cache_descr);
1380 -
1381 - pages_metadata = pages_on_disk * sizeof(struct rrdeng_page_descr);
1382 -
1383 - /* This is an empirical estimation for Judy array indexing and extent structures */
1384 - index_metadata = pages_on_disk * 58;
1385 -
1386 - rrddim_set_by_pointer(st_ram_usage, rd_cached, populated_pages - API_producers);
1387 - rrddim_set_by_pointer(st_ram_usage, rd_pinned, API_producers);
1388 - rrddim_set_by_pointer(st_ram_usage, rd_cache_metadata, cache_metadata);
1389 - rrddim_set_by_pointer(st_ram_usage, rd_pages_metadata, pages_metadata);
1390 - rrddim_set_by_pointer(st_ram_usage, rd_index_metadata, index_metadata);
1391 - rrdset_done(st_ram_usage);
1392 - }
2323 }
2324 }
1395 -#endif
2325 }
2326
2327 static void update_strings_charts() {
@@ -2113,6 +3042,7 @@ static struct worker_utilization all_workers_utilization[] = {
3042 { .name = "STREAMRCV", .family = "workers streaming receive", .priority = 1000000 },
3043 { .name = "STREAMSND", .family = "workers streaming send", .priority = 1000000 },
3044 { .name = "DBENGINE", .family = "workers dbengine instances", .priority = 1000000 },
3045 + { .name = "LIBUV", .family = "workers libuv threadpool", .priority = 1000000 },
3046 { .name = "WEB", .family = "workers web server", .priority = 1000000 },
3047 { .name = "ACLKQUERY", .family = "workers aclk query", .priority = 1000000 },
3048 { .name = "ACLKSYNC", .family = "workers aclk host sync", .priority = 1000000 },
@@ -2671,6 +3601,9 @@ static void worker_utilization_charts_callback(void *ptr
3601 // find the worker_thread in the list
3602 struct worker_thread *wt = worker_thread_find_or_create(wu, pid);
3603
3604 + if(utilization_usec > duration_usec)
3605 + utilization_usec = duration_usec;
3606 +
3607 wt->enabled = true;
3608 wt->busy_time = utilization_usec;
3609 wt->jobs_started = jobs_started;
@@ -2842,22 +3775,19 @@ void *global_statistics_main(void *ptr)
3775 // to make sure we are not close to any other thread
3776 hb.randomness = 0;
3777
2845 - while (!netdata_exit) {
3778 + while (service_running(SERVICE_COLLECTORS)) {
3779 worker_is_idle();
3780 heartbeat_next(&hb, step);
3781
3782 worker_is_busy(WORKER_JOB_GLOBAL);
3783 global_statistics_charts();
3784
2852 - worker_is_busy(WORKER_JOB_SQLITE3);
2853 - sqlite3_statistics_charts();
2854 -
3785 worker_is_busy(WORKER_JOB_REGISTRY);
3786 registry_statistics();
3787
3788 if(dbengine_enabled) {
3789 worker_is_busy(WORKER_JOB_DBENGINE);
2860 - dbengine_statistics_charts();
3790 + dbengine2_statistics_charts();
3791 }
3792
3793 worker_is_busy(WORKER_JOB_HEARTBEAT);
@@ -2912,7 +3842,7 @@ void *global_statistics_workers_main(void *ptr)
3842 heartbeat_t hb;
3843 heartbeat_init(&hb);
3844
2915 - while (!netdata_exit) {
3845 + while (service_running(SERVICE_COLLECTORS)) {
3846 worker_is_idle();
3847 heartbeat_next(&hb, step);
3848
@@ -2924,3 +3854,45 @@ void *global_statistics_workers_main(void *ptr)
3854 return NULL;
3855 }
3856
3857 +// ---------------------------------------------------------------------------------------------------------------------
3858 +// sqlite3 thread
3859 +
3860 +static void global_statistics_sqlite3_cleanup(void *ptr)
3861 +{
3862 + worker_unregister();
3863 +
3864 + struct netdata_static_thread *static_thread = (struct netdata_static_thread *)ptr;
3865 + static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
3866 +
3867 + info("cleaning up...");
3868 +
3869 + static_thread->enabled = NETDATA_MAIN_THREAD_EXITED;
3870 +}
3871 +
3872 +void *global_statistics_sqlite3_main(void *ptr)
3873 +{
3874 + global_statistics_register_workers();
3875 +
3876 + netdata_thread_cleanup_push(global_statistics_sqlite3_cleanup, ptr);
3877 +
3878 + int update_every =
3879 + (int)config_get_number(CONFIG_SECTION_GLOBAL_STATISTICS, "update every", localhost->rrd_update_every);
3880 + if (update_every < localhost->rrd_update_every)
3881 + update_every = localhost->rrd_update_every;
3882 +
3883 + usec_t step = update_every * USEC_PER_SEC;
3884 + heartbeat_t hb;
3885 + heartbeat_init(&hb);
3886 +
3887 + while (service_running(SERVICE_COLLECTORS)) {
3888 + worker_is_idle();
3889 + heartbeat_next(&hb, step);
3890 +
3891 + worker_is_busy(WORKER_JOB_SQLITE3);
3892 + sqlite3_statistics_charts();
3893 + }
3894 +
3895 + netdata_thread_cleanup_pop(1);
3896 + return NULL;
3897 +}
3898 +
daemon/main.c
+363 -24
@@ -8,6 +8,8 @@ bool unittest_running = false;
8 int netdata_zero_metrics_enabled;
9 int netdata_anonymous_statistics_enabled;
10
11 +int libuv_worker_threads = MIN_LIBUV_WORKER_THREADS;
12 +
13 struct netdata_static_thread *static_threads;
14
15 struct config netdata_config = {
@@ -23,11 +25,267 @@ struct config netdata_config = {
25 }
26 };
27
26 -void netdata_cleanup_and_exit(int ret) {
27 - // enabling this, is wrong
28 - // because the threads will be cancelled while cleaning up
29 - // netdata_exit = 1;
28 +typedef struct service_thread {
29 + pid_t tid;
30 + SERVICE_THREAD_TYPE type;
31 + SERVICE_TYPE services;
32 + char name[NETDATA_THREAD_NAME_MAX + 1];
33 + bool cancelled;
34 +
35 + union {
36 + netdata_thread_t netdata_thread;
37 + uv_thread_t uv_thread;
38 + };
39 +
40 + force_quit_t force_quit_callback;
41 + request_quit_t request_quit_callback;
42 + void *data;
43 +} SERVICE_THREAD;
44 +
45 +struct service_globals {
46 + SERVICE_TYPE running;
47 + SPINLOCK lock;
48 + Pvoid_t pid_judy;
49 +} service_globals = {
50 + .running = ~0,
51 + .pid_judy = NULL,
52 +};
53 +
54 +SERVICE_THREAD *service_register(SERVICE_THREAD_TYPE thread_type, request_quit_t request_quit_callback, force_quit_t force_quit_callback, void *data, bool update __maybe_unused) {
55 + SERVICE_THREAD *sth = NULL;
56 + pid_t tid = gettid();
57 +
58 + netdata_spinlock_lock(&service_globals.lock);
59 + Pvoid_t *PValue = JudyLIns(&service_globals.pid_judy, tid, PJE0);
60 + if(!*PValue) {
61 + sth = callocz(1, sizeof(SERVICE_THREAD));
62 + sth->tid = tid;
63 + sth->type = thread_type;
64 + sth->request_quit_callback = request_quit_callback;
65 + sth->force_quit_callback = force_quit_callback;
66 + sth->data = data;
67 + os_thread_get_current_name_np(sth->name);
68 + *PValue = sth;
69 +
70 + switch(thread_type) {
71 + case SERVICE_THREAD_TYPE_NETDATA:
72 + sth->netdata_thread = netdata_thread_self();
73 + break;
74 +
75 + case SERVICE_THREAD_TYPE_LIBUV:
76 + sth->uv_thread = uv_thread_self();
77 + break;
78 + }
79 + }
80 + else {
81 + sth = *PValue;
82 + }
83 + netdata_spinlock_unlock(&service_globals.lock);
84 +
85 + return sth;
86 +}
87 +
88 +void service_exits(void) {
89 + pid_t tid = gettid();
90 +
91 + netdata_spinlock_lock(&service_globals.lock);
92 + Pvoid_t *PValue = JudyLGet(service_globals.pid_judy, tid, PJE0);
93 + if(PValue) {
94 + freez(*PValue);
95 + JudyLDel(&service_globals.pid_judy, tid, PJE0);
96 + }
97 + netdata_spinlock_unlock(&service_globals.lock);
98 +}
99 +
100 +bool service_running(SERVICE_TYPE service) {
101 + static __thread SERVICE_THREAD *sth = NULL;
102 +
103 + if(unlikely(!sth))
104 + sth = service_register(SERVICE_THREAD_TYPE_NETDATA, NULL, NULL, NULL, false);
105 +
106 + if(netdata_exit)
107 + __atomic_store_n(&service_globals.running, 0, __ATOMIC_RELAXED);
108 +
109 + if(service == 0)
110 + service = sth->services;
111 +
112 + sth->services |= service;
113 +
114 + return ((__atomic_load_n(&service_globals.running, __ATOMIC_RELAXED) & service) == service);
115 +}
116 +
117 +void service_signal_exit(SERVICE_TYPE service) {
118 + __atomic_and_fetch(&service_globals.running, ~(service), __ATOMIC_RELAXED);
119 +
120 + netdata_spinlock_lock(&service_globals.lock);
121 +
122 + Pvoid_t *PValue;
123 + Word_t tid = 0;
124 + bool first = true;
125 + while((PValue = JudyLFirstThenNext(service_globals.pid_judy, &tid, &first))) {
126 + SERVICE_THREAD *sth = *PValue;
127 +
128 + if((sth->services & service) && sth->request_quit_callback) {
129 + netdata_spinlock_unlock(&service_globals.lock);
130 + sth->request_quit_callback(sth->data);
131 + netdata_spinlock_lock(&service_globals.lock);
132 + continue;
133 + }
134 + }
135 +
136 + netdata_spinlock_unlock(&service_globals.lock);
137 +}
138 +
139 +static void service_to_buffer(BUFFER *wb, SERVICE_TYPE service) {
140 + if(service & SERVICE_MAINTENANCE)
141 + buffer_strcat(wb, "MAINTENANCE ");
142 + if(service & SERVICE_COLLECTORS)
143 + buffer_strcat(wb, "COLLECTORS ");
144 + if(service & SERVICE_ML_TRAINING)
145 + buffer_strcat(wb, "ML_TRAINING ");
146 + if(service & SERVICE_ML_PREDICTION)
147 + buffer_strcat(wb, "ML_PREDICTION ");
148 + if(service & SERVICE_REPLICATION)
149 + buffer_strcat(wb, "REPLICATION ");
150 + if(service & ABILITY_DATA_QUERIES)
151 + buffer_strcat(wb, "DATA_QUERIES ");
152 + if(service & ABILITY_WEB_REQUESTS)
153 + buffer_strcat(wb, "WEB_REQUESTS ");
154 + if(service & SERVICE_WEB_SERVER)
155 + buffer_strcat(wb, "WEB_SERVER ");
156 + if(service & SERVICE_ACLK)
157 + buffer_strcat(wb, "ACLK ");
158 + if(service & SERVICE_HEALTH)
159 + buffer_strcat(wb, "HEALTH ");
160 + if(service & SERVICE_STREAMING)
161 + buffer_strcat(wb, "STREAMING ");
162 + if(service & ABILITY_STREAMING_CONNECTIONS)
163 + buffer_strcat(wb, "STREAMING_CONNECTIONS ");
164 + if(service & SERVICE_CONTEXT)
165 + buffer_strcat(wb, "CONTEXT ");
166 + if(service & SERVICE_ANALYTICS)
167 + buffer_strcat(wb, "ANALYTICS ");
168 + if(service & SERVICE_EXPORTERS)
169 + buffer_strcat(wb, "EXPORTERS ");
170 +}
171 +
172 +static bool service_wait_exit(SERVICE_TYPE service, usec_t timeout_ut) {
173 + BUFFER *service_list = buffer_create(1024);
174 + BUFFER *thread_list = buffer_create(1024);
175 + usec_t started_ut = now_monotonic_usec(), ended_ut;
176 + size_t running;
177 + SERVICE_TYPE running_services = 0;
178 +
179 + // cancel the threads
180 + running = 0;
181 + running_services = 0;
182 + {
183 + buffer_flush(thread_list);
184 +
185 + netdata_spinlock_lock(&service_globals.lock);
186 +
187 + Pvoid_t *PValue;
188 + Word_t tid = 0;
189 + bool first = true;
190 + while((PValue = JudyLFirstThenNext(service_globals.pid_judy, &tid, &first))) {
191 + SERVICE_THREAD *sth = *PValue;
192 + if(sth->services & service && sth->tid != gettid() && !sth->cancelled) {
193 + sth->cancelled = true;
194 +
195 + switch(sth->type) {
196 + case SERVICE_THREAD_TYPE_NETDATA:
197 + netdata_thread_cancel(sth->netdata_thread);
198 + break;
199 +
200 + case SERVICE_THREAD_TYPE_LIBUV:
201 + break;
202 + }
203 +
204 + if(running)
205 + buffer_strcat(thread_list, ", ");
206 +
207 + buffer_sprintf(thread_list, "'%s' (%d)", sth->name, sth->tid);
208 +
209 + running++;
210 + running_services |= sth->services & service;
211 +
212 + if(sth->force_quit_callback) {
213 + netdata_spinlock_unlock(&service_globals.lock);
214 + sth->force_quit_callback(sth->data);
215 + netdata_spinlock_lock(&service_globals.lock);
216 + continue;
217 + }
218 + }
219 + }
220 +
221 + netdata_spinlock_unlock(&service_globals.lock);
222 + }
223 +
224 + service_signal_exit(service);
225
226 + // signal them to stop
227 + size_t last_running = 0;
228 + size_t stale_time_ut = 0;
229 + usec_t sleep_ut = 500 * USEC_PER_MS;
230 + do {
231 + if(running != last_running)
232 + stale_time_ut = 0;
233 +
234 + last_running = running;
235 + running = 0;
236 + running_services = 0;
237 + buffer_flush(thread_list);
238 +
239 + netdata_spinlock_lock(&service_globals.lock);
240 +
241 + Pvoid_t *PValue;
242 + Word_t tid = 0;
243 + bool first = true;
244 + while((PValue = JudyLFirstThenNext(service_globals.pid_judy, &tid, &first))) {
245 + SERVICE_THREAD *sth = *PValue;
246 + if(sth->services & service && sth->tid != gettid()) {
247 + if(running)
248 + buffer_strcat(thread_list, ", ");
249 +
250 + buffer_sprintf(thread_list, "'%s' (%d)", sth->name, sth->tid);
251 +
252 + running_services |= sth->services & service;
253 + running++;
254 + }
255 + }
256 +
257 + netdata_spinlock_unlock(&service_globals.lock);
258 +
259 + if(running) {
260 + buffer_flush(service_list);
261 + service_to_buffer(service_list, running_services);
262 + info("SERVICE CONTROL: waiting for the following %zu services [ %s] to exit: %s",
263 + running, buffer_tostring(service_list),
264 + running <= 10 ? buffer_tostring(thread_list) : "");
265 + sleep_usec(sleep_ut);
266 + stale_time_ut += sleep_ut;
267 + }
268 +
269 + ended_ut = now_monotonic_usec();
270 + } while(running && (ended_ut - started_ut < timeout_ut || stale_time_ut < timeout_ut));
271 +
272 + if(running) {
273 + buffer_flush(service_list);
274 + service_to_buffer(service_list, running_services);
275 + info("SERVICE CONTROL: "
276 + "the following %zu service(s) [ %s] take too long to exit: %s; "
277 + "giving up on them...",
278 + running, buffer_tostring(service_list),
279 + buffer_tostring(thread_list));
280 + }
281 +
282 + buffer_free(thread_list);
283 + buffer_free(service_list);
284 +
285 + return (running == 0);
286 +}
287 +
288 +void netdata_cleanup_and_exit(int ret) {
289 error_log_limit_unlimited();
290 info("EXIT: netdata prepares to exit with code %d...", ret);
291
@@ -39,38 +297,82 @@ void netdata_cleanup_and_exit(int ret) {
297 snprintfz(agent_incomplete_shutdown_file, FILENAME_MAX, "%s/.agent_incomplete_shutdown", netdata_configured_varlib_dir);
298 (void) rename(agent_crash_file, agent_incomplete_shutdown_file);
299
42 - // cleanup/save the database and exit
300 + service_signal_exit(
301 + SERVICE_MAINTENANCE
302 + | ABILITY_DATA_QUERIES
303 + | ABILITY_WEB_REQUESTS
304 + | ABILITY_STREAMING_CONNECTIONS
305 + | SERVICE_ACLK
306 + );
307 +
308 + service_wait_exit(
309 + SERVICE_REPLICATION
310 + | SERVICE_EXPORTERS
311 + | SERVICE_ML_TRAINING
312 + | SERVICE_HEALTH
313 + | SERVICE_WEB_SERVER
314 + , 3 * USEC_PER_SEC);
315 +
316 + service_wait_exit(
317 + SERVICE_COLLECTORS
318 + | SERVICE_STREAMING
319 + , 3 * USEC_PER_SEC);
320 +
321 + service_wait_exit(
322 + SERVICE_ML_PREDICTION
323 + | SERVICE_CONTEXT
324 + , 3 * USEC_PER_SEC);
325 +
326 + service_wait_exit(
327 + SERVICE_MAINTENANCE
328 + , 3 * USEC_PER_SEC);
329 +
330 info("EXIT: cleaning up the database...");
331 rrdhost_cleanup_all();
332
46 - if(!ret) {
47 - // exit cleanly
333 + info("EXIT: metasync shutdown prepare...");
334 + metadata_sync_shutdown_prepare();
335
49 - // stop everything
50 - info("EXIT: stopping static threads...");
336 #ifdef ENABLE_ACLK
52 - aclk_sync_exit_all();
337 + aclk_sync_exit_all();
338 #endif
54 - cancel_main_threads();
339
56 - // free the database
57 - info("EXIT: freeing database memory...");
340 + service_wait_exit(
341 + SERVICE_ACLK
342 + , 3 * USEC_PER_SEC);
343 +
344 + // stop everything else
345 + service_wait_exit(~0, 10 * USEC_PER_SEC);
346 +
347 + info("EXIT: stopping static threads...");
348 + cancel_main_threads();
349 +
350 + if(!ret) {
351 + // exit cleanly
352 +
353 #ifdef ENABLE_DBENGINE
354 if(dbengine_enabled) {
355 + info("EXIT: flushing dbengine...");
356 for (size_t tier = 0; tier < storage_tiers; tier++)
357 rrdeng_prepare_exit(multidb_ctx[tier]);
358 }
359 #endif
64 - metadata_sync_shutdown_prepare();
360 +
361 + // free the database
362 + info("EXIT: freeing database memory...");
363 rrdhost_free_all();
364 +
365 metadata_sync_shutdown();
366 +
367 #ifdef ENABLE_DBENGINE
368 if(dbengine_enabled) {
369 + info("EXIT: stopping dbengine...");
370 for (size_t tier = 0; tier < storage_tiers; tier++)
371 rrdeng_exit(multidb_ctx[tier]);
372 }
373 #endif
374 }
375 +
376 sql_close_context_database();
377 sql_close_database();
378
@@ -679,8 +981,9 @@ static void get_netdata_configured_variables() {
981 // ------------------------------------------------------------------------
982 // get default Database Engine page cache size in MiB
983
682 - db_engine_use_malloc = config_get_boolean(CONFIG_SECTION_DB, "dbengine page cache with malloc", CONFIG_BOOLEAN_YES);
984 default_rrdeng_page_cache_mb = (int) config_get_number(CONFIG_SECTION_DB, "dbengine page cache size MB", default_rrdeng_page_cache_mb);
985 + db_engine_journal_check = config_get_boolean(CONFIG_SECTION_DB, "dbengine enable journal integrity check", CONFIG_BOOLEAN_NO);
986 +
987 if(default_rrdeng_page_cache_mb < RRDENG_MIN_PAGE_CACHE_SIZE_MB) {
988 error("Invalid page cache size %d given. Defaulting to %d.", default_rrdeng_page_cache_mb, RRDENG_MIN_PAGE_CACHE_SIZE_MB);
989 default_rrdeng_page_cache_mb = RRDENG_MIN_PAGE_CACHE_SIZE_MB;
@@ -731,14 +1034,14 @@ static void get_netdata_configured_variables() {
1034
1035 // --------------------------------------------------------------------
1036
734 - rrdset_free_obsolete_time = config_get_number(CONFIG_SECTION_DB, "cleanup obsolete charts after secs", rrdset_free_obsolete_time);
1037 + rrdset_free_obsolete_time_s = config_get_number(CONFIG_SECTION_DB, "cleanup obsolete charts after secs", rrdset_free_obsolete_time_s);
1038 // Current chart locking and invalidation scheme doesn't prevent Netdata from segmentation faults if a short
1039 // cleanup delay is set. Extensive stress tests showed that 10 seconds is quite a safe delay. Look at
1040 // https://github.com/netdata/netdata/pull/11222#issuecomment-868367920 for more information.
738 - if (rrdset_free_obsolete_time < 10) {
739 - rrdset_free_obsolete_time = 10;
1041 + if (rrdset_free_obsolete_time_s < 10) {
1042 + rrdset_free_obsolete_time_s = 10;
1043 info("The \"cleanup obsolete charts after seconds\" option was set to 10 seconds.");
741 - config_set_number(CONFIG_SECTION_DB, "cleanup obsolete charts after secs", rrdset_free_obsolete_time);
1044 + config_set_number(CONFIG_SECTION_DB, "cleanup obsolete charts after secs", rrdset_free_obsolete_time_s);
1045 }
1046
1047 gap_when_lost_iterations_above = (int)config_get_number(CONFIG_SECTION_DB, "gap when lost iterations above", gap_when_lost_iterations_above);
@@ -746,6 +1049,7 @@ static void get_netdata_configured_variables() {
1049 gap_when_lost_iterations_above = 1;
1050 config_set_number(CONFIG_SECTION_DB, "gap when lost iterations above", gap_when_lost_iterations_above);
1051 }
1052 + gap_when_lost_iterations_above += 2;
1053
1054 // --------------------------------------------------------------------
1055 // get various system parameters
@@ -874,6 +1178,10 @@ void post_conf_load(char **user)
1178 appconfig_get(&cloud_config, CONFIG_SECTION_GLOBAL, "cloud base url", DEFAULT_CLOUD_BASE_URL);
1179 }
1180
1181 +int pgc_unittest(void);
1182 +int mrg_unittest(void);
1183 +int julytest(void);
1184 +
1185 int main(int argc, char **argv) {
1186 int i;
1187 int config_loaded = 0;
@@ -1001,7 +1309,7 @@ int main(int argc, char **argv) {
1309 default_health_enabled = 0;
1310 storage_tiers = 1;
1311 registry_init();
1004 - if(rrd_init("unittest", NULL)) {
1312 + if(rrd_init("unittest", NULL, true)) {
1313 fprintf(stderr, "rrd_init failed for unittest\n");
1314 return 1;
1315 }
@@ -1056,6 +1364,18 @@ int main(int argc, char **argv) {
1364 unittest_running = true;
1365 return metadata_unittest();
1366 }
1367 + else if(strcmp(optarg, "pgctest") == 0) {
1368 + unittest_running = true;
1369 + return pgc_unittest();
1370 + }
1371 + else if(strcmp(optarg, "mrgtest") == 0) {
1372 + unittest_running = true;
1373 + return mrg_unittest();
1374 + }
1375 + else if(strcmp(optarg, "julytest") == 0) {
1376 + unittest_running = true;
1377 + return julytest();
1378 + }
1379 else if(strncmp(optarg, createdataset_string, strlen(createdataset_string)) == 0) {
1380 optarg += strlen(createdataset_string);
1381 unsigned history_seconds = strtoul(optarg, NULL, 0);
@@ -1345,9 +1665,29 @@ int main(int argc, char **argv) {
1665 // initialize the system clocks
1666 clocks_init();
1667
1348 - // prepare configuration environment variables for the plugins
1668 + // set libuv worker threads
1669 + libuv_worker_threads = get_system_cpus() * 2;
1670 +
1671 + if(libuv_worker_threads < MIN_LIBUV_WORKER_THREADS)
1672 + libuv_worker_threads = MIN_LIBUV_WORKER_THREADS;
1673 +
1674 + if(libuv_worker_threads > MAX_LIBUV_WORKER_THREADS)
1675 + libuv_worker_threads = MAX_LIBUV_WORKER_THREADS;
1676 +
1677
1350 - setenv("UV_THREADPOOL_SIZE", config_get(CONFIG_SECTION_GLOBAL, "libuv worker threads", "16"), 1);
1678 + libuv_worker_threads = config_get_number(CONFIG_SECTION_GLOBAL, "libuv worker threads", libuv_worker_threads);
1679 + if(libuv_worker_threads < MIN_LIBUV_WORKER_THREADS) {
1680 + libuv_worker_threads = MIN_LIBUV_WORKER_THREADS;
1681 + config_set_number(CONFIG_SECTION_GLOBAL, "libuv worker threads", libuv_worker_threads);
1682 + }
1683 +
1684 + {
1685 + char buf[20 + 1];
1686 + snprintfz(buf, 20, "%d", libuv_worker_threads);
1687 + setenv("UV_THREADPOOL_SIZE", buf, 1);
1688 + }
1689 +
1690 + // prepare configuration environment variables for the plugins
1691 get_netdata_configured_variables();
1692 set_global_environment();
1693
@@ -1426,7 +1766,6 @@ int main(int argc, char **argv) {
1766 // setup threads configs
1767 default_stacksize = netdata_threads_init();
1768
1429 -
1769 // --------------------------------------------------------------------
1770 // check which threads are enabled and initialize them
1771
@@ -1504,7 +1843,7 @@ int main(int argc, char **argv) {
1843 system_info->hops = 0;
1844 get_install_type(&system_info->install_type, &system_info->prebuilt_arch, &system_info->prebuilt_dist);
1845
1507 - if(rrd_init(netdata_configured_hostname, system_info))
1846 + if(rrd_init(netdata_configured_hostname, system_info, false))
1847 fatal("Cannot initialize localhost instance with name '%s'.", netdata_configured_hostname);
1848
1849 char agent_crash_file[FILENAME_MAX + 1];
daemon/main.h
+30
@@ -27,4 +27,34 @@ int killpid(pid_t pid);
27 void netdata_cleanup_and_exit(int ret) NORETURN;
28 void send_statistics(const char *action, const char *action_result, const char *action_data);
29
30 +typedef enum {
31 + ABILITY_DATA_QUERIES = (1 << 0),
32 + ABILITY_WEB_REQUESTS = (1 << 1),
33 + ABILITY_STREAMING_CONNECTIONS = (1 << 2),
34 + SERVICE_MAINTENANCE = (1 << 3),
35 + SERVICE_COLLECTORS = (1 << 4),
36 + SERVICE_ML_TRAINING = (1 << 5),
37 + SERVICE_ML_PREDICTION = (1 << 6),
38 + SERVICE_REPLICATION = (1 << 7),
39 + SERVICE_WEB_SERVER = (1 << 8),
40 + SERVICE_ACLK = (1 << 9),
41 + SERVICE_HEALTH = (1 << 10),
42 + SERVICE_STREAMING = (1 << 11),
43 + SERVICE_CONTEXT = (1 << 12),
44 + SERVICE_ANALYTICS = (1 << 13),
45 + SERVICE_EXPORTERS = (1 << 14),
46 +} SERVICE_TYPE;
47 +
48 +typedef enum {
49 + SERVICE_THREAD_TYPE_NETDATA,
50 + SERVICE_THREAD_TYPE_LIBUV,
51 +} SERVICE_THREAD_TYPE;
52 +
53 +typedef void (*force_quit_t)(void *data);
54 +typedef void (*request_quit_t)(void *data);
55 +
56 +void service_exits(void);
57 +bool service_running(SERVICE_TYPE service);
58 +struct service_thread *service_register(SERVICE_THREAD_TYPE thread_type, request_quit_t request_quit_callback, force_quit_t force_quit_callback, void *data, bool update __maybe_unused);
59 +
60 #endif /* NETDATA_MAIN_H */
daemon/service.c
+20 -12
@@ -22,6 +22,10 @@
22 #define WORKER_JOB_SAVE_CHART 13
23 #define WORKER_JOB_DELETE_CHART 14
24 #define WORKER_JOB_FREE_DIMENSION 15
25 +#define WORKER_JOB_PGC_MAIN_EVICT 16
26 +#define WORKER_JOB_PGC_MAIN_FLUSH 17
27 +#define WORKER_JOB_PGC_OPEN_EVICT 18
28 +#define WORKER_JOB_PGC_OPEN_FLUSH 19
29
30 static void svc_rrddim_obsolete_to_archive(RRDDIM *rd) {
31 RRDSET *st = rd->rrdset;
@@ -83,7 +87,7 @@ static bool svc_rrdset_archive_obsolete_dimensions(RRDSET *st, bool all_dimensio
87 dfe_start_write(st->rrddim_root_index, rd) {
88 if(unlikely(
89 all_dimensions ||
86 - (rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE) && (rd->last_collected_time.tv_sec + rrdset_free_obsolete_time < now))
90 + (rrddim_flag_check(rd, RRDDIM_FLAG_OBSOLETE) && (rd->last_collected_time.tv_sec + rrdset_free_obsolete_time_s < now))
91 )) {
92
93 if(dictionary_acquired_item_references(rd_dfe.item) == 1) {
@@ -142,9 +146,9 @@ static void svc_rrdhost_cleanup_obsolete_charts(RRDHOST *host) {
146 continue;
147
148 if(unlikely(rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)
145 - && st->last_accessed_time + rrdset_free_obsolete_time < now
146 - && st->last_updated.tv_sec + rrdset_free_obsolete_time < now
147 - && st->last_collected_time.tv_sec + rrdset_free_obsolete_time < now
149 + && st->last_accessed_time_s + rrdset_free_obsolete_time_s < now
150 + && st->last_updated.tv_sec + rrdset_free_obsolete_time_s < now
151 + && st->last_collected_time.tv_sec + rrdset_free_obsolete_time_s < now
152 )) {
153 svc_rrdset_obsolete_to_archive(st);
154 }
@@ -166,10 +170,10 @@ static void svc_rrdset_check_obsoletion(RRDHOST *host) {
170 if(rrdset_is_replicating(st))
171 continue;
172
169 - last_entry_t = rrdset_last_entry_t(st);
173 + last_entry_t = rrdset_last_entry_s(st);
174
171 - if(last_entry_t && last_entry_t < host->senders_connect_time &&
172 - host->senders_connect_time + TIME_TO_RUN_OBSOLETIONS_ON_CHILD_CONNECT + ITERATIONS_TO_RUN_OBSOLETIONS_ON_CHILD_CONNECT * st->update_every
175 + if(last_entry_t && last_entry_t < host->child_connect_time &&
176 + host->child_connect_time + TIME_TO_RUN_OBSOLETIONS_ON_CHILD_CONNECT + ITERATIONS_TO_RUN_OBSOLETIONS_ON_CHILD_CONNECT * st->update_every
177 < now)
178
179 rrdset_is_obsolete(st);
@@ -196,10 +200,10 @@ static void svc_rrd_cleanup_obsolete_charts_from_all_hosts() {
200 && host->trigger_chart_obsoletion_check
201 && (
202 (
199 - host->senders_last_chart_command
200 - && host->senders_last_chart_command + host->health_delay_up_to < now_realtime_sec()
203 + host->child_last_chart_command
204 + && host->child_last_chart_command + host->health_delay_up_to < now_realtime_sec()
205 )
202 - || (host->senders_connect_time + TIME_TO_RUN_OBSOLETIONS_ON_CHILD_CONNECT < now_realtime_sec())
206 + || (host->child_connect_time + TIME_TO_RUN_OBSOLETIONS_ON_CHILD_CONNECT < now_realtime_sec())
207 )
208 ) {
209 svc_rrdset_check_obsoletion(host);
@@ -238,7 +242,7 @@ restart_after_removal:
242 }
243
244 worker_is_busy(WORKER_JOB_FREE_HOST);
241 - rrdhost_free(host, 0);
245 + rrdhost_free___while_having_rrd_wrlock(host, false);
246 goto restart_after_removal;
247 }
248
@@ -276,6 +280,10 @@ void *service_main(void *ptr)
280 worker_register_job_name(WORKER_JOB_SAVE_CHART, "save chart");
281 worker_register_job_name(WORKER_JOB_DELETE_CHART, "delete chart");
282 worker_register_job_name(WORKER_JOB_FREE_DIMENSION, "free dimension");
283 + worker_register_job_name(WORKER_JOB_PGC_MAIN_EVICT, "main cache evictions");
284 + worker_register_job_name(WORKER_JOB_PGC_MAIN_FLUSH, "main cache flushes");
285 + worker_register_job_name(WORKER_JOB_PGC_OPEN_EVICT, "open cache evictions");
286 + worker_register_job_name(WORKER_JOB_PGC_OPEN_FLUSH, "open cache flushes");
287
288 netdata_thread_cleanup_push(service_main_cleanup, ptr);
289 heartbeat_t hb;
@@ -284,7 +292,7 @@ void *service_main(void *ptr)
292
293 debug(D_SYSTEM, "Service thread starts");
294
287 - while (!netdata_exit) {
295 + while (service_running(SERVICE_MAINTENANCE)) {
296 worker_is_idle();
297 heartbeat_next(&hb, step);
298
daemon/static_threads.c
+12
@@ -7,6 +7,7 @@ void *analytics_main(void *ptr);
7 void *cpuidlejitter_main(void *ptr);
8 void *global_statistics_main(void *ptr);
9 void *global_statistics_workers_main(void *ptr);
10 +void *global_statistics_sqlite3_main(void *ptr);
11 void *health_main(void *ptr);
12 void *pluginsd_main(void *ptr);
13 void *service_main(void *ptr);
@@ -66,6 +67,17 @@ const struct netdata_static_thread static_threads_common[] = {
67 .init_routine = NULL,
68 .start_routine = global_statistics_workers_main
69 },
70 + {
71 + .name = "SQLITE3_STATS",
72 + .config_section = CONFIG_SECTION_PLUGINS,
73 + .config_name = "netdata monitoring",
74 + .env_name = "NETDATA_INTERNALS_MONITORING",
75 + .global_variable = &global_statistics_enabled,
76 + .enabled = 1,
77 + .thread = NULL,
78 + .init_routine = NULL,
79 + .start_routine = global_statistics_sqlite3_main
80 + },
81 {
82 .name = "PLUGINSD",
83 .config_section = NULL,
daemon/unit_test.c
+13 -12
@@ -1289,7 +1289,7 @@ int run_test(struct test *test)
1289 fprintf(stderr, " %s/%s: checking position %lu (at %"PRId64" secs), expecting value " NETDATA_DOUBLE_FORMAT
1290 ", found " NETDATA_DOUBLE_FORMAT ", %s\n",
1291 test->name, rrddim_name(rd), c+1,
1292 - (int64_t)((rrdset_first_entry_t(st) + c * st->update_every) - time_start),
1292 + (int64_t)((rrdset_first_entry_s(st) + c * st->update_every) - time_start),
1293 n, v, (same)?"OK":"### E R R O R ###");
1294
1295 if(!same) errors++;
@@ -1301,7 +1301,7 @@ int run_test(struct test *test)
1301 fprintf(stderr, " %s/%s: checking position %lu (at %"PRId64" secs), expecting value " NETDATA_DOUBLE_FORMAT
1302 ", found " NETDATA_DOUBLE_FORMAT ", %s\n",
1303 test->name, rrddim_name(rd2), c+1,
1304 - (int64_t)((rrdset_first_entry_t(st) + c * st->update_every) - time_start),
1304 + (int64_t)((rrdset_first_entry_s(st) + c * st->update_every) - time_start),
1305 n, v, (same)?"OK":"### E R R O R ###");
1306 if(!same) errors++;
1307 }
@@ -1775,7 +1775,6 @@ static inline void rrddim_set_by_pointer_fake_time(RRDDIM *rd, collected_number
1775 static RRDHOST *dbengine_rrdhost_find_or_create(char *name)
1776 {
1777 /* We don't want to drop metrics when generating load, we prefer to block data generation itself */
1778 - rrdeng_drop_metrics_under_page_cache_pressure = 0;
1778
1779 return rrdhost_find_or_create(
1780 name
@@ -1932,7 +1931,7 @@ static int test_dbengine_check_metrics(RRDSET *st[CHARTS], RRDDIM *rd[CHARTS][DI
1931 time_now = time_start + (c + 1) * update_every;
1932 for (i = 0 ; i < CHARTS ; ++i) {
1933 for (j = 0; j < DIMS; ++j) {
1935 - rd[i][j]->tiers[0]->query_ops->init(rd[i][j]->tiers[0]->db_metric_handle, &handle, time_now, time_now + QUERY_BATCH * update_every);
1934 + rd[i][j]->tiers[0]->query_ops->init(rd[i][j]->tiers[0]->db_metric_handle, &handle, time_now, time_now + QUERY_BATCH * update_every, STORAGE_PRIORITY_NORMAL);
1935 for (k = 0; k < QUERY_BATCH; ++k) {
1936 last = ((collected_number)i * DIMS) * REGION_POINTS[current_region] +
1937 j * REGION_POINTS[current_region] + c + k;
@@ -1940,8 +1939,8 @@ static int test_dbengine_check_metrics(RRDSET *st[CHARTS], RRDDIM *rd[CHARTS][DI
1939
1940 STORAGE_POINT sp = rd[i][j]->tiers[0]->query_ops->next_metric(&handle);
1941 value = sp.sum;
1943 - time_retrieved = sp.start_time;
1944 - end_time = sp.end_time;
1942 + time_retrieved = sp.start_time_s;
1943 + end_time = sp.end_time_s;
1944
1945 same = (roundndd(value) == roundndd(expected)) ? 1 : 0;
1946 if(!same) {
@@ -1993,7 +1992,8 @@ static int test_dbengine_check_rrdr(RRDSET *st[CHARTS], RRDDIM *rd[CHARTS][DIMS]
1992 ONEWAYALLOC *owa = onewayalloc_create(0);
1993 RRDR *r = rrd2rrdr_legacy(owa, st[i], points, time_start, time_end,
1994 RRDR_GROUPING_AVERAGE, 0, RRDR_OPTION_NATURAL_POINTS,
1996 - NULL, NULL, 0, 0, QUERY_SOURCE_UNITTEST);
1995 + NULL, NULL, 0, 0,
1996 + QUERY_SOURCE_UNITTEST, STORAGE_PRIORITY_NORMAL);
1997 if (!r) {
1998 fprintf(stderr, " DB-engine unittest %s: empty RRDR on region %d ### E R R O R ###\n", rrdset_name(st[i]), current_region);
1999 return ++errors;
@@ -2131,7 +2131,8 @@ int test_dbengine(void)
2131 ONEWAYALLOC *owa = onewayalloc_create(0);
2132 RRDR *r = rrd2rrdr_legacy(owa, st[i], points, time_start[0] + update_every,
2133 time_end[REGIONS - 1], RRDR_GROUPING_AVERAGE, 0,
2134 - RRDR_OPTION_NATURAL_POINTS, NULL, NULL, 0, 0, QUERY_SOURCE_UNITTEST);
2134 + RRDR_OPTION_NATURAL_POINTS, NULL, NULL, 0, 0,
2135 + QUERY_SOURCE_UNITTEST, STORAGE_PRIORITY_NORMAL);
2136
2137 if (!r) {
2138 fprintf(stderr, " DB-engine unittest %s: empty RRDR ### E R R O R ###\n", rrdset_name(st[i]));
@@ -2329,7 +2330,7 @@ void generate_dbengine_dataset(unsigned history_seconds)
2330 }
2331 freez(thread_info);
2332 rrd_wrlock();
2332 - rrdhost_free(host, 1);
2333 + rrdhost_free___while_having_rrd_wrlock(host, true);
2334 rrd_unlock();
2335 }
2336
@@ -2389,7 +2390,7 @@ static void query_dbengine_chart(void *arg)
2390 time_before = MIN(time_after + duration, time_max); /* up to 1 hour queries */
2391 }
2392
2392 - rd->tiers[0]->query_ops->init(rd->tiers[0]->db_metric_handle, &handle, time_after, time_before);
2393 + rd->tiers[0]->query_ops->init(rd->tiers[0]->db_metric_handle, &handle, time_after, time_before, STORAGE_PRIORITY_NORMAL);
2394 ++thread_info->queries_nr;
2395 for (time_now = time_after ; time_now <= time_before ; time_now += update_every) {
2396 generatedv = generate_dbengine_chart_value(i, j, time_now);
@@ -2407,8 +2408,8 @@ static void query_dbengine_chart(void *arg)
2408
2409 STORAGE_POINT sp = rd->tiers[0]->query_ops->next_metric(&handle);
2410 value = sp.sum;
2410 - time_retrieved = sp.start_time;
2411 - end_time = sp.end_time;
2411 + time_retrieved = sp.start_time_s;
2412 + end_time = sp.end_time_s;
2413
2414 if (!netdata_double_isnumber(value)) {
2415 if (!thread_info->delete_old_data) { /* data validation only when we don't delete */
database/engine/cache.c new
+2587
@@ -0,0 +1,2587 @@
1 +#include "cache.h"
2 +
3 +/* STATES AND TRANSITIONS
4 + *
5 + * entry | entry
6 + * v v
7 + * HOT -> DIRTY --> CLEAN --> EVICT
8 + * v | v
9 + * flush | evict
10 + * v | v
11 + * save | free
12 + * callback | callback
13 + *
14 + */
15 +
16 +typedef int32_t REFCOUNT;
17 +#define REFCOUNT_DELETING (-100)
18 +
19 +// to use arrayalloc uncomment the following line:
20 +#define PGC_WITH_ARAL 1
21 +
22 +typedef enum __attribute__ ((__packed__)) {
23 + // mutually exclusive flags
24 + PGC_PAGE_CLEAN = (1 << 0), // none of the following
25 + PGC_PAGE_DIRTY = (1 << 1), // contains unsaved data
26 + PGC_PAGE_HOT = (1 << 2), // currently being collected
27 +
28 + // flags related to various actions on each page
29 + PGC_PAGE_IS_BEING_DELETED = (1 << 3),
30 + PGC_PAGE_IS_BEING_MIGRATED_TO_V2 = (1 << 4),
31 + PGC_PAGE_HAS_NO_DATA_IGNORE_ACCESSES = (1 << 5),
32 + PGC_PAGE_HAS_BEEN_ACCESSED = (1 << 6),
33 +} PGC_PAGE_FLAGS;
34 +
35 +#define page_flag_check(page, flag) (__atomic_load_n(&((page)->flags), __ATOMIC_ACQUIRE) & (flag))
36 +#define page_flag_set(page, flag) __atomic_or_fetch(&((page)->flags), flag, __ATOMIC_RELEASE)
37 +#define page_flag_clear(page, flag) __atomic_and_fetch(&((page)->flags), ~(flag), __ATOMIC_RELEASE)
38 +
39 +#define page_get_status_flags(page) page_flag_check(page, PGC_PAGE_HOT | PGC_PAGE_DIRTY | PGC_PAGE_CLEAN)
40 +#define is_page_hot(page) (page_get_status_flags(page) == PGC_PAGE_HOT)
41 +#define is_page_dirty(page) (page_get_status_flags(page) == PGC_PAGE_DIRTY)
42 +#define is_page_clean(page) (page_get_status_flags(page) == PGC_PAGE_CLEAN)
43 +
44 +struct pgc_page {
45 + // indexing data
46 + Word_t section;
47 + Word_t metric_id;
48 + time_t start_time_s;
49 + time_t end_time_s;
50 + uint32_t update_every_s;
51 + uint32_t assumed_size;
52 +
53 + REFCOUNT refcount;
54 + uint16_t accesses; // counts the number of accesses on this page
55 + PGC_PAGE_FLAGS flags;
56 + SPINLOCK transition_spinlock; // when the page changes between HOT, DIRTY, CLEAN, we have to get this lock
57 +
58 + struct {
59 + struct pgc_page *next;
60 + struct pgc_page *prev;
61 + } link;
62 +
63 + void *data;
64 + uint8_t custom_data[];
65 +
66 + // IMPORTANT!
67 + // THIS STRUCTURE NEEDS TO BE INITIALIZED BY HAND!
68 +};
69 +
70 +struct pgc_linked_list {
71 + SPINLOCK spinlock;
72 + union {
73 + PGC_PAGE *base;
74 + Pvoid_t sections_judy;
75 + };
76 + PGC_PAGE_FLAGS flags;
77 + size_t version;
78 + size_t last_version_checked;
79 + bool linked_list_in_sections_judy; // when true, we use 'sections_judy', otherwise we use 'base'
80 + struct pgc_queue_statistics *stats;
81 +};
82 +
83 +struct pgc {
84 + struct {
85 + size_t partitions;
86 + size_t clean_size;
87 + size_t max_dirty_pages_per_call;
88 + size_t max_pages_per_inline_eviction;
89 + size_t max_skip_pages_per_inline_eviction;
90 + size_t max_flushes_inline;
91 + size_t max_workers_evict_inline;
92 + size_t additional_bytes_per_page;
93 + free_clean_page_callback pgc_free_clean_cb;
94 + save_dirty_page_callback pgc_save_dirty_cb;
95 + PGC_OPTIONS options;
96 +
97 + size_t severe_pressure_per1000;
98 + size_t aggressive_evict_per1000;
99 + size_t healthy_size_per1000;
100 + size_t evict_low_threshold_per1000;
101 +
102 + dynamic_target_cache_size_callback dynamic_target_size_cb;
103 + } config;
104 +
105 +#ifdef PGC_WITH_ARAL
106 + ARAL *aral;
107 +#endif
108 +
109 + PGC_CACHE_LINE_PADDING(0);
110 +
111 + struct pgc_index {
112 + netdata_rwlock_t rwlock;
113 + Pvoid_t sections_judy;
114 + } *index;
115 +
116 + PGC_CACHE_LINE_PADDING(1);
117 +
118 + struct {
119 + SPINLOCK spinlock;
120 + size_t per1000;
121 + } usage;
122 +
123 + PGC_CACHE_LINE_PADDING(2);
124 +
125 + struct pgc_linked_list clean; // LRU is applied here to free memory from the cache
126 +
127 + PGC_CACHE_LINE_PADDING(3);
128 +
129 + struct pgc_linked_list dirty; // in the dirty list, pages are ordered the way they were marked dirty
130 +
131 + PGC_CACHE_LINE_PADDING(4);
132 +
133 + struct pgc_linked_list hot; // in the hot list, pages are order the way they were marked hot
134 +
135 + PGC_CACHE_LINE_PADDING(5);
136 +
137 + struct pgc_statistics stats; // statistics
138 +
139 +#ifdef NETDATA_PGC_POINTER_CHECK
140 + PGC_CACHE_LINE_PADDING(6);
141 + netdata_mutex_t global_pointer_registry_mutex;
142 + Pvoid_t global_pointer_registry;
143 +#endif
144 +};
145 +
146 +
147 +
148 +// ----------------------------------------------------------------------------
149 +// validate each pointer is indexed once - internal checks only
150 +
151 +static inline void pointer_index_init(PGC *cache __maybe_unused) {
152 +#ifdef NETDATA_PGC_POINTER_CHECK
153 + netdata_mutex_init(&cache->global_pointer_registry_mutex);
154 +#else
155 + ;
156 +#endif
157 +}
158 +
159 +static inline void pointer_destroy_index(PGC *cache __maybe_unused) {
160 +#ifdef NETDATA_PGC_POINTER_CHECK
161 + netdata_mutex_lock(&cache->global_pointer_registry_mutex);
162 + JudyHSFreeArray(&cache->global_pointer_registry, PJE0);
163 + netdata_mutex_unlock(&cache->global_pointer_registry_mutex);
164 +#else
165 + ;
166 +#endif
167 +}
168 +static inline void pointer_add(PGC *cache __maybe_unused, PGC_PAGE *page __maybe_unused) {
169 +#ifdef NETDATA_PGC_POINTER_CHECK
170 + netdata_mutex_lock(&cache->global_pointer_registry_mutex);
171 + Pvoid_t *PValue = JudyHSIns(&cache->global_pointer_registry, &page, sizeof(void *), PJE0);
172 + if(*PValue != NULL)
173 + fatal("pointer already exists in registry");
174 + *PValue = page;
175 + netdata_mutex_unlock(&cache->global_pointer_registry_mutex);
176 +#else
177 + ;
178 +#endif
179 +}
180 +
181 +static inline void pointer_check(PGC *cache __maybe_unused, PGC_PAGE *page __maybe_unused) {
182 +#ifdef NETDATA_PGC_POINTER_CHECK
183 + netdata_mutex_lock(&cache->global_pointer_registry_mutex);
184 + Pvoid_t *PValue = JudyHSGet(cache->global_pointer_registry, &page, sizeof(void *));
185 + if(PValue == NULL)
186 + fatal("pointer is not found in registry");
187 + netdata_mutex_unlock(&cache->global_pointer_registry_mutex);
188 +#else
189 + ;
190 +#endif
191 +}
192 +
193 +static inline void pointer_del(PGC *cache __maybe_unused, PGC_PAGE *page __maybe_unused) {
194 +#ifdef NETDATA_PGC_POINTER_CHECK
195 + netdata_mutex_lock(&cache->global_pointer_registry_mutex);
196 + int ret = JudyHSDel(&cache->global_pointer_registry, &page, sizeof(void *), PJE0);
197 + if(!ret)
198 + fatal("pointer to be deleted does not exist in registry");
199 + netdata_mutex_unlock(&cache->global_pointer_registry_mutex);
200 +#else
201 + ;
202 +#endif
203 +}
204 +
205 +// ----------------------------------------------------------------------------
206 +// locking
207 +
208 +static size_t pgc_indexing_partition(PGC *cache, Word_t metric_id) {
209 + static __thread Word_t last_metric_id = 0;
210 + static __thread size_t last_partition = 0;
211 +
212 + if(metric_id == last_metric_id || cache->config.partitions == 1)
213 + return last_partition;
214 +
215 + last_metric_id = metric_id;
216 + last_partition = indexing_partition(metric_id, cache->config.partitions);
217 +
218 + return last_partition;
219 +}
220 +
221 +static void pgc_index_read_lock(PGC *cache, size_t partition) {
222 + netdata_rwlock_rdlock(&cache->index[partition].rwlock);
223 +}
224 +static void pgc_index_read_unlock(PGC *cache, size_t partition) {
225 + netdata_rwlock_unlock(&cache->index[partition].rwlock);
226 +}
227 +//static bool pgc_index_write_trylock(PGC *cache, size_t partition) {
228 +// return !netdata_rwlock_trywrlock(&cache->index[partition].rwlock);
229 +//}
230 +static void pgc_index_write_lock(PGC *cache, size_t partition) {
231 + netdata_rwlock_wrlock(&cache->index[partition].rwlock);
232 +}
233 +static void pgc_index_write_unlock(PGC *cache, size_t partition) {
234 + netdata_rwlock_unlock(&cache->index[partition].rwlock);
235 +}
236 +
237 +static inline bool pgc_ll_trylock(PGC *cache __maybe_unused, struct pgc_linked_list *ll) {
238 + return netdata_spinlock_trylock(&ll->spinlock);
239 +}
240 +
241 +static inline void pgc_ll_lock(PGC *cache __maybe_unused, struct pgc_linked_list *ll) {
242 + netdata_spinlock_lock(&ll->spinlock);
243 +}
244 +
245 +static inline void pgc_ll_unlock(PGC *cache __maybe_unused, struct pgc_linked_list *ll) {
246 + netdata_spinlock_unlock(&ll->spinlock);
247 +}
248 +
249 +static inline bool page_transition_trylock(PGC *cache __maybe_unused, PGC_PAGE *page) {
250 + return netdata_spinlock_trylock(&page->transition_spinlock);
251 +}
252 +
253 +static inline void page_transition_lock(PGC *cache __maybe_unused, PGC_PAGE *page) {
254 + netdata_spinlock_lock(&page->transition_spinlock);
255 +}
256 +
257 +static inline void page_transition_unlock(PGC *cache __maybe_unused, PGC_PAGE *page) {
258 + netdata_spinlock_unlock(&page->transition_spinlock);
259 +}
260 +
261 +// ----------------------------------------------------------------------------
262 +// evictions control
263 +
264 +static inline size_t cache_usage_per1000(PGC *cache, size_t *size_to_evict) {
265 +
266 + if(size_to_evict)
267 + netdata_spinlock_lock(&cache->usage.spinlock);
268 +
269 + else if(!netdata_spinlock_trylock(&cache->usage.spinlock))
270 + return __atomic_load_n(&cache->usage.per1000, __ATOMIC_RELAXED);
271 +
272 + size_t current_cache_size;
273 + size_t wanted_cache_size;
274 + size_t per1000;
275 +
276 + if(cache->config.options & PGC_OPTIONS_AUTOSCALE) {
277 + size_t dirty_max = __atomic_load_n(&cache->dirty.stats->max_size, __ATOMIC_RELAXED);
278 + size_t hot_max = __atomic_load_n(&cache->hot.stats->max_size, __ATOMIC_RELAXED);
279 + size_t dirty = __atomic_load_n(&cache->dirty.stats->size, __ATOMIC_RELAXED);
280 + size_t hot = __atomic_load_n(&cache->hot.stats->size, __ATOMIC_RELAXED);
281 +
282 + size_t max_size1 = MAX(hot_max, hot) * 2;
283 + size_t max_size2 = hot_max + ((dirty_max < hot_max / 2) ? hot_max / 2 : dirty_max * 2);
284 + wanted_cache_size = MIN(max_size1, max_size2);
285 +
286 + if(cache->config.dynamic_target_size_cb) {
287 + size_t wanted_cache_size_cb = cache->config.dynamic_target_size_cb();
288 + if(wanted_cache_size_cb > wanted_cache_size)
289 + wanted_cache_size = wanted_cache_size_cb;
290 + }
291 +
292 + if (wanted_cache_size < hot + dirty + cache->config.clean_size)
293 + wanted_cache_size = hot + dirty + cache->config.clean_size;
294 + }
295 + else {
296 + size_t dirty = __atomic_load_n(&cache->dirty.stats->size, __ATOMIC_RELAXED);
297 + size_t hot = __atomic_load_n(&cache->hot.stats->size, __ATOMIC_RELAXED);
298 +
299 + wanted_cache_size = hot + dirty + cache->config.clean_size;
300 + }
301 +
302 + current_cache_size = __atomic_load_n(&cache->stats.size, __ATOMIC_RELAXED);
303 +
304 + per1000 = current_cache_size * 1000 / wanted_cache_size;
305 +
306 + __atomic_store_n(&cache->usage.per1000, per1000, __ATOMIC_RELAXED);
307 + __atomic_store_n(&cache->stats.wanted_cache_size, wanted_cache_size, __ATOMIC_RELAXED);
308 + __atomic_store_n(&cache->stats.current_cache_size, current_cache_size, __ATOMIC_RELAXED);
309 +
310 + if(size_to_evict) {
311 + size_t target = wanted_cache_size / 1000 * cache->config.evict_low_threshold_per1000;
312 + if(current_cache_size > target)
313 + *size_to_evict = current_cache_size - target;
314 + else
315 + *size_to_evict = 0;
316 + }
317 +
318 + netdata_spinlock_unlock(&cache->usage.spinlock);
319 + return per1000;
320 +}
321 +
322 +static inline bool cache_under_severe_pressure(PGC *cache) {
323 + if(unlikely(cache_usage_per1000(cache, NULL) >= cache->config.severe_pressure_per1000)) {
324 + __atomic_add_fetch(&cache->stats.events_cache_under_severe_pressure, 1, __ATOMIC_RELAXED);
325 + return true;
326 + }
327 +
328 + return false;
329 +}
330 +
331 +static inline bool cache_needs_space_aggressively(PGC *cache) {
332 + if(unlikely(cache_usage_per1000(cache, NULL) >= cache->config.aggressive_evict_per1000)) {
333 + __atomic_add_fetch(&cache->stats.events_cache_needs_space_aggressively, 1, __ATOMIC_RELAXED);
334 + return true;
335 + }
336 +
337 + return false;
338 +}
339 +
340 +#define cache_above_healthy_limit(cache) (cache_usage_per1000(cache, NULL) >= (cache)->config.healthy_size_per1000)
341 +
342 +typedef bool (*evict_filter)(PGC_PAGE *page, void *data);
343 +static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evict, bool wait, bool all_of_them, evict_filter filter, void *data);
344 +#define evict_pages(cache, max_skip, max_evict, wait, all_of_them) evict_pages_with_filter(cache, max_skip, max_evict, wait, all_of_them, NULL, NULL)
345 +
346 +static inline void evict_on_clean_page_added(PGC *cache __maybe_unused) {
347 + if((cache->config.options & PGC_OPTIONS_EVICT_PAGES_INLINE) || cache_needs_space_aggressively(cache)) {
348 + evict_pages(cache,
349 + cache->config.max_skip_pages_per_inline_eviction,
350 + cache->config.max_pages_per_inline_eviction,
351 + false, false);
352 + }
353 +}
354 +
355 +static inline void evict_on_page_release_when_permitted(PGC *cache __maybe_unused) {
356 + if (unlikely((cache->config.options & PGC_OPTIONS_EVICT_PAGES_INLINE) || cache_needs_space_aggressively(cache))) {
357 + evict_pages(cache,
358 + cache->config.max_skip_pages_per_inline_eviction,
359 + cache->config.max_pages_per_inline_eviction,
360 + false, false);
361 + }
362 +}
363 +
364 +// ----------------------------------------------------------------------------
365 +// flushing control
366 +
367 +static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wait, bool all_of_them);
368 +
369 +static inline bool flushing_critical(PGC *cache) {
370 + if(unlikely(__atomic_load_n(&cache->dirty.stats->size, __ATOMIC_RELAXED) > __atomic_load_n(&cache->hot.stats->max_size, __ATOMIC_RELAXED))) {
371 + __atomic_add_fetch(&cache->stats.events_flush_critical, 1, __ATOMIC_RELAXED);
372 + return true;
373 + }
374 +
375 + return false;
376 +}
377 +
378 +// ----------------------------------------------------------------------------
379 +// helpers
380 +
381 +static size_t page_assumed_size(PGC *cache, size_t size) {
382 + return size + (sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page + sizeof(Word_t) * 3);
383 +}
384 +
385 +static size_t page_size_from_assumed_size(PGC *cache, size_t assumed_size) {
386 + return assumed_size - (sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page + sizeof(Word_t) * 3);
387 +}
388 +
389 +// ----------------------------------------------------------------------------
390 +// Linked list management
391 +
392 +static inline void atomic_set_max(size_t *max, size_t desired) {
393 + size_t expected;
394 +
395 + expected = __atomic_load_n(max, __ATOMIC_RELAXED);
396 +
397 + do {
398 +
399 + if(expected >= desired)
400 + return;
401 +
402 + } while(!__atomic_compare_exchange_n(max, &expected, desired,
403 + false, __ATOMIC_RELAXED, __ATOMIC_RELAXED));
404 +}
405 +
406 +struct section_pages {
407 + SPINLOCK migration_to_v2_spinlock;
408 + size_t entries;
409 + size_t size;
410 + PGC_PAGE *base;
411 +};
412 +
413 +static ARAL section_pages_aral = {
414 + .filename = NULL,
415 + .cache_dir = NULL,
416 + .use_mmap = false,
417 + .initial_elements = 16384 / sizeof(struct section_pages),
418 + .requested_element_size = sizeof(struct section_pages),
419 +};
420 +
421 +static void pgc_stats_ll_judy_change(PGC *cache, struct pgc_linked_list *ll, size_t mem_before_judyl, size_t mem_after_judyl) {
422 + if(mem_after_judyl > mem_before_judyl) {
423 + __atomic_add_fetch(&ll->stats->size, mem_after_judyl - mem_before_judyl, __ATOMIC_RELAXED);
424 + __atomic_add_fetch(&cache->stats.size, mem_after_judyl - mem_before_judyl, __ATOMIC_RELAXED);
425 + }
426 + else if(mem_after_judyl < mem_before_judyl) {
427 + __atomic_sub_fetch(&ll->stats->size, mem_before_judyl - mem_after_judyl, __ATOMIC_RELAXED);
428 + __atomic_sub_fetch(&cache->stats.size, mem_before_judyl - mem_after_judyl, __ATOMIC_RELAXED);
429 + }
430 +}
431 +
432 +static void pgc_stats_index_judy_change(PGC *cache, size_t mem_before_judyl, size_t mem_after_judyl) {
433 + if(mem_after_judyl > mem_before_judyl) {
434 + __atomic_add_fetch(&cache->stats.size, mem_after_judyl - mem_before_judyl, __ATOMIC_RELAXED);
435 + }
436 + else if(mem_after_judyl < mem_before_judyl) {
437 + __atomic_sub_fetch(&cache->stats.size, mem_before_judyl - mem_after_judyl, __ATOMIC_RELAXED);
438 + }
439 +}
440 +
441 +static void pgc_ll_add(PGC *cache __maybe_unused, struct pgc_linked_list *ll, PGC_PAGE *page, bool having_lock) {
442 + if(!having_lock)
443 + pgc_ll_lock(cache, ll);
444 +
445 + internal_fatal(page_get_status_flags(page) != 0,
446 + "DBENGINE CACHE: invalid page flags, the page has %d, but it is should be %d",
447 + page_get_status_flags(page),
448 + 0);
449 +
450 + if(ll->linked_list_in_sections_judy) {
451 + size_t mem_before_judyl, mem_after_judyl;
452 +
453 + mem_before_judyl = JudyLMemUsed(ll->sections_judy);
454 + Pvoid_t *section_pages_pptr = JudyLIns(&ll->sections_judy, page->section, PJE0);
455 + mem_after_judyl = JudyLMemUsed(ll->sections_judy);
456 +
457 + struct section_pages *sp = *section_pages_pptr;
458 + if(!sp) {
459 + // sp = callocz(1, sizeof(struct section_pages));
460 + sp = arrayalloc_mallocz(&section_pages_aral);
461 + memset(sp, 0, sizeof(struct section_pages));
462 +
463 + *section_pages_pptr = sp;
464 +
465 + mem_after_judyl += sizeof(struct section_pages);
466 + }
467 + pgc_stats_ll_judy_change(cache, ll, mem_before_judyl, mem_after_judyl);
468 +
469 + sp->entries++;
470 + sp->size += page->assumed_size;
471 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(sp->base, page, link.prev, link.next);
472 +
473 + if((sp->entries % cache->config.max_dirty_pages_per_call) == 0)
474 + ll->version++;
475 + }
476 + else {
477 + // CLEAN pages end up here.
478 + // - New pages created as CLEAN, always have 1 access.
479 + // - DIRTY pages made CLEAN, depending on their accesses may be appended (accesses > 0) or prepended (accesses = 0).
480 +
481 + if(page->accesses || page_flag_check(page, PGC_PAGE_HAS_BEEN_ACCESSED | PGC_PAGE_HAS_NO_DATA_IGNORE_ACCESSES) == PGC_PAGE_HAS_BEEN_ACCESSED) {
482 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(ll->base, page, link.prev, link.next);
483 + page_flag_clear(page, PGC_PAGE_HAS_BEEN_ACCESSED);
484 + }
485 + else
486 + DOUBLE_LINKED_LIST_PREPEND_UNSAFE(ll->base, page, link.prev, link.next);
487 +
488 + ll->version++;
489 + }
490 +
491 + page_flag_set(page, ll->flags);
492 +
493 + if(!having_lock)
494 + pgc_ll_unlock(cache, ll);
495 +
496 + size_t entries = __atomic_add_fetch(&ll->stats->entries, 1, __ATOMIC_RELAXED);
497 + size_t size = __atomic_add_fetch(&ll->stats->size, page->assumed_size, __ATOMIC_RELAXED);
498 + __atomic_add_fetch(&ll->stats->added_entries, 1, __ATOMIC_RELAXED);
499 + __atomic_add_fetch(&ll->stats->added_size, page->assumed_size, __ATOMIC_RELAXED);
500 +
501 + atomic_set_max(&ll->stats->max_entries, entries);
502 + atomic_set_max(&ll->stats->max_size, size);
503 +}
504 +
505 +static void pgc_ll_del(PGC *cache __maybe_unused, struct pgc_linked_list *ll, PGC_PAGE *page, bool having_lock) {
506 + __atomic_sub_fetch(&ll->stats->entries, 1, __ATOMIC_RELAXED);
507 + __atomic_sub_fetch(&ll->stats->size, page->assumed_size, __ATOMIC_RELAXED);
508 + __atomic_add_fetch(&ll->stats->removed_entries, 1, __ATOMIC_RELAXED);
509 + __atomic_add_fetch(&ll->stats->removed_size, page->assumed_size, __ATOMIC_RELAXED);
510 +
511 + if(!having_lock)
512 + pgc_ll_lock(cache, ll);
513 +
514 + internal_fatal(page_get_status_flags(page) != ll->flags,
515 + "DBENGINE CACHE: invalid page flags, the page has %d, but it is should be %d",
516 + page_get_status_flags(page),
517 + ll->flags);
518 +
519 + page_flag_clear(page, ll->flags);
520 +
521 + if(ll->linked_list_in_sections_judy) {
522 + Pvoid_t *section_pages_pptr = JudyLGet(ll->sections_judy, page->section, PJE0);
523 + internal_fatal(!section_pages_pptr, "DBENGINE CACHE: page should be in Judy LL, but it is not");
524 +
525 + struct section_pages *sp = *section_pages_pptr;
526 + sp->entries--;
527 + sp->size -= page->assumed_size;
528 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(sp->base, page, link.prev, link.next);
529 +
530 + if(!sp->base) {
531 + size_t mem_before_judyl, mem_after_judyl;
532 +
533 + mem_before_judyl = JudyLMemUsed(ll->sections_judy);
534 + int rc = JudyLDel(&ll->sections_judy, page->section, PJE0);
535 + mem_after_judyl = JudyLMemUsed(ll->sections_judy);
536 +
537 + if(!rc)
538 + fatal("DBENGINE CACHE: cannot delete section from Judy LL");
539 +
540 + // freez(sp);
541 + arrayalloc_freez(&section_pages_aral, sp);
542 + mem_after_judyl -= sizeof(struct section_pages);
543 + pgc_stats_ll_judy_change(cache, ll, mem_before_judyl, mem_after_judyl);
544 + }
545 + }
546 + else {
547 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(ll->base, page, link.prev, link.next);
548 + ll->version++;
549 + }
550 +
551 + if(!having_lock)
552 + pgc_ll_unlock(cache, ll);
553 +}
554 +
555 +static void page_has_been_accessed(PGC *cache, PGC_PAGE *page) {
556 + PGC_PAGE_FLAGS flags = page_flag_check(page, PGC_PAGE_CLEAN | PGC_PAGE_HAS_NO_DATA_IGNORE_ACCESSES);
557 +
558 + if (!(flags & PGC_PAGE_HAS_NO_DATA_IGNORE_ACCESSES)) {
559 + __atomic_add_fetch(&page->accesses, 1, __ATOMIC_RELAXED);
560 +
561 + if (flags & PGC_PAGE_CLEAN) {
562 + if(pgc_ll_trylock(cache, &cache->clean)) {
563 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(cache->clean.base, page, link.prev, link.next);
564 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(cache->clean.base, page, link.prev, link.next);
565 + pgc_ll_unlock(cache, &cache->clean);
566 + page_flag_clear(page, PGC_PAGE_HAS_BEEN_ACCESSED);
567 + }
568 + else
569 + page_flag_set(page, PGC_PAGE_HAS_BEEN_ACCESSED);
570 + }
571 + }
572 +}
573 +
574 +
575 +// ----------------------------------------------------------------------------
576 +// state transitions
577 +
578 +static inline void page_set_clean(PGC *cache, PGC_PAGE *page, bool having_transition_lock, bool having_clean_lock) {
579 + if(!having_transition_lock)
580 + page_transition_lock(cache, page);
581 +
582 + PGC_PAGE_FLAGS flags = page_get_status_flags(page);
583 +
584 + if(flags & PGC_PAGE_CLEAN) {
585 + if(!having_transition_lock)
586 + page_transition_unlock(cache, page);
587 + return;
588 + }
589 +
590 + if(flags & PGC_PAGE_HOT)
591 + pgc_ll_del(cache, &cache->hot, page, false);
592 +
593 + if(flags & PGC_PAGE_DIRTY)
594 + pgc_ll_del(cache, &cache->dirty, page, false);
595 +
596 + // first add to linked list, the set the flag (required for move_page_last())
597 + pgc_ll_add(cache, &cache->clean, page, having_clean_lock);
598 +
599 + if(!having_transition_lock)
600 + page_transition_unlock(cache, page);
601 +}
602 +
603 +static void page_set_dirty(PGC *cache, PGC_PAGE *page, bool having_hot_lock) {
604 + if(!having_hot_lock)
605 + // to avoid deadlocks, we have to get the hot lock before the page transition
606 + // since this is what all_hot_to_dirty() does
607 + pgc_ll_lock(cache, &cache->hot);
608 +
609 + page_transition_lock(cache, page);
610 +
611 + PGC_PAGE_FLAGS flags = page_get_status_flags(page);
612 +
613 + if(flags & PGC_PAGE_DIRTY) {
614 + page_transition_unlock(cache, page);
615 +
616 + if(!having_hot_lock)
617 + // we don't need the hot lock anymore
618 + pgc_ll_unlock(cache, &cache->hot);
619 +
620 + return;
621 + }
622 +
623 + if(likely(flags & PGC_PAGE_HOT))
624 + pgc_ll_del(cache, &cache->hot, page, true);
625 +
626 + if(!having_hot_lock)
627 + // we don't need the hot lock anymore
628 + pgc_ll_unlock(cache, &cache->hot);
629 +
630 + if(unlikely(flags & PGC_PAGE_CLEAN))
631 + pgc_ll_del(cache, &cache->clean, page, false);
632 +
633 + // first add to linked list, the set the flag (required for move_page_last())
634 + pgc_ll_add(cache, &cache->dirty, page, false);
635 +
636 + page_transition_unlock(cache, page);
637 +}
638 +
639 +static inline void page_set_hot(PGC *cache, PGC_PAGE *page) {
640 + page_transition_lock(cache, page);
641 +
642 + PGC_PAGE_FLAGS flags = page_get_status_flags(page);
643 +
644 + if(flags & PGC_PAGE_HOT) {
645 + page_transition_unlock(cache, page);
646 + return;
647 + }
648 +
649 + if(flags & PGC_PAGE_DIRTY)
650 + pgc_ll_del(cache, &cache->dirty, page, false);
651 +
652 + if(flags & PGC_PAGE_CLEAN)
653 + pgc_ll_del(cache, &cache->clean, page, false);
654 +
655 + // first add to linked list, the set the flag (required for move_page_last())
656 + pgc_ll_add(cache, &cache->hot, page, false);
657 +
658 + page_transition_unlock(cache, page);
659 +}
660 +
661 +
662 +// ----------------------------------------------------------------------------
663 +// Referencing
664 +
665 +static inline size_t PGC_REFERENCED_PAGES(PGC *cache) {
666 + return __atomic_load_n(&cache->stats.referenced_entries, __ATOMIC_RELAXED);
667 +}
668 +
669 +static inline void PGC_REFERENCED_PAGES_PLUS1(PGC *cache, PGC_PAGE *page) {
670 + __atomic_add_fetch(&cache->stats.referenced_entries, 1, __ATOMIC_RELAXED);
671 + __atomic_add_fetch(&cache->stats.referenced_size, page->assumed_size, __ATOMIC_RELAXED);
672 +}
673 +
674 +static inline void PGC_REFERENCED_PAGES_MINUS1(PGC *cache, size_t assumed_size) {
675 + __atomic_sub_fetch(&cache->stats.referenced_entries, 1, __ATOMIC_RELAXED);
676 + __atomic_sub_fetch(&cache->stats.referenced_size, assumed_size, __ATOMIC_RELAXED);
677 +}
678 +
679 +// If the page is not already acquired,
680 +// YOU HAVE TO HAVE THE QUEUE (hot, dirty, clean) THE PAGE IS IN, L O C K E D !
681 +// If you don't have it locked, NOTHING PREVENTS THIS PAGE FOR VANISHING WHILE THIS IS CALLED!
682 +static inline bool page_acquire(PGC *cache, PGC_PAGE *page) {
683 + __atomic_add_fetch(&cache->stats.acquires, 1, __ATOMIC_RELAXED);
684 +
685 + REFCOUNT expected, desired;
686 +
687 + expected = __atomic_load_n(&page->refcount, __ATOMIC_RELAXED);
688 + size_t spins = 0;
689 +
690 + do {
691 + spins++;
692 +
693 + if(unlikely(expected < 0))
694 + return false;
695 +
696 + desired = expected + 1;
697 +
698 + } while(!__atomic_compare_exchange_n(&page->refcount, &expected, desired, false, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED));
699 +
700 + if(unlikely(spins > 1))
701 + __atomic_add_fetch(&cache->stats.acquire_spins, spins - 1, __ATOMIC_RELAXED);
702 +
703 + if(desired == 1)
704 + PGC_REFERENCED_PAGES_PLUS1(cache, page);
705 +
706 + return true;
707 +}
708 +
709 +static inline void page_release(PGC *cache, PGC_PAGE *page, bool evict_if_necessary) {
710 + __atomic_add_fetch(&cache->stats.releases, 1, __ATOMIC_RELAXED);
711 +
712 + size_t assumed_size = page->assumed_size; // take the size before we release it
713 + REFCOUNT expected, desired;
714 +
715 + expected = __atomic_load_n(&page->refcount, __ATOMIC_RELAXED);
716 +
717 + size_t spins = 0;
718 + do {
719 + spins++;
720 +
721 + internal_fatal(expected <= 0,
722 + "DBENGINE CACHE: trying to release a page with reference counter %d", expected);
723 +
724 + desired = expected - 1;
725 +
726 + } while(!__atomic_compare_exchange_n(&page->refcount, &expected, desired, false, __ATOMIC_RELEASE, __ATOMIC_RELAXED));
727 +
728 + if(unlikely(spins > 1))
729 + __atomic_add_fetch(&cache->stats.release_spins, spins - 1, __ATOMIC_RELAXED);
730 +
731 + if(desired == 0) {
732 + PGC_REFERENCED_PAGES_MINUS1(cache, assumed_size);
733 +
734 + if(evict_if_necessary)
735 + evict_on_page_release_when_permitted(cache);
736 + }
737 +}
738 +
739 +static inline bool non_acquired_page_get_for_deletion___while_having_clean_locked(PGC *cache __maybe_unused, PGC_PAGE *page) {
740 + __atomic_add_fetch(&cache->stats.acquires_for_deletion, 1, __ATOMIC_RELAXED);
741 +
742 + internal_fatal(!is_page_clean(page),
743 + "DBENGINE CACHE: only clean pages can be deleted");
744 +
745 + REFCOUNT expected, desired;
746 +
747 + expected = __atomic_load_n(&page->refcount, __ATOMIC_RELAXED);
748 + size_t spins = 0;
749 + bool delete_it;
750 +
751 + do {
752 + spins++;
753 +
754 + if (expected == 0) {
755 + desired = REFCOUNT_DELETING;
756 + delete_it = true;
757 + }
758 + else {
759 + delete_it = false;
760 + break;
761 + }
762 +
763 + } while(!__atomic_compare_exchange_n(&page->refcount, &expected, desired, false, __ATOMIC_RELEASE, __ATOMIC_RELAXED));
764 +
765 + if(delete_it) {
766 + // we can delete this page
767 + internal_fatal(page_flag_check(page, PGC_PAGE_IS_BEING_DELETED),
768 + "DBENGINE CACHE: page is already being deleted");
769 +
770 + page_flag_set(page, PGC_PAGE_IS_BEING_DELETED);
771 + }
772 +
773 + if(unlikely(spins > 1))
774 + __atomic_add_fetch(&cache->stats.delete_spins, spins - 1, __ATOMIC_RELAXED);
775 +
776 + return delete_it;
777 +}
778 +
779 +static inline bool acquired_page_get_for_deletion_or_release_it(PGC *cache __maybe_unused, PGC_PAGE *page) {
780 + __atomic_add_fetch(&cache->stats.acquires_for_deletion, 1, __ATOMIC_RELAXED);
781 +
782 + size_t assumed_size = page->assumed_size; // take the size before we release it
783 +
784 + REFCOUNT expected, desired;
785 +
786 + expected = __atomic_load_n(&page->refcount, __ATOMIC_RELAXED);
787 + size_t spins = 0;
788 + bool delete_it;
789 +
790 + do {
791 + spins++;
792 +
793 + internal_fatal(expected < 1,
794 + "DBENGINE CACHE: page to be deleted should be acquired by the caller.");
795 +
796 + if (expected == 1) {
797 + // we are the only one having this page referenced
798 + desired = REFCOUNT_DELETING;
799 + delete_it = true;
800 + }
801 + else {
802 + // this page cannot be deleted
803 + desired = expected - 1;
804 + delete_it = false;
805 + }
806 +
807 + } while(!__atomic_compare_exchange_n(&page->refcount, &expected, desired, false, __ATOMIC_RELEASE, __ATOMIC_RELAXED));
808 +
809 + if(delete_it) {
810 + PGC_REFERENCED_PAGES_MINUS1(cache, assumed_size);
811 +
812 + // we can delete this page
813 + internal_fatal(page_flag_check(page, PGC_PAGE_IS_BEING_DELETED),
814 + "DBENGINE CACHE: page is already being deleted");
815 +
816 + page_flag_set(page, PGC_PAGE_IS_BEING_DELETED);
817 + }
818 +
819 + if(unlikely(spins > 1))
820 + __atomic_add_fetch(&cache->stats.delete_spins, spins - 1, __ATOMIC_RELAXED);
821 +
822 + return delete_it;
823 +}
824 +
825 +
826 +// ----------------------------------------------------------------------------
827 +// Indexing
828 +
829 +static void free_this_page(PGC *cache, PGC_PAGE *page) {
830 + // call the callback to free the user supplied memory
831 + cache->config.pgc_free_clean_cb(cache, (PGC_ENTRY){
832 + .section = page->section,
833 + .metric_id = page->metric_id,
834 + .start_time_s = page->start_time_s,
835 + .end_time_s = __atomic_load_n(&page->end_time_s, __ATOMIC_RELAXED),
836 + .update_every_s = page->update_every_s,
837 + .size = page_size_from_assumed_size(cache, page->assumed_size),
838 + .hot = (is_page_hot(page)) ? true : false,
839 + .data = page->data,
840 + .custom_data = (cache->config.additional_bytes_per_page) ? page->custom_data : NULL,
841 + });
842 +
843 + // update statistics
844 + __atomic_add_fetch(&cache->stats.removed_entries, 1, __ATOMIC_RELAXED);
845 + __atomic_add_fetch(&cache->stats.removed_size, page->assumed_size, __ATOMIC_RELAXED);
846 +
847 + __atomic_sub_fetch(&cache->stats.entries, 1, __ATOMIC_RELAXED);
848 + __atomic_sub_fetch(&cache->stats.size, page->assumed_size, __ATOMIC_RELAXED);
849 +
850 + // free our memory
851 +#ifdef PGC_WITH_ARAL
852 + arrayalloc_freez(cache->aral, page);
853 +#else
854 + freez(page);
855 +#endif
856 +}
857 +
858 +static void remove_this_page_from_index_unsafe(PGC *cache, PGC_PAGE *page, size_t partition) {
859 + // remove it from the Judy arrays
860 +
861 + pointer_check(cache, page);
862 +
863 + internal_fatal(page_flag_check(page, PGC_PAGE_HOT | PGC_PAGE_DIRTY | PGC_PAGE_CLEAN),
864 + "DBENGINE CACHE: page to be removed from the cache is still in the linked-list");
865 +
866 + internal_fatal(!page_flag_check(page, PGC_PAGE_IS_BEING_DELETED),
867 + "DBENGINE CACHE: page to be removed from the index, is not marked for deletion");
868 +
869 + internal_fatal(partition != pgc_indexing_partition(cache, page->metric_id),
870 + "DBENGINE CACHE: attempted to remove this page from the wrong partition of the cache");
871 +
872 + Pvoid_t *metrics_judy_pptr = JudyLGet(cache->index[partition].sections_judy, page->section, PJE0);
873 + if(unlikely(!metrics_judy_pptr))
874 + fatal("DBENGINE CACHE: section '%lu' should exist, but it does not.", page->section);
875 +
876 + Pvoid_t *pages_judy_pptr = JudyLGet(*metrics_judy_pptr, page->metric_id, PJE0);
877 + if(unlikely(!pages_judy_pptr))
878 + fatal("DBENGINE CACHE: metric '%lu' in section '%lu' should exist, but it does not.",
879 + page->metric_id, page->section);
880 +
881 + Pvoid_t *page_ptr = JudyLGet(*pages_judy_pptr, page->start_time_s, PJE0);
882 + if(unlikely(!page_ptr))
883 + fatal("DBENGINE CACHE: page with start time '%ld' of metric '%lu' in section '%lu' should exist, but it does not.",
884 + page->start_time_s, page->metric_id, page->section);
885 +
886 + PGC_PAGE *found_page = *page_ptr;
887 + if(unlikely(found_page != page))
888 + fatal("DBENGINE CACHE: page with start time '%ld' of metric '%lu' in section '%lu' should exist, but the index returned a different address.",
889 + page->start_time_s, page->metric_id, page->section);
890 +
891 + size_t mem_before_judyl = 0, mem_after_judyl = 0;
892 +
893 + mem_before_judyl += JudyLMemUsed(*pages_judy_pptr);
894 + if(unlikely(!JudyLDel(pages_judy_pptr, page->start_time_s, PJE0)))
895 + fatal("DBENGINE CACHE: page with start time '%ld' of metric '%lu' in section '%lu' exists, but cannot be deleted.",
896 + page->start_time_s, page->metric_id, page->section);
897 + mem_after_judyl += JudyLMemUsed(*pages_judy_pptr);
898 +
899 + mem_before_judyl += JudyLMemUsed(*metrics_judy_pptr);
900 + if(!*pages_judy_pptr && !JudyLDel(metrics_judy_pptr, page->metric_id, PJE0))
901 + fatal("DBENGINE CACHE: metric '%lu' in section '%lu' exists and is empty, but cannot be deleted.",
902 + page->metric_id, page->section);
903 + mem_after_judyl += JudyLMemUsed(*metrics_judy_pptr);
904 +
905 + mem_before_judyl += JudyLMemUsed(cache->index[partition].sections_judy);
906 + if(!*metrics_judy_pptr && !JudyLDel(&cache->index[partition].sections_judy, page->section, PJE0))
907 + fatal("DBENGINE CACHE: section '%lu' exists and is empty, but cannot be deleted.", page->section);
908 + mem_after_judyl += JudyLMemUsed(cache->index[partition].sections_judy);
909 +
910 + pgc_stats_index_judy_change(cache, mem_before_judyl, mem_after_judyl);
911 +
912 + pointer_del(cache, page);
913 +}
914 +
915 +static void remove_and_free_page_not_in_any_queue_and_acquired_for_deletion(PGC *cache, PGC_PAGE *page) {
916 + size_t partition = pgc_indexing_partition(cache, page->metric_id);
917 + pgc_index_write_lock(cache, partition);
918 + remove_this_page_from_index_unsafe(cache, page, partition);
919 + pgc_index_write_unlock(cache, partition);
920 + free_this_page(cache, page);
921 +}
922 +
923 +static bool make_acquired_page_clean_and_evict_or_page_release(PGC *cache, PGC_PAGE *page) {
924 + pointer_check(cache, page);
925 +
926 + page_transition_lock(cache, page);
927 + pgc_ll_lock(cache, &cache->clean);
928 +
929 + // make it clean - it does not have any accesses, so it will be prepended
930 + page_set_clean(cache, page, true, true);
931 +
932 + if(!acquired_page_get_for_deletion_or_release_it(cache, page)) {
933 + pgc_ll_unlock(cache, &cache->clean);
934 + page_transition_unlock(cache, page);
935 + return false;
936 + }
937 +
938 + // remove it from the linked list
939 + pgc_ll_del(cache, &cache->clean, page, true);
940 + pgc_ll_unlock(cache, &cache->clean);
941 + page_transition_unlock(cache, page);
942 +
943 + remove_and_free_page_not_in_any_queue_and_acquired_for_deletion(cache, page);
944 +
945 + return true;
946 +}
947 +
948 +// returns true, when there is more work to do
949 +static bool evict_pages_with_filter(PGC *cache, size_t max_skip, size_t max_evict, bool wait, bool all_of_them, evict_filter filter, void *data) {
950 + if(!all_of_them && !cache_above_healthy_limit(cache))
951 + // don't bother - not enough to do anything
952 + return false;
953 +
954 + size_t workers_running = __atomic_add_fetch(&cache->stats.workers_evict, 1, __ATOMIC_RELAXED);
955 + if(!wait && !all_of_them && workers_running > cache->config.max_workers_evict_inline) {
956 + __atomic_sub_fetch(&cache->stats.workers_evict, 1, __ATOMIC_RELAXED);
957 + return false;
958 + }
959 +
960 + internal_fatal(cache->clean.linked_list_in_sections_judy,
961 + "wrong clean pages configuration - clean pages need to have a linked list, not a judy array");
962 +
963 + if(unlikely(!max_skip))
964 + max_skip = SIZE_MAX;
965 + else if(unlikely(max_skip < 2))
966 + max_skip = 2;
967 +
968 + if(unlikely(!max_evict))
969 + max_evict = SIZE_MAX;
970 + else if(unlikely(max_evict < 2))
971 + max_evict = 2;
972 +
973 + PGC_PAGE *pages_to_evict = NULL;
974 + size_t total_pages_evicted = 0;
975 + size_t total_pages_skipped = 0;
976 + bool stopped_before_finishing = false;
977 + size_t spins = 0;
978 +
979 + do {
980 + if(++spins > 1)
981 + __atomic_add_fetch(&cache->stats.evict_spins, 1, __ATOMIC_RELAXED);
982 +
983 + size_t max_size_to_evict = 0;
984 + if (all_of_them)
985 + max_size_to_evict = SIZE_MAX;
986 +
987 + else
988 + cache_usage_per1000(cache, &max_size_to_evict);
989 +
990 + if (!max_size_to_evict)
991 + break;
992 +
993 + // check if we have to stop
994 + if(total_pages_evicted >= max_evict && !all_of_them) {
995 + stopped_before_finishing = true;
996 + break;
997 + }
998 +
999 + if(!all_of_them && !wait) {
1000 + if(!pgc_ll_trylock(cache, &cache->clean)) {
1001 + stopped_before_finishing = true;
1002 + goto premature_exit;
1003 + }
1004 +
1005 + // at this point we have the clean lock
1006 + }
1007 + else
1008 + pgc_ll_lock(cache, &cache->clean);
1009 +
1010 + // find a page to evict
1011 + pages_to_evict = NULL;
1012 + for(PGC_PAGE *page = cache->clean.base, *next = NULL, *first_page_we_relocated = NULL; page ; page = next) {
1013 + next = page->link.next;
1014 +
1015 + if(unlikely(page == first_page_we_relocated))
1016 + // we did a complete loop on all pages
1017 + break;
1018 +
1019 + if(unlikely(page_flag_check(page, PGC_PAGE_HAS_BEEN_ACCESSED | PGC_PAGE_HAS_NO_DATA_IGNORE_ACCESSES) == PGC_PAGE_HAS_BEEN_ACCESSED)) {
1020 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(cache->clean.base, page, link.prev, link.next);
1021 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(cache->clean.base, page, link.prev, link.next);
1022 + page_flag_clear(page, PGC_PAGE_HAS_BEEN_ACCESSED);
1023 + continue;
1024 + }
1025 +
1026 + if(unlikely(filter && !filter(page, data)))
1027 + continue;
1028 +
1029 + if(non_acquired_page_get_for_deletion___while_having_clean_locked(cache, page)) {
1030 + // we can delete this page
1031 +
1032 + // remove it from the clean list
1033 + pgc_ll_del(cache, &cache->clean, page, true);
1034 +
1035 + __atomic_add_fetch(&cache->stats.evicting_entries, 1, __ATOMIC_RELAXED);
1036 + __atomic_add_fetch(&cache->stats.evicting_size, page->assumed_size, __ATOMIC_RELAXED);
1037 +
1038 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(pages_to_evict, page, link.prev, link.next);
1039 +
1040 + if(!all_of_them)
1041 + // we do it one-by-one to avoid locking clean queue for way too long
1042 + break;
1043 + }
1044 + else {
1045 + // we can't delete this page
1046 +
1047 + if(!first_page_we_relocated)
1048 + first_page_we_relocated = page;
1049 +
1050 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(cache->clean.base, page, link.prev, link.next);
1051 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(cache->clean.base, page, link.prev, link.next);
1052 +
1053 + // check if we have to stop
1054 + if(unlikely(++total_pages_skipped >= max_skip && !all_of_them)) {
1055 + stopped_before_finishing = true;
1056 + break;
1057 + }
1058 + }
1059 + }
1060 + pgc_ll_unlock(cache, &cache->clean);
1061 +
1062 + if(likely(pages_to_evict)) {
1063 + // remove them from the index
1064 +
1065 + for(PGC_PAGE *page = pages_to_evict, *next = NULL; page ; page = next) {
1066 + next = page->link.next;
1067 +
1068 + size_t page_size = page->assumed_size;
1069 +
1070 + size_t partition = pgc_indexing_partition(cache, page->metric_id);
1071 + pgc_index_write_lock(cache, partition);
1072 + remove_this_page_from_index_unsafe(cache, page, partition);
1073 + pgc_index_write_unlock(cache, partition);
1074 + free_this_page(cache, page);
1075 +
1076 + __atomic_sub_fetch(&cache->stats.evicting_entries, 1, __ATOMIC_RELAXED);
1077 + __atomic_sub_fetch(&cache->stats.evicting_size, page_size, __ATOMIC_RELAXED);
1078 +
1079 + total_pages_evicted++;
1080 + }
1081 + }
1082 + else
1083 + break;
1084 +
1085 + } while(all_of_them || (total_pages_evicted < max_evict && total_pages_skipped < max_skip));
1086 +
1087 + if(all_of_them && !filter) {
1088 + pgc_ll_lock(cache, &cache->clean);
1089 + if(cache->clean.stats->entries) {
1090 + error_limit_static_global_var(erl, 1, 0);
1091 + error_limit(&erl, "DBENGINE CACHE: cannot free all clean pages, %zu are still in the clean queue",
1092 + cache->clean.stats->entries);
1093 + }
1094 + pgc_ll_unlock(cache, &cache->clean);
1095 + }
1096 +
1097 +premature_exit:
1098 + if(unlikely(total_pages_skipped))
1099 + __atomic_add_fetch(&cache->stats.evict_skipped, total_pages_skipped, __ATOMIC_RELAXED);
1100 +
1101 + __atomic_sub_fetch(&cache->stats.workers_evict, 1, __ATOMIC_RELAXED);
1102 +
1103 + return stopped_before_finishing;
1104 +}
1105 +
1106 +static PGC_PAGE *page_add(PGC *cache, PGC_ENTRY *entry, bool *added) {
1107 + __atomic_add_fetch(&cache->stats.workers_add, 1, __ATOMIC_RELAXED);
1108 +
1109 +#ifdef PGC_WITH_ARAL
1110 + PGC_PAGE *allocation = arrayalloc_mallocz(cache->aral);
1111 +#endif
1112 + PGC_PAGE *page;
1113 + size_t spins = 0;
1114 +
1115 + do {
1116 + if(++spins > 1)
1117 + __atomic_add_fetch(&cache->stats.insert_spins, 1, __ATOMIC_RELAXED);
1118 +
1119 + size_t partition = pgc_indexing_partition(cache, entry->metric_id);
1120 + pgc_index_write_lock(cache, partition);
1121 +
1122 + size_t mem_before_judyl = 0, mem_after_judyl = 0;
1123 +
1124 + mem_before_judyl += JudyLMemUsed(cache->index[partition].sections_judy);
1125 + Pvoid_t *metrics_judy_pptr = JudyLIns(&cache->index[partition].sections_judy, entry->section, PJE0);
1126 + if(unlikely(!metrics_judy_pptr || metrics_judy_pptr == PJERR))
1127 + fatal("DBENGINE CACHE: corrupted sections judy array");
1128 + mem_after_judyl += JudyLMemUsed(cache->index[partition].sections_judy);
1129 +
1130 + mem_before_judyl += JudyLMemUsed(*metrics_judy_pptr);
1131 + Pvoid_t *pages_judy_pptr = JudyLIns(metrics_judy_pptr, entry->metric_id, PJE0);
1132 + if(unlikely(!pages_judy_pptr || pages_judy_pptr == PJERR))
1133 + fatal("DBENGINE CACHE: corrupted pages judy array");
1134 + mem_after_judyl += JudyLMemUsed(*metrics_judy_pptr);
1135 +
1136 + mem_before_judyl += JudyLMemUsed(*pages_judy_pptr);
1137 + Pvoid_t *page_ptr = JudyLIns(pages_judy_pptr, entry->start_time_s, PJE0);
1138 + if(unlikely(!page_ptr || page_ptr == PJERR))
1139 + fatal("DBENGINE CACHE: corrupted page in judy array");
1140 + mem_after_judyl += JudyLMemUsed(*pages_judy_pptr);
1141 +
1142 + pgc_stats_index_judy_change(cache, mem_before_judyl, mem_after_judyl);
1143 +
1144 + page = *page_ptr;
1145 +
1146 + if (likely(!page)) {
1147 +#ifdef PGC_WITH_ARAL
1148 + page = allocation;
1149 + allocation = NULL;
1150 +#else
1151 + page = mallocz(sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page);
1152 +#endif
1153 + page->refcount = 1;
1154 + page->accesses = (entry->hot) ? 0 : 1;
1155 + page->flags = 0;
1156 + page->section = entry->section;
1157 + page->metric_id = entry->metric_id;
1158 + page->start_time_s = entry->start_time_s;
1159 + page->end_time_s = entry->end_time_s,
1160 + page->update_every_s = entry->update_every_s,
1161 + page->data = entry->data;
1162 + page->assumed_size = page_assumed_size(cache, entry->size);
1163 + netdata_spinlock_init(&page->transition_spinlock);
1164 + page->link.prev = NULL;
1165 + page->link.next = NULL;
1166 +
1167 + if(cache->config.additional_bytes_per_page) {
1168 + if(entry->custom_data)
1169 + memcpy(page->custom_data, entry->custom_data, cache->config.additional_bytes_per_page);
1170 + else
1171 + memset(page->custom_data, 0, cache->config.additional_bytes_per_page);
1172 + }
1173 +
1174 + // put it in the index
1175 + *page_ptr = page;
1176 + pointer_add(cache, page);
1177 + pgc_index_write_unlock(cache, partition);
1178 +
1179 + if (entry->hot)
1180 + page_set_hot(cache, page);
1181 + else
1182 + page_set_clean(cache, page, false, false);
1183 +
1184 + PGC_REFERENCED_PAGES_PLUS1(cache, page);
1185 +
1186 + // update statistics
1187 + __atomic_add_fetch(&cache->stats.added_entries, 1, __ATOMIC_RELAXED);
1188 + __atomic_add_fetch(&cache->stats.added_size, page->assumed_size, __ATOMIC_RELAXED);
1189 +
1190 + __atomic_add_fetch(&cache->stats.entries, 1, __ATOMIC_RELAXED);
1191 + __atomic_add_fetch(&cache->stats.size, page->assumed_size, __ATOMIC_RELAXED);
1192 +
1193 + if(added)
1194 + *added = true;
1195 + }
1196 + else {
1197 + if (!page_acquire(cache, page))
1198 + page = NULL;
1199 +
1200 + else if(added)
1201 + *added = false;
1202 +
1203 + pgc_index_write_unlock(cache, partition);
1204 +
1205 + if(unlikely(!page)) {
1206 + // now that we don't have the lock,
1207 + // give it some time for the old page to go away
1208 + struct timespec ns = { .tv_sec = 0, .tv_nsec = 1 };
1209 + nanosleep(&ns, NULL);
1210 + }
1211 + }
1212 +
1213 + } while(!page);
1214 +
1215 +#ifdef PGC_WITH_ARAL
1216 + if(allocation)
1217 + arrayalloc_freez(cache->aral, allocation);
1218 +#endif
1219 +
1220 + __atomic_sub_fetch(&cache->stats.workers_add, 1, __ATOMIC_RELAXED);
1221 +
1222 + if(!entry->hot)
1223 + evict_on_clean_page_added(cache);
1224 +
1225 + if((cache->config.options & PGC_OPTIONS_FLUSH_PAGES_INLINE) || flushing_critical(cache)) {
1226 + flush_pages(cache, cache->config.max_flushes_inline, PGC_SECTION_ALL,
1227 + false, false);
1228 + }
1229 +
1230 + return page;
1231 +}
1232 +
1233 +static PGC_PAGE *page_find_and_acquire(PGC *cache, Word_t section, Word_t metric_id, time_t start_time_s, PGC_SEARCH method) {
1234 + __atomic_add_fetch(&cache->stats.workers_search, 1, __ATOMIC_RELAXED);
1235 +
1236 + size_t *stats_hit_ptr, *stats_miss_ptr;
1237 +
1238 + if(method == PGC_SEARCH_CLOSEST) {
1239 + __atomic_add_fetch(&cache->stats.searches_closest, 1, __ATOMIC_RELAXED);
1240 + stats_hit_ptr = &cache->stats.searches_closest_hits;
1241 + stats_miss_ptr = &cache->stats.searches_closest_misses;
1242 + }
1243 + else {
1244 + __atomic_add_fetch(&cache->stats.searches_exact, 1, __ATOMIC_RELAXED);
1245 + stats_hit_ptr = &cache->stats.searches_exact_hits;
1246 + stats_miss_ptr = &cache->stats.searches_exact_misses;
1247 + }
1248 +
1249 + PGC_PAGE *page = NULL;
1250 + size_t partition = pgc_indexing_partition(cache, metric_id);
1251 +
1252 + pgc_index_read_lock(cache, partition);
1253 +
1254 + Pvoid_t *metrics_judy_pptr = JudyLGet(cache->index[partition].sections_judy, section, PJE0);
1255 + if(unlikely(metrics_judy_pptr == PJERR))
1256 + fatal("DBENGINE CACHE: corrupted sections judy array");
1257 +
1258 + if(unlikely(!metrics_judy_pptr)) {
1259 + // section does not exist
1260 + goto cleanup;
1261 + }
1262 +
1263 + Pvoid_t *pages_judy_pptr = JudyLGet(*metrics_judy_pptr, metric_id, PJE0);
1264 + if(unlikely(pages_judy_pptr == PJERR))
1265 + fatal("DBENGINE CACHE: corrupted pages judy array");
1266 +
1267 + if(unlikely(!pages_judy_pptr)) {
1268 + // metric does not exist
1269 + goto cleanup;
1270 + }
1271 +
1272 + switch(method) {
1273 + default:
1274 + case PGC_SEARCH_CLOSEST: {
1275 + Pvoid_t *page_ptr = JudyLGet(*pages_judy_pptr, start_time_s, PJE0);
1276 + if (unlikely(page_ptr == PJERR))
1277 + fatal("DBENGINE CACHE: corrupted page in pages judy array");
1278 +
1279 + if (page_ptr)
1280 + page = *page_ptr;
1281 +
1282 + else {
1283 + Word_t time = start_time_s;
1284 +
1285 + // find the previous page
1286 + page_ptr = JudyLLast(*pages_judy_pptr, &time, PJE0);
1287 + if(unlikely(page_ptr == PJERR))
1288 + fatal("DBENGINE CACHE: corrupted page in pages judy array #2");
1289 +
1290 + if(page_ptr) {
1291 + // found a page starting before our timestamp
1292 + // check if our timestamp is included
1293 + page = *page_ptr;
1294 + if(start_time_s > page->end_time_s)
1295 + // it is not good for us
1296 + page = NULL;
1297 + }
1298 +
1299 + if(!page) {
1300 + // find the next page then...
1301 + time = start_time_s;
1302 + page_ptr = JudyLNext(*pages_judy_pptr, &time, PJE0);
1303 + if(page_ptr)
1304 + page = *page_ptr;
1305 + }
1306 + }
1307 + }
1308 + break;
1309 +
1310 + case PGC_SEARCH_EXACT: {
1311 + Pvoid_t *page_ptr = JudyLGet(*pages_judy_pptr, start_time_s, PJE0);
1312 + if (unlikely(page_ptr == PJERR))
1313 + fatal("DBENGINE CACHE: corrupted page in pages judy array");
1314 +
1315 + if (page_ptr)
1316 + page = *page_ptr;
1317 + }
1318 + break;
1319 +
1320 + case PGC_SEARCH_FIRST: {
1321 + Word_t time = start_time_s;
1322 + Pvoid_t *page_ptr = JudyLFirst(*pages_judy_pptr, &time, PJE0);
1323 + if (unlikely(page_ptr == PJERR))
1324 + fatal("DBENGINE CACHE: corrupted page in pages judy array");
1325 +
1326 + if (page_ptr)
1327 + page = *page_ptr;
1328 + }
1329 + break;
1330 +
1331 + case PGC_SEARCH_NEXT: {
1332 + Word_t time = start_time_s;
1333 + Pvoid_t *page_ptr = JudyLNext(*pages_judy_pptr, &time, PJE0);
1334 + if (unlikely(page_ptr == PJERR))
1335 + fatal("DBENGINE CACHE: corrupted page in pages judy array");
1336 +
1337 + if (page_ptr)
1338 + page = *page_ptr;
1339 + }
1340 + break;
1341 +
1342 + case PGC_SEARCH_LAST: {
1343 + Word_t time = start_time_s;
1344 + Pvoid_t *page_ptr = JudyLLast(*pages_judy_pptr, &time, PJE0);
1345 + if (unlikely(page_ptr == PJERR))
1346 + fatal("DBENGINE CACHE: corrupted page in pages judy array");
1347 +
1348 + if (page_ptr)
1349 + page = *page_ptr;
1350 + }
1351 + break;
1352 +
1353 + case PGC_SEARCH_PREV: {
1354 + Word_t time = start_time_s;
1355 + Pvoid_t *page_ptr = JudyLPrev(*pages_judy_pptr, &time, PJE0);
1356 + if (unlikely(page_ptr == PJERR))
1357 + fatal("DBENGINE CACHE: corrupted page in pages judy array");
1358 +
1359 + if (page_ptr)
1360 + page = *page_ptr;
1361 + }
1362 + break;
1363 + }
1364 +
1365 + if(page) {
1366 + pointer_check(cache, page);
1367 +
1368 + if(!page_acquire(cache, page)) {
1369 + // this page is not good to use
1370 + page = NULL;
1371 + }
1372 + }
1373 +
1374 +cleanup:
1375 + pgc_index_read_unlock(cache, partition);
1376 +
1377 + if(page) {
1378 + __atomic_add_fetch(stats_hit_ptr, 1, __ATOMIC_RELAXED);
1379 + page_has_been_accessed(cache, page);
1380 + }
1381 + else
1382 + __atomic_add_fetch(stats_miss_ptr, 1, __ATOMIC_RELAXED);
1383 +
1384 + __atomic_sub_fetch(&cache->stats.workers_search, 1, __ATOMIC_RELAXED);
1385 +
1386 + return page;
1387 +}
1388 +
1389 +static void all_hot_pages_to_dirty(PGC *cache, Word_t section) {
1390 + pgc_ll_lock(cache, &cache->hot);
1391 +
1392 + bool first = true;
1393 + Word_t last_section = (section == PGC_SECTION_ALL) ? 0 : section;
1394 + Pvoid_t *section_pages_pptr;
1395 + while ((section_pages_pptr = JudyLFirstThenNext(cache->hot.sections_judy, &last_section, &first))) {
1396 + if(section != PGC_SECTION_ALL && last_section != section)
1397 + break;
1398 +
1399 + struct section_pages *sp = *section_pages_pptr;
1400 +
1401 + PGC_PAGE *page = sp->base;
1402 + while(page) {
1403 + PGC_PAGE *next = page->link.next;
1404 +
1405 + if(page_acquire(cache, page)) {
1406 + page_set_dirty(cache, page, true);
1407 + page_release(cache, page, false);
1408 + // page ptr may be invalid now
1409 + }
1410 +
1411 + page = next;
1412 + }
1413 + }
1414 + pgc_ll_unlock(cache, &cache->hot);
1415 +}
1416 +
1417 +// returns true when there is more work to do
1418 +static bool flush_pages(PGC *cache, size_t max_flushes, Word_t section, bool wait, bool all_of_them) {
1419 + internal_fatal(!cache->dirty.linked_list_in_sections_judy,
1420 + "wrong dirty pages configuration - dirty pages need to have a judy array, not a linked list");
1421 +
1422 + if(!all_of_them && !wait) {
1423 + // we have been called from a data collection thread
1424 + // let's not waste its time...
1425 +
1426 + if(!pgc_ll_trylock(cache, &cache->dirty)) {
1427 + // we would block, so give up...
1428 + return true;
1429 + }
1430 +
1431 + // we got the lock at this point
1432 + }
1433 + else
1434 + pgc_ll_lock(cache, &cache->dirty);
1435 +
1436 + size_t optimal_flush_size = cache->config.max_dirty_pages_per_call;
1437 + size_t dirty_version_at_entry = cache->dirty.version;
1438 + if(!all_of_them && (cache->dirty.stats->entries < optimal_flush_size || cache->dirty.last_version_checked == dirty_version_at_entry)) {
1439 + pgc_ll_unlock(cache, &cache->dirty);
1440 + return false;
1441 + }
1442 +
1443 + __atomic_add_fetch(&cache->stats.workers_flush, 1, __ATOMIC_RELAXED);
1444 +
1445 + bool have_dirty_lock = true;
1446 +
1447 + if(all_of_them || !max_flushes)
1448 + max_flushes = SIZE_MAX;
1449 +
1450 + Word_t last_section = (section == PGC_SECTION_ALL) ? 0 : section;
1451 + size_t flushes_so_far = 0;
1452 + Pvoid_t *section_pages_pptr;
1453 + bool stopped_before_finishing = false;
1454 + size_t spins = 0;
1455 + bool first = true;
1456 +
1457 + while (have_dirty_lock && (section_pages_pptr = JudyLFirstThenNext(cache->dirty.sections_judy, &last_section, &first))) {
1458 + if(section != PGC_SECTION_ALL && last_section != section)
1459 + break;
1460 +
1461 + struct section_pages *sp = *section_pages_pptr;
1462 + if(!all_of_them && sp->entries < optimal_flush_size)
1463 + continue;
1464 +
1465 + if(!all_of_them && flushes_so_far > max_flushes) {
1466 + stopped_before_finishing = true;
1467 + break;
1468 + }
1469 +
1470 + if(++spins > 1)
1471 + __atomic_add_fetch(&cache->stats.flush_spins, 1, __ATOMIC_RELAXED);
1472 +
1473 + PGC_ENTRY array[optimal_flush_size];
1474 + PGC_PAGE *pages[optimal_flush_size];
1475 + size_t pages_added = 0, pages_added_size = 0;
1476 + size_t pages_removed_dirty = 0, pages_removed_dirty_size = 0;
1477 + size_t pages_cancelled = 0, pages_cancelled_size = 0;
1478 + size_t pages_made_clean = 0, pages_made_clean_size = 0;
1479 +
1480 + PGC_PAGE *page = sp->base;
1481 + while (page && pages_added < optimal_flush_size) {
1482 + PGC_PAGE *next = page->link.next;
1483 +
1484 + internal_fatal(page_get_status_flags(page) != PGC_PAGE_DIRTY,
1485 + "DBENGINE CACHE: page should be in the dirty list before saved");
1486 +
1487 + if (page_acquire(cache, page)) {
1488 + internal_fatal(page_get_status_flags(page) != PGC_PAGE_DIRTY,
1489 + "DBENGINE CACHE: page should be in the dirty list before saved");
1490 +
1491 + internal_fatal(page->section != last_section,
1492 + "DBENGINE CACHE: dirty page is not in the right section (tier)");
1493 +
1494 + if(!page_transition_trylock(cache, page)) {
1495 + page_release(cache, page, false);
1496 + // page ptr may be invalid now
1497 + }
1498 + else {
1499 + pages[pages_added] = page;
1500 + array[pages_added] = (PGC_ENTRY) {
1501 + .section = page->section,
1502 + .metric_id = page->metric_id,
1503 + .start_time_s = page->start_time_s,
1504 + .end_time_s = __atomic_load_n(&page->end_time_s, __ATOMIC_RELAXED),
1505 + .update_every_s = page->update_every_s,
1506 + .size = page_size_from_assumed_size(cache, page->assumed_size),
1507 + .data = page->data,
1508 + .custom_data = (cache->config.additional_bytes_per_page) ? page->custom_data : NULL,
1509 + .hot = false,
1510 + };
1511 +
1512 + pages_added_size += page->assumed_size;
1513 + pages_added++;
1514 + }
1515 + }
1516 +
1517 + page = next;
1518 + }
1519 +
1520 + // do we have enough to save?
1521 + if(all_of_them || pages_added == optimal_flush_size) {
1522 + // we should do it
1523 +
1524 + for (size_t i = 0; i < pages_added; i++) {
1525 + PGC_PAGE *tpg = pages[i];
1526 +
1527 + internal_fatal(page_get_status_flags(tpg) != PGC_PAGE_DIRTY,
1528 + "DBENGINE CACHE: page should be in the dirty list before saved");
1529 +
1530 + // remove it from the dirty list
1531 + pgc_ll_del(cache, &cache->dirty, tpg, true);
1532 +
1533 + __atomic_add_fetch(&cache->stats.flushing_entries, 1, __ATOMIC_RELAXED);
1534 + __atomic_add_fetch(&cache->stats.flushing_size, tpg->assumed_size, __ATOMIC_RELAXED);
1535 +
1536 + pages_removed_dirty_size += tpg->assumed_size;
1537 + pages_removed_dirty++;
1538 + }
1539 +
1540 + // next time, repeat the same section (tier)
1541 + first = true;
1542 + }
1543 + else {
1544 + // we can't do it
1545 +
1546 + for (size_t i = 0; i < pages_added; i++) {
1547 + PGC_PAGE *tpg = pages[i];
1548 +
1549 + internal_fatal(page_get_status_flags(tpg) != PGC_PAGE_DIRTY,
1550 + "DBENGINE CACHE: page should be in the dirty list before saved");
1551 +
1552 + pages_cancelled_size += tpg->assumed_size;
1553 + pages_cancelled++;
1554 +
1555 + page_transition_unlock(cache, tpg);
1556 + page_release(cache, tpg, false);
1557 + // page ptr may be invalid now
1558 + }
1559 +
1560 + __atomic_add_fetch(&cache->stats.flushes_cancelled, pages_cancelled, __ATOMIC_RELAXED);
1561 + __atomic_add_fetch(&cache->stats.flushes_cancelled_size, pages_cancelled_size, __ATOMIC_RELAXED);
1562 +
1563 + internal_fatal(pages_added != pages_cancelled || pages_added_size != pages_cancelled_size,
1564 + "DBENGINE CACHE: flushing cancel pages mismatch");
1565 +
1566 + // next time, continue to the next section (tier)
1567 + first = false;
1568 + continue;
1569 + }
1570 +
1571 + pgc_ll_unlock(cache, &cache->dirty);
1572 + have_dirty_lock = false;
1573 +
1574 + // call the callback to save them
1575 + // it may take some time, so let's release the lock
1576 + cache->config.pgc_save_dirty_cb(cache, array, pages, pages_added);
1577 + flushes_so_far++;
1578 +
1579 + __atomic_add_fetch(&cache->stats.flushes_completed, pages_added, __ATOMIC_RELAXED);
1580 + __atomic_add_fetch(&cache->stats.flushes_completed_size, pages_added_size, __ATOMIC_RELAXED);
1581 +
1582 + size_t pages_to_evict = 0; (void)pages_to_evict;
1583 + for (size_t i = 0; i < pages_added; i++) {
1584 + PGC_PAGE *tpg = pages[i];
1585 +
1586 + internal_fatal(page_get_status_flags(tpg) != 0,
1587 + "DBENGINE CACHE: page should not be in any list while it is being saved");
1588 +
1589 + __atomic_sub_fetch(&cache->stats.flushing_entries, 1, __ATOMIC_RELAXED);
1590 + __atomic_sub_fetch(&cache->stats.flushing_size, tpg->assumed_size, __ATOMIC_RELAXED);
1591 +
1592 + pages_made_clean_size += tpg->assumed_size;
1593 + pages_made_clean++;
1594 +
1595 + if(!tpg->accesses)
1596 + pages_to_evict++;
1597 +
1598 + page_set_clean(cache, tpg, true, false);
1599 + page_transition_unlock(cache, tpg);
1600 + page_release(cache, tpg, false);
1601 + // tpg ptr may be invalid now
1602 + }
1603 +
1604 + internal_fatal(pages_added != pages_made_clean || pages_added != pages_removed_dirty ||
1605 + pages_added_size != pages_made_clean_size || pages_added_size != pages_removed_dirty_size
1606 + , "DBENGINE CACHE: flushing pages mismatch");
1607 +
1608 + if(!all_of_them && !wait) {
1609 + if(pgc_ll_trylock(cache, &cache->dirty))
1610 + have_dirty_lock = true;
1611 +
1612 + else {
1613 + stopped_before_finishing = true;
1614 + have_dirty_lock = false;
1615 + }
1616 + }
1617 + else {
1618 + pgc_ll_lock(cache, &cache->dirty);
1619 + have_dirty_lock = true;
1620 + }
1621 + }
1622 +
1623 + if(have_dirty_lock) {
1624 + if(!stopped_before_finishing && dirty_version_at_entry > cache->dirty.last_version_checked)
1625 + cache->dirty.last_version_checked = dirty_version_at_entry;
1626 +
1627 + pgc_ll_unlock(cache, &cache->dirty);
1628 + }
1629 +
1630 + __atomic_sub_fetch(&cache->stats.workers_flush, 1, __ATOMIC_RELAXED);
1631 +
1632 + return stopped_before_finishing;
1633 +}
1634 +
1635 +void free_all_unreferenced_clean_pages(PGC *cache) {
1636 + evict_pages(cache, 0, 0, true, true);
1637 +}
1638 +
1639 +// ----------------------------------------------------------------------------
1640 +// public API
1641 +
1642 +PGC *pgc_create(size_t clean_size_bytes, free_clean_page_callback pgc_free_cb,
1643 + size_t max_dirty_pages_per_flush, save_dirty_page_callback pgc_save_dirty_cb,
1644 + size_t max_pages_per_inline_eviction, size_t max_skip_pages_per_inline_eviction,
1645 + size_t max_flushes_inline,
1646 + PGC_OPTIONS options, size_t partitions, size_t additional_bytes_per_page) {
1647 +
1648 + if(max_pages_per_inline_eviction < 2)
1649 + max_pages_per_inline_eviction = 2;
1650 +
1651 + if(max_dirty_pages_per_flush < 1)
1652 + max_dirty_pages_per_flush = 1;
1653 +
1654 + if(max_flushes_inline * max_dirty_pages_per_flush < 2)
1655 + max_flushes_inline = 2;
1656 +
1657 + PGC *cache = callocz(1, sizeof(PGC));
1658 + cache->config.options = options;
1659 + cache->config.clean_size = (clean_size_bytes < 1 * 1024 * 1024) ? 1 * 1024 * 1024 : clean_size_bytes;
1660 + cache->config.pgc_free_clean_cb = pgc_free_cb;
1661 + cache->config.max_dirty_pages_per_call = max_dirty_pages_per_flush,
1662 + cache->config.pgc_save_dirty_cb = pgc_save_dirty_cb;
1663 + cache->config.max_pages_per_inline_eviction = (max_pages_per_inline_eviction < 2) ? 2 : max_pages_per_inline_eviction;
1664 + cache->config.max_skip_pages_per_inline_eviction = (max_skip_pages_per_inline_eviction < 2) ? 2 : max_skip_pages_per_inline_eviction;
1665 + cache->config.max_flushes_inline = (max_flushes_inline < 1) ? 1 : max_flushes_inline;
1666 + cache->config.partitions = partitions < 1 ? (size_t)get_system_cpus() : partitions;
1667 + cache->config.additional_bytes_per_page = additional_bytes_per_page;
1668 +
1669 + cache->config.max_workers_evict_inline = 10;
1670 + cache->config.severe_pressure_per1000 = 1000;
1671 + cache->config.aggressive_evict_per1000 = 990;
1672 + cache->config.healthy_size_per1000 = 980;
1673 + cache->config.evict_low_threshold_per1000 = 970;
1674 +
1675 + cache->index = callocz(cache->config.partitions, sizeof(struct pgc_index));
1676 +
1677 + for(size_t part = 0; part < cache->config.partitions ; part++)
1678 + netdata_rwlock_init(&cache->index[part].rwlock);
1679 +
1680 + netdata_spinlock_init(&cache->hot.spinlock);
1681 + netdata_spinlock_init(&cache->dirty.spinlock);
1682 + netdata_spinlock_init(&cache->clean.spinlock);
1683 +
1684 + cache->hot.flags = PGC_PAGE_HOT;
1685 + cache->hot.linked_list_in_sections_judy = true;
1686 + cache->hot.stats = &cache->stats.queues.hot;
1687 +
1688 + cache->dirty.flags = PGC_PAGE_DIRTY;
1689 + cache->dirty.linked_list_in_sections_judy = true;
1690 + cache->dirty.stats = &cache->stats.queues.dirty;
1691 +
1692 + cache->clean.flags = PGC_PAGE_CLEAN;
1693 + cache->clean.linked_list_in_sections_judy = false;
1694 + cache->clean.stats = &cache->stats.queues.clean;
1695 +
1696 +#ifdef PGC_WITH_ARAL
1697 + cache->aral = arrayalloc_create(sizeof(PGC_PAGE) + cache->config.additional_bytes_per_page, 65536 / sizeof(PGC_PAGE),
1698 + NULL, NULL, false, false);
1699 +#endif
1700 +
1701 + pointer_index_init(cache);
1702 +
1703 + return cache;
1704 +}
1705 +
1706 +void pgc_flush_all_hot_and_dirty_pages(PGC *cache, Word_t section) {
1707 + all_hot_pages_to_dirty(cache, section);
1708 +
1709 + // save all dirty pages to make them clean
1710 + flush_pages(cache, 0, section, true, true);
1711 +}
1712 +
1713 +void pgc_destroy(PGC *cache) {
1714 + // convert all hot pages to dirty
1715 + all_hot_pages_to_dirty(cache, PGC_SECTION_ALL);
1716 +
1717 + // save all dirty pages to make them clean
1718 + flush_pages(cache, 0, PGC_SECTION_ALL, true, true);
1719 +
1720 + // free all unreferenced clean pages
1721 + free_all_unreferenced_clean_pages(cache);
1722 +
1723 + if(PGC_REFERENCED_PAGES(cache))
1724 + error("DBENGINE CACHE: there are %zu referenced cache pages - leaving the cache allocated", PGC_REFERENCED_PAGES(cache));
1725 + else {
1726 + pointer_destroy_index(cache);
1727 +#ifdef PGC_WITH_ARAL
1728 + arrayalloc_destroy(cache->aral);
1729 +#endif
1730 + freez(cache);
1731 + }
1732 +}
1733 +
1734 +PGC_PAGE *pgc_page_add_and_acquire(PGC *cache, PGC_ENTRY entry, bool *added) {
1735 + return page_add(cache, &entry, added);
1736 +}
1737 +
1738 +PGC_PAGE *pgc_page_dup(PGC *cache, PGC_PAGE *page) {
1739 + if(!page_acquire(cache, page))
1740 + fatal("DBENGINE CACHE: tried to dup a page that is not acquired!");
1741 +
1742 + return page;
1743 +}
1744 +
1745 +void pgc_page_release(PGC *cache, PGC_PAGE *page) {
1746 + page_release(cache, page, is_page_clean(page));
1747 +}
1748 +
1749 +void pgc_page_hot_to_dirty_and_release(PGC *cache, PGC_PAGE *page) {
1750 + __atomic_add_fetch(&cache->stats.workers_hot2dirty, 1, __ATOMIC_RELAXED);
1751 +
1752 +//#ifdef NETDATA_INTERNAL_CHECKS
1753 +// page_transition_lock(cache, page);
1754 +// internal_fatal(!is_page_hot(page), "DBENGINE CACHE: called %s() but page is not hot", __FUNCTION__ );
1755 +// page_transition_unlock(cache, page);
1756 +//#endif
1757 +
1758 + // make page dirty
1759 + page_set_dirty(cache, page, false);
1760 +
1761 + // release the page
1762 + page_release(cache, page, true);
1763 + // page ptr may be invalid now
1764 +
1765 + __atomic_sub_fetch(&cache->stats.workers_hot2dirty, 1, __ATOMIC_RELAXED);
1766 +
1767 + // flush, if we have to
1768 + if((cache->config.options & PGC_OPTIONS_FLUSH_PAGES_INLINE) || flushing_critical(cache)) {
1769 + flush_pages(cache, cache->config.max_flushes_inline, PGC_SECTION_ALL,
1770 + false, false);
1771 + }
1772 +}
1773 +
1774 +bool pgc_page_to_clean_evict_or_release(PGC *cache, PGC_PAGE *page) {
1775 + bool ret;
1776 +
1777 + __atomic_add_fetch(&cache->stats.workers_hot2dirty, 1, __ATOMIC_RELAXED);
1778 +
1779 + // prevent accesses from increasing the accesses counter
1780 + page_flag_set(page, PGC_PAGE_HAS_NO_DATA_IGNORE_ACCESSES);
1781 +
1782 + // zero the accesses counter
1783 + __atomic_store_n(&page->accesses, 0, __ATOMIC_RELEASE);
1784 +
1785 + // if there are no other references to it, evict it immediately
1786 + if(make_acquired_page_clean_and_evict_or_page_release(cache, page)) {
1787 + __atomic_add_fetch(&cache->stats.hot_empty_pages_evicted_immediately, 1, __ATOMIC_RELAXED);
1788 + ret = true;
1789 + }
1790 + else {
1791 + __atomic_add_fetch(&cache->stats.hot_empty_pages_evicted_later, 1, __ATOMIC_RELAXED);
1792 + ret = false;
1793 + }
1794 +
1795 + __atomic_sub_fetch(&cache->stats.workers_hot2dirty, 1, __ATOMIC_RELAXED);
1796 +
1797 + return ret;
1798 +}
1799 +
1800 +Word_t pgc_page_section(PGC_PAGE *page) {
1801 + return page->section;
1802 +}
1803 +
1804 +Word_t pgc_page_metric(PGC_PAGE *page) {
1805 + return page->metric_id;
1806 +}
1807 +
1808 +time_t pgc_page_start_time_s(PGC_PAGE *page) {
1809 + return page->start_time_s;
1810 +}
1811 +
1812 +time_t pgc_page_end_time_s(PGC_PAGE *page) {
1813 + return page->end_time_s;
1814 +}
1815 +
1816 +time_t pgc_page_update_every_s(PGC_PAGE *page) {
1817 + return page->update_every_s;
1818 +}
1819 +
1820 +time_t pgc_page_fix_update_every(PGC_PAGE *page, time_t update_every_s) {
1821 + if(page->update_every_s == 0)
1822 + page->update_every_s = update_every_s;
1823 +
1824 + return page->update_every_s;
1825 +}
1826 +
1827 +time_t pgc_page_fix_end_time_s(PGC_PAGE *page, time_t end_time_s) {
1828 + page->end_time_s = end_time_s;
1829 + return page->end_time_s;
1830 +}
1831 +
1832 +void *pgc_page_data(PGC_PAGE *page) {
1833 + return page->data;
1834 +}
1835 +
1836 +void *pgc_page_custom_data(PGC *cache, PGC_PAGE *page) {
1837 + if(cache->config.additional_bytes_per_page)
1838 + return page->custom_data;
1839 +
1840 + return NULL;
1841 +}
1842 +
1843 +size_t pgc_page_data_size(PGC *cache, PGC_PAGE *page) {
1844 + return page_size_from_assumed_size(cache, page->assumed_size);
1845 +}
1846 +
1847 +bool pgc_is_page_hot(PGC_PAGE *page) {
1848 + return is_page_hot(page);
1849 +}
1850 +
1851 +bool pgc_is_page_dirty(PGC_PAGE *page) {
1852 + return is_page_dirty(page);
1853 +}
1854 +
1855 +bool pgc_is_page_clean(PGC_PAGE *page) {
1856 + return is_page_clean(page);
1857 +}
1858 +
1859 +void pgc_reset_hot_max(PGC *cache) {
1860 + size_t entries = __atomic_load_n(&cache->hot.stats->entries, __ATOMIC_RELAXED);
1861 + size_t size = __atomic_load_n(&cache->hot.stats->size, __ATOMIC_RELAXED);
1862 +
1863 + __atomic_store_n(&cache->hot.stats->max_entries, entries, __ATOMIC_RELAXED);
1864 + __atomic_store_n(&cache->hot.stats->max_size, size, __ATOMIC_RELAXED);
1865 +
1866 + size_t size_to_evict = 0;
1867 + cache_usage_per1000(cache, &size_to_evict);
1868 + evict_pages(cache, 0, 0, true, false);
1869 +}
1870 +
1871 +void pgc_set_dynamic_target_cache_size_callback(PGC *cache, dynamic_target_cache_size_callback callback) {
1872 + cache->config.dynamic_target_size_cb = callback;
1873 +
1874 + size_t size_to_evict = 0;
1875 + cache_usage_per1000(cache, &size_to_evict);
1876 + evict_pages(cache, 0, 0, true, false);
1877 +}
1878 +
1879 +size_t pgc_get_current_cache_size(PGC *cache) {
1880 + cache_usage_per1000(cache, NULL);
1881 + return __atomic_load_n(&cache->stats.current_cache_size, __ATOMIC_RELAXED);
1882 +}
1883 +
1884 +size_t pgc_get_wanted_cache_size(PGC *cache) {
1885 + cache_usage_per1000(cache, NULL);
1886 + return __atomic_load_n(&cache->stats.wanted_cache_size, __ATOMIC_RELAXED);
1887 +}
1888 +
1889 +bool pgc_evict_pages(PGC *cache, size_t max_skip, size_t max_evict) {
1890 + bool under_pressure = cache_under_severe_pressure(cache);
1891 + return evict_pages(cache,
1892 + under_pressure ? 0 : max_skip,
1893 + under_pressure ? 0 : max_evict,
1894 + true, false);
1895 +}
1896 +
1897 +bool pgc_flush_pages(PGC *cache, size_t max_flushes) {
1898 + bool under_pressure = flushing_critical(cache);
1899 + return flush_pages(cache, under_pressure ? 0 : max_flushes, PGC_SECTION_ALL, true, false);
1900 +}
1901 +
1902 +void pgc_page_hot_set_end_time_s(PGC *cache __maybe_unused, PGC_PAGE *page, time_t end_time_s) {
1903 + internal_fatal(!is_page_hot(page),
1904 + "DBENGINE CACHE: end_time_s update on non-hot page");
1905 +
1906 + internal_fatal(end_time_s <= __atomic_load_n(&page->end_time_s, __ATOMIC_RELAXED),
1907 + "DBENGINE CACHE: end_time_s is not bigger than existing");
1908 +
1909 + __atomic_store_n(&page->end_time_s, end_time_s, __ATOMIC_RELAXED);
1910 +
1911 +#ifdef PGC_COUNT_POINTS_COLLECTED
1912 + __atomic_add_fetch(&cache->stats.points_collected, 1, __ATOMIC_RELAXED);
1913 +#endif
1914 +}
1915 +
1916 +PGC_PAGE *pgc_page_get_and_acquire(PGC *cache, Word_t section, Word_t metric_id, time_t start_time_s, PGC_SEARCH method) {
1917 + return page_find_and_acquire(cache, section, metric_id, start_time_s, method);
1918 +}
1919 +
1920 +struct pgc_statistics pgc_get_statistics(PGC *cache) {
1921 + // FIXME - get the statistics atomically
1922 + return cache->stats;
1923 +}
1924 +
1925 +void pgc_open_cache_to_journal_v2(PGC *cache, Word_t section, unsigned datafile_fileno, uint8_t type, migrate_to_v2_callback cb, void *data) {
1926 + __atomic_add_fetch(&cache->stats.workers_jv2_flush, 1, __ATOMIC_RELAXED);
1927 +
1928 + pgc_ll_lock(cache, &cache->hot);
1929 +
1930 + Pvoid_t JudyL_metrics = NULL;
1931 + Pvoid_t JudyL_extents_pos = NULL;
1932 +
1933 + size_t count_of_unique_extents = 0;
1934 + size_t count_of_unique_metrics = 0;
1935 + size_t count_of_unique_pages = 0;
1936 +
1937 + size_t master_extent_index_id = 0;
1938 +
1939 + Pvoid_t *section_pages_pptr = JudyLGet(cache->hot.sections_judy, section, PJE0);
1940 + if(!section_pages_pptr) {
1941 + pgc_ll_unlock(cache, &cache->hot);
1942 + return;
1943 + }
1944 +
1945 + struct section_pages *sp = *section_pages_pptr;
1946 + if(!netdata_spinlock_trylock(&sp->migration_to_v2_spinlock)) {
1947 + internal_fatal(true, "DBENGINE: migration to journal v2 is already running for this section");
1948 + pgc_ll_unlock(cache, &cache->hot);
1949 + return;
1950 + }
1951 +
1952 + for(PGC_PAGE *page = sp->base; page ; page = page->link.next) {
1953 + struct extent_io_data *xio = (struct extent_io_data *)page->custom_data;
1954 + if(xio->fileno != datafile_fileno) continue;
1955 +
1956 + if(page_flag_check(page, PGC_PAGE_IS_BEING_MIGRATED_TO_V2)) {
1957 + internal_fatal(true, "Migration to journal v2: page has already been migrated to v2");
1958 + continue;
1959 + }
1960 +
1961 + if(!page_transition_trylock(cache, page)) {
1962 + internal_fatal(true, "Migration to journal v2: cannot get page transition lock");
1963 + continue;
1964 + }
1965 +
1966 + if(!page_acquire(cache, page)) {
1967 + internal_fatal(true, "Migration to journal v2: cannot acquire page for migration to v2");
1968 + continue;
1969 + }
1970 +
1971 + page_flag_set(page, PGC_PAGE_IS_BEING_MIGRATED_TO_V2);
1972 +
1973 + pgc_ll_unlock(cache, &cache->hot);
1974 +
1975 + // update the extents JudyL
1976 +
1977 + size_t current_extent_index_id;
1978 + Pvoid_t *PValue = JudyLIns(&JudyL_extents_pos, xio->pos, PJE0);
1979 + if(!PValue || *PValue == PJERR)
1980 + fatal("Corrupted JudyL extents pos");
1981 +
1982 + struct jv2_extents_info *ei;
1983 + if(!*PValue) {
1984 + ei = callocz(1, sizeof(struct jv2_extents_info));
1985 + ei->pos = xio->pos;
1986 + ei->bytes = xio->bytes;
1987 + ei->number_of_pages = 1;
1988 + ei->index = master_extent_index_id++;
1989 + *PValue = ei;
1990 +
1991 + count_of_unique_extents++;
1992 + }
1993 + else {
1994 + ei = *PValue;
1995 + ei->number_of_pages++;
1996 + }
1997 +
1998 + current_extent_index_id = ei->index;
1999 +
2000 + // update the metrics JudyL
2001 +
2002 + PValue = JudyLIns(&JudyL_metrics, page->metric_id, PJE0);
2003 + if(!PValue || *PValue == PJERR)
2004 + fatal("Corrupted JudyL metrics");
2005 +
2006 + struct jv2_metrics_info *mi;
2007 + if(!*PValue) {
2008 + mi = callocz(1, sizeof(struct jv2_metrics_info));
2009 + mi->uuid = mrg_metric_uuid(main_mrg, (METRIC *)page->metric_id);
2010 + mi->first_time_s = page->start_time_s;
2011 + mi->last_time_s = page->end_time_s;
2012 + mi->number_of_pages = 1;
2013 + *PValue = mi;
2014 +
2015 + count_of_unique_metrics++;
2016 + }
2017 + else {
2018 + mi = *PValue;
2019 + mi->number_of_pages++;
2020 + if(page->start_time_s < mi->first_time_s)
2021 + mi->first_time_s = page->start_time_s;
2022 + if(page->end_time_s > mi->last_time_s)
2023 + mi->last_time_s = page->end_time_s;
2024 + }
2025 +
2026 + PValue = JudyLIns(&mi->JudyL_pages_by_start_time, page->start_time_s, PJE0);
2027 + if(!PValue || *PValue == PJERR)
2028 + fatal("Corrupted JudyL metric pages");
2029 +
2030 + if(!*PValue) {
2031 + struct jv2_page_info *pi = callocz(1, (sizeof(struct jv2_page_info)));
2032 + pi->start_time_s = page->start_time_s;
2033 + pi->end_time_s = page->end_time_s;
2034 + pi->update_every_s = page->update_every_s;
2035 + pi->page_length = page_size_from_assumed_size(cache, page->assumed_size);
2036 + pi->page = page;
2037 + pi->extent_index = current_extent_index_id;
2038 + pi->custom_data = (cache->config.additional_bytes_per_page) ? page->custom_data : NULL;
2039 + *PValue = pi;
2040 +
2041 + count_of_unique_pages++;
2042 + }
2043 + else {
2044 + // impossible situation
2045 + internal_fatal(true, "Page is already in JudyL metric pages");
2046 + page_flag_clear(page, PGC_PAGE_IS_BEING_MIGRATED_TO_V2);
2047 + page_transition_unlock(cache, page);
2048 + page_release(cache, page, false);
2049 + }
2050 +
2051 + pgc_ll_lock(cache, &cache->hot);
2052 + }
2053 +
2054 + netdata_spinlock_unlock(&sp->migration_to_v2_spinlock);
2055 + pgc_ll_unlock(cache, &cache->hot);
2056 +
2057 + // callback
2058 + cb(section, datafile_fileno, type, JudyL_metrics, JudyL_extents_pos, count_of_unique_extents, count_of_unique_metrics, count_of_unique_pages, data);
2059 +
2060 + {
2061 + Pvoid_t *PValue1;
2062 + bool metric_id_first = true;
2063 + Word_t metric_id = 0;
2064 + while ((PValue1 = JudyLFirstThenNext(JudyL_metrics, &metric_id, &metric_id_first))) {
2065 + struct jv2_metrics_info *mi = *PValue1;
2066 +
2067 + Pvoid_t *PValue2;
2068 + bool start_time_first = true;
2069 + Word_t start_time = 0;
2070 + while ((PValue2 = JudyLFirstThenNext(mi->JudyL_pages_by_start_time, &start_time, &start_time_first))) {
2071 + struct jv2_page_info *pi = *PValue2;
2072 + page_transition_unlock(cache, pi->page);
2073 + pgc_page_hot_to_dirty_and_release(cache, pi->page);
2074 + // make_acquired_page_clean_and_evict_or_page_release(cache, pi->page);
2075 + freez(pi);
2076 + }
2077 +
2078 + JudyLFreeArray(&mi->JudyL_pages_by_start_time, PJE0);
2079 + freez(mi);
2080 + }
2081 + JudyLFreeArray(&JudyL_metrics, PJE0);
2082 + }
2083 +
2084 + {
2085 + Pvoid_t *PValue;
2086 + bool extent_pos_first = true;
2087 + Word_t extent_pos = 0;
2088 + while ((PValue = JudyLFirstThenNext(JudyL_extents_pos, &extent_pos, &extent_pos_first))) {
2089 + struct jv2_extents_info *ei = *PValue;
2090 + freez(ei);
2091 + }
2092 + JudyLFreeArray(&JudyL_extents_pos, PJE0);
2093 + }
2094 +
2095 + __atomic_sub_fetch(&cache->stats.workers_jv2_flush, 1, __ATOMIC_RELAXED);
2096 +}
2097 +
2098 +static bool match_page_data(PGC_PAGE *page, void *data) {
2099 + return (page->data == data);
2100 +}
2101 +
2102 +void pgc_open_evict_clean_pages_of_datafile(PGC *cache, struct rrdengine_datafile *datafile) {
2103 + evict_pages_with_filter(cache, 0, 0, true, true, match_page_data, datafile);
2104 +}
2105 +
2106 +size_t pgc_count_clean_pages_having_data_ptr(PGC *cache, Word_t section, void *ptr) {
2107 + size_t found = 0;
2108 +
2109 + pgc_ll_lock(cache, &cache->clean);
2110 + for(PGC_PAGE *page = cache->clean.base; page ;page = page->link.next)
2111 + found += (page->data == ptr && page->section == section) ? 1 : 0;
2112 + pgc_ll_unlock(cache, &cache->clean);
2113 +
2114 + return found;
2115 +}
2116 +
2117 +size_t pgc_count_hot_pages_having_data_ptr(PGC *cache, Word_t section, void *ptr) {
2118 + size_t found = 0;
2119 +
2120 + pgc_ll_lock(cache, &cache->hot);
2121 + Pvoid_t *section_pages_pptr = JudyLGet(cache->hot.sections_judy, section, PJE0);
2122 + if(section_pages_pptr) {
2123 + struct section_pages *sp = *section_pages_pptr;
2124 + for(PGC_PAGE *page = sp->base; page ;page = page->link.next)
2125 + found += (page->data == ptr) ? 1 : 0;
2126 + }
2127 + pgc_ll_unlock(cache, &cache->hot);
2128 +
2129 + return found;
2130 +}
2131 +
2132 +// ----------------------------------------------------------------------------
2133 +// unittest
2134 +
2135 +static void unittest_free_clean_page_callback(PGC *cache __maybe_unused, PGC_ENTRY entry __maybe_unused) {
2136 + ;
2137 +}
2138 +
2139 +static void unittest_save_dirty_page_callback(PGC *cache __maybe_unused, PGC_ENTRY *entries_array __maybe_unused, PGC_PAGE **pages_array __maybe_unused, size_t entries __maybe_unused) {
2140 + ;
2141 +}
2142 +
2143 +#ifdef PGC_STRESS_TEST
2144 +
2145 +struct {
2146 + bool stop;
2147 + PGC *cache;
2148 + PGC_PAGE **metrics;
2149 + size_t clean_metrics;
2150 + size_t hot_metrics;
2151 + time_t first_time_t;
2152 + time_t last_time_t;
2153 + size_t cache_size;
2154 + size_t query_threads;
2155 + size_t collect_threads;
2156 + size_t partitions;
2157 + size_t points_per_page;
2158 + time_t time_per_collection_ut;
2159 + time_t time_per_query_ut;
2160 + time_t time_per_flush_ut;
2161 + PGC_OPTIONS options;
2162 + char rand_statebufs[1024];
2163 + struct random_data *random_data;
2164 +} pgc_uts = {
2165 + .stop = false,
2166 + .metrics = NULL,
2167 + .clean_metrics = 100000,
2168 + .hot_metrics = 1000000,
2169 + .first_time_t = 100000000,
2170 + .last_time_t = 0,
2171 + .cache_size = 0, // get the default (8MB)
2172 + .collect_threads = 16,
2173 + .query_threads = 16,
2174 + .partitions = 0, // get the default (system cpus)
2175 + .options = PGC_OPTIONS_AUTOSCALE,/* PGC_OPTIONS_FLUSH_PAGES_INLINE | PGC_OPTIONS_EVICT_PAGES_INLINE,*/
2176 + .points_per_page = 10,
2177 + .time_per_collection_ut = 1000000,
2178 + .time_per_query_ut = 250,
2179 + .time_per_flush_ut = 100,
2180 + .rand_statebufs = {},
2181 + .random_data = NULL,
2182 +};
2183 +
2184 +void *unittest_stress_test_collector(void *ptr) {
2185 + size_t id = *((size_t *)ptr);
2186 +
2187 + size_t metric_start = pgc_uts.clean_metrics;
2188 + size_t metric_end = pgc_uts.clean_metrics + pgc_uts.hot_metrics;
2189 + size_t number_of_metrics = metric_end - metric_start;
2190 + size_t per_collector_metrics = number_of_metrics / pgc_uts.collect_threads;
2191 + metric_start = metric_start + per_collector_metrics * id + 1;
2192 + metric_end = metric_start + per_collector_metrics - 1;
2193 +
2194 + time_t start_time_t = pgc_uts.first_time_t + 1;
2195 +
2196 + heartbeat_t hb;
2197 + heartbeat_init(&hb);
2198 +
2199 + while(!__atomic_load_n(&pgc_uts.stop, __ATOMIC_RELAXED)) {
2200 + // info("COLLECTOR %zu: collecting metrics %zu to %zu, from %ld to %lu", id, metric_start, metric_end, start_time_t, start_time_t + pgc_uts.points_per_page);
2201 +
2202 + netdata_thread_disable_cancelability();
2203 +
2204 + for (size_t i = metric_start; i < metric_end; i++) {
2205 + bool added;
2206 +
2207 + pgc_uts.metrics[i] = pgc_page_add_and_acquire(pgc_uts.cache, (PGC_ENTRY) {
2208 + .section = 1,
2209 + .metric_id = i,
2210 + .start_time_t = start_time_t,
2211 + .end_time_t = start_time_t,
2212 + .update_every = 1,
2213 + .size = 4096,
2214 + .data = NULL,
2215 + .hot = true,
2216 + }, &added);
2217 +
2218 + if(!pgc_is_page_hot(pgc_uts.metrics[i]) || !added) {
2219 + pgc_page_release(pgc_uts.cache, pgc_uts.metrics[i]);
2220 + pgc_uts.metrics[i] = NULL;
2221 + }
2222 + }
2223 +
2224 + time_t end_time_t = start_time_t + (time_t)pgc_uts.points_per_page;
2225 + while(++start_time_t <= end_time_t && !__atomic_load_n(&pgc_uts.stop, __ATOMIC_RELAXED)) {
2226 + heartbeat_next(&hb, pgc_uts.time_per_collection_ut);
2227 +
2228 + for (size_t i = metric_start; i < metric_end; i++) {
2229 + if(pgc_uts.metrics[i])
2230 + pgc_page_hot_set_end_time_t(pgc_uts.cache, pgc_uts.metrics[i], start_time_t);
2231 + }
2232 +
2233 + __atomic_store_n(&pgc_uts.last_time_t, start_time_t, __ATOMIC_RELAXED);
2234 + }
2235 +
2236 + for (size_t i = metric_start; i < metric_end; i++) {
2237 + if (pgc_uts.metrics[i]) {
2238 + if(i % 10 == 0)
2239 + pgc_page_to_clean_evict_or_release(pgc_uts.cache, pgc_uts.metrics[i]);
2240 + else
2241 + pgc_page_hot_to_dirty_and_release(pgc_uts.cache, pgc_uts.metrics[i]);
2242 + }
2243 + }
2244 +
2245 + netdata_thread_enable_cancelability();
2246 + }
2247 +
2248 + return ptr;
2249 +}
2250 +
2251 +void *unittest_stress_test_queries(void *ptr) {
2252 + size_t id = *((size_t *)ptr);
2253 + struct random_data *random_data = &pgc_uts.random_data[id];
2254 +
2255 + size_t start = 0;
2256 + size_t end = pgc_uts.clean_metrics + pgc_uts.hot_metrics;
2257 +
2258 + while(!__atomic_load_n(&pgc_uts.stop, __ATOMIC_RELAXED)) {
2259 + netdata_thread_disable_cancelability();
2260 +
2261 + int32_t random_number;
2262 + random_r(random_data, &random_number);
2263 +
2264 + size_t metric_id = random_number % (end - start);
2265 + time_t start_time_t = pgc_uts.first_time_t;
2266 + time_t end_time_t = __atomic_load_n(&pgc_uts.last_time_t, __ATOMIC_RELAXED);
2267 + if(end_time_t <= start_time_t)
2268 + end_time_t = start_time_t + 1;
2269 + size_t pages = (end_time_t - start_time_t) / pgc_uts.points_per_page + 1;
2270 +
2271 + PGC_PAGE *array[pages];
2272 + for(size_t i = 0; i < pages ;i++)
2273 + array[i] = NULL;
2274 +
2275 + // find the pages the cache has
2276 + for(size_t i = 0; i < pages ;i++) {
2277 + time_t page_start_time = start_time_t + (time_t)(i * pgc_uts.points_per_page);
2278 + array[i] = pgc_page_get_and_acquire(pgc_uts.cache, 1, metric_id,
2279 + page_start_time, (i < pages - 1)?PGC_SEARCH_EXACT:PGC_SEARCH_CLOSEST);
2280 + }
2281 +
2282 + // load the rest of the pages
2283 + for(size_t i = 0; i < pages ;i++) {
2284 + if(array[i]) continue;
2285 +
2286 + time_t page_start_time = start_time_t + (time_t)(i * pgc_uts.points_per_page);
2287 + array[i] = pgc_page_add_and_acquire(pgc_uts.cache, (PGC_ENTRY) {
2288 + .section = 1,
2289 + .metric_id = metric_id,
2290 + .start_time_t = page_start_time,
2291 + .end_time_t = page_start_time + (time_t)pgc_uts.points_per_page,
2292 + .update_every = 1,
2293 + .size = 4096,
2294 + .data = NULL,
2295 + .hot = false,
2296 + }, NULL);
2297 + }
2298 +
2299 + // do the query
2300 + // ...
2301 + struct timespec work_duration = {.tv_sec = 0, .tv_nsec = pgc_uts.time_per_query_ut * NSEC_PER_USEC };
2302 + nanosleep(&work_duration, NULL);
2303 +
2304 + // release the pages
2305 + for(size_t i = 0; i < pages ;i++) {
2306 + if(!array[i]) continue;
2307 + pgc_page_release(pgc_uts.cache, array[i]);
2308 + array[i] = NULL;
2309 + }
2310 +
2311 + netdata_thread_enable_cancelability();
2312 + }
2313 +
2314 + return ptr;
2315 +}
2316 +
2317 +void *unittest_stress_test_service(void *ptr) {
2318 + heartbeat_t hb;
2319 + heartbeat_init(&hb);
2320 + while(!__atomic_load_n(&pgc_uts.stop, __ATOMIC_RELAXED)) {
2321 + heartbeat_next(&hb, 1 * USEC_PER_SEC);
2322 +
2323 + pgc_flush_pages(pgc_uts.cache, 1000);
2324 + pgc_evict_pages(pgc_uts.cache, 0, 0);
2325 + }
2326 + return ptr;
2327 +}
2328 +
2329 +static void unittest_stress_test_save_dirty_page_callback(PGC *cache __maybe_unused, PGC_ENTRY *entries_array __maybe_unused, PGC_PAGE **pages_array __maybe_unused, size_t entries __maybe_unused) {
2330 + // info("SAVE %zu pages", entries);
2331 + if(!pgc_uts.stop) {
2332 + usec_t t = pgc_uts.time_per_flush_ut;
2333 +
2334 + if(t > 0) {
2335 + struct timespec work_duration = {
2336 + .tv_sec = t / USEC_PER_SEC,
2337 + .tv_nsec = (long) ((t % USEC_PER_SEC) * NSEC_PER_USEC)
2338 + };
2339 +
2340 + nanosleep(&work_duration, NULL);
2341 + }
2342 + }
2343 +}
2344 +
2345 +void unittest_stress_test(void) {
2346 + pgc_uts.cache = pgc_create(pgc_uts.cache_size * 1024 * 1024,
2347 + unittest_free_clean_page_callback,
2348 + 64, unittest_stress_test_save_dirty_page_callback,
2349 + 1000, 10000, 1,
2350 + pgc_uts.options, pgc_uts.partitions, 0);
2351 +
2352 + pgc_uts.metrics = callocz(pgc_uts.clean_metrics + pgc_uts.hot_metrics, sizeof(PGC_PAGE *));
2353 +
2354 + pthread_t service_thread;
2355 + netdata_thread_create(&service_thread, "SERVICE",
2356 + NETDATA_THREAD_OPTION_JOINABLE | NETDATA_THREAD_OPTION_DONT_LOG,
2357 + unittest_stress_test_service, NULL);
2358 +
2359 + pthread_t collect_threads[pgc_uts.collect_threads];
2360 + size_t collect_thread_ids[pgc_uts.collect_threads];
2361 + for(size_t i = 0; i < pgc_uts.collect_threads ;i++) {
2362 + collect_thread_ids[i] = i;
2363 + char buffer[100 + 1];
2364 + snprintfz(buffer, 100, "COLLECT_%zu", i);
2365 + netdata_thread_create(&collect_threads[i], buffer,
2366 + NETDATA_THREAD_OPTION_JOINABLE | NETDATA_THREAD_OPTION_DONT_LOG,
2367 + unittest_stress_test_collector, &collect_thread_ids[i]);
2368 + }
2369 +
2370 + pthread_t queries_threads[pgc_uts.query_threads];
2371 + size_t query_thread_ids[pgc_uts.query_threads];
2372 + pgc_uts.random_data = callocz(pgc_uts.query_threads, sizeof(struct random_data));
2373 + for(size_t i = 0; i < pgc_uts.query_threads ;i++) {
2374 + query_thread_ids[i] = i;
2375 + char buffer[100 + 1];
2376 + snprintfz(buffer, 100, "QUERY_%zu", i);
2377 + initstate_r(1, pgc_uts.rand_statebufs, 1024, &pgc_uts.random_data[i]);
2378 + netdata_thread_create(&queries_threads[i], buffer,
2379 + NETDATA_THREAD_OPTION_JOINABLE | NETDATA_THREAD_OPTION_DONT_LOG,
2380 + unittest_stress_test_queries, &query_thread_ids[i]);
2381 + }
2382 +
2383 + heartbeat_t hb;
2384 + heartbeat_init(&hb);
2385 +
2386 + struct {
2387 + size_t entries;
2388 + size_t added;
2389 + size_t deleted;
2390 + size_t referenced;
2391 +
2392 + size_t hot_entries;
2393 + size_t hot_added;
2394 + size_t hot_deleted;
2395 +
2396 + size_t dirty_entries;
2397 + size_t dirty_added;
2398 + size_t dirty_deleted;
2399 +
2400 + size_t clean_entries;
2401 + size_t clean_added;
2402 + size_t clean_deleted;
2403 +
2404 + size_t searches_exact;
2405 + size_t searches_exact_hits;
2406 + size_t searches_closest;
2407 + size_t searches_closest_hits;
2408 +
2409 + size_t collections;
2410 +
2411 + size_t events_cache_under_severe_pressure;
2412 + size_t events_cache_needs_space_90;
2413 + size_t events_flush_critical;
2414 + } stats = {}, old_stats = {};
2415 +
2416 + for(int i = 0; i < 86400 ;i++) {
2417 + heartbeat_next(&hb, 1 * USEC_PER_SEC);
2418 +
2419 + old_stats = stats;
2420 + stats.entries = __atomic_load_n(&pgc_uts.cache->stats.entries, __ATOMIC_RELAXED);
2421 + stats.added = __atomic_load_n(&pgc_uts.cache->stats.added_entries, __ATOMIC_RELAXED);
2422 + stats.deleted = __atomic_load_n(&pgc_uts.cache->stats.removed_entries, __ATOMIC_RELAXED);
2423 + stats.referenced = __atomic_load_n(&pgc_uts.cache->stats.referenced_entries, __ATOMIC_RELAXED);
2424 +
2425 + stats.hot_entries = __atomic_load_n(&pgc_uts.cache->hot.stats->entries, __ATOMIC_RELAXED);
2426 + stats.hot_added = __atomic_load_n(&pgc_uts.cache->hot.stats->added_entries, __ATOMIC_RELAXED);
2427 + stats.hot_deleted = __atomic_load_n(&pgc_uts.cache->hot.stats->removed_entries, __ATOMIC_RELAXED);
2428 +
2429 + stats.dirty_entries = __atomic_load_n(&pgc_uts.cache->dirty.stats->entries, __ATOMIC_RELAXED);
2430 + stats.dirty_added = __atomic_load_n(&pgc_uts.cache->dirty.stats->added_entries, __ATOMIC_RELAXED);
2431 + stats.dirty_deleted = __atomic_load_n(&pgc_uts.cache->dirty.stats->removed_entries, __ATOMIC_RELAXED);
2432 +
2433 + stats.clean_entries = __atomic_load_n(&pgc_uts.cache->clean.stats->entries, __ATOMIC_RELAXED);
2434 + stats.clean_added = __atomic_load_n(&pgc_uts.cache->clean.stats->added_entries, __ATOMIC_RELAXED);
2435 + stats.clean_deleted = __atomic_load_n(&pgc_uts.cache->clean.stats->removed_entries, __ATOMIC_RELAXED);
2436 +
2437 + stats.searches_exact = __atomic_load_n(&pgc_uts.cache->stats.searches_exact, __ATOMIC_RELAXED);
2438 + stats.searches_exact_hits = __atomic_load_n(&pgc_uts.cache->stats.searches_exact_hits, __ATOMIC_RELAXED);
2439 +
2440 + stats.searches_closest = __atomic_load_n(&pgc_uts.cache->stats.searches_closest, __ATOMIC_RELAXED);
2441 + stats.searches_closest_hits = __atomic_load_n(&pgc_uts.cache->stats.searches_closest_hits, __ATOMIC_RELAXED);
2442 +
2443 + stats.events_cache_under_severe_pressure = __atomic_load_n(&pgc_uts.cache->stats.events_cache_under_severe_pressure, __ATOMIC_RELAXED);
2444 + stats.events_cache_needs_space_90 = __atomic_load_n(&pgc_uts.cache->stats.events_cache_needs_space_aggressively, __ATOMIC_RELAXED);
2445 + stats.events_flush_critical = __atomic_load_n(&pgc_uts.cache->stats.events_flush_critical, __ATOMIC_RELAXED);
2446 +
2447 + size_t searches_exact = stats.searches_exact - old_stats.searches_exact;
2448 + size_t searches_closest = stats.searches_closest - old_stats.searches_closest;
2449 +
2450 + size_t hit_exact = stats.searches_exact_hits - old_stats.searches_exact_hits;
2451 + size_t hit_closest = stats.searches_closest_hits - old_stats.searches_closest_hits;
2452 +
2453 + double hit_exact_pc = (searches_exact > 0) ? (double)hit_exact * 100.0 / (double)searches_exact : 0.0;
2454 + double hit_closest_pc = (searches_closest > 0) ? (double)hit_closest * 100.0 / (double)searches_closest : 0.0;
2455 +
2456 +#ifdef PGC_COUNT_POINTS_COLLECTED
2457 + stats.collections = __atomic_load_n(&pgc_uts.cache->stats.points_collected, __ATOMIC_RELAXED);
2458 +#endif
2459 +
2460 + char *cache_status = "N";
2461 + if(stats.events_cache_under_severe_pressure > old_stats.events_cache_under_severe_pressure)
2462 + cache_status = "F";
2463 + else if(stats.events_cache_needs_space_90 > old_stats.events_cache_needs_space_90)
2464 + cache_status = "f";
2465 +
2466 + char *flushing_status = "N";
2467 + if(stats.events_flush_critical > old_stats.events_flush_critical)
2468 + flushing_status = "F";
2469 +
2470 + info("PGS %5zuk +%4zuk/-%4zuk "
2471 + "| RF %5zuk "
2472 + "| HOT %5zuk +%4zuk -%4zuk "
2473 + "| DRT %s %5zuk +%4zuk -%4zuk "
2474 + "| CLN %s %5zuk +%4zuk -%4zuk "
2475 + "| SRCH %4zuk %4zuk, HIT %4.1f%% %4.1f%% "
2476 +#ifdef PGC_COUNT_POINTS_COLLECTED
2477 + "| CLCT %8.4f Mps"
2478 +#endif
2479 + , stats.entries / 1000
2480 + , (stats.added - old_stats.added) / 1000, (stats.deleted - old_stats.deleted) / 1000
2481 + , stats.referenced / 1000
2482 + , stats.hot_entries / 1000, (stats.hot_added - old_stats.hot_added) / 1000, (stats.hot_deleted - old_stats.hot_deleted) / 1000
2483 + , flushing_status
2484 + , stats.dirty_entries / 1000
2485 + , (stats.dirty_added - old_stats.dirty_added) / 1000, (stats.dirty_deleted - old_stats.dirty_deleted) / 1000
2486 + , cache_status
2487 + , stats.clean_entries / 1000
2488 + , (stats.clean_added - old_stats.clean_added) / 1000, (stats.clean_deleted - old_stats.clean_deleted) / 1000
2489 + , searches_exact / 1000, searches_closest / 1000
2490 + , hit_exact_pc, hit_closest_pc
2491 +#ifdef PGC_COUNT_POINTS_COLLECTED
2492 + , (double)(stats.collections - old_stats.collections) / 1000.0 / 1000.0
2493 +#endif
2494 + );
2495 + }
2496 + info("Waiting for threads to stop...");
2497 + __atomic_store_n(&pgc_uts.stop, true, __ATOMIC_RELAXED);
2498 +
2499 + netdata_thread_join(service_thread, NULL);
2500 +
2501 + for(size_t i = 0; i < pgc_uts.collect_threads ;i++)
2502 + netdata_thread_join(collect_threads[i],NULL);
2503 +
2504 + for(size_t i = 0; i < pgc_uts.query_threads ;i++)
2505 + netdata_thread_join(queries_threads[i],NULL);
2506 +
2507 + pgc_destroy(pgc_uts.cache);
2508 +
2509 + freez(pgc_uts.metrics);
2510 + freez(pgc_uts.random_data);
2511 +}
2512 +#endif
2513 +
2514 +int pgc_unittest(void) {
2515 + PGC *cache = pgc_create(32 * 1024 * 1024, unittest_free_clean_page_callback,
2516 + 64, unittest_save_dirty_page_callback,
2517 + 10, 1000, 10,
2518 + PGC_OPTIONS_DEFAULT, 1, 11);
2519 +
2520 + // FIXME - unit tests
2521 + // - add clean page
2522 + // - add clean page again (should not add it)
2523 + // - release page (should decrement counters)
2524 + // - add hot page
2525 + // - add hot page again (should not add it)
2526 + // - turn hot page to dirty, with and without a reference counter to it
2527 + // - dirty pages are saved once there are enough of them
2528 + // - find page exact
2529 + // - find page (should return last)
2530 + // - find page (should return next)
2531 + // - page cache full (should evict)
2532 + // - on destroy, turn hot pages to dirty and save them
2533 +
2534 + PGC_PAGE *page1 = pgc_page_add_and_acquire(cache, (PGC_ENTRY){
2535 + .section = 1,
2536 + .metric_id = 10,
2537 + .start_time_s = 100,
2538 + .end_time_s = 1000,
2539 + .size = 4096,
2540 + .data = NULL,
2541 + .hot = false,
2542 + .custom_data = (uint8_t *)"0123456789",
2543 + }, NULL);
2544 +
2545 + if(strcmp(pgc_page_custom_data(cache, page1), "0123456789") != 0)
2546 + fatal("custom data do not work");
2547 +
2548 + memcpy(pgc_page_custom_data(cache, page1), "ABCDEFGHIJ", 11);
2549 + if(strcmp(pgc_page_custom_data(cache, page1), "ABCDEFGHIJ") != 0)
2550 + fatal("custom data do not work");
2551 +
2552 + pgc_page_release(cache, page1);
2553 +
2554 + PGC_PAGE *page2 = pgc_page_add_and_acquire(cache, (PGC_ENTRY){
2555 + .section = 2,
2556 + .metric_id = 10,
2557 + .start_time_s = 1001,
2558 + .end_time_s = 2000,
2559 + .size = 4096,
2560 + .data = NULL,
2561 + .hot = true,
2562 + }, NULL);
2563 +
2564 + pgc_page_hot_set_end_time_s(cache, page2, 2001);
2565 + pgc_page_hot_to_dirty_and_release(cache, page2);
2566 +
2567 + PGC_PAGE *page3 = pgc_page_add_and_acquire(cache, (PGC_ENTRY){
2568 + .section = 3,
2569 + .metric_id = 10,
2570 + .start_time_s = 1001,
2571 + .end_time_s = 2000,
2572 + .size = 4096,
2573 + .data = NULL,
2574 + .hot = true,
2575 + }, NULL);
2576 +
2577 + pgc_page_hot_set_end_time_s(cache, page3, 2001);
2578 + pgc_page_hot_to_dirty_and_release(cache, page3);
2579 +
2580 + pgc_destroy(cache);
2581 +
2582 +#ifdef PGC_STRESS_TEST
2583 + unittest_stress_test();
2584 +#endif
2585 +
2586 + return 0;
2587 +}
database/engine/cache.h new
+238
@@ -0,0 +1,238 @@
1 +#ifndef DBENGINE_CACHE_H
2 +#define DBENGINE_CACHE_H
3 +
4 +#include "../rrd.h"
5 +
6 +// CACHE COMPILE TIME CONFIGURATION
7 +// #define PGC_COUNT_POINTS_COLLECTED 1
8 +
9 +typedef struct pgc PGC;
10 +typedef struct pgc_page PGC_PAGE;
11 +
12 +typedef enum __attribute__ ((__packed__)) {
13 + PGC_OPTIONS_NONE = 0,
14 + PGC_OPTIONS_EVICT_PAGES_INLINE = (1 << 0),
15 + PGC_OPTIONS_FLUSH_PAGES_INLINE = (1 << 1),
16 + PGC_OPTIONS_AUTOSCALE = (1 << 2),
17 +} PGC_OPTIONS;
18 +
19 +#define PGC_OPTIONS_DEFAULT (PGC_OPTIONS_EVICT_PAGES_INLINE | PGC_OPTIONS_FLUSH_PAGES_INLINE | PGC_OPTIONS_AUTOSCALE)
20 +
21 +typedef struct pgc_entry {
22 + Word_t section; // the section this belongs to
23 + Word_t metric_id; // the metric this belongs to
24 + time_t start_time_s; // the start time of the page
25 + time_t end_time_s; // the end time of the page
26 + size_t size; // the size in bytes of the allocation, outside the cache
27 + void *data; // a pointer to data outside the cache
28 + uint32_t update_every_s; // the update every of the page
29 + bool hot; // true if this entry is currently being collected
30 + uint8_t *custom_data;
31 +} PGC_ENTRY;
32 +
33 +#define PGC_CACHE_LINE_PADDING(x) uint8_t padding##x[128]
34 +
35 +struct pgc_queue_statistics {
36 + size_t entries;
37 + size_t size;
38 +
39 + PGC_CACHE_LINE_PADDING(1);
40 +
41 + size_t max_entries;
42 + size_t max_size;
43 +
44 + PGC_CACHE_LINE_PADDING(2);
45 +
46 + size_t added_entries;
47 + size_t added_size;
48 +
49 + PGC_CACHE_LINE_PADDING(3);
50 +
51 + size_t removed_entries;
52 + size_t removed_size;
53 +
54 + PGC_CACHE_LINE_PADDING(4);
55 +};
56 +
57 +struct pgc_statistics {
58 + size_t wanted_cache_size;
59 + size_t current_cache_size;
60 +
61 + PGC_CACHE_LINE_PADDING(1);
62 +
63 + size_t added_entries;
64 + size_t added_size;
65 +
66 + PGC_CACHE_LINE_PADDING(2);
67 +
68 + size_t removed_entries;
69 + size_t removed_size;
70 +
71 + PGC_CACHE_LINE_PADDING(3);
72 +
73 + size_t entries; // all the entries (includes clean, dirty, host)
74 + size_t size; // all the entries (includes clean, dirty, host)
75 +
76 + size_t evicting_entries;
77 + size_t evicting_size;
78 +
79 + size_t flushing_entries;
80 + size_t flushing_size;
81 +
82 + PGC_CACHE_LINE_PADDING(4);
83 +
84 + size_t acquires;
85 + PGC_CACHE_LINE_PADDING(4a);
86 + size_t releases;
87 + PGC_CACHE_LINE_PADDING(4b);
88 + size_t acquires_for_deletion;
89 + PGC_CACHE_LINE_PADDING(4c);
90 +
91 + size_t referenced_entries; // all the entries currently referenced
92 + size_t referenced_size; // all the entries currently referenced
93 +
94 + PGC_CACHE_LINE_PADDING(5);
95 +
96 + size_t searches_exact;
97 + size_t searches_exact_hits;
98 + size_t searches_exact_misses;
99 +
100 + PGC_CACHE_LINE_PADDING(6);
101 +
102 + size_t searches_closest;
103 + size_t searches_closest_hits;
104 + size_t searches_closest_misses;
105 +
106 + PGC_CACHE_LINE_PADDING(7);
107 +
108 + size_t flushes_completed;
109 + size_t flushes_completed_size;
110 + size_t flushes_cancelled;
111 + size_t flushes_cancelled_size;
112 +
113 +#ifdef PGC_COUNT_POINTS_COLLECTED
114 + PGC_CACHE_LINE_PADDING(8);
115 + size_t points_collected;
116 +#endif
117 +
118 + PGC_CACHE_LINE_PADDING(9);
119 +
120 + size_t insert_spins;
121 + size_t evict_spins;
122 + size_t release_spins;
123 + size_t acquire_spins;
124 + size_t delete_spins;
125 + size_t flush_spins;
126 +
127 + PGC_CACHE_LINE_PADDING(10);
128 +
129 + size_t workers_search;
130 + size_t workers_add;
131 + size_t workers_evict;
132 + size_t workers_flush;
133 + size_t workers_jv2_flush;
134 + size_t workers_hot2dirty;
135 +
136 + size_t evict_skipped;
137 + size_t hot_empty_pages_evicted_immediately;
138 + size_t hot_empty_pages_evicted_later;
139 +
140 + PGC_CACHE_LINE_PADDING(11);
141 +
142 + // events
143 + size_t events_cache_under_severe_pressure;
144 + size_t events_cache_needs_space_aggressively;
145 + size_t events_flush_critical;
146 +
147 + PGC_CACHE_LINE_PADDING(12);
148 +
149 + struct {
150 + PGC_CACHE_LINE_PADDING(0);
151 + struct pgc_queue_statistics hot;
152 + PGC_CACHE_LINE_PADDING(1);
153 + struct pgc_queue_statistics dirty;
154 + PGC_CACHE_LINE_PADDING(2);
155 + struct pgc_queue_statistics clean;
156 + PGC_CACHE_LINE_PADDING(3);
157 + } queues;
158 +};
159 +
160 +
161 +typedef void (*free_clean_page_callback)(PGC *cache, PGC_ENTRY entry);
162 +typedef void (*save_dirty_page_callback)(PGC *cache, PGC_ENTRY *entries_array, PGC_PAGE **pages_array, size_t entries);
163 +
164 +// create a cache
165 +PGC *pgc_create(size_t clean_size_bytes, free_clean_page_callback pgc_free_clean_cb,
166 + size_t max_dirty_pages_per_flush, save_dirty_page_callback pgc_save_dirty_cb,
167 + size_t max_pages_per_inline_eviction, size_t max_skip_pages_per_inline_eviction,
168 + size_t max_flushes_inline,
169 + PGC_OPTIONS options, size_t partitions, size_t additional_bytes_per_page);
170 +
171 +// destroy the cache
172 +void pgc_destroy(PGC *cache);
173 +
174 +#define PGC_SECTION_ALL ((Word_t)0)
175 +void pgc_flush_all_hot_and_dirty_pages(PGC *cache, Word_t section);
176 +
177 +// add a page to the cache and return a pointer to it
178 +PGC_PAGE *pgc_page_add_and_acquire(PGC *cache, PGC_ENTRY entry, bool *added);
179 +
180 +// get another reference counter on an already referenced page
181 +PGC_PAGE *pgc_page_dup(PGC *cache, PGC_PAGE *page);
182 +
183 +// release a page (all pointers to it are now invalid)
184 +void pgc_page_release(PGC *cache, PGC_PAGE *page);
185 +
186 +// mark a hot page dirty, and release it
187 +void pgc_page_hot_to_dirty_and_release(PGC *cache, PGC_PAGE *page);
188 +
189 +// find a page from the cache
190 +typedef enum {
191 + PGC_SEARCH_EXACT,
192 + PGC_SEARCH_CLOSEST,
193 + PGC_SEARCH_FIRST,
194 + PGC_SEARCH_NEXT,
195 + PGC_SEARCH_LAST,
196 + PGC_SEARCH_PREV,
197 +} PGC_SEARCH;
198 +
199 +PGC_PAGE *pgc_page_get_and_acquire(PGC *cache, Word_t section, Word_t metric_id, time_t start_time_s, PGC_SEARCH method);
200 +
201 +// get information from an acquired page
202 +Word_t pgc_page_section(PGC_PAGE *page);
203 +Word_t pgc_page_metric(PGC_PAGE *page);
204 +time_t pgc_page_start_time_s(PGC_PAGE *page);
205 +time_t pgc_page_end_time_s(PGC_PAGE *page);
206 +time_t pgc_page_update_every_s(PGC_PAGE *page);
207 +time_t pgc_page_fix_update_every(PGC_PAGE *page, time_t update_every_s);
208 +time_t pgc_page_fix_end_time_s(PGC_PAGE *page, time_t end_time_s);
209 +void *pgc_page_data(PGC_PAGE *page);
210 +void *pgc_page_custom_data(PGC *cache, PGC_PAGE *page);
211 +size_t pgc_page_data_size(PGC *cache, PGC_PAGE *page);
212 +bool pgc_is_page_hot(PGC_PAGE *page);
213 +bool pgc_is_page_dirty(PGC_PAGE *page);
214 +bool pgc_is_page_clean(PGC_PAGE *page);
215 +void pgc_reset_hot_max(PGC *cache);
216 +size_t pgc_get_current_cache_size(PGC *cache);
217 +size_t pgc_get_wanted_cache_size(PGC *cache);
218 +
219 +// resetting the end time of a hot page
220 +void pgc_page_hot_set_end_time_s(PGC *cache, PGC_PAGE *page, time_t end_time_s);
221 +bool pgc_page_to_clean_evict_or_release(PGC *cache, PGC_PAGE *page);
222 +
223 +typedef void (*migrate_to_v2_callback)(Word_t section, unsigned datafile_fileno, uint8_t type, Pvoid_t JudyL_metrics, Pvoid_t JudyL_extents_pos, size_t count_of_unique_extents, size_t count_of_unique_metrics, size_t count_of_unique_pages, void *data);
224 +void pgc_open_cache_to_journal_v2(PGC *cache, Word_t section, unsigned datafile_fileno, uint8_t type, migrate_to_v2_callback cb, void *data);
225 +void pgc_open_evict_clean_pages_of_datafile(PGC *cache, struct rrdengine_datafile *datafile);
226 +size_t pgc_count_clean_pages_having_data_ptr(PGC *cache, Word_t section, void *ptr);
227 +size_t pgc_count_hot_pages_having_data_ptr(PGC *cache, Word_t section, void *ptr);
228 +
229 +typedef size_t (*dynamic_target_cache_size_callback)(void);
230 +void pgc_set_dynamic_target_cache_size_callback(PGC *cache, dynamic_target_cache_size_callback callback);
231 +
232 +// return true when there is more work to do
233 +bool pgc_evict_pages(PGC *cache, size_t max_skip, size_t max_evict);
234 +bool pgc_flush_pages(PGC *cache, size_t max_flushes);
235 +
236 +struct pgc_statistics pgc_get_statistics(PGC *cache);
237 +
238 +#endif // DBENGINE_CACHE_H
database/engine/datafile.c
+222 -81
@@ -1,52 +1,169 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2 #include "rrdengine.h"
3
4 -void df_extent_insert(struct extent_info *extent)
5 -{
6 - struct rrdengine_datafile *datafile = extent->datafile;
7 -
8 - if (likely(NULL != datafile->extents.last)) {
9 - datafile->extents.last->next = extent;
10 - }
11 - if (unlikely(NULL == datafile->extents.first)) {
12 - datafile->extents.first = extent;
13 - }
14 - datafile->extents.last = extent;
15 -}
16 -
4 void datafile_list_insert(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile)
5 {
19 - if (likely(NULL != ctx->datafiles.last)) {
20 - ctx->datafiles.last->next = datafile;
21 - }
22 - if (unlikely(NULL == ctx->datafiles.first)) {
23 - ctx->datafiles.first = datafile;
24 - }
25 - ctx->datafiles.last = datafile;
6 + uv_rwlock_wrlock(&ctx->datafiles.rwlock);
7 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(ctx->datafiles.first, datafile, prev, next);
8 + uv_rwlock_wrunlock(&ctx->datafiles.rwlock);
9 }
10
28 -void datafile_list_delete(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile)
11 +void datafile_list_delete_unsafe(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile)
12 {
30 - struct rrdengine_datafile *next;
31 -
32 - next = datafile->next;
33 - fatal_assert((NULL != next) && (ctx->datafiles.first == datafile) && (ctx->datafiles.last != datafile));
34 - ctx->datafiles.first = next;
13 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(ctx->datafiles.first, datafile, prev, next);
14 }
15
16
38 -static void datafile_init(struct rrdengine_datafile *datafile, struct rrdengine_instance *ctx,
39 - unsigned tier, unsigned fileno)
17 +static struct rrdengine_datafile *datafile_alloc_and_init(struct rrdengine_instance *ctx, unsigned tier, unsigned fileno)
18 {
19 fatal_assert(tier == 1);
20 +
21 + struct rrdengine_datafile *datafile = callocz(1, sizeof(struct rrdengine_datafile));
22 +
23 datafile->tier = tier;
24 datafile->fileno = fileno;
44 - datafile->file = (uv_file)0;
45 - datafile->pos = 0;
46 - datafile->extents.first = datafile->extents.last = NULL; /* will be populated by journalfile */
47 - datafile->journalfile = NULL;
48 - datafile->next = NULL;
25 + fatal_assert(0 == uv_rwlock_init(&datafile->extent_rwlock));
26 datafile->ctx = ctx;
27 +
28 + datafile->users.spinlock = NETDATA_SPINLOCK_INITIALIZER;
29 + datafile->users.available = true;
30 +
31 + datafile->extent_exclusive_access.spinlock = NETDATA_SPINLOCK_INITIALIZER;
32 +
33 + datafile->writers.spinlock = NETDATA_SPINLOCK_INITIALIZER;
34 +
35 + return datafile;
36 +}
37 +
38 +void datafile_acquire_dup(struct rrdengine_datafile *df) {
39 + netdata_spinlock_lock(&df->users.spinlock);
40 +
41 + if(!df->users.lockers)
42 + fatal("DBENGINE: datafile is not acquired to duplicate");
43 +
44 + df->users.lockers++;
45 +
46 + netdata_spinlock_unlock(&df->users.spinlock);
47 +}
48 +
49 +bool datafile_acquire(struct rrdengine_datafile *df, DATAFILE_ACQUIRE_REASONS reason) {
50 + bool ret;
51 +
52 + netdata_spinlock_lock(&df->users.spinlock);
53 +
54 + if(df->users.available) {
55 + ret = true;
56 + df->users.lockers++;
57 + df->users.lockers_by_reason[reason]++;
58 + }
59 + else
60 + ret = false;
61 +
62 + netdata_spinlock_unlock(&df->users.spinlock);
63 +
64 + return ret;
65 +}
66 +
67 +void datafile_release(struct rrdengine_datafile *df, DATAFILE_ACQUIRE_REASONS reason) {
68 + netdata_spinlock_lock(&df->users.spinlock);
69 + if(!df->users.lockers)
70 + fatal("DBENGINE DATAFILE: cannot release a datafile that is not acquired");
71 +
72 + df->users.lockers--;
73 + df->users.lockers_by_reason[reason]--;
74 + netdata_spinlock_unlock(&df->users.spinlock);
75 +}
76 +
77 +bool datafile_acquire_for_deletion(struct rrdengine_datafile *df) {
78 + bool can_be_deleted = false;
79 +
80 + netdata_spinlock_lock(&df->users.spinlock);
81 + df->users.available = false;
82 +
83 + if(!df->users.lockers)
84 + can_be_deleted = true;
85 +
86 + else {
87 + // there are lockers
88 +
89 + // evict any pages referencing this in the open cache
90 + netdata_spinlock_unlock(&df->users.spinlock);
91 + pgc_open_evict_clean_pages_of_datafile(open_cache, df);
92 + netdata_spinlock_lock(&df->users.spinlock);
93 +
94 + if(!df->users.lockers)
95 + can_be_deleted = true;
96 +
97 + else {
98 + // there are lockers still
99 +
100 + // count the number of pages referencing this in the open cache
101 + netdata_spinlock_unlock(&df->users.spinlock);
102 + usec_t time_to_scan_ut = now_monotonic_usec();
103 + size_t clean_pages_in_open_cache = pgc_count_clean_pages_having_data_ptr(open_cache, (Word_t)df->ctx, df);
104 + size_t hot_pages_in_open_cache = pgc_count_hot_pages_having_data_ptr(open_cache, (Word_t)df->ctx, df);
105 + time_to_scan_ut = now_monotonic_usec() - time_to_scan_ut;
106 + netdata_spinlock_lock(&df->users.spinlock);
107 +
108 + if(!df->users.lockers)
109 + can_be_deleted = true;
110 +
111 + else if(!clean_pages_in_open_cache && !hot_pages_in_open_cache) {
112 + // no pages in the open cache related to this datafile
113 +
114 + time_t now_s = now_monotonic_sec();
115 +
116 + if(!df->users.time_to_evict) {
117 + // first time we did the above
118 + df->users.time_to_evict = now_s + 120;
119 + internal_error(true, "DBENGINE: datafile %u of tier %d is not used by any open cache pages, "
120 + "but it has %u lockers (oc:%u, pd:%u), "
121 + "%zu clean and %zu hot open cache pages "
122 + "- will be deleted shortly "
123 + "(scanned open cache in %llu usecs)",
124 + df->fileno, df->ctx->tier,
125 + df->users.lockers,
126 + df->users.lockers_by_reason[DATAFILE_ACQUIRE_OPEN_CACHE],
127 + df->users.lockers_by_reason[DATAFILE_ACQUIRE_PAGE_DETAILS],
128 + clean_pages_in_open_cache,
129 + hot_pages_in_open_cache,
130 + time_to_scan_ut);
131 + }
132 +
133 + else if(now_s > df->users.time_to_evict) {
134 + // time expired, lets remove it
135 + can_be_deleted = true;
136 + internal_error(true, "DBENGINE: datafile %u of tier %d is not used by any open cache pages, "
137 + "but it has %u lockers (oc:%u, pd:%u), "
138 + "%zu clean and %zu hot open cache pages "
139 + "- will be deleted now "
140 + "(scanned open cache in %llu usecs)",
141 + df->fileno, df->ctx->tier,
142 + df->users.lockers,
143 + df->users.lockers_by_reason[DATAFILE_ACQUIRE_OPEN_CACHE],
144 + df->users.lockers_by_reason[DATAFILE_ACQUIRE_PAGE_DETAILS],
145 + clean_pages_in_open_cache,
146 + hot_pages_in_open_cache,
147 + time_to_scan_ut);
148 + }
149 + }
150 + else
151 + internal_error(true, "DBENGINE: datafile %u of tier %d "
152 + "has %u lockers (oc:%u, pd:%u), "
153 + "%zu clean and %zu hot open cache pages "
154 + "(scanned open cache in %llu usecs)",
155 + df->fileno, df->ctx->tier,
156 + df->users.lockers,
157 + df->users.lockers_by_reason[DATAFILE_ACQUIRE_OPEN_CACHE],
158 + df->users.lockers_by_reason[DATAFILE_ACQUIRE_PAGE_DETAILS],
159 + clean_pages_in_open_cache,
160 + hot_pages_in_open_cache,
161 + time_to_scan_ut);
162 + }
163 + }
164 + netdata_spinlock_unlock(&df->users.spinlock);
165 +
166 + return can_be_deleted;
167 }
168
169 void generate_datafilepath(struct rrdengine_datafile *datafile, char *str, size_t maxlen)
@@ -66,7 +183,7 @@ int close_data_file(struct rrdengine_datafile *datafile)
183
184 ret = uv_fs_close(NULL, &req, datafile->file, NULL);
185 if (ret < 0) {
69 - error("uv_fs_close(%s): %s", path, uv_strerror(ret));
186 + error("DBENGINE: uv_fs_close(%s): %s", path, uv_strerror(ret));
187 ++ctx->stats.fs_errors;
188 rrd_stat_atomic_add(&global_fs_errors, 1);
189 }
@@ -86,7 +203,7 @@ int unlink_data_file(struct rrdengine_datafile *datafile)
203
204 ret = uv_fs_unlink(NULL, &req, path, NULL);
205 if (ret < 0) {
89 - error("uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
206 + error("DBENGINE: uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
207 ++ctx->stats.fs_errors;
208 rrd_stat_atomic_add(&global_fs_errors, 1);
209 }
@@ -97,7 +214,7 @@ int unlink_data_file(struct rrdengine_datafile *datafile)
214 return ret;
215 }
216
100 -int destroy_data_file(struct rrdengine_datafile *datafile)
217 +int destroy_data_file_unsafe(struct rrdengine_datafile *datafile)
218 {
219 struct rrdengine_instance *ctx = datafile->ctx;
220 uv_fs_t req;
@@ -108,7 +225,7 @@ int destroy_data_file(struct rrdengine_datafile *datafile)
225
226 ret = uv_fs_ftruncate(NULL, &req, datafile->file, 0, NULL);
227 if (ret < 0) {
111 - error("uv_fs_ftruncate(%s): %s", path, uv_strerror(ret));
228 + error("DBENGINE: uv_fs_ftruncate(%s): %s", path, uv_strerror(ret));
229 ++ctx->stats.fs_errors;
230 rrd_stat_atomic_add(&global_fs_errors, 1);
231 }
@@ -116,7 +233,7 @@ int destroy_data_file(struct rrdengine_datafile *datafile)
233
234 ret = uv_fs_close(NULL, &req, datafile->file, NULL);
235 if (ret < 0) {
119 - error("uv_fs_close(%s): %s", path, uv_strerror(ret));
236 + error("DBENGINE: uv_fs_close(%s): %s", path, uv_strerror(ret));
237 ++ctx->stats.fs_errors;
238 rrd_stat_atomic_add(&global_fs_errors, 1);
239 }
@@ -124,7 +241,7 @@ int destroy_data_file(struct rrdengine_datafile *datafile)
241
242 ret = uv_fs_unlink(NULL, &req, path, NULL);
243 if (ret < 0) {
127 - error("uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
244 + error("DBENGINE: uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
245 ++ctx->stats.fs_errors;
246 rrd_stat_atomic_add(&global_fs_errors, 1);
247 }
@@ -157,7 +274,7 @@ int create_data_file(struct rrdengine_datafile *datafile)
274
275 ret = posix_memalign((void *)&superblock, RRDFILE_ALIGNMENT, sizeof(*superblock));
276 if (unlikely(ret)) {
160 - fatal("posix_memalign:%s", strerror(ret));
277 + fatal("DBENGINE: posix_memalign:%s", strerror(ret));
278 }
279 memset(superblock, 0, sizeof(*superblock));
280 (void) strncpy(superblock->magic_number, RRDENG_DF_MAGIC, RRDENG_MAGIC_SZ);
@@ -169,14 +286,14 @@ int create_data_file(struct rrdengine_datafile *datafile)
286 ret = uv_fs_write(NULL, &req, file, &iov, 1, 0, NULL);
287 if (ret < 0) {
288 fatal_assert(req.result < 0);
172 - error("uv_fs_write: %s", uv_strerror(ret));
289 + error("DBENGINE: uv_fs_write: %s", uv_strerror(ret));
290 ++ctx->stats.io_errors;
291 rrd_stat_atomic_add(&global_io_errors, 1);
292 }
293 uv_fs_req_cleanup(&req);
294 posix_memfree(superblock);
295 if (ret < 0) {
179 - destroy_data_file(datafile);
296 + destroy_data_file_unsafe(datafile);
297 return ret;
298 }
299
@@ -196,13 +313,13 @@ static int check_data_file_superblock(uv_file file)
313
314 ret = posix_memalign((void *)&superblock, RRDFILE_ALIGNMENT, sizeof(*superblock));
315 if (unlikely(ret)) {
199 - fatal("posix_memalign:%s", strerror(ret));
316 + fatal("DBENGINE: posix_memalign:%s", strerror(ret));
317 }
318 iov = uv_buf_init((void *)superblock, sizeof(*superblock));
319
320 ret = uv_fs_read(NULL, &req, file, &iov, 1, 0, NULL);
321 if (ret < 0) {
205 - error("uv_fs_read: %s", uv_strerror(ret));
322 + error("DBENGINE: uv_fs_read: %s", uv_strerror(ret));
323 uv_fs_req_cleanup(&req);
324 goto error;
325 }
@@ -212,7 +329,7 @@ static int check_data_file_superblock(uv_file file)
329 if (strncmp(superblock->magic_number, RRDENG_DF_MAGIC, RRDENG_MAGIC_SZ) ||
330 strncmp(superblock->version, RRDENG_DF_VER, RRDENG_VER_SZ) ||
331 superblock->tier != 1) {
215 - error("File has invalid superblock.");
332 + error("DBENGINE: file has invalid superblock.");
333 ret = UV_EINVAL;
334 } else {
335 ret = 0;
@@ -238,7 +355,7 @@ static int load_data_file(struct rrdengine_datafile *datafile)
355 rrd_stat_atomic_add(&global_fs_errors, 1);
356 return fd;
357 }
241 - info("Initializing data file \"%s\".", path);
358 + info("DBENGINE: initializing data file \"%s\".", path);
359
360 ret = check_file_properties(file, &file_size, sizeof(struct rrdeng_df_sb));
361 if (ret)
@@ -254,14 +371,14 @@ static int load_data_file(struct rrdengine_datafile *datafile)
371 datafile->file = file;
372 datafile->pos = file_size;
373
257 - info("Data file \"%s\" initialized (size:%"PRIu64").", path, file_size);
374 + info("DBENGINE: data file \"%s\" initialized (size:%"PRIu64").", path, file_size);
375 return 0;
376
377 error:
378 error = ret;
379 ret = uv_fs_close(NULL, &req, file, NULL);
380 if (ret < 0) {
264 - error("uv_fs_close(%s): %s", path, uv_strerror(ret));
381 + error("DBENGINE: uv_fs_close(%s): %s", path, uv_strerror(ret));
382 ++ctx->stats.fs_errors;
383 rrd_stat_atomic_add(&global_fs_errors, 1);
384 }
@@ -295,21 +412,18 @@ static int scan_data_files(struct rrdengine_instance *ctx)
412 if (ret < 0) {
413 fatal_assert(req.result < 0);
414 uv_fs_req_cleanup(&req);
298 - error("uv_fs_scandir(%s): %s", ctx->dbfiles_path, uv_strerror(ret));
415 + error("DBENGINE: uv_fs_scandir(%s): %s", ctx->dbfiles_path, uv_strerror(ret));
416 ++ctx->stats.fs_errors;
417 rrd_stat_atomic_add(&global_fs_errors, 1);
418 return ret;
419 }
303 - info("Found %d files in path %s", ret, ctx->dbfiles_path);
420 + info("DBENGINE: found %d files in path %s", ret, ctx->dbfiles_path);
421
422 datafiles = callocz(MIN(ret, MAX_DATAFILES), sizeof(*datafiles));
423 for (matched_files = 0 ; UV_EOF != uv_fs_scandir_next(&req, &dent) && matched_files < MAX_DATAFILES ; ) {
307 - info("Scanning file \"%s/%s\"", ctx->dbfiles_path, dent.name);
424 ret = sscanf(dent.name, DATAFILE_PREFIX RRDENG_FILE_NUMBER_SCAN_TMPL DATAFILE_EXTENSION, &tier, &no);
425 if (2 == ret) {
310 - info("Matched file \"%s/%s\"", ctx->dbfiles_path, dent.name);
311 - datafile = mallocz(sizeof(*datafile));
312 - datafile_init(datafile, ctx, tier, no);
426 + datafile = datafile_alloc_and_init(ctx, tier, no);
427 datafiles[matched_files++] = datafile;
428 }
429 }
@@ -320,7 +434,7 @@ static int scan_data_files(struct rrdengine_instance *ctx)
434 return 0;
435 }
436 if (matched_files == MAX_DATAFILES) {
323 - error("Warning: hit maximum database engine file limit of %d files", MAX_DATAFILES);
437 + error("DBENGINE: warning: hit maximum database engine file limit of %d files", MAX_DATAFILES);
438 }
439 qsort(datafiles, matched_files, sizeof(*datafiles), scan_data_files_cmp);
440 /* TODO: change this when tiering is implemented */
@@ -346,16 +460,16 @@ static int scan_data_files(struct rrdengine_instance *ctx)
460 if (must_delete_pair) {
461 char path[RRDENG_PATH_MAX];
462
349 - error("Deleting invalid data and journal file pair.");
463 + error("DBENGINE: deleting invalid data and journal file pair.");
464 ret = unlink_journal_file(journalfile);
465 if (!ret) {
466 generate_journalfilepath(datafile, path, sizeof(path));
353 - info("Deleted journal file \"%s\".", path);
467 + info("DBENGINE: deleted journal file \"%s\".", path);
468 }
469 ret = unlink_data_file(datafile);
470 if (!ret) {
471 generate_datafilepath(datafile, path, sizeof(path));
358 - info("Deleted data file \"%s\".", path);
472 + info("DBENGINE: deleted data file \"%s\".", path);
473 }
474 freez(journalfile);
475 freez(datafile);
@@ -373,20 +487,20 @@ static int scan_data_files(struct rrdengine_instance *ctx)
487 }
488
489 /* Creates a datafile and a journalfile pair */
376 -int create_new_datafile_pair(struct rrdengine_instance *ctx, unsigned tier, unsigned fileno)
490 +int create_new_datafile_pair(struct rrdengine_instance *ctx)
491 {
492 struct rrdengine_datafile *datafile;
493 struct rrdengine_journalfile *journalfile;
494 + unsigned fileno = __atomic_load_n(&ctx->last_fileno, __ATOMIC_RELAXED) + 1;
495 int ret;
496 char path[RRDENG_PATH_MAX];
497
383 - info("Creating new data and journal files in path %s", ctx->dbfiles_path);
384 - datafile = mallocz(sizeof(*datafile));
385 - datafile_init(datafile, ctx, tier, fileno);
498 + info("DBENGINE: creating new data and journal files in path %s", ctx->dbfiles_path);
499 + datafile = datafile_alloc_and_init(ctx, 1, fileno);
500 ret = create_data_file(datafile);
501 if (!ret) {
502 generate_datafilepath(datafile, path, sizeof(path));
389 - info("Created data file \"%s\".", path);
503 + info("DBENGINE: created data file \"%s\".", path);
504 } else {
505 goto error_after_datafile;
506 }
@@ -397,20 +511,24 @@ int create_new_datafile_pair(struct rrdengine_instance *ctx, unsigned tier, unsi
511 ret = create_journal_file(journalfile, datafile);
512 if (!ret) {
513 generate_journalfilepath(datafile, path, sizeof(path));
400 - info("Created journal file \"%s\".", path);
514 + info("DBENGINE: created journal file \"%s\".", path);
515 } else {
516 goto error_after_journalfile;
517 }
518 datafile_list_insert(ctx, datafile);
519 ctx->disk_space += datafile->pos + journalfile->pos;
520
521 + __atomic_add_fetch(&ctx->last_fileno, 1, __ATOMIC_RELAXED);
522 +
523 return 0;
524
525 error_after_journalfile:
410 - destroy_data_file(datafile);
526 + destroy_data_file_unsafe(datafile);
527 freez(journalfile);
528 error_after_datafile:
529 freez(datafile);
530 +
531 + uv_rwlock_wrunlock(&ctx->datafiles.rwlock);
532 return ret;
533 }
534
@@ -421,40 +539,63 @@ int init_data_files(struct rrdengine_instance *ctx)
539 {
540 int ret;
541
542 + fatal_assert(0 == uv_rwlock_init(&ctx->datafiles.rwlock));
543 + __atomic_store_n(&ctx->journal_initialization, true, __ATOMIC_RELAXED);
544 ret = scan_data_files(ctx);
545 if (ret < 0) {
426 - error("Failed to scan path \"%s\".", ctx->dbfiles_path);
546 + error("DBENGINE: failed to scan path \"%s\".", ctx->dbfiles_path);
547 return ret;
548 } else if (0 == ret) {
429 - info("Data files not found, creating in path \"%s\".", ctx->dbfiles_path);
430 - ret = create_new_datafile_pair(ctx, 1, 1);
549 + info("DBENGINE: data files not found, creating in path \"%s\".", ctx->dbfiles_path);
550 + ctx->last_fileno = 0;
551 + ret = create_new_datafile_pair(ctx);
552 if (ret) {
432 - error("Failed to create data and journal files in path \"%s\".", ctx->dbfiles_path);
553 + error("DBENGINE: failed to create data and journal files in path \"%s\".", ctx->dbfiles_path);
554 return ret;
555 }
435 - ctx->last_fileno = 1;
556 }
557 + else if(ctx->create_new_datafile_pair)
558 + create_new_datafile_pair(ctx);
559
560 + pgc_reset_hot_max(open_cache);
561 + ctx->create_new_datafile_pair = false;
562 + __atomic_store_n(&ctx->journal_initialization, false, __ATOMIC_RELAXED);
563 return 0;
564 }
565
566 void finalize_data_files(struct rrdengine_instance *ctx)
567 {
443 - struct rrdengine_datafile *datafile, *next_datafile;
444 - struct rrdengine_journalfile *journalfile;
445 - struct extent_info *extent, *next_extent;
446 -
447 - for (datafile = ctx->datafiles.first ; datafile != NULL ; datafile = next_datafile) {
448 - journalfile = datafile->journalfile;
449 - next_datafile = datafile->next;
568 + do {
569 + struct rrdengine_datafile *datafile = ctx->datafiles.first;
570 + struct rrdengine_journalfile *journalfile = datafile->journalfile;
571
451 - for (extent = datafile->extents.first ; extent != NULL ; extent = next_extent) {
452 - next_extent = extent->next;
453 - freez(extent);
572 + while(!datafile_acquire_for_deletion(datafile) && datafile != ctx->datafiles.first->prev) {
573 + info("Waiting to acquire data file %u of tier %d to close it...", datafile->fileno, ctx->tier);
574 + sleep_usec(500 * USEC_PER_MS);
575 }
576 +
577 + bool available = false;
578 + do {
579 + uv_rwlock_wrlock(&ctx->datafiles.rwlock);
580 + netdata_spinlock_lock(&datafile->writers.spinlock);
581 + available = (datafile->writers.running || datafile->writers.flushed_to_open_running) ? false : true;
582 +
583 + if(!available) {
584 + netdata_spinlock_unlock(&datafile->writers.spinlock);
585 + uv_rwlock_wrunlock(&ctx->datafiles.rwlock);
586 + info("Waiting for writers to data file %u of tier %d to finish...", datafile->fileno, ctx->tier);
587 + sleep_usec(500 * USEC_PER_MS);
588 + }
589 + } while(!available);
590 +
591 close_journal_file(journalfile, datafile);
592 close_data_file(datafile);
593 + datafile_list_delete_unsafe(ctx, datafile);
594 + netdata_spinlock_unlock(&datafile->writers.spinlock);
595 + uv_rwlock_wrunlock(&ctx->datafiles.rwlock);
596 +
597 freez(journalfile);
598 freez(datafile);
459 - }
599 +
600 + } while(ctx->datafiles.first);
601 }
database/engine/datafile.h
+47 -25
@@ -13,54 +13,76 @@ struct rrdengine_instance;
13 #define DATAFILE_PREFIX "datafile-"
14 #define DATAFILE_EXTENSION ".ndf"
15
16 -#define MAX_DATAFILE_SIZE (1073741824LU)
17 -#define MIN_DATAFILE_SIZE (4194304LU)
16 +#ifndef MAX_DATAFILE_SIZE
17 +#define MAX_DATAFILE_SIZE (512LU * 1024LU * 1024LU)
18 +#endif
19 +#if MIN_DATAFILE_SIZE > MAX_DATAFILE_SIZE
20 +#error MIN_DATAFILE_SIZE > MAX_DATAFILE_SIZE
21 +#endif
22 +
23 +#define MIN_DATAFILE_SIZE (4LU * 1024LU * 1024LU)
24 #define MAX_DATAFILES (65536) /* Supports up to 64TiB for now */
19 -#define TARGET_DATAFILES (20)
25 +#define TARGET_DATAFILES (50)
26
27 #define DATAFILE_IDEAL_IO_SIZE (1048576U)
28
23 -struct extent_info {
24 - uint64_t offset;
25 - uint32_t size;
26 - uint8_t number_of_pages;
27 - struct rrdengine_datafile *datafile;
28 - struct extent_info *next;
29 - struct rrdeng_page_descr *pages[];
30 -};
31 -
32 -struct rrdengine_df_extents {
33 - /* the extent list is sorted based on disk offset */
34 - struct extent_info *first;
35 - struct extent_info *last;
36 -};
37 -
29 /* only one event loop is supported for now */
30 struct rrdengine_datafile {
31 unsigned tier;
32 unsigned fileno;
33 uv_file file;
34 uint64_t pos;
35 + uv_rwlock_t extent_rwlock;
36 struct rrdengine_instance *ctx;
45 - struct rrdengine_df_extents extents;
37 struct rrdengine_journalfile *journalfile;
38 + struct rrdengine_datafile *prev;
39 struct rrdengine_datafile *next;
40 +
41 + struct {
42 + SPINLOCK spinlock;
43 + size_t running;
44 + size_t flushed_to_open_running;
45 + } writers;
46 +
47 + // exclusive access to extents
48 + struct {
49 + SPINLOCK spinlock;
50 + unsigned lockers;
51 + Pvoid_t extents_JudyL;
52 + } extent_exclusive_access;
53 +
54 + struct {
55 + SPINLOCK spinlock;
56 + unsigned lockers;
57 + unsigned lockers_by_reason[2];
58 + bool available;
59 + time_t time_to_evict;
60 + } users;
61 };
62
63 +typedef enum __attribute__ ((__packed__)) {
64 + DATAFILE_ACQUIRE_OPEN_CACHE = 0,
65 + DATAFILE_ACQUIRE_PAGE_DETAILS = 1,
66 +} DATAFILE_ACQUIRE_REASONS;
67 +
68 +void datafile_acquire_dup(struct rrdengine_datafile *df);
69 +bool datafile_acquire(struct rrdengine_datafile *df, DATAFILE_ACQUIRE_REASONS reason);
70 +void datafile_release(struct rrdengine_datafile *df, DATAFILE_ACQUIRE_REASONS reason);
71 +bool datafile_acquire_for_deletion(struct rrdengine_datafile *df);
72 +
73 struct rrdengine_datafile_list {
51 - struct rrdengine_datafile *first; /* oldest */
52 - struct rrdengine_datafile *last; /* newest */
74 + uv_rwlock_t rwlock;
75 + struct rrdengine_datafile *first; /* oldest - the newest with ->first->prev */
76 };
77
55 -void df_extent_insert(struct extent_info *extent);
78 void datafile_list_insert(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile);
57 -void datafile_list_delete(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile);
79 +void datafile_list_delete_unsafe(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile);
80 void generate_datafilepath(struct rrdengine_datafile *datafile, char *str, size_t maxlen);
81 int close_data_file(struct rrdengine_datafile *datafile);
82 int unlink_data_file(struct rrdengine_datafile *datafile);
61 -int destroy_data_file(struct rrdengine_datafile *datafile);
83 +int destroy_data_file_unsafe(struct rrdengine_datafile *datafile);
84 int create_data_file(struct rrdengine_datafile *datafile);
63 -int create_new_datafile_pair(struct rrdengine_instance *ctx, unsigned tier, unsigned fileno);
85 +int create_new_datafile_pair(struct rrdengine_instance *ctx);
86 int init_data_files(struct rrdengine_instance *ctx);
87 void finalize_data_files(struct rrdengine_instance *ctx);
88
database/engine/journalfile.c
+815 -192
@@ -1,96 +1,116 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2 #include "rrdengine.h"
3
4 +
5 +// DBENGINE2: Helper
6 +
7 +static void update_metric_retention_and_granularity_by_uuid(
8 + struct rrdengine_instance *ctx, uuid_t *uuid,
9 + time_t first_time_s, time_t last_time_s,
10 + time_t update_every_s, time_t now_s)
11 +{
12 + if(last_time_s > now_s) {
13 + error_limit_static_global_var(erl, 1, 0);
14 + error_limit(&erl, "DBENGINE JV2: wrong last time on-disk (%ld - %ld, now %ld), "
15 + "fixing last time to now",
16 + first_time_s, last_time_s, now_s);
17 + last_time_s = now_s;
18 + }
19 +
20 + if(first_time_s > last_time_s) {
21 + error_limit_static_global_var(erl, 1, 0);
22 + error_limit(&erl, "DBENGINE JV2: wrong first time on-disk (%ld - %ld, now %ld), "
23 + "fixing first time to last time",
24 + first_time_s, last_time_s, now_s);
25 +
26 + first_time_s = last_time_s;
27 + }
28 +
29 + if(first_time_s == 0 ||
30 + last_time_s == 0
31 + ) {
32 + error_limit_static_global_var(erl, 1, 0);
33 + error_limit(&erl, "DBENGINE JV2: zero on-disk timestamps (%ld - %ld, now %ld), "
34 + "using them as-is",
35 + first_time_s, last_time_s, now_s);
36 + }
37 +
38 + MRG_ENTRY entry = {
39 + .section = (Word_t)ctx,
40 + .first_time_s = first_time_s,
41 + .last_time_s = last_time_s,
42 + .latest_update_every_s = update_every_s
43 + };
44 + uuid_copy(entry.uuid, *uuid);
45 +
46 + bool added;
47 + METRIC *metric = mrg_metric_add_and_acquire(main_mrg, entry, &added);
48 +
49 + if (likely(!added))
50 + mrg_metric_expand_retention(main_mrg, metric, first_time_s, last_time_s, update_every_s);
51 +
52 + mrg_metric_release(main_mrg, metric);
53 +}
54 +
55 static void flush_transaction_buffer_cb(uv_fs_t* req)
56 {
6 - struct generic_io_descriptor *io_descr = req->data;
7 - struct rrdengine_worker_config* wc = req->loop->data;
8 - struct rrdengine_instance *ctx = wc->ctx;
57 + worker_is_busy(RRDENG_FLUSH_TRANSACTION_BUFFER_CB);
58 +
59 + WAL *wal = req->data;
60 + struct generic_io_descriptor *io_descr = &wal->io_descr;
61 + struct rrdengine_instance *ctx = io_descr->ctx;
62
63 debug(D_RRDENGINE, "%s: Journal block was written to disk.", __func__);
64 if (req->result < 0) {
65 ++ctx->stats.io_errors;
66 rrd_stat_atomic_add(&global_io_errors, 1);
14 - error("%s: uv_fs_write: %s", __func__, uv_strerror((int)req->result));
67 + error("DBENGINE: %s: uv_fs_write: %s", __func__, uv_strerror((int)req->result));
68 } else {
69 debug(D_RRDENGINE, "%s: Journal block was written to disk.", __func__);
70 }
71
72 uv_fs_req_cleanup(req);
20 - posix_memfree(io_descr->buf);
21 - freez(io_descr);
73 + wal_release(wal);
74 +
75 + __atomic_sub_fetch(&ctx->worker_config.atomics.extents_currently_being_flushed, 1, __ATOMIC_RELAXED);
76 +
77 + worker_is_idle();
78 }
79
80 /* Careful to always call this before creating a new journal file */
25 -void wal_flush_transaction_buffer(struct rrdengine_worker_config* wc)
81 +void wal_flush_transaction_buffer(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile, WAL *wal, uv_loop_t *loop)
82 {
27 - struct rrdengine_instance *ctx = wc->ctx;
83 int ret;
84 struct generic_io_descriptor *io_descr;
30 - unsigned pos, size;
31 - struct rrdengine_journalfile *journalfile;
85 + struct rrdengine_journalfile *journalfile = datafile->journalfile;
86
33 - if (unlikely(NULL == ctx->commit_log.buf || 0 == ctx->commit_log.buf_pos)) {
34 - return;
35 - }
36 - /* care with outstanding transactions when switching journal files */
37 - journalfile = ctx->datafiles.last->journalfile;
38 -
39 - io_descr = mallocz(sizeof(*io_descr));
40 - pos = ctx->commit_log.buf_pos;
41 - size = ctx->commit_log.buf_size;
42 - if (pos < size) {
87 + io_descr = &wal->io_descr;
88 + io_descr->ctx = ctx;
89 + if (wal->size < wal->buf_size) {
90 /* simulate an empty transaction to skip the rest of the block */
44 - *(uint8_t *) (ctx->commit_log.buf + pos) = STORE_PADDING;
91 + *(uint8_t *) (wal->buf + wal->size) = STORE_PADDING;
92 }
46 - io_descr->buf = ctx->commit_log.buf;
47 - io_descr->bytes = size;
93 + io_descr->buf = wal->buf;
94 + io_descr->bytes = wal->buf_size;
95 io_descr->pos = journalfile->pos;
49 - io_descr->req.data = io_descr;
96 + io_descr->req.data = wal;
97 + io_descr->data = journalfile;
98 io_descr->completion = NULL;
99
52 - io_descr->iov = uv_buf_init((void *)io_descr->buf, size);
53 - ret = uv_fs_write(wc->loop, &io_descr->req, journalfile->file, &io_descr->iov, 1,
100 + io_descr->iov = uv_buf_init((void *)io_descr->buf, wal->buf_size);
101 + ret = uv_fs_write(loop, &io_descr->req, journalfile->file, &io_descr->iov, 1,
102 journalfile->pos, flush_transaction_buffer_cb);
103 fatal_assert(-1 != ret);
56 - journalfile->pos += RRDENG_BLOCK_SIZE;
57 - ctx->disk_space += RRDENG_BLOCK_SIZE;
58 - ctx->commit_log.buf = NULL;
59 - ctx->stats.io_write_bytes += RRDENG_BLOCK_SIZE;
104 + journalfile->pos += wal->buf_size;
105 + ctx->disk_space += wal->buf_size;
106 + ctx->stats.io_write_bytes += wal->buf_size;
107 ++ctx->stats.io_write_requests;
108 }
109
63 -void * wal_get_transaction_buffer(struct rrdengine_worker_config* wc, unsigned size)
110 +void generate_journalfilepath_v2(struct rrdengine_datafile *datafile, char *str, size_t maxlen)
111 {
65 - struct rrdengine_instance *ctx = wc->ctx;
66 - int ret;
67 - unsigned buf_pos = 0, buf_size;
68 -
69 - fatal_assert(size);
70 - if (ctx->commit_log.buf) {
71 - unsigned remaining;
72 -
73 - buf_pos = ctx->commit_log.buf_pos;
74 - buf_size = ctx->commit_log.buf_size;
75 - remaining = buf_size - buf_pos;
76 - if (size > remaining) {
77 - /* we need a new buffer */
78 - wal_flush_transaction_buffer(wc);
79 - }
80 - }
81 - if (NULL == ctx->commit_log.buf) {
82 - buf_size = ALIGN_BYTES_CEILING(size);
83 - ret = posix_memalign((void *)&ctx->commit_log.buf, RRDFILE_ALIGNMENT, buf_size);
84 - if (unlikely(ret)) {
85 - fatal("posix_memalign:%s", strerror(ret));
86 - }
87 - memset(ctx->commit_log.buf, 0, buf_size);
88 - buf_pos = ctx->commit_log.buf_pos = 0;
89 - ctx->commit_log.buf_size = buf_size;
90 - }
91 - ctx->commit_log.buf_pos += size;
92 -
93 - return ctx->commit_log.buf + buf_pos;
112 + (void) snprintfz(str, maxlen, "%s/" WALFILE_PREFIX RRDENG_FILE_NUMBER_PRINT_TMPL WALFILE_EXTENSION_V2,
113 + datafile->ctx->dbfiles_path, datafile->tier, datafile->fileno);
114 }
115
116 void generate_journalfilepath(struct rrdengine_datafile *datafile, char *str, size_t maxlen)
@@ -104,28 +124,51 @@ void journalfile_init(struct rrdengine_journalfile *journalfile, struct rrdengin
124 journalfile->file = (uv_file)0;
125 journalfile->pos = 0;
126 journalfile->datafile = datafile;
127 + SET_JOURNAL_DATA(journalfile, 0);
128 + SET_JOURNAL_DATA_SIZE(journalfile, 0);
129 + journalfile->data = NULL;
130 }
131
109 -int close_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
132 +static int close_uv_file(struct rrdengine_datafile *datafile, uv_file file)
133 {
111 - struct rrdengine_instance *ctx = datafile->ctx;
112 - uv_fs_t req;
134 int ret;
135 char path[RRDENG_PATH_MAX];
136
116 - generate_journalfilepath(datafile, path, sizeof(path));
117 -
118 - ret = uv_fs_close(NULL, &req, journalfile->file, NULL);
137 + uv_fs_t req;
138 + ret = uv_fs_close(NULL, &req, file, NULL);
139 if (ret < 0) {
120 - error("uv_fs_close(%s): %s", path, uv_strerror(ret));
121 - ++ctx->stats.fs_errors;
140 + generate_journalfilepath(datafile, path, sizeof(path));
141 + error("DBENGINE: uv_fs_close(%s): %s", path, uv_strerror(ret));
142 + ++datafile->ctx->stats.fs_errors;
143 rrd_stat_atomic_add(&global_fs_errors, 1);
144 }
145 uv_fs_req_cleanup(&req);
125 -
146 return ret;
147 }
148
149 +int close_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
150 +{
151 + struct rrdengine_instance *ctx = datafile->ctx;
152 + char path[RRDENG_PATH_MAX];
153 +
154 + void *journal_data = GET_JOURNAL_DATA(journalfile);
155 + size_t journal_data_size = GET_JOURNAL_DATA_SIZE(journalfile);
156 +
157 + if (likely(journal_data)) {
158 + if (munmap(journal_data, journal_data_size)) {
159 + generate_journalfilepath_v2(datafile, path, sizeof(path));
160 + error("DBENGINE: failed to unmap journal index file for %s", path);
161 + ++ctx->stats.fs_errors;
162 + rrd_stat_atomic_add(&global_fs_errors, 1);
163 + }
164 + SET_JOURNAL_DATA(journalfile, 0);
165 + SET_JOURNAL_DATA_SIZE(journalfile, 0);
166 + return 0;
167 + }
168 +
169 + return close_uv_file(datafile, journalfile->file);
170 +}
171 +
172 int unlink_journal_file(struct rrdengine_journalfile *journalfile)
173 {
174 struct rrdengine_datafile *datafile = journalfile->datafile;
@@ -138,7 +181,7 @@ int unlink_journal_file(struct rrdengine_journalfile *journalfile)
181
182 ret = uv_fs_unlink(NULL, &req, path, NULL);
183 if (ret < 0) {
141 - error("uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
184 + error("DBENGINE: uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
185 ++ctx->stats.fs_errors;
186 rrd_stat_atomic_add(&global_fs_errors, 1);
187 }
@@ -149,26 +192,32 @@ int unlink_journal_file(struct rrdengine_journalfile *journalfile)
192 return ret;
193 }
194
152 -int destroy_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
195 +int destroy_journal_file_unsafe(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
196 {
197 struct rrdengine_instance *ctx = datafile->ctx;
198 uv_fs_t req;
199 int ret;
200 char path[RRDENG_PATH_MAX];
201 + char path_v2[RRDENG_PATH_MAX];
202
203 generate_journalfilepath(datafile, path, sizeof(path));
204 + generate_journalfilepath_v2(datafile, path_v2, sizeof(path));
205
206 + if (journalfile->file) {
207 ret = uv_fs_ftruncate(NULL, &req, journalfile->file, 0, NULL);
208 if (ret < 0) {
163 - error("uv_fs_ftruncate(%s): %s", path, uv_strerror(ret));
209 + error("DBENGINE: uv_fs_ftruncate(%s): %s", path, uv_strerror(ret));
210 ++ctx->stats.fs_errors;
211 rrd_stat_atomic_add(&global_fs_errors, 1);
212 }
213 uv_fs_req_cleanup(&req);
214 + (void) close_uv_file(datafile, journalfile->file);
215 + }
216
169 - ret = uv_fs_close(NULL, &req, journalfile->file, NULL);
217 + // This is the new journal v2 index file
218 + ret = uv_fs_unlink(NULL, &req, path_v2, NULL);
219 if (ret < 0) {
171 - error("uv_fs_close(%s): %s", path, uv_strerror(ret));
220 + error("DBENGINE: uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
221 ++ctx->stats.fs_errors;
222 rrd_stat_atomic_add(&global_fs_errors, 1);
223 }
@@ -176,13 +225,23 @@ int destroy_journal_file(struct rrdengine_journalfile *journalfile, struct rrden
225
226 ret = uv_fs_unlink(NULL, &req, path, NULL);
227 if (ret < 0) {
179 - error("uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
228 + error("DBENGINE: uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
229 ++ctx->stats.fs_errors;
230 rrd_stat_atomic_add(&global_fs_errors, 1);
231 }
232 uv_fs_req_cleanup(&req);
233
234 ++ctx->stats.journalfile_deletions;
235 + ++ctx->stats.journalfile_deletions;
236 +
237 + void *journal_data = GET_JOURNAL_DATA(journalfile);
238 + size_t journal_data_size = GET_JOURNAL_DATA_SIZE(journalfile);
239 +
240 + if (journal_data) {
241 + if (munmap(journal_data, journal_data_size)) {
242 + error("DBENGINE: failed to unmap index file %s", path_v2);
243 + }
244 + }
245
246 return ret;
247 }
@@ -209,7 +268,7 @@ int create_journal_file(struct rrdengine_journalfile *journalfile, struct rrdeng
268
269 ret = posix_memalign((void *)&superblock, RRDFILE_ALIGNMENT, sizeof(*superblock));
270 if (unlikely(ret)) {
212 - fatal("posix_memalign:%s", strerror(ret));
271 + fatal("DBENGINE: posix_memalign:%s", strerror(ret));
272 }
273 memset(superblock, 0, sizeof(*superblock));
274 (void) strncpy(superblock->magic_number, RRDENG_JF_MAGIC, RRDENG_MAGIC_SZ);
@@ -220,14 +279,14 @@ int create_journal_file(struct rrdengine_journalfile *journalfile, struct rrdeng
279 ret = uv_fs_write(NULL, &req, file, &iov, 1, 0, NULL);
280 if (ret < 0) {
281 fatal_assert(req.result < 0);
223 - error("uv_fs_write: %s", uv_strerror(ret));
282 + error("DBENGINE: uv_fs_write: %s", uv_strerror(ret));
283 ++ctx->stats.io_errors;
284 rrd_stat_atomic_add(&global_io_errors, 1);
285 }
286 uv_fs_req_cleanup(&req);
287 posix_memfree(superblock);
288 if (ret < 0) {
230 - destroy_journal_file(journalfile, datafile);
289 + destroy_journal_file_unsafe(journalfile, datafile);
290 return ret;
291 }
292
@@ -247,13 +306,13 @@ static int check_journal_file_superblock(uv_file file)
306
307 ret = posix_memalign((void *)&superblock, RRDFILE_ALIGNMENT, sizeof(*superblock));
308 if (unlikely(ret)) {
250 - fatal("posix_memalign:%s", strerror(ret));
309 + fatal("DBENGINE: posix_memalign:%s", strerror(ret));
310 }
311 iov = uv_buf_init((void *)superblock, sizeof(*superblock));
312
313 ret = uv_fs_read(NULL, &req, file, &iov, 1, 0, NULL);
314 if (ret < 0) {
256 - error("uv_fs_read: %s", uv_strerror(ret));
315 + error("DBENGINE: uv_fs_read: %s", uv_strerror(ret));
316 uv_fs_req_cleanup(&req);
317 goto error;
318 }
@@ -262,7 +321,7 @@ static int check_journal_file_superblock(uv_file file)
321
322 if (strncmp(superblock->magic_number, RRDENG_JF_MAGIC, RRDENG_MAGIC_SZ) ||
323 strncmp(superblock->version, RRDENG_JF_VER, RRDENG_VER_SZ)) {
265 - error("File has invalid superblock.");
324 + error("DBENGINE: File has invalid superblock.");
325 ret = UV_EINVAL;
326 } else {
327 ret = 0;
@@ -272,15 +331,10 @@ static int check_journal_file_superblock(uv_file file)
331 return ret;
332 }
333
275 -static void restore_extent_metadata(struct rrdengine_instance *ctx, struct rrdengine_journalfile *journalfile,
276 - void *buf, unsigned max_size)
334 +static void restore_extent_metadata(struct rrdengine_instance *ctx, struct rrdengine_journalfile *journalfile, void *buf, unsigned max_size)
335 {
336 static BITMAP256 page_error_map;
279 - struct page_cache *pg_cache = &ctx->pg_cache;
280 - unsigned i, count, payload_length, descr_size, valid_pages;
281 - struct rrdeng_page_descr *descr;
282 - struct extent_info *extent;
283 - /* persistent structures */
337 + unsigned i, count, payload_length, descr_size;
338 struct rrdeng_jf_store_data *jf_metric_data;
339
340 jf_metric_data = buf;
@@ -288,117 +342,63 @@ static void restore_extent_metadata(struct rrdengine_instance *ctx, struct rrden
342 descr_size = sizeof(*jf_metric_data->descr) * count;
343 payload_length = sizeof(*jf_metric_data) + descr_size;
344 if (payload_length > max_size) {
291 - error("Corrupted transaction payload.");
345 + error("DBENGINE: corrupted transaction payload.");
346 return;
347 }
348
295 - extent = mallocz(sizeof(*extent) + count * sizeof(extent->pages[0]));
296 - extent->offset = jf_metric_data->extent_offset;
297 - extent->size = jf_metric_data->extent_size;
298 - extent->datafile = journalfile->datafile;
299 - extent->next = NULL;
300 -
301 - for (i = 0, valid_pages = 0 ; i < count ; ++i) {
349 + time_t now_s = now_realtime_sec();
350 + for (i = 0; i < count ; ++i) {
351 uuid_t *temp_id;
303 - Pvoid_t *PValue;
304 - struct pg_cache_page_index *page_index = NULL;
352 uint8_t page_type = jf_metric_data->descr[i].type;
353
354 if (page_type > PAGE_TYPE_MAX) {
355 if (!bitmap256_get_bit(&page_error_map, page_type)) {
309 - error("Unknown page type %d encountered.", page_type);
356 + error("DBENGINE: unknown page type %d encountered.", page_type);
357 bitmap256_set_bit(&page_error_map, page_type, 1);
358 }
359 continue;
360 }
314 - uint64_t start_time_ut = jf_metric_data->descr[i].start_time_ut;
315 - uint64_t end_time_ut = jf_metric_data->descr[i].end_time_ut;
316 - size_t entries = jf_metric_data->descr[i].page_length / page_type_size[page_type];
317 - time_t update_every_s = (entries > 1) ? ((end_time_ut - start_time_ut) / USEC_PER_SEC / (entries - 1)) : 0;
318 -
319 - if (unlikely(start_time_ut > end_time_ut)) {
320 - ctx->load_errors[LOAD_ERRORS_PAGE_FLIPPED_TIME].counter++;
321 - if(ctx->load_errors[LOAD_ERRORS_PAGE_FLIPPED_TIME].latest_end_time_ut < end_time_ut)
322 - ctx->load_errors[LOAD_ERRORS_PAGE_FLIPPED_TIME].latest_end_time_ut = end_time_ut;
323 - continue;
324 - }
361
326 - if (unlikely(start_time_ut == end_time_ut && entries != 1)) {
327 - ctx->load_errors[LOAD_ERRORS_PAGE_EQUAL_TIME].counter++;
328 - if(ctx->load_errors[LOAD_ERRORS_PAGE_EQUAL_TIME].latest_end_time_ut < end_time_ut)
329 - ctx->load_errors[LOAD_ERRORS_PAGE_EQUAL_TIME].latest_end_time_ut = end_time_ut;
330 - continue;
331 - }
362 + temp_id = (uuid_t *)jf_metric_data->descr[i].uuid;
363 + METRIC *metric = mrg_metric_get_and_acquire(main_mrg, temp_id, (Word_t) ctx);
364
333 - if (unlikely(!entries)) {
334 - ctx->load_errors[LOAD_ERRORS_PAGE_ZERO_ENTRIES].counter++;
335 - if(ctx->load_errors[LOAD_ERRORS_PAGE_ZERO_ENTRIES].latest_end_time_ut < end_time_ut)
336 - ctx->load_errors[LOAD_ERRORS_PAGE_ZERO_ENTRIES].latest_end_time_ut = end_time_ut;
337 - continue;
338 - }
365 + struct rrdeng_extent_page_descr *descr = &jf_metric_data->descr[i];
366 + VALIDATED_PAGE_DESCRIPTOR vd = validate_extent_page_descr(
367 + descr, now_s,
368 + (metric) ? mrg_metric_get_update_every_s(main_mrg, metric) : 0,
369 + false);
370
340 - if(entries > 1 && update_every_s == 0) {
341 - ctx->load_errors[LOAD_ERRORS_PAGE_UPDATE_ZERO].counter++;
342 - if(ctx->load_errors[LOAD_ERRORS_PAGE_UPDATE_ZERO].latest_end_time_ut < end_time_ut)
343 - ctx->load_errors[LOAD_ERRORS_PAGE_UPDATE_ZERO].latest_end_time_ut = end_time_ut;
371 + if(!vd.data_on_disk_valid) {
372 + mrg_metric_release(main_mrg, metric);
373 continue;
374 }
375
347 - if(start_time_ut + update_every_s * USEC_PER_SEC * (entries - 1) != end_time_ut) {
348 - ctx->load_errors[LOAD_ERRORS_PAGE_FLEXY_TIME].counter++;
349 - if(ctx->load_errors[LOAD_ERRORS_PAGE_FLEXY_TIME].latest_end_time_ut < end_time_ut)
350 - ctx->load_errors[LOAD_ERRORS_PAGE_FLEXY_TIME].latest_end_time_ut = end_time_ut;
351 -
352 - // let this be
353 - // end_time_ut = start_time_ut + update_every_s * USEC_PER_SEC * (entries - 1);
376 + bool update_metric_time = true;
377 + if (!metric) {
378 + MRG_ENTRY entry = {
379 + .section = (Word_t)ctx,
380 + .first_time_s = vd.start_time_s,
381 + .last_time_s = vd.end_time_s,
382 + .latest_update_every_s = vd.update_every_s,
383 + };
384 + uuid_copy(entry.uuid, *temp_id);
385 +
386 + bool added;
387 + metric = mrg_metric_add_and_acquire(main_mrg, entry, &added);
388 + if(added)
389 + update_metric_time = false;
390 }
391 + Word_t metric_id = mrg_metric_id(main_mrg, metric);
392
356 - temp_id = (uuid_t *)jf_metric_data->descr[i].uuid;
357 -
358 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
359 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, temp_id, sizeof(uuid_t));
360 - if (likely(NULL != PValue)) {
361 - page_index = *PValue;
362 - }
363 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
364 - if (NULL == PValue) {
365 - /* First time we see the UUID */
366 - uv_rwlock_wrlock(&pg_cache->metrics_index.lock);
367 - PValue = JudyHSIns(&pg_cache->metrics_index.JudyHS_array, temp_id, sizeof(uuid_t), PJE0);
368 - fatal_assert(NULL == *PValue); /* TODO: figure out concurrency model */
369 - *PValue = page_index = create_page_index(temp_id, ctx);
370 - page_index->prev = pg_cache->metrics_index.last_page_index;
371 - pg_cache->metrics_index.last_page_index = page_index;
372 - uv_rwlock_wrunlock(&pg_cache->metrics_index.lock);
373 - }
374 -
375 - descr = pg_cache_create_descr();
376 - descr->page_length = jf_metric_data->descr[i].page_length;
377 - descr->start_time_ut = start_time_ut;
378 - descr->end_time_ut = end_time_ut;
379 - descr->update_every_s = (update_every_s > 0) ? (uint32_t)update_every_s : (page_index->latest_update_every_s);
380 - descr->id = &page_index->id;
381 - descr->extent = extent;
382 - descr->type = page_type;
383 - extent->pages[valid_pages++] = descr;
384 - pg_cache_insert(ctx, page_index, descr);
385 -
386 - if(page_index->latest_time_ut == descr->end_time_ut)
387 - page_index->latest_update_every_s = descr->update_every_s;
388 -
389 - if(descr->update_every_s == 0)
390 - fatal(
391 - "DBENGINE: page descriptor update every is zero, end_time_ut = %llu, start_time_ut = %llu, entries = %zu",
392 - (unsigned long long)end_time_ut, (unsigned long long)start_time_ut, entries);
393 - }
393 + if (update_metric_time)
394 + mrg_metric_expand_retention(main_mrg, metric, vd.start_time_s, vd.end_time_s, vd.update_every_s);
395
395 - extent->number_of_pages = valid_pages;
396 + pgc_open_add_hot_page(
397 + (Word_t)ctx, metric_id, vd.start_time_s, vd.end_time_s, vd.update_every_s,
398 + journalfile->datafile,
399 + jf_metric_data->extent_offset, jf_metric_data->extent_size, jf_metric_data->descr[i].page_length);
400
397 - if (likely(valid_pages))
398 - df_extent_insert(extent);
399 - else {
400 - freez(extent);
401 - ctx->load_errors[LOAD_ERRORS_DROPPED_EXTENT].counter++;
401 + mrg_metric_release(main_mrg, metric);
402 }
403 }
404
@@ -424,14 +424,14 @@ static unsigned replay_transaction(struct rrdengine_instance *ctx, struct rrdeng
424 return 0;
425 }
426 if (sizeof(*jf_header) > max_size) {
427 - error("Corrupted transaction record, skipping.");
427 + error("DBENGINE: corrupted transaction record, skipping.");
428 return 0;
429 }
430 *id = jf_header->id;
431 payload_length = jf_header->payload_length;
432 size_bytes = sizeof(*jf_header) + payload_length + sizeof(*jf_trailer);
433 if (size_bytes > max_size) {
434 - error("Corrupted transaction record, skipping.");
434 + error("DBENGINE: corrupted transaction record, skipping.");
435 return 0;
436 }
437 jf_trailer = buf + sizeof(*jf_header) + payload_length;
@@ -440,7 +440,7 @@ static unsigned replay_transaction(struct rrdengine_instance *ctx, struct rrdeng
440 ret = crc32cmp(jf_trailer->checksum, crc);
441 debug(D_RRDENGINE, "Transaction %"PRIu64" was read from disk. CRC32 check: %s", *id, ret ? "FAILED" : "SUCCEEDED");
442 if (unlikely(ret)) {
443 - error("Transaction %"PRIu64" was read from disk. CRC32 check: FAILED", *id);
443 + error("DBENGINE: transaction %"PRIu64" was read from disk. CRC32 check: FAILED", *id);
444 return size_bytes;
445 }
446 switch (jf_header->type) {
@@ -449,7 +449,7 @@ static unsigned replay_transaction(struct rrdengine_instance *ctx, struct rrdeng
449 restore_extent_metadata(ctx, journalfile, buf + sizeof(*jf_header), payload_length);
450 break;
451 default:
452 - error("Unknown transaction type. Skipping record.");
452 + error("DBENGINE: unknown transaction type, skipping record.");
453 break;
454 }
455
@@ -483,7 +483,7 @@ static uint64_t iterate_transactions(struct rrdengine_instance *ctx, struct rrde
483 if (unlikely(!journal_is_mmapped)) {
484 ret = posix_memalign((void *)&buf, RRDFILE_ALIGNMENT, READAHEAD_BYTES);
485 if (unlikely(ret))
486 - fatal("posix_memalign:%s", strerror(ret));
486 + fatal("DBENGINE: posix_memalign:%s", strerror(ret));
487 }
488 else
489 buf = journalfile->data + sizeof(struct rrdeng_jf_sb);
@@ -493,7 +493,7 @@ static uint64_t iterate_transactions(struct rrdengine_instance *ctx, struct rrde
493 iov = uv_buf_init(buf, size_bytes);
494 ret = uv_fs_read(NULL, &req, file, &iov, 1, pos, NULL);
495 if (ret < 0) {
496 - error("uv_fs_read: pos=%" PRIu64 ", %s", pos, uv_strerror(ret));
496 + error("DBENGINE: uv_fs_read: pos=%" PRIu64 ", %s", pos, uv_strerror(ret));
497 uv_fs_req_cleanup(&req);
498 goto skip_file;
499 }
@@ -524,6 +524,609 @@ skip_file:
524 return max_id;
525 }
526
527 +// Checks that the extent list checksum is valid
528 +static int check_journal_v2_extent_list (void *data_start, size_t file_size)
529 +{
530 + UNUSED(file_size);
531 + uLong crc;
532 +
533 + struct journal_v2_header *j2_header = (void *) data_start;
534 + struct journal_v2_block_trailer *journal_v2_trailer;
535 +
536 + journal_v2_trailer = (struct journal_v2_block_trailer *) ((uint8_t *) data_start + j2_header->extent_trailer_offset);
537 + crc = crc32(0L, Z_NULL, 0);
538 + crc = crc32(crc, (uint8_t *) data_start + j2_header->extent_offset, j2_header->extent_count * sizeof(struct journal_extent_list));
539 + if (unlikely(crc32cmp(journal_v2_trailer->checksum, crc))) {
540 + error("DBENGINE: extent list CRC32 check: FAILED");
541 + return 1;
542 + }
543 +
544 + return 0;
545 +}
546 +
547 +// Checks that the metric list (UUIDs) checksum is valid
548 +static int check_journal_v2_metric_list(void *data_start, size_t file_size)
549 +{
550 + UNUSED(file_size);
551 + uLong crc;
552 +
553 + struct journal_v2_header *j2_header = (void *) data_start;
554 + struct journal_v2_block_trailer *journal_v2_trailer;
555 +
556 + journal_v2_trailer = (struct journal_v2_block_trailer *) ((uint8_t *) data_start + j2_header->metric_trailer_offset);
557 + crc = crc32(0L, Z_NULL, 0);
558 + crc = crc32(crc, (uint8_t *) data_start + j2_header->metric_offset, j2_header->metric_count * sizeof(struct journal_metric_list));
559 + if (unlikely(crc32cmp(journal_v2_trailer->checksum, crc))) {
560 + error("DBENGINE: metric list CRC32 check: FAILED");
561 + return 1;
562 + }
563 + return 0;
564 +}
565 +
566 +//
567 +// Return
568 +// 0 Ok
569 +// 1 Invalid
570 +// 2 Force rebuild
571 +// 3 skip
572 +
573 +static int check_journal_v2_file(void *data_start, size_t file_size, uint32_t original_size)
574 +{
575 + int rc;
576 + uLong crc;
577 +
578 + struct journal_v2_header *j2_header = (void *) data_start;
579 + struct journal_v2_block_trailer *journal_v2_trailer;
580 +
581 + if (j2_header->magic == JOURVAL_V2_REBUILD_MAGIC)
582 + return 2;
583 +
584 + if (j2_header->magic == JOURVAL_V2_SKIP_MAGIC)
585 + return 3;
586 +
587 + // Magic failure
588 + if (j2_header->magic != JOURVAL_V2_MAGIC)
589 + return 1;
590 +
591 + if (j2_header->total_file_size != file_size)
592 + return 1;
593 +
594 + if (original_size && j2_header->original_file_size != original_size)
595 + return 1;
596 +
597 + journal_v2_trailer = (struct journal_v2_block_trailer *) ((uint8_t *) data_start + file_size - sizeof(*journal_v2_trailer));
598 +
599 + crc = crc32(0L, Z_NULL, 0);
600 + crc = crc32(crc, (void *) j2_header, sizeof(*j2_header));
601 +
602 + rc = crc32cmp(journal_v2_trailer->checksum, crc);
603 + if (unlikely(rc)) {
604 + error("DBENGINE: file CRC32 check: FAILED");
605 + return 1;
606 + }
607 +
608 + rc = check_journal_v2_extent_list(data_start, file_size);
609 + if (rc) return 1;
610 +
611 + rc = check_journal_v2_metric_list(data_start, file_size);
612 + if (rc) return 1;
613 +
614 + if (!db_engine_journal_check)
615 + return 0;
616 +
617 + // Verify complete UUID chain
618 +
619 + struct journal_metric_list *metric = (void *) (data_start + j2_header->metric_offset);
620 +
621 + unsigned verified = 0;
622 + unsigned entries;
623 + unsigned total_pages = 0;
624 +
625 + info("DBENGINE: checking %u metrics that exist in the journal", j2_header->metric_count);
626 + for (entries = 0; entries < j2_header->metric_count; entries++) {
627 +
628 + char uuid_str[UUID_STR_LEN];
629 + uuid_unparse_lower(metric->uuid, uuid_str);
630 + struct journal_page_header *metric_list_header = (void *) (data_start + metric->page_offset);
631 + struct journal_page_header local_metric_list_header = *metric_list_header;
632 +
633 + local_metric_list_header.crc = JOURVAL_V2_MAGIC;
634 +
635 + crc = crc32(0L, Z_NULL, 0);
636 + crc = crc32(crc, (void *) &local_metric_list_header, sizeof(local_metric_list_header));
637 + rc = crc32cmp(metric_list_header->checksum, crc);
638 +
639 + if (!rc) {
640 + struct journal_v2_block_trailer *journal_trailer =
641 + (void *) data_start + metric->page_offset + sizeof(struct journal_page_header) + (metric_list_header->entries * sizeof(struct journal_page_list));
642 +
643 + crc = crc32(0L, Z_NULL, 0);
644 + crc = crc32(crc, (uint8_t *) metric_list_header + sizeof(struct journal_page_header), metric_list_header->entries * sizeof(struct journal_page_list));
645 + rc = crc32cmp(journal_trailer->checksum, crc);
646 + internal_error(rc, "DBENGINE: index %u : %s entries %u at offset %u verified, DATA CRC computed %lu, stored %u", entries, uuid_str, metric->entries, metric->page_offset,
647 + crc, metric_list_header->crc);
648 + if (!rc) {
649 + total_pages += metric_list_header->entries;
650 + verified++;
651 + }
652 + }
653 +
654 + metric++;
655 + if (((uint8_t *) metric - (uint8_t *) data_start) > (uint32_t) file_size) {
656 + info("DBENGINE: verification failed EOF reached -- total entries %u, verified %u", entries, verified);
657 + return 1;
658 + }
659 + }
660 +
661 + if (entries != verified) {
662 + info("DBENGINE: verification failed -- total entries %u, verified %u", entries, verified);
663 + return 1;
664 + }
665 + info("DBENGINE: verification succeeded -- total entries %u, verified %u (%u total pages)", entries, verified, total_pages);
666 +
667 + return 0;
668 +}
669 +
670 +int load_journal_file_v2(struct rrdengine_instance *ctx, struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
671 +{
672 + int ret, fd;
673 + uint64_t file_size;
674 + char path[RRDENG_PATH_MAX];
675 + struct stat statbuf;
676 + uint32_t original_file_size = 0;
677 +
678 + generate_journalfilepath(datafile, path, sizeof(path));
679 + ret = stat(path, &statbuf);
680 + if (!ret)
681 + original_file_size = (uint32_t)statbuf.st_size;
682 +
683 + generate_journalfilepath_v2(datafile, path, sizeof(path));
684 +
685 + fd = open(path, O_RDONLY);
686 + if (fd < 0) {
687 + if (errno == ENOENT)
688 + return 1;
689 + ++ctx->stats.fs_errors;
690 + rrd_stat_atomic_add(&global_fs_errors, 1);
691 + error("DBENGINE: failed to open '%s'", path);
692 + return 1;
693 + }
694 +
695 + ret = fstat(fd, &statbuf);
696 + if (ret) {
697 + error("DBENGINE: failed to get file information for '%s'", path);
698 + close(fd);
699 + return 1;
700 + }
701 +
702 + file_size = (size_t)statbuf.st_size;
703 +
704 + if (file_size < sizeof(struct journal_v2_header)) {
705 + error_report("Invalid file %s. Not the expected size", path);
706 + close(fd);
707 + return 1;
708 + }
709 +
710 + usec_t start_loading = now_realtime_usec();
711 + uint8_t *data_start = mmap(NULL, file_size, PROT_READ, MAP_SHARED, fd, 0);
712 + if (data_start == MAP_FAILED) {
713 + close(fd);
714 + return 1;
715 + }
716 + close(fd);
717 +
718 + info("DBENGINE: checking integrity of '%s'", path);
719 + int rc = check_journal_v2_file(data_start, file_size, original_file_size);
720 + if (unlikely(rc)) {
721 + if (rc == 2)
722 + error_report("File %s needs to be rebuilt", path);
723 + else if (rc == 3)
724 + error_report("File %s will be skipped", path);
725 + else
726 + error_report("File %s is invalid and it will be rebuilt", path);
727 +
728 + if (unlikely(munmap(data_start, file_size)))
729 + error("DBENGINE: failed to unmap '%s'", path);
730 +
731 + return rc;
732 + }
733 +
734 + struct journal_v2_header *j2_header = (void *) data_start;
735 + uint32_t entries = j2_header->metric_count;
736 +
737 + if (unlikely(!entries)) {
738 + if (unlikely(munmap(data_start, file_size)))
739 + error("DBENGINE: failed to unmap '%s'", path);
740 +
741 + return 1;
742 + }
743 +
744 + madvise_dontfork(data_start, file_size);
745 + madvise_dontdump(data_start, file_size);
746 +
747 + struct journal_metric_list *metric = (struct journal_metric_list *) (data_start + j2_header->metric_offset);
748 +
749 + // Initialize the journal file to be able to access the data
750 + SET_JOURNAL_DATA(journalfile, data_start);
751 + SET_JOURNAL_DATA_SIZE(journalfile, file_size);
752 +
753 + time_t header_start_time_s = (time_t) (j2_header->start_time_ut / USEC_PER_SEC);
754 +
755 + time_t now_s = now_realtime_sec();
756 + for (size_t i=0; i < entries; i++) {
757 + time_t start_time_s = header_start_time_s + metric->delta_start_s;
758 + time_t end_time_s = header_start_time_s + metric->delta_end_s;
759 + time_t update_every_s = (metric->entries > 1) ? ((end_time_s - start_time_s) / (entries - 1)) : 0;
760 + update_metric_retention_and_granularity_by_uuid(
761 + ctx, &metric->uuid, start_time_s, end_time_s, update_every_s, now_s);
762 +
763 +#ifdef NETDATA_INTERNAL_CHECKS
764 + struct journal_page_header *metric_list_header = (void *) (data_start + metric->page_offset);
765 + fatal_assert(uuid_compare(metric_list_header->uuid, metric->uuid) == 0);
766 + fatal_assert(metric->entries == metric_list_header->entries);
767 +#endif
768 + metric++;
769 + }
770 +
771 + info("DBENGINE: journal file '%s' loaded (size:%"PRIu64") with %u metrics in %d ms", path, file_size, entries,
772 + (int) ((now_realtime_usec() - start_loading) / USEC_PER_MS));
773 +
774 + // File is OK load it
775 + return 0;
776 +}
777 +
778 +struct journal_metric_list_to_sort {
779 + struct jv2_metrics_info *metric_info;
780 +};
781 +
782 +static int journal_metric_compare (const void *item1, const void *item2)
783 +{
784 + const struct jv2_metrics_info *metric1 = ((struct journal_metric_list_to_sort *) item1)->metric_info;
785 + const struct jv2_metrics_info *metric2 = ((struct journal_metric_list_to_sort *) item2)->metric_info;
786 +
787 + return uuid_compare(*(metric1->uuid), *(metric2->uuid));
788 +}
789 +
790 +
791 +// Write list of extents for the journalfile
792 +void *journal_v2_write_extent_list(Pvoid_t JudyL_extents_pos, void *data)
793 +{
794 + Pvoid_t *PValue;
795 + struct journal_extent_list *j2_extent_base = (void *) data;
796 + struct jv2_extents_info *ext_info;
797 +
798 + bool first = true;
799 + Word_t pos = 0;
800 + size_t count = 0;
801 + while ((PValue = JudyLFirstThenNext(JudyL_extents_pos, &pos, &first))) {
802 + ext_info = *PValue;
803 + size_t index = ext_info->index;
804 + j2_extent_base[index].file_index = 0;
805 + j2_extent_base[index].datafile_offset = ext_info->pos;
806 + j2_extent_base[index].datafile_size = ext_info->bytes;
807 + j2_extent_base[index].pages = ext_info->number_of_pages;
808 + count++;
809 + }
810 + return j2_extent_base + count;
811 +}
812 +
813 +static int verify_journal_space(struct journal_v2_header *j2_header, void *data, uint32_t bytes)
814 +{
815 + if ((unsigned long)(((uint8_t *) data - (uint8_t *) j2_header->data) + bytes) > (j2_header->total_file_size - sizeof(struct journal_v2_block_trailer)))
816 + return 1;
817 +
818 + return 0;
819 +}
820 +
821 +void *journal_v2_write_metric_page(struct journal_v2_header *j2_header, void *data, struct jv2_metrics_info *metric_info, uint32_t pages_offset)
822 +{
823 + struct journal_metric_list *metric = (void *) data;
824 +
825 + if (verify_journal_space(j2_header, data, sizeof(*metric)))
826 + return NULL;
827 +
828 + uuid_copy(metric->uuid, *metric_info->uuid);
829 + metric->entries = metric_info->number_of_pages;
830 + metric->page_offset = pages_offset;
831 + metric->delta_start_s = (uint32_t)(metric_info->first_time_s - (time_t)(j2_header->start_time_ut / USEC_PER_SEC));
832 + metric->delta_end_s = (uint32_t)(metric_info->last_time_s - (time_t)(j2_header->start_time_ut / USEC_PER_SEC));
833 +
834 + return ++metric;
835 +}
836 +
837 +void *journal_v2_write_data_page_header(struct journal_v2_header *j2_header __maybe_unused, void *data, struct jv2_metrics_info *metric_info, uint32_t uuid_offset)
838 +{
839 + struct journal_page_header *data_page_header = (void *) data;
840 + uLong crc;
841 +
842 + uuid_copy(data_page_header->uuid, *metric_info->uuid);
843 + data_page_header->entries = metric_info->number_of_pages;
844 + data_page_header->uuid_offset = uuid_offset; // data header OFFSET poings to METRIC in the directory
845 + data_page_header->crc = JOURVAL_V2_MAGIC;
846 + crc = crc32(0L, Z_NULL, 0);
847 + crc = crc32(crc, (void *) data_page_header, sizeof(*data_page_header));
848 + crc32set(data_page_header->checksum, crc);
849 + return ++data_page_header;
850 +}
851 +
852 +void *journal_v2_write_data_page_trailer(struct journal_v2_header *j2_header __maybe_unused, void *data, void *page_header)
853 +{
854 + struct journal_page_header *data_page_header = (void *) page_header;
855 + struct journal_v2_block_trailer *journal_trailer = (void *) data;
856 + uLong crc;
857 +
858 + crc = crc32(0L, Z_NULL, 0);
859 + crc = crc32(crc, (uint8_t *) page_header + sizeof(struct journal_page_header), data_page_header->entries * sizeof(struct journal_page_list));
860 + crc32set(journal_trailer->checksum, crc);
861 + return ++journal_trailer;
862 +}
863 +
864 +void *journal_v2_write_data_page(struct journal_v2_header *j2_header, void *data, struct jv2_page_info *page_info)
865 +{
866 + struct journal_page_list *data_page = data;
867 +
868 + if (verify_journal_space(j2_header, data, sizeof(*data_page)))
869 + return NULL;
870 +
871 + struct extent_io_data *ei = page_info->custom_data;
872 +
873 + data_page->delta_start_s = (uint32_t) (page_info->start_time_s - (time_t) (j2_header->start_time_ut) / USEC_PER_SEC);
874 + data_page->delta_end_s = (uint32_t) (page_info->end_time_s - (time_t) (j2_header->start_time_ut) / USEC_PER_SEC);
875 + data_page->extent_index = page_info->extent_index;
876 +
877 + data_page->update_every_s = page_info->update_every_s;
878 + data_page->page_length = (uint16_t) (ei ? ei->page_length : page_info->page_length);
879 + data_page->type = 0;
880 +
881 + return ++data_page;
882 +}
883 +
884 +// Must be recorded in metric_info->entries
885 +void *journal_v2_write_descriptors(struct journal_v2_header *j2_header, void *data, struct jv2_metrics_info *metric_info)
886 +{
887 + Pvoid_t *PValue;
888 +
889 + struct journal_page_list *data_page = (void *)data;
890 + // We need to write all descriptors with index metric_info->min_index_time_s, metric_info->max_index_time_s
891 + // that belong to this journal file
892 + Pvoid_t JudyL_array = metric_info->JudyL_pages_by_start_time;
893 +
894 + Word_t index_time = 0;
895 + bool first = true;
896 + struct jv2_page_info *page_info;
897 + while ((PValue = JudyLFirstThenNext(JudyL_array, &index_time, &first))) {
898 + page_info = *PValue;
899 + // Write one descriptor and return the next data page location
900 + data_page = journal_v2_write_data_page(j2_header, (void *)data_page, page_info);
901 + if (NULL == data_page)
902 + break;
903 + }
904 + return data_page;
905 +}
906 +
907 +// Migrate the journalfile pointed by datafile
908 +// activate : make the new file active immediately
909 +// journafile data will be set and descriptors (if deleted) will be repopulated as needed
910 +// startup : if the migration is done during agent startup
911 +// this will allow us to optimize certain things
912 +
913 +void do_migrate_to_v2_callback(Word_t section, unsigned datafile_fileno __maybe_unused, uint8_t type __maybe_unused,
914 + Pvoid_t JudyL_metrics, Pvoid_t JudyL_extents_pos,
915 + size_t number_of_extents, size_t number_of_metrics, size_t number_of_pages, void *user_data)
916 +{
917 + char path[RRDENG_PATH_MAX];
918 + Pvoid_t *PValue;
919 + struct rrdengine_instance *ctx = (struct rrdengine_instance *) section;
920 + struct rrdengine_journalfile *journalfile = (struct rrdengine_journalfile *) user_data;
921 + struct rrdengine_datafile *datafile = journalfile->datafile;
922 + time_t min_time_s = LLONG_MAX;
923 + time_t max_time_s = 0;
924 + struct jv2_metrics_info *metric_info;
925 +
926 + generate_journalfilepath_v2(datafile, path, sizeof(path));
927 +
928 + info("DBENGINE: indexing file '%s': extents %lu, metrics %lu, pages %lu",
929 + path,
930 + number_of_extents,
931 + number_of_metrics,
932 + number_of_pages);
933 +
934 +#ifdef NETDATA_INTERNAL_CHECKS
935 + usec_t start_loading = now_realtime_usec();
936 +#endif
937 +
938 + size_t total_file_size = 0;
939 + total_file_size += (sizeof(struct journal_v2_header) + JOURNAL_V2_HEADER_PADDING_SZ);
940 +
941 + // Extents will start here
942 + uint32_t extent_offset = total_file_size;
943 + total_file_size += (number_of_extents * sizeof(struct journal_extent_list));
944 +
945 + uint32_t extent_offset_trailer = total_file_size;
946 + total_file_size += sizeof(struct journal_v2_block_trailer);
947 +
948 + // UUID list will start here
949 + uint32_t metrics_offset = total_file_size;
950 + total_file_size += (number_of_metrics * sizeof(struct journal_metric_list));
951 +
952 + // UUID list trailer
953 + uint32_t metric_offset_trailer = total_file_size;
954 + total_file_size += sizeof(struct journal_v2_block_trailer);
955 +
956 + // descr @ time will start here
957 + uint32_t pages_offset = total_file_size;
958 + total_file_size += (number_of_pages * (sizeof(struct journal_page_list) + sizeof(struct journal_page_header) + sizeof(struct journal_v2_block_trailer)));
959 +
960 + // File trailer
961 + uint32_t trailer_offset = total_file_size;
962 + total_file_size += sizeof(struct journal_v2_block_trailer);
963 +
964 + uint8_t *data_start = netdata_mmap(path, total_file_size, MAP_SHARED, 0, false);
965 + uint8_t *data = data_start;
966 +
967 + memset(data_start, 0, extent_offset);
968 +
969 + // Write header
970 + struct journal_v2_header j2_header;
971 + memset(&j2_header, 0, sizeof(j2_header));
972 +
973 + j2_header.magic = JOURVAL_V2_MAGIC;
974 + j2_header.start_time_ut = 0;
975 + j2_header.end_time_ut = 0;
976 + j2_header.extent_count = number_of_extents;
977 + j2_header.extent_offset = extent_offset;
978 + j2_header.metric_count = number_of_metrics;
979 + j2_header.metric_offset = metrics_offset;
980 + j2_header.page_count = number_of_pages;
981 + j2_header.page_offset = pages_offset;
982 + j2_header.extent_trailer_offset = extent_offset_trailer;
983 + j2_header.metric_trailer_offset = metric_offset_trailer;
984 + j2_header.total_file_size = total_file_size;
985 + j2_header.original_file_size = (uint32_t) journalfile->pos;
986 + j2_header.data = data_start; // Used during migration
987 +
988 + struct journal_v2_block_trailer *journal_v2_trailer;
989 +
990 + data = journal_v2_write_extent_list(JudyL_extents_pos, data_start + extent_offset);
991 + internal_error(true, "DBENGINE: write extent list so far %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
992 +
993 + fatal_assert(data == data_start + extent_offset_trailer);
994 +
995 + // Calculate CRC for extents
996 + journal_v2_trailer = (struct journal_v2_block_trailer *) (data_start + extent_offset_trailer);
997 + uLong crc;
998 + crc = crc32(0L, Z_NULL, 0);
999 + crc = crc32(crc, (uint8_t *) data_start + extent_offset, number_of_extents * sizeof(struct journal_extent_list));
1000 + crc32set(journal_v2_trailer->checksum, crc);
1001 +
1002 + internal_error(true, "DBENGINE: CALCULATE CRC FOR EXTENT %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1003 + // Skip the trailer, point to the metrics off
1004 + data += sizeof(struct journal_v2_block_trailer);
1005 +
1006 + // Sanity check -- we must be at the metrics_offset
1007 + fatal_assert(data == data_start + metrics_offset);
1008 +
1009 + // Allocate array to sort UUIDs and keep them sorted in the journal because we want to do binary search when we do lookups
1010 + struct journal_metric_list_to_sort *uuid_list = mallocz(number_of_metrics * sizeof(struct journal_metric_list_to_sort));
1011 +
1012 + Word_t Index = 0;
1013 + size_t count = 0;
1014 + bool first_then_next = true;
1015 + while ((PValue = JudyLFirstThenNext(JudyL_metrics, &Index, &first_then_next))) {
1016 + metric_info = *PValue;
1017 +
1018 + fatal_assert(count < number_of_metrics);
1019 + uuid_list[count++].metric_info = metric_info;
1020 + min_time_s = MIN(min_time_s, metric_info->first_time_s);
1021 + max_time_s = MAX(max_time_s, metric_info->last_time_s);
1022 + }
1023 +
1024 + // Store in the header
1025 + j2_header.start_time_ut = min_time_s * USEC_PER_SEC;
1026 + j2_header.end_time_ut = max_time_s * USEC_PER_SEC;
1027 +
1028 + qsort(&uuid_list[0], number_of_metrics, sizeof(struct journal_metric_list_to_sort), journal_metric_compare);
1029 + internal_error(true, "DBENGINE: traverse and qsort UUID %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1030 +
1031 + uint32_t resize_file_to = total_file_size;
1032 +
1033 + for (Index = 0; Index < number_of_metrics; Index++) {
1034 + metric_info = uuid_list[Index].metric_info;
1035 +
1036 + // Calculate current UUID offset from start of file. We will store this in the data page header
1037 + uint32_t uuid_offset = data - data_start;
1038 +
1039 + // Write the UUID we are processing
1040 + data = (void *) journal_v2_write_metric_page(&j2_header, data, metric_info, pages_offset);
1041 + if (unlikely(!data))
1042 + break;
1043 +
1044 + // Next we will write
1045 + // Header
1046 + // Detailed entries (descr @ time)
1047 + // Trailer (checksum)
1048 +
1049 + // Keep the page_list_header, to be used for migration when where agent is running
1050 + metric_info->page_list_header = pages_offset;
1051 + // Write page header
1052 + void *metric_page = journal_v2_write_data_page_header(&j2_header, data_start + pages_offset, metric_info, uuid_offset);
1053 +
1054 + // Start writing descr @ time
1055 + void *page_trailer = journal_v2_write_descriptors(&j2_header, metric_page, metric_info);
1056 + if (unlikely(!page_trailer))
1057 + break;
1058 +
1059 + // Trailer (checksum)
1060 + uint8_t *next_page_address = journal_v2_write_data_page_trailer(&j2_header, page_trailer, data_start + pages_offset);
1061 +
1062 + // Calculate start of the pages start for next descriptor
1063 + pages_offset += (metric_info->number_of_pages * (sizeof(struct journal_page_list)) + sizeof(struct journal_page_header) + sizeof(struct journal_v2_block_trailer));
1064 + // Verify we are at the right location
1065 + if (pages_offset != (next_page_address - data_start)) {
1066 + // make sure checks fail so that we abort
1067 + data = data_start;
1068 + break;
1069 + }
1070 + }
1071 +
1072 + if (data == data_start + metric_offset_trailer) {
1073 + internal_error(true, "DBENGINE: WRITE METRICS AND PAGES %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1074 +
1075 + // Calculate CRC for metrics
1076 + journal_v2_trailer = (struct journal_v2_block_trailer *)(data_start + metric_offset_trailer);
1077 + crc = crc32(0L, Z_NULL, 0);
1078 + crc =
1079 + crc32(crc, (uint8_t *)data_start + metrics_offset, number_of_metrics * sizeof(struct journal_metric_list));
1080 + crc32set(journal_v2_trailer->checksum, crc);
1081 + internal_error(true, "DBENGINE: CALCULATE CRC FOR UUIDs %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1082 +
1083 + // Prepare to write checksum for the file
1084 + j2_header.data = NULL;
1085 + journal_v2_trailer = (struct journal_v2_block_trailer *)(data_start + trailer_offset);
1086 + crc = crc32(0L, Z_NULL, 0);
1087 + crc = crc32(crc, (void *)&j2_header, sizeof(j2_header));
1088 + crc32set(journal_v2_trailer->checksum, crc);
1089 +
1090 + // Write header to the file
1091 + memcpy(data_start, &j2_header, sizeof(j2_header));
1092 +
1093 + internal_error(true, "DBENGINE: FILE COMPLETED --------> %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1094 +
1095 + info("DBENGINE: migrated journal file '%s', file size %lu", path, total_file_size);
1096 +
1097 + SET_JOURNAL_DATA(journalfile, data_start);
1098 + SET_JOURNAL_DATA_SIZE(journalfile, total_file_size);
1099 +
1100 + internal_error(true, "DBENGINE: ACTIVATING NEW INDEX JNL %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1101 + ctx->disk_space += total_file_size;
1102 + freez(uuid_list);
1103 + return;
1104 + }
1105 + else {
1106 + info("DBENGINE: failed to build index '%s', file will be skipped", path);
1107 + j2_header.data = NULL;
1108 + j2_header.magic = JOURVAL_V2_SKIP_MAGIC;
1109 + memcpy(data_start, &j2_header, sizeof(j2_header));
1110 + resize_file_to = sizeof(j2_header);
1111 + }
1112 +
1113 + netdata_munmap(data_start, total_file_size);
1114 + freez(uuid_list);
1115 +
1116 + if (likely(resize_file_to == total_file_size))
1117 + return;
1118 +
1119 + int ret = truncate(path, (long) resize_file_to);
1120 + if (ret < 0) {
1121 + ctx->disk_space += total_file_size;
1122 + ++ctx->stats.fs_errors;
1123 + rrd_stat_atomic_add(&global_fs_errors, 1);
1124 + error("DBENGINE: failed to resize file '%s'", path);
1125 + }
1126 + else
1127 + ctx->disk_space += sizeof(struct journal_v2_header);
1128 +}
1129 +
1130 int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfile *journalfile,
1131 struct rrdengine_datafile *datafile)
1132 {
@@ -533,14 +1136,21 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
1136 uint64_t file_size, max_id;
1137 char path[RRDENG_PATH_MAX];
1138
1139 + // Do not try to load the latest file (always rebuild and live migrate)
1140 + if (datafile->fileno != ctx->last_fileno) {
1141 + if (!load_journal_file_v2(ctx, journalfile, datafile))
1142 + return 0;
1143 + }
1144 +
1145 generate_journalfilepath(datafile, path, sizeof(path));
1146 +
1147 + // If it is not the last file, open read only
1148 fd = open_file_direct_io(path, O_RDWR, &file);
1149 if (fd < 0) {
1150 ++ctx->stats.fs_errors;
1151 rrd_stat_atomic_add(&global_fs_errors, 1);
1152 return fd;
1153 }
543 - info("Loading journal file \"%s\".", path);
1154
1155 ret = check_file_properties(file, &file_size, sizeof(struct rrdeng_df_sb));
1156 if (ret)
@@ -548,30 +1158,45 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
1158 file_size = ALIGN_BYTES_FLOOR(file_size);
1159
1160 ret = check_journal_file_superblock(file);
551 - if (ret)
1161 + if (ret) {
1162 + info("DBENGINE: invalid journal file '%s' ; superblock check failed.", path);
1163 goto error;
1164 + }
1165 ctx->stats.io_read_bytes += sizeof(struct rrdeng_jf_sb);
1166 ++ctx->stats.io_read_requests;
1167
1168 journalfile->file = file;
1169 journalfile->pos = file_size;
558 - journalfile->data = netdata_mmap(path, file_size, MAP_SHARED, 0);
559 - info("Loading journal file \"%s\" using %s.", path, journalfile->data?"MMAP":"uv_fs_read");
1170 +
1171 + journalfile->data = netdata_mmap(path, file_size, MAP_SHARED, 0, !(datafile->fileno == ctx->last_fileno));
1172 + info("DBENGINE: loading journal file '%s' using %s.", path, journalfile->data?"MMAP":"uv_fs_read");
1173
1174 max_id = iterate_transactions(ctx, journalfile);
1175
1176 ctx->commit_log.transaction_id = MAX(ctx->commit_log.transaction_id, max_id + 1);
1177
565 - info("Journal file \"%s\" loaded (size:%"PRIu64").", path, file_size);
1178 + info("DBENGINE: journal file '%s' loaded (size:%"PRIu64").", path, file_size);
1179 if (likely(journalfile->data))
1180 netdata_munmap(journalfile->data, file_size);
1181 +
1182 + bool is_last_file = (ctx->last_fileno == journalfile->datafile->fileno);
1183 + if (is_last_file && journalfile->datafile->pos <= rrdeng_target_data_file_size(ctx) / 3) {
1184 + ctx->create_new_datafile_pair = false;
1185 + return 0;
1186 + }
1187 +
1188 + pgc_open_cache_to_journal_v2(open_cache, (Word_t) ctx, (int) datafile->fileno, ctx->page_type, do_migrate_to_v2_callback, (void *) datafile->journalfile);
1189 +
1190 + if (is_last_file)
1191 + ctx->create_new_datafile_pair = true;
1192 +
1193 return 0;
1194
570 - error:
1195 +error:
1196 error = ret;
1197 ret = uv_fs_close(NULL, &req, file, NULL);
1198 if (ret < 0) {
574 - error("uv_fs_close(%s): %s", path, uv_strerror(ret));
1199 + error("DBENGINE: uv_fs_close(%s): %s", path, uv_strerror(ret));
1200 ++ctx->stats.fs_errors;
1201 rrd_stat_atomic_add(&global_fs_errors, 1);
1202 }
@@ -581,7 +1206,5 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
1206
1207 void init_commit_log(struct rrdengine_instance *ctx)
1208 {
584 - ctx->commit_log.buf = NULL;
585 - ctx->commit_log.buf_pos = 0;
1209 ctx->commit_log.transaction_id = 1;
1210 }
database/engine/journalfile.h
+88 -8
@@ -13,37 +13,117 @@ struct rrdengine_journalfile;
13
14 #define WALFILE_PREFIX "journalfile-"
15 #define WALFILE_EXTENSION ".njf"
16 +#define WALFILE_EXTENSION_V2 ".njfv2"
17
18 +#define is_descr_journal_v2(descr) ((descr)->extent_entry != NULL)
19
20 /* only one event loop is supported for now */
21 struct rrdengine_journalfile {
22 uv_file file;
23 uint64_t pos;
24 void *data;
25 + void *journal_data; // MMAPed file of journal v2
26 + uint32_t journal_data_size; // Total file size mapped
27 struct rrdengine_datafile *datafile;
28 };
29
30 +
31 +// Journal v2 structures
32 +
33 +#define JOURVAL_V2_MAGIC (0x01221019)
34 +#define JOURVAL_V2_REBUILD_MAGIC (0x00221019)
35 +#define JOURVAL_V2_SKIP_MAGIC (0x02221019)
36 +
37 +struct journal_v2_block_trailer {
38 + union {
39 + uint8_t checksum[CHECKSUM_SZ]; /* CRC32 */
40 + uint32_t crc;
41 + };
42 +};
43 +
44 +// Journal V2
45 +// 28 bytes
46 +struct journal_page_header {
47 + union {
48 + uint8_t checksum[CHECKSUM_SZ]; // CRC check
49 + uint32_t crc;
50 + };
51 + uint32_t uuid_offset; // Points back to the UUID list which should point here (UUIDs should much)
52 + uint32_t entries; // Entries
53 + uuid_t uuid; // Which UUID this is
54 +};
55 +
56 +// 20 bytes
57 +struct journal_page_list {
58 + uint32_t delta_start_s; // relative to the start time of journal
59 + uint32_t delta_end_s; // relative to delta_start
60 + uint32_t extent_index; // Index to the extent (extent list) (bytes from BASE)
61 + uint32_t update_every_s;
62 + uint16_t page_length;
63 + uint8_t type;
64 +};
65 +
66 +// UUID_LIST
67 +// 32 bytes
68 +struct journal_metric_list {
69 + uuid_t uuid;
70 + uint32_t entries; // Number of entries
71 + uint32_t page_offset; // OFFSET that contains entries * struct( journal_page_list )
72 + uint32_t delta_start_s; // Min time of metric
73 + uint32_t delta_end_s; // Max time of metric (to be used to populate page_index)
74 +};
75 +
76 +// 16 bytes
77 +struct journal_extent_list {
78 + uint64_t datafile_offset; // Datafile offset to find the extent
79 + uint32_t datafile_size; // Size of the extent
80 + uint16_t file_index; // which file index is this datafile[index]
81 + uint8_t pages; // number of pages (not all are necesssarily valid)
82 +};
83 +
84 +// 72 bytes
85 +struct journal_v2_header {
86 + uint32_t magic;
87 + usec_t start_time_ut; // Min start time of journal
88 + usec_t end_time_ut; // Maximum end time of journal
89 + uint32_t extent_count; // Count of extents
90 + uint32_t extent_offset;
91 + uint32_t metric_count; // Count of metrics (unique UUIDS)
92 + uint32_t metric_offset;
93 + uint32_t page_count; // Total count of pages (descriptors @ time)
94 + uint32_t page_offset;
95 + uint32_t extent_trailer_offset; // CRC for entent list
96 + uint32_t metric_trailer_offset; // CRC for metric list
97 + uint32_t original_file_size; // This is the original journal file
98 + uint32_t total_file_size; // This is the total file size
99 + void *data; // Used when building the index
100 +};
101 +
102 +#define JOURNAL_V2_HEADER_PADDING_SZ (RRDENG_BLOCK_SIZE - (sizeof(struct journal_v2_header)))
103 +
104 +
105 +
106 /* only one event loop is supported for now */
107 struct transaction_commit_log {
108 uint64_t transaction_id;
29 -
30 - /* outstanding transaction buffer */
31 - void *buf;
32 - unsigned buf_pos;
33 - unsigned buf_size;
109 };
110
111 +struct wal;
112 +
113 void generate_journalfilepath(struct rrdengine_datafile *datafile, char *str, size_t maxlen);
114 +void generate_journalfilepath_v2(struct rrdengine_datafile *datafile, char *str, size_t maxlen);
115 void journalfile_init(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
38 -void *wal_get_transaction_buffer(struct rrdengine_worker_config* wc, unsigned size);
39 -void wal_flush_transaction_buffer(struct rrdengine_worker_config* wc);
116 +void wal_flush_transaction_buffer(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile, struct wal *wal, uv_loop_t *loop);
117 int close_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
118 int unlink_journal_file(struct rrdengine_journalfile *journalfile);
42 -int destroy_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
119 +int destroy_journal_file_unsafe(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
120 int create_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
121 int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfile *journalfile,
122 struct rrdengine_datafile *datafile);
123 void init_commit_log(struct rrdengine_instance *ctx);
124
125 +void do_migrate_to_v2_callback(Word_t section, unsigned datafile_fileno __maybe_unused, uint8_t type __maybe_unused,
126 + Pvoid_t JudyL_metrics, Pvoid_t JudyL_extents_pos,
127 + size_t number_of_extents, size_t number_of_metrics, size_t number_of_pages, void *user_data);
128
129 #endif /* NETDATA_JOURNALFILE_H */
\ No newline at end of file
database/engine/metric.c new
+643
@@ -0,0 +1,643 @@
1 +#include "metric.h"
2 +
3 +typedef int32_t REFCOUNT;
4 +#define REFCOUNT_DELETING (-100)
5 +
6 +struct metric {
7 + uuid_t uuid; // never changes
8 + Word_t section; // never changes
9 + time_t first_time_s; //
10 + time_t latest_time_s_clean; // archived pages latest time
11 + time_t latest_time_s_hot; // latest time of the currently collected page
12 + uint32_t latest_update_every_s; //
13 + SPINLOCK timestamps_lock; // protects the 3 timestamps
14 +
15 + // THIS IS allocated with malloc()
16 + // YOU HAVE TO INITIALIZE IT YOURSELF !
17 +};
18 +
19 +struct mrg {
20 + struct pgc_index {
21 + ARAL *aral;
22 + netdata_rwlock_t rwlock;
23 + Pvoid_t uuid_judy; // each UUID has a JudyL of sections (tiers)
24 + } index;
25 +
26 + struct mrg_statistics stats;
27 +};
28 +
29 +static inline void MRG_STATS_DUPLICATE_ADD(MRG *mrg) {
30 + __atomic_add_fetch(&mrg->stats.additions_duplicate, 1, __ATOMIC_RELAXED);
31 +}
32 +
33 +static inline void MRG_STATS_ADDED_METRIC(MRG *mrg) {
34 + __atomic_add_fetch(&mrg->stats.entries, 1, __ATOMIC_RELAXED);
35 + __atomic_add_fetch(&mrg->stats.additions, 1, __ATOMIC_RELAXED);
36 + __atomic_add_fetch(&mrg->stats.size, sizeof(METRIC), __ATOMIC_RELAXED);
37 +}
38 +
39 +static inline void MRG_STATS_DELETED_METRIC(MRG *mrg) {
40 + __atomic_sub_fetch(&mrg->stats.entries, 1, __ATOMIC_RELAXED);
41 + __atomic_sub_fetch(&mrg->stats.size, sizeof(METRIC), __ATOMIC_RELAXED);
42 + __atomic_add_fetch(&mrg->stats.deletions, 1, __ATOMIC_RELAXED);
43 +}
44 +
45 +static inline void MRG_STATS_SEARCH_HIT(MRG *mrg) {
46 + __atomic_add_fetch(&mrg->stats.search_hits, 1, __ATOMIC_RELAXED);
47 +}
48 +
49 +static inline void MRG_STATS_SEARCH_MISS(MRG *mrg) {
50 + __atomic_add_fetch(&mrg->stats.search_misses, 1, __ATOMIC_RELAXED);
51 +}
52 +
53 +static inline void MRG_STATS_DELETE_MISS(MRG *mrg) {
54 + __atomic_add_fetch(&mrg->stats.delete_misses, 1, __ATOMIC_RELAXED);
55 +}
56 +
57 +static void mrg_index_read_lock(MRG *mrg) {
58 + netdata_rwlock_rdlock(&mrg->index.rwlock);
59 +}
60 +static void mrg_index_read_unlock(MRG *mrg) {
61 + netdata_rwlock_unlock(&mrg->index.rwlock);
62 +}
63 +static void mrg_index_write_lock(MRG *mrg) {
64 + netdata_rwlock_wrlock(&mrg->index.rwlock);
65 +}
66 +static void mrg_index_write_unlock(MRG *mrg) {
67 + netdata_rwlock_unlock(&mrg->index.rwlock);
68 +}
69 +
70 +static inline void mrg_stats_size_judyl_change(MRG *mrg, size_t mem_before_judyl, size_t mem_after_judyl) {
71 + if(mem_after_judyl > mem_before_judyl)
72 + __atomic_add_fetch(&mrg->stats.size, mem_after_judyl - mem_before_judyl, __ATOMIC_RELAXED);
73 + else if(mem_after_judyl < mem_before_judyl)
74 + __atomic_sub_fetch(&mrg->stats.size, mem_before_judyl - mem_after_judyl, __ATOMIC_RELAXED);
75 +}
76 +
77 +static inline void mrg_stats_size_judyhs_added_uuid(MRG *mrg) {
78 + __atomic_add_fetch(&mrg->stats.size, sizeof(uuid_t) * 3, __ATOMIC_RELAXED);
79 +}
80 +
81 +static inline void mrg_stats_size_judyhs_removed_uuid(MRG *mrg) {
82 + __atomic_sub_fetch(&mrg->stats.size, sizeof(uuid_t) * 3, __ATOMIC_RELAXED);
83 +}
84 +
85 +static METRIC *metric_add(MRG *mrg, MRG_ENTRY *entry, bool *ret) {
86 + mrg_index_write_lock(mrg);
87 +
88 + size_t mem_before_judyl, mem_after_judyl;
89 +
90 + Pvoid_t *sections_judy_pptr = JudyHSIns(&mrg->index.uuid_judy, &entry->uuid, sizeof(uuid_t), PJE0);
91 + if(!sections_judy_pptr || sections_judy_pptr == PJERR)
92 + fatal("DBENGINE METRIC: corrupted UUIDs JudyHS array");
93 +
94 + if(!*sections_judy_pptr)
95 + mrg_stats_size_judyhs_added_uuid(mrg);
96 +
97 + mem_before_judyl = JudyLMemUsed(*sections_judy_pptr);
98 + Pvoid_t *PValue = JudyLIns(sections_judy_pptr, entry->section, PJE0);
99 + mem_after_judyl = JudyLMemUsed(*sections_judy_pptr);
100 + mrg_stats_size_judyl_change(mrg, mem_before_judyl, mem_after_judyl);
101 +
102 + if(!PValue || PValue == PJERR)
103 + fatal("DBENGINE METRIC: corrupted section JudyL array");
104 +
105 + if(*PValue != NULL) {
106 + METRIC *metric = *PValue;
107 + mrg_index_write_unlock(mrg);
108 +
109 + if(ret)
110 + *ret = false;
111 +
112 + MRG_STATS_DUPLICATE_ADD(mrg);
113 + return metric;
114 + }
115 +
116 + METRIC *metric = arrayalloc_mallocz(mrg->index.aral);
117 + uuid_copy(metric->uuid, entry->uuid);
118 + metric->section = entry->section;
119 + metric->first_time_s = entry->first_time_s;
120 + metric->latest_time_s_clean = entry->last_time_s;
121 + metric->latest_time_s_hot = 0;
122 + metric->latest_update_every_s = entry->latest_update_every_s;
123 + metric->timestamps_lock = NETDATA_SPINLOCK_INITIALIZER;
124 + *PValue = metric;
125 +
126 + mrg_index_write_unlock(mrg);
127 +
128 + if(ret)
129 + *ret = true;
130 +
131 + MRG_STATS_ADDED_METRIC(mrg);
132 +
133 + return metric;
134 +}
135 +
136 +static METRIC *metric_get(MRG *mrg, uuid_t *uuid, Word_t section) {
137 + mrg_index_read_lock(mrg);
138 +
139 + Pvoid_t *sections_judy_pptr = JudyHSGet(mrg->index.uuid_judy, uuid, sizeof(uuid_t));
140 + if(!sections_judy_pptr) {
141 + mrg_index_read_unlock(mrg);
142 + MRG_STATS_SEARCH_MISS(mrg);
143 + return NULL;
144 + }
145 +
146 + Pvoid_t *PValue = JudyLGet(*sections_judy_pptr, section, PJE0);
147 + if(!PValue) {
148 + mrg_index_read_unlock(mrg);
149 + MRG_STATS_SEARCH_MISS(mrg);
150 + return NULL;
151 + }
152 +
153 + METRIC *metric = *PValue;
154 +
155 + mrg_index_read_unlock(mrg);
156 +
157 + MRG_STATS_SEARCH_HIT(mrg);
158 + return metric;
159 +}
160 +
161 +static bool metric_del(MRG *mrg, METRIC *metric) {
162 + size_t mem_before_judyl, mem_after_judyl;
163 +
164 + mrg_index_write_lock(mrg);
165 +
166 + Pvoid_t *sections_judy_pptr = JudyHSGet(mrg->index.uuid_judy, &metric->uuid, sizeof(uuid_t));
167 + if(!sections_judy_pptr || !*sections_judy_pptr) {
168 + mrg_index_write_unlock(mrg);
169 + MRG_STATS_DELETE_MISS(mrg);
170 + return false;
171 + }
172 +
173 + mem_before_judyl = JudyLMemUsed(*sections_judy_pptr);
174 + int rc = JudyLDel(sections_judy_pptr, metric->section, PJE0);
175 + mem_after_judyl = JudyLMemUsed(*sections_judy_pptr);
176 + mrg_stats_size_judyl_change(mrg, mem_before_judyl, mem_after_judyl);
177 +
178 + if(!rc) {
179 + mrg_index_write_unlock(mrg);
180 + MRG_STATS_DELETE_MISS(mrg);
181 + return false;
182 + }
183 +
184 + if(!*sections_judy_pptr) {
185 + rc = JudyHSDel(&mrg->index.uuid_judy, &metric->uuid, sizeof(uuid_t), PJE0);
186 + if(!rc)
187 + fatal("DBENGINE METRIC: cannot delete UUID from JudyHS");
188 + mrg_stats_size_judyhs_removed_uuid(mrg);
189 + }
190 +
191 + // arrayalloc is running lockless here
192 + arrayalloc_freez(mrg->index.aral, metric);
193 +
194 + mrg_index_write_unlock(mrg);
195 +
196 + MRG_STATS_DELETED_METRIC(mrg);
197 +
198 + return true;
199 +}
200 +
201 +// ----------------------------------------------------------------------------
202 +// public API
203 +
204 +MRG *mrg_create(void) {
205 + MRG *mrg = callocz(1, sizeof(MRG));
206 + netdata_rwlock_init(&mrg->index.rwlock);
207 + mrg->index.aral = arrayalloc_create(sizeof(METRIC), 65536 / sizeof(METRIC), NULL, NULL, false, true);
208 + mrg->stats.size = sizeof(MRG);
209 + return mrg;
210 +}
211 +
212 +void mrg_destroy(MRG *mrg __maybe_unused) {
213 + // no destruction possible
214 + // we can't traverse the metrics list
215 +
216 + // to delete entries, the caller needs to keep pointers to them
217 + // and delete them one by one
218 +
219 + ;
220 +}
221 +
222 +METRIC *mrg_metric_add_and_acquire(MRG *mrg, MRG_ENTRY entry, bool *ret) {
223 + // FIXME - support refcount
224 +
225 +// internal_fatal(entry.latest_time_s > now_realtime_sec(),
226 +// "DBENGINE METRIC: metric latest time is in the future");
227 +
228 + return metric_add(mrg, &entry, ret);
229 +}
230 +
231 +METRIC *mrg_metric_get_and_acquire(MRG *mrg, uuid_t *uuid, Word_t section) {
232 + // FIXME - support refcount
233 + return metric_get(mrg, uuid, section);
234 +}
235 +
236 +bool mrg_metric_release_and_delete(MRG *mrg, METRIC *metric) {
237 + // FIXME - support refcount
238 + return metric_del(mrg, metric);
239 +}
240 +
241 +METRIC *mrg_metric_dup(MRG *mrg __maybe_unused, METRIC *metric) {
242 + // FIXME - duplicate refcount
243 + return metric;
244 +}
245 +
246 +void mrg_metric_release(MRG *mrg __maybe_unused, METRIC *metric __maybe_unused) {
247 + // FIXME - release refcount
248 +
249 +}
250 +
251 +Word_t mrg_metric_id(MRG *mrg __maybe_unused, METRIC *metric) {
252 + return (Word_t)metric;
253 +}
254 +
255 +uuid_t *mrg_metric_uuid(MRG *mrg __maybe_unused, METRIC *metric) {
256 + return &metric->uuid;
257 +}
258 +
259 +Word_t mrg_metric_section(MRG *mrg __maybe_unused, METRIC *metric) {
260 + return metric->section;
261 +}
262 +
263 +bool mrg_metric_set_first_time_s(MRG *mrg __maybe_unused, METRIC *metric, time_t first_time_s) {
264 + netdata_spinlock_lock(&metric->timestamps_lock);
265 + metric->first_time_s = first_time_s;
266 + netdata_spinlock_unlock(&metric->timestamps_lock);
267 +
268 + return true;
269 +}
270 +
271 +void mrg_metric_expand_retention(MRG *mrg __maybe_unused, METRIC *metric, time_t first_time_s, time_t last_time_s, time_t update_every_s) {
272 +
273 + internal_fatal(first_time_s > now_realtime_sec() + 1,
274 + "DBENGINE METRIC: metric first time is in the future");
275 + internal_fatal(last_time_s > now_realtime_sec() + 1,
276 + "DBENGINE METRIC: metric last time is in the future");
277 +
278 + netdata_spinlock_lock(&metric->timestamps_lock);
279 +
280 + if(first_time_s && (!metric->first_time_s || first_time_s < metric->first_time_s))
281 + metric->first_time_s = first_time_s;
282 +
283 + if(last_time_s && (!metric->latest_time_s_clean || last_time_s > metric->latest_time_s_clean)) {
284 + metric->latest_time_s_clean = last_time_s;
285 +
286 + if(update_every_s)
287 + metric->latest_update_every_s = update_every_s;
288 + }
289 + else if(!metric->latest_update_every_s && update_every_s)
290 + metric->latest_update_every_s = update_every_s;
291 +
292 + netdata_spinlock_unlock(&metric->timestamps_lock);
293 +}
294 +
295 +bool mrg_metric_set_first_time_s_if_zero(MRG *mrg __maybe_unused, METRIC *metric, time_t first_time_s) {
296 + bool ret = false;
297 +
298 + netdata_spinlock_lock(&metric->timestamps_lock);
299 + if(!metric->first_time_s) {
300 + metric->first_time_s = first_time_s;
301 +
302 +// if(unlikely(metric->latest_time_s_clean < metric->first_time_s))
303 +// metric->latest_time_s_clean = metric->first_time_s;
304 +//
305 +// if(unlikely(metric->latest_time_s_hot < metric->first_time_s))
306 +// metric->latest_time_s_hot = metric->first_time_s;
307 +
308 + ret = true;
309 + }
310 + netdata_spinlock_unlock(&metric->timestamps_lock);
311 +
312 + return ret;
313 +}
314 +
315 +time_t mrg_metric_get_first_time_s(MRG *mrg __maybe_unused, METRIC *metric) {
316 + time_t first_time_s;
317 + netdata_spinlock_lock(&metric->timestamps_lock);
318 + first_time_s = metric->first_time_s;
319 + if(!first_time_s) {
320 + if(metric->latest_time_s_clean)
321 + first_time_s = metric->latest_time_s_clean;
322 +
323 + if(!first_time_s || metric->latest_time_s_hot < metric->latest_time_s_clean)
324 + first_time_s = metric->latest_time_s_hot;
325 + }
326 + netdata_spinlock_unlock(&metric->timestamps_lock);
327 +
328 + return first_time_s;
329 +}
330 +
331 +bool mrg_metric_set_clean_latest_time_s(MRG *mrg __maybe_unused, METRIC *metric, time_t latest_time_s) {
332 + netdata_spinlock_lock(&metric->timestamps_lock);
333 +
334 + internal_fatal(latest_time_s > now_realtime_sec() + 1,
335 + "DBENGINE METRIC: metric latest time is in the future");
336 +
337 + internal_fatal(metric->latest_time_s_clean > latest_time_s,
338 + "DBENGINE METRIC: metric new clean latest time is older than the previous one");
339 +
340 + metric->latest_time_s_clean = latest_time_s;
341 +
342 + if(unlikely(!metric->first_time_s))
343 + metric->first_time_s = latest_time_s;
344 +
345 +// if(unlikely(metric->first_time_s > latest_time_s))
346 +// metric->first_time_s = latest_time_s;
347 +
348 + netdata_spinlock_unlock(&metric->timestamps_lock);
349 + return true;
350 +}
351 +
352 +bool mrg_metric_set_hot_latest_time_s(MRG *mrg __maybe_unused, METRIC *metric, time_t latest_time_s) {
353 +// internal_fatal(latest_time_s > now_realtime_sec(),
354 +// "DBENGINE METRIC: metric latest time is in the future");
355 +
356 + netdata_spinlock_lock(&metric->timestamps_lock);
357 + metric->latest_time_s_hot = latest_time_s;
358 +
359 + if(unlikely(!metric->first_time_s))
360 + metric->first_time_s = latest_time_s;
361 +
362 +// if(unlikely(metric->first_time_s > latest_time_s))
363 +// metric->first_time_s = latest_time_s;
364 +
365 + netdata_spinlock_unlock(&metric->timestamps_lock);
366 + return true;
367 +}
368 +
369 +time_t mrg_metric_get_latest_time_s(MRG *mrg __maybe_unused, METRIC *metric) {
370 + time_t max;
371 + netdata_spinlock_lock(&metric->timestamps_lock);
372 + max = MAX(metric->latest_time_s_clean, metric->latest_time_s_hot);
373 + netdata_spinlock_unlock(&metric->timestamps_lock);
374 + return max;
375 +}
376 +
377 +bool mrg_metric_set_update_every(MRG *mrg __maybe_unused, METRIC *metric, time_t update_every_s) {
378 + if(!update_every_s)
379 + return false;
380 +
381 + netdata_spinlock_lock(&metric->timestamps_lock);
382 + metric->latest_update_every_s = update_every_s;
383 + netdata_spinlock_unlock(&metric->timestamps_lock);
384 +
385 + return true;
386 +}
387 +
388 +bool mrg_metric_set_update_every_s_if_zero(MRG *mrg __maybe_unused, METRIC *metric, time_t update_every_s) {
389 + if(!update_every_s)
390 + return false;
391 +
392 + netdata_spinlock_lock(&metric->timestamps_lock);
393 + if(!metric->latest_update_every_s)
394 + metric->latest_update_every_s = update_every_s;
395 + netdata_spinlock_unlock(&metric->timestamps_lock);
396 +
397 + return true;
398 +}
399 +
400 +time_t mrg_metric_get_update_every_s(MRG *mrg __maybe_unused, METRIC *metric) {
401 + time_t update_every_s;
402 +
403 + netdata_spinlock_lock(&metric->timestamps_lock);
404 + update_every_s = metric->latest_update_every_s;
405 + netdata_spinlock_unlock(&metric->timestamps_lock);
406 +
407 + return update_every_s;
408 +}
409 +
410 +struct mrg_statistics mrg_get_statistics(MRG *mrg) {
411 + // FIXME - use atomics
412 + return mrg->stats;
413 +}
414 +
415 +// ----------------------------------------------------------------------------
416 +// unit test
417 +
418 +#ifdef MRG_STRESS_TEST
419 +
420 +static void mrg_stress(MRG *mrg, size_t entries, size_t sections) {
421 + bool ret;
422 +
423 + info("DBENGINE METRIC: stress testing %zu entries on %zu sections...", entries, sections);
424 +
425 + METRIC *array[entries][sections];
426 + for(size_t i = 0; i < entries ; i++) {
427 + MRG_ENTRY e = {
428 + .first_time_s = (time_t)(i + 1),
429 + .latest_time_s = (time_t)(i + 2),
430 + .latest_update_every_s = (time_t)(i + 3),
431 + };
432 + uuid_generate_random(e.uuid);
433 +
434 + for(size_t section = 0; section < sections ;section++) {
435 + e.section = section;
436 + array[i][section] = mrg_metric_add_and_acquire(mrg, e, &ret);
437 + if(!ret)
438 + fatal("DBENGINE METRIC: failed to add metric %zu, section %zu", i, section);
439 +
440 + if(mrg_metric_add_and_acquire(mrg, e, &ret) != array[i][section])
441 + fatal("DBENGINE METRIC: adding the same metric twice, returns a different metric");
442 +
443 + if(ret)
444 + fatal("DBENGINE METRIC: adding the same metric twice, returns success");
445 +
446 + if(mrg_metric_get_and_acquire(mrg, &e.uuid, e.section) != array[i][section])
447 + fatal("DBENGINE METRIC: cannot get back the same metric");
448 +
449 + if(uuid_compare(*mrg_metric_uuid(mrg, array[i][section]), e.uuid) != 0)
450 + fatal("DBENGINE METRIC: uuids do not match");
451 + }
452 + }
453 +
454 + for(size_t i = 0; i < entries ; i++) {
455 + for (size_t section = 0; section < sections; section++) {
456 + uuid_t uuid;
457 + uuid_generate_random(uuid);
458 +
459 + if(mrg_metric_get_and_acquire(mrg, &uuid, section))
460 + fatal("DBENGINE METRIC: found non-existing uuid");
461 +
462 + if(mrg_metric_id(mrg, array[i][section]) != (Word_t)array[i][section])
463 + fatal("DBENGINE METRIC: metric id does not match");
464 +
465 + if(mrg_metric_get_first_time_s(mrg, array[i][section]) != (time_t)(i + 1))
466 + fatal("DBENGINE METRIC: wrong first time returned");
467 + if(mrg_metric_get_latest_time_s(mrg, array[i][section]) != (time_t)(i + 2))
468 + fatal("DBENGINE METRIC: wrong latest time returned");
469 + if(mrg_metric_get_update_every_s(mrg, array[i][section]) != (time_t)(i + 3))
470 + fatal("DBENGINE METRIC: wrong latest time returned");
471 +
472 + if(!mrg_metric_set_first_time_s(mrg, array[i][section], (time_t)((i + 1) * 2)))
473 + fatal("DBENGINE METRIC: cannot set first time");
474 + if(!mrg_metric_set_clean_latest_time_s(mrg, array[i][section], (time_t) ((i + 1) * 3)))
475 + fatal("DBENGINE METRIC: cannot set latest time");
476 + if(!mrg_metric_set_update_every(mrg, array[i][section], (time_t)((i + 1) * 4)))
477 + fatal("DBENGINE METRIC: cannot set update every");
478 +
479 + if(mrg_metric_get_first_time_s(mrg, array[i][section]) != (time_t)((i + 1) * 2))
480 + fatal("DBENGINE METRIC: wrong first time returned");
481 + if(mrg_metric_get_latest_time_s(mrg, array[i][section]) != (time_t)((i + 1) * 3))
482 + fatal("DBENGINE METRIC: wrong latest time returned");
483 + if(mrg_metric_get_update_every_s(mrg, array[i][section]) != (time_t)((i + 1) * 4))
484 + fatal("DBENGINE METRIC: wrong latest time returned");
485 + }
486 + }
487 +
488 + for(size_t i = 0; i < entries ; i++) {
489 + for (size_t section = 0; section < sections; section++) {
490 + if(!mrg_metric_release_and_delete(mrg, array[i][section]))
491 + fatal("DBENGINE METRIC: failed to delete metric");
492 + }
493 + }
494 +}
495 +
496 +static void *mrg_stress_test_thread1(void *ptr) {
497 + MRG *mrg = ptr;
498 +
499 + for(int i = 0; i < 5 ; i++)
500 + mrg_stress(mrg, 10000, 5);
501 +
502 + return ptr;
503 +}
504 +
505 +static void *mrg_stress_test_thread2(void *ptr) {
506 + MRG *mrg = ptr;
507 +
508 + for(int i = 0; i < 10 ; i++)
509 + mrg_stress(mrg, 500, 50);
510 +
511 + return ptr;
512 +}
513 +
514 +static void *mrg_stress_test_thread3(void *ptr) {
515 + MRG *mrg = ptr;
516 +
517 + for(int i = 0; i < 50 ; i++)
518 + mrg_stress(mrg, 5000, 1);
519 +
520 + return ptr;
521 +}
522 +#endif
523 +
524 +int mrg_unittest(void) {
525 + MRG *mrg = mrg_create();
526 + METRIC *metric1, *metric2;
527 + bool ret;
528 +
529 + MRG_ENTRY entry = {
530 + .section = 1,
531 + .first_time_s = 2,
532 + .last_time_s = 3,
533 + .latest_update_every_s = 4,
534 + };
535 + uuid_generate(entry.uuid);
536 + metric1 = mrg_metric_add_and_acquire(mrg, entry, &ret);
537 + if(!ret)
538 + fatal("DBENGINE METRIC: failed to add metric");
539 +
540 + // add the same metric again
541 + if(mrg_metric_add_and_acquire(mrg, entry, &ret) != metric1)
542 + fatal("DBENGINE METRIC: adding the same metric twice, does not return the same pointer");
543 + if(ret)
544 + fatal("DBENGINE METRIC: managed to add the same metric twice");
545 +
546 + if(mrg_metric_get_and_acquire(mrg, &entry.uuid, entry.section) != metric1)
547 + fatal("DBENGINE METRIC: cannot find the metric added");
548 +
549 + // add the same metric again
550 + if(mrg_metric_add_and_acquire(mrg, entry, &ret) != metric1)
551 + fatal("DBENGINE METRIC: adding the same metric twice, does not return the same pointer");
552 + if(ret)
553 + fatal("DBENGINE METRIC: managed to add the same metric twice");
554 +
555 + // add the same metric in another section
556 + entry.section = 0;
557 + metric2 = mrg_metric_add_and_acquire(mrg, entry, &ret);
558 + if(!ret)
559 + fatal("DBENGINE METRIC: failed to add metric in different section");
560 +
561 + // add the same metric again
562 + if(mrg_metric_add_and_acquire(mrg, entry, &ret) != metric2)
563 + fatal("DBENGINE METRIC: adding the same metric twice (section 0), does not return the same pointer");
564 + if(ret)
565 + fatal("DBENGINE METRIC: managed to add the same metric twice in (section 0)");
566 +
567 + if(mrg_metric_get_and_acquire(mrg, &entry.uuid, entry.section) != metric2)
568 + fatal("DBENGINE METRIC: cannot find the metric added (section 0)");
569 +
570 + // delete the first metric
571 + if(!mrg_metric_release_and_delete(mrg, metric1))
572 + fatal("DBENGINE METRIC: cannot delete the first metric");
573 +
574 + if(mrg_metric_get_and_acquire(mrg, &entry.uuid, entry.section) != metric2)
575 + fatal("DBENGINE METRIC: cannot find the metric added (section 0), after deleting the first one");
576 +
577 + // delete the first metric again - metric1 pointer is invalid now
578 + if(mrg_metric_release_and_delete(mrg, metric1))
579 + fatal("DBENGINE METRIC: deleted again an already deleted metric");
580 +
581 + // find the section 0 metric again
582 + if(mrg_metric_get_and_acquire(mrg, &entry.uuid, entry.section) != metric2)
583 + fatal("DBENGINE METRIC: cannot find the metric added (section 0), after deleting the first one twice");
584 +
585 + // delete the second metric
586 + if(!mrg_metric_release_and_delete(mrg, metric2))
587 + fatal("DBENGINE METRIC: cannot delete the second metric");
588 +
589 + // delete the second metric again
590 + if(mrg_metric_release_and_delete(mrg, metric2))
591 + fatal("DBENGINE METRIC: managed to delete an already deleted metric");
592 +
593 + if(mrg->stats.entries != 0)
594 + fatal("DBENGINE METRIC: invalid entries counter");
595 +
596 +#ifdef MRG_STRESS_TEST
597 + usec_t started_ut = now_realtime_usec();
598 + pthread_t thread1;
599 + netdata_thread_create(&thread1, "TH1",
600 + NETDATA_THREAD_OPTION_JOINABLE | NETDATA_THREAD_OPTION_DONT_LOG,
601 + mrg_stress_test_thread1, mrg);
602 +
603 + pthread_t thread2;
604 + netdata_thread_create(&thread2, "TH2",
605 + NETDATA_THREAD_OPTION_JOINABLE | NETDATA_THREAD_OPTION_DONT_LOG,
606 + mrg_stress_test_thread2, mrg);
607 +
608 + pthread_t thread3;
609 + netdata_thread_create(&thread3, "TH3",
610 + NETDATA_THREAD_OPTION_JOINABLE | NETDATA_THREAD_OPTION_DONT_LOG,
611 + mrg_stress_test_thread3, mrg);
612 +
613 +
614 + sleep_usec(5 * USEC_PER_SEC);
615 +
616 + netdata_thread_cancel(thread1);
617 + netdata_thread_cancel(thread2);
618 + netdata_thread_cancel(thread3);
619 +
620 + netdata_thread_join(thread1, NULL);
621 + netdata_thread_join(thread2, NULL);
622 + netdata_thread_join(thread3, NULL);
623 + usec_t ended_ut = now_realtime_usec();
624 +
625 + info("DBENGINE METRIC: did %zu additions, %zu duplicate additions, "
626 + "%zu deletions, %zu wrong deletions, "
627 + "%zu successful searches, %zu wrong searches, "
628 + "%zu successful pointer validations, %zu wrong pointer validations "
629 + "in %llu usecs",
630 + mrg->stats.additions, mrg->stats.additions_duplicate,
631 + mrg->stats.deletions, mrg->stats.delete_misses,
632 + mrg->stats.search_hits, mrg->stats.search_misses,
633 + mrg->stats.pointer_validation_hits, mrg->stats.pointer_validation_misses,
634 + ended_ut - started_ut);
635 +
636 +#endif
637 +
638 + mrg_destroy(mrg);
639 +
640 + info("DBENGINE METRIC: all tests passed!");
641 +
642 + return 0;
643 +}
database/engine/metric.h new
+60
@@ -0,0 +1,60 @@
1 +#ifndef DBENGINE_METRIC_H
2 +#define DBENGINE_METRIC_H
3 +
4 +#include "../rrd.h"
5 +
6 +typedef struct metric METRIC;
7 +typedef struct mrg MRG;
8 +
9 +typedef struct mrg_entry {
10 + uuid_t uuid;
11 + Word_t section;
12 + time_t first_time_s;
13 + time_t last_time_s;
14 + uint32_t latest_update_every_s;
15 +} MRG_ENTRY;
16 +
17 +struct mrg_statistics {
18 + size_t entries;
19 + size_t size; // memory without indexing
20 + size_t additions;
21 + size_t additions_duplicate;
22 + size_t deletions;
23 + size_t delete_misses;
24 + size_t search_hits;
25 + size_t search_misses;
26 + size_t pointer_validation_hits;
27 + size_t pointer_validation_misses;
28 +};
29 +
30 +MRG *mrg_create(void);
31 +void mrg_destroy(MRG *mrg);
32 +
33 +METRIC *mrg_metric_dup(MRG *mrg, METRIC *metric);
34 +void mrg_metric_release(MRG *mrg, METRIC *metric);
35 +
36 +METRIC *mrg_metric_add_and_acquire(MRG *mrg, MRG_ENTRY entry, bool *ret);
37 +METRIC *mrg_metric_get_and_acquire(MRG *mrg, uuid_t *uuid, Word_t section);
38 +bool mrg_metric_release_and_delete(MRG *mrg, METRIC *metric);
39 +
40 +Word_t mrg_metric_id(MRG *mrg, METRIC *metric);
41 +uuid_t *mrg_metric_uuid(MRG *mrg, METRIC *metric);
42 +Word_t mrg_metric_section(MRG *mrg, METRIC *metric);
43 +
44 +bool mrg_metric_set_first_time_s(MRG *mrg, METRIC *metric, time_t first_time_s);
45 +bool mrg_metric_set_first_time_s_if_zero(MRG *mrg, METRIC *metric, time_t first_time_s);
46 +time_t mrg_metric_get_first_time_s(MRG *mrg, METRIC *metric);
47 +void mrg_metric_expand_retention(MRG *mrg __maybe_unused, METRIC *metric, time_t first_time_s, time_t last_time_s, time_t update_every_s);
48 +
49 +bool mrg_metric_set_clean_latest_time_s(MRG *mrg, METRIC *metric, time_t latest_time_s);
50 +bool mrg_metric_set_hot_latest_time_s(MRG *mrg, METRIC *metric, time_t latest_time_s);
51 +time_t mrg_metric_get_latest_time_s(MRG *mrg, METRIC *metric);
52 +
53 +bool mrg_metric_set_update_every(MRG *mrg, METRIC *metric, time_t update_every_s);
54 +time_t mrg_metric_get_update_every_s(MRG *mrg, METRIC *metric);
55 +
56 +bool mrg_metric_set_update_every_s_if_zero(MRG *mrg, METRIC *metric, time_t update_every_s);
57 +
58 +struct mrg_statistics mrg_get_statistics(MRG *mrg);
59 +
60 +#endif // DBENGINE_METRIC_H
database/engine/pagecache.c
+885 -1123
@@ -3,1311 +3,1073 @@
3
4 #include "rrdengine.h"
5
6 -ARAL page_descr_aral = {
7 - .requested_element_size = sizeof(struct rrdeng_page_descr),
8 - .initial_elements = 20000,
9 - .filename = "page_descriptors",
10 - .cache_dir = &netdata_configured_cache_dir,
11 - .use_mmap = false,
12 - .internal.initialized = false
13 -};
14 -
15 -void rrdeng_page_descr_aral_go_singlethreaded(void) {
16 - page_descr_aral.internal.lockless = true;
17 -}
18 -void rrdeng_page_descr_aral_go_multithreaded(void) {
19 - page_descr_aral.internal.lockless = false;
20 -}
6 +MRG *main_mrg = NULL;
7 +PGC *main_cache = NULL;
8 +PGC *open_cache = NULL;
9 +PGC *extent_cache = NULL;
10 +struct rrdeng_cache_efficiency_stats rrdeng_cache_efficiency_stats = {};
11
22 -struct rrdeng_page_descr *rrdeng_page_descr_mallocz(void) {
23 - struct rrdeng_page_descr *descr;
24 - descr = arrayalloc_mallocz(&page_descr_aral);
25 - return descr;
12 +static void main_cache_free_clean_page_callback(PGC *cache __maybe_unused, PGC_ENTRY entry __maybe_unused)
13 +{
14 + // Release storage associated with the page
15 + freez(entry.data);
16 }
17
28 -void rrdeng_page_descr_freez(struct rrdeng_page_descr *descr) {
29 - arrayalloc_freez(&page_descr_aral, descr);
30 -}
18 +static void main_cache_flush_dirty_page_callback(PGC *cache __maybe_unused, PGC_ENTRY *entries_array __maybe_unused, PGC_PAGE **pages_array __maybe_unused, size_t entries __maybe_unused)
19 +{
20 + struct rrdengine_instance *ctx = (struct rrdengine_instance *) entries_array[0].section;
21
32 -void rrdeng_page_descr_use_malloc(void) {
33 - if(page_descr_aral.internal.initialized)
34 - error("DBENGINE: cannot change ARAL allocation policy after it has been initialized.");
35 - else
36 - page_descr_aral.use_mmap = false;
37 -}
22 + // mark ctx as having flushing in progress
23 + __atomic_add_fetch(&ctx->worker_config.atomics.extents_currently_being_flushed, 1, __ATOMIC_RELAXED);
24
39 -void rrdeng_page_descr_use_mmap(void) {
40 - if(page_descr_aral.internal.initialized)
41 - error("DBENGINE: cannot change ARAL allocation policy after it has been initialized.");
42 - else
43 - page_descr_aral.use_mmap = true;
44 -}
25 + size_t bytes_per_point = PAGE_POINT_CTX_SIZE_BYTES(ctx);
26
46 -bool rrdeng_page_descr_is_mmap(void) {
47 - return page_descr_aral.use_mmap;
48 -}
27 + struct page_descr_with_data *base = NULL;
28
50 -/* Forward declarations */
51 -static int pg_cache_try_evict_one_page_unsafe(struct rrdengine_instance *ctx);
29 + for (size_t Index = 0 ; Index < entries; Index++) {
30 + time_t start_time_s = entries_array[Index].start_time_s;
31 + time_t end_time_s = entries_array[Index].end_time_s;
32 + struct page_descr_with_data *descr = page_descriptor_get();
33
53 -/* always inserts into tail */
54 -static inline void pg_cache_replaceQ_insert_unsafe(struct rrdengine_instance *ctx,
55 - struct rrdeng_page_descr *descr)
56 -{
57 - struct page_cache *pg_cache = &ctx->pg_cache;
58 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
34 + descr->id = mrg_metric_uuid(main_mrg, (METRIC *) entries_array[Index].metric_id);
35 + descr->metric_id = entries_array[Index].metric_id;
36 + descr->start_time_ut = start_time_s * USEC_PER_SEC;
37 + descr->end_time_ut = end_time_s * USEC_PER_SEC;
38 + descr->update_every_s = entries_array[Index].update_every_s;
39 + descr->type = ctx->page_type;
40
60 - if (likely(NULL != pg_cache->replaceQ.tail)) {
61 - pg_cache_descr->prev = pg_cache->replaceQ.tail;
62 - pg_cache->replaceQ.tail->next = pg_cache_descr;
63 - }
64 - if (unlikely(NULL == pg_cache->replaceQ.head)) {
65 - pg_cache->replaceQ.head = pg_cache_descr;
66 - }
67 - pg_cache->replaceQ.tail = pg_cache_descr;
68 -}
41 + descr->page_length = (end_time_s - (start_time_s - descr->update_every_s)) / descr->update_every_s * bytes_per_point;
42
70 -static inline void pg_cache_replaceQ_delete_unsafe(struct rrdengine_instance *ctx,
71 - struct rrdeng_page_descr *descr)
72 -{
73 - struct page_cache *pg_cache = &ctx->pg_cache;
74 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr, *prev, *next;
43 + if(descr->page_length > entries_array[Index].size) {
44 + descr->page_length = entries_array[Index].size;
45
76 - prev = pg_cache_descr->prev;
77 - next = pg_cache_descr->next;
46 + error_limit_static_global_var(erl, 1, 0);
47 + error_limit(&erl, "DBENGINE: page exceeds the maximum size, adjusting it to max.");
48 + }
49
79 - if (likely(NULL != prev)) {
80 - prev->next = next;
81 - }
82 - if (likely(NULL != next)) {
83 - next->prev = prev;
84 - }
85 - if (unlikely(pg_cache_descr == pg_cache->replaceQ.head)) {
86 - pg_cache->replaceQ.head = next;
87 - }
88 - if (unlikely(pg_cache_descr == pg_cache->replaceQ.tail)) {
89 - pg_cache->replaceQ.tail = prev;
90 - }
91 - pg_cache_descr->prev = pg_cache_descr->next = NULL;
92 -}
50 + memcpy(descr->page, pgc_page_data(pages_array[Index]), descr->page_length);
51 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(base, descr, link.prev, link.next);
52
94 -void pg_cache_replaceQ_insert(struct rrdengine_instance *ctx,
95 - struct rrdeng_page_descr *descr)
96 -{
97 - struct page_cache *pg_cache = &ctx->pg_cache;
53 + internal_fatal(descr->page_length > RRDENG_BLOCK_SIZE, "DBENGINE: faulty page length calculation");
54 + }
55
99 - uv_rwlock_wrlock(&pg_cache->replaceQ.lock);
100 - pg_cache_replaceQ_insert_unsafe(ctx, descr);
101 - uv_rwlock_wrunlock(&pg_cache->replaceQ.lock);
56 + struct completion completion;
57 + completion_init(&completion);
58 + rrdeng_enq_cmd(ctx, RRDENG_OPCODE_FLUSH_PAGES, base, &completion, STORAGE_PRIORITY_CRITICAL);
59 + completion_wait_for(&completion);
60 + completion_destroy(&completion);
61 }
62
104 -void pg_cache_replaceQ_delete(struct rrdengine_instance *ctx,
105 - struct rrdeng_page_descr *descr)
106 -{
107 - struct page_cache *pg_cache = &ctx->pg_cache;
108 -
109 - uv_rwlock_wrlock(&pg_cache->replaceQ.lock);
110 - pg_cache_replaceQ_delete_unsafe(ctx, descr);
111 - uv_rwlock_wrunlock(&pg_cache->replaceQ.lock);
112 -}
113 -void pg_cache_replaceQ_set_hot(struct rrdengine_instance *ctx,
114 - struct rrdeng_page_descr *descr)
63 +static void open_cache_free_clean_page_callback(PGC *cache __maybe_unused, PGC_ENTRY entry __maybe_unused)
64 {
116 - struct page_cache *pg_cache = &ctx->pg_cache;
117 -
118 - uv_rwlock_wrlock(&pg_cache->replaceQ.lock);
119 - pg_cache_replaceQ_delete_unsafe(ctx, descr);
120 - pg_cache_replaceQ_insert_unsafe(ctx, descr);
121 - uv_rwlock_wrunlock(&pg_cache->replaceQ.lock);
65 + struct rrdengine_datafile *datafile = entry.data;
66 + datafile_release(datafile, DATAFILE_ACQUIRE_OPEN_CACHE);
67 }
68
124 -struct rrdeng_page_descr *pg_cache_create_descr(void)
69 +static void open_cache_flush_dirty_page_callback(PGC *cache __maybe_unused, PGC_ENTRY *entries_array __maybe_unused, PGC_PAGE **pages_array __maybe_unused, size_t entries __maybe_unused)
70 {
126 - struct rrdeng_page_descr *descr;
127 -
128 - descr = rrdeng_page_descr_mallocz();
129 - descr->page_length = 0;
130 - descr->start_time_ut = INVALID_TIME;
131 - descr->end_time_ut = INVALID_TIME;
132 - descr->id = NULL;
133 - descr->extent = NULL;
134 - descr->pg_cache_descr_state = 0;
135 - descr->pg_cache_descr = NULL;
136 - descr->update_every_s = 0;
137 -
138 - return descr;
71 + ;
72 }
73
141 -/* The caller must hold page descriptor lock. */
142 -void pg_cache_wake_up_waiters_unsafe(struct rrdeng_page_descr *descr)
74 +static void extent_cache_free_clean_page_callback(PGC *cache __maybe_unused, PGC_ENTRY entry __maybe_unused)
75 {
144 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
145 - if (pg_cache_descr->waiters)
146 - uv_cond_broadcast(&pg_cache_descr->cond);
76 + freez(entry.data);
77 }
78
149 -void pg_cache_wake_up_waiters(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
79 +static void extent_cache_flush_dirty_page_callback(PGC *cache __maybe_unused, PGC_ENTRY *entries_array __maybe_unused, PGC_PAGE **pages_array __maybe_unused, size_t entries __maybe_unused)
80 {
151 - rrdeng_page_descr_mutex_lock(ctx, descr);
152 - pg_cache_wake_up_waiters_unsafe(descr);
153 - rrdeng_page_descr_mutex_unlock(ctx, descr);
81 + ;
82 }
83
156 -/*
157 - * The caller must hold page descriptor lock.
158 - * The lock will be released and re-acquired. The descriptor is not guaranteed
159 - * to exist after this function returns.
160 - */
161 -void pg_cache_wait_event_unsafe(struct rrdeng_page_descr *descr)
162 -{
163 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
84 +typedef enum {
85 + PAGE_IS_IN_THE_PAST = -1,
86 + PAGE_IS_IN_RANGE = 0,
87 + PAGE_IS_IN_THE_FUTURE = 1,
88 +} TIME_RANGE_COMPARE;
89
165 - ++pg_cache_descr->waiters;
166 - uv_cond_wait(&pg_cache_descr->cond, &pg_cache_descr->mutex);
167 - --pg_cache_descr->waiters;
168 -}
90 +static inline TIME_RANGE_COMPARE is_page_in_time_range(time_t page_first_time_s, time_t page_last_time_s, time_t wanted_start_time_s, time_t wanted_end_time_s) {
91 + // page_first_time_s <= wanted_end_time_s && page_last_time_s >= wanted_start_time_s
92
170 -/*
171 - * The caller must hold page descriptor lock.
172 - * The lock will be released and re-acquired. The descriptor is not guaranteed
173 - * to exist after this function returns.
174 - * Returns UV_ETIMEDOUT if timeout_sec seconds pass.
175 - */
176 -int pg_cache_timedwait_event_unsafe(struct rrdeng_page_descr *descr, uint64_t timeout_sec)
177 -{
178 - int ret;
179 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
93 + if(page_last_time_s < wanted_start_time_s)
94 + return PAGE_IS_IN_THE_PAST;
95
181 - ++pg_cache_descr->waiters;
182 - ret = uv_cond_timedwait(&pg_cache_descr->cond, &pg_cache_descr->mutex, timeout_sec * NSEC_PER_SEC);
183 - --pg_cache_descr->waiters;
96 + if(page_first_time_s > wanted_end_time_s)
97 + return PAGE_IS_IN_THE_FUTURE;
98
185 - return ret;
99 + return PAGE_IS_IN_RANGE;
100 }
101
188 -/*
189 - * Returns page flags.
190 - * The lock will be released and re-acquired. The descriptor is not guaranteed
191 - * to exist after this function returns.
192 - */
193 -unsigned long pg_cache_wait_event(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
102 +static int journal_metric_uuid_compare(const void *key, const void *metric)
103 {
195 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
196 - unsigned long flags;
197 -
198 - rrdeng_page_descr_mutex_lock(ctx, descr);
199 - pg_cache_wait_event_unsafe(descr);
200 - flags = pg_cache_descr->flags;
201 - rrdeng_page_descr_mutex_unlock(ctx, descr);
202 -
203 - return flags;
104 + return uuid_compare(*(uuid_t *) key, ((struct journal_metric_list *) metric)->uuid);
105 }
106
206 -/*
207 - * The caller must hold page descriptor lock.
208 - */
209 -int pg_cache_can_get_unsafe(struct rrdeng_page_descr *descr, int exclusive_access)
210 -{
211 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
107 +static inline struct page_details *pdc_find_page_for_time(
108 + Pcvoid_t PArray,
109 + time_t wanted_time_s,
110 + size_t *gaps,
111 + PDC_PAGE_STATUS stop_at
112 +) {
113 + Word_t PIndexF = wanted_time_s, PIndexL = wanted_time_s;
114 + Pvoid_t *PValueF, *PValueL;
115 + struct page_details *pdF = NULL, *pdL = NULL;
116 + bool firstF = true, firstL = true;
117 +
118 + while ((PValueF = PDCJudyLFirstThenNext(PArray, &PIndexF, &firstF))) {
119 + pdF = *PValueF;
120 +
121 + PDC_PAGE_STATUS status = __atomic_load_n(&pdF->status, __ATOMIC_ACQUIRE);
122 + if (!(status & (PDC_PAGE_FAILED | PDC_PAGE_SKIP | PDC_PAGE_INVALID | PDC_PAGE_RELEASED | stop_at)))
123 + break;
124
213 - if ((pg_cache_descr->flags & (RRD_PAGE_LOCKED | RRD_PAGE_READ_PENDING)) ||
214 - (exclusive_access && pg_cache_descr->refcnt)) {
215 - return 0;
125 + pdF = NULL;
126 }
127
218 - return 1;
219 -}
128 + while ((PValueL = PDCJudyLLastThenPrev(PArray, &PIndexL, &firstL))) {
129 + pdL = *PValueL;
130
221 -/*
222 - * The caller must hold page descriptor lock.
223 - * Gets a reference to the page descriptor.
224 - * Returns 1 on success and 0 on failure.
225 - */
226 -int pg_cache_try_get_unsafe(struct rrdeng_page_descr *descr, int exclusive_access)
227 -{
228 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
131 + PDC_PAGE_STATUS status = __atomic_load_n(&pdL->status, __ATOMIC_ACQUIRE);
132 + if(status & stop_at) {
133 + // don't go all the way back to the beginning - stop at the last processed
134 + pdL = NULL;
135 + break;
136 + }
137
230 - if (!pg_cache_can_get_unsafe(descr, exclusive_access))
231 - return 0;
138 + if (!(status & (PDC_PAGE_FAILED | PDC_PAGE_SKIP | PDC_PAGE_INVALID | PDC_PAGE_RELEASED)))
139 + break;
140
233 - if (exclusive_access)
234 - pg_cache_descr->flags |= RRD_PAGE_LOCKED;
235 - ++pg_cache_descr->refcnt;
141 + pdL = NULL;
142 + }
143
237 - return 1;
238 -}
144 + TIME_RANGE_COMPARE rcF = (pdF) ? is_page_in_time_range(pdF->first_time_s, pdF->last_time_s, wanted_time_s, wanted_time_s) : PAGE_IS_IN_THE_FUTURE;
145 + TIME_RANGE_COMPARE rcL = (pdL) ? is_page_in_time_range(pdL->first_time_s, pdL->last_time_s, wanted_time_s, wanted_time_s) : PAGE_IS_IN_THE_PAST;
146
240 -/*
241 - * The caller must hold the page descriptor lock.
242 - * This function may block doing cleanup.
243 - */
244 -void pg_cache_put_unsafe(struct rrdeng_page_descr *descr)
245 -{
246 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
147 + if (!pdF || pdF == pdL) {
148 + // F is missing, or they are the same
149 + // return L
150 + (*gaps) += (rcL == PAGE_IS_IN_RANGE) ? 0 : 1;
151 + return pdL;
152 + }
153
248 - pg_cache_descr->flags &= ~RRD_PAGE_LOCKED;
249 - if (0 == --pg_cache_descr->refcnt) {
250 - pg_cache_wake_up_waiters_unsafe(descr);
154 + if (!pdL) {
155 + // L is missing
156 + // return F
157 + (*gaps) += (rcF == PAGE_IS_IN_RANGE) ? 0 : 1;
158 + return pdF;
159 }
252 -}
160
254 -/*
255 - * This function may block doing cleanup.
256 - */
257 -void pg_cache_put(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
258 -{
259 - rrdeng_page_descr_mutex_lock(ctx, descr);
260 - pg_cache_put_unsafe(descr);
261 - rrdeng_page_descr_mutex_unlock(ctx, descr);
262 -}
161 + if (rcF == rcL) {
162 + // both are on the same side,
163 + // but they are different pages
164
264 -/* The caller must hold the page cache lock */
265 -static void pg_cache_release_pages_unsafe(struct rrdengine_instance *ctx, unsigned number)
266 -{
267 - struct page_cache *pg_cache = &ctx->pg_cache;
165 + switch (rcF) {
166 + case PAGE_IS_IN_RANGE:
167 + // pick the higher resolution
168 + if (pdF->update_every_s && pdF->update_every_s < pdL->update_every_s)
169 + return pdF;
170
269 - pg_cache->populated_pages -= number;
270 -}
171 + if (pdL->update_every_s && pdL->update_every_s < pdF->update_every_s)
172 + return pdL;
173
272 -static void pg_cache_release_pages(struct rrdengine_instance *ctx, unsigned number)
273 -{
274 - struct page_cache *pg_cache = &ctx->pg_cache;
174 + // same resolution - pick the one that starts earlier
175 + if (pdL->first_time_s < pdF->first_time_s)
176 + return pdL;
177
276 - uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
277 - pg_cache_release_pages_unsafe(ctx, number);
278 - uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
279 -}
178 + return pdF;
179 + break;
180
281 -/*
282 - * This function returns the maximum number of pages allowed in the page cache.
283 - */
284 -unsigned long pg_cache_hard_limit(struct rrdengine_instance *ctx)
285 -{
286 - return ctx->max_cache_pages + (unsigned long)ctx->metric_API_max_producers;
287 -}
181 + case PAGE_IS_IN_THE_FUTURE:
182 + (*gaps)++;
183
289 -/*
290 - * This function returns the low watermark number of pages in the page cache. The page cache should strive to keep the
291 - * number of pages below that number.
292 - */
293 -unsigned long pg_cache_soft_limit(struct rrdengine_instance *ctx)
294 -{
295 - return ctx->cache_pages_low_watermark + (unsigned long)ctx->metric_API_max_producers;
296 -}
184 + // pick the one that starts earlier
185 + if (pdL->first_time_s < pdF->first_time_s)
186 + return pdL;
187
298 -/*
299 - * This function returns the maximum number of dirty pages that are committed to be written to disk allowed in the page
300 - * cache.
301 - */
302 -unsigned long pg_cache_committed_hard_limit(struct rrdengine_instance *ctx)
303 -{
304 - /* We remove the active pages of the producers from the calculation and only allow the extra pinned pages */
305 - return ctx->cache_pages_low_watermark + (unsigned long)ctx->metric_API_max_producers;
306 -}
188 + return pdF;
189 + break;
190
308 -/*
309 - * This function will block until it reserves #number populated pages.
310 - * It will trigger evictions or dirty page flushing if the pg_cache_hard_limit() limit is hit.
311 - */
312 -static void pg_cache_reserve_pages(struct rrdengine_instance *ctx, unsigned number)
313 -{
314 - struct page_cache *pg_cache = &ctx->pg_cache;
315 - unsigned failures = 0;
316 - const unsigned FAILURES_CEILING = 10; /* truncates exponential backoff to (2^FAILURES_CEILING x slot) */
317 - unsigned long exp_backoff_slot_usec = USEC_PER_MS * 10;
318 -
319 - assert(number < ctx->max_cache_pages);
320 -
321 - uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
322 - if (pg_cache->populated_pages + number >= pg_cache_hard_limit(ctx) + 1)
323 - debug(D_RRDENGINE, "==Page cache full. Reserving %u pages.==",
324 - number);
325 - while (pg_cache->populated_pages + number >= pg_cache_hard_limit(ctx) + 1) {
326 -
327 - if (!pg_cache_try_evict_one_page_unsafe(ctx)) {
328 - /* failed to evict */
329 - struct completion compl;
330 - struct rrdeng_cmd cmd;
331 -
332 - ++failures;
333 - uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
334 -
335 - completion_init(&compl);
336 - cmd.opcode = RRDENG_FLUSH_PAGES;
337 - cmd.completion = &compl;
338 - rrdeng_enq_cmd(&ctx->worker_config, &cmd);
339 - /* wait for some pages to be flushed */
340 - debug(D_RRDENGINE, "%s: waiting for pages to be written to disk before evicting.", __func__);
341 - completion_wait_for(&compl);
342 - completion_destroy(&compl);
343 -
344 - if (unlikely(failures > 1)) {
345 - unsigned long slots, usecs_to_sleep;
346 - /* exponential backoff */
347 - slots = random() % (2LU << MIN(failures, FAILURES_CEILING));
348 - usecs_to_sleep = slots * exp_backoff_slot_usec;
349 -
350 - if (usecs_to_sleep >= USEC_PER_SEC)
351 - error("Page cache is full. Sleeping for %llu second(s).", usecs_to_sleep / USEC_PER_SEC);
352 -
353 - (void)sleep_usec(usecs_to_sleep);
354 - }
355 - uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
191 + default:
192 + case PAGE_IS_IN_THE_PAST:
193 + (*gaps)++;
194 + return NULL;
195 + break;
196 }
197 }
358 - pg_cache->populated_pages += number;
359 - uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
360 -}
198
362 -/*
363 - * This function will attempt to reserve #number populated pages.
364 - * It may trigger evictions if the pg_cache_soft_limit() limit is hit.
365 - * Returns 0 on failure and 1 on success.
366 - */
367 -static int pg_cache_try_reserve_pages(struct rrdengine_instance *ctx, unsigned number)
368 -{
369 - struct page_cache *pg_cache = &ctx->pg_cache;
370 - unsigned count = 0;
371 - int ret = 0;
372 -
373 - assert(number < ctx->max_cache_pages);
374 -
375 - uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
376 - if (pg_cache->populated_pages + number >= pg_cache_soft_limit(ctx) + 1) {
377 - debug(D_RRDENGINE,
378 - "==Page cache full. Trying to reserve %u pages.==",
379 - number);
380 - do {
381 - if (!pg_cache_try_evict_one_page_unsafe(ctx))
382 - break;
383 - ++count;
384 - } while (pg_cache->populated_pages + number >= pg_cache_soft_limit(ctx) + 1);
385 - debug(D_RRDENGINE, "Evicted %u pages.", count);
199 + if(rcF == PAGE_IS_IN_RANGE) {
200 + // (*gaps) += 0;
201 + return pdF;
202 }
203
388 - if (pg_cache->populated_pages + number < pg_cache_hard_limit(ctx) + 1) {
389 - pg_cache->populated_pages += number;
390 - ret = 1; /* success */
204 + if(rcL == PAGE_IS_IN_RANGE) {
205 + // (*gaps) += 0;
206 + return pdL;
207 }
392 - uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
208
394 - return ret;
395 -}
209 + if(rcF == PAGE_IS_IN_THE_FUTURE) {
210 + (*gaps)++;
211 + return pdF;
212 + }
213
397 -/* The caller must hold the page cache and the page descriptor locks in that order */
398 -static void pg_cache_evict_unsafe(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
399 -{
400 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
214 + if(rcL == PAGE_IS_IN_THE_FUTURE) {
215 + (*gaps)++;
216 + return pdL;
217 + }
218
402 - dbengine_page_free(pg_cache_descr->page);
403 - pg_cache_descr->page = NULL;
404 - pg_cache_descr->flags &= ~RRD_PAGE_POPULATED;
405 - pg_cache_release_pages_unsafe(ctx, 1);
406 - ++ctx->stats.pg_cache_evictions;
219 + // impossible case
220 + (*gaps)++;
221 + return NULL;
222 }
223
409 -/*
410 - * The caller must hold the page cache lock.
411 - * Lock order: page cache -> replaceQ -> page descriptor
412 - * This function iterates all pages and tries to evict one.
413 - * If it fails it sets in_flight_descr to the oldest descriptor that has write-back in progress,
414 - * or it sets it to NULL if no write-back is in progress.
415 - *
416 - * Returns 1 on success and 0 on failure.
417 - */
418 -static int pg_cache_try_evict_one_page_unsafe(struct rrdengine_instance *ctx)
419 -{
420 - struct page_cache *pg_cache = &ctx->pg_cache;
421 - unsigned long old_flags;
422 - struct rrdeng_page_descr *descr;
423 - struct page_cache_descr *pg_cache_descr = NULL;
224 +static size_t get_page_list_from_pgc(PGC *cache, METRIC *metric, struct rrdengine_instance *ctx,
225 + time_t wanted_start_time_s, time_t wanted_end_time_s,
226 + Pvoid_t *JudyL_page_array, size_t *cache_gaps,
227 + bool open_cache_mode, PDC_PAGE_STATUS tags) {
228
425 - uv_rwlock_wrlock(&pg_cache->replaceQ.lock);
426 - for (pg_cache_descr = pg_cache->replaceQ.head ; NULL != pg_cache_descr ; pg_cache_descr = pg_cache_descr->next) {
427 - descr = pg_cache_descr->descr;
229 + size_t pages_found_in_cache = 0;
230 + Word_t metric_id = mrg_metric_id(main_mrg, metric);
231
429 - rrdeng_page_descr_mutex_lock(ctx, descr);
430 - old_flags = pg_cache_descr->flags;
431 - if ((old_flags & RRD_PAGE_POPULATED) && !(old_flags & RRD_PAGE_DIRTY) && pg_cache_try_get_unsafe(descr, 1)) {
432 - /* must evict */
433 - pg_cache_evict_unsafe(ctx, descr);
434 - pg_cache_put_unsafe(descr);
435 - pg_cache_replaceQ_delete_unsafe(ctx, descr);
232 + time_t now_s = wanted_start_time_s;
233 + time_t dt_s = mrg_metric_get_update_every_s(main_mrg, metric);
234
437 - rrdeng_page_descr_mutex_unlock(ctx, descr);
438 - uv_rwlock_wrunlock(&pg_cache->replaceQ.lock);
235 + if(!dt_s)
236 + dt_s = default_rrd_update_every;
237
440 - rrdeng_try_deallocate_pg_cache_descr(ctx, descr);
238 + time_t previous_page_end_time_s = now_s - dt_s;
239 + bool first = true;
240
442 - return 1;
443 - }
444 - rrdeng_page_descr_mutex_unlock(ctx, descr);
445 - }
446 - uv_rwlock_wrunlock(&pg_cache->replaceQ.lock);
241 + do {
242 + PGC_PAGE *page = pgc_page_get_and_acquire(
243 + cache, (Word_t)ctx, (Word_t)metric_id, now_s,
244 + (first) ? PGC_SEARCH_CLOSEST : PGC_SEARCH_NEXT);
245
448 - /* failed to evict */
449 - return 0;
450 -}
246 + first = false;
247
452 -/**
453 - * Deletes a page from the database.
454 - * Callers of this function need to make sure they're not deleting the same descriptor concurrently.
455 - * @param ctx is the database instance.
456 - * @param descr is the page descriptor.
457 - * @param remove_dirty must be non-zero if the page to be deleted is dirty.
458 - * @param is_exclusive_holder must be non-zero if the caller holds an exclusive page reference.
459 - * @param metric_id is set to the metric the page belongs to, if it's safe to delete the metric and metric_id is not
460 - * NULL. Otherwise, metric_id is not set.
461 - * @return 1 if it's safe to delete the metric, 0 otherwise.
462 - */
463 -uint8_t pg_cache_punch_hole(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr, uint8_t remove_dirty,
464 - uint8_t is_exclusive_holder, uuid_t *metric_id)
465 -{
466 - struct page_cache *pg_cache = &ctx->pg_cache;
467 - struct page_cache_descr *pg_cache_descr = NULL;
468 - Pvoid_t *PValue;
469 - struct pg_cache_page_index *page_index = NULL;
470 - int ret;
471 - uint8_t can_delete_metric = 0;
472 -
473 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
474 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, descr->id, sizeof(uuid_t));
475 - fatal_assert(NULL != PValue);
476 - page_index = *PValue;
477 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
478 -
479 - uv_rwlock_wrlock(&page_index->lock);
480 - ret = JudyLDel(&page_index->JudyL_array, (Word_t)(descr->start_time_ut / USEC_PER_SEC), PJE0);
481 - if (unlikely(0 == ret)) {
482 - uv_rwlock_wrunlock(&page_index->lock);
483 - if (unlikely(debug_flags & D_RRDENGINE)) {
484 - print_page_descr(descr);
485 - }
486 - goto destroy;
487 - }
488 - --page_index->page_count;
489 - if (!page_index->writers && !page_index->page_count) {
490 - can_delete_metric = 1;
491 - if (metric_id) {
492 - memcpy(metric_id, page_index->id, sizeof(uuid_t));
248 + if(!page) {
249 + if(previous_page_end_time_s < wanted_end_time_s)
250 + (*cache_gaps)++;
251 +
252 + break;
253 }
494 - }
495 - uv_rwlock_wrunlock(&page_index->lock);
496 - fatal_assert(1 == ret);
497 -
498 - uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
499 - ++ctx->stats.pg_cache_deletions;
500 - --pg_cache->page_descriptors;
501 - uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
502 -
503 - rrdeng_page_descr_mutex_lock(ctx, descr);
504 - pg_cache_descr = descr->pg_cache_descr;
505 - if (!is_exclusive_holder) {
506 - /* If we don't hold an exclusive page reference get one */
507 - while (!pg_cache_try_get_unsafe(descr, 1)) {
508 - debug(D_RRDENGINE, "%s: Waiting for locked page:", __func__);
509 - if (unlikely(debug_flags & D_RRDENGINE))
510 - print_page_cache_descr(descr, "", true);
511 - pg_cache_wait_event_unsafe(descr);
254 +
255 + time_t page_start_time_s = pgc_page_start_time_s(page);
256 + time_t page_end_time_s = pgc_page_end_time_s(page);
257 + time_t page_update_every_s = pgc_page_update_every_s(page);
258 + size_t page_length = pgc_page_data_size(cache, page);
259 +
260 + if(!page_update_every_s)
261 + page_update_every_s = dt_s;
262 +
263 + if(is_page_in_time_range(page_start_time_s, page_end_time_s, wanted_start_time_s, wanted_end_time_s) != PAGE_IS_IN_RANGE) {
264 + // not a useful page for this query
265 + pgc_page_release(cache, page);
266 + page = NULL;
267 +
268 + if(previous_page_end_time_s < wanted_end_time_s)
269 + (*cache_gaps)++;
270 +
271 + break;
272 }
513 - }
514 - if (remove_dirty) {
515 - pg_cache_descr->flags &= ~RRD_PAGE_DIRTY;
516 - } else {
517 - /* even a locked page could be dirty */
518 - while (unlikely(pg_cache_descr->flags & RRD_PAGE_DIRTY)) {
519 - debug(D_RRDENGINE, "%s: Found dirty page, waiting for it to be flushed:", __func__);
520 - if (unlikely(debug_flags & D_RRDENGINE))
521 - print_page_cache_descr(descr, "", true);
522 - pg_cache_wait_event_unsafe(descr);
273 +
274 + if (page_start_time_s - previous_page_end_time_s > dt_s)
275 + (*cache_gaps)++;
276 +
277 + Pvoid_t *PValue = PDCJudyLIns(JudyL_page_array, (Word_t) page_start_time_s, PJE0);
278 + if (!PValue || PValue == PJERR)
279 + fatal("DBENGINE: corrupted judy array in %s()", __FUNCTION__ );
280 +
281 + if (unlikely(*PValue)) {
282 + struct page_details *pd = *PValue;
283 + UNUSED(pd);
284 +
285 +// internal_error(
286 +// pd->first_time_s != page_first_time_s ||
287 +// pd->last_time_s != page_last_time_s ||
288 +// pd->update_every_s != page_update_every_s,
289 +// "DBENGINE: duplicate page with different retention in %s cache "
290 +// "1st: %ld to %ld, ue %u, size %u "
291 +// "2nd: %ld to %ld, ue %ld size %zu "
292 +// "- ignoring the second",
293 +// cache == open_cache ? "open" : "main",
294 +// pd->first_time_s, pd->last_time_s, pd->update_every_s, pd->page_length,
295 +// page_first_time_s, page_last_time_s, page_update_every_s, page_length);
296 +
297 + pgc_page_release(cache, page);
298 }
524 - }
525 - rrdeng_page_descr_mutex_unlock(ctx, descr);
526 -
527 - while (unlikely(pg_cache_descr->flags & RRD_PAGE_READ_PENDING)) {
528 - error_limit_static_global_var(erl, 1, 0);
529 - error_limit(&erl, "%s: Found page with READ PENDING, waiting for read to complete", __func__);
530 - if (unlikely(debug_flags & D_RRDENGINE))
531 - print_page_cache_descr(descr, "", true);
532 - pg_cache_wait_event_unsafe(descr);
533 - }
299 + else {
300 +
301 + internal_fatal(pgc_page_metric(page) != metric_id, "Wrong metric id in page found in cache");
302 + internal_fatal(pgc_page_section(page) != (Word_t)ctx, "Wrong section in page found in cache");
303 +
304 + struct page_details *pd = page_details_get();
305 + pd->metric_id = metric_id;
306 + pd->first_time_s = page_start_time_s;
307 + pd->last_time_s = page_end_time_s;
308 + pd->page_length = page_length;
309 + pd->update_every_s = page_update_every_s;
310 + pd->page = (open_cache_mode) ? NULL : page;
311 + pd->status |= ((pd->page) ? (PDC_PAGE_READY | PDC_PAGE_PRELOADED) : 0) | tags;
312 +
313 + if(open_cache_mode) {
314 + struct rrdengine_datafile *datafile = pgc_page_data(page);
315 + if(datafile_acquire(datafile, DATAFILE_ACQUIRE_PAGE_DETAILS)) { // for pd
316 + struct extent_io_data *xio = (struct extent_io_data *) pgc_page_custom_data(cache, page);
317 + pd->datafile.ptr = pgc_page_data(page);
318 + pd->datafile.file = xio->file;
319 + pd->datafile.extent.pos = xio->pos;
320 + pd->datafile.extent.bytes = xio->bytes;
321 + pd->datafile.fileno = pd->datafile.ptr->fileno;
322 + pd->status |= PDC_PAGE_DATAFILE_ACQUIRED | PDC_PAGE_DISK_PENDING;
323 + }
324 + else {
325 + pd->status |= PDC_PAGE_FAILED | PDC_PAGE_FAILED_TO_ACQUIRE_DATAFILE;
326 + }
327 + pgc_page_release(cache, page);
328 + }
329
535 - if (pg_cache_descr->flags & RRD_PAGE_POPULATED) {
536 - /* only after locking can it be safely deleted from LRU */
537 - pg_cache_replaceQ_delete(ctx, descr);
330 + *PValue = pd;
331
539 - uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
540 - pg_cache_evict_unsafe(ctx, descr);
541 - uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
542 - }
543 - pg_cache_put(ctx, descr);
544 - rrdeng_try_deallocate_pg_cache_descr(ctx, descr);
545 - while (descr->pg_cache_descr_state & PG_CACHE_DESCR_ALLOCATED) {
546 - rrdeng_try_deallocate_pg_cache_descr(ctx, descr); /* spin */
547 - (void)sleep_usec(1000); /* 1 msec */
548 - }
549 -destroy:
550 - rrdeng_page_descr_freez(descr);
551 - pg_cache_update_metric_times(page_index);
332 + pages_found_in_cache++;
333 + }
334
553 - return can_delete_metric;
554 -}
335 + // prepare for the next iteration
336 + previous_page_end_time_s = page_end_time_s;
337
556 -static inline int is_page_in_time_range(struct rrdeng_page_descr *descr, usec_t start_time, usec_t end_time)
557 -{
558 - usec_t pg_start, pg_end;
338 + if(page_update_every_s > 0)
339 + dt_s = page_update_every_s;
340
560 - pg_start = descr->start_time_ut;
561 - pg_end = descr->end_time_ut;
341 + // we are going to as for the NEXT page
342 + // so, set this to our first time
343 + now_s = page_start_time_s;
344
563 - return (pg_start < start_time && pg_end >= start_time) ||
564 - (pg_start >= start_time && pg_start <= end_time);
565 -}
345 + } while(now_s <= wanted_end_time_s);
346
567 -static inline int is_point_in_time_in_page(struct rrdeng_page_descr *descr, usec_t point_in_time)
568 -{
569 - return (point_in_time >= descr->start_time_ut && point_in_time <= descr->end_time_ut);
347 + return pages_found_in_cache;
348 }
349
572 -/* The caller must hold the page index lock */
573 -static inline struct rrdeng_page_descr *
574 - find_first_page_in_time_range(struct pg_cache_page_index *page_index, usec_t start_time, usec_t end_time)
575 -{
576 - struct rrdeng_page_descr *descr = NULL;
350 +static size_t list_has_time_gaps(
351 + struct rrdengine_instance *ctx,
352 + METRIC *metric,
353 + Pvoid_t JudyL_page_array,
354 + time_t wanted_start_time_s,
355 + time_t wanted_end_time_s,
356 + size_t *pages_total,
357 + size_t *pages_found_pass4,
358 + size_t *pages_pending,
359 + size_t *pages_overlapping,
360 + time_t *optimal_end_time_s
361 +) {
362 + // we will recalculate these, so zero them
363 + *pages_pending = 0;
364 + *pages_overlapping = 0;
365 + *optimal_end_time_s = 0;
366 +
367 + bool first;
368 Pvoid_t *PValue;
578 - Word_t Index;
579 -
580 - Index = (Word_t)(start_time / USEC_PER_SEC);
581 - PValue = JudyLLast(page_index->JudyL_array, &Index, PJE0);
582 - if (likely(NULL != PValue)) {
583 - descr = *PValue;
584 - if (is_page_in_time_range(descr, start_time, end_time)) {
585 - return descr;
586 - }
369 + Word_t this_page_start_time;
370 + struct page_details *pd;
371 +
372 + size_t gaps = 0;
373 + Word_t metric_id = mrg_metric_id(main_mrg, metric);
374 +
375 + // ------------------------------------------------------------------------
376 + // PASS 1: remove the preprocessing flags from the pages in PDC
377 +
378 + first = true;
379 + this_page_start_time = 0;
380 + while((PValue = PDCJudyLFirstThenNext(JudyL_page_array, &this_page_start_time, &first))) {
381 + pd = *PValue;
382 + pd->status &= ~(PDC_PAGE_SKIP|PDC_PAGE_PREPROCESSED);
383 }
384
589 - Index = (Word_t)(start_time / USEC_PER_SEC);
590 - PValue = JudyLFirst(page_index->JudyL_array, &Index, PJE0);
591 - if (likely(NULL != PValue)) {
592 - descr = *PValue;
593 - if (is_page_in_time_range(descr, start_time, end_time)) {
594 - return descr;
385 + // ------------------------------------------------------------------------
386 + // PASS 2: emulate processing to find the useful pages
387 +
388 + time_t now_s = wanted_start_time_s;
389 + time_t dt_s = mrg_metric_get_update_every_s(main_mrg, metric);
390 + if(!dt_s)
391 + dt_s = default_rrd_update_every;
392 +
393 + size_t pages_pass2 = 0, pages_pass3 = 0;
394 + while((pd = pdc_find_page_for_time(
395 + JudyL_page_array, now_s, &gaps,
396 + PDC_PAGE_PREPROCESSED))) {
397 +
398 + pd->status |= PDC_PAGE_PREPROCESSED;
399 + pages_pass2++;
400 +
401 + if(pd->update_every_s)
402 + dt_s = pd->update_every_s;
403 +
404 + now_s = pd->last_time_s + dt_s;
405 + if(now_s > wanted_end_time_s) {
406 + *optimal_end_time_s = pd->last_time_s;
407 + break;
408 }
409 }
410
598 - return NULL;
599 -}
411 + // ------------------------------------------------------------------------
412 + // PASS 3: mark as skipped all the pages not useful
413
601 -/* Update metric oldest and latest timestamps efficiently when adding new values */
602 -void pg_cache_add_new_metric_time(struct pg_cache_page_index *page_index, struct rrdeng_page_descr *descr)
603 -{
604 - usec_t oldest_time = page_index->oldest_time_ut;
605 - usec_t latest_time = page_index->latest_time_ut;
414 + first = true;
415 + this_page_start_time = 0;
416 + while((PValue = PDCJudyLFirstThenNext(JudyL_page_array, &this_page_start_time, &first))) {
417 + pd = *PValue;
418
607 - if (unlikely(oldest_time == INVALID_TIME || descr->start_time_ut < oldest_time)) {
608 - page_index->oldest_time_ut = descr->start_time_ut;
609 - }
610 - if (likely(descr->end_time_ut > latest_time || latest_time == INVALID_TIME)) {
611 - page_index->latest_time_ut = descr->end_time_ut;
612 - }
613 -}
419 + internal_fatal(pd->metric_id != metric_id, "pd has wrong metric_id");
420
615 -/* Update metric oldest and latest timestamps when removing old values */
616 -void pg_cache_update_metric_times(struct pg_cache_page_index *page_index)
617 -{
618 - Pvoid_t *firstPValue, *lastPValue;
619 - Word_t firstIndex, lastIndex;
620 - struct rrdeng_page_descr *descr;
621 - usec_t oldest_time = INVALID_TIME;
622 - usec_t latest_time = INVALID_TIME;
623 -
624 - uv_rwlock_rdlock(&page_index->lock);
625 - /* Find first page in range */
626 - firstIndex = (Word_t)0;
627 - firstPValue = JudyLFirst(page_index->JudyL_array, &firstIndex, PJE0);
628 - if (likely(NULL != firstPValue)) {
629 - descr = *firstPValue;
630 - oldest_time = descr->start_time_ut;
631 - }
632 - lastIndex = (Word_t)-1;
633 - lastPValue = JudyLLast(page_index->JudyL_array, &lastIndex, PJE0);
634 - if (likely(NULL != lastPValue)) {
635 - descr = *lastPValue;
636 - latest_time = descr->end_time_ut;
637 - }
638 - uv_rwlock_rdunlock(&page_index->lock);
421 + if(!(pd->status & PDC_PAGE_PREPROCESSED)) {
422 + (*pages_overlapping)++;
423 + pd->status |= PDC_PAGE_SKIP;
424 + pd->status &= ~(PDC_PAGE_READY | PDC_PAGE_DISK_PENDING);
425 + continue;
426 + }
427
640 - if (unlikely(NULL == firstPValue)) {
641 - fatal_assert(NULL == lastPValue);
642 - page_index->oldest_time_ut = page_index->latest_time_ut = INVALID_TIME;
643 - return;
644 - }
645 - page_index->oldest_time_ut = oldest_time;
646 - page_index->latest_time_ut = latest_time;
647 -}
428 + pages_pass3++;
429
649 -/* If index is NULL lookup by UUID (descr->id) */
650 -void pg_cache_insert(struct rrdengine_instance *ctx, struct pg_cache_page_index *index,
651 - struct rrdeng_page_descr *descr)
652 -{
653 - struct page_cache *pg_cache = &ctx->pg_cache;
654 - Pvoid_t *PValue;
655 - struct pg_cache_page_index *page_index;
656 - unsigned long pg_cache_descr_state = descr->pg_cache_descr_state;
657 -
658 - if (0 != pg_cache_descr_state) {
659 - /* there is page cache descriptor pre-allocated state */
660 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
661 -
662 - fatal_assert(pg_cache_descr_state & PG_CACHE_DESCR_ALLOCATED);
663 - if (pg_cache_descr->flags & RRD_PAGE_POPULATED) {
664 - pg_cache_reserve_pages(ctx, 1);
665 - if (!(pg_cache_descr->flags & RRD_PAGE_DIRTY))
666 - pg_cache_replaceQ_insert(ctx, descr);
430 + if(!pd->page) {
431 + pd->page = pgc_page_get_and_acquire(main_cache, (Word_t) ctx, (Word_t) metric_id, pd->first_time_s, PGC_SEARCH_EXACT);
432 +
433 + if(pd->page) {
434 + (*pages_found_pass4)++;
435 +
436 + pd->status &= ~PDC_PAGE_DISK_PENDING;
437 + pd->status |= PDC_PAGE_READY | PDC_PAGE_PRELOADED | PDC_PAGE_PRELOADED_PASS4;
438 + }
439 + else {
440 + (*pages_pending)++;
441 +
442 + if (pd->status & PDC_PAGE_DISK_PENDING) {
443 + internal_fatal(!pd->datafile.ptr, "datafile is NULL");
444 + internal_fatal(!pd->datafile.extent.bytes, "datafile.extent.bytes zero");
445 + internal_fatal(!pd->datafile.extent.pos, "datafile.extent.pos is zero");
446 + internal_fatal(!pd->datafile.fileno, "datafile.fileno is zero");
447 + }
448 + else
449 + internal_fatal(!(pd->status & PDC_PAGE_FAILED),
450 + "DBENGINE: pdc has a disk pending page, without proper tagging");
451 + }
452 + }
453 + else {
454 + pd->status &= ~PDC_PAGE_DISK_PENDING;
455 + pd->status |= (PDC_PAGE_READY | PDC_PAGE_PRELOADED);
456 }
457 }
458
670 - if (unlikely(NULL == index)) {
671 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
672 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, descr->id, sizeof(uuid_t));
673 - fatal_assert(NULL != PValue);
674 - page_index = *PValue;
675 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
676 - } else {
677 - page_index = index;
678 - }
459 + internal_fatal(pages_pass2 != pages_pass3,
460 + "DBENGINE: page count does not match");
461 +
462 + *pages_total = pages_pass2;
463
680 - uv_rwlock_wrlock(&page_index->lock);
681 - PValue = JudyLIns(&page_index->JudyL_array, (Word_t)(descr->start_time_ut / USEC_PER_SEC), PJE0);
682 - *PValue = descr;
683 - ++page_index->page_count;
684 - pg_cache_add_new_metric_time(page_index, descr);
685 - uv_rwlock_wrunlock(&page_index->lock);
686 -
687 - uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
688 - ++ctx->stats.pg_cache_insertions;
689 - ++pg_cache->page_descriptors;
690 - uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
464 + return gaps;
465 }
466
693 -usec_t pg_cache_oldest_time_in_range(struct rrdengine_instance *ctx, uuid_t *id, usec_t start_time_ut, usec_t end_time_ut)
694 -{
695 - struct page_cache *pg_cache = &ctx->pg_cache;
696 - struct rrdeng_page_descr *descr = NULL;
697 - Pvoid_t *PValue;
698 - struct pg_cache_page_index *page_index = NULL;
467 +typedef void (*page_found_callback)(PGC_PAGE *page, void *data);
468 +size_t get_page_list_from_journal_v2(struct rrdengine_instance *ctx, METRIC *metric, usec_t start_time_ut, usec_t end_time_ut, page_found_callback callback, void *callback_data) {
469 + uuid_t *uuid = mrg_metric_uuid(main_mrg, metric);
470 + Word_t metric_id = mrg_metric_id(main_mrg, metric);
471
700 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
701 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, id, sizeof(uuid_t));
702 - if (likely(NULL != PValue)) {
703 - page_index = *PValue;
704 - }
705 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
706 - if (NULL == PValue) {
707 - return INVALID_TIME;
708 - }
472 + time_t wanted_start_time_s = (time_t)(start_time_ut / USEC_PER_SEC);
473 + time_t wanted_end_time_s = (time_t)(end_time_ut / USEC_PER_SEC);
474
710 - uv_rwlock_rdlock(&page_index->lock);
711 - descr = find_first_page_in_time_range(page_index, start_time_ut, end_time_ut);
712 - if (NULL == descr) {
713 - uv_rwlock_rdunlock(&page_index->lock);
714 - return INVALID_TIME;
715 - }
716 - uv_rwlock_rdunlock(&page_index->lock);
717 - return descr->start_time_ut;
718 -}
475 + size_t pages_found = 0;
476
720 -/**
721 - * Return page information for the first page before point_in_time that satisfies the filter.
722 - * @param ctx DB context
723 - * @param page_index page index of a metric
724 - * @param point_in_time_ut the pages that are searched must be older than this timestamp
725 - * @param filter decides if the page satisfies the caller's criteria
726 - * @param page_info the result of the search is set in this pointer
727 - */
728 -void pg_cache_get_filtered_info_prev(struct rrdengine_instance *ctx, struct pg_cache_page_index *page_index,
729 - usec_t point_in_time_ut, pg_cache_page_info_filter_t *filter,
730 - struct rrdeng_page_info *page_info)
731 -{
732 - struct page_cache *pg_cache = &ctx->pg_cache;
733 - struct rrdeng_page_descr *descr = NULL;
734 - Pvoid_t *PValue;
735 - Word_t Index;
477 + uv_rwlock_rdlock(&ctx->datafiles.rwlock);
478 + struct rrdengine_datafile *datafile;
479 + for(datafile = ctx->datafiles.first; datafile ; datafile = datafile->next) {
480 + struct journal_v2_header *journal_header = (struct journal_v2_header *) GET_JOURNAL_DATA(datafile->journalfile);
481
737 - (void)pg_cache;
738 - fatal_assert(NULL != page_index);
482 + if (!journal_header)
483 + continue;
484
740 - Index = (Word_t)(point_in_time_ut / USEC_PER_SEC);
741 - uv_rwlock_rdlock(&page_index->lock);
742 - do {
743 - PValue = JudyLPrev(page_index->JudyL_array, &Index, PJE0);
744 - descr = unlikely(NULL == PValue) ? NULL : *PValue;
745 - } while (descr != NULL && !filter(descr));
746 - if (unlikely(NULL == descr)) {
747 - page_info->page_length = 0;
748 - page_info->start_time_ut = INVALID_TIME;
749 - page_info->end_time_ut = INVALID_TIME;
750 - } else {
751 - page_info->page_length = descr->page_length;
752 - page_info->start_time_ut = descr->start_time_ut;
753 - page_info->end_time_ut = descr->end_time_ut;
485 + time_t journal_start_time_s = (time_t)(journal_header->start_time_ut / USEC_PER_SEC);
486 + time_t journal_end_time_s = (time_t)(journal_header->end_time_ut / USEC_PER_SEC);
487 +
488 + // is the datafile within our time-range?
489 + TIME_RANGE_COMPARE jrc = is_page_in_time_range(journal_start_time_s, journal_end_time_s, wanted_start_time_s, wanted_end_time_s);
490 + if(jrc != PAGE_IS_IN_RANGE)
491 + continue;
492 +
493 + // the datafile possibly contains useful data for this query
494 +
495 + size_t journal_metric_count = (size_t)journal_header->metric_count;
496 + struct journal_metric_list *uuid_list = (struct journal_metric_list *)((uint8_t *) journal_header + journal_header->metric_offset);
497 + struct journal_metric_list *uuid_entry = bsearch(uuid,uuid_list,journal_metric_count,sizeof(*uuid_list), journal_metric_uuid_compare);
498 +
499 + if (unlikely(!uuid_entry))
500 + // our UUID is not in this datafile
501 + continue;
502 +
503 + struct journal_page_header *page_list_header = (struct journal_page_header *) ((uint8_t *) journal_header + uuid_entry->page_offset);
504 + struct journal_page_list *page_list = (struct journal_page_list *)((uint8_t *) page_list_header + sizeof(*page_list_header));
505 + struct journal_extent_list *extent_list = (void *)((uint8_t *)journal_header + journal_header->extent_offset);
506 + uint32_t uuid_page_entries = page_list_header->entries;
507 +
508 + for (uint32_t index = 0; index < uuid_page_entries; index++) {
509 + struct journal_page_list *page_entry_in_journal = &page_list[index];
510 +
511 + time_t page_first_time_s = page_entry_in_journal->delta_start_s + journal_start_time_s;
512 + time_t page_last_time_s = page_entry_in_journal->delta_end_s + journal_start_time_s;
513 +
514 + TIME_RANGE_COMPARE prc = is_page_in_time_range(page_first_time_s, page_last_time_s, wanted_start_time_s, wanted_end_time_s);
515 + if(prc == PAGE_IS_IN_THE_PAST)
516 + continue;
517 +
518 + if(prc == PAGE_IS_IN_THE_FUTURE)
519 + break;
520 +
521 + time_t page_update_every_s = page_entry_in_journal->update_every_s;
522 + size_t page_length = page_entry_in_journal->page_length;
523 +
524 + if(datafile_acquire(datafile, DATAFILE_ACQUIRE_OPEN_CACHE)) { //for open cache item
525 + // add this page to open cache
526 + bool added = false;
527 + struct extent_io_data ei = {
528 + .pos = extent_list[page_entry_in_journal->extent_index].datafile_offset,
529 + .bytes = extent_list[page_entry_in_journal->extent_index].datafile_size,
530 + .page_length = page_length,
531 + .file = datafile->file,
532 + .fileno = datafile->fileno,
533 + };
534 +
535 + PGC_PAGE *page = pgc_page_add_and_acquire(open_cache, (PGC_ENTRY) {
536 + .hot = false,
537 + .section = (Word_t) ctx,
538 + .metric_id = metric_id,
539 + .start_time_s = page_first_time_s,
540 + .end_time_s = page_last_time_s,
541 + .update_every_s = page_update_every_s,
542 + .data = datafile,
543 + .size = 0,
544 + .custom_data = (uint8_t *) &ei,
545 + }, &added);
546 +
547 + if(!added)
548 + datafile_release(datafile, DATAFILE_ACQUIRE_OPEN_CACHE);
549 +
550 + callback(page, callback_data);
551 +
552 + pgc_page_release(open_cache, page);
553 +
554 + pages_found++;
555 + }
556 + }
557 }
755 - uv_rwlock_rdunlock(&page_index->lock);
558 + uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
559 +
560 + return pages_found;
561 }
562
758 -/**
759 - * Searches for an unallocated page without triggering disk I/O. Attempts to reserve the page and get a reference.
760 - * @param ctx DB context
761 - * @param id lookup by UUID
762 - * @param start_time_ut exact starting time in usec
763 - * @param ret_page_indexp Sets the page index pointer (*ret_page_indexp) for the given UUID.
764 - * @return the page descriptor or NULL on failure. It can fail if:
765 - * 1. The page is already allocated to the page cache.
766 - * 2. It did not succeed to get a reference.
767 - * 3. It did not succeed to reserve a spot in the page cache.
768 - */
769 -struct rrdeng_page_descr *pg_cache_lookup_unpopulated_and_lock(struct rrdengine_instance *ctx, uuid_t *id,
770 - usec_t start_time_ut)
771 -{
772 - struct page_cache *pg_cache = &ctx->pg_cache;
773 - struct rrdeng_page_descr *descr = NULL;
774 - struct page_cache_descr *pg_cache_descr = NULL;
775 - unsigned long flags;
776 - Pvoid_t *PValue;
777 - struct pg_cache_page_index *page_index = NULL;
778 - Word_t Index;
563 +void add_page_details_from_journal_v2(PGC_PAGE *page, void *JudyL_pptr) {
564 + struct rrdengine_datafile *datafile = pgc_page_data(page);
565
780 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
781 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, id, sizeof(uuid_t));
782 - if (likely(NULL != PValue)) {
783 - page_index = *PValue;
784 - }
785 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
566 + if(!datafile_acquire(datafile, DATAFILE_ACQUIRE_PAGE_DETAILS)) // for pd
567 + return;
568
787 - if ((NULL == PValue) || !pg_cache_try_reserve_pages(ctx, 1)) {
788 - /* Failed to find page or failed to reserve a spot in the cache */
789 - return NULL;
569 + Pvoid_t *PValue = PDCJudyLIns(JudyL_pptr, pgc_page_start_time_s(page), PJE0);
570 + if (!PValue || PValue == PJERR)
571 + fatal("DBENGINE: corrupted judy array");
572 +
573 + if (unlikely(*PValue)) {
574 + datafile_release(datafile, DATAFILE_ACQUIRE_PAGE_DETAILS);
575 + return;
576 }
577
792 - uv_rwlock_rdlock(&page_index->lock);
793 - Index = (Word_t)(start_time_ut / USEC_PER_SEC);
794 - PValue = JudyLGet(page_index->JudyL_array, Index, PJE0);
795 - if (likely(NULL != PValue)) {
796 - descr = *PValue;
578 + Word_t metric_id = pgc_page_metric(page);
579 +
580 + // let's add it to the judy
581 + struct extent_io_data *ei = pgc_page_custom_data(open_cache, page);
582 + struct page_details *pd = page_details_get();
583 + *PValue = pd;
584 +
585 + pd->datafile.extent.pos = ei->pos;
586 + pd->datafile.extent.bytes = ei->bytes;
587 + pd->datafile.file = ei->file;
588 + pd->datafile.fileno = ei->fileno;
589 + pd->first_time_s = pgc_page_start_time_s(page);
590 + pd->last_time_s = pgc_page_end_time_s(page);
591 + pd->datafile.ptr = datafile;
592 + pd->page_length = ei->page_length;
593 + pd->update_every_s = pgc_page_update_every_s(page);
594 + pd->metric_id = metric_id;
595 + pd->status |= PDC_PAGE_DISK_PENDING | PDC_PAGE_SOURCE_JOURNAL_V2 | PDC_PAGE_DATAFILE_ACQUIRED;
596 +}
597 +
598 +// Return a judyL will all pages that have start_time_ut and end_time_ut
599 +// Pvalue of the judy will be the end time for that page
600 +// DBENGINE2:
601 +#define time_delta(finish, pass) do { if(pass) { usec_t t = pass; (pass) = (finish) - (pass); (finish) = t; } } while(0)
602 +static Pvoid_t get_page_list(
603 + struct rrdengine_instance *ctx,
604 + METRIC *metric,
605 + usec_t start_time_ut,
606 + usec_t end_time_ut,
607 + size_t *pages_to_load,
608 + time_t *optimal_end_time_s
609 +) {
610 + *optimal_end_time_s = 0;
611 +
612 + Pvoid_t JudyL_page_array = (Pvoid_t) NULL;
613 +
614 + time_t wanted_start_time_s = (time_t)(start_time_ut / USEC_PER_SEC);
615 + time_t wanted_end_time_s = (time_t)(end_time_ut / USEC_PER_SEC);
616 +
617 + size_t pages_found_in_main_cache = 0,
618 + pages_found_in_open_cache = 0,
619 + pages_found_in_journals_v2 = 0,
620 + pages_found_pass4 = 0,
621 + pages_pending = 0,
622 + pages_overlapping = 0,
623 + pages_total = 0;
624 +
625 + size_t cache_gaps = 0, query_gaps = 0;
626 + bool done_v2 = false, done_open = false;
627 +
628 + usec_t pass1_ut = 0, pass2_ut = 0, pass3_ut = 0, pass4_ut = 0;
629 +
630 + // --------------------------------------------------------------
631 + // PASS 1: Check what the main page cache has available
632 +
633 + pass1_ut = now_monotonic_usec();
634 + size_t pages_pass1 = get_page_list_from_pgc(main_cache, metric, ctx, wanted_start_time_s, wanted_end_time_s,
635 + &JudyL_page_array, &cache_gaps,
636 + false, PDC_PAGE_PRELOADED_PASS1 | PDC_PAGE_SOURCE_MAIN_CACHE);
637 + query_gaps += cache_gaps;
638 + pages_found_in_main_cache += pages_pass1;
639 + pages_total += pages_pass1;
640 +
641 + if(pages_found_in_main_cache && !cache_gaps) {
642 + query_gaps = list_has_time_gaps(ctx, metric, JudyL_page_array, wanted_start_time_s, wanted_end_time_s,
643 + &pages_total, &pages_found_pass4, &pages_pending, &pages_overlapping,
644 + optimal_end_time_s);
645 +
646 + if (pages_total && !query_gaps)
647 + goto we_are_done;
648 }
798 - if (NULL == PValue || 0 == descr->page_length) {
799 - /* Failed to find non-empty page */
800 - uv_rwlock_rdunlock(&page_index->lock);
649
802 - pg_cache_release_pages(ctx, 1);
803 - return NULL;
650 + // --------------------------------------------------------------
651 + // PASS 2: Check what the open journal page cache has available
652 + // these will be loaded from disk
653 +
654 + pass2_ut = now_monotonic_usec();
655 + size_t pages_pass2 = get_page_list_from_pgc(open_cache, metric, ctx, wanted_start_time_s, wanted_end_time_s,
656 + &JudyL_page_array, &cache_gaps,
657 + true, PDC_PAGE_SOURCE_OPEN_CACHE);
658 + query_gaps += cache_gaps;
659 + pages_found_in_open_cache += pages_pass2;
660 + pages_total += pages_pass2;
661 + done_open = true;
662 +
663 + if(pages_found_in_open_cache) {
664 + query_gaps = list_has_time_gaps(ctx, metric, JudyL_page_array, wanted_start_time_s, wanted_end_time_s,
665 + &pages_total, &pages_found_pass4, &pages_pending, &pages_overlapping,
666 + optimal_end_time_s);
667 +
668 + if (pages_total && !query_gaps)
669 + goto we_are_done;
670 }
671
806 - rrdeng_page_descr_mutex_lock(ctx, descr);
807 - pg_cache_descr = descr->pg_cache_descr;
808 - flags = pg_cache_descr->flags;
809 - uv_rwlock_rdunlock(&page_index->lock);
672 + // --------------------------------------------------------------
673 + // PASS 3: Check Journal v2 to fill the gaps
674 +
675 + pass3_ut = now_monotonic_usec();
676 + size_t pages_pass3 = get_page_list_from_journal_v2(ctx, metric, start_time_ut, end_time_ut,
677 + add_page_details_from_journal_v2, &JudyL_page_array);
678 + pages_found_in_journals_v2 += pages_pass3;
679 + pages_total += pages_pass3;
680 + done_v2 = true;
681 +
682 + // --------------------------------------------------------------
683 + // PASS 4: Check the cache again
684 + // and calculate the time gaps in the query
685 + // THIS IS REQUIRED AFTER JOURNAL V2 LOOKUP
686 +
687 + pass4_ut = now_monotonic_usec();
688 + query_gaps = list_has_time_gaps(ctx, metric, JudyL_page_array, wanted_start_time_s, wanted_end_time_s,
689 + &pages_total, &pages_found_pass4, &pages_pending, &pages_overlapping,
690 + optimal_end_time_s);
691 +
692 +we_are_done:
693 +
694 + if(pages_to_load)
695 + *pages_to_load = pages_pending;
696 +
697 + usec_t finish_ut = now_monotonic_usec();
698 + time_delta(finish_ut, pass4_ut);
699 + time_delta(finish_ut, pass3_ut);
700 + time_delta(finish_ut, pass2_ut);
701 + time_delta(finish_ut, pass1_ut);
702 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.prep_time_in_main_cache_lookup, pass1_ut, __ATOMIC_RELAXED);
703 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.prep_time_in_open_cache_lookup, pass2_ut, __ATOMIC_RELAXED);
704 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.prep_time_in_journal_v2_lookup, pass3_ut, __ATOMIC_RELAXED);
705 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.prep_time_in_pass4_lookup, pass4_ut, __ATOMIC_RELAXED);
706 +
707 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.queries, 1, __ATOMIC_RELAXED);
708 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.queries_planned_with_gaps, (query_gaps) ? 1 : 0, __ATOMIC_RELAXED);
709 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.queries_open, done_open ? 1 : 0, __ATOMIC_RELAXED);
710 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.queries_journal_v2, done_v2 ? 1 : 0, __ATOMIC_RELAXED);
711 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_total, pages_total, __ATOMIC_RELAXED);
712 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_meta_source_main_cache, pages_found_in_main_cache, __ATOMIC_RELAXED);
713 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_meta_source_open_cache, pages_found_in_open_cache, __ATOMIC_RELAXED);
714 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_meta_source_journal_v2, pages_found_in_journals_v2, __ATOMIC_RELAXED);
715 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_data_source_main_cache, pages_found_in_main_cache, __ATOMIC_RELAXED);
716 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_pending_found_in_cache_at_pass4, pages_found_pass4, __ATOMIC_RELAXED);
717 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_to_load_from_disk, pages_pending, __ATOMIC_RELAXED);
718 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_overlapping_skipped, pages_overlapping, __ATOMIC_RELAXED);
719 +
720 + return JudyL_page_array;
721 +}
722
811 - if ((flags & RRD_PAGE_POPULATED) || !pg_cache_try_get_unsafe(descr, 1)) {
812 - /* Failed to get reference or page is already populated */
813 - rrdeng_page_descr_mutex_unlock(ctx, descr);
723 +inline void rrdeng_prep_wait(PDC *pdc) {
724 + if (unlikely(pdc && !pdc->prep_done)) {
725 + usec_t started_ut = now_monotonic_usec();
726 + completion_wait_for(&pdc->prep_completion);
727 + pdc->prep_done = true;
728 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.query_time_wait_for_prep, now_monotonic_usec() - started_ut, __ATOMIC_RELAXED);
729 + }
730 +}
731
815 - pg_cache_release_pages(ctx, 1);
816 - return NULL;
732 +void rrdeng_prep_query(PDC *pdc) {
733 + size_t pages_to_load = 0;
734 + pdc->page_list_JudyL = get_page_list(pdc->ctx, pdc->metric,
735 + pdc->start_time_s * USEC_PER_SEC,
736 + pdc->end_time_s * USEC_PER_SEC,
737 + &pages_to_load,
738 + &pdc->optimal_end_time_s);
739 +
740 + if (pages_to_load && pdc->page_list_JudyL) {
741 + pdc_acquire(pdc); // we get 1 for the 1st worker in the chain: do_read_page_list_work()
742 + pdc->preload_all_extent_pages = false;
743 + usec_t start_ut = now_monotonic_usec();
744 +// if(likely(priority == STORAGE_PRIORITY_BEST_EFFORT))
745 +// dbengine_load_page_list_directly(ctx, handle->pdc);
746 +// else
747 + pdc_route_asynchronously(pdc->ctx, pdc);
748 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.prep_time_to_route, now_monotonic_usec() - start_ut, __ATOMIC_RELAXED);
749 }
818 - /* success */
819 - rrdeng_page_descr_mutex_unlock(ctx, descr);
820 - rrd_stat_atomic_add(&ctx->stats.pg_cache_misses, 1);
750 + else
751 + completion_mark_complete(&pdc->page_completion);
752 +
753 + completion_mark_complete(&pdc->prep_completion);
754
822 - return descr;
755 + pdc_release_and_destroy_if_unreferenced(pdc, true, true);
756 }
757
758 /**
759 * Searches for pages in a time range and triggers disk I/O if necessary and possible.
827 - * Does not get a reference.
760 * @param ctx DB context
829 - * @param id UUID
761 + * @param handle query handle as initialized
762 * @param start_time_ut inclusive starting time in usec
763 * @param end_time_ut inclusive ending time in usec
832 - * @param page_info_arrayp It allocates (*page_arrayp) and populates it with information of pages that overlap
833 - * with the time range [start_time,end_time]. The caller must free (*page_info_arrayp) with freez().
834 - * If page_info_arrayp is set to NULL nothing was allocated.
835 - * @param ret_page_indexp Sets the page index pointer (*ret_page_indexp) for the given UUID.
836 - * @return the number of pages that overlap with the time range [start_time,end_time].
764 + * @return 1 / 0 (pages found or not found)
765 */
838 -unsigned pg_cache_preload(struct rrdengine_instance *ctx, uuid_t *id, usec_t start_time_ut, usec_t end_time_ut,
839 - struct rrdeng_page_info **page_info_arrayp, struct pg_cache_page_index **ret_page_indexp)
840 -{
841 - struct page_cache *pg_cache = &ctx->pg_cache;
842 - struct rrdeng_page_descr *descr = NULL, *preload_array[PAGE_CACHE_MAX_PRELOAD_PAGES];
843 - struct page_cache_descr *pg_cache_descr = NULL;
844 - unsigned i, j, k, preload_count, count, page_info_array_max_size;
845 - unsigned long flags;
846 - Pvoid_t *PValue;
847 - struct pg_cache_page_index *page_index = NULL;
848 - Word_t Index;
849 - uint8_t failed_to_reserve;
850 -
851 - fatal_assert(NULL != ret_page_indexp);
766 +void pg_cache_preload(struct rrdeng_query_handle *handle) {
767 + if (unlikely(!handle || !handle->metric))
768 + return;
769
853 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
854 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, id, sizeof(uuid_t));
855 - if (likely(NULL != PValue)) {
856 - *ret_page_indexp = page_index = *PValue;
770 + __atomic_add_fetch(&handle->ctx->inflight_queries, 1, __ATOMIC_RELAXED);
771 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.currently_running_queries, 1, __ATOMIC_RELAXED);
772 + handle->pdc = pdc_get();
773 + handle->pdc->metric = mrg_metric_dup(main_mrg, handle->metric);
774 + handle->pdc->start_time_s = handle->start_time_s;
775 + handle->pdc->end_time_s = handle->end_time_s;
776 + handle->pdc->priority = handle->priority;
777 + handle->pdc->optimal_end_time_s = handle->end_time_s;
778 + handle->pdc->ctx = handle->ctx;
779 + handle->pdc->refcount = 1;
780 + netdata_spinlock_init(&handle->pdc->refcount_spinlock);
781 + completion_init(&handle->pdc->prep_completion);
782 + completion_init(&handle->pdc->page_completion);
783 +
784 + if(ctx_is_available_for_queries(handle->ctx)) {
785 + handle->pdc->refcount++; // we get 1 for the query thread and 1 for the prep thread
786 + rrdeng_enq_cmd(handle->ctx, RRDENG_OPCODE_PREP_QUERY, handle->pdc, NULL, handle->priority);
787 }
858 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
859 - if (NULL == PValue) {
860 - debug(D_RRDENGINE, "%s: No page was found to attempt preload.", __func__);
861 - *ret_page_indexp = NULL;
862 - return 0;
788 + else {
789 + completion_mark_complete(&handle->pdc->prep_completion);
790 + completion_mark_complete(&handle->pdc->page_completion);
791 }
792 +}
793
865 - uv_rwlock_rdlock(&page_index->lock);
866 - descr = find_first_page_in_time_range(page_index, start_time_ut, end_time_ut);
867 - if (NULL == descr) {
868 - uv_rwlock_rdunlock(&page_index->lock);
869 - debug(D_RRDENGINE, "%s: No page was found to attempt preload.", __func__);
870 - *ret_page_indexp = NULL;
871 - return 0;
872 - } else {
873 - Index = (Word_t)(descr->start_time_ut / USEC_PER_SEC);
874 - }
875 - if (page_info_arrayp) {
876 - page_info_array_max_size = PAGE_CACHE_MAX_PRELOAD_PAGES * sizeof(struct rrdeng_page_info);
877 - *page_info_arrayp = mallocz(page_info_array_max_size);
878 - }
794 +/*
795 + * Searches for the first page between start_time and end_time and gets a reference.
796 + * start_time and end_time are inclusive.
797 + * If index is NULL lookup by UUID (id).
798 + */
799 +struct pgc_page *pg_cache_lookup_next(
800 + struct rrdengine_instance *ctx,
801 + PDC *pdc,
802 + time_t now_s,
803 + time_t last_update_every_s,
804 + size_t *entries
805 +) {
806 + if (unlikely(!pdc))
807 + return NULL;
808
880 - for (count = 0, preload_count = 0 ;
881 - descr != NULL && is_page_in_time_range(descr, start_time_ut, end_time_ut) ;
882 - PValue = JudyLNext(page_index->JudyL_array, &Index, PJE0),
883 - descr = unlikely(NULL == PValue) ? NULL : *PValue) {
884 - /* Iterate all pages in range */
809 + rrdeng_prep_wait(pdc);
810
886 - if (unlikely(0 == descr->page_length))
887 - continue;
888 - if (page_info_arrayp) {
889 - if (unlikely(count >= page_info_array_max_size / sizeof(struct rrdeng_page_info))) {
890 - page_info_array_max_size += PAGE_CACHE_MAX_PRELOAD_PAGES * sizeof(struct rrdeng_page_info);
891 - *page_info_arrayp = reallocz(*page_info_arrayp, page_info_array_max_size);
892 - }
893 - (*page_info_arrayp)[count].start_time_ut = descr->start_time_ut;
894 - (*page_info_arrayp)[count].end_time_ut = descr->end_time_ut;
895 - (*page_info_arrayp)[count].page_length = descr->page_length;
896 - }
897 - ++count;
898 -
899 - rrdeng_page_descr_mutex_lock(ctx, descr);
900 - pg_cache_descr = descr->pg_cache_descr;
901 - flags = pg_cache_descr->flags;
902 - if (pg_cache_can_get_unsafe(descr, 0)) {
903 - if (flags & RRD_PAGE_POPULATED) {
904 - /* success */
905 - rrdeng_page_descr_mutex_unlock(ctx, descr);
906 - debug(D_RRDENGINE, "%s: Page was found in memory.", __func__);
907 - continue;
908 - }
909 - }
910 - if (!(flags & RRD_PAGE_POPULATED) && pg_cache_try_get_unsafe(descr, 1)) {
911 - preload_array[preload_count++] = descr;
912 - if (PAGE_CACHE_MAX_PRELOAD_PAGES == preload_count) {
913 - rrdeng_page_descr_mutex_unlock(ctx, descr);
914 - break;
915 - }
916 - }
917 - rrdeng_page_descr_mutex_unlock(ctx, descr);
811 + if (unlikely(!pdc->page_list_JudyL))
812 + return NULL;
813
919 - }
920 - uv_rwlock_rdunlock(&page_index->lock);
814 + usec_t start_ut = now_monotonic_usec();
815 + size_t gaps = 0;
816 + bool waited = false, preloaded;
817 + PGC_PAGE *page = NULL;
818
922 - failed_to_reserve = 0;
923 - for (i = 0 ; i < preload_count && !failed_to_reserve ; ++i) {
924 - struct rrdeng_cmd cmd;
925 - struct rrdeng_page_descr *next;
819 + while(!page) {
820 + bool page_from_pd = false;
821 + preloaded = false;
822 + struct page_details *pd = pdc_find_page_for_time(
823 + pdc->page_list_JudyL, now_s, &gaps,
824 + PDC_PAGE_PROCESSED);
825
927 - descr = preload_array[i];
928 - if (NULL == descr) {
929 - continue;
930 - }
931 - if (!pg_cache_try_reserve_pages(ctx, 1)) {
932 - failed_to_reserve = 1;
826 + if (!pd)
827 break;
934 - }
935 - cmd.opcode = RRDENG_READ_EXTENT;
936 - cmd.read_extent.page_cache_descr[0] = descr;
937 - /* don't use this page again */
938 - preload_array[i] = NULL;
939 - for (j = 0, k = 1 ; j < preload_count ; ++j) {
940 - next = preload_array[j];
941 - if (NULL == next) {
942 - continue;
943 - }
944 - if (descr->extent == next->extent) {
945 - /* same extent, consolidate */
946 - if (!pg_cache_try_reserve_pages(ctx, 1)) {
947 - failed_to_reserve = 1;
948 - break;
949 - }
950 - cmd.read_extent.page_cache_descr[k++] = next;
951 - /* don't use this page again */
952 - preload_array[j] = NULL;
953 - }
954 - }
955 - cmd.read_extent.page_count = k;
956 - rrdeng_enq_cmd(&ctx->worker_config, &cmd);
957 - }
958 - if (failed_to_reserve) {
959 - debug(D_RRDENGINE, "%s: Failed to reserve enough memory, canceling I/O.", __func__);
960 - for (i = 0 ; i < preload_count ; ++i) {
961 - descr = preload_array[i];
962 - if (NULL == descr) {
963 - continue;
964 - }
965 - pg_cache_put(ctx, descr);
966 - }
967 - }
968 - if (!preload_count) {
969 - /* no such page */
970 - debug(D_RRDENGINE, "%s: No page was eligible to attempt preload.", __func__);
971 - }
972 - if (unlikely(0 == count && page_info_arrayp)) {
973 - freez(*page_info_arrayp);
974 - *page_info_arrayp = NULL;
975 - }
976 - return count;
977 -}
828
979 -/*
980 - * Searches for a page and gets a reference.
981 - * When point_in_time is INVALID_TIME get any page.
982 - * If index is NULL lookup by UUID (id).
983 - */
984 -struct rrdeng_page_descr *
985 - pg_cache_lookup(struct rrdengine_instance *ctx, struct pg_cache_page_index *index, uuid_t *id,
986 - usec_t point_in_time_ut)
987 -{
988 - struct page_cache *pg_cache = &ctx->pg_cache;
989 - struct rrdeng_page_descr *descr = NULL;
990 - struct page_cache_descr *pg_cache_descr = NULL;
991 - unsigned long flags;
992 - Pvoid_t *PValue;
993 - struct pg_cache_page_index *page_index = NULL;
994 - Word_t Index;
995 - uint8_t page_not_in_cache;
996 -
997 - if (unlikely(NULL == index)) {
998 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
999 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, id, sizeof(uuid_t));
1000 - if (likely(NULL != PValue)) {
1001 - page_index = *PValue;
1002 - }
1003 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
1004 - if (NULL == PValue) {
1005 - return NULL;
1006 - }
1007 - } else {
1008 - page_index = index;
1009 - }
1010 - pg_cache_reserve_pages(ctx, 1);
1011 -
1012 - page_not_in_cache = 0;
1013 - uv_rwlock_rdlock(&page_index->lock);
1014 - while (1) {
1015 - Index = (Word_t)(point_in_time_ut / USEC_PER_SEC);
1016 - PValue = JudyLLast(page_index->JudyL_array, &Index, PJE0);
1017 - if (likely(NULL != PValue)) {
1018 - descr = *PValue;
1019 - }
1020 - if (NULL == PValue ||
1021 - 0 == descr->page_length ||
1022 - (INVALID_TIME != point_in_time_ut &&
1023 - !is_point_in_time_in_page(descr, point_in_time_ut))) {
1024 - /* non-empty page not found */
1025 - uv_rwlock_rdunlock(&page_index->lock);
1026 -
1027 - pg_cache_release_pages(ctx, 1);
1028 - return NULL;
1029 - }
1030 - rrdeng_page_descr_mutex_lock(ctx, descr);
1031 - pg_cache_descr = descr->pg_cache_descr;
1032 - flags = pg_cache_descr->flags;
1033 - if ((flags & RRD_PAGE_POPULATED) && pg_cache_try_get_unsafe(descr, 0)) {
1034 - /* success */
1035 - rrdeng_page_descr_mutex_unlock(ctx, descr);
1036 - debug(D_RRDENGINE, "%s: Page was found in memory.", __func__);
1037 - break;
1038 - }
1039 - if (!(flags & RRD_PAGE_POPULATED) && pg_cache_try_get_unsafe(descr, 1)) {
1040 - struct rrdeng_cmd cmd;
829 + page = pd->page;
830 + page_from_pd = true;
831 + preloaded = pdc_page_status_check(pd, PDC_PAGE_PRELOADED);
832 + if(!page) {
833 + if(!completion_is_done(&pdc->page_completion)) {
834 + page = pgc_page_get_and_acquire(main_cache, (Word_t)ctx,
835 + pd->metric_id, pd->first_time_s, PGC_SEARCH_EXACT);
836 + page_from_pd = false;
837 + preloaded = pdc_page_status_check(pd, PDC_PAGE_PRELOADED);
838 + }
839
1042 - uv_rwlock_rdunlock(&page_index->lock);
840 + if(!page) {
841 + pdc->completed_jobs =
842 + completion_wait_for_a_job(&pdc->page_completion, pdc->completed_jobs);
843
1044 - cmd.opcode = RRDENG_READ_PAGE;
1045 - cmd.read_page.page_cache_descr = descr;
1046 - rrdeng_enq_cmd(&ctx->worker_config, &cmd);
844 + page = pd->page;
845 + page_from_pd = true;
846 + preloaded = pdc_page_status_check(pd, PDC_PAGE_PRELOADED);
847 + waited = true;
848 + }
849
1048 - debug(D_RRDENGINE, "%s: Waiting for page to be asynchronously read from disk:", __func__);
1049 - if(unlikely(debug_flags & D_RRDENGINE))
1050 - print_page_cache_descr(descr, "", true);
1051 - while (!(pg_cache_descr->flags & RRD_PAGE_POPULATED)) {
1052 - pg_cache_wait_event_unsafe(descr);
850 + if(!page || pdc_page_status_check(pd, PDC_PAGE_FAILED | PDC_PAGE_SKIP | PDC_PAGE_INVALID)) {
851 + page = NULL;
852 + continue;
853 }
1054 - /* success */
1055 - /* Downgrade exclusive reference to allow other readers */
1056 - pg_cache_descr->flags &= ~RRD_PAGE_LOCKED;
1057 - pg_cache_wake_up_waiters_unsafe(descr);
1058 - rrdeng_page_descr_mutex_unlock(ctx, descr);
1059 - rrd_stat_atomic_add(&ctx->stats.pg_cache_misses, 1);
1060 - return descr;
854 }
1062 - uv_rwlock_rdunlock(&page_index->lock);
1063 - debug(D_RRDENGINE, "%s: Waiting for page to be unlocked:", __func__);
1064 - if(unlikely(debug_flags & D_RRDENGINE))
1065 - print_page_cache_descr(descr, "", true);
1066 - if (!(flags & RRD_PAGE_POPULATED))
1067 - page_not_in_cache = 1;
1068 - pg_cache_wait_event_unsafe(descr);
1069 - rrdeng_page_descr_mutex_unlock(ctx, descr);
1070 -
1071 - /* reset scan to find again */
1072 - uv_rwlock_rdlock(&page_index->lock);
1073 - }
1074 - uv_rwlock_rdunlock(&page_index->lock);
855
1076 - if (!(flags & RRD_PAGE_DIRTY))
1077 - pg_cache_replaceQ_set_hot(ctx, descr);
1078 - pg_cache_release_pages(ctx, 1);
1079 - if (page_not_in_cache)
1080 - rrd_stat_atomic_add(&ctx->stats.pg_cache_misses, 1);
1081 - else
1082 - rrd_stat_atomic_add(&ctx->stats.pg_cache_hits, 1);
1083 - return descr;
1084 -}
856 + // we now have page
857
1086 -/*
1087 - * Searches for the first page between start_time and end_time and gets a reference.
1088 - * start_time and end_time are inclusive.
1089 - * If index is NULL lookup by UUID (id).
1090 - */
1091 -struct rrdeng_page_descr *
1092 -pg_cache_lookup_next(struct rrdengine_instance *ctx, struct pg_cache_page_index *index, uuid_t *id,
1093 - usec_t start_time_ut, usec_t end_time_ut)
1094 -{
1095 - struct page_cache *pg_cache = &ctx->pg_cache;
1096 - struct rrdeng_page_descr *descr = NULL;
1097 - struct page_cache_descr *pg_cache_descr = NULL;
1098 - unsigned long flags;
1099 - Pvoid_t *PValue;
1100 - struct pg_cache_page_index *page_index = NULL;
1101 - uint8_t page_not_in_cache;
1102 -
1103 - if (unlikely(NULL == index)) {
1104 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
1105 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, id, sizeof(uuid_t));
1106 - if (likely(NULL != PValue)) {
1107 - page_index = *PValue;
1108 - }
1109 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
1110 - if (NULL == PValue) {
1111 - return NULL;
1112 - }
1113 - } else {
1114 - page_index = index;
1115 - }
1116 - pg_cache_reserve_pages(ctx, 1);
1117 -
1118 - page_not_in_cache = 0;
1119 - uv_rwlock_rdlock(&page_index->lock);
1120 - int retry_count = 0;
1121 - while (1) {
1122 - descr = find_first_page_in_time_range(page_index, start_time_ut, end_time_ut);
1123 - if (NULL == descr || 0 == descr->page_length || retry_count == default_rrdeng_page_fetch_retries) {
1124 - /* non-empty page not found */
1125 - if (retry_count == default_rrdeng_page_fetch_retries)
1126 - error_report("Page cache timeout while waiting for page %p : returning FAIL", descr);
1127 - uv_rwlock_rdunlock(&page_index->lock);
1128 -
1129 - pg_cache_release_pages(ctx, 1);
1130 - return NULL;
858 + time_t page_start_time_s = pgc_page_start_time_s(page);
859 + time_t page_end_time_s = pgc_page_end_time_s(page);
860 + time_t page_update_every_s = pgc_page_update_every_s(page);
861 + size_t page_length = pgc_page_data_size(main_cache, page);
862 +
863 + if(unlikely(page_start_time_s == INVALID_TIME || page_end_time_s == INVALID_TIME)) {
864 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_zero_time_skipped, 1, __ATOMIC_RELAXED);
865 + pgc_page_to_clean_evict_or_release(main_cache, page);
866 + pdc_page_status_set(pd, PDC_PAGE_INVALID | PDC_PAGE_RELEASED);
867 + pd->page = page = NULL;
868 + continue;
869 }
1132 - rrdeng_page_descr_mutex_lock(ctx, descr);
1133 - pg_cache_descr = descr->pg_cache_descr;
1134 - flags = pg_cache_descr->flags;
1135 - if ((flags & RRD_PAGE_POPULATED) && pg_cache_try_get_unsafe(descr, 0)) {
1136 - /* success */
1137 - rrdeng_page_descr_mutex_unlock(ctx, descr);
1138 - debug(D_RRDENGINE, "%s: Page was found in memory.", __func__);
1139 - break;
870 + else if(page_length > RRDENG_BLOCK_SIZE) {
871 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_invalid_size_skipped, 1, __ATOMIC_RELAXED);
872 + pgc_page_to_clean_evict_or_release(main_cache, page);
873 + pdc_page_status_set(pd, PDC_PAGE_INVALID | PDC_PAGE_RELEASED);
874 + pd->page = page = NULL;
875 + continue;
876 }
1141 - if (!(flags & RRD_PAGE_POPULATED) && pg_cache_try_get_unsafe(descr, 1)) {
1142 - struct rrdeng_cmd cmd;
877 + else {
878 + if (unlikely(page_update_every_s <= 0 || page_update_every_s > 86400)) {
879 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_invalid_update_every_fixed, 1, __ATOMIC_RELAXED);
880 + pd->update_every_s = page_update_every_s = pgc_page_fix_update_every(page, last_update_every_s);
881 + }
882
1144 - uv_rwlock_rdunlock(&page_index->lock);
883 + size_t entries_by_size = page_entries_by_size(page_length, PAGE_POINT_CTX_SIZE_BYTES(ctx));
884 + size_t entries_by_time = page_entries_by_time(page_start_time_s, page_end_time_s, page_update_every_s);
885 + if(unlikely(entries_by_size < entries_by_time)) {
886 + time_t fixed_page_end_time_s = (time_t)(page_start_time_s + (entries_by_size - 1) * page_update_every_s);
887 + pd->last_time_s = page_end_time_s = pgc_page_fix_end_time_s(page, fixed_page_end_time_s);
888 + entries_by_time = (page_end_time_s - (page_start_time_s - page_update_every_s)) / page_update_every_s;
889
1146 - cmd.opcode = RRDENG_READ_PAGE;
1147 - cmd.read_page.page_cache_descr = descr;
1148 - rrdeng_enq_cmd(&ctx->worker_config, &cmd);
890 + internal_fatal(entries_by_size != entries_by_time, "DBENGINE: wrong entries by time again!");
891
1150 - debug(D_RRDENGINE, "%s: Waiting for page to be asynchronously read from disk:", __func__);
1151 - if(unlikely(debug_flags & D_RRDENGINE))
1152 - print_page_cache_descr(descr, "", true);
1153 - while (!(pg_cache_descr->flags & RRD_PAGE_POPULATED)) {
1154 - pg_cache_wait_event_unsafe(descr);
892 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_invalid_entries_fixed, 1, __ATOMIC_RELAXED);
893 }
1156 - /* success */
1157 - /* Downgrade exclusive reference to allow other readers */
1158 - pg_cache_descr->flags &= ~RRD_PAGE_LOCKED;
1159 - pg_cache_wake_up_waiters_unsafe(descr);
1160 - rrdeng_page_descr_mutex_unlock(ctx, descr);
1161 - rrd_stat_atomic_add(&ctx->stats.pg_cache_misses, 1);
1162 - return descr;
894 + *entries = entries_by_time;
895 }
1164 - uv_rwlock_rdunlock(&page_index->lock);
1165 - debug(D_RRDENGINE, "%s: Waiting for page to be unlocked:", __func__);
1166 - if(unlikely(debug_flags & D_RRDENGINE))
1167 - print_page_cache_descr(descr, "", true);
1168 - if (!(flags & RRD_PAGE_POPULATED))
1169 - page_not_in_cache = 1;
1170 -
1171 - if (pg_cache_timedwait_event_unsafe(descr, default_rrdeng_page_fetch_timeout) == UV_ETIMEDOUT) {
1172 - error_report("Page cache timeout while waiting for page %p : retry count = %d", descr, retry_count);
1173 - ++retry_count;
896 +
897 + if(unlikely(page_end_time_s < now_s)) {
898 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_past_time_skipped, 1, __ATOMIC_RELAXED);
899 + pgc_page_release(main_cache, page);
900 + pdc_page_status_set(pd, PDC_PAGE_SKIP | PDC_PAGE_RELEASED);
901 + pd->page = page = NULL;
902 + continue;
903 }
1175 - rrdeng_page_descr_mutex_unlock(ctx, descr);
904
1177 - /* reset scan to find again */
1178 - uv_rwlock_rdlock(&page_index->lock);
905 + if(page_from_pd)
906 + // PDC_PAGE_RELEASED is for pdc_destroy() to not release the page twice - the caller will release it
907 + pdc_page_status_set(pd, PDC_PAGE_RELEASED | PDC_PAGE_PROCESSED);
908 + else
909 + pdc_page_status_set(pd, PDC_PAGE_PROCESSED);
910 }
1180 - uv_rwlock_rdunlock(&page_index->lock);
911
1182 - if (!(flags & RRD_PAGE_DIRTY))
1183 - pg_cache_replaceQ_set_hot(ctx, descr);
1184 - pg_cache_release_pages(ctx, 1);
1185 - if (page_not_in_cache)
1186 - rrd_stat_atomic_add(&ctx->stats.pg_cache_misses, 1);
1187 - else
1188 - rrd_stat_atomic_add(&ctx->stats.pg_cache_hits, 1);
1189 - return descr;
1190 -}
912 + if(gaps && !pdc->executed_with_gaps)
913 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.queries_executed_with_gaps, 1, __ATOMIC_RELAXED);
914 + pdc->executed_with_gaps = +gaps;
915
1192 -struct pg_cache_page_index *create_page_index(uuid_t *id, struct rrdengine_instance *ctx)
1193 -{
1194 - struct pg_cache_page_index *page_index;
1195 -
1196 - page_index = mallocz(sizeof(*page_index));
1197 - page_index->JudyL_array = (Pvoid_t) NULL;
1198 - uuid_copy(page_index->id, *id);
1199 - fatal_assert(0 == uv_rwlock_init(&page_index->lock));
1200 - page_index->oldest_time_ut = INVALID_TIME;
1201 - page_index->latest_time_ut = INVALID_TIME;
1202 - page_index->prev = NULL;
1203 - page_index->page_count = 0;
1204 - page_index->refcount = 0;
1205 - page_index->writers = 0;
1206 - page_index->ctx = ctx;
1207 - page_index->latest_update_every_s = default_rrd_update_every;
1208 -
1209 - return page_index;
1210 -}
916 + if(page) {
917 + if(waited)
918 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.page_next_wait_loaded, 1, __ATOMIC_RELAXED);
919 + else
920 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.page_next_nowait_loaded, 1, __ATOMIC_RELAXED);
921 + }
922 + else {
923 + if(waited)
924 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.page_next_wait_failed, 1, __ATOMIC_RELAXED);
925 + else
926 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.page_next_nowait_failed, 1, __ATOMIC_RELAXED);
927 + }
928
1212 -static void init_metrics_index(struct rrdengine_instance *ctx)
1213 -{
1214 - struct page_cache *pg_cache = &ctx->pg_cache;
929 + if(waited) {
930 + if(preloaded)
931 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.query_time_to_slow_preload_next_page, now_monotonic_usec() - start_ut, __ATOMIC_RELAXED);
932 + else
933 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.query_time_to_slow_disk_next_page, now_monotonic_usec() - start_ut, __ATOMIC_RELAXED);
934 + }
935 + else {
936 + if(preloaded)
937 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.query_time_to_fast_preload_next_page, now_monotonic_usec() - start_ut, __ATOMIC_RELAXED);
938 + else
939 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.query_time_to_fast_disk_next_page, now_monotonic_usec() - start_ut, __ATOMIC_RELAXED);
940 + }
941
1216 - pg_cache->metrics_index.JudyHS_array = (Pvoid_t) NULL;
1217 - pg_cache->metrics_index.last_page_index = NULL;
1218 - fatal_assert(0 == uv_rwlock_init(&pg_cache->metrics_index.lock));
942 + return page;
943 }
944
1221 -static void init_replaceQ(struct rrdengine_instance *ctx)
1222 -{
1223 - struct page_cache *pg_cache = &ctx->pg_cache;
945 +void pgc_open_add_hot_page(Word_t section, Word_t metric_id, time_t start_time_s, time_t end_time_s, time_t update_every_s,
946 + struct rrdengine_datafile *datafile, uint64_t extent_offset, unsigned extent_size, uint32_t page_length) {
947 +
948 + if(!datafile_acquire(datafile, DATAFILE_ACQUIRE_OPEN_CACHE)) // for open cache item
949 + fatal("DBENGINE: cannot acquire datafile to put page in open cache");
950 +
951 + struct extent_io_data ext_io_data = {
952 + .file = datafile->file,
953 + .fileno = datafile->fileno,
954 + .pos = extent_offset,
955 + .bytes = extent_size,
956 + .page_length = page_length
957 + };
958 +
959 + PGC_ENTRY page_entry = {
960 + .hot = true,
961 + .section = section,
962 + .metric_id = metric_id,
963 + .start_time_s = start_time_s,
964 + .end_time_s = end_time_s,
965 + .update_every_s = update_every_s,
966 + .size = 0,
967 + .data = datafile,
968 + .custom_data = (uint8_t *) &ext_io_data,
969 + };
970 +
971 + internal_fatal(!datafile->fileno, "DBENGINE: datafile supplied does not have a number");
972 +
973 + bool added = true;
974 + PGC_PAGE *page = pgc_page_add_and_acquire(open_cache, page_entry, &added);
975 + int tries = 100;
976 + while(!added && page_entry.end_time_s > pgc_page_end_time_s(page) && tries--) {
977 + pgc_page_to_clean_evict_or_release(open_cache, page);
978 + page = pgc_page_add_and_acquire(open_cache, page_entry, &added);
979 + }
980
1225 - pg_cache->replaceQ.head = NULL;
1226 - pg_cache->replaceQ.tail = NULL;
1227 - fatal_assert(0 == uv_rwlock_init(&pg_cache->replaceQ.lock));
1228 -}
981 + if(!added) {
982 + datafile_release(datafile, DATAFILE_ACQUIRE_OPEN_CACHE);
983
1230 -static void init_committed_page_index(struct rrdengine_instance *ctx)
1231 -{
1232 - struct page_cache *pg_cache = &ctx->pg_cache;
984 + internal_fatal(page_entry.end_time_s > pgc_page_end_time_s(page),
985 + "DBENGINE: cannot add longer page to open cache");
986 + }
987
1234 - pg_cache->committed_page_index.JudyL_array = (Pvoid_t) NULL;
1235 - fatal_assert(0 == uv_rwlock_init(&pg_cache->committed_page_index.lock));
1236 - pg_cache->committed_page_index.latest_corr_id = 0;
1237 - pg_cache->committed_page_index.nr_committed_pages = 0;
988 + pgc_page_release(open_cache, (PGC_PAGE *)page);
989 }
990
1240 -void init_page_cache(struct rrdengine_instance *ctx)
1241 -{
1242 - struct page_cache *pg_cache = &ctx->pg_cache;
991 +size_t dynamic_open_cache_size(void) {
992 + size_t main_cache_size = pgc_get_wanted_cache_size(main_cache);
993 + size_t target_size = main_cache_size / 100 * 5;
994
1244 - pg_cache->page_descriptors = 0;
1245 - pg_cache->populated_pages = 0;
1246 - fatal_assert(0 == uv_rwlock_init(&pg_cache->pg_cache_rwlock));
995 + if(target_size < 2 * 1024 * 1024)
996 + target_size = 2 * 1024 * 1024;
997
1248 - init_metrics_index(ctx);
1249 - init_replaceQ(ctx);
1250 - init_committed_page_index(ctx);
998 + return target_size;
999 }
1000
1253 -void free_page_cache(struct rrdengine_instance *ctx)
1254 -{
1255 - struct page_cache *pg_cache = &ctx->pg_cache;
1256 - Pvoid_t *PValue;
1257 - struct pg_cache_page_index *page_index, *prev_page_index;
1258 - Word_t Index;
1259 - struct rrdeng_page_descr *descr;
1260 - struct page_cache_descr *pg_cache_descr;
1261 -
1262 - // if we are exiting, the OS will recover all memory so do not slow down the shutdown process
1263 - // Do the cleanup if we are compiling with NETDATA_INTERNAL_CHECKS
1264 - // This affects the reporting of dbengine statistics which are available in real time
1265 - // via the /api/v1/dbengine_stats endpoint
1266 -#ifndef NETDATA_DBENGINE_FREE
1267 - if (netdata_exit)
1268 - return;
1269 -#endif
1270 - Word_t metrics_index_bytes = 0, pages_index_bytes = 0, pages_dirty_index_bytes = 0;
1001 +size_t dynamic_extent_cache_size(void) {
1002 + size_t main_cache_size = pgc_get_wanted_cache_size(main_cache);
1003 + size_t target_size = main_cache_size / 100 * 5;
1004
1272 - /* Free committed page index */
1273 - pages_dirty_index_bytes = JudyLFreeArray(&pg_cache->committed_page_index.JudyL_array, PJE0);
1274 - fatal_assert(NULL == pg_cache->committed_page_index.JudyL_array);
1005 + if(target_size < 3 * 1024 * 1024)
1006 + target_size = 3 * 1024 * 1024;
1007
1276 - for (page_index = pg_cache->metrics_index.last_page_index ;
1277 - page_index != NULL ;
1278 - page_index = prev_page_index) {
1008 + return target_size;
1009 +}
1010
1280 - prev_page_index = page_index->prev;
1011 +void init_page_cache(void)
1012 +{
1013 + static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER;
1014 + static bool initialized = false;
1015
1282 - /* Find first page in range */
1283 - Index = (Word_t) 0;
1284 - PValue = JudyLFirst(page_index->JudyL_array, &Index, PJE0);
1285 - descr = unlikely(NULL == PValue) ? NULL : *PValue;
1016 + netdata_spinlock_lock(&spinlock);
1017 + if (!initialized) {
1018 + initialized = true;
1019
1287 - while (descr != NULL) {
1288 - /* Iterate all page descriptors of this metric */
1020 + main_mrg = mrg_create();
1021
1290 - if (descr->pg_cache_descr_state & PG_CACHE_DESCR_ALLOCATED) {
1291 - /* Check rrdenglocking.c */
1292 - pg_cache_descr = descr->pg_cache_descr;
1293 - if (pg_cache_descr->flags & RRD_PAGE_POPULATED) {
1294 - dbengine_page_free(pg_cache_descr->page);
1295 - }
1296 - rrdeng_destroy_pg_cache_descr(ctx, pg_cache_descr);
1297 - }
1298 - rrdeng_page_descr_freez(descr);
1022 + size_t target_cache_size = (size_t)default_rrdeng_page_cache_mb * 1024ULL * 1024ULL;
1023 + size_t main_cache_size = (target_cache_size / 100) * 95;
1024 + size_t open_cache_size = 0;
1025 + size_t extent_cache_size = (target_cache_size / 100) * 5;
1026
1300 - PValue = JudyLNext(page_index->JudyL_array, &Index, PJE0);
1301 - descr = unlikely(NULL == PValue) ? NULL : *PValue;
1027 + if(extent_cache_size < 3 * 1024 * 1024) {
1028 + extent_cache_size = 3 * 1024 * 1024;
1029 + main_cache_size = target_cache_size - extent_cache_size;
1030 }
1031
1304 - /* Free page index */
1305 - pages_index_bytes += JudyLFreeArray(&page_index->JudyL_array, PJE0);
1306 - fatal_assert(NULL == page_index->JudyL_array);
1307 - freez(page_index);
1032 + main_cache = pgc_create(
1033 + main_cache_size,
1034 + main_cache_free_clean_page_callback,
1035 + (size_t) rrdeng_pages_per_extent,
1036 + main_cache_flush_dirty_page_callback,
1037 + 20, //
1038 + 1000, //
1039 + 5, // don't delay too much other threads
1040 + PGC_OPTIONS_AUTOSCALE, // AUTOSCALE = 2x max hot pages
1041 + 0, // 0 = as many as the system cpus
1042 + 0
1043 + );
1044 +
1045 + open_cache = pgc_create(
1046 + open_cache_size, // the default is 1MB
1047 + open_cache_free_clean_page_callback,
1048 + 1,
1049 + open_cache_flush_dirty_page_callback,
1050 + 20, //
1051 + 1000, //
1052 + 3, // don't delay too much other threads
1053 + PGC_OPTIONS_AUTOSCALE | PGC_OPTIONS_EVICT_PAGES_INLINE | PGC_OPTIONS_FLUSH_PAGES_INLINE,
1054 + 0, // 0 = as many as the system cpus
1055 + sizeof(struct extent_io_data)
1056 + );
1057 + pgc_set_dynamic_target_cache_size_callback(open_cache, dynamic_open_cache_size);
1058 +
1059 + extent_cache = pgc_create(
1060 + extent_cache_size,
1061 + extent_cache_free_clean_page_callback,
1062 + 1,
1063 + extent_cache_flush_dirty_page_callback,
1064 + 5, //
1065 + 100, //
1066 + 2, // don't delay too much other threads
1067 + PGC_OPTIONS_AUTOSCALE | PGC_OPTIONS_EVICT_PAGES_INLINE | PGC_OPTIONS_FLUSH_PAGES_INLINE,
1068 + 0, // 0 = as many as the system cpus
1069 + 0
1070 + );
1071 + pgc_set_dynamic_target_cache_size_callback(extent_cache, dynamic_extent_cache_size);
1072 }
1309 - /* Free metrics index */
1310 - metrics_index_bytes = JudyHSFreeArray(&pg_cache->metrics_index.JudyHS_array, PJE0);
1311 - fatal_assert(NULL == pg_cache->metrics_index.JudyHS_array);
1312 - info("Freed %lu bytes of memory from page cache.", pages_dirty_index_bytes + pages_index_bytes + metrics_index_bytes);
1073 +
1074 + netdata_spinlock_unlock(&spinlock);
1075 }
database/engine/pagecache.h
+30 -216
@@ -5,66 +5,39 @@
5
6 #include "rrdengine.h"
7
8 +extern struct mrg *main_mrg;
9 +extern struct pgc *main_cache;
10 +extern struct pgc *open_cache;
11 +extern struct pgc *extent_cache;
12 +
13 /* Forward declarations */
14 struct rrdengine_instance;
10 -struct extent_info;
11 -struct rrdeng_page_descr;
15
16 #define INVALID_TIME (0)
17 #define MAX_PAGE_CACHE_FETCH_RETRIES (3)
18 #define PAGE_CACHE_FETCH_WAIT_TIMEOUT (3)
19
17 -/* Page flags */
18 -#define RRD_PAGE_DIRTY (1LU << 0)
19 -#define RRD_PAGE_LOCKED (1LU << 1)
20 -#define RRD_PAGE_READ_PENDING (1LU << 2)
21 -#define RRD_PAGE_WRITE_PENDING (1LU << 3)
22 -#define RRD_PAGE_POPULATED (1LU << 4)
23 -
24 -struct page_cache_descr {
25 - struct rrdeng_page_descr *descr; /* parent descriptor */
26 - void *page;
27 - unsigned long flags;
28 - struct page_cache_descr *prev; /* LRU */
29 - struct page_cache_descr *next; /* LRU */
30 -
31 - unsigned refcnt;
32 - uv_mutex_t mutex; /* always take it after the page cache lock or after the commit lock */
33 - uv_cond_t cond;
34 - unsigned waiters;
35 -};
20 +extern struct rrdeng_cache_efficiency_stats rrdeng_cache_efficiency_stats;
21
37 -/* Page cache descriptor flags, state = 0 means no descriptor */
38 -#define PG_CACHE_DESCR_ALLOCATED (1LU << 0)
39 -#define PG_CACHE_DESCR_DESTROY (1LU << 1)
40 -#define PG_CACHE_DESCR_LOCKED (1LU << 2)
41 -#define PG_CACHE_DESCR_SHIFT (3)
42 -#define PG_CACHE_DESCR_USERS_MASK (((unsigned long)-1) << PG_CACHE_DESCR_SHIFT)
43 -#define PG_CACHE_DESCR_FLAGS_MASK (((unsigned long)-1) >> (BITS_PER_ULONG - PG_CACHE_DESCR_SHIFT))
44 -
45 -/*
46 - * Page cache descriptor state bits (works for both 32-bit and 64-bit architectures):
47 - *
48 - * 63 ... 31 ... 3 | 2 | 1 | 0|
49 - * -----------------------------+------------+------------+-----------|
50 - * number of descriptor users | DESTROY | LOCKED | ALLOCATED |
51 - */
52 -struct rrdeng_page_descr {
53 - uuid_t *id; /* never changes */
54 - struct extent_info *extent;
55 -
56 - /* points to ephemeral page cache descriptor if the page resides in the cache */
57 - struct page_cache_descr *pg_cache_descr;
58 -
59 - /* Compare-And-Swap target for page cache descriptor allocation algorithm */
60 - volatile unsigned long pg_cache_descr_state;
61 -
62 - /* page information */
22 +struct page_descr_with_data {
23 + uuid_t *id;
24 + Word_t metric_id;
25 usec_t start_time_ut;
26 usec_t end_time_ut;
65 - uint32_t update_every_s:24;
27 uint8_t type;
28 + uint32_t update_every_s;
29 uint32_t page_length;
30 + uint8_t page[RRDENG_BLOCK_SIZE];
31 +
32 + struct {
33 + struct page_descr_with_data *prev;
34 + struct page_descr_with_data *next;
35 + } link;
36 +
37 + struct {
38 + struct page_descr_with_data *prev;
39 + struct page_descr_with_data *next;
40 + } cache;
41 };
42
43 #define PAGE_INFO_SCRATCH_SZ (8)
@@ -76,179 +49,20 @@ struct rrdeng_page_info {
49 uint32_t page_length;
50 };
51
79 -/* returns 1 for success, 0 for failure */
80 -typedef int pg_cache_page_info_filter_t(struct rrdeng_page_descr *);
81 -
82 -#define PAGE_CACHE_MAX_PRELOAD_PAGES (256)
83 -
52 struct pg_alignment {
85 - uint32_t page_length;
53 + uint32_t page_position;
54 uint32_t refcount;
55 };
56
89 -/* maps time ranges to pages */
90 -struct pg_cache_page_index {
91 - uuid_t id;
92 - /*
93 - * care: JudyL_array indices are converted from useconds to seconds to fit in one word in 32-bit architectures
94 - * TODO: examine if we want to support better granularity than seconds
95 - */
96 - Pvoid_t JudyL_array;
97 - Word_t page_count;
98 - unsigned short refcount;
99 - unsigned short writers;
100 - uv_rwlock_t lock;
101 -
102 - /*
103 - * Only one effective writer, data deletion workqueue.
104 - * It's also written during the DB loading phase.
105 - */
106 - usec_t oldest_time_ut;
107 -
108 - /*
109 - * Only one effective writer, data collection thread.
110 - * It's also written by the data deletion workqueue when data collection is disabled for this metric.
111 - */
112 - usec_t latest_time_ut;
113 -
114 - struct rrdengine_instance *ctx;
115 - uint32_t latest_update_every_s;
116 -
117 - struct pg_cache_page_index *prev;
118 -};
119 -
120 -/* maps UUIDs to page indices */
121 -struct pg_cache_metrics_index {
122 - uv_rwlock_t lock;
123 - Pvoid_t JudyHS_array;
124 - struct pg_cache_page_index *last_page_index;
125 -};
126 -
127 -/* gathers dirty pages to be written on disk */
128 -struct pg_cache_committed_page_index {
129 - uv_rwlock_t lock;
130 -
131 - Pvoid_t JudyL_array;
57 +struct rrdeng_query_handle;
58 +struct page_details_control;
59
133 - /*
134 - * Dirty page correlation ID is a hint. Dirty pages that are correlated should have
135 - * a small correlation ID difference. Dirty pages in memory should never have the
136 - * same ID at the same time for correctness.
137 - */
138 - Word_t latest_corr_id;
139 -
140 - unsigned nr_committed_pages;
141 -};
142 -
143 -/*
144 - * Gathers populated pages to be evicted.
145 - * Relies on page cache descriptors being there as it uses their memory.
146 - */
147 -struct pg_cache_replaceQ {
148 - uv_rwlock_t lock; /* LRU lock */
149 -
150 - struct page_cache_descr *head; /* LRU */
151 - struct page_cache_descr *tail; /* MRU */
152 -};
153 -
154 -struct page_cache { /* TODO: add statistics */
155 - uv_rwlock_t pg_cache_rwlock; /* page cache lock */
156 -
157 - struct pg_cache_metrics_index metrics_index;
158 - struct pg_cache_committed_page_index committed_page_index;
159 - struct pg_cache_replaceQ replaceQ;
160 -
161 - unsigned page_descriptors;
162 - unsigned populated_pages;
163 -};
164 -
165 -void pg_cache_wake_up_waiters_unsafe(struct rrdeng_page_descr *descr);
166 -void pg_cache_wake_up_waiters(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
167 -void pg_cache_wait_event_unsafe(struct rrdeng_page_descr *descr);
168 -unsigned long pg_cache_wait_event(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
169 -void pg_cache_replaceQ_insert(struct rrdengine_instance *ctx,
170 - struct rrdeng_page_descr *descr);
171 -void pg_cache_replaceQ_delete(struct rrdengine_instance *ctx,
172 - struct rrdeng_page_descr *descr);
173 -void pg_cache_replaceQ_set_hot(struct rrdengine_instance *ctx,
174 - struct rrdeng_page_descr *descr);
175 -struct rrdeng_page_descr *pg_cache_create_descr(void);
176 -int pg_cache_try_get_unsafe(struct rrdeng_page_descr *descr, int exclusive_access);
177 -void pg_cache_put_unsafe(struct rrdeng_page_descr *descr);
178 -void pg_cache_put(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
179 -void pg_cache_insert(struct rrdengine_instance *ctx, struct pg_cache_page_index *index,
180 - struct rrdeng_page_descr *descr);
181 -uint8_t pg_cache_punch_hole(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr,
182 - uint8_t remove_dirty, uint8_t is_exclusive_holder, uuid_t *metric_id);
183 -usec_t pg_cache_oldest_time_in_range(struct rrdengine_instance *ctx, uuid_t *id,
184 - usec_t start_time_ut, usec_t end_time_ut);
185 -void pg_cache_get_filtered_info_prev(struct rrdengine_instance *ctx, struct pg_cache_page_index *page_index,
186 - usec_t point_in_time_ut, pg_cache_page_info_filter_t *filter,
187 - struct rrdeng_page_info *page_info);
188 -struct rrdeng_page_descr *pg_cache_lookup_unpopulated_and_lock(struct rrdengine_instance *ctx, uuid_t *id,
189 - usec_t start_time_ut);
190 -unsigned
191 - pg_cache_preload(struct rrdengine_instance *ctx, uuid_t *id, usec_t start_time_ut, usec_t end_time_ut,
192 - struct rrdeng_page_info **page_info_arrayp, struct pg_cache_page_index **ret_page_indexp);
193 -struct rrdeng_page_descr *
194 - pg_cache_lookup(struct rrdengine_instance *ctx, struct pg_cache_page_index *index, uuid_t *id,
195 - usec_t point_in_time_ut);
196 -struct rrdeng_page_descr *
197 - pg_cache_lookup_next(struct rrdengine_instance *ctx, struct pg_cache_page_index *index, uuid_t *id,
198 - usec_t start_time_ut, usec_t end_time_ut);
199 -struct pg_cache_page_index *create_page_index(uuid_t *id, struct rrdengine_instance *ctx);
200 -void init_page_cache(struct rrdengine_instance *ctx);
201 -void free_page_cache(struct rrdengine_instance *ctx);
202 -void pg_cache_add_new_metric_time(struct pg_cache_page_index *page_index, struct rrdeng_page_descr *descr);
203 -void pg_cache_update_metric_times(struct pg_cache_page_index *page_index);
204 -unsigned long pg_cache_hard_limit(struct rrdengine_instance *ctx);
205 -unsigned long pg_cache_soft_limit(struct rrdengine_instance *ctx);
206 -unsigned long pg_cache_committed_hard_limit(struct rrdengine_instance *ctx);
207 -
208 -void rrdeng_page_descr_aral_go_singlethreaded(void);
209 -void rrdeng_page_descr_aral_go_multithreaded(void);
210 -void rrdeng_page_descr_use_malloc(void);
211 -void rrdeng_page_descr_use_mmap(void);
212 -bool rrdeng_page_descr_is_mmap(void);
213 -struct rrdeng_page_descr *rrdeng_page_descr_mallocz(void);
214 -void rrdeng_page_descr_freez(struct rrdeng_page_descr *descr);
215 -
216 -static inline void
217 - pg_cache_atomic_get_pg_info(struct rrdeng_page_descr *descr, usec_t *end_time_ut_p, uint32_t *page_lengthp)
218 -{
219 - usec_t end_time_ut, old_end_time_ut;
220 - uint32_t page_length;
60 +void rrdeng_prep_wait(struct page_details_control *pdc);
61 +void rrdeng_prep_query(struct page_details_control *pdc);
62 +void pg_cache_preload(struct rrdeng_query_handle *handle);
63 +struct pgc_page *pg_cache_lookup_next(struct rrdengine_instance *ctx, struct page_details_control *pdc, time_t now_s, time_t last_update_every_s, size_t *entries);
64 +void init_page_cache(void);
65
222 - if (NULL == descr->extent) {
223 - /* this page is currently being modified, get consistent info locklessly */
224 - do {
225 - end_time_ut = descr->end_time_ut;
226 - __sync_synchronize();
227 - old_end_time_ut = end_time_ut;
228 - page_length = descr->page_length;
229 - __sync_synchronize();
230 - end_time_ut = descr->end_time_ut;
231 - __sync_synchronize();
232 - } while ((end_time_ut != old_end_time_ut || (end_time_ut & 1) != 0));
233 -
234 - *end_time_ut_p = end_time_ut;
235 - *page_lengthp = page_length;
236 - } else {
237 - *end_time_ut_p = descr->end_time_ut;
238 - *page_lengthp = descr->page_length;
239 - }
240 -}
241 -
242 -/* The caller must hold a reference to the page and must have already set the new data */
243 -static inline void pg_cache_atomic_set_pg_info(struct rrdeng_page_descr *descr, usec_t end_time_ut, uint32_t page_length)
244 -{
245 - fatal_assert(!(end_time_ut & 1));
246 - __sync_synchronize();
247 - descr->end_time_ut |= 1; /* mark start of uncertainty period by adding 1 microsecond */
248 - __sync_synchronize();
249 - descr->page_length = page_length;
250 - __sync_synchronize();
251 - descr->end_time_ut = end_time_ut; /* mark end of uncertainty period */
252 -}
66 +void pgc_open_add_hot_page(Word_t section, Word_t metric_id, time_t start_time_s, time_t end_time_s, time_t update_every_s, struct rrdengine_datafile *datafile, uint64_t extent_offset, unsigned extent_size, uint32_t page_length);
67
68 #endif /* NETDATA_PAGECACHE_H */
database/engine/pdc.c new
+1221
@@ -0,0 +1,1221 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +#define NETDATA_RRD_INTERNALS
3 +#include "pdc.h"
4 +
5 +struct extent_page_details_list {
6 + uv_file file;
7 + uint64_t extent_offset;
8 + uint32_t extent_size;
9 + unsigned number_of_pages_in_JudyL;
10 + Pvoid_t page_details_by_metric_id_JudyL;
11 + struct page_details_control *pdc;
12 + struct rrdengine_datafile *datafile;
13 +
14 + struct {
15 + struct extent_page_details_list *prev;
16 + struct extent_page_details_list *next;
17 + } cache;
18 +};
19 +
20 +typedef struct datafile_extent_offset_list {
21 + uv_file file;
22 + unsigned fileno;
23 + Pvoid_t extent_pd_list_by_extent_offset_JudyL;
24 +
25 + struct {
26 + struct datafile_extent_offset_list *prev;
27 + struct datafile_extent_offset_list *next;
28 + } cache;
29 +} DEOL;
30 +
31 +// ----------------------------------------------------------------------------
32 +// PDC cache
33 +
34 +static struct {
35 + struct {
36 + SPINLOCK spinlock;
37 + PDC *available_items;
38 + size_t available;
39 + } protected;
40 +
41 + struct {
42 + size_t allocated;
43 + } atomics;
44 +} pdc_globals = {
45 + .protected = {
46 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
47 + .available_items = NULL,
48 + .available = 0,
49 + },
50 + .atomics = {
51 + .allocated = 0,
52 + },
53 +};
54 +
55 +void pdc_cleanup(void) {
56 + netdata_spinlock_lock(&pdc_globals.protected.spinlock);
57 +
58 + while(pdc_globals.protected.available_items && pdc_globals.protected.available > (size_t)libuv_worker_threads) {
59 + PDC *item = pdc_globals.protected.available_items;
60 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(pdc_globals.protected.available_items, item, cache.prev, cache.next);
61 + freez(item);
62 + pdc_globals.protected.available--;
63 + __atomic_sub_fetch(&pdc_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
64 + }
65 +
66 + netdata_spinlock_unlock(&pdc_globals.protected.spinlock);
67 +}
68 +
69 +PDC *pdc_get(void) {
70 + PDC *pdc = NULL;
71 +
72 + netdata_spinlock_lock(&pdc_globals.protected.spinlock);
73 +
74 + if(likely(pdc_globals.protected.available_items)) {
75 + pdc = pdc_globals.protected.available_items;
76 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(pdc_globals.protected.available_items, pdc, cache.prev, cache.next);
77 + pdc_globals.protected.available--;
78 + }
79 +
80 + netdata_spinlock_unlock(&pdc_globals.protected.spinlock);
81 +
82 + if(unlikely(!pdc)) {
83 + pdc = mallocz(sizeof(PDC));
84 + __atomic_add_fetch(&pdc_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
85 + }
86 +
87 + memset(pdc, 0, sizeof(PDC));
88 + return pdc;
89 +}
90 +
91 +static void pdc_release(PDC *pdc) {
92 + if(unlikely(!pdc)) return;
93 +
94 + netdata_spinlock_lock(&pdc_globals.protected.spinlock);
95 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(pdc_globals.protected.available_items, pdc, cache.prev, cache.next);
96 + pdc_globals.protected.available++;
97 + netdata_spinlock_unlock(&pdc_globals.protected.spinlock);
98 +}
99 +
100 +size_t pdc_cache_size(void) {
101 + return __atomic_load_n(&pdc_globals.atomics.allocated, __ATOMIC_RELAXED) * sizeof(PDC);
102 +}
103 +
104 +// ----------------------------------------------------------------------------
105 +// PD cache
106 +
107 +static struct {
108 + struct {
109 + SPINLOCK spinlock;
110 + struct page_details *available_items;
111 + size_t available;
112 + } protected;
113 +
114 + struct {
115 + size_t allocated;
116 + } atomics;
117 +} page_details_globals = {
118 + .protected = {
119 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
120 + .available_items = NULL,
121 + .available = 0,
122 + },
123 + .atomics = {
124 + .allocated = 0,
125 + },
126 +};
127 +
128 +void page_details_cleanup(void) {
129 + netdata_spinlock_lock(&page_details_globals.protected.spinlock);
130 +
131 + while(page_details_globals.protected.available_items && page_details_globals.protected.available > (size_t)libuv_worker_threads * 2) {
132 + struct page_details *item = page_details_globals.protected.available_items;
133 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(page_details_globals.protected.available_items, item, cache.prev, cache.next);
134 + freez(item);
135 + page_details_globals.protected.available--;
136 + __atomic_sub_fetch(&page_details_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
137 + }
138 +
139 + netdata_spinlock_unlock(&page_details_globals.protected.spinlock);
140 +}
141 +
142 +struct page_details *page_details_get(void) {
143 + struct page_details *pd = NULL;
144 +
145 + netdata_spinlock_lock(&page_details_globals.protected.spinlock);
146 +
147 + if(likely(page_details_globals.protected.available_items)) {
148 + pd = page_details_globals.protected.available_items;
149 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(page_details_globals.protected.available_items, pd, cache.prev, cache.next);
150 + page_details_globals.protected.available--;
151 + }
152 +
153 + netdata_spinlock_unlock(&page_details_globals.protected.spinlock);
154 +
155 + if(unlikely(!pd)) {
156 + pd = mallocz(sizeof(struct page_details));
157 + __atomic_add_fetch(&page_details_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
158 + }
159 +
160 + memset(pd, 0, sizeof(struct page_details));
161 + return pd;
162 +}
163 +
164 +static void page_details_release(struct page_details *pd) {
165 + if(unlikely(!pd)) return;
166 +
167 + netdata_spinlock_lock(&page_details_globals.protected.spinlock);
168 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(page_details_globals.protected.available_items, pd, cache.prev, cache.next);
169 + page_details_globals.protected.available++;
170 + netdata_spinlock_unlock(&page_details_globals.protected.spinlock);
171 +}
172 +
173 +size_t pd_cache_size(void) {
174 + return __atomic_load_n(&page_details_globals.atomics.allocated, __ATOMIC_RELAXED) * sizeof(struct page_details);
175 +}
176 +
177 +// ----------------------------------------------------------------------------
178 +// epdl cache
179 +
180 +static struct {
181 + struct {
182 + SPINLOCK spinlock;
183 + EPDL *available_items;
184 + size_t available;
185 + } protected;
186 +
187 + struct {
188 + size_t allocated;
189 + } atomics;
190 +} epdl_globals = {
191 + .protected = {
192 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
193 + .available_items = NULL,
194 + .available = 0,
195 + },
196 + .atomics = {
197 + .allocated = 0,
198 + },
199 +};
200 +
201 +void epdl_cleanup(void) {
202 + netdata_spinlock_lock(&epdl_globals.protected.spinlock);
203 +
204 + while(epdl_globals.protected.available_items && epdl_globals.protected.available > 100) {
205 + EPDL *item = epdl_globals.protected.available_items;
206 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(epdl_globals.protected.available_items, item, cache.prev, cache.next);
207 + freez(item);
208 + epdl_globals.protected.available--;
209 + __atomic_sub_fetch(&epdl_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
210 + }
211 +
212 + netdata_spinlock_unlock(&epdl_globals.protected.spinlock);
213 +}
214 +
215 +static EPDL *epdl_get(void) {
216 + EPDL *epdl = NULL;
217 +
218 + netdata_spinlock_lock(&epdl_globals.protected.spinlock);
219 +
220 + if(likely(epdl_globals.protected.available_items)) {
221 + epdl = epdl_globals.protected.available_items;
222 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(epdl_globals.protected.available_items, epdl, cache.prev, cache.next);
223 + epdl_globals.protected.available--;
224 + }
225 +
226 + netdata_spinlock_unlock(&epdl_globals.protected.spinlock);
227 +
228 + if(unlikely(!epdl)) {
229 + epdl = mallocz(sizeof(EPDL));
230 + __atomic_add_fetch(&epdl_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
231 + }
232 +
233 + memset(epdl, 0, sizeof(EPDL));
234 + return epdl;
235 +}
236 +
237 +static void epdl_release(EPDL *epdl) {
238 + if(unlikely(!epdl)) return;
239 +
240 + netdata_spinlock_lock(&epdl_globals.protected.spinlock);
241 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(epdl_globals.protected.available_items, epdl, cache.prev, cache.next);
242 + epdl_globals.protected.available++;
243 + netdata_spinlock_unlock(&epdl_globals.protected.spinlock);
244 +}
245 +
246 +size_t epdl_cache_size(void) {
247 + return __atomic_load_n(&epdl_globals.atomics.allocated, __ATOMIC_RELAXED) * sizeof(EPDL);
248 +}
249 +
250 +// ----------------------------------------------------------------------------
251 +// deol cache
252 +
253 +static struct {
254 + struct {
255 + SPINLOCK spinlock;
256 + DEOL *available_items;
257 + size_t available;
258 + } protected;
259 +
260 + struct {
261 + size_t allocated;
262 + } atomics;
263 +} deol_globals = {
264 + .protected = {
265 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
266 + .available_items = NULL,
267 + .available = 0,
268 + },
269 + .atomics = {
270 + .allocated = 0,
271 + },
272 +};
273 +
274 +void deol_cleanup(void) {
275 + netdata_spinlock_lock(&deol_globals.protected.spinlock);
276 +
277 + while(deol_globals.protected.available_items && deol_globals.protected.available > 100) {
278 + DEOL *item = deol_globals.protected.available_items;
279 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(deol_globals.protected.available_items, item, cache.prev, cache.next);
280 + freez(item);
281 + deol_globals.protected.available--;
282 + __atomic_sub_fetch(&deol_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
283 + }
284 +
285 + netdata_spinlock_unlock(&deol_globals.protected.spinlock);
286 +}
287 +
288 +static DEOL *deol_get(void) {
289 + DEOL *deol = NULL;
290 +
291 + netdata_spinlock_lock(&deol_globals.protected.spinlock);
292 +
293 + if(likely(deol_globals.protected.available_items)) {
294 + deol = deol_globals.protected.available_items;
295 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(deol_globals.protected.available_items, deol, cache.prev, cache.next);
296 + deol_globals.protected.available--;
297 + }
298 +
299 + netdata_spinlock_unlock(&deol_globals.protected.spinlock);
300 +
301 + if(unlikely(!deol)) {
302 + deol = mallocz(sizeof(DEOL));
303 + __atomic_add_fetch(&deol_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
304 + }
305 +
306 + memset(deol, 0, sizeof(DEOL));
307 + return deol;
308 +}
309 +
310 +static void deol_release(DEOL *deol) {
311 + if(unlikely(!deol)) return;
312 +
313 + netdata_spinlock_lock(&deol_globals.protected.spinlock);
314 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(deol_globals.protected.available_items, deol, cache.prev, cache.next);
315 + deol_globals.protected.available++;
316 + netdata_spinlock_unlock(&deol_globals.protected.spinlock);
317 +}
318 +
319 +size_t deol_cache_size(void) {
320 + return __atomic_load_n(&deol_globals.atomics.allocated, __ATOMIC_RELAXED) * sizeof(DEOL);
321 +}
322 +
323 +// ----------------------------------------------------------------------------
324 +// extent with buffer cache
325 +
326 +static struct {
327 + struct {
328 + SPINLOCK spinlock;
329 + struct extent_buffer *available_items;
330 + size_t available;
331 + } protected;
332 +
333 + struct {
334 + size_t allocated;
335 + size_t allocated_bytes;
336 + } atomics;
337 +
338 + size_t max_size;
339 +
340 +} extent_buffer_globals = {
341 + .protected = {
342 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
343 + .available_items = NULL,
344 + .available = 0,
345 + },
346 + .atomics = {
347 + .allocated = 0,
348 + .allocated_bytes = 0,
349 + },
350 + .max_size = MAX_PAGES_PER_EXTENT * RRDENG_BLOCK_SIZE,
351 +};
352 +
353 +void extent_buffer_init(void) {
354 + size_t max_extent_uncompressed = MAX_PAGES_PER_EXTENT * RRDENG_BLOCK_SIZE;
355 + size_t max_size = (size_t)LZ4_compressBound(MAX_PAGES_PER_EXTENT * RRDENG_BLOCK_SIZE);
356 + if(max_size < max_extent_uncompressed)
357 + max_size = max_extent_uncompressed;
358 +
359 + extent_buffer_globals.max_size = max_size;
360 +}
361 +
362 +void extent_buffer_cleanup(void) {
363 + netdata_spinlock_lock(&extent_buffer_globals.protected.spinlock);
364 +
365 + while(extent_buffer_globals.protected.available_items && extent_buffer_globals.protected.available > 1) {
366 + struct extent_buffer *item = extent_buffer_globals.protected.available_items;
367 + size_t bytes = sizeof(struct extent_buffer) + item->bytes;
368 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(extent_buffer_globals.protected.available_items, item, cache.prev, cache.next);
369 + freez(item);
370 + extent_buffer_globals.protected.available--;
371 + __atomic_sub_fetch(&extent_buffer_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
372 + __atomic_sub_fetch(&extent_buffer_globals.atomics.allocated_bytes, bytes, __ATOMIC_RELAXED);
373 + }
374 +
375 + netdata_spinlock_unlock(&extent_buffer_globals.protected.spinlock);
376 +}
377 +
378 +struct extent_buffer *extent_buffer_get(size_t size) {
379 + internal_fatal(size > extent_buffer_globals.max_size, "DBENGINE: extent size is too big");
380 +
381 + struct extent_buffer *eb = NULL;
382 +
383 + if(size < extent_buffer_globals.max_size)
384 + size = extent_buffer_globals.max_size;
385 +
386 + netdata_spinlock_lock(&extent_buffer_globals.protected.spinlock);
387 + if(likely(extent_buffer_globals.protected.available_items)) {
388 + eb = extent_buffer_globals.protected.available_items;
389 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(extent_buffer_globals.protected.available_items, eb, cache.prev, cache.next);
390 + extent_buffer_globals.protected.available--;
391 + }
392 + netdata_spinlock_unlock(&extent_buffer_globals.protected.spinlock);
393 +
394 + if(unlikely(eb && eb->bytes < size)) {
395 + size_t bytes = sizeof(struct extent_buffer) + eb->bytes;
396 + freez(eb);
397 + eb = NULL;
398 + __atomic_sub_fetch(&extent_buffer_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
399 + __atomic_sub_fetch(&extent_buffer_globals.atomics.allocated_bytes, bytes, __ATOMIC_RELAXED);
400 + }
401 +
402 + if(unlikely(!eb)) {
403 + size_t bytes = sizeof(struct extent_buffer) + size;
404 + eb = mallocz(bytes);
405 + eb->bytes = size;
406 + __atomic_add_fetch(&extent_buffer_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
407 + __atomic_add_fetch(&extent_buffer_globals.atomics.allocated_bytes, bytes, __ATOMIC_RELAXED);
408 + }
409 +
410 + return eb;
411 +}
412 +
413 +void extent_buffer_release(struct extent_buffer *eb) {
414 + if(unlikely(!eb)) return;
415 +
416 + netdata_spinlock_lock(&extent_buffer_globals.protected.spinlock);
417 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(extent_buffer_globals.protected.available_items, eb, cache.prev, cache.next);
418 + extent_buffer_globals.protected.available++;
419 + netdata_spinlock_unlock(&extent_buffer_globals.protected.spinlock);
420 +}
421 +
422 +size_t extent_buffer_cache_size(void) {
423 + return __atomic_load_n(&extent_buffer_globals.atomics.allocated_bytes, __ATOMIC_RELAXED);
424 +}
425 +
426 +// ----------------------------------------------------------------------------
427 +// epdl logic
428 +
429 +static void epdl_destroy(EPDL *epdl)
430 +{
431 + Pvoid_t *pd_by_start_time_s_JudyL;
432 + Word_t metric_id_index = 0;
433 + bool metric_id_first = true;
434 + while ((pd_by_start_time_s_JudyL = PDCJudyLFirstThenNext(
435 + epdl->page_details_by_metric_id_JudyL,
436 + &metric_id_index, &metric_id_first)))
437 + PDCJudyLFreeArray(pd_by_start_time_s_JudyL, PJE0);
438 +
439 + PDCJudyLFreeArray(&epdl->page_details_by_metric_id_JudyL, PJE0);
440 + epdl_release(epdl);
441 +}
442 +
443 +static void epdl_mark_all_not_loaded_pages_as_failed(EPDL *epdl, PDC_PAGE_STATUS tags, size_t *statistics_counter)
444 +{
445 + size_t pages_matched = 0;
446 +
447 + Word_t metric_id_index = 0;
448 + bool metric_id_first = true;
449 + Pvoid_t *pd_by_start_time_s_JudyL;
450 + while((pd_by_start_time_s_JudyL = PDCJudyLFirstThenNext(epdl->page_details_by_metric_id_JudyL, &metric_id_index, &metric_id_first))) {
451 +
452 + Word_t start_time_index = 0;
453 + bool start_time_first = true;
454 + Pvoid_t *PValue;
455 + while ((PValue = PDCJudyLFirstThenNext(*pd_by_start_time_s_JudyL, &start_time_index, &start_time_first))) {
456 + struct page_details *pd = *PValue;
457 +
458 + if(!pd->page) {
459 + pdc_page_status_set(pd, PDC_PAGE_FAILED | tags);
460 + pages_matched++;
461 + }
462 + }
463 + }
464 +
465 + if(pages_matched && statistics_counter)
466 + __atomic_add_fetch(statistics_counter, pages_matched, __ATOMIC_RELAXED);
467 +}
468 +
469 +static bool epdl_check_if_pages_are_already_in_cache(struct rrdengine_instance *ctx, EPDL *epdl, PDC_PAGE_STATUS tags)
470 +{
471 + size_t count_remaining = 0;
472 + size_t found = 0;
473 +
474 + Word_t metric_id_index = 0;
475 + bool metric_id_first = true;
476 + Pvoid_t *pd_by_start_time_s_JudyL;
477 + while((pd_by_start_time_s_JudyL = PDCJudyLFirstThenNext(epdl->page_details_by_metric_id_JudyL, &metric_id_index, &metric_id_first))) {
478 +
479 + Word_t start_time_index = 0;
480 + bool start_time_first = true;
481 + Pvoid_t *PValue;
482 + while ((PValue = PDCJudyLFirstThenNext(*pd_by_start_time_s_JudyL, &start_time_index, &start_time_first))) {
483 + struct page_details *pd = *PValue;
484 + if (pd->page)
485 + continue;
486 +
487 + pd->page = pgc_page_get_and_acquire(main_cache, (Word_t) ctx, pd->metric_id, pd->first_time_s, PGC_SEARCH_EXACT);
488 + if (pd->page) {
489 + found++;
490 + pdc_page_status_set(pd, PDC_PAGE_READY | tags);
491 + }
492 + else
493 + count_remaining++;
494 + }
495 + }
496 +
497 + if(found) {
498 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_load_ok_preloaded, found, __ATOMIC_RELAXED);
499 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_data_source_main_cache, found, __ATOMIC_RELAXED);
500 + }
501 +
502 + return count_remaining == 0;
503 +}
504 +
505 +// ----------------------------------------------------------------------------
506 +// PDC logic
507 +
508 +static void pdc_destroy(PDC *pdc) {
509 + mrg_metric_release(main_mrg, pdc->metric);
510 + completion_destroy(&pdc->prep_completion);
511 + completion_destroy(&pdc->page_completion);
512 +
513 + Pvoid_t *PValue;
514 + struct page_details *pd;
515 + Word_t time_index = 0;
516 + bool first_then_next = true;
517 + size_t unroutable = 0;
518 + while((PValue = PDCJudyLFirstThenNext(pdc->page_list_JudyL, &time_index, &first_then_next))) {
519 + pd = *PValue;
520 +
521 + // no need for atomics here - we are done...
522 + PDC_PAGE_STATUS status = pd->status;
523 +
524 + if(status & PDC_PAGE_DATAFILE_ACQUIRED) {
525 + datafile_release(pd->datafile.ptr, DATAFILE_ACQUIRE_PAGE_DETAILS);
526 + pd->datafile.ptr = NULL;
527 + }
528 +
529 + internal_fatal(pd->datafile.ptr, "DBENGINE: page details has a datafile.ptr that is not released.");
530 +
531 + if(!pd->page && !(status & (PDC_PAGE_READY | PDC_PAGE_FAILED | PDC_PAGE_RELEASED | PDC_PAGE_SKIP | PDC_PAGE_INVALID))) {
532 + // pdc_page_status_set(pd, PDC_PAGE_FAILED);
533 + unroutable++;
534 + }
535 +
536 + if(pd->page && !(status & PDC_PAGE_RELEASED)) {
537 + pgc_page_release(main_cache, pd->page);
538 + // pdc_page_status_set(pd, PDC_PAGE_RELEASED);
539 + }
540 +
541 + page_details_release(pd);
542 + }
543 +
544 + PDCJudyLFreeArray(&pdc->page_list_JudyL, PJE0);
545 +
546 + __atomic_sub_fetch(&rrdeng_cache_efficiency_stats.currently_running_queries, 1, __ATOMIC_RELAXED);
547 + __atomic_sub_fetch(&pdc->ctx->inflight_queries, 1, __ATOMIC_RELAXED);
548 + pdc_release(pdc);
549 +
550 + if(unroutable)
551 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_load_fail_unroutable, unroutable, __ATOMIC_RELAXED);
552 +}
553 +
554 +void pdc_acquire(PDC *pdc) {
555 + netdata_spinlock_lock(&pdc->refcount_spinlock);
556 +
557 + if(pdc->refcount < 1)
558 + fatal("DBENGINE: pdc is not referenced and cannot be acquired");
559 +
560 + pdc->refcount++;
561 + netdata_spinlock_unlock(&pdc->refcount_spinlock);
562 +}
563 +
564 +bool pdc_release_and_destroy_if_unreferenced(PDC *pdc, bool worker, bool router __maybe_unused) {
565 + netdata_spinlock_lock(&pdc->refcount_spinlock);
566 +
567 + if(pdc->refcount <= 0)
568 + fatal("DBENGINE: pdc is not referenced and cannot be released");
569 +
570 + pdc->refcount--;
571 +
572 + if (pdc->refcount <= 1 && worker) {
573 + // when 1 refcount is remaining, and we are a worker,
574 + // we can mark the job completed:
575 + // - if the remaining refcount is from the query caller, we will wake it up
576 + // - if the remaining refcount is from another worker, the query thread is already away
577 + completion_mark_complete(&pdc->page_completion);
578 + }
579 +
580 + if (pdc->refcount == 0) {
581 + netdata_spinlock_unlock(&pdc->refcount_spinlock);
582 + pdc_destroy(pdc);
583 + return true;
584 + }
585 +
586 + netdata_spinlock_unlock(&pdc->refcount_spinlock);
587 + return false;
588 +}
589 +
590 +void pdc_to_epdl_router(struct rrdengine_instance *ctx, PDC *pdc, execute_extent_page_details_list_t exec_first_extent_list, execute_extent_page_details_list_t exec_rest_extent_list)
591 +{
592 + Pvoid_t *PValue;
593 + Pvoid_t *PValue1;
594 + Pvoid_t *PValue2;
595 + Word_t time_index = 0;
596 + struct page_details *pd = NULL;
597 +
598 + // this is the entire page list
599 + // Lets do some deduplication
600 + // 1. Per datafile
601 + // 2. Per extent
602 + // 3. Pages per extent will be added to the cache either as acquired or not
603 +
604 + Pvoid_t JudyL_datafile_list = NULL;
605 +
606 + DEOL *deol;
607 + EPDL *epdl;
608 +
609 + if (pdc->page_list_JudyL) {
610 + bool first_then_next = true;
611 + while((PValue = PDCJudyLFirstThenNext(pdc->page_list_JudyL, &time_index, &first_then_next))) {
612 + pd = *PValue;
613 +
614 + internal_fatal(!pd,
615 + "DBENGINE: pdc page list has an empty page details entry");
616 +
617 + if (!(pd->status & PDC_PAGE_DISK_PENDING))
618 + continue;
619 +
620 + internal_fatal(!(pd->status & PDC_PAGE_DATAFILE_ACQUIRED),
621 + "DBENGINE: page details has not acquired the datafile");
622 +
623 + internal_fatal((pd->status & (PDC_PAGE_READY | PDC_PAGE_FAILED)),
624 + "DBENGINE: page details has disk pending flag but it is ready/failed");
625 +
626 + internal_fatal(pd->page,
627 + "DBENGINE: page details has a page linked to it, but it is marked for loading");
628 +
629 + PValue1 = PDCJudyLIns(&JudyL_datafile_list, pd->datafile.fileno, PJE0);
630 + if (PValue1 && !*PValue1) {
631 + *PValue1 = deol = deol_get();
632 + deol->extent_pd_list_by_extent_offset_JudyL = NULL;
633 + deol->fileno = pd->datafile.fileno;
634 + }
635 + else
636 + deol = *PValue1;
637 +
638 + PValue2 = PDCJudyLIns(&deol->extent_pd_list_by_extent_offset_JudyL, pd->datafile.extent.pos, PJE0);
639 + if (PValue2 && !*PValue2) {
640 + *PValue2 = epdl = epdl_get();
641 + epdl->page_details_by_metric_id_JudyL = NULL;
642 + epdl->number_of_pages_in_JudyL = 0;
643 + epdl->file = pd->datafile.file;
644 + epdl->extent_offset = pd->datafile.extent.pos;
645 + epdl->extent_size = pd->datafile.extent.bytes;
646 + epdl->datafile = pd->datafile.ptr;
647 + }
648 + else
649 + epdl = *PValue2;
650 +
651 + epdl->number_of_pages_in_JudyL++;
652 +
653 + Pvoid_t *pd_by_first_time_s_judyL = PDCJudyLIns(&epdl->page_details_by_metric_id_JudyL, pd->metric_id, PJE0);
654 + Pvoid_t *pd_pptr = PDCJudyLIns(pd_by_first_time_s_judyL, pd->first_time_s, PJE0);
655 + *pd_pptr = pd;
656 + }
657 +
658 + size_t extent_list_no = 0;
659 + Word_t datafile_no = 0;
660 + first_then_next = true;
661 + while((PValue = PDCJudyLFirstThenNext(JudyL_datafile_list, &datafile_no, &first_then_next))) {
662 + deol = *PValue;
663 +
664 + bool first_then_next_extent = true;
665 + Word_t pos = 0;
666 + while ((PValue = PDCJudyLFirstThenNext(deol->extent_pd_list_by_extent_offset_JudyL, &pos, &first_then_next_extent))) {
667 + epdl = *PValue;
668 + internal_fatal(!epdl, "DBENGINE: extent_list is not populated properly");
669 +
670 + // The extent page list can be dispatched to a worker
671 + // It will need to populate the cache with "acquired" pages that are in the list (pd) only
672 + // the rest of the extent pages will be added to the cache butnot acquired
673 +
674 + pdc_acquire(pdc); // we do this for the next worker: do_read_extent_work()
675 + epdl->pdc = pdc;
676 +
677 + if(extent_list_no++ == 0)
678 + exec_first_extent_list(ctx, epdl, pdc->priority);
679 + else
680 + exec_rest_extent_list(ctx, epdl, pdc->priority);
681 + }
682 + PDCJudyLFreeArray(&deol->extent_pd_list_by_extent_offset_JudyL, PJE0);
683 + deol_release(deol);
684 + }
685 + PDCJudyLFreeArray(&JudyL_datafile_list, PJE0);
686 + }
687 +
688 + pdc_release_and_destroy_if_unreferenced(pdc, true, true);
689 +}
690 +
691 +static bool datafile_get_exclusive_access_to_extent(EPDL *epdl) {
692 + struct rrdengine_datafile *df = epdl->datafile;
693 + bool is_it_mine = false;
694 +
695 + while(!is_it_mine) {
696 + netdata_spinlock_lock(&df->extent_exclusive_access.spinlock);
697 + if(!df->users.available) {
698 + netdata_spinlock_unlock(&df->extent_exclusive_access.spinlock);
699 + return false;
700 + }
701 + Pvoid_t *PValue = JudyLIns(&df->extent_exclusive_access.extents_JudyL, epdl->extent_offset, PJE0);
702 + if (!*PValue) {
703 + *(Word_t *) PValue = gettid();
704 + df->extent_exclusive_access.lockers++;
705 + is_it_mine = true;
706 + }
707 + netdata_spinlock_unlock(&df->extent_exclusive_access.spinlock);
708 +
709 + if(!is_it_mine) {
710 + static const struct timespec ns = { .tv_sec = 0, .tv_nsec = 1 };
711 + nanosleep(&ns, NULL);
712 + }
713 + }
714 + return true;
715 +}
716 +
717 +static void datafile_release_exclusive_access_to_extent(EPDL *epdl) {
718 + struct rrdengine_datafile *df = epdl->datafile;
719 +
720 + netdata_spinlock_lock(&df->extent_exclusive_access.spinlock);
721 +
722 +#ifdef NETDATA_INTERNAL_CHECKS
723 + Pvoid_t *PValue = JudyLGet(df->extent_exclusive_access.extents_JudyL, epdl->extent_offset, PJE0);
724 + if (*(Word_t *) PValue != (Word_t)gettid())
725 + fatal("DBENGINE: exclusive extent access is not mine");
726 +#endif
727 +
728 + int rc = JudyLDel(&df->extent_exclusive_access.extents_JudyL, epdl->extent_offset, PJE0);
729 + if (!rc)
730 + fatal("DBENGINE: cannot find my exclusive access");
731 +
732 + df->extent_exclusive_access.lockers--;
733 + netdata_spinlock_unlock(&df->extent_exclusive_access.spinlock);
734 +}
735 +
736 +static void fill_page_with_nulls(void *page, uint32_t page_length, uint8_t type) {
737 + switch(type) {
738 + case PAGE_METRICS: {
739 + storage_number n = pack_storage_number(NAN, SN_FLAG_NONE);
740 + storage_number *array = (storage_number *)page;
741 + size_t slots = page_length / sizeof(n);
742 + for(size_t i = 0; i < slots ; i++)
743 + array[i] = n;
744 + }
745 + break;
746 +
747 + case PAGE_TIER: {
748 + storage_number_tier1_t n = {
749 + .min_value = NAN,
750 + .max_value = NAN,
751 + .sum_value = NAN,
752 + .count = 1,
753 + .anomaly_count = 0,
754 + };
755 + storage_number_tier1_t *array = (storage_number_tier1_t *)page;
756 + size_t slots = page_length / sizeof(n);
757 + for(size_t i = 0; i < slots ; i++)
758 + array[i] = n;
759 + }
760 + break;
761 +
762 + default: {
763 + static bool logged = false;
764 + if(!logged) {
765 + error("DBENGINE: cannot fill page with nulls on unknown page type id %d", type);
766 + logged = true;
767 + }
768 + memset(page, 0, page_length);
769 + }
770 + }
771 +}
772 +
773 +inline VALIDATED_PAGE_DESCRIPTOR validate_extent_page_descr(const struct rrdeng_extent_page_descr *descr, time_t now_s, time_t overwrite_zero_update_every_s, bool have_read_error) {
774 + VALIDATED_PAGE_DESCRIPTOR vd = {
775 + .start_time_s = (time_t) (descr->start_time_ut / USEC_PER_SEC),
776 + .end_time_s = (time_t) (descr->end_time_ut / USEC_PER_SEC),
777 + .page_length = descr->page_length,
778 + .type = descr->type,
779 + };
780 + vd.point_size = page_type_size[vd.type];
781 + vd.entries = page_entries_by_size(vd.page_length, vd.point_size);
782 + vd.update_every_s = (vd.entries > 1) ? ((vd.end_time_s - vd.start_time_s) / (time_t)(vd.entries - 1)) : overwrite_zero_update_every_s;
783 +
784 + bool is_valid = true;
785 +
786 + // another such set of checks exists in
787 + // update_metric_retention_and_granularity_by_uuid()
788 +
789 + if( have_read_error ||
790 + vd.page_length == 0 ||
791 + vd.page_length > RRDENG_BLOCK_SIZE ||
792 + vd.start_time_s > vd.end_time_s ||
793 + vd.end_time_s > now_s ||
794 + vd.start_time_s == 0 ||
795 + vd.end_time_s == 0 ||
796 + (vd.start_time_s == vd.end_time_s && vd.entries > 1) ||
797 + (vd.update_every_s == 0 && vd.entries > 1)
798 + ) {
799 + is_valid = false;
800 +
801 + error_limit_static_global_var(erl, 1, 0);
802 + error_limit(&erl, "DBENGINE: ignoring invalid page of type %u from %ld to %ld (now %ld), update every %ld, page length %zu, point size %zu, entries %zu.",
803 + vd.type, vd.start_time_s, vd.end_time_s, now_s, vd.update_every_s, vd.page_length, vd.point_size, vd.entries);
804 + }
805 + else {
806 + if (vd.update_every_s) {
807 + size_t entries_by_time = page_entries_by_time(vd.start_time_s, vd.end_time_s, vd.update_every_s);
808 +
809 + if (vd.entries != entries_by_time) {
810 + if (overwrite_zero_update_every_s < vd.update_every_s)
811 + vd.update_every_s = overwrite_zero_update_every_s;
812 +
813 + time_t new_end_time_s = (time_t)(vd.start_time_s + (vd.entries - 1) * vd.update_every_s);
814 +
815 + if(new_end_time_s <= vd.end_time_s) {
816 + // end time is wrong
817 + vd.end_time_s = new_end_time_s;
818 + }
819 + else {
820 + // update every is wrong
821 + vd.update_every_s = overwrite_zero_update_every_s;
822 + vd.end_time_s = (time_t)(vd.start_time_s + (vd.entries - 1) * vd.update_every_s);
823 + }
824 + }
825 + }
826 + else
827 + vd.update_every_s = overwrite_zero_update_every_s;
828 + }
829 +
830 + if(!is_valid) {
831 + if(vd.start_time_s == vd.end_time_s) {
832 + vd.page_length = vd.point_size;
833 + vd.entries = 1;
834 + }
835 + else {
836 + vd.page_length = vd.point_size * 2;
837 + vd.update_every_s = vd.end_time_s - vd.start_time_s;
838 + vd.entries = 2;
839 + }
840 + }
841 +
842 + vd.data_on_disk_valid = is_valid;
843 + return vd;
844 +}
845 +
846 +static bool epdl_populate_pages_from_extent_data(
847 + struct rrdengine_instance *ctx,
848 + void *data,
849 + size_t data_length,
850 + EPDL *epdl,
851 + bool preload_all_pages,
852 + bool worker,
853 + PDC_PAGE_STATUS tags,
854 + bool cached_extent)
855 +{
856 + int ret;
857 + unsigned i, count;
858 + void *uncompressed_buf = NULL;
859 + uint32_t payload_length, payload_offset, trailer_offset, uncompressed_payload_length = 0;
860 + bool have_read_error = false;
861 + /* persistent structures */
862 + struct rrdeng_df_extent_header *header;
863 + struct rrdeng_df_extent_trailer *trailer;
864 + struct extent_buffer *eb = NULL;
865 + uLong crc;
866 +
867 + bool can_use_data = true;
868 + if(data_length < sizeof(*header) + sizeof(header->descr[0]) + sizeof(*trailer)) {
869 + can_use_data = false;
870 + }
871 + else {
872 + header = data;
873 + payload_length = header->payload_length;
874 + count = header->number_of_pages;
875 + payload_offset = sizeof(*header) + sizeof(header->descr[0]) * count;
876 + trailer_offset = data_length - sizeof(*trailer);
877 + trailer = data + trailer_offset;
878 + }
879 +
880 + if( !can_use_data ||
881 + count < 1 ||
882 + count > MAX_PAGES_PER_EXTENT ||
883 + (header->compression_algorithm != RRD_NO_COMPRESSION && header->compression_algorithm != RRD_LZ4) ||
884 + (payload_length != trailer_offset - payload_offset) ||
885 + (data_length != payload_offset + payload_length + sizeof(*trailer))
886 + ) {
887 +
888 + error_limit_static_global_var(erl, 1, 0);
889 + error_limit(&erl, "%s: Extent at offset %"PRIu64" (%u bytes) was read from datafile %u, but header is INVALID", __func__,
890 + epdl->extent_offset, epdl->extent_size, epdl->datafile->fileno);
891 +
892 + return false;
893 + }
894 +
895 + crc = crc32(0L, Z_NULL, 0);
896 + crc = crc32(crc, data, epdl->extent_size - sizeof(*trailer));
897 + ret = crc32cmp(trailer->checksum, crc);
898 + if (unlikely(ret)) {
899 + ++ctx->stats.io_errors;
900 + rrd_stat_atomic_add(&global_io_errors, 1);
901 + have_read_error = true;
902 +
903 + error_limit_static_global_var(erl, 1, 0);
904 + error_limit(&erl, "%s: Extent at offset %"PRIu64" (%u bytes) was read from datafile %u, but CRC32 check FAILED", __func__,
905 + epdl->extent_offset, epdl->extent_size, epdl->datafile->fileno);
906 + }
907 +
908 + if(worker)
909 + worker_is_busy(UV_EVENT_EXT_DECOMPRESSION);
910 +
911 + if (likely(!have_read_error && RRD_NO_COMPRESSION != header->compression_algorithm)) {
912 + // find the uncompressed extent size
913 + uncompressed_payload_length = 0;
914 + for (i = 0; i < count; ++i) {
915 + size_t page_length = header->descr[i].page_length;
916 + if(page_length > RRDENG_BLOCK_SIZE) {
917 + have_read_error = true;
918 + break;
919 + }
920 +
921 + uncompressed_payload_length += header->descr[i].page_length;
922 + }
923 +
924 + if(unlikely(uncompressed_payload_length > MAX_PAGES_PER_EXTENT * RRDENG_BLOCK_SIZE))
925 + have_read_error = true;
926 +
927 + if(likely(!have_read_error)) {
928 + eb = extent_buffer_get(uncompressed_payload_length);
929 + uncompressed_buf = eb->data;
930 +
931 + ret = LZ4_decompress_safe(data + payload_offset, uncompressed_buf,
932 + (int) payload_length, (int) uncompressed_payload_length);
933 + ctx->stats.before_decompress_bytes += payload_length;
934 + ctx->stats.after_decompress_bytes += ret;
935 + debug(D_RRDENGINE, "LZ4 decompressed %u bytes to %d bytes.", payload_length, ret);
936 + }
937 + }
938 +
939 + size_t stats_data_from_main_cache = 0;
940 + size_t stats_data_from_extent = 0;
941 + size_t stats_load_compressed = 0;
942 + size_t stats_load_uncompressed = 0;
943 + size_t stats_load_invalid_page = 0;
944 + size_t stats_cache_hit_while_inserting = 0;
945 + size_t stats_cache_hit_before_allocation = 0;
946 +
947 + uint32_t page_offset = 0, page_length;
948 + time_t now_s = now_realtime_sec();
949 + for (i = 0; i < count; i++, page_offset += page_length) {
950 + page_length = header->descr[i].page_length;
951 + time_t start_time_s = (time_t) (header->descr[i].start_time_ut / USEC_PER_SEC);
952 +
953 + if(!page_length || !start_time_s) {
954 + error_limit_static_global_var(erl, 1, 0);
955 + error_limit(&erl, "%s: Extent at offset %"PRIu64" (%u bytes) was read from datafile %u, having page %u (out of %u) EMPTY",
956 + __func__, epdl->extent_offset, epdl->extent_size, epdl->datafile->fileno, i, count);
957 + continue;
958 + }
959 +
960 + if(worker)
961 + worker_is_busy(UV_EVENT_METRIC_LOOKUP);
962 +
963 + METRIC *metric = mrg_metric_get_and_acquire(main_mrg, &header->descr[i].uuid, (Word_t)ctx);
964 + Word_t metric_id = (Word_t)metric;
965 + if(!metric) {
966 + error_limit_static_global_var(erl, 1, 0);
967 + error_limit(&erl, "%s: Extent at offset %"PRIu64" (%u bytes) was read from datafile %u, having page %u (out of %u) for unknown UUID",
968 + __func__, epdl->extent_offset, epdl->extent_size, epdl->datafile->fileno, i, count);
969 + continue;
970 + }
971 + mrg_metric_release(main_mrg, metric);
972 +
973 + if(worker)
974 + worker_is_busy(UV_EVENT_PAGE_LOOKUP);
975 +
976 + struct page_details *pd = NULL;
977 + Pvoid_t *pd_by_start_time_s_judyL = PDCJudyLGet(epdl->page_details_by_metric_id_JudyL, metric_id, PJE0);
978 + internal_fatal(pd_by_start_time_s_judyL == PJERR, "DBENGINE: corrupted extent metrics JudyL");
979 +
980 + if(pd_by_start_time_s_judyL && *pd_by_start_time_s_judyL) {
981 + Pvoid_t *pd_pptr = PDCJudyLGet(*pd_by_start_time_s_judyL, start_time_s, PJE0);
982 + internal_fatal(pd_pptr == PJERR, "DBENGINE: corrupted metric page details JudyHS");
983 +
984 + if(pd_pptr && *pd_pptr) {
985 + pd = *pd_pptr;
986 + internal_fatal(metric_id != pd->metric_id, "DBENGINE: metric ids do not match");
987 + }
988 + }
989 +
990 + if(!pd && !preload_all_pages)
991 + continue;
992 +
993 + VALIDATED_PAGE_DESCRIPTOR vd = validate_extent_page_descr(
994 + &header->descr[i], now_s,
995 + (pd) ? pd->update_every_s : 0,
996 + have_read_error);
997 +
998 + if(worker)
999 + worker_is_busy(UV_EVENT_PAGE_POPULATION);
1000 +
1001 + PGC_PAGE *page = pgc_page_get_and_acquire(main_cache, (Word_t)ctx, metric_id, start_time_s, PGC_SEARCH_EXACT);
1002 + if (!page) {
1003 + void *page_data = dbengine_page_alloc(ctx, vd.page_length);
1004 +
1005 + if (unlikely(!vd.data_on_disk_valid)) {
1006 + fill_page_with_nulls(page_data, vd.page_length, vd.type);
1007 + stats_load_invalid_page++;
1008 + }
1009 +
1010 + else if (RRD_NO_COMPRESSION == header->compression_algorithm) {
1011 + memcpy(page_data, data + payload_offset + page_offset, (size_t) vd.page_length);
1012 + stats_load_uncompressed++;
1013 + }
1014 +
1015 + else {
1016 + if(unlikely(page_offset + vd.page_length > uncompressed_payload_length)) {
1017 + error_limit_static_global_var(erl, 10, 0);
1018 + error_limit(&erl,
1019 + "DBENGINE: page %u offset %u + page length %zu exceeds the uncompressed buffer size %u",
1020 + i, page_offset, vd.page_length, uncompressed_payload_length);
1021 +
1022 + fill_page_with_nulls(page_data, vd.page_length, vd.type);
1023 + stats_load_invalid_page++;
1024 + }
1025 + else {
1026 + memcpy(page_data, uncompressed_buf + page_offset, vd.page_length);
1027 + stats_load_compressed++;
1028 + }
1029 + }
1030 +
1031 + PGC_ENTRY page_entry = {
1032 + .hot = false,
1033 + .section = (Word_t)ctx,
1034 + .metric_id = metric_id,
1035 + .start_time_s = vd.start_time_s,
1036 + .end_time_s = vd.end_time_s,
1037 + .update_every_s = vd.update_every_s,
1038 + .size = (size_t) vd.page_length,
1039 + .data = page_data
1040 + };
1041 +
1042 + bool added = true;
1043 + page = pgc_page_add_and_acquire(main_cache, page_entry, &added);
1044 + if (false == added) {
1045 + dbengine_page_free(page_data);
1046 + stats_cache_hit_while_inserting++;
1047 + stats_data_from_main_cache++;
1048 + }
1049 + else
1050 + stats_data_from_extent++;
1051 + }
1052 + else {
1053 + stats_cache_hit_before_allocation++;
1054 + stats_data_from_main_cache++;
1055 + }
1056 +
1057 + if (pd) {
1058 + pd->page = page;
1059 + pd->page_length = pgc_page_data_size(main_cache, page);
1060 + pdc_page_status_set(pd, PDC_PAGE_READY | tags);
1061 + }
1062 + else
1063 + pgc_page_release(main_cache, page);
1064 + }
1065 +
1066 + if(stats_data_from_main_cache)
1067 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_data_source_main_cache, stats_data_from_main_cache, __ATOMIC_RELAXED);
1068 +
1069 + if(cached_extent)
1070 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_data_source_extent_cache, stats_data_from_extent, __ATOMIC_RELAXED);
1071 + else
1072 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_data_source_disk, stats_data_from_extent, __ATOMIC_RELAXED);
1073 +
1074 + if(stats_cache_hit_before_allocation)
1075 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_load_ok_loaded_but_cache_hit_before_allocation, stats_cache_hit_before_allocation, __ATOMIC_RELAXED);
1076 +
1077 + if(stats_cache_hit_while_inserting)
1078 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_load_ok_loaded_but_cache_hit_while_inserting, stats_cache_hit_while_inserting, __ATOMIC_RELAXED);
1079 +
1080 + if(stats_load_compressed)
1081 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_load_ok_compressed, stats_load_compressed, __ATOMIC_RELAXED);
1082 +
1083 + if(stats_load_uncompressed)
1084 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_load_ok_uncompressed, stats_load_uncompressed, __ATOMIC_RELAXED);
1085 +
1086 + if(stats_load_invalid_page)
1087 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_load_fail_invalid_page_in_extent, stats_load_invalid_page, __ATOMIC_RELAXED);
1088 +
1089 + if(worker)
1090 + worker_is_idle();
1091 +
1092 + extent_buffer_release(eb);
1093 +
1094 + return true;
1095 +}
1096 +
1097 +void epdl_find_extent_and_populate_pages(struct rrdengine_instance *ctx, EPDL *epdl, bool worker) {
1098 + struct page_details_control *pdc = epdl->pdc;
1099 +
1100 + bool extent_exclusive = false;
1101 +
1102 + if(pdc->preload_all_extent_pages) {
1103 + if (!datafile_get_exclusive_access_to_extent(epdl)) {
1104 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.pages_load_fail_datafile_not_available, 1, __ATOMIC_RELAXED);
1105 + goto cleanup;
1106 + }
1107 + extent_exclusive = true;
1108 + }
1109 +
1110 + if (epdl_check_if_pages_are_already_in_cache(ctx, epdl, PDC_PAGE_PRELOADED_WORKER))
1111 + goto cleanup;
1112 +
1113 + if(__atomic_load_n(&pdc->workers_should_stop, __ATOMIC_RELAXED))
1114 + goto cleanup;
1115 +
1116 + if(worker)
1117 + worker_is_busy(UV_EVENT_EXTENT_CACHE);
1118 +
1119 + PDC_PAGE_STATUS not_loaded_pages_tag = 0, loaded_pages_tag = 0;
1120 + bool extent_found_in_cache = false;
1121 +
1122 + void *extent_compressed_data = NULL;
1123 + PGC_PAGE *extent_cache_page = pgc_page_get_and_acquire(
1124 + extent_cache, (Word_t)ctx,
1125 + (Word_t)epdl->datafile->fileno, (time_t)epdl->extent_offset,
1126 + PGC_SEARCH_EXACT);
1127 +
1128 + if(extent_cache_page) {
1129 + extent_compressed_data = pgc_page_data(extent_cache_page);
1130 + internal_fatal(epdl->extent_size != pgc_page_data_size(extent_cache, extent_cache_page),
1131 + "DBENGINE: cache size does not match the expected size");
1132 +
1133 + loaded_pages_tag |= PDC_PAGE_LOADED_FROM_EXTENT_CACHE;
1134 + not_loaded_pages_tag |= PDC_PAGE_LOADED_FROM_EXTENT_CACHE;
1135 + extent_found_in_cache = true;
1136 + }
1137 + else {
1138 + if(worker)
1139 + worker_is_busy(UV_EVENT_EXTENT_MMAP);
1140 +
1141 + off_t map_start = ALIGN_BYTES_FLOOR(epdl->extent_offset);
1142 + size_t length = ALIGN_BYTES_CEILING(epdl->extent_offset + epdl->extent_size) - map_start;
1143 +
1144 + void *mmap_data = mmap(NULL, length, PROT_READ, MAP_SHARED, epdl->file, map_start);
1145 + if(mmap_data != MAP_FAILED) {
1146 + extent_compressed_data = mmap_data + (epdl->extent_offset - map_start);
1147 +
1148 + void *copied_extent_compressed_data = mallocz(epdl->extent_size);
1149 + memcpy(copied_extent_compressed_data, extent_compressed_data, epdl->extent_size);
1150 +
1151 + int ret = munmap(mmap_data, length);
1152 + fatal_assert(0 == ret);
1153 +
1154 + if(worker)
1155 + worker_is_busy(UV_EVENT_EXTENT_CACHE);
1156 +
1157 + bool added = false;
1158 + extent_cache_page = pgc_page_add_and_acquire(extent_cache, (PGC_ENTRY) {
1159 + .hot = false,
1160 + .section = (Word_t) ctx,
1161 + .metric_id = (Word_t) epdl->datafile->fileno,
1162 + .start_time_s = (time_t) epdl->extent_offset,
1163 + .size = epdl->extent_size,
1164 + .end_time_s = 0,
1165 + .update_every_s = 0,
1166 + .data = copied_extent_compressed_data,
1167 + }, &added);
1168 +
1169 + if (!added) {
1170 + freez(copied_extent_compressed_data);
1171 + internal_fatal(epdl->extent_size != pgc_page_data_size(extent_cache, extent_cache_page),
1172 + "DBENGINE: cache size does not match the expected size");
1173 + }
1174 +
1175 + extent_compressed_data = pgc_page_data(extent_cache_page);
1176 +
1177 + loaded_pages_tag |= PDC_PAGE_LOADED_FROM_DISK;
1178 + not_loaded_pages_tag |= PDC_PAGE_LOADED_FROM_DISK;
1179 + }
1180 + }
1181 +
1182 + if(extent_compressed_data) {
1183 + // Need to decompress and then process the pagelist
1184 + bool extent_used = epdl_populate_pages_from_extent_data(
1185 + ctx, extent_compressed_data, epdl->extent_size,
1186 + epdl, pdc->preload_all_extent_pages,
1187 + worker, loaded_pages_tag, extent_found_in_cache);
1188 +
1189 + if(extent_used) {
1190 + // since the extent was used, all the pages that are not
1191 + // loaded from this extent, were not found in the extent
1192 + not_loaded_pages_tag |= PDC_PAGE_FAILED_UUID_NOT_IN_EXTENT;
1193 + }
1194 + else
1195 + not_loaded_pages_tag |= PDC_PAGE_FAILED_INVALID_EXTENT;
1196 + }
1197 + else
1198 + not_loaded_pages_tag |= PDC_PAGE_FAILED_TO_MAP_EXTENT;
1199 +
1200 +
1201 + // mark all pending pages as failed
1202 + epdl_mark_all_not_loaded_pages_as_failed(
1203 + epdl, not_loaded_pages_tag,
1204 + &rrdeng_cache_efficiency_stats.pages_load_fail_cant_mmap_extent);
1205 +
1206 + if(extent_cache_page)
1207 + pgc_page_release(extent_cache, extent_cache_page);
1208 +
1209 + cleanup:
1210 + if(extent_exclusive)
1211 + datafile_release_exclusive_access_to_extent(epdl);
1212 +
1213 + completion_mark_complete_a_job(&epdl->pdc->page_completion);
1214 + pdc_release_and_destroy_if_unreferenced(pdc, true, false);
1215 +
1216 + // Free the Judy that holds the requested pagelist and the extents
1217 + epdl_destroy(epdl);
1218 +
1219 + if(worker)
1220 + worker_is_idle();
1221 +}
database/engine/pdc.h new
+62
@@ -0,0 +1,62 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef DBENGINE_PDC_H
4 +#define DBENGINE_PDC_H
5 +
6 +#include "../engine/rrdengine.h"
7 +
8 +#ifdef PDC_USE_JULYL
9 +#define PDCJudyLIns JulyLIns
10 +#define PDCJudyLGet JulyLGet
11 +#define PDCJudyLFirst JulyLFirst
12 +#define PDCJudyLNext JulyLNext
13 +#define PDCJudyLLast JulyLLast
14 +#define PDCJudyLPrev JulyLPrev
15 +#define PDCJudyLFirstThenNext JulyLFirstThenNext
16 +#define PDCJudyLLastThenPrev JulyLLastThenPrev
17 +#define PDCJudyLFreeArray JulyLFreeArray
18 +#else
19 +#define PDCJudyLIns JudyLIns
20 +#define PDCJudyLGet JudyLGet
21 +#define PDCJudyLFirst JudyLFirst
22 +#define PDCJudyLNext JudyLNext
23 +#define PDCJudyLLast JudyLLast
24 +#define PDCJudyLPrev JudyLPrev
25 +#define PDCJudyLFirstThenNext JudyLFirstThenNext
26 +#define PDCJudyLLastThenPrev JudyLLastThenPrev
27 +#define PDCJudyLFreeArray JudyLFreeArray
28 +#endif
29 +
30 +typedef struct extent_page_details_list EPDL;
31 +typedef void (*execute_extent_page_details_list_t)(struct rrdengine_instance *ctx, EPDL *epdl, enum storage_priority priority);
32 +void pdc_to_epdl_router(struct rrdengine_instance *ctx, struct page_details_control *pdc, execute_extent_page_details_list_t exec_first_extent_list, execute_extent_page_details_list_t exec_rest_extent_list);
33 +void epdl_find_extent_and_populate_pages(struct rrdengine_instance *ctx, EPDL *epdl, bool worker);
34 +
35 +size_t pdc_cache_size(void);
36 +size_t pd_cache_size(void);
37 +size_t epdl_cache_size(void);
38 +size_t deol_cache_size(void);
39 +size_t extent_buffer_cache_size(void);
40 +
41 +void pdc_cleanup(void);
42 +void page_details_cleanup(void);
43 +void epdl_cleanup(void);
44 +void deol_cleanup(void);
45 +void extent_buffer_cleanup(void);
46 +
47 +struct extent_buffer {
48 + size_t bytes;
49 +
50 + struct {
51 + struct extent_buffer *prev;
52 + struct extent_buffer *next;
53 + } cache;
54 +
55 + uint8_t data[];
56 +};
57 +
58 +void extent_buffer_init(void);
59 +struct extent_buffer *extent_buffer_get(size_t size);
60 +void extent_buffer_release(struct extent_buffer *eb);
61 +
62 +#endif // DBENGINE_PDC_H
database/engine/rrdengine.c
+1285 -1166
@@ -2,6 +2,7 @@
2 #define NETDATA_RRD_INTERNALS
3
4 #include "rrdengine.h"
5 +#include "pdc.h"
6
7 rrdeng_stats_t global_io_errors = 0;
8 rrdeng_stats_t global_fs_errors = 0;
@@ -11,31 +12,34 @@ rrdeng_stats_t global_flushing_pressure_page_deletions = 0;
12
13 unsigned rrdeng_pages_per_extent = MAX_PAGES_PER_EXTENT;
14
14 -#if WORKER_UTILIZATION_MAX_JOB_TYPES < (RRDENG_MAX_OPCODE + 2)
15 +#if WORKER_UTILIZATION_MAX_JOB_TYPES < (RRDENG_OPCODE_MAX + 2)
16 #error Please increase WORKER_UTILIZATION_MAX_JOB_TYPES to at least (RRDENG_MAX_OPCODE + 2)
17 #endif
18
18 -void *dbengine_page_alloc() {
19 - void *page = NULL;
20 - if (unlikely(db_engine_use_malloc))
21 - page = mallocz(RRDENG_BLOCK_SIZE);
22 - else {
23 - page = netdata_mmap(NULL, RRDENG_BLOCK_SIZE, MAP_PRIVATE, enable_ksm);
24 - if(!page) fatal("Cannot allocate dbengine page cache page, with mmap()");
25 - }
26 - return page;
27 -}
28 -
29 -void dbengine_page_free(void *page) {
30 - if (unlikely(db_engine_use_malloc))
31 - freez(page);
32 - else
33 - netdata_munmap(page, RRDENG_BLOCK_SIZE);
34 -}
19 +struct rrdeng_main {
20 + uv_thread_t thread;
21 + uv_loop_t loop;
22 + uv_async_t async;
23 + uv_timer_t timer;
24 + pid_t tid;
25 +
26 + time_t last_buffers_cleanup_s;
27 +
28 + bool flush_running;
29 + bool evict_running;
30 +} rrdeng_main = {
31 + .thread = 0,
32 + .loop = {},
33 + .async = {},
34 + .timer = {},
35 + .last_buffers_cleanup_s = 0,
36 + .flush_running = false,
37 + .evict_running = false,
38 +};
39
40 static void sanity_check(void)
41 {
38 - BUILD_BUG_ON(WORKER_UTILIZATION_MAX_JOB_TYPES < (RRDENG_MAX_OPCODE + 2));
42 + BUILD_BUG_ON(WORKER_UTILIZATION_MAX_JOB_TYPES < (RRDENG_OPCODE_MAX + 2));
43
44 /* Magic numbers must fit in the super-blocks */
45 BUILD_BUG_ON(strlen(RRDENG_DF_MAGIC) > RRDENG_MAGIC_SZ);
@@ -54,519 +58,623 @@ static void sanity_check(void)
58 BUILD_BUG_ON(MAX_PAGES_PER_EXTENT > 255);
59
60 /* extent cache count must fit in 32 bits */
57 - BUILD_BUG_ON(MAX_CACHED_EXTENTS > 32);
61 +// BUILD_BUG_ON(MAX_CACHED_EXTENTS > 32);
62
63 /* page info scratch space must be able to hold 2 32-bit integers */
64 BUILD_BUG_ON(sizeof(((struct rrdeng_page_info *)0)->scratch) < 2 * sizeof(uint32_t));
65 }
66
63 -/* always inserts into tail */
64 -static inline void xt_cache_replaceQ_insert(struct rrdengine_worker_config* wc,
65 - struct extent_cache_element *xt_cache_elem)
66 -{
67 - struct extent_cache *xt_cache = &wc->xt_cache;
67 +// ----------------------------------------------------------------------------
68 +// work request cache
69
69 - xt_cache_elem->prev = NULL;
70 - xt_cache_elem->next = NULL;
70 +typedef void (*work_cb)(struct rrdengine_instance *ctx, void *data, struct completion *completion, uv_work_t* req);
71 +typedef void (*after_work_cb)(struct rrdengine_instance *ctx, void *data, struct completion *completion, uv_work_t* req, int status);
72
72 - if (likely(NULL != xt_cache->replaceQ_tail)) {
73 - xt_cache_elem->prev = xt_cache->replaceQ_tail;
74 - xt_cache->replaceQ_tail->next = xt_cache_elem;
75 - }
76 - if (unlikely(NULL == xt_cache->replaceQ_head)) {
77 - xt_cache->replaceQ_head = xt_cache_elem;
73 +struct rrdeng_work {
74 + uv_work_t req;
75 +
76 + struct rrdengine_instance *ctx;
77 + void *data;
78 + struct completion *completion;
79 +
80 + work_cb work_cb;
81 + after_work_cb after_work_cb;
82 + enum rrdeng_opcode opcode;
83 +
84 + struct {
85 + struct rrdeng_work *prev;
86 + struct rrdeng_work *next;
87 + } cache;
88 +};
89 +
90 +static struct {
91 + struct {
92 + SPINLOCK spinlock;
93 + struct rrdeng_work *available_items;
94 + size_t available;
95 + } protected;
96 +
97 + struct {
98 + size_t allocated;
99 + size_t dispatched;
100 + size_t executing;
101 + size_t pending_cb;
102 + } atomics;
103 +} work_request_globals = {
104 + .protected = {
105 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
106 + .available_items = NULL,
107 + .available = 0,
108 + },
109 + .atomics = {
110 + .allocated = 0,
111 + .dispatched = 0,
112 + .executing = 0,
113 + },
114 +};
115 +
116 +static inline bool work_request_full(void) {
117 + return __atomic_load_n(&work_request_globals.atomics.dispatched, __ATOMIC_RELAXED) >= (size_t)(libuv_worker_threads - RESERVED_LIBUV_WORKER_THREADS);
118 +}
119 +
120 +static void work_request_cleanup(void) {
121 + netdata_spinlock_lock(&work_request_globals.protected.spinlock);
122 + while(work_request_globals.protected.available_items && work_request_globals.protected.available > (size_t)libuv_worker_threads) {
123 + struct rrdeng_work *item = work_request_globals.protected.available_items;
124 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(work_request_globals.protected.available_items, item, cache.prev, cache.next);
125 + freez(item);
126 + work_request_globals.protected.available--;
127 + __atomic_sub_fetch(&work_request_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
128 }
79 - xt_cache->replaceQ_tail = xt_cache_elem;
129 + netdata_spinlock_unlock(&work_request_globals.protected.spinlock);
130 }
131
82 -static inline void xt_cache_replaceQ_delete(struct rrdengine_worker_config* wc,
83 - struct extent_cache_element *xt_cache_elem)
84 -{
85 - struct extent_cache *xt_cache = &wc->xt_cache;
86 - struct extent_cache_element *prev, *next;
132 +static inline void work_done(struct rrdeng_work *work_request) {
133 + netdata_spinlock_lock(&work_request_globals.protected.spinlock);
134 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(work_request_globals.protected.available_items, work_request, cache.prev, cache.next);
135 + work_request_globals.protected.available++;
136 + netdata_spinlock_unlock(&work_request_globals.protected.spinlock);
137 +}
138
88 - prev = xt_cache_elem->prev;
89 - next = xt_cache_elem->next;
139 +void work_standard_worker(uv_work_t *req) {
140 + __atomic_add_fetch(&work_request_globals.atomics.executing, 1, __ATOMIC_RELAXED);
141
91 - if (likely(NULL != prev)) {
92 - prev->next = next;
93 - }
94 - if (likely(NULL != next)) {
95 - next->prev = prev;
142 + register_libuv_worker_jobs();
143 + worker_is_busy(UV_EVENT_WORKER_INIT);
144 +
145 + struct rrdeng_work *work_request = req->data;
146 + work_request->work_cb(work_request->ctx, work_request->data, work_request->completion, req);
147 + worker_is_idle();
148 +
149 + __atomic_sub_fetch(&work_request_globals.atomics.dispatched, 1, __ATOMIC_RELAXED);
150 + __atomic_sub_fetch(&work_request_globals.atomics.executing, 1, __ATOMIC_RELAXED);
151 + __atomic_add_fetch(&work_request_globals.atomics.pending_cb, 1, __ATOMIC_RELAXED);
152 +
153 + // signal the event loop a worker is available
154 + fatal_assert(0 == uv_async_send(&rrdeng_main.async));
155 +}
156 +
157 +void after_work_standard_callback(uv_work_t* req, int status) {
158 + struct rrdeng_work *work_request = req->data;
159 +
160 + worker_is_busy(RRDENG_OPCODE_MAX + work_request->opcode);
161 +
162 + if(work_request->after_work_cb)
163 + work_request->after_work_cb(work_request->ctx, work_request->data, work_request->completion, req, status);
164 +
165 + work_done(work_request);
166 + __atomic_sub_fetch(&work_request_globals.atomics.pending_cb, 1, __ATOMIC_RELAXED);
167 +
168 + worker_is_idle();
169 +}
170 +
171 +static bool work_dispatch(struct rrdengine_instance *ctx, void *data, struct completion *completion, enum rrdeng_opcode opcode, work_cb work_cb, after_work_cb after_work_cb) {
172 + struct rrdeng_work *work_request = NULL;
173 +
174 + internal_fatal(rrdeng_main.tid != gettid(), "work_dispatch() can only be run from the event loop thread");
175 +
176 + netdata_spinlock_lock(&work_request_globals.protected.spinlock);
177 +
178 + if(likely(work_request_globals.protected.available_items)) {
179 + work_request = work_request_globals.protected.available_items;
180 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(work_request_globals.protected.available_items, work_request, cache.prev, cache.next);
181 + work_request_globals.protected.available--;
182 }
97 - if (unlikely(xt_cache_elem == xt_cache->replaceQ_head)) {
98 - xt_cache->replaceQ_head = next;
183 +
184 + netdata_spinlock_unlock(&work_request_globals.protected.spinlock);
185 +
186 + if(unlikely(!work_request)) {
187 + work_request = mallocz(sizeof(struct rrdeng_work));
188 + __atomic_add_fetch(&work_request_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
189 }
100 - if (unlikely(xt_cache_elem == xt_cache->replaceQ_tail)) {
101 - xt_cache->replaceQ_tail = prev;
190 +
191 + memset(work_request, 0, sizeof(struct rrdeng_work));
192 + work_request->req.data = work_request;
193 + work_request->ctx = ctx;
194 + work_request->data = data;
195 + work_request->completion = completion;
196 + work_request->work_cb = work_cb;
197 + work_request->after_work_cb = after_work_cb;
198 + work_request->opcode = opcode;
199 +
200 + if(uv_queue_work(&rrdeng_main.loop, &work_request->req, work_standard_worker, after_work_standard_callback)) {
201 + internal_fatal(true, "DBENGINE: cannot queue work");
202 + work_done(work_request);
203 + return false;
204 }
103 - xt_cache_elem->prev = xt_cache_elem->next = NULL;
104 -}
205
106 -static inline void xt_cache_replaceQ_set_hot(struct rrdengine_worker_config* wc,
107 - struct extent_cache_element *xt_cache_elem)
108 -{
109 - xt_cache_replaceQ_delete(wc, xt_cache_elem);
110 - xt_cache_replaceQ_insert(wc, xt_cache_elem);
206 + __atomic_add_fetch(&work_request_globals.atomics.dispatched, 1, __ATOMIC_RELAXED);
207 +
208 + return true;
209 }
210
113 -/* Returns the index of the cached extent if it was successfully inserted in the extent cache, otherwise -1 */
114 -static int try_insert_into_xt_cache(struct rrdengine_worker_config* wc, struct extent_info *extent)
115 -{
116 - struct extent_cache *xt_cache = &wc->xt_cache;
117 - struct extent_cache_element *xt_cache_elem;
118 - unsigned idx;
119 - int ret;
211 +// ----------------------------------------------------------------------------
212 +// page descriptor cache
213 +
214 +static struct {
215 + struct {
216 + SPINLOCK spinlock;
217 + struct page_descr_with_data *available_items;
218 + size_t available;
219 + } protected;
220 +
221 + struct {
222 + size_t allocated;
223 + } atomics;
224 +} page_descriptor_globals = {
225 + .protected = {
226 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
227 + .available_items = NULL,
228 + .available = 0,
229 + },
230 + .atomics = {
231 + .allocated = 0,
232 + },
233 +};
234
121 - ret = find_first_zero(xt_cache->allocation_bitmap);
122 - if (-1 == ret || ret >= MAX_CACHED_EXTENTS) {
123 - for (xt_cache_elem = xt_cache->replaceQ_head ; NULL != xt_cache_elem ; xt_cache_elem = xt_cache_elem->next) {
124 - idx = xt_cache_elem - xt_cache->extent_array;
125 - if (!check_bit(xt_cache->inflight_bitmap, idx)) {
126 - xt_cache_replaceQ_delete(wc, xt_cache_elem);
127 - break;
128 - }
129 - }
130 - if (NULL == xt_cache_elem)
131 - return -1;
132 - } else {
133 - idx = (unsigned)ret;
134 - xt_cache_elem = &xt_cache->extent_array[idx];
235 +static void page_descriptor_cleanup(void) {
236 + netdata_spinlock_lock(&page_descriptor_globals.protected.spinlock);
237 +
238 + while(page_descriptor_globals.protected.available_items && page_descriptor_globals.protected.available > MAX_PAGES_PER_EXTENT) {
239 + struct page_descr_with_data *item = page_descriptor_globals.protected.available_items;
240 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(page_descriptor_globals.protected.available_items, item, cache.prev, cache.next);
241 + freez(item);
242 + page_descriptor_globals.protected.available--;
243 + __atomic_sub_fetch(&page_descriptor_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
244 }
136 - xt_cache_elem->extent = extent;
137 - xt_cache_elem->fileno = extent->datafile->fileno;
138 - xt_cache_elem->inflight_io_descr = NULL;
139 - xt_cache_replaceQ_insert(wc, xt_cache_elem);
140 - modify_bit(&xt_cache->allocation_bitmap, idx, 1);
245
142 - return (int)idx;
246 + netdata_spinlock_unlock(&page_descriptor_globals.protected.spinlock);
247 }
248
145 -/**
146 - * Returns 0 if the cached extent was found in the extent cache, 1 otherwise.
147 - * Sets *idx to point to the position of the extent inside the cache.
148 - **/
149 -static uint8_t lookup_in_xt_cache(struct rrdengine_worker_config* wc, struct extent_info *extent, unsigned *idx)
150 -{
151 - struct extent_cache *xt_cache = &wc->xt_cache;
152 - struct extent_cache_element *xt_cache_elem;
153 - unsigned i;
249 +struct page_descr_with_data *page_descriptor_get(void) {
250 + struct page_descr_with_data *descr = NULL;
251
155 - for (i = 0 ; i < MAX_CACHED_EXTENTS ; ++i) {
156 - xt_cache_elem = &xt_cache->extent_array[i];
157 - if (check_bit(xt_cache->allocation_bitmap, i) && xt_cache_elem->extent == extent &&
158 - xt_cache_elem->fileno == extent->datafile->fileno) {
159 - *idx = i;
160 - return 0;
161 - }
252 + netdata_spinlock_lock(&page_descriptor_globals.protected.spinlock);
253 +
254 + if(likely(page_descriptor_globals.protected.available_items)) {
255 + descr = page_descriptor_globals.protected.available_items;
256 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(page_descriptor_globals.protected.available_items, descr, cache.prev, cache.next);
257 + page_descriptor_globals.protected.available--;
258 }
163 - return 1;
164 -}
259
166 -#if 0 /* disabled code */
167 -static void delete_from_xt_cache(struct rrdengine_worker_config* wc, unsigned idx)
168 -{
169 - struct extent_cache *xt_cache = &wc->xt_cache;
170 - struct extent_cache_element *xt_cache_elem;
171 -
172 - xt_cache_elem = &xt_cache->extent_array[idx];
173 - xt_cache_replaceQ_delete(wc, xt_cache_elem);
174 - xt_cache_elem->extent = NULL;
175 - modify_bit(&wc->xt_cache.allocation_bitmap, idx, 0); /* invalidate it */
176 - modify_bit(&wc->xt_cache.inflight_bitmap, idx, 0); /* not in-flight anymore */
260 + netdata_spinlock_unlock(&page_descriptor_globals.protected.spinlock);
261 +
262 + if(unlikely(!descr)) {
263 + descr = mallocz(sizeof(struct page_descr_with_data));
264 + __atomic_add_fetch(&page_descriptor_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
265 + }
266 +
267 + memset(descr, 0, sizeof(struct page_descr_with_data));
268 + return descr;
269 }
178 -#endif
270
180 -void enqueue_inflight_read_to_xt_cache(struct rrdengine_worker_config* wc, unsigned idx,
181 - struct extent_io_descriptor *xt_io_descr)
182 -{
183 - struct extent_cache *xt_cache = &wc->xt_cache;
184 - struct extent_cache_element *xt_cache_elem;
185 - struct extent_io_descriptor *old_next;
186 -
187 - xt_cache_elem = &xt_cache->extent_array[idx];
188 - old_next = xt_cache_elem->inflight_io_descr->next;
189 - xt_cache_elem->inflight_io_descr->next = xt_io_descr;
190 - xt_io_descr->next = old_next;
271 +static inline void page_descriptor_release(struct page_descr_with_data *descr) {
272 + if(unlikely(!descr)) return;
273 +
274 + netdata_spinlock_lock(&page_descriptor_globals.protected.spinlock);
275 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(page_descriptor_globals.protected.available_items, descr, cache.prev, cache.next);
276 + page_descriptor_globals.protected.available++;
277 + netdata_spinlock_unlock(&page_descriptor_globals.protected.spinlock);
278 }
279
193 -void read_cached_extent_cb(struct rrdengine_worker_config* wc, unsigned idx, struct extent_io_descriptor *xt_io_descr)
194 -{
195 - unsigned i, j, page_offset;
196 - struct rrdengine_instance *ctx = wc->ctx;
197 - struct rrdeng_page_descr *descr;
198 - struct page_cache_descr *pg_cache_descr;
199 - void *page;
200 - struct extent_info *extent = xt_io_descr->descr_array[0]->extent;
201 -
202 - for (i = 0 ; i < xt_io_descr->descr_count; ++i) {
203 - page = dbengine_page_alloc();
204 - descr = xt_io_descr->descr_array[i];
205 - for (j = 0, page_offset = 0 ; j < extent->number_of_pages ; ++j) {
206 - /* care, we don't hold the descriptor mutex */
207 - if (!uuid_compare(*extent->pages[j]->id, *descr->id) &&
208 - extent->pages[j]->page_length == descr->page_length &&
209 - extent->pages[j]->start_time_ut == descr->start_time_ut &&
210 - extent->pages[j]->end_time_ut == descr->end_time_ut) {
211 - break;
212 - }
213 - page_offset += extent->pages[j]->page_length;
280 +// ----------------------------------------------------------------------------
281 +// extent io descriptor cache
282 +
283 +static struct {
284 + struct {
285 + SPINLOCK spinlock;
286 + struct extent_io_descriptor *available_items;
287 + size_t available;
288 + } protected;
289 +
290 + struct {
291 + size_t allocated;
292 + } atomics;
293 +
294 +} extent_io_descriptor_globals = {
295 + .protected = {
296 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
297 + .available_items = NULL,
298 + .available = 0,
299 + },
300 + .atomics = {
301 + .allocated = 0,
302 + },
303 +};
304
215 - }
216 - /* care, we don't hold the descriptor mutex */
217 - (void) memcpy(page, wc->xt_cache.extent_array[idx].pages + page_offset, descr->page_length);
218 -
219 - rrdeng_page_descr_mutex_lock(ctx, descr);
220 - pg_cache_descr = descr->pg_cache_descr;
221 - pg_cache_descr->page = page;
222 - pg_cache_descr->flags |= RRD_PAGE_POPULATED;
223 - pg_cache_descr->flags &= ~RRD_PAGE_READ_PENDING;
224 - rrdeng_page_descr_mutex_unlock(ctx, descr);
225 - pg_cache_replaceQ_insert(ctx, descr);
226 - if (xt_io_descr->release_descr) {
227 - pg_cache_put(ctx, descr);
228 - } else {
229 - debug(D_RRDENGINE, "%s: Waking up waiters.", __func__);
230 - pg_cache_wake_up_waiters(ctx, descr);
231 - }
305 +static void extent_io_descriptor_cleanup(void) {
306 + netdata_spinlock_lock(&extent_io_descriptor_globals.protected.spinlock);
307 + while(extent_io_descriptor_globals.protected.available_items && extent_io_descriptor_globals.protected.available > (size_t)libuv_worker_threads) {
308 + struct extent_io_descriptor *item = extent_io_descriptor_globals.protected.available_items;
309 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(extent_io_descriptor_globals.protected.available_items, item, cache.prev, cache.next);
310 + freez(item);
311 + extent_io_descriptor_globals.protected.available--;
312 + __atomic_sub_fetch(&extent_io_descriptor_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
313 }
233 - if (xt_io_descr->completion)
234 - completion_mark_complete(xt_io_descr->completion);
235 - freez(xt_io_descr);
314 + netdata_spinlock_unlock(&extent_io_descriptor_globals.protected.spinlock);
315 }
316
238 -static void fill_page_with_nulls(void *page, uint32_t page_length, uint8_t type) {
239 - switch(type) {
240 - case PAGE_METRICS: {
241 - storage_number n = pack_storage_number(NAN, SN_FLAG_NONE);
242 - storage_number *array = (storage_number *)page;
243 - size_t slots = page_length / sizeof(n);
244 - for(size_t i = 0; i < slots ; i++)
245 - array[i] = n;
246 - }
247 - break;
317 +static struct extent_io_descriptor *extent_io_descriptor_get(void) {
318 + struct extent_io_descriptor *xt_io_descr = NULL;
319
249 - case PAGE_TIER: {
250 - storage_number_tier1_t n = {
251 - .min_value = NAN,
252 - .max_value = NAN,
253 - .sum_value = NAN,
254 - .count = 1,
255 - .anomaly_count = 0,
256 - };
257 - storage_number_tier1_t *array = (storage_number_tier1_t *)page;
258 - size_t slots = page_length / sizeof(n);
259 - for(size_t i = 0; i < slots ; i++)
260 - array[i] = n;
261 - }
262 - break;
320 + netdata_spinlock_lock(&extent_io_descriptor_globals.protected.spinlock);
321
264 - default: {
265 - static bool logged = false;
266 - if(!logged) {
267 - error("DBENGINE: cannot fill page with nulls on unknown page type id %d", type);
268 - logged = true;
269 - }
270 - memset(page, 0, page_length);
271 - }
322 + if(likely(extent_io_descriptor_globals.protected.available_items)) {
323 + xt_io_descr = extent_io_descriptor_globals.protected.available_items;
324 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(extent_io_descriptor_globals.protected.available_items, xt_io_descr, cache.prev, cache.next);
325 + extent_io_descriptor_globals.protected.available--;
326 }
327 +
328 + netdata_spinlock_unlock(&extent_io_descriptor_globals.protected.spinlock);
329 +
330 + if(unlikely(!xt_io_descr)) {
331 + xt_io_descr = mallocz(sizeof(struct extent_io_descriptor));
332 + __atomic_add_fetch(&extent_io_descriptor_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
333 + }
334 +
335 + memset(xt_io_descr, 0, sizeof(struct extent_io_descriptor));
336 + return xt_io_descr;
337 }
338
275 -struct rrdeng_page_descr *get_descriptor(struct pg_cache_page_index *page_index, time_t start_time_s)
276 -{
277 - uv_rwlock_rdlock(&page_index->lock);
278 - Pvoid_t *PValue = JudyLGet(page_index->JudyL_array, start_time_s, PJE0);
279 - struct rrdeng_page_descr *descr = unlikely(NULL == PValue) ? NULL : *PValue;
280 - uv_rwlock_rdunlock(&page_index->lock);
281 - return descr;
282 -};
339 +static inline void extent_io_descriptor_release(struct extent_io_descriptor *xt_io_descr) {
340 + if(unlikely(!xt_io_descr)) return;
341
284 -static void do_extent_processing (struct rrdengine_worker_config *wc, struct extent_io_descriptor *xt_io_descr, bool read_failed)
285 -{
286 - struct rrdengine_instance *ctx = wc->ctx;
287 - struct page_cache *pg_cache = &ctx->pg_cache;
288 - struct rrdeng_page_descr *descr;
289 - struct page_cache_descr *pg_cache_descr;
290 - int ret;
291 - unsigned i, j, count;
292 - void *page, *uncompressed_buf = NULL;
293 - uint32_t payload_length, payload_offset, page_offset, uncompressed_payload_length = 0;
294 - uint8_t have_read_error = 0;
295 - /* persistent structures */
296 - struct rrdeng_df_extent_header *header;
297 - struct rrdeng_df_extent_trailer *trailer;
298 - uLong crc;
342 + netdata_spinlock_lock(&extent_io_descriptor_globals.protected.spinlock);
343 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(extent_io_descriptor_globals.protected.available_items, xt_io_descr, cache.prev, cache.next);
344 + extent_io_descriptor_globals.protected.available++;
345 + netdata_spinlock_unlock(&extent_io_descriptor_globals.protected.spinlock);
346 +}
347
300 - header = xt_io_descr->buf;
301 - payload_length = header->payload_length;
302 - count = header->number_of_pages;
303 - payload_offset = sizeof(*header) + sizeof(header->descr[0]) * count;
304 - trailer = xt_io_descr->buf + xt_io_descr->bytes - sizeof(*trailer);
348 +// ----------------------------------------------------------------------------
349 +// query handle cache
350 +
351 +static struct {
352 + struct {
353 + SPINLOCK spinlock;
354 + struct rrdeng_query_handle *available_items;
355 + size_t available;
356 + } protected;
357 +
358 + struct {
359 + size_t allocated;
360 + } atomics;
361 +} rrdeng_query_handle_globals = {
362 + .protected = {
363 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
364 + .available_items = NULL,
365 + .available = 0,
366 + },
367 + .atomics = {
368 + .allocated = 0,
369 + },
370 +};
371
306 - if (unlikely(read_failed)) {
307 - struct rrdengine_datafile *datafile = xt_io_descr->descr_array[0]->extent->datafile;
372 +static void rrdeng_query_handle_cleanup(void) {
373 + netdata_spinlock_lock(&rrdeng_query_handle_globals.protected.spinlock);
374
309 - ++ctx->stats.io_errors;
310 - rrd_stat_atomic_add(&global_io_errors, 1);
311 - have_read_error = 1;
312 - error("%s: uv_fs_read - extent at offset %"PRIu64"(%u) in datafile %u-%u.", __func__, xt_io_descr->pos,
313 - xt_io_descr->bytes, datafile->tier, datafile->fileno);
314 - goto after_crc_check;
315 - }
316 - crc = crc32(0L, Z_NULL, 0);
317 - crc = crc32(crc, xt_io_descr->buf, xt_io_descr->bytes - sizeof(*trailer));
318 - ret = crc32cmp(trailer->checksum, crc);
319 -#ifdef NETDATA_INTERNAL_CHECKS
320 - {
321 - struct rrdengine_datafile *datafile = xt_io_descr->descr_array[0]->extent->datafile;
322 - debug(D_RRDENGINE, "%s: Extent at offset %"PRIu64"(%u) was read from datafile %u-%u. CRC32 check: %s", __func__,
323 - xt_io_descr->pos, xt_io_descr->bytes, datafile->tier, datafile->fileno, ret ? "FAILED" : "SUCCEEDED");
375 + while(rrdeng_query_handle_globals.protected.available_items && rrdeng_query_handle_globals.protected.available > 10) {
376 + struct rrdeng_query_handle *item = rrdeng_query_handle_globals.protected.available_items;
377 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rrdeng_query_handle_globals.protected.available_items, item, cache.prev, cache.next);
378 + freez(item);
379 + rrdeng_query_handle_globals.protected.available--;
380 + __atomic_sub_fetch(&rrdeng_query_handle_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
381 }
325 -#endif
326 - if (unlikely(ret)) {
327 - struct rrdengine_datafile *datafile = xt_io_descr->descr_array[0]->extent->datafile;
382
329 - ++ctx->stats.io_errors;
330 - rrd_stat_atomic_add(&global_io_errors, 1);
331 - have_read_error = 1;
332 - error("%s: Extent at offset %"PRIu64"(%u) was read from datafile %u-%u. CRC32 check: FAILED", __func__,
333 - xt_io_descr->pos, xt_io_descr->bytes, datafile->tier, datafile->fileno);
383 + netdata_spinlock_unlock(&rrdeng_query_handle_globals.protected.spinlock);
384 +}
385 +
386 +struct rrdeng_query_handle *rrdeng_query_handle_get(void) {
387 + struct rrdeng_query_handle *handle = NULL;
388 +
389 + netdata_spinlock_lock(&rrdeng_query_handle_globals.protected.spinlock);
390 +
391 + if(likely(rrdeng_query_handle_globals.protected.available_items)) {
392 + handle = rrdeng_query_handle_globals.protected.available_items;
393 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rrdeng_query_handle_globals.protected.available_items, handle, cache.prev, cache.next);
394 + rrdeng_query_handle_globals.protected.available--;
395 }
396
336 -after_crc_check:
337 - if (!have_read_error && RRD_NO_COMPRESSION != header->compression_algorithm) {
338 - uncompressed_payload_length = 0;
339 - for (i = 0 ; i < count ; ++i) {
340 - uncompressed_payload_length += header->descr[i].page_length;
341 - }
342 - uncompressed_buf = mallocz(uncompressed_payload_length);
343 - ret = LZ4_decompress_safe(xt_io_descr->buf + payload_offset, uncompressed_buf,
344 - payload_length, uncompressed_payload_length);
345 - ctx->stats.before_decompress_bytes += payload_length;
346 - ctx->stats.after_decompress_bytes += ret;
347 - debug(D_RRDENGINE, "LZ4 decompressed %u bytes to %d bytes.", payload_length, ret);
348 - /* care, we don't hold the descriptor mutex */
397 + netdata_spinlock_unlock(&rrdeng_query_handle_globals.protected.spinlock);
398 +
399 + if(unlikely(!handle)) {
400 + handle = mallocz(sizeof(struct rrdeng_query_handle));
401 + __atomic_add_fetch(&rrdeng_query_handle_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
402 }
350 - {
351 - uint8_t xt_is_cached = 0;
352 - unsigned xt_idx;
353 - struct extent_info *extent = xt_io_descr->descr_array[0]->extent;
354 -
355 - xt_is_cached = !lookup_in_xt_cache(wc, extent, &xt_idx);
356 - if (xt_is_cached && check_bit(wc->xt_cache.inflight_bitmap, xt_idx)) {
357 - struct extent_cache *xt_cache = &wc->xt_cache;
358 - struct extent_cache_element *xt_cache_elem = &xt_cache->extent_array[xt_idx];
359 - struct extent_io_descriptor *curr, *next;
360 -
361 - if (have_read_error) {
362 - memset(xt_cache_elem->pages, 0, sizeof(xt_cache_elem->pages));
363 - } else if (RRD_NO_COMPRESSION == header->compression_algorithm) {
364 - (void)memcpy(xt_cache_elem->pages, xt_io_descr->buf + payload_offset, payload_length);
365 - } else {
366 - (void)memcpy(xt_cache_elem->pages, uncompressed_buf, uncompressed_payload_length);
367 - }
368 - /* complete all connected in-flight read requests */
369 - for (curr = xt_cache_elem->inflight_io_descr->next ; curr ; curr = next) {
370 - next = curr->next;
371 - read_cached_extent_cb(wc, xt_idx, curr);
372 - }
373 - xt_cache_elem->inflight_io_descr = NULL;
374 - modify_bit(&xt_cache->inflight_bitmap, xt_idx, 0); /* not in-flight anymore */
375 - }
403 +
404 + memset(handle, 0, sizeof(struct rrdeng_query_handle));
405 + return handle;
406 +}
407 +
408 +void rrdeng_query_handle_release(struct rrdeng_query_handle *handle) {
409 + if(unlikely(!handle)) return;
410 +
411 + netdata_spinlock_lock(&rrdeng_query_handle_globals.protected.spinlock);
412 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(rrdeng_query_handle_globals.protected.available_items, handle, cache.prev, cache.next);
413 + rrdeng_query_handle_globals.protected.available++;
414 + netdata_spinlock_unlock(&rrdeng_query_handle_globals.protected.spinlock);
415 +}
416 +
417 +// ----------------------------------------------------------------------------
418 +// WAL cache
419 +
420 +static struct {
421 + struct {
422 + SPINLOCK spinlock;
423 + WAL *available_items;
424 + size_t available;
425 + } protected;
426 +
427 + struct {
428 + size_t allocated;
429 + } atomics;
430 +} wal_globals = {
431 + .protected = {
432 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
433 + .available_items = NULL,
434 + .available = 0,
435 + },
436 + .atomics = {
437 + .allocated = 0,
438 + },
439 +};
440 +
441 +static void wal_cleanup(void) {
442 + netdata_spinlock_lock(&wal_globals.protected.spinlock);
443 +
444 + while(wal_globals.protected.available_items && wal_globals.protected.available > storage_tiers) {
445 + WAL *wal = wal_globals.protected.available_items;
446 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(wal_globals.protected.available_items, wal, cache.prev, cache.next);
447 + posix_memfree(wal->buf);
448 + freez(wal);
449 + wal_globals.protected.available--;
450 + __atomic_sub_fetch(&wal_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
451 }
452
378 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
379 - Pvoid_t *PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, xt_io_descr->descr_array[0]->id, sizeof(uuid_t));
380 - struct pg_cache_page_index *page_index = likely( NULL != PValue) ? *PValue : NULL;
381 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
382 -
383 -
384 - for (i = 0, page_offset = 0; i < count; page_offset += header->descr[i++].page_length) {
385 - uint8_t is_prefetched_page;
386 - descr = NULL;
387 - for (j = 0 ; j < xt_io_descr->descr_count; ++j) {
388 - struct rrdeng_page_descr descrj;
389 -
390 - descrj = xt_io_descr->descr_read_array[j];
391 - /* care, we don't hold the descriptor mutex */
392 - if (!uuid_compare(*(uuid_t *) header->descr[i].uuid, *descrj.id) &&
393 - header->descr[i].page_length == descrj.page_length &&
394 - header->descr[i].start_time_ut == descrj.start_time_ut &&
395 - header->descr[i].end_time_ut == descrj.end_time_ut) {
396 - //descr = descrj;
397 - descr = get_descriptor(page_index, (time_t) (descrj.start_time_ut / USEC_PER_SEC));
398 - if (unlikely(!descr)) {
399 - error_limit_static_thread_var(erl, 1, 0);
400 - error_limit(&erl, "%s: Required descriptor is not in the page index anymore", __FUNCTION__);
401 - }
402 - break;
403 - }
404 - }
405 - is_prefetched_page = 0;
406 - if (!descr) { /* This extent page has not been requested. Try populating it for locality (best effort). */
407 - descr = pg_cache_lookup_unpopulated_and_lock(ctx, (uuid_t *)header->descr[i].uuid,
408 - header->descr[i].start_time_ut);
409 - if (!descr)
410 - continue; /* Failed to reserve a suitable page */
411 - is_prefetched_page = 1;
412 - }
413 - page = dbengine_page_alloc();
414 -
415 - /* care, we don't hold the descriptor mutex */
416 - if (have_read_error) {
417 - fill_page_with_nulls(page, descr->page_length, descr->type);
418 - } else if (RRD_NO_COMPRESSION == header->compression_algorithm) {
419 - (void) memcpy(page, xt_io_descr->buf + payload_offset + page_offset, descr->page_length);
420 - } else {
421 - (void) memcpy(page, uncompressed_buf + page_offset, descr->page_length);
422 - }
423 - rrdeng_page_descr_mutex_lock(ctx, descr);
424 - pg_cache_descr = descr->pg_cache_descr;
425 - pg_cache_descr->page = page;
426 - pg_cache_descr->flags |= RRD_PAGE_POPULATED;
427 - pg_cache_descr->flags &= ~RRD_PAGE_READ_PENDING;
428 - rrdeng_page_descr_mutex_unlock(ctx, descr);
429 - pg_cache_replaceQ_insert(ctx, descr);
430 - if (xt_io_descr->release_descr || is_prefetched_page) {
431 - pg_cache_put(ctx, descr);
432 - } else {
433 - debug(D_RRDENGINE, "%s: Waking up waiters.", __func__);
434 - pg_cache_wake_up_waiters(ctx, descr);
435 - }
453 + netdata_spinlock_unlock(&wal_globals.protected.spinlock);
454 +}
455 +
456 +WAL *wal_get(struct rrdengine_instance *ctx, unsigned size) {
457 + if(!size || size > RRDENG_BLOCK_SIZE)
458 + fatal("DBENGINE: invalid WAL size requested");
459 +
460 + WAL *wal = NULL;
461 +
462 + netdata_spinlock_lock(&wal_globals.protected.spinlock);
463 +
464 + if(likely(wal_globals.protected.available_items)) {
465 + wal = wal_globals.protected.available_items;
466 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(wal_globals.protected.available_items, wal, cache.prev, cache.next);
467 + wal_globals.protected.available--;
468 }
437 - if (!have_read_error && RRD_NO_COMPRESSION != header->compression_algorithm) {
438 - freez(uncompressed_buf);
469 +
470 + uint64_t transaction_id = ctx->commit_log.transaction_id++;
471 + netdata_spinlock_unlock(&wal_globals.protected.spinlock);
472 +
473 + if(unlikely(!wal)) {
474 + wal = mallocz(sizeof(WAL));
475 + wal->buf_size = RRDENG_BLOCK_SIZE;
476 + int ret = posix_memalign((void *)&wal->buf, RRDFILE_ALIGNMENT, wal->buf_size);
477 + if (unlikely(ret))
478 + fatal("DBENGINE: posix_memalign:%s", strerror(ret));
479 + __atomic_add_fetch(&wal_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
480 }
440 - if (xt_io_descr->completion)
441 - completion_mark_complete(xt_io_descr->completion);
481 +
482 + // these need to survive
483 + unsigned buf_size = wal->buf_size;
484 + void *buf = wal->buf;
485 +
486 + memset(wal, 0, sizeof(WAL));
487 +
488 + // put them back
489 + wal->buf_size = buf_size;
490 + wal->buf = buf;
491 +
492 + memset(wal->buf, 0, wal->buf_size);
493 +
494 + wal->transaction_id = transaction_id;
495 + wal->size = size;
496 +
497 + return wal;
498 }
499
444 -static void read_extent_cb(uv_fs_t *req)
445 -{
446 - struct rrdengine_worker_config *wc = req->loop->data;
447 - struct extent_io_descriptor *xt_io_descr;
500 +void wal_release(WAL *wal) {
501 + if(unlikely(!wal)) return;
502
449 - xt_io_descr = req->data;
450 - do_extent_processing(wc, xt_io_descr, req->result < 0);
451 - uv_fs_req_cleanup(req);
452 - posix_memfree(xt_io_descr->buf);
453 - freez(xt_io_descr);
503 + netdata_spinlock_lock(&wal_globals.protected.spinlock);
504 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(wal_globals.protected.available_items, wal, cache.prev, cache.next);
505 + wal_globals.protected.available++;
506 + netdata_spinlock_unlock(&wal_globals.protected.spinlock);
507 }
508
456 -static void read_mmap_extent_cb(uv_work_t *req, int status __maybe_unused)
457 -{
458 - struct rrdengine_worker_config *wc = req->loop->data;
459 - struct rrdengine_instance *ctx = wc->ctx;
460 - struct extent_io_descriptor *xt_io_descr;
461 - xt_io_descr = req->data;
509 +// ----------------------------------------------------------------------------
510 +// command queue cache
511
463 - if (likely(xt_io_descr->map_base)) {
464 - do_extent_processing(wc, xt_io_descr, false);
465 - munmap(xt_io_descr->map_base, xt_io_descr->map_length);
466 - freez(xt_io_descr);
467 - return;
468 - }
512 +struct rrdeng_cmd {
513 + struct rrdengine_instance *ctx;
514 + enum rrdeng_opcode opcode;
515 + void *data;
516 + struct completion *completion;
517 + enum storage_priority priority;
518 +
519 + struct {
520 + struct rrdeng_cmd *prev;
521 + struct rrdeng_cmd *next;
522 + } cache;
523 +};
524
470 - // MMAP failed, so do uv_fs_read
471 - int ret = posix_memalign((void *)&xt_io_descr->buf, RRDFILE_ALIGNMENT, ALIGN_BYTES_CEILING(xt_io_descr->bytes));
472 - if (unlikely(ret)) {
473 - fatal("posix_memalign:%s", strerror(ret));
525 +static struct {
526 + struct {
527 + SPINLOCK spinlock;
528 + struct rrdeng_cmd *available_items;
529 + size_t available;
530 +
531 + struct {
532 + size_t allocated;
533 + } atomics;
534 + } cache;
535 +
536 + struct {
537 + SPINLOCK spinlock;
538 + size_t waiting;
539 + struct rrdeng_cmd *waiting_items_by_priority[STORAGE_PRIO_MAX_DONT_USE];
540 + size_t executed_by_priority[STORAGE_PRIO_MAX_DONT_USE];
541 + } queue;
542 +
543 +
544 +} rrdeng_cmd_globals = {
545 + .cache = {
546 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
547 + .available_items = NULL,
548 + .available = 0,
549 + .atomics = {
550 + .allocated = 0,
551 + },
552 + },
553 + .queue = {
554 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
555 + .waiting = 0,
556 + },
557 +};
558 +
559 +static void rrdeng_cmd_cleanup(void) {
560 + netdata_spinlock_lock(&rrdeng_cmd_globals.cache.spinlock);
561 + while(rrdeng_cmd_globals.cache.available_items && rrdeng_cmd_globals.cache.available > 100) {
562 + struct rrdeng_cmd *item = rrdeng_cmd_globals.cache.available_items;
563 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rrdeng_cmd_globals.cache.available_items, item, cache.prev, cache.next);
564 + freez(item);
565 + rrdeng_cmd_globals.cache.available--;
566 + __atomic_sub_fetch(&rrdeng_cmd_globals.cache.atomics.allocated, 1, __ATOMIC_RELAXED);
567 }
475 - unsigned real_io_size = ALIGN_BYTES_CEILING( xt_io_descr->bytes);
476 - xt_io_descr->iov = uv_buf_init((void *)xt_io_descr->buf, real_io_size);
477 - xt_io_descr->req.data = xt_io_descr;
478 - ret = uv_fs_read(req->loop, &xt_io_descr->req, xt_io_descr->file, &xt_io_descr->iov, 1, (unsigned) xt_io_descr->pos, read_extent_cb);
479 - fatal_assert(-1 != ret);
480 - ctx->stats.io_read_bytes += real_io_size;
481 - ctx->stats.io_read_extent_bytes += real_io_size;
568 + netdata_spinlock_unlock(&rrdeng_cmd_globals.cache.spinlock);
569 }
570
484 -static void do_mmap_read_extent(uv_work_t *req)
485 -{
486 - struct extent_io_descriptor *xt_io_descr = (struct extent_io_descriptor * )req->data;
487 - struct rrdengine_worker_config *wc = req->loop->data;
488 - struct rrdengine_instance *ctx = wc->ctx;
489 -
490 - off_t map_start = ALIGN_BYTES_FLOOR(xt_io_descr->pos);
491 - size_t length = ALIGN_BYTES_CEILING(xt_io_descr->pos + xt_io_descr->bytes) - map_start;
492 - unsigned real_io_size = xt_io_descr->bytes;
493 -
494 - void *data = mmap(NULL, length, PROT_READ, MAP_SHARED, xt_io_descr->file, map_start);
495 - if (likely(data != MAP_FAILED)) {
496 - xt_io_descr->map_base = data;
497 - xt_io_descr->map_length = length;
498 - xt_io_descr->buf = data + (xt_io_descr->pos - map_start);
499 - ctx->stats.io_read_bytes += real_io_size;
500 - ctx->stats.io_read_extent_bytes += real_io_size;
571 +void rrdeng_enq_cmd(struct rrdengine_instance *ctx, enum rrdeng_opcode opcode, void *data, struct completion *completion, STORAGE_PRIORITY priority) {
572 + struct rrdeng_cmd *cmd = NULL;
573 +
574 + if(unlikely(priority >= STORAGE_PRIO_MAX_DONT_USE))
575 + priority = STORAGE_PRIORITY_NORMAL;
576 +
577 + netdata_spinlock_lock(&rrdeng_cmd_globals.cache.spinlock);
578 + if(likely(rrdeng_cmd_globals.cache.available_items)) {
579 + cmd = rrdeng_cmd_globals.cache.available_items;
580 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rrdeng_cmd_globals.cache.available_items, cmd, cache.prev, cache.next);
581 + rrdeng_cmd_globals.cache.available--;
582 }
502 -}
583 + netdata_spinlock_unlock(&rrdeng_cmd_globals.cache.spinlock);
584
504 -static void do_read_extent(struct rrdengine_worker_config* wc,
505 - struct rrdeng_page_descr **descr,
506 - unsigned count,
507 - uint8_t release_descr)
508 -{
509 - struct rrdengine_instance *ctx = wc->ctx;
510 - struct page_cache_descr *pg_cache_descr;
511 - int ret;
512 - unsigned i, size_bytes, pos;
513 - struct extent_io_descriptor *xt_io_descr;
514 - struct rrdengine_datafile *datafile;
515 - struct extent_info *extent = descr[0]->extent;
516 - uint8_t xt_is_cached = 0, xt_is_inflight = 0;
517 - unsigned xt_idx;
518 -
519 - datafile = extent->datafile;
520 - pos = extent->offset;
521 - size_bytes = extent->size;
522 -
523 - xt_io_descr = callocz(1, sizeof(*xt_io_descr));
524 - for (i = 0 ; i < count; ++i) {
525 - rrdeng_page_descr_mutex_lock(ctx, descr[i]);
526 - pg_cache_descr = descr[i]->pg_cache_descr;
527 - pg_cache_descr->flags |= RRD_PAGE_READ_PENDING;
528 - rrdeng_page_descr_mutex_unlock(ctx, descr[i]);
529 - xt_io_descr->descr_array[i] = descr[i];
530 - xt_io_descr->descr_read_array[i] = *(descr[i]);
585 + if(unlikely(!cmd)) {
586 + cmd = mallocz(sizeof(struct rrdeng_cmd));
587 + __atomic_add_fetch(&rrdeng_cmd_globals.cache.atomics.allocated, 1, __ATOMIC_RELAXED);
588 }
532 - xt_io_descr->descr_count = count;
533 - xt_io_descr->file = datafile->file;
534 - xt_io_descr->bytes = size_bytes;
535 - xt_io_descr->pos = pos;
536 - xt_io_descr->req_worker.data = xt_io_descr;
537 - xt_io_descr->completion = NULL;
538 - xt_io_descr->release_descr = release_descr;
539 - xt_io_descr->buf = NULL;
540 -
541 - xt_is_cached = !lookup_in_xt_cache(wc, extent, &xt_idx);
542 - if (xt_is_cached) {
543 - xt_cache_replaceQ_set_hot(wc, &wc->xt_cache.extent_array[xt_idx]);
544 - xt_is_inflight = check_bit(wc->xt_cache.inflight_bitmap, xt_idx);
545 - if (xt_is_inflight) {
546 - enqueue_inflight_read_to_xt_cache(wc, xt_idx, xt_io_descr);
547 - return;
548 - }
549 - return read_cached_extent_cb(wc, xt_idx, xt_io_descr);
550 - } else {
551 - ret = try_insert_into_xt_cache(wc, extent);
552 - if (-1 != ret) {
553 - xt_idx = (unsigned)ret;
554 - modify_bit(&wc->xt_cache.inflight_bitmap, xt_idx, 1);
555 - wc->xt_cache.extent_array[xt_idx].inflight_io_descr = xt_io_descr;
589 +
590 + memset(cmd, 0, sizeof(struct rrdeng_cmd));
591 + cmd->ctx = ctx;
592 + cmd->opcode = opcode;
593 + cmd->data = data;
594 + cmd->completion = completion;
595 + cmd->priority = priority;
596 +
597 + netdata_spinlock_lock(&rrdeng_cmd_globals.queue.spinlock);
598 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(rrdeng_cmd_globals.queue.waiting_items_by_priority[priority], cmd, cache.prev, cache.next);
599 + rrdeng_cmd_globals.queue.waiting++;
600 + netdata_spinlock_unlock(&rrdeng_cmd_globals.queue.spinlock);
601 +
602 + fatal_assert(0 == uv_async_send(&rrdeng_main.async));
603 +}
604 +
605 +static inline bool rrdeng_cmd_has_waiting_opcodes_in_lower_priorities(STORAGE_PRIORITY priority, STORAGE_PRIORITY max_priority) {
606 + for(; priority <= max_priority ; priority++)
607 + if(rrdeng_cmd_globals.queue.waiting_items_by_priority[priority])
608 + return true;
609 +
610 + return false;
611 +}
612 +
613 +static inline struct rrdeng_cmd rrdeng_deq_cmd(void) {
614 + struct rrdeng_cmd *cmd = NULL;
615 +
616 + STORAGE_PRIORITY max_priority = work_request_full() ? STORAGE_PRIORITY_CRITICAL : STORAGE_PRIORITY_BEST_EFFORT;
617 +
618 + // find an opcode to execute from the queue
619 + netdata_spinlock_lock(&rrdeng_cmd_globals.queue.spinlock);
620 + for(STORAGE_PRIORITY priority = STORAGE_PRIORITY_CRITICAL; priority <= max_priority ; priority++) {
621 + cmd = rrdeng_cmd_globals.queue.waiting_items_by_priority[priority];
622 + if(cmd) {
623 +
624 + // avoid starvation of lower priorities
625 + if(unlikely(priority > STORAGE_PRIORITY_CRITICAL &&
626 + priority < STORAGE_PRIORITY_BEST_EFFORT &&
627 + ++rrdeng_cmd_globals.queue.executed_by_priority[priority] % 50 == 0 &&
628 + rrdeng_cmd_has_waiting_opcodes_in_lower_priorities(priority + 1, max_priority))) {
629 + // let the others run 2% of the requests
630 + cmd = NULL;
631 + continue;
632 + }
633 +
634 + // remove it from the queue
635 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rrdeng_cmd_globals.queue.waiting_items_by_priority[priority], cmd, cache.prev, cache.next);
636 + rrdeng_cmd_globals.queue.waiting--;
637 + break;
638 }
639 }
640 + netdata_spinlock_unlock(&rrdeng_cmd_globals.queue.spinlock);
641
559 - ret = uv_queue_work(wc->loop, &xt_io_descr->req_worker, do_mmap_read_extent, read_mmap_extent_cb);
560 - fatal_assert(-1 != ret);
642 + struct rrdeng_cmd ret;
643 + if(cmd) {
644 + // copy it, to return it
645 + ret = *cmd;
646
562 - ++ctx->stats.io_read_requests;
563 - ++ctx->stats.io_read_extents;
564 - ctx->stats.pg_cache_backfills += count;
647 + // put it in the cache
648 + netdata_spinlock_lock(&rrdeng_cmd_globals.cache.spinlock);
649 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(rrdeng_cmd_globals.cache.available_items, cmd, cache.prev, cache.next);
650 + rrdeng_cmd_globals.cache.available++;
651 + netdata_spinlock_unlock(&rrdeng_cmd_globals.cache.spinlock);
652 + }
653 + else
654 + ret = (struct rrdeng_cmd) {
655 + .ctx = NULL,
656 + .opcode = RRDENG_OPCODE_NOOP,
657 + .priority = STORAGE_PRIORITY_BEST_EFFORT,
658 + .completion = NULL,
659 + .data = NULL,
660 + };
661 +
662 + return ret;
663 }
664
567 -static void commit_data_extent(struct rrdengine_worker_config* wc, struct extent_io_descriptor *xt_io_descr)
568 -{
569 - struct rrdengine_instance *ctx = wc->ctx;
665 +
666 +// ----------------------------------------------------------------------------
667 +
668 +void *dbengine_page_alloc(struct rrdengine_instance *ctx __maybe_unused, size_t size) {
669 + void *page = mallocz(size);
670 + return page;
671 +}
672 +
673 +void dbengine_page_free(void *page) {
674 + freez(page);
675 +}
676 +
677 +static void commit_data_extent(struct rrdengine_instance *ctx, struct extent_io_descriptor *xt_io_descr) {
678 unsigned count, payload_length, descr_size, size_bytes;
679 void *buf;
680 /* persistent structures */
@@ -582,12 +690,13 @@ static void commit_data_extent(struct rrdengine_worker_config* wc, struct extent
690 payload_length = sizeof(*jf_metric_data) + descr_size;
691 size_bytes = sizeof(*jf_header) + payload_length + sizeof(*jf_trailer);
692
585 - buf = wal_get_transaction_buffer(wc, size_bytes);
693 + xt_io_descr->wal = wal_get(ctx, size_bytes);
694 + buf = xt_io_descr->wal->buf;
695
696 jf_header = buf;
697 jf_header->type = STORE_DATA;
698 jf_header->reserved = 0;
590 - jf_header->id = ctx->commit_log.transaction_id++;
699 + jf_header->id = xt_io_descr->wal->transaction_id;
700 jf_header->payload_length = payload_length;
701
702 jf_metric_data = buf + sizeof(*jf_header);
@@ -602,265 +711,141 @@ static void commit_data_extent(struct rrdengine_worker_config* wc, struct extent
711 crc32set(jf_trailer->checksum, crc);
712 }
713
605 -static void do_commit_transaction(struct rrdengine_worker_config* wc, uint8_t type, void *data)
606 -{
607 - switch (type) {
608 - case STORE_DATA:
609 - commit_data_extent(wc, (struct extent_io_descriptor *)data);
610 - break;
611 - default:
612 - fatal_assert(type == STORE_DATA);
613 - break;
614 - }
615 -}
616 -
617 -static void after_invalidate_oldest_committed(struct rrdengine_worker_config* wc)
618 -{
619 - int error;
714 +static void after_extent_flushed_to_open(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
715 + if(completion)
716 + completion_mark_complete(completion);
717
621 - error = uv_thread_join(wc->now_invalidating_dirty_pages);
622 - if (error) {
623 - error("uv_thread_join(): %s", uv_strerror(error));
624 - }
625 - freez(wc->now_invalidating_dirty_pages);
626 - wc->now_invalidating_dirty_pages = NULL;
627 - wc->cleanup_thread_invalidating_dirty_pages = 0;
718 + if(ctx_is_available_for_queries(ctx))
719 + rrdeng_enq_cmd(ctx, RRDENG_OPCODE_DATABASE_ROTATE, NULL, NULL, STORAGE_PRIORITY_CRITICAL);
720 }
721
630 -static void invalidate_oldest_committed(void *arg)
631 -{
632 - struct rrdengine_instance *ctx = arg;
633 - struct rrdengine_worker_config *wc = &ctx->worker_config;
634 - struct page_cache *pg_cache = &ctx->pg_cache;
635 - int ret;
636 - struct rrdeng_page_descr *descr;
637 - struct page_cache_descr *pg_cache_descr;
638 - Pvoid_t *PValue;
639 - Word_t Index;
640 - unsigned nr_committed_pages;
641 -
642 - do {
643 - uv_rwlock_wrlock(&pg_cache->committed_page_index.lock);
644 - for (Index = 0,
645 - PValue = JudyLFirst(pg_cache->committed_page_index.JudyL_array, &Index, PJE0),
646 - descr = unlikely(NULL == PValue) ? NULL : *PValue;
722 +static void extent_flushed_to_open_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *uv_work_req __maybe_unused) {
723 + worker_is_busy(UV_EVENT_FLUSHED_TO_OPEN);
724
648 - descr != NULL;
649 -
650 - PValue = JudyLNext(pg_cache->committed_page_index.JudyL_array, &Index, PJE0),
651 - descr = unlikely(NULL == PValue) ? NULL : *PValue) {
652 - fatal_assert(0 != descr->page_length);
725 + uv_fs_t *uv_fs_request = data;
726 + struct extent_io_descriptor *xt_io_descr = uv_fs_request->data;
727 + struct page_descr_with_data *descr;
728 + struct rrdengine_datafile *datafile;
729 + unsigned i;
730
654 - rrdeng_page_descr_mutex_lock(ctx, descr);
655 - pg_cache_descr = descr->pg_cache_descr;
656 - if (!(pg_cache_descr->flags & RRD_PAGE_WRITE_PENDING) && pg_cache_try_get_unsafe(descr, 1)) {
657 - rrdeng_page_descr_mutex_unlock(ctx, descr);
731 + if (uv_fs_request->result < 0) {
732 + __atomic_add_fetch(&ctx->stats.io_errors, 1, __ATOMIC_RELAXED);
733 + rrd_stat_atomic_add(&global_io_errors, 1);
734 + error("DBENGINE: %s: uv_fs_write: %s", __func__, uv_strerror((int)uv_fs_request->result));
735 + }
736 + datafile = xt_io_descr->datafile;
737
659 - ret = JudyLDel(&pg_cache->committed_page_index.JudyL_array, Index, PJE0);
660 - fatal_assert(1 == ret);
661 - break;
662 - }
663 - rrdeng_page_descr_mutex_unlock(ctx, descr);
664 - }
665 - uv_rwlock_wrunlock(&pg_cache->committed_page_index.lock);
738 + bool still_running = ctx_is_available_for_queries(ctx);
739
667 - if (!descr) {
668 - info("Failed to invalidate any dirty pages to relieve page cache pressure.");
740 + for (i = 0 ; i < xt_io_descr->descr_count ; ++i) {
741 + descr = xt_io_descr->descr_array[i];
742
670 - goto out;
671 - }
672 - pg_cache_punch_hole(ctx, descr, 1, 1, NULL);
673 -
674 - uv_rwlock_wrlock(&pg_cache->committed_page_index.lock);
675 - nr_committed_pages = --pg_cache->committed_page_index.nr_committed_pages;
676 - uv_rwlock_wrunlock(&pg_cache->committed_page_index.lock);
677 - rrd_stat_atomic_add(&ctx->stats.flushing_pressure_page_deletions, 1);
678 - rrd_stat_atomic_add(&global_flushing_pressure_page_deletions, 1);
679 -
680 - } while (nr_committed_pages >= pg_cache_committed_hard_limit(ctx));
681 -out:
682 - wc->cleanup_thread_invalidating_dirty_pages = 1;
683 - /* wake up event loop */
684 - fatal_assert(0 == uv_async_send(&wc->async));
685 -}
743 + if (likely(still_running))
744 + pgc_open_add_hot_page(
745 + (Word_t)ctx, descr->metric_id,
746 + (time_t) (descr->start_time_ut / USEC_PER_SEC),
747 + (time_t) (descr->end_time_ut / USEC_PER_SEC),
748 + descr->update_every_s,
749 + datafile,
750 + xt_io_descr->pos, xt_io_descr->bytes, descr->page_length);
751
687 -void rrdeng_invalidate_oldest_committed(struct rrdengine_worker_config* wc)
688 -{
689 - struct rrdengine_instance *ctx = wc->ctx;
690 - struct page_cache *pg_cache = &ctx->pg_cache;
691 - unsigned nr_committed_pages;
692 - int error;
752 + page_descriptor_release(descr);
753 + }
754
694 - if (unlikely(ctx->quiesce != NO_QUIESCE)) /* Shutting down */
695 - return;
755 + uv_fs_req_cleanup(uv_fs_request);
756 + posix_memfree(xt_io_descr->buf);
757 + extent_io_descriptor_release(xt_io_descr);
758
697 - uv_rwlock_rdlock(&pg_cache->committed_page_index.lock);
698 - nr_committed_pages = pg_cache->committed_page_index.nr_committed_pages;
699 - uv_rwlock_rdunlock(&pg_cache->committed_page_index.lock);
759 + netdata_spinlock_lock(&datafile->writers.spinlock);
760 + datafile->writers.flushed_to_open_running--;
761 + netdata_spinlock_unlock(&datafile->writers.spinlock);
762
701 - if (nr_committed_pages >= pg_cache_committed_hard_limit(ctx)) {
702 - /* delete the oldest page in memory */
703 - if (wc->now_invalidating_dirty_pages) {
704 - /* already deleting a page */
705 - return;
706 - }
707 - errno = 0;
708 - error("Failed to flush dirty buffers quickly enough in dbengine instance \"%s\". "
709 - "Metric data are being deleted, please reduce disk load or use a faster disk.", ctx->dbfiles_path);
710 -
711 - wc->now_invalidating_dirty_pages = mallocz(sizeof(*wc->now_invalidating_dirty_pages));
712 - wc->cleanup_thread_invalidating_dirty_pages = 0;
713 -
714 - error = uv_thread_create(wc->now_invalidating_dirty_pages, invalidate_oldest_committed, ctx);
715 - if (error) {
716 - error("uv_thread_create(): %s", uv_strerror(error));
717 - freez(wc->now_invalidating_dirty_pages);
718 - wc->now_invalidating_dirty_pages = NULL;
719 - }
720 - }
763 + if(datafile->fileno != __atomic_load_n(&ctx->last_fileno, __ATOMIC_RELAXED) && still_running)
764 + // we just finished a flushing on a datafile that is not the active one
765 + rrdeng_enq_cmd(ctx, RRDENG_OPCODE_JOURNAL_FILE_INDEX, datafile, NULL, STORAGE_PRIORITY_CRITICAL);
766 }
767
723 -void flush_pages_cb(uv_fs_t* req)
724 -{
725 - struct rrdengine_worker_config* wc = req->loop->data;
726 - struct rrdengine_instance *ctx = wc->ctx;
727 - struct page_cache *pg_cache = &ctx->pg_cache;
728 - struct extent_io_descriptor *xt_io_descr;
729 - struct rrdeng_page_descr *descr;
730 - struct page_cache_descr *pg_cache_descr;
731 - unsigned i, count;
768 +// Main event loop callback
769 +static void extent_flush_io_callback(uv_fs_t *uv_fs_request) {
770 + worker_is_busy(RRDENG_OPCODE_MAX + RRDENG_OPCODE_FLUSH_PAGES);
771 + struct extent_io_descriptor *xt_io_descr = uv_fs_request->data;
772 + struct rrdengine_datafile *datafile = xt_io_descr->datafile;
773 + struct rrdengine_instance *ctx = datafile->ctx;
774
733 - xt_io_descr = req->data;
734 - if (req->result < 0) {
735 - ++ctx->stats.io_errors;
736 - rrd_stat_atomic_add(&global_io_errors, 1);
737 - error("%s: uv_fs_write: %s", __func__, uv_strerror((int)req->result));
738 - }
739 -#ifdef NETDATA_INTERNAL_CHECKS
740 - {
741 - struct rrdengine_datafile *datafile = xt_io_descr->descr_array[0]->extent->datafile;
742 - debug(D_RRDENGINE, "%s: Extent at offset %"PRIu64"(%u) was written to datafile %u-%u. Waking up waiters.",
743 - __func__, xt_io_descr->pos, xt_io_descr->bytes, datafile->tier, datafile->fileno);
744 - }
745 -#endif
746 - count = xt_io_descr->descr_count;
747 - for (i = 0 ; i < count ; ++i) {
748 - /* care, we don't hold the descriptor mutex */
749 - descr = xt_io_descr->descr_array[i];
775 + wal_flush_transaction_buffer(ctx, xt_io_descr->datafile, xt_io_descr->wal, &rrdeng_main.loop);
776
751 - pg_cache_replaceQ_insert(ctx, descr);
777 + netdata_spinlock_lock(&datafile->writers.spinlock);
778 + datafile->writers.running--;
779
753 - rrdeng_page_descr_mutex_lock(ctx, descr);
754 - pg_cache_descr = descr->pg_cache_descr;
755 - pg_cache_descr->flags &= ~(RRD_PAGE_DIRTY | RRD_PAGE_WRITE_PENDING);
756 - /* wake up waiters, care no reference being held */
757 - pg_cache_wake_up_waiters_unsafe(descr);
758 - rrdeng_page_descr_mutex_unlock(ctx, descr);
759 - }
760 - if (xt_io_descr->completion)
761 - completion_mark_complete(xt_io_descr->completion);
762 - uv_fs_req_cleanup(req);
763 - posix_memfree(xt_io_descr->buf);
764 - freez(xt_io_descr);
780 + datafile->writers.flushed_to_open_running++;
781 + rrdeng_enq_cmd(xt_io_descr->ctx, RRDENG_OPCODE_FLUSHED_TO_OPEN, uv_fs_request, xt_io_descr->completion, STORAGE_PRIORITY_CRITICAL);
782
766 - uv_rwlock_wrlock(&pg_cache->committed_page_index.lock);
767 - pg_cache->committed_page_index.nr_committed_pages -= count;
768 - uv_rwlock_wrunlock(&pg_cache->committed_page_index.lock);
769 - wc->inflight_dirty_pages -= count;
783 + netdata_spinlock_unlock(&datafile->writers.spinlock);
784 +
785 + worker_is_idle();
786 }
787
788 /*
773 - * completion must be NULL or valid.
774 - * Returns 0 when no flushing can take place.
775 - * Returns datafile bytes to be written on successful flushing initiation.
789 + * Take a page list in a judy array and write them
790 */
777 -static int do_flush_pages(struct rrdengine_worker_config* wc, int force, struct completion *completion)
778 -{
779 - struct rrdengine_instance *ctx = wc->ctx;
780 - struct page_cache *pg_cache = &ctx->pg_cache;
791 +static unsigned do_flush_extent(struct rrdengine_instance *ctx, struct page_descr_with_data *base, struct completion *completion) {
792 int ret;
793 int compressed_size, max_compressed_size = 0;
794 unsigned i, count, size_bytes, pos, real_io_size;
795 uint32_t uncompressed_payload_length, payload_offset;
785 - struct rrdeng_page_descr *descr, *eligible_pages[MAX_PAGES_PER_EXTENT];
786 - struct page_cache_descr *pg_cache_descr;
796 + struct page_descr_with_data *descr, *eligible_pages[MAX_PAGES_PER_EXTENT];
797 struct extent_io_descriptor *xt_io_descr;
798 + struct extent_buffer *eb = NULL;
799 void *compressed_buf = NULL;
789 - Word_t descr_commit_idx_array[MAX_PAGES_PER_EXTENT];
790 - Pvoid_t *PValue;
800 Word_t Index;
801 uint8_t compression_algorithm = ctx->global_compress_alg;
793 - struct extent_info *extent;
802 struct rrdengine_datafile *datafile;
803 /* persistent structures */
804 struct rrdeng_df_extent_header *header;
805 struct rrdeng_df_extent_trailer *trailer;
806 uLong crc;
807
800 - if (force) {
801 - debug(D_RRDENGINE, "Asynchronous flushing of extent has been forced by page pressure.");
802 - }
803 - uv_rwlock_wrlock(&pg_cache->committed_page_index.lock);
804 - for (Index = 0, count = 0, uncompressed_payload_length = 0,
805 - PValue = JudyLFirst(pg_cache->committed_page_index.JudyL_array, &Index, PJE0),
806 - descr = unlikely(NULL == PValue) ? NULL : *PValue ;
807 -
808 - descr != NULL && count != rrdeng_pages_per_extent;
809 -
810 - PValue = JudyLNext(pg_cache->committed_page_index.JudyL_array, &Index, PJE0),
811 - descr = unlikely(NULL == PValue) ? NULL : *PValue) {
812 - uint8_t page_write_pending;
813 -
814 - fatal_assert(0 != descr->page_length);
815 - page_write_pending = 0;
816 -
817 - rrdeng_page_descr_mutex_lock(ctx, descr);
818 - pg_cache_descr = descr->pg_cache_descr;
819 - if (!(pg_cache_descr->flags & RRD_PAGE_WRITE_PENDING)) {
820 - page_write_pending = 1;
821 - /* care, no reference being held */
822 - pg_cache_descr->flags |= RRD_PAGE_WRITE_PENDING;
823 - uncompressed_payload_length += descr->page_length;
824 - descr_commit_idx_array[count] = Index;
825 - eligible_pages[count++] = descr;
826 - }
827 - rrdeng_page_descr_mutex_unlock(ctx, descr);
808 + for(descr = base, Index = 0, count = 0, uncompressed_payload_length = 0;
809 + descr && count != rrdeng_pages_per_extent;
810 + descr = descr->link.next, Index++) {
811 +
812 + uncompressed_payload_length += descr->page_length;
813 + eligible_pages[count++] = descr;
814
829 - if (page_write_pending) {
830 - ret = JudyLDel(&pg_cache->committed_page_index.JudyL_array, Index, PJE0);
831 - fatal_assert(1 == ret);
832 - }
815 }
834 - uv_rwlock_wrunlock(&pg_cache->committed_page_index.lock);
816
817 if (!count) {
837 - debug(D_RRDENGINE, "%s: no pages eligible for flushing.", __func__);
818 if (completion)
819 completion_mark_complete(completion);
820 +
821 + __atomic_sub_fetch(&ctx->worker_config.atomics.extents_currently_being_flushed, 1, __ATOMIC_RELAXED);
822 return 0;
823 }
842 - wc->inflight_dirty_pages += count;
824
844 - xt_io_descr = mallocz(sizeof(*xt_io_descr));
825 + xt_io_descr = extent_io_descriptor_get();
826 + xt_io_descr->ctx = ctx;
827 payload_offset = sizeof(*header) + count * sizeof(header->descr[0]);
828 switch (compression_algorithm) {
847 - case RRD_NO_COMPRESSION:
848 - size_bytes = payload_offset + uncompressed_payload_length + sizeof(*trailer);
849 - break;
850 - default: /* Compress */
851 - fatal_assert(uncompressed_payload_length < LZ4_MAX_INPUT_SIZE);
852 - max_compressed_size = LZ4_compressBound(uncompressed_payload_length);
853 - compressed_buf = mallocz(max_compressed_size);
854 - size_bytes = payload_offset + MAX(uncompressed_payload_length, (unsigned)max_compressed_size) + sizeof(*trailer);
855 - break;
829 + case RRD_NO_COMPRESSION:
830 + size_bytes = payload_offset + uncompressed_payload_length + sizeof(*trailer);
831 + break;
832 +
833 + default: /* Compress */
834 + fatal_assert(uncompressed_payload_length < LZ4_MAX_INPUT_SIZE);
835 + max_compressed_size = LZ4_compressBound(uncompressed_payload_length);
836 + eb = extent_buffer_get(max_compressed_size);
837 + compressed_buf = eb->data;
838 + size_bytes = payload_offset + MAX(uncompressed_payload_length, (unsigned)max_compressed_size) + sizeof(*trailer);
839 + break;
840 }
841 +
842 ret = posix_memalign((void *)&xt_io_descr->buf, RRDFILE_ALIGNMENT, ALIGN_BYTES_CEILING(size_bytes));
843 if (unlikely(ret)) {
859 - fatal("posix_memalign:%s", strerror(ret));
844 + fatal("DBENGINE: posix_memalign:%s", strerror(ret));
845 /* freez(xt_io_descr);*/
846 }
847 memset(xt_io_descr->buf, 0, ALIGN_BYTES_CEILING(size_bytes));
863 - (void) memcpy(xt_io_descr->descr_array, eligible_pages, sizeof(struct rrdeng_page_descr *) * count);
848 + (void) memcpy(xt_io_descr->descr_array, eligible_pages, sizeof(struct page_descr_with_data *) * count);
849 xt_io_descr->descr_count = count;
850
851 pos = 0;
@@ -869,17 +854,7 @@ static int do_flush_pages(struct rrdengine_worker_config* wc, int force, struct
854 header->number_of_pages = count;
855 pos += sizeof(*header);
856
872 - extent = mallocz(sizeof(*extent) + count * sizeof(extent->pages[0]));
873 - datafile = ctx->datafiles.last; /* TODO: check for exceeded size quota */
874 - extent->offset = datafile->pos;
875 - extent->number_of_pages = count;
876 - extent->datafile = datafile;
877 - extent->next = NULL;
878 -
857 for (i = 0 ; i < count ; ++i) {
880 - /* This is here for performance reasons */
881 - xt_io_descr->descr_commit_idx_array[i] = descr_commit_idx_array[i];
882 -
858 descr = xt_io_descr->descr_array[i];
859 header->descr[i].type = descr->type;
860 uuid_copy(*(uuid_t *)header->descr[i].uuid, *descr->id);
@@ -890,35 +865,56 @@ static int do_flush_pages(struct rrdengine_worker_config* wc, int force, struct
865 }
866 for (i = 0 ; i < count ; ++i) {
867 descr = xt_io_descr->descr_array[i];
893 - /* care, we don't hold the descriptor mutex */
894 - (void) memcpy(xt_io_descr->buf + pos, descr->pg_cache_descr->page, descr->page_length);
895 - descr->extent = extent;
896 - extent->pages[i] = descr;
897 -
868 + (void) memcpy(xt_io_descr->buf + pos, descr->page, descr->page_length);
869 pos += descr->page_length;
870 }
900 - df_extent_insert(extent);
871
872 switch (compression_algorithm) {
903 - case RRD_NO_COMPRESSION:
904 - header->payload_length = uncompressed_payload_length;
905 - break;
906 - default: /* Compress */
907 - compressed_size = LZ4_compress_default(xt_io_descr->buf + payload_offset, compressed_buf,
873 + case RRD_NO_COMPRESSION:
874 + header->payload_length = uncompressed_payload_length;
875 + break;
876 + default: /* Compress */
877 + compressed_size = LZ4_compress_default(xt_io_descr->buf + payload_offset, compressed_buf,
878 uncompressed_payload_length, max_compressed_size);
909 - ctx->stats.before_compress_bytes += uncompressed_payload_length;
910 - ctx->stats.after_compress_bytes += compressed_size;
911 - debug(D_RRDENGINE, "LZ4 compressed %"PRIu32" bytes to %d bytes.", uncompressed_payload_length, compressed_size);
912 - (void) memcpy(xt_io_descr->buf + payload_offset, compressed_buf, compressed_size);
913 - freez(compressed_buf);
914 - size_bytes = payload_offset + compressed_size + sizeof(*trailer);
915 - header->payload_length = compressed_size;
879 + ctx->stats.before_compress_bytes += uncompressed_payload_length;
880 + ctx->stats.after_compress_bytes += compressed_size;
881 + debug(D_RRDENGINE, "LZ4 compressed %"PRIu32" bytes to %d bytes.", uncompressed_payload_length, compressed_size);
882 + (void) memcpy(xt_io_descr->buf + payload_offset, compressed_buf, compressed_size);
883 + extent_buffer_release(eb);
884 + size_bytes = payload_offset + compressed_size + sizeof(*trailer);
885 + header->payload_length = compressed_size;
886 break;
887 }
918 - extent->size = size_bytes;
888 +
889 + // get the latest datafile
890 + uv_rwlock_rdlock(&ctx->datafiles.rwlock);
891 + datafile = ctx->datafiles.first->prev;
892 + netdata_spinlock_lock(&datafile->writers.spinlock);
893 + uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
894 +
895 + if(ctx_is_available_for_queries(ctx) && datafile->pos > rrdeng_target_data_file_size(ctx)) {
896 + static SPINLOCK sp = NETDATA_SPINLOCK_INITIALIZER;
897 + netdata_spinlock_lock(&sp);
898 + if(create_new_datafile_pair(ctx) == 0)
899 + rrdeng_enq_cmd(ctx, RRDENG_OPCODE_JOURNAL_FILE_INDEX, datafile, NULL, STORAGE_PRIORITY_CRITICAL);
900 + netdata_spinlock_unlock(&sp);
901 +
902 + // unlock the old datafile
903 + netdata_spinlock_unlock(&datafile->writers.spinlock);
904 +
905 + // get the new datafile
906 + uv_rwlock_rdlock(&ctx->datafiles.rwlock);
907 + datafile = ctx->datafiles.first->prev;
908 + netdata_spinlock_lock(&datafile->writers.spinlock);
909 + uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
910 + }
911 +
912 + datafile->writers.running++;
913 +
914 + xt_io_descr->datafile = datafile;
915 xt_io_descr->bytes = size_bytes;
916 xt_io_descr->pos = datafile->pos;
921 - xt_io_descr->req.data = xt_io_descr;
917 + xt_io_descr->uv_fs_request.data = xt_io_descr;
918 xt_io_descr->completion = completion;
919
920 trailer = xt_io_descr->buf + size_bytes - sizeof(*trailer);
@@ -928,249 +924,306 @@ static int do_flush_pages(struct rrdengine_worker_config* wc, int force, struct
924
925 real_io_size = ALIGN_BYTES_CEILING(size_bytes);
926 xt_io_descr->iov = uv_buf_init((void *)xt_io_descr->buf, real_io_size);
931 - ret = uv_fs_write(wc->loop, &xt_io_descr->req, datafile->file, &xt_io_descr->iov, 1, datafile->pos, flush_pages_cb);
932 - fatal_assert(-1 != ret);
927 +
928 ctx->stats.io_write_bytes += real_io_size;
929 ++ctx->stats.io_write_requests;
930 ctx->stats.io_write_extent_bytes += real_io_size;
931 ++ctx->stats.io_write_extents;
937 - do_commit_transaction(wc, STORE_DATA, xt_io_descr);
938 - datafile->pos += ALIGN_BYTES_CEILING(size_bytes);
939 - ctx->disk_space += ALIGN_BYTES_CEILING(size_bytes);
940 - rrdeng_test_quota(wc);
932 + commit_data_extent(ctx, xt_io_descr);
933 + datafile->pos += real_io_size;
934 + ctx->disk_space += real_io_size;
935 + ctx->last_flush_fileno = datafile->fileno;
936 +
937 + ret = uv_fs_write(&rrdeng_main.loop, &xt_io_descr->uv_fs_request, datafile->file, &xt_io_descr->iov,
938 + 1, xt_io_descr->pos, extent_flush_io_callback);
939 +
940 + fatal_assert(-1 != ret);
941 +
942 + netdata_spinlock_unlock(&datafile->writers.spinlock);
943
942 - return ALIGN_BYTES_CEILING(size_bytes);
944 + return real_io_size;
945 }
946
945 -static void after_delete_old_data(struct rrdengine_worker_config* wc)
947 +static void after_database_rotate(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
948 + ctx->worker_config.now_deleting_files = false;
949 +}
950 +
951 +struct uuid_first_time_s {
952 + uuid_t *uuid;
953 + time_t first_time_s;
954 + time_t last_time_s;
955 + METRIC *metric;
956 +};
957 +
958 +static int journal_metric_uuid_compare(const void *key, const void *metric)
959 {
947 - struct rrdengine_instance *ctx = wc->ctx;
948 - struct rrdengine_datafile *datafile;
949 - struct rrdengine_journalfile *journalfile;
950 - unsigned deleted_bytes, journalfile_bytes, datafile_bytes;
951 - int ret, error;
960 + return uuid_compare(*(uuid_t *) key, ((struct journal_metric_list *) metric)->uuid);
961 +}
962 +
963 +void find_uuid_first_time(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile, Pvoid_t metric_first_time_JudyL)
964 +{
965 + if (unlikely(!datafile))
966 + return;
967 +
968 + unsigned v2_count = 0;
969 + unsigned journalfile_count = 0;
970 + uv_rwlock_rdlock(&ctx->datafiles.rwlock);
971 + while (datafile) {
972 + struct journal_v2_header *journal_header = (struct journal_v2_header *) GET_JOURNAL_DATA(datafile->journalfile);
973 + if (!journal_header || !datafile->users.available) {
974 + datafile = datafile->next;
975 + continue;
976 + }
977 +
978 + time_t journal_start_time_s = (time_t) (journal_header->start_time_ut / USEC_PER_SEC);
979 + size_t journal_metric_count = (size_t)journal_header->metric_count;
980 + struct journal_metric_list *uuid_list = (struct journal_metric_list *)((uint8_t *) journal_header + journal_header->metric_offset);
981 +
982 + Word_t index = 0;
983 + bool first_then_next = true;
984 + Pvoid_t *PValue;
985 + while ((PValue = JudyLFirstThenNext(metric_first_time_JudyL, &index, &first_then_next))) {
986 + struct uuid_first_time_s *uuid_first_t_entry = *PValue;
987 +
988 + struct journal_metric_list *uuid_entry = bsearch(uuid_first_t_entry->uuid,uuid_list,journal_metric_count,sizeof(*uuid_list), journal_metric_uuid_compare);
989 +
990 + if (unlikely(!uuid_entry))
991 + continue;
992 +
993 + time_t first_time_s = uuid_entry->delta_start_s + journal_start_time_s;
994 + time_t last_time_s = uuid_entry->delta_end_s + journal_start_time_s;
995 + uuid_first_t_entry->first_time_s = MIN(uuid_first_t_entry->first_time_s , first_time_s);
996 + uuid_first_t_entry->last_time_s = MAX(uuid_first_t_entry->last_time_s , last_time_s);
997 + v2_count++;
998 + }
999 + journalfile_count++;
1000 + datafile = datafile->next;
1001 + }
1002 + uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
1003 +
1004 + // Let's scan the open cache for almost exact match
1005 + bool first_then_next = true;
1006 + Pvoid_t *PValue;
1007 + Word_t index = 0;
1008 + unsigned open_cache_count = 0;
1009 + while ((PValue = JudyLFirstThenNext(metric_first_time_JudyL, &index, &first_then_next))) {
1010 + struct uuid_first_time_s *uuid_first_t_entry = *PValue;
1011 +
1012 + PGC_PAGE *page = pgc_page_get_and_acquire(
1013 + open_cache, (Word_t)ctx,
1014 + (Word_t)uuid_first_t_entry->metric, uuid_first_t_entry->last_time_s,
1015 + PGC_SEARCH_CLOSEST);
1016 +
1017 + if (page) {
1018 + time_t first_time_s = pgc_page_start_time_s(page);
1019 + time_t last_time_s = pgc_page_end_time_s(page);
1020 + uuid_first_t_entry->first_time_s = MIN(uuid_first_t_entry->first_time_s, first_time_s);
1021 + uuid_first_t_entry->last_time_s = MAX(uuid_first_t_entry->last_time_s, last_time_s);
1022 + pgc_page_release(open_cache, page);
1023 + open_cache_count++;
1024 + }
1025 + }
1026 + info("DBENGINE: processed %u journalfiles and matched %u metric pages in v2 files and %u in open cache", journalfile_count,
1027 + v2_count, open_cache_count);
1028 +}
1029 +
1030 +static void update_metrics_first_time_s(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile_to_delete, struct rrdengine_datafile *first_datafile_remaining, bool worker) {
1031 + if(worker)
1032 + worker_is_busy(UV_EVENT_ANALYZE_V2);
1033 +
1034 + struct rrdengine_journalfile *journal_file = datafile_to_delete->journalfile;
1035 + struct journal_v2_header *journal_header = (struct journal_v2_header *)GET_JOURNAL_DATA(journal_file);
1036 + struct journal_metric_list *uuid_list = (struct journal_metric_list *)((uint8_t *) journal_header + journal_header->metric_offset);
1037 +
1038 + Pvoid_t metric_first_time_JudyL = (Pvoid_t) NULL;
1039 + Pvoid_t *PValue;
1040 +
1041 + unsigned count = 0;
1042 + struct uuid_first_time_s *uuid_first_t_entry;
1043 + for (uint32_t index = 0; index < journal_header->metric_count; ++index) {
1044 + METRIC *metric = mrg_metric_get_and_acquire(main_mrg, &uuid_list[index].uuid, (Word_t) ctx);
1045 + if (!metric)
1046 + continue;
1047 +
1048 + PValue = JudyLIns(&metric_first_time_JudyL, (Word_t) index, PJE0);
1049 + fatal_assert(NULL != PValue);
1050 + if (!*PValue) {
1051 + uuid_first_t_entry = mallocz(sizeof(*uuid_first_t_entry));
1052 + uuid_first_t_entry->metric = metric;
1053 + uuid_first_t_entry->first_time_s = mrg_metric_get_first_time_s(main_mrg, metric);
1054 + uuid_first_t_entry->last_time_s = mrg_metric_get_latest_time_s(main_mrg, metric);
1055 + uuid_first_t_entry->uuid = mrg_metric_uuid(main_mrg, metric);
1056 + *PValue = uuid_first_t_entry;
1057 + count++;
1058 + }
1059 + }
1060 +
1061 + info("DBENGINE: recalculating retention for %u metrics", count);
1062 +
1063 + // Update the first time / last time for all metrics we plan to delete
1064 +
1065 + if(worker)
1066 + worker_is_busy(UV_EVENT_RETENTION_V2);
1067 +
1068 + find_uuid_first_time(ctx, first_datafile_remaining, metric_first_time_JudyL);
1069 +
1070 + if(worker)
1071 + worker_is_busy(UV_EVENT_RETENTION_UPDATE);
1072 +
1073 + info("DBENGINE: updating metric registry retention for %u metrics", count);
1074 +
1075 + Word_t index = 0;
1076 + bool first_then_next = true;
1077 + while ((PValue = JudyLFirstThenNext(metric_first_time_JudyL, &index, &first_then_next))) {
1078 + uuid_first_t_entry = *PValue;
1079 + mrg_metric_set_first_time_s(main_mrg, uuid_first_t_entry->metric, uuid_first_t_entry->first_time_s);
1080 + mrg_metric_release(main_mrg, uuid_first_t_entry->metric);
1081 + freez(uuid_first_t_entry);
1082 + }
1083 +
1084 + JudyLFreeArray(&metric_first_time_JudyL, PJE0);
1085 +
1086 + if(worker)
1087 + worker_is_idle();
1088 +}
1089 +
1090 +static void datafile_delete(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile, bool worker) {
1091 + if(worker)
1092 + worker_is_busy(UV_EVENT_DATAFILE_ACQUIRE);
1093 +
1094 + bool datafile_got_for_deletion = datafile_acquire_for_deletion(datafile);
1095 +
1096 + if (ctx_is_available_for_queries(ctx))
1097 + update_metrics_first_time_s(ctx, datafile, datafile->next, worker);
1098 +
1099 + while (!datafile_got_for_deletion) {
1100 + if(worker)
1101 + worker_is_busy(UV_EVENT_DATAFILE_ACQUIRE);
1102 +
1103 + datafile_got_for_deletion = datafile_acquire_for_deletion(datafile);
1104 +
1105 + if (!datafile_got_for_deletion) {
1106 + info("DBENGINE: waiting for data file '%s/"
1107 + DATAFILE_PREFIX RRDENG_FILE_NUMBER_PRINT_TMPL DATAFILE_EXTENSION
1108 + "' to be available for deletion, "
1109 + "it is in use currently by %u users.",
1110 + ctx->dbfiles_path, ctx->datafiles.first->tier, ctx->datafiles.first->fileno, datafile->users.lockers);
1111 +
1112 + sleep_usec(1 * USEC_PER_SEC);
1113 + }
1114 + }
1115 +
1116 + info("DBENGINE: deleting data file '%s/"
1117 + DATAFILE_PREFIX RRDENG_FILE_NUMBER_PRINT_TMPL DATAFILE_EXTENSION
1118 + "'.",
1119 + ctx->dbfiles_path, ctx->datafiles.first->tier, ctx->datafiles.first->fileno);
1120 +
1121 + if(worker)
1122 + worker_is_busy(UV_EVENT_DATAFILE_DELETE);
1123 +
1124 + struct rrdengine_journalfile *journal_file;
1125 + unsigned deleted_bytes, journal_file_bytes, datafile_bytes;
1126 + int ret;
1127 char path[RRDENG_PATH_MAX];
1128
954 - datafile = ctx->datafiles.first;
955 - journalfile = datafile->journalfile;
1129 + uv_rwlock_wrlock(&ctx->datafiles.rwlock);
1130 +
1131 + journal_file = datafile->journalfile;
1132 datafile_bytes = datafile->pos;
957 - journalfile_bytes = journalfile->pos;
958 - deleted_bytes = 0;
1133 + journal_file_bytes = journal_file->pos;
1134 + deleted_bytes = GET_JOURNAL_DATA_SIZE(journal_file);
1135
960 - info("Deleting data and journal file pair.");
961 - datafile_list_delete(ctx, datafile);
962 - ret = destroy_journal_file(journalfile, datafile);
1136 + info("DBENGINE: deleting data and journal files to maintain disk quota");
1137 + datafile_list_delete_unsafe(ctx, datafile);
1138 + ret = destroy_journal_file_unsafe(journal_file, datafile);
1139 if (!ret) {
1140 generate_journalfilepath(datafile, path, sizeof(path));
965 - info("Deleted journal file \"%s\".", path);
966 - deleted_bytes += journalfile_bytes;
1141 + info("DBENGINE: deleted journal file \"%s\".", path);
1142 + generate_journalfilepath_v2(datafile, path, sizeof(path));
1143 + info("DBENGINE: deleted journal file \"%s\".", path);
1144 + deleted_bytes += journal_file_bytes;
1145 }
968 - ret = destroy_data_file(datafile);
1146 + ret = destroy_data_file_unsafe(datafile);
1147 if (!ret) {
1148 generate_datafilepath(datafile, path, sizeof(path));
971 - info("Deleted data file \"%s\".", path);
1149 + info("DBENGINE: deleted data file \"%s\".", path);
1150 deleted_bytes += datafile_bytes;
1151 }
974 - freez(journalfile);
1152 + freez(journal_file);
1153 freez(datafile);
1154
1155 ctx->disk_space -= deleted_bytes;
978 - info("Reclaimed %u bytes of disk space.", deleted_bytes);
979 -
980 - error = uv_thread_join(wc->now_deleting_files);
981 - if (error) {
982 - error("uv_thread_join(): %s", uv_strerror(error));
983 - }
984 - freez(wc->now_deleting_files);
985 - /* unfreeze command processing */
986 - wc->now_deleting_files = NULL;
1156 + info("DBENGINE: reclaimed %u bytes of disk space.", deleted_bytes);
1157 + uv_rwlock_wrunlock(&ctx->datafiles.rwlock);
1158
988 - wc->cleanup_thread_deleting_files = 0;
1159 rrdcontext_db_rotation();
1160 +}
1161
991 - /* interrupt event loop */
992 - uv_stop(wc->loop);
1162 +static void database_rotate_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *uv_work_req __maybe_unused) {
1163 + datafile_delete(ctx, ctx->datafiles.first, true);
1164 }
1165
995 -static void delete_old_data(void *arg)
996 -{
997 - struct rrdengine_instance *ctx = arg;
998 - struct rrdengine_worker_config* wc = &ctx->worker_config;
999 - struct rrdengine_datafile *datafile;
1000 - struct extent_info *extent, *next;
1001 - struct rrdeng_page_descr *descr;
1002 - unsigned count, i;
1003 - uint8_t can_delete_metric;
1004 - uuid_t metric_id;
1005 -
1006 - /* Safe to use since it will be deleted after we are done */
1007 - datafile = ctx->datafiles.first;
1008 -
1009 - for (extent = datafile->extents.first ; extent != NULL ; extent = next) {
1010 - count = extent->number_of_pages;
1011 - for (i = 0 ; i < count ; ++i) {
1012 - descr = extent->pages[i];
1013 - can_delete_metric = pg_cache_punch_hole(ctx, descr, 0, 0, &metric_id);
1014 - if (unlikely(can_delete_metric)) {
1015 - /*
1016 - * If the metric is empty, has no active writers and if the metadata log has been initialized then
1017 - * attempt to delete the corresponding netdata dimension.
1018 - */
1019 - metaqueue_delete_dimension_uuid(&metric_id);
1020 - }
1021 - }
1022 - next = extent->next;
1023 - freez(extent);
1024 - }
1025 - wc->cleanup_thread_deleting_files = 1;
1026 - /* wake up event loop */
1027 - fatal_assert(0 == uv_async_send(&wc->async));
1166 +static void after_flush_all_hot_and_dirty_pages_of_section(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1167 + ;
1168 }
1169
1030 -void rrdeng_test_quota(struct rrdengine_worker_config* wc)
1031 -{
1032 - struct rrdengine_instance *ctx = wc->ctx;
1033 - struct rrdengine_datafile *datafile;
1034 - unsigned current_size, target_size;
1035 - uint8_t out_of_space, only_one_datafile;
1036 - int ret, error;
1037 -
1038 - out_of_space = 0;
1039 - /* Do not allow the pinned pages to exceed the disk space quota to avoid deadlocks */
1040 - if (unlikely(ctx->disk_space > MAX(ctx->max_disk_space, 2 * ctx->metric_API_max_producers * RRDENG_BLOCK_SIZE))) {
1041 - out_of_space = 1;
1042 - }
1043 - datafile = ctx->datafiles.last;
1044 - current_size = datafile->pos;
1045 - target_size = ctx->max_disk_space / TARGET_DATAFILES;
1046 - target_size = MIN(target_size, MAX_DATAFILE_SIZE);
1047 - target_size = MAX(target_size, MIN_DATAFILE_SIZE);
1048 - only_one_datafile = (datafile == ctx->datafiles.first) ? 1 : 0;
1049 - if (unlikely(current_size >= target_size || (out_of_space && only_one_datafile))) {
1050 - /* Finalize data and journal file and create a new pair */
1051 - wal_flush_transaction_buffer(wc);
1052 - ret = create_new_datafile_pair(ctx, 1, ctx->last_fileno + 1);
1053 - if (likely(!ret)) {
1054 - ++ctx->last_fileno;
1055 - }
1056 - }
1057 - if (unlikely(out_of_space && NO_QUIESCE == ctx->quiesce)) {
1058 - /* delete old data */
1059 - if (wc->now_deleting_files) {
1060 - /* already deleting data */
1061 - return;
1062 - }
1063 - if (NULL == ctx->datafiles.first->next) {
1064 - error("Cannot delete data file \"%s/"DATAFILE_PREFIX RRDENG_FILE_NUMBER_PRINT_TMPL DATAFILE_EXTENSION"\""
1065 - " to reclaim space, there are no other file pairs left.",
1066 - ctx->dbfiles_path, ctx->datafiles.first->tier, ctx->datafiles.first->fileno);
1067 - return;
1068 - }
1069 - info("Deleting data file \"%s/"DATAFILE_PREFIX RRDENG_FILE_NUMBER_PRINT_TMPL DATAFILE_EXTENSION"\".",
1070 - ctx->dbfiles_path, ctx->datafiles.first->tier, ctx->datafiles.first->fileno);
1071 - wc->now_deleting_files = mallocz(sizeof(*wc->now_deleting_files));
1072 - wc->cleanup_thread_deleting_files = 0;
1073 -
1074 - error = uv_thread_create(wc->now_deleting_files, delete_old_data, ctx);
1075 - if (error) {
1076 - error("uv_thread_create(): %s", uv_strerror(error));
1077 - freez(wc->now_deleting_files);
1078 - wc->now_deleting_files = NULL;
1079 - }
1080 - }
1170 +static void flush_all_hot_and_dirty_pages_of_section_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *uv_work_req __maybe_unused) {
1171 + pgc_flush_all_hot_and_dirty_pages(main_cache, (Word_t)ctx);
1172 + completion_mark_complete(&ctx->quiesce_completion);
1173 }
1174
1083 -static inline int rrdeng_threads_alive(struct rrdengine_worker_config* wc)
1084 -{
1085 - if (wc->now_invalidating_dirty_pages || wc->now_deleting_files) {
1086 - return 1;
1087 - }
1088 - return 0;
1175 +static void after_ctx_shutdown(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1176 + ;
1177 }
1178
1091 -static void rrdeng_cleanup_finished_threads(struct rrdengine_worker_config* wc)
1092 -{
1093 - struct rrdengine_instance *ctx = wc->ctx;
1179 +static void ctx_shutdown_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *uv_work_req __maybe_unused) {
1180 + completion_wait_for(&ctx->quiesce_completion);
1181 + completion_destroy(&ctx->quiesce_completion);
1182
1095 - if (unlikely(wc->cleanup_thread_invalidating_dirty_pages)) {
1096 - after_invalidate_oldest_committed(wc);
1097 - }
1098 - if (unlikely(wc->cleanup_thread_deleting_files)) {
1099 - after_delete_old_data(wc);
1100 - }
1101 - if (unlikely(SET_QUIESCE == ctx->quiesce && !rrdeng_threads_alive(wc))) {
1102 - ctx->quiesce = QUIESCED;
1103 - completion_mark_complete(&ctx->rrdengine_completion);
1104 - }
1183 + while(__atomic_load_n(&ctx->worker_config.atomics.extents_currently_being_flushed, __ATOMIC_RELAXED) ||
1184 + __atomic_load_n(&ctx->inflight_queries, __ATOMIC_RELAXED))
1185 + sleep_usec(1 * USEC_PER_MS);
1186 +
1187 + completion_mark_complete(completion);
1188 }
1189
1107 -/* return 0 on success */
1108 -int init_rrd_files(struct rrdengine_instance *ctx)
1109 -{
1110 - int ret = init_data_files(ctx);
1111 -
1112 - BUFFER *wb = buffer_create(1000);
1113 - size_t all_errors = 0;
1114 - usec_t now = now_realtime_usec();
1115 -
1116 - if(ctx->load_errors[LOAD_ERRORS_PAGE_FLIPPED_TIME].counter) {
1117 - buffer_sprintf(wb, "%s%zu pages had start time > end time (latest: %llu secs ago)"
1118 - , (all_errors)?", ":""
1119 - , ctx->load_errors[LOAD_ERRORS_PAGE_FLIPPED_TIME].counter
1120 - , (now - ctx->load_errors[LOAD_ERRORS_PAGE_FLIPPED_TIME].latest_end_time_ut) / USEC_PER_SEC
1121 - );
1122 - all_errors += ctx->load_errors[LOAD_ERRORS_PAGE_FLIPPED_TIME].counter;
1123 - }
1190 +static void cache_flush_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *uv_work_req __maybe_unused) {
1191 + if (!main_cache)
1192 + return;
1193
1125 - if(ctx->load_errors[LOAD_ERRORS_PAGE_EQUAL_TIME].counter) {
1126 - buffer_sprintf(wb, "%s%zu pages had start time = end time with more than 1 entries (latest: %llu secs ago)"
1127 - , (all_errors)?", ":""
1128 - , ctx->load_errors[LOAD_ERRORS_PAGE_EQUAL_TIME].counter
1129 - , (now - ctx->load_errors[LOAD_ERRORS_PAGE_EQUAL_TIME].latest_end_time_ut) / USEC_PER_SEC
1130 - );
1131 - all_errors += ctx->load_errors[LOAD_ERRORS_PAGE_EQUAL_TIME].counter;
1132 - }
1194 + worker_is_busy(UV_EVENT_FLUSH_MAIN);
1195 + pgc_flush_pages(main_cache, 0);
1196 +}
1197
1134 - if(ctx->load_errors[LOAD_ERRORS_PAGE_ZERO_ENTRIES].counter) {
1135 - buffer_sprintf(wb, "%s%zu pages had zero points (latest: %llu secs ago)"
1136 - , (all_errors)?", ":""
1137 - , ctx->load_errors[LOAD_ERRORS_PAGE_ZERO_ENTRIES].counter
1138 - , (now - ctx->load_errors[LOAD_ERRORS_PAGE_ZERO_ENTRIES].latest_end_time_ut) / USEC_PER_SEC
1139 - );
1140 - all_errors += ctx->load_errors[LOAD_ERRORS_PAGE_ZERO_ENTRIES].counter;
1141 - }
1198 +static void cache_evict_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *req __maybe_unused) {
1199 + if (!main_cache)
1200 + return;
1201
1143 - if(ctx->load_errors[LOAD_ERRORS_PAGE_UPDATE_ZERO].counter) {
1144 - buffer_sprintf(wb, "%s%zu pages had update every == 0 with entries > 1 (latest: %llu secs ago)"
1145 - , (all_errors)?", ":""
1146 - , ctx->load_errors[LOAD_ERRORS_PAGE_UPDATE_ZERO].counter
1147 - , (now - ctx->load_errors[LOAD_ERRORS_PAGE_UPDATE_ZERO].latest_end_time_ut) / USEC_PER_SEC
1148 - );
1149 - all_errors += ctx->load_errors[LOAD_ERRORS_PAGE_UPDATE_ZERO].counter;
1150 - }
1202 + worker_is_busy(UV_EVENT_EVICT_MAIN);
1203 + pgc_evict_pages(main_cache, 0, 0);
1204 +}
1205
1152 - if(ctx->load_errors[LOAD_ERRORS_PAGE_FLEXY_TIME].counter) {
1153 - buffer_sprintf(wb, "%s%zu pages had a different number of points compared to their timestamps (latest: %llu secs ago; these page have been loaded)"
1154 - , (all_errors)?", ":""
1155 - , ctx->load_errors[LOAD_ERRORS_PAGE_FLEXY_TIME].counter
1156 - , (now - ctx->load_errors[LOAD_ERRORS_PAGE_FLEXY_TIME].latest_end_time_ut) / USEC_PER_SEC
1157 - );
1158 - all_errors += ctx->load_errors[LOAD_ERRORS_PAGE_FLEXY_TIME].counter;
1159 - }
1206 +static void after_prep_query(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1207 + ;
1208 +}
1209
1161 - if(ctx->load_errors[LOAD_ERRORS_DROPPED_EXTENT].counter) {
1162 - buffer_sprintf(wb, "%s%zu extents have been dropped because they didn't have any valid pages"
1163 - , (all_errors)?", ":""
1164 - , ctx->load_errors[LOAD_ERRORS_DROPPED_EXTENT].counter
1165 - );
1166 - all_errors += ctx->load_errors[LOAD_ERRORS_DROPPED_EXTENT].counter;
1167 - }
1210 +static void query_prep_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *req __maybe_unused) {
1211 + worker_is_busy(UV_EVENT_PREP_QUERY);
1212 + PDC *pdc = data;
1213 + rrdeng_prep_query(pdc);
1214 +}
1215
1169 - if(all_errors)
1170 - info("DBENGINE: tier %d: %s", ctx->tier, buffer_tostring(wb));
1216 +unsigned rrdeng_target_data_file_size(struct rrdengine_instance *ctx) {
1217 + unsigned target_size = ctx->max_disk_space / TARGET_DATAFILES;
1218 + target_size = MIN(target_size, MAX_DATAFILE_SIZE);
1219 + target_size = MAX(target_size, MIN_DATAFILE_SIZE);
1220 + return target_size;
1221 +}
1222
1172 - buffer_free(wb);
1173 - return ret;
1223 +/* return 0 on success */
1224 +int init_rrd_files(struct rrdengine_instance *ctx)
1225 +{
1226 + return init_data_files(ctx);
1227 }
1228
1229 void finalize_rrd_files(struct rrdengine_instance *ctx)
@@ -1178,331 +1231,397 @@ void finalize_rrd_files(struct rrdengine_instance *ctx)
1231 return finalize_data_files(ctx);
1232 }
1233
1181 -void rrdeng_init_cmd_queue(struct rrdengine_worker_config* wc)
1234 +void async_cb(uv_async_t *handle)
1235 {
1183 - wc->cmd_queue.head = wc->cmd_queue.tail = 0;
1184 - wc->queue_size = 0;
1185 - fatal_assert(0 == uv_cond_init(&wc->cmd_cond));
1186 - fatal_assert(0 == uv_mutex_init(&wc->cmd_mutex));
1236 + uv_stop(handle->loop);
1237 + uv_update_time(handle->loop);
1238 + debug(D_RRDENGINE, "%s called, active=%d.", __func__, uv_is_active((uv_handle_t *)handle));
1239 }
1240
1189 -void rrdeng_enq_cmd(struct rrdengine_worker_config* wc, struct rrdeng_cmd *cmd)
1190 -{
1191 - unsigned queue_size;
1241 +#define TIMER_PERIOD_MS (1000)
1242
1193 - /* wait for free space in queue */
1194 - uv_mutex_lock(&wc->cmd_mutex);
1195 - while ((queue_size = wc->queue_size) == RRDENG_CMD_Q_MAX_SIZE) {
1196 - uv_cond_wait(&wc->cmd_cond, &wc->cmd_mutex);
1197 - }
1198 - fatal_assert(queue_size < RRDENG_CMD_Q_MAX_SIZE);
1199 - /* enqueue command */
1200 - wc->cmd_queue.cmd_array[wc->cmd_queue.tail] = *cmd;
1201 - wc->cmd_queue.tail = wc->cmd_queue.tail != RRDENG_CMD_Q_MAX_SIZE - 1 ?
1202 - wc->cmd_queue.tail + 1 : 0;
1203 - wc->queue_size = queue_size + 1;
1204 - uv_mutex_unlock(&wc->cmd_mutex);
1205 -
1206 - /* wake up event loop */
1207 - fatal_assert(0 == uv_async_send(&wc->async));
1243 +
1244 +static void extent_read_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *uv_work_req __maybe_unused) {
1245 + EPDL *epdl = data;
1246 + epdl_find_extent_and_populate_pages(ctx, epdl, true);
1247 }
1248
1210 -struct rrdeng_cmd rrdeng_deq_cmd(struct rrdengine_worker_config* wc)
1211 -{
1212 - struct rrdeng_cmd ret;
1213 - unsigned queue_size;
1214 -
1215 - uv_mutex_lock(&wc->cmd_mutex);
1216 - queue_size = wc->queue_size;
1217 - if (queue_size == 0) {
1218 - ret.opcode = RRDENG_NOOP;
1219 - } else {
1220 - /* dequeue command */
1221 - ret = wc->cmd_queue.cmd_array[wc->cmd_queue.head];
1222 - if (queue_size == 1) {
1223 - wc->cmd_queue.head = wc->cmd_queue.tail = 0;
1224 - } else {
1225 - wc->cmd_queue.head = wc->cmd_queue.head != RRDENG_CMD_Q_MAX_SIZE - 1 ?
1226 - wc->cmd_queue.head + 1 : 0;
1249 +static void epdl_populate_pages_asynchronously(struct rrdengine_instance *ctx, EPDL *epdl, STORAGE_PRIORITY priority) {
1250 + rrdeng_enq_cmd(ctx, RRDENG_OPCODE_EXTENT_READ, epdl, NULL, priority);
1251 +}
1252 +
1253 +void pdc_route_asynchronously(struct rrdengine_instance *ctx, struct page_details_control *pdc) {
1254 + pdc_to_epdl_router(ctx, pdc, epdl_populate_pages_asynchronously, epdl_populate_pages_asynchronously);
1255 +}
1256 +
1257 +void epdl_populate_pages_synchronously(struct rrdengine_instance *ctx, EPDL *epdl, enum storage_priority priority __maybe_unused) {
1258 + epdl_find_extent_and_populate_pages(ctx, epdl, false);
1259 +}
1260 +
1261 +void pdc_route_synchronously(struct rrdengine_instance *ctx, struct page_details_control *pdc) {
1262 + pdc_to_epdl_router(ctx, pdc, epdl_populate_pages_synchronously, epdl_populate_pages_synchronously);
1263 +}
1264 +
1265 +#define MAX_RETRIES_TO_START_INDEX (100)
1266 +static void journal_v2_indexing_tp_worker(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t *uv_work_req __maybe_unused) {
1267 + unsigned count = 0;
1268 + worker_is_busy(UV_EVENT_JOURNAL_INDEX_WAIT);
1269 +
1270 + while (ctx->worker_config.now_deleting_files && count++ < MAX_RETRIES_TO_START_INDEX)
1271 + sleep_usec(100 * USEC_PER_MS);
1272 +
1273 + if (count == MAX_RETRIES_TO_START_INDEX) {
1274 + worker_is_idle();
1275 + return;
1276 + }
1277 +
1278 + struct rrdengine_datafile *datafile = ctx->datafiles.first;
1279 + worker_is_busy(UV_EVENT_JOURNAL_INDEX);
1280 + count = 0;
1281 + while (datafile && datafile->fileno != ctx->last_fileno && datafile->fileno != ctx->last_flush_fileno) {
1282 +
1283 + netdata_spinlock_lock(&datafile->writers.spinlock);
1284 + bool available = (datafile->writers.running || datafile->writers.flushed_to_open_running) ? false : true;
1285 + netdata_spinlock_unlock(&datafile->writers.spinlock);
1286 +
1287 + if(!available)
1288 + continue;
1289 +
1290 + if (unlikely(!GET_JOURNAL_DATA(datafile->journalfile))) {
1291 + info("DBENGINE: journal file %u is ready to be indexed", datafile->fileno);
1292 + pgc_open_cache_to_journal_v2(open_cache, (Word_t) ctx, (int) datafile->fileno, ctx->page_type, do_migrate_to_v2_callback, (void *) datafile->journalfile);
1293 + count++;
1294 }
1228 - wc->queue_size = queue_size - 1;
1295
1230 - /* wake up producers */
1231 - uv_cond_signal(&wc->cmd_cond);
1296 + datafile = datafile->next;
1297 +
1298 + if (unlikely(!ctx_is_available_for_queries(ctx)))
1299 + break;
1300 }
1233 - uv_mutex_unlock(&wc->cmd_mutex);
1301
1235 - return ret;
1302 + errno = 0;
1303 + internal_error(count, "DBENGINE: journal indexing done; %u files processed", count);
1304 +
1305 + worker_is_idle();
1306 }
1307
1238 -static void load_configuration_dynamic(void)
1239 -{
1240 - unsigned read_num = (unsigned)config_get_number(CONFIG_SECTION_DB, "dbengine pages per extent", MAX_PAGES_PER_EXTENT);
1241 - if (read_num > 0 && read_num <= MAX_PAGES_PER_EXTENT)
1242 - rrdeng_pages_per_extent = read_num;
1243 - else {
1244 - error("Invalid dbengine pages per extent %u given. Using %u.", read_num, rrdeng_pages_per_extent);
1245 - config_set_number(CONFIG_SECTION_DB, "dbengine pages per extent", rrdeng_pages_per_extent);
1246 - }
1308 +static void after_do_cache_flush(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1309 + rrdeng_main.flush_running = false;
1310 }
1311
1249 -void async_cb(uv_async_t *handle)
1250 -{
1251 - uv_stop(handle->loop);
1252 - uv_update_time(handle->loop);
1253 - debug(D_RRDENGINE, "%s called, active=%d.", __func__, uv_is_active((uv_handle_t *)handle));
1312 +static void after_do_cache_evict(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1313 + rrdeng_main.evict_running = false;
1314 }
1315
1256 -/* Flushes dirty pages when timer expires */
1257 -#define TIMER_PERIOD_MS (1000)
1316 +static void after_extent_read(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1317 + ;
1318 +}
1319
1259 -void timer_cb(uv_timer_t* handle)
1260 -{
1261 - worker_is_busy(RRDENG_MAX_OPCODE + 1);
1320 +static void after_journal_v2_indexing(struct rrdengine_instance *ctx __maybe_unused, void *data __maybe_unused, struct completion *completion __maybe_unused, uv_work_t* req __maybe_unused, int status __maybe_unused) {
1321 + ctx->worker_config.migration_to_v2_running = false;
1322 + rrdeng_enq_cmd(ctx, RRDENG_OPCODE_DATABASE_ROTATE, NULL, NULL, STORAGE_PRIORITY_CRITICAL);
1323 +}
1324
1263 - struct rrdengine_worker_config* wc = handle->data;
1264 - struct rrdengine_instance *ctx = wc->ctx;
1325 +struct rrdeng_buffer_sizes rrdeng_get_buffer_sizes(void) {
1326 + return (struct rrdeng_buffer_sizes) {
1327 + .opcodes = __atomic_load_n(&rrdeng_cmd_globals.cache.atomics.allocated, __ATOMIC_RELAXED) * sizeof(struct rrdeng_cmd),
1328 + .handles = __atomic_load_n(&rrdeng_query_handle_globals.atomics.allocated, __ATOMIC_RELAXED) * sizeof(struct rrdeng_query_handle),
1329 + .descriptors = __atomic_load_n(&page_descriptor_globals.atomics.allocated, __ATOMIC_RELAXED) * sizeof(struct page_descr_with_data),
1330 + .wal = __atomic_load_n(&wal_globals.atomics.allocated, __ATOMIC_RELAXED) * (sizeof(WAL) + RRDENG_BLOCK_SIZE),
1331 + .workers = __atomic_load_n(&work_request_globals.atomics.allocated, __ATOMIC_RELAXED) * sizeof(struct rrdeng_work),
1332 + .pdc = pdc_cache_size(),
1333 + .xt_io = __atomic_load_n(&extent_io_descriptor_globals.atomics.allocated, __ATOMIC_RELAXED) * sizeof(struct extent_io_descriptor),
1334 + .xt_buf = extent_buffer_cache_size(),
1335 + .epdl = epdl_cache_size(),
1336 + .deol = deol_cache_size(),
1337 + .pd = pd_cache_size(),
1338 +#ifdef PDC_USE_JULYL
1339 + .julyl = julyl_cache_size(),
1340 +#endif
1341 + };
1342 +}
1343
1344 +void timer_cb(uv_timer_t* handle) {
1345 + worker_is_busy(RRDENG_TIMER_CB);
1346 uv_stop(handle->loop);
1347 uv_update_time(handle->loop);
1268 - rrdeng_test_quota(wc);
1269 - debug(D_RRDENGINE, "%s: timeout reached.", __func__);
1270 - if (likely(!wc->now_deleting_files && !wc->now_invalidating_dirty_pages)) {
1271 - /* There is free space so we can write to disk and we are not actively deleting dirty buffers */
1272 - struct page_cache *pg_cache = &ctx->pg_cache;
1273 - unsigned long total_bytes, bytes_written, nr_committed_pages, bytes_to_write = 0, producers, low_watermark,
1274 - high_watermark;
1275 -
1276 - uv_rwlock_rdlock(&pg_cache->committed_page_index.lock);
1277 - nr_committed_pages = pg_cache->committed_page_index.nr_committed_pages;
1278 - uv_rwlock_rdunlock(&pg_cache->committed_page_index.lock);
1279 - producers = ctx->metric_API_max_producers;
1280 - /* are flushable pages more than 25% of the maximum page cache size */
1281 - high_watermark = (ctx->max_cache_pages * 25LLU) / 100;
1282 - low_watermark = (ctx->max_cache_pages * 5LLU) / 100; /* 5%, must be smaller than high_watermark */
1283 -
1284 - /* Flush more pages only if disk can keep up */
1285 - if (wc->inflight_dirty_pages < high_watermark + producers) {
1286 - if (nr_committed_pages > producers &&
1287 - /* committed to be written pages are more than the produced number */
1288 - nr_committed_pages - producers > high_watermark) {
1289 - /* Flushing speed must increase to stop page cache from filling with dirty pages */
1290 - bytes_to_write = (nr_committed_pages - producers - low_watermark) * RRDENG_BLOCK_SIZE;
1291 - }
1292 - bytes_to_write = MAX(DATAFILE_IDEAL_IO_SIZE, bytes_to_write);
1348
1294 - debug(D_RRDENGINE, "Flushing pages to disk.");
1295 - for (total_bytes = bytes_written = do_flush_pages(wc, 0, NULL);
1296 - bytes_written && (total_bytes < bytes_to_write);
1297 - total_bytes += bytes_written) {
1298 - bytes_written = do_flush_pages(wc, 0, NULL);
1299 - }
1300 - }
1301 - }
1302 - load_configuration_dynamic();
1303 -#ifdef NETDATA_INTERNAL_CHECKS
1304 - {
1305 - char buf[4096];
1306 - debug(D_RRDENGINE, "%s", get_rrdeng_statistics(wc->ctx, buf, sizeof(buf)));
1307 - }
1349 + worker_set_metric(RRDENG_OPCODES_WAITING, (NETDATA_DOUBLE)rrdeng_cmd_globals.queue.waiting);
1350 + worker_set_metric(RRDENG_WORKS_DISPATCHED, (NETDATA_DOUBLE)__atomic_load_n(&work_request_globals.atomics.dispatched, __ATOMIC_RELAXED));
1351 + worker_set_metric(RRDENG_WORKS_EXECUTING, (NETDATA_DOUBLE)__atomic_load_n(&work_request_globals.atomics.executing, __ATOMIC_RELAXED));
1352 +
1353 + rrdeng_enq_cmd(NULL, RRDENG_OPCODE_FLUSH_INIT, NULL, NULL, STORAGE_PRIORITY_CRITICAL);
1354 + rrdeng_enq_cmd(NULL, RRDENG_OPCODE_EVICT_INIT, NULL, NULL, STORAGE_PRIORITY_CRITICAL);
1355 +
1356 + time_t now_s = now_monotonic_sec();
1357 + if(now_s - rrdeng_main.last_buffers_cleanup_s > 600) {
1358 + rrdeng_main.last_buffers_cleanup_s = now_s;
1359 +
1360 + work_request_cleanup();
1361 + page_descriptor_cleanup();
1362 + extent_io_descriptor_cleanup();
1363 + rrdeng_cmd_cleanup();
1364 + pdc_cleanup();
1365 + page_details_cleanup();
1366 + rrdeng_query_handle_cleanup();
1367 + wal_cleanup();
1368 + extent_buffer_cleanup();
1369 + epdl_cleanup();
1370 + deol_cleanup();
1371 +#ifdef PDC_USE_JULYL
1372 + julyl_cleanup();
1373 #endif
1374 + }
1375
1376 worker_is_idle();
1377 }
1378
1313 -#define MAX_CMD_BATCH_SIZE (256)
1379 +bool rrdeng_dbengine_spawn(struct rrdengine_instance *ctx) {
1380 + static bool spawned = false;
1381
1315 -void rrdeng_worker(void* arg)
1316 -{
1317 - worker_register("DBENGINE");
1318 - worker_register_job_name(RRDENG_NOOP, "noop");
1319 - worker_register_job_name(RRDENG_READ_PAGE, "page read");
1320 - worker_register_job_name(RRDENG_READ_EXTENT, "extent read");
1321 - worker_register_job_name(RRDENG_COMMIT_PAGE, "commit");
1322 - worker_register_job_name(RRDENG_FLUSH_PAGES, "flush");
1323 - worker_register_job_name(RRDENG_SHUTDOWN, "shutdown");
1324 - worker_register_job_name(RRDENG_INVALIDATE_OLDEST_MEMORY_PAGE, "page lru");
1325 - worker_register_job_name(RRDENG_QUIESCE, "quiesce");
1326 - worker_register_job_name(RRDENG_MAX_OPCODE, "cleanup");
1327 - worker_register_job_name(RRDENG_MAX_OPCODE + 1, "timer");
1328 -
1329 - struct rrdengine_worker_config* wc = arg;
1330 - struct rrdengine_instance *ctx = wc->ctx;
1331 - uv_loop_t* loop;
1332 - int shutdown, ret;
1333 - enum rrdeng_opcode opcode;
1334 - uv_timer_t timer_req;
1335 - struct rrdeng_cmd cmd;
1336 - unsigned cmd_batch_size;
1382 + if(!spawned) {
1383 + int ret;
1384
1338 - rrdeng_init_cmd_queue(wc);
1385 + ret = uv_loop_init(&rrdeng_main.loop);
1386 + if (ret) {
1387 + error("DBENGINE: uv_loop_init(): %s", uv_strerror(ret));
1388 + return false;
1389 + }
1390 + rrdeng_main.loop.data = &rrdeng_main;
1391
1340 - loop = wc->loop = mallocz(sizeof(uv_loop_t));
1341 - ret = uv_loop_init(loop);
1342 - if (ret) {
1343 - error("uv_loop_init(): %s", uv_strerror(ret));
1344 - goto error_after_loop_init;
1345 - }
1346 - loop->data = wc;
1392 + ret = uv_async_init(&rrdeng_main.loop, &rrdeng_main.async, async_cb);
1393 + if (ret) {
1394 + error("DBENGINE: uv_async_init(): %s", uv_strerror(ret));
1395 + fatal_assert(0 == uv_loop_close(&rrdeng_main.loop));
1396 + return false;
1397 + }
1398 + rrdeng_main.async.data = &rrdeng_main;
1399 +
1400 + ret = uv_timer_init(&rrdeng_main.loop, &rrdeng_main.timer);
1401 + if (ret) {
1402 + error("DBENGINE: uv_timer_init(): %s", uv_strerror(ret));
1403 + uv_close((uv_handle_t *)&rrdeng_main.async, NULL);
1404 + fatal_assert(0 == uv_loop_close(&rrdeng_main.loop));
1405 + return false;
1406 + }
1407 + rrdeng_main.timer.data = &rrdeng_main;
1408
1348 - ret = uv_async_init(wc->loop, &wc->async, async_cb);
1349 - if (ret) {
1350 - error("uv_async_init(): %s", uv_strerror(ret));
1351 - goto error_after_async_init;
1409 + fatal_assert(0 == uv_thread_create(&rrdeng_main.thread, dbengine_event_loop, &rrdeng_main));
1410 + spawned = true;
1411 }
1353 - wc->async.data = wc;
1412
1355 - wc->now_deleting_files = NULL;
1356 - wc->cleanup_thread_deleting_files = 0;
1413 + ctx->worker_config.now_deleting_files = false;
1414 + ctx->worker_config.migration_to_v2_running = false;
1415 + ctx->worker_config.atomics.extents_currently_being_flushed = 0;
1416
1358 - wc->now_invalidating_dirty_pages = NULL;
1359 - wc->cleanup_thread_invalidating_dirty_pages = 0;
1360 - wc->inflight_dirty_pages = 0;
1417 + return true;
1418 +}
1419
1362 - /* dirty page flushing timer */
1363 - ret = uv_timer_init(loop, &timer_req);
1364 - if (ret) {
1365 - error("uv_timer_init(): %s", uv_strerror(ret));
1366 - goto error_after_timer_init;
1367 - }
1368 - timer_req.data = wc;
1420 +void dbengine_event_loop(void* arg) {
1421 + sanity_check();
1422 + uv_thread_set_name_np(pthread_self(), "DBENGINE");
1423
1370 - wc->error = 0;
1371 - /* wake up initialization thread */
1372 - completion_mark_complete(&ctx->rrdengine_completion);
1424 + worker_register("DBENGINE");
1425
1374 - fatal_assert(0 == uv_timer_start(&timer_req, timer_cb, TIMER_PERIOD_MS, TIMER_PERIOD_MS));
1375 - shutdown = 0;
1376 - int set_name = 0;
1377 - while (likely(shutdown == 0 || rrdeng_threads_alive(wc))) {
1426 + // opcode jobs
1427 + worker_register_job_name(RRDENG_OPCODE_NOOP, "noop");
1428 +
1429 + worker_register_job_name(RRDENG_OPCODE_EXTENT_READ, "extent read");
1430 + worker_register_job_name(RRDENG_OPCODE_PREP_QUERY, "prep query");
1431 + worker_register_job_name(RRDENG_OPCODE_FLUSH_PAGES, "flush pages");
1432 + worker_register_job_name(RRDENG_OPCODE_FLUSHED_TO_OPEN, "flushed to open");
1433 + worker_register_job_name(RRDENG_OPCODE_FLUSH_INIT, "flush init");
1434 + worker_register_job_name(RRDENG_OPCODE_EVICT_INIT, "evict init");
1435 + //worker_register_job_name(RRDENG_OPCODE_DATAFILE_CREATE, "datafile create");
1436 + worker_register_job_name(RRDENG_OPCODE_JOURNAL_FILE_INDEX, "journal file index");
1437 + worker_register_job_name(RRDENG_OPCODE_DATABASE_ROTATE, "db rotate");
1438 + worker_register_job_name(RRDENG_OPCODE_CTX_SHUTDOWN, "ctx shutdown");
1439 + worker_register_job_name(RRDENG_OPCODE_CTX_QUIESCE, "ctx quiesce");
1440 +
1441 + worker_register_job_name(RRDENG_OPCODE_MAX, "get opcode");
1442 +
1443 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_EXTENT_READ, "extent read cb");
1444 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_PREP_QUERY, "prep query cb");
1445 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_FLUSH_PAGES, "flush pages cb");
1446 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_FLUSHED_TO_OPEN, "flushed to open cb");
1447 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_FLUSH_INIT, "flush init cb");
1448 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_EVICT_INIT, "evict init cb");
1449 + //worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_DATAFILE_CREATE, "datafile create cb");
1450 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_JOURNAL_FILE_INDEX, "journal file index cb");
1451 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_DATABASE_ROTATE, "db rotate cb");
1452 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_CTX_SHUTDOWN, "ctx shutdown cb");
1453 + worker_register_job_name(RRDENG_OPCODE_MAX + RRDENG_OPCODE_CTX_QUIESCE, "ctx quiesce cb");
1454 +
1455 + // special jobs
1456 + worker_register_job_name(RRDENG_TIMER_CB, "timer");
1457 + worker_register_job_name(RRDENG_FLUSH_TRANSACTION_BUFFER_CB, "transaction buffer flush cb");
1458 +
1459 + worker_register_job_custom_metric(RRDENG_OPCODES_WAITING, "opcodes waiting", "opcodes", WORKER_METRIC_ABSOLUTE);
1460 + worker_register_job_custom_metric(RRDENG_WORKS_DISPATCHED, "works dispatched", "works", WORKER_METRIC_ABSOLUTE);
1461 + worker_register_job_custom_metric(RRDENG_WORKS_EXECUTING, "works executing", "works", WORKER_METRIC_ABSOLUTE);
1462 +
1463 + extent_buffer_init();
1464 +
1465 + struct rrdeng_main *main = arg;
1466 + enum rrdeng_opcode opcode;
1467 + struct rrdeng_cmd cmd;
1468 + main->tid = gettid();
1469 +
1470 + fatal_assert(0 == uv_timer_start(&main->timer, timer_cb, TIMER_PERIOD_MS, TIMER_PERIOD_MS));
1471 +
1472 + bool shutdown = false;
1473 + while (likely(!shutdown)) {
1474 worker_is_idle();
1379 - uv_run(loop, UV_RUN_DEFAULT);
1380 - worker_is_busy(RRDENG_MAX_OPCODE);
1381 - rrdeng_cleanup_finished_threads(wc);
1475 + uv_run(&main->loop, UV_RUN_DEFAULT);
1476
1477 /* wait for commands */
1384 - cmd_batch_size = 0;
1478 do {
1386 - /*
1387 - * Avoid starving the loop when there are too many commands coming in.
1388 - * timer_cb will interrupt the loop again to allow serving more commands.
1389 - */
1390 - if (unlikely(cmd_batch_size >= MAX_CMD_BATCH_SIZE))
1391 - break;
1392 -
1393 - cmd = rrdeng_deq_cmd(wc);
1479 + worker_is_busy(RRDENG_OPCODE_MAX);
1480 + cmd = rrdeng_deq_cmd();
1481 opcode = cmd.opcode;
1395 - ++cmd_batch_size;
1482
1397 - if(likely(opcode != RRDENG_NOOP))
1398 - worker_is_busy(opcode);
1483 + worker_is_busy(opcode);
1484
1485 switch (opcode) {
1401 - case RRDENG_NOOP:
1402 - /* the command queue was empty, do nothing */
1403 - break;
1404 - case RRDENG_SHUTDOWN:
1405 - shutdown = 1;
1406 - break;
1407 - case RRDENG_QUIESCE:
1408 - ctx->drop_metrics_under_page_cache_pressure = 0;
1409 - ctx->quiesce = SET_QUIESCE;
1410 - fatal_assert(0 == uv_timer_stop(&timer_req));
1411 - uv_close((uv_handle_t *)&timer_req, NULL);
1412 - while (do_flush_pages(wc, 1, NULL)) {
1413 - ; /* Force flushing of all committed pages. */
1486 + case RRDENG_OPCODE_EXTENT_READ: {
1487 + struct rrdengine_instance *ctx = cmd.ctx;
1488 + EPDL *epdl = cmd.data;
1489 + work_dispatch(ctx, epdl, NULL, opcode, extent_read_tp_worker, after_extent_read);
1490 + break;
1491 }
1415 - wal_flush_transaction_buffer(wc);
1416 - if (!rrdeng_threads_alive(wc)) {
1417 - ctx->quiesce = QUIESCED;
1418 - completion_mark_complete(&ctx->rrdengine_completion);
1492 +
1493 + case RRDENG_OPCODE_PREP_QUERY: {
1494 + struct rrdengine_instance *ctx = cmd.ctx;
1495 + PDC *pdc = cmd.data;
1496 + work_dispatch(ctx, pdc, NULL, opcode, query_prep_tp_worker, after_prep_query);
1497 + break;
1498 }
1420 - break;
1421 - case RRDENG_READ_PAGE:
1422 - do_read_extent(wc, &cmd.read_page.page_cache_descr, 1, 0);
1423 - break;
1424 - case RRDENG_READ_EXTENT:
1425 - do_read_extent(wc, cmd.read_extent.page_cache_descr, cmd.read_extent.page_count, 1);
1426 - if (unlikely(!set_name)) {
1427 - set_name = 1;
1428 - uv_thread_set_name_np(ctx->worker_config.thread, "DBENGINE");
1499 +
1500 + case RRDENG_OPCODE_FLUSH_PAGES: {
1501 + struct rrdengine_instance *ctx = cmd.ctx;
1502 + struct page_descr_with_data *base = cmd.data;
1503 + struct completion *completion = cmd.completion; // optional
1504 + // for the datafile and the journalfile
1505 + do_flush_extent(ctx, base, completion);
1506 + break;
1507 }
1430 - break;
1431 - case RRDENG_COMMIT_PAGE:
1432 - do_commit_transaction(wc, STORE_DATA, NULL);
1433 - break;
1434 - case RRDENG_FLUSH_PAGES: {
1435 - if (wc->now_invalidating_dirty_pages) {
1436 - /* Do not flush if the disk cannot keep up */
1437 - completion_mark_complete(cmd.completion);
1438 - } else {
1439 - (void)do_flush_pages(wc, 1, cmd.completion);
1508 +
1509 + case RRDENG_OPCODE_FLUSHED_TO_OPEN: {
1510 + struct rrdengine_instance *ctx = cmd.ctx;
1511 + uv_fs_t *uv_fs_request = cmd.data;
1512 + struct extent_io_descriptor *xt_io_descr = uv_fs_request->data;
1513 + struct completion *completion = xt_io_descr->completion;
1514 + work_dispatch(ctx, uv_fs_request, completion, opcode, extent_flushed_to_open_tp_worker, after_extent_flushed_to_open);
1515 + break;
1516 + }
1517 +
1518 + case RRDENG_OPCODE_FLUSH_INIT: {
1519 + if(!rrdeng_main.flush_running) {
1520 +
1521 + rrdeng_main.flush_running = true;
1522 + if(!work_dispatch(NULL, NULL, NULL, opcode, cache_flush_tp_worker, after_do_cache_flush))
1523 + rrdeng_main.flush_running = false;
1524 +
1525 + }
1526 + break;
1527 + }
1528 +
1529 + case RRDENG_OPCODE_EVICT_INIT: {
1530 + if(!rrdeng_main.evict_running) {
1531 +
1532 + rrdeng_main.evict_running = true;
1533 + if (!work_dispatch(NULL, NULL, NULL, opcode, cache_evict_tp_worker, after_do_cache_evict))
1534 + rrdeng_main.evict_running = false;
1535 +
1536 + }
1537 + break;
1538 + }
1539 +
1540 +// case RRDENG_OPCODE_DATAFILE_CREATE: {
1541 +// struct rrdengine_instance *ctx = cmd.ctx;
1542 +// struct rrdengine_datafile *datafile = ctx->datafiles.first->prev;
1543 +// if(datafile->pos > rrdeng_target_data_file_size(ctx) &&
1544 +// create_new_datafile_pair(ctx, 1, ctx->last_fileno + 1) == 0) {
1545 +// ++ctx->last_fileno;
1546 +// rrdeng_enq_cmd(ctx, RRDENG_OPCODE_JOURNAL_FILE_INDEX, datafile, NULL, STORAGE_PRIORITY_CRITICAL);
1547 +// }
1548 +// break;
1549 +// }
1550 +
1551 + case RRDENG_OPCODE_JOURNAL_FILE_INDEX: {
1552 + struct rrdengine_instance *ctx = cmd.ctx;
1553 + struct rrdengine_datafile *datafile = cmd.data;
1554 + if(!ctx->worker_config.migration_to_v2_running) {
1555 +
1556 + ctx->worker_config.migration_to_v2_running = true;
1557 + if (!work_dispatch(ctx, datafile, NULL, opcode, journal_v2_indexing_tp_worker, after_journal_v2_indexing))
1558 + ctx->worker_config.migration_to_v2_running = false;
1559 +
1560 + }
1561 + break;
1562 + }
1563 +
1564 + case RRDENG_OPCODE_DATABASE_ROTATE: {
1565 + struct rrdengine_instance *ctx = cmd.ctx;
1566 + if (!ctx->worker_config.now_deleting_files &&
1567 + ctx->datafiles.first->next != NULL &&
1568 + ctx->datafiles.first->next->next != NULL &&
1569 + ctx->disk_space > MAX(ctx->max_disk_space, 2 * ctx->metric_API_max_producers * RRDENG_BLOCK_SIZE)) {
1570 +
1571 + ctx->worker_config.now_deleting_files = true;
1572 + if(!work_dispatch(ctx, NULL, NULL, opcode, database_rotate_tp_worker, after_database_rotate))
1573 + ctx->worker_config.now_deleting_files = false;
1574 +
1575 + }
1576 + break;
1577 + }
1578 +
1579 + case RRDENG_OPCODE_CTX_QUIESCE: {
1580 + // a ctx will shutdown shortly
1581 + struct rrdengine_instance *ctx = cmd.ctx;
1582 + __atomic_store_n(&ctx->quiesce, SET_QUIESCE, __ATOMIC_RELEASE);
1583 + work_dispatch(ctx, NULL, NULL, opcode,
1584 + flush_all_hot_and_dirty_pages_of_section_tp_worker,
1585 + after_flush_all_hot_and_dirty_pages_of_section);
1586 + break;
1587 + }
1588 +
1589 + case RRDENG_OPCODE_CTX_SHUTDOWN: {
1590 + // a ctx is shutting down
1591 + struct rrdengine_instance *ctx = cmd.ctx;
1592 + struct completion *completion = cmd.completion;
1593 + work_dispatch(ctx, NULL, completion, opcode, ctx_shutdown_tp_worker, after_ctx_shutdown);
1594 + break;
1595 + }
1596 +
1597 + case RRDENG_OPCODE_NOOP: {
1598 + /* the command queue was empty, do nothing */
1599 + break;
1600 + }
1601 +
1602 + // not opcodes
1603 + case RRDENG_OPCODE_MAX:
1604 + default: {
1605 + internal_fatal(true, "DBENGINE: unknown opcode");
1606 + break;
1607 }
1441 - break;
1442 - case RRDENG_INVALIDATE_OLDEST_MEMORY_PAGE:
1443 - rrdeng_invalidate_oldest_committed(wc);
1444 - break;
1445 - }
1446 - default:
1447 - debug(D_RRDENGINE, "%s: default.", __func__);
1448 - break;
1608 }
1450 - } while (opcode != RRDENG_NOOP);
1609 +
1610 + } while (opcode != RRDENG_OPCODE_NOOP);
1611 }
1612
1613 /* cleanup operations of the event loop */
1454 - info("Shutting down RRD engine event loop for tier %d", ctx->tier);
1614 + info("DBENGINE: shutting down dbengine thread");
1615
1616 /*
1617 * uv_async_send after uv_close does not seem to crash in linux at the moment,
1618 * it is however undocumented behaviour and we need to be aware if this becomes
1619 * an issue in the future.
1620 */
1461 - uv_close((uv_handle_t *)&wc->async, NULL);
1462 -
1463 - while (do_flush_pages(wc, 1, NULL)) {
1464 - ; /* Force flushing of all committed pages. */
1465 - }
1466 - wal_flush_transaction_buffer(wc);
1467 - uv_run(loop, UV_RUN_DEFAULT);
1468 -
1469 - info("Shutting down RRD engine event loop for tier %d complete", ctx->tier);
1470 - /* TODO: don't let the API block by waiting to enqueue commands */
1471 - uv_cond_destroy(&wc->cmd_cond);
1472 -/* uv_mutex_destroy(&wc->cmd_mutex); */
1473 - fatal_assert(0 == uv_loop_close(loop));
1474 - freez(loop);
1475 -
1621 + uv_close((uv_handle_t *)&main->async, NULL);
1622 + uv_timer_stop(&main->timer);
1623 + uv_close((uv_handle_t *)&main->timer, NULL);
1624 + uv_run(&main->loop, UV_RUN_DEFAULT);
1625 + uv_loop_close(&main->loop);
1626 worker_unregister();
1477 - return;
1478 -
1479 -error_after_timer_init:
1480 - uv_close((uv_handle_t *)&wc->async, NULL);
1481 -error_after_async_init:
1482 - fatal_assert(0 == uv_loop_close(loop));
1483 -error_after_loop_init:
1484 - freez(loop);
1485 -
1486 - wc->error = UV_EAGAIN;
1487 - /* wake up initialization thread */
1488 - completion_mark_complete(&ctx->rrdengine_completion);
1489 - worker_unregister();
1490 -}
1491 -
1492 -/* C entry point for development purposes
1493 - * make "LDFLAGS=-errdengine_main"
1494 - */
1495 -void rrdengine_main(void)
1496 -{
1497 - int ret;
1498 - struct rrdengine_instance *ctx;
1499 -
1500 - sanity_check();
1501 - ret = rrdeng_init(NULL, &ctx, "/tmp", RRDENG_MIN_PAGE_CACHE_SIZE_MB, RRDENG_MIN_DISK_SPACE_MB, 0);
1502 - if (ret) {
1503 - exit(ret);
1504 - }
1505 - rrdeng_exit(ctx);
1506 - fprintf(stderr, "Hello world!");
1507 - exit(0);
1627 }
database/engine/rrdengine.h
+281 -130
@@ -19,11 +19,9 @@
19 #include "journalfile.h"
20 #include "rrdengineapi.h"
21 #include "pagecache.h"
22 -#include "rrdenglocking.h"
23 -
24 -#ifdef NETDATA_RRD_INTERNALS
25 -
26 -#endif /* NETDATA_RRD_INTERNALS */
22 +#include "metric.h"
23 +#include "cache.h"
24 +#include "pdc.h"
25
26 extern unsigned rrdeng_pages_per_extent;
27
@@ -32,153 +30,288 @@ struct rrdengine_instance;
30
31 #define MAX_PAGES_PER_EXTENT (64) /* TODO: can go higher only when journal supports bigger than 4KiB transactions */
32
33 +#define GET_JOURNAL_DATA(x) __atomic_load_n(&(x)->journal_data, __ATOMIC_ACQUIRE)
34 +#define GET_JOURNAL_DATA_SIZE(x) __atomic_load_n(&(x)->journal_data_size, __ATOMIC_ACQUIRE)
35 +#define SET_JOURNAL_DATA(x, y) __atomic_store_n(&(x)->journal_data, (y), __ATOMIC_RELEASE)
36 +#define SET_JOURNAL_DATA_SIZE(x, y) __atomic_store_n(&(x)->journal_data_size, (y), __ATOMIC_RELEASE)
37 +
38 #define RRDENG_FILE_NUMBER_SCAN_TMPL "%1u-%10u"
39 #define RRDENG_FILE_NUMBER_PRINT_TMPL "%1.1u-%10.10u"
40
41 +typedef struct page_details_control {
42 + struct rrdengine_instance *ctx;
43 + struct metric *metric;
44 +
45 + struct completion prep_completion;
46 + struct completion page_completion; // sync between the query thread and the workers
47 +
48 + Pvoid_t page_list_JudyL; // the list of page details
49 + unsigned completed_jobs; // the number of jobs completed last time the query thread checked
50 + bool preload_all_extent_pages; // true to preload all the pages on each extent involved in the query
51 + bool workers_should_stop; // true when the query thread left and the workers should stop
52 + bool prep_done;
53 +
54 + SPINLOCK refcount_spinlock; // spinlock to protect refcount
55 + int32_t refcount; // the number of workers currently working on this request + 1 for the query thread
56 + size_t executed_with_gaps;
57 +
58 + time_t start_time_s;
59 + time_t end_time_s;
60 + STORAGE_PRIORITY priority;
61 +
62 + time_t optimal_end_time_s;
63 +
64 + struct {
65 + struct page_details_control *prev;
66 + struct page_details_control *next;
67 + } cache;
68 +} PDC;
69 +
70 +PDC *pdc_get(void);
71 +
72 +typedef enum __attribute__ ((__packed__)) {
73 + // final status for all pages
74 + // if a page does not have one of these, it is considered unroutable
75 + PDC_PAGE_READY = (1 << 0), // ready to be processed (pd->page is not null)
76 + PDC_PAGE_FAILED = (1 << 1), // failed to be loaded (pd->page is null)
77 + PDC_PAGE_SKIP = (1 << 2), // don't use this page, it is not good for us
78 + PDC_PAGE_INVALID = (1 << 3), // don't use this page, it is invalid
79 +
80 + // other statuses for tracking issues
81 + PDC_PAGE_PREPROCESSED = (1 << 4), // used during preprocessing
82 + PDC_PAGE_PROCESSED = (1 << 5), // processed by the query caller
83 + PDC_PAGE_RELEASED = (1 << 6), // already released
84 +
85 + // data found in cache (preloaded) or on disk?
86 + PDC_PAGE_PRELOADED = (1 << 7), // data found in memory
87 + PDC_PAGE_DISK_PENDING = (1 << 8), // data need to be loaded from disk
88 +
89 + // worker related statuses
90 + PDC_PAGE_FAILED_INVALID_EXTENT = (1 << 9),
91 + PDC_PAGE_FAILED_UUID_NOT_IN_EXTENT = (1 << 10),
92 + PDC_PAGE_FAILED_TO_MAP_EXTENT = (1 << 11),
93 + PDC_PAGE_FAILED_TO_ACQUIRE_DATAFILE= (1 << 12),
94 +
95 + PDC_PAGE_LOADED_FROM_EXTENT_CACHE = (1 << 13),
96 + PDC_PAGE_LOADED_FROM_DISK = (1 << 14),
97 +
98 + PDC_PAGE_PRELOADED_PASS1 = (1 << 15),
99 + PDC_PAGE_PRELOADED_PASS4 = (1 << 16),
100 + PDC_PAGE_PRELOADED_WORKER = (1 << 17),
101 +
102 + PDC_PAGE_SOURCE_MAIN_CACHE = (1 << 19),
103 + PDC_PAGE_SOURCE_OPEN_CACHE = (1 << 19),
104 + PDC_PAGE_SOURCE_JOURNAL_V2 = (1 << 20),
105 +
106 + // datafile acquired
107 + PDC_PAGE_DATAFILE_ACQUIRED = (1 << 30),
108 +} PDC_PAGE_STATUS;
109 +
110 +struct page_details {
111 + struct {
112 + struct rrdengine_datafile *ptr;
113 + uv_file file;
114 + unsigned fileno;
115 +
116 + struct {
117 + uint64_t pos;
118 + uint32_t bytes;
119 + } extent;
120 + } datafile;
121 +
122 + struct pgc_page *page;
123 + Word_t metric_id;
124 + time_t first_time_s;
125 + time_t last_time_s;
126 + uint32_t update_every_s;
127 + uint16_t page_length;
128 + PDC_PAGE_STATUS status;
129 +
130 + struct {
131 + struct page_details *prev;
132 + struct page_details *next;
133 + } cache;
134 +};
135 +
136 +struct page_details *page_details_get(void);
137 +
138 +#define pdc_page_status_check(pd, flag) (__atomic_load_n(&((pd)->status), __ATOMIC_ACQUIRE) & (flag))
139 +#define pdc_page_status_set(pd, flag) __atomic_or_fetch(&((pd)->status), flag, __ATOMIC_RELEASE)
140 +#define pdc_page_status_clear(pd, flag) __atomic_and_fetch(&((od)->status), ~(flag), __ATOMIC_RELEASE)
141 +
142 +struct jv2_extents_info {
143 + size_t index;
144 + uint64_t pos;
145 + unsigned bytes;
146 + size_t number_of_pages;
147 +};
148 +
149 +struct jv2_metrics_info {
150 + uuid_t *uuid;
151 + uint32_t page_list_header;
152 + time_t first_time_s;
153 + time_t last_time_s;
154 + size_t number_of_pages;
155 + Pvoid_t JudyL_pages_by_start_time;
156 +};
157 +
158 +struct jv2_page_info {
159 + time_t start_time_s;
160 + time_t end_time_s;
161 + time_t update_every_s;
162 + size_t page_length;
163 + uint32_t extent_index;
164 + void *custom_data;
165 +
166 + // private
167 + struct pgc_page *page;
168 +};
169 +
170 +typedef enum __attribute__ ((__packed__)) {
171 + RRDENG_CHO_UNALIGNED = (1 << 0), // set when this metric is not page aligned according to page alignment
172 + RRDENG_FIRST_PAGE_ALLOCATED = (1 << 1), // set when this metric has allocated its first page
173 +} RRDENG_COLLECT_HANDLE_OPTIONS;
174 +
175 struct rrdeng_collect_handle {
39 - struct pg_cache_page_index *page_index;
40 - struct rrdeng_page_descr *descr;
41 - unsigned long page_correlation_id;
42 - // set to 1 when this dimension is not page aligned with the other dimensions in the chart
43 - uint8_t unaligned_page;
176 + struct metric *metric;
177 + struct pgc_page *page;
178 struct pg_alignment *alignment;
179 + RRDENG_COLLECT_HANDLE_OPTIONS options;
180 + uint8_t type;
181 + // 2 bytes remaining here for future use
182 + uint32_t page_entries_max;
183 + uint32_t page_position; // keep track of the current page size, to make sure we don't exceed it
184 + usec_t page_end_time_ut;
185 + usec_t update_every_ut;
186 };
187
188 struct rrdeng_query_handle {
48 - struct rrdeng_page_descr *descr;
189 + struct metric *metric;
190 + struct pgc_page *page;
191 struct rrdengine_instance *ctx;
50 - struct pg_cache_page_index *page_index;
51 - time_t wanted_start_time_s;
192 + storage_number *metric_data;
193 + struct page_details_control *pdc;
194 +
195 + // the request
196 + time_t start_time_s;
197 + time_t end_time_s;
198 + STORAGE_PRIORITY priority;
199 +
200 + // internal data
201 time_t now_s;
202 + time_t dt_s;
203 +
204 unsigned position;
205 unsigned entries;
55 - storage_number *page;
56 - usec_t page_end_time_ut;
57 - uint32_t page_length;
58 - time_t dt_s;
206 +
207 + struct {
208 + struct rrdeng_query_handle *prev;
209 + struct rrdeng_query_handle *next;
210 + } cache;
211 +
212 +#ifdef NETDATA_INTERNAL_CHECKS
213 + usec_t started_time_s;
214 + pid_t query_pid;
215 + struct rrdeng_query_handle *prev, *next;
216 +#endif
217 };
218
61 -typedef enum {
62 - RRDENGINE_STATUS_UNINITIALIZED = 0,
63 - RRDENGINE_STATUS_INITIALIZING,
64 - RRDENGINE_STATUS_INITIALIZED
65 -} rrdengine_state_t;
219 +struct rrdeng_query_handle *rrdeng_query_handle_get(void);
220 +void rrdeng_query_handle_release(struct rrdeng_query_handle *handle);
221
222 enum rrdeng_opcode {
223 /* can be used to return empty status or flush the command queue */
69 - RRDENG_NOOP = 0,
70 -
71 - RRDENG_READ_PAGE,
72 - RRDENG_READ_EXTENT,
73 - RRDENG_COMMIT_PAGE,
74 - RRDENG_FLUSH_PAGES,
75 - RRDENG_SHUTDOWN,
76 - RRDENG_INVALIDATE_OLDEST_MEMORY_PAGE,
77 - RRDENG_QUIESCE,
78 -
79 - RRDENG_MAX_OPCODE
80 -};
81 -
82 -struct rrdeng_read_page {
83 - struct rrdeng_page_descr *page_cache_descr;
84 -};
85 -
86 -struct rrdeng_read_extent {
87 - struct rrdeng_page_descr *page_cache_descr[MAX_PAGES_PER_EXTENT];
88 - int page_count;
224 + RRDENG_OPCODE_NOOP = 0,
225 +
226 + RRDENG_OPCODE_EXTENT_READ,
227 + RRDENG_OPCODE_PREP_QUERY,
228 + RRDENG_OPCODE_FLUSH_PAGES,
229 + RRDENG_OPCODE_FLUSHED_TO_OPEN,
230 + RRDENG_OPCODE_FLUSH_INIT,
231 + RRDENG_OPCODE_EVICT_INIT,
232 + //RRDENG_OPCODE_DATAFILE_CREATE,
233 + RRDENG_OPCODE_JOURNAL_FILE_INDEX,
234 + RRDENG_OPCODE_DATABASE_ROTATE,
235 + RRDENG_OPCODE_CTX_SHUTDOWN,
236 + RRDENG_OPCODE_CTX_QUIESCE,
237 +
238 + RRDENG_OPCODE_MAX
239 };
240
91 -struct rrdeng_cmd {
92 - enum rrdeng_opcode opcode;
93 - union {
94 - struct rrdeng_read_page read_page;
95 - struct rrdeng_read_extent read_extent;
96 - struct completion *completion;
97 - };
98 -};
99 -
100 -#define RRDENG_CMD_Q_MAX_SIZE (2048)
101 -
102 -struct rrdeng_cmdqueue {
103 - unsigned head, tail;
104 - struct rrdeng_cmd cmd_array[RRDENG_CMD_Q_MAX_SIZE];
241 +// WORKERS IDS:
242 +// RRDENG_MAX_OPCODE : reserved for the cleanup
243 +// RRDENG_MAX_OPCODE + opcode : reserved for the callbacks of each opcode
244 +// RRDENG_MAX_OPCODE + RRDENG_MAX_OPCODE : reserved for the timer
245 +#define RRDENG_TIMER_CB (RRDENG_OPCODE_MAX + RRDENG_OPCODE_MAX)
246 +#define RRDENG_FLUSH_TRANSACTION_BUFFER_CB (RRDENG_TIMER_CB + 1)
247 +#define RRDENG_OPCODES_WAITING (RRDENG_TIMER_CB + 2)
248 +#define RRDENG_WORKS_DISPATCHED (RRDENG_TIMER_CB + 3)
249 +#define RRDENG_WORKS_EXECUTING (RRDENG_TIMER_CB + 4)
250 +
251 +struct extent_io_data {
252 + unsigned fileno;
253 + uv_file file;
254 + uint64_t pos;
255 + unsigned bytes;
256 + uint16_t page_length;
257 };
258
259 struct extent_io_descriptor {
108 - uv_fs_t req;
109 - uv_work_t req_worker;
260 + struct rrdengine_instance *ctx;
261 + uv_fs_t uv_fs_request;
262 uv_buf_t iov;
263 uv_file file;
264 void *buf;
113 - void *map_base;
114 - size_t map_length;
265 + struct wal *wal;
266 uint64_t pos;
267 unsigned bytes;
268 struct completion *completion;
269 unsigned descr_count;
119 - int release_descr;
120 - struct rrdeng_page_descr *descr_array[MAX_PAGES_PER_EXTENT];
121 - struct rrdeng_page_descr descr_read_array[MAX_PAGES_PER_EXTENT];
122 - Word_t descr_commit_idx_array[MAX_PAGES_PER_EXTENT];
270 + struct page_descr_with_data *descr_array[MAX_PAGES_PER_EXTENT];
271 + struct rrdengine_datafile *datafile;
272 struct extent_io_descriptor *next; /* multiple requests to be served by the same cached extent */
273 +
274 + struct {
275 + struct extent_io_descriptor *prev;
276 + struct extent_io_descriptor *next;
277 + } cache;
278 };
279
280 struct generic_io_descriptor {
281 + struct rrdengine_instance *ctx;
282 uv_fs_t req;
283 uv_buf_t iov;
284 void *buf;
285 + void *data;
286 uint64_t pos;
287 unsigned bytes;
288 struct completion *completion;
289 };
290
135 -struct extent_cache_element {
136 - struct extent_info *extent; /* The ABA problem is avoided with the help of fileno below */
137 - unsigned fileno;
138 - struct extent_cache_element *prev; /* LRU */
139 - struct extent_cache_element *next; /* LRU */
140 - struct extent_io_descriptor *inflight_io_descr; /* I/O descriptor for in-flight extent */
141 - uint8_t pages[MAX_PAGES_PER_EXTENT * RRDENG_BLOCK_SIZE];
142 -};
143 -
144 -#define MAX_CACHED_EXTENTS 16 /* cannot be over 32 to fit in 32-bit architectures */
291 +typedef struct wal {
292 + uint64_t transaction_id;
293 + void *buf;
294 + size_t size;
295 + size_t buf_size;
296 + struct generic_io_descriptor io_descr;
297
146 -/* Initialize by setting the structure to zero */
147 -struct extent_cache {
148 - struct extent_cache_element extent_array[MAX_CACHED_EXTENTS];
149 - unsigned allocation_bitmap; /* 1 if the corresponding position in the extent_array is allocated */
150 - unsigned inflight_bitmap; /* 1 if the corresponding position in the extent_array is waiting for I/O */
298 + struct {
299 + struct wal *prev;
300 + struct wal *next;
301 + } cache;
302 +} WAL;
303
152 - struct extent_cache_element *replaceQ_head; /* LRU */
153 - struct extent_cache_element *replaceQ_tail; /* MRU */
154 -};
304 +WAL *wal_get(struct rrdengine_instance *ctx, unsigned size);
305 +void wal_release(WAL *wal);
306
307 struct rrdengine_worker_config {
157 - struct rrdengine_instance *ctx;
158 -
159 - uv_thread_t thread;
160 - uv_loop_t* loop;
161 - uv_async_t async;
162 -
163 - /* file deletion thread */
164 - uv_thread_t *now_deleting_files;
165 - unsigned long cleanup_thread_deleting_files; /* set to 0 when now_deleting_files is still running */
166 -
167 - /* dirty page deletion thread */
168 - uv_thread_t *now_invalidating_dirty_pages;
169 - /* set to 0 when now_invalidating_dirty_pages is still running */
170 - unsigned long cleanup_thread_invalidating_dirty_pages;
171 - unsigned inflight_dirty_pages;
172 -
173 - /* FIFO command queue */
174 - uv_mutex_t cmd_mutex;
175 - uv_cond_t cmd_cond;
176 - volatile unsigned queue_size;
177 - struct rrdeng_cmdqueue cmd_queue;
308 + bool now_deleting_files;
309 + bool migration_to_v2_running;
310
179 - struct extent_cache xt_cache;
180 -
181 - int error;
311 + struct {
312 + // non-zero until we commit data to disk (both datafile and journal file)
313 + unsigned extents_currently_being_flushed;
314 + } atomics;
315 };
316
317 /*
@@ -231,20 +364,10 @@ extern rrdeng_stats_t global_flushing_pressure_page_deletions; /* number of dele
364 #define SET_QUIESCE (1) /* set it before shutting down the instance, quiesce long running operations */
365 #define QUIESCED (2) /* is set after all threads have finished running */
366
234 -typedef enum {
235 - LOAD_ERRORS_PAGE_FLIPPED_TIME = 0,
236 - LOAD_ERRORS_PAGE_EQUAL_TIME = 1,
237 - LOAD_ERRORS_PAGE_ZERO_ENTRIES = 2,
238 - LOAD_ERRORS_PAGE_UPDATE_ZERO = 3,
239 - LOAD_ERRORS_PAGE_FLEXY_TIME = 4,
240 - LOAD_ERRORS_DROPPED_EXTENT = 5,
241 -} INVALID_PAGE_ID;
242 -
367 struct rrdengine_instance {
368 struct rrdengine_worker_config worker_config;
369 struct completion rrdengine_completion;
246 - struct page_cache pg_cache;
247 - uint8_t drop_metrics_under_page_cache_pressure; /* boolean */
370 + bool journal_initialization;
371 uint8_t global_compress_alg;
372 struct transaction_commit_log commit_log;
373 struct rrdengine_datafile_list datafiles;
@@ -255,29 +378,57 @@ struct rrdengine_instance {
378 uint64_t max_disk_space;
379 int tier;
380 unsigned last_fileno; /* newest index of datafile and journalfile */
258 - unsigned long max_cache_pages;
259 - unsigned long cache_pages_low_watermark;
381 + unsigned last_flush_fileno;
382 unsigned long metric_API_max_producers;
383
262 - uint8_t quiesce; /* set to SET_QUIESCE before shutdown of the engine */
384 + bool create_new_datafile_pair;
385 + uint8_t quiesce; /* set to SET_QUIESCE before shutdown of the engine */
386 uint8_t page_type; /* Default page type for this context */
387
265 - struct rrdengine_statistics stats;
388 + struct completion quiesce_completion;
389
267 - struct {
268 - size_t counter;
269 - usec_t latest_end_time_ut;
270 - } load_errors[6];
390 + size_t inflight_queries;
391 + struct rrdengine_statistics stats;
392 };
393
273 -void *dbengine_page_alloc(void);
394 +#define ctx_is_available_for_queries(ctx) (__atomic_load_n(&(ctx)->quiesce, __ATOMIC_RELAXED) == NO_QUIESCE)
395 +
396 +void *dbengine_page_alloc(struct rrdengine_instance *ctx, size_t size);
397 void dbengine_page_free(void *page);
398
399 int init_rrd_files(struct rrdengine_instance *ctx);
400 void finalize_rrd_files(struct rrdengine_instance *ctx);
278 -void rrdeng_test_quota(struct rrdengine_worker_config* wc);
279 -void rrdeng_worker(void* arg);
280 -void rrdeng_enq_cmd(struct rrdengine_worker_config* wc, struct rrdeng_cmd *cmd);
281 -struct rrdeng_cmd rrdeng_deq_cmd(struct rrdengine_worker_config* wc);
401 +bool rrdeng_dbengine_spawn(struct rrdengine_instance *ctx);
402 +void dbengine_event_loop(void *arg);
403 +void rrdeng_enq_cmd(struct rrdengine_instance *ctx, enum rrdeng_opcode opcode, void *data, struct completion *completion, enum storage_priority priority);
404 +
405 +void pdc_route_asynchronously(struct rrdengine_instance *ctx, struct page_details_control *pdc);
406 +void pdc_route_synchronously(struct rrdengine_instance *ctx, struct page_details_control *pdc);
407 +
408 +void pdc_acquire(PDC *pdc);
409 +bool pdc_release_and_destroy_if_unreferenced(PDC *pdc, bool worker, bool router);
410 +
411 +unsigned rrdeng_target_data_file_size(struct rrdengine_instance *ctx);
412 +
413 +struct page_descr_with_data *page_descriptor_get(void);
414 +
415 +typedef struct validated_page_descriptor {
416 + time_t start_time_s;
417 + time_t end_time_s;
418 + time_t update_every_s;
419 + size_t page_length;
420 + size_t point_size;
421 + size_t entries;
422 + uint8_t type;
423 + bool data_on_disk_valid;
424 +} VALIDATED_PAGE_DESCRIPTOR;
425 +
426 +#define page_entries_by_time(start_time_s, end_time_s, update_every_s) \
427 + ((update_every_s) ? (((end_time_s) - ((start_time_s) - (update_every_s))) / (update_every_s)) : 1)
428 +
429 +#define page_entries_by_size(page_length_in_bytes, point_size_in_bytes) \
430 + ((page_length_in_bytes) / (point_size_in_bytes))
431 +
432 +VALIDATED_PAGE_DESCRIPTOR validate_extent_page_descr(const struct rrdeng_extent_page_descr *descr, time_t now_s, time_t overwrite_zero_update_every_s, bool have_read_error);
433
434 #endif /* NETDATA_RRDENGINE_H */
database/engine/rrdengineapi.c
+520 -703
@@ -1,6 +1,5 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2 #include "rrdengine.h"
3 -#include "../storage_engine.h"
3
4 /* Default global database instance */
5 struct rrdengine_instance multidb_ctx_storage_tier0;
@@ -8,11 +7,16 @@ struct rrdengine_instance multidb_ctx_storage_tier1;
7 struct rrdengine_instance multidb_ctx_storage_tier2;
8 struct rrdengine_instance multidb_ctx_storage_tier3;
9 struct rrdengine_instance multidb_ctx_storage_tier4;
10 +
11 +#define mrg_metric_ctx(metric) (struct rrdengine_instance *)mrg_metric_section(main_mrg, metric)
12 +
13 +
14 #if RRD_STORAGE_TIERS != 5
15 #error RRD_STORAGE_TIERS is not 5 - you need to add allocations here
16 #endif
17 struct rrdengine_instance *multidb_ctx[RRD_STORAGE_TIERS];
18 uint8_t tier_page_type[RRD_STORAGE_TIERS] = {PAGE_METRICS, PAGE_TIER, PAGE_TIER, PAGE_TIER, PAGE_TIER};
19 +size_t tier_page_size[RRD_STORAGE_TIERS] = {4096, 2048, 384, 384, 384};
20
21 #if PAGE_TYPE_MAX != 1
22 #error PAGE_TYPE_MAX is not 1 - you need to add allocations here
@@ -27,14 +31,12 @@ __attribute__((constructor)) void initialize_multidb_ctx(void) {
31 multidb_ctx[4] = &multidb_ctx_storage_tier4;
32 }
33
30 -int db_engine_use_malloc = 0;
34 int default_rrdeng_page_fetch_timeout = 3;
35 int default_rrdeng_page_fetch_retries = 3;
36 int default_rrdeng_page_cache_mb = 32;
37 +int db_engine_journal_check = 0;
38 int default_rrdeng_disk_quota_mb = 256;
39 int default_multidb_disk_quota_mb = 256;
36 -/* Default behaviour is to unblock data collection if the page cache is full of dirty pages by dropping metrics */
37 -uint8_t rrdeng_drop_metrics_under_page_cache_pressure = 1;
40
41 // ----------------------------------------------------------------------------
42 // metrics groups
@@ -90,111 +92,77 @@ void rrdeng_generate_legacy_uuid(const char *dim_id, const char *chart_id, uuid_
92 memcpy(ret_uuid, hash_value, sizeof(uuid_t));
93 }
94
93 -/* Transform legacy UUID to be unique across hosts deterministically */
94 -void rrdeng_convert_legacy_uuid_to_multihost(char machine_guid[GUID_LEN + 1], uuid_t *legacy_uuid, uuid_t *ret_uuid)
95 -{
96 - EVP_MD_CTX *evpctx;
97 - unsigned char hash_value[EVP_MAX_MD_SIZE];
98 - unsigned int hash_len;
99 -
100 - evpctx = EVP_MD_CTX_create();
101 - EVP_DigestInit_ex(evpctx, EVP_sha256(), NULL);
102 - EVP_DigestUpdate(evpctx, machine_guid, GUID_LEN);
103 - EVP_DigestUpdate(evpctx, *legacy_uuid, sizeof(uuid_t));
104 - EVP_DigestFinal_ex(evpctx, hash_value, &hash_len);
105 - EVP_MD_CTX_destroy(evpctx);
106 - fatal_assert(hash_len > sizeof(uuid_t));
107 - memcpy(ret_uuid, hash_value, sizeof(uuid_t));
108 -}
109 -
110 -STORAGE_METRIC_HANDLE *rrdeng_metric_get_legacy(STORAGE_INSTANCE *db_instance, const char *rd_id, const char *st_id) {
95 +static METRIC *rrdeng_metric_get_legacy(STORAGE_INSTANCE *db_instance, const char *rd_id, const char *st_id) {
96 + struct rrdengine_instance *ctx = (struct rrdengine_instance *)db_instance;
97 uuid_t legacy_uuid;
98 rrdeng_generate_legacy_uuid(rd_id, st_id, &legacy_uuid);
113 - return rrdeng_metric_get(db_instance, &legacy_uuid);
99 + return mrg_metric_get_and_acquire(main_mrg, &legacy_uuid, (Word_t) ctx);
100 }
101
102 // ----------------------------------------------------------------------------
103 // metric handle
104
105 void rrdeng_metric_release(STORAGE_METRIC_HANDLE *db_metric_handle) {
120 - struct pg_cache_page_index *page_index = (struct pg_cache_page_index *)db_metric_handle;
121 -
122 - __atomic_sub_fetch(&page_index->refcount, 1, __ATOMIC_SEQ_CST);
106 + METRIC *metric = (METRIC *)db_metric_handle;
107 + mrg_metric_release(main_mrg, metric);
108 }
109
110 STORAGE_METRIC_HANDLE *rrdeng_metric_dup(STORAGE_METRIC_HANDLE *db_metric_handle) {
126 - struct pg_cache_page_index *page_index = (struct pg_cache_page_index *)db_metric_handle;
127 - __atomic_add_fetch(&page_index->refcount, 1, __ATOMIC_SEQ_CST);
128 - return db_metric_handle;
111 + METRIC *metric = (METRIC *)db_metric_handle;
112 + return (STORAGE_METRIC_HANDLE *) mrg_metric_dup(main_mrg, metric);
113 }
114
115 STORAGE_METRIC_HANDLE *rrdeng_metric_get(STORAGE_INSTANCE *db_instance, uuid_t *uuid) {
116 struct rrdengine_instance *ctx = (struct rrdengine_instance *)db_instance;
133 - struct page_cache *pg_cache = &ctx->pg_cache;
134 - struct pg_cache_page_index *page_index = NULL;
135 -
136 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
137 - Pvoid_t *PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, uuid, sizeof(uuid_t));
138 - if (likely(NULL != PValue))
139 - page_index = *PValue;
140 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
141 -
142 - if (likely(page_index))
143 - __atomic_add_fetch(&page_index->refcount, 1, __ATOMIC_SEQ_CST);
144 -
145 - return (STORAGE_METRIC_HANDLE *)page_index;
117 + return (STORAGE_METRIC_HANDLE *) mrg_metric_get_and_acquire(main_mrg, uuid, (Word_t) ctx);
118 }
119
148 -STORAGE_METRIC_HANDLE *rrdeng_metric_create(STORAGE_INSTANCE *db_instance, uuid_t *uuid) {
120 +static METRIC *rrdeng_metric_create(STORAGE_INSTANCE *db_instance, uuid_t *uuid) {
121 internal_fatal(!db_instance, "DBENGINE: db_instance is NULL");
122
123 struct rrdengine_instance *ctx = (struct rrdengine_instance *)db_instance;
152 - struct pg_cache_page_index *page_index;
153 - struct page_cache *pg_cache = &ctx->pg_cache;
154 -
155 - uv_rwlock_wrlock(&pg_cache->metrics_index.lock);
156 - Pvoid_t *PValue = JudyHSIns(&pg_cache->metrics_index.JudyHS_array, uuid, sizeof(uuid_t), PJE0);
157 - fatal_assert(NULL == *PValue); /* TODO: figure out concurrency model */
158 - *PValue = page_index = create_page_index(uuid, ctx);
159 - page_index->prev = pg_cache->metrics_index.last_page_index;
160 - pg_cache->metrics_index.last_page_index = page_index;
161 - page_index->refcount = 1;
162 - uv_rwlock_wrunlock(&pg_cache->metrics_index.lock);
163 -
164 - return (STORAGE_METRIC_HANDLE *)page_index;
124 + MRG_ENTRY entry = {
125 + .section = (Word_t)ctx,
126 + .first_time_s = 0,
127 + .last_time_s = 0,
128 + .latest_update_every_s = 0,
129 + };
130 + uuid_copy(entry.uuid, *uuid);
131 +
132 + METRIC *metric = mrg_metric_add_and_acquire(main_mrg, entry, NULL);
133 + return metric;
134 }
135
136 STORAGE_METRIC_HANDLE *rrdeng_metric_get_or_create(RRDDIM *rd, STORAGE_INSTANCE *db_instance) {
168 - STORAGE_METRIC_HANDLE *db_metric_handle;
169 -
170 - db_metric_handle = rrdeng_metric_get(db_instance, &rd->metric_uuid);
171 - if(!db_metric_handle) {
172 - db_metric_handle = rrdeng_metric_get_legacy(db_instance, rrddim_id(rd), rrdset_id(rd->rrdset));
173 - if(db_metric_handle) {
174 - struct pg_cache_page_index *page_index = (struct pg_cache_page_index *)db_metric_handle;
175 - uuid_copy(rd->metric_uuid, page_index->id);
176 - }
137 + struct rrdengine_instance *ctx = (struct rrdengine_instance *)db_instance;
138 + METRIC *metric;
139 +
140 + metric = mrg_metric_get_and_acquire(main_mrg, &rd->metric_uuid, (Word_t) ctx);
141 + if(!metric) {
142 + metric = rrdeng_metric_get_legacy(db_instance, rrddim_id(rd), rrdset_id(rd->rrdset));
143 + if(metric)
144 + uuid_copy(rd->metric_uuid, *mrg_metric_uuid(main_mrg, metric));
145 }
178 - if(!db_metric_handle)
179 - db_metric_handle = rrdeng_metric_create(db_instance, &rd->metric_uuid);
146 +
147 + if(!metric)
148 + metric = rrdeng_metric_create(db_instance, &rd->metric_uuid);
149
150 #ifdef NETDATA_INTERNAL_CHECKS
182 - struct pg_cache_page_index *page_index = (struct pg_cache_page_index *)db_metric_handle;
183 - if(uuid_compare(rd->metric_uuid, page_index->id) != 0) {
151 + if(uuid_compare(rd->metric_uuid, *mrg_metric_uuid(main_mrg, metric)) != 0) {
152 char uuid1[UUID_STR_LEN + 1];
153 char uuid2[UUID_STR_LEN + 1];
154
155 uuid_unparse(rd->metric_uuid, uuid1);
188 - uuid_unparse(page_index->id, uuid2);
189 - fatal("DBENGINE: uuids do not match, asked for metric '%s', but got page_index of metric '%s'", uuid1, uuid2);
156 + uuid_unparse(*mrg_metric_uuid(main_mrg, metric), uuid2);
157 + fatal("DBENGINE: uuids do not match, asked for metric '%s', but got metric '%s'", uuid1, uuid2);
158 }
159
192 - struct rrdengine_instance *ctx = (struct rrdengine_instance *)db_instance;
193 - if(page_index->ctx != ctx)
194 - fatal("DBENGINE: mixed up rrdengine instances, asked for metric from %p, got from %p", ctx, page_index->ctx);
160 + if(mrg_metric_ctx(metric) != ctx)
161 + fatal("DBENGINE: mixed up db instances, asked for metric from %p, got from %p",
162 + ctx, mrg_metric_ctx(metric));
163 #endif
164
197 - return db_metric_handle;
165 + return (STORAGE_METRIC_HANDLE *)metric;
166 }
167
168
@@ -206,45 +174,43 @@ STORAGE_METRIC_HANDLE *rrdeng_metric_get_or_create(RRDDIM *rd, STORAGE_INSTANCE
174 * The handle must be released with rrdeng_store_metric_final().
175 */
176 STORAGE_COLLECT_HANDLE *rrdeng_store_metric_init(STORAGE_METRIC_HANDLE *db_metric_handle, uint32_t update_every, STORAGE_METRICS_GROUP *smg) {
209 - struct pg_cache_page_index *page_index = (struct pg_cache_page_index *)db_metric_handle;
177 + METRIC *metric = mrg_metric_dup(main_mrg, (METRIC *)db_metric_handle);
178 struct rrdeng_collect_handle *handle;
179
180 handle = callocz(1, sizeof(struct rrdeng_collect_handle));
213 - handle->page_index = page_index;
214 - handle->descr = NULL;
215 - handle->unaligned_page = 0;
216 - page_index->latest_update_every_s = update_every;
181 + handle->metric = metric;
182 + handle->page = NULL;
183 + handle->page_position = 0;
184 + handle->page_entries_max = 0;
185 + handle->update_every_ut = update_every * USEC_PER_SEC;
186 +
187 + mrg_metric_set_update_every(main_mrg, metric, update_every);
188
189 handle->alignment = (struct pg_alignment *)smg;
190 rrdeng_page_alignment_acquire(handle->alignment);
191
221 - uv_rwlock_wrlock(&page_index->lock);
222 - ++page_index->writers;
223 - uv_rwlock_wrunlock(&page_index->lock);
224 -
192 return (STORAGE_COLLECT_HANDLE *)handle;
193 }
194
195 /* The page must be populated and referenced */
229 -static int page_has_only_empty_metrics(struct rrdeng_page_descr *descr)
230 -{
231 - switch(descr->type) {
196 +static bool page_has_only_empty_metrics(struct rrdeng_collect_handle *handle) {
197 + switch(handle->type) {
198 case PAGE_METRICS: {
233 - size_t slots = descr->page_length / PAGE_POINT_SIZE_BYTES(descr);
234 - storage_number *array = (storage_number *)descr->pg_cache_descr->page;
199 + size_t slots = handle->page_position;
200 + storage_number *array = (storage_number *)pgc_page_data(handle->page);
201 for (size_t i = 0 ; i < slots; ++i) {
202 if(does_storage_number_exist(array[i]))
237 - return 0;
203 + return false;
204 }
205 }
206 break;
207
208 case PAGE_TIER: {
243 - size_t slots = descr->page_length / PAGE_POINT_SIZE_BYTES(descr);
244 - storage_number_tier1_t *array = (storage_number_tier1_t *)descr->pg_cache_descr->page;
209 + size_t slots = handle->page_position;
210 + storage_number_tier1_t *array = (storage_number_tier1_t *)pgc_page_data(handle->page);
211 for (size_t i = 0 ; i < slots; ++i) {
212 if(fpclassify(array[i].sum_value) != FP_NAN)
247 - return 0;
213 + return false;
214 }
215 }
216 break;
@@ -252,43 +218,118 @@ static int page_has_only_empty_metrics(struct rrdeng_page_descr *descr)
218 default: {
219 static bool logged = false;
220 if(!logged) {
255 - error("DBENGINE: cannot check page for nulls on unknown page type id %d", descr->type);
221 + error("DBENGINE: cannot check page for nulls on unknown page type id %d", (mrg_metric_ctx(handle->metric))->page_type);
222 logged = true;
223 }
258 - return 0;
224 + return false;
225 }
226 }
227
262 - return 1;
228 + return true;
229 }
230
231 void rrdeng_store_metric_flush_current_page(STORAGE_COLLECT_HANDLE *collection_handle) {
232 struct rrdeng_collect_handle *handle = (struct rrdeng_collect_handle *)collection_handle;
267 - // struct rrdeng_metric_handle *metric_handle = (struct rrdeng_metric_handle *)handle->metric_handle;
268 - struct rrdengine_instance *ctx = handle->page_index->ctx;
269 - struct rrdeng_page_descr *descr = handle->descr;
270 -
271 - if (unlikely(!ctx)) return;
272 - if (unlikely(!descr)) return;
273 -
274 - if (likely(descr->page_length)) {
275 - int page_is_empty;
276 -
277 - rrd_stat_atomic_add(&ctx->stats.metric_API_producers, -1);
278 -
279 - page_is_empty = page_has_only_empty_metrics(descr);
280 - if (page_is_empty) {
281 - print_page_cache_descr(descr, "Page has empty metrics only, deleting", true);
282 - pg_cache_put(ctx, descr);
283 - pg_cache_punch_hole(ctx, descr, 1, 0, NULL);
284 - } else
285 - rrdeng_commit_page(ctx, descr, handle->page_correlation_id);
286 - } else {
287 - dbengine_page_free(descr->pg_cache_descr->page);
288 - rrdeng_destroy_pg_cache_descr(ctx, descr->pg_cache_descr);
289 - rrdeng_page_descr_freez(descr);
233 +
234 + if (unlikely(!handle->page))
235 + return;
236 +
237 + if(!handle->page_position || page_has_only_empty_metrics(handle))
238 + pgc_page_to_clean_evict_or_release(main_cache, handle->page);
239 +
240 + else {
241 + mrg_metric_set_clean_latest_time_s(main_mrg, handle->metric, pgc_page_end_time_s(handle->page));
242 + pgc_page_hot_to_dirty_and_release(main_cache, handle->page);
243 + }
244 +
245 + mrg_metric_set_hot_latest_time_s(main_mrg, handle->metric, 0);
246 +
247 + handle->page = NULL;
248 + handle->page_position = 0;
249 + handle->page_entries_max = 0;
250 +
251 + internal_fatal((time_t)(handle->update_every_ut / USEC_PER_SEC) != mrg_metric_get_update_every_s(main_mrg, handle->metric),
252 + "DBENGINE: the collection handle update every and the metric registry update every are not the same");
253 +}
254 +
255 +static void rrdeng_store_metric_create_new_page(struct rrdeng_collect_handle *handle, struct rrdengine_instance *ctx, usec_t point_in_time_ut, void *data, size_t data_size) {
256 +time_t point_in_time_s = (time_t)(point_in_time_ut / USEC_PER_SEC);
257 + time_t update_every_s = (time_t)(handle->update_every_ut / USEC_PER_SEC);
258 +
259 + PGC_ENTRY page_entry = {
260 + .section = (Word_t) ctx,
261 + .metric_id = mrg_metric_id(main_mrg, handle->metric),
262 + .start_time_s = point_in_time_s,
263 + .end_time_s = point_in_time_s,
264 + .size = data_size,
265 + .data = data,
266 + .update_every_s = update_every_s,
267 + .hot = true
268 + };
269 +
270 + bool added = true;
271 + PGC_PAGE *page = pgc_page_add_and_acquire(main_cache, page_entry, &added);
272 + if (unlikely(!added)) {
273 + internal_fatal(!pgc_is_page_hot(page),
274 + "DBENGINE CACHE: requested to add a hot page to the main cache, "
275 + "but the page returned is not hot");
276 +
277 + if(unlikely(pgc_page_data_size(main_cache, page) < PAGE_POINT_CTX_SIZE_BYTES(ctx)))
278 + fatal("DBENGINE: hot page returned from main cache does not have the size for storing 1 point");
279 +
280 + // copy the point in data
281 + memcpy(pgc_page_data(page), data, PAGE_POINT_CTX_SIZE_BYTES(ctx));
282 +
283 + // free data
284 + dbengine_page_free(page_entry.data);
285 +
286 + handle->page_entries_max = pgc_page_data_size(main_cache, page) / PAGE_POINT_CTX_SIZE_BYTES(ctx);
287 + }
288 + else
289 + handle->page_entries_max = data_size / PAGE_POINT_CTX_SIZE_BYTES(ctx);
290 +
291 + handle->page_end_time_ut = point_in_time_ut;
292 + handle->page_position = 1; // zero is already in our data
293 + handle->page = page;
294 +}
295 +
296 +static void *rrdeng_alloc_new_metric_data(struct rrdeng_collect_handle *handle, size_t *data_size) {
297 + struct rrdengine_instance *ctx = mrg_metric_ctx(handle->metric);
298 + size_t size;
299 +
300 + if(handle->options & RRDENG_FIRST_PAGE_ALLOCATED) {
301 + // any page except the first
302 + size = tier_page_size[ctx->tier];
303 + }
304 + else {
305 + // the first page
306 + handle->options |= RRDENG_FIRST_PAGE_ALLOCATED;
307 + size_t max_size = tier_page_size[ctx->tier];
308 + size_t max_slots = max_size / PAGE_POINT_CTX_SIZE_BYTES(ctx);
309 + size_t min_slots = max_slots / 5;
310 + size_t distribution = max_slots - min_slots;
311 + size_t this_page_end_slot = indexing_partition((Word_t)handle->alignment, distribution);
312 +
313 + size_t current_end_slot = (size_t)now_monotonic_sec() % distribution;
314 +
315 + if(current_end_slot < this_page_end_slot)
316 + this_page_end_slot -= current_end_slot;
317 + else if(current_end_slot > this_page_end_slot)
318 + this_page_end_slot = (max_slots - current_end_slot) + this_page_end_slot;
319 +
320 + size_t final_slots = min_slots + this_page_end_slot;
321 +
322 + if(final_slots > max_slots)
323 + final_slots = max_slots;
324 +
325 + if(final_slots < min_slots)
326 + final_slots = min_slots;
327 +
328 + size = final_slots * PAGE_POINT_CTX_SIZE_BYTES(ctx);
329 }
291 - handle->descr = NULL;
330 +
331 + *data_size = size;
332 + return dbengine_page_alloc(ctx, size);
333 }
334
335 static void rrdeng_store_metric_next_internal(STORAGE_COLLECT_HANDLE *collection_handle,
@@ -301,137 +342,94 @@ static void rrdeng_store_metric_next_internal(STORAGE_COLLECT_HANDLE *collection
342 SN_FLAGS flags)
343 {
344 struct rrdeng_collect_handle *handle = (struct rrdeng_collect_handle *)collection_handle;
304 - struct pg_cache_page_index *page_index = handle->page_index;
305 - struct rrdengine_instance *ctx = handle->page_index->ctx;
306 - struct page_cache *pg_cache = &ctx->pg_cache;
307 - struct rrdeng_page_descr *descr = handle->descr;
345 + struct rrdengine_instance *ctx = mrg_metric_ctx(handle->metric);
346
309 - void *page;
310 - uint8_t must_flush_unaligned_page = 0, perfect_page_alignment = 0;
347 + bool perfect_page_alignment = false;
348 + void *data;
349 + size_t data_size;
350
312 - if (descr) {
351 + if(likely(handle->page)) {
352 /* Make alignment decisions */
314 -
315 -#ifdef NETDATA_INTERNAL_CHECKS
316 - if(descr->end_time_ut + page_index->latest_update_every_s * USEC_PER_SEC != point_in_time_ut) {
317 - char buffer[200 + 1];
318 - snprintfz(buffer, 200,
319 - "metrics collected are %s, end_time_ut = %llu, point_in_time_ut = %llu, update_every = %u, delta = %llu",
320 - (point_in_time_ut / USEC_PER_SEC - descr->end_time_ut / USEC_PER_SEC > page_index->latest_update_every_s)?"far apart":"not aligned",
321 - descr->end_time_ut / USEC_PER_SEC,
322 - point_in_time_ut / USEC_PER_SEC,
323 - page_index->latest_update_every_s,
324 - point_in_time_ut / USEC_PER_SEC - descr->end_time_ut / USEC_PER_SEC);
325 - print_page_cache_descr(descr, buffer, false);
326 - }
327 -#endif
328 -
329 - if (descr->page_length == handle->alignment->page_length) {
353 + if (handle->page_position == handle->alignment->page_position) {
354 /* this is the leading dimension that defines chart alignment */
331 - perfect_page_alignment = 1;
355 + perfect_page_alignment = true;
356 }
357 +
358 /* is the metric far enough out of alignment with the others? */
334 - if (unlikely(descr->page_length + PAGE_POINT_SIZE_BYTES(descr) < handle->alignment->page_length)) {
335 - handle->unaligned_page = 1;
336 - print_page_cache_descr(descr, "Metric page is not aligned with chart", true);
337 - }
338 - if (unlikely(handle->unaligned_page &&
359 + if (unlikely(handle->page_position + 1 < handle->alignment->page_position))
360 + handle->options |= RRDENG_CHO_UNALIGNED;
361 +
362 + if (unlikely((handle->options & RRDENG_CHO_UNALIGNED) &&
363 /* did the other metrics change page? */
340 - handle->alignment->page_length <= PAGE_POINT_SIZE_BYTES(descr))) {
341 - print_page_cache_descr(descr, "must_flush_unaligned_page = 1", true);
342 - must_flush_unaligned_page = 1;
343 - handle->unaligned_page = 0;
344 - }
345 - }
346 - if (unlikely(NULL == descr ||
347 - descr->page_length + PAGE_POINT_SIZE_BYTES(descr) > RRDENG_BLOCK_SIZE ||
348 - must_flush_unaligned_page)) {
364 + handle->alignment->page_position <= 1)) {
365 + handle->options &= ~RRDENG_CHO_UNALIGNED;
366
350 - if(descr) {
351 - print_page_cache_descr(descr, "flushing metric", true);
367 rrdeng_store_metric_flush_current_page(collection_handle);
353 - }
354 -
355 - page = rrdeng_create_page(ctx, &page_index->id, &descr);
356 - fatal_assert(page);
357 -
358 - descr->update_every_s = page_index->latest_update_every_s;
359 - handle->descr = descr;
368
361 - handle->page_correlation_id = rrd_atomic_fetch_add(&pg_cache->committed_page_index.latest_corr_id, 1);
362 -
363 - if (0 == handle->alignment->page_length) {
364 - /* this is the leading dimension that defines chart alignment */
365 - perfect_page_alignment = 1;
369 + data = rrdeng_alloc_new_metric_data(handle, &data_size);
370 + }
371 + else {
372 + data = pgc_page_data(handle->page);
373 + data_size = pgc_page_data_size(main_cache, handle->page);
374 }
375 }
376 + else
377 + data = rrdeng_alloc_new_metric_data(handle, &data_size);
378
369 - page = descr->pg_cache_descr->page;
370 -
371 - switch (descr->type) {
379 + switch (ctx->page_type) {
380 case PAGE_METRICS: {
373 - ((storage_number *)page)[descr->page_length / PAGE_POINT_SIZE_BYTES(descr)] = pack_storage_number(n, flags);
381 + storage_number *tier0_metric_data = data;
382 + tier0_metric_data[handle->page_position] = pack_storage_number(n, flags);
383 }
384 break;
385
386 case PAGE_TIER: {
387 + storage_number_tier1_t *tier12_metric_data = data;
388 storage_number_tier1_t number_tier1;
389 number_tier1.sum_value = (float)n;
390 number_tier1.min_value = (float)min_value;
391 number_tier1.max_value = (float)max_value;
392 number_tier1.anomaly_count = anomaly_count;
393 number_tier1.count = count;
384 - ((storage_number_tier1_t *)page)[descr->page_length / PAGE_POINT_SIZE_BYTES(descr)] = number_tier1;
394 + tier12_metric_data[handle->page_position] = number_tier1;
395 }
396 break;
397
398 default: {
399 static bool logged = false;
400 if(!logged) {
391 - error("DBENGINE: cannot store metric on unknown page type id %d", descr->type);
401 + error("DBENGINE: cannot store metric on unknown page type id %d", ctx->page_type);
402 logged = true;
403 }
404 }
405 break;
406 }
407
398 - pg_cache_atomic_set_pg_info(descr, point_in_time_ut, descr->page_length + PAGE_POINT_SIZE_BYTES(descr));
408 + if(unlikely(!handle->page)){
409 + rrdeng_store_metric_create_new_page(handle, ctx, point_in_time_ut, data, data_size);
410 + // handle->position is set to 1 already
411
400 - if (perfect_page_alignment)
401 - handle->alignment->page_length = descr->page_length;
402 - if (unlikely(INVALID_TIME == descr->start_time_ut)) {
403 - unsigned long new_metric_API_producers, old_metric_API_max_producers, ret_metric_API_max_producers;
404 - descr->start_time_ut = point_in_time_ut;
405 -
406 - new_metric_API_producers = rrd_atomic_add_fetch(&ctx->stats.metric_API_producers, 1);
407 - while (unlikely(new_metric_API_producers > (old_metric_API_max_producers = ctx->metric_API_max_producers))) {
408 - /* Increase ctx->metric_API_max_producers */
409 - ret_metric_API_max_producers = ulong_compare_and_swap(&ctx->metric_API_max_producers,
410 - old_metric_API_max_producers,
411 - new_metric_API_producers);
412 - if (old_metric_API_max_producers == ret_metric_API_max_producers) {
413 - /* success */
414 - break;
415 - }
412 + if (0 == handle->alignment->page_position) {
413 + /* this is the leading dimension that defines chart alignment */
414 + perfect_page_alignment = true;
415 }
416 + }
417 + else {
418 + // update an existing page
419 + pgc_page_hot_set_end_time_s(main_cache, handle->page, (time_t) (point_in_time_ut / USEC_PER_SEC));
420 + handle->page_end_time_ut = point_in_time_ut;
421
418 - pg_cache_insert(ctx, page_index, descr);
419 - } else {
420 - pg_cache_add_new_metric_time(page_index, descr);
422 + if(unlikely(++handle->page_position >= handle->page_entries_max)) {
423 + internal_fatal(handle->page_position > handle->page_entries_max, "DBENGINE: exceeded page max number of points");
424 + rrdeng_store_metric_flush_current_page(collection_handle);
425 + }
426 }
427
423 -// {
424 -// unsigned char u[16] = { 0x0C, 0x0A, 0x40, 0xD6, 0x2A, 0x43, 0x4A, 0x7C, 0x95, 0xF7, 0xD1, 0x1E, 0x0C, 0x9E, 0x8A, 0xE7 };
425 -// if(uuid_compare(u, page_index->id) == 0) {
426 -// char buffer[100];
427 -// snprintfz(buffer, 100, "store system.cpu, collect:%u, page_index first:%u, last:%u",
428 -// (uint32_t)(point_in_time / USEC_PER_SEC),
429 -// (uint32_t)(page_index->oldest_time / USEC_PER_SEC),
430 -// (uint32_t)(page_index->latest_time / USEC_PER_SEC));
431 -//
432 -// print_page_cache_descr(descr, buffer, false);
433 -// }
434 -// }
428 + if (perfect_page_alignment)
429 + handle->alignment->page_position = handle->page_position;
430 +
431 + // update the metric information
432 + mrg_metric_set_hot_latest_time_s(main_mrg, handle->metric, (time_t) (point_in_time_ut / USEC_PER_SEC));
433 }
434
435 void rrdeng_store_metric_next(STORAGE_COLLECT_HANDLE *collection_handle,
@@ -444,230 +442,207 @@ void rrdeng_store_metric_next(STORAGE_COLLECT_HANDLE *collection_handle,
442 SN_FLAGS flags)
443 {
444 struct rrdeng_collect_handle *handle = (struct rrdeng_collect_handle *)collection_handle;
447 - struct pg_cache_page_index *page_index = handle->page_index;
448 - struct rrdeng_page_descr *descr = handle->descr;
449 -
450 - if(likely(descr)) {
451 - usec_t last_point_in_time_ut = descr->end_time_ut;
452 - usec_t update_every_ut = page_index->latest_update_every_s * USEC_PER_SEC;
453 - size_t points_gap = (point_in_time_ut <= last_point_in_time_ut) ?
454 - (size_t)0 :
455 - (size_t)((point_in_time_ut - last_point_in_time_ut) / update_every_ut);
456 -
457 - if(unlikely(points_gap != 1)) {
458 - if (unlikely(points_gap <= 0)) {
459 - time_t now = now_realtime_sec();
460 - static __thread size_t counter = 0;
461 - static __thread time_t last_time_logged = 0;
462 - counter++;
463 -
464 - if(now - last_time_logged > 600) {
465 - error("DBENGINE: collected point is in the past (repeated %zu times in the last %zu secs). Ignoring these data collection points.",
466 - counter, (size_t)(last_time_logged?(now - last_time_logged):0));
467 -
468 - last_time_logged = now;
469 - counter = 0;
470 - }
471 - return;
472 - }
445
474 - size_t point_size = PAGE_POINT_SIZE_BYTES(descr);
475 - size_t page_size_in_points = RRDENG_BLOCK_SIZE / point_size;
476 - size_t used_points = descr->page_length / point_size;
477 - size_t remaining_points_in_page = page_size_in_points - used_points;
446 + if(likely(handle->page_end_time_ut + handle->update_every_ut == point_in_time_ut)) {
447 + // happy path
448 + ;
449 + }
450 + else if(unlikely(point_in_time_ut < handle->page_end_time_ut)) {
451 + error_limit_static_global_var(erl, 1, 0);
452 + error_limit(&erl, "DBENGINE: new point at %llu is older than the last collected %llu, ignoring it",
453 + point_in_time_ut, handle->page_end_time_ut);
454 + return;
455 + }
456
479 - bool new_point_is_aligned = true;
480 - if(unlikely((point_in_time_ut - last_point_in_time_ut) / points_gap != update_every_ut))
481 - new_point_is_aligned = false;
457 + else if(unlikely(point_in_time_ut == handle->page_end_time_ut)) {
458 + error_limit_static_global_var(erl, 1, 0);
459 + error_limit(&erl, "DBENGINE: new point time %llu has the same timestamp to the last collected point, ignoring it",
460 + point_in_time_ut);
461 + return;
462 + }
463
483 - if(unlikely(points_gap > remaining_points_in_page || !new_point_is_aligned)) {
484 -// char buffer[200];
485 -// snprintfz(buffer, 200, "data collection skipped %zu points, last stored point %llu, new point %llu, update every %d. Cutting page.",
486 -// points_gap, last_point_in_time_ut / USEC_PER_SEC, point_in_time_ut / USEC_PER_SEC, page_index->latest_update_every_s);
487 -// print_page_cache_descr(descr, buffer, false);
464 + else if(handle->page) {
465 + size_t points_gap = (point_in_time_ut - handle->page_end_time_ut) / handle->update_every_ut;
466 + size_t page_remaining_points = handle->page_entries_max - handle->page_position;
467
489 - rrdeng_store_metric_flush_current_page(collection_handle);
490 - }
491 - else {
492 -// char buffer[200];
493 -// snprintfz(buffer, 200, "data collection skipped %zu points, last stored point %llu, new point %llu, update every %d. Filling the gap.",
494 -// points_gap, last_point_in_time_ut / USEC_PER_SEC, point_in_time_ut / USEC_PER_SEC, page_index->latest_update_every_s);
495 -// print_page_cache_descr(descr, buffer, false);
496 -
497 - // loop to fill the gap
498 - usec_t step_ut = page_index->latest_update_every_s * USEC_PER_SEC;
499 - usec_t last_point_filled_ut = last_point_in_time_ut + step_ut;
500 -
501 - while (last_point_filled_ut < point_in_time_ut) {
502 - rrdeng_store_metric_next_internal(
503 - collection_handle, last_point_filled_ut, NAN, NAN, NAN,
468 + if(points_gap > page_remaining_points)
469 + rrdeng_store_metric_flush_current_page(collection_handle);
470 + else {
471 + // loop to fill the gap
472 + usec_t last_point_filled_ut = handle->page_end_time_ut + handle->update_every_ut;
473 +
474 + while (last_point_filled_ut < point_in_time_ut) {
475 + rrdeng_store_metric_next_internal(
476 + collection_handle, last_point_filled_ut,
477 + NAN, NAN, NAN,
478 1, 0, SN_EMPTY_SLOT);
479
506 - last_point_filled_ut += step_ut;
507 - }
480 + last_point_filled_ut += handle->update_every_ut;
481 }
482 }
483 }
484
485 + internal_fatal((time_t)(handle->update_every_ut / USEC_PER_SEC) !=
486 + mrg_metric_get_update_every_s(main_mrg, handle->metric),
487 + "DBENGINE: the collection handle update every and the metric registry update every are not the same");
488 +
489 +// FIXME - is this a problem?
490 +// internal_fatal((point_in_time_ut - handle->page_end_time_ut) % handle->update_every_ut,
491 +// "DBENGINE: new point is not aligned to update every");
492 +
493 rrdeng_store_metric_next_internal(collection_handle, point_in_time_ut, n, min_value, max_value, count, anomaly_count, flags);
494 }
495
515 -
496 /*
497 * Releases the database reference from the handle for storing metrics.
498 * Returns 1 if it's safe to delete the dimension.
499 */
500 int rrdeng_store_metric_finalize(STORAGE_COLLECT_HANDLE *collection_handle) {
501 struct rrdeng_collect_handle *handle = (struct rrdeng_collect_handle *)collection_handle;
522 - struct pg_cache_page_index *page_index = handle->page_index;
523 -
524 - uint8_t can_delete_metric = 0;
502
503 rrdeng_store_metric_flush_current_page(collection_handle);
527 - uv_rwlock_wrlock(&page_index->lock);
528 -
529 - if (!--page_index->writers && !page_index->page_count)
530 - can_delete_metric = 1;
531 -
532 - uv_rwlock_wrunlock(&page_index->lock);
533 -
504 rrdeng_page_alignment_release(handle->alignment);
505 + mrg_metric_release(main_mrg, handle->metric);
506 freez(handle);
507
537 - return can_delete_metric;
508 + return 0;
509 }
510
511 void rrdeng_store_metric_change_collection_frequency(STORAGE_COLLECT_HANDLE *collection_handle, int update_every) {
512 struct rrdeng_collect_handle *handle = (struct rrdeng_collect_handle *)collection_handle;
542 - struct pg_cache_page_index *page_index = handle->page_index;
513 + METRIC *metric = handle->metric;
514 + usec_t update_every_ut = update_every * USEC_PER_SEC;
515 +
516 + internal_fatal((time_t)(handle->update_every_ut / USEC_PER_SEC) != mrg_metric_get_update_every_s(main_mrg, metric),
517 + "DBENGINE: the collection handle update every and the metric registry update every are not the same");
518 +
519 + if(update_every_ut == handle->update_every_ut)
520 + return;
521 +
522 rrdeng_store_metric_flush_current_page(collection_handle);
544 - uv_rwlock_rdlock(&page_index->lock);
545 - page_index->latest_update_every_s = update_every;
546 - uv_rwlock_rdunlock(&page_index->lock);
523 + mrg_metric_set_update_every(main_mrg, metric, update_every);
524 + handle->update_every_ut = update_every_ut;
525 }
526
527 // ----------------------------------------------------------------------------
528 // query ops
529
552 -//static inline uint32_t *pginfo_to_dt(struct rrdeng_page_info *page_info)
553 -//{
554 -// return (uint32_t *)&page_info->scratch[0];
555 -//}
556 -//
557 -//static inline uint32_t *pginfo_to_points(struct rrdeng_page_info *page_info)
558 -//{
559 -// return (uint32_t *)&page_info->scratch[sizeof(uint32_t)];
560 -//}
561 -//
530 +#ifdef NETDATA_INTERNAL_CHECKS
531 +SPINLOCK global_query_handle_spinlock = NETDATA_SPINLOCK_INITIALIZER;
532 +static struct rrdeng_query_handle *global_query_handle_ll = NULL;
533 +static void register_query_handle(struct rrdeng_query_handle *handle) {
534 + handle->query_pid = gettid();
535 + handle->started_time_s = now_realtime_sec();
536 +
537 + netdata_spinlock_lock(&global_query_handle_spinlock);
538 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(global_query_handle_ll, handle, prev, next);
539 + netdata_spinlock_unlock(&global_query_handle_spinlock);
540 +}
541 +static void unregister_query_handle(struct rrdeng_query_handle *handle) {
542 + netdata_spinlock_lock(&global_query_handle_spinlock);
543 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(global_query_handle_ll, handle, prev, next);
544 + netdata_spinlock_unlock(&global_query_handle_spinlock);
545 +}
546 +#else
547 +static void register_query_handle(struct rrdeng_query_handle *handle __maybe_unused) {
548 + ;
549 +}
550 +static void unregister_query_handle(struct rrdeng_query_handle *handle __maybe_unused) {
551 + ;
552 +}
553 +#endif
554 +
555 /*
556 * Gets a handle for loading metrics from the database.
557 * The handle must be released with rrdeng_load_metric_final().
558 */
566 -void rrdeng_load_metric_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *rrdimm_handle, time_t start_time_s, time_t end_time_s)
559 +void rrdeng_load_metric_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *rrddim_handle, time_t start_time_s, time_t end_time_s, STORAGE_PRIORITY priority)
560 {
568 - struct pg_cache_page_index *page_index = (struct pg_cache_page_index *)db_metric_handle;
569 - struct rrdengine_instance *ctx = page_index->ctx;
561 + usec_t started_ut = now_monotonic_usec();
562
571 - // fprintf(stderr, "%s: %s/%s start time %ld, end time %ld\n", __FUNCTION__ , rd->rrdset->name, rd->name, start_time, end_time);
563 + netdata_thread_disable_cancelability();
564
565 + METRIC *metric = (METRIC *)db_metric_handle;
566 + struct rrdengine_instance *ctx = mrg_metric_ctx(metric);
567 struct rrdeng_query_handle *handle;
574 - unsigned pages_nr;
568
576 - if(!page_index->latest_update_every_s)
577 - page_index->latest_update_every_s = default_rrd_update_every;
569 + mrg_metric_set_update_every_s_if_zero(main_mrg, metric, default_rrd_update_every);
570
579 - rrdimm_handle->start_time_s = start_time_s;
580 - rrdimm_handle->end_time_s = end_time_s;
571 + handle = rrdeng_query_handle_get();
572 + register_query_handle(handle);
573
582 - handle = callocz(1, sizeof(struct rrdeng_query_handle));
583 - handle->wanted_start_time_s = start_time_s;
584 - handle->now_s = start_time_s;
585 - handle->position = 0;
586 - handle->ctx = ctx;
587 - handle->descr = NULL;
588 - handle->dt_s = page_index->latest_update_every_s;
589 - rrdimm_handle->handle = (STORAGE_QUERY_HANDLE *)handle;
590 - pages_nr = pg_cache_preload(ctx, &page_index->id, start_time_s * USEC_PER_SEC, end_time_s * USEC_PER_SEC,
591 - NULL, &handle->page_index);
592 - if (unlikely(NULL == handle->page_index || 0 == pages_nr))
593 - // there are no metrics to load
594 - handle->wanted_start_time_s = INVALID_TIME;
595 -}
574 + if(unlikely(priority == STORAGE_PRIORITY_CRITICAL))
575 + // critical is reserved for dbengine internal use
576 + priority = STORAGE_PRIORITY_HIGH;
577
597 -static int rrdeng_load_page_next(struct storage_engine_query_handle *rrdimm_handle, bool debug_this __maybe_unused) {
598 - struct rrdeng_query_handle *handle = (struct rrdeng_query_handle *)rrdimm_handle->handle;
578 + handle->ctx = ctx;
579 + handle->metric = metric;
580 + handle->start_time_s = start_time_s;
581 + handle->end_time_s = end_time_s;
582 + handle->priority = priority;
583 + handle->now_s = start_time_s;
584
600 - struct rrdengine_instance *ctx = handle->ctx;
601 - struct rrdeng_page_descr *descr = handle->descr;
585 + handle->dt_s = mrg_metric_get_update_every_s(main_mrg, metric);
586 + if(!handle->dt_s)
587 + handle->dt_s = default_rrd_update_every;
588
603 - uint32_t page_length;
604 - usec_t page_end_time_ut;
605 - unsigned position;
589 + rrddim_handle->handle = (STORAGE_QUERY_HANDLE *)handle;
590 + rrddim_handle->start_time_s = start_time_s;
591 + rrddim_handle->end_time_s = end_time_s;
592 + rrddim_handle->priority = priority;
593
607 - if (likely(descr)) {
608 - // Drop old page's reference
594 + pg_cache_preload(handle);
595
610 -#ifdef NETDATA_INTERNAL_CHECKS
611 - rrd_stat_atomic_add(&ctx->stats.metric_API_consumers, -1);
612 -#endif
596 + __atomic_add_fetch(&rrdeng_cache_efficiency_stats.query_time_init, now_monotonic_usec() - started_ut, __ATOMIC_RELAXED);
597 +}
598
614 - pg_cache_put(ctx, descr);
615 - handle->descr = NULL;
616 - handle->wanted_start_time_s = (time_t)((handle->page_end_time_ut / USEC_PER_SEC) + handle->dt_s);
599 +static bool rrdeng_load_page_next(struct storage_engine_query_handle *rrddim_handle, bool debug_this __maybe_unused) {
600 + struct rrdeng_query_handle *handle = (struct rrdeng_query_handle *)rrddim_handle->handle;
601 + struct rrdengine_instance *ctx = handle->ctx;
602
618 - if (unlikely(handle->wanted_start_time_s > rrdimm_handle->end_time_s))
619 - return 1;
603 + if (likely(handle->page)) {
604 + // we have a page to release
605 + pgc_page_release(main_cache, handle->page);
606 + handle->page = NULL;
607 }
608
622 - usec_t wanted_start_time_ut = handle->wanted_start_time_s * USEC_PER_SEC;
623 - descr = pg_cache_lookup_next(ctx, handle->page_index, &handle->page_index->id,
624 - wanted_start_time_ut, rrdimm_handle->end_time_s * USEC_PER_SEC);
625 - if (NULL == descr)
626 - return 1;
609 + if (unlikely(handle->now_s > rrddim_handle->end_time_s))
610 + return false;
611
628 -#ifdef NETDATA_INTERNAL_CHECKS
629 - rrd_stat_atomic_add(&ctx->stats.metric_API_consumers, 1);
630 -#endif
612 + size_t entries;
613 + handle->page = pg_cache_lookup_next(ctx, handle->pdc, handle->now_s, handle->dt_s, &entries);
614 + if (unlikely(!handle->page))
615 + return false;
616
632 - handle->descr = descr;
633 - pg_cache_atomic_get_pg_info(descr, &page_end_time_ut, &page_length);
634 - if (unlikely(INVALID_TIME == descr->start_time_ut || INVALID_TIME == page_end_time_ut || 0 == descr->update_every_s)) {
635 - error("DBENGINE: discarding invalid page descriptor (start_time = %llu, end_time = %llu, update_every_s = %d)",
636 - descr->start_time_ut, page_end_time_ut, descr->update_every_s);
637 - return 1;
638 - }
617 + time_t page_start_time_s = pgc_page_start_time_s(handle->page);
618 + time_t page_end_time_s = pgc_page_end_time_s(handle->page);
619 + time_t page_update_every_s = pgc_page_update_every_s(handle->page);
620
640 - if (unlikely(descr->start_time_ut != page_end_time_ut && wanted_start_time_ut > descr->start_time_ut)) {
641 - // we're in the middle of the page somewhere
642 - unsigned entries = page_length / PAGE_POINT_SIZE_BYTES(descr);
643 - position = ((uint64_t)(wanted_start_time_ut - descr->start_time_ut)) * (entries - 1) /
644 - (page_end_time_ut - descr->start_time_ut);
621 + unsigned position;
622 + if(likely(handle->now_s >= page_start_time_s && handle->now_s <= page_end_time_s)) {
623 +
624 + if(unlikely(entries == 1 || page_start_time_s == page_end_time_s))
625 + position = 0;
626 + else
627 + position = (handle->now_s - page_start_time_s) * (entries - 1) / (page_end_time_s - page_start_time_s);
628 +
629 + internal_fatal(position >= entries, "DBENGINE: wrong page position calculation");
630 }
646 - else
631 + else if(handle->now_s < page_start_time_s) {
632 + handle->now_s = page_start_time_s;
633 position = 0;
634 + }
635 + else {
636 + internal_fatal(true, "DBENGINE: this page is entirely in our past and should not be accepted for this query in the first place");
637 + handle->now_s = page_end_time_s;
638 + position = entries - 1;
639 + }
640
649 - handle->page_end_time_ut = page_end_time_ut;
650 - handle->page_length = page_length;
651 - handle->entries = page_length / PAGE_POINT_SIZE_BYTES(descr);
652 - handle->page = descr->pg_cache_descr->page;
653 - handle->dt_s = descr->update_every_s;
641 + handle->entries = entries;
642 handle->position = position;
655 -
656 -// if(debug_this)
657 -// info("DBENGINE: rrdeng_load_page_next(), "
658 -// "position:%d, "
659 -// "start_time_ut:%llu, "
660 -// "page_end_time_ut:%llu, "
661 -// "next_page_time_ut:%llu, "
662 -// "in_out:%s"
663 -// , position
664 -// , descr->start_time_ut
665 -// , page_end_time_ut
666 -// ,
667 -// wanted_start_time_ut, in_out?"true":"false"
668 -// );
669 -
670 - return 0;
643 + handle->metric_data = pgc_page_data((PGC_PAGE *)handle->page);
644 + handle->dt_s = page_update_every_s;
645 + return true;
646 }
647
648 // Returns the metric and sets its timestamp into current_time
@@ -675,75 +650,28 @@ static int rrdeng_load_page_next(struct storage_engine_query_handle *rrdimm_hand
650 // IT IS REQUIRED TO **ALWAYS** KEEP TRACK OF TIME, EVEN OUTSIDE THE DATABASE BOUNDARIES
651 STORAGE_POINT rrdeng_load_metric_next(struct storage_engine_query_handle *rrddim_handle) {
652 struct rrdeng_query_handle *handle = (struct rrdeng_query_handle *)rrddim_handle->handle;
678 - // struct rrdeng_metric_handle *metric_handle = handle->metric_handle;
679 -
680 - struct rrdeng_page_descr *descr = handle->descr;
681 - time_t now = handle->now_s + handle->dt_s;
682 -
683 -// bool debug_this = false;
684 -// {
685 -// unsigned char u[16] = { 0x0C, 0x0A, 0x40, 0xD6, 0x2A, 0x43, 0x4A, 0x7C, 0x95, 0xF7, 0xD1, 0x1E, 0x0C, 0x9E, 0x8A, 0xE7 };
686 -// if(uuid_compare(u, handle->page_index->id) == 0) {
687 -// char buffer[100];
688 -// snprintfz(buffer, 100, "load system.cpu, now:%u, dt:%u, position:%u page_index first:%u, last:%u",
689 -// (uint32_t)(now),
690 -// (uint32_t)(handle->dt_s),
691 -// (uint32_t)(handle->position),
692 -// (uint32_t)(handle->page_index->oldest_time / USEC_PER_SEC),
693 -// (uint32_t)(handle->page_index->latest_time / USEC_PER_SEC));
694 -//
695 -// print_page_cache_descr(descr, buffer, false);
696 -// debug_this = true;
697 -// }
698 -// }
699 -
653 STORAGE_POINT sp;
701 - unsigned position = handle->position + 1;
702 - storage_number_tier1_t tier1_value;
703 -
704 - if (unlikely(INVALID_TIME == handle->wanted_start_time_s)) {
705 - handle->wanted_start_time_s = INVALID_TIME;
706 - handle->now_s = now;
707 - storage_point_empty(sp, now - handle->dt_s, now);
708 - return sp;
654 +
655 + if (unlikely(handle->now_s > rrddim_handle->end_time_s)) {
656 + storage_point_empty(sp, handle->now_s - handle->dt_s, handle->now_s);
657 + goto prepare_for_next_iteration;
658 }
659
711 - if (unlikely(!descr || position >= handle->entries)) {
660 + if (unlikely(!handle->page || handle->position >= handle->entries)) {
661 // We need to get a new page
713 - if(rrdeng_load_page_next(rrddim_handle, false)) {
714 - // next calls will not load any more metrics
715 - handle->wanted_start_time_s = INVALID_TIME;
716 - handle->now_s = now;
717 - storage_point_empty(sp, now - handle->dt_s, now);
718 - return sp;
719 - }
662
721 - descr = handle->descr;
722 - position = handle->position;
723 - now = (time_t)((descr->start_time_ut / USEC_PER_SEC) + position * descr->update_every_s);
724 -
725 -// if(debug_this) {
726 -// char buffer[100];
727 -// snprintfz(buffer, 100, "NEW PAGE system.cpu, now:%u, dt:%u, position:%u page_index first:%u, last:%u",
728 -// (uint32_t)(now),
729 -// (uint32_t)(handle->dt_s),
730 -// (uint32_t)(handle->position),
731 -// (uint32_t)(handle->page_index->oldest_time / USEC_PER_SEC),
732 -// (uint32_t)(handle->page_index->latest_time / USEC_PER_SEC));
733 -//
734 -// print_page_cache_descr(descr, buffer, false);
735 -// }
663 + if (!rrdeng_load_page_next(rrddim_handle, false)) {
664 + storage_point_empty(sp, handle->now_s - handle->dt_s, handle->now_s);
665 + goto prepare_for_next_iteration;
666 + }
667 }
668
738 - sp.start_time = now - handle->dt_s;
739 - sp.end_time = now;
669 + sp.start_time_s = handle->now_s - handle->dt_s;
670 + sp.end_time_s = handle->now_s;
671
741 - handle->position = position;
742 - handle->now_s = now;
743 -
744 - switch(descr->type) {
672 + switch(handle->ctx->page_type) {
673 case PAGE_METRICS: {
746 - storage_number n = handle->page[position];
674 + storage_number n = handle->metric_data[handle->position];
675 sp.min = sp.max = sp.sum = unpack_storage_number(n);
676 sp.flags = n & SN_USER_FLAGS;
677 sp.count = 1;
@@ -752,7 +680,7 @@ STORAGE_POINT rrdeng_load_metric_next(struct storage_engine_query_handle *rrddim
680 break;
681
682 case PAGE_TIER: {
755 - tier1_value = ((storage_number_tier1_t *)handle->page)[position];
683 + storage_number_tier1_t tier1_value = ((storage_number_tier1_t *)handle->metric_data)[handle->position];
684 sp.flags = tier1_value.anomaly_count ? SN_FLAG_NONE : SN_FLAG_NOT_ANOMALOUS;
685 sp.count = tier1_value.count;
686 sp.anomaly_count = tier1_value.anomaly_count;
@@ -766,204 +694,99 @@ STORAGE_POINT rrdeng_load_metric_next(struct storage_engine_query_handle *rrddim
694 default: {
695 static bool logged = false;
696 if(!logged) {
769 - error("DBENGINE: unknown page type %d found. Cannot decode it. Ignoring its metrics.", descr->type);
697 + error("DBENGINE: unknown page type %d found. Cannot decode it. Ignoring its metrics.", handle->ctx->page_type);
698 logged = true;
699 }
772 - storage_point_empty(sp, sp.start_time, sp.end_time);
700 + storage_point_empty(sp, sp.start_time_s, sp.end_time_s);
701 }
702 break;
703 }
704
777 - if (unlikely(now >= rrddim_handle->end_time_s)) {
778 - // next calls will not load any more metrics
779 - handle->wanted_start_time_s = INVALID_TIME;
780 - }
705 +prepare_for_next_iteration:
706 + internal_fatal(sp.end_time_s < rrddim_handle->start_time_s, "DBENGINE: this point is too old for this query");
707 + internal_fatal(sp.end_time_s < handle->now_s, "DBENGINE: this point is too old for this point in time");
708
782 -// if(debug_this)
783 -// info("DBENGINE: returning point: "
784 -// "time from %ld to %ld // query from %ld to %ld // wanted_start_time_s %ld"
785 -// , sp.start_time, sp.end_time
786 -// , rrddim_handle->start_time_s, rrddim_handle->end_time_s
787 -// , handle->wanted_start_time_s
788 -// );
709 + handle->now_s += handle->dt_s;
710 + handle->position++;
711
712 return sp;
713 }
714
793 -int rrdeng_load_metric_is_finished(struct storage_engine_query_handle *rrdimm_handle)
794 -{
795 - struct rrdeng_query_handle *handle = (struct rrdeng_query_handle *)rrdimm_handle->handle;
796 - return (INVALID_TIME == handle->wanted_start_time_s);
715 +int rrdeng_load_metric_is_finished(struct storage_engine_query_handle *rrddim_handle) {
716 + struct rrdeng_query_handle *handle = (struct rrdeng_query_handle *)rrddim_handle->handle;
717 + return (handle->now_s > rrddim_handle->end_time_s);
718 }
719
720 /*
721 * Releases the database reference from the handle for loading metrics.
722 */
802 -void rrdeng_load_metric_finalize(struct storage_engine_query_handle *rrdimm_handle)
723 +void rrdeng_load_metric_finalize(struct storage_engine_query_handle *rrddim_handle)
724 {
804 - struct rrdeng_query_handle *handle = (struct rrdeng_query_handle *)rrdimm_handle->handle;
805 - struct rrdengine_instance *ctx = handle->ctx;
806 - struct rrdeng_page_descr *descr = handle->descr;
725 + struct rrdeng_query_handle *handle = (struct rrdeng_query_handle *)rrddim_handle->handle;
726
808 - if (descr) {
809 -#ifdef NETDATA_INTERNAL_CHECKS
810 - rrd_stat_atomic_add(&ctx->stats.metric_API_consumers, -1);
811 -#endif
812 - pg_cache_put(ctx, descr);
813 - }
727 + if (handle->page)
728 + pgc_page_release(main_cache, handle->page);
729
815 - // whatever is allocated at rrdeng_load_metric_init() should be freed here
816 - freez(handle);
817 - rrdimm_handle->handle = NULL;
818 -}
730 + if(!pdc_release_and_destroy_if_unreferenced(handle->pdc, false, false)) {
731 + __atomic_store_n(&handle->pdc->workers_should_stop, true, __ATOMIC_RELAXED);
732 + }
733
820 -time_t rrdeng_metric_latest_time(STORAGE_METRIC_HANDLE *db_metric_handle) {
821 - struct pg_cache_page_index *page_index = (struct pg_cache_page_index *)db_metric_handle;
822 - return (time_t)(page_index->latest_time_ut / USEC_PER_SEC);
823 -}
824 -time_t rrdeng_metric_oldest_time(STORAGE_METRIC_HANDLE *db_metric_handle) {
825 - struct pg_cache_page_index *page_index = (struct pg_cache_page_index *)db_metric_handle;
826 - return (time_t)(page_index->oldest_time_ut / USEC_PER_SEC);
734 + unregister_query_handle(handle);
735 + rrdeng_query_handle_release(handle);
736 + rrddim_handle->handle = NULL;
737 + netdata_thread_enable_cancelability();
738 }
739
829 -int rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *si, uuid_t *dim_uuid, time_t *first_entry_t, time_t *last_entry_t)
830 -{
831 - struct page_cache *pg_cache;
832 - struct rrdengine_instance *ctx;
833 - Pvoid_t *PValue;
834 - struct pg_cache_page_index *page_index = NULL;
740 +time_t rrdeng_load_align_to_optimal_before(struct storage_engine_query_handle *rrddim_handle) {
741 + struct rrdeng_query_handle *handle = (struct rrdeng_query_handle *)rrddim_handle->handle;
742
836 - ctx = (struct rrdengine_instance *)si;
837 - if (unlikely(!ctx)) {
838 - error("DBENGINE: invalid STORAGE INSTANCE to %s()", __FUNCTION__);
839 - return 1;
743 + if(handle->pdc) {
744 + rrdeng_prep_wait(handle->pdc);
745 + if (handle->pdc->optimal_end_time_s > rrddim_handle->end_time_s)
746 + rrddim_handle->end_time_s = handle->pdc->optimal_end_time_s;
747 }
841 - pg_cache = &ctx->pg_cache;
748
843 - uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
844 - PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, dim_uuid, sizeof(uuid_t));
845 - if (likely(NULL != PValue)) {
846 - page_index = *PValue;
847 - }
848 - uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
749 + return rrddim_handle->end_time_s;
750 +}
751
850 - if (likely(page_index)) {
851 - *first_entry_t = page_index->oldest_time_ut / USEC_PER_SEC;
852 - *last_entry_t = page_index->latest_time_ut / USEC_PER_SEC;
853 - return 0;
854 - }
752 +time_t rrdeng_metric_latest_time(STORAGE_METRIC_HANDLE *db_metric_handle) {
753 + METRIC *metric = (METRIC *)db_metric_handle;
754 + time_t latest_time_s = 0;
755
856 - return 1;
857 -}
756 + if (metric)
757 + latest_time_s = mrg_metric_get_latest_time_s(main_mrg, metric);
758
859 -/* Also gets a reference for the page */
860 -void *rrdeng_create_page(struct rrdengine_instance *ctx, uuid_t *id, struct rrdeng_page_descr **ret_descr)
861 -{
862 - struct rrdeng_page_descr *descr;
863 - struct page_cache_descr *pg_cache_descr;
864 - void *page;
865 - /* TODO: check maximum number of pages in page cache limit */
866 -
867 - descr = pg_cache_create_descr();
868 - descr->id = id; /* TODO: add page type: metric, log, something? */
869 - descr->type = ctx->page_type;
870 - page = dbengine_page_alloc(); /*TODO: add page size */
871 - rrdeng_page_descr_mutex_lock(ctx, descr);
872 - pg_cache_descr = descr->pg_cache_descr;
873 - pg_cache_descr->page = page;
874 - pg_cache_descr->flags = RRD_PAGE_DIRTY /*| RRD_PAGE_LOCKED */ | RRD_PAGE_POPULATED /* | BEING_COLLECTED */;
875 - pg_cache_descr->refcnt = 1;
876 -
877 - debug(D_RRDENGINE, "Created new page:");
878 - if (unlikely(debug_flags & D_RRDENGINE))
879 - print_page_cache_descr(descr, "", true);
880 - rrdeng_page_descr_mutex_unlock(ctx, descr);
881 - *ret_descr = descr;
882 - return page;
759 + return latest_time_s;
760 }
761
885 -/* The page must not be empty */
886 -void rrdeng_commit_page(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr,
887 - Word_t page_correlation_id)
888 -{
889 - struct page_cache *pg_cache = &ctx->pg_cache;
890 - Pvoid_t *PValue;
891 - unsigned nr_committed_pages;
762 +time_t rrdeng_metric_oldest_time(STORAGE_METRIC_HANDLE *db_metric_handle) {
763 + METRIC *metric = (METRIC *)db_metric_handle;
764
893 - if (unlikely(NULL == descr)) {
894 - debug(D_RRDENGINE, "%s: page descriptor is NULL, page has already been force-committed.", __func__);
895 - return;
896 - }
897 - fatal_assert(descr->page_length);
898 -
899 - uv_rwlock_wrlock(&pg_cache->committed_page_index.lock);
900 - PValue = JudyLIns(&pg_cache->committed_page_index.JudyL_array, page_correlation_id, PJE0);
901 - *PValue = descr;
902 - nr_committed_pages = ++pg_cache->committed_page_index.nr_committed_pages;
903 - uv_rwlock_wrunlock(&pg_cache->committed_page_index.lock);
904 -
905 - if (nr_committed_pages >= pg_cache_hard_limit(ctx) / 2) {
906 - /* over 50% of pages have not been committed yet */
907 -
908 - if (ctx->drop_metrics_under_page_cache_pressure &&
909 - nr_committed_pages >= pg_cache_committed_hard_limit(ctx)) {
910 - /* 100% of pages are dirty */
911 - struct rrdeng_cmd cmd;
912 -
913 - cmd.opcode = RRDENG_INVALIDATE_OLDEST_MEMORY_PAGE;
914 - rrdeng_enq_cmd(&ctx->worker_config, &cmd);
915 - } else {
916 - if (0 == (unsigned long) ctx->stats.pg_cache_over_half_dirty_events) {
917 - /* only print the first time */
918 - errno = 0;
919 - error("Failed to flush dirty buffers quickly enough in dbengine instance \"%s\". "
920 - "Metric data at risk of not being stored in the database, "
921 - "please reduce disk load or use a faster disk.", ctx->dbfiles_path);
922 - }
923 - rrd_stat_atomic_add(&ctx->stats.pg_cache_over_half_dirty_events, 1);
924 - rrd_stat_atomic_add(&global_pg_cache_over_half_dirty_events, 1);
925 - }
926 - }
765 + time_t oldest_time_s = 0;
766 + if (metric)
767 + oldest_time_s = mrg_metric_get_first_time_s(main_mrg, metric);
768
928 - pg_cache_put(ctx, descr);
769 + return oldest_time_s;
770 }
771
931 -/* Gets a reference for the page */
932 -void *rrdeng_get_latest_page(struct rrdengine_instance *ctx, uuid_t *id, void **handle)
772 +bool rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *db_instance, uuid_t *dim_uuid, time_t *first_entry_s, time_t *last_entry_s)
773 {
934 - struct rrdeng_page_descr *descr;
935 - struct page_cache_descr *pg_cache_descr;
936 -
937 - debug(D_RRDENGINE, "Reading existing page:");
938 - descr = pg_cache_lookup(ctx, NULL, id, INVALID_TIME);
939 - if (NULL == descr) {
940 - *handle = NULL;
941 -
942 - return NULL;
774 + struct rrdengine_instance *ctx = (struct rrdengine_instance *)db_instance;
775 + if (unlikely(!ctx)) {
776 + error("DBENGINE: invalid STORAGE INSTANCE to %s()", __FUNCTION__);
777 + return false;
778 }
944 - *handle = descr;
945 - pg_cache_descr = descr->pg_cache_descr;
779
947 - return pg_cache_descr->page;
948 -}
780 + METRIC *metric = mrg_metric_get_and_acquire(main_mrg, dim_uuid, (Word_t) ctx);
781 + if (unlikely(!metric))
782 + return false;
783
950 -/* Gets a reference for the page */
951 -void *rrdeng_get_page(struct rrdengine_instance *ctx, uuid_t *id, usec_t point_in_time_ut, void **handle)
952 -{
953 - struct rrdeng_page_descr *descr;
954 - struct page_cache_descr *pg_cache_descr;
955 -
956 - debug(D_RRDENGINE, "Reading existing page:");
957 - descr = pg_cache_lookup(ctx, NULL, id, point_in_time_ut);
958 - if (NULL == descr) {
959 - *handle = NULL;
784 + *first_entry_s = mrg_metric_get_first_time_s(main_mrg, metric);
785 + *last_entry_s = mrg_metric_get_latest_time_s(main_mrg, metric);
786
961 - return NULL;
962 - }
963 - *handle = descr;
964 - pg_cache_descr = descr->pg_cache_descr;
787 + mrg_metric_release(main_mrg, metric);
788
966 - return pg_cache_descr->page;
789 + return true;
790 }
791
792 /*
@@ -977,13 +800,11 @@ void rrdeng_get_37_statistics(struct rrdengine_instance *ctx, unsigned long long
800 if (ctx == NULL)
801 return;
802
980 - struct page_cache *pg_cache = &ctx->pg_cache;
981 -
803 array[0] = (uint64_t)ctx->stats.metric_API_producers;
804 array[1] = (uint64_t)ctx->stats.metric_API_consumers;
984 - array[2] = (uint64_t)pg_cache->page_descriptors;
985 - array[3] = (uint64_t)pg_cache->populated_pages;
986 - array[4] = (uint64_t)pg_cache->committed_page_index.nr_committed_pages;
805 + array[2] = 0;
806 + array[3] = 0;
807 + array[4] = 0;
808 array[5] = (uint64_t)ctx->stats.pg_cache_insertions;
809 array[6] = (uint64_t)ctx->stats.pg_cache_deletions;
810 array[7] = (uint64_t)ctx->stats.pg_cache_hits;
@@ -1016,14 +837,9 @@ void rrdeng_get_37_statistics(struct rrdengine_instance *ctx, unsigned long long
837 array[34] = (uint64_t)global_pg_cache_over_half_dirty_events;
838 array[35] = (uint64_t)ctx->stats.flushing_pressure_page_deletions;
839 array[36] = (uint64_t)global_flushing_pressure_page_deletions;
1019 - fatal_assert(RRDENG_NR_STATS == 37);
1020 -}
840 + array[37] = 0; //(uint64_t)pg_cache->active_descriptors;
841
1022 -/* Releases reference to page */
1023 -void rrdeng_put_page(struct rrdengine_instance *ctx, void *handle)
1024 -{
1025 - (void)ctx;
1026 - pg_cache_put(ctx, (struct rrdeng_page_descr *)handle);
842 + fatal_assert(RRDENG_NR_STATS == 38);
843 }
844
845 /*
@@ -1062,9 +878,6 @@ int rrdeng_init(RRDHOST *host, struct rrdengine_instance **ctxp, char *dbfiles_p
878 ctx->global_compress_alg = RRD_LZ4;
879 if (page_cache_mb < RRDENG_MIN_PAGE_CACHE_SIZE_MB)
880 page_cache_mb = RRDENG_MIN_PAGE_CACHE_SIZE_MB;
1065 - ctx->max_cache_pages = page_cache_mb * (1048576LU / RRDENG_BLOCK_SIZE);
1066 - /* try to keep 5% of the page cache free */
1067 - ctx->cache_pages_low_watermark = (ctx->max_cache_pages * 95LLU) / 100;
881 if (disk_space_mb < RRDENG_MIN_DISK_SPACE_MB)
882 disk_space_mb = RRDENG_MIN_DISK_SPACE_MB;
883 ctx->max_disk_space = disk_space_mb * 1048576LLU;
@@ -1075,46 +888,29 @@ int rrdeng_init(RRDHOST *host, struct rrdengine_instance **ctxp, char *dbfiles_p
888 else
889 strncpyz(ctx->machine_guid, host->machine_guid, GUID_LEN);
890
1078 - ctx->drop_metrics_under_page_cache_pressure = rrdeng_drop_metrics_under_page_cache_pressure;
891 ctx->metric_API_max_producers = 0;
892 ctx->quiesce = NO_QUIESCE;
893 ctx->host = host;
894
895 memset(&ctx->worker_config, 0, sizeof(ctx->worker_config));
1084 - ctx->worker_config.ctx = ctx;
1085 - init_page_cache(ctx);
896 + init_page_cache();
897 init_commit_log(ctx);
898 error = init_rrd_files(ctx);
1088 - if (error) {
1089 - goto error_after_init_rrd_files;
1090 - }
899 + if (!error) {
900
1092 - completion_init(&ctx->rrdengine_completion);
1093 - fatal_assert(0 == uv_thread_create(&ctx->worker_config.thread, rrdeng_worker, &ctx->worker_config));
1094 - /* wait for worker thread to initialize */
1095 - completion_wait_for(&ctx->rrdengine_completion);
1096 - completion_destroy(&ctx->rrdengine_completion);
1097 - uv_thread_set_name_np(ctx->worker_config.thread, "LIBUV_WORKER");
1098 - if (ctx->worker_config.error) {
1099 - goto error_after_rrdeng_worker;
1100 - }
1101 -// error = metalog_init(ctx);
1102 -// if (error) {
1103 -// error("Failed to initialize metadata log file event loop.");
1104 -// goto error_after_rrdeng_worker;
1105 -// }
901 + if(rrdeng_dbengine_spawn(ctx))
902 + // success - we run this ctx too
903 + return 0;
904
1107 - return 0;
905 + finalize_rrd_files(ctx);
906 + }
907
1109 -error_after_rrdeng_worker:
1110 - finalize_rrd_files(ctx);
1111 -error_after_init_rrd_files:
1112 - free_page_cache(ctx);
908 if (!is_storage_engine_shared((STORAGE_INSTANCE *)ctx)) {
909 freez(ctx);
910 if (ctxp)
911 *ctxp = NULL;
912 }
913 +
914 rrd_stat_atomic_add(&rrdeng_reserved_file_descriptors, -RRDENG_FD_BUDGET_PER_INSTANCE);
915 return UV_EIO;
916 }
@@ -1122,23 +918,23 @@ error_after_init_rrd_files:
918 /*
919 * Returns 0 on success, 1 on error
920 */
1125 -int rrdeng_exit(struct rrdengine_instance *ctx)
1126 -{
1127 - struct rrdeng_cmd cmd;
1128 -
1129 - if (NULL == ctx) {
921 +int rrdeng_exit(struct rrdengine_instance *ctx) {
922 + if (NULL == ctx)
923 return 1;
1131 - }
924
1133 - /* TODO: add page to page cache */
1134 - cmd.opcode = RRDENG_SHUTDOWN;
1135 - rrdeng_enq_cmd(&ctx->worker_config, &cmd);
925 + // FIXME - ktsaou - properly cleanup ctx
926 + // 1. make sure all collectors are stopped
927 + // 2. make new queries will not be accepted
928 + // 3. flush this section of the main cache
929 + // 4. then wait for completion
930
1137 - fatal_assert(0 == uv_thread_join(&ctx->worker_config.thread));
931 + struct completion completion = {};
932 + completion_init(&completion);
933 + rrdeng_enq_cmd(ctx, RRDENG_OPCODE_CTX_SHUTDOWN, NULL, &completion, STORAGE_PRIORITY_BEST_EFFORT);
934 + completion_wait_for(&completion);
935 + completion_destroy(&completion);
936
937 finalize_rrd_files(ctx);
1140 - //metalog_exit(ctx->metalog_ctx);
1141 - free_page_cache(ctx);
938
939 if(!is_storage_engine_shared((STORAGE_INSTANCE *)ctx))
940 freez(ctx);
@@ -1147,76 +943,96 @@ int rrdeng_exit(struct rrdengine_instance *ctx)
943 return 0;
944 }
945
1150 -void rrdeng_prepare_exit(struct rrdengine_instance *ctx)
1151 -{
1152 - struct rrdeng_cmd cmd;
1153 -
1154 - if (NULL == ctx) {
946 +void rrdeng_prepare_exit(struct rrdengine_instance *ctx) {
947 + if (NULL == ctx)
948 return;
1156 - }
949
1158 - completion_init(&ctx->rrdengine_completion);
1159 - cmd.opcode = RRDENG_QUIESCE;
1160 - rrdeng_enq_cmd(&ctx->worker_config, &cmd);
950 + // FIXME - ktsaou - properly cleanup ctx
951 + // 1. make sure all collectors are stopped
952
1162 - /* wait for dbengine to quiesce */
1163 - completion_wait_for(&ctx->rrdengine_completion);
1164 - completion_destroy(&ctx->rrdengine_completion);
1165 -
1166 - //metalog_prepare_exit(ctx->metalog_ctx);
953 + completion_init(&ctx->quiesce_completion);
954 + rrdeng_enq_cmd(ctx, RRDENG_OPCODE_CTX_QUIESCE, NULL, NULL, STORAGE_PRIORITY_CRITICAL);
955 }
956
1169 -RRDENG_SIZE_STATS rrdeng_size_statistics(struct rrdengine_instance *ctx) {
1170 - RRDENG_SIZE_STATS stats = { 0 };
957 +static void populate_v2_statistics(struct rrdengine_datafile *datafile, RRDENG_SIZE_STATS *stats)
958 +{
959 + void *data_start = GET_JOURNAL_DATA(datafile->journalfile);
960 + if (unlikely(!data_start))
961 + return;
962
1172 - for(struct pg_cache_page_index *page_index = ctx->pg_cache.metrics_index.last_page_index;
1173 - page_index != NULL ;page_index = page_index->prev) {
1174 - stats.metrics++;
1175 - stats.metrics_pages += page_index->page_count;
963 + struct journal_v2_header *j2_header = (void *) data_start;
964 +
965 + stats->extents += j2_header->extent_count;
966 +
967 + unsigned entries;
968 + struct journal_extent_list *extent_list = (void *) (data_start + j2_header->extent_offset);
969 + for (entries = 0; entries < j2_header->extent_count; entries++) {
970 + stats->extents_compressed_bytes += extent_list->datafile_size;
971 + stats->extents_pages += extent_list->pages;
972 + extent_list++;
973 }
974
1178 - for(struct rrdengine_datafile *df = ctx->datafiles.first; df ;df = df->next) {
1179 - stats.datafiles++;
975 + struct journal_metric_list *metric = (void *) (data_start + j2_header->metric_offset);
976 + time_t journal_start_time_s = (time_t) (j2_header->start_time_ut / USEC_PER_SEC);
977
1181 - for(struct extent_info *ei = df->extents.first; ei ; ei = ei->next) {
1182 - stats.extents++;
1183 - stats.extents_compressed_bytes += ei->size;
978 + stats->metrics += j2_header->metric_count;
979 + for (entries = 0; entries < j2_header->metric_count; entries++) {
980
1185 - for(int p = 0; p < ei->number_of_pages ;p++) {
1186 - struct rrdeng_page_descr *descr = ei->pages[p];
981 + struct journal_page_header *metric_list_header = (void *) (data_start + metric->page_offset);
982 + stats->metrics_pages += metric_list_header->entries;
983 + struct journal_page_list *descr = (void *) (data_start + metric->page_offset + sizeof(struct journal_page_header));
984 + for (uint32_t idx=0; idx < metric_list_header->entries; idx++) {
985
1188 - usec_t update_every_usec;
986 + time_t update_every_s;
987
1190 - size_t points = descr->page_length / PAGE_POINT_SIZE_BYTES(descr);
988 + size_t points = descr->page_length / PAGE_POINT_CTX_SIZE_BYTES(datafile->ctx);
989
1192 - if(likely(points > 1))
1193 - update_every_usec = (descr->end_time_ut - descr->start_time_ut) / (points - 1);
1194 - else {
1195 - update_every_usec = default_rrd_update_every * get_tier_grouping(ctx->tier) * USEC_PER_SEC;
1196 - stats.single_point_pages++;
1197 - }
990 + time_t start_time_s = journal_start_time_s + descr->delta_start_s;
991 + time_t end_time_s = journal_start_time_s + descr->delta_end_s;
992
1199 - time_t duration_secs = (time_t)((descr->end_time_ut - descr->start_time_ut + update_every_usec)/USEC_PER_SEC);
993 + if(likely(points > 1))
994 + update_every_s = (time_t) ((end_time_s - start_time_s) / (points - 1));
995 + else {
996 + update_every_s = (time_t) (default_rrd_update_every * get_tier_grouping(datafile->ctx->tier));
997 + stats->single_point_pages++;
998 + }
999
1201 - stats.extents_pages++;
1202 - stats.pages_uncompressed_bytes += descr->page_length;
1203 - stats.pages_duration_secs += duration_secs;
1204 - stats.points += points;
1000 + time_t duration_s = (time_t)((end_time_s - start_time_s + update_every_s));
1001
1206 - stats.page_types[descr->type].pages++;
1207 - stats.page_types[descr->type].pages_uncompressed_bytes += descr->page_length;
1208 - stats.page_types[descr->type].pages_duration_secs += duration_secs;
1209 - stats.page_types[descr->type].points += points;
1002 + stats->pages_uncompressed_bytes += descr->page_length;
1003 + stats->pages_duration_secs += duration_s;
1004 + stats->points += points;
1005
1211 - if(!stats.first_t || (descr->start_time_ut - update_every_usec) < stats.first_t)
1212 - stats.first_t = (descr->start_time_ut - update_every_usec) / USEC_PER_SEC;
1006 + stats->page_types[descr->type].pages++;
1007 + stats->page_types[descr->type].pages_uncompressed_bytes += descr->page_length;
1008 + stats->page_types[descr->type].pages_duration_secs += duration_s;
1009 + stats->page_types[descr->type].points += points;
1010
1214 - if(!stats.last_t || descr->end_time_ut > stats.last_t)
1215 - stats.last_t = descr->end_time_ut / USEC_PER_SEC;
1216 - }
1011 + if(!stats->first_time_s || (start_time_s - update_every_s) < stats->first_time_s)
1012 + stats->first_time_s = (start_time_s - update_every_s);
1013 +
1014 + if(!stats->last_time_s || end_time_s > stats->last_time_s)
1015 + stats->last_time_s = end_time_s;
1016 +
1017 + descr++;
1018 }
1019 + metric++;
1020 }
1021 +}
1022 +
1023 +RRDENG_SIZE_STATS rrdeng_size_statistics(struct rrdengine_instance *ctx) {
1024 + RRDENG_SIZE_STATS stats = { 0 };
1025 +
1026 + uv_rwlock_rdlock(&ctx->datafiles.rwlock);
1027 + for(struct rrdengine_datafile *df = ctx->datafiles.first; df ;df = df->next) {
1028 + stats.datafiles++;
1029
1030 + if (GET_JOURNAL_DATA(df->journalfile)) {
1031 + // FIXME: Rework statistics based only on V2
1032 + populate_v2_statistics(df, &stats);
1033 + }
1034 + }
1035 + uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
1036
1037 stats.currently_collected_metrics = ctx->stats.metric_API_producers;
1038 stats.max_concurrently_collected_metrics = ctx->metric_API_max_producers;
@@ -1228,7 +1044,7 @@ RRDENG_SIZE_STATS rrdeng_size_statistics(struct rrdengine_instance *ctx) {
1044 stats.disk_space = ctx->disk_space;
1045 stats.max_disk_space = ctx->max_disk_space;
1046
1231 - stats.database_retention_secs = (time_t)(stats.last_t - stats.first_t);
1047 + stats.database_retention_secs = (time_t)(stats.last_time_s - stats.first_time_s);
1048
1049 if(stats.extents_pages)
1050 stats.average_page_size_bytes = (double)stats.pages_uncompressed_bytes / (double)stats.extents_pages;
@@ -1252,21 +1068,22 @@ RRDENG_SIZE_STATS rrdeng_size_statistics(struct rrdengine_instance *ctx) {
1068 }
1069 }
1070
1255 - stats.sizeof_metric = struct_natural_alignment(sizeof(struct pg_cache_page_index) + sizeof(struct pg_alignment));
1256 - stats.sizeof_page = struct_natural_alignment(sizeof(struct rrdeng_page_descr));
1071 +// stats.sizeof_metric = 0;
1072 stats.sizeof_datafile = struct_natural_alignment(sizeof(struct rrdengine_datafile)) + struct_natural_alignment(sizeof(struct rrdengine_journalfile));
1258 - stats.sizeof_page_in_cache = struct_natural_alignment(sizeof(struct page_cache_descr));
1073 + stats.sizeof_page_in_cache = 0; // struct_natural_alignment(sizeof(struct page_cache_descr));
1074 stats.sizeof_point_data = page_type_size[ctx->page_type];
1260 - stats.sizeof_page_data = RRDENG_BLOCK_SIZE;
1075 + stats.sizeof_page_data = tier_page_size[ctx->tier];
1076 stats.pages_per_extent = rrdeng_pages_per_extent;
1077
1263 - stats.sizeof_extent = sizeof(struct extent_info);
1264 - stats.sizeof_page_in_extent = sizeof(struct rrdeng_page_descr *);
1265 -
1266 - stats.sizeof_metric_in_index = 40;
1267 - stats.sizeof_page_in_index = 24;
1078 +// stats.sizeof_metric_in_index = 40;
1079 +// stats.sizeof_page_in_index = 24;
1080
1081 stats.default_granularity_secs = (size_t)default_rrd_update_every * get_tier_grouping(ctx->tier);
1082
1083 return stats;
1084 }
1085 +
1086 +struct rrdeng_cache_efficiency_stats rrdeng_get_cache_efficiency_stats(void) {
1087 + // FIXME - make cache efficiency stats atomic
1088 + return rrdeng_cache_efficiency_stats;
1089 +}
database/engine/rrdengineapi.h
+102 -30
@@ -8,7 +8,7 @@
8 #define RRDENG_MIN_PAGE_CACHE_SIZE_MB (8)
9 #define RRDENG_MIN_DISK_SPACE_MB (64)
10
11 -#define RRDENG_NR_STATS (37)
11 +#define RRDENG_NR_STATS (38)
12
13 #define RRDENG_FD_BUDGET_PER_INSTANCE (50)
14
@@ -16,26 +16,16 @@ extern int db_engine_use_malloc;
16 extern int default_rrdeng_page_fetch_timeout;
17 extern int default_rrdeng_page_fetch_retries;
18 extern int default_rrdeng_page_cache_mb;
19 +extern int db_engine_journal_indexing;
20 +extern int db_engine_journal_check;
21 extern int default_rrdeng_disk_quota_mb;
22 extern int default_multidb_disk_quota_mb;
21 -extern uint8_t rrdeng_drop_metrics_under_page_cache_pressure;
23 extern struct rrdengine_instance *multidb_ctx[RRD_STORAGE_TIERS];
24 extern size_t page_type_size[];
25 +extern size_t tier_page_size[];
26
27 #define PAGE_POINT_SIZE_BYTES(x) page_type_size[(x)->type]
26 -
27 -struct rrdeng_region_info {
28 - time_t start_time_s;
29 - int update_every;
30 - unsigned points;
31 -};
32 -
33 -void *rrdeng_create_page(struct rrdengine_instance *ctx, uuid_t *id, struct rrdeng_page_descr **ret_descr);
34 -void rrdeng_commit_page(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr,
35 - Word_t page_correlation_id);
36 -void *rrdeng_get_latest_page(struct rrdengine_instance *ctx, uuid_t *id, void **handle);
37 -void *rrdeng_get_page(struct rrdengine_instance *ctx, uuid_t *id, usec_t point_in_time_ut, void **handle);
38 -void rrdeng_put_page(struct rrdengine_instance *ctx, void *handle);
28 +#define PAGE_POINT_CTX_SIZE_BYTES(x) page_type_size[(x)->page_type]
29
30 void rrdeng_generate_legacy_uuid(const char *dim_id, const char *chart_id, uuid_t *ret_uuid);
31 void rrdeng_convert_legacy_uuid_to_multihost(char machine_guid[GUID_LEN + 1], uuid_t *legacy_uuid,
@@ -44,8 +34,6 @@ void rrdeng_convert_legacy_uuid_to_multihost(char machine_guid[GUID_LEN + 1], uu
34
35 STORAGE_METRIC_HANDLE *rrdeng_metric_get_or_create(RRDDIM *rd, STORAGE_INSTANCE *db_instance);
36 STORAGE_METRIC_HANDLE *rrdeng_metric_get(STORAGE_INSTANCE *db_instance, uuid_t *uuid);
47 -STORAGE_METRIC_HANDLE *rrdeng_metric_create(STORAGE_INSTANCE *db_instance, uuid_t *uuid);
48 -STORAGE_METRIC_HANDLE *rrdeng_metric_get_legacy(STORAGE_INSTANCE *db_instance, const char *rd_id, const char *st_id);
37 void rrdeng_metric_release(STORAGE_METRIC_HANDLE *db_metric_handle);
38 STORAGE_METRIC_HANDLE *rrdeng_metric_dup(STORAGE_METRIC_HANDLE *db_metric_handle);
39
@@ -60,15 +48,16 @@ void rrdeng_store_metric_next(STORAGE_COLLECT_HANDLE *collection_handle, usec_t
48 SN_FLAGS flags);
49 int rrdeng_store_metric_finalize(STORAGE_COLLECT_HANDLE *collection_handle);
50
63 -void rrdeng_load_metric_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *rrdimm_handle,
64 - time_t start_time_s, time_t end_time_s);
51 +void rrdeng_load_metric_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *rrddim_handle,
52 + time_t start_time_s, time_t end_time_s, STORAGE_PRIORITY priority);
53 STORAGE_POINT rrdeng_load_metric_next(struct storage_engine_query_handle *rrddim_handle);
54
55
68 -int rrdeng_load_metric_is_finished(struct storage_engine_query_handle *rrdimm_handle);
69 -void rrdeng_load_metric_finalize(struct storage_engine_query_handle *rrdimm_handle);
56 +int rrdeng_load_metric_is_finished(struct storage_engine_query_handle *rrddim_handle);
57 +void rrdeng_load_metric_finalize(struct storage_engine_query_handle *rrddim_handle);
58 time_t rrdeng_metric_latest_time(STORAGE_METRIC_HANDLE *db_metric_handle);
59 time_t rrdeng_metric_oldest_time(STORAGE_METRIC_HANDLE *db_metric_handle);
60 +time_t rrdeng_load_align_to_optimal_before(struct storage_engine_query_handle *rrddim_handle);
61
62 void rrdeng_get_37_statistics(struct rrdengine_instance *ctx, unsigned long long *array);
63
@@ -78,7 +67,7 @@ int rrdeng_init(RRDHOST *host, struct rrdengine_instance **ctxp, char *dbfiles_p
67
68 int rrdeng_exit(struct rrdengine_instance *ctx);
69 void rrdeng_prepare_exit(struct rrdengine_instance *ctx);
81 -int rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *si, uuid_t *dim_uuid, time_t *first_entry_t, time_t *last_entry_t);
70 +bool rrdeng_metric_retention_by_uuid(STORAGE_INSTANCE *db_instance, uuid_t *dim_uuid, time_t *first_entry_s, time_t *last_entry_s);
71
72 extern STORAGE_METRICS_GROUP *rrdeng_metrics_group_get(STORAGE_INSTANCE *db_instance, uuid_t *uuid);
73 extern void rrdeng_metrics_group_release(STORAGE_INSTANCE *db_instance, STORAGE_METRICS_GROUP *smg);
@@ -86,12 +75,6 @@ extern void rrdeng_metrics_group_release(STORAGE_INSTANCE *db_instance, STORAGE_
75 typedef struct rrdengine_size_statistics {
76 size_t default_granularity_secs;
77
89 - size_t sizeof_metric;
90 - size_t sizeof_metric_in_index;
91 - size_t sizeof_page;
92 - size_t sizeof_page_in_index;
93 - size_t sizeof_extent;
94 - size_t sizeof_page_in_extent;
78 size_t sizeof_datafile;
79 size_t sizeof_page_in_cache;
80 size_t sizeof_point_data;
@@ -119,8 +102,8 @@ typedef struct rrdengine_size_statistics {
102
103 size_t single_point_pages;
104
122 - usec_t first_t;
123 - usec_t last_t;
105 + time_t first_time_s;
106 + time_t last_time_s;
107
108 size_t currently_collected_metrics;
109 size_t max_concurrently_collected_metrics;
@@ -139,6 +122,95 @@ typedef struct rrdengine_size_statistics {
122 double average_page_size_bytes;
123 } RRDENG_SIZE_STATS;
124
125 +struct rrdeng_cache_efficiency_stats {
126 + size_t queries;
127 + size_t queries_planned_with_gaps;
128 + size_t queries_executed_with_gaps;
129 + size_t queries_open;
130 + size_t queries_journal_v2;
131 +
132 + size_t currently_running_queries;
133 +
134 + // query planner output of the queries
135 + size_t pages_total;
136 + size_t pages_to_load_from_disk;
137 +
138 + // pages metadata sources
139 + size_t pages_meta_source_main_cache;
140 + size_t pages_meta_source_open_cache;
141 + size_t pages_meta_source_journal_v2;
142 +
143 + // preloading
144 + size_t pages_pending_found_in_cache_at_pass4;
145 + size_t page_next_wait_failed;
146 + size_t page_next_wait_loaded;
147 + size_t page_next_nowait_failed;
148 + size_t page_next_nowait_loaded;
149 +
150 + // pages data sources
151 + size_t pages_data_source_main_cache;
152 + size_t pages_data_source_disk;
153 + size_t pages_data_source_extent_cache; // loaded by a cached extent
154 +
155 + // cache hits at different points
156 + size_t pages_load_ok_loaded_but_cache_hit_before_allocation; // found in cache after loading, before allocating
157 + size_t pages_load_ok_loaded_but_cache_hit_while_inserting; // found in cache while inserting it (conflict)
158 +
159 + // loading
160 + size_t pages_load_ok_uncompressed;
161 + size_t pages_load_ok_compressed;
162 + size_t pages_load_ok_preloaded;
163 + size_t pages_load_fail_invalid_page_in_extent;
164 + size_t pages_load_fail_cant_mmap_extent;
165 + size_t pages_load_fail_datafile_not_available;
166 + size_t pages_load_fail_unroutable;
167 + size_t pages_load_fail_uuid_not_found;
168 + size_t pages_load_fail_invalid_extent;
169 +
170 + // timings for query preparation
171 + size_t prep_time_to_route;
172 + size_t prep_time_in_main_cache_lookup;
173 + size_t prep_time_in_open_cache_lookup;
174 + size_t prep_time_in_journal_v2_lookup;
175 + size_t prep_time_in_pass4_lookup;
176 +
177 + // timings the query thread experiences
178 + size_t query_time_init;
179 + size_t query_time_wait_for_prep;
180 + size_t query_time_to_slow_disk_next_page;
181 + size_t query_time_to_fast_disk_next_page;
182 + size_t query_time_to_slow_preload_next_page;
183 + size_t query_time_to_fast_preload_next_page;
184 +
185 + // query issues
186 + size_t pages_zero_time_skipped;
187 + size_t pages_past_time_skipped;
188 + size_t pages_overlapping_skipped;
189 + size_t pages_invalid_size_skipped;
190 + size_t pages_invalid_update_every_fixed;
191 + size_t pages_invalid_entries_fixed;
192 +};
193 +
194 +struct rrdeng_buffer_sizes {
195 + size_t workers;
196 + size_t pdc;
197 + size_t wal;
198 + size_t descriptors;
199 + size_t xt_io;
200 + size_t xt_buf;
201 + size_t handles;
202 + size_t opcodes;
203 + size_t epdl;
204 + size_t deol;
205 + size_t pd;
206 +#ifdef PDC_USE_JULYL
207 + size_t julyl;
208 +#endif
209 +};
210 +
211 +struct rrdeng_buffer_sizes rrdeng_get_buffer_sizes(void);
212 +struct rrdeng_cache_efficiency_stats rrdeng_get_cache_efficiency_stats(void);
213 +
214 RRDENG_SIZE_STATS rrdeng_size_statistics(struct rrdengine_instance *ctx);
215
216 #endif /* NETDATA_RRDENGINEAPI_H */
database/engine/rrdenginelib.c
+65 -68
@@ -4,68 +4,68 @@
4 #define BUFSIZE (512)
5
6 /* Caller must hold descriptor lock */
7 -void print_page_cache_descr(struct rrdeng_page_descr *descr, const char *msg, bool log_debug)
8 -{
9 - if(log_debug && !(debug_flags & D_RRDENGINE))
10 - return;
11 -
12 - BUFFER *wb = buffer_create(512);
13 -
14 - if(!descr) {
15 - buffer_sprintf(wb, "DBENGINE: %s : descr is NULL", msg);
16 - }
17 - else {
18 - struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
19 - char uuid_str[UUID_STR_LEN];
20 -
21 - uuid_unparse_lower(*descr->id, uuid_str);
22 - buffer_sprintf(wb, "DBENGINE: %s : page(%p) metric:%s, len:%"PRIu32", time:%"PRIu64"->%"PRIu64", update_every:%u, type:%u, xt_offset:",
23 - msg,
24 - pg_cache_descr->page, uuid_str,
25 - descr->page_length,
26 - (uint64_t)descr->start_time_ut,
27 - (uint64_t)descr->end_time_ut,
28 - (uint32_t)descr->update_every_s,
29 - (uint32_t)descr->type
30 - );
31 - if (!descr->extent) {
32 - buffer_strcat(wb, "N/A");
33 - } else {
34 - buffer_sprintf(wb, "%"PRIu64, descr->extent->offset);
35 - }
36 -
37 - buffer_sprintf(wb, ", flags:0x%2.2lX refcnt:%u", pg_cache_descr->flags, pg_cache_descr->refcnt);
38 - }
39 -
40 - if(log_debug)
41 - debug(D_RRDENGINE, "%s", buffer_tostring(wb));
42 - else
43 - internal_error(true, "%s", buffer_tostring(wb));
44 -
45 - buffer_free(wb);
46 -}
47 -
48 -void print_page_descr(struct rrdeng_page_descr *descr)
49 -{
50 - char uuid_str[UUID_STR_LEN];
51 - char str[BUFSIZE + 1];
52 - int pos = 0;
53 -
54 - uuid_unparse_lower(*descr->id, uuid_str);
55 - pos += snprintfz(str, BUFSIZE - pos, "id=%s\n"
56 - "--->len:%"PRIu32" time:%"PRIu64"->%"PRIu64" xt_offset:",
57 - uuid_str,
58 - descr->page_length,
59 - (uint64_t)descr->start_time_ut,
60 - (uint64_t)descr->end_time_ut);
61 - if (!descr->extent) {
62 - pos += snprintfz(str + pos, BUFSIZE - pos, "N/A");
63 - } else {
64 - pos += snprintfz(str + pos, BUFSIZE - pos, "%"PRIu64, descr->extent->offset);
65 - }
66 - snprintfz(str + pos, BUFSIZE - pos, "\n\n");
67 - fputs(str, stderr);
68 -}
7 +//void print_page_cache_descr(struct rrdeng_page_descr *descr, const char *msg, bool log_debug)
8 +//{
9 +// if(log_debug && !(debug_flags & D_RRDENGINE))
10 +// return;
11 +//
12 +// BUFFER *wb = buffer_create(512);
13 +//
14 +// if(!descr) {
15 +// buffer_sprintf(wb, "DBENGINE: %s : descr is NULL", msg);
16 +// }
17 +// else {
18 +// struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
19 +// char uuid_str[UUID_STR_LEN];
20 +//
21 +// uuid_unparse_lower(*descr->id, uuid_str);
22 +// buffer_sprintf(wb, "DBENGINE: %s : page(%p) metric:%s, len:%"PRIu32", time:%"PRIu64"->%"PRIu64", update_every:%u, type:%u, xt_offset:",
23 +// msg,
24 +// pg_cache_descr->page, uuid_str,
25 +// descr->page_length,
26 +// (uint64_t)descr->start_time_ut,
27 +// (uint64_t)descr->end_time_ut,
28 +// (uint32_t)descr->update_every_s,
29 +// (uint32_t)descr->type
30 +// );
31 +// if (!descr->extent) {
32 +// buffer_strcat(wb, "N/A");
33 +// } else {
34 +// buffer_sprintf(wb, "%"PRIu64, descr->extent->offset);
35 +// }
36 +//
37 +// buffer_sprintf(wb, ", flags:0x%2.2lX refcnt:%u", pg_cache_descr->flags, pg_cache_descr->refcnt);
38 +// }
39 +//
40 +// if(log_debug)
41 +// debug(D_RRDENGINE, "%s", buffer_tostring(wb));
42 +// else
43 +// internal_error(true, "%s", buffer_tostring(wb));
44 +//
45 +// buffer_free(wb);
46 +//}
47 +//
48 +//void print_page_descr(struct rrdeng_page_descr *descr)
49 +//{
50 +// char uuid_str[UUID_STR_LEN];
51 +// char str[BUFSIZE + 1];
52 +// int pos = 0;
53 +//
54 +// uuid_unparse_lower(*descr->id, uuid_str);
55 +// pos += snprintfz(str, BUFSIZE - pos, "id=%s\n"
56 +// "--->len:%"PRIu32" time:%"PRIu64"->%"PRIu64" xt_offset:",
57 +// uuid_str,
58 +// descr->page_length,
59 +// (uint64_t)descr->start_time_ut,
60 +// (uint64_t)descr->end_time_ut);
61 +// if (!descr->extent) {
62 +// pos += snprintfz(str + pos, BUFSIZE - pos, "N/A");
63 +// } else {
64 +// pos += snprintfz(str + pos, BUFSIZE - pos, "%"PRIu64, descr->extent->offset);
65 +// }
66 +// snprintfz(str + pos, BUFSIZE - pos, "\n\n");
67 +// fputs(str, stderr);
68 +//}
69
70 int check_file_properties(uv_file file, uint64_t *file_size, size_t min_size)
71 {
@@ -144,9 +144,6 @@ int open_file_for_io(char *path, int flags, uv_file *file, int direct)
144
145 char *get_rrdeng_statistics(struct rrdengine_instance *ctx, char *str, size_t size)
146 {
147 - struct page_cache *pg_cache;
148 -
149 - pg_cache = &ctx->pg_cache;
147 snprintfz(str, size,
148 "metric_API_producers: %ld\n"
149 "metric_API_consumers: %ld\n"
@@ -187,10 +184,10 @@ char *get_rrdeng_statistics(struct rrdengine_instance *ctx, char *str, size_t si
184 "global_flushing_pressure_page_deletions: %ld\n",
185 (long)ctx->stats.metric_API_producers,
186 (long)ctx->stats.metric_API_consumers,
190 - (long)pg_cache->page_descriptors,
187 + 0L,
188 (long)ctx->stats.page_cache_descriptors,
192 - (long)pg_cache->populated_pages,
193 - (long)pg_cache->committed_page_index.nr_committed_pages,
189 + 0L,
190 + 0L,
191 (long)ctx->stats.pg_cache_insertions,
192 (long)ctx->stats.pg_cache_deletions,
193 (long)ctx->stats.pg_cache_hits,
database/engine/rrdenginelib.h
-3
@@ -6,7 +6,6 @@
6 #include "libnetdata/libnetdata.h"
7
8 /* Forward declarations */
9 -struct rrdeng_page_descr;
9 struct rrdengine_instance;
10
11 #define STR_HELPER(x) #x
@@ -83,8 +82,6 @@ static inline void crc32set(void *crcp, uLong crc)
82 *(uint32_t *)crcp = crc;
83 }
84
86 -void print_page_cache_descr(struct rrdeng_page_descr *descr, const char *msg, bool log_debug);
87 -void print_page_descr(struct rrdeng_page_descr *descr);
85 int check_file_properties(uv_file file, uint64_t *file_size, size_t min_size);
86 int open_file_for_io(char *path, int flags, uv_file *file, int direct);
87 static inline int open_file_direct_io(char *path, int flags, uv_file *file)
database/engine/rrdenglocking.c deleted
-241
@@ -1,241 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -#include "rrdengine.h"
3 -
4 -struct page_cache_descr *rrdeng_create_pg_cache_descr(struct rrdengine_instance *ctx)
5 -{
6 - struct page_cache_descr *pg_cache_descr;
7 -
8 - pg_cache_descr = mallocz(sizeof(*pg_cache_descr));
9 - rrd_stat_atomic_add(&ctx->stats.page_cache_descriptors, 1);
10 - pg_cache_descr->page = NULL;
11 - pg_cache_descr->flags = 0;
12 - pg_cache_descr->prev = pg_cache_descr->next = NULL;
13 - pg_cache_descr->refcnt = 0;
14 - pg_cache_descr->waiters = 0;
15 - fatal_assert(0 == uv_cond_init(&pg_cache_descr->cond));
16 - fatal_assert(0 == uv_mutex_init(&pg_cache_descr->mutex));
17 -
18 - return pg_cache_descr;
19 -}
20 -
21 -void rrdeng_destroy_pg_cache_descr(struct rrdengine_instance *ctx, struct page_cache_descr *pg_cache_descr)
22 -{
23 - uv_cond_destroy(&pg_cache_descr->cond);
24 - uv_mutex_destroy(&pg_cache_descr->mutex);
25 - freez(pg_cache_descr);
26 - rrd_stat_atomic_add(&ctx->stats.page_cache_descriptors, -1);
27 -}
28 -
29 -/* also allocates page cache descriptor if missing */
30 -void rrdeng_page_descr_mutex_lock(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
31 -{
32 - unsigned long old_state, old_users, new_state, ret_state;
33 - struct page_cache_descr *pg_cache_descr = NULL;
34 - uint8_t we_locked;
35 -
36 - we_locked = 0;
37 - while (1) { /* spin */
38 - old_state = descr->pg_cache_descr_state;
39 - old_users = old_state >> PG_CACHE_DESCR_SHIFT;
40 -
41 - if (unlikely(we_locked)) {
42 - fatal_assert(old_state & PG_CACHE_DESCR_LOCKED);
43 - new_state = (1 << PG_CACHE_DESCR_SHIFT) | PG_CACHE_DESCR_ALLOCATED;
44 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, old_state, new_state);
45 - if (old_state == ret_state) {
46 - /* success */
47 - break;
48 - }
49 - continue; /* spin */
50 - }
51 - if (old_state & PG_CACHE_DESCR_LOCKED) {
52 - fatal_assert(0 == old_users);
53 - continue; /* spin */
54 - }
55 - if (0 == old_state) {
56 - /* no page cache descriptor has been allocated */
57 -
58 - if (NULL == pg_cache_descr) {
59 - pg_cache_descr = rrdeng_create_pg_cache_descr(ctx);
60 - }
61 - new_state = PG_CACHE_DESCR_LOCKED;
62 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, 0, new_state);
63 - if (0 == ret_state) {
64 - we_locked = 1;
65 - descr->pg_cache_descr = pg_cache_descr;
66 - pg_cache_descr->descr = descr;
67 - pg_cache_descr = NULL; /* make sure we don't free pg_cache_descr */
68 - /* retry */
69 - continue;
70 - }
71 - continue; /* spin */
72 - }
73 - /* page cache descriptor is already allocated */
74 - if (unlikely(!(old_state & PG_CACHE_DESCR_ALLOCATED))) {
75 - fatal("Invalid page cache descriptor locking state:%#lX", old_state);
76 - }
77 - new_state = (old_users + 1) << PG_CACHE_DESCR_SHIFT;
78 - new_state |= old_state & PG_CACHE_DESCR_FLAGS_MASK;
79 -
80 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, old_state, new_state);
81 - if (old_state == ret_state) {
82 - /* success */
83 - break;
84 - }
85 - /* spin */
86 - }
87 -
88 - if (pg_cache_descr) {
89 - rrdeng_destroy_pg_cache_descr(ctx, pg_cache_descr);
90 - }
91 - pg_cache_descr = descr->pg_cache_descr;
92 - uv_mutex_lock(&pg_cache_descr->mutex);
93 -}
94 -
95 -void rrdeng_page_descr_mutex_unlock(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
96 -{
97 - unsigned long old_state, new_state, ret_state, old_users;
98 - struct page_cache_descr *pg_cache_descr, *delete_pg_cache_descr = NULL;
99 - uint8_t we_locked;
100 -
101 - uv_mutex_unlock(&descr->pg_cache_descr->mutex);
102 -
103 - we_locked = 0;
104 - while (1) { /* spin */
105 - old_state = descr->pg_cache_descr_state;
106 - old_users = old_state >> PG_CACHE_DESCR_SHIFT;
107 -
108 - if (unlikely(we_locked)) {
109 - fatal_assert(0 == old_users);
110 -
111 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, old_state, 0);
112 - if (old_state == ret_state) {
113 - /* success */
114 - rrdeng_destroy_pg_cache_descr(ctx, delete_pg_cache_descr);
115 - return;
116 - }
117 - continue; /* spin */
118 - }
119 - if (old_state & PG_CACHE_DESCR_LOCKED) {
120 - fatal_assert(0 == old_users);
121 - continue; /* spin */
122 - }
123 - fatal_assert(old_state & PG_CACHE_DESCR_ALLOCATED);
124 - pg_cache_descr = descr->pg_cache_descr;
125 - /* caller is the only page cache descriptor user and there are no pending references on the page */
126 - if ((old_state & PG_CACHE_DESCR_DESTROY) && (1 == old_users) &&
127 - !pg_cache_descr->flags && !pg_cache_descr->refcnt) {
128 - fatal_assert(!pg_cache_descr->waiters);
129 -
130 - new_state = PG_CACHE_DESCR_LOCKED;
131 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, old_state, new_state);
132 - if (old_state == ret_state) {
133 - we_locked = 1;
134 - delete_pg_cache_descr = pg_cache_descr;
135 - descr->pg_cache_descr = NULL;
136 - /* retry */
137 - continue;
138 - }
139 - continue; /* spin */
140 - }
141 - fatal_assert(old_users > 0);
142 - new_state = (old_users - 1) << PG_CACHE_DESCR_SHIFT;
143 - new_state |= old_state & PG_CACHE_DESCR_FLAGS_MASK;
144 -
145 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, old_state, new_state);
146 - if (old_state == ret_state) {
147 - /* success */
148 - break;
149 - }
150 - /* spin */
151 - }
152 -}
153 -
154 -/*
155 - * Tries to deallocate page cache descriptor. If it fails, it postpones deallocation by setting the
156 - * PG_CACHE_DESCR_DESTROY flag which will be eventually cleared by a different context after doing
157 - * the deallocation.
158 - */
159 -void rrdeng_try_deallocate_pg_cache_descr(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
160 -{
161 - unsigned long old_state, new_state, ret_state, old_users;
162 - struct page_cache_descr *pg_cache_descr = NULL;
163 - uint8_t just_locked, can_free, must_unlock;
164 -
165 - just_locked = 0;
166 - can_free = 0;
167 - must_unlock = 0;
168 - while (1) { /* spin */
169 - old_state = descr->pg_cache_descr_state;
170 - old_users = old_state >> PG_CACHE_DESCR_SHIFT;
171 -
172 - if (unlikely(just_locked)) {
173 - fatal_assert(0 == old_users);
174 -
175 - must_unlock = 1;
176 - just_locked = 0;
177 - /* Try deallocate if there are no pending references on the page */
178 - if (!pg_cache_descr->flags && !pg_cache_descr->refcnt) {
179 - fatal_assert(!pg_cache_descr->waiters);
180 -
181 - descr->pg_cache_descr = NULL;
182 - can_free = 1;
183 - /* success */
184 - continue;
185 - }
186 - continue; /* spin */
187 - }
188 - if (unlikely(must_unlock)) {
189 - fatal_assert(0 == old_users);
190 -
191 - if (can_free) {
192 - /* success */
193 - new_state = 0;
194 - } else {
195 - new_state = old_state | PG_CACHE_DESCR_DESTROY;
196 - new_state &= ~PG_CACHE_DESCR_LOCKED;
197 - }
198 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, old_state, new_state);
199 - if (old_state == ret_state) {
200 - /* unlocked */
201 - if (can_free)
202 - rrdeng_destroy_pg_cache_descr(ctx, pg_cache_descr);
203 - return;
204 - }
205 - continue; /* spin */
206 - }
207 - if (!(old_state & PG_CACHE_DESCR_ALLOCATED)) {
208 - /* don't do anything */
209 - return;
210 - }
211 - if (old_state & PG_CACHE_DESCR_LOCKED) {
212 - fatal_assert(0 == old_users);
213 - continue; /* spin */
214 - }
215 - /* caller is the only page cache descriptor user */
216 - if (0 == old_users) {
217 - new_state = old_state | PG_CACHE_DESCR_LOCKED;
218 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, old_state, new_state);
219 - if (old_state == ret_state) {
220 - just_locked = 1;
221 - pg_cache_descr = descr->pg_cache_descr;
222 - /* retry */
223 - continue;
224 - }
225 - continue; /* spin */
226 - }
227 - if (old_state & PG_CACHE_DESCR_DESTROY) {
228 - /* don't do anything */
229 - return;
230 - }
231 - /* plant PG_CACHE_DESCR_DESTROY so that other contexts eventually free the page cache descriptor */
232 - new_state = old_state | PG_CACHE_DESCR_DESTROY;
233 -
234 - ret_state = ulong_compare_and_swap(&descr->pg_cache_descr_state, old_state, new_state);
235 - if (old_state == ret_state) {
236 - /* success */
237 - return;
238 - }
239 - /* spin */
240 - }
241 -}
\ No newline at end of file
database/engine/rrdenglocking.h deleted
-17
@@ -1,17 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -#ifndef NETDATA_RRDENGLOCKING_H
4 -#define NETDATA_RRDENGLOCKING_H
5 -
6 -#include "rrdengine.h"
7 -
8 -/* Forward declarations */
9 -struct page_cache_descr;
10 -
11 -struct page_cache_descr *rrdeng_create_pg_cache_descr(struct rrdengine_instance *ctx);
12 -void rrdeng_destroy_pg_cache_descr(struct rrdengine_instance *ctx, struct page_cache_descr *pg_cache_descr);
13 -void rrdeng_page_descr_mutex_lock(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
14 -void rrdeng_page_descr_mutex_unlock(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
15 -void rrdeng_try_deallocate_pg_cache_descr(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
16 -
17 -#endif /* NETDATA_RRDENGLOCKING_H */
\ No newline at end of file
database/ram/rrddim_mem.c
+242 -96
@@ -21,85 +21,209 @@ void rrddim_metrics_group_release(STORAGE_INSTANCE *db_instance __maybe_unused,
21 // ----------------------------------------------------------------------------
22 // RRDDIM legacy data collection functions
23
24 +struct mem_metric_handle {
25 + RRDDIM *rd;
26 +
27 + size_t counter;
28 + size_t entries;
29 + size_t current_entry;
30 + time_t last_updated_s;
31 + time_t update_every_s;
32 +
33 + int32_t refcount;
34 +};
35 +
36 +static void update_metric_handle_from_rrddim(struct mem_metric_handle *mh, RRDDIM *rd) {
37 + mh->counter = rd->rrdset->counter;
38 + mh->entries = rd->rrdset->entries;
39 + mh->current_entry = rd->rrdset->current_entry;
40 + mh->last_updated_s = rd->rrdset->last_updated.tv_sec;
41 + mh->update_every_s = rd->rrdset->update_every;
42 +}
43 +
44 +static void check_metric_handle_from_rrddim(struct mem_metric_handle *mh) {
45 + RRDDIM *rd = mh->rd; (void)rd;
46 + internal_fatal(mh->entries != (size_t)rd->rrdset->entries, "RRDDIM: entries do not match");
47 + internal_fatal(mh->update_every_s != rd->rrdset->update_every, "RRDDIM: update every does not match");
48 +}
49 +
50 STORAGE_METRIC_HANDLE *
51 rrddim_metric_get_or_create(RRDDIM *rd, STORAGE_INSTANCE *db_instance __maybe_unused) {
26 - STORAGE_METRIC_HANDLE *t = rrddim_metric_get(db_instance, &rd->metric_uuid);
27 - if(!t) {
52 + struct mem_metric_handle *mh = (struct mem_metric_handle *)rrddim_metric_get(db_instance, &rd->metric_uuid);
53 + while(!mh) {
54 netdata_rwlock_wrlock(&rrddim_JudyHS_rwlock);
55 Pvoid_t *PValue = JudyHSIns(&rrddim_JudyHS_array, &rd->metric_uuid, sizeof(uuid_t), PJE0);
30 - fatal_assert(NULL == *PValue);
31 - *PValue = rd;
32 - t = (STORAGE_METRIC_HANDLE *)rd;
56 + mh = *PValue;
57 + if(!mh) {
58 + mh = callocz(1, sizeof(struct mem_metric_handle));
59 + mh->rd = rd;
60 + mh->refcount = 1;
61 + update_metric_handle_from_rrddim(mh, rd);
62 + *PValue = mh;
63 + }
64 + else {
65 + if(__atomic_add_fetch(&mh->refcount, 1, __ATOMIC_RELAXED) <= 0)
66 + mh = NULL;
67 + }
68 netdata_rwlock_unlock(&rrddim_JudyHS_rwlock);
69 }
70
36 - if((RRDDIM *)t != rd)
37 - fatal("RRDDIM_MEM: incorrect pointer returned from index.");
71 + internal_fatal(mh->rd != rd, "RRDDIM_MEM: incorrect pointer returned from index.");
72
39 - return (STORAGE_METRIC_HANDLE *)rd;
73 + return (STORAGE_METRIC_HANDLE *)mh;
74 }
75
76 STORAGE_METRIC_HANDLE *
77 rrddim_metric_get(STORAGE_INSTANCE *db_instance __maybe_unused, uuid_t *uuid) {
44 - RRDDIM *rd = NULL;
78 + struct mem_metric_handle *mh = NULL;
79 netdata_rwlock_rdlock(&rrddim_JudyHS_rwlock);
80 Pvoid_t *PValue = JudyHSGet(rrddim_JudyHS_array, uuid, sizeof(uuid_t));
47 - if (likely(NULL != PValue))
48 - rd = *PValue;
81 + if (likely(NULL != PValue)) {
82 + mh = *PValue;
83 + if(__atomic_add_fetch(&mh->refcount, 1, __ATOMIC_RELAXED) <= 0)
84 + mh = NULL;
85 + }
86 netdata_rwlock_unlock(&rrddim_JudyHS_rwlock);
87
51 - return (STORAGE_METRIC_HANDLE *)rd;
88 + return (STORAGE_METRIC_HANDLE *)mh;
89 }
90
91 STORAGE_METRIC_HANDLE *rrddim_metric_dup(STORAGE_METRIC_HANDLE *db_metric_handle) {
92 + struct mem_metric_handle *mh = (struct mem_metric_handle *)db_metric_handle;
93 + __atomic_add_fetch(&mh->refcount, 1, __ATOMIC_RELAXED);
94 return db_metric_handle;
95 }
96
97 void rrddim_metric_release(STORAGE_METRIC_HANDLE *db_metric_handle __maybe_unused) {
59 - RRDDIM *rd = (RRDDIM *)db_metric_handle;
98 + struct mem_metric_handle *mh = (struct mem_metric_handle *)db_metric_handle;
99
61 - netdata_rwlock_wrlock(&rrddim_JudyHS_rwlock);
62 - JudyHSDel(&rrddim_JudyHS_array, &rd->metric_uuid, sizeof(uuid_t), PJE0);
63 - netdata_rwlock_unlock(&rrddim_JudyHS_rwlock);
100 + if(__atomic_sub_fetch(&mh->refcount, 1, __ATOMIC_RELAXED) == 0) {
101 + // we are the last one holding this
102 +
103 + int32_t expected = 0;
104 + if(__atomic_compare_exchange_n(&mh->refcount, &expected, -99999, false, __ATOMIC_RELAXED, __ATOMIC_RELAXED)) {
105 + // we can delete it
106 +
107 + RRDDIM *rd = mh->rd;
108 + netdata_rwlock_wrlock(&rrddim_JudyHS_rwlock);
109 + JudyHSDel(&rrddim_JudyHS_array, &rd->metric_uuid, sizeof(uuid_t), PJE0);
110 + netdata_rwlock_unlock(&rrddim_JudyHS_rwlock);
111 + }
112 + }
113 }
114
66 -void rrddim_store_metric_change_collection_frequency(STORAGE_COLLECT_HANDLE *collection_handle, int update_every __maybe_unused) {
115 +bool rrddim_metric_retention_by_uuid(STORAGE_INSTANCE *db_instance __maybe_unused, uuid_t *uuid, time_t *first_entry_s, time_t *last_entry_s) {
116 + STORAGE_METRIC_HANDLE *db_metric_handle = rrddim_metric_get(db_instance, uuid);
117 + if(!db_metric_handle)
118 + return false;
119 +
120 + *first_entry_s = rrddim_query_oldest_time_s(db_metric_handle);
121 + *last_entry_s = rrddim_query_latest_time_s(db_metric_handle);
122 +
123 + return true;
124 +}
125 +
126 +void rrddim_store_metric_change_collection_frequency(STORAGE_COLLECT_HANDLE *collection_handle, int update_every) {
127 + struct mem_collect_handle *ch = (struct mem_collect_handle *)collection_handle;
128 + struct mem_metric_handle *mh = (struct mem_metric_handle *)ch->db_metric_handle;
129 +
130 rrddim_store_metric_flush(collection_handle);
131 + mh->update_every_s = update_every;
132 }
133
134 STORAGE_COLLECT_HANDLE *rrddim_collect_init(STORAGE_METRIC_HANDLE *db_metric_handle, uint32_t update_every __maybe_unused, STORAGE_METRICS_GROUP *smg __maybe_unused) {
71 - RRDDIM *rd = (RRDDIM *)db_metric_handle;
72 - rd->db[rd->rrdset->current_entry] = pack_storage_number(NAN, SN_FLAG_NONE);
135 + struct mem_metric_handle *mh = (struct mem_metric_handle *)db_metric_handle;
136 + RRDDIM *rd = mh->rd;
137 +
138 + update_metric_handle_from_rrddim(mh, rd);
139 + internal_fatal(mh->update_every_s != update_every, "RRDDIM: update requested does not match the dimension");
140 +
141 struct mem_collect_handle *ch = callocz(1, sizeof(struct mem_collect_handle));
142 ch->rd = rd;
143 + ch->db_metric_handle = db_metric_handle;
144 +
145 return (STORAGE_COLLECT_HANDLE *)ch;
146 }
147
78 -void rrddim_collect_store_metric(STORAGE_COLLECT_HANDLE *collection_handle, usec_t point_in_time, NETDATA_DOUBLE number,
79 - NETDATA_DOUBLE min_value,
80 - NETDATA_DOUBLE max_value,
81 - uint16_t count,
82 - uint16_t anomaly_count,
83 - SN_FLAGS flags)
84 -{
85 - UNUSED(point_in_time);
86 - UNUSED(min_value);
87 - UNUSED(max_value);
88 - UNUSED(count);
89 - UNUSED(anomaly_count);
148 +void rrddim_store_metric_flush(STORAGE_COLLECT_HANDLE *collection_handle) {
149 + struct mem_collect_handle *ch = (struct mem_collect_handle *)collection_handle;
150 + struct mem_metric_handle *mh = (struct mem_metric_handle *)ch->db_metric_handle;
151 +
152 + RRDDIM *rd = mh->rd;
153 + size_t entries = mh->entries;
154 + storage_number empty = pack_storage_number(NAN, SN_FLAG_NONE);
155 +
156 + for(size_t i = 0; i < entries ;i++)
157 + rd->db[i] = empty;
158
159 + mh->counter = 0;
160 + mh->last_updated_s = 0;
161 + mh->current_entry = 0;
162 +}
163 +
164 +static inline void rrddim_fill_the_gap(STORAGE_COLLECT_HANDLE *collection_handle, time_t now_collect_s) {
165 struct mem_collect_handle *ch = (struct mem_collect_handle *)collection_handle;
92 - RRDDIM *rd = ch->rd;
93 - rd->db[rd->rrdset->current_entry] = pack_storage_number(number, flags);
166 + struct mem_metric_handle *mh = (struct mem_metric_handle *)ch->db_metric_handle;
167 +
168 + RRDDIM *rd = mh->rd;
169 +
170 + internal_fatal(ch->rd != mh->rd, "RRDDIM: dimensions do not match");
171 + check_metric_handle_from_rrddim(mh);
172 +
173 + size_t entries = mh->entries;
174 + time_t update_every_s = mh->update_every_s;
175 + time_t last_stored_s = mh->last_updated_s;
176 + size_t gap_entries = (now_collect_s - last_stored_s) / update_every_s;
177 + if(gap_entries >= entries)
178 + rrddim_store_metric_flush(collection_handle);
179 +
180 + else {
181 + storage_number empty = pack_storage_number(NAN, SN_FLAG_NONE);
182 + size_t current_entry = mh->current_entry;
183 + time_t now_store_s = last_stored_s + update_every_s;
184 +
185 + // fill the dimension
186 + size_t c;
187 + for(c = 0; c < entries && now_store_s <= now_collect_s ; now_store_s += update_every_s, c++) {
188 + rd->db[current_entry++] = empty;
189 +
190 + if(unlikely(current_entry >= entries))
191 + current_entry = 0;
192 + }
193 + mh->counter += c;
194 + mh->current_entry = current_entry;
195 + mh->last_updated_s = now_store_s;
196 + }
197 }
198
96 -void rrddim_store_metric_flush(STORAGE_COLLECT_HANDLE *collection_handle) {
199 +void rrddim_collect_store_metric(STORAGE_COLLECT_HANDLE *collection_handle,
200 + usec_t point_in_time_ut,
201 + NETDATA_DOUBLE number,
202 + NETDATA_DOUBLE min_value __maybe_unused,
203 + NETDATA_DOUBLE max_value __maybe_unused,
204 + uint16_t count __maybe_unused,
205 + uint16_t anomaly_count __maybe_unused,
206 + SN_FLAGS flags)
207 +{
208 struct mem_collect_handle *ch = (struct mem_collect_handle *)collection_handle;
209 + struct mem_metric_handle *mh = (struct mem_metric_handle *)ch->db_metric_handle;
210
211 RRDDIM *rd = ch->rd;
100 - for(int i = 0; i < rd->rrdset->entries ;i++)
101 - rd->db[i] = SN_EMPTY_SLOT;
212 + time_t point_in_time_s = (time_t)(point_in_time_ut / USEC_PER_SEC);
213
214 + internal_fatal(ch->rd != mh->rd, "RRDDIM: dimensions do not match");
215 + check_metric_handle_from_rrddim(mh);
216 +
217 + if(unlikely(point_in_time_s <= mh->last_updated_s))
218 + return;
219 +
220 + if(unlikely(mh->last_updated_s && point_in_time_s - mh->update_every_s > mh->last_updated_s))
221 + rrddim_fill_the_gap(collection_handle, point_in_time_s);
222 +
223 + rd->db[mh->current_entry] = pack_storage_number(number, flags);
224 + mh->counter++;
225 + mh->current_entry = (mh->current_entry + 1) >= mh->entries ? 0 : mh->current_entry + 1;
226 + mh->last_updated_s = point_in_time_s;
227 }
228
229 int rrddim_collect_finalize(STORAGE_COLLECT_HANDLE *collection_handle) {
@@ -109,41 +233,44 @@ int rrddim_collect_finalize(STORAGE_COLLECT_HANDLE *collection_handle) {
233
234 // ----------------------------------------------------------------------------
235
112 -// get the total duration in seconds of the round robin database
113 -#define rrddim_duration(st) (( (time_t)(rd)->rrdset->counter >= (time_t)(rd)->rrdset->entries ? (time_t)(rd)->rrdset->entries : (time_t)(rd)->rrdset->counter ) * (time_t)(rd)->rrdset->update_every)
236 +// get the total duration in seconds of the round-robin database
237 +#define metric_duration(mh) (( (time_t)(mh)->counter >= (time_t)(mh)->entries ? (time_t)(mh)->entries : (time_t)(mh)->counter ) * (time_t)(mh)->update_every_s)
238
115 -// get the last slot updated in the round robin database
116 -#define rrddim_last_slot(rd) ((size_t)(((rd)->rrdset->current_entry == 0) ? (rd)->rrdset->entries - 1 : (rd)->rrdset->current_entry - 1))
239 +// get the last slot updated in the round-robin database
240 +#define rrddim_last_slot(mh) ((size_t)(((mh)->current_entry == 0) ? (mh)->entries - 1 : (mh)->current_entry - 1))
241
242 // return the slot that has the oldest value
119 -#define rrddim_first_slot(rd) ((size_t)((rd)->rrdset->counter >= (size_t)(rd)->rrdset->entries ? (rd)->rrdset->current_entry : 0))
243 +#define rrddim_first_slot(mh) ((size_t)((mh)->counter >= (size_t)(mh)->entries ? (mh)->current_entry : 0))
244
121 -// get the slot of the round robin database, for the given timestamp (t)
122 -// it always returns a valid slot, although may not be for the time requested if the time is outside the round robin database
245 +// get the slot of the round-robin database, for the given timestamp (t)
246 +// it always returns a valid slot, although it may not be for the time requested if the time is outside the round-robin database
247 // only valid when not using dbengine
124 -static inline size_t rrddim_time2slot(RRDDIM *rd, time_t t) {
248 +static inline size_t rrddim_time2slot(STORAGE_METRIC_HANDLE *db_metric_handle, time_t t) {
249 + struct mem_metric_handle *mh = (struct mem_metric_handle *)db_metric_handle;
250 + RRDDIM *rd = mh->rd;
251 +
252 size_t ret = 0;
126 - time_t last_entry_t = rrddim_query_latest_time((STORAGE_METRIC_HANDLE *)rd);
127 - time_t first_entry_t = rrddim_query_oldest_time((STORAGE_METRIC_HANDLE *)rd);
128 - size_t entries = rd->rrdset->entries;
129 - size_t first_slot = rrddim_first_slot(rd);
130 - size_t last_slot = rrddim_last_slot(rd);
131 - size_t update_every = rd->rrdset->update_every;
132 -
133 - if(t >= last_entry_t) {
253 + time_t last_entry_s = rrddim_query_latest_time_s(db_metric_handle);
254 + time_t first_entry_s = rrddim_query_oldest_time_s(db_metric_handle);
255 + size_t entries = mh->entries;
256 + size_t first_slot = rrddim_first_slot(mh);
257 + size_t last_slot = rrddim_last_slot(mh);
258 + size_t update_every = mh->update_every_s;
259 +
260 + if(t >= last_entry_s) {
261 // the requested time is after the last entry we have
262 ret = last_slot;
263 }
264 else {
138 - if(t <= first_entry_t) {
265 + if(t <= first_entry_s) {
266 // the requested time is before the first entry we have
267 ret = first_slot;
268 }
269 else {
143 - if(last_slot >= (size_t)((last_entry_t - t) / update_every))
144 - ret = last_slot - ((last_entry_t - t) / update_every);
270 + if(last_slot >= (size_t)((last_entry_s - t) / update_every))
271 + ret = last_slot - ((last_entry_s - t) / update_every);
272 else
146 - ret = last_slot - ((last_entry_t - t) / update_every) + entries;
273 + ret = last_slot - ((last_entry_s - t) / update_every) + entries;
274 }
275 }
276
@@ -155,15 +282,18 @@ static inline size_t rrddim_time2slot(RRDDIM *rd, time_t t) {
282 return ret;
283 }
284
158 -// get the timestamp of a specific slot in the round robin database
285 +// get the timestamp of a specific slot in the round-robin database
286 // only valid when not using dbengine
160 -static inline time_t rrddim_slot2time(RRDDIM *rd, size_t slot) {
287 +static inline time_t rrddim_slot2time(STORAGE_METRIC_HANDLE *db_metric_handle, size_t slot) {
288 + struct mem_metric_handle *mh = (struct mem_metric_handle *)db_metric_handle;
289 + RRDDIM *rd = mh->rd;
290 +
291 time_t ret;
162 - time_t last_entry_t = rrddim_query_latest_time((STORAGE_METRIC_HANDLE *)rd);
163 - time_t first_entry_t = rrddim_query_oldest_time((STORAGE_METRIC_HANDLE *)rd);
164 - size_t entries = rd->rrdset->entries;
165 - size_t last_slot = rrddim_last_slot(rd);
166 - size_t update_every = rd->rrdset->update_every;
292 + time_t last_entry_s = rrddim_query_latest_time_s(db_metric_handle);
293 + time_t first_entry_s = rrddim_query_oldest_time_s(db_metric_handle);
294 + size_t entries = mh->entries;
295 + size_t last_slot = rrddim_last_slot(mh);
296 + size_t update_every = mh->update_every_s;
297
298 if(slot >= entries) {
299 error("INTERNAL ERROR: caller of rrddim_slot2time() gives invalid slot %zu", slot);
@@ -171,18 +301,22 @@ static inline time_t rrddim_slot2time(RRDDIM *rd, size_t slot) {
301 }
302
303 if(slot > last_slot)
174 - ret = last_entry_t - (time_t)(update_every * (last_slot - slot + entries));
304 + ret = last_entry_s - (time_t)(update_every * (last_slot - slot + entries));
305 else
176 - ret = last_entry_t - (time_t)(update_every * (last_slot - slot));
306 + ret = last_entry_s - (time_t)(update_every * (last_slot - slot));
307 +
308 + if(unlikely(ret < first_entry_s)) {
309 + error("INTERNAL ERROR: rrddim_slot2time() on dimension '%s' of chart '%s' returned time (%ld) too far in the past (before first_entry_s %ld) for slot %zu",
310 + rrddim_name(rd), rrdset_id(rd->rrdset), ret, first_entry_s, slot);
311
178 - if(unlikely(ret < first_entry_t)) {
179 - error("INTERNAL ERROR: rrddim_slot2time() on %s returns time too far in the past", rrddim_name(rd));
180 - ret = first_entry_t;
312 + ret = first_entry_s;
313 }
314
183 - if(unlikely(ret > last_entry_t)) {
184 - error("INTERNAL ERROR: rrddim_slot2time() on %s returns time into the future", rrddim_name(rd));
185 - ret = last_entry_t;
315 + if(unlikely(ret > last_entry_s)) {
316 + error("INTERNAL ERROR: rrddim_slot2time() on dimension '%s' of chart '%s' returned time (%ld) too far into the future (after last_entry_s %ld) for slot %zu",
317 + rrddim_name(rd), rrdset_id(rd->rrdset), ret, last_entry_s, slot);
318 +
319 + ret = last_entry_s;
320 }
321
322 return ret;
@@ -191,20 +325,24 @@ static inline time_t rrddim_slot2time(RRDDIM *rd, size_t slot) {
325 // ----------------------------------------------------------------------------
326 // RRDDIM legacy database query functions
327
194 -void rrddim_query_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *handle, time_t start_time, time_t end_time) {
195 - RRDDIM *rd = (RRDDIM *)db_metric_handle;
328 +void rrddim_query_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *handle, time_t start_time_s, time_t end_time_s, STORAGE_PRIORITY priority __maybe_unused) {
329 + struct mem_metric_handle *mh = (struct mem_metric_handle *)db_metric_handle;
330
197 - handle->rd = rd;
198 - handle->start_time_s = start_time;
199 - handle->end_time_s = end_time;
331 + check_metric_handle_from_rrddim(mh);
332 +
333 + handle->start_time_s = start_time_s;
334 + handle->end_time_s = end_time_s;
335 + handle->priority = priority;
336 struct mem_query_handle* h = mallocz(sizeof(struct mem_query_handle));
201 - h->slot = rrddim_time2slot(rd, start_time);
202 - h->last_slot = rrddim_time2slot(rd, end_time);
203 - h->dt = rd->rrdset->update_every;
337 + h->db_metric_handle = db_metric_handle;
338 +
339 + h->slot = rrddim_time2slot(db_metric_handle, start_time_s);
340 + h->last_slot = rrddim_time2slot(db_metric_handle, end_time_s);
341 + h->dt = mh->update_every_s;
342
205 - h->next_timestamp = start_time;
206 - h->slot_timestamp = rrddim_slot2time(rd, h->slot);
207 - h->last_timestamp = rrddim_slot2time(rd, h->last_slot);
343 + h->next_timestamp = start_time_s;
344 + h->slot_timestamp = rrddim_slot2time(db_metric_handle, h->slot);
345 + h->last_timestamp = rrddim_slot2time(db_metric_handle, h->last_slot);
346
347 // info("RRDDIM QUERY INIT: start %ld, end %ld, next %ld, first %ld, last %ld, dt %ld", start_time, end_time, h->next_timestamp, h->slot_timestamp, h->last_timestamp, h->dt);
348
@@ -215,9 +353,11 @@ void rrddim_query_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_e
353 // IT IS REQUIRED TO **ALWAYS** SET ALL RETURN VALUES (current_time, end_time, flags)
354 // IT IS REQUIRED TO **ALWAYS** KEEP TRACK OF TIME, EVEN OUTSIDE THE DATABASE BOUNDARIES
355 STORAGE_POINT rrddim_query_next_metric(struct storage_engine_query_handle *handle) {
218 - RRDDIM *rd = handle->rd;
356 struct mem_query_handle* h = (struct mem_query_handle*)handle->handle;
220 - size_t entries = rd->rrdset->entries;
357 + struct mem_metric_handle *mh = (struct mem_metric_handle *)h->db_metric_handle;
358 + RRDDIM *rd = mh->rd;
359 +
360 + size_t entries = mh->entries;
361 size_t slot = h->slot;
362
363 STORAGE_POINT sp;
@@ -227,16 +367,16 @@ STORAGE_POINT rrddim_query_next_metric(struct storage_engine_query_handle *handl
367 h->next_timestamp += h->dt;
368
369 // set this timestamp for our caller
230 - sp.start_time = this_timestamp - h->dt;
231 - sp.end_time = this_timestamp;
370 + sp.start_time_s = this_timestamp - h->dt;
371 + sp.end_time_s = this_timestamp;
372
373 if(unlikely(this_timestamp < h->slot_timestamp)) {
234 - storage_point_empty(sp, sp.start_time, sp.end_time);
374 + storage_point_empty(sp, sp.start_time_s, sp.end_time_s);
375 return sp;
376 }
377
378 if(unlikely(this_timestamp > h->last_timestamp)) {
239 - storage_point_empty(sp, sp.start_time, sp.end_time);
379 + storage_point_empty(sp, sp.start_time_s, sp.end_time_s);
380 return sp;
381 }
382
@@ -254,24 +394,30 @@ STORAGE_POINT rrddim_query_next_metric(struct storage_engine_query_handle *handl
394 }
395
396 int rrddim_query_is_finished(struct storage_engine_query_handle *handle) {
257 - struct mem_query_handle* h = (struct mem_query_handle*)handle->handle;
397 + struct mem_query_handle *h = (struct mem_query_handle*)handle->handle;
398 return (h->next_timestamp > handle->end_time_s);
399 }
400
401 void rrddim_query_finalize(struct storage_engine_query_handle *handle) {
402 #ifdef NETDATA_INTERNAL_CHECKS
403 + struct mem_query_handle *h = (struct mem_query_handle*)handle->handle;
404 + struct mem_metric_handle *mh = (struct mem_metric_handle *)h->db_metric_handle;
405 if(!rrddim_query_is_finished(handle))
264 - error("QUERY: query for chart '%s' dimension '%s' has been stopped unfinished", rrdset_id(handle->rd->rrdset), rrddim_name(handle->rd));
406 + error("QUERY: query for chart '%s' dimension '%s' has been stopped unfinished", rrdset_id(mh->rd->rrdset), rrddim_name(mh->rd));
407 #endif
408 freez(handle->handle);
409 }
410
269 -time_t rrddim_query_latest_time(STORAGE_METRIC_HANDLE *db_metric_handle) {
270 - RRDDIM *rd = (RRDDIM *)db_metric_handle;
271 - return rd->rrdset->last_updated.tv_sec;
411 +time_t rrddim_query_align_to_optimal_before(struct storage_engine_query_handle *rrddim_handle) {
412 + return rrddim_handle->end_time_s;
413 +}
414 +
415 +time_t rrddim_query_latest_time_s(STORAGE_METRIC_HANDLE *db_metric_handle) {
416 + struct mem_metric_handle *mh = (struct mem_metric_handle *)db_metric_handle;
417 + return mh->last_updated_s;
418 }
419
274 -time_t rrddim_query_oldest_time(STORAGE_METRIC_HANDLE *db_metric_handle) {
275 - RRDDIM *rd = (RRDDIM *)db_metric_handle;
276 - return (time_t)(rd->rrdset->last_updated.tv_sec - rrddim_duration(rd));
420 +time_t rrddim_query_oldest_time_s(STORAGE_METRIC_HANDLE *db_metric_handle) {
421 + struct mem_metric_handle *mh = (struct mem_metric_handle *)db_metric_handle;
422 + return (time_t)(mh->last_updated_s - metric_duration(mh));
423 }
database/ram/rrddim_mem.h
+9 -6
@@ -6,12 +6,12 @@
6 #include "database/rrd.h"
7
8 struct mem_collect_handle {
9 + STORAGE_METRIC_HANDLE *db_metric_handle;
10 RRDDIM *rd;
10 - long slot;
11 - long entries;
11 };
12
13 struct mem_query_handle {
14 + STORAGE_METRIC_HANDLE *db_metric_handle;
15 time_t dt;
16 time_t next_timestamp;
17 time_t last_timestamp;
@@ -25,12 +25,14 @@ STORAGE_METRIC_HANDLE *rrddim_metric_get(STORAGE_INSTANCE *db_instance, uuid_t *
25 STORAGE_METRIC_HANDLE *rrddim_metric_dup(STORAGE_METRIC_HANDLE *db_metric_handle);
26 void rrddim_metric_release(STORAGE_METRIC_HANDLE *db_metric_handle);
27
28 +bool rrddim_metric_retention_by_uuid(STORAGE_INSTANCE *db_instance, uuid_t *uuid, time_t *first_entry_s, time_t *last_entry_s);
29 +
30 STORAGE_METRICS_GROUP *rrddim_metrics_group_get(STORAGE_INSTANCE *db_instance, uuid_t *uuid);
31 void rrddim_metrics_group_release(STORAGE_INSTANCE *db_instance, STORAGE_METRICS_GROUP *smg);
32
33 STORAGE_COLLECT_HANDLE *rrddim_collect_init(STORAGE_METRIC_HANDLE *db_metric_handle, uint32_t update_every, STORAGE_METRICS_GROUP *smg);
34 void rrddim_store_metric_change_collection_frequency(STORAGE_COLLECT_HANDLE *collection_handle, int update_every);
33 -void rrddim_collect_store_metric(STORAGE_COLLECT_HANDLE *collection_handle, usec_t point_in_time, NETDATA_DOUBLE number,
35 +void rrddim_collect_store_metric(STORAGE_COLLECT_HANDLE *collection_handle, usec_t point_in_time_ut, NETDATA_DOUBLE number,
36 NETDATA_DOUBLE min_value,
37 NETDATA_DOUBLE max_value,
38 uint16_t count,
@@ -39,11 +41,12 @@ void rrddim_collect_store_metric(STORAGE_COLLECT_HANDLE *collection_handle, usec
41 void rrddim_store_metric_flush(STORAGE_COLLECT_HANDLE *collection_handle);
42 int rrddim_collect_finalize(STORAGE_COLLECT_HANDLE *collection_handle);
43
42 -void rrddim_query_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *handle, time_t start_time, time_t end_time);
44 +void rrddim_query_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *handle, time_t start_time_s, time_t end_time_s, STORAGE_PRIORITY priority);
45 STORAGE_POINT rrddim_query_next_metric(struct storage_engine_query_handle *handle);
46 int rrddim_query_is_finished(struct storage_engine_query_handle *handle);
47 void rrddim_query_finalize(struct storage_engine_query_handle *handle);
46 -time_t rrddim_query_latest_time(STORAGE_METRIC_HANDLE *db_metric_handle);
47 -time_t rrddim_query_oldest_time(STORAGE_METRIC_HANDLE *db_metric_handle);
48 +time_t rrddim_query_latest_time_s(STORAGE_METRIC_HANDLE *db_metric_handle);
49 +time_t rrddim_query_oldest_time_s(STORAGE_METRIC_HANDLE *db_metric_handle);
50 +time_t rrddim_query_align_to_optimal_before(struct storage_engine_query_handle *rrddim_handle);
51
52 #endif
database/rrd.h
+131 -191
@@ -34,8 +34,8 @@ typedef struct ml_host ml_host_t;
34 typedef struct ml_chart ml_chart_t;
35 typedef struct ml_dimension ml_dimension_t;
36
37 -typedef enum {
38 - QUERY_SOURCE_UNKNOWN,
37 +typedef enum __attribute__ ((__packed__)) {
38 + QUERY_SOURCE_UNKNOWN = 0,
39 QUERY_SOURCE_API_DATA,
40 QUERY_SOURCE_API_BADGE,
41 QUERY_SOURCE_API_WEIGHTS,
@@ -44,6 +44,16 @@ typedef enum {
44 QUERY_SOURCE_UNITTEST,
45 } QUERY_SOURCE;
46
47 +typedef enum __attribute__ ((__packed__)) storage_priority {
48 + STORAGE_PRIORITY_CRITICAL = 0,
49 + STORAGE_PRIORITY_HIGH,
50 + STORAGE_PRIORITY_NORMAL,
51 + STORAGE_PRIORITY_LOW,
52 + STORAGE_PRIORITY_BEST_EFFORT,
53 +
54 + STORAGE_PRIO_MAX_DONT_USE,
55 +} STORAGE_PRIORITY;
56 +
57 // forward declarations
58 struct rrddim_tier;
59
@@ -53,6 +63,32 @@ struct rrdengine_instance;
63 struct pg_cache_page_index;
64 #endif
65
66 +// ----------------------------------------------------------------------------
67 +// memory mode
68 +
69 +typedef enum __attribute__ ((__packed__)) rrd_memory_mode {
70 + RRD_MEMORY_MODE_NONE = 0,
71 + RRD_MEMORY_MODE_RAM = 1,
72 + RRD_MEMORY_MODE_MAP = 2,
73 + RRD_MEMORY_MODE_SAVE = 3,
74 + RRD_MEMORY_MODE_ALLOC = 4,
75 + RRD_MEMORY_MODE_DBENGINE = 5,
76 +
77 + // this is 8-bit
78 +} RRD_MEMORY_MODE;
79 +
80 +#define RRD_MEMORY_MODE_NONE_NAME "none"
81 +#define RRD_MEMORY_MODE_RAM_NAME "ram"
82 +#define RRD_MEMORY_MODE_MAP_NAME "map"
83 +#define RRD_MEMORY_MODE_SAVE_NAME "save"
84 +#define RRD_MEMORY_MODE_ALLOC_NAME "alloc"
85 +#define RRD_MEMORY_MODE_DBENGINE_NAME "dbengine"
86 +
87 +extern RRD_MEMORY_MODE default_rrd_memory_mode;
88 +
89 +const char *rrd_memory_mode_name(RRD_MEMORY_MODE id);
90 +RRD_MEMORY_MODE rrd_memory_mode_id(const char *name);
91 +
92 #include "daemon/common.h"
93 #include "web/api/queries/query.h"
94 #include "web/api/queries/rrdr.h"
@@ -64,6 +100,32 @@ struct pg_cache_page_index;
100 #include "streaming/rrdpush.h"
101 #include "aclk/aclk_rrdhost_state.h"
102 #include "sqlite/sqlite_health.h"
103 +
104 +typedef struct storage_query_handle STORAGE_QUERY_HANDLE;
105 +
106 +// iterator state for RRD dimension data queries
107 +struct storage_engine_query_handle {
108 + time_t start_time_s;
109 + time_t end_time_s;
110 + STORAGE_PRIORITY priority;
111 + STORAGE_QUERY_HANDLE* handle;
112 +};
113 +
114 +typedef struct storage_point {
115 + NETDATA_DOUBLE min; // when count > 1, this is the minimum among them
116 + NETDATA_DOUBLE max; // when count > 1, this is the maximum among them
117 + NETDATA_DOUBLE sum; // the point sum - divided by count gives the average
118 +
119 + // end_time - start_time = point duration
120 + time_t start_time_s; // the time the point starts
121 + time_t end_time_s; // the time the point ends
122 +
123 + unsigned count; // the number of original points aggregated
124 + unsigned anomaly_count; // the number of original points found anomalous
125 +
126 + SN_FLAGS flags; // flags stored with the point
127 +} STORAGE_POINT;
128 +
129 #include "rrdcontext.h"
130
131 extern bool unittest_running;
@@ -71,27 +133,14 @@ extern bool dbengine_enabled;
133 extern size_t storage_tiers;
134 extern size_t storage_tiers_grouping_iterations[RRD_STORAGE_TIERS];
135
74 -typedef enum {
75 - RRD_BACKFILL_NONE,
136 +typedef enum __attribute__ ((__packed__)) {
137 + RRD_BACKFILL_NONE = 0,
138 RRD_BACKFILL_FULL,
139 RRD_BACKFILL_NEW
140 } RRD_BACKFILL;
141
142 extern RRD_BACKFILL storage_tiers_backfill[RRD_STORAGE_TIERS];
143
82 -enum {
83 - CONTEXT_FLAGS_ARCHIVE = 0x01,
84 - CONTEXT_FLAGS_CHART = 0x02,
85 - CONTEXT_FLAGS_CONTEXT = 0x04
86 -};
87 -
88 -struct context_param {
89 - RRDDIM *rd;
90 - time_t first_entry_t;
91 - time_t last_entry_t;
92 - uint8_t flags;
93 -};
94 -
144 #define UPDATE_EVERY 1
145 #define UPDATE_EVERY_MAX 3600
146
@@ -101,7 +150,12 @@ struct context_param {
150 extern int default_rrd_update_every;
151 extern int default_rrd_history_entries;
152 extern int gap_when_lost_iterations_above;
104 -extern time_t rrdset_free_obsolete_time;
153 +extern time_t rrdset_free_obsolete_time_s;
154 +
155 +#define MIN_LIBUV_WORKER_THREADS 16
156 +#define MAX_LIBUV_WORKER_THREADS 128
157 +#define RESERVED_LIBUV_WORKER_THREADS 6
158 +extern int libuv_worker_threads;
159
160 #define RRD_ID_LENGTH_MAX 200
161
@@ -111,10 +165,10 @@ typedef long long total_number;
165 // ----------------------------------------------------------------------------
166 // chart types
167
114 -typedef enum rrdset_type {
168 +typedef enum __attribute__ ((__packed__)) rrdset_type {
169 RRDSET_TYPE_LINE = 0,
170 RRDSET_TYPE_AREA = 1,
117 - RRDSET_TYPE_STACKED = 2
171 + RRDSET_TYPE_STACKED = 2,
172 } RRDSET_TYPE;
173
174 #define RRDSET_TYPE_LINE_NAME "line"
@@ -125,37 +179,10 @@ RRDSET_TYPE rrdset_type_id(const char *name);
179 const char *rrdset_type_name(RRDSET_TYPE chart_type);
180
181
128 -// ----------------------------------------------------------------------------
129 -// memory mode
130 -
131 -typedef enum rrd_memory_mode {
132 - RRD_MEMORY_MODE_NONE = 0,
133 - RRD_MEMORY_MODE_RAM = 1,
134 - RRD_MEMORY_MODE_MAP = 2,
135 - RRD_MEMORY_MODE_SAVE = 3,
136 - RRD_MEMORY_MODE_ALLOC = 4,
137 - RRD_MEMORY_MODE_DBENGINE = 5,
138 -
139 - // this is 8-bit
140 -} RRD_MEMORY_MODE;
141 -
142 -#define RRD_MEMORY_MODE_NONE_NAME "none"
143 -#define RRD_MEMORY_MODE_RAM_NAME "ram"
144 -#define RRD_MEMORY_MODE_MAP_NAME "map"
145 -#define RRD_MEMORY_MODE_SAVE_NAME "save"
146 -#define RRD_MEMORY_MODE_ALLOC_NAME "alloc"
147 -#define RRD_MEMORY_MODE_DBENGINE_NAME "dbengine"
148 -
149 -extern RRD_MEMORY_MODE default_rrd_memory_mode;
150 -
151 -const char *rrd_memory_mode_name(RRD_MEMORY_MODE id);
152 -RRD_MEMORY_MODE rrd_memory_mode_id(const char *name);
153 -
154 -
182 // ----------------------------------------------------------------------------
183 // algorithms types
184
158 -typedef enum rrd_algorithm {
185 +typedef enum __attribute__ ((__packed__)) rrd_algorithm {
186 RRD_ALGORITHM_ABSOLUTE = 0,
187 RRD_ALGORITHM_INCREMENTAL = 1,
188 RRD_ALGORITHM_PCENT_OVER_DIFF_TOTAL = 2,
@@ -186,7 +213,7 @@ DICTIONARY *rrdfamily_rrdvars_dict(const RRDFAMILY_ACQUIRED *rf);
213 // flags & options
214
215 // options are permanent configuration options (no atomics to alter/access them)
189 -typedef enum rrddim_options {
216 +typedef enum __attribute__ ((__packed__)) rrddim_options {
217 RRDDIM_OPTION_NONE = 0,
218 RRDDIM_OPTION_HIDDEN = (1 << 0), // this dimension will not be offered to callers
219 RRDDIM_OPTION_DONT_DETECT_RESETS_OR_OVERFLOWS = (1 << 1), // do not offer RESET or OVERFLOW info to callers
@@ -200,7 +227,7 @@ typedef enum rrddim_options {
227 #define rrddim_option_clear(rd, option) (rd)->options &= ~(option)
228
229 // flags are runtime changing status flags (atomics are required to alter/access them)
203 -typedef enum rrddim_flags {
230 +typedef enum __attribute__ ((__packed__)) rrddim_flags {
231 RRDDIM_FLAG_NONE = 0,
232 RRDDIM_FLAG_PENDING_HEALTH_INITIALIZATION = (1 << 0),
233
@@ -219,7 +246,7 @@ typedef enum rrddim_flags {
246 #define rrddim_flag_set(rd, flag) __atomic_or_fetch(&((rd)->flags), (flag), __ATOMIC_SEQ_CST)
247 #define rrddim_flag_clear(rd, flag) __atomic_and_fetch(&((rd)->flags), ~(flag), __ATOMIC_SEQ_CST)
248
222 -typedef enum rrdlabel_source {
249 +typedef enum __attribute__ ((__packed__)) rrdlabel_source {
250 RRDLABEL_SRC_AUTO = (1 << 0), // set when Netdata found the label by some automation
251 RRDLABEL_SRC_CONFIG = (1 << 1), // set when the user configured the label
252 RRDLABEL_SRC_K8S = (1 << 2), // set when this label is found from k8s (RRDLABEL_SRC_AUTO should also be set)
@@ -279,10 +306,10 @@ struct rrddim {
306 STRING *id; // the id of this dimension (for internal identification)
307 STRING *name; // the name of this dimension (as presented to user)
308
282 - RRD_ALGORITHM algorithm:8; // the algorithm that is applied to add new collected values
283 - RRDDIM_OPTIONS options:8; // permanent configuration options
284 - RRD_MEMORY_MODE rrd_memory_mode:8; // the memory mode for this dimension
285 - /*RRDDIM_FLAGS*/ uint8_t flags; // run time changing status flags
309 + RRD_ALGORITHM algorithm; // the algorithm that is applied to add new collected values
310 + RRDDIM_OPTIONS options; // permanent configuration options
311 + RRD_MEMORY_MODE rrd_memory_mode; // the memory mode for this dimension
312 + RRDDIM_FLAGS flags; // run time changing status flags
313
314 bool updated; // 1 when the dimension has been updated since the last processing
315 bool exposed; // 1 when set what have sent this dimension to the central netdata
@@ -362,37 +389,22 @@ void rrddim_memory_file_save(RRDDIM *rd);
389
390 // ----------------------------------------------------------------------------
391
365 -typedef struct storage_point {
366 - NETDATA_DOUBLE min; // when count > 1, this is the minimum among them
367 - NETDATA_DOUBLE max; // when count > 1, this is the maximum among them
368 - NETDATA_DOUBLE sum; // the point sum - divided by count gives the average
369 -
370 - // end_time - start_time = point duration
371 - time_t start_time; // the time the point starts
372 - time_t end_time; // the time the point ends
373 -
374 - unsigned count; // the number of original points aggregated
375 - unsigned anomaly_count; // the number of original points found anomalous
376 -
377 - SN_FLAGS flags; // flags stored with the point
378 -} STORAGE_POINT;
379 -
392 #define storage_point_unset(x) do { \
393 (x).min = (x).max = (x).sum = NAN; \
394 (x).count = 0; \
395 (x).anomaly_count = 0; \
396 (x).flags = SN_FLAG_NONE; \
385 - (x).start_time = 0; \
386 - (x).end_time = 0; \
397 + (x).start_time_s = 0; \
398 + (x).end_time_s = 0; \
399 } while(0)
400
389 -#define storage_point_empty(x, start_t, end_t) do { \
401 +#define storage_point_empty(x, start_s, end_s) do { \
402 (x).min = (x).max = (x).sum = NAN; \
403 (x).count = 1; \
404 (x).anomaly_count = 0; \
405 (x).flags = SN_FLAG_NONE; \
394 - (x).start_time = start_t; \
395 - (x).end_time = end_t; \
406 + (x).start_time_s = start_s; \
407 + (x).end_time_s = end_s; \
408 } while(0)
409
410 #define storage_point_is_unset(x) (!(x).count)
@@ -402,10 +414,6 @@ typedef struct storage_point {
414 // engine-specific iterator state for dimension data collection
415 typedef struct storage_collect_handle STORAGE_COLLECT_HANDLE;
416
405 -// ----------------------------------------------------------------------------
406 -// engine-specific iterator state for dimension data queries
407 -typedef struct storage_query_handle STORAGE_QUERY_HANDLE;
408 -
417 // ------------------------------------------------------------------------
418 // function pointers that handle data collection
419 struct storage_engine_collect_ops {
@@ -430,18 +438,11 @@ struct storage_engine_collect_ops {
438 };
439
440 // ----------------------------------------------------------------------------
433 -// iterator state for RRD dimension data queries
434 -struct storage_engine_query_handle {
435 - RRDDIM *rd;
436 - time_t start_time_s;
437 - time_t end_time_s;
438 - STORAGE_QUERY_HANDLE* handle;
439 -};
441
442 // function pointers that handle database queries
443 struct storage_engine_query_ops {
444 // run this before starting a series of next_metric() database queries
444 - void (*init)(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *handle, time_t start_time, time_t end_time);
445 + void (*init)(STORAGE_METRIC_HANDLE *db_metric_handle, struct storage_engine_query_handle *handle, time_t start_time_s, time_t end_time_s, STORAGE_PRIORITY priority);
446
447 // run this to load each metric number from the database
448 STORAGE_POINT (*next_metric)(struct storage_engine_query_handle *handle);
@@ -453,10 +454,14 @@ struct storage_engine_query_ops {
454 void (*finalize)(struct storage_engine_query_handle *handle);
455
456 // get the timestamp of the last entry of this metric
456 - time_t (*latest_time)(STORAGE_METRIC_HANDLE *db_metric_handle);
457 + time_t (*latest_time_s)(STORAGE_METRIC_HANDLE *db_metric_handle);
458
459 // get the timestamp of the first entry of this metric
459 - time_t (*oldest_time)(STORAGE_METRIC_HANDLE *db_metric_handle);
460 + time_t (*oldest_time_s)(STORAGE_METRIC_HANDLE *db_metric_handle);
461 +
462 + // adapt 'before' timestamp to the optimal for the query
463 + // can only move 'before' ahead (to the future)
464 + time_t (*align_to_optimal_before)(struct storage_engine_query_handle *handle);
465 };
466
467 typedef struct storage_engine STORAGE_ENGINE;
@@ -469,6 +474,7 @@ typedef struct storage_engine_api {
474 STORAGE_METRIC_HANDLE *(*metric_get_or_create)(RRDDIM *rd, STORAGE_INSTANCE *instance);
475 void (*metric_release)(STORAGE_METRIC_HANDLE *);
476 STORAGE_METRIC_HANDLE *(*metric_dup)(STORAGE_METRIC_HANDLE *);
477 + bool (*metric_retention_by_uuid)(STORAGE_INSTANCE *db_instance, uuid_t *uuid, time_t *first_entry_s, time_t *last_entry_s);
478
479 // operations
480 struct storage_engine_collect_ops collect_ops;
@@ -492,12 +498,12 @@ struct rrddim_tier {
498 STORAGE_METRIC_HANDLE *db_metric_handle; // the metric handle inside the database
499 STORAGE_COLLECT_HANDLE *db_collection_handle; // the data collection handle
500 STORAGE_POINT virtual_point;
495 - time_t next_point_time;
501 + time_t next_point_time_s;
502 struct storage_engine_collect_ops *collect_ops;
503 struct storage_engine_query_ops *query_ops;
504 };
505
500 -void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, size_t tier, time_t now);
506 +void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, size_t tier, time_t now_s);
507
508 // ----------------------------------------------------------------------------
509 // these loop macros make sure the linked list is accessed with the right lock
@@ -521,7 +527,7 @@ void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, size_t tier, time_t now);
527 // flags are set/unset in a manner that is not thread safe
528 // and may lead to missing information.
529
524 -typedef enum rrdset_flags {
530 +typedef enum __attribute__ ((__packed__)) rrdset_flags {
531 RRDSET_FLAG_DETAIL = (1 << 1), // if set, the data set should be considered as a detail of another
532 // (the master data set should be the one that has the same family and is not detail)
533 RRDSET_FLAG_DEBUG = (1 << 2), // enables or disables debugging for a chart
@@ -606,10 +612,6 @@ struct rrdset {
612
613 DICTIONARY *rrddim_root_index; // dimensions index
614
609 - int gap_when_lost_iterations_above; // after how many lost iterations a gap should be stored
610 - // netdata will interpolate values for gaps lower than this
611 - // TODO - use the global - all charts have the same value
612 -
615 STORAGE_METRICS_GROUP *storage_metrics_groups[RRD_STORAGE_TIERS];
616
617 // ------------------------------------------------------------------------
@@ -623,10 +625,12 @@ struct rrdset {
625 // ------------------------------------------------------------------------
626 // data collection members
627
628 + SPINLOCK data_collection_lock;
629 +
630 size_t counter; // the number of times we added values to this database
631 size_t counter_done; // the number of times rrdset_done() has been called
632
629 - time_t last_accessed_time; // the last time this RRDSET has been accessed
633 + time_t last_accessed_time_s; // the last time this RRDSET has been accessed
634
635 usec_t usec_since_last_update; // the time in microseconds since the last collection of data
636
@@ -640,7 +644,7 @@ struct rrdset {
644 // ------------------------------------------------------------------------
645 // data collection - streaming to parents, temp variables
646
643 - time_t upstream_resync_time; // the timestamp up to which we should resync clock upstream
647 + time_t upstream_resync_time_s; // the timestamp up to which we should resync clock upstream
648
649 // ------------------------------------------------------------------------
650 // db mode SAVE, MAP specifics
@@ -648,7 +652,6 @@ struct rrdset {
652 // (RRDSET_DB_STATE ptr to an undefined structure, and a call to clean this up during destruction)
653
654 char *cache_dir; // the directory to store dimensions
651 - unsigned long memsize; // how much mem we have allocated for this (without dimensions)
655 void *st_on_file; // compatibility with V019 RRDSET files
656
657 // ------------------------------------------------------------------------
@@ -738,7 +741,7 @@ bool rrdset_memory_load_or_create_map_save(RRDSET *st_on_file, RRD_MEMORY_MODE m
741 // flags are set/unset in a manner that is not thread safe
742 // and may lead to missing information.
743
741 -typedef enum rrdhost_flags {
744 +typedef enum __attribute__ ((__packed__)) rrdhost_flags {
745 // Orphan, Archived and Obsolete flags
746 RRDHOST_FLAG_ORPHAN = (1 << 10), // this host is orphan (not receiving data)
747 RRDHOST_FLAG_ARCHIVED = (1 << 11), // The host is archived, no collected charts yet
@@ -751,7 +754,6 @@ typedef enum rrdhost_flags {
754 RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED = (1 << 16), // When set, the host is connected to a parent
755 RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS = (1 << 17), // when set, rrdset_done() should push metrics to parent
756 RRDHOST_FLAG_RRDPUSH_SENDER_LOGGED_STATUS = (1 << 18), // when set, we have logged the status of metrics streaming
754 - RRDHOST_FLAG_RRDPUSH_SENDER_JOIN = (1 << 19), // When set, we want to join the sender thread
757
758 // Health
759 RRDHOST_FLAG_PENDING_HEALTH_INITIALIZATION = (1 << 20), // contains charts and dims with uninitialized variables
@@ -765,8 +767,11 @@ typedef enum rrdhost_flags {
767 RRDHOST_FLAG_ACLK_STREAM_CONTEXTS = (1 << 24), // when set, we should send ACLK stream context updates
768 // Metadata
769 RRDHOST_FLAG_METADATA_UPDATE = (1 << 25), // metadata needs to be stored in the database
770 + RRDHOST_FLAG_METADATA_LABELS = (1 << 26), // metadata needs to be stored in the database
771 + RRDHOST_FLAG_METADATA_INFO = (1 << 27), // metadata needs to be stored in the database
772 + RRDHOST_FLAG_METADATA_CLAIMID = (1 << 28), // metadata needs to be stored in the database
773
769 - RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED = ( 1 << 26), // set when the receiver part is disconnected
774 + RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED = (1 << 29), // set when the receiver part is disconnected
775 } RRDHOST_FLAGS;
776
777 #define rrdhost_flag_check(host, flag) (__atomic_load_n(&((host)->flags), __ATOMIC_SEQ_CST) & (flag))
@@ -780,7 +785,7 @@ typedef enum rrdhost_flags {
785 #define rrdset_debug(st, fmt, args...) debug_dummy()
786 #endif
787
783 -typedef enum {
788 +typedef enum __attribute__ ((__packed__)) {
789 // Indexing
790 RRDHOST_OPTION_INDEXED_MACHINE_GUID = (1 << 0), // when set, we have indexed its machine guid
791 RRDHOST_OPTION_INDEXED_HOSTNAME = (1 << 1), // when set, we have indexed its hostname
@@ -987,10 +992,10 @@ struct rrdhost {
992 // ------------------------------------------------------------------------
993 // streaming of data from remote hosts - rrdpush receiver
994
990 - time_t senders_connect_time; // the time the last sender was connected
991 - time_t senders_last_chart_command; // the time of the last CHART streaming command
992 - time_t senders_disconnected_time; // the time the last sender was disconnected
993 - int senders_count; // number of senders currently streaming
995 + time_t child_connect_time; // the time the last sender was connected
996 + time_t child_last_chart_command; // the time of the last CHART streaming command
997 + time_t child_disconnected_time; // the time the last sender was disconnected
998 + int connected_children_count; // number of senders currently streaming
999
1000 struct receiver_state *receiver;
1001 netdata_mutex_t receiver_lock;
@@ -1002,7 +1007,6 @@ struct rrdhost {
1007
1008 unsigned int health_enabled; // 1 when this host has health enabled
1009 bool health_spawn; // true when health thread is running
1005 - netdata_thread_t health_thread; // the health thread
1010 unsigned int aclk_alert_reloaded; // 1 on thread start and health reload, 0 after removed are sent
1011 time_t health_delay_up_to; // a timestamp to delay alarms processing up to
1012 STRING *health_default_exec; // the full path of the alarms notifications program
@@ -1027,7 +1031,7 @@ struct rrdhost {
1031 // ------------------------------------------------------------------------
1032 // locks
1033
1030 - netdata_rwlock_t rrdhost_rwlock; // lock for this RRDHOST (protects rrdset_root linked list)
1034 + SPINLOCK rrdhost_update_lock;
1035
1036 // ------------------------------------------------------------------------
1037 // ML handle
@@ -1075,10 +1079,6 @@ extern RRDHOST *localhost;
1079 #define rrdhost_program_name(host) string2str((host)->program_name)
1080 #define rrdhost_program_version(host) string2str((host)->program_version)
1081
1078 -#define rrdhost_rdlock(host) netdata_rwlock_rdlock(&((host)->rrdhost_rwlock))
1079 -#define rrdhost_wrlock(host) netdata_rwlock_wrlock(&((host)->rrdhost_rwlock))
1080 -#define rrdhost_unlock(host) netdata_rwlock_unlock(&((host)->rrdhost_rwlock))
1081 -
1082 #define rrdhost_aclk_state_lock(host) netdata_mutex_lock(&((host)->aclk_state_lock))
1083 #define rrdhost_aclk_state_unlock(host) netdata_mutex_unlock(&((host)->aclk_state_lock))
1084
@@ -1093,16 +1093,15 @@ extern RRDHOST *localhost;
1093 #define rrdhost_sender_replicating_charts_zero(host) (__atomic_store_n(&((host)->rrdpush_sender_replicating_charts), 0, __ATOMIC_RELAXED))
1094
1095 extern DICTIONARY *rrdhost_root_index;
1096 -long rrdhost_hosts_available(void);
1096 +size_t rrdhost_hosts_available(void);
1097
1098 // ----------------------------------------------------------------------------
1099 -// these loop macros make sure the linked list is accessed with the right lock
1099
1100 #define rrdhost_foreach_read(var) \
1102 - for((var) = localhost, rrd_check_rdlock(); var ; (var) = (var)->next)
1101 + for((var) = localhost; var ; (var) = (var)->next)
1102
1103 #define rrdhost_foreach_write(var) \
1105 - for((var) = localhost, rrd_check_wrlock(); var ; (var) = (var)->next)
1104 + for((var) = localhost; var ; (var) = (var)->next)
1105
1106
1107 // ----------------------------------------------------------------------------
@@ -1125,10 +1124,9 @@ void rrddim_index_destroy(RRDSET *st);
1124
1125 // ----------------------------------------------------------------------------
1126
1128 -extern size_t rrd_hosts_available;
1129 -extern time_t rrdhost_free_orphan_time;
1127 +extern time_t rrdhost_free_orphan_time_s;
1128
1131 -int rrd_init(char *hostname, struct rrdhost_system_info *system_info);
1129 +int rrd_init(char *hostname, struct rrdhost_system_info *system_info, bool unittest);
1130
1131 RRDHOST *rrdhost_find_by_hostname(const char *hostname);
1132 RRDHOST *rrdhost_find_by_guid(const char *guid);
@@ -1159,57 +1157,8 @@ RRDHOST *rrdhost_find_or_create(
1157 , bool is_archived
1158 );
1159
1162 -void rrdhost_update(RRDHOST *host
1163 - , const char *hostname
1164 - , const char *registry_hostname
1165 - , const char *guid
1166 - , const char *os
1167 - , const char *timezone
1168 - , const char *abbrev_timezone
1169 - , int32_t utc_offset
1170 - , const char *tags
1171 - , const char *program_name
1172 - , const char *program_version
1173 - , int update_every
1174 - , long history
1175 - , RRD_MEMORY_MODE mode
1176 - , unsigned int health_enabled
1177 - , unsigned int rrdpush_enabled
1178 - , char *rrdpush_destination
1179 - , char *rrdpush_api_key
1180 - , char *rrdpush_send_charts_matching
1181 - , bool rrdpush_enable_replication
1182 - , time_t rrdpush_seconds_to_replicate
1183 - , time_t rrdpush_replication_step
1184 - , struct rrdhost_system_info *system_info
1185 -);
1186 -
1160 int rrdhost_set_system_info_variable(struct rrdhost_system_info *system_info, char *name, char *value);
1161
1189 -#if defined(NETDATA_INTERNAL_CHECKS) && defined(NETDATA_VERIFY_LOCKS)
1190 -void __rrdhost_check_wrlock(RRDHOST *host, const char *file, const char *function, const unsigned long line);
1191 -void __rrdhost_check_rdlock(RRDHOST *host, const char *file, const char *function, const unsigned long line);
1192 -void __rrdset_check_rdlock(RRDSET *st, const char *file, const char *function, const unsigned long line);
1193 -void __rrdset_check_wrlock(RRDSET *st, const char *file, const char *function, const unsigned long line);
1194 -void __rrd_check_rdlock(const char *file, const char *function, const unsigned long line);
1195 -void __rrd_check_wrlock(const char *file, const char *function, const unsigned long line);
1196 -
1197 -#define rrdhost_check_rdlock(host) __rrdhost_check_rdlock(host, __FILE__, __FUNCTION__, __LINE__)
1198 -#define rrdhost_check_wrlock(host) __rrdhost_check_wrlock(host, __FILE__, __FUNCTION__, __LINE__)
1199 -#define rrdset_check_rdlock(st) __rrdset_check_rdlock(st, __FILE__, __FUNCTION__, __LINE__)
1200 -#define rrdset_check_wrlock(st) __rrdset_check_wrlock(st, __FILE__, __FUNCTION__, __LINE__)
1201 -#define rrd_check_rdlock() __rrd_check_rdlock(__FILE__, __FUNCTION__, __LINE__)
1202 -#define rrd_check_wrlock() __rrd_check_wrlock(__FILE__, __FUNCTION__, __LINE__)
1203 -
1204 -#else
1205 -#define rrdhost_check_rdlock(host) (void)0
1206 -#define rrdhost_check_wrlock(host) (void)0
1207 -#define rrdset_check_rdlock(st) (void)0
1208 -#define rrdset_check_wrlock(st) (void)0
1209 -#define rrd_check_rdlock() (void)0
1210 -#define rrd_check_wrlock() (void)0
1211 -#endif
1212 -
1162 // ----------------------------------------------------------------------------
1163 // RRDSET functions
1164
@@ -1242,15 +1191,15 @@ void rrdhost_save_all(void);
1191 void rrdhost_cleanup_all(void);
1192
1193 void rrdhost_system_info_free(struct rrdhost_system_info *system_info);
1245 -void rrdhost_free(RRDHOST *host, bool force);
1194 +void rrdhost_free___while_having_rrd_wrlock(RRDHOST *host, bool force);
1195 void rrdhost_save_charts(RRDHOST *host);
1196 void rrdhost_delete_charts(RRDHOST *host);
1197
1249 -int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, time_t now);
1198 +int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, time_t now_s);
1199
1200 void rrdset_update_heterogeneous_flag(RRDSET *st);
1201
1253 -time_t rrdset_set_update_every(RRDSET *st, time_t update_every);
1202 +time_t rrdset_set_update_every_s(RRDSET *st, time_t update_every_s);
1203
1204 RRDSET *rrdset_find(RRDHOST *host, const char *id);
1205 #define rrdset_find_localhost(id) rrdset_find(localhost, id)
@@ -1301,16 +1250,17 @@ void rrdset_isnot_obsolete(RRDSET *st);
1250 #define rrdset_is_available_for_exporting_and_alarms(st) (!rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE) && !rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && rrdset_number_of_dimensions(st))
1251 #define rrdset_is_archived(st) (rrdset_flag_check(st, RRDSET_FLAG_ARCHIVED) && rrdset_number_of_dimensions(st))
1252
1304 -time_t rrddim_first_entry_t(RRDDIM *rd);
1305 -time_t rrddim_first_entry_t_of_tier(RRDDIM *rd, size_t tier);
1306 -time_t rrddim_last_entry_t(RRDDIM *rd);
1307 -time_t rrddim_last_entry_t_of_tier(RRDDIM *rd, size_t tier);
1253 +time_t rrddim_first_entry_s(RRDDIM *rd);
1254 +time_t rrddim_first_entry_s_of_tier(RRDDIM *rd, size_t tier);
1255 +time_t rrddim_last_entry_s(RRDDIM *rd);
1256 +time_t rrddim_last_entry_s_of_tier(RRDDIM *rd, size_t tier);
1257
1309 -time_t rrdset_first_entry_t(RRDSET *st);
1310 -time_t rrdset_first_entry_t_of_tier(RRDSET *st, size_t tier);
1311 -time_t rrdset_last_entry_t(RRDSET *st);
1258 +time_t rrdset_first_entry_s(RRDSET *st);
1259 +time_t rrdset_first_entry_s_of_tier(RRDSET *st, size_t tier);
1260 +time_t rrdset_last_entry_s(RRDSET *st);
1261 +time_t rrdset_last_entry_s_of_tier(RRDSET *st, size_t tier);
1262
1313 -time_t rrdhost_last_entry_t(RRDHOST *h);
1263 +void rrdset_get_retention_of_tier_for_collected_chart(RRDSET *st, time_t *first_time_s, time_t *last_time_s, time_t now_s, size_t tier);
1264
1265 // ----------------------------------------------------------------------------
1266 // RRD DIMENSION functions
@@ -1348,9 +1298,6 @@ collected_number rrddim_timed_set_by_pointer(RRDSET *st, RRDDIM *rd, struct time
1298 collected_number rrddim_set_by_pointer(RRDSET *st, RRDDIM *rd, collected_number value);
1299 collected_number rrddim_set(RRDSET *st, const char *id, collected_number value);
1300
1351 -#ifdef ENABLE_ACLK
1352 -time_t calc_dimension_liveness(RRDDIM *rd, time_t now);
1353 -#endif
1301 long align_entries_to_pagesize(RRD_MEMORY_MODE mode, long entries);
1302
1303 #ifdef NETDATA_LOG_COLLECTION_ERRORS
@@ -1381,14 +1328,6 @@ void rrddim_free(RRDSET *st, RRDDIM *rd);
1328 void rrdset_reset(RRDSET *st);
1329 void rrdset_delete_obsolete_dimensions(RRDSET *st);
1330
1384 -RRDHOST *rrdhost_create(
1385 - const char *hostname, const char *registry_hostname, const char *guid, const char *os, const char *timezone,
1386 - const char *abbrev_timezone, int32_t utc_offset,const char *tags, const char *program_name, const char *program_version,
1387 - int update_every, long entries, RRD_MEMORY_MODE memory_mode, unsigned int health_enabled, unsigned int rrdpush_enabled,
1388 - char *rrdpush_destination, char *rrdpush_api_key, char *rrdpush_send_charts_matching,
1389 - bool rrdpush_enable_replication, time_t rrdpush_seconds_to_replicate, time_t rrdpush_replication_step,
1390 - struct rrdhost_system_info *system_info, int is_localhost, bool is_archived);
1391 -
1331 #endif /* NETDATA_RRD_INTERNALS */
1332
1333 void set_host_properties(
@@ -1397,6 +1336,7 @@ void set_host_properties(
1336 const char *program_name, const char *program_version);
1337
1338 size_t get_tier_grouping(size_t tier);
1339 +void store_metric_collection_completed(void);
1340
1341 // ----------------------------------------------------------------------------
1342 // RRD DB engine declarations
database/rrdcontext.c
+195 -180
@@ -221,8 +221,8 @@ typedef struct rrdmetric {
221
222 RRDDIM *rrddim;
223
224 - time_t first_time_t;
225 - time_t last_time_t;
224 + time_t first_time_s;
225 + time_t last_time_s;
226 RRD_FLAGS flags;
227
228 struct rrdinstance *ri;
@@ -240,10 +240,10 @@ typedef struct rrdinstance {
240 RRDSET_TYPE chart_type;
241
242 RRD_FLAGS flags; // flags related to this instance
243 - time_t first_time_t;
244 - time_t last_time_t;
243 + time_t first_time_s;
244 + time_t last_time_s;
245
246 - int update_every; // data collection frequency
246 + time_t update_every_s; // data collection frequency
247 RRDSET *rrdset; // pointer to RRDSET when collected, or NULL
248
249 DICTIONARY *rrdlabels; // linked to RRDSET->chart_labels or own version
@@ -269,8 +269,8 @@ typedef struct rrdcontext {
269 RRDSET_TYPE chart_type;
270
271 RRD_FLAGS flags;
272 - time_t first_time_t;
273 - time_t last_time_t;
272 + time_t first_time_s;
273 + time_t last_time_s;
274
275 VERSIONED_CONTEXT_DATA hub;
276
@@ -522,20 +522,20 @@ static bool rrdmetric_conflict_callback(const DICTIONARY_ITEM *item __maybe_unus
522 uuid_unparse(rm->uuid, uuid1);
523 uuid_unparse(rm_new->uuid, uuid2);
524
525 - time_t old_first_time_t = 0;
526 - time_t old_last_time_t = 0;
525 + time_t old_first_time_s = 0;
526 + time_t old_last_time_s = 0;
527 if(rrdmetric_update_retention(rm)) {
528 - old_first_time_t = rm->first_time_t;
529 - old_last_time_t = rm->last_time_t;
528 + old_first_time_s = rm->first_time_s;
529 + old_last_time_s = rm->last_time_s;
530 }
531
532 uuid_copy(rm->uuid, rm_new->uuid);
533
534 - time_t new_first_time_t = 0;
535 - time_t new_last_time_t = 0;
534 + time_t new_first_time_s = 0;
535 + time_t new_last_time_s = 0;
536 if(rrdmetric_update_retention(rm)) {
537 - new_first_time_t = rm->first_time_t;
538 - new_last_time_t = rm->last_time_t;
537 + new_first_time_s = rm->first_time_s;
538 + new_last_time_s = rm->last_time_s;
539 }
540
541 internal_error(true,
@@ -543,8 +543,8 @@ static bool rrdmetric_conflict_callback(const DICTIONARY_ITEM *item __maybe_unus
543 , string2str(rm->id)
544 , string2str(rm->ri->id)
545 , rrdhost_hostname(rm->ri->rc->rrdhost)
546 - , uuid1, old_first_time_t, old_last_time_t, old_last_time_t - old_first_time_t
547 - , uuid2, new_first_time_t, new_last_time_t, new_last_time_t - new_first_time_t
546 + , uuid1, old_first_time_s, old_last_time_s, old_last_time_s - old_first_time_s
547 + , uuid2, new_first_time_s, new_last_time_s, new_last_time_s - new_first_time_s
548 );
549 #else
550 uuid_copy(rm->uuid, rm_new->uuid);
@@ -576,13 +576,13 @@ static bool rrdmetric_conflict_callback(const DICTIONARY_ITEM *item __maybe_unus
576 rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_METADATA);
577 }
578
579 - if(!rm->first_time_t || (rm_new->first_time_t && rm_new->first_time_t < rm->first_time_t)) {
580 - rm->first_time_t = rm_new->first_time_t;
579 + if(!rm->first_time_s || (rm_new->first_time_s && rm_new->first_time_s < rm->first_time_s)) {
580 + rm->first_time_s = rm_new->first_time_s;
581 rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
582 }
583
584 - if(!rm->last_time_t || (rm_new->last_time_t && rm_new->last_time_t > rm->last_time_t)) {
585 - rm->last_time_t = rm_new->last_time_t;
584 + if(!rm->last_time_s || (rm_new->last_time_s && rm_new->last_time_s > rm->last_time_s)) {
585 + rm->last_time_s = rm_new->last_time_s;
586 rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
587 }
588
@@ -863,8 +863,8 @@ static bool rrdinstance_conflict_callback(const DICTIONARY_ITEM *item __maybe_un
863 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_METADATA);
864 }
865
866 - if(ri->update_every != ri_new->update_every) {
867 - ri->update_every = ri_new->update_every;
866 + if(ri->update_every_s != ri_new->update_every_s) {
867 + ri->update_every_s = ri_new->update_every_s;
868 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_METADATA);
869 }
870
@@ -936,8 +936,8 @@ static void rrdinstance_trigger_updates(RRDINSTANCE *ri, const char *function) {
936 ri->priority = st->priority;
937 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_METADATA);
938 }
939 - if(unlikely(st->update_every != ri->update_every)) {
940 - ri->update_every = st->update_every;
939 + if(unlikely(st->update_every != ri->update_every_s)) {
940 + ri->update_every_s = st->update_every;
941 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_METADATA);
942 }
943 }
@@ -980,7 +980,7 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
980 .title = string_dup(st->title),
981 .chart_type = st->chart_type,
982 .priority = st->priority,
983 - .update_every = st->update_every,
983 + .update_every_s = st->update_every,
984 .flags = RRD_FLAG_NONE, // no need for atomics
985 .rrdset = st,
986 };
@@ -1018,8 +1018,8 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
1018 RRDMETRIC *rm_old = rrdmetric_acquired_value(rd->rrdmetric);
1019 rrd_flags_replace(rm_old, RRD_FLAG_DELETED|RRD_FLAG_UPDATED|RRD_FLAG_LIVE_RETENTION|RRD_FLAG_UPDATE_REASON_UNUSED|RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
1020 rm_old->rrddim = NULL;
1021 - rm_old->first_time_t = 0;
1022 - rm_old->last_time_t = 0;
1021 + rm_old->first_time_s = 0;
1022 + rm_old->last_time_s = 0;
1023
1024 rrdmetric_release(rd->rrdmetric);
1025 rd->rrdmetric = NULL;
@@ -1034,8 +1034,8 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
1034
1035 rrd_flags_replace(ri_old, RRD_FLAG_OWN_LABELS|RRD_FLAG_DELETED|RRD_FLAG_UPDATED|RRD_FLAG_LIVE_RETENTION|RRD_FLAG_UPDATE_REASON_UNUSED|RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
1036 ri_old->rrdset = NULL;
1037 - ri_old->first_time_t = 0;
1038 - ri_old->last_time_t = 0;
1037 + ri_old->first_time_s = 0;
1038 + ri_old->last_time_s = 0;
1039
1040 rrdinstance_trigger_updates(ri_old, __FUNCTION__ );
1041 rrdinstance_release(ria_old);
@@ -1045,8 +1045,8 @@ static inline void rrdinstance_from_rrdset(RRDSET *st) {
1045 if(!dictionary_entries(rc_old->rrdinstances) && !dictionary_stats_referenced_items(rc_old->rrdinstances)) {
1046 rrdcontext_lock(rc_old);
1047 rc_old->flags = ((rc_old->flags & RRD_FLAG_QUEUED)?RRD_FLAG_QUEUED:RRD_FLAG_NONE)|RRD_FLAG_DELETED|RRD_FLAG_UPDATED|RRD_FLAG_LIVE_RETENTION|RRD_FLAG_UPDATE_REASON_UNUSED|RRD_FLAG_UPDATE_REASON_ZERO_RETENTION;
1048 - rc_old->first_time_t = 0;
1049 - rc_old->last_time_t = 0;
1048 + rc_old->first_time_s = 0;
1049 + rc_old->last_time_s = 0;
1050 rrdcontext_unlock(rc_old);
1051 rrdcontext_trigger_updates(rc_old, __FUNCTION__ );
1052 }
@@ -1224,13 +1224,13 @@ static void rrdcontext_insert_callback(const DICTIONARY_ITEM *item __maybe_unuse
1224
1225 rc->version = rc->hub.version;
1226 rc->priority = rc->hub.priority;
1227 - rc->first_time_t = (time_t)rc->hub.first_time_t;
1228 - rc->last_time_t = (time_t)rc->hub.last_time_t;
1227 + rc->first_time_s = (time_t)rc->hub.first_time_s;
1228 + rc->last_time_s = (time_t)rc->hub.last_time_s;
1229
1230 - if(rc->hub.deleted || !rc->hub.first_time_t)
1230 + if(rc->hub.deleted || !rc->hub.first_time_s)
1231 rrd_flag_set_deleted(rc, RRD_FLAG_NONE);
1232 else {
1233 - if (rc->last_time_t == 0)
1233 + if (rc->last_time_s == 0)
1234 rrd_flag_set_collected(rc);
1235 else
1236 rrd_flag_set_archived(rc);
@@ -1734,8 +1734,8 @@ struct rrdcontext_to_json {
1734 SIMPLE_PATTERN *chart_dimensions;
1735 size_t written;
1736 time_t now;
1737 - time_t combined_first_time_t;
1738 - time_t combined_last_time_t;
1737 + time_t combined_first_time_s;
1738 + time_t combined_last_time_s;
1739 RRD_FLAGS combined_flags;
1740 };
1741
@@ -1751,10 +1751,10 @@ static inline int rrdmetric_to_json_callback(const DICTIONARY_ITEM *item, void *
1751 if(unlikely(rrd_flag_is_deleted(rm) && !(options & RRDCONTEXT_OPTION_SHOW_DELETED)))
1752 return 0;
1753
1754 - if(after && (!rm->last_time_t || after > rm->last_time_t))
1754 + if(after && (!rm->last_time_s || after > rm->last_time_s))
1755 return 0;
1756
1757 - if(before && (!rm->first_time_t || before < rm->first_time_t))
1757 + if(before && (!rm->first_time_s || before < rm->first_time_s))
1758 return 0;
1759
1760 if(t->chart_dimensions
@@ -1764,14 +1764,14 @@ static inline int rrdmetric_to_json_callback(const DICTIONARY_ITEM *item, void *
1764
1765 if(t->written) {
1766 buffer_strcat(wb, ",\n");
1767 - t->combined_first_time_t = MIN(t->combined_first_time_t, rm->first_time_t);
1768 - t->combined_last_time_t = MAX(t->combined_last_time_t, rm->last_time_t);
1767 + t->combined_first_time_s = MIN(t->combined_first_time_s, rm->first_time_s);
1768 + t->combined_last_time_s = MAX(t->combined_last_time_s, rm->last_time_s);
1769 t->combined_flags |= rrd_flags_get(rm);
1770 }
1771 else {
1772 buffer_strcat(wb, "\n");
1773 - t->combined_first_time_t = rm->first_time_t;
1774 - t->combined_last_time_t = rm->last_time_t;
1773 + t->combined_first_time_s = rm->first_time_s;
1774 + t->combined_last_time_s = rm->last_time_s;
1775 t->combined_flags = rrd_flags_get(rm);
1776 }
1777
@@ -1789,8 +1789,8 @@ static inline int rrdmetric_to_json_callback(const DICTIONARY_ITEM *item, void *
1789 ",\n\t\t\t\t\t\t\t\"last_time_t\":%lld"
1790 ",\n\t\t\t\t\t\t\t\"collected\":%s"
1791 , string2str(rm->name)
1792 - , (long long)rm->first_time_t
1793 - , rrd_flag_is_collected(rm) ? (long long)t->now : (long long)rm->last_time_t
1792 + , (long long)rm->first_time_s
1793 + , rrd_flag_is_collected(rm) ? (long long)t->now : (long long)rm->last_time_s
1794 , rrd_flag_is_collected(rm) ? "true" : "false"
1795 );
1796
@@ -1826,10 +1826,10 @@ static inline int rrdinstance_to_json_callback(const DICTIONARY_ITEM *item, void
1826 if(unlikely(rrd_flag_is_deleted(ri) && !(options & RRDCONTEXT_OPTION_SHOW_DELETED)))
1827 return 0;
1828
1829 - if(after && (!ri->last_time_t || after > ri->last_time_t))
1829 + if(after && (!ri->last_time_s || after > ri->last_time_s))
1830 return 0;
1831
1832 - if(before && (!ri->first_time_t || before < ri->first_time_t))
1832 + if(before && (!ri->first_time_s || before < ri->first_time_s))
1833 return 0;
1834
1835 if(t_parent->chart_label_key && !rrdlabels_match_simple_pattern_parsed(ri->rrdlabels, t_parent->chart_label_key, '\0'))
@@ -1838,8 +1838,8 @@ static inline int rrdinstance_to_json_callback(const DICTIONARY_ITEM *item, void
1838 if(t_parent->chart_labels_filter && !rrdlabels_match_simple_pattern_parsed(ri->rrdlabels, t_parent->chart_labels_filter, ':'))
1839 return 0;
1840
1841 - time_t first_time_t = ri->first_time_t;
1842 - time_t last_time_t = ri->last_time_t;
1841 + time_t first_time_s = ri->first_time_s;
1842 + time_t last_time_s = ri->last_time_s;
1843 RRD_FLAGS flags = rrd_flags_get(ri);
1844
1845 BUFFER *wb_metrics = NULL;
@@ -1865,21 +1865,21 @@ static inline int rrdinstance_to_json_callback(const DICTIONARY_ITEM *item, void
1865 return 0;
1866 }
1867
1868 - first_time_t = t_metrics.combined_first_time_t;
1869 - last_time_t = t_metrics.combined_last_time_t;
1868 + first_time_s = t_metrics.combined_first_time_s;
1869 + last_time_s = t_metrics.combined_last_time_s;
1870 flags = t_metrics.combined_flags;
1871 }
1872
1873 if(t_parent->written) {
1874 buffer_strcat(wb, ",\n");
1875 - t_parent->combined_first_time_t = MIN(t_parent->combined_first_time_t, first_time_t);
1876 - t_parent->combined_last_time_t = MAX(t_parent->combined_last_time_t, last_time_t);
1875 + t_parent->combined_first_time_s = MIN(t_parent->combined_first_time_s, first_time_s);
1876 + t_parent->combined_last_time_s = MAX(t_parent->combined_last_time_s, last_time_s);
1877 t_parent->combined_flags |= flags;
1878 }
1879 else {
1880 buffer_strcat(wb, "\n");
1881 - t_parent->combined_first_time_t = first_time_t;
1882 - t_parent->combined_last_time_t = last_time_t;
1881 + t_parent->combined_first_time_s = first_time_s;
1882 + t_parent->combined_last_time_s = last_time_s;
1883 t_parent->combined_flags = flags;
1884 }
1885
@@ -1899,7 +1899,7 @@ static inline int rrdinstance_to_json_callback(const DICTIONARY_ITEM *item, void
1899 ",\n\t\t\t\t\t\"family\":\"%s\""
1900 ",\n\t\t\t\t\t\"chart_type\":\"%s\""
1901 ",\n\t\t\t\t\t\"priority\":%u"
1902 - ",\n\t\t\t\t\t\"update_every\":%d"
1902 + ",\n\t\t\t\t\t\"update_every\":%ld"
1903 ",\n\t\t\t\t\t\"first_time_t\":%lld"
1904 ",\n\t\t\t\t\t\"last_time_t\":%lld"
1905 ",\n\t\t\t\t\t\"collected\":%s"
@@ -1910,9 +1910,9 @@ static inline int rrdinstance_to_json_callback(const DICTIONARY_ITEM *item, void
1910 , string2str(ri->family)
1911 , rrdset_type_name(ri->chart_type)
1912 , ri->priority
1913 - , ri->update_every
1914 - , (long long)first_time_t
1915 - , (flags & RRD_FLAG_COLLECTED) ? (long long)t_parent->now : (long long)last_time_t
1913 + , ri->update_every_s
1914 + , (long long)first_time_s
1915 + , (flags & RRD_FLAG_COLLECTED) ? (long long)t_parent->now : (long long)last_time_s
1916 , (flags & RRD_FLAG_COLLECTED) ? "true" : "false"
1917 );
1918
@@ -1967,14 +1967,14 @@ static inline int rrdcontext_to_json_callback(const DICTIONARY_ITEM *item, void
1967 if(options & RRDCONTEXT_OPTION_DEEPSCAN)
1968 rrdcontext_recalculate_context_retention(rc, RRD_FLAG_NONE, false);
1969
1970 - if(after && (!rc->last_time_t || after > rc->last_time_t))
1970 + if(after && (!rc->last_time_s || after > rc->last_time_s))
1971 return 0;
1972
1973 - if(before && (!rc->first_time_t || before < rc->first_time_t))
1973 + if(before && (!rc->first_time_s || before < rc->first_time_s))
1974 return 0;
1975
1976 - time_t first_time_t = rc->first_time_t;
1977 - time_t last_time_t = rc->last_time_t;
1976 + time_t first_time_s = rc->first_time_s;
1977 + time_t last_time_s = rc->last_time_s;
1978 RRD_FLAGS flags = rrd_flags_get(rc);
1979
1980 BUFFER *wb_instances = NULL;
@@ -2003,8 +2003,8 @@ static inline int rrdcontext_to_json_callback(const DICTIONARY_ITEM *item, void
2003 return 0;
2004 }
2005
2006 - first_time_t = t_instances.combined_first_time_t;
2007 - last_time_t = t_instances.combined_last_time_t;
2006 + first_time_s = t_instances.combined_first_time_s;
2007 + last_time_s = t_instances.combined_last_time_s;
2008 flags = t_instances.combined_flags;
2009 }
2010
@@ -2034,8 +2034,8 @@ static inline int rrdcontext_to_json_callback(const DICTIONARY_ITEM *item, void
2034 , string2str(rc->family)
2035 , rrdset_type_name(rc->chart_type)
2036 , rc->priority
2037 - , (long long)first_time_t
2038 - , (flags & RRD_FLAG_COLLECTED) ? (long long)t_parent->now : (long long)last_time_t
2037 + , (long long)first_time_s
2038 + , (flags & RRD_FLAG_COLLECTED) ? (long long)t_parent->now : (long long)last_time_s
2039 , (flags & RRD_FLAG_COLLECTED) ? "true" : "false"
2040 );
2041
@@ -2319,11 +2319,25 @@ void query_target_release(QUERY_TARGET *qt) {
2319 string_freez(qt->query.array[i].chart.name);
2320 qt->query.array[i].chart.name = NULL;
2321
2322 + // reset the plans
2323 + for(size_t p = 0; p < qt->query.array[i].plan.used; p++) {
2324 + internal_fatal(qt->query.array[i].plan.array[p].initialized &&
2325 + !qt->query.array[i].plan.array[p].finalized,
2326 + "QUERY: left-over initialized plan");
2327 +
2328 + qt->query.array[i].plan.array[p].initialized = false;
2329 + qt->query.array[i].plan.array[p].finalized = false;
2330 + }
2331 + qt->query.array[i].plan.used = 0;
2332 +
2333 + // reset the tiers
2334 for(size_t tier = 0; tier < storage_tiers ;tier++) {
2335 if(qt->query.array[i].tiers[tier].db_metric_handle) {
2336 STORAGE_ENGINE *eng = qt->query.array[i].tiers[tier].eng;
2337 eng->api.metric_release(qt->query.array[i].tiers[tier].db_metric_handle);
2338 qt->query.array[i].tiers[tier].db_metric_handle = NULL;
2339 + qt->query.array[i].tiers[tier].weight = 0;
2340 + qt->query.array[i].tiers[tier].eng = NULL;
2341 }
2342 }
2343 }
@@ -2357,9 +2371,9 @@ void query_target_release(QUERY_TARGET *qt) {
2371 qt->contexts.used = 0;
2372 qt->hosts.used = 0;
2373
2360 - qt->db.minimum_latest_update_every = 0;
2361 - qt->db.first_time_t = 0;
2362 - qt->db.last_time_t = 0;
2374 + qt->db.minimum_latest_update_every_s = 0;
2375 + qt->db.first_time_s = 0;
2376 + qt->db.last_time_s = 0;
2377
2378 qt->id[0] = '\0';
2379
@@ -2407,22 +2421,22 @@ static void query_target_add_metric(QUERY_TARGET_LOCALS *qtl, RRDMETRIC_ACQUIRED
2421 if(!queryable_instance)
2422 return;
2423
2410 - time_t common_first_time_t = 0;
2411 - time_t common_last_time_t = 0;
2412 - time_t common_update_every = 0;
2424 + time_t common_first_time_s = 0;
2425 + time_t common_last_time_s = 0;
2426 + time_t common_update_every_s = 0;
2427 size_t tiers_added = 0;
2428 struct {
2429 STORAGE_ENGINE *eng;
2430 STORAGE_METRIC_HANDLE *db_metric_handle;
2417 - time_t db_first_time_t;
2418 - time_t db_last_time_t;
2419 - time_t db_update_every;
2431 + time_t db_first_time_s;
2432 + time_t db_last_time_s;
2433 + time_t db_update_every_s;
2434 } tier_retention[storage_tiers];
2435
2436 for (size_t tier = 0; tier < storage_tiers; tier++) {
2437 STORAGE_ENGINE *eng = qtl->host->db[tier].eng;
2438 tier_retention[tier].eng = eng;
2425 - tier_retention[tier].db_update_every = (time_t) (qtl->host->db[tier].tier_grouping * ri->update_every);
2439 + tier_retention[tier].db_update_every_s = (time_t) (qtl->host->db[tier].tier_grouping * ri->update_every_s);
2440
2441 if(rm->rrddim && rm->rrddim->tiers[tier] && rm->rrddim->tiers[tier]->db_metric_handle)
2442 tier_retention[tier].db_metric_handle = eng->api.metric_dup(rm->rrddim->tiers[tier]->db_metric_handle);
@@ -2430,38 +2444,38 @@ static void query_target_add_metric(QUERY_TARGET_LOCALS *qtl, RRDMETRIC_ACQUIRED
2444 tier_retention[tier].db_metric_handle = eng->api.metric_get(qtl->host->db[tier].instance, &rm->uuid);
2445
2446 if(tier_retention[tier].db_metric_handle) {
2433 - tier_retention[tier].db_first_time_t = tier_retention[tier].eng->api.query_ops.oldest_time(tier_retention[tier].db_metric_handle);
2434 - tier_retention[tier].db_last_time_t = tier_retention[tier].eng->api.query_ops.latest_time(tier_retention[tier].db_metric_handle);
2447 + tier_retention[tier].db_first_time_s = tier_retention[tier].eng->api.query_ops.oldest_time_s(tier_retention[tier].db_metric_handle);
2448 + tier_retention[tier].db_last_time_s = tier_retention[tier].eng->api.query_ops.latest_time_s(tier_retention[tier].db_metric_handle);
2449
2436 - if(!common_first_time_t)
2437 - common_first_time_t = tier_retention[tier].db_first_time_t;
2438 - else if(tier_retention[tier].db_first_time_t)
2439 - common_first_time_t = MIN(common_first_time_t, tier_retention[tier].db_first_time_t);
2450 + if(!common_first_time_s)
2451 + common_first_time_s = tier_retention[tier].db_first_time_s;
2452 + else if(tier_retention[tier].db_first_time_s)
2453 + common_first_time_s = MIN(common_first_time_s, tier_retention[tier].db_first_time_s);
2454
2441 - if(!common_last_time_t)
2442 - common_last_time_t = tier_retention[tier].db_last_time_t;
2455 + if(!common_last_time_s)
2456 + common_last_time_s = tier_retention[tier].db_last_time_s;
2457 else
2444 - common_last_time_t = MAX(common_last_time_t, tier_retention[tier].db_last_time_t);
2458 + common_last_time_s = MAX(common_last_time_s, tier_retention[tier].db_last_time_s);
2459
2446 - if(!common_update_every)
2447 - common_update_every = tier_retention[tier].db_update_every;
2448 - else if(tier_retention[tier].db_update_every)
2449 - common_update_every = MIN(common_update_every, tier_retention[tier].db_update_every);
2460 + if(!common_update_every_s)
2461 + common_update_every_s = tier_retention[tier].db_update_every_s;
2462 + else if(tier_retention[tier].db_update_every_s)
2463 + common_update_every_s = MIN(common_update_every_s, tier_retention[tier].db_update_every_s);
2464
2465 tiers_added++;
2466 }
2467 else {
2454 - tier_retention[tier].db_first_time_t = 0;
2455 - tier_retention[tier].db_last_time_t = 0;
2456 - tier_retention[tier].db_update_every = 0;
2468 + tier_retention[tier].db_first_time_s = 0;
2469 + tier_retention[tier].db_last_time_s = 0;
2470 + tier_retention[tier].db_update_every_s = 0;
2471 }
2472 }
2473
2474 bool release_retention = true;
2475 bool timeframe_matches =
2476 (tiers_added
2463 - && (common_first_time_t - common_update_every * 2) <= qt->window.before
2464 - && (common_last_time_t + common_update_every * 2) >= qt->window.after
2477 + && (common_first_time_s - common_update_every_s * 2) <= qt->window.before
2478 + && (common_last_time_s + common_update_every_s * 2) >= qt->window.after
2479 ) ? true : false;
2480
2481 if(timeframe_matches) {
@@ -2506,7 +2520,7 @@ static void query_target_add_metric(QUERY_TARGET_LOCALS *qtl, RRDMETRIC_ACQUIRED
2520 // let's add it to the query metrics
2521
2522 if(ri->rrdset)
2509 - ri->rrdset->last_accessed_time = qtl->start_s;
2523 + ri->rrdset->last_accessed_time_s = qtl->start_s;
2524
2525 if (qt->query.used == qt->query.size) {
2526 qt->query.size = (qt->query.size) ? qt->query.size * 2 : 1;
@@ -2514,6 +2528,7 @@ static void query_target_add_metric(QUERY_TARGET_LOCALS *qtl, RRDMETRIC_ACQUIRED
2528 }
2529 QUERY_METRIC *qm = &qt->query.array[qt->query.used++];
2530
2531 + qm->plan.used = 0;
2532 qm->dimension.options = options;
2533
2534 qm->link.host = qtl->host;
@@ -2527,18 +2542,18 @@ static void query_target_add_metric(QUERY_TARGET_LOCALS *qtl, RRDMETRIC_ACQUIRED
2542 qm->dimension.id = string_dup(rm->id);
2543 qm->dimension.name = string_dup(rm->name);
2544
2530 - if (!qt->db.first_time_t || common_first_time_t < qt->db.first_time_t)
2531 - qt->db.first_time_t = common_first_time_t;
2545 + if (!qt->db.first_time_s || common_first_time_s < qt->db.first_time_s)
2546 + qt->db.first_time_s = common_first_time_s;
2547
2533 - if (!qt->db.last_time_t || common_last_time_t > qt->db.last_time_t)
2534 - qt->db.last_time_t = common_last_time_t;
2548 + if (!qt->db.last_time_s || common_last_time_s > qt->db.last_time_s)
2549 + qt->db.last_time_s = common_last_time_s;
2550
2551 for (size_t tier = 0; tier < storage_tiers; tier++) {
2552 qm->tiers[tier].eng = tier_retention[tier].eng;
2553 qm->tiers[tier].db_metric_handle = tier_retention[tier].db_metric_handle;
2539 - qm->tiers[tier].db_first_time_t = tier_retention[tier].db_first_time_t;
2540 - qm->tiers[tier].db_last_time_t = tier_retention[tier].db_last_time_t;
2541 - qm->tiers[tier].db_update_every = tier_retention[tier].db_update_every;
2554 + qm->tiers[tier].db_first_time_s = tier_retention[tier].db_first_time_s;
2555 + qm->tiers[tier].db_last_time_s = tier_retention[tier].db_last_time_s;
2556 + qm->tiers[tier].db_update_every_s = tier_retention[tier].db_update_every_s;
2557 }
2558 release_retention = false;
2559 }
@@ -2569,8 +2584,8 @@ static void query_target_add_instance(QUERY_TARGET_LOCALS *qtl, RRDINSTANCE_ACQU
2584
2585 qtl->ria = qt->instances.array[qt->instances.used++] = rrdinstance_acquired_dup(ria);
2586
2572 - if(qt->db.minimum_latest_update_every == 0 || ri->update_every < qt->db.minimum_latest_update_every)
2573 - qt->db.minimum_latest_update_every = ri->update_every;
2587 + if(qt->db.minimum_latest_update_every_s == 0 || ri->update_every_s < qt->db.minimum_latest_update_every_s)
2588 + qt->db.minimum_latest_update_every_s = ri->update_every_s;
2589
2590 if(queryable_instance) {
2591 if ((qt->instances.chart_label_key_pattern && !rrdlabels_match_simple_pattern_parsed(ri->rrdlabels, qt->instances.chart_label_key_pattern, ':')) ||
@@ -2761,6 +2776,9 @@ void query_target_generate_name(QUERY_TARGET *qt) {
2776 }
2777
2778 QUERY_TARGET *query_target_create(QUERY_TARGET_REQUEST *qtr) {
2779 + if(!service_running(ABILITY_DATA_QUERIES))
2780 + return NULL;
2781 +
2782 QUERY_TARGET *qt = &thread_query_target;
2783
2784 if(qt->used)
@@ -2791,7 +2809,7 @@ QUERY_TARGET *query_target_create(QUERY_TARGET_REQUEST *qtr) {
2809 .charts_labels_filter = qt->request.charts_labels_filter,
2810 };
2811
2794 - qt->db.minimum_latest_update_every = 0; // it will be updated by query_target_add_query()
2812 + qt->db.minimum_latest_update_every_s = 0; // it will be updated by query_target_add_query()
2813
2814 // prepare all the patterns
2815 qt->hosts.pattern = is_valid_sp(qtl.hosts) ? simple_pattern_create(qtl.hosts, ",|\t\r\n\f\v", SIMPLE_PATTERN_EXACT) : NULL;
@@ -2913,7 +2931,7 @@ static void rrdinstance_load_chart_callback(SQL_CHART_DATA *sc, void *data) {
2931 .family = string_strdupz(sc->family),
2932 .chart_type = sc->chart_type,
2933 .priority = sc->priority,
2916 - .update_every = sc->update_every,
2934 + .update_every_s = sc->update_every,
2935 .flags = RRD_FLAG_ARCHIVED | RRD_FLAG_UPDATE_REASON_LOAD_SQL, // no need for atomics
2936 };
2937 uuid_copy(tri.uuid, sc->chart_id);
@@ -3003,7 +3021,7 @@ static uint64_t rrdcontext_version_hash_with_callback(
3021 // when the context is being collected,
3022 // rc->hub.last_time_t is already zero
3023
3006 - hash += rc->hub.version + rc->hub.last_time_t - rc->hub.first_time_t;
3024 + hash += rc->hub.version + rc->hub.last_time_s - rc->hub.first_time_s;
3025
3026 rrdcontext_unlock(rc);
3027
@@ -3048,17 +3066,16 @@ static bool rrdmetric_update_retention(RRDMETRIC *rm) {
3066 time_t min_first_time_t = LONG_MAX, max_last_time_t = 0;
3067
3068 if(rm->rrddim) {
3051 - min_first_time_t = rrddim_first_entry_t(rm->rrddim);
3052 - max_last_time_t = rrddim_last_entry_t(rm->rrddim);
3069 + min_first_time_t = rrddim_first_entry_s(rm->rrddim);
3070 + max_last_time_t = rrddim_last_entry_s(rm->rrddim);
3071 }
3054 -#ifdef ENABLE_DBENGINE
3055 - else if (dbengine_enabled) {
3072 + else {
3073 RRDHOST *rrdhost = rm->ri->rc->rrdhost;
3074 for (size_t tier = 0; tier < storage_tiers; tier++) {
3058 - if(!rrdhost->db[tier].instance) continue;
3075 + STORAGE_ENGINE *eng = rrdhost->db[tier].eng;
3076
3077 time_t first_time_t, last_time_t;
3061 - if (rrdeng_metric_retention_by_uuid(rrdhost->db[tier].instance, &rm->uuid, &first_time_t, &last_time_t) == 0) {
3078 + if (eng->api.metric_retention_by_uuid(rrdhost->db[tier].instance, &rm->uuid, &first_time_t, &last_time_t)) {
3079 if (first_time_t < min_first_time_t)
3080 min_first_time_t = first_time_t;
3081
@@ -3067,17 +3084,15 @@ static bool rrdmetric_update_retention(RRDMETRIC *rm) {
3084 }
3085 }
3086 }
3070 - else {
3071 - // cannot get retention
3087 +
3088 + if((min_first_time_t == LONG_MAX || min_first_time_t == 0) && max_last_time_t == 0)
3089 return false;
3073 - }
3074 -#endif
3090
3091 if(min_first_time_t == LONG_MAX)
3092 min_first_time_t = 0;
3093
3094 if(min_first_time_t > max_last_time_t) {
3080 - internal_error(true, "RRDMETRIC: retention of '%s' is flipped", string2str(rm->id));
3095 + internal_error(true, "RRDMETRIC: retention of '%s' is flipped, first_time_t = %ld, last_time_t = %ld", string2str(rm->id), min_first_time_t, max_last_time_t);
3096 time_t tmp = min_first_time_t;
3097 min_first_time_t = max_last_time_t;
3098 max_last_time_t = tmp;
@@ -3085,17 +3100,17 @@ static bool rrdmetric_update_retention(RRDMETRIC *rm) {
3100
3101 // check if retention changed
3102
3088 - if (min_first_time_t != rm->first_time_t) {
3089 - rm->first_time_t = min_first_time_t;
3103 + if (min_first_time_t != rm->first_time_s) {
3104 + rm->first_time_s = min_first_time_t;
3105 rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
3106 }
3107
3093 - if (max_last_time_t != rm->last_time_t) {
3094 - rm->last_time_t = max_last_time_t;
3108 + if (max_last_time_t != rm->last_time_s) {
3109 + rm->last_time_s = max_last_time_t;
3110 rrd_flag_set_updated(rm, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
3111 }
3112
3098 - if(unlikely(!rm->first_time_t && !rm->last_time_t))
3113 + if(unlikely(!rm->first_time_s && !rm->last_time_s))
3114 rrd_flag_set_deleted(rm, RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
3115
3116 rrd_flag_set(rm, RRD_FLAG_LIVE_RETENTION);
@@ -3114,7 +3129,7 @@ static inline bool rrdmetric_should_be_deleted(RRDMETRIC *rm) {
3129 return false;
3130
3131 rrdmetric_update_retention(rm);
3117 - if(rm->first_time_t || rm->last_time_t)
3132 + if(rm->first_time_s || rm->last_time_s)
3133 return false;
3134
3135 return true;
@@ -3136,7 +3151,7 @@ static inline bool rrdinstance_should_be_deleted(RRDINSTANCE *ri) {
3151 if(unlikely(dictionary_entries(ri->rrdmetrics) != 0))
3152 return false;
3153
3139 - if(ri->first_time_t || ri->last_time_t)
3154 + if(ri->first_time_s || ri->last_time_s)
3155 return false;
3156
3157 return true;
@@ -3155,7 +3170,7 @@ static inline bool rrdcontext_should_be_deleted(RRDCONTEXT *rc) {
3170 if(unlikely(dictionary_entries(rc->rrdinstances) != 0))
3171 return false;
3172
3158 - if(unlikely(rc->first_time_t || rc->last_time_t))
3173 + if(unlikely(rc->first_time_s || rc->last_time_s))
3174 return false;
3175
3176 return true;
@@ -3180,7 +3195,7 @@ static void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jo
3195
3196 RRDCONTEXT *rc;
3197 dfe_start_reentrant((DICTIONARY *)host->rrdctx, rc) {
3183 - if(unlikely(netdata_exit)) break;
3198 + if(unlikely(!service_running(SERVICE_CONTEXT))) break;
3199
3200 if(worker_jobs) worker_is_busy(WORKER_JOB_CLEANUP);
3201
@@ -3188,7 +3203,7 @@ static void rrdcontext_garbage_collect_single_host(RRDHOST *host, bool worker_jo
3203
3204 RRDINSTANCE *ri;
3205 dfe_start_reentrant(rc->rrdinstances, ri) {
3191 - if(unlikely(netdata_exit)) break;
3206 + if(unlikely(!service_running(SERVICE_CONTEXT))) break;
3207
3208 RRDMETRIC *rm;
3209 dfe_start_write(ri->rrdmetrics, rm) {
@@ -3304,7 +3319,7 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL
3319 if(dictionary_entries(ri->rrdmetrics) > 0) {
3320 RRDMETRIC *rm;
3321 dfe_start_read((DICTIONARY *)ri->rrdmetrics, rm) {
3307 - if(unlikely(netdata_exit)) break;
3322 + if(unlikely(!service_running(SERVICE_CONTEXT))) break;
3323
3324 RRD_FLAGS reason_to_pass = reason;
3325 if(rrd_flag_check(ri, RRD_FLAG_UPDATE_REASON_UPDATE_RETENTION))
@@ -3320,16 +3335,16 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL
3335 continue;
3336 }
3337
3323 - if(!currently_collected && rrd_flag_check(rm, RRD_FLAG_COLLECTED) && rm->first_time_t)
3338 + if(!currently_collected && rrd_flag_check(rm, RRD_FLAG_COLLECTED) && rm->first_time_s)
3339 currently_collected = true;
3340
3341 metrics_active++;
3342
3328 - if (rm->first_time_t && rm->first_time_t < min_first_time_t)
3329 - min_first_time_t = rm->first_time_t;
3343 + if (rm->first_time_s && rm->first_time_s < min_first_time_t)
3344 + min_first_time_t = rm->first_time_s;
3345
3331 - if (rm->last_time_t && rm->last_time_t > max_last_time_t)
3332 - max_last_time_t = rm->last_time_t;
3346 + if (rm->last_time_s && rm->last_time_s > max_last_time_t)
3347 + max_last_time_t = rm->last_time_s;
3348 }
3349 dfe_done(rm);
3350 }
@@ -3342,13 +3357,13 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL
3357 if(unlikely(!metrics_active)) {
3358 // no metrics available
3359
3345 - if(ri->first_time_t) {
3346 - ri->first_time_t = 0;
3360 + if(ri->first_time_s) {
3361 + ri->first_time_s = 0;
3362 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
3363 }
3364
3350 - if(ri->last_time_t) {
3351 - ri->last_time_t = 0;
3365 + if(ri->last_time_s) {
3366 + ri->last_time_s = 0;
3367 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
3368 }
3369
@@ -3361,13 +3376,13 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL
3376 min_first_time_t = 0;
3377
3378 if (unlikely(min_first_time_t == 0 || max_last_time_t == 0)) {
3364 - if(ri->first_time_t) {
3365 - ri->first_time_t = 0;
3379 + if(ri->first_time_s) {
3380 + ri->first_time_s = 0;
3381 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
3382 }
3383
3369 - if(ri->last_time_t) {
3370 - ri->last_time_t = 0;
3384 + if(ri->last_time_s) {
3385 + ri->last_time_s = 0;
3386 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
3387 }
3388
@@ -3377,13 +3392,13 @@ static void rrdinstance_post_process_updates(RRDINSTANCE *ri, bool force, RRD_FL
3392 else {
3393 rrd_flag_clear(ri, RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
3394
3380 - if (unlikely(ri->first_time_t != min_first_time_t)) {
3381 - ri->first_time_t = min_first_time_t;
3395 + if (unlikely(ri->first_time_s != min_first_time_t)) {
3396 + ri->first_time_s = min_first_time_t;
3397 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
3398 }
3399
3385 - if (unlikely(ri->last_time_t != max_last_time_t)) {
3386 - ri->last_time_t = max_last_time_t;
3400 + if (unlikely(ri->last_time_s != max_last_time_t)) {
3401 + ri->last_time_s = max_last_time_t;
3402 rrd_flag_set_updated(ri, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
3403 }
3404
@@ -3411,7 +3426,7 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG
3426 if(dictionary_entries(rc->rrdinstances) > 0) {
3427 RRDINSTANCE *ri;
3428 dfe_start_reentrant(rc->rrdinstances, ri) {
3414 - if(unlikely(netdata_exit)) break;
3429 + if(unlikely(!service_running(SERVICE_CONTEXT))) break;
3430
3431 RRD_FLAGS reason_to_pass = reason;
3432 if(rrd_flag_check(rc, RRD_FLAG_UPDATE_REASON_UPDATE_RETENTION))
@@ -3430,7 +3445,7 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG
3445 continue;
3446 }
3447
3433 - if(unlikely(!currently_collected && rrd_flag_is_collected(ri) && ri->first_time_t))
3448 + if(unlikely(!currently_collected && rrd_flag_is_collected(ri) && ri->first_time_s))
3449 currently_collected = true;
3450
3451 internal_error(rc->units != ri->units,
@@ -3443,11 +3458,11 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG
3458 if (ri->priority >= RRDCONTEXT_MINIMUM_ALLOWED_PRIORITY && ri->priority < min_priority)
3459 min_priority = ri->priority;
3460
3446 - if (ri->first_time_t && ri->first_time_t < min_first_time_t)
3447 - min_first_time_t = ri->first_time_t;
3461 + if (ri->first_time_s && ri->first_time_s < min_first_time_t)
3462 + min_first_time_t = ri->first_time_s;
3463
3449 - if (ri->last_time_t && ri->last_time_t > max_last_time_t)
3450 - max_last_time_t = ri->last_time_t;
3464 + if (ri->last_time_s && ri->last_time_s > max_last_time_t)
3465 + max_last_time_t = ri->last_time_s;
3466 }
3467 dfe_done(ri);
3468 }
@@ -3476,13 +3491,13 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG
3491 if(unlikely(!instances_active)) {
3492 // we had some instances, but they are gone now...
3493
3479 - if(rc->first_time_t) {
3480 - rc->first_time_t = 0;
3494 + if(rc->first_time_s) {
3495 + rc->first_time_s = 0;
3496 rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
3497 }
3498
3484 - if(rc->last_time_t) {
3485 - rc->last_time_t = 0;
3499 + if(rc->last_time_s) {
3500 + rc->last_time_s = 0;
3501 rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
3502 }
3503
@@ -3495,13 +3510,13 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG
3510 min_first_time_t = 0;
3511
3512 if (unlikely(min_first_time_t == 0 && max_last_time_t == 0)) {
3498 - if(rc->first_time_t) {
3499 - rc->first_time_t = 0;
3513 + if(rc->first_time_s) {
3514 + rc->first_time_s = 0;
3515 rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
3516 }
3517
3503 - if(rc->last_time_t) {
3504 - rc->last_time_t = 0;
3518 + if(rc->last_time_s) {
3519 + rc->last_time_s = 0;
3520 rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
3521 }
3522
@@ -3510,13 +3525,13 @@ static void rrdcontext_post_process_updates(RRDCONTEXT *rc, bool force, RRD_FLAG
3525 else {
3526 rrd_flag_clear(rc, RRD_FLAG_UPDATE_REASON_ZERO_RETENTION);
3527
3513 - if (unlikely(rc->first_time_t != min_first_time_t)) {
3514 - rc->first_time_t = min_first_time_t;
3528 + if (unlikely(rc->first_time_s != min_first_time_t)) {
3529 + rc->first_time_s = min_first_time_t;
3530 rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_FIRST_TIME_T);
3531 }
3532
3518 - if (rc->last_time_t != max_last_time_t) {
3519 - rc->last_time_t = max_last_time_t;
3533 + if (rc->last_time_s != max_last_time_t) {
3534 + rc->last_time_s = max_last_time_t;
3535 rrd_flag_set_updated(rc, RRD_FLAG_UPDATE_REASON_CHANGED_LAST_TIME_T);
3536 }
3537
@@ -3583,7 +3598,7 @@ static void rrdcontext_post_process_queued_contexts(RRDHOST *host) {
3598
3599 RRDCONTEXT *rc;
3600 dfe_start_reentrant((DICTIONARY *)host->rrdctx_post_processing_queue, rc) {
3586 - if(unlikely(netdata_exit)) break;
3601 + if(unlikely(!service_running(SERVICE_CONTEXT))) break;
3602
3603 rrdcontext_dequeue_from_post_processing(rc);
3604 rrdcontext_post_process_updates(rc, false, RRD_FLAG_NONE, true);
@@ -3612,8 +3627,8 @@ static void rrdcontext_message_send_unsafe(RRDCONTEXT *rc, bool snapshot __maybe
3627 rc->hub.family = string2str(rc->family);
3628 rc->hub.chart_type = rrdset_type_name(rc->chart_type);
3629 rc->hub.priority = rc->priority;
3615 - rc->hub.first_time_t = rc->first_time_t;
3616 - rc->hub.last_time_t = rrd_flag_is_collected(rc) ? 0 : rc->last_time_t;
3630 + rc->hub.first_time_s = rc->first_time_s;
3631 + rc->hub.last_time_s = rrd_flag_is_collected(rc) ? 0 : rc->last_time_s;
3632 rc->hub.deleted = rrd_flag_is_deleted(rc) ? true : false;
3633
3634 #ifdef ENABLE_ACLK
@@ -3625,8 +3640,8 @@ static void rrdcontext_message_send_unsafe(RRDCONTEXT *rc, bool snapshot __maybe
3640 .family = rc->hub.family,
3641 .chart_type = rc->hub.chart_type,
3642 .priority = rc->hub.priority,
3628 - .first_entry = rc->hub.first_time_t,
3629 - .last_entry = rc->hub.last_time_t,
3643 + .first_entry = rc->hub.first_time_s,
3644 + .last_entry = rc->hub.last_time_s,
3645 .deleted = rc->hub.deleted,
3646 };
3647
@@ -3680,10 +3695,10 @@ static bool check_if_cloud_version_changed_unsafe(RRDCONTEXT *rc, bool sending _
3695 if(unlikely(rc->priority != rc->hub.priority))
3696 priority_changed = true;
3697
3683 - if(unlikely((uint64_t)rc->first_time_t != rc->hub.first_time_t))
3698 + if(unlikely((uint64_t)rc->first_time_s != rc->hub.first_time_s))
3699 first_time_changed = true;
3700
3686 - if(unlikely((uint64_t)((flags & RRD_FLAG_COLLECTED) ? 0 : rc->last_time_t) != rc->hub.last_time_t))
3701 + if(unlikely((uint64_t)((flags & RRD_FLAG_COLLECTED) ? 0 : rc->last_time_s) != rc->hub.last_time_s))
3702 last_time_changed = true;
3703
3704 if(unlikely(((flags & RRD_FLAG_DELETED) ? true : false) != rc->hub.deleted))
@@ -3702,8 +3717,8 @@ static bool check_if_cloud_version_changed_unsafe(RRDCONTEXT *rc, bool sending _
3717 string2str(rc->family), family_changed ? " (CHANGED)" : "",
3718 rrdset_type_name(rc->chart_type), chart_type_changed ? " (CHANGED)" : "",
3719 rc->priority, priority_changed ? " (CHANGED)" : "",
3705 - rc->first_time_t, first_time_changed ? " (CHANGED)" : "",
3706 - (flags & RRD_FLAG_COLLECTED) ? 0 : rc->last_time_t, last_time_changed ? " (CHANGED)" : "",
3720 + rc->first_time_s, first_time_changed ? " (CHANGED)" : "",
3721 + (flags & RRD_FLAG_COLLECTED) ? 0 : rc->last_time_s, last_time_changed ? " (CHANGED)" : "",
3722 (flags & RRD_FLAG_DELETED) ? "true" : "false", deleted_changed ? " (CHANGED)" : "",
3723 sending ? (now_realtime_usec() - rc->queue.queued_ut) / USEC_PER_MS : 0,
3724 sending ? (rc->queue.scheduled_dispatch_ut - rc->queue.queued_ut) / USEC_PER_MS : 0
@@ -3764,7 +3779,7 @@ static void rrdcontext_dispatch_queued_contexts_to_hub(RRDHOST *host, usec_t now
3779
3780 RRDCONTEXT *rc;
3781 dfe_start_reentrant((DICTIONARY *)host->rrdctx_hub_queue, rc) {
3767 - if(unlikely(netdata_exit)) break;
3782 + if(unlikely(!service_running(SERVICE_CONTEXT))) break;
3783
3784 if(unlikely(messages_added >= MESSAGES_PER_BUNDLE_TO_SEND_TO_HUB_PER_HOST))
3785 break;
@@ -3831,7 +3846,7 @@ static void rrdcontext_dispatch_queued_contexts_to_hub(RRDHOST *host, usec_t now
3846 dfe_done(rc);
3847
3848 #ifdef ENABLE_ACLK
3834 - if(!netdata_exit && bundle) {
3849 + if(service_running(SERVICE_CONTEXT) && bundle) {
3850 // we have a bundle to send messages
3851
3852 // update the version hash
@@ -3882,11 +3897,11 @@ void *rrdcontext_main(void *ptr) {
3897 heartbeat_init(&hb);
3898 usec_t step = RRDCONTEXT_WORKER_THREAD_HEARTBEAT_USEC;
3899
3885 - while (!netdata_exit) {
3900 + while (service_running(SERVICE_CONTEXT)) {
3901 worker_is_idle();
3902 heartbeat_next(&hb, step);
3903
3889 - if(unlikely(netdata_exit)) break;
3904 + if(unlikely(!service_running(SERVICE_CONTEXT))) break;
3905
3906 usec_t now_ut = now_realtime_usec();
3907
@@ -3902,7 +3917,7 @@ void *rrdcontext_main(void *ptr) {
3917 rrd_rdlock();
3918 RRDHOST *host;
3919 rrdhost_foreach_read(host) {
3905 - if(unlikely(netdata_exit)) break;
3920 + if(unlikely(!service_running(SERVICE_CONTEXT))) break;
3921
3922 worker_is_busy(WORKER_JOB_HOSTS);
3923
database/rrdcontext.h
+27 -6
@@ -118,15 +118,35 @@ DICTIONARY *rrdcontext_all_metrics_to_dict(RRDHOST *host, SIMPLE_PATTERN *contex
118 // ----------------------------------------------------------------------------
119 // public API for queries
120
121 +typedef struct query_plan_entry {
122 + size_t tier;
123 + time_t after;
124 + time_t before;
125 + struct storage_engine_query_handle handle;
126 + STORAGE_POINT (*next_metric)(struct storage_engine_query_handle *handle);
127 + int (*is_finished)(struct storage_engine_query_handle *handle);
128 + void (*finalize)(struct storage_engine_query_handle *handle);
129 + bool initialized;
130 + bool finalized;
131 +} QUERY_PLAN_ENTRY;
132 +
133 +#define QUERY_PLANS_MAX (RRD_STORAGE_TIERS * 2)
134 +
135 typedef struct query_metric {
136 struct query_metric_tier {
137 struct storage_engine *eng;
138 STORAGE_METRIC_HANDLE *db_metric_handle;
125 - time_t db_first_time_t; // the oldest timestamp available for this tier
126 - time_t db_last_time_t; // the latest timestamp available for this tier
127 - time_t db_update_every; // latest update every for this tier
139 + time_t db_first_time_s; // the oldest timestamp available for this tier
140 + time_t db_last_time_s; // the latest timestamp available for this tier
141 + time_t db_update_every_s; // latest update every for this tier
142 + long weight;
143 } tiers[RRD_STORAGE_TIERS];
144
145 + struct {
146 + size_t used;
147 + QUERY_PLAN_ENTRY array[QUERY_PLANS_MAX];
148 + } plan;
149 +
150 struct {
151 RRDHOST *host;
152 RRDCONTEXT_ACQUIRED *rca;
@@ -172,6 +192,7 @@ typedef struct query_target_request {
192 time_t resampling_time;
193 size_t tier;
194 QUERY_SOURCE query_source;
195 + STORAGE_PRIORITY priority;
196 } QUERY_TARGET_REQUEST;
197
198 typedef struct query_target {
@@ -198,9 +219,9 @@ typedef struct query_target {
219 } window;
220
221 struct {
201 - time_t first_time_t; // the combined first_time_t of all metrics in the query, across all tiers
202 - time_t last_time_t; // the combined last_time_T of all metrics in the query, across all tiers
203 - time_t minimum_latest_update_every; // the min update every of the metrics in the query
222 + time_t first_time_s; // the combined first_time_t of all metrics in the query, across all tiers
223 + time_t last_time_s; // the combined last_time_T of all metrics in the query, across all tiers
224 + time_t minimum_latest_update_every_s; // the min update every of the metrics in the query
225 } db;
226
227 struct {
database/rrddim.c
+24 -23
@@ -64,7 +64,7 @@ static void rrddim_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, v
64 size_t entries = st->entries;
65 if(!entries) entries = 5;
66
67 - rd->db = netdata_mmap(NULL, entries * sizeof(storage_number), MAP_PRIVATE, 1);
67 + rd->db = netdata_mmap(NULL, entries * sizeof(storage_number), MAP_PRIVATE, 1, false);
68 if(!rd->db) {
69 info("Failed to use memory mode ram for chart '%s', dimension '%s', falling back to alloc", rrdset_name(st), rrddim_name(rd));
70 ctr->memory_mode = RRD_MEMORY_MODE_ALLOC;
@@ -285,7 +285,6 @@ static void rrddim_react_callback(const DICTIONARY_ITEM *item __maybe_unused, vo
285
286 if(ctr->react_action & (RRDDIM_REACT_UPDATED | RRDDIM_REACT_NEW)) {
287 rrddim_flag_set(rd, RRDDIM_FLAG_METADATA_UPDATE);
288 - rrdset_flag_set(rd->rrdset, RRDSET_FLAG_METADATA_UPDATE);
288 rrdhost_flag_set(rd->rrdset->rrdhost, RRDHOST_FLAG_METADATA_UPDATE);
289 }
290
@@ -420,44 +419,45 @@ inline int rrddim_set_divisor(RRDSET *st, RRDDIM *rd, collected_number divisor)
419
420 // ----------------------------------------------------------------------------
421
423 -time_t rrddim_last_entry_t_of_tier(RRDDIM *rd, size_t tier) {
422 +time_t rrddim_last_entry_s_of_tier(RRDDIM *rd, size_t tier) {
423 if(unlikely(tier > storage_tiers || !rd->tiers[tier]))
424 return 0;
425
427 - return rd->tiers[tier]->query_ops->latest_time(rd->tiers[tier]->db_metric_handle);
426 + return rd->tiers[tier]->query_ops->latest_time_s(rd->tiers[tier]->db_metric_handle);
427 }
428
430 -time_t rrddim_last_entry_t(RRDDIM *rd) {
431 - time_t latest = rd->tiers[0]->query_ops->latest_time(rd->tiers[0]->db_metric_handle);
429 +// get the timestamp of the last entry in the round-robin database
430 +time_t rrddim_last_entry_s(RRDDIM *rd) {
431 + time_t latest_time_s = rrddim_last_entry_s_of_tier(rd, 0);
432
433 for(size_t tier = 1; tier < storage_tiers ;tier++) {
434 if(unlikely(!rd->tiers[tier])) continue;
435
436 - time_t t = rd->tiers[tier]->query_ops->latest_time(rd->tiers[tier]->db_metric_handle);
437 - if(t > latest)
438 - latest = t;
436 + time_t t = rrddim_last_entry_s_of_tier(rd, tier);
437 + if(t > latest_time_s)
438 + latest_time_s = t;
439 }
440
441 - return latest;
441 + return latest_time_s;
442 }
443
444 -time_t rrddim_first_entry_t_of_tier(RRDDIM *rd, size_t tier) {
444 +time_t rrddim_first_entry_s_of_tier(RRDDIM *rd, size_t tier) {
445 if(unlikely(tier > storage_tiers || !rd->tiers[tier]))
446 return 0;
447
448 - return rd->tiers[tier]->query_ops->oldest_time(rd->tiers[tier]->db_metric_handle);
448 + return rd->tiers[tier]->query_ops->oldest_time_s(rd->tiers[tier]->db_metric_handle);
449 }
450
451 -time_t rrddim_first_entry_t(RRDDIM *rd) {
452 - time_t oldest = 0;
451 +time_t rrddim_first_entry_s(RRDDIM *rd) {
452 + time_t oldest_time_s = 0;
453
454 for(size_t tier = 0; tier < storage_tiers ;tier++) {
455 - time_t t = rrddim_first_entry_t_of_tier(rd, tier);
456 - if(t != 0 && (oldest == 0 || t < oldest))
457 - oldest = t;
455 + time_t t = rrddim_first_entry_s_of_tier(rd, tier);
456 + if(t != 0 && (oldest_time_s == 0 || t < oldest_time_s))
457 + oldest_time_s = t;
458 }
459
460 - return oldest;
460 + return oldest_time_s;
461 }
462
463 RRDDIM *rrddim_add_custom(RRDSET *st
@@ -504,8 +504,8 @@ int rrddim_hide(RRDSET *st, const char *id) {
504 return 1;
505 }
506 if (!rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN)) {
507 - rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);
508 - metaqueue_dimension_update_flags(rd);
507 + rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN | RRDDIM_FLAG_METADATA_UPDATE);
508 + rrdhost_flag_set(rd->rrdset->rrdhost, RRDHOST_FLAG_METADATA_UPDATE);
509 }
510
511 rrddim_option_set(rd, RRDDIM_OPTION_HIDDEN);
@@ -524,7 +524,8 @@ int rrddim_unhide(RRDSET *st, const char *id) {
524 }
525 if (rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN)) {
526 rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
527 - metaqueue_dimension_update_flags(rd);
527 + rrddim_flag_set(rd, RRDDIM_FLAG_METADATA_UPDATE);
528 + rrdhost_flag_set(rd->rrdset->rrdhost, RRDHOST_FLAG_METADATA_UPDATE);
529 }
530
531 rrddim_option_clear(rd, RRDDIM_OPTION_HIDDEN);
@@ -694,8 +695,8 @@ bool rrddim_memory_load_or_create_map_save(RRDSET *st, RRDDIM *rd, RRD_MEMORY_MO
695 rrdset_strncpyz_name(filename, rrddim_id(rd), FILENAME_MAX);
696 snprintfz(fullfilename, FILENAME_MAX, "%s/%s.db", st->cache_dir, filename);
697
697 - rd_on_file = (struct rrddim_map_save_v019 *)netdata_mmap(fullfilename, size,
698 - ((memory_mode == RRD_MEMORY_MODE_MAP) ? MAP_SHARED : MAP_PRIVATE), 1);
698 + rd_on_file = (struct rrddim_map_save_v019 *)netdata_mmap(
699 + fullfilename, size, ((memory_mode == RRD_MEMORY_MODE_MAP) ? MAP_SHARED : MAP_PRIVATE), 1, false);
700
701 if(unlikely(!rd_on_file)) return false;
702
database/rrdhost.c
+183 -225
@@ -3,6 +3,8 @@
3 #define NETDATA_RRD_INTERNALS
4 #include "rrd.h"
5
6 +static void rrdhost_streaming_sender_structures_init(RRDHOST *host);
7 +
8 bool dbengine_enabled = false; // will become true if and when dbengine is initialized
9 size_t storage_tiers = 3;
10 size_t storage_tiers_grouping_iterations[RRD_STORAGE_TIERS] = { 1, 60, 60, 60, 60 };
@@ -24,11 +26,10 @@ size_t get_tier_grouping(size_t tier) {
26 }
27
28 RRDHOST *localhost = NULL;
27 -size_t rrd_hosts_available = 0;
29 netdata_rwlock_t rrd_rwlock = NETDATA_RWLOCK_INITIALIZER;
30
30 -time_t rrdset_free_obsolete_time = 3600;
31 -time_t rrdhost_free_orphan_time = 3600;
31 +time_t rrdset_free_obsolete_time_s = 3600;
32 +time_t rrdhost_free_orphan_time_s = 3600;
33
34 bool is_storage_engine_shared(STORAGE_INSTANCE *engine) {
35 #ifdef ENABLE_DBENGINE
@@ -63,7 +64,7 @@ static inline void rrdhost_init() {
64 // ----------------------------------------------------------------------------
65 // RRDHOST index by UUID
66
66 -inline long rrdhost_hosts_available(void) {
67 +inline size_t rrdhost_hosts_available(void) {
68 return dictionary_entries(rrdhost_root_index);
69 }
70
@@ -139,7 +140,7 @@ static inline void rrdhost_init_tags(RRDHOST *host, const char *tags) {
140 string_freez(old);
141 }
142
142 -static inline void rrdhost_init_hostname(RRDHOST *host, const char *hostname) {
143 +static inline void rrdhost_init_hostname(RRDHOST *host, const char *hostname, bool add_to_index) {
144 if(unlikely(hostname && !*hostname)) hostname = NULL;
145
146 if(host->hostname && hostname && !strcmp(rrdhost_hostname(host), hostname))
@@ -151,7 +152,8 @@ static inline void rrdhost_init_hostname(RRDHOST *host, const char *hostname) {
152 host->hostname = string_strdupz(hostname?hostname:"localhost");
153 string_freez(old);
154
154 - rrdhost_index_add_hostname(host);
155 + if(add_to_index)
156 + rrdhost_index_add_hostname(host);
157 }
158
159 static inline void rrdhost_init_os(RRDHOST *host, const char *os) {
@@ -211,7 +213,7 @@ static void rrdhost_initialize_rrdpush_sender(RRDHOST *host,
213 if(rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key) {
214 rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_INITIALIZED);
215
214 - sender_init(host);
216 + rrdhost_streaming_sender_structures_init(host);
217
218 #ifdef ENABLE_HTTPS
219 host->sender->ssl.conn = NULL;
@@ -230,35 +232,34 @@ static void rrdhost_initialize_rrdpush_sender(RRDHOST *host,
232 rrdhost_option_clear(host, RRDHOST_OPTION_SENDER_ENABLED);
233 }
234
233 -RRDHOST *rrdhost_create(const char *hostname,
234 - const char *registry_hostname,
235 - const char *guid,
236 - const char *os,
237 - const char *timezone,
238 - const char *abbrev_timezone,
239 - int32_t utc_offset,
240 - const char *tags,
241 - const char *program_name,
242 - const char *program_version,
243 - int update_every,
244 - long entries,
245 - RRD_MEMORY_MODE memory_mode,
246 - unsigned int health_enabled,
247 - unsigned int rrdpush_enabled,
248 - char *rrdpush_destination,
249 - char *rrdpush_api_key,
250 - char *rrdpush_send_charts_matching,
251 - bool rrdpush_enable_replication,
252 - time_t rrdpush_seconds_to_replicate,
253 - time_t rrdpush_replication_step,
254 - struct rrdhost_system_info *system_info,
255 - int is_localhost,
256 - bool archived
235 +static RRDHOST *rrdhost_create(
236 + const char *hostname,
237 + const char *registry_hostname,
238 + const char *guid,
239 + const char *os,
240 + const char *timezone,
241 + const char *abbrev_timezone,
242 + int32_t utc_offset,
243 + const char *tags,
244 + const char *program_name,
245 + const char *program_version,
246 + int update_every,
247 + long entries,
248 + RRD_MEMORY_MODE memory_mode,
249 + unsigned int health_enabled,
250 + unsigned int rrdpush_enabled,
251 + char *rrdpush_destination,
252 + char *rrdpush_api_key,
253 + char *rrdpush_send_charts_matching,
254 + bool rrdpush_enable_replication,
255 + time_t rrdpush_seconds_to_replicate,
256 + time_t rrdpush_replication_step,
257 + struct rrdhost_system_info *system_info,
258 + int is_localhost,
259 + bool archived
260 ) {
261 debug(D_RRDHOST, "Host '%s': adding with guid '%s'", hostname, guid);
262
260 - rrd_check_wrlock();
261 -
263 if(memory_mode == RRD_MEMORY_MODE_DBENGINE && !dbengine_enabled) {
264 error("memory mode 'dbengine' is not enabled, but host '%s' is configured for it. Falling back to 'alloc'", hostname);
265 memory_mode = RRD_MEMORY_MODE_ALLOC;
@@ -278,7 +279,7 @@ int is_legacy = 1;
279 set_host_properties(host, (update_every > 0)?update_every:1, memory_mode, registry_hostname, os,
280 tags, timezone, abbrev_timezone, utc_offset, program_name, program_version);
281
281 - rrdhost_init_hostname(host, hostname);
282 + rrdhost_init_hostname(host, hostname, false);
283
284 host->rrd_history_entries = align_entries_to_pagesize(memory_mode, entries);
285 host->health_enabled = ((memory_mode == RRD_MEMORY_MODE_NONE)) ? 0 : health_enabled;
@@ -312,7 +313,6 @@ int is_legacy = 1;
313 break;
314 }
315
315 - netdata_rwlock_init(&host->rrdhost_rwlock);
316 netdata_mutex_init(&host->aclk_state_lock);
317 netdata_mutex_init(&host->receiver_lock);
318
@@ -356,18 +356,8 @@ int is_legacy = 1;
356 if(!host->rrdvars)
357 host->rrdvars = rrdvariables_create();
358
359 - RRDHOST *t = rrdhost_index_add_by_guid(host);
360 - if(t != host) {
361 - error("Host '%s': cannot add host with machine guid '%s' to index. It already exists as host '%s' with machine guid '%s'.", rrdhost_hostname(host), host->machine_guid, rrdhost_hostname(t), t->machine_guid);
362 - rrdhost_free(host, 1);
363 - return NULL;
364 - }
365 -
366 - if (likely(!uuid_parse(host->machine_guid, host->host_uuid))) {
367 - if(!archived)
368 - metaqueue_host_update_info(host->machine_guid);
359 + if (likely(!uuid_parse(host->machine_guid, host->host_uuid)))
360 sql_load_node_id(host);
370 - }
361 else
362 error_report("Host machine GUID %s is not valid", host->machine_guid);
363
@@ -385,9 +375,12 @@ int is_legacy = 1;
375
376 snprintfz(dbenginepath, FILENAME_MAX, "%s/dbengine", host->cache_dir);
377 ret = mkdir(dbenginepath, 0775);
378 +
379 if (ret != 0 && errno != EEXIST)
380 error("Host '%s': cannot create directory '%s'", rrdhost_hostname(host), dbenginepath);
390 - else ret = 0; // succeed
381 + else
382 + ret = 0; // succeed
383 +
384 if (is_legacy) {
385 // initialize legacy dbengine instance as needed
386
@@ -422,15 +415,17 @@ int is_legacy = 1;
415 host->db[tier].tier_grouping = get_tier_grouping(tier);
416 }
417 }
418 +
419 if (ret) { // check legacy or multihost initialization success
420 error(
421 "Host '%s': cannot initialize host with machine guid '%s'. Failed to initialize DB engine at '%s'.",
422 rrdhost_hostname(host), host->machine_guid, host->cache_dir);
429 - rrdhost_free(host, 1);
430 - host = NULL;
431 - //rrd_hosts_available++; //TODO: maybe we want this?
423
433 - return host;
424 + rrd_wrlock();
425 + rrdhost_free___while_having_rrd_wrlock(host, true);
426 + rrd_unlock();
427 +
428 + return NULL;
429 }
430
431 #else
@@ -454,14 +449,6 @@ int is_legacy = 1;
449 #endif
450 }
451
457 - // ------------------------------------------------------------------------
458 - // link it and add it to the index
459 -
460 - if(is_localhost)
461 - DOUBLE_LINKED_LIST_PREPEND_UNSAFE(localhost, host, prev, next);
462 - else
463 - DOUBLE_LINKED_LIST_APPEND_UNSAFE(localhost, host, prev, next);
464 -
452 // ------------------------------------------------------------------------
453 // init new ML host and update system_info to let upstreams know
454 // about ML functionality
@@ -473,6 +460,30 @@ int is_legacy = 1;
460 host->system_info->mc_version = enable_metric_correlations ? metric_correlations_version : 0;
461 }
462
463 + // ------------------------------------------------------------------------
464 + // link it and add it to the index
465 +
466 + rrd_wrlock();
467 +
468 + RRDHOST *t = rrdhost_index_add_by_guid(host);
469 + if(t != host) {
470 + error("Host '%s': cannot add host with machine guid '%s' to index. It already exists as host '%s' with machine guid '%s'.", rrdhost_hostname(host), host->machine_guid, rrdhost_hostname(t), t->machine_guid);
471 + rrdhost_free___while_having_rrd_wrlock(host, true);
472 + rrd_unlock();
473 + return NULL;
474 + }
475 +
476 + rrdhost_index_add_hostname(host);
477 +
478 + if(is_localhost)
479 + DOUBLE_LINKED_LIST_PREPEND_UNSAFE(localhost, host, prev, next);
480 + else
481 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(localhost, host, prev, next);
482 +
483 + rrd_unlock();
484 +
485 + // ------------------------------------------------------------------------
486 +
487 info("Host '%s' (at registry as '%s') with guid '%s' initialized"
488 ", os '%s'"
489 ", timezone '%s'"
@@ -511,23 +522,21 @@ int is_legacy = 1;
522 , string2str(host->health_default_exec)
523 , string2str(host->health_default_recipient)
524 );
514 - if(!archived)
515 - metaqueue_host_update_system_info(host);
525
517 - rrd_hosts_available++;
526 + if(!archived)
527 + rrdhost_flag_set(host,RRDHOST_FLAG_METADATA_INFO | RRDHOST_FLAG_METADATA_UPDATE);
528
529 rrdhost_load_rrdcontext_data(host);
530 if (!archived) {
531 ml_host_new(host);
532 ml_start_anomaly_detection_threads(host);
533 } else
524 - rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED);
525 -
534 + rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED | RRDHOST_FLAG_ORPHAN);
535
536 return host;
537 }
538
530 -void rrdhost_update(RRDHOST *host
539 +static void rrdhost_update(RRDHOST *host
540 , const char *hostname
541 , const char *registry_hostname
542 , const char *guid
@@ -554,11 +563,16 @@ void rrdhost_update(RRDHOST *host
563 {
564 UNUSED(guid);
565
566 + netdata_spinlock_lock(&host->rrdhost_update_lock);
567 +
568 host->health_enabled = (mode == RRD_MEMORY_MODE_NONE) ? 0 : health_enabled;
569
559 - rrdhost_system_info_free(host->system_info);
560 - host->system_info = system_info;
561 - metaqueue_host_update_system_info(host);
570 + {
571 + struct rrdhost_system_info *old = host->system_info;
572 + host->system_info = system_info;
573 + rrdhost_flag_set(host, RRDHOST_FLAG_METADATA_INFO | RRDHOST_FLAG_METADATA_CLAIMID | RRDHOST_FLAG_METADATA_UPDATE);
574 + rrdhost_system_info_free(old);
575 + }
576
577 rrdhost_init_os(host, os);
578 rrdhost_init_timezone(host, timezone, abbrev_timezone, utc_offset);
@@ -568,7 +582,7 @@ void rrdhost_update(RRDHOST *host
582
583 if(strcmp(rrdhost_hostname(host), hostname) != 0) {
584 info("Host '%s' has been renamed to '%s'. If this is not intentional it may mean multiple hosts are using the same machine_guid.", rrdhost_hostname(host), hostname);
571 - rrdhost_init_hostname(host, hostname);
585 + rrdhost_init_hostname(host, hostname, true);
586 }
587
588 if(strcmp(rrdhost_program_name(host), program_name) != 0) {
@@ -629,17 +643,17 @@ void rrdhost_update(RRDHOST *host
643 host->rrdpush_seconds_to_replicate = rrdpush_seconds_to_replicate;
644 host->rrdpush_replication_step = rrdpush_replication_step;
645
632 - rrd_hosts_available++;
633 -
646 ml_host_new(host);
647 ml_start_anomaly_detection_threads(host);
636 -
648 +
649 rrdhost_load_rrdcontext_data(host);
650 info("Host %s is not in archived mode anymore", rrdhost_hostname(host));
651 }
652
653 if (health_enabled)
654 health_thread_spawn(host);
655 +
656 + netdata_spinlock_unlock(&host->rrdhost_update_lock);
657 }
658
659 RRDHOST *rrdhost_find_or_create(
@@ -669,15 +683,18 @@ RRDHOST *rrdhost_find_or_create(
683 ) {
684 debug(D_RRDHOST, "Searching for host '%s' with guid '%s'", hostname, guid);
685
672 - rrd_wrlock();
686 RRDHOST *host = rrdhost_find_by_guid(guid);
687 if (unlikely(host && host->rrd_memory_mode != mode && rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))) {
688 /* If a legacy memory mode instantiates all dbengine state must be discarded to avoid inconsistencies */
689 error("Archived host '%s' has memory mode '%s', but the wanted one is '%s'. Discarding archived state.",
690 rrdhost_hostname(host), rrd_memory_mode_name(host->rrd_memory_mode), rrd_memory_mode_name(mode));
678 - rrdhost_free(host, 1);
691 +
692 + rrd_wrlock();
693 + rrdhost_free___while_having_rrd_wrlock(host, true);
694 host = NULL;
695 + rrd_unlock();
696 }
697 +
698 if(!host) {
699 host = rrdhost_create(
700 hostname
@@ -707,6 +724,7 @@ RRDHOST *rrdhost_find_or_create(
724 );
725 }
726 else {
727 +
728 rrdhost_update(host
729 , hostname
730 , registry_hostname
@@ -730,19 +748,13 @@ RRDHOST *rrdhost_find_or_create(
748 , rrdpush_seconds_to_replicate
749 , rrdpush_replication_step
750 , system_info);
733 - }
734 - if (host) {
735 - rrdhost_wrlock(host);
736 - rrdhost_flag_clear(host, RRDHOST_FLAG_ORPHAN);
737 - host->senders_disconnected_time = 0;
738 - rrdhost_unlock(host);
739 - }
751
741 - rrd_unlock();
752 + }
753
754 return host;
755 }
745 -inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, time_t now) {
756 +
757 +inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, time_t now_s) {
758 if(host != protected_host
759 && host != localhost
760 && rrdhost_receiver_replicating_charts(host) == 0
@@ -750,8 +762,8 @@ inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, tim
762 && rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN)
763 && !rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED)
764 && !host->receiver
753 - && host->senders_disconnected_time
754 - && host->senders_disconnected_time + rrdhost_free_orphan_time < now)
765 + && host->child_disconnected_time
766 + && host->child_disconnected_time + rrdhost_free_orphan_time_s < now_s)
767 return 1;
768
769 return 0;
@@ -762,6 +774,14 @@ inline int rrdhost_should_be_removed(RRDHOST *host, RRDHOST *protected_host, tim
774
775 void dbengine_init(char *hostname) {
776 #ifdef ENABLE_DBENGINE
777 + unsigned read_num = (unsigned)config_get_number(CONFIG_SECTION_DB, "dbengine pages per extent", MAX_PAGES_PER_EXTENT);
778 + if (read_num > 0 && read_num <= MAX_PAGES_PER_EXTENT)
779 + rrdeng_pages_per_extent = read_num;
780 + else {
781 + error("Invalid dbengine pages per extent %u given. Using %u.", read_num, rrdeng_pages_per_extent);
782 + config_set_number(CONFIG_SECTION_DB, "dbengine pages per extent", rrdeng_pages_per_extent);
783 + }
784 +
785 storage_tiers = config_get_number(CONFIG_SECTION_DB, "storage tiers", storage_tiers);
786 if(storage_tiers < 1) {
787 error("At least 1 storage tier is required. Assuming 1.");
@@ -788,11 +808,6 @@ void dbengine_init(char *hostname) {
808 config_set_number(CONFIG_SECTION_DB, "dbengine page fetch retries", default_rrdeng_page_fetch_retries);
809 }
810
791 - if(config_get_boolean(CONFIG_SECTION_DB, "dbengine page descriptors in file mapped memory", rrdeng_page_descr_is_mmap()) == CONFIG_BOOLEAN_YES)
792 - rrdeng_page_descr_use_mmap();
793 - else
794 - rrdeng_page_descr_use_malloc();
795 -
811 size_t created_tiers = 0;
812 char dbenginepath[FILENAME_MAX + 1];
813 char dbengineconfig[200 + 1];
@@ -884,7 +899,7 @@ void dbengine_init(char *hostname) {
899 #endif
900 }
901
887 -int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
902 +int rrd_init(char *hostname, struct rrdhost_system_info *system_info, bool unittest) {
903 rrdhost_init();
904
905 if (unlikely(sql_init_database(DB_CHECK_NONE, system_info ? 0 : 1))) {
@@ -897,7 +912,7 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
912 error_report("Failed to initialize context metadata database");
913 }
914
900 - if (unlikely(strcmp(hostname, "unittest") == 0)) {
915 + if (unlikely(unittest)) {
916 dbengine_enabled = true;
917 }
918 else {
@@ -905,11 +920,11 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
920 rrdpush_init();
921
922 if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE || rrdpush_receiver_needs_dbengine()) {
908 - info("Initializing dbengine...");
923 + info("DBENGINE: Initializing ...");
924 dbengine_init(hostname);
925 }
926 else {
912 - info("Not initializing dbengine...");
927 + info("DBENGINE: Not initializing ...");
928 storage_tiers = 1;
929 }
930
@@ -927,42 +942,41 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
942 }
943 }
944
930 - metadata_sync_init();
945 + if(!unittest)
946 + metadata_sync_init();
947 +
948 debug(D_RRDHOST, "Initializing localhost with hostname '%s'", hostname);
932 - rrd_wrlock();
949 localhost = rrdhost_create(
934 - hostname
935 - , registry_get_this_machine_hostname()
950 + hostname
951 + , registry_get_this_machine_hostname()
952 , registry_get_this_machine_guid()
953 , os_type
938 - , netdata_configured_timezone
939 - , netdata_configured_abbrev_timezone
940 - , netdata_configured_utc_offset
941 - , ""
942 - , program_name
943 - , program_version
944 - , default_rrd_update_every
945 - , default_rrd_history_entries
946 - , default_rrd_memory_mode
947 - , default_health_enabled
948 - , default_rrdpush_enabled
949 - , default_rrdpush_destination
950 - , default_rrdpush_api_key
951 - , default_rrdpush_send_charts_matching
952 - , default_rrdpush_enable_replication
953 - , default_rrdpush_seconds_to_replicate
954 - , default_rrdpush_replication_step
955 - , system_info
956 - , 1
957 - , 0
954 + , netdata_configured_timezone
955 + , netdata_configured_abbrev_timezone
956 + , netdata_configured_utc_offset
957 + , ""
958 + , program_name
959 + , program_version
960 + , default_rrd_update_every
961 + , default_rrd_history_entries
962 + , default_rrd_memory_mode
963 + , default_health_enabled
964 + , default_rrdpush_enabled
965 + , default_rrdpush_destination
966 + , default_rrdpush_api_key
967 + , default_rrdpush_send_charts_matching
968 + , default_rrdpush_enable_replication
969 + , default_rrdpush_seconds_to_replicate
970 + , default_rrdpush_replication_step
971 + , system_info
972 + , 1
973 + , 0
974 );
975 +
976 if (unlikely(!localhost)) {
960 - rrd_unlock();
977 return 1;
978 }
979
964 - rrd_unlock();
965 -
980 if (likely(system_info)) {
981 migrate_localhost(&localhost->host_uuid);
982 sql_aclk_sync_init();
@@ -971,42 +985,6 @@ int rrd_init(char *hostname, struct rrdhost_system_info *system_info) {
985 return localhost==NULL;
986 }
987
974 -// ----------------------------------------------------------------------------
975 -// RRDHOST - lock validations
976 -// there are only used when NETDATA_INTERNAL_CHECKS is set
977 -
978 -void __rrdhost_check_rdlock(RRDHOST *host, const char *file, const char *function, const unsigned long line) {
979 - debug(D_RRDHOST, "Checking read lock on host '%s'", rrdhost_hostname(host));
980 -
981 - int ret = netdata_rwlock_trywrlock(&host->rrdhost_rwlock);
982 - if(ret == 0)
983 - fatal("RRDHOST '%s' should be read-locked, but it is not, at function %s() at line %lu of file '%s'", rrdhost_hostname(host), function, line, file);
984 -}
985 -
986 -void __rrdhost_check_wrlock(RRDHOST *host, const char *file, const char *function, const unsigned long line) {
987 - debug(D_RRDHOST, "Checking write lock on host '%s'", rrdhost_hostname(host));
988 -
989 - int ret = netdata_rwlock_tryrdlock(&host->rrdhost_rwlock);
990 - if(ret == 0)
991 - fatal("RRDHOST '%s' should be write-locked, but it is not, at function %s() at line %lu of file '%s'", rrdhost_hostname(host), function, line, file);
992 -}
993 -
994 -void __rrd_check_rdlock(const char *file, const char *function, const unsigned long line) {
995 - debug(D_RRDHOST, "Checking read lock on all RRDs");
996 -
997 - int ret = netdata_rwlock_trywrlock(&rrd_rwlock);
998 - if(ret == 0)
999 - fatal("RRDs should be read-locked, but it are not, at function %s() at line %lu of file '%s'", function, line, file);
1000 -}
1001 -
1002 -void __rrd_check_wrlock(const char *file, const char *function, const unsigned long line) {
1003 - debug(D_RRDHOST, "Checking write lock on all RRDs");
1004 -
1005 - int ret = netdata_rwlock_tryrdlock(&rrd_rwlock);
1006 - if(ret == 0)
1007 - fatal("RRDs should be write-locked, but it are not, at function %s() at line %lu of file '%s'", function, line, file);
1008 -}
1009 -
988 // ----------------------------------------------------------------------------
989 // RRDHOST - free
990
@@ -1046,16 +1024,41 @@ void rrdhost_system_info_free(struct rrdhost_system_info *system_info) {
1024 }
1025 }
1026
1049 -void destroy_receiver_state(struct receiver_state *rpt);
1027 +static void rrdhost_streaming_sender_structures_init(RRDHOST *host)
1028 +{
1029 + if (host->sender)
1030 + return;
1031 +
1032 + host->sender = callocz(1, sizeof(*host->sender));
1033 + host->sender->host = host;
1034 + host->sender->buffer = cbuffer_new(CBUFFER_INITIAL_SIZE, 1024 * 1024);
1035 + host->sender->capabilities = STREAM_OUR_CAPABILITIES;
1036 +
1037 + host->sender->rrdpush_sender_pipe[PIPE_READ] = -1;
1038 + host->sender->rrdpush_sender_pipe[PIPE_WRITE] = -1;
1039 + host->sender->rrdpush_sender_socket = -1;
1040
1051 -void stop_streaming_sender(RRDHOST *host)
1041 +#ifdef ENABLE_COMPRESSION
1042 + if(default_compression_enabled) {
1043 + host->sender->flags |= SENDER_FLAG_COMPRESSION;
1044 + host->sender->compressor = create_compressor();
1045 + }
1046 + else
1047 + host->sender->flags &= ~SENDER_FLAG_COMPRESSION;
1048 +#endif
1049 +
1050 + netdata_mutex_init(&host->sender->mutex);
1051 + replication_init_sender(host->sender);
1052 +}
1053 +
1054 +static void rrdhost_streaming_sender_structures_free(RRDHOST *host)
1055 {
1056 rrdhost_option_clear(host, RRDHOST_OPTION_SENDER_ENABLED);
1057
1058 if (unlikely(!host->sender))
1059 return;
1060
1058 - rrdpush_sender_thread_stop(host); // stop a possibly running thread
1061 + rrdpush_sender_thread_stop(host, "HOST CLEANUP", true); // stop a possibly running thread
1062 cbuffer_free(host->sender->buffer);
1063 #ifdef ENABLE_COMPRESSION
1064 if (host->sender->compressor)
@@ -1067,38 +1070,29 @@ void stop_streaming_sender(RRDHOST *host)
1070 rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_INITIALIZED);
1071 }
1072
1070 -void stop_streaming_receiver(RRDHOST *host)
1071 -{
1072 - netdata_mutex_lock(&host->receiver_lock);
1073 - if (host->receiver) {
1074 - if (!host->receiver->exited)
1075 - netdata_thread_cancel(host->receiver->thread);
1076 - netdata_mutex_unlock(&host->receiver_lock);
1077 - struct receiver_state *rpt = host->receiver;
1078 - while (host->receiver && !rpt->exited)
1079 - sleep_usec(50 * USEC_PER_MS);
1080 - // If the receiver detached from the host then its thread will destroy the state
1081 - if (host->receiver == rpt)
1082 - destroy_receiver_state(host->receiver);
1083 - } else
1084 - netdata_mutex_unlock(&host->receiver_lock);
1085 -}
1086 -
1087 -void rrdhost_free(RRDHOST *host, bool force) {
1073 +void rrdhost_free___while_having_rrd_wrlock(RRDHOST *host, bool force) {
1074 if(!host) return;
1075
1090 - if (netdata_exit || force)
1076 + if (netdata_exit || force) {
1077 info("Freeing all memory for host '%s'...", rrdhost_hostname(host));
1078
1093 - rrd_check_wrlock(); // make sure the RRDs are write locked
1079 + // ------------------------------------------------------------------------
1080 + // first remove it from the indexes, so that it will not be discoverable
1081 +
1082 + rrdhost_index_del_hostname(host);
1083 + rrdhost_index_del_by_guid(host);
1084 +
1085 + if (host->prev)
1086 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(localhost, host, prev, next);
1087 + }
1088
1089 // ------------------------------------------------------------------------
1090 // clean up streaming
1091
1098 - stop_streaming_sender(host);
1092 + rrdhost_streaming_sender_structures_free(host);
1093
1094 if (netdata_exit || force)
1101 - stop_streaming_receiver(host);
1095 + stop_streaming_receiver(host, "HOST CLEANUP");
1096
1097
1098 // ------------------------------------------------------------------------
@@ -1106,9 +1100,6 @@ void rrdhost_free(RRDHOST *host, bool force) {
1100
1101 rrdcalc_delete_all(host);
1102
1109 -
1110 - rrdhost_wrlock(host); // lock this RRDHOST
1111 -
1103 // ------------------------------------------------------------------------
1104 // release its children resources
1105
@@ -1145,8 +1136,7 @@ void rrdhost_free(RRDHOST *host, bool force) {
1136
1137 if (!netdata_exit && !force) {
1138 info("Setting archive mode for host '%s'...", rrdhost_hostname(host));
1148 - rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED);
1149 - rrdhost_unlock(host);
1139 + rrdhost_flag_set(host, RRDHOST_FLAG_ARCHIVED | RRDHOST_FLAG_ORPHAN);
1140 return;
1141 }
1142
@@ -1165,17 +1155,6 @@ void rrdhost_free(RRDHOST *host, bool force) {
1155 }
1156 #endif
1157
1168 - // ------------------------------------------------------------------------
1169 - // remove it from the indexes
1170 -
1171 - rrdhost_index_del_hostname(host);
1172 - rrdhost_index_del_by_guid(host);
1173 -
1174 - // ------------------------------------------------------------------------
1175 - // unlink it from the host
1176 -
1177 - DOUBLE_LINKED_LIST_REMOVE_UNSAFE(localhost, host, prev, next);
1178 -
1158 // ------------------------------------------------------------------------
1159 // free it
1160
@@ -1200,9 +1179,7 @@ void rrdhost_free(RRDHOST *host, bool force) {
1179 freez(host->health_log_filename);
1180 string_freez(host->registry_hostname);
1181 simple_pattern_free(host->rrdpush_send_charts_matching);
1203 - rrdhost_unlock(host);
1182 netdata_rwlock_destroy(&host->health_log.alarm_log_rwlock);
1205 - netdata_rwlock_destroy(&host->rrdhost_rwlock);
1183 freez(host->node_id);
1184
1185 rrdfamily_index_destroy(host);
@@ -1217,7 +1194,6 @@ void rrdhost_free(RRDHOST *host, bool force) {
1194 if (wc)
1195 wc->is_orphan = 0;
1196 #endif
1220 - rrd_hosts_available--;
1197 }
1198
1199 void rrdhost_free_all(void) {
@@ -1225,10 +1201,10 @@ void rrdhost_free_all(void) {
1201
1202 /* Make sure child-hosts are released before the localhost. */
1203 while(localhost && localhost->next)
1228 - rrdhost_free(localhost->next, 1);
1204 + rrdhost_free___while_having_rrd_wrlock(localhost->next, true);
1205
1206 if(localhost)
1231 - rrdhost_free(localhost, 1);
1207 + rrdhost_free___while_having_rrd_wrlock(localhost, true);
1208
1209 rrd_unlock();
1210 }
@@ -1316,8 +1292,7 @@ static void rrdhost_load_auto_labels(void) {
1292
1293 health_add_host_labels();
1294
1319 - rrdlabels_add(
1320 - labels, "_is_parent", (localhost->senders_count > 0) ? "true" : "false", RRDLABEL_SRC_AUTO);
1295 + rrdlabels_add(labels, "_is_parent", (localhost->connected_children_count > 0) ? "true" : "false", RRDLABEL_SRC_AUTO);
1296
1297 if (localhost->rrdpush_send_destination)
1298 rrdlabels_add(labels, "_streams_to", localhost->rrdpush_send_destination, RRDLABEL_SRC_AUTO);
@@ -1395,8 +1370,7 @@ void reload_host_labels(void) {
1370 rrdhost_load_kubernetes_labels();
1371 rrdhost_load_auto_labels();
1372
1398 - rrdlabels_remove_all_unmarked(localhost->rrdlabels);
1399 - metaqueue_store_host_labels(localhost->machine_guid);
1373 + rrdhost_flag_set(localhost,RRDHOST_FLAG_METADATA_LABELS | RRDHOST_FLAG_METADATA_UPDATE);
1374
1375 health_label_log_save(localhost);
1376
@@ -1457,7 +1431,7 @@ void rrdhost_cleanup_charts(RRDHOST *host) {
1431 // RRDHOST - save all hosts to disk
1432
1433 void rrdhost_save_all(void) {
1460 - info("Saving database [%zu hosts(s)]...", rrd_hosts_available);
1434 + info("Saving database [%zu hosts(s)]...", rrdhost_hosts_available());
1435
1436 rrd_rdlock();
1437
@@ -1472,7 +1446,7 @@ void rrdhost_save_all(void) {
1446 // RRDHOST - save or delete all hosts from disk
1447
1448 void rrdhost_cleanup_all(void) {
1475 - info("Cleaning up database [%zu hosts(s)]...", rrd_hosts_available);
1449 + info("Cleaning up database [%zu hosts(s)]...", rrdhost_hosts_available());
1450
1451 rrd_rdlock();
1452
@@ -1626,19 +1600,3 @@ int rrdhost_set_system_info_variable(struct rrdhost_system_info *system_info, ch
1600
1601 return res;
1602 }
1629 -
1630 -// Added for gap-filling, if this proves to be a bottleneck in large-scale systems then we will need to cache
1631 -// the last entry times as the metric updates, but let's see if it is a problem first.
1632 -time_t rrdhost_last_entry_t(RRDHOST *h) {
1633 - RRDSET *st;
1634 - time_t result = 0;
1635 -
1636 - rrdset_foreach_read(st, h) {
1637 - time_t st_last = rrdset_last_entry_t(st);
1638 -
1639 - if (st_last > result)
1640 - result = st_last;
1641 - }
1642 - rrdset_foreach_done(st);
1643 - return result;
1644 -}
database/rrdlabels.c
+2 -1
@@ -964,7 +964,8 @@ void rrdset_update_rrdlabels(RRDSET *st, DICTIONARY *new_rrdlabels) {
964 if (new_rrdlabels)
965 rrdlabels_migrate_to_these(st->rrdlabels, new_rrdlabels);
966
967 - metaqueue_chart_labels(st);
967 + rrdset_flag_set(st, RRDSET_FLAG_METADATA_UPDATE);
968 + rrdhost_flag_set(st->rrdhost, RRDHOST_FLAG_METADATA_UPDATE);
969 }
970
971
database/rrdset.c
+136 -152
@@ -28,6 +28,8 @@ static inline void rrdset_index_del_name(RRDHOST *host, RRDSET *st) {
28 }
29
30 static inline RRDSET *rrdset_index_find_name(RRDHOST *host, const char *name) {
31 + if (unlikely(!host->rrdset_root_index_name))
32 + return NULL;
33 return dictionary_get(host->rrdset_root_index_name, name);
34 }
35
@@ -132,9 +134,10 @@ static void rrdset_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, v
134 st->rrd_memory_mode = ctr->memory_mode;
135
136 st->chart_type = ctr->chart_type;
135 - st->gap_when_lost_iterations_above = (int) (gap_when_lost_iterations_above + 2);
137 st->rrdhost = host;
138
139 + netdata_spinlock_init(&st->data_collection_lock);
140 +
141 st->flags = RRDSET_FLAG_SYNC_CLOCK
142 | RRDSET_FLAG_INDEXED_ID
143 | RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED
@@ -287,7 +290,7 @@ static bool rrdset_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused,
290 }
291
292 if (unlikely(st->update_every != ctr->update_every)) {
290 - rrdset_set_update_every(st, ctr->update_every);
293 + rrdset_set_update_every_s(st, ctr->update_every);
294 ctr->react_action |= RRDSET_REACT_UPDATED;
295 }
296
@@ -361,7 +364,7 @@ static void rrdset_react_callback(const DICTIONARY_ITEM *item __maybe_unused, vo
364 RRDSET *st = rrdset;
365 RRDHOST *host = st->rrdhost;
366
364 - st->last_accessed_time = now_realtime_sec();
367 + st->last_accessed_time_s = now_realtime_sec();
368
369 if(host->health_enabled && (ctr->react_action & (RRDSET_REACT_NEW | RRDSET_REACT_CHART_ACTIVATED))) {
370 rrdset_flag_set(st, RRDSET_FLAG_PENDING_HEALTH_INITIALIZATION);
@@ -436,6 +439,8 @@ static inline void rrdset_index_del(RRDHOST *host, RRDSET *st) {
439
440 static RRDSET *rrdset_index_find(RRDHOST *host, const char *id) {
441 // TODO - the name index should have an acquired dictionary item, not just a pointer to RRDSET
442 + if (unlikely(!host->rrdset_root_index))
443 + return NULL;
444 return dictionary_get(host->rrdset_root_index, id);
445 }
446
@@ -447,7 +452,7 @@ inline RRDSET *rrdset_find(RRDHOST *host, const char *id) {
452 RRDSET *st = rrdset_index_find(host, id);
453
454 if(st)
450 - st->last_accessed_time = now_realtime_sec();
455 + st->last_accessed_time_s = now_realtime_sec();
456
457 return(st);
458 }
@@ -526,51 +531,103 @@ int rrdset_reset_name(RRDSET *st, const char *name) {
531 }
532
533 // get the timestamp of the last entry in the round-robin database
529 -time_t rrdset_last_entry_t(RRDSET *st) {
534 +time_t rrdset_last_entry_s(RRDSET *st) {
535 RRDDIM *rd;
531 - time_t last_entry_t = 0;
536 + time_t last_entry_s = 0;
537
538 rrddim_foreach_read(rd, st) {
534 - time_t t = rrddim_last_entry_t(rd);
535 - if(t > last_entry_t) last_entry_t = t;
539 + time_t t = rrddim_last_entry_s(rd);
540 + if(t > last_entry_s) last_entry_s = t;
541 }
542 rrddim_foreach_done(rd);
543
539 - return last_entry_t;
544 + return last_entry_s;
545 +}
546 +
547 +time_t rrdset_last_entry_s_of_tier(RRDSET *st, size_t tier) {
548 + RRDDIM *rd;
549 + time_t last_entry_s = 0;
550 +
551 + rrddim_foreach_read(rd, st) {
552 + time_t t = rrddim_last_entry_s_of_tier(rd, tier);
553 + if(t > last_entry_s) last_entry_s = t;
554 + }
555 + rrddim_foreach_done(rd);
556 +
557 + return last_entry_s;
558 }
559
560 // get the timestamp of first entry in the round-robin database
543 -time_t rrdset_first_entry_t(RRDSET *st) {
561 +time_t rrdset_first_entry_s(RRDSET *st) {
562 RRDDIM *rd;
545 - time_t first_entry_t = LONG_MAX;
563 + time_t first_entry_s = LONG_MAX;
564
565 rrddim_foreach_read(rd, st) {
548 - time_t t = rrddim_first_entry_t(rd);
549 - if(t < first_entry_t)
550 - first_entry_t = t;
566 + time_t t = rrddim_first_entry_s(rd);
567 + if(t < first_entry_s)
568 + first_entry_s = t;
569 }
570 rrddim_foreach_done(rd);
571
554 - if (unlikely(LONG_MAX == first_entry_t)) return 0;
555 - return first_entry_t;
572 + if (unlikely(LONG_MAX == first_entry_s)) return 0;
573 + return first_entry_s;
574 }
575
558 -time_t rrdset_first_entry_t_of_tier(RRDSET *st, size_t tier) {
576 +time_t rrdset_first_entry_s_of_tier(RRDSET *st, size_t tier) {
577 if(unlikely(tier > storage_tiers))
578 return 0;
579
580 RRDDIM *rd;
563 - time_t first_entry_t = LONG_MAX;
581 + time_t first_entry_s = LONG_MAX;
582
583 rrddim_foreach_read(rd, st) {
566 - time_t t = rrddim_first_entry_t_of_tier(rd, tier);
567 - if(t && t < first_entry_t)
568 - first_entry_t = t;
584 + time_t t = rrddim_first_entry_s_of_tier(rd, tier);
585 + if(t && t < first_entry_s)
586 + first_entry_s = t;
587 }
588 rrddim_foreach_done(rd);
589
572 - if (unlikely(LONG_MAX == first_entry_t)) return 0;
573 - return first_entry_t;
590 + if (unlikely(LONG_MAX == first_entry_s)) return 0;
591 + return first_entry_s;
592 +}
593 +
594 +void rrdset_get_retention_of_tier_for_collected_chart(RRDSET *st, time_t *first_time_s, time_t *last_time_s, time_t now_s, size_t tier) {
595 + if(!now_s)
596 + now_s = now_realtime_sec();
597 +
598 + time_t db_first_entry_s = rrdset_first_entry_s_of_tier(st, tier);
599 + time_t db_last_entry_s = st->last_updated.tv_sec; // we assume this is a collected RRDSET
600 +
601 + if(unlikely(!db_last_entry_s)) {
602 + db_last_entry_s = rrdset_last_entry_s_of_tier(st, tier);
603 +
604 + if (unlikely(!db_last_entry_s))
605 + // we assume this is a collected RRDSET
606 + db_last_entry_s = now_s;
607 + }
608 +
609 + if(unlikely(db_last_entry_s > now_s)) {
610 + internal_error(true,
611 + "RRDSET: 'host:%s/chart:%s' latest db time %ld is in the future, adjusting it to now %ld",
612 + rrdhost_hostname(st->rrdhost), rrdset_id(st),
613 + db_last_entry_s, now_s);
614 + db_last_entry_s = now_s;
615 + }
616 +
617 + if(unlikely(db_first_entry_s && db_last_entry_s && db_first_entry_s >= db_last_entry_s)) {
618 + internal_error(true,
619 + "RRDSET: 'host:%s/chart:%s' oldest db time %ld is equal or bigger than latest db time %ld, adjusting it last updated time - update every",
620 + rrdhost_hostname(st->rrdhost), rrdset_id(st),
621 + db_first_entry_s, db_last_entry_s);
622 + db_first_entry_s = db_last_entry_s - st->update_every;
623 + }
624 +
625 + if(unlikely(!db_first_entry_s && db_last_entry_s))
626 + // this can be the case on the first data collection of a chart
627 + db_first_entry_s = db_last_entry_s;
628 +
629 + *first_time_s = db_first_entry_s;
630 + *last_time_s = db_last_entry_s;
631 }
632
633 inline void rrdset_is_obsolete(RRDSET *st) {
@@ -583,7 +640,7 @@ inline void rrdset_is_obsolete(RRDSET *st) {
640 rrdset_flag_set(st, RRDSET_FLAG_OBSOLETE);
641 rrdhost_flag_set(st->rrdhost, RRDHOST_FLAG_PENDING_OBSOLETE_CHARTS);
642
586 - st->last_accessed_time = now_realtime_sec();
643 + st->last_accessed_time_s = now_realtime_sec();
644
645 rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
646
@@ -597,7 +654,7 @@ inline void rrdset_is_obsolete(RRDSET *st) {
654 inline void rrdset_isnot_obsolete(RRDSET *st) {
655 if(unlikely((rrdset_flag_check(st, RRDSET_FLAG_OBSOLETE)))) {
656 rrdset_flag_clear(st, RRDSET_FLAG_OBSOLETE);
600 - st->last_accessed_time = now_realtime_sec();
657 + st->last_accessed_time_s = now_realtime_sec();
658
659 rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED);
660
@@ -814,7 +871,7 @@ RRDSET *rrdset_create_custom(
871 , long history_entries
872 ) {
873 if (host != localhost)
817 - host->senders_last_chart_command = now_realtime_sec();
874 + host->child_last_chart_command = now_realtime_sec();
875
876 if(!type || !type[0])
877 fatal("Cannot create rrd stats without a type: id '%s', name '%s', family '%s', context '%s', title '%s', units '%s', plugin '%s', module '%s'."
@@ -925,15 +982,8 @@ void rrdset_timed_next(RRDSET *st, struct timeval now, usec_t duration_since_las
982 );
983 #endif
984
928 - st->last_collected_time.tv_sec = now.tv_sec - st->update_every;
929 - st->last_collected_time.tv_usec = now.tv_usec;
930 - last_collected_time_align(st);
931 -
932 - st->last_updated.tv_sec = now.tv_sec - st->update_every;
933 - st->last_updated.tv_usec = now.tv_usec;
934 - last_updated_time_align(st);
985 + duration_since_last_update = 0;
986
936 - duration_since_last_update = st->update_every * USEC_PER_SEC;
987 #ifdef NETDATA_INTERNAL_CHECKS
988 if(!discard_reason) discard_reason = "COLLECTION TIME IN FUTURE";
989 #endif
@@ -946,6 +996,7 @@ void rrdset_timed_next(RRDSET *st, struct timeval now, usec_t duration_since_las
996 #endif
997
998 duration_since_last_update = (usec_t)since_last_usec;
999 +
1000 #ifdef NETDATA_INTERNAL_CHECKS
1001 if(!discard_reason) discard_reason = "COLLECTION TIME TOO FAR IN THE PAST";
1002 #endif
@@ -954,16 +1005,16 @@ void rrdset_timed_next(RRDSET *st, struct timeval now, usec_t duration_since_las
1005 #ifdef NETDATA_INTERNAL_CHECKS
1006 if(since_last_usec > 0 && (susec_t) duration_since_last_update < since_last_usec) {
1007 static __thread susec_t min_delta = USEC_PER_SEC * 3600, permanent_min_delta = 0;
957 - static __thread time_t last_t = 0;
1008 + static __thread time_t last_time_s = 0;
1009
1010 // the first time initialize it so that it will make the check later
960 - if(last_t == 0) last_t = now.tv_sec + 60;
1011 + if(last_time_s == 0) last_time_s = now.tv_sec + 60;
1012
1013 susec_t delta = since_last_usec - (susec_t) duration_since_last_update;
1014 if(delta < min_delta) min_delta = delta;
1015
965 - if(now.tv_sec >= last_t + 60) {
966 - last_t = now.tv_sec;
1016 + if(now.tv_sec >= last_time_s + 60) {
1017 + last_time_s = now.tv_sec;
1018
1019 if(min_delta > permanent_min_delta) {
1020 info("MINIMUM MICROSECONDS DELTA of thread %d increased from %lld to %lld (+%lld)", gettid(), permanent_min_delta, min_delta, min_delta - permanent_min_delta);
@@ -1034,7 +1085,7 @@ static inline usec_t rrdset_update_last_collected_time(RRDSET *st) {
1085 return last_collect_ut;
1086 }
1087
1037 -static inline usec_t rrdset_init_last_updated_time(RRDSET *st) {
1088 +static inline void rrdset_init_last_updated_time(RRDSET *st) {
1089 // copy the last collected time to last updated time
1090 st->last_updated.tv_sec = st->last_collected_time.tv_sec;
1091 st->last_updated.tv_usec = st->last_collected_time.tv_usec;
@@ -1043,31 +1094,25 @@ static inline usec_t rrdset_init_last_updated_time(RRDSET *st) {
1094 st->last_updated.tv_sec -= st->update_every;
1095
1096 last_updated_time_align(st);
1046 -
1047 - usec_t last_updated_ut = st->last_updated.tv_sec * USEC_PER_SEC + st->last_updated.tv_usec;
1048 -
1049 - rrdset_debug(st, "initialized last updated time to %0.3" NETDATA_DOUBLE_MODIFIER, (NETDATA_DOUBLE)last_updated_ut / USEC_PER_SEC);
1050 -
1051 - return last_updated_ut;
1097 }
1098
1099 static __thread size_t rrdset_done_statistics_points_stored_per_tier[RRD_STORAGE_TIERS];
1100
1056 -static inline time_t tier_next_point_time(RRDDIM *rd, struct rrddim_tier *t, time_t now) {
1101 +static inline time_t tier_next_point_time_s(RRDDIM *rd, struct rrddim_tier *t, time_t now_s) {
1102 time_t loop = (time_t)rd->update_every * (time_t)t->tier_grouping;
1058 - return now + loop - ((now + loop) % loop);
1103 + return now_s + loop - ((now_s + loop) % loop);
1104 }
1105
1106 void store_metric_at_tier(RRDDIM *rd, size_t tier, struct rrddim_tier *t, STORAGE_POINT sp, usec_t now_ut __maybe_unused) {
1062 - if (unlikely(!t->next_point_time))
1063 - t->next_point_time = tier_next_point_time(rd, t, sp.end_time);
1107 + if (unlikely(!t->next_point_time_s))
1108 + t->next_point_time_s = tier_next_point_time_s(rd, t, sp.end_time_s);
1109
1065 - if(unlikely(sp.start_time > t->next_point_time)) {
1110 + if(unlikely(sp.start_time_s > t->next_point_time_s)) {
1111 if (likely(!storage_point_is_unset(t->virtual_point))) {
1112
1113 t->collect_ops->store_metric(
1114 t->db_collection_handle,
1070 - t->next_point_time * USEC_PER_SEC,
1115 + t->next_point_time_s * USEC_PER_SEC,
1116 t->virtual_point.sum,
1117 t->virtual_point.min,
1118 t->virtual_point.max,
@@ -1078,7 +1123,7 @@ void store_metric_at_tier(RRDDIM *rd, size_t tier, struct rrddim_tier *t, STORAG
1123 else {
1124 t->collect_ops->store_metric(
1125 t->db_collection_handle,
1081 - t->next_point_time * USEC_PER_SEC,
1126 + t->next_point_time_s * USEC_PER_SEC,
1127 NAN,
1128 NAN,
1129 NAN,
@@ -1088,15 +1133,15 @@ void store_metric_at_tier(RRDDIM *rd, size_t tier, struct rrddim_tier *t, STORAG
1133
1134 rrdset_done_statistics_points_stored_per_tier[tier]++;
1135 t->virtual_point.count = 0; // make the point unset
1091 - t->next_point_time = tier_next_point_time(rd, t, sp.end_time);
1136 + t->next_point_time_s = tier_next_point_time_s(rd, t, sp.end_time_s);
1137 }
1138
1139 // merge the dates into our virtual point
1095 - if (unlikely(sp.start_time < t->virtual_point.start_time))
1096 - t->virtual_point.start_time = sp.start_time;
1140 + if (unlikely(sp.start_time_s < t->virtual_point.start_time_s))
1141 + t->virtual_point.start_time_s = sp.start_time_s;
1142
1098 - if (likely(sp.end_time > t->virtual_point.end_time))
1099 - t->virtual_point.end_time = sp.end_time;
1143 + if (likely(sp.end_time_s > t->virtual_point.end_time_s))
1144 + t->virtual_point.end_time_s = sp.end_time_s;
1145
1146 // merge the values into our virtual point
1147 if (likely(!storage_point_is_empty(sp))) {
@@ -1151,11 +1196,11 @@ void rrddim_store_metric(RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n,
1196 rd->tiers[0]->collect_ops->store_metric(rd->tiers[0]->db_collection_handle, point_end_time_ut, n, 0, 0, 1, 0, flags);
1197 rrdset_done_statistics_points_stored_per_tier[0]++;
1198
1154 - time_t now = (time_t)(point_end_time_ut / USEC_PER_SEC);
1199 + time_t now_s = (time_t)(point_end_time_ut / USEC_PER_SEC);
1200
1201 STORAGE_POINT sp = {
1157 - .start_time = now - rd->update_every,
1158 - .end_time = now,
1202 + .start_time_s = now_s - rd->update_every,
1203 + .end_time_s = now_s,
1204 .min = n,
1205 .max = n,
1206 .sum = n,
@@ -1172,7 +1217,7 @@ void rrddim_store_metric(RRDDIM *rd, usec_t point_end_time_ut, NETDATA_DOUBLE n,
1217 if(!rrddim_option_check(rd, RRDDIM_OPTION_BACKFILLED_HIGH_TIERS)) {
1218 // we have not collected this tier before
1219 // let's fill any gap that may exist
1175 - rrdr_fill_tier_gap_from_smaller_tiers(rd, tier, now);
1220 + rrdr_fill_tier_gap_from_smaller_tiers(rd, tier, now_s);
1221 rrddim_option_set(rd, RRDDIM_OPTION_BACKFILLED_HIGH_TIERS);
1222 }
1223
@@ -1339,20 +1384,20 @@ static inline size_t rrdset_done_interpolate(
1384 break;
1385 }
1386
1342 - time_t current_time = (time_t) (next_store_ut / USEC_PER_SEC);
1387 + time_t current_time_s = (time_t) (next_store_ut / USEC_PER_SEC);
1388
1389 if(unlikely(!store_this_entry)) {
1345 - (void) ml_is_anomalous(rd, current_time, 0, false);
1390 + (void) ml_is_anomalous(rd, current_time_s, 0, false);
1391
1392 rrddim_store_metric(rd, next_store_ut, NAN, SN_FLAG_NONE);
1393 rrdcontext_collected_rrddim(rd);
1394 continue;
1395 }
1396
1352 - if(likely(rd->updated && rd->collections_counter > 1 && iterations < st->gap_when_lost_iterations_above)) {
1397 + if(likely(rd->updated && rd->collections_counter > 1 && iterations < gap_when_lost_iterations_above)) {
1398 uint32_t dim_storage_flags = storage_flags;
1399
1355 - if (ml_is_anomalous(rd, current_time, new_value, true)) {
1400 + if (ml_is_anomalous(rd, current_time_s, new_value, true)) {
1401 // clear anomaly bit: 0 -> is anomalous, 1 -> not anomalous
1402 dim_storage_flags &= ~((storage_number)SN_FLAG_NOT_ANOMALOUS);
1403 }
@@ -1362,7 +1407,7 @@ static inline size_t rrdset_done_interpolate(
1407 rd->last_stored_value = new_value;
1408 }
1409 else {
1365 - (void) ml_is_anomalous(rd, current_time, 0, false);
1410 + (void) ml_is_anomalous(rd, current_time_s, 0, false);
1411
1412 rrdset_debug(st, "%s: STORE[%ld] = NON EXISTING ", rrddim_name(rd), current_entry);
1413
@@ -1401,36 +1446,6 @@ static inline size_t rrdset_done_interpolate(
1446 return stored_entries;
1447 }
1448
1404 -static inline void rrdset_done_fill_the_gap(RRDSET *st) {
1405 - usec_t update_every_ut = st->update_every * USEC_PER_SEC;
1406 - usec_t now_collect_ut = st->last_collected_time.tv_sec * USEC_PER_SEC + st->last_collected_time.tv_usec;
1407 -
1408 - long c = 0, entries = st->entries;
1409 - RRDDIM *rd;
1410 - rrddim_foreach_read(rd, st) {
1411 - usec_t next_store_ut = (st->last_updated.tv_sec + st->update_every) * USEC_PER_SEC;
1412 - long current_entry = st->current_entry;
1413 -
1414 - for(c = 0; c < entries && next_store_ut <= now_collect_ut ; next_store_ut += update_every_ut, c++) {
1415 - rd->db[current_entry] = pack_storage_number(NAN, SN_FLAG_NONE);
1416 - current_entry = ((current_entry + 1) >= entries) ? 0 : current_entry + 1;
1417 -
1418 - rrdset_debug(st, "%s: STORE[%ld] = NON EXISTING (FILLED THE GAP)", rrddim_name(rd), current_entry);
1419 - }
1420 - }
1421 - rrddim_foreach_done(rd);
1422 -
1423 - if(c > 0) {
1424 - c--;
1425 - st->last_updated.tv_sec += c * st->update_every;
1426 -
1427 - st->current_entry += c;
1428 - st->counter += c;
1429 - if(st->current_entry >= st->entries)
1430 - st->current_entry -= st->entries;
1431 - }
1432 -}
1433 -
1449 void rrdset_done(RRDSET *st) {
1450 struct timeval now;
1451
@@ -1439,10 +1454,12 @@ void rrdset_done(RRDSET *st) {
1454 }
1455
1456 void rrdset_timed_done(RRDSET *st, struct timeval now, bool pending_rrdset_next) {
1442 - if(unlikely(netdata_exit)) return;
1457 + if(unlikely(!service_running(SERVICE_COLLECTORS))) return;
1458 +
1459 + netdata_spinlock_lock(&st->data_collection_lock);
1460
1461 if (pending_rrdset_next)
1445 - rrdset_next(st);
1462 + rrdset_timed_next(st, now, 0ULL);
1463
1464 debug(D_RRD_CALLS, "rrdset_done() for chart '%s'", rrdset_name(st));
1465
@@ -1531,29 +1548,6 @@ void rrdset_timed_done(RRDSET *st, struct timeval now, bool pending_rrdset_next)
1548 first_entry = 1;
1549 }
1550
1534 -#ifdef ENABLE_DBENGINE
1535 - // check if we will re-write the entire page
1536 - if(unlikely(st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE &&
1537 - dt_usec(&st->last_collected_time, &st->last_updated) > (RRDENG_BLOCK_SIZE / sizeof(storage_number)) * update_every_ut)) {
1538 - info(
1539 - "'%s': too old data (last updated at %" PRId64 ".%" PRId64 ", last collected at %" PRId64 ".%" PRId64 "). "
1540 - "Resetting it. Will not store the next entry.",
1541 - rrdset_id(st),
1542 - (int64_t)st->last_updated.tv_sec,
1543 - (int64_t)st->last_updated.tv_usec,
1544 - (int64_t)st->last_collected_time.tv_sec,
1545 - (int64_t)st->last_collected_time.tv_usec);
1546 - rrdset_reset(st);
1547 - rrdset_init_last_updated_time(st);
1548 -
1549 - st->usec_since_last_update = update_every_ut;
1550 -
1551 - // the first entry should not be stored
1552 - store_this_entry = 0;
1553 - first_entry = 1;
1554 - }
1555 -#endif
1556 -
1551 // these are the 3 variables that will help us in interpolation
1552 // last_stored_ut = the last time we added a value to the storage
1553 // now_collect_ut = the time the current value has been collected
@@ -1563,21 +1557,11 @@ void rrdset_timed_done(RRDSET *st, struct timeval now, bool pending_rrdset_next)
1557 next_store_ut = (st->last_updated.tv_sec + st->update_every) * USEC_PER_SEC;
1558
1559 if(unlikely(!st->counter_done)) {
1566 - // if we have not collected metrics this session (st->counter_done == 0)
1567 - // and we have collected metrics for this chart in the past (st->counter != 0)
1568 - // fill the gap (the chart has been just loaded from disk)
1569 - if(unlikely(st->counter) && st->rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE) {
1570 - // TODO this should be inside the storage engine
1571 - rrdset_done_fill_the_gap(st);
1572 - last_stored_ut = st->last_updated.tv_sec * USEC_PER_SEC + st->last_updated.tv_usec;
1573 - next_store_ut = (st->last_updated.tv_sec + st->update_every) * USEC_PER_SEC;
1574 - }
1575 - if (st->rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE) {
1576 - // set a fake last_updated to jump to current time
1577 - rrdset_init_last_updated_time(st);
1578 - last_stored_ut = st->last_updated.tv_sec * USEC_PER_SEC + st->last_updated.tv_usec;
1579 - next_store_ut = (st->last_updated.tv_sec + st->update_every) * USEC_PER_SEC;
1580 - }
1560 + // set a fake last_updated to jump to current time
1561 + rrdset_init_last_updated_time(st);
1562 +
1563 + last_stored_ut = st->last_updated.tv_sec * USEC_PER_SEC + st->last_updated.tv_usec;
1564 + next_store_ut = (st->last_updated.tv_sec + st->update_every) * USEC_PER_SEC;
1565
1566 if(unlikely(rrdset_flag_check(st, RRDSET_FLAG_STORE_FIRST))) {
1567 store_this_entry = 1;
@@ -1927,6 +1911,8 @@ after_second_database_work:
1911 );
1912 }
1913
1914 + netdata_spinlock_unlock(&st->data_collection_lock);
1915 +
1916 // ALL DONE ABOUT THE DATA UPDATE
1917 // --------------------------------------------------------------------
1918
@@ -1958,13 +1944,13 @@ after_second_database_work:
1944 store_metric_collection_completed();
1945 }
1946
1961 -time_t rrdset_set_update_every(RRDSET *st, time_t update_every) {
1947 +time_t rrdset_set_update_every_s(RRDSET *st, time_t update_every_s) {
1948
1949 internal_error(true, "RRDSET '%s' switching update every from %d to %d",
1964 - rrdset_id(st), (int)st->update_every, (int)update_every);
1950 + rrdset_id(st), (int)st->update_every, (int)update_every_s);
1951
1966 - time_t prev_update_every = st->update_every;
1967 - st->update_every = update_every;
1952 + time_t prev_update_every_s = st->update_every;
1953 + st->update_every = update_every_s;
1954
1955 // switch update every to the storage engine
1956 RRDDIM *rd;
@@ -1974,13 +1960,13 @@ time_t rrdset_set_update_every(RRDSET *st, time_t update_every) {
1960 rd->tiers[tier]->collect_ops->change_collection_frequency(rd->tiers[tier]->db_collection_handle, (int)(st->rrdhost->db[tier].tier_grouping * st->update_every));
1961 }
1962
1977 - assert(rd->update_every == prev_update_every &&
1963 + assert(rd->update_every == prev_update_every_s &&
1964 "chart's update every differs from the update every of its dimensions");
1965 rd->update_every = st->update_every;
1966 }
1967 rrddim_foreach_done(rd);
1968
1983 - return prev_update_every;
1969 + return prev_update_every_s;
1970 }
1971
1972 // ----------------------------------------------------------------------------
@@ -2028,8 +2014,8 @@ struct rrdset_map_save_v019 {
2014 size_t counter; // NEEDS TO BE UPDATED - maintained on load
2015 size_t counter_done; // ignored
2016 union { //
2031 - time_t last_accessed_time; // ignored
2032 - time_t last_entry_t; // ignored
2017 + time_t last_accessed_time_s; // ignored
2018 + time_t last_entry_s; // ignored
2019 }; //
2020 time_t upstream_resync_time; // ignored
2021 void *plugin_name; // ignored
@@ -2109,13 +2095,11 @@ bool rrdset_memory_load_or_create_map_save(RRDSET *st, RRD_MEMORY_MODE memory_mo
2095
2096 unsigned long size = sizeof(struct rrdset_map_save_v019);
2097 struct rrdset_map_save_v019 *st_on_file = (struct rrdset_map_save_v019 *)netdata_mmap(
2112 - fullfilename, size,
2113 - ((memory_mode == RRD_MEMORY_MODE_MAP) ? MAP_SHARED : MAP_PRIVATE),
2114 - 0);
2098 + fullfilename, size, ((memory_mode == RRD_MEMORY_MODE_MAP) ? MAP_SHARED : MAP_PRIVATE), 0, false);
2099
2100 if(!st_on_file) return false;
2101
2118 - time_t now = now_realtime_sec();
2102 + time_t now_s = now_realtime_sec();
2103
2104 st_on_file->magic[sizeof(RRDSET_MAGIC_V019)] = '\0';
2105 if(strcmp(st_on_file->magic, RRDSET_MAGIC_V019) != 0) {
@@ -2134,13 +2118,13 @@ bool rrdset_memory_load_or_create_map_save(RRDSET *st, RRD_MEMORY_MODE memory_mo
2118 error("File '%s' does not have the desired granularity. Clearing it.", fullfilename);
2119 memset(st_on_file, 0, size);
2120 }
2137 - else if((now - st_on_file->last_updated.tv_sec) > st->update_every * st->entries) {
2121 + else if((now_s - st_on_file->last_updated.tv_sec) > st->update_every * st->entries) {
2122 info("File '%s' is too old. Clearing it.", fullfilename);
2123 memset(st_on_file, 0, size);
2124 }
2141 - else if(st_on_file->last_updated.tv_sec > now + st->update_every) {
2142 - error("File '%s' refers to the future by %zd secs. Resetting it to now.", fullfilename, (ssize_t)(st_on_file->last_updated.tv_sec - now));
2143 - st_on_file->last_updated.tv_sec = now;
2125 + else if(st_on_file->last_updated.tv_sec > now_s + st->update_every) {
2126 + error("File '%s' refers to the future by %zd secs. Resetting it to now.", fullfilename, (ssize_t)(st_on_file->last_updated.tv_sec - now_s));
2127 + st_on_file->last_updated.tv_sec = now_s;
2128 }
2129
2130 if(st_on_file->current_entry >= st_on_file->entries)
database/sqlite/sqlite_aclk.c
+150 -181
@@ -10,10 +10,140 @@ void sanity_check(void) {
10 BUILD_BUG_ON(WORKER_UTILIZATION_MAX_JOB_TYPES < ACLK_MAX_ENUMERATIONS_DEFINED);
11 }
12
13 -const char *aclk_sync_config[] = {
13 +static int sql_check_aclk_table(void *data, int argc, char **argv, char **column)
14 +{
15 + struct aclk_database_worker_config *wc = data;
16 + UNUSED(argc);
17 + UNUSED(column);
18
15 - NULL,
16 -};
19 + debug(D_ACLK_SYNC,"Scheduling aclk sync table check for node %s", (char *) argv[0]);
20 + struct aclk_database_cmd cmd;
21 + memset(&cmd, 0, sizeof(cmd));
22 + cmd.opcode = ACLK_DATABASE_DELETE_HOST;
23 + cmd.data = strdupz((char *) argv[0]);
24 + aclk_database_enq_cmd_noblock(wc, &cmd);
25 + return 0;
26 +}
27 +
28 +#define SQL_SELECT_ACLK_ACTIVE_LIST "SELECT REPLACE(SUBSTR(name,19),'_','-') FROM sqlite_schema " \
29 + "WHERE name LIKE 'aclk_chart_latest_%' AND type IN ('table');"
30 +
31 +static void sql_check_aclk_table_list(struct aclk_database_worker_config *wc)
32 +{
33 + char *err_msg = NULL;
34 + debug(D_ACLK_SYNC,"Cleaning tables for nodes that do not exist");
35 + int rc = sqlite3_exec_monitored(db_meta, SQL_SELECT_ACLK_ACTIVE_LIST, sql_check_aclk_table, (void *) wc, &err_msg);
36 + if (rc != SQLITE_OK) {
37 + error_report("Query failed when trying to check for obsolete ACLK sync tables, %s", err_msg);
38 + sqlite3_free(err_msg);
39 + }
40 +}
41 +
42 +static void sql_maint_aclk_sync_database(struct aclk_database_worker_config *wc, struct aclk_database_cmd cmd)
43 +{
44 + UNUSED(cmd);
45 +
46 + debug(D_ACLK, "Checking database for %s", wc->host_guid);
47 +
48 + BUFFER *sql = buffer_create(ACLK_SYNC_QUERY_SIZE);
49 +
50 + buffer_sprintf(sql,"DELETE FROM aclk_alert_%s WHERE date_submitted IS NOT NULL AND "
51 + "CAST(date_cloud_ack AS INT) < unixepoch()-%d;", wc->uuid_str, ACLK_DELETE_ACK_ALERTS_INTERNAL);
52 + db_execute(buffer_tostring(sql));
53 +
54 + buffer_free(sql);
55 +}
56 +
57 +
58 +#define SQL_SELECT_HOST_BY_UUID "SELECT host_id FROM host WHERE host_id = @host_id;"
59 +
60 +static int is_host_available(uuid_t *host_id)
61 +{
62 + sqlite3_stmt *res = NULL;
63 + int rc;
64 +
65 + if (unlikely(!db_meta)) {
66 + if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
67 + error_report("Database has not been initialized");
68 + return 1;
69 + }
70 +
71 + rc = sqlite3_prepare_v2(db_meta, SQL_SELECT_HOST_BY_UUID, -1, &res, 0);
72 + if (unlikely(rc != SQLITE_OK)) {
73 + error_report("Failed to prepare statement to select node instance information for a node");
74 + return 1;
75 + }
76 +
77 + rc = sqlite3_bind_blob(res, 1, host_id, sizeof(*host_id), SQLITE_STATIC);
78 + if (unlikely(rc != SQLITE_OK)) {
79 + error_report("Failed to bind host_id parameter to select node instance information");
80 + goto failed;
81 + }
82 + rc = sqlite3_step_monitored(res);
83 +
84 +failed:
85 + if (unlikely(sqlite3_finalize(res) != SQLITE_OK))
86 + error_report("Failed to finalize the prepared statement when checking host existence");
87 +
88 + return (rc == SQLITE_ROW);
89 +}
90 +
91 +// OPCODE: ACLK_DATABASE_DELETE_HOST
92 +void sql_delete_aclk_table_list(struct aclk_database_worker_config *wc, struct aclk_database_cmd cmd)
93 +{
94 + UNUSED(wc);
95 + char uuid_str[GUID_LEN + 1];
96 + char host_str[GUID_LEN + 1];
97 +
98 + int rc;
99 + uuid_t host_uuid;
100 + char *host_guid = (char *)cmd.data;
101 +
102 + if (unlikely(!host_guid))
103 + return;
104 +
105 + rc = uuid_parse(host_guid, host_uuid);
106 + freez(host_guid);
107 + if (rc)
108 + return;
109 +
110 + uuid_unparse_lower(host_uuid, host_str);
111 + uuid_unparse_lower_fix(&host_uuid, uuid_str);
112 +
113 + debug(D_ACLK_SYNC, "Checking if I should delete aclk tables for node %s", host_str);
114 +
115 + if (is_host_available(&host_uuid)) {
116 + debug(D_ACLK_SYNC, "Host %s exists, not deleting aclk sync tables", host_str);
117 + return;
118 + }
119 +
120 + debug(D_ACLK_SYNC, "Host %s does NOT exist, can delete aclk sync tables", host_str);
121 +
122 + sqlite3_stmt *res = NULL;
123 + BUFFER *sql = buffer_create(ACLK_SYNC_QUERY_SIZE);
124 +
125 + buffer_sprintf(sql,"SELECT 'drop '||type||' IF EXISTS '||name||';' FROM sqlite_schema " \
126 + "WHERE name LIKE 'aclk_%%_%s' AND type IN ('table', 'trigger', 'index');", uuid_str);
127 +
128 + rc = sqlite3_prepare_v2(db_meta, buffer_tostring(sql), -1, &res, 0);
129 + if (rc != SQLITE_OK) {
130 + error_report("Failed to prepare statement to clean up aclk tables");
131 + goto fail;
132 + }
133 + buffer_flush(sql);
134 +
135 + while (sqlite3_step_monitored(res) == SQLITE_ROW)
136 + buffer_strcat(sql, (char *) sqlite3_column_text(res, 0));
137 +
138 + rc = sqlite3_finalize(res);
139 + if (unlikely(rc != SQLITE_OK))
140 + error_report("Failed to finalize statement to clean up aclk tables, rc = %d", rc);
141 +
142 + db_execute(buffer_tostring(sql));
143 +
144 +fail:
145 + buffer_free(sql);
146 +}
147
148 uv_mutex_t aclk_async_lock;
149 struct aclk_database_worker_config *aclk_thread_head = NULL;
@@ -38,7 +168,6 @@ void aclk_add_worker_thread(struct aclk_database_worker_config *wc)
168 aclk_thread_head = wc;
169 }
170 uv_mutex_unlock(&aclk_async_lock);
41 - return;
171 }
172
173 void aclk_del_worker_thread(struct aclk_database_worker_config *wc)
@@ -53,7 +182,6 @@ void aclk_del_worker_thread(struct aclk_database_worker_config *wc)
182 if (*tmp)
183 *tmp = wc->next;
184 uv_mutex_unlock(&aclk_async_lock);
56 - return;
185 }
186
187 int aclk_worker_thread_exists(char *guid)
@@ -199,7 +327,6 @@ void aclk_sync_exit_all()
327 uv_mutex_unlock(&aclk_async_lock);
328 }
329
202 -#ifdef ENABLE_ACLK
330 enum {
331 IDX_HOST_ID,
332 IDX_HOSTNAME,
@@ -268,9 +395,9 @@ static int create_host_callback(void *data, int argc, char **argv, char **column
395 #endif
396 return 0;
397 }
271 -#endif
398
273 -int aclk_start_sync_thread(void *data, int argc, char **argv, char **column)
399 +#ifdef ENABLE_ACLK
400 +static int aclk_start_sync_thread(void *data, int argc, char **argv, char **column)
401 {
402 char uuid_str[GUID_LEN + 1];
403 UNUSED(data);
@@ -286,10 +413,9 @@ int aclk_start_sync_thread(void *data, int argc, char **argv, char **column)
413 sql_create_aclk_table(host, (uuid_t *) argv[0], (uuid_t *) argv[1]);
414 return 0;
415 }
289 -
416 +#endif
417 void sql_aclk_sync_init(void)
418 {
292 -#ifdef ENABLE_ACLK
419 char *err_msg = NULL;
420 int rc;
421
@@ -301,21 +427,7 @@ void sql_aclk_sync_init(void)
427 return;
428 }
429
304 - info("SQLite aclk sync initialization");
305 -
306 - for (int i = 0; aclk_sync_config[i]; i++) {
307 - debug(D_ACLK_SYNC, "Executing %s", aclk_sync_config[i]);
308 - rc = sqlite3_exec_monitored(db_meta, aclk_sync_config[i], 0, 0, &err_msg);
309 - if (rc != SQLITE_OK) {
310 - error_report("SQLite error aclk sync initialization setup, rc = %d (%s)", rc, err_msg);
311 - error_report("SQLite failed statement %s", aclk_sync_config[i]);
312 - sqlite3_free(err_msg);
313 - return;
314 - }
315 - }
316 - info("SQLite aclk sync initialization completed");
317 - fatal_assert(0 == uv_mutex_init(&aclk_async_lock));
318 -
430 + info("Creating archived hosts");
431 rc = sqlite3_exec_monitored(db_meta, "SELECT host_id, hostname, registry_hostname, update_every, os, "
432 "timezone, tags, hops, memory_mode, abbrev_timezone, utc_offset, program_name, "
433 "program_version, entries, health_enabled FROM host WHERE hops >0;",
@@ -325,14 +437,16 @@ void sql_aclk_sync_init(void)
437 sqlite3_free(err_msg);
438 }
439
440 +#ifdef ENABLE_ACLK
441 + fatal_assert(0 == uv_mutex_init(&aclk_async_lock));
442 rc = sqlite3_exec_monitored(db_meta, "SELECT ni.host_id, ni.node_id FROM host h, node_instance ni WHERE "
443 "h.host_id = ni.host_id AND ni.node_id IS NOT NULL;", aclk_start_sync_thread, NULL, &err_msg);
444 if (rc != SQLITE_OK) {
445 error_report("SQLite error when starting ACLK sync threads, rc = %d (%s)", rc, err_msg);
446 sqlite3_free(err_msg);
447 }
448 + info("ACLK sync initialization completed");
449 #endif
335 - return;
450 }
451
452 static void async_cb(uv_async_t *handle)
@@ -374,9 +488,7 @@ static void timer_cb(uv_timer_t* handle)
488 #endif
489 }
490
377 -#define MAX_CMD_BATCH_SIZE (256)
378 -
379 -void aclk_database_worker(void *arg)
491 +static void aclk_database_worker(void *arg)
492 {
493 worker_register("ACLKSYNC");
494 worker_register_job_name(ACLK_DATABASE_NOOP, "noop");
@@ -398,15 +510,12 @@ void aclk_database_worker(void *arg)
510 enum aclk_database_opcode opcode;
511 uv_timer_t timer_req;
512 struct aclk_database_cmd cmd;
401 - unsigned cmd_batch_size;
402 -
403 - //aclk_database_init_cmd_queue(wc);
513
514 char threadname[NETDATA_THREAD_NAME_MAX+1];
515 if (wc->host)
407 - snprintfz(threadname, NETDATA_THREAD_NAME_MAX, "AS_%s", rrdhost_hostname(wc->host));
516 + snprintfz(threadname, NETDATA_THREAD_NAME_MAX, "ACLK[%s]", rrdhost_hostname(wc->host));
517 else {
409 - snprintfz(threadname, NETDATA_THREAD_NAME_MAX, "AS_%s", wc->uuid_str);
518 + snprintfz(threadname, NETDATA_THREAD_NAME_MAX, "ACLK[%s]", wc->uuid_str);
519 threadname[11] = '\0';
520 }
521 uv_thread_set_name_np(wc->thread, threadname);
@@ -449,17 +558,13 @@ void aclk_database_worker(void *arg)
558 uv_run(loop, UV_RUN_DEFAULT);
559
560 /* wait for commands */
452 - cmd_batch_size = 0;
561 do {
454 - if (unlikely(cmd_batch_size >= MAX_CMD_BATCH_SIZE))
455 - break;
562 cmd = aclk_database_deq_cmd(wc);
563
564 if (netdata_exit)
565 break;
566
567 opcode = cmd.opcode;
462 - ++cmd_batch_size;
568
569 if(likely(opcode != ACLK_DATABASE_NOOP))
570 worker_is_busy(opcode);
@@ -535,7 +640,7 @@ void aclk_database_worker(void *arg)
640 wc->host = rrdhost_find_by_guid(wc->host_guid);
641 if (wc->host) {
642 info("HOST %s (%s) detected as active", rrdhost_hostname(wc->host), wc->host_guid);
538 - snprintfz(threadname, NETDATA_THREAD_NAME_MAX, "AS_%s", rrdhost_hostname(wc->host));
643 + snprintfz(threadname, NETDATA_THREAD_NAME_MAX, "ACLK[%s]", rrdhost_hostname(wc->host));
644 uv_thread_set_name_np(wc->thread, threadname);
645 wc->host->dbsync_worker = wc;
646 if (unlikely(!wc->hostname))
@@ -584,10 +689,8 @@ void aclk_database_worker(void *arg)
689 info("Shutting down ACLK sync event loop complete for host %s", wc->host_guid);
690 /* TODO: don't let the API block by waiting to enqueue commands */
691 uv_cond_destroy(&wc->cmd_cond);
587 -/* uv_mutex_destroy(&wc->cmd_mutex); */
588 - //fatal_assert(0 == uv_loop_close(loop));
589 - int rc;
692
693 + int rc;
694 do {
695 rc = uv_loop_close(loop);
696 } while (rc != UV_EBUSY);
@@ -648,6 +751,10 @@ void sql_create_aclk_table(RRDHOST *host, uuid_t *host_uuid, uuid_t *node_id)
751 if (likely(host)) {
752 host->dbsync_worker = (void *)wc;
753 wc->hostname = strdupz(rrdhost_hostname(host));
754 + if (node_id && !host->node_id) {
755 + host->node_id = mallocz(sizeof(*host->node_id));
756 + uuid_copy(*host->node_id, *node_id);
757 + }
758 }
759 else
760 wc->hostname = get_hostname_by_node_id(wc->node_id);
@@ -663,142 +770,4 @@ void sql_create_aclk_table(RRDHOST *host, uuid_t *host_uuid, uuid_t *node_id)
770 UNUSED(host_uuid);
771 UNUSED(node_id);
772 #endif
666 - return;
667 -}
668 -
669 -void sql_maint_aclk_sync_database(struct aclk_database_worker_config *wc, struct aclk_database_cmd cmd)
670 -{
671 - UNUSED(cmd);
672 -
673 - debug(D_ACLK, "Checking database for %s", wc->host_guid);
674 -
675 - BUFFER *sql = buffer_create(ACLK_SYNC_QUERY_SIZE);
676 -
677 - buffer_sprintf(sql,"DELETE FROM aclk_alert_%s WHERE date_submitted IS NOT NULL AND "
678 - "CAST(date_cloud_ack AS INT) < unixepoch()-%d;", wc->uuid_str, ACLK_DELETE_ACK_ALERTS_INTERNAL);
679 - db_execute(buffer_tostring(sql));
680 -
681 - buffer_free(sql);
682 - return;
683 -}
684 -
685 -#define SQL_SELECT_HOST_BY_UUID "SELECT host_id FROM host WHERE host_id = @host_id;"
686 -
687 -static int is_host_available(uuid_t *host_id)
688 -{
689 - sqlite3_stmt *res = NULL;
690 - int rc;
691 -
692 - if (unlikely(!db_meta)) {
693 - if (default_rrd_memory_mode == RRD_MEMORY_MODE_DBENGINE)
694 - error_report("Database has not been initialized");
695 - return 1;
696 - }
697 -
698 - rc = sqlite3_prepare_v2(db_meta, SQL_SELECT_HOST_BY_UUID, -1, &res, 0);
699 - if (unlikely(rc != SQLITE_OK)) {
700 - error_report("Failed to prepare statement to select node instance information for a node");
701 - return 1;
702 - }
703 -
704 - rc = sqlite3_bind_blob(res, 1, host_id, sizeof(*host_id), SQLITE_STATIC);
705 - if (unlikely(rc != SQLITE_OK)) {
706 - error_report("Failed to bind host_id parameter to select node instance information");
707 - goto failed;
708 - }
709 - rc = sqlite3_step_monitored(res);
710 -
711 - failed:
712 - if (unlikely(sqlite3_finalize(res) != SQLITE_OK))
713 - error_report("Failed to finalize the prepared statement when checking host existence");
714 -
715 - return (rc == SQLITE_ROW);
716 -}
717 -
718 -// OPCODE: ACLK_DATABASE_DELETE_HOST
719 -void sql_delete_aclk_table_list(struct aclk_database_worker_config *wc, struct aclk_database_cmd cmd)
720 -{
721 - UNUSED(wc);
722 - char uuid_str[GUID_LEN + 1];
723 - char host_str[GUID_LEN + 1];
724 -
725 - int rc;
726 - uuid_t host_uuid;
727 - char *host_guid = (char *)cmd.data;
728 -
729 - if (unlikely(!host_guid))
730 - return;
731 -
732 - rc = uuid_parse(host_guid, host_uuid);
733 - freez(host_guid);
734 - if (rc)
735 - return;
736 -
737 - uuid_unparse_lower(host_uuid, host_str);
738 - uuid_unparse_lower_fix(&host_uuid, uuid_str);
739 -
740 - debug(D_ACLK_SYNC, "Checking if I should delete aclk tables for node %s", host_str);
741 -
742 - if (is_host_available(&host_uuid)) {
743 - debug(D_ACLK_SYNC, "Host %s exists, not deleting aclk sync tables", host_str);
744 - return;
745 - }
746 -
747 - debug(D_ACLK_SYNC, "Host %s does NOT exist, can delete aclk sync tables", host_str);
748 -
749 - sqlite3_stmt *res = NULL;
750 - BUFFER *sql = buffer_create(ACLK_SYNC_QUERY_SIZE);
751 -
752 - buffer_sprintf(sql,"SELECT 'drop '||type||' IF EXISTS '||name||';' FROM sqlite_schema " \
753 - "WHERE name LIKE 'aclk_%%_%s' AND type IN ('table', 'trigger', 'index');", uuid_str);
754 -
755 - rc = sqlite3_prepare_v2(db_meta, buffer_tostring(sql), -1, &res, 0);
756 - if (rc != SQLITE_OK) {
757 - error_report("Failed to prepare statement to clean up aclk tables");
758 - goto fail;
759 - }
760 - buffer_flush(sql);
761 -
762 - while (sqlite3_step_monitored(res) == SQLITE_ROW)
763 - buffer_strcat(sql, (char *) sqlite3_column_text(res, 0));
764 -
765 - rc = sqlite3_finalize(res);
766 - if (unlikely(rc != SQLITE_OK))
767 - error_report("Failed to finalize statement to clean up aclk tables, rc = %d", rc);
768 -
769 - db_execute(buffer_tostring(sql));
770 -
771 -fail:
772 - buffer_free(sql);
773 - return;
774 -}
775 -
776 -static int sql_check_aclk_table(void *data, int argc, char **argv, char **column)
777 -{
778 - struct aclk_database_worker_config *wc = data;
779 - UNUSED(argc);
780 - UNUSED(column);
781 -
782 - debug(D_ACLK_SYNC,"Scheduling aclk sync table check for node %s", (char *) argv[0]);
783 - struct aclk_database_cmd cmd;
784 - memset(&cmd, 0, sizeof(cmd));
785 - cmd.opcode = ACLK_DATABASE_DELETE_HOST;
786 - cmd.data = strdupz((char *) argv[0]);
787 - aclk_database_enq_cmd_noblock(wc, &cmd);
788 - return 0;
789 -}
790 -
791 -#define SQL_SELECT_ACLK_ACTIVE_LIST "SELECT REPLACE(SUBSTR(name,19),'_','-') FROM sqlite_schema " \
792 - "WHERE name LIKE 'aclk_chart_latest_%' AND type IN ('table');"
793 -
794 -void sql_check_aclk_table_list(struct aclk_database_worker_config *wc)
795 -{
796 - char *err_msg = NULL;
797 - debug(D_ACLK_SYNC,"Cleaning tables for nodes that do not exist");
798 - int rc = sqlite3_exec_monitored(db_meta, SQL_SELECT_ACLK_ACTIVE_LIST, sql_check_aclk_table, (void *) wc, &err_msg);
799 - if (rc != SQLITE_OK) {
800 - error_report("Query failed when trying to check for obsolete ACLK sync tables, %s", err_msg);
801 - sqlite3_free(err_msg);
802 - }
803 - return;
804 -}
773 +}
\ No newline at end of file
database/sqlite/sqlite_aclk.h
+1 -4
@@ -99,7 +99,7 @@ struct aclk_database_cmd {
99 struct aclk_completion *completion;
100 };
101
102 -#define ACLK_DATABASE_CMD_Q_MAX_SIZE (16384)
102 +#define ACLK_DATABASE_CMD_Q_MAX_SIZE (1024)
103
104 struct aclk_database_cmdqueue {
105 unsigned head, tail;
@@ -166,9 +166,6 @@ int aclk_database_enq_cmd_noblock(struct aclk_database_worker_config *wc, struct
166 void aclk_database_enq_cmd(struct aclk_database_worker_config *wc, struct aclk_database_cmd *cmd);
167 void sql_create_aclk_table(RRDHOST *host, uuid_t *host_uuid, uuid_t *node_id);
168 void sql_aclk_sync_init(void);
169 -void sql_check_aclk_table_list(struct aclk_database_worker_config *wc);
170 -void sql_delete_aclk_table_list(struct aclk_database_worker_config *wc, struct aclk_database_cmd cmd);
171 -void sql_maint_aclk_sync_database(struct aclk_database_worker_config *wc, struct aclk_database_cmd cmd);
169 int claimed();
170 void aclk_sync_exit_all();
171 struct aclk_database_worker_config *find_inactive_wc_by_node_id(char *node_id);
database/sqlite/sqlite_aclk_alert.c
+3
@@ -133,6 +133,9 @@ done:
133 // and handle both cases
134 int sql_queue_alarm_to_aclk(RRDHOST *host, ALARM_ENTRY *ae, int skip_filter)
135 {
136 + if(!service_running(SERVICE_ACLK))
137 + return 0;
138 +
139 if (!claimed())
140 return 0;
141
database/sqlite/sqlite_context.c
+12 -12
@@ -283,8 +283,8 @@ void ctx_get_context_list(uuid_t *host_uuid, void (*dict_cb)(VERSIONED_CONTEXT_D
283 context_data.chart_type = (char *) sqlite3_column_text(res, 3);
284 context_data.units = (char *) sqlite3_column_text(res, 4);
285 context_data.priority = sqlite3_column_int64(res, 5);
286 - context_data.first_time_t = sqlite3_column_int64(res, 6);
287 - context_data.last_time_t = sqlite3_column_int64(res, 7);
286 + context_data.first_time_s = sqlite3_column_int64(res, 6);
287 + context_data.last_time_s = sqlite3_column_int64(res, 7);
288 context_data.deleted = sqlite3_column_int(res, 8);
289 context_data.family = (char *) sqlite3_column_text(res, 9);
290 dict_cb(&context_data, data);
@@ -360,13 +360,13 @@ int ctx_store_context(uuid_t *host_uuid, VERSIONED_CONTEXT_DATA *context_data)
360 goto skip_store;
361 }
362
363 - rc = sqlite3_bind_int64(res, 8, (time_t) context_data->first_time_t);
363 + rc = sqlite3_bind_int64(res, 8, (time_t) context_data->first_time_s);
364 if (unlikely(rc != SQLITE_OK)) {
365 error_report("Failed to bind first_time_t to store context details");
366 goto skip_store;
367 }
368
369 - rc = sqlite3_bind_int64(res, 9, (time_t) context_data->last_time_t);
369 + rc = sqlite3_bind_int64(res, 9, (time_t) context_data->last_time_s);
370 if (unlikely(rc != SQLITE_OK)) {
371 error_report("Failed to bind last_time_t to store context details");
372 goto skip_store;
@@ -478,8 +478,8 @@ static void dict_ctx_get_context_list_cb(VERSIONED_CONTEXT_DATA *context_data, v
478 context_data->chart_type,
479 context_data->units,
480 context_data->priority,
481 - context_data->first_time_t,
482 - context_data->last_time_t,
481 + context_data->first_time_s,
482 + context_data->last_time_s,
483 context_data->deleted,
484 context_data->family);
485 }
@@ -504,8 +504,8 @@ int ctx_unittest(void)
504 context_data.family = strdupz("TestContextFamily");
505 context_data.priority = 50000;
506 context_data.deleted = 0;
507 - context_data.first_time_t = 1657781000;
508 - context_data.last_time_t = 1657781100;
507 + context_data.first_time_s = 1657781000;
508 + context_data.last_time_s = 1657781100;
509 context_data.version = now_realtime_usec();
510
511 if (likely(!ctx_store_context(&host_uuid, &context_data)))
@@ -519,8 +519,8 @@ int ctx_unittest(void)
519 info("Entry %s not inserted", context_data.id);
520
521 // This will change end time
522 - context_data.first_time_t = 1657781000;
523 - context_data.last_time_t = 1657782001;
522 + context_data.first_time_s = 1657781000;
523 + context_data.last_time_s = 1657782001;
524 if (likely(!ctx_update_context(&host_uuid, &context_data)))
525 info("Entry %s updated", context_data.id);
526 else
@@ -530,8 +530,8 @@ int ctx_unittest(void)
530 info("List context end after insert");
531
532 // This will change start time
533 - context_data.first_time_t = 1657782000;
534 - context_data.last_time_t = 1657782001;
533 + context_data.first_time_s = 1657782000;
534 + context_data.last_time_s = 1657782001;
535 if (likely(!ctx_update_context(&host_uuid, &context_data)))
536 info("Entry %s updated", context_data.id);
537 else
database/sqlite/sqlite_context.h
+2 -2
@@ -45,8 +45,8 @@ typedef struct versioned_context_data {
45
46 uint64_t priority; // the chart priority of the context
47
48 - uint64_t first_time_t; // the first entry in the database, in seconds
49 - uint64_t last_time_t; // the last point in the database, in seconds
48 + uint64_t first_time_s; // the first entry in the database, in seconds
49 + uint64_t last_time_s; // the last point in the database, in seconds
50
51 bool deleted; // true when this is deleted
52
database/sqlite/sqlite_functions.c
+2 -2
@@ -664,10 +664,10 @@ void sql_rrdset2json(RRDHOST *host, BUFFER *wb)
664 , dimensions
665 , (size_t) 0
666 , (size_t) 0
667 - , rrd_hosts_available
667 + , rrdhost_hosts_available()
668 );
669
670 - if(unlikely(rrd_hosts_available > 1)) {
670 + if(unlikely(rrdhost_hosts_available() > 1)) {
671 rrd_rdlock();
672
673 size_t found = 0;
database/sqlite/sqlite_metadata.c
+78 -293
@@ -4,9 +4,9 @@
4
5 // SQL statements
6
7 -#define SQL_STORE_CLAIM_ID "insert into node_instance " \
8 - "(host_id, claim_id, date_created) values (@host_id, @claim_id, unixepoch()) " \
9 - "on conflict(host_id) do update set claim_id = excluded.claim_id;"
7 +#define SQL_STORE_CLAIM_ID "INSERT INTO node_instance " \
8 + "(host_id, claim_id, date_created) VALUES (@host_id, @claim_id, unixepoch()) " \
9 + "ON CONFLICT(host_id) DO UPDATE SET claim_id = excluded.claim_id;"
10
11 #define SQL_DELETE_HOST_LABELS "DELETE FROM host_label WHERE host_id = @uuid;"
12
@@ -56,24 +56,12 @@
56
57 #define MAX_METADATA_CLEANUP (500) // Maximum metadata write operations (e.g deletes before retrying)
58 #define METADATA_MAX_BATCH_SIZE (512) // Maximum commands to execute before running the event loop
59 -#define METADATA_MAX_TRANSACTION_BATCH (128) // Maximum commands to add in a transaction
59
60 enum metadata_opcode {
61 METADATA_DATABASE_NOOP = 0,
62 METADATA_DATABASE_TIMER,
64 - METADATA_ADD_CHART,
65 - METADATA_ADD_CHART_LABEL,
66 - METADATA_ADD_DIMENSION,
63 METADATA_DEL_DIMENSION,
68 - METADATA_ADD_DIMENSION_OPTION,
69 - METADATA_ADD_HOST_SYSTEM_INFO,
70 - METADATA_ADD_HOST_INFO,
64 METADATA_STORE_CLAIM_ID,
72 - METADATA_STORE_HOST_LABELS,
73 - METADATA_STORE_BUFFER,
74 -
75 - METADATA_SKIP_TRANSACTION, // Dummy -- OPCODES less than this one can be in a tranasction
76 -
65 METADATA_SCAN_HOSTS,
66 METADATA_MAINTENANCE,
67 METADATA_SYNC_SHUTDOWN,
@@ -105,14 +93,14 @@ typedef enum {
93
94 struct metadata_wc {
95 uv_thread_t thread;
96 + uv_loop_t *loop;
97 + uv_async_t async;
98 + uv_timer_t timer_req;
99 time_t check_metadata_after;
100 time_t check_hosts_after;
101 volatile unsigned queue_size;
111 - uv_loop_t *loop;
112 - uv_async_t async;
102 METADATA_FLAG flags;
103 uint64_t row_id;
115 - uv_timer_t timer_req;
104 struct completion init_complete;
105 /* FIFO command queue */
106 uv_mutex_t cmd_mutex;
@@ -417,49 +405,6 @@ static BUFFER *sql_store_host_system_info(RRDHOST *host)
405 }
406
407
420 -/*
421 - * Store set option for a dimension
422 - */
423 -static int sql_set_dimension_option(uuid_t *dim_uuid, char *option)
424 -{
425 - sqlite3_stmt *res = NULL;
426 - int rc;
427 -
428 - if (unlikely(!db_meta)) {
429 - if (default_rrd_memory_mode != RRD_MEMORY_MODE_DBENGINE)
430 - return 0;
431 - error_report("Database has not been initialized");
432 - return 1;
433 - }
434 -
435 - rc = sqlite3_prepare_v2(db_meta, "UPDATE dimension SET options = @options WHERE dim_id = @dim_id", -1, &res, 0);
436 - if (unlikely(rc != SQLITE_OK)) {
437 - error_report("Failed to prepare statement to update dimension options");
438 - return 0;
439 - };
440 -
441 - rc = sqlite3_bind_blob(res, 2, dim_uuid, sizeof(*dim_uuid), SQLITE_STATIC);
442 - if (unlikely(rc != SQLITE_OK))
443 - goto bind_fail;
444 -
445 - if (!option || !strcmp(option,"unhide"))
446 - rc = sqlite3_bind_null(res, 1);
447 - else
448 - rc = sqlite3_bind_text(res, 1, option, -1, SQLITE_STATIC);
449 - if (unlikely(rc != SQLITE_OK))
450 - goto bind_fail;
451 -
452 - rc = execute_insert(res);
453 - if (unlikely(rc != SQLITE_DONE))
454 - error_report("Failed to update dimension option, rc = %d", rc);
455 -
456 -bind_fail:
457 - rc = sqlite3_finalize(res);
458 - if (unlikely(rc != SQLITE_OK))
459 - error_report("Failed to finalize statement in update dimension options, rc = %d", rc);
460 - return 0;
461 -}
462 -
408 /*
409 * Store a chart in the database
410 */
@@ -668,19 +613,23 @@ bind_fail:
613 static bool dimension_can_be_deleted(uuid_t *dim_uuid)
614 {
615 #ifdef ENABLE_DBENGINE
671 - bool no_retention = true;
672 - for (size_t tier = 0; tier < storage_tiers; tier++) {
673 - if (!multidb_ctx[tier])
674 - continue;
675 - time_t first_time_t = 0, last_time_t = 0;
676 - if (rrdeng_metric_retention_by_uuid((void *) multidb_ctx[tier], dim_uuid, &first_time_t, &last_time_t) == 0) {
677 - if (first_time_t > 0) {
678 - no_retention = false;
679 - break;
616 + if(dbengine_enabled) {
617 + bool no_retention = true;
618 + for (size_t tier = 0; tier < storage_tiers; tier++) {
619 + if (!multidb_ctx[tier])
620 + continue;
621 + time_t first_time_t = 0, last_time_t = 0;
622 + if (rrdeng_metric_retention_by_uuid((void *) multidb_ctx[tier], dim_uuid, &first_time_t, &last_time_t)) {
623 + if (first_time_t > 0) {
624 + no_retention = false;
625 + break;
626 + }
627 }
628 }
629 + return no_retention;
630 }
683 - return no_retention;
631 + else
632 + return false;
633 #else
634 return false;
635 #endif
@@ -817,7 +766,7 @@ static void metadata_enq_cmd(struct metadata_wc *wc, struct metadata_cmd *cmd)
766 (void) uv_async_send(&wc->async);
767 }
768
820 -static struct metadata_cmd metadata_deq_cmd(struct metadata_wc *wc, enum metadata_opcode *next_opcode)
769 +static struct metadata_cmd metadata_deq_cmd(struct metadata_wc *wc)
770 {
771 struct metadata_cmd ret;
772 unsigned queue_size;
@@ -828,7 +777,6 @@ static struct metadata_cmd metadata_deq_cmd(struct metadata_wc *wc, enum metadat
777 memset(&ret, 0, sizeof(ret));
778 ret.opcode = METADATA_DATABASE_NOOP;
779 ret.completion = NULL;
831 - *next_opcode = METADATA_DATABASE_NOOP;
780 } else {
781 /* dequeue command */
782 ret = wc->cmd_queue.cmd_array[wc->cmd_queue.head];
@@ -840,10 +788,6 @@ static struct metadata_cmd metadata_deq_cmd(struct metadata_wc *wc, enum metadat
788 wc->cmd_queue.head + 1 : 0;
789 }
790 wc->queue_size = queue_size - 1;
843 - if (wc->queue_size > 0)
844 - *next_opcode = wc->cmd_queue.cmd_array[wc->cmd_queue.head].opcode;
845 - else
846 - *next_opcode = METADATA_DATABASE_NOOP;
791 /* wake up producers */
792 uv_cond_signal(&wc->cmd_cond);
793 }
@@ -892,10 +836,15 @@ static void after_metadata_cleanup(uv_work_t *req, int status)
836 struct metadata_wc *wc = req->data;
837 metadata_flag_clear(wc, METADATA_FLAG_CLEANUP);
838 }
839 +
840 static void start_metadata_cleanup(uv_work_t *req)
841 {
842 + register_libuv_worker_jobs();
843 +
844 + worker_is_busy(UV_EVENT_METADATA_CLEANUP);
845 struct metadata_wc *wc = req->data;
846 check_dimension_metadata(wc);
847 + worker_is_idle();
848 }
849
850 struct scan_metadata_payload {
@@ -965,6 +914,11 @@ static bool metadata_scan_host(RRDHOST *host, uint32_t max_count) {
914 if(rrddim_flag_check(rd, RRDDIM_FLAG_METADATA_UPDATE)) {
915 rrddim_flag_clear(rd, RRDDIM_FLAG_METADATA_UPDATE);
916
917 + if (rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN))
918 + rrddim_flag_set(rd, RRDDIM_FLAG_META_HIDDEN);
919 + else
920 + rrddim_flag_clear(rd, RRDDIM_FLAG_META_HIDDEN);
921 +
922 rc = sql_store_dimension(
923 &rd->metric_uuid,
924 &rd->rrdset->chart_uuid,
@@ -990,17 +944,54 @@ static bool metadata_scan_host(RRDHOST *host, uint32_t max_count) {
944 // Worker thread to scan hosts for pending metadata to store
945 static void start_metadata_hosts(uv_work_t *req __maybe_unused)
946 {
947 + register_libuv_worker_jobs();
948 +
949 RRDHOST *host;
950
951 struct scan_metadata_payload *data = req->data;
952 struct metadata_wc *wc = data->wc;
953
954 bool run_again = false;
955 + worker_is_busy(UV_EVENT_METADATA_STORE);
956 dfe_start_reentrant(rrdhost_root_index, host) {
957 if (rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED) || !rrdhost_flag_check(host, RRDHOST_FLAG_METADATA_UPDATE))
958 continue;
959 internal_error(true, "METADATA: Scanning host %s", rrdhost_hostname(host));
1003 - rrdhost_flag_clear(host,RRDHOST_FLAG_METADATA_UPDATE);
960 +
961 + if (unlikely(rrdhost_flag_check(host, RRDHOST_FLAG_METADATA_LABELS))) {
962 + rrdhost_flag_clear(host, RRDHOST_FLAG_METADATA_LABELS);
963 + int rc = exec_statement_with_uuid(SQL_DELETE_HOST_LABELS, &host->host_uuid);
964 + if (likely(rc == SQLITE_OK)) {
965 + BUFFER *work_buffer = buffer_create(1024);
966 + struct query_build tmp = {.sql = work_buffer, .count = 0};
967 + uuid_unparse_lower(host->host_uuid, tmp.uuid_str);
968 + rrdlabels_walkthrough_read(host->rrdlabels, host_label_store_to_sql_callback, &tmp);
969 + db_execute(buffer_tostring(work_buffer));
970 + buffer_free(work_buffer);
971 + }
972 + }
973 +
974 + if (unlikely(rrdhost_flag_check(host, RRDHOST_FLAG_METADATA_CLAIMID))) {
975 + rrdhost_flag_clear(host, RRDHOST_FLAG_METADATA_CLAIMID);
976 + uuid_t uuid;
977 + if (likely(host->aclk_state.claimed_id && !uuid_parse(host->aclk_state.claimed_id, uuid)))
978 + store_claim_id(&host->host_uuid, &uuid);
979 + else
980 + store_claim_id(&host->host_uuid, NULL);
981 + }
982 +
983 + if (unlikely(rrdhost_flag_check(host, RRDHOST_FLAG_METADATA_INFO))) {
984 + rrdhost_flag_clear(host, RRDHOST_FLAG_METADATA_INFO);
985 +
986 + BUFFER *work_buffer = sql_store_host_system_info(host);
987 + db_execute(buffer_tostring(work_buffer));
988 + buffer_free(work_buffer);
989 +
990 + int rc = sql_store_host_info(host);
991 + if (unlikely(rc))
992 + error_report("Failed to store host info in the database for %s", string2str(host->hostname));
993 + }
994 +
995 if (unlikely(metadata_scan_host(host, data->max_count))) {
996 run_again = true;
997 rrdhost_flag_set(host,RRDHOST_FLAG_METADATA_UPDATE);
@@ -1008,10 +999,12 @@ static void start_metadata_hosts(uv_work_t *req __maybe_unused)
999 }
1000 }
1001 dfe_done(host);
1002 +
1003 if (unlikely(run_again))
1004 wc->check_hosts_after = now_realtime_sec() + METADATA_HOST_CHECK_IMMEDIATE;
1005 else
1006 wc->check_hosts_after = now_realtime_sec() + METADATA_HOST_CHECK_INTERVAL;
1007 + worker_is_idle();
1008 }
1009
1010 static void metadata_event_loop(void *arg)
@@ -1019,23 +1012,15 @@ static void metadata_event_loop(void *arg)
1012 worker_register("METASYNC");
1013 worker_register_job_name(METADATA_DATABASE_NOOP, "noop");
1014 worker_register_job_name(METADATA_DATABASE_TIMER, "timer");
1022 - worker_register_job_name(METADATA_ADD_CHART, "add chart");
1023 - worker_register_job_name(METADATA_ADD_CHART_LABEL, "add chart label");
1024 - worker_register_job_name(METADATA_ADD_DIMENSION, "add dimension");
1015 worker_register_job_name(METADATA_DEL_DIMENSION, "delete dimension");
1026 - worker_register_job_name(METADATA_ADD_DIMENSION_OPTION, "dimension option");
1027 - worker_register_job_name(METADATA_ADD_HOST_SYSTEM_INFO, "host system info");
1028 - worker_register_job_name(METADATA_ADD_HOST_INFO, "host info");
1016 worker_register_job_name(METADATA_STORE_CLAIM_ID, "add claim id");
1030 - worker_register_job_name(METADATA_STORE_HOST_LABELS, "host labels");
1017 worker_register_job_name(METADATA_MAINTENANCE, "maintenance");
1018
1033 -
1019 int ret;
1020 uv_loop_t *loop;
1021 unsigned cmd_batch_size;
1022 struct metadata_wc *wc = arg;
1038 - enum metadata_opcode opcode, next_opcode;
1023 + enum metadata_opcode opcode;
1024 uv_work_t metadata_cleanup_worker;
1025
1026 uv_thread_set_name_np(wc->thread, "METASYNC");
@@ -1073,21 +1058,11 @@ static void metadata_event_loop(void *arg)
1058 wc->check_hosts_after = now_realtime_sec() + METADATA_HOST_CHECK_FIRST_CHECK;
1059
1060 int shutdown = 0;
1076 - int in_transaction = 0;
1077 - int commands_in_transaction = 0;
1078 - // This can be used in the event loop for all opcodes (not workers)
1079 - BUFFER *work_buffer = buffer_create(1024);
1061 wc->row_id = 0;
1062 completion_mark_complete(&wc->init_complete);
1063
1064 while (shutdown == 0 || (wc->flags & METADATA_WORKER_BUSY)) {
1084 - RRDDIM *rd = NULL;
1085 - RRDSET *st = NULL;
1086 - RRDHOST *host = NULL;
1087 - DICTIONARY_ITEM *dict_item = NULL;
1088 - BUFFER *buffer = NULL;
1065 uuid_t *uuid;
1090 - int rc;
1066
1067 worker_is_idle();
1068 uv_run(loop, UV_RUN_DEFAULT);
@@ -1098,7 +1073,7 @@ static void metadata_event_loop(void *arg)
1073 if (unlikely(cmd_batch_size >= METADATA_MAX_BATCH_SIZE))
1074 break;
1075
1101 - cmd = metadata_deq_cmd(wc, &next_opcode);
1076 + cmd = metadata_deq_cmd(wc);
1077 opcode = cmd.opcode;
1078
1079 if (unlikely(opcode == METADATA_DATABASE_NOOP && metadata_flag_check(wc, METADATA_FLAG_SHUTDOWN))) {
@@ -1108,130 +1083,32 @@ static void metadata_event_loop(void *arg)
1083
1084 ++cmd_batch_size;
1085
1111 - // If we are not in transaction and this command is the same with the next ; start a transaction
1112 - if (!in_transaction && opcode < METADATA_SKIP_TRANSACTION && opcode == next_opcode) {
1113 - if (opcode != METADATA_DATABASE_NOOP) {
1114 - in_transaction = 1;
1115 - db_execute("BEGIN TRANSACTION;");
1116 - }
1117 - }
1118 -
1119 - if (likely(in_transaction)) {
1120 - commands_in_transaction++;
1121 - }
1122 -
1086 if (likely(opcode != METADATA_DATABASE_NOOP))
1124 - worker_is_busy(opcode);
1087 + worker_is_busy(opcode);
1088
1089 switch (opcode) {
1090 case METADATA_DATABASE_NOOP:
1091 case METADATA_DATABASE_TIMER:
1092 break;
1130 - case METADATA_ADD_CHART:
1131 - dict_item = (DICTIONARY_ITEM * ) cmd.param[0];
1132 - st = (RRDSET *) dictionary_acquired_item_value(dict_item);
1133 -
1134 - rc = sql_store_chart(
1135 - &st->chart_uuid,
1136 - &st->rrdhost->host_uuid,
1137 - string2str(st->parts.type),
1138 - string2str(st->parts.id),
1139 - string2str(st->parts.name),
1140 - rrdset_family(st),
1141 - rrdset_context(st),
1142 - rrdset_title(st),
1143 - rrdset_units(st),
1144 - rrdset_plugin_name(st),
1145 - rrdset_module_name(st),
1146 - st->priority,
1147 - st->update_every,
1148 - st->chart_type,
1149 - st->rrd_memory_mode,
1150 - st->entries);
1151 -
1152 - if (unlikely(rc))
1153 - error_report("Failed to store chart %s", rrdset_id(st));
1154 -
1155 - dictionary_acquired_item_release(st->rrdhost->rrdset_root_index, dict_item);
1156 - break;
1157 - case METADATA_ADD_CHART_LABEL:
1158 - dict_item = (DICTIONARY_ITEM * ) cmd.param[0];
1159 - st = (RRDSET *) dictionary_acquired_item_value(dict_item);
1160 - check_and_update_chart_labels(st, work_buffer);
1161 - dictionary_acquired_item_release(st->rrdhost->rrdset_root_index, dict_item);
1162 - break;
1163 - case METADATA_ADD_DIMENSION:
1164 - dict_item = (DICTIONARY_ITEM * ) cmd.param[0];
1165 - rd = (RRDDIM *) dictionary_acquired_item_value(dict_item);
1166 -
1167 - rc = sql_store_dimension(
1168 - &rd->metric_uuid,
1169 - &rd->rrdset->chart_uuid,
1170 - string2str(rd->id),
1171 - string2str(rd->name),
1172 - rd->multiplier,
1173 - rd->divisor,
1174 - rd->algorithm,
1175 - rrddim_option_check(rd, RRDDIM_OPTION_HIDDEN));
1176 -
1177 - if (unlikely(rc))
1178 - error_report("Failed to store dimension %s", rrddim_id(rd));
1179 -
1180 - dictionary_acquired_item_release(rd->rrdset->rrddim_root_index, dict_item);
1181 - break;
1093 +
1094 case METADATA_DEL_DIMENSION:
1095 uuid = (uuid_t *) cmd.param[0];
1096 if (likely(dimension_can_be_deleted(uuid)))
1097 delete_dimension_uuid(uuid);
1098 freez(uuid);
1099 break;
1188 - case METADATA_ADD_DIMENSION_OPTION:
1189 - dict_item = (DICTIONARY_ITEM * ) cmd.param[0];
1190 - rd = (RRDDIM *) dictionary_acquired_item_value(dict_item);
1191 - rc = sql_set_dimension_option(
1192 - &rd->metric_uuid, rrddim_flag_check(rd, RRDDIM_FLAG_META_HIDDEN) ? "hidden" : NULL);
1193 - if (unlikely(rc))
1194 - error_report("Failed to store dimension option for %s", string2str(rd->id));
1195 - dictionary_acquired_item_release(rd->rrdset->rrddim_root_index, dict_item);
1196 - break;
1197 - case METADATA_ADD_HOST_SYSTEM_INFO:
1198 - buffer = (BUFFER *) cmd.param[0];
1199 - db_execute(buffer_tostring(buffer));
1200 - buffer_free(buffer);
1201 - break;
1202 - case METADATA_ADD_HOST_INFO:
1203 - dict_item = (DICTIONARY_ITEM * ) cmd.param[0];
1204 - host = (RRDHOST *) dictionary_acquired_item_value(dict_item);
1205 - rc = sql_store_host_info(host);
1206 - if (unlikely(rc))
1207 - error_report("Failed to store host info in the database for %s", string2str(host->hostname));
1208 - dictionary_acquired_item_release(rrdhost_root_index, dict_item);
1209 - break;
1100 case METADATA_STORE_CLAIM_ID:
1101 store_claim_id((uuid_t *) cmd.param[0], (uuid_t *) cmd.param[1]);
1102 freez((void *) cmd.param[0]);
1103 freez((void *) cmd.param[1]);
1104 break;
1215 - case METADATA_STORE_HOST_LABELS:
1216 - dict_item = (DICTIONARY_ITEM * ) cmd.param[0];
1217 - host = (RRDHOST *) dictionary_acquired_item_value(dict_item);
1218 - rc = exec_statement_with_uuid(SQL_DELETE_HOST_LABELS, &host->host_uuid);
1219 -
1220 - if (likely(rc == SQLITE_OK)) {
1221 - buffer_flush(work_buffer);
1222 - struct query_build tmp = {.sql = work_buffer, .count = 0};
1223 - uuid_unparse_lower(host->host_uuid, tmp.uuid_str);
1224 - rrdlabels_walkthrough_read(host->rrdlabels, host_label_store_to_sql_callback, &tmp);
1225 - db_execute(buffer_tostring(work_buffer));
1226 - }
1227 -
1228 - dictionary_acquired_item_release(rrdhost_root_index, dict_item);
1229 - break;
1230 -
1105 case METADATA_SCAN_HOSTS:
1106 if (unlikely(metadata_flag_check(wc, METADATA_FLAG_SCANNING_HOSTS)))
1107 break;
1108
1109 + if (unittest_running)
1110 + break;
1111 +
1112 struct scan_metadata_payload *data = mallocz(sizeof(*data));
1113 data->request.data = data;
1114 data->wc = wc;
@@ -1255,11 +1132,6 @@ static void metadata_event_loop(void *arg)
1132 metadata_flag_clear(wc, METADATA_FLAG_SCANNING_HOSTS);
1133 }
1134 break;
1258 - case METADATA_STORE_BUFFER:
1259 - buffer = (BUFFER *) cmd.param[0];
1260 - db_execute(buffer_tostring(buffer));
1261 - buffer_free(buffer);
1262 - break;
1135 case METADATA_MAINTENANCE:
1136 if (unlikely(metadata_flag_check(wc, METADATA_FLAG_CLEANUP)))
1137 break;
@@ -1279,11 +1151,6 @@ static void metadata_event_loop(void *arg)
1151 default:
1152 break;
1153 }
1282 - if (in_transaction && (commands_in_transaction >= METADATA_MAX_TRANSACTION_BATCH || opcode != next_opcode)) {
1283 - in_transaction = 0;
1284 - db_execute("COMMIT TRANSACTION;");
1285 - commands_in_transaction = 0;
1286 - }
1154
1155 if (cmd.completion)
1156 completion_mark_complete(cmd.completion);
@@ -1302,8 +1169,6 @@ static void metadata_event_loop(void *arg)
1169 uv_run(loop, UV_RUN_DEFAULT);
1170
1171 uv_cond_destroy(&wc->cmd_cond);
1305 - /* uv_mutex_destroy(&wc->cmd_mutex); */
1306 - //fatal_assert(0 == uv_loop_close(loop));
1172 int rc;
1173
1174 do {
@@ -1313,7 +1178,6 @@ static void metadata_event_loop(void *arg)
1178 freez(loop);
1179 worker_unregister();
1180
1316 - buffer_free(work_buffer);
1181 info("METADATA: Shutting down event loop");
1182 completion_mark_complete(&wc->init_complete);
1183 return;
@@ -1408,50 +1272,6 @@ static inline void queue_metadata_cmd(enum metadata_opcode opcode, const void *p
1272 }
1273
1274 // Public
1411 -void metaqueue_chart_update(RRDSET *st)
1412 -{
1413 - const DICTIONARY_ITEM *acquired_st = dictionary_get_and_acquire_item(st->rrdhost->rrdset_root_index, string2str(st->id));
1414 - queue_metadata_cmd(METADATA_ADD_CHART, acquired_st, NULL);
1415 -}
1416 -
1417 -//
1418 -// RD may not be collected, so we may store it needlessly
1419 -void metaqueue_dimension_update(RRDDIM *rd)
1420 -{
1421 - const DICTIONARY_ITEM *acquired_rd =
1422 - dictionary_get_and_acquire_item(rd->rrdset->rrddim_root_index, string2str(rd->id));
1423 -
1424 - if (unlikely(rrdset_flag_check(rd->rrdset, RRDSET_FLAG_METADATA_UPDATE))) {
1425 - metaqueue_chart_update(rd->rrdset);
1426 - rrdset_flag_clear(rd->rrdset, RRDSET_FLAG_METADATA_UPDATE);
1427 - }
1428 -
1429 - queue_metadata_cmd(METADATA_ADD_DIMENSION, acquired_rd, NULL);
1430 -}
1431 -
1432 -void metaqueue_dimension_update_flags(RRDDIM *rd)
1433 -{
1434 - const DICTIONARY_ITEM *acquired_rd =
1435 - dictionary_get_and_acquire_item(rd->rrdset->rrddim_root_index, string2str(rd->id));
1436 - queue_metadata_cmd(METADATA_ADD_DIMENSION_OPTION, acquired_rd, NULL);
1437 -}
1438 -
1439 -void metaqueue_host_update_system_info(RRDHOST *host)
1440 -{
1441 - BUFFER *work_buffer = sql_store_host_system_info(host);
1442 -
1443 - if (unlikely(!work_buffer))
1444 - return;
1445 -
1446 - queue_metadata_cmd(METADATA_ADD_HOST_SYSTEM_INFO, work_buffer, NULL);
1447 -}
1448 -
1449 -void metaqueue_host_update_info(const char *machine_guid)
1450 -{
1451 - const DICTIONARY_ITEM *acquired_host = dictionary_get_and_acquire_item(rrdhost_root_index, machine_guid);
1452 - queue_metadata_cmd(METADATA_ADD_HOST_INFO, acquired_host, NULL);
1453 -}
1454 -
1275 void metaqueue_delete_dimension_uuid(uuid_t *uuid)
1276 {
1277 if (unlikely(!metasync_worker.loop))
@@ -1461,40 +1281,6 @@ void metaqueue_delete_dimension_uuid(uuid_t *uuid)
1281 queue_metadata_cmd(METADATA_DEL_DIMENSION, use_uuid, NULL);
1282 }
1283
1464 -void metaqueue_store_claim_id(uuid_t *host_uuid, uuid_t *claim_uuid)
1465 -{
1466 - if (unlikely(!host_uuid))
1467 - return;
1468 -
1469 - uuid_t *local_host_uuid = mallocz(sizeof(*host_uuid));
1470 - uuid_t *local_claim_uuid = NULL;
1471 -
1472 - uuid_copy(*local_host_uuid, *host_uuid);
1473 - if (likely(claim_uuid)) {
1474 - local_claim_uuid = mallocz(sizeof(*claim_uuid));
1475 - uuid_copy(*local_claim_uuid, *claim_uuid);
1476 - }
1477 - queue_metadata_cmd(METADATA_STORE_CLAIM_ID, local_host_uuid, local_claim_uuid);
1478 -}
1479 -
1480 -void metaqueue_store_host_labels(const char *machine_guid)
1481 -{
1482 - const DICTIONARY_ITEM *acquired_host = dictionary_get_and_acquire_item(rrdhost_root_index, machine_guid);
1483 - queue_metadata_cmd(METADATA_STORE_HOST_LABELS, acquired_host, NULL);
1484 -}
1485 -
1486 -void metaqueue_buffer(BUFFER *buffer)
1487 -{
1488 - queue_metadata_cmd(METADATA_STORE_BUFFER, buffer, NULL);
1489 -}
1490 -
1491 -void metaqueue_chart_labels(RRDSET *st)
1492 -{
1493 - const DICTIONARY_ITEM *acquired_st = dictionary_get_and_acquire_item(st->rrdhost->rrdset_root_index, string2str(st->id));
1494 - queue_metadata_cmd(METADATA_ADD_CHART_LABEL, acquired_st, NULL);
1495 -}
1496 -
1497 -
1284 //
1285 // unitests
1286 //
@@ -1558,7 +1344,6 @@ static void *metadata_unittest_threads(void)
1344 void *retval;
1345 netdata_thread_join(threads[i], &retval);
1346 }
1561 -// uv_async_send(&metasync_worker.async);
1347 sleep_usec(5 * USEC_PER_SEC);
1348
1349 fprintf(stderr, "Added %u elements, processed %u\n", tu.added, tu.processed);
database/sqlite/sqlite_metadata.h
-9
@@ -11,17 +11,8 @@ void metadata_sync_init(void);
11 void metadata_sync_shutdown(void);
12 void metadata_sync_shutdown_prepare(void);
13
14 -void metaqueue_dimension_update(RRDDIM *rd);
15 -void metaqueue_chart_update(RRDSET *st);
16 -void metaqueue_dimension_update_flags(RRDDIM *rd);
17 -void metaqueue_host_update_system_info(RRDHOST *host);
18 -void metaqueue_host_update_info(const char *machine_guid);
14 void metaqueue_delete_dimension_uuid(uuid_t *uuid);
20 -void metaqueue_store_claim_id(uuid_t *host_uuid, uuid_t *claim_uuid);
21 -void metaqueue_store_host_labels(const char *machine_guid);
22 -void metaqueue_chart_labels(RRDSET *st);
15 void migrate_localhost(uuid_t *host_uuid);
24 -void metaqueue_buffer(BUFFER *buffer);
16
17 // UNIT TEST
18 int metadata_unittest(void);
database/storage_engine.c
+29 -21
@@ -6,23 +6,24 @@
6 #include "engine/rrdengineapi.h"
7 #endif
8
9 -#define im_collect_ops { \
10 - .init = rrddim_collect_init,\
11 - .store_metric = rrddim_collect_store_metric,\
12 - .flush = rrddim_store_metric_flush,\
13 - .finalize = rrddim_collect_finalize, \
9 +#define im_collect_ops { \
10 + .init = rrddim_collect_init, \
11 + .store_metric = rrddim_collect_store_metric, \
12 + .flush = rrddim_store_metric_flush, \
13 + .finalize = rrddim_collect_finalize, \
14 .change_collection_frequency = rrddim_store_metric_change_collection_frequency, \
15 - .metrics_group_get = rrddim_metrics_group_get, \
16 - .metrics_group_release = rrddim_metrics_group_release, \
15 + .metrics_group_get = rrddim_metrics_group_get, \
16 + .metrics_group_release = rrddim_metrics_group_release, \
17 }
18
19 -#define im_query_ops { \
20 - .init = rrddim_query_init, \
21 - .next_metric = rrddim_query_next_metric, \
22 - .is_finished = rrddim_query_is_finished, \
23 - .finalize = rrddim_query_finalize, \
24 - .latest_time = rrddim_query_latest_time, \
25 - .oldest_time = rrddim_query_oldest_time \
19 +#define im_query_ops { \
20 + .init = rrddim_query_init, \
21 + .next_metric = rrddim_query_next_metric, \
22 + .is_finished = rrddim_query_is_finished, \
23 + .finalize = rrddim_query_finalize, \
24 + .latest_time_s = rrddim_query_latest_time_s, \
25 + .oldest_time_s = rrddim_query_oldest_time_s, \
26 + .align_to_optimal_before = rrddim_query_align_to_optimal_before, \
27 }
28
29 static STORAGE_ENGINE engines[] = {
@@ -34,8 +35,9 @@ static STORAGE_ENGINE engines[] = {
35 .metric_get_or_create = rrddim_metric_get_or_create,
36 .metric_dup = rrddim_metric_dup,
37 .metric_release = rrddim_metric_release,
38 + .metric_retention_by_uuid = rrddim_metric_retention_by_uuid,
39 .collect_ops = im_collect_ops,
38 - .query_ops = im_query_ops
40 + .query_ops = im_query_ops,
41 }
42 },
43 {
@@ -46,8 +48,9 @@ static STORAGE_ENGINE engines[] = {
48 .metric_get_or_create = rrddim_metric_get_or_create,
49 .metric_dup = rrddim_metric_dup,
50 .metric_release = rrddim_metric_release,
51 + .metric_retention_by_uuid = rrddim_metric_retention_by_uuid,
52 .collect_ops = im_collect_ops,
50 - .query_ops = im_query_ops
53 + .query_ops = im_query_ops,
54 }
55 },
56 {
@@ -58,8 +61,9 @@ static STORAGE_ENGINE engines[] = {
61 .metric_get_or_create = rrddim_metric_get_or_create,
62 .metric_dup = rrddim_metric_dup,
63 .metric_release = rrddim_metric_release,
64 + .metric_retention_by_uuid = rrddim_metric_retention_by_uuid,
65 .collect_ops = im_collect_ops,
62 - .query_ops = im_query_ops
66 + .query_ops = im_query_ops,
67 }
68 },
69 {
@@ -70,8 +74,9 @@ static STORAGE_ENGINE engines[] = {
74 .metric_get_or_create = rrddim_metric_get_or_create,
75 .metric_dup = rrddim_metric_dup,
76 .metric_release = rrddim_metric_release,
77 + .metric_retention_by_uuid = rrddim_metric_retention_by_uuid,
78 .collect_ops = im_collect_ops,
74 - .query_ops = im_query_ops
79 + .query_ops = im_query_ops,
80 }
81 },
82 {
@@ -82,8 +87,9 @@ static STORAGE_ENGINE engines[] = {
87 .metric_get_or_create = rrddim_metric_get_or_create,
88 .metric_dup = rrddim_metric_dup,
89 .metric_release = rrddim_metric_release,
90 + .metric_retention_by_uuid = rrddim_metric_retention_by_uuid,
91 .collect_ops = im_collect_ops,
86 - .query_ops = im_query_ops
92 + .query_ops = im_query_ops,
93 }
94 },
95 #ifdef ENABLE_DBENGINE
@@ -95,6 +101,7 @@ static STORAGE_ENGINE engines[] = {
101 .metric_get_or_create = rrdeng_metric_get_or_create,
102 .metric_dup = rrdeng_metric_dup,
103 .metric_release = rrdeng_metric_release,
104 + .metric_retention_by_uuid = rrdeng_metric_retention_by_uuid,
105 .collect_ops = {
106 .init = rrdeng_store_metric_init,
107 .store_metric = rrdeng_store_metric_next,
@@ -109,8 +116,9 @@ static STORAGE_ENGINE engines[] = {
116 .next_metric = rrdeng_load_metric_next,
117 .is_finished = rrdeng_load_metric_is_finished,
118 .finalize = rrdeng_load_metric_finalize,
112 - .latest_time = rrdeng_metric_latest_time,
113 - .oldest_time = rrdeng_metric_oldest_time
119 + .latest_time_s = rrdeng_metric_latest_time,
120 + .oldest_time_s = rrdeng_metric_oldest_time,
121 + .align_to_optimal_before = rrdeng_load_align_to_optimal_before,
122 }
123 }
124 },
exporting/exporting_engine.c
+1 -1
@@ -197,7 +197,7 @@ void *exporting_main(void *ptr)
197 heartbeat_t hb;
198 heartbeat_init(&hb);
199
200 - while (!netdata_exit) {
200 + while (service_running(SERVICE_EXPORTERS)) {
201 heartbeat_next(&hb, step_ut);
202 engine->now = now_realtime_sec();
203
exporting/process_data.c
+3 -3
@@ -77,8 +77,8 @@ NETDATA_DOUBLE exporting_calculate_value_from_stored_data(
77 time_t before = instance->before;
78
79 // find the edges of the rrd database for this chart
80 - time_t first_t = rd->tiers[0]->query_ops->oldest_time(rd->tiers[0]->db_metric_handle);
81 - time_t last_t = rd->tiers[0]->query_ops->latest_time(rd->tiers[0]->db_metric_handle);
80 + time_t first_t = rd->tiers[0]->query_ops->oldest_time_s(rd->tiers[0]->db_metric_handle);
81 + time_t last_t = rd->tiers[0]->query_ops->latest_time_s(rd->tiers[0]->db_metric_handle);
82 time_t update_every = st->update_every;
83 struct storage_engine_query_handle handle;
84
@@ -126,7 +126,7 @@ NETDATA_DOUBLE exporting_calculate_value_from_stored_data(
126 size_t counter = 0;
127 NETDATA_DOUBLE sum = 0;
128
129 - for (rd->tiers[0]->query_ops->init(rd->tiers[0]->db_metric_handle, &handle, after, before); !rd->tiers[0]->query_ops->is_finished(&handle);) {
129 + for (rd->tiers[0]->query_ops->init(rd->tiers[0]->db_metric_handle, &handle, after, before, STORAGE_PRIORITY_LOW); !rd->tiers[0]->query_ops->is_finished(&handle);) {
130 STORAGE_POINT sp = rd->tiers[0]->query_ops->next_metric(&handle);
131 points_read++;
132
exporting/tests/test_exporting_engine.h
-3
@@ -55,9 +55,6 @@ int __wrap_connect_to_one_of(
55 size_t *reconnects_counter,
56 char *connected_to,
57 size_t connected_to_size);
58 -void __rrdhost_check_rdlock(RRDHOST *host, const char *file, const char *function, const unsigned long line);
59 -void __rrdset_check_rdlock(RRDSET *st, const char *file, const char *function, const unsigned long line);
60 -void __rrd_check_rdlock(const char *file, const char *function, const unsigned long line);
58 time_t __mock_rrddim_query_oldest_time(STORAGE_METRIC_HANDLE *db_metric_handle);
59 time_t __mock_rrddim_query_latest_time(STORAGE_METRIC_HANDLE *db_metric_handle);
60 void __mock_rrddim_query_init(STORAGE_METRIC_HANDLE *db_metric_handle, struct rrddim_query_handle *handle, time_t start_time, time_t end_time);
health/health.c
+16 -14
@@ -693,8 +693,8 @@ static inline int rrdcalc_isrunnable(RRDCALC *rc, time_t now, time_t *next_run)
693 }
694
695 int update_every = rc->rrdset->update_every;
696 - time_t first = rrdset_first_entry_t(rc->rrdset);
697 - time_t last = rrdset_last_entry_t(rc->rrdset);
696 + time_t first = rrdset_first_entry_s(rc->rrdset);
697 + time_t last = rrdset_last_entry_s(rc->rrdset);
698
699 if(unlikely(now + update_every < first /* || now - update_every > last */)) {
700 debug(D_HEALTH
@@ -742,13 +742,16 @@ static void health_thread_cleanup(void *ptr) {
742 struct health_state *h = ptr;
743 h->host->health_spawn = 0;
744
745 - netdata_thread_cancel(netdata_thread_self());
745 log_health("[%s]: Health thread ended.", rrdhost_hostname(h->host));
746 debug(D_HEALTH, "HEALTH %s: Health thread ended.", rrdhost_hostname(h->host));
747 }
748
749 static void initialize_health(RRDHOST *host, int is_localhost) {
751 - if(!host->health_enabled || rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH)) return;
750 + if(!host->health_enabled ||
751 + rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH) ||
752 + !service_running(SERVICE_HEALTH))
753 + return;
754 +
755 rrdhost_flag_set(host, RRDHOST_FLAG_INITIALIZED_HEALTH);
756
757 log_health("[%s]: Initializing health.", rrdhost_hostname(host));
@@ -840,7 +843,7 @@ static void health_sleep(time_t next_run, unsigned int loop __maybe_unused, RRDH
843 if(now < next_run) {
844 worker_is_idle();
845 debug(D_HEALTH, "Health monitoring iteration no %u done. Next iteration in %d secs", loop, (int) (next_run - now));
843 - while (now < next_run && host->health_enabled && !netdata_exit) {
846 + while (now < next_run && host->health_enabled && service_running(SERVICE_HEALTH)) {
847 sleep_usec(USEC_PER_SEC);
848 now = now_realtime_sec();
849 }
@@ -1014,15 +1017,13 @@ void *health_main(void *ptr) {
1017
1018 bool health_running_logged = false;
1019
1017 - rrdhost_rdlock(host); //CHECK
1020 rrdcalc_delete_alerts_not_matching_host_labels_from_this_host(host);
1019 - rrdhost_unlock(host);
1021
1022 unsigned int loop = 0;
1023 #ifdef ENABLE_ACLK
1024 unsigned int marked_aclk_reload_loop = 0;
1025 #endif
1025 - while(!netdata_exit && host->health_enabled) {
1026 + while(service_running(SERVICE_HEALTH) && host->health_enabled) {
1027 loop++;
1028 debug(D_HEALTH, "Health monitoring iteration no %u started", loop);
1029
@@ -1183,7 +1184,7 @@ void *health_main(void *ptr) {
1184 &rc->db_after,&rc->db_before,
1185 NULL, NULL, NULL,
1186 &value_is_null, NULL, 0, 0,
1186 - QUERY_SOURCE_HEALTH);
1187 + QUERY_SOURCE_HEALTH, STORAGE_PRIORITY_LOW);
1188
1189 if (unlikely(ret != 200)) {
1190 // database lookup failed
@@ -1259,7 +1260,7 @@ void *health_main(void *ptr) {
1260 }
1261 foreach_rrdcalc_in_rrdhost_done(rc);
1262
1262 - if (unlikely(runnable && !netdata_exit)) {
1263 + if (unlikely(runnable && service_running(SERVICE_HEALTH))) {
1264 foreach_rrdcalc_in_rrdhost_read(host, rc) {
1265 if (unlikely(!(rc->run_flags & RRDCALC_FLAG_RUNNABLE)))
1266 continue;
@@ -1512,7 +1513,7 @@ void *health_main(void *ptr) {
1513 foreach_rrdcalc_in_rrdhost_done(rc);
1514 }
1515
1515 - if (unlikely(netdata_exit))
1516 + if (unlikely(!service_running(SERVICE_HEALTH)))
1517 break;
1518
1519 // execute notifications
@@ -1520,7 +1521,7 @@ void *health_main(void *ptr) {
1521 worker_is_busy(WORKER_HEALTH_JOB_ALARM_LOG_PROCESS);
1522 health_alarm_log_process(host);
1523
1523 - if (unlikely(netdata_exit)) {
1524 + if (unlikely(!service_running(SERVICE_HEALTH))) {
1525 // wait for all notifications to finish before allowing health to be cleaned up
1526 ALARM_ENTRY *ae;
1527 while (NULL != (ae = alarm_notifications_in_progress.head)) {
@@ -1543,7 +1544,7 @@ void *health_main(void *ptr) {
1544 }
1545 #endif
1546
1546 - if(unlikely(netdata_exit))
1547 + if(unlikely(!service_running(SERVICE_HEALTH)))
1548 break;
1549
1550 health_sleep(next_run, loop, host);
@@ -1574,7 +1575,8 @@ void health_thread_spawn(RRDHOST * host) {
1575 struct health_state *health = callocz(1, sizeof(*health));
1576 health->host = host;
1577
1577 - if(netdata_thread_create(&host->health_thread, tag, NETDATA_THREAD_OPTION_JOINABLE, health_main, (void *) health)) {
1578 + netdata_thread_t health_thread;
1579 + if(netdata_thread_create(&health_thread, tag, NETDATA_THREAD_OPTION_DEFAULT, health_main, (void *) health)) {
1580 log_health("[%s]: Failed to create new thread for client.", rrdhost_hostname(host));
1581 error("HEALTH [%s]: Failed to create new thread for client.", rrdhost_hostname(host));
1582 }
health/health.h
-1
@@ -82,7 +82,6 @@ void health_alarm_log_add_entry(RRDHOST *host, ALARM_ENTRY *ae);
82
83 struct health_state {
84 RRDHOST *host;
85 - netdata_thread_t thread;
85 };
86
87 void health_readdir(RRDHOST *host, const char *user_path, const char *stock_path, const char *subpath);
health/health_config.c
+2 -1
@@ -1171,7 +1171,8 @@ void sql_refresh_hashes(void)
1171 }
1172
1173 void health_readdir(RRDHOST *host, const char *user_path, const char *stock_path, const char *subpath) {
1174 - if(unlikely(!host->health_enabled) && !rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH)) {
1174 + if(unlikely((!host->health_enabled) && !rrdhost_flag_check(host, RRDHOST_FLAG_INITIALIZED_HEALTH)) ||
1175 + !service_running(SERVICE_HEALTH)) {
1176 debug(D_HEALTH, "CONFIG health is not enabled for host '%s'", rrdhost_hostname(host));
1177 return;
1178 }
libnetdata/Makefile.am
+1
@@ -15,6 +15,7 @@ SUBDIRS = \
15 ebpf \
16 eval \
17 json \
18 + july \
19 health \
20 locks \
21 log \
libnetdata/arrayalloc/arrayalloc.c
+1 -1
@@ -210,7 +210,7 @@ static void arrayalloc_add_page(ARAL *ar TRACE_ALLOCATIONS_FUNCTION_DEFINITION_P
210 char filename[FILENAME_MAX + 1];
211 snprintfz(filename, FILENAME_MAX, "%s/array_alloc.mmap/%s.%zu", *ar->cache_dir, ar->filename, ar->internal.file_number);
212 page->filename = strdupz(filename);
213 - page->data = netdata_mmap(page->filename, page->size, MAP_SHARED, 0);
213 + page->data = netdata_mmap(page->filename, page->size, MAP_SHARED, 0, false);
214 if (unlikely(!page->data))
215 fatal("Cannot allocate arrayalloc buffer of size %zu on filename '%s'", page->size, page->filename);
216 }
libnetdata/clocks/clocks.c
+31 -11
@@ -189,9 +189,13 @@ void sleep_to_absolute_time(usec_t usec) {
189 .tv_nsec = (suseconds_t)((usec % USEC_PER_SEC) * NSEC_PER_USEC)
190 };
191
192 + errno = 0;
193 int ret = 0;
194 while( (ret = clock_nanosleep(clock, TIMER_ABSTIME, &req, NULL)) != 0 ) {
194 - if(ret == EINTR) continue;
195 + if(ret == EINTR) {
196 + errno = 0;
197 + continue;
198 + }
199 else {
200 if (ret == EINVAL) {
201 if (!einval_printed) {
@@ -313,7 +317,7 @@ usec_t heartbeat_next(heartbeat_t *hb, usec_t tick) {
317 // sleep_usec() has a loop to guarantee we will sleep for at least the requested time.
318 // According the specs, when we sleep for a relative time, clock adjustments should not affect the duration
319 // we sleep.
316 - sleep_usec(next - now);
320 + sleep_usec_with_now(next - now, now);
321 now = now_realtime_usec();
322 dt = now - hb->realtime;
323
@@ -342,7 +346,7 @@ usec_t heartbeat_next(heartbeat_t *hb, usec_t tick) {
346 return dt;
347 }
348
345 -void sleep_usec(usec_t usec) {
349 +void sleep_usec_with_now(usec_t usec, usec_t started_ut) {
350 // we expect microseconds (1.000.000 per second)
351 // but timespec is nanoseconds (1.000.000.000 per second)
352 struct timespec rem = { 0, 0 }, req = {
@@ -350,21 +354,37 @@ void sleep_usec(usec_t usec) {
354 .tv_nsec = (suseconds_t) ((usec % USEC_PER_SEC) * NSEC_PER_USEC)
355 };
356
353 -#ifdef __linux__
354 - while (clock_nanosleep(CLOCK_REALTIME, 0, &req, &rem) != 0) {
355 -#else
357 + // make sure errno is not EINTR
358 + errno = 0;
359 +
360 + if(!started_ut)
361 + started_ut = now_realtime_usec();
362 +
363 + usec_t end_ut = started_ut + usec;
364 +
365 while (nanosleep(&req, &rem) != 0) {
357 -#endif
366 if (likely(errno == EINTR && (rem.tv_sec || rem.tv_nsec))) {
367 req = rem;
368 rem = (struct timespec){ 0, 0 };
369 +
370 + // break an infinite loop
371 + errno = 0;
372 +
373 + usec_t now_ut = now_realtime_usec();
374 + if(now_ut >= end_ut)
375 + break;
376 +
377 + usec_t remaining_ut = (usec_t)req.tv_sec * USEC_PER_SEC + (usec_t)req.tv_nsec * NSEC_PER_USEC > usec;
378 + usec_t check_ut = now_ut - started_ut;
379 + if(remaining_ut > check_ut) {
380 + req = (struct timespec){
381 + .tv_sec = (time_t) ( check_ut / USEC_PER_SEC),
382 + .tv_nsec = (suseconds_t) ((check_ut % USEC_PER_SEC) * NSEC_PER_USEC)
383 + };
384 + }
385 }
386 else {
363 -#ifdef __linux__
364 - error("Cannot clock_nanosleep(CLOCK_REALTIME) for %llu microseconds.", usec);
365 -#else
387 error("Cannot nanosleep() for %llu microseconds.", usec);
367 -#endif
388 break;
389 }
390 }
libnetdata/clocks/clocks.h
+2 -1
@@ -141,7 +141,8 @@ usec_t heartbeat_next(heartbeat_t *hb, usec_t tick);
141
142 void heartbeat_statistics(usec_t *min_ptr, usec_t *max_ptr, usec_t *average_ptr, size_t *count_ptr);
143
144 -void sleep_usec(usec_t usec);
144 +void sleep_usec_with_now(usec_t usec, usec_t started_ut);
145 +#define sleep_usec(usec) sleep_usec_with_now(usec, 0);
146
147 void clocks_init(void);
148
libnetdata/completion/completion.c
+30
@@ -5,6 +5,7 @@
5 void completion_init(struct completion *p)
6 {
7 p->completed = 0;
8 + p->completed_jobs = 0;
9 fatal_assert(0 == uv_cond_init(&p->cond));
10 fatal_assert(0 == uv_mutex_init(&p->mutex));
11 }
@@ -32,3 +33,32 @@ void completion_mark_complete(struct completion *p)
33 uv_cond_broadcast(&p->cond);
34 uv_mutex_unlock(&p->mutex);
35 }
36 +
37 +unsigned completion_wait_for_a_job(struct completion *p, unsigned completed_jobs)
38 +{
39 + uv_mutex_lock(&p->mutex);
40 + while (0 == p->completed && p->completed_jobs <= completed_jobs) {
41 + uv_cond_wait(&p->cond, &p->mutex);
42 + }
43 + completed_jobs = p->completed_jobs;
44 + uv_mutex_unlock(&p->mutex);
45 +
46 + return completed_jobs;
47 +}
48 +
49 +void completion_mark_complete_a_job(struct completion *p)
50 +{
51 + uv_mutex_lock(&p->mutex);
52 + p->completed_jobs++;
53 + uv_cond_broadcast(&p->cond);
54 + uv_mutex_unlock(&p->mutex);
55 +}
56 +
57 +bool completion_is_done(struct completion *p)
58 +{
59 + bool ret;
60 + uv_mutex_lock(&p->mutex);
61 + ret = p->completed;
62 + uv_mutex_unlock(&p->mutex);
63 + return ret;
64 +}
libnetdata/completion/completion.h
+5
@@ -9,6 +9,7 @@ struct completion {
9 uv_mutex_t mutex;
10 uv_cond_t cond;
11 volatile unsigned completed;
12 + volatile unsigned completed_jobs;
13 };
14
15 void completion_init(struct completion *p);
@@ -19,4 +20,8 @@ void completion_wait_for(struct completion *p);
20
21 void completion_mark_complete(struct completion *p);
22
23 +unsigned completion_wait_for_a_job(struct completion *p, unsigned completed_jobs);
24 +void completion_mark_complete_a_job(struct completion *p);
25 +bool completion_is_done(struct completion *p);
26 +
27 #endif /* NETDATA_COMPLETION_H */
libnetdata/dictionary/dictionary.c
+28 -4
@@ -1234,11 +1234,29 @@ static inline size_t item_get_name_len(const DICTIONARY_ITEM *item) {
1234 return strlen(item->caller_name);
1235 }
1236
1237 +static ARAL dict_items_aral = {
1238 + .filename = NULL,
1239 + .cache_dir = NULL,
1240 + .use_mmap = false,
1241 + .initial_elements = 65536 / sizeof(DICTIONARY_ITEM),
1242 + .requested_element_size = sizeof(DICTIONARY_ITEM),
1243 +};
1244 +
1245 +static ARAL dict_shared_items_aral = {
1246 + .filename = NULL,
1247 + .cache_dir = NULL,
1248 + .use_mmap = false,
1249 + .initial_elements = 65536 / sizeof(DICTIONARY_ITEM_SHARED),
1250 + .requested_element_size = sizeof(DICTIONARY_ITEM_SHARED),
1251 +};
1252 +
1253 static DICTIONARY_ITEM *dict_item_create(DICTIONARY *dict __maybe_unused, size_t *allocated_bytes, DICTIONARY_ITEM *master_item) {
1254 DICTIONARY_ITEM *item;
1255
1256 size_t size = sizeof(DICTIONARY_ITEM);
1241 - item = callocz(1, size);
1257 +// item = callocz(1, size);
1258 + item = arrayalloc_mallocz(&dict_items_aral);
1259 + memset(item, 0, sizeof(DICTIONARY_ITEM));
1260
1261 #ifdef NETDATA_INTERNAL_CHECKS
1262 item->creator_pid = gettid();
@@ -1257,7 +1275,10 @@ static DICTIONARY_ITEM *dict_item_create(DICTIONARY *dict __maybe_unused, size_t
1275 }
1276 else {
1277 size = sizeof(DICTIONARY_ITEM_SHARED);
1260 - item->shared = callocz(1, size);
1278 + // item->shared = callocz(1, size);
1279 + item->shared = arrayalloc_mallocz(&dict_shared_items_aral);
1280 + memset(item->shared, 0, sizeof(DICTIONARY_ITEM_SHARED));
1281 +
1282 item->shared->links = 1;
1283 *allocated_bytes += size;
1284 }
@@ -1396,12 +1417,15 @@ static size_t dict_item_free_with_hooks(DICTIONARY *dict, DICTIONARY_ITEM *item)
1417 }
1418 value_size += item->shared->value_len;
1419
1399 - freez(item->shared);
1420 + // freez(item->shared);
1421 + arrayalloc_freez(&dict_shared_items_aral, item->shared);
1422 item->shared = NULL;
1423 item_size += sizeof(DICTIONARY_ITEM_SHARED);
1424 }
1425
1404 - freez(item);
1426 + // freez(item);
1427 + arrayalloc_freez(&dict_items_aral, item);
1428 +
1429 item_size += sizeof(DICTIONARY_ITEM);
1430
1431 DICTIONARY_STATS_MINUS_MEMORY(dict, key_size, item_size, value_size);
libnetdata/july/Makefile.am new
+8
@@ -0,0 +1,8 @@
1 +# SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +AUTOMAKE_OPTIONS = subdir-objects
4 +MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5 +
6 +dist_noinst_DATA = \
7 + README.md \
8 + $(NULL)
libnetdata/july/README.md new
+10
@@ -0,0 +1,10 @@
1 +<!--
2 +custom_edit_url: https://github.com/netdata/netdata/edit/master/libnetdata/july/README.md
3 +-->
4 +
5 +
6 +# July
7 +
8 +An interface similar to `Judy` that uses minimal allocations (that can be cached)
9 +for items that are mainly appended (just a few insertions in the middle)
10 +
libnetdata/july/july.c new
+447
@@ -0,0 +1,447 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "july.h"
4 +
5 +#define JULYL_MIN_ENTRIES 10
6 +
7 +struct JulyL_item {
8 + Word_t index;
9 + void *value;
10 +};
11 +
12 +struct JulyL {
13 + size_t entries;
14 + size_t used;
15 +
16 + // statistics
17 + size_t bytes;
18 + size_t bytes_moved;
19 + size_t reallocs;
20 +
21 + struct {
22 + struct JulyL *prev;
23 + struct JulyL *next;
24 + } cache;
25 +
26 + struct JulyL_item array[];
27 +};
28 +
29 +// ----------------------------------------------------------------------------
30 +// JulyL cache
31 +
32 +static struct {
33 + struct {
34 + SPINLOCK spinlock;
35 + struct JulyL *available_items;
36 + size_t available;
37 + } protected;
38 +
39 + struct {
40 + size_t bytes;
41 + size_t allocated;
42 + size_t bytes_moved;
43 + size_t reallocs;
44 + } atomics;
45 +} julyl_globals = {
46 + .protected = {
47 + .spinlock = NETDATA_SPINLOCK_INITIALIZER,
48 + .available_items = NULL,
49 + .available = 0,
50 + },
51 + .atomics = {
52 + .bytes = 0,
53 + .allocated = 0,
54 + .bytes_moved = 0,
55 + .reallocs = 0,
56 + },
57 +};
58 +
59 +void julyl_cleanup(void) {
60 + netdata_spinlock_lock(&julyl_globals.protected.spinlock);
61 +
62 + while(julyl_globals.protected.available_items && julyl_globals.protected.available > 10) {
63 + struct JulyL *item = julyl_globals.protected.available_items;
64 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(julyl_globals.protected.available_items, item, cache.prev, cache.next);
65 + size_t bytes = item->bytes;
66 + freez(item);
67 + julyl_globals.protected.available--;
68 + __atomic_sub_fetch(&julyl_globals.atomics.bytes, bytes, __ATOMIC_RELAXED);
69 + __atomic_sub_fetch(&julyl_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
70 + }
71 +
72 + netdata_spinlock_unlock(&julyl_globals.protected.spinlock);
73 +}
74 +
75 +struct JulyL *julyl_get(void) {
76 + struct JulyL *j;
77 +
78 + netdata_spinlock_lock(&julyl_globals.protected.spinlock);
79 +
80 + j = julyl_globals.protected.available_items;
81 + if(likely(j)) {
82 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(julyl_globals.protected.available_items, j, cache.prev, cache.next);
83 + julyl_globals.protected.available--;
84 + }
85 +
86 + netdata_spinlock_unlock(&julyl_globals.protected.spinlock);
87 +
88 + if(unlikely(!j)) {
89 + size_t bytes = sizeof(struct JulyL) + JULYL_MIN_ENTRIES * sizeof(struct JulyL_item);
90 + j = mallocz(bytes);
91 + j->bytes = bytes;
92 + j->entries = JULYL_MIN_ENTRIES;
93 + __atomic_add_fetch(&julyl_globals.atomics.bytes, bytes, __ATOMIC_RELAXED);
94 + __atomic_add_fetch(&julyl_globals.atomics.allocated, 1, __ATOMIC_RELAXED);
95 + }
96 +
97 + j->used = 0;
98 + j->bytes_moved = 0;
99 + j->reallocs = 0;
100 + j->cache.next = j->cache.prev = NULL;
101 + return j;
102 +}
103 +
104 +static void julyl_release(struct JulyL *j) {
105 + if(unlikely(!j)) return;
106 +
107 + __atomic_add_fetch(&julyl_globals.atomics.bytes_moved, j->bytes_moved, __ATOMIC_RELAXED);
108 + __atomic_add_fetch(&julyl_globals.atomics.reallocs, j->reallocs, __ATOMIC_RELAXED);
109 +
110 + netdata_spinlock_lock(&julyl_globals.protected.spinlock);
111 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(julyl_globals.protected.available_items, j, cache.prev, cache.next);
112 + julyl_globals.protected.available++;
113 + netdata_spinlock_unlock(&julyl_globals.protected.spinlock);
114 +}
115 +
116 +size_t julyl_cache_size(void) {
117 + return __atomic_load_n(&julyl_globals.atomics.bytes, __ATOMIC_RELAXED);
118 +}
119 +
120 +size_t julyl_bytes_moved(void) {
121 + return __atomic_load_n(&julyl_globals.atomics.bytes_moved, __ATOMIC_RELAXED);
122 +}
123 +
124 +// ----------------------------------------------------------------------------
125 +// JulyL
126 +
127 +size_t JulyLGet_binary_search_position_of_index(const struct JulyL *July, Word_t Index) {
128 + // return the position of the first item >= Index
129 +
130 + size_t left = 0;
131 + size_t right = July->used;
132 + while(left < right) {
133 + size_t middle = (left + right) >> 1;
134 +
135 + if(July->array[middle].index > Index)
136 + right = middle;
137 +
138 + else
139 + left = middle + 1;
140 + }
141 +
142 + internal_fatal(left > July->used, "JULY: invalid position returned");
143 +
144 + if(left > 0 && July->array[left - 1].index == Index)
145 + return left - 1;
146 +
147 + internal_fatal( (left < July->used && July->array[left].index < Index) ||
148 + (left > 0 && July->array[left - 1].index >= Index)
149 + , "JULY: wrong item returned");
150 +
151 + return left;
152 +}
153 +
154 +PPvoid_t JulyLGet(Pcvoid_t PArray, Word_t Index, PJError_t PJError __maybe_unused) {
155 + const struct JulyL *July = PArray;
156 + if(!July)
157 + return NULL;
158 +
159 + size_t pos = JulyLGet_binary_search_position_of_index(July, Index);
160 +
161 + if(unlikely(pos >= July->used || July->array[pos].index != Index))
162 + return NULL;
163 +
164 + return (PPvoid_t)&July->array[pos].value;
165 +}
166 +
167 +PPvoid_t JulyLIns(PPvoid_t PPArray, Word_t Index, PJError_t PJError __maybe_unused) {
168 + struct JulyL *July = *PPArray;
169 + if(unlikely(!July)) {
170 + July = julyl_get();
171 + July->used = 0;
172 + *PPArray = July;
173 + }
174 +
175 + size_t pos = JulyLGet_binary_search_position_of_index(July, Index);
176 +
177 + if((pos == July->used || July->array[pos].index != Index)) {
178 + // we have to add this entry
179 +
180 + if (unlikely(July->used == July->entries)) {
181 + // we have to expand the array
182 + size_t bytes = sizeof(struct JulyL) + July->entries * 2 * sizeof(struct JulyL_item);
183 + __atomic_add_fetch(&julyl_globals.atomics.bytes, bytes - July->bytes, __ATOMIC_RELAXED);
184 + July = reallocz(July, bytes);
185 + July->bytes = bytes;
186 + July->entries *= 2;
187 + July->reallocs++;
188 + *PPArray = July;
189 + }
190 +
191 + if (unlikely(pos != July->used)) {
192 + // we have to shift some members to make room
193 + size_t size = (July->used - pos) * sizeof(struct JulyL_item);
194 + memmove(&July->array[pos + 1], &July->array[pos], size);
195 + July->bytes_moved += size;
196 + }
197 +
198 + July->used++;
199 + July->array[pos].value = NULL;
200 + July->array[pos].index = Index;
201 + }
202 +
203 + return &July->array[pos].value;
204 +}
205 +
206 +PPvoid_t JulyLFirst(Pcvoid_t PArray, Word_t *Index, PJError_t PJError __maybe_unused) {
207 + const struct JulyL *July = PArray;
208 + if(!July)
209 + return NULL;
210 +
211 + size_t pos = JulyLGet_binary_search_position_of_index(July, *Index);
212 + // pos is >= Index
213 +
214 + if(unlikely(pos == July->used))
215 + return NULL;
216 +
217 + *Index = July->array[pos].index;
218 + return (PPvoid_t)&July->array[pos].value;
219 +}
220 +
221 +PPvoid_t JulyLNext(Pcvoid_t PArray, Word_t *Index, PJError_t PJError __maybe_unused) {
222 + const struct JulyL *July = PArray;
223 + if(!July)
224 + return NULL;
225 +
226 + size_t pos = JulyLGet_binary_search_position_of_index(July, *Index);
227 + // pos is >= Index
228 +
229 + if(unlikely(pos == July->used))
230 + return NULL;
231 +
232 + if(July->array[pos].index == *Index) {
233 + pos++;
234 +
235 + if(unlikely(pos == July->used))
236 + return NULL;
237 + }
238 +
239 + *Index = July->array[pos].index;
240 + return (PPvoid_t)&July->array[pos].value;
241 +}
242 +
243 +PPvoid_t JulyLLast(Pcvoid_t PArray, Word_t *Index, PJError_t PJError __maybe_unused) {
244 + const struct JulyL *July = PArray;
245 + if(!July)
246 + return NULL;
247 +
248 + size_t pos = JulyLGet_binary_search_position_of_index(July, *Index);
249 + // pos is >= Index
250 +
251 + if(pos > 0 && (pos == July->used || July->array[pos].index > *Index))
252 + pos--;
253 +
254 + if(unlikely(pos == 0 && July->array[0].index > *Index))
255 + return NULL;
256 +
257 + *Index = July->array[pos].index;
258 + return (PPvoid_t)&July->array[pos].value;
259 +}
260 +
261 +PPvoid_t JulyLPrev(Pcvoid_t PArray, Word_t *Index, PJError_t PJError __maybe_unused) {
262 + const struct JulyL *July = PArray;
263 + if(!July)
264 + return NULL;
265 +
266 + size_t pos = JulyLGet_binary_search_position_of_index(July, *Index);
267 + // pos is >= Index
268 +
269 + if(unlikely(pos == 0 || July->used == 0))
270 + return NULL;
271 +
272 + // get the previous one
273 + pos--;
274 +
275 + *Index = July->array[pos].index;
276 + return (PPvoid_t)&July->array[pos].value;
277 +}
278 +
279 +Word_t JulyLFreeArray(PPvoid_t PPArray, PJError_t PJError __maybe_unused) {
280 + struct JulyL *July = *PPArray;
281 + if(unlikely(!July))
282 + return 0;
283 +
284 + size_t bytes = July->bytes;
285 + julyl_release(July);
286 + *PPArray = NULL;
287 + return bytes;
288 +}
289 +
290 +// ----------------------------------------------------------------------------
291 +// unittest
292 +
293 +#define item_index(i) (((i) * 2) + 100)
294 +
295 +int julytest(void) {
296 + Word_t entries = 10000;
297 + Pvoid_t array = NULL;
298 +
299 + // test additions
300 + for(Word_t i = 0; i < entries ;i++) {
301 + Pvoid_t *PValue = JulyLIns(&array, item_index(i), PJE0);
302 + if(!PValue)
303 + fatal("JULY: cannot insert item %lu", item_index(i));
304 +
305 + *PValue = (void *)(item_index(i));
306 + }
307 +
308 + // test successful finds
309 + for(Word_t i = 0; i < entries ;i++) {
310 + Pvoid_t *PValue = JulyLGet(array, item_index(i), PJE0);
311 + if(!PValue)
312 + fatal("JULY: cannot find item %lu", item_index(i));
313 +
314 + if(*PValue != (void *)(item_index(i)))
315 + fatal("JULY: item %lu has the value %lu", item_index(i), (unsigned long)(*PValue));
316 + }
317 +
318 + // test finding the first item
319 + for(Word_t i = 0; i < entries ;i++) {
320 + Word_t index = item_index(i);
321 + Pvoid_t *PValue = JulyLFirst(array, &index, PJE0);
322 + if(!PValue)
323 + fatal("JULY: cannot find first item %lu", item_index(i));
324 +
325 + if(*PValue != (void *)(item_index(i)))
326 + fatal("JULY: item %lu has the value %lu", item_index(i), (unsigned long)(*PValue));
327 +
328 + if(index != item_index(i))
329 + fatal("JULY: item %lu has index %lu", item_index(i), index);
330 + }
331 +
332 + // test finding the next item
333 + for(Word_t i = 0; i < entries - 1 ;i++) {
334 + Word_t index = item_index(i);
335 + Pvoid_t *PValue = JulyLNext(array, &index, PJE0);
336 + if(!PValue)
337 + fatal("JULY: cannot find next item %lu", item_index(i));
338 +
339 + if(*PValue != (void *)(item_index(i + 1)))
340 + fatal("JULY: item %lu next has the value %lu", item_index(i), (unsigned long)(*PValue));
341 +
342 + if(index != item_index(i + 1))
343 + fatal("JULY: item %lu next has index %lu", item_index(i), index);
344 + }
345 +
346 + // test finding the last item
347 + for(Word_t i = 0; i < entries ;i++) {
348 + Word_t index = item_index(i);
349 + Pvoid_t *PValue = JulyLLast(array, &index, PJE0);
350 + if(!PValue)
351 + fatal("JULY: cannot find last item %lu", item_index(i));
352 +
353 + if(*PValue != (void *)(item_index(i)))
354 + fatal("JULY: item %lu has the value %lu", item_index(i), (unsigned long)(*PValue));
355 +
356 + if(index != item_index(i))
357 + fatal("JULY: item %lu has index %lu", item_index(i), index);
358 + }
359 +
360 + // test finding the prev item
361 + for(Word_t i = 1; i < entries ;i++) {
362 + Word_t index = item_index(i);
363 + Pvoid_t *PValue = JulyLPrev(array, &index, PJE0);
364 + if(!PValue)
365 + fatal("JULY: cannot find prev item %lu", item_index(i));
366 +
367 + if(*PValue != (void *)(item_index(i - 1)))
368 + fatal("JULY: item %lu prev has the value %lu", item_index(i), (unsigned long)(*PValue));
369 +
370 + if(index != item_index(i - 1))
371 + fatal("JULY: item %lu prev has index %lu", item_index(i), index);
372 + }
373 +
374 + // test full traversal forward
375 + {
376 + Word_t i = 0;
377 + Word_t index = 0;
378 + bool first = true;
379 + Pvoid_t *PValue;
380 + while((PValue = JulyLFirstThenNext(array, &index, &first))) {
381 + if(*PValue != (void *)(item_index(i)))
382 + fatal("JULY: item %lu traversal has the value %lu", item_index(i), (unsigned long)(*PValue));
383 +
384 + if(index != item_index(i))
385 + fatal("JULY: item %lu traversal has index %lu", item_index(i), index);
386 +
387 + i++;
388 + }
389 +
390 + if(i != entries)
391 + fatal("JULY: expected to forward traverse %lu entries, but traversed %lu", entries, i);
392 + }
393 +
394 + // test full traversal backward
395 + {
396 + Word_t i = 0;
397 + Word_t index = (Word_t)(-1);
398 + bool first = true;
399 + Pvoid_t *PValue;
400 + while((PValue = JulyLLastThenPrev(array, &index, &first))) {
401 + if(*PValue != (void *)(item_index(entries - i - 1)))
402 + fatal("JULY: item %lu traversal has the value %lu", item_index(i), (unsigned long)(*PValue));
403 +
404 + if(index != item_index(entries - i - 1))
405 + fatal("JULY: item %lu traversal has index %lu", item_index(i), index);
406 +
407 + i++;
408 + }
409 +
410 + if(i != entries)
411 + fatal("JULY: expected to back traverse %lu entries, but traversed %lu", entries, i);
412 + }
413 +
414 + // test finding non-existing first item
415 + for(Word_t i = 0; i < entries ;i++) {
416 + Word_t index = item_index(i) - 1;
417 + Pvoid_t *PValue = JulyLFirst(array, &index, PJE0);
418 + if(!PValue)
419 + fatal("JULY: cannot find first item %lu", item_index(i) - 1);
420 +
421 + if(*PValue != (void *)(item_index(i)))
422 + fatal("JULY: item %lu has the value %lu", item_index(i), (unsigned long)(*PValue));
423 +
424 + if(index != item_index(i))
425 + fatal("JULY: item %lu has index %lu", item_index(i), index);
426 + }
427 +
428 + // test finding non-existing last item
429 + for(Word_t i = 0; i < entries ;i++) {
430 + Word_t index = item_index(i) + 1;
431 + Pvoid_t *PValue = JulyLLast(array, &index, PJE0);
432 + if(!PValue)
433 + fatal("JULY: cannot find last item %lu", item_index(i) + 1);
434 +
435 + if(*PValue != (void *)(item_index(i)))
436 + fatal("JULY: item %lu has the value %lu", item_index(i), (unsigned long)(*PValue));
437 +
438 + if(index != item_index(i))
439 + fatal("JULY: item %lu has index %lu", item_index(i), index);
440 + }
441 +
442 + JulyLFreeArray(&array, PJE0);
443 +
444 + return 0;
445 +}
446 +
447 +
libnetdata/july/july.h new
+40
@@ -0,0 +1,40 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_JULY_H
4 +#define NETDATA_JULY_H 1
5 +
6 +#include "../libnetdata.h"
7 +
8 +// #define PDC_USE_JULYL 1
9 +
10 +PPvoid_t JulyLGet(Pcvoid_t PArray, Word_t Index, PJError_t PJError);
11 +PPvoid_t JulyLIns(PPvoid_t PPArray, Word_t Index, PJError_t PJError);
12 +PPvoid_t JulyLFirst(Pcvoid_t PArray, Word_t *Index, PJError_t PJError);
13 +PPvoid_t JulyLNext(Pcvoid_t PArray, Word_t *Index, PJError_t PJError);
14 +PPvoid_t JulyLLast(Pcvoid_t PArray, Word_t *Index, PJError_t PJError);
15 +PPvoid_t JulyLPrev(Pcvoid_t PArray, Word_t *Index, PJError_t PJError);
16 +Word_t JulyLFreeArray(PPvoid_t PPArray, PJError_t PJError);
17 +
18 +static inline PPvoid_t JulyLFirstThenNext(Pcvoid_t PArray, Word_t * PIndex, bool *first) {
19 + if(unlikely(*first)) {
20 + *first = false;
21 + return JulyLFirst(PArray, PIndex, PJE0);
22 + }
23 +
24 + return JulyLNext(PArray, PIndex, PJE0);
25 +}
26 +
27 +static inline PPvoid_t JulyLLastThenPrev(Pcvoid_t PArray, Word_t * PIndex, bool *first) {
28 + if(unlikely(*first)) {
29 + *first = false;
30 + return JulyLLast(PArray, PIndex, PJE0);
31 + }
32 +
33 + return JulyLPrev(PArray, PIndex, PJE0);
34 +}
35 +
36 +void julyl_cleanup(void);
37 +size_t julyl_cache_size(void);
38 +size_t julyl_bytes_moved(void);
39 +
40 +#endif // NETDATA_JULY_H
libnetdata/libnetdata.c
+9 -16
@@ -1176,7 +1176,7 @@ static int memory_file_open(const char *filename, size_t size) {
1176 return fd;
1177 }
1178
1179 -static inline int madvise_sequential(void *mem, size_t len) {
1179 +inline int madvise_sequential(void *mem, size_t len) {
1180 static int logger = 1;
1181 int ret = madvise(mem, len, MADV_SEQUENTIAL);
1182
@@ -1184,7 +1184,7 @@ static inline int madvise_sequential(void *mem, size_t len) {
1184 return ret;
1185 }
1186
1187 -static inline int madvise_dontfork(void *mem, size_t len) {
1187 +inline int madvise_dontfork(void *mem, size_t len) {
1188 static int logger = 1;
1189 int ret = madvise(mem, len, MADV_DONTFORK);
1190
@@ -1192,7 +1192,7 @@ static inline int madvise_dontfork(void *mem, size_t len) {
1192 return ret;
1193 }
1194
1195 -static inline int madvise_willneed(void *mem, size_t len) {
1195 +inline int madvise_willneed(void *mem, size_t len) {
1196 static int logger = 1;
1197 int ret = madvise(mem, len, MADV_WILLNEED);
1198
@@ -1200,24 +1200,19 @@ static inline int madvise_willneed(void *mem, size_t len) {
1200 return ret;
1201 }
1202
1203 +inline int madvise_dontdump(void *mem __maybe_unused, size_t len __maybe_unused) {
1204 #if __linux__
1204 -static inline int madvise_dontdump(void *mem, size_t len) {
1205 static int logger = 1;
1206 int ret = madvise(mem, len, MADV_DONTDUMP);
1207
1208 if (ret != 0 && logger-- > 0) error("madvise(MADV_DONTDUMP) failed.");
1209 return ret;
1210 -}
1210 #else
1212 -static inline int madvise_dontdump(void *mem, size_t len) {
1213 - UNUSED(mem);
1214 - UNUSED(len);
1215 -
1211 return 0;
1217 -}
1212 #endif
1213 +}
1214
1220 -static inline int madvise_mergeable(void *mem, size_t len) {
1215 +inline int madvise_mergeable(void *mem __maybe_unused, size_t len __maybe_unused) {
1216 #ifdef MADV_MERGEABLE
1217 static int logger = 1;
1218 int ret = madvise(mem, len, MADV_MERGEABLE);
@@ -1225,14 +1220,12 @@ static inline int madvise_mergeable(void *mem, size_t len) {
1220 if (ret != 0 && logger-- > 0) error("madvise(MADV_MERGEABLE) failed.");
1221 return ret;
1222 #else
1228 - UNUSED(mem);
1229 - UNUSED(len);
1230 -
1223 return 0;
1224 #endif
1225 }
1226
1235 -void *netdata_mmap(const char *filename, size_t size, int flags, int ksm) {
1227 +void *netdata_mmap(const char *filename, size_t size, int flags, int ksm, bool read_only)
1228 +{
1229 // info("netdata_mmap('%s', %zu", filename, size);
1230
1231 // MAP_SHARED is used in memory mode map
@@ -1271,7 +1264,7 @@ void *netdata_mmap(const char *filename, size_t size, int flags, int ksm) {
1264 fd_for_mmap = -1;
1265 }
1266
1274 - mem = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, fd_for_mmap, 0);
1267 + mem = mmap(NULL, size, read_only ? PROT_READ : PROT_READ | PROT_WRITE, flags, fd_for_mmap, 0);
1268 if (mem != MAP_FAILED) {
1269
1270 #ifdef NETDATA_TRACE_ALLOCATIONS
libnetdata/libnetdata.h
+78 -1
@@ -301,6 +301,12 @@ char *mystrsep(char **ptr, char *s);
301 char *trim(char *s); // remove leading and trailing spaces; may return NULL
302 char *trim_all(char *buffer); // like trim(), but also remove duplicate spaces inside the string; may return NULL
303
304 +int madvise_sequential(void *mem, size_t len);
305 +int madvise_dontfork(void *mem, size_t len);
306 +int madvise_willneed(void *mem, size_t len);
307 +int madvise_dontdump(void *mem, size_t len);
308 +int madvise_mergeable(void *mem, size_t len);
309 +
310 int vsnprintfz(char *dst, size_t n, const char *fmt, va_list args);
311 int snprintfz(char *dst, size_t n, const char *fmt, ...) PRINTFLIKE(3, 4);
312
@@ -335,7 +341,7 @@ void posix_memfree(void *ptr);
341 void json_escape_string(char *dst, const char *src, size_t size);
342 void json_fix_string(char *s);
343
338 -void *netdata_mmap(const char *filename, size_t size, int flags, int ksm);
344 +void *netdata_mmap(const char *filename, size_t size, int flags, int ksm, bool read_only);
345 int netdata_munmap(void *ptr, size_t size);
346 int memory_file_save(const char *filename, void *mem, size_t size);
347
@@ -422,6 +428,7 @@ void netdata_cleanup_and_exit(int ret) NORETURN;
428 void send_statistics(const char *action, const char *action_result, const char *action_data);
429 extern char *netdata_configured_host_prefix;
430 #include "libjudy/src/Judy.h"
431 +#include "july/july.h"
432 #include "os.h"
433 #include "storage_number/storage_number.h"
434 #include "threads/threads.h"
@@ -500,6 +507,76 @@ struct malloc_trace {
507 };
508 #endif // NETDATA_TRACE_ALLOCATIONS
509
510 +static inline PPvoid_t JudyLFirstThenNext(Pcvoid_t PArray, Word_t * PIndex, bool *first) {
511 + if(unlikely(*first)) {
512 + *first = false;
513 + return JudyLFirst(PArray, PIndex, PJE0);
514 + }
515 +
516 + return JudyLNext(PArray, PIndex, PJE0);
517 +}
518 +
519 +static inline PPvoid_t JudyLLastThenPrev(Pcvoid_t PArray, Word_t * PIndex, bool *first) {
520 + if(unlikely(*first)) {
521 + *first = false;
522 + return JudyLLast(PArray, PIndex, PJE0);
523 + }
524 +
525 + return JudyLPrev(PArray, PIndex, PJE0);
526 +}
527 +
528 +static inline size_t indexing_partition_old(Word_t ptr, Word_t modulo) {
529 + size_t total = 0;
530 +
531 + total += (ptr & 0xff) >> 0;
532 + total += (ptr & 0xff00) >> 8;
533 + total += (ptr & 0xff0000) >> 16;
534 + total += (ptr & 0xff000000) >> 24;
535 +
536 + if(sizeof(Word_t) > 4) {
537 + total += (ptr & 0xff00000000) >> 32;
538 + total += (ptr & 0xff0000000000) >> 40;
539 + total += (ptr & 0xff000000000000) >> 48;
540 + total += (ptr & 0xff00000000000000) >> 56;
541 + }
542 +
543 + return (total % modulo);
544 +}
545 +
546 +static uint32_t murmur32(uint32_t h) __attribute__((const));
547 +static inline uint32_t murmur32(uint32_t h) {
548 + h ^= h >> 16;
549 + h *= 0x85ebca6b;
550 + h ^= h >> 13;
551 + h *= 0xc2b2ae35;
552 + h ^= h >> 16;
553 +
554 + return h;
555 +}
556 +
557 +static uint64_t murmur64(uint64_t h) __attribute__((const));
558 +static inline uint64_t murmur64(uint64_t k) {
559 + k ^= k >> 33;
560 + k *= 0xff51afd7ed558ccdUL;
561 + k ^= k >> 33;
562 + k *= 0xc4ceb9fe1a85ec53UL;
563 + k ^= k >> 33;
564 +
565 + return k;
566 +}
567 +
568 +static inline size_t indexing_partition(Word_t ptr, Word_t modulo) __attribute__((const));
569 +static inline size_t indexing_partition(Word_t ptr, Word_t modulo) {
570 + if(sizeof(Word_t) == 8) {
571 + uint64_t hash = murmur64(ptr);
572 + return hash % modulo;
573 + }
574 + else {
575 + uint32_t hash = murmur32(ptr);
576 + return hash % modulo;
577 + }
578 +}
579 +
580 # ifdef __cplusplus
581 }
582 # endif
libnetdata/locks/locks.c
+91 -348
@@ -15,8 +15,6 @@
15 #ifndef NETDATA_THREAD_LOCKS_ARRAY_SIZE
16 #define NETDATA_THREAD_LOCKS_ARRAY_SIZE 10
17 #endif
18 -static __thread netdata_rwlock_t *netdata_thread_locks[NETDATA_THREAD_LOCKS_ARRAY_SIZE];
19 -
18
19 #endif // NETDATA_TRACE_RWLOCKS
20
@@ -120,8 +118,6 @@ int __netdata_mutex_unlock(netdata_mutex_t *mutex) {
118
119 #ifdef NETDATA_TRACE_RWLOCKS
120
123 -#warning NETDATA_TRACE_RWLOCKS ENABLED - EXPECT A LOT OF OUTPUT
124 -
121 int netdata_mutex_init_debug(const char *file __maybe_unused, const char *function __maybe_unused,
122 const unsigned long line __maybe_unused, netdata_mutex_t *mutex) {
123 debug(D_LOCKS, "MUTEX_LOCK: netdata_mutex_init(%p) from %lu@%s, %s()", mutex, line, file, function);
@@ -289,6 +285,10 @@ void netdata_spinlock_init(SPINLOCK *spinlock) {
285 void netdata_spinlock_lock(SPINLOCK *spinlock) {
286 static const struct timespec ns = { .tv_sec = 0, .tv_nsec = 1 };
287
288 +#ifdef NETDATA_INTERNAL_CHECKS
289 + size_t spins = 0;
290 +#endif
291 +
292 netdata_thread_disable_cancelability();
293
294 for(int i = 1;
@@ -296,254 +296,104 @@ void netdata_spinlock_lock(SPINLOCK *spinlock) {
296 __atomic_test_and_set(&spinlock->locked, __ATOMIC_ACQUIRE)
297 ; i++
298 ) {
299 +
300 +#ifdef NETDATA_INTERNAL_CHECKS
301 + spins++;
302 +#endif
303 if(unlikely(i == 8)) {
304 i = 0;
305 nanosleep(&ns, NULL);
306 }
307 }
308 +
309 // we have the lock
310 +
311 +#ifdef NETDATA_INTERNAL_CHECKS
312 + spinlock->spins += spins;
313 + spinlock->locker_pid = gettid();
314 +#endif
315 }
316
317 void netdata_spinlock_unlock(SPINLOCK *spinlock) {
318 +#ifdef NETDATA_INTERNAL_CHECKS
319 + spinlock->locker_pid = 0;
320 +#endif
321 __atomic_clear(&spinlock->locked, __ATOMIC_RELEASE);
322 netdata_thread_enable_cancelability();
323 }
324
312 -#ifdef NETDATA_TRACE_RWLOCKS
325 +bool netdata_spinlock_trylock(SPINLOCK *spinlock) {
326 + netdata_thread_disable_cancelability();
327
314 -// ----------------------------------------------------------------------------
315 -// lockers list
328 + if(!__atomic_load_n(&spinlock->locked, __ATOMIC_RELAXED) &&
329 + !__atomic_test_and_set(&spinlock->locked, __ATOMIC_ACQUIRE))
330 + // we got the lock
331 + return true;
332
317 -void not_supported_by_posix_rwlocks(const char *file, const char *function, const unsigned long line, netdata_rwlock_t *rwlock, char locktype, const char *reason) {
318 - __netdata_mutex_lock(&rwlock->lockers_mutex);
319 - fprintf(stderr,
320 - "RW_LOCK FATAL ON LOCK %p: %d '%s' (function %s() %lu@%s) attempts to acquire a '%c' lock, but it is not supported by POSIX because: %s. At this attempt, the task is holding %zu rwlocks and %zu mutexes. There are %zu readers and %zu writers holding this lock:\n",
321 - rwlock,
322 - gettid(), netdata_thread_tag(),
323 - function, line, file,
324 - locktype,
325 - reason,
326 - netdata_locks_acquired_rwlocks, netdata_locks_acquired_mutexes,
327 - rwlock->readers, rwlock->writers);
328 -
329 - int i;
330 - usec_t now = now_monotonic_high_precision_usec();
331 - netdata_rwlock_locker *p;
332 - for(i = 1, p = rwlock->lockers; p ;p = p->next, i++) {
333 - fprintf(stderr,
334 - " => %i: RW_LOCK %p: process %d '%s' (function %s() %lu@%s) is having %zu '%c' lock for %llu usec.\n",
335 - i, rwlock,
336 - p->pid, p->tag,
337 - p->function, p->line, p->file,
338 - p->callers, p->lock,
339 - (now - p->start_s));
340 - }
341 - __netdata_mutex_unlock(&rwlock->lockers_mutex);
333 + // we didn't get the lock
334 + return false;
335 }
336
344 -static void log_rwlock_lockers(const char *file, const char *function, const unsigned long line, netdata_rwlock_t *rwlock, const char *reason, char locktype) {
345 -
346 - // this function can only be used by one thread at a time
347 - // because otherwise, the threads may deadlock waiting for each other
348 - static netdata_mutex_t log_lockers_mutex = NETDATA_MUTEX_INITIALIZER;
349 - __netdata_mutex_lock(&log_lockers_mutex);
350 -
351 - // now work on this locker
352 - __netdata_mutex_lock(&rwlock->lockers_mutex);
353 - fprintf(stderr,
354 - "RW_LOCK ON LOCK %p: %d '%s' (function %s() %lu@%s) %s a '%c' lock (while holding %zu rwlocks and %zu mutexes). There are %zu readers and %zu writers holding this lock:\n",
355 - rwlock,
356 - gettid(), netdata_thread_tag(),
357 - function, line, file,
358 - reason, locktype,
359 - netdata_locks_acquired_rwlocks, netdata_locks_acquired_mutexes,
360 - rwlock->readers, rwlock->writers);
361 -
362 - int i;
363 - usec_t now = now_monotonic_high_precision_usec();
364 - netdata_rwlock_locker *p;
365 - for(i = 1, p = rwlock->lockers; p ;p = p->next, i++) {
366 - fprintf(stderr,
367 - " => %i: RW_LOCK %p: process %d '%s' (function %s() %lu@%s) is having %zu '%c' lock for %llu usec.\n",
368 - i, rwlock,
369 - p->pid, p->tag,
370 - p->function, p->line, p->file,
371 - p->callers, p->lock,
372 - (now - p->start_s));
373 -
374 - if(p->all_caller_locks) {
375 - // find the lock in the netdata_thread_locks[]
376 - // and remove it
377 - int k;
378 - for(k = 0; k < NETDATA_THREAD_LOCKS_ARRAY_SIZE ;k++) {
379 - if (p->all_caller_locks[k] && p->all_caller_locks[k] != rwlock) {
380 -
381 - // lock the other lock lockers list
382 - __netdata_mutex_lock(&p->all_caller_locks[k]->lockers_mutex);
383 -
384 - // print the list of lockers of the other lock
385 - netdata_rwlock_locker *r;
386 - int j;
387 - for(j = 1, r = p->all_caller_locks[k]->lockers; r ;r = r->next, j++) {
388 - fprintf(
389 - stderr,
390 - " ~~~> %i: RW_LOCK %p: process %d '%s' (function %s() %lu@%s) is having %zu '%c' lock for %llu usec.\n",
391 - j,
392 - p->all_caller_locks[k],
393 - r->pid,
394 - r->tag,
395 - r->function,
396 - r->line,
397 - r->file,
398 - r->callers,
399 - r->lock,
400 - (now - r->start_s));
401 - }
402 -
403 - // unlock the other lock lockers list
404 - __netdata_mutex_unlock(&p->all_caller_locks[k]->lockers_mutex);
405 - }
406 - }
407 - }
337 +#ifdef NETDATA_TRACE_RWLOCKS
338
409 - }
410 - __netdata_mutex_unlock(&rwlock->lockers_mutex);
339 +// ----------------------------------------------------------------------------
340 +// lockers list
341
412 - // unlock this function for other threads
413 - __netdata_mutex_unlock(&log_lockers_mutex);
414 -}
415 -
416 -static netdata_rwlock_locker *add_rwlock_locker(const char *file, const char *function, const unsigned long line, netdata_rwlock_t *rwlock, char lock_type) {
417 - netdata_rwlock_locker *p = mallocz(sizeof(netdata_rwlock_locker));
418 - p->pid = gettid();
419 - p->tag = netdata_thread_tag();
420 - p->lock = lock_type;
421 - p->file = file;
422 - p->function = function;
423 - p->line = line;
424 - p->callers = 1;
425 - p->all_caller_locks = netdata_thread_locks;
426 - p->start_s = now_monotonic_high_precision_usec();
427 -
428 - // find a slot in the netdata_thread_locks[]
429 - int i;
430 - for(i = 0; i < NETDATA_THREAD_LOCKS_ARRAY_SIZE ;i++) {
431 - if (!netdata_thread_locks[i]) {
432 - netdata_thread_locks[i] = rwlock;
433 - break;
434 - }
435 - }
342 +static netdata_rwlock_locker *find_rwlock_locker(const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
343 + pid_t pid = gettid();
344 + netdata_rwlock_locker *locker = NULL;
345
346 __netdata_mutex_lock(&rwlock->lockers_mutex);
438 - p->next = rwlock->lockers;
439 - rwlock->lockers = p;
440 - if(lock_type == 'R') rwlock->readers++;
441 - if(lock_type == 'W') rwlock->writers++;
347 + Pvoid_t *PValue = JudyLGet(rwlock->lockers_pid_JudyL, pid, PJE0);
348 + if(PValue && *PValue)
349 + locker = *PValue;
350 __netdata_mutex_unlock(&rwlock->lockers_mutex);
351
444 - return p;
352 + return locker;
353 }
354
447 -static void remove_rwlock_locker(const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock, netdata_rwlock_locker *locker) {
448 - usec_t end_s = now_monotonic_high_precision_usec();
355 +static netdata_rwlock_locker *add_rwlock_locker(const char *file, const char *function, const unsigned long line, netdata_rwlock_t *rwlock, LOCKER_REQUEST lock_type) {
356 + netdata_rwlock_locker *locker;
357
450 - if(locker->callers == 0)
451 - fprintf(stderr,
452 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) callers should be positive but it is zero\n",
453 - rwlock,
454 - locker->pid, locker->tag,
455 - locker->function, locker->line, locker->file);
456 -
457 - if(locker->callers > 1 && locker->lock != 'R')
458 - fprintf(stderr,
459 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) only 'R' locks support multiple holders, but here we have %zu callers holding a '%c' lock.\n",
460 - rwlock,
461 - locker->pid, locker->tag,
462 - locker->function, locker->line, locker->file,
463 - locker->callers, locker->lock);
464 -
465 - __netdata_mutex_lock(&rwlock->lockers_mutex);
466 - locker->callers--;
467 -
468 - if(!locker->callers) {
469 - int doit = 0;
470 -
471 - if (rwlock->lockers == locker) {
472 - rwlock->lockers = locker->next;
473 - doit = 1;
474 - } else {
475 - netdata_rwlock_locker *p;
476 - for (p = rwlock->lockers; p && p->next != locker; p = p->next)
477 - ;
478 - if (p && p->next == locker) {
479 - p->next = locker->next;
480 - doit = 1;
481 - }
482 - }
483 - if(doit) {
484 - if(locker->lock == 'R') rwlock->readers--;
485 - if(locker->lock == 'W') rwlock->writers--;
486 - }
358 + locker = find_rwlock_locker(file, function, line, rwlock);
359 + if(locker) {
360 + locker->lock |= lock_type;
361 + locker->refcount++;
362 + }
363 + else {
364 + locker = mallocz(sizeof(netdata_rwlock_locker));
365 + locker->pid = gettid();
366 + locker->tag = netdata_thread_tag();
367 + locker->refcount = 1;
368 + locker->lock = lock_type;
369 + locker->got_it = false;
370 + locker->file = file;
371 + locker->function = function;
372 + locker->line = line;
373
488 - if(!doit) {
489 - fprintf(stderr,
490 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) with %zu x '%c' lock is not found.\n",
491 - rwlock,
492 - locker->pid, locker->tag,
493 - locker->function, locker->line, locker->file,
494 - locker->callers, locker->lock);
495 - }
496 - else {
497 - // find the lock in the netdata_thread_locks[]
498 - // and remove it
499 - int i;
500 - for(i = 0; i < NETDATA_THREAD_LOCKS_ARRAY_SIZE ;i++) {
501 - if (netdata_thread_locks[i] == rwlock)
502 - netdata_thread_locks[i] = NULL;
503 - }
504 -
505 - if(end_s - locker->start_s >= NETDATA_TRACE_RWLOCKS_HOLD_TIME_TO_IGNORE_USEC)
506 - fprintf(stderr,
507 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) holded a '%c' for %llu usec.\n",
508 - rwlock,
509 - locker->pid, locker->tag,
510 - locker->function, locker->line, locker->file,
511 - locker->lock, end_s - locker->start_s);
512 -
513 - freez(locker);
514 - }
374 + __netdata_mutex_lock(&rwlock->lockers_mutex);
375 + DOUBLE_LINKED_LIST_APPEND_UNSAFE(rwlock->lockers, locker, prev, next);
376 + Pvoid_t *PValue = JudyLIns(&rwlock->lockers_pid_JudyL, locker->pid, PJE0);
377 + *PValue = locker;
378 + if (lock_type == RWLOCK_REQUEST_READ || lock_type == RWLOCK_REQUEST_TRYREAD) rwlock->readers++;
379 + if (lock_type == RWLOCK_REQUEST_WRITE || lock_type == RWLOCK_REQUEST_TRYWRITE) rwlock->writers++;
380 + __netdata_mutex_unlock(&rwlock->lockers_mutex);
381 }
382
517 - __netdata_mutex_unlock(&rwlock->lockers_mutex);
383 + return locker;
384 }
385
520 -static netdata_rwlock_locker *find_rwlock_locker(const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
521 - pid_t pid = gettid();
522 - netdata_rwlock_locker *p;
523 -
386 +static void remove_rwlock_locker(const char *file __maybe_unused, const char *function __maybe_unused, const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock, netdata_rwlock_locker *locker) {
387 __netdata_mutex_lock(&rwlock->lockers_mutex);
525 - for(p = rwlock->lockers; p ;p = p->next) {
526 - if(p->pid == pid) break;
388 + locker->refcount--;
389 + if(!locker->refcount) {
390 + DOUBLE_LINKED_LIST_REMOVE_UNSAFE(rwlock->lockers, locker, prev, next);
391 + JudyLDel(&rwlock->lockers_pid_JudyL, locker->pid, PJE0);
392 + if (locker->lock == RWLOCK_REQUEST_READ || locker->lock == RWLOCK_REQUEST_TRYREAD) rwlock->readers--;
393 + else if (locker->lock == RWLOCK_REQUEST_WRITE || locker->lock == RWLOCK_REQUEST_TRYWRITE) rwlock->writers--;
394 + freez(locker);
395 }
396 __netdata_mutex_unlock(&rwlock->lockers_mutex);
529 -
530 - return p;
531 -}
532 -
533 -static netdata_rwlock_locker *update_or_add_rwlock_locker(const char *file, const char *function, const unsigned long line, netdata_rwlock_t *rwlock, netdata_rwlock_locker *locker, char locktype) {
534 - if(!locker) {
535 - return add_rwlock_locker(file, function, line, rwlock, locktype);
536 - }
537 - else if(locker->lock == 'R' && locktype == 'R') {
538 - __netdata_mutex_lock(&rwlock->lockers_mutex);
539 - locker->callers++;
540 - __netdata_mutex_unlock(&rwlock->lockers_mutex);
541 - return locker;
542 - }
543 - else {
544 - not_supported_by_posix_rwlocks(file, function, line, rwlock, locktype, "DEADLOCK - WANTS TO CHANGE LOCK TYPE BUT ALREADY HAS THIS LOCKED");
545 - return locker;
546 - }
397 }
398
399 // ----------------------------------------------------------------------------
@@ -551,84 +401,41 @@ static netdata_rwlock_locker *update_or_add_rwlock_locker(const char *file, cons
401
402 int netdata_rwlock_destroy_debug(const char *file __maybe_unused, const char *function __maybe_unused,
403 const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
554 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_destroy(%p) from %lu@%s, %s()", rwlock, line, file, function);
555 -
556 - if(rwlock->readers)
557 - error("RW_LOCK: destroying a rwlock with %zu readers in it", rwlock->readers);
558 - if(rwlock->writers)
559 - error("RW_LOCK: destroying a rwlock with %zu writers in it", rwlock->writers);
404
405 int ret = __netdata_rwlock_destroy(rwlock);
406 if(!ret) {
407 while (rwlock->lockers)
408 remove_rwlock_locker(file, function, line, rwlock, rwlock->lockers);
565 -
566 - if (rwlock->readers)
567 - error("RW_LOCK: internal error - empty rwlock with %zu readers in it", rwlock->readers);
568 - if (rwlock->writers)
569 - error("RW_LOCK: internal error - empty rwlock with %zu writers in it", rwlock->writers);
409 }
410
572 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_destroy(%p) = %d, from %lu@%s, %s()", rwlock, ret, line, file, function);
573 -
411 return ret;
412 }
413
414 int netdata_rwlock_init_debug(const char *file __maybe_unused, const char *function __maybe_unused,
415 const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
579 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_init(%p) from %lu@%s, %s()", rwlock, line, file, function);
416
417 int ret = __netdata_rwlock_init(rwlock);
418 if(!ret) {
419 __netdata_mutex_init(&rwlock->lockers_mutex);
420 + rwlock->lockers_pid_JudyL = NULL;
421 rwlock->lockers = NULL;
422 rwlock->readers = 0;
423 rwlock->writers = 0;
424 }
425
589 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_init(%p) = %d, from %lu@%s, %s()", rwlock, ret, line, file, function);
590 -
426 return ret;
427 }
428
429 int netdata_rwlock_rdlock_debug(const char *file __maybe_unused, const char *function __maybe_unused,
430 const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
431
597 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_rdlock(%p) from %lu@%s, %s()", rwlock, line, file, function);
598 -
599 - netdata_rwlock_locker *locker = find_rwlock_locker(file, function, line, rwlock);
600 -
601 -#ifdef NETDATA_TRACE_RWLOCKS_LOG_NESTED
602 - if(locker && locker->lock == 'R') {
603 - log_rwlock_lockers(file, function, line, rwlock, "NESTED READ LOCK REQUEST", 'R');
604 - }
605 -#endif // NETDATA_TRACE_RWLOCKS_LOG_NESTED
432 + netdata_rwlock_locker *locker = add_rwlock_locker(file, function, line, rwlock, RWLOCK_REQUEST_READ);
433
607 - int log = 0;
608 - if(rwlock->writers) {
609 - log_rwlock_lockers(file, function, line, rwlock, "WANTS", 'R');
610 - log = 1;
611 - }
612 -
613 - usec_t start_s = now_monotonic_high_precision_usec();
434 int ret = __netdata_rwlock_rdlock(rwlock);
615 - usec_t end_s = now_monotonic_high_precision_usec();
616 -
617 - if(!ret) {
618 - locker = update_or_add_rwlock_locker(file, function, line, rwlock, locker, 'R');
619 - if(log) log_rwlock_lockers(file, function, line, rwlock, "GOT", 'R');
620 -
621 - }
622 -
623 - if(end_s - start_s >= NETDATA_TRACE_RWLOCKS_WAIT_TIME_TO_IGNORE_USEC)
624 - fprintf(stderr,
625 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) WAITED for a READ lock for %llu usec.\n",
626 - rwlock,
627 - gettid(), netdata_thread_tag(),
628 - function, line, file,
629 - end_s - start_s);
630 -
631 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_rdlock(%p) = %d in %llu usec, from %lu@%s, %s()", rwlock, ret, end_s - start_s, line, file, function);
435 + if(!ret)
436 + locker->got_it = true;
437 + else
438 + remove_rwlock_locker(file, function, line, rwlock, locker);
439
440 return ret;
441 }
@@ -636,36 +443,13 @@ int netdata_rwlock_rdlock_debug(const char *file __maybe_unused, const char *fun
443 int netdata_rwlock_wrlock_debug(const char *file __maybe_unused, const char *function __maybe_unused,
444 const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
445
639 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_wrlock(%p) from %lu@%s, %s()", rwlock, line, file, function);
640 -
641 - netdata_rwlock_locker *locker = find_rwlock_locker(file, function, line, rwlock);
642 - if(locker)
643 - not_supported_by_posix_rwlocks(file, function, line, rwlock, 'W', "DEADLOCK - WANTS A WRITE LOCK BUT ALREADY HAVE THIS LOCKED");
446 + netdata_rwlock_locker *locker = add_rwlock_locker(file, function, line, rwlock, RWLOCK_REQUEST_WRITE);
447
645 - int log = 0;
646 - if(rwlock->readers) {
647 - log_rwlock_lockers(file, function, line, rwlock, "WANTS", 'W');
648 - log = 1;
649 - }
650 -
651 - usec_t start_s = now_monotonic_high_precision_usec();
448 int ret = __netdata_rwlock_wrlock(rwlock);
653 - usec_t end_s = now_monotonic_high_precision_usec();
654 -
655 - if(!ret){
656 - locker = update_or_add_rwlock_locker(file, function, line, rwlock, locker, 'W');
657 - if(log) log_rwlock_lockers(file, function, line, rwlock, "GOT", 'W');
658 - }
659 -
660 - if(end_s - start_s >= NETDATA_TRACE_RWLOCKS_WAIT_TIME_TO_IGNORE_USEC)
661 - fprintf(stderr,
662 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) WAITED for a WRITE lock for %llu usec.\n",
663 - rwlock,
664 - gettid(), netdata_thread_tag(),
665 - function, line, file,
666 - end_s - start_s);
667 -
668 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_wrlock(%p) = %d in %llu usec, from %lu@%s, %s()", rwlock, ret, end_s - start_s, line, file, function);
449 + if(!ret)
450 + locker->got_it = true;
451 + else
452 + remove_rwlock_locker(file, function, line, rwlock, locker);
453
454 return ret;
455 }
@@ -673,83 +457,42 @@ int netdata_rwlock_wrlock_debug(const char *file __maybe_unused, const char *fun
457 int netdata_rwlock_unlock_debug(const char *file __maybe_unused, const char *function __maybe_unused,
458 const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
459
676 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_unlock(%p) from %lu@%s, %s()", rwlock, line, file, function);
677 -
460 netdata_rwlock_locker *locker = find_rwlock_locker(file, function, line, rwlock);
461 +
462 if(unlikely(!locker))
680 - not_supported_by_posix_rwlocks(file, function, line, rwlock, 'U', "UNLOCK WITHOUT LOCK");
463 + fatal("UNLOCK WITHOUT LOCK");
464
682 - usec_t start_s = now_monotonic_high_precision_usec();
465 int ret = __netdata_rwlock_unlock(rwlock);
684 - usec_t end_s = now_monotonic_high_precision_usec();
685 -
686 - if(end_s - start_s >= NETDATA_TRACE_RWLOCKS_WAIT_TIME_TO_IGNORE_USEC)
687 - fprintf(stderr,
688 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) WAITED to UNLOCK for %llu usec.\n",
689 - rwlock,
690 - gettid(), netdata_thread_tag(),
691 - function, line, file,
692 - end_s - start_s);
693 -
694 - if(likely(!ret && locker)) remove_rwlock_locker(file, function, line, rwlock, locker);
695 -
696 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_unlock(%p) = %d in %llu usec, from %lu@%s, %s()", rwlock, ret, end_s - start_s, line, file, function);
466 + if(likely(!ret))
467 + remove_rwlock_locker(file, function, line, rwlock, locker);
468
469 return ret;
470 }
471
472 int netdata_rwlock_tryrdlock_debug(const char *file __maybe_unused, const char *function __maybe_unused,
473 const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
703 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_tryrdlock(%p) from %lu@%s, %s()", rwlock, line, file, function);
474
705 - netdata_rwlock_locker *locker = find_rwlock_locker(file, function, line, rwlock);
706 - if(locker && locker->lock == 'W')
707 - not_supported_by_posix_rwlocks(file, function, line, rwlock, 'R', "DEADLOCK - WANTS A READ LOCK BUT IT HAS A WRITE LOCK ALREADY");
475 + netdata_rwlock_locker *locker = add_rwlock_locker(file, function, line, rwlock, RWLOCK_REQUEST_TRYREAD);
476
709 - usec_t start_s = now_monotonic_high_precision_usec();
477 int ret = __netdata_rwlock_tryrdlock(rwlock);
711 - usec_t end_s = now_monotonic_high_precision_usec();
712 -
478 if(!ret)
714 - locker = update_or_add_rwlock_locker(file, function, line, rwlock, locker, 'R');
715 -
716 - if(end_s - start_s >= NETDATA_TRACE_RWLOCKS_WAIT_TIME_TO_IGNORE_USEC)
717 - fprintf(stderr,
718 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) WAITED to TRYREAD for %llu usec.\n",
719 - rwlock,
720 - gettid(), netdata_thread_tag(),
721 - function, line, file,
722 - end_s - start_s);
723 -
724 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_tryrdlock(%p) = %d in %llu usec, from %lu@%s, %s()", rwlock, ret, end_s - start_s, line, file, function);
479 + locker->got_it = true;
480 + else
481 + remove_rwlock_locker(file, function, line, rwlock, locker);
482
483 return ret;
484 }
485
486 int netdata_rwlock_trywrlock_debug(const char *file __maybe_unused, const char *function __maybe_unused,
487 const unsigned long line __maybe_unused, netdata_rwlock_t *rwlock) {
731 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_trywrlock(%p) from %lu@%s, %s()", rwlock, line, file, function);
488
733 - netdata_rwlock_locker *locker = find_rwlock_locker(file, function, line, rwlock);
734 - if(locker)
735 - not_supported_by_posix_rwlocks(file, function, line, rwlock, 'W', "ALREADY HAS THIS LOCK");
489 + netdata_rwlock_locker *locker = add_rwlock_locker(file, function, line, rwlock, RWLOCK_REQUEST_TRYWRITE);
490
737 - usec_t start_s = now_monotonic_high_precision_usec();
491 int ret = __netdata_rwlock_trywrlock(rwlock);
739 - usec_t end_s = now_monotonic_high_precision_usec();
740 -
492 if(!ret)
742 - locker = update_or_add_rwlock_locker(file, function, line, rwlock, locker, 'W');
743 -
744 - if(end_s - start_s >= NETDATA_TRACE_RWLOCKS_WAIT_TIME_TO_IGNORE_USEC)
745 - fprintf(stderr,
746 - "RW_LOCK ON LOCK %p: %d, '%s' (function %s() %lu@%s) WAITED to TRYWRITE for %llu usec.\n",
747 - rwlock,
748 - gettid(), netdata_thread_tag(),
749 - function, line, file,
750 - end_s - start_s);
751 -
752 - debug(D_LOCKS, "RW_LOCK: netdata_rwlock_trywrlock(%p) = %d in %llu usec, from %lu@%s, %s()", rwlock, ret, end_s - start_s, line, file, function);
493 + locker->got_it = true;
494 + else
495 + remove_rwlock_locker(file, function, line, rwlock, locker);
496
497 return ret;
498 }
libnetdata/locks/locks.h
+22 -7
@@ -11,24 +11,37 @@ typedef pthread_mutex_t netdata_mutex_t;
11
12 typedef struct netdata_spinlock {
13 bool locked;
14 +#ifdef NETDATA_INTERNAL_CHECKS
15 + size_t spins;
16 + pid_t locker_pid;
17 +#endif
18 } SPINLOCK;
15 -#define NETDATA_SPINLOCK_INITIALIZER (SPINLOCK){ .locked = false }
19 +#define NETDATA_SPINLOCK_INITIALIZER (SPINLOCK) { .locked = false }
20 +
21 void netdata_spinlock_init(SPINLOCK *spinlock);
22 void netdata_spinlock_lock(SPINLOCK *spinlock);
23 void netdata_spinlock_unlock(SPINLOCK *spinlock);
24 +bool netdata_spinlock_trylock(SPINLOCK *spinlock);
25
26 #ifdef NETDATA_TRACE_RWLOCKS
27 +
28 +typedef enum {
29 + RWLOCK_REQUEST_READ = (1 << 0),
30 + RWLOCK_REQUEST_WRITE = (1 << 1),
31 + RWLOCK_REQUEST_TRYREAD = (1 << 2),
32 + RWLOCK_REQUEST_TRYWRITE = (1 << 3),
33 +} LOCKER_REQUEST;
34 +
35 typedef struct netdata_rwlock_locker {
36 + LOCKER_REQUEST lock;
37 + bool got_it;
38 pid_t pid;
39 + size_t refcount;
40 const char *tag;
24 - char lock; // 'R', 'W'
41 const char *file;
42 const char *function;
43 unsigned long line;
28 - size_t callers;
29 - usec_t start_s;
30 - struct netdata_rwlock_t **all_caller_locks;
31 - struct netdata_rwlock_locker *next;
44 + struct netdata_rwlock_locker *next, *prev;
45 } netdata_rwlock_locker;
46
47 typedef struct netdata_rwlock_t {
@@ -37,6 +50,7 @@ typedef struct netdata_rwlock_t {
50 size_t writers; // the number of writers on the lock
51 netdata_mutex_t lockers_mutex; // a mutex to protect the linked list of the lock holding threads
52 netdata_rwlock_locker *lockers; // the linked list of the lock holding threads
53 + Pvoid_t lockers_pid_JudyL;
54 } netdata_rwlock_t;
55
56 #define NETDATA_RWLOCK_INITIALIZER { \
@@ -44,7 +58,8 @@ typedef struct netdata_rwlock_t {
58 .readers = 0, \
59 .writers = 0, \
60 .lockers_mutex = NETDATA_MUTEX_INITIALIZER, \
47 - .lockers = NULL \
61 + .lockers = NULL, \
62 + .lockers_pid_JudyL = NULL, \
63 }
64
65 #else // NETDATA_TRACE_RWLOCKS
libnetdata/required_dummies.h
+1
@@ -36,6 +36,7 @@ int health_variable_lookup(STRING *variable, struct rrdcalc *rc, NETDATA_DOUBLE
36 void rrdset_thread_rda_free(void){};
37 void sender_thread_buffer_free(void){};
38 void query_target_free(void){};
39 +void service_exits(void){};
40
41 // required by get_system_cpus()
42 char *netdata_configured_host_prefix = "";
libnetdata/socket/socket.c
+2 -1
@@ -1633,6 +1633,7 @@ void poll_events(LISTEN_SOCKETS *sockets
1633 , int (*rcv_callback)(POLLINFO * /*pi*/, short int * /*events*/)
1634 , int (*snd_callback)(POLLINFO * /*pi*/, short int * /*events*/)
1635 , void (*tmr_callback)(void * /*timer_data*/)
1636 + , bool (*check_to_stop_callback)(void)
1637 , SIMPLE_PATTERN *access_list
1638 , int allow_dns
1639 , void *data
@@ -1715,7 +1716,7 @@ void poll_events(LISTEN_SOCKETS *sockets
1716
1717 netdata_thread_cleanup_push(poll_events_cleanup, &p);
1718
1718 - while(!netdata_exit) {
1719 + while(!check_to_stop_callback()) {
1720 if(unlikely(timer_usec)) {
1721 now_usec = now_boottime_usec();
1722
libnetdata/socket/socket.h
+13 -12
@@ -10,18 +10,18 @@
10 #endif
11
12 typedef enum web_client_acl {
13 - WEB_CLIENT_ACL_NONE = 0,
14 - WEB_CLIENT_ACL_NOCHECK = 0,
15 - WEB_CLIENT_ACL_DASHBOARD = 1 << 0,
16 - WEB_CLIENT_ACL_REGISTRY = 1 << 1,
17 - WEB_CLIENT_ACL_BADGE = 1 << 2,
18 - WEB_CLIENT_ACL_MGMT = 1 << 3,
19 - WEB_CLIENT_ACL_STREAMING = 1 << 4,
20 - WEB_CLIENT_ACL_NETDATACONF = 1 << 5,
21 - WEB_CLIENT_ACL_SSL_OPTIONAL = 1 << 6,
22 - WEB_CLIENT_ACL_SSL_FORCE = 1 << 7,
23 - WEB_CLIENT_ACL_SSL_DEFAULT = 1 << 8,
24 - WEB_CLIENT_ACL_ACLK = 1 << 9,
13 + WEB_CLIENT_ACL_NONE = (0),
14 + WEB_CLIENT_ACL_NOCHECK = (0),
15 + WEB_CLIENT_ACL_DASHBOARD = (1 << 0),
16 + WEB_CLIENT_ACL_REGISTRY = (1 << 1),
17 + WEB_CLIENT_ACL_BADGE = (1 << 2),
18 + WEB_CLIENT_ACL_MGMT = (1 << 3),
19 + WEB_CLIENT_ACL_STREAMING = (1 << 4),
20 + WEB_CLIENT_ACL_NETDATACONF = (1 << 5),
21 + WEB_CLIENT_ACL_SSL_OPTIONAL = (1 << 6),
22 + WEB_CLIENT_ACL_SSL_FORCE = (1 << 7),
23 + WEB_CLIENT_ACL_SSL_DEFAULT = (1 << 8),
24 + WEB_CLIENT_ACL_ACLK = (1 << 9),
25 } WEB_CLIENT_ACL;
26
27 #define WEB_CLIENT_ACL_ALL 0xFFFF
@@ -202,6 +202,7 @@ void poll_events(LISTEN_SOCKETS *sockets
202 , int (*rcv_callback)(POLLINFO *pi, short int *events)
203 , int (*snd_callback)(POLLINFO *pi, short int *events)
204 , void (*tmr_callback)(void *timer_data)
205 + , bool (*check_to_stop_callback)(void)
206 , SIMPLE_PATTERN *access_list
207 , int allow_dns
208 , void *data
libnetdata/threads/threads.c
+21 -18
@@ -2,8 +2,7 @@
2
3 #include "../libnetdata.h"
4
5 -static size_t default_stacksize = 0, wanted_stacksize = 0;
6 -static pthread_attr_t *attr = NULL;
5 +static pthread_attr_t *netdata_threads_attr = NULL;
6
7 // ----------------------------------------------------------------------------
8 // per thread data
@@ -69,46 +68,48 @@ size_t netdata_threads_init(void) {
68 // --------------------------------------------------------------------
69 // get the required stack size of the threads of netdata
70
72 - attr = callocz(1, sizeof(pthread_attr_t));
73 - i = pthread_attr_init(attr);
71 + netdata_threads_attr = callocz(1, sizeof(pthread_attr_t));
72 + i = pthread_attr_init(netdata_threads_attr);
73 if(i != 0)
74 fatal("pthread_attr_init() failed with code %d.", i);
75
77 - i = pthread_attr_getstacksize(attr, &default_stacksize);
76 + size_t stacksize = 0;
77 + i = pthread_attr_getstacksize(netdata_threads_attr, &stacksize);
78 if(i != 0)
79 fatal("pthread_attr_getstacksize() failed with code %d.", i);
80 else
81 - debug(D_OPTIONS, "initial pthread stack size is %zu bytes", default_stacksize);
81 + debug(D_OPTIONS, "initial pthread stack size is %zu bytes", stacksize);
82
83 - return default_stacksize;
83 + return stacksize;
84 }
85
86 // ----------------------------------------------------------------------------
87 // late initialization
88
89 void netdata_threads_init_after_fork(size_t stacksize) {
90 - wanted_stacksize = stacksize;
90 int i;
91
92 // ------------------------------------------------------------------------
94 - // set default pthread stack size
93 + // set pthread stack size
94
96 - if(attr && default_stacksize < wanted_stacksize && wanted_stacksize > 0) {
97 - i = pthread_attr_setstacksize(attr, wanted_stacksize);
95 + if(netdata_threads_attr && stacksize > (size_t)PTHREAD_STACK_MIN) {
96 + i = pthread_attr_setstacksize(netdata_threads_attr, stacksize);
97 if(i != 0)
99 - fatal("pthread_attr_setstacksize() to %zu bytes, failed with code %d.", wanted_stacksize, i);
98 + error("pthread_attr_setstacksize() to %zu bytes, failed with code %d.", stacksize, i);
99 else
101 - debug(D_SYSTEM, "Successfully set pthread stacksize to %zu bytes", wanted_stacksize);
100 + info("Set threads stack size to %zu bytes", stacksize);
101 }
102 + else
103 + error("Invalid pthread stacksize %zu", stacksize);
104 }
105
105 -
106 // ----------------------------------------------------------------------------
107 // netdata_thread_create
108
109 -extern void rrdset_thread_rda_free(void);
110 -extern void sender_thread_buffer_free(void);
111 -extern void query_target_free(void);
109 +void rrdset_thread_rda_free(void);
110 +void sender_thread_buffer_free(void);
111 +void query_target_free(void);
112 +void service_exits(void);
113
114 static void thread_cleanup(void *ptr) {
115 if(netdata_thread != ptr) {
@@ -123,6 +124,8 @@ static void thread_cleanup(void *ptr) {
124 rrdset_thread_rda_free();
125 query_target_free();
126 thread_cache_destroy();
127 + service_exits();
128 + worker_unregister();
129
130 freez((void *)netdata_thread->tag);
131 netdata_thread->tag = NULL;
@@ -214,7 +217,7 @@ int netdata_thread_create(netdata_thread_t *thread, const char *tag, NETDATA_THR
217 info->start_routine = start_routine;
218 info->options = options;
219
217 - int ret = pthread_create(thread, attr, thread_start, info);
220 + int ret = pthread_create(thread, netdata_threads_attr, thread_start, info);
221 if(ret != 0)
222 error("failed to create new thread for %s. pthread_create() failed with code %d", tag, ret);
223
ml/ADCharts.cc
+62 -61
@@ -21,16 +21,16 @@ void ml::updateDimensionsChart(RRDHOST *RH, const MachineLearningStats &MLS) {
21 NameSS << "machine_learning_status_on_" << rrdhost_hostname(localhost);
22
23 MachineLearningStatusRS = rrdset_create(
24 - RH,
25 - "netdata", // type
24 + RH,
25 + "netdata", // type
26 IdSS.str().c_str(), // id
27 NameSS.str().c_str(), // name
28 - "ml", // family
28 + NETDATA_ML_CHART_FAMILY, // family
29 "netdata.machine_learning_status", // ctx
30 "Machine learning status", // title
31 "dimensions", // units
32 - "netdata", // plugin
33 - "ml", // module
32 + NETDATA_ML_PLUGIN, // plugin
33 + NETDATA_ML_MODULE_TRAINING, // module
34 NETDATA_ML_CHART_PRIO_MACHINE_LEARNING_STATUS, // priority
35 RH->rrd_update_every, // update_every
36 RRDSET_TYPE_LINE // chart_type
@@ -65,16 +65,16 @@ void ml::updateDimensionsChart(RRDHOST *RH, const MachineLearningStats &MLS) {
65 NameSS << "metric_types_on_" << rrdhost_hostname(localhost);
66
67 MetricTypesRS = rrdset_create(
68 - RH,
69 - "netdata", // type
68 + RH,
69 + "netdata", // type
70 IdSS.str().c_str(), // id
71 NameSS.str().c_str(), // name
72 - "ml", // family
72 + NETDATA_ML_CHART_FAMILY, // family
73 "netdata.metric_types", // ctx
74 "Dimensions by metric type", // title
75 "dimensions", // units
76 - "netdata", // plugin
77 - "ml", // module
76 + NETDATA_ML_PLUGIN, // plugin
77 + NETDATA_ML_MODULE_TRAINING, // module
78 NETDATA_ML_CHART_PRIO_METRIC_TYPES, // priority
79 RH->rrd_update_every, // update_every
80 RRDSET_TYPE_LINE // chart_type
@@ -109,16 +109,16 @@ void ml::updateDimensionsChart(RRDHOST *RH, const MachineLearningStats &MLS) {
109 NameSS << "training_status_on_" << rrdhost_hostname(localhost);
110
111 TrainingStatusRS = rrdset_create(
112 - RH,
113 - "netdata", // type
112 + RH,
113 + "netdata", // type
114 IdSS.str().c_str(), // id
115 NameSS.str().c_str(), // name
116 - "ml", // family
116 + NETDATA_ML_CHART_FAMILY, // family
117 "netdata.training_status", // ctx
118 "Training status of dimensions", // title
119 "dimensions", // units
120 - "netdata", // plugin
121 - "ml", // module
120 + NETDATA_ML_PLUGIN, // plugin
121 + NETDATA_ML_MODULE_TRAINING, // module
122 NETDATA_ML_CHART_PRIO_TRAINING_STATUS, // priority
123 RH->rrd_update_every, // update_every
124 RRDSET_TYPE_LINE // chart_type
@@ -156,16 +156,16 @@ void ml::updateDimensionsChart(RRDHOST *RH, const MachineLearningStats &MLS) {
156 NameSS << "dimensions_on_" << rrdhost_hostname(localhost);
157
158 PredictionRS = rrdset_create(
159 - RH,
160 - "anomaly_detection", // type
159 + RH,
160 + "anomaly_detection", // type
161 IdSS.str().c_str(), // id
162 NameSS.str().c_str(), // name
163 "dimensions", // family
164 "anomaly_detection.dimensions", // ctx
165 "Anomaly detection dimensions", // title
166 "dimensions", // units
167 - "netdata", // plugin
168 - "ml", // module
167 + NETDATA_ML_PLUGIN, // plugin
168 + NETDATA_ML_MODULE_TRAINING, // module
169 ML_CHART_PRIO_DIMENSIONS, // priority
170 RH->rrd_update_every, // update_every
171 RRDSET_TYPE_LINE // chart_type
@@ -195,16 +195,16 @@ void ml::updateHostAndDetectionRateCharts(RRDHOST *RH, collected_number AnomalyR
195 NameSS << "anomaly_rate_on_" << rrdhost_hostname(localhost);
196
197 HostRateRS = rrdset_create(
198 - RH,
199 - "anomaly_detection", // type
198 + RH,
199 + "anomaly_detection", // type
200 IdSS.str().c_str(), // id
201 NameSS.str().c_str(), // name
202 "anomaly_rate", // family
203 "anomaly_detection.anomaly_rate", // ctx
204 "Percentage of anomalous dimensions", // title
205 "percentage", // units
206 - "netdata", // plugin
207 - "ml", // module
206 + NETDATA_ML_PLUGIN, // plugin
207 + NETDATA_ML_MODULE_DETECTION, // module
208 ML_CHART_PRIO_ANOMALY_RATE, // priority
209 RH->rrd_update_every, // update_every
210 RRDSET_TYPE_LINE // chart_type
@@ -229,16 +229,16 @@ void ml::updateHostAndDetectionRateCharts(RRDHOST *RH, collected_number AnomalyR
229 NameSS << "anomaly_detection_on_" << rrdhost_hostname(localhost);
230
231 AnomalyDetectionRS = rrdset_create(
232 - RH,
233 - "anomaly_detection", // type
232 + RH,
233 + "anomaly_detection", // type
234 IdSS.str().c_str(), // id
235 NameSS.str().c_str(), // name
236 "anomaly_detection", // family
237 "anomaly_detection.detector_events", // ctx
238 "Anomaly detection events", // title
239 "percentage", // units
240 - "netdata", // plugin
241 - "ml", // module
240 + NETDATA_ML_PLUGIN, // plugin
241 + NETDATA_ML_MODULE_DETECTION, // module
242 ML_CHART_PRIO_DETECTOR_EVENTS, // priority
243 RH->rrd_update_every, // update_every
244 RRDSET_TYPE_LINE // chart_type
@@ -271,20 +271,21 @@ void ml::updateHostAndDetectionRateCharts(RRDHOST *RH, collected_number AnomalyR
271 NULL /* group options */,
272 0, /* timeout */
273 0, /* tier */
274 - QUERY_SOURCE_ML
274 + QUERY_SOURCE_ML,
275 + STORAGE_PRIORITY_BEST_EFFORT
276 );
277
278 if(R) {
278 - assert(R->d == 1 && R->n == 1 && R->rows == 1);
279 -
280 - static thread_local bool PrevAboveThreshold = false;
281 - bool AboveThreshold = R->v[0] >= Cfg.HostAnomalyRateThreshold;
282 - bool NewAnomalyEvent = AboveThreshold && !PrevAboveThreshold;
283 - PrevAboveThreshold = AboveThreshold;
284 -
285 - rrddim_set_by_pointer(AnomalyDetectionRS, AboveThresholdRD, AboveThreshold);
286 - rrddim_set_by_pointer(AnomalyDetectionRS, NewAnomalyEventRD, NewAnomalyEvent);
287 - rrdset_done(AnomalyDetectionRS);
279 + if(R->d == 1 && R->n == 1 && R->rows == 1) {
280 + static thread_local bool PrevAboveThreshold = false;
281 + bool AboveThreshold = R->v[0] >= Cfg.HostAnomalyRateThreshold;
282 + bool NewAnomalyEvent = AboveThreshold && !PrevAboveThreshold;
283 + PrevAboveThreshold = AboveThreshold;
284 +
285 + rrddim_set_by_pointer(AnomalyDetectionRS, AboveThresholdRD, AboveThreshold);
286 + rrddim_set_by_pointer(AnomalyDetectionRS, NewAnomalyEventRD, NewAnomalyEvent);
287 + rrdset_done(AnomalyDetectionRS);
288 + }
289
290 rrdr_free(OWA, R);
291 }
@@ -309,15 +310,15 @@ void ml::updateResourceUsageCharts(RRDHOST *RH, const struct rusage &PredictionR
310 NameSS << "prediction_usage_for_" << rrdhost_hostname(RH);
311
312 RS = rrdset_create_localhost(
312 - "netdata", // type
313 + "netdata", // type
314 IdSS.str().c_str(), // id
315 NameSS.str().c_str(), // name
315 - "ml", // family
316 + NETDATA_ML_CHART_FAMILY, // family
317 "netdata.prediction_usage", // ctx
318 "Prediction resource usage", // title
319 "milliseconds/s", // units
319 - "netdata", // plugin
320 - "ml", // module
320 + NETDATA_ML_PLUGIN, // plugin
321 + NETDATA_ML_MODULE_PREDICTION, // module
322 NETDATA_ML_CHART_PRIO_PREDICTION_USAGE, // priority
323 RH->rrd_update_every, // update_every
324 RRDSET_TYPE_STACKED // chart_type
@@ -350,15 +351,15 @@ void ml::updateResourceUsageCharts(RRDHOST *RH, const struct rusage &PredictionR
351 NameSS << "training_usage_for_" << rrdhost_hostname(RH);
352
353 RS = rrdset_create_localhost(
353 - "netdata", // type
354 + "netdata", // type
355 IdSS.str().c_str(), // id
356 NameSS.str().c_str(), // name
356 - "ml", // family
357 + NETDATA_ML_CHART_FAMILY, // family
358 "netdata.training_usage", // ctx
359 "Training resource usage", // title
360 "milliseconds/s", // units
360 - "netdata", // plugin
361 - "ml", // module
361 + NETDATA_ML_PLUGIN, // plugin
362 + NETDATA_ML_MODULE_TRAINING, // module
363 NETDATA_ML_CHART_PRIO_TRAINING_USAGE, // priority
364 RH->rrd_update_every, // update_every
365 RRDSET_TYPE_STACKED // chart_type
@@ -393,16 +394,16 @@ void ml::updateTrainingStatisticsChart(RRDHOST *RH, const TrainingStats &TS) {
394 NameSS << "queue_stats_on_" << rrdhost_hostname(localhost);
395
396 RS = rrdset_create(
396 - RH,
397 - "netdata", // type
397 + RH,
398 + "netdata", // type
399 IdSS.str().c_str(), // id
400 NameSS.str().c_str(), // name
400 - "ml", // family
401 + NETDATA_ML_CHART_FAMILY, // family
402 "netdata.queue_stats", // ctx
403 "Training queue stats", // title
404 "items", // units
404 - "netdata", // plugin
405 - "ml", // module
405 + NETDATA_ML_PLUGIN, // plugin
406 + NETDATA_ML_MODULE_TRAINING, // module
407 NETDATA_ML_CHART_PRIO_QUEUE_STATS, // priority
408 RH->rrd_update_every, // update_every
409 RRDSET_TYPE_LINE// chart_type
@@ -436,16 +437,16 @@ void ml::updateTrainingStatisticsChart(RRDHOST *RH, const TrainingStats &TS) {
437 NameSS << "training_time_stats_on_" << rrdhost_hostname(localhost);
438
439 RS = rrdset_create(
439 - RH,
440 - "netdata", // type
440 + RH,
441 + "netdata", // type
442 IdSS.str().c_str(), // id
443 NameSS.str().c_str(), // name
443 - "ml", // family
444 + NETDATA_ML_CHART_FAMILY, // family
445 "netdata.training_time_stats", // ctx
446 "Training time stats", // title
447 "milliseconds", // units
447 - "netdata", // plugin
448 - "ml", // module
448 + NETDATA_ML_PLUGIN, // plugin
449 + NETDATA_ML_MODULE_TRAINING, // module
450 NETDATA_ML_CHART_PRIO_TRAINING_TIME_STATS, // priority
451 RH->rrd_update_every, // update_every
452 RRDSET_TYPE_LINE// chart_type
@@ -483,16 +484,16 @@ void ml::updateTrainingStatisticsChart(RRDHOST *RH, const TrainingStats &TS) {
484 NameSS << "training_results_on_" << rrdhost_hostname(localhost);
485
486 RS = rrdset_create(
486 - RH,
487 - "netdata", // type
487 + RH,
488 + "netdata", // type
489 IdSS.str().c_str(), // id
490 NameSS.str().c_str(), // name
490 - "ml", // family
491 + NETDATA_ML_CHART_FAMILY, // family
492 "netdata.training_results", // ctx
493 "Training results", // title
494 "events", // units
494 - "netdata", // plugin
495 - "ml", // module
495 + NETDATA_ML_PLUGIN, // plugin
496 + NETDATA_ML_MODULE_TRAINING, // module
497 NETDATA_ML_CHART_PRIO_TRAINING_RESULTS, // priority
498 RH->rrd_update_every, // update_every
499 RRDSET_TYPE_LINE// chart_type
ml/Dimension.cc
+3 -3
@@ -70,8 +70,8 @@ std::pair<CalculatedNumber *, TrainingResponse> Dimension::getCalculatedNumbers(
70 TrainingResp.FirstEntryOnRequest = TrainingReq.FirstEntryOnRequest;
71 TrainingResp.LastEntryOnRequest = TrainingReq.LastEntryOnRequest;
72
73 - TrainingResp.FirstEntryOnResponse = rrddim_first_entry_t_of_tier(RD, 0);
74 - TrainingResp.LastEntryOnResponse = rrddim_last_entry_t_of_tier(RD, 0);
73 + TrainingResp.FirstEntryOnResponse = rrddim_first_entry_s_of_tier(RD, 0);
74 + TrainingResp.LastEntryOnResponse = rrddim_last_entry_s_of_tier(RD, 0);
75
76 size_t MinN = Cfg.MinTrainSamples;
77 size_t MaxN = Cfg.MaxTrainSamples;
@@ -195,7 +195,7 @@ TrainingResult Dimension::trainModel(const TrainingRequest &TrainingReq) {
195 MT = MetricType::Constant;
196 TS = TrainingStatus::Trained;
197 TR = TrainingResp;
198 - LastTrainingTime = rrddim_last_entry_t(RD);
198 + LastTrainingTime = rrddim_last_entry_s(RD);
199 }
200
201 delete[] CNs;
ml/Dimension.h
+5 -5
@@ -166,11 +166,11 @@ public:
166 private:
167 TrainingRequest getTrainingRequest(time_t CurrT) const {
168 return TrainingRequest {
169 - string_dup(RD->rrdset->id),
170 - string_dup(RD->id),
171 - CurrT,
172 - rrddim_first_entry_t(RD),
173 - rrddim_last_entry_t(RD)
169 + string_dup(RD->rrdset->id),
170 + string_dup(RD->id),
171 + CurrT,
172 + rrddim_first_entry_s(RD),
173 + rrddim_last_entry_s(RD)
174 };
175 }
176
ml/Host.cc
+52 -10
@@ -54,7 +54,15 @@ void Host::getModelsAsJson(nlohmann::json &Json) {
54 }
55 }
56
57 +#define WORKER_JOB_DETECTION_PREP 0
58 +#define WORKER_JOB_DETECTION_DIM_CHART 1
59 +#define WORKER_JOB_DETECTION_HOST_CHART 2
60 +#define WORKER_JOB_DETECTION_STATS 3
61 +#define WORKER_JOB_DETECTION_RESOURCES 4
62 +
63 void Host::detectOnce() {
64 + worker_is_busy(WORKER_JOB_DETECTION_PREP);
65 +
66 MLS = {};
67 MachineLearningStats MLSCopy = {};
68 TrainingStats TSCopy = {};
@@ -134,13 +142,20 @@ void Host::detectOnce() {
142 TSCopy.RemainingUT = 0;
143 }
144
145 + worker_is_busy(WORKER_JOB_DETECTION_DIM_CHART);
146 updateDimensionsChart(RH, MLSCopy);
147 +
148 + worker_is_busy(WORKER_JOB_DETECTION_HOST_CHART);
149 updateHostAndDetectionRateCharts(RH, HostAnomalyRate * 10000.0);
150
151 +#ifdef NETDATA_ML_RESOURCE_CHARTS
152 + worker_is_busy(WORKER_JOB_DETECTION_RESOURCES);
153 struct rusage PredictionRU;
154 getrusage(RUSAGE_THREAD, &PredictionRU);
155 updateResourceUsageCharts(RH, PredictionRU, TSCopy.TrainingRU);
156 +#endif
157
158 + worker_is_busy(WORKER_JOB_DETECTION_STATS);
159 updateTrainingStatisticsChart(RH, TSCopy);
160 }
161
@@ -150,7 +165,6 @@ public:
165 RRDDIM_ACQUIRED *AcqRD = nullptr;
166 Dimension *D = nullptr;
167
153 - rrdhost_rdlock(RH);
168 RRDSET *RS = rrdset_find(RH, string2str(ChartId));
169 if (RS) {
170 AcqRD = rrddim_find_and_acquire(RS, string2str(DimensionId));
@@ -160,7 +174,6 @@ public:
174 D = reinterpret_cast<Dimension *>(RD->ml_dimension);
175 }
176 }
163 - rrdhost_unlock(RH);
177
178 return AcquiredDimension(AcqRD, D);
179 }
@@ -190,8 +203,19 @@ void Host::scheduleForTraining(TrainingRequest TR) {
203 TrainingQueue.push(TR);
204 }
205
206 +#define WORKER_JOB_TRAINING_FIND 0
207 +#define WORKER_JOB_TRAINING_TRAIN 1
208 +#define WORKER_JOB_TRAINING_STATS 2
209 +
210 void Host::train() {
194 - while (!netdata_exit) {
211 + worker_register("MLTRAIN");
212 + worker_register_job_name(WORKER_JOB_TRAINING_FIND, "find");
213 + worker_register_job_name(WORKER_JOB_TRAINING_TRAIN, "train");
214 + worker_register_job_name(WORKER_JOB_TRAINING_STATS, "stats");
215 +
216 + service_register(SERVICE_THREAD_TYPE_NETDATA, NULL, (force_quit_t )ml_stop_anomaly_detection_threads, RH, true);
217 +
218 + while (service_running(SERVICE_ML_TRAINING)) {
219 auto P = TrainingQueue.pop();
220 TrainingRequest TrainingReq = P.first;
221 size_t Size = P.second;
@@ -200,15 +224,21 @@ void Host::train() {
224 if (AllottedUT > USEC_PER_SEC)
225 AllottedUT = USEC_PER_SEC;
226
203 - usec_t StartUT = now_realtime_usec();
227 + usec_t StartUT = now_monotonic_usec();
228 TrainingResult TrainingRes;
229 {
230 + worker_is_busy(WORKER_JOB_TRAINING_FIND);
231 AcquiredDimension AcqDim = AcquiredDimension::find(RH, TrainingReq.ChartId, TrainingReq.DimensionId);
232 +
233 + worker_is_busy(WORKER_JOB_TRAINING_TRAIN);
234 TrainingRes = AcqDim.train(TrainingReq);
235 +
236 string_freez(TrainingReq.ChartId);
237 string_freez(TrainingReq.DimensionId);
238 }
211 - usec_t ConsumedUT = now_realtime_usec() - StartUT;
239 + usec_t ConsumedUT = now_monotonic_usec() - StartUT;
240 +
241 + worker_is_busy(WORKER_JOB_TRAINING_STATS);
242
243 usec_t RemainingUT = 0;
244 if (ConsumedUT < AllottedUT)
@@ -249,15 +279,27 @@ void Host::train() {
279 }
280 }
281
282 + worker_is_idle();
283 std::this_thread::sleep_for(std::chrono::microseconds{RemainingUT});
284 + worker_is_busy(0);
285 }
286 }
287
288 void Host::detect() {
289 + worker_register("MLDETECT");
290 + worker_register_job_name(WORKER_JOB_DETECTION_PREP, "prep");
291 + worker_register_job_name(WORKER_JOB_DETECTION_DIM_CHART, "dim chart");
292 + worker_register_job_name(WORKER_JOB_DETECTION_HOST_CHART, "host chart");
293 + worker_register_job_name(WORKER_JOB_DETECTION_STATS, "stats");
294 + worker_register_job_name(WORKER_JOB_DETECTION_RESOURCES, "resources");
295 +
296 + service_register(SERVICE_THREAD_TYPE_NETDATA, NULL, (force_quit_t )ml_stop_anomaly_detection_threads, RH, true);
297 +
298 heartbeat_t HB;
299 heartbeat_init(&HB);
300
260 - while (!netdata_exit) {
301 + while (service_running((SERVICE_TYPE)(SERVICE_ML_PREDICTION | SERVICE_COLLECTORS))) {
302 + worker_is_idle();
303 heartbeat_next(&HB, RH->rrd_update_every * USEC_PER_SEC);
304 detectOnce();
305 }
@@ -294,10 +336,10 @@ void Host::startAnomalyDetectionThreads() {
336 char Tag[NETDATA_THREAD_TAG_MAX + 1];
337
338 snprintfz(Tag, NETDATA_THREAD_TAG_MAX, "TRAIN[%s]", rrdhost_hostname(RH));
297 - netdata_thread_create(&TrainingThread, Tag, NETDATA_THREAD_OPTION_JOINABLE, train_main, static_cast<void *>(this));
339 + netdata_thread_create(&TrainingThread, Tag, NETDATA_THREAD_OPTION_DEFAULT, train_main, static_cast<void *>(this));
340
341 snprintfz(Tag, NETDATA_THREAD_TAG_MAX, "DETECT[%s]", rrdhost_hostname(RH));
300 - netdata_thread_create(&DetectionThread, Tag, NETDATA_THREAD_OPTION_JOINABLE, detect_main, static_cast<void *>(this));
342 + netdata_thread_create(&DetectionThread, Tag, NETDATA_THREAD_OPTION_DEFAULT, detect_main, static_cast<void *>(this));
343 }
344
345 void Host::stopAnomalyDetectionThreads() {
@@ -311,8 +353,8 @@ void Host::stopAnomalyDetectionThreads() {
353 // Signal the training queue to stop popping-items
354 TrainingQueue.signal();
355 netdata_thread_cancel(TrainingThread);
314 - netdata_thread_join(TrainingThread, nullptr);
356 + // netdata_thread_join(TrainingThread, nullptr);
357
358 netdata_thread_cancel(DetectionThread);
317 - netdata_thread_join(DetectionThread, nullptr);
359 + // netdata_thread_join(DetectionThread, nullptr);
360 }
ml/Query.h
+4 -4
@@ -12,15 +12,15 @@ public:
12 }
13
14 time_t latestTime() {
15 - return Ops->latest_time(RD->tiers[0]->db_metric_handle);
15 + return Ops->latest_time_s(RD->tiers[0]->db_metric_handle);
16 }
17
18 time_t oldestTime() {
19 - return Ops->oldest_time(RD->tiers[0]->db_metric_handle);
19 + return Ops->oldest_time_s(RD->tiers[0]->db_metric_handle);
20 }
21
22 void init(time_t AfterT, time_t BeforeT) {
23 - Ops->init(RD->tiers[0]->db_metric_handle, &Handle, AfterT, BeforeT);
23 + Ops->init(RD->tiers[0]->db_metric_handle, &Handle, AfterT, BeforeT, STORAGE_PRIORITY_BEST_EFFORT);
24 Initialized = true;
25 points_read = 0;
26 }
@@ -40,7 +40,7 @@ public:
40 std::pair<time_t, CalculatedNumber> nextMetric() {
41 points_read++;
42 STORAGE_POINT sp = Ops->next_metric(&Handle);
43 - return { sp.end_time, sp.sum / sp.count };
43 + return {sp.end_time_s, sp.sum / sp.count };
44 }
45
46 private:
ml/SamplesBuffer.cc
+4 -7
@@ -54,12 +54,12 @@ void SamplesBuffer::diffSamples() {
54
55 void SamplesBuffer::smoothSamples() {
56 // Holds the mean value of each window
57 - CalculatedNumber *AccCNs = new CalculatedNumber[NumDimsPerSample]();
58 - Sample Acc(AccCNs, NumDimsPerSample);
57 + CalculatedNumber AccCNs[1] = { 0 };
58 + Sample Acc(AccCNs, 1);
59
60 // Used to avoid clobbering the accumulator when moving the window
61 - CalculatedNumber *TmpCNs = new CalculatedNumber[NumDimsPerSample]();
62 - Sample Tmp(TmpCNs, NumDimsPerSample);
61 + CalculatedNumber TmpCNs[1] = { 0 };
62 + Sample Tmp(TmpCNs, 1);
63
64 CalculatedNumber Factor = (CalculatedNumber) 1 / SmoothN;
65
@@ -88,9 +88,6 @@ void SamplesBuffer::smoothSamples() {
88 Acc.copy(Tmp);
89 Acc.scale(Factor);
90 }
91 -
92 - delete[] AccCNs;
93 - delete[] TmpCNs;
91 }
92
93 void SamplesBuffer::lagSamples() {
ml/SamplesBuffer.h
+3 -1
@@ -86,7 +86,9 @@ public:
86 DiffN(DiffN), SmoothN(SmoothN), LagN(LagN),
87 SamplingRatio(SamplingRatio), RandNums(RandNums),
88 BytesPerSample(NumDimsPerSample * sizeof(CalculatedNumber)),
89 - Preprocessed(false) {};
89 + Preprocessed(false) {
90 + assert(NumDimsPerSample == 1 && "SamplesBuffer supports only one dimension per sample");
91 + };
92
93 void preprocess(std::vector<DSample> &Samples);
94 void preprocess(DSample &Feature);
ml/ml.cc
-4
@@ -52,8 +52,6 @@ void ml_host_new(RRDHOST *RH) {
52
53 Host *H = new Host(RH);
54 RH->ml_host = reinterpret_cast<ml_host_t *>(H);
55 -
56 - H->startAnomalyDetectionThreads();
55 }
56
57 void ml_host_delete(RRDHOST *RH) {
@@ -61,8 +59,6 @@ void ml_host_delete(RRDHOST *RH) {
59 if (!H)
60 return;
61
64 - H->stopAnomalyDetectionThreads();
65 -
62 delete H;
63 RH->ml_host = nullptr;
64 }
streaming/receiver.c
+338 -273
@@ -16,7 +16,8 @@
16
17 extern struct config stream_config;
18
19 -void destroy_receiver_state(struct receiver_state *rpt) {
19 +void receiver_state_free(struct receiver_state *rpt) {
20 +
21 freez(rpt->key);
22 freez(rpt->hostname);
23 freez(rpt->registry_hostname);
@@ -29,43 +30,21 @@ void destroy_receiver_state(struct receiver_state *rpt) {
30 freez(rpt->client_port);
31 freez(rpt->program_name);
32 freez(rpt->program_version);
33 +
34 #ifdef ENABLE_HTTPS
33 - if(rpt->ssl.conn){
35 + if(rpt->ssl.conn)
36 SSL_free(rpt->ssl.conn);
35 - }
37 #endif
38 +
39 #ifdef ENABLE_COMPRESSION
40 if (rpt->decompressor)
41 rpt->decompressor->destroy(&rpt->decompressor);
42 #endif
41 - freez(rpt);
42 -}
43 -
44 -static void rrdpush_receiver_thread_cleanup(void *ptr) {
45 - worker_unregister();
43
47 - static __thread int executed = 0;
48 - if(!executed) {
49 - executed = 1;
50 - struct receiver_state *rpt = (struct receiver_state *) ptr;
51 - // If the shutdown sequence has started, and this receiver is still attached to the host then we cannot touch
52 - // the host pointer as it is unpredictable when the RRDHOST is deleted. Do the cleanup from rrdhost_free().
53 - if (netdata_exit && rpt->host) {
54 - rpt->exited = 1;
55 - return;
56 - }
44 + if(rpt->system_info)
45 + rrdhost_system_info_free(rpt->system_info);
46
58 - // Make sure that we detach this thread and don't kill a freshly arriving receiver
59 - if (!netdata_exit && rpt->host) {
60 - netdata_mutex_lock(&rpt->host->receiver_lock);
61 - if (rpt->host->receiver == rpt)
62 - rpt->host->receiver = NULL;
63 - netdata_mutex_unlock(&rpt->host->receiver_lock);
64 - }
65 -
66 - info("STREAM %s [receive from [%s]:%s]: receive thread ended (task id %d)", rpt->hostname, rpt->client_ip, rpt->client_port, gettid());
67 - destroy_receiver_state(rpt);
68 - }
47 + freez(rpt);
48 }
49
50 #include "collectors/plugins.d/pluginsd_parser.h"
@@ -105,11 +84,10 @@ PARSER_RC streaming_claimed_id(char **words, size_t num_words, void *user)
84 if (host->aclk_state.claimed_id)
85 freez(host->aclk_state.claimed_id);
86 host->aclk_state.claimed_id = strcmp(claim_id_str, "NULL") ? strdupz(claim_id_str) : NULL;
108 -
109 - metaqueue_store_claim_id(&host->host_uuid, host->aclk_state.claimed_id ? &uuid : NULL);
110 -
87 rrdhost_aclk_state_unlock(host);
88
89 + rrdhost_flag_set(host, RRDHOST_FLAG_METADATA_CLAIMID |RRDHOST_FLAG_METADATA_UPDATE);
90 +
91 rrdpush_claimed_id(host);
92
93 return PARSER_RC_OK;
@@ -390,39 +368,50 @@ static size_t streaming_parser(struct receiver_state *rpt, struct plugind *cd, i
368
369 size_t read_buffer_start = 0;
370 char buffer[PLUGINSD_LINE_MAX + 2] = "";
393 - while(!netdata_exit) {
371 + while(service_running(SERVICE_STREAMING)) {
372 + netdata_thread_testcancel();
373 +
374 if(!receiver_next_line(rpt, buffer, PLUGINSD_LINE_MAX + 2, &read_buffer_start)) {
375 bool have_new_data;
396 - if(compressed_connection)
376 + if(likely(compressed_connection))
377 have_new_data = receiver_read_compressed(rpt);
378 else
379 have_new_data = receiver_read_uncompressed(rpt);
380
401 - if(!have_new_data)
381 + if(unlikely(!have_new_data)) {
382 + if(!rpt->exit.reason)
383 + rpt->exit.reason = "SOCKET READ ERROR";
384 +
385 break;
386 + }
387
388 rpt->last_msg_t = now_realtime_sec();
389 continue;
390 }
391
408 - if(unlikely(netdata_exit)) {
409 - internal_error(true, "exiting...");
392 + if(unlikely(!service_running(SERVICE_STREAMING))) {
393 + if(!rpt->exit.reason)
394 + rpt->exit.reason = "NETDATA EXIT";
395 goto done;
396 }
412 - if(unlikely(rpt->shutdown)) {
413 - internal_error(true, "parser shutdown...");
397 + if(unlikely(rpt->exit.shutdown)) {
398 + if(!rpt->exit.reason)
399 + rpt->exit.reason = "SHUTDOWN REQUESTED";
400 +
401 goto done;
402 }
403
404 if (unlikely(parser_action(parser, buffer))) {
405 internal_error(true, "parser_action() failed on keyword '%s'.", buffer);
406 +
407 + if(!rpt->exit.reason)
408 + rpt->exit.reason = "PARSER FAILED";
409 +
410 break;
411 }
412 }
413
414 done:
424 - internal_error(true, "Streaming receiver thread stopping...");
425 -
415 result = user.count;
416
417 // free parser with the pop function
@@ -431,103 +420,236 @@ done:
420 return result;
421 }
422
434 -static void rrdpush_receiver_replication_reset(struct receiver_state *rpt) {
423 +static void rrdpush_receiver_replication_reset(RRDHOST *host) {
424 RRDSET *st;
436 - rrdset_foreach_read(st, rpt->host) {
425 + rrdset_foreach_read(st, host) {
426 rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS);
427 rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
428 }
429 rrdset_foreach_done(st);
441 - rrdhost_receiver_replicating_charts_zero(rpt->host);
430 + rrdhost_receiver_replicating_charts_zero(host);
431 +}
432 +
433 +bool rrdhost_set_receiver(RRDHOST *host, struct receiver_state *rpt) {
434 + bool signal_rrdcontext = false;
435 + bool set_this = false;
436 +
437 + netdata_mutex_lock(&host->receiver_lock);
438 +
439 + if (!host->receiver || host->receiver == rpt) {
440 + rrdhost_flag_clear(host, RRDHOST_FLAG_ORPHAN);
441 +
442 + host->receiver = rpt;
443 + rpt->host = host;
444 +
445 + host->child_connect_time = now_realtime_sec();
446 + host->child_disconnected_time = 0;
447 + host->child_last_chart_command = 0;
448 + host->trigger_chart_obsoletion_check = 1;
449 +
450 + if (rpt->config.health_enabled != CONFIG_BOOLEAN_NO) {
451 + if (rpt->config.alarms_delay > 0) {
452 + host->health_delay_up_to = now_realtime_sec() + rpt->config.alarms_delay;
453 + log_health(
454 + "[%s]: Postponing health checks for %" PRId64 " seconds, because it was just connected.",
455 + rrdhost_hostname(host),
456 + (int64_t) rpt->config.alarms_delay);
457 + }
458 + }
459 +
460 + signal_rrdcontext = true;
461 + rrdpush_receiver_replication_reset(host);
462 +
463 + rrdhost_flag_clear(rpt->host, RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED);
464 +
465 + set_this = true;
466 + }
467 +
468 + netdata_mutex_unlock(&host->receiver_lock);
469 +
470 + if(signal_rrdcontext)
471 + rrdcontext_host_child_connected(host);
472 +
473 + return set_this;
474 +}
475 +
476 +static void rrdhost_clear_receiver(struct receiver_state *rpt) {
477 + bool signal_rrdcontext = false;
478 +
479 + RRDHOST *host = rpt->host;
480 + if(host) {
481 + netdata_mutex_lock(&host->receiver_lock);
482 +
483 + // Make sure that we detach this thread and don't kill a freshly arriving receiver
484 + if(host->receiver == rpt) {
485 + host->trigger_chart_obsoletion_check = 0;
486 + host->child_connect_time = 0;
487 + host->child_disconnected_time = now_realtime_sec();
488 +
489 + if (rpt->config.health_enabled == CONFIG_BOOLEAN_AUTO)
490 + host->health_enabled = 0;
491 +
492 + rrdpush_sender_thread_stop(host, "RECEIVER LEFT", false);
493 +
494 + signal_rrdcontext = true;
495 + rrdpush_receiver_replication_reset(host);
496 +
497 + if (host->receiver == rpt)
498 + host->receiver = NULL;
499 +
500 + rrdhost_flag_set(host, RRDHOST_FLAG_ORPHAN);
501 + }
502 +
503 + netdata_mutex_unlock(&host->receiver_lock);
504 +
505 + if(signal_rrdcontext)
506 + rrdcontext_host_child_disconnected(host);
507 + }
508 +}
509 +
510 +bool stop_streaming_receiver(RRDHOST *host, const char *reason) {
511 + bool ret = false;
512 +
513 + netdata_mutex_lock(&host->receiver_lock);
514 +
515 + if(host->receiver) {
516 + if(!host->receiver->exit.shutdown) {
517 + host->receiver->exit.shutdown = true;
518 + host->receiver->exit.reason = reason;
519 + shutdown(host->receiver->fd, SHUT_RDWR);
520 + }
521 +
522 + netdata_thread_cancel(host->receiver->thread);
523 + }
524 +
525 + int count = 2000;
526 + while (host->receiver && count-- > 0) {
527 + netdata_mutex_unlock(&host->receiver_lock);
528 +
529 + // let the lock for the receiver thread to exit
530 + sleep_usec(1 * USEC_PER_MS);
531 +
532 + netdata_mutex_lock(&host->receiver_lock);
533 + }
534 +
535 + if(host->receiver)
536 + error("STREAM '%s' [receive from [%s]:%s]: "
537 + "thread %d takes too long to stop, giving up..."
538 + , rrdhost_hostname(host)
539 + , host->receiver->client_ip, host->receiver->client_port
540 + , gettid());
541 + else
542 + ret = true;
543 +
544 + netdata_mutex_unlock(&host->receiver_lock);
545 +
546 + return ret;
547 +}
548 +
549 +void rrdpush_receive_log_status(struct receiver_state *rpt, const char *msg, const char *status) {
550 +
551 + log_stream_connection(rpt->client_ip, rpt->client_port,
552 + (rpt->key && *rpt->key)? rpt->key : "-",
553 + (rpt->machine_guid && *rpt->machine_guid) ? rpt->machine_guid : "-",
554 + (rpt->hostname && *rpt->hostname) ? rpt->hostname : "-",
555 + status);
556 +
557 + info("STREAM '%s' [receive from [%s]:%s]: "
558 + "%s. "
559 + "STATUS: %s%s%s%s"
560 + , rpt->hostname
561 + , rpt->client_ip, rpt->client_port
562 + , msg
563 + , status
564 + , rpt->exit.reason?" (":""
565 + , rpt->exit.reason?rpt->exit.reason:""
566 + , rpt->exit.reason?")":""
567 + );
568 +
569 +}
570 +
571 +static void rrdhost_reset_destinations(RRDHOST *host) {
572 + for (struct rrdpush_destinations *d = host->destinations; d; d = d->next)
573 + d->postpone_reconnection_until = 0;
574 }
575
576 static int rrdpush_receive(struct receiver_state *rpt)
577 {
446 - int history = default_rrd_history_entries;
447 - RRD_MEMORY_MODE mode = default_rrd_memory_mode;
448 - int health_enabled = default_health_enabled;
449 - int rrdpush_enabled = default_rrdpush_enabled;
450 - char *rrdpush_destination = default_rrdpush_destination;
451 - char *rrdpush_api_key = default_rrdpush_api_key;
452 - char *rrdpush_send_charts_matching = default_rrdpush_send_charts_matching;
453 - bool rrdpush_enable_replication = default_rrdpush_enable_replication;
454 - time_t rrdpush_seconds_to_replicate = default_rrdpush_seconds_to_replicate;
455 - time_t rrdpush_replication_step = default_rrdpush_replication_step;
456 - time_t alarms_delay = 60;
457 -
458 - rpt->update_every = (int)appconfig_get_number(&stream_config, rpt->machine_guid, "update every", rpt->update_every);
459 - if(rpt->update_every < 0) rpt->update_every = 1;
460 -
461 - history = (int)appconfig_get_number(&stream_config, rpt->key, "default history", history);
462 - history = (int)appconfig_get_number(&stream_config, rpt->machine_guid, "history", history);
463 - if(history < 5) history = 5;
464 -
465 - mode = rrd_memory_mode_id(appconfig_get(&stream_config, rpt->key, "default memory mode", rrd_memory_mode_name(mode)));
466 - mode = rrd_memory_mode_id(appconfig_get(&stream_config, rpt->machine_guid, "memory mode", rrd_memory_mode_name(mode)));
467 -
468 - if (unlikely(mode == RRD_MEMORY_MODE_DBENGINE && !dbengine_enabled)) {
469 - error("STREAM %s [receive from %s:%s]: dbengine is not enabled, falling back to default.", rpt->hostname, rpt->client_ip, rpt->client_port);
470 - mode = default_rrd_memory_mode;
578 + rpt->config.mode = default_rrd_memory_mode;
579 + rpt->config.history = default_rrd_history_entries;
580 +
581 + rpt->config.health_enabled = (int)default_health_enabled;
582 + rpt->config.alarms_delay = 60;
583 +
584 + rpt->config.rrdpush_enabled = (int)default_rrdpush_enabled;
585 + rpt->config.rrdpush_destination = default_rrdpush_destination;
586 + rpt->config.rrdpush_api_key = default_rrdpush_api_key;
587 + rpt->config.rrdpush_send_charts_matching = default_rrdpush_send_charts_matching;
588 +
589 + rpt->config.rrdpush_enable_replication = default_rrdpush_enable_replication;
590 + rpt->config.rrdpush_seconds_to_replicate = default_rrdpush_seconds_to_replicate;
591 + rpt->config.rrdpush_replication_step = default_rrdpush_replication_step;
592 +
593 + rpt->config.update_every = (int)appconfig_get_number(&stream_config, rpt->machine_guid, "update every", rpt->config.update_every);
594 + if(rpt->config.update_every < 0) rpt->config.update_every = 1;
595 +
596 + rpt->config.history = (int)appconfig_get_number(&stream_config, rpt->key, "default history", rpt->config.history);
597 + rpt->config.history = (int)appconfig_get_number(&stream_config, rpt->machine_guid, "history", rpt->config.history);
598 + if(rpt->config.history < 5) rpt->config.history = 5;
599 +
600 + rpt->config.mode = rrd_memory_mode_id(appconfig_get(&stream_config, rpt->key, "default memory mode", rrd_memory_mode_name(rpt->config.mode)));
601 + rpt->config.mode = rrd_memory_mode_id(appconfig_get(&stream_config, rpt->machine_guid, "memory mode", rrd_memory_mode_name(rpt->config.mode)));
602 +
603 + if (unlikely(rpt->config.mode == RRD_MEMORY_MODE_DBENGINE && !dbengine_enabled)) {
604 + error("STREAM '%s' [receive from %s:%s]: "
605 + "dbengine is not enabled, falling back to default."
606 + , rpt->hostname
607 + , rpt->client_ip, rpt->client_port
608 + );
609 +
610 + rpt->config.mode = default_rrd_memory_mode;
611 }
612
473 - health_enabled = appconfig_get_boolean_ondemand(&stream_config, rpt->key, "health enabled by default", health_enabled);
474 - health_enabled = appconfig_get_boolean_ondemand(&stream_config, rpt->machine_guid, "health enabled", health_enabled);
613 + rpt->config.health_enabled = appconfig_get_boolean_ondemand(&stream_config, rpt->key, "health enabled by default", rpt->config.health_enabled);
614 + rpt->config.health_enabled = appconfig_get_boolean_ondemand(&stream_config, rpt->machine_guid, "health enabled", rpt->config.health_enabled);
615
476 - alarms_delay = appconfig_get_number(&stream_config, rpt->key, "default postpone alarms on connect seconds", alarms_delay);
477 - alarms_delay = appconfig_get_number(&stream_config, rpt->machine_guid, "postpone alarms on connect seconds", alarms_delay);
616 + rpt->config.alarms_delay = appconfig_get_number(&stream_config, rpt->key, "default postpone alarms on connect seconds", rpt->config.alarms_delay);
617 + rpt->config.alarms_delay = appconfig_get_number(&stream_config, rpt->machine_guid, "postpone alarms on connect seconds", rpt->config.alarms_delay);
618
479 - rrdpush_enabled = appconfig_get_boolean(&stream_config, rpt->key, "default proxy enabled", rrdpush_enabled);
480 - rrdpush_enabled = appconfig_get_boolean(&stream_config, rpt->machine_guid, "proxy enabled", rrdpush_enabled);
619 + rpt->config.rrdpush_enabled = appconfig_get_boolean(&stream_config, rpt->key, "default proxy enabled", rpt->config.rrdpush_enabled);
620 + rpt->config.rrdpush_enabled = appconfig_get_boolean(&stream_config, rpt->machine_guid, "proxy enabled", rpt->config.rrdpush_enabled);
621
482 - rrdpush_destination = appconfig_get(&stream_config, rpt->key, "default proxy destination", rrdpush_destination);
483 - rrdpush_destination = appconfig_get(&stream_config, rpt->machine_guid, "proxy destination", rrdpush_destination);
622 + rpt->config.rrdpush_destination = appconfig_get(&stream_config, rpt->key, "default proxy destination", rpt->config.rrdpush_destination);
623 + rpt->config.rrdpush_destination = appconfig_get(&stream_config, rpt->machine_guid, "proxy destination", rpt->config.rrdpush_destination);
624
485 - rrdpush_api_key = appconfig_get(&stream_config, rpt->key, "default proxy api key", rrdpush_api_key);
486 - rrdpush_api_key = appconfig_get(&stream_config, rpt->machine_guid, "proxy api key", rrdpush_api_key);
625 + rpt->config.rrdpush_api_key = appconfig_get(&stream_config, rpt->key, "default proxy api key", rpt->config.rrdpush_api_key);
626 + rpt->config.rrdpush_api_key = appconfig_get(&stream_config, rpt->machine_guid, "proxy api key", rpt->config.rrdpush_api_key);
627
488 - rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->key, "default proxy send charts matching", rrdpush_send_charts_matching);
489 - rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->machine_guid, "proxy send charts matching", rrdpush_send_charts_matching);
628 + rpt->config.rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->key, "default proxy send charts matching", rpt->config.rrdpush_send_charts_matching);
629 + rpt->config.rrdpush_send_charts_matching = appconfig_get(&stream_config, rpt->machine_guid, "proxy send charts matching", rpt->config.rrdpush_send_charts_matching);
630
491 - rrdpush_enable_replication = appconfig_get_boolean(&stream_config, rpt->key, "enable replication", rrdpush_enable_replication);
492 - rrdpush_enable_replication = appconfig_get_boolean(&stream_config, rpt->machine_guid, "enable replication", rrdpush_enable_replication);
631 + rpt->config.rrdpush_enable_replication = appconfig_get_boolean(&stream_config, rpt->key, "enable replication", rpt->config.rrdpush_enable_replication);
632 + rpt->config.rrdpush_enable_replication = appconfig_get_boolean(&stream_config, rpt->machine_guid, "enable replication", rpt->config.rrdpush_enable_replication);
633
494 - rrdpush_seconds_to_replicate = appconfig_get_number(&stream_config, rpt->key, "seconds to replicate", rrdpush_seconds_to_replicate);
495 - rrdpush_seconds_to_replicate = appconfig_get_number(&stream_config, rpt->machine_guid, "seconds to replicate", rrdpush_seconds_to_replicate);
634 + rpt->config.rrdpush_seconds_to_replicate = appconfig_get_number(&stream_config, rpt->key, "seconds to replicate", rpt->config.rrdpush_seconds_to_replicate);
635 + rpt->config.rrdpush_seconds_to_replicate = appconfig_get_number(&stream_config, rpt->machine_guid, "seconds to replicate", rpt->config.rrdpush_seconds_to_replicate);
636
497 - rrdpush_replication_step = appconfig_get_number(&stream_config, rpt->key, "seconds per replication step", rrdpush_replication_step);
498 - rrdpush_replication_step = appconfig_get_number(&stream_config, rpt->machine_guid, "seconds per replication step", rrdpush_replication_step);
637 + rpt->config.rrdpush_replication_step = appconfig_get_number(&stream_config, rpt->key, "seconds per replication step", rpt->config.rrdpush_replication_step);
638 + rpt->config.rrdpush_replication_step = appconfig_get_number(&stream_config, rpt->machine_guid, "seconds per replication step", rpt->config.rrdpush_replication_step);
639
640 #ifdef ENABLE_COMPRESSION
501 - unsigned int rrdpush_compression = default_compression_enabled;
502 - rrdpush_compression = appconfig_get_boolean(&stream_config, rpt->key, "enable compression", rrdpush_compression);
503 - rrdpush_compression = appconfig_get_boolean(&stream_config, rpt->machine_guid, "enable compression", rrdpush_compression);
504 - rpt->rrdpush_compression = (rrdpush_compression && default_compression_enabled);
641 + rpt->config.rrdpush_compression = default_compression_enabled;
642 + rpt->config.rrdpush_compression = appconfig_get_boolean(&stream_config, rpt->key, "enable compression", rpt->config.rrdpush_compression);
643 + rpt->config.rrdpush_compression = appconfig_get_boolean(&stream_config, rpt->machine_guid, "enable compression", rpt->config.rrdpush_compression);
644 + rpt->rrdpush_compression = (rpt->config.rrdpush_compression && default_compression_enabled);
645 #endif //ENABLE_COMPRESSION
646
647 (void)appconfig_set_default(&stream_config, rpt->machine_guid, "host tags", (rpt->tags)?rpt->tags:"");
648
509 - if (strcmp(rpt->machine_guid, localhost->machine_guid) == 0) {
510 - log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->machine_guid, rpt->hostname, "DENIED - ATTEMPT TO RECEIVE METRICS FROM MACHINE_GUID IDENTICAL TO PARENT");
511 - error("STREAM %s [receive from %s:%s]: denied to receive metrics, machine GUID [%s] is my own. Did you copy the parent/proxy machine GUID to a child, or is this an inter-agent loop?", rpt->hostname, rpt->client_ip, rpt->client_port, rpt->machine_guid);
512 - char initial_response[HTTP_HEADER_SIZE + 1];
513 - snprintfz(initial_response, HTTP_HEADER_SIZE, "%s", START_STREAMING_ERROR_SAME_LOCALHOST);
514 -#ifdef ENABLE_HTTPS
515 - if(send_timeout(&rpt->ssl, rpt->fd, initial_response, strlen(initial_response), 0, 60) != (ssize_t)strlen(initial_response)) {
516 -#else
517 - if(send_timeout(rpt->fd, initial_response, strlen(initial_response), 0, 60) != strlen(initial_response)) {
518 -#endif
519 - log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->host->machine_guid, rrdhost_hostname(rpt->host), "FAILED - CANNOT REPLY");
520 - error("STREAM %s [receive from [%s]:%s]: cannot send command.", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
521 - close(rpt->fd);
522 - return 0;
523 - }
524 - close(rpt->fd);
525 - return 0;
526 - }
527 -
528 - if (rpt->host==NULL) {
529 -
530 - rpt->host = rrdhost_find_or_create(
649 + // find the host for this receiver
650 + {
651 + // this will also update the host with our system_info
652 + RRDHOST *host = rrdhost_find_or_create(
653 rpt->hostname
654 , rpt->registry_hostname
655 , rpt->machine_guid
@@ -538,76 +660,41 @@ static int rrdpush_receive(struct receiver_state *rpt)
660 , rpt->tags
661 , rpt->program_name
662 , rpt->program_version
541 - , rpt->update_every
542 - , history
543 - , mode
544 - , (unsigned int)(health_enabled != CONFIG_BOOLEAN_NO)
545 - , (unsigned int)(rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key)
546 - , rrdpush_destination
547 - , rrdpush_api_key
548 - , rrdpush_send_charts_matching
549 - , rrdpush_enable_replication
550 - , rrdpush_seconds_to_replicate
551 - , rrdpush_replication_step
663 + , rpt->config.update_every
664 + , rpt->config.history
665 + , rpt->config.mode
666 + , (unsigned int)(rpt->config.health_enabled != CONFIG_BOOLEAN_NO)
667 + , (unsigned int)(rpt->config.rrdpush_enabled && rpt->config.rrdpush_destination && *rpt->config.rrdpush_destination && rpt->config.rrdpush_api_key && *rpt->config.rrdpush_api_key)
668 + , rpt->config.rrdpush_destination
669 + , rpt->config.rrdpush_api_key
670 + , rpt->config.rrdpush_send_charts_matching
671 + , rpt->config.rrdpush_enable_replication
672 + , rpt->config.rrdpush_seconds_to_replicate
673 + , rpt->config.rrdpush_replication_step
674 , rpt->system_info
675 , 0
676 );
677
556 - if(!rpt->host) {
678 + if(!host) {
679 + rrdpush_receive_log_status(rpt, "failed to find/create host structure", "INTERNAL ERROR DROPPING CONNECTION");
680 close(rpt->fd);
558 - log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->machine_guid, rpt->hostname, "FAILED - CANNOT ACQUIRE HOST");
559 - error("STREAM %s [receive from [%s]:%s]: failed to find/create host structure.", rpt->hostname, rpt->client_ip, rpt->client_port);
681 return 1;
682 }
683
563 - netdata_mutex_lock(&rpt->host->receiver_lock);
564 - if (rpt->host->receiver == NULL)
565 - rpt->host->receiver = rpt;
566 - else {
567 - error("Multiple receivers connected for %s concurrently, cancelling this one...", rpt->machine_guid);
568 - netdata_mutex_unlock(&rpt->host->receiver_lock);
684 + // system_info has been consumed by the host structure
685 + rpt->system_info = NULL;
686 +
687 + if(!rrdhost_set_receiver(host, rpt)) {
688 + rrdpush_receive_log_status(rpt, "host is already served by another receiver", "DUPLICATE RECEIVER DROPPING CONNECTION");
689 close(rpt->fd);
570 - log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->machine_guid, rpt->hostname, "FAILED - BEATEN TO HOST CREATION");
690 return 1;
691 }
573 - netdata_mutex_unlock(&rpt->host->receiver_lock);
574 - }
575 - else {
576 - rrd_wrlock();
577 - rrdhost_update(
578 - rpt->host,
579 - rpt->hostname,
580 - rpt->registry_hostname,
581 - rpt->machine_guid,
582 - rpt->os,
583 - rpt->timezone,
584 - rpt->abbrev_timezone,
585 - rpt->utc_offset,
586 - rpt->tags,
587 - rpt->program_name,
588 - rpt->program_version,
589 - rpt->update_every,
590 - history,
591 - mode,
592 - (unsigned int)(health_enabled != CONFIG_BOOLEAN_NO),
593 - (unsigned int)(rrdpush_enabled && rrdpush_destination && *rrdpush_destination && rrdpush_api_key && *rrdpush_api_key),
594 - rrdpush_destination,
595 - rrdpush_api_key,
596 - rrdpush_send_charts_matching,
597 - rrdpush_enable_replication,
598 - rrdpush_seconds_to_replicate,
599 - rrdpush_replication_step,
600 - rpt->system_info);
601 - rrd_unlock();
692 }
693
694 #ifdef NETDATA_INTERNAL_CHECKS
605 - int ssl = 0;
606 -#ifdef ENABLE_HTTPS
607 - if (rpt->ssl.conn != NULL)
608 - ssl = 1;
609 -#endif
610 - info("STREAM %s [receive from [%s]:%s]: client willing to stream metrics for host '%s' with machine_guid '%s': update every = %d, history = %ld, memory mode = %s, health %s,%s tags '%s'"
695 + info("STREAM '%s' [receive from [%s]:%s]: "
696 + "client willing to stream metrics for host '%s' with machine_guid '%s': "
697 + "update every = %d, history = %ld, memory mode = %s, health %s,%s tags '%s'"
698 , rpt->hostname
699 , rpt->client_ip
700 , rpt->client_port
@@ -616,8 +703,12 @@ static int rrdpush_receive(struct receiver_state *rpt)
703 , rpt->host->rrd_update_every
704 , rpt->host->rrd_history_entries
705 , rrd_memory_mode_name(rpt->host->rrd_memory_mode)
619 - , (health_enabled == CONFIG_BOOLEAN_NO)?"disabled":((health_enabled == CONFIG_BOOLEAN_YES)?"enabled":"auto")
620 - , ssl ? " SSL," : ""
706 + , (rpt->config.health_enabled == CONFIG_BOOLEAN_NO)?"disabled":((rpt->config.health_enabled == CONFIG_BOOLEAN_YES)?"enabled":"auto")
707 +#ifdef ENABLE_HTTPS
708 + , (rpt->ssl.conn != NULL) ? " SSL," : ""
709 +#else
710 + , ""
711 +#endif
712 , rrdhost_tags(rpt->host)
713 );
714 #endif // NETDATA_INTERNAL_CHECKS
@@ -648,76 +739,60 @@ static int rrdpush_receive(struct receiver_state *rpt)
739 }
740 #endif
741
651 - // info("STREAM %s [receive from [%s]:%s]: initializing communication...", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
652 - char initial_response[HTTP_HEADER_SIZE];
653 - if (stream_has_capability(rpt, STREAM_CAP_VCAPS)) {
654 - log_receiver_capabilities(rpt);
655 - sprintf(initial_response, "%s%u", START_STREAMING_PROMPT_VN, rpt->capabilities);
656 - }
657 - else if (stream_has_capability(rpt, STREAM_CAP_VN)) {
658 - log_receiver_capabilities(rpt);
659 - sprintf(initial_response, "%s%d", START_STREAMING_PROMPT_VN, stream_capabilities_to_vn(rpt->capabilities));
660 - } else if (stream_has_capability(rpt, STREAM_CAP_V2)) {
661 - log_receiver_capabilities(rpt);
662 - sprintf(initial_response, "%s", START_STREAMING_PROMPT_V2);
663 - } else { // stream_has_capability(rpt, STREAM_CAP_V1)
664 - log_receiver_capabilities(rpt);
665 - sprintf(initial_response, "%s", START_STREAMING_PROMPT_V1);
666 - }
667 - debug(D_STREAM, "Initial response to %s: %s", rpt->client_ip, initial_response);
742 + {
743 + // info("STREAM %s [receive from [%s]:%s]: initializing communication...", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
744 + char initial_response[HTTP_HEADER_SIZE];
745 + if (stream_has_capability(rpt, STREAM_CAP_VCAPS)) {
746 + log_receiver_capabilities(rpt);
747 + sprintf(initial_response, "%s%u", START_STREAMING_PROMPT_VN, rpt->capabilities);
748 + }
749 + else if (stream_has_capability(rpt, STREAM_CAP_VN)) {
750 + log_receiver_capabilities(rpt);
751 + sprintf(initial_response, "%s%d", START_STREAMING_PROMPT_VN, stream_capabilities_to_vn(rpt->capabilities));
752 + }
753 + else if (stream_has_capability(rpt, STREAM_CAP_V2)) {
754 + log_receiver_capabilities(rpt);
755 + sprintf(initial_response, "%s", START_STREAMING_PROMPT_V2);
756 + }
757 + else { // stream_has_capability(rpt, STREAM_CAP_V1)
758 + log_receiver_capabilities(rpt);
759 + sprintf(initial_response, "%s", START_STREAMING_PROMPT_V1);
760 + }
761 +
762 + debug(D_STREAM, "Initial response to %s: %s", rpt->client_ip, initial_response);
763 + if(send_timeout(
764 #ifdef ENABLE_HTTPS
669 - if(send_timeout(&rpt->ssl, rpt->fd, initial_response, strlen(initial_response), 0, 60) != (ssize_t)strlen(initial_response)) {
670 -#else
671 - if(send_timeout(rpt->fd, initial_response, strlen(initial_response), 0, 60) != strlen(initial_response)) {
765 + &rpt->ssl,
766 #endif
673 - log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->host->machine_guid, rrdhost_hostname(rpt->host), "FAILED - CANNOT REPLY");
674 - error("STREAM %s [receive from [%s]:%s]: cannot send ready command.", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
675 - close(rpt->fd);
676 - return 0;
677 - }
767 + rpt->fd, initial_response, strlen(initial_response), 0, 60) != (ssize_t)strlen(initial_response)) {
768
679 - // remove the non-blocking flag from the socket
680 - if(sock_delnonblock(rpt->fd) < 0)
681 - error("STREAM %s [receive from [%s]:%s]: cannot remove the non-blocking flag from socket %d", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port, rpt->fd);
682 -
683 - struct timeval timeout;
684 - timeout.tv_sec = 600;
685 - timeout.tv_usec = 0;
686 - if (unlikely(setsockopt(rpt->fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof timeout) != 0))
687 - error("STREAM %s [receive from [%s]:%s]: cannot set timeout for socket %d", rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port, rpt->fd);
688 -
689 - rrdhost_wrlock(rpt->host);
690 -/* if(rpt->host->connected_senders > 0) {
691 - rrdhost_unlock(rpt->host);
692 - log_stream_connection(rpt->client_ip, rpt->client_port, rpt->key, rpt->host->machine_guid, rpt->host->hostname, "REJECTED - ALREADY CONNECTED");
693 - info("STREAM %s [receive from [%s]:%s]: multiple streaming connections for the same host detected. Rejecting new connection.", rpt->host->hostname, rpt->client_ip, rpt->client_port);
694 - fclose(fp);
695 - return 0;
696 - }
697 -*/
698 -
699 -// rpt->host->connected_senders++;
700 - if(health_enabled != CONFIG_BOOLEAN_NO) {
701 - if(alarms_delay > 0) {
702 - rpt->host->health_delay_up_to = now_realtime_sec() + alarms_delay;
703 - log_health(
704 - "[%s]: Postponing health checks for %" PRId64 " seconds, because it was just connected.",
705 - rrdhost_hostname(rpt->host),
706 - (int64_t)alarms_delay);
769 + rrdpush_receive_log_status(rpt, "cannot reply back", "CANT REPLY DROPPING CONNECTION");
770 + close(rpt->fd);
771 + return 0;
772 }
773 }
709 - rpt->host->senders_connect_time = now_realtime_sec();
710 - rpt->host->senders_last_chart_command = 0;
711 - rpt->host->trigger_chart_obsoletion_check = 1;
774
713 - rrdhost_unlock(rpt->host);
775 + {
776 + // remove the non-blocking flag from the socket
777 + if(sock_delnonblock(rpt->fd) < 0)
778 + error("STREAM '%s' [receive from [%s]:%s]: "
779 + "cannot remove the non-blocking flag from socket %d"
780 + , rrdhost_hostname(rpt->host)
781 + , rpt->client_ip, rpt->client_port
782 + , rpt->fd);
783
715 - // call the plugins.d processor to receive the metrics
716 - info("STREAM %s [receive from [%s]:%s]: receiving metrics...",
717 - rrdhost_hostname(rpt->host), rpt->client_ip, rpt->client_port);
784 + struct timeval timeout;
785 + timeout.tv_sec = 600;
786 + timeout.tv_usec = 0;
787 + if (unlikely(setsockopt(rpt->fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof timeout) != 0))
788 + error("STREAM '%s' [receive from [%s]:%s]: "
789 + "cannot set timeout for socket %d"
790 + , rrdhost_hostname(rpt->host)
791 + , rpt->client_ip, rpt->client_port
792 + , rpt->fd);
793 + }
794
719 - log_stream_connection(rpt->client_ip, rpt->client_port,
720 - rpt->key, rpt->host->machine_guid, rrdhost_hostname(rpt->host), "CONNECTED");
795 + rrdpush_receive_log_status(rpt, "ready to receive data", "CONNECTED");
796
797 cd.capabilities = rpt->capabilities;
798
@@ -728,12 +803,10 @@ static int rrdpush_receive(struct receiver_state *rpt)
803 aclk_host_state_update(rpt->host, 1);
804 #endif
805
731 - rrdhost_set_is_parent_label(++localhost->senders_count);
806 + rrdhost_set_is_parent_label(++localhost->connected_children_count);
807
733 - rrdpush_receiver_replication_reset(rpt);
734 - rrdcontext_host_child_connected(rpt->host);
735 -
736 - rrdhost_flag_clear(rpt->host, RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED);
808 + // let it reconnect to parent immediately
809 + rrdhost_reset_destinations(rpt->host);
810
811 size_t count = streaming_parser(rpt, &cd, rpt->fd,
812 #ifdef ENABLE_HTTPS
@@ -745,15 +818,14 @@ static int rrdpush_receive(struct receiver_state *rpt)
818
819 rrdhost_flag_set(rpt->host, RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED);
820
748 - log_stream_connection(rpt->client_ip, rpt->client_port,
749 - rpt->key, rpt->host->machine_guid, rpt->hostname,
750 - "DISCONNECTED");
821 + if(!rpt->exit.reason)
822 + rpt->exit.reason = "PARSER EXIT";
823
752 - error("STREAM %s [receive from [%s]:%s]: disconnected (completed %zu updates).",
753 - rpt->hostname, rpt->client_ip, rpt->client_port, count);
754 -
755 - rrdcontext_host_child_disconnected(rpt->host);
756 - rrdpush_receiver_replication_reset(rpt);
824 + {
825 + char msg[100 + 1];
826 + snprintfz(msg, 100, "disconnected (completed %zu updates)", count);
827 + rrdpush_receive_log_status(rpt, msg, "DISCONNECTED");
828 + }
829
830 #ifdef ENABLE_ACLK
831 // in case we have cloud connection we inform cloud
@@ -762,48 +834,41 @@ static int rrdpush_receive(struct receiver_state *rpt)
834 aclk_host_state_update(rpt->host, 0);
835 #endif
836
765 - rrdhost_set_is_parent_label(--localhost->senders_count);
766 -
767 - // During a shutdown there is cleanup code in rrdhost that will cancel the sender thread
768 - if (!netdata_exit && rpt->host) {
769 - rrd_rdlock();
770 - rrdhost_wrlock(rpt->host);
771 - netdata_mutex_lock(&rpt->host->receiver_lock);
772 - if (rpt->host->receiver == rpt) {
773 - rpt->host->senders_connect_time = 0;
774 - rpt->host->trigger_chart_obsoletion_check = 0;
775 - rpt->host->senders_disconnected_time = now_realtime_sec();
776 - rrdhost_flag_set(rpt->host, RRDHOST_FLAG_ORPHAN);
777 - if(health_enabled == CONFIG_BOOLEAN_AUTO)
778 - rpt->host->health_enabled = 0;
779 - }
780 - rrdhost_unlock(rpt->host);
781 - if (rpt->host->receiver == rpt) {
782 - rrdpush_sender_thread_stop(rpt->host);
783 - }
784 - netdata_mutex_unlock(&rpt->host->receiver_lock);
785 - rrd_unlock();
786 - }
837 + rrdhost_set_is_parent_label(--localhost->connected_children_count);
838
839 // cleanup
840 close(rpt->fd);
841 return (int)count;
842 }
843
844 +static void rrdpush_receiver_thread_cleanup(void *ptr) {
845 + struct receiver_state *rpt = (struct receiver_state *) ptr;
846 + worker_unregister();
847 +
848 + rrdhost_clear_receiver(rpt);
849 +
850 + info("STREAM '%s' [receive from [%s]:%s]: "
851 + "receive thread ended (task id %d)"
852 + , rpt->hostname ? rpt->hostname : "-"
853 + , rpt->client_ip ? rpt->client_ip : "-", rpt->client_port ? rpt->client_port : "-"
854 + , gettid());
855 +
856 + receiver_state_free(rpt);
857 +}
858 +
859 void *rrdpush_receiver_thread(void *ptr) {
860 netdata_thread_cleanup_push(rrdpush_receiver_thread_cleanup, ptr);
861
796 - struct receiver_state *rpt = (struct receiver_state *)ptr;
797 - info("STREAM %s [%s]:%s: receive thread created (task id %d)", rpt->hostname, rpt->client_ip, rpt->client_port, gettid());
798 -
862 worker_register("STREAMRCV");
863 worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_READ, "received bytes", "bytes/s", WORKER_METRIC_INCREMENT);
864 worker_register_job_custom_metric(WORKER_RECEIVER_JOB_BYTES_UNCOMPRESSED, "uncompressed bytes", "bytes/s", WORKER_METRIC_INCREMENT);
865 worker_register_job_custom_metric(WORKER_RECEIVER_JOB_REPLICATION_COMPLETION, "replication completion", "%", WORKER_METRIC_ABSOLUTE);
866 +
867 + struct receiver_state *rpt = (struct receiver_state *)ptr;
868 + info("STREAM %s [%s]:%s: receive thread created (task id %d)", rpt->hostname, rpt->client_ip, rpt->client_port, gettid());
869 +
870 rrdpush_receive(rpt);
804 - worker_unregister();
871
872 netdata_thread_cleanup_pop(1);
873 return NULL;
874 }
809 -
streaming/replication.c
+480 -253
@@ -11,20 +11,21 @@
11 #define WORKER_JOB_QUERYING 2
12 #define WORKER_JOB_DELETE_ENTRY 3
13 #define WORKER_JOB_FIND_CHART 4
14 -#define WORKER_JOB_CHECK_CONSISTENCY 5
15 -#define WORKER_JOB_BUFFER_COMMIT 6
16 -#define WORKER_JOB_CLEANUP 7
17 -#define WORKER_JOB_WAIT 8
14 +#define WORKER_JOB_PREPARE_QUERY 5
15 +#define WORKER_JOB_CHECK_CONSISTENCY 6
16 +#define WORKER_JOB_BUFFER_COMMIT 7
17 +#define WORKER_JOB_CLEANUP 8
18 +#define WORKER_JOB_WAIT 9
19
20 // master thread worker jobs
20 -#define WORKER_JOB_STATISTICS 9
21 -#define WORKER_JOB_CUSTOM_METRIC_PENDING_REQUESTS 10
22 -#define WORKER_JOB_CUSTOM_METRIC_SKIPPED_NO_ROOM 11
23 -#define WORKER_JOB_CUSTOM_METRIC_COMPLETION 12
24 -#define WORKER_JOB_CUSTOM_METRIC_ADDED 13
25 -#define WORKER_JOB_CUSTOM_METRIC_DONE 14
26 -#define WORKER_JOB_CUSTOM_METRIC_SENDER_RESETS 15
27 -#define WORKER_JOB_CUSTOM_METRIC_SENDER_FULL 16
21 +#define WORKER_JOB_STATISTICS 10
22 +#define WORKER_JOB_CUSTOM_METRIC_PENDING_REQUESTS 11
23 +#define WORKER_JOB_CUSTOM_METRIC_SKIPPED_NO_ROOM 12
24 +#define WORKER_JOB_CUSTOM_METRIC_COMPLETION 13
25 +#define WORKER_JOB_CUSTOM_METRIC_ADDED 14
26 +#define WORKER_JOB_CUSTOM_METRIC_DONE 15
27 +#define WORKER_JOB_CUSTOM_METRIC_SENDER_RESETS 16
28 +#define WORKER_JOB_CUSTOM_METRIC_SENDER_FULL 17
29
30 #define ITERATIONS_IDLE_WITHOUT_PENDING_TO_RUN_SENDER_VERIFICATION 30
31 #define SECONDS_TO_RESET_POINT_IN_TIME 10
@@ -51,137 +52,313 @@ struct replication_dimension {
52 STORAGE_POINT sp;
53 struct storage_engine_query_handle handle;
54 bool enabled;
55 + bool skip;
56
57 DICTIONARY *dict;
58 const DICTIONARY_ITEM *rda;
59 RRDDIM *rd;
60 };
61
60 -static time_t replicate_chart_timeframe(BUFFER *wb, RRDSET *st, time_t after, time_t before, bool enable_streaming, time_t wall_clock_time) {
62 +struct replication_query {
63 + RRDSET *st;
64 +
65 + struct {
66 + time_t first_entry_t;
67 + time_t last_entry_t;
68 + } db;
69 +
70 + struct { // what the parent requested
71 + time_t after;
72 + time_t before;
73 + bool enable_streaming;
74 + } request;
75 +
76 + struct { // what the child will do
77 + time_t after;
78 + time_t before;
79 + bool enable_streaming;
80 +
81 + bool locked_data_collection;
82 + bool execute;
83 + } query;
84 +
85 + time_t wall_clock_time;
86 +
87 + size_t points_read;
88 + size_t points_generated;
89 +
90 + struct storage_engine_query_ops *ops;
91 + struct replication_request *rq;
92 +
93 + size_t dimensions;
94 + struct replication_dimension data[];
95 +};
96 +
97 +static struct replication_query *replication_query_prepare(
98 + RRDSET *st,
99 + time_t db_first_entry,
100 + time_t db_last_entry,
101 + time_t requested_after,
102 + time_t requested_before,
103 + time_t query_after,
104 + time_t query_before,
105 + bool enable_streaming,
106 + time_t wall_clock_time
107 +) {
108 size_t dimensions = rrdset_number_of_dimensions(st);
62 - size_t points_read = 0, points_generated = 0;
109 + struct replication_query *q = callocz(1, sizeof(struct replication_query) + dimensions * sizeof(struct replication_dimension));
110 + q->dimensions = dimensions;
111 + q->st = st;
112
64 - struct storage_engine_query_ops *ops = &st->rrdhost->db[0].eng->api.query_ops;
65 - struct replication_dimension data[dimensions];
66 - memset(data, 0, sizeof(data));
113 + q->db.first_entry_t = db_first_entry;
114 + q->db.last_entry_t = db_last_entry;
115
68 - if(enable_streaming && st->last_updated.tv_sec > before) {
69 - internal_error(true, "STREAM_SENDER REPLAY: 'host:%s/chart:%s' has start_streaming = true, adjusting replication before timestamp from %llu to %llu",
70 - rrdhost_hostname(st->rrdhost), rrdset_id(st),
71 - (unsigned long long)before,
72 - (unsigned long long)st->last_updated.tv_sec
73 - );
74 - before = st->last_updated.tv_sec;
116 + q->request.after = requested_after,
117 + q->request.before = requested_before,
118 + q->request.enable_streaming = enable_streaming,
119 +
120 + q->query.after = query_after;
121 + q->query.before = query_before;
122 + q->query.enable_streaming = enable_streaming;
123 +
124 + q->wall_clock_time = wall_clock_time;
125 +
126 + if (!q->dimensions || !q->query.after || !q->query.before) {
127 + q->query.execute = false;
128 + q->dimensions = 0;
129 + return q;
130 + }
131 +
132 + if(q->query.enable_streaming) {
133 + netdata_spinlock_lock(&st->data_collection_lock);
134 + q->query.locked_data_collection = true;
135 +
136 + if (st->last_updated.tv_sec > q->query.before) {
137 + internal_error(true,
138 + "STREAM_SENDER REPLAY: 'host:%s/chart:%s' "
139 + "has start_streaming = true, "
140 + "adjusting replication before timestamp from %llu to %llu",
141 + rrdhost_hostname(st->rrdhost), rrdset_id(st),
142 + (unsigned long long) q->query.before,
143 + (unsigned long long) st->last_updated.tv_sec
144 + );
145 + q->query.before = st->last_updated.tv_sec;
146 + }
147 }
148
149 + q->ops = &st->rrdhost->db[0].eng->api.query_ops;
150 +
151 // prepare our array of dimensions
78 - {
79 - RRDDIM *rd;
80 - rrddim_foreach_read(rd, st) {
81 - if(unlikely(!rd || !rd_dfe.item || !rd->exposed))
82 - continue;
152 + size_t count = 0;
153 + RRDDIM *rd;
154 + rrddim_foreach_read(rd, st) {
155 + if (unlikely(!rd || !rd_dfe.item || !rd->exposed))
156 + continue;
157
84 - if (unlikely(rd_dfe.counter >= dimensions)) {
85 - internal_error(true, "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s' has more dimensions than the replicated ones",
86 - rrdhost_hostname(st->rrdhost), rrdset_id(st));
87 - break;
88 - }
158 + if (unlikely(rd_dfe.counter >= q->dimensions)) {
159 + internal_error(true,
160 + "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s' has more dimensions than the replicated ones",
161 + rrdhost_hostname(st->rrdhost), rrdset_id(st));
162 + break;
163 + }
164
90 - struct replication_dimension *d = &data[rd_dfe.counter];
165 + struct replication_dimension *d = &q->data[rd_dfe.counter];
166
92 - d->dict = rd_dfe.dict;
93 - d->rda = dictionary_acquired_item_dup(rd_dfe.dict, rd_dfe.item);
94 - d->rd = rd;
167 + d->dict = rd_dfe.dict;
168 + d->rda = dictionary_acquired_item_dup(rd_dfe.dict, rd_dfe.item);
169 + d->rd = rd;
170
96 - ops->init(rd->tiers[0]->db_metric_handle, &d->handle, after, before);
97 - d->enabled = true;
171 + q->ops->init(rd->tiers[0]->db_metric_handle, &d->handle, q->query.after, q->query.before,
172 + q->query.locked_data_collection ? STORAGE_PRIORITY_HIGH : STORAGE_PRIORITY_LOW);
173 + d->enabled = true;
174 + d->skip = false;
175 + count++;
176 + }
177 + rrddim_foreach_done(rd);
178 +
179 + if(!count) {
180 + // no data for this chart
181 +
182 + q->query.execute = false;
183 +
184 + if(q->query.locked_data_collection) {
185 + netdata_spinlock_unlock(&st->data_collection_lock);
186 + q->query.locked_data_collection = false;
187 }
99 - rrddim_foreach_done(rd);
188 +
189 + }
190 + else {
191 + // we have data for this chart
192 +
193 + q->query.execute = true;
194 + }
195 +
196 + return q;
197 +}
198 +
199 +static time_t replication_query_finalize(struct replication_query *q, bool executed) {
200 + time_t query_before = q->query.before;
201 + size_t dimensions = q->dimensions;
202 +
203 + // release all the dictionary items acquired
204 + // finalize the queries
205 + size_t queries = 0;
206 +
207 + for (size_t i = 0; i < dimensions; i++) {
208 + struct replication_dimension *d = &q->data[i];
209 + if (unlikely(!d->enabled)) continue;
210 +
211 + q->ops->finalize(&d->handle);
212 +
213 + dictionary_acquired_item_release(d->dict, d->rda);
214 +
215 + // update global statistics
216 + queries++;
217 + }
218 +
219 + if(q->query.locked_data_collection) {
220 + netdata_spinlock_unlock(&q->st->data_collection_lock);
221 + q->query.locked_data_collection = false;
222 + }
223 +
224 + if(executed) {
225 + netdata_spinlock_lock(&replication_queries.spinlock);
226 + replication_queries.queries_started += queries;
227 + replication_queries.queries_finished += queries;
228 + replication_queries.points_read += q->points_read;
229 + replication_queries.points_generated += q->points_generated;
230 + netdata_spinlock_unlock(&replication_queries.spinlock);
231 }
232
102 - time_t now = after + 1, actual_after = 0, actual_before = 0; (void)actual_before;
233 + freez(q);
234 +
235 + return query_before;
236 +}
237 +
238 +static void replication_query_align_to_optimal_before(struct replication_query *q) {
239 + if(!q->query.execute || q->query.enable_streaming)
240 + return;
241 +
242 + size_t dimensions = q->dimensions;
243 + time_t expanded_before = 0;
244 +
245 + for (size_t i = 0; i < dimensions; i++) {
246 + struct replication_dimension *d = &q->data[i];
247 + if(unlikely(!d->enabled)) continue;
248 +
249 + time_t new_before = q->ops->align_to_optimal_before(&d->handle);
250 + if (!expanded_before || new_before < expanded_before)
251 + expanded_before = new_before;
252 + }
253 +
254 + if(expanded_before > q->query.before && // it is later than the original
255 + (expanded_before - q->query.before) / q->st->update_every < 1024 && // it is reasonable (up to a page)
256 + expanded_before < q->st->last_updated.tv_sec && // it is not the chart's last updated time
257 + expanded_before < q->wall_clock_time) // it is not later than the wall clock time
258 + q->query.before = expanded_before;
259 +}
260 +
261 +static time_t replication_query_execute_and_finalize(BUFFER *wb, struct replication_query *q) {
262 + if(!q->query.execute)
263 + return replication_query_finalize(q, false);
264 +
265 + replication_query_align_to_optimal_before(q);
266 +
267 + time_t after = q->query.after;
268 + time_t before = q->query.before;
269 + size_t dimensions = q->dimensions;
270 + struct storage_engine_query_ops *ops = q->ops;
271 + time_t wall_clock_time = q->wall_clock_time;
272 +
273 + size_t points_read = q->points_read, points_generated = q->points_generated;
274 +
275 +#ifdef NETDATA_LOG_REPLICATION_REQUESTS
276 + time_t actual_after = 0, actual_before = 0;
277 +#endif
278 +
279 + time_t now = after + 1;
280 while(now <= before) {
281 time_t min_start_time = 0, min_end_time = 0;
282 for (size_t i = 0; i < dimensions ;i++) {
106 - struct replication_dimension *d = &data[i];
107 - if(unlikely(!d->enabled)) continue;
283 + struct replication_dimension *d = &q->data[i];
284 + if(unlikely(!d->enabled || d->skip)) continue;
285
286 // fetch the first valid point for the dimension
110 - int max_skip = 100;
111 - while(d->sp.end_time < now && !ops->is_finished(&d->handle) && max_skip-- > 0) {
287 + int max_skip = 1000;
288 + while(d->sp.end_time_s < now && !ops->is_finished(&d->handle) && max_skip-- >= 0) {
289 d->sp = ops->next_metric(&d->handle);
290 points_read++;
291 }
292
116 - internal_error(max_skip <= 0,
117 - "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s/dim:%s': db does not advance the query beyond time %llu",
118 - rrdhost_hostname(st->rrdhost), rrdset_id(st), rrddim_id(d->rd), (unsigned long long) now);
293 + if(max_skip <= 0) {
294 + d->skip = true;
295
120 - if(unlikely(d->sp.end_time < now || storage_point_is_unset(d->sp) || storage_point_is_empty(d->sp)))
121 - continue;
296 + error_limit_static_global_var(erl, 1, 0);
297 + error_limit(&erl,
298 + "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s/dim:%s': db does not advance the query beyond time %llu (tried 1000 times to get the next point and always got back a point in the past)",
299 + rrdhost_hostname(q->st->rrdhost), rrdset_id(q->st), rrddim_id(d->rd),
300 + (unsigned long long) now);
301
123 - if(unlikely(!min_start_time)) {
124 - min_start_time = d->sp.start_time;
125 - min_end_time = d->sp.end_time;
126 - }
127 - else {
128 - min_start_time = MIN(min_start_time, d->sp.start_time);
129 - min_end_time = MIN(min_end_time, d->sp.end_time);
302 + continue;
303 }
131 - }
304
133 - if(unlikely(min_start_time > wall_clock_time + 1 || min_end_time > wall_clock_time + st->update_every + 1)) {
134 - internal_error(true,
135 - "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s': db provided future start time %llu or end time %llu (now is %llu)",
136 - rrdhost_hostname(st->rrdhost), rrdset_id(st),
137 - (unsigned long long)min_start_time,
138 - (unsigned long long)min_end_time,
139 - (unsigned long long)wall_clock_time);
140 - break;
305 + if(d->sp.end_time_s < now)
306 + // this dimension does not have any more data
307 + continue;
308 +
309 + if(unlikely(!min_start_time))
310 + min_start_time = d->sp.start_time_s;
311 +
312 + if(unlikely(!min_end_time))
313 + min_end_time = d->sp.end_time_s;
314 +
315 + min_start_time = MIN(min_start_time, d->sp.start_time_s);
316 + min_end_time = MIN(min_end_time, d->sp.end_time_s);
317 }
318
143 - if(unlikely(min_end_time < now)) {
144 -#ifdef NETDATA_LOG_REPLICATION_REQUESTS
145 - internal_error(true,
146 - "STREAM_SENDER REPLAY: 'host:%s/chart:%s': no data on any dimension beyond time %llu",
147 - rrdhost_hostname(st->rrdhost), rrdset_id(st), (unsigned long long)now);
148 -#endif // NETDATA_LOG_REPLICATION_REQUESTS
319 + if(unlikely(min_end_time < now))
320 break;
150 - }
321
152 - if(unlikely(min_end_time <= min_start_time))
153 - min_start_time = min_end_time - st->update_every;
322 + if(likely(min_start_time <= now)) {
323 + // we have a valid point
324 +
325 + if (unlikely(min_end_time <= min_start_time))
326 + min_start_time = min_end_time - q->st->update_every;
327 +
328 +#ifdef NETDATA_LOG_REPLICATION_REQUESTS
329 + if (unlikely(!actual_after))
330 + actual_after = min_end_time;
331
155 - if(unlikely(!actual_after)) {
156 - actual_after = min_end_time;
157 - actual_before = min_end_time;
158 - }
159 - else
332 actual_before = min_end_time;
333 +#endif
334
162 - buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_BEGIN " '' %llu %llu %llu\n"
163 - , (unsigned long long)min_start_time
164 - , (unsigned long long)min_end_time
165 - , (unsigned long long)wall_clock_time
166 - );
335 + buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_BEGIN " '' %llu %llu %llu\n",
336 + (unsigned long long) min_start_time,
337 + (unsigned long long) min_end_time,
338 + (unsigned long long) wall_clock_time
339 + );
340
168 - // output the replay values for this time
169 - for (size_t i = 0; i < dimensions ;i++) {
170 - struct replication_dimension *d = &data[i];
171 - if(unlikely(!d->enabled)) continue;
341 + // output the replay values for this time
342 + for (size_t i = 0; i < dimensions; i++) {
343 + struct replication_dimension *d = &q->data[i];
344 + if (unlikely(!d->enabled)) continue;
345
173 - if(likely(d->sp.start_time <= min_end_time && d->sp.end_time >= min_end_time))
174 - buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_SET " \"%s\" " NETDATA_DOUBLE_FORMAT " \"%s\"\n",
175 - rrddim_id(d->rd), d->sp.sum, d->sp.flags & SN_FLAG_RESET ? "R" : "");
346 + if (likely( d->sp.start_time_s <= min_end_time &&
347 + d->sp.end_time_s >= min_end_time &&
348 + !storage_point_is_unset(d->sp) &&
349 + !storage_point_is_empty(d->sp))) {
350
177 - else
178 - buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_SET " \"%s\" NAN \"E\"\n",
179 - rrddim_id(d->rd));
351 + buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_SET " \"%s\" " NETDATA_DOUBLE_FORMAT " \"%s\"\n",
352 + rrddim_id(d->rd), d->sp.sum, d->sp.flags & SN_FLAG_RESET ? "R" : "");
353
181 - points_generated++;
182 - }
354 + points_generated++;
355 + }
356 + }
357
184 - now = min_end_time + 1;
358 + now = min_end_time + 1;
359 + }
360 + else
361 + now = min_start_time;
362 }
363
364 #ifdef NETDATA_LOG_REPLICATION_REQUESTS
@@ -202,32 +379,12 @@ static time_t replicate_chart_timeframe(BUFFER *wb, RRDSET *st, time_t after, ti
379 (unsigned long long)after, (unsigned long long)before);
380 #endif // NETDATA_LOG_REPLICATION_REQUESTS
381
205 - // release all the dictionary items acquired
206 - // finalize the queries
207 - size_t queries = 0;
208 - for(size_t i = 0; i < dimensions ;i++) {
209 - struct replication_dimension *d = &data[i];
210 - if(unlikely(!d->enabled)) continue;
211 -
212 - ops->finalize(&d->handle);
213 -
214 - dictionary_acquired_item_release(d->dict, d->rda);
215 -
216 - // update global statistics
217 - queries++;
218 - }
219 -
220 - netdata_spinlock_lock(&replication_queries.spinlock);
221 - replication_queries.queries_started += queries;
222 - replication_queries.queries_finished += queries;
223 - replication_queries.points_read += points_read;
224 - replication_queries.points_generated += points_generated;
225 - netdata_spinlock_unlock(&replication_queries.spinlock);
226 -
227 - return before;
382 + q->points_read = points_read;
383 + q->points_generated = points_generated;
384 + return replication_query_finalize(q, true);
385 }
386
230 -static void replicate_chart_collection_state(BUFFER *wb, RRDSET *st) {
387 +static void replication_send_chart_collection_state(BUFFER *wb, RRDSET *st) {
388 RRDDIM *rd;
389 rrddim_foreach_read(rd, st) {
390 if(!rd->exposed) continue;
@@ -248,56 +405,23 @@ static void replicate_chart_collection_state(BUFFER *wb, RRDSET *st) {
405 );
406 }
407
251 -bool replicate_chart_response(RRDHOST *host, RRDSET *st, bool start_streaming, time_t after, time_t before) {
252 - time_t query_after = after;
253 - time_t query_before = before;
254 - time_t now = now_realtime_sec();
255 - time_t tolerance = 2; // sometimes from the time we get this value, to the time we check,
256 - // a data collection has been made
257 - // so, we give this tolerance to detect invalid timestamps
258 -
259 - // find the first entry we have
260 - time_t first_entry_local = rrdset_first_entry_t(st);
261 - if(first_entry_local > now + tolerance) {
262 - internal_error(true,
263 - "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s' db first time %llu is in the future (now is %llu)",
264 - rrdhost_hostname(st->rrdhost), rrdset_id(st),
265 - (unsigned long long)first_entry_local, (unsigned long long)now);
266 - first_entry_local = now;
267 - }
408 +static struct replication_query *replication_response_prepare(RRDSET *st, bool start_streaming, time_t requested_after, time_t requested_before) {
409 + time_t query_after = requested_after;
410 + time_t query_before = requested_before;
411 + time_t wall_clock_time = now_realtime_sec();
412
269 - if (query_after < first_entry_local)
270 - query_after = first_entry_local;
271 -
272 - // find the latest entry we have
273 - time_t last_entry_local = st->last_updated.tv_sec;
274 - if(!last_entry_local) {
275 - internal_error(true,
276 - "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s' RRDSET reports last updated time zero.",
277 - rrdhost_hostname(st->rrdhost), rrdset_id(st));
278 - last_entry_local = rrdset_last_entry_t(st);
279 - if(!last_entry_local) {
280 - internal_error(true,
281 - "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s' db reports last time zero.",
282 - rrdhost_hostname(st->rrdhost), rrdset_id(st));
283 - last_entry_local = now;
284 - }
285 - }
413 + time_t db_first_entry, db_last_entry;
414 + rrdset_get_retention_of_tier_for_collected_chart(st, &db_first_entry, &db_last_entry, wall_clock_time, 0);
415
287 - if(last_entry_local > now + tolerance) {
288 - internal_error(true,
289 - "STREAM_SENDER REPLAY ERROR: 'host:%s/chart:%s' last updated time %llu is in the future (now is %llu)",
290 - rrdhost_hostname(st->rrdhost), rrdset_id(st),
291 - (unsigned long long)last_entry_local, (unsigned long long)now);
292 - last_entry_local = now;
293 - }
416 + if (query_after < db_first_entry)
417 + query_after = db_first_entry;
418
295 - if (query_before > last_entry_local)
296 - query_before = last_entry_local;
419 + if (query_before > db_last_entry)
420 + query_before = db_last_entry;
421
422 // if the parent asked us to start streaming, then fill the rest with the data that we have
423 if (start_streaming)
300 - query_before = last_entry_local;
424 + query_before = db_last_entry;
425
426 if (query_after > query_before) {
427 time_t tmp = query_before;
@@ -305,7 +429,29 @@ bool replicate_chart_response(RRDHOST *host, RRDSET *st, bool start_streaming, t
429 query_after = tmp;
430 }
431
308 - bool enable_streaming = (start_streaming || query_before == last_entry_local || !after || !before) ? true : false;
432 + bool enable_streaming = (start_streaming || query_before == db_last_entry || !requested_after || !requested_before) ? true : false;
433 +
434 + return replication_query_prepare(
435 + st,
436 + db_first_entry, db_last_entry,
437 + requested_after, requested_before,
438 + query_after, query_before, enable_streaming,
439 + wall_clock_time);
440 +}
441 +
442 +void replication_response_cancel_and_finalize(struct replication_query *q) {
443 + replication_query_finalize(q, false);
444 +}
445 +
446 +static bool sender_is_still_connected_for_this_request(struct replication_request *rq);
447 +
448 +bool replication_response_execute_and_finalize(struct replication_query *q) {
449 + struct replication_request *rq = q->rq;
450 + RRDSET *st = q->st;
451 + RRDHOST *host = st->rrdhost;
452 + time_t after = q->request.after;
453 + time_t before; // the query will report this
454 + bool enable_streaming = q->query.enable_streaming;
455
456 // we might want to optimize this by filling a temporary buffer
457 // and copying the result to the host's buffer in order to avoid
@@ -314,25 +460,22 @@ bool replicate_chart_response(RRDHOST *host, RRDSET *st, bool start_streaming, t
460
461 buffer_sprintf(wb, PLUGINSD_KEYWORD_REPLAY_BEGIN " \"%s\"\n", rrdset_id(st));
462
317 - if(after != 0 && before != 0)
318 - before = replicate_chart_timeframe(wb, st, query_after, query_before, enable_streaming, now);
319 - else {
320 - after = 0;
321 - before = 0;
322 - enable_streaming = true;
323 - }
463 + bool locked_data_collection = q->query.locked_data_collection;
464 + q->query.locked_data_collection = false;
465 +
466 + before = replication_query_execute_and_finalize(wb, q);
467 +
468 + // IMPORTANT: q is invalid now
469 + q = NULL;
470
471 // get again the world clock time
326 - time_t world_clock_time = now_realtime_sec();
327 - if(enable_streaming) {
328 - if(now < world_clock_time) {
329 - // we needed time to execute this request
330 - // so, the parent will need to replicate more data
331 - enable_streaming = false;
332 - }
333 - else
334 - replicate_chart_collection_state(wb, st);
335 - }
472 + if(enable_streaming)
473 + replication_send_chart_collection_state(wb, st);
474 +
475 + // get a fresh retention to send to the parent
476 + time_t wall_clock_time = now_realtime_sec();
477 + time_t db_first_entry, db_last_entry;
478 + rrdset_get_retention_of_tier_for_collected_chart(st, &db_first_entry, &db_last_entry, wall_clock_time, 0);
479
480 // end with first/last entries we have, and the first start time and
481 // last end time of the data we sent
@@ -342,7 +485,7 @@ bool replicate_chart_response(RRDHOST *host, RRDSET *st, bool start_streaming, t
485 (int)st->update_every
486
487 // child first db time, child end db time
345 - , (unsigned long long)first_entry_local, (unsigned long long)last_entry_local
488 + , (unsigned long long)db_first_entry, (unsigned long long)db_last_entry
489
490 // start streaming boolean
491 , enable_streaming ? "true" : "false"
@@ -351,13 +494,37 @@ bool replicate_chart_response(RRDHOST *host, RRDSET *st, bool start_streaming, t
494 , (unsigned long long)after, (unsigned long long)before
495
496 // child world clock time
354 - , (unsigned long long)world_clock_time
497 + , (unsigned long long)wall_clock_time
498 );
499
500 worker_is_busy(WORKER_JOB_BUFFER_COMMIT);
501 sender_commit(host->sender, wb);
502 worker_is_busy(WORKER_JOB_CLEANUP);
503
504 + if(enable_streaming) {
505 + if(sender_is_still_connected_for_this_request(rq)) {
506 + // enable normal streaming if we have to
507 + // but only if the sender buffer has not been flushed since we started
508 +
509 + if(rrdset_flag_check(st, RRDSET_FLAG_SENDER_REPLICATION_IN_PROGRESS)) {
510 + rrdset_flag_clear(st, RRDSET_FLAG_SENDER_REPLICATION_IN_PROGRESS);
511 + rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
512 + rrdhost_sender_replicating_charts_minus_one(st->rrdhost);
513 +
514 +#ifdef NETDATA_LOG_REPLICATION_REQUESTS
515 + internal_error(true, "STREAM_SENDER REPLAY: 'host:%s/chart:%s' streaming starts",
516 + rrdhost_hostname(st->rrdhost), rrdset_id(st));
517 +#endif
518 + }
519 + else
520 + internal_error(true, "REPLAY ERROR: 'host:%s/chart:%s' received start streaming command, but the chart is not in progress replicating",
521 + rrdhost_hostname(st->rrdhost), rrdset_id(st));
522 + }
523 + }
524 +
525 + if(locked_data_collection)
526 + netdata_spinlock_unlock(&st->data_collection_lock);
527 +
528 return enable_streaming;
529 }
530
@@ -478,8 +645,8 @@ bool replicate_chart_request(send_command callback, void *callback_data, RRDHOST
645 },
646
647 .local_db = {
481 - .first_entry_t = rrdset_first_entry_t(st),
482 - .last_entry_t = rrdset_last_entry_t(st),
648 + .first_entry_t = rrdset_first_entry_s(st),
649 + .last_entry_t = rrdset_last_entry_s(st),
650 .last_entry_t_adjusted_to_now = false,
651 .now = now_realtime_sec(),
652 },
@@ -585,13 +752,20 @@ struct replication_request {
752 STRING *chart_id; // the chart of the request
753 time_t after; // the start time of the query (maybe zero) key for sorting (JudyL)
754 time_t before; // the end time of the query (maybe zero)
588 - bool start_streaming; // true, when the parent wants to send the rest of the data (before is overwritten) and enable normal streaming
755
756 usec_t sender_last_flush_ut; // the timestamp of the sender, at the time we indexed this request
757 Word_t unique_id; // auto-increment, later requests have bigger
592 - bool found; // used as a result boolean for the find call
758 +
759 + bool start_streaming; // true, when the parent wants to send the rest of the data (before is overwritten) and enable normal streaming
760 bool indexed_in_judy; // true when the request is indexed in judy
761 bool not_indexed_buffer_full; // true when the request is not indexed because the sender is full
762 + bool not_indexed_preprocessing; // true when the request is not indexed, but it is pending in preprocessing
763 +
764 + // prepare ahead members - preprocessing
765 + bool found; // used as a result boolean for the find call
766 + bool executed; // used to detect if we have skipped requests while preprocessing
767 + RRDSET *st; // caching of the chart during preprocessing
768 + struct replication_query *q; // the preprocessing query initialization
769 };
770
771 // replication sort entry in JudyL array
@@ -734,6 +908,7 @@ static struct replication_sort_entry *replication_sort_entry_create_unsafe(struc
908 rq->unique_id = rse->unique_id;
909 rq->indexed_in_judy = false;
910 rq->not_indexed_buffer_full = false;
911 + rq->not_indexed_preprocessing = false;
912 return rse;
913 }
914
@@ -747,6 +922,7 @@ static void replication_sort_entry_add(struct replication_request *rq) {
922 if(rrdpush_sender_replication_buffer_full_get(rq->sender)) {
923 rq->indexed_in_judy = false;
924 rq->not_indexed_buffer_full = true;
925 + rq->not_indexed_preprocessing = false;
926 replication_globals.unsafe.pending_no_room++;
927 replication_recursive_unlock();
928 return;
@@ -780,6 +956,7 @@ static void replication_sort_entry_add(struct replication_request *rq) {
956 *item = rse;
957 rq->indexed_in_judy = true;
958 rq->not_indexed_buffer_full = false;
959 + rq->not_indexed_preprocessing = false;
960
961 if(!replication_globals.unsafe.first_time_t || rq->after < replication_globals.unsafe.first_time_t)
962 replication_globals.unsafe.first_time_t = rq->after;
@@ -787,7 +964,7 @@ static void replication_sort_entry_add(struct replication_request *rq) {
964 replication_recursive_unlock();
965 }
966
790 -static bool replication_sort_entry_unlink_and_free_unsafe(struct replication_sort_entry *rse, Pvoid_t **inner_judy_ppptr) {
967 +static bool replication_sort_entry_unlink_and_free_unsafe(struct replication_sort_entry *rse, Pvoid_t **inner_judy_ppptr, bool preprocessing) {
968 fatal_when_replication_is_not_locked_for_me();
969
970 bool inner_judy_deleted = false;
@@ -798,6 +975,7 @@ static bool replication_sort_entry_unlink_and_free_unsafe(struct replication_sor
975 rrdpush_sender_pending_replication_requests_minus_one(rse->rq->sender);
976
977 rse->rq->indexed_in_judy = false;
978 + rse->rq->not_indexed_preprocessing = preprocessing;
979
980 // delete it from the inner judy
981 JudyLDel(*inner_judy_ppptr, rse->rq->unique_id, PJE0);
@@ -826,7 +1004,7 @@ static void replication_sort_entry_del(struct replication_request *rq, bool buff
1004 Pvoid_t *our_item_pptr = JudyLGet(*inner_judy_pptr, rq->unique_id, PJE0);
1005 if (our_item_pptr) {
1006 rse_to_delete = *our_item_pptr;
829 - replication_sort_entry_unlink_and_free_unsafe(rse_to_delete, &inner_judy_pptr);
1007 + replication_sort_entry_unlink_and_free_unsafe(rse_to_delete, &inner_judy_pptr, false);
1008
1009 if(buffer_full) {
1010 replication_globals.unsafe.pending_no_room++;
@@ -844,13 +1022,6 @@ static void replication_sort_entry_del(struct replication_request *rq, bool buff
1022 replication_recursive_unlock();
1023 }
1024
847 -static inline PPvoid_t JudyLFirstOrNext(Pcvoid_t PArray, Word_t * PIndex, bool first) {
848 - if(unlikely(first))
849 - return JudyLFirst(PArray, PIndex, PJE0);
850 -
851 - return JudyLNext(PArray, PIndex, PJE0);
852 -}
853 -
1025 static struct replication_request replication_request_get_first_available() {
1026 Pvoid_t *inner_judy_pptr;
1027
@@ -881,7 +1052,7 @@ static struct replication_request replication_request_get_first_available() {
1052 }
1053
1054 bool find_same_after = true;
884 - while (!rq_to_return.found && (inner_judy_pptr = JudyLFirstOrNext(replication_globals.unsafe.queue.JudyL_array, &replication_globals.unsafe.queue.after, find_same_after))) {
1055 + while (!rq_to_return.found && (inner_judy_pptr = JudyLFirstThenNext(replication_globals.unsafe.queue.JudyL_array, &replication_globals.unsafe.queue.after, &find_same_after))) {
1056 Pvoid_t *our_item_pptr;
1057
1058 if(unlikely(round == 2 && replication_globals.unsafe.queue.after > started_after))
@@ -898,14 +1069,11 @@ static struct replication_request replication_request_get_first_available() {
1069 // set the return result to found
1070 rq_to_return.found = true;
1071
901 - if (replication_sort_entry_unlink_and_free_unsafe(rse, &inner_judy_pptr))
1072 + if (replication_sort_entry_unlink_and_free_unsafe(rse, &inner_judy_pptr, true))
1073 // we removed the item from the outer JudyL
1074 break;
1075 }
1076
906 - // call JudyLNext from now on
907 - find_same_after = false;
908 -
1077 // prepare for the next iteration on the outer loop
1078 replication_globals.unsafe.queue.unique_id = 0;
1079 }
@@ -945,7 +1113,7 @@ static bool replication_request_conflict_callback(const DICTIONARY_ITEM *item __
1113
1114 replication_recursive_lock();
1115
948 - if(!rq->indexed_in_judy && rq->not_indexed_buffer_full) {
1116 + if(!rq->indexed_in_judy && rq->not_indexed_buffer_full && !rq->not_indexed_preprocessing) {
1117 // we can replace this command
1118 internal_error(
1119 true,
@@ -958,7 +1126,7 @@ static bool replication_request_conflict_callback(const DICTIONARY_ITEM *item __
1126 rq->before = rq_new->before;
1127 rq->start_streaming = rq_new->start_streaming;
1128 }
961 - else if(!rq->indexed_in_judy) {
1129 + else if(!rq->indexed_in_judy && !rq->not_indexed_preprocessing) {
1130 replication_sort_entry_add(rq);
1131 internal_error(
1132 true,
@@ -1001,49 +1169,44 @@ static void replication_request_delete_callback(const DICTIONARY_ITEM *item __ma
1169 string_freez(rq->chart_id);
1170 }
1171
1172 +static bool sender_is_still_connected_for_this_request(struct replication_request *rq) {
1173 + return rq->sender_last_flush_ut == rrdpush_sender_get_flush_time(rq->sender);
1174 +};
1175 +
1176 static bool replication_execute_request(struct replication_request *rq, bool workers) {
1177 bool ret = false;
1178
1007 - if(likely(workers))
1008 - worker_is_busy(WORKER_JOB_FIND_CHART);
1179 + if(!rq->st) {
1180 + if(likely(workers))
1181 + worker_is_busy(WORKER_JOB_FIND_CHART);
1182
1010 - RRDSET *st = rrdset_find(rq->sender->host, string2str(rq->chart_id));
1011 - if(!st) {
1183 + rq->st = rrdset_find(rq->sender->host, string2str(rq->chart_id));
1184 + }
1185 +
1186 + if(!rq->st) {
1187 internal_error(true, "REPLAY ERROR: 'host:%s/chart:%s' not found",
1188 rrdhost_hostname(rq->sender->host), string2str(rq->chart_id));
1189
1190 goto cleanup;
1191 }
1192
1018 - if(likely(workers))
1019 - worker_is_busy(WORKER_JOB_QUERYING);
1020 -
1193 netdata_thread_disable_cancelability();
1194
1023 - // send the replication data
1024 - bool start_streaming = replicate_chart_response(
1025 - st->rrdhost, st, rq->start_streaming, rq->after, rq->before);
1195 + if(!rq->q) {
1196 + if(likely(workers))
1197 + worker_is_busy(WORKER_JOB_PREPARE_QUERY);
1198
1027 - netdata_thread_enable_cancelability();
1199 + rq->q = replication_response_prepare(rq->st, rq->start_streaming, rq->after, rq->before);
1200 + }
1201
1029 - if(start_streaming && rq->sender_last_flush_ut == rrdpush_sender_get_flush_time(rq->sender)) {
1030 - // enable normal streaming if we have to
1031 - // but only if the sender buffer has not been flushed since we started
1202 + if(likely(workers))
1203 + worker_is_busy(WORKER_JOB_QUERYING);
1204
1033 - if(rrdset_flag_check(st, RRDSET_FLAG_SENDER_REPLICATION_IN_PROGRESS)) {
1034 - rrdset_flag_clear(st, RRDSET_FLAG_SENDER_REPLICATION_IN_PROGRESS);
1035 - rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
1036 - rrdhost_sender_replicating_charts_minus_one(st->rrdhost);
1205 + // send the replication data
1206 + rq->q->rq = rq;
1207 + replication_response_execute_and_finalize(rq->q);
1208
1038 -#ifdef NETDATA_LOG_REPLICATION_REQUESTS
1039 - internal_error(true, "STREAM_SENDER REPLAY: 'host:%s/chart:%s' streaming starts",
1040 - rrdhost_hostname(st->rrdhost), rrdset_id(st));
1041 -#endif
1042 - }
1043 - else
1044 - internal_error(true, "REPLAY ERROR: 'host:%s/chart:%s' received start streaming command, but the chart is not in progress replicating",
1045 - rrdhost_hostname(st->rrdhost), string2str(rq->chart_id));
1046 - }
1209 + netdata_thread_enable_cancelability();
1210
1211 __atomic_add_fetch(&replication_globals.atomic.executed, 1, __ATOMIC_RELAXED);
1212
@@ -1068,6 +1231,7 @@ void replication_add_request(struct sender_state *sender, const char *chart_id,
1231 .sender_last_flush_ut = rrdpush_sender_get_flush_time(sender),
1232 .indexed_in_judy = false,
1233 .not_indexed_buffer_full = false,
1234 + .not_indexed_preprocessing = false,
1235 };
1236
1237 if(start_streaming && rrdpush_sender_get_buffer_used_percent(sender) <= STREAMING_START_MAX_SENDER_BUFFER_PERCENTAGE_ALLOWED)
@@ -1079,9 +1243,7 @@ void replication_add_request(struct sender_state *sender, const char *chart_id,
1243
1244 void replication_sender_delete_pending_requests(struct sender_state *sender) {
1245 // allow the dictionary destructor to go faster on locks
1082 - replication_recursive_lock();
1246 dictionary_flush(sender->replication.requests);
1084 - replication_recursive_unlock();
1247 }
1248
1249 void replication_init_sender(struct sender_state *sender) {
@@ -1107,9 +1269,8 @@ void replication_recalculate_buffer_used_ratio_unsafe(struct sender_state *s) {
1269
1270 struct replication_request *rq;
1271 dfe_start_read(s->replication.requests, rq) {
1110 - if(rq->indexed_in_judy && !rq->not_indexed_buffer_full) {
1272 + if(rq->indexed_in_judy)
1273 replication_sort_entry_del(rq, true);
1112 - }
1274 }
1275 dfe_done(rq);
1276
@@ -1122,9 +1283,8 @@ void replication_recalculate_buffer_used_ratio_unsafe(struct sender_state *s) {
1283
1284 struct replication_request *rq;
1285 dfe_start_read(s->replication.requests, rq) {
1125 - if(!rq->indexed_in_judy && rq->not_indexed_buffer_full) {
1286 + if(!rq->indexed_in_judy && (rq->not_indexed_buffer_full || rq->not_indexed_preprocessing))
1287 replication_sort_entry_add(rq);
1127 - }
1288 }
1289 dfe_done(rq);
1290
@@ -1214,6 +1374,7 @@ static void replication_initialize_workers(bool master) {
1374 worker_register_job_name(WORKER_JOB_QUERYING, "querying");
1375 worker_register_job_name(WORKER_JOB_DELETE_ENTRY, "dict delete");
1376 worker_register_job_name(WORKER_JOB_FIND_CHART, "find chart");
1377 + worker_register_job_name(WORKER_JOB_PREPARE_QUERY, "prepare query");
1378 worker_register_job_name(WORKER_JOB_CHECK_CONSISTENCY, "check consistency");
1379 worker_register_job_name(WORKER_JOB_BUFFER_COMMIT, "commit");
1380 worker_register_job_name(WORKER_JOB_CLEANUP, "cleanup");
@@ -1236,23 +1397,89 @@ static void replication_initialize_workers(bool master) {
1397 #define REQUEST_CHART_NOT_FOUND (-2)
1398
1399 static int replication_execute_next_pending_request(void) {
1239 - worker_is_busy(WORKER_JOB_FIND_NEXT);
1240 - struct replication_request rq = replication_request_get_first_available();
1400 + static __thread int max_requests_ahead = 0;
1401 + static __thread struct replication_request *rqs = NULL;
1402 + static __thread int rqs_last_executed = 0, rqs_last_prepared = 0;
1403 + static __thread size_t queue_rounds = 0; (void)queue_rounds;
1404 + struct replication_request *rq;
1405 +
1406 + if(unlikely(!rqs)) {
1407 + max_requests_ahead = libuv_worker_threads * 2;
1408 + if(max_requests_ahead < 2)
1409 + max_requests_ahead = 2;
1410 +
1411 + rqs = callocz(max_requests_ahead, sizeof(struct replication_request));
1412 + }
1413 +
1414 + // fill the queue
1415 + do {
1416 + if(++rqs_last_prepared >= max_requests_ahead) {
1417 + rqs_last_prepared = 0;
1418 + queue_rounds++;
1419 + }
1420 +
1421 + internal_fatal(queue_rounds > 1 && !rqs[rqs_last_prepared].executed,
1422 + "REPLAY FATAL: query has not been executed!");
1423 +
1424 + worker_is_busy(WORKER_JOB_FIND_NEXT);
1425 + rqs[rqs_last_prepared] = replication_request_get_first_available();
1426 + rq = &rqs[rqs_last_prepared];
1427 +
1428 + if(rq->found) {
1429 + if (!rq->st) {
1430 + worker_is_busy(WORKER_JOB_FIND_CHART);
1431 + rq->st = rrdset_find(rq->sender->host, string2str(rq->chart_id));
1432 + }
1433
1242 - if(unlikely(!rq.found)) {
1434 + if (rq->st && !rq->q) {
1435 + worker_is_busy(WORKER_JOB_PREPARE_QUERY);
1436 + rq->q = replication_response_prepare(rq->st, rq->start_streaming, rq->after, rq->before);
1437 + }
1438 + }
1439 +
1440 + } while(rq->found && rqs_last_prepared != rqs_last_executed);
1441 +
1442 + // pick the first usable
1443 + do {
1444 + if (++rqs_last_executed >= max_requests_ahead)
1445 + rqs_last_executed = 0;
1446 +
1447 + rq = &rqs[rqs_last_executed];
1448 + rq->executed = true;
1449 +
1450 + if(rq->found) {
1451 + if (rq->sender_last_flush_ut != rrdpush_sender_get_flush_time(rq->sender)) {
1452 + // the sender has reconnected since this request was queued,
1453 + // we can safely throw it away, since the parent will resend it
1454 + replication_response_cancel_and_finalize(rq->q);
1455 + rq->found = false;
1456 + }
1457 + else if (rrdpush_sender_replication_buffer_full_get(rq->sender)) {
1458 + // the sender buffer is full, so we can ignore this request,
1459 + // it has already been marked as 'preprocessed' in the dictionary,
1460 + // and the sender will put it back in when there is
1461 + // enough room in the buffer for processing replication requests
1462 + replication_response_cancel_and_finalize(rq->q);
1463 + rq->found = false;
1464 + }
1465 + else {
1466 + // we can execute this,
1467 + // delete it from the dictionary
1468 + worker_is_busy(WORKER_JOB_DELETE_ENTRY);
1469 + dictionary_del(rq->sender->replication.requests, string2str(rq->chart_id));
1470 + }
1471 + }
1472 +
1473 + } while(!rq->found && rqs_last_executed != rqs_last_prepared);
1474 +
1475 + if(unlikely(!rq->found)) {
1476 worker_is_idle();
1477 return REQUEST_QUEUE_EMPTY;
1478 }
1479
1247 - // delete the request from the dictionary
1248 - worker_is_busy(WORKER_JOB_DELETE_ENTRY);
1249 - if(!dictionary_del(rq.sender->replication.requests, string2str(rq.chart_id)))
1250 - error("REPLAY ERROR: 'host:%s/chart:%s' failed to be deleted from sender pending charts index",
1251 - rrdhost_hostname(rq.sender->host), string2str(rq.chart_id));
1252 -
1253 - replication_set_latest_first_time(rq.after);
1480 + replication_set_latest_first_time(rq->after);
1481
1255 - if(unlikely(!replication_execute_request(&rq, true))) {
1482 + if(unlikely(!replication_execute_request(rq, true))) {
1483 worker_is_idle();
1484 return REQUEST_CHART_NOT_FOUND;
1485 }
@@ -1270,7 +1497,7 @@ static void *replication_worker_thread(void *ptr) {
1497
1498 netdata_thread_cleanup_push(replication_worker_cleanup, ptr);
1499
1273 - while(!netdata_exit) {
1500 + while(service_running(SERVICE_REPLICATION)) {
1501 if(unlikely(replication_execute_next_pending_request() == REQUEST_QUEUE_EMPTY)) {
1502 worker_is_busy(WORKER_JOB_WAIT);
1503 worker_is_idle();
@@ -1333,7 +1560,7 @@ void *replication_thread_main(void *ptr __maybe_unused) {
1560 size_t last_executed = 0;
1561 size_t last_sender_resets = 0;
1562
1336 - while(!netdata_exit) {
1563 + while(service_running(SERVICE_REPLICATION)) {
1564
1565 // statistics
1566 usec_t now_mono_ut = now_monotonic_usec();
streaming/rrdpush.c
+327 -245
@@ -108,7 +108,7 @@ int rrdpush_init() {
108 default_rrdpush_seconds_to_replicate = config_get_number(CONFIG_SECTION_DB, "seconds to replicate", default_rrdpush_seconds_to_replicate);
109 default_rrdpush_replication_step = config_get_number(CONFIG_SECTION_DB, "seconds per replication step", default_rrdpush_replication_step);
110
111 - rrdhost_free_orphan_time = config_get_number(CONFIG_SECTION_DB, "cleanup orphan hosts after secs", rrdhost_free_orphan_time);
111 + rrdhost_free_orphan_time_s = config_get_number(CONFIG_SECTION_DB, "cleanup orphan hosts after secs", rrdhost_free_orphan_time_s);
112
113 #ifdef ENABLE_COMPRESSION
114 default_compression_enabled = (unsigned int)appconfig_get_boolean(&stream_config, CONFIG_SECTION_STREAM,
@@ -295,40 +295,14 @@ static inline bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
295 rrdsetvar_print_to_streaming_custom_chart_variables(st, wb);
296
297 if (stream_has_capability(host->sender, STREAM_CAP_REPLICATION)) {
298 - time_t first_entry_local = rrdset_first_entry_t_of_tier(st, 0);
299 - time_t last_entry_local = st->last_updated.tv_sec;
300 -
301 - if(unlikely(!last_entry_local))
302 - last_entry_local = rrdset_last_entry_t(st);
298 + time_t db_first_time_t, db_last_time_t;
299
300 time_t now = now_realtime_sec();
305 - if(unlikely(last_entry_local > now)) {
306 - internal_error(true,
307 - "RRDSET REPLAY ERROR: 'host:%s/chart:%s' last updated time %ld is in the future, adjusting it to now %ld",
308 - rrdhost_hostname(st->rrdhost), rrdset_id(st),
309 - last_entry_local, now);
310 - last_entry_local = now;
311 - }
312 -
313 - if(unlikely(first_entry_local && last_entry_local && first_entry_local >= last_entry_local)) {
314 - internal_error(true,
315 - "RRDSET REPLAY ERROR: 'host:%s/chart:%s' first updated time %ld is equal or bigger than last updated time %ld, adjusting it last updated time - update every",
316 - rrdhost_hostname(st->rrdhost), rrdset_id(st),
317 - first_entry_local, last_entry_local);
318 - first_entry_local = last_entry_local - st->update_every;
319 - }
320 -
321 - if(unlikely(!first_entry_local && last_entry_local)) {
322 - internal_error(true,
323 - "RRDSET REPLAY ERROR: 'host:%s/chart:%s' first time %ld, last time %ld, setting both to last time",
324 - rrdhost_hostname(st->rrdhost), rrdset_id(st),
325 - first_entry_local, last_entry_local);
326 - first_entry_local = last_entry_local;
327 - }
301 + rrdset_get_retention_of_tier_for_collected_chart(st, &db_first_time_t, &db_last_time_t, now, 0);
302
303 buffer_sprintf(wb, PLUGINSD_KEYWORD_CHART_DEFINITION_END " %llu %llu %llu\n",
330 - (unsigned long long)first_entry_local,
331 - (unsigned long long)last_entry_local,
304 + (unsigned long long)db_first_time_t,
305 + (unsigned long long)db_last_time_t,
306 (unsigned long long)now);
307
308 rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_IN_PROGRESS);
@@ -342,7 +316,7 @@ static inline bool rrdpush_send_chart_definition(BUFFER *wb, RRDSET *st) {
316 #endif
317 }
318
345 - st->upstream_resync_time = st->last_collected_time.tv_sec + (remote_clock_resync_iterations * st->update_every);
319 + st->upstream_resync_time_s = st->last_collected_time.tv_sec + (remote_clock_resync_iterations * st->update_every);
320 return replication_progress;
321 }
322
@@ -352,7 +326,7 @@ static void rrdpush_send_chart_metrics(BUFFER *wb, RRDSET *st, struct sender_sta
326 buffer_fast_strcat(wb, rrdset_id(st), string_strlen(st->id));
327 buffer_fast_strcat(wb, "\" ", 2);
328
355 - if(stream_has_capability(s, STREAM_CAP_REPLICATION) || st->last_collected_time.tv_sec > st->upstream_resync_time)
329 + if(stream_has_capability(s, STREAM_CAP_REPLICATION) || st->last_collected_time.tv_sec > st->upstream_resync_time_s)
330 buffer_print_llu(wb, st->usec_since_last_update);
331 else
332 buffer_fast_strcat(wb, "0", 1);
@@ -581,25 +555,16 @@ void rrdpush_destinations_free(RRDHOST *host) {
555
556 // Either the receiver lost the connection or the host is being destroyed.
557 // The sender mutex guards thread creation, any spurious data is wiped on reconnection.
584 -void rrdpush_sender_thread_stop(RRDHOST *host) {
585 -
558 +void rrdpush_sender_thread_stop(RRDHOST *host, const char *reason, bool wait) {
559 if (!host->sender)
560 return;
561
562 netdata_mutex_lock(&host->sender->mutex);
590 - netdata_thread_t thr = 0;
563
564 if(rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN)) {
593 - rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN);
594 -
595 - info("STREAM %s [send]: signaling sending thread to stop...", rrdhost_hostname(host));
596 -
597 - // signal the thread that we want to join it
598 - rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_JOIN);
565
600 - // copy the thread id, so that we will be waiting for the right one
601 - // even if a new one has been spawn
602 - thr = host->rrdpush_sender_thread;
566 + host->sender->exit.shutdown = true;
567 + host->sender->exit.reason = reason;
568
569 // signal it to cancel
570 netdata_thread_cancel(host->rrdpush_sender_thread);
@@ -607,11 +572,14 @@ void rrdpush_sender_thread_stop(RRDHOST *host) {
572
573 netdata_mutex_unlock(&host->sender->mutex);
574
610 - if(thr != 0) {
611 - info("STREAM %s [send]: waiting for the sending thread to stop...", rrdhost_hostname(host));
612 - void *result;
613 - netdata_thread_join(thr, &result);
614 - info("STREAM %s [send]: sending thread has exited.", rrdhost_hostname(host));
575 + if(wait) {
576 + netdata_mutex_lock(&host->sender->mutex);
577 + while(host->sender->tid) {
578 + netdata_mutex_unlock(&host->sender->mutex);
579 + sleep_usec(10 * USEC_PER_MS);
580 + netdata_mutex_lock(&host->sender->mutex);
581 + }
582 + netdata_mutex_unlock(&host->sender->mutex);
583 }
584 }
585
@@ -631,7 +599,7 @@ static void rrdpush_sender_thread_spawn(RRDHOST *host) {
599 char tag[NETDATA_THREAD_TAG_MAX + 1];
600 snprintfz(tag, NETDATA_THREAD_TAG_MAX, "STREAM_SENDER[%s]", rrdhost_hostname(host));
601
634 - if(netdata_thread_create(&host->rrdpush_sender_thread, tag, NETDATA_THREAD_OPTION_JOINABLE, rrdpush_sender_thread, (void *) host->sender))
602 + if(netdata_thread_create(&host->rrdpush_sender_thread, tag, NETDATA_THREAD_OPTION_DEFAULT, rrdpush_sender_thread, (void *) host->sender))
603 error("STREAM %s [send]: failed to create new thread for client.", rrdhost_hostname(host));
604 else
605 rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN);
@@ -645,7 +613,7 @@ int rrdpush_receiver_permission_denied(struct web_client *w) {
613 // to prevent an attacker from gaining info about the error
614 buffer_flush(w->response.data);
615 buffer_sprintf(w->response.data, "You are not permitted to access this. Check the logs for more info.");
648 - return 401;
616 + return HTTP_RESP_UNAUTHORIZED;
617 }
618
619 int rrdpush_receiver_too_busy_now(struct web_client *w) {
@@ -653,21 +621,38 @@ int rrdpush_receiver_too_busy_now(struct web_client *w) {
621 // to prevent an attacker from gaining info about the error
622 buffer_flush(w->response.data);
623 buffer_sprintf(w->response.data, "The server is too busy now to accept this request. Try later.");
656 - return 503;
624 + return HTTP_RESP_SERVICE_UNAVAILABLE;
625 }
626
627 void *rrdpush_receiver_thread(void *ptr);
628 int rrdpush_receiver_thread_spawn(struct web_client *w, char *url) {
661 - // info("clients wants to STREAM metrics.");
629
663 - char *key = NULL, *hostname = NULL, *registry_hostname = NULL, *machine_guid = NULL, *os = "unknown", *timezone = "unknown", *abbrev_timezone = "UTC", *tags = NULL;
664 - int32_t utc_offset = 0;
665 - int update_every = default_rrd_update_every;
666 - uint32_t stream_version = UINT_MAX;
667 - char buf[GUID_LEN + 1];
630 + if(!service_running(ABILITY_STREAMING_CONNECTIONS))
631 + return rrdpush_receiver_too_busy_now(w);
632 +
633 + struct receiver_state *rpt = callocz(1, sizeof(*rpt));
634 + rpt->last_msg_t = now_realtime_sec();
635 + rpt->capabilities = STREAM_CAP_INVALID;
636 +
637 + rpt->system_info = callocz(1, sizeof(struct rrdhost_system_info));
638 + rpt->system_info->hops = 1;
639 +
640 + rpt->fd = w->ifd;
641 + rpt->client_ip = strdupz(w->client_ip);
642 + rpt->client_port = strdupz(w->client_port);
643 +
644 + rpt->config.update_every = default_rrd_update_every;
645 +
646 +#ifdef ENABLE_HTTPS
647 + rpt->ssl.conn = w->ssl.conn;
648 + rpt->ssl.flags = w->ssl.flags;
649 +
650 + w->ssl.conn = NULL;
651 + w->ssl.flags = NETDATA_SSL_START;
652 +#endif
653 +
654 + // parse the parameters and fill rpt and rpt->system_info
655
669 - struct rrdhost_system_info *system_info = callocz(1, sizeof(struct rrdhost_system_info));
670 - system_info->hops = 1;
656 while(url) {
657 char *value = mystrsep(&url, "&");
658 if(!value || !*value) continue;
@@ -676,178 +661,307 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *url) {
661 if(!name || !*name) continue;
662 if(!value || !*value) continue;
663
679 - if(!strcmp(name, "key"))
680 - key = value;
681 - else if(!strcmp(name, "hostname"))
682 - hostname = value;
683 - else if(!strcmp(name, "registry_hostname"))
684 - registry_hostname = value;
685 - else if(!strcmp(name, "machine_guid"))
686 - machine_guid = value;
664 + if(!strcmp(name, "key") && !rpt->key)
665 + rpt->key = strdupz(value);
666 +
667 + else if(!strcmp(name, "hostname") && !rpt->hostname)
668 + rpt->hostname = strdupz(value);
669 +
670 + else if(!strcmp(name, "registry_hostname") && !rpt->registry_hostname)
671 + rpt->registry_hostname = strdupz(value);
672 +
673 + else if(!strcmp(name, "machine_guid") && !rpt->machine_guid)
674 + rpt->machine_guid = strdupz(value);
675 +
676 else if(!strcmp(name, "update_every"))
688 - update_every = (int)strtoul(value, NULL, 0);
689 - else if(!strcmp(name, "os"))
690 - os = value;
691 - else if(!strcmp(name, "timezone"))
692 - timezone = value;
693 - else if(!strcmp(name, "abbrev_timezone"))
694 - abbrev_timezone = value;
677 + rpt->config.update_every = (int)strtoul(value, NULL, 0);
678 +
679 + else if(!strcmp(name, "os") && !rpt->os)
680 + rpt->os = strdupz(value);
681 +
682 + else if(!strcmp(name, "timezone") && !rpt->timezone)
683 + rpt->timezone = strdupz(value);
684 +
685 + else if(!strcmp(name, "abbrev_timezone") && !rpt->abbrev_timezone)
686 + rpt->abbrev_timezone = strdupz(value);
687 +
688 else if(!strcmp(name, "utc_offset"))
696 - utc_offset = (int32_t)strtol(value, NULL, 0);
689 + rpt->utc_offset = (int32_t)strtol(value, NULL, 0);
690 +
691 else if(!strcmp(name, "hops"))
698 - system_info->hops = (uint16_t) strtoul(value, NULL, 0);
692 + rpt->system_info->hops = (uint16_t) strtoul(value, NULL, 0);
693 +
694 else if(!strcmp(name, "ml_capable"))
700 - system_info->ml_capable = strtoul(value, NULL, 0);
695 + rpt->system_info->ml_capable = strtoul(value, NULL, 0);
696 +
697 else if(!strcmp(name, "ml_enabled"))
702 - system_info->ml_enabled = strtoul(value, NULL, 0);
698 + rpt->system_info->ml_enabled = strtoul(value, NULL, 0);
699 +
700 else if(!strcmp(name, "mc_version"))
704 - system_info->mc_version = strtoul(value, NULL, 0);
705 - else if(!strcmp(name, "tags"))
706 - tags = value;
707 - else if(!strcmp(name, "ver"))
708 - stream_version = convert_stream_version_to_capabilities(strtoul(value, NULL, 0));
701 + rpt->system_info->mc_version = strtoul(value, NULL, 0);
702 +
703 + else if(!strcmp(name, "tags") && !rpt->tags)
704 + rpt->tags = strdupz(value);
705 +
706 + else if(!strcmp(name, "ver") && (rpt->capabilities & STREAM_CAP_INVALID))
707 + rpt->capabilities = convert_stream_version_to_capabilities(strtoul(value, NULL, 0));
708 +
709 else {
710 // An old Netdata child does not have a compatible streaming protocol, map to something sane.
711 if (!strcmp(name, "NETDATA_SYSTEM_OS_NAME"))
712 name = "NETDATA_HOST_OS_NAME";
713 +
714 else if (!strcmp(name, "NETDATA_SYSTEM_OS_ID"))
715 name = "NETDATA_HOST_OS_ID";
716 +
717 else if (!strcmp(name, "NETDATA_SYSTEM_OS_ID_LIKE"))
718 name = "NETDATA_HOST_OS_ID_LIKE";
719 +
720 else if (!strcmp(name, "NETDATA_SYSTEM_OS_VERSION"))
721 name = "NETDATA_HOST_OS_VERSION";
722 +
723 else if (!strcmp(name, "NETDATA_SYSTEM_OS_VERSION_ID"))
724 name = "NETDATA_HOST_OS_VERSION_ID";
725 +
726 else if (!strcmp(name, "NETDATA_SYSTEM_OS_DETECTION"))
727 name = "NETDATA_HOST_OS_DETECTION";
723 - else if(!strcmp(name, "NETDATA_PROTOCOL_VERSION") && stream_version == UINT_MAX) {
724 - stream_version = convert_stream_version_to_capabilities(1);
725 - }
728
727 - if (unlikely(rrdhost_set_system_info_variable(system_info, name, value))) {
728 - info("STREAM [receive from [%s]:%s]: request has parameter '%s' = '%s', which is not used.",
729 - w->client_ip, w->client_port, name, value);
729 + else if(!strcmp(name, "NETDATA_PROTOCOL_VERSION") && (rpt->capabilities & STREAM_CAP_INVALID))
730 + rpt->capabilities = convert_stream_version_to_capabilities(1);
731 +
732 + if (unlikely(rrdhost_set_system_info_variable(rpt->system_info, name, value))) {
733 + info("STREAM '%s' [receive from [%s]:%s]: "
734 + "request has parameter '%s' = '%s', which is not used."
735 + , (rpt->hostname && *rpt->hostname) ? rpt->hostname : "-"
736 + , rpt->client_ip, rpt->client_port
737 + , name, value);
738 }
739 }
740 }
741
734 - if (stream_version == UINT_MAX)
735 - stream_version = convert_stream_version_to_capabilities(0);
742 + if (rpt->capabilities & STREAM_CAP_INVALID)
743 + // no version is supplied, assume version 0;
744 + rpt->capabilities = convert_stream_version_to_capabilities(0);
745
737 - if(!key || !*key) {
738 - rrdhost_system_info_free(system_info);
739 - log_stream_connection(w->client_ip, w->client_port, (key && *key)?key:"-", (machine_guid && *machine_guid)?machine_guid:"-", (hostname && *hostname)?hostname:"-", "ACCESS DENIED - NO KEY");
740 - error("STREAM [receive from [%s]:%s]: request without an API key. Forbidding access.", w->client_ip, w->client_port);
741 - return rrdpush_receiver_permission_denied(w);
746 + // find the program name and version
747 + if(w->user_agent && w->user_agent[0]) {
748 + char *t = strchr(w->user_agent, '/');
749 + if(t && *t) {
750 + *t = '\0';
751 + t++;
752 + }
753 +
754 + rpt->program_name = strdupz(w->user_agent);
755 + if(t && *t) rpt->program_version = strdupz(t);
756 }
757
744 - if(!hostname || !*hostname) {
745 - rrdhost_system_info_free(system_info);
746 - log_stream_connection(w->client_ip, w->client_port, (key && *key)?key:"-", (machine_guid && *machine_guid)?machine_guid:"-", (hostname && *hostname)?hostname:"-", "ACCESS DENIED - NO HOSTNAME");
747 - error("STREAM [receive from [%s]:%s]: request without a hostname. Forbidding access.", w->client_ip, w->client_port);
758 + // check if we should accept this connection
759 +
760 + if(!rpt->key || !*rpt->key) {
761 + rrdpush_receive_log_status(
762 + rpt,
763 + "request without an API key",
764 + "NO API KEY PERMISSION DENIED");
765 +
766 + receiver_state_free(rpt);
767 return rrdpush_receiver_permission_denied(w);
768 }
769
751 - if(!machine_guid || !*machine_guid) {
752 - rrdhost_system_info_free(system_info);
753 - log_stream_connection(w->client_ip, w->client_port, (key && *key)?key:"-", (machine_guid && *machine_guid)?machine_guid:"-", (hostname && *hostname)?hostname:"-", "ACCESS DENIED - NO MACHINE GUID");
754 - error("STREAM [receive from [%s]:%s]: request without a machine GUID. Forbidding access.", w->client_ip, w->client_port);
770 + if(!rpt->hostname || !*rpt->hostname) {
771 + rrdpush_receive_log_status(
772 + rpt,
773 + "request without a hostname",
774 + "NO HOSTNAME PERMISSION DENIED");
775 +
776 + receiver_state_free(rpt);
777 return rrdpush_receiver_permission_denied(w);
778 }
779
758 - if(regenerate_guid(key, buf) == -1) {
759 - rrdhost_system_info_free(system_info);
760 - log_stream_connection(w->client_ip, w->client_port, key, machine_guid, hostname, "ACCESS DENIED - INVALID KEY");
761 - error("STREAM [receive from [%s]:%s]: API key '%s' is not valid GUID (use the command uuidgen to generate one). Forbidding access.", w->client_ip, w->client_port, key);
780 + if(!rpt->registry_hostname)
781 + rpt->registry_hostname = strdupz(rpt->hostname);
782 +
783 + if(!rpt->machine_guid || !*rpt->machine_guid) {
784 + rrdpush_receive_log_status(
785 + rpt,
786 + "request without a machine GUID",
787 + "NO MACHINE GUID PERMISSION DENIED");
788 +
789 + receiver_state_free(rpt);
790 return rrdpush_receiver_permission_denied(w);
791 }
792
765 - if(regenerate_guid(machine_guid, buf) == -1) {
766 - rrdhost_system_info_free(system_info);
767 - log_stream_connection(w->client_ip, w->client_port, key, machine_guid, hostname, "ACCESS DENIED - INVALID MACHINE GUID");
768 - error("STREAM [receive from [%s]:%s]: machine GUID '%s' is not GUID. Forbidding access.", w->client_ip, w->client_port, machine_guid);
769 - return rrdpush_receiver_permission_denied(w);
793 + {
794 + char buf[GUID_LEN + 1];
795 +
796 + if (regenerate_guid(rpt->key, buf) == -1) {
797 + rrdpush_receive_log_status(
798 + rpt,
799 + "API key is not a valid UUID (use the command uuidgen to generate one)",
800 + "INVALID API KEY PERMISSION DENIED");
801 +
802 + receiver_state_free(rpt);
803 + return rrdpush_receiver_permission_denied(w);
804 + }
805 +
806 + if (regenerate_guid(rpt->machine_guid, buf) == -1) {
807 + rrdpush_receive_log_status(
808 + rpt,
809 + "machine GUID is not a valid UUID",
810 + "INVALID MACHINE GUID PERMISSION DENIED");
811 +
812 + receiver_state_free(rpt);
813 + return rrdpush_receiver_permission_denied(w);
814 + }
815 }
816
772 - const char *api_key_type = appconfig_get(&stream_config, key, "type", "api");
817 + const char *api_key_type = appconfig_get(&stream_config, rpt->key, "type", "api");
818 if(!api_key_type || !*api_key_type) api_key_type = "unknown";
819 if(strcmp(api_key_type, "api") != 0) {
775 - rrdhost_system_info_free(system_info);
776 - log_stream_connection(w->client_ip, w->client_port, key, machine_guid, hostname, "ACCESS DENIED - API KEY GIVEN IS NOT API KEY");
777 - error("STREAM [receive from [%s]:%s]: API key '%s' is a %s GUID. Forbidding access.", w->client_ip, w->client_port, key, api_key_type);
820 + rrdpush_receive_log_status(
821 + rpt,
822 + "API key is a machine GUID",
823 + "INVALID API KEY PERMISSION DENIED");
824 +
825 + receiver_state_free(rpt);
826 return rrdpush_receiver_permission_denied(w);
827 }
828
781 - if(!appconfig_get_boolean(&stream_config, key, "enabled", 0)) {
782 - rrdhost_system_info_free(system_info);
783 - log_stream_connection(w->client_ip, w->client_port, key, machine_guid, hostname, "ACCESS DENIED - KEY NOT ENABLED");
784 - error("STREAM [receive from [%s]:%s]: API key '%s' is not allowed. Forbidding access.", w->client_ip, w->client_port, key);
829 + if(!appconfig_get_boolean(&stream_config, rpt->key, "enabled", 0)) {
830 + rrdpush_receive_log_status(
831 + rpt,
832 + "API key is not enabled",
833 + "API KEY DISABLED PERMISSION DENIED");
834 +
835 + receiver_state_free(rpt);
836 return rrdpush_receiver_permission_denied(w);
837 }
838
839 {
789 - SIMPLE_PATTERN *key_allow_from = simple_pattern_create(appconfig_get(&stream_config, key, "allow from", "*"), NULL, SIMPLE_PATTERN_EXACT);
840 + SIMPLE_PATTERN *key_allow_from = simple_pattern_create(
841 + appconfig_get(&stream_config, rpt->key, "allow from", "*"),
842 + NULL, SIMPLE_PATTERN_EXACT);
843 +
844 if(key_allow_from) {
845 if(!simple_pattern_matches(key_allow_from, w->client_ip)) {
846 simple_pattern_free(key_allow_from);
793 - rrdhost_system_info_free(system_info);
794 - log_stream_connection(w->client_ip, w->client_port, key, machine_guid, hostname, "ACCESS DENIED - KEY NOT ALLOWED FROM THIS IP");
795 - error("STREAM [receive from [%s]:%s]: API key '%s' is not permitted from this IP. Forbidding access.", w->client_ip, w->client_port, key);
847 +
848 + rrdpush_receive_log_status(
849 + rpt,
850 + "API key is not allowed from this IP",
851 + "NOT ALLOWED IP PERMISSION DENIED");
852 +
853 + receiver_state_free(rpt);
854 return rrdpush_receiver_permission_denied(w);
855 }
856 +
857 simple_pattern_free(key_allow_from);
858 }
859 }
860
802 - const char *machine_guid_type = appconfig_get(&stream_config, machine_guid, "type", "machine");
803 - if(!machine_guid_type || !*machine_guid_type) machine_guid_type = "unknown";
804 - if(strcmp(machine_guid_type, "machine") != 0) {
805 - rrdhost_system_info_free(system_info);
806 - log_stream_connection(w->client_ip, w->client_port, key, machine_guid, hostname, "ACCESS DENIED - MACHINE GUID GIVEN IS NOT A MACHINE GUID");
807 - error("STREAM [receive from [%s]:%s]: machine GUID '%s' is a %s GUID. Forbidding access.", w->client_ip, w->client_port, machine_guid, machine_guid_type);
808 - return rrdpush_receiver_permission_denied(w);
861 + {
862 + const char *machine_guid_type = appconfig_get(&stream_config, rpt->machine_guid, "type", "machine");
863 + if (!machine_guid_type || !*machine_guid_type) machine_guid_type = "unknown";
864 +
865 + if (strcmp(machine_guid_type, "machine") != 0) {
866 + rrdpush_receive_log_status(
867 + rpt,
868 + "machine GUID is an API key",
869 + "INVALID MACHINE GUID PERMISSION DENIED");
870 +
871 + receiver_state_free(rpt);
872 + return rrdpush_receiver_permission_denied(w);
873 + }
874 }
875
811 - if(!appconfig_get_boolean(&stream_config, machine_guid, "enabled", 1)) {
812 - rrdhost_system_info_free(system_info);
813 - log_stream_connection(w->client_ip, w->client_port, key, machine_guid, hostname, "ACCESS DENIED - MACHINE GUID NOT ENABLED");
814 - error("STREAM [receive from [%s]:%s]: machine GUID '%s' is not allowed. Forbidding access.", w->client_ip, w->client_port, machine_guid);
876 + if(!appconfig_get_boolean(&stream_config, rpt->machine_guid, "enabled", 1)) {
877 + rrdpush_receive_log_status(
878 + rpt,
879 + "machine GUID is not enabled",
880 + "MACHINE GUID DISABLED PERMISSION DENIED");
881 +
882 + receiver_state_free(rpt);
883 return rrdpush_receiver_permission_denied(w);
884 }
885
886 {
819 - SIMPLE_PATTERN *machine_allow_from = simple_pattern_create(appconfig_get(&stream_config, machine_guid, "allow from", "*"), NULL, SIMPLE_PATTERN_EXACT);
887 + SIMPLE_PATTERN *machine_allow_from = simple_pattern_create(
888 + appconfig_get(&stream_config, rpt->machine_guid, "allow from", "*"),
889 + NULL, SIMPLE_PATTERN_EXACT);
890 +
891 if(machine_allow_from) {
892 if(!simple_pattern_matches(machine_allow_from, w->client_ip)) {
893 simple_pattern_free(machine_allow_from);
823 - rrdhost_system_info_free(system_info);
824 - log_stream_connection(w->client_ip, w->client_port, key, machine_guid, hostname, "ACCESS DENIED - MACHINE GUID NOT ALLOWED FROM THIS IP");
825 - error("STREAM [receive from [%s]:%s]: Machine GUID '%s' is not permitted from this IP. Forbidding access.", w->client_ip, w->client_port, machine_guid);
894 +
895 + rrdpush_receive_log_status(
896 + rpt,
897 + "machine GUID is not allowed from this IP",
898 + "NOT ALLOWED IP PERMISSION DENIED");
899 +
900 + receiver_state_free(rpt);
901 return rrdpush_receiver_permission_denied(w);
902 }
903 +
904 simple_pattern_free(machine_allow_from);
905 }
906 }
907
908 + if (strcmp(rpt->machine_guid, localhost->machine_guid) == 0) {
909 +
910 + rrdpush_receive_log_status(
911 + rpt,
912 + "machine GUID is my own",
913 + "LOCALHOST PERMISSION DENIED");
914 +
915 + char initial_response[HTTP_HEADER_SIZE + 1];
916 + snprintfz(initial_response, HTTP_HEADER_SIZE, "%s", START_STREAMING_ERROR_SAME_LOCALHOST);
917 +
918 + if(send_timeout(
919 +#ifdef ENABLE_HTTPS
920 + &rpt->ssl,
921 +#endif
922 + rpt->fd, initial_response, strlen(initial_response), 0, 60) != (ssize_t)strlen(initial_response)) {
923 +
924 + error("STREAM '%s' [receive from [%s]:%s]: "
925 + "failed to reply."
926 + , rpt->hostname
927 + , rpt->client_ip, rpt->client_port
928 + );
929 + }
930 +
931 + close(rpt->fd);
932 + receiver_state_free(rpt);
933 + return web_client_socket_is_now_used_for_streaming(w);
934 + }
935 +
936 if(unlikely(web_client_streaming_rate_t > 0)) {
833 - static netdata_mutex_t stream_rate_mutex = NETDATA_MUTEX_INITIALIZER;
834 - static volatile time_t last_stream_accepted_t = 0;
937 + static SPINLOCK spinlock = NETDATA_SPINLOCK_INITIALIZER;
938 + static time_t last_stream_accepted_t = 0;
939
836 - netdata_mutex_lock(&stream_rate_mutex);
940 time_t now = now_realtime_sec();
941 + netdata_spinlock_lock(&spinlock);
942
943 if(unlikely(last_stream_accepted_t == 0))
944 last_stream_accepted_t = now;
945
946 if(now - last_stream_accepted_t < web_client_streaming_rate_t) {
843 - netdata_mutex_unlock(&stream_rate_mutex);
844 - rrdhost_system_info_free(system_info);
845 - error("STREAM [receive from [%s]:%s]: too busy to accept new streaming request. Will be allowed in %ld secs.", w->client_ip, w->client_port, (long)(web_client_streaming_rate_t - (now - last_stream_accepted_t)));
947 + netdata_spinlock_unlock(&spinlock);
948 +
949 + char msg[100 + 1];
950 + snprintfz(msg, 100,
951 + "rate limit, will accept new connection in %ld secs",
952 + (long)(web_client_streaming_rate_t - (now - last_stream_accepted_t)));
953 +
954 + rrdpush_receive_log_status(
955 + rpt,
956 + msg,
957 + "RATE LIMIT TRY LATER");
958 +
959 + receiver_state_free(rpt);
960 return rrdpush_receiver_too_busy_now(w);
961 }
962
963 last_stream_accepted_t = now;
850 - netdata_mutex_unlock(&stream_rate_mutex);
964 + netdata_spinlock_unlock(&spinlock);
965 }
966
967 /*
@@ -858,117 +972,85 @@ int rrdpush_receiver_thread_spawn(struct web_client *w, char *url) {
972 * lock to prevent race-hazard (two threads try to create the host concurrently, one wins and the other does a
973 * lookup to the now-attached structure).
974 */
861 - struct receiver_state *rpt = callocz(1, sizeof(*rpt));
975
863 - rrd_rdlock();
864 - RRDHOST *host = rrdhost_find_by_guid(machine_guid);
865 - if (unlikely(host && rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))) /* Ignore archived hosts. */
866 - host = NULL;
867 - if (host) {
868 - rrdhost_wrlock(host);
869 - netdata_mutex_lock(&host->receiver_lock);
870 - rrdhost_flag_clear(host, RRDHOST_FLAG_ORPHAN);
871 - host->senders_disconnected_time = 0;
872 - if (host->receiver != NULL) {
873 - time_t age = now_realtime_sec() - host->receiver->last_msg_t;
874 - if (age > 30) {
875 - host->receiver->shutdown = 1;
876 - shutdown(host->receiver->fd, SHUT_RDWR);
877 - host->receiver = NULL; // Thread holds reference to structure
878 - info(
879 - "STREAM %s [receive from [%s]:%s]: multiple connections for same host detected - "
880 - "existing connection is dead (%"PRId64" sec), accepting new connection.",
881 - rrdhost_hostname(host),
882 - w->client_ip,
883 - w->client_port,
884 - (int64_t)age);
885 - }
886 - else {
887 - netdata_mutex_unlock(&host->receiver_lock);
888 - rrdhost_unlock(host);
889 - rrd_unlock();
890 - log_stream_connection(w->client_ip, w->client_port, key, host->machine_guid, rrdhost_hostname(host),
891 - "REJECTED - ALREADY CONNECTED");
892 - info(
893 - "STREAM %s [receive from [%s]:%s]: multiple connections for same host detected - "
894 - "existing connection is active (within last %"PRId64" sec), rejecting new connection.",
895 - rrdhost_hostname(host),
896 - w->client_ip,
897 - w->client_port,
898 - (int64_t)age);
899 - // Have not set WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET - caller should clean up
900 - buffer_flush(w->response.data);
901 - buffer_strcat(w->response.data, "This GUID is already streaming to this server");
902 - freez(rpt);
903 - return 409;
976 + {
977 + time_t age;
978 + bool receiver_stale = false;
979 + bool receiver_working = false;
980 +
981 + rrd_rdlock();
982 + RRDHOST *host = rrdhost_find_by_guid(rpt->machine_guid);
983 + if (unlikely(host && rrdhost_flag_check(host, RRDHOST_FLAG_ARCHIVED))) /* Ignore archived hosts. */
984 + host = NULL;
985 +
986 + if (host) {
987 + netdata_mutex_lock(&host->receiver_lock);
988 + if (host->receiver) {
989 + age = now_realtime_sec() - host->receiver->last_msg_t;
990 +
991 + if (age < 30)
992 + receiver_working = true;
993 + else
994 + receiver_stale = true;
995 }
996 + netdata_mutex_unlock(&host->receiver_lock);
997 }
906 - host->receiver = rpt;
907 - netdata_mutex_unlock(&host->receiver_lock);
908 - rrdhost_unlock(host);
909 - }
910 - rrd_unlock();
911 -
912 - rpt->last_msg_t = now_realtime_sec();
913 -
914 - rpt->host = host;
915 - rpt->fd = w->ifd;
916 - rpt->key = strdupz(key);
917 - rpt->hostname = strdupz(hostname);
918 - rpt->registry_hostname = strdupz((registry_hostname && *registry_hostname)?registry_hostname:hostname);
919 - rpt->machine_guid = strdupz(machine_guid);
920 - rpt->os = strdupz(os);
921 - rpt->timezone = strdupz(timezone);
922 - rpt->abbrev_timezone = strdupz(abbrev_timezone);
923 - rpt->utc_offset = utc_offset;
924 - rpt->tags = (tags)?strdupz(tags):NULL;
925 - rpt->client_ip = strdupz(w->client_ip);
926 - rpt->client_port = strdupz(w->client_port);
927 - rpt->update_every = update_every;
928 - rpt->system_info = system_info;
929 - rpt->capabilities = stream_version;
930 -#ifdef ENABLE_HTTPS
931 - rpt->ssl.conn = w->ssl.conn;
932 - rpt->ssl.flags = w->ssl.flags;
933 -
934 - w->ssl.conn = NULL;
935 - w->ssl.flags = NETDATA_SSL_START;
936 -#endif
937 -
938 - if(w->user_agent && w->user_agent[0]) {
939 - char *t = strchr(w->user_agent, '/');
940 - if(t && *t) {
941 - *t = '\0';
942 - t++;
998 + rrd_unlock();
999 +
1000 + if (receiver_stale && stop_streaming_receiver(host, "STALE RECEIVER")) {
1001 + // we stopped the receiver
1002 + // we can proceed with this connection
1003 + receiver_stale = false;
1004 +
1005 + info("STREAM '%s' [receive from [%s]:%s]: "
1006 + "stopped previous stale receiver to accept this one."
1007 + , rpt->hostname
1008 + , rpt->client_ip, rpt->client_port
1009 + );
1010 }
1011
945 - rpt->program_name = strdupz(w->user_agent);
946 - if(t && *t) rpt->program_version = strdupz(t);
1012 + if (receiver_working || receiver_stale) {
1013 + // another receiver is already connected
1014 + // try again later
1015 +
1016 + char msg[200 + 1];
1017 + snprintfz(msg, 200,
1018 + "multiple connections for same host, "
1019 + "old connection was used %ld secs ago%s",
1020 + age, receiver_stale ? " (signaled old receiver to stop)" : " (new connection not accepted)");
1021 +
1022 + rrdpush_receive_log_status(
1023 + rpt,
1024 + msg,
1025 + "ALREADY CONNECTED");
1026 +
1027 + // Have not set WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET - caller should clean up
1028 + buffer_flush(w->response.data);
1029 + buffer_strcat(w->response.data, "This GUID is already streaming to this server");
1030 + receiver_state_free(rpt);
1031 + return HTTP_RESP_CONFLICT;
1032 + }
1033 }
1034
949 -
950 -
1035 debug(D_SYSTEM, "starting STREAM receive thread.");
1036
1037 char tag[FILENAME_MAX + 1];
1038 snprintfz(tag, FILENAME_MAX, "STREAM_RECEIVER[%s,[%s]:%s]", rpt->hostname, w->client_ip, w->client_port);
1039
956 - if(netdata_thread_create(&rpt->thread, tag, NETDATA_THREAD_OPTION_DEFAULT, rrdpush_receiver_thread, (void *)rpt))
957 - error("Failed to create new STREAM receive thread for client.");
1040 + if(netdata_thread_create(&rpt->thread, tag, NETDATA_THREAD_OPTION_DEFAULT, rrdpush_receiver_thread, (void *)rpt)) {
1041 + rrdpush_receive_log_status(
1042 + rpt,
1043 + "can't create receiver thread",
1044 + "INTERNAL SERVER ERROR");
1045
959 - // prevent the caller from closing the streaming socket
960 - if(web_server_mode == WEB_SERVER_MODE_STATIC_THREADED) {
961 - web_client_flag_set(w, WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET);
962 - }
963 - else {
964 - if(w->ifd == w->ofd)
965 - w->ifd = w->ofd = -1;
966 - else
967 - w->ifd = -1;
1046 + buffer_flush(w->response.data);
1047 + buffer_strcat(w->response.data, "Can't handle this request");
1048 + receiver_state_free(rpt);
1049 + return HTTP_RESP_INTERNAL_SERVER_ERROR;
1050 }
1051
970 - buffer_flush(w->response.data);
971 - return 200;
1052 + // prevent the caller from closing the streaming socket
1053 + return web_client_socket_is_now_used_for_streaming(w);
1054 }
1055
1056 static void stream_capabilities_to_string(BUFFER *wb, STREAM_CAPABILITIES caps) {
streaming/rrdpush.h
+40 -8
@@ -3,12 +3,14 @@
3 #ifndef NETDATA_RRDPUSH_H
4 #define NETDATA_RRDPUSH_H 1
5
6 -#include "database/rrd.h"
6 #include "libnetdata/libnetdata.h"
8 -#include "web/server/web_client.h"
7 #include "daemon/common.h"
8 +#include "web/server/web_client.h"
9 +#include "database/rrd.h"
10
11 #define CONNECTED_TO_SIZE 100
12 +#define CBUFFER_INITIAL_SIZE (16 * 1024)
13 +#define THREAD_BUFFER_INITIAL_SIZE (CBUFFER_INITIAL_SIZE * 4)
14
15 // ----------------------------------------------------------------------------
16 // obsolete versions - do not use anymore
@@ -22,6 +24,9 @@
24
25 typedef enum {
26 // do not use the first 3 bits
27 + // they used to be versions 1, 2 and 3
28 + // before we introduce capabilities
29 +
30 STREAM_CAP_V1 = (1 << 3), // v1 = the oldest protocol
31 STREAM_CAP_V2 = (1 << 4), // v2 = the second version of the protocol (with host labels)
32 STREAM_CAP_VN = (1 << 5), // version negotiation supported (for versions 3, 4, 5 of the protocol)
@@ -37,6 +42,7 @@ typedef enum {
42 STREAM_CAP_REPLICATION = (1 << 12), // replication supported
43 STREAM_CAP_BINARY = (1 << 13), // streaming supports binary data
44
45 + STREAM_CAP_INVALID = (1 << 30), // used as an invalid value for capabilities when this is set
46 // this must be signed int, so don't use the last bit
47 // needed for negotiating errors between parent and child
48 } STREAM_CAPABILITIES;
@@ -162,6 +168,11 @@ struct sender_state {
168 struct netdata_ssl ssl; // structure used to encrypt the connection
169 #endif
170
171 + struct {
172 + bool shutdown;
173 + const char *reason;
174 + } exit;
175 +
176 struct {
177 DICTIONARY *requests; // de-duplication of replication requests, per chart
178
@@ -216,13 +227,32 @@ struct receiver_state {
227 char *program_name; // Duplicated in pluginsd
228 char *program_version;
229 struct rrdhost_system_info *system_info;
219 - int update_every;
230 STREAM_CAPABILITIES capabilities;
231 time_t last_msg_t;
232 char read_buffer[PLUGINSD_LINE_MAX + 1];
233 int read_len;
224 - unsigned int shutdown:1; // Tell the thread to exit
225 - unsigned int exited; // Indicates that the thread has exited (NOT A BITFIELD!)
234 +
235 + struct {
236 + bool shutdown; // signal the streaming parser to exit
237 + const char *reason; // the reason of disconnection to log
238 + } exit;
239 +
240 + struct {
241 + RRD_MEMORY_MODE mode;
242 + int history;
243 + int update_every;
244 + int health_enabled; // CONFIG_BOOLEAN_YES, CONFIG_BOOLEAN_NO, CONFIG_BOOLEAN_AUTO
245 + time_t alarms_delay;
246 + int rrdpush_enabled;
247 + char *rrdpush_api_key; // DONT FREE - it is allocated in appconfig
248 + char *rrdpush_send_charts_matching; // DONT FREE - it is allocated in appconfig
249 + bool rrdpush_enable_replication;
250 + time_t rrdpush_seconds_to_replicate;
251 + time_t rrdpush_replication_step;
252 + char *rrdpush_destination; // DONT FREE - it is allocated in appconfig
253 + unsigned int rrdpush_compression;
254 + } config;
255 +
256 #ifdef ENABLE_HTTPS
257 struct netdata_ssl ssl;
258 #endif
@@ -260,8 +290,6 @@ extern unsigned int remote_clock_resync_iterations;
290 void rrdpush_destinations_init(RRDHOST *host);
291 void rrdpush_destinations_free(RRDHOST *host);
292
263 -void sender_init(RRDHOST *host);
264 -
293 BUFFER *sender_start(struct sender_state *s);
294 void sender_commit(struct sender_state *s, BUFFER *wb);
295 void sender_cancel(struct sender_state *s);
@@ -275,7 +303,7 @@ void rrdpush_send_host_labels(RRDHOST *host);
303 void rrdpush_claimed_id(RRDHOST *host);
304
305 int rrdpush_receiver_thread_spawn(struct web_client *w, char *url);
278 -void rrdpush_sender_thread_stop(RRDHOST *host);
306 +void rrdpush_sender_thread_stop(RRDHOST *host, const char *reason, bool wait);
307
308 void rrdpush_sender_send_this_host_variable_now(RRDHOST *host, const RRDVAR_ACQUIRED *rva);
309 void log_stream_connection(const char *client_ip, const char *client_port, const char *api_key, const char *machine_guid, const char *host, const char *msg);
@@ -295,11 +323,15 @@ struct compressor_state *create_compressor();
323 struct decompressor_state *create_decompressor();
324 #endif
325
326 +void rrdpush_receive_log_status(struct receiver_state *rpt, const char *msg, const char *status);
327 void log_receiver_capabilities(struct receiver_state *rpt);
328 void log_sender_capabilities(struct sender_state *s);
329 STREAM_CAPABILITIES convert_stream_version_to_capabilities(int32_t version);
330 int32_t stream_capabilities_to_vn(uint32_t caps);
331
332 +void receiver_state_free(struct receiver_state *rpt);
333 +bool stop_streaming_receiver(RRDHOST *host, const char *reason);
334 +
335 #include "replication.h"
336
337 #endif //NETDATA_RRDPUSH_H
streaming/sender.c
+115 -59
@@ -36,22 +36,31 @@ extern char *netdata_ssl_ca_file;
36
37 static __thread BUFFER *sender_thread_buffer = NULL;
38 static __thread bool sender_thread_buffer_used = false;
39 +static __thread bool sender_thread_buffer_recreate = false;
40
41 void sender_thread_buffer_free(void) {
41 - if(sender_thread_buffer) {
42 - buffer_free(sender_thread_buffer);
43 - sender_thread_buffer = NULL;
44 - }
42 + buffer_free(sender_thread_buffer);
43 + sender_thread_buffer = NULL;
44 }
45
46 // Collector thread starting a transmission
47 BUFFER *sender_start(struct sender_state *s __maybe_unused) {
49 - if(!sender_thread_buffer)
50 - sender_thread_buffer = buffer_create(1024);
51 -
52 - if(sender_thread_buffer_used)
48 + if(unlikely(sender_thread_buffer_used))
49 fatal("STREAMING: thread buffer is used multiple times concurrently.");
50
51 + if(unlikely(sender_thread_buffer_recreate)) {
52 + sender_thread_buffer_recreate = false;
53 + if(sender_thread_buffer && sender_thread_buffer->size > THREAD_BUFFER_INITIAL_SIZE) {
54 + buffer_free(sender_thread_buffer);
55 + sender_thread_buffer = NULL;
56 + }
57 + }
58 +
59 + if(!sender_thread_buffer) {
60 + sender_thread_buffer = buffer_create(THREAD_BUFFER_INITIAL_SIZE);
61 + sender_thread_buffer_recreate = false;
62 + }
63 +
64 sender_thread_buffer_used = true;
65 buffer_flush(sender_thread_buffer);
66 return sender_thread_buffer;
@@ -227,7 +236,7 @@ static void rrdpush_sender_thread_reset_all_charts(RRDHOST *host) {
236 rrdset_flag_clear(st, RRDSET_FLAG_UPSTREAM_EXPOSED | RRDSET_FLAG_SENDER_REPLICATION_IN_PROGRESS);
237 rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
238
230 - st->upstream_resync_time = 0;
239 + st->upstream_resync_time_s = 0;
240
241 RRDDIM *rd;
242 rrddim_foreach_read(rd, st)
@@ -740,7 +749,13 @@ static bool attempt_to_connect(struct sender_state *state)
749 state->sent_bytes_on_this_connection = 0;
750
751 // slow re-connection on repeating errors
743 - sleep_usec(USEC_PER_SEC * state->reconnect_delay); // seconds
752 + usec_t now_ut = now_monotonic_usec();
753 + usec_t end_ut = now_ut + USEC_PER_SEC * state->reconnect_delay;
754 + while(now_ut < end_ut) {
755 + netdata_thread_testcancel();
756 + sleep_usec(500 * USEC_PER_MS); // seconds
757 + now_ut = now_monotonic_usec();
758 + }
759
760 return false;
761 }
@@ -878,7 +893,7 @@ void execute_commands(struct sender_state *s) {
893 log_access("STREAM: %d from '%s' for host '%s': %s",
894 gettid(), s->connected_to, rrdhost_hostname(s->host), start);
895
881 - internal_error(true, "STREAM %s [send to %s] received command over connection: %s", rrdhost_hostname(s->host), s->connected_to, start);
896 + // internal_error(true, "STREAM %s [send to %s] received command over connection: %s", rrdhost_hostname(s->host), s->connected_to, start);
897
898 char *words[PLUGINSD_MAX_WORDS] = { NULL };
899 size_t num_words = pluginsd_split_words(start, words, PLUGINSD_MAX_WORDS, NULL, NULL, 0);
@@ -1021,59 +1036,83 @@ void rrdpush_signal_sender_to_wake_up(struct sender_state *s) {
1036 }
1037 }
1038
1024 -static void rrdpush_sender_thread_cleanup_callback(void *ptr) {
1025 - struct rrdpush_sender_thread_data *data = ptr;
1026 - worker_unregister();
1027 -
1028 - RRDHOST *host = data->host;
1039 +static bool rrdhost_set_sender(RRDHOST *host) {
1040 + if(unlikely(!host->sender)) return false;
1041
1042 + bool ret = false;
1043 netdata_mutex_lock(&host->sender->mutex);
1044 + if(!host->sender->tid) {
1045 + rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED | RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
1046 + rrdhost_flag_set(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN);
1047 + host->sender->tid = gettid();
1048 + ret = true;
1049 + }
1050 + netdata_mutex_unlock(&host->sender->mutex);
1051
1032 - info("STREAM %s [send]: sending thread cleans up...", rrdhost_hostname(host));
1052 + return ret;
1053 +}
1054
1034 - rrdpush_sender_thread_close_socket(host);
1035 - rrdpush_sender_pipe_close(host, host->sender->rrdpush_sender_pipe, false);
1055 +static void rrdhost_clear_sender___while_having_sender_mutex(RRDHOST *host) {
1056 + if(unlikely(!host->sender)) return;
1057 +
1058 + if(host->sender->tid == gettid()) {
1059 + host->sender->tid = 0;
1060 + host->sender->exit.shutdown = false;
1061 + host->sender->exit.reason = NULL;
1062 + rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN | RRDHOST_FLAG_RRDPUSH_SENDER_CONNECTED | RRDHOST_FLAG_RRDPUSH_SENDER_READY_4_METRICS);
1063 + }
1064 +}
1065
1037 - if(!rrdhost_flag_check(host, RRDHOST_FLAG_RRDPUSH_SENDER_JOIN)) {
1038 - info("STREAM %s [send]: sending thread detaches itself.", rrdhost_hostname(host));
1039 - netdata_thread_detach(netdata_thread_self());
1066 +static bool rrdhost_sender_should_exit(struct sender_state *s) {
1067 + // check for outstanding cancellation requests
1068 + netdata_thread_testcancel();
1069 +
1070 + if(unlikely(!service_running(SERVICE_STREAMING))) {
1071 + if(!s->exit.reason)
1072 + s->exit.reason = "NETDATA EXIT";
1073 + return true;
1074 }
1075
1042 - rrdhost_flag_clear(host, RRDHOST_FLAG_RRDPUSH_SENDER_SPAWN);
1076 + if(unlikely(!rrdhost_has_rrdpush_sender_enabled(s->host))) {
1077 + if(!s->exit.reason)
1078 + s->exit.reason = "NON STREAMABLE HOST";
1079 + return true;
1080 + }
1081
1044 - info("STREAM %s [send]: sending thread now exits.", rrdhost_hostname(host));
1082 + if(unlikely(s->exit.shutdown)) {
1083 + if(!s->exit.reason)
1084 + s->exit.reason = "SENDER SHUTDOWN REQUESTED";
1085 + return true;
1086 + }
1087
1046 - netdata_mutex_unlock(&host->sender->mutex);
1088 + if(unlikely(rrdhost_flag_check(s->host, RRDHOST_FLAG_ORPHAN))) {
1089 + if(!s->exit.reason)
1090 + s->exit.reason = "RECEIVER LEFT";
1091 + return true;
1092 + }
1093
1048 - freez(data->pipe_buffer);
1049 - freez(data);
1094 + return false;
1095 }
1096
1052 -void sender_init(RRDHOST *host)
1053 -{
1054 - if (host->sender)
1055 - return;
1097 +static void rrdpush_sender_thread_cleanup_callback(void *ptr) {
1098 + struct rrdpush_sender_thread_data *s = ptr;
1099 + worker_unregister();
1100
1057 - host->sender = callocz(1, sizeof(*host->sender));
1058 - host->sender->host = host;
1059 - host->sender->buffer = cbuffer_new(1024, 1024 * 1024);
1060 - host->sender->capabilities = STREAM_OUR_CAPABILITIES;
1101 + RRDHOST *host = s->host;
1102
1062 - host->sender->rrdpush_sender_pipe[PIPE_READ] = -1;
1063 - host->sender->rrdpush_sender_pipe[PIPE_WRITE] = -1;
1064 - host->sender->rrdpush_sender_socket = -1;
1103 + netdata_mutex_lock(&host->sender->mutex);
1104 + info("STREAM %s [send]: sending thread exits %s",
1105 + rrdhost_hostname(host),
1106 + host->sender->exit.reason ? host->sender->exit.reason : "");
1107
1066 -#ifdef ENABLE_COMPRESSION
1067 - if(default_compression_enabled) {
1068 - host->sender->flags |= SENDER_FLAG_COMPRESSION;
1069 - host->sender->compressor = create_compressor();
1070 - }
1071 - else
1072 - host->sender->flags &= ~SENDER_FLAG_COMPRESSION;
1073 -#endif
1108 + rrdpush_sender_thread_close_socket(host);
1109 + rrdpush_sender_pipe_close(host, host->sender->rrdpush_sender_pipe, false);
1110 +
1111 + rrdhost_clear_sender___while_having_sender_mutex(host);
1112 + netdata_mutex_unlock(&host->sender->mutex);
1113
1075 - netdata_mutex_init(&host->sender->mutex);
1076 - replication_init_sender(host->sender);
1114 + freez(s->pipe_buffer);
1115 + freez(s);
1116 }
1117
1118 void *rrdpush_sender_thread(void *ptr) {
@@ -1105,13 +1144,18 @@ void *rrdpush_sender_thread(void *ptr) {
1144 worker_register_job_custom_metric(WORKER_SENDER_JOB_REPLAY_DICT_SIZE, "replication dict entries", "entries", WORKER_METRIC_ABSOLUTE);
1145
1146 struct sender_state *s = ptr;
1108 - s->tid = gettid();
1147
1148 if(!rrdhost_has_rrdpush_sender_enabled(s->host) || !s->host->rrdpush_send_destination ||
1149 !*s->host->rrdpush_send_destination || !s->host->rrdpush_send_api_key ||
1150 !*s->host->rrdpush_send_api_key) {
1151 error("STREAM %s [send]: thread created (task id %d), but host has streaming disabled.",
1114 - rrdhost_hostname(s->host), s->tid);
1152 + rrdhost_hostname(s->host), gettid());
1153 + return NULL;
1154 + }
1155 +
1156 + if(!rrdhost_set_sender(s->host)) {
1157 + error("STREAM %s [send]: thread created (task id %d), but there is another sender running for this host.",
1158 + rrdhost_hostname(s->host), gettid());
1159 return NULL;
1160 }
1161
@@ -1127,7 +1171,7 @@ void *rrdpush_sender_thread(void *ptr) {
1171 }
1172 #endif
1173
1130 - info("STREAM %s [send]: thread created (task id %d)", rrdhost_hostname(s->host), s->tid);
1174 + info("STREAM %s [send]: thread created (task id %d)", rrdhost_hostname(s->host), gettid());
1175
1176 s->timeout = (int)appconfig_get_number(
1177 &stream_config, CONFIG_SECTION_STREAM, "timeout seconds", 600);
@@ -1168,14 +1212,9 @@ void *rrdpush_sender_thread(void *ptr) {
1212 thread_data->sender_state = s;
1213 thread_data->host = s->host;
1214
1171 - // reset our cleanup flags
1172 - rrdhost_flag_clear(s->host, RRDHOST_FLAG_RRDPUSH_SENDER_JOIN);
1173 -
1215 netdata_thread_cleanup_push(rrdpush_sender_thread_cleanup_callback, thread_data);
1216
1176 - for(; rrdhost_has_rrdpush_sender_enabled(s->host) && !netdata_exit ;) {
1177 - // check for outstanding cancellation requests
1178 - netdata_thread_testcancel();
1217 + while(!rrdhost_sender_should_exit(s)) {
1218
1219 // The connection attempt blocks (after which we use the socket in nonblocking)
1220 if(unlikely(s->rrdpush_sender_socket == -1)) {
@@ -1186,9 +1225,12 @@ void *rrdpush_sender_thread(void *ptr) {
1225 s->buffer->read = 0;
1226 s->buffer->write = 0;
1227
1189 - if(unlikely(!attempt_to_connect(s)))
1228 + if(!attempt_to_connect(s))
1229 continue;
1230
1231 + if(rrdhost_sender_should_exit(s))
1232 + break;
1233 +
1234 s->last_traffic_seen_t = now_monotonic_sec();
1235 rrdpush_claimed_id(s->host);
1236 rrdpush_send_host_labels(s->host);
@@ -1213,6 +1255,17 @@ void *rrdpush_sender_thread(void *ptr) {
1255 netdata_mutex_lock(&s->mutex);
1256 size_t outstanding = cbuffer_next_unsafe(s->host->sender->buffer, NULL);
1257 size_t available = cbuffer_available_size_unsafe(s->host->sender->buffer);
1258 + if(unlikely(!outstanding && s->host->sender->buffer->size > CBUFFER_INITIAL_SIZE)) {
1259 + static __thread time_t last_reset_time_t = 0;
1260 + time_t now_t = now_monotonic_sec();
1261 + if(now_t - last_reset_time_t > 600) {
1262 + last_reset_time_t = now_t;
1263 + size_t max = s->host->sender->buffer->max_size;
1264 + cbuffer_free(s->host->sender->buffer);
1265 + s->host->sender->buffer = cbuffer_new(CBUFFER_INITIAL_SIZE, max);
1266 + sender_thread_buffer_recreate = true;
1267 + }
1268 + }
1269 netdata_mutex_unlock(&s->mutex);
1270
1271 worker_set_metric(WORKER_SENDER_JOB_BUFFER_RATIO, (NETDATA_DOUBLE)(s->host->sender->buffer->max_size - available) * 100.0 / (NETDATA_DOUBLE)s->host->sender->buffer->max_size);
@@ -1248,12 +1301,14 @@ void *rrdpush_sender_thread(void *ptr) {
1301 .revents = 0,
1302 }
1303 };
1304 +
1305 int poll_rc = poll(fds, 2, 1000);
1306
1307 debug(D_STREAM, "STREAM: poll() finished collector=%d socket=%d (current chunk %zu bytes)...",
1308 fds[Collector].revents, fds[Socket].revents, outstanding);
1309
1256 - if(unlikely(netdata_exit)) break;
1310 + if(unlikely(rrdhost_sender_should_exit(s)))
1311 + break;
1312
1313 internal_error(fds[Collector].fd != s->rrdpush_sender_pipe[PIPE_READ],
1314 "STREAM %s [send to %s]: pipe changed after poll().", rrdhost_hostname(s->host), s->connected_to);
@@ -1263,6 +1318,7 @@ void *rrdpush_sender_thread(void *ptr) {
1318
1319 // Spurious wake-ups without error - loop again
1320 if (poll_rc == 0 || ((poll_rc == -1) && (errno == EAGAIN || errno == EINTR))) {
1321 + netdata_thread_testcancel();
1322 debug(D_STREAM, "Spurious wakeup");
1323 continue;
1324 }
web/api/badges/web_buffer_svg.c
+3 -3
@@ -969,7 +969,7 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
969 ret = HTTP_RESP_OK;
970 goto cleanup;
971 }
972 - st->last_accessed_time = now_realtime_sec();
972 + st->last_accessed_time_s = now_realtime_sec();
973
974 if(alarm) {
975 rca = rrdcalc_from_rrdset_get(st, alarm);
@@ -1110,14 +1110,14 @@ int web_client_api_request_v1_badge(RRDHOST *host, struct web_client *w, char *u
1110 ret = HTTP_RESP_INTERNAL_SERVER_ERROR;
1111
1112 // if the collected value is too old, don't calculate its value
1113 - if (rrdset_last_entry_t(st) >= (now_realtime_sec() - (st->update_every * st->gap_when_lost_iterations_above)))
1113 + if (rrdset_last_entry_s(st) >= (now_realtime_sec() - (st->update_every * gap_when_lost_iterations_above)))
1114 ret = rrdset2value_api_v1(st, w->response.data, &n,
1115 (dimensions) ? buffer_tostring(dimensions) : NULL,
1116 points, after, before, group, group_options, 0, options,
1117 NULL, &latest_timestamp,
1118 NULL, NULL, NULL,
1119 &value_is_null, NULL, 0, 0,
1120 - QUERY_SOURCE_API_BADGE);
1120 + QUERY_SOURCE_API_BADGE, STORAGE_PRIORITY_NORMAL);
1121
1122 // if the value cannot be calculated, show empty badge
1123 if (ret != HTTP_RESP_OK) {
web/api/exporters/shell/allmetrics_shell.c
+1 -1
@@ -127,7 +127,7 @@ void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_s
127 rrdset_family(st),
128 rrdset_context(st),
129 rrdset_units(st),
130 - (int64_t)rrdset_last_entry_t(st));
130 + (int64_t) rrdset_last_entry_s(st));
131
132 chart_counter++;
133 dimension_counter = 0;
web/api/formatters/charts2json.c
+4 -4
@@ -78,7 +78,7 @@ void charts2json(RRDHOST *host, BUFFER *wb, int skip_volatile, int show_archived
78 rrdset2json(st, wb, &dimensions, &memory, skip_volatile);
79
80 c++;
81 - st->last_accessed_time = now;
81 + st->last_accessed_time_s = now;
82 }
83 }
84 rrdset_foreach_done(st);
@@ -102,10 +102,10 @@ void charts2json(RRDHOST *host, BUFFER *wb, int skip_volatile, int show_archived
102 , dimensions
103 , alarms
104 , memory
105 - , rrd_hosts_available
105 + , rrdhost_hosts_available()
106 );
107
108 - if(unlikely(rrd_hosts_available > 1)) {
108 + if(unlikely(rrdhost_hosts_available() > 1)) {
109 rrd_rdlock();
110
111 size_t found = 0;
@@ -178,7 +178,7 @@ void chartcollectors2json(RRDHOST *host, BUFFER *wb) {
178 };
179 sprintf(name, "%s:%s", col.plugin, col.module);
180 dictionary_set(dict, name, &col, sizeof(struct collector));
181 - st->last_accessed_time = now;
181 + st->last_accessed_time_s = now;
182 }
183 }
184 rrdset_foreach_done(st);
web/api/formatters/json_wrapper.c
+3 -3
@@ -70,9 +70,9 @@ void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb, uint32_t format, RRDR_OPTIONS
70 , kq, kq, sq, qt->id, sq
71 , kq, kq, sq, qt->id, sq
72 , kq, kq, (long long)r->update_every
73 - , kq, kq, (long long)qt->db.minimum_latest_update_every
74 - , kq, kq, (long long)qt->db.first_time_t
75 - , kq, kq, (long long)qt->db.last_time_t
73 + , kq, kq, (long long)qt->db.minimum_latest_update_every_s
74 + , kq, kq, (long long)qt->db.first_time_s
75 + , kq, kq, (long long)qt->db.last_time_s
76 , kq, kq, (long long)r->before
77 , kq, kq, (long long)r->after
78 , kq, kq, sq, web_client_api_request_v1_data_group_to_string(group_method), sq
web/api/formatters/rrd2json.c
+3 -1
@@ -77,6 +77,7 @@ int rrdset2value_api_v1(
77 , time_t timeout
78 , size_t tier
79 , QUERY_SOURCE query_source
80 + , STORAGE_PRIORITY priority
81 ) {
82 int ret = HTTP_RESP_INTERNAL_SERVER_ERROR;
83
@@ -94,7 +95,8 @@ int rrdset2value_api_v1(
95 group_options,
96 timeout,
97 tier,
97 - query_source);
98 + query_source,
99 + priority);
100
101 if(!r) {
102 if(value_is_null) *value_is_null = 1;
web/api/formatters/rrd2json.h
+1
@@ -79,6 +79,7 @@ int rrdset2value_api_v1(
79 , time_t timeout
80 , size_t tier
81 , QUERY_SOURCE query_source
82 + , STORAGE_PRIORITY priority
83 );
84
85 #endif /* NETDATA_RRD2JSON_H */
web/api/formatters/rrdset2json.c
+3 -3
@@ -25,8 +25,8 @@ void chart_labels2json(RRDSET *st, BUFFER *wb, size_t indentation)
25 // generate JSON for the /api/v1/chart API call
26
27 void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memory_used, int skip_volatile) {
28 - time_t first_entry_t = rrdset_first_entry_t(st);
29 - time_t last_entry_t = rrdset_last_entry_t(st);
28 + time_t first_entry_t = rrdset_first_entry_s(st);
29 + time_t last_entry_t = rrdset_last_entry_s(st);
30
31 buffer_sprintf(
32 wb,
@@ -83,7 +83,7 @@ void rrdset2json(RRDSET *st, BUFFER *wb, size_t *dimensions_count, size_t *memor
83 "\t\t\t\"dimensions\": {\n",
84 st->update_every);
85
86 - unsigned long memory = sizeof(RRDSET) + st->memsize;
86 + unsigned long memory = sizeof(RRDSET);
87
88 size_t dimensions = 0;
89 RRDDIM *rd;
web/api/formatters/value/value.c
+2 -1
@@ -106,7 +106,7 @@ QUERY_VALUE rrdmetric2value(RRDHOST *host,
106 struct rrdcontext_acquired *rca, struct rrdinstance_acquired *ria, struct rrdmetric_acquired *rma,
107 time_t after, time_t before,
108 RRDR_OPTIONS options, RRDR_GROUPING group_method, const char *group_options,
109 - size_t tier, time_t timeout, QUERY_SOURCE query_source
109 + size_t tier, time_t timeout, QUERY_SOURCE query_source, STORAGE_PRIORITY priority
110 ) {
111 QUERY_TARGET_REQUEST qtr = {
112 .host = host,
@@ -122,6 +122,7 @@ QUERY_VALUE rrdmetric2value(RRDHOST *host,
122 .tier = tier,
123 .timeout = timeout,
124 .query_source = query_source,
125 + .priority = priority,
126 };
127
128 ONEWAYALLOC *owa = onewayalloc_create(16 * 1024);
web/api/formatters/value/value.h
+1 -1
@@ -23,7 +23,7 @@ QUERY_VALUE rrdmetric2value(RRDHOST *host,
23 struct rrdcontext_acquired *rca, struct rrdinstance_acquired *ria, struct rrdmetric_acquired *rma,
24 time_t after, time_t before,
25 RRDR_OPTIONS options, RRDR_GROUPING group_method, const char *group_options,
26 - size_t tier, time_t timeout, QUERY_SOURCE query_source
26 + size_t tier, time_t timeout, QUERY_SOURCE query_source, STORAGE_PRIORITY priority
27 );
28
29 NETDATA_DOUBLE rrdr2value(RRDR *r, long i, RRDR_OPTIONS options, int *all_values_are_null, NETDATA_DOUBLE *anomaly_rate);
web/api/queries/query.c
+254 -211
@@ -694,7 +694,7 @@ static inline void rrdr_done(RRDR *r, long rrdr_line) {
694 // tier management
695
696 static bool query_metric_is_valid_tier(QUERY_METRIC *qm, size_t tier) {
697 - if(!qm->tiers[tier].db_metric_handle || !qm->tiers[tier].db_first_time_t || !qm->tiers[tier].db_last_time_t || !qm->tiers[tier].db_update_every)
697 + if(!qm->tiers[tier].db_metric_handle || !qm->tiers[tier].db_first_time_s || !qm->tiers[tier].db_last_time_s || !qm->tiers[tier].db_update_every_s)
698 return false;
699
700 return true;
@@ -705,11 +705,11 @@ static size_t query_metric_first_working_tier(QUERY_METRIC *qm) {
705
706 // find the db time-range for this tier for all metrics
707 STORAGE_METRIC_HANDLE *db_metric_handle = qm->tiers[tier].db_metric_handle;
708 - time_t first_t = qm->tiers[tier].db_first_time_t;
709 - time_t last_t = qm->tiers[tier].db_last_time_t;
710 - time_t update_every = qm->tiers[tier].db_update_every;
708 + time_t first_time_s = qm->tiers[tier].db_first_time_s;
709 + time_t last_time_s = qm->tiers[tier].db_last_time_s;
710 + time_t update_every_s = qm->tiers[tier].db_update_every_s;
711
712 - if(!db_metric_handle || !first_t || !last_t || !update_every)
712 + if(!db_metric_handle || !first_time_s || !last_time_s || !update_every_s)
713 continue;
714
715 return tier;
@@ -718,17 +718,17 @@ static size_t query_metric_first_working_tier(QUERY_METRIC *qm) {
718 return 0;
719 }
720
721 -static long query_plan_points_coverage_weight(time_t db_first_t, time_t db_last_t, time_t db_update_every, time_t after_wanted, time_t before_wanted, size_t points_wanted, size_t tier __maybe_unused) {
722 - if(db_first_t == 0 || db_last_t == 0 || db_update_every == 0)
721 +static long query_plan_points_coverage_weight(time_t db_first_time_s, time_t db_last_time_s, time_t db_update_every_s, time_t after_wanted, time_t before_wanted, size_t points_wanted, size_t tier __maybe_unused) {
722 + if(db_first_time_s == 0 || db_last_time_s == 0 || db_update_every_s == 0)
723 return -LONG_MAX;
724
725 - time_t common_first_t = MAX(db_first_t, after_wanted);
726 - time_t common_last_t = MIN(db_last_t, before_wanted);
725 + time_t common_first_t = MAX(db_first_time_s, after_wanted);
726 + time_t common_last_t = MIN(db_last_time_s, before_wanted);
727
728 long time_coverage = (common_last_t - common_first_t) * 1000000 / (before_wanted - after_wanted);
729 size_t points_wanted_in_coverage = points_wanted * time_coverage / 1000000;
730
731 - long points_available = (common_last_t - common_first_t) / db_update_every;
731 + long points_available = (common_last_t - common_first_t) / db_update_every_s;
732 long points_delta = (long)(points_available - points_wanted_in_coverage);
733 long points_coverage = (points_delta < 0) ? (long)(points_available * time_coverage / points_wanted_in_coverage) : time_coverage;
734
@@ -748,27 +748,25 @@ static size_t query_metric_best_tier_for_timeframe(QUERY_METRIC *qm, time_t afte
748 if(unlikely(after_wanted == before_wanted || points_wanted <= 0))
749 return query_metric_first_working_tier(qm);
750
751 - long weight[storage_tiers];
752 -
751 for(size_t tier = 0; tier < storage_tiers ; tier++) {
752
753 // find the db time-range for this tier for all metrics
754 STORAGE_METRIC_HANDLE *db_metric_handle = qm->tiers[tier].db_metric_handle;
757 - time_t first_t = qm->tiers[tier].db_first_time_t;
758 - time_t last_t = qm->tiers[tier].db_last_time_t;
759 - time_t update_every = qm->tiers[tier].db_update_every;
755 + time_t first_time_s = qm->tiers[tier].db_first_time_s;
756 + time_t last_time_s = qm->tiers[tier].db_last_time_s;
757 + time_t update_every_s = qm->tiers[tier].db_update_every_s;
758
761 - if(!db_metric_handle || !first_t || !last_t || !update_every) {
762 - weight[tier] = -LONG_MAX;
759 + if(!db_metric_handle || !first_time_s || !last_time_s || !update_every_s) {
760 + qm->tiers[tier].weight = -LONG_MAX;
761 continue;
762 }
763
766 - weight[tier] = query_plan_points_coverage_weight(first_t, last_t, update_every, after_wanted, before_wanted, points_wanted, tier);
764 + qm->tiers[tier].weight = query_plan_points_coverage_weight(first_time_s, last_time_s, update_every_s, after_wanted, before_wanted, points_wanted, tier);
765 }
766
767 size_t best_tier = 0;
768 for(size_t tier = 1; tier < storage_tiers ; tier++) {
771 - if(weight[tier] >= weight[best_tier])
769 + if(qm->tiers[tier].weight >= qm->tiers[best_tier].weight)
770 best_tier = tier;
771 }
772
@@ -788,38 +786,38 @@ static size_t rrddim_find_best_tier_for_timeframe(QUERY_TARGET *qt, time_t after
786
787 for(size_t tier = 0; tier < storage_tiers ; tier++) {
788
791 - time_t common_first_t = 0;
792 - time_t common_last_t = 0;
793 - time_t common_update_every = 0;
789 + time_t common_first_time_s = 0;
790 + time_t common_last_time_s = 0;
791 + time_t common_update_every_s = 0;
792
793 // find the db time-range for this tier for all metrics
794 for(size_t i = 0, used = qt->query.used; i < used ; i++) {
795 QUERY_METRIC *qm = &qt->query.array[i];
796
799 - time_t first_t = qm->tiers[tier].db_first_time_t;
800 - time_t last_t = qm->tiers[tier].db_last_time_t;
801 - time_t update_every = qm->tiers[tier].db_update_every;
797 + time_t first_time_s = qm->tiers[tier].db_first_time_s;
798 + time_t last_time_s = qm->tiers[tier].db_last_time_s;
799 + time_t update_every_s = qm->tiers[tier].db_update_every_s;
800
803 - if(!first_t || !last_t || !update_every)
801 + if(!first_time_s || !last_time_s || !update_every_s)
802 continue;
803
806 - if(!common_first_t)
807 - common_first_t = first_t;
804 + if(!common_first_time_s)
805 + common_first_time_s = first_time_s;
806 else
809 - common_first_t = MIN(first_t, common_first_t);
807 + common_first_time_s = MIN(first_time_s, common_first_time_s);
808
811 - if(!common_last_t)
812 - common_last_t = last_t;
809 + if(!common_last_time_s)
810 + common_last_time_s = last_time_s;
811 else
814 - common_last_t = MAX(last_t, common_last_t);
812 + common_last_time_s = MAX(last_time_s, common_last_time_s);
813
816 - if(!common_update_every)
817 - common_update_every = update_every;
814 + if(!common_update_every_s)
815 + common_update_every_s = update_every_s;
816 else
819 - common_update_every = MIN(update_every, common_update_every);
817 + common_update_every_s = MIN(update_every_s, common_update_every_s);
818 }
819
822 - weight[tier] = query_plan_points_coverage_weight(common_first_t, common_last_t, common_update_every, after_wanted, before_wanted, points_wanted, tier);
820 + weight[tier] = query_plan_points_coverage_weight(common_first_time_s, common_last_time_s, common_update_every_s, after_wanted, before_wanted, points_wanted, tier);
821 }
822
823 size_t best_tier = 0;
@@ -842,19 +840,19 @@ static time_t rrdset_find_natural_update_every_for_timeframe(QUERY_TARGET *qt, t
840 best_tier = rrddim_find_best_tier_for_timeframe(qt, after_wanted, before_wanted, points_wanted);
841
842 // find the db minimum update every for this tier for all metrics
845 - time_t common_update_every = default_rrd_update_every;
843 + time_t common_update_every_s = default_rrd_update_every;
844 for(size_t i = 0, used = qt->query.used; i < used ; i++) {
845 QUERY_METRIC *qm = &qt->query.array[i];
846
849 - time_t update_every = qm->tiers[best_tier].db_update_every;
847 + time_t update_every_s = qm->tiers[best_tier].db_update_every_s;
848
849 if(!i)
852 - common_update_every = update_every;
850 + common_update_every_s = update_every_s;
851 else
854 - common_update_every = MIN(update_every, common_update_every);
852 + common_update_every_s = MIN(update_every_s, common_update_every_s);
853 }
854
857 - return common_update_every;
855 + return common_update_every_s;
856 }
857
858 // ----------------------------------------------------------------------------
@@ -888,17 +886,6 @@ QUERY_POINT QUERY_POINT_EMPTY = {
886 #define query_point_set_id(point, point_id) debug_dummy()
887 #endif
888
891 -typedef struct query_plan_entry {
892 - size_t tier;
893 - time_t after;
894 - time_t before;
895 -} QUERY_PLAN_ENTRY;
896 -
897 -typedef struct query_plan {
898 - size_t entries;
899 - QUERY_PLAN_ENTRY data[RRD_STORAGE_TIERS*2];
900 -} QUERY_PLAN;
901 -
889 typedef struct query_engine_ops {
890 // configuration
891 RRDR *r;
@@ -908,14 +895,13 @@ typedef struct query_engine_ops {
895 TIER_QUERY_FETCH tier_query_fetch;
896
897 // query planer
911 - QUERY_PLAN plan;
898 size_t current_plan;
899 time_t current_plan_expire_time;
900
901 // storage queries
902 size_t tier;
903 struct query_metric_tier *tier_ptr;
918 - struct storage_engine_query_handle handle;
904 + struct storage_engine_query_handle *handle;
905 STORAGE_POINT (*next_metric)(struct storage_engine_query_handle *handle);
906 int (*is_finished)(struct storage_engine_query_handle *handle);
907 void (*finalize)(struct storage_engine_query_handle *handle);
@@ -937,30 +923,78 @@ typedef struct query_engine_ops {
923 // ----------------------------------------------------------------------------
924 // query planer
925
940 -#define query_plan_should_switch_plan(ops, now) ((now) >= (ops).current_plan_expire_time)
926 +#define query_plan_should_switch_plan(ops, now) ((now) >= (ops)->current_plan_expire_time)
927 +
928 +static void query_planer_initialize_plans(QUERY_ENGINE_OPS *ops) {
929 + QUERY_METRIC *qm = ops->qm;
930 +
931 + for(size_t p = 0; p < qm->plan.used ; p++) {
932 + time_t after = qm->plan.array[p].after;
933 + time_t before = qm->plan.array[p].before;
934 +
935 + size_t tier = qm->plan.array[p].tier;
936 + struct query_metric_tier *tier_ptr = &qm->tiers[tier];
937 + tier_ptr->eng->api.query_ops.init(
938 + tier_ptr->db_metric_handle,
939 + &qm->plan.array[p].handle,
940 + after, before,
941 + ops->r->internal.qt->request.priority);
942 +
943 + qm->plan.array[p].next_metric = tier_ptr->eng->api.query_ops.next_metric;
944 + qm->plan.array[p].is_finished = tier_ptr->eng->api.query_ops.is_finished;
945 + qm->plan.array[p].finalize = tier_ptr->eng->api.query_ops.finalize;
946 + qm->plan.array[p].initialized = true;
947 + qm->plan.array[p].finalized = false;
948 + }
949 +}
950 +
951 +static void query_planer_finalize_plan(QUERY_ENGINE_OPS *ops, size_t plan_id) {
952 + QUERY_METRIC *qm = ops->qm;
953 +
954 + if(qm->plan.array[plan_id].initialized && !qm->plan.array[plan_id].finalized) {
955 + qm->plan.array[plan_id].finalize(&qm->plan.array[plan_id].handle);
956 + qm->plan.array[plan_id].initialized = false;
957 + qm->plan.array[plan_id].finalized = true;
958 + qm->plan.array[plan_id].next_metric = NULL;
959 + qm->plan.array[plan_id].is_finished = NULL;
960 + qm->plan.array[plan_id].finalize = NULL;
961 +
962 + if(ops->current_plan == plan_id) {
963 + ops->next_metric = NULL;
964 + ops->is_finished = NULL;
965 + ops->finalize = NULL;
966 + }
967 + }
968 +}
969 +
970 +static void query_planer_finalize_remaining_plans(QUERY_ENGINE_OPS *ops) {
971 + QUERY_METRIC *qm = ops->qm;
972
942 -static void query_planer_activate_plan(QUERY_ENGINE_OPS *ops, size_t plan_id, time_t overwrite_after) {
943 - if(unlikely(plan_id >= ops->plan.entries))
944 - plan_id = ops->plan.entries - 1;
973 + for(size_t p = 0; p < qm->plan.used ; p++)
974 + query_planer_finalize_plan(ops, p);
975 +}
976
946 - time_t after = ops->plan.data[plan_id].after;
947 - time_t before = ops->plan.data[plan_id].before;
977 +static void query_planer_activate_plan(QUERY_ENGINE_OPS *ops, size_t plan_id, time_t overwrite_after __maybe_unused) {
978 + QUERY_METRIC *qm = ops->qm;
979
949 - if(overwrite_after > after && overwrite_after < before)
950 - after = overwrite_after;
980 + internal_fatal(plan_id >= qm->plan.used, "QUERY: invalid plan_id given");
981 + internal_fatal(!qm->plan.array[plan_id].initialized, "QUERY: plan has not been initialized");
982 + internal_fatal(qm->plan.array[plan_id].finalized, "QUERY: plan has been finalized");
983
952 - ops->tier = ops->plan.data[plan_id].tier;
953 - ops->tier_ptr = &ops->qm->tiers[ops->tier];
954 - ops->tier_ptr->eng->api.query_ops.init(ops->tier_ptr->db_metric_handle, &ops->handle, after, before);
955 - ops->next_metric = ops->tier_ptr->eng->api.query_ops.next_metric;
956 - ops->is_finished = ops->tier_ptr->eng->api.query_ops.is_finished;
957 - ops->finalize = ops->tier_ptr->eng->api.query_ops.finalize;
984 + ops->tier = qm->plan.array[plan_id].tier;
985 + ops->tier_ptr = &qm->tiers[ops->tier];
986 + ops->handle = &qm->plan.array[plan_id].handle;
987 + ops->next_metric = qm->plan.array[plan_id].next_metric;
988 + ops->is_finished = qm->plan.array[plan_id].is_finished;
989 + ops->finalize = qm->plan.array[plan_id].finalize;
990 ops->current_plan = plan_id;
959 - ops->current_plan_expire_time = ops->plan.data[plan_id].before;
991 + ops->current_plan_expire_time = qm->plan.array[plan_id].before;
992 }
993
994 static void query_planer_next_plan(QUERY_ENGINE_OPS *ops, time_t now, time_t last_point_end_time) {
963 - internal_error(now < ops->current_plan_expire_time && now < ops->plan.data[ops->current_plan].before,
995 + QUERY_METRIC *qm = ops->qm;
996 +
997 + internal_fatal(now < ops->current_plan_expire_time && now < qm->plan.array[ops->current_plan].before,
998 "QUERY: switching query plan too early!");
999
1000 size_t old_plan = ops->current_plan;
@@ -969,7 +1003,7 @@ static void query_planer_next_plan(QUERY_ENGINE_OPS *ops, time_t now, time_t las
1003 do {
1004 ops->current_plan++;
1005
972 - if (ops->current_plan >= ops->plan.entries) {
1006 + if (ops->current_plan >= qm->plan.used) {
1007 ops->current_plan = old_plan;
1008 ops->current_plan_expire_time = ops->r->internal.qt->window.before;
1009 // let the query run with current plan
@@ -977,23 +1011,16 @@ static void query_planer_next_plan(QUERY_ENGINE_OPS *ops, time_t now, time_t las
1011 return;
1012 }
1013
980 - next_plan_before_time = ops->plan.data[ops->current_plan].before;
1014 + next_plan_before_time = qm->plan.array[ops->current_plan].before;
1015 } while(now >= next_plan_before_time || last_point_end_time >= next_plan_before_time);
1016
983 - if(!query_metric_is_valid_tier(ops->qm, ops->plan.data[ops->current_plan].tier)) {
1017 + if(!query_metric_is_valid_tier(qm, qm->plan.array[ops->current_plan].tier)) {
1018 ops->current_plan = old_plan;
1019 ops->current_plan_expire_time = ops->r->internal.qt->window.before;
1020 return;
1021 }
1022
989 - if(ops->finalize) {
990 - ops->finalize(&ops->handle);
991 - ops->finalize = NULL;
992 - ops->is_finished = NULL;
993 - }
994 -
995 - // internal_error(true, "QUERY: switched plan to %zu (all is %zu), previous expiration was %ld, this starts at %ld, now is %ld, last_point_end_time %ld", ops->current_plan, ops->plan.entries, ops->plan.data[ops->current_plan-1].before, ops->plan.data[ops->current_plan].after, now, last_point_end_time);
996 -
1023 + query_planer_finalize_plan(ops, old_plan);
1024 query_planer_activate_plan(ops, ops->current_plan, MIN(now, last_point_end_time));
1025 }
1026
@@ -1004,59 +1031,62 @@ static int compare_query_plan_entries_on_start_time(const void *a, const void *b
1031 }
1032
1033 static bool query_plan(QUERY_ENGINE_OPS *ops, time_t after_wanted, time_t before_wanted, size_t points_wanted) {
1007 - //BUFFER *wb = buffer_create(1000);
1008 - //buffer_sprintf(wb, "QUERY PLAN for chart '%s' dimension '%s', from %ld to %ld:", rd->rrdset->name, rd->name, after_wanted, before_wanted);
1034 + QUERY_METRIC *qm = ops->qm;
1035
1036 // put our selected tier as the first plan
1037 size_t selected_tier;
1038
1039 if(ops->r->internal.query_options & RRDR_OPTION_SELECTED_TIER
1040 && ops->r->internal.qt->window.tier < storage_tiers
1015 - && query_metric_is_valid_tier(ops->qm, ops->r->internal.qt->window.tier)) {
1041 + && query_metric_is_valid_tier(qm, ops->r->internal.qt->window.tier)) {
1042 selected_tier = ops->r->internal.qt->window.tier;
1043 }
1044 else {
1019 - selected_tier = query_metric_best_tier_for_timeframe(ops->qm, after_wanted, before_wanted, points_wanted);
1045 + selected_tier = query_metric_best_tier_for_timeframe(qm, after_wanted, before_wanted, points_wanted);
1046
1047 if(ops->r->internal.query_options & RRDR_OPTION_SELECTED_TIER)
1048 ops->r->internal.query_options &= ~RRDR_OPTION_SELECTED_TIER;
1049 +
1050 + if(!query_metric_is_valid_tier(qm, selected_tier))
1051 + return false;
1052 }
1053
1025 - ops->plan.entries = 1;
1026 - ops->plan.data[0].tier = selected_tier;
1027 - ops->plan.data[0].after = ops->qm->tiers[selected_tier].db_first_time_t;
1028 - ops->plan.data[0].before = ops->qm->tiers[selected_tier].db_last_time_t;
1054 + qm->plan.used = 1;
1055 + qm->plan.array[0].tier = selected_tier;
1056 + qm->plan.array[0].after = qm->tiers[selected_tier].db_first_time_s;
1057 + qm->plan.array[0].before = qm->tiers[selected_tier].db_last_time_s;
1058
1059 if(!(ops->r->internal.query_options & RRDR_OPTION_SELECTED_TIER)) {
1060 // the selected tier
1032 - time_t selected_tier_first_time_t = ops->plan.data[0].after;
1033 - time_t selected_tier_last_time_t = ops->plan.data[0].before;
1034 -
1035 - //buffer_sprintf(wb, ": SELECTED tier %zu, from %ld to %ld", selected_tier, ops->plan.data[0].after, ops->plan.data[0].before);
1061 + time_t selected_tier_first_time_s = qm->plan.array[0].after;
1062 + time_t selected_tier_last_time_s = qm->plan.array[0].before;
1063
1064 // check if our selected tier can start the query
1038 - if (selected_tier_first_time_t > after_wanted) {
1065 + if (selected_tier_first_time_s > after_wanted) {
1066 // we need some help from other tiers
1067 for (size_t tr = (int)selected_tier + 1; tr < storage_tiers; tr++) {
1041 - if(!query_metric_is_valid_tier(ops->qm, tr))
1068 + if(!query_metric_is_valid_tier(qm, tr))
1069 continue;
1070
1071 // find the first time of this tier
1045 - time_t first_time_t = ops->qm->tiers[tr].db_first_time_t;
1046 -
1047 - //buffer_sprintf(wb, ": EVAL AFTER tier %d, %ld", tier, first_time_t);
1072 + time_t first_time_s = qm->tiers[tr].db_first_time_s;
1073
1074 // can it help?
1050 - if (first_time_t < selected_tier_first_time_t) {
1075 + if (first_time_s < selected_tier_first_time_s) {
1076 // it can help us add detail at the beginning of the query
1077 QUERY_PLAN_ENTRY t = {
1078 .tier = tr,
1054 - .after = (first_time_t < after_wanted) ? after_wanted : first_time_t,
1055 - .before = selected_tier_first_time_t};
1056 - ops->plan.data[ops->plan.entries++] = t;
1079 + .after = (first_time_s < after_wanted) ? after_wanted : first_time_s,
1080 + .before = selected_tier_first_time_s,
1081 + .initialized = false,
1082 + .finalized = false,
1083 + };
1084 + qm->plan.array[qm->plan.used++] = t;
1085 +
1086 + internal_fatal(!t.after || !t.before, "QUERY: invalid plan selected");
1087
1088 // prepare for the tier
1059 - selected_tier_first_time_t = t.after;
1089 + selected_tier_first_time_s = t.after;
1090
1091 if (t.after <= after_wanted)
1092 break;
@@ -1065,28 +1095,33 @@ static bool query_plan(QUERY_ENGINE_OPS *ops, time_t after_wanted, time_t before
1095 }
1096
1097 // check if our selected tier can finish the query
1068 - if (selected_tier_last_time_t < before_wanted) {
1098 + if (selected_tier_last_time_s < before_wanted) {
1099 // we need some help from other tiers
1100 for (int tr = (int)selected_tier - 1; tr >= 0; tr--) {
1071 - if(!query_metric_is_valid_tier(ops->qm, tr))
1101 + if(!query_metric_is_valid_tier(qm, tr))
1102 continue;
1103
1104 // find the last time of this tier
1075 - time_t last_time_t = ops->qm->tiers[tr].db_last_time_t;
1105 + time_t last_time_s = qm->tiers[tr].db_last_time_s;
1106
1077 - //buffer_sprintf(wb, ": EVAL BEFORE tier %d, %ld", tier, last_time_t);
1107 + //buffer_sprintf(wb, ": EVAL BEFORE tier %d, %ld", tier, last_time_s);
1108
1109 // can it help?
1080 - if (last_time_t > selected_tier_last_time_t) {
1110 + if (last_time_s > selected_tier_last_time_s) {
1111 // it can help us add detail at the end of the query
1112 QUERY_PLAN_ENTRY t = {
1113 .tier = tr,
1084 - .after = selected_tier_last_time_t,
1085 - .before = (last_time_t > before_wanted) ? before_wanted : last_time_t};
1086 - ops->plan.data[ops->plan.entries++] = t;
1114 + .after = selected_tier_last_time_s,
1115 + .before = (last_time_s > before_wanted) ? before_wanted : last_time_s,
1116 + .initialized = false,
1117 + .finalized = false,
1118 + };
1119 + qm->plan.array[qm->plan.used++] = t;
1120
1121 // prepare for the tier
1089 - selected_tier_last_time_t = t.before;
1122 + selected_tier_last_time_s = t.before;
1123 +
1124 + internal_fatal(!t.after || !t.before, "QUERY: invalid plan selected");
1125
1126 if (t.before >= before_wanted)
1127 break;
@@ -1096,26 +1131,20 @@ static bool query_plan(QUERY_ENGINE_OPS *ops, time_t after_wanted, time_t before
1131 }
1132
1133 // sort the query plan
1099 - if(ops->plan.entries > 1)
1100 - qsort(&ops->plan.data, ops->plan.entries, sizeof(QUERY_PLAN_ENTRY), compare_query_plan_entries_on_start_time);
1134 + if(qm->plan.used > 1)
1135 + qsort(&qm->plan.array, qm->plan.used, sizeof(QUERY_PLAN_ENTRY), compare_query_plan_entries_on_start_time);
1136
1137 // make sure it has the whole timeframe we need
1103 - if(ops->plan.data[0].after < after_wanted)
1104 - ops->plan.data[0].after = after_wanted;
1105 -
1106 - if(ops->plan.data[ops->plan.entries - 1].before > before_wanted)
1107 - ops->plan.data[ops->plan.entries - 1].before = before_wanted;
1138 + if(qm->plan.array[0].after < after_wanted)
1139 + qm->plan.array[0].after = after_wanted;
1140
1109 - //buffer_sprintf(wb, ": FINAL STEPS %zu", ops->plan.entries);
1141 + if(qm->plan.array[qm->plan.used - 1].before > before_wanted)
1142 + qm->plan.array[qm->plan.used - 1].before = before_wanted;
1143
1111 - //for(size_t i = 0; i < ops->plan.entries ;i++)
1112 - // buffer_sprintf(wb, ": STEP %zu = use tier %zu from %ld to %ld", i+1, ops->plan.data[i].tier, ops->plan.data[i].after, ops->plan.data[i].before);
1113 -
1114 - //internal_error(true, "%s", buffer_tostring(wb));
1115 -
1116 - if(!query_metric_is_valid_tier(ops->qm, ops->plan.data[0].tier))
1144 + if(!query_metric_is_valid_tier(qm, qm->plan.array[0].tier))
1145 return false;
1146
1147 + query_planer_initialize_plans(ops);
1148 query_planer_activate_plan(ops, 0, 0);
1149
1150 return true;
@@ -1146,24 +1175,45 @@ static bool query_plan(QUERY_ENGINE_OPS *ops, time_t after_wanted, time_t before
1175 #define query_add_point_to_group(r, point, ops) do { \
1176 if(likely(netdata_double_isnumber((point).value))) { \
1177 if(likely(fpclassify((point).value) != FP_ZERO)) \
1149 - (ops).group_points_non_zero++; \
1178 + (ops)->group_points_non_zero++; \
1179 \
1180 if(unlikely((point).flags & SN_FLAG_RESET)) \
1152 - (ops).group_value_flags |= RRDR_VALUE_RESET; \
1181 + (ops)->group_value_flags |= RRDR_VALUE_RESET; \
1182 \
1154 - (ops).grouping_add(r, (point).value); \
1183 + (ops)->grouping_add(r, (point).value); \
1184 } \
1185 \
1157 - (ops).group_points_added++; \
1158 - (ops).group_anomaly_rate += (point).anomaly; \
1186 + (ops)->group_points_added++; \
1187 + (ops)->group_anomaly_rate += (point).anomaly; \
1188 } while(0)
1189
1161 -static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1190 +static QUERY_ENGINE_OPS *rrd2rrdr_query_prep(RRDR *r, size_t dim_id_in_rrdr) {
1191 + QUERY_TARGET *qt = r->internal.qt;
1192 +
1193 + QUERY_ENGINE_OPS *ops = onewayalloc_mallocz(r->internal.owa, sizeof(QUERY_ENGINE_OPS));
1194 + *ops = (QUERY_ENGINE_OPS) {
1195 + .r = r,
1196 + .qm = &qt->query.array[dim_id_in_rrdr],
1197 + .grouping_add = r->internal.grouping_add,
1198 + .grouping_flush = r->internal.grouping_flush,
1199 + .tier_query_fetch = r->internal.tier_query_fetch,
1200 + .view_update_every = r->update_every,
1201 + .query_granularity = (time_t)(r->update_every / r->group),
1202 + .group_value_flags = RRDR_VALUE_NOTHING,
1203 + };
1204 +
1205 + if(!query_plan(ops, qt->window.after, qt->window.before, qt->window.points))
1206 + return NULL;
1207 +
1208 + return ops;
1209 +}
1210 +
1211 +static void rrd2rrdr_query_execute(RRDR *r, size_t dim_id_in_rrdr, QUERY_ENGINE_OPS *ops) {
1212 QUERY_TARGET *qt = r->internal.qt;
1163 - QUERY_METRIC *qm = &qt->query.array[dim_id_in_rrdr];
1213 + QUERY_METRIC *qm = &qt->query.array[dim_id_in_rrdr]; (void)qm;
1214 size_t points_wanted = qt->window.points;
1215 time_t after_wanted = qt->window.after;
1166 - time_t before_wanted = qt->window.before;
1216 + time_t before_wanted = qt->window.before; (void)before_wanted;
1217
1218 // bool debug_this = false;
1219 // if(strcmp("user", string2str(rd->id)) == 0 && strcmp("system.cpu", string2str(rd->rrdset->id)) == 0)
@@ -1174,39 +1224,25 @@ static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1224
1225 size_t points_added = 0;
1226
1177 - QUERY_ENGINE_OPS ops = {
1178 - .r = r,
1179 - .qm = qm,
1180 - .grouping_add = r->internal.grouping_add,
1181 - .grouping_flush = r->internal.grouping_flush,
1182 - .tier_query_fetch = r->internal.tier_query_fetch,
1183 - .view_update_every = r->update_every,
1184 - .query_granularity = (time_t)(r->update_every / r->group),
1185 - .group_value_flags = RRDR_VALUE_NOTHING
1186 - };
1187 -
1227 long rrdr_line = -1;
1228 bool use_anomaly_bit_as_value = (r->internal.query_options & RRDR_OPTION_ANOMALY_BIT) ? true : false;
1229
1191 - if(!query_plan(&ops, after_wanted, before_wanted, points_wanted))
1192 - return;
1193 -
1230 NETDATA_DOUBLE min = r->min, max = r->max;
1231
1232 QUERY_POINT last2_point = QUERY_POINT_EMPTY;
1233 QUERY_POINT last1_point = QUERY_POINT_EMPTY;
1234 QUERY_POINT new_point = QUERY_POINT_EMPTY;
1235
1200 - time_t now_start_time = after_wanted - ops.query_granularity;
1201 - time_t now_end_time = after_wanted + ops.view_update_every - ops.query_granularity;
1236 + time_t now_start_time = after_wanted - ops->query_granularity;
1237 + time_t now_end_time = after_wanted + ops->view_update_every - ops->query_granularity;
1238
1239 size_t db_points_read_since_plan_switch = 0; (void)db_points_read_since_plan_switch;
1240
1241 // The main loop, based on the query granularity we need
1206 - for( ; points_added < points_wanted ; now_start_time = now_end_time, now_end_time += ops.view_update_every) {
1242 + for( ; points_added < points_wanted ; now_start_time = now_end_time, now_end_time += ops->view_update_every) {
1243
1244 if(unlikely(query_plan_should_switch_plan(ops, now_end_time))) {
1209 - query_planer_next_plan(&ops, now_end_time, new_point.end_time);
1245 + query_planer_next_plan(ops, now_end_time, new_point.end_time);
1246 db_points_read_since_plan_switch = 0;
1247 }
1248
@@ -1219,7 +1255,7 @@ static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1255 last1_point = new_point;
1256 }
1257
1222 - if(unlikely(ops.is_finished(&ops.handle))) {
1258 + if(unlikely(ops->is_finished(ops->handle))) {
1259 if(count_same_end_time != 0) {
1260 last2_point = last1_point;
1261 last1_point = new_point;
@@ -1236,15 +1272,15 @@ static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1272 // fetch the new point
1273 {
1274 db_points_read_since_plan_switch++;
1239 - STORAGE_POINT sp = ops.next_metric(&ops.handle);
1275 + STORAGE_POINT sp = ops->next_metric(ops->handle);
1276
1241 - ops.db_points_read_per_tier[ops.tier]++;
1242 - ops.db_total_points_read++;
1277 + ops->db_points_read_per_tier[ops->tier]++;
1278 + ops->db_total_points_read++;
1279
1244 - new_point.start_time = sp.start_time;
1245 - new_point.end_time = sp.end_time;
1280 + new_point.start_time = sp.start_time_s;
1281 + new_point.end_time = sp.end_time_s;
1282 new_point.anomaly = sp.count ? (NETDATA_DOUBLE)sp.anomaly_count * 100.0 / (NETDATA_DOUBLE)sp.count : 0.0;
1247 - query_point_set_id(new_point, ops.db_total_points_read);
1283 + query_point_set_id(new_point, ops->db_total_points_read);
1284
1285 // if(debug_this)
1286 // info("QUERY: got point %zu, from time %ld to %ld // now from %ld to %ld // query from %ld to %ld",
@@ -1257,7 +1293,7 @@ static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1293 new_point.value = new_point.anomaly;
1294
1295 else {
1260 - switch (ops.tier_query_fetch) {
1296 + switch (ops->tier_query_fetch) {
1297 default:
1298 case TIER_QUERY_FETCH_AVERAGE:
1299 new_point.value = sp.sum / sp.count;
@@ -1288,7 +1324,7 @@ static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1324 internal_error(true, "QUERY: '%s', dimension '%s' next_metric() returned point %zu start time %ld, end time %ld, that are both equal",
1325 qt->id, string2str(qm->dimension.id), new_point.id, new_point.start_time, new_point.end_time);
1326
1291 - new_point.start_time = new_point.end_time - ops.tier_ptr->db_update_every;
1327 + new_point.start_time = new_point.end_time - ops->tier_ptr->db_update_every_s;
1328 }
1329
1330 // check if the db is advancing the query
@@ -1352,7 +1388,7 @@ static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1388
1389 size_t iterations = 0;
1390 for ( ; now_end_time <= new_point.end_time && points_added < points_wanted ;
1355 - now_end_time += ops.view_update_every, iterations++) {
1391 + now_end_time += ops->view_update_every, iterations++) {
1392
1393 // now_start_time is wrong in this loop
1394 // but, we don't need it
@@ -1411,20 +1447,20 @@ static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1447 RRDR_VALUE_FLAGS *rrdr_value_options_ptr = &r->o[rrdr_o_v_index];
1448
1449 // update the dimension options
1414 - if(likely(ops.group_points_non_zero))
1450 + if(likely(ops->group_points_non_zero))
1451 r->od[dim_id_in_rrdr] |= RRDR_DIMENSION_NONZERO;
1452
1453 // store the specific point options
1418 - *rrdr_value_options_ptr = ops.group_value_flags;
1454 + *rrdr_value_options_ptr = ops->group_value_flags;
1455
1456 // store the group value
1421 - NETDATA_DOUBLE group_value = ops.grouping_flush(r, rrdr_value_options_ptr);
1457 + NETDATA_DOUBLE group_value = ops->grouping_flush(r, rrdr_value_options_ptr);
1458 r->v[rrdr_o_v_index] = group_value;
1459
1460 // we only store uint8_t anomaly rates,
1461 // so let's get double precision by storing
1462 // anomaly rates in the range 0 - 200
1427 - r->ar[rrdr_o_v_index] = ops.group_anomaly_rate / (NETDATA_DOUBLE)ops.group_points_added;
1463 + r->ar[rrdr_o_v_index] = ops->group_anomaly_rate / (NETDATA_DOUBLE)ops->group_points_added;
1464
1465 if(likely(points_added || dim_id_in_rrdr)) {
1466 // find the min/max across all dimensions
@@ -1440,72 +1476,71 @@ static inline void rrd2rrdr_do_dimension(RRDR *r, size_t dim_id_in_rrdr) {
1476 }
1477
1478 points_added++;
1443 - ops.group_points_added = 0;
1444 - ops.group_value_flags = RRDR_VALUE_NOTHING;
1445 - ops.group_points_non_zero = 0;
1446 - ops.group_anomaly_rate = 0;
1479 + ops->group_points_added = 0;
1480 + ops->group_value_flags = RRDR_VALUE_NOTHING;
1481 + ops->group_points_non_zero = 0;
1482 + ops->group_anomaly_rate = 0;
1483 }
1484 // the loop above increased "now" by query_granularity,
1485 // but the main loop will increase it too,
1486 // so, let's undo the last iteration of this loop
1487 if(iterations)
1452 - now_end_time -= ops.view_update_every;
1488 + now_end_time -= ops->view_update_every;
1489 }
1454 - ops.finalize(&ops.handle);
1490 + query_planer_finalize_remaining_plans(ops);
1491
1492 r->internal.result_points_generated += points_added;
1457 - r->internal.db_points_read += ops.db_total_points_read;
1493 + r->internal.db_points_read += ops->db_total_points_read;
1494 for(size_t tr = 0; tr < storage_tiers ; tr++)
1459 - r->internal.tier_points_read[tr] += ops.db_points_read_per_tier[tr];
1495 + r->internal.tier_points_read[tr] += ops->db_points_read_per_tier[tr];
1496
1497 r->min = min;
1498 r->max = max;
1499 r->before = max_date;
1464 - r->after = min_date - ops.view_update_every + ops.query_granularity;
1500 + r->after = min_date - ops->view_update_every + ops->query_granularity;
1501 rrdr_done(r, rrdr_line);
1502
1503 internal_error(points_added != points_wanted,
1504 "QUERY: '%s', dimension '%s', requested %zu points, but RRDR added %zu (%zu db points read).",
1505 qt->id, string2str(qm->dimension.id),
1470 - (size_t)points_wanted, (size_t)points_added, ops.db_total_points_read);
1506 + (size_t)points_wanted, (size_t)points_added, ops->db_total_points_read);
1507 }
1508
1509 // ----------------------------------------------------------------------------
1510 // fill the gap of a tier
1511
1512 void store_metric_at_tier(RRDDIM *rd, size_t tier, struct rrddim_tier *t, STORAGE_POINT sp, usec_t now_ut);
1477 -void store_metric_collection_completed(void);
1513
1479 -void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, size_t tier, time_t now) {
1514 +void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, size_t tier, time_t now_s) {
1515 if(unlikely(tier >= storage_tiers)) return;
1516 if(storage_tiers_backfill[tier] == RRD_BACKFILL_NONE) return;
1517
1518 struct rrddim_tier *t = rd->tiers[tier];
1519 if(unlikely(!t)) return;
1520
1486 - time_t latest_time_t = t->query_ops->latest_time(t->db_metric_handle);
1521 + time_t latest_time_s = t->query_ops->latest_time_s(t->db_metric_handle);
1522 time_t granularity = (time_t)t->tier_grouping * (time_t)rd->update_every;
1488 - time_t time_diff = now - latest_time_t;
1523 + time_t time_diff = now_s - latest_time_s;
1524
1525 // if the user wants only NEW backfilling, and we don't have any data
1491 - if(storage_tiers_backfill[tier] == RRD_BACKFILL_NEW && latest_time_t <= 0) return;
1526 + if(storage_tiers_backfill[tier] == RRD_BACKFILL_NEW && latest_time_s <= 0) return;
1527
1528 // there is really nothing we can do
1494 - if(now <= latest_time_t || time_diff < granularity) return;
1529 + if(now_s <= latest_time_s || time_diff < granularity) return;
1530
1531 struct storage_engine_query_handle handle;
1532
1533 // for each lower tier
1534 for(int read_tier = (int)tier - 1; read_tier >= 0 ; read_tier--){
1500 - time_t smaller_tier_first_time = rd->tiers[read_tier]->query_ops->oldest_time(rd->tiers[read_tier]->db_metric_handle);
1501 - time_t smaller_tier_last_time = rd->tiers[read_tier]->query_ops->latest_time(rd->tiers[read_tier]->db_metric_handle);
1502 - if(smaller_tier_last_time <= latest_time_t) continue; // it is as bad as we are
1535 + time_t smaller_tier_first_time = rd->tiers[read_tier]->query_ops->oldest_time_s(rd->tiers[read_tier]->db_metric_handle);
1536 + time_t smaller_tier_last_time = rd->tiers[read_tier]->query_ops->latest_time_s(rd->tiers[read_tier]->db_metric_handle);
1537 + if(smaller_tier_last_time <= latest_time_s) continue; // it is as bad as we are
1538
1504 - long after_wanted = (latest_time_t < smaller_tier_first_time) ? smaller_tier_first_time : latest_time_t;
1539 + long after_wanted = (latest_time_s < smaller_tier_first_time) ? smaller_tier_first_time : latest_time_s;
1540 long before_wanted = smaller_tier_last_time;
1541
1542 struct rrddim_tier *tmp = rd->tiers[read_tier];
1508 - tmp->query_ops->init(tmp->db_metric_handle, &handle, after_wanted, before_wanted);
1543 + tmp->query_ops->init(tmp->db_metric_handle, &handle, after_wanted, before_wanted, STORAGE_PRIORITY_HIGH);
1544
1545 size_t points_read = 0;
1546
@@ -1514,9 +1549,9 @@ void rrdr_fill_tier_gap_from_smaller_tiers(RRDDIM *rd, size_t tier, time_t now)
1549 STORAGE_POINT sp = tmp->query_ops->next_metric(&handle);
1550 points_read++;
1551
1517 - if(sp.end_time > latest_time_t) {
1518 - latest_time_t = sp.end_time;
1519 - store_metric_at_tier(rd, tier, t, sp, sp.end_time * USEC_PER_SEC);
1552 + if(sp.end_time_s > latest_time_s) {
1553 + latest_time_s = sp.end_time_s;
1554 + store_metric_at_tier(rd, tier, t, sp, sp.end_time_s * USEC_PER_SEC);
1555 }
1556 }
1557
@@ -1551,12 +1586,12 @@ static void rrd2rrdr_log_request_response_metadata(RRDR *r
1586 , const char *msg
1587 ) {
1588
1554 - time_t first_entry_t = r->internal.qt->db.first_time_t;
1555 - time_t last_entry_t = r->internal.qt->db.last_time_t;
1589 + time_t first_entry_s = r->internal.qt->db.first_time_s;
1590 + time_t last_entry_s = r->internal.qt->db.last_time_s;
1591
1592 internal_error(
1558 - true,
1559 - "rrd2rrdr() on %s update every %ld with %s grouping %s (group: %zu, resampling_time: %ld, resampling_group: %zu), "
1593 + true,
1594 + "rrd2rrdr() on %s update every %ld with %s grouping %s (group: %zu, resampling_time: %ld, resampling_group: %zu), "
1595 "after (got: %ld, want: %ld, req: %ld, db: %ld), "
1596 "before (got: %ld, want: %ld, req: %ld, db: %ld), "
1597 "duration (got: %ld, want: %ld, req: %ld, db: %ld), "
@@ -1576,19 +1611,19 @@ static void rrd2rrdr_log_request_response_metadata(RRDR *r
1611 , r->after
1612 , after_wanted
1613 , after_requested
1579 - , first_entry_t
1614 + , first_entry_s
1615
1616 // before
1617 , r->before
1618 , before_wanted
1619 , before_requested
1585 - , last_entry_t
1620 + , last_entry_s
1621
1622 // duration
1623 , (long)(r->before - r->after + r->internal.qt->window.query_granularity)
1624 , (long)(before_wanted - after_wanted + r->internal.qt->window.query_granularity)
1625 , (long)before_requested - after_requested
1591 - , (long)((last_entry_t - first_entry_t) + r->internal.qt->window.query_granularity)
1626 + , (long)((last_entry_s - first_entry_s) + r->internal.qt->window.query_granularity)
1627
1628 // points
1629 , r->rows
@@ -1708,7 +1743,7 @@ bool query_target_calculate_window(QUERY_TARGET *qt) {
1743 time_t resampling_time_requested = qt->request.resampling_time;
1744 RRDR_OPTIONS options = qt->request.options;
1745 size_t tier = qt->request.tier;
1711 - time_t update_every = qt->db.minimum_latest_update_every;
1746 + time_t update_every = qt->db.minimum_latest_update_every_s;
1747
1748 // RULES
1749 // points_requested = 0
@@ -1763,30 +1798,30 @@ bool query_target_calculate_window(QUERY_TARGET *qt) {
1798 if (after_wanted == 0 || before_wanted == 0) {
1799 relative_period_requested = true;
1800
1766 - time_t first_entry_t = qt->db.first_time_t;
1767 - time_t last_entry_t = qt->db.last_time_t;
1801 + time_t first_entry_s = qt->db.first_time_s;
1802 + time_t last_entry_s = qt->db.last_time_s;
1803
1769 - if (first_entry_t == 0 || last_entry_t == 0) {
1770 - internal_error(true, "QUERY: no data detected on query '%s' (db first_entry_t = %ld, last_entry_t = %ld", qt->id, first_entry_t, last_entry_t);
1804 + if (first_entry_s == 0 || last_entry_s == 0) {
1805 + internal_error(true, "QUERY: no data detected on query '%s' (db first_entry_t = %ld, last_entry_t = %ld", qt->id, first_entry_s, last_entry_s);
1806 query_debug_log_free();
1807 return false;
1808 }
1809
1775 - query_debug_log(":first_entry_t %ld, last_entry_t %ld", first_entry_t, last_entry_t);
1810 + query_debug_log(":first_entry_t %ld, last_entry_t %ld", first_entry_s, last_entry_s);
1811
1812 if (after_wanted == 0) {
1778 - after_wanted = first_entry_t;
1813 + after_wanted = first_entry_s;
1814 query_debug_log(":zero after_wanted %ld", after_wanted);
1815 }
1816
1817 if (before_wanted == 0) {
1783 - before_wanted = last_entry_t;
1818 + before_wanted = last_entry_s;
1819 before_is_aligned_to_db_end = true;
1820 query_debug_log(":zero before_wanted %ld", before_wanted);
1821 }
1822
1823 if (points_wanted == 0) {
1789 - points_wanted = (last_entry_t - first_entry_t) / update_every;
1824 + points_wanted = (last_entry_s - first_entry_s) / update_every;
1825 query_debug_log(":zero points_wanted %zu", points_wanted);
1826 }
1827 }
@@ -1804,7 +1839,7 @@ bool query_target_calculate_window(QUERY_TARGET *qt) {
1839 update_every = rrdset_find_natural_update_every_for_timeframe(
1840 qt, after_wanted, before_wanted, points_wanted, options, tier);
1841
1807 - if (update_every <= 0) update_every = qt->db.minimum_latest_update_every;
1842 + if (update_every <= 0) update_every = qt->db.minimum_latest_update_every_s;
1843 query_debug_log(":natural update every %ld", update_every);
1844 }
1845
@@ -1975,7 +2010,8 @@ RRDR *rrd2rrdr_legacy(
2010 ONEWAYALLOC *owa,
2011 RRDSET *st, size_t points, time_t after, time_t before,
2012 RRDR_GROUPING group_method, time_t resampling_time, RRDR_OPTIONS options, const char *dimensions,
1978 - const char *group_options, time_t timeout, size_t tier, QUERY_SOURCE query_source) {
2013 + const char *group_options, time_t timeout, size_t tier, QUERY_SOURCE query_source,
2014 + STORAGE_PRIORITY priority) {
2015
2016 QUERY_TARGET_REQUEST qtr = {
2017 .st = st,
@@ -1990,6 +2026,7 @@ RRDR *rrd2rrdr_legacy(
2026 .timeout = timeout,
2027 .tier = tier,
2028 .query_source = query_source,
2029 + .priority = priority,
2030 };
2031
2032 return rrd2rrdr(owa, query_target_create(&qtr));
@@ -2056,16 +2093,22 @@ RRDR *rrd2rrdr(ONEWAYALLOC *owa, QUERY_TARGET *qt) {
2093 if (qt->request.timeout)
2094 now_realtime_timeval(&query_start_time);
2095
2096 + QUERY_ENGINE_OPS **ops = onewayalloc_callocz(r->internal.owa, qt->query.used, sizeof(QUERY_ENGINE_OPS *));
2097 + for(size_t c = 0, max = qt->query.used; c < max ; c++)
2098 + ops[c] = rrd2rrdr_query_prep(r, c);
2099 +
2100 for(size_t c = 0, max = qt->query.used; c < max ; c++) {
2101 // set the query target dimension options to rrdr
2102 r->od[c] = qt->query.array[c].dimension.options;
2103
2063 - r->od[c] |= RRDR_DIMENSION_SELECTED;
2064 -
2104 // reset the grouping for the new dimension
2105 r->internal.grouping_reset(r);
2106
2068 - rrd2rrdr_do_dimension(r, c);
2107 + if(ops[c]) {
2108 + r->od[c] |= RRDR_DIMENSION_SELECTED;
2109 + rrd2rrdr_query_execute(r, c, ops[c]);
2110 + }
2111 +
2112 if (qt->request.timeout)
2113 now_realtime_timeval(&query_current_time);
2114
web/api/queries/rrdr.h
+2 -1
@@ -138,7 +138,8 @@ RRDR *rrd2rrdr_legacy(
138 ONEWAYALLOC *owa,
139 RRDSET *st, size_t points, time_t after, time_t before,
140 RRDR_GROUPING group_method, time_t resampling_time, RRDR_OPTIONS options, const char *dimensions,
141 - const char *group_options, time_t timeout, size_t tier, QUERY_SOURCE query_source);
141 + const char *group_options, time_t timeout, size_t tier, QUERY_SOURCE query_source,
142 + STORAGE_PRIORITY priority);
143
144 RRDR *rrd2rrdr(ONEWAYALLOC *owa, struct query_target *qt);
145 bool query_target_calculate_window(struct query_target *qt);
web/api/queries/weights.c
+14 -4
@@ -520,6 +520,7 @@ NETDATA_DOUBLE *rrd2rrdr_ks2(
520 .group_options = group_options,
521 .tier = tier,
522 .query_source = QUERY_SOURCE_API_WEIGHTS,
523 + .priority = STORAGE_PRIORITY_NORMAL,
524 };
525
526 RRDR *r = rrd2rrdr(owa, query_target_create(&qtr));
@@ -638,7 +639,9 @@ static void rrdset_metric_correlations_volume(
639
640 options |= RRDR_OPTION_MATCH_IDS | RRDR_OPTION_ABSOLUTE | RRDR_OPTION_NATURAL_POINTS;
641
641 - QUERY_VALUE baseline_average = rrdmetric2value(host, rca, ria, rma, baseline_after, baseline_before, options, group_method, group_options, tier, 0, QUERY_SOURCE_API_WEIGHTS);
642 + QUERY_VALUE baseline_average = rrdmetric2value(host, rca, ria, rma, baseline_after, baseline_before,
643 + options, group_method, group_options, tier, 0,
644 + QUERY_SOURCE_API_WEIGHTS, STORAGE_PRIORITY_NORMAL);
645 merge_query_value_to_stats(&baseline_average, stats);
646
647 if(!netdata_double_isnumber(baseline_average.value)) {
@@ -646,7 +649,9 @@ static void rrdset_metric_correlations_volume(
649 baseline_average.value = 0.0;
650 }
651
649 - QUERY_VALUE highlight_average = rrdmetric2value(host, rca, ria, rma, after, before, options, group_method, group_options, tier, 0, QUERY_SOURCE_API_WEIGHTS);
652 + QUERY_VALUE highlight_average = rrdmetric2value(host, rca, ria, rma, after, before,
653 + options, group_method, group_options, tier, 0,
654 + QUERY_SOURCE_API_WEIGHTS, STORAGE_PRIORITY_NORMAL);
655 merge_query_value_to_stats(&highlight_average, stats);
656
657 if(!netdata_double_isnumber(highlight_average.value))
@@ -659,7 +664,9 @@ static void rrdset_metric_correlations_volume(
664
665 char highlight_countif_options[50 + 1];
666 snprintfz(highlight_countif_options, 50, "%s" NETDATA_DOUBLE_FORMAT, highlight_average.value < baseline_average.value ? "<" : ">", baseline_average.value);
662 - QUERY_VALUE highlight_countif = rrdmetric2value(host, rca, ria, rma, after, before, options, RRDR_GROUPING_COUNTIF, highlight_countif_options, tier, 0, QUERY_SOURCE_API_WEIGHTS);
667 + QUERY_VALUE highlight_countif = rrdmetric2value(host, rca, ria, rma, after, before,
668 + options, RRDR_GROUPING_COUNTIF, highlight_countif_options, tier, 0,
669 + QUERY_SOURCE_API_WEIGHTS, STORAGE_PRIORITY_NORMAL);
670 merge_query_value_to_stats(&highlight_countif, stats);
671
672 if(!netdata_double_isnumber(highlight_countif.value)) {
@@ -700,7 +707,10 @@ static void rrdset_weights_anomaly_rate(
707
708 options |= RRDR_OPTION_MATCH_IDS | RRDR_OPTION_ANOMALY_BIT | RRDR_OPTION_NATURAL_POINTS;
709
703 - QUERY_VALUE qv = rrdmetric2value(host, rca, ria, rma, after, before, options, group_method, group_options, tier, 0, QUERY_SOURCE_API_WEIGHTS);
710 + QUERY_VALUE qv = rrdmetric2value(host, rca, ria, rma, after, before,
711 + options, group_method, group_options, tier, 0,
712 + QUERY_SOURCE_API_WEIGHTS, STORAGE_PRIORITY_NORMAL);
713 +
714 merge_query_value_to_stats(&qv, stats);
715
716 if(netdata_double_isnumber(qv.value))
web/api/web_api_v1.c
+7 -20
@@ -388,7 +388,7 @@ inline int web_client_api_request_single_chart(RRDHOST *host, struct web_client
388 }
389
390 w->response.data->contenttype = CT_APPLICATION_JSON;
391 - st->last_accessed_time = now_realtime_sec();
391 + st->last_accessed_time_s = now_realtime_sec();
392 callback(st, w->response.data);
393 return HTTP_RESP_OK;
394
@@ -752,6 +752,7 @@ inline int web_client_api_request_v1_data(RRDHOST *host, struct web_client *w, c
752 .chart_label_key = chart_label_key,
753 .charts_labels_filter = chart_labels_filter,
754 .query_source = QUERY_SOURCE_API_DATA,
755 + .priority = STORAGE_PRIORITY_NORMAL,
756 };
757 qt = query_target_create(&qtr);
758
@@ -1068,15 +1069,13 @@ static inline void web_client_api_request_v1_info_mirrored_hosts(BUFFER *wb) {
1069 if (count > 0)
1070 buffer_strcat(wb, ",\n");
1071
1071 - netdata_mutex_lock(&host->receiver_lock);
1072 buffer_sprintf(
1073 wb, "\t\t{ \"guid\": \"%s\", \"hostname\": \"%s\", \"reachable\": %s, \"hops\": %d"
1074 , host->machine_guid
1075 , rrdhost_hostname(host)
1076 - , (host->receiver || host == localhost) ? "true" : "false"
1076 + , (host == localhost || !rrdhost_flag_check(host, RRDHOST_FLAG_ORPHAN)) ? "true" : "false"
1077 , host->system_info ? host->system_info->hops : (host == localhost) ? 0 : 1
1078 );
1079 - netdata_mutex_unlock(&host->receiver_lock);
1079
1080 rrdhost_aclk_state_lock(host);
1081 if (host->aclk_state.claimed_id)
@@ -1519,12 +1518,6 @@ static void web_client_api_v1_dbengine_stats_for_tier(BUFFER *wb, size_t tier) {
1518
1519 buffer_sprintf(wb,
1520 "\n\t\t\"default_granularity_secs\":%zu"
1522 - ",\n\t\t\"sizeof_metric\":%zu"
1523 - ",\n\t\t\"sizeof_metric_in_index\":%zu"
1524 - ",\n\t\t\"sizeof_page\":%zu"
1525 - ",\n\t\t\"sizeof_page_in_index\":%zu"
1526 - ",\n\t\t\"sizeof_extent\":%zu"
1527 - ",\n\t\t\"sizeof_page_in_extent\":%zu"
1521 ",\n\t\t\"sizeof_datafile\":%zu"
1522 ",\n\t\t\"sizeof_page_in_cache\":%zu"
1523 ",\n\t\t\"sizeof_point_data\":%zu"
@@ -1540,8 +1533,8 @@ static void web_client_api_v1_dbengine_stats_for_tier(BUFFER *wb, size_t tier) {
1533 ",\n\t\t\"pages_uncompressed_bytes\":%zu"
1534 ",\n\t\t\"pages_duration_secs\":%lld"
1535 ",\n\t\t\"single_point_pages\":%zu"
1543 - ",\n\t\t\"first_t\":%llu"
1544 - ",\n\t\t\"last_t\":%llu"
1536 + ",\n\t\t\"first_t\":%ld"
1537 + ",\n\t\t\"last_t\":%ld"
1538 ",\n\t\t\"database_retention_secs\":%lld"
1539 ",\n\t\t\"average_compression_savings\":%0.2f"
1540 ",\n\t\t\"average_point_duration_secs\":%0.2f"
@@ -1554,12 +1547,6 @@ static void web_client_api_v1_dbengine_stats_for_tier(BUFFER *wb, size_t tier) {
1547 ",\n\t\t\"disk_space\":%zu"
1548 ",\n\t\t\"max_disk_space\":%zu"
1549 , stats.default_granularity_secs
1557 - , stats.sizeof_metric
1558 - , stats.sizeof_metric_in_index
1559 - , stats.sizeof_page
1560 - , stats.sizeof_page_in_index
1561 - , stats.sizeof_extent
1562 - , stats.sizeof_page_in_extent
1550 , stats.sizeof_datafile
1551 , stats.sizeof_page_in_cache
1552 , stats.sizeof_point_data
@@ -1575,8 +1562,8 @@ static void web_client_api_v1_dbengine_stats_for_tier(BUFFER *wb, size_t tier) {
1562 , stats.pages_uncompressed_bytes
1563 , (long long)stats.pages_duration_secs
1564 , stats.single_point_pages
1578 - , stats.first_t
1579 - , stats.last_t
1565 + , stats.first_time_s
1566 + , stats.last_time_s
1567 , (long long)stats.database_retention_secs
1568 , stats.average_compression_savings
1569 , stats.average_point_duration_secs
web/server/static/static-threaded.c
+35 -30
@@ -307,7 +307,7 @@ static int web_server_rcv_callback(POLLINFO *pi, short int *events) {
307 web_client_send(w);
308 }
309
310 - if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) {
310 + else if(unlikely(w->mode == WEB_CLIENT_MODE_FILECOPY)) {
311 if(w->pollinfo_filecopy_slot == 0) {
312 debug(D_WEB_CLIENT, "%llu: FILECOPY DETECTED ON FD %d", w->id, pi->fd);
313
@@ -408,6 +408,10 @@ static void socket_listen_main_static_threaded_worker_cleanup(void *ptr) {
408 worker_unregister();
409 }
410
411 +static bool web_server_should_stop(void) {
412 + return !service_running(SERVICE_WEB_SERVER);
413 +}
414 +
415 void *socket_listen_main_static_threaded_worker(void *ptr) {
416 worker_private = (struct web_server_static_threaded_worker *)ptr;
417 worker_private->running = 1;
@@ -430,6 +434,7 @@ void *socket_listen_main_static_threaded_worker(void *ptr) {
434 , web_server_rcv_callback
435 , web_server_snd_callback
436 , NULL
437 + , web_server_should_stop
438 , web_allow_connections_from
439 , web_allow_connections_dns
440 , NULL
@@ -452,35 +457,35 @@ static void socket_listen_main_static_threaded_cleanup(void *ptr) {
457 struct netdata_static_thread *static_thread = (struct netdata_static_thread *)ptr;
458 static_thread->enabled = NETDATA_MAIN_THREAD_EXITING;
459
455 - int i, found = 0;
456 - usec_t max = 2 * USEC_PER_SEC, step = 50000;
457 -
458 - // we start from 1, - 0 is self
459 - for(i = 1; i < static_threaded_workers_count; i++) {
460 - if(static_workers_private_data[i].running) {
461 - found++;
462 - info("stopping worker %d", i + 1);
463 - netdata_thread_cancel(static_workers_private_data[i].thread);
464 - }
465 - else
466 - info("found stopped worker %d", i + 1);
467 - }
468 -
469 - while(found && max > 0) {
470 - max -= step;
471 - info("Waiting %d static web threads to finish...", found);
472 - sleep_usec(step);
473 - found = 0;
474 -
475 - // we start from 1, - 0 is self
476 - for(i = 1; i < static_threaded_workers_count; i++) {
477 - if (static_workers_private_data[i].running)
478 - found++;
479 - }
480 - }
481 -
482 - if(found)
483 - error("%d static web threads are taking too long to finish. Giving up.", found);
460 +// int i, found = 0;
461 +// usec_t max = 2 * USEC_PER_SEC, step = 50000;
462 +//
463 +// // we start from 1, - 0 is self
464 +// for(i = 1; i < static_threaded_workers_count; i++) {
465 +// if(static_workers_private_data[i].running) {
466 +// found++;
467 +// info("stopping worker %d", i + 1);
468 +// netdata_thread_cancel(static_workers_private_data[i].thread);
469 +// }
470 +// else
471 +// info("found stopped worker %d", i + 1);
472 +// }
473 +//
474 +// while(found && max > 0) {
475 +// max -= step;
476 +// info("Waiting %d static web threads to finish...", found);
477 +// sleep_usec(step);
478 +// found = 0;
479 +//
480 +// // we start from 1, - 0 is self
481 +// for(i = 1; i < static_threaded_workers_count; i++) {
482 +// if (static_workers_private_data[i].running)
483 +// found++;
484 +// }
485 +// }
486 +//
487 +// if(found)
488 +// error("%d static web threads are taking too long to finish. Giving up.", found);
489
490 info("closing all web server sockets...");
491 listen_sockets_close(&api_sockets);
web/server/web_client.c
+24
@@ -1314,6 +1314,9 @@ static inline int web_client_switch_host(RRDHOST *host, struct web_client *w, ch
1314 }
1315
1316 static inline int web_client_process_url(RRDHOST *host, struct web_client *w, char *url) {
1317 + if(unlikely(!service_running(ABILITY_WEB_REQUESTS)))
1318 + return web_client_permission_denied(w);
1319 +
1320 static uint32_t
1321 hash_api = 0,
1322 hash_netdata_conf = 0,
@@ -1935,3 +1938,24 @@ ssize_t web_client_receive(struct web_client *w)
1938
1939 return(bytes);
1940 }
1941 +
1942 +
1943 +int web_client_socket_is_now_used_for_streaming(struct web_client *w) {
1944 + // prevent the web_client from closing the streaming socket
1945 +
1946 + WEB_CLIENT_IS_DEAD(w);
1947 +
1948 + if(web_server_mode == WEB_SERVER_MODE_STATIC_THREADED) {
1949 + web_client_flag_set(w, WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET);
1950 + }
1951 + else {
1952 + if(w->ifd == w->ofd)
1953 + w->ifd = w->ofd = -1;
1954 + else
1955 + w->ifd = -1;
1956 + }
1957 +
1958 + buffer_flush(w->response.data);
1959 +
1960 + return HTTP_RESP_OK;
1961 +}
web/server/web_client.h
+6 -1
@@ -19,13 +19,16 @@ extern int web_enable_gzip, web_gzip_level, web_gzip_strategy;
19
20 // HTTP_CODES 4XX Client Errors
21 #define HTTP_RESP_BAD_REQUEST 400
22 +#define HTTP_RESP_UNAUTHORIZED 401
23 #define HTTP_RESP_FORBIDDEN 403
24 #define HTTP_RESP_NOT_FOUND 404
25 +#define HTTP_RESP_CONFLICT 409
26 #define HTTP_RESP_PRECOND_FAIL 412
27
28 // HTTP_CODES 5XX Server Errors
29 #define HTTP_RESP_INTERNAL_SERVER_ERROR 500
28 -#define HTTP_RESP_BACKEND_FETCH_FAILED 503
30 +#define HTTP_RESP_BACKEND_FETCH_FAILED 503 // 503 is right
31 +#define HTTP_RESP_SERVICE_UNAVAILABLE 503 // 503 is right
32 #define HTTP_RESP_GATEWAY_TIMEOUT 504
33 #define HTTP_RESP_BACKEND_RESPONSE_INVALID 591
34
@@ -206,6 +209,8 @@ int mysendfile(struct web_client *w, char *filename);
209 void web_client_build_http_header(struct web_client *w);
210 char *strip_control_characters(char *url);
211
212 +int web_client_socket_is_now_used_for_streaming(struct web_client *w);
213 +
214 #include "daemon/common.h"
215
216 #endif