New journal disk based indexing for agent memory reduction (#13885)
* Add read only option to netdata_mmap so files are accessed ousing PROT_READ * Initial functions to write the new journal file and switch to the new indexing * Cleanup code, add parameters to pg_cache_punch_hole to avoid updating page latets oldest times pg_cache insert to have parameter if page index locked needs to be done Page eviction functions will try to deallocate the descriptor as well (pg_cache_punch_hole without page_index time updates) Cleanup messages during startup * Cleanup messages during startup * Disbale extent caching for now, add placeholder for journal indexing and activation while the agent is running * Add main function to populate descriptors by checking the new journal indexing * prevent crash * fix for binary search crash * Avoid Time-of-check time-of-use filesystem race condition * always add a page * populate fixes - it is still incomplete * pg_cache_insert returns the descriptor that ends up in the page_index * Add populate next (Fix 1) * Fix compilation warnings, reactivate extent caching * Add populate next (Fix 2) * Add populate next (Fix 3) switch to the next entry or journal file when asking to populate descriptor with next * Fix resource leak and wrong sizeof * Rework page population (part 1) * Additional checksums added / journal validation * Cleanup (part 1) * Locking added and Cleanup (part 2) * Close journal file after new journal index activation * Skip warning when compiling without NETDATA_INTERNAL_CHECKS * Ignore empty index file (header and trailer and no metrics) * Try to remove all evicted descriptors (may prevent slight memory increase) * Evict pages also when we succesfully do try_reserve * Precache pages and cleanup * Add a separate cleanup thread to release unused descriptors * Check existence of key correctly * Fix total file size calculation * Statistics for journal descriptors * Track and release jourval v2 descriptors * Do not try to allocate pages for locality if under pressure * Do not track v2 descriptors when populating the page_index * Track page descriptors as they are inserted in the page index (per journal file) Scan journal files for pending items to cleanup Cleanup v2 descriptors only if they are not populated Check before adding to page cache to avoid memory allocation /free * Close journal file that has been processed and migrated to the new index Check for valid file before trying to truncate / close. This file has been closed during startup * Better calculation for the number of prefetched data pages based on the query end time Code cleanup and comments Add v2 populated descriptor expiration based on journal access time * Code cleanup * Faster indexing Better journal validation (more sanity checks) Detect new datafile/ journal creation and trigger index generation Switch to the new index / mark descriptors in memory as needed Update journal access time when a descriptor is returned Code cleanup (part 1) * Re activate descriptor clean Code cleanup * Allow locality precaching * Allow locality precaching for the same page alignment * Descriptor cleanup internal changed * Disable locality precaching * Precache only if not under pressure / internal cleanup at 60 seconds * Remove unused functions * Migrate on startup always Make sure the metric uuid is valid (we have a page_index) Prevent crash if no datafile is available when logging an error Remove unused functions * New warn limit for precaching Stress test v2 descriptor cleanup - Every 1s cleanup if it doesnt exist in cache - 60s cache eviction * Arrayalloc internal checks on free activated with NETDATA_ARRAYALLOC_INTERNAL_CHECKS Ability to add DESCRIPTOR_EXPIRATION_TIME and DESCRIPTOR_INTERVAL_CLEANUP during compile Defaults DESCRIPTOR_INTERVAL_CLEANUP = 60 and DESCRIPTOR_EXPIRATION_TIME = 600 * Lookup page index correctly * Calculate index time once * Detect a duplicate page when doing cache insert and during flushing of pages * Better logging * Descriptor validation (extent vs page index) when building an index file while the agent is running * Mark invalid entries in the journal v2 file * Schedule an index rebuild if a descriptor is found without an extent in the timerange we are processing Release descriptor lock to prevent random shutdown locks * Proper unlock * Skip descriptor cleanup when journal file v2 migration is running * Fix page cache statistics Remove multiple entries of the page_index from the page cache Cleanup * Adjust preload pages on pg_cache_next. Handle invalid descriptor properly Unlock properly * Better handling of invalid pages Journal indexing during runtime will scan all files to find potential ones to index * Reactivate migration on startup Evict descriptors to cause migration Don't count the entries in page index (calculate when processing the extent list) Check for valid extent since we may set the extent to NULL on startup if it is invalid Better structure init Address valgrind issues * Add don't fork/dump option * Add separate lock to protect accessing a datafile's extent list Comment out some unused code (for now) Abort descriptor cleanup if we are force flushing pages (page cache under pressure) * Check for index and schedule when data flush completes Configure max datafile size during compilation Keep a separate JudyL array for descriptors Skip quota test if we are deleting descriptors or explicitly flushing pages under pressure * Fix * set function when waiters are waken up * add the line number to trace the deadlock * add thread id * add wait list * init to zero * disable thread cancelability inside dbengine rrdeng_load_page_next() * make sure the owner is the thread * disable thread cancelability for replication as a whole * Check and queue indexing after first page flush * Queue indexing after a small delay to allow some time for page flushing * tracing of waiters only when compiled with internal checks * Mark descr with extent_entry * Return page timeout * Check if a journalfile is ready to be indexed Migrate the descriptors or evict if possible Compilation warning fix * Use page index if indexing during startup Mark if journalfile should be checked depending on whether we can migrate or delete a page during indexing * require 3x max message size as sender buffer * fix for the msg of the adaptive buffer size * fix for the msg of the duplicate replication commands * Disable descriptor deletion during migration * Detect descriptor with same start page time * sender sorts replication requests before fullfilling them; receiver does not send duplicate replication requests * dbengine never allows past timestamps to be collected * do not accept values same as last data point stored in dbengine * replicate non-overlapping ranges * a better replication logic to avoid sending overlapping data to parents * Do not start journal migration in parallel * Always update page index times * Fix page index first / last times on load * internal log when replication responses do not match the requests or when replication commands are sent while there are others inflight * do not log out of bounds RBEGIN if it is the last replication command we sent * better checking of past data collection points * better checking of past data collection points - optimized * fix corruption during decompression of streaming * Add config to disable journal indexing Add config parameter for detailed journal integrity check (Metric chain validation check during startup) pg cache insert drop check for existing page Fix crc calculation for metric headers * children disable compression globally, only when the compression gives an error * turn boolean member into RRDHOST OPTION * Compilation warnings * Remove unused code * replication sender statistics * replication sender statistics set to 100% when no replication requests are pending * Fix casting warning Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
Stelios Fragkakis committed
Nov 15, 2022 at 23:00 UTC
224b051a2b2bab39a4b536e531ab9ca590bf31bb
30 files changed
+2320
-651
collectors/plugins.d/pluginsd_parser.c
+30
-2
@@ -293,9 +293,25 @@ PARSER_RC pluginsd_chart_definition_end(char **words, size_t num_words, void *us
293
// rrdhost_hostname(host), rrdset_id(st),
294
// (unsigned long long)first_entry_child, (unsigned long long)last_entry_child);
295
296
- rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
296
+ bool ok = true;
297
+ if(!rrdset_flag_check(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS)) {
298
+
299
+#ifdef NETDATA_INTERNAL_CHECKS
300
+ st->replay.start_streaming = false;
301
+ st->replay.after = 0;
302
+ st->replay.before = 0;
303
+#endif
304
+
305
+ rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
306
+ rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS);
307
+
308
+ ok = replicate_chart_request(send_to_plugin, user_object->parser, host, st, first_entry_child,
309
+ last_entry_child, 0, 0);
310
+ }
311
+ else {
312
+ internal_error(true, "RRDSET: not sending duplicate replication request for chart '%s'", rrdset_id(st));
313
+ }
314
298
- bool ok = replicate_chart_request(send_to_plugin, user_object->parser, host, st, first_entry_child, last_entry_child, 0, 0);
315
return ok ? PARSER_RC_OK : PARSER_RC_ERROR;
316
}
317
@@ -875,6 +891,11 @@ PARSER_RC pluginsd_replay_rrdset_begin(char **words, size_t num_words, void *use
891
time_t start_time = strtol(start_time_str, NULL, 0);
892
time_t end_time = strtol(end_time_str, NULL, 0);
893
894
+ internal_error(
895
+ (!st->replay.start_streaming && (end_time < st->replay.after || start_time > st->replay.before)),
896
+ "REPLAY: received a " PLUGINSD_KEYWORD_REPLAY_BEGIN " on chart '%s' ('%s') on host '%s', from %ld to %ld, which does not match our request (%ld to %ld).",
897
+ rrdset_name(st), rrdset_id(st), rrdhost_hostname(st->rrdhost), start_time, end_time, st->replay.after, st->replay.before);
898
+
899
if(start_time && end_time) {
900
if (start_time > end_time) {
901
error("REPLAY: requested a " PLUGINSD_KEYWORD_REPLAY_BEGIN " on chart '%s' ('%s') on host '%s', but timings are invalid (%ld to %ld). Disabling it.",
@@ -1135,11 +1156,18 @@ PARSER_RC pluginsd_replay_end(char **words, size_t num_words, void *user)
1156
st->counter++;
1157
st->counter_done++;
1158
1159
+#ifdef NETDATA_INTERNAL_CHECKS
1160
+ st->replay.start_streaming = false;
1161
+ st->replay.after = 0;
1162
+ st->replay.before = 0;
1163
+#endif
1164
+
1165
if (start_streaming) {
1166
if (st->update_every != update_every_child)
1167
rrdset_set_update_every(st, update_every_child);
1168
1169
rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
1170
+ rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS);
1171
rrdset_flag_clear(st, RRDSET_FLAG_SYNC_CLOCK);
1172
return PARSER_RC_OK;
1173
}
daemon/global_statistics.c
+3
@@ -865,6 +865,7 @@ static void dbengine_statistics_charts(void) {
865
866
{
867
static RRDSET *st_long_term_pages = NULL;
868
+ static RRDDIM *rd_memory = NULL;
869
static RRDDIM *rd_total = NULL;
870
static RRDDIM *rd_insertions = NULL;
871
static RRDDIM *rd_deletions = NULL;
@@ -885,6 +886,7 @@ static void dbengine_statistics_charts(void) {
886
localhost->rrd_update_every,
887
RRDSET_TYPE_LINE);
888
889
+ rd_memory = rrddim_add(st_long_term_pages, "journal v2 descriptors", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
890
rd_total = rrddim_add(st_long_term_pages, "total", NULL, 1, 1, RRD_ALGORITHM_ABSOLUTE);
891
rd_insertions = rrddim_add(st_long_term_pages, "insertions", NULL, 1, 1, RRD_ALGORITHM_INCREMENTAL);
892
rd_deletions = rrddim_add(st_long_term_pages, "deletions", NULL, -1, 1, RRD_ALGORITHM_INCREMENTAL);
@@ -893,6 +895,7 @@ static void dbengine_statistics_charts(void) {
895
} else
896
rrdset_next(st_long_term_pages);
897
898
+ rrddim_set_by_pointer(st_long_term_pages, rd_memory, (collected_number)stats_array[37]);
899
rrddim_set_by_pointer(st_long_term_pages, rd_total, (collected_number)stats_array[2]);
900
rrddim_set_by_pointer(st_long_term_pages, rd_insertions, (collected_number)stats_array[5]);
901
rrddim_set_by_pointer(st_long_term_pages, rd_deletions, (collected_number)stats_array[6]);
daemon/main.c
+3
@@ -680,6 +680,9 @@ static void get_netdata_configured_variables() {
680
681
db_engine_use_malloc = config_get_boolean(CONFIG_SECTION_DB, "dbengine page cache with malloc", CONFIG_BOOLEAN_NO);
682
default_rrdeng_page_cache_mb = (int) config_get_number(CONFIG_SECTION_DB, "dbengine page cache size MB", default_rrdeng_page_cache_mb);
683
+ db_engine_journal_indexing = config_get_boolean(CONFIG_SECTION_DB, "dbengine enable journal indexing", CONFIG_BOOLEAN_YES);
684
+ db_engine_journal_check = config_get_boolean(CONFIG_SECTION_DB, "dbengine enable journal integrity check", CONFIG_BOOLEAN_NO);
685
+
686
if(default_rrdeng_page_cache_mb < RRDENG_MIN_PAGE_CACHE_SIZE_MB) {
687
error("Invalid page cache size %d given. Defaulting to %d.", default_rrdeng_page_cache_mb, RRDENG_MIN_PAGE_CACHE_SIZE_MB);
688
default_rrdeng_page_cache_mb = RRDENG_MIN_PAGE_CACHE_SIZE_MB;
database/engine/datafile.c
+34
-7
@@ -1,9 +1,29 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
#include "rrdengine.h"
3
4
+void df_extent_delete_all_unsafe(struct rrdengine_datafile *datafile)
5
+{
6
+ struct extent_info *extent = datafile->extents.first, *next_extent;
7
+
8
+ char path[RRDENG_PATH_MAX];
9
+
10
+ generate_journalfilepath_v2(datafile, path, sizeof(path));
11
+ internal_error(true, "Deleting extents of file %s", path);
12
+ unsigned count = 0;
13
+ while (extent) {
14
+ next_extent = extent->next;
15
+ freez(extent);
16
+ count++;
17
+ extent = next_extent;
18
+ }
19
+ datafile->extents.first = NULL;
20
+ internal_error(true, "Deleted %u extents of file %s", count, path);
21
+}
22
+
23
void df_extent_insert(struct extent_info *extent)
24
{
25
struct rrdengine_datafile *datafile = extent->datafile;
26
+ uv_rwlock_wrlock(&datafile->extent_rwlock);
27
28
if (likely(NULL != datafile->extents.last)) {
29
datafile->extents.last->next = extent;
@@ -12,10 +32,14 @@ void df_extent_insert(struct extent_info *extent)
32
datafile->extents.first = extent;
33
}
34
datafile->extents.last = extent;
35
+
36
+ uv_rwlock_wrunlock(&datafile->extent_rwlock);
37
}
38
39
void datafile_list_insert(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile)
40
{
41
+ uv_rwlock_wrlock(&ctx->datafiles.rwlock);
42
+
43
if (likely(NULL != ctx->datafiles.last)) {
44
ctx->datafiles.last->next = datafile;
45
}
@@ -23,12 +47,13 @@ void datafile_list_insert(struct rrdengine_instance *ctx, struct rrdengine_dataf
47
ctx->datafiles.first = datafile;
48
}
49
ctx->datafiles.last = datafile;
50
+
51
+ uv_rwlock_wrunlock(&ctx->datafiles.rwlock);
52
}
53
28
-void datafile_list_delete(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile)
54
+void datafile_list_delete_unsafe(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile)
55
{
56
struct rrdengine_datafile *next;
31
-
57
next = datafile->next;
58
fatal_assert((NULL != next) && (ctx->datafiles.first == datafile) && (ctx->datafiles.last != datafile));
59
ctx->datafiles.first = next;
@@ -44,6 +69,7 @@ static void datafile_init(struct rrdengine_datafile *datafile, struct rrdengine_
69
datafile->file = (uv_file)0;
70
datafile->pos = 0;
71
datafile->extents.first = datafile->extents.last = NULL; /* will be populated by journalfile */
72
+ fatal_assert(0 == uv_rwlock_init(&datafile->extent_rwlock));
73
datafile->journalfile = NULL;
74
datafile->next = NULL;
75
datafile->ctx = ctx;
@@ -97,7 +123,7 @@ int unlink_data_file(struct rrdengine_datafile *datafile)
123
return ret;
124
}
125
100
-int destroy_data_file(struct rrdengine_datafile *datafile)
126
+int destroy_data_file_unsafe(struct rrdengine_datafile *datafile)
127
{
128
struct rrdengine_instance *ctx = datafile->ctx;
129
uv_fs_t req;
@@ -176,7 +202,7 @@ int create_data_file(struct rrdengine_datafile *datafile)
202
uv_fs_req_cleanup(&req);
203
posix_memfree(superblock);
204
if (ret < 0) {
179
- destroy_data_file(datafile);
205
+ destroy_data_file_unsafe(datafile);
206
return ret;
207
}
208
@@ -304,10 +330,8 @@ static int scan_data_files(struct rrdengine_instance *ctx)
330
331
datafiles = callocz(MIN(ret, MAX_DATAFILES), sizeof(*datafiles));
332
for (matched_files = 0 ; UV_EOF != uv_fs_scandir_next(&req, &dent) && matched_files < MAX_DATAFILES ; ) {
307
- info("Scanning file \"%s/%s\"", ctx->dbfiles_path, dent.name);
333
ret = sscanf(dent.name, DATAFILE_PREFIX RRDENG_FILE_NUMBER_SCAN_TMPL DATAFILE_EXTENSION, &tier, &no);
334
if (2 == ret) {
310
- info("Matched file \"%s/%s\"", ctx->dbfiles_path, dent.name);
335
datafile = mallocz(sizeof(*datafile));
336
datafile_init(datafile, ctx, tier, no);
337
datafiles[matched_files++] = datafile;
@@ -337,6 +361,7 @@ static int scan_data_files(struct rrdengine_instance *ctx)
361
journalfile = mallocz(sizeof(*journalfile));
362
datafile->journalfile = journalfile;
363
journalfile_init(journalfile, datafile);
364
+ journalfile->file_index = i;
365
ret = load_journal_file(ctx, journalfile, datafile);
366
if (0 != ret) {
367
if (!must_delete_pair) /* If datafile is still open close it */
@@ -346,6 +371,7 @@ static int scan_data_files(struct rrdengine_instance *ctx)
371
if (must_delete_pair) {
372
char path[RRDENG_PATH_MAX];
373
374
+ // TODO: Also delete the version 2
375
error("Deleting invalid data and journal file pair.");
376
ret = unlink_journal_file(journalfile);
377
if (!ret) {
@@ -407,7 +433,7 @@ int create_new_datafile_pair(struct rrdengine_instance *ctx, unsigned tier, unsi
433
return 0;
434
435
error_after_journalfile:
410
- destroy_data_file(datafile);
436
+ destroy_data_file_unsafe(datafile);
437
freez(journalfile);
438
error_after_datafile:
439
freez(datafile);
@@ -421,6 +447,7 @@ int init_data_files(struct rrdengine_instance *ctx)
447
{
448
int ret;
449
450
+ fatal_assert(0 == uv_rwlock_init(&ctx->datafiles.rwlock));
451
ret = scan_data_files(ctx);
452
if (ret < 0) {
453
error("Failed to scan path \"%s\".", ctx->dbfiles_path);
database/engine/datafile.h
+12
-2
@@ -13,7 +13,13 @@ struct rrdengine_instance;
13
#define DATAFILE_PREFIX "datafile-"
14
#define DATAFILE_EXTENSION ".ndf"
15
16
+#ifndef MAX_DATAFILE_SIZE
17
#define MAX_DATAFILE_SIZE (1073741824LU)
18
+#endif
19
+#if MIN_DATAFILE_SIZE > MAX_DATAFILE_SIZE
20
+#error MIN_DATAFILE_SIZE > MAX_DATAFILE_SIZE
21
+#endif
22
+
23
#define MIN_DATAFILE_SIZE (4194304LU)
24
#define MAX_DATAFILES (65536) /* Supports up to 64TiB for now */
25
#define TARGET_DATAFILES (20)
@@ -26,6 +32,7 @@ struct extent_info {
32
uint8_t number_of_pages;
33
struct rrdengine_datafile *datafile;
34
struct extent_info *next;
35
+ uint32_t index; // This is the entent index for version 2
36
struct rrdeng_page_descr *pages[];
37
};
38
@@ -41,6 +48,7 @@ struct rrdengine_datafile {
48
unsigned fileno;
49
uv_file file;
50
uint64_t pos;
51
+ uv_rwlock_t extent_rwlock;
52
struct rrdengine_instance *ctx;
53
struct rrdengine_df_extents extents;
54
struct rrdengine_journalfile *journalfile;
@@ -48,20 +56,22 @@ struct rrdengine_datafile {
56
};
57
58
struct rrdengine_datafile_list {
59
+ uv_rwlock_t rwlock;
60
struct rrdengine_datafile *first; /* oldest */
61
struct rrdengine_datafile *last; /* newest */
62
};
63
64
void df_extent_insert(struct extent_info *extent);
65
void datafile_list_insert(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile);
57
-void datafile_list_delete(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile);
66
+void datafile_list_delete_unsafe(struct rrdengine_instance *ctx, struct rrdengine_datafile *datafile);
67
void generate_datafilepath(struct rrdengine_datafile *datafile, char *str, size_t maxlen);
68
int close_data_file(struct rrdengine_datafile *datafile);
69
int unlink_data_file(struct rrdengine_datafile *datafile);
61
-int destroy_data_file(struct rrdengine_datafile *datafile);
70
+int destroy_data_file_unsafe(struct rrdengine_datafile *datafile);
71
int create_data_file(struct rrdengine_datafile *datafile);
72
int create_new_datafile_pair(struct rrdengine_instance *ctx, unsigned tier, unsigned fileno);
73
int init_data_files(struct rrdengine_instance *ctx);
74
void finalize_data_files(struct rrdengine_instance *ctx);
75
+void df_extent_delete_all_unsafe(struct rrdengine_datafile *datafile);
76
77
#endif /* NETDATA_DATAFILE_H */
\ No newline at end of file
database/engine/journalfile.c
+1092
-34
@@ -1,6 +1,25 @@
1
// SPDX-License-Identifier: GPL-3.0-or-later
2
#include "rrdengine.h"
3
4
+void queue_journalfile_v2_migration(struct rrdengine_worker_config *wc)
5
+{
6
+ struct rrdeng_work *work_request;
7
+
8
+ if (unlikely(wc->running_journal_migration))
9
+ return;
10
+
11
+ work_request = mallocz(sizeof(*work_request));
12
+ work_request->req.data = work_request;
13
+ work_request->wc = wc;
14
+ work_request->count = 0;
15
+ work_request->rerun = false;
16
+ wc->running_journal_migration = 1;
17
+ if (unlikely(uv_queue_work(wc->loop, &work_request->req, start_journal_indexing, after_journal_indexing))) {
18
+ freez(work_request);
19
+ wc->running_journal_migration = 0;
20
+ }
21
+}
22
+
23
static void flush_transaction_buffer_cb(uv_fs_t* req)
24
{
25
struct generic_io_descriptor *io_descr = req->data;
@@ -47,6 +66,7 @@ void wal_flush_transaction_buffer(struct rrdengine_worker_config* wc)
66
io_descr->bytes = size;
67
io_descr->pos = journalfile->pos;
68
io_descr->req.data = io_descr;
69
+ io_descr->data = journalfile;
70
io_descr->completion = NULL;
71
72
io_descr->iov = uv_buf_init((void *)io_descr->buf, size);
@@ -93,6 +113,12 @@ void * wal_get_transaction_buffer(struct rrdengine_worker_config* wc, unsigned s
113
return ctx->commit_log.buf + buf_pos;
114
}
115
116
+void generate_journalfilepath_v2(struct rrdengine_datafile *datafile, char *str, size_t maxlen)
117
+{
118
+ (void) snprintfz(str, maxlen, "%s/" WALFILE_PREFIX RRDENG_FILE_NUMBER_PRINT_TMPL WALFILE_EXTENSION_V2,
119
+ datafile->ctx->dbfiles_path, datafile->tier, datafile->fileno);
120
+}
121
+
122
void generate_journalfilepath(struct rrdengine_datafile *datafile, char *str, size_t maxlen)
123
{
124
(void) snprintfz(str, maxlen, "%s/" WALFILE_PREFIX RRDENG_FILE_NUMBER_PRINT_TMPL WALFILE_EXTENSION,
@@ -104,28 +130,51 @@ void journalfile_init(struct rrdengine_journalfile *journalfile, struct rrdengin
130
journalfile->file = (uv_file)0;
131
journalfile->pos = 0;
132
journalfile->datafile = datafile;
133
+ journalfile->journal_data = NULL;
134
+ journalfile->journal_data_size = 0;
135
+ journalfile->JudyL_array = (Pvoid_t) NULL;
136
+ journalfile->data = NULL;
137
+ journalfile->file_index = 0;
138
+ journalfile->last_access = 0;
139
}
140
109
-int close_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
141
+static int close_uv_file(struct rrdengine_datafile *datafile, uv_file file)
142
{
111
- struct rrdengine_instance *ctx = datafile->ctx;
112
- uv_fs_t req;
143
int ret;
144
char path[RRDENG_PATH_MAX];
145
116
- generate_journalfilepath(datafile, path, sizeof(path));
117
-
118
- ret = uv_fs_close(NULL, &req, journalfile->file, NULL);
146
+ uv_fs_t req;
147
+ ret = uv_fs_close(NULL, &req, file, NULL);
148
if (ret < 0) {
149
+ generate_journalfilepath(datafile, path, sizeof(path));
150
error("uv_fs_close(%s): %s", path, uv_strerror(ret));
121
- ++ctx->stats.fs_errors;
151
+ ++datafile->ctx->stats.fs_errors;
152
rrd_stat_atomic_add(&global_fs_errors, 1);
153
}
154
uv_fs_req_cleanup(&req);
125
-
155
return ret;
156
}
157
158
+int close_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
159
+{
160
+ struct rrdengine_instance *ctx = datafile->ctx;
161
+ char path[RRDENG_PATH_MAX];
162
+
163
+ if (likely(journalfile->journal_data)) {
164
+ if (munmap(journalfile->journal_data, journalfile->journal_data_size)) {
165
+ generate_journalfilepath_v2(datafile, path, sizeof(path));
166
+ error("Failed to unmap journal index file for %s", path);
167
+ ++ctx->stats.fs_errors;
168
+ rrd_stat_atomic_add(&global_fs_errors, 1);
169
+ }
170
+ journalfile->journal_data = NULL;
171
+ journalfile->journal_data_size = 0;
172
+ return 0;
173
+ }
174
+
175
+ return close_uv_file(datafile, journalfile->file);
176
+}
177
+
178
int unlink_journal_file(struct rrdengine_journalfile *journalfile)
179
{
180
struct rrdengine_datafile *datafile = journalfile->datafile;
@@ -149,14 +198,16 @@ int unlink_journal_file(struct rrdengine_journalfile *journalfile)
198
return ret;
199
}
200
152
-int destroy_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
201
+int destroy_journal_file_unsafe(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
202
{
203
struct rrdengine_instance *ctx = datafile->ctx;
204
uv_fs_t req;
205
int ret;
206
char path[RRDENG_PATH_MAX];
207
+ char path_v2[RRDENG_PATH_MAX];
208
209
generate_journalfilepath(datafile, path, sizeof(path));
210
+ generate_journalfilepath_v2(datafile, path_v2, sizeof(path));
211
212
ret = uv_fs_ftruncate(NULL, &req, journalfile->file, 0, NULL);
213
if (ret < 0) {
@@ -166,9 +217,12 @@ int destroy_journal_file(struct rrdengine_journalfile *journalfile, struct rrden
217
}
218
uv_fs_req_cleanup(&req);
219
169
- ret = uv_fs_close(NULL, &req, journalfile->file, NULL);
220
+ (void) close_uv_file(datafile, journalfile->file);
221
+
222
+ // This is the new journal v2 index file
223
+ ret = uv_fs_unlink(NULL, &req, path_v2, NULL);
224
if (ret < 0) {
171
- error("uv_fs_close(%s): %s", path, uv_strerror(ret));
225
+ error("uv_fs_fsunlink(%s): %s", path, uv_strerror(ret));
226
++ctx->stats.fs_errors;
227
rrd_stat_atomic_add(&global_fs_errors, 1);
228
}
@@ -182,8 +236,15 @@ int destroy_journal_file(struct rrdengine_journalfile *journalfile, struct rrden
236
}
237
uv_fs_req_cleanup(&req);
238
239
+ ++ctx->stats.journalfile_deletions;
240
++ctx->stats.journalfile_deletions;
241
242
+ if (journalfile->journal_data) {
243
+ if (munmap(journalfile->journal_data, journalfile->journal_data_size)) {
244
+ error("Failed to unmap index file %s", path_v2);
245
+ }
246
+ }
247
+
248
return ret;
249
}
250
@@ -227,7 +288,7 @@ int create_journal_file(struct rrdengine_journalfile *journalfile, struct rrdeng
288
uv_fs_req_cleanup(&req);
289
posix_memfree(superblock);
290
if (ret < 0) {
230
- destroy_journal_file(journalfile, datafile);
291
+ destroy_journal_file_unsafe(journalfile, datafile);
292
return ret;
293
}
294
@@ -366,36 +427,67 @@ static void restore_extent_metadata(struct rrdengine_instance *ctx, struct rrden
427
uv_rwlock_wrlock(&pg_cache->metrics_index.lock);
428
PValue = JudyHSIns(&pg_cache->metrics_index.JudyHS_array, temp_id, sizeof(uuid_t), PJE0);
429
fatal_assert(NULL == *PValue); /* TODO: figure out concurrency model */
369
- *PValue = page_index = create_page_index(temp_id, ctx);
370
- page_index->prev = pg_cache->metrics_index.last_page_index;
371
- pg_cache->metrics_index.last_page_index = page_index;
430
+ *PValue = page_index = create_page_index(temp_id, ctx);
431
+ page_index->prev = pg_cache->metrics_index.last_page_index;
432
+ pg_cache->metrics_index.last_page_index = page_index;
433
uv_rwlock_wrunlock(&pg_cache->metrics_index.lock);
434
}
435
375
- descr = pg_cache_create_descr();
436
+ // Lookup this descriptor
437
+ Word_t p_time = start_time_ut / USEC_PER_SEC;
438
+ uv_rwlock_rdlock(&page_index->lock);
439
+ PValue = JudyLFirst(page_index->JudyL_array, &p_time, PJE0);
440
+ descr = (NULL == PValue) ? NULL: *PValue;
441
+ uv_rwlock_rdunlock(&page_index->lock);
442
+
443
+ bool descr_found = false;
444
+ if (unlikely(descr && descr->start_time_ut == start_time_ut)) {
445
+ // We have this descriptor already
446
+ descr_found = true;
447
+
448
+#ifdef NETDATA_INTERNAL_CHECKS
449
+ char uuid_str[UUID_STR_LEN];
450
+ uuid_unparse_lower(page_index->id, uuid_str);
451
+ internal_error(true, "REMOVING UUID %s with %lu, %lu length=%u (fileno=%u), extent database offset = %lu, size = %u",
452
+ uuid_str, start_time_ut, end_time_ut, jf_metric_data->descr[i].page_length, descr->extent->datafile->fileno,
453
+ descr->extent->offset, descr->extent->size);
454
+ internal_error(true, "APPLYING UUID %s with %lu, %lu length=%u (fileno=%u), extent database offset = %lu, size = %u",
455
+ uuid_str, start_time_ut, end_time_ut, jf_metric_data->descr[i].page_length, extent->datafile->fileno,
456
+ extent->offset, extent->size);
457
+#endif
458
+ // Remove entry from previous extent
459
+ unlink_descriptor_extent_unsafe(descr);
460
+ internal_error(true, "REMOVING UUID %s with %lu OK", uuid_str, start_time_ut);
461
+ }
462
+ else {
463
+ descr = pg_cache_create_descr();
464
+ descr->id = &page_index->id;
465
+ }
466
+
467
descr->page_length = jf_metric_data->descr[i].page_length;
468
descr->start_time_ut = start_time_ut;
469
descr->end_time_ut = end_time_ut;
470
descr->update_every_s = (update_every_s > 0) ? (uint32_t)update_every_s : (page_index->latest_update_every_s);
380
- descr->id = &page_index->id;
471
+
472
descr->extent = extent;
473
descr->type = page_type;
474
extent->pages[valid_pages++] = descr;
384
- pg_cache_insert(ctx, page_index, descr);
475
+ if (likely(!descr_found))
476
+ (void)pg_cache_insert(ctx, page_index, descr, true);
477
386
- if(page_index->latest_time_ut == descr->end_time_ut)
478
+ if (page_index->latest_time_ut == descr->end_time_ut)
479
page_index->latest_update_every_s = descr->update_every_s;
480
481
if(descr->update_every_s == 0)
390
- fatal(
391
- "DBENGINE: page descriptor update every is zero, end_time_ut = %llu, start_time_ut = %llu, entries = %zu",
482
+ fatal("DBENGINE: page descriptor update every is zero, end_time_ut = %llu, start_time_ut = %llu, entries = %zu",
483
(unsigned long long)end_time_ut, (unsigned long long)start_time_ut, entries);
484
}
485
486
extent->number_of_pages = valid_pages;
487
397
- if (likely(valid_pages))
488
+ if (likely(valid_pages)) {
489
df_extent_insert(extent);
490
+ }
491
else {
492
freez(extent);
493
ctx->load_errors[LOAD_ERRORS_DROPPED_EXTENT].counter++;
@@ -444,13 +536,13 @@ static unsigned replay_transaction(struct rrdengine_instance *ctx, struct rrdeng
536
return size_bytes;
537
}
538
switch (jf_header->type) {
447
- case STORE_DATA:
448
- debug(D_RRDENGINE, "Replaying transaction %"PRIu64"", jf_header->id);
449
- restore_extent_metadata(ctx, journalfile, buf + sizeof(*jf_header), payload_length);
450
- break;
451
- default:
452
- error("Unknown transaction type. Skipping record.");
453
- break;
539
+ case STORE_DATA:
540
+ debug(D_RRDENGINE, "Replaying transaction %"PRIu64"", jf_header->id);
541
+ restore_extent_metadata(ctx, journalfile, buf + sizeof(*jf_header), payload_length);
542
+ break;
543
+ default:
544
+ error("Unknown transaction type. Skipping record.");
545
+ break;
546
}
547
548
return size_bytes;
@@ -518,12 +610,905 @@ static uint64_t iterate_transactions(struct rrdengine_instance *ctx, struct rrde
610
if (likely(journal_is_mmapped))
611
buf += size_bytes;
612
}
521
-skip_file:
613
+ skip_file:
614
if (unlikely(!journal_is_mmapped))
615
posix_memfree(buf);
616
return max_id;
617
}
618
619
+bool unlink_descriptor_extent_unsafe(struct rrdeng_page_descr *descr)
620
+{
621
+ if (unlikely(!descr || !descr->extent))
622
+ return true;
623
+
624
+ struct extent_info *extent = descr->extent;
625
+ for (uint8_t index = 0; index < extent->number_of_pages; index++) {
626
+ if (extent->pages[index] == descr) {
627
+ extent->pages[index] = NULL;
628
+ return true;
629
+ }
630
+ }
631
+ return false;
632
+}
633
+
634
+// Checks that the extent list checksum is valid
635
+static int check_journal_v2_extent_list (void *data_start, size_t file_size)
636
+{
637
+ UNUSED(file_size);
638
+ uLong crc;
639
+
640
+ struct journal_v2_header *j2_header = (void *) data_start;
641
+ struct journal_v2_block_trailer *journal_v2_trailer;
642
+
643
+ journal_v2_trailer = (struct journal_v2_block_trailer *) ((uint8_t *) data_start + j2_header->extent_trailer_offset);
644
+ crc = crc32(0L, Z_NULL, 0);
645
+ crc = crc32(crc, (uint8_t *) data_start + j2_header->extent_offset, j2_header->extent_count * sizeof(struct journal_extent_list));
646
+ if (unlikely(crc32cmp(journal_v2_trailer->checksum, crc))) {
647
+ error("Extent list CRC32 check: FAILED");
648
+ return 1;
649
+ }
650
+
651
+ return 0;
652
+}
653
+
654
+// Checks that the metric list (UUIDs) checksum is valid
655
+static int check_journal_v2_metric_list(void *data_start, size_t file_size)
656
+{
657
+ UNUSED(file_size);
658
+ uLong crc;
659
+
660
+ struct journal_v2_header *j2_header = (void *) data_start;
661
+ struct journal_v2_block_trailer *journal_v2_trailer;
662
+
663
+ journal_v2_trailer = (struct journal_v2_block_trailer *) ((uint8_t *) data_start + j2_header->metric_trailer_offset);
664
+ crc = crc32(0L, Z_NULL, 0);
665
+ crc = crc32(crc, (uint8_t *) data_start + j2_header->metric_offset, j2_header->metric_count * sizeof(struct journal_metric_list));
666
+ if (unlikely(crc32cmp(journal_v2_trailer->checksum, crc))) {
667
+ error("Metric list CRC32 check: FAILED");
668
+ return 1;
669
+ }
670
+ return 0;
671
+}
672
+
673
+static int check_journal_v2_file(void *data_start, size_t file_size, uint32_t original_size)
674
+{
675
+ int rc;
676
+ uLong crc;
677
+
678
+ struct journal_v2_header *j2_header = (void *) data_start;
679
+ struct journal_v2_block_trailer *journal_v2_trailer;
680
+
681
+ if (j2_header->magic == JOURVAL_V2_REBUILD_MAGIC)
682
+ return 2;
683
+
684
+ // Magic failure
685
+ if (j2_header->magic != JOURVAL_V2_MAGIC)
686
+ return 1;
687
+
688
+ if (j2_header->total_file_size != file_size)
689
+ return 1;
690
+
691
+ if (original_size && j2_header->original_file_size != original_size)
692
+ return 1;
693
+
694
+ journal_v2_trailer = (struct journal_v2_block_trailer *) ((uint8_t *) data_start + file_size - sizeof(*journal_v2_trailer));
695
+
696
+ crc = crc32(0L, Z_NULL, 0);
697
+ crc = crc32(crc, (void *) j2_header, sizeof(*j2_header));
698
+
699
+ rc = crc32cmp(journal_v2_trailer->checksum, crc);
700
+ if (unlikely(rc)) {
701
+ error("File CRC32 check: FAILED");
702
+ return 1;
703
+ }
704
+
705
+ rc = check_journal_v2_extent_list(data_start, file_size);
706
+ if (rc) return 1;
707
+
708
+ rc = check_journal_v2_metric_list(data_start, file_size);
709
+ if (rc) return 1;
710
+
711
+ if (!db_engine_journal_check)
712
+ return 0;
713
+
714
+ // Verify complete UUID chain
715
+
716
+ struct journal_metric_list *metric = (void *) (data_start + j2_header->metric_offset);
717
+
718
+ unsigned verified = 0;
719
+ unsigned entries;
720
+ unsigned total_pages = 0;
721
+
722
+ info("Checking %u metrics that exist in the journal", j2_header->metric_count);
723
+ for (entries = 0; entries < j2_header->metric_count; entries++) {
724
+
725
+ char uuid_str[UUID_STR_LEN];
726
+ uuid_unparse_lower(metric->uuid, uuid_str);
727
+ struct journal_page_header *metric_list_header = (void *) (data_start + metric->page_offset);
728
+ struct journal_page_header local_metric_list_header = *metric_list_header;
729
+
730
+ local_metric_list_header.crc = JOURVAL_V2_MAGIC;
731
+
732
+ crc = crc32(0L, Z_NULL, 0);
733
+ crc = crc32(crc, (void *) &local_metric_list_header, sizeof(local_metric_list_header));
734
+ rc = crc32cmp(metric_list_header->checksum, crc);
735
+
736
+ internal_error(true, "Index %u : %s entries %u at offset %u (%llu -- %llu) verified, HEADER CRC computed %lu, stored %u", entries, uuid_str, metric->entries, metric->page_offset,
737
+ j2_header->start_time_ut + (usec_t) metric->delta_start * USEC_PER_SEC, j2_header->start_time_ut + (usec_t) metric->delta_end * USEC_PER_SEC,
738
+ crc, metric_list_header->crc);
739
+ if (!rc) {
740
+ struct journal_v2_block_trailer *journal_trailer =
741
+ (void *) data_start + metric->page_offset + sizeof(struct journal_page_header) + (metric_list_header->entries * sizeof(struct journal_page_list));
742
+
743
+ crc = crc32(0L, Z_NULL, 0);
744
+ crc = crc32(crc, (uint8_t *) metric_list_header + sizeof(struct journal_page_header), metric_list_header->entries * sizeof(struct journal_page_list));
745
+ rc = crc32cmp(journal_trailer->checksum, crc);
746
+ internal_error(rc, "Index %u : %s entries %u at offset %u verified, DATA CRC computed %lu, stored %u", entries, uuid_str, metric->entries, metric->page_offset,
747
+ crc, metric_list_header->crc);
748
+ if (!rc) {
749
+ total_pages += metric_list_header->entries;
750
+ verified++;
751
+ }
752
+ }
753
+
754
+ metric++;
755
+ if (((uint8_t *) metric - (uint8_t *) data_start) > (uint32_t) file_size) {
756
+ info("Verification failed EOF reached -- total entries %u, verified %u", entries, verified);
757
+ return 1;
758
+ }
759
+ }
760
+
761
+ if (entries != verified) {
762
+ info("Verification failed -- total entries %u, verified %u", entries, verified);
763
+ return 1;
764
+ }
765
+ info("Verification succeeded -- total entries %u, verified %u (%u total pages)", entries, verified, total_pages);
766
+
767
+ return 0;
768
+}
769
+
770
+int load_journal_file_v2(struct rrdengine_instance *ctx, struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile)
771
+{
772
+ struct page_cache *pg_cache = &ctx->pg_cache;
773
+ struct pg_cache_page_index *page_index;
774
+ int ret, fd;
775
+ uint64_t file_size;
776
+ char path[RRDENG_PATH_MAX];
777
+ struct stat statbuf;
778
+ uint32_t original_file_size = 0;
779
+
780
+ generate_journalfilepath(datafile, path, sizeof(path));
781
+ ret = stat(path, &statbuf);
782
+ if (!ret)
783
+ original_file_size = (uint32_t)statbuf.st_size;
784
+
785
+ generate_journalfilepath_v2(datafile, path, sizeof(path));
786
+
787
+ fd = open(path, O_RDONLY);
788
+ if (fd < 0) {
789
+ if (errno == ENOENT)
790
+ return 1;
791
+ ++ctx->stats.fs_errors;
792
+ rrd_stat_atomic_add(&global_fs_errors, 1);
793
+ error("Failed to open %s", path);
794
+ return 1;
795
+ }
796
+
797
+ ret = fstat(fd, &statbuf);
798
+ if (ret) {
799
+ error("Failed to get file information for %s", path);
800
+ close(fd);
801
+ return 1;
802
+ }
803
+
804
+ file_size = (size_t)statbuf.st_size;
805
+
806
+ if (file_size < sizeof(struct journal_v2_header)) {
807
+ error_report("Invalid file %s. Not the expected size", path);
808
+ close(fd);
809
+ return 1;
810
+ }
811
+
812
+ usec_t start_loading = now_realtime_usec();
813
+ uint8_t *data_start = mmap(NULL, file_size, PROT_READ, MAP_SHARED, fd, 0);
814
+ if (data_start == MAP_FAILED) {
815
+ close(fd);
816
+ return 1;
817
+ }
818
+ close(fd);
819
+
820
+ info("Checking integrity of %s", path);
821
+ int rc = check_journal_v2_file(data_start, file_size, original_file_size);
822
+ if (rc) {
823
+ if (rc == 2)
824
+ error_report("File %s needs to be rebuilt", path);
825
+ else
826
+ error_report("File %s is invalid", path);
827
+ if (unlikely(munmap(data_start, file_size)))
828
+ error("Failed to unmap %s", path);
829
+ return 1;
830
+ }
831
+
832
+ struct journal_v2_header *j2_header = (void *) data_start;
833
+
834
+ size_t entries = j2_header->metric_count;
835
+
836
+ if (!entries) {
837
+ if (unlikely(munmap(data_start, file_size)))
838
+ error("Failed to unmap %s", path);
839
+ return 1;
840
+ }
841
+
842
+ rc = madvise(data_start, file_size, MADV_DONTFORK);
843
+ if (rc)
844
+ error("MADV_DONTFORK: setting failed");
845
+
846
+ rc = madvise(data_start, file_size, MADV_DONTDUMP);
847
+ if (rc)
848
+ error("MADV_DONTDUMP: setting failed");
849
+
850
+ struct journal_metric_list *metric = (struct journal_metric_list *) (data_start + j2_header->metric_offset);
851
+
852
+ uv_rwlock_wrlock(&pg_cache->metrics_index.lock);
853
+
854
+ // Initialize the journal file to be able to access the data
855
+ journalfile->journal_data = data_start;
856
+ journalfile->journal_data_size = file_size;
857
+
858
+ usec_t header_start_time = j2_header->start_time_ut;
859
+ for (size_t i=0; i < entries; i++) {
860
+ Pvoid_t *PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, metric->uuid, sizeof(uuid_t));
861
+ if (likely(NULL != PValue)) {
862
+ page_index = *PValue;
863
+ }
864
+ else {
865
+ PValue = JudyHSIns(&pg_cache->metrics_index.JudyHS_array, metric->uuid, sizeof(uuid_t), PJE0);
866
+ fatal_assert(NULL == *PValue);
867
+ *PValue = page_index = create_page_index(&metric->uuid, ctx);
868
+ page_index->oldest_time_ut = LLONG_MAX;
869
+ page_index->latest_time_ut = 0;
870
+ page_index->prev = pg_cache->metrics_index.last_page_index;
871
+ pg_cache->metrics_index.last_page_index = page_index;
872
+ }
873
+
874
+ usec_t metric_start_ut = header_start_time + (usec_t ) metric->delta_start * USEC_PER_SEC;
875
+ usec_t metric_end_ut = header_start_time + (usec_t ) metric->delta_end * USEC_PER_SEC;
876
+
877
+ if (page_index->oldest_time_ut > metric_start_ut)
878
+ page_index->oldest_time_ut = metric_start_ut;
879
+
880
+ if (page_index->latest_time_ut < metric_end_ut)
881
+ page_index->latest_time_ut = metric_end_ut;
882
+
883
+ ++page_index->page_count;
884
+ ++pg_cache->page_descriptors;
885
+ metric++;
886
+ }
887
+
888
+ uv_rwlock_wrunlock(&pg_cache->metrics_index.lock);
889
+
890
+ info("Journal file \"%s\" loaded (size:%"PRIu64") with %lu metrics in %d ms",
891
+ path, file_size, entries,
892
+ (int) ((now_realtime_usec() - start_loading) / USEC_PER_MS));
893
+ return 0;
894
+}
895
+
896
+
897
+struct metric_info_s {
898
+ uuid_t *id;
899
+ struct pg_cache_page_index *page_index;
900
+ uint32_t entries;
901
+ time_t min_index_time_s;
902
+ time_t max_index_time_s;
903
+ usec_t min_time_ut;
904
+ usec_t max_time_ut;
905
+ uint32_t page_list_header;
906
+ Pvoid_t JudyL_array;
907
+};
908
+
909
+struct journal_metric_list_to_sort {
910
+ struct metric_info_s *metric_info;
911
+};
912
+
913
+static int journal_metric_compare (const void *item1, const void *item2)
914
+{
915
+ const struct metric_info_s *metric1 = ((struct journal_metric_list_to_sort *) item1)->metric_info;
916
+ const struct metric_info_s *metric2 = ((struct journal_metric_list_to_sort *) item2)->metric_info;
917
+
918
+ return uuid_compare(*(metric1->id), *(metric2->id));
919
+}
920
+
921
+
922
+// Write list of extents for the journalfile
923
+void *journal_v2_write_extent_list(struct rrdengine_journalfile *journalfile, void *data)
924
+{
925
+ struct extent_info *extent = journalfile->datafile->extents.first;
926
+ struct journal_extent_list *j2_extent = (void *) data;
927
+ while (extent) {
928
+ j2_extent->datafile_offset = extent->offset;
929
+ j2_extent->datafile_size = extent->size;
930
+ j2_extent->pages = extent->number_of_pages;
931
+ j2_extent->file_index = journalfile->file_index;
932
+ j2_extent++;
933
+ extent = extent->next;
934
+ };
935
+ return j2_extent;
936
+}
937
+
938
+static int verify_journal_space(struct journal_v2_header *j2_header, void *data, uint32_t bytes)
939
+{
940
+ if ((unsigned long)(((uint8_t *) data - (uint8_t *) j2_header->data) + bytes) > (j2_header->total_file_size - sizeof(struct journal_v2_block_trailer)))
941
+ return 1;
942
+
943
+ return 0;
944
+}
945
+
946
+void *journal_v2_write_metric_page(struct journal_v2_header *j2_header, void *data, struct metric_info_s *metric_info, uint32_t pages_offset)
947
+{
948
+ struct journal_metric_list *metric = (void *) data;
949
+
950
+ if (verify_journal_space(j2_header, data, sizeof(*metric)))
951
+ return NULL;
952
+
953
+ uuid_copy(metric->uuid, *metric_info->id);
954
+ metric->entries = metric_info->entries;
955
+ metric->page_offset = pages_offset;
956
+ metric->delta_start = (metric_info->min_time_ut - j2_header->start_time_ut) / USEC_PER_SEC;
957
+ metric->delta_end = (metric_info->max_time_ut - j2_header->start_time_ut) / USEC_PER_SEC;
958
+
959
+ return ++metric;
960
+}
961
+
962
+void *journal_v2_write_data_page_header(struct journal_v2_header *j2_header __maybe_unused, void *data, struct metric_info_s *metric_info, uint32_t uuid_offset)
963
+{
964
+ struct journal_page_header *data_page_header = (void *) data;
965
+ uLong crc;
966
+
967
+ uuid_copy(data_page_header->uuid, *metric_info->id);
968
+ data_page_header->entries = metric_info->entries;
969
+ data_page_header->uuid_offset = uuid_offset; // data header OFFSET poings to METRIC in the directory
970
+ data_page_header->crc = JOURVAL_V2_MAGIC;
971
+ crc = crc32(0L, Z_NULL, 0);
972
+ crc = crc32(crc, (void *) data_page_header, sizeof(*data_page_header));
973
+ crc32set(data_page_header->checksum, crc);
974
+ return ++data_page_header;
975
+}
976
+
977
+void *journal_v2_write_data_page_trailer(struct journal_v2_header *j2_header __maybe_unused, void *data, void *page_header)
978
+{
979
+ struct journal_page_header *data_page_header = (void *) page_header;
980
+ struct journal_v2_block_trailer *journal_trailer = (void *) data;
981
+ uLong crc;
982
+
983
+ crc = crc32(0L, Z_NULL, 0);
984
+ crc = crc32(crc, (uint8_t *) page_header + sizeof(struct journal_page_header), data_page_header->entries * sizeof(struct journal_page_list));
985
+ crc32set(journal_trailer->checksum, crc);
986
+ return ++journal_trailer;
987
+}
988
+
989
+void *journal_v2_write_data_page(struct journal_v2_header *j2_header, void *data, struct rrdeng_page_descr *descr)
990
+{
991
+ if (unlikely(!descr))
992
+ return data;
993
+
994
+ struct journal_page_list *data_page = data;
995
+
996
+ // verify that we can write number of bytes
997
+ if (verify_journal_space(j2_header, data, sizeof(*data_page)))
998
+ return NULL;
999
+
1000
+ fatal_assert(descr->extent != NULL);
1001
+
1002
+ uint32_t extent_index = unlikely(NULL == descr->extent) ? UINT32_MAX : descr->extent->index;
1003
+
1004
+ data_page->delta_start_s = (descr->start_time_ut - j2_header->start_time_ut) / USEC_PER_SEC;
1005
+ data_page->delta_end_s = (descr->end_time_ut - j2_header->start_time_ut) / USEC_PER_SEC;
1006
+ data_page->extent_index = extent_index;
1007
+ data_page->update_every_s = (uint16_t) descr->update_every_s;
1008
+ data_page->page_length = descr->page_length;
1009
+ data_page->type = descr->type;
1010
+
1011
+ // Rebuild on start to resolve unknown entry
1012
+ if (unlikely(UINT32_MAX == extent_index))
1013
+ j2_header->magic = JOURVAL_V2_REBUILD_MAGIC;
1014
+
1015
+ return ++data_page;
1016
+}
1017
+
1018
+// For a page_index write all descr @ time entries
1019
+// Must be recorded in metric_info->entries
1020
+void *journal_v2_write_descriptors(struct journal_v2_header *j2_header, void *data, struct metric_info_s *metric_info, struct rrdengine_journalfile *journalfile)
1021
+{
1022
+ struct rrdeng_page_descr *descr;
1023
+ Pvoid_t *PValue;
1024
+
1025
+ struct journal_page_list *data_page = (void *)data;
1026
+ struct page_cache *pg_cache = &journalfile->datafile->ctx->pg_cache;
1027
+ struct pg_cache_page_index *page_index;
1028
+
1029
+ uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
1030
+ PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, metric_info->id, sizeof(uuid_t));
1031
+ page_index = (NULL == PValue) ? NULL : *PValue;
1032
+ uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
1033
+
1034
+ if (page_index == NULL)
1035
+ return data_page;
1036
+
1037
+ // We need to write all descriptors with index metric_info->min_index_time_s, metric_info->max_index_time_s
1038
+ // that belong to this journal file
1039
+
1040
+ Word_t index_time = metric_info->min_index_time_s;
1041
+ unsigned entries = 0;
1042
+
1043
+ uv_rwlock_rdlock(&page_index->lock);
1044
+
1045
+ Pvoid_t JudyL_array = metric_info->JudyL_array ? metric_info->JudyL_array : page_index->JudyL_array;
1046
+
1047
+ // Need page_index lock if running live
1048
+ for (PValue = JudyLFirst(JudyL_array, &index_time, PJE0),
1049
+ descr = unlikely(NULL == PValue) ? NULL : *PValue;
1050
+ descr != NULL;
1051
+ PValue = JudyLNext(JudyL_array, &index_time, PJE0),
1052
+ descr = unlikely(NULL == PValue) ? NULL : *PValue) {
1053
+
1054
+ if (unlikely((time_t) index_time > metric_info->max_index_time_s) || entries == metric_info->entries)
1055
+ break;
1056
+
1057
+ // Write one descriptor and return the next data page location
1058
+ data_page = journal_v2_write_data_page(j2_header, (void *)data_page, descr);
1059
+
1060
+ entries++;
1061
+ if (unlikely(!data_page))
1062
+ break;
1063
+ }
1064
+
1065
+ uv_rwlock_rdunlock(&page_index->lock);
1066
+ return data_page;
1067
+}
1068
+
1069
+static void journal_v2_remove_active_descriptors(struct rrdengine_journalfile *journalfile, struct metric_info_s *metric_info, bool startup)
1070
+{
1071
+ if (true == startup) {
1072
+ // This is during startup, so we are the only ones accessing the structures
1073
+ // thats why we can safely remote the entire page_index->JudyL_array
1074
+ struct rrdeng_page_descr *descr;
1075
+ Word_t index_time;
1076
+ Pvoid_t *PValue;
1077
+ struct pg_cache_page_index *page_index;
1078
+
1079
+ page_index = metric_info->page_index;
1080
+
1081
+ for (index_time = 0, PValue = JudyLFirst(page_index->JudyL_array, &index_time, PJE0),
1082
+ descr = unlikely(NULL == PValue) ? NULL : *PValue; descr != NULL;
1083
+ PValue = JudyLNext(page_index->JudyL_array, &index_time, PJE0),
1084
+ descr = unlikely(NULL == PValue) ? NULL : *PValue) {
1085
+
1086
+ rrdeng_page_descr_freez(descr);
1087
+ }
1088
+ (void)JudyLFreeArray(&page_index->JudyL_array, PJE0);
1089
+ }
1090
+ else {
1091
+ // This is during runtime
1092
+ struct rrdeng_page_descr *descr;
1093
+ Pvoid_t *PValue;
1094
+ struct pg_cache_page_index *page_index = metric_info->page_index;
1095
+ struct page_cache *pg_cache = &page_index->ctx->pg_cache;
1096
+ struct rrdengine_instance *ctx = page_index->ctx;
1097
+
1098
+ Word_t index_time = metric_info->min_index_time_s;
1099
+ uint32_t metric_info_offset = metric_info->page_list_header;
1100
+
1101
+ struct journal_page_header *page_list_header = (struct journal_page_header *) ((uint8_t *) journalfile->journal_data + metric_info_offset);
1102
+ struct journal_v2_header *journal_header = (struct journal_v2_header *) journalfile->journal_data;
1103
+ // Sanity check that we refer to the same UUID
1104
+ fatal_assert(uuid_compare(page_list_header->uuid, *metric_info->id) == 0);
1105
+
1106
+ struct journal_page_list *page_list = (struct journal_page_list *)((uint8_t *) page_list_header + sizeof(*page_list_header));
1107
+ struct journal_extent_list *extent_list = (void *)((uint8_t *)journal_header + journal_header->extent_offset);
1108
+
1109
+ uint32_t index = 0;
1110
+ uint32_t entries = page_list_header->entries;
1111
+
1112
+ uv_rwlock_rdlock(&page_index->lock);
1113
+
1114
+ bool mark_journalfile_for_expiration_check = false;
1115
+ for (PValue = JudyLFirst(metric_info->JudyL_array, &index_time, PJE0),
1116
+ descr = unlikely(NULL == PValue) ? NULL : *PValue;
1117
+ descr != NULL;
1118
+ PValue = JudyLNext(metric_info->JudyL_array, &index_time, PJE0),
1119
+ descr = unlikely(NULL == PValue) ? NULL : *PValue) {
1120
+
1121
+ if (unlikely((time_t) index_time > metric_info->max_index_time_s) || index == entries)
1122
+ break;
1123
+
1124
+ if (descr->extent_entry || (!descr->extent_entry && descr->extent && descr->extent->datafile->journalfile != journalfile))
1125
+ continue;
1126
+
1127
+ struct journal_page_list *page_entry = &page_list[index++];
1128
+
1129
+ if (likely(page_entry->extent_index != UINT32_MAX)) {
1130
+
1131
+ fatal_assert(descr->extent->offset == extent_list[page_entry->extent_index].datafile_offset);
1132
+ fatal_assert(descr->extent->size == extent_list[page_entry->extent_index].datafile_size);
1133
+
1134
+ rrdeng_page_descr_mutex_lock(ctx, descr);
1135
+ while (!pg_cache_try_get_unsafe(descr, 1)) {
1136
+ pg_cache_wait_event_unsafe(descr);
1137
+ }
1138
+
1139
+ descr->extent_entry = &extent_list[page_entry->extent_index];
1140
+ descr->extent = NULL;
1141
+ descr->file = journalfile->datafile->file;
1142
+ ++pg_cache->active_descriptors;
1143
+ pg_cache_put_unsafe(descr);
1144
+ rrdeng_try_deallocate_pg_cache_descr(ctx, descr);
1145
+ rrdeng_page_descr_mutex_unlock(ctx, descr);
1146
+ mark_journalfile_for_expiration_check = true;
1147
+ }
1148
+ }
1149
+
1150
+ if (mark_journalfile_for_expiration_check) {
1151
+ uint32_t page_offset = (uint8_t *)page_list_header - (uint8_t *)journalfile->journal_data;
1152
+ mark_journalfile_descriptor(pg_cache, journalfile, page_offset, 1);
1153
+ }
1154
+
1155
+ uv_rwlock_rdunlock(&page_index->lock);
1156
+ }
1157
+}
1158
+
1159
+bool descriptor_is_corrupted(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
1160
+{
1161
+ struct page_cache *pg_cache = &ctx->pg_cache;
1162
+ struct pg_cache_page_index *page_index = get_page_index(pg_cache, descr->id);
1163
+
1164
+ if (unlikely(!page_index))
1165
+ return true;
1166
+
1167
+ time_t index_time_s = (time_t) (descr->start_time_ut / USEC_PER_SEC);
1168
+ struct rrdeng_page_descr *idx_descr = get_descriptor(page_index, index_time_s);
1169
+
1170
+ bool is_corrupted = (idx_descr != descr);
1171
+
1172
+#ifdef NETDATA_INTERNAL_CHECKS
1173
+ char uuid_str[UUID_STR_LEN];
1174
+ uuid_unparse_lower(page_index->id, uuid_str);
1175
+ internal_error(is_corrupted, "Descriptor corrupted (Extent %p Judy %p) @ %ld", descr, idx_descr, index_time_s);
1176
+#endif
1177
+
1178
+ return is_corrupted;
1179
+}
1180
+
1181
+static bool journalfile_ready_to_index(struct rrdengine_datafile *datafile)
1182
+{
1183
+ struct extent_info *extent = datafile->extents.first;
1184
+ while (extent) {
1185
+ uint8_t extent_pages = extent->number_of_pages;
1186
+ for (uint8_t index = 0; index < extent_pages; index++) {
1187
+ struct rrdeng_page_descr *descr = extent->pages[index];
1188
+ if (unlikely(!descr))
1189
+ continue;
1190
+ if (unlikely(!descr->extent))
1191
+ return false;
1192
+ }
1193
+ extent = extent->next;
1194
+ }
1195
+ return true;
1196
+}
1197
+
1198
+// Migrate the journalfile pointed by datafile
1199
+// activate : make the new file active immediately
1200
+// journafile data will be set and descriptors (if deleted) will be repopulated as needed
1201
+// startup : if the migration is done during agent startup
1202
+// this will allow us to optimize certain things
1203
+void migrate_journal_file_v2(struct rrdengine_datafile *datafile, bool activate, bool startup)
1204
+{
1205
+ char path[RRDENG_PATH_MAX];
1206
+ size_t number_of_extents = 0; // Number of extents
1207
+ size_t number_of_metrics = 0; // Number of unique metrics (UUIDS)
1208
+ size_t number_of_pages = 0; // Total number of descriptors @ time
1209
+ Pvoid_t *PValue;
1210
+ Pvoid_t metrics_JudyL_array = NULL;
1211
+ Pvoid_t metrics_JudyHS_array = NULL;
1212
+ struct rrdengine_instance *ctx = datafile->ctx;
1213
+ struct rrdengine_journalfile *journalfile = datafile->journalfile;
1214
+ usec_t min_time_ut = LLONG_MAX;
1215
+ usec_t max_time_ut = 0;
1216
+ struct metric_info_s *metric_info;
1217
+
1218
+ // Do nothing if we already have a mmaped file
1219
+ if (unlikely(journalfile->journal_data))
1220
+ return;
1221
+
1222
+ generate_journalfilepath_v2(datafile, path, sizeof(path));
1223
+ info("Indexing file %s", path);
1224
+
1225
+#ifdef NETDATA_INTERNAL_CHECKS
1226
+ usec_t start_loading = now_realtime_usec();
1227
+#endif
1228
+
1229
+ if (false == startup)
1230
+ uv_rwlock_rdlock(&ctx->datafiles.rwlock);
1231
+
1232
+ struct extent_info *extent = datafile->extents.first;
1233
+ while (extent) {
1234
+ uint8_t extent_pages = extent->number_of_pages;
1235
+ for (uint8_t index = 0; index < extent_pages; index++) {
1236
+ struct rrdeng_page_descr *descr = extent->pages[index];
1237
+
1238
+ if (unlikely(!descr))
1239
+ continue;
1240
+
1241
+ if (false == startup) {
1242
+ if (unlikely(descriptor_is_corrupted(ctx, descr))) {
1243
+ extent->pages[index] = NULL;
1244
+ continue;
1245
+ }
1246
+ }
1247
+
1248
+ PValue = JudyHSGet(metrics_JudyHS_array, descr->id, sizeof(uuid_t));
1249
+ if (likely(NULL != PValue)) {
1250
+ metric_info = *PValue;
1251
+ }
1252
+ else {
1253
+ PValue = JudyHSIns(&metrics_JudyHS_array, descr->id, sizeof(uuid_t), PJE0);
1254
+ *PValue = metric_info = mallocz(sizeof(*metric_info));
1255
+
1256
+ metric_info->entries =0;
1257
+ metric_info->min_time_ut = LLONG_MAX;
1258
+ metric_info->max_time_ut = 0;
1259
+ metric_info->min_index_time_s = LLONG_MAX;
1260
+ metric_info->max_index_time_s = 0;
1261
+ metric_info->id = descr->id;
1262
+ metric_info->page_index = NULL;
1263
+ metric_info->page_list_header = 0;
1264
+ metric_info->JudyL_array = (Pvoid_t) NULL;
1265
+
1266
+ PValue = JudyLIns(&metrics_JudyL_array,number_of_metrics, PJE0);
1267
+ *PValue = metric_info;
1268
+ number_of_metrics++;
1269
+ }
1270
+ time_t current_index_time_s = (time_t) (descr->start_time_ut / USEC_PER_SEC);
1271
+
1272
+ if (metric_info->min_time_ut > descr->start_time_ut) {
1273
+ metric_info->min_time_ut = descr->start_time_ut;
1274
+ metric_info->min_index_time_s = current_index_time_s;
1275
+ }
1276
+
1277
+ metric_info->max_index_time_s= MAX(metric_info->max_index_time_s, current_index_time_s);
1278
+ metric_info->max_time_ut = MAX(metric_info->max_time_ut , descr->end_time_ut);
1279
+
1280
+ if (false == startup) {
1281
+ PValue = JudyLIns(&metric_info->JudyL_array, current_index_time_s, PJE0);
1282
+ fatal_assert(NULL != PValue && NULL == *PValue);
1283
+ *PValue = descr;
1284
+ }
1285
+
1286
+ metric_info->entries++;
1287
+ number_of_pages++;
1288
+
1289
+ // Maintain the min max times to add to the journal header
1290
+ min_time_ut = MIN(min_time_ut, descr->start_time_ut);
1291
+ max_time_ut = MAX(max_time_ut, descr->end_time_ut);
1292
+ }
1293
+ extent->index = number_of_extents++;
1294
+ extent = extent->next;
1295
+ }
1296
+ info("File %s has %lu extents", path, number_of_extents);
1297
+
1298
+ if (false == startup)
1299
+ uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
1300
+
1301
+ internal_error(true, "Scan and extbuild metric %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1302
+
1303
+ // Calculate total jourval file size
1304
+ size_t total_file_size = 0;
1305
+ total_file_size += (sizeof(struct journal_v2_header) + JOURNAL_V2_HEADER_PADDING_SZ);
1306
+
1307
+ // Extents will start here
1308
+ uint32_t extent_offset = total_file_size;
1309
+ total_file_size += (number_of_extents * sizeof(struct journal_extent_list));
1310
+
1311
+ uint32_t extent_offset_trailer = total_file_size;
1312
+ total_file_size += sizeof(struct journal_v2_block_trailer);
1313
+
1314
+ // UUID list will start here
1315
+ uint32_t metrics_offset = total_file_size;
1316
+ total_file_size += (number_of_metrics * sizeof(struct journal_metric_list));
1317
+
1318
+ // UUID list trailer
1319
+ uint32_t metric_offset_trailer = total_file_size;
1320
+ total_file_size += sizeof(struct journal_v2_block_trailer);
1321
+
1322
+ // descr @ time will start here
1323
+ uint32_t pages_offset = total_file_size;
1324
+ total_file_size += (number_of_pages * (sizeof(struct journal_page_list) + sizeof(struct journal_page_header) + sizeof(struct journal_v2_block_trailer)));
1325
+
1326
+ // File trailer
1327
+ uint32_t trailer_offset = total_file_size;
1328
+ total_file_size += sizeof(struct journal_v2_block_trailer);
1329
+
1330
+ uint8_t *data_start = netdata_mmap(path, total_file_size, MAP_SHARED, 0, false);
1331
+ uint8_t *data = data_start;
1332
+
1333
+ memset(data_start, 0, extent_offset);
1334
+
1335
+ // Write header
1336
+ struct journal_v2_header j2_header;
1337
+ memset(&j2_header, 0, sizeof(j2_header));
1338
+
1339
+ j2_header.magic = JOURVAL_V2_MAGIC;
1340
+ j2_header.start_time_ut = min_time_ut;
1341
+ j2_header.end_time_ut = max_time_ut;
1342
+ j2_header.extent_count = number_of_extents;
1343
+ j2_header.extent_offset = extent_offset;
1344
+ j2_header.metric_count = number_of_metrics;
1345
+ j2_header.metric_offset = metrics_offset;
1346
+ j2_header.page_count = number_of_pages;
1347
+ j2_header.page_offset = pages_offset;
1348
+ j2_header.extent_trailer_offset = extent_offset_trailer;
1349
+ j2_header.metric_trailer_offset = metric_offset_trailer;
1350
+ j2_header.total_file_size = total_file_size;
1351
+ j2_header.original_file_size = (uint32_t) journalfile->pos;
1352
+ j2_header.data = data_start; // Used during migration
1353
+
1354
+ struct journal_v2_block_trailer *journal_v2_trailer;
1355
+
1356
+ // Write all the extents we have
1357
+ data = journal_v2_write_extent_list(journalfile, data_start + extent_offset);
1358
+ internal_error(true, "Write extent list so far %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1359
+
1360
+ fatal_assert(data == data_start + extent_offset_trailer);
1361
+
1362
+ // Calculate CRC for extents
1363
+ journal_v2_trailer = (struct journal_v2_block_trailer *) (data_start + extent_offset_trailer);
1364
+ uLong crc;
1365
+ crc = crc32(0L, Z_NULL, 0);
1366
+ crc = crc32(crc, (uint8_t *) data_start + extent_offset, number_of_extents * sizeof(struct journal_extent_list));
1367
+ crc32set(journal_v2_trailer->checksum, crc);
1368
+
1369
+ internal_error(true, "CALCULATE CRC FOR EXTENT %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1370
+ // Skip the trailer, point to the metrics off
1371
+ data += sizeof(struct journal_v2_block_trailer);
1372
+
1373
+ // Sanity check -- we must be at the metrics_offset
1374
+ fatal_assert(data == data_start + metrics_offset);
1375
+
1376
+ // Allocate array to sort UUIDs and keep them sorted in the journal because we want to do binary search when we do lookups
1377
+ struct journal_metric_list_to_sort *uuid_list = mallocz(number_of_metrics * sizeof(struct journal_metric_list_to_sort));
1378
+
1379
+ Word_t Index;
1380
+ struct page_cache *pg_cache = &ctx->pg_cache;
1381
+ struct pg_cache_page_index *page_index;
1382
+ for (Index = 0, PValue = JudyLFirst(metrics_JudyL_array, &Index, PJE0),
1383
+ metric_info = unlikely(NULL == PValue) ? NULL : *PValue;
1384
+ metric_info != NULL;
1385
+ PValue = JudyLNext(metrics_JudyL_array, &Index, PJE0),
1386
+ metric_info = unlikely(NULL == PValue) ? NULL : *PValue) {
1387
+
1388
+ fatal_assert(Index < number_of_metrics);
1389
+ uuid_list[Index].metric_info = metric_info;
1390
+
1391
+ if (false == startup)
1392
+ uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
1393
+
1394
+ PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, metric_info->id, sizeof(uuid_t));
1395
+ page_index = (NULL == PValue) ? NULL : *PValue;
1396
+
1397
+ if (false == startup)
1398
+ uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
1399
+
1400
+ fatal_assert(NULL != page_index);
1401
+
1402
+ metric_info->page_index = page_index;
1403
+
1404
+ }
1405
+ // Cleanup judy arrays we no longer need
1406
+ JudyLFreeArray(&metrics_JudyL_array, PJE0);
1407
+ JudyHSFreeArray(&metrics_JudyHS_array, PJE0);
1408
+
1409
+ qsort(&uuid_list[0], number_of_metrics, sizeof(struct journal_metric_list_to_sort), journal_metric_compare);
1410
+ internal_error(true, "Traverse and qsort UUID %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1411
+ // Write sorted UUID LIST
1412
+ // The loop will write a single UUID entry then
1413
+ // Write all entries (descr @ time) for that UUID at the proper location (header, number of entries, trailer)
1414
+ // Move on to write the next UUID
1415
+ // Write trailer after the UUID list
1416
+ for (Index = 0; Index < number_of_metrics; Index++) {
1417
+ metric_info = uuid_list[Index].metric_info;
1418
+
1419
+ // Calculate current UUID offset from start of file. We will store this in the data page header
1420
+ uint32_t uuid_offset = data - data_start;
1421
+
1422
+ // Write the UUID we are processing
1423
+ data = (void *) journal_v2_write_metric_page(&j2_header, data, metric_info, pages_offset);
1424
+ if (unlikely(!data))
1425
+ break;
1426
+
1427
+ // Next we will write
1428
+ // Header
1429
+ // Detailed entries (descr @ time)
1430
+ // Trailer (checksum)
1431
+
1432
+ // Keep the page_list_header, to be used for migration when where agent is running
1433
+ metric_info->page_list_header = pages_offset;
1434
+ // Write page header
1435
+ void *metric_page = journal_v2_write_data_page_header(&j2_header, data_start + pages_offset, metric_info, uuid_offset);
1436
+
1437
+ // Start writing descr @ time
1438
+ void *page_trailer = journal_v2_write_descriptors(&j2_header, metric_page, metric_info, journalfile);
1439
+ if (unlikely(!page_trailer))
1440
+ break;
1441
+
1442
+ // Trailer (checksum)
1443
+ uint8_t *next_page_address = journal_v2_write_data_page_trailer(&j2_header, page_trailer, data_start + pages_offset);
1444
+
1445
+ // Calculate start of the pages start for next descriptor
1446
+ pages_offset += (metric_info->entries * (sizeof(struct journal_page_list)) + sizeof(struct journal_page_header) + sizeof(struct journal_v2_block_trailer));
1447
+ // Verify we are at the right location
1448
+ fatal_assert(pages_offset == (next_page_address - data_start));
1449
+ }
1450
+ // Data should be at the UUID trailer offset
1451
+ fatal_assert(data == data_start + metric_offset_trailer);
1452
+
1453
+ internal_error(true, "WRITE METRICS AND PAGES %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1454
+
1455
+ // Calculate CRC for metrics
1456
+ journal_v2_trailer = (struct journal_v2_block_trailer *) (data_start + metric_offset_trailer);
1457
+ crc = crc32(0L, Z_NULL, 0);
1458
+ crc = crc32(crc, (uint8_t *) data_start + metrics_offset, number_of_metrics * sizeof(struct journal_metric_list));
1459
+ crc32set(journal_v2_trailer->checksum, crc);
1460
+ internal_error(true, "CALCULATE CRC FOR UUIDs %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1461
+
1462
+ // Prepare to write checksum for the file
1463
+ j2_header.data = NULL;
1464
+ journal_v2_trailer = (struct journal_v2_block_trailer *) (data_start + trailer_offset);
1465
+ crc = crc32(0L, Z_NULL, 0);
1466
+ crc = crc32(crc, (void *) &j2_header, sizeof(j2_header));
1467
+ crc32set(journal_v2_trailer->checksum, crc);
1468
+
1469
+ // Write header to the file
1470
+ memcpy(data_start, &j2_header, sizeof(j2_header));
1471
+
1472
+ internal_error(true, "FILE COMPLETED --------> %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1473
+
1474
+ info("Migrated journal file %s, File size %lu", path, total_file_size);
1475
+
1476
+ if (activate) {
1477
+ journalfile->journal_data = data_start;
1478
+ journalfile->journal_data_size = total_file_size;
1479
+
1480
+ // HERE we need to remove old descriptors and activate the new ones
1481
+ {
1482
+ for (Index = 0; Index < number_of_metrics; Index++) {
1483
+ journal_v2_remove_active_descriptors(journalfile, uuid_list[Index].metric_info, startup);
1484
+
1485
+ if (false == startup)
1486
+ JudyLFreeArray(&uuid_list[Index].metric_info->JudyL_array, PJE0);
1487
+
1488
+ freez(uuid_list[Index].metric_info);
1489
+ }
1490
+ internal_error(true, "ACTIVATING NEW INDEX JNL %llu", (now_realtime_usec() - start_loading) / USEC_PER_MS);
1491
+
1492
+ if (false == startup) {
1493
+ uv_rwlock_wrlock(&journalfile->datafile->extent_rwlock);
1494
+ df_extent_delete_all_unsafe(journalfile->datafile);
1495
+ uv_rwlock_wrunlock(&journalfile->datafile->extent_rwlock);
1496
+ }
1497
+ else
1498
+ df_extent_delete_all_unsafe(journalfile->datafile);
1499
+ }
1500
+
1501
+ ctx->disk_space += total_file_size;
1502
+ }
1503
+ else {
1504
+ // If we failed and didnt process the entire list, free the rest
1505
+ for (Index = 0; Index < number_of_metrics; Index++)
1506
+ freez(uuid_list[Index].metric_info);
1507
+ netdata_munmap(data_start, total_file_size);
1508
+ }
1509
+ freez(uuid_list);
1510
+}
1511
+
1512
int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfile *journalfile,
1513
struct rrdengine_datafile *datafile)
1514
{
@@ -532,15 +1517,24 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
1517
int ret, fd, error;
1518
uint64_t file_size, max_id;
1519
char path[RRDENG_PATH_MAX];
1520
+ int should_try_migration = 0;
1521
+
1522
+ // Do not try to load the latest file (always rebuild and live migrate)
1523
+ if (datafile->fileno != ctx->last_fileno && db_engine_journal_indexing) {
1524
+ if (!(should_try_migration = load_journal_file_v2(ctx, journalfile, datafile))) {
1525
+ return 0;
1526
+ }
1527
+ }
1528
1529
generate_journalfilepath(datafile, path, sizeof(path));
537
- fd = open_file_direct_io(path, O_RDWR, &file);
1530
+
1531
+ // If it is not the last file, open read only
1532
+ fd = open_file_direct_io(path, datafile->fileno == ctx->last_fileno ? O_RDWR : O_RDONLY, &file);
1533
if (fd < 0) {
1534
++ctx->stats.fs_errors;
1535
rrd_stat_atomic_add(&global_fs_errors, 1);
1536
return fd;
1537
}
543
- info("Loading journal file \"%s\".", path);
1538
1539
ret = check_file_properties(file, &file_size, sizeof(struct rrdeng_df_sb));
1540
if (ret)
@@ -548,14 +1542,17 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
1542
file_size = ALIGN_BYTES_FLOOR(file_size);
1543
1544
ret = check_journal_file_superblock(file);
551
- if (ret)
1545
+ if (ret) {
1546
+ info("Invalid journal file \"%s\" ; superblock check failed.", path);
1547
goto error;
1548
+ }
1549
ctx->stats.io_read_bytes += sizeof(struct rrdeng_jf_sb);
1550
++ctx->stats.io_read_requests;
1551
1552
journalfile->file = file;
1553
journalfile->pos = file_size;
558
- journalfile->data = netdata_mmap(path, file_size, MAP_SHARED, 0);
1554
+
1555
+ journalfile->data = netdata_mmap(path, file_size, MAP_SHARED, 0, !(datafile->fileno == ctx->last_fileno));
1556
info("Loading journal file \"%s\" using %s.", path, journalfile->data?"MMAP":"uv_fs_read");
1557
1558
max_id = iterate_transactions(ctx, journalfile);
@@ -565,6 +1562,16 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
1562
info("Journal file \"%s\" loaded (size:%"PRIu64").", path, file_size);
1563
if (likely(journalfile->data))
1564
netdata_munmap(journalfile->data, file_size);
1565
+
1566
+ // Don't Index the last file
1567
+ if (ctx->last_fileno == journalfile->datafile->fileno || !db_engine_journal_indexing)
1568
+ return 0;
1569
+
1570
+ if (should_try_migration == 1)
1571
+ migrate_journal_file_v2(datafile, true, true);
1572
+ else
1573
+ error_report("File %s cannot be migrated to the new journal format. Index will be allocated in memory", path);
1574
+
1575
return 0;
1576
1577
error:
@@ -579,6 +1586,57 @@ int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfi
1586
return error;
1587
}
1588
1589
+void after_journal_indexing(uv_work_t *req, int status)
1590
+{
1591
+ struct rrdeng_work *work_request = req->data;
1592
+ struct rrdengine_worker_config *wc = work_request->wc;
1593
+
1594
+ if (likely(status != UV_ECANCELED)) {
1595
+ errno = 0;
1596
+ if (likely(work_request->count))
1597
+ internal_error(true, "Journal indexing done; %u files processed", work_request->count);
1598
+ }
1599
+ wc->running_journal_migration = 0;
1600
+ wc->run_indexing= work_request->rerun;
1601
+ freez(work_request);
1602
+}
1603
+
1604
+#define MAX_RETRIES_TO_START_INDEX (100)
1605
+void start_journal_indexing(uv_work_t *req)
1606
+{
1607
+ struct rrdeng_work *work_request = req->data;
1608
+ struct rrdengine_worker_config *wc = work_request->wc;
1609
+ struct rrdengine_instance *ctx = wc->ctx;
1610
+
1611
+ unsigned count = 0;
1612
+ while ((wc->now_deleting_files || wc->now_deleting_descriptors) && count++ < MAX_RETRIES_TO_START_INDEX)
1613
+ sleep_usec(100 * USEC_PER_MS);
1614
+
1615
+ if (count == MAX_RETRIES_TO_START_INDEX)
1616
+ return;
1617
+
1618
+ struct rrdengine_datafile *datafile = ctx->datafiles.first;
1619
+
1620
+ while (datafile && datafile->fileno != ctx->last_fileno) {
1621
+ if (unlikely(!datafile->journalfile->journal_data)) {
1622
+ bool ready_to_index = journalfile_ready_to_index(datafile);
1623
+ if (ready_to_index) {
1624
+ info("Journal file %u is ready to be indexed", datafile->fileno);
1625
+ migrate_journal_file_v2(datafile, true, false);
1626
+ ++work_request->count;
1627
+ }
1628
+ else {
1629
+ info("Journal file %u is not ready to be indexed", datafile->fileno);
1630
+ work_request->rerun = true;
1631
+ sleep_usec(100 * USEC_PER_MS);
1632
+ }
1633
+ }
1634
+ datafile = datafile->next;
1635
+ if (unlikely(NO_QUIESCE != ctx->quiesce))
1636
+ break;
1637
+ }
1638
+}
1639
+
1640
void init_commit_log(struct rrdengine_instance *ctx)
1641
{
1642
ctx->commit_log.buf = NULL;
database/engine/journalfile.h
+86
-2
@@ -13,16 +13,98 @@ struct rrdengine_journalfile;
13
14
#define WALFILE_PREFIX "journalfile-"
15
#define WALFILE_EXTENSION ".njf"
16
+#define WALFILE_EXTENSION_V2 ".njfv2"
17
18
+#define is_descr_journal_v2(descr) ((descr)->extent_entry != NULL)
19
20
/* only one event loop is supported for now */
21
struct rrdengine_journalfile {
22
uv_file file;
23
uint64_t pos;
24
void *data;
25
+ uint16_t file_index; // File index
26
+ void *journal_data; // MMAPed file of journal v2
27
+ uint32_t journal_data_size; // Total file size mapped
28
+ Pvoid_t JudyL_array; // Used to track active v2 descriptors
29
+ time_t last_access; // Last access for v2 descriptors to decide cleanup
30
struct rrdengine_datafile *datafile;
31
};
32
33
+
34
+// Journal v2 structures
35
+
36
+#define JOURVAL_V2_MAGIC (0x01221019)
37
+#define JOURVAL_V2_REBUILD_MAGIC (0x00221019)
38
+
39
+struct journal_v2_block_trailer {
40
+ union {
41
+ uint8_t checksum[CHECKSUM_SZ]; /* CRC32 */
42
+ uint32_t crc;
43
+ };
44
+};
45
+
46
+// Journal V2
47
+// 28 bytes
48
+struct journal_page_header {
49
+ union {
50
+ uint8_t checksum[4]; // CRC check
51
+ uint32_t crc;
52
+ };
53
+ uint32_t uuid_offset; // Points back to the UUID list which should point here (UUIDs should much)
54
+ uint32_t entries; // Entries
55
+ uuid_t uuid; // Which UUID this is
56
+};
57
+
58
+// 20 bytes
59
+struct journal_page_list {
60
+ uint32_t delta_start_s; // relative to the start time of journal
61
+ uint32_t delta_end_s; // relative to delta_start
62
+ uint32_t extent_index; // Index to the extent (extent list) (bytes from BASE)
63
+ uint16_t update_every_s;
64
+ uint16_t page_length;
65
+ uint8_t type;
66
+};
67
+
68
+// UUID_LIST
69
+// 32 bytes
70
+struct journal_metric_list {
71
+ uuid_t uuid;
72
+ uint32_t entries; // Number of entries
73
+ uint32_t page_offset; // OFFSET that contains entries * struct( journal_page_list )
74
+ uint32_t delta_start; // Min time of metric
75
+ uint32_t delta_end; // Max time of metric (to be used to populate page_index)
76
+};
77
+
78
+// 16 bytes
79
+struct journal_extent_list {
80
+ uint64_t datafile_offset; // Datafile offset to find the extent
81
+ uint32_t datafile_size; // Size of the extent
82
+ uint16_t file_index; // which file index is this datafile[index]
83
+ uint8_t pages; // number of pages (not all are necesssarily valid)
84
+};
85
+
86
+// 72 bytes
87
+struct journal_v2_header {
88
+ uint32_t magic;
89
+ usec_t start_time_ut; // Min start time of journal
90
+ usec_t end_time_ut; // Maximum end time of journal
91
+ uint32_t extent_count; // Count of extents
92
+ uint32_t extent_offset;
93
+ uint32_t metric_count; // Count of metrics (unique UUIDS)
94
+ uint32_t metric_offset;
95
+ uint32_t page_count; // Total count of pages (descriptors @ time)
96
+ uint32_t page_offset;
97
+ uint32_t extent_trailer_offset; // CRC for entent list
98
+ uint32_t metric_trailer_offset; // CRC for metric list
99
+ uint32_t original_file_size; // This is the original journal file
100
+ uint32_t total_file_size; // This is the total file size
101
+ void *data; // Used when building the index
102
+};
103
+
104
+#define JOURNAL_V2_HEADER_PADDING_SZ (RRDENG_BLOCK_SIZE - (sizeof(struct journal_v2_header)))
105
+
106
+
107
+
108
/* only one event loop is supported for now */
109
struct transaction_commit_log {
110
uint64_t transaction_id;
@@ -34,16 +116,18 @@ struct transaction_commit_log {
116
};
117
118
void generate_journalfilepath(struct rrdengine_datafile *datafile, char *str, size_t maxlen);
119
+void generate_journalfilepath_v2(struct rrdengine_datafile *datafile, char *str, size_t maxlen);
120
void journalfile_init(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
121
void *wal_get_transaction_buffer(struct rrdengine_worker_config* wc, unsigned size);
122
void wal_flush_transaction_buffer(struct rrdengine_worker_config* wc);
123
int close_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
124
int unlink_journal_file(struct rrdengine_journalfile *journalfile);
42
-int destroy_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
125
+int destroy_journal_file_unsafe(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
126
int create_journal_file(struct rrdengine_journalfile *journalfile, struct rrdengine_datafile *datafile);
127
int load_journal_file(struct rrdengine_instance *ctx, struct rrdengine_journalfile *journalfile,
128
struct rrdengine_datafile *datafile);
129
void init_commit_log(struct rrdengine_instance *ctx);
47
-
130
+bool unlink_descriptor_extent_unsafe(struct rrdeng_page_descr *descr);
131
+void queue_journalfile_v2_migration(struct rrdengine_worker_config *wc);
132
133
#endif /* NETDATA_JOURNALFILE_H */
\ No newline at end of file
database/engine/pagecache.c
+402
-167
@@ -134,6 +134,9 @@ struct rrdeng_page_descr *pg_cache_create_descr(void)
134
descr->pg_cache_descr_state = 0;
135
descr->pg_cache_descr = NULL;
136
descr->update_every_s = 0;
137
+ descr->extent_entry = NULL;
138
+ descr->type = 0;
139
+ descr->file = -1;
140
141
return descr;
142
}
@@ -158,13 +161,41 @@ void pg_cache_wake_up_waiters(struct rrdengine_instance *ctx, struct rrdeng_page
161
* The lock will be released and re-acquired. The descriptor is not guaranteed
162
* to exist after this function returns.
163
*/
164
+#ifdef NETDATA_INTERNAL_CHECKS
165
+void pg_cache_wait_event_unsafe_with_trace(struct rrdeng_page_descr *descr, const char *function, size_t line)
166
+#else
167
void pg_cache_wait_event_unsafe(struct rrdeng_page_descr *descr)
168
+#endif
169
{
170
struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
171
172
+#ifdef NETDATA_INTERNAL_CHECKS
173
+ if(pg_cache_descr->owner.tid != gettid())
174
+ fatal("DBENGINE: pg_cache_descr is not locked by me in %s(). It is locked by thread %u, I am %u",
175
+ __FUNCTION__, (unsigned)pg_cache_descr->owner.tid, (unsigned)gettid());
176
+
177
+ struct pg_cache_waiter w = {
178
+ .line = line,
179
+ .function = function,
180
+ .tid = gettid(),
181
+ .next = NULL,
182
+ .prev = NULL,
183
+ };
184
+
185
+ DOUBLE_LINKED_LIST_PREPEND_UNSAFE(pg_cache_descr->wait_list, &w, prev, next);
186
+#endif
187
+
188
++pg_cache_descr->waiters;
189
uv_cond_wait(&pg_cache_descr->cond, &pg_cache_descr->mutex);
190
--pg_cache_descr->waiters;
191
+
192
+#ifdef NETDATA_INTERNAL_CHECKS
193
+ DOUBLE_LINKED_LIST_REMOVE_UNSAFE(pg_cache_descr->wait_list, &w, prev, next);
194
+
195
+ pg_cache_descr->owner.function = function;
196
+ pg_cache_descr->owner.line = line;
197
+ pg_cache_descr->owner.tid = gettid();
198
+#endif
199
}
200
201
/*
@@ -173,15 +204,43 @@ void pg_cache_wait_event_unsafe(struct rrdeng_page_descr *descr)
204
* to exist after this function returns.
205
* Returns UV_ETIMEDOUT if timeout_sec seconds pass.
206
*/
207
+#ifdef NETDATA_INTERNAL_CHECKS
208
+int pg_cache_timedwait_event_unsafe_with_trace(struct rrdeng_page_descr *descr, uint64_t timeout_sec, const char *function, size_t line)
209
+#else
210
int pg_cache_timedwait_event_unsafe(struct rrdeng_page_descr *descr, uint64_t timeout_sec)
211
+#endif
212
{
213
int ret;
214
struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr;
215
216
+#ifdef NETDATA_INTERNAL_CHECKS
217
+ if(pg_cache_descr->owner.tid != gettid())
218
+ fatal("DBENGINE: pg_cache_descr is not locked by me in %s(). It is locked by thread %u, I am %u",
219
+ __FUNCTION__, (unsigned)pg_cache_descr->owner.tid, (unsigned)gettid());
220
+
221
+ struct pg_cache_waiter w = {
222
+ .line = line,
223
+ .function = function,
224
+ .tid = gettid(),
225
+ .next = NULL,
226
+ .prev = NULL,
227
+ };
228
+
229
+ DOUBLE_LINKED_LIST_PREPEND_UNSAFE(pg_cache_descr->wait_list, &w, prev, next);
230
+#endif
231
+
232
++pg_cache_descr->waiters;
233
ret = uv_cond_timedwait(&pg_cache_descr->cond, &pg_cache_descr->mutex, timeout_sec * NSEC_PER_SEC);
234
--pg_cache_descr->waiters;
235
236
+#ifdef NETDATA_INTERNAL_CHECKS
237
+ DOUBLE_LINKED_LIST_REMOVE_UNSAFE(pg_cache_descr->wait_list, &w, prev, next);
238
+
239
+ pg_cache_descr->owner.function = function;
240
+ pg_cache_descr->owner.line = line;
241
+ pg_cache_descr->owner.tid = gettid();
242
+#endif
243
+
244
return ret;
245
}
246
@@ -295,6 +354,11 @@ unsigned long pg_cache_soft_limit(struct rrdengine_instance *ctx)
354
return ctx->cache_pages_low_watermark + (unsigned long)ctx->metric_API_max_producers;
355
}
356
357
+unsigned long pg_cache_warn_limit(struct rrdengine_instance *ctx)
358
+{
359
+ return ctx->cache_pages_warn_watermark + (unsigned long)ctx->metric_API_max_producers;
360
+}
361
+
362
/*
363
* This function returns the maximum number of dirty pages that are committed to be written to disk allowed in the page
364
* cache.
@@ -322,9 +386,10 @@ static void pg_cache_reserve_pages(struct rrdengine_instance *ctx, unsigned numb
386
if (pg_cache->populated_pages + number >= pg_cache_hard_limit(ctx) + 1)
387
debug(D_RRDENGINE, "==Page cache full. Reserving %u pages.==",
388
number);
389
+
390
while (pg_cache->populated_pages + number >= pg_cache_hard_limit(ctx) + 1) {
391
327
- if (!pg_cache_try_evict_one_page_unsafe(ctx)) {
392
+ if (!(pg_cache_try_evict_one_page_unsafe(ctx))) {
393
/* failed to evict */
394
struct completion compl;
395
struct rrdeng_cmd cmd;
@@ -390,7 +455,6 @@ static int pg_cache_try_reserve_pages(struct rrdengine_instance *ctx, unsigned n
455
ret = 1; /* success */
456
}
457
uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
393
-
458
return ret;
459
}
460
@@ -413,8 +477,8 @@ static void pg_cache_evict_unsafe(struct rrdengine_instance *ctx, struct rrdeng_
477
* If it fails it sets in_flight_descr to the oldest descriptor that has write-back in progress,
478
* or it sets it to NULL if no write-back is in progress.
479
*
416
- * Returns 1 on success and 0 on failure.
417
- */
480
+* Returns 1 on success and 0 on failure.
481
+*/
482
static int pg_cache_try_evict_one_page_unsafe(struct rrdengine_instance *ctx)
483
{
484
struct page_cache *pg_cache = &ctx->pg_cache;
@@ -458,10 +522,16 @@ static int pg_cache_try_evict_one_page_unsafe(struct rrdengine_instance *ctx)
522
* @param is_exclusive_holder must be non-zero if the caller holds an exclusive page reference.
523
* @param metric_id is set to the metric the page belongs to, if it's safe to delete the metric and metric_id is not
524
* NULL. Otherwise, metric_id is not set.
525
+ * @spin True to keep trying to release the page, false to try once
526
* @return 1 if it's safe to delete the metric, 0 otherwise.
527
*/
463
-uint8_t pg_cache_punch_hole(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr, uint8_t remove_dirty,
464
- uint8_t is_exclusive_holder, uuid_t *metric_id)
528
+uint8_t pg_cache_punch_hole(
529
+ struct rrdengine_instance *ctx,
530
+ struct rrdeng_page_descr *descr,
531
+ uint8_t remove_dirty,
532
+ uint8_t is_exclusive_holder,
533
+ uuid_t(*metric_id),
534
+ bool update_page_duration)
535
{
536
struct page_cache *pg_cache = &ctx->pg_cache;
537
struct page_cache_descr *pg_cache_descr = NULL;
@@ -485,7 +555,8 @@ uint8_t pg_cache_punch_hole(struct rrdengine_instance *ctx, struct rrdeng_page_d
555
}
556
goto destroy;
557
}
488
- --page_index->page_count;
558
+ if (update_page_duration)
559
+ --page_index->page_count;
560
if (!page_index->writers && !page_index->page_count) {
561
can_delete_metric = 1;
562
if (metric_id) {
@@ -497,7 +568,11 @@ uint8_t pg_cache_punch_hole(struct rrdengine_instance *ctx, struct rrdeng_page_d
568
569
uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
570
++ctx->stats.pg_cache_deletions;
500
- --pg_cache->page_descriptors;
571
+ if (update_page_duration)
572
+ --pg_cache->page_descriptors;
573
+
574
+ if (is_descr_journal_v2(descr))
575
+ --pg_cache->active_descriptors;
576
uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
577
578
rrdeng_page_descr_mutex_lock(ctx, descr);
@@ -512,7 +587,7 @@ uint8_t pg_cache_punch_hole(struct rrdengine_instance *ctx, struct rrdeng_page_d
587
}
588
}
589
if (remove_dirty) {
515
- pg_cache_descr->flags &= ~RRD_PAGE_DIRTY;
590
+ pg_cache_descr->flags &= ~(RRD_PAGE_DIRTY | RRD_PAGE_INVALID);
591
} else {
592
/* even a locked page could be dirty */
593
while (unlikely(pg_cache_descr->flags & RRD_PAGE_DIRTY)) {
@@ -540,7 +615,8 @@ uint8_t pg_cache_punch_hole(struct rrdengine_instance *ctx, struct rrdeng_page_d
615
}
616
destroy:
617
rrdeng_page_descr_freez(descr);
543
- pg_cache_update_metric_times(page_index);
618
+ if (update_page_duration)
619
+ pg_cache_update_metric_times(page_index);
620
621
return can_delete_metric;
622
}
@@ -556,33 +632,272 @@ static inline int is_page_in_time_range(struct rrdeng_page_descr *descr, usec_t
632
(pg_start >= start_time && pg_start <= end_time);
633
}
634
559
-static inline int is_point_in_time_in_page(struct rrdeng_page_descr *descr, usec_t point_in_time)
635
+static uint32_t find_matching_page(struct journal_page_header *page_list_header, uint32_t delta_start_time_s)
636
{
561
- return (point_in_time >= descr->start_time_ut && point_in_time <= descr->end_time_ut);
637
+ uint32_t left = 0;
638
+ uint32_t right = page_list_header->entries;
639
+
640
+ while (left < right) {
641
+ struct journal_page_list *page_list = (struct journal_page_list *) ((uint8_t *) page_list_header + sizeof(*page_list_header));
642
+ struct journal_page_list *page_entry;
643
+
644
+ uint32_t middle_delta_start_s;
645
+ uint32_t middle_delta_end_s;
646
+
647
+ uint32_t middle = (left + right) >> 1;
648
+
649
+ page_entry = &page_list[middle];
650
+ middle_delta_start_s = page_entry->delta_start_s;
651
+ middle_delta_end_s = page_entry->delta_end_s;
652
+
653
+ if (delta_start_time_s >= middle_delta_start_s && delta_start_time_s <= middle_delta_end_s)
654
+ return middle;
655
+
656
+ if (delta_start_time_s < middle_delta_end_s)
657
+ right = middle;
658
+ else if(delta_start_time_s > middle_delta_end_s)
659
+ left = middle + 1;
660
+ else
661
+ return middle;
662
+ }
663
+ return right;
664
}
665
564
-/* The caller must hold the page index lock */
565
-static inline struct rrdeng_page_descr *
566
- find_first_page_in_time_range(struct pg_cache_page_index *page_index, usec_t start_time, usec_t end_time)
666
+bool descr_exists_unsafe( struct pg_cache_page_index *page_index, time_t start_time_s)
667
{
668
+ return (NULL != JudyLGet(page_index->JudyL_array, start_time_s, PJE0));
669
+}
670
+
671
+void mark_journalfile_descriptor( struct page_cache *pg_cache, struct rrdengine_journalfile *journalfile, uint32_t page_offset, uint32_t Index)
672
+{
673
+ Pvoid_t *PValue;
674
+
675
+ uv_rwlock_wrlock(&pg_cache->v2_lock);
676
+ PValue = JudyLIns(&journalfile->JudyL_array, (Word_t)page_offset, PJE0);
677
+ *(uint32_t *)PValue = (Index + 1);
678
+ journalfile->last_access = now_realtime_sec();
679
+ uv_rwlock_wrunlock(&pg_cache->v2_lock);
680
+}
681
+
682
+static void update_journal_access_time(struct rrdengine_journalfile *journalfile, struct pg_cache_page_index *page_index, struct rrdeng_page_descr *descr)
683
+{
684
+ if (journalfile) {
685
+ journalfile->last_access = now_realtime_sec();
686
+ return;
687
+ }
688
+
689
+ if (unlikely(!page_index || !descr))
690
+ return;
691
+
692
+ if (!is_descr_journal_v2(descr))
693
+ return;
694
+
695
+ struct rrdengine_instance *ctx = page_index->ctx;
696
+
697
+ uv_rwlock_rdlock(&ctx->datafiles.rwlock);
698
+ struct rrdengine_datafile *datafile = ctx->datafiles.first;
699
+ while (datafile) {
700
+ journalfile = datafile->journalfile;
701
+ if (!journalfile->journal_data) {
702
+ datafile = datafile->next;
703
+ continue;
704
+ }
705
+ if (datafile->file == descr->file) {
706
+ journalfile->last_access = now_realtime_sec();
707
+ break;
708
+ }
709
+ datafile = datafile->next;
710
+ }
711
+ uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
712
+}
713
+
714
+// Note: We have read lock on page index
715
+// We release and escalate to write lock
716
+// Return to read lock when done
717
+static struct rrdeng_page_descr *add_pages_from_timerange(
718
+ struct journal_page_header *page_list_header,
719
+ uint32_t delta_start_time_s,
720
+ uint32_t delta_end_time_s,
721
+ usec_t journal_start_time_ut,
722
+ struct pg_cache_page_index *page_index,
723
+ struct journal_extent_list *extent_list,
724
+ struct rrdengine_datafile *datafile,
725
+ uint32_t cache_pages)
726
+{
727
+
728
+ struct rrdengine_instance *ctx = page_index->ctx;
729
+ struct page_cache *pg_cache = &ctx->pg_cache;
730
+ time_t journal_start_time_s = (time_t)(journal_start_time_ut / USEC_PER_SEC);
731
+ struct journal_page_list *page_list = (struct journal_page_list *)((uint8_t *) page_list_header + sizeof(*page_list_header));
732
+
733
+ uint32_t pos = find_matching_page(page_list_header, delta_start_time_s);
734
+
735
+ // This is the page offset that we will store to check for v2 descriptors later on
736
+ uint32_t page_offset = (uint8_t *) page_list_header - (uint8_t *) datafile->journalfile->journal_data;
737
+ uint32_t entries = page_list_header->entries;
738
+ uint32_t pages_to_cache = MIN(pos + cache_pages, entries);
739
+
740
struct rrdeng_page_descr *descr = NULL;
741
+
742
+ bool journal_updated = false;
743
+ bool rw_lock_acquired = false;
744
+
745
+ // We will cache pages_to_cache pages or until our end time is out of range
746
+ for (uint32_t x = pos; x < pages_to_cache; x++) {
747
+
748
+ struct journal_page_list *page_entry = &page_list[x];
749
+
750
+ if (page_entry->extent_index == UINT32_MAX)
751
+ continue;
752
+
753
+ if (delta_end_time_s < page_entry->delta_start_s)
754
+ break;
755
+
756
+ time_t index_time_s = (time_t) (journal_start_time_s + page_entry->delta_start_s);
757
+
758
+ if (!descr_exists_unsafe(page_index, index_time_s)) {
759
+ struct rrdeng_page_descr *new_descr = pg_cache_create_descr();
760
+ new_descr->page_length = page_entry->page_length;
761
+ new_descr->start_time_ut = index_time_s * USEC_PER_SEC;
762
+ new_descr->end_time_ut = (journal_start_time_s + page_entry->delta_end_s) * USEC_PER_SEC;
763
+ new_descr->id = &page_index->id;
764
+ new_descr->extent = NULL;
765
+ new_descr->extent_entry = &extent_list[page_entry->extent_index];
766
+ new_descr->type = page_entry->type;
767
+ new_descr->update_every_s = page_entry->update_every_s;
768
+ new_descr->file = datafile->file;
769
+
770
+ if (false == rw_lock_acquired) {
771
+ uv_rwlock_rdunlock(&page_index->lock);
772
+ uv_rwlock_wrlock(&page_index->lock);
773
+ rw_lock_acquired = true;
774
+ }
775
+
776
+ struct rrdeng_page_descr *added_descr = pg_cache_insert(ctx, page_index, new_descr, false);
777
+ if (unlikely(added_descr != new_descr))
778
+ rrdeng_page_descr_freez(new_descr);
779
+
780
+ if (!descr) {
781
+ descr = added_descr;
782
+ // Mark the area to check
783
+ mark_journalfile_descriptor(pg_cache, datafile->journalfile, page_offset, x);
784
+ journal_updated = true;
785
+ }
786
+ }
787
+ }
788
+
789
+ if (!journal_updated)
790
+ update_journal_access_time(datafile->journalfile, NULL, NULL);
791
+
792
+ // Check if we have switched to rw lock for the page index and switch back
793
+ if (rw_lock_acquired) {
794
+ uv_rwlock_wrunlock(&page_index->lock);
795
+ uv_rwlock_rdlock(&page_index->lock);
796
+ }
797
+ return descr;
798
+};
799
+
800
+static int journal_metric_uuid_compare(const void *key, const void *metric)
801
+{
802
+ return uuid_compare(*(uuid_t *) key, ((struct journal_metric_list *) metric)->uuid);
803
+}
804
+
805
+//
806
+// Steps
807
+// 1. Find which journal has the start time within its range
808
+// 2. Find the UUID in that journal
809
+// 3. Find the array of times for that UUID (convert from the journal header to the offset needed)
810
+// Note: We have page_index lock
811
+// cache pages is the maximum pages to fetch (create metadata for)
812
+// This will be limited by the end_time or if we run out of pages in the matching journal
813
+// pages that could be precached but exist in another journal will not be precached
814
+static struct rrdeng_page_descr *populate_page_index(
815
+ struct pg_cache_page_index *page_index,
816
+ usec_t start_time_ut,
817
+ usec_t end_time_ut,
818
+ uint32_t cache_pages)
819
+{
820
+ struct rrdengine_instance *ctx = page_index->ctx;
821
+
822
+ uv_rwlock_rdlock(&ctx->datafiles.rwlock);
823
+
824
+ struct rrdengine_datafile *datafile = ctx->datafiles.first;
825
+ while (datafile) {
826
+ struct journal_v2_header *journal_header = (struct journal_v2_header *) datafile->journalfile->journal_data;
827
+ if (!journal_header) {
828
+ datafile = datafile->next;
829
+ continue;
830
+ }
831
+ if (start_time_ut >= journal_header->start_time_ut && start_time_ut <= journal_header->end_time_ut) {
832
+
833
+ struct journal_metric_list *uuid_list = (struct journal_metric_list *)((uint8_t *) journal_header + journal_header->metric_offset);
834
+
835
+ struct journal_metric_list *uuid_entry = bsearch(
836
+ &page_index->id,
837
+ uuid_list,
838
+ (size_t)journal_header->metric_count,
839
+ sizeof(struct journal_metric_list),
840
+ journal_metric_uuid_compare);
841
+
842
+ uint32_t delta_start_time = (start_time_ut - journal_header->start_time_ut) / USEC_PER_SEC;
843
+ uint32_t delta_end_time = (end_time_ut - journal_header->start_time_ut) / USEC_PER_SEC;
844
+
845
+ if (uuid_entry && ((delta_start_time >= uuid_entry->delta_start && delta_start_time <= uuid_entry->delta_end))) {
846
+
847
+ struct journal_page_header *page_list_header = (struct journal_page_header *) ((uint8_t *) journal_header + uuid_entry->page_offset);
848
+ struct rrdeng_page_descr *descr = add_pages_from_timerange(
849
+ page_list_header,
850
+ delta_start_time,
851
+ delta_end_time,
852
+ journal_header->start_time_ut,
853
+ page_index,
854
+ (void *)((uint8_t *)journal_header + journal_header->extent_offset),
855
+ datafile,
856
+ cache_pages);
857
+
858
+ uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
859
+ return descr;
860
+ }
861
+ }
862
+ datafile = datafile->next;
863
+ }
864
+
865
+ uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
866
+ return NULL;
867
+}
868
+
869
+/* The caller must hold the page index lock */
870
+static inline struct rrdeng_page_descr *find_first_page_in_time_range(
871
+ struct pg_cache_page_index *page_index,
872
+ usec_t start_time_ut,
873
+ usec_t end_time_ut,
874
+ uint32_t cache_pages)
875
+{
876
+ struct rrdeng_page_descr *descr= NULL;
877
+
878
Pvoid_t *PValue;
879
Word_t Index;
880
572
- Index = (Word_t)(start_time / USEC_PER_SEC);
881
+ Index = (Word_t) (start_time_ut / USEC_PER_SEC);
882
PValue = JudyLLast(page_index->JudyL_array, &Index, PJE0);
883
if (likely(NULL != PValue)) {
884
descr = *PValue;
576
- if (is_page_in_time_range(descr, start_time, end_time)) {
885
+ if (is_page_in_time_range(descr, start_time_ut, end_time_ut)) {
886
+ update_journal_access_time(NULL, page_index, descr);
887
return descr;
888
}
889
}
890
581
- Index = (Word_t)(start_time / USEC_PER_SEC);
891
+ descr = populate_page_index(page_index, start_time_ut, end_time_ut, cache_pages);
892
+ if (descr)
893
+ return descr;
894
+
895
+ Index = (Word_t) (start_time_ut / USEC_PER_SEC);
896
PValue = JudyLFirst(page_index->JudyL_array, &Index, PJE0);
897
if (likely(NULL != PValue)) {
898
descr = *PValue;
585
- if (is_page_in_time_range(descr, start_time, end_time)) {
899
+ if (is_page_in_time_range(descr, start_time_ut, end_time_ut)) {
900
+ update_journal_access_time(NULL, page_index, descr);
901
return descr;
902
}
903
}
@@ -638,9 +953,13 @@ void pg_cache_update_metric_times(struct pg_cache_page_index *page_index)
953
page_index->latest_time_ut = latest_time;
954
}
955
956
+
957
/* If index is NULL lookup by UUID (descr->id) */
642
-void pg_cache_insert(struct rrdengine_instance *ctx, struct pg_cache_page_index *index,
643
- struct rrdeng_page_descr *descr)
958
+struct rrdeng_page_descr *pg_cache_insert(
959
+ struct rrdengine_instance *ctx,
960
+ struct pg_cache_page_index *index,
961
+ struct rrdeng_page_descr *descr,
962
+ bool lock_and_count)
963
{
964
struct page_cache *pg_cache = &ctx->pg_cache;
965
Pvoid_t *PValue;
@@ -669,44 +988,34 @@ void pg_cache_insert(struct rrdengine_instance *ctx, struct pg_cache_page_index
988
page_index = index;
989
}
990
672
- uv_rwlock_wrlock(&page_index->lock);
991
+ if (lock_and_count)
992
+ uv_rwlock_wrlock(&page_index->lock);
993
+
994
PValue = JudyLIns(&page_index->JudyL_array, (Word_t)(descr->start_time_ut / USEC_PER_SEC), PJE0);
995
+ fatal_assert(NULL != PValue);
996
+
997
+ if (unlikely(*PValue) && !lock_and_count)
998
+ return *PValue;
999
+
1000
*PValue = descr;
675
- ++page_index->page_count;
676
- pg_cache_add_new_metric_time(page_index, descr);
677
- uv_rwlock_wrunlock(&page_index->lock);
1001
+ if (lock_and_count)
1002
+ ++page_index->page_count;
1003
679
- uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
680
- ++ctx->stats.pg_cache_insertions;
681
- ++pg_cache->page_descriptors;
682
- uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
683
-}
1004
+ pg_cache_add_new_metric_time(page_index, descr);
1005
685
-usec_t pg_cache_oldest_time_in_range(struct rrdengine_instance *ctx, uuid_t *id, usec_t start_time_ut, usec_t end_time_ut)
686
-{
687
- struct page_cache *pg_cache = &ctx->pg_cache;
688
- struct rrdeng_page_descr *descr = NULL;
689
- Pvoid_t *PValue;
690
- struct pg_cache_page_index *page_index = NULL;
1006
+ if (lock_and_count)
1007
+ uv_rwlock_wrunlock(&page_index->lock);
1008
692
- uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
693
- PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, id, sizeof(uuid_t));
694
- if (likely(NULL != PValue)) {
695
- page_index = *PValue;
696
- }
697
- uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
698
- if (NULL == PValue) {
699
- return INVALID_TIME;
1009
+ uv_rwlock_wrlock(&pg_cache->pg_cache_rwlock);
1010
+ if (lock_and_count) {
1011
+ ++ctx->stats.pg_cache_insertions;
1012
+ ++pg_cache->page_descriptors;
1013
}
1014
+ if (is_descr_journal_v2(descr))
1015
+ ++pg_cache->active_descriptors;
1016
+ uv_rwlock_wrunlock(&pg_cache->pg_cache_rwlock);
1017
702
- uv_rwlock_rdlock(&page_index->lock);
703
- descr = find_first_page_in_time_range(page_index, start_time_ut, end_time_ut);
704
- if (NULL == descr) {
705
- uv_rwlock_rdunlock(&page_index->lock);
706
- return INVALID_TIME;
707
- }
708
- uv_rwlock_rdunlock(&page_index->lock);
709
- return descr->start_time_ut;
1018
+ return descr;
1019
}
1020
1021
/**
@@ -758,8 +1067,11 @@ void pg_cache_get_filtered_info_prev(struct rrdengine_instance *ctx, struct pg_c
1067
* 2. It did not succeed to get a reference.
1068
* 3. It did not succeed to reserve a spot in the page cache.
1069
*/
761
-struct rrdeng_page_descr *pg_cache_lookup_unpopulated_and_lock(struct rrdengine_instance *ctx, uuid_t *id,
762
- usec_t start_time_ut)
1070
+struct rrdeng_page_descr *pg_cache_lookup_unpopulated_and_lock(
1071
+ struct rrdengine_instance *ctx,
1072
+ uuid_t(*id),
1073
+ usec_t start_time_ut,
1074
+ struct pg_alignment *alignment)
1075
{
1076
struct page_cache *pg_cache = &ctx->pg_cache;
1077
struct rrdeng_page_descr *descr = NULL;
@@ -776,6 +1088,11 @@ struct rrdeng_page_descr *pg_cache_lookup_unpopulated_and_lock(struct rrdengine_
1088
}
1089
uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
1090
1091
+ if (page_index && page_index->alignment && alignment && page_index->alignment != alignment) {
1092
+ if (pg_cache->populated_pages >= pg_cache_warn_limit(ctx))
1093
+ return NULL;
1094
+ }
1095
+
1096
if ((NULL == PValue) || !pg_cache_try_reserve_pages(ctx, 1)) {
1097
/* Failed to find page or failed to reserve a spot in the cache */
1098
return NULL;
@@ -855,7 +1172,7 @@ unsigned pg_cache_preload(struct rrdengine_instance *ctx, uuid_t *id, usec_t sta
1172
}
1173
1174
uv_rwlock_rdlock(&page_index->lock);
858
- descr = find_first_page_in_time_range(page_index, start_time_ut, end_time_ut);
1175
+ descr = find_first_page_in_time_range(page_index, start_time_ut, end_time_ut, PAGE_CACHE_MAX_PRELOAD_PAGES);
1176
if (NULL == descr) {
1177
uv_rwlock_rdunlock(&page_index->lock);
1178
debug(D_RRDENGINE, "%s: No page was found to attempt preload.", __func__);
@@ -869,12 +1186,18 @@ unsigned pg_cache_preload(struct rrdengine_instance *ctx, uuid_t *id, usec_t sta
1186
*page_info_arrayp = mallocz(page_info_array_max_size);
1187
}
1188
1189
+ struct rrdeng_page_descr *last_descr = NULL;
1190
for (count = 0, preload_count = 0 ;
1191
descr != NULL && is_page_in_time_range(descr, start_time_ut, end_time_ut) ;
1192
PValue = JudyLNext(page_index->JudyL_array, &Index, PJE0),
1193
descr = unlikely(NULL == PValue) ? NULL : *PValue) {
1194
/* Iterate all pages in range */
1195
1196
+ if (last_descr == descr)
1197
+ break;
1198
+
1199
+ last_descr = descr;
1200
+
1201
if (unlikely(0 == descr->page_length))
1202
continue;
1203
if (page_info_arrayp) {
@@ -907,7 +1230,6 @@ unsigned pg_cache_preload(struct rrdengine_instance *ctx, uuid_t *id, usec_t sta
1230
}
1231
}
1232
rrdeng_page_descr_mutex_unlock(ctx, descr);
910
-
1233
}
1234
uv_rwlock_rdunlock(&page_index->lock);
1235
@@ -933,7 +1255,8 @@ unsigned pg_cache_preload(struct rrdengine_instance *ctx, uuid_t *id, usec_t sta
1255
if (NULL == next) {
1256
continue;
1257
}
936
- if (descr->extent == next->extent) {
1258
+ if ((descr->extent && descr->extent == next->extent) ||
1259
+ ((descr->extent_entry && descr->extent_entry == next->extent_entry))) {
1260
/* same extent, consolidate */
1261
if (!pg_cache_try_reserve_pages(ctx, 1)) {
1262
failed_to_reserve = 1;
@@ -968,113 +1291,6 @@ unsigned pg_cache_preload(struct rrdengine_instance *ctx, uuid_t *id, usec_t sta
1291
return count;
1292
}
1293
971
-/*
972
- * Searches for a page and gets a reference.
973
- * When point_in_time is INVALID_TIME get any page.
974
- * If index is NULL lookup by UUID (id).
975
- */
976
-struct rrdeng_page_descr *
977
- pg_cache_lookup(struct rrdengine_instance *ctx, struct pg_cache_page_index *index, uuid_t *id,
978
- usec_t point_in_time_ut)
979
-{
980
- struct page_cache *pg_cache = &ctx->pg_cache;
981
- struct rrdeng_page_descr *descr = NULL;
982
- struct page_cache_descr *pg_cache_descr = NULL;
983
- unsigned long flags;
984
- Pvoid_t *PValue;
985
- struct pg_cache_page_index *page_index = NULL;
986
- Word_t Index;
987
- uint8_t page_not_in_cache;
988
-
989
- if (unlikely(NULL == index)) {
990
- uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
991
- PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, id, sizeof(uuid_t));
992
- if (likely(NULL != PValue)) {
993
- page_index = *PValue;
994
- }
995
- uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
996
- if (NULL == PValue) {
997
- return NULL;
998
- }
999
- } else {
1000
- page_index = index;
1001
- }
1002
- pg_cache_reserve_pages(ctx, 1);
1003
-
1004
- page_not_in_cache = 0;
1005
- uv_rwlock_rdlock(&page_index->lock);
1006
- while (1) {
1007
- Index = (Word_t)(point_in_time_ut / USEC_PER_SEC);
1008
- PValue = JudyLLast(page_index->JudyL_array, &Index, PJE0);
1009
- if (likely(NULL != PValue)) {
1010
- descr = *PValue;
1011
- }
1012
- if (NULL == PValue ||
1013
- 0 == descr->page_length ||
1014
- (INVALID_TIME != point_in_time_ut &&
1015
- !is_point_in_time_in_page(descr, point_in_time_ut))) {
1016
- /* non-empty page not found */
1017
- uv_rwlock_rdunlock(&page_index->lock);
1018
-
1019
- pg_cache_release_pages(ctx, 1);
1020
- return NULL;
1021
- }
1022
- rrdeng_page_descr_mutex_lock(ctx, descr);
1023
- pg_cache_descr = descr->pg_cache_descr;
1024
- flags = pg_cache_descr->flags;
1025
- if ((flags & RRD_PAGE_POPULATED) && pg_cache_try_get_unsafe(descr, 0)) {
1026
- /* success */
1027
- rrdeng_page_descr_mutex_unlock(ctx, descr);
1028
- debug(D_RRDENGINE, "%s: Page was found in memory.", __func__);
1029
- break;
1030
- }
1031
- if (!(flags & RRD_PAGE_POPULATED) && pg_cache_try_get_unsafe(descr, 1)) {
1032
- struct rrdeng_cmd cmd;
1033
-
1034
- uv_rwlock_rdunlock(&page_index->lock);
1035
-
1036
- cmd.opcode = RRDENG_READ_PAGE;
1037
- cmd.read_page.page_cache_descr = descr;
1038
- rrdeng_enq_cmd(&ctx->worker_config, &cmd);
1039
-
1040
- debug(D_RRDENGINE, "%s: Waiting for page to be asynchronously read from disk:", __func__);
1041
- if(unlikely(debug_flags & D_RRDENGINE))
1042
- print_page_cache_descr(descr, "", true);
1043
- while (!(pg_cache_descr->flags & RRD_PAGE_POPULATED)) {
1044
- pg_cache_wait_event_unsafe(descr);
1045
- }
1046
- /* success */
1047
- /* Downgrade exclusive reference to allow other readers */
1048
- pg_cache_descr->flags &= ~RRD_PAGE_LOCKED;
1049
- pg_cache_wake_up_waiters_unsafe(descr);
1050
- rrdeng_page_descr_mutex_unlock(ctx, descr);
1051
- rrd_stat_atomic_add(&ctx->stats.pg_cache_misses, 1);
1052
- return descr;
1053
- }
1054
- uv_rwlock_rdunlock(&page_index->lock);
1055
- debug(D_RRDENGINE, "%s: Waiting for page to be unlocked:", __func__);
1056
- if(unlikely(debug_flags & D_RRDENGINE))
1057
- print_page_cache_descr(descr, "", true);
1058
- if (!(flags & RRD_PAGE_POPULATED))
1059
- page_not_in_cache = 1;
1060
- pg_cache_wait_event_unsafe(descr);
1061
- rrdeng_page_descr_mutex_unlock(ctx, descr);
1062
-
1063
- /* reset scan to find again */
1064
- uv_rwlock_rdlock(&page_index->lock);
1065
- }
1066
- uv_rwlock_rdunlock(&page_index->lock);
1067
-
1068
- if (!(flags & RRD_PAGE_DIRTY))
1069
- pg_cache_replaceQ_set_hot(ctx, descr);
1070
- pg_cache_release_pages(ctx, 1);
1071
- if (page_not_in_cache)
1072
- rrd_stat_atomic_add(&ctx->stats.pg_cache_misses, 1);
1073
- else
1074
- rrd_stat_atomic_add(&ctx->stats.pg_cache_hits, 1);
1075
- return descr;
1076
-}
1077
-
1294
/*
1295
* Searches for the first page between start_time and end_time and gets a reference.
1296
* start_time and end_time are inclusive.
@@ -1111,7 +1327,7 @@ pg_cache_lookup_next(struct rrdengine_instance *ctx, struct pg_cache_page_index
1327
uv_rwlock_rdlock(&page_index->lock);
1328
int retry_count = 0;
1329
while (1) {
1114
- descr = find_first_page_in_time_range(page_index, start_time_ut, end_time_ut);
1330
+ descr = find_first_page_in_time_range(page_index, start_time_ut, end_time_ut, PAGE_CACHE_MAX_PRELOAD_PAGES);
1331
if (NULL == descr || 0 == descr->page_length || retry_count == default_rrdeng_page_fetch_retries) {
1332
/* non-empty page not found */
1333
if (retry_count == default_rrdeng_page_fetch_retries)
@@ -1124,6 +1340,22 @@ pg_cache_lookup_next(struct rrdengine_instance *ctx, struct pg_cache_page_index
1340
rrdeng_page_descr_mutex_lock(ctx, descr);
1341
pg_cache_descr = descr->pg_cache_descr;
1342
flags = pg_cache_descr->flags;
1343
+
1344
+ if ((flags & RRD_PAGE_INVALID)) {
1345
+ bool can_drop_page = pg_cache_try_get_unsafe(descr, 1);
1346
+ rrdeng_page_descr_mutex_unlock(ctx, descr);
1347
+
1348
+ uv_rwlock_rdunlock(&page_index->lock);
1349
+ pg_cache_release_pages(ctx, 1);
1350
+
1351
+ if (likely(can_drop_page)) {
1352
+ info("Dropping invalid page descr=%lu - pg_cache=%lu - Ref=%u", descr->pg_cache_descr_state,
1353
+ descr->pg_cache_descr->flags, descr->pg_cache_descr->refcnt);
1354
+ pg_cache_punch_hole(ctx, descr, 0, 1, NULL, false);
1355
+ }
1356
+ return NULL;
1357
+ }
1358
+
1359
if ((flags & RRD_PAGE_POPULATED) && pg_cache_try_get_unsafe(descr, 0)) {
1360
/* success */
1361
rrdeng_page_descr_mutex_unlock(ctx, descr);
@@ -1165,7 +1397,6 @@ pg_cache_lookup_next(struct rrdengine_instance *ctx, struct pg_cache_page_index
1397
++retry_count;
1398
}
1399
rrdeng_page_descr_mutex_unlock(ctx, descr);
1168
-
1400
/* reset scan to find again */
1401
uv_rwlock_rdlock(&page_index->lock);
1402
}
@@ -1186,9 +1417,9 @@ struct pg_cache_page_index *create_page_index(uuid_t *id, struct rrdengine_insta
1417
struct pg_cache_page_index *page_index;
1418
1419
page_index = mallocz(sizeof(*page_index));
1420
+ fatal_assert(0 == uv_rwlock_init(&page_index->lock));
1421
page_index->JudyL_array = (Pvoid_t) NULL;
1422
uuid_copy(page_index->id, *id);
1191
- fatal_assert(0 == uv_rwlock_init(&page_index->lock));
1423
page_index->oldest_time_ut = INVALID_TIME;
1424
page_index->latest_time_ut = INVALID_TIME;
1425
page_index->prev = NULL;
@@ -1235,12 +1466,15 @@ void init_page_cache(struct rrdengine_instance *ctx)
1466
struct page_cache *pg_cache = &ctx->pg_cache;
1467
1468
pg_cache->page_descriptors = 0;
1469
+ pg_cache->active_descriptors = 0;
1470
pg_cache->populated_pages = 0;
1471
fatal_assert(0 == uv_rwlock_init(&pg_cache->pg_cache_rwlock));
1472
1473
init_metrics_index(ctx);
1474
init_replaceQ(ctx);
1475
init_committed_page_index(ctx);
1476
+
1477
+ fatal_assert(0 == uv_rwlock_init(&pg_cache->v2_lock));
1478
}
1479
1480
void free_page_cache(struct rrdengine_instance *ctx)
@@ -1304,3 +1538,4 @@ void free_page_cache(struct rrdengine_instance *ctx)
1538
fatal_assert(NULL == pg_cache->metrics_index.JudyHS_array);
1539
info("Freed %lu bytes of memory from page cache.", pages_dirty_index_bytes + pages_index_bytes + metrics_index_bytes);
1540
}
1541
+
database/engine/pagecache.h
+51
-12
@@ -20,6 +20,18 @@ struct rrdeng_page_descr;
20
#define RRD_PAGE_READ_PENDING (1LU << 2)
21
#define RRD_PAGE_WRITE_PENDING (1LU << 3)
22
#define RRD_PAGE_POPULATED (1LU << 4)
23
+#define RRD_PAGE_INVALID (1LU << 5)
24
+
25
+#ifdef NETDATA_INTERNAL_CHECKS
26
+struct pg_cache_waiter {
27
+ const char *function;
28
+ size_t line;
29
+ pid_t tid;
30
+
31
+ struct pg_cache_waiter *next;
32
+ struct pg_cache_waiter *prev;
33
+};
34
+#endif
35
36
struct page_cache_descr {
37
struct rrdeng_page_descr *descr; /* parent descriptor */
@@ -29,6 +41,12 @@ struct page_cache_descr {
41
struct page_cache_descr *next; /* LRU */
42
43
unsigned refcnt;
44
+
45
+#ifdef NETDATA_INTERNAL_CHECKS
46
+ struct pg_cache_waiter *wait_list;
47
+ struct pg_cache_waiter owner;
48
+#endif
49
+
50
uv_mutex_t mutex; /* always take it after the page cache lock or after the commit lock */
51
uv_cond_t cond;
52
unsigned waiters;
@@ -49,6 +67,7 @@ struct page_cache_descr {
67
* -----------------------------+------------+------------+-----------|
68
* number of descriptor users | DESTROY | LOCKED | ALLOCATED |
69
*/
70
+
71
struct rrdeng_page_descr {
72
uuid_t *id; /* never changes */
73
struct extent_info *extent;
@@ -65,6 +84,8 @@ struct rrdeng_page_descr {
84
uint32_t update_every_s:24;
85
uint8_t type;
86
uint32_t page_length;
87
+ uv_file file; // This is the datafile this descriptor belongs
88
+ void *extent_entry;
89
};
90
91
#define PAGE_INFO_SCRATCH_SZ (8)
@@ -79,7 +100,7 @@ struct rrdeng_page_info {
100
/* returns 1 for success, 0 for failure */
101
typedef int pg_cache_page_info_filter_t(struct rrdeng_page_descr *);
102
82
-#define PAGE_CACHE_MAX_PRELOAD_PAGES (256)
103
+#define PAGE_CACHE_MAX_PRELOAD_PAGES (64)
104
105
struct pg_alignment {
106
uint32_t page_length;
@@ -154,18 +175,30 @@ struct pg_cache_replaceQ {
175
176
struct page_cache { /* TODO: add statistics */
177
uv_rwlock_t pg_cache_rwlock; /* page cache lock */
178
+ uv_rwlock_t v2_lock;
179
180
struct pg_cache_metrics_index metrics_index;
181
struct pg_cache_committed_page_index committed_page_index;
182
struct pg_cache_replaceQ replaceQ;
183
184
unsigned page_descriptors;
185
+ unsigned active_descriptors;
186
unsigned populated_pages;
187
};
188
189
void pg_cache_wake_up_waiters_unsafe(struct rrdeng_page_descr *descr);
190
void pg_cache_wake_up_waiters(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
191
+
192
+#ifdef NETDATA_INTERNAL_CHECKS
193
+#define pg_cache_wait_event_unsafe(descr) pg_cache_wait_event_unsafe_with_trace(descr, __FUNCTION__, __LINE__)
194
+void pg_cache_wait_event_unsafe_with_trace(struct rrdeng_page_descr *descr, const char *function, size_t line);
195
+#define pg_cache_timedwait_event_unsafe(descr, timeout_sec) pg_cache_timedwait_event_unsafe_with_trace(descr, timeout_sec, __FUNCTION__, __LINE__)
196
+int pg_cache_timedwait_event_unsafe_with_trace(struct rrdeng_page_descr *descr, uint64_t timeout_sec, const char *function, size_t line);
197
+#else
198
void pg_cache_wait_event_unsafe(struct rrdeng_page_descr *descr);
199
+int pg_cache_timedwait_event_unsafe(struct rrdeng_page_descr *descr, uint64_t timeout_sec);
200
+#endif
201
+
202
unsigned long pg_cache_wait_event(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
203
void pg_cache_replaceQ_insert(struct rrdengine_instance *ctx,
204
struct rrdeng_page_descr *descr);
@@ -177,23 +210,27 @@ struct rrdeng_page_descr *pg_cache_create_descr(void);
210
int pg_cache_try_get_unsafe(struct rrdeng_page_descr *descr, int exclusive_access);
211
void pg_cache_put_unsafe(struct rrdeng_page_descr *descr);
212
void pg_cache_put(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
180
-void pg_cache_insert(struct rrdengine_instance *ctx, struct pg_cache_page_index *index,
181
- struct rrdeng_page_descr *descr);
182
-uint8_t pg_cache_punch_hole(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr,
183
- uint8_t remove_dirty, uint8_t is_exclusive_holder, uuid_t *metric_id);
184
-usec_t pg_cache_oldest_time_in_range(struct rrdengine_instance *ctx, uuid_t *id,
185
- usec_t start_time_ut, usec_t end_time_ut);
213
+struct rrdeng_page_descr *pg_cache_insert(
214
+ struct rrdengine_instance *ctx,
215
+ struct pg_cache_page_index *index,
216
+ struct rrdeng_page_descr *descr,
217
+ bool lock_and_count);
218
+
219
+uint8_t pg_cache_punch_hole(
220
+ struct rrdengine_instance *ctx,
221
+ struct rrdeng_page_descr *descr,
222
+ uint8_t remove_dirty,
223
+ uint8_t is_exclusive_holder,
224
+ uuid_t(*metric_id),
225
+ bool update_page_duration);
226
+
227
void pg_cache_get_filtered_info_prev(struct rrdengine_instance *ctx, struct pg_cache_page_index *page_index,
228
usec_t point_in_time_ut, pg_cache_page_info_filter_t *filter,
229
struct rrdeng_page_info *page_info);
189
-struct rrdeng_page_descr *pg_cache_lookup_unpopulated_and_lock(struct rrdengine_instance *ctx, uuid_t *id,
190
- usec_t start_time_ut);
230
+struct rrdeng_page_descr *pg_cache_lookup_unpopulated_and_lock(struct rrdengine_instance *ctx, uuid_t(*id), usec_t start_time_ut, struct pg_alignment *alignment);
231
unsigned
232
pg_cache_preload(struct rrdengine_instance *ctx, uuid_t *id, usec_t start_time_ut, usec_t end_time_ut,
233
struct rrdeng_page_info **page_info_arrayp, struct pg_cache_page_index **ret_page_indexp);
194
-struct rrdeng_page_descr *
195
- pg_cache_lookup(struct rrdengine_instance *ctx, struct pg_cache_page_index *index, uuid_t *id,
196
- usec_t point_in_time_ut);
234
struct rrdeng_page_descr *
235
pg_cache_lookup_next(struct rrdengine_instance *ctx, struct pg_cache_page_index *index, uuid_t *id,
236
usec_t start_time_ut, usec_t end_time_ut);
@@ -204,6 +241,7 @@ void pg_cache_add_new_metric_time(struct pg_cache_page_index *page_index, struct
241
void pg_cache_update_metric_times(struct pg_cache_page_index *page_index);
242
unsigned long pg_cache_hard_limit(struct rrdengine_instance *ctx);
243
unsigned long pg_cache_soft_limit(struct rrdengine_instance *ctx);
244
+unsigned long pg_cache_warn_limit(struct rrdengine_instance *ctx);
245
unsigned long pg_cache_committed_hard_limit(struct rrdengine_instance *ctx);
246
247
void rrdeng_page_descr_aral_go_singlethreaded(void);
@@ -213,6 +251,7 @@ void rrdeng_page_descr_use_mmap(void);
251
bool rrdeng_page_descr_is_mmap(void);
252
struct rrdeng_page_descr *rrdeng_page_descr_mallocz(void);
253
void rrdeng_page_descr_freez(struct rrdeng_page_descr *descr);
254
+void mark_journalfile_descriptor( struct page_cache *pg_cache, struct rrdengine_journalfile *journalfile, uint32_t page_offset, uint32_t Index);
255
256
static inline void
257
pg_cache_atomic_get_pg_info(struct rrdeng_page_descr *descr, usec_t *end_time_ut_p, uint32_t *page_lengthp)
database/engine/rrdengine.c
+313
-257
@@ -20,7 +20,7 @@ void *dbengine_page_alloc() {
20
if (unlikely(db_engine_use_malloc))
21
page = mallocz(RRDENG_BLOCK_SIZE);
22
else {
23
- page = netdata_mmap(NULL, RRDENG_BLOCK_SIZE, MAP_PRIVATE, enable_ksm);
23
+ page = netdata_mmap(NULL, RRDENG_BLOCK_SIZE, MAP_PRIVATE, enable_ksm, false);
24
if(!page) fatal("Cannot allocate dbengine page cache page, with mmap()");
25
}
26
return page;
@@ -60,181 +60,6 @@ static void sanity_check(void)
60
BUILD_BUG_ON(sizeof(((struct rrdeng_page_info *)0)->scratch) < 2 * sizeof(uint32_t));
61
}
62
63
-/* always inserts into tail */
64
-static inline void xt_cache_replaceQ_insert(struct rrdengine_worker_config* wc,
65
- struct extent_cache_element *xt_cache_elem)
66
-{
67
- struct extent_cache *xt_cache = &wc->xt_cache;
68
-
69
- xt_cache_elem->prev = NULL;
70
- xt_cache_elem->next = NULL;
71
-
72
- if (likely(NULL != xt_cache->replaceQ_tail)) {
73
- xt_cache_elem->prev = xt_cache->replaceQ_tail;
74
- xt_cache->replaceQ_tail->next = xt_cache_elem;
75
- }
76
- if (unlikely(NULL == xt_cache->replaceQ_head)) {
77
- xt_cache->replaceQ_head = xt_cache_elem;
78
- }
79
- xt_cache->replaceQ_tail = xt_cache_elem;
80
-}
81
-
82
-static inline void xt_cache_replaceQ_delete(struct rrdengine_worker_config* wc,
83
- struct extent_cache_element *xt_cache_elem)
84
-{
85
- struct extent_cache *xt_cache = &wc->xt_cache;
86
- struct extent_cache_element *prev, *next;
87
-
88
- prev = xt_cache_elem->prev;
89
- next = xt_cache_elem->next;
90
-
91
- if (likely(NULL != prev)) {
92
- prev->next = next;
93
- }
94
- if (likely(NULL != next)) {
95
- next->prev = prev;
96
- }
97
- if (unlikely(xt_cache_elem == xt_cache->replaceQ_head)) {
98
- xt_cache->replaceQ_head = next;
99
- }
100
- if (unlikely(xt_cache_elem == xt_cache->replaceQ_tail)) {
101
- xt_cache->replaceQ_tail = prev;
102
- }
103
- xt_cache_elem->prev = xt_cache_elem->next = NULL;
104
-}
105
-
106
-static inline void xt_cache_replaceQ_set_hot(struct rrdengine_worker_config* wc,
107
- struct extent_cache_element *xt_cache_elem)
108
-{
109
- xt_cache_replaceQ_delete(wc, xt_cache_elem);
110
- xt_cache_replaceQ_insert(wc, xt_cache_elem);
111
-}
112
-
113
-/* Returns the index of the cached extent if it was successfully inserted in the extent cache, otherwise -1 */
114
-static int try_insert_into_xt_cache(struct rrdengine_worker_config* wc, struct extent_info *extent)
115
-{
116
- struct extent_cache *xt_cache = &wc->xt_cache;
117
- struct extent_cache_element *xt_cache_elem;
118
- unsigned idx;
119
- int ret;
120
-
121
- ret = find_first_zero(xt_cache->allocation_bitmap);
122
- if (-1 == ret || ret >= MAX_CACHED_EXTENTS) {
123
- for (xt_cache_elem = xt_cache->replaceQ_head ; NULL != xt_cache_elem ; xt_cache_elem = xt_cache_elem->next) {
124
- idx = xt_cache_elem - xt_cache->extent_array;
125
- if (!check_bit(xt_cache->inflight_bitmap, idx)) {
126
- xt_cache_replaceQ_delete(wc, xt_cache_elem);
127
- break;
128
- }
129
- }
130
- if (NULL == xt_cache_elem)
131
- return -1;
132
- } else {
133
- idx = (unsigned)ret;
134
- xt_cache_elem = &xt_cache->extent_array[idx];
135
- }
136
- xt_cache_elem->extent = extent;
137
- xt_cache_elem->fileno = extent->datafile->fileno;
138
- xt_cache_elem->inflight_io_descr = NULL;
139
- xt_cache_replaceQ_insert(wc, xt_cache_elem);
140
- modify_bit(&xt_cache->allocation_bitmap, idx, 1);
141
-
142
- return (int)idx;
143
-}
144
-
145
-/**
146
- * Returns 0 if the cached extent was found in the extent cache, 1 otherwise.
147
- * Sets *idx to point to the position of the extent inside the cache.
148
- **/
149
-static uint8_t lookup_in_xt_cache(struct rrdengine_worker_config* wc, struct extent_info *extent, unsigned *idx)
150
-{
151
- struct extent_cache *xt_cache = &wc->xt_cache;
152
- struct extent_cache_element *xt_cache_elem;
153
- unsigned i;
154
-
155
- for (i = 0 ; i < MAX_CACHED_EXTENTS ; ++i) {
156
- xt_cache_elem = &xt_cache->extent_array[i];
157
- if (check_bit(xt_cache->allocation_bitmap, i) && xt_cache_elem->extent == extent &&
158
- xt_cache_elem->fileno == extent->datafile->fileno) {
159
- *idx = i;
160
- return 0;
161
- }
162
- }
163
- return 1;
164
-}
165
-
166
-#if 0 /* disabled code */
167
-static void delete_from_xt_cache(struct rrdengine_worker_config* wc, unsigned idx)
168
-{
169
- struct extent_cache *xt_cache = &wc->xt_cache;
170
- struct extent_cache_element *xt_cache_elem;
171
-
172
- xt_cache_elem = &xt_cache->extent_array[idx];
173
- xt_cache_replaceQ_delete(wc, xt_cache_elem);
174
- xt_cache_elem->extent = NULL;
175
- modify_bit(&wc->xt_cache.allocation_bitmap, idx, 0); /* invalidate it */
176
- modify_bit(&wc->xt_cache.inflight_bitmap, idx, 0); /* not in-flight anymore */
177
-}
178
-#endif
179
-
180
-void enqueue_inflight_read_to_xt_cache(struct rrdengine_worker_config* wc, unsigned idx,
181
- struct extent_io_descriptor *xt_io_descr)
182
-{
183
- struct extent_cache *xt_cache = &wc->xt_cache;
184
- struct extent_cache_element *xt_cache_elem;
185
- struct extent_io_descriptor *old_next;
186
-
187
- xt_cache_elem = &xt_cache->extent_array[idx];
188
- old_next = xt_cache_elem->inflight_io_descr->next;
189
- xt_cache_elem->inflight_io_descr->next = xt_io_descr;
190
- xt_io_descr->next = old_next;
191
-}
192
-
193
-void read_cached_extent_cb(struct rrdengine_worker_config* wc, unsigned idx, struct extent_io_descriptor *xt_io_descr)
194
-{
195
- unsigned i, j, page_offset;
196
- struct rrdengine_instance *ctx = wc->ctx;
197
- struct rrdeng_page_descr *descr;
198
- struct page_cache_descr *pg_cache_descr;
199
- void *page;
200
- struct extent_info *extent = xt_io_descr->descr_array[0]->extent;
201
-
202
- for (i = 0 ; i < xt_io_descr->descr_count; ++i) {
203
- page = dbengine_page_alloc();
204
- descr = xt_io_descr->descr_array[i];
205
- for (j = 0, page_offset = 0 ; j < extent->number_of_pages ; ++j) {
206
- /* care, we don't hold the descriptor mutex */
207
- if (!uuid_compare(*extent->pages[j]->id, *descr->id) &&
208
- extent->pages[j]->page_length == descr->page_length &&
209
- extent->pages[j]->start_time_ut == descr->start_time_ut &&
210
- extent->pages[j]->end_time_ut == descr->end_time_ut) {
211
- break;
212
- }
213
- page_offset += extent->pages[j]->page_length;
214
-
215
- }
216
- /* care, we don't hold the descriptor mutex */
217
- (void) memcpy(page, wc->xt_cache.extent_array[idx].pages + page_offset, descr->page_length);
218
-
219
- rrdeng_page_descr_mutex_lock(ctx, descr);
220
- pg_cache_descr = descr->pg_cache_descr;
221
- pg_cache_descr->page = page;
222
- pg_cache_descr->flags |= RRD_PAGE_POPULATED;
223
- pg_cache_descr->flags &= ~RRD_PAGE_READ_PENDING;
224
- rrdeng_page_descr_mutex_unlock(ctx, descr);
225
- pg_cache_replaceQ_insert(ctx, descr);
226
- if (xt_io_descr->release_descr) {
227
- pg_cache_put(ctx, descr);
228
- } else {
229
- debug(D_RRDENGINE, "%s: Waking up waiters.", __func__);
230
- pg_cache_wake_up_waiters(ctx, descr);
231
- }
232
- }
233
- if (xt_io_descr->completion)
234
- completion_mark_complete(xt_io_descr->completion);
235
- freez(xt_io_descr);
236
-}
237
-
63
static void fill_page_with_nulls(void *page, uint32_t page_length, uint8_t type) {
64
switch(type) {
65
case PAGE_METRICS: {
@@ -272,11 +97,30 @@ static void fill_page_with_nulls(void *page, uint32_t page_length, uint8_t type)
97
}
98
}
99
100
+unsigned int getdatafile_fileno(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
101
+{
102
+ if (descr->extent)
103
+ return descr->extent->datafile->fileno;
104
+
105
+ uv_rwlock_rdlock(&ctx->datafiles.rwlock);
106
+ struct rrdengine_datafile *datafile = ctx->datafiles.first;
107
+ while (datafile) {
108
+ struct rrdengine_journalfile *journalfile = datafile->journalfile;
109
+ if (journalfile->journal_data && datafile->file == descr->file)
110
+ break;
111
+ datafile = datafile->next;
112
+ }
113
+ uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
114
+
115
+ return datafile ? datafile->fileno : 0;
116
+}
117
+
118
static void do_extent_processing (struct rrdengine_worker_config *wc, struct extent_io_descriptor *xt_io_descr, bool read_failed)
119
{
120
struct rrdengine_instance *ctx = wc->ctx;
278
- struct rrdeng_page_descr *descr;
121
+ struct rrdeng_page_descr *descr = NULL;
122
struct page_cache_descr *pg_cache_descr;
123
+ struct page_cache *pg_cache = &ctx->pg_cache;
124
int ret;
125
unsigned i, j, count;
126
void *page, *uncompressed_buf = NULL;
@@ -294,33 +138,29 @@ static void do_extent_processing (struct rrdengine_worker_config *wc, struct ext
138
trailer = xt_io_descr->buf + xt_io_descr->bytes - sizeof(*trailer);
139
140
if (unlikely(read_failed)) {
297
- struct rrdengine_datafile *datafile = xt_io_descr->descr_array[0]->extent->datafile;
298
-
141
++ctx->stats.io_errors;
142
rrd_stat_atomic_add(&global_io_errors, 1);
143
have_read_error = 1;
302
- error("%s: uv_fs_read - extent at offset %"PRIu64"(%u) in datafile %u-%u.", __func__, xt_io_descr->pos,
303
- xt_io_descr->bytes, datafile->tier, datafile->fileno);
144
+ error("%s: uv_fs_read - extent at offset %"PRIu64"(%u) in datafile %u.", __func__, xt_io_descr->pos,
145
+ xt_io_descr->bytes, getdatafile_fileno(ctx, xt_io_descr->descr_array[0]));
146
goto after_crc_check;
147
}
148
crc = crc32(0L, Z_NULL, 0);
149
crc = crc32(crc, xt_io_descr->buf, xt_io_descr->bytes - sizeof(*trailer));
150
ret = crc32cmp(trailer->checksum, crc);
151
#ifdef NETDATA_INTERNAL_CHECKS
310
- {
152
+ if(xt_io_descr->descr_array[0]->extent) {
153
struct rrdengine_datafile *datafile = xt_io_descr->descr_array[0]->extent->datafile;
154
debug(D_RRDENGINE, "%s: Extent at offset %"PRIu64"(%u) was read from datafile %u-%u. CRC32 check: %s", __func__,
155
xt_io_descr->pos, xt_io_descr->bytes, datafile->tier, datafile->fileno, ret ? "FAILED" : "SUCCEEDED");
156
}
157
#endif
158
if (unlikely(ret)) {
317
- struct rrdengine_datafile *datafile = xt_io_descr->descr_array[0]->extent->datafile;
318
-
159
++ctx->stats.io_errors;
160
rrd_stat_atomic_add(&global_io_errors, 1);
161
have_read_error = 1;
322
- error("%s: Extent at offset %"PRIu64"(%u) was read from datafile %u-%u. CRC32 check: FAILED", __func__,
323
- xt_io_descr->pos, xt_io_descr->bytes, datafile->tier, datafile->fileno);
162
+ error("%s: Extent at offset %"PRIu64"(%u) was read from datafile %u. CRC32 check: FAILED", __func__,
163
+ xt_io_descr->pos, xt_io_descr->bytes, getdatafile_fileno(ctx, xt_io_descr->descr_array[0]));
164
}
165
166
after_crc_check:
@@ -337,33 +177,13 @@ after_crc_check:
177
debug(D_RRDENGINE, "LZ4 decompressed %u bytes to %d bytes.", payload_length, ret);
178
/* care, we don't hold the descriptor mutex */
179
}
340
- {
341
- uint8_t xt_is_cached = 0;
342
- unsigned xt_idx;
343
- struct extent_info *extent = xt_io_descr->descr_array[0]->extent;
344
-
345
- xt_is_cached = !lookup_in_xt_cache(wc, extent, &xt_idx);
346
- if (xt_is_cached && check_bit(wc->xt_cache.inflight_bitmap, xt_idx)) {
347
- struct extent_cache *xt_cache = &wc->xt_cache;
348
- struct extent_cache_element *xt_cache_elem = &xt_cache->extent_array[xt_idx];
349
- struct extent_io_descriptor *curr, *next;
350
-
351
- if (have_read_error) {
352
- memset(xt_cache_elem->pages, 0, sizeof(xt_cache_elem->pages));
353
- } else if (RRD_NO_COMPRESSION == header->compression_algorithm) {
354
- (void)memcpy(xt_cache_elem->pages, xt_io_descr->buf + payload_offset, payload_length);
355
- } else {
356
- (void)memcpy(xt_cache_elem->pages, uncompressed_buf, uncompressed_payload_length);
357
- }
358
- /* complete all connected in-flight read requests */
359
- for (curr = xt_cache_elem->inflight_io_descr->next ; curr ; curr = next) {
360
- next = curr->next;
361
- read_cached_extent_cb(wc, xt_idx, curr);
362
- }
363
- xt_cache_elem->inflight_io_descr = NULL;
364
- modify_bit(&xt_cache->inflight_bitmap, xt_idx, 0); /* not in-flight anymore */
365
- }
366
- }
180
+
181
+ uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
182
+ Pvoid_t *PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, xt_io_descr->descr_array[0]->id, sizeof(uuid_t));
183
+ struct pg_cache_page_index *page_index = likely( NULL != PValue) ? *PValue : NULL;
184
+ uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
185
+
186
+ struct pg_alignment *alignment = likely(NULL != page_index) ? page_index->alignment : NULL;
187
188
for (i = 0, page_offset = 0; i < count; page_offset += header->descr[i++].page_length) {
189
uint8_t is_prefetched_page;
@@ -378,13 +198,13 @@ after_crc_check:
198
header->descr[i].start_time_ut == descrj->start_time_ut &&
199
header->descr[i].end_time_ut == descrj->end_time_ut) {
200
descr = descrj;
201
+ bitmap256_set_bit(&xt_io_descr->descr_array_wakeup, j, 0);
202
break;
203
}
204
}
205
is_prefetched_page = 0;
206
if (!descr) { /* This extent page has not been requested. Try populating it for locality (best effort). */
386
- descr = pg_cache_lookup_unpopulated_and_lock(ctx, (uuid_t *)header->descr[i].uuid,
387
- header->descr[i].start_time_ut);
207
+ descr = pg_cache_lookup_unpopulated_and_lock(ctx, (uuid_t *)header->descr[i].uuid, header->descr[i].start_time_ut, alignment);
208
if (!descr)
209
continue; /* Failed to reserve a suitable page */
210
is_prefetched_page = 1;
@@ -413,6 +233,18 @@ after_crc_check:
233
pg_cache_wake_up_waiters(ctx, descr);
234
}
235
}
236
+ for (j = 0 ; j < xt_io_descr->descr_count; ++j) {
237
+ descr = xt_io_descr->descr_array[j];
238
+ if (unlikely(bitmap256_get_bit(&xt_io_descr->descr_array_wakeup, j))) {
239
+ rrdeng_page_descr_mutex_lock(ctx, descr);
240
+ if (!(descr->pg_cache_descr->flags & RRD_PAGE_POPULATED)) {
241
+ descr->pg_cache_descr->flags &= ~RRD_PAGE_READ_PENDING;
242
+ descr->pg_cache_descr->flags |= RRD_PAGE_INVALID;
243
+ }
244
+ rrdeng_page_descr_mutex_unlock(ctx, descr);
245
+ pg_cache_wake_up_waiters(ctx, descr);
246
+ }
247
+ }
248
if (!have_read_error && RRD_NO_COMPRESSION != header->compression_algorithm) {
249
freez(uncompressed_buf);
250
}
@@ -492,12 +324,21 @@ static void do_read_extent(struct rrdengine_worker_config* wc,
324
struct extent_io_descriptor *xt_io_descr;
325
struct rrdengine_datafile *datafile;
326
struct extent_info *extent = descr[0]->extent;
495
- uint8_t xt_is_cached = 0, xt_is_inflight = 0;
496
- unsigned xt_idx;
327
+ uv_file file_to_use;
328
498
- datafile = extent->datafile;
499
- pos = extent->offset;
500
- size_bytes = extent->size;
329
+ struct journal_extent_list *extent_entry = (struct journal_extent_list *) descr[0]->extent_entry;
330
+
331
+ if (likely(extent_entry)) {
332
+ file_to_use = descr[0]->file;
333
+ pos = extent_entry->datafile_offset;
334
+ size_bytes = extent_entry->datafile_size;
335
+ }
336
+ else {
337
+ datafile = extent->datafile;
338
+ file_to_use = datafile->file;
339
+ pos = extent->offset;
340
+ size_bytes = extent->size;
341
+ }
342
343
xt_io_descr = callocz(1, sizeof(*xt_io_descr));
344
for (i = 0 ; i < count; ++i) {
@@ -506,9 +347,10 @@ static void do_read_extent(struct rrdengine_worker_config* wc,
347
pg_cache_descr->flags |= RRD_PAGE_READ_PENDING;
348
rrdeng_page_descr_mutex_unlock(ctx, descr[i]);
349
xt_io_descr->descr_array[i] = descr[i];
350
+ bitmap256_set_bit(&xt_io_descr->descr_array_wakeup, i, 1);
351
}
352
xt_io_descr->descr_count = count;
511
- xt_io_descr->file = datafile->file;
353
+ xt_io_descr->file = file_to_use;
354
xt_io_descr->bytes = size_bytes;
355
xt_io_descr->pos = pos;
356
xt_io_descr->req_worker.data = xt_io_descr;
@@ -516,24 +358,6 @@ static void do_read_extent(struct rrdengine_worker_config* wc,
358
xt_io_descr->release_descr = release_descr;
359
xt_io_descr->buf = NULL;
360
519
- xt_is_cached = !lookup_in_xt_cache(wc, extent, &xt_idx);
520
- if (xt_is_cached) {
521
- xt_cache_replaceQ_set_hot(wc, &wc->xt_cache.extent_array[xt_idx]);
522
- xt_is_inflight = check_bit(wc->xt_cache.inflight_bitmap, xt_idx);
523
- if (xt_is_inflight) {
524
- enqueue_inflight_read_to_xt_cache(wc, xt_idx, xt_io_descr);
525
- return;
526
- }
527
- return read_cached_extent_cb(wc, xt_idx, xt_io_descr);
528
- } else {
529
- ret = try_insert_into_xt_cache(wc, extent);
530
- if (-1 != ret) {
531
- xt_idx = (unsigned)ret;
532
- modify_bit(&wc->xt_cache.inflight_bitmap, xt_idx, 1);
533
- wc->xt_cache.extent_array[xt_idx].inflight_io_descr = xt_io_descr;
534
- }
535
- }
536
-
361
ret = uv_queue_work(wc->loop, &xt_io_descr->req_worker, do_mmap_read_extent, read_mmap_extent_cb);
362
fatal_assert(-1 != ret);
363
@@ -647,7 +471,7 @@ static void invalidate_oldest_committed(void *arg)
471
472
goto out;
473
}
650
- pg_cache_punch_hole(ctx, descr, 1, 1, NULL);
474
+ pg_cache_punch_hole(ctx, descr, 1, 1, NULL, true);
475
476
uv_rwlock_wrlock(&pg_cache->committed_page_index.lock);
477
nr_committed_pages = --pg_cache->committed_page_index.nr_committed_pages;
@@ -698,6 +522,152 @@ void rrdeng_invalidate_oldest_committed(struct rrdengine_worker_config* wc)
522
}
523
}
524
525
+struct pg_cache_page_index *get_page_index(struct page_cache *pg_cache, uuid_t *uuid)
526
+{
527
+ uv_rwlock_rdlock(&pg_cache->metrics_index.lock);
528
+ Pvoid_t *PValue = JudyHSGet(pg_cache->metrics_index.JudyHS_array, uuid, sizeof(uuid_t));
529
+ struct pg_cache_page_index *page_index = (NULL == PValue) ? NULL : *PValue;
530
+ uv_rwlock_rdunlock(&pg_cache->metrics_index.lock);
531
+ return page_index;
532
+}
533
+
534
+struct rrdeng_page_descr *get_descriptor(struct pg_cache_page_index *page_index, time_t start_time_s)
535
+{
536
+ uv_rwlock_rdlock(&page_index->lock);
537
+ Pvoid_t *PValue = JudyLGet(page_index->JudyL_array, start_time_s, PJE0);
538
+ struct rrdeng_page_descr *descr = unlikely(NULL == PValue) ? NULL : *PValue;
539
+ uv_rwlock_rdunlock(&page_index->lock);
540
+ return descr;
541
+};
542
+
543
+static bool try_to_remove_v2_descriptor( struct rrdengine_instance *ctx, struct pg_cache_page_index *page_index, time_t start_time_s, bool expired)
544
+{
545
+ struct rrdeng_page_descr *descr = get_descriptor(page_index, start_time_s);
546
+ if (unlikely(!descr))
547
+ return true;
548
+
549
+ rrdeng_page_descr_mutex_lock(ctx, descr);
550
+ unsigned flags = descr->pg_cache_descr->flags & RRD_PAGE_POPULATED;
551
+ if ((!flags || expired) && pg_cache_try_get_unsafe(descr, 1)) {
552
+ rrdeng_page_descr_mutex_unlock(ctx, descr);
553
+ pg_cache_punch_hole(ctx, descr, 0, 1, NULL, false);
554
+ return true;
555
+ }
556
+ rrdeng_page_descr_mutex_unlock(ctx, descr);
557
+
558
+ return false;
559
+}
560
+
561
+#ifndef DESCRIPTOR_EXPIRATION_TIME
562
+#define DESCRIPTOR_EXPIRATION_TIME (365 * 86400)
563
+#endif
564
+
565
+static void check_journal_file(struct rrdengine_journalfile *journalfile, size_t count)
566
+{
567
+ struct rrdengine_instance *ctx = journalfile->datafile->ctx;
568
+ struct page_cache *pg_cache = &ctx->pg_cache;
569
+
570
+ Pvoid_t *PValue;
571
+ Word_t page_address;
572
+ uint32_t Index;
573
+
574
+ struct journal_v2_header *journal_header = (struct journal_v2_header *) journalfile->journal_data;
575
+ time_t journal_start_time_s = (time_t) (journal_header->start_time_ut / USEC_PER_SEC);
576
+
577
+ uv_rwlock_rdlock(&pg_cache->v2_lock);
578
+
579
+ bool expired = ((now_realtime_sec() - journalfile->last_access) > DESCRIPTOR_EXPIRATION_TIME);
580
+
581
+ unsigned count_evicted = 0;
582
+ for (page_address = 0,
583
+ PValue = JudyLFirst(journalfile->JudyL_array, &page_address, PJE0),
584
+ Index = unlikely(NULL == PValue) ? 0 : *(uint32_t *) PValue;
585
+ Index ;
586
+ PValue = JudyLNext(journalfile->JudyL_array, &page_address, PJE0),
587
+ Index = unlikely(NULL == PValue) ? 0 : *(uint32_t *) PValue) {
588
+
589
+ uv_rwlock_rdunlock(&pg_cache->v2_lock);
590
+
591
+ // Assume we will evict everything
592
+ bool all_evicted = true;
593
+ // Get the page index will be working on
594
+ struct journal_page_header *page_list_header = journalfile->journal_data + page_address;
595
+ struct pg_cache_page_index *page_index = get_page_index(pg_cache, &page_list_header->uuid);
596
+
597
+ if (likely(page_index)) {
598
+ struct journal_page_list *page_list = (struct journal_page_list *) ((void *) page_list_header + sizeof(*page_list_header));
599
+ uint32_t entries = page_list_header->entries;
600
+
601
+ // First try to target the marked entry; Note marked entry is recorded +1
602
+ struct journal_page_list *page_entry = &page_list[Index - 1];
603
+ time_t index_time_s = journal_start_time_s + page_entry->delta_start_s;
604
+ all_evicted = try_to_remove_v2_descriptor(ctx, page_index, index_time_s, expired);
605
+
606
+ // Try to scan range ; all need to return evicted
607
+ for (uint32_t x = 0; all_evicted && x < entries; x++) {
608
+ index_time_s = journal_start_time_s + (&page_list[x])->delta_start_s;
609
+ all_evicted = all_evicted && try_to_remove_v2_descriptor(ctx, page_index, index_time_s, expired);
610
+ }
611
+ }
612
+
613
+ if (all_evicted) {
614
+ uv_rwlock_wrlock(&pg_cache->v2_lock);
615
+ (void) JudyLDel(&journalfile->JudyL_array, page_address, PJE0);
616
+ uv_rwlock_wrunlock(&pg_cache->v2_lock);
617
+ ++count_evicted;
618
+ }
619
+
620
+ uv_rwlock_rdlock(&pg_cache->v2_lock);
621
+ if (count_evicted > count / 10)
622
+ break;
623
+ }
624
+
625
+ uv_rwlock_rdunlock(&pg_cache->v2_lock);
626
+}
627
+
628
+void after_delete_descriptors(struct rrdengine_worker_config* wc)
629
+{
630
+ int error = uv_thread_join(wc->now_deleting_descriptors);
631
+ if (error)
632
+ error("uv_thread_join(): %s", uv_strerror(error));
633
+ freez(wc->now_deleting_descriptors);
634
+ wc->now_deleting_descriptors = NULL;
635
+ wc->cleanup_deleting_descriptors = 0;
636
+ /* interrupt event loop */
637
+ uv_stop(wc->loop);
638
+}
639
+
640
+void delete_descriptors(void *arg)
641
+{
642
+ struct rrdengine_instance *ctx = arg;
643
+ struct page_cache *pg_cache = &ctx->pg_cache;
644
+ struct rrdengine_worker_config *wc = &ctx->worker_config;
645
+
646
+ uv_rwlock_rdlock(&ctx->datafiles.rwlock);
647
+ struct rrdengine_datafile *datafile = ctx->datafiles.first;
648
+ struct rrdengine_journalfile *journalfile;
649
+ while (datafile) {
650
+ journalfile = datafile->journalfile;
651
+ if (!journalfile->journal_data) {
652
+ datafile = datafile->next;
653
+ continue;
654
+ }
655
+
656
+ uv_rwlock_rdlock(&pg_cache->v2_lock);
657
+ Word_t count = JudyLCount(journalfile->JudyL_array, 0, -1, PJE0);
658
+ uv_rwlock_rdunlock(&pg_cache->v2_lock);
659
+
660
+ if (unlikely(count))
661
+ check_journal_file(journalfile, count);
662
+
663
+ datafile = datafile->next;
664
+ }
665
+
666
+ uv_rwlock_rdunlock(&ctx->datafiles.rwlock);
667
+ wc->cleanup_deleting_descriptors = 1;
668
+ fatal_assert(0 == uv_async_send(&wc->async));
669
+}
670
+
671
void flush_pages_cb(uv_fs_t* req)
672
{
673
struct rrdengine_worker_config* wc = req->loop->data;
@@ -794,15 +764,43 @@ static int do_flush_pages(struct rrdengine_worker_config* wc, int force, struct
764
765
rrdeng_page_descr_mutex_lock(ctx, descr);
766
pg_cache_descr = descr->pg_cache_descr;
797
- if (!(pg_cache_descr->flags & RRD_PAGE_WRITE_PENDING)) {
767
+ bool invalid_page = ((pg_cache_descr->flags & RRD_PAGE_INVALID) == RRD_PAGE_INVALID);
768
+ if (likely(false == invalid_page)) {
769
+ if (!(pg_cache_descr->flags & RRD_PAGE_WRITE_PENDING)) {
770
+ page_write_pending = 1;
771
+ /* care, no reference being held */
772
+ pg_cache_descr->flags |= RRD_PAGE_WRITE_PENDING;
773
+ uncompressed_payload_length += descr->page_length;
774
+ descr_commit_idx_array[count] = Index;
775
+ eligible_pages[count++] = descr;
776
+ }
777
+ rrdeng_page_descr_mutex_unlock(ctx, descr);
778
+ } else {
779
+#ifdef NETDATA_INTERNAL_CHECKS
780
+ {
781
+ char uuid_str[UUID_STR_LEN];
782
+ uuid_unparse_lower(*descr->id, uuid_str);
783
+ internal_error(true, "SKIPPING page %p marked as invalid for %s with %llu, %llu length=%u",
784
+ descr, uuid_str, descr->start_time_ut, descr->end_time_ut, descr->page_length);
785
+
786
+ struct pg_cache_page_index *page_index = get_page_index(pg_cache, descr->id);
787
+ struct rrdeng_page_descr *in_cache_descr = get_descriptor(page_index,(time_t) (descr->start_time_ut / USEC_PER_SEC ));
788
+
789
+ if (likely(in_cache_descr))
790
+ internal_error(true, "MEMORY page %p found %s with %llu, %llu length=%u",
791
+ in_cache_descr, uuid_str, in_cache_descr->start_time_ut / USEC_PER_SEC, in_cache_descr->end_time_ut / USEC_PER_SEC, in_cache_descr->page_length);
792
+ else
793
+ internal_error(true, "MEMORY page not found for %s", uuid_str);
794
+ }
795
+#endif
796
+ if (pg_cache_try_get_unsafe(descr, 1)) {
797
+ rrdeng_page_descr_mutex_unlock(ctx, descr);
798
+ pg_cache_punch_hole(ctx, descr, 0, 1, NULL, false);
799
+ }
800
+ else
801
+ rrdeng_page_descr_mutex_unlock(ctx, descr);
802
page_write_pending = 1;
799
- /* care, no reference being held */
800
- pg_cache_descr->flags |= RRD_PAGE_WRITE_PENDING;
801
- uncompressed_payload_length += descr->page_length;
802
- descr_commit_idx_array[count] = Index;
803
- eligible_pages[count++] = descr;
803
}
805
- rrdeng_page_descr_mutex_unlock(ctx, descr);
804
805
if (page_write_pending) {
806
ret = JudyLDel(&pg_cache->committed_page_index.JudyL_array, Index, PJE0);
@@ -929,21 +927,25 @@ static void after_delete_old_data(struct rrdengine_worker_config* wc)
927
int ret, error;
928
char path[RRDENG_PATH_MAX];
929
930
+ uv_rwlock_wrlock(&ctx->datafiles.rwlock);
931
+
932
datafile = ctx->datafiles.first;
933
journalfile = datafile->journalfile;
934
datafile_bytes = datafile->pos;
935
journalfile_bytes = journalfile->pos;
936
- deleted_bytes = 0;
936
+ deleted_bytes = journalfile->journal_data_size;
937
938
- info("Deleting data and journal file pair.");
939
- datafile_list_delete(ctx, datafile);
940
- ret = destroy_journal_file(journalfile, datafile);
938
+ info("Deleting data and journal files");
939
+ datafile_list_delete_unsafe(ctx, datafile);
940
+ ret = destroy_journal_file_unsafe(journalfile, datafile);
941
if (!ret) {
942
generate_journalfilepath(datafile, path, sizeof(path));
943
info("Deleted journal file \"%s\".", path);
944
+ generate_journalfilepath_v2(datafile, path, sizeof(path));
945
+ info("Deleted journal file \"%s\".", path);
946
deleted_bytes += journalfile_bytes;
947
}
946
- ret = destroy_data_file(datafile);
948
+ ret = destroy_data_file_unsafe(datafile);
949
if (!ret) {
950
generate_datafilepath(datafile, path, sizeof(path));
951
info("Deleted data file \"%s\".", path);
@@ -964,6 +966,9 @@ static void after_delete_old_data(struct rrdengine_worker_config* wc)
966
wc->now_deleting_files = NULL;
967
968
wc->cleanup_thread_deleting_files = 0;
969
+
970
+ uv_rwlock_wrunlock(&ctx->datafiles.rwlock);
971
+
972
rrdcontext_db_rotation();
973
974
/* interrupt event loop */
@@ -982,13 +987,18 @@ static void delete_old_data(void *arg)
987
uuid_t metric_id;
988
989
/* Safe to use since it will be deleted after we are done */
990
+
991
datafile = ctx->datafiles.first;
992
993
+ uv_rwlock_wrlock(&datafile->extent_rwlock);
994
for (extent = datafile->extents.first ; extent != NULL ; extent = next) {
995
count = extent->number_of_pages;
996
for (i = 0 ; i < count ; ++i) {
997
descr = extent->pages[i];
991
- can_delete_metric = pg_cache_punch_hole(ctx, descr, 0, 0, &metric_id);
998
+ if (unlikely(!descr))
999
+ continue;
1000
+
1001
+ can_delete_metric = pg_cache_punch_hole(ctx, descr, 0, 0, &metric_id, true);
1002
if (unlikely(can_delete_metric)) {
1003
/*
1004
* If the metric is empty, has no active writers and if the metadata log has been initialized then
@@ -1000,19 +1010,31 @@ static void delete_old_data(void *arg)
1010
next = extent->next;
1011
freez(extent);
1012
}
1013
+ uv_rwlock_wrunlock(&datafile->extent_rwlock);
1014
+
1015
wc->cleanup_thread_deleting_files = 1;
1016
/* wake up event loop */
1017
fatal_assert(0 == uv_async_send(&wc->async));
1018
}
1019
1020
+#define DESCRIPTOR_INITIAL_CLEANUP (60)
1021
+#ifndef DESCRIPTOR_INTERVAL_CLEANUP
1022
+#define DESCRIPTOR_INTERVAL_CLEANUP (1)
1023
+#endif
1024
+
1025
void rrdeng_test_quota(struct rrdengine_worker_config* wc)
1026
{
1027
+ static time_t next_descriptor_cleanup = 0;
1028
struct rrdengine_instance *ctx = wc->ctx;
1029
struct rrdengine_datafile *datafile;
1030
unsigned current_size, target_size;
1031
uint8_t out_of_space, only_one_datafile;
1032
int ret, error;
1033
1034
+ if (0 == next_descriptor_cleanup) {
1035
+ next_descriptor_cleanup = now_realtime_sec() + DESCRIPTOR_INITIAL_CLEANUP;
1036
+ }
1037
+
1038
out_of_space = 0;
1039
/* Do not allow the pinned pages to exceed the disk space quota to avoid deadlocks */
1040
if (unlikely(ctx->disk_space > MAX(ctx->max_disk_space, 2 * ctx->metric_API_max_producers * RRDENG_BLOCK_SIZE))) {
@@ -1024,14 +1046,33 @@ void rrdeng_test_quota(struct rrdengine_worker_config* wc)
1046
target_size = MIN(target_size, MAX_DATAFILE_SIZE);
1047
target_size = MAX(target_size, MIN_DATAFILE_SIZE);
1048
only_one_datafile = (datafile == ctx->datafiles.first) ? 1 : 0;
1027
- if (unlikely(current_size >= target_size || (out_of_space && only_one_datafile))) {
1049
+
1050
+ if (unlikely(current_size >= target_size || (out_of_space && only_one_datafile)) && !wc->now_deleting_descriptors) {
1051
/* Finalize data and journal file and create a new pair */
1052
+ struct rrdengine_journalfile *journalfile = unlikely(NULL == ctx->datafiles.last) ? NULL : ctx->datafiles.last->journalfile;
1053
wal_flush_transaction_buffer(wc);
1054
ret = create_new_datafile_pair(ctx, 1, ctx->last_fileno + 1);
1055
if (likely(!ret)) {
1056
++ctx->last_fileno;
1057
+ if (likely(journalfile && db_engine_journal_indexing))
1058
+ wc->run_indexing = true;
1059
}
1060
}
1061
+
1062
+ if (db_engine_journal_indexing && !wc->now_deleting_files && !wc->now_deleting_descriptors && !wc->running_journal_migration && !out_of_space &&
1063
+ NO_QUIESCE == ctx->quiesce && next_descriptor_cleanup < now_realtime_sec()) {
1064
+ next_descriptor_cleanup = now_realtime_sec() + DESCRIPTOR_INTERVAL_CLEANUP;
1065
+
1066
+ wc->now_deleting_descriptors = mallocz(sizeof(*wc->now_deleting_descriptors));
1067
+ wc->cleanup_deleting_descriptors = 0;
1068
+ error = uv_thread_create(wc->now_deleting_descriptors, delete_descriptors, ctx);
1069
+ if (error) {
1070
+ error("uv_thread_create(): %s", uv_strerror(error));
1071
+ freez(wc->now_deleting_descriptors);
1072
+ wc->now_deleting_descriptors = NULL;
1073
+ }
1074
+ }
1075
+
1076
if (unlikely(out_of_space && NO_QUIESCE == ctx->quiesce)) {
1077
/* delete old data */
1078
if (wc->now_deleting_files) {
@@ -1060,7 +1101,7 @@ void rrdeng_test_quota(struct rrdengine_worker_config* wc)
1101
1102
static inline int rrdeng_threads_alive(struct rrdengine_worker_config* wc)
1103
{
1063
- if (wc->now_invalidating_dirty_pages || wc->now_deleting_files) {
1104
+ if (wc->now_invalidating_dirty_pages || wc->now_deleting_files || wc->now_deleting_descriptors) {
1105
return 1;
1106
}
1107
return 0;
@@ -1076,6 +1117,10 @@ static void rrdeng_cleanup_finished_threads(struct rrdengine_worker_config* wc)
1117
if (unlikely(wc->cleanup_thread_deleting_files)) {
1118
after_delete_old_data(wc);
1119
}
1120
+
1121
+ if (unlikely(wc->cleanup_deleting_descriptors))
1122
+ after_delete_descriptors(wc);
1123
+
1124
if (unlikely(SET_QUIESCE == ctx->quiesce && !rrdeng_threads_alive(wc))) {
1125
ctx->quiesce = QUIESCED;
1126
completion_mark_complete(&ctx->rrdengine_completion);
@@ -1244,8 +1289,9 @@ void timer_cb(uv_timer_t* handle)
1289
uv_stop(handle->loop);
1290
uv_update_time(handle->loop);
1291
rrdeng_test_quota(wc);
1292
+
1293
debug(D_RRDENGINE, "%s: timeout reached.", __func__);
1248
- if (likely(!wc->now_deleting_files && !wc->now_invalidating_dirty_pages)) {
1294
+ if (likely(!wc->now_deleting_files && !wc->now_invalidating_dirty_pages && !wc->now_deleting_descriptors && !wc->running_journal_migration)) {
1295
/* There is free space so we can write to disk and we are not actively deleting dirty buffers */
1296
struct page_cache *pg_cache = &ctx->pg_cache;
1297
unsigned long total_bytes, bytes_written, nr_committed_pages, bytes_to_write = 0, producers, low_watermark,
@@ -1277,6 +1323,12 @@ void timer_cb(uv_timer_t* handle)
1323
}
1324
}
1325
}
1326
+
1327
+ if (true == wc->run_indexing) {
1328
+ wc->run_indexing = false;
1329
+ queue_journalfile_v2_migration(wc);
1330
+ }
1331
+
1332
load_configuration_dynamic();
1333
#ifdef NETDATA_INTERNAL_CHECKS
1334
{
@@ -1284,7 +1336,6 @@ void timer_cb(uv_timer_t* handle)
1336
debug(D_RRDENGINE, "%s", get_rrdeng_statistics(wc->ctx, buf, sizeof(buf)));
1337
}
1338
#endif
1287
-
1339
worker_is_idle();
1340
}
1341
@@ -1333,9 +1384,14 @@ void rrdeng_worker(void* arg)
1384
wc->now_deleting_files = NULL;
1385
wc->cleanup_thread_deleting_files = 0;
1386
1387
+ wc->now_deleting_descriptors = NULL;
1388
+ wc->cleanup_deleting_descriptors = 0;
1389
+ wc->running_journal_migration = 0;
1390
+
1391
wc->now_invalidating_dirty_pages = NULL;
1392
wc->cleanup_thread_invalidating_dirty_pages = 0;
1393
wc->inflight_dirty_pages = 0;
1394
+ wc->run_indexing = false;
1395
1396
/* dirty page flushing timer */
1397
ret = uv_timer_init(loop, &timer_req);
database/engine/rrdengine.h
+33
-12
@@ -92,12 +92,22 @@ struct rrdeng_cmd {
92
union {
93
struct rrdeng_read_page read_page;
94
struct rrdeng_read_extent read_extent;
95
+ struct rrdengine_journalfile *journalfile;
96
struct completion *completion;
97
};
98
};
99
100
#define RRDENG_CMD_Q_MAX_SIZE (2048)
101
102
+struct rrdeng_work {
103
+ uv_work_t req;
104
+ struct rrdengine_worker_config *wc;
105
+ void *data;
106
+ uint32_t count;
107
+ bool rerun;
108
+ struct completion *completion;
109
+};
110
+
111
struct rrdeng_cmdqueue {
112
unsigned head, tail;
113
struct rrdeng_cmd cmd_array[RRDENG_CMD_Q_MAX_SIZE];
@@ -117,6 +127,7 @@ struct extent_io_descriptor {
127
unsigned descr_count;
128
int release_descr;
129
struct rrdeng_page_descr *descr_array[MAX_PAGES_PER_EXTENT];
130
+ BITMAP256 descr_array_wakeup;
131
Word_t descr_commit_idx_array[MAX_PAGES_PER_EXTENT];
132
struct extent_io_descriptor *next; /* multiple requests to be served by the same cached extent */
133
};
@@ -125,6 +136,7 @@ struct generic_io_descriptor {
136
uv_fs_t req;
137
uv_buf_t iov;
138
void *buf;
139
+ void *data;
140
uint64_t pos;
141
unsigned bytes;
142
struct completion *completion;
@@ -133,8 +145,8 @@ struct generic_io_descriptor {
145
struct extent_cache_element {
146
struct extent_info *extent; /* The ABA problem is avoided with the help of fileno below */
147
unsigned fileno;
136
- struct extent_cache_element *prev; /* LRU */
137
- struct extent_cache_element *next; /* LRU */
148
+ struct extent_cache_element *prev; /* LRU */
149
+ struct extent_cache_element *next; /* LRU */
150
struct extent_io_descriptor *inflight_io_descr; /* I/O descriptor for in-flight extent */
151
uint8_t pages[MAX_PAGES_PER_EXTENT * RRDENG_BLOCK_SIZE];
152
};
@@ -145,7 +157,7 @@ struct extent_cache_element {
157
struct extent_cache {
158
struct extent_cache_element extent_array[MAX_CACHED_EXTENTS];
159
unsigned allocation_bitmap; /* 1 if the corresponding position in the extent_array is allocated */
148
- unsigned inflight_bitmap; /* 1 if the corresponding position in the extent_array is waiting for I/O */
160
+ unsigned inflight_bitmap; /* 1 if the corresponding position in the extent_array is waiting for I/O */
161
162
struct extent_cache_element *replaceQ_head; /* LRU */
163
struct extent_cache_element *replaceQ_tail; /* MRU */
@@ -155,18 +167,23 @@ struct rrdengine_worker_config {
167
struct rrdengine_instance *ctx;
168
169
uv_thread_t thread;
158
- uv_loop_t* loop;
170
+ uv_loop_t *loop;
171
uv_async_t async;
172
173
/* file deletion thread */
174
uv_thread_t *now_deleting_files;
175
+ uv_thread_t *now_deleting_descriptors;
176
unsigned long cleanup_thread_deleting_files; /* set to 0 when now_deleting_files is still running */
177
+ unsigned long cleanup_deleting_descriptors; /* set to 0 when now_deleting_descriptors is still running */
178
+
179
+ unsigned long running_journal_migration;
180
181
/* dirty page deletion thread */
182
uv_thread_t *now_invalidating_dirty_pages;
183
/* set to 0 when now_invalidating_dirty_pages is still running */
184
unsigned long cleanup_thread_invalidating_dirty_pages;
185
unsigned inflight_dirty_pages;
186
+ bool run_indexing;
187
188
/* FIFO command queue */
189
uv_mutex_t cmd_mutex;
@@ -225,9 +242,9 @@ extern rrdeng_stats_t rrdeng_reserved_file_descriptors;
242
extern rrdeng_stats_t global_pg_cache_over_half_dirty_events;
243
extern rrdeng_stats_t global_flushing_pressure_page_deletions; /* number of deleted pages */
244
228
-#define NO_QUIESCE (0) /* initial state when all operations function normally */
245
+#define NO_QUIESCE (0) /* initial state when all operations function normally */
246
#define SET_QUIESCE (1) /* set it before shutting down the instance, quiesce long running operations */
230
-#define QUIESCED (2) /* is set after all threads have finished running */
247
+#define QUIESCED (2) /* is set after all threads have finished running */
248
249
typedef enum {
250
LOAD_ERRORS_PAGE_FLIPPED_TIME = 0,
@@ -255,9 +272,10 @@ struct rrdengine_instance {
272
unsigned last_fileno; /* newest index of datafile and journalfile */
273
unsigned long max_cache_pages;
274
unsigned long cache_pages_low_watermark;
275
+ unsigned long cache_pages_warn_watermark;
276
unsigned long metric_API_max_producers;
277
260
- uint8_t quiesce; /* set to SET_QUIESCE before shutdown of the engine */
278
+ uint8_t quiesce; /* set to SET_QUIESCE before shutdown of the engine */
279
uint8_t page_type; /* Default page type for this context */
280
281
struct rrdengine_statistics stats;
@@ -273,9 +291,12 @@ void dbengine_page_free(void *page);
291
292
int init_rrd_files(struct rrdengine_instance *ctx);
293
void finalize_rrd_files(struct rrdengine_instance *ctx);
276
-void rrdeng_test_quota(struct rrdengine_worker_config* wc);
277
-void rrdeng_worker(void* arg);
278
-void rrdeng_enq_cmd(struct rrdengine_worker_config* wc, struct rrdeng_cmd *cmd);
279
-struct rrdeng_cmd rrdeng_deq_cmd(struct rrdengine_worker_config* wc);
280
-
294
+void rrdeng_test_quota(struct rrdengine_worker_config *wc);
295
+void rrdeng_worker(void *arg);
296
+void rrdeng_enq_cmd(struct rrdengine_worker_config *wc, struct rrdeng_cmd *cmd);
297
+struct rrdeng_cmd rrdeng_deq_cmd(struct rrdengine_worker_config *wc);
298
+void after_journal_indexing(uv_work_t *req, int status);
299
+void start_journal_indexing(uv_work_t *req);
300
+struct pg_cache_page_index *get_page_index(struct page_cache *pg_cache, uuid_t *uuid);
301
+struct rrdeng_page_descr *get_descriptor(struct pg_cache_page_index *page_index, time_t start_time_s);
302
#endif /* NETDATA_RRDENGINE_H */
database/engine/rrdengineapi.c
+18
-69
@@ -31,6 +31,8 @@ int db_engine_use_malloc = 0;
31
int default_rrdeng_page_fetch_timeout = 3;
32
int default_rrdeng_page_fetch_retries = 3;
33
int default_rrdeng_page_cache_mb = 32;
34
+int db_engine_journal_indexing = 1;
35
+int db_engine_journal_check = 0;
36
int default_rrdeng_disk_quota_mb = 256;
37
int default_multidb_disk_quota_mb = 256;
38
/* Default behaviour is to unblock data collection if the page cache is full of dirty pages by dropping metrics */
@@ -285,7 +287,7 @@ void rrdeng_store_metric_flush_current_page(STORAGE_COLLECT_HANDLE *collection_h
287
if (page_is_empty) {
288
print_page_cache_descr(descr, "Page has empty metrics only, deleting", true);
289
pg_cache_put(ctx, descr);
288
- pg_cache_punch_hole(ctx, descr, 1, 0, NULL);
290
+ pg_cache_punch_hole(ctx, descr, 1, 0, NULL, true);
291
} else
292
rrdeng_commit_page(ctx, descr, handle->page_correlation_id);
293
} else {
@@ -420,7 +422,7 @@ static void rrdeng_store_metric_next_internal(STORAGE_COLLECT_HANDLE *collection
422
}
423
}
424
423
- pg_cache_insert(ctx, page_index, descr);
425
+ (void )pg_cache_insert(ctx, page_index, descr, true);
426
} else {
427
pg_cache_add_new_metric_time(page_index, descr);
428
}
@@ -461,18 +463,9 @@ void rrdeng_store_metric_next(STORAGE_COLLECT_HANDLE *collection_handle,
463
464
if(unlikely(points_gap != 1)) {
465
if (unlikely(points_gap <= 0)) {
464
- time_t now = now_realtime_sec();
465
- static __thread size_t counter = 0;
466
- static __thread time_t last_time_logged = 0;
467
- counter++;
468
-
469
- if(now - last_time_logged > 600) {
470
- error("DBENGINE: collected point is in the past (repeated %zu times in the last %zu secs). Ignoring these data collection points.",
471
- counter, (size_t)(last_time_logged?(now - last_time_logged):0));
472
-
473
- last_time_logged = now;
474
- counter = 0;
475
- }
466
+ error_limit_static_global_var(erl, 1, 0);
467
+ error_limit(&erl, "DBENGINE: ignoring past collected point at %llu, which is in the past of the current page end time %llu",
468
+ point_in_time_ut, last_point_in_time_ut);
469
return;
470
}
471
@@ -500,7 +493,7 @@ void rrdeng_store_metric_next(STORAGE_COLLECT_HANDLE *collection_handle,
493
// print_page_cache_descr(descr, buffer, false);
494
495
// loop to fill the gap
503
- usec_t step_ut = page_index->latest_update_every_s * USEC_PER_SEC;
496
+ usec_t step_ut = update_every_ut;
497
usec_t last_point_filled_ut = last_point_in_time_ut + step_ut;
498
499
while (last_point_filled_ut < point_in_time_ut) {
@@ -513,6 +506,12 @@ void rrdeng_store_metric_next(STORAGE_COLLECT_HANDLE *collection_handle,
506
}
507
}
508
}
509
+ else if(unlikely(point_in_time_ut <= page_index->latest_time_ut && page_index->latest_time_ut != INVALID_TIME)) {
510
+ error_limit_static_global_var(erl, 1, 0);
511
+ error_limit(&erl, "DBENGINE: ignoring past collected point at %llu, while is in the past of latest value in the database %llu",
512
+ point_in_time_ut, page_index->latest_time_ut);
513
+ return;
514
+ }
515
516
rrdeng_store_metric_next_internal(collection_handle, point_in_time_ut, n, min_value, max_value, count, anomaly_count, flags);
517
}
@@ -551,16 +550,6 @@ void rrdeng_store_metric_change_collection_frequency(STORAGE_COLLECT_HANDLE *col
550
// ----------------------------------------------------------------------------
551
// query ops
552
554
-//static inline uint32_t *pginfo_to_dt(struct rrdeng_page_info *page_info)
555
-//{
556
-// return (uint32_t *)&page_info->scratch[0];
557
-//}
558
-//
559
-//static inline uint32_t *pginfo_to_points(struct rrdeng_page_info *page_info)
560
-//{
561
-// return (uint32_t *)&page_info->scratch[sizeof(uint32_t)];
562
-//}
563
-//
553
/*
554
* Gets a handle for loading metrics from the database.
555
* The handle must be released with rrdeng_load_metric_final().
@@ -624,6 +613,7 @@ static int rrdeng_load_page_next(struct storage_engine_query_handle *rrdimm_hand
613
usec_t wanted_start_time_ut = handle->wanted_start_time_s * USEC_PER_SEC;
614
descr = pg_cache_lookup_next(ctx, handle->page_index, &handle->page_index->id,
615
wanted_start_time_ut, rrdimm_handle->end_time_s * USEC_PER_SEC);
616
+
617
if (NULL == descr)
618
return 1;
619
@@ -930,44 +920,6 @@ void rrdeng_commit_page(struct rrdengine_instance *ctx, struct rrdeng_page_descr
920
pg_cache_put(ctx, descr);
921
}
922
933
-/* Gets a reference for the page */
934
-void *rrdeng_get_latest_page(struct rrdengine_instance *ctx, uuid_t *id, void **handle)
935
-{
936
- struct rrdeng_page_descr *descr;
937
- struct page_cache_descr *pg_cache_descr;
938
-
939
- debug(D_RRDENGINE, "Reading existing page:");
940
- descr = pg_cache_lookup(ctx, NULL, id, INVALID_TIME);
941
- if (NULL == descr) {
942
- *handle = NULL;
943
-
944
- return NULL;
945
- }
946
- *handle = descr;
947
- pg_cache_descr = descr->pg_cache_descr;
948
-
949
- return pg_cache_descr->page;
950
-}
951
-
952
-/* Gets a reference for the page */
953
-void *rrdeng_get_page(struct rrdengine_instance *ctx, uuid_t *id, usec_t point_in_time_ut, void **handle)
954
-{
955
- struct rrdeng_page_descr *descr;
956
- struct page_cache_descr *pg_cache_descr;
957
-
958
- debug(D_RRDENGINE, "Reading existing page:");
959
- descr = pg_cache_lookup(ctx, NULL, id, point_in_time_ut);
960
- if (NULL == descr) {
961
- *handle = NULL;
962
-
963
- return NULL;
964
- }
965
- *handle = descr;
966
- pg_cache_descr = descr->pg_cache_descr;
967
-
968
- return pg_cache_descr->page;
969
-}
970
-
923
/*
924
* Gathers Database Engine statistics.
925
* Careful when modifying this function.
@@ -1018,7 +970,8 @@ void rrdeng_get_37_statistics(struct rrdengine_instance *ctx, unsigned long long
970
array[34] = (uint64_t)global_pg_cache_over_half_dirty_events;
971
array[35] = (uint64_t)ctx->stats.flushing_pressure_page_deletions;
972
array[36] = (uint64_t)global_flushing_pressure_page_deletions;
1021
- fatal_assert(RRDENG_NR_STATS == 37);
973
+ array[37] = (uint64_t)pg_cache->active_descriptors;
974
+ fatal_assert(RRDENG_NR_STATS == 38);
975
}
976
977
/* Releases reference to page */
@@ -1067,6 +1020,7 @@ int rrdeng_init(RRDHOST *host, struct rrdengine_instance **ctxp, char *dbfiles_p
1020
ctx->max_cache_pages = page_cache_mb * (1048576LU / RRDENG_BLOCK_SIZE);
1021
/* try to keep 5% of the page cache free */
1022
ctx->cache_pages_low_watermark = (ctx->max_cache_pages * 95LLU) / 100;
1023
+ ctx->cache_pages_warn_watermark = (ctx->max_cache_pages * 90LLU) / 100;
1024
if (disk_space_mb < RRDENG_MIN_DISK_SPACE_MB)
1025
disk_space_mb = RRDENG_MIN_DISK_SPACE_MB;
1026
ctx->max_disk_space = disk_space_mb * 1048576LLU;
@@ -1100,11 +1054,6 @@ int rrdeng_init(RRDHOST *host, struct rrdengine_instance **ctxp, char *dbfiles_p
1054
if (ctx->worker_config.error) {
1055
goto error_after_rrdeng_worker;
1056
}
1103
-// error = metalog_init(ctx);
1104
-// if (error) {
1105
-// error("Failed to initialize metadata log file event loop.");
1106
-// goto error_after_rrdeng_worker;
1107
-// }
1057
1058
return 0;
1059
database/engine/rrdengineapi.h
+3
-1
@@ -8,7 +8,7 @@
8
#define RRDENG_MIN_PAGE_CACHE_SIZE_MB (8)
9
#define RRDENG_MIN_DISK_SPACE_MB (64)
10
11
-#define RRDENG_NR_STATS (37)
11
+#define RRDENG_NR_STATS (38)
12
13
#define RRDENG_FD_BUDGET_PER_INSTANCE (50)
14
@@ -16,6 +16,8 @@ extern int db_engine_use_malloc;
16
extern int default_rrdeng_page_fetch_timeout;
17
extern int default_rrdeng_page_fetch_retries;
18
extern int default_rrdeng_page_cache_mb;
19
+extern int db_engine_journal_indexing;
20
+extern int db_engine_journal_check;
21
extern int default_rrdeng_disk_quota_mb;
22
extern int default_multidb_disk_quota_mb;
23
extern uint8_t rrdeng_drop_metrics_under_page_cache_pressure;
database/engine/rrdenglocking.c
+22
-2
@@ -12,6 +12,10 @@ struct page_cache_descr *rrdeng_create_pg_cache_descr(struct rrdengine_instance
12
pg_cache_descr->prev = pg_cache_descr->next = NULL;
13
pg_cache_descr->refcnt = 0;
14
pg_cache_descr->waiters = 0;
15
+#ifdef NETDATA_INTERNAL_CHECKS
16
+ pg_cache_descr->wait_list = NULL;
17
+ pg_cache_descr->owner.function = NULL;
18
+#endif
19
fatal_assert(0 == uv_cond_init(&pg_cache_descr->cond));
20
fatal_assert(0 == uv_mutex_init(&pg_cache_descr->mutex));
21
@@ -27,7 +31,11 @@ void rrdeng_destroy_pg_cache_descr(struct rrdengine_instance *ctx, struct page_c
31
}
32
33
/* also allocates page cache descriptor if missing */
34
+#ifdef NETDATA_INTERNAL_CHECKS
35
+void rrdeng_page_descr_mutex_lock_with_trace(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr, const char *function, size_t line)
36
+#else
37
void rrdeng_page_descr_mutex_lock(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
38
+#endif
39
{
40
unsigned long old_state, old_users, new_state, ret_state;
41
struct page_cache_descr *pg_cache_descr = NULL;
@@ -90,15 +98,27 @@ void rrdeng_page_descr_mutex_lock(struct rrdengine_instance *ctx, struct rrdeng_
98
}
99
pg_cache_descr = descr->pg_cache_descr;
100
uv_mutex_lock(&pg_cache_descr->mutex);
101
+
102
+#ifdef NETDATA_INTERNAL_CHECKS
103
+ pg_cache_descr->owner.function = function;
104
+ pg_cache_descr->owner.line = line;
105
+ pg_cache_descr->owner.tid = gettid();
106
+#endif
107
}
108
109
void rrdeng_page_descr_mutex_unlock(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr)
110
{
111
unsigned long old_state, new_state, ret_state, old_users;
98
- struct page_cache_descr *pg_cache_descr, *delete_pg_cache_descr = NULL;
112
+ struct page_cache_descr *pg_cache_descr = descr->pg_cache_descr, *delete_pg_cache_descr = NULL;
113
uint8_t we_locked;
114
101
- uv_mutex_unlock(&descr->pg_cache_descr->mutex);
115
+#ifdef NETDATA_INTERNAL_CHECKS
116
+ pg_cache_descr->owner.function = "UNLOCKED";
117
+ pg_cache_descr->owner.line = 0;
118
+ pg_cache_descr->owner.tid = 0;
119
+#endif
120
+
121
+ uv_mutex_unlock(&pg_cache_descr->mutex);
122
123
we_locked = 0;
124
while (1) { /* spin */
database/engine/rrdenglocking.h
+7
@@ -10,7 +10,14 @@ struct page_cache_descr;
10
11
struct page_cache_descr *rrdeng_create_pg_cache_descr(struct rrdengine_instance *ctx);
12
void rrdeng_destroy_pg_cache_descr(struct rrdengine_instance *ctx, struct page_cache_descr *pg_cache_descr);
13
+
14
+#ifdef NETDATA_INTERNAL_CHECKS
15
+#define rrdeng_page_descr_mutex_lock(ctx, descr) rrdeng_page_descr_mutex_lock_with_trace(ctx, descr, __FUNCTION__, __LINE__)
16
+void rrdeng_page_descr_mutex_lock_with_trace(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr, const char *function, size_t line);
17
+#else
18
void rrdeng_page_descr_mutex_lock(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
19
+#endif
20
+
21
void rrdeng_page_descr_mutex_unlock(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
22
void rrdeng_try_deallocate_pg_cache_descr(struct rrdengine_instance *ctx, struct rrdeng_page_descr *descr);
23
database/rrd.h
+12
-2
@@ -533,8 +533,9 @@ typedef enum rrdset_flags {
533
534
RRDSET_FLAG_SENDER_REPLICATION_FINISHED = (1 << 22), // the sending side has completed replication
535
RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED = (1 << 23), // the receiving side has completed replication
536
+ RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS = (1 << 24), // the receiving side has replication in progress
537
537
- RRDSET_FLAG_UPSTREAM_SEND_VARIABLES = (1 << 24), // a custom variable has been updated and needs to be exposed to parent
538
+ RRDSET_FLAG_UPSTREAM_SEND_VARIABLES = (1 << 25), // a custom variable has been updated and needs to be exposed to parent
539
} RRDSET_FLAGS;
540
541
#define rrdset_flag_check(st, flag) (__atomic_load_n(&((st)->flags), __ATOMIC_SEQ_CST) & (flag))
@@ -658,6 +659,14 @@ struct rrdset {
659
netdata_rwlock_t rwlock; // protection for RRDCALC *base
660
RRDCALC *base; // double linked list of RRDCALC related to this RRDSET
661
} alerts;
662
+
663
+#ifdef NETDATA_INTERNAL_CHECKS
664
+ struct {
665
+ bool start_streaming;
666
+ time_t after;
667
+ time_t before;
668
+ } replay;
669
+#endif
670
};
671
672
#define rrdset_plugin_name(st) string2str((st)->plugin_name)
@@ -757,6 +766,8 @@ typedef enum {
766
// Configuration options
767
RRDHOST_OPTION_DELETE_OBSOLETE_CHARTS = (1 << 3), // delete files of obsolete charts
768
RRDHOST_OPTION_DELETE_ORPHAN_HOST = (1 << 4), // delete the entire host when orphan
769
+
770
+ RRDHOST_OPTION_REPLICATION = (1 << 5), // when set, we support replication for this host
771
} RRDHOST_OPTIONS;
772
773
#define rrdhost_option_check(host, flag) ((host)->options & (flag))
@@ -937,7 +948,6 @@ struct rrdhost {
948
struct rrdpush_destinations *destination; // the current destination from the above list
949
SIMPLE_PATTERN *rrdpush_send_charts_matching; // pattern to match the charts to be sent
950
940
- bool rrdpush_enable_replication; // enable replication
951
time_t rrdpush_seconds_to_replicate; // max time we want to replicate from the child
952
time_t rrdpush_replication_step; // seconds per replication step
953
database/rrddim.c
+3
-3
@@ -64,7 +64,7 @@ static void rrddim_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, v
64
size_t entries = st->entries;
65
if(!entries) entries = 5;
66
67
- rd->db = netdata_mmap(NULL, entries * sizeof(storage_number), MAP_PRIVATE, 1);
67
+ rd->db = netdata_mmap(NULL, entries * sizeof(storage_number), MAP_PRIVATE, 1, false);
68
if(!rd->db) {
69
info("Failed to use memory mode ram for chart '%s', dimension '%s', falling back to alloc", rrdset_name(st), rrddim_name(rd));
70
ctr->memory_mode = RRD_MEMORY_MODE_ALLOC;
@@ -666,8 +666,8 @@ bool rrddim_memory_load_or_create_map_save(RRDSET *st, RRDDIM *rd, RRD_MEMORY_MO
666
rrdset_strncpyz_name(filename, rrddim_id(rd), FILENAME_MAX);
667
snprintfz(fullfilename, FILENAME_MAX, "%s/%s.db", st->cache_dir, filename);
668
669
- rd_on_file = (struct rrddim_map_save_v019 *)netdata_mmap(fullfilename, size,
670
- ((memory_mode == RRD_MEMORY_MODE_MAP) ? MAP_SHARED : MAP_PRIVATE), 1);
669
+ rd_on_file = (struct rrddim_map_save_v019 *)netdata_mmap(
670
+ fullfilename, size, ((memory_mode == RRD_MEMORY_MODE_MAP) ? MAP_SHARED : MAP_PRIVATE), 1, false);
671
672
if(unlikely(!rd_on_file)) return false;
673
database/rrdhost.c
+10
-3
@@ -290,7 +290,11 @@ int is_legacy = 1;
290
host, rrdpush_enabled, rrdpush_destination, rrdpush_api_key, rrdpush_send_charts_matching);
291
}
292
293
- host->rrdpush_enable_replication = rrdpush_enable_replication;
293
+ if(rrdpush_enable_replication)
294
+ rrdhost_option_set(host, RRDHOST_OPTION_REPLICATION);
295
+ else
296
+ rrdhost_option_clear(host, RRDHOST_OPTION_REPLICATION);
297
+
298
host->rrdpush_seconds_to_replicate = rrdpush_seconds_to_replicate;
299
host->rrdpush_replication_step = rrdpush_replication_step;
300
@@ -616,11 +620,14 @@ void rrdhost_update(RRDHOST *host
620
rrdcalctemplate_index_init(host);
621
rrdcalc_rrdhost_index_init(host);
622
619
- host->rrdpush_enable_replication = rrdpush_enable_replication;
623
+ if(rrdpush_enable_replication)
624
+ rrdhost_option_set(host, RRDHOST_OPTION_REPLICATION);
625
+ else
626
+ rrdhost_option_clear(host, RRDHOST_OPTION_REPLICATION);
627
+
628
host->rrdpush_seconds_to_replicate = rrdpush_seconds_to_replicate;
629
host->rrdpush_replication_step = rrdpush_replication_step;
630
623
-
631
rrd_hosts_available++;
632
ml_new_host(host);
633
rrdhost_load_rrdcontext_data(host);
database/rrdset.c
+1
-3
@@ -2012,9 +2012,7 @@ bool rrdset_memory_load_or_create_map_save(RRDSET *st, RRD_MEMORY_MODE memory_mo
2012
2013
unsigned long size = sizeof(struct rrdset_map_save_v019);
2014
struct rrdset_map_save_v019 *st_on_file = (struct rrdset_map_save_v019 *)netdata_mmap(
2015
- fullfilename, size,
2016
- ((memory_mode == RRD_MEMORY_MODE_MAP) ? MAP_SHARED : MAP_PRIVATE),
2017
- 0);
2015
+ fullfilename, size, ((memory_mode == RRD_MEMORY_MODE_MAP) ? MAP_SHARED : MAP_PRIVATE), 0, false);
2016
2017
if(!st_on_file) return false;
2018
libnetdata/arrayalloc/arrayalloc.c
+2
-2
@@ -206,7 +206,7 @@ static void arrayalloc_add_page(ARAL *ar) {
206
char filename[FILENAME_MAX + 1];
207
snprintfz(filename, FILENAME_MAX, "%s/array_alloc.mmap/%s.%zu", *ar->cache_dir, ar->filename, ar->internal.file_number);
208
page->filename = strdupz(filename);
209
- page->data = netdata_mmap(page->filename, page->size, MAP_SHARED, 0);
209
+ page->data = netdata_mmap(page->filename, page->size, MAP_SHARED, 0, false);
210
if (unlikely(!page->data))
211
fatal("Cannot allocate arrayalloc buffer of size %zu on filename '%s'", page->size, page->filename);
212
}
@@ -344,7 +344,7 @@ void arrayalloc_freez(ARAL *ar, void *ptr) {
344
#endif
345
}
346
347
-#ifdef NETDATA_INTERNAL_CHECKS
347
+#ifdef NETDATA_ARRAYALLOC_INTERNAL_CHECKS
348
{
349
// find the page ptr belongs
350
ARAL_PAGE *page2 = find_page_with_allocation_internal_check(ar, ptr);
libnetdata/dictionary/dictionary.c
+7
-2
@@ -2171,11 +2171,13 @@ int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const D
2171
// ----------------------------------------------------------------------------
2172
// sorted walkthrough
2173
2174
+typedef int (*qsort_compar)(const void *item1, const void *item2);
2175
+
2176
static int dictionary_sort_compar(const void *item1, const void *item2) {
2177
return strcmp(item_get_name((*(DICTIONARY_ITEM **)item1)), item_get_name((*(DICTIONARY_ITEM **)item2)));
2178
}
2179
2178
-int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *entry, void *data), void *data) {
2180
+int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *entry, void *data), void *data, dictionary_sorted_compar compar) {
2181
if(unlikely(!dict || !callback)) return 0;
2182
2183
if(unlikely(is_dictionary_destroyed(dict))) {
@@ -2200,7 +2202,10 @@ int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(
2202
if(unlikely(i != entries))
2203
entries = i;
2204
2203
- qsort(array, entries, sizeof(DICTIONARY_ITEM *), dictionary_sort_compar);
2205
+ if(compar)
2206
+ qsort(array, entries, sizeof(DICTIONARY_ITEM *), (qsort_compar)compar);
2207
+ else
2208
+ qsort(array, entries, sizeof(DICTIONARY_ITEM *), dictionary_sort_compar);
2209
2210
bool callit = true;
2211
int ret = 0, r;
libnetdata/dictionary/dictionary.h
+5
-3
@@ -230,9 +230,11 @@ size_t dictionary_acquired_item_references(DICT_ITEM_CONST DICTIONARY_ITEM *item
230
#define dictionary_walkthrough_write(dict, callback, data) dictionary_walkthrough_rw(dict, 'w', callback, data)
231
int dictionary_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *value, void *data), void *data);
232
233
-#define dictionary_sorted_walkthrough_read(dict, callback, data) dictionary_sorted_walkthrough_rw(dict, 'r', callback, data)
234
-#define dictionary_sorted_walkthrough_write(dict, callback, data) dictionary_sorted_walkthrough_rw(dict, 'w', callback, data)
235
-int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *entry, void *data), void *data);
233
+typedef int (*dictionary_sorted_compar)(const DICTIONARY_ITEM **item1, const DICTIONARY_ITEM **item2);
234
+
235
+#define dictionary_sorted_walkthrough_read(dict, callback, data) dictionary_sorted_walkthrough_rw(dict, 'r', callback, data, NULL)
236
+#define dictionary_sorted_walkthrough_write(dict, callback, data) dictionary_sorted_walkthrough_rw(dict, 'w', callback, data, NULL)
237
+int dictionary_sorted_walkthrough_rw(DICTIONARY *dict, char rw, int (*callback)(const DICTIONARY_ITEM *item, void *entry, void *data), void *data, dictionary_sorted_compar compar);
238
239
// ----------------------------------------------------------------------------
240
// Traverse with foreach
libnetdata/libnetdata.c
+3
-2
@@ -1232,7 +1232,8 @@ static inline int madvise_mergeable(void *mem, size_t len) {
1232
#endif
1233
}
1234
1235
-void *netdata_mmap(const char *filename, size_t size, int flags, int ksm) {
1235
+void *netdata_mmap(const char *filename, size_t size, int flags, int ksm, bool read_only)
1236
+{
1237
// info("netdata_mmap('%s', %zu", filename, size);
1238
1239
// MAP_SHARED is used in memory mode map
@@ -1271,7 +1272,7 @@ void *netdata_mmap(const char *filename, size_t size, int flags, int ksm) {
1272
fd_for_mmap = -1;
1273
}
1274
1274
- mem = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, fd_for_mmap, 0);
1275
+ mem = mmap(NULL, size, read_only ? PROT_READ : PROT_READ | PROT_WRITE, flags, fd_for_mmap, 0);
1276
if (mem != MAP_FAILED) {
1277
1278
#ifdef NETDATA_TRACE_ALLOCATIONS
libnetdata/libnetdata.h
+1
-1
@@ -335,7 +335,7 @@ void posix_memfree(void *ptr);
335
void json_escape_string(char *dst, const char *src, size_t size);
336
void json_fix_string(char *s);
337
338
-void *netdata_mmap(const char *filename, size_t size, int flags, int ksm);
338
+void *netdata_mmap(const char *filename, size_t size, int flags, int ksm, bool read_only);
339
int netdata_munmap(void *ptr, size_t size);
340
int memory_file_save(const char *filename, void *mem, size_t size);
341
streaming/compression.c
+1
@@ -5,6 +5,7 @@
5
6
#define STREAM_COMPRESSION_MSG "STREAM_COMPRESSION"
7
8
+// signature MUST end with a newline
9
#define SIGNATURE ((uint32_t)('z' | 0x80) | (0x80 << 8) | (0x80 << 16) | ('\n' << 24))
10
#define SIGNATURE_MASK ((uint32_t)0xff | (0x80 << 8) | (0x80 << 16) | (0xff << 24))
11
#define SIGNATURE_SIZE 4
streaming/receiver.c
+22
-3
@@ -240,8 +240,11 @@ static int receiver_read(struct receiver_state *r, FILE *fp) {
240
return 0;
241
}
242
243
+ // for compressed streams, the compression signature header ends with a new line
244
+ // so, here we read a single line from the stream.
245
+
246
int ret = 0;
244
- if (read_stream(r, fp, r->read_buffer + r->read_len, sizeof(r->read_buffer) - r->read_len - 1, &ret)) {
247
+ if (read_stream(r, fp, r->read_buffer + r->read_len, sizeof(r->read_buffer) - r->read_len, &ret)) {
248
internal_error(true, "read_stream() failed (1).");
249
return 1;
250
}
@@ -284,7 +287,7 @@ static int receiver_read(struct receiver_state *r, FILE *fp) {
287
}
288
289
// Fill read buffer with decompressed data
287
- r->read_len = r->decompressor->get(r->decompressor, r->read_buffer, sizeof(r->read_buffer));
290
+ r->read_len += (int)r->decompressor->get(r->decompressor, r->read_buffer + r->read_len, sizeof(r->read_buffer) - r->read_len);
291
return 0;
292
}
293
@@ -724,8 +727,15 @@ static int rrdpush_receive(struct receiver_state *rpt)
727
728
rrdhost_set_is_parent_label(++localhost->senders_count);
729
727
- rrdcontext_host_child_connected(rpt->host);
730
+ if(stream_has_capability(rpt->host->receiver, STREAM_CAP_REPLICATION)) {
731
+ RRDSET *st;
732
+ rrdset_foreach_read(st, rpt->host) {
733
+ rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS | RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
734
+ }
735
+ rrdset_foreach_done(st);
736
+ }
737
738
+ rrdcontext_host_child_connected(rpt->host);
739
740
rrdhost_flag_clear(rpt->host, RRDHOST_FLAG_RRDPUSH_RECEIVER_DISCONNECTED);
741
@@ -746,6 +756,15 @@ static int rrdpush_receive(struct receiver_state *rpt)
756
error("STREAM %s [receive from [%s]:%s]: disconnected (completed %zu updates).",
757
rpt->hostname, rpt->client_ip, rpt->client_port, count);
758
759
+ if(stream_has_capability(rpt->host->receiver, STREAM_CAP_REPLICATION)) {
760
+ RRDSET *st;
761
+ rrdset_foreach_read(st, rpt->host) {
762
+ rrdset_flag_clear(st, RRDSET_FLAG_RECEIVER_REPLICATION_IN_PROGRESS);
763
+ rrdset_flag_set(st, RRDSET_FLAG_RECEIVER_REPLICATION_FINISHED);
764
+ }
765
+ rrdset_foreach_done(st);
766
+ }
767
+
768
rrdcontext_host_child_disconnected(rpt->host);
769
770
#ifdef ENABLE_ACLK
streaming/replication.c
+14
-2
@@ -42,7 +42,7 @@ static time_t replicate_chart_timeframe(BUFFER *wb, RRDSET *st, time_t after, ti
42
rrddim_foreach_done(rd);
43
}
44
45
- time_t now = after, actual_after = 0, actual_before = 0;
45
+ time_t now = after + 1, actual_after = 0, actual_before = 0;
46
while(now <= before) {
47
time_t min_start_time = 0, min_end_time = 0;
48
for (size_t i = 0; i < dimensions && data[i].rd; i++) {
@@ -252,6 +252,18 @@ static bool send_replay_chart_cmd(send_command callback, void *callback_data, RR
252
}
253
#endif
254
255
+#ifdef NETDATA_INTERNAL_CHECKS
256
+ internal_error(
257
+ st->replay.after != 0 || st->replay.before != 0,
258
+ "REPLAY: host '%s', chart '%s': sending replication request, while there is another inflight",
259
+ rrdhost_hostname(st->rrdhost), rrdset_id(st)
260
+ );
261
+
262
+ st->replay.start_streaming = start_streaming;
263
+ st->replay.after = after;
264
+ st->replay.before = before;
265
+#endif
266
+
267
debug(D_REPLICATION, PLUGINSD_KEYWORD_REPLAY_CHART " \"%s\" \"%s\" %llu %llu\n",
268
rrdset_id(st), start_streaming ? "true" : "false", (unsigned long long)after, (unsigned long long)before);
269
@@ -277,7 +289,7 @@ bool replicate_chart_request(send_command callback, void *callback_data, RRDHOST
289
290
// if replication is disabled, send an empty replication request
291
// asking no data
280
- if (!host->rrdpush_enable_replication) {
292
+ if (unlikely(!rrdhost_option_check(host, RRDHOST_OPTION_REPLICATION))) {
293
internal_error(true,
294
"REPLAY: host '%s', chart '%s': sending empty replication request because replication is disabled",
295
rrdhost_hostname(host), rrdset_id(st));
streaming/rrdpush.h
+2
@@ -170,6 +170,8 @@ struct sender_state {
170
#endif
171
172
DICTIONARY *replication_requests;
173
+ time_t replication_first_time;
174
+ time_t replication_min_time;
175
};
176
177
struct receiver_state {
streaming/sender.c
+127
-57
@@ -21,9 +21,14 @@
21
#define WORKER_SENDER_JOB_BUFFER_RATIO 15
22
#define WORKER_SENDER_JOB_BYTES_RECEIVED 16
23
#define WORKER_SENDER_JOB_BYTES_SENT 17
24
-
25
-#if WORKER_UTILIZATION_MAX_JOB_TYPES < 18
26
-#error WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 18
24
+#define WORKER_SENDER_JOB_REPLAY_REQUEST 18
25
+#define WORKER_SENDER_JOB_REPLAY_RESPONSE 19
26
+#define WORKER_SENDER_JOB_REPLAY_QUEUE_SIZE 20
27
+#define WORKER_SENDER_JOB_REPLAY_COMPLETION 21
28
+#define WORKER_SENDER_JOB_FUNCTION 22
29
+
30
+#if WORKER_UTILIZATION_MAX_JOB_TYPES < 23
31
+#error WORKER_UTILIZATION_MAX_JOB_TYPES has to be at least 23
32
#endif
33
34
extern struct config stream_config;
@@ -81,6 +86,8 @@ static inline void deactivate_compression(struct sender_state *s) {
86
}
87
#endif
88
89
+#define SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE 3
90
+
91
// Collector thread finishing a transmission
92
void sender_commit(struct sender_state *s, BUFFER *wb) {
93
@@ -100,15 +107,15 @@ void sender_commit(struct sender_state *s, BUFFER *wb) {
107
108
netdata_mutex_lock(&s->mutex);
109
103
- if(unlikely(s->host->sender->buffer->max_size < (buffer_strlen(wb) + 1) * 2)) {
104
- error("STREAM %s [send to %s]: max buffer size of %zu is too small for data of size %zu. Increasing the max buffer size to twice the max data size.",
105
- rrdhost_hostname(s->host), s->connected_to, s->host->sender->buffer->max_size, buffer_strlen(wb) + 1);
110
+ if(unlikely(s->host->sender->buffer->max_size < (src_len + 1) * SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE)) {
111
+ info("STREAM %s [send to %s]: max buffer size of %zu is too small for a data message of size %zu. Increasing the max buffer size to %d times the max data message size.",
112
+ rrdhost_hostname(s->host), s->connected_to, s->host->sender->buffer->max_size, buffer_strlen(wb) + 1, SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE);
113
107
- s->host->sender->buffer->max_size = (buffer_strlen(wb) + 1) * 2;
114
+ s->host->sender->buffer->max_size = (src_len + 1) * SENDER_BUFFER_ADAPT_TO_TIMES_MAX_SIZE;
115
}
116
117
#ifdef ENABLE_COMPRESSION
111
- if (s->flags & SENDER_FLAG_COMPRESSION && s->compressor) {
118
+ if (stream_has_capability(s, STREAM_CAP_COMPRESSION) && s->compressor) {
119
while(src_len) {
120
size_t size_to_compress = src_len;
121
@@ -468,7 +475,7 @@ static bool rrdpush_sender_thread_connect_to_parent(RRDHOST *host, int default_p
475
476
#ifdef ENABLE_COMPRESSION
477
// If we don't want compression, remove it from our capabilities
471
- if(!(s->flags & SENDER_FLAG_COMPRESSION) && stream_has_capability(s, STREAM_CAP_COMPRESSION))
478
+ if(!(s->flags & SENDER_FLAG_COMPRESSION))
479
s->capabilities &= ~STREAM_CAP_COMPRESSION;
480
#endif // ENABLE_COMPRESSION
481
@@ -664,20 +671,12 @@ static bool rrdpush_sender_thread_connect_to_parent(RRDHOST *host, int default_p
671
return false;
672
673
#ifdef ENABLE_COMPRESSION
667
- // if the stream does not have compression capability,
668
- // shut it down for us too.
669
- // FIXME - this means that if there are multiple parents and one of them does not support compression
670
- // we are going to shut it down for all of them eventually...
671
- if(!stream_has_capability(s, STREAM_CAP_COMPRESSION))
672
- s->flags &= ~SENDER_FLAG_COMPRESSION;
673
-
674
- if(s->flags & SENDER_FLAG_COMPRESSION) {
675
- if(s->compressor)
674
+ if(stream_has_capability(s, STREAM_CAP_COMPRESSION)) {
675
+ if(!s->compressor)
676
+ s->compressor = create_compressor();
677
+ else
678
s->compressor->reset(s->compressor);
679
}
678
- else
679
- info("STREAM %s [send to %s]: compression is disabled on this connection.", rrdhost_hostname(host), s->connected_to);
680
-
680
#endif //ENABLE_COMPRESSION
681
682
log_sender_capabilities(s);
@@ -881,6 +880,8 @@ void execute_commands(struct sender_state *s) {
880
const char *keyword = get_word(words, num_words, 0);
881
882
if(keyword && strcmp(keyword, PLUGINSD_KEYWORD_FUNCTION) == 0) {
883
+ worker_is_busy(WORKER_SENDER_JOB_FUNCTION);
884
+
885
char *transaction = get_word(words, num_words, 1);
886
char *timeout_s = get_word(words, num_words, 2);
887
char *function = get_word(words, num_words, 3);
@@ -909,7 +910,10 @@ void execute_commands(struct sender_state *s) {
910
stream_execute_function_callback(wb, code, tmp);
911
}
912
}
912
- } else if (keyword && strcmp(keyword, PLUGINSD_KEYWORD_REPLAY_CHART) == 0) {
913
+ }
914
+ else if (keyword && strcmp(keyword, PLUGINSD_KEYWORD_REPLAY_CHART) == 0) {
915
+ worker_is_busy(WORKER_SENDER_JOB_REPLAY_REQUEST);
916
+
917
const char *chart_id = get_word(words, num_words, 1);
918
const char *start_streaming = get_word(words, num_words, 2);
919
const char *after = get_word(words, num_words, 3);
@@ -932,7 +936,8 @@ void execute_commands(struct sender_state *s) {
936
};
937
dictionary_set(s->replication_requests, chart_id, &tmp, sizeof(struct replication_request));
938
}
935
- } else {
939
+ }
940
+ else {
941
error("STREAM %s [send to %s] received unknown command over connection: %s", rrdhost_hostname(s->host), s->connected_to, words[0]?words[0]:"(unset)");
942
}
943
@@ -1045,10 +1050,12 @@ static bool replication_request_conflict_callback(const DICTIONARY_ITEM *item, v
1050
struct replication_request *rr = old_value;
1051
struct replication_request *rr_new = new_value;
1052
1048
- error("STREAM %s [send to %s]: duplicate replication command received for chart '%s' (existing from %llu to %llu [%s], new from %llu to %llu [%s])",
1049
- rrdhost_hostname(s->host), s->connected_to, dictionary_acquired_item_name(item),
1050
- (unsigned long long)rr->after, (unsigned long long)rr->before, rr->start_streaming?"true":"false",
1051
- (unsigned long long)rr_new->after, (unsigned long long)rr_new->before, rr_new->start_streaming?"true":"false");
1053
+ internal_error(
1054
+ true,
1055
+ "STREAM %s [send to %s]: duplicate replication command received for chart '%s' (existing from %llu to %llu [%s], new from %llu to %llu [%s])",
1056
+ rrdhost_hostname(s->host), s->connected_to, dictionary_acquired_item_name(item),
1057
+ (unsigned long long)rr->after, (unsigned long long)rr->before, rr->start_streaming?"true":"false",
1058
+ (unsigned long long)rr_new->after, (unsigned long long)rr_new->before, rr_new->start_streaming?"true":"false");
1059
1060
bool updated = false;
1061
@@ -1092,6 +1099,8 @@ void sender_init(RRDHOST *host)
1099
host->sender->flags |= SENDER_FLAG_COMPRESSION;
1100
host->sender->compressor = create_compressor();
1101
}
1102
+ else
1103
+ host->sender->flags &= ~SENDER_FLAG_COMPRESSION;
1104
#endif
1105
1106
netdata_mutex_init(&host->sender->mutex);
@@ -1110,45 +1119,98 @@ static size_t sender_buffer_used_percent(struct sender_state *s) {
1119
return (s->host->sender->buffer->max_size - available) * 100 / s->host->sender->buffer->max_size;
1120
}
1121
1122
+int replication_request_compar(const DICTIONARY_ITEM **item1, const DICTIONARY_ITEM **item2) {
1123
+ struct replication_request *rr1 = dictionary_acquired_item_value(*item1);
1124
+ struct replication_request *rr2 = dictionary_acquired_item_value(*item2);
1125
+
1126
+ time_t after1 = rr1->after;
1127
+ time_t after2 = rr2->after;
1128
+
1129
+ if(after1 < after2)
1130
+ return -1;
1131
+ if(after1 > after2)
1132
+ return 1;
1133
+
1134
+ return 0;
1135
+}
1136
+
1137
+int process_one_replication_request(const DICTIONARY_ITEM *item, void *value, void *data) {
1138
+ struct sender_state *s = data;
1139
+
1140
+ size_t used_percent = sender_buffer_used_percent(s);
1141
+ if(used_percent >= 50) return -1; // signal the traversal to stop
1142
+
1143
+ worker_is_busy(WORKER_SENDER_JOB_REPLAY_RESPONSE);
1144
+
1145
+ struct replication_request *rr = value;
1146
+ const char *name = dictionary_acquired_item_name(item);
1147
+
1148
+ // delete it from the dictionary
1149
+ // the current item is referenced - it will not go away until the next iteration of the dfe loop
1150
+ dictionary_del(s->replication_requests, name);
1151
+
1152
+ // find the chart
1153
+ RRDSET *st = rrdset_find(s->host, name);
1154
+ if(unlikely(!st)) {
1155
+ internal_error(true,
1156
+ "STREAM %s [send to %s]: cannot find chart '%s' to satisfy pending replication command."
1157
+ , rrdhost_hostname(s->host), s->connected_to, name);
1158
+ return 0;
1159
+ }
1160
+
1161
+ if(rr->after < s->replication_first_time || !s->replication_first_time)
1162
+ s->replication_first_time = rr->after;
1163
+
1164
+ if(rr->before < s->replication_min_time || !s->replication_min_time)
1165
+ s->replication_min_time = rr->before;
1166
+
1167
+ netdata_thread_disable_cancelability();
1168
+
1169
+ // send the replication data
1170
+ bool start_streaming = replicate_chart_response(st->rrdhost, st,
1171
+ rr->start_streaming, rr->after, rr->before);
1172
+
1173
+ netdata_thread_enable_cancelability();
1174
+
1175
+ // enable normal streaming if we have to
1176
+ if (start_streaming) {
1177
+ debug(D_REPLICATION, "Enabling metric streaming for chart %s.%s",
1178
+ rrdhost_hostname(s->host), rrdset_id(st));
1179
+
1180
+ rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
1181
+ }
1182
+
1183
+ return 1;
1184
+}
1185
+
1186
static void process_replication_requests(struct sender_state *s) {
1114
- if(dictionary_entries(s->replication_requests) == 0)
1115
- return;
1187
+ size_t entries = dictionary_entries(s->replication_requests);
1188
1117
- struct replication_request *rr;
1118
- dfe_start_write(s->replication_requests, rr) {
1119
- size_t used_percent = sender_buffer_used_percent(s);
1120
- if(used_percent > 50) break;
1121
-
1122
- // delete it from the dictionary
1123
- // the current item is referenced - it will not go away until the next iteration of the dfe loop
1124
- dictionary_del(s->replication_requests, rr_dfe.name);
1125
-
1126
- // find the chart
1127
- RRDSET *st = rrdset_find(s->host, rr_dfe.name);
1128
- if(unlikely(!st)) {
1129
- internal_error(true,
1130
- "STREAM %s [send to %s]: cannot find chart '%s' to satisfy pending replication command."
1131
- , rrdhost_hostname(s->host), s->connected_to, rr_dfe.name);
1132
- continue;
1133
- }
1189
+ worker_set_metric(WORKER_SENDER_JOB_REPLAY_QUEUE_SIZE, (NETDATA_DOUBLE)entries);
1190
1135
- netdata_thread_disable_cancelability();
1191
+ if(!entries) {
1192
+ worker_set_metric(WORKER_SENDER_JOB_REPLAY_COMPLETION, 100.0);
1193
+ return;
1194
+ }
1195
1137
- // send the replication data
1138
- bool start_streaming = replicate_chart_response(st->rrdhost, st,
1139
- rr->start_streaming, rr->after, rr->before);
1196
+ s->replication_min_time = 0;
1197
1141
- netdata_thread_enable_cancelability();
1198
+ int count = dictionary_sorted_walkthrough_rw(s->replication_requests, DICTIONARY_LOCK_WRITE,
1199
+ process_one_replication_request, s,
1200
+ replication_request_compar);
1201
1143
- // enable normal streaming if we have to
1144
- if (start_streaming) {
1145
- debug(D_REPLICATION, "Enabling metric streaming for chart %s.%s",
1146
- rrdhost_hostname(s->host), rrdset_id(st));
1202
+ if(count != 0 && s->replication_min_time && s->replication_first_time) {
1203
+ time_t now = now_realtime_sec();
1204
+ if(now > s->replication_first_time && now >= s->replication_min_time) {
1205
+ time_t completed = s->replication_min_time - s->replication_first_time;
1206
+ time_t all_duration = now - s->replication_first_time;
1207
1148
- rrdset_flag_set(st, RRDSET_FLAG_SENDER_REPLICATION_FINISHED);
1208
+ NETDATA_DOUBLE percent = (NETDATA_DOUBLE) completed * 100.0 / (NETDATA_DOUBLE) all_duration;
1209
+ worker_set_metric(WORKER_SENDER_JOB_REPLAY_COMPLETION, percent);
1210
}
1211
}
1151
- dfe_done(rr);
1212
+
1213
+ worker_is_idle();
1214
}
1215
1216
void *rrdpush_sender_thread(void *ptr) {
@@ -1171,9 +1233,17 @@ void *rrdpush_sender_thread(void *ptr) {
1233
worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_NO_COMPRESSION, "disconnect no compression");
1234
worker_register_job_name(WORKER_SENDER_JOB_DISCONNECT_BAD_HANDSHAKE, "disconnect bad handshake");
1235
1236
+ worker_register_job_name(WORKER_SENDER_JOB_REPLAY_REQUEST, "replay request");
1237
+ worker_register_job_name(WORKER_SENDER_JOB_REPLAY_RESPONSE, "replay response");
1238
+ worker_register_job_name(WORKER_SENDER_JOB_FUNCTION, "function");
1239
+
1240
worker_register_job_custom_metric(WORKER_SENDER_JOB_BUFFER_RATIO, "used buffer ratio", "%", WORKER_METRIC_ABSOLUTE);
1241
worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_RECEIVED, "bytes received", "bytes/s", WORKER_METRIC_INCREMENTAL);
1242
worker_register_job_custom_metric(WORKER_SENDER_JOB_BYTES_SENT, "bytes sent", "bytes/s", WORKER_METRIC_INCREMENTAL);
1243
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_REPLAY_COMPLETION, "replication completion", "%", WORKER_METRIC_ABSOLUTE);
1244
+ worker_register_job_custom_metric(WORKER_SENDER_JOB_REPLAY_QUEUE_SIZE, "replications pending", "commands", WORKER_METRIC_ABSOLUTE);
1245
+
1246
+ worker_set_metric(WORKER_SENDER_JOB_REPLAY_COMPLETION, 100.0);
1247
1248
struct sender_state *s = ptr;
1249
s->tid = gettid();
web/api/web_api_v1.c
+1
-1
@@ -1226,7 +1226,7 @@ inline int web_client_api_request_v1_info_fill_buffer(RRDHOST *host, BUFFER *wb)
1226
#ifdef ENABLE_COMPRESSION
1227
if(host->sender){
1228
buffer_strcat(wb, "\t\"stream-compression\": ");
1229
- buffer_strcat(wb, (host->sender->flags & SENDER_FLAG_COMPRESSION) ? "true" : "false");
1229
+ buffer_strcat(wb, stream_has_capability(host->sender, STREAM_CAP_COMPRESSION) ? "true" : "false");
1230
buffer_strcat(wb, ",\n");
1231
}else{
1232
buffer_strcat(wb, "\t\"stream-compression\": null,\n");