Reinstate systemd-journal plugin. (#21729)
* Revert "Refactor function progress: move timeout and progress authority to the plugin runtime and agent (#21723)" This reverts commit 09cbde151aaafa5d15b9340d10e656aceb4d445f. * Revert "Improve log message when deferred response is too big. (#21720)" This reverts commit 453bc9d4ad22bd7fcde41d754d6d7187957e193a. * Revert "Use info-level log for undhandled modification events. (#21719)" This reverts commit 986a187b20f866ca801b9614ae56b9d52840e45d. * Revert "Handle fields with high-cardinality and big payloads. (#21716)" This reverts commit 80a5a9f63ea05ce378cf051c809076a21f519557. * Revert "Set status flag of active journal file to archived on shutdown. (#21707)" This reverts commit c2cd26f16658e128a5785309958b2f26d3166941. * Revert "Make new journal viewer plugin a mandatory dependency for native packages to ensure clean upgrades. (#21701)" This reverts commit 3ee561b00229bdc188d8c83d87d97030700ac2b1. * Revert "Document logs functionality of otel plugin (#21705)" This reverts commit 83ac4e53cbc7903a77acb9d817a3c3d7a4fc3bac. * Revert "Update README links to correct otel-plugin paths (#21699)" This reverts commit 088c20a3f928a196124df2e772d3ff8a8d9943d2. * Revert "Add object size and bounds validation when reading journal files. (#21697)" This reverts commit e482ca8f6c1dc6d877dc1d429826f31d5cacc31e. * Revert "Handle transient systemd state flag changes. (#21695)" This reverts commit 5638b80d28bf0cfaa90f6391b332cb58ead20efc. * Revert "Keep window manager state consistent on `mmap` failures. (#21693)" This reverts commit c56d3582e5a65dbe287f2cc4ee76646e9f2211a2. * Revert "Always capture backtrace on panic. (#21681)" This reverts commit 191de6f12fccd7b360c42acdf4a2531876569680. * Revert "Use debug level for logging of requests. (#21625)" This reverts commit 2e0dd48debe9193a220d623ead640de86f275d09. * Revert "Assortement of journal-viewer plugin fixes. (#21612)" This reverts commit 8f048c8029ccf46686044ba189778d6197d718e8. * Revert "Fix 32-bit builds of journal-viewer plugin. (#21580)" This reverts commit d98adc041a201f7f13f63717bdd33df74292e65c. * Revert "Fix response when no log entries have been found. (#21581)" This reverts commit d0c06b899b92a84c4b83de2c854dcd6602b9f71f. * Revert "Proper decompression of data object payloads. (#21578)" This reverts commit 7f4ce41c142b38ccbe2a6bc0268858e38b541736. * Revert "bring back systemd-journal.plugin setcap (from source install) (#21569)" This reverts commit ad6ee9c2617c74ecb77b5539975058997c758da8. * Revert "Use host-prefixed log directories under containers. (#21568)" This reverts commit 1fac5653d541f1880c805e3ebb0720fbbd2b0e50. * Revert "Allow systemd-journal plugin if journal-viewer does not exist. (#21561)" This reverts commit 2b180ba017d777a9c9fbfc0c092b621ddde186f4. * Revert "Add sentry error reporting (#21558)" This reverts commit ffcec41d4e844d0ca5fcb08e286356857e38b1b5. * Revert "OTEL logs (#21356)" This reverts commit d0905d9b99b9c6d8bd24847b020641f9b4455f55. * Add Rust cargo workspace for journal-viewer and otel plugins alongside systemd-journal.plugin Copy the Rust crates from origin/master into src/crates/ as an independent cargo workspace. This brings back the journal-viewer plugin, otel plugin, and their supporting crates (journal-core, journal-engine, journal-index, journal-registry, journal-log-writer, netdata-plugin, etc.) without modifying the existing systemd-journal.plugin or its Rust FFI backend under src/crates/jf/. The outer workspace's Cargo.toml excludes the jf/ directory so the two workspaces remain independent: - src/crates/jf/: used exclusively by the C-based systemd-journal.plugin - src/crates/: the new cargo workspace for journal-viewer and otel plugins * Switch otel-plugin to build from src/crates/ workspace instead of src/crates/jf/ Split the corrosion_import_crate call so that each Rust workspace is imported independently: - journal_reader_ffi from src/crates/jf/ (for systemd-journal.plugin) - otel-plugin from src/crates/ (the main cargo workspace) Also add the otel-plugin rustflags for tracing_unstable and static builds, and update config install paths and docs map to point to src/crates/netdata-otel/otel-plugin/. * Add full build and packaging support for journal-viewer-plugin Add ENABLE_PLUGIN_JOURNAL_VIEWER as a new CMake option, independent of the existing ENABLE_PLUGIN_SYSTEMD_JOURNAL. Both plugins coexist without conflicts or replacements. CMake (CMakeLists.txt): - New ENABLE_PLUGIN_JOURNAL_VIEWER option (default: ON) - Import journal-viewer-plugin from the main src/crates/ workspace via Corrosion, alongside otel-plugin - Rustflags: --cfg=tracing_unstable, io_uring_skip_arch_check on 32-bit, crt-static for static builds - Install target and config file processing for journal-viewer.yaml Packaging: - Debian: new netdata-plugin-journal-viewer package with postinst for cap_dac_read_search capability (Packaging.cmake, postinst) - RPM: new journal-viewer subpackage with caps and file entries (netdata.spec.in) Installer and build entry points: - netdata-installer.sh: --enable/--disable-plugin-journal-viewer CLI flags and setcap/permissions block for the plugin binary - packaging/installer/functions.sh: PLUGIN_JOURNAL_VIEWER CMake feature - packaging/docker/Dockerfile: --enable-plugin-journal-viewer in build - packaging/makeself/jobs/70-netdata-git.install.sh: enable for all architectures except armv6l The plugin defaults to disabled in the installer (like otel) and is explicitly enabled in Docker and static/makeself builds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Rename journal-viewer-plugin to otel-signal-viewer-plugin The journal-viewer name was confusing alongside systemd-journal.plugin since both implied journal log viewing. Rename to otel-signal-viewer to reflect the plugin's actual purpose: viewing OTel signals (logs now, traces in the future). Renamed across all layers: - Rust crate: package name, binary name, directory, config file - CMake: ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER option, Corrosion targets, rustflags, install targets, config processing - RPM: subpackage, file lists, exclusions - Debian: package definition, postinst directory - Installer: CLI flags (--enable/--disable-plugin-otel-signal-viewer), variable (ENABLE_OTEL_SIGNAL_VIEWER), permissions block - Docker: installer flag, chmod loop - Makeself: static build flags - Rust source: function name, chart contexts, plugin runtime name, config filename references - Documentation: map.yaml label, crate README * Fix installer flag * Update docs/.map/map.yaml Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * Remove duplicate source entry * Guard Recommends tag against old RPM on Amazon Linux 2 * Disable otel-signal-viewer plugin on Windows. * Bring back function cancellation on timeout and useful diagnostic. * Reference correct plugin in README.md * Remove RUSTFLAGS from static builds. * Update function description. * Update otel-signal-viewer configuration file. * Replace runtime relative path resolution with CMake template substitution. Both the otel and otel-signal-viewer plugins used a resolve_relative_path() helper to turn relative config paths into absolute ones at runtime, using Netdata environment variables as the base directory. The otel-signal-viewer plugin already had a .yaml.in template with CMake variables for its paths, making its resolve_relative_path() call redundant (the paths were already absolute after CMake substitution). Unify both plugins on the CMake approach: - Convert otel.yaml to otel.yaml.in with @configdir_POST@ and @logdir_POST@ so paths are absolute at install time. - Add configure_file() in CMakeLists.txt for the otel plugin config (matching the existing pattern used by otel-signal-viewer). - Remove resolve_relative_path() from both plugins since all stock config paths are now absolute after CMake processing. User-edited configs must use absolute paths going forward, otherwise relative paths will be resolved relative to the plugin's runtime path. * Address packaging review comments. - Restore BUILD_ARCH ARG in Dockerfile for correct arch in cross-build CI. - Deduplicate NETDATA_CMAKE_OPTIONS in makeself static build script. - Add Obsoletes for journal-viewer in RPM otel-signal-viewer package. - Add Replaces/Conflicts for journal-viewer in DEB otel-signal-viewer package. * Finish restoring BUILD_ARCH for Docker images. * Fix up dependencies for native packages properly. In reality we actually need to use a transitional package to migrate users back to the journal viewer instead of obsoleting the old package. * Make the signal viewer plugin a mandatory dependency of the OTel plugin. * Address Cubic review comments. * Fix DEB package dependencies. * Update src/collectors/systemd-journal.plugin/passive_journal_centralization_guide_no_encryption.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Revert "Remove RUSTFLAGS from static builds." This reverts commit 3ff5ae308a02632fe131299b3abf0a9e91ecbb5b. The per-target corrosion_add_target_rustflags() for journal_reader_ffi does not work for static builds on Alpine/musl because Corrosion determines native-static-libs during CMake configure time, before target-specific rustflags are applied. Without crt-static during that extraction, Rust reports -lgcc_s as a required native library, but Alpine doesn't ship libgcc_s.a (only libgcc_s.so). The -static linker flag then fails to find it. The global RUSTFLAGS export in 70-netdata-git.install.sh is set before CMake runs, so Corrosion's configure-time execute_process inherits it from the environment and correctly reports -lgcc_eh (which exists as a static archive) instead of -lgcc_s. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix static build permissions for otel plugins. The static build post-installer was missing otel-plugin and otel-signal-viewer-plugin from the ownership/permissions setup, causing the netdata user to get "Permission denied" when trying to execute them. The DEB packages handled this correctly via their postinst scripts, but the static build equivalent in install-or-update.sh was never updated when these plugins were added. * Fix missing chmod 4750 fallback for otel-signal-viewer-plugin when setcap is unavailable. The else branch (no setcap) was missing the fallback chmod for otel-signal-viewer-plugin, so it would lack needed privileges on systems without setcap. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> Co-authored-by: Austin S. Hemmelgarn <austin@netdata.cloud> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
vkalintiris committed
Feb 11, 2026 at 05:44 UTC
1c0787be6b109087f6f10cbc2acf7352fdc4ca76
111 files changed
+19204
-344
.github/labeler.yml
+1
-2
@@ -195,8 +195,7 @@ collectors/systemd-journal:
195
- any:
196
- changed-files:
197
- any-glob-to-any-file:
198
- - src/crates/netdata-log-viewer/**
199
- - docs/logs/**
198
+ - src/collectors/systemd-journal.plugin/**
199
200
collectors/tc:
201
- any:
CMakeLists.txt
+110
-42
@@ -132,7 +132,7 @@ mark_as_advanced(ENABLE_DASHBOARD)
132
# Data collection plugins
133
option(ENABLE_PLUGIN_GO "Enable metric collectors written in Go" ${DEFAULT_FEATURE_STATE})
134
option(ENABLE_PLUGIN_OTEL "Enable collection of OpenTelemetry metrics and logs" ${DEFAULT_FEATURE_STATE})
135
-option(ENABLE_NETDATA_JOURNAL_FILE_READER "Enable journal viewer plugin" ${DEFAULT_FEATURE_STATE})
135
+cmake_dependent_option(ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER "Enable OTel signal viewer plugin" ${DEFAULT_FEATURE_STATE} "NOT OS_WINDOWS" False)
136
option(ENABLE_PLUGIN_PYTHON "Enable metric collectors written in Python" ${DEFAULT_FEATURE_STATE})
137
138
cmake_dependent_option(ENABLE_PLUGIN_APPS "Enable per-process resource usage monitoring" ${DEFAULT_FEATURE_STATE} "OS_LINUX OR OS_FREEBSD OR OS_MACOS OR OS_WINDOWS" False)
@@ -151,6 +151,7 @@ cmake_dependent_option(ENABLE_PLUGIN_NETWORK_VIEWER "Enable network viewer funct
151
cmake_dependent_option(ENABLE_PLUGIN_NFACCT "Enable Linux NFACCT metric collection" ${DEFAULT_FEATURE_STATE} "OS_LINUX" False)
152
cmake_dependent_option(ENABLE_PLUGIN_PERF "Enable Linux performance counter monitoring" ${DEFAULT_FEATURE_STATE} "OS_LINUX" False)
153
cmake_dependent_option(ENABLE_PLUGIN_SLABINFO "Enable Linux kernel SLAB allocator monitoring" ${DEFAULT_FEATURE_STATE} "OS_LINUX" False)
154
+cmake_dependent_option(ENABLE_PLUGIN_SYSTEMD_JOURNAL "Enable systemd journal log collection" ${DEFAULT_FEATURE_STATE} "OS_LINUX" False)
155
cmake_dependent_option(ENABLE_PLUGIN_SYSTEMD_UNITS "Enable systemd units information collection" ${DEFAULT_FEATURE_STATE} "OS_LINUX" False)
156
cmake_dependent_option(ENABLE_PLUGIN_XENSTAT "Enable Xen domain monitoring" ${DEFAULT_FEATURE_STATE} "OS_LINUX" False)
157
@@ -184,9 +185,10 @@ mark_as_advanced(ENABLE_LIBUNWIND)
185
cmake_dependent_option(FORCE_LEGACY_LIBBPF "Force usage of libbpf 0.0.9 instead of the latest version." False "ENABLE_PLUGIN_EBPF" False)
186
mark_as_advanced(FORCE_LEGACY_LIBBPF)
187
188
+cmake_dependent_option(ENABLE_NETDATA_JOURNAL_FILE_READER "Enable netdata's journal file reader implementation" False "ENABLE_PLUGIN_SYSTEMD_JOURNAL" False)
189
190
# Setup Rust/Corrosion for plugins that need it
189
-if(ENABLE_NETDATA_JOURNAL_FILE_READER OR ENABLE_PLUGIN_OTEL)
191
+if(ENABLE_NETDATA_JOURNAL_FILE_READER OR ENABLE_PLUGIN_OTEL OR ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER)
192
include(FetchContent)
193
FetchContent_Declare(
194
Corrosion
@@ -195,41 +197,48 @@ if(ENABLE_NETDATA_JOURNAL_FILE_READER OR ENABLE_PLUGIN_OTEL)
197
)
198
FetchContent_MakeAvailable(Corrosion)
199
198
- if(ENABLE_NETDATA_JOURNAL_FILE_READER AND ENABLE_PLUGIN_OTEL)
199
- corrosion_import_crate(MANIFEST_PATH src/crates/Cargo.toml
200
- CRATES journal-viewer-plugin otel-plugin)
201
- elseif(ENABLE_NETDATA_JOURNAL_FILE_READER)
202
- corrosion_import_crate(MANIFEST_PATH src/crates/Cargo.toml
203
- CRATES journal-viewer-plugin)
204
- elseif(ENABLE_PLUGIN_OTEL)
200
+ if(ENABLE_NETDATA_JOURNAL_FILE_READER)
201
+ corrosion_import_crate(MANIFEST_PATH src/crates/jf/Cargo.toml
202
+ CRATES journal_reader_ffi)
203
+ endif()
204
+
205
+ # Import crates from the main cargo workspace
206
+ set(_WORKSPACE_CRATES "")
207
+ if(ENABLE_PLUGIN_OTEL)
208
+ list(APPEND _WORKSPACE_CRATES otel-plugin)
209
+ endif()
210
+ if(ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER)
211
+ list(APPEND _WORKSPACE_CRATES otel-signal-viewer-plugin)
212
+ endif()
213
+ if(_WORKSPACE_CRATES)
214
corrosion_import_crate(MANIFEST_PATH src/crates/Cargo.toml
206
- CRATES otel-plugin)
215
+ CRATES ${_WORKSPACE_CRATES})
216
endif()
217
209
- if(ENABLE_NETDATA_JOURNAL_FILE_READER)
210
- set(JOURNAL_VIEWER_RUSTFLAGS --cfg=tracing_unstable)
218
+ if(ENABLE_PLUGIN_OTEL)
219
+ if(STATIC_BUILD)
220
+ corrosion_add_target_rustflags(otel-plugin --cfg=tracing_unstable "-C" "target-feature=+crt-static")
221
+ else()
222
+ corrosion_add_target_rustflags(otel-plugin --cfg=tracing_unstable)
223
+ endif()
224
+ endif()
225
+
226
+ if(ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER)
227
+ set(OTEL_SIGNAL_VIEWER_RUSTFLAGS --cfg=tracing_unstable)
228
229
# We depend (transitively) on the `io-uring` crate which doesn't provide
230
# prebuilt bindings for 32-bit arches. Considering this is a compile-time
231
# and not a runtime dependency (because we don't use foyer's io-uring
232
# engine), we can simply skip the check.
233
if(CMAKE_SIZEOF_VOID_P EQUAL 4 AND CMAKE_SYSTEM_NAME STREQUAL "Linux")
217
- list(APPEND JOURNAL_VIEWER_RUSTFLAGS --cfg=io_uring_skip_arch_check)
234
+ list(APPEND OTEL_SIGNAL_VIEWER_RUSTFLAGS --cfg=io_uring_skip_arch_check)
235
endif()
236
237
if(STATIC_BUILD)
221
- list(APPEND JOURNAL_VIEWER_RUSTFLAGS "-C" "target-feature=+crt-static")
238
+ list(APPEND OTEL_SIGNAL_VIEWER_RUSTFLAGS "-C" "target-feature=+crt-static")
239
endif()
240
224
- corrosion_add_target_rustflags(journal-viewer-plugin ${JOURNAL_VIEWER_RUSTFLAGS})
225
- endif()
226
-
227
- if(ENABLE_PLUGIN_OTEL)
228
- if(STATIC_BUILD)
229
- corrosion_add_target_rustflags(otel-plugin --cfg=tracing_unstable "-C" "target-feature=+crt-static")
230
- else()
231
- corrosion_add_target_rustflags(otel-plugin --cfg=tracing_unstable)
232
- endif()
241
+ corrosion_add_target_rustflags(otel-signal-viewer-plugin ${OTEL_SIGNAL_VIEWER_RUSTFLAGS})
242
endif()
243
endif()
244
@@ -1708,6 +1717,22 @@ set(STATSD_PLUGIN_FILES
1717
src/collectors/statsd.plugin/statsd.c
1718
)
1719
1720
+set(SYSTEMD_JOURNAL_PLUGIN_FILES
1721
+ src/collectors/systemd-journal.plugin/systemd-journal-fstat.c
1722
+ src/collectors/systemd-journal.plugin/systemd-internals.h
1723
+ src/collectors/systemd-journal.plugin/systemd-main.c
1724
+ src/collectors/systemd-journal.plugin/systemd-journal.c
1725
+ src/collectors/systemd-journal.plugin/systemd-journal-annotations.c
1726
+ src/collectors/systemd-journal.plugin/systemd-journal-files.c
1727
+ src/collectors/systemd-journal.plugin/systemd-journal-watcher.c
1728
+ src/collectors/systemd-journal.plugin/systemd-journal-dyncfg.c
1729
+ src/collectors/systemd-journal.plugin/provider/netdata_provider.c
1730
+ src/collectors/systemd-journal.plugin/provider/netdata_provider.h
1731
+ src/collectors/systemd-journal.plugin/provider/rust_provider.h
1732
+ src/libnetdata/os/system-maps/system-services.h
1733
+ src/collectors/systemd-journal.plugin/systemd-journal-sampling.h
1734
+)
1735
+
1736
set(SYSTEMD_UNITS_PLUGIN_FILES
1737
src/collectors/systemd-units.plugin/plugin_systemd_units.c
1738
)
@@ -2758,6 +2783,36 @@ if(ENABLE_PLUGIN_CGROUP_NETWORK)
2783
DESTINATION usr/libexec/netdata/plugins.d)
2784
endif()
2785
2786
+# Enable rust implementation if we don't have systemd and we want the journal plugin
2787
+if(ENABLE_PLUGIN_SYSTEMD_JOURNAL AND NOT SYSTEMD_FOUND)
2788
+ if (NOT ENABLE_NETDATA_JOURNAL_FILE_READER)
2789
+ message(WARNING "Systemd journal package not found, will try netdata's journal reader which requires cargo.")
2790
+ set(ENABLE_NETDATA_JOURNAL_FILE_READER True)
2791
+ endif()
2792
+endif()
2793
+
2794
+if(ENABLE_PLUGIN_SYSTEMD_JOURNAL)
2795
+ add_executable(systemd-journal.plugin ${SYSTEMD_JOURNAL_PLUGIN_FILES})
2796
+
2797
+ if(ENABLE_NETDATA_JOURNAL_FILE_READER)
2798
+ target_compile_definitions(systemd-journal.plugin PRIVATE HAVE_RUST_PROVIDER)
2799
+ target_link_libraries(systemd-journal.plugin journal_reader_ffi)
2800
+ endif()
2801
+
2802
+ target_link_libraries(systemd-journal.plugin libnetdata)
2803
+
2804
+ install(TARGETS systemd-journal.plugin
2805
+ COMPONENT plugin-systemd-journal
2806
+ DESTINATION usr/libexec/netdata/plugins.d)
2807
+
2808
+ if(BUILD_FOR_PACKAGING)
2809
+ install(FILES
2810
+ ${PKG_FILES_PATH}/copyright
2811
+ COMPONENT plugin-systemd-journal
2812
+ DESTINATION usr/share/doc/netdata-plugin-systemd-journal)
2813
+ endif()
2814
+endif()
2815
+
2816
if(ENABLE_PLUGIN_SYSTEMD_UNITS)
2817
if(NOT SYSTEMD_FOUND)
2818
message(FATAL_ERROR "Systemd units plugin requires systemd, but systemd was not found.")
@@ -3225,10 +3280,6 @@ if(ENABLE_PLUGIN_OTEL)
3280
RUNTIME DESTINATION usr/libexec/netdata/plugins.d
3281
COMPONENT plugin-otel)
3282
3228
- install(FILES src/crates/netdata-otel/otel-plugin/configs/otel.yaml
3229
- COMPONENT plugin-otel
3230
- DESTINATION usr/lib/netdata/conf.d)
3231
-
3283
install(FILES src/crates/netdata-otel/otel-plugin/configs/otel.d/v1/metrics/hostmetrics-receiver.yaml
3284
COMPONENT plugin-otel
3285
DESTINATION usr/lib/netdata/conf.d/otel.d/v1/metrics)
@@ -3237,14 +3288,14 @@ if(ENABLE_PLUGIN_OTEL)
3288
endif()
3289
3290
#
3240
-# journal-viewer-plugin
3291
+# otel-signal-viewer
3292
#
3242
-# Handle journal-viewer plugin
3243
-if(ENABLE_NETDATA_JOURNAL_FILE_READER)
3244
- corrosion_install(TARGETS journal-viewer-plugin
3293
+# Handle OTel signal viewer plugin
3294
+if(ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER)
3295
+ corrosion_install(TARGETS otel-signal-viewer-plugin
3296
PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE
3297
RUNTIME DESTINATION usr/libexec/netdata/plugins.d
3247
- COMPONENT plugin-journal-viewer)
3298
+ COMPONENT plugin-otel-signal-viewer)
3299
endif()
3300
3301
#
@@ -3321,16 +3372,6 @@ set(logdir_POST "${LOG_DIR}")
3372
set(netdata_user_POST "${NETDATA_USER}")
3373
set(netdata_group_POST "${NETDATA_USER}")
3374
3324
-if(ENABLE_NETDATA_JOURNAL_FILE_READER)
3325
- configure_file(src/crates/netdata-log-viewer/journal-viewer-plugin/configs/journal-viewer.yaml.in
3326
- src/crates/netdata-log-viewer/journal-viewer-plugin/configs/journal-viewer.yaml
3327
- @ONLY)
3328
-
3329
- install(FILES ${CMAKE_BINARY_DIR}/src/crates/netdata-log-viewer/journal-viewer-plugin/configs/journal-viewer.yaml
3330
- COMPONENT plugin-journal-viewer
3331
- DESTINATION usr/lib/netdata/conf.d)
3332
-endif()
3333
-
3375
if(NOT OS_WINDOWS)
3376
configure_file(src/claim/netdata-claim.sh.in src/claim/netdata-claim.sh @ONLY)
3377
install(PROGRAMS
@@ -3362,6 +3403,32 @@ install(PROGRAMS
3403
COMPONENT netdata
3404
DESTINATION usr/libexec/netdata/plugins.d)
3405
3406
+#
3407
+# otel config
3408
+#
3409
+if(ENABLE_PLUGIN_OTEL)
3410
+ configure_file(src/crates/netdata-otel/otel-plugin/configs/otel.yaml.in
3411
+ src/crates/netdata-otel/otel-plugin/configs/otel.yaml
3412
+ @ONLY)
3413
+
3414
+ install(FILES ${CMAKE_BINARY_DIR}/src/crates/netdata-otel/otel-plugin/configs/otel.yaml
3415
+ COMPONENT plugin-otel
3416
+ DESTINATION usr/lib/netdata/conf.d)
3417
+endif()
3418
+
3419
+#
3420
+# otel-signal-viewer config
3421
+#
3422
+if(ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER)
3423
+ configure_file(src/crates/netdata-log-viewer/otel-signal-viewer-plugin/configs/otel-signal-viewer.yaml.in
3424
+ src/crates/netdata-log-viewer/otel-signal-viewer-plugin/configs/otel-signal-viewer.yaml
3425
+ @ONLY)
3426
+
3427
+ install(FILES ${CMAKE_BINARY_DIR}/src/crates/netdata-log-viewer/otel-signal-viewer-plugin/configs/otel-signal-viewer.yaml
3428
+ COMPONENT plugin-otel-signal-viewer
3429
+ DESTINATION usr/lib/netdata/conf.d)
3430
+endif()
3431
+
3432
#
3433
# statsd
3434
#
@@ -3749,6 +3816,7 @@ endif()
3816
3817
# confs
3818
install(FILES
3819
+ src/collectors/systemd-journal.plugin/schema.d/systemd-journal%3Amonitored-directories.json
3820
src/health/schema.d/health%3Aalert%3Aprototype.json
3821
COMPONENT netdata
3822
DESTINATION usr/lib/netdata/conf.d/schema.d)
docs/.map/map.csv
+3
-3
@@ -139,15 +139,15 @@ https://github.com/netdata/netdata/edit/master/src/web/api/exporters/shell/READM
139
,,,,,
140
,,,,,
141
logs_integrations,,,,,
142
-https://github.com/netdata/netdata/edit/master/docs/logs/README.md,Journal Viewer Plugin,Published,Logs/Systemd Journal Logs,,View and analyze logs available in systemd journal
143
-https://github.com/netdata/netdata/edit/master/docs/logs/forward_secure_sealing.md,Forward Secure Sealing (FSS) in Systemd-Journal,Published,Logs/Systemd Journal Logs,,
142
+https://github.com/netdata/netdata/edit/master/src/collectors/systemd-journal.plugin/README.md,Systemd Journal Plugin Reference,Published,Logs/Systemd Journal Logs,,View and analyze logs available in systemd journal
143
+https://github.com/netdata/netdata/edit/master/src/collectors/systemd-journal.plugin/forward_secure_sealing.md,Forward Secure Sealing (FSS) in Systemd-Journal,Published,Logs/Systemd Journal Logs,,
144
https://github.com/netdata/netdata/edit/master/src/collectors/windows-events.plugin/README.md,Windows Events Plugin Reference,Published,Logs/Windows Event Logs,,
145
https://github.com/netdata/netdata/edit/master/src/collectors/log2journal/README.md,log2journal,Published,Logs/log2journal,,
146
https://github.com/netdata/netdata/edit/master/src/libnetdata/log/systemd-cat-native.md,systemd-cat-native,Published,Logs/systemd-cat-native,,
147
https://github.com/netdata/netdata/edit/master/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/README.md,Logs Centralization Points with systemd-journald,Published,Logs/Logs Centralization Points with systemd-journald,,
148
https://github.com/netdata/netdata/edit/master/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/passive-journal-centralization-with-encryption-using-self-signed-certificates.md,Passive journal centralization with encryption using self-signed certificates,Published,Logs/Logs Centralization Points with systemd-journald,,
149
https://github.com/netdata/netdata/edit/master/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/passive-journal-centralization-without-encryption.md,Passive journal centralization without encryption,Published,Logs/Logs Centralization Points with systemd-journald,,
150
-https://github.com/netdata/netdata/edit/master/docs/logs/active_journal_centralization_guide_no_encryption.md,Active journal source without encryption,Published,Logs/Logs Centralization Points with systemd-journald,,
150
+https://github.com/netdata/netdata/edit/master/src/collectors/systemd-journal.plugin/active_journal_centralization_guide_no_encryption.md,Active journal source without encryption,Published,Logs/Logs Centralization Points with systemd-journald,,
151
,,,,,
152
,,,,,
153
https://github.com/netdata/netdata/edit/master/docs/top-monitoring-netdata-functions.md,Top Consumers,Published,Top Consumers,,Present the Netdata Functions what these are and why they should be used.
docs/category-overview-pages/working-with-logs.md
+1
-1
@@ -2,7 +2,7 @@
2
3
This section talks about the ways Netdata collects and visualizes logs.
4
5
-The [journal viewer plugin](/docs/logs/README.md) is the core Netdata component for reading systemd journal logs.
5
+The [systemd journal plugin](/src/collectors/systemd-journal.plugin) is the core Netdata component for reading systemd journal logs.
6
7
For structured logs, Netdata provides tools like [log2journal](/src/collectors/log2journal/README.md) and [systemd-cat-native](/src/libnetdata/log/systemd-cat-native.md) to convert them into compatible systemd journal entries.
8
docs/dashboards-and-charts/logs-tab.md
+8
-8
@@ -1,16 +1,16 @@
1
# Logs tab
2
3
-The Logs tab is using the [`journal-viewer` plugin](/docs/logs/README.md), to present a structured view into your infrastructure's `systemd` logs.
3
+The Logs tab is using the [`systemd` journal plugin](/src/collectors/systemd-journal.plugin/README.md), to present a structured view into your infrastructure's `systemd` logs.
4
5
We have a thorough section explaining how you can [work with logs](https://learn.netdata.cloud/docs/logs), detailing how the plugin works, and what other utilities are used under the hood to provide you with the visualizations and the log entries.
6
7
-The [`journal-viewer` plugin](/docs/logs/README.md) documentation has information about:
7
+The [`systemd` journal plugin](/src/collectors/systemd-journal.plugin/README.md) documentation has information about:
8
9
-- [Key features the plugin provides](/docs/logs/README.md#key-features)
10
-- [Journal sources](/docs/logs/README.md#journal-sources)
11
-- [Journal fields](/docs/logs/README.md#journal-fields)
12
-- [Full-text search](/docs/logs/README.md#full-text-search)
13
-- [Query performance](/docs/logs/README.md#query-performance)
14
-- [Performance at scale](/docs/logs/README.md#performance-at-scale)
9
+- [Key features the plugin provides](/src/collectors/systemd-journal.plugin/README.md#key-features)
10
+- [Journal sources](/src/collectors/systemd-journal.plugin/README.md#journal-sources)
11
+- [Journal fields](/src/collectors/systemd-journal.plugin/README.md#journal-fields)
12
+- [Full-text search](/src/collectors/systemd-journal.plugin/README.md#full-text-search)
13
+- [Query performance](/src/collectors/systemd-journal.plugin/README.md#query-performance)
14
+- [Performance at scale](/src/collectors/systemd-journal.plugin/README.md#performance-at-scale)
15
16
We recommend you to read through that document, to better understand how the plugin and the visualizations work.
docs/developer-and-contributor-corner/dyncfg.md
+5
-4
@@ -150,12 +150,13 @@ The health module manages alert definitions through DynCfg:
150
</details>
151
152
<details>
153
-<summary><strong>journal-viewer-plugin (External Plugin, Rust)</strong></summary><br/>
153
+<summary><strong>systemd-journal.plugin (External Plugin, C)</strong></summary><br/>
154
155
-External Rust plugin that provides systemd journal log viewing and analysis:
155
+External C plugin that manages journal directory configurations:
156
157
-- **Location**: `src/crates/netdata-log-viewer/journal-viewer-plugin/`
158
-- **Use Case**: Viewing and analyzing systemd journal logs
157
+- **File**: `src/collectors/systemd-journal.plugin/systemd-journal-dyncfg.c`
158
+- **Pattern**: SINGLE configuration type
159
+- **Use Case**: Managing journal directory paths
160
161
<br/>
162
</details>
docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/passive-journal-centralization-with-encryption-using-self-signed-certificates.md
+2
-2
@@ -20,7 +20,7 @@ This means that if both certificates are issued by the same certificate authorit
20
21
## Self-signed certificates
22
23
-To simplify the process of creating and managing self-signed certificates, we have created [this bash script](https://github.com/netdata/netdata/blob/master/docs/logs/systemd-journal-self-signed-certs.sh).
23
+To simplify the process of creating and managing self-signed certificates, we have created [this bash script](https://github.com/netdata/netdata/blob/master/src/collectors/systemd-journal.plugin/systemd-journal-self-signed-certs.sh).
24
25
This helps to also automate the distribution of the certificates to your servers (it generates a new bash script for each of your servers, which includes everything required, including the certificates).
26
@@ -34,7 +34,7 @@ On the server that will issue the certificates (usually the centralization serve
34
sudo apt-get install systemd-journal-remote openssl
35
36
# download the script and make it executable
37
-curl >systemd-journal-self-signed-certs.sh "https://raw.githubusercontent.com/netdata/netdata/master/docs/logs/systemd-journal-self-signed-certs.sh"
37
+curl >systemd-journal-self-signed-certs.sh "https://raw.githubusercontent.com/netdata/netdata/master/src/collectors/systemd-journal.plugin/systemd-journal-self-signed-certs.sh"
38
chmod 750 systemd-journal-self-signed-certs.sh
39
```
40
docs/security-and-privacy-design/netdata-kubernetes.md
+1
-1
@@ -26,7 +26,7 @@ This architecture ensures that the components responsible for broader network ex
26
Even within the `child` agent container, which might be granted elevated permissions by Kubernetes (e.g., host mounts, capabilities), Netdata enforces strict internal privilege separation:
27
28
- **Unprivileged Core Daemon:** The main `netdata` daemon process, which orchestrates data collection, communicates with the parent, and handles basic tasks, runs as a **non-root user** (typically the `netdata` user). Key internal plugins responsible for collecting core metrics, such as `proc.plugin` (for `/proc` based metrics) and `cgroups.plugin` (for container metrics via `/sys/fs/cgroup`), also run as threads within this unprivileged daemon process.
29
-- **Dedicated Privileged Helpers/Plugins:** Operations that require elevated privileges or specific capabilities to access protected host resources (e.g., detailed process information via `apps.plugin`, certain network operations via `network-viewer.plugin` or `cgroup-network`, logs querying via `journal-viewer-plugin`) are delegated to **dedicated external plugins or helper processes**.
29
+- **Dedicated Privileged Helpers/Plugins:** Operations that require elevated privileges or specific capabilities to access protected host resources (e.g., detailed process information via `apps.plugin`, certain network operations via `network-viewer.plugin` or `cgroup-network`, logs querying via `systemd-journal.plugin`) are delegated to **dedicated external plugins or helper processes**.
30
- **Isolation via `setuid`:** These external plugins/helpers run as separate, isolated processes spawned by the main daemon. Within the container, the specific helper executables requiring elevated rights are configured with the **`setuid` root** permission bit. This standard Linux mechanism allows only these designated programs to execute specific, pre-defined tasks with root privileges _within the container's environment_.
31
- **Contained Capabilities:** Consequently, even if the _container_ is granted capabilities like `SYS_ADMIN` or `SYS_PTRACE`, the unprivileged main `netdata` daemon and its non-`setuid` plugins **cannot** directly utilize these capabilities. Only the specific `setuid` root helpers designed for tasks needing those privileges can effectively leverage them.
32
- **Narrowly Scoped Tasks:** Each privileged helper is designed to perform a narrow, hard-coded task – typically reading specific system information and passing it back to the main daemon.
docs/top-monitoring-netdata-functions.md
+2
-2
@@ -17,8 +17,8 @@ Beyond their primary roles of collecting metrics, collectors can execute specifi
17
| Network-connections | Real-time monitoring of all network connections, showing established connections, ports, protocols, and connection states across TCP/UDP services. | `netstat`, `ss` | yes | [network-viewer](https://github.com/netdata/netdata/tree/master/src/collectors/network-viewer.plugin) |
18
| Network-interfaces | Network traffic, packet drop rates, interface states, MTU, speed, and duplex mode for all network interfaces. | `bmon`, `bwm-ng` | no | [proc](https://github.com/netdata/netdata/tree/master/src/collectors/proc.plugin#readme) |
19
| Processes | Real-time information about the system's resource usage, including CPU utilization, memory consumption, and disk IO for every running process. | `top`, `htop` | yes | [apps](/src/collectors/apps.plugin/README.md) |
20
-| Systemd-journal | Viewing, exploring and analyzing systemd journal logs. | `journalctl` | yes | [journal-viewer](/docs/logs/README.md) |
21
-| Systemd-list-units | Information about all systemd units, including their active state, description, whether or not they are enabled, and more. | `systemctl list-units` | yes | [journal-viewer](/docs/logs/README.md) |
20
+| Systemd-journal | Viewing, exploring and analyzing systemd journal logs. | `journalctl` | yes | [systemd-journal](https://github.com/netdata/netdata/tree/master/src/collectors/systemd-journal.plugin#readme) |
21
+| Systemd-list-units | Information about all systemd units, including their active state, description, whether or not they are enabled, and more. | `systemctl list-units` | yes | [systemd-journal](https://github.com/netdata/netdata/tree/master/src/collectors/systemd-journal.plugin#readme) |
22
| Systemd-services | System resource utilization for all running systemd services: CPU, memory, and disk IO. | `systemd-cgtop` | no | [cgroups](https://github.com/netdata/netdata/tree/master/src/collectors/cgroups.plugin#readme) |
23
| Netdata-api-calls | Real-time tracing of API calls made to the Netdata Agent. It provides information on query, source, status, elapsed time, and more. | | yes | |
24
| Netdata-streaming | Comprehensive overview of all Netdata children instances, offering detailed information about their status, replication completion time, and many more. | | yes | |
integrations/logs/metadata.yaml
+1
-1
@@ -3,7 +3,7 @@
3
- id: "logs-systemd-journal"
4
meta:
5
name: "Systemd Journal Logs"
6
- link: "https://github.com/netdata/netdata/blob/master/docs/logs/README.md"
6
+ link: "https://github.com/netdata/netdata/blob/master/src/collectors/systemd-journal.plugin/README.md"
7
categories:
8
- logs
9
icon_filename: "netdata.png"
netdata-installer.sh
+14
-9
@@ -213,10 +213,13 @@ USAGE: ${PROGRAM} [options]
213
--disable-plugin-nfacct Explicitly disable the nfacct plugin.
214
--enable-plugin-xenstat Enable the xenstat plugin. Default: enable it when libxenstat and libyajl are available.
215
--disable-plugin-xenstat Explicitly disable the xenstat plugin.
216
+ --enable-plugin-systemd-journal Enable the systemd journal plugin. Default: enable it when libsystemd is available.
217
+ --disable-plugin-systemd-journal Explicitly disable the systemd journal plugin.
218
+ --internal-systemd-journal Enable the internal journal file reader instead of using libsystemd
219
--enable-plugin-otel Enable the Netdata OpenTelemetry plugin. Default: disabled
220
--disable-plugin-otel Explicitly disable the Netdata OpenTelemetry plugin.
218
- --enable-plugin-systemd-journal Enable the systemd journal reader plugin. Default: enabled if Rust is available and the build isn’t known broken
219
- --disable-plugin-systemd-journal Explicitly disable the systemd journal reader plugin.
221
+ --enable-plugin-otel-signal-viewer Enable the OTel signal viewer plugin. Default: disabled
222
+ --disable-plugin-otel-signal-viewer Explicitly disable the OTel signal viewer plugin.
223
--enable-plugin-ibm Enable the IBM ecosystem monitoring plugin. Default: disabled
224
--disable-plugin-ibm Explicitly disable the IBM ecosystem monitoring plugin.
225
--enable-exporting-kinesis Enable AWS Kinesis exporting connector. Default: enable it when libaws_cpp_sdk_kinesis
@@ -261,6 +264,7 @@ ENABLE_GO=1
264
ENABLE_PYTHON=1
265
ENABLE_CHARTS=1
266
ENABLE_OTEL=0
267
+ENABLE_OTEL_SIGNAL_VIEWER=0
268
ENABLE_IBM=0
269
FORCE_LEGACY_CXX=0
270
NETDATA_CMAKE_OPTIONS="${NETDATA_CMAKE_OPTIONS-}"
@@ -300,8 +304,11 @@ while [ -n "${1}" ]; do
304
"--disable-plugin-xenstat") ENABLE_XENSTAT=0 ;;
305
"--enable-plugin-systemd-journal") ENABLE_SYSTEMD_JOURNAL=1 ;;
306
"--disable-plugin-systemd-journal") ENABLE_SYSTEMD_JOURNAL=0 ;;
307
+ "--internal-systemd-journal") USE_RUST_JOURNAL_FILE=1 ;;
308
"--enable-plugin-otel") ENABLE_OTEL=1 ;;
309
"--disable-plugin-otel") ENABLE_OTEL=0 ;;
310
+ "--enable-plugin-otel-signal-viewer") ENABLE_OTEL_SIGNAL_VIEWER=1 ;;
311
+ "--disable-plugin-otel-signal-viewer") ENABLE_OTEL_SIGNAL_VIEWER=0 ;;
312
"--enable-plugin-ibm") ENABLE_IBM=1 ;;
313
"--disable-plugin-ibm") ENABLE_IBM=0 ;;
314
"--enable-exporting-kinesis" | "--enable-backend-kinesis")
@@ -817,8 +824,6 @@ if [ "$(id -u)" -eq 0 ]; then
824
fi
825
fi
826
820
- # The Rust-based 'journal-viewer' plugin was added in 2.8-nightly to replace 'systemd-journal'.
821
- # TODO: After 2.9 release and proper deprecation announcement, remove this.
827
if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/systemd-journal.plugin" ]; then
828
run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/systemd-journal.plugin"
829
capabilities=0
@@ -834,18 +839,18 @@ if [ "$(id -u)" -eq 0 ]; then
839
fi
840
fi
841
837
- if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/journal-viewer-plugin" ]; then
838
- run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/journal-viewer-plugin"
842
+ if [ -f "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin" ]; then
843
+ run chown "root:${NETDATA_GROUP}" "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin"
844
capabilities=0
845
if ! iscontainer && command -v setcap 1> /dev/null 2>&1; then
841
- run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/journal-viewer-plugin"
842
- if run setcap cap_dac_read_search+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/journal-viewer-plugin"; then
846
+ run chmod 0750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin"
847
+ if run setcap cap_dac_read_search+ep "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin"; then
848
capabilities=1
849
fi
850
fi
851
852
if [ $capabilities -eq 0 ]; then
848
- run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/journal-viewer-plugin"
853
+ run chmod 4750 "${NETDATA_PREFIX}/usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin"
854
fi
855
fi
856
netdata.spec.in
+58
-17
@@ -270,11 +270,11 @@ Suggests: %{name}-plugin-freeipmi = %{version}
270
%if %{_have_cups}
271
Suggests: %{name}-plugin-cups = %{version}
272
%endif
273
-Requires: %{name}-plugin-journal-viewer = %{version}
273
+Recommends: %{name}-plugin-systemd-journal = %{version}
274
Recommends: %{name}-plugin-systemd-units = %{version}
275
Recommends: %{name}-plugin-network-viewer = %{version}
276
%else
277
-Requires: %{name}-plugin-journal-viewer = %{version}
277
+Requires: %{name}-plugin-systemd-journal = %{version}
278
Requires: %{name}-plugin-network-viewer = %{version}
279
%endif
280
@@ -327,6 +327,10 @@ BuildRequires: golang >= 1.25
327
%endif
328
# end - go.d.plugin plugin dependencies
329
330
+# systemd-journal dependencies
331
+BuildRequires: pkgconfig(libsystemd)
332
+# end - systemd-journal dependencies
333
+
334
# Prometheus remote write dependencies
335
%if 0%{?suse_version}
336
BuildRequires: snappy-devel
@@ -698,9 +702,12 @@ rm -rf "${RPM_BUILD_ROOT}"
702
# perf belongs to a different sub-package
703
%exclude %{_libexecdir}/%{name}/plugins.d/perf.plugin
704
701
-# journal-viewer belongs to a different sub-package
702
-%exclude %{_libexecdir}/%{name}/plugins.d/journal-viewer-plugin
703
-%exclude %{_libdir}/%{name}/conf.d/journal-viewer.yaml
705
+# systemd-journal belongs to a different sub-package
706
+%exclude %{_libexecdir}/%{name}/plugins.d/systemd-journal.plugin
707
+
708
+# otel-signal-viewer belongs to a different sub-package
709
+%exclude %{_libexecdir}/%{name}/plugins.d/otel-signal-viewer-plugin
710
+%exclude %{_libdir}/%{name}/conf.d/otel-signal-viewer.yaml
711
712
# xenstat belongs to a different sub-package
713
%exclude %{_libexecdir}/%{name}/plugins.d/xenstat.plugin
@@ -805,7 +812,7 @@ Summary: The eBPF metrics collection plugin for the Netdata Agent
812
Group: Applications/System
813
Requires: %{name} = %{version}
814
Conflicts: %{name} < %{version}
808
-%if 0%{?centos_ver} != 7
815
+%if 0%{?centos_ver} != 7 && 0%{?amazon_linux} != 2
816
Recommends: %{name}-plugin-apps = %{version}
817
Recommends: %{name}-ebpf-legacy-code >= %{version}
818
%else
@@ -3119,25 +3126,34 @@ Requires(pre): %{name}-user >= %{version}
3126
%caps(cap_dac_read_search=ep) %attr(0750,root,netdata) %{_libexecdir}/%{name}/plugins.d/debugfs.plugin
3127
3128
%package plugin-journal-viewer
3122
-Summary: The journal viewer plugin for the Netdata Agent
3129
+Summary: Transitional dummy package
3130
+Group: Applications/System
3131
+Requires: %{name}-plugin-systemd-journal
3132
+
3133
+%description plugin-journal-viewer
3134
+ This package simply ensures a clean upgrade to the renamed
3135
+ netdata-plugin-systemd-journal package. Once that package is installed,
3136
+ you can safely remove this one.
3137
+
3138
+%files plugin-journal-viewer
3139
+
3140
+%package plugin-systemd-journal
3141
+Summary: The systemd-journal plugin for the Netdata Agent
3142
Group: Applications/System
3143
Requires: %{name} = %{version}
3144
Conflicts: %{name} < %{version}
3126
-Conflicts: %{name}-plugin-systemd-journal
3127
-Obsoletes: %{name}-plugin-systemd-journal < %{version}
3145
%if ! %{_have_sysuser}
3146
Requires(pre): %{name}-user >= %{version}
3147
%endif
3148
3132
-%description plugin-journal-viewer
3133
- This plugin provides systemd journal log viewing and querying functionality
3134
- with histogram analysis and faceted search capabilities for the Netdata Agent.
3149
+%description plugin-systemd-journal
3150
+ This plugin allows the Netdata Agent to present entries from the systemd
3151
+ journal on Netdata Cloud or the local Agent Dashboard.
3152
3136
-%files plugin-journal-viewer
3153
+%files plugin-systemd-journal
3154
%defattr(0750,root,netdata,0750)
3138
-# CAP_DAC_READ_SEARCH required for reading journal files.
3139
-%caps(cap_dac_read_search=ep) %attr(0750,root,netdata) %{_libexecdir}/%{name}/plugins.d/journal-viewer-plugin
3140
-%attr(0640,root,netdata) %{_libdir}/%{name}/conf.d/journal-viewer.yaml
3155
+# CAP_DAC_READ_SEARCH required for data collection.
3156
+%caps(cap_dac_read_search=ep) %attr(0750,root,netdata) %{_libexecdir}/%{name}/plugins.d/systemd-journal.plugin
3157
3158
%if 0%{?centos_ver} != 7 && 0%{?amazon_linux} != 2
3159
%package plugin-systemd-units
@@ -3182,7 +3198,7 @@ Group: Applications/System
3198
Requires: %{name} = %{version}
3199
Conflicts: %{name} < %{version}
3200
%if 0%{?_have_ebpf}
3185
-%if 0%{?centos_ver} != 7
3201
+%if 0%{?centos_ver} != 7 && 0%{?amazon_linux} != 2
3202
Recommends: %{name}-plugin-ebpf = %{version}
3203
%else
3204
Requires: %{name}-plugin-ebpf = %{version}
@@ -3222,6 +3238,7 @@ Requires(pre): %{name}-user >= %{version}
3238
Summary: The Open Telemetry plugin for the Netdata Agent
3239
Group: Applications/System
3240
Requires: %{name} >= %{version}
3241
+Requires: %{name}-plugin-otel-signal-viewer = %{version}
3242
Conflicts: %{name} < %{version}
3243
%if ! %{_have_sysuser}
3244
Requires(pre): %{name}-user >= %{version}
@@ -3237,6 +3254,27 @@ Requires(pre): %{name}-user >= %{version}
3254
%attr(0750,root,netdata) %{_libexecdir}/%{name}/plugins.d/otel-plugin
3255
%endif
3256
3257
+%if %{_have_rust}
3258
+%package plugin-otel-signal-viewer
3259
+Summary: The OTel signal viewer plugin for the Netdata Agent
3260
+Group: Applications/System
3261
+Requires: %{name} = %{version}
3262
+Conflicts: %{name} < %{version}
3263
+%if ! %{_have_sysuser}
3264
+Requires(pre): %{name}-user >= %{version}
3265
+%endif
3266
+
3267
+%description plugin-otel-signal-viewer
3268
+ This plugin provides OTel signal viewing and querying functionality
3269
+ with histogram analysis and faceted search capabilities for the Netdata Agent.
3270
+
3271
+%files plugin-otel-signal-viewer
3272
+%defattr(0750,root,netdata,0750)
3273
+# CAP_DAC_READ_SEARCH required for reading journal files.
3274
+%caps(cap_dac_read_search=ep) %attr(0750,root,netdata) %{_libexecdir}/%{name}/plugins.d/otel-signal-viewer-plugin
3275
+%attr(0640,root,netdata) %{_libdir}/%{name}/conf.d/otel-signal-viewer.yaml
3276
+%endif
3277
+
3278
%package user
3279
Summary: User and group accounts for the Netdata Agent
3280
Group: Applications/System
@@ -3280,6 +3318,9 @@ done
3318
%attr(0644,root,root) %{_sysusersdir}/%{name}.conf
3319
3320
%changelog
3321
+* Tue Feb 10 2026 Austin Hemmelgarn <austin@netdata.cloud> 0.0.0-38
3322
+- Add new otel-signal-viewer plugin
3323
+- Add back systemd-journal plugin with journal-viewer plugin as a transitional package
3324
* Tue Nov 11 2025 Austin Hemmelgarn <austin@netdata.cloud> 0.0.0-37
3325
- Update to use sysusers infrastructure when available.
3326
* Fri Oct 24 2025 Austin Hemmelgarn <austin@netdata.cloud> 0.0.0-36
packaging/cmake/Modules/Packaging.cmake
+48
-22
@@ -97,7 +97,7 @@ set(CPACK_DEBIAN_NETDATA_PACKAGE_PREDEPENDS "netdata-user, libcap2-bin")
97
set(CPACK_DEBIAN_NETDATA_PACKAGE_SUGGESTS
98
"netdata-plugin-cups, netdata-plugin-freeipmi, netdata-plugin-ibm")
99
set(CPACK_DEBIAN_NETDATA_PACKAGE_RECOMMENDS
100
- "netdata-plugin-systemd-units, \
100
+ "netdata-plugin-systemd-journal, netdata-plugin-systemd-units, \
101
netdata-plugin-network-viewer")
102
set(CPACK_DEBIAN_NETDATA_PACKAGE_CONFLICTS
103
"netdata-core, netdata-plugins-bash, netdata-plugins-python, netdata-web")
@@ -110,10 +110,6 @@ if(ENABLE_PLUGIN_OTEL)
110
list(APPEND _main_deps "netdata-plugin-otel")
111
endif()
112
113
-if(ENABLE_NETDATA_JOURNAL_FILE_READER)
114
- list(APPEND _main_deps "netdata-plugin-journal-viewer")
115
-endif()
116
-
113
if(ENABLE_PLUGIN_CHARTS)
114
list(APPEND _main_deps "netdata-plugin-chartsd")
115
endif()
@@ -429,6 +425,7 @@ set(CPACK_COMPONENT_PLUGIN-OTEL_DESCRIPTION
425
426
set(CPACK_DEBIAN_PLUGIN-OTEL_PACKAGE_NAME "netdata-plugin-otel")
427
set(CPACK_DEBIAN_PLUGIN-OTEL_PACKAGE_SECTION "net")
428
+set(CPACK_DEBIAN_PLUGIN-OTEL_PACKAGE_DEPENDS "netdata-plugin-otel-signal-viewer (= ${CPACK_PACKAGE_VERSION})")
429
set(CPACK_DEBIAN_PLUGIN-OTEL_PACKAGE_CONFLICTS "netdata (<< 1.40)")
430
set(CPACK_DEBIAN_PLUGIN-OTEL_PACKAGE_PREDEPENDS "netdata-user")
431
@@ -437,6 +434,25 @@ set(CPACK_DEBIAN_PLUGIN-OTEL_PACKAGE_CONTROL_EXTRA
434
435
set(CPACK_DEBIAN_PLUGIN-OTEL_DEBUGINFO_PACKAGE Off)
436
437
+#
438
+# otel-signal-viewer
439
+#
440
+
441
+set(CPACK_COMPONENT_PLUGIN-OTEL-SIGNAL-VIEWER_DEPENDS "netdata")
442
+set(CPACK_COMPONENT_PLUGIN-OTEL-SIGNAL-VIEWER_DESCRIPTION
443
+ "The OTel signal viewer plugin for the Netdata Agent
444
+ This plugin provides OTel signal viewing and querying functionality
445
+ with histogram analysis and faceted search capabilities.")
446
+
447
+set(CPACK_DEBIAN_PLUGIN-OTEL-SIGNAL-VIEWER_PACKAGE_NAME "netdata-plugin-otel-signal-viewer")
448
+set(CPACK_DEBIAN_PLUGIN-OTEL-SIGNAL-VIEWER_PACKAGE_SECTION "net")
449
+set(CPACK_DEBIAN_PLUGIN-OTEL-SIGNAL-VIEWER_PACKAGE_PREDEPENDS "libcap2-bin, adduser")
450
+
451
+set(CPACK_DEBIAN_PLUGIN-OTEL-SIGNAL-VIEWER_PACKAGE_CONTROL_EXTRA
452
+ "${PKG_FILES_PATH}/deb/plugin-otel-signal-viewer/postinst")
453
+
454
+set(CPACK_DEBIAN_PLUGIN-OTEL-SIGNAL-VIEWER_DEBUGINFO_PACKAGE Off)
455
+
456
#
457
# nfacct.plugin
458
#
@@ -522,27 +538,33 @@ set(CPACK_DEBIAN_PLUGIN-SLABINFO_PACKAGE_CONTROL_EXTRA
538
set(CPACK_DEBIAN_PLUGIN-SLABINFO-DEBUGINFO_PACKAGE On)
539
540
#
525
-# journal-viewer.plugin
541
+# systemd-journal.plugin
542
#
543
528
-set(CPACK_COMPONENT_PLUGIN-JOURNAL-VIEWER_DEPENDS "netdata")
529
-set(CPACK_COMPONENT_PLUGIN-JOURNAL-VIEWER_DESCRIPTION
530
- "The journal viewer plugin for the Netdata Agent
531
- This plugin provides systemd journal log viewing and querying functionality
532
- with histogram analysis and faceted search capabilities.")
544
+set(CPACK_COMPONENT_PLUGIN-SYSTEMD-JOURNAL_DEPENDS "netdata")
545
+set(CPACK_COMPONENT_PLUGIN-SYSTEMD-JOURNAL_DESCRIPTION
546
+ "The systemd-journal collector for the Netdata Agent
547
+ This plugin allows the Netdata Agent to present logs from the systemd
548
+ journal on Netdata Cloud or the local Agent dashboard.")
549
534
-set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_PACKAGE_NAME "netdata-plugin-journal-viewer")
535
-set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_PACKAGE_SECTION "net")
536
-set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_PACKAGE_PREDEPENDS "libcap2-bin, adduser")
550
+set(CPACK_DEBIAN_PLUGIN-SYSTEMD-JOURNAL_PACKAGE_NAME "netdata-plugin-systemd-journal")
551
+set(CPACK_DEBIAN_PLUGIN-SYSTEMD-JOURNAL_PACKAGE_SECTION "net")
552
+set(CPACK_DEBIAN_PLUGIN-SYSTEMD-JOURNAL_PACKAGE_PREDEPENDS "libcap2-bin, adduser")
553
+
554
+set(CPACK_DEBIAN_PLUGIN-SYSTEMD-JOURNAL_PACKAGE_CONTROL_EXTRA
555
+ "${PKG_FILES_PATH}/deb/plugin-systemd-journal/postinst")
556
538
-set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_PACKAGE_CONTROL_EXTRA
539
- "${PKG_FILES_PATH}/deb/plugin-journal-viewer/postinst")
557
+set(CPACK_DEBIAN_PLUGIN-SYSTEMD-JOURNAL_DEBUGINFO_PACKAGE On)
558
541
-# Replaces old systemd-journal plugin
542
-set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_PACKAGE_CONFLICTS "netdata-plugin-systemd-journal")
543
-set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_PACKAGE_REPLACES "netdata-plugin-systemd-journal")
559
+set(CPACK_COMPONENT_PLUGIN-JOURNAL-VIEWER_DESCRIPTION
560
+ "Transitional dummy package.
561
+ This package simply ensures a clean upgrade to the renamed
562
+ netdata-plugin-systemd-journal package. Once that package is installed,
563
+ you can safely remove this one.")
564
545
-set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_DEBUGINFO_PACKAGE Off)
565
+set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_PACKAGE_NAME "netdata-plugin-journal-viewer")
566
+set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_PACKAGE_SECTION "net")
567
+set(CPACK_DEBIAN_PLUGIN-JOURNAL-VIEWER_DEPENDS "netdata-plugin-systemd-journal (= ${CPACK_PACKAGE_VERSION})")
568
569
#
570
# systemd-units.plugin
@@ -634,8 +656,9 @@ endif()
656
if(ENABLE_PLUGIN_SLABINFO)
657
list(APPEND CPACK_COMPONENTS_ALL "plugin-slabinfo")
658
endif()
637
-if(ENABLE_PLUGIN_OTEL)
638
- list(APPEND CPACK_COMPONENTS_ALL "plugin-journal-viewer")
659
+if(ENABLE_PLUGIN_SYSTEMD_JOURNAL)
660
+ list(APPEND CPACK_COMPONENTS_ALL "plugin-journal-viewer")
661
+ list(APPEND CPACK_COMPONENTS_ALL "plugin-systemd-journal")
662
endif()
663
if(ENABLE_PLUGIN_SYSTEMD_UNITS)
664
list(APPEND CPACK_COMPONENTS_ALL "plugin-systemd-units")
@@ -646,5 +669,8 @@ endif()
669
if(ENABLE_PLUGIN_OTEL)
670
list(APPEND CPACK_COMPONENTS_ALL "plugin-otel")
671
endif()
672
+if(ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER)
673
+ list(APPEND CPACK_COMPONENTS_ALL "plugin-otel-signal-viewer")
674
+endif()
675
676
include(CPack)
packaging/cmake/pkg-files/deb/plugin-journal-viewer/postinst
deleted
-15
@@ -1,15 +0,0 @@
1
-#!/bin/sh
2
-
3
-set -e
4
-
5
-case "$1" in
6
- configure|reconfigure)
7
- chown root:netdata /usr/libexec/netdata/plugins.d/journal-viewer-plugin
8
- chmod 0750 /usr/libexec/netdata/plugins.d/journal-viewer-plugin
9
- if ! setcap "cap_dac_read_search=eip" /usr/libexec/netdata/plugins.d/journal-viewer-plugin; then
10
- chmod -f 4750 /usr/libexec/netdata/plugins.d/journal-viewer-plugin
11
- fi
12
- ;;
13
-esac
14
-
15
-exit 0
packaging/cmake/pkg-files/deb/plugin-otel-signal-viewer/postinst
new
+15
@@ -0,0 +1,15 @@
1
+#!/bin/sh
2
+
3
+set -e
4
+
5
+case "$1" in
6
+ configure|reconfigure)
7
+ chown root:netdata /usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin
8
+ chmod 0750 /usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin
9
+ if ! setcap "cap_dac_read_search=eip" /usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin; then
10
+ chmod -f 4750 /usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin
11
+ fi
12
+ ;;
13
+esac
14
+
15
+exit 0
packaging/cmake/pkg-files/deb/plugin-systemd-journal/postinst
new
+15
@@ -0,0 +1,15 @@
1
+#!/bin/sh
2
+
3
+set -e
4
+
5
+case "$1" in
6
+ configure|reconfigure)
7
+ chown root:netdata /usr/libexec/netdata/plugins.d/systemd-journal.plugin
8
+ chmod 0750 /usr/libexec/netdata/plugins.d/systemd-journal.plugin
9
+ if ! setcap "cap_dac_read_search=eip" /usr/libexec/netdata/plugins.d/systemd-journal.plugin; then
10
+ chmod -f 4750 /usr/libexec/netdata/plugins.d/systemd-journal.plugin
11
+ fi
12
+ ;;
13
+esac
14
+
15
+exit 0
packaging/docker/Dockerfile
+24
-26
@@ -29,19 +29,18 @@ WORKDIR /opt/netdata.git
29
30
# Install from source
31
RUN chmod +x netdata-installer.sh && \
32
- cp -rp /deps/* /usr/local/ && \
33
- BUILD_ARCH="${BUILD_ARCH:-"$(uname -m)"}" && \
34
- /bin/echo -e "INSTALL_TYPE='oci'\nPREBUILT_ARCH='${BUILD_ARCH}'" > ./system/.install-type && \
35
- EXTRA_INSTALL_OPTS="${EXTRA_INSTALL_OPTS} --enable-plugin-systemd-journal" && \
36
- NETDATA_BUILD_DIR=/build \
37
- CFLAGS="$(packaging/docker/gen-cflags.sh)" LDFLAGS="-Wl,--gc-sections" ./netdata-installer.sh --dont-wait --dont-start-it \
38
- --use-system-protobuf \
39
- --disable-ebpf \
40
- --enable-plugin-otel \
41
- ${EXTRA_INSTALL_OPTS} \
42
- --install-no-prefix / \
43
- "$([ "$RELEASE_CHANNEL" = stable ] && echo --stable-channel)" && \
44
- rm -rf /build /root/.cargo
32
+ cp -rp /deps/* /usr/local/ && \
33
+ BUILD_ARCH="${BUILD_ARCH:-"$(uname -m)"}" && \
34
+ /bin/echo -e "INSTALL_TYPE='oci'\nPREBUILT_ARCH='${BUILD_ARCH}'" > ./system/.install-type && \
35
+ CFLAGS="$(packaging/docker/gen-cflags.sh)" LDFLAGS="-Wl,--gc-sections" ./netdata-installer.sh --dont-wait --dont-start-it \
36
+ --use-system-protobuf \
37
+ --disable-ebpf \
38
+ --enable-plugin-otel \
39
+ --enable-plugin-otel-signal-viewer \
40
+ --internal-systemd-journal \
41
+ ${EXTRA_INSTALL_OPTS} \
42
+ --install-no-prefix / \
43
+ "$([ "$RELEASE_CHANNEL" = stable ] && echo --stable-channel)"
44
45
# files to one directory
46
RUN mkdir -p /app/usr/sbin/ \
@@ -116,20 +115,20 @@ COPY --from=builder /app /
115
# Create netdata user and apply the permissions as described in
116
# https://docs.netdata.cloud/docs/netdata-security/#netdata-directories, but own everything by root group due to https://github.com/netdata/netdata/pull/6543
117
# hadolint ignore=DL3013
119
-RUN addgroup --stdoutmsglevel=info --gid ${NETDATA_GID} --system "${DOCKER_GRP}" && \
120
- adduser --stdoutmsglevel=info --system --no-create-home --shell /usr/sbin/nologin --uid ${NETDATA_UID} --home /etc/netdata --group "${DOCKER_USR}" && \
121
- chown -vR root:root \
118
+RUN addgroup --gid ${NETDATA_GID} --system "${DOCKER_GRP}" && \
119
+ adduser --system --no-create-home --shell /usr/sbin/nologin --uid ${NETDATA_UID} --home /etc/netdata --group "${DOCKER_USR}" && \
120
+ chown -R root:root \
121
/etc/netdata \
122
/usr/share/netdata \
123
/usr/libexec/netdata && \
125
- chown -vR netdata:root \
124
+ chown -R netdata:root \
125
/usr/lib/netdata \
126
/var/cache/netdata \
127
/var/lib/netdata \
128
/var/log/netdata && \
130
- chown -vR netdata:netdata /var/lib/netdata/cloud.d && \
131
- chmod -v 0700 /var/lib/netdata/cloud.d && \
132
- chmod -v 0755 /usr/libexec/netdata/plugins.d/*.plugin && \
129
+ chown -R netdata:netdata /var/lib/netdata/cloud.d && \
130
+ chmod 0700 /var/lib/netdata/cloud.d && \
131
+ chmod 0755 /usr/libexec/netdata/plugins.d/*.plugin && \
132
for name in cgroup-network \
133
local-listeners \
134
apps.plugin \
@@ -141,14 +140,13 @@ RUN addgroup --stdoutmsglevel=info --gid ${NETDATA_GID} --system "${DOCKER_GRP}"
140
slabinfo.plugin \
141
network-viewer.plugin \
142
otel-plugin \
144
- journal-viewer-plugin; do \
145
- if [ -f "/usr/libexec/netdata/plugins.d/$name" ] ; then \
146
- chmod -v 4755 "/usr/libexec/netdata/plugins.d/$name"; \
147
- fi \
143
+ otel-signal-viewer-plugin \
144
+ systemd-journal.plugin; do \
145
+ [ -f "/usr/libexec/netdata/plugins.d/$name" ] && chmod 4755 "/usr/libexec/netdata/plugins.d/$name"; \
146
done && \
147
# Group write permissions due to: https://github.com/netdata/netdata/pull/6543
150
- find /var/lib/netdata /var/cache/netdata -type d -exec chmod -v 0770 {} \; && \
151
- find /var/lib/netdata /var/cache/netdata -type f -exec chmod -v 0660 {} \; && \
148
+ find /var/lib/netdata /var/cache/netdata -type d -exec chmod 0770 {} \; && \
149
+ find /var/lib/netdata /var/cache/netdata -type f -exec chmod 0660 {} \; && \
150
cp -va /etc/netdata /etc/netdata.stock
151
152
ENTRYPOINT ["/usr/sbin/run.sh"]
packaging/docker/README.md
+3
-3
@@ -38,8 +38,8 @@ The Netdata container requires specific **privileges** and **mounts** to provide
38
| go.d.plugin | /var/log | Web servers logs tailing. See [weblog](https://github.com/netdata/go.d.plugin/tree/master/modules/weblog#readme) collector. |
39
| apps.plugin | /etc/passwd, /etc/group | Monitoring of host system resource usage by each user and user group. |
40
| proc.plugin | /proc | Host system monitoring (CPU, memory, network interfaces, disks, etc.). |
41
-| journal-viewer-plugin | /var/log | Viewing, exploring and analyzing systemd journal logs. |
42
-| systemd-units.plugin | /run/dbus | Systemd-list-units function: information about all systemd units, including their active state, description, whether they are enabled, and more. |
41
+| systemd-journal.plugin | /var/log | Viewing, exploring and analyzing systemd journal logs. |
42
+| systemd-units.plugin | /run/dbus | Systemd-list-units function: information about all systemd units, including their active state, description, whether they are enabled, and more. |
43
| go.d.plugin | /run/dbus | [go.d/systemdunits](https://github.com/netdata/go.d.plugin/tree/master/modules/systemdunits#readme) |
44
45
</details>
@@ -475,7 +475,7 @@ The following components won't work:
475
- freeipmi.plugin
476
- perf.plugin
477
- slabinfo.plugin
478
-- journal-viewer-plugin
478
+- systemd-journal.plugin
479
480
This method creates a [volume](https://docs.docker.com/storage/volumes/) for Netdata's configuration files
481
_within the container_ at `/etc/netdata`.
packaging/installer/functions.sh
+14
-3
@@ -330,14 +330,24 @@ prepare_cmake_options() {
330
fi
331
332
if [ -z "${ENABLE_SYSTEMD_JOURNAL}" ]; then
333
- if command -v rustc > /dev/null 2>&1; then
334
- ENABLE_SYSTEMD_JOURNAL=1
333
+ if check_for_module libsystemd; then
334
+ if check_for_module libelogind; then
335
+ ENABLE_SYSTEMD_JOURNAL=0
336
+ else
337
+ ENABLE_SYSTEMD_JOURNAL=1
338
+ fi
339
else
340
ENABLE_SYSTEMD_JOURNAL=0
341
fi
342
fi
343
340
- enable_feature NETDATA_JOURNAL_FILE_READER "${ENABLE_SYSTEMD_JOURNAL}"
344
+ enable_feature PLUGIN_SYSTEMD_JOURNAL "${ENABLE_SYSTEMD_JOURNAL}"
345
+
346
+ if [ "${ENABLE_SYSTEMD_JOURNAL}" -eq 1 ] && [ -n "${USE_RUST_JOURNAL_FILE}" ]; then
347
+ enable_feature NETDATA_JOURNAL_FILE_READER 1
348
+ else
349
+ enable_feature NETDATA_JOURNAL_FILE_READER 0
350
+ fi
351
352
if check_for_module 'libsystemd >= 221'; then
353
enable_feature PLUGIN_SYSTEMD_UNITS 1
@@ -368,6 +378,7 @@ prepare_cmake_options() {
378
enable_feature ML "${NETDATA_ENABLE_ML:-1}"
379
enable_feature PLUGIN_APPS "${ENABLE_APPS:-1}"
380
enable_feature PLUGIN_OTEL "${ENABLE_OTEL:-0}"
381
+ enable_feature PLUGIN_OTEL_SIGNAL_VIEWER "${ENABLE_OTEL_SIGNAL_VIEWER:-0}"
382
enable_feature PLUGIN_IBM "${ENABLE_IBM:-0}"
383
384
check_for_feature EXPORTER_PROMETHEUS_REMOTE_WRITE "${EXPORTER_PROMETHEUS}" snappy
packaging/installer/netdata-updater.sh
+13
@@ -1262,6 +1262,19 @@ update_binpkg() {
1262
# shellcheck disable=SC2086
1263
env ${env} ${pm_cmd} ${upgrade_subcmd} ${pkg_install_opts} netdata >&3 2>&3 || fatal "Failed to update Netdata package." U000F
1264
1265
+ if ${pkg_installed_check} systemd > /dev/null 2>&1; then
1266
+ if [ "${NETDATA_NO_SYSTEMD_JOURNAL}" -eq 0 ]; then
1267
+ if ! ${pkg_installed_check} netdata-plugin-systemd-journal > /dev/null 2>&1; then
1268
+ env ${env} ${pm_cmd} ${install_subcmd} ${pkg_install_opts} netdata-plugin-systemd-journal >&3 2>&3
1269
+
1270
+ if [ -n "${mark_auto_cmd}" ]; then
1271
+ # shellcheck disable=SC2086
1272
+ env ${env} ${mark_auto_cmd} netdata-plugin-systemd-journal >&3 2>&3
1273
+ fi
1274
+ fi
1275
+ fi
1276
+ fi
1277
+
1278
current_version="$(get_current_version)"
1279
latest_version="$(get_latest_version)"
1280
packaging/makeself/install-or-update.sh
+9
-1
@@ -180,7 +180,7 @@ fi
180
181
progress "changing plugins ownership and permissions"
182
183
-for x in ndsudo apps.plugin perf.plugin slabinfo.plugin debugfs.plugin freeipmi.plugin ioping cgroup-network local-listeners network-viewer.plugin ebpf.plugin nfacct.plugin xenstat.plugin python.d.plugin charts.d.plugin go.d.plugin ioping.plugin cgroup-network-helper.sh; do
183
+for x in ndsudo apps.plugin perf.plugin slabinfo.plugin debugfs.plugin freeipmi.plugin ioping cgroup-network local-listeners network-viewer.plugin ebpf.plugin nfacct.plugin xenstat.plugin python.d.plugin charts.d.plugin go.d.plugin ioping.plugin cgroup-network-helper.sh otel-plugin otel-signal-viewer-plugin; do
184
f="usr/libexec/netdata/plugins.d/${x}"
185
if [ -f "${f}" ]; then
186
run chown root:${NETDATA_GROUP} "${f}"
@@ -209,11 +209,19 @@ if command -v setcap >/dev/null 2>&1; then
209
if ! run setcap "${perf_caps}" "usr/libexec/netdata/plugins.d/perf.plugin"; then
210
run chmod 4750 "usr/libexec/netdata/plugins.d/perf.plugin"
211
fi
212
+ if [ -f "usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin" ]; then
213
+ if ! run setcap "cap_dac_read_search=eip" "usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin"; then
214
+ run chmod 4750 "usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin"
215
+ fi
216
+ fi
217
else
218
for x in apps.plugin perf.plugin slabinfo.plugin debugfs.plugin; do
219
f="usr/libexec/netdata/plugins.d/${x}"
220
run chmod 4750 "${f}"
221
done
222
+ if [ -f "usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin" ]; then
223
+ run chmod 4750 "usr/libexec/netdata/plugins.d/otel-signal-viewer-plugin"
224
+ fi
225
fi
226
227
for x in ndsudo freeipmi.plugin ioping cgroup-network local-listeners network-viewer.plugin ebpf.plugin nfacct.plugin xenstat.plugin; do
packaging/makeself/jobs/70-netdata-git.install.sh
+7
-3
@@ -22,14 +22,18 @@ export IS_NETDATA_STATIC_BINARY="yes"
22
NETDATA_BUILD_DIR="$(build_path netdata)"
23
export NETDATA_BUILD_DIR
24
25
-export NETDATA_CMAKE_OPTIONS="-DSTATIC_BUILD=On -DENABLE_LIBBACKTRACE=On"
25
+# Needed to make Rust play nice with our static builds
26
+# Once Cargo's profile-rustflags feature is a bit more widespread, we should switch to using that to specify this.
27
+export RUSTFLAGS="-C target-feature=+crt-static"
28
29
case "${BUILDARCH}" in
30
armv6l)
29
- export INSTALLER_ARGS="--disable-plugin-otel --disable-plugin-systemd-journal"
31
+ export NETDATA_CMAKE_OPTIONS="-DSTATIC_BUILD=On -DENABLE_LIBBACKTRACE=On"
32
+ export INSTALLER_ARGS="--disable-plugin-systemd-journal --disable-plugin-otel --disable-plugin-otel-signal-viewer"
33
;;
34
*)
32
- export INSTALLER_ARGS="--enable-plugin-otel --enable-plugin-systemd-journal"
35
+ export NETDATA_CMAKE_OPTIONS="-DSTATIC_BUILD=On -DENABLE_LIBBACKTRACE=On"
36
+ export INSTALLER_ARGS="--enable-plugin-systemd-journal --internal-systemd-journal --enable-plugin-otel --enable-plugin-otel-signal-viewer"
37
;;
38
esac
39
packaging/windows/compile-on-windows.sh
+1
-1
@@ -43,7 +43,7 @@ CFLAGS="${BUILD_CFLAGS}" /usr/bin/cmake \
43
-DENABLE_PLUGIN_GO=On \
44
-DENABLE_EXPORTER_PROMETHEUS_REMOTE_WRITE=Off \
45
-DENABLE_PLUGIN_OTEL=Off \
46
- -DENABLE_NETDATA_JOURNAL_FILE_READER=Off \
46
+ -DENABLE_PLUGIN_SYSTEMD_JOURNAL=Off \
47
-DENABLE_BUNDLED_JSONC=On \
48
-DENABLE_BUNDLED_PROTOBUF=Off \
49
-DRust_COMPILER=/ucrt64/bin/rustc \
src/collectors/README.md
+1
@@ -27,6 +27,7 @@ This section outlines the required privileges and how they are configured in dif
27
|------------------------|-------------------------------------------------|-----------------------------------------------------|
28
| apps.plugin | CAP_DAC_READ_SEARCH, CAP_SYS_PTRACE | setuid root |
29
| debugfs.plugin | CAP_DAC_READ_SEARCH | setuid root |
30
+| systemd-journal.plugin | CAP_DAC_READ_SEARCH | setuid root |
31
| perf.plugin | CAP_PERFMON | setuid root |
32
| slabinfo.plugin | CAP_DAC_READ_SEARCH | setuid root |
33
| go.d.plugin | CAP_DAC_READ_SEARCH, CAP_NET_ADMIN, CAP_NET_RAW | setuid root |
src/collectors/systemd-journal.plugin/README.md
renamed
+4
-4
@@ -1,8 +1,8 @@
1
-# `journal-viewer` plugin
1
+# `systemd` journal plugin
2
3
[KEY FEATURES](#key-features) | [PREREQUISITES](#prerequisites) | [JOURNAL SOURCES](#journal-sources) | [JOURNAL FIELDS](#journal-fields) | [VISUALIZATION](#visualization-capabilities) | [PLAY MODE](#play-mode) | [FULL TEXT SEARCH](#full-text-search) | [QUERY PERFORMANCE](#query-performance) | [PERFORMANCE AT SCALE](#performance-at-scale) | [BEST PRACTICES](#best-practices-for-better-performance) | [CONFIGURATION](#configuration-and-maintenance) | [FAQ](#faq) | [HOW TO TROUBLESHOOT COMMON ISSUES](#how-to-troubleshoot-common-issues) | [HOW TO VERIFY SETUP](#how-to-verify-setup)
4
5
-The `journal-viewer` plugin provides an efficient way to view, explore, and analyze `systemd` journal logs directly from the Netdata dashboard. It combines powerful filtering, real-time updates, and visual analysis tools to help you troubleshoot system issues effectively.
5
+The `systemd` journal plugin provides an efficient way to view, explore, and analyze `systemd` journal logs directly from the Netdata dashboard. It combines powerful filtering, real-time updates, and visual analysis tools to help you troubleshoot system issues effectively.
6
7
8
@@ -544,7 +544,7 @@ Sampling ensures responsiveness at scale, but selecting sources and filters rema
544
545
| Error Message | Meaning | Solution |
546
|--------------|---------|----------|
547
-| "Plugin not available" | The journal-viewer plugin isn't loaded | Check your Netdata installation type (must not be Alpine or static) |
547
+| "Plugin not available" | The systemd-journal plugin isn't loaded | Check your Netdata installation type (must not be Alpine or static) |
548
| "Unable to open journal" | Permission issues accessing journal files | Ensure Netdata has proper permissions for journal directories |
549
| "Timeout while querying" | Query is taking too long to complete | Reduce the query scope with filters or shorter timeframes |
550
| "No sources detected" | Cannot find valid journal files | Check journal file locations and setup |
@@ -558,7 +558,7 @@ Sampling ensures responsiveness at scale, but selecting sources and filters rema
558
sudo netdata -W plugins
559
```
560
561
-Check that the `journal-viewer-plugin` is listed as active.
561
+Check that the `systemd-journal` plugin is listed as active.
562
563
### How to confirm journal sources are detected
564
src/collectors/systemd-journal.plugin/active_journal_centralization_guide_no_encryption.md
renamed
src/collectors/systemd-journal.plugin/forward_secure_sealing.md
renamed
src/collectors/systemd-journal.plugin/passive_journal_centralization_guide_no_encryption.md
new
+150
@@ -0,0 +1,150 @@
1
+# Passive journal centralization without encryption
2
+
3
+This page will guide you through creating a passive journal centralization setup without the use of encryption.
4
+
5
+Once you centralize your infrastructure logs to a server, Netdata will automatically detect all the logs from all servers and organize them in sources.
6
+With the setup described in this document, journal files are identified by the IPs of the clients sending the logs. Netdata will automatically do
7
+reverse DNS lookups to find the names of the server and name the sources on the dashboard accordingly.
8
+
9
+A _passive_ journal server waits for clients to push their metrics to it, so in this setup we will:
10
+
11
+1. configure `systemd-journal-remote` on the server, to listen for incoming connections.
12
+2. configure `systemd-journal-upload` on the clients, to push their logs to the server.
13
+
14
+> ⚠️ **IMPORTANT**<br/>
15
+> These instructions will copy your logs to a central server, without any encryption or authorization.<br/>
16
+> DO NOT USE THIS ON NON-TRUSTED NETWORKS.
17
+
18
+## Server configuration
19
+
20
+On the centralization server install `systemd-journal-remote`:
21
+
22
+```bash
23
+# change this according to your distro
24
+sudo apt-get install systemd-journal-remote
25
+```
26
+
27
+Make sure the journal transfer protocol is `http`:
28
+
29
+```bash
30
+sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/
31
+
32
+# edit it to make sure it says:
33
+# --listen-http=-3
34
+# not:
35
+# --listen-https=-3
36
+sudo nano /etc/systemd/system/systemd-journal-remote.service
37
+
38
+# reload systemd
39
+sudo systemctl daemon-reload
40
+```
41
+
42
+Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`
43
+
44
+```bash
45
+# edit the socket file
46
+sudo systemctl edit systemd-journal-remote.socket
47
+```
48
+
49
+and add the following lines into the instructed place, and choose your desired port; save and exit.
50
+
51
+```bash
52
+[Socket]
53
+ListenStream=<DESIRED_PORT>
54
+```
55
+
56
+Finally, enable it, so that it will start automatically upon receiving a connection:
57
+
58
+```bash
59
+# enable systemd-journal-remote
60
+sudo systemctl enable --now systemd-journal-remote.socket
61
+sudo systemctl enable systemd-journal-remote.service
62
+```
63
+
64
+`systemd-journal-remote` is now listening for incoming journals from remote hosts.
65
+
66
+## Client configuration
67
+
68
+On the clients, install `systemd-journal-remote` (it includes `systemd-journal-upload`):
69
+
70
+```bash
71
+# change this according to your distro
72
+sudo apt-get install systemd-journal-remote
73
+```
74
+
75
+Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:
76
+
77
+```text
78
+[Upload]
79
+URL=http://centralization.server.ip:19532
80
+```
81
+
82
+Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:
83
+
84
+```bash
85
+sudo systemctl edit systemd-journal-upload
86
+```
87
+
88
+At the top, add:
89
+
90
+```text
91
+[Service]
92
+Restart=always
93
+```
94
+
95
+Enable and start `systemd-journal-upload`, like this:
96
+
97
+```bash
98
+sudo systemctl enable systemd-journal-upload
99
+sudo systemctl start systemd-journal-upload
100
+```
101
+
102
+## Verify it works
103
+
104
+To verify the central server is receiving logs, run this on the central server:
105
+
106
+```bash
107
+sudo ls -l /var/log/journal/remote/
108
+```
109
+
110
+You should see new files from the client's IP.
111
+
112
+Also, `systemctl status systemd-journal-remote` should show something like this:
113
+
114
+```bash
115
+systemd-journal-remote.service - Journal Remote Sink Service
116
+ Loaded: loaded (/etc/systemd/system/systemd-journal-remote.service; indirect; preset: disabled)
117
+ Active: active (running) since Sun 2023-10-15 14:29:46 EEST; 2h 24min ago
118
+TriggeredBy: ● systemd-journal-remote.socket
119
+ Docs: man:systemd-journal-remote(8)
120
+ man:journal-remote.conf(5)
121
+ Main PID: 2118153 (systemd-journal)
122
+ Status: "Processing requests..."
123
+ Tasks: 1 (limit: 154152)
124
+ Memory: 2.2M
125
+ CPU: 71ms
126
+ CGroup: /system.slice/systemd-journal-remote.service
127
+ └─2118153 /usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/
128
+```
129
+
130
+Note the `status: "Processing requests..."` and the PID under `CGroup`.
131
+
132
+On the client `systemctl status systemd-journal-upload` should show something like this:
133
+
134
+```bash
135
+● systemd-journal-upload.service - Journal Remote Upload Service
136
+ Loaded: loaded (/lib/systemd/system/systemd-journal-upload.service; enabled; vendor preset: disabled)
137
+ Drop-In: /etc/systemd/system/systemd-journal-upload.service.d
138
+ └─override.conf
139
+ Active: active (running) since Sun 2023-10-15 10:39:04 UTC; 3h 17min ago
140
+ Docs: man:systemd-journal-upload(8)
141
+ Main PID: 4169 (systemd-journal)
142
+ Status: "Processing input..."
143
+ Tasks: 1 (limit: 13868)
144
+ Memory: 3.5M
145
+ CPU: 1.081s
146
+ CGroup: /system.slice/systemd-journal-upload.service
147
+ └─4169 /lib/systemd/systemd-journal-upload --save-state
148
+```
149
+
150
+Note the `Status: "Processing input..."` and the PID under `CGroup`.
src/collectors/systemd-journal.plugin/passive_journal_centralization_guide_self_signed_certs.md
new
+249
@@ -0,0 +1,249 @@
1
+# Passive journal centralization with encryption using self-signed certificates
2
+
3
+This page will guide you through creating a **passive** journal centralization setup using **self-signed certificates** for encryption and authorization.
4
+
5
+Once you centralize your infrastructure logs to a server, Netdata will automatically detect all the logs from all servers and organize them in sources. With the setup described in this document, on recent systemd versions, Netdata will automatically name all remote sources using the names of the clients, as they are described at their certificates (on older versions, the names will be IPs or reverse DNS lookups of the IPs).
6
+
7
+A **passive** journal server waits for clients to push their metrics to it, so in this setup we will:
8
+
9
+1. configure a certificates authority and issue self-signed certificates for your servers.
10
+2. configure `systemd-journal-remote` on the server, to listen for incoming connections.
11
+3. configure `systemd-journal-upload` on the clients, to push their logs to the server.
12
+
13
+Keep in mind that the authorization involved works like this:
14
+
15
+1. The server (`systemd-journal-remote`) validates that the client (`systemd-journal-upload`) uses a trusted certificate (a certificate issued by the same certificate authority as its own).
16
+ So, **the server will accept logs from any client having a valid certificate**.
17
+2. The client (`systemd-journal-upload`) validates that the receiver (`systemd-journal-remote`) uses a trusted certificate (like the server does) and it also checks that the hostname or IP of the URL specified to its configuration, matches one of the names or IPs of the server it gets connected to. So, **the client does a validation that it connected to the right server**, using the URL hostname against the names and IPs of the server on its certificate.
18
+
19
+This means, that if both certificates are issued by the same certificate authority, only the client can potentially reject the server.
20
+
21
+## Self-signed certificates
22
+
23
+To simplify the process of creating and managing self-signed certificates, we have created [this bash script](https://github.com/netdata/netdata/blob/master/src/collectors/systemd-journal.plugin/systemd-journal-self-signed-certs.sh).
24
+
25
+This helps to also automate the distribution of the certificates to your servers (it generates a new bash script for each of your servers, which includes everything required, including the certificates).
26
+
27
+We suggest to keep this script and all the involved certificates at the journals centralization server, in the directory `/etc/ssl/systemd-journal`, so that you can make future changes as required. If you prefer to keep the certificate authority and all the certificates at a more secure location, just use the script on that location.
28
+
29
+On the server that will issue the certificates (usually the centralizaton server), do the following:
30
+
31
+```bash
32
+# install systemd-journal-remote to add the users and groups required and openssl for the certs
33
+# change this according to your distro
34
+sudo apt-get install systemd-journal-remote openssl
35
+
36
+# download the script and make it executable
37
+curl >systemd-journal-self-signed-certs.sh "https://raw.githubusercontent.com/netdata/netdata/master/src/collectors/systemd-journal.plugin/systemd-journal-self-signed-certs.sh"
38
+chmod 750 systemd-journal-self-signed-certs.sh
39
+```
40
+
41
+To create certificates for your servers, run this:
42
+
43
+```bash
44
+sudo ./systemd-journal-self-signed-certs.sh "server1" "DNS:hostname1" "IP:10.0.0.1"
45
+```
46
+
47
+Where:
48
+
49
+- `server1` is the canonical name of the server. On newer systemd version, this name will be used by `systemd-journal-remote` and Netdata when you view the logs on the dashboard.
50
+- `DNS:hostname1` is a DNS name that the server is reachable at. Add `"DNS:xyz"` multiple times to define multiple DNS names for the server.
51
+- `IP:10.0.0.1` is an IP that the server is reachable at. Add `"IP:xyz"` multiple times to define multiple IPs for the server.
52
+
53
+Repeat this process to create the certificates for all your servers. You can add servers as required, at any time in the future.
54
+
55
+Existing certificates are never re-generated. Typically certificates need to be revoked and new ones to be issued. But `systemd-journal-remote` tools do not support handling revocations. So, the only option you have to re-issue a certificate is to delete its files in `/etc/ssl/systemd-journal` and run the script again to create a new one.
56
+
57
+Once you run the script of each of your servers, in `/etc/ssl/systemd-journal` you will find shell scripts named `runme-on-XXX.sh`, where `XXX` are the canonical names of your servers.
58
+
59
+These `runme-on-XXX.sh` include everything to install the certificates, fix their file permissions to be accessible by `systemd-journal-remote` and `systemd-journal-upload`, and update `/etc/systemd/journal-remote.conf` and `/etc/systemd/journal-upload.conf`.
60
+
61
+You can copy and paste (or `scp`) these scripts on your server and each of your clients:
62
+
63
+```bash
64
+sudo scp /etc/ssl/systemd-journal/runme-on-XXX.sh XXX:/tmp/
65
+```
66
+
67
+For the rest of this guide, we assume that you have copied the right `runme-on-XXX.sh` at the `/tmp` of all the servers for which you issued certificates.
68
+
69
+### note about certificates file permissions
70
+
71
+It is worth noting that `systemd-journal` certificates need to be owned by `systemd-journal-remote:systemd-journal`.
72
+
73
+Both the user `systemd-journal-remote` and the group `systemd-journal` are automatically added by the `systemd-journal-remote` package. However, `systemd-journal-upload` (and `systemd-journal-gatewayd` - that is not used in this guide) use dynamic users. Thankfully they are added to the `systemd-journal` remote group.
74
+
75
+So, by having the certificates owned by `systemd-journal-remote:systemd-journal`, satisfies both `systemd-journal-remote` which is not in the `systemd-journal` group, and `systemd-journal-upload` (and `systemd-journal-gatewayd`) which use dynamic users.
76
+
77
+You don't need to do anything about it (the scripts take care of everything), but it is worth noting how this works.
78
+
79
+## Server configuration
80
+
81
+On the centralization server install `systemd-journal-remote`:
82
+
83
+```bash
84
+# change this according to your distro
85
+sudo apt-get install systemd-journal-remote
86
+```
87
+
88
+Make sure the journal transfer protocol is `https`:
89
+
90
+```bash
91
+sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/
92
+
93
+# edit it to make sure it says:
94
+# --listen-https=-3
95
+# not:
96
+# --listen-http=-3
97
+sudo nano /etc/systemd/system/systemd-journal-remote.service
98
+
99
+# reload systemd
100
+sudo systemctl daemon-reload
101
+```
102
+
103
+Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`
104
+
105
+```bash
106
+# edit the socket file
107
+sudo systemctl edit systemd-journal-remote.socket
108
+```
109
+
110
+and add the following lines into the instructed place, and choose your desired port; save and exit.
111
+
112
+```bash
113
+[Socket]
114
+ListenStream=<DESIRED_PORT>
115
+```
116
+
117
+Next, run the `runme-on-XXX.sh` script on the server:
118
+
119
+```bash
120
+# if you run the certificate authority on the server:
121
+sudo /etc/ssl/systemd-journal/runme-on-XXX.sh
122
+
123
+# if you run the certificate authority elsewhere,
124
+# assuming you have coped the runme-on-XXX.sh script (as described above):
125
+sudo bash /tmp/runme-on-XXX.sh
126
+```
127
+
128
+This will install the certificates in `/etc/ssl/systemd-journal`, set the right file permissions, and update `/etc/systemd/journal-remote.conf` and `/etc/systemd/journal-upload.conf` to use the right certificate files.
129
+
130
+Finally, enable it, so that it will start automatically upon receiving a connection:
131
+
132
+```bash
133
+# enable systemd-journal-remote
134
+sudo systemctl enable --now systemd-journal-remote.socket
135
+sudo systemctl enable systemd-journal-remote.service
136
+```
137
+
138
+`systemd-journal-remote` is now listening for incoming journals from remote hosts.
139
+
140
+> When done, remember to `rm /tmp/runme-on-*.sh` to make sure your certificates are secure.
141
+
142
+## Client configuration
143
+
144
+On the clients, install `systemd-journal-remote` (it includes `systemd-journal-upload`):
145
+
146
+```bash
147
+# change this according to your distro
148
+sudo apt-get install systemd-journal-remote
149
+```
150
+
151
+Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:
152
+
153
+```text
154
+[Upload]
155
+URL=https://centralization.server.ip:19532
156
+```
157
+
158
+Make sure that `centralization.server.ip` is one of the `DNS:` or `IP:` parameters you defined when you created the centralization server certificates. If it is not, the client may reject to connect.
159
+
160
+Next, edit `systemd-journal-upload.service`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:
161
+
162
+```bash
163
+sudo systemctl edit systemd-journal-upload.service
164
+```
165
+
166
+At the top, add:
167
+
168
+```text
169
+[Service]
170
+Restart=always
171
+```
172
+
173
+Enable `systemd-journal-upload.service`, like this:
174
+
175
+```bash
176
+sudo systemctl enable systemd-journal-upload.service
177
+```
178
+
179
+Assuming that you have in `/tmp` the relevant `runme-on-XXX.sh` script for this client, run:
180
+
181
+```bash
182
+sudo bash /tmp/runme-on-XXX.sh
183
+```
184
+
185
+This will install the certificates in `/etc/ssl/systemd-journal`, set the right file permissions, and update `/etc/systemd/journal-remote.conf` and `/etc/systemd/journal-upload.conf` to use the right certificate files.
186
+
187
+Finally, restart `systemd-journal-upload.service`:
188
+
189
+```bash
190
+sudo systemctl restart systemd-journal-upload.service
191
+```
192
+
193
+The client should now be pushing logs to the central server.
194
+
195
+> When done, remember to `rm /tmp/runme-on-*.sh` to make sure your certificates are secure.
196
+
197
+Here it is in action, in Netdata:
198
+
199
+
200
+
201
+## Verify it works
202
+
203
+To verify the central server is receiving logs, run this on the central server:
204
+
205
+```bash
206
+sudo ls -l /var/log/journal/remote/
207
+```
208
+
209
+Depending on the `systemd` version you use, you should see new files from the clients' canonical names (as defined at their certificates) or IPs.
210
+
211
+Also, `systemctl status systemd-journal-remote` should show something like this:
212
+
213
+```bash
214
+systemd-journal-remote.service - Journal Remote Sink Service
215
+ Loaded: loaded (/etc/systemd/system/systemd-journal-remote.service; indirect; preset: disabled)
216
+ Active: active (running) since Sun 2023-10-15 14:29:46 EEST; 2h 24min ago
217
+TriggeredBy: ● systemd-journal-remote.socket
218
+ Docs: man:systemd-journal-remote(8)
219
+ man:journal-remote.conf(5)
220
+ Main PID: 2118153 (systemd-journal)
221
+ Status: "Processing requests..."
222
+ Tasks: 1 (limit: 154152)
223
+ Memory: 2.2M
224
+ CPU: 71ms
225
+ CGroup: /system.slice/systemd-journal-remote.service
226
+ └─2118153 /usr/lib/systemd/systemd-journal-remote --listen-https=-3 --output=/var/log/journal/remote/
227
+```
228
+
229
+Note the `status: "Processing requests..."` and the PID under `CGroup`.
230
+
231
+On the client `systemctl status systemd-journal-upload` should show something like this:
232
+
233
+```bash
234
+● systemd-journal-upload.service - Journal Remote Upload Service
235
+ Loaded: loaded (/lib/systemd/system/systemd-journal-upload.service; enabled; vendor preset: disabled)
236
+ Drop-In: /etc/systemd/system/systemd-journal-upload.service.d
237
+ └─override.conf
238
+ Active: active (running) since Sun 2023-10-15 10:39:04 UTC; 3h 17min ago
239
+ Docs: man:systemd-journal-upload(8)
240
+ Main PID: 4169 (systemd-journal)
241
+ Status: "Processing input..."
242
+ Tasks: 1 (limit: 13868)
243
+ Memory: 3.5M
244
+ CPU: 1.081s
245
+ CGroup: /system.slice/systemd-journal-upload.service
246
+ └─4169 /lib/systemd/systemd-journal-upload --save-state
247
+```
248
+
249
+Note the `Status: "Processing input..."` and the PID under `CGroup`.
src/collectors/systemd-journal.plugin/provider/netdata_provider.c
new
+182
@@ -0,0 +1,182 @@
1
+#include "netdata_provider.h"
2
+
3
+int32_t nsd_id128_from_string(const char *s, NsdId128 *ret)
4
+{
5
+#if defined(HAVE_RUST_PROVIDER)
6
+ return rsd_id128_from_string(s, (struct RsdId128 *) ret);
7
+#else
8
+ return sd_id128_from_string(s, (sd_id128_t *) ret);
9
+#endif
10
+}
11
+
12
+int32_t nsd_id128_equal(NsdId128 a, NsdId128 b)
13
+{
14
+#if defined(HAVE_RUST_PROVIDER)
15
+ return rsd_id128_equal(a, b);
16
+#else
17
+ return sd_id128_equal(a, b);
18
+#endif
19
+}
20
+
21
+int nsd_journal_open_files(NsdJournal **ret, const char *const *paths, int flags)
22
+{
23
+#if defined(HAVE_RUST_PROVIDER)
24
+ return rsd_journal_open_files(ret, paths, flags);
25
+#else
26
+ return sd_journal_open_files(ret, paths, flags);
27
+#endif
28
+}
29
+
30
+void nsd_journal_close(NsdJournal *j)
31
+{
32
+#if defined(HAVE_RUST_PROVIDER)
33
+ rsd_journal_close(j);
34
+#else
35
+ sd_journal_close(j);
36
+#endif
37
+}
38
+
39
+int nsd_journal_seek_head(NsdJournal *j)
40
+{
41
+#if defined(HAVE_RUST_PROVIDER)
42
+ return rsd_journal_seek_head(j);
43
+#else
44
+ return sd_journal_seek_head(j);
45
+#endif
46
+}
47
+
48
+int nsd_journal_seek_tail(NsdJournal *j)
49
+{
50
+#if defined(HAVE_RUST_PROVIDER)
51
+ return rsd_journal_seek_tail(j);
52
+#else
53
+ return sd_journal_seek_tail(j);
54
+#endif
55
+}
56
+
57
+int nsd_journal_seek_realtime_usec(NsdJournal *j, uint64_t usec)
58
+{
59
+#if defined(HAVE_RUST_PROVIDER)
60
+ return rsd_journal_seek_realtime_usec(j, usec);
61
+#else
62
+ return sd_journal_seek_realtime_usec(j, usec);
63
+#endif
64
+}
65
+
66
+int nsd_journal_next(NsdJournal *j)
67
+{
68
+#if defined(HAVE_RUST_PROVIDER)
69
+ return rsd_journal_next(j);
70
+#else
71
+ return sd_journal_next(j);
72
+#endif
73
+}
74
+
75
+int nsd_journal_previous(NsdJournal *j)
76
+{
77
+#if defined(HAVE_RUST_PROVIDER)
78
+ return rsd_journal_previous(j);
79
+#else
80
+ return sd_journal_previous(j);
81
+#endif
82
+}
83
+
84
+#if defined(HAVE_SD_JOURNAL_GET_SEQNUM)
85
+int nsd_journal_get_seqnum(NsdJournal *j, uint64_t *ret_seqnum, NsdId128 *ret_seqnum_id)
86
+{
87
+#if defined(HAVE_RUST_PROVIDER)
88
+ return rsd_journal_get_seqnum(j, ret_seqnum, ret_seqnum_id);
89
+#else
90
+ return sd_journal_get_seqnum(j, ret_seqnum, ret_seqnum_id);
91
+#endif
92
+}
93
+#endif /* HAVE_SD_JOURNAL_GET_SEQNUM */
94
+
95
+int nsd_journal_get_realtime_usec(NsdJournal *j, uint64_t *ret)
96
+{
97
+#if defined(HAVE_RUST_PROVIDER)
98
+ return rsd_journal_get_realtime_usec(j, ret);
99
+#else
100
+ return sd_journal_get_realtime_usec(j, ret);
101
+#endif
102
+}
103
+
104
+#if defined(HAVE_SD_JOURNAL_RESTART_FIELDS)
105
+int nsd_journal_enumerate_fields(NsdJournal *j, const char **field)
106
+{
107
+#if defined(HAVE_RUST_PROVIDER)
108
+ return rsd_journal_enumerate_fields(j, field);
109
+#else
110
+ return sd_journal_enumerate_fields(j, field);
111
+#endif
112
+}
113
+#endif /* HAVE_SD_JOURNAL_RESTART_FIELDS */
114
+
115
+#if defined(HAVE_SD_JOURNAL_RESTART_FIELDS)
116
+void nsd_journal_restart_fields(NsdJournal *j)
117
+{
118
+#if defined(HAVE_RUST_PROVIDER)
119
+ rsd_journal_restart_fields(j);
120
+#else
121
+ sd_journal_restart_fields(j);
122
+#endif
123
+}
124
+#endif /* HAVE_SD_JOURNAL_RESTART_FIELDS */
125
+
126
+#if defined(HAVE_SD_JOURNAL_RESTART_FIELDS)
127
+int nsd_journal_query_unique(NsdJournal *j, const char *field)
128
+{
129
+#if defined(HAVE_RUST_PROVIDER)
130
+ return rsd_journal_query_unique(j, field);
131
+#else
132
+ return sd_journal_query_unique(j, field);
133
+#endif
134
+}
135
+#endif /* HAVE_SD_JOURNAL_RESTART_FIELDS */
136
+
137
+#if defined(HAVE_SD_JOURNAL_RESTART_FIELDS)
138
+void nsd_journal_restart_unique(NsdJournal *j)
139
+{
140
+#if defined(HAVE_RUST_PROVIDER)
141
+ rsd_journal_restart_unique(j);
142
+#else
143
+ sd_journal_restart_unique(j);
144
+#endif
145
+}
146
+#endif /* HAVE_SD_JOURNAL_RESTART_FIELDS */
147
+
148
+int nsd_journal_add_match(NsdJournal *j, const void *data, uintptr_t size)
149
+{
150
+#if defined(HAVE_RUST_PROVIDER)
151
+ return rsd_journal_add_match(j, data, size);
152
+#else
153
+ return sd_journal_add_match(j, data, size);
154
+#endif
155
+}
156
+
157
+int nsd_journal_add_conjunction(NsdJournal *j)
158
+{
159
+#if defined(HAVE_RUST_PROVIDER)
160
+ return rsd_journal_add_conjunction(j);
161
+#else
162
+ return sd_journal_add_conjunction(j);
163
+#endif
164
+}
165
+
166
+int nsd_journal_add_disjunction(NsdJournal *j)
167
+{
168
+#if defined(HAVE_RUST_PROVIDER)
169
+ return rsd_journal_add_disjunction(j);
170
+#else
171
+ return sd_journal_add_disjunction(j);
172
+#endif
173
+}
174
+
175
+void nsd_journal_flush_matches(NsdJournal *j)
176
+{
177
+#if defined(HAVE_RUST_PROVIDER)
178
+ rsd_journal_flush_matches(j);
179
+#else
180
+ sd_journal_flush_matches(j);
181
+#endif
182
+}
src/collectors/systemd-journal.plugin/provider/netdata_provider.h
new
+78
@@ -0,0 +1,78 @@
1
+#ifndef ND_SD_JOURNAL_PROVIDER_NETDATA_H
2
+#define ND_SD_JOURNAL_PROVIDER_NETDATA_H
3
+
4
+#ifdef __cplusplus
5
+extern "C" {
6
+#endif /* __cplusplus */
7
+
8
+#include "config.h"
9
+
10
+#if defined(HAVE_RUST_PROVIDER)
11
+ #include "rust_provider.h"
12
+#else
13
+ #include <systemd/sd-journal.h>
14
+#endif
15
+
16
+#if defined(HAVE_RUST_PROVIDER)
17
+ typedef struct RsdJournal NsdJournal;
18
+
19
+ #define NSD_JOURNAL_FOREACH_DATA(j, data, l) RSD_JOURNAL_FOREACH_DATA(j, data, l)
20
+ #define NSD_JOURNAL_FOREACH_UNIQUE(j, data, l) RSD_JOURNAL_FOREACH_UNIQUE(j, data, l)
21
+ #define NSD_JOURNAL_FOREACH_FIELD(j, field) RSD_JOURNAL_FOREACH_FIELD(j, field)
22
+
23
+ typedef struct RsdId128 NsdId128;
24
+
25
+ #define NSD_ID128_NULL RSD_ID128_NULL
26
+ #define NSD_ID128_STRING_MAX RSD_ID128_STRING_MAX
27
+ #define NSD_ID128_UUID_STRING_MAX RSD_ID128_UUID_STRING_MAX
28
+#else
29
+ typedef struct sd_journal NsdJournal;
30
+ typedef sd_id128_t NsdId128;
31
+
32
+ #define NSD_ID128_NULL SD_ID128_NULL
33
+ #define NSD_ID128_STRING_MAX SD_ID128_STRING_MAX
34
+ #define NSD_ID128_UUID_STRING_MAX SD_ID128_UUID_STRING_MAX
35
+
36
+ #define NSD_JOURNAL_FOREACH_DATA(j, data, l) SD_JOURNAL_FOREACH_DATA(j, data, l)
37
+ #if defined(HAVE_SD_JOURNAL_RESTART_FIELDS)
38
+ #define NSD_JOURNAL_FOREACH_UNIQUE(j, data, l) SD_JOURNAL_FOREACH_UNIQUE(j, data, l)
39
+ #define NSD_JOURNAL_FOREACH_FIELD(j, field) SD_JOURNAL_FOREACH_FIELD(j, field)
40
+ #endif
41
+#endif
42
+
43
+int32_t nsd_id128_from_string(const char *s, NsdId128 *ret);
44
+int32_t nsd_id128_equal(NsdId128 a, NsdId128 b);
45
+
46
+int nsd_journal_open_files(NsdJournal **ret, const char *const *paths, int flags);
47
+void nsd_journal_close(NsdJournal *j);
48
+
49
+int nsd_journal_seek_head(NsdJournal *j);
50
+int nsd_journal_seek_tail(NsdJournal *j);
51
+int nsd_journal_seek_realtime_usec(NsdJournal *j, uint64_t usec);
52
+
53
+int nsd_journal_next(NsdJournal *j);
54
+int nsd_journal_previous(NsdJournal *j);
55
+
56
+#if defined(HAVE_SD_JOURNAL_GET_SEQNUM)
57
+int nsd_journal_get_seqnum(NsdJournal *j, uint64_t *ret_seqnum, NsdId128 *ret_seqnum_id);
58
+#endif
59
+int nsd_journal_get_realtime_usec(NsdJournal *j, uint64_t *ret);
60
+
61
+#if defined(HAVE_SD_JOURNAL_RESTART_FIELDS)
62
+int nsd_journal_enumerate_fields(NsdJournal *j, const char **field);
63
+void nsd_journal_restart_fields(NsdJournal *j);
64
+
65
+int nsd_journal_query_unique(NsdJournal *j, const char *field);
66
+void nsd_journal_restart_unique(NsdJournal *j);
67
+#endif /* HAVE_SD_JOURNAL_RESTART_FIELDS */
68
+
69
+int nsd_journal_add_match(NsdJournal *j, const void *data, uintptr_t size);
70
+int nsd_journal_add_conjunction(NsdJournal *j);
71
+int nsd_journal_add_disjunction(NsdJournal *j);
72
+void nsd_journal_flush_matches(NsdJournal *j);
73
+
74
+#ifdef __cplusplus
75
+}
76
+#endif /* __cplusplus */
77
+
78
+#endif /* ND_SD_JOURNAL_PROVIDER_NETDATA_H */
src/collectors/systemd-journal.plugin/provider/rust_provider.h
new
+27
@@ -0,0 +1,27 @@
1
+#ifndef ND_SD_JOURNAL_PROVIDER_RUST_H
2
+#define ND_SD_JOURNAL_PROVIDER_RUST_H
3
+
4
+#include "crates/jf/journal_reader_ffi/journal_reader_ffi.h"
5
+
6
+#define RSD_ID128_NULL ((const RsdId128){.bytes = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}})
7
+#define RSD_ID128_STRING_MAX 33U
8
+#define RSD_ID128_UUID_STRING_MAX 37U
9
+
10
+#define RSD_JOURNAL_FOREACH_DATA(j, data, l) \
11
+ for (rsd_journal_restart_data(j); rsd_journal_enumerate_available_data((j), &(data), &(l)) > 0;)
12
+
13
+#define RSD_JOURNAL_FOREACH_UNIQUE(j, data, l) \
14
+ for (rsd_journal_restart_unique(j); rsd_journal_enumerate_available_unique((j), &(data), &(l)) > 0;)
15
+
16
+#define RSD_JOURNAL_FOREACH_FIELD(j, field) \
17
+ for (rsd_journal_restart_fields(j); rsd_journal_enumerate_fields((j), &(field)) > 0;)
18
+
19
+#ifndef HAVE_SD_JOURNAL_RESTART_FIELDS
20
+#define HAVE_SD_JOURNAL_RESTART_FIELDS 1
21
+#endif
22
+
23
+#ifndef HAVE_SD_JOURNAL_GET_SEQNUM
24
+#define HAVE_SD_JOURNAL_GET_SEQNUM 1
25
+#endif
26
+
27
+#endif /* ND_SD_JOURNAL_PROVIDER_RUST_H */
src/collectors/systemd-journal.plugin/schema.d/systemd-journal%3Amonitored-directories.json
new
+38
@@ -0,0 +1,38 @@
1
+{
2
+ "jsonSchema": {
3
+ "$schema": "http://json-schema.org/draft-07/schema#",
4
+ "type": "object",
5
+ "properties": {
6
+ "journalDirectories": {
7
+ "title": "systemd-journal directories",
8
+ "description": "The list of directories `systemd-journald` and `systemd-journal-remote` store journal files. Netdata monitors these directories to automatically detect changes.",
9
+ "type": "array",
10
+ "items": {
11
+ "title": "Absolute Path",
12
+ "type": "string",
13
+ "pattern": "^/.+$"
14
+ },
15
+ "maxItems": 100,
16
+ "uniqueItems": true
17
+ }
18
+ },
19
+ "required": [
20
+ "journalDirectories"
21
+ ]
22
+ },
23
+ "uiSchema": {
24
+ "journalDirectories": {
25
+ "ui:listFlavour": "list",
26
+ "ui:options": {
27
+ "addable": true,
28
+ "orderable": false,
29
+ "removable": true
30
+ },
31
+ "items": {
32
+ "ui:placeholder": "Enter absolute directory path",
33
+ "ui:widget": "text",
34
+ "ui:emptyValue": ""
35
+ }
36
+ }
37
+ }
38
+}
src/collectors/systemd-journal.plugin/systemd-internals.h
new
+174
@@ -0,0 +1,174 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_COLLECTORS_SYSTEMD_INTERNALS_H
4
+#define NETDATA_COLLECTORS_SYSTEMD_INTERNALS_H
5
+
6
+#include "collectors/all.h"
7
+#include "libnetdata/libnetdata.h"
8
+#include "provider/netdata_provider.h"
9
+
10
+#include <linux/capability.h>
11
+#include <syslog.h>
12
+
13
+#define ND_SD_JOURNAL_FUNCTION_DESCRIPTION "View, search and analyze systemd journal entries."
14
+#define ND_SD_JOURNAL_FUNCTION_NAME "systemd-journal"
15
+#define ND_SD_JOURNAL_DEFAULT_TIMEOUT 60
16
+#define ND_SD_JOURNAL_ENABLE_ESTIMATIONS_FILE_PERCENTAGE 0.01
17
+#define ND_SD_JOURNAL_EXECUTE_WATCHER_PENDING_EVERY_MS 250
18
+#define ND_SD_JOURNAL_ALL_FILES_SCAN_EVERY_USEC (5 * 60 * USEC_PER_SEC)
19
+
20
+#define ND_SD_UNITS_FUNCTION_DESCRIPTION "Lists all systemd units (services, timers, mounts, etc.) with their current state and status."
21
+#define ND_SD_UNITS_FUNCTION_NAME "systemd-list-units"
22
+#define ND_SD_UNITS_DEFAULT_TIMEOUT 30
23
+
24
+extern __thread size_t fstat_thread_calls;
25
+extern __thread size_t fstat_thread_cached_responses;
26
+void fstat_cache_enable_on_thread(void);
27
+void fstat_cache_disable_on_thread(void);
28
+
29
+extern netdata_mutex_t stdout_mutex;
30
+
31
+typedef enum {
32
+ ND_SD_JOURNAL_NO_FILE_MATCHED,
33
+ ND_SD_JOURNAL_FAILED_TO_OPEN,
34
+ ND_SD_JOURNAL_FAILED_TO_SEEK,
35
+ ND_SD_JOURNAL_TIMED_OUT,
36
+ ND_SD_JOURNAL_OK,
37
+ ND_SD_JOURNAL_NOT_MODIFIED,
38
+ ND_SD_JOURNAL_CANCELLED,
39
+} ND_SD_JOURNAL_STATUS;
40
+
41
+typedef enum {
42
+ ND_SD_JF_NONE = 0,
43
+ ND_SD_JF_ALL = (1 << 0),
44
+ ND_SD_JF_LOCAL_ALL = (1 << 1),
45
+ ND_SD_JF_REMOTE_ALL = (1 << 2),
46
+ ND_SD_JF_LOCAL_SYSTEM = (1 << 3),
47
+ ND_SD_JF_LOCAL_USER = (1 << 4),
48
+ ND_SD_JF_LOCAL_NAMESPACE = (1 << 5),
49
+ ND_SD_JF_LOCAL_OTHER = (1 << 6),
50
+} SD_JOURNAL_FILE_SOURCE_TYPE;
51
+
52
+struct nd_journal_file {
53
+ const char *filename;
54
+ size_t filename_len;
55
+ STRING *source;
56
+ SD_JOURNAL_FILE_SOURCE_TYPE source_type;
57
+ usec_t file_last_modified_ut;
58
+ usec_t msg_first_ut;
59
+ usec_t msg_last_ut;
60
+ size_t size;
61
+ bool logged_failure;
62
+ bool logged_journalctl_failure;
63
+ usec_t max_journal_vs_realtime_delta_ut;
64
+
65
+ usec_t last_scan_monotonic_ut;
66
+ usec_t last_scan_header_vs_last_modified_ut;
67
+
68
+ uint64_t first_seqnum;
69
+ uint64_t last_seqnum;
70
+ NsdId128 first_writer_id;
71
+ NsdId128 last_writer_id;
72
+
73
+ uint64_t messages_in_file;
74
+};
75
+
76
+#define ND_SD_JF_SOURCE_ALL_NAME "all"
77
+#define ND_SD_JF_SOURCE_LOCAL_NAME "all-local-logs"
78
+#define ND_SD_JF_SOURCE_LOCAL_SYSTEM_NAME "all-local-system-logs"
79
+#define ND_SD_JF_SOURCE_LOCAL_USERS_NAME "all-local-user-logs"
80
+#define ND_SD_JF_SOURCE_LOCAL_OTHER_NAME "all-uncategorized"
81
+#define ND_SD_JF_SOURCE_NAMESPACES_NAME "all-local-namespaces"
82
+#define ND_SD_JF_SOURCE_REMOTES_NAME "all-remote-systems"
83
+
84
+#define ND_SD_JOURNAL_OPEN_FLAGS (0)
85
+
86
+#define JOURNAL_VS_REALTIME_DELTA_DEFAULT_UT (5 * USEC_PER_SEC) // assume a 5-seconds latency
87
+#define JOURNAL_VS_REALTIME_DELTA_MAX_UT (2 * 60 * USEC_PER_SEC) // up to 2-minutes latency
88
+
89
+extern DICTIONARY *nd_journal_files_registry;
90
+extern DICTIONARY *used_hashes_registry;
91
+extern DICTIONARY *boot_ids_to_first_ut;
92
+
93
+int nd_journal_file_dict_items_backward_compar(const void *a, const void *b);
94
+int nd_journal_file_dict_items_forward_compar(const void *a, const void *b);
95
+void buffer_json_journal_versions(BUFFER *wb);
96
+void available_journal_file_sources_to_json_array(BUFFER *wb);
97
+bool nd_journal_files_completed_once(void);
98
+void nd_journal_files_registry_update(void);
99
+void nd_journal_directory_scan_recursively(DICTIONARY *files, DICTIONARY *dirs, const char *dirname, int depth);
100
+
101
+FACET_ROW_SEVERITY syslog_priority_to_facet_severity(FACETS *facets, FACET_ROW *row, void *data);
102
+
103
+void nd_sd_journal_dynamic_row_id(
104
+ FACETS *facets,
105
+ BUFFER *json_array,
106
+ FACET_ROW_KEY_VALUE *rkv,
107
+ FACET_ROW *row,
108
+ void *data);
109
+void nd_sd_journal_transform_priority(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
110
+void nd_sd_journal_transform_syslog_facility(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
111
+void nd_sd_journal_transform_errno(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
112
+void nd_sd_journal_transform_boot_id(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
113
+void nd_sd_journal_transform_uid(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
114
+void nd_sd_journal_transform_gid(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
115
+void nd_sd_journal_transform_cap_effective(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
116
+void nd_sd_journal_transform_timestamp_usec(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
117
+
118
+usec_t nd_journal_file_update_annotation_boot_id(NsdJournal *j, struct nd_journal_file *njf, const char *boot_id);
119
+
120
+#define MAX_JOURNAL_DIRECTORIES 100
121
+struct journal_directory {
122
+ STRING *path;
123
+};
124
+extern struct journal_directory journal_directories[MAX_JOURNAL_DIRECTORIES];
125
+
126
+void nd_journal_init_files_and_directories(void);
127
+void function_systemd_journal(
128
+ const char *transaction,
129
+ char *function,
130
+ usec_t *stop_monotonic_ut,
131
+ bool *cancelled,
132
+ BUFFER *payload,
133
+ HTTP_ACCESS access __maybe_unused,
134
+ const char *source,
135
+ void *data);
136
+void nd_journal_file_update_header(const char *filename, struct nd_journal_file *njf);
137
+
138
+void nd_sd_journal_annotations_init(void);
139
+void nd_sd_journal_transform_message_id(FACETS *facets, BUFFER *wb, FACETS_TRANSFORMATION_SCOPE scope, void *data);
140
+
141
+void nd_journal_watcher_main(void *arg);
142
+void nd_journal_watcher_restart(void);
143
+
144
+static inline bool parse_journal_field(
145
+ const char *data,
146
+ size_t data_length,
147
+ const char **key,
148
+ size_t *key_length,
149
+ const char **value,
150
+ size_t *value_length)
151
+{
152
+ const char *k = data;
153
+ const char *equal = strchr(k, '=');
154
+ if (unlikely(!equal))
155
+ return false;
156
+
157
+ size_t kl = equal - k;
158
+
159
+ const char *v = ++equal;
160
+ size_t vl = data_length - kl - 1;
161
+
162
+ *key = k;
163
+ *key_length = kl;
164
+ *value = v;
165
+ *value_length = vl;
166
+
167
+ return true;
168
+}
169
+
170
+void nd_systemd_journal_dyncfg_init(struct functions_evloop_globals *wg);
171
+
172
+bool is_journal_file(const char *filename, ssize_t len, const char **start_of_extension);
173
+
174
+#endif //NETDATA_COLLECTORS_SYSTEMD_INTERNALS_H
src/collectors/systemd-journal.plugin/systemd-journal-annotations.c
new
+737
@@ -0,0 +1,737 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "systemd-internals.h"
4
+
5
+const char *errno_map[] = {
6
+ [1] = "1 (EPERM)", // "Operation not permitted",
7
+ [2] = "2 (ENOENT)", // "No such file or directory",
8
+ [3] = "3 (ESRCH)", // "No such process",
9
+ [4] = "4 (EINTR)", // "Interrupted system call",
10
+ [5] = "5 (EIO)", // "Input/output error",
11
+ [6] = "6 (ENXIO)", // "No such device or address",
12
+ [7] = "7 (E2BIG)", // "Argument list too long",
13
+ [8] = "8 (ENOEXEC)", // "Exec format error",
14
+ [9] = "9 (EBADF)", // "Bad file descriptor",
15
+ [10] = "10 (ECHILD)", // "No child processes",
16
+ [11] = "11 (EAGAIN)", // "Resource temporarily unavailable",
17
+ [12] = "12 (ENOMEM)", // "Cannot allocate memory",
18
+ [13] = "13 (EACCES)", // "Permission denied",
19
+ [14] = "14 (EFAULT)", // "Bad address",
20
+ [15] = "15 (ENOTBLK)", // "Block device required",
21
+ [16] = "16 (EBUSY)", // "Device or resource busy",
22
+ [17] = "17 (EEXIST)", // "File exists",
23
+ [18] = "18 (EXDEV)", // "Invalid cross-device link",
24
+ [19] = "19 (ENODEV)", // "No such device",
25
+ [20] = "20 (ENOTDIR)", // "Not a directory",
26
+ [21] = "21 (EISDIR)", // "Is a directory",
27
+ [22] = "22 (EINVAL)", // "Invalid argument",
28
+ [23] = "23 (ENFILE)", // "Too many open files in system",
29
+ [24] = "24 (EMFILE)", // "Too many open files",
30
+ [25] = "25 (ENOTTY)", // "Inappropriate ioctl for device",
31
+ [26] = "26 (ETXTBSY)", // "Text file busy",
32
+ [27] = "27 (EFBIG)", // "File too large",
33
+ [28] = "28 (ENOSPC)", // "No space left on device",
34
+ [29] = "29 (ESPIPE)", // "Illegal seek",
35
+ [30] = "30 (EROFS)", // "Read-only file system",
36
+ [31] = "31 (EMLINK)", // "Too many links",
37
+ [32] = "32 (EPIPE)", // "Broken pipe",
38
+ [33] = "33 (EDOM)", // "Numerical argument out of domain",
39
+ [34] = "34 (ERANGE)", // "Numerical result out of range",
40
+ [35] = "35 (EDEADLK)", // "Resource deadlock avoided",
41
+ [36] = "36 (ENAMETOOLONG)", // "File name too long",
42
+ [37] = "37 (ENOLCK)", // "No locks available",
43
+ [38] = "38 (ENOSYS)", // "Function not implemented",
44
+ [39] = "39 (ENOTEMPTY)", // "Directory not empty",
45
+ [40] = "40 (ELOOP)", // "Too many levels of symbolic links",
46
+ [42] = "42 (ENOMSG)", // "No message of desired type",
47
+ [43] = "43 (EIDRM)", // "Identifier removed",
48
+ [44] = "44 (ECHRNG)", // "Channel number out of range",
49
+ [45] = "45 (EL2NSYNC)", // "Level 2 not synchronized",
50
+ [46] = "46 (EL3HLT)", // "Level 3 halted",
51
+ [47] = "47 (EL3RST)", // "Level 3 reset",
52
+ [48] = "48 (ELNRNG)", // "Link number out of range",
53
+ [49] = "49 (EUNATCH)", // "Protocol driver not attached",
54
+ [50] = "50 (ENOCSI)", // "No CSI structure available",
55
+ [51] = "51 (EL2HLT)", // "Level 2 halted",
56
+ [52] = "52 (EBADE)", // "Invalid exchange",
57
+ [53] = "53 (EBADR)", // "Invalid request descriptor",
58
+ [54] = "54 (EXFULL)", // "Exchange full",
59
+ [55] = "55 (ENOANO)", // "No anode",
60
+ [56] = "56 (EBADRQC)", // "Invalid request code",
61
+ [57] = "57 (EBADSLT)", // "Invalid slot",
62
+ [59] = "59 (EBFONT)", // "Bad font file format",
63
+ [60] = "60 (ENOSTR)", // "Device not a stream",
64
+ [61] = "61 (ENODATA)", // "No data available",
65
+ [62] = "62 (ETIME)", // "Timer expired",
66
+ [63] = "63 (ENOSR)", // "Out of streams resources",
67
+ [64] = "64 (ENONET)", // "Machine is not on the network",
68
+ [65] = "65 (ENOPKG)", // "Package not installed",
69
+ [66] = "66 (EREMOTE)", // "Object is remote",
70
+ [67] = "67 (ENOLINK)", // "Link has been severed",
71
+ [68] = "68 (EADV)", // "Advertise error",
72
+ [69] = "69 (ESRMNT)", // "Srmount error",
73
+ [70] = "70 (ECOMM)", // "Communication error on send",
74
+ [71] = "71 (EPROTO)", // "Protocol error",
75
+ [72] = "72 (EMULTIHOP)", // "Multihop attempted",
76
+ [73] = "73 (EDOTDOT)", // "RFS specific error",
77
+ [74] = "74 (EBADMSG)", // "Bad message",
78
+ [75] = "75 (EOVERFLOW)", // "Value too large for defined data type",
79
+ [76] = "76 (ENOTUNIQ)", // "Name not unique on network",
80
+ [77] = "77 (EBADFD)", // "File descriptor in bad state",
81
+ [78] = "78 (EREMCHG)", // "Remote address changed",
82
+ [79] = "79 (ELIBACC)", // "Can not access a needed shared library",
83
+ [80] = "80 (ELIBBAD)", // "Accessing a corrupted shared library",
84
+ [81] = "81 (ELIBSCN)", // ".lib section in a.out corrupted",
85
+ [82] = "82 (ELIBMAX)", // "Attempting to link in too many shared libraries",
86
+ [83] = "83 (ELIBEXEC)", // "Cannot exec a shared library directly",
87
+ [84] = "84 (EILSEQ)", // "Invalid or incomplete multibyte or wide character",
88
+ [85] = "85 (ERESTART)", // "Interrupted system call should be restarted",
89
+ [86] = "86 (ESTRPIPE)", // "Streams pipe error",
90
+ [87] = "87 (EUSERS)", // "Too many users",
91
+ [88] = "88 (ENOTSOCK)", // "Socket operation on non-socket",
92
+ [89] = "89 (EDESTADDRREQ)", // "Destination address required",
93
+ [90] = "90 (EMSGSIZE)", // "Message too long",
94
+ [91] = "91 (EPROTOTYPE)", // "Protocol wrong type for socket",
95
+ [92] = "92 (ENOPROTOOPT)", // "Protocol not available",
96
+ [93] = "93 (EPROTONOSUPPORT)", // "Protocol not supported",
97
+ [94] = "94 (ESOCKTNOSUPPORT)", // "Socket type not supported",
98
+ [95] = "95 (ENOTSUP)", // "Operation not supported",
99
+ [96] = "96 (EPFNOSUPPORT)", // "Protocol family not supported",
100
+ [97] = "97 (EAFNOSUPPORT)", // "Address family not supported by protocol",
101
+ [98] = "98 (EADDRINUSE)", // "Address already in use",
102
+ [99] = "99 (EADDRNOTAVAIL)", // "Cannot assign requested address",
103
+ [100] = "100 (ENETDOWN)", // "Network is down",
104
+ [101] = "101 (ENETUNREACH)", // "Network is unreachable",
105
+ [102] = "102 (ENETRESET)", // "Network dropped connection on reset",
106
+ [103] = "103 (ECONNABORTED)", // "Software caused connection abort",
107
+ [104] = "104 (ECONNRESET)", // "Connection reset by peer",
108
+ [105] = "105 (ENOBUFS)", // "No buffer space available",
109
+ [106] = "106 (EISCONN)", // "Transport endpoint is already connected",
110
+ [107] = "107 (ENOTCONN)", // "Transport endpoint is not connected",
111
+ [108] = "108 (ESHUTDOWN)", // "Cannot send after transport endpoint shutdown",
112
+ [109] = "109 (ETOOMANYREFS)", // "Too many references: cannot splice",
113
+ [110] = "110 (ETIMEDOUT)", // "Connection timed out",
114
+ [111] = "111 (ECONNREFUSED)", // "Connection refused",
115
+ [112] = "112 (EHOSTDOWN)", // "Host is down",
116
+ [113] = "113 (EHOSTUNREACH)", // "No route to host",
117
+ [114] = "114 (EALREADY)", // "Operation already in progress",
118
+ [115] = "115 (EINPROGRESS)", // "Operation now in progress",
119
+ [116] = "116 (ESTALE)", // "Stale file handle",
120
+ [117] = "117 (EUCLEAN)", // "Structure needs cleaning",
121
+ [118] = "118 (ENOTNAM)", // "Not a XENIX named type file",
122
+ [119] = "119 (ENAVAIL)", // "No XENIX semaphores available",
123
+ [120] = "120 (EISNAM)", // "Is a named type file",
124
+ [121] = "121 (EREMOTEIO)", // "Remote I/O error",
125
+ [122] = "122 (EDQUOT)", // "Disk quota exceeded",
126
+ [123] = "123 (ENOMEDIUM)", // "No medium found",
127
+ [124] = "124 (EMEDIUMTYPE)", // "Wrong medium type",
128
+ [125] = "125 (ECANCELED)", // "Operation canceled",
129
+ [126] = "126 (ENOKEY)", // "Required key not available",
130
+ [127] = "127 (EKEYEXPIRED)", // "Key has expired",
131
+ [128] = "128 (EKEYREVOKED)", // "Key has been revoked",
132
+ [129] = "129 (EKEYREJECTED)", // "Key was rejected by service",
133
+ [130] = "130 (EOWNERDEAD)", // "Owner died",
134
+ [131] = "131 (ENOTRECOVERABLE)", // "State not recoverable",
135
+ [132] = "132 (ERFKILL)", // "Operation not possible due to RF-kill",
136
+ [133] = "133 (EHWPOISON)", // "Memory page has hardware error",
137
+};
138
+
139
+const char *linux_capabilities[] = {
140
+ [CAP_CHOWN] = "CHOWN",
141
+ [CAP_DAC_OVERRIDE] = "DAC_OVERRIDE",
142
+ [CAP_DAC_READ_SEARCH] = "DAC_READ_SEARCH",
143
+ [CAP_FOWNER] = "FOWNER",
144
+ [CAP_FSETID] = "FSETID",
145
+ [CAP_KILL] = "KILL",
146
+ [CAP_SETGID] = "SETGID",
147
+ [CAP_SETUID] = "SETUID",
148
+ [CAP_SETPCAP] = "SETPCAP",
149
+ [CAP_LINUX_IMMUTABLE] = "LINUX_IMMUTABLE",
150
+ [CAP_NET_BIND_SERVICE] = "NET_BIND_SERVICE",
151
+ [CAP_NET_BROADCAST] = "NET_BROADCAST",
152
+ [CAP_NET_ADMIN] = "NET_ADMIN",
153
+ [CAP_NET_RAW] = "NET_RAW",
154
+ [CAP_IPC_LOCK] = "IPC_LOCK",
155
+ [CAP_IPC_OWNER] = "IPC_OWNER",
156
+ [CAP_SYS_MODULE] = "SYS_MODULE",
157
+ [CAP_SYS_RAWIO] = "SYS_RAWIO",
158
+ [CAP_SYS_CHROOT] = "SYS_CHROOT",
159
+ [CAP_SYS_PTRACE] = "SYS_PTRACE",
160
+ [CAP_SYS_PACCT] = "SYS_PACCT",
161
+ [CAP_SYS_ADMIN] = "SYS_ADMIN",
162
+ [CAP_SYS_BOOT] = "SYS_BOOT",
163
+ [CAP_SYS_NICE] = "SYS_NICE",
164
+ [CAP_SYS_RESOURCE] = "SYS_RESOURCE",
165
+ [CAP_SYS_TIME] = "SYS_TIME",
166
+ [CAP_SYS_TTY_CONFIG] = "SYS_TTY_CONFIG",
167
+ [CAP_MKNOD] = "MKNOD",
168
+ [CAP_LEASE] = "LEASE",
169
+ [CAP_AUDIT_WRITE] = "AUDIT_WRITE",
170
+ [CAP_AUDIT_CONTROL] = "AUDIT_CONTROL",
171
+ [CAP_SETFCAP] = "SETFCAP",
172
+ [CAP_MAC_OVERRIDE] = "MAC_OVERRIDE",
173
+ [CAP_MAC_ADMIN] = "MAC_ADMIN",
174
+ [CAP_SYSLOG] = "SYSLOG",
175
+ [CAP_WAKE_ALARM] = "WAKE_ALARM",
176
+ [CAP_BLOCK_SUSPEND] = "BLOCK_SUSPEND",
177
+ [37 /*CAP_AUDIT_READ*/] = "AUDIT_READ",
178
+ [38 /*CAP_PERFMON*/] = "PERFMON",
179
+ [39 /*CAP_BPF*/] = "BPF",
180
+ [40 /* CAP_CHECKPOINT_RESTORE */] = "CHECKPOINT_RESTORE",
181
+};
182
+
183
+static const char *syslog_facility_to_name(int facility)
184
+{
185
+ switch (facility) {
186
+ case LOG_FAC(LOG_KERN):
187
+ return "kern";
188
+ case LOG_FAC(LOG_USER):
189
+ return "user";
190
+ case LOG_FAC(LOG_MAIL):
191
+ return "mail";
192
+ case LOG_FAC(LOG_DAEMON):
193
+ return "daemon";
194
+ case LOG_FAC(LOG_AUTH):
195
+ return "auth";
196
+ case LOG_FAC(LOG_SYSLOG):
197
+ return "syslog";
198
+ case LOG_FAC(LOG_LPR):
199
+ return "lpr";
200
+ case LOG_FAC(LOG_NEWS):
201
+ return "news";
202
+ case LOG_FAC(LOG_UUCP):
203
+ return "uucp";
204
+ case LOG_FAC(LOG_CRON):
205
+ return "cron";
206
+ case LOG_FAC(LOG_AUTHPRIV):
207
+ return "authpriv";
208
+ case LOG_FAC(LOG_FTP):
209
+ return "ftp";
210
+ case LOG_FAC(LOG_LOCAL0):
211
+ return "local0";
212
+ case LOG_FAC(LOG_LOCAL1):
213
+ return "local1";
214
+ case LOG_FAC(LOG_LOCAL2):
215
+ return "local2";
216
+ case LOG_FAC(LOG_LOCAL3):
217
+ return "local3";
218
+ case LOG_FAC(LOG_LOCAL4):
219
+ return "local4";
220
+ case LOG_FAC(LOG_LOCAL5):
221
+ return "local5";
222
+ case LOG_FAC(LOG_LOCAL6):
223
+ return "local6";
224
+ case LOG_FAC(LOG_LOCAL7):
225
+ return "local7";
226
+ default:
227
+ return NULL;
228
+ }
229
+}
230
+
231
+static const char *syslog_priority_to_name(int priority)
232
+{
233
+ switch (priority) {
234
+ case LOG_ALERT:
235
+ return "alert";
236
+ case LOG_CRIT:
237
+ return "critical";
238
+ case LOG_DEBUG:
239
+ return "debug";
240
+ case LOG_EMERG:
241
+ return "panic";
242
+ case LOG_ERR:
243
+ return "error";
244
+ case LOG_INFO:
245
+ return "info";
246
+ case LOG_NOTICE:
247
+ return "notice";
248
+ case LOG_WARNING:
249
+ return "warning";
250
+ default:
251
+ return NULL;
252
+ }
253
+}
254
+
255
+FACET_ROW_SEVERITY
256
+syslog_priority_to_facet_severity(FACETS *facets __maybe_unused, FACET_ROW *row, void *data __maybe_unused)
257
+{
258
+ // same to
259
+ // https://github.com/systemd/systemd/blob/aab9e4b2b86905a15944a1ac81e471b5b7075932/src/basic/terminal-util.c#L1501
260
+ // function get_log_colors()
261
+
262
+ FACET_ROW_KEY_VALUE *priority_rkv = dictionary_get(row->dict, "PRIORITY");
263
+ if (!priority_rkv || priority_rkv->empty)
264
+ return FACET_ROW_SEVERITY_NORMAL;
265
+
266
+ int priority = str2i(buffer_tostring(priority_rkv->wb));
267
+
268
+ if (priority <= LOG_ERR)
269
+ return FACET_ROW_SEVERITY_CRITICAL;
270
+
271
+ else if (priority <= LOG_WARNING)
272
+ return FACET_ROW_SEVERITY_WARNING;
273
+
274
+ else if (priority <= LOG_NOTICE)
275
+ return FACET_ROW_SEVERITY_NOTICE;
276
+
277
+ else if (priority >= LOG_DEBUG)
278
+ return FACET_ROW_SEVERITY_DEBUG;
279
+
280
+ return FACET_ROW_SEVERITY_NORMAL;
281
+}
282
+
283
+void nd_sd_journal_transform_syslog_facility(
284
+ FACETS *facets __maybe_unused,
285
+ BUFFER *wb,
286
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
287
+ void *data __maybe_unused)
288
+{
289
+ const char *v = buffer_tostring(wb);
290
+ if (*v && isdigit(*v)) {
291
+ int facility = str2i(buffer_tostring(wb));
292
+ const char *name = syslog_facility_to_name(facility);
293
+ if (name) {
294
+ buffer_flush(wb);
295
+ buffer_strcat(wb, name);
296
+ }
297
+ }
298
+}
299
+
300
+void nd_sd_journal_transform_priority(
301
+ FACETS *facets __maybe_unused,
302
+ BUFFER *wb,
303
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
304
+ void *data __maybe_unused)
305
+{
306
+ if (scope == FACETS_TRANSFORM_FACET_SORT)
307
+ return;
308
+
309
+ const char *v = buffer_tostring(wb);
310
+ if (*v && isdigit(*v)) {
311
+ int priority = str2i(buffer_tostring(wb));
312
+ const char *name = syslog_priority_to_name(priority);
313
+ if (name) {
314
+ buffer_flush(wb);
315
+ buffer_strcat(wb, name);
316
+ }
317
+ }
318
+}
319
+
320
+void nd_sd_journal_transform_errno(
321
+ FACETS *facets __maybe_unused,
322
+ BUFFER *wb,
323
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
324
+ void *data __maybe_unused)
325
+{
326
+ if (scope == FACETS_TRANSFORM_FACET_SORT)
327
+ return;
328
+
329
+ const char *v = buffer_tostring(wb);
330
+ if (*v && isdigit(*v)) {
331
+ unsigned err_no = str2u(buffer_tostring(wb));
332
+ if (err_no > 0 && err_no < sizeof(errno_map) / sizeof(*errno_map)) {
333
+ const char *name = errno_map[err_no];
334
+ if (name) {
335
+ buffer_flush(wb);
336
+ buffer_strcat(wb, name);
337
+ }
338
+ }
339
+ }
340
+}
341
+
342
+DICTIONARY *boot_ids_to_first_ut = NULL;
343
+
344
+void nd_sd_journal_transform_boot_id(
345
+ FACETS *facets __maybe_unused,
346
+ BUFFER *wb,
347
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
348
+ void *data __maybe_unused)
349
+{
350
+ const char *boot_id = buffer_tostring(wb);
351
+ if (*boot_id && isxdigit(*boot_id)) {
352
+ usec_t ut = UINT64_MAX;
353
+ usec_t *p_ut = dictionary_get(boot_ids_to_first_ut, boot_id);
354
+ if (!p_ut) {
355
+#ifndef HAVE_SD_JOURNAL_RESTART_FIELDS
356
+ struct nd_journal_file *njf;
357
+ dfe_start_read(nd_journal_files_registry, njf)
358
+ {
359
+ const char *files[2] = {
360
+ [0] = njf_dfe.name,
361
+ [1] = NULL,
362
+ };
363
+
364
+ sd_journal *j = NULL;
365
+ int r = sd_journal_open_files(&j, files, ND_SD_JOURNAL_OPEN_FLAGS);
366
+ if (r < 0 || !j) {
367
+ internal_error(
368
+ true,
369
+ "JOURNAL: while looking for the first timestamp of boot_id '%s', "
370
+ "sd_journal_open_files('%s') returned %d",
371
+ boot_id,
372
+ jf_dfe.name,
373
+ r);
374
+ continue;
375
+ }
376
+
377
+ ut = nd_journal_file_update_annotation_boot_id(j, njf, boot_id);
378
+ sd_journal_close(j);
379
+ }
380
+ dfe_done(njf);
381
+#endif
382
+ } else
383
+ ut = *p_ut;
384
+
385
+ if (ut && ut != UINT64_MAX) {
386
+ char buffer[RFC3339_MAX_LENGTH];
387
+ rfc3339_datetime_ut(buffer, sizeof(buffer), ut, 0, true);
388
+
389
+ switch (scope) {
390
+ default:
391
+ case FACETS_TRANSFORM_DATA:
392
+ case FACETS_TRANSFORM_VALUE:
393
+ buffer_sprintf(wb, " (%s) ", buffer);
394
+ break;
395
+
396
+ case FACETS_TRANSFORM_FACET:
397
+ case FACETS_TRANSFORM_FACET_SORT:
398
+ case FACETS_TRANSFORM_HISTOGRAM:
399
+ buffer_flush(wb);
400
+ buffer_sprintf(wb, "%s", buffer);
401
+ break;
402
+ }
403
+ }
404
+ }
405
+}
406
+
407
+void nd_sd_journal_transform_uid(
408
+ FACETS *facets __maybe_unused,
409
+ BUFFER *wb,
410
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
411
+ void *data __maybe_unused)
412
+{
413
+ if (scope == FACETS_TRANSFORM_FACET_SORT)
414
+ return;
415
+
416
+ const char *v = buffer_tostring(wb);
417
+ if (*v && isdigit(*v)) {
418
+ uid_t uid = str2i(buffer_tostring(wb));
419
+ CACHED_USERNAME cu = cached_username_get_by_uid(uid);
420
+ buffer_contents_replace(wb, string2str(cu.username), string_strlen(cu.username));
421
+ cached_username_release(cu);
422
+ }
423
+}
424
+
425
+void nd_sd_journal_transform_gid(
426
+ FACETS *facets __maybe_unused,
427
+ BUFFER *wb,
428
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
429
+ void *data __maybe_unused)
430
+{
431
+ if (scope == FACETS_TRANSFORM_FACET_SORT)
432
+ return;
433
+
434
+ const char *v = buffer_tostring(wb);
435
+ if (*v && isdigit(*v)) {
436
+ gid_t gid = str2i(buffer_tostring(wb));
437
+ CACHED_GROUPNAME cg = cached_groupname_get_by_gid(gid);
438
+ buffer_contents_replace(wb, string2str(cg.groupname), string_strlen(cg.groupname));
439
+ cached_groupname_release(cg);
440
+ }
441
+}
442
+
443
+void nd_sd_journal_transform_cap_effective(
444
+ FACETS *facets __maybe_unused,
445
+ BUFFER *wb,
446
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
447
+ void *data __maybe_unused)
448
+{
449
+ if (scope == FACETS_TRANSFORM_FACET_SORT)
450
+ return;
451
+
452
+ const char *v = buffer_tostring(wb);
453
+ if (*v && isdigit(*v)) {
454
+ uint64_t cap = strtoul(buffer_tostring(wb), NULL, 16);
455
+ if (cap) {
456
+ buffer_fast_strcat(wb, " (", 2);
457
+ for (size_t i = 0, added = 0; i < sizeof(linux_capabilities) / sizeof(linux_capabilities[0]); i++) {
458
+ if (linux_capabilities[i] && (cap & (1ULL << i))) {
459
+ if (added)
460
+ buffer_fast_strcat(wb, " | ", 3);
461
+
462
+ buffer_strcat(wb, linux_capabilities[i]);
463
+ added++;
464
+ }
465
+ }
466
+ buffer_fast_strcat(wb, ")", 1);
467
+ }
468
+ }
469
+}
470
+
471
+void nd_sd_journal_transform_timestamp_usec(
472
+ FACETS *facets __maybe_unused,
473
+ BUFFER *wb,
474
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
475
+ void *data __maybe_unused)
476
+{
477
+ if (scope == FACETS_TRANSFORM_FACET_SORT)
478
+ return;
479
+
480
+ const char *v = buffer_tostring(wb);
481
+ if (*v && isdigit(*v)) {
482
+ uint64_t ut = str2ull(buffer_tostring(wb), NULL);
483
+ if (ut) {
484
+ char buffer[RFC3339_MAX_LENGTH];
485
+ rfc3339_datetime_ut(buffer, sizeof(buffer), ut, 6, true);
486
+ buffer_sprintf(wb, " (%s)", buffer);
487
+ }
488
+ }
489
+}
490
+
491
+void nd_sd_journal_dynamic_row_id(
492
+ FACETS *facets __maybe_unused,
493
+ BUFFER *json_array,
494
+ FACET_ROW_KEY_VALUE *rkv,
495
+ FACET_ROW *row,
496
+ void *data __maybe_unused)
497
+{
498
+ FACET_ROW_KEY_VALUE *pid_rkv = dictionary_get(row->dict, "_PID");
499
+ const char *pid = pid_rkv ? buffer_tostring(pid_rkv->wb) : FACET_VALUE_UNSET;
500
+
501
+ const char *identifier = NULL;
502
+ FACET_ROW_KEY_VALUE *container_name_rkv = dictionary_get(row->dict, "CONTAINER_NAME");
503
+ if (container_name_rkv && !container_name_rkv->empty)
504
+ identifier = buffer_tostring(container_name_rkv->wb);
505
+
506
+ if (!identifier) {
507
+ FACET_ROW_KEY_VALUE *syslog_identifier_rkv = dictionary_get(row->dict, "SYSLOG_IDENTIFIER");
508
+ if (syslog_identifier_rkv && !syslog_identifier_rkv->empty)
509
+ identifier = buffer_tostring(syslog_identifier_rkv->wb);
510
+
511
+ if (!identifier) {
512
+ FACET_ROW_KEY_VALUE *comm_rkv = dictionary_get(row->dict, "_COMM");
513
+ if (comm_rkv && !comm_rkv->empty)
514
+ identifier = buffer_tostring(comm_rkv->wb);
515
+ }
516
+ }
517
+
518
+ buffer_flush(rkv->wb);
519
+
520
+ if (!identifier || !*identifier)
521
+ buffer_strcat(rkv->wb, FACET_VALUE_UNSET);
522
+ else if (!pid || !*pid)
523
+ buffer_sprintf(rkv->wb, "%s", identifier);
524
+ else
525
+ buffer_sprintf(rkv->wb, "%s[%s]", identifier, pid);
526
+
527
+ buffer_json_add_array_item_string(json_array, buffer_tostring(rkv->wb));
528
+}
529
+
530
+struct message_id_info {
531
+ const char *msg;
532
+};
533
+
534
+static DICTIONARY *known_journal_messages_ids = NULL;
535
+
536
+#define msgid_into_dict(uuid, message) \
537
+ do { \
538
+ i.msg = message; \
539
+ dictionary_set(known_journal_messages_ids, uuid, &i, sizeof(i)); \
540
+ } while (0)
541
+
542
+static void nd_sd_journal_message_ids_init(void)
543
+{
544
+ known_journal_messages_ids = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
545
+ struct message_id_info i = {0};
546
+
547
+ // systemd
548
+ // https://github.com/systemd/systemd/blob/main/catalog/systemd.catalog.in
549
+ msgid_into_dict("f77379a8490b408bbe5f6940505a777b", "Journal started");
550
+ msgid_into_dict("d93fb3c9c24d451a97cea615ce59c00b", "Journal stopped");
551
+ msgid_into_dict("a596d6fe7bfa4994828e72309e95d61e", "Journal messages suppressed");
552
+ msgid_into_dict("e9bf28e6e834481bb6f48f548ad13606", "Journal messages missed");
553
+ msgid_into_dict("ec387f577b844b8fa948f33cad9a75e6", "Journal disk space usage");
554
+ msgid_into_dict("fc2e22bc6ee647b6b90729ab34a250b1", "Coredump");
555
+ msgid_into_dict("5aadd8e954dc4b1a8c954d63fd9e1137", "Coredump truncated");
556
+ msgid_into_dict("1f4e0a44a88649939aaea34fc6da8c95", "Backtrace"); // not found in systemd catalog
557
+ msgid_into_dict("8d45620c1a4348dbb17410da57c60c66", "User Session created");
558
+ msgid_into_dict("3354939424b4456d9802ca8333ed424a", "User Session terminated");
559
+ msgid_into_dict("fcbefc5da23d428093f97c82a9290f7b", "Seat started");
560
+ msgid_into_dict("e7852bfe46784ed0accde04bc864c2d5", "Seat removed");
561
+ msgid_into_dict("24d8d4452573402496068381a6312df2", "VM or container started");
562
+ msgid_into_dict("58432bd3bace477cb514b56381b8a758", "VM or container stopped");
563
+ msgid_into_dict("c7a787079b354eaaa9e77b371893cd27", "Time change");
564
+ msgid_into_dict("45f82f4aef7a4bbf942ce861d1f20990", "Timezone change");
565
+ msgid_into_dict("50876a9db00f4c40bde1a2ad381c3a1b", "System configuration issues");
566
+ msgid_into_dict("b07a249cd024414a82dd00cd181378ff", "System start-up completed");
567
+ msgid_into_dict("eed00a68ffd84e31882105fd973abdd1", "User start-up completed");
568
+ msgid_into_dict("6bbd95ee977941e497c48be27c254128", "Sleep start");
569
+ msgid_into_dict("8811e6df2a8e40f58a94cea26f8ebf14", "Sleep stop");
570
+ msgid_into_dict("98268866d1d54a499c4e98921d93bc40", "System shutdown initiated");
571
+ msgid_into_dict("c14aaf76ec284a5fa1f105f88dfb061c", "System factory reset initiated");
572
+ msgid_into_dict("d9ec5e95e4b646aaaea2fd05214edbda", "Container init crashed");
573
+ msgid_into_dict("3ed0163e868a4417ab8b9e210407a96c", "System reboot failed after crash");
574
+ msgid_into_dict("645c735537634ae0a32b15a7c6cba7d4", "Init execution froze");
575
+ msgid_into_dict("5addb3a06a734d3396b794bf98fb2d01", "Init crashed no coredump");
576
+ msgid_into_dict("5c9e98de4ab94c6a9d04d0ad793bd903", "Init crashed no fork");
577
+ msgid_into_dict("5e6f1f5e4db64a0eaee3368249d20b94", "Init crashed unknown signal");
578
+ msgid_into_dict("83f84b35ee264f74a3896a9717af34cb", "Init crashed systemd signal");
579
+ msgid_into_dict("3a73a98baf5b4b199929e3226c0be783", "Init crashed process signal");
580
+ msgid_into_dict("2ed18d4f78ca47f0a9bc25271c26adb4", "Init crashed waitpid failed");
581
+ msgid_into_dict("56b1cd96f24246c5b607666fda952356", "Init crashed coredump failed");
582
+ msgid_into_dict("4ac7566d4d7548f4981f629a28f0f829", "Init crashed coredump");
583
+ msgid_into_dict("38e8b1e039ad469291b18b44c553a5b7", "Crash shell failed to fork");
584
+ msgid_into_dict("872729b47dbe473eb768ccecd477beda", "Crash shell failed to execute");
585
+ msgid_into_dict("658a67adc1c940b3b3316e7e8628834a", "Selinux failed");
586
+ msgid_into_dict("e6f456bd92004d9580160b2207555186", "Battery low warning");
587
+ msgid_into_dict("267437d33fdd41099ad76221cc24a335", "Battery low powering off");
588
+ msgid_into_dict("79e05b67bc4545d1922fe47107ee60c5", "Manager mainloop failed");
589
+ msgid_into_dict("dbb136b10ef4457ba47a795d62f108c9", "Manager no xdgdir path");
590
+ msgid_into_dict("ed158c2df8884fa584eead2d902c1032", "Init failed to drop capability bounding set of usermode");
591
+ msgid_into_dict("42695b500df048298bee37159caa9f2e", "Init failed to drop capability bounding set");
592
+ msgid_into_dict("bfc2430724ab44499735b4f94cca9295", "User manager can't disable new privileges");
593
+ msgid_into_dict("59288af523be43a28d494e41e26e4510", "Manager failed to start default target");
594
+ msgid_into_dict("689b4fcc97b4486ea5da92db69c9e314", "Manager failed to isolate default target");
595
+ msgid_into_dict("5ed836f1766f4a8a9fc5da45aae23b29", "Manager failed to collect passed file descriptors");
596
+ msgid_into_dict("6a40fbfbd2ba4b8db02fb40c9cd090d7", "Init failed to fix up environment variables");
597
+ msgid_into_dict("0e54470984ac419689743d957a119e2e", "Manager failed to allocate");
598
+ msgid_into_dict("d67fa9f847aa4b048a2ae33535331adb", "Manager failed to write Smack");
599
+ msgid_into_dict("af55a6f75b544431b72649f36ff6d62c", "System shutdown critical error");
600
+ msgid_into_dict("d18e0339efb24a068d9c1060221048c2", "Init failed to fork off valgrind");
601
+ msgid_into_dict("7d4958e842da4a758f6c1cdc7b36dcc5", "Unit starting");
602
+ msgid_into_dict("39f53479d3a045ac8e11786248231fbf", "Unit started");
603
+ msgid_into_dict("be02cf6855d2428ba40df7e9d022f03d", "Unit failed");
604
+ msgid_into_dict("de5b426a63be47a7b6ac3eaac82e2f6f", "Unit stopping");
605
+ msgid_into_dict("9d1aaa27d60140bd96365438aad20286", "Unit stopped");
606
+ msgid_into_dict("d34d037fff1847e6ae669a370e694725", "Unit reloading");
607
+ msgid_into_dict("7b05ebc668384222baa8881179cfda54", "Unit reloaded");
608
+ msgid_into_dict("5eb03494b6584870a536b337290809b3", "Unit restart scheduled");
609
+ msgid_into_dict("ae8f7b866b0347b9af31fe1c80b127c0", "Unit resources");
610
+ msgid_into_dict("7ad2d189f7e94e70a38c781354912448", "Unit success");
611
+ msgid_into_dict("0e4284a0caca4bfc81c0bb6786972673", "Unit skipped");
612
+ msgid_into_dict("d9b373ed55a64feb8242e02dbe79a49c", "Unit failure result");
613
+ msgid_into_dict("641257651c1b4ec9a8624d7a40a9e1e7", "Process execution failed");
614
+ msgid_into_dict("98e322203f7a4ed290d09fe03c09fe15", "Unit process exited");
615
+ msgid_into_dict("0027229ca0644181a76c4e92458afa2e", "Syslog forward missed");
616
+ msgid_into_dict("1dee0369c7fc4736b7099b38ecb46ee7", "Mount point is not empty");
617
+ msgid_into_dict("d989611b15e44c9dbf31e3c81256e4ed", "Unit oomd kill"); // not found in systemd catalog
618
+ msgid_into_dict("fe6faa94e7774663a0da52717891d8ef", "Unit out of memory");
619
+ msgid_into_dict("b72ea4a2881545a0b50e200e55b9b06f", "Lid opened");
620
+ msgid_into_dict("b72ea4a2881545a0b50e200e55b9b070", "Lid closed");
621
+ msgid_into_dict("f5f416b862074b28927a48c3ba7d51ff", "System docked");
622
+ msgid_into_dict("51e171bd585248568110144c517cca53", "System undocked");
623
+ msgid_into_dict("b72ea4a2881545a0b50e200e55b9b071", "Power key");
624
+ msgid_into_dict("3e0117101eb243c1b9a50db3494ab10b", "Power key long press");
625
+ msgid_into_dict("9fa9d2c012134ec385451ffe316f97d0", "Reboot key");
626
+ msgid_into_dict("f1c59a58c9d943668965c337caec5975", "Reboot key long press");
627
+ msgid_into_dict("b72ea4a2881545a0b50e200e55b9b072", "Suspend key");
628
+ msgid_into_dict("bfdaf6d312ab4007bc1fe40a15df78e8", "Suspend key long press");
629
+ msgid_into_dict("b72ea4a2881545a0b50e200e55b9b073", "Hibernate key");
630
+ msgid_into_dict("167836df6f7f428e98147227b2dc8945", "Hibernate key long press");
631
+ msgid_into_dict("c772d24e9a884cbeb9ea12625c306c01", "Invalid configuration"); // not found in systemd catalog
632
+ msgid_into_dict("1675d7f172174098b1108bf8c7dc8f5d", "DNSSEC validation failed");
633
+ msgid_into_dict("4d4408cfd0d144859184d1e65d7c8a65", "DNSSEC trust anchor revoked");
634
+ msgid_into_dict("36db2dfa5a9045e1bd4af5f93e1cf057", "DNSSEC turned off");
635
+ msgid_into_dict("b61fdac612e94b9182285b998843061f", "Username unsafe");
636
+ msgid_into_dict("1b3bb94037f04bbf81028e135a12d293", "Mount point path not suitable");
637
+ msgid_into_dict("010190138f494e29a0ef6669749531aa", "Device path not suitable"); // not found in systemd catalog
638
+ msgid_into_dict("b480325f9c394a7b802c231e51a2752c", "Nobody user unsuitable");
639
+ msgid_into_dict("1c0454c1bd2241e0ac6fefb4bc631433", "Systemd udev settle deprecated");
640
+ msgid_into_dict("7c8a41f37b764941a0e1780b1be2f037", "Time initial sync");
641
+ msgid_into_dict("7db73c8af0d94eeb822ae04323fe6ab6", "Time initial bump");
642
+ msgid_into_dict("9e7066279dc8403da79ce4b1a69064b2", "Shutdown scheduled");
643
+ msgid_into_dict("249f6fb9e6e2428c96f3f0875681ffa3", "Shutdown canceled");
644
+ msgid_into_dict("3f7d5ef3e54f4302b4f0b143bb270cab", "TPM PCR Extended");
645
+ msgid_into_dict("f9b0be465ad540d0850ad32172d57c21", "Memory Trimmed");
646
+ msgid_into_dict("a8fa8dacdb1d443e9503b8be367a6adb", "SysV Service Found");
647
+ msgid_into_dict("187c62eb1e7f463bb530394f52cb090f", "Portable Service attached");
648
+ msgid_into_dict("76c5c754d628490d8ecba4c9d042112b", "Portable Service detached");
649
+ msgid_into_dict("9cf56b8baf9546cf9478783a8de42113", "systemd-networkd sysctl changed by foreign process");
650
+ msgid_into_dict("ad7089f928ac4f7ea00c07457d47ba8a", "SRK into TPM authorization failure");
651
+ msgid_into_dict("b2bcbaf5edf948e093ce50bbea0e81ec", "Secure Attention Key (SAK) was pressed");
652
+
653
+ // dbus
654
+ // https://github.com/bus1/dbus-broker/blob/main/src/catalog/catalog-ids.h
655
+ msgid_into_dict("7fc63312330b479bb32e598d47cef1a8", "dbus activate no unit");
656
+ msgid_into_dict("ee9799dab1e24d81b7bee7759a543e1b", "dbus activate masked unit");
657
+ msgid_into_dict("a0fa58cafd6f4f0c8d003d16ccf9e797", "dbus broker exited");
658
+ msgid_into_dict("c8c6cde1c488439aba371a664353d9d8", "dbus dirwatch");
659
+ msgid_into_dict("8af3357071af4153af414daae07d38e7", "dbus dispatch stats");
660
+ msgid_into_dict("199d4300277f495f84ba4028c984214c", "dbus no sopeergroup");
661
+ msgid_into_dict("b209c0d9d1764ab38d13b8e00d1784d6", "dbus protocol violation");
662
+ msgid_into_dict("6fa70fa776044fa28be7a21daf42a108", "dbus receive failed");
663
+ msgid_into_dict("0ce0fa61d1a9433dabd67417f6b8e535", "dbus service failed open");
664
+ msgid_into_dict("24dc708d9e6a4226a3efe2033bb744de", "dbus service invalid");
665
+ msgid_into_dict("f15d2347662d483ea9bcd8aa1a691d28", "dbus sighup");
666
+
667
+ // gnome
668
+ // https://gitlab.gnome.org/GNOME/gnome-session/-/blob/main/gnome-session/gsm-manager.c
669
+ msgid_into_dict("0ce153587afa4095832d233c17a88001", "Gnome SM startup succeeded");
670
+ msgid_into_dict("10dd2dc188b54a5e98970f56499d1f73", "Gnome SM unrecoverable failure");
671
+
672
+ // gnome-shell
673
+ // https://gitlab.gnome.org/GNOME/gnome-shell/-/blob/main/js/ui/main.js#L56
674
+ msgid_into_dict("f3ea493c22934e26811cd62abe8e203a", "Gnome shell started");
675
+
676
+ // flathub
677
+ // https://docs.flatpak.org/de/latest/flatpak-command-reference.html
678
+ msgid_into_dict("c7b39b1e006b464599465e105b361485", "Flatpak cache");
679
+
680
+ // ???
681
+ msgid_into_dict("75ba3deb0af041a9a46272ff85d9e73e", "Flathub pulls");
682
+ msgid_into_dict("f02bce89a54e4efab3a94a797d26204a", "Flathub pull errors");
683
+
684
+ // ??
685
+ msgid_into_dict("dd11929c788e48bdbb6276fb5f26b08a", "Boltd starting");
686
+
687
+ // Netdata
688
+ msgid_into_dict("1e6061a9fbd44501b3ccc368119f2b69", "Netdata startup");
689
+ msgid_into_dict("ed4cdb8f1beb4ad3b57cb3cae2d162fa", "Netdata connection from child");
690
+ msgid_into_dict("6e2e3839067648968b646045dbf28d66", "Netdata connection to parent");
691
+ msgid_into_dict("9ce0cb58ab8b44df82c4bf1ad9ee22de", "Netdata alert transition");
692
+ msgid_into_dict("6db0018e83e34320ae2a659d78019fb7", "Netdata alert notification");
693
+ msgid_into_dict("23e93dfccbf64e11aac858b9410d8a82", "Netdata fatal message");
694
+ msgid_into_dict("8ddaf5ba33a74078b609250db1e951f3", "Sensor state transition");
695
+ msgid_into_dict("ec87a56120d5431bace51e2fb8bba243", "Netdata log flood protection");
696
+ msgid_into_dict("acb33cb95778476baac702eb7e4e151d", "Netdata Cloud connection");
697
+ msgid_into_dict("d1f59606dd4d41e3b217a0cfcae8e632", "Netdata extreme cardinality");
698
+ msgid_into_dict("02f47d350af5449197bf7a95b605a468", "Netdata exit reason");
699
+ msgid_into_dict("4fdf40816c124623a032b7fe73beacb8", "Netdata dynamic configuration");
700
+}
701
+
702
+void nd_sd_journal_transform_message_id(
703
+ FACETS *facets __maybe_unused,
704
+ BUFFER *wb,
705
+ FACETS_TRANSFORMATION_SCOPE scope __maybe_unused,
706
+ void *data __maybe_unused)
707
+{
708
+ const char *message_id = buffer_tostring(wb);
709
+ struct message_id_info *i = dictionary_get(known_journal_messages_ids, message_id);
710
+
711
+ if (!i)
712
+ return;
713
+
714
+ switch (scope) {
715
+ default:
716
+ case FACETS_TRANSFORM_DATA:
717
+ case FACETS_TRANSFORM_VALUE:
718
+ buffer_sprintf(wb, " (%s)", i->msg);
719
+ break;
720
+
721
+ case FACETS_TRANSFORM_FACET:
722
+ case FACETS_TRANSFORM_FACET_SORT:
723
+ case FACETS_TRANSFORM_HISTOGRAM:
724
+ buffer_flush(wb);
725
+ buffer_strcat(wb, i->msg);
726
+ break;
727
+ }
728
+}
729
+
730
+void nd_sd_journal_annotations_init(void)
731
+{
732
+ cached_usernames_init();
733
+ cached_groupnames_init();
734
+ update_cached_host_users();
735
+ update_cached_host_groups();
736
+ nd_sd_journal_message_ids_init();
737
+}
src/collectors/systemd-journal.plugin/systemd-journal-dyncfg.c
new
+177
@@ -0,0 +1,177 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "systemd-internals.h"
4
+
5
+#define JOURNAL_DIRECTORIES_JSON_NODE "journalDirectories"
6
+
7
+static bool is_directory(const char *dir)
8
+{
9
+ struct stat statbuf;
10
+ if (stat(dir, &statbuf) != 0) {
11
+ // Error in stat() means the path probably doesn't exist or can't be accessed.
12
+ return false;
13
+ }
14
+ // S_ISDIR macro is true if the path is a directory.
15
+ return S_ISDIR(statbuf.st_mode) ? true : false;
16
+}
17
+
18
+static const char *is_valid_dir(const char *dir)
19
+{
20
+ if (strcmp(dir, "/") == 0)
21
+ return "/ is not acceptable";
22
+
23
+ if (!strstartswith(dir, "/"))
24
+ return "only directories starting with / are accepted";
25
+
26
+ if (strstr(dir, "/./"))
27
+ return "directory contains /./";
28
+
29
+ if (strstr(dir, "/../") || strendswith(dir, "/.."))
30
+ return "directory contains /../";
31
+
32
+ if (strstartswith(dir, "/dev/") || strcmp(dir, "/dev") == 0)
33
+ return "directory contains /dev";
34
+
35
+ if (strstartswith(dir, "/proc/") || strcmp(dir, "/proc") == 0)
36
+ return "directory contains /proc";
37
+
38
+ if (strstartswith(dir, "/sys/") || strcmp(dir, "/sys") == 0)
39
+ return "directory contains /sys";
40
+
41
+ if (strstartswith(dir, "/etc/") || strcmp(dir, "/etc") == 0)
42
+ return "directory contains /etc";
43
+
44
+ if (strstartswith(dir, "/lib/") || strcmp(dir, "/lib") == 0 || strstartswith(dir, "/lib32/") ||
45
+ strcmp(dir, "/lib32") == 0 || strstartswith(dir, "/lib64/") || strcmp(dir, "/lib64") == 0)
46
+ return "directory contains /lib";
47
+
48
+ return NULL;
49
+}
50
+
51
+static int systemd_journal_directories_dyncfg_update(BUFFER *result, BUFFER *payload)
52
+{
53
+ if (!payload || !buffer_strlen(payload))
54
+ return dyncfg_default_response(result, HTTP_RESP_BAD_REQUEST, "empty payload received");
55
+
56
+ CLEAN_JSON_OBJECT *jobj = json_tokener_parse(buffer_tostring(payload));
57
+ if (!jobj)
58
+ return dyncfg_default_response(result, HTTP_RESP_BAD_REQUEST, "cannot parse json payload");
59
+
60
+ struct json_object *journalDirectories;
61
+ json_object_object_get_ex(jobj, JOURNAL_DIRECTORIES_JSON_NODE, &journalDirectories);
62
+
63
+ if (json_object_get_type(journalDirectories) != json_type_array)
64
+ return dyncfg_default_response(
65
+ result, HTTP_RESP_BAD_REQUEST, "member " JOURNAL_DIRECTORIES_JSON_NODE " is not an array");
66
+
67
+ size_t n_directories = json_object_array_length(journalDirectories);
68
+ if (n_directories > MAX_JOURNAL_DIRECTORIES)
69
+ return dyncfg_default_response(result, HTTP_RESP_BAD_REQUEST, "too many directories configured");
70
+
71
+ // validate the directories
72
+ for (size_t i = 0; i < n_directories; i++) {
73
+ struct json_object *dir = json_object_array_get_idx(journalDirectories, i);
74
+ const char *s = json_object_get_string(dir);
75
+ if (s && *s) {
76
+ const char *msg = is_valid_dir(s);
77
+ if (msg)
78
+ return dyncfg_default_response(result, HTTP_RESP_BAD_REQUEST, msg);
79
+ }
80
+ }
81
+
82
+ size_t added = 0, not_found = 0;
83
+ for (size_t i = 0; i < n_directories; i++) {
84
+ struct json_object *dir = json_object_array_get_idx(journalDirectories, i);
85
+ const char *s = json_object_get_string(dir);
86
+ if (s && *s) {
87
+ string_freez(journal_directories[added].path);
88
+ journal_directories[added++].path = string_strdupz(s);
89
+
90
+ if (!is_directory(s))
91
+ not_found++;
92
+ }
93
+ }
94
+
95
+ if (!added)
96
+ return dyncfg_default_response(result, HTTP_RESP_BAD_REQUEST, "no directories in the payload");
97
+ else {
98
+ for (size_t i = added; i < MAX_JOURNAL_DIRECTORIES; i++) {
99
+ string_freez(journal_directories[i].path);
100
+ journal_directories[i].path = NULL;
101
+ }
102
+ }
103
+
104
+ nd_journal_watcher_restart();
105
+
106
+ return dyncfg_default_response(
107
+ result, HTTP_RESP_OK, not_found ? "added, but some directories are not found in the filesystem" : "");
108
+}
109
+
110
+static int systemd_journal_directories_dyncfg_get(BUFFER *wb)
111
+{
112
+ buffer_flush(wb);
113
+ buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
114
+
115
+ buffer_json_member_add_array(wb, JOURNAL_DIRECTORIES_JSON_NODE);
116
+ for (size_t i = 0; i < MAX_JOURNAL_DIRECTORIES; i++) {
117
+ if (!journal_directories[i].path)
118
+ break;
119
+
120
+ buffer_json_add_array_item_string(wb, string2str(journal_directories[i].path));
121
+ }
122
+ buffer_json_array_close(wb);
123
+
124
+ buffer_json_finalize(wb);
125
+ return HTTP_RESP_OK;
126
+}
127
+
128
+static int systemd_journal_directories_dyncfg_cb(
129
+ const char *transaction,
130
+ const char *id,
131
+ DYNCFG_CMDS cmd,
132
+ const char *add_name __maybe_unused,
133
+ BUFFER *payload,
134
+ usec_t *stop_monotonic_ut __maybe_unused,
135
+ bool *cancelled __maybe_unused,
136
+ BUFFER *result,
137
+ HTTP_ACCESS access __maybe_unused,
138
+ const char *source __maybe_unused,
139
+ void *data __maybe_unused)
140
+{
141
+ CLEAN_BUFFER *action = buffer_create(100, NULL);
142
+ dyncfg_cmds2buffer(cmd, action);
143
+
144
+ if (cmd == DYNCFG_CMD_GET)
145
+ return systemd_journal_directories_dyncfg_get(result);
146
+
147
+ if (cmd == DYNCFG_CMD_UPDATE)
148
+ return systemd_journal_directories_dyncfg_update(result, payload);
149
+
150
+ nd_log(
151
+ NDLS_COLLECTORS,
152
+ NDLP_ERR,
153
+ "DYNCFG: unhandled transaction '%s', id '%s' cmd '%s', payload: %s",
154
+ transaction,
155
+ id,
156
+ buffer_tostring(action),
157
+ payload ? buffer_tostring(payload) : "");
158
+
159
+ return dyncfg_default_response(result, HTTP_RESP_BAD_REQUEST, "the command is not handled by this plugin");
160
+}
161
+
162
+void nd_systemd_journal_dyncfg_init(struct functions_evloop_globals *wg)
163
+{
164
+ functions_evloop_dyncfg_add(
165
+ wg,
166
+ "systemd-journal:monitored-directories",
167
+ "/logs/systemd-journal",
168
+ DYNCFG_STATUS_RUNNING,
169
+ DYNCFG_TYPE_SINGLE,
170
+ DYNCFG_SOURCE_TYPE_INTERNAL,
171
+ "internal",
172
+ DYNCFG_CMD_SCHEMA | DYNCFG_CMD_GET | DYNCFG_CMD_UPDATE,
173
+ HTTP_ACCESS_NONE,
174
+ HTTP_ACCESS_NONE,
175
+ systemd_journal_directories_dyncfg_cb,
176
+ NULL);
177
+}
src/collectors/systemd-journal.plugin/systemd-journal-files.c
new
+872
@@ -0,0 +1,872 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "collectors/systemd-journal.plugin/provider/netdata_provider.h"
4
+#include "systemd-internals.h"
5
+
6
+#define ND_SD_JOURNAL_MAX_SOURCE_LEN 64
7
+#define VAR_LOG_JOURNAL_MAX_DEPTH 10
8
+
9
+struct journal_directory journal_directories[MAX_JOURNAL_DIRECTORIES] = {0};
10
+DICTIONARY *nd_journal_files_registry = NULL;
11
+DICTIONARY *used_hashes_registry = NULL;
12
+
13
+static usec_t systemd_journal_session = 0;
14
+
15
+void buffer_json_journal_versions(BUFFER *wb)
16
+{
17
+ buffer_json_member_add_object(wb, "versions");
18
+ {
19
+ buffer_json_member_add_uint64(
20
+ wb, "sources", systemd_journal_session + dictionary_version(nd_journal_files_registry));
21
+ }
22
+ buffer_json_object_close(wb);
23
+}
24
+
25
+static bool journal_sd_id128_parse(const char *in, NsdId128 *ret)
26
+{
27
+ while (isspace(*in))
28
+ in++;
29
+
30
+ char uuid[33];
31
+ strncpyz(uuid, in, 32);
32
+ uuid[32] = '\0';
33
+
34
+ if (strlen(uuid) == 32) {
35
+ NsdId128 read;
36
+ if (nsd_id128_from_string(uuid, &read) == 0) {
37
+ *ret = read;
38
+ return true;
39
+ }
40
+ }
41
+
42
+ return false;
43
+}
44
+
45
+usec_t
46
+nd_journal_file_update_annotation_boot_id(NsdJournal *j, struct nd_journal_file *njf __maybe_unused, const char *boot_id)
47
+{
48
+ usec_t ut = UINT64_MAX;
49
+ int r;
50
+
51
+ char m[100];
52
+ size_t len = snprintfz(m, sizeof(m), "_BOOT_ID=%s", boot_id);
53
+
54
+ nsd_journal_flush_matches(j);
55
+
56
+ r = nsd_journal_add_match(j, m, len);
57
+ if (r < 0) {
58
+ errno = -r;
59
+ internal_error(
60
+ true,
61
+ "JOURNAL: while looking for the first timestamp of boot_id '%s', "
62
+ "sd_journal_add_match('%s') on file '%s' returned %d",
63
+ boot_id,
64
+ m,
65
+ njf->filename,
66
+ r);
67
+ return UINT64_MAX;
68
+ }
69
+
70
+ r = nsd_journal_seek_head(j);
71
+ if (r < 0) {
72
+ errno = -r;
73
+ internal_error(
74
+ true,
75
+ "JOURNAL: while looking for the first timestamp of boot_id '%s', "
76
+ "sd_journal_seek_head() on file '%s' returned %d",
77
+ boot_id,
78
+ njf->filename,
79
+ r);
80
+ return UINT64_MAX;
81
+ }
82
+
83
+ r = nsd_journal_next(j);
84
+ if (r < 0) {
85
+ errno = -r;
86
+ internal_error(
87
+ true,
88
+ "JOURNAL: while looking for the first timestamp of boot_id '%s', "
89
+ "sd_journal_next() on file '%s' returned %d",
90
+ boot_id,
91
+ njf->filename,
92
+ r);
93
+ return UINT64_MAX;
94
+ } else if (r == 0) {
95
+ internal_error(
96
+ true,
97
+ "GVD: while looking for the first timestamp of boot_id '%s', "
98
+ "sd_journal_next() on file '%s' returned %d",
99
+ boot_id,
100
+ njf->filename,
101
+ r);
102
+ return UINT64_MAX;
103
+ }
104
+
105
+ r = nsd_journal_get_realtime_usec(j, &ut);
106
+ if (r < 0 || !ut || ut == UINT64_MAX) {
107
+ errno = -r;
108
+ internal_error(
109
+ r != -EADDRNOTAVAIL,
110
+ "JOURNAL: while looking for the first timestamp of boot_id '%s', "
111
+ "sd_journal_get_realtime_usec() on file '%s' returned %d",
112
+ boot_id,
113
+ njf->filename,
114
+ r);
115
+ return UINT64_MAX;
116
+ }
117
+
118
+ if (ut && ut != UINT64_MAX) {
119
+ dictionary_set(boot_ids_to_first_ut, boot_id, &ut, sizeof(ut));
120
+ return ut;
121
+ }
122
+
123
+ return UINT64_MAX;
124
+}
125
+
126
+static void
127
+nd_journal_file_get_boot_id_annotations(NsdJournal *j __maybe_unused, struct nd_journal_file *njf __maybe_unused)
128
+{
129
+#ifdef HAVE_SD_JOURNAL_RESTART_FIELDS
130
+ nsd_journal_flush_matches(j);
131
+
132
+ int r = nsd_journal_query_unique(j, "_BOOT_ID");
133
+ if (r < 0) {
134
+ errno = -r;
135
+ internal_error(
136
+ true,
137
+ "JOURNAL: while querying for the unique _BOOT_ID values, "
138
+ "sd_journal_query_unique() on file '%s' returned %d",
139
+ njf->filename,
140
+ r);
141
+ errno = -r;
142
+ return;
143
+ }
144
+
145
+ const void *data = NULL;
146
+ size_t data_length;
147
+
148
+ DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED);
149
+
150
+ NSD_JOURNAL_FOREACH_UNIQUE(j, data, data_length)
151
+ {
152
+ const char *key, *value;
153
+ size_t key_length, value_length;
154
+
155
+ if (!parse_journal_field(data, data_length, &key, &key_length, &value, &value_length))
156
+ continue;
157
+
158
+ if (value_length != 32)
159
+ continue;
160
+
161
+ char buf[33];
162
+ memcpy(buf, value, 32);
163
+ buf[32] = '\0';
164
+
165
+ dictionary_set(dict, buf, NULL, 0);
166
+ }
167
+
168
+ void *nothing;
169
+ dfe_start_read(dict, nothing)
170
+ {
171
+ nd_journal_file_update_annotation_boot_id(j, njf, nothing_dfe.name);
172
+ }
173
+ dfe_done(nothing);
174
+
175
+ dictionary_destroy(dict);
176
+#endif
177
+}
178
+
179
+void nd_journal_file_update_header(const char *filename, struct nd_journal_file *njf)
180
+{
181
+ if (njf->last_scan_header_vs_last_modified_ut == njf->file_last_modified_ut)
182
+ return;
183
+
184
+ fstat_cache_enable_on_thread();
185
+
186
+ const char *files[2] = {
187
+ [0] = filename,
188
+ [1] = NULL,
189
+ };
190
+
191
+ NsdJournal *j = NULL;
192
+ if (nsd_journal_open_files(&j, files, ND_SD_JOURNAL_OPEN_FLAGS) < 0 || !j) {
193
+ netdata_log_error("JOURNAL: cannot open file '%s' to update msg_ut", filename);
194
+ fstat_cache_disable_on_thread();
195
+
196
+ if (!njf->logged_failure) {
197
+ netdata_log_error(
198
+ "cannot open journal file '%s', using file timestamps to understand time-frame.", filename);
199
+ njf->logged_failure = true;
200
+ }
201
+
202
+ njf->msg_first_ut = 0;
203
+ njf->msg_last_ut = njf->file_last_modified_ut;
204
+ njf->last_scan_header_vs_last_modified_ut = njf->file_last_modified_ut;
205
+ return;
206
+ }
207
+
208
+ usec_t first_ut = 0, last_ut = 0;
209
+ uint64_t first_seqnum = 0, last_seqnum = 0;
210
+ NsdId128 first_writer_id = NSD_ID128_NULL, last_writer_id = NSD_ID128_NULL;
211
+
212
+ if (nsd_journal_seek_head(j) < 0 || nsd_journal_next(j) < 0 || nsd_journal_get_realtime_usec(j, &first_ut) < 0 ||
213
+ !first_ut) {
214
+ internal_error(true, "cannot find the timestamp of the first message in '%s'", filename);
215
+ first_ut = 0;
216
+ }
217
+#ifdef HAVE_SD_JOURNAL_GET_SEQNUM
218
+ else {
219
+ if (nsd_journal_get_seqnum(j, &first_seqnum, &first_writer_id) < 0 || !first_seqnum) {
220
+ internal_error(true, "cannot find the first seqnums of the first message in '%s'", filename);
221
+ first_seqnum = 0;
222
+ memset(&first_writer_id, 0, sizeof(first_writer_id));
223
+ }
224
+ }
225
+#endif
226
+
227
+ if (nsd_journal_seek_tail(j) < 0 || nsd_journal_previous(j) < 0 || nsd_journal_get_realtime_usec(j, &last_ut) < 0 ||
228
+ !last_ut) {
229
+ internal_error(true, "cannot find the timestamp of the last message in '%s'", filename);
230
+ last_ut = njf->file_last_modified_ut;
231
+ }
232
+#ifdef HAVE_SD_JOURNAL_GET_SEQNUM
233
+ else {
234
+ if (nsd_journal_get_seqnum(j, &last_seqnum, &last_writer_id) < 0 || !last_seqnum) {
235
+ internal_error(true, "cannot find the last seqnums of the first message in '%s'", filename);
236
+ last_seqnum = 0;
237
+ memset(&last_writer_id, 0, sizeof(last_writer_id));
238
+ }
239
+ }
240
+#endif
241
+
242
+ if (first_ut > last_ut) {
243
+ internal_error(true, "timestamps are flipped in file '%s'", filename);
244
+ usec_t t = first_ut;
245
+ first_ut = last_ut;
246
+ last_ut = t;
247
+ }
248
+
249
+ if (!first_seqnum || !first_ut) {
250
+ // extract these from the filename - if possible
251
+
252
+ const char *at = strchr(filename, '@');
253
+ if (at) {
254
+ const char *dash_seqnum = strchr(at + 1, '-');
255
+ if (dash_seqnum) {
256
+ const char *dash_first_msg_ut = strchr(dash_seqnum + 1, '-');
257
+ if (dash_first_msg_ut) {
258
+ const char *dot_journal = NULL;
259
+ if (is_journal_file(filename, -1, &dot_journal) && dot_journal && dot_journal > dash_first_msg_ut) {
260
+ if (dash_seqnum - at - 1 == 32 && dash_first_msg_ut - dash_seqnum - 1 == 16 &&
261
+ dot_journal - dash_first_msg_ut - 1 == 16) {
262
+ NsdId128 writer;
263
+ if (journal_sd_id128_parse(at + 1, &writer)) {
264
+ char *endptr = NULL;
265
+ uint64_t seqnum = strtoul(dash_seqnum + 1, &endptr, 16);
266
+ if (endptr == dash_first_msg_ut) {
267
+ uint64_t ts = strtoul(dash_first_msg_ut + 1, &endptr, 16);
268
+ if (endptr == dot_journal) {
269
+ first_seqnum = seqnum;
270
+ first_writer_id = writer;
271
+ first_ut = ts;
272
+ }
273
+ }
274
+ }
275
+ }
276
+ }
277
+ }
278
+ }
279
+ }
280
+ }
281
+
282
+ njf->first_seqnum = first_seqnum;
283
+ njf->last_seqnum = last_seqnum;
284
+
285
+ njf->first_writer_id = first_writer_id;
286
+ njf->last_writer_id = last_writer_id;
287
+
288
+ njf->msg_first_ut = first_ut;
289
+ njf->msg_last_ut = last_ut;
290
+
291
+ if (!njf->msg_last_ut)
292
+ njf->msg_last_ut = njf->file_last_modified_ut;
293
+
294
+ if (last_seqnum > first_seqnum) {
295
+ if (!nsd_id128_equal(first_writer_id, last_writer_id)) {
296
+ njf->messages_in_file = 0;
297
+ nd_log(
298
+ NDLS_COLLECTORS,
299
+ NDLP_NOTICE,
300
+ "The writers of the first and the last message in file '%s' differ.",
301
+ filename);
302
+ } else
303
+ njf->messages_in_file = last_seqnum - first_seqnum + 1;
304
+ } else
305
+ njf->messages_in_file = 0;
306
+
307
+ nd_journal_file_get_boot_id_annotations(j, njf);
308
+ nsd_journal_close(j);
309
+ fstat_cache_disable_on_thread();
310
+
311
+ njf->last_scan_header_vs_last_modified_ut = njf->file_last_modified_ut;
312
+}
313
+
314
+static STRING *string_strdupz_source(const char *s, const char *e, size_t max_len, const char *prefix)
315
+{
316
+ char buf[max_len];
317
+ size_t len;
318
+ char *dst = buf;
319
+
320
+ if (prefix) {
321
+ len = strlen(prefix);
322
+ memcpy(buf, prefix, len);
323
+ dst = &buf[len];
324
+ max_len -= len;
325
+ }
326
+
327
+ len = e - s;
328
+ if (len >= max_len)
329
+ len = max_len - 1;
330
+ memcpy(dst, s, len);
331
+ dst[len] = '\0';
332
+ buf[max_len - 1] = '\0';
333
+
334
+ for (size_t i = 0; buf[i]; i++)
335
+ if (!is_netdata_api_valid_character(buf[i]))
336
+ buf[i] = '_';
337
+
338
+ return string_strdupz(buf);
339
+}
340
+
341
+static void files_registry_insert_cb(const DICTIONARY_ITEM *item, void *value, void *data __maybe_unused)
342
+{
343
+ struct nd_journal_file *njf = value;
344
+ njf->filename = dictionary_acquired_item_name(item);
345
+ njf->filename_len = strlen(njf->filename);
346
+ njf->source_type = ND_SD_JF_ALL;
347
+
348
+ // based on the filename
349
+ // decide the source to show to the user
350
+ const char *s = strrchr(njf->filename, '/');
351
+ if (s) {
352
+ if (strstr(njf->filename, "/remote/")) {
353
+ njf->source_type |= ND_SD_JF_REMOTE_ALL;
354
+
355
+ if (strncmp(s, "/remote-", 8) == 0) {
356
+ s = &s[8]; // skip "/remote-"
357
+
358
+ char *e = strchr(s, '@');
359
+ if (!e)
360
+ is_journal_file(s, -1, (const char **)&e);
361
+
362
+ if (e) {
363
+ const char *d = s;
364
+ for (; d < e && (isdigit(*d) || *d == '.' || *d == ':'); d++)
365
+ ;
366
+ if (d == e) {
367
+ // a valid IP address
368
+ char ip[e - s + 1];
369
+ memcpy(ip, s, e - s);
370
+ ip[e - s] = '\0';
371
+ char buf[ND_SD_JOURNAL_MAX_SOURCE_LEN];
372
+ if (ip_to_hostname(ip, buf, sizeof(buf)))
373
+ njf->source =
374
+ string_strdupz_source(buf, &buf[strlen(buf)], ND_SD_JOURNAL_MAX_SOURCE_LEN, "remote-");
375
+ else {
376
+ internal_error(true, "Cannot find the hostname for IP '%s'", ip);
377
+ njf->source = string_strdupz_source(s, e, ND_SD_JOURNAL_MAX_SOURCE_LEN, "remote-");
378
+ }
379
+ } else
380
+ njf->source = string_strdupz_source(s, e, ND_SD_JOURNAL_MAX_SOURCE_LEN, "remote-");
381
+ }
382
+ }
383
+ } else {
384
+ njf->source_type |= ND_SD_JF_LOCAL_ALL;
385
+
386
+ const char *t = s - 1;
387
+ while (t >= njf->filename && *t != '.' && *t != '/')
388
+ t--;
389
+
390
+ if (t >= njf->filename && *t == '.') {
391
+ njf->source_type |= ND_SD_JF_LOCAL_NAMESPACE;
392
+ njf->source = string_strdupz_source(t + 1, s, ND_SD_JOURNAL_MAX_SOURCE_LEN, "namespace-");
393
+ } else if (strncmp(s, "/system", 7) == 0)
394
+ njf->source_type |= ND_SD_JF_LOCAL_SYSTEM;
395
+
396
+ else if (strncmp(s, "/user", 5) == 0)
397
+ njf->source_type |= ND_SD_JF_LOCAL_USER;
398
+
399
+ else
400
+ njf->source_type |= ND_SD_JF_LOCAL_OTHER;
401
+ }
402
+ } else
403
+ njf->source_type |= ND_SD_JF_LOCAL_ALL | ND_SD_JF_LOCAL_OTHER;
404
+
405
+ njf->msg_last_ut = njf->file_last_modified_ut;
406
+
407
+ nd_log(NDLS_COLLECTORS, NDLP_DEBUG, "Journal file added to the journal files registry: '%s'", njf->filename);
408
+}
409
+
410
+static bool files_registry_conflict_cb(
411
+ const DICTIONARY_ITEM *item __maybe_unused,
412
+ void *old_value,
413
+ void *new_value,
414
+ void *data __maybe_unused)
415
+{
416
+ struct nd_journal_file *njf_old = old_value;
417
+ struct nd_journal_file *njf_new = new_value;
418
+
419
+ if (njf_new->last_scan_monotonic_ut > njf_old->last_scan_monotonic_ut)
420
+ njf_old->last_scan_monotonic_ut = njf_new->last_scan_monotonic_ut;
421
+
422
+ if (njf_new->file_last_modified_ut > njf_old->file_last_modified_ut) {
423
+ njf_old->file_last_modified_ut = njf_new->file_last_modified_ut;
424
+ njf_old->size = njf_new->size;
425
+
426
+ njf_old->msg_last_ut = njf_old->file_last_modified_ut;
427
+ }
428
+
429
+ return false;
430
+}
431
+
432
+struct nd_journal_file_source {
433
+ usec_t first_ut;
434
+ usec_t last_ut;
435
+ size_t count;
436
+ uint64_t size;
437
+};
438
+
439
+#define print_duration(dst, dst_len, pos, remaining, duration, one, many, printed) \
440
+ do { \
441
+ if ((remaining) > (duration)) { \
442
+ uint64_t _count = (remaining) / (duration); \
443
+ uint64_t _rem = (remaining) - (_count * (duration)); \
444
+ (pos) += snprintfz( \
445
+ &(dst)[pos], \
446
+ (dst_len) - (pos), \
447
+ "%s%s%" PRIu64 " %s", \
448
+ (printed) ? ", " : "", \
449
+ _rem ? "" : "and ", \
450
+ _count, \
451
+ _count > 1 ? (many) : (one)); \
452
+ (remaining) = _rem; \
453
+ (printed) = true; \
454
+ } \
455
+ } while (0)
456
+
457
+static int nd_journal_file_to_json_array_cb(const DICTIONARY_ITEM *item, void *entry, void *data)
458
+{
459
+ struct nd_journal_file_source *nd_jfs = entry;
460
+ BUFFER *wb = data;
461
+
462
+ const char *name = dictionary_acquired_item_name(item);
463
+
464
+ buffer_json_add_array_item_object(wb);
465
+ {
466
+ char size_for_humans[128];
467
+ size_snprintf(size_for_humans, sizeof(size_for_humans), nd_jfs->size, "B", false);
468
+
469
+ char duration_for_humans[128];
470
+ duration_snprintf(
471
+ duration_for_humans,
472
+ sizeof(duration_for_humans),
473
+ (time_t)((nd_jfs->last_ut - nd_jfs->first_ut) / USEC_PER_SEC),
474
+ "s",
475
+ true);
476
+
477
+ char last_ut[RFC3339_MAX_LENGTH];
478
+ rfc3339_datetime_ut(last_ut, sizeof(last_ut), nd_jfs->last_ut, 0, true);
479
+
480
+ char info[1024];
481
+ snprintfz(
482
+ info,
483
+ sizeof(info),
484
+ "%zu files, total size %s, covering %s, last entry at %s",
485
+ nd_jfs->count,
486
+ size_for_humans,
487
+ duration_for_humans,
488
+ last_ut);
489
+
490
+ buffer_json_member_add_string(wb, "id", name);
491
+ buffer_json_member_add_string(wb, "name", name);
492
+ buffer_json_member_add_string(wb, "pill", size_for_humans);
493
+ buffer_json_member_add_string(wb, "info", info);
494
+ }
495
+ buffer_json_object_close(wb); // options object
496
+
497
+ return 1;
498
+}
499
+
500
+static bool nd_journal_file_merge_sizes(
501
+ const DICTIONARY_ITEM *item __maybe_unused,
502
+ void *old_value,
503
+ void *new_value,
504
+ void *data __maybe_unused)
505
+{
506
+ struct nd_journal_file_source *jfs = old_value, *njfs = new_value;
507
+ jfs->count += njfs->count;
508
+ jfs->size += njfs->size;
509
+
510
+ if (njfs->first_ut && njfs->first_ut < jfs->first_ut)
511
+ jfs->first_ut = njfs->first_ut;
512
+
513
+ if (njfs->last_ut && njfs->last_ut > jfs->last_ut)
514
+ jfs->last_ut = njfs->last_ut;
515
+
516
+ return false;
517
+}
518
+
519
+void available_journal_file_sources_to_json_array(BUFFER *wb)
520
+{
521
+ DICTIONARY *dict = dictionary_create(
522
+ DICT_OPTION_SINGLE_THREADED | DICT_OPTION_NAME_LINK_DONT_CLONE | DICT_OPTION_DONT_OVERWRITE_VALUE);
523
+ dictionary_register_conflict_callback(dict, nd_journal_file_merge_sizes, NULL);
524
+
525
+ struct nd_journal_file_source njfs_tmp = {0};
526
+
527
+ struct nd_journal_file *njf;
528
+ dfe_start_read(nd_journal_files_registry, njf)
529
+ {
530
+ njfs_tmp.first_ut = njf->msg_first_ut;
531
+ njfs_tmp.last_ut = njf->msg_last_ut;
532
+ njfs_tmp.count = 1;
533
+ njfs_tmp.size = njf->size;
534
+
535
+ dictionary_set(dict, ND_SD_JF_SOURCE_ALL_NAME, &njfs_tmp, sizeof(njfs_tmp));
536
+
537
+ if (njf->source_type & ND_SD_JF_LOCAL_ALL)
538
+ dictionary_set(dict, ND_SD_JF_SOURCE_LOCAL_NAME, &njfs_tmp, sizeof(njfs_tmp));
539
+ if (njf->source_type & ND_SD_JF_LOCAL_SYSTEM)
540
+ dictionary_set(dict, ND_SD_JF_SOURCE_LOCAL_SYSTEM_NAME, &njfs_tmp, sizeof(njfs_tmp));
541
+ if (njf->source_type & ND_SD_JF_LOCAL_USER)
542
+ dictionary_set(dict, ND_SD_JF_SOURCE_LOCAL_USERS_NAME, &njfs_tmp, sizeof(njfs_tmp));
543
+ if (njf->source_type & ND_SD_JF_LOCAL_OTHER)
544
+ dictionary_set(dict, ND_SD_JF_SOURCE_LOCAL_OTHER_NAME, &njfs_tmp, sizeof(njfs_tmp));
545
+ if (njf->source_type & ND_SD_JF_LOCAL_NAMESPACE)
546
+ dictionary_set(dict, ND_SD_JF_SOURCE_NAMESPACES_NAME, &njfs_tmp, sizeof(njfs_tmp));
547
+ if (njf->source_type & ND_SD_JF_REMOTE_ALL)
548
+ dictionary_set(dict, ND_SD_JF_SOURCE_REMOTES_NAME, &njfs_tmp, sizeof(njfs_tmp));
549
+ if (njf->source)
550
+ dictionary_set(dict, string2str(njf->source), &njfs_tmp, sizeof(njfs_tmp));
551
+ }
552
+ dfe_done(jf);
553
+
554
+ dictionary_sorted_walkthrough_read(dict, nd_journal_file_to_json_array_cb, wb);
555
+
556
+ dictionary_destroy(dict);
557
+}
558
+
559
+static void files_registry_delete_cb(const DICTIONARY_ITEM *item, void *value, void *data __maybe_unused)
560
+{
561
+ struct nd_journal_file *njf = value;
562
+ const char *filename = dictionary_acquired_item_name(item);
563
+ (void)filename;
564
+
565
+ internal_error(true, "removed journal file '%s'", filename);
566
+ string_freez(njf->source);
567
+}
568
+
569
+#define EXT_DOT_JOURNAL ".journal"
570
+#define EXT_DOT_JOURNAL_TILDA ".journal~"
571
+
572
+static struct {
573
+ const char *ext;
574
+ ssize_t len;
575
+} valid_journal_extension[] = {
576
+ {.ext = EXT_DOT_JOURNAL, .len = sizeof(EXT_DOT_JOURNAL) - 1},
577
+ {.ext = EXT_DOT_JOURNAL_TILDA, .len = sizeof(EXT_DOT_JOURNAL_TILDA) - 1},
578
+};
579
+
580
+bool is_journal_file(const char *filename, ssize_t len, const char **start_of_extension)
581
+{
582
+ if (len < 0)
583
+ len = (ssize_t)strlen(filename);
584
+
585
+ for (size_t i = 0; i < _countof(valid_journal_extension); i++) {
586
+ const char *ext = valid_journal_extension[i].ext;
587
+ ssize_t elen = valid_journal_extension[i].len;
588
+
589
+ if (len > elen && strcmp(filename + len - elen, ext) == 0) {
590
+ if (start_of_extension)
591
+ *start_of_extension = filename + len - elen;
592
+ return true;
593
+ }
594
+ }
595
+
596
+ if (start_of_extension)
597
+ *start_of_extension = NULL;
598
+
599
+ return false;
600
+}
601
+
602
+void nd_journal_directory_scan_recursively(DICTIONARY *files, DICTIONARY *dirs, const char *dirname, int depth)
603
+{
604
+ if (depth > VAR_LOG_JOURNAL_MAX_DEPTH)
605
+ return;
606
+
607
+ DIR *dir;
608
+ struct dirent *entry;
609
+ char full_path[FILENAME_MAX];
610
+
611
+ // Open the directory.
612
+ if ((dir = opendir(dirname)) == NULL) {
613
+ if (errno != ENOENT && errno != ENOTDIR)
614
+ netdata_log_error("Cannot opendir() '%s'", dirname);
615
+ return;
616
+ }
617
+
618
+ bool existing = false;
619
+ bool *found = dictionary_set(dirs, dirname, &existing, sizeof(existing));
620
+ if (*found)
621
+ return;
622
+ *found = true;
623
+
624
+ // Read each entry in the directory.
625
+ while ((entry = readdir(dir)) != NULL) {
626
+ if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
627
+ continue;
628
+
629
+ ssize_t len = snprintfz(full_path, sizeof(full_path), "%s/%s", dirname, entry->d_name);
630
+
631
+ if (entry->d_type == DT_DIR) {
632
+ nd_journal_directory_scan_recursively(files, dirs, full_path, depth++);
633
+ } else if (entry->d_type == DT_REG && is_journal_file(full_path, len, NULL)) {
634
+ if (files)
635
+ dictionary_set(files, full_path, NULL, 0);
636
+
637
+ send_newline_and_flush(&stdout_mutex);
638
+ } else if (entry->d_type == DT_LNK) {
639
+ struct stat info;
640
+ if (stat(full_path, &info) == -1)
641
+ continue;
642
+
643
+ if (S_ISDIR(info.st_mode)) {
644
+ // The symbolic link points to a directory
645
+ char resolved_path[FILENAME_MAX + 1];
646
+ if (realpath(full_path, resolved_path) != NULL) {
647
+ nd_journal_directory_scan_recursively(files, dirs, resolved_path, depth++);
648
+ }
649
+ } else if (S_ISREG(info.st_mode) && is_journal_file(full_path, len, NULL)) {
650
+ if (files)
651
+ dictionary_set(files, full_path, NULL, 0);
652
+
653
+ send_newline_and_flush(&stdout_mutex);
654
+ }
655
+ }
656
+ }
657
+
658
+ closedir(dir);
659
+}
660
+
661
+static size_t nd_journal_files_scans = 0;
662
+bool nd_journal_files_completed_once(void)
663
+{
664
+ return nd_journal_files_scans > 0;
665
+}
666
+
667
+int filenames_compar(const void *a, const void *b)
668
+{
669
+ const char *p1 = *(const char **)a;
670
+ const char *p2 = *(const char **)b;
671
+
672
+ const char *at1 = strchr(p1, '@');
673
+ const char *at2 = strchr(p2, '@');
674
+
675
+ if (!at1 && at2)
676
+ return -1;
677
+
678
+ if (at1 && !at2)
679
+ return 1;
680
+
681
+ if (!at1 && !at2)
682
+ return strcmp(p1, p2);
683
+
684
+ const char *dash1 = strrchr(at1, '-');
685
+ const char *dash2 = strrchr(at2, '-');
686
+
687
+ if (!dash1 || !dash2)
688
+ return strcmp(p1, p2);
689
+
690
+ uint64_t ts1 = strtoul(dash1 + 1, NULL, 16);
691
+ uint64_t ts2 = strtoul(dash2 + 1, NULL, 16);
692
+
693
+ if (ts1 > ts2)
694
+ return -1;
695
+
696
+ if (ts1 < ts2)
697
+ return 1;
698
+
699
+ return -strcmp(p1, p2);
700
+}
701
+
702
+void nd_journal_files_registry_update(void)
703
+{
704
+ static SPINLOCK spinlock = SPINLOCK_INITIALIZER;
705
+
706
+ if (spinlock_trylock(&spinlock)) {
707
+ usec_t scan_monotonic_ut = now_monotonic_usec();
708
+
709
+ DICTIONARY *files = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
710
+ DICTIONARY *dirs = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
711
+
712
+ for (unsigned i = 0; i < MAX_JOURNAL_DIRECTORIES; i++) {
713
+ if (!journal_directories[i].path)
714
+ break;
715
+ nd_journal_directory_scan_recursively(files, dirs, string2str(journal_directories[i].path), 0);
716
+ }
717
+
718
+ const char **array = mallocz(sizeof(const char *) * dictionary_entries(files));
719
+ size_t used = 0;
720
+
721
+ void *x;
722
+ dfe_start_read(files, x)
723
+ {
724
+ if (used >= dictionary_entries(files))
725
+ continue;
726
+ array[used++] = x_dfe.name;
727
+ }
728
+ dfe_done(x);
729
+
730
+ qsort(array, used, sizeof(const char *), filenames_compar);
731
+
732
+ for (size_t i = 0; i < used; i++) {
733
+ const char *full_path = array[i];
734
+
735
+ struct stat info;
736
+ if (stat(full_path, &info) == -1)
737
+ continue;
738
+
739
+ struct nd_journal_file njf_tmp = {
740
+ .file_last_modified_ut = info.st_mtim.tv_sec * USEC_PER_SEC + info.st_mtim.tv_nsec / NSEC_PER_USEC,
741
+ .last_scan_monotonic_ut = scan_monotonic_ut,
742
+ .size = info.st_size,
743
+ .max_journal_vs_realtime_delta_ut = JOURNAL_VS_REALTIME_DELTA_DEFAULT_UT,
744
+ };
745
+ struct nd_journal_file *njf =
746
+ dictionary_set(nd_journal_files_registry, full_path, &njf_tmp, sizeof(njf_tmp));
747
+ nd_journal_file_update_header(njf->filename, njf);
748
+ }
749
+ freez(array);
750
+ dictionary_destroy(files);
751
+ dictionary_destroy(dirs);
752
+
753
+ struct nd_journal_file *njf;
754
+ dfe_start_write(nd_journal_files_registry, njf)
755
+ {
756
+ if (njf->last_scan_monotonic_ut < scan_monotonic_ut)
757
+ dictionary_del(nd_journal_files_registry, njf_dfe.name);
758
+ }
759
+ dfe_done(njf);
760
+ dictionary_garbage_collect(nd_journal_files_registry);
761
+
762
+ nd_journal_files_scans++;
763
+ spinlock_unlock(&spinlock);
764
+
765
+ internal_error(
766
+ true,
767
+ "Journal library scan completed in %.3f ms",
768
+ (double)(now_monotonic_usec() - scan_monotonic_ut) / (double)USEC_PER_MS);
769
+ }
770
+}
771
+
772
+// ----------------------------------------------------------------------------
773
+
774
+int nd_journal_file_dict_items_backward_compar(const void *a, const void *b)
775
+{
776
+ const DICTIONARY_ITEM **ad = (const DICTIONARY_ITEM **)a, **bd = (const DICTIONARY_ITEM **)b;
777
+ struct nd_journal_file *njf_lhs = dictionary_acquired_item_value(*ad);
778
+ struct nd_journal_file *njf_rhs = dictionary_acquired_item_value(*bd);
779
+
780
+ // compare the last message timestamps
781
+ if (njf_lhs->msg_last_ut < njf_rhs->msg_last_ut)
782
+ return 1;
783
+
784
+ if (njf_lhs->msg_last_ut > njf_rhs->msg_last_ut)
785
+ return -1;
786
+
787
+ // compare the file last modification timestamps
788
+ if (njf_lhs->file_last_modified_ut < njf_rhs->file_last_modified_ut)
789
+ return 1;
790
+
791
+ if (njf_lhs->file_last_modified_ut > njf_rhs->file_last_modified_ut)
792
+ return -1;
793
+
794
+ // compare the first message timestamps
795
+ if (njf_lhs->msg_first_ut < njf_rhs->msg_first_ut)
796
+ return 1;
797
+
798
+ if (njf_lhs->msg_first_ut > njf_rhs->msg_first_ut)
799
+ return -1;
800
+
801
+ return 0;
802
+}
803
+
804
+int nd_journal_file_dict_items_forward_compar(const void *a, const void *b)
805
+{
806
+ return -nd_journal_file_dict_items_backward_compar(a, b);
807
+}
808
+
809
+static bool boot_id_conflict_cb(
810
+ const DICTIONARY_ITEM *item __maybe_unused,
811
+ void *old_value,
812
+ void *new_value,
813
+ void *data __maybe_unused)
814
+{
815
+ usec_t *old_usec = old_value;
816
+ usec_t *new_usec = new_value;
817
+
818
+ if (*new_usec < *old_usec) {
819
+ *old_usec = *new_usec;
820
+ return true;
821
+ }
822
+
823
+ return false;
824
+}
825
+
826
+void nd_journal_init_files_and_directories(void)
827
+{
828
+ unsigned d = 0;
829
+
830
+ // ------------------------------------------------------------------------
831
+ // setup the journal directories
832
+
833
+ journal_directories[d++].path = string_strdupz("/run/log/journal");
834
+ journal_directories[d++].path = string_strdupz("/var/log/journal");
835
+
836
+ if (*netdata_configured_host_prefix) {
837
+ char path[PATH_MAX];
838
+ snprintfz(path, sizeof(path), "%s/var/log/journal", netdata_configured_host_prefix);
839
+ journal_directories[d++].path = string_strdupz(path);
840
+ snprintfz(path, sizeof(path), "%s/run/log/journal", netdata_configured_host_prefix);
841
+ journal_directories[d++].path = string_strdupz(path);
842
+ }
843
+
844
+ const char *netdata_configured_log_dir = getenv("NETDATA_LOG_DIR");
845
+ if (netdata_configured_log_dir != NULL) {
846
+ char path[PATH_MAX];
847
+ snprintfz(path, sizeof(path), "%s/otel", netdata_configured_host_prefix);
848
+ journal_directories[d++].path = string_strdupz(path);
849
+ };
850
+
851
+ // terminate the list
852
+ journal_directories[d].path = NULL;
853
+
854
+ // ------------------------------------------------------------------------
855
+ // initialize the used hashes files registry
856
+
857
+ used_hashes_registry = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE);
858
+
859
+ systemd_journal_session = (now_realtime_usec() / USEC_PER_SEC) * USEC_PER_SEC;
860
+
861
+ nd_journal_files_registry = dictionary_create_advanced(
862
+ DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE, NULL, sizeof(struct nd_journal_file));
863
+
864
+ dictionary_register_insert_callback(nd_journal_files_registry, files_registry_insert_cb, NULL);
865
+ dictionary_register_delete_callback(nd_journal_files_registry, files_registry_delete_cb, NULL);
866
+ dictionary_register_conflict_callback(nd_journal_files_registry, files_registry_conflict_cb, NULL);
867
+
868
+ boot_ids_to_first_ut =
869
+ dictionary_create_advanced(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE, NULL, sizeof(usec_t));
870
+
871
+ dictionary_register_conflict_callback(boot_ids_to_first_ut, boot_id_conflict_cb, NULL);
872
+}
src/collectors/systemd-journal.plugin/systemd-journal-fstat.c
new
+93
@@ -0,0 +1,93 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "systemd-internals.h"
4
+
5
+#if !defined(HAVE_RUST_PROVIDER)
6
+
7
+// ----------------------------------------------------------------------------
8
+// fstat64 overloading to speed up libsystemd
9
+// https://github.com/systemd/systemd/pull/29261
10
+
11
+#include <dlfcn.h>
12
+#include <sys/stat.h>
13
+
14
+#define FSTAT_CACHE_MAX 1024
15
+struct fdstat64_cache_entry {
16
+ bool enabled;
17
+ bool updated;
18
+ int err_no;
19
+ struct stat64 stat;
20
+ int ret;
21
+ size_t cached_count;
22
+ size_t session;
23
+};
24
+
25
+struct fdstat64_cache_entry fstat64_cache[FSTAT_CACHE_MAX] = {0};
26
+__thread size_t fstat_thread_calls = 0;
27
+__thread size_t fstat_thread_cached_responses = 0;
28
+static __thread bool enable_thread_fstat = false;
29
+static __thread size_t fstat_caching_thread_session = 0;
30
+static size_t fstat_caching_global_session = 0;
31
+
32
+void fstat_cache_enable_on_thread(void)
33
+{
34
+ fstat_caching_thread_session = __atomic_add_fetch(&fstat_caching_global_session, 1, __ATOMIC_ACQUIRE);
35
+ enable_thread_fstat = true;
36
+}
37
+
38
+void fstat_cache_disable_on_thread(void)
39
+{
40
+ fstat_caching_thread_session = __atomic_add_fetch(&fstat_caching_global_session, 1, __ATOMIC_RELEASE);
41
+ enable_thread_fstat = false;
42
+}
43
+
44
+int fstat64(int fd, struct stat64 *buf)
45
+{
46
+ static int (*real_fstat)(int, struct stat64 *) = NULL;
47
+ if (!real_fstat)
48
+ real_fstat = dlsym(RTLD_NEXT, "fstat64");
49
+
50
+ fstat_thread_calls++;
51
+
52
+ if (fd >= 0 && fd < FSTAT_CACHE_MAX) {
53
+ if (enable_thread_fstat && fstat64_cache[fd].session != fstat_caching_thread_session) {
54
+ fstat64_cache[fd].session = fstat_caching_thread_session;
55
+ fstat64_cache[fd].enabled = true;
56
+ fstat64_cache[fd].updated = false;
57
+ }
58
+
59
+ if (fstat64_cache[fd].enabled && fstat64_cache[fd].updated &&
60
+ fstat64_cache[fd].session == fstat_caching_thread_session) {
61
+ fstat_thread_cached_responses++;
62
+ errno = fstat64_cache[fd].err_no;
63
+ *buf = fstat64_cache[fd].stat;
64
+ fstat64_cache[fd].cached_count++;
65
+ return fstat64_cache[fd].ret;
66
+ }
67
+ }
68
+
69
+ int ret = real_fstat(fd, buf);
70
+
71
+ if (fd >= 0 && fd < FSTAT_CACHE_MAX && fstat64_cache[fd].enabled &&
72
+ fstat64_cache[fd].session == fstat_caching_thread_session) {
73
+ fstat64_cache[fd].ret = ret;
74
+ fstat64_cache[fd].updated = true;
75
+ fstat64_cache[fd].err_no = errno;
76
+ fstat64_cache[fd].stat = *buf;
77
+ }
78
+
79
+ return ret;
80
+}
81
+
82
+#else // HAVE_RUST_PROVIDER
83
+
84
+// When using Rust provider, disable fstat caching entirely since
85
+// we will not rely on libsystemd.
86
+
87
+__thread size_t fstat_thread_calls = 0;
88
+__thread size_t fstat_thread_cached_responses = 0;
89
+
90
+void fstat_cache_enable_on_thread(void) { }
91
+void fstat_cache_disable_on_thread(void) { }
92
+
93
+#endif
src/collectors/systemd-journal.plugin/systemd-journal-sampling.h
new
+415
@@ -0,0 +1,415 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#ifndef NETDATA_ND_SD_JOURNAL_SAMPLING_H
4
+#define NETDATA_ND_SD_JOURNAL_SAMPLING_H
5
+
6
+// ----------------------------------------------------------------------------
7
+// sampling support
8
+
9
+static inline void sampling_query_init(LOGS_QUERY_STATUS *lqs, FACETS *facets)
10
+{
11
+ if (!lqs->rq.sampling)
12
+ return;
13
+
14
+ if (!lqs->rq.slice) {
15
+ // the user is doing a full data query
16
+ // disable sampling
17
+ lqs->rq.sampling = 0;
18
+ return;
19
+ }
20
+
21
+ if (lqs->rq.data_only) {
22
+ // the user is doing a data query
23
+ // disable sampling
24
+ lqs->rq.sampling = 0;
25
+ return;
26
+ }
27
+
28
+ if (!lqs->c.files_matched) {
29
+ // no files have been matched
30
+ // disable sampling
31
+ lqs->rq.sampling = 0;
32
+ return;
33
+ }
34
+
35
+ lqs->c.samples.slots = facets_histogram_slots(facets);
36
+ if (lqs->c.samples.slots < 2)
37
+ lqs->c.samples.slots = 2;
38
+ if (lqs->c.samples.slots > ND_SD_JOURNAL_SAMPLING_SLOTS)
39
+ lqs->c.samples.slots = ND_SD_JOURNAL_SAMPLING_SLOTS;
40
+
41
+ if (!lqs->rq.after_ut || !lqs->rq.before_ut || lqs->rq.after_ut >= lqs->rq.before_ut) {
42
+ // we don't have enough information for sampling
43
+ lqs->rq.sampling = 0;
44
+ return;
45
+ }
46
+
47
+ usec_t delta = lqs->rq.before_ut - lqs->rq.after_ut;
48
+ usec_t step = delta / facets_histogram_slots(facets) - 1;
49
+ if (step < 1)
50
+ step = 1;
51
+
52
+ lqs->c.samples_per_time_slot.start_ut = lqs->rq.after_ut;
53
+ lqs->c.samples_per_time_slot.end_ut = lqs->rq.before_ut;
54
+ lqs->c.samples_per_time_slot.step_ut = step;
55
+
56
+ // the minimum number of rows to enable sampling
57
+ lqs->c.samples.enable_after_samples = lqs->rq.sampling / 2;
58
+
59
+ size_t files_matched = lqs->c.files_matched;
60
+ if (!files_matched)
61
+ files_matched = 1;
62
+
63
+ // the minimum number of rows per file to enable sampling
64
+ lqs->c.samples_per_file.enable_after_samples = (lqs->rq.sampling / 4) / files_matched;
65
+ if (lqs->c.samples_per_file.enable_after_samples < lqs->rq.entries)
66
+ lqs->c.samples_per_file.enable_after_samples = lqs->rq.entries;
67
+
68
+ // the minimum number of rows per time slot to enable sampling
69
+ lqs->c.samples_per_time_slot.enable_after_samples = (lqs->rq.sampling / 4) / lqs->c.samples.slots;
70
+ if (lqs->c.samples_per_time_slot.enable_after_samples < lqs->rq.entries)
71
+ lqs->c.samples_per_time_slot.enable_after_samples = lqs->rq.entries;
72
+}
73
+
74
+static inline void sampling_file_init(LOGS_QUERY_STATUS *lqs, struct nd_journal_file *jf __maybe_unused)
75
+{
76
+ lqs->c.samples_per_file.sampled = 0;
77
+ lqs->c.samples_per_file.unsampled = 0;
78
+ lqs->c.samples_per_file.estimated = 0;
79
+ lqs->c.samples_per_file.every = 0;
80
+ lqs->c.samples_per_file.skipped = 0;
81
+ lqs->c.samples_per_file.recalibrate = 0;
82
+}
83
+
84
+static inline size_t sampling_file_lines_scanned_so_far(LOGS_QUERY_STATUS *lqs)
85
+{
86
+ size_t sampled = lqs->c.samples_per_file.sampled + lqs->c.samples_per_file.unsampled;
87
+ if (!sampled)
88
+ sampled = 1;
89
+ return sampled;
90
+}
91
+
92
+static inline void sampling_running_file_query_overlapping_timeframe_ut(
93
+ LOGS_QUERY_STATUS *lqs,
94
+ struct nd_journal_file *jf,
95
+ FACETS_ANCHOR_DIRECTION direction,
96
+ usec_t msg_ut,
97
+ usec_t *after_ut,
98
+ usec_t *before_ut)
99
+{
100
+ // find the overlap of the query and file timeframes
101
+ // taking into account the first message we encountered
102
+
103
+ usec_t oldest_ut, newest_ut;
104
+ if (direction == FACETS_ANCHOR_DIRECTION_FORWARD) {
105
+ // the first message we know (oldest)
106
+ oldest_ut = lqs->c.query_file.first_msg_ut ? lqs->c.query_file.first_msg_ut : jf->msg_first_ut;
107
+ if (!oldest_ut)
108
+ oldest_ut = lqs->c.query_file.start_ut;
109
+
110
+ if (jf->msg_last_ut)
111
+ newest_ut = MIN(lqs->c.query_file.stop_ut, jf->msg_last_ut);
112
+ else if (jf->file_last_modified_ut)
113
+ newest_ut = MIN(lqs->c.query_file.stop_ut, jf->file_last_modified_ut);
114
+ else
115
+ newest_ut = lqs->c.query_file.stop_ut;
116
+
117
+ if (msg_ut < oldest_ut)
118
+ oldest_ut = msg_ut - 1;
119
+ } else /* BACKWARD */ {
120
+ // the latest message we know (newest)
121
+ newest_ut = lqs->c.query_file.first_msg_ut ? lqs->c.query_file.first_msg_ut : jf->msg_last_ut;
122
+ if (!newest_ut)
123
+ newest_ut = lqs->c.query_file.start_ut;
124
+
125
+ if (jf->msg_first_ut)
126
+ oldest_ut = MAX(lqs->c.query_file.stop_ut, jf->msg_first_ut);
127
+ else
128
+ oldest_ut = lqs->c.query_file.stop_ut;
129
+
130
+ if (newest_ut < msg_ut)
131
+ newest_ut = msg_ut + 1;
132
+ }
133
+
134
+ *after_ut = oldest_ut;
135
+ *before_ut = newest_ut;
136
+}
137
+
138
+static inline double sampling_running_file_query_progress_by_time(
139
+ LOGS_QUERY_STATUS *lqs,
140
+ struct nd_journal_file *jf,
141
+ FACETS_ANCHOR_DIRECTION direction,
142
+ usec_t msg_ut)
143
+{
144
+ usec_t after_ut, before_ut, elapsed_ut;
145
+ sampling_running_file_query_overlapping_timeframe_ut(lqs, jf, direction, msg_ut, &after_ut, &before_ut);
146
+
147
+ if (direction == FACETS_ANCHOR_DIRECTION_FORWARD)
148
+ elapsed_ut = msg_ut - after_ut;
149
+ else
150
+ elapsed_ut = before_ut - msg_ut;
151
+
152
+ usec_t total_ut = before_ut - after_ut;
153
+ double progress = (double)elapsed_ut / (double)total_ut;
154
+
155
+ return progress;
156
+}
157
+
158
+static inline usec_t sampling_running_file_query_remaining_time(
159
+ LOGS_QUERY_STATUS *lqs,
160
+ struct nd_journal_file *jf,
161
+ FACETS_ANCHOR_DIRECTION direction,
162
+ usec_t msg_ut,
163
+ usec_t *total_time_ut,
164
+ usec_t *remaining_start_ut,
165
+ usec_t *remaining_end_ut)
166
+{
167
+ usec_t after_ut, before_ut;
168
+ sampling_running_file_query_overlapping_timeframe_ut(lqs, jf, direction, msg_ut, &after_ut, &before_ut);
169
+
170
+ // since we have a timestamp in msg_ut
171
+ // this timestamp can extend the overlap
172
+ if (msg_ut <= after_ut)
173
+ after_ut = msg_ut - 1;
174
+
175
+ if (msg_ut >= before_ut)
176
+ before_ut = msg_ut + 1;
177
+
178
+ // return the remaining duration
179
+ usec_t remaining_from_ut, remaining_to_ut;
180
+ if (direction == FACETS_ANCHOR_DIRECTION_FORWARD) {
181
+ remaining_from_ut = msg_ut;
182
+ remaining_to_ut = before_ut;
183
+ } else {
184
+ remaining_from_ut = after_ut;
185
+ remaining_to_ut = msg_ut;
186
+ }
187
+
188
+ usec_t remaining_ut = remaining_to_ut - remaining_from_ut;
189
+
190
+ if (total_time_ut)
191
+ *total_time_ut = (before_ut > after_ut) ? before_ut - after_ut : 1;
192
+
193
+ if (remaining_start_ut)
194
+ *remaining_start_ut = remaining_from_ut;
195
+
196
+ if (remaining_end_ut)
197
+ *remaining_end_ut = remaining_to_ut;
198
+
199
+ return remaining_ut;
200
+}
201
+
202
+static inline size_t sampling_running_file_query_estimate_remaining_lines_by_time(
203
+ LOGS_QUERY_STATUS *lqs,
204
+ struct nd_journal_file *jf,
205
+ FACETS_ANCHOR_DIRECTION direction,
206
+ usec_t msg_ut)
207
+{
208
+ size_t scanned_lines = sampling_file_lines_scanned_so_far(lqs);
209
+
210
+ // Calculate the proportion of time covered
211
+ usec_t total_time_ut, remaining_start_ut, remaining_end_ut;
212
+ usec_t remaining_time_ut = sampling_running_file_query_remaining_time(
213
+ lqs, jf, direction, msg_ut, &total_time_ut, &remaining_start_ut, &remaining_end_ut);
214
+ if (total_time_ut == 0)
215
+ total_time_ut = 1;
216
+
217
+ double proportion_by_time = (double)(total_time_ut - remaining_time_ut) / (double)total_time_ut;
218
+
219
+ if (proportion_by_time == 0 || proportion_by_time > 1.0 || !isfinite(proportion_by_time))
220
+ proportion_by_time = 1.0;
221
+
222
+ // Estimate the total number of lines in the file
223
+ size_t expected_matching_logs_by_time = (size_t)((double)scanned_lines / proportion_by_time);
224
+
225
+ if (jf->messages_in_file && expected_matching_logs_by_time > jf->messages_in_file)
226
+ expected_matching_logs_by_time = jf->messages_in_file;
227
+
228
+ // Calculate the estimated number of remaining lines
229
+ size_t remaining_logs_by_time = expected_matching_logs_by_time - scanned_lines;
230
+ if (remaining_logs_by_time < 1)
231
+ remaining_logs_by_time = 1;
232
+
233
+ // nd_log(NDLS_COLLECTORS, NDLP_INFO,
234
+ // "JOURNAL ESTIMATION: '%s' "
235
+ // "scanned_lines=%zu [sampled=%zu, unsampled=%zu, estimated=%zu], "
236
+ // "file [%"PRIu64" - %"PRIu64", duration %"PRId64", known lines in file %zu], "
237
+ // "query [%"PRIu64" - %"PRIu64", duration %"PRId64"], "
238
+ // "first message read from the file at %"PRIu64", current message at %"PRIu64", "
239
+ // "proportion of time %.2f %%, "
240
+ // "expected total lines in file %zu, "
241
+ // "remaining lines %zu, "
242
+ // "remaining time %"PRIu64" [%"PRIu64" - %"PRIu64", duration %"PRId64"]"
243
+ // , jf->filename
244
+ // , scanned_lines, fqs->samples_per_file.sampled, fqs->samples_per_file.unsampled, fqs->samples_per_file.estimated
245
+ // , jf->msg_first_ut, jf->msg_last_ut, jf->msg_last_ut - jf->msg_first_ut, jf->messages_in_file
246
+ // , fqs->query_file.start_ut, fqs->query_file.stop_ut, fqs->query_file.stop_ut - fqs->query_file.start_ut
247
+ // , fqs->query_file.first_msg_ut, msg_ut
248
+ // , proportion_by_time * 100.0
249
+ // , expected_matching_logs_by_time
250
+ // , remaining_logs_by_time
251
+ // , remaining_time_ut, remaining_start_ut, remaining_end_ut, remaining_end_ut - remaining_start_ut
252
+ // );
253
+
254
+ return remaining_logs_by_time;
255
+}
256
+
257
+static inline size_t sampling_running_file_query_estimate_remaining_lines(
258
+ NsdJournal *j __maybe_unused,
259
+ LOGS_QUERY_STATUS *lqs,
260
+ struct nd_journal_file *jf,
261
+ FACETS_ANCHOR_DIRECTION direction,
262
+ usec_t msg_ut)
263
+{
264
+ size_t remaining_logs_by_seqnum = 0;
265
+
266
+#ifdef HAVE_SD_JOURNAL_GET_SEQNUM
267
+ size_t expected_matching_logs_by_seqnum = 0;
268
+ double proportion_by_seqnum = 0.0;
269
+ uint64_t current_msg_seqnum;
270
+ NsdId128 current_msg_writer;
271
+ if (!lqs->c.query_file.first_msg_seqnum || nsd_journal_get_seqnum(j, ¤t_msg_seqnum, ¤t_msg_writer) < 0) {
272
+ lqs->c.query_file.first_msg_seqnum = 0;
273
+ lqs->c.query_file.first_msg_writer = NSD_ID128_NULL;
274
+ } else if (jf->messages_in_file) {
275
+ size_t scanned_lines = sampling_file_lines_scanned_so_far(lqs);
276
+
277
+ double proportion_of_all_lines_so_far;
278
+ if (direction == FACETS_ANCHOR_DIRECTION_FORWARD)
279
+ proportion_of_all_lines_so_far = (double)scanned_lines / (double)(current_msg_seqnum - jf->first_seqnum);
280
+ else
281
+ proportion_of_all_lines_so_far = (double)scanned_lines / (double)(jf->last_seqnum - current_msg_seqnum);
282
+
283
+ if (proportion_of_all_lines_so_far > 1.0)
284
+ proportion_of_all_lines_so_far = 1.0;
285
+
286
+ expected_matching_logs_by_seqnum = (size_t)(proportion_of_all_lines_so_far * (double)jf->messages_in_file);
287
+
288
+ proportion_by_seqnum = (double)scanned_lines / (double)expected_matching_logs_by_seqnum;
289
+
290
+ if (proportion_by_seqnum == 0 || proportion_by_seqnum > 1.0 || !isfinite(proportion_by_seqnum))
291
+ proportion_by_seqnum = 1.0;
292
+
293
+ remaining_logs_by_seqnum = expected_matching_logs_by_seqnum - scanned_lines;
294
+ if (!remaining_logs_by_seqnum)
295
+ remaining_logs_by_seqnum = 1;
296
+ }
297
+#endif
298
+
299
+ if (remaining_logs_by_seqnum)
300
+ return remaining_logs_by_seqnum;
301
+
302
+ return sampling_running_file_query_estimate_remaining_lines_by_time(lqs, jf, direction, msg_ut);
303
+}
304
+
305
+static inline void sampling_decide_file_sampling_every(
306
+ NsdJournal *j,
307
+ LOGS_QUERY_STATUS *lqs,
308
+ struct nd_journal_file *jf,
309
+ FACETS_ANCHOR_DIRECTION direction,
310
+ usec_t msg_ut)
311
+{
312
+ size_t files_matched = lqs->c.files_matched;
313
+ if (!files_matched)
314
+ files_matched = 1;
315
+
316
+ size_t remaining_lines = sampling_running_file_query_estimate_remaining_lines(j, lqs, jf, direction, msg_ut);
317
+ size_t wanted_samples = (lqs->rq.sampling / 2) / files_matched;
318
+ if (!wanted_samples)
319
+ wanted_samples = 1;
320
+
321
+ lqs->c.samples_per_file.every = remaining_lines / wanted_samples;
322
+
323
+ if (lqs->c.samples_per_file.every < 1)
324
+ lqs->c.samples_per_file.every = 1;
325
+}
326
+
327
+typedef enum {
328
+ SAMPLING_STOP_AND_ESTIMATE = -1,
329
+ SAMPLING_FULL = 0,
330
+ SAMPLING_SKIP_FIELDS = 1,
331
+} sampling_t;
332
+
333
+static inline sampling_t is_row_in_sample(
334
+ NsdJournal *j,
335
+ LOGS_QUERY_STATUS *lqs,
336
+ struct nd_journal_file *jf,
337
+ usec_t msg_ut,
338
+ FACETS_ANCHOR_DIRECTION direction,
339
+ bool candidate_to_keep)
340
+{
341
+ if (!lqs->rq.sampling || candidate_to_keep)
342
+ return SAMPLING_FULL;
343
+
344
+ if (unlikely(msg_ut < lqs->c.samples_per_time_slot.start_ut))
345
+ msg_ut = lqs->c.samples_per_time_slot.start_ut;
346
+ if (unlikely(msg_ut > lqs->c.samples_per_time_slot.end_ut))
347
+ msg_ut = lqs->c.samples_per_time_slot.end_ut;
348
+
349
+ size_t slot = (msg_ut - lqs->c.samples_per_time_slot.start_ut) / lqs->c.samples_per_time_slot.step_ut;
350
+ if (slot >= lqs->c.samples.slots)
351
+ slot = lqs->c.samples.slots - 1;
352
+
353
+ bool should_sample = false;
354
+
355
+ if (lqs->c.samples.sampled < lqs->c.samples.enable_after_samples ||
356
+ lqs->c.samples_per_file.sampled < lqs->c.samples_per_file.enable_after_samples ||
357
+ lqs->c.samples_per_time_slot.sampled[slot] < lqs->c.samples_per_time_slot.enable_after_samples)
358
+ should_sample = true;
359
+
360
+ else if (lqs->c.samples_per_file.recalibrate >= ND_SD_JOURNAL_SAMPLING_RECALIBRATE || !lqs->c.samples_per_file.every) {
361
+ // this is the first to be unsampled for this file
362
+ sampling_decide_file_sampling_every(j, lqs, jf, direction, msg_ut);
363
+ lqs->c.samples_per_file.recalibrate = 0;
364
+ should_sample = true;
365
+ } else {
366
+ // we sample 1 every fqs->samples_per_file.every
367
+ if (lqs->c.samples_per_file.skipped >= lqs->c.samples_per_file.every) {
368
+ lqs->c.samples_per_file.skipped = 0;
369
+ should_sample = true;
370
+ } else
371
+ lqs->c.samples_per_file.skipped++;
372
+ }
373
+
374
+ if (should_sample) {
375
+ lqs->c.samples.sampled++;
376
+ lqs->c.samples_per_file.sampled++;
377
+ lqs->c.samples_per_time_slot.sampled[slot]++;
378
+
379
+ return SAMPLING_FULL;
380
+ }
381
+
382
+ lqs->c.samples_per_file.recalibrate++;
383
+
384
+ lqs->c.samples.unsampled++;
385
+ lqs->c.samples_per_file.unsampled++;
386
+ lqs->c.samples_per_time_slot.unsampled[slot]++;
387
+
388
+ if (lqs->c.samples_per_file.unsampled > lqs->c.samples_per_file.sampled) {
389
+ double progress_by_time = sampling_running_file_query_progress_by_time(lqs, jf, direction, msg_ut);
390
+
391
+ if (progress_by_time > ND_SD_JOURNAL_ENABLE_ESTIMATIONS_FILE_PERCENTAGE)
392
+ return SAMPLING_STOP_AND_ESTIMATE;
393
+ }
394
+
395
+ return SAMPLING_SKIP_FIELDS;
396
+}
397
+
398
+static inline void sampling_update_running_query_file_estimates(
399
+ FACETS *facets,
400
+ NsdJournal *j,
401
+ LOGS_QUERY_STATUS *lqs,
402
+ struct nd_journal_file *jf,
403
+ usec_t msg_ut,
404
+ FACETS_ANCHOR_DIRECTION direction)
405
+{
406
+ usec_t total_time_ut, remaining_start_ut, remaining_end_ut;
407
+ sampling_running_file_query_remaining_time(
408
+ lqs, jf, direction, msg_ut, &total_time_ut, &remaining_start_ut, &remaining_end_ut);
409
+ size_t remaining_lines = sampling_running_file_query_estimate_remaining_lines(j, lqs, jf, direction, msg_ut);
410
+ facets_update_estimations(facets, remaining_start_ut, remaining_end_ut, remaining_lines);
411
+ lqs->c.samples.estimated += remaining_lines;
412
+ lqs->c.samples_per_file.estimated += remaining_lines;
413
+}
414
+
415
+#endif //NETDATA_ND_SD_JOURNAL_SAMPLING_H
src/collectors/systemd-journal.plugin/systemd-journal-self-signed-certs.sh
renamed
src/collectors/systemd-journal.plugin/systemd-journal-watcher.c
new
+613
@@ -0,0 +1,613 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "systemd-internals.h"
4
+#include <sys/inotify.h>
5
+
6
+#define INITIAL_WATCHES 256
7
+
8
+#define WATCH_FOR (IN_CREATE | IN_MODIFY | IN_DELETE | IN_DELETE_SELF | IN_MOVED_FROM | IN_MOVED_TO | IN_UNMOUNT)
9
+
10
+typedef uint32_t INOTIFY_MASK;
11
+
12
+ENUM_STR_MAP_DEFINE(INOTIFY_MASK) = {
13
+ // helpers (combine multiple flags)
14
+ // must be first in the list
15
+ {.id = IN_ALL_EVENTS, .name = "IN_ALL_EVENTS"},
16
+ {.id = IN_CLOSE, .name = "IN_CLOSE"},
17
+ {.id = IN_MOVE, .name = "IN_MOVE"},
18
+
19
+ // individual flags
20
+ {.id = IN_ACCESS, .name = "IN_ACCESS"},
21
+ {.id = IN_MODIFY, .name = "IN_MODIFY"},
22
+ {.id = IN_ATTRIB, .name = "IN_ATTRIB"},
23
+ {.id = IN_CLOSE_WRITE, .name = "IN_CLOSE_WRITE"},
24
+ {.id = IN_CLOSE_NOWRITE, .name = "IN_CLOSE_NOWRITE"},
25
+ {.id = IN_OPEN, .name = "IN_OPEN"},
26
+ {.id = IN_MOVED_FROM, .name = "IN_MOVED_FROM"},
27
+ {.id = IN_MOVED_TO, .name = "IN_MOVED_TO"},
28
+ {.id = IN_CREATE, .name = "IN_CREATE"},
29
+ {.id = IN_DELETE, .name = "IN_DELETE"},
30
+ {.id = IN_DELETE_SELF, .name = "IN_DELETE_SELF"},
31
+ {.id = IN_MOVE_SELF, .name = "IN_MOVE_SELF"},
32
+ {.id = IN_UNMOUNT, .name = "IN_UNMOUNT"},
33
+ {.id = IN_Q_OVERFLOW, .name = "IN_Q_OVERFLOW"},
34
+ {.id = IN_IGNORED, .name = "IN_IGNORED"},
35
+ {.id = IN_ONLYDIR, .name = "IN_ONLYDIR"},
36
+ {.id = IN_DONT_FOLLOW, .name = "IN_DONT_FOLLOW"},
37
+ {.id = IN_EXCL_UNLINK, .name = "IN_EXCL_UNLINK"},
38
+#ifdef IN_MASK_CREATE
39
+ {.id = IN_MASK_CREATE, .name = "IN_MASK_CREATE"},
40
+#endif
41
+ {.id = IN_MASK_ADD, .name = "IN_MASK_ADD"},
42
+ {.id = IN_ISDIR, .name = "IN_ISDIR"},
43
+ {.id = IN_ONESHOT, .name = "IN_ONESHOT"},
44
+
45
+ // terminator
46
+ {.id = 0, .name = NULL}};
47
+
48
+BITMAP_STR_DEFINE_FUNCTIONS(INOTIFY_MASK, 0, "UNKNOWN");
49
+
50
+DEFINE_JUDYL_TYPED(SYMLINKED_DIRS, STRING *);
51
+
52
+typedef struct watch_entry {
53
+ int slot;
54
+
55
+ int wd; // Watch descriptor
56
+ char *path; // Dynamically allocated path
57
+
58
+ struct watch_entry *next; // for the free list
59
+} WatchEntry;
60
+
61
+typedef struct {
62
+ WatchEntry *watchList;
63
+ WatchEntry *freeList;
64
+ int watchCount;
65
+ int watchListSize;
66
+
67
+ size_t errors;
68
+
69
+ SYMLINKED_DIRS_JudyLSet symlinkedDirs;
70
+ DICTIONARY *pending;
71
+} Watcher;
72
+
73
+static WatchEntry *get_slot(Watcher *watcher)
74
+{
75
+ WatchEntry *t;
76
+
77
+ if (watcher->freeList != NULL) {
78
+ t = watcher->freeList;
79
+ watcher->freeList = t->next;
80
+ t->next = NULL;
81
+ return t;
82
+ }
83
+
84
+ if (watcher->watchCount == watcher->watchListSize) {
85
+ watcher->watchListSize *= 2;
86
+ watcher->watchList = reallocz(watcher->watchList, watcher->watchListSize * sizeof(WatchEntry));
87
+ }
88
+
89
+ watcher->watchList[watcher->watchCount] = (WatchEntry){
90
+ .slot = watcher->watchCount,
91
+ .wd = -1,
92
+ .path = NULL,
93
+ .next = NULL,
94
+ };
95
+ t = &watcher->watchList[watcher->watchCount];
96
+ watcher->watchCount++;
97
+
98
+ return t;
99
+}
100
+
101
+static void free_slot(Watcher *watcher, WatchEntry *t)
102
+{
103
+ t->wd = -1;
104
+ freez(t->path);
105
+ t->path = NULL;
106
+
107
+ // link it to the free list
108
+ t->next = watcher->freeList;
109
+ watcher->freeList = t;
110
+}
111
+
112
+static int add_watch(Watcher *watcher, int inotifyFd, const char *path)
113
+{
114
+ WatchEntry *t = get_slot(watcher);
115
+
116
+ errno_clear();
117
+ t->wd = inotify_add_watch(inotifyFd, path, WATCH_FOR);
118
+ if (t->wd == -1) {
119
+ nd_log(NDLS_COLLECTORS, NDLP_ERR, "JOURNAL WATCHER: cannot watch directory: '%s'", path);
120
+
121
+ free_slot(watcher, t);
122
+
123
+ struct stat info;
124
+ if (stat(path, &info) == 0 && S_ISDIR(info.st_mode)) {
125
+ // the directory exists, but we failed to add the watch
126
+ // increase errors
127
+ watcher->errors++;
128
+ }
129
+ } else {
130
+ t->path = strdupz(path);
131
+
132
+ nd_log(NDLS_COLLECTORS, NDLP_DEBUG, "JOURNAL WATCHER: watching directory: '%s'", path);
133
+ }
134
+ return t->wd;
135
+}
136
+
137
+static void remove_watch(Watcher *watcher, int inotifyFd, int wd)
138
+{
139
+ errno_clear();
140
+
141
+ int i;
142
+ for (i = 0; i < watcher->watchCount; ++i) {
143
+ if (watcher->watchList[i].wd == wd) {
144
+ nd_log(
145
+ NDLS_COLLECTORS,
146
+ NDLP_DEBUG,
147
+ "JOURNAL WATCHER: removing watch from directory: '%s'",
148
+ watcher->watchList[i].path);
149
+
150
+ if (inotify_rm_watch(inotifyFd, watcher->watchList[i].wd) == -1)
151
+ nd_log(NDLS_COLLECTORS, NDLP_ERR, "JOURNAL WATCHER: inotify_rm_watch() returned -1");
152
+
153
+ free_slot(watcher, &watcher->watchList[i]);
154
+ return;
155
+ }
156
+ }
157
+
158
+ nd_log(NDLS_COLLECTORS, NDLP_WARNING, "JOURNAL WATCHER: cannot find directory watch %d to remove.", wd);
159
+}
160
+
161
+static void free_watches(Watcher *watcher, int inotifyFd)
162
+{
163
+ for (int i = 0; i < watcher->watchCount; ++i) {
164
+ if (watcher->watchList[i].wd != -1) {
165
+ if (inotify_rm_watch(inotifyFd, watcher->watchList[i].wd) == -1)
166
+ nd_log(NDLS_COLLECTORS, NDLP_ERR, "JOURNAL WATCHER: inotify_rm_watch() returned -1");
167
+ free_slot(watcher, &watcher->watchList[i]);
168
+ }
169
+ }
170
+ freez(watcher->watchList);
171
+ watcher->watchList = NULL;
172
+
173
+ dictionary_destroy(watcher->pending);
174
+ watcher->pending = NULL;
175
+}
176
+
177
+static void free_symlinked_dirs(Watcher *watcher)
178
+{
179
+ Word_t idx = 0;
180
+ STRING *value;
181
+ while ((value = SYMLINKED_DIRS_FIRST(&watcher->symlinkedDirs, &idx))) {
182
+ SYMLINKED_DIRS_DEL(&watcher->symlinkedDirs, idx);
183
+ STRING *key = (STRING *)idx;
184
+ string_freez(key);
185
+ string_freez(value);
186
+ }
187
+}
188
+
189
+static char *get_path_from_wd(Watcher *watcher, int wd)
190
+{
191
+ for (int i = 0; i < watcher->watchCount; ++i) {
192
+ if (watcher->watchList[i].wd == wd)
193
+ return watcher->watchList[i].path;
194
+ }
195
+ return NULL;
196
+}
197
+
198
+static bool is_directory_watched(Watcher *watcher, const char *path)
199
+{
200
+ for (int i = 0; i < watcher->watchCount; ++i) {
201
+ if (watcher->watchList[i].wd != -1 && strcmp(watcher->watchList[i].path, path) == 0) {
202
+ return true;
203
+ }
204
+ }
205
+ return false;
206
+}
207
+
208
+static void watch_directory_and_subdirectories(Watcher *watcher, int inotifyFd, const char *basePath)
209
+{
210
+ DICTIONARY *dirs = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
211
+
212
+ // First resolve any symlinks in the base path
213
+ char real_path[PATH_MAX];
214
+ if (realpath(basePath, real_path) == NULL) {
215
+ // If realpath fails, try using the original path
216
+ strncpyz(real_path, basePath, sizeof(real_path));
217
+ }
218
+
219
+ nd_journal_directory_scan_recursively(NULL, dirs, real_path, 0);
220
+
221
+ void *x;
222
+ dfe_start_read(dirs, x)
223
+ {
224
+ const char *dirname = x_dfe.name;
225
+ char resolved_path[PATH_MAX];
226
+
227
+ // Resolve symlinks for each subdirectory
228
+ if (realpath(dirname, resolved_path) != NULL) {
229
+ // Check if this directory is already being watched
230
+ if (!is_directory_watched(watcher, resolved_path)) {
231
+ add_watch(watcher, inotifyFd, resolved_path);
232
+ }
233
+ } else {
234
+ // If realpath fails, try with original path
235
+ if (!is_directory_watched(watcher, dirname)) {
236
+ add_watch(watcher, inotifyFd, dirname);
237
+ }
238
+ }
239
+ }
240
+ dfe_done(x);
241
+
242
+ dictionary_destroy(dirs);
243
+}
244
+
245
+static bool is_subpath(const char *path, const char *subpath)
246
+{
247
+ // Use strncmp to compare the paths
248
+ if (strncmp(path, subpath, strlen(path)) == 0) {
249
+ // Ensure that the next character is a '/' or '\0'
250
+ char next_char = subpath[strlen(path)];
251
+ return next_char == '/' || next_char == '\0';
252
+ }
253
+
254
+ return false;
255
+}
256
+
257
+void remove_directory_watch(Watcher *watcher, int inotifyFd, const char *dirPath)
258
+{
259
+ for (int i = 0; i < watcher->watchCount; ++i) {
260
+ WatchEntry *t = &watcher->watchList[i];
261
+ if (t->wd != -1 && is_subpath(dirPath, t->path)) {
262
+ if (inotify_rm_watch(inotifyFd, t->wd) == -1)
263
+ nd_log(
264
+ NDLS_COLLECTORS, NDLP_ERR, "JOURNAL WATCHER: inotify_rm_watch() on path '%s' returned -1", t->path);
265
+ else
266
+ nd_log(NDLS_COLLECTORS, NDLP_DEBUG, "JOURNAL WATCHER: stopped watching directory '%s'", t->path);
267
+ free_slot(watcher, t);
268
+ }
269
+ }
270
+
271
+ struct nd_journal_file *njf;
272
+ dfe_start_write(nd_journal_files_registry, njf)
273
+ {
274
+ if (is_subpath(dirPath, njf->filename))
275
+ dictionary_del(nd_journal_files_registry, njf->filename);
276
+ }
277
+ dfe_done(njf);
278
+
279
+ dictionary_garbage_collect(nd_journal_files_registry);
280
+}
281
+
282
+void process_event(Watcher *watcher, int inotifyFd, struct inotify_event *event)
283
+{
284
+ errno_clear();
285
+
286
+ if (!event->len) {
287
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
288
+ INOTIFY_MASK_2buffer(wb, event->mask, ", ");
289
+ nd_log(
290
+ NDLS_COLLECTORS,
291
+ NDLP_NOTICE,
292
+ "JOURNAL WATCHER: received event with mask %u (%s) and len %u (this is zero) - ignoring it.",
293
+ event->mask,
294
+ buffer_tostring(wb),
295
+ event->len);
296
+ return;
297
+ }
298
+
299
+ char *dirPath = get_path_from_wd(watcher, event->wd);
300
+ if (!dirPath) {
301
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
302
+ INOTIFY_MASK_2buffer(wb, event->mask, ", ");
303
+ nd_log(
304
+ NDLS_COLLECTORS,
305
+ NDLP_NOTICE,
306
+ "JOURNAL WATCHER: received event with mask %u (%s) and len %u for path: '%s' - "
307
+ "but we can't find its watch descriptor - ignoring it.",
308
+ event->mask,
309
+ buffer_tostring(wb),
310
+ event->len,
311
+ event->name);
312
+ return;
313
+ }
314
+
315
+#if 0
316
+ {
317
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
318
+ INOTIFY_MASK_2buffer(wb, event->mask, ", ");
319
+ nd_log(NDLS_COLLECTORS, NDLP_DEBUG,
320
+ "JOURNAL WATCHER: received event with mask %u (%s) for path: '%s' inside '%s'",
321
+ event->mask, buffer_tostring(wb), event->name, dirPath);
322
+ }
323
+#endif
324
+
325
+ if (event->mask & IN_DELETE_SELF) {
326
+ remove_watch(watcher, inotifyFd, event->wd);
327
+ return;
328
+ }
329
+
330
+ static __thread char fullPath[PATH_MAX];
331
+ snprintfz(fullPath, sizeof(fullPath), "%s/%s", dirPath, event->name);
332
+
333
+ bool is_dir = event->mask & IN_ISDIR;
334
+ char resolved_path[PATH_MAX];
335
+ const char *path_to_use = fullPath;
336
+
337
+ if (event->mask & (IN_CREATE | IN_MOVED_TO)) {
338
+ // Give the system a moment to establish the symlink
339
+ sleep_usec(1000); // 1ms sleep
340
+
341
+ struct stat st;
342
+ if (lstat(fullPath, &st) == 0) {
343
+ if (S_ISLNK(st.st_mode)) {
344
+ // It's a symlink - resolve it
345
+ if (realpath(fullPath, resolved_path) != NULL) {
346
+ path_to_use = resolved_path;
347
+
348
+ // Check if it points to a directory
349
+ if (stat(resolved_path, &st) == 0 && S_ISDIR(st.st_mode)) {
350
+ is_dir = true;
351
+
352
+ STRING *fullPathString = string_strdupz(fullPath);
353
+ STRING *symlinked = SYMLINKED_DIRS_GET(&watcher->symlinkedDirs, (uintptr_t)fullPathString);
354
+ if (!symlinked) {
355
+ SYMLINKED_DIRS_SET(
356
+ &watcher->symlinkedDirs, (uintptr_t)fullPathString, string_strdupz(resolved_path));
357
+
358
+ // we leave fullPathString allocated, as it's now in the JudyL set
359
+
360
+ nd_log(
361
+ NDLS_COLLECTORS,
362
+ NDLP_DEBUG,
363
+ "JOURNAL WATCHER: New symlinked directory created: '%s' -> '%s'",
364
+ fullPath,
365
+ resolved_path);
366
+ } else if (string_strcmp(symlinked, resolved_path) != 0) {
367
+ SYMLINKED_DIRS_SET(
368
+ &watcher->symlinkedDirs, (uintptr_t)fullPathString, string_strdupz(resolved_path));
369
+
370
+ nd_log(
371
+ NDLS_COLLECTORS,
372
+ NDLP_DEBUG,
373
+ "JOURNAL WATCHER: Updated symlinked directory: '%s' -> '%s' (was '%s')",
374
+ fullPath,
375
+ resolved_path,
376
+ string2str(symlinked));
377
+
378
+ string_freez(symlinked);
379
+
380
+ // we need to free this, since it was already in the JudyL set
381
+ string_freez(fullPathString);
382
+ } else
383
+ string_freez(fullPathString); // we don't need it anymore
384
+ }
385
+ }
386
+ }
387
+ }
388
+ } else if (event->mask & IN_DELETE) {
389
+ // Check if it was a symlink
390
+ STRING *fullPathString = string_strdupz(fullPath);
391
+ STRING *symlinked = SYMLINKED_DIRS_GET(&watcher->symlinkedDirs, (uintptr_t)fullPathString);
392
+ if (symlinked) {
393
+ strncpyz(resolved_path, string2str(symlinked), sizeof(resolved_path) - 1);
394
+ path_to_use = resolved_path;
395
+ SYMLINKED_DIRS_DEL(&watcher->symlinkedDirs, (uintptr_t)fullPathString);
396
+ string_freez(fullPathString); // to remove also the one referenced in the JudyL set
397
+ string_freez(symlinked);
398
+ is_dir = true;
399
+
400
+ nd_log(
401
+ NDLS_COLLECTORS,
402
+ NDLP_DEBUG,
403
+ "JOURNAL WATCHER: Deleted symlinked directory: '%s' -> '%s'",
404
+ fullPath,
405
+ resolved_path);
406
+ }
407
+ string_freez(fullPathString); // the one we allocated above
408
+ }
409
+
410
+ if (is_dir) {
411
+ if (event->mask & (IN_DELETE | IN_MOVED_FROM)) {
412
+ nd_log(NDLS_COLLECTORS, NDLP_DEBUG, "JOURNAL WATCHER: Directory deleted or moved out: '%s'", path_to_use);
413
+
414
+ remove_directory_watch(watcher, inotifyFd, path_to_use);
415
+ } else if (event->mask & (IN_CREATE | IN_MOVED_TO)) {
416
+ nd_log(
417
+ NDLS_COLLECTORS, NDLP_DEBUG, "JOURNAL WATCHER: New directory created or moved in: '%s'", path_to_use);
418
+
419
+ watch_directory_and_subdirectories(watcher, inotifyFd, path_to_use);
420
+ } else {
421
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
422
+ INOTIFY_MASK_2buffer(wb, event->mask, ", ");
423
+ nd_log(
424
+ NDLS_COLLECTORS,
425
+ NDLP_WARNING,
426
+ "JOURNAL WATCHER: Received unhandled event with mask %u (%s) for directory '%s'",
427
+ event->mask,
428
+ buffer_tostring(wb),
429
+ path_to_use);
430
+ }
431
+ } else if (is_journal_file(event->name, (ssize_t)strlen(event->name), NULL)) {
432
+ dictionary_set(watcher->pending, path_to_use, NULL, 0);
433
+ } else {
434
+ CLEAN_BUFFER *wb = buffer_create(0, NULL);
435
+ INOTIFY_MASK_2buffer(wb, event->mask, ", ");
436
+ nd_log(
437
+ NDLS_COLLECTORS,
438
+ NDLP_DEBUG,
439
+ "JOURNAL WATCHER: ignoring event with mask %u (%s) for file '%s' ('%s')",
440
+ event->mask,
441
+ buffer_tostring(wb),
442
+ path_to_use,
443
+ fullPath);
444
+ }
445
+}
446
+
447
+static void process_pending(Watcher *watcher)
448
+{
449
+ errno_clear();
450
+
451
+ void *x;
452
+ dfe_start_write(watcher->pending, x)
453
+ {
454
+ struct stat info;
455
+ const char *fullPath = x_dfe.name;
456
+
457
+ if (stat(fullPath, &info) != 0) {
458
+ nd_log(
459
+ NDLS_COLLECTORS,
460
+ NDLP_DEBUG,
461
+ "JOURNAL WATCHER: file '%s' no longer exists, removing it from the registry",
462
+ fullPath);
463
+
464
+ dictionary_del(nd_journal_files_registry, fullPath);
465
+ } else if (S_ISREG(info.st_mode)) {
466
+ // nd_log(NDLS_COLLECTORS, NDLP_DEBUG,
467
+ // "JOURNAL WATCHER: file '%s' has been added/updated, updating the registry",
468
+ // fullPath);
469
+
470
+ struct nd_journal_file t = {
471
+ .file_last_modified_ut = info.st_mtim.tv_sec * USEC_PER_SEC + info.st_mtim.tv_nsec / NSEC_PER_USEC,
472
+ .last_scan_monotonic_ut = now_monotonic_usec(),
473
+ .size = info.st_size,
474
+ .max_journal_vs_realtime_delta_ut = JOURNAL_VS_REALTIME_DELTA_DEFAULT_UT,
475
+ };
476
+ struct nd_journal_file *jf = dictionary_set(nd_journal_files_registry, fullPath, &t, sizeof(t));
477
+ nd_journal_file_update_header(jf->filename, jf);
478
+ }
479
+
480
+ dictionary_del(watcher->pending, fullPath);
481
+ }
482
+ dfe_done(x);
483
+
484
+ dictionary_garbage_collect(watcher->pending);
485
+}
486
+
487
+size_t journal_watcher_wanted_session_id = 0;
488
+
489
+void nd_journal_watcher_restart(void)
490
+{
491
+ __atomic_add_fetch(&journal_watcher_wanted_session_id, 1, __ATOMIC_RELAXED);
492
+}
493
+
494
+static bool process_inotify_events(struct buffered_reader *reader, Watcher *watcher, int inotifyFd)
495
+{
496
+ errno_clear();
497
+
498
+ bool unmount_event = false;
499
+ ssize_t processed = 0;
500
+
501
+ // Process as many complete events as we can
502
+ while (processed + (ssize_t)sizeof(struct inotify_event) <= reader->read_len) {
503
+ struct inotify_event *event = (struct inotify_event *)(reader->read_buffer + processed);
504
+
505
+ if (event->len > NAME_MAX + 1) {
506
+ // The event length is impossibly large
507
+ nd_log(
508
+ NDLS_COLLECTORS,
509
+ NDLP_ERR,
510
+ "JOURNAL WATCHER: received impossibly large event length %u - restarting",
511
+ event->len);
512
+ return true; // force a restart
513
+ }
514
+
515
+ // Check if we have the complete event including the name
516
+ ssize_t total_size = (ssize_t)sizeof(struct inotify_event) + event->len;
517
+ if (processed + total_size > reader->read_len)
518
+ break; // Wait for more data
519
+
520
+ if (event->mask & IN_UNMOUNT) {
521
+ unmount_event = true;
522
+ break;
523
+ }
524
+
525
+ process_event(watcher, inotifyFd, event);
526
+ processed += total_size;
527
+ }
528
+
529
+ // If we have unprocessed data, move it to the start
530
+ if (processed < reader->read_len) {
531
+ memmove(reader->read_buffer, reader->read_buffer + processed, reader->read_len - processed);
532
+ reader->read_len -= processed;
533
+ } else
534
+ reader->read_len = 0;
535
+
536
+ reader->read_buffer[reader->read_len] = '\0';
537
+ return unmount_event;
538
+}
539
+
540
+void nd_journal_watcher_main(void *arg __maybe_unused)
541
+{
542
+ while (1) {
543
+ size_t journal_watcher_session_id = __atomic_load_n(&journal_watcher_wanted_session_id, __ATOMIC_RELAXED);
544
+
545
+ Watcher watcher = {
546
+ .watchList = mallocz(INITIAL_WATCHES * sizeof(WatchEntry)),
547
+ .freeList = NULL,
548
+ .watchCount = 0,
549
+ .watchListSize = INITIAL_WATCHES,
550
+ .pending = dictionary_create(DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_SINGLE_THREADED),
551
+ .errors = 0,
552
+ };
553
+
554
+ int inotifyFd = inotify_init();
555
+ if (inotifyFd < 0) {
556
+ nd_log(NDLS_COLLECTORS, NDLP_ERR, "inotify_init() failed.");
557
+ free_watches(&watcher, inotifyFd);
558
+ return;
559
+ }
560
+
561
+ for (unsigned i = 0; i < MAX_JOURNAL_DIRECTORIES; i++) {
562
+ if (!journal_directories[i].path)
563
+ break;
564
+ watch_directory_and_subdirectories(&watcher, inotifyFd, string2str(journal_directories[i].path));
565
+ }
566
+
567
+ usec_t last_headers_update_ut = now_monotonic_usec();
568
+ struct buffered_reader reader;
569
+ buffered_reader_init(&reader);
570
+
571
+ while (journal_watcher_session_id == __atomic_load_n(&journal_watcher_wanted_session_id, __ATOMIC_RELAXED)) {
572
+ buffered_reader_ret_t rc =
573
+ buffered_reader_read_timeout(&reader, inotifyFd, ND_SD_JOURNAL_EXECUTE_WATCHER_PENDING_EVERY_MS, false);
574
+
575
+ if (rc == BUFFERED_READER_READ_OK || rc == BUFFERED_READER_READ_BUFFER_FULL) {
576
+ if (process_inotify_events(&reader, &watcher, inotifyFd))
577
+ break;
578
+ } else if (rc != BUFFERED_READER_READ_POLL_TIMEOUT) {
579
+ nd_log(
580
+ NDLS_COLLECTORS,
581
+ NDLP_ERR,
582
+ "JOURNAL WATCHER: cannot read inotify events, buffered_reader_read_timeout() returned %d - "
583
+ "restarting the watcher.",
584
+ rc);
585
+ break;
586
+ }
587
+
588
+ usec_t ut = now_monotonic_usec();
589
+ if (dictionary_entries(watcher.pending) &&
590
+ (rc == BUFFERED_READER_READ_POLL_TIMEOUT ||
591
+ last_headers_update_ut + (ND_SD_JOURNAL_EXECUTE_WATCHER_PENDING_EVERY_MS * USEC_PER_MS) <= ut)) {
592
+ process_pending(&watcher);
593
+ last_headers_update_ut = ut;
594
+ }
595
+
596
+ if (watcher.errors) {
597
+ nd_log(
598
+ NDLS_COLLECTORS,
599
+ NDLP_NOTICE,
600
+ "JOURNAL WATCHER: there were errors in setting up inotify watches - restarting the watcher.");
601
+ }
602
+ }
603
+
604
+ close(inotifyFd);
605
+ free_watches(&watcher, inotifyFd);
606
+ free_symlinked_dirs(&watcher);
607
+
608
+ // this will scan the directories and cleanup the registry
609
+ nd_journal_files_registry_update();
610
+
611
+ sleep_usec(2 * USEC_PER_SEC);
612
+ }
613
+}
src/collectors/systemd-journal.plugin/systemd-journal.c
new
+1179
@@ -0,0 +1,1179 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+/*
4
+ * TODO
5
+ * _UDEV_DEVLINK is frequently set more than once per field - support multi-value faces
6
+ *
7
+ */
8
+
9
+#include "collectors/systemd-journal.plugin/provider/netdata_provider.h"
10
+#include "systemd-internals.h"
11
+
12
+#define ND_SD_JOURNAL_FUNCTION_DESCRIPTION "View, search and analyze systemd journal entries."
13
+#define ND_SD_JOURNAL_FUNCTION_NAME "systemd-journal"
14
+#define ND_SD_JOURNAL_SAMPLING_SLOTS 1000
15
+#define ND_SD_JOURNAL_SAMPLING_RECALIBRATE 10000
16
+
17
+#ifdef HAVE_SD_JOURNAL_RESTART_FIELDS
18
+#define LQS_DEFAULT_SLICE_MODE 1
19
+#else
20
+#define LQS_DEFAULT_SLICE_MODE 0
21
+#endif
22
+
23
+// functions needed by LQS
24
+static SD_JOURNAL_FILE_SOURCE_TYPE get_internal_source_type(const char *value);
25
+
26
+// structures needed by LQS
27
+struct lqs_extension {
28
+ struct {
29
+ usec_t start_ut;
30
+ usec_t stop_ut;
31
+ usec_t first_msg_ut;
32
+
33
+ NsdId128 first_msg_writer;
34
+ uint64_t first_msg_seqnum;
35
+ } query_file;
36
+
37
+ struct {
38
+ uint32_t enable_after_samples;
39
+ uint32_t slots;
40
+ uint32_t sampled;
41
+ uint32_t unsampled;
42
+ uint32_t estimated;
43
+ } samples;
44
+
45
+ struct {
46
+ uint32_t enable_after_samples;
47
+ uint32_t every;
48
+ uint32_t skipped;
49
+ uint32_t recalibrate;
50
+ uint32_t sampled;
51
+ uint32_t unsampled;
52
+ uint32_t estimated;
53
+ } samples_per_file;
54
+
55
+ struct {
56
+ usec_t start_ut;
57
+ usec_t end_ut;
58
+ usec_t step_ut;
59
+ uint32_t enable_after_samples;
60
+ uint32_t sampled[ND_SD_JOURNAL_SAMPLING_SLOTS];
61
+ uint32_t unsampled[ND_SD_JOURNAL_SAMPLING_SLOTS];
62
+ } samples_per_time_slot;
63
+
64
+ // per file progress info
65
+ // size_t cached_count;
66
+
67
+ // progress statistics
68
+ usec_t matches_setup_ut;
69
+ size_t rows_useful;
70
+ size_t rows_read;
71
+ size_t bytes_read;
72
+ size_t files_matched;
73
+ size_t file_working;
74
+};
75
+
76
+// prepare LQS
77
+#define LQS_FUNCTION_NAME ND_SD_JOURNAL_FUNCTION_NAME
78
+#define LQS_FUNCTION_DESCRIPTION ND_SD_JOURNAL_FUNCTION_DESCRIPTION
79
+#define LQS_DEFAULT_ITEMS_PER_QUERY 200
80
+#define LQS_DEFAULT_ITEMS_SAMPLING 1000000
81
+#define LQS_SOURCE_TYPE SD_JOURNAL_FILE_SOURCE_TYPE
82
+#define LQS_SOURCE_TYPE_ALL ND_SD_JF_ALL
83
+#define LQS_SOURCE_TYPE_NONE ND_SD_JF_NONE
84
+#define LQS_PARAMETER_SOURCE_NAME "Journal Sources" // this is how it is shown to users
85
+#define LQS_FUNCTION_GET_INTERNAL_SOURCE_TYPE(value) get_internal_source_type(value)
86
+#define LQS_FUNCTION_SOURCE_TO_JSON_ARRAY(wb) available_journal_file_sources_to_json_array(wb)
87
+#include "libnetdata/facets/logs_query_status.h"
88
+
89
+#include "systemd-journal-sampling.h"
90
+
91
+#define FACET_MAX_VALUE_LENGTH 8192
92
+#define ND_SD_JOURNAL_DEFAULT_TIMEOUT 60
93
+#define ND_SD_JOURNAL_PROGRESS_EVERY_UT (250 * USEC_PER_MS)
94
+#define JOURNAL_KEY_ND_JOURNAL_FILE "ND_JOURNAL_FILE"
95
+#define JOURNAL_KEY_ND_JOURNAL_PROCESS "ND_JOURNAL_PROCESS"
96
+#define JOURNAL_DEFAULT_DIRECTION FACETS_ANCHOR_DIRECTION_BACKWARD
97
+#define SYSTEMD_ALWAYS_VISIBLE_KEYS NULL
98
+
99
+#define SYSTEMD_KEYS_EXCLUDED_FROM_FACETS \
100
+ "!MESSAGE_ID" \
101
+ "|*MESSAGE*" \
102
+ "|*TIMESTAMP*" \
103
+ "|__*" \
104
+ ""
105
+
106
+#define SYSTEMD_KEYS_INCLUDED_IN_FACETS \
107
+ \
108
+ /* --- USER JOURNAL FIELDS --- */ \
109
+ \
110
+ /* "|MESSAGE" */ \
111
+ "|MESSAGE_ID" \
112
+ "|PRIORITY" \
113
+ "|CODE_FILE" /* "|CODE_LINE" */ \
114
+ "|CODE_FUNC" \
115
+ "|ERRNO" /* "|INVOCATION_ID" */ /* "|USER_INVOCATION_ID" */ \
116
+ "|SYSLOG_FACILITY" \
117
+ "|SYSLOG_IDENTIFIER" /* "|SYSLOG_PID" */ /* "|SYSLOG_TIMESTAMP" */ /* "|SYSLOG_RAW" */ /* "!DOCUMENTATION" */ /* "|TID" */ \
118
+ "|UNIT" \
119
+ "|USER_UNIT" \
120
+ "|UNIT_RESULT" /* undocumented */ \
121
+ \
122
+ /* --- TRUSTED JOURNAL FIELDS --- */ \
123
+ \
124
+ /* "|_PID" */ \
125
+ "|_UID" \
126
+ "|_GID" \
127
+ "|_COMM" \
128
+ "|_EXE" /* "|_CMDLINE" */ \
129
+ "|_CAP_EFFECTIVE" /* "|_AUDIT_SESSION" */ \
130
+ "|_AUDIT_LOGINUID" \
131
+ "|_SYSTEMD_CGROUP" \
132
+ "|_SYSTEMD_SLICE" \
133
+ "|_SYSTEMD_UNIT" \
134
+ "|_SYSTEMD_USER_UNIT" \
135
+ "|_SYSTEMD_USER_SLICE" \
136
+ "|_SYSTEMD_SESSION" \
137
+ "|_SYSTEMD_OWNER_UID" \
138
+ "|_SELINUX_CONTEXT" /* "|_SOURCE_REALTIME_TIMESTAMP" */ \
139
+ "|_BOOT_ID" \
140
+ "|_MACHINE_ID" /* "|_SYSTEMD_INVOCATION_ID" */ \
141
+ "|_HOSTNAME" \
142
+ "|_TRANSPORT" \
143
+ "|_STREAM_ID" /* "|LINE_BREAK" */ \
144
+ "|_NAMESPACE" \
145
+ "|_RUNTIME_SCOPE" \
146
+ \
147
+ /* --- KERNEL JOURNAL FIELDS --- */ \
148
+ \
149
+ /* "|_KERNEL_DEVICE" */ \
150
+ "|_KERNEL_SUBSYSTEM" /* "|_UDEV_SYSNAME" */ \
151
+ "|_UDEV_DEVNODE" /* "|_UDEV_DEVLINK" */ \
152
+ \
153
+ /* --- LOGGING ON BEHALF --- */ \
154
+ \
155
+ "|OBJECT_UID" \
156
+ "|OBJECT_GID" \
157
+ "|OBJECT_COMM" \
158
+ "|OBJECT_EXE" /* "|OBJECT_CMDLINE" */ /* "|OBJECT_AUDIT_SESSION" */ \
159
+ "|OBJECT_AUDIT_LOGINUID" \
160
+ "|OBJECT_SYSTEMD_CGROUP" \
161
+ "|OBJECT_SYSTEMD_SESSION" \
162
+ "|OBJECT_SYSTEMD_OWNER_UID" \
163
+ "|OBJECT_SYSTEMD_UNIT" \
164
+ "|OBJECT_SYSTEMD_USER_UNIT" \
165
+ \
166
+ /* --- CORE DUMPS --- */ \
167
+ \
168
+ "|COREDUMP_COMM" \
169
+ "|COREDUMP_UNIT" \
170
+ "|COREDUMP_USER_UNIT" \
171
+ "|COREDUMP_SIGNAL_NAME" \
172
+ "|COREDUMP_CGROUP" \
173
+ \
174
+ /* --- DOCKER --- */ \
175
+ \
176
+ "|CONTAINER_ID" /* "|CONTAINER_ID_FULL" */ \
177
+ "|CONTAINER_NAME" \
178
+ "|CONTAINER_TAG" \
179
+ "|IMAGE_NAME" /* undocumented */ /* "|CONTAINER_PARTIAL_MESSAGE" */ \
180
+ \
181
+ /* --- NETDATA --- */ \
182
+ \
183
+ "|ND_NIDL_NODE" \
184
+ "|ND_NIDL_CONTEXT" \
185
+ "|ND_LOG_SOURCE" /*"|ND_MODULE" */ \
186
+ "|ND_ALERT_NAME" \
187
+ "|ND_ALERT_CLASS" \
188
+ "|ND_ALERT_COMPONENT" \
189
+ "|ND_ALERT_TYPE" \
190
+ "|ND_ALERT_STATUS" \
191
+ \
192
+ ""
193
+
194
+static SD_JOURNAL_FILE_SOURCE_TYPE get_internal_source_type(const char *value)
195
+{
196
+ if (strcmp(value, ND_SD_JF_SOURCE_ALL_NAME) == 0)
197
+ return ND_SD_JF_ALL;
198
+ else if (strcmp(value, ND_SD_JF_SOURCE_LOCAL_NAME) == 0)
199
+ return ND_SD_JF_LOCAL_ALL;
200
+ else if (strcmp(value, ND_SD_JF_SOURCE_REMOTES_NAME) == 0)
201
+ return ND_SD_JF_REMOTE_ALL;
202
+ else if (strcmp(value, ND_SD_JF_SOURCE_NAMESPACES_NAME) == 0)
203
+ return ND_SD_JF_LOCAL_NAMESPACE;
204
+ else if (strcmp(value, ND_SD_JF_SOURCE_LOCAL_SYSTEM_NAME) == 0)
205
+ return ND_SD_JF_LOCAL_SYSTEM;
206
+ else if (strcmp(value, ND_SD_JF_SOURCE_LOCAL_USERS_NAME) == 0)
207
+ return ND_SD_JF_LOCAL_USER;
208
+ else if (strcmp(value, ND_SD_JF_SOURCE_LOCAL_OTHER_NAME) == 0)
209
+ return ND_SD_JF_LOCAL_OTHER;
210
+
211
+ return ND_SD_JF_NONE;
212
+}
213
+
214
+static inline bool nd_sd_journal_seek_to(NsdJournal *j, usec_t timestamp)
215
+{
216
+ if (nsd_journal_seek_realtime_usec(j, timestamp) < 0) {
217
+ netdata_log_error("SYSTEMD-JOURNAL: Failed to seek to %" PRIu64, timestamp);
218
+ if (nsd_journal_seek_tail(j) < 0) {
219
+ netdata_log_error("SYSTEMD-JOURNAL: Failed to seek to journal's tail");
220
+ return false;
221
+ }
222
+ }
223
+
224
+ return true;
225
+}
226
+
227
+#define JD_SOURCE_REALTIME_TIMESTAMP "_SOURCE_REALTIME_TIMESTAMP"
228
+
229
+static inline size_t
230
+nd_sd_journal_process_row(NsdJournal *j, FACETS *facets, struct nd_journal_file *njf, usec_t *msg_ut)
231
+{
232
+ const void *data;
233
+ size_t length, bytes = 0;
234
+
235
+ facets_add_key_value_length(
236
+ facets, JOURNAL_KEY_ND_JOURNAL_FILE, sizeof(JOURNAL_KEY_ND_JOURNAL_FILE) - 1, njf->filename, njf->filename_len);
237
+
238
+ NSD_JOURNAL_FOREACH_DATA(j, data, length)
239
+ {
240
+ const char *key, *value;
241
+ size_t key_length, value_length;
242
+
243
+ if (!parse_journal_field(data, length, &key, &key_length, &value, &value_length))
244
+ continue;
245
+
246
+#ifdef NETDATA_INTERNAL_CHECKS
247
+ usec_t origin_journal_ut = *msg_ut;
248
+#endif
249
+ if (unlikely(
250
+ key_length == sizeof(JD_SOURCE_REALTIME_TIMESTAMP) - 1 &&
251
+ memcmp(key, JD_SOURCE_REALTIME_TIMESTAMP, sizeof(JD_SOURCE_REALTIME_TIMESTAMP) - 1) == 0)) {
252
+ usec_t ut = str2ull(value, NULL);
253
+ if (ut && ut < *msg_ut) {
254
+ usec_t delta = *msg_ut - ut;
255
+ *msg_ut = ut;
256
+
257
+ if (delta > JOURNAL_VS_REALTIME_DELTA_MAX_UT)
258
+ delta = JOURNAL_VS_REALTIME_DELTA_MAX_UT;
259
+
260
+ // update max_journal_vs_realtime_delta_ut if the delta increased
261
+ usec_t expected = njf->max_journal_vs_realtime_delta_ut;
262
+ do {
263
+ if (delta <= expected)
264
+ break;
265
+ } while (!__atomic_compare_exchange_n(
266
+ &njf->max_journal_vs_realtime_delta_ut, &expected, delta, false, __ATOMIC_RELAXED, __ATOMIC_RELAXED));
267
+
268
+ internal_error(
269
+ delta > expected,
270
+ "increased max_journal_vs_realtime_delta_ut from %" PRIu64 " to %" PRIu64 ", "
271
+ "journal %" PRIu64 ", actual %" PRIu64 " (delta %" PRIu64 ")",
272
+ expected,
273
+ delta,
274
+ origin_journal_ut,
275
+ *msg_ut,
276
+ origin_journal_ut - (*msg_ut));
277
+ }
278
+ }
279
+
280
+ bytes += length;
281
+ facets_add_key_value_length(
282
+ facets,
283
+ key,
284
+ key_length,
285
+ value,
286
+ value_length <= FACET_MAX_VALUE_LENGTH ? value_length : FACET_MAX_VALUE_LENGTH);
287
+ }
288
+
289
+ return bytes;
290
+}
291
+
292
+#define FUNCTION_PROGRESS_UPDATE_ROWS(rows_read, rows) __atomic_fetch_add(&(rows_read), rows, __ATOMIC_RELAXED)
293
+#define FUNCTION_PROGRESS_UPDATE_BYTES(bytes_read, bytes) __atomic_fetch_add(&(bytes_read), bytes, __ATOMIC_RELAXED)
294
+#define FUNCTION_PROGRESS_EVERY_ROWS (1ULL << 13)
295
+#define FUNCTION_DATA_ONLY_CHECK_EVERY_ROWS (1ULL << 7)
296
+
297
+static inline ND_SD_JOURNAL_STATUS check_stop(const bool *cancelled, const usec_t *stop_monotonic_ut)
298
+{
299
+ if (cancelled && __atomic_load_n(cancelled, __ATOMIC_RELAXED)) {
300
+ internal_error(true, "Function has been cancelled");
301
+ return ND_SD_JOURNAL_CANCELLED;
302
+ }
303
+
304
+ if (now_monotonic_usec() > __atomic_load_n(stop_monotonic_ut, __ATOMIC_RELAXED)) {
305
+ internal_error(true, "Function timed out");
306
+ return ND_SD_JOURNAL_TIMED_OUT;
307
+ }
308
+
309
+ return ND_SD_JOURNAL_OK;
310
+}
311
+
312
+ND_SD_JOURNAL_STATUS nd_sd_journal_query_backward(
313
+ NsdJournal *j,
314
+ BUFFER *wb __maybe_unused,
315
+ FACETS *facets,
316
+ struct nd_journal_file *njf,
317
+ LOGS_QUERY_STATUS *fqs)
318
+{
319
+ usec_t anchor_delta = __atomic_load_n(&njf->max_journal_vs_realtime_delta_ut, __ATOMIC_RELAXED);
320
+ lqs_query_timeframe(fqs, anchor_delta);
321
+ usec_t start_ut = fqs->query.start_ut;
322
+ usec_t stop_ut = fqs->query.stop_ut;
323
+ bool stop_when_full = fqs->query.stop_when_full;
324
+
325
+ fqs->c.query_file.start_ut = start_ut;
326
+ fqs->c.query_file.stop_ut = stop_ut;
327
+
328
+ if (!nd_sd_journal_seek_to(j, start_ut))
329
+ return ND_SD_JOURNAL_FAILED_TO_SEEK;
330
+
331
+ size_t errors_no_timestamp = 0;
332
+ usec_t latest_msg_ut = 0; // the biggest timestamp we have seen so far
333
+ usec_t first_msg_ut = 0; // the first message we got from the db
334
+ size_t row_counter = 0, last_row_counter = 0, rows_useful = 0;
335
+ size_t bytes = 0, last_bytes = 0;
336
+
337
+ usec_t last_usec_from = 0;
338
+ usec_t last_usec_to = 0;
339
+
340
+ ND_SD_JOURNAL_STATUS status = ND_SD_JOURNAL_OK;
341
+
342
+ facets_rows_begin(facets);
343
+ while (status == ND_SD_JOURNAL_OK && nsd_journal_previous(j) > 0) {
344
+ usec_t msg_ut = 0;
345
+ if (nsd_journal_get_realtime_usec(j, &msg_ut) < 0 || !msg_ut) {
346
+ errors_no_timestamp++;
347
+ continue;
348
+ }
349
+
350
+ if (unlikely(msg_ut > start_ut))
351
+ continue;
352
+
353
+ if (unlikely(msg_ut < stop_ut))
354
+ break;
355
+
356
+ if (unlikely(msg_ut > latest_msg_ut))
357
+ latest_msg_ut = msg_ut;
358
+
359
+ if (unlikely(!first_msg_ut)) {
360
+ first_msg_ut = msg_ut;
361
+ fqs->c.query_file.first_msg_ut = msg_ut;
362
+
363
+#ifdef HAVE_SD_JOURNAL_GET_SEQNUM
364
+ if (nsd_journal_get_seqnum(j, &fqs->c.query_file.first_msg_seqnum, &fqs->c.query_file.first_msg_writer) <
365
+ 0) {
366
+ fqs->c.query_file.first_msg_seqnum = 0;
367
+ fqs->c.query_file.first_msg_writer = NSD_ID128_NULL;
368
+ }
369
+#endif
370
+ }
371
+
372
+ sampling_t sample = is_row_in_sample(
373
+ j, fqs, njf, msg_ut, FACETS_ANCHOR_DIRECTION_BACKWARD, facets_row_candidate_to_keep(facets, msg_ut));
374
+
375
+ if (sample == SAMPLING_FULL) {
376
+ bytes += nd_sd_journal_process_row(j, facets, njf, &msg_ut);
377
+
378
+ // make sure each line gets a unique timestamp
379
+ if (unlikely(msg_ut >= last_usec_from && msg_ut <= last_usec_to))
380
+ msg_ut = --last_usec_from;
381
+ else
382
+ last_usec_from = last_usec_to = msg_ut;
383
+
384
+ if (facets_row_finished(facets, msg_ut))
385
+ rows_useful++;
386
+
387
+ row_counter++;
388
+ if (unlikely(
389
+ (row_counter % FUNCTION_DATA_ONLY_CHECK_EVERY_ROWS) == 0 && stop_when_full &&
390
+ facets_rows(facets) >= fqs->rq.entries)) {
391
+ // stop the data only query
392
+ usec_t oldest = facets_row_oldest_ut(facets);
393
+ if (oldest && msg_ut < (oldest - anchor_delta))
394
+ break;
395
+ }
396
+
397
+ if (unlikely(row_counter % FUNCTION_PROGRESS_EVERY_ROWS == 0)) {
398
+ FUNCTION_PROGRESS_UPDATE_ROWS(fqs->c.rows_read, row_counter - last_row_counter);
399
+ last_row_counter = row_counter;
400
+
401
+ FUNCTION_PROGRESS_UPDATE_BYTES(fqs->c.bytes_read, bytes - last_bytes);
402
+ last_bytes = bytes;
403
+
404
+ status = check_stop(fqs->cancelled, fqs->stop_monotonic_ut);
405
+ }
406
+ } else if (sample == SAMPLING_SKIP_FIELDS)
407
+ facets_row_finished_unsampled(facets, msg_ut);
408
+ else {
409
+ sampling_update_running_query_file_estimates(facets, j, fqs, njf, msg_ut, FACETS_ANCHOR_DIRECTION_BACKWARD);
410
+ break;
411
+ }
412
+ }
413
+
414
+ FUNCTION_PROGRESS_UPDATE_ROWS(fqs->c.rows_read, row_counter - last_row_counter);
415
+ FUNCTION_PROGRESS_UPDATE_BYTES(fqs->c.bytes_read, bytes - last_bytes);
416
+
417
+ fqs->c.rows_useful += rows_useful;
418
+
419
+ if (errors_no_timestamp)
420
+ netdata_log_error("SYSTEMD-JOURNAL: %zu lines did not have timestamps", errors_no_timestamp);
421
+
422
+ if (latest_msg_ut > fqs->last_modified)
423
+ fqs->last_modified = latest_msg_ut;
424
+
425
+ return status;
426
+}
427
+
428
+ND_SD_JOURNAL_STATUS nd_sd_journal_query_forward(
429
+ NsdJournal *j,
430
+ BUFFER *wb __maybe_unused,
431
+ FACETS *facets,
432
+ struct nd_journal_file *njf,
433
+ LOGS_QUERY_STATUS *fqs)
434
+{
435
+ usec_t anchor_delta = __atomic_load_n(&njf->max_journal_vs_realtime_delta_ut, __ATOMIC_RELAXED);
436
+ lqs_query_timeframe(fqs, anchor_delta);
437
+ usec_t start_ut = fqs->query.start_ut;
438
+ usec_t stop_ut = fqs->query.stop_ut;
439
+ bool stop_when_full = fqs->query.stop_when_full;
440
+
441
+ fqs->c.query_file.start_ut = start_ut;
442
+ fqs->c.query_file.stop_ut = stop_ut;
443
+
444
+ if (!nd_sd_journal_seek_to(j, start_ut))
445
+ return ND_SD_JOURNAL_FAILED_TO_SEEK;
446
+
447
+ size_t errors_no_timestamp = 0;
448
+ usec_t latest_msg_ut = 0; // the biggest timestamp we have seen so far
449
+ usec_t first_msg_ut = 0; // the first message we got from the db
450
+ size_t row_counter = 0, last_row_counter = 0, rows_useful = 0;
451
+ size_t bytes = 0, last_bytes = 0;
452
+
453
+ usec_t last_usec_from = 0;
454
+ usec_t last_usec_to = 0;
455
+
456
+ ND_SD_JOURNAL_STATUS status = ND_SD_JOURNAL_OK;
457
+
458
+ facets_rows_begin(facets);
459
+ while (status == ND_SD_JOURNAL_OK && nsd_journal_next(j) > 0) {
460
+ usec_t msg_ut = 0;
461
+ if (nsd_journal_get_realtime_usec(j, &msg_ut) < 0 || !msg_ut) {
462
+ errors_no_timestamp++;
463
+ continue;
464
+ }
465
+
466
+ if (unlikely(msg_ut < start_ut))
467
+ continue;
468
+
469
+ if (unlikely(msg_ut > stop_ut))
470
+ break;
471
+
472
+ if (likely(msg_ut > latest_msg_ut))
473
+ latest_msg_ut = msg_ut;
474
+
475
+ if (unlikely(!first_msg_ut)) {
476
+ first_msg_ut = msg_ut;
477
+ fqs->c.query_file.first_msg_ut = msg_ut;
478
+ }
479
+
480
+ sampling_t sample = is_row_in_sample(
481
+ j, fqs, njf, msg_ut, FACETS_ANCHOR_DIRECTION_FORWARD, facets_row_candidate_to_keep(facets, msg_ut));
482
+
483
+ if (sample == SAMPLING_FULL) {
484
+ bytes += nd_sd_journal_process_row(j, facets, njf, &msg_ut);
485
+
486
+ // make sure each line gets a unique timestamp
487
+ if (unlikely(msg_ut >= last_usec_from && msg_ut <= last_usec_to))
488
+ msg_ut = ++last_usec_to;
489
+ else
490
+ last_usec_from = last_usec_to = msg_ut;
491
+
492
+ if (facets_row_finished(facets, msg_ut))
493
+ rows_useful++;
494
+
495
+ row_counter++;
496
+ if (unlikely(
497
+ (row_counter % FUNCTION_DATA_ONLY_CHECK_EVERY_ROWS) == 0 && stop_when_full &&
498
+ facets_rows(facets) >= fqs->rq.entries)) {
499
+ // stop the data only query
500
+ usec_t newest = facets_row_newest_ut(facets);
501
+ if (newest && msg_ut > (newest + anchor_delta))
502
+ break;
503
+ }
504
+
505
+ if (unlikely(row_counter % FUNCTION_PROGRESS_EVERY_ROWS == 0)) {
506
+ FUNCTION_PROGRESS_UPDATE_ROWS(fqs->c.rows_read, row_counter - last_row_counter);
507
+ last_row_counter = row_counter;
508
+
509
+ FUNCTION_PROGRESS_UPDATE_BYTES(fqs->c.bytes_read, bytes - last_bytes);
510
+ last_bytes = bytes;
511
+
512
+ status = check_stop(fqs->cancelled, fqs->stop_monotonic_ut);
513
+ }
514
+ } else if (sample == SAMPLING_SKIP_FIELDS)
515
+ facets_row_finished_unsampled(facets, msg_ut);
516
+ else {
517
+ sampling_update_running_query_file_estimates(facets, j, fqs, njf, msg_ut, FACETS_ANCHOR_DIRECTION_FORWARD);
518
+ break;
519
+ }
520
+ }
521
+
522
+ FUNCTION_PROGRESS_UPDATE_ROWS(fqs->c.rows_read, row_counter - last_row_counter);
523
+ FUNCTION_PROGRESS_UPDATE_BYTES(fqs->c.bytes_read, bytes - last_bytes);
524
+
525
+ fqs->c.rows_useful += rows_useful;
526
+
527
+ if (errors_no_timestamp)
528
+ netdata_log_error("SYSTEMD-JOURNAL: %zu lines did not have timestamps", errors_no_timestamp);
529
+
530
+ if (latest_msg_ut > fqs->last_modified)
531
+ fqs->last_modified = latest_msg_ut;
532
+
533
+ return status;
534
+}
535
+
536
+bool nd_sd_journal_check_if_modified_since(NsdJournal *j, usec_t seek_to, usec_t last_modified)
537
+{
538
+ // return true, if data have been modified since the timestamp
539
+
540
+ if (!last_modified || !seek_to)
541
+ return false;
542
+
543
+ if (!nd_sd_journal_seek_to(j, seek_to))
544
+ return false;
545
+
546
+ usec_t first_msg_ut = 0;
547
+ while (nsd_journal_previous(j) > 0) {
548
+ usec_t msg_ut;
549
+ if (nsd_journal_get_realtime_usec(j, &msg_ut) < 0)
550
+ continue;
551
+
552
+ first_msg_ut = msg_ut;
553
+ break;
554
+ }
555
+
556
+ return first_msg_ut != last_modified;
557
+}
558
+
559
+#ifdef HAVE_SD_JOURNAL_RESTART_FIELDS
560
+static bool netdata_systemd_filtering_by_journal(NsdJournal *j, FACETS *facets, LOGS_QUERY_STATUS *lqs)
561
+{
562
+ const char *field = NULL;
563
+ const void *data = NULL;
564
+ size_t data_length;
565
+ size_t added_keys = 0;
566
+ size_t failures = 0;
567
+ size_t filters_added = 0;
568
+
569
+ NSD_JOURNAL_FOREACH_FIELD(j, field)
570
+ { // for each key
571
+ bool interesting;
572
+
573
+ if (lqs->rq.data_only)
574
+ interesting = facets_key_name_is_filter(facets, field);
575
+ else
576
+ interesting = facets_key_name_is_facet(facets, field);
577
+
578
+ if (interesting) {
579
+ if (nsd_journal_query_unique(j, field) >= 0) {
580
+ bool added_this_key = false;
581
+ size_t added_values = 0;
582
+
583
+ NSD_JOURNAL_FOREACH_UNIQUE(j, data, data_length)
584
+ { // for each value of the key
585
+ const char *key, *value;
586
+ size_t key_length, value_length;
587
+
588
+ if (!parse_journal_field(data, data_length, &key, &key_length, &value, &value_length))
589
+ continue;
590
+
591
+ facets_add_possible_value_name_to_key(facets, key, key_length, value, value_length);
592
+
593
+ if (!facets_key_name_value_length_is_selected(facets, key, key_length, value, value_length))
594
+ continue;
595
+
596
+ if (added_keys && !added_this_key) {
597
+ if (nsd_journal_add_conjunction(j) < 0) // key AND key AND key
598
+ failures++;
599
+
600
+ added_this_key = true;
601
+ added_keys++;
602
+ } else if (added_values)
603
+ if (nsd_journal_add_disjunction(j) < 0) // value OR value OR value
604
+ failures++;
605
+
606
+ if (nsd_journal_add_match(j, data, data_length) < 0)
607
+ failures++;
608
+
609
+ if (!added_keys) {
610
+ added_keys++;
611
+ added_this_key = true;
612
+ }
613
+
614
+ added_values++;
615
+ filters_added++;
616
+ }
617
+ }
618
+ }
619
+ }
620
+
621
+ if (failures) {
622
+ lqs_log_error(lqs, "failed to setup journal filter, will run the full query.");
623
+ nsd_journal_flush_matches(j);
624
+ return true;
625
+ }
626
+
627
+ return filters_added ? true : false;
628
+}
629
+#endif // HAVE_SD_JOURNAL_RESTART_FIELDS
630
+
631
+static ND_SD_JOURNAL_STATUS nd_sd_journal_query_one_file(
632
+ const char *filename,
633
+ BUFFER *wb,
634
+ FACETS *facets,
635
+ struct nd_journal_file *njf,
636
+ LOGS_QUERY_STATUS *fqs)
637
+{
638
+ NsdJournal *j = NULL;
639
+ errno_clear();
640
+
641
+ fstat_cache_enable_on_thread();
642
+
643
+ const char *paths[2] = {
644
+ [0] = filename,
645
+ [1] = NULL,
646
+ };
647
+
648
+ if (nsd_journal_open_files(&j, paths, ND_SD_JOURNAL_OPEN_FLAGS) < 0 || !j) {
649
+ netdata_log_error("JOURNAL: cannot open file '%s' for query", filename);
650
+ fstat_cache_disable_on_thread();
651
+ return ND_SD_JOURNAL_FAILED_TO_OPEN;
652
+ }
653
+
654
+ ND_SD_JOURNAL_STATUS status;
655
+ bool matches_filters = true;
656
+
657
+#ifdef HAVE_SD_JOURNAL_RESTART_FIELDS
658
+ if (fqs->rq.slice) {
659
+ usec_t started = now_monotonic_usec();
660
+
661
+ matches_filters = netdata_systemd_filtering_by_journal(j, facets, fqs) || !fqs->rq.filters;
662
+ usec_t ended = now_monotonic_usec();
663
+
664
+ fqs->c.matches_setup_ut += (ended - started);
665
+ }
666
+#endif // HAVE_SD_JOURNAL_RESTART_FIELDS
667
+
668
+ if (matches_filters) {
669
+ if (fqs->rq.direction == FACETS_ANCHOR_DIRECTION_FORWARD)
670
+ status = nd_sd_journal_query_forward(j, wb, facets, njf, fqs);
671
+ else
672
+ status = nd_sd_journal_query_backward(j, wb, facets, njf, fqs);
673
+ } else
674
+ status = ND_SD_JOURNAL_NO_FILE_MATCHED;
675
+
676
+ nsd_journal_close(j);
677
+ fstat_cache_disable_on_thread();
678
+
679
+ return status;
680
+}
681
+
682
+static bool jf_is_mine(struct nd_journal_file *njf, LOGS_QUERY_STATUS *fqs)
683
+{
684
+ if ((fqs->rq.source_type == ND_SD_JF_NONE && !fqs->rq.sources) || (njf->source_type & fqs->rq.source_type) ||
685
+ (fqs->rq.sources && simple_pattern_matches(fqs->rq.sources, string2str(njf->source)))) {
686
+ if (!njf->msg_last_ut)
687
+ // the file is not scanned yet, or the timestamps have not been updated,
688
+ // so we don't know if it can contribute or not - let's add it.
689
+ return true;
690
+
691
+ usec_t anchor_delta = JOURNAL_VS_REALTIME_DELTA_MAX_UT;
692
+ usec_t first_ut = njf->msg_first_ut - anchor_delta;
693
+ usec_t last_ut = njf->msg_last_ut + anchor_delta;
694
+
695
+ if (last_ut >= fqs->rq.after_ut && first_ut <= fqs->rq.before_ut)
696
+ return true;
697
+ }
698
+
699
+ return false;
700
+}
701
+
702
+static int nd_sd_journal_query(BUFFER *wb, LOGS_QUERY_STATUS *lqs)
703
+{
704
+ FACETS *facets = lqs->facets;
705
+
706
+ ND_SD_JOURNAL_STATUS status = ND_SD_JOURNAL_NO_FILE_MATCHED;
707
+ struct nd_journal_file *njf;
708
+
709
+ lqs->c.files_matched = 0;
710
+ lqs->c.file_working = 0;
711
+ lqs->c.rows_useful = 0;
712
+ lqs->c.rows_read = 0;
713
+ lqs->c.bytes_read = 0;
714
+
715
+ size_t files_used = 0;
716
+ size_t files_max = dictionary_entries(nd_journal_files_registry);
717
+ const DICTIONARY_ITEM *file_items[files_max];
718
+
719
+ // count the files
720
+ bool files_are_newer = false;
721
+ dfe_start_read(nd_journal_files_registry, njf)
722
+ {
723
+ if (!jf_is_mine(njf, lqs))
724
+ continue;
725
+
726
+ file_items[files_used++] = dictionary_acquired_item_dup(nd_journal_files_registry, njf_dfe.item);
727
+
728
+ if (njf->msg_last_ut > lqs->rq.if_modified_since)
729
+ files_are_newer = true;
730
+ }
731
+ dfe_done(jf);
732
+
733
+ lqs->c.files_matched = files_used;
734
+
735
+ if (lqs->rq.if_modified_since && !files_are_newer) {
736
+ // release the files
737
+ for (size_t f = 0; f < files_used; f++)
738
+ dictionary_acquired_item_release(nd_journal_files_registry, file_items[f]);
739
+
740
+ return rrd_call_function_error(wb, "No new data since the previous call.", HTTP_RESP_NOT_MODIFIED);
741
+ }
742
+
743
+ // sort the files, so that they are optimal for facets
744
+ if (files_used >= 2) {
745
+ if (lqs->rq.direction == FACETS_ANCHOR_DIRECTION_BACKWARD)
746
+ qsort(file_items, files_used, sizeof(const DICTIONARY_ITEM *), nd_journal_file_dict_items_backward_compar);
747
+ else
748
+ qsort(file_items, files_used, sizeof(const DICTIONARY_ITEM *), nd_journal_file_dict_items_forward_compar);
749
+ }
750
+
751
+ bool partial = false;
752
+ usec_t query_started_ut = now_monotonic_usec();
753
+ usec_t started_ut = query_started_ut;
754
+ usec_t ended_ut = started_ut;
755
+ usec_t duration_ut = 0, max_duration_ut = 0;
756
+ usec_t progress_duration_ut = 0;
757
+
758
+ sampling_query_init(lqs, facets);
759
+
760
+ buffer_json_member_add_array(wb, "_journal_files");
761
+ for (size_t f = 0; f < files_used; f++) {
762
+ const char *filename = dictionary_acquired_item_name(file_items[f]);
763
+ njf = dictionary_acquired_item_value(file_items[f]);
764
+
765
+ if (!jf_is_mine(njf, lqs))
766
+ continue;
767
+
768
+ started_ut = ended_ut;
769
+
770
+ // do not even try to do the query if we expect it to pass the timeout
771
+ if (ended_ut + max_duration_ut * 3 >= *lqs->stop_monotonic_ut) {
772
+ partial = true;
773
+ status = ND_SD_JOURNAL_TIMED_OUT;
774
+ break;
775
+ }
776
+
777
+ lqs->c.file_working++;
778
+
779
+ size_t fs_calls = fstat_thread_calls;
780
+ size_t fs_cached = fstat_thread_cached_responses;
781
+ size_t rows_useful = lqs->c.rows_useful;
782
+ size_t rows_read = lqs->c.rows_read;
783
+ size_t bytes_read = lqs->c.bytes_read;
784
+ size_t matches_setup_ut = lqs->c.matches_setup_ut;
785
+
786
+ sampling_file_init(lqs, njf);
787
+
788
+ ND_SD_JOURNAL_STATUS tmp_status = nd_sd_journal_query_one_file(filename, wb, facets, njf, lqs);
789
+
790
+ rows_useful = lqs->c.rows_useful - rows_useful;
791
+ rows_read = lqs->c.rows_read - rows_read;
792
+ bytes_read = lqs->c.bytes_read - bytes_read;
793
+ matches_setup_ut = lqs->c.matches_setup_ut - matches_setup_ut;
794
+ fs_calls = fstat_thread_calls - fs_calls;
795
+ fs_cached = fstat_thread_cached_responses - fs_cached;
796
+
797
+ ended_ut = now_monotonic_usec();
798
+ duration_ut = ended_ut - started_ut;
799
+
800
+ if (duration_ut > max_duration_ut)
801
+ max_duration_ut = duration_ut;
802
+
803
+ progress_duration_ut += duration_ut;
804
+ if (progress_duration_ut >= ND_SD_JOURNAL_PROGRESS_EVERY_UT) {
805
+ progress_duration_ut = 0;
806
+ netdata_mutex_lock(&stdout_mutex);
807
+ pluginsd_function_progress_to_stdout(lqs->rq.transaction, f + 1, files_used);
808
+ netdata_mutex_unlock(&stdout_mutex);
809
+ }
810
+
811
+ buffer_json_add_array_item_object(wb); // journal file
812
+ {
813
+ // information about the file
814
+ buffer_json_member_add_string(wb, "_filename", filename);
815
+ buffer_json_member_add_uint64(wb, "_source_type", njf->source_type);
816
+ buffer_json_member_add_string(wb, "_source", string2str(njf->source));
817
+ buffer_json_member_add_uint64(wb, "_last_modified_ut", njf->file_last_modified_ut);
818
+ buffer_json_member_add_uint64(wb, "_msg_first_ut", njf->msg_first_ut);
819
+ buffer_json_member_add_uint64(wb, "_msg_last_ut", njf->msg_last_ut);
820
+ buffer_json_member_add_uint64(wb, "_journal_vs_realtime_delta_ut", njf->max_journal_vs_realtime_delta_ut);
821
+
822
+ // information about the current use of the file
823
+ buffer_json_member_add_uint64(wb, "duration_ut", ended_ut - started_ut);
824
+ buffer_json_member_add_uint64(wb, "rows_read", rows_read);
825
+ buffer_json_member_add_uint64(wb, "rows_useful", rows_useful);
826
+ buffer_json_member_add_double(
827
+ wb, "rows_per_second", (double)rows_read / (double)duration_ut * (double)USEC_PER_SEC);
828
+ buffer_json_member_add_uint64(wb, "bytes_read", bytes_read);
829
+ buffer_json_member_add_double(
830
+ wb, "bytes_per_second", (double)bytes_read / (double)duration_ut * (double)USEC_PER_SEC);
831
+ buffer_json_member_add_uint64(wb, "duration_matches_ut", matches_setup_ut);
832
+ buffer_json_member_add_uint64(wb, "fstat_query_calls", fs_calls);
833
+ buffer_json_member_add_uint64(wb, "fstat_query_cached_responses", fs_cached);
834
+
835
+ if (lqs->rq.sampling) {
836
+ buffer_json_member_add_object(wb, "_sampling");
837
+ {
838
+ buffer_json_member_add_uint64(wb, "sampled", lqs->c.samples_per_file.sampled);
839
+ buffer_json_member_add_uint64(wb, "unsampled", lqs->c.samples_per_file.unsampled);
840
+ buffer_json_member_add_uint64(wb, "estimated", lqs->c.samples_per_file.estimated);
841
+ }
842
+ buffer_json_object_close(wb); // _sampling
843
+ }
844
+ }
845
+ buffer_json_object_close(wb); // journal file
846
+
847
+ bool stop = false;
848
+ switch (tmp_status) {
849
+ case ND_SD_JOURNAL_OK:
850
+ case ND_SD_JOURNAL_NO_FILE_MATCHED:
851
+ status = (status == ND_SD_JOURNAL_OK) ? ND_SD_JOURNAL_OK : tmp_status;
852
+ break;
853
+
854
+ case ND_SD_JOURNAL_FAILED_TO_OPEN:
855
+ case ND_SD_JOURNAL_FAILED_TO_SEEK:
856
+ partial = true;
857
+ if (status == ND_SD_JOURNAL_NO_FILE_MATCHED)
858
+ status = tmp_status;
859
+ break;
860
+
861
+ case ND_SD_JOURNAL_CANCELLED:
862
+ case ND_SD_JOURNAL_TIMED_OUT:
863
+ partial = true;
864
+ stop = true;
865
+ status = tmp_status;
866
+ break;
867
+
868
+ case ND_SD_JOURNAL_NOT_MODIFIED:
869
+ internal_fatal(true, "this should never be returned here");
870
+ break;
871
+ }
872
+
873
+ if (stop)
874
+ break;
875
+ }
876
+ buffer_json_array_close(wb); // _journal_files
877
+
878
+ // release the files
879
+ for (size_t f = 0; f < files_used; f++)
880
+ dictionary_acquired_item_release(nd_journal_files_registry, file_items[f]);
881
+
882
+ switch (status) {
883
+ case ND_SD_JOURNAL_OK:
884
+ if (lqs->rq.if_modified_since && !lqs->c.rows_useful)
885
+ return rrd_call_function_error(
886
+ wb, "No additional useful data since the previous call.", HTTP_RESP_NOT_MODIFIED);
887
+ break;
888
+
889
+ case ND_SD_JOURNAL_TIMED_OUT:
890
+ case ND_SD_JOURNAL_NO_FILE_MATCHED:
891
+ break;
892
+
893
+ case ND_SD_JOURNAL_CANCELLED:
894
+ return rrd_call_function_error(wb, "Request cancelled.", HTTP_RESP_CLIENT_CLOSED_REQUEST);
895
+
896
+ case ND_SD_JOURNAL_NOT_MODIFIED:
897
+ return rrd_call_function_error(wb, "No new data since the previous call.", HTTP_RESP_NOT_MODIFIED);
898
+
899
+ case ND_SD_JOURNAL_FAILED_TO_OPEN:
900
+ return rrd_call_function_error(wb, "Failed to open systemd journal file.", HTTP_RESP_INTERNAL_SERVER_ERROR);
901
+
902
+ case ND_SD_JOURNAL_FAILED_TO_SEEK:
903
+ return rrd_call_function_error(
904
+ wb, "Failed to seek in systemd journal file.", HTTP_RESP_INTERNAL_SERVER_ERROR);
905
+
906
+ default:
907
+ return rrd_call_function_error(wb, "Unknown status", HTTP_RESP_INTERNAL_SERVER_ERROR);
908
+ }
909
+
910
+ buffer_json_member_add_uint64(wb, "status", HTTP_RESP_OK);
911
+ buffer_json_member_add_boolean(wb, "partial", partial);
912
+ buffer_json_member_add_string(wb, "type", "table");
913
+
914
+ // build a message for the query
915
+ if (!lqs->rq.data_only) {
916
+ CLEAN_BUFFER *msg = buffer_create(0, NULL);
917
+ CLEAN_BUFFER *msg_description = buffer_create(0, NULL);
918
+ ND_LOG_FIELD_PRIORITY msg_priority = NDLP_INFO;
919
+
920
+ if (!nd_journal_files_completed_once()) {
921
+ buffer_strcat(msg, "Journals are still being scanned. ");
922
+ buffer_strcat(
923
+ msg_description,
924
+ "LIBRARY SCAN: The journal files are still being scanned, you are probably viewing incomplete data. ");
925
+ msg_priority = NDLP_WARNING;
926
+ }
927
+
928
+ if (partial) {
929
+ buffer_strcat(msg, "Query timed-out, incomplete data. ");
930
+ buffer_strcat(
931
+ msg_description,
932
+ "QUERY TIMEOUT: The query timed out and may not include all the data of the selected window. ");
933
+ msg_priority = NDLP_WARNING;
934
+ }
935
+
936
+ if (lqs->c.samples.estimated || lqs->c.samples.unsampled) {
937
+ double percent = (double)(lqs->c.samples.sampled * 100.0 /
938
+ (lqs->c.samples.estimated + lqs->c.samples.unsampled + lqs->c.samples.sampled));
939
+ buffer_sprintf(msg, "%.2f%% real data", percent);
940
+ buffer_sprintf(msg_description, "ACTUAL DATA: The filters counters reflect %0.2f%% of the data. ", percent);
941
+ msg_priority = MIN(msg_priority, NDLP_NOTICE);
942
+ }
943
+
944
+ if (lqs->c.samples.unsampled) {
945
+ double percent = (double)(lqs->c.samples.unsampled * 100.0 /
946
+ (lqs->c.samples.estimated + lqs->c.samples.unsampled + lqs->c.samples.sampled));
947
+ buffer_sprintf(msg, ", %.2f%% unsampled", percent);
948
+ buffer_sprintf(
949
+ msg_description,
950
+ "UNSAMPLED DATA: %0.2f%% of the events exist and have been counted, but their values have not been evaluated, so they are not included in the filters counters. ",
951
+ percent);
952
+ msg_priority = MIN(msg_priority, NDLP_NOTICE);
953
+ }
954
+
955
+ if (lqs->c.samples.estimated) {
956
+ double percent = (double)(lqs->c.samples.estimated * 100.0 /
957
+ (lqs->c.samples.estimated + lqs->c.samples.unsampled + lqs->c.samples.sampled));
958
+ buffer_sprintf(msg, ", %.2f%% estimated", percent);
959
+ buffer_sprintf(
960
+ msg_description,
961
+ "ESTIMATED DATA: The query selected a large amount of data, so to avoid delaying too much, the presented data are estimated by %0.2f%%. ",
962
+ percent);
963
+ msg_priority = MIN(msg_priority, NDLP_NOTICE);
964
+ }
965
+
966
+ buffer_json_member_add_object(wb, "message");
967
+ if (buffer_tostring(msg)) {
968
+ buffer_json_member_add_string(wb, "title", buffer_tostring(msg));
969
+ buffer_json_member_add_string(wb, "description", buffer_tostring(msg_description));
970
+ buffer_json_member_add_string(wb, "status", nd_log_id2priority(msg_priority));
971
+ }
972
+ // else send an empty object if there is nothing to tell
973
+ buffer_json_object_close(wb); // message
974
+ }
975
+
976
+ if (!lqs->rq.data_only) {
977
+ buffer_json_member_add_time_t(wb, "update_every", 1);
978
+ buffer_json_member_add_string(wb, "help", ND_SD_JOURNAL_FUNCTION_DESCRIPTION);
979
+ }
980
+
981
+ if (!lqs->rq.data_only || lqs->rq.tail)
982
+ buffer_json_member_add_uint64(wb, "last_modified", lqs->last_modified);
983
+
984
+ facets_sort_and_reorder_keys(facets);
985
+ facets_report(facets, wb, used_hashes_registry);
986
+
987
+ wb->expires = now_realtime_sec() + (lqs->rq.data_only ? 3600 : 0);
988
+ buffer_json_member_add_time_t(wb, "expires", wb->expires);
989
+
990
+ buffer_json_member_add_object(wb, "_fstat_caching");
991
+ {
992
+ buffer_json_member_add_uint64(wb, "calls", fstat_thread_calls);
993
+ buffer_json_member_add_uint64(wb, "cached", fstat_thread_cached_responses);
994
+ }
995
+ buffer_json_object_close(wb); // _fstat_caching
996
+
997
+ if (lqs->rq.sampling) {
998
+ buffer_json_member_add_object(wb, "_sampling");
999
+ {
1000
+ buffer_json_member_add_uint64(wb, "sampled", lqs->c.samples.sampled);
1001
+ buffer_json_member_add_uint64(wb, "unsampled", lqs->c.samples.unsampled);
1002
+ buffer_json_member_add_uint64(wb, "estimated", lqs->c.samples.estimated);
1003
+ }
1004
+ buffer_json_object_close(wb); // _sampling
1005
+ }
1006
+
1007
+ wb->content_type = CT_APPLICATION_JSON;
1008
+ wb->response_code = HTTP_RESP_OK;
1009
+ return wb->response_code;
1010
+}
1011
+
1012
+static void systemd_journal_register_transformations(LOGS_QUERY_STATUS *lqs)
1013
+{
1014
+ FACETS *facets = lqs->facets;
1015
+ LOGS_QUERY_REQUEST *rq = &lqs->rq;
1016
+
1017
+ // ----------------------------------------------------------------------------------------------------------------
1018
+ // register the fields in the order you want them on the dashboard
1019
+
1020
+ facets_register_row_severity(facets, syslog_priority_to_facet_severity, NULL);
1021
+
1022
+ facets_register_key_name(facets, "_HOSTNAME", rq->default_facet | FACET_KEY_OPTION_VISIBLE);
1023
+
1024
+ facets_register_dynamic_key_name(
1025
+ facets,
1026
+ JOURNAL_KEY_ND_JOURNAL_PROCESS,
1027
+ FACET_KEY_OPTION_NEVER_FACET | FACET_KEY_OPTION_VISIBLE,
1028
+ nd_sd_journal_dynamic_row_id,
1029
+ NULL);
1030
+
1031
+ facets_register_key_name(
1032
+ facets,
1033
+ "MESSAGE",
1034
+ FACET_KEY_OPTION_NEVER_FACET | FACET_KEY_OPTION_MAIN_TEXT | FACET_KEY_OPTION_VISIBLE | FACET_KEY_OPTION_FTS);
1035
+
1036
+ facets_register_key_name_transformation(
1037
+ facets,
1038
+ "PRIORITY",
1039
+ rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW | FACET_KEY_OPTION_EXPANDED_FILTER,
1040
+ nd_sd_journal_transform_priority,
1041
+ NULL);
1042
+
1043
+ facets_register_key_name_transformation(
1044
+ facets,
1045
+ "SYSLOG_FACILITY",
1046
+ rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW | FACET_KEY_OPTION_EXPANDED_FILTER,
1047
+ nd_sd_journal_transform_syslog_facility,
1048
+ NULL);
1049
+
1050
+ facets_register_key_name_transformation(
1051
+ facets, "ERRNO", rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_errno, NULL);
1052
+
1053
+ facets_register_key_name(facets, JOURNAL_KEY_ND_JOURNAL_FILE, FACET_KEY_OPTION_NEVER_FACET);
1054
+
1055
+ facets_register_key_name(facets, "SYSLOG_IDENTIFIER", rq->default_facet);
1056
+
1057
+ facets_register_key_name(facets, "UNIT", rq->default_facet);
1058
+
1059
+ facets_register_key_name(facets, "USER_UNIT", rq->default_facet);
1060
+
1061
+ facets_register_key_name_transformation(
1062
+ facets,
1063
+ "MESSAGE_ID",
1064
+ rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW | FACET_KEY_OPTION_EXPANDED_FILTER,
1065
+ nd_sd_journal_transform_message_id,
1066
+ NULL);
1067
+
1068
+ facets_register_key_name_transformation(
1069
+ facets, "_BOOT_ID", rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_boot_id, NULL);
1070
+
1071
+ facets_register_key_name_transformation(
1072
+ facets,
1073
+ "_SYSTEMD_OWNER_UID",
1074
+ rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW,
1075
+ nd_sd_journal_transform_uid,
1076
+ NULL);
1077
+
1078
+ facets_register_key_name_transformation(
1079
+ facets, "_UID", rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_uid, NULL);
1080
+
1081
+ facets_register_key_name_transformation(
1082
+ facets,
1083
+ "OBJECT_SYSTEMD_OWNER_UID",
1084
+ rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW,
1085
+ nd_sd_journal_transform_uid,
1086
+ NULL);
1087
+
1088
+ facets_register_key_name_transformation(
1089
+ facets, "OBJECT_UID", rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_uid, NULL);
1090
+
1091
+ facets_register_key_name_transformation(
1092
+ facets, "_GID", rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_gid, NULL);
1093
+
1094
+ facets_register_key_name_transformation(
1095
+ facets, "OBJECT_GID", rq->default_facet | FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_gid, NULL);
1096
+
1097
+ facets_register_key_name_transformation(
1098
+ facets, "_CAP_EFFECTIVE", FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_cap_effective, NULL);
1099
+
1100
+ facets_register_key_name_transformation(
1101
+ facets, "_AUDIT_LOGINUID", FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_uid, NULL);
1102
+
1103
+ facets_register_key_name_transformation(
1104
+ facets, "OBJECT_AUDIT_LOGINUID", FACET_KEY_OPTION_TRANSFORM_VIEW, nd_sd_journal_transform_uid, NULL);
1105
+
1106
+ facets_register_key_name_transformation(
1107
+ facets,
1108
+ "_SOURCE_REALTIME_TIMESTAMP",
1109
+ FACET_KEY_OPTION_TRANSFORM_VIEW,
1110
+ nd_sd_journal_transform_timestamp_usec,
1111
+ NULL);
1112
+}
1113
+
1114
+void function_systemd_journal(
1115
+ const char *transaction,
1116
+ char *function,
1117
+ usec_t *stop_monotonic_ut,
1118
+ bool *cancelled,
1119
+ BUFFER *payload,
1120
+ HTTP_ACCESS access __maybe_unused,
1121
+ const char *source __maybe_unused,
1122
+ void *data __maybe_unused)
1123
+{
1124
+ fstat_thread_calls = 0;
1125
+ fstat_thread_cached_responses = 0;
1126
+
1127
+#ifdef HAVE_SD_JOURNAL_RESTART_FIELDS
1128
+ bool have_slice = true;
1129
+#else
1130
+ bool have_slice = false;
1131
+#endif // HAVE_SD_JOURNAL_RESTART_FIELDS
1132
+
1133
+ LOGS_QUERY_STATUS tmp_fqs = {
1134
+ .facets = lqs_facets_create(
1135
+ LQS_DEFAULT_ITEMS_PER_QUERY,
1136
+ FACETS_OPTION_ALL_KEYS_FTS | FACETS_OPTION_HASH_IDS,
1137
+ SYSTEMD_ALWAYS_VISIBLE_KEYS,
1138
+ SYSTEMD_KEYS_INCLUDED_IN_FACETS,
1139
+ SYSTEMD_KEYS_EXCLUDED_FROM_FACETS,
1140
+ have_slice),
1141
+
1142
+ .rq = LOGS_QUERY_REQUEST_DEFAULTS(transaction, LQS_DEFAULT_SLICE_MODE, JOURNAL_DEFAULT_DIRECTION),
1143
+
1144
+ .cancelled = cancelled,
1145
+ .stop_monotonic_ut = stop_monotonic_ut,
1146
+ };
1147
+ LOGS_QUERY_STATUS *lqs = &tmp_fqs;
1148
+
1149
+ CLEAN_BUFFER *wb = lqs_create_output_buffer();
1150
+
1151
+ // ------------------------------------------------------------------------
1152
+ // parse the parameters
1153
+
1154
+ if (lqs_request_parse_and_validate(lqs, wb, function, payload, have_slice, "PRIORITY")) {
1155
+ systemd_journal_register_transformations(lqs);
1156
+
1157
+ // ------------------------------------------------------------------------
1158
+ // add versions to the response
1159
+
1160
+ buffer_json_journal_versions(wb);
1161
+
1162
+ // ------------------------------------------------------------------------
1163
+ // run the request
1164
+
1165
+ if (lqs->rq.info)
1166
+ lqs_info_response(wb, lqs->facets);
1167
+ else {
1168
+ nd_sd_journal_query(wb, lqs);
1169
+ if (wb->response_code == HTTP_RESP_OK)
1170
+ buffer_json_finalize(wb);
1171
+ }
1172
+ }
1173
+
1174
+ netdata_mutex_lock(&stdout_mutex);
1175
+ pluginsd_function_result_to_stdout(transaction, wb);
1176
+ netdata_mutex_unlock(&stdout_mutex);
1177
+
1178
+ lqs_cleanup(lqs);
1179
+}
src/collectors/systemd-journal.plugin/systemd-main.c
new
+126
@@ -0,0 +1,126 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+#include "systemd-internals.h"
4
+#include "libnetdata/required_dummies.h"
5
+
6
+#define ND_SD_JOURNAL_WORKER_THREADS 5
7
+
8
+netdata_mutex_t stdout_mutex;
9
+
10
+static void __attribute__((constructor)) init_mutex(void) {
11
+ netdata_mutex_init(&stdout_mutex);
12
+}
13
+
14
+static void __attribute__((destructor)) destroy_mutex(void) {
15
+ netdata_mutex_destroy(&stdout_mutex);
16
+}
17
+
18
+static bool plugin_should_exit = false;
19
+
20
+static bool journal_data_directories_exist()
21
+{
22
+ struct stat st;
23
+ for (unsigned i = 0; i < MAX_JOURNAL_DIRECTORIES && journal_directories[i].path; i++) {
24
+ if ((stat(string2str(journal_directories[i].path), &st) == 0) && S_ISDIR(st.st_mode))
25
+ return true;
26
+ }
27
+ return false;
28
+}
29
+
30
+int main(int argc __maybe_unused, char **argv __maybe_unused)
31
+{
32
+ nd_thread_tag_set("sd-jrnl.plugin");
33
+ nd_log_initialize_for_external_plugins("systemd-journal.plugin");
34
+ netdata_threads_init_for_external_plugins(0);
35
+
36
+ netdata_configured_host_prefix = getenv("NETDATA_HOST_PREFIX");
37
+ if (verify_netdata_host_prefix(true) == -1)
38
+ exit(1);
39
+
40
+ // ------------------------------------------------------------------------
41
+ // initialization
42
+
43
+ nd_sd_journal_annotations_init();
44
+ nd_journal_init_files_and_directories();
45
+
46
+ if (!journal_data_directories_exist()) {
47
+ nd_log_collector(NDLP_INFO, "unable to locate journal data directories. Exiting...");
48
+ fprintf(stdout, "DISABLE\n");
49
+ fflush(stdout);
50
+ exit(0);
51
+ }
52
+
53
+ // ------------------------------------------------------------------------
54
+ // debug
55
+
56
+ if (argc == 2 && strcmp(argv[1], "debug") == 0) {
57
+ nd_journal_files_registry_update();
58
+
59
+ bool cancelled = false;
60
+ usec_t stop_monotonic_ut = now_monotonic_usec() + 600 * USEC_PER_SEC;
61
+ char buf[] =
62
+ "systemd-journal after:-8640000 before:0 direction:backward last:200 data_only:false slice:true facets: source:all";
63
+ function_systemd_journal("123", buf, &stop_monotonic_ut, &cancelled, NULL, HTTP_ACCESS_ALL, NULL, NULL);
64
+ exit(1);
65
+ }
66
+
67
+ // ------------------------------------------------------------------------
68
+ // watcher thread
69
+
70
+ nd_thread_create("SDWATCH", NETDATA_THREAD_OPTION_DONT_LOG, nd_journal_watcher_main, NULL);
71
+
72
+ // ------------------------------------------------------------------------
73
+ // the event loop for functions
74
+
75
+ struct functions_evloop_globals *wg =
76
+ functions_evloop_init(ND_SD_JOURNAL_WORKER_THREADS, "SDJ", &stdout_mutex, &plugin_should_exit, NULL);
77
+
78
+ functions_evloop_add_function(
79
+ wg, ND_SD_JOURNAL_FUNCTION_NAME, function_systemd_journal, ND_SD_JOURNAL_DEFAULT_TIMEOUT, NULL);
80
+
81
+ nd_systemd_journal_dyncfg_init(wg);
82
+
83
+ // ------------------------------------------------------------------------
84
+ // register functions to netdata
85
+
86
+ netdata_mutex_lock(&stdout_mutex);
87
+
88
+ fprintf(
89
+ stdout,
90
+ PLUGINSD_KEYWORD_FUNCTION " GLOBAL \"%s\" %d \"%s\" \"logs\" " HTTP_ACCESS_FORMAT " %d\n",
91
+ ND_SD_JOURNAL_FUNCTION_NAME,
92
+ ND_SD_JOURNAL_DEFAULT_TIMEOUT,
93
+ ND_SD_JOURNAL_FUNCTION_DESCRIPTION,
94
+ (HTTP_ACCESS_FORMAT_CAST)(HTTP_ACCESS_SIGNED_ID | HTTP_ACCESS_SAME_SPACE | HTTP_ACCESS_SENSITIVE_DATA),
95
+ RRDFUNCTIONS_PRIORITY_DEFAULT);
96
+
97
+ fflush(stdout);
98
+ netdata_mutex_unlock(&stdout_mutex);
99
+
100
+ // ------------------------------------------------------------------------
101
+
102
+ usec_t send_newline_ut = 0;
103
+ usec_t since_last_scan_ut =
104
+ ND_SD_JOURNAL_ALL_FILES_SCAN_EVERY_USEC * 2; // something big to trigger scanning at start
105
+ const bool tty = isatty(fileno(stdout)) == 1;
106
+
107
+ heartbeat_t hb;
108
+ heartbeat_init(&hb, USEC_PER_SEC);
109
+ while (!__atomic_load_n(&plugin_should_exit, __ATOMIC_ACQUIRE)) {
110
+ if (since_last_scan_ut > ND_SD_JOURNAL_ALL_FILES_SCAN_EVERY_USEC) {
111
+ nd_journal_files_registry_update();
112
+ since_last_scan_ut = 0;
113
+ }
114
+
115
+ usec_t dt_ut = heartbeat_next(&hb);
116
+ since_last_scan_ut += dt_ut;
117
+ send_newline_ut += dt_ut;
118
+
119
+ if (!tty && send_newline_ut > USEC_PER_SEC) {
120
+ send_newline_and_flush(&stdout_mutex);
121
+ send_newline_ut = 0;
122
+ }
123
+ }
124
+
125
+ exit(0);
126
+}
src/crates/Cargo.lock
+30
-30
@@ -1717,36 +1717,6 @@ dependencies = [
1717
"walkdir",
1718
]
1719
1720
-[[package]]
1721
-name = "journal-viewer-plugin"
1722
-version = "0.1.3"
1723
-dependencies = [
1724
- "anyhow",
1725
- "async-trait",
1726
- "bytesize",
1727
- "bytesize-serde",
1728
- "foyer",
1729
- "journal-core",
1730
- "journal-function",
1731
- "journal-index",
1732
- "journal-registry",
1733
- "netdata-plugin-error",
1734
- "netdata-plugin-protocol",
1735
- "netdata-plugin-schema",
1736
- "notify",
1737
- "num_cpus",
1738
- "parking_lot",
1739
- "rt",
1740
- "schemars",
1741
- "serde",
1742
- "serde_json",
1743
- "serde_yaml",
1744
- "thiserror",
1745
- "tokio",
1746
- "tokio-util",
1747
- "tracing",
1748
-]
1749
-
1720
[[package]]
1721
name = "js-sys"
1722
version = "0.3.82"
@@ -2277,6 +2247,36 @@ dependencies = [
2247
"tracing",
2248
]
2249
2250
+[[package]]
2251
+name = "otel-signal-viewer-plugin"
2252
+version = "0.1.3"
2253
+dependencies = [
2254
+ "anyhow",
2255
+ "async-trait",
2256
+ "bytesize",
2257
+ "bytesize-serde",
2258
+ "foyer",
2259
+ "journal-core",
2260
+ "journal-function",
2261
+ "journal-index",
2262
+ "journal-registry",
2263
+ "netdata-plugin-error",
2264
+ "netdata-plugin-protocol",
2265
+ "netdata-plugin-schema",
2266
+ "notify",
2267
+ "num_cpus",
2268
+ "parking_lot",
2269
+ "rt",
2270
+ "schemars",
2271
+ "serde",
2272
+ "serde_json",
2273
+ "serde_yaml",
2274
+ "thiserror",
2275
+ "tokio",
2276
+ "tokio-util",
2277
+ "tracing",
2278
+]
2279
+
2280
[[package]]
2281
name = "panic-message"
2282
version = "0.3.0"
src/crates/Cargo.toml
+3
-2
@@ -1,5 +1,6 @@
1
[workspace]
2
resolver = "2"
3
+exclude = ["jf"]
4
members = [
5
# Individual crates
6
"journal-common",
@@ -21,7 +22,7 @@ members = [
22
23
# Netdata log viewer workspace members
24
"netdata-log-viewer/journal-function",
24
- "netdata-log-viewer/journal-viewer-plugin",
25
+ "netdata-log-viewer/otel-signal-viewer-plugin",
26
27
# Netdata OTEL workspace members
28
"netdata-otel/otel-plugin",
@@ -146,7 +147,7 @@ rt = { path = "netdata-plugin/rt" }
147
148
# Netdata log viewer crates
149
journal-function = { path = "netdata-log-viewer/journal-function" }
149
-journal-viewer-plugin = { path = "netdata-log-viewer/journal-viewer-plugin" }
150
+otel-signal-viewer-plugin = { path = "netdata-log-viewer/otel-signal-viewer-plugin" }
151
152
# Netdata OTEL crates
153
otel-plugin = { path = "netdata-otel/otel-plugin" }
src/crates/jf/.gitignore
new
+3
@@ -0,0 +1,3 @@
1
+target/
2
+.idea/
3
+journal_reader_ffi/journal_reader_ffi.h
src/crates/jf/Cargo.lock
new
+2221
@@ -0,0 +1,2221 @@
1
+# This file is automatically @generated by Cargo.
2
+# It is not intended for manual editing.
3
+version = 4
4
+
5
+[[package]]
6
+name = "addr2line"
7
+version = "0.25.1"
8
+source = "registry+https://github.com/rust-lang/crates.io-index"
9
+checksum = "1b5d307320b3181d6d7954e663bd7c774a838b8220fe0593c86d9fb09f498b4b"
10
+dependencies = [
11
+ "gimli",
12
+]
13
+
14
+[[package]]
15
+name = "adler2"
16
+version = "2.0.1"
17
+source = "registry+https://github.com/rust-lang/crates.io-index"
18
+checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
19
+
20
+[[package]]
21
+name = "aho-corasick"
22
+version = "1.1.3"
23
+source = "registry+https://github.com/rust-lang/crates.io-index"
24
+checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916"
25
+dependencies = [
26
+ "memchr",
27
+]
28
+
29
+[[package]]
30
+name = "android_system_properties"
31
+version = "0.1.5"
32
+source = "registry+https://github.com/rust-lang/crates.io-index"
33
+checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311"
34
+dependencies = [
35
+ "libc",
36
+]
37
+
38
+[[package]]
39
+name = "anstream"
40
+version = "0.6.20"
41
+source = "registry+https://github.com/rust-lang/crates.io-index"
42
+checksum = "3ae563653d1938f79b1ab1b5e668c87c76a9930414574a6583a7b7e11a8e6192"
43
+dependencies = [
44
+ "anstyle",
45
+ "anstyle-parse",
46
+ "anstyle-query",
47
+ "anstyle-wincon",
48
+ "colorchoice",
49
+ "is_terminal_polyfill",
50
+ "utf8parse",
51
+]
52
+
53
+[[package]]
54
+name = "anstyle"
55
+version = "1.0.13"
56
+source = "registry+https://github.com/rust-lang/crates.io-index"
57
+checksum = "5192cca8006f1fd4f7237516f40fa183bb07f8fbdfedaa0036de5ea9b0b45e78"
58
+
59
+[[package]]
60
+name = "anstyle-parse"
61
+version = "0.2.7"
62
+source = "registry+https://github.com/rust-lang/crates.io-index"
63
+checksum = "4e7644824f0aa2c7b9384579234ef10eb7efb6a0deb83f9630a49594dd9c15c2"
64
+dependencies = [
65
+ "utf8parse",
66
+]
67
+
68
+[[package]]
69
+name = "anstyle-query"
70
+version = "1.1.4"
71
+source = "registry+https://github.com/rust-lang/crates.io-index"
72
+checksum = "9e231f6134f61b71076a3eab506c379d4f36122f2af15a9ff04415ea4c3339e2"
73
+dependencies = [
74
+ "windows-sys 0.60.2",
75
+]
76
+
77
+[[package]]
78
+name = "anstyle-wincon"
79
+version = "3.0.10"
80
+source = "registry+https://github.com/rust-lang/crates.io-index"
81
+checksum = "3e0633414522a32ffaac8ac6cc8f748e090c5717661fddeea04219e2344f5f2a"
82
+dependencies = [
83
+ "anstyle",
84
+ "once_cell_polyfill",
85
+ "windows-sys 0.60.2",
86
+]
87
+
88
+[[package]]
89
+name = "anyhow"
90
+version = "1.0.100"
91
+source = "registry+https://github.com/rust-lang/crates.io-index"
92
+checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
93
+
94
+[[package]]
95
+name = "async-trait"
96
+version = "0.1.89"
97
+source = "registry+https://github.com/rust-lang/crates.io-index"
98
+checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb"
99
+dependencies = [
100
+ "proc-macro2",
101
+ "quote",
102
+ "syn",
103
+]
104
+
105
+[[package]]
106
+name = "atomic-waker"
107
+version = "1.1.2"
108
+source = "registry+https://github.com/rust-lang/crates.io-index"
109
+checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0"
110
+
111
+[[package]]
112
+name = "atty"
113
+version = "0.2.14"
114
+source = "registry+https://github.com/rust-lang/crates.io-index"
115
+checksum = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8"
116
+dependencies = [
117
+ "hermit-abi",
118
+ "libc",
119
+ "winapi",
120
+]
121
+
122
+[[package]]
123
+name = "autocfg"
124
+version = "1.5.0"
125
+source = "registry+https://github.com/rust-lang/crates.io-index"
126
+checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
127
+
128
+[[package]]
129
+name = "axum"
130
+version = "0.8.5"
131
+source = "registry+https://github.com/rust-lang/crates.io-index"
132
+checksum = "98e529aee37b5c8206bb4bf4c44797127566d72f76952c970bd3d1e85de8f4e2"
133
+dependencies = [
134
+ "axum-core",
135
+ "bytes",
136
+ "futures-util",
137
+ "http",
138
+ "http-body",
139
+ "http-body-util",
140
+ "itoa",
141
+ "matchit",
142
+ "memchr",
143
+ "mime",
144
+ "percent-encoding",
145
+ "pin-project-lite",
146
+ "serde_core",
147
+ "sync_wrapper",
148
+ "tower",
149
+ "tower-layer",
150
+ "tower-service",
151
+]
152
+
153
+[[package]]
154
+name = "axum-core"
155
+version = "0.5.4"
156
+source = "registry+https://github.com/rust-lang/crates.io-index"
157
+checksum = "0ac7a6beb1182c7e30253ee75c3e918080bfb83f5a3023bcdf7209d85fd147e6"
158
+dependencies = [
159
+ "bytes",
160
+ "futures-core",
161
+ "http",
162
+ "http-body",
163
+ "http-body-util",
164
+ "mime",
165
+ "pin-project-lite",
166
+ "sync_wrapper",
167
+ "tower-layer",
168
+ "tower-service",
169
+]
170
+
171
+[[package]]
172
+name = "backtrace"
173
+version = "0.3.76"
174
+source = "registry+https://github.com/rust-lang/crates.io-index"
175
+checksum = "bb531853791a215d7c62a30daf0dde835f381ab5de4589cfe7c649d2cbe92bd6"
176
+dependencies = [
177
+ "addr2line",
178
+ "cfg-if",
179
+ "libc",
180
+ "miniz_oxide",
181
+ "object",
182
+ "rustc-demangle",
183
+ "windows-link",
184
+]
185
+
186
+[[package]]
187
+name = "base64"
188
+version = "0.21.7"
189
+source = "registry+https://github.com/rust-lang/crates.io-index"
190
+checksum = "9d297deb1925b89f2ccc13d7635fa0714f12c87adce1c75356b39ca9b7178567"
191
+
192
+[[package]]
193
+name = "base64"
194
+version = "0.22.1"
195
+source = "registry+https://github.com/rust-lang/crates.io-index"
196
+checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
197
+
198
+[[package]]
199
+name = "bitflags"
200
+version = "2.9.4"
201
+source = "registry+https://github.com/rust-lang/crates.io-index"
202
+checksum = "2261d10cca569e4643e526d8dc2e62e433cc8aba21ab764233731f8d369bf394"
203
+
204
+[[package]]
205
+name = "bumpalo"
206
+version = "3.19.0"
207
+source = "registry+https://github.com/rust-lang/crates.io-index"
208
+checksum = "46c5e41b57b8bba42a04676d81cb89e9ee8e859a1a66f80a5a72e1cb76b34d43"
209
+
210
+[[package]]
211
+name = "bytes"
212
+version = "1.10.1"
213
+source = "registry+https://github.com/rust-lang/crates.io-index"
214
+checksum = "d71b6127be86fdcfddb610f7182ac57211d4b18a3e9c82eb2d17662f2227ad6a"
215
+
216
+[[package]]
217
+name = "bytesize"
218
+version = "1.3.3"
219
+source = "registry+https://github.com/rust-lang/crates.io-index"
220
+checksum = "2e93abca9e28e0a1b9877922aacb20576e05d4679ffa78c3d6dc22a26a216659"
221
+
222
+[[package]]
223
+name = "bytesize-serde"
224
+version = "0.2.1"
225
+source = "registry+https://github.com/rust-lang/crates.io-index"
226
+checksum = "86d1eb2fd2668859e9785b99700c66b7ea9fdeda35d99f95827a4e5493440178"
227
+dependencies = [
228
+ "bytesize",
229
+ "serde",
230
+]
231
+
232
+[[package]]
233
+name = "cbindgen"
234
+version = "0.28.0"
235
+source = "registry+https://github.com/rust-lang/crates.io-index"
236
+checksum = "eadd868a2ce9ca38de7eeafdcec9c7065ef89b42b32f0839278d55f35c54d1ff"
237
+dependencies = [
238
+ "clap",
239
+ "heck 0.4.1",
240
+ "indexmap",
241
+ "log",
242
+ "proc-macro2",
243
+ "quote",
244
+ "serde",
245
+ "serde_json",
246
+ "syn",
247
+ "tempfile",
248
+ "toml",
249
+]
250
+
251
+[[package]]
252
+name = "cc"
253
+version = "1.2.39"
254
+source = "registry+https://github.com/rust-lang/crates.io-index"
255
+checksum = "e1354349954c6fc9cb0deab020f27f783cf0b604e8bb754dc4658ecf0d29c35f"
256
+dependencies = [
257
+ "find-msvc-tools",
258
+ "shlex",
259
+]
260
+
261
+[[package]]
262
+name = "cfg-if"
263
+version = "1.0.3"
264
+source = "registry+https://github.com/rust-lang/crates.io-index"
265
+checksum = "2fd1289c04a9ea8cb22300a459a72a385d7c73d3259e2ed7dcb2af674838cfa9"
266
+
267
+[[package]]
268
+name = "chrono"
269
+version = "0.4.42"
270
+source = "registry+https://github.com/rust-lang/crates.io-index"
271
+checksum = "145052bdd345b87320e369255277e3fb5152762ad123a901ef5c262dd38fe8d2"
272
+dependencies = [
273
+ "iana-time-zone",
274
+ "js-sys",
275
+ "num-traits",
276
+ "serde",
277
+ "wasm-bindgen",
278
+ "windows-link",
279
+]
280
+
281
+[[package]]
282
+name = "clap"
283
+version = "4.5.48"
284
+source = "registry+https://github.com/rust-lang/crates.io-index"
285
+checksum = "e2134bb3ea021b78629caa971416385309e0131b351b25e01dc16fb54e1b5fae"
286
+dependencies = [
287
+ "clap_builder",
288
+ "clap_derive",
289
+]
290
+
291
+[[package]]
292
+name = "clap_builder"
293
+version = "4.5.48"
294
+source = "registry+https://github.com/rust-lang/crates.io-index"
295
+checksum = "c2ba64afa3c0a6df7fa517765e31314e983f51dda798ffba27b988194fb65dc9"
296
+dependencies = [
297
+ "anstream",
298
+ "anstyle",
299
+ "clap_lex",
300
+ "strsim",
301
+]
302
+
303
+[[package]]
304
+name = "clap_derive"
305
+version = "4.5.47"
306
+source = "registry+https://github.com/rust-lang/crates.io-index"
307
+checksum = "bbfd7eae0b0f1a6e63d4b13c9c478de77c2eb546fba158ad50b4203dc24b9f9c"
308
+dependencies = [
309
+ "heck 0.5.0",
310
+ "proc-macro2",
311
+ "quote",
312
+ "syn",
313
+]
314
+
315
+[[package]]
316
+name = "clap_lex"
317
+version = "0.7.5"
318
+source = "registry+https://github.com/rust-lang/crates.io-index"
319
+checksum = "b94f61472cee1439c0b966b47e3aca9ae07e45d070759512cd390ea2bebc6675"
320
+
321
+[[package]]
322
+name = "colorchoice"
323
+version = "1.0.4"
324
+source = "registry+https://github.com/rust-lang/crates.io-index"
325
+checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75"
326
+
327
+[[package]]
328
+name = "core-foundation-sys"
329
+version = "0.8.7"
330
+source = "registry+https://github.com/rust-lang/crates.io-index"
331
+checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
332
+
333
+[[package]]
334
+name = "crc32fast"
335
+version = "1.5.0"
336
+source = "registry+https://github.com/rust-lang/crates.io-index"
337
+checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511"
338
+dependencies = [
339
+ "cfg-if",
340
+]
341
+
342
+[[package]]
343
+name = "either"
344
+version = "1.15.0"
345
+source = "registry+https://github.com/rust-lang/crates.io-index"
346
+checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
347
+
348
+[[package]]
349
+name = "equivalent"
350
+version = "1.0.2"
351
+source = "registry+https://github.com/rust-lang/crates.io-index"
352
+checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
353
+
354
+[[package]]
355
+name = "errno"
356
+version = "0.3.14"
357
+source = "registry+https://github.com/rust-lang/crates.io-index"
358
+checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
359
+dependencies = [
360
+ "libc",
361
+ "windows-sys 0.61.1",
362
+]
363
+
364
+[[package]]
365
+name = "error"
366
+version = "0.1.0"
367
+dependencies = [
368
+ "static_assertions",
369
+ "thiserror",
370
+ "zerocopy 0.9.0-alpha.0",
371
+]
372
+
373
+[[package]]
374
+name = "fastrand"
375
+version = "2.3.0"
376
+source = "registry+https://github.com/rust-lang/crates.io-index"
377
+checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be"
378
+
379
+[[package]]
380
+name = "find-msvc-tools"
381
+version = "0.1.2"
382
+source = "registry+https://github.com/rust-lang/crates.io-index"
383
+checksum = "1ced73b1dacfc750a6db6c0a0c3a3853c8b41997e2e2c563dc90804ae6867959"
384
+
385
+[[package]]
386
+name = "fixedbitset"
387
+version = "0.5.7"
388
+source = "registry+https://github.com/rust-lang/crates.io-index"
389
+checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99"
390
+
391
+[[package]]
392
+name = "flate2"
393
+version = "1.1.2"
394
+source = "registry+https://github.com/rust-lang/crates.io-index"
395
+checksum = "4a3d7db9596fecd151c5f638c0ee5d5bd487b6e0ea232e5dc96d5250f6f94b1d"
396
+dependencies = [
397
+ "crc32fast",
398
+ "miniz_oxide",
399
+]
400
+
401
+[[package]]
402
+name = "flatten-serde-json"
403
+version = "1.22.1"
404
+source = "git+https://github.com/meilisearch/meilisearch?tag=v1.22.1#077ec2ab11bb4daefcb57f89eab9cff16e075fdc"
405
+dependencies = [
406
+ "serde_json",
407
+]
408
+
409
+[[package]]
410
+name = "flatten_otel"
411
+version = "0.1.0"
412
+dependencies = [
413
+ "flatten-serde-json",
414
+ "opentelemetry-proto",
415
+ "serde_json",
416
+]
417
+
418
+[[package]]
419
+name = "flog-otel"
420
+version = "0.1.0"
421
+dependencies = [
422
+ "anyhow",
423
+ "chrono",
424
+ "clap",
425
+ "opentelemetry-proto",
426
+ "prost",
427
+ "serde",
428
+ "serde_json",
429
+ "tokio",
430
+ "tonic",
431
+ "tonic-build",
432
+ "tracing",
433
+ "tracing-subscriber",
434
+ "uuid",
435
+]
436
+
437
+[[package]]
438
+name = "fnv"
439
+version = "1.0.7"
440
+source = "registry+https://github.com/rust-lang/crates.io-index"
441
+checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
442
+
443
+[[package]]
444
+name = "futures-channel"
445
+version = "0.3.31"
446
+source = "registry+https://github.com/rust-lang/crates.io-index"
447
+checksum = "2dff15bf788c671c1934e366d07e30c1814a8ef514e1af724a602e8a2fbe1b10"
448
+dependencies = [
449
+ "futures-core",
450
+]
451
+
452
+[[package]]
453
+name = "futures-core"
454
+version = "0.3.31"
455
+source = "registry+https://github.com/rust-lang/crates.io-index"
456
+checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e"
457
+
458
+[[package]]
459
+name = "futures-executor"
460
+version = "0.3.31"
461
+source = "registry+https://github.com/rust-lang/crates.io-index"
462
+checksum = "1e28d1d997f585e54aebc3f97d39e72338912123a67330d723fdbb564d646c9f"
463
+dependencies = [
464
+ "futures-core",
465
+ "futures-task",
466
+ "futures-util",
467
+]
468
+
469
+[[package]]
470
+name = "futures-macro"
471
+version = "0.3.31"
472
+source = "registry+https://github.com/rust-lang/crates.io-index"
473
+checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650"
474
+dependencies = [
475
+ "proc-macro2",
476
+ "quote",
477
+ "syn",
478
+]
479
+
480
+[[package]]
481
+name = "futures-sink"
482
+version = "0.3.31"
483
+source = "registry+https://github.com/rust-lang/crates.io-index"
484
+checksum = "e575fab7d1e0dcb8d0c7bcf9a63ee213816ab51902e6d244a95819acacf1d4f7"
485
+
486
+[[package]]
487
+name = "futures-task"
488
+version = "0.3.31"
489
+source = "registry+https://github.com/rust-lang/crates.io-index"
490
+checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988"
491
+
492
+[[package]]
493
+name = "futures-util"
494
+version = "0.3.31"
495
+source = "registry+https://github.com/rust-lang/crates.io-index"
496
+checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81"
497
+dependencies = [
498
+ "futures-core",
499
+ "futures-macro",
500
+ "futures-sink",
501
+ "futures-task",
502
+ "pin-project-lite",
503
+ "pin-utils",
504
+ "slab",
505
+]
506
+
507
+[[package]]
508
+name = "getrandom"
509
+version = "0.2.16"
510
+source = "registry+https://github.com/rust-lang/crates.io-index"
511
+checksum = "335ff9f135e4384c8150d6f27c6daed433577f86b4750418338c01a1a2528592"
512
+dependencies = [
513
+ "cfg-if",
514
+ "libc",
515
+ "wasi 0.11.1+wasi-snapshot-preview1",
516
+]
517
+
518
+[[package]]
519
+name = "getrandom"
520
+version = "0.3.3"
521
+source = "registry+https://github.com/rust-lang/crates.io-index"
522
+checksum = "26145e563e54f2cadc477553f1ec5ee650b00862f0a58bcd12cbdc5f0ea2d2f4"
523
+dependencies = [
524
+ "cfg-if",
525
+ "libc",
526
+ "r-efi",
527
+ "wasi 0.14.7+wasi-0.2.4",
528
+]
529
+
530
+[[package]]
531
+name = "gimli"
532
+version = "0.32.3"
533
+source = "registry+https://github.com/rust-lang/crates.io-index"
534
+checksum = "e629b9b98ef3dd8afe6ca2bd0f89306cec16d43d907889945bc5d6687f2f13c7"
535
+
536
+[[package]]
537
+name = "h2"
538
+version = "0.4.12"
539
+source = "registry+https://github.com/rust-lang/crates.io-index"
540
+checksum = "f3c0b69cfcb4e1b9f1bf2f53f95f766e4661169728ec61cd3fe5a0166f2d1386"
541
+dependencies = [
542
+ "atomic-waker",
543
+ "bytes",
544
+ "fnv",
545
+ "futures-core",
546
+ "futures-sink",
547
+ "http",
548
+ "indexmap",
549
+ "slab",
550
+ "tokio",
551
+ "tokio-util",
552
+ "tracing",
553
+]
554
+
555
+[[package]]
556
+name = "hashbrown"
557
+version = "0.16.0"
558
+source = "registry+https://github.com/rust-lang/crates.io-index"
559
+checksum = "5419bdc4f6a9207fbeba6d11b604d481addf78ecd10c11ad51e76c2f6482748d"
560
+
561
+[[package]]
562
+name = "heck"
563
+version = "0.4.1"
564
+source = "registry+https://github.com/rust-lang/crates.io-index"
565
+checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8"
566
+
567
+[[package]]
568
+name = "heck"
569
+version = "0.5.0"
570
+source = "registry+https://github.com/rust-lang/crates.io-index"
571
+checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
572
+
573
+[[package]]
574
+name = "hermit-abi"
575
+version = "0.1.19"
576
+source = "registry+https://github.com/rust-lang/crates.io-index"
577
+checksum = "62b467343b94ba476dcb2500d242dadbb39557df889310ac77c5d99100aaac33"
578
+dependencies = [
579
+ "libc",
580
+]
581
+
582
+[[package]]
583
+name = "hex"
584
+version = "0.4.3"
585
+source = "registry+https://github.com/rust-lang/crates.io-index"
586
+checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
587
+
588
+[[package]]
589
+name = "http"
590
+version = "1.3.1"
591
+source = "registry+https://github.com/rust-lang/crates.io-index"
592
+checksum = "f4a85d31aea989eead29a3aaf9e1115a180df8282431156e533de47660892565"
593
+dependencies = [
594
+ "bytes",
595
+ "fnv",
596
+ "itoa",
597
+]
598
+
599
+[[package]]
600
+name = "http-body"
601
+version = "1.0.1"
602
+source = "registry+https://github.com/rust-lang/crates.io-index"
603
+checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184"
604
+dependencies = [
605
+ "bytes",
606
+ "http",
607
+]
608
+
609
+[[package]]
610
+name = "http-body-util"
611
+version = "0.1.3"
612
+source = "registry+https://github.com/rust-lang/crates.io-index"
613
+checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a"
614
+dependencies = [
615
+ "bytes",
616
+ "futures-core",
617
+ "http",
618
+ "http-body",
619
+ "pin-project-lite",
620
+]
621
+
622
+[[package]]
623
+name = "httparse"
624
+version = "1.10.1"
625
+source = "registry+https://github.com/rust-lang/crates.io-index"
626
+checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
627
+
628
+[[package]]
629
+name = "httpdate"
630
+version = "1.0.3"
631
+source = "registry+https://github.com/rust-lang/crates.io-index"
632
+checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
633
+
634
+[[package]]
635
+name = "humantime"
636
+version = "2.3.0"
637
+source = "registry+https://github.com/rust-lang/crates.io-index"
638
+checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424"
639
+
640
+[[package]]
641
+name = "humantime-serde"
642
+version = "1.1.1"
643
+source = "registry+https://github.com/rust-lang/crates.io-index"
644
+checksum = "57a3db5ea5923d99402c94e9feb261dc5ee9b4efa158b0315f788cf549cc200c"
645
+dependencies = [
646
+ "humantime",
647
+ "serde",
648
+]
649
+
650
+[[package]]
651
+name = "hyper"
652
+version = "1.7.0"
653
+source = "registry+https://github.com/rust-lang/crates.io-index"
654
+checksum = "eb3aa54a13a0dfe7fbe3a59e0c76093041720fdc77b110cc0fc260fafb4dc51e"
655
+dependencies = [
656
+ "atomic-waker",
657
+ "bytes",
658
+ "futures-channel",
659
+ "futures-core",
660
+ "h2",
661
+ "http",
662
+ "http-body",
663
+ "httparse",
664
+ "httpdate",
665
+ "itoa",
666
+ "pin-project-lite",
667
+ "pin-utils",
668
+ "smallvec",
669
+ "tokio",
670
+ "want",
671
+]
672
+
673
+[[package]]
674
+name = "hyper-timeout"
675
+version = "0.5.2"
676
+source = "registry+https://github.com/rust-lang/crates.io-index"
677
+checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0"
678
+dependencies = [
679
+ "hyper",
680
+ "hyper-util",
681
+ "pin-project-lite",
682
+ "tokio",
683
+ "tower-service",
684
+]
685
+
686
+[[package]]
687
+name = "hyper-util"
688
+version = "0.1.17"
689
+source = "registry+https://github.com/rust-lang/crates.io-index"
690
+checksum = "3c6995591a8f1380fcb4ba966a252a4b29188d51d2b89e3a252f5305be65aea8"
691
+dependencies = [
692
+ "bytes",
693
+ "futures-channel",
694
+ "futures-core",
695
+ "futures-util",
696
+ "http",
697
+ "http-body",
698
+ "hyper",
699
+ "libc",
700
+ "pin-project-lite",
701
+ "socket2 0.6.0",
702
+ "tokio",
703
+ "tower-service",
704
+ "tracing",
705
+]
706
+
707
+[[package]]
708
+name = "iana-time-zone"
709
+version = "0.1.64"
710
+source = "registry+https://github.com/rust-lang/crates.io-index"
711
+checksum = "33e57f83510bb73707521ebaffa789ec8caf86f9657cad665b092b581d40e9fb"
712
+dependencies = [
713
+ "android_system_properties",
714
+ "core-foundation-sys",
715
+ "iana-time-zone-haiku",
716
+ "js-sys",
717
+ "log",
718
+ "wasm-bindgen",
719
+ "windows-core",
720
+]
721
+
722
+[[package]]
723
+name = "iana-time-zone-haiku"
724
+version = "0.1.2"
725
+source = "registry+https://github.com/rust-lang/crates.io-index"
726
+checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f"
727
+dependencies = [
728
+ "cc",
729
+]
730
+
731
+[[package]]
732
+name = "indexmap"
733
+version = "2.11.4"
734
+source = "registry+https://github.com/rust-lang/crates.io-index"
735
+checksum = "4b0f83760fb341a774ed326568e19f5a863af4a952def8c39f9ab92fd95b88e5"
736
+dependencies = [
737
+ "equivalent",
738
+ "hashbrown",
739
+]
740
+
741
+[[package]]
742
+name = "io-uring"
743
+version = "0.7.10"
744
+source = "registry+https://github.com/rust-lang/crates.io-index"
745
+checksum = "046fa2d4d00aea763528b4950358d0ead425372445dc8ff86312b3c69ff7727b"
746
+dependencies = [
747
+ "bitflags",
748
+ "cfg-if",
749
+ "libc",
750
+]
751
+
752
+[[package]]
753
+name = "is_terminal_polyfill"
754
+version = "1.70.1"
755
+source = "registry+https://github.com/rust-lang/crates.io-index"
756
+checksum = "7943c866cc5cd64cbc25b2e01621d07fa8eb2a1a23160ee81ce38704e97b8ecf"
757
+
758
+[[package]]
759
+name = "itertools"
760
+version = "0.14.0"
761
+source = "registry+https://github.com/rust-lang/crates.io-index"
762
+checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285"
763
+dependencies = [
764
+ "either",
765
+]
766
+
767
+[[package]]
768
+name = "itoa"
769
+version = "1.0.15"
770
+source = "registry+https://github.com/rust-lang/crates.io-index"
771
+checksum = "4a5f13b858c8d314ee3e8f639011f7ccefe71f97f96e50151fb991f267928e2c"
772
+
773
+[[package]]
774
+name = "journal_file"
775
+version = "0.1.0"
776
+dependencies = [
777
+ "error",
778
+ "hex",
779
+ "memmap2",
780
+ "rand",
781
+ "ruzstd",
782
+ "siphasher",
783
+ "tempfile",
784
+ "twox-hash",
785
+ "window_manager",
786
+ "zerocopy 0.9.0-alpha.0",
787
+]
788
+
789
+[[package]]
790
+name = "journal_log"
791
+version = "0.1.0"
792
+dependencies = [
793
+ "error",
794
+ "journal_file",
795
+ "memmap2",
796
+ "tempfile",
797
+ "uuid",
798
+]
799
+
800
+[[package]]
801
+name = "journal_reader_ffi"
802
+version = "0.1.0"
803
+dependencies = [
804
+ "cbindgen",
805
+ "error",
806
+ "journal_file",
807
+ "memmap2",
808
+ "serde_json",
809
+ "sigbus",
810
+]
811
+
812
+[[package]]
813
+name = "js-sys"
814
+version = "0.3.81"
815
+source = "registry+https://github.com/rust-lang/crates.io-index"
816
+checksum = "ec48937a97411dcb524a265206ccd4c90bb711fca92b2792c407f268825b9305"
817
+dependencies = [
818
+ "once_cell",
819
+ "wasm-bindgen",
820
+]
821
+
822
+[[package]]
823
+name = "lazy_static"
824
+version = "1.5.0"
825
+source = "registry+https://github.com/rust-lang/crates.io-index"
826
+checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
827
+
828
+[[package]]
829
+name = "libc"
830
+version = "0.2.176"
831
+source = "registry+https://github.com/rust-lang/crates.io-index"
832
+checksum = "58f929b4d672ea937a23a1ab494143d968337a5f47e56d0815df1e0890ddf174"
833
+
834
+[[package]]
835
+name = "linux-raw-sys"
836
+version = "0.11.0"
837
+source = "registry+https://github.com/rust-lang/crates.io-index"
838
+checksum = "df1d3c3b53da64cf5760482273a98e575c651a67eec7f77df96b5b642de8f039"
839
+
840
+[[package]]
841
+name = "log"
842
+version = "0.4.28"
843
+source = "registry+https://github.com/rust-lang/crates.io-index"
844
+checksum = "34080505efa8e45a4b816c349525ebe327ceaa8559756f0356cba97ef3bf7432"
845
+
846
+[[package]]
847
+name = "matchit"
848
+version = "0.8.4"
849
+source = "registry+https://github.com/rust-lang/crates.io-index"
850
+checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3"
851
+
852
+[[package]]
853
+name = "memchr"
854
+version = "2.7.6"
855
+source = "registry+https://github.com/rust-lang/crates.io-index"
856
+checksum = "f52b00d39961fc5b2736ea853c9cc86238e165017a493d1d5c8eac6bdc4cc273"
857
+
858
+[[package]]
859
+name = "memmap2"
860
+version = "0.9.8"
861
+source = "registry+https://github.com/rust-lang/crates.io-index"
862
+checksum = "843a98750cd611cc2965a8213b53b43e715f13c37a9e096c6408e69990961db7"
863
+dependencies = [
864
+ "libc",
865
+]
866
+
867
+[[package]]
868
+name = "mime"
869
+version = "0.3.17"
870
+source = "registry+https://github.com/rust-lang/crates.io-index"
871
+checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
872
+
873
+[[package]]
874
+name = "miniz_oxide"
875
+version = "0.8.9"
876
+source = "registry+https://github.com/rust-lang/crates.io-index"
877
+checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316"
878
+dependencies = [
879
+ "adler2",
880
+]
881
+
882
+[[package]]
883
+name = "mio"
884
+version = "1.0.4"
885
+source = "registry+https://github.com/rust-lang/crates.io-index"
886
+checksum = "78bed444cc8a2160f01cbcf811ef18cac863ad68ae8ca62092e8db51d51c761c"
887
+dependencies = [
888
+ "libc",
889
+ "wasi 0.11.1+wasi-snapshot-preview1",
890
+ "windows-sys 0.59.0",
891
+]
892
+
893
+[[package]]
894
+name = "multimap"
895
+version = "0.10.1"
896
+source = "registry+https://github.com/rust-lang/crates.io-index"
897
+checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084"
898
+
899
+[[package]]
900
+name = "nu-ansi-term"
901
+version = "0.50.1"
902
+source = "registry+https://github.com/rust-lang/crates.io-index"
903
+checksum = "d4a28e057d01f97e61255210fcff094d74ed0466038633e95017f5beb68e4399"
904
+dependencies = [
905
+ "windows-sys 0.52.0",
906
+]
907
+
908
+[[package]]
909
+name = "num-traits"
910
+version = "0.2.19"
911
+source = "registry+https://github.com/rust-lang/crates.io-index"
912
+checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
913
+dependencies = [
914
+ "autocfg",
915
+]
916
+
917
+[[package]]
918
+name = "object"
919
+version = "0.37.3"
920
+source = "registry+https://github.com/rust-lang/crates.io-index"
921
+checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe"
922
+dependencies = [
923
+ "memchr",
924
+]
925
+
926
+[[package]]
927
+name = "once_cell"
928
+version = "1.21.3"
929
+source = "registry+https://github.com/rust-lang/crates.io-index"
930
+checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d"
931
+
932
+[[package]]
933
+name = "once_cell_polyfill"
934
+version = "1.70.1"
935
+source = "registry+https://github.com/rust-lang/crates.io-index"
936
+checksum = "a4895175b425cb1f87721b59f0f286c2092bd4af812243672510e1ac53e2e0ad"
937
+
938
+[[package]]
939
+name = "opentelemetry"
940
+version = "0.30.0"
941
+source = "registry+https://github.com/rust-lang/crates.io-index"
942
+checksum = "aaf416e4cb72756655126f7dd7bb0af49c674f4c1b9903e80c009e0c37e552e6"
943
+dependencies = [
944
+ "futures-core",
945
+ "futures-sink",
946
+ "js-sys",
947
+ "pin-project-lite",
948
+ "thiserror",
949
+ "tracing",
950
+]
951
+
952
+[[package]]
953
+name = "opentelemetry-proto"
954
+version = "0.30.0"
955
+source = "registry+https://github.com/rust-lang/crates.io-index"
956
+checksum = "2e046fd7660710fe5a05e8748e70d9058dc15c94ba914e7c4faa7c728f0e8ddc"
957
+dependencies = [
958
+ "base64 0.22.1",
959
+ "hex",
960
+ "opentelemetry",
961
+ "opentelemetry_sdk",
962
+ "prost",
963
+ "serde",
964
+ "tonic",
965
+]
966
+
967
+[[package]]
968
+name = "opentelemetry_sdk"
969
+version = "0.30.0"
970
+source = "registry+https://github.com/rust-lang/crates.io-index"
971
+checksum = "11f644aa9e5e31d11896e024305d7e3c98a88884d9f8919dbf37a9991bc47a4b"
972
+dependencies = [
973
+ "futures-channel",
974
+ "futures-executor",
975
+ "futures-util",
976
+ "opentelemetry",
977
+ "percent-encoding",
978
+ "rand",
979
+ "serde_json",
980
+ "thiserror",
981
+]
982
+
983
+[[package]]
984
+name = "otel-plugin"
985
+version = "0.1.0"
986
+dependencies = [
987
+ "anyhow",
988
+ "atty",
989
+ "base64 0.21.7",
990
+ "bytesize",
991
+ "bytesize-serde",
992
+ "clap",
993
+ "flatten-serde-json",
994
+ "flatten_otel",
995
+ "humantime",
996
+ "humantime-serde",
997
+ "journal_file",
998
+ "journal_log",
999
+ "memmap2",
1000
+ "opentelemetry-proto",
1001
+ "prost",
1002
+ "regex",
1003
+ "serde",
1004
+ "serde_json",
1005
+ "serde_regex",
1006
+ "serde_yaml",
1007
+ "tokio",
1008
+ "tonic",
1009
+ "uuid",
1010
+]
1011
+
1012
+[[package]]
1013
+name = "percent-encoding"
1014
+version = "2.3.2"
1015
+source = "registry+https://github.com/rust-lang/crates.io-index"
1016
+checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
1017
+
1018
+[[package]]
1019
+name = "petgraph"
1020
+version = "0.7.1"
1021
+source = "registry+https://github.com/rust-lang/crates.io-index"
1022
+checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772"
1023
+dependencies = [
1024
+ "fixedbitset",
1025
+ "indexmap",
1026
+]
1027
+
1028
+[[package]]
1029
+name = "pin-project"
1030
+version = "1.1.10"
1031
+source = "registry+https://github.com/rust-lang/crates.io-index"
1032
+checksum = "677f1add503faace112b9f1373e43e9e054bfdd22ff1a63c1bc485eaec6a6a8a"
1033
+dependencies = [
1034
+ "pin-project-internal",
1035
+]
1036
+
1037
+[[package]]
1038
+name = "pin-project-internal"
1039
+version = "1.1.10"
1040
+source = "registry+https://github.com/rust-lang/crates.io-index"
1041
+checksum = "6e918e4ff8c4549eb882f14b3a4bc8c8bc93de829416eacf579f1207a8fbf861"
1042
+dependencies = [
1043
+ "proc-macro2",
1044
+ "quote",
1045
+ "syn",
1046
+]
1047
+
1048
+[[package]]
1049
+name = "pin-project-lite"
1050
+version = "0.2.16"
1051
+source = "registry+https://github.com/rust-lang/crates.io-index"
1052
+checksum = "3b3cff922bd51709b605d9ead9aa71031d81447142d828eb4a6eba76fe619f9b"
1053
+
1054
+[[package]]
1055
+name = "pin-utils"
1056
+version = "0.1.0"
1057
+source = "registry+https://github.com/rust-lang/crates.io-index"
1058
+checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
1059
+
1060
+[[package]]
1061
+name = "ppv-lite86"
1062
+version = "0.2.21"
1063
+source = "registry+https://github.com/rust-lang/crates.io-index"
1064
+checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9"
1065
+dependencies = [
1066
+ "zerocopy 0.8.27",
1067
+]
1068
+
1069
+[[package]]
1070
+name = "prettyplease"
1071
+version = "0.2.37"
1072
+source = "registry+https://github.com/rust-lang/crates.io-index"
1073
+checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"
1074
+dependencies = [
1075
+ "proc-macro2",
1076
+ "syn",
1077
+]
1078
+
1079
+[[package]]
1080
+name = "proc-macro2"
1081
+version = "1.0.101"
1082
+source = "registry+https://github.com/rust-lang/crates.io-index"
1083
+checksum = "89ae43fd86e4158d6db51ad8e2b80f313af9cc74f5c0e03ccb87de09998732de"
1084
+dependencies = [
1085
+ "unicode-ident",
1086
+]
1087
+
1088
+[[package]]
1089
+name = "prost"
1090
+version = "0.13.5"
1091
+source = "registry+https://github.com/rust-lang/crates.io-index"
1092
+checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5"
1093
+dependencies = [
1094
+ "bytes",
1095
+ "prost-derive",
1096
+]
1097
+
1098
+[[package]]
1099
+name = "prost-build"
1100
+version = "0.13.5"
1101
+source = "registry+https://github.com/rust-lang/crates.io-index"
1102
+checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
1103
+dependencies = [
1104
+ "heck 0.5.0",
1105
+ "itertools",
1106
+ "log",
1107
+ "multimap",
1108
+ "once_cell",
1109
+ "petgraph",
1110
+ "prettyplease",
1111
+ "prost",
1112
+ "prost-types",
1113
+ "regex",
1114
+ "syn",
1115
+ "tempfile",
1116
+]
1117
+
1118
+[[package]]
1119
+name = "prost-derive"
1120
+version = "0.13.5"
1121
+source = "registry+https://github.com/rust-lang/crates.io-index"
1122
+checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
1123
+dependencies = [
1124
+ "anyhow",
1125
+ "itertools",
1126
+ "proc-macro2",
1127
+ "quote",
1128
+ "syn",
1129
+]
1130
+
1131
+[[package]]
1132
+name = "prost-types"
1133
+version = "0.13.5"
1134
+source = "registry+https://github.com/rust-lang/crates.io-index"
1135
+checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16"
1136
+dependencies = [
1137
+ "prost",
1138
+]
1139
+
1140
+[[package]]
1141
+name = "quote"
1142
+version = "1.0.41"
1143
+source = "registry+https://github.com/rust-lang/crates.io-index"
1144
+checksum = "ce25767e7b499d1b604768e7cde645d14cc8584231ea6b295e9c9eb22c02e1d1"
1145
+dependencies = [
1146
+ "proc-macro2",
1147
+]
1148
+
1149
+[[package]]
1150
+name = "r-efi"
1151
+version = "5.3.0"
1152
+source = "registry+https://github.com/rust-lang/crates.io-index"
1153
+checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f"
1154
+
1155
+[[package]]
1156
+name = "rand"
1157
+version = "0.9.2"
1158
+source = "registry+https://github.com/rust-lang/crates.io-index"
1159
+checksum = "6db2770f06117d490610c7488547d543617b21bfa07796d7a12f6f1bd53850d1"
1160
+dependencies = [
1161
+ "rand_chacha",
1162
+ "rand_core",
1163
+]
1164
+
1165
+[[package]]
1166
+name = "rand_chacha"
1167
+version = "0.9.0"
1168
+source = "registry+https://github.com/rust-lang/crates.io-index"
1169
+checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
1170
+dependencies = [
1171
+ "ppv-lite86",
1172
+ "rand_core",
1173
+]
1174
+
1175
+[[package]]
1176
+name = "rand_core"
1177
+version = "0.9.3"
1178
+source = "registry+https://github.com/rust-lang/crates.io-index"
1179
+checksum = "99d9a13982dcf210057a8a78572b2217b667c3beacbf3a0d8b454f6f82837d38"
1180
+dependencies = [
1181
+ "getrandom 0.3.3",
1182
+]
1183
+
1184
+[[package]]
1185
+name = "regex"
1186
+version = "1.11.3"
1187
+source = "registry+https://github.com/rust-lang/crates.io-index"
1188
+checksum = "8b5288124840bee7b386bc413c487869b360b2b4ec421ea56425128692f2a82c"
1189
+dependencies = [
1190
+ "aho-corasick",
1191
+ "memchr",
1192
+ "regex-automata",
1193
+ "regex-syntax",
1194
+]
1195
+
1196
+[[package]]
1197
+name = "regex-automata"
1198
+version = "0.4.11"
1199
+source = "registry+https://github.com/rust-lang/crates.io-index"
1200
+checksum = "833eb9ce86d40ef33cb1306d8accf7bc8ec2bfea4355cbdebb3df68b40925cad"
1201
+dependencies = [
1202
+ "aho-corasick",
1203
+ "memchr",
1204
+ "regex-syntax",
1205
+]
1206
+
1207
+[[package]]
1208
+name = "regex-syntax"
1209
+version = "0.8.6"
1210
+source = "registry+https://github.com/rust-lang/crates.io-index"
1211
+checksum = "caf4aa5b0f434c91fe5c7f1ecb6a5ece2130b02ad2a590589dda5146df959001"
1212
+
1213
+[[package]]
1214
+name = "ring"
1215
+version = "0.17.14"
1216
+source = "registry+https://github.com/rust-lang/crates.io-index"
1217
+checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
1218
+dependencies = [
1219
+ "cc",
1220
+ "cfg-if",
1221
+ "getrandom 0.2.16",
1222
+ "libc",
1223
+ "untrusted",
1224
+ "windows-sys 0.52.0",
1225
+]
1226
+
1227
+[[package]]
1228
+name = "rustc-demangle"
1229
+version = "0.1.26"
1230
+source = "registry+https://github.com/rust-lang/crates.io-index"
1231
+checksum = "56f7d92ca342cea22a06f2121d944b4fd82af56988c270852495420f961d4ace"
1232
+
1233
+[[package]]
1234
+name = "rustix"
1235
+version = "1.1.2"
1236
+source = "registry+https://github.com/rust-lang/crates.io-index"
1237
+checksum = "cd15f8a2c5551a84d56efdc1cd049089e409ac19a3072d5037a17fd70719ff3e"
1238
+dependencies = [
1239
+ "bitflags",
1240
+ "errno",
1241
+ "libc",
1242
+ "linux-raw-sys",
1243
+ "windows-sys 0.61.1",
1244
+]
1245
+
1246
+[[package]]
1247
+name = "rustls"
1248
+version = "0.23.32"
1249
+source = "registry+https://github.com/rust-lang/crates.io-index"
1250
+checksum = "cd3c25631629d034ce7cd9940adc9d45762d46de2b0f57193c4443b92c6d4d40"
1251
+dependencies = [
1252
+ "log",
1253
+ "once_cell",
1254
+ "ring",
1255
+ "rustls-pki-types",
1256
+ "rustls-webpki",
1257
+ "subtle",
1258
+ "zeroize",
1259
+]
1260
+
1261
+[[package]]
1262
+name = "rustls-pki-types"
1263
+version = "1.12.0"
1264
+source = "registry+https://github.com/rust-lang/crates.io-index"
1265
+checksum = "229a4a4c221013e7e1f1a043678c5cc39fe5171437c88fb47151a21e6f5b5c79"
1266
+dependencies = [
1267
+ "zeroize",
1268
+]
1269
+
1270
+[[package]]
1271
+name = "rustls-webpki"
1272
+version = "0.103.6"
1273
+source = "registry+https://github.com/rust-lang/crates.io-index"
1274
+checksum = "8572f3c2cb9934231157b45499fc41e1f58c589fdfb81a844ba873265e80f8eb"
1275
+dependencies = [
1276
+ "ring",
1277
+ "rustls-pki-types",
1278
+ "untrusted",
1279
+]
1280
+
1281
+[[package]]
1282
+name = "rustversion"
1283
+version = "1.0.22"
1284
+source = "registry+https://github.com/rust-lang/crates.io-index"
1285
+checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
1286
+
1287
+[[package]]
1288
+name = "ruzstd"
1289
+version = "0.8.1"
1290
+source = "registry+https://github.com/rust-lang/crates.io-index"
1291
+checksum = "3640bec8aad418d7d03c72ea2de10d5c646a598f9883c7babc160d91e3c1b26c"
1292
+dependencies = [
1293
+ "twox-hash",
1294
+]
1295
+
1296
+[[package]]
1297
+name = "ryu"
1298
+version = "1.0.20"
1299
+source = "registry+https://github.com/rust-lang/crates.io-index"
1300
+checksum = "28d3b2b1366ec20994f1fd18c3c594f05c5dd4bc44d8bb0c1c632c8d6829481f"
1301
+
1302
+[[package]]
1303
+name = "serde"
1304
+version = "1.0.228"
1305
+source = "registry+https://github.com/rust-lang/crates.io-index"
1306
+checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
1307
+dependencies = [
1308
+ "serde_core",
1309
+ "serde_derive",
1310
+]
1311
+
1312
+[[package]]
1313
+name = "serde_core"
1314
+version = "1.0.228"
1315
+source = "registry+https://github.com/rust-lang/crates.io-index"
1316
+checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
1317
+dependencies = [
1318
+ "serde_derive",
1319
+]
1320
+
1321
+[[package]]
1322
+name = "serde_derive"
1323
+version = "1.0.228"
1324
+source = "registry+https://github.com/rust-lang/crates.io-index"
1325
+checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
1326
+dependencies = [
1327
+ "proc-macro2",
1328
+ "quote",
1329
+ "syn",
1330
+]
1331
+
1332
+[[package]]
1333
+name = "serde_json"
1334
+version = "1.0.145"
1335
+source = "registry+https://github.com/rust-lang/crates.io-index"
1336
+checksum = "402a6f66d8c709116cf22f558eab210f5a50187f702eb4d7e5ef38d9a7f1c79c"
1337
+dependencies = [
1338
+ "indexmap",
1339
+ "itoa",
1340
+ "memchr",
1341
+ "ryu",
1342
+ "serde",
1343
+ "serde_core",
1344
+]
1345
+
1346
+[[package]]
1347
+name = "serde_regex"
1348
+version = "1.1.0"
1349
+source = "registry+https://github.com/rust-lang/crates.io-index"
1350
+checksum = "a8136f1a4ea815d7eac4101cfd0b16dc0cb5e1fe1b8609dfd728058656b7badf"
1351
+dependencies = [
1352
+ "regex",
1353
+ "serde",
1354
+]
1355
+
1356
+[[package]]
1357
+name = "serde_spanned"
1358
+version = "0.6.9"
1359
+source = "registry+https://github.com/rust-lang/crates.io-index"
1360
+checksum = "bf41e0cfaf7226dca15e8197172c295a782857fcb97fad1808a166870dee75a3"
1361
+dependencies = [
1362
+ "serde",
1363
+]
1364
+
1365
+[[package]]
1366
+name = "serde_yaml"
1367
+version = "0.9.34+deprecated"
1368
+source = "registry+https://github.com/rust-lang/crates.io-index"
1369
+checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47"
1370
+dependencies = [
1371
+ "indexmap",
1372
+ "itoa",
1373
+ "ryu",
1374
+ "serde",
1375
+ "unsafe-libyaml",
1376
+]
1377
+
1378
+[[package]]
1379
+name = "sharded-slab"
1380
+version = "0.1.7"
1381
+source = "registry+https://github.com/rust-lang/crates.io-index"
1382
+checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6"
1383
+dependencies = [
1384
+ "lazy_static",
1385
+]
1386
+
1387
+[[package]]
1388
+name = "shlex"
1389
+version = "1.3.0"
1390
+source = "registry+https://github.com/rust-lang/crates.io-index"
1391
+checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
1392
+
1393
+[[package]]
1394
+name = "sigbus"
1395
+version = "0.1.0"
1396
+dependencies = [
1397
+ "error",
1398
+ "libc",
1399
+]
1400
+
1401
+[[package]]
1402
+name = "signal-hook-registry"
1403
+version = "1.4.6"
1404
+source = "registry+https://github.com/rust-lang/crates.io-index"
1405
+checksum = "b2a4719bff48cee6b39d12c020eeb490953ad2443b7055bd0b21fca26bd8c28b"
1406
+dependencies = [
1407
+ "libc",
1408
+]
1409
+
1410
+[[package]]
1411
+name = "siphasher"
1412
+version = "1.0.1"
1413
+source = "registry+https://github.com/rust-lang/crates.io-index"
1414
+checksum = "56199f7ddabf13fe5074ce809e7d3f42b42ae711800501b5b16ea82ad029c39d"
1415
+
1416
+[[package]]
1417
+name = "slab"
1418
+version = "0.4.11"
1419
+source = "registry+https://github.com/rust-lang/crates.io-index"
1420
+checksum = "7a2ae44ef20feb57a68b23d846850f861394c2e02dc425a50098ae8c90267589"
1421
+
1422
+[[package]]
1423
+name = "smallvec"
1424
+version = "1.15.1"
1425
+source = "registry+https://github.com/rust-lang/crates.io-index"
1426
+checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03"
1427
+
1428
+[[package]]
1429
+name = "socket2"
1430
+version = "0.5.10"
1431
+source = "registry+https://github.com/rust-lang/crates.io-index"
1432
+checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678"
1433
+dependencies = [
1434
+ "libc",
1435
+ "windows-sys 0.52.0",
1436
+]
1437
+
1438
+[[package]]
1439
+name = "socket2"
1440
+version = "0.6.0"
1441
+source = "registry+https://github.com/rust-lang/crates.io-index"
1442
+checksum = "233504af464074f9d066d7b5416c5f9b894a5862a6506e306f7b816cdd6f1807"
1443
+dependencies = [
1444
+ "libc",
1445
+ "windows-sys 0.59.0",
1446
+]
1447
+
1448
+[[package]]
1449
+name = "static_assertions"
1450
+version = "1.1.0"
1451
+source = "registry+https://github.com/rust-lang/crates.io-index"
1452
+checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
1453
+
1454
+[[package]]
1455
+name = "strsim"
1456
+version = "0.11.1"
1457
+source = "registry+https://github.com/rust-lang/crates.io-index"
1458
+checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
1459
+
1460
+[[package]]
1461
+name = "subtle"
1462
+version = "2.6.1"
1463
+source = "registry+https://github.com/rust-lang/crates.io-index"
1464
+checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
1465
+
1466
+[[package]]
1467
+name = "syn"
1468
+version = "2.0.106"
1469
+source = "registry+https://github.com/rust-lang/crates.io-index"
1470
+checksum = "ede7c438028d4436d71104916910f5bb611972c5cfd7f89b8300a8186e6fada6"
1471
+dependencies = [
1472
+ "proc-macro2",
1473
+ "quote",
1474
+ "unicode-ident",
1475
+]
1476
+
1477
+[[package]]
1478
+name = "sync_wrapper"
1479
+version = "1.0.2"
1480
+source = "registry+https://github.com/rust-lang/crates.io-index"
1481
+checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
1482
+
1483
+[[package]]
1484
+name = "tempfile"
1485
+version = "3.23.0"
1486
+source = "registry+https://github.com/rust-lang/crates.io-index"
1487
+checksum = "2d31c77bdf42a745371d260a26ca7163f1e0924b64afa0b688e61b5a9fa02f16"
1488
+dependencies = [
1489
+ "fastrand",
1490
+ "getrandom 0.3.3",
1491
+ "once_cell",
1492
+ "rustix",
1493
+ "windows-sys 0.61.1",
1494
+]
1495
+
1496
+[[package]]
1497
+name = "thiserror"
1498
+version = "2.0.17"
1499
+source = "registry+https://github.com/rust-lang/crates.io-index"
1500
+checksum = "f63587ca0f12b72a0600bcba1d40081f830876000bb46dd2337a3051618f4fc8"
1501
+dependencies = [
1502
+ "thiserror-impl",
1503
+]
1504
+
1505
+[[package]]
1506
+name = "thiserror-impl"
1507
+version = "2.0.17"
1508
+source = "registry+https://github.com/rust-lang/crates.io-index"
1509
+checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913"
1510
+dependencies = [
1511
+ "proc-macro2",
1512
+ "quote",
1513
+ "syn",
1514
+]
1515
+
1516
+[[package]]
1517
+name = "thread_local"
1518
+version = "1.1.9"
1519
+source = "registry+https://github.com/rust-lang/crates.io-index"
1520
+checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185"
1521
+dependencies = [
1522
+ "cfg-if",
1523
+]
1524
+
1525
+[[package]]
1526
+name = "tokio"
1527
+version = "1.47.1"
1528
+source = "registry+https://github.com/rust-lang/crates.io-index"
1529
+checksum = "89e49afdadebb872d3145a5638b59eb0691ea23e46ca484037cfab3b76b95038"
1530
+dependencies = [
1531
+ "backtrace",
1532
+ "bytes",
1533
+ "io-uring",
1534
+ "libc",
1535
+ "mio",
1536
+ "pin-project-lite",
1537
+ "signal-hook-registry",
1538
+ "slab",
1539
+ "socket2 0.6.0",
1540
+ "tokio-macros",
1541
+ "windows-sys 0.59.0",
1542
+]
1543
+
1544
+[[package]]
1545
+name = "tokio-macros"
1546
+version = "2.5.0"
1547
+source = "registry+https://github.com/rust-lang/crates.io-index"
1548
+checksum = "6e06d43f1345a3bcd39f6a56dbb7dcab2ba47e68e8ac134855e7e2bdbaf8cab8"
1549
+dependencies = [
1550
+ "proc-macro2",
1551
+ "quote",
1552
+ "syn",
1553
+]
1554
+
1555
+[[package]]
1556
+name = "tokio-rustls"
1557
+version = "0.26.4"
1558
+source = "registry+https://github.com/rust-lang/crates.io-index"
1559
+checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61"
1560
+dependencies = [
1561
+ "rustls",
1562
+ "tokio",
1563
+]
1564
+
1565
+[[package]]
1566
+name = "tokio-stream"
1567
+version = "0.1.17"
1568
+source = "registry+https://github.com/rust-lang/crates.io-index"
1569
+checksum = "eca58d7bba4a75707817a2c44174253f9236b2d5fbd055602e9d5c07c139a047"
1570
+dependencies = [
1571
+ "futures-core",
1572
+ "pin-project-lite",
1573
+ "tokio",
1574
+]
1575
+
1576
+[[package]]
1577
+name = "tokio-util"
1578
+version = "0.7.16"
1579
+source = "registry+https://github.com/rust-lang/crates.io-index"
1580
+checksum = "14307c986784f72ef81c89db7d9e28d6ac26d16213b109ea501696195e6e3ce5"
1581
+dependencies = [
1582
+ "bytes",
1583
+ "futures-core",
1584
+ "futures-sink",
1585
+ "pin-project-lite",
1586
+ "tokio",
1587
+]
1588
+
1589
+[[package]]
1590
+name = "toml"
1591
+version = "0.8.23"
1592
+source = "registry+https://github.com/rust-lang/crates.io-index"
1593
+checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362"
1594
+dependencies = [
1595
+ "serde",
1596
+ "serde_spanned",
1597
+ "toml_datetime",
1598
+ "toml_edit",
1599
+]
1600
+
1601
+[[package]]
1602
+name = "toml_datetime"
1603
+version = "0.6.11"
1604
+source = "registry+https://github.com/rust-lang/crates.io-index"
1605
+checksum = "22cddaf88f4fbc13c51aebbf5f8eceb5c7c5a9da2ac40a13519eb5b0a0e8f11c"
1606
+dependencies = [
1607
+ "serde",
1608
+]
1609
+
1610
+[[package]]
1611
+name = "toml_edit"
1612
+version = "0.22.27"
1613
+source = "registry+https://github.com/rust-lang/crates.io-index"
1614
+checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
1615
+dependencies = [
1616
+ "indexmap",
1617
+ "serde",
1618
+ "serde_spanned",
1619
+ "toml_datetime",
1620
+ "toml_write",
1621
+ "winnow",
1622
+]
1623
+
1624
+[[package]]
1625
+name = "toml_write"
1626
+version = "0.1.2"
1627
+source = "registry+https://github.com/rust-lang/crates.io-index"
1628
+checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801"
1629
+
1630
+[[package]]
1631
+name = "tonic"
1632
+version = "0.13.1"
1633
+source = "registry+https://github.com/rust-lang/crates.io-index"
1634
+checksum = "7e581ba15a835f4d9ea06c55ab1bd4dce26fc53752c69a04aac00703bfb49ba9"
1635
+dependencies = [
1636
+ "async-trait",
1637
+ "axum",
1638
+ "base64 0.22.1",
1639
+ "bytes",
1640
+ "flate2",
1641
+ "h2",
1642
+ "http",
1643
+ "http-body",
1644
+ "http-body-util",
1645
+ "hyper",
1646
+ "hyper-timeout",
1647
+ "hyper-util",
1648
+ "percent-encoding",
1649
+ "pin-project",
1650
+ "prost",
1651
+ "socket2 0.5.10",
1652
+ "tokio",
1653
+ "tokio-rustls",
1654
+ "tokio-stream",
1655
+ "tower",
1656
+ "tower-layer",
1657
+ "tower-service",
1658
+ "tracing",
1659
+]
1660
+
1661
+[[package]]
1662
+name = "tonic-build"
1663
+version = "0.13.1"
1664
+source = "registry+https://github.com/rust-lang/crates.io-index"
1665
+checksum = "eac6f67be712d12f0b41328db3137e0d0757645d8904b4cb7d51cd9c2279e847"
1666
+dependencies = [
1667
+ "prettyplease",
1668
+ "proc-macro2",
1669
+ "prost-build",
1670
+ "prost-types",
1671
+ "quote",
1672
+ "syn",
1673
+]
1674
+
1675
+[[package]]
1676
+name = "tower"
1677
+version = "0.5.2"
1678
+source = "registry+https://github.com/rust-lang/crates.io-index"
1679
+checksum = "d039ad9159c98b70ecfd540b2573b97f7f52c3e8d9f8ad57a24b916a536975f9"
1680
+dependencies = [
1681
+ "futures-core",
1682
+ "futures-util",
1683
+ "indexmap",
1684
+ "pin-project-lite",
1685
+ "slab",
1686
+ "sync_wrapper",
1687
+ "tokio",
1688
+ "tokio-util",
1689
+ "tower-layer",
1690
+ "tower-service",
1691
+ "tracing",
1692
+]
1693
+
1694
+[[package]]
1695
+name = "tower-layer"
1696
+version = "0.3.3"
1697
+source = "registry+https://github.com/rust-lang/crates.io-index"
1698
+checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e"
1699
+
1700
+[[package]]
1701
+name = "tower-service"
1702
+version = "0.3.3"
1703
+source = "registry+https://github.com/rust-lang/crates.io-index"
1704
+checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3"
1705
+
1706
+[[package]]
1707
+name = "tracing"
1708
+version = "0.1.41"
1709
+source = "registry+https://github.com/rust-lang/crates.io-index"
1710
+checksum = "784e0ac535deb450455cbfa28a6f0df145ea1bb7ae51b821cf5e7927fdcfbdd0"
1711
+dependencies = [
1712
+ "pin-project-lite",
1713
+ "tracing-attributes",
1714
+ "tracing-core",
1715
+]
1716
+
1717
+[[package]]
1718
+name = "tracing-attributes"
1719
+version = "0.1.30"
1720
+source = "registry+https://github.com/rust-lang/crates.io-index"
1721
+checksum = "81383ab64e72a7a8b8e13130c49e3dab29def6d0c7d76a03087b3cf71c5c6903"
1722
+dependencies = [
1723
+ "proc-macro2",
1724
+ "quote",
1725
+ "syn",
1726
+]
1727
+
1728
+[[package]]
1729
+name = "tracing-core"
1730
+version = "0.1.34"
1731
+source = "registry+https://github.com/rust-lang/crates.io-index"
1732
+checksum = "b9d12581f227e93f094d3af2ae690a574abb8a2b9b7a96e7cfe9647b2b617678"
1733
+dependencies = [
1734
+ "once_cell",
1735
+ "valuable",
1736
+]
1737
+
1738
+[[package]]
1739
+name = "tracing-log"
1740
+version = "0.2.0"
1741
+source = "registry+https://github.com/rust-lang/crates.io-index"
1742
+checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3"
1743
+dependencies = [
1744
+ "log",
1745
+ "once_cell",
1746
+ "tracing-core",
1747
+]
1748
+
1749
+[[package]]
1750
+name = "tracing-subscriber"
1751
+version = "0.3.20"
1752
+source = "registry+https://github.com/rust-lang/crates.io-index"
1753
+checksum = "2054a14f5307d601f88daf0553e1cbf472acc4f2c51afab632431cdcd72124d5"
1754
+dependencies = [
1755
+ "nu-ansi-term",
1756
+ "sharded-slab",
1757
+ "smallvec",
1758
+ "thread_local",
1759
+ "tracing-core",
1760
+ "tracing-log",
1761
+]
1762
+
1763
+[[package]]
1764
+name = "try-lock"
1765
+version = "0.2.5"
1766
+source = "registry+https://github.com/rust-lang/crates.io-index"
1767
+checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
1768
+
1769
+[[package]]
1770
+name = "twox-hash"
1771
+version = "2.1.2"
1772
+source = "registry+https://github.com/rust-lang/crates.io-index"
1773
+checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c"
1774
+
1775
+[[package]]
1776
+name = "unicode-ident"
1777
+version = "1.0.19"
1778
+source = "registry+https://github.com/rust-lang/crates.io-index"
1779
+checksum = "f63a545481291138910575129486daeaf8ac54aee4387fe7906919f7830c7d9d"
1780
+
1781
+[[package]]
1782
+name = "unsafe-libyaml"
1783
+version = "0.2.11"
1784
+source = "registry+https://github.com/rust-lang/crates.io-index"
1785
+checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861"
1786
+
1787
+[[package]]
1788
+name = "untrusted"
1789
+version = "0.9.0"
1790
+source = "registry+https://github.com/rust-lang/crates.io-index"
1791
+checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
1792
+
1793
+[[package]]
1794
+name = "utf8parse"
1795
+version = "0.2.2"
1796
+source = "registry+https://github.com/rust-lang/crates.io-index"
1797
+checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
1798
+
1799
+[[package]]
1800
+name = "uuid"
1801
+version = "1.18.1"
1802
+source = "registry+https://github.com/rust-lang/crates.io-index"
1803
+checksum = "2f87b8aa10b915a06587d0dec516c282ff295b475d94abf425d62b57710070a2"
1804
+dependencies = [
1805
+ "getrandom 0.3.3",
1806
+ "js-sys",
1807
+ "wasm-bindgen",
1808
+]
1809
+
1810
+[[package]]
1811
+name = "valuable"
1812
+version = "0.1.1"
1813
+source = "registry+https://github.com/rust-lang/crates.io-index"
1814
+checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
1815
+
1816
+[[package]]
1817
+name = "want"
1818
+version = "0.3.1"
1819
+source = "registry+https://github.com/rust-lang/crates.io-index"
1820
+checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
1821
+dependencies = [
1822
+ "try-lock",
1823
+]
1824
+
1825
+[[package]]
1826
+name = "wasi"
1827
+version = "0.11.1+wasi-snapshot-preview1"
1828
+source = "registry+https://github.com/rust-lang/crates.io-index"
1829
+checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
1830
+
1831
+[[package]]
1832
+name = "wasi"
1833
+version = "0.14.7+wasi-0.2.4"
1834
+source = "registry+https://github.com/rust-lang/crates.io-index"
1835
+checksum = "883478de20367e224c0090af9cf5f9fa85bed63a95c1abf3afc5c083ebc06e8c"
1836
+dependencies = [
1837
+ "wasip2",
1838
+]
1839
+
1840
+[[package]]
1841
+name = "wasip2"
1842
+version = "1.0.1+wasi-0.2.4"
1843
+source = "registry+https://github.com/rust-lang/crates.io-index"
1844
+checksum = "0562428422c63773dad2c345a1882263bbf4d65cf3f42e90921f787ef5ad58e7"
1845
+dependencies = [
1846
+ "wit-bindgen",
1847
+]
1848
+
1849
+[[package]]
1850
+name = "wasm-bindgen"
1851
+version = "0.2.104"
1852
+source = "registry+https://github.com/rust-lang/crates.io-index"
1853
+checksum = "c1da10c01ae9f1ae40cbfac0bac3b1e724b320abfcf52229f80b547c0d250e2d"
1854
+dependencies = [
1855
+ "cfg-if",
1856
+ "once_cell",
1857
+ "rustversion",
1858
+ "wasm-bindgen-macro",
1859
+ "wasm-bindgen-shared",
1860
+]
1861
+
1862
+[[package]]
1863
+name = "wasm-bindgen-backend"
1864
+version = "0.2.104"
1865
+source = "registry+https://github.com/rust-lang/crates.io-index"
1866
+checksum = "671c9a5a66f49d8a47345ab942e2cb93c7d1d0339065d4f8139c486121b43b19"
1867
+dependencies = [
1868
+ "bumpalo",
1869
+ "log",
1870
+ "proc-macro2",
1871
+ "quote",
1872
+ "syn",
1873
+ "wasm-bindgen-shared",
1874
+]
1875
+
1876
+[[package]]
1877
+name = "wasm-bindgen-macro"
1878
+version = "0.2.104"
1879
+source = "registry+https://github.com/rust-lang/crates.io-index"
1880
+checksum = "7ca60477e4c59f5f2986c50191cd972e3a50d8a95603bc9434501cf156a9a119"
1881
+dependencies = [
1882
+ "quote",
1883
+ "wasm-bindgen-macro-support",
1884
+]
1885
+
1886
+[[package]]
1887
+name = "wasm-bindgen-macro-support"
1888
+version = "0.2.104"
1889
+source = "registry+https://github.com/rust-lang/crates.io-index"
1890
+checksum = "9f07d2f20d4da7b26400c9f4a0511e6e0345b040694e8a75bd41d578fa4421d7"
1891
+dependencies = [
1892
+ "proc-macro2",
1893
+ "quote",
1894
+ "syn",
1895
+ "wasm-bindgen-backend",
1896
+ "wasm-bindgen-shared",
1897
+]
1898
+
1899
+[[package]]
1900
+name = "wasm-bindgen-shared"
1901
+version = "0.2.104"
1902
+source = "registry+https://github.com/rust-lang/crates.io-index"
1903
+checksum = "bad67dc8b2a1a6e5448428adec4c3e84c43e561d8c9ee8a9e5aabeb193ec41d1"
1904
+dependencies = [
1905
+ "unicode-ident",
1906
+]
1907
+
1908
+[[package]]
1909
+name = "winapi"
1910
+version = "0.3.9"
1911
+source = "registry+https://github.com/rust-lang/crates.io-index"
1912
+checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
1913
+dependencies = [
1914
+ "winapi-i686-pc-windows-gnu",
1915
+ "winapi-x86_64-pc-windows-gnu",
1916
+]
1917
+
1918
+[[package]]
1919
+name = "winapi-i686-pc-windows-gnu"
1920
+version = "0.4.0"
1921
+source = "registry+https://github.com/rust-lang/crates.io-index"
1922
+checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
1923
+
1924
+[[package]]
1925
+name = "winapi-x86_64-pc-windows-gnu"
1926
+version = "0.4.0"
1927
+source = "registry+https://github.com/rust-lang/crates.io-index"
1928
+checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
1929
+
1930
+[[package]]
1931
+name = "window_manager"
1932
+version = "0.1.0"
1933
+dependencies = [
1934
+ "error",
1935
+ "memmap2",
1936
+]
1937
+
1938
+[[package]]
1939
+name = "windows-core"
1940
+version = "0.62.1"
1941
+source = "registry+https://github.com/rust-lang/crates.io-index"
1942
+checksum = "6844ee5416b285084d3d3fffd743b925a6c9385455f64f6d4fa3031c4c2749a9"
1943
+dependencies = [
1944
+ "windows-implement",
1945
+ "windows-interface",
1946
+ "windows-link",
1947
+ "windows-result",
1948
+ "windows-strings",
1949
+]
1950
+
1951
+[[package]]
1952
+name = "windows-implement"
1953
+version = "0.60.1"
1954
+source = "registry+https://github.com/rust-lang/crates.io-index"
1955
+checksum = "edb307e42a74fb6de9bf3a02d9712678b22399c87e6fa869d6dfcd8c1b7754e0"
1956
+dependencies = [
1957
+ "proc-macro2",
1958
+ "quote",
1959
+ "syn",
1960
+]
1961
+
1962
+[[package]]
1963
+name = "windows-interface"
1964
+version = "0.59.2"
1965
+source = "registry+https://github.com/rust-lang/crates.io-index"
1966
+checksum = "c0abd1ddbc6964ac14db11c7213d6532ef34bd9aa042c2e5935f59d7908b46a5"
1967
+dependencies = [
1968
+ "proc-macro2",
1969
+ "quote",
1970
+ "syn",
1971
+]
1972
+
1973
+[[package]]
1974
+name = "windows-link"
1975
+version = "0.2.0"
1976
+source = "registry+https://github.com/rust-lang/crates.io-index"
1977
+checksum = "45e46c0661abb7180e7b9c281db115305d49ca1709ab8242adf09666d2173c65"
1978
+
1979
+[[package]]
1980
+name = "windows-result"
1981
+version = "0.4.0"
1982
+source = "registry+https://github.com/rust-lang/crates.io-index"
1983
+checksum = "7084dcc306f89883455a206237404d3eaf961e5bd7e0f312f7c91f57eb44167f"
1984
+dependencies = [
1985
+ "windows-link",
1986
+]
1987
+
1988
+[[package]]
1989
+name = "windows-strings"
1990
+version = "0.5.0"
1991
+source = "registry+https://github.com/rust-lang/crates.io-index"
1992
+checksum = "7218c655a553b0bed4426cf54b20d7ba363ef543b52d515b3e48d7fd55318dda"
1993
+dependencies = [
1994
+ "windows-link",
1995
+]
1996
+
1997
+[[package]]
1998
+name = "windows-sys"
1999
+version = "0.52.0"
2000
+source = "registry+https://github.com/rust-lang/crates.io-index"
2001
+checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
2002
+dependencies = [
2003
+ "windows-targets 0.52.6",
2004
+]
2005
+
2006
+[[package]]
2007
+name = "windows-sys"
2008
+version = "0.59.0"
2009
+source = "registry+https://github.com/rust-lang/crates.io-index"
2010
+checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
2011
+dependencies = [
2012
+ "windows-targets 0.52.6",
2013
+]
2014
+
2015
+[[package]]
2016
+name = "windows-sys"
2017
+version = "0.60.2"
2018
+source = "registry+https://github.com/rust-lang/crates.io-index"
2019
+checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb"
2020
+dependencies = [
2021
+ "windows-targets 0.53.4",
2022
+]
2023
+
2024
+[[package]]
2025
+name = "windows-sys"
2026
+version = "0.61.1"
2027
+source = "registry+https://github.com/rust-lang/crates.io-index"
2028
+checksum = "6f109e41dd4a3c848907eb83d5a42ea98b3769495597450cf6d153507b166f0f"
2029
+dependencies = [
2030
+ "windows-link",
2031
+]
2032
+
2033
+[[package]]
2034
+name = "windows-targets"
2035
+version = "0.52.6"
2036
+source = "registry+https://github.com/rust-lang/crates.io-index"
2037
+checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
2038
+dependencies = [
2039
+ "windows_aarch64_gnullvm 0.52.6",
2040
+ "windows_aarch64_msvc 0.52.6",
2041
+ "windows_i686_gnu 0.52.6",
2042
+ "windows_i686_gnullvm 0.52.6",
2043
+ "windows_i686_msvc 0.52.6",
2044
+ "windows_x86_64_gnu 0.52.6",
2045
+ "windows_x86_64_gnullvm 0.52.6",
2046
+ "windows_x86_64_msvc 0.52.6",
2047
+]
2048
+
2049
+[[package]]
2050
+name = "windows-targets"
2051
+version = "0.53.4"
2052
+source = "registry+https://github.com/rust-lang/crates.io-index"
2053
+checksum = "2d42b7b7f66d2a06854650af09cfdf8713e427a439c97ad65a6375318033ac4b"
2054
+dependencies = [
2055
+ "windows-link",
2056
+ "windows_aarch64_gnullvm 0.53.0",
2057
+ "windows_aarch64_msvc 0.53.0",
2058
+ "windows_i686_gnu 0.53.0",
2059
+ "windows_i686_gnullvm 0.53.0",
2060
+ "windows_i686_msvc 0.53.0",
2061
+ "windows_x86_64_gnu 0.53.0",
2062
+ "windows_x86_64_gnullvm 0.53.0",
2063
+ "windows_x86_64_msvc 0.53.0",
2064
+]
2065
+
2066
+[[package]]
2067
+name = "windows_aarch64_gnullvm"
2068
+version = "0.52.6"
2069
+source = "registry+https://github.com/rust-lang/crates.io-index"
2070
+checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
2071
+
2072
+[[package]]
2073
+name = "windows_aarch64_gnullvm"
2074
+version = "0.53.0"
2075
+source = "registry+https://github.com/rust-lang/crates.io-index"
2076
+checksum = "86b8d5f90ddd19cb4a147a5fa63ca848db3df085e25fee3cc10b39b6eebae764"
2077
+
2078
+[[package]]
2079
+name = "windows_aarch64_msvc"
2080
+version = "0.52.6"
2081
+source = "registry+https://github.com/rust-lang/crates.io-index"
2082
+checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
2083
+
2084
+[[package]]
2085
+name = "windows_aarch64_msvc"
2086
+version = "0.53.0"
2087
+source = "registry+https://github.com/rust-lang/crates.io-index"
2088
+checksum = "c7651a1f62a11b8cbd5e0d42526e55f2c99886c77e007179efff86c2b137e66c"
2089
+
2090
+[[package]]
2091
+name = "windows_i686_gnu"
2092
+version = "0.52.6"
2093
+source = "registry+https://github.com/rust-lang/crates.io-index"
2094
+checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
2095
+
2096
+[[package]]
2097
+name = "windows_i686_gnu"
2098
+version = "0.53.0"
2099
+source = "registry+https://github.com/rust-lang/crates.io-index"
2100
+checksum = "c1dc67659d35f387f5f6c479dc4e28f1d4bb90ddd1a5d3da2e5d97b42d6272c3"
2101
+
2102
+[[package]]
2103
+name = "windows_i686_gnullvm"
2104
+version = "0.52.6"
2105
+source = "registry+https://github.com/rust-lang/crates.io-index"
2106
+checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
2107
+
2108
+[[package]]
2109
+name = "windows_i686_gnullvm"
2110
+version = "0.53.0"
2111
+source = "registry+https://github.com/rust-lang/crates.io-index"
2112
+checksum = "9ce6ccbdedbf6d6354471319e781c0dfef054c81fbc7cf83f338a4296c0cae11"
2113
+
2114
+[[package]]
2115
+name = "windows_i686_msvc"
2116
+version = "0.52.6"
2117
+source = "registry+https://github.com/rust-lang/crates.io-index"
2118
+checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
2119
+
2120
+[[package]]
2121
+name = "windows_i686_msvc"
2122
+version = "0.53.0"
2123
+source = "registry+https://github.com/rust-lang/crates.io-index"
2124
+checksum = "581fee95406bb13382d2f65cd4a908ca7b1e4c2f1917f143ba16efe98a589b5d"
2125
+
2126
+[[package]]
2127
+name = "windows_x86_64_gnu"
2128
+version = "0.52.6"
2129
+source = "registry+https://github.com/rust-lang/crates.io-index"
2130
+checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
2131
+
2132
+[[package]]
2133
+name = "windows_x86_64_gnu"
2134
+version = "0.53.0"
2135
+source = "registry+https://github.com/rust-lang/crates.io-index"
2136
+checksum = "2e55b5ac9ea33f2fc1716d1742db15574fd6fc8dadc51caab1c16a3d3b4190ba"
2137
+
2138
+[[package]]
2139
+name = "windows_x86_64_gnullvm"
2140
+version = "0.52.6"
2141
+source = "registry+https://github.com/rust-lang/crates.io-index"
2142
+checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
2143
+
2144
+[[package]]
2145
+name = "windows_x86_64_gnullvm"
2146
+version = "0.53.0"
2147
+source = "registry+https://github.com/rust-lang/crates.io-index"
2148
+checksum = "0a6e035dd0599267ce1ee132e51c27dd29437f63325753051e71dd9e42406c57"
2149
+
2150
+[[package]]
2151
+name = "windows_x86_64_msvc"
2152
+version = "0.52.6"
2153
+source = "registry+https://github.com/rust-lang/crates.io-index"
2154
+checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
2155
+
2156
+[[package]]
2157
+name = "windows_x86_64_msvc"
2158
+version = "0.53.0"
2159
+source = "registry+https://github.com/rust-lang/crates.io-index"
2160
+checksum = "271414315aff87387382ec3d271b52d7ae78726f5d44ac98b4f4030c91880486"
2161
+
2162
+[[package]]
2163
+name = "winnow"
2164
+version = "0.7.13"
2165
+source = "registry+https://github.com/rust-lang/crates.io-index"
2166
+checksum = "21a0236b59786fed61e2a80582dd500fe61f18b5dca67a4a067d0bc9039339cf"
2167
+dependencies = [
2168
+ "memchr",
2169
+]
2170
+
2171
+[[package]]
2172
+name = "wit-bindgen"
2173
+version = "0.46.0"
2174
+source = "registry+https://github.com/rust-lang/crates.io-index"
2175
+checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59"
2176
+
2177
+[[package]]
2178
+name = "zerocopy"
2179
+version = "0.8.27"
2180
+source = "registry+https://github.com/rust-lang/crates.io-index"
2181
+checksum = "0894878a5fa3edfd6da3f88c4805f4c8558e2b996227a3d864f47fe11e38282c"
2182
+dependencies = [
2183
+ "zerocopy-derive 0.8.27",
2184
+]
2185
+
2186
+[[package]]
2187
+name = "zerocopy"
2188
+version = "0.9.0-alpha.0"
2189
+source = "registry+https://github.com/rust-lang/crates.io-index"
2190
+checksum = "3c9c378180b968f2e98efa0c4bba8c49fd4cd247dc82a7f01e72dc137d447d8c"
2191
+dependencies = [
2192
+ "zerocopy-derive 0.9.0-alpha.0",
2193
+]
2194
+
2195
+[[package]]
2196
+name = "zerocopy-derive"
2197
+version = "0.8.27"
2198
+source = "registry+https://github.com/rust-lang/crates.io-index"
2199
+checksum = "88d2b8d9c68ad2b9e4340d7832716a4d21a22a1154777ad56ea55c51a9cf3831"
2200
+dependencies = [
2201
+ "proc-macro2",
2202
+ "quote",
2203
+ "syn",
2204
+]
2205
+
2206
+[[package]]
2207
+name = "zerocopy-derive"
2208
+version = "0.9.0-alpha.0"
2209
+source = "registry+https://github.com/rust-lang/crates.io-index"
2210
+checksum = "36811c5ea90dd441f941919fc99163e702f150a8e1495d72a268106a0100df09"
2211
+dependencies = [
2212
+ "proc-macro2",
2213
+ "quote",
2214
+ "syn",
2215
+]
2216
+
2217
+[[package]]
2218
+name = "zeroize"
2219
+version = "1.8.2"
2220
+source = "registry+https://github.com/rust-lang/crates.io-index"
2221
+checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0"
src/crates/jf/Cargo.toml
new
+66
@@ -0,0 +1,66 @@
1
+[workspace]
2
+resolver = "2"
3
+members = [
4
+ "error",
5
+ "journal_file",
6
+ "journal_reader_ffi",
7
+ "journal_log",
8
+ "window_manager",
9
+ "sigbus",
10
+ "otel-plugin",
11
+ "flatten_otel",
12
+ "flog-otel",
13
+]
14
+
15
+[workspace.package]
16
+version = "0.1.0"
17
+edition = "2021"
18
+rust-version = "1.85"
19
+
20
+[workspace.dependencies]
21
+memmap2 = "0.9"
22
+zerocopy = { version = "0.9.0-alpha.0", features = ["derive"] }
23
+thiserror = "2"
24
+rand = "0.9"
25
+siphasher = "1.0"
26
+twox-hash = { version = "2.1", default-features = false, features = ["std"] }
27
+static_assertions = "1.1"
28
+duct = "0.13"
29
+serde = { version = "1.0", features = ["derive"] }
30
+serde_json = "1.0"
31
+walkdir = "2"
32
+ruzstd = "0.8"
33
+libc = "0.2"
34
+hex = "0.4"
35
+tempfile = "3"
36
+indexmap = "2.9"
37
+uuid = { version = "1.0", features = ["v4", "rng"] }
38
+chrono = { version = "0.4", features = ["serde"] }
39
+tracing = "0.1"
40
+tracing-subscriber = "0.3"
41
+
42
+# otel-plugin
43
+prost = "0.13"
44
+regex = "1.10"
45
+tokio = { version = "1.45", features = ["rt-multi-thread", "process", "io-util"] }
46
+tokio-stream = "0.1"
47
+tonic = "0.13"
48
+tonic-build = "0.13"
49
+prost-build = "0.13"
50
+hyper-util = "0.1"
51
+tower = "0.5"
52
+opentelemetry-proto = "0.30"
53
+atty = "0.2"
54
+anyhow = "1.0"
55
+humantime-serde = "1.1"
56
+humantime = "2.2"
57
+bytesize = "1.0"
58
+bytesize-serde = "0.2"
59
+
60
+clap = { version = "4.5", features = ["derive", "env"] }
61
+
62
+flatten-serde-json = { git = "https://github.com/meilisearch/meilisearch", tag = "v1.22.1" }
63
+
64
+[profile.release]
65
+lto = true
66
+codegen-units = 1
src/crates/jf/LICENSE
new
+684
@@ -0,0 +1,684 @@
1
+ GNU GENERAL PUBLIC LICENSE
2
+ Version 3, 29 June 2007
3
+
4
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
5
+ Everyone is permitted to copy and distribute verbatim copies
6
+ of this license document, but changing it is not allowed.
7
+
8
+ Preamble
9
+
10
+ The GNU General Public License is a free, copyleft license for
11
+software and other kinds of works.
12
+
13
+ The licenses for most software and other practical works are designed
14
+to take away your freedom to share and change the works. By contrast,
15
+the GNU General Public License is intended to guarantee your freedom to
16
+share and change all versions of a program--to make sure it remains free
17
+software for all its users. We, the Free Software Foundation, use the
18
+GNU General Public License for most of our software; it applies also to
19
+any other work released this way by its authors. You can apply it to
20
+your programs, too.
21
+
22
+ When we speak of free software, we are referring to freedom, not
23
+price. Our General Public Licenses are designed to make sure that you
24
+have the freedom to distribute copies of free software (and charge for
25
+them if you wish), that you receive source code or can get it if you
26
+want it, that you can change the software or use pieces of it in new
27
+free programs, and that you know you can do these things.
28
+
29
+ To protect your rights, we need to prevent others from denying you
30
+these rights or asking you to surrender the rights. Therefore, you have
31
+certain responsibilities if you distribute copies of the software, or if
32
+you modify it: responsibilities to respect the freedom of others.
33
+
34
+ For example, if you distribute copies of such a program, whether
35
+gratis or for a fee, you must pass on to the recipients the same
36
+freedoms that you received. You must make sure that they, too, receive
37
+or can get the source code. And you must show them these terms so they
38
+know their rights.
39
+
40
+ Developers that use the GNU GPL protect your rights with two steps:
41
+(1) assert copyright on the software, and (2) offer you this License
42
+giving you legal permission to copy, distribute and/or modify it.
43
+
44
+ For the developers' and authors' protection, the GPL clearly explains
45
+that there is no warranty for this free software. For both users' and
46
+authors' sake, the GPL requires that modified versions be marked as
47
+changed, so that their problems will not be attributed erroneously to
48
+authors of previous versions.
49
+
50
+ Some devices are designed to deny users access to install or run
51
+modified versions of the software inside them, although the manufacturer
52
+can do so. This is fundamentally incompatible with the aim of
53
+protecting users' freedom to change the software. The systematic
54
+pattern of such abuse occurs in the area of products for individuals to
55
+use, which is precisely where it is most unacceptable. Therefore, we
56
+have designed this version of the GPL to prohibit the practice for those
57
+products. If such problems arise substantially in other domains, we
58
+stand ready to extend this provision to those domains in future versions
59
+of the GPL, as needed to protect the freedom of users.
60
+
61
+ Finally, every program is threatened constantly by software patents.
62
+States should not allow patents to restrict development and use of
63
+software on general-purpose computers, but in those that do, we wish to
64
+avoid the special danger that patents applied to a free program could
65
+make it effectively proprietary. To prevent this, the GPL assures that
66
+patents cannot be used to render the program non-free.
67
+
68
+ The precise terms and conditions for copying, distribution and
69
+modification follow.
70
+
71
+ TERMS AND CONDITIONS
72
+
73
+ 0. Definitions.
74
+
75
+ "This License" refers to version 3 of the GNU General Public License.
76
+
77
+ "Copyright" also means copyright-like laws that apply to other kinds of
78
+works, such as semiconductor masks.
79
+
80
+ "The Program" refers to any copyrightable work licensed under this
81
+License. Each licensee is addressed as "you". "Licensees" and
82
+"recipients" may be individuals or organizations.
83
+
84
+ To "modify" a work means to copy from or adapt all or part of the work
85
+in a fashion requiring copyright permission, other than the making of an
86
+exact copy. The resulting work is called a "modified version" of the
87
+earlier work or a work "based on" the earlier work.
88
+
89
+ A "covered work" means either the unmodified Program or a work based
90
+on the Program.
91
+
92
+ To "propagate" a work means to do anything with it that, without
93
+permission, would make you directly or secondarily liable for
94
+infringement under applicable copyright law, except executing it on a
95
+computer or modifying a private copy. Propagation includes copying,
96
+distribution (with or without modification), making available to the
97
+public, and in some countries other activities as well.
98
+
99
+ To "convey" a work means any kind of propagation that enables other
100
+parties to make or receive copies. Mere interaction with a user through
101
+a computer network, with no transfer of a copy, is not conveying.
102
+
103
+ An interactive user interface displays "Appropriate Legal Notices"
104
+to the extent that it includes a convenient and prominently visible
105
+feature that (1) displays an appropriate copyright notice, and (2)
106
+tells the user that there is no warranty for the work (except to the
107
+extent that warranties are provided), that licensees may convey the
108
+work under this License, and how to view a copy of this License. If
109
+the interface presents a list of user commands or options, such as a
110
+menu, a prominent item in the list meets this criterion.
111
+
112
+ 1. Source Code.
113
+
114
+ The "source code" for a work means the preferred form of the work
115
+for making modifications to it. "Object code" means any non-source
116
+form of a work.
117
+
118
+ A "Standard Interface" means an interface that either is an official
119
+standard defined by a recognized standards body, or, in the case of
120
+interfaces specified for a particular programming language, one that
121
+is widely used among developers working in that language.
122
+
123
+ The "System Libraries" of an executable work include anything, other
124
+than the work as a whole, that (a) is included in the normal form of
125
+packaging a Major Component, but which is not part of that Major
126
+Component, and (b) serves only to enable use of the work with that
127
+Major Component, or to implement a Standard Interface for which an
128
+implementation is available to the public in source code form. A
129
+"Major Component", in this context, means a major essential component
130
+(kernel, window system, and so on) of the specific operating system
131
+(if any) on which the executable work runs, or a compiler used to
132
+produce the work, or an object code interpreter used to run it.
133
+
134
+ The "Corresponding Source" for a work in object code form means all
135
+the source code needed to generate, install, and (for an executable
136
+work) run the object code and to modify the work, including scripts to
137
+control those activities. However, it does not include the work's
138
+System Libraries, or general-purpose tools or generally available free
139
+programs which are used unmodified in performing those activities but
140
+which are not part of the work. For example, Corresponding Source
141
+includes interface definition files associated with source files for
142
+the work, and the source code for shared libraries and dynamically
143
+linked subprograms that the work is specifically designed to require,
144
+such as by intimate data communication or control flow between those
145
+subprograms and other parts of the work.
146
+
147
+ The Corresponding Source need not include anything that users
148
+can regenerate automatically from other parts of the Corresponding
149
+Source.
150
+
151
+ The Corresponding Source for a work in source code form is that
152
+same work.
153
+
154
+ 2. Basic Permissions.
155
+
156
+ All rights granted under this License are granted for the term of
157
+copyright on the Program, and are irrevocable provided the stated
158
+conditions are met. This License explicitly affirms your unlimited
159
+permission to run the unmodified Program. The output from running a
160
+covered work is covered by this License only if the output, given its
161
+content, constitutes a covered work. This License acknowledges your
162
+rights of fair use or other equivalent, as provided by copyright law.
163
+
164
+ You may make, run and propagate covered works that you do not
165
+convey, without conditions so long as your license otherwise remains
166
+in force. You may convey covered works to others for the sole purpose
167
+of having them make modifications exclusively for you, or provide you
168
+with facilities for running those works, provided that you comply with
169
+the terms of this License in conveying all material for which you do
170
+not control copyright. Those thus making or running the covered works
171
+for you must do so exclusively on your behalf, under your direction
172
+and control, on terms that prohibit them from making any copies of
173
+your copyrighted material outside their relationship with you.
174
+
175
+ Conveying under any other circumstances is permitted solely under
176
+the conditions stated below. Sublicensing is not allowed; section 10
177
+makes it unnecessary.
178
+
179
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
180
+
181
+ No covered work shall be deemed part of an effective technological
182
+measure under any applicable law fulfilling obligations under article
183
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
184
+similar laws prohibiting or restricting circumvention of such
185
+measures.
186
+
187
+ When you convey a covered work, you waive any legal power to forbid
188
+circumvention of technological measures to the extent such circumvention
189
+is effected by exercising rights under this License with respect to
190
+the covered work, and you disclaim any intention to limit operation or
191
+modification of the work as a means of enforcing, against the work's
192
+users, your or third parties' legal rights to forbid circumvention of
193
+technological measures.
194
+
195
+ 4. Conveying Verbatim Copies.
196
+
197
+ You may convey verbatim copies of the Program's source code as you
198
+receive it, in any medium, provided that you conspicuously and
199
+appropriately publish on each copy an appropriate copyright notice;
200
+keep intact all notices stating that this License and any
201
+non-permissive terms added in accord with section 7 apply to the code;
202
+keep intact all notices of the absence of any warranty; and give all
203
+recipients a copy of this License along with the Program.
204
+
205
+ You may charge any price or no price for each copy that you convey,
206
+and you may offer support or warranty protection for a fee.
207
+
208
+ 5. Conveying Modified Source Versions.
209
+
210
+ You may convey a work based on the Program, or the modifications to
211
+produce it from the Program, in the form of source code under the
212
+terms of section 4, provided that you also meet all of these conditions:
213
+
214
+ a) The work must carry prominent notices stating that you modified
215
+ it, and giving a relevant date.
216
+
217
+ b) The work must carry prominent notices stating that it is
218
+ released under this License and any conditions added under section
219
+ 7. This requirement modifies the requirement in section 4 to
220
+ "keep intact all notices".
221
+
222
+ c) You must license the entire work, as a whole, under this
223
+ License to anyone who comes into possession of a copy. This
224
+ License will therefore apply, along with any applicable section 7
225
+ additional terms, to the whole of the work, and all its parts,
226
+ regardless of how they are packaged. This License gives no
227
+ permission to license the work in any other way, but it does not
228
+ invalidate such permission if you have separately received it.
229
+
230
+ d) If the work has interactive user interfaces, each must display
231
+ Appropriate Legal Notices; however, if the Program has interactive
232
+ interfaces that do not display Appropriate Legal Notices, your
233
+ work need not make them do so.
234
+
235
+ A compilation of a covered work with other separate and independent
236
+works, which are not by their nature extensions of the covered work,
237
+and which are not combined with it such as to form a larger program,
238
+in or on a volume of a storage or distribution medium, is called an
239
+"aggregate" if the compilation and its resulting copyright are not
240
+used to limit the access or legal rights of the compilation's users
241
+beyond what the individual works permit. Inclusion of a covered work
242
+in an aggregate does not cause this License to apply to the other
243
+parts of the aggregate.
244
+
245
+ 6. Conveying Non-Source Forms.
246
+
247
+ You may convey a covered work in object code form under the terms
248
+of sections 4 and 5, provided that you also convey the
249
+machine-readable Corresponding Source under the terms of this License,
250
+in one of these ways:
251
+
252
+ a) Convey the object code in, or embodied in, a physical product
253
+ (including a physical distribution medium), accompanied by the
254
+ Corresponding Source fixed on a durable physical medium
255
+ customarily used for software interchange.
256
+
257
+ b) Convey the object code in, or embodied in, a physical product
258
+ (including a physical distribution medium), accompanied by a
259
+ written offer, valid for at least three years and valid for as
260
+ long as you offer spare parts or customer support for that product
261
+ model, to give anyone who possesses the object code either (1) a
262
+ copy of the Corresponding Source for all the software in the
263
+ product that is covered by this License, on a durable physical
264
+ medium customarily used for software interchange, for a price no
265
+ more than your reasonable cost of physically performing this
266
+ conveying of source, or (2) access to copy the
267
+ Corresponding Source from a network server at no charge.
268
+
269
+ c) Convey individual copies of the object code with a copy of the
270
+ written offer to provide the Corresponding Source. This
271
+ alternative is allowed only occasionally and noncommercially, and
272
+ only if you received the object code with such an offer, in accord
273
+ with subsection 6b.
274
+
275
+ d) Convey the object code by offering access from a designated
276
+ place (gratis or for a charge), and offer equivalent access to the
277
+ Corresponding Source in the same way through the same place at no
278
+ further charge. You need not require recipients to copy the
279
+ Corresponding Source along with the object code. If the place to
280
+ copy the object code is a network server, the Corresponding Source
281
+ may be on a different server (operated by you or a third party)
282
+ that supports equivalent copying facilities, provided you maintain
283
+ clear directions next to the object code saying where to find the
284
+ Corresponding Source. Regardless of what server hosts the
285
+ Corresponding Source, you remain obligated to ensure that it is
286
+ available for as long as needed to satisfy these requirements.
287
+
288
+ e) Convey the object code using peer-to-peer transmission, provided
289
+ you inform other peers where the object code and Corresponding
290
+ Source of the work are being offered to the general public at no
291
+ charge under subsection 6d.
292
+
293
+ A separable portion of the object code, whose source code is excluded
294
+from the Corresponding Source as a System Library, need not be
295
+included in conveying the object code work.
296
+
297
+ A "User Product" is either (1) a "consumer product", which means any
298
+tangible personal property which is normally used for personal, family,
299
+or household purposes, or (2) anything designed or sold for incorporation
300
+into a dwelling. In determining whether a product is a consumer product,
301
+doubtful cases shall be resolved in favor of coverage. For a particular
302
+product received by a particular user, "normally used" refers to a
303
+typical or common use of that class of product, regardless of the status
304
+of the particular user or of the way in which the particular user
305
+actually uses, or expects or is expected to use, the product. A product
306
+is a consumer product regardless of whether the product has substantial
307
+commercial, industrial or non-consumer uses, unless such uses represent
308
+the only significant mode of use of the product.
309
+
310
+ "Installation Information" for a User Product means any methods,
311
+procedures, authorization keys, or other information required to install
312
+and execute modified versions of a covered work in that User Product from
313
+a modified version of its Corresponding Source. The information must
314
+suffice to ensure that the continued functioning of the modified object
315
+code is in no case prevented or interfered with solely because
316
+modification has been made.
317
+
318
+ If you convey an object code work under this section in, or with, or
319
+specifically for use in, a User Product, and the conveying occurs as
320
+part of a transaction in which the right of possession and use of the
321
+User Product is transferred to the recipient in perpetuity or for a
322
+fixed term (regardless of how the transaction is characterized), the
323
+Corresponding Source conveyed under this section must be accompanied
324
+by the Installation Information. But this requirement does not apply
325
+if neither you nor any third party retains the ability to install
326
+modified object code on the User Product (for example, the work has
327
+been installed in ROM).
328
+
329
+ The requirement to provide Installation Information does not include a
330
+requirement to continue to provide support service, warranty, or updates
331
+for a work that has been modified or installed by the recipient, or for
332
+the User Product in which it has been modified or installed. Access to a
333
+network may be denied when the modification itself materially and
334
+adversely affects the operation of the network or violates the rules and
335
+protocols for communication across the network.
336
+
337
+ Corresponding Source conveyed, and Installation Information provided,
338
+in accord with this section must be in a format that is publicly
339
+documented (and with an implementation available to the public in
340
+source code form), and must require no special password or key for
341
+unpacking, reading or copying.
342
+
343
+ 7. Additional Terms.
344
+
345
+ "Additional permissions" are terms that supplement the terms of this
346
+License by making exceptions from one or more of its conditions.
347
+Additional permissions that are applicable to the entire Program shall
348
+be treated as though they were included in this License, to the extent
349
+that they are valid under applicable law. If additional permissions
350
+apply only to part of the Program, that part may be used separately
351
+under those permissions, but the entire Program remains governed by
352
+this License without regard to the additional permissions.
353
+
354
+ When you convey a copy of a covered work, you may at your option
355
+remove any additional permissions from that copy, or from any part of
356
+it. (Additional permissions may be written to require their own
357
+removal in certain cases when you modify the work.) You may place
358
+additional permissions on material, added by you to a covered work,
359
+for which you have or can give appropriate copyright permission.
360
+
361
+ Notwithstanding any other provision of this License, for material you
362
+add to a covered work, you may (if authorized by the copyright holders of
363
+that material) supplement the terms of this License with terms:
364
+
365
+ a) Disclaiming warranty or limiting liability differently from the
366
+ terms of sections 15 and 16 of this License; or
367
+
368
+ b) Requiring preservation of specified reasonable legal notices or
369
+ author attributions in that material or in the Appropriate Legal
370
+ Notices displayed by works containing it; or
371
+
372
+ c) Prohibiting misrepresentation of the origin of that material, or
373
+ requiring that modified versions of such material be marked in
374
+ reasonable ways as different from the original version; or
375
+
376
+ d) Limiting the use for publicity purposes of names of licensors or
377
+ authors of the material; or
378
+
379
+ e) Declining to grant rights under trademark law for use of some
380
+ trade names, trademarks, or service marks; or
381
+
382
+ f) Requiring indemnification of licensors and authors of that
383
+ material by anyone who conveys the material (or modified versions of
384
+ it) with contractual assumptions of liability to the recipient, for
385
+ any liability that these contractual assumptions directly impose on
386
+ those licensors and authors.
387
+
388
+ All other non-permissive additional terms are considered "further
389
+restrictions" within the meaning of section 10. If the Program as you
390
+received it, or any part of it, contains a notice stating that it is
391
+governed by this License along with a term that is a further
392
+restriction, you may remove that term. If a license document contains
393
+a further restriction but permits relicensing or conveying under this
394
+License, you may add to a covered work material governed by the terms
395
+of that license document, provided that the further restriction does
396
+not survive such relicensing or conveying.
397
+
398
+ If you add terms to a covered work in accord with this section, you
399
+must place, in the relevant source files, a statement of the
400
+additional terms that apply to those files, or a notice indicating
401
+where to find the applicable terms.
402
+
403
+ Additional terms, permissive or non-permissive, may be stated in the
404
+form of a separately written license, or stated as exceptions;
405
+the above requirements apply either way.
406
+
407
+ 8. Termination.
408
+
409
+ You may not propagate or modify a covered work except as expressly
410
+provided under this License. Any attempt otherwise to propagate or
411
+modify it is void, and will automatically terminate your rights under
412
+this License (including any patent licenses granted under the third
413
+paragraph of section 11).
414
+
415
+ However, if you cease all violation of this License, then your
416
+license from a particular copyright holder is reinstated (a)
417
+provisionally, unless and until the copyright holder explicitly and
418
+finally terminates your license, and (b) permanently, if the copyright
419
+holder fails to notify you of the violation by some reasonable means
420
+prior to 60 days after the cessation.
421
+
422
+ Moreover, your license from a particular copyright holder is
423
+reinstated permanently if the copyright holder notifies you of the
424
+violation by some reasonable means, this is the first time you have
425
+received notice of violation of this License (for any work) from that
426
+copyright holder, and you cure the violation prior to 30 days after
427
+your receipt of the notice.
428
+
429
+ Termination of your rights under this section does not terminate the
430
+licenses of parties who have received copies or rights from you under
431
+this License. If your rights have been terminated and not permanently
432
+reinstated, you do not qualify to receive new licenses for the same
433
+material under section 10.
434
+
435
+ 9. Acceptance Not Required for Having Copies.
436
+
437
+ You are not required to accept this License in order to receive or
438
+run a copy of the Program. Ancillary propagation of a covered work
439
+occurring solely as a consequence of using peer-to-peer transmission
440
+to receive a copy likewise does not require acceptance. However,
441
+nothing other than this License grants you permission to propagate or
442
+modify any covered work. These actions infringe copyright if you do
443
+not accept this License. Therefore, by modifying or propagating a
444
+covered work, you indicate your acceptance of this License to do so.
445
+
446
+ 10. Automatic Licensing of Downstream Recipients.
447
+
448
+ Each time you convey a covered work, the recipient automatically
449
+receives a license from the original licensors, to run, modify and
450
+propagate that work, subject to this License. You are not responsible
451
+for enforcing compliance by third parties with this License.
452
+
453
+ An "entity transaction" is a transaction transferring control of an
454
+organization, or substantially all assets of one, or subdividing an
455
+organization, or merging organizations. If propagation of a covered
456
+work results from an entity transaction, each party to that
457
+transaction who receives a copy of the work also receives whatever
458
+licenses to the work the party's predecessor in interest had or could
459
+give under the previous paragraph, plus a right to possession of the
460
+Corresponding Source of the work from the predecessor in interest, if
461
+the predecessor has it or can get it with reasonable efforts.
462
+
463
+ You may not impose any further restrictions on the exercise of the
464
+rights granted or affirmed under this License. For example, you may
465
+not impose a license fee, royalty, or other charge for exercise of
466
+rights granted under this License, and you may not initiate litigation
467
+(including a cross-claim or counterclaim in a lawsuit) alleging that
468
+any patent claim is infringed by making, using, selling, offering for
469
+sale, or importing the Program or any portion of it.
470
+
471
+ 11. Patents.
472
+
473
+ A "contributor" is a copyright holder who authorizes use under this
474
+License of the Program or a work on which the Program is based. The
475
+work thus licensed is called the contributor's "contributor version".
476
+
477
+ A contributor's "essential patent claims" are all patent claims
478
+owned or controlled by the contributor, whether already acquired or
479
+hereafter acquired, that would be infringed by some manner, permitted
480
+by this License, of making, using, or selling its contributor version,
481
+but do not include claims that would be infringed only as a
482
+consequence of further modification of the contributor version. For
483
+purposes of this definition, "control" includes the right to grant
484
+patent sublicenses in a manner consistent with the requirements of
485
+this License.
486
+
487
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
488
+patent license under the contributor's essential patent claims, to
489
+make, use, sell, offer for sale, import and otherwise run, modify and
490
+propagate the contents of its contributor version.
491
+
492
+ In the following three paragraphs, a "patent license" is any express
493
+agreement or commitment, however denominated, not to enforce a patent
494
+(such as an express permission to practice a patent or covenant not to
495
+sue for patent infringement). To "grant" such a patent license to a
496
+party means to make such an agreement or commitment not to enforce a
497
+patent against the party.
498
+
499
+ If you convey a covered work, knowingly relying on a patent license,
500
+and the Corresponding Source of the work is not available for anyone
501
+to copy, free of charge and under the terms of this License, through a
502
+publicly available network server or other readily accessible means,
503
+then you must either (1) cause the Corresponding Source to be so
504
+available, or (2) arrange to deprive yourself of the benefit of the
505
+patent license for this particular work, or (3) arrange, in a manner
506
+consistent with the requirements of this License, to extend the patent
507
+license to downstream recipients. "Knowingly relying" means you have
508
+actual knowledge that, but for the patent license, your conveying the
509
+covered work in a country, or your recipient's use of the covered work
510
+in a country, would infringe one or more identifiable patents in that
511
+country that you have reason to believe are valid.
512
+
513
+ If, pursuant to or in connection with a single transaction or
514
+arrangement, you convey, or propagate by procuring conveyance of, a
515
+covered work, and grant a patent license to some of the parties
516
+receiving the covered work authorizing them to use, propagate, modify
517
+or convey a specific copy of the covered work, then the patent license
518
+you grant is automatically extended to all recipients of the covered
519
+work and works based on it.
520
+
521
+ A patent license is "discriminatory" if it does not include within
522
+the scope of its coverage, prohibits the exercise of, or is
523
+conditioned on the non-exercise of one or more of the rights that are
524
+specifically granted under this License. You may not convey a covered
525
+work if you are a party to an arrangement with a third party that is
526
+in the business of distributing software, under which you make payment
527
+to the third party based on the extent of your activity of conveying
528
+the work, and under which the third party grants, to any of the
529
+parties who would receive the covered work from you, a discriminatory
530
+patent license (a) in connection with copies of the covered work
531
+conveyed by you (or copies made from those copies), or (b) primarily
532
+for and in connection with specific products or compilations that
533
+contain the covered work, unless you entered into that arrangement,
534
+or that patent license was granted, prior to 28 March 2007.
535
+
536
+ Nothing in this License shall be construed as excluding or limiting
537
+any implied license or other defenses to infringement that may
538
+otherwise be available to you under applicable patent law.
539
+
540
+ 12. No Surrender of Others' Freedom.
541
+
542
+ If conditions are imposed on you (whether by court order, agreement or
543
+otherwise) that contradict the conditions of this License, they do not
544
+excuse you from the conditions of this License. If you cannot convey a
545
+covered work so as to satisfy simultaneously your obligations under this
546
+License and any other pertinent obligations, then as a consequence you may
547
+not convey it at all. For example, if you agree to terms that obligate you
548
+to collect a royalty for further conveying from those to whom you convey
549
+the Program, the only way you could satisfy both those terms and this
550
+License would be to refrain entirely from conveying the Program.
551
+
552
+ 13. Use with the GNU Affero General Public License.
553
+
554
+ Notwithstanding any other provision of this License, you have
555
+permission to link or combine any covered work with a work licensed
556
+under version 3 of the GNU Affero General Public License into a single
557
+combined work, and to convey the resulting work. The terms of this
558
+License will continue to apply to the part which is the covered work,
559
+but the special requirements of the GNU Affero General Public License,
560
+section 13, concerning interaction through a network will apply to the
561
+combination as such.
562
+
563
+ 14. Revised Versions of this License.
564
+
565
+ The Free Software Foundation may publish revised and/or new versions of
566
+the GNU General Public License from time to time. Such new versions will
567
+be similar in spirit to the present version, but may differ in detail to
568
+address new problems or concerns.
569
+
570
+ Each version is given a distinguishing version number. If the
571
+Program specifies that a certain numbered version of the GNU General
572
+Public License "or any later version" applies to it, you have the
573
+option of following the terms and conditions either of that numbered
574
+version or of any later version published by the Free Software
575
+Foundation. If the Program does not specify a version number of the
576
+GNU General Public License, you may choose any version ever published
577
+by the Free Software Foundation.
578
+
579
+ If the Program specifies that a proxy can decide which future
580
+versions of the GNU General Public License can be used, that proxy's
581
+public statement of acceptance of a version permanently authorizes you
582
+to choose that version for the Program.
583
+
584
+ Later license versions may give you additional or different
585
+permissions. However, no additional obligations are imposed on any
586
+author or copyright holder as a result of your choosing to follow a
587
+later version.
588
+
589
+ 15. Disclaimer of Warranty.
590
+
591
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
592
+APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
593
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
594
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
595
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
596
+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
597
+IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
598
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
599
+
600
+ 16. Limitation of Liability.
601
+
602
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
603
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
604
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
605
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
606
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
607
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
608
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
609
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
610
+SUCH DAMAGES.
611
+
612
+ 17. Interpretation of Sections 15 and 16.
613
+
614
+ If the disclaimer of warranty and limitation of liability provided
615
+above cannot be given local legal effect according to their terms,
616
+reviewing courts shall apply local law that most closely approximates
617
+an absolute waiver of all civil liability in connection with the
618
+Program, unless a warranty or assumption of liability accompanies a
619
+copy of the Program in return for a fee.
620
+
621
+ END OF TERMS AND CONDITIONS
622
+
623
+ How to Apply These Terms to Your New Programs
624
+
625
+ If you develop a new program, and you want it to be of the greatest
626
+possible use to the public, the best way to achieve this is to make it
627
+free software which everyone can redistribute and change under these terms.
628
+
629
+ To do so, attach the following notices to the program. It is safest
630
+to attach them to the start of each source file to most effectively
631
+state the exclusion of warranty; and each file should have at least
632
+the "copyright" line and a pointer to where the full notice is found.
633
+
634
+ {one line to give the program's name and a brief idea of what it does.}
635
+ Copyright (C) {year} {name of author}
636
+
637
+ This program is free software: you can redistribute it and/or modify
638
+ it under the terms of the GNU General Public License as published by
639
+ the Free Software Foundation, either version 3 of the License, or
640
+ (at your option) any later version.
641
+
642
+ This program is distributed in the hope that it will be useful,
643
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
644
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
645
+ GNU General Public License for more details.
646
+
647
+ You should have received a copy of the GNU General Public License
648
+ along with this program. If not, see <http://www.gnu.org/licenses/>.
649
+
650
+Also add information on how to contact you by electronic and paper mail.
651
+
652
+ If the program does terminal interaction, make it output a short
653
+notice like this when it starts in an interactive mode:
654
+
655
+ {project} Copyright (C) {year} {fullname}
656
+ This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
657
+ This is free software, and you are welcome to redistribute it
658
+ under certain conditions; type `show c' for details.
659
+
660
+The hypothetical commands `show w' and `show c' should show the appropriate
661
+parts of the General Public License. Of course, your program's commands
662
+might be different; for a GUI interface, you would use an "about box".
663
+
664
+ You should also get your employer (if you work as a programmer) or school,
665
+if any, to sign a "copyright disclaimer" for the program, if necessary.
666
+For more information on this, and how to apply and follow the GNU GPL, see
667
+<http://www.gnu.org/licenses/>.
668
+
669
+ The GNU General Public License does not permit incorporating your program
670
+into proprietary programs. If your program is a subroutine library, you
671
+may consider it more useful to permit linking proprietary applications with
672
+the library. If this is what you want to do, use the GNU Lesser General
673
+Public License instead of this License. But first, please read
674
+<http://www.gnu.org/philosophy/why-not-lgpl.html>.
675
+
676
+---------------------------------------------------------------------------
677
+
678
+Note:
679
+Individual files contain the following tag instead of the full license text.
680
+
681
+ SPDX-License-Identifier: GPL-3.0-or-later
682
+
683
+This enables machine processing of license information based on the SPDX
684
+License Identifiers that are here available: http://spdx.org/licenses/
src/crates/jf/error/Cargo.toml
new
+10
@@ -0,0 +1,10 @@
1
+[package]
2
+name = "error"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+static_assertions = { workspace = true }
9
+thiserror = { workspace = true }
10
+zerocopy = { workspace = true }
src/crates/jf/error/src/lib.rs
new
+141
@@ -0,0 +1,141 @@
1
+#[macro_use]
2
+extern crate static_assertions;
3
+
4
+use std::io;
5
+use thiserror::Error;
6
+
7
+#[derive(Error, Debug)]
8
+pub enum JournalError {
9
+ #[error("invalid magic number")]
10
+ InvalidMagicNumber,
11
+
12
+ #[error("invalid journal file state")]
13
+ InvalidJournalFileState,
14
+
15
+ #[error("invalid object type")]
16
+ InvalidObjectType,
17
+
18
+ #[error("invalid object location")]
19
+ InvalidObjectLocation,
20
+
21
+ #[error("invalid zerocopy size")]
22
+ InvalidZeroCopySize,
23
+
24
+ #[error("previous object is still in use")]
25
+ ValueGuardInUse,
26
+
27
+ #[error("i/o error during object operation: {0}")]
28
+ Io(#[from] io::Error),
29
+
30
+ #[error("missing hash table")]
31
+ MissingHashTable,
32
+
33
+ #[error("missing object from hash table")]
34
+ MissingObjectFromHashTable,
35
+
36
+ #[error("invalid offset array offset")]
37
+ InvalidOffsetArrayOffset,
38
+
39
+ #[error("invalid offset array index")]
40
+ InvalidOffsetArrayIndex,
41
+
42
+ #[error("empty offset array list")]
43
+ EmptyOffsetArrayList,
44
+
45
+ #[error("empty offset array node")]
46
+ EmptyOffsetArrayNode,
47
+
48
+ #[error("empty inline cursor")]
49
+ EmptyInlineCursor,
50
+
51
+ #[error("unset cursor")]
52
+ UnsetCursor,
53
+
54
+ #[error("malformed filter")]
55
+ MalformedFilter,
56
+
57
+ #[error("Invalid field")]
58
+ InvalidField,
59
+
60
+ #[error("Decompressor error")]
61
+ DecompressorError,
62
+
63
+ #[error("out of bounds index")]
64
+ OutOfBoundsIndex,
65
+
66
+ #[error("invalid offset")]
67
+ InvalidOffset,
68
+
69
+ #[error("zerocopy failure")]
70
+ ZerocopyFailure,
71
+
72
+ #[error("sigbus handler error")]
73
+ SigbusHandlerError,
74
+
75
+ #[error("unknown compression method")]
76
+ UnknownCompressionMethod,
77
+
78
+ #[error("ffi error")]
79
+ InvalidFfiOp,
80
+
81
+ #[error("uuid encoding/decoding")]
82
+ UuidSerde,
83
+
84
+ #[error("invalid filename")]
85
+ InvalidFilename,
86
+
87
+ #[error("system time error")]
88
+ SystemTimeError,
89
+
90
+ #[error("directory not found")]
91
+ DirectoryNotFound,
92
+
93
+ #[error("not a directory")]
94
+ NotADirectory,
95
+}
96
+
97
+const_assert!(std::mem::size_of::<JournalError>() <= 16);
98
+
99
+impl JournalError {
100
+ pub fn to_error_code(&self) -> i32 {
101
+ match self {
102
+ JournalError::InvalidMagicNumber => -1,
103
+ JournalError::InvalidJournalFileState => -2,
104
+ JournalError::InvalidObjectType => -3,
105
+ JournalError::InvalidObjectLocation => -4,
106
+ JournalError::InvalidZeroCopySize => -5,
107
+ JournalError::ValueGuardInUse => -6,
108
+ JournalError::Io(_) => -7,
109
+ JournalError::MissingHashTable => -8,
110
+ JournalError::MissingObjectFromHashTable => -9,
111
+ JournalError::InvalidOffsetArrayOffset => -10,
112
+ JournalError::InvalidOffsetArrayIndex => -11,
113
+ JournalError::EmptyOffsetArrayList => -12,
114
+ JournalError::EmptyOffsetArrayNode => -13,
115
+ JournalError::EmptyInlineCursor => -14,
116
+ JournalError::UnsetCursor => -15,
117
+ JournalError::MalformedFilter => -16,
118
+ JournalError::InvalidField => -17,
119
+ JournalError::DecompressorError => -18,
120
+ JournalError::OutOfBoundsIndex => -19,
121
+ JournalError::InvalidOffset => -20,
122
+ JournalError::ZerocopyFailure => -21,
123
+ JournalError::SigbusHandlerError => -22,
124
+ JournalError::UnknownCompressionMethod => -23,
125
+ JournalError::InvalidFfiOp => -24,
126
+ JournalError::UuidSerde => -25,
127
+ JournalError::InvalidFilename => -26,
128
+ JournalError::SystemTimeError => -27,
129
+ JournalError::DirectoryNotFound => -28,
130
+ JournalError::NotADirectory => -29,
131
+ }
132
+ }
133
+}
134
+
135
+impl<T: zerocopy::KnownLayout> From<zerocopy::SizeError<&[u8], T>> for JournalError {
136
+ fn from(_: zerocopy::SizeError<&[u8], T>) -> Self {
137
+ JournalError::InvalidZeroCopySize
138
+ }
139
+}
140
+
141
+pub type Result<T> = std::result::Result<T, JournalError>;
src/crates/jf/flatten_otel/Cargo.toml
new
+10
@@ -0,0 +1,10 @@
1
+[package]
2
+name = "flatten_otel"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+opentelemetry-proto = { workspace = true, features = ["logs", "metrics", "with-serde"] }
9
+flatten-serde-json = { workspace = true }
10
+serde_json = { workspace = true }
src/crates/jf/flatten_otel/src/lib.rs
new
+86
@@ -0,0 +1,86 @@
1
+use serde_json::{Map as JsonMap, Value as JsonValue};
2
+
3
+use opentelemetry_proto::tonic::{
4
+ common::v1::{AnyValue, InstrumentationScope, KeyValue},
5
+ resource::v1::Resource,
6
+};
7
+
8
+mod logs;
9
+mod metrics;
10
+
11
+pub use logs::{json_from_export_logs_service_request, json_from_log_record};
12
+pub use metrics::flatten_metrics_request;
13
+
14
+fn json_from_key_value_list(kvl: &Vec<KeyValue>) -> JsonMap<String, JsonValue> {
15
+ let mut map = JsonMap::new();
16
+
17
+ for kv in kvl {
18
+ if let Some(any_value) = &kv.value {
19
+ map.insert(kv.key.clone(), json_from_any_value(any_value));
20
+ } else {
21
+ map.insert(kv.key.clone(), JsonValue::Null);
22
+ }
23
+ }
24
+
25
+ flatten_serde_json::flatten(&map)
26
+}
27
+
28
+fn json_from_any_value(any_value: &AnyValue) -> JsonValue {
29
+ use opentelemetry_proto::tonic::common::v1::any_value::Value;
30
+
31
+ match &any_value.value {
32
+ Some(Value::StringValue(s)) => JsonValue::String(s.clone()),
33
+ Some(Value::BoolValue(b)) => JsonValue::Bool(*b),
34
+ Some(Value::IntValue(i)) => JsonValue::Number(
35
+ serde_json::Number::from_f64(*i as f64).unwrap_or_else(|| serde_json::Number::from(0)),
36
+ ),
37
+ Some(Value::DoubleValue(d)) => JsonValue::Number(
38
+ serde_json::Number::from_f64(*d).unwrap_or_else(|| serde_json::Number::from(0)),
39
+ ),
40
+ Some(Value::ArrayValue(array)) => {
41
+ let values: Vec<JsonValue> = array.values.iter().map(json_from_any_value).collect();
42
+ JsonValue::Array(values)
43
+ }
44
+ Some(Value::KvlistValue(kvl)) => JsonValue::Object(json_from_key_value_list(&kvl.values)),
45
+ Some(Value::BytesValue(_bytes)) => {
46
+ todo!("Add support for byte values");
47
+ }
48
+ None => JsonValue::Null,
49
+ }
50
+}
51
+
52
+fn json_from_resource(jm: &mut JsonMap<String, JsonValue>, resource: &Resource) {
53
+ if !resource.attributes.is_empty() {
54
+ let resource_attrs = json_from_key_value_list(&resource.attributes);
55
+ for (key, value) in resource_attrs {
56
+ jm.insert(format!("resource.attributes.{}", key), value);
57
+ }
58
+ }
59
+}
60
+
61
+fn json_from_instrumentation_scope(
62
+ jm: &mut JsonMap<String, JsonValue>,
63
+ scope: &InstrumentationScope,
64
+) {
65
+ if !scope.name.is_empty() {
66
+ jm.insert(
67
+ "scope.name".to_string(),
68
+ JsonValue::String(scope.name.clone()),
69
+ );
70
+ }
71
+
72
+ if !scope.version.is_empty() {
73
+ jm.insert(
74
+ "scope.version".to_string(),
75
+ JsonValue::String(scope.version.clone()),
76
+ );
77
+ }
78
+
79
+ if !scope.attributes.is_empty() {
80
+ let scope_attrs = json_from_key_value_list(&scope.attributes);
81
+
82
+ for (key, value) in scope_attrs {
83
+ jm.insert(format!("scope.attributes.{}", key), value);
84
+ }
85
+ }
86
+}
src/crates/jf/flatten_otel/src/logs.rs
new
+126
@@ -0,0 +1,126 @@
1
+use serde_json::{Map as JsonMap, Value as JsonValue};
2
+
3
+use opentelemetry_proto::tonic::{
4
+ collector::logs::v1::ExportLogsServiceRequest, logs::v1::LogRecord,
5
+};
6
+
7
+use crate::{
8
+ json_from_any_value, json_from_instrumentation_scope, json_from_key_value_list,
9
+ json_from_resource,
10
+};
11
+
12
+pub fn json_from_log_record(jm: &mut JsonMap<String, JsonValue>, log_record: &LogRecord) {
13
+ // Add log record fields with "log." prefix
14
+ jm.insert(
15
+ "log.time_unix_nano".to_string(),
16
+ JsonValue::Number(serde_json::Number::from(log_record.time_unix_nano)),
17
+ );
18
+ jm.insert(
19
+ "log.observed_time_unix_nano".to_string(),
20
+ JsonValue::Number(serde_json::Number::from(log_record.observed_time_unix_nano)),
21
+ );
22
+ jm.insert(
23
+ "log.severity_number".to_string(),
24
+ JsonValue::Number(serde_json::Number::from(log_record.severity_number)),
25
+ );
26
+ jm.insert(
27
+ "log.severity_text".to_string(),
28
+ JsonValue::String(log_record.severity_text.clone()),
29
+ );
30
+
31
+ // Add body if present
32
+ if let Some(body) = &log_record.body {
33
+ let mut temp_map = JsonMap::new();
34
+ temp_map.insert("body".to_string(), json_from_any_value(body));
35
+
36
+ let flattened_body = flatten_serde_json::flatten(&temp_map);
37
+ for (key, value) in flattened_body {
38
+ if key != "body" {
39
+ jm.insert(format!("log.{}", key), value);
40
+ }
41
+ }
42
+ }
43
+
44
+ // Add event name
45
+ jm.insert(
46
+ "log.event_name".to_string(),
47
+ JsonValue::String(log_record.event_name.clone()),
48
+ );
49
+
50
+ // Add log record attributes
51
+ let log_attrs = json_from_key_value_list(&log_record.attributes);
52
+ for (key, value) in log_attrs {
53
+ jm.insert(format!("log.attributes.{}", key), value);
54
+ }
55
+
56
+ jm.insert(
57
+ "log.dropped_attributes_count".to_string(),
58
+ JsonValue::Number(serde_json::Number::from(
59
+ log_record.dropped_attributes_count,
60
+ )),
61
+ );
62
+ jm.insert(
63
+ "log.flags".to_string(),
64
+ JsonValue::Number(serde_json::Number::from(log_record.flags)),
65
+ );
66
+
67
+ // Add trace_id and span_id as hex strings
68
+ // if !log_record.trace_id.is_empty() {
69
+ // jm.insert(
70
+ // "log.trace_id".to_string(),
71
+ // JsonValue::String(hex::encode(&log_record.trace_id)),
72
+ // );
73
+ // }
74
+ // if !log_record.span_id.is_empty() {
75
+ // jm.insert(
76
+ // "log.span_id".to_string(),
77
+ // JsonValue::String(hex::encode(&log_record.span_id)),
78
+ // );
79
+ // }
80
+}
81
+
82
+// TODO: this does not belong here, it should be part of the service.
83
+pub fn json_from_export_logs_service_request(request: &ExportLogsServiceRequest) -> JsonValue {
84
+ let mut items = Vec::new();
85
+
86
+ for resource_logs in &request.resource_logs {
87
+ for scope_logs in &resource_logs.scope_logs {
88
+ for log_record in &scope_logs.log_records {
89
+ let mut jm = JsonMap::new();
90
+
91
+ // Add resource information
92
+ if let Some(resource) = resource_logs.resource.as_ref() {
93
+ json_from_resource(&mut jm, resource);
94
+ }
95
+
96
+ // Add resource schema URL
97
+ if !resource_logs.schema_url.is_empty() {
98
+ jm.insert(
99
+ "resource.schema_url".to_string(),
100
+ JsonValue::String(resource_logs.schema_url.clone()),
101
+ );
102
+ }
103
+
104
+ // Add scope information
105
+ if let Some(scope) = scope_logs.scope.as_ref() {
106
+ json_from_instrumentation_scope(&mut jm, scope);
107
+ }
108
+
109
+ // Add scope schema URL
110
+ if !scope_logs.schema_url.is_empty() {
111
+ jm.insert(
112
+ "scope.schema_url".to_string(),
113
+ JsonValue::String(scope_logs.schema_url.clone()),
114
+ );
115
+ }
116
+
117
+ // Add log record information
118
+ json_from_log_record(&mut jm, log_record);
119
+
120
+ items.push(JsonValue::Object(jm));
121
+ }
122
+ }
123
+ }
124
+
125
+ JsonValue::Array(items)
126
+}
src/crates/jf/flatten_otel/src/metrics.rs
new
+284
@@ -0,0 +1,284 @@
1
+use serde_json::{json, Map as JsonMap, Value as JsonValue};
2
+
3
+use opentelemetry_proto::tonic::{
4
+ collector::metrics::v1::ExportMetricsServiceRequest,
5
+ metrics::v1::{
6
+ metric::Data, AggregationTemporality, Gauge, Histogram, HistogramDataPoint, Metric,
7
+ NumberDataPoint, ResourceMetrics, ScopeMetrics, Sum,
8
+ },
9
+};
10
+
11
+use crate::{json_from_instrumentation_scope, json_from_key_value_list, json_from_resource};
12
+
13
+pub fn flatten_metrics_request(
14
+ req: &ExportMetricsServiceRequest,
15
+) -> Vec<JsonMap<String, JsonValue>> {
16
+ req.resource_metrics
17
+ .iter()
18
+ .flat_map(flatten_resource_metrics)
19
+ .collect()
20
+}
21
+
22
+fn flatten_resource_metrics(resource_metrics: &ResourceMetrics) -> Vec<JsonMap<String, JsonValue>> {
23
+ resource_metrics
24
+ .scope_metrics
25
+ .iter()
26
+ .flat_map(|scope_metrics| {
27
+ let mut flattened_metrics = flatten_scope_metrics(scope_metrics);
28
+
29
+ if let Some(resource) = &resource_metrics.resource {
30
+ flattened_metrics
31
+ .iter_mut()
32
+ .for_each(|jm| json_from_resource(jm, resource));
33
+ }
34
+
35
+ flattened_metrics
36
+ })
37
+ .collect()
38
+}
39
+
40
+fn flatten_scope_metrics(scope_metrics: &ScopeMetrics) -> Vec<JsonMap<String, JsonValue>> {
41
+ scope_metrics
42
+ .metrics
43
+ .iter()
44
+ .flat_map(|metric| {
45
+ let mut flattened_metrics = flatten_metric(metric);
46
+
47
+ if let Some(scope) = &scope_metrics.scope {
48
+ flattened_metrics
49
+ .iter_mut()
50
+ .for_each(|jm| json_from_instrumentation_scope(jm, scope));
51
+ }
52
+
53
+ flattened_metrics
54
+ })
55
+ .collect()
56
+}
57
+
58
+fn flatten_metric(metric: &Metric) -> Vec<JsonMap<String, JsonValue>> {
59
+ let Some(data) = metric.data.as_ref() else {
60
+ return Vec::new();
61
+ };
62
+
63
+ let mut flattened_metrics = match data {
64
+ Data::Gauge(gauge) => flatten_gauge(gauge),
65
+ Data::Sum(sum) => flatten_sum(sum),
66
+ Data::Histogram(histogram) => flatten_histogram(histogram),
67
+ Data::ExponentialHistogram(_) | Data::Summary(_) => {
68
+ eprintln!(
69
+ "Summary and exponential histogram metrics are not supported yet ('{}')",
70
+ metric.name
71
+ );
72
+ return Vec::new();
73
+ }
74
+ };
75
+
76
+ for jm in flattened_metrics.iter_mut() {
77
+ // Add metric metadata
78
+ jm.insert(
79
+ "metric.name".to_string(),
80
+ JsonValue::String(metric.name.clone()),
81
+ );
82
+ jm.insert(
83
+ "metric.description".to_string(),
84
+ JsonValue::String(metric.description.clone()),
85
+ );
86
+ jm.insert(
87
+ "metric.unit".to_string(),
88
+ JsonValue::String(metric.unit.clone()),
89
+ );
90
+
91
+ for (key, value) in json_from_key_value_list(&metric.metadata) {
92
+ jm.insert(format!("metric.metadata.{}", key), value);
93
+ }
94
+ }
95
+
96
+ flattened_metrics
97
+}
98
+
99
+fn flatten_gauge(gauge: &Gauge) -> Vec<JsonMap<String, JsonValue>> {
100
+ let mut flattened_metrics = Vec::new();
101
+
102
+ for data_point in &gauge.data_points {
103
+ let mut jm = flatten_number_data_point(data_point);
104
+
105
+ if jm.is_empty() {
106
+ continue;
107
+ }
108
+
109
+ jm.insert(
110
+ "metric.type".to_string(),
111
+ JsonValue::String("gauge".to_string()),
112
+ );
113
+
114
+ flattened_metrics.push(jm);
115
+ }
116
+
117
+ flattened_metrics
118
+}
119
+
120
+fn flatten_sum(sum: &Sum) -> Vec<JsonMap<String, JsonValue>> {
121
+ let mut flattened_metrics = Vec::new();
122
+
123
+ let aggregation_temporality = match sum.aggregation_temporality {
124
+ x if x == AggregationTemporality::Unspecified as i32 => "unspecified",
125
+ x if x == AggregationTemporality::Delta as i32 => "delta",
126
+ x if x == AggregationTemporality::Cumulative as i32 => "cumulative",
127
+ _ => "unknown",
128
+ };
129
+
130
+ for data_point in &sum.data_points {
131
+ let mut jm = flatten_number_data_point(data_point);
132
+
133
+ if jm.is_empty() {
134
+ continue;
135
+ }
136
+
137
+ jm.insert(
138
+ "metric.type".to_string(),
139
+ JsonValue::String("sum".to_string()),
140
+ );
141
+ jm.insert(
142
+ "metric.aggregation_temporality".to_string(),
143
+ JsonValue::String(aggregation_temporality.to_string()),
144
+ );
145
+ jm.insert(
146
+ "metric.is_monotonic".to_string(),
147
+ JsonValue::Bool(sum.is_monotonic),
148
+ );
149
+
150
+ flattened_metrics.push(jm);
151
+ }
152
+
153
+ flattened_metrics
154
+}
155
+
156
+fn flatten_histogram(histogram: &Histogram) -> Vec<JsonMap<String, JsonValue>> {
157
+ let mut flattened_metrics = Vec::new();
158
+
159
+ let aggregation_temporality = match histogram.aggregation_temporality {
160
+ x if x == AggregationTemporality::Unspecified as i32 => "unspecified",
161
+ x if x == AggregationTemporality::Delta as i32 => "delta",
162
+ x if x == AggregationTemporality::Cumulative as i32 => "cumulative",
163
+ _ => "unknown",
164
+ };
165
+
166
+ for data_point in &histogram.data_points {
167
+ let mut bucket_maps = flatten_histogram_data_point(data_point);
168
+
169
+ for jm in bucket_maps.iter_mut() {
170
+ jm.insert(
171
+ "metric.type".to_string(),
172
+ JsonValue::String("histogram".to_string()),
173
+ );
174
+ jm.insert(
175
+ "metric.aggregation_temporality".to_string(),
176
+ JsonValue::String(aggregation_temporality.to_string()),
177
+ );
178
+ }
179
+
180
+ flattened_metrics.extend(bucket_maps);
181
+ }
182
+
183
+ flattened_metrics
184
+}
185
+
186
+fn flatten_number_data_point(ndp: &NumberDataPoint) -> JsonMap<String, JsonValue> {
187
+ let mut jm = JsonMap::new();
188
+
189
+ let Some(value) = &ndp.value else {
190
+ return jm;
191
+ };
192
+
193
+ match value {
194
+ opentelemetry_proto::tonic::metrics::v1::number_data_point::Value::AsDouble(d) => {
195
+ jm.insert("metric.value".to_string(), json!(d));
196
+ }
197
+ opentelemetry_proto::tonic::metrics::v1::number_data_point::Value::AsInt(i) => {
198
+ jm.insert("metric.value".to_string(), JsonValue::Number((*i).into()));
199
+ }
200
+ };
201
+
202
+ jm.insert(
203
+ "metric.start_time_unix_nano".to_string(),
204
+ JsonValue::Number(ndp.start_time_unix_nano.into()),
205
+ );
206
+ jm.insert(
207
+ "metric.time_unix_nano".to_string(),
208
+ JsonValue::Number(ndp.time_unix_nano.into()),
209
+ );
210
+
211
+ for (key, value) in json_from_key_value_list(&ndp.attributes) {
212
+ jm.insert(format!("metric.attributes.{}", key), value);
213
+ }
214
+
215
+ if !ndp.exemplars.is_empty() {
216
+ // todo!...
217
+ }
218
+
219
+ if ndp.flags != 0 {
220
+ jm.insert(
221
+ "metric.flags".to_string(),
222
+ JsonValue::Number(ndp.flags.into()),
223
+ );
224
+ }
225
+
226
+ jm
227
+}
228
+
229
+fn flatten_histogram_data_point(hdp: &HistogramDataPoint) -> Vec<JsonMap<String, JsonValue>> {
230
+ let mut results = Vec::new();
231
+
232
+ if hdp.bucket_counts.is_empty() || hdp.explicit_bounds.is_empty() {
233
+ return results;
234
+ }
235
+
236
+ // Create base map with common fields
237
+ let mut base_map = JsonMap::new();
238
+ base_map.insert(
239
+ "metric.start_time_unix_nano".to_string(),
240
+ JsonValue::Number(hdp.start_time_unix_nano.into()),
241
+ );
242
+ base_map.insert(
243
+ "metric.time_unix_nano".to_string(),
244
+ JsonValue::Number(hdp.time_unix_nano.into()),
245
+ );
246
+
247
+ // Add attributes
248
+ for (key, value) in json_from_key_value_list(&hdp.attributes) {
249
+ base_map.insert(format!("metric.attributes.{}", key), value);
250
+ }
251
+
252
+ // Handle regular buckets
253
+ for (&bound, &count) in hdp.explicit_bounds.iter().zip(hdp.bucket_counts.iter()) {
254
+ let mut bucket_map = base_map.clone();
255
+
256
+ // Set dimension name to bucket identifier
257
+ let bucket_name = format!("{}", bound);
258
+ bucket_map.insert(
259
+ "metric.attributes._nd_dimension".to_string(),
260
+ JsonValue::String("bucket".to_string()),
261
+ );
262
+ bucket_map.insert("bucket".to_string(), JsonValue::String(bucket_name.clone()));
263
+ bucket_map.insert("metric.value".to_string(), JsonValue::from(count));
264
+
265
+ results.push(bucket_map);
266
+ }
267
+
268
+ // Handle +Inf bucket if it exists
269
+ if hdp.bucket_counts.len() > hdp.explicit_bounds.len() {
270
+ let mut inf_map = base_map.clone();
271
+ let inf_count = hdp.bucket_counts[hdp.bucket_counts.len() - 1];
272
+
273
+ inf_map.insert(
274
+ "metric.attributes._nd_dimension".to_string(),
275
+ JsonValue::String("bucket".to_string()),
276
+ );
277
+ inf_map.insert("bucket".to_string(), JsonValue::String("+Inf".to_string()));
278
+ inf_map.insert("metric.value".to_string(), JsonValue::from(inf_count));
279
+
280
+ results.push(inf_map);
281
+ }
282
+
283
+ results
284
+}
src/crates/jf/flog-otel/Cargo.toml
new
+22
@@ -0,0 +1,22 @@
1
+[package]
2
+name = "flog-otel"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+tokio.workspace = true
9
+serde.workspace = true
10
+serde_json.workspace = true
11
+chrono.workspace = true
12
+clap.workspace = true
13
+uuid.workspace = true
14
+anyhow.workspace = true
15
+tracing.workspace = true
16
+tracing-subscriber.workspace = true
17
+tonic.workspace = true
18
+prost.workspace = true
19
+opentelemetry-proto.workspace = true
20
+
21
+[build-dependencies]
22
+tonic-build.workspace = true
src/crates/jf/flog-otel/src/main.rs
new
+312
@@ -0,0 +1,312 @@
1
+use anyhow::Result;
2
+use chrono::{DateTime, Utc};
3
+use clap::Parser;
4
+use opentelemetry_proto::tonic::collector::logs::v1::{
5
+ logs_service_client::LogsServiceClient, ExportLogsServiceRequest,
6
+};
7
+use opentelemetry_proto::tonic::common::v1::{any_value, AnyValue, KeyValue};
8
+use opentelemetry_proto::tonic::logs::v1::{LogRecord, ResourceLogs, ScopeLogs};
9
+use opentelemetry_proto::tonic::resource::v1::Resource;
10
+use serde::Deserialize;
11
+use std::process::Stdio;
12
+use std::sync::Arc;
13
+use tokio::io::{AsyncBufReadExt, BufReader};
14
+use tokio::sync::Mutex;
15
+use tokio::time::{Duration, Instant};
16
+use tracing::{error, info, warn};
17
+use tracing_subscriber;
18
+use uuid::Uuid;
19
+
20
+#[derive(Parser, Debug)]
21
+#[command(author, version, about, long_about = None)]
22
+struct Args {
23
+ #[arg(short, long, default_value = "1048576")]
24
+ rate_limit_bytes: u64,
25
+
26
+ #[arg(short, long, default_value = "http://127.0.0.1:4317")]
27
+ otel_endpoint: String,
28
+
29
+ #[arg(short, long, default_value = "1000")]
30
+ log_count: u32,
31
+
32
+ #[arg(long, default_value = "false")]
33
+ loop_forever: bool,
34
+}
35
+
36
+#[derive(Deserialize, Debug)]
37
+struct FlogEntry {
38
+ host: String,
39
+ #[serde(rename = "user-identifier")]
40
+ user_identifier: String,
41
+ datetime: String,
42
+ method: String,
43
+ request: String,
44
+ protocol: String,
45
+ status: u16,
46
+ bytes: u64,
47
+ referer: String,
48
+}
49
+
50
+struct RateLimiter {
51
+ bytes_sent: u64,
52
+ last_reset: Instant,
53
+ limit: u64,
54
+}
55
+
56
+impl RateLimiter {
57
+ fn new(limit: u64) -> Self {
58
+ Self {
59
+ bytes_sent: 0,
60
+ last_reset: Instant::now(),
61
+ limit,
62
+ }
63
+ }
64
+
65
+ async fn can_send(&mut self, size: u64) -> bool {
66
+ if self.last_reset.elapsed() >= Duration::from_secs(1) {
67
+ self.bytes_sent = 0;
68
+ self.last_reset = Instant::now();
69
+ }
70
+
71
+ if self.bytes_sent + size <= self.limit {
72
+ self.bytes_sent += size;
73
+ true
74
+ } else {
75
+ false
76
+ }
77
+ }
78
+}
79
+
80
+fn parse_flog_datetime(datetime_str: &str) -> Result<DateTime<Utc>> {
81
+ let dt = chrono::DateTime::parse_from_str(datetime_str, "%d/%b/%Y:%H:%M:%S %z")?;
82
+ Ok(dt.with_timezone(&Utc))
83
+}
84
+
85
+fn flog_to_otel(entry: FlogEntry) -> Result<LogRecord> {
86
+ let dt = parse_flog_datetime(&entry.datetime)?;
87
+ let time_unix_nano = dt.timestamp_nanos_opt().unwrap_or(0) as u64;
88
+
89
+ let severity_number = match entry.status {
90
+ 200..=299 => 9, // INFO
91
+ 300..=399 => 5, // DEBUG
92
+ 400..=499 => 13, // WARN
93
+ 500..=599 => 17, // ERROR
94
+ _ => 9, // INFO
95
+ };
96
+
97
+ let log_message = format!(
98
+ "{} {} {} {} {} {}",
99
+ entry.host, entry.method, entry.request, entry.protocol, entry.status, entry.bytes
100
+ );
101
+
102
+ let attributes = vec![
103
+ KeyValue {
104
+ key: "host".to_string(),
105
+ value: Some(AnyValue {
106
+ value: Some(any_value::Value::StringValue(entry.host)),
107
+ }),
108
+ },
109
+ KeyValue {
110
+ key: "method".to_string(),
111
+ value: Some(AnyValue {
112
+ value: Some(any_value::Value::StringValue(entry.method)),
113
+ }),
114
+ },
115
+ KeyValue {
116
+ key: "request".to_string(),
117
+ value: Some(AnyValue {
118
+ value: Some(any_value::Value::StringValue(entry.request)),
119
+ }),
120
+ },
121
+ KeyValue {
122
+ key: "protocol".to_string(),
123
+ value: Some(AnyValue {
124
+ value: Some(any_value::Value::StringValue(entry.protocol)),
125
+ }),
126
+ },
127
+ KeyValue {
128
+ key: "status".to_string(),
129
+ value: Some(AnyValue {
130
+ value: Some(any_value::Value::IntValue(entry.status as i64)),
131
+ }),
132
+ },
133
+ KeyValue {
134
+ key: "response_bytes".to_string(),
135
+ value: Some(AnyValue {
136
+ value: Some(any_value::Value::IntValue(entry.bytes as i64)),
137
+ }),
138
+ },
139
+ KeyValue {
140
+ key: "referer".to_string(),
141
+ value: Some(AnyValue {
142
+ value: Some(any_value::Value::StringValue(entry.referer)),
143
+ }),
144
+ },
145
+ KeyValue {
146
+ key: "user_identifier".to_string(),
147
+ value: Some(AnyValue {
148
+ value: Some(any_value::Value::StringValue(entry.user_identifier)),
149
+ }),
150
+ },
151
+ ];
152
+
153
+ let trace_id_bytes = Uuid::new_v4().as_u128().to_be_bytes().to_vec();
154
+ let span_id_bytes = (Uuid::new_v4().as_u128() as u64).to_be_bytes().to_vec();
155
+
156
+ Ok(LogRecord {
157
+ time_unix_nano,
158
+ severity_number: severity_number as i32,
159
+ severity_text: String::new(),
160
+ body: Some(AnyValue {
161
+ value: Some(any_value::Value::StringValue(log_message)),
162
+ }),
163
+ attributes,
164
+ dropped_attributes_count: 0,
165
+ flags: 0,
166
+ trace_id: trace_id_bytes,
167
+ span_id: span_id_bytes,
168
+ ..Default::default()
169
+ })
170
+}
171
+
172
+async fn send_otel_logs(logs: Vec<LogRecord>, endpoint: &str) -> Result<()> {
173
+ let mut client = LogsServiceClient::connect(endpoint.to_string()).await?;
174
+
175
+ let resource_logs = vec![ResourceLogs {
176
+ resource: Some(Resource {
177
+ attributes: vec![
178
+ KeyValue {
179
+ key: "service.name".to_string(),
180
+ value: Some(AnyValue {
181
+ value: Some(any_value::Value::StringValue(
182
+ "flog-otel-wrapper".to_string(),
183
+ )),
184
+ }),
185
+ },
186
+ KeyValue {
187
+ key: "service.version".to_string(),
188
+ value: Some(AnyValue {
189
+ value: Some(any_value::Value::StringValue("0.1.0".to_string())),
190
+ }),
191
+ },
192
+ ],
193
+ dropped_attributes_count: 0,
194
+ entity_refs: vec![],
195
+ }),
196
+ scope_logs: vec![ScopeLogs {
197
+ scope: Some(
198
+ opentelemetry_proto::tonic::common::v1::InstrumentationScope {
199
+ name: "flog-otel-wrapper".to_string(),
200
+ version: "0.1.0".to_string(),
201
+ attributes: vec![],
202
+ dropped_attributes_count: 0,
203
+ },
204
+ ),
205
+ log_records: logs,
206
+ schema_url: String::new(),
207
+ }],
208
+ schema_url: String::new(),
209
+ }];
210
+
211
+ let request = tonic::Request::new(ExportLogsServiceRequest { resource_logs });
212
+
213
+ let response = client.export(request).await?;
214
+
215
+ if response.get_ref().partial_success.is_some() {
216
+ let partial_success = response.get_ref().partial_success.as_ref().unwrap();
217
+ if partial_success.rejected_log_records > 0 {
218
+ warn!("Some logs were rejected: {}", partial_success.error_message);
219
+ }
220
+ }
221
+
222
+ info!("Successfully sent logs to OTEL collector via gRPC");
223
+ Ok(())
224
+}
225
+
226
+#[tokio::main]
227
+async fn main() -> Result<()> {
228
+ tracing_subscriber::fmt::init();
229
+
230
+ let args = Args::parse();
231
+ let rate_limiter = Arc::new(Mutex::new(RateLimiter::new(args.rate_limit_bytes)));
232
+
233
+ info!(
234
+ "Starting flog-otel-wrapper with rate limit: {} bytes/sec",
235
+ args.rate_limit_bytes
236
+ );
237
+ info!("OTEL endpoint: {}", args.otel_endpoint);
238
+
239
+ loop {
240
+ let log_count_str = args.log_count.to_string();
241
+ let mut flog_args = vec!["-f", "json", "-t", "stdout"];
242
+
243
+ if args.loop_forever {
244
+ flog_args.extend(&["-l"]);
245
+ } else {
246
+ flog_args.extend(&["-n", &log_count_str]);
247
+ }
248
+
249
+ let mut child = tokio::process::Command::new("flog")
250
+ .args(&flog_args)
251
+ .stdout(Stdio::piped())
252
+ .spawn()?;
253
+
254
+ let stdout = child.stdout.take().unwrap();
255
+ let reader = BufReader::new(stdout);
256
+ let mut lines = reader.lines();
257
+
258
+ let mut log_batch = Vec::new();
259
+ const BATCH_SIZE: usize = 100;
260
+
261
+ while let Some(line) = lines.next_line().await? {
262
+ match serde_json::from_str::<FlogEntry>(&line) {
263
+ Ok(entry) => match flog_to_otel(entry) {
264
+ Ok(otel_log) => {
265
+ log_batch.push(otel_log);
266
+
267
+ if log_batch.len() >= BATCH_SIZE {
268
+ let batch_size = (log_batch.len() * 1024) as u64;
269
+
270
+ let mut limiter = rate_limiter.lock().await;
271
+ if limiter.can_send(batch_size).await {
272
+ drop(limiter);
273
+
274
+ if let Err(e) =
275
+ send_otel_logs(log_batch.clone(), &args.otel_endpoint).await
276
+ {
277
+ error!("Failed to send OTEL logs: {}", e);
278
+ }
279
+ log_batch.clear();
280
+ } else {
281
+ drop(limiter);
282
+ info!("Rate limit reached, waiting 1 second...");
283
+ tokio::time::sleep(Duration::from_secs(1)).await;
284
+ }
285
+ }
286
+ }
287
+ Err(e) => error!("Failed to convert to OTEL format: {}", e),
288
+ },
289
+ Err(e) => error!("Failed to parse JSON: {}", e),
290
+ }
291
+ }
292
+
293
+ if !log_batch.is_empty() {
294
+ let batch_size = (log_batch.len() * 1024) as u64;
295
+ let mut limiter = rate_limiter.lock().await;
296
+ if limiter.can_send(batch_size).await {
297
+ drop(limiter);
298
+ if let Err(e) = send_otel_logs(log_batch, &args.otel_endpoint).await {
299
+ error!("Failed to send final OTEL logs: {}", e);
300
+ }
301
+ }
302
+ }
303
+
304
+ child.wait().await?;
305
+
306
+ if !args.loop_forever {
307
+ break;
308
+ }
309
+ }
310
+
311
+ Ok(())
312
+}
src/crates/jf/journal_file/Cargo.toml
new
+19
@@ -0,0 +1,19 @@
1
+[package]
2
+name = "journal_file"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+error = { path = "../error" }
9
+window_manager = { path = "../window_manager" }
10
+memmap2 = { workspace = true }
11
+ruzstd = { workspace = true }
12
+siphasher = { workspace = true }
13
+twox-hash = { workspace = true }
14
+zerocopy = { workspace = true }
15
+hex = { workspace = true }
16
+rand = { workspace = true }
17
+
18
+[dev-dependencies]
19
+tempfile = { workspace = true }
src/crates/jf/journal_file/src/cursor.rs
new
+237
@@ -0,0 +1,237 @@
1
+use crate::{file::JournalFile, filter::FilterExpr, offset_array, offset_array::Direction};
2
+use error::{JournalError, Result};
3
+use std::num::NonZeroU64;
4
+use window_manager::MemoryMap;
5
+
6
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
7
+pub enum Location {
8
+ Head,
9
+ Tail,
10
+ Realtime(u64),
11
+ Monotonic(u64, [u8; 16]),
12
+ Seqnum(u64, Option<[u8; 16]>),
13
+ XorHash(u64),
14
+ ResolvedEntry(NonZeroU64),
15
+}
16
+
17
+impl Default for Location {
18
+ fn default() -> Self {
19
+ Self::Head
20
+ }
21
+}
22
+
23
+#[derive(Debug)]
24
+pub struct JournalCursor {
25
+ pub location: Location,
26
+ pub filter_expr: Option<FilterExpr>,
27
+ pub array_cursor: Option<offset_array::Cursor>,
28
+}
29
+
30
+impl JournalCursor {
31
+ #[allow(clippy::new_without_default)]
32
+ pub fn new() -> Self {
33
+ Self {
34
+ location: Location::Head,
35
+ filter_expr: None,
36
+ array_cursor: None,
37
+ }
38
+ }
39
+
40
+ pub fn set_location(&mut self, location: Location) {
41
+ self.location = location;
42
+ self.array_cursor = None;
43
+ }
44
+
45
+ pub fn set_filter(&mut self, filter_expr: FilterExpr) {
46
+ self.filter_expr = Some(filter_expr);
47
+ // FIXME: should we set cursor to None?
48
+ }
49
+
50
+ pub fn clear_filter(&mut self) {
51
+ self.filter_expr = None;
52
+ self.array_cursor = None;
53
+ self.set_location(Location::Head);
54
+ }
55
+
56
+ pub fn step<M: MemoryMap>(
57
+ &mut self,
58
+ journal_file: &JournalFile<M>,
59
+ direction: Direction,
60
+ ) -> Result<bool> {
61
+ let new_location = if self.filter_expr.is_some() {
62
+ self.resolve_filter_location(journal_file, direction)?
63
+ } else {
64
+ self.resolve_array_cursor(journal_file, direction)?
65
+ };
66
+
67
+ if let Some(location) = new_location {
68
+ self.location = location;
69
+ Ok(true)
70
+ } else {
71
+ Ok(false)
72
+ }
73
+ }
74
+
75
+ pub fn position(&self) -> Result<NonZeroU64> {
76
+ match self.location {
77
+ Location::ResolvedEntry(entry_offset) => Ok(entry_offset),
78
+ _ => Err(JournalError::UnsetCursor),
79
+ }
80
+ }
81
+
82
+ fn resolve_array_cursor<M: MemoryMap>(
83
+ &mut self,
84
+ journal_file: &JournalFile<M>,
85
+ direction: Direction,
86
+ ) -> Result<Option<Location>> {
87
+ let new_location = match (self.location, direction) {
88
+ (Location::Head, Direction::Forward) => {
89
+ let entry_list = journal_file
90
+ .entry_list()
91
+ .ok_or(JournalError::InvalidOffsetArrayOffset)?;
92
+
93
+ let cursor = entry_list.cursor_head();
94
+ if let Some(offset) = cursor.value(journal_file)? {
95
+ self.array_cursor = Some(cursor);
96
+ Some(Location::ResolvedEntry(offset))
97
+ } else {
98
+ None
99
+ }
100
+ }
101
+ (Location::Head, Direction::Backward) => None,
102
+ (Location::Tail, Direction::Forward) => None,
103
+ (Location::Tail, Direction::Backward) => {
104
+ let entry_list = journal_file
105
+ .entry_list()
106
+ .ok_or(JournalError::InvalidOffsetArrayOffset)?;
107
+
108
+ let cursor = entry_list.cursor_tail(journal_file)?;
109
+ if let Some(offset) = cursor.value(journal_file)? {
110
+ self.array_cursor = Some(cursor);
111
+ Some(Location::ResolvedEntry(offset))
112
+ } else {
113
+ None
114
+ }
115
+ }
116
+ (Location::Realtime(realtime), _) => {
117
+ let entry_list = journal_file
118
+ .entry_list()
119
+ .ok_or(JournalError::InvalidOffsetArrayOffset)?;
120
+
121
+ let predicate = |entry_offset| {
122
+ let entry_object = journal_file.entry_ref(entry_offset)?;
123
+ Ok(entry_object.header.realtime < realtime)
124
+ };
125
+
126
+ let cursor = entry_list
127
+ .directed_partition_point(journal_file, predicate, Direction::Forward)?
128
+ .map(Ok)
129
+ .unwrap_or_else(|| entry_list.cursor_tail(journal_file))?;
130
+
131
+ if let Some(offset) = cursor.value(journal_file)? {
132
+ self.array_cursor = Some(cursor);
133
+ Some(Location::ResolvedEntry(offset))
134
+ } else {
135
+ None
136
+ }
137
+ }
138
+ (Location::ResolvedEntry(_), Direction::Forward) => {
139
+ let Some(cursor) = self.array_cursor.unwrap().next(journal_file)? else {
140
+ return Ok(None);
141
+ };
142
+
143
+ if let Some(offset) = cursor.value(journal_file)? {
144
+ self.array_cursor = Some(cursor);
145
+ Some(Location::ResolvedEntry(offset))
146
+ } else {
147
+ None
148
+ }
149
+ }
150
+ (Location::ResolvedEntry(_), Direction::Backward) => {
151
+ let Some(cursor) = self.array_cursor.unwrap().previous(journal_file)? else {
152
+ return Ok(None);
153
+ };
154
+
155
+ if let Some(offset) = cursor.value(journal_file)? {
156
+ self.array_cursor = Some(cursor);
157
+ Some(Location::ResolvedEntry(offset))
158
+ } else {
159
+ None
160
+ }
161
+ }
162
+ _ => {
163
+ unimplemented!()
164
+ }
165
+ };
166
+
167
+ Ok(new_location)
168
+ }
169
+
170
+ fn resolve_filter_location<M: MemoryMap>(
171
+ &mut self,
172
+ journal_file: &JournalFile<M>,
173
+ direction: Direction,
174
+ ) -> Result<Option<Location>> {
175
+ let filter_expr = self.filter_expr.as_mut().unwrap();
176
+
177
+ let resolved_location = match (self.location, direction) {
178
+ (Location::Head, Direction::Forward) => filter_expr
179
+ .head()
180
+ .next(journal_file, NonZeroU64::MIN)?
181
+ .map(Location::ResolvedEntry),
182
+ (Location::Head, Direction::Backward) => None,
183
+ (Location::Tail, Direction::Forward) => None,
184
+ (Location::Tail, Direction::Backward) => filter_expr
185
+ .tail(journal_file)?
186
+ .previous(journal_file, NonZeroU64::MAX)?
187
+ .map(Location::ResolvedEntry),
188
+ (Location::Realtime(realtime), direction) => {
189
+ let entry_list = journal_file
190
+ .entry_list()
191
+ .ok_or(JournalError::InvalidOffsetArrayOffset)?;
192
+
193
+ let predicate = |entry_offset| {
194
+ let entry_object = journal_file.entry_ref(entry_offset)?;
195
+ Ok(entry_object.header.realtime < realtime)
196
+ };
197
+
198
+ let cursor = entry_list
199
+ .directed_partition_point(journal_file, predicate, Direction::Forward)?
200
+ .map(Ok)
201
+ .unwrap_or_else(|| entry_list.cursor_tail(journal_file))?;
202
+
203
+ if let Some(entry_offset) = cursor.value(journal_file)? {
204
+ match direction {
205
+ Direction::Forward => filter_expr
206
+ .head()
207
+ .next(journal_file, entry_offset)?
208
+ .map(Location::ResolvedEntry),
209
+ Direction::Backward => filter_expr
210
+ .tail(journal_file)?
211
+ .previous(journal_file, entry_offset)?
212
+ .map(Location::ResolvedEntry),
213
+ }
214
+ } else {
215
+ None
216
+ }
217
+ }
218
+ (Location::ResolvedEntry(location_offset), Direction::Forward) => filter_expr
219
+ .next(journal_file, location_offset.saturating_add(1))?
220
+ .map(Location::ResolvedEntry),
221
+ (Location::ResolvedEntry(location_offset), Direction::Backward) => {
222
+ if let Some(needle_offset) = NonZeroU64::new(location_offset.get() - 1) {
223
+ filter_expr
224
+ .previous(journal_file, needle_offset)?
225
+ .map(Location::ResolvedEntry)
226
+ } else {
227
+ None
228
+ }
229
+ }
230
+ _ => {
231
+ unimplemented!();
232
+ }
233
+ };
234
+
235
+ Ok(resolved_location)
236
+ }
237
+}
src/crates/jf/journal_file/src/file.rs
new
+1141
@@ -0,0 +1,1141 @@
1
+#![allow(unused_imports, clippy::field_reassign_with_default)]
2
+
3
+use crate::hash;
4
+use crate::object::*;
5
+use crate::offset_array;
6
+use error::{JournalError, Result};
7
+use std::cell::{RefCell, UnsafeCell};
8
+use std::fs::{File, OpenOptions};
9
+use std::marker::PhantomData;
10
+use std::num::NonZero;
11
+use std::num::NonZeroI128;
12
+use std::num::NonZeroU64;
13
+use std::path::Path;
14
+use window_manager::{MemoryMap, MemoryMapMut, WindowManager};
15
+use zerocopy::{ByteSlice, FromBytes, SplitByteSlice, SplitByteSliceMut};
16
+
17
+#[cfg(debug_assertions)]
18
+use std::backtrace::Backtrace;
19
+
20
+use crate::value_guard::ValueGuard;
21
+
22
+fn read_host_file(filename: &str) -> Result<String> {
23
+ match std::fs::read_to_string(filename) {
24
+ Ok(contents) => Ok(contents),
25
+ Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
26
+ let filename = format!("/host/{}", filename);
27
+ Ok(std::fs::read_to_string(filename)?)
28
+ }
29
+ Err(e) => Err(e.into()),
30
+ }
31
+}
32
+
33
+#[cfg(target_os = "linux")]
34
+pub fn load_machine_id() -> Result<[u8; 16]> {
35
+ let content = read_host_file("/etc/machine-id")?;
36
+ let decoded = hex::decode(content.trim()).map_err(|_| JournalError::UuidSerde)?;
37
+ let bytes: [u8; 16] = decoded.try_into().map_err(|_| JournalError::UuidSerde)?;
38
+ Ok(bytes)
39
+}
40
+
41
+#[cfg(target_os = "macos")]
42
+pub fn load_machine_id() -> Result<[u8; 16]> {
43
+ use std::process::Command;
44
+
45
+ let output = Command::new("system_profiler")
46
+ .arg("SPHardwareDataType")
47
+ .output()
48
+ .map_err(|_| JournalError::UuidSerde)?;
49
+
50
+ if output.status.success() {
51
+ let output_str = String::from_utf8_lossy(&output.stdout);
52
+ for line in output_str.lines() {
53
+ if line.contains("Hardware UUID:") {
54
+ if let Some(uuid_str) = line.split("Hardware UUID:").nth(1) {
55
+ let uuid_str = uuid_str.trim();
56
+ let hex_str: String = uuid_str.chars().filter(|c| *c != '-').collect();
57
+
58
+ if hex_str.len() == 32 {
59
+ let mut bytes = [0u8; 16];
60
+ for i in 0..16 {
61
+ let hex_pair = &hex_str[i * 2..i * 2 + 2];
62
+ bytes[i] = u8::from_str_radix(hex_pair, 16)
63
+ .map_err(|_| JournalError::UuidSerde)?;
64
+ }
65
+ return Ok(bytes);
66
+ }
67
+ }
68
+ }
69
+ }
70
+ }
71
+
72
+ Err(JournalError::UuidSerde)
73
+}
74
+
75
+#[cfg(not(any(target_os = "linux", target_os = "macos")))]
76
+pub fn load_machine_id() -> Result<[u8; 16]> {
77
+ Err(JournalError::UuidSerde)
78
+}
79
+
80
+#[cfg(target_os = "linux")]
81
+pub fn load_boot_id() -> Result<[u8; 16]> {
82
+ let content = std::fs::read_to_string("/proc/sys/kernel/random/boot_id")?;
83
+
84
+ let uuid_str = content.trim();
85
+ let hex_str: String = uuid_str.chars().filter(|c| *c != '-').collect();
86
+
87
+ if hex_str.len() != 32 {
88
+ return Err(JournalError::UuidSerde);
89
+ }
90
+
91
+ let mut bytes = [0u8; 16];
92
+ for i in 0..16 {
93
+ let hex_pair = &hex_str[i * 2..i * 2 + 2];
94
+ bytes[i] = u8::from_str_radix(hex_pair, 16).map_err(|_| JournalError::UuidSerde)?;
95
+ }
96
+
97
+ Ok(bytes)
98
+}
99
+
100
+#[cfg(target_os = "macos")]
101
+pub fn load_boot_id() -> Result<[u8; 16]> {
102
+ use std::process::Command;
103
+
104
+ let output = Command::new("sysctl")
105
+ .arg("-n")
106
+ .arg("kern.boottime")
107
+ .output()
108
+ .map_err(|_| JournalError::UuidSerde)?;
109
+
110
+ if output.status.success() {
111
+ let output_str = String::from_utf8_lossy(&output.stdout);
112
+ // Parse "{ sec = 1753988677, usec = 131097 } Thu Jul 31 22:04:37 2025"
113
+ // Extract sec and usec values
114
+ if let (Some(sec_start), Some(usec_start)) =
115
+ (output_str.find("sec = "), output_str.find("usec = "))
116
+ {
117
+ let sec_str = &output_str[sec_start + 6..];
118
+ let sec_end = sec_str.find(',').unwrap_or(sec_str.len());
119
+ let sec_str = &sec_str[..sec_end].trim();
120
+
121
+ let usec_str = &output_str[usec_start + 7..];
122
+ let usec_end = usec_str.find(' ').unwrap_or(usec_str.len());
123
+ let usec_str = &usec_str[..usec_end].trim();
124
+
125
+ if let (Ok(sec), Ok(usec)) = (sec_str.parse::<u64>(), usec_str.parse::<u64>()) {
126
+ // Create a deterministic UUID from boot time
127
+ // Use sec in first 8 bytes, usec in next 4 bytes, pad remaining with zeros
128
+ let mut bytes = [0u8; 16];
129
+ bytes[0..8].copy_from_slice(&sec.to_be_bytes());
130
+ bytes[8..12].copy_from_slice(&(usec as u32).to_be_bytes());
131
+ // bytes[12..16] remain zero-filled for consistency
132
+ return Ok(bytes);
133
+ }
134
+ }
135
+ }
136
+
137
+ Err(JournalError::UuidSerde)
138
+}
139
+
140
+#[cfg(not(any(target_os = "linux", target_os = "macos")))]
141
+pub fn load_boot_id() -> Result<[u8; 16]> {
142
+ Err(JournalError::UuidSerde)
143
+}
144
+
145
+// Size to pad objects to (8 bytes)
146
+const OBJECT_ALIGNMENT: u64 = 8;
147
+
148
+pub trait BucketVisitor<'a> {
149
+ type Object: JournalObject<&'a [u8]> + HashableObject;
150
+ type Output;
151
+
152
+ /// Called for each object in the bucket. Return Some(output) to stop iteration,
153
+ /// or None to continue to the next object.
154
+ fn visit(&mut self, object: &ValueGuard<'a, Self::Object>) -> Result<Option<Self::Output>>;
155
+}
156
+
157
+struct PayloadMatcher<'data, T> {
158
+ payload: &'data [u8],
159
+ hash: u64,
160
+ _phantom: PhantomData<T>,
161
+}
162
+
163
+impl<'data, B: ByteSlice> PayloadMatcher<'data, DataObject<B>> {
164
+ fn data_matcher(payload: &'data [u8], hash: u64) -> Self {
165
+ Self {
166
+ payload,
167
+ hash,
168
+ _phantom: PhantomData::<DataObject<B>>,
169
+ }
170
+ }
171
+}
172
+
173
+impl<'data, B: ByteSlice> PayloadMatcher<'data, FieldObject<B>> {
174
+ fn field_matcher(payload: &'data [u8], hash: u64) -> Self {
175
+ Self {
176
+ payload,
177
+ hash,
178
+ _phantom: PhantomData::<FieldObject<B>>,
179
+ }
180
+ }
181
+}
182
+
183
+impl<'a, T> BucketVisitor<'a> for PayloadMatcher<'_, T>
184
+where
185
+ T: JournalObject<&'a [u8]> + HashableObject,
186
+{
187
+ type Object = T;
188
+ type Output = NonZeroU64;
189
+
190
+ fn visit(&mut self, object: &ValueGuard<'a, Self::Object>) -> Result<Option<Self::Output>> {
191
+ if object.hash() == self.hash && object.get_payload() == self.payload {
192
+ Ok(Some(object.offset()))
193
+ } else {
194
+ Ok(None)
195
+ }
196
+ }
197
+}
198
+
199
+#[derive(Debug, Clone)]
200
+pub struct JournalFileOptions {
201
+ machine_id: [u8; 16],
202
+ boot_id: [u8; 16],
203
+ seqnum_id: [u8; 16],
204
+ file_id: [u8; 16],
205
+ window_size: u64,
206
+ data_hash_table_buckets: usize,
207
+ field_hash_table_buckets: usize,
208
+ enable_keyed_hash: bool,
209
+}
210
+
211
+impl JournalFileOptions {
212
+ pub fn new(
213
+ machine_id: [u8; 16],
214
+ boot_id: [u8; 16],
215
+ seqnum_id: [u8; 16],
216
+ file_id: [u8; 16],
217
+ ) -> Self {
218
+ Self {
219
+ machine_id,
220
+ boot_id,
221
+ seqnum_id,
222
+ file_id,
223
+ window_size: 64 * 1024,
224
+ data_hash_table_buckets: 4096,
225
+ field_hash_table_buckets: 512,
226
+ enable_keyed_hash: true,
227
+ }
228
+ }
229
+
230
+ pub fn with_window_size(mut self, size: u64) -> Self {
231
+ assert_eq!(size % OBJECT_ALIGNMENT, 0);
232
+ assert_eq!(size % 4096, 0, "Window size must be page-aligned");
233
+ self.window_size = size;
234
+ self
235
+ }
236
+
237
+ pub fn with_data_hash_table_buckets(mut self, buckets: usize) -> Self {
238
+ assert!(
239
+ buckets.is_power_of_two(),
240
+ "Hash table buckets should be a power of two"
241
+ );
242
+ self.data_hash_table_buckets = buckets;
243
+ self
244
+ }
245
+
246
+ pub fn with_field_hash_table_buckets(mut self, buckets: usize) -> Self {
247
+ assert!(
248
+ buckets.is_power_of_two(),
249
+ "Hash table buckets should be a power of two"
250
+ );
251
+ self.field_hash_table_buckets = buckets;
252
+ self
253
+ }
254
+
255
+ pub fn with_keyed_hash(mut self, enabled: bool) -> Self {
256
+ self.enable_keyed_hash = enabled;
257
+ self
258
+ }
259
+
260
+ pub fn create<M: MemoryMapMut>(self, path: impl AsRef<Path>) -> Result<JournalFile<M>> {
261
+ JournalFile::create(path, self)
262
+ }
263
+}
264
+
265
+/// Hash table bucket utilization statistics
266
+#[derive(Debug, Clone, Copy)]
267
+pub struct BucketUtilization {
268
+ pub data_occupied: usize,
269
+ pub data_total: usize,
270
+ pub field_occupied: usize,
271
+ pub field_total: usize,
272
+}
273
+
274
+impl BucketUtilization {
275
+ pub fn data_utilization(&self) -> f64 {
276
+ if self.data_total == 0 {
277
+ 0.0
278
+ } else {
279
+ self.data_occupied as f64 / self.data_total as f64
280
+ }
281
+ }
282
+
283
+ pub fn field_utilization(&self) -> f64 {
284
+ if self.field_total == 0 {
285
+ 0.0
286
+ } else {
287
+ self.field_occupied as f64 / self.field_total as f64
288
+ }
289
+ }
290
+}
291
+
292
+///
293
+/// A reader for systemd journal files that efficiently maps small regions of the file into memory.
294
+///
295
+/// # Memory Management
296
+///
297
+/// This implementation uses a window-based memory mapping strategy similar to systemd's original
298
+/// implementation. Instead of mapping the entire file, it maintains a small set of memory-mapped
299
+/// windows and reuses them as needed.
300
+///
301
+/// # Concurrency and Safety
302
+///
303
+/// `JournalFile` uses interior mutability to provide a safe API with the following characteristics:
304
+///
305
+/// - The window manager is wrapped in an `UnsafeCell` to allow mutation through a shared reference.
306
+/// - A single `RefCell<bool>` guards access to ensure only one object can be active at a time.
307
+/// - Methods like `data_object()` return a `ValueGuard<T>` that automatically releases the lock
308
+/// when dropped.
309
+///
310
+/// This design ensures that memory safety is maintained even though references to memory-mapped
311
+/// regions could be invalidated when new objects are created.
312
+pub struct JournalFile<M: MemoryMap> {
313
+ // Persistent memory maps for journal header and data/field hash tables
314
+ header_map: M,
315
+ data_hash_table_map: Option<M>,
316
+ field_hash_table_map: Option<M>,
317
+
318
+ // Window manager for other objects
319
+ window_manager: UnsafeCell<WindowManager<M>>,
320
+
321
+ // Flag to track if any object is in use
322
+ object_in_use: RefCell<bool>,
323
+
324
+ #[cfg(debug_assertions)]
325
+ prev_backtrace: RefCell<Backtrace>,
326
+ #[cfg(debug_assertions)]
327
+ backtrace: RefCell<Backtrace>,
328
+}
329
+
330
+fn map_hash_table<M: MemoryMap>(
331
+ file: &File,
332
+ offset: Option<NonZeroU64>,
333
+ size: Option<NonZeroU64>,
334
+) -> Result<Option<M>> {
335
+ let (Some(offset), Some(size)) = (offset, size) else {
336
+ return Ok(None);
337
+ };
338
+
339
+ if offset.get() <= std::mem::size_of::<JournalHeader>() as u64 {
340
+ return Err(JournalError::InvalidObjectLocation);
341
+ }
342
+ if size.get() <= std::mem::size_of::<ObjectHeader>() as u64 {
343
+ return Err(JournalError::InvalidObjectLocation);
344
+ }
345
+
346
+ let offset = offset.get() - std::mem::size_of::<ObjectHeader>() as u64;
347
+ let size = std::mem::size_of::<ObjectHeader>() as u64 + size.get();
348
+ M::create(file, offset, size).map(Some)
349
+}
350
+
351
+impl<M: MemoryMap> JournalFile<M> {
352
+ pub fn visit_bucket<'a, H, V>(
353
+ &'a self,
354
+ hash_table: Option<H>,
355
+ hash: u64,
356
+ mut visitor: V,
357
+ ) -> Result<Option<V::Output>>
358
+ where
359
+ H: HashTable<Object = V::Object>,
360
+ V: BucketVisitor<'a>,
361
+ {
362
+ let hash_table = hash_table.ok_or(JournalError::MissingHashTable)?;
363
+ let bucket = hash_table.hash_item_ref(hash);
364
+ let mut object_offset = bucket.head_hash_offset;
365
+
366
+ while let Some(offset) = object_offset {
367
+ let object_guard = self.journal_object_ref::<V::Object>(offset)?;
368
+
369
+ if let Some(output) = visitor.visit(&object_guard)? {
370
+ return Ok(Some(output));
371
+ }
372
+
373
+ object_offset = object_guard.next_hash_offset();
374
+ }
375
+
376
+ Ok(None)
377
+ }
378
+
379
+ pub fn open(path: impl AsRef<Path>, window_size: u64) -> Result<Self> {
380
+ debug_assert_eq!(window_size % OBJECT_ALIGNMENT, 0);
381
+
382
+ // Open file and check its size
383
+ let file = OpenOptions::new().read(true).write(false).open(&path)?;
384
+
385
+ // Create a memory map for the header
386
+ let header_size = std::mem::size_of::<JournalHeader>() as u64;
387
+ let header_map = M::create(&file, 0, header_size)?;
388
+ let header = JournalHeader::ref_from_prefix(&header_map).unwrap().0;
389
+ if header.signature != *b"LPKSHHRH" {
390
+ return Err(JournalError::InvalidMagicNumber);
391
+ }
392
+
393
+ // Initialize the hash table maps if they exist
394
+ let data_hash_table_map = map_hash_table(
395
+ &file,
396
+ header.data_hash_table_offset,
397
+ header.data_hash_table_size,
398
+ )?;
399
+ let field_hash_table_map = map_hash_table(
400
+ &file,
401
+ header.field_hash_table_offset,
402
+ header.field_hash_table_size,
403
+ )?;
404
+
405
+ // Create window manager for the rest of the objects
406
+ let window_manager = UnsafeCell::new(WindowManager::new(file, window_size, 32)?);
407
+
408
+ Ok(JournalFile {
409
+ header_map,
410
+ data_hash_table_map,
411
+ field_hash_table_map,
412
+ window_manager,
413
+ object_in_use: RefCell::new(false),
414
+
415
+ #[cfg(debug_assertions)]
416
+ prev_backtrace: RefCell::new(Backtrace::capture()),
417
+ #[cfg(debug_assertions)]
418
+ backtrace: RefCell::new(Backtrace::capture()),
419
+ })
420
+ }
421
+
422
+ pub fn hash(&self, data: &[u8]) -> u64 {
423
+ let is_keyed_hash = self
424
+ .journal_header_ref()
425
+ .has_incompatible_flag(HeaderIncompatibleFlags::KeyedHash);
426
+
427
+ hash::journal_hash_data(
428
+ data,
429
+ is_keyed_hash,
430
+ if is_keyed_hash {
431
+ Some(&self.journal_header_ref().file_id)
432
+ } else {
433
+ None
434
+ },
435
+ )
436
+ }
437
+
438
+ pub fn entry_list(&self) -> Option<offset_array::List> {
439
+ let head_offset = self.journal_header_ref().entry_array_offset?;
440
+ let total_items =
441
+ std::num::NonZeroUsize::new(self.journal_header_ref().n_entries as usize)?;
442
+ Some(offset_array::List::new(head_offset, total_items))
443
+ }
444
+
445
+ pub fn journal_header_ref(&self) -> &JournalHeader {
446
+ JournalHeader::ref_from_prefix(&self.header_map).unwrap().0
447
+ }
448
+
449
+ pub fn data_hash_table_map(&self) -> Option<&M> {
450
+ self.data_hash_table_map.as_ref()
451
+ }
452
+ pub fn field_hash_table_map(&self) -> Option<&M> {
453
+ self.field_hash_table_map.as_ref()
454
+ }
455
+
456
+ pub fn data_hash_table_ref(&self) -> Option<DataHashTable<&[u8]>> {
457
+ self.data_hash_table_map
458
+ .as_ref()
459
+ .and_then(|m| DataHashTable::<&[u8]>::from_data(m, false))
460
+ }
461
+
462
+ pub fn field_hash_table_ref(&self) -> Option<FieldHashTable<&[u8]>> {
463
+ self.field_hash_table_map
464
+ .as_ref()
465
+ .and_then(|m| FieldHashTable::<&[u8]>::from_data(m, false))
466
+ }
467
+
468
+ pub fn object_header_ref(&self, position: NonZeroU64) -> Result<&ObjectHeader> {
469
+ let size_needed = std::mem::size_of::<ObjectHeader>() as u64;
470
+ let window_manager = unsafe { &mut *self.window_manager.get() };
471
+ let header_slice = window_manager.get_slice(position.get(), size_needed)?;
472
+ Ok(ObjectHeader::ref_from_bytes(header_slice).unwrap())
473
+ }
474
+
475
+ fn object_data_ref(&self, offset: NonZeroU64, size_needed: u64) -> Result<&[u8]> {
476
+ let window_manager = unsafe { &mut *self.window_manager.get() };
477
+ let object_slice = window_manager.get_slice(offset.get(), size_needed)?;
478
+ Ok(object_slice)
479
+ }
480
+
481
+ fn journal_object_ref<'a, T>(&'a self, offset: NonZeroU64) -> Result<ValueGuard<'a, T>>
482
+ where
483
+ T: JournalObject<&'a [u8]>,
484
+ {
485
+ // Check if any object is already in use
486
+ let mut is_in_use = self.object_in_use.borrow_mut();
487
+ if *is_in_use {
488
+ #[cfg(debug_assertions)]
489
+ {
490
+ eprintln!(
491
+ "Value is in use. Current Backtrace: {:?}, Previous Backtrace: {:?}",
492
+ self.backtrace.borrow().to_string(),
493
+ self.prev_backtrace.borrow().to_string()
494
+ );
495
+ }
496
+ return Err(JournalError::ValueGuardInUse);
497
+ }
498
+
499
+ #[cfg(debug_assertions)]
500
+ {
501
+ self.backtrace.swap(&self.prev_backtrace);
502
+ let _ = self.backtrace.replace(Backtrace::force_capture());
503
+ }
504
+
505
+ let is_compact = self
506
+ .journal_header_ref()
507
+ .has_incompatible_flag(HeaderIncompatibleFlags::Compact);
508
+
509
+ let size_needed = {
510
+ let header = self.object_header_ref(offset)?;
511
+ header.size
512
+ };
513
+
514
+ let data = self.object_data_ref(offset, size_needed)?;
515
+ let Some(value) = T::from_data(data, is_compact) else {
516
+ return Err(JournalError::ZerocopyFailure);
517
+ };
518
+
519
+ // Mark as in use
520
+ *is_in_use = true;
521
+
522
+ Ok(ValueGuard::new(offset, value, &self.object_in_use))
523
+ }
524
+
525
+ pub fn offset_array_ref(
526
+ &self,
527
+ offset: NonZeroU64,
528
+ ) -> Result<ValueGuard<'_, OffsetArrayObject<&[u8]>>> {
529
+ self.journal_object_ref(offset)
530
+ }
531
+
532
+ pub fn field_ref(&self, offset: NonZeroU64) -> Result<ValueGuard<'_, FieldObject<&[u8]>>> {
533
+ self.journal_object_ref(offset)
534
+ }
535
+
536
+ pub fn entry_ref(&self, offset: NonZeroU64) -> Result<ValueGuard<'_, EntryObject<&[u8]>>> {
537
+ self.journal_object_ref(offset)
538
+ }
539
+
540
+ pub fn data_ref(&self, offset: NonZeroU64) -> Result<ValueGuard<'_, DataObject<&[u8]>>> {
541
+ self.journal_object_ref(offset)
542
+ }
543
+
544
+ pub fn tag_ref(&self, offset: NonZeroU64) -> Result<ValueGuard<'_, TagObject<&[u8]>>> {
545
+ self.journal_object_ref(offset)
546
+ }
547
+
548
+ pub fn find_data_offset(&self, hash: u64, payload: &[u8]) -> Result<Option<NonZeroU64>> {
549
+ let visitor = PayloadMatcher::data_matcher(payload, hash);
550
+ self.visit_bucket(self.data_hash_table_ref(), hash, visitor)
551
+ }
552
+
553
+ pub fn find_field_offset(&self, hash: u64, payload: &[u8]) -> Result<Option<NonZeroU64>> {
554
+ let visitor = PayloadMatcher::field_matcher(payload, hash);
555
+ self.visit_bucket(self.field_hash_table_ref(), hash, visitor)
556
+ }
557
+
558
+ /// Run a directed partition point query on a data object's entry array
559
+ ///
560
+ /// This finds the first/last entry (depending on direction) that satisfies the given predicate
561
+ /// in the entry array chain of the data object.
562
+ pub fn data_object_directed_partition_point<F>(
563
+ &self,
564
+ data_offset: NonZeroU64,
565
+ predicate: F,
566
+ direction: offset_array::Direction,
567
+ ) -> Result<Option<NonZeroU64>>
568
+ where
569
+ F: Fn(NonZeroU64) -> Result<bool>,
570
+ {
571
+ let Some(cursor) = self.data_ref(data_offset)?.inlined_cursor() else {
572
+ return Ok(None);
573
+ };
574
+
575
+ let Some(best_match) = cursor.directed_partition_point(self, predicate, direction)? else {
576
+ return Ok(None);
577
+ };
578
+
579
+ best_match.value(self)
580
+ }
581
+
582
+ /// Creates an iterator over all field objects in the field hash table
583
+ pub fn fields(&self) -> FieldIterator<'_, M> {
584
+ // Get the field hash table
585
+ let field_hash_table = self.field_hash_table_ref();
586
+
587
+ // Initialize with the first bucket
588
+ let mut iterator = FieldIterator {
589
+ journal: self,
590
+ field_hash_table,
591
+ current_bucket_index: 0,
592
+ next_field_offset: None,
593
+ };
594
+
595
+ // Find the first non-empty bucket
596
+ iterator.advance_to_next_nonempty_bucket();
597
+
598
+ iterator
599
+ }
600
+
601
+ /// Creates an iterator over all DATA objects for the specified field
602
+ pub fn field_data_objects<'a>(
603
+ &'a self,
604
+ field_name: &'a [u8],
605
+ ) -> Result<FieldDataIterator<'a, M>> {
606
+ // Find the field offset by name
607
+ let field_hash = self.hash(field_name);
608
+ let Some(field_offset) = self.find_field_offset(field_hash, field_name)? else {
609
+ return Ok(FieldDataIterator {
610
+ journal: self,
611
+ current_data_offset: None,
612
+ });
613
+ };
614
+
615
+ // Get the field object to access its head_data_offset
616
+ let field_guard = self.field_ref(field_offset)?;
617
+ let head_data_offset = field_guard.header.head_data_offset;
618
+
619
+ // Create the iterator
620
+ Ok(FieldDataIterator {
621
+ journal: self,
622
+ current_data_offset: head_data_offset,
623
+ })
624
+ }
625
+
626
+ /// Creates an iterator over all DATA objects for a specific entry
627
+ pub fn entry_data_objects(&self, entry_offset: NonZeroU64) -> Result<EntryDataIterator<'_, M>> {
628
+ // Get the entry object to determine how many data items it has
629
+ let entry_guard = self.entry_ref(entry_offset)?;
630
+
631
+ // Get the total number of items
632
+ let total_items = match &entry_guard.items {
633
+ EntryItemsType::Regular(items) => items.len(),
634
+ EntryItemsType::Compact(items) => items.len(),
635
+ };
636
+
637
+ // Create the iterator
638
+ Ok(EntryDataIterator {
639
+ journal: self,
640
+ entry_offset: Some(entry_offset),
641
+ current_index: 0,
642
+ total_items,
643
+ })
644
+ }
645
+
646
+ /// Get hash table bucket utilization statistics
647
+ pub fn bucket_utilization(&self) -> Option<BucketUtilization> {
648
+ let data_hash_table = self.data_hash_table_ref()?;
649
+ let data_total = data_hash_table.items.len();
650
+ let data_occupied = data_hash_table
651
+ .items
652
+ .iter()
653
+ .filter(|item| item.head_hash_offset.is_some())
654
+ .count();
655
+
656
+ let field_hash_table = self.field_hash_table_ref()?;
657
+ let field_total = field_hash_table.items.len();
658
+ let field_occupied = field_hash_table
659
+ .items
660
+ .iter()
661
+ .filter(|item| item.head_hash_offset.is_some())
662
+ .count();
663
+
664
+ Some(BucketUtilization {
665
+ data_occupied,
666
+ data_total,
667
+ field_occupied,
668
+ field_total,
669
+ })
670
+ }
671
+}
672
+
673
+impl<M: MemoryMapMut> JournalFile<M> {
674
+ pub fn create(path: impl AsRef<Path>, options: JournalFileOptions) -> Result<Self> {
675
+ let file = OpenOptions::new()
676
+ .create(true)
677
+ .truncate(true)
678
+ .read(true)
679
+ .write(true)
680
+ .open(&path)?;
681
+
682
+ // Calculate hash table sizes
683
+ let data_hash_table_size =
684
+ options.data_hash_table_buckets * std::mem::size_of::<HashItem>();
685
+ let field_hash_table_size =
686
+ options.field_hash_table_buckets * std::mem::size_of::<HashItem>();
687
+
688
+ // Calculate hash table offsets
689
+ let data_hash_table_offset = std::mem::size_of::<JournalHeader>() as u64
690
+ + std::mem::size_of::<ObjectHeader>() as u64;
691
+ let field_hash_table_offset = data_hash_table_offset
692
+ + data_hash_table_size as u64
693
+ + std::mem::size_of::<ObjectHeader>() as u64;
694
+
695
+ // Create header with options configuration
696
+ let mut header = JournalHeader::default();
697
+ header.signature = *b"LPKSHHRH";
698
+
699
+ // Set flags based on options configuration
700
+ if options.enable_keyed_hash {
701
+ header.incompatible_flags |= HeaderIncompatibleFlags::KeyedHash as u32;
702
+ }
703
+
704
+ // Set hash table configuration
705
+ header.data_hash_table_offset = NonZeroU64::new(data_hash_table_offset);
706
+ header.data_hash_table_size = NonZeroU64::new(data_hash_table_size as u64);
707
+ header.field_hash_table_offset = NonZeroU64::new(field_hash_table_offset);
708
+ header.field_hash_table_size = NonZeroU64::new(field_hash_table_size as u64);
709
+
710
+ // Set other header fields
711
+ header.tail_object_offset =
712
+ NonZeroU64::new(data_hash_table_offset + data_hash_table_size as u64);
713
+ header.header_size = std::mem::size_of::<JournalHeader>() as u64;
714
+ header.n_objects = 2;
715
+ header.arena_size =
716
+ field_hash_table_offset + field_hash_table_size as u64 - header.header_size;
717
+
718
+ // Set IDs from options
719
+ header.machine_id = options.machine_id;
720
+ header.tail_entry_boot_id = options.boot_id;
721
+ header.file_id = options.file_id;
722
+ header.seqnum_id = options.seqnum_id;
723
+
724
+ // Create memory maps for hash tables
725
+ let data_hash_table_map = map_hash_table(
726
+ &file,
727
+ header.data_hash_table_offset,
728
+ header.data_hash_table_size,
729
+ )?;
730
+ let field_hash_table_map = map_hash_table(
731
+ &file,
732
+ header.field_hash_table_offset,
733
+ header.field_hash_table_size,
734
+ )?;
735
+
736
+ // Create header memory map and write header
737
+ let header_size = std::mem::size_of::<JournalHeader>() as u64;
738
+ let mut header_map = M::create(&file, 0, header_size)?;
739
+ {
740
+ let header_mut = JournalHeader::mut_from_prefix(&mut header_map).unwrap().0;
741
+ *header_mut = header;
742
+ }
743
+
744
+ // Create window manager for the rest of the objects
745
+ let window_manager = UnsafeCell::new(WindowManager::new(file, options.window_size, 32)?);
746
+
747
+ let jf = JournalFile {
748
+ header_map,
749
+ data_hash_table_map,
750
+ field_hash_table_map,
751
+ window_manager,
752
+ object_in_use: RefCell::new(false),
753
+
754
+ #[cfg(debug_assertions)]
755
+ prev_backtrace: RefCell::new(Backtrace::capture()),
756
+ #[cfg(debug_assertions)]
757
+ backtrace: RefCell::new(Backtrace::capture()),
758
+ };
759
+
760
+ // write data hash table object header info
761
+ {
762
+ let offset = NonZeroU64::new(
763
+ header.data_hash_table_offset.unwrap().get()
764
+ - std::mem::size_of::<ObjectHeader>() as u64,
765
+ )
766
+ .unwrap();
767
+ let size = header.data_hash_table_size.unwrap().get()
768
+ + std::mem::size_of::<ObjectHeader>() as u64;
769
+
770
+ let object_header = jf.object_header_mut(offset)?;
771
+ object_header.type_ = ObjectType::DataHashTable as u8;
772
+ object_header.size = size
773
+ }
774
+
775
+ // write field hash table object header info
776
+ {
777
+ let offset = NonZeroU64::new(
778
+ header.field_hash_table_offset.unwrap().get()
779
+ - std::mem::size_of::<ObjectHeader>() as u64,
780
+ )
781
+ .unwrap();
782
+ let size = header.field_hash_table_size.unwrap().get()
783
+ + std::mem::size_of::<ObjectHeader>() as u64;
784
+
785
+ let object_header = jf.object_header_mut(offset)?;
786
+ object_header.type_ = ObjectType::FieldHashTable as u8;
787
+ object_header.size = size
788
+ }
789
+
790
+ Ok(jf)
791
+ }
792
+
793
+ pub fn journal_header_mut(&mut self) -> &mut JournalHeader {
794
+ JournalHeader::mut_from_prefix(&mut self.header_map)
795
+ .unwrap()
796
+ .0
797
+ }
798
+
799
+ pub fn data_hash_table_mut(&mut self) -> Option<DataHashTable<&mut [u8]>> {
800
+ self.data_hash_table_map
801
+ .as_mut()
802
+ .and_then(|m| DataHashTable::<&mut [u8]>::from_data_mut(m, false))
803
+ }
804
+
805
+ pub fn field_hash_table_mut(&mut self) -> Option<FieldHashTable<&mut [u8]>> {
806
+ self.field_hash_table_map
807
+ .as_mut()
808
+ .and_then(|m| FieldHashTable::<&mut [u8]>::from_data_mut(m, false))
809
+ }
810
+
811
+ fn object_header_mut(&self, offset: NonZeroU64) -> Result<&mut ObjectHeader> {
812
+ let size_needed = std::mem::size_of::<ObjectHeader>() as u64;
813
+ let window_manager = unsafe { &mut *self.window_manager.get() };
814
+ let header_slice = window_manager.get_slice_mut(offset.get(), size_needed)?;
815
+ Ok(ObjectHeader::mut_from_bytes(header_slice).unwrap())
816
+ }
817
+
818
+ fn object_data_mut(&self, offset: NonZeroU64, size_needed: u64) -> Result<&mut [u8]> {
819
+ let window_manager = unsafe { &mut *self.window_manager.get() };
820
+ let object_slice = window_manager.get_slice_mut(offset.get(), size_needed)?;
821
+ Ok(object_slice)
822
+ }
823
+
824
+ fn journal_object_mut<'a, T>(
825
+ &'a self,
826
+ type_: ObjectType,
827
+ offset: NonZeroU64,
828
+ size: Option<u64>,
829
+ ) -> Result<ValueGuard<'a, T>>
830
+ where
831
+ T: JournalObjectMut<&'a mut [u8]>,
832
+ {
833
+ // Check if any object is already in use
834
+ let mut is_in_use = self.object_in_use.borrow_mut();
835
+ if *is_in_use {
836
+ #[cfg(debug_assertions)]
837
+ {
838
+ eprintln!(
839
+ "Value is in use. Current Backtrace: {:?}, Previous Backtrace: {:?}",
840
+ self.backtrace.borrow().to_string(),
841
+ self.prev_backtrace.borrow().to_string()
842
+ );
843
+ }
844
+ return Err(JournalError::ValueGuardInUse);
845
+ }
846
+
847
+ #[cfg(debug_assertions)]
848
+ {
849
+ self.backtrace.swap(&self.prev_backtrace);
850
+ let _ = self.backtrace.replace(Backtrace::force_capture());
851
+ }
852
+
853
+ let is_compact = self
854
+ .journal_header_ref()
855
+ .has_incompatible_flag(HeaderIncompatibleFlags::Compact);
856
+
857
+ let size_needed = match size {
858
+ Some(size) => {
859
+ let header = self.object_header_mut(offset)?;
860
+ header.type_ = type_ as u8;
861
+ header.size = size;
862
+ size
863
+ }
864
+ None => {
865
+ let header = self.object_header_ref(offset)?;
866
+ if header.type_ != type_ as u8 {
867
+ return Err(JournalError::InvalidObjectType);
868
+ }
869
+ header.size
870
+ }
871
+ };
872
+
873
+ let data = self.object_data_mut(offset, size_needed)?;
874
+ let value = T::from_data_mut(data, is_compact).ok_or(JournalError::ZerocopyFailure)?;
875
+
876
+ // Mark as in use
877
+ *is_in_use = true;
878
+ Ok(ValueGuard::new(offset, value, &self.object_in_use))
879
+ }
880
+
881
+ pub fn offset_array_mut(
882
+ &self,
883
+ offset: NonZeroU64,
884
+ capacity: Option<NonZeroU64>,
885
+ ) -> Result<ValueGuard<'_, OffsetArrayObject<&mut [u8]>>> {
886
+ let size = capacity.map(|c| {
887
+ let mut size = std::mem::size_of::<OffsetArrayObjectHeader>() as u64;
888
+
889
+ let is_compact = self
890
+ .journal_header_ref()
891
+ .has_incompatible_flag(HeaderIncompatibleFlags::Compact);
892
+ if is_compact {
893
+ size += c.get() * std::mem::size_of::<u32>() as u64;
894
+ } else {
895
+ size += c.get() * std::mem::size_of::<u64>() as u64;
896
+ }
897
+
898
+ size
899
+ });
900
+
901
+ let offset_array = self.journal_object_mut(ObjectType::EntryArray, offset, size);
902
+ offset_array
903
+ }
904
+
905
+ pub fn field_mut(
906
+ &self,
907
+ offset: NonZeroU64,
908
+ size: Option<u64>,
909
+ ) -> Result<ValueGuard<'_, FieldObject<&mut [u8]>>> {
910
+ let size = size.map(|n| std::mem::size_of::<FieldObjectHeader>() as u64 + n);
911
+ self.journal_object_mut(ObjectType::Field, offset, size)
912
+ }
913
+
914
+ pub fn entry_mut(
915
+ &self,
916
+ offset: NonZeroU64,
917
+ size: Option<u64>,
918
+ ) -> Result<ValueGuard<'_, EntryObject<&mut [u8]>>> {
919
+ let size = size.map(|n| std::mem::size_of::<DataObjectHeader>() as u64 + n);
920
+ self.journal_object_mut(ObjectType::Entry, offset, size)
921
+ }
922
+
923
+ pub fn data_mut(
924
+ &self,
925
+ offset: NonZeroU64,
926
+ size: Option<u64>,
927
+ ) -> Result<ValueGuard<'_, DataObject<&mut [u8]>>> {
928
+ let size = size.map(|n| std::mem::size_of::<DataObjectHeader>() as u64 + n);
929
+ self.journal_object_mut(ObjectType::Data, offset, size)
930
+ }
931
+
932
+ pub fn tag_mut(
933
+ &self,
934
+ offset: NonZeroU64,
935
+ new: bool,
936
+ ) -> Result<ValueGuard<'_, TagObject<&mut [u8]>>> {
937
+ let size = if new {
938
+ Some(std::mem::size_of::<TagObjectHeader>() as u64)
939
+ } else {
940
+ None
941
+ };
942
+ self.journal_object_mut(ObjectType::Tag, offset, size)
943
+ }
944
+}
945
+
946
+macro_rules! impl_hash_table_set_tail_offset {
947
+ (
948
+ $method_name:ident,
949
+ $hash_table_ref:ident,
950
+ $hash_table_mut:ident,
951
+ $object_mut:ident
952
+ ) => {
953
+ pub fn $method_name(&mut self, hash: u64, object_offset: NonZeroU64) -> Result<()> {
954
+ let hash_item = {
955
+ let Some(ht) = self.$hash_table_ref() else {
956
+ return Err(JournalError::MissingHashTable);
957
+ };
958
+ *ht.hash_item_ref(hash)
959
+ };
960
+
961
+ if let Some(tail_hash_offset) = hash_item.tail_hash_offset {
962
+ let mut tail_object = self.$object_mut(tail_hash_offset, None)?;
963
+ tail_object.set_next_hash_offset(object_offset);
964
+ }
965
+
966
+ let Some(mut ht) = self.$hash_table_mut() else {
967
+ return Err(JournalError::MissingHashTable);
968
+ };
969
+
970
+ let hash_item = ht.hash_item_mut(hash);
971
+ if hash_item.head_hash_offset.is_none() {
972
+ hash_item.head_hash_offset = Some(object_offset);
973
+ }
974
+ hash_item.tail_hash_offset = Some(object_offset);
975
+
976
+ Ok(())
977
+ }
978
+ };
979
+}
980
+
981
+impl<M: MemoryMapMut> JournalFile<M> {
982
+ impl_hash_table_set_tail_offset!(
983
+ data_hash_table_set_tail_offset,
984
+ data_hash_table_ref,
985
+ data_hash_table_mut,
986
+ data_mut
987
+ );
988
+
989
+ impl_hash_table_set_tail_offset!(
990
+ field_hash_table_set_tail_offset,
991
+ field_hash_table_ref,
992
+ field_hash_table_mut,
993
+ field_mut
994
+ );
995
+}
996
+
997
+/// Iterator that walks through all field objects in the field hash table
998
+pub struct FieldIterator<'a, M: MemoryMap> {
999
+ journal: &'a JournalFile<M>,
1000
+ field_hash_table: Option<FieldHashTable<&'a [u8]>>,
1001
+ current_bucket_index: usize,
1002
+ next_field_offset: Option<NonZeroU64>,
1003
+}
1004
+
1005
+impl<M: MemoryMap> FieldIterator<'_, M> {
1006
+ /// Advances to the next non-empty bucket
1007
+ fn advance_to_next_nonempty_bucket(&mut self) {
1008
+ // If we don't have a hash table, there's nothing to iterate
1009
+ let Some(hash_table) = &self.field_hash_table else {
1010
+ return;
1011
+ };
1012
+
1013
+ let items = &hash_table.items;
1014
+
1015
+ // Find the next non-empty bucket
1016
+ while self.current_bucket_index < items.len() {
1017
+ let bucket = items[self.current_bucket_index];
1018
+ if bucket.head_hash_offset.is_some() {
1019
+ self.next_field_offset = bucket.head_hash_offset;
1020
+ return;
1021
+ }
1022
+ self.current_bucket_index += 1;
1023
+ }
1024
+
1025
+ // No more non-empty buckets
1026
+ self.next_field_offset = None;
1027
+ }
1028
+}
1029
+
1030
+impl<'a, M: MemoryMap> Iterator for FieldIterator<'a, M> {
1031
+ type Item = Result<ValueGuard<'a, FieldObject<&'a [u8]>>>;
1032
+
1033
+ fn next(&mut self) -> Option<Self::Item> {
1034
+ let offset = self.next_field_offset?;
1035
+
1036
+ match self.journal.field_ref(offset) {
1037
+ Ok(field_guard) => {
1038
+ // Get the next field offset before we return the guard
1039
+ self.next_field_offset = field_guard.header.next_hash_offset;
1040
+
1041
+ // If we've reached the end of the chain, move to the next bucket
1042
+ if self.next_field_offset.is_none() {
1043
+ self.current_bucket_index += 1;
1044
+ self.advance_to_next_nonempty_bucket();
1045
+ }
1046
+
1047
+ Some(Ok(field_guard))
1048
+ }
1049
+ Err(e) => {
1050
+ self.next_field_offset = None;
1051
+ Some(Err(e))
1052
+ }
1053
+ }
1054
+ }
1055
+}
1056
+
1057
+/// Iterator that walks through all DATA objects for a specific field
1058
+pub struct FieldDataIterator<'a, M: MemoryMap> {
1059
+ journal: &'a JournalFile<M>,
1060
+ current_data_offset: Option<NonZeroU64>,
1061
+}
1062
+
1063
+impl<'a, M: MemoryMap> Iterator for FieldDataIterator<'a, M> {
1064
+ type Item = Result<ValueGuard<'a, DataObject<&'a [u8]>>>;
1065
+
1066
+ fn next(&mut self) -> Option<Self::Item> {
1067
+ let data_offset = self.current_data_offset?;
1068
+
1069
+ match self.journal.data_ref(data_offset) {
1070
+ Ok(data_guard) => {
1071
+ // Get the next data offset before we return the guard
1072
+ self.current_data_offset = data_guard.header.next_field_offset;
1073
+ Some(Ok(data_guard))
1074
+ }
1075
+ Err(e) => {
1076
+ self.current_data_offset = None;
1077
+ Some(Err(e))
1078
+ }
1079
+ }
1080
+ }
1081
+}
1082
+
1083
+/// Iterator that walks through all DATA objects for a specific entry
1084
+pub struct EntryDataIterator<'a, M: MemoryMap> {
1085
+ journal: &'a JournalFile<M>,
1086
+ entry_offset: Option<NonZeroU64>,
1087
+ current_index: usize,
1088
+ total_items: usize,
1089
+}
1090
+
1091
+impl<'a, M: MemoryMap> Iterator for EntryDataIterator<'a, M> {
1092
+ type Item = Result<ValueGuard<'a, DataObject<&'a [u8]>>>;
1093
+
1094
+ fn next(&mut self) -> Option<Self::Item> {
1095
+ let entry_offset = self.entry_offset?;
1096
+
1097
+ // If we've reached the end of the data indices, return None
1098
+ if self.current_index >= self.total_items {
1099
+ return None;
1100
+ }
1101
+
1102
+ // Get the entry object to access the data offset
1103
+ match self.journal.entry_ref(entry_offset) {
1104
+ Ok(entry_guard) => {
1105
+ let idx = self.current_index;
1106
+ self.current_index += 1;
1107
+
1108
+ let data_offset = match &entry_guard.items {
1109
+ EntryItemsType::Regular(items) => {
1110
+ if idx >= items.len() {
1111
+ return None;
1112
+ }
1113
+ items[idx].object_offset
1114
+ }
1115
+ EntryItemsType::Compact(items) => {
1116
+ if idx >= items.len() {
1117
+ return None;
1118
+ }
1119
+ items[idx].object_offset as u64
1120
+ }
1121
+ };
1122
+
1123
+ let data_offset = NonZeroU64::new(data_offset)?;
1124
+
1125
+ // Drop the entry guard before obtaining the data object
1126
+ drop(entry_guard);
1127
+
1128
+ // Try to get the data object
1129
+ match self.journal.data_ref(data_offset) {
1130
+ Ok(data_guard) => Some(Ok(data_guard)),
1131
+ Err(e) => Some(Err(e)),
1132
+ }
1133
+ }
1134
+ Err(e) => {
1135
+ // If we can't read the entry, return the error and stop iteration
1136
+ self.current_index = self.total_items;
1137
+ Some(Err(e))
1138
+ }
1139
+ }
1140
+ }
1141
+}
src/crates/jf/journal_file/src/filter.rs
new
+438
@@ -0,0 +1,438 @@
1
+use crate::{
2
+ file::JournalFile,
3
+ offset_array::{Direction, InlinedCursor},
4
+};
5
+use error::{JournalError, Result};
6
+use std::num::NonZeroU64;
7
+use window_manager::MemoryMap;
8
+
9
+#[derive(Clone, Debug)]
10
+pub enum FilterExpr {
11
+ Match(u64, Option<InlinedCursor>),
12
+ Conjunction(Vec<FilterExpr>),
13
+ Disjunction(Vec<FilterExpr>),
14
+}
15
+
16
+impl FilterExpr {
17
+ pub fn lookup<M: MemoryMap>(
18
+ &self,
19
+ journal_file: &JournalFile<M>,
20
+ needle_offset: u64,
21
+ direction: Direction,
22
+ ) -> Result<Option<u64>> {
23
+ let Some(needle_offset) = NonZeroU64::new(needle_offset) else {
24
+ return Err(JournalError::InvalidOffset);
25
+ };
26
+
27
+ let predicate =
28
+ move |entry_offset: NonZeroU64| -> Result<bool> { Ok(entry_offset < needle_offset) };
29
+
30
+ match self {
31
+ FilterExpr::Match(data_offset, _) => {
32
+ let Some(data_offset) = NonZeroU64::new(*data_offset) else {
33
+ return Err(JournalError::InvalidOffset);
34
+ };
35
+ let entry_offset = journal_file.data_object_directed_partition_point(
36
+ data_offset,
37
+ predicate,
38
+ direction,
39
+ )?;
40
+ Ok(entry_offset.map(|x| x.get()))
41
+ }
42
+ FilterExpr::Conjunction(filter_exprs) => {
43
+ let mut current_offset = needle_offset;
44
+
45
+ loop {
46
+ let previous_offset = current_offset;
47
+
48
+ for filter_expr in filter_exprs {
49
+ if direction == Direction::Backward {
50
+ current_offset = current_offset.saturating_add(1);
51
+ }
52
+
53
+ match filter_expr.lookup(journal_file, current_offset.get(), direction)? {
54
+ Some(new_offset) => {
55
+ if new_offset == 0 {
56
+ panic!("Wtf");
57
+ }
58
+ current_offset = NonZeroU64::new(new_offset).unwrap();
59
+ }
60
+ None => return Ok(None),
61
+ }
62
+ }
63
+
64
+ if current_offset == previous_offset {
65
+ return Ok(Some(current_offset.get()));
66
+ }
67
+ }
68
+ }
69
+ FilterExpr::Disjunction(filter_exprs) => {
70
+ let cmp = match direction {
71
+ Direction::Forward => std::cmp::min,
72
+ Direction::Backward => std::cmp::max,
73
+ };
74
+
75
+ filter_exprs.iter().try_fold(None, |acc, expr| {
76
+ let result = expr.lookup(journal_file, needle_offset.get(), direction)?;
77
+
78
+ Ok(match (acc, result) {
79
+ (None, Some(offset)) => Some(offset),
80
+ (Some(best), Some(offset)) => Some(cmp(best, offset)),
81
+ (acc, None) => acc,
82
+ })
83
+ })
84
+ }
85
+ }
86
+ }
87
+
88
+ pub fn head(&mut self) -> &mut Self {
89
+ match self {
90
+ FilterExpr::Match(_, None) => (),
91
+ FilterExpr::Match(_, Some(ic)) => {
92
+ *ic = ic.head();
93
+ }
94
+ FilterExpr::Conjunction(filter_exprs) => {
95
+ for filter_expr in filter_exprs.iter_mut() {
96
+ filter_expr.head();
97
+ }
98
+ }
99
+ FilterExpr::Disjunction(filter_exprs) => {
100
+ for filter_expr in filter_exprs.iter_mut() {
101
+ filter_expr.head();
102
+ }
103
+ }
104
+ }
105
+
106
+ self
107
+ }
108
+
109
+ pub fn tail<M: MemoryMap>(&mut self, journal_file: &JournalFile<M>) -> Result<&mut Self> {
110
+ match self {
111
+ FilterExpr::Match(_, None) => (),
112
+ FilterExpr::Match(_, Some(ic)) => {
113
+ *ic = ic.tail(journal_file)?;
114
+ }
115
+ FilterExpr::Conjunction(filter_exprs) => {
116
+ for filter_expr in filter_exprs.iter_mut() {
117
+ filter_expr.tail(journal_file)?;
118
+ }
119
+ }
120
+ FilterExpr::Disjunction(filter_exprs) => {
121
+ for filter_expr in filter_exprs.iter_mut() {
122
+ filter_expr.tail(journal_file)?;
123
+ }
124
+ }
125
+ }
126
+
127
+ Ok(self)
128
+ }
129
+
130
+ // Returns the offset of the next matching entry, if any, with an offset
131
+ // greater or equal to the needle offset.
132
+ pub fn next<M: MemoryMap>(
133
+ &mut self,
134
+ journal_file: &JournalFile<M>,
135
+ needle_offset: NonZeroU64,
136
+ ) -> Result<Option<NonZeroU64>> {
137
+ match self {
138
+ FilterExpr::Match(_, None) => Ok(None),
139
+ FilterExpr::Match(_, Some(ic)) => ic.next_until(journal_file, needle_offset),
140
+ FilterExpr::Conjunction(filter_exprs) => {
141
+ let mut needle_offset = needle_offset;
142
+
143
+ loop {
144
+ let previous_offset = needle_offset;
145
+
146
+ for fe in filter_exprs.iter_mut() {
147
+ if let Some(new_offset) = fe.next(journal_file, needle_offset)? {
148
+ needle_offset = new_offset;
149
+ } else {
150
+ return Ok(None);
151
+ }
152
+ }
153
+
154
+ if needle_offset == previous_offset {
155
+ return Ok(Some(needle_offset));
156
+ }
157
+ }
158
+ }
159
+ FilterExpr::Disjunction(filter_exprs) => {
160
+ let mut best_offset: Option<NonZeroU64> = None;
161
+
162
+ for fe in filter_exprs.iter_mut() {
163
+ if let Some(fe_offset) = fe.next(journal_file, needle_offset)? {
164
+ best_offset = match best_offset {
165
+ Some(offset) => Some(fe_offset.min(offset)),
166
+ None => Some(fe_offset),
167
+ };
168
+ }
169
+ }
170
+
171
+ Ok(best_offset)
172
+ }
173
+ }
174
+ }
175
+
176
+ // Returns the offset of the previous matching entry, if any, with an offset
177
+ // less or equal to the needle offset.
178
+ pub fn previous<M: MemoryMap>(
179
+ &mut self,
180
+ journal_file: &JournalFile<M>,
181
+ needle_offset: NonZeroU64,
182
+ ) -> Result<Option<NonZeroU64>> {
183
+ match self {
184
+ FilterExpr::Match(_, None) => Ok(None),
185
+ FilterExpr::Match(_, Some(ic)) => ic.previous_until(journal_file, needle_offset),
186
+ FilterExpr::Conjunction(filter_exprs) => {
187
+ let mut needle_offset = needle_offset;
188
+
189
+ loop {
190
+ let previous_offset = needle_offset;
191
+
192
+ for fe in filter_exprs.iter_mut().rev() {
193
+ if let Some(new_offset) = fe.previous(journal_file, needle_offset)? {
194
+ needle_offset = new_offset;
195
+ } else {
196
+ return Ok(None);
197
+ }
198
+ }
199
+
200
+ if needle_offset == previous_offset {
201
+ return Ok(Some(needle_offset));
202
+ }
203
+ }
204
+ }
205
+ FilterExpr::Disjunction(filter_exprs) => {
206
+ let mut best_offset: Option<NonZeroU64> = None;
207
+
208
+ for fe in filter_exprs.iter_mut() {
209
+ if let Some(fe_offset) = fe.previous(journal_file, needle_offset)? {
210
+ best_offset = match best_offset {
211
+ Some(offset) => Some(fe_offset.max(offset)),
212
+ None => Some(fe_offset),
213
+ };
214
+ }
215
+ }
216
+
217
+ Ok(best_offset)
218
+ }
219
+ }
220
+ }
221
+
222
+ pub fn dump<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<String> {
223
+ let mut output = String::new();
224
+ self.dump_internal(journal_file, 0, &mut output)?;
225
+ Ok(output)
226
+ }
227
+
228
+ /// Helper function for format_data_objects that handles nested expressions and indentation
229
+ fn dump_internal<M: MemoryMap>(
230
+ &self,
231
+ journal_file: &JournalFile<M>,
232
+ indent_level: usize,
233
+ output: &mut String,
234
+ ) -> Result<()> {
235
+ let indent = " ".repeat(indent_level);
236
+
237
+ match self {
238
+ FilterExpr::Match(data_offset, inlined_cursor) => {
239
+ // Load the data object
240
+ let Some(data_offset) = NonZeroU64::new(*data_offset) else {
241
+ return Err(JournalError::InvalidOffset);
242
+ };
243
+ let data_object = journal_file.data_ref(data_offset)?;
244
+
245
+ // Get the payload as a string if possible
246
+ let payload_bytes = data_object.payload_bytes();
247
+ let payload_str = String::from_utf8_lossy(payload_bytes);
248
+
249
+ // Format the data offset and payload
250
+ output.push_str(&format!(
251
+ "{}Match[offset=0x{:x}]: {}\n",
252
+ indent, data_offset, payload_str
253
+ ));
254
+
255
+ // Format cursor information if available
256
+ if let Some(ic) = inlined_cursor {
257
+ output.push_str(&format!("{} Cursor: {:?}\n", indent, ic));
258
+ }
259
+ }
260
+ FilterExpr::Conjunction(filter_exprs) => {
261
+ output.push_str(&format!("{}Conjunction (AND) {{\n", indent));
262
+ for expr in filter_exprs {
263
+ expr.dump_internal(journal_file, indent_level + 1, output)?;
264
+ }
265
+ output.push_str(&format!("{}}}\n", indent));
266
+ }
267
+ FilterExpr::Disjunction(filter_exprs) => {
268
+ output.push_str(&format!("{}Disjunction (OR) {{\n", indent));
269
+ for expr in filter_exprs {
270
+ expr.dump_internal(journal_file, indent_level + 1, output)?;
271
+ }
272
+ output.push_str(&format!("{}}}\n", indent));
273
+ }
274
+ }
275
+
276
+ Ok(())
277
+ }
278
+}
279
+
280
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
281
+pub enum LogicalOp {
282
+ Conjunction,
283
+ Disjunction,
284
+}
285
+
286
+#[derive(Debug)]
287
+pub struct JournalFilter {
288
+ filter_expr: Option<FilterExpr>,
289
+ current_matches: Vec<Vec<u8>>,
290
+ current_op: LogicalOp,
291
+}
292
+
293
+impl Default for JournalFilter {
294
+ fn default() -> Self {
295
+ Self {
296
+ filter_expr: None,
297
+ current_matches: Vec::new(),
298
+ current_op: LogicalOp::Conjunction,
299
+ }
300
+ }
301
+}
302
+
303
+impl JournalFilter {
304
+ fn extract_key(kv_pair: &[u8]) -> Option<&[u8]> {
305
+ if let Some(equal_pos) = kv_pair.iter().position(|&b| b == b'=') {
306
+ Some(&kv_pair[..equal_pos])
307
+ } else {
308
+ None
309
+ }
310
+ }
311
+
312
+ fn convert_current_matches<M: MemoryMap>(
313
+ &mut self,
314
+ journal_file: &JournalFile<M>,
315
+ ) -> Result<Option<FilterExpr>> {
316
+ if self.current_matches.is_empty() {
317
+ return Ok(None);
318
+ }
319
+
320
+ let mut elements = Vec::new();
321
+ let mut i = 0;
322
+
323
+ while i < self.current_matches.len() {
324
+ let current_key = Self::extract_key(&self.current_matches[i]).unwrap_or(&[]);
325
+ let start = i;
326
+
327
+ // Find all matches with the same key
328
+ while i < self.current_matches.len()
329
+ && Self::extract_key(&self.current_matches[i]).unwrap_or(&[]) == current_key
330
+ {
331
+ i += 1;
332
+ }
333
+
334
+ // If we have multiple values for this key, create a disjunction
335
+ if i - start > 1 {
336
+ let mut matches = Vec::with_capacity(i - start);
337
+ for idx in start..i {
338
+ let data = self.current_matches[idx].as_slice();
339
+ let hash = journal_file.hash(data);
340
+ let Some(offset) = journal_file.find_data_offset(hash, data)? else {
341
+ // TODO: should we really return an error?
342
+ return Err(JournalError::InvalidOffset);
343
+ };
344
+
345
+ let ic = journal_file.data_ref(offset)?.inlined_cursor();
346
+ matches.push(FilterExpr::Match(offset.get(), ic));
347
+ }
348
+ elements.push(FilterExpr::Disjunction(matches));
349
+ } else {
350
+ let data = self.current_matches[start].as_slice();
351
+ let hash = journal_file.hash(data);
352
+ let Some(offset) = journal_file.find_data_offset(hash, data)? else {
353
+ // TODO: should we really return an error?
354
+ return Err(JournalError::InvalidOffset);
355
+ };
356
+
357
+ let ic = journal_file.data_ref(offset)?.inlined_cursor();
358
+ elements.push(FilterExpr::Match(offset.get(), ic));
359
+ }
360
+ }
361
+
362
+ self.current_matches.clear();
363
+
364
+ match elements.len() {
365
+ 0 => panic!("Could not create filter elements from current matches"),
366
+ 1 => Ok(Some(elements.remove(0))),
367
+ _ => Ok(Some(FilterExpr::Conjunction(elements))),
368
+ }
369
+ }
370
+
371
+ pub fn add_match(&mut self, kv_pair: &[u8]) {
372
+ if kv_pair.contains(&b'=') {
373
+ let new_item = kv_pair.to_vec();
374
+ let new_key = Self::extract_key(&new_item).unwrap_or(&[]);
375
+
376
+ // Find the insertion position using binary search
377
+ let pos = self
378
+ .current_matches
379
+ .binary_search_by(|item| {
380
+ let key = Self::extract_key(item).unwrap_or(&[]);
381
+ key.cmp(new_key)
382
+ })
383
+ .unwrap_or_else(|e| e);
384
+
385
+ // Insert at the found position
386
+ self.current_matches.insert(pos, new_item);
387
+ }
388
+ }
389
+
390
+ pub fn set_operation<M: MemoryMap>(
391
+ &mut self,
392
+ journal_file: &JournalFile<M>,
393
+ op: LogicalOp,
394
+ ) -> Result<()> {
395
+ let new_expr = self.convert_current_matches(journal_file)?;
396
+ if new_expr.is_none() {
397
+ self.current_op = op;
398
+ return Ok(());
399
+ }
400
+
401
+ if self.filter_expr.is_none() {
402
+ self.filter_expr = new_expr;
403
+ self.current_op = op;
404
+ return Ok(());
405
+ }
406
+
407
+ let new_expr = new_expr.unwrap();
408
+ let current_expr = self.filter_expr.take().unwrap();
409
+
410
+ self.filter_expr = Some(match (current_expr, self.current_op) {
411
+ (FilterExpr::Disjunction(mut exprs), LogicalOp::Disjunction) => {
412
+ exprs.push(new_expr);
413
+ FilterExpr::Disjunction(exprs)
414
+ }
415
+ (FilterExpr::Conjunction(mut exprs), LogicalOp::Conjunction) => {
416
+ exprs.push(new_expr);
417
+ FilterExpr::Conjunction(exprs)
418
+ }
419
+ (current_expr, LogicalOp::Disjunction) => {
420
+ FilterExpr::Disjunction(vec![current_expr, new_expr])
421
+ }
422
+ (current_expr, LogicalOp::Conjunction) => {
423
+ FilterExpr::Conjunction(vec![current_expr, new_expr])
424
+ }
425
+ });
426
+
427
+ self.current_op = op;
428
+ Ok(())
429
+ }
430
+
431
+ pub fn build<M: MemoryMap>(&mut self, journal_file: &JournalFile<M>) -> Result<FilterExpr> {
432
+ self.set_operation(journal_file, self.current_op)?;
433
+
434
+ self.current_matches.clear();
435
+ self.current_op = LogicalOp::Conjunction;
436
+ self.filter_expr.take().ok_or(JournalError::MalformedFilter)
437
+ }
438
+}
src/crates/jf/journal_file/src/hash.rs
new
+31
@@ -0,0 +1,31 @@
1
+use siphasher::sip::SipHasher24;
2
+use std::hash::Hasher;
3
+
4
+fn jenkins_hash64(data: &[u8]) -> u64 {
5
+ // FIXME: user real jenkins hasher
6
+ let mut hasher = twox_hash::XxHash64::default();
7
+ hasher.write(data);
8
+ hasher.finish()
9
+}
10
+
11
+fn siphash24(data: &[u8], key: &[u8; 16]) -> u64 {
12
+ let k0 = u64::from_le_bytes(key[0..8].try_into().unwrap());
13
+ let k1 = u64::from_le_bytes(key[8..16].try_into().unwrap());
14
+
15
+ let mut hasher = SipHasher24::new_with_keys(k0, k1);
16
+ hasher.write(data);
17
+ hasher.finish()
18
+}
19
+
20
+pub fn journal_hash_data(data: &[u8], is_keyed_hash: bool, file_id: Option<&[u8; 16]>) -> u64 {
21
+ if is_keyed_hash {
22
+ if let Some(file_id) = file_id {
23
+ siphash24(data, file_id)
24
+ } else {
25
+ // FIXME: verify fallback behaviour
26
+ jenkins_hash64(data)
27
+ }
28
+ } else {
29
+ jenkins_hash64(data)
30
+ }
31
+}
src/crates/jf/journal_file/src/journal_file.rs
new
+816
@@ -0,0 +1,816 @@
1
+#![allow(clippy::field_reassign_with_default)]
2
+
3
+use crate::hash;
4
+use crate::object::*;
5
+use crate::offset_array;
6
+use error::{JournalError, Result};
7
+use std::cell::{RefCell, UnsafeCell};
8
+use std::fs::OpenOptions;
9
+use std::path::Path;
10
+use window_manager::{MemoryMap, MemoryMapMut, WindowManager};
11
+use zerocopy::FromBytes;
12
+
13
+#[cfg(debug_assertions)]
14
+use std::backtrace::Backtrace;
15
+
16
+use crate::value_guard::ValueGuard;
17
+
18
+// Size to pad objects to (8 bytes)
19
+const OBJECT_ALIGNMENT: u64 = 8;
20
+
21
+/// A reader for systemd journal files that efficiently maps small regions of the file into memory.
22
+///
23
+/// # Memory Management
24
+///
25
+/// This implementation uses a window-based memory mapping strategy similar to systemd's original
26
+/// implementation. Instead of mapping the entire file, it maintains a small set of memory-mapped
27
+/// windows and reuses them as needed.
28
+///
29
+/// # Concurrency and Safety
30
+///
31
+/// `JournalFile` uses interior mutability to provide a safe API with the following characteristics:
32
+///
33
+/// - The window manager is wrapped in an `UnsafeCell` to allow mutation through a shared reference.
34
+/// - A single `RefCell<bool>` guards access to ensure only one object can be active at a time.
35
+/// - Methods like `data_object()` return a `ValueGuard<T>` that automatically releases the lock
36
+/// when dropped.
37
+///
38
+/// This design ensures that memory safety is maintained even though references to memory-mapped
39
+/// regions could be invalidated when new objects are created.
40
+pub struct JournalFile<M: MemoryMap> {
41
+ // Persistent memory maps for journal header and data/field hash tables
42
+ header_map: M,
43
+ data_hash_table_map: Option<M>,
44
+ field_hash_table_map: Option<M>,
45
+
46
+ // Window manager for other objects
47
+ window_manager: UnsafeCell<WindowManager<M>>,
48
+
49
+ // Flag to track if any object is in use
50
+ object_in_use: RefCell<bool>,
51
+
52
+ #[cfg(debug_assertions)]
53
+ prev_backtrace: RefCell<Backtrace>,
54
+ #[cfg(debug_assertions)]
55
+ backtrace: RefCell<Backtrace>,
56
+}
57
+
58
+impl<M: MemoryMapMut> JournalFile<M> {
59
+ pub fn create(path: impl AsRef<Path>, window_size: u64) -> Result<Self> {
60
+ debug_assert_eq!(window_size % OBJECT_ALIGNMENT, 0);
61
+
62
+ let file = OpenOptions::new()
63
+ .create(true)
64
+ .truncate(true)
65
+ .read(true)
66
+ .write(true)
67
+ .open(&path)?;
68
+
69
+ let mut header = JournalHeader::default();
70
+ header.signature = *b"LPKSHHRH";
71
+
72
+ header.data_hash_table_offset = std::mem::size_of::<JournalHeader>() as u64
73
+ + std::mem::size_of::<ObjectHeader>() as u64;
74
+ header.data_hash_table_size = 4096 * std::mem::size_of::<HashItem>() as u64;
75
+
76
+ header.field_hash_table_offset = header.data_hash_table_offset
77
+ + header.data_hash_table_size
78
+ + std::mem::size_of::<ObjectHeader>() as u64;
79
+ header.field_hash_table_size = 512 * std::mem::size_of::<HashItem>() as u64;
80
+
81
+ debug_assert_eq!(header.data_hash_table_offset % OBJECT_ALIGNMENT, 0);
82
+ debug_assert_eq!(header.data_hash_table_size % OBJECT_ALIGNMENT, 0);
83
+ let data_hash_table_map = header.map_data_hash_table(&file)?;
84
+
85
+ debug_assert_eq!(header.field_hash_table_offset % OBJECT_ALIGNMENT, 0);
86
+ debug_assert_eq!(header.field_hash_table_size % OBJECT_ALIGNMENT, 0);
87
+ let field_hash_table_map = header.map_field_hash_table(&file)?;
88
+
89
+ header.tail_object_offset = header.data_hash_table_offset + header.data_hash_table_size;
90
+ header.n_objects = 2;
91
+
92
+ debug_assert_eq!(header.tail_object_offset % OBJECT_ALIGNMENT, 0);
93
+
94
+ let header_size = std::mem::size_of::<JournalHeader>() as u64;
95
+ let mut header_map = M::create(&file, 0, header_size)?;
96
+ {
97
+ let header_mut = JournalHeader::mut_from_prefix(&mut header_map).unwrap().0;
98
+ *header_mut = header;
99
+ }
100
+
101
+ // Create window manager for the rest of the objects
102
+ let window_manager = UnsafeCell::new(WindowManager::new(file, window_size, 32)?);
103
+
104
+ Ok(JournalFile {
105
+ header_map,
106
+ data_hash_table_map,
107
+ field_hash_table_map,
108
+ window_manager,
109
+ object_in_use: RefCell::new(false),
110
+
111
+ #[cfg(debug_assertions)]
112
+ prev_backtrace: RefCell::new(Backtrace::capture()),
113
+ #[cfg(debug_assertions)]
114
+ backtrace: RefCell::new(Backtrace::capture()),
115
+ })
116
+ }
117
+
118
+ pub fn journal_header_mut(&mut self) -> &mut JournalHeader {
119
+ JournalHeader::mut_from_prefix(&mut self.header_map)
120
+ .unwrap()
121
+ .0
122
+ }
123
+
124
+ pub fn data_hash_table_mut(&mut self) -> Option<HashTableObject<&mut [u8]>> {
125
+ self.data_hash_table_map
126
+ .as_mut()
127
+ .map(|m| HashTableObject::<&mut [u8]>::from_data_mut(m, false))
128
+ }
129
+
130
+ pub fn field_hash_table_mut(&mut self) -> Option<HashTableObject<&mut [u8]>> {
131
+ self.field_hash_table_map
132
+ .as_mut()
133
+ .map(|m| HashTableObject::<&mut [u8]>::from_data_mut(m, false))
134
+ }
135
+
136
+ fn object_header_mut(&self, position: u64) -> Result<&mut ObjectHeader> {
137
+ let size_needed = std::mem::size_of::<ObjectHeader>() as u64;
138
+ let window_manager = unsafe { &mut *self.window_manager.get() };
139
+ let header_slice = window_manager.get_slice_mut(position, size_needed)?;
140
+ Ok(ObjectHeader::mut_from_bytes(header_slice).unwrap())
141
+ }
142
+
143
+ fn object_data_mut(&self, position: u64, size_needed: u64) -> Result<&mut [u8]> {
144
+ let window_manager = unsafe { &mut *self.window_manager.get() };
145
+ let object_slice = window_manager.get_slice_mut(position, size_needed)?;
146
+ Ok(object_slice)
147
+ }
148
+
149
+ fn journal_object_mut<'a, T>(
150
+ &'a self,
151
+ type_: ObjectType,
152
+ position: u64,
153
+ size: Option<u64>,
154
+ ) -> Result<ValueGuard<'a, T>>
155
+ where
156
+ T: JournalObjectMut<&'a mut [u8]>,
157
+ {
158
+ // Check if any object is already in use
159
+ let mut is_in_use = self.object_in_use.borrow_mut();
160
+ if *is_in_use {
161
+ #[cfg(debug_assertions)]
162
+ {
163
+ eprintln!(
164
+ "Value is in use. Current Backtrace: {:?}, Previous Backtrace: {:?}",
165
+ self.backtrace.borrow().to_string(),
166
+ self.prev_backtrace.borrow().to_string()
167
+ );
168
+ }
169
+ return Err(JournalError::ValueGuardInUse);
170
+ }
171
+
172
+ #[cfg(debug_assertions)]
173
+ {
174
+ self.backtrace.swap(&self.prev_backtrace);
175
+ let _ = self.backtrace.replace(Backtrace::force_capture());
176
+ }
177
+
178
+ let is_compact = self
179
+ .journal_header_ref()
180
+ .has_incompatible_flag(HeaderIncompatibleFlags::Compact);
181
+
182
+ let size_needed = match size {
183
+ Some(size) => {
184
+ let header = self.object_header_mut(position)?;
185
+ header.type_ = type_ as u8;
186
+ header.size = size;
187
+ size
188
+ }
189
+ None => {
190
+ let header = self.object_header_ref(position)?;
191
+ if header.type_ != type_ as u8 {
192
+ return Err(JournalError::InvalidObjectType);
193
+ }
194
+ header.size
195
+ }
196
+ };
197
+
198
+ let data = self.object_data_mut(position, size_needed)?;
199
+ let object = T::from_data_mut(data, is_compact);
200
+
201
+ // Mark as in use
202
+ *is_in_use = true;
203
+ Ok(ValueGuard::new(object, &self.object_in_use))
204
+ }
205
+
206
+ pub fn offset_array_mut(
207
+ &self,
208
+ position: u64,
209
+ capacity: Option<u64>,
210
+ ) -> Result<ValueGuard<OffsetArrayObject<&mut [u8]>>> {
211
+ let size = capacity.map(|c| {
212
+ let mut size = std::mem::size_of::<OffsetArrayObjectHeader>() as u64;
213
+
214
+ let is_compact = self
215
+ .journal_header_ref()
216
+ .has_incompatible_flag(HeaderIncompatibleFlags::Compact);
217
+ if is_compact {
218
+ size += c * std::mem::size_of::<u32>() as u64;
219
+ } else {
220
+ size += c * std::mem::size_of::<u64>() as u64;
221
+ }
222
+
223
+ size
224
+ });
225
+
226
+ let offset_array = self.journal_object_mut(ObjectType::EntryArray, position, size);
227
+ offset_array
228
+ }
229
+
230
+ pub fn field_mut(
231
+ &self,
232
+ position: u64,
233
+ size: Option<u64>,
234
+ ) -> Result<ValueGuard<FieldObject<&mut [u8]>>> {
235
+ let size = size.map(|n| std::mem::size_of::<FieldObjectHeader>() as u64 + n);
236
+ self.journal_object_mut(ObjectType::Field, position, size)
237
+ }
238
+
239
+ pub fn entry_mut(&self, position: u64) -> Result<ValueGuard<EntryObject<&mut [u8]>>> {
240
+ self.journal_object_mut(ObjectType::Entry, position, None)
241
+ }
242
+
243
+ pub fn data_mut(
244
+ &self,
245
+ position: u64,
246
+ size: Option<u64>,
247
+ ) -> Result<ValueGuard<DataObject<&mut [u8]>>> {
248
+ let size = size.map(|n| std::mem::size_of::<DataObjectHeader>() as u64 + n);
249
+ self.journal_object_mut(ObjectType::Data, position, size)
250
+ }
251
+
252
+ pub fn tag_mut(&self, position: u64, new: bool) -> Result<ValueGuard<TagObject<&mut [u8]>>> {
253
+ let size = if new {
254
+ Some(std::mem::size_of::<TagObjectHeader>() as u64)
255
+ } else {
256
+ None
257
+ };
258
+ self.journal_object_mut(ObjectType::Tag, position, size)
259
+ }
260
+}
261
+
262
+impl<M: MemoryMap> JournalFile<M> {
263
+ pub fn open(path: impl AsRef<Path>, window_size: u64) -> Result<Self> {
264
+ debug_assert_eq!(window_size % OBJECT_ALIGNMENT, 0);
265
+
266
+ // Open file and check its size
267
+ let file = OpenOptions::new().read(true).write(false).open(&path)?;
268
+
269
+ // Create a memory map for the header
270
+ let header_size = std::mem::size_of::<JournalHeader>() as u64;
271
+ let header_map = M::create(&file, 0, header_size)?;
272
+ let header = JournalHeader::ref_from_prefix(&header_map).unwrap().0;
273
+ if header.signature != *b"LPKSHHRH" {
274
+ return Err(JournalError::InvalidMagicNumber);
275
+ }
276
+
277
+ // Initialize the hash table maps if they exist
278
+ let data_hash_table_map = header.map_data_hash_table(&file)?;
279
+ let field_hash_table_map = header.map_field_hash_table(&file)?;
280
+
281
+ // Create window manager for the rest of the objects
282
+ let window_manager = UnsafeCell::new(WindowManager::new(file, window_size, 32)?);
283
+
284
+ Ok(JournalFile {
285
+ header_map,
286
+ data_hash_table_map,
287
+ field_hash_table_map,
288
+ window_manager,
289
+ object_in_use: RefCell::new(false),
290
+
291
+ #[cfg(debug_assertions)]
292
+ prev_backtrace: RefCell::new(Backtrace::capture()),
293
+ #[cfg(debug_assertions)]
294
+ backtrace: RefCell::new(Backtrace::capture()),
295
+ })
296
+ }
297
+
298
+ pub fn hash(&self, data: &[u8]) -> u64 {
299
+ let is_keyed_hash = self
300
+ .journal_header_ref()
301
+ .has_incompatible_flag(HeaderIncompatibleFlags::KeyedHash);
302
+
303
+ hash::journal_hash_data(
304
+ data,
305
+ is_keyed_hash,
306
+ if is_keyed_hash {
307
+ Some(&self.journal_header_ref().file_id)
308
+ } else {
309
+ None
310
+ },
311
+ )
312
+ }
313
+
314
+ pub fn entry_list(&self) -> Option<offset_array::List> {
315
+ let head_offset = std::num::NonZeroU64::new(self.journal_header_ref().entry_array_offset)?;
316
+ let total_items =
317
+ std::num::NonZeroUsize::new(self.journal_header_ref().n_entries as usize)?;
318
+ Some(offset_array::List::new(head_offset, total_items))
319
+ }
320
+
321
+ pub fn journal_header_ref(&self) -> &JournalHeader {
322
+ JournalHeader::ref_from_prefix(&self.header_map).unwrap().0
323
+ }
324
+
325
+ pub fn data_hash_table_ref(&self) -> Option<HashTableObject<&[u8]>> {
326
+ self.data_hash_table_map
327
+ .as_ref()
328
+ .and_then(|m| HashTableObject::<&[u8]>::from_data(m, false))
329
+ }
330
+
331
+ pub fn field_hash_table_ref(&self) -> Option<HashTableObject<&[u8]>> {
332
+ self.field_hash_table_map
333
+ .as_ref()
334
+ .and_then(|m| HashTableObject::<&[u8]>::from_data(m, false))
335
+ }
336
+
337
+ fn object_header_ref(&self, position: u64) -> Result<&ObjectHeader> {
338
+ let size_needed = std::mem::size_of::<ObjectHeader>() as u64;
339
+ let window_manager = unsafe { &mut *self.window_manager.get() };
340
+ let header_slice = window_manager.get_slice(position, size_needed)?;
341
+ Ok(ObjectHeader::ref_from_bytes(header_slice).unwrap())
342
+ }
343
+
344
+ fn object_data_ref(&self, position: u64, size_needed: u64) -> Result<&[u8]> {
345
+ let window_manager = unsafe { &mut *self.window_manager.get() };
346
+ let object_slice = window_manager.get_slice(position, size_needed)?;
347
+ Ok(object_slice)
348
+ }
349
+
350
+ fn journal_object_ref<'a, T>(&'a self, position: u64) -> Result<ValueGuard<'a, T>>
351
+ where
352
+ T: JournalObject<&'a [u8]>,
353
+ {
354
+ // Check if any object is already in use
355
+ let mut is_in_use = self.object_in_use.borrow_mut();
356
+ if *is_in_use {
357
+ #[cfg(debug_assertions)]
358
+ {
359
+ eprintln!(
360
+ "Value is in use. Current Backtrace: {:?}, Previous Backtrace: {:?}",
361
+ self.backtrace.borrow().to_string(),
362
+ self.prev_backtrace.borrow().to_string()
363
+ );
364
+ }
365
+ return Err(JournalError::ValueGuardInUse);
366
+ }
367
+
368
+ #[cfg(debug_assertions)]
369
+ {
370
+ self.backtrace.swap(&self.prev_backtrace);
371
+ let _ = self.backtrace.replace(Backtrace::force_capture());
372
+ }
373
+
374
+ let is_compact = self
375
+ .journal_header_ref()
376
+ .has_incompatible_flag(HeaderIncompatibleFlags::Compact);
377
+
378
+ let size_needed = {
379
+ let header = self.object_header_ref(position)?;
380
+ header.size
381
+ };
382
+
383
+ let data = self.object_data_ref(position, size_needed)?;
384
+ let Some(object) = T::from_data(data, is_compact) else {
385
+ return Err(JournalError::ZerocopyFailure);
386
+ };
387
+
388
+ // Mark as in use
389
+ *is_in_use = true;
390
+
391
+ Ok(ValueGuard::new(object, &self.object_in_use))
392
+ }
393
+
394
+ pub fn offset_array_ref(&self, position: u64) -> Result<ValueGuard<OffsetArrayObject<&[u8]>>> {
395
+ self.journal_object_ref(position)
396
+ }
397
+
398
+ pub fn field_ref(&self, position: u64) -> Result<ValueGuard<FieldObject<&[u8]>>> {
399
+ self.journal_object_ref(position)
400
+ }
401
+
402
+ pub fn entry_ref(&self, position: u64) -> Result<ValueGuard<EntryObject<&[u8]>>> {
403
+ self.journal_object_ref(position)
404
+ }
405
+
406
+ pub fn data_ref(&self, position: u64) -> Result<ValueGuard<DataObject<&[u8]>>> {
407
+ self.journal_object_ref(position)
408
+ }
409
+
410
+ pub fn tag_ref(&self, position: u64) -> Result<ValueGuard<TagObject<&[u8]>>> {
411
+ self.journal_object_ref(position)
412
+ }
413
+
414
+ fn lookup_hash_table<'a, T, F>(
415
+ &'a self,
416
+ hash_table: Option<HashTableObject<&[u8]>>,
417
+ data: &[u8],
418
+ hash: u64,
419
+ fetch_fn: F,
420
+ ) -> Result<u64>
421
+ where
422
+ T: HashableObject,
423
+ F: Fn(u64) -> Result<ValueGuard<'a, T>>,
424
+ {
425
+ let hash_table = hash_table.ok_or(JournalError::MissingHashTable)?;
426
+
427
+ // Find the right bucket in the hash table
428
+ let hash_table_size = hash_table.items.len();
429
+ let bucket_idx = (hash % hash_table_size as u64) as usize;
430
+
431
+ // Get the head object offset from the bucket
432
+ let bucket = hash_table.items[bucket_idx];
433
+ let mut object_offset = bucket.head_hash_offset;
434
+
435
+ // Traverse the linked list of objects in this bucket
436
+ while object_offset != 0 {
437
+ match fetch_fn(object_offset) {
438
+ Ok(object_guard) => {
439
+ // Check if this is the object we're looking for
440
+ if object_guard.hash() == hash && object_guard.get_payload() == data {
441
+ return Ok(object_offset);
442
+ }
443
+
444
+ // Move to the next object in the chain
445
+ object_offset = object_guard.next_hash_offset();
446
+ }
447
+ Err(e) => {
448
+ return Err(e);
449
+ }
450
+ }
451
+ }
452
+
453
+ Err(JournalError::MissingObjectFromHashTable)
454
+ }
455
+
456
+ /// Finds a field object by name and returns its offset
457
+ pub fn find_field_offset_by_name(&self, field_name: &[u8], hash: u64) -> Result<u64> {
458
+ self.lookup_hash_table::<FieldObject<&[u8]>, _>(
459
+ self.field_hash_table_ref(),
460
+ field_name,
461
+ hash,
462
+ |offset| self.field_ref(offset),
463
+ )
464
+ }
465
+
466
+ /// Finds a data object by payload and returns its offset
467
+ pub fn find_data_offset_by_payload(&self, payload: &[u8], hash: u64) -> Result<u64> {
468
+ self.lookup_hash_table::<DataObject<&[u8]>, _>(
469
+ self.data_hash_table_ref(),
470
+ payload,
471
+ hash,
472
+ |offset| self.data_ref(offset),
473
+ )
474
+ }
475
+
476
+ /// Run a directed partition point query on a data object's entry array
477
+ ///
478
+ /// This finds the first/last entry (depending on direction) that satisfies the given predicate
479
+ /// in the entry array chain of the data object.
480
+ pub fn data_object_directed_partition_point<F>(
481
+ &self,
482
+ data_offset: u64,
483
+ predicate: F,
484
+ direction: offset_array::Direction,
485
+ ) -> Result<Option<u64>>
486
+ where
487
+ F: Fn(u64) -> Result<bool>,
488
+ {
489
+ let Some(cursor) = self.data_ref(data_offset)?.inlined_cursor() else {
490
+ return Ok(None);
491
+ };
492
+
493
+ let best_match = cursor.directed_partition_point(self, predicate, direction)?;
494
+
495
+ // Convert the result to an entry offset
496
+ best_match.map(|c| c.value(self)).transpose()
497
+ }
498
+
499
+ /// Creates an iterator over all offsets of an offset array list
500
+ pub fn array_offsets(&self, position: u64) -> Result<OffsetArrayListIterator<'_, M>> {
501
+ Ok(OffsetArrayListIterator {
502
+ journal: self,
503
+ offset: position,
504
+ capacity: if position == 0 {
505
+ 0
506
+ } else {
507
+ self.offset_array_ref(position)?.capacity()
508
+ },
509
+ index: 0,
510
+ })
511
+ }
512
+
513
+ /// Creates an iterator over all entry offsets in the journal
514
+ pub fn entry_offsets(&self) -> Result<OffsetArrayListIterator<'_, M>> {
515
+ self.array_offsets(self.journal_header_ref().entry_array_offset)
516
+ }
517
+
518
+ /// Creates an iterator over all field objects in the field hash table
519
+ pub fn fields(&self) -> FieldIterator<'_, M> {
520
+ // Get the field hash table
521
+ let field_hash_table = self.field_hash_table_ref();
522
+
523
+ // Initialize with the first bucket
524
+ let mut iterator = FieldIterator {
525
+ journal: self,
526
+ field_hash_table,
527
+ current_bucket_index: 0,
528
+ next_field_offset: 0,
529
+ };
530
+
531
+ // Find the first non-empty bucket
532
+ iterator.advance_to_next_nonempty_bucket();
533
+
534
+ iterator
535
+ }
536
+
537
+ /// Creates an iterator over all DATA objects for the specified field
538
+ pub fn field_data_objects<'a>(
539
+ &'a self,
540
+ field_name: &'a [u8],
541
+ ) -> Result<FieldDataIterator<'a, M>> {
542
+ // Find the field offset by name
543
+ let field_hash = self.hash(field_name);
544
+ let field_offset = self.find_field_offset_by_name(field_name, field_hash)?;
545
+
546
+ // Get the field object to access its head_data_offset
547
+ let field_guard = self.field_ref(field_offset)?;
548
+ let head_data_offset = field_guard.header.head_data_offset;
549
+
550
+ // Create the iterator
551
+ Ok(FieldDataIterator {
552
+ journal: self,
553
+ current_data_offset: head_data_offset,
554
+ })
555
+ }
556
+
557
+ /// Creates an iterator over all DATA objects for a specific entry
558
+ pub fn entry_data_objects(&self, entry_offset: u64) -> Result<EntryDataIterator<'_, M>> {
559
+ // Get the entry object to determine how many data items it has
560
+ let entry_guard = self.entry_ref(entry_offset)?;
561
+
562
+ // Get the total number of items
563
+ let total_items = match &entry_guard.items {
564
+ EntryItemsType::Regular(items) => items.len(),
565
+ EntryItemsType::Compact(items) => items.len(),
566
+ };
567
+
568
+ // Create the iterator
569
+ Ok(EntryDataIterator {
570
+ journal: self,
571
+ entry_offset,
572
+ current_index: 0,
573
+ total_items,
574
+ })
575
+ }
576
+}
577
+
578
+/*
579
+ * Offset array iteration
580
+*/
581
+
582
+/// Iterator that returns all offsets in an offset array list
583
+pub struct OffsetArrayListIterator<'a, M: MemoryMap> {
584
+ journal: &'a JournalFile<M>,
585
+ offset: u64,
586
+ capacity: usize,
587
+ index: usize,
588
+}
589
+
590
+impl<M: MemoryMap> Iterator for OffsetArrayListIterator<'_, M> {
591
+ type Item = Result<u64>;
592
+
593
+ fn next(&mut self) -> Option<Self::Item> {
594
+ // If we've reached the end of offsets in the offset array list, return None
595
+ if self.offset == 0 {
596
+ return None;
597
+ }
598
+
599
+ // Check if we need to move to the next offset array
600
+ if self.index >= self.capacity {
601
+ // Get the next offset array offset
602
+ let next_offset = match self.journal.offset_array_ref(self.offset) {
603
+ Ok(array_guard) => array_guard.header.next_offset_array,
604
+ Err(e) => return Some(Err(e)),
605
+ };
606
+
607
+ // If there's no next offset array, we're done
608
+ if next_offset == 0 {
609
+ self.offset = 0;
610
+ return None;
611
+ }
612
+
613
+ // Set up the next offset array
614
+ match self.journal.offset_array_ref(next_offset) {
615
+ Ok(array_guard) => {
616
+ self.offset = next_offset;
617
+ self.capacity = array_guard.capacity();
618
+ self.index = 0;
619
+ }
620
+ Err(e) => return Some(Err(e)),
621
+ }
622
+ }
623
+
624
+ // Get the current offset from the offset array
625
+ let offset = match self.journal.offset_array_ref(self.offset) {
626
+ Ok(array_guard) => match &array_guard.items {
627
+ OffsetsType::Regular(offsets) => {
628
+ if self.index < offsets.len() {
629
+ offsets[self.index]
630
+ } else {
631
+ 0
632
+ }
633
+ }
634
+ OffsetsType::Compact(offsets) => {
635
+ if self.index < offsets.len() {
636
+ offsets[self.index] as u64
637
+ } else {
638
+ 0
639
+ }
640
+ }
641
+ },
642
+ Err(e) => return Some(Err(e)),
643
+ };
644
+
645
+ // Increment index for next iteration
646
+ self.index += 1;
647
+
648
+ // If offset is zero, we've reached the end of all entries
649
+ if offset == 0 {
650
+ self.offset = 0;
651
+ return None;
652
+ }
653
+
654
+ Some(Ok(offset))
655
+ }
656
+}
657
+
658
+/// Iterator that walks through all field objects in the field hash table
659
+pub struct FieldIterator<'a, M: MemoryMap> {
660
+ journal: &'a JournalFile<M>,
661
+ field_hash_table: Option<HashTableObject<&'a [u8]>>,
662
+ current_bucket_index: usize,
663
+ next_field_offset: u64,
664
+}
665
+
666
+impl<M: MemoryMap> FieldIterator<'_, M> {
667
+ /// Advances to the next non-empty bucket
668
+ fn advance_to_next_nonempty_bucket(&mut self) {
669
+ // If we don't have a hash table, there's nothing to iterate
670
+ let Some(hash_table) = &self.field_hash_table else {
671
+ return;
672
+ };
673
+
674
+ let items = &hash_table.items;
675
+ let num_buckets = items.len();
676
+
677
+ // Find the next non-empty bucket
678
+ while self.current_bucket_index < num_buckets {
679
+ let bucket = items[self.current_bucket_index];
680
+ if bucket.head_hash_offset != 0 {
681
+ // Found a non-empty bucket
682
+ self.next_field_offset = bucket.head_hash_offset;
683
+ return;
684
+ }
685
+ self.current_bucket_index += 1;
686
+ }
687
+
688
+ // No more non-empty buckets
689
+ self.next_field_offset = 0;
690
+ }
691
+}
692
+
693
+impl<'a, M: MemoryMap> Iterator for FieldIterator<'a, M> {
694
+ type Item = Result<ValueGuard<'a, FieldObject<&'a [u8]>>>;
695
+
696
+ fn next(&mut self) -> Option<Self::Item> {
697
+ // If we've reached the end, return None
698
+ if self.next_field_offset == 0 {
699
+ return None;
700
+ }
701
+
702
+ // Save the current offset to return
703
+ let offset = self.next_field_offset;
704
+
705
+ // Try to get the field object
706
+ match self.journal.field_ref(offset) {
707
+ Ok(field_guard) => {
708
+ // Get the next field offset before we return the guard
709
+ self.next_field_offset = field_guard.header.next_hash_offset;
710
+
711
+ // If we've reached the end of the chain, move to the next bucket
712
+ if self.next_field_offset == 0 {
713
+ self.current_bucket_index += 1;
714
+ self.advance_to_next_nonempty_bucket();
715
+ }
716
+
717
+ Some(Ok(field_guard))
718
+ }
719
+ Err(e) => {
720
+ // If we can't read the field, return the error and stop iteration
721
+ self.next_field_offset = 0;
722
+ Some(Err(e))
723
+ }
724
+ }
725
+ }
726
+}
727
+
728
+/// Iterator that walks through all DATA objects for a specific field
729
+pub struct FieldDataIterator<'a, M: MemoryMap> {
730
+ journal: &'a JournalFile<M>,
731
+ current_data_offset: u64,
732
+}
733
+
734
+impl<'a, M: MemoryMap> Iterator for FieldDataIterator<'a, M> {
735
+ type Item = Result<ValueGuard<'a, DataObject<&'a [u8]>>>;
736
+
737
+ fn next(&mut self) -> Option<Self::Item> {
738
+ // If we've reached the end, return None
739
+ if self.current_data_offset == 0 {
740
+ return None;
741
+ }
742
+
743
+ // Save the current offset to return
744
+ let offset = self.current_data_offset;
745
+
746
+ // Try to get the data object
747
+ match self.journal.data_ref(offset) {
748
+ Ok(data_guard) => {
749
+ // Get the next data offset before we return the guard
750
+ self.current_data_offset = data_guard.header.next_field_offset;
751
+ Some(Ok(data_guard))
752
+ }
753
+ Err(e) => {
754
+ // If we can't read the data object, return the error and stop iteration
755
+ self.current_data_offset = 0;
756
+ Some(Err(e))
757
+ }
758
+ }
759
+ }
760
+}
761
+
762
+/// Iterator that walks through all DATA objects for a specific entry
763
+pub struct EntryDataIterator<'a, M: MemoryMap> {
764
+ journal: &'a JournalFile<M>,
765
+ entry_offset: u64,
766
+ current_index: usize,
767
+ total_items: usize,
768
+}
769
+
770
+impl<'a, M: MemoryMap> Iterator for EntryDataIterator<'a, M> {
771
+ type Item = Result<ValueGuard<'a, DataObject<&'a [u8]>>>;
772
+
773
+ fn next(&mut self) -> Option<Self::Item> {
774
+ // If we've reached the end of the data indices, return None
775
+ if self.current_index >= self.total_items {
776
+ return None;
777
+ }
778
+
779
+ // Get the entry object to access the data offset
780
+ match self.journal.entry_ref(self.entry_offset) {
781
+ Ok(entry_guard) => {
782
+ let idx = self.current_index;
783
+ self.current_index += 1;
784
+
785
+ let data_offset = match &entry_guard.items {
786
+ EntryItemsType::Regular(items) => {
787
+ if idx >= items.len() {
788
+ return None;
789
+ }
790
+ items[idx].object_offset
791
+ }
792
+ EntryItemsType::Compact(items) => {
793
+ if idx >= items.len() {
794
+ return None;
795
+ }
796
+ items[idx].object_offset as u64
797
+ }
798
+ };
799
+
800
+ // Drop the entry guard before obtaining the data object
801
+ drop(entry_guard);
802
+
803
+ // Try to get the data object
804
+ match self.journal.data_ref(data_offset) {
805
+ Ok(data_guard) => Some(Ok(data_guard)),
806
+ Err(e) => Some(Err(e)),
807
+ }
808
+ }
809
+ Err(e) => {
810
+ // If we can't read the entry, return the error and stop iteration
811
+ self.current_index = self.total_items;
812
+ Some(Err(e))
813
+ }
814
+ }
815
+ }
816
+}
src/crates/jf/journal_file/src/lib.rs
new
+35
@@ -0,0 +1,35 @@
1
+// Modules - keep some public for advanced usage
2
+pub mod cursor;
3
+pub mod file;
4
+pub mod filter;
5
+mod hash;
6
+mod object;
7
+pub mod offset_array;
8
+pub mod reader;
9
+mod value_guard;
10
+pub mod writer;
11
+
12
+// Core functionality
13
+pub use file::{load_boot_id, BucketUtilization, JournalFile, JournalFileOptions};
14
+pub use reader::JournalReader;
15
+pub use writer::JournalWriter;
16
+
17
+// Essential types for working with readers
18
+pub use cursor::Location;
19
+pub use offset_array::Direction;
20
+
21
+// Advanced filtering (for users who need it)
22
+pub use cursor::JournalCursor;
23
+pub use filter::{FilterExpr, JournalFilter, LogicalOp};
24
+
25
+// For FFI compatibility and advanced object manipulation
26
+pub use object::HashableObject;
27
+
28
+// Re-export commonly needed external types
29
+pub use memmap2::{Mmap, MmapMut};
30
+
31
+// Internal utilities that might be needed
32
+pub use crate::hash::journal_hash_data;
33
+
34
+// Internal re-exports needed by the crate itself (not part of public API)
35
+pub(crate) use object::*;
src/crates/jf/journal_file/src/object.rs
new
+941
@@ -0,0 +1,941 @@
1
+use crate::offset_array::{Cursor, InlinedCursor, List};
2
+use error::{JournalError, Result};
3
+use std::num::{NonZeroU32, NonZeroU64, NonZeroUsize};
4
+use zerocopy::{
5
+ ByteSlice, ByteSliceMut, FromBytes, Immutable, IntoBytes, KnownLayout, Ref, SplitByteSlice,
6
+ SplitByteSliceMut,
7
+};
8
+
9
+pub trait HashableObject {
10
+ /// Get the hash value of this object
11
+ fn hash(&self) -> u64;
12
+
13
+ /// Get the payload data for matching
14
+ fn get_payload(&self) -> &[u8];
15
+
16
+ /// Get the offset to the next object in the hash chain
17
+ fn next_hash_offset(&self) -> Option<NonZeroU64>;
18
+
19
+ /// Get the object type
20
+ fn object_type() -> ObjectType;
21
+}
22
+
23
+pub trait HashableObjectMut: HashableObject {
24
+ /// Set the offset to the next object in the hash chain
25
+ fn set_next_hash_offset(&mut self, offset: NonZeroU64);
26
+
27
+ /// Set the payload of the object
28
+ fn set_payload(&mut self, data: &[u8]);
29
+}
30
+
31
+/// Trait for hash table operations
32
+pub trait HashTable {
33
+ /// The type of objects stored in this hash table
34
+ type Object: HashableObject;
35
+
36
+ /// Get the hash item for a given hash value
37
+ fn hash_item_ref(&self, hash: u64) -> &HashItem;
38
+
39
+ /// Get the length of the hash table (number of buckets)
40
+ fn len(&self) -> usize;
41
+
42
+ /// Make clippy happy
43
+ fn is_empty(&self) -> bool {
44
+ todo!()
45
+ }
46
+}
47
+
48
+/// Trait for mutable hash table operations
49
+pub trait HashTableMut: HashTable {
50
+ /// Get a mutable reference to the hash item for a given hash value
51
+ fn hash_item_mut(&mut self, hash: u64) -> &mut HashItem;
52
+}
53
+
54
+pub struct DataHashTable<B: ByteSlice> {
55
+ pub header: Ref<B, ObjectHeader>,
56
+ pub items: Ref<B, [HashItem]>,
57
+}
58
+
59
+pub struct FieldHashTable<B: ByteSlice> {
60
+ pub header: Ref<B, ObjectHeader>,
61
+ pub items: Ref<B, [HashItem]>,
62
+}
63
+
64
+// Implement HashTable for DataHashTable
65
+impl<B: ByteSlice> HashTable for DataHashTable<B> {
66
+ type Object = DataObject<B>;
67
+
68
+ fn hash_item_ref(&self, hash: u64) -> &HashItem {
69
+ let bucket_index = hash as usize % self.items.len();
70
+ &self.items[bucket_index]
71
+ }
72
+
73
+ fn len(&self) -> usize {
74
+ self.items.len()
75
+ }
76
+}
77
+
78
+// Implement HashTable for FieldHashTable
79
+impl<B: ByteSlice> HashTable for FieldHashTable<B> {
80
+ type Object = FieldObject<B>;
81
+
82
+ fn hash_item_ref(&self, hash: u64) -> &HashItem {
83
+ let bucket_index = hash as usize % self.items.len();
84
+ &self.items[bucket_index]
85
+ }
86
+
87
+ fn len(&self) -> usize {
88
+ self.items.len()
89
+ }
90
+}
91
+
92
+// Implement HashTableMut for DataHashTable
93
+impl<B: ByteSliceMut> HashTableMut for DataHashTable<B> {
94
+ fn hash_item_mut(&mut self, hash: u64) -> &mut HashItem {
95
+ let bucket_index = hash as usize % self.items.len();
96
+ &mut self.items[bucket_index]
97
+ }
98
+}
99
+
100
+// Implement HashTableMut for FieldHashTable
101
+impl<B: ByteSliceMut> HashTableMut for FieldHashTable<B> {
102
+ fn hash_item_mut(&mut self, hash: u64) -> &mut HashItem {
103
+ let bucket_index = hash as usize % self.items.len();
104
+ &mut self.items[bucket_index]
105
+ }
106
+}
107
+
108
+// Implement JournalObject for DataHashTable
109
+impl<B: SplitByteSlice> JournalObject<B> for DataHashTable<B> {
110
+ fn from_data(data: B, _is_compact: bool) -> Option<Self> {
111
+ let (header_data, items_data) = data.split_at(std::mem::size_of::<ObjectHeader>()).ok()?;
112
+
113
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
114
+ let items = zerocopy::Ref::from_bytes(items_data).ok()?;
115
+
116
+ Some(DataHashTable { header, items })
117
+ }
118
+}
119
+
120
+// Implement JournalObjectMut for DataHashTable
121
+impl<B: SplitByteSliceMut> JournalObjectMut<B> for DataHashTable<B> {
122
+ fn from_data_mut(data: B, _is_compact: bool) -> Option<Self> {
123
+ let (header_data, items_data) = data.split_at(std::mem::size_of::<ObjectHeader>()).ok()?;
124
+
125
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
126
+ let items = zerocopy::Ref::from_bytes(items_data).ok()?;
127
+
128
+ Some(DataHashTable { header, items })
129
+ }
130
+}
131
+
132
+// Implement JournalObject for FieldHashTable
133
+impl<B: SplitByteSlice> JournalObject<B> for FieldHashTable<B> {
134
+ fn from_data(data: B, _is_compact: bool) -> Option<Self> {
135
+ let (header_data, items_data) = data.split_at(std::mem::size_of::<ObjectHeader>()).ok()?;
136
+
137
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
138
+ let items = zerocopy::Ref::from_bytes(items_data).ok()?;
139
+
140
+ Some(FieldHashTable { header, items })
141
+ }
142
+}
143
+
144
+// Implement JournalObjectMut for FieldHashTable
145
+impl<B: SplitByteSliceMut> JournalObjectMut<B> for FieldHashTable<B> {
146
+ fn from_data_mut(data: B, _is_compact: bool) -> Option<Self> {
147
+ let (header_data, items_data) = data.split_at(std::mem::size_of::<ObjectHeader>()).ok()?;
148
+
149
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
150
+ let items = zerocopy::Ref::from_bytes(items_data).ok()?;
151
+
152
+ Some(FieldHashTable { header, items })
153
+ }
154
+}
155
+
156
+impl<B: ByteSlice> HashableObject for FieldObject<B> {
157
+ fn hash(&self) -> u64 {
158
+ self.header.hash
159
+ }
160
+
161
+ fn get_payload(&self) -> &[u8] {
162
+ &self.payload
163
+ }
164
+
165
+ fn next_hash_offset(&self) -> Option<NonZeroU64> {
166
+ self.header.next_hash_offset
167
+ }
168
+
169
+ fn object_type() -> ObjectType {
170
+ ObjectType::Field
171
+ }
172
+}
173
+
174
+impl HashableObjectMut for FieldObject<&mut [u8]> {
175
+ fn set_next_hash_offset(&mut self, next_hash_offset: NonZeroU64) {
176
+ self.header.next_hash_offset = Some(next_hash_offset);
177
+ }
178
+
179
+ fn set_payload(&mut self, data: &[u8]) {
180
+ self.payload.copy_from_slice(data);
181
+ }
182
+}
183
+
184
+impl<B: ByteSlice> HashableObject for DataObject<B> {
185
+ fn hash(&self) -> u64 {
186
+ self.header.hash
187
+ }
188
+
189
+ fn get_payload(&self) -> &[u8] {
190
+ self.payload_bytes()
191
+ }
192
+
193
+ fn next_hash_offset(&self) -> Option<NonZeroU64> {
194
+ self.header.next_hash_offset
195
+ }
196
+
197
+ fn object_type() -> ObjectType {
198
+ ObjectType::Data
199
+ }
200
+}
201
+
202
+impl HashableObjectMut for DataObject<&mut [u8]> {
203
+ fn set_next_hash_offset(&mut self, next_hash_offset: NonZeroU64) {
204
+ self.header.next_hash_offset = Some(next_hash_offset);
205
+ }
206
+
207
+ fn set_payload(&mut self, data: &[u8]) {
208
+ match &mut self.payload {
209
+ DataPayloadType::Regular(payload) => {
210
+ payload.copy_from_slice(data);
211
+ }
212
+ DataPayloadType::Compact { payload, .. } => {
213
+ payload.copy_from_slice(data);
214
+ }
215
+ };
216
+ }
217
+}
218
+
219
+/// Trait to standardize creation of journal objects from byte slices
220
+pub trait JournalObject<B: SplitByteSlice>: Sized {
221
+ /// Create a new journal object from a byte slice
222
+ fn from_data(data: B, is_compact: bool) -> Option<Self>;
223
+}
224
+
225
+pub trait JournalObjectMut<B: SplitByteSliceMut>: JournalObject<B> {
226
+ /// Create a new journal object from a byte slice
227
+ fn from_data_mut(data: B, is_compact: bool) -> Option<Self>;
228
+}
229
+
230
+pub enum HeaderIncompatibleFlags {
231
+ CompressedXz = 1 << 0,
232
+ CompressedLz4 = 1 << 1,
233
+ KeyedHash = 1 << 2,
234
+ CompressedZstd = 1 << 3,
235
+ Compact = 1 << 4,
236
+}
237
+
238
+pub enum HeaderCompatibleFlags {
239
+ Sealed = 1 << 0,
240
+ TailEntryBootId = 1 << 1,
241
+}
242
+
243
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
244
+pub enum JournalState {
245
+ Offline = 0,
246
+ Online = 1,
247
+ Archived = 2,
248
+}
249
+
250
+impl TryFrom<u8> for JournalState {
251
+ type Error = JournalError;
252
+
253
+ fn try_from(value: u8) -> Result<Self> {
254
+ match value {
255
+ 0 => Ok(JournalState::Offline),
256
+ 1 => Ok(JournalState::Online),
257
+ 2 => Ok(JournalState::Archived),
258
+ _ => Err(JournalError::InvalidJournalFileState),
259
+ }
260
+ }
261
+}
262
+
263
+#[derive(Default, Debug, Clone, Copy, FromBytes, IntoBytes, Immutable, KnownLayout)]
264
+#[repr(C)]
265
+pub struct JournalHeader {
266
+ pub signature: [u8; 8], // "LPKSHHRH"
267
+ pub compatible_flags: u32, // Compatible extension flags
268
+ pub incompatible_flags: u32, // Incompatible extension flags
269
+ pub state: u8, // File state (offline=0, online=1, archived=2)
270
+ pub reserved: [u8; 7], // Reserved space
271
+ pub file_id: [u8; 16], // Unique ID for this file
272
+ pub machine_id: [u8; 16], // Machine ID this belongs to
273
+ pub tail_entry_boot_id: [u8; 16], // Boot ID of the last entry
274
+ pub seqnum_id: [u8; 16], // Sequence number ID
275
+ pub header_size: u64, // Size of the header
276
+ pub arena_size: u64, // Size of the data arena
277
+ pub data_hash_table_offset: Option<NonZeroU64>, // Offset of the data hash table
278
+ pub data_hash_table_size: Option<NonZeroU64>, // Size of the data hash table
279
+ pub field_hash_table_offset: Option<NonZeroU64>, // Offset of the field hash table
280
+ pub field_hash_table_size: Option<NonZeroU64>, // Size of the field hash table
281
+ pub tail_object_offset: Option<NonZeroU64>, // Offset of the last object
282
+ pub n_objects: u64, // Number of objects
283
+ pub n_entries: u64, // Number of entries
284
+ pub tail_entry_seqnum: u64, // Sequence number of the last entry
285
+ pub head_entry_seqnum: u64, // Sequence number of the first entry
286
+ pub entry_array_offset: Option<NonZeroU64>, // Offset of the entry array
287
+ pub head_entry_realtime: u64, // Realtime timestamp of the first entry
288
+ pub tail_entry_realtime: u64, // Realtime timestamp of the last entry
289
+ pub tail_entry_monotonic: u64, // Monotonic timestamp of the last entry
290
+}
291
+
292
+/*
293
+ NOTE: For the time being, we do not need the following fields.
294
+
295
+ // Added in 187
296
+ pub n_data: u64, // Number of data objects
297
+ pub n_fields: u64, // Number of field objects
298
+ // Added in 189
299
+ pub n_tags: u64, // Number of tag objects
300
+ pub n_entry_arrays: u64, // Number of entry array objects
301
+ // Added in 246
302
+ pub data_hash_chain_depth: u64, // Deepest chain in data hash table
303
+ pub field_hash_chain_depth: u64, // Deepest chain in field hash table
304
+ // Added in 252
305
+ pub tail_entry_array_offset: u32, // Offset to the tail entry array
306
+ pub tail_entry_array_n_entries: u32, // Number of entries in the tail entry array
307
+ // Added in 254
308
+ pub tail_entry_offset: u64, // Offset to the tail entry
309
+*/
310
+
311
+impl JournalHeader {
312
+ pub fn has_incompatible_flag(&self, flag: HeaderIncompatibleFlags) -> bool {
313
+ (self.incompatible_flags & flag as u32) != 0
314
+ }
315
+
316
+ pub fn has_compatible_flag(&self, flag: HeaderCompatibleFlags) -> bool {
317
+ (self.compatible_flags & flag as u32) != 0
318
+ }
319
+}
320
+
321
+pub enum ObjectFlags {
322
+ CompressedXz = 1 << 0,
323
+ CompressedLz4 = 1 << 1,
324
+ CompressedZstd = 1 << 2,
325
+}
326
+
327
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
328
+#[repr(u8)]
329
+pub enum ObjectType {
330
+ Unused = 0,
331
+ Data = 1,
332
+ Field = 2,
333
+ Entry = 3,
334
+ DataHashTable = 4,
335
+ FieldHashTable = 5,
336
+ EntryArray = 6,
337
+ Tag = 7,
338
+}
339
+
340
+impl TryFrom<u8> for ObjectType {
341
+ type Error = JournalError;
342
+
343
+ fn try_from(value: u8) -> Result<Self> {
344
+ match value {
345
+ 0 => Ok(ObjectType::Unused),
346
+ 1 => Ok(ObjectType::Data),
347
+ 2 => Ok(ObjectType::Field),
348
+ 3 => Ok(ObjectType::Entry),
349
+ 4 => Ok(ObjectType::DataHashTable),
350
+ 5 => Ok(ObjectType::FieldHashTable),
351
+ 6 => Ok(ObjectType::EntryArray),
352
+ 7 => Ok(ObjectType::Tag),
353
+ _ => Err(JournalError::InvalidObjectType),
354
+ }
355
+ }
356
+}
357
+
358
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
359
+#[repr(C)]
360
+pub struct ObjectHeader {
361
+ pub type_: u8,
362
+ pub flags: u8,
363
+ pub reserved: [u8; 6],
364
+ pub size: u64,
365
+}
366
+
367
+impl ObjectHeader {
368
+ pub fn xz_compressed(&self) -> bool {
369
+ (self.flags & ObjectFlags::CompressedXz as u8) != 0
370
+ }
371
+
372
+ pub fn lz4_compressed(&self) -> bool {
373
+ (self.flags & ObjectFlags::CompressedLz4 as u8) != 0
374
+ }
375
+
376
+ pub fn zstd_compressed(&self) -> bool {
377
+ (self.flags & ObjectFlags::CompressedZstd as u8) != 0
378
+ }
379
+
380
+ pub fn is_compressed(&self) -> bool {
381
+ self.zstd_compressed() | self.lz4_compressed() | self.xz_compressed()
382
+ }
383
+
384
+ pub fn aligned_size(&self) -> u64 {
385
+ (self.size + 7) & !7
386
+ }
387
+}
388
+
389
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
390
+#[repr(C)]
391
+pub struct FieldObjectHeader {
392
+ pub object_header: ObjectHeader,
393
+ pub hash: u64,
394
+ pub next_hash_offset: Option<NonZeroU64>,
395
+ pub head_data_offset: Option<NonZeroU64>,
396
+}
397
+
398
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
399
+#[repr(C)]
400
+pub struct OffsetArrayObjectHeader {
401
+ pub object_header: ObjectHeader,
402
+ pub next_offset_array: Option<NonZeroU64>,
403
+}
404
+
405
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
406
+#[repr(C)]
407
+pub struct HashItem {
408
+ pub head_hash_offset: Option<NonZeroU64>,
409
+ pub tail_hash_offset: Option<NonZeroU64>,
410
+}
411
+
412
+#[derive(Debug)]
413
+pub struct FieldObject<B: ByteSlice> {
414
+ pub header: Ref<B, FieldObjectHeader>,
415
+ pub payload: B,
416
+}
417
+
418
+impl<B: SplitByteSlice> JournalObject<B> for FieldObject<B> {
419
+ fn from_data(data: B, _is_compact: bool) -> Option<Self> {
420
+ let (header, payload) = zerocopy::Ref::from_prefix(data).ok()?;
421
+ Some(FieldObject { header, payload })
422
+ }
423
+}
424
+
425
+impl<B: SplitByteSliceMut> JournalObjectMut<B> for FieldObject<B> {
426
+ fn from_data_mut(data: B, _is_compact: bool) -> Option<Self> {
427
+ let (header, payload) = zerocopy::Ref::from_prefix(data).ok()?;
428
+ Some(FieldObject { header, payload })
429
+ }
430
+}
431
+
432
+pub enum OffsetsType<B: ByteSlice> {
433
+ Regular(Ref<B, [Option<NonZeroU64>]>),
434
+ Compact(Ref<B, [Option<NonZeroU32>]>),
435
+}
436
+
437
+impl<B: ByteSlice> OffsetsType<B> {
438
+ pub fn get(&self, index: usize) -> Option<NonZeroU64> {
439
+ match self {
440
+ OffsetsType::Regular(offsets) => offsets[index],
441
+ OffsetsType::Compact(offsets) => offsets[index].map(NonZeroU64::from),
442
+ }
443
+ }
444
+}
445
+
446
+impl<B: ByteSliceMut> OffsetsType<B> {
447
+ pub fn set(&mut self, index: usize, value: NonZeroU64) {
448
+ match self {
449
+ OffsetsType::Regular(offsets) => offsets[index] = Some(value),
450
+ OffsetsType::Compact(offsets) => offsets[index] = NonZeroU32::new(value.get() as u32),
451
+ }
452
+ }
453
+}
454
+
455
+impl<B: ByteSlice> std::fmt::Debug for OffsetsType<B> {
456
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
457
+ match self {
458
+ OffsetsType::Regular(items) => write!(f, "Regular({} items)", items.len()),
459
+ OffsetsType::Compact(items) => write!(f, "Compact({} items)", items.len()),
460
+ }
461
+ }
462
+}
463
+
464
+pub struct OffsetArrayObject<B: ByteSlice> {
465
+ pub header: Ref<B, OffsetArrayObjectHeader>,
466
+ pub items: OffsetsType<B>,
467
+}
468
+
469
+impl<B: ByteSlice> OffsetArrayObject<B> {
470
+ pub fn capacity(&self) -> usize {
471
+ match &self.items {
472
+ OffsetsType::Regular(offsets) => offsets.len(),
473
+ OffsetsType::Compact(offsets) => offsets.len(),
474
+ }
475
+ }
476
+
477
+ pub fn len(&self, remaining_items: usize) -> usize {
478
+ self.capacity().min(remaining_items)
479
+ }
480
+
481
+ pub fn is_empty(&self, remaining_items: usize) -> bool {
482
+ self.len(remaining_items) == 0
483
+ }
484
+
485
+ pub fn get(&self, index: usize, remaining_items: usize) -> Result<Option<NonZeroU64>> {
486
+ if self.is_empty(remaining_items) {
487
+ return Err(JournalError::EmptyOffsetArrayNode);
488
+ }
489
+
490
+ Ok(self.items.get(index))
491
+ }
492
+}
493
+
494
+impl<B: ByteSliceMut> OffsetArrayObject<B> {
495
+ pub fn set(&mut self, index: usize, offset: NonZeroU64) -> Result<()> {
496
+ if index >= self.capacity() {
497
+ return Err(JournalError::OutOfBoundsIndex);
498
+ }
499
+
500
+ self.items.set(index, offset);
501
+ Ok(())
502
+ }
503
+}
504
+
505
+impl<B: ByteSlice> std::fmt::Debug for OffsetArrayObject<B> {
506
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
507
+ f.debug_struct("JournalHeader")
508
+ .field("header", &self.header)
509
+ .finish()
510
+ }
511
+}
512
+
513
+impl<B: SplitByteSlice> JournalObject<B> for OffsetArrayObject<B> {
514
+ fn from_data(data: B, is_compact: bool) -> Option<Self> {
515
+ let (header_data, items_data) = data
516
+ .split_at(std::mem::size_of::<OffsetArrayObjectHeader>())
517
+ .ok()?;
518
+
519
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
520
+
521
+ let items_type = if is_compact {
522
+ let compact_items = zerocopy::Ref::from_bytes(items_data).ok()?;
523
+ OffsetsType::Compact(compact_items)
524
+ } else {
525
+ let regular_items = zerocopy::Ref::from_bytes(items_data).ok()?;
526
+ OffsetsType::Regular(regular_items)
527
+ };
528
+
529
+ Some(OffsetArrayObject {
530
+ header,
531
+ items: items_type,
532
+ })
533
+ }
534
+}
535
+
536
+impl<B: SplitByteSliceMut> JournalObjectMut<B> for OffsetArrayObject<B> {
537
+ fn from_data_mut(data: B, is_compact: bool) -> Option<Self> {
538
+ let (header_data, items_data) = data
539
+ .split_at(std::mem::size_of::<OffsetArrayObjectHeader>())
540
+ .ok()?;
541
+
542
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
543
+
544
+ let items_type = if is_compact {
545
+ let compact_items = zerocopy::Ref::from_bytes(items_data).ok()?;
546
+ OffsetsType::Compact(compact_items)
547
+ } else {
548
+ let regular_items = zerocopy::Ref::from_bytes(items_data).ok()?;
549
+ OffsetsType::Regular(regular_items)
550
+ };
551
+
552
+ Some(OffsetArrayObject {
553
+ header,
554
+ items: items_type,
555
+ })
556
+ }
557
+}
558
+
559
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
560
+#[repr(C)]
561
+pub struct EntryObjectHeader {
562
+ pub object_header: ObjectHeader,
563
+ pub seqnum: u64,
564
+ pub realtime: u64,
565
+ pub monotonic: u64,
566
+ pub boot_id: [u8; 16], // UUID/128-bit ID
567
+ pub xor_hash: u64,
568
+}
569
+
570
+// For regular (non-compact) format - an array of these follows the header
571
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
572
+#[repr(C)]
573
+pub struct RegularEntryItem {
574
+ pub object_offset: u64,
575
+ pub hash: u64,
576
+}
577
+
578
+// For compact format - an array of these follows the header
579
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
580
+#[repr(C)]
581
+pub struct CompactEntryItem {
582
+ pub object_offset: u32,
583
+}
584
+
585
+pub enum EntryItemsType<B: ByteSlice> {
586
+ Regular(Ref<B, [RegularEntryItem]>),
587
+ Compact(Ref<B, [CompactEntryItem]>),
588
+}
589
+
590
+impl<B: ByteSliceMut> EntryItemsType<B> {
591
+ pub fn set(&mut self, index: usize, object_offset: NonZeroU64, hash: Option<u64>) {
592
+ match self {
593
+ EntryItemsType::Regular(entry_items) => {
594
+ entry_items[index].object_offset = object_offset.get();
595
+ entry_items[index].hash = hash.unwrap();
596
+ }
597
+ EntryItemsType::Compact(entry_items) => {
598
+ debug_assert!(hash.is_none());
599
+ assert!(object_offset.get() < u32::MAX as u64);
600
+ entry_items[index].object_offset = object_offset.get() as u32;
601
+ }
602
+ }
603
+ }
604
+}
605
+
606
+impl<B: ByteSlice> EntryItemsType<B> {
607
+ pub fn get(&self, index: usize) -> u64 {
608
+ match self {
609
+ EntryItemsType::Regular(entry_items) => entry_items[index].object_offset,
610
+ EntryItemsType::Compact(entry_items) => entry_items[index].object_offset as u64,
611
+ }
612
+ }
613
+
614
+ pub fn len(&self) -> usize {
615
+ match self {
616
+ EntryItemsType::Regular(entry_items) => entry_items.len(),
617
+ EntryItemsType::Compact(entry_items) => entry_items.len(),
618
+ }
619
+ }
620
+
621
+ pub fn is_empty(&self) -> bool {
622
+ match self {
623
+ EntryItemsType::Regular(entry_items) => entry_items.is_empty(),
624
+ EntryItemsType::Compact(entry_items) => entry_items.is_empty(),
625
+ }
626
+ }
627
+}
628
+
629
+impl<B: ByteSlice> std::fmt::Debug for EntryItemsType<B> {
630
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
631
+ match self {
632
+ EntryItemsType::Regular(items) => write!(f, "Regular({} items)", items.len()),
633
+ EntryItemsType::Compact(items) => write!(f, "Compact({} items)", items.len()),
634
+ }
635
+ }
636
+}
637
+
638
+pub struct EntryObject<B: ByteSlice> {
639
+ pub header: Ref<B, EntryObjectHeader>,
640
+ pub items: EntryItemsType<B>,
641
+}
642
+
643
+impl<B: ByteSlice> std::fmt::Debug for EntryObject<B> {
644
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
645
+ f.debug_struct("EntryObject")
646
+ .field("header", &self.header)
647
+ .field("items", &self.items)
648
+ .finish()
649
+ }
650
+}
651
+
652
+impl<B: SplitByteSlice> JournalObject<B> for EntryObject<B> {
653
+ fn from_data(data: B, is_compact: bool) -> Option<Self> {
654
+ let (header_data, items_data) = data
655
+ .split_at(std::mem::size_of::<EntryObjectHeader>())
656
+ .ok()?;
657
+
658
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
659
+
660
+ let items_type = if is_compact {
661
+ let compact_items = zerocopy::Ref::from_bytes(items_data).ok()?;
662
+ EntryItemsType::Compact(compact_items)
663
+ } else {
664
+ let regular_items = zerocopy::Ref::from_bytes(items_data).ok()?;
665
+ EntryItemsType::Regular(regular_items)
666
+ };
667
+
668
+ Some(EntryObject {
669
+ header,
670
+ items: items_type,
671
+ })
672
+ }
673
+}
674
+
675
+impl<B: SplitByteSliceMut> JournalObjectMut<B> for EntryObject<B> {
676
+ fn from_data_mut(data: B, is_compact: bool) -> Option<Self> {
677
+ let (header_data, items_data) = data
678
+ .split_at(std::mem::size_of::<EntryObjectHeader>())
679
+ .ok()?;
680
+
681
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
682
+
683
+ let items_type = if is_compact {
684
+ let compact_items = zerocopy::Ref::from_bytes(items_data).ok()?;
685
+ EntryItemsType::Compact(compact_items)
686
+ } else {
687
+ let regular_items = zerocopy::Ref::from_bytes(items_data).ok()?;
688
+ EntryItemsType::Regular(regular_items)
689
+ };
690
+
691
+ Some(EntryObject {
692
+ header,
693
+ items: items_type,
694
+ })
695
+ }
696
+}
697
+
698
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
699
+#[repr(C)]
700
+pub struct DataObjectHeader {
701
+ pub object_header: ObjectHeader,
702
+ pub hash: u64,
703
+ pub next_hash_offset: Option<NonZeroU64>,
704
+ pub next_field_offset: Option<NonZeroU64>,
705
+ pub entry_offset: Option<NonZeroU64>,
706
+ pub entry_array_offset: Option<NonZeroU64>,
707
+ pub n_entries: Option<NonZeroU64>,
708
+}
709
+
710
+impl DataObjectHeader {
711
+ pub fn xz_compressed(&self) -> bool {
712
+ self.object_header.xz_compressed()
713
+ }
714
+
715
+ pub fn lz4_compressed(&self) -> bool {
716
+ self.object_header.lz4_compressed()
717
+ }
718
+
719
+ pub fn zstd_compressed(&self) -> bool {
720
+ self.object_header.zstd_compressed()
721
+ }
722
+
723
+ pub fn is_compressed(&self) -> bool {
724
+ self.object_header.is_compressed()
725
+ }
726
+
727
+ pub fn inlined_cursor(&self) -> Option<InlinedCursor> {
728
+ let inlined_offset = self.entry_offset?;
729
+ let cursor = match self.n_entries?.get() {
730
+ 1 => None,
731
+ n => {
732
+ let total_items = unsafe { NonZeroUsize::new_unchecked(n as usize - 1) };
733
+ Some(Cursor::at_head(List::new(
734
+ self.entry_array_offset?,
735
+ total_items,
736
+ )))
737
+ }
738
+ };
739
+ Some(InlinedCursor::new(inlined_offset, cursor))
740
+ }
741
+}
742
+
743
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable, PartialEq, Eq)]
744
+#[repr(C)]
745
+pub struct CompactDataFields {
746
+ pub tail_entry_array_offset: u32,
747
+ pub tail_entry_array_n_entries: u32,
748
+}
749
+
750
+#[derive(PartialEq, Eq)]
751
+pub enum DataPayloadType<B: ByteSlice> {
752
+ Regular(B),
753
+ Compact {
754
+ compact_fields: Ref<B, CompactDataFields>,
755
+ payload: B,
756
+ },
757
+}
758
+
759
+impl<B: ByteSlice> std::fmt::Debug for DataPayloadType<B> {
760
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
761
+ match self {
762
+ DataPayloadType::Regular(payload) => write!(f, "Regular({} bytes)", payload.len()),
763
+ DataPayloadType::Compact {
764
+ compact_fields,
765
+ payload,
766
+ } => write!(
767
+ f,
768
+ "Compact(fields: {:?}, payload: {} bytes)",
769
+ compact_fields,
770
+ payload.len()
771
+ ),
772
+ }
773
+ }
774
+}
775
+
776
+// Complete Data Object structure
777
+pub struct DataObject<B: ByteSlice> {
778
+ pub header: Ref<B, DataObjectHeader>,
779
+ pub payload: DataPayloadType<B>,
780
+}
781
+
782
+impl<B: ByteSlice> std::fmt::Debug for DataObject<B> {
783
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
784
+ f.debug_struct("DataObject")
785
+ .field("header", &self.header)
786
+ .field("payload", &self.payload)
787
+ .finish()
788
+ }
789
+}
790
+
791
+impl<B: SplitByteSlice> JournalObject<B> for DataObject<B> {
792
+ fn from_data(data: B, is_compact: bool) -> Option<Self> {
793
+ let (header_data, remaining_data) = data
794
+ .split_at(std::mem::size_of::<DataObjectHeader>())
795
+ .ok()?;
796
+
797
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
798
+
799
+ let payload = if is_compact {
800
+ let (fields_data, payload_data) = remaining_data
801
+ .split_at(std::mem::size_of::<CompactDataFields>())
802
+ .ok()?;
803
+
804
+ let compact_fields = zerocopy::Ref::from_bytes(fields_data).ok()?;
805
+
806
+ DataPayloadType::Compact {
807
+ compact_fields,
808
+ payload: payload_data,
809
+ }
810
+ } else {
811
+ DataPayloadType::Regular(remaining_data)
812
+ };
813
+
814
+ Some(DataObject { header, payload })
815
+ }
816
+}
817
+
818
+impl<B: SplitByteSliceMut> JournalObjectMut<B> for DataObject<B> {
819
+ fn from_data_mut(data: B, is_compact: bool) -> Option<Self> {
820
+ let (header_data, remaining_data) = data
821
+ .split_at(std::mem::size_of::<DataObjectHeader>())
822
+ .ok()?;
823
+
824
+ let header = zerocopy::Ref::from_bytes(header_data).ok()?;
825
+
826
+ let payload = if is_compact {
827
+ let (fields_data, payload_data) = remaining_data
828
+ .split_at(std::mem::size_of::<CompactDataFields>())
829
+ .ok()?;
830
+
831
+ let compact_fields = zerocopy::Ref::from_bytes(fields_data).ok()?;
832
+
833
+ DataPayloadType::Compact {
834
+ compact_fields,
835
+ payload: payload_data,
836
+ }
837
+ } else {
838
+ DataPayloadType::Regular(remaining_data)
839
+ };
840
+
841
+ Some(DataObject { header, payload })
842
+ }
843
+}
844
+
845
+impl<B: ByteSlice> DataObject<B> {
846
+ pub fn payload_bytes(&self) -> &[u8] {
847
+ match &self.payload {
848
+ DataPayloadType::Regular(payload) => payload,
849
+ DataPayloadType::Compact { payload, .. } => payload,
850
+ }
851
+ }
852
+
853
+ pub fn inlined_cursor(&self) -> Option<InlinedCursor> {
854
+ self.header.inlined_cursor()
855
+ }
856
+
857
+ pub fn is_compressed(&self) -> bool {
858
+ self.header.is_compressed()
859
+ }
860
+
861
+ pub fn xz_compressed(&self) -> bool {
862
+ self.header.xz_compressed()
863
+ }
864
+
865
+ pub fn lz4_compressed(&self) -> bool {
866
+ self.header.lz4_compressed()
867
+ }
868
+
869
+ pub fn zstd_compressed(&self) -> bool {
870
+ self.header.zstd_compressed()
871
+ }
872
+
873
+ pub fn decompress(&self, buf: &mut Vec<u8>) -> Result<usize> {
874
+ debug_assert!(self.is_compressed());
875
+
876
+ if self.zstd_compressed() {
877
+ use ruzstd::decoding::StreamingDecoder;
878
+ use ruzstd::io::Read;
879
+
880
+ let payload = self.payload_bytes();
881
+ let mut decoder =
882
+ StreamingDecoder::new(payload).map_err(|_| JournalError::DecompressorError)?;
883
+
884
+ buf.clear();
885
+ decoder
886
+ .read_to_end(buf)
887
+ .map_err(|_| JournalError::DecompressorError)
888
+ } else {
889
+ Err(JournalError::UnknownCompressionMethod)
890
+ }
891
+ }
892
+}
893
+
894
+// SHA-256 HMAC is 32 bytes (256 bits)
895
+pub const TAG_LENGTH: usize = 256 / 8;
896
+
897
+#[derive(Debug, Copy, Clone, FromBytes, IntoBytes, KnownLayout, Immutable)]
898
+#[repr(C)]
899
+pub struct TagObjectHeader {
900
+ pub object_header: ObjectHeader,
901
+ pub seqnum: u64,
902
+ pub epoch: u64,
903
+ pub tag: [u8; TAG_LENGTH], // SHA-256 HMAC
904
+}
905
+
906
+pub struct TagObject<B: ByteSlice> {
907
+ pub header: Ref<B, TagObjectHeader>,
908
+}
909
+
910
+impl<B: ByteSlice> std::fmt::Debug for TagObject<B> {
911
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
912
+ f.debug_struct("TagObject")
913
+ .field("header", &self.header)
914
+ .finish()
915
+ }
916
+}
917
+
918
+impl<B: SplitByteSlice> JournalObject<B> for TagObject<B> {
919
+ fn from_data(data: B, _is_compact: bool) -> Option<Self> {
920
+ let header = zerocopy::Ref::from_bytes(data).ok()?;
921
+ Some(TagObject { header })
922
+ }
923
+}
924
+
925
+impl<B: SplitByteSliceMut> JournalObjectMut<B> for TagObject<B> {
926
+ fn from_data_mut(data: B, _is_compact: bool) -> Option<Self> {
927
+ let header = zerocopy::Ref::from_bytes(data).ok()?;
928
+ Some(TagObject { header })
929
+ }
930
+}
931
+
932
+impl<B: ByteSlice> TagObject<B> {
933
+ // Helper function to format tag as hex string
934
+ pub fn tag_as_hex(&self) -> String {
935
+ self.header
936
+ .tag
937
+ .iter()
938
+ .map(|b| format!("{:02x}", b))
939
+ .collect()
940
+ }
941
+}
src/crates/jf/journal_file/src/offset_array.rs
new
+686
@@ -0,0 +1,686 @@
1
+use crate::file::JournalFile;
2
+use error::{JournalError, Result};
3
+use std::num::{NonZeroU64, NonZeroUsize};
4
+use window_manager::MemoryMap;
5
+
6
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
7
+pub enum Direction {
8
+ Forward,
9
+ Backward,
10
+}
11
+
12
+/// A reference to a single array of offsets in the journal file
13
+pub struct Node {
14
+ offset: NonZeroU64,
15
+ next_offset: Option<NonZeroU64>,
16
+ capacity: NonZeroUsize,
17
+ // Number of items remaining in this array and subsequent arrays
18
+ remaining_items: NonZeroUsize,
19
+}
20
+
21
+impl Node {
22
+ /// Create a new offset array reference
23
+ fn new<M: MemoryMap>(
24
+ journal_file: &JournalFile<M>,
25
+ offset: NonZeroU64,
26
+ remaining_items: NonZeroUsize,
27
+ ) -> Result<Self> {
28
+ let array = journal_file.offset_array_ref(offset)?;
29
+ let capacity =
30
+ NonZeroUsize::new(array.capacity()).ok_or(JournalError::EmptyOffsetArrayNode)?;
31
+
32
+ Ok(Self {
33
+ offset,
34
+ next_offset: array.header.next_offset_array,
35
+ capacity,
36
+ remaining_items,
37
+ })
38
+ }
39
+
40
+ /// Get the offset of this array in the file
41
+ pub fn offset(&self) -> NonZeroU64 {
42
+ self.offset
43
+ }
44
+
45
+ /// Get the maximum number of items this array can hold
46
+ pub fn capacity(&self) -> NonZeroUsize {
47
+ self.capacity
48
+ }
49
+
50
+ /// Get the number of items available in this array
51
+ pub fn len(&self) -> NonZeroUsize {
52
+ self.capacity.min(self.remaining_items)
53
+ }
54
+
55
+ /// Check if this array has a next array in the chain
56
+ pub fn has_next(&self) -> bool {
57
+ self.next_offset.is_some() && self.remaining_items > self.len()
58
+ }
59
+
60
+ /// Get the next array in the chain, if any
61
+ pub fn next<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Option<Self>> {
62
+ if !self.has_next() {
63
+ return Ok(None);
64
+ }
65
+
66
+ let next_offset = self.next_offset.unwrap();
67
+ let remaining_items = {
68
+ let n = self.remaining_items.get().saturating_sub(self.len().get());
69
+ NonZeroUsize::new(n).ok_or(JournalError::EmptyOffsetArrayNode)?
70
+ };
71
+ let node = Self::new(journal_file, next_offset, remaining_items);
72
+
73
+ Some(node).transpose()
74
+ }
75
+
76
+ /// Get an item at the specified index
77
+ pub fn get<M: MemoryMap>(
78
+ &self,
79
+ journal_file: &JournalFile<M>,
80
+ index: usize,
81
+ ) -> Result<Option<NonZeroU64>> {
82
+ if index >= self.len().get() {
83
+ return Err(JournalError::InvalidOffsetArrayIndex);
84
+ }
85
+
86
+ let array = journal_file.offset_array_ref(self.offset)?;
87
+ array.get(index, self.remaining_items.get())
88
+ }
89
+
90
+ /// Returns the first index where the predicate returns false, or array length if
91
+ /// the predicate is true for all elements
92
+ pub fn partition_point<M, F>(
93
+ &self,
94
+ journal_file: &JournalFile<M>,
95
+ left: usize,
96
+ right: usize,
97
+ predicate: F,
98
+ ) -> Result<usize>
99
+ where
100
+ M: MemoryMap,
101
+ F: Fn(NonZeroU64) -> Result<bool>,
102
+ {
103
+ let mut left = left;
104
+ let mut right = right;
105
+
106
+ debug_assert!(left <= right);
107
+ debug_assert!(right <= self.len().get());
108
+
109
+ while left != right {
110
+ let mid = left.midpoint(right);
111
+ let Some(offset) = self.get(journal_file, mid)? else {
112
+ return Err(JournalError::InvalidOffset);
113
+ };
114
+
115
+ if predicate(offset)? {
116
+ left = mid + 1;
117
+ } else {
118
+ right = mid;
119
+ }
120
+ }
121
+
122
+ Ok(left)
123
+ }
124
+
125
+ /// Find the forward or backward (depending on direction) position that matches the predicate.
126
+ pub fn directed_partition_point<M, F>(
127
+ &self,
128
+ journal_file: &JournalFile<M>,
129
+ left: usize,
130
+ right: usize,
131
+ predicate: F,
132
+ direction: Direction,
133
+ ) -> Result<Option<usize>>
134
+ where
135
+ M: MemoryMap,
136
+ F: Fn(NonZeroU64) -> Result<bool>,
137
+ {
138
+ let index = self.partition_point(journal_file, left, right, predicate)?;
139
+
140
+ Ok(match direction {
141
+ Direction::Forward => {
142
+ if index < self.len().get() {
143
+ Some(index)
144
+ } else {
145
+ None
146
+ }
147
+ }
148
+ Direction::Backward => {
149
+ if index > 0 {
150
+ Some(index - 1)
151
+ } else {
152
+ None
153
+ }
154
+ }
155
+ })
156
+ }
157
+}
158
+
159
+impl std::fmt::Debug for Node {
160
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
161
+ let next_offset = self.next_offset.map(|x| x.get()).unwrap_or(0);
162
+
163
+ f.debug_struct("Node")
164
+ .field("offset", &format!("0x{:x}", self.offset))
165
+ .field("next_offset", &format!("0x{:x}", next_offset))
166
+ .field("capacity", &self.capacity)
167
+ .field("len", &self.len())
168
+ .field("remaining_items", &self.remaining_items)
169
+ .finish()
170
+ }
171
+}
172
+
173
+/// A linked list of offset arrays
174
+#[derive(Copy, Clone)]
175
+pub struct List {
176
+ head_offset: NonZeroU64,
177
+ total_items: NonZeroUsize,
178
+}
179
+
180
+impl std::fmt::Debug for List {
181
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
182
+ f.debug_struct("List")
183
+ .field("head_offset", &format!("0x{:x}", self.head_offset))
184
+ .field("total_items", &self.total_items)
185
+ .finish()
186
+ }
187
+}
188
+
189
+impl List {
190
+ /// Create a new list from head offset and total items
191
+ pub fn new(head_offset: NonZeroU64, total_items: NonZeroUsize) -> Self {
192
+ Self {
193
+ head_offset,
194
+ total_items,
195
+ }
196
+ }
197
+
198
+ /// Get the head array of this chain
199
+ pub fn head<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Node> {
200
+ Node::new(journal_file, self.head_offset, self.total_items)
201
+ }
202
+
203
+ /// Get the tail array of this list by traversing from head to tail
204
+ pub fn tail<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Node> {
205
+ let mut current = self.head(journal_file)?;
206
+
207
+ while let Some(next) = current.next(journal_file)? {
208
+ current = next;
209
+ }
210
+
211
+ Ok(current)
212
+ }
213
+
214
+ /// Get a cursor at the first position in the chain
215
+ pub fn cursor_head(self) -> Cursor {
216
+ Cursor::at_head(self)
217
+ }
218
+
219
+ /// Get a cursor at the last position in the chain
220
+ pub fn cursor_tail<M: MemoryMap>(self, journal_file: &JournalFile<M>) -> Result<Cursor> {
221
+ Cursor::at_tail(journal_file, self)
222
+ }
223
+
224
+ /// Finds the first/last array item position where the predicate function becomes false
225
+ /// in a chain of offset arrays.
226
+ ///
227
+ /// # Parameters
228
+ /// * `predicate` - Function that takes an array item value and returns true if the search should continue.
229
+ /// * `direction` - Direction of the search (Forward or Backward)
230
+ pub fn directed_partition_point<M, F>(
231
+ self,
232
+ journal_file: &JournalFile<M>,
233
+ predicate: F,
234
+ direction: Direction,
235
+ ) -> Result<Option<Cursor>>
236
+ where
237
+ M: MemoryMap,
238
+ F: Fn(NonZeroU64) -> Result<bool>,
239
+ {
240
+ let mut last_cursor: Option<Cursor> = None;
241
+
242
+ let mut node = self.head(journal_file)?;
243
+
244
+ loop {
245
+ let left = 0;
246
+ let right = node.len().get();
247
+
248
+ if let Some(index) =
249
+ node.directed_partition_point(journal_file, left, right, &predicate, direction)?
250
+ {
251
+ let cursor = Cursor::at_position(
252
+ journal_file,
253
+ self,
254
+ node.offset,
255
+ index,
256
+ node.remaining_items,
257
+ )?;
258
+
259
+ match direction {
260
+ Direction::Forward => {
261
+ return Ok(Some(cursor));
262
+ }
263
+ Direction::Backward => {
264
+ // In backward direction, save this match and continue
265
+ // to ensure we'll find the last match
266
+ last_cursor = Some(cursor);
267
+
268
+ // If this match is at the end of the array and there's a next array,
269
+ // we should check the next array as well
270
+ if index == node.len().get() - 1 && node.has_next() {
271
+ // continue;
272
+ } else {
273
+ return Ok(last_cursor);
274
+ }
275
+ }
276
+ }
277
+ } else if direction == Direction::Backward {
278
+ // No match in this array for backward direction
279
+ return Ok(last_cursor);
280
+ }
281
+
282
+ if let Some(nd) = node.next(journal_file)? {
283
+ node = nd;
284
+ } else {
285
+ break;
286
+ }
287
+ }
288
+
289
+ // For backward direction, return the last match we found (if any)
290
+ if direction == Direction::Backward {
291
+ return Ok(last_cursor);
292
+ }
293
+
294
+ // No match found in any array
295
+ Ok(None)
296
+ }
297
+}
298
+
299
+/// A cursor pointing to a specific position within an offset array chain
300
+#[derive(Clone, Copy)]
301
+pub struct Cursor {
302
+ list: List,
303
+ array_offset: NonZeroU64,
304
+ array_index: usize,
305
+ remaining_items: NonZeroUsize,
306
+}
307
+
308
+impl Cursor {
309
+ pub fn head(&self) -> Self {
310
+ Self::at_head(self.list)
311
+ }
312
+
313
+ /// Create a cursor at the head of the chain
314
+ pub fn at_head(list: List) -> Self {
315
+ Self {
316
+ list,
317
+ array_offset: list.head_offset,
318
+ array_index: 0,
319
+ remaining_items: list.total_items,
320
+ }
321
+ }
322
+
323
+ /// Create a cursor at the tail of the chain
324
+ pub fn at_tail<M: MemoryMap>(journal_file: &JournalFile<M>, list: List) -> Result<Self> {
325
+ let mut current_array = list.head(journal_file)?;
326
+
327
+ while let Some(next_array) = current_array.next(journal_file)? {
328
+ current_array = next_array;
329
+ }
330
+
331
+ Ok(Self {
332
+ list,
333
+ array_offset: current_array.offset,
334
+ array_index: current_array.len().get() - 1,
335
+ remaining_items: current_array.len(),
336
+ })
337
+ }
338
+
339
+ /// Create a cursor at a specific position
340
+ pub fn at_position<M: MemoryMap>(
341
+ journal_file: &JournalFile<M>,
342
+ offset_array_list: List,
343
+ array_offset: NonZeroU64,
344
+ array_index: usize,
345
+ remaining_items: NonZeroUsize,
346
+ ) -> Result<Self> {
347
+ debug_assert!(offset_array_list.total_items >= remaining_items);
348
+
349
+ // Verify the array exists
350
+ let array = Node::new(journal_file, array_offset, remaining_items)?;
351
+
352
+ // Verify the index is valid
353
+ if array_index >= array.len().get() {
354
+ return Err(JournalError::InvalidOffsetArrayIndex);
355
+ }
356
+
357
+ Ok(Self {
358
+ list: offset_array_list,
359
+ array_offset,
360
+ array_index,
361
+ remaining_items,
362
+ })
363
+ }
364
+
365
+ /// Get the current array this cursor points to
366
+ pub fn node<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Node> {
367
+ Node::new(journal_file, self.array_offset, self.remaining_items)
368
+ }
369
+
370
+ pub fn value<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Option<NonZeroU64>> {
371
+ self.node(journal_file)?.get(journal_file, self.array_index)
372
+ }
373
+
374
+ /// Move to the next position
375
+ pub fn next<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Option<Self>> {
376
+ let array_node = self.node(journal_file)?;
377
+
378
+ // FIXME: overtly defensive/expensive...
379
+ if self.array_index + 1 < array_node.len().get() {
380
+ // Next item is in the same array
381
+ return Ok(Some(Self {
382
+ list: self.list,
383
+ array_offset: self.array_offset,
384
+ array_index: self.array_index + 1,
385
+ remaining_items: self.remaining_items,
386
+ }));
387
+ }
388
+
389
+ if !array_node.has_next() {
390
+ return Ok(None);
391
+ }
392
+
393
+ let next_array = array_node.next(journal_file)?.unwrap();
394
+
395
+ match NonZeroUsize::new(
396
+ self.remaining_items
397
+ .get()
398
+ .saturating_sub(array_node.len().get()),
399
+ ) {
400
+ None => Ok(None),
401
+ Some(remaining_items) => Ok(Some(Self {
402
+ list: self.list,
403
+ array_offset: next_array.offset,
404
+ array_index: 0,
405
+ remaining_items,
406
+ })),
407
+ }
408
+ }
409
+
410
+ /// Move to the previous position
411
+ pub fn previous<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Option<Self>> {
412
+ if self.array_index > 0 {
413
+ // Previous item is in the same array
414
+ return Ok(Some(Self {
415
+ list: self.list,
416
+ array_offset: self.array_offset,
417
+ array_index: self.array_index - 1,
418
+ remaining_items: self.remaining_items,
419
+ }));
420
+ }
421
+
422
+ if self.array_offset == self.list.head_offset {
423
+ return Ok(None);
424
+ }
425
+
426
+ let mut node = self.list.head(journal_file)?;
427
+ while node.has_next() {
428
+ if node.next_offset == Some(self.array_offset) {
429
+ return Ok(Some(Self {
430
+ list: self.list,
431
+ array_offset: node.offset,
432
+ array_index: node.len().get() - 1,
433
+ remaining_items: node.remaining_items,
434
+ }));
435
+ }
436
+
437
+ node = node.next(journal_file)?.unwrap();
438
+ }
439
+
440
+ Err(JournalError::InvalidOffsetArrayOffset)
441
+ }
442
+}
443
+
444
+impl std::fmt::Debug for Cursor {
445
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
446
+ f.debug_struct("Cursor")
447
+ .field("array_offset", &format!("0x{:x}", self.array_offset))
448
+ .field("array_index", &self.array_index)
449
+ .field("remaining_items", &self.remaining_items)
450
+ .finish()
451
+ }
452
+}
453
+
454
+#[derive(Debug, Copy, Clone)]
455
+pub struct InlinedCursor {
456
+ inlined_offset: NonZeroU64,
457
+ cursor: Option<Cursor>,
458
+ at_inlined_offset: bool,
459
+}
460
+
461
+impl InlinedCursor {
462
+ pub fn new(inlined_offset: NonZeroU64, cursor: Option<Cursor>) -> Self {
463
+ Self {
464
+ inlined_offset,
465
+ cursor,
466
+ at_inlined_offset: true,
467
+ }
468
+ }
469
+
470
+ pub fn head(&self) -> Self {
471
+ Self {
472
+ inlined_offset: self.inlined_offset,
473
+ cursor: self.cursor.as_ref().map(|c| c.head()),
474
+ at_inlined_offset: true,
475
+ }
476
+ }
477
+
478
+ pub fn tail<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Self> {
479
+ // Start with a copy of the current cursor
480
+ let mut result = *self;
481
+
482
+ // If we have an entry array list cursor, move it to the tail
483
+ if let Some(cursor) = self.cursor {
484
+ result.cursor = Some(cursor.list.cursor_tail(journal_file)?);
485
+ result.at_inlined_offset = false;
486
+ }
487
+
488
+ Ok(result)
489
+ }
490
+
491
+ fn next<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Option<Self>> {
492
+ // Case 1: We're at the inlined entry, move to the first array entry
493
+ if self.at_inlined_offset {
494
+ if self.cursor.is_some() {
495
+ return Ok(Some(Self {
496
+ inlined_offset: self.inlined_offset,
497
+ cursor: self.cursor,
498
+ at_inlined_offset: false,
499
+ }));
500
+ } else {
501
+ return Ok(None);
502
+ }
503
+ }
504
+
505
+ // Case 2: We're already in the entry array
506
+ if let Some(current_cursor) = self.cursor.as_ref() {
507
+ let next_cursor = current_cursor.next(journal_file)?;
508
+
509
+ if next_cursor.is_some() {
510
+ return Ok(Some(Self {
511
+ inlined_offset: self.inlined_offset,
512
+ cursor: next_cursor,
513
+ at_inlined_offset: false,
514
+ }));
515
+ } else {
516
+ return Ok(None);
517
+ }
518
+ }
519
+
520
+ // No more entries
521
+ Ok(None)
522
+ }
523
+
524
+ fn previous<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Option<Self>> {
525
+ if self.at_inlined_offset {
526
+ return Ok(None);
527
+ }
528
+
529
+ if let Some(current_cursor) = self.cursor {
530
+ // Try to move to the previous position in the array
531
+ if let Some(prev_cursor) = current_cursor.previous(journal_file)? {
532
+ // We can move back within the array
533
+ let mut ic = *self;
534
+ ic.cursor = Some(prev_cursor);
535
+ return Ok(Some(ic));
536
+ } else {
537
+ // We're at the first array position, move to the inlined entry
538
+ let mut ic = *self;
539
+ ic.at_inlined_offset = true;
540
+ return Ok(Some(ic));
541
+ }
542
+ }
543
+
544
+ unreachable!();
545
+ }
546
+
547
+ pub fn value<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<Option<NonZeroU64>> {
548
+ // Case 1: We're at the inlined entry
549
+ if self.at_inlined_offset {
550
+ return Ok(Some(self.inlined_offset));
551
+ }
552
+
553
+ // Case 2: We're in the entry array
554
+ if let Some(cursor) = self.cursor {
555
+ return cursor.value(journal_file);
556
+ }
557
+
558
+ unreachable!();
559
+ }
560
+
561
+ pub fn next_until<M: MemoryMap>(
562
+ &mut self,
563
+ journal_file: &JournalFile<M>,
564
+ offset: NonZeroU64,
565
+ ) -> Result<Option<NonZeroU64>> {
566
+ let Some(current_offset) = self.value(journal_file)? else {
567
+ return Ok(None);
568
+ };
569
+
570
+ if current_offset >= offset {
571
+ return Ok(Some(current_offset));
572
+ }
573
+
574
+ while let Some(ic) = self.next(journal_file)? {
575
+ *self = ic;
576
+
577
+ let Some(current_offset) = self.value(journal_file)? else {
578
+ break;
579
+ };
580
+
581
+ if current_offset >= offset {
582
+ return Ok(Some(current_offset));
583
+ }
584
+ }
585
+
586
+ Ok(None)
587
+ }
588
+
589
+ pub fn previous_until<M: MemoryMap>(
590
+ &mut self,
591
+ journal_file: &JournalFile<M>,
592
+ offset: NonZeroU64,
593
+ ) -> Result<Option<NonZeroU64>> {
594
+ let Some(current_offset) = self.value(journal_file)? else {
595
+ return Ok(None);
596
+ };
597
+
598
+ if current_offset <= offset {
599
+ return Ok(Some(current_offset));
600
+ }
601
+
602
+ while let Some(ic) = self.previous(journal_file)? {
603
+ *self = ic;
604
+
605
+ let Some(current_offset) = ic.value(journal_file)? else {
606
+ break;
607
+ };
608
+
609
+ if current_offset <= offset {
610
+ return Ok(Some(current_offset));
611
+ }
612
+ }
613
+
614
+ Ok(None)
615
+ }
616
+
617
+ pub fn directed_partition_point<M, F>(
618
+ &self,
619
+ journal_file: &JournalFile<M>,
620
+ predicate: F,
621
+ direction: Direction,
622
+ ) -> Result<Option<Self>>
623
+ where
624
+ M: MemoryMap,
625
+ F: Fn(NonZeroU64) -> Result<bool>,
626
+ {
627
+ // Variables to track our best match
628
+ let mut best_match: Option<Self> = None;
629
+
630
+ // Handle the inlined entry based on direction
631
+ match direction {
632
+ Direction::Forward => {
633
+ if !predicate(self.inlined_offset)? {
634
+ return Ok(Some(self.head()));
635
+ }
636
+ }
637
+ Direction::Backward => {
638
+ if predicate(self.inlined_offset)? {
639
+ // If predicate is true for inlined entry and we're going backward,
640
+ // this is potentially our best match
641
+ best_match = Some(self.head());
642
+ }
643
+ }
644
+ }
645
+
646
+ // If we have an array cursor, check it too using binary search
647
+ if let Some(cursor) = self.cursor {
648
+ let ic = cursor
649
+ .list
650
+ .directed_partition_point(journal_file, predicate, direction)?;
651
+
652
+ if let Some(ic) = ic {
653
+ // Create a new InlinedCursor with this array cursor
654
+ let array_match = Self {
655
+ inlined_offset: self.inlined_offset,
656
+ cursor: Some(ic),
657
+ at_inlined_offset: false,
658
+ };
659
+
660
+ // Compare with our current best match
661
+ if best_match.is_none() {
662
+ best_match = Some(array_match);
663
+ } else {
664
+ // Choose the better match based on direction
665
+ let best_offset = best_match.as_ref().unwrap().value(journal_file)?;
666
+ let array_offset = array_match.value(journal_file)?;
667
+
668
+ match direction {
669
+ Direction::Forward => {
670
+ if array_offset < best_offset {
671
+ best_match = Some(array_match);
672
+ }
673
+ }
674
+ Direction::Backward => {
675
+ if array_offset > best_offset {
676
+ best_match = Some(array_match);
677
+ }
678
+ }
679
+ }
680
+ }
681
+ }
682
+ }
683
+
684
+ Ok(best_match)
685
+ }
686
+}
src/crates/jf/journal_file/src/reader.rs
new
+194
@@ -0,0 +1,194 @@
1
+use crate::{
2
+ cursor::{JournalCursor, Location},
3
+ file::{EntryDataIterator, FieldDataIterator, FieldIterator, JournalFile},
4
+ filter::{JournalFilter, LogicalOp},
5
+ object::{DataObject, FieldObject},
6
+ offset_array::Direction,
7
+ value_guard::ValueGuard,
8
+};
9
+use error::Result;
10
+use std::num::NonZeroU64;
11
+use window_manager::MemoryMap;
12
+
13
+pub struct JournalReader<'a, M: MemoryMap> {
14
+ cursor: JournalCursor,
15
+
16
+ filter: Option<JournalFilter>,
17
+ field_iterator: Option<FieldIterator<'a, M>>,
18
+ field_data_iterator: Option<FieldDataIterator<'a, M>>,
19
+ entry_data_iterator: Option<EntryDataIterator<'a, M>>,
20
+
21
+ field_guard: Option<ValueGuard<'a, FieldObject<&'a [u8]>>>,
22
+ data_guard: Option<ValueGuard<'a, DataObject<&'a [u8]>>>,
23
+}
24
+
25
+impl<M: MemoryMap> std::fmt::Debug for JournalReader<'_, M> {
26
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
27
+ f.debug_struct("JournalReader")
28
+ // .field("cursor", &self.cursor)
29
+ .field("field_guard", &self.field_guard)
30
+ .field("data_guard", &self.data_guard)
31
+ .finish()
32
+ }
33
+}
34
+
35
+impl<M: MemoryMap> Default for JournalReader<'_, M> {
36
+ fn default() -> Self {
37
+ Self {
38
+ cursor: JournalCursor::new(),
39
+ filter: None,
40
+ field_iterator: None,
41
+ field_data_iterator: None,
42
+ entry_data_iterator: None,
43
+ field_guard: None,
44
+ data_guard: None,
45
+ }
46
+ }
47
+}
48
+
49
+impl<'a, M: MemoryMap> JournalReader<'a, M> {
50
+ pub fn dump(&self, journal_file: &'a JournalFile<M>) -> Result<String> {
51
+ if let Some(filter_expr) = self.cursor.filter_expr.as_ref() {
52
+ filter_expr.dump(journal_file)
53
+ } else {
54
+ Ok(String::from("no filter expr"))
55
+ }
56
+ }
57
+
58
+ pub fn set_location(&mut self, location: Location) {
59
+ self.cursor.set_location(location)
60
+ }
61
+
62
+ pub fn step(&mut self, journal_file: &'a JournalFile<M>, direction: Direction) -> Result<bool> {
63
+ self.drop_guards();
64
+
65
+ if let Some(filter) = self.filter.as_mut() {
66
+ let filter_expr = filter.build(journal_file)?;
67
+ self.cursor.set_filter(filter_expr);
68
+ self.filter = None;
69
+ }
70
+
71
+ self.cursor.step(journal_file, direction)
72
+ }
73
+
74
+ pub fn add_match(&mut self, data: &[u8]) {
75
+ self.filter.get_or_insert_default().add_match(data);
76
+ }
77
+
78
+ pub fn add_conjunction(&mut self, journal_file: &'a JournalFile<M>) -> Result<()> {
79
+ self.filter
80
+ .get_or_insert_default()
81
+ .set_operation(journal_file, LogicalOp::Conjunction)
82
+ }
83
+
84
+ pub fn add_disjunction(&mut self, journal_file: &'a JournalFile<M>) -> Result<()> {
85
+ self.filter
86
+ .get_or_insert_default()
87
+ .set_operation(journal_file, LogicalOp::Disjunction)
88
+ }
89
+
90
+ pub fn flush_matches(&mut self) {
91
+ self.cursor.clear_filter();
92
+ self.filter = None;
93
+ }
94
+
95
+ pub fn get_realtime_usec(&self, journal_file: &'a JournalFile<M>) -> Result<u64> {
96
+ let entry_offset = self.cursor.position()?;
97
+ let entry_object = journal_file.entry_ref(entry_offset)?;
98
+ Ok(entry_object.header.realtime)
99
+ }
100
+
101
+ pub fn get_seqnum(&self, journal_file: &'a JournalFile<M>) -> Result<(u64, [u8; 16])> {
102
+ let entry_offset = self.cursor.position()?;
103
+ let entry_object = journal_file.entry_ref(entry_offset)?;
104
+ Ok((
105
+ entry_object.header.seqnum,
106
+ journal_file.journal_header_ref().seqnum_id,
107
+ ))
108
+ }
109
+
110
+ pub fn get_entry_offset(&self) -> Result<NonZeroU64> {
111
+ self.cursor.position()
112
+ }
113
+
114
+ fn drop_guards(&mut self) {
115
+ self.field_guard.take();
116
+ self.data_guard.take();
117
+ }
118
+
119
+ pub fn fields_restart(&mut self) {
120
+ self.drop_guards();
121
+ self.field_iterator = None;
122
+ }
123
+
124
+ pub fn fields_enumerate(
125
+ &mut self,
126
+ journal_file: &'a JournalFile<M>,
127
+ ) -> Result<Option<&ValueGuard<'_, FieldObject<&'a [u8]>>>> {
128
+ self.drop_guards();
129
+
130
+ if self.field_iterator.is_none() {
131
+ self.field_iterator = Some(journal_file.fields());
132
+ }
133
+
134
+ if let Some(iter) = &mut self.field_iterator {
135
+ self.field_guard = iter.next().transpose()?;
136
+ Ok(self.field_guard.as_ref())
137
+ } else {
138
+ Ok(None)
139
+ }
140
+ }
141
+
142
+ pub fn field_data_query_unique(
143
+ &mut self,
144
+ journal_file: &'a JournalFile<M>,
145
+ field_name: &'a [u8],
146
+ ) -> Result<()> {
147
+ self.drop_guards();
148
+
149
+ self.field_data_iterator = Some(journal_file.field_data_objects(field_name)?);
150
+ Ok(())
151
+ }
152
+
153
+ pub fn field_data_restart(&mut self) {
154
+ self.drop_guards();
155
+ }
156
+
157
+ pub fn field_data_enumerate(
158
+ &mut self,
159
+ _: &'a JournalFile<M>,
160
+ ) -> Result<Option<&ValueGuard<'_, DataObject<&'a [u8]>>>> {
161
+ self.drop_guards();
162
+
163
+ if let Some(iter) = &mut self.field_data_iterator {
164
+ self.data_guard = iter.next().transpose()?;
165
+ Ok(self.data_guard.as_ref())
166
+ } else {
167
+ Ok(None)
168
+ }
169
+ }
170
+
171
+ pub fn entry_data_restart(&mut self) {
172
+ self.drop_guards();
173
+ self.entry_data_iterator = None;
174
+ }
175
+
176
+ pub fn entry_data_enumerate(
177
+ &mut self,
178
+ journal_file: &'a JournalFile<M>,
179
+ ) -> Result<Option<&ValueGuard<'_, DataObject<&'a [u8]>>>> {
180
+ self.drop_guards();
181
+
182
+ if self.entry_data_iterator.is_none() {
183
+ let entry_offset = self.cursor.position()?;
184
+ self.entry_data_iterator = Some(journal_file.entry_data_objects(entry_offset)?);
185
+ }
186
+
187
+ if let Some(iter) = &mut self.entry_data_iterator {
188
+ self.data_guard = iter.next().transpose()?;
189
+ Ok(self.data_guard.as_ref())
190
+ } else {
191
+ Ok(None)
192
+ }
193
+ }
194
+}
src/crates/jf/journal_file/src/value_guard.rs
new
+99
@@ -0,0 +1,99 @@
1
+use std::cell::RefCell;
2
+use std::ops::{Deref, DerefMut};
3
+
4
+/// A guard that ensures exclusive access to objects obtained from a shared memory window.
5
+///
6
+/// # Purpose
7
+///
8
+/// `ValueGuard` enforces that only one journal object can be accessed at a time.
9
+/// This is necessary because:
10
+///
11
+/// 1. The underlying window manager uses a limited number of memory-mapped windows
12
+/// that are shared between all objects.
13
+/// 2. Creating a new object could invalidate memory references held by previously
14
+/// created objects, even though the objects themselves are immutable.
15
+///
16
+/// # Usage
17
+///
18
+/// When a `ValueGuard<T>` is returned from methods like `data_object()`, it provides
19
+/// read-only access to the underlying object via the `Deref` trait. When the guard
20
+/// is dropped (goes out of scope), it automatically releases the lock, allowing new
21
+/// objects to be created.
22
+///
23
+/// # Safety and Interior Mutability
24
+///
25
+/// Although the returned objects are immutable, the window manager that provides
26
+/// the memory they reference uses interior mutability (via `UnsafeCell`) to reuse
27
+/// memory-mapped regions. This guard ensures that objects are not accessed after
28
+/// their underlying memory might have been repurposed.
29
+#[derive(Debug)]
30
+pub struct ValueGuard<'a, T> {
31
+ offset: NonZeroU64,
32
+ value: T,
33
+ in_use_flag: &'a RefCell<bool>,
34
+}
35
+
36
+impl<'a, T> ValueGuard<'a, T> {
37
+ pub fn new(offset: NonZeroU64, value: T, in_use_flag: &'a RefCell<bool>) -> Self {
38
+ Self {
39
+ offset,
40
+ value,
41
+ in_use_flag,
42
+ }
43
+ }
44
+
45
+ pub fn offset(&self) -> NonZeroU64 {
46
+ self.offset
47
+ }
48
+}
49
+
50
+impl<T> Deref for ValueGuard<'_, T> {
51
+ type Target = T;
52
+
53
+ fn deref(&self) -> &Self::Target {
54
+ &self.value
55
+ }
56
+}
57
+
58
+impl<T> DerefMut for ValueGuard<'_, T> {
59
+ fn deref_mut(&mut self) -> &mut Self::Target {
60
+ &mut self.value
61
+ }
62
+}
63
+
64
+impl<T> Drop for ValueGuard<'_, T> {
65
+ fn drop(&mut self) {
66
+ *self.in_use_flag.borrow_mut() = false;
67
+ }
68
+}
69
+
70
+use crate::{HashableObject, HashableObjectMut};
71
+use std::num::NonZeroU64;
72
+
73
+impl<T: HashableObject> HashableObject for ValueGuard<'_, T> {
74
+ fn hash(&self) -> u64 {
75
+ self.value.hash()
76
+ }
77
+
78
+ fn get_payload(&self) -> &[u8] {
79
+ self.value.get_payload()
80
+ }
81
+
82
+ fn next_hash_offset(&self) -> Option<NonZeroU64> {
83
+ self.value.next_hash_offset()
84
+ }
85
+
86
+ fn object_type() -> crate::ObjectType {
87
+ T::object_type()
88
+ }
89
+}
90
+
91
+impl<T: HashableObjectMut> HashableObjectMut for ValueGuard<'_, T> {
92
+ fn set_next_hash_offset(&mut self, offset: NonZeroU64) {
93
+ self.value.set_next_hash_offset(offset);
94
+ }
95
+
96
+ fn set_payload(&mut self, data: &[u8]) {
97
+ self.value.set_payload(data);
98
+ }
99
+}
src/crates/jf/journal_file/src/writer.rs
new
+700
@@ -0,0 +1,700 @@
1
+#![allow(unused_imports, dead_code)]
2
+
3
+use crate::{
4
+ journal_hash_data, CompactEntryItem, DataHashTable, DataObject, DataObjectHeader,
5
+ DataPayloadType, EntryObject, EntryObjectHeader, FieldHashTable, FieldObject,
6
+ FieldObjectHeader, HashItem, HashTable, HashTableMut, HashableObject, HashableObjectMut,
7
+ HeaderIncompatibleFlags, JournalFile, JournalFileOptions, JournalHeader, JournalState,
8
+ ObjectHeader, ObjectType, RegularEntryItem,
9
+};
10
+use error::{JournalError, Result};
11
+use memmap2::MmapMut;
12
+use rand::{seq::IndexedRandom, Rng};
13
+use std::num::{NonZeroU64, NonZeroUsize};
14
+use std::path::Path;
15
+use window_manager::MemoryMapMut;
16
+use zerocopy::{FromBytes, IntoBytes};
17
+
18
+const OBJECT_ALIGNMENT: u64 = 8;
19
+
20
+#[derive(Debug, Clone, Copy)]
21
+struct EntryItem {
22
+ offset: NonZeroU64,
23
+ hash: u64,
24
+}
25
+
26
+pub struct JournalWriter {
27
+ tail_object_offset: NonZeroU64,
28
+ append_offset: NonZeroU64,
29
+ next_seqnum: u64,
30
+ num_written_objects: u64,
31
+ entry_items: Vec<EntryItem>,
32
+ first_entry_monotonic: Option<u64>,
33
+}
34
+
35
+impl JournalWriter {
36
+ /// Get current file size in bytes
37
+ pub fn current_file_size(&self) -> u64 {
38
+ self.append_offset.get()
39
+ }
40
+
41
+ /// Get the monotonic timestamp of the first entry written to this file
42
+ pub fn first_entry_monotonic(&self) -> Option<u64> {
43
+ self.first_entry_monotonic
44
+ }
45
+
46
+ pub fn new(journal_file: &mut JournalFile<MmapMut>) -> Result<Self> {
47
+ let (append_offset, next_seqnum) = {
48
+ let header = journal_file.journal_header_ref();
49
+
50
+ let Some(tail_object_offset) = header.tail_object_offset else {
51
+ return Err(JournalError::InvalidMagicNumber);
52
+ };
53
+
54
+ let tail_object = journal_file.object_header_ref(tail_object_offset)?;
55
+
56
+ (
57
+ tail_object_offset.saturating_add(tail_object.size),
58
+ header.tail_entry_seqnum + 1,
59
+ )
60
+ };
61
+
62
+ Ok(Self {
63
+ tail_object_offset: journal_file
64
+ .journal_header_ref()
65
+ .tail_object_offset
66
+ .unwrap(),
67
+ append_offset,
68
+ next_seqnum,
69
+ num_written_objects: 0,
70
+ entry_items: Vec::with_capacity(128),
71
+ first_entry_monotonic: None,
72
+ })
73
+ }
74
+
75
+ pub fn add_entry(
76
+ &mut self,
77
+ journal_file: &mut JournalFile<MmapMut>,
78
+ items: &[&[u8]],
79
+ realtime: u64,
80
+ monotonic: u64,
81
+ boot_id: [u8; 16],
82
+ ) -> Result<()> {
83
+ let header = journal_file.journal_header_ref();
84
+ assert!(header.has_incompatible_flag(HeaderIncompatibleFlags::KeyedHash));
85
+
86
+ // Write the data/field objects while computing the entry's xor-hash
87
+ // and storing each data object's offset/hash
88
+ let mut xor_hash = 0;
89
+ {
90
+ self.entry_items.clear();
91
+ for payload in items {
92
+ let offset = self.add_data(journal_file, payload)?;
93
+ let hash = {
94
+ let data_guard = journal_file.data_ref(offset)?;
95
+ data_guard.hash()
96
+ };
97
+
98
+ let entry_item = EntryItem { offset, hash };
99
+ self.entry_items.push(entry_item);
100
+
101
+ xor_hash ^= journal_hash_data(payload, true, None);
102
+ }
103
+
104
+ self.entry_items
105
+ .sort_unstable_by(|a, b| a.offset.cmp(&b.offset));
106
+ self.entry_items.dedup_by(|a, b| a.offset == b.offset);
107
+ }
108
+
109
+ // write the entry itself
110
+ let entry_offset = self.append_offset;
111
+ let entry_size = {
112
+ let size = Some(self.entry_items.len() as u64 * 16);
113
+ let mut entry_guard = journal_file.entry_mut(entry_offset, size)?;
114
+
115
+ entry_guard.header.seqnum = self.next_seqnum;
116
+ entry_guard.header.xor_hash = xor_hash;
117
+ entry_guard.header.boot_id = boot_id;
118
+ entry_guard.header.monotonic = monotonic;
119
+ entry_guard.header.realtime = realtime;
120
+
121
+ // set each entry item
122
+ for (index, entry_item) in self.entry_items.iter().enumerate() {
123
+ entry_guard
124
+ .items
125
+ .set(index, entry_item.offset, Some(entry_item.hash));
126
+ }
127
+
128
+ entry_guard.header.object_header.aligned_size()
129
+ };
130
+ self.object_added(entry_offset, entry_size);
131
+
132
+ self.append_to_entry_array(journal_file, entry_offset)?;
133
+ for entry_item_index in 0..self.entry_items.len() {
134
+ self.link_data_to_entry(journal_file, entry_offset, entry_item_index)?;
135
+ }
136
+
137
+ self.entry_added(
138
+ journal_file.journal_header_mut(),
139
+ realtime,
140
+ monotonic,
141
+ boot_id,
142
+ );
143
+
144
+ Ok(())
145
+ }
146
+
147
+ fn object_added(&mut self, object_offset: NonZeroU64, object_size: u64) {
148
+ self.tail_object_offset = object_offset;
149
+ self.append_offset = object_offset.saturating_add(object_size);
150
+ self.num_written_objects += 1;
151
+ }
152
+
153
+ fn entry_added(
154
+ &mut self,
155
+ header: &mut JournalHeader,
156
+ realtime: u64,
157
+ monotonic: u64,
158
+ boot_id: [u8; 16],
159
+ ) {
160
+ header.n_entries += 1;
161
+ header.n_objects += self.num_written_objects;
162
+ header.tail_object_offset = Some(self.tail_object_offset);
163
+ header.arena_size = self.append_offset.get() - header.header_size;
164
+
165
+ if header.head_entry_seqnum == 0 {
166
+ header.head_entry_seqnum = self.next_seqnum;
167
+ }
168
+ if header.head_entry_realtime == 0 {
169
+ header.head_entry_realtime = realtime;
170
+ }
171
+ if self.first_entry_monotonic.is_none() {
172
+ self.first_entry_monotonic = Some(monotonic);
173
+ }
174
+
175
+ header.tail_entry_seqnum = self.next_seqnum;
176
+ header.tail_entry_realtime = realtime;
177
+ header.tail_entry_monotonic = monotonic;
178
+ header.tail_entry_boot_id = boot_id;
179
+
180
+ self.next_seqnum += 1;
181
+ self.num_written_objects = 0;
182
+ }
183
+
184
+ fn add_data(
185
+ &mut self,
186
+ journal_file: &mut JournalFile<MmapMut>,
187
+ payload: &[u8],
188
+ ) -> Result<NonZeroU64> {
189
+ let hash = journal_file.hash(payload);
190
+
191
+ match journal_file.find_data_offset(hash, payload)? {
192
+ Some(data_offset) => Ok(data_offset),
193
+ None => {
194
+ // We will have to write the new data object at the current
195
+ // tail offset
196
+ let data_offset = self.append_offset;
197
+ let data_size = {
198
+ let mut data_guard =
199
+ journal_file.data_mut(data_offset, Some(payload.len() as u64))?;
200
+
201
+ data_guard.header.hash = hash;
202
+ data_guard.set_payload(payload);
203
+ data_guard.header.object_header.aligned_size()
204
+ };
205
+
206
+ self.object_added(data_offset, data_size);
207
+
208
+ // Update hash table
209
+ journal_file.data_hash_table_set_tail_offset(hash, data_offset)?;
210
+
211
+ // Add the field object, if we have any
212
+ if let Some(equals_pos) = payload.iter().position(|&b| b == b'=') {
213
+ let field_offset = self.add_field(journal_file, &payload[..equals_pos])?;
214
+
215
+ // Link data object to the linked-list
216
+ {
217
+ let head_data_offset = {
218
+ let field_guard = journal_file.field_ref(field_offset)?;
219
+ field_guard.header.head_data_offset
220
+ };
221
+
222
+ let mut data_guard = journal_file.data_mut(data_offset, None)?;
223
+ data_guard.header.next_field_offset = head_data_offset;
224
+ }
225
+
226
+ // Link field to the head of the linked list
227
+ {
228
+ let mut field_guard = journal_file.field_mut(field_offset, None)?;
229
+ field_guard.header.head_data_offset = Some(data_offset);
230
+ };
231
+ }
232
+
233
+ Ok(data_offset)
234
+ }
235
+ }
236
+ }
237
+
238
+ fn add_field(
239
+ &mut self,
240
+ journal_file: &mut JournalFile<MmapMut>,
241
+ payload: &[u8],
242
+ ) -> Result<NonZeroU64> {
243
+ let hash = journal_file.hash(payload);
244
+
245
+ match journal_file.find_field_offset(hash, payload)? {
246
+ Some(field_offset) => Ok(field_offset),
247
+ None => {
248
+ // We will have to write the new field object at the current
249
+ // tail offset
250
+ let field_offset = self.append_offset;
251
+ let field_size = {
252
+ let mut field_guard =
253
+ journal_file.field_mut(field_offset, Some(payload.len() as u64))?;
254
+
255
+ field_guard.header.hash = hash;
256
+ field_guard.set_payload(payload);
257
+ field_guard.header.object_header.aligned_size()
258
+ };
259
+ self.object_added(field_offset, field_size);
260
+
261
+ // Update hash table
262
+ journal_file.field_hash_table_set_tail_offset(hash, field_offset)?;
263
+
264
+ // Return the offset where we wrote the newly added data object
265
+ Ok(field_offset)
266
+ }
267
+ }
268
+ }
269
+
270
+ fn allocate_new_array(
271
+ &mut self,
272
+ journal_file: &JournalFile<MmapMut>,
273
+ capacity: NonZeroU64,
274
+ ) -> Result<NonZeroU64> {
275
+ // let new_capacity = previous_capacity.saturating_mul(NonZeroU64::new(2).unwrap());
276
+
277
+ let array_offset = self.append_offset;
278
+ let array_size = {
279
+ let array_guard = journal_file.offset_array_mut(array_offset, Some(capacity))?;
280
+
281
+ array_guard.header.object_header.aligned_size()
282
+ };
283
+ self.object_added(array_offset, array_size);
284
+
285
+ Ok(array_offset)
286
+ }
287
+
288
+ fn append_to_entry_array(
289
+ &mut self,
290
+ journal_file: &mut JournalFile<MmapMut>,
291
+ entry_offset: NonZeroU64,
292
+ ) -> Result<()> {
293
+ let entry_array_offset = journal_file.journal_header_ref().entry_array_offset;
294
+
295
+ if entry_array_offset.is_none() {
296
+ journal_file.journal_header_mut().entry_array_offset = {
297
+ let array_offset =
298
+ self.allocate_new_array(journal_file, NonZeroU64::new(4096).unwrap())?;
299
+ let mut array_guard = journal_file.offset_array_mut(array_offset, None)?;
300
+ array_guard.set(0, entry_offset)?;
301
+ Some(array_offset)
302
+ };
303
+ } else {
304
+ let tail_node = {
305
+ let entry_list = journal_file
306
+ .entry_list()
307
+ .ok_or(JournalError::EmptyOffsetArrayList)?;
308
+ entry_list.tail(journal_file)?
309
+ };
310
+
311
+ if tail_node.len() < tail_node.capacity() {
312
+ let mut array_guard = journal_file.offset_array_mut(tail_node.offset(), None)?;
313
+ array_guard.set(tail_node.len().get(), entry_offset)?;
314
+ } else {
315
+ let new_array_offset = {
316
+ let new_capacity = tail_node.capacity().get().saturating_mul(2) as u64;
317
+ let new_array_offset = self
318
+ .allocate_new_array(journal_file, NonZeroU64::new(new_capacity).unwrap())?;
319
+ let mut array_guard = journal_file.offset_array_mut(new_array_offset, None)?;
320
+ array_guard.set(0, entry_offset)?;
321
+
322
+ new_array_offset
323
+ };
324
+
325
+ // Link the old tail to the new array
326
+ {
327
+ let mut array_guard =
328
+ journal_file.offset_array_mut(tail_node.offset(), None)?;
329
+ array_guard.header.next_offset_array = Some(new_array_offset);
330
+ }
331
+ }
332
+ }
333
+
334
+ Ok(())
335
+ }
336
+
337
+ fn append_to_data_entry_array(
338
+ &mut self,
339
+ journal_file: &mut JournalFile<MmapMut>,
340
+ mut array_offset: NonZeroU64,
341
+ entry_offset: NonZeroU64,
342
+ current_count: u64,
343
+ ) -> Result<()> {
344
+ // Navigate to the tail of the array chain
345
+ let mut current_index = 0u64;
346
+ #[allow(unused_assignments)]
347
+ let mut tail_offset = array_offset;
348
+
349
+ loop {
350
+ let array_guard = journal_file.offset_array_ref(array_offset)?;
351
+ let capacity = array_guard.capacity() as u64;
352
+
353
+ if current_index + capacity >= current_count {
354
+ // This is the tail array
355
+ tail_offset = array_offset;
356
+ break;
357
+ }
358
+
359
+ current_index += capacity;
360
+
361
+ let Some(next_offset) = array_guard.header.next_offset_array else {
362
+ // This shouldn't happen if counts are correct
363
+ return Err(JournalError::InvalidOffsetArrayOffset);
364
+ };
365
+
366
+ array_offset = next_offset;
367
+ }
368
+
369
+ // Try to add to the tail array
370
+ let tail_capacity = {
371
+ let tail_guard = journal_file.offset_array_ref(tail_offset)?;
372
+ tail_guard.capacity() as u64
373
+ };
374
+
375
+ let entries_in_tail = current_count - current_index;
376
+
377
+ if entries_in_tail < tail_capacity {
378
+ // There's space in the tail array
379
+ let mut tail_guard = journal_file.offset_array_mut(tail_offset, None)?;
380
+ tail_guard.set(entries_in_tail as usize, entry_offset)?;
381
+ } else {
382
+ // Need to create a new array
383
+ let new_capacity = NonZeroU64::new(tail_capacity * 2).unwrap(); // Double the size
384
+ let new_array_offset = self.allocate_new_array(journal_file, new_capacity)?;
385
+
386
+ // Link the old tail to the new array
387
+ let mut tail_guard = journal_file.offset_array_mut(tail_offset, None)?;
388
+ tail_guard.header.next_offset_array = Some(new_array_offset);
389
+ drop(tail_guard);
390
+
391
+ // Add entry to the new array
392
+ let mut new_array_guard = journal_file.offset_array_mut(new_array_offset, None)?;
393
+ new_array_guard.set(0, entry_offset)?;
394
+ }
395
+
396
+ Ok(())
397
+ }
398
+
399
+ fn link_data_to_entry(
400
+ &mut self,
401
+ journal_file: &mut JournalFile<MmapMut>,
402
+ entry_offset: NonZeroU64,
403
+ entry_item_index: usize,
404
+ ) -> Result<()> {
405
+ let data_offset = self.entry_items[entry_item_index].offset;
406
+ let mut data_guard = journal_file.data_mut(data_offset, None)?;
407
+
408
+ match data_guard.header.n_entries {
409
+ None => {
410
+ data_guard.header.entry_offset = Some(entry_offset);
411
+ data_guard.header.n_entries = NonZeroU64::new(1);
412
+ }
413
+ Some(n_entries) => {
414
+ match n_entries.get() {
415
+ 0 => {
416
+ unreachable!();
417
+ }
418
+ 1 => {
419
+ drop(data_guard);
420
+
421
+ // Create new entry array with initial capacity
422
+ let array_capacity = NonZeroU64::new(64).unwrap();
423
+ let array_offset = self.allocate_new_array(journal_file, array_capacity)?;
424
+
425
+ // Load new array and set its first entry offset
426
+ {
427
+ let mut array_guard =
428
+ journal_file.offset_array_mut(array_offset, None)?;
429
+ array_guard.set(0, entry_offset)?;
430
+ }
431
+
432
+ // Update data object to point to the array
433
+ let mut data_guard = journal_file.data_mut(data_offset, None)?;
434
+ data_guard.header.entry_array_offset = Some(array_offset);
435
+ data_guard.header.n_entries = NonZeroU64::new(2);
436
+ }
437
+ x => {
438
+ // There's already an entry array, append to it
439
+ let current_count = x - 1;
440
+ let array_offset = data_guard.header.entry_array_offset.unwrap();
441
+
442
+ // Drop the data guard to avoid borrow conflicts
443
+ drop(data_guard);
444
+
445
+ // Find the tail of the entry array chain and append
446
+ self.append_to_data_entry_array(
447
+ journal_file,
448
+ array_offset,
449
+ entry_offset,
450
+ current_count,
451
+ )?;
452
+
453
+ // Update the count
454
+ let mut data_guard = journal_file.data_mut(data_offset, None)?;
455
+ data_guard.header.n_entries = NonZeroU64::new(x + 1);
456
+ }
457
+ }
458
+ }
459
+ }
460
+
461
+ Ok(())
462
+ }
463
+}
464
+
465
+#[cfg(test)]
466
+mod tests {
467
+ use super::*;
468
+ use crate::{load_boot_id, Direction, JournalFile, JournalReader, Location};
469
+ use memmap2::Mmap;
470
+ use std::collections::HashMap;
471
+ use tempfile::NamedTempFile;
472
+
473
+ fn generate_uuid() -> [u8; 16] {
474
+ use rand::Rng;
475
+ let mut rng = rand::rng();
476
+ rng.random()
477
+ }
478
+
479
+ #[test]
480
+ fn test_write_and_read_journal_entries() -> Result<()> {
481
+ // Create test data - a hash map with key/values to add to the journal file
482
+ let mut test_data = HashMap::new();
483
+ test_data.insert(
484
+ "MESSAGE",
485
+ vec!["Hello, world!", "Another message", "Final message"],
486
+ );
487
+ test_data.insert("PRIORITY", vec!["6", "4", "3"]);
488
+ test_data.insert(
489
+ "_SYSTEMD_UNIT",
490
+ vec!["test.service", "other.service", "test.service"],
491
+ );
492
+ test_data.insert("_PID", vec!["1234", "5678", "9999"]);
493
+
494
+ // Create a temporary file for the journal
495
+ let temp_file = NamedTempFile::new().map_err(JournalError::Io).unwrap();
496
+ let journal_path = temp_file.path();
497
+
498
+ // Step 1: Create and write to the journal
499
+ let boot_id = load_boot_id().unwrap_or([1; 16]); // Use real boot_id or fallback
500
+ let num_entries = test_data.values().next().unwrap().len();
501
+
502
+ let options = JournalFileOptions::new(
503
+ generate_uuid(),
504
+ generate_uuid(),
505
+ generate_uuid(),
506
+ generate_uuid(),
507
+ );
508
+ let mut journal_file = JournalFile::create(journal_path, options)?;
509
+ let iterations = 5000;
510
+ for _ in 0..iterations {
511
+ let mut writer = JournalWriter::new(&mut journal_file)?;
512
+
513
+ // Write entries to the journal
514
+ for i in 0..num_entries {
515
+ let mut entry_data = Vec::new();
516
+
517
+ // Build the entry data for this index
518
+ for (key, values) in &test_data {
519
+ let kv_pair = format!("{}={}", key, values[i]);
520
+ entry_data.push(kv_pair.into_bytes());
521
+ }
522
+
523
+ // Convert to slice references for the writer
524
+ let entry_refs: Vec<&[u8]> = entry_data.iter().map(|v| v.as_slice()).collect();
525
+
526
+ // Write the entry with timestamps
527
+ let realtime = 1000000 + (i as u64 * 1000); // Mock realtime in microseconds
528
+ let monotonic = 500000 + (i as u64 * 1000); // Mock monotonic time
529
+
530
+ writer.add_entry(&mut journal_file, &entry_refs, realtime, monotonic, boot_id)?;
531
+ }
532
+ }
533
+
534
+ // Step 2: Read back and verify the journal contents
535
+ {
536
+ let journal_file = JournalFile::<Mmap>::open(journal_path, 8 * 1024)?;
537
+ let mut reader = JournalReader::default();
538
+
539
+ let hdr = journal_file.journal_header_ref();
540
+ println!("Header: {:#?}", hdr);
541
+
542
+ // Start from the head
543
+ reader.set_location(Location::Head);
544
+
545
+ let mut entries_read = 0;
546
+ while reader.step(&journal_file, Direction::Forward)? {
547
+ println!("Reading entry {}", entries_read);
548
+
549
+ // Verify timestamps
550
+ let realtime = reader.get_realtime_usec(&journal_file)?;
551
+ let expected_realtime = 1000000 + ((entries_read % 3) * 1000);
552
+ assert_eq!(
553
+ realtime, expected_realtime,
554
+ "Realtime mismatch for entry {}",
555
+ entries_read
556
+ );
557
+
558
+ let (seqnum, _seqnum_id) = reader.get_seqnum(&journal_file)?;
559
+ assert_eq!(
560
+ seqnum,
561
+ entries_read + 1,
562
+ "Sequence number mismatch for entry {}",
563
+ entries_read
564
+ );
565
+
566
+ // Read all data for this entry
567
+ let mut entry_fields = HashMap::new();
568
+ reader.entry_data_restart();
569
+
570
+ while let Some(data_guard) = reader.entry_data_enumerate(&journal_file)? {
571
+ let payload = data_guard.payload_bytes();
572
+ let payload_str = String::from_utf8_lossy(payload);
573
+
574
+ if let Some(eq_pos) = payload_str.find('=') {
575
+ let key = &payload_str[..eq_pos];
576
+ let value = &payload_str[eq_pos + 1..];
577
+ entry_fields.insert(key.to_string(), value.to_string());
578
+ }
579
+ }
580
+
581
+ // Verify the data matches what we wrote
582
+ for (key, values) in &test_data {
583
+ let expected_value = &values[entries_read as usize % 3];
584
+ let actual_value = entry_fields.get(*key).unwrap_or_else(|| {
585
+ panic!("Missing key '{}' in entry {}", key, entries_read)
586
+ });
587
+
588
+ assert_eq!(
589
+ actual_value, expected_value,
590
+ "Value mismatch for key '{}' in entry {}",
591
+ key, entries_read
592
+ );
593
+ }
594
+
595
+ println!("Read entry {}", entries_read);
596
+ entries_read += 1;
597
+ }
598
+
599
+ assert_eq!(
600
+ entries_read as usize,
601
+ num_entries * iterations,
602
+ "Number of entries read doesn't match written"
603
+ );
604
+ }
605
+
606
+ // Step 3: Test filtering by specific fields
607
+ {
608
+ let journal_file = JournalFile::<Mmap>::open(journal_path, 64 * 1024)?;
609
+ let mut reader = JournalReader::default();
610
+
611
+ // Test filtering by _SYSTEMD_UNIT=test.service
612
+ reader.add_match(b"_SYSTEMD_UNIT=test.service");
613
+ reader.set_location(Location::Head);
614
+
615
+ let mut filtered_entries = 0;
616
+ while reader.step(&journal_file, Direction::Forward)? {
617
+ // Verify this entry actually contains the filter match
618
+ reader.entry_data_restart();
619
+ let mut found_match = false;
620
+
621
+ while let Some(data_guard) = reader.entry_data_enumerate(&journal_file)? {
622
+ let payload = data_guard.payload_bytes();
623
+ if payload == b"_SYSTEMD_UNIT=test.service" {
624
+ found_match = true;
625
+ break;
626
+ }
627
+ }
628
+
629
+ assert!(
630
+ found_match,
631
+ "Filtered entry doesn't contain the expected field"
632
+ );
633
+ filtered_entries += 1;
634
+ }
635
+
636
+ // Should find 2 entries with _SYSTEMD_UNIT=test.service (entries 0 and 2)
637
+ assert_eq!(filtered_entries, 2, "Expected 2 filtered entries");
638
+ }
639
+
640
+ println!("✅ All tests passed!");
641
+ Ok(())
642
+ }
643
+
644
+ #[test]
645
+ fn test_field_enumeration() -> Result<()> {
646
+ // Create a simple journal with known fields
647
+ let temp_file = NamedTempFile::new().map_err(JournalError::Io)?;
648
+ let journal_path = temp_file.path();
649
+
650
+ let test_fields = vec!["MESSAGE", "PRIORITY", "_SYSTEMD_UNIT"];
651
+ let boot_id = [1; 16];
652
+
653
+ // Write a single entry with multiple fields
654
+ {
655
+ let options = JournalFileOptions::new(
656
+ generate_uuid(),
657
+ generate_uuid(),
658
+ generate_uuid(),
659
+ generate_uuid(),
660
+ );
661
+
662
+ let mut journal_file = JournalFile::create(journal_path, options)?;
663
+ let mut writer = JournalWriter::new(&mut journal_file)?;
664
+
665
+ let entry_data = vec![
666
+ b"MESSAGE=Test message".as_slice(),
667
+ b"PRIORITY=6".as_slice(),
668
+ b"_SYSTEMD_UNIT=test.service".as_slice(),
669
+ ];
670
+
671
+ writer.add_entry(&mut journal_file, &entry_data, 1000000, 500000, boot_id)?;
672
+ }
673
+
674
+ // Read back and enumerate fields
675
+ {
676
+ let journal_file = JournalFile::<Mmap>::open(journal_path, 8 * 1024)?;
677
+ let mut reader = JournalReader::default();
678
+
679
+ let mut found_fields = Vec::new();
680
+ reader.fields_restart();
681
+
682
+ while let Some(field_guard) = reader.fields_enumerate(&journal_file)? {
683
+ let field_name = String::from_utf8_lossy(field_guard.payload);
684
+ found_fields.push(field_name.to_string());
685
+ }
686
+
687
+ // Verify all expected fields were found
688
+ for expected_field in &test_fields {
689
+ assert!(
690
+ found_fields.contains(&expected_field.to_string()),
691
+ "Expected field '{}' not found. Found: {:?}",
692
+ expected_field,
693
+ found_fields
694
+ );
695
+ }
696
+ }
697
+
698
+ Ok(())
699
+ }
700
+}
src/crates/jf/journal_log/Cargo.toml
new
+14
@@ -0,0 +1,14 @@
1
+[package]
2
+name = "journal_log"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+error = { path = "../error" }
9
+journal_file = { path = "../journal_file" }
10
+memmap2 = { workspace = true }
11
+uuid = { version = "1.0", features = ["v4", "rng"] }
12
+
13
+[dev-dependencies]
14
+tempfile = { workspace = true }
src/crates/jf/journal_log/src/lib.rs
new
+632
@@ -0,0 +1,632 @@
1
+use error::{JournalError, Result};
2
+use journal_file::{
3
+ load_boot_id, BucketUtilization, JournalFile, JournalFileOptions, JournalWriter,
4
+};
5
+use memmap2::MmapMut;
6
+use std::cmp::Ordering;
7
+use std::ffi::OsStr;
8
+use std::path::{Path, PathBuf};
9
+use std::time::{Duration, SystemTime, UNIX_EPOCH};
10
+
11
+#[derive(Debug, Clone, PartialEq, Eq)]
12
+pub struct JournalFileInfo {
13
+ pub path: PathBuf,
14
+ pub timestamp: SystemTime,
15
+ pub counter: u64,
16
+ pub size: Option<u64>,
17
+}
18
+
19
+impl JournalFileInfo {
20
+ pub fn from_path(path: impl AsRef<Path>) -> Result<Self> {
21
+ let path = path.as_ref();
22
+ let filename = path
23
+ .file_name()
24
+ .and_then(OsStr::to_str)
25
+ .ok_or(JournalError::InvalidFilename)?;
26
+
27
+ let (timestamp, counter) = Self::parse_filename(filename)?;
28
+
29
+ Ok(Self {
30
+ path: path.to_path_buf(),
31
+ timestamp,
32
+ counter,
33
+ size: None,
34
+ })
35
+ }
36
+
37
+ pub fn from_parts(
38
+ timestamp: SystemTime,
39
+ counter: u64,
40
+ size: Option<u64>,
41
+ ) -> Result<JournalFileInfo> {
42
+ let duration = timestamp
43
+ .duration_since(UNIX_EPOCH)
44
+ .map_err(|_| JournalError::SystemTimeError)?;
45
+ let micros = duration.as_secs() * 1_000_000 + duration.subsec_micros() as u64;
46
+ let path = PathBuf::from(format!("journal-{}-{}.journal", micros, counter));
47
+
48
+ Ok(Self {
49
+ path,
50
+ timestamp,
51
+ counter,
52
+ size,
53
+ })
54
+ }
55
+
56
+ /// Parse timestamp and counter from filename
57
+ /// Expected format: "journal-{timestamp_micros}-{counter}.journal"
58
+ fn parse_filename(filename: &str) -> Result<(SystemTime, u64)> {
59
+ let name = filename.strip_suffix(".journal").unwrap_or(filename);
60
+
61
+ if let Some(stripped) = name.strip_prefix("journal-") {
62
+ let parts: Vec<&str> = stripped.split('-').collect();
63
+ if parts.len() == 2 {
64
+ let timestamp_micros: u64 = parts[0]
65
+ .parse()
66
+ .map_err(|_| JournalError::InvalidFilename)?;
67
+ let counter: u64 = parts[1]
68
+ .parse()
69
+ .map_err(|_| JournalError::InvalidFilename)?;
70
+
71
+ let timestamp = UNIX_EPOCH + Duration::from_micros(timestamp_micros);
72
+ return Ok((timestamp, counter));
73
+ }
74
+ }
75
+
76
+ Err(JournalError::InvalidFilename)
77
+ }
78
+
79
+ /// Get file size, loading from filesystem if not cached
80
+ pub fn get_size(&mut self) -> Result<u64> {
81
+ if let Some(size) = self.size {
82
+ Ok(size)
83
+ } else {
84
+ let metadata = std::fs::metadata(&self.path)?;
85
+ let size = metadata.len();
86
+ self.size = Some(size);
87
+ Ok(size)
88
+ }
89
+ }
90
+}
91
+
92
+// Implement ordering based on counter (for detecting duplicates and ordering)
93
+impl Ord for JournalFileInfo {
94
+ fn cmp(&self, other: &Self) -> Ordering {
95
+ // Order by counter only - this enables duplicate detection
96
+ self.counter.cmp(&other.counter)
97
+ }
98
+}
99
+
100
+impl PartialOrd for JournalFileInfo {
101
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
102
+ Some(self.cmp(other))
103
+ }
104
+}
105
+
106
+/// Determines when an active journal file should be sealed
107
+#[derive(Debug, Copy, Clone, Default)]
108
+pub struct RotationPolicy {
109
+ /// Maximum file size before rotating (in bytes)
110
+ pub size_of_journal_file: Option<u64>,
111
+ /// Maximum duration that entries in a single file can span
112
+ pub duration_of_journal_file: Option<Duration>,
113
+}
114
+
115
+impl RotationPolicy {
116
+ pub fn with_size_of_journal_file(mut self, size_of_journal_file: u64) -> Self {
117
+ self.size_of_journal_file = Some(size_of_journal_file);
118
+ self
119
+ }
120
+
121
+ pub fn with_duration_of_journal_file(mut self, duration_of_journal_file: Duration) -> Self {
122
+ self.duration_of_journal_file = Some(duration_of_journal_file);
123
+ self
124
+ }
125
+}
126
+
127
+/// Retention policy - determines when files should be removed
128
+#[derive(Debug, Copy, Clone, Default)]
129
+pub struct RetentionPolicy {
130
+ /// Maximum number of journal files to keep
131
+ pub number_of_journal_files: Option<usize>,
132
+ /// Maximum total size of all journal files (in bytes)
133
+ pub size_of_journal_files: Option<u64>,
134
+ /// Maximum age of files to keep
135
+ pub duration_of_journal_files: Option<Duration>,
136
+}
137
+
138
+impl RetentionPolicy {
139
+ pub fn with_number_of_journal_files(mut self, number_of_journal_files: usize) -> Self {
140
+ self.number_of_journal_files = Some(number_of_journal_files);
141
+ self
142
+ }
143
+
144
+ pub fn with_size_of_journal_files(mut self, size_of_journal_files: u64) -> Self {
145
+ self.size_of_journal_files = Some(size_of_journal_files);
146
+ self
147
+ }
148
+
149
+ pub fn with_duration_of_journal_files(mut self, duration_of_journal_files: Duration) -> Self {
150
+ self.duration_of_journal_files = Some(duration_of_journal_files);
151
+ self
152
+ }
153
+}
154
+
155
+/// Configuration for journal directory management
156
+#[derive(Debug, Clone)]
157
+pub struct JournalDirectoryConfig {
158
+ /// Directory path where journal files are stored
159
+ pub directory: PathBuf,
160
+ /// Policy for when to rotate active files
161
+ pub rotation_policy: RotationPolicy,
162
+ /// Policy for when to remove old files
163
+ pub retention_policy: RetentionPolicy,
164
+}
165
+
166
+impl JournalDirectoryConfig {
167
+ pub fn new(directory: impl Into<PathBuf>) -> Self {
168
+ Self {
169
+ directory: directory.into(),
170
+ rotation_policy: RotationPolicy::default(),
171
+ retention_policy: RetentionPolicy::default(),
172
+ }
173
+ }
174
+
175
+ pub fn with_sealing_policy(mut self, policy: RotationPolicy) -> Self {
176
+ self.rotation_policy = policy;
177
+ self
178
+ }
179
+
180
+ pub fn with_retention_policy(mut self, policy: RetentionPolicy) -> Self {
181
+ self.retention_policy = policy;
182
+ self
183
+ }
184
+}
185
+
186
+/// Manages a directory of journal files with automatic cleanup and sealing
187
+#[derive(Debug)]
188
+pub struct JournalDirectory {
189
+ config: JournalDirectoryConfig,
190
+ /// Files ordered by counter (oldest counter first)
191
+ files: Vec<JournalFileInfo>,
192
+ /// Next counter value for new files
193
+ next_counter: u64,
194
+ /// Cached total size of all files
195
+ total_size: u64,
196
+}
197
+
198
+impl JournalDirectory {
199
+ /// Scan the directory and load existing journal files
200
+ pub fn with_config(config: JournalDirectoryConfig) -> Result<Self> {
201
+ // Create directory if it does not already exist.
202
+ if !config.directory.exists() {
203
+ std::fs::create_dir_all(&config.directory)?;
204
+ } else if !config.directory.is_dir() {
205
+ return Err(JournalError::NotADirectory);
206
+ }
207
+
208
+ let mut journal_directory = Self {
209
+ config,
210
+ files: Vec::new(),
211
+ next_counter: 0,
212
+ total_size: 0,
213
+ };
214
+
215
+ // Read all .journal files from directory
216
+ for entry in std::fs::read_dir(&journal_directory.config.directory)? {
217
+ let entry = entry?;
218
+ let file_path = entry.path();
219
+
220
+ if file_path.extension() != Some(OsStr::new("journal")) {
221
+ continue;
222
+ }
223
+
224
+ match JournalFileInfo::from_path(&file_path) {
225
+ Ok(mut file_info) => {
226
+ // Load the actual file size from filesystem
227
+ let file_size = file_info.get_size().unwrap_or(0);
228
+ journal_directory.total_size += file_size;
229
+ journal_directory.next_counter =
230
+ journal_directory.next_counter.max(file_info.counter + 1);
231
+ journal_directory.files.push(file_info);
232
+ }
233
+ Err(_) => {
234
+ // Skip files with invalid names
235
+ continue;
236
+ }
237
+ }
238
+ }
239
+
240
+ // Sort files by counter to maintain order
241
+ journal_directory.files.sort();
242
+
243
+ Ok(journal_directory)
244
+ }
245
+
246
+ pub fn directory_path(&self) -> &Path {
247
+ &self.config.directory
248
+ }
249
+
250
+ pub fn get_full_path(&self, file_info: &JournalFileInfo) -> PathBuf {
251
+ if file_info.path.is_absolute() {
252
+ file_info.path.clone()
253
+ } else {
254
+ self.config.directory.join(&file_info.path)
255
+ }
256
+ }
257
+
258
+ // Get information about all the files in the journal directory
259
+ pub fn files(&self) -> Vec<JournalFileInfo> {
260
+ self.files.clone()
261
+ }
262
+
263
+ /// Add a new journal file to the directory representation
264
+ pub fn new_file(&mut self, existing_file: Option<JournalFileInfo>) -> Result<JournalFileInfo> {
265
+ let timestamp = SystemTime::now();
266
+ let new_file = JournalFileInfo::from_parts(timestamp, self.next_counter, None)?;
267
+
268
+ self.files.push(new_file.clone());
269
+ self.next_counter += 1;
270
+
271
+ if let Some(existing_file) = existing_file {
272
+ self.total_size += existing_file.size.unwrap_or(0);
273
+ }
274
+
275
+ Ok(new_file)
276
+ }
277
+
278
+ /// Remove the oldest file (by counter) from both filesystem and tracking
279
+ fn remove_oldest_file(&mut self) -> Result<()> {
280
+ if let Some(oldest_file) = self.files.first() {
281
+ let file_path = self.get_full_path(oldest_file);
282
+ let file_size = oldest_file.size.unwrap_or(0);
283
+
284
+ // Remove from filesystem
285
+ if let Err(e) = std::fs::remove_file(&file_path) {
286
+ // Log error but continue cleanup - file might already be deleted
287
+ eprintln!(
288
+ "Warning: Failed to remove journal file {:?}: {}",
289
+ file_path, e
290
+ );
291
+ }
292
+
293
+ // Remove from tracking and update total size
294
+ self.files.remove(0);
295
+ self.total_size = self.total_size.saturating_sub(file_size);
296
+ }
297
+
298
+ Ok(())
299
+ }
300
+
301
+ /// Remove files older than the specified cutoff time
302
+ fn remove_files_older_than(&mut self, cutoff_time: SystemTime) -> Result<()> {
303
+ // Find files older than cutoff time
304
+ let mut files_to_remove = Vec::new();
305
+ for (index, file) in self.files.iter().enumerate() {
306
+ if file.timestamp <= cutoff_time {
307
+ files_to_remove.push(index);
308
+ }
309
+ }
310
+
311
+ // Remove files in reverse order to maintain indices
312
+ for &index in files_to_remove.iter().rev() {
313
+ let file = &self.files[index];
314
+ let file_path = self.get_full_path(file);
315
+ let file_size = file.size.unwrap_or(0);
316
+
317
+ // Remove from filesystem
318
+ if let Err(e) = std::fs::remove_file(&file_path) {
319
+ // Log error but continue cleanup
320
+ eprintln!(
321
+ "Warning: Failed to remove journal file {:?}: {}",
322
+ file_path, e
323
+ );
324
+ }
325
+
326
+ // Remove from tracking and update total size
327
+ self.files.remove(index);
328
+ self.total_size = self.total_size.saturating_sub(file_size);
329
+ }
330
+
331
+ Ok(())
332
+ }
333
+
334
+ /// Enforce the retention policy by removing old files
335
+ pub fn enforce_retention_policy(&mut self) -> Result<()> {
336
+ let policy = self.config.retention_policy;
337
+
338
+ // 1. Remove by file count limit
339
+ if let Some(max_files) = policy.number_of_journal_files {
340
+ while self.files.len() > max_files {
341
+ self.remove_oldest_file()?;
342
+ }
343
+ }
344
+
345
+ // 2. Remove by total size limit
346
+ if let Some(max_total_size) = policy.size_of_journal_files {
347
+ while self.total_size > max_total_size && !self.files.is_empty() {
348
+ self.remove_oldest_file()?;
349
+ }
350
+ }
351
+
352
+ // 3. Remove by entry age limit
353
+ if let Some(max_entry_age) = policy.duration_of_journal_files {
354
+ let cutoff_time = SystemTime::now()
355
+ .checked_sub(max_entry_age)
356
+ .unwrap_or(SystemTime::UNIX_EPOCH);
357
+ self.remove_files_older_than(cutoff_time)?;
358
+ }
359
+
360
+ Ok(())
361
+ }
362
+}
363
+
364
+fn generate_uuid() -> [u8; 16] {
365
+ uuid::Uuid::new_v4().into_bytes()
366
+}
367
+
368
+/// Configuration for JournalLog
369
+#[derive(Debug, Clone)]
370
+pub struct JournalLogConfig {
371
+ /// Directory where journal files are stored
372
+ pub journal_dir: PathBuf,
373
+ /// Policy for when to rotate active files
374
+ pub rotation_policy: RotationPolicy,
375
+ /// Policy for when to remove old files
376
+ pub retention_policy: RetentionPolicy,
377
+}
378
+
379
+impl JournalLogConfig {
380
+ pub fn new(journal_dir: impl Into<PathBuf>) -> Self {
381
+ Self {
382
+ journal_dir: journal_dir.into(),
383
+ rotation_policy: RotationPolicy::default()
384
+ .with_size_of_journal_file(100 * 1024 * 1024) // 100MB
385
+ .with_duration_of_journal_file(Duration::from_secs(2 * 3600)), // 2 hours
386
+ retention_policy: RetentionPolicy::default()
387
+ .with_number_of_journal_files(10) // 10 files
388
+ .with_size_of_journal_files(1024 * 1024 * 1024) // 1GB
389
+ .with_duration_of_journal_files(Duration::from_secs(7 * 24 * 3600)), // 7 days
390
+ }
391
+ }
392
+
393
+ pub fn with_rotation_policy(mut self, policy: RotationPolicy) -> Self {
394
+ self.rotation_policy = policy;
395
+ self
396
+ }
397
+
398
+ pub fn with_retention_policy(mut self, policy: RetentionPolicy) -> Self {
399
+ self.retention_policy = policy;
400
+ self
401
+ }
402
+}
403
+
404
+pub struct JournalLog {
405
+ directory: JournalDirectory,
406
+ current_file: Option<JournalFile<MmapMut>>,
407
+ current_writer: Option<JournalWriter>,
408
+ current_file_info: Option<JournalFileInfo>,
409
+ machine_id: [u8; 16],
410
+ boot_id: [u8; 16],
411
+ seqnum_id: [u8; 16],
412
+ previous_bucket_utilization: Option<BucketUtilization>,
413
+}
414
+
415
+/// Calculate optimal bucket sizes based on previous file utilization or rotation policy
416
+fn calculate_bucket_sizes(
417
+ previous_utilization: Option<&BucketUtilization>,
418
+ rotation_policy: &RotationPolicy,
419
+) -> (usize, usize) {
420
+ if let Some(utilization) = previous_utilization {
421
+ let data_utilization = utilization.data_utilization();
422
+ let field_utilization = utilization.field_utilization();
423
+
424
+ let data_buckets = if data_utilization > 0.75 {
425
+ (utilization.data_total * 2).next_power_of_two()
426
+ } else if data_utilization < 0.25 && utilization.data_total > 4096 {
427
+ (utilization.data_total / 2).next_power_of_two()
428
+ } else {
429
+ utilization.data_total
430
+ };
431
+
432
+ let field_buckets = if field_utilization > 0.75 {
433
+ (utilization.field_total * 2).next_power_of_two()
434
+ } else if field_utilization < 0.25 && utilization.field_total > 512 {
435
+ (utilization.field_total / 2).next_power_of_two()
436
+ } else {
437
+ utilization.field_total
438
+ };
439
+
440
+ (data_buckets, field_buckets)
441
+ } else {
442
+ // Initial sizing based on rotation policy max file size
443
+ let max_file_size = rotation_policy
444
+ .size_of_journal_file
445
+ .unwrap_or(8 * 1024 * 1024);
446
+
447
+ // 16 MiB -> 4096 data buckets
448
+ let data_buckets = (max_file_size / 4096).max(1024).next_power_of_two() as usize;
449
+ let field_buckets = 128; // Assume ~8:1 data:field ratio
450
+
451
+ (data_buckets, field_buckets)
452
+ }
453
+}
454
+
455
+impl JournalLog {
456
+ pub fn new(config: JournalLogConfig) -> Result<Self> {
457
+ let journal_config = JournalDirectoryConfig::new(&config.journal_dir)
458
+ .with_sealing_policy(config.rotation_policy)
459
+ .with_retention_policy(config.retention_policy);
460
+
461
+ let mut directory = JournalDirectory::with_config(journal_config)?;
462
+
463
+ // Enforce retention policy on startup to clean up any old files
464
+ directory.enforce_retention_policy()?;
465
+
466
+ let machine_id = journal_file::file::load_machine_id()?;
467
+ let boot_id = load_boot_id()?;
468
+ // TODO: Use NETDATA_INVOCATION_ID
469
+ let seqnum_id = generate_uuid();
470
+
471
+ Ok(JournalLog {
472
+ directory,
473
+ current_file: None,
474
+ current_writer: None,
475
+ current_file_info: None,
476
+ machine_id,
477
+ boot_id,
478
+ seqnum_id,
479
+ previous_bucket_utilization: None,
480
+ })
481
+ }
482
+
483
+ fn ensure_active_journal(&mut self) -> Result<()> {
484
+ // Check if rotation is needed before writing
485
+ if let Some(writer) = &self.current_writer {
486
+ if self.should_rotate(writer) {
487
+ self.rotate_current_file()?;
488
+ }
489
+ }
490
+
491
+ if self.current_file.is_none() {
492
+ // Create a new journal file
493
+ let file_info = self.directory.new_file(None)?;
494
+
495
+ // Get the full path for the journal file
496
+ let file_path = self.directory.get_full_path(&file_info);
497
+
498
+ // Calculate optimal bucket sizes based on previous file utilization
499
+ let (data_buckets, field_buckets) = calculate_bucket_sizes(
500
+ self.previous_bucket_utilization.as_ref(),
501
+ &self.directory.config.rotation_policy,
502
+ );
503
+
504
+ let options = JournalFileOptions::new(
505
+ self.machine_id,
506
+ self.boot_id,
507
+ self.seqnum_id,
508
+ generate_uuid(),
509
+ )
510
+ .with_window_size(8 * 1024 * 1024)
511
+ .with_data_hash_table_buckets(data_buckets)
512
+ .with_field_hash_table_buckets(field_buckets)
513
+ .with_keyed_hash(true);
514
+
515
+ let mut journal_file = JournalFile::create(&file_path, options)?;
516
+ let writer = JournalWriter::new(&mut journal_file)?;
517
+
518
+ self.current_file = Some(journal_file);
519
+ self.current_writer = Some(writer);
520
+ self.current_file_info = Some(file_info);
521
+
522
+ // Enforce retention policy after creating new file to account for the new file count
523
+ self.directory.enforce_retention_policy()?;
524
+ }
525
+
526
+ Ok(())
527
+ }
528
+
529
+ /// Checks if we have to rotate. Prioritizes file size over file creation
530
+ /// time.
531
+ fn should_rotate(&self, writer: &JournalWriter) -> bool {
532
+ let policy = self.directory.config.rotation_policy;
533
+
534
+ // Check if the file size went over the limit
535
+ if let Some(max_size) = policy.size_of_journal_file {
536
+ if writer.current_file_size() >= max_size {
537
+ return true;
538
+ }
539
+ }
540
+
541
+ // Check if the time span between first and last entries exceeds the limit
542
+ let Some(file) = &self.current_file else {
543
+ return false;
544
+ };
545
+ let Some(max_entry_span) = policy.duration_of_journal_file else {
546
+ return false;
547
+ };
548
+ let Some(first_monotonic) = writer.first_entry_monotonic() else {
549
+ return false;
550
+ };
551
+
552
+ let header = file.journal_header_ref();
553
+ let last_monotonic = header.tail_entry_monotonic;
554
+
555
+ // Convert monotonic timestamps (microseconds) to duration
556
+ let entry_span = if last_monotonic >= first_monotonic {
557
+ Duration::from_micros(last_monotonic - first_monotonic)
558
+ } else {
559
+ return false;
560
+ };
561
+
562
+ if entry_span >= max_entry_span {
563
+ return true;
564
+ }
565
+
566
+ false
567
+ }
568
+
569
+ fn rotate_current_file(&mut self) -> Result<()> {
570
+ // Capture bucket utilization before closing the file
571
+ if let Some(file) = &self.current_file {
572
+ self.previous_bucket_utilization = file.bucket_utilization();
573
+ }
574
+
575
+ // Update the current file's size in our tracking before closing
576
+ if let (Some(file_info), Some(writer)) = (&mut self.current_file_info, &self.current_writer)
577
+ {
578
+ let current_size = writer.current_file_size();
579
+ file_info.size = Some(current_size);
580
+
581
+ // Update the size in the directory's file list
582
+ if let Some(tracked_file) = self
583
+ .directory
584
+ .files
585
+ .iter_mut()
586
+ .find(|f| f.counter == file_info.counter)
587
+ {
588
+ let old_size = tracked_file.size.unwrap_or(0);
589
+ tracked_file.size = Some(current_size);
590
+
591
+ // Update total size tracking
592
+ self.directory.total_size = self
593
+ .directory
594
+ .total_size
595
+ .saturating_sub(old_size)
596
+ .saturating_add(current_size);
597
+ }
598
+ }
599
+
600
+ // Close current file
601
+ self.current_file = None;
602
+ self.current_writer = None;
603
+ self.current_file_info = None;
604
+
605
+ // Next call to ensure_active_journal() will create new file
606
+ Ok(())
607
+ }
608
+
609
+ pub fn write_entry(&mut self, items: &[&[u8]]) -> Result<()> {
610
+ if items.is_empty() {
611
+ return Ok(());
612
+ }
613
+
614
+ self.ensure_active_journal()?;
615
+
616
+ let journal_file = self.current_file.as_mut().unwrap();
617
+ let writer = self.current_writer.as_mut().unwrap();
618
+
619
+ let now = SystemTime::now();
620
+ let realtime = now
621
+ .duration_since(UNIX_EPOCH)
622
+ .unwrap_or_default()
623
+ .as_micros() as u64;
624
+
625
+ // Use realtime for monotonic as well for simplicity
626
+ let monotonic = realtime;
627
+
628
+ writer.add_entry(journal_file, items, realtime, monotonic, self.boot_id)?;
629
+
630
+ Ok(())
631
+ }
632
+}
src/crates/jf/journal_reader/Cargo.toml
new
+10
@@ -0,0 +1,10 @@
1
+[package]
2
+name = "journal_reader"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+error = { path = "../error" }
9
+journal_file = { path = "../journal_file" }
10
+window_manager = { path = "../window_manager" }
src/crates/jf/journal_reader/src/journal_cursor.rs
new
+214
@@ -0,0 +1,214 @@
1
+use crate::journal_filter::FilterExpr;
2
+use error::{JournalError, Result};
3
+use journal_file::{offset_array, offset_array::Direction, JournalFile};
4
+use window_manager::MemoryMap;
5
+
6
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
7
+pub enum Location {
8
+ Head,
9
+ Tail,
10
+ Realtime(u64),
11
+ Monotonic(u64, [u8; 16]),
12
+ Seqnum(u64, Option<[u8; 16]>),
13
+ XorHash(u64),
14
+ ResolvedEntry(u64),
15
+}
16
+
17
+impl Default for Location {
18
+ fn default() -> Self {
19
+ Self::Head
20
+ }
21
+}
22
+
23
+#[derive(Debug)]
24
+pub struct JournalCursor {
25
+ pub location: Location,
26
+ pub filter_expr: Option<FilterExpr>,
27
+ pub array_cursor: Option<offset_array::Cursor>,
28
+}
29
+
30
+impl JournalCursor {
31
+ #[allow(clippy::new_without_default)]
32
+ pub fn new() -> Self {
33
+ Self {
34
+ location: Location::Head,
35
+ filter_expr: None,
36
+ array_cursor: None,
37
+ }
38
+ }
39
+
40
+ pub fn set_location(&mut self, location: Location) {
41
+ self.location = location;
42
+ self.array_cursor = None;
43
+ }
44
+
45
+ pub fn set_filter(&mut self, filter_expr: FilterExpr) {
46
+ self.filter_expr = Some(filter_expr);
47
+ // FIXME: should we set cursor to None?
48
+ }
49
+
50
+ pub fn clear_filter(&mut self) {
51
+ self.filter_expr = None;
52
+ self.array_cursor = None;
53
+ self.set_location(Location::Head);
54
+ }
55
+
56
+ pub fn step<M: MemoryMap>(
57
+ &mut self,
58
+ journal_file: &JournalFile<M>,
59
+ direction: Direction,
60
+ ) -> Result<bool> {
61
+ let new_location = if self.filter_expr.is_some() {
62
+ self.resolve_filter_location(journal_file, direction)?
63
+ } else {
64
+ self.resolve_array_cursor(journal_file, direction)?
65
+ };
66
+
67
+ if let Some(location) = new_location {
68
+ self.location = location;
69
+ Ok(true)
70
+ } else {
71
+ Ok(false)
72
+ }
73
+ }
74
+
75
+ pub fn position(&self) -> Result<u64> {
76
+ match self.location {
77
+ Location::ResolvedEntry(entry_offset) => Ok(entry_offset),
78
+ _ => Err(JournalError::UnsetCursor),
79
+ }
80
+ }
81
+
82
+ fn resolve_array_cursor<M: MemoryMap>(
83
+ &mut self,
84
+ journal_file: &JournalFile<M>,
85
+ direction: Direction,
86
+ ) -> Result<Option<Location>> {
87
+ let new_location = match (self.location, direction) {
88
+ (Location::Head, Direction::Forward) => {
89
+ let entry_list = journal_file
90
+ .entry_list()
91
+ .ok_or(JournalError::InvalidOffsetArrayOffset)?;
92
+
93
+ let cursor = entry_list.cursor_head();
94
+ let offset = cursor.value(journal_file)?;
95
+ self.array_cursor = Some(cursor);
96
+ Some(Location::ResolvedEntry(offset))
97
+ }
98
+ (Location::Head, Direction::Backward) => None,
99
+ (Location::Tail, Direction::Forward) => None,
100
+ (Location::Tail, Direction::Backward) => {
101
+ let entry_list = journal_file
102
+ .entry_list()
103
+ .ok_or(JournalError::InvalidOffsetArrayOffset)?;
104
+
105
+ let cursor = entry_list.cursor_tail(journal_file)?;
106
+ let offset = cursor.value(journal_file)?;
107
+ self.array_cursor = Some(cursor);
108
+ Some(Location::ResolvedEntry(offset))
109
+ }
110
+ (Location::Realtime(realtime), _) => {
111
+ let entry_list = journal_file
112
+ .entry_list()
113
+ .ok_or(JournalError::InvalidOffsetArrayOffset)?;
114
+
115
+ let predicate = |entry_offset| {
116
+ let entry_object = journal_file.entry_ref(entry_offset)?;
117
+ Ok(entry_object.header.realtime < realtime)
118
+ };
119
+
120
+ let cursor = entry_list
121
+ .directed_partition_point(journal_file, predicate, Direction::Forward)?
122
+ .map(Ok)
123
+ .unwrap_or_else(|| entry_list.cursor_tail(journal_file))?;
124
+
125
+ let offset = cursor.value(journal_file)?;
126
+ self.array_cursor = Some(cursor);
127
+ Some(Location::ResolvedEntry(offset))
128
+ }
129
+ (Location::ResolvedEntry(_), Direction::Forward) => {
130
+ let Some(cursor) = self.array_cursor.unwrap().next(journal_file)? else {
131
+ return Ok(None);
132
+ };
133
+
134
+ let offset = cursor.value(journal_file)?;
135
+ self.array_cursor = Some(cursor);
136
+ Some(Location::ResolvedEntry(offset))
137
+ }
138
+ (Location::ResolvedEntry(_), Direction::Backward) => {
139
+ let Some(cursor) = self.array_cursor.unwrap().previous(journal_file)? else {
140
+ return Ok(None);
141
+ };
142
+
143
+ let offset = cursor.value(journal_file)?;
144
+ self.array_cursor = Some(cursor);
145
+ Some(Location::ResolvedEntry(offset))
146
+ }
147
+ _ => {
148
+ unimplemented!()
149
+ }
150
+ };
151
+
152
+ Ok(new_location)
153
+ }
154
+
155
+ fn resolve_filter_location<M: MemoryMap>(
156
+ &mut self,
157
+ journal_file: &JournalFile<M>,
158
+ direction: Direction,
159
+ ) -> Result<Option<Location>> {
160
+ let filter_expr = self.filter_expr.as_mut().unwrap();
161
+
162
+ let resolved_location = match (self.location, direction) {
163
+ (Location::Head, Direction::Forward) => filter_expr
164
+ .head()
165
+ .next(journal_file, u64::MIN)?
166
+ .map(Location::ResolvedEntry),
167
+ (Location::Head, Direction::Backward) => None,
168
+ (Location::Tail, Direction::Forward) => None,
169
+ (Location::Tail, Direction::Backward) => filter_expr
170
+ .tail(journal_file)?
171
+ .previous(journal_file, u64::MAX)?
172
+ .map(Location::ResolvedEntry),
173
+ (Location::Realtime(realtime), direction) => {
174
+ let entry_list = journal_file
175
+ .entry_list()
176
+ .ok_or(JournalError::InvalidOffsetArrayOffset)?;
177
+
178
+ let predicate = |entry_offset| {
179
+ let entry_object = journal_file.entry_ref(entry_offset)?;
180
+ Ok(entry_object.header.realtime < realtime)
181
+ };
182
+
183
+ let cursor = entry_list
184
+ .directed_partition_point(journal_file, predicate, Direction::Forward)?
185
+ .map(Ok)
186
+ .unwrap_or_else(|| entry_list.cursor_tail(journal_file))?;
187
+
188
+ let entry_offset = cursor.value(journal_file)?;
189
+
190
+ match direction {
191
+ Direction::Forward => filter_expr
192
+ .head()
193
+ .next(journal_file, entry_offset)?
194
+ .map(Location::ResolvedEntry),
195
+ Direction::Backward => filter_expr
196
+ .tail(journal_file)?
197
+ .previous(journal_file, entry_offset)?
198
+ .map(Location::ResolvedEntry),
199
+ }
200
+ }
201
+ (Location::ResolvedEntry(location_offset), Direction::Forward) => filter_expr
202
+ .next(journal_file, location_offset + 1)?
203
+ .map(Location::ResolvedEntry),
204
+ (Location::ResolvedEntry(location_offset), Direction::Backward) => filter_expr
205
+ .previous(journal_file, location_offset - 1)?
206
+ .map(Location::ResolvedEntry),
207
+ _ => {
208
+ unimplemented!();
209
+ }
210
+ };
211
+
212
+ Ok(resolved_location)
213
+ }
214
+}
src/crates/jf/journal_reader/src/journal_filter.rs
new
+416
@@ -0,0 +1,416 @@
1
+use error::{JournalError, Result};
2
+use journal_file::{
3
+ offset_array::{Direction, InlinedCursor},
4
+ JournalFile,
5
+};
6
+use window_manager::MemoryMap;
7
+
8
+#[derive(Clone, Debug)]
9
+pub enum FilterExpr {
10
+ Match(u64, Option<InlinedCursor>),
11
+ Conjunction(Vec<FilterExpr>),
12
+ Disjunction(Vec<FilterExpr>),
13
+}
14
+
15
+impl FilterExpr {
16
+ pub fn lookup<M: MemoryMap>(
17
+ &self,
18
+ journal_file: &JournalFile<M>,
19
+ needle_offset: u64,
20
+ direction: Direction,
21
+ ) -> Result<Option<u64>> {
22
+ let predicate =
23
+ move |entry_offset: u64| -> Result<bool> { Ok(entry_offset < needle_offset) };
24
+
25
+ match self {
26
+ FilterExpr::Match(data_offset, _) => {
27
+ let entry_offset = journal_file.data_object_directed_partition_point(
28
+ *data_offset,
29
+ predicate,
30
+ direction,
31
+ )?;
32
+ Ok(entry_offset)
33
+ }
34
+ FilterExpr::Conjunction(filter_exprs) => {
35
+ let mut current_offset = needle_offset;
36
+
37
+ loop {
38
+ let previous_offset = current_offset;
39
+
40
+ for filter_expr in filter_exprs {
41
+ if direction == Direction::Backward {
42
+ current_offset = current_offset.saturating_add(1);
43
+ }
44
+
45
+ match filter_expr.lookup(journal_file, current_offset, direction)? {
46
+ Some(new_offset) => current_offset = new_offset,
47
+ None => return Ok(None),
48
+ }
49
+ }
50
+
51
+ if current_offset == previous_offset {
52
+ return Ok(Some(current_offset));
53
+ }
54
+ }
55
+ }
56
+ FilterExpr::Disjunction(filter_exprs) => {
57
+ let cmp = match direction {
58
+ Direction::Forward => std::cmp::min,
59
+ Direction::Backward => std::cmp::max,
60
+ };
61
+
62
+ filter_exprs.iter().try_fold(None, |acc, expr| {
63
+ let result = expr.lookup(journal_file, needle_offset, direction)?;
64
+
65
+ Ok(match (acc, result) {
66
+ (None, Some(offset)) => Some(offset),
67
+ (Some(best), Some(offset)) => Some(cmp(best, offset)),
68
+ (acc, None) => acc,
69
+ })
70
+ })
71
+ }
72
+ }
73
+ }
74
+
75
+ pub fn head(&mut self) -> &mut Self {
76
+ match self {
77
+ FilterExpr::Match(_, None) => (),
78
+ FilterExpr::Match(_, Some(ic)) => {
79
+ *ic = ic.head();
80
+ }
81
+ FilterExpr::Conjunction(filter_exprs) => {
82
+ for filter_expr in filter_exprs.iter_mut() {
83
+ filter_expr.head();
84
+ }
85
+ }
86
+ FilterExpr::Disjunction(filter_exprs) => {
87
+ for filter_expr in filter_exprs.iter_mut() {
88
+ filter_expr.head();
89
+ }
90
+ }
91
+ }
92
+
93
+ self
94
+ }
95
+
96
+ pub fn tail<M: MemoryMap>(&mut self, journal_file: &JournalFile<M>) -> Result<&mut Self> {
97
+ match self {
98
+ FilterExpr::Match(_, None) => (),
99
+ FilterExpr::Match(_, Some(ic)) => {
100
+ *ic = ic.tail(journal_file)?;
101
+ }
102
+ FilterExpr::Conjunction(filter_exprs) => {
103
+ for filter_expr in filter_exprs.iter_mut() {
104
+ filter_expr.tail(journal_file)?;
105
+ }
106
+ }
107
+ FilterExpr::Disjunction(filter_exprs) => {
108
+ for filter_expr in filter_exprs.iter_mut() {
109
+ filter_expr.tail(journal_file)?;
110
+ }
111
+ }
112
+ }
113
+
114
+ Ok(self)
115
+ }
116
+
117
+ // Returns the offset of the next matching entry, if any, with an offset
118
+ // greater or equal to the needle offset.
119
+ pub fn next<M: MemoryMap>(
120
+ &mut self,
121
+ journal_file: &JournalFile<M>,
122
+ needle_offset: u64,
123
+ ) -> Result<Option<u64>> {
124
+ match self {
125
+ FilterExpr::Match(_, None) => Ok(None),
126
+ FilterExpr::Match(_, Some(ic)) => ic.next_until(journal_file, needle_offset),
127
+ FilterExpr::Conjunction(filter_exprs) => {
128
+ let mut needle_offset = needle_offset;
129
+
130
+ loop {
131
+ let previous_offset = needle_offset;
132
+
133
+ for fe in filter_exprs.iter_mut() {
134
+ if let Some(new_offset) = fe.next(journal_file, needle_offset)? {
135
+ needle_offset = new_offset;
136
+ } else {
137
+ return Ok(None);
138
+ }
139
+ }
140
+
141
+ if needle_offset == previous_offset {
142
+ return Ok(Some(needle_offset));
143
+ }
144
+ }
145
+ }
146
+ FilterExpr::Disjunction(filter_exprs) => {
147
+ let mut best_offset: Option<u64> = None;
148
+
149
+ for fe in filter_exprs.iter_mut() {
150
+ if let Some(fe_offset) = fe.next(journal_file, needle_offset)? {
151
+ best_offset = match best_offset {
152
+ Some(offset) => Some(fe_offset.min(offset)),
153
+ None => Some(fe_offset),
154
+ };
155
+ }
156
+ }
157
+
158
+ Ok(best_offset)
159
+ }
160
+ }
161
+ }
162
+
163
+ // Returns the offset of the previous matching entry, if any, with an offset
164
+ // less or equal to the needle offset.
165
+ pub fn previous<M: MemoryMap>(
166
+ &mut self,
167
+ journal_file: &JournalFile<M>,
168
+ needle_offset: u64,
169
+ ) -> Result<Option<u64>> {
170
+ match self {
171
+ FilterExpr::Match(_, None) => Ok(None),
172
+ FilterExpr::Match(_, Some(ic)) => ic.previous_until(journal_file, needle_offset),
173
+ FilterExpr::Conjunction(filter_exprs) => {
174
+ let mut needle_offset = needle_offset;
175
+
176
+ loop {
177
+ let previous_offset = needle_offset;
178
+
179
+ for fe in filter_exprs.iter_mut().rev() {
180
+ if let Some(new_offset) = fe.previous(journal_file, needle_offset)? {
181
+ needle_offset = new_offset;
182
+ } else {
183
+ return Ok(None);
184
+ }
185
+ }
186
+
187
+ if needle_offset == previous_offset {
188
+ return Ok(Some(needle_offset));
189
+ }
190
+ }
191
+ }
192
+ FilterExpr::Disjunction(filter_exprs) => {
193
+ let mut best_offset: Option<u64> = None;
194
+
195
+ for fe in filter_exprs.iter_mut() {
196
+ if let Some(fe_offset) = fe.previous(journal_file, needle_offset)? {
197
+ best_offset = match best_offset {
198
+ Some(offset) => Some(fe_offset.max(offset)),
199
+ None => Some(fe_offset),
200
+ };
201
+ }
202
+ }
203
+
204
+ Ok(best_offset)
205
+ }
206
+ }
207
+ }
208
+
209
+ pub fn dump<M: MemoryMap>(&self, journal_file: &JournalFile<M>) -> Result<String> {
210
+ let mut output = String::new();
211
+ self.dump_internal(journal_file, 0, &mut output)?;
212
+ Ok(output)
213
+ }
214
+
215
+ /// Helper function for format_data_objects that handles nested expressions and indentation
216
+ fn dump_internal<M: MemoryMap>(
217
+ &self,
218
+ journal_file: &JournalFile<M>,
219
+ indent_level: usize,
220
+ output: &mut String,
221
+ ) -> Result<()> {
222
+ let indent = " ".repeat(indent_level);
223
+
224
+ match self {
225
+ FilterExpr::Match(data_offset, inlined_cursor) => {
226
+ // Load the data object
227
+ let data_object = journal_file.data_ref(*data_offset)?;
228
+
229
+ // Get the payload as a string if possible
230
+ let payload_bytes = data_object.payload_bytes();
231
+ let payload_str = String::from_utf8_lossy(payload_bytes);
232
+
233
+ // Format the data offset and payload
234
+ output.push_str(&format!(
235
+ "{}Match[offset=0x{:x}]: {}\n",
236
+ indent, data_offset, payload_str
237
+ ));
238
+
239
+ // Format cursor information if available
240
+ if let Some(ic) = inlined_cursor {
241
+ output.push_str(&format!("{} Cursor: {:?}\n", indent, ic));
242
+ }
243
+ }
244
+ FilterExpr::Conjunction(filter_exprs) => {
245
+ output.push_str(&format!("{}Conjunction (AND) {{\n", indent));
246
+ for expr in filter_exprs {
247
+ expr.dump_internal(journal_file, indent_level + 1, output)?;
248
+ }
249
+ output.push_str(&format!("{}}}\n", indent));
250
+ }
251
+ FilterExpr::Disjunction(filter_exprs) => {
252
+ output.push_str(&format!("{}Disjunction (OR) {{\n", indent));
253
+ for expr in filter_exprs {
254
+ expr.dump_internal(journal_file, indent_level + 1, output)?;
255
+ }
256
+ output.push_str(&format!("{}}}\n", indent));
257
+ }
258
+ }
259
+
260
+ Ok(())
261
+ }
262
+}
263
+
264
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
265
+pub enum LogicalOp {
266
+ Conjunction,
267
+ Disjunction,
268
+}
269
+
270
+#[derive(Debug)]
271
+pub struct JournalFilter {
272
+ filter_expr: Option<FilterExpr>,
273
+ current_matches: Vec<Vec<u8>>,
274
+ current_op: LogicalOp,
275
+}
276
+
277
+impl Default for JournalFilter {
278
+ fn default() -> Self {
279
+ Self {
280
+ filter_expr: None,
281
+ current_matches: Vec::new(),
282
+ current_op: LogicalOp::Conjunction,
283
+ }
284
+ }
285
+}
286
+
287
+impl JournalFilter {
288
+ fn extract_key(kv_pair: &[u8]) -> Option<&[u8]> {
289
+ if let Some(equal_pos) = kv_pair.iter().position(|&b| b == b'=') {
290
+ Some(&kv_pair[..equal_pos])
291
+ } else {
292
+ None
293
+ }
294
+ }
295
+
296
+ fn convert_current_matches<M: MemoryMap>(
297
+ &mut self,
298
+ journal_file: &JournalFile<M>,
299
+ ) -> Result<Option<FilterExpr>> {
300
+ if self.current_matches.is_empty() {
301
+ return Ok(None);
302
+ }
303
+
304
+ let mut elements = Vec::new();
305
+ let mut i = 0;
306
+
307
+ while i < self.current_matches.len() {
308
+ let current_key = Self::extract_key(&self.current_matches[i]).unwrap_or(&[]);
309
+ let start = i;
310
+
311
+ // Find all matches with the same key
312
+ while i < self.current_matches.len()
313
+ && Self::extract_key(&self.current_matches[i]).unwrap_or(&[]) == current_key
314
+ {
315
+ i += 1;
316
+ }
317
+
318
+ // If we have multiple values for this key, create a disjunction
319
+ if i - start > 1 {
320
+ let mut matches = Vec::with_capacity(i - start);
321
+ for idx in start..i {
322
+ let data = self.current_matches[idx].as_slice();
323
+ let hash = journal_file.hash(data);
324
+ let offset = journal_file.find_data_offset_by_payload(data, hash)?;
325
+
326
+ let ic = journal_file.data_ref(offset)?.inlined_cursor();
327
+ matches.push(FilterExpr::Match(offset, ic));
328
+ }
329
+ elements.push(FilterExpr::Disjunction(matches));
330
+ } else {
331
+ let data = self.current_matches[start].as_slice();
332
+ let hash = journal_file.hash(data);
333
+ let offset = journal_file.find_data_offset_by_payload(data, hash)?;
334
+
335
+ let ic = journal_file.data_ref(offset)?.inlined_cursor();
336
+ elements.push(FilterExpr::Match(offset, ic));
337
+ }
338
+ }
339
+
340
+ self.current_matches.clear();
341
+
342
+ match elements.len() {
343
+ 0 => panic!("Could not create filter elements from current matches"),
344
+ 1 => Ok(Some(elements.remove(0))),
345
+ _ => Ok(Some(FilterExpr::Conjunction(elements))),
346
+ }
347
+ }
348
+
349
+ pub fn add_match(&mut self, kv_pair: &[u8]) {
350
+ if kv_pair.contains(&b'=') {
351
+ let new_item = kv_pair.to_vec();
352
+ let new_key = Self::extract_key(&new_item).unwrap_or(&[]);
353
+
354
+ // Find the insertion position using binary search
355
+ let pos = self
356
+ .current_matches
357
+ .binary_search_by(|item| {
358
+ let key = Self::extract_key(item).unwrap_or(&[]);
359
+ key.cmp(new_key)
360
+ })
361
+ .unwrap_or_else(|e| e);
362
+
363
+ // Insert at the found position
364
+ self.current_matches.insert(pos, new_item);
365
+ }
366
+ }
367
+
368
+ pub fn set_operation<M: MemoryMap>(
369
+ &mut self,
370
+ journal_file: &JournalFile<M>,
371
+ op: LogicalOp,
372
+ ) -> Result<()> {
373
+ let new_expr = self.convert_current_matches(journal_file)?;
374
+ if new_expr.is_none() {
375
+ self.current_op = op;
376
+ return Ok(());
377
+ }
378
+
379
+ if self.filter_expr.is_none() {
380
+ self.filter_expr = new_expr;
381
+ self.current_op = op;
382
+ return Ok(());
383
+ }
384
+
385
+ let new_expr = new_expr.unwrap();
386
+ let current_expr = self.filter_expr.take().unwrap();
387
+
388
+ self.filter_expr = Some(match (current_expr, self.current_op) {
389
+ (FilterExpr::Disjunction(mut exprs), LogicalOp::Disjunction) => {
390
+ exprs.push(new_expr);
391
+ FilterExpr::Disjunction(exprs)
392
+ }
393
+ (FilterExpr::Conjunction(mut exprs), LogicalOp::Conjunction) => {
394
+ exprs.push(new_expr);
395
+ FilterExpr::Conjunction(exprs)
396
+ }
397
+ (current_expr, LogicalOp::Disjunction) => {
398
+ FilterExpr::Disjunction(vec![current_expr, new_expr])
399
+ }
400
+ (current_expr, LogicalOp::Conjunction) => {
401
+ FilterExpr::Conjunction(vec![current_expr, new_expr])
402
+ }
403
+ });
404
+
405
+ self.current_op = op;
406
+ Ok(())
407
+ }
408
+
409
+ pub fn build<M: MemoryMap>(&mut self, journal_file: &JournalFile<M>) -> Result<FilterExpr> {
410
+ self.set_operation(journal_file, self.current_op)?;
411
+
412
+ self.current_matches.clear();
413
+ self.current_op = LogicalOp::Conjunction;
414
+ self.filter_expr.take().ok_or(JournalError::MalformedFilter)
415
+ }
416
+}
src/crates/jf/journal_reader/src/lib.rs
new
+197
@@ -0,0 +1,197 @@
1
+use error::Result;
2
+use journal_file::{
3
+ offset_array, DataObject, EntryDataIterator, FieldDataIterator, FieldIterator, FieldObject,
4
+ JournalFile, ValueGuard,
5
+};
6
+use window_manager::MemoryMap;
7
+
8
+pub mod journal_filter;
9
+use journal_filter::{JournalFilter, LogicalOp};
10
+pub mod journal_cursor;
11
+use journal_cursor::JournalCursor;
12
+pub use journal_cursor::Location;
13
+
14
+pub use offset_array::Direction;
15
+
16
+pub struct JournalReader<'a, M: MemoryMap> {
17
+ cursor: JournalCursor,
18
+
19
+ filter: Option<JournalFilter>,
20
+ field_iterator: Option<FieldIterator<'a, M>>,
21
+ field_data_iterator: Option<FieldDataIterator<'a, M>>,
22
+ entry_data_iterator: Option<EntryDataIterator<'a, M>>,
23
+
24
+ field_guard: Option<ValueGuard<'a, FieldObject<&'a [u8]>>>,
25
+ data_guard: Option<ValueGuard<'a, DataObject<&'a [u8]>>>,
26
+}
27
+
28
+impl<M: MemoryMap> std::fmt::Debug for JournalReader<'_, M> {
29
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
30
+ f.debug_struct("JournalReader")
31
+ // .field("cursor", &self.cursor)
32
+ .field("field_guard", &self.field_guard)
33
+ .field("data_guard", &self.data_guard)
34
+ .finish()
35
+ }
36
+}
37
+
38
+impl<M: MemoryMap> Default for JournalReader<'_, M> {
39
+ fn default() -> Self {
40
+ Self {
41
+ cursor: JournalCursor::new(),
42
+ filter: None,
43
+ field_iterator: None,
44
+ field_data_iterator: None,
45
+ entry_data_iterator: None,
46
+ field_guard: None,
47
+ data_guard: None,
48
+ }
49
+ }
50
+}
51
+
52
+impl<'a, M: MemoryMap> JournalReader<'a, M> {
53
+ pub fn dump(&self, journal_file: &'a JournalFile<M>) -> Result<String> {
54
+ if let Some(filter_expr) = self.cursor.filter_expr.as_ref() {
55
+ filter_expr.dump(journal_file)
56
+ } else {
57
+ Ok(String::from("no filter expr"))
58
+ }
59
+ }
60
+
61
+ pub fn set_location(&mut self, location: Location) {
62
+ self.cursor.set_location(location)
63
+ }
64
+
65
+ pub fn step(&mut self, journal_file: &'a JournalFile<M>, direction: Direction) -> Result<bool> {
66
+ self.drop_guards();
67
+
68
+ if let Some(filter) = self.filter.as_mut() {
69
+ let filter_expr = filter.build(journal_file)?;
70
+ self.cursor.set_filter(filter_expr);
71
+ self.filter = None;
72
+ }
73
+
74
+ self.cursor.step(journal_file, direction)
75
+ }
76
+
77
+ pub fn add_match(&mut self, data: &[u8]) {
78
+ self.filter.get_or_insert_default().add_match(data);
79
+ }
80
+
81
+ pub fn add_conjunction(&mut self, journal_file: &'a JournalFile<M>) -> Result<()> {
82
+ self.filter
83
+ .get_or_insert_default()
84
+ .set_operation(journal_file, LogicalOp::Conjunction)
85
+ }
86
+
87
+ pub fn add_disjunction(&mut self, journal_file: &'a JournalFile<M>) -> Result<()> {
88
+ self.filter
89
+ .get_or_insert_default()
90
+ .set_operation(journal_file, LogicalOp::Disjunction)
91
+ }
92
+
93
+ pub fn flush_matches(&mut self) {
94
+ self.cursor.clear_filter();
95
+ self.filter = None;
96
+ }
97
+
98
+ pub fn get_realtime_usec(&self, journal_file: &'a JournalFile<M>) -> Result<u64> {
99
+ let entry_offset = self.cursor.position()?;
100
+ let entry_object = journal_file.entry_ref(entry_offset)?;
101
+ Ok(entry_object.header.realtime)
102
+ }
103
+
104
+ pub fn get_seqnum(&self, journal_file: &'a JournalFile<M>) -> Result<(u64, [u8; 16])> {
105
+ let entry_offset = self.cursor.position()?;
106
+ let entry_object = journal_file.entry_ref(entry_offset)?;
107
+ Ok((
108
+ entry_object.header.seqnum,
109
+ journal_file.journal_header_ref().seqnum_id,
110
+ ))
111
+ }
112
+
113
+ pub fn get_entry_offset(&self) -> Result<u64> {
114
+ self.cursor.position()
115
+ }
116
+
117
+ fn drop_guards(&mut self) {
118
+ self.field_guard.take();
119
+ self.data_guard.take();
120
+ }
121
+
122
+ pub fn fields_restart(&mut self) {
123
+ self.drop_guards();
124
+ self.field_iterator = None;
125
+ }
126
+
127
+ pub fn fields_enumerate(
128
+ &mut self,
129
+ journal_file: &'a JournalFile<M>,
130
+ ) -> Result<Option<&ValueGuard<FieldObject<&'a [u8]>>>> {
131
+ self.drop_guards();
132
+
133
+ if self.field_iterator.is_none() {
134
+ self.field_iterator = Some(journal_file.fields());
135
+ }
136
+
137
+ if let Some(iter) = &mut self.field_iterator {
138
+ self.field_guard = iter.next().transpose()?;
139
+ Ok(self.field_guard.as_ref())
140
+ } else {
141
+ Ok(None)
142
+ }
143
+ }
144
+
145
+ pub fn field_data_query_unique(
146
+ &mut self,
147
+ journal_file: &'a JournalFile<M>,
148
+ field_name: &'a [u8],
149
+ ) -> Result<()> {
150
+ self.drop_guards();
151
+
152
+ self.field_data_iterator = Some(journal_file.field_data_objects(field_name)?);
153
+ Ok(())
154
+ }
155
+
156
+ pub fn field_data_restart(&mut self) {
157
+ self.drop_guards();
158
+ }
159
+
160
+ pub fn field_data_enumerate(
161
+ &mut self,
162
+ _: &'a JournalFile<M>,
163
+ ) -> Result<Option<&ValueGuard<DataObject<&'a [u8]>>>> {
164
+ self.drop_guards();
165
+
166
+ if let Some(iter) = &mut self.field_data_iterator {
167
+ self.data_guard = iter.next().transpose()?;
168
+ Ok(self.data_guard.as_ref())
169
+ } else {
170
+ Ok(None)
171
+ }
172
+ }
173
+
174
+ pub fn entry_data_restart(&mut self) {
175
+ self.drop_guards();
176
+ self.entry_data_iterator = None;
177
+ }
178
+
179
+ pub fn entry_data_enumerate(
180
+ &mut self,
181
+ journal_file: &'a JournalFile<M>,
182
+ ) -> Result<Option<&ValueGuard<DataObject<&'a [u8]>>>> {
183
+ self.drop_guards();
184
+
185
+ if self.entry_data_iterator.is_none() {
186
+ let entry_offset = self.cursor.position()?;
187
+ self.entry_data_iterator = Some(journal_file.entry_data_objects(entry_offset)?);
188
+ }
189
+
190
+ if let Some(iter) = &mut self.entry_data_iterator {
191
+ self.data_guard = iter.next().transpose()?;
192
+ Ok(self.data_guard.as_ref())
193
+ } else {
194
+ Ok(None)
195
+ }
196
+ }
197
+}
src/crates/jf/journal_reader_ffi/Cargo.toml
new
+19
@@ -0,0 +1,19 @@
1
+[package]
2
+name = "journal_reader_ffi"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+error = { path = "../error" }
9
+# journal_reader integrated into journal_file
10
+journal_file = { path = "../journal_file" }
11
+memmap2 = { workspace = true }
12
+serde_json = { workspace = true }
13
+sigbus = { path = "../sigbus" }
14
+
15
+[build-dependencies]
16
+cbindgen = "0.28.0"
17
+
18
+[lib]
19
+crate-type = ["staticlib"]
src/crates/jf/journal_reader_ffi/build.rs
new
+21
@@ -0,0 +1,21 @@
1
+extern crate cbindgen;
2
+
3
+use std::env;
4
+
5
+fn main() {
6
+ let crate_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
7
+
8
+ cbindgen::Builder::new()
9
+ .with_crate(crate_dir)
10
+ .with_language(cbindgen::Language::C)
11
+ .with_cpp_compat(true)
12
+ .with_include_guard("JOURNAL_READER_FFI_H")
13
+ .rename_item("SdJournal", "sd_journal")
14
+ .rename_item("SdId128", "sd_id128_t")
15
+ .generate()
16
+ .expect("Unable to generate bindings")
17
+ .write_to_file("journal_reader_ffi.h");
18
+
19
+ println!("cargo:rerun-if-changed=cbindgen.toml");
20
+ println!("cargo:rerun-if-changed=src/");
21
+}
src/crates/jf/journal_reader_ffi/src/lib.rs
new
+448
@@ -0,0 +1,448 @@
1
+use journal_file::{Direction, HashableObject, JournalFile, JournalReader, Location};
2
+use memmap2::Mmap;
3
+use std::ffi::{c_char, c_int, c_void, CStr};
4
+
5
+#[repr(C)]
6
+#[derive(Debug, Clone, Copy)]
7
+pub struct RsdId128 {
8
+ pub bytes: [u8; 16],
9
+}
10
+
11
+fn unhexchar(c: u8) -> Result<u8, i32> {
12
+ match c {
13
+ b'0'..=b'9' => Ok(c - b'0'),
14
+ b'a'..=b'f' => Ok(c - b'a' + 10),
15
+ b'A'..=b'F' => Ok(c - b'A' + 10),
16
+ _ => Err(-22), // -EINVAL
17
+ }
18
+}
19
+
20
+#[no_mangle]
21
+unsafe extern "C" fn rsd_id128_from_string(s: *const c_char, ret: *mut RsdId128) -> i32 {
22
+ debug_assert!(!s.is_null());
23
+ debug_assert!(!ret.is_null());
24
+
25
+ let c_str = match CStr::from_ptr(s).to_str() {
26
+ Ok(s) => s,
27
+ Err(_) => return -1,
28
+ };
29
+
30
+ let res = &mut *ret;
31
+ let mut n: usize = 0;
32
+ let mut i: usize = 0;
33
+ let mut is_guid = false;
34
+
35
+ let bytes = c_str.as_bytes();
36
+
37
+ while n < 16 {
38
+ if i >= bytes.len() {
39
+ return -1;
40
+ }
41
+
42
+ if bytes[i] == b'-' {
43
+ if i == 8 {
44
+ is_guid = true;
45
+ } else if i == 13 || i == 18 || i == 23 {
46
+ if !is_guid {
47
+ return -1;
48
+ }
49
+ } else {
50
+ return -1;
51
+ }
52
+
53
+ i += 1;
54
+ continue;
55
+ }
56
+
57
+ if i + 1 >= bytes.len() {
58
+ return -1;
59
+ }
60
+
61
+ let a = match unhexchar(bytes[i]) {
62
+ Ok(val) => val,
63
+ Err(e) => return e,
64
+ };
65
+ i += 1;
66
+
67
+ let b = match unhexchar(bytes[i]) {
68
+ Ok(val) => val,
69
+ Err(e) => return e,
70
+ };
71
+ i += 1;
72
+
73
+ res.bytes[n] = (a << 4) | b;
74
+ n += 1;
75
+ }
76
+
77
+ let expected_len = if is_guid { 36 } else { 32 };
78
+ if i != expected_len || i >= bytes.len() || bytes[i] != 0 {
79
+ return -1;
80
+ }
81
+
82
+ 0
83
+}
84
+
85
+#[no_mangle]
86
+pub extern "C" fn rsd_id128_equal(a: RsdId128, b: RsdId128) -> i32 {
87
+ (a.bytes == b.bytes) as i32
88
+}
89
+
90
+impl PartialEq for RsdId128 {
91
+ fn eq(&self, other: &Self) -> bool {
92
+ self.bytes == other.bytes
93
+ }
94
+}
95
+
96
+impl Eq for RsdId128 {}
97
+
98
+struct RsdJournal<'a> {
99
+ journal_file: Box<JournalFile<Mmap>>,
100
+ reader: JournalReader<'a, Mmap>,
101
+ field_buffer: Vec<u8>,
102
+ decompressed_payload: Vec<u8>,
103
+}
104
+
105
+#[no_mangle]
106
+unsafe extern "C" fn rsd_journal_open_files(
107
+ ret: *mut *mut RsdJournal,
108
+ paths: *const *const c_char,
109
+ _flags: c_int,
110
+) -> c_int {
111
+ debug_assert!(!ret.is_null());
112
+ debug_assert!(!paths.is_null());
113
+
114
+ if sigbus::install_handler().is_err() {
115
+ eprintln!("Failed to install sigbus handler");
116
+ }
117
+
118
+ // Get the first path
119
+ let path_ptr = *paths;
120
+ if path_ptr.is_null() {
121
+ return error::JournalError::InvalidFfiOp.to_error_code();
122
+ }
123
+
124
+ // Convert C string to Rust string
125
+ let path = match CStr::from_ptr(path_ptr).to_str() {
126
+ Ok(s) => s,
127
+ Err(_) => {
128
+ return error::JournalError::InvalidFfiOp.to_error_code();
129
+ }
130
+ };
131
+
132
+ // Create the ObjectFile
133
+ let window_size = 512 * 1024 * 1024;
134
+ let journal_file = match JournalFile::<Mmap>::open(path, window_size) {
135
+ Ok(f) => Box::new(f),
136
+ Err(e) => {
137
+ return e.to_error_code();
138
+ }
139
+ };
140
+
141
+ let journal = Box::new(RsdJournal {
142
+ reader: JournalReader::default(),
143
+ journal_file,
144
+ field_buffer: Vec::with_capacity(256),
145
+ decompressed_payload: Vec::new(),
146
+ });
147
+
148
+ // Pass ownership to the caller
149
+ *ret = Box::into_raw(journal);
150
+
151
+ 0
152
+}
153
+
154
+#[no_mangle]
155
+unsafe extern "C" fn rsd_journal_close(j: *mut RsdJournal) {
156
+ debug_assert!(!j.is_null());
157
+ let _ = Box::from_raw(j);
158
+}
159
+
160
+#[no_mangle]
161
+unsafe extern "C" fn rsd_journal_seek_head(j: *mut RsdJournal) -> c_int {
162
+ debug_assert!(!j.is_null());
163
+ let journal = &mut *j;
164
+ journal.reader.set_location(Location::Head);
165
+ 0
166
+}
167
+
168
+#[no_mangle]
169
+unsafe extern "C" fn rsd_journal_seek_tail(j: *mut RsdJournal) -> c_int {
170
+ debug_assert!(!j.is_null());
171
+ let journal = &mut *j;
172
+ journal.reader.set_location(Location::Tail);
173
+ 0
174
+}
175
+
176
+#[no_mangle]
177
+unsafe extern "C" fn rsd_journal_seek_realtime_usec(j: *mut RsdJournal, usec: u64) -> c_int {
178
+ debug_assert!(!j.is_null());
179
+ let journal = &mut *j;
180
+ journal.reader.set_location(Location::Realtime(usec));
181
+ 0
182
+}
183
+
184
+#[no_mangle]
185
+unsafe extern "C" fn rsd_journal_next(j: *mut RsdJournal) -> c_int {
186
+ debug_assert!(!j.is_null());
187
+ let journal = &mut *j;
188
+
189
+ match journal
190
+ .reader
191
+ .step(&journal.journal_file, Direction::Forward)
192
+ {
193
+ Ok(has_entry) => {
194
+ if has_entry {
195
+ 1
196
+ } else {
197
+ 0
198
+ }
199
+ }
200
+ Err(e) => e.to_error_code(),
201
+ }
202
+}
203
+
204
+#[no_mangle]
205
+unsafe extern "C" fn rsd_journal_previous(j: *mut RsdJournal) -> c_int {
206
+ debug_assert!(!j.is_null());
207
+ let journal = &mut *j;
208
+
209
+ match journal
210
+ .reader
211
+ .step(&journal.journal_file, Direction::Backward)
212
+ {
213
+ Ok(has_entry) => {
214
+ if has_entry {
215
+ 1
216
+ } else {
217
+ 0
218
+ }
219
+ }
220
+ Err(e) => e.to_error_code(),
221
+ }
222
+}
223
+
224
+#[no_mangle]
225
+unsafe extern "C" fn rsd_journal_get_seqnum(
226
+ j: *mut RsdJournal,
227
+ ret_seqnum: *mut u64,
228
+ ret_seqnum_id: *mut RsdId128,
229
+) -> c_int {
230
+ debug_assert!(!j.is_null());
231
+ debug_assert!(!ret_seqnum.is_null());
232
+ debug_assert!(!ret_seqnum_id.is_null());
233
+
234
+ let journal = &mut *j;
235
+ match journal.reader.get_seqnum(&journal.journal_file) {
236
+ Ok((seqnum, boot_id)) => {
237
+ *ret_seqnum = seqnum;
238
+
239
+ if !ret_seqnum_id.is_null() {
240
+ *ret_seqnum_id = RsdId128 { bytes: boot_id };
241
+ }
242
+
243
+ 0
244
+ }
245
+ Err(e) => e.to_error_code(),
246
+ }
247
+}
248
+
249
+#[no_mangle]
250
+unsafe extern "C" fn rsd_journal_get_realtime_usec(j: *mut RsdJournal, ret: *mut u64) -> c_int {
251
+ debug_assert!(!j.is_null());
252
+ debug_assert!(!ret.is_null());
253
+
254
+ let journal = &mut *j;
255
+
256
+ match journal.reader.get_realtime_usec(&journal.journal_file) {
257
+ Ok(realtime) => {
258
+ *ret = realtime;
259
+ 0
260
+ }
261
+ Err(e) => e.to_error_code(),
262
+ }
263
+}
264
+
265
+#[no_mangle]
266
+unsafe extern "C" fn rsd_journal_restart_data(j: *mut RsdJournal) {
267
+ debug_assert!(!j.is_null());
268
+
269
+ let journal = &mut *j;
270
+ journal.reader.entry_data_restart();
271
+}
272
+
273
+#[no_mangle]
274
+unsafe extern "C" fn rsd_journal_enumerate_available_data(
275
+ j: *mut RsdJournal,
276
+ data: *mut *const c_void,
277
+ l: *mut usize,
278
+) -> c_int {
279
+ debug_assert!(!j.is_null());
280
+ debug_assert!(!data.is_null());
281
+ debug_assert!(!l.is_null());
282
+
283
+ let journal = &mut *j;
284
+
285
+ match journal.reader.entry_data_enumerate(&journal.journal_file) {
286
+ Ok(Some(data_guard)) => {
287
+ if data_guard.is_compressed() {
288
+ return match data_guard.decompress(&mut journal.decompressed_payload) {
289
+ Ok(n) => {
290
+ *l = n;
291
+ *data = journal.decompressed_payload.as_ptr() as *const c_void;
292
+ 1
293
+ }
294
+ Err(error::JournalError::UnknownCompressionMethod) => {
295
+ eprintln!("unknown compression method");
296
+ -1
297
+ }
298
+ Err(_) => -1,
299
+ };
300
+ } else {
301
+ let payload = data_guard.payload_bytes();
302
+ *l = payload.len();
303
+ *data = payload.as_ptr() as *const c_void;
304
+ }
305
+ 1
306
+ }
307
+ Ok(None) => 0,
308
+ Err(e) => e.to_error_code(),
309
+ }
310
+}
311
+
312
+#[no_mangle]
313
+unsafe extern "C" fn rsd_journal_restart_fields(j: *mut RsdJournal) {
314
+ debug_assert!(!j.is_null());
315
+
316
+ let journal = &mut *j;
317
+ journal.reader.fields_restart();
318
+}
319
+
320
+#[no_mangle]
321
+unsafe extern "C" fn rsd_journal_enumerate_fields(
322
+ j: *mut RsdJournal,
323
+ field: *mut *const c_char,
324
+) -> c_int {
325
+ debug_assert!(!j.is_null());
326
+ debug_assert!(!field.is_null());
327
+
328
+ let journal = &mut *j;
329
+
330
+ match journal.reader.fields_enumerate(&journal.journal_file) {
331
+ Ok(Some(field_guard)) => {
332
+ let field_name = field_guard.get_payload();
333
+
334
+ journal.field_buffer.clear();
335
+ journal.field_buffer.extend_from_slice(field_name);
336
+ journal.field_buffer.push(0);
337
+ *field = journal.field_buffer.as_ptr() as *const c_char;
338
+
339
+ 1
340
+ }
341
+ Ok(None) => 0,
342
+ Err(e) => e.to_error_code(),
343
+ }
344
+}
345
+
346
+#[no_mangle]
347
+unsafe extern "C" fn rsd_journal_query_unique(j: *mut RsdJournal, field: *const c_char) -> c_int {
348
+ debug_assert!(!j.is_null());
349
+ debug_assert!(!field.is_null());
350
+
351
+ let journal = &mut *j;
352
+ let field_cstr = CStr::from_ptr(field);
353
+ let field_name = field_cstr.to_bytes();
354
+
355
+ match journal
356
+ .reader
357
+ .field_data_query_unique(&journal.journal_file, field_name)
358
+ {
359
+ Ok(_) => 0,
360
+ Err(e) => e.to_error_code(),
361
+ }
362
+}
363
+
364
+#[no_mangle]
365
+unsafe extern "C" fn rsd_journal_restart_unique(j: *mut RsdJournal) {
366
+ debug_assert!(!j.is_null());
367
+ let journal = &mut *j;
368
+ journal.reader.field_data_restart();
369
+}
370
+
371
+#[no_mangle]
372
+unsafe extern "C" fn rsd_journal_enumerate_available_unique(
373
+ j: *mut RsdJournal,
374
+ data: *mut *const c_void,
375
+ l: *mut usize,
376
+) -> c_int {
377
+ debug_assert!(!j.is_null());
378
+ debug_assert!(!data.is_null());
379
+ debug_assert!(!l.is_null());
380
+
381
+ let journal = &mut *j;
382
+
383
+ match journal.reader.field_data_enumerate(&journal.journal_file) {
384
+ Ok(Some(data_guard)) => {
385
+ let payload = data_guard.payload_bytes();
386
+ *data = payload.as_ptr() as *const c_void;
387
+ *l = payload.len();
388
+
389
+ 1
390
+ }
391
+ Ok(None) => 0,
392
+ Err(e) => e.to_error_code(),
393
+ }
394
+}
395
+
396
+#[no_mangle]
397
+unsafe extern "C" fn rsd_journal_add_match(
398
+ j: *mut RsdJournal,
399
+ data: *const c_void,
400
+ size: usize,
401
+) -> c_int {
402
+ debug_assert!(!j.is_null());
403
+ debug_assert!(!data.is_null());
404
+
405
+ let journal = &mut *j;
406
+
407
+ let data_slice = if size == 0 {
408
+ let mut len = 0;
409
+ let data_ptr = data as *const u8;
410
+ while *data_ptr.add(len) != 0 {
411
+ len += 1;
412
+ }
413
+ std::slice::from_raw_parts(data as *const u8, len)
414
+ } else {
415
+ std::slice::from_raw_parts(data as *const u8, size)
416
+ };
417
+
418
+ journal.reader.add_match(data_slice);
419
+ 0
420
+}
421
+
422
+#[no_mangle]
423
+unsafe extern "C" fn rsd_journal_add_conjunction(j: *mut RsdJournal) -> c_int {
424
+ debug_assert!(!j.is_null());
425
+ let journal = &mut *j;
426
+ match journal.reader.add_conjunction(&journal.journal_file) {
427
+ Ok(_) => 0,
428
+ Err(e) => e.to_error_code(),
429
+ }
430
+}
431
+
432
+#[no_mangle]
433
+unsafe extern "C" fn rsd_journal_add_disjunction(j: *mut RsdJournal) -> c_int {
434
+ debug_assert!(!j.is_null());
435
+
436
+ let journal = &mut *j;
437
+ match journal.reader.add_disjunction(&journal.journal_file) {
438
+ Ok(_) => 0,
439
+ Err(e) => e.to_error_code(),
440
+ }
441
+}
442
+
443
+#[no_mangle]
444
+unsafe extern "C" fn rsd_journal_flush_matches(j: *mut RsdJournal) {
445
+ debug_assert!(!j.is_null());
446
+ let journal = &mut *j;
447
+ journal.reader.flush_matches();
448
+}
src/crates/jf/otel-plugin/Cargo.toml
new
+31
@@ -0,0 +1,31 @@
1
+[package]
2
+name = "otel-plugin"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+tokio = { workspace = true }
9
+prost = { workspace = true }
10
+tonic = { workspace = true, features = ["gzip", "tls-ring"] }
11
+opentelemetry-proto = { workspace = true, features = ["metrics", "logs"] }
12
+
13
+flatten_otel = { path = "../flatten_otel" }
14
+journal_file = { path = "../journal_file" }
15
+journal_log = { path = "../journal_log" }
16
+memmap2 = { workspace = true }
17
+serde_json = { workspace = true, features = ["preserve_order"] }
18
+uuid = { version = "1.0", features = ["v4", "rng"] }
19
+flatten-serde-json = { workspace = true }
20
+base64 = "0.21"
21
+regex = { workspace = true }
22
+serde_yaml = "0.9.34"
23
+serde_regex = "1.1.0"
24
+serde = { version = "1.0.219", features = ["derive"] }
25
+atty = { workspace = true }
26
+clap = { workspace = true }
27
+anyhow = { workspace = true }
28
+humantime-serde = { workspace = true }
29
+humantime = { workspace = true }
30
+bytesize = { workspace = true }
31
+bytesize-serde = { workspace = true }
src/crates/jf/otel-plugin/README.md
new
+82
@@ -0,0 +1,82 @@
1
+# OpenTelemetry Metrics (otel.plugin)
2
+
3
+`otel.plugin` is a [Netdata](https://github.com/netdata/netdata) external plugin,
4
+enabling users to ingest, store and visualize OpenTelemetry metrics in charts.
5
+
6
+## Configuration
7
+
8
+Edit the [otel.yml](https://github.com/netdata/netdata/blob/master/src/crates/jf/otel-plugin/configs/otel.yml)
9
+configuration file using `edit-config` from the Netdata
10
+[config directory](/docs/netdata-agent/configuration/README.md#locate-your-config-directory),
11
+which is typically located under `/etc/netdata`.
12
+
13
+```bash
14
+cd /etc/netdata # Replace this path with your Netdata config directory
15
+sudo ./edit-config otel.yml
16
+```
17
+
18
+### gRPC Endpoint
19
+
20
+By default `otel.plugin` listens for incoming OTLP-formatted metrics on
21
+`localhost:4317` via gRPC. Users can set up a secure TLS connection by
22
+updating the TLS configuration in the `endpoint` section:
23
+
24
+```yaml
25
+endpoint:
26
+ # gRPC endpoint to listen on for OpenTelemetry data
27
+ path: "127.0.0.1:4317"
28
+
29
+ # Path to TLS certificate file (enables TLS when provided)
30
+ tls_cert_path: null
31
+
32
+ # Path to TLS private key file (required when TLS certificate is provided)
33
+ tls_key_path: null
34
+
35
+ # Path to TLS CA certificate file for client authentication (optional)
36
+ tls_ca_cert_path: null
37
+```
38
+
39
+### Metrics
40
+
41
+The `metrics` section allows users to specify the directory containing
42
+configuration files for mapping OpenTelemetry metrics to Netdata chart
43
+instances, and the number of metric samples the `otel.plugin` will use for
44
+detecting their collection interval:
45
+
46
+```yaml
47
+metrics:
48
+ # Directory with configuration files for mapping OTEL metrics to Netdata charts
49
+ # (relative paths are resolved based on Netdata's user configuration directory)
50
+ chart_configs_dir: otel.d/v1/metrics/
51
+
52
+ # Number of samples to buffer for collection interval detection
53
+ buffer_samples: 10
54
+```
55
+
56
+## Mapping OpenTelemetry metrics to Netdata chart instances
57
+
58
+Without an explicit mapping, the `otel.plugin` defaults to creating distinct
59
+chart instances based on the attributes of each data point in a metric. Users
60
+can place their YAML chart configuration files under `otel.d/v1/metrics` to
61
+override, or fine-tune, the default mapping.
62
+
63
+For each instrumentation scope and metric name, the configuration defines
64
+the attributes that the `otel.plugin` will use when creating new chart
65
+instances and dimension names.
66
+
67
+For example, the following bit from the
68
+[otel.d/v1/metrics/hostmetrics.yml](https://github.com/netdata/netdata/blob/master/src/crates/jf/otel-plugin/configs/otel.d/v1/metrics/hostmetrics-receiver.yml)
69
+ configuration file for the [hostmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/hostmetricsreceiver/internal/scraper/networkscraper/documentation.md) receiver:
70
+```yaml
71
+select:
72
+ instrumentation_scope_name: hostmetricsreceiver.*networkscraper
73
+ metric_name: system.network.connections
74
+extract:
75
+ chart_instance_pattern: metric.attributes.protocol
76
+ dimension_name: metric.attributes.state
77
+```
78
+will apply to metrics whose instrumentation scope and metric names match the
79
+corresponding regular expressions specified in the values of the
80
+`instrumentation_scope_name` and `metric_name` keys. Similarly, the values of
81
+the `protocol` and `state` attributes of each data point in the matched metric
82
+will be used to create a new chart instance with the proper dimension names.
src/crates/jf/otel-plugin/configs/otel.d/v1/metrics/hostmetrics-receiver.yml
new
+172
@@ -0,0 +1,172 @@
1
+configs:
2
+ - select:
3
+ instrumentation_scope_name: .*hostmetricsreceiver.*networkscraper$
4
+ metric_name: system\.network\.connections
5
+ extract:
6
+ chart_instance_pattern: metric.attributes.protocol
7
+ dimension_name: metric.attributes.state
8
+ - select:
9
+ instrumentation_scope_name: .*hostmetricsreceiver.*networkscraper$
10
+ metric_name: system\.network\.dropped
11
+ extract:
12
+ chart_instance_pattern: metric.attributes.device
13
+ dimension_name: metric.attributes.direction
14
+ - select:
15
+ instrumentation_scope_name: .*hostmetricsreceiver.*networkscraper$
16
+ metric_name: system\.network\.errors
17
+ extract:
18
+ chart_instance_pattern: metric.attributes.device
19
+ dimension_name: metric.attributes.direction
20
+ - select:
21
+ instrumentation_scope_name: .*hostmetricsreceiver.*networkscraper$
22
+ metric_name: system\.network\.io
23
+ extract:
24
+ chart_instance_pattern: metric.attributes.device
25
+ dimension_name: metric.attributes.direction
26
+ - select:
27
+ instrumentation_scope_name: .*hostmetricsreceiver.*networkscraper$
28
+ metric_name: system\.network\.packets
29
+ extract:
30
+ chart_instance_pattern: metric.attributes.device
31
+ dimension_name: metric.attributes.direction
32
+ - select:
33
+ instrumentation_scope_name: .*hostmetricsreceiver.*cpuscraper$
34
+ metric_name: system\.cpu\.time
35
+ extract:
36
+ chart_instance_pattern: metric.attributes.cpu
37
+ dimension_name: metric.attributes.state
38
+ - select:
39
+ instrumentation_scope_name: .*hostmetricsreceiver.*cpuscraper$
40
+ metric_name: system\.cpu\.frequency
41
+ extract:
42
+ chart_instance_pattern: metric.attributes.cpu
43
+ - select:
44
+ instrumentation_scope_name: .*hostmetricsreceiver.*cpuscraper$
45
+ metric_name: system\.cpu\.utilization
46
+ extract:
47
+ chart_instance_pattern: metric.attributes.cpu
48
+ dimension_name: metric.attributes.state
49
+ - select:
50
+ instrumentation_scope_name: .*hostmetricsreceiver.*diskscraper$
51
+ metric_name: system\.disk\.io$
52
+ extract:
53
+ chart_instance_pattern: metric.attributes.device
54
+ dimension_name: metric.attributes.direction
55
+ - select:
56
+ instrumentation_scope_name: .*hostmetricsreceiver.*diskscraper$
57
+ metric_name: system\.disk\.io_time
58
+ extract:
59
+ chart_instance_pattern: metric.attributes.device
60
+ - select:
61
+ instrumentation_scope_name: .*hostmetricsreceiver.*diskscraper$
62
+ metric_name: system\.disk\.merged
63
+ extract:
64
+ chart_instance_pattern: metric.attributes.device
65
+ dimension_name: metric.attributes.direction
66
+ - select:
67
+ instrumentation_scope_name: .*hostmetricsreceiver.*diskscraper$
68
+ metric_name: system\.disk\.operation_time
69
+ extract:
70
+ chart_instance_pattern: metric.attributes.device
71
+ dimension_name: metric.attributes.direction
72
+ - select:
73
+ instrumentation_scope_name: .*hostmetricsreceiver.*diskscraper$
74
+ metric_name: system\.disk\.operations
75
+ extract:
76
+ chart_instance_pattern: metric.attributes.device
77
+ dimension_name: metric.attributes.direction
78
+ - select:
79
+ instrumentation_scope_name: .*hostmetricsreceiver.*diskscraper$
80
+ metric_name: system\.disk\.pending_operations
81
+ extract:
82
+ chart_instance_pattern: metric.attributes.device
83
+ - select:
84
+ instrumentation_scope_name: .*hostmetricsreceiver.*diskscraper$
85
+ metric_name: system\.disk\.weighted_io
86
+ extract:
87
+ chart_instance_pattern: metric.attributes.device
88
+ - select:
89
+ instrumentation_scope_name: .*hostmetricsreceiver.*filesystemscraper$
90
+ metric_name: system\.filesystem\.inodes\.usage
91
+ extract:
92
+ chart_instance_pattern: metric.attributes.mountpoint
93
+ dimension_name: metric.attributes.state
94
+ - select:
95
+ instrumentation_scope_name: .*hostmetricsreceiver.*filesystemscraper$
96
+ metric_name: system\.filesystem\.usage
97
+ extract:
98
+ chart_instance_pattern: metric.attributes.mountpoint
99
+ dimension_name: metric.attributes.state
100
+ - select:
101
+ instrumentation_scope_name: .*hostmetricsreceiver.*filesystemscraper$
102
+ metric_name: system\.filesystem\.utilization
103
+ extract:
104
+ chart_instance_pattern: metric.attributes.mountpoint
105
+ - select:
106
+ instrumentation_scope_name: .*hostmetricsreceiver.*memoryscraper$
107
+ metric_name: system\.memory\.utilization
108
+ extract:
109
+ dimension_name: metric.attributes.state
110
+ - select:
111
+ instrumentation_scope_name: .*hostmetricsreceiver.*pagingscraper$
112
+ metric_name: system\.paging\.faults
113
+ extract:
114
+ dimension_name: metric.attributes.type
115
+ - select:
116
+ instrumentation_scope_name: .*hostmetricsreceiver.*pagingscraper$
117
+ metric_name: system\.paging\.operations
118
+ extract:
119
+ chart_instance_pattern: metric.attributes.type
120
+ dimension_name: metric.attributes.direction
121
+ - select:
122
+ instrumentation_scope_name: .*hostmetricsreceiver.*pagingscraper$
123
+ metric_name: system\.paging\.usage
124
+ extract:
125
+ chart_instance_pattern: metric.attributes.device
126
+ dimension_name: metric.attributes.state
127
+ - select:
128
+ instrumentation_scope_name: .*hostmetricsreceiver.*pagingscraper$
129
+ metric_name: system\.paging\.utilization
130
+ extract:
131
+ chart_instance_pattern: metric.attributes.device
132
+ dimension_name: metric.attributes.state
133
+ - select:
134
+ instrumentation_scope_name: .*hostmetricsreceiver.*processesscraper$
135
+ metric_name: system\.processes\.count
136
+ extract:
137
+ dimension_name: metric.attributes.status
138
+ - select:
139
+ instrumentation_scope_name: .*hostmetricsreceiver.*processscraper$
140
+ metric_name: process\.cpu\.time
141
+ extract:
142
+ dimension_name: metric.attributes.state
143
+ - select:
144
+ instrumentation_scope_name: .*hostmetricsreceiver.*processscraper$
145
+ metric_name: process\.disk\.io
146
+ extract:
147
+ dimension_name: metric.attributes.direction
148
+ - select:
149
+ instrumentation_scope_name: .*hostmetricsreceiver.*processscraper$
150
+ metric_name: process\.context_switches
151
+ extract:
152
+ dimension_name: metric.attributes.type
153
+ - select:
154
+ instrumentation_scope_name: .*hostmetricsreceiver.*processscraper$
155
+ metric_name: process\.cpu\.utilization
156
+ extract:
157
+ dimension_name: metric.attributes.state
158
+ - select:
159
+ instrumentation_scope_name: .*hostmetricsreceiver.*processscraper$
160
+ metric_name: process\.disk\.operations
161
+ extract:
162
+ dimension_name: metric.attributes.direction
163
+ - select:
164
+ instrumentation_scope_name: .*hostmetricsreceiver.*processscraper$
165
+ metric_name: process\.paging\.faults
166
+ extract:
167
+ dimension_name: metric.attributes.type
168
+ - select:
169
+ instrumentation_scope_name: .*hostmetricsreceiver.*processscraper$
170
+ metric_name: process\.paging\.faults
171
+ extract:
172
+ dimension_name: metric.attributes.type
src/crates/jf/otel-plugin/configs/otel.yml
renamed
-4
@@ -47,7 +47,3 @@ logs:
47
48
# Maximum time span that entries in a single journal file can cover (e.g., "2 hours", "1h", "30m").
49
duration_of_journal_file: "2 hours"
50
-
51
- # Store the complete OTLP JSON representation in the OTLP_JSON field.
52
- # When enabled, each log entry includes the full original JSON message for debugging and reprocessing.
53
- store_otlp_json: false
src/crates/jf/otel-plugin/src/chart_config.rs
new
+171
@@ -0,0 +1,171 @@
1
+use anyhow::{Context, Result};
2
+use regex::Regex;
3
+use serde::{Deserialize, Serialize};
4
+use serde_json::{Map as JsonMap, Value as JsonValue};
5
+use std::fs;
6
+use std::path::Path;
7
+
8
+#[derive(Debug, Clone, Serialize, Deserialize)]
9
+pub struct SelectCriteria {
10
+ #[serde(with = "serde_regex", skip_serializing_if = "Option::is_none", default)]
11
+ pub instrumentation_scope_name: Option<Regex>,
12
+
13
+ #[serde(with = "serde_regex", skip_serializing_if = "Option::is_none", default)]
14
+ pub instrumentation_scope_version: Option<Regex>,
15
+
16
+ #[serde(with = "serde_regex")]
17
+ pub metric_name: Regex,
18
+}
19
+
20
+#[derive(Debug, Clone, Serialize, Deserialize)]
21
+pub struct ExtractPattern {
22
+ #[serde(skip_serializing_if = "Option::is_none")]
23
+ pub chart_instance_pattern: Option<String>,
24
+
25
+ #[serde(skip_serializing_if = "Option::is_none")]
26
+ pub dimension_name: Option<String>,
27
+}
28
+
29
+#[derive(Debug, Clone, Serialize, Deserialize)]
30
+pub struct ChartConfig {
31
+ pub select: SelectCriteria,
32
+ pub extract: ExtractPattern,
33
+}
34
+
35
+impl ChartConfig {
36
+ pub fn matches(&self, json_map: &JsonMap<String, JsonValue>) -> bool {
37
+ if let Some(scope_regex) = &self.select.instrumentation_scope_name {
38
+ if let Some(JsonValue::String(scope_name)) = json_map.get("scope.name") {
39
+ if !scope_regex.is_match(scope_name) {
40
+ return false;
41
+ }
42
+ } else {
43
+ return false;
44
+ }
45
+ }
46
+
47
+ if let Some(version_regex) = &self.select.instrumentation_scope_version {
48
+ if let Some(JsonValue::String(scope_version)) = json_map.get("scope.version") {
49
+ if !version_regex.is_match(scope_version) {
50
+ return false;
51
+ }
52
+ } else {
53
+ return false;
54
+ }
55
+ }
56
+
57
+ if let Some(JsonValue::String(metric_name)) = json_map.get("metric.name") {
58
+ self.select.metric_name.is_match(metric_name)
59
+ } else {
60
+ false
61
+ }
62
+ }
63
+}
64
+
65
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
66
+pub struct ChartConfigs {
67
+ configs: Vec<ChartConfig>,
68
+}
69
+
70
+#[derive(Debug, Default, Clone)]
71
+pub struct ChartConfigManager {
72
+ stock: ChartConfigs,
73
+ user: ChartConfigs,
74
+}
75
+
76
+impl ChartConfigManager {
77
+ pub fn with_default_configs() -> Self {
78
+ let mut manager = Self::default();
79
+ manager.load_stock_config();
80
+ manager
81
+ }
82
+
83
+ pub fn find_matching_config(
84
+ &self,
85
+ json_map: &JsonMap<String, JsonValue>,
86
+ ) -> Option<&ChartConfig> {
87
+ // Chaining-order is important. We want to priority user configurations
88
+ // and fall back to stock configurations if they are missing.
89
+ self.user
90
+ .configs
91
+ .iter()
92
+ .chain(self.stock.configs.iter())
93
+ .find(|config| config.matches(json_map))
94
+ }
95
+
96
+ fn load_stock_config(&mut self) {
97
+ const DEFAULT_CONFIGS_YAML: &str =
98
+ include_str!("../configs/otel.d/v1/metrics/hostmetrics-receiver.yml");
99
+
100
+ match serde_yaml::from_str::<ChartConfigs>(DEFAULT_CONFIGS_YAML) {
101
+ Ok(configs) => {
102
+ self.stock = configs;
103
+ }
104
+ Err(e) => {
105
+ eprintln!("Failed to parse default configs YAML: {}", e);
106
+ }
107
+ }
108
+ }
109
+
110
+ pub fn load_user_configs<P: AsRef<Path>>(&mut self, config_dir: P) -> Result<()> {
111
+ // check dir
112
+ let config_path = config_dir.as_ref();
113
+ if !config_path.exists() {
114
+ return Err(anyhow::anyhow!(
115
+ "Configuration directory does not exist: {}",
116
+ config_path.display()
117
+ ));
118
+ }
119
+ if !config_path.is_dir() {
120
+ return Err(anyhow::anyhow!(
121
+ "Configuration path is not a directory: {}",
122
+ config_path.display()
123
+ ));
124
+ }
125
+
126
+ // collect the yaml files
127
+ let mut config_files: Vec<_> = std::fs::read_dir(config_path)
128
+ .with_context(|| {
129
+ format!(
130
+ "Failed to read chart config directory: {}",
131
+ config_path.display()
132
+ )
133
+ })?
134
+ .filter_map(|entry| {
135
+ let entry = entry.ok()?;
136
+ let path = entry.path();
137
+ if path.is_file()
138
+ && matches!(
139
+ path.extension().and_then(|s| s.to_str()),
140
+ Some("yml" | "yaml")
141
+ )
142
+ {
143
+ Some(path)
144
+ } else {
145
+ None
146
+ }
147
+ })
148
+ .collect();
149
+ config_files.sort();
150
+
151
+ // deserialize them
152
+ self.user = ChartConfigs::default();
153
+ for path in config_files {
154
+ match fs::read_to_string(&path) {
155
+ Ok(contents) => match serde_yaml::from_str::<ChartConfigs>(&contents) {
156
+ Ok(chart_configs) => {
157
+ self.user.configs.extend(chart_configs.configs);
158
+ }
159
+ Err(e) => {
160
+ eprintln!("Failed to parse YAML file {}: {}", path.display(), e);
161
+ }
162
+ },
163
+ Err(e) => {
164
+ eprintln!("Failed to read file {}: {}", path.display(), e);
165
+ }
166
+ }
167
+ }
168
+
169
+ Ok(())
170
+ }
171
+}
src/crates/jf/otel-plugin/src/flattened_point.rs
new
+171
@@ -0,0 +1,171 @@
1
+use serde_json::{Map as JsonMap, Value as JsonValue};
2
+use std::hash::{Hash, Hasher};
3
+
4
+use crate::regex_cache::RegexCache;
5
+
6
+#[derive(Default, Debug)]
7
+pub struct FlattenedPoint {
8
+ pub attributes: JsonMap<String, JsonValue>,
9
+
10
+ pub nd_instance_name: String,
11
+ pub nd_dimension_name: String,
12
+
13
+ pub metric_name: String,
14
+ pub metric_description: String,
15
+ pub metric_unit: String,
16
+ pub metric_type: String,
17
+
18
+ pub metric_time_unix_nano: u64,
19
+ pub metric_value: f64,
20
+
21
+ pub metric_is_monotonic: Option<bool>,
22
+}
23
+
24
+use crate::chart_config::ChartConfig;
25
+
26
+impl FlattenedPoint {
27
+ pub fn new(
28
+ mut json_map: JsonMap<String, JsonValue>,
29
+ chart_config: Option<&ChartConfig>,
30
+ regex_cache: &RegexCache,
31
+ ) -> Option<Self> {
32
+ let Some(JsonValue::String(metric_name)) = json_map.remove("metric.name") else {
33
+ debug_assert!(false, "metric.name missing from json map");
34
+ return None;
35
+ };
36
+
37
+ let Some(JsonValue::String(metric_description)) = json_map.remove("metric.description")
38
+ else {
39
+ debug_assert!(false, "metric.description missing from json map");
40
+ return None;
41
+ };
42
+
43
+ let Some(JsonValue::String(metric_unit)) = json_map.remove("metric.unit") else {
44
+ debug_assert!(false, "metric.unit missing from json map");
45
+ return None;
46
+ };
47
+
48
+ let Some(JsonValue::String(metric_type)) = json_map.remove("metric.type") else {
49
+ debug_assert!(false, "metric.type missing from json map");
50
+ return None;
51
+ };
52
+
53
+ // Ignore start_time_unix for the time being.
54
+ json_map.remove("metric.start_time_unix_nano");
55
+
56
+ let Some(metric_time_unix_nano) = json_map
57
+ .remove("metric.time_unix_nano")
58
+ .and_then(|v| v.as_u64())
59
+ else {
60
+ debug_assert!(false, "metric.time_unix_nano missing from json map");
61
+ return None;
62
+ };
63
+
64
+ let Some(metric_value) = json_map.remove("metric.value").and_then(|v| v.as_f64()) else {
65
+ debug_assert!(false, "metric.value missing from json map");
66
+ return None;
67
+ };
68
+
69
+ let metric_is_monotonic = json_map
70
+ .remove("metric.is_monotonic")
71
+ .and_then(|v| v.as_bool());
72
+
73
+ if let Some(config) = chart_config {
74
+ if let Some(chart_instance_pattern) = &config.extract.chart_instance_pattern {
75
+ if !json_map.contains_key("metric.attributes._nd_chart_instance") {
76
+ json_map.insert(
77
+ "metric.attributes._nd_chart_instance".to_string(),
78
+ JsonValue::String(chart_instance_pattern.clone()),
79
+ );
80
+ }
81
+ }
82
+
83
+ if let Some(dimension_name) = &config.extract.dimension_name {
84
+ if !json_map.contains_key("metric.attributes._nd_dimension") {
85
+ json_map.insert(
86
+ "metric.attributes._nd_dimension".to_string(),
87
+ JsonValue::String(dimension_name.clone()),
88
+ );
89
+ }
90
+ }
91
+ }
92
+
93
+ let nd_dimension_name = {
94
+ let nd_dimension_key = json_map
95
+ .remove("metric.attributes._nd_dimension")
96
+ .and_then(|v| v.as_str().map(String::from));
97
+
98
+ if let Some(key) = nd_dimension_key {
99
+ match json_map.remove(&key) {
100
+ Some(JsonValue::String(s)) => s.clone(),
101
+ Some(JsonValue::Number(n)) => n.to_string(),
102
+ Some(JsonValue::Bool(b)) => b.to_string(),
103
+ Some(value) => {
104
+ eprintln!("Only strings/number/bool values can be used for dimension name >>>{:#?}<<<", value);
105
+ return None;
106
+ }
107
+ _ => {
108
+ eprintln!(
109
+ "Dimension key >>>{:?}<<< not found in flattened representation.",
110
+ key
111
+ );
112
+ return None;
113
+ }
114
+ }
115
+ } else {
116
+ String::from("value")
117
+ }
118
+ };
119
+
120
+ let nd_instance_name = {
121
+ let nd_chart_instance = json_map
122
+ .remove("metric.attributes._nd_chart_instance")
123
+ .and_then(|v| v.as_str().map(String::from))
124
+ .and_then(|s| regex_cache.get(&s).ok());
125
+
126
+ let mut matched_values = vec![metric_name.clone()];
127
+ if let Some(pattern) = nd_chart_instance {
128
+ for (key, value) in &json_map {
129
+ if pattern.is_match(key) {
130
+ let value_str = match value {
131
+ JsonValue::String(s) => s.clone(),
132
+ JsonValue::Number(n) => n.to_string(),
133
+ JsonValue::Bool(b) => b.to_string(),
134
+ JsonValue::Null => "null".to_string(),
135
+ _ => serde_json::to_string(value).unwrap_or_default(),
136
+ };
137
+ matched_values.push(value_str);
138
+ }
139
+ }
140
+ }
141
+
142
+ let name = matched_values.join(".");
143
+
144
+ let hash = {
145
+ use std::hash::DefaultHasher;
146
+
147
+ let mut state = DefaultHasher::new();
148
+ name.hash(&mut state);
149
+ json_map.hash(&mut state);
150
+ metric_unit.hash(&mut state);
151
+ metric_type.hash(&mut state);
152
+ state.finish()
153
+ };
154
+
155
+ format!("{name}.{hash:016x}")
156
+ };
157
+
158
+ Some(Self {
159
+ attributes: json_map,
160
+ nd_instance_name,
161
+ nd_dimension_name,
162
+ metric_name,
163
+ metric_description: metric_description.replace('\'', "\""),
164
+ metric_unit,
165
+ metric_type,
166
+ metric_time_unix_nano,
167
+ metric_value,
168
+ metric_is_monotonic,
169
+ })
170
+ }
171
+}
src/crates/jf/otel-plugin/src/logs_service.rs
new
+99
@@ -0,0 +1,99 @@
1
+use anyhow::{Context, Result};
2
+use flatten_otel::json_from_export_logs_service_request;
3
+use journal_log::{JournalLog, JournalLogConfig, RetentionPolicy, RotationPolicy};
4
+use opentelemetry_proto::tonic::collector::logs::v1::{
5
+ logs_service_server::LogsService, ExportLogsServiceRequest, ExportLogsServiceResponse,
6
+};
7
+use serde_json::Value;
8
+use std::sync::{Arc, Mutex};
9
+use tonic::{Request, Response, Status};
10
+
11
+use crate::plugin_config::PluginConfig;
12
+
13
+pub struct NetdataLogsService {
14
+ journal_log: Arc<Mutex<JournalLog>>,
15
+}
16
+
17
+impl NetdataLogsService {
18
+ pub fn new(plugin_config: PluginConfig) -> Result<Self> {
19
+ let logs_config = plugin_config.logs;
20
+
21
+ let rotation_policy = RotationPolicy::default()
22
+ .with_size_of_journal_file(logs_config.size_of_journal_file.as_u64())
23
+ .with_duration_of_journal_file(logs_config.duration_of_journal_file);
24
+
25
+ let retention_policy = RetentionPolicy::default()
26
+ .with_number_of_journal_files(logs_config.number_of_journal_files)
27
+ .with_size_of_journal_files(logs_config.size_of_journal_files.as_u64())
28
+ .with_duration_of_journal_files(logs_config.duration_of_journal_files);
29
+
30
+ let journal_config = JournalLogConfig::new(&logs_config.journal_dir)
31
+ .with_rotation_policy(rotation_policy)
32
+ .with_retention_policy(retention_policy);
33
+
34
+ let journal_log = Arc::new(Mutex::new(JournalLog::new(journal_config).with_context(
35
+ || {
36
+ format!(
37
+ "Failed to create journal log for directory: {}",
38
+ logs_config.journal_dir
39
+ )
40
+ },
41
+ )?));
42
+ Ok(NetdataLogsService { journal_log })
43
+ }
44
+
45
+ fn json_to_entry_data(&self, json_value: &Value) -> Vec<Vec<u8>> {
46
+ let mut entry_data = Vec::new();
47
+
48
+ if let Value::Object(obj) = json_value {
49
+ for (key, value) in obj {
50
+ let value_str = match value {
51
+ Value::String(s) => s.clone(),
52
+ Value::Number(n) => n.to_string(),
53
+ Value::Bool(b) => b.to_string(),
54
+ Value::Null => "null".to_string(),
55
+ _ => serde_json::to_string(value).unwrap_or_default(),
56
+ };
57
+
58
+ let kv_pair = format!("{}={}", key, value_str);
59
+ entry_data.push(kv_pair.into_bytes());
60
+ }
61
+ }
62
+
63
+ entry_data
64
+ }
65
+}
66
+
67
+#[tonic::async_trait]
68
+impl LogsService for NetdataLogsService {
69
+ async fn export(
70
+ &self,
71
+ request: Request<ExportLogsServiceRequest>,
72
+ ) -> Result<Response<ExportLogsServiceResponse>, Status> {
73
+ let req = request.into_inner();
74
+
75
+ let json_array = json_from_export_logs_service_request(&req);
76
+
77
+ if let Value::Array(entries) = json_array {
78
+ for entry in entries {
79
+ let entry_data = self.json_to_entry_data(&entry);
80
+ if !entry_data.is_empty() {
81
+ let entry_refs: Vec<&[u8]> = entry_data.iter().map(|v| v.as_slice()).collect();
82
+ if let Err(e) = self.journal_log.lock().unwrap().write_entry(&entry_refs) {
83
+ eprintln!("Failed to write log entry: {}", e);
84
+ return Err(Status::internal(format!(
85
+ "Failed to write log entry: {}",
86
+ e
87
+ )));
88
+ }
89
+ }
90
+ }
91
+ }
92
+
93
+ let reply = ExportLogsServiceResponse {
94
+ partial_success: None,
95
+ };
96
+
97
+ Ok(Response::new(reply))
98
+ }
99
+}
src/crates/jf/otel-plugin/src/main.rs
new
+104
@@ -0,0 +1,104 @@
1
+use anyhow::{Context, Result};
2
+use opentelemetry_proto::tonic::collector::{
3
+ logs::v1::logs_service_server::LogsServiceServer,
4
+ metrics::v1::metrics_service_server::MetricsServiceServer,
5
+};
6
+use tonic::transport::{Identity, Server, ServerTlsConfig};
7
+
8
+mod chart_config;
9
+mod flattened_point;
10
+mod netdata_chart;
11
+mod netdata_env;
12
+mod regex_cache;
13
+mod samples_table;
14
+
15
+mod plugin_config;
16
+use crate::plugin_config::PluginConfig;
17
+
18
+mod logs_service;
19
+use crate::logs_service::NetdataLogsService;
20
+
21
+mod metrics_service;
22
+use crate::metrics_service::NetdataMetricsService;
23
+
24
+async fn send_keepalive_periodically() {
25
+ let mut interval = tokio::time::interval(tokio::time::Duration::from_secs(60));
26
+
27
+ loop {
28
+ interval.tick().await;
29
+ println!("PLUGIN_KEEPALIVE");
30
+ }
31
+}
32
+
33
+#[tokio::main]
34
+async fn main() -> Result<()> {
35
+ let config = PluginConfig::new().context("Failed to initialize plugin configuration")?;
36
+
37
+ let addr =
38
+ config.endpoint.path.parse().with_context(|| {
39
+ format!("Failed to parse endpoint address: {}", config.endpoint.path)
40
+ })?;
41
+ let metrics_service =
42
+ NetdataMetricsService::new(config.clone()).context("Failed to create metrics service")?;
43
+ let logs_service =
44
+ NetdataLogsService::new(config.clone()).context("Failed to create logs service")?;
45
+
46
+ println!("TRUST_DURATIONS 1");
47
+
48
+ let mut server_builder = Server::builder();
49
+
50
+ // Configure TLS if provided
51
+ if let (Some(cert_path), Some(key_path)) = (
52
+ &config.endpoint.tls_cert_path,
53
+ &config.endpoint.tls_key_path,
54
+ ) {
55
+ let cert = std::fs::read(cert_path)
56
+ .with_context(|| format!("Failed to read TLS certificate from: {}", cert_path))?;
57
+ let key = std::fs::read(key_path)
58
+ .with_context(|| format!("Failed to read TLS private key from: {}", key_path))?;
59
+ let identity = Identity::from_pem(cert, key);
60
+
61
+ let mut tls_config_builder = ServerTlsConfig::new().identity(identity);
62
+
63
+ // If CA certificate is provided, enable client authentication
64
+ if let Some(ref ca_cert_path) = config.endpoint.tls_ca_cert_path {
65
+ let ca_cert = std::fs::read(ca_cert_path)
66
+ .with_context(|| format!("Failed to read CA certificate from: {}", ca_cert_path))?;
67
+ tls_config_builder =
68
+ tls_config_builder.client_ca_root(tonic::transport::Certificate::from_pem(ca_cert));
69
+ }
70
+
71
+ server_builder = server_builder
72
+ .tls_config(tls_config_builder)
73
+ .context("Failed to configure TLS")?;
74
+ } else {
75
+ eprintln!(
76
+ "TLS disabled, using insecure connection on endpoint: {}",
77
+ config.endpoint.path
78
+ );
79
+ }
80
+
81
+ let server = server_builder
82
+ .add_service(
83
+ MetricsServiceServer::new(metrics_service)
84
+ .accept_compressed(tonic::codec::CompressionEncoding::Gzip),
85
+ )
86
+ .add_service(
87
+ LogsServiceServer::new(logs_service)
88
+ .accept_compressed(tonic::codec::CompressionEncoding::Gzip),
89
+ )
90
+ .serve(addr);
91
+
92
+ let keepalive = send_keepalive_periodically();
93
+
94
+ tokio::select! {
95
+ result = server => {
96
+ result.with_context(|| format!("Failed to serve gRPC server on {}", addr))?;
97
+ }
98
+ _ = keepalive => {
99
+ // This should never complete
100
+ }
101
+ }
102
+
103
+ Ok(())
104
+}
src/crates/jf/otel-plugin/src/metrics_service.rs
new
+140
@@ -0,0 +1,140 @@
1
+use anyhow::{Context, Result};
2
+use flatten_otel::flatten_metrics_request;
3
+use opentelemetry_proto::tonic::collector::metrics::v1::{
4
+ metrics_service_server::MetricsService, ExportMetricsServiceRequest,
5
+ ExportMetricsServiceResponse,
6
+};
7
+use std::collections::HashMap;
8
+use std::sync::Arc;
9
+use tokio::sync::RwLock;
10
+use tonic::{Request, Response, Status};
11
+
12
+use crate::chart_config::ChartConfigManager;
13
+use crate::flattened_point::FlattenedPoint;
14
+use crate::netdata_chart::NetdataChart;
15
+use crate::plugin_config::PluginConfig;
16
+use crate::regex_cache::RegexCache;
17
+
18
+#[derive(Default)]
19
+pub struct NetdataMetricsService {
20
+ regex_cache: RegexCache,
21
+ charts: Arc<RwLock<HashMap<String, NetdataChart>>>,
22
+ config: Arc<PluginConfig>,
23
+ chart_config_manager: ChartConfigManager,
24
+ call_count: std::sync::atomic::AtomicU64,
25
+}
26
+
27
+impl NetdataMetricsService {
28
+ pub fn new(config: PluginConfig) -> Result<Self> {
29
+ let mut chart_config_manager = ChartConfigManager::with_default_configs();
30
+
31
+ // Load user chart configs if directory is specified
32
+ if let Some(chart_configs_dir) = &config.metrics.chart_configs_dir {
33
+ chart_config_manager
34
+ .load_user_configs(chart_configs_dir)
35
+ .with_context(|| {
36
+ format!(
37
+ "Failed to load chart configs from directory: {}",
38
+ chart_configs_dir
39
+ )
40
+ })?;
41
+ }
42
+
43
+ Ok(Self {
44
+ regex_cache: RegexCache::default(),
45
+ charts: Arc::default(),
46
+ config: Arc::new(config),
47
+ chart_config_manager,
48
+ call_count: std::sync::atomic::AtomicU64::new(0),
49
+ })
50
+ }
51
+
52
+ async fn cleanup_stale_charts(&self, max_age: std::time::Duration) {
53
+ let now = std::time::SystemTime::now();
54
+
55
+ let mut guard = self.charts.write().await;
56
+ guard.retain(|_, chart| {
57
+ let Some(chart_time) = chart.last_collection_time() else {
58
+ return true;
59
+ };
60
+
61
+ now.duration_since(chart_time)
62
+ .unwrap_or(std::time::Duration::ZERO)
63
+ < max_age
64
+ });
65
+ }
66
+}
67
+
68
+#[tonic::async_trait]
69
+impl MetricsService for NetdataMetricsService {
70
+ async fn export(
71
+ &self,
72
+ request: Request<ExportMetricsServiceRequest>,
73
+ ) -> Result<Response<ExportMetricsServiceResponse>, Status> {
74
+ let req = request.into_inner();
75
+
76
+ let flattened_points = flatten_metrics_request(&req)
77
+ .into_iter()
78
+ .filter_map(|jm| {
79
+ let cfg = self.chart_config_manager.find_matching_config(&jm);
80
+ FlattenedPoint::new(jm, cfg, &self.regex_cache)
81
+ })
82
+ .collect::<Vec<_>>();
83
+
84
+ if self.config.metrics.print_flattened {
85
+ // Just print the flattened points
86
+ for fp in &flattened_points {
87
+ println!("{:#?}", fp);
88
+ }
89
+
90
+ return Ok(Response::new(ExportMetricsServiceResponse {
91
+ partial_success: None,
92
+ }));
93
+ }
94
+
95
+ // ingest
96
+ {
97
+ let mut newly_created_charts = 0;
98
+
99
+ for fp in flattened_points.iter() {
100
+ let mut guard = self.charts.write().await;
101
+
102
+ if let Some(netdata_chart) = guard.get_mut(&fp.nd_instance_name) {
103
+ netdata_chart.ingest(fp);
104
+ } else if newly_created_charts < self.config.metrics.throttle_charts {
105
+ let mut netdata_chart =
106
+ NetdataChart::from_flattened_point(fp, self.config.metrics.buffer_samples);
107
+ netdata_chart.ingest(fp);
108
+ guard.insert(fp.nd_instance_name.clone(), netdata_chart);
109
+
110
+ newly_created_charts += 1;
111
+ }
112
+ }
113
+ }
114
+
115
+ // process
116
+ {
117
+ let mut guard = self.charts.write().await;
118
+
119
+ for netdata_chart in guard.values_mut() {
120
+ netdata_chart.process();
121
+ }
122
+ }
123
+
124
+ // cleanup stale charts
125
+ {
126
+ let prev_count = self
127
+ .call_count
128
+ .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
129
+
130
+ if prev_count % 60 == 0 {
131
+ let one_hour = std::time::Duration::from_secs(3600);
132
+ self.cleanup_stale_charts(one_hour).await;
133
+ }
134
+ }
135
+
136
+ Ok(Response::new(ExportMetricsServiceResponse {
137
+ partial_success: None,
138
+ }))
139
+ }
140
+}
src/crates/jf/otel-plugin/src/netdata_chart.rs
new
+296
@@ -0,0 +1,296 @@
1
+use serde_json::{Map as JsonMap, Value as JsonValue};
2
+
3
+use crate::flattened_point::FlattenedPoint;
4
+use crate::samples_table::{CollectionInterval, SamplesTable};
5
+
6
+#[derive(Debug, Default, Clone)]
7
+enum ChartState {
8
+ #[default]
9
+ Uninitialized,
10
+ InGap,
11
+ Initialized,
12
+ Empty,
13
+}
14
+
15
+#[derive(Debug)]
16
+pub struct NetdataChart {
17
+ chart_id: String,
18
+ metric_name: String,
19
+ metric_description: String,
20
+ metric_unit: String,
21
+ metric_type: String,
22
+ is_monotonic: Option<bool>,
23
+ attributes: JsonMap<String, JsonValue>,
24
+
25
+ samples_table: SamplesTable,
26
+ last_samples_table_interval: Option<CollectionInterval>,
27
+ last_collection_interval: Option<CollectionInterval>,
28
+ chart_state: ChartState,
29
+ samples_threshold: usize,
30
+
31
+ multiplier: i32,
32
+ divisor: i32,
33
+}
34
+
35
+impl NetdataChart {
36
+ pub fn from_flattened_point(fp: &FlattenedPoint, samples_threshold: usize) -> Self {
37
+ Self {
38
+ chart_id: fp.nd_instance_name.clone(),
39
+ metric_name: fp.metric_name.clone(),
40
+ metric_description: fp.metric_description.clone(),
41
+ metric_unit: fp.metric_unit.clone(),
42
+ metric_type: fp.metric_type.clone(),
43
+ attributes: fp.attributes.clone(),
44
+ is_monotonic: fp.metric_is_monotonic,
45
+
46
+ samples_table: SamplesTable::default(),
47
+ last_samples_table_interval: None,
48
+ last_collection_interval: None,
49
+ chart_state: ChartState::Uninitialized,
50
+
51
+ samples_threshold,
52
+
53
+ multiplier: 1,
54
+ divisor: 1,
55
+ }
56
+ }
57
+
58
+ fn is_histogram(&self) -> bool {
59
+ self.metric_type == "histogram"
60
+ }
61
+
62
+ pub fn ingest(&mut self, fp: &FlattenedPoint) {
63
+ let dimension_name = &fp.nd_dimension_name;
64
+ let value = fp.metric_value;
65
+ let unix_time = fp.metric_time_unix_nano;
66
+
67
+ let new_dimension = self.samples_table.insert(dimension_name, unix_time, value);
68
+
69
+ if new_dimension {
70
+ self.chart_state = ChartState::Uninitialized;
71
+ self.last_samples_table_interval = None;
72
+ self.last_collection_interval = None;
73
+ }
74
+ }
75
+
76
+ fn initialize(&mut self) -> bool {
77
+ // Clean up stale samples if we have a previous interval
78
+ if let Some(ci) = &self.last_samples_table_interval {
79
+ self.samples_table.drop_stale_samples(ci);
80
+ }
81
+
82
+ // Check if we have enough samples to determine frequency
83
+ if self.samples_table.total_samples() < self.samples_threshold {
84
+ return false;
85
+ }
86
+
87
+ // Store the old interval before calculating the new one
88
+ let old_lci = self.last_collection_interval;
89
+
90
+ // Set up collection intervals
91
+ self.last_samples_table_interval =
92
+ self.samples_table
93
+ .collection_interval()
94
+ .map(|ci| CollectionInterval {
95
+ end_time: ci.end_time - ci.update_every.get(),
96
+ update_every: ci.update_every,
97
+ });
98
+
99
+ self.last_collection_interval = self
100
+ .last_samples_table_interval
101
+ .and_then(|ci| ci.aligned_interval());
102
+
103
+ (self.multiplier, self.divisor) = self.samples_table.scaling_factors();
104
+
105
+ // Check if we need to emit a chart definition
106
+ if let Some(new_lci) = &self.last_collection_interval {
107
+ if let Some(old_lci) = old_lci {
108
+ if old_lci.update_every != new_lci.update_every {
109
+ // Update every changed, emit the chart definition again
110
+ self.emit_chart_definition();
111
+ }
112
+ } else {
113
+ // No previous collection interval, we need to emit the
114
+ // chart definition first
115
+ self.emit_chart_definition();
116
+ }
117
+ }
118
+
119
+ true
120
+ }
121
+
122
+ pub fn process(&mut self) {
123
+ loop {
124
+ match &self.chart_state {
125
+ ChartState::Uninitialized | ChartState::InGap => {
126
+ if !self.initialize() {
127
+ return;
128
+ }
129
+
130
+ self.chart_state = ChartState::Initialized;
131
+ }
132
+ ChartState::Initialized => {
133
+ self.chart_state = self.process_next_interval();
134
+ }
135
+ ChartState::Empty => {
136
+ self.chart_state = ChartState::Initialized;
137
+ return;
138
+ }
139
+ }
140
+ }
141
+ }
142
+
143
+ fn emit_chart_definition(&self) {
144
+ let ci = self.last_collection_interval.unwrap();
145
+ let ue = ci.update_every;
146
+
147
+ let type_id = &self.chart_id;
148
+ let name = "";
149
+ let title = &self.metric_description;
150
+ let units = &self.metric_unit;
151
+ let context = format!("otel.{}", &self.metric_name);
152
+ let family = self.metric_name.replace('.', "/");
153
+ let chart_type = if self.is_histogram() {
154
+ "heatmap"
155
+ } else {
156
+ "line"
157
+ };
158
+ let priority = 1;
159
+ let update_every = std::time::Duration::from_nanos(ue.get()).as_secs();
160
+
161
+ println!(
162
+ "CHART {type_id} '{name}' '{title}' '{units}' '{family}' '{context}' {chart_type} {priority} {update_every}"
163
+ );
164
+
165
+ for (key, value) in self.attributes.iter() {
166
+ let value_str = match value {
167
+ JsonValue::String(s) => s.clone(),
168
+ JsonValue::Number(n) => n.to_string(),
169
+ JsonValue::Bool(b) => b.to_string(),
170
+ _ => continue,
171
+ };
172
+
173
+ println!("CLABEL '{key}' '{value_str}' 1");
174
+ }
175
+ println!("CLABEL_COMMIT");
176
+
177
+ // Emit dimensions
178
+ if self.is_histogram() {
179
+ let mut dimension_names = self.samples_table.iter_dimensions().collect::<Vec<_>>();
180
+
181
+ dimension_names.sort_by(|a, b| {
182
+ let a_val = if *a == "+Inf" {
183
+ f64::INFINITY
184
+ } else {
185
+ a.parse::<f64>().unwrap()
186
+ };
187
+ let b_val = if *b == "+Inf" {
188
+ f64::INFINITY
189
+ } else {
190
+ b.parse::<f64>().unwrap()
191
+ };
192
+ a_val.partial_cmp(&b_val).unwrap()
193
+ });
194
+
195
+ for dimension_name in dimension_names {
196
+ let algorithm = match self.is_monotonic {
197
+ Some(true) => "incremental",
198
+ _ => "absolute",
199
+ };
200
+ println!(
201
+ "DIMENSION {} {} {} 1 {}",
202
+ dimension_name, dimension_name, algorithm, self.divisor
203
+ );
204
+ }
205
+ } else {
206
+ for dimension_name in self.samples_table.iter_dimensions() {
207
+ let algorithm = match self.is_monotonic {
208
+ Some(true) => "incremental",
209
+ _ => "absolute",
210
+ };
211
+ println!(
212
+ "DIMENSION {} {} {} 1 {}",
213
+ dimension_name, dimension_name, algorithm, self.divisor
214
+ );
215
+ }
216
+ }
217
+ }
218
+
219
+ fn process_next_interval(&mut self) -> ChartState {
220
+ let lsti = match &self.last_samples_table_interval {
221
+ Some(interval) => interval,
222
+ None => return ChartState::Empty,
223
+ };
224
+
225
+ let lci = match &self.last_collection_interval {
226
+ Some(interval) => interval,
227
+ None => return ChartState::Empty,
228
+ };
229
+
230
+ // Clean stale samples
231
+ self.samples_table.drop_stale_samples(lsti);
232
+ if self.samples_table.is_empty() {
233
+ return ChartState::Empty;
234
+ }
235
+
236
+ // Check for gaps
237
+ let have_gap = self
238
+ .samples_table
239
+ .iter_samples_buffers()
240
+ .all(|sb| sb.first().is_none_or(|sp| lsti.is_in_gap(sp)));
241
+
242
+ if have_gap {
243
+ return ChartState::InGap;
244
+ }
245
+
246
+ // Collect samples to emit
247
+ let mut samples_to_emit = Vec::new();
248
+ for (dimension_name, sb) in &mut self.samples_table.iter_mut() {
249
+ if let Some(sp) = sb.first() {
250
+ if lsti.is_on_time(sp) {
251
+ if let Some(sample) = sb.pop() {
252
+ samples_to_emit.push((dimension_name.clone(), sample.value));
253
+ }
254
+ }
255
+ }
256
+ }
257
+
258
+ // Emit data if we have samples
259
+ if !samples_to_emit.is_empty() {
260
+ self.emit_begin(lci.update_every.get());
261
+ for (dimension_name, value) in samples_to_emit {
262
+ self.emit_set(&dimension_name, value);
263
+ }
264
+ self.emit_end();
265
+ }
266
+
267
+ // Move to next interval
268
+ self.last_samples_table_interval = Some(lsti.next_interval());
269
+ self.last_collection_interval = Some(lci.next_interval());
270
+
271
+ ChartState::Initialized
272
+ }
273
+
274
+ fn emit_begin(&self, update_every: u64) {
275
+ let ue = std::time::Duration::from_nanos(update_every).as_micros() as u64;
276
+ println!("BEGIN {} {}", self.chart_id, ue);
277
+ }
278
+
279
+ fn emit_set(&self, dimension_name: &str, value: f64) {
280
+ println!("SET {} {}", dimension_name, value * self.divisor as f64);
281
+ }
282
+
283
+ fn emit_end(&self) {
284
+ let collection_time = std::time::Duration::from_nanos(
285
+ self.last_collection_interval.unwrap().collection_time(),
286
+ )
287
+ .as_secs();
288
+ println!("END {collection_time}");
289
+ }
290
+
291
+ pub fn last_collection_time(&self) -> Option<std::time::SystemTime> {
292
+ self.last_collection_interval.as_ref().map(|lci| {
293
+ std::time::UNIX_EPOCH + std::time::Duration::from_nanos(lci.collection_time())
294
+ })
295
+ }
296
+}
src/crates/jf/otel-plugin/src/netdata_env.rs
new
+204
@@ -0,0 +1,204 @@
1
+#![allow(dead_code)]
2
+
3
+use std::env;
4
+use std::path::PathBuf;
5
+
6
+#[derive(Debug, Clone, Default)]
7
+pub struct NetdataEnv {
8
+ pub user_config_dir: Option<PathBuf>,
9
+ pub stock_config_dir: Option<PathBuf>,
10
+ pub plugins_dir: Option<PathBuf>,
11
+ pub user_plugins_dirs: Option<Vec<PathBuf>>,
12
+ pub web_dir: Option<PathBuf>,
13
+ pub cache_dir: Option<PathBuf>,
14
+ pub log_dir: Option<PathBuf>,
15
+ pub host_prefix: Option<String>,
16
+ pub debug_flags: Option<String>,
17
+ pub update_every: Option<u64>,
18
+ pub invocation_id: Option<String>,
19
+ pub log_method: Option<LogMethod>,
20
+ pub log_format: Option<LogFormat>,
21
+ pub log_level: Option<LogLevel>,
22
+ pub syslog_facility: Option<SyslogFacility>,
23
+ pub errors_throttle_period: Option<u64>,
24
+ pub errors_per_period: Option<u64>,
25
+ pub systemd_journal_path: Option<PathBuf>,
26
+}
27
+
28
+#[derive(Debug, Clone)]
29
+pub enum LogMethod {
30
+ Syslog,
31
+ Journal,
32
+ Stderr,
33
+ None,
34
+}
35
+
36
+#[derive(Debug, Clone)]
37
+pub enum LogFormat {
38
+ Journal,
39
+ Logfmt,
40
+ Json,
41
+}
42
+
43
+#[derive(Debug, Clone)]
44
+pub enum LogLevel {
45
+ Emergency,
46
+ Alert,
47
+ Critical,
48
+ Error,
49
+ Warning,
50
+ Notice,
51
+ Info,
52
+ Debug,
53
+}
54
+
55
+#[derive(Debug, Clone)]
56
+pub enum SyslogFacility {
57
+ Auth,
58
+ Authpriv,
59
+ Cron,
60
+ Daemon,
61
+ Ftp,
62
+ Kern,
63
+ Lpr,
64
+ Mail,
65
+ News,
66
+ Syslog,
67
+ User,
68
+ Uucp,
69
+ Local0,
70
+ Local1,
71
+ Local2,
72
+ Local3,
73
+ Local4,
74
+ Local5,
75
+ Local6,
76
+ Local7,
77
+}
78
+
79
+impl NetdataEnv {
80
+ pub fn from_environment() -> Self {
81
+ Self {
82
+ user_config_dir: env::var("NETDATA_USER_CONFIG_DIR").ok().map(PathBuf::from),
83
+ stock_config_dir: env::var("NETDATA_STOCK_CONFIG_DIR").ok().map(PathBuf::from),
84
+ plugins_dir: env::var("NETDATA_PLUGINS_DIR").ok().map(PathBuf::from),
85
+ user_plugins_dirs: env::var("NETDATA_USER_PLUGINS_DIRS")
86
+ .ok()
87
+ .map(|s| s.split(':').map(PathBuf::from).collect()),
88
+ web_dir: env::var("NETDATA_WEB_DIR").ok().map(PathBuf::from),
89
+ cache_dir: env::var("NETDATA_CACHE_DIR").ok().map(PathBuf::from),
90
+ log_dir: env::var("NETDATA_LOG_DIR").ok().map(PathBuf::from),
91
+ host_prefix: env::var("NETDATA_HOST_PREFIX").ok(),
92
+ debug_flags: env::var("NETDATA_DEBUG_FLAGS").ok(),
93
+ update_every: env::var("NETDATA_UPDATE_EVERY")
94
+ .ok()
95
+ .and_then(|s| s.parse().ok()),
96
+ invocation_id: env::var("NETDATA_INVOCATION_ID").ok(),
97
+ log_method: env::var("NETDATA_LOG_METHOD")
98
+ .ok()
99
+ .and_then(|s| s.parse().ok()),
100
+ log_format: env::var("NETDATA_LOG_FORMAT")
101
+ .ok()
102
+ .and_then(|s| s.parse().ok()),
103
+ log_level: env::var("NETDATA_LOG_LEVEL")
104
+ .ok()
105
+ .and_then(|s| s.parse().ok()),
106
+ syslog_facility: env::var("NETDATA_SYSLOG_FACILITY")
107
+ .ok()
108
+ .and_then(|s| s.parse().ok()),
109
+ errors_throttle_period: env::var("NETDATA_ERRORS_THROTTLE_PERIOD")
110
+ .ok()
111
+ .and_then(|s| s.parse().ok()),
112
+ errors_per_period: env::var("NETDATA_ERRORS_PER_PERIOD")
113
+ .ok()
114
+ .and_then(|s| s.parse().ok()),
115
+ systemd_journal_path: env::var("NETDATA_SYSTEMD_JOURNAL_PATH")
116
+ .ok()
117
+ .map(PathBuf::from),
118
+ }
119
+ }
120
+
121
+ pub fn running_under_netdata(&self) -> bool {
122
+ // we are overtly cautious, just one check would suffice
123
+ self.user_config_dir.is_some()
124
+ || self.stock_config_dir.is_some()
125
+ || self.plugins_dir.is_some()
126
+ || self.invocation_id.is_some()
127
+ }
128
+}
129
+
130
+// Implement FromStr for the enums
131
+impl std::str::FromStr for LogMethod {
132
+ type Err = String;
133
+
134
+ fn from_str(s: &str) -> Result<Self, Self::Err> {
135
+ match s.to_lowercase().as_str() {
136
+ "syslog" => Ok(LogMethod::Syslog),
137
+ "journal" => Ok(LogMethod::Journal),
138
+ "stderr" => Ok(LogMethod::Stderr),
139
+ "none" => Ok(LogMethod::None),
140
+ _ => Err(format!("Invalid log method: {}", s)),
141
+ }
142
+ }
143
+}
144
+
145
+impl std::str::FromStr for LogFormat {
146
+ type Err = String;
147
+
148
+ fn from_str(s: &str) -> Result<Self, Self::Err> {
149
+ match s.to_lowercase().as_str() {
150
+ "journal" => Ok(LogFormat::Journal),
151
+ "logfmt" => Ok(LogFormat::Logfmt),
152
+ "json" => Ok(LogFormat::Json),
153
+ _ => Err(format!("Invalid log format: {}", s)),
154
+ }
155
+ }
156
+}
157
+
158
+impl std::str::FromStr for LogLevel {
159
+ type Err = String;
160
+
161
+ fn from_str(s: &str) -> Result<Self, Self::Err> {
162
+ match s.to_lowercase().as_str() {
163
+ "emergency" => Ok(LogLevel::Emergency),
164
+ "alert" => Ok(LogLevel::Alert),
165
+ "critical" => Ok(LogLevel::Critical),
166
+ "error" => Ok(LogLevel::Error),
167
+ "warning" => Ok(LogLevel::Warning),
168
+ "notice" => Ok(LogLevel::Notice),
169
+ "info" => Ok(LogLevel::Info),
170
+ "debug" => Ok(LogLevel::Debug),
171
+ _ => Err(format!("Invalid log level: {}", s)),
172
+ }
173
+ }
174
+}
175
+
176
+impl std::str::FromStr for SyslogFacility {
177
+ type Err = String;
178
+
179
+ fn from_str(s: &str) -> Result<Self, Self::Err> {
180
+ match s.to_lowercase().as_str() {
181
+ "auth" => Ok(SyslogFacility::Auth),
182
+ "authpriv" => Ok(SyslogFacility::Authpriv),
183
+ "cron" => Ok(SyslogFacility::Cron),
184
+ "daemon" => Ok(SyslogFacility::Daemon),
185
+ "ftp" => Ok(SyslogFacility::Ftp),
186
+ "kern" => Ok(SyslogFacility::Kern),
187
+ "lpr" => Ok(SyslogFacility::Lpr),
188
+ "mail" => Ok(SyslogFacility::Mail),
189
+ "news" => Ok(SyslogFacility::News),
190
+ "syslog" => Ok(SyslogFacility::Syslog),
191
+ "user" => Ok(SyslogFacility::User),
192
+ "uucp" => Ok(SyslogFacility::Uucp),
193
+ "local0" => Ok(SyslogFacility::Local0),
194
+ "local1" => Ok(SyslogFacility::Local1),
195
+ "local2" => Ok(SyslogFacility::Local2),
196
+ "local3" => Ok(SyslogFacility::Local3),
197
+ "local4" => Ok(SyslogFacility::Local4),
198
+ "local5" => Ok(SyslogFacility::Local5),
199
+ "local6" => Ok(SyslogFacility::Local6),
200
+ "local7" => Ok(SyslogFacility::Local7),
201
+ _ => Err(format!("Invalid syslog facility: {}", s)),
202
+ }
203
+ }
204
+}
src/crates/jf/otel-plugin/src/plugin_config.rs
new
+299
@@ -0,0 +1,299 @@
1
+use crate::netdata_env::NetdataEnv;
2
+
3
+use anyhow::{Context, Result};
4
+use bytesize::ByteSize;
5
+use clap::Parser;
6
+use serde::{Deserialize, Serialize};
7
+use std::fs;
8
+use std::path::Path;
9
+use std::time::Duration;
10
+
11
+#[derive(Parser, Debug, Clone, Serialize, Deserialize)]
12
+#[serde(deny_unknown_fields)]
13
+pub struct EndpointConfig {
14
+ /// gRPC endpoint to listen on
15
+ #[arg(long = "otel-endpoint", default_value = "127.0.0.1:4317")]
16
+ pub path: String,
17
+
18
+ /// Path to TLS certificate file (enables TLS when provided)
19
+ #[arg(long = "otel-tls-cert-path")]
20
+ pub tls_cert_path: Option<String>,
21
+
22
+ /// Path to TLS private key file (required when TLS certificate is provided)
23
+ #[arg(long = "otel-tls-key-path")]
24
+ pub tls_key_path: Option<String>,
25
+
26
+ /// Path to TLS CA certificate file for client authentication (optional)
27
+ #[arg(long = "otel-tls-ca-cert-path")]
28
+ pub tls_ca_cert_path: Option<String>,
29
+}
30
+
31
+impl Default for EndpointConfig {
32
+ fn default() -> Self {
33
+ Self {
34
+ path: String::from("127.0.0.1:4317"),
35
+ tls_cert_path: None,
36
+ tls_key_path: None,
37
+ tls_ca_cert_path: None,
38
+ }
39
+ }
40
+}
41
+
42
+#[derive(Parser, Debug, Clone, Serialize, Deserialize)]
43
+#[serde(deny_unknown_fields)]
44
+pub struct MetricsConfig {
45
+ /// Print flattened metrics to stdout for debugging
46
+ #[arg(long = "otel-metrics-print-flattened")]
47
+ pub print_flattened: bool,
48
+
49
+ /// Number of samples to buffer for collection interval detection
50
+ #[arg(long = "otel-metrics-buffer-samples", default_value = "10")]
51
+ pub buffer_samples: usize,
52
+
53
+ /// Maximum number of new charts to create per collection interval
54
+ #[arg(long = "otel-metrics-throttle-charts", default_value = "100")]
55
+ pub throttle_charts: usize,
56
+
57
+ /// Directory with configuration files for mapping OTEL metrics to Netdata charts
58
+ #[arg(long = "otel-metrics-charts-configs-dir")]
59
+ pub chart_configs_dir: Option<String>,
60
+}
61
+
62
+impl Default for MetricsConfig {
63
+ fn default() -> Self {
64
+ Self {
65
+ print_flattened: false,
66
+ buffer_samples: 10,
67
+ throttle_charts: 100,
68
+ chart_configs_dir: None,
69
+ }
70
+ }
71
+}
72
+
73
+/// Parse a duration string for clap (e.g., "7 days", "1 week", "168h")
74
+fn parse_duration(s: &str) -> Result<Duration, String> {
75
+ humantime::parse_duration(s).map_err(|e| {
76
+ format!(
77
+ "Invalid duration format: '{}'. Use formats like '7 days', '1 week', '168h'. Error: {}",
78
+ s, e
79
+ )
80
+ })
81
+}
82
+
83
+/// Parse a bytesize string for clap (e.g., "100MB", "1.5GB", "512MiB")
84
+fn parse_bytesize(s: &str) -> Result<ByteSize, String> {
85
+ s.parse().map_err(|e| {
86
+ format!(
87
+ "Invalid size format: '{}'. Use formats like '100MB', '1.5GB', '512MiB'. Error: {}",
88
+ s, e
89
+ )
90
+ })
91
+}
92
+
93
+#[derive(Parser, Debug, Clone, Serialize, Deserialize)]
94
+#[serde(deny_unknown_fields)]
95
+pub struct LogsConfig {
96
+ /// Directory to store journal files for logs
97
+ #[arg(long = "otel-logs-journal-dir")]
98
+ pub journal_dir: String,
99
+
100
+ /// Maximum file size for journal files (accepts human-readable sizes like "100MB", "1.5GB")
101
+ #[arg(
102
+ long = "otel-logs-rotation-size-of-journal-file",
103
+ default_value = "100MB",
104
+ value_parser = parse_bytesize
105
+ )]
106
+ #[serde(with = "bytesize_serde")]
107
+ pub size_of_journal_file: ByteSize,
108
+
109
+ /// Maximum number of journal files to keep
110
+ #[arg(
111
+ long = "otel-logs-retention-number-of-journal-files",
112
+ default_value = "10"
113
+ )]
114
+ pub number_of_journal_files: usize,
115
+
116
+ /// Maximum total size for all journal files (accepts human-readable sizes like "1GB", "500MB")
117
+ #[arg(
118
+ long = "otel-logs-retention-size-of-journal-files",
119
+ default_value = "1GB",
120
+ value_parser = parse_bytesize
121
+ )]
122
+ #[serde(with = "bytesize_serde")]
123
+ pub size_of_journal_files: ByteSize,
124
+
125
+ /// Maximum age for journal entries (accepts human-readable durations like "7 days", "1 week", "168h")
126
+ #[arg(
127
+ long = "otel-logs-retention-duration-of-journal-files",
128
+ default_value = "7 days",
129
+ value_parser = parse_duration
130
+ )]
131
+ #[serde(with = "humantime_serde")]
132
+ pub duration_of_journal_files: Duration,
133
+
134
+ /// Maximum duration that entries in a single journal file can span (accepts human-readable durations like "2 hours", "1h", "30m")
135
+ #[arg(
136
+ long = "otel-logs-rotation-duration-of-journal-file",
137
+ default_value = "2 hours",
138
+ value_parser = parse_duration
139
+ )]
140
+ #[serde(with = "humantime_serde")]
141
+ pub duration_of_journal_file: Duration,
142
+}
143
+
144
+impl Default for LogsConfig {
145
+ fn default() -> Self {
146
+ Self {
147
+ journal_dir: String::from("/tmp/netdata-journals"),
148
+ size_of_journal_file: ByteSize::mb(100),
149
+ number_of_journal_files: 10,
150
+ size_of_journal_files: ByteSize::gb(1),
151
+ duration_of_journal_files: Duration::from_secs(7 * 24 * 60 * 60), // 7 days
152
+ duration_of_journal_file: Duration::from_secs(2 * 60 * 60), // 2 hours
153
+ }
154
+ }
155
+}
156
+
157
+#[derive(Default, Debug, Parser, Clone, Serialize, Deserialize)]
158
+#[command(name = "otel-plugin")]
159
+#[command(about = "OpenTelemetry metrics and logs plugin.")]
160
+#[command(version = "0.1")]
161
+#[serde(deny_unknown_fields)]
162
+pub struct PluginConfig {
163
+ // endpoint configuration (includes grpc endpoint and tls)
164
+ #[command(flatten)]
165
+ #[serde(rename = "endpoint")]
166
+ pub endpoint: EndpointConfig,
167
+
168
+ // metrics
169
+ #[command(flatten)]
170
+ #[serde(rename = "metrics")]
171
+ pub metrics: MetricsConfig,
172
+
173
+ // logs
174
+ #[command(flatten)]
175
+ #[serde(rename = "logs")]
176
+ pub logs: LogsConfig,
177
+
178
+ /// Collection interval (ignored)
179
+ #[arg(hide = true, help = "Collection interval in seconds (ignored)")]
180
+ #[serde(skip)]
181
+ pub _update_frequency: Option<u32>,
182
+
183
+ // netdata env variables
184
+ #[arg(skip)]
185
+ #[serde(skip)]
186
+ pub _netdata_env: NetdataEnv,
187
+}
188
+
189
+impl PluginConfig {
190
+ pub fn new() -> Result<Self> {
191
+ let netdata_env = NetdataEnv::from_environment();
192
+
193
+ let mut config = if netdata_env.running_under_netdata() {
194
+ // Try user config first, fallback to stock config
195
+ let user_config = netdata_env
196
+ .user_config_dir
197
+ .as_ref()
198
+ .map(|path| path.join("otel.yml"))
199
+ .and_then(|path| {
200
+ Self::from_yaml_file(&path)
201
+ .with_context(|| format!("Loading user config from {}", path.display()))
202
+ .ok()
203
+ });
204
+
205
+ if let Some(config) = user_config {
206
+ config
207
+ } else if let Some(stock_path) = netdata_env
208
+ .stock_config_dir
209
+ .as_ref()
210
+ .map(|p| p.join("otel.yml"))
211
+ {
212
+ Self::from_yaml_file(&stock_path).with_context(|| {
213
+ format!("Loading stock config from {}", stock_path.display())
214
+ })?
215
+ } else {
216
+ anyhow::bail!("No configuration directories available");
217
+ }
218
+ } else {
219
+ // load from CLI args
220
+ Self::parse()
221
+ };
222
+
223
+ // Resolve relative paths
224
+ if let Some(charts_config_dir) = &mut config.metrics.chart_configs_dir {
225
+ *charts_config_dir =
226
+ resolve_relative_path(charts_config_dir, netdata_env.user_config_dir.as_deref());
227
+ }
228
+ config.logs.journal_dir =
229
+ resolve_relative_path(&config.logs.journal_dir, netdata_env.log_dir.as_deref());
230
+
231
+ // Validate configuration
232
+ if config.metrics.buffer_samples == 0 {
233
+ anyhow::bail!("buffer_samples must be greater than 0");
234
+ }
235
+
236
+ if config.metrics.throttle_charts == 0 {
237
+ anyhow::bail!("throttle_charts must be greater than 0");
238
+ }
239
+
240
+ // Validate endpoint format (basic check)
241
+ if !config.endpoint.path.contains(':') {
242
+ anyhow::bail!(
243
+ "endpoint must be in format host:port, got: {}",
244
+ config.endpoint.path
245
+ );
246
+ }
247
+
248
+ // Validate TLS configuration
249
+ match (
250
+ &config.endpoint.tls_cert_path,
251
+ &config.endpoint.tls_key_path,
252
+ ) {
253
+ (Some(cert_path), Some(key_path)) => {
254
+ if cert_path.is_empty() {
255
+ anyhow::bail!("TLS certificate path cannot be empty when provided");
256
+ }
257
+ if key_path.is_empty() {
258
+ anyhow::bail!("TLS private key path cannot be empty when provided");
259
+ }
260
+ }
261
+ (Some(_), None) => {
262
+ anyhow::bail!(
263
+ "TLS private key path must be provided when TLS certificate is provided"
264
+ );
265
+ }
266
+ (None, Some(_)) => {
267
+ anyhow::bail!(
268
+ "TLS certificate path must be provided when TLS private key is provided"
269
+ );
270
+ }
271
+ (None, None) => {
272
+ // TLS disabled, which is fine
273
+ }
274
+ }
275
+
276
+ Ok(config)
277
+ }
278
+
279
+ pub fn from_yaml_file<P: AsRef<Path>>(path: P) -> Result<Self> {
280
+ let path = path.as_ref();
281
+ let contents = fs::read_to_string(path)
282
+ .with_context(|| format!("Failed to read config file: {}", path.display()))?;
283
+ let config: PluginConfig = serde_yaml::from_str(&contents)
284
+ .with_context(|| format!("Failed to parse YAML config file: {}", path.display()))?;
285
+ Ok(config)
286
+ }
287
+}
288
+
289
+// Helper function to resolve relative paths
290
+fn resolve_relative_path(path: &str, base_dir: Option<&Path>) -> String {
291
+ let path = Path::new(path);
292
+ if path.is_absolute() {
293
+ path.to_string_lossy().to_string()
294
+ } else if let Some(base) = base_dir {
295
+ base.join(path).to_string_lossy().to_string()
296
+ } else {
297
+ path.to_string_lossy().to_string()
298
+ }
299
+}
src/crates/jf/otel-plugin/src/regex_cache.rs
new
+22
@@ -0,0 +1,22 @@
1
+use regex::Regex;
2
+use std::collections::HashMap;
3
+use std::sync::{Arc, Mutex};
4
+
5
+#[derive(Default, Debug)]
6
+pub struct RegexCache {
7
+ cache: Arc<Mutex<HashMap<String, Regex>>>,
8
+}
9
+
10
+impl RegexCache {
11
+ pub fn get(&self, pattern: &str) -> Result<Regex, regex::Error> {
12
+ let mut cache = self.cache.lock().unwrap();
13
+
14
+ if let Some(regex) = cache.get(pattern) {
15
+ return Ok(regex.clone());
16
+ }
17
+
18
+ let compiled_regex = Regex::new(pattern)?;
19
+ cache.insert(pattern.to_string(), compiled_regex.clone());
20
+ Ok(compiled_regex)
21
+ }
22
+}
src/crates/jf/otel-plugin/src/samples_table.rs
new
+221
@@ -0,0 +1,221 @@
1
+use std::collections::HashMap;
2
+use std::num::NonZeroU64;
3
+
4
+#[derive(Default, Clone, Copy, PartialEq, Debug)]
5
+pub struct SamplePoint {
6
+ unix_time: u64,
7
+ pub value: f64,
8
+}
9
+
10
+#[derive(Copy, Clone, Debug)]
11
+pub struct CollectionInterval {
12
+ pub end_time: u64,
13
+ pub update_every: NonZeroU64,
14
+}
15
+
16
+impl CollectionInterval {
17
+ fn from_samples(sample_points: &[SamplePoint]) -> Option<Self> {
18
+ if sample_points.len() < 2 {
19
+ return None;
20
+ }
21
+
22
+ let collection_time = sample_points[0].unix_time;
23
+ let mut update_every = u64::MAX;
24
+
25
+ for w in sample_points.windows(2) {
26
+ update_every = update_every.min(w[1].unix_time - w[0].unix_time);
27
+ }
28
+
29
+ NonZeroU64::new(update_every).map(|update_every| Self {
30
+ end_time: collection_time,
31
+ update_every,
32
+ })
33
+ }
34
+
35
+ pub fn next_interval(&self) -> Self {
36
+ Self {
37
+ end_time: self.end_time + self.update_every.get(),
38
+ update_every: self.update_every,
39
+ }
40
+ }
41
+
42
+ pub fn collection_time(&self) -> u64 {
43
+ self.end_time + self.update_every.get()
44
+ }
45
+
46
+ fn is_stale(&self, sp: &SamplePoint) -> bool {
47
+ sp.unix_time < self.end_time
48
+ }
49
+
50
+ pub fn is_on_time(&self, sp: &SamplePoint) -> bool {
51
+ let window = self.update_every.get() / 4;
52
+ let window_start = self.end_time + self.update_every.get() - window;
53
+ let window_end = self.end_time + self.update_every.get() + window;
54
+
55
+ sp.unix_time >= window_start && sp.unix_time <= window_end
56
+ }
57
+
58
+ pub fn is_in_gap(&self, sp: &SamplePoint) -> bool {
59
+ !self.is_stale(sp) && !self.is_on_time(sp)
60
+ }
61
+
62
+ pub fn aligned_interval(&self) -> Option<Self> {
63
+ let dur = std::time::Duration::from_nanos(self.end_time);
64
+ let end_time = dur.as_secs() + u64::from(dur.subsec_millis() >= 500);
65
+
66
+ let dur = std::time::Duration::from_nanos(self.update_every.get());
67
+ let update_every = dur.as_secs() + u64::from(dur.subsec_millis() >= 500);
68
+
69
+ Self::from_secs(end_time, update_every)
70
+ }
71
+
72
+ fn from_secs(end_time: u64, update_every: u64) -> Option<Self> {
73
+ let end_time = std::time::Duration::from_secs(end_time).as_nanos() as u64;
74
+ let update_every = std::time::Duration::from_secs(update_every).as_nanos() as u64;
75
+
76
+ NonZeroU64::new(update_every).map(|update_every| Self {
77
+ end_time,
78
+ update_every,
79
+ })
80
+ }
81
+}
82
+
83
+#[derive(Debug, Default, Clone)]
84
+pub struct SamplesBuffer(Vec<SamplePoint>);
85
+
86
+impl SamplesBuffer {
87
+ pub fn push(&mut self, sp: SamplePoint) {
88
+ match self.0.binary_search_by_key(&sp.unix_time, |p| p.unix_time) {
89
+ Ok(idx) => self.0[idx] = sp,
90
+ Err(idx) => self.0.insert(idx, sp),
91
+ }
92
+ }
93
+
94
+ pub fn pop(&mut self) -> Option<SamplePoint> {
95
+ if self.0.is_empty() {
96
+ None
97
+ } else {
98
+ Some(self.0.remove(0))
99
+ }
100
+ }
101
+
102
+ fn is_empty(&self) -> bool {
103
+ self.0.is_empty()
104
+ }
105
+
106
+ pub fn first(&self) -> Option<&SamplePoint> {
107
+ self.0.first()
108
+ }
109
+
110
+ pub fn len(&self) -> usize {
111
+ self.0.len()
112
+ }
113
+
114
+ pub fn drop_stale_samples(&mut self, ci: &CollectionInterval) -> usize {
115
+ let split_idx = self
116
+ .0
117
+ .iter()
118
+ .position(|sp| !ci.is_stale(sp))
119
+ .unwrap_or(self.0.len());
120
+
121
+ self.0.drain(..split_idx);
122
+
123
+ split_idx
124
+ }
125
+
126
+ pub fn collection_interval(&self) -> Option<CollectionInterval> {
127
+ CollectionInterval::from_samples(&self.0)
128
+ }
129
+}
130
+
131
+#[derive(Debug, Default)]
132
+pub struct SamplesTable {
133
+ dimensions: HashMap<String, SamplesBuffer>,
134
+}
135
+
136
+impl SamplesTable {
137
+ pub fn insert(&mut self, dimension: &str, unix_time: u64, value: f64) -> bool {
138
+ let sp = SamplePoint { unix_time, value };
139
+
140
+ let is_new_dimension = if let Some(sb) = self.dimensions.get_mut(dimension) {
141
+ sb.push(sp);
142
+ false
143
+ } else {
144
+ let mut sb = SamplesBuffer::default();
145
+ sb.push(sp);
146
+ self.dimensions.insert(dimension.to_string(), sb);
147
+ true
148
+ };
149
+
150
+ is_new_dimension
151
+ }
152
+
153
+ pub fn is_empty(&self) -> bool {
154
+ self.dimensions.values().all(|sb| sb.is_empty())
155
+ }
156
+
157
+ pub fn total_samples(&self) -> usize {
158
+ self.dimensions
159
+ .values()
160
+ .map(|sb| sb.len())
161
+ .max()
162
+ .unwrap_or(0)
163
+ }
164
+
165
+ pub fn drop_stale_samples(&mut self, ci: &CollectionInterval) -> usize {
166
+ let mut dropped_samples = 0;
167
+
168
+ for sb in self.dimensions.values_mut() {
169
+ dropped_samples += sb.drop_stale_samples(ci);
170
+ }
171
+
172
+ dropped_samples
173
+ }
174
+
175
+ pub fn collection_interval(&self) -> Option<CollectionInterval> {
176
+ self.dimensions
177
+ .values()
178
+ .filter_map(|sb| sb.collection_interval())
179
+ .min_by_key(|ci| ci.collection_time())
180
+ }
181
+
182
+ pub fn iter_mut(&mut self) -> impl Iterator<Item = (&String, &mut SamplesBuffer)> {
183
+ self.dimensions.iter_mut()
184
+ }
185
+
186
+ pub fn iter_dimensions(&self) -> impl Iterator<Item = &String> {
187
+ self.dimensions.keys()
188
+ }
189
+
190
+ pub fn iter_samples_buffers(&self) -> impl Iterator<Item = &SamplesBuffer> {
191
+ self.dimensions.values()
192
+ }
193
+
194
+ // returns multiplier/divisor
195
+ pub fn scaling_factors(&self) -> (i32, i32) {
196
+ let mut has_nonzero = false;
197
+
198
+ for buffer in self.dimensions.values() {
199
+ for sample in &buffer.0 {
200
+ let value = sample.value;
201
+
202
+ // Check if value is outside the -100 to 100 range
203
+ if !(-100.0..=100.0).contains(&value) {
204
+ return (1, 1);
205
+ }
206
+
207
+ // Check for non-zero values
208
+ if value != 0.0 {
209
+ has_nonzero = true;
210
+ }
211
+ }
212
+ }
213
+
214
+ // Return 1/1000 scaling if all values are in range and at least one is non-zero
215
+ if has_nonzero {
216
+ (1, 1000)
217
+ } else {
218
+ (1, 1)
219
+ }
220
+ }
221
+}
src/crates/jf/sigbus/Cargo.toml
new
+9
@@ -0,0 +1,9 @@
1
+[package]
2
+name = "sigbus"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+error = { path = "../error" }
9
+libc = { workspace = true }
src/crates/jf/sigbus/src/lib.rs
new
+49
@@ -0,0 +1,49 @@
1
+use error::{JournalError, Result};
2
+use std::sync::atomic::{AtomicBool, Ordering};
3
+use std::sync::OnceLock;
4
+
5
+static SIGBUS_OCCURRED: AtomicBool = AtomicBool::new(false);
6
+static HANDLER_INSTALLED: OnceLock<i32> = OnceLock::new();
7
+
8
+extern "C" fn sigbus_handler(
9
+ _sig: libc::c_int,
10
+ info: *mut libc::siginfo_t,
11
+ _ucontext: *mut libc::c_void,
12
+) {
13
+ unsafe {
14
+ let si = &*info;
15
+ let fault_addr = si.si_addr();
16
+
17
+ let page_addr = (fault_addr as usize & !(4096 - 1)) as *mut libc::c_void;
18
+ libc::mmap(
19
+ page_addr,
20
+ 4096,
21
+ libc::PROT_READ,
22
+ libc::MAP_PRIVATE | libc::MAP_ANONYMOUS | libc::MAP_FIXED,
23
+ -1,
24
+ 0,
25
+ );
26
+
27
+ SIGBUS_OCCURRED.store(true, Ordering::Relaxed);
28
+ }
29
+}
30
+
31
+pub fn signalled() -> bool {
32
+ SIGBUS_OCCURRED.load(Ordering::Relaxed)
33
+}
34
+
35
+pub fn install_handler() -> Result<()> {
36
+ let rc = HANDLER_INSTALLED.get_or_init(|| unsafe {
37
+ let mut sa: libc::sigaction = std::mem::zeroed();
38
+
39
+ sa.sa_flags = libc::SA_SIGINFO;
40
+ sa.sa_sigaction = sigbus_handler as usize;
41
+
42
+ libc::sigaction(libc::SIGBUS, &sa, std::ptr::null_mut())
43
+ });
44
+
45
+ match rc {
46
+ -1 => Err(JournalError::SigbusHandlerError),
47
+ _ => Ok(()),
48
+ }
49
+}
src/crates/jf/window_manager/Cargo.toml
new
+9
@@ -0,0 +1,9 @@
1
+[package]
2
+name = "window_manager"
3
+version.workspace = true
4
+edition.workspace = true
5
+rust-version.workspace = true
6
+
7
+[dependencies]
8
+error = { path = "../error" }
9
+memmap2 = { workspace = true }
src/crates/jf/window_manager/src/lib.rs
new
+235
@@ -0,0 +1,235 @@
1
+use error::Result;
2
+use memmap2::{Mmap, MmapMut, MmapOptions};
3
+use std::fs::File;
4
+use std::ops::{Deref, DerefMut};
5
+
6
+const PAGE_SIZE: u64 = 4096;
7
+
8
+pub trait MemoryMap: Deref<Target = [u8]> {
9
+ fn create(file: &File, offset: u64, size: u64) -> Result<Self>
10
+ where
11
+ Self: Sized;
12
+}
13
+
14
+pub trait MemoryMapMut: MemoryMap + DerefMut {}
15
+
16
+impl MemoryMap for Mmap {
17
+ fn create(file: &File, offset: u64, size: u64) -> Result<Self> {
18
+ let mmap = unsafe {
19
+ MmapOptions::new()
20
+ .offset(offset)
21
+ .len(size as usize)
22
+ .map(file)?
23
+ };
24
+
25
+ Ok(mmap)
26
+ }
27
+}
28
+
29
+impl MemoryMap for MmapMut {
30
+ fn create(file: &File, offset: u64, size: u64) -> Result<Self> {
31
+ let required_size = offset + size;
32
+
33
+ if required_size > file.metadata()?.len() {
34
+ file.set_len(required_size)?;
35
+ }
36
+
37
+ let mmap = unsafe {
38
+ MmapOptions::new()
39
+ .offset(offset)
40
+ .len(size as usize)
41
+ .map_mut(file)?
42
+ };
43
+
44
+ Ok(mmap)
45
+ }
46
+}
47
+
48
+impl MemoryMapMut for MmapMut {}
49
+
50
+struct Window<M: MemoryMap> {
51
+ offset: u64,
52
+ size: u64,
53
+ mmap: M,
54
+}
55
+
56
+impl<M: MemoryMap> std::fmt::Debug for Window<M> {
57
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
58
+ f.debug_struct("Window")
59
+ .field("offset", &self.offset)
60
+ .field("size", &self.size)
61
+ .finish()
62
+ }
63
+}
64
+
65
+impl<M: MemoryMap> Window<M> {
66
+ fn end_offset(&self) -> u64 {
67
+ self.offset + self.size
68
+ }
69
+
70
+ fn contains(&self, position: u64) -> bool {
71
+ position >= self.offset && position < self.end_offset()
72
+ }
73
+
74
+ fn contains_range(&self, position: u64, size: u64) -> bool {
75
+ position >= self.offset && position + size <= self.end_offset()
76
+ }
77
+
78
+ fn get_slice(&self, position: u64, size: u64) -> &[u8] {
79
+ debug_assert!(self.contains_range(position, size));
80
+
81
+ let offset = (position - self.offset) as usize;
82
+ &self.mmap[offset..offset + size as usize]
83
+ }
84
+}
85
+
86
+impl<M: MemoryMapMut> Window<M> {
87
+ pub fn get_mut_slice(&mut self, position: u64, size: u64) -> &mut [u8] {
88
+ debug_assert!(self.contains_range(position, size));
89
+
90
+ let offset = (position - self.offset) as usize;
91
+ &mut self.mmap[offset..offset + size as usize]
92
+ }
93
+}
94
+
95
+pub struct WindowManager<M: MemoryMap> {
96
+ file: File,
97
+ _file_size: u64,
98
+ chunk_size: u64,
99
+ active_window_idx: Option<usize>,
100
+ max_windows: usize,
101
+ windows: Vec<Window<M>>,
102
+}
103
+
104
+impl<M: MemoryMap> WindowManager<M> {
105
+ pub fn new(file: File, chunk_size: u64, max_windows: usize) -> Result<Self> {
106
+ debug_assert!(chunk_size != 0 && (chunk_size % PAGE_SIZE) == 0);
107
+ debug_assert!(max_windows != 0);
108
+
109
+ let _file_size = file.metadata()?.len();
110
+
111
+ Ok(WindowManager {
112
+ file,
113
+ _file_size,
114
+ chunk_size,
115
+ max_windows,
116
+ windows: Vec::new(),
117
+ active_window_idx: None,
118
+ })
119
+ }
120
+
121
+ fn get_chunk_aligned_start(&self, position: u64) -> u64 {
122
+ (position / self.chunk_size) * self.chunk_size
123
+ }
124
+
125
+ fn get_chunk_aligned_end(&self, position: u64) -> u64 {
126
+ position.div_ceil(self.chunk_size) * self.chunk_size
127
+ }
128
+
129
+ fn create_window(&self, window_start: u64, chunk_count: u64) -> Result<Window<M>> {
130
+ debug_assert_ne!(chunk_count, 0);
131
+
132
+ let size = chunk_count * self.chunk_size;
133
+ let mmap = M::create(&self.file, window_start, size)?;
134
+ Ok(Window {
135
+ offset: window_start,
136
+ size,
137
+ mmap,
138
+ })
139
+ }
140
+
141
+ fn find_window_to_evict(&self) -> usize {
142
+ if self.active_window_idx == Some(0) && self.windows.len() > 1 {
143
+ 1
144
+ } else {
145
+ 0
146
+ }
147
+ }
148
+
149
+ fn lookup_window_by_range(&self, position: u64, size_needed: u64) -> Option<usize> {
150
+ if let Some(idx) = self.active_window_idx {
151
+ if self.windows[idx].contains_range(position, size_needed) {
152
+ return Some(idx);
153
+ }
154
+ }
155
+
156
+ for (idx, window) in self.windows.iter().enumerate() {
157
+ if window.contains_range(position, size_needed) {
158
+ return Some(idx);
159
+ }
160
+ }
161
+
162
+ None
163
+ }
164
+
165
+ fn lookup_window_by_position(&self, position: u64) -> Option<usize> {
166
+ if let Some(idx) = self.active_window_idx {
167
+ if self.windows[idx].contains(position) {
168
+ return Some(idx);
169
+ }
170
+ }
171
+
172
+ for (idx, window) in self.windows.iter().enumerate() {
173
+ if window.contains(position) {
174
+ return Some(idx);
175
+ }
176
+ }
177
+
178
+ None
179
+ }
180
+
181
+ fn get_window(&mut self, position: u64, size_needed: u64) -> Result<&mut Window<M>> {
182
+ if let Some(idx) = self.lookup_window_by_range(position, size_needed) {
183
+ // Use the existing window
184
+ Ok(&mut self.windows[idx])
185
+ } else if let Some(idx) = self.lookup_window_by_position(position) {
186
+ // Remap the window
187
+
188
+ let window = self.windows.remove(idx);
189
+
190
+ let window_start = window.offset;
191
+ let window_end = self.get_chunk_aligned_end(position + size_needed);
192
+ let num_chunks = (window_end - window_start) / self.chunk_size;
193
+
194
+ let new_window = self.create_window(window_start, num_chunks)?;
195
+
196
+ self.windows.push(new_window);
197
+ self.active_window_idx = Some(self.windows.len() - 1);
198
+ Ok(self.windows.last_mut().unwrap())
199
+ } else {
200
+ // Create a brand new window
201
+
202
+ if self.windows.len() >= self.max_windows {
203
+ self.windows.remove(self.find_window_to_evict());
204
+ }
205
+
206
+ // NOTE: the active window index might have been invalidated. In
207
+ // the scope that follows, we should not use code that relies on it.
208
+ {
209
+ // Calculate window start for this position
210
+ let window_start = self.get_chunk_aligned_start(position);
211
+ let window_end = self.get_chunk_aligned_end(position + size_needed);
212
+ let num_chunks = (window_end - window_start) / self.chunk_size;
213
+
214
+ let new_window = self.create_window(window_start, num_chunks)?;
215
+
216
+ self.windows.push(new_window);
217
+ }
218
+
219
+ self.active_window_idx = Some(self.windows.len() - 1);
220
+ Ok(self.windows.last_mut().unwrap())
221
+ }
222
+ }
223
+
224
+ pub fn get_slice(&mut self, position: u64, size: u64) -> Result<&[u8]> {
225
+ let window = self.get_window(position, size)?;
226
+ Ok(window.get_slice(position, size))
227
+ }
228
+}
229
+
230
+impl<M: MemoryMapMut> WindowManager<M> {
231
+ pub fn get_slice_mut(&mut self, position: u64, size: u64) -> Result<&mut [u8]> {
232
+ let window = self.get_window(position, size)?;
233
+ Ok(window.get_mut_slice(position, size))
234
+ }
235
+}
src/crates/netdata-log-viewer/README.md
+9
-9
@@ -17,7 +17,7 @@ This plugin provides a `systemd-journal` function that Netdata can call to query
17
│ (plugin protocol)
18
↓
19
┌──────────────────────────┐
20
-│ journal-viewer-plugin │
20
+│ otel-signal-viewer-plugin │
21
│ │
22
│ ┌───────────────┐ │
23
│ │ Journal │ │
@@ -69,7 +69,7 @@ docker run -d --name jaeger \
69
### Building
70
71
```bash
72
-cargo build --bin journal-viewer-plugin --release
72
+cargo build --bin otel-signal-viewer-plugin --release
73
```
74
75
### Running
@@ -81,7 +81,7 @@ The plugin is designed to be spawned by Netdata:
81
# Configure in /etc/netdata/netdata.conf:
82
83
[plugins]
84
- journal-viewer = yes
84
+ otel-signal-viewer-plugin = yes
85
```
86
87
For development/testing:
@@ -106,7 +106,7 @@ See [DEVELOPMENT.md](./DEVELOPMENT.md) for comprehensive documentation.
106
### Development Workflow
107
108
1. **Make changes** to the plugin code
109
-2. **Rebuild** (fast, seconds): `cargo build --bin journal-viewer-plugin`
109
+2. **Rebuild** (fast, seconds): `cargo build --bin otel-signal-viewer-plugin`
110
3. **Restart** Netdata: `sudo systemctl restart netdata`
111
4. **View traces** in Jaeger: http://localhost:16686
112
5. **Check logs**: `sudo journalctl -u netdata -f`
@@ -124,7 +124,7 @@ See [DEVELOPMENT.md](./DEVELOPMENT.md) for comprehensive documentation.
124
```
125
netdata-log-viewer/
126
├── histogram-service/ # Core business logic (library)
127
-├── journal-viewer-plugin/ # Netdata plugin (binary)
127
+├── otel-signal-viewer-plugin/ # Netdata plugin (binary)
128
├── types/ # Shared request/response types
129
├── watcher-plugin/ # DEPRECATED - no longer needed
130
├── lv/ # DEPRECATED - no longer needed
@@ -142,7 +142,7 @@ The plugin is configured at compile time with sensible defaults:
142
- **Memory cache**: 10,000 entries
143
- **Disk cache**: 64 MiB
144
145
-To customize, edit `create_shared_state()` in `journal-viewer-plugin/src/main.rs`.
145
+To customize, edit `create_shared_state()` in `otel-signal-viewer-plugin/src/main.rs`.
146
147
## Observability
148
@@ -162,13 +162,13 @@ Control log verbosity with `RUST_LOG`:
162
export RUST_LOG="debug"
163
164
# Selective logging
165
-export RUST_LOG="journal_viewer_plugin=trace,histogram_service=debug,journal=info"
165
+export RUST_LOG="otel_signal_viewer=trace,histogram_service=debug,journal=info"
166
```
167
168
### Metrics (Netdata)
169
170
The plugin reports its own metrics:
171
-- `journal_viewer.journal_calls` - Successful/failed/cancelled function calls
171
+- `otel_signal_viewer.journal_calls` - Successful/failed/cancelled function calls
172
173
## Function Interface
174
@@ -229,7 +229,7 @@ The plugin exposes the `systemd-journal` function:
229
sudo tail -f /var/log/netdata/error.log
230
231
# Run manually
232
-sudo -u netdata /path/to/journal-viewer-plugin
232
+sudo -u netdata /path/to/otel-signal-viewer-plugin
233
```
234
235
### No traces in Jaeger
src/crates/netdata-log-viewer/otel-signal-viewer-plugin/Cargo.toml
renamed
+2
-2
@@ -1,5 +1,5 @@
1
[package]
2
-name = "journal-viewer-plugin"
2
+name = "otel-signal-viewer-plugin"
3
version.workspace = true
4
edition.workspace = true
5
rust-version.workspace = true
@@ -8,7 +8,7 @@ rust-version.workspace = true
8
workspace = true
9
10
[[bin]]
11
-name = "journal-viewer-plugin"
11
+name = "otel-signal-viewer-plugin"
12
path = "src/main.rs"
13
14
[dependencies]
src/crates/netdata-log-viewer/otel-signal-viewer-plugin/configs/otel-signal-viewer.yaml.in
renamed
+7
-22
@@ -1,47 +1,35 @@
1
-# Log Viewer Plugin Configuration
2
-# This file configures the systemd journal log viewer plugin for Netdata
1
+# OpenTelemetry Viewer Plugin Configuration
2
+# This file configures the OpenTelemetry viewer plugin for Netdata
3
4
journal:
5
# Paths to journal directories to watch and index. Multiple directories can
6
# be specified to monitor different journal sources. At least one path must
7
# be specified
8
paths:
9
- - "/var/log/journal"
10
- - "/run/log/journal"
9
- "@localstatedir_POST@/log/netdata/otel/v1"
10
11
cache:
12
# Directory to store the hybrid cache (memory + disk backed by Foyer)
15
- # Relative paths are resolved based on Netdata's cache directory
16
- # Default: /var/cache/netdata/log-viewer
17
- directory: "@cachedir_POST@/log-viewer"
13
+ directory: "@cachedir_POST@/otel-signal-viewer"
14
19
- # Memory cache capacity (number of indexed journal entries to keep in memory)
15
+ # Memory cache capacity (number of indexed journal files to keep in memory)
16
# Higher values improve query performance but use more RAM
21
- # Default: 1000
17
memory_capacity: 1000
18
19
# Disk cache size (maximum size of the disk-backed cache)
25
- # Accepts human-readable sizes: "32MB", "64MB", "1GB", etc.
20
# Higher values improve performance for large journal files
27
- # Default: 32MB
21
disk_capacity: "32MB"
22
23
# Cache block size (size of individual cache blocks in Foyer)
31
- # Accepts human-readable sizes: "4MB", "8MB", "16MB", etc.
32
- # Trade-off between I/O efficiency and memory granularity
33
- # Default: 4MB
24
block_size: "4MB"
25
26
# Number of background workers for indexing journal files
37
- # Higher values speed up indexing but use more CPU
27
# Default: number of CPU cores (auto-detected)
28
# Uncomment to override:
40
- # workers: 24
29
+ # workers: 4
30
31
# Queue capacity for pending indexing requests
32
# Controls backpressure on the indexing system
44
- # Default: 100
33
queue_capacity: 100
34
35
indexing:
@@ -50,12 +38,9 @@ indexing:
38
# truncated to prevent unbounded memory growth. This protects against
39
# high-cardinality fields (e.g., MESSAGE with millions of unique values)
40
# causing memory exhaustion during indexing.
53
- # Default: 500
41
max_unique_values_per_field: 500
42
43
# Maximum payload size (in bytes) for field values to index.
57
- # Field values with payloads larger than this limit (or compressed values)
58
- # will be skipped. This prevents large binary data or encoded content
59
- # from consuming excessive memory.
60
- # Default: 100
44
+ # Field values with payloads larger than this limit will be skipped.
45
+ # This prevents field values with large payloads from consuming excessive memory.
46
max_field_payload_size: 100
src/crates/netdata-log-viewer/otel-signal-viewer-plugin/src/catalog.rs
renamed
+2
-2
@@ -698,8 +698,8 @@ impl FunctionHandler for CatalogFunction {
698
// update the `rt` crate as well.
699
700
let mut func_decl = FunctionDeclaration::new(
701
- "journal-viewer",
702
- "Query and visualize journal log entries with histograms and facets",
701
+ "otel-signal-viewer",
702
+ "Query and visualize OpenTelemetry logs",
703
);
704
func_decl.global = true;
705
func_decl.tags = Some(String::from("logs"));
src/crates/netdata-log-viewer/otel-signal-viewer-plugin/src/charts.rs
renamed
+9
-9
@@ -1,4 +1,4 @@
1
-//! Chart definitions for the journal-viewer plugin
1
+//! Chart definitions for the otel-signal-viewer plugin
2
//!
3
//! This module contains all Netdata chart metric structures that track
4
//! plugin performance, cache utilization, and request characteristics.
@@ -35,12 +35,12 @@ impl Metrics {
35
/// Metrics for tracking journal function calls
36
#[derive(JsonSchema, NetdataChart, Default, Clone, PartialEq, Serialize, Deserialize)]
37
#[schemars(
38
- extend("x-chart-id" = "journal_viewer.journal_calls"),
38
+ extend("x-chart-id" = "otel_signal_viewer.journal_calls"),
39
extend("x-chart-title" = "Journal Function Calls"),
40
extend("x-chart-units" = "calls/s"),
41
extend("x-chart-type" = "line"),
42
extend("x-chart-family" = "requests"),
43
- extend("x-chart-context" = "journal_viewer.journal_calls"),
43
+ extend("x-chart-context" = "otel_signal_viewer.journal_calls"),
44
)]
45
pub struct JournalCallMetrics {
46
#[schemars(extend("x-dimension-algorithm" = "incremental"))]
@@ -54,12 +54,12 @@ pub struct JournalCallMetrics {
54
/// Metrics for tracking cache sizes
55
#[derive(JsonSchema, NetdataChart, Default, Clone, PartialEq, Serialize, Deserialize)]
56
#[schemars(
57
- extend("x-chart-id" = "journal_viewer.bucket_lru_cache_size"),
57
+ extend("x-chart-id" = "otel_signal_viewer.bucket_lru_cache_size"),
58
extend("x-chart-title" = "LRU Bucket Cache Size"),
59
extend("x-chart-units" = "entries"),
60
extend("x-chart-type" = "line"),
61
extend("x-chart-family" = "cache"),
62
- extend("x-chart-context" = "journal_viewer.bucket_lru_cache_size"),
62
+ extend("x-chart-context" = "otel_signal_viewer.bucket_lru_cache_size"),
63
)]
64
pub struct BucketCacheSizeMetrics {
65
#[schemars(extend("x-dimension-algorithm" = "absolute"))]
@@ -71,12 +71,12 @@ pub struct BucketCacheSizeMetrics {
71
/// Metrics for tracking bucket response types
72
#[derive(JsonSchema, NetdataChart, Default, Clone, PartialEq, Serialize, Deserialize)]
73
#[schemars(
74
- extend("x-chart-id" = "journal_viewer.bucket_responses"),
74
+ extend("x-chart-id" = "otel_signal_viewer.bucket_responses"),
75
extend("x-chart-title" = "Bucket Response Types"),
76
extend("x-chart-units" = "buckets/s"),
77
extend("x-chart-type" = "stacked"),
78
extend("x-chart-family" = "responses"),
79
- extend("x-chart-context" = "journal_viewer.bucket_responses"),
79
+ extend("x-chart-context" = "otel_signal_viewer.bucket_responses"),
80
)]
81
pub struct BucketResponseMetrics {
82
#[schemars(extend("x-dimension-algorithm" = "incremental"))]
@@ -88,12 +88,12 @@ pub struct BucketResponseMetrics {
88
/// Metrics for tracking histogram request characteristics
89
#[derive(JsonSchema, NetdataChart, Default, Clone, PartialEq, Serialize, Deserialize)]
90
#[schemars(
91
- extend("x-chart-id" = "journal_viewer.histogram_requests"),
91
+ extend("x-chart-id" = "otel_signal_viewer.histogram_requests"),
92
extend("x-chart-title" = "Histogram Request Details"),
93
extend("x-chart-units" = "count/s"),
94
extend("x-chart-type" = "line"),
95
extend("x-chart-family" = "requests"),
96
- extend("x-chart-context" = "journal_viewer.histogram_requests"),
96
+ extend("x-chart-context" = "otel_signal_viewer.histogram_requests"),
97
)]
98
pub struct HistogramRequestMetrics {
99
#[schemars(extend("x-dimension-algorithm" = "incremental"))]
src/crates/netdata-log-viewer/otel-signal-viewer-plugin/src/main.rs
renamed
+2
-2
@@ -1,4 +1,4 @@
1
-//! journal-viewer-plugin standalone binary
1
+//! otel-signal-viewer standalone binary
2
3
use journal_function::IndexingLimits;
4
use journal_registry::Monitor;
@@ -54,7 +54,7 @@ async fn run_plugin() -> std::result::Result<(), Box<dyn std::error::Error>> {
54
config.indexing.max_field_payload_size
55
);
56
57
- let mut runtime = PluginRuntime::new("journal-viewer");
57
+ let mut runtime = PluginRuntime::new("otel-signal-viewer");
58
info!("plugin runtime created");
59
60
let (monitor, notify_rx) = match Monitor::new() {
src/crates/netdata-log-viewer/otel-signal-viewer-plugin/src/plugin_config.rs
renamed
+2
-18
@@ -138,7 +138,7 @@ impl PluginConfig {
138
let user_config = netdata_env
139
.user_config_dir
140
.as_ref()
141
- .map(|path| path.join("journal-viewer.yaml"))
141
+ .map(|path| path.join("otel-signal-viewer.yaml"))
142
.and_then(|path| {
143
if path.exists() {
144
Config::from_yaml_file(&path)
@@ -154,7 +154,7 @@ impl PluginConfig {
154
} else if let Some(stock_path) = netdata_env
155
.stock_config_dir
156
.as_ref()
157
- .map(|p| p.join("journal-viewer.yaml"))
157
+ .map(|p| p.join("otel-signal-viewer.yaml"))
158
{
159
if stock_path.exists() {
160
Config::from_yaml_file(&stock_path).with_context(|| {
@@ -173,10 +173,6 @@ impl PluginConfig {
173
Config::default()
174
};
175
176
- // Resolve relative paths
177
- config.cache.directory =
178
- resolve_relative_path(&config.cache.directory, netdata_env.cache_dir.as_deref());
179
-
176
// Add host-prefixed journal paths for containerized environments
177
if let Some(ref host_prefix) = netdata_env.host_prefix {
178
if !host_prefix.is_empty() {
@@ -259,18 +255,6 @@ impl Config {
255
}
256
}
257
262
-/// Helper function to resolve relative paths against a base directory
263
-fn resolve_relative_path(path: &str, base_dir: Option<&Path>) -> String {
264
- let path = Path::new(path);
265
- if path.is_absolute() {
266
- path.to_string_lossy().to_string()
267
- } else if let Some(base) = base_dir {
268
- base.join(path).to_string_lossy().to_string()
269
- } else {
270
- path.to_string_lossy().to_string()
271
- }
272
-}
273
-
258
/// Standard journal paths that should be expanded with host prefix in containerized environments
259
const STANDARD_JOURNAL_PATHS: &[&str] = &["/var/log/journal", "/run/log/journal"];
260
src/crates/netdata-otel/otel-plugin/configs/otel.yaml.in
new
+51
@@ -0,0 +1,51 @@
1
+# OpenTelemetry Plugin Configuration
2
+# This file configures the OpenTelemetry metrics and logs ingestion for Netdata
3
+
4
+endpoint:
5
+ # gRPC endpoint to listen on for OpenTelemetry data
6
+ path: "127.0.0.1:4317"
7
+
8
+ # Path to TLS certificate file (enables TLS when provided)
9
+ tls_cert_path: null
10
+
11
+ # Path to TLS private key file (required when TLS certificate is provided)
12
+ tls_key_path: null
13
+
14
+ # Path to TLS CA certificate file for client authentication (optional)
15
+ tls_ca_cert_path: null
16
+
17
+metrics:
18
+ # Print flattened metrics to stdout for debugging, instead of ingesting them
19
+ print_flattened: false
20
+
21
+ # Number of samples to buffer for collection interval detection
22
+ buffer_samples: 10
23
+
24
+ # Maximum number of new charts to create per collection interval
25
+ throttle_charts: 100
26
+
27
+ # Directory with configuration files for mapping OTEL metrics to Netdata charts
28
+ chart_configs_dir: @configdir_POST@/otel.d/v1/metrics
29
+
30
+logs:
31
+ # Directory to store journal files for logs.
32
+ journal_dir: @logdir_POST@/otel/v1
33
+
34
+ # Maximum file size for individual journal files (e.g., "100MB", "1.5GB").
35
+ size_of_journal_file: "100MB"
36
+
37
+ # Maximum number of journal files to keep.
38
+ number_of_journal_files: 10
39
+
40
+ # Maximum total size for all journal files combined (e.g., "1GB", "500MB")
41
+ size_of_journal_files: "1GB"
42
+
43
+ # Maximum age for journal entries (e.g., "7 days", "1 week", "168h").
44
+ duration_of_journal_files: "7 days"
45
+
46
+ # Maximum time span that entries in a single journal file can cover (e.g., "2 hours", "1h", "30m").
47
+ duration_of_journal_file: "2 hours"
48
+
49
+ # Store the complete OTLP JSON representation in the OTLP_JSON field.
50
+ # When enabled, each log entry includes the full original JSON message for debugging and reprocessing.
51
+ store_otlp_json: false
src/crates/netdata-otel/otel-plugin/src/plugin_config.rs
+1
-21
@@ -211,7 +211,7 @@ impl PluginConfig {
211
pub fn new() -> Result<Self> {
212
let netdata_env = NetdataEnv::from_environment();
213
214
- let mut config = if netdata_env.running_under_netdata() {
214
+ let config = if netdata_env.running_under_netdata() {
215
// Try user config first, fallback to stock config
216
let user_config = netdata_env
217
.user_config_dir
@@ -241,14 +241,6 @@ impl PluginConfig {
241
Self::parse()
242
};
243
244
- // Resolve relative paths
245
- if let Some(charts_config_dir) = &mut config.metrics.chart_configs_dir {
246
- *charts_config_dir =
247
- resolve_relative_path(charts_config_dir, netdata_env.user_config_dir.as_deref());
248
- }
249
- config.logs.journal_dir =
250
- resolve_relative_path(&config.logs.journal_dir, netdata_env.log_dir.as_deref());
251
-
244
// Validate configuration
245
if config.metrics.buffer_samples == 0 {
246
anyhow::bail!("buffer_samples must be greater than 0");
@@ -306,15 +298,3 @@ impl PluginConfig {
298
Ok(config)
299
}
300
}
309
-
310
-// Helper function to resolve relative paths
311
-fn resolve_relative_path(path: &str, base_dir: Option<&Path>) -> String {
312
- let path = Path::new(path);
313
- if path.is_absolute() {
314
- path.to_string_lossy().to_string()
315
- } else if let Some(base) = base_dir {
316
- base.join(path).to_string_lossy().to_string()
317
- } else {
318
- path.to_string_lossy().to_string()
319
- }
320
-}
src/crates/netdata-plugin/rt/src/lib.rs
+1
-1
@@ -1001,7 +1001,7 @@ impl<R: AsyncRead + Unpin + Send, W: AsyncWrite + Unpin + Send> PluginRuntime<R,
1001
// we convert the frontend request from a GET to POST.
1002
let mut function_call = function_call;
1003
{
1004
- if function_call.name == "journal-viewer" {
1004
+ if function_call.name == "otel-signal-viewer" {
1005
if !function_call.args.is_empty() {
1006
let mut map = serde_json::Map::new();
1007
map.insert("info".to_string(), serde_json::json!(true));
src/daemon/dyncfg/README.md
+9
-5
@@ -417,13 +417,17 @@ The health module uses the high-level API with IDs like:
417
418
It supports multiple configuration objects, validation of alert definitions, and conversion between different configuration formats (JSON and traditional Netdata health configuration syntax).
419
420
-### journal-viewer-plugin (External Plugin)
420
+### systemd-journal.plugin (External Plugin)
421
422
-The journal-viewer-plugin is an external plugin written in Rust that provides systemd journal log viewing and analysis:
422
+The systemd-journal.plugin is an external plugin written in C that uses DynCfg to manage journal directory configurations:
423
424
-- **Location**: `src/crates/netdata-log-viewer/journal-viewer-plugin/`
425
-- Provides systemd journal log querying and visualization capabilities
426
-- Implements the Netdata plugin protocol for communication with the agent
424
+- `src/collectors/systemd-journal.plugin/systemd-journal-dyncfg.c`: Implements DynCfg integration
425
+- Registers as `systemd-journal:monitored-directories` ID
426
+- Uses a SINGLE configuration type for its directory list
427
+- Provides validation of directory paths for security
428
+- Implements GET and UPDATE commands
429
+
430
+This is a good example of an external plugin using the DynCfg system for a single configuration object.
431
432
### go.d.plugin (External Plugin)
433
src/libnetdata/facets/README.md
+1
-1
@@ -263,7 +263,7 @@ Log databases often contain out-of-order entries:
263
## Integration with Netdata Architecture
264
265
### Plugin Communication
266
-- The facets library runs within plugins
266
+- The facets library runs within plugins (systemd-journal.plugin, windows-events.plugin)
267
- Plugins can run anywhere in the Netdata ecosystem (parent nodes, child nodes, etc.)
268
- Communication with plugins happens via **rrdfunctions** - Netdata's function execution framework
269
- rrdfunctions provide the transport layer to send requests to plugins and receive responses
src/libnetdata/log/README.md
+1
-1
@@ -284,7 +284,7 @@ Netdata assigns unique UUIDs to specific event types for easy filtering and corr
284
| `d1f59606-dd4d-41e3-b217-a0cfcae8e632` | Extreme Cardinality | Metric cardinality exceeds safe limits |
285
| `4fdf4081-6c12-4623-a032-b7fe73beacb8` | User Configuration | Dynamic configuration changed by user |
286
287
-You can view these events using the Netdata `log-viewer` plugin at the `MESSAGE_ID` filter,
287
+You can view these events using the Netdata systemd-journal.plugin at the `MESSAGE_ID` filter,
288
or using `journalctl` like this:
289
290
```bash
src/libnetdata/log/nd_log-internals.c
-2
@@ -778,8 +778,6 @@ int nd_log_collectors_fd(void) {
778
if(nd_log.sources[NDLS_COLLECTORS].method == NDLM_FILE && nd_log.sources[NDLS_COLLECTORS].fd != -1)
779
return nd_log.sources[NDLS_COLLECTORS].fd;
780
781
- // For NDLM_JOURNAL, return STDERR_FILENO so the log-forwarder
782
- // can read from the pipe and properly format messages for journal
781
return STDERR_FILENO;
782
}
783
src/plugins.d/plugins_d.c
+17
-43
@@ -241,31 +241,31 @@ static void pluginsd_main_cleanup(void *pptr) {
241
worker_unregister();
242
}
243
244
+static bool is_plugin(char *dst, size_t dst_size, const char *filename) {
245
#if defined(OS_WINDOWS)
245
-static const char *plugin_suffixes[] = {
246
- ".plugin.exe",
247
- "_plugin.exe",
248
- "-plugin.exe",
249
- ".plugin",
250
- NULL
251
-};
246
+ const char *suffixes[] = {
247
+ ".plugin.exe",
248
+ "_plugin.exe",
249
+ "-plugin.exe",
250
+ ".plugin",
251
+ NULL
252
+ };
253
#else
253
-static const char *plugin_suffixes[] = {
254
- ".plugin",
255
- "_plugin",
256
- "-plugin",
257
- NULL
258
-};
254
+ const char *suffixes[] = {
255
+ ".plugin",
256
+ "_plugin",
257
+ "-plugin",
258
+ NULL
259
+ };
260
#endif
261
261
-static bool is_plugin(char *dst, size_t dst_size, const char *filename) {
262
size_t filename_len = strlen(filename);
263
264
- for (int i = 0; plugin_suffixes[i] != NULL; i++) {
265
- size_t suffix_len = strlen(plugin_suffixes[i]);
264
+ for (int i = 0; suffixes[i] != NULL; i++) {
265
+ size_t suffix_len = strlen(suffixes[i]);
266
267
if (filename_len > suffix_len &&
268
- strcmp(plugin_suffixes[i], &filename[filename_len - suffix_len]) == 0) {
268
+ strcmp(suffixes[i], &filename[filename_len - suffix_len]) == 0) {
269
snprintfz(dst, dst_size, "%.*s", (int)(filename_len - suffix_len), filename);
270
return true;
271
}
@@ -274,20 +274,6 @@ static bool is_plugin(char *dst, size_t dst_size, const char *filename) {
274
return false;
275
}
276
277
-static bool plugin_exists(const char *pluginname) {
278
- char path[FILENAME_MAX + 1];
279
-
280
- for (int idx = 0; idx < PLUGINSD_MAX_DIRECTORIES && plugin_directories[idx]; idx++) {
281
- for (int i = 0; plugin_suffixes[i] != NULL; i++) {
282
- snprintfz(path, sizeof(path), "%s/%s%s", plugin_directories[idx], pluginname, plugin_suffixes[i]);
283
- if (access(path, F_OK) == 0)
284
- return true;
285
- }
286
- }
287
-
288
- return false;
289
-}
290
-
277
void *pluginsd_main(void *ptr) {
278
CLEANUP_FUNCTION_REGISTER(pluginsd_main_cleanup) cleanup_ptr = ptr;
279
@@ -308,8 +294,6 @@ void *pluginsd_main(void *ptr) {
294
// so that we don't log broken directories on each loop
295
int directory_errors[PLUGINSD_MAX_DIRECTORIES] = { 0 };
296
311
- bool journal_viewer_exists = plugin_exists("journal-viewer");
312
-
297
while (service_running(SERVICE_COLLECTORS)) {
298
int idx;
299
const char *directory_name;
@@ -344,16 +328,6 @@ void *pluginsd_main(void *ptr) {
328
continue;
329
}
330
347
- // Blacklist obsolete plugins that have been replaced
348
- if (strcmp(pluginname, "systemd-journal") == 0 && journal_viewer_exists) {
349
- static bool logged = false;
350
- if (!logged) {
351
- netdata_log_info("plugin '%s' has been replaced by 'journal-viewer' and is no longer supported", file->d_name);
352
- logged = true;
353
- }
354
- continue;
355
- }
356
-
331
int enabled = inicfg_get_boolean(&netdata_config, CONFIG_SECTION_PLUGINS, pluginname, automatic_run);
332
if (unlikely(!enabled)) {
333
netdata_log_debug(D_PLUGINSD, "plugin '%s' is not enabled", file->d_name);