@cryptotaxi247 / netdata-1 / commits / e599f4679

Improve systemd journal logs documentation (#20184)

kanelatechnical committed Apr 28, 2025 at 14:49 UTC e599f467972de07440420f2516582033fe8c13e9
1 file changed +415 -324
src/collectors/systemd-journal.plugin/README.md
+415 -324
@@ -1,472 +1,563 @@
1 # `systemd` journal plugin
2
3 -[KEY FEATURES](#key-features) | [JOURNAL SOURCES](#journal-sources) | [JOURNAL FIELDS](#journal-fields) |
4 -[PLAY MODE](#play-mode) | [FULL TEXT SEARCH](#full-text-search) | [PERFORMANCE](#query-performance) |
5 -[CONFIGURATION](#configuration-and-maintenance) | [FAQ](#faq)
3 +[KEY FEATURES](#key-features) | [PREREQUISITES](#prerequisites) | [JOURNAL SOURCES](#journal-sources) | [JOURNAL FIELDS](#journal-fields) | [VISUALIZATION](#visualization-capabilities) | [PLAY MODE](#play-mode) | [FULL TEXT SEARCH](#full-text-search) | [QUERY PERFORMANCE](#query-performance) | [PERFORMANCE AT SCALE](#performance-at-scale) | [BEST PRACTICES](#best-practices-for-better-performance) | [CONFIGURATION](#configuration-and-maintenance) | [FAQ](#faq) | [HOW TO TROUBLESHOOT COMMON ISSUES](#how-to-troubleshoot-common-issues) | [HOW TO VERIFY SETUP](#how-to-verify-setup)
4
7 -The `systemd` journal plugin by Netdata makes viewing, exploring and analyzing `systemd` journal logs simple and
8 -efficient.
9 -It automatically discovers available journal sources, allows advanced filtering, offers interactive visual
10 -representations and supports exploring the logs of both individual servers and the logs on infrastructure wide
11 -journal centralization servers.
5 +The `systemd` journal plugin provides an efficient way to view, explore, and analyze `systemd` journal logs directly from the Netdata dashboard. It combines powerful filtering, real-time updates, and visual analysis tools to help you troubleshoot system issues effectively.
6
13 -![image](https://github.com/netdata/netdata/assets/2662304/691b7470-ec56-430c-8b81-0c9e49012679)
7 +![Netdata systemd journal plugin interface](https://github.com/netdata/netdata/assets/2662304/691b7470-ec56-430c-8b81-0c9e49012679)
8
15 -## Key features
9 +---
10
17 -- Works on both **individual servers** and **journal centralization servers**.
18 -- Supports `persistent` and `volatile` journals.
19 -- Supports `system`, `user`, `namespaces` and `remote` journals.
20 -- Allows filtering on **any journal field** or **field value**, for any time-frame.
21 -- Allows **full text search** (`grep`) on all journal fields, for any time-frame.
22 -- Provides a **histogram** for log entries over time, with a breakdown per field-value, for any field and any
23 - time-frame.
24 -- Works directly on journal files, without any other third-party components.
25 -- Supports coloring log entries, the same way `journalctl` does.
26 -- In PLAY mode provides the same experience as `journalctl -f`, showing new log entries immediately after they are
27 - received.
11 +## Key features
12
29 -### Prerequisites
13 +- **Unified view of logs** from multiple sources (system, user, namespace, remote)
14 +- **Real-time streaming** with PLAY mode for continuous monitoring
15 +- **Powerful filtering** on all journal fields with counters showing matching entries
16 +- **Full-text search** with wildcard and pattern matching
17 +- **Visual analysis** with interactive histograms showing log frequency
18 +- **Enriched field display** for improved readability (priorities, UIDs, timestamps)
19 +- **High performance** with intelligent sampling for large datasets
20 +- **Zero configuration** for immediate use on supported systems
21 +- **Multi-node support** for centralized log analysis
22 +- **UI-based exploration** without needing to learn complex journalctl syntax
23 +- **Integrated with Netdata's dashboard** for correlation with metrics
24
31 -`systemd-journal.plugin` is a Netdata Function Plugin.
25 +---
26
33 -To protect your privacy, as with all Netdata Functions, a free Netdata Cloud user account is required to access it.
34 -For more information, check [this discussion](https://github.com/netdata/netdata/discussions/16136).
27 +## Prerequisites
28
36 -### Limitations
29 +| Requirement | Details |
30 +|-------------|---------|
31 +| Netdata Agent v1.44+ | This plugin requires Netdata version 1.44 or newer |
32 +| Netdata Cloud account | Required to use Netdata Functions, including this plugin |
33 +| **Not supported in**: | Static builds (use Debian-based containers instead) |
34
38 -#### Plugin availability
35 +:::tip
36
40 -The following are limitations related to the availability of the plugin:
37 +This plugin is a Netdata Function Plugin. A free Netdata Cloud account is required. See the [Netdata Functions discussion](https://github.com/netdata/netdata/discussions/16136).
38
42 -- Netdata versions prior to 1.44 shipped in a docker container do not include this plugin.
43 - The problem is that `libsystemd` is not available in Alpine Linux (there is a `libsystemd`, but it is a dummy that
44 - returns failure on all calls). Starting with Netdata version 1.44, Netdata containers use a Debian base image
45 - making this plugin available when Netdata is running in a container.
46 -- For the same reason (lack of `systemd` support for Alpine Linux), the plugin is not available on `static` builds of
47 - Netdata (which are based on `muslc`, not `glibc`). If your Netdata is installed in `/opt/netdata` you most likely have
48 - a static build of Netdata.
49 -- On old systemd systems (like Centos 7), the plugin runs always in "full data query" mode, which makes it slower. The
50 - reason, is that systemd API is missing some important calls we need to use the field indexes of `systemd` journal.
51 - However, when running in this mode, the plugin offers also negative matches on the data (like filtering for all logs
52 - that do not have set some field), and this is the reason "full data query" mode is also offered as an option even on
53 - newer versions of `systemd`.
39 +:::
40
55 -#### `systemd` journal features
41 +The plugin is designed for native package installations, source installations, and Docker installations (Debian-based). If using Docker, make sure you're using the Debian-based containers.
42
57 -The following are limitations related to the features of `systemd` journal:
43 +---
44
59 -- This plugin assumes that binary field values are text fields with newlines in them. `systemd-journal` has the ability
60 - to support binary fields without specifying the nature of the binary data. However, binary fields are commonly used
61 - to store log entries that include multiple lines of text. The plugin treats all binary fields are multi-line text.
62 -- This plugin does not support multiple values per field for any given log entry. `systemd` journal has the ability to
63 - accept the same field key, multiple times, with multiple values on a single log entry. This plugin will present the
64 - last value and ignore the others for this log entry.
65 -- This plugin will only read journal files located in `/var/log/journal` or `/run/log/journal`. `systemd-journal-remote` has the
66 - ability to store journal files anywhere (user configured). If journal files are not located in `/var/log/journal`
67 - or `/run/log/journal` (and any of their subdirectories), the plugin will not find them. A simple solution is to link
68 - the other directories somewhere inside `/var/log/journal`. The plugin will pick them up, even if a subdirectory of
69 - `/var/log/journal` is a link to a directory outside `/var/log/journal`.
45 +## Journal sources
46
71 -Other than the above, this plugin supports all features of `systemd` journals.
47 +The plugin automatically detects available journal sources based on the journal files in `/var/log/journal` (persistent logs) and `/run/log/journal` (volatile logs).
48
73 -## Journal Sources
49 +![Journal sources selection interface](https://github.com/netdata/netdata/assets/2662304/28e63a3e-6809-4586-b3b0-80755f340e31)
50
75 -The plugin automatically detects the available journal sources, based on the journal files available in
76 -`/var/log/journal` (persistent logs) and `/run/log/journal` (volatile logs).
51 +By default, all sources merge into a unified view of log messages.
52
78 -![journal-sources](https://github.com/netdata/netdata/assets/2662304/28e63a3e-6809-4586-b3b0-80755f340e31)
53 +:::tip
54
80 -The plugin, by default, merges all journal sources to provide a unified view of all log messages available.
55 +Select a specific source before analyzing logs in depth to improve query performance.
56
82 -> To improve query performance, we recommend selecting the relevant journal source, before doing more analysis on the
83 -> logs.
57 +:::
58
85 -### `system` journals
59 +### System journals
60
87 -`system` journals are the default journals available on all `systemd` based systems.
61 +Default journals on all `systemd`-based systems. Includes:
62
89 -`system` journals contain:
63 +- Kernel log messages (`kmsg`)
64 +- Audit records
65 +- Syslog messages via `systemd-journald`
66 +- Output from service units
67 +- Native journal API messages
68
91 -- kernel log messages (via `kmsg`),
92 -- audit records, originating from the kernel audit subsystem,
93 -- messages received by `systemd-journald` via `syslog`,
94 -- messages received via the standard output and error of service units,
95 -- structured messages received via the native journal API.
69 +### User journals
70
97 -### `user` journals
71 +- Shows journal files for all users, not just the current user
72 +- Each regular user (UID > 999) typically has their own user journal
73 +- Merged into `remote` journals on centralization servers
74
99 -Unlike `journalctl`, the Netdata plugin allows viewing, exploring and querying the journal files of **all users**.
75 +### Namespace journals
76
101 -By default, each user, with a UID outside the range of system users (0–999), dynamic service users,
102 -and the nobody user (65,534), will get their own set of `user` journal files. For more information about
103 -this policy, check [Users, Groups, UIDs and GIDs on systemd Systems](https://systemd.io/UIDS-GIDS/).
77 +- Isolate log streams per project or service
78 +- Set with `LogNamespace=` in systemd unit files
79 +- Requires special setup to propagate to central servers
80
105 -Keep in mind that `user` journals are merged with the `system` journals when they are propagated to a journal
106 -centralization server. So, at the centralization server, the `remote` journals contain both the `system` and `user`
107 -journals of the sender.
81 +### Remote journals
82
109 -### `namespaces` journals
83 +- Created by `systemd-journal-remote`
84 +- Typically named by sender IP, then resolved to hostname
85
111 -The plugin auto-detects the namespaces available and provides a list of all namespaces at the "sources" list on the UI.
86 +---
87
113 -Journal namespaces are both a mechanism for logically isolating the log stream of projects consisting
114 -of one or more services from the rest of the system and a mechanism for improving performance.
88 +## Journal fields
89
116 -`systemd` service units may be assigned to a specific journal namespace through the `LogNamespace=` unit file setting.
90 +`systemd` journals support dynamic fields per log entry. All fields and values are indexed for fast querying.
91
118 -Keep in mind that namespaces require special configuration to be propagated to a journal centralization server.
119 -This makes them a little more difficult to handle, from the administration perspective.
92 +Fields are enriched for readability:
93
121 -### `remote` journals
94 +| Field | Enrichment applied |
95 +|-------|-------------------|
96 +| `_BOOT_ID` | Timestamp of first message for boot |
97 +| `PRIORITY` | Human-readable priority name |
98 +| `SYSLOG_FACILITY` | Named facility |
99 +| `ERRNO` | Error name |
100 +| UID and GID fields | Resolved to user and group names |
101 +| `_CAP_EFFECTIVE` | Human-readable capabilities |
102 +| `_SOURCE_REALTIME_TIMESTAMP` | UTC timestamp |
103 +| `MESSAGE_ID` | Well-known event name if known |
104
123 -Remote journals are created by `systemd-journal-remote`. This `systemd` feature allows creating logs centralization
124 -points within your infrastructure, based exclusively on `systemd`.
105 +:::tip
106
126 -Usually `remote` journals are named by the IP of the server sending these logs. The Netdata plugin automatically
127 -extracts these IPs and performs a reverse DNS lookup to find their hostnames. When this is successful,
128 -`remote` journals are named by the hostnames of the origin servers.
107 +Enrichments are visual only and not searchable. UID/GID values are based on the system where the plugin runs.
108
130 -For information about configuring a journal centralization server,
131 -check [this FAQ item](#how-do-i-configure-a-journal-centralization-server).
109 +:::
110
133 -## Journal Fields
111 +### Fields in the table
112
135 -`systemd` journals are designed to support multiple fields per log entry. The power of `systemd` journals is that,
136 -unlike other log management systems, it supports dynamic and variable fields for each log message,
137 -while all fields and their values are indexed for fast querying.
113 +Use the ⚙️ icon above the table to select fields to display as columns.
114
139 -This means that each application can log messages annotated with its own unique fields and values, and `systemd`
140 -journals will automatically index all of them, without any configuration or manual action.
115 +![Table field customization interface](https://github.com/netdata/netdata/assets/2662304/cd75fb55-6821-43d4-a2aa-033792c7f7ac)
116
142 -For a description of the most frequent fields found in `systemd` journals, check `man systemd.journal-fields`.
117 +The table view provides a powerful way to analyze logs with:
118 +- Scrollable list of log entries with customizable columns
119 +- Color-coded PRIORITY levels for quick identification of issues
120 +- Clickable entries for detailed viewing
121 +- Time-ordered display (newest first by default)
122 +- Pagination controls to navigate large datasets
123
144 -Fields found in the journal files are automatically added to the UI in multiple places to help you explore
145 -and filter the data.
124 +### Fields in the sidebar
125
147 -The plugin automatically enriches certain fields to make them more user-friendly:
126 +Click a log entry to open the right-hand info panel showing all fields for that entry.
127
149 -- `_BOOT_ID`: the hex value is annotated with the timestamp of the first message encountered for this boot id.
150 -- `PRIORITY`: the numeric value is replaced with the human-readable name of each priority.
151 -- `SYSLOG_FACILITY`: the encoded value is replaced with the human-readable name of each facility.
152 -- `ERRNO`: the numeric value is annotated with the short name of each value.
153 -- `_UID` `_AUDIT_LOGINUID`, `_SYSTEMD_OWNER_UID`, `OBJECT_UID`, `OBJECT_SYSTEMD_OWNER_UID`, `OBJECT_AUDIT_LOGINUID`:
154 - the local user database is consulted to annotate them with usernames.
155 -- `_GID`, `OBJECT_GID`: the local group database is consulted to annotate them with group names.
156 -- `_CAP_EFFECTIVE`: the encoded value is annotated with a human-readable list of the linux capabilities.
157 -- `_SOURCE_REALTIME_TIMESTAMP`: the numeric value is annotated with human-readable datetime in UTC.
158 -- `MESSAGE_ID`: for the known `MESSAGE_ID`s, the value is replaced with the well-known name of the event.
128 +![Sidebar field display](https://github.com/netdata/netdata/assets/2662304/3207794c-a61b-444c-8ffe-6c07cbc90ae2)
129
160 -The values of all other fields are presented as found in the journals.
130 +The sidebar shows:
131 +- Every field present in the selected journal entry
132 +- Raw and enriched field values
133 +- Copyable text for sharing or further analysis
134 +- Quick filtering options for any field value
135
162 -> IMPORTANT:
163 -> The UID and GID annotations are added during presentation and are taken from the server running the plugin.
164 -> For `remote` sources, the names presented may not reflect the actual user and group names on the origin server.
165 -> The numeric value will still be visible though, as-is on the origin server.
136 +### Fields as filters
137
167 -The annotations are not searchable with full-text search. They are only added for the presentation of the fields.
138 +The plugin offers select fields as filters with counters. Field allowlists and blocklists protect performance.
139
169 -### Journal fields as columns in the table
140 +:::tip
141
171 -All journal fields available in the journal files are offered as columns on the UI. Use the gear button above the table:
142 +"Full data queries" mode enables negative/empty matches but may slow performance.
143
173 -![image](https://github.com/netdata/netdata/assets/2662304/cd75fb55-6821-43d4-a2aa-033792c7f7ac)
144 +:::
145
175 -### Journal fields as additional info to each log entry
146 +![Field filtering interface](https://github.com/netdata/netdata/assets/2662304/ac710d46-07c2-487b-8ce3-e7f767b9ae0f)
147
177 -When you click a log line, the `info` sidebar will open on the right of the screen, to provide the full list of fields
178 -related to this log line. You can close this `info` sidebar, by selecting the filter icon at its top.
148 +Key filter features:
149 +- Real-time counters showing matching entry counts
150 +- Multi-select capability for each field
151 +- Toggleable inclusion/exclusion mode
152 +- Persistent filter selections across page reloads
153
180 -![image](https://github.com/netdata/netdata/assets/2662304/3207794c-a61b-444c-8ffe-6c07cbc90ae2)
154 +### Fields as histogram sources
155
182 -### Journal fields as filters
156 +Histograms visualize log frequency per field value over time. Supports:
157
184 -The plugin presents a select list of fields as filters to the query, with counters for each of the possible values
185 -for the field. This list can be used to quickly check which fields and values are available for the entire time-frame
186 -of the query.
158 +- Zoom
159 +- Pan
160 +- Click-to-navigate
161
188 -Internally, the plugin has:
162 +![Log frequency histogram](https://github.com/netdata/netdata/assets/2662304/d3dcb1d1-daf4-49cf-9663-91b5b3099c2d)
163
190 -1. An allowlist of fields to be presented as filters.
191 -2. A blocklist of fields to prevent them from becoming filters. This list includes fields with a very high
192 - cardinality, like timestamps, unique message ids, etc. This is mainly for protecting the server's performance,
193 - to avoid building in memory indexes for the fields that almost each of their values is unique.
164 +---
165
195 -Keep in mind that the values presented in the filters and their sorting are affected by the "full data queries"
196 -setting:
166 +## Visualization capabilities
167
198 -![image](https://github.com/netdata/netdata/assets/2662304/ac710d46-07c2-487b-8ce3-e7f767b9ae0f)
168 +The plugin offers several visualization features to help you understand and navigate your logs effectively.
169
200 -When "full data queries" is off, empty values are hidden and cannot be selected. This is due to a limitation of
201 -`libsystemd` that does not allow negative or empty matches. Also, values with zero counters may appear in the list.
170 +### Timeline view
171
203 -When "full data queries" is on, Netdata is applying all filtering to the data (not `libsystemd`), but this means
204 -that all the data of the entire time-frame, without any filtering applied, have to be read by the plugin to prepare
205 -the response required. So, "full data queries" can be significantly slower over long time-frames.
172 +The timeline at the top of the interface shows:
173 +- Log frequency distribution over time
174 +- Interactive zoom and pan controls
175 +- Time selection capabilities
176 +- Anomaly highlighting
177
207 -### Journal fields as histogram sources
178 +### Histograms
179
209 -The plugin visualizes log entry distribution over time using a histogram.
180 +Field-specific histograms provide:
181 +- Visual breakdown of log entries by field value
182 +- Color-coded frequency indicators
183 +- Click-to-filter capability
184 +- Time correlation with the main timeline
185
211 -The data source of this histogram can be any of the fields that are available as filters.
212 -For each of the values this field has, across the entire time-frame of the query, the histogram will get corresponding
213 -dimensions, showing the number of log entries, per value, over time.
186 +### Color coding
187
215 -The granularity of the histogram is adjusted automatically to have about 150 columns visible on screen.
188 +The plugin uses color to enhance readability:
189 +- Priority levels (emergency, alert, critical, etc.) have distinct colors
190 +- Selected filters are highlighted
191 +- Active elements use consistent color indicators
192 +- Error states and warnings have clear visual differentiation
193
217 -The histogram presented by the plugin is interactive:
194 +### UI navigation
195
219 -- **Zoom**, either with the global date-time picker or the zoom tool in the histogram's toolbox.
220 -- **Pan**, either with global date-time picker or by dragging with the mouse the chart to the left or the right.
221 -- **Click**, to quickly jump to the highlighted point in time in the log entries.
196 +The interface offers several ways to navigate logs:
197 +- Scroll through paginated results
198 +- Jump to specific timeframes
199 +- Click on histogram bars to focus on specific values
200 +- Use filter panels to narrow down results
201 +- Toggle between data views
202
223 -![image](https://github.com/netdata/netdata/assets/2662304/d3dcb1d1-daf4-49cf-9663-91b5b3099c2d)
203 +---
204
205 ## PLAY mode
206
227 -The plugin supports PLAY mode to continuously update the screen with new log entries found in the journal files.
228 -Hit the "play" button at the top of the Netdata dashboard screen.
207 +The plugin supports PLAY mode for real-time log streaming. Click the ▶️ button at the top of the dashboard to activate it.
208 +
209 +- Continuously updates the screen with newly received logs
210 +- Works for both single nodes and centralized log servers
211 +
212 +:::tip
213
230 -On centralized log servers, PLAY mode provides a unified view of all the new logs encountered across the entire
231 -infrastructure,
232 -from all hosts sending logs to the central logs server via `systemd-remote`.
214 +**PLAY** mode offers a similar experience to `journalctl -f`, but with visual enhancements.
215
234 -## Full-text search
216 +:::
217
236 -The plugin supports searching through all fields of log entries using text patterns.
218 +---
219
238 -**Pattern Matching**:
220 +## Full text search
221
240 -- All patterns use "contains" matching (not exact match).
241 -- Patterns support wildcards (`*`) to match any characters.
242 -- Example: `error` matches "error", "errors", "error_count", etc.
243 -- Example with wildcard: `a*b` matches anything containing `a` followed by `b`, like `acb`, `a_long_b`.
222 +The plugin supports full-text search using flexible pattern matching:
223
245 -**Multiple Patterns**:
224 +| Feature | Description |
225 +|---------|-------------|
226 +| Contains match | Default pattern style (e.g., `error` matches `error`, `error_count`) |
227 +| Wildcards | `*` matches any characters (e.g., `a*b` matches `acb`, `a_long_b`) |
228 +| Multiple patterns | Separate with `\|` for OR logic (e.g., `error\|warning` matches lines containing either "error" OR "warning") |
229 +| Negation | Prefix with `!` to exclude (e.g., `!systemd\|*` excludes lines with `systemd`) |
230
247 -- Separate patterns with `|` (pipe character).
248 -- Multiple patterns are combined with OR logic.
249 -- Example: `error|warning` matches entries containing either `error` OR `warning`.
231 +:::tip
232
251 -**Negative Patterns**:
233 +Full-text search applies across all fields. Combine with filters for precise results.
234
253 -- Prefix pattern with `!` to exclude matching entries.
254 -- Example: `!systemd|*` matches any entry that does NOT contain `systemd`.
235 +:::
236 +
237 +---
238
239 ## Query performance
240
258 -Journal files are designed to be accessed by multiple readers and one writer, concurrently.
241 +The plugin reads journal files directly using `libsystemd`, supporting concurrent readers and one writer.
242 +
243 +Query performance depends on several factors:
244
260 -Readers (like this Netdata plugin), open the journal files and `libsystemd`, behind the scenes, maps regions
261 -of the files into memory, to satisfy each query.
245 +| Factor | Impact on Performance |
246 +|--------|----------------------|
247 +| Number of journal files queried | Fewer files lead to faster queries |
248 +| Disk speed | Faster disks improve query times |
249 +| Available memory | More memory allows better caching |
250 +| Filters applied | Using fewer filters speeds up the query |
251
263 -On logs aggregation servers, the performance of the queries depends on the following factors:
252 +For best performance:
253 +- Keep the visible **timeframe short**
254 +- **Limit the number of rows** displayed
255 +- **Apply filters** to reduce the dataset
256 +- **Use specific sources** instead of querying across all journals
257
265 -1. The **number of files** involved in each query.
258 +---
259
267 - This is why we suggest selecting a source when possible.
260 +## Performance at scale
261
269 -2. The **speed of the disks** hosting the journal files.
262 +The plugin handles large datasets efficiently using a sampling algorithm, ensuring responsive queries even on busy log servers.
263
271 - Journal files perform a lot of reading while querying, so the faster the disks, the faster the query will finish.
264 +### How sampling works
265
273 -3. The **memory available** for caching parts of the files.
266 +| Step | Description |
267 +|------|-------------|
268 +| 1 | **Fully evaluates the latest 500,000 log entries** |
269 +| 2 | **Distributes evaluation** across journal files for **up to 1 million entries** |
270 +| 3 | **Marks additional entries** as `[unsampled]` beyond the evaluation budget |
271 +| 4 | **Estimates counts** as `[estimated]` once unsampled limits are hit |
272 +| 5 | Uses sequence numbers (if available) for **precise estimation** |
273 +| 6 | **Continues responsive histogram generation** while managing performance |
274
275 - Increased memory will help the kernel cache the most frequently used parts of the journal files, avoiding disk I/O
276 - and speeding up queries.
275 +The plugin uses a sophisticated algorithm that prioritizes newer logs while maintaining reasonable estimates for historical data. This approach ensures that even with terabytes of journal data, the interface remains responsive and usable.
276
278 -4. The **number of filters** applied.
277 +At scale, this plugin achieves up to **25–30x faster query performance** compared to `journalctl`, especially on multi-journal queries.
278
280 - Queries are significantly faster when just a few filters are selected.
279 +:::tip
280
282 -In general, for a faster experience, **keep a low number of rows within the visible timeframe**.
281 +The sampling algorithm is designed to be resilient to large datasets. Even if you see `[unsampled]` or `[estimated]` indicators, the results remain statistically representative of the full dataset.
282
284 -Even with long timeframes and a few tens of thousands of matching log entries, filtering will typically yield results in under a second. Large datasets with millions of entries can impact performance and lead to longer query times.
283 +:::
284
286 -The plugin aborts journal queries when your browser cancels inflight requests. This allows you to work on the UI
287 -while there are background queries running.
285 +---
286
289 -At the time of this writing, this Netdata plugin is about 25–30 times faster than `journalctl` on queries that access
290 -multiple journal files, over long time-frames.
287 +## Best practices for better performance
288
292 -During the development of this plugin, we submitted, to `systemd`, a number of patches to improve `journalctl`
293 -performance by a factor of 14:
289 +`systemd-journal` is designed for **reliability first** and **performance second**. It uses deduplication, field linking, and compression to minimize disk footprint, but the structure of journal files can still result in higher disk I/O during queries.
290
295 -- <https://github.com/systemd/systemd/pull/29365>
296 -- <https://github.com/systemd/systemd/pull/29366>
297 -- <https://github.com/systemd/systemd/pull/29261>
291 +### Filesystem and storage recommendations
292
299 -However, even after these patches are merged, `journalctl` will still be 2x slower than this Netdata plugin,
300 -on multi-journal queries.
293 +| Recommendation | Benefit |
294 +|---------------|---------|
295 +| Use compressed filesystems (`ext4`, `btrfs`, `zfs`) | Reduces disk I/O by minimizing file size |
296 +| Use SSD or NVMe storage | Speeds up journal file reads |
297 +| Avoid small fragmented journal files | Prevents query slowdowns on busy centralization servers |
298
302 -The problem lies in the way `libsystemd` handles multi-journal file queries. To overcome this problem,
303 -the Netdata plugin queries each file individually, and then it merges the results to be returned.
304 -This is transparent, thanks to the `facets` library in `libnetdata` that handles on-the-fly indexing, filtering,
305 -and searching of any dataset, independently of its source.
299 +### Memory and caching
300
307 -## Performance at scale
301 +| Recommendation | Benefit |
302 +|---------------|---------|
303 +| Allocate more RAM for the system | Improves OS caching of journal files |
304 +| Query the same timeframe repeatedly | Benefits from cached journal data |
305 +| Limit query timeframes on large datasets | Reduces memory overhead and improves speed |
306
309 -On busy logs servers, or when querying long timeframes that match millions of log entries, the plugin has a sampling
310 -algorithm to allow it to respond promptly. It works like this:
311 -
312 -1. The latest 500k log entries are queried in full, evaluating all the fields of every single log entry. This evaluation
313 - allows counting the unique values per field, updating the counters next to each value at the filters section of the
314 - dashboard.
315 -2. When the latest 500k log entries have been processed and there is more data to read, the plugin evenly divides 500k
316 - more log entries to the number of journal files matched by the query. So, it will continue to evaluate all the fields
317 - of all log entries, up to the budget per file, aiming to fully query one million log entries in total.
318 -3. When the budget is hit for a given file, the plugin continues to scan log entries, but this time it does not evaluate
319 - the fields and their values, so the counters per field and value are not updated. These unsampled log entries are
320 - shown in the histogram with the label `[unsampled]`.
321 -4. The plugin continues to count `[unsampled]` entries until as many as sampled entries have been evaluated and at least
322 - 1% of the journal file has been processed.
323 -5. When the `[unsampled]` budget is exhausted, the plugin stops processing the journal file and based on the processing
324 - completed so far and the number of entries in the journal file, it estimates the remaining number of log entries in
325 - that file. This is shown as `[estimated]` at the histogram.
326 -6. In systemd versions 254 or later, the plugin fetches the unique sequence number of each log entry and calculates the percentage of the file matched by the query, versus the total number of the log entries in the journal file.
327 -7. In systemd versions prior to 254, the plugin estimates the number of entries the journal file contributes to the
328 - query, using the number of log entries matched it vs. the total duration the log file has entries for.
329 -
330 -The plugin efficiently handles datasets with tens of millions of log entries, providing real-time responses and accurate visualizations.
331 -
332 -By analyzing the latest 500 thousand log entries and 1% of all journal files, the plugin provides an accurate representation of the entire timeframe.
333 -
334 -Keep in mind that although the plugin is quite effective and responds promptly when there are hundreds of journal files
335 -matching a query, response times may be longer when there are several thousands of smaller files. Systemd versions 254+
336 -attempt to solve this problem by allowing `systemd-journal-remote` to create larger files. However, for systemd
337 -versions prior to 254, `systemd-journal-remote` creates files of up to 32MB each, which when running very busy
338 -journals centralization servers aggregating several thousands of log entries per second, the number of files can grow
339 -to several dozens of thousands quickly. In such setups, the plugin should ideally skip processing journal files
340 -entirely, relying solely on the estimations of the sequence of files each file is part of. However, this has not been
341 -implemented yet. To improve the query performance in such setups, the user has to query smaller timeframes.
342 -
343 -Another optimization taking place in huge journal centralization points is the initial scan of the database. The plugin
344 -needs to know the list of all journal files available, including the details of the first and the last message in each
345 -of them. When there are several thousands of files in a directory (like it usually happens in `/var/log/journal/remote`),
346 -directory listing and examination of each file can take a considerable amount of time (even `ls -l` takes minutes).
347 -To work around this problem, the plugin uses `inotify` to receive file updates immediately and scans the library from
348 -the newest to the oldest file, allowing the user interface to work immediately after startup, for the most recent
349 -timeframes.
350 -
351 -### Best practices for better performance
352 -
353 -systemd-journal has been designed **first to be reliable** and then to be fast. It includes several mechanisms to ensure
354 -minimal data loss under all conditions (e.g. disk corruption, tampering, forward secure sealing) and despite the fact
355 -that it utilizes several techniques to require minimal disk footprint (like deduplication of log entries, linking of
356 -values and fields, compression) the disk footprint of journal files remains significantly higher compared to other log
357 -management solutions.
358 -
359 -The higher disk footprint results in higher disk I/O during querying, since a lot more data has to be read from disk to
360 -evaluate a query. Query performance at scale can greatly benefit from using a compressed filesystem (ext4, btrfs, zfs)
361 -to store systemd-journal files.
362 -
363 -systemd-journal files are cached by the operating system. There is no database server to serve queries. Each file is
364 -opened and the query runs by directly accessing the data in it.
365 -
366 -Therefore, systemd-journal relies on the caching layer of the operating system to optimize query performance. The more
367 -RAM the system has, although it will not be reported as `used` (it will be reported as `cache`), the faster the queries
368 -will get. The first time a timeframe is accessed, the query performance will be slower, but further queries on the same
369 -timeframe will be significantly faster since journal data are now cached in memory.
370 -
371 -So, on busy logs centralization systems, queries performance can be improved significantly by using a compressed
372 -filesystem for storing the journal files, and higher amounts of RAM.
307 +:::tip
308 +
309 +Journal data is cached by the operating system. The more RAM available for caching, the faster your queries will be.
310 +
311 +:::
312 +
313 +### Query strategies
314 +
315 +| Strategy | Why it helps |
316 +|----------|-------------|
317 +| Narrow the timeframe of your queries | Minimizes data scanned per request |
318 +| Use specific filters and source selections | Limits the scope of journal files being queried |
319 +| Limit the number of rows returned in the UI | Keeps response times fast and manageable |
320 +| Enable PLAY mode only when necessary | Reduces continuous query load on the system |
321 +
322 +---
323
324 ## Configuration and maintenance
325
376 -This Netdata plugin does not require any configuration or maintenance.
326 +The Netdata `systemd` journal plugin is designed to work **out of the box** with minimal configuration.
327 +
328 +### Requirements
329 +
330 +| Requirement | Description |
331 +|------------|-------------|
332 +| Netdata Agent | Installed on the node or centralization server |
333 +| Journal files | Located in `/var/log/journal` (persistent) or `/run/log/journal` (volatile) |
334 +| Netdata Cloud account | Required to access Netdata Functions, including this plugin |
335 +
336 +:::tip
337 +
338 +No additional configuration is required for this plugin to operate on supported systems.
339 +
340 +:::
341 +
342 +### Maintenance considerations
343 +
344 +| Task | Purpose |
345 +|------|---------|
346 +| **Keep Netdata up to date** | Ensures plugin compatibility and performance optimizations |
347 +| **Monitor disk usage** of journal files | Prevents performance issues caused by excessive log volume |
348 +| **Verify** journal file **locations** | Confirms the plugin can access the intended sources |
349 +| **Review** source selections **periodically** | Adjusts scope as infrastructure changes |
350 +
351 +---
352
353 ## FAQ
354
355 ### Can I use this plugin on journal centralization servers?
356
382 -Yes. You can centralize your logs using `systemd-journal-remote`, and then install Netdata
383 -on this logs centralization server to explore the logs of all your infrastructure.
357 +Yes — you can centralize your logs using `systemd-journal-remote` and install Netdata on the centralization server to explore logs from your entire infrastructure.
358 +The plugin provides **multi-node views** and allows you to combine logs from multiple servers.
359 +
360 +:::tip
361
385 -This plugin will automatically provide multi-node views of your logs and also allow you to combine the logs
386 -of multiple servers, as you see fit.
362 +For details on configuring a journal centralization server, see the [journal centralization setup guide](/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/passive-journal-centralization-without-encryption.md).
363
388 -Check [configuring a logs centralization server](#how-do-i-configure-a-journal-centralization-server).
364 +:::
365
390 -### Can I use this plugin from a parent Netdata?
366 +### Can I use this plugin from a parent Netdata node?
367
392 -Yes. When your nodes are connected to a Netdata parent, all their functions are available
393 -via the parent's UI. So, from the parent UI, you can access the functions of all your nodes.
368 +Yes — if your nodes are connected to a Netdata parent, all their functions are accessible via the parent's UI.
369 +This includes access to the `systemd` journal plugin for each child node.
370
395 -Keep in mind that to protect your privacy, to access Netdata functions, you need a
396 -free Netdata Cloud account.
371 +### Does this plugin expose any data to Netdata Cloud?
372
398 -### Is any of my data exposed to Netdata Cloud from this plugin?
373 +No — when accessing the Agent directly, **no data is exposed to Netdata Cloud**.
374 +The Cloud account is only used for authentication. Data flows directly from your Netdata Agent to your web browser.
375
400 -No. When you access the Agent directly, none of your data passes through Netdata Cloud.
401 -You need a free Netdata Cloud account only to verify your identity and enable the use of
402 -Netdata Functions. Once this is done, all the data flow directly from your Netdata Agent
403 -to your web browser.
376 +:::tip
377
405 -Also check [this discussion](https://github.com/netdata/netdata/discussions/16136).
378 +When using `https://app.netdata.cloud`, communication is encrypted but data is not stored in Netdata Cloud.
379 +See [this discussion](https://github.com/netdata/netdata/discussions/16136) for more details.
380
407 -When you access Netdata via `https://app.netdata.cloud`, your data travel via Netdata Cloud,
408 -but they are not stored in Netdata Cloud. This is to allow you to access your Netdata Agents from
409 -anywhere. All communication from/to Netdata Cloud is encrypted.
381 +:::
382
383 ### What are `volatile` and `persistent` journals?
384
413 -`systemd` `journald` allows creating both `volatile` journals in a `tmpfs` ram drive,
414 -and `persistent` journals stored on disk.
385 +- **Persistent journals** are stored on disk in `/var/log/journal`
386 +- **Volatile journals** are kept in memory in `/run/log/journal` and cleared on reboot
387
416 -`volatile` journals are particularly useful when the system monitored is sensitive to
417 -disk I/O, or does not have any writable disks at all.
388 +:::tip
389
419 -For more information check `man systemd-journald`.
390 +For more, check `man systemd-journald`.
391
421 -### I centralize my logs with Loki. Why use Netdata for my journals?
392 +:::
393
423 -`systemd` journals have almost infinite cardinality at their labels, and all of them are indexed,
424 -even if every single message has unique fields and values.
394 +### I centralize my logs with Loki. Why use Netdata for journals?
395
426 -When you send `systemd` journal logs to Loki, even if you use the `relabel_rules` argument to
427 -`loki.source.journal` with a JSON format, you need to specify which of the fields from journald
428 -you want inherited by Loki. This means you need to know the most important fields beforehand.
429 -At the same time you loose all the flexibility `systemd` journal provides:
430 -**indexing on all fields and all their values**.
396 +`systemd` journals support **dynamic, high-cardinality labels** with all fields indexed by default.
397 +When sending logs to Loki, you must predefine which fields to include, reducing flexibility.
398
432 -Loki generally assumes that all logs are like a table. All entries in a stream share the same
433 -fields. But journald does exactly the opposite. Each log entry is unique and may have its own unique fields.
399 +Netdata reads journals directly, providing:
400
435 -So, Loki and `systemd-journal` are good for different use cases.
401 +| Feature | Netdata journal plugin | Loki |
402 +|---------|----------------------|------|
403 +| **Indexed on all fields** | ✔️ Yes | ❌ Only selected labels |
404 +| Supports **dynamic fields** | ✔️ Yes | ❌ Assumes fixed schema |
405 +| Requires **configuration** | ❌ No | ✔️ Yes (relabel rules, label selection) |
406
437 -`systemd-journal` already runs in your systems. You use it today. It is there inside all your systems
438 -collecting the system and applications logs. And for its use case, it has advantages over other
439 -centralization solutions. So, why not use it?
407 +:::tip
408
441 -### Is it worth building a `systemd` logs centralization server?
409 +Loki and `systemd` journals serve different use cases — they can complement, not replace, each other.
410
443 -Yes. It is straightforward, fast, and the software to do it is already in your systems.
411 +:::
412
445 -For application and system logs, `systemd` journal is ideal and the visibility you can get
446 -by centralizing your system logs and the use of this Netdata plugin, is unparalleled.
413 +### Is it worth setting up a `systemd` logs centralization server?
414 +
415 +Yes — the tools required are included in modern Linux systems, and setup is straightforward.
416 +Centralized logs provide high visibility with minimal overhead.
417
418 ### How do I configure a journal centralization server?
419
450 -A short summary to get journal server running can be found below.
451 -There are two strategies you can apply, when it comes down to a centralized server for `systemd` journal logs.
420 +Two main strategies:
421 +
422 +| Strategy | Description |
423 +|----------|------------|
424 +| **Active sources** | Central server **fetches logs** from each node |
425 +| **Passive sources** | Nodes **push logs** to the central server |
426 +
427 +:::tip
428 +
429 +See the [passive journal centralization without encryption guide](/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/passive-journal-centralization-without-encryption.md)
430 +or the [encrypted setup guide](https://github.com/netdata/netdata/blob/master/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald).
431 +
432 +:::
433 +
434 +### Limitations when using centralization
435 +
436 +| Limitation | Notes |
437 +|------------|-------|
438 +| Namespaces not supported by Docker | [Related issue](https://github.com/moby/moby/issues/41879) |
439 +| `systemd-journal-upload` does not handle namespaces automatically | Requires manual configuration per namespace |
440 +
441 +### How can I report bugs or request features?
442 +
443 +If you encounter issues or have ideas for improvements:
444 +
445 +1. Check the [existing GitHub issues](https://github.com/netdata/netdata/issues)
446 +2. Submit a new issue with detailed reproduction steps
447 +3. For feature requests, describe your use case clearly
448 +
449 +The plugin is actively maintained, and feedback helps improve it for everyone.
450 +
451 +### Can I customize the plugin's appearance or behavior?
452 +
453 +Currently, customization options are limited to:
454 +- Column selection in the table view
455 +- Filter configurations
456 +- Time range selection
457 +- Source selection
458 +
459 +Additional customization features may be added in future releases based on user feedback.
460 +
461 +---
462 +
463 +## How to troubleshoot common issues
464 +
465 +Use the following solutions to resolve common issues with the systemd journal plugin.
466 +
467 +### How to resolve plugin availability problems
468 +
469 +| Possible Cause | Solution |
470 +|---------------|----------|
471 +| Running an older Netdata version (pre-1.44) | Update Netdata to version **1.44 or later** |
472 +| Using Alpine-based Docker container | Use the **Debian-based Netdata container** (from 1.44+) |
473 +| Using a static build of Netdata (`/opt/netdata`) | Switch to a package-based installation or source build |
474 +| Missing `libsystemd` or required dependencies | Make sure `libsystemd` is installed on the host |
475 +
476 +### How to fix slow or timing out queries
477 +
478 +| Possible Cause | Solution |
479 +|---------------|----------|
480 +| Querying too many journal files at once | Select specific **sources** before running your query |
481 +| Long timeframes selected | Narrow the **timeframe** to improve performance |
482 +| Low disk speed or insufficient RAM | Use faster disks and increase memory for better caching |
483 +| Too many active filters | Reduce the number of **filters** applied |
484 +
485 +:::tip
486 +
487 +Sampling ensures responsiveness at scale, but selecting sources and filters remains the best way to optimize performance.
488 +
489 +:::
490 +
491 +### How to solve missing journal sources
492 +
493 +| Possible Cause | Solution |
494 +|---------------|----------|
495 +| Journals stored outside default paths | Create a **symlink** to `/var/log/journal` or `/run/log/journal` |
496 +| Journals not persistent across reboots | Configure `systemd-journald` to enable **persistent logs** with `Storage=persistent` in `/etc/systemd/journald.conf` |
497 +
498 +### How to address missing or incomplete logs
499 +
500 +| Possible Cause | Solution |
501 +|---------------|----------|
502 +| Journals rotated or deleted | Ensure **persistent storage** is enabled |
503 +| Misconfigured journal centralization | Check `systemd-journal-remote` and `systemd-journal-upload` settings |
504 +| Namespace logs not forwarded | Manually configure forwarding for **each namespace** |
505 +
506 +### How to fix UI rendering issues
507 +
508 +| Possible Cause | Solution |
509 +|---------------|----------|
510 +| Outdated browser | Update to the latest version of Chrome, Firefox, Safari, or Edge |
511 +| Zoom level affecting layout | Reset browser zoom to 100% |
512 +| Ad blockers or script blockers | Temporarily disable to test if they're interfering |
513 +| Network connectivity issues | Check network connections to Netdata server |
514 +
515 +### How to understand error messages
516 +
517 +| Error Message | Meaning | Solution |
518 +|--------------|---------|----------|
519 +| "Plugin not available" | The systemd-journal plugin isn't loaded | Check your Netdata installation type (must not be Alpine or static) |
520 +| "Unable to open journal" | Permission issues accessing journal files | Ensure Netdata has proper permissions for journal directories |
521 +| "Timeout while querying" | Query is taking too long to complete | Reduce the query scope with filters or shorter timeframes |
522 +| "No sources detected" | Cannot find valid journal files | Check journal file locations and setup |
523 +| "Source selection failed" | Selected source cannot be accessed | Verify the source exists and permissions are correct |
524 +
525 +---
526 +
527 +## How to verify setup
528 +
529 +### How to check if the plugin is running
530 +
531 +```bash
532 +sudo netdata -W plugins
533 +```
534 +
535 +Check that the `systemd-journal` plugin is listed as active.
536 +
537 +### How to confirm journal sources are detected
538 +
539 +1. Open the **Logs** tab in the Netdata UI
540 +2. Use the **Sources** filter on the right sidebar
541 +3. Ensure you can see your expected sources (e.g., `system`, `user`, `remote`, or specific namespaces)
542
453 -1. _Active sources_, where the centralized server fetches the logs from each server
454 -2. _Passive sources_, where the centralized server accepts a log stream from an individual server.
543 +:::tip
544
456 -For more options and reference to documentation, check `man systemd-journal-remote` and `man systemd-journal-upload`.
545 +If sources are missing, check the journal file locations and verify symlinks if needed.
546
458 -#### _passive_ journal centralization without encryption
547 +:::
548
460 -If you want to set up your own passive journal centralization setup without encryption, [check out guide on it](/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/passive-journal-centralization-without-encryption.md).
549 +### How to test basic queries
550
462 -#### _passive_ journal centralization with encryption using self-signed certificates
551 +- Apply a **simple filter** (like `PRIORITY=3`) and confirm logs are returned
552 +- Use **full-text search** (e.g., search for `error`) and verify results populate correctly
553 +- Toggle **PLAY mode** to confirm live logs are streaming
554
464 -If you want to set up your own passive journal centralization setup using self-signed certificates for encryption, [check out guide on it](https://github.com/netdata/netdata/blob/master/docs/observability-centralization-points/logs-centralization-points-with-systemd-journald).
555 +### How to validate plugin logs
556
466 -#### Limitations when using a logs centralization server
557 +Check the Netdata Agent logs for plugin startup messages:
558
468 -As of this writing `namespaces` support by `systemd` is limited:
559 +```bash
560 +sudo journalctl -u netdata | grep journal
561 +```
562
470 -- Docker containers cannot log to namespaces. Check [this issue](https://github.com/moby/moby/issues/41879).
471 -- `systemd-journal-upload` automatically uploads `system` and `user` journals, but not `namespaces` journals. For this
472 - you need to spawn a `systemd-journal-upload` per namespace.
563 +Look for lines confirming the journal plugin started successfully and detected sources.
\ No newline at end of file