Systemd-journal (#21222)
Kanela committed
Oct 28, 2025 at 12:40 UTC
33ef20264b0aac8d787b46401457ece83693a060
1 file changed
+45
-42
src/collectors/systemd-journal.plugin/README.md
+45
-42
@@ -6,8 +6,6 @@ The `systemd` journal plugin provides an efficient way to view, explore, and ana
6
7
8
9
----
10
-
9
## Key features
10
11
- **Unified view of logs** from multiple sources (system, user, namespace, remote)
@@ -22,8 +20,6 @@ The `systemd` journal plugin provides an efficient way to view, explore, and ana
20
- **UI-based exploration** without needing to learn complex journalctl syntax
21
- **Integrated with Netdata's dashboard** for correlation with metrics
22
25
----
26
-
23
## Prerequisites
24
25
| Requirement | Details |
@@ -40,8 +36,6 @@ This plugin is a Netdata Function Plugin. A free Netdata Cloud account is requir
36
37
The plugin is designed for native package installations, source installations, and Docker installations (Debian-based). If using Docker, make sure you're using the Debian-based containers.
38
43
----
44
-
39
## Journal sources
40
41
The plugin automatically detects available journal sources based on the journal files in `/var/log/journal` (persistent logs) and `/run/log/journal` (volatile logs).
@@ -83,8 +77,6 @@ Default journals on all `systemd`-based systems. Includes:
77
- Created by `systemd-journal-remote`
78
- Typically named by sender IP, then resolved to hostname
79
86
----
87
-
80
## Journal fields
81
82
`systemd` journals support dynamic fields per log entry. All fields and values are indexed for fast querying.
@@ -161,8 +153,6 @@ Histograms visualize log frequency per field value over time. Supports:
153
154
155
164
----
165
-
156
## Visualization capabilities
157
158
The plugin offers several visualization features to help you understand and navigate your logs effectively.
@@ -200,8 +190,6 @@ The interface offers several ways to navigate logs:
190
- Use filter panels to narrow down results
191
- Toggle between data views
192
203
----
204
-
193
## PLAY mode
194
195
The plugin supports PLAY mode for real-time log streaming. Click the ▶️ button at the top of the dashboard to activate it.
@@ -215,8 +203,6 @@ The plugin supports PLAY mode for real-time log streaming. Click the ▶️ butt
203
204
:::
205
218
----
219
-
206
## Full text search
207
208
The plugin supports full-text search using flexible pattern matching:
@@ -234,8 +220,6 @@ Full-text search applies across all fields. Combine with filters for precise res
220
221
:::
222
237
----
238
-
223
## Query performance
224
225
The plugin reads journal files directly using `libsystemd`, supporting concurrent readers and one writer.
@@ -255,8 +239,6 @@ For best performance:
239
- **Apply filters** to reduce the dataset
240
- **Use specific sources** instead of querying across all journals
241
258
----
259
-
242
## Performance at scale
243
244
The plugin handles large datasets efficiently using a sampling algorithm, ensuring responsive queries even on busy log servers.
@@ -282,31 +264,30 @@ The sampling algorithm is designed to be resilient to large datasets. Even if yo
264
265
:::
266
285
-#### Accuracy implications
286
-Netdata’s sampling budget evaluates **up to 1,000,000 log entries** before it ever marks rows as `[unsampled]`. The proportion of the dataset we examine is:
267
+### Accuracy implications
268
+
269
+Netdata's sampling budget evaluates **up to 1,000,000 log entries** before it ever marks rows as `[unsampled]`. The proportion of the dataset we examine is:
270
271
```
272
evaluated_entries = min(total_entries, 1_000_000)
273
evaluated_ratio = evaluated_entries / total_entries
274
```
275
293
-Because the sampling set is so large, percentage breakdowns stay tight even on massive datasets. For a 10 M–entry window where 60 % of logs share a value, the 95 % confidence interval around that percentage is:
276
+Because the sampling set is so large, percentage breakdowns stay tight even on massive datasets. For a 10 M–entry window where 60 % of logs share a value, the 95 % confidence interval around that percentage is:
277
278
```
279
standard_error ≈ sqrt(p * (1 - p) / evaluated_entries)
297
-CI95 ≈ 1.96 * standard_error = 1.96 * sqrt(0.6 * 0.4 / 1_000_000) ≈ ±0.9 %
280
+CI95 ≈ 1.96 * standard_error = 1.96 * sqrt(0.6 * 0.4 / 1_000_000) ≈ ±0.9 %
281
```
282
283
By contrast, evaluating only 5,000 entries (a small-sample approach typical of many log explorers when speed is prioritized) would yield:
284
285
```
303
-CI95 ≈ 1.96 * sqrt(0.6 * 0.4 / 5_000) ≈ ±8.7 %
286
+CI95 ≈ 1.96 * sqrt(0.6 * 0.4 / 5_000) ≈ ±8.7 %
287
```
288
289
The result is that even at extreme scale, mainly because Netdata samples 200x more data, it can provide significantly more accurate estimations on value distributions, at comparable performance.
290
308
----
309
-
291
## Best practices for better performance
292
293
`systemd-journal` is designed for **reliability first** and **performance second**. It uses deduplication, field linking, and compression to minimize disk footprint, but the structure of journal files can still result in higher disk I/O during queries.
@@ -342,8 +323,6 @@ Journal data is cached by the operating system. The more RAM available for cachi
323
| Limit the number of rows returned in the UI | Keeps response times fast and manageable |
324
| Enable PLAY mode only when necessary | Reduces continuous query load on the system |
325
345
----
346
-
326
## Configuration and maintenance
327
328
The Netdata `systemd` journal plugin is designed to work **out of the box** with minimal configuration.
@@ -371,11 +350,10 @@ No additional configuration is required for this plugin to operate on supported
350
| **Verify** journal file **locations** | Confirms the plugin can access the intended sources |
351
| **Review** source selections **periodically** | Adjusts scope as infrastructure changes |
352
374
----
375
-
353
## FAQ
354
378
-### Can I use this plugin on journal centralization servers?
355
+<details>
356
+<summary><strong>Can I use this plugin on journal centralization servers?</strong></summary>
357
358
Yes — you can centralize your logs using `systemd-journal-remote` and install Netdata on the centralization server to explore logs from your entire infrastructure.
359
The plugin provides **multi-node views** and allows you to combine logs from multiple servers.
@@ -386,12 +364,18 @@ For details on configuring a journal centralization server, see the [journal cen
364
365
:::
366
389
-### Can I use this plugin from a parent Netdata node?
367
+</details>
368
+
369
+<details>
370
+<summary><strong>Can I use this plugin from a parent Netdata node?</strong></summary>
371
372
Yes — if your nodes are connected to a Netdata parent, all their functions are accessible via the parent's UI.
373
This includes access to the `systemd` journal plugin for each child node.
374
394
-### Does this plugin expose any data to Netdata Cloud?
375
+</details>
376
+
377
+<details>
378
+<summary><strong>Does this plugin expose any data to Netdata Cloud?</strong></summary>
379
380
No — when accessing the Agent directly, **no data is exposed to Netdata Cloud**.
381
The Cloud account is only used for authentication. Data flows directly from your Netdata Agent to your web browser.
@@ -403,7 +387,10 @@ See [this discussion](https://github.com/netdata/netdata/discussions/16136) for
387
388
:::
389
406
-### What are `volatile` and `persistent` journals?
390
+</details>
391
+
392
+<details>
393
+<summary><strong>What are `volatile` and `persistent` journals?</strong></summary>
394
395
- **Persistent journals** are stored on disk in `/var/log/journal`
396
- **Volatile journals** are kept in memory in `/run/log/journal` and cleared on reboot
@@ -414,7 +401,10 @@ For more, check `man systemd-journald`.
401
402
:::
403
417
-### I centralize my logs with Loki. Why use Netdata for journals?
404
+</details>
405
+
406
+<details>
407
+<summary><strong>I centralize my logs with Loki. Why use Netdata for journals?</strong></summary>
408
409
`systemd` journals support **dynamic, high-cardinality labels** with all fields indexed by default.
410
When sending logs to Loki, you must predefine which fields to include, reducing flexibility.
@@ -433,12 +423,18 @@ Loki and `systemd` journals serve different use cases — they can complement, n
423
424
:::
425
436
-### Is it worth setting up a `systemd` logs centralization server?
426
+</details>
427
+
428
+<details>
429
+<summary><strong>Is it worth setting up a `systemd` logs centralization server?</strong></summary>
430
431
Yes — the tools required are included in modern Linux systems, and setup is straightforward.
432
Centralized logs provide high visibility with minimal overhead.
433
441
-### How do I configure a journal centralization server?
434
+</details>
435
+
436
+<details>
437
+<summary><strong>How do I configure a journal centralization server?</strong></summary>
438
439
Two main strategies:
440
@@ -454,14 +450,20 @@ or the [encrypted setup guide](https://github.com/netdata/netdata/blob/master/do
450
451
:::
452
457
-### Limitations when using centralization
453
+</details>
454
+
455
+<details>
456
+<summary><strong>What are the limitations when using centralization?</strong></summary>
457
458
| Limitation | Notes |
459
|------------|-------|
460
| Namespaces not supported by Docker | [Related issue](https://github.com/moby/moby/issues/41879) |
461
| `systemd-journal-upload` does not handle namespaces automatically | Requires manual configuration per namespace |
462
464
-### How can I report bugs or request features?
463
+</details>
464
+
465
+<details>
466
+<summary><strong>How can I report bugs or request features?</strong></summary>
467
468
If you encounter issues or have ideas for improvements:
469
@@ -471,7 +473,10 @@ If you encounter issues or have ideas for improvements:
473
474
The plugin is actively maintained, and feedback helps improve it for everyone.
475
474
-### Can I customize the plugin's appearance or behavior?
476
+</details>
477
+
478
+<details>
479
+<summary><strong>Can I customize the plugin's appearance or behavior?</strong></summary>
480
481
Currently, customization options are limited to:
482
- Column selection in the table view
@@ -481,7 +486,7 @@ Currently, customization options are limited to:
486
487
Additional customization features may be added in future releases based on user feedback.
488
484
----
489
+</details>
490
491
## How to troubleshoot common issues
492
@@ -545,8 +550,6 @@ Sampling ensures responsiveness at scale, but selecting sources and filters rema
550
| "No sources detected" | Cannot find valid journal files | Check journal file locations and setup |
551
| "Source selection failed" | Selected source cannot be accessed | Verify the source exists and permissions are correct |
552
548
----
549
-
553
## How to verify setup
554
555
### How to check if the plugin is running