d08615593 · cryptotaxi247/netdata-1

@@ -40,31 +40,34 @@ For more information check [this discussion](https://github.com/netdata/netdata/

40

41 The following are limitations related to the availability of the plugin:

42

43 -- This plugin is not available when Netdata is installed in a container. The problem is that `libsystemd` is not

44 - available in Alpine Linux (there is a `libsystemd`, but it is a dummy that returns failure on all calls). We plan to

45 - change this, by shipping Netdata containers based on Debian.

43 +- Netdata versions prior to 1.44 shipped in a docker container do not include this plugin.

44 + The problem is that `libsystemd` is not available in Alpine Linux (there is a `libsystemd`, but it is a dummy that

45 + returns failure on all calls). Starting with Netdata version 1.44, Netdata containers use a Debian base image

46 + making this plugin available when Netdata is running in a container.

47 - For the same reason (lack of `systemd` support for Alpine Linux), the plugin is not available on `static` builds of

47 - Netdata (which are based on `muslc`, not `glibc`).

48 + Netdata (which are based on `muslc`, not `glibc`). If your Netdata is installed in `/opt/netdata` you most likely have

49 + a static build of Netdata.

50 - On old systemd systems (like Centos 7), the plugin runs always in "full data query" mode, which makes it slower. The

51 reason, is that systemd API is missing some important calls we need to use the field indexes of `systemd` journal.

52 However, when running in this mode, the plugin offers also negative matches on the data (like filtering for all logs

53 that do not have set some field), and this is the reason "full data query" mode is also offered as an option even on

54 newer versions of `systemd`.

55

54 -To use the plugin, install one of our native distribution packages, or install it from source.

55 -

56 #### `systemd` journal features

57

58 The following are limitations related to the features of `systemd` journal:

59

60 -- This plugin does not support binary field values. `systemd` journal has the ability to assign fields with binary data.

61 - This plugin assumes all fields contain text values (text in this context includes numbers).

60 +- This plugin assumes that binary field values are text fields with newlines in them. `systemd-journal` has the ability

61 + to support binary fields, without specifying the nature of the binary data. However, binary fields are commonly used

62 + to store log entries that include multiple lines of text. The plugin treats all binary fields are multi-line text.

63 - This plugin does not support multiple values per field for any given log entry. `systemd` journal has the ability to

64 accept the same field key, multiple times, with multiple values on a single log entry. This plugin will present the

65 last value and ignore the others for this log entry.

66 - This plugin will only read journal files located in `/var/log/journal` or `/run/log/journal`. `systemd-journal-remote` has the

67 ability to store journal files anywhere (user configured). If journal files are not located in `/var/log/journal`

67 - or `/run/log/journal` (and any of their subdirectories), the plugin will not find them.

68 + or `/run/log/journal` (and any of their subdirectories), the plugin will not find them. A simple solution is to link

69 + the other directories somewhere inside `/var/log/journal`. The plugin will pick them up, even if a sub-directory of

70 + `/var/log/journal` is a link to a directory outside `/var/log/journal`.

71

72 Other than the above, this plugin supports all features of `systemd` journals.

73

@@ -153,6 +156,7 @@ The plugin automatically enriches certain fields to make them more user-friendly

156 - `_GID`, `OBJECT_GID`: the local group database is consulted to annotate them with group names.

157 - `_CAP_EFFECTIVE`: the encoded value is annotated with a human-readable list of the linux capabilities.

158 - `_SOURCE_REALTIME_TIMESTAMP`: the numeric value is annotated with human-readable datetime in UTC.

159 +- `MESSAGE_ID`: for the known `MESSAGE_ID`s, the value is replaced with the well known name of the event.

160

161 The values of all other fields are presented as found in the journals.

162

@@ -295,6 +299,70 @@ the Netdata plugin queries each file individually and it then it merges the resu

299 This is transparent, thanks to the `facets` library in `libnetdata` that handles on-the-fly indexing, filtering,

300 and searching of any dataset, independently of its source.

301

302 +## Performance at scale

303 +

304 +On busy logs servers, or when querying long timeframes that match millions of log entries, the plugin has a sampling

305 +algorithm to allow it respond promptly. It works like this:

306 +

307 +1. The latest 500k log entries are queried in full, meaning that the log entries are queried in full, evaluating all the

308 + fields of every single log entry. This evaluation allows counting the unique values per field, updating the counters

309 + next to each value at the filters section of the dashboard.

310 +2. When the latest 500k log entries have been processed and there are more data to read, the plugin divides evenly 500k

311 + more log entries to the number of journal files matched by the query. So, it will continue to evaluate all the fields

312 + of all log entries, up to the budget per file, aiming to fully query 1 million log entries in total.

313 +3. When the budget is hit for a given file, the plugin continues to scan log entries, but this time it does not evaluate

314 + the fields and their values, so the counters per field and value are not updated. These unsampled log entries are

315 + shown in the histogram with the label `[unsampled]`.

316 +4. The plugin continues to count `[unsampled]` entries until as many as sampled entries have been evaluated and at least

317 + 1% of the journal file has been processed.

318 +5. When the `[unsampled]` budget is exhausted, the plugin stops processing the journal file and based on the processing

319 + completed so far and the number of entries in the journal file, it estimates the remaining number of log entries in

320 + that file. This is shown as `[estimated]` at the histogram.

321 +6. In systemd versions 254 or later, the plugin fetches the unique sequence number of each log entry and calculates the

322 + the percentage of the file matched by the query, versus the total number of the log entries in the journal file.

323 +7. In systemd versions prior to 254, the plugin estimates the number of entries the journal file contributes to the

324 + query, using the amount of log entries matched it vs. the total duration the log file has entries for.

325 +

326 +The above allow the plugin to respond promptly even when the number of log entries in the journal files is several

327 +dozens millions, while providing accurate estimations of the log entries over time at the histogram and enough counters

328 +at the fields filtering section to help users get an overview of the whole timeframe.

329 +

330 +The fact that the latest 500k log entries and 1% of all journal files (which are spread over time) have been fully

331 +evaluated, including counting the number of appearances for each field value, the plugin usually provides an accurate

332 +representation of the whole timeframe.

333 +

334 +Keep in mind that although the plugin is quite effective and responds promptly when there are hundreds of journal files

335 +matching a query, response times may be longer when there are several thousands of smaller files. systemd versions 254+

336 +attempt to solve this problem by allowing `systemd-journal-remote` to create larger files. However, for systemd

337 +versions prior to 254, `systemd-journal-remote` creates files of up to 32MB each, which when running very busy

338 +journals centralization servers aggregating several thousands of log entries per second, the number of files can grow

339 +to several dozens of thousands quickly. In such setups, the plugin should ideally skip processing journal files

340 +entirely, relying solely on the estimations of the sequence of files each file is part of. However, this has not been

341 +implemented yet. To improve the query performance in such setups, the user has to query smaller timeframes.

342 +

343 +Another optimization taking place in huge journal centralization points, is the initial scan of the database. The plugin

344 +needs to know the list of all journal files available, including the details of the first and the last message in each

345 +of them. When there are several thousands of files in a directory (like it usually happens in `/var/log/journal/remote`),

346 +directory listing and examination of each file can take a considerable amount of time (even `ls -l` takes minutes).

347 +To work around this problem, the plugin uses `inotify` to receive file updates immediately and scans the library from

348 +the newest to the oldest file, allowing the user interface to work immediately after startup, for the most recent

349 +timeframes.

350 +

351 +systemd-journal has been designed first to be reliable and then to be fast. It includes several mechanisms to ensure

352 +minimal data loss under all conditions (e.g. disk corruption, tampering, forward secure sealing) and despite the fact

353 +that it utilizes several techniques to require minimal disk footprint (like deduplication of log entries, linking of

354 +values and fields, compression) the disk footprint of journal files remains significantly higher compared to other log

355 +management solutions. The higher disk footprint results in higher disk I/O during querying, since a lot more data have

356 +to read from disk to evaluate a query. Query performance at scale can greatly be improved by utilizing a compressed

357 +filesystem (ext4, btrfs, zfs) to store systemd-journal files.

358 +

359 +systemd-journal files are cached by the operating system. There is no database server to serve queries. Each file is

360 +opened and the query runs by directly accessing the data in it. Therefore systemd-journal relies on the caching

361 +layer of the operating system to optimize query performance. The more RAM the system has, although it will not be

362 +reported as `used` (it will be reported as `cache`), the faster the queries will get. The first time a timeframe is

363 +accessed the query performance will be slower, but further queries on the same timeframe will be significantly faster

364 +since journal data are now cached in memory.

365 +

366 ## Configuration and maintenance

367

368 This Netdata plugin does not require any configuration or maintenance.

Update README.md