docs required for release (#16546)
* improve journal plugin docs * added systemd-cat-native.md * updated log2journal docs * added screenshot to log2journal
Costa Tsaousis committed
Dec 5, 2023 at 17:05 UTC
261837989ef9f125cd3d47321b751c5f5cb83cc7
5 files changed
+631
-238
collectors/log2journal/README.md
+220
-169
@@ -6,6 +6,11 @@ By combining these tools you can create advanced log processing pipelines sendin
6
7
The process involves the usual piping of shell commands, to get and process the log files in realtime.
8
9
+The result is like this: nginx logs into systemd-journal:
10
+
11
+
12
+
13
+
14
The overall process looks like this:
15
16
```bash
@@ -14,7 +19,7 @@ tail -F /var/log/nginx/*.log |\ # outputs log lines
19
systemd-cat-native # send to local/remote journald
20
```
21
17
-Let's see the steps:
22
+These are the steps:
23
24
1. `tail -F /var/log/nginx/*.log`<br/>this command will tail all `*.log` files in `/var/log/nginx/`. We use `-F` instead of `-f` to ensure that files will still be tailed after log rotation.
25
2. `log2joural` is a Netdata program. It reads log entries and extracts fields, according to the PCRE2 pattern it accepts. It can also apply some basic operations on the fields, like injecting new fields or duplicating existing ones or rewriting their values. The output of `log2journal` is in Systemd Journal Export Format, and it looks like this:
@@ -28,9 +33,38 @@ Let's see the steps:
33
3. `systemd-cat-native` is a Netdata program. I can send the logs to a local `systemd-journald` (journal namespaces supported), or to a remote `systemd-journal-remote`.
34
35
31
-## YAML configuration
36
+## Processing pipeline
37
+
38
+The sequence of processing in Netdata's `log2journal` is designed to methodically transform and prepare log data for export in the systemd Journal Export Format. This transformation occurs through a pipeline of stages, each with a specific role in processing the log entries. Here's a description of each stage in the sequence:
39
+
40
+1. **Input**<br/>
41
+ The tool reads one log line at a time from the input source. It supports different input formats such as JSON, logfmt, and free-form logs defined by PCRE2 patterns.
42
+
43
+2. **Extract Fields and Values**<br/>
44
+ Based on the input format (JSON, logfmt, or custom pattern), it extracts fields and their values from each log line. In the case of JSON and logfmt, it automatically extracts all fields. For custom patterns, it uses PCRE2 regular expressions, and fields are extracted based on sub-expressions defined in the pattern.
45
+
46
+3. **Transliteration**<br/>
47
+ Extracted fields are transliterated to the limited character set accepted by systemd-journal: capitals A-Z, digits 0-9, underscores.
48
+
49
+4. **Apply Optional Prefix**<br/>
50
+ If a prefix is specified, it is added to all keys. This happens before any other processing so that all subsequent matches and manipulations take the prefix into account.
51
+
52
+5. **Rename Fields**<br/>
53
+ Renames fields as specified in the configuration. This is used to change the names of the fields to match desired or required naming conventions.
54
55
+6. **Inject New Fields**<br/>
56
+ New fields are injected into the log data. This can include constants or values derived from other fields, using variable substitution.
57
58
+7. **Rewrite Field Values**<br/>
59
+ Applies rewriting rules to alter the values of the fields. This can involve complex transformations, including regular expressions and variable substitutions. The rewrite rules can also inject new fields into the data.
60
+
61
+8. **Filter Fields**<br/>
62
+ Fields are filtered based on include and exclude patterns. This stage selects which fields are to be sent to the journal, allowing for selective logging.
63
+
64
+9. **Output**<br/>
65
+ Finally, the processed log data is output in the Journal Export Format. This format is compatible with systemd's journaling system and can be sent to local or remote systemd journal systems, by piping the output of `log2journal` to `systemd-cat-native`.
66
+
67
+This pipeline ensures a flexible and comprehensive approach to log processing, allowing for a wide range of modifications and customizations to fit various logging requirements. Each stage builds upon the previous one, enabling complex log transformations and enrichments before the data is exported to the systemd journal.
68
69
## Real-life example
70
@@ -318,7 +352,7 @@ tail -n $last -F /var/log/nginx/*access.log \
352
353
```
354
321
-Netdata log2journal v1.43.0-313-gd79fbac6a
355
+Netdata log2journal v1.43.0-341-gdac4df856
356
357
Convert logs to systemd Journal Export Format.
358
@@ -333,15 +367,53 @@ Options:
367
--file /path/to/file.yaml or -f /path/to/file.yaml
368
Read yaml configuration file for instructions.
369
336
- --config CONFIG_NAME
337
- Run with the internal configuration named CONFIG_NAME.
338
- Available internal configs:
370
+ --config CONFIG_NAME or -c CONFIG_NAME
371
+ Run with the internal YAML configuration named CONFIG_NAME.
372
+ Available internal YAML configs:
373
340
- nginx-combined nginx-json
374
+ nginx-combined nginx-json default
375
342
- --show-config
343
- Show the configuration in YAML format before starting the job.
344
- This is also an easy way to convert command line parameters to yaml.
376
+--------------------------------------------------------------------------------
377
+ INPUT PROCESSING
378
+
379
+ PATTERN
380
+ PATTERN should be a valid PCRE2 regular expression.
381
+ RE2 regular expressions (like the ones usually used in Go applications),
382
+ are usually valid PCRE2 patterns too.
383
+ Sub-expressions without named groups are evaluated, but their matches are
384
+ not added to the output.
385
+
386
+ - JSON mode
387
+ JSON mode is enabled when the pattern is set to: json
388
+ Field names are extracted from the JSON logs and are converted to the
389
+ format expected by Journal Export Format (all caps, only _ is allowed).
390
+
391
+ - logfmt mode
392
+ logfmt mode is enabled when the pattern is set to: logfmt
393
+ Field names are extracted from the logfmt logs and are converted to the
394
+ format expected by Journal Export Format (all caps, only _ is allowed).
395
+
396
+ All keys extracted from the input, are transliterated to match Journal
397
+ semantics (capital A-Z, digits 0-9, underscore).
398
+
399
+ In a YAML file:
400
+ ```yaml
401
+ pattern: 'PCRE2 pattern | json | logfmt'
402
+ ```
403
+
404
+--------------------------------------------------------------------------------
405
+ GLOBALS
406
+
407
+ --prefix PREFIX
408
+ Prefix all fields with PREFIX. The PREFIX is added before any other
409
+ processing, so that the extracted keys have to be matched with the PREFIX in
410
+ them. PREFIX is NOT transliterated and it is assumed to be systemd-journal
411
+ friendly.
412
+
413
+ In a YAML file:
414
+ ```yaml
415
+ prefix: 'PREFIX_' # prepend all keys with this prefix.
416
+ ```
417
418
--filename-key KEY
419
Add a field with KEY as the key and the current filename as value.
@@ -349,51 +421,151 @@ Options:
421
and tail matches multiple filenames.
422
To inject the filename when tailing a single file, use --inject.
423
352
- --unmatched-key KEY
353
- Include unmatched log entries in the output with KEY as the field name.
354
- Use this to include unmatched entries to the output stream.
355
- Usually it should be set to --unmatched-key=MESSAGE so that the
356
- unmatched entry will appear as the log message in the journals.
357
- Use --inject-unmatched to inject additional fields to unmatched lines.
424
+ In a YAML file:
425
+ ```yaml
426
+ filename:
427
+ key: KEY
428
+ ```
429
+
430
+--------------------------------------------------------------------------------
431
+ RENAMING OF KEYS
432
+
433
+ --rename NEW=OLD
434
+ Rename fields. OLD has been transliterated and PREFIX has been added.
435
+ NEW is assumed to be systemd journal friendly.
436
+
437
+ Up to 512 renaming rules are allowed.
438
+
439
+ In a YAML file:
440
+ ```yaml
441
+ rename:
442
+ - new_key: KEY1
443
+ old_key: KEY2 # transliterated with PREFIX added
444
+ - new_key: KEY3
445
+ old_key: KEY4 # transliterated with PREFIX added
446
+ # add as many as required
447
+ ```
448
359
- --inject LINE
449
+--------------------------------------------------------------------------------
450
+ INJECTING NEW KEYS
451
+
452
+ --inject KEY=VALUE
453
Inject constant fields to the output (both matched and unmatched logs).
454
--inject entries are added to unmatched lines too, when their key is
455
not used in --inject-unmatched (--inject-unmatched override --inject).
456
+ VALUE can use variable like ${OTHER_KEY} to be replaced with the values
457
+ of other keys available.
458
459
Up to 512 fields can be injected.
460
366
- --inject-unmatched LINE
367
- Inject lines into the output for each unmatched log entry.
368
- Usually, --inject-unmatched=PRIORITY=3 is needed to mark the unmatched
369
- lines as errors, so that they can easily be spotted in the journals.
461
+ In a YAML file:
462
+ ```yaml
463
+ inject:
464
+ - key: KEY1
465
+ value: 'VALUE1'
466
+ - key: KEY2
467
+ value: '${KEY3}${KEY4}' # gets the values of KEY3 and KEY4
468
+ # add as many as required
469
+ ```
470
371
- Up to 512 such lines can be injected.
471
+--------------------------------------------------------------------------------
472
+ REWRITING KEY VALUES
473
373
- --rewrite KEY=/SearchPattern/ReplacePattern
474
+ --rewrite KEY=/MATCH/REPLACE[/OPTIONS]
475
Apply a rewrite rule to the values of a specific key.
476
The first character after KEY= is the separator, which should also
376
- be used between the search pattern and the replacement pattern.
377
- The search pattern is a PCRE2 regular expression, and the replacement
378
- pattern supports literals and named capture groups from the search pattern.
477
+ be used between the MATCH, REPLACE and OPTIONS.
478
+
479
+ OPTIONS can be a comma separated list of `non-empty`, `dont-stop` and
480
+ `inject`.
481
+
482
+ When `non-empty` is given, MATCH is expected to be a variable
483
+ substitution using `${KEY1}${KEY2}`. Once the substitution is completed
484
+ the rule is matching the KEY only if the result is not empty.
485
+ When `non-empty` is not set, the MATCH string is expected to be a PCRE2
486
+ regular expression to be checked against the KEY value. This PCRE2
487
+ pattern may include named groups to extract parts of the KEY's value.
488
+
489
+ REPLACE supports variable substitution like `${variable}` against MATCH
490
+ named groups (when MATCH is a PCRE2 pattern) and `${KEY}` against the
491
+ keys defined so far.
492
493
Example:
494
--rewrite DATE=/^(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})$/
495
${day}/${month}/${year}
383
- This will rewrite dates in the format YYYY-MM-DD to DD/MM/YYYY.
496
+ The above will rewrite dates in the format YYYY-MM-DD to DD/MM/YYYY.
497
385
- Only one rewrite rule is applied per key; the sequence of rewrites stops
386
- for the key once a rule matches it. This allows providing a sequence of
387
- independent rewriting rules for the same key, matching the different
498
+ Only one rewrite rule is applied per key; the sequence of rewrites for a
499
+ given key, stops once a rule matches it. This allows providing a sequence
500
+ of independent rewriting rules for the same key, matching the different
501
values the key may get, and also provide a catch-all rewrite rule at the
389
- end, for setting the key value if no other rule matched it.
390
-
391
- Duplication of keys with the values of multiple other keys, combined with
392
- multiple value rewriting rules, allows creating complex rules for adding
393
- new keys, based on the values of existing keys.
502
+ end, for setting the key value if no other rule matched it. The rewrite
503
+ rule can allow processing more rewrite rules when OPTIONS includes
504
+ the keyword 'dont-stop'.
505
506
Up to 512 rewriting rules are allowed.
507
508
+ In a YAML file:
509
+ ```yaml
510
+ rewrite:
511
+ # the order if these rules in important - processed top to bottom
512
+ - key: KEY1
513
+ match: 'PCRE2 PATTERN WITH NAMED GROUPS'
514
+ value: 'all match fields and input keys as ${VARIABLE}'
515
+ inject: BOOLEAN # yes = inject the field, don't just rewrite it
516
+ stop: BOOLEAN # no = continue processing, don't stop if matched
517
+ - key: KEY2
518
+ non_empty: '${KEY3}${KEY4}' # match only if this evaluates to non empty
519
+ value: 'all input keys as ${VARIABLE}'
520
+ inject: BOOLEAN # yes = inject the field, don't just rewrite it
521
+ stop: BOOLEAN # no = continue processing, don't stop if matched
522
+ # add as many rewrites as required
523
+ ```
524
+
525
+ By default rewrite rules are applied only on fields already defined.
526
+ This allows shipping YAML files that include more rewrites than are
527
+ required for a specific input file.
528
+ Rewrite rules however allow injecting new fields when OPTIONS include
529
+ the keyword `inject` or in YAML `inject: yes` is given.
530
+
531
+ MATCH on the command line can be empty to define an unconditional rule.
532
+ Similarly, `match` and `non_empty` can be omitted in the YAML file.
533
+--------------------------------------------------------------------------------
534
+ UNMATCHED LINES
535
+
536
+ --unmatched-key KEY
537
+ Include unmatched log entries in the output with KEY as the field name.
538
+ Use this to include unmatched entries to the output stream.
539
+ Usually it should be set to --unmatched-key=MESSAGE so that the
540
+ unmatched entry will appear as the log message in the journals.
541
+ Use --inject-unmatched to inject additional fields to unmatched lines.
542
+
543
+ In a YAML file:
544
+ ```yaml
545
+ unmatched:
546
+ key: MESSAGE # inject the error log as MESSAGE
547
+ ```
548
+
549
+ --inject-unmatched LINE
550
+ Inject lines into the output for each unmatched log entry.
551
+ Usually, --inject-unmatched=PRIORITY=3 is needed to mark the unmatched
552
+ lines as errors, so that they can easily be spotted in the journals.
553
+
554
+ Up to 512 such lines can be injected.
555
+
556
+ In a YAML file:
557
+ ```yaml
558
+ unmatched:
559
+ key: MESSAGE # inject the error log as MESSAGE
560
+ inject::
561
+ - key: KEY1
562
+ value: 'VALUE1'
563
+ # add as many constants as required
564
+ ```
565
+
566
+--------------------------------------------------------------------------------
567
+ FILTERING
568
+
569
--include PATTERN
570
Include only keys matching the PCRE2 PATTERN.
571
Useful when parsing JSON of logfmt logs, to include only the keys given.
@@ -408,37 +580,22 @@ Options:
580
exclude wins and the key will not be added, like a pipeline, we first
581
include it and then exclude it.
582
411
- --prefix PREFIX
412
- Prefix all fields with PREFIX. The PREFIX is added before processing
413
- duplications, renames and rewrites, so that the extracted keys have to
414
- be matched with the PREFIX in them.
415
- PREFIX is assumed to be systemd-journal friendly.
416
-
417
- --rename NEW=OLD
418
- Rename fields, before rewriting their values.
583
+ In a YAML file:
584
+ ```yaml
585
+ filter:
586
+ include: 'PCRE2 PATTERN MATCHING KEY NAMES TO INCLUDE'
587
+ exclude: 'PCRE2 PATTERN MATCHING KEY NAMES TO EXCLUDE'
588
+ ```
589
420
- Up to 512 renaming rules are allowed.
590
+--------------------------------------------------------------------------------
591
+ OTHER
592
593
-h, or --help
594
Display this help and exit.
595
425
- PATTERN
426
- PATTERN should be a valid PCRE2 regular expression.
427
- RE2 regular expressions (like the ones usually used in Go applications),
428
- are usually valid PCRE2 patterns too.
429
- Sub-expressions without named groups are evaluated, but their matches are
430
- not added to the output.
431
-
432
- JSON mode
433
- JSON mode is enabled when the pattern is set to: json
434
- Field names are extracted from the JSON logs and are converted to the
435
- format expected by Journal Export Format (all caps, only _ is allowed).
436
-
437
- logfmt mode
438
- logfmt mode is enabled when the pattern is set to: logfmt
439
- Field names are extracted from the logfmt logs and are converted to the
440
- format expected by Journal Export Format (all caps, only _ is allowed).
441
-
596
+ --show-config
597
+ Show the configuration in YAML format before starting the job.
598
+ This is also an easy way to convert command line parameters to yaml.
599
600
The program accepts all parameters as both --option=value and --option value.
601
@@ -456,7 +613,7 @@ This is a simple diagram of the pipeline taking place:
613
+---------------------------------------------------+
614
| EXTRACT FIELDS AND VALUES |
615
| JSON, logfmt, or pattern based |
459
- | (apply optional PREFIX) |
616
+ | (apply optional PREFIX - all keys use capitals) |
617
+---------------------------------------------------+
618
v v v v v v
619
+---------------------------------------------------+
@@ -524,110 +681,4 @@ You can find the most common fields at 'man systemd.journal-fields'.
681
682
## `systemd-cat-native` options
683
527
-```
528
-
529
-Netdata systemd-cat-native v1.40.0-1214-gae733dd49
530
-
531
-This program reads from its standard input, lines in the format:
532
-
533
-KEY1=VALUE1\n
534
-KEY2=VALUE2\n
535
-KEYN=VALUEN\n
536
-\n
537
-
538
-and sends them to systemd-journal.
539
-
540
- - Binary journal fields are not accepted at its input
541
- - Binary journal fields can be generated after newline processing
542
- - Messages have to be separated by an empty line
543
- - Keys starting with underscore are not accepted (by journald)
544
- - Other rules imposed by systemd-journald are imposed (by journald)
545
-
546
-Usage:
547
-
548
- systemd-cat-native
549
- [--newline=STRING]
550
- [--log-as-netdata|-N]
551
- [--namespace=NAMESPACE] [--socket=PATH]
552
- [--url=URL [--key=FILENAME] [--cert=FILENAME] [--trust=FILENAME|all]]
553
-
554
-The program has the following modes of logging:
555
-
556
- * Log to a local systemd-journald or stderr
557
-
558
- This is the default mode. If systemd-journald is available, logs will be
559
- sent to systemd, otherwise logs will be printed on stderr, using logfmt
560
- formatting. Options --socket and --namespace are available to configure
561
- the journal destination:
562
-
563
- --socket=PATH
564
- The path of a systemd-journald UNIX socket.
565
- The program will use the default systemd-journald socket when this
566
- option is not used.
567
-
568
- --namespace=NAMESPACE
569
- The name of a configured and running systemd-journald namespace.
570
- The program will produce the socket path based on its internal
571
- defaults, to send the messages to the systemd journal namespace.
572
-
573
- * Log as Netdata, enabled with --log-as-netdata or -N
574
-
575
- In this mode the program uses environment variables set by Netdata for
576
- the log destination. Only log fields defined by Netdata are accepted.
577
- If the environment variables expected by Netdata are not found, it
578
- falls back to stderr logging in logfmt format.
579
-
580
- * Log to a systemd-journal-remote TCP socket, enabled with --url=URL
581
-
582
- In this mode, the program will directly sent logs to a remote systemd
583
- journal (systemd-journal-remote expected at the destination)
584
- This mode is available even when the local system does not support
585
- systemd, or even it is not Linux, allowing a remote Linux systemd
586
- journald to become the logs database of the local system.
587
-
588
- Unfortunately systemd-journal-remote does not accept compressed
589
- data over the network, so the stream will be uncompressed.
590
-
591
- --url=URL
592
- The destination systemd-journal-remote address and port, similarly
593
- to what /etc/systemd/journal-upload.conf accepts.
594
- Usually it is in the form: https://ip.address:19532
595
- Both http and https URLs are accepted. When using https, the
596
- following additional options are accepted:
597
-
598
- --key=FILENAME
599
- The filename of the private key of the server.
600
- The default is: /etc/ssl/private/journal-upload.pem
601
-
602
- --cert=FILENAME
603
- The filename of the public key of the server.
604
- The default is: /etc/ssl/certs/journal-upload.pem
605
-
606
- --trust=FILENAME | all
607
- The filename of the trusted CA public key.
608
- The default is: /etc/ssl/ca/trusted.pem
609
- The keyword 'all' can be used to trust all CAs.
610
-
611
- --keep-trying
612
- Keep trying to send the message, if the remote journal is not there.
613
-
614
- NEWLINES PROCESSING
615
- systemd-journal logs entries may have newlines in them. However the
616
- Journal Export Format uses binary formatted data to achieve this,
617
- making it hard for text processing.
618
-
619
- To overcome this limitation, this program allows single-line text
620
- formatted values at its input, to be binary formatted multi-line Journal
621
- Export Format at its output.
622
-
623
- To achieve that it allows replacing a given string to a newline.
624
- The parameter --newline=STRING allows setting the string to be replaced
625
- with newlines.
626
-
627
- For example by setting --newline='{NEWLINE}', the program will replace
628
- all occurrences of {NEWLINE} with the newline character, within each
629
- VALUE of the KEY=VALUE lines. Once this this done, the program will
630
- switch the field to the binary Journal Export Format before sending the
631
- log event to systemd-journal.
632
-
633
-```
684
+Read [the manual of systemd-cat-native](../../libnetdata/log/systemd-cat-native.md).
collectors/log2journal/log2journal-help.c
+181
-57
@@ -61,8 +61,8 @@ void log_job_command_line_help(const char *name) {
61
printf(" Read yaml configuration file for instructions.\n");
62
printf("\n");
63
printf(" --config CONFIG_NAME or -c CONFIG_NAME\n");
64
- printf(" Run with the internal configuration named CONFIG_NAME.\n");
65
- printf(" Available internal configs:\n");
64
+ printf(" Run with the internal YAML configuration named CONFIG_NAME.\n");
65
+ printf(" Available internal YAML configs:\n");
66
printf("\n");
67
config_dir_print_available();
68
printf("\n");
@@ -71,9 +71,47 @@ void log_job_command_line_help(const char *name) {
71
printf(" YAML configuration parsing is not compiled in this binary.\n");
72
printf("\n");
73
#endif
74
- printf(" --show-config\n");
75
- printf(" Show the configuration in YAML format before starting the job.\n");
76
- printf(" This is also an easy way to convert command line parameters to yaml.\n");
74
+ printf("--------------------------------------------------------------------------------\n");
75
+ printf(" INPUT PROCESSING\n");
76
+ printf("\n");
77
+ printf(" PATTERN\n");
78
+ printf(" PATTERN should be a valid PCRE2 regular expression.\n");
79
+ printf(" RE2 regular expressions (like the ones usually used in Go applications),\n");
80
+ printf(" are usually valid PCRE2 patterns too.\n");
81
+ printf(" Sub-expressions without named groups are evaluated, but their matches are\n");
82
+ printf(" not added to the output.\n");
83
+ printf("\n");
84
+ printf(" - JSON mode\n");
85
+ printf(" JSON mode is enabled when the pattern is set to: json\n");
86
+ printf(" Field names are extracted from the JSON logs and are converted to the\n");
87
+ printf(" format expected by Journal Export Format (all caps, only _ is allowed).\n");
88
+ printf("\n");
89
+ printf(" - logfmt mode\n");
90
+ printf(" logfmt mode is enabled when the pattern is set to: logfmt\n");
91
+ printf(" Field names are extracted from the logfmt logs and are converted to the\n");
92
+ printf(" format expected by Journal Export Format (all caps, only _ is allowed).\n");
93
+ printf("\n");
94
+ printf(" All keys extracted from the input, are transliterated to match Journal\n");
95
+ printf(" semantics (capital A-Z, digits 0-9, underscore).\n");
96
+ printf("\n");
97
+ printf(" In a YAML file:\n");
98
+ printf(" ```yaml\n");
99
+ printf(" pattern: 'PCRE2 pattern | json | logfmt'\n");
100
+ printf(" ```\n");
101
+ printf("\n");
102
+ printf("--------------------------------------------------------------------------------\n");
103
+ printf(" GLOBALS\n");
104
+ printf("\n");
105
+ printf(" --prefix PREFIX\n");
106
+ printf(" Prefix all fields with PREFIX. The PREFIX is added before any other\n");
107
+ printf(" processing, so that the extracted keys have to be matched with the PREFIX in\n");
108
+ printf(" them. PREFIX is NOT transliterated and it is assumed to be systemd-journal\n");
109
+ printf(" friendly.\n");
110
+ printf("\n");
111
+ printf(" In a YAML file:\n");
112
+ printf(" ```yaml\n");
113
+ printf(" prefix: 'PREFIX_' # prepend all keys with this prefix.\n");
114
+ printf(" ```\n");
115
printf("\n");
116
printf(" --filename-key KEY\n");
117
printf(" Add a field with KEY as the key and the current filename as value.\n");
@@ -81,51 +119,152 @@ void log_job_command_line_help(const char *name) {
119
printf(" and tail matches multiple filenames.\n");
120
printf(" To inject the filename when tailing a single file, use --inject.\n");
121
printf("\n");
84
- printf(" --unmatched-key KEY\n");
85
- printf(" Include unmatched log entries in the output with KEY as the field name.\n");
86
- printf(" Use this to include unmatched entries to the output stream.\n");
87
- printf(" Usually it should be set to --unmatched-key=MESSAGE so that the\n");
88
- printf(" unmatched entry will appear as the log message in the journals.\n");
89
- printf(" Use --inject-unmatched to inject additional fields to unmatched lines.\n");
122
+ printf(" In a YAML file:\n");
123
+ printf(" ```yaml\n");
124
+ printf(" filename:\n");
125
+ printf(" key: KEY\n");
126
+ printf(" ```\n");
127
+ printf("\n");
128
+ printf("--------------------------------------------------------------------------------\n");
129
+ printf(" RENAMING OF KEYS\n");
130
+ printf("\n");
131
+ printf(" --rename NEW=OLD\n");
132
+ printf(" Rename fields. OLD has been transliterated and PREFIX has been added.\n");
133
+ printf(" NEW is assumed to be systemd journal friendly.\n");
134
printf("\n");
91
- printf(" --inject LINE\n");
135
+ printf(" Up to %d renaming rules are allowed.\n", MAX_RENAMES);
136
+ printf("\n");
137
+ printf(" In a YAML file:\n");
138
+ printf(" ```yaml\n");
139
+ printf(" rename:\n");
140
+ printf(" - new_key: KEY1\n");
141
+ printf(" old_key: KEY2 # transliterated with PREFIX added\n");
142
+ printf(" - new_key: KEY3\n");
143
+ printf(" old_key: KEY4 # transliterated with PREFIX added\n");
144
+ printf(" # add as many as required\n");
145
+ printf(" ```\n");
146
+ printf("\n");
147
+ printf("--------------------------------------------------------------------------------\n");
148
+ printf(" INJECTING NEW KEYS\n");
149
+ printf("\n");
150
+ printf(" --inject KEY=VALUE\n");
151
printf(" Inject constant fields to the output (both matched and unmatched logs).\n");
152
printf(" --inject entries are added to unmatched lines too, when their key is\n");
153
printf(" not used in --inject-unmatched (--inject-unmatched override --inject).\n");
154
+ printf(" VALUE can use variable like ${OTHER_KEY} to be replaced with the values\n");
155
+ printf(" of other keys available.\n");
156
printf("\n");
157
printf(" Up to %d fields can be injected.\n", MAX_INJECTIONS);
158
printf("\n");
98
- printf(" --inject-unmatched LINE\n");
99
- printf(" Inject lines into the output for each unmatched log entry.\n");
100
- printf(" Usually, --inject-unmatched=PRIORITY=3 is needed to mark the unmatched\n");
101
- printf(" lines as errors, so that they can easily be spotted in the journals.\n");
159
+ printf(" In a YAML file:\n");
160
+ printf(" ```yaml\n");
161
+ printf(" inject:\n");
162
+ printf(" - key: KEY1\n");
163
+ printf(" value: 'VALUE1'\n");
164
+ printf(" - key: KEY2\n");
165
+ printf(" value: '${KEY3}${KEY4}' # gets the values of KEY3 and KEY4\n");
166
+ printf(" # add as many as required\n");
167
+ printf(" ```\n");
168
printf("\n");
103
- printf(" Up to %d such lines can be injected.\n", MAX_INJECTIONS);
169
+ printf("--------------------------------------------------------------------------------\n");
170
+ printf(" REWRITING KEY VALUES\n");
171
printf("\n");
105
- printf(" --rewrite KEY=/SearchPattern/ReplacePattern\n");
172
+ printf(" --rewrite KEY=/MATCH/REPLACE[/OPTIONS]\n");
173
printf(" Apply a rewrite rule to the values of a specific key.\n");
174
printf(" The first character after KEY= is the separator, which should also\n");
108
- printf(" be used between the search pattern and the replacement pattern.\n");
109
- printf(" The search pattern is a PCRE2 regular expression, and the replacement\n");
110
- printf(" pattern supports literals and named capture groups from the search pattern.\n");
175
+ printf(" be used between the MATCH, REPLACE and OPTIONS.\n");
176
+ printf("\n");
177
+ printf(" OPTIONS can be a comma separated list of `non-empty`, `dont-stop` and\n");
178
+ printf(" `inject`.\n");
179
+ printf("\n");
180
+ printf(" When `non-empty` is given, MATCH is expected to be a variable\n");
181
+ printf(" substitution using `${KEY1}${KEY2}`. Once the substitution is completed\n");
182
+ printf(" the rule is matching the KEY only if the result is not empty.\n");
183
+ printf(" When `non-empty` is not set, the MATCH string is expected to be a PCRE2\n");
184
+ printf(" regular expression to be checked against the KEY value. This PCRE2\n");
185
+ printf(" pattern may include named groups to extract parts of the KEY's value.\n");
186
+ printf("\n");
187
+ printf(" REPLACE supports variable substitution like `${variable}` against MATCH\n");
188
+ printf(" named groups (when MATCH is a PCRE2 pattern) and `${KEY}` against the\n");
189
+ printf(" keys defined so far.\n");
190
printf("\n");
191
printf(" Example:\n");
192
printf(" --rewrite DATE=/^(?<year>\\d{4})-(?<month>\\d{2})-(?<day>\\d{2})$/\n");
193
printf(" ${day}/${month}/${year}\n");
115
- printf(" This will rewrite dates in the format YYYY-MM-DD to DD/MM/YYYY.\n");
194
+ printf(" The above will rewrite dates in the format YYYY-MM-DD to DD/MM/YYYY.\n");
195
printf("\n");
117
- printf(" Only one rewrite rule is applied per key; the sequence of rewrites stops\n");
118
- printf(" for the key once a rule matches it. This allows providing a sequence of\n");
119
- printf(" independent rewriting rules for the same key, matching the different\n");
196
+ printf(" Only one rewrite rule is applied per key; the sequence of rewrites for a\n");
197
+ printf(" given key, stops once a rule matches it. This allows providing a sequence\n");
198
+ printf(" of independent rewriting rules for the same key, matching the different\n");
199
printf(" values the key may get, and also provide a catch-all rewrite rule at the\n");
121
- printf(" end, for setting the key value if no other rule matched it.\n");
122
- printf("\n");
123
- printf(" Duplication of keys with the values of multiple other keys, combined with\n");
124
- printf(" multiple value rewriting rules, allows creating complex rules for adding\n");
125
- printf(" new keys, based on the values of existing keys.\n");
200
+ printf(" end, for setting the key value if no other rule matched it. The rewrite\n");
201
+ printf(" rule can allow processing more rewrite rules when OPTIONS includes\n");
202
+ printf(" the keyword 'dont-stop'.\n");
203
printf("\n");
204
printf(" Up to %d rewriting rules are allowed.\n", MAX_REWRITES);
205
printf("\n");
206
+ printf(" In a YAML file:\n");
207
+ printf(" ```yaml\n");
208
+ printf(" rewrite:\n");
209
+ printf(" # the order if these rules in important - processed top to bottom\n");
210
+ printf(" - key: KEY1\n");
211
+ printf(" match: 'PCRE2 PATTERN WITH NAMED GROUPS'\n");
212
+ printf(" value: 'all match fields and input keys as ${VARIABLE}'\n");
213
+ printf(" inject: BOOLEAN # yes = inject the field, don't just rewrite it\n");
214
+ printf(" stop: BOOLEAN # no = continue processing, don't stop if matched\n");
215
+ printf(" - key: KEY2\n");
216
+ printf(" non_empty: '${KEY3}${KEY4}' # match only if this evaluates to non empty\n");
217
+ printf(" value: 'all input keys as ${VARIABLE}'\n");
218
+ printf(" inject: BOOLEAN # yes = inject the field, don't just rewrite it\n");
219
+ printf(" stop: BOOLEAN # no = continue processing, don't stop if matched\n");
220
+ printf(" # add as many rewrites as required\n");
221
+ printf(" ```\n");
222
+ printf("\n");
223
+ printf(" By default rewrite rules are applied only on fields already defined.\n");
224
+ printf(" This allows shipping YAML files that include more rewrites than are\n");
225
+ printf(" required for a specific input file.\n");
226
+ printf(" Rewrite rules however allow injecting new fields when OPTIONS include\n");
227
+ printf(" the keyword `inject` or in YAML `inject: yes` is given.\n");
228
+ printf("\n");
229
+ printf(" MATCH on the command line can be empty to define an unconditional rule.\n");
230
+ printf(" Similarly, `match` and `non_empty` can be omitted in the YAML file.");
231
+ printf("\n");
232
+ printf("--------------------------------------------------------------------------------\n");
233
+ printf(" UNMATCHED LINES\n");
234
+ printf("\n");
235
+ printf(" --unmatched-key KEY\n");
236
+ printf(" Include unmatched log entries in the output with KEY as the field name.\n");
237
+ printf(" Use this to include unmatched entries to the output stream.\n");
238
+ printf(" Usually it should be set to --unmatched-key=MESSAGE so that the\n");
239
+ printf(" unmatched entry will appear as the log message in the journals.\n");
240
+ printf(" Use --inject-unmatched to inject additional fields to unmatched lines.\n");
241
+ printf("\n");
242
+ printf(" In a YAML file:\n");
243
+ printf(" ```yaml\n");
244
+ printf(" unmatched:\n");
245
+ printf(" key: MESSAGE # inject the error log as MESSAGE\n");
246
+ printf(" ```\n");
247
+ printf("\n");
248
+ printf(" --inject-unmatched LINE\n");
249
+ printf(" Inject lines into the output for each unmatched log entry.\n");
250
+ printf(" Usually, --inject-unmatched=PRIORITY=3 is needed to mark the unmatched\n");
251
+ printf(" lines as errors, so that they can easily be spotted in the journals.\n");
252
+ printf("\n");
253
+ printf(" Up to %d such lines can be injected.\n", MAX_INJECTIONS);
254
+ printf("\n");
255
+ printf(" In a YAML file:\n");
256
+ printf(" ```yaml\n");
257
+ printf(" unmatched:\n");
258
+ printf(" key: MESSAGE # inject the error log as MESSAGE\n");
259
+ printf(" inject::\n");
260
+ printf(" - key: KEY1\n");
261
+ printf(" value: 'VALUE1'\n");
262
+ printf(" # add as many constants as required\n");
263
+ printf(" ```\n");
264
+ printf("\n");
265
+ printf("--------------------------------------------------------------------------------\n");
266
+ printf(" FILTERING\n");
267
+ printf("\n");
268
printf(" --include PATTERN\n");
269
printf(" Include only keys matching the PCRE2 PATTERN.\n");
270
printf(" Useful when parsing JSON of logfmt logs, to include only the keys given.\n");
@@ -140,37 +279,22 @@ void log_job_command_line_help(const char *name) {
279
printf(" exclude wins and the key will not be added, like a pipeline, we first\n");
280
printf(" include it and then exclude it.\n");
281
printf("\n");
143
- printf(" --prefix PREFIX\n");
144
- printf(" Prefix all fields with PREFIX. The PREFIX is added before processing\n");
145
- printf(" duplications, renames and rewrites, so that the extracted keys have to\n");
146
- printf(" be matched with the PREFIX in them.\n");
147
- printf(" PREFIX is assumed to be systemd-journal friendly.\n");
148
- printf("\n");
149
- printf(" --rename NEW=OLD\n");
150
- printf(" Rename fields, before rewriting their values.\n");
282
+ printf(" In a YAML file:\n");
283
+ printf(" ```yaml\n");
284
+ printf(" filter:\n");
285
+ printf(" include: 'PCRE2 PATTERN MATCHING KEY NAMES TO INCLUDE'\n");
286
+ printf(" exclude: 'PCRE2 PATTERN MATCHING KEY NAMES TO EXCLUDE'\n");
287
+ printf(" ```\n");
288
printf("\n");
152
- printf(" Up to %d renaming rules are allowed.\n", MAX_RENAMES);
289
+ printf("--------------------------------------------------------------------------------\n");
290
+ printf(" OTHER\n");
291
printf("\n");
292
printf(" -h, or --help\n");
293
printf(" Display this help and exit.\n");
294
printf("\n");
157
- printf(" PATTERN\n");
158
- printf(" PATTERN should be a valid PCRE2 regular expression.\n");
159
- printf(" RE2 regular expressions (like the ones usually used in Go applications),\n");
160
- printf(" are usually valid PCRE2 patterns too.\n");
161
- printf(" Sub-expressions without named groups are evaluated, but their matches are\n");
162
- printf(" not added to the output.\n");
163
- printf("\n");
164
- printf(" JSON mode\n");
165
- printf(" JSON mode is enabled when the pattern is set to: json\n");
166
- printf(" Field names are extracted from the JSON logs and are converted to the\n");
167
- printf(" format expected by Journal Export Format (all caps, only _ is allowed).\n");
168
- printf("\n");
169
- printf(" logfmt mode\n");
170
- printf(" logfmt mode is enabled when the pattern is set to: logfmt\n");
171
- printf(" Field names are extracted from the logfmt logs and are converted to the\n");
172
- printf(" format expected by Journal Export Format (all caps, only _ is allowed).\n");
173
- printf("\n");
295
+ printf(" --show-config\n");
296
+ printf(" Show the configuration in YAML format before starting the job.\n");
297
+ printf(" This is also an easy way to convert command line parameters to yaml.\n");
298
printf("\n");
299
printf("The program accepts all parameters as both --option=value and --option value.\n");
300
printf("\n");
@@ -188,7 +312,7 @@ void log_job_command_line_help(const char *name) {
312
printf(" +---------------------------------------------------+ \n");
313
printf(" | EXTRACT FIELDS AND VALUES | \n");
314
printf(" | JSON, logfmt, or pattern based | \n");
191
- printf(" | (apply optional PREFIX) | \n");
315
+ printf(" | (apply optional PREFIX - all keys use capitals) | \n");
316
printf(" +---------------------------------------------------+ \n");
317
printf(" v v v v v v \n");
318
printf(" +---------------------------------------------------+ \n");
collectors/systemd-journal.plugin/README.md
+20
-12
@@ -304,9 +304,9 @@ and searching of any dataset, independently of its source.
304
On busy logs servers, or when querying long timeframes that match millions of log entries, the plugin has a sampling
305
algorithm to allow it respond promptly. It works like this:
306
307
-1. The latest 500k log entries are queried in full, meaning that the log entries are queried in full, evaluating all the
308
- fields of every single log entry. This evaluation allows counting the unique values per field, updating the counters
309
- next to each value at the filters section of the dashboard.
307
+1. The latest 500k log entries are queried in full, evaluating all the fields of every single log entry. This evaluation
308
+ allows counting the unique values per field, updating the counters next to each value at the filters section of the
309
+ dashboard.
310
2. When the latest 500k log entries have been processed and there are more data to read, the plugin divides evenly 500k
311
more log entries to the number of journal files matched by the query. So, it will continue to evaluate all the fields
312
of all log entries, up to the budget per file, aiming to fully query 1 million log entries in total.
@@ -348,20 +348,28 @@ To work around this problem, the plugin uses `inotify` to receive file updates i
348
the newest to the oldest file, allowing the user interface to work immediately after startup, for the most recent
349
timeframes.
350
351
-systemd-journal has been designed first to be reliable and then to be fast. It includes several mechanisms to ensure
351
+### Best practices for better performance
352
+
353
+systemd-journal has been designed **first to be reliable** and then to be fast. It includes several mechanisms to ensure
354
minimal data loss under all conditions (e.g. disk corruption, tampering, forward secure sealing) and despite the fact
355
that it utilizes several techniques to require minimal disk footprint (like deduplication of log entries, linking of
356
values and fields, compression) the disk footprint of journal files remains significantly higher compared to other log
355
-management solutions. The higher disk footprint results in higher disk I/O during querying, since a lot more data have
356
-to read from disk to evaluate a query. Query performance at scale can greatly be improved by utilizing a compressed
357
-filesystem (ext4, btrfs, zfs) to store systemd-journal files.
357
+management solutions.
358
+
359
+The higher disk footprint results in higher disk I/O during querying, since a lot more data have to read from disk to
360
+evaluate a query. Query performance at scale can greatly benefit by utilizing a compressed filesystem (ext4, btrfs, zfs)
361
+to store systemd-journal files.
362
363
systemd-journal files are cached by the operating system. There is no database server to serve queries. Each file is
360
-opened and the query runs by directly accessing the data in it. Therefore systemd-journal relies on the caching
361
-layer of the operating system to optimize query performance. The more RAM the system has, although it will not be
362
-reported as `used` (it will be reported as `cache`), the faster the queries will get. The first time a timeframe is
363
-accessed the query performance will be slower, but further queries on the same timeframe will be significantly faster
364
-since journal data are now cached in memory.
364
+opened and the query runs by directly accessing the data in it.
365
+
366
+Therefore systemd-journal relies on the caching layer of the operating system to optimize query performance. The more
367
+RAM the system has, although it will not be reported as `used` (it will be reported as `cache`), the faster the queries
368
+will get. The first time a timeframe is accessed the query performance will be slower, but further queries on the same
369
+timeframe will be significantly faster since journal data are now cached in memory.
370
+
371
+So, on busy logs centralization systems, queries performance can be improved significantly by using a compressed
372
+filesystem for storing the journal files, and higher amounts of RAM.
373
374
## Configuration and maintenance
375
libnetdata/log/Makefile.am
+1
@@ -5,4 +5,5 @@ MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5
6
dist_noinst_DATA = \
7
README.md \
8
+ systemd-cat-native.md \
9
$(NULL)
libnetdata/log/systemd-cat-native.md
new
+209
@@ -0,0 +1,209 @@
1
+# systemd-cat-native
2
+
3
+`systemd` includes a utility called `systemd-cat`. This utility reads log lines from its standard input and sends them
4
+to the local systemd journal. Its key limitation is that despite the fact that systemd journals support structured logs,
5
+this command does not support sending structured logs to it.
6
+
7
+`systemd-cat-native` is a Netdata supplied utility to push structured logs to systemd journals. Key features:
8
+
9
+- reads [Journal Export Format](https://systemd.io/JOURNAL_EXPORT_FORMATS/) formatted log entries
10
+- converts text fields into binary journal multiline log fields
11
+- sends logs to any of these:
12
+ - local default `systemd-journald`,
13
+ - local namespace `systemd-journald`,
14
+ - remote `systemd-journal-remote` using HTTP or HTTPS, the same way `systemd-journal-upload` does.
15
+- is the standard external logger of Netdata shell scripts
16
+
17
+## Simple use:
18
+
19
+```bash
20
+printf "MESSAGE=hello world\nPRIORITY=6\n\n" | systemd-cat-native
21
+```
22
+
23
+The result:
24
+
25
+
26
+
27
+
28
+Sending `PRIORITY=3` (error):
29
+
30
+```bash
31
+printf "MESSAGE=hey, this is error\nPRIORITY=3\n\n" | systemd-cat-native
32
+```
33
+
34
+The result:
35
+
36
+
37
+Sending multi-line log entries (in this example we replace the text `--NEWLINE--` with a newline in the log entry):
38
+
39
+```bash
40
+printf "MESSAGE=hello--NEWLINE--world\nPRIORITY=6\n\n" | systemd-cat-native --newline='--NEWLINE--'
41
+```
42
+
43
+The result:
44
+
45
+
46
+
47
+
48
+Processing the standard `\n` string can be tricky due to shell escaping. This works, but note that
49
+we have to add a lot of backslashes to printf.
50
+
51
+```bash
52
+printf "MESSAGE=hello\\\\nworld\nPRIORITY=6\n\n" | systemd-cat-native --newline='\n'
53
+```
54
+
55
+`systemd-cat-native` needs to receive it like this for newline processing to work:
56
+
57
+```bash
58
+# printf "MESSAGE=hello\\\\nworld\nPRIORITY=6\n\n"
59
+MESSAGE=hello\nworld
60
+PRIORITY=6
61
+
62
+```
63
+
64
+## Best practices
65
+
66
+These are the rules about fields, enforced by `systemd-journald`:
67
+
68
+- field names can be up to **64 characters**,
69
+- field values can be up to **48k characters**,
70
+- the only allowed field characters are **A-Z**, **0-9** and **underscore**,
71
+- the **first** character of fields cannot be a **digit**
72
+- **protected** journal fields start with underscore:
73
+ * they are accepted by `systemd-journal-remote`,
74
+ * they are **NOT** accepted by a local `systemd-journald`.
75
+
76
+For best results, always include these fields:
77
+
78
+- `MESSAGE=TEXT`<br/>
79
+ The `MESSAGE` is the body of the log entry.
80
+ This field is what we usually see in our logs.
81
+
82
+- `PRIORITY=NUMBER`<br/>
83
+ `PRIORITY` sets the severity of the log entry.<br/>
84
+ `0=emerg, 1=alert, 2=crit, 3=err, 4=warn, 5=notice, 6=info, 7=debug`
85
+ - Emergency events (0) are usually broadcast to all terminals.
86
+ - Emergency, alert, critical, and error (0-3) are usually colored red.
87
+ - Warning (4) entries are usually colored yellow.
88
+ - Notice (5) entries are usually bold or have a brighter white color.
89
+ - Info (6) entries are the default.
90
+ - Debug (7) entries are usually grayed or dimmed.
91
+
92
+- `SYSLOG_IDENTIFIER=NAME`<br/>
93
+ `SYSLOG_IDENTIFIER` sets the name of application.
94
+ Use something descriptive, like: `SYSLOG_IDENTIFIER=myapp`
95
+
96
+You can find the most common fields at `man systemd.journal-fields`.
97
+
98
+
99
+## Usage
100
+
101
+```
102
+Netdata systemd-cat-native v1.43.0-333-g5af71b875
103
+
104
+This program reads from its standard input, lines in the format:
105
+
106
+KEY1=VALUE1\n
107
+KEY2=VALUE2\n
108
+KEYN=VALUEN\n
109
+\n
110
+
111
+and sends them to systemd-journal.
112
+
113
+ - Binary journal fields are not accepted at its input
114
+ - Binary journal fields can be generated after newline processing
115
+ - Messages have to be separated by an empty line
116
+ - Keys starting with underscore are not accepted (by journald)
117
+ - Other rules imposed by systemd-journald are imposed (by journald)
118
+
119
+Usage:
120
+
121
+ systemd-cat-native
122
+ [--newline=STRING]
123
+ [--log-as-netdata|-N]
124
+ [--namespace=NAMESPACE] [--socket=PATH]
125
+ [--url=URL [--key=FILENAME] [--cert=FILENAME] [--trust=FILENAME|all]]
126
+
127
+The program has the following modes of logging:
128
+
129
+ * Log to a local systemd-journald or stderr
130
+
131
+ This is the default mode. If systemd-journald is available, logs will be
132
+ sent to systemd, otherwise logs will be printed on stderr, using logfmt
133
+ formatting. Options --socket and --namespace are available to configure
134
+ the journal destination:
135
+
136
+ --socket=PATH
137
+ The path of a systemd-journald UNIX socket.
138
+ The program will use the default systemd-journald socket when this
139
+ option is not used.
140
+
141
+ --namespace=NAMESPACE
142
+ The name of a configured and running systemd-journald namespace.
143
+ The program will produce the socket path based on its internal
144
+ defaults, to send the messages to the systemd journal namespace.
145
+
146
+ * Log as Netdata, enabled with --log-as-netdata or -N
147
+
148
+ In this mode the program uses environment variables set by Netdata for
149
+ the log destination. Only log fields defined by Netdata are accepted.
150
+ If the environment variables expected by Netdata are not found, it
151
+ falls back to stderr logging in logfmt format.
152
+
153
+ * Log to a systemd-journal-remote TCP socket, enabled with --url=URL
154
+
155
+ In this mode, the program will directly sent logs to a remote systemd
156
+ journal (systemd-journal-remote expected at the destination)
157
+ This mode is available even when the local system does not support
158
+ systemd, or even it is not Linux, allowing a remote Linux systemd
159
+ journald to become the logs database of the local system.
160
+
161
+ Unfortunately systemd-journal-remote does not accept compressed
162
+ data over the network, so the stream will be uncompressed.
163
+
164
+ --url=URL
165
+ The destination systemd-journal-remote address and port, similarly
166
+ to what /etc/systemd/journal-upload.conf accepts.
167
+ Usually it is in the form: https://ip.address:19532
168
+ Both http and https URLs are accepted. When using https, the
169
+ following additional options are accepted:
170
+
171
+ --key=FILENAME
172
+ The filename of the private key of the server.
173
+ The default is: /etc/ssl/private/journal-upload.pem
174
+
175
+ --cert=FILENAME
176
+ The filename of the public key of the server.
177
+ The default is: /etc/ssl/certs/journal-upload.pem
178
+
179
+ --trust=FILENAME | all
180
+ The filename of the trusted CA public key.
181
+ The default is: /etc/ssl/ca/trusted.pem
182
+ The keyword 'all' can be used to trust all CAs.
183
+
184
+ --namespace=NAMESPACE
185
+ Set the namespace of the messages sent.
186
+
187
+ --keep-trying
188
+ Keep trying to send the message, if the remote journal is not there.
189
+
190
+ NEWLINES PROCESSING
191
+ systemd-journal logs entries may have newlines in them. However the
192
+ Journal Export Format uses binary formatted data to achieve this,
193
+ making it hard for text processing.
194
+
195
+ To overcome this limitation, this program allows single-line text
196
+ formatted values at its input, to be binary formatted multi-line Journal
197
+ Export Format at its output.
198
+
199
+ To achieve that it allows replacing a given string to a newline.
200
+ The parameter --newline=STRING allows setting the string to be replaced
201
+ with newlines.
202
+
203
+ For example by setting --newline='--NEWLINE--', the program will replace
204
+ all occurrences of --NEWLINE-- with the newline character, within each
205
+ VALUE of the KEY=VALUE lines. Once this this done, the program will
206
+ switch the field to the binary Journal Export Format before sending the
207
+ log event to systemd-journal.
208
+
209
+```
\ No newline at end of file