Update README.md
Costa Tsaousis committed
Oct 15, 2023 at 20:57 UTC
a7dc81d661656ab3318bbdd29b79475e3ffaae0e
1 file changed
+271
-287
collectors/systemd-journal.plugin/README.md
+271
-287
@@ -1,3 +1,4 @@
1
+
2
# `systemd` journal plugin
3
4
[KEY FEATURES](#key-features) | [JOURNAL SOURCES](#journal-sources) | [JOURNAL FIELDS](#journal-fields) |
@@ -366,312 +367,295 @@ by centralizing your system logs and the use of this Netdata plugin, is unparall
367
### How do I configure a journals' centralization server?
368
369
A short summary to get journal server running can be found below.
369
-There are two strategies you can apply, when it comes down to a centralized server for systemd journal logs.
370
+There are two strategies you can apply, when it comes down to a centralized server for `systemd` journal logs.
371
372
1. _Active sources_, where the centralized server fetches the logs from each individual server
373
2. _Passive sources_, where the centralized server accepts a log stream from an individual server.
374
375
For more options and reference to documentation, check `man systemd-journal-remote` and `man systemd-journal-upload`.
376
376
-We will focus on providing some instructions on setting up a _passive_ centralized server.
377
+#### _passive_ journals' centralization without encryption
378
378
-⚠️ Two things to keep always in mind:
379
+> ℹ️ _passive_ is a journal server that waits for clients to push their metrics to it.
380
380
-1. `systemd-journal-remote` doesn't provide a mechanism to authorize each individual server to write its logs to
381
- the parent server. Especially in public-faced servers you need to make sure that the endpoints of this service
382
- are protected from "bad actors" (for instance; on the centralization server, allow traffic to the
383
- `systemd-journal-remote` specific port (`19532`) only from each individual server)
384
-2. Even with TLS enabled on the centralization server, we don't advise you to push systemd journal logs over the public
385
- network. Prefer cleaner approaches, for instance, create one centralization server per one specific subnet of your
386
- VPC.
381
+> ⚠️ **IMPORTANT**
382
+> These instructions will copy your logs to a central server, without any encryption or authorization.
383
+> DO NOT USE THIS ON NON-TRUSTED NETWORKS.
384
388
-#### Configuring a journals' centralization server
385
+##### _passive_ server, without encryption
386
390
-On the centralization server install `systemd-journal-remote`, and enable it with `systemctl`, like this:
387
+On the centralization server install `systemd-journal-remote`:
388
389
```sh
390
# change this according to your distro
391
sudo apt-get install systemd-journal-remote
392
+```
393
+
394
+Make sure the journal transfer protocol is `http`:
395
+
396
+```sh
397
+sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/
398
+
399
+# edit it to make sure it says:
400
+# --listen-http=-3
401
+# not:
402
+# --listen-https=-3
403
+sudo nano /etc/systemd/system/systemd-journal-remote.service
404
+
405
+# reload systemd
406
+sudo systemctl daemon-reload
407
+```
408
+
409
+Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`
410
+
411
+```sh
412
+# edit the socket file
413
+sudo systemctl edit systemd-journal-remote.socket
414
+```
415
+
416
+and add the following lines into the instructed place, and choose your desired port; save and exit.
417
+
418
+```sh
419
+[Socket]
420
+ListenStream=<DESIRED_PORT>
421
+```
422
+
423
+Finally, enable it, so that it will start automatically upon receiving a connection:
424
+
425
+```
426
+# enable systemd-journal-remote
427
+sudo systemctl enable --now systemd-journal-remote.socket
428
+sudo systemctl enable systemd-journal-remote.service
429
+```
430
+
431
+`systemd-journal-remote` is now listening for incoming journals from remote hosts.
432
+
433
+##### _passive_ client, without encryption
434
+
435
+On the clients, install `systemd-journal-remote`:
436
+
437
+```sh
438
+# change this according to your distro
439
+sudo apt-get install systemd-journal-remote
440
+```
441
+
442
+Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:
443
+
444
+```
445
+[Upload]
446
+URL=http://centralization.server.ip:19532
447
+```
448
+
449
+Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:
450
+
451
+```sh
452
+sudo systemctl edit systemd-journal-upload
453
+```
454
+
455
+At the top, add:
456
+
457
+```
458
+[Service]
459
+Restart=always
460
+```
461
+
462
+Enable and start `systemd-journal-upload`, like this:
463
+
464
+```sh
465
+sudo systemctl enable systemd-journal-upload
466
+sudo systemctl start systemd-journal-upload
467
+```
468
+
469
+##### verify it works
470
+
471
+To verify the central server is receiving logs, run this on the central server:
472
+
473
+```sh
474
+sudo ls -l /var/log/journal/remote/
475
+```
476
+
477
+You should see new files from the client's IP.
478
+
479
+Also, `systemctl status systemd-journal-remote` should show something like this:
480
+
481
+```
482
+systemd-journal-remote.service - Journal Remote Sink Service
483
+ Loaded: loaded (/etc/systemd/system/systemd-journal-remote.service; indirect; preset: disabled)
484
+ Active: active (running) since Sun 2023-10-15 14:29:46 EEST; 2h 24min ago
485
+TriggeredBy: ● systemd-journal-remote.socket
486
+ Docs: man:systemd-journal-remote(8)
487
+ man:journal-remote.conf(5)
488
+ Main PID: 2118153 (systemd-journal)
489
+ Status: "Processing requests..."
490
+ Tasks: 1 (limit: 154152)
491
+ Memory: 2.2M
492
+ CPU: 71ms
493
+ CGroup: /system.slice/systemd-journal-remote.service
494
+ └─2118153 /usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/
495
+```
496
+
497
+Note the `status: "Processing requests..."` and the PID under `CGroup`.
498
+
499
+On the client `systemctl status systemd-journal-upload` should show something like this:
500
+
501
+```
502
+● systemd-journal-upload.service - Journal Remote Upload Service
503
+ Loaded: loaded (/lib/systemd/system/systemd-journal-upload.service; enabled; vendor preset: disabled)
504
+ Drop-In: /etc/systemd/system/systemd-journal-upload.service.d
505
+ └─override.conf
506
+ Active: active (running) since Sun 2023-10-15 10:39:04 UTC; 3h 17min ago
507
+ Docs: man:systemd-journal-upload(8)
508
+ Main PID: 4169 (systemd-journal)
509
+ Status: "Processing input..."
510
+ Tasks: 1 (limit: 13868)
511
+ Memory: 3.5M
512
+ CPU: 1.081s
513
+ CGroup: /system.slice/systemd-journal-upload.service
514
+ └─4169 /lib/systemd/systemd-journal-upload --save-state
515
+```
516
+
517
+Note the `Status: "Processing input..."` and the PID under `CGroup`.
518
+
519
+#### _passive_ journals' centralization with encryption using self-signed certificates
520
+
521
+> ℹ️ _passive_ is a journal server that waits for clients to push their metrics to it.
522
+
523
+##### _passive_ server, with encryption and self-singed certificates
524
+
525
+On the centralization server install `systemd-journal-remote` and `openssl`:
526
+
527
+```sh
528
+# change this according to your distro
529
+sudo apt-get install systemd-journal-remote openssl
530
+```
531
396
-# enable receiving
532
+Make sure the journal transfer protocol is `https`:
533
+
534
+```sh
535
+sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/
536
+
537
+# edit it to make sure it says:
538
+# --listen-https=-3
539
+# not:
540
+# --listen-http=-3
541
+sudo nano /etc/systemd/system/systemd-journal-remote.service
542
+
543
+# reload systemd
544
+sudo systemctl daemon-reload
545
+```
546
+
547
+Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`
548
+
549
+```sh
550
+# edit the socket file
551
+sudo systemctl edit systemd-journal-remote.socket
552
+```
553
+
554
+and add the following lines into the instructed place, and choose your desired port; save and exit.
555
+
556
+```sh
557
+[Socket]
558
+ListenStream=<DESIRED_PORT>
559
+```
560
+
561
+Finally, enable it, so that it will start automatically upon receiving a connection:
562
+
563
+```sh
564
+# enable systemd-journal-remote
565
sudo systemctl enable --now systemd-journal-remote.socket
566
sudo systemctl enable systemd-journal-remote.service
567
```
568
401
-`systemd-journal-remote` is now listening for incoming journals from remote hosts, on port `19532`.
402
-Please note that `systemd-journal-remote` supports using secure connections.
403
-
404
-##### Centralization server without TLS (use case; only for secure intranets)
405
-
406
-1. To change the protocol of the journal transfer (from HTTPS, which is the default to HTTP), edit the service file of
407
- the `systemd-journal-remote` service.
408
-
409
- ```sh
410
- sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/
411
-
412
- # edit it
413
- # --listen-http=-3 specifies the incoming journal for http.
414
- # If you want to use https, change it to --listen-https=-3.
415
- nano /etc/systemd/system/systemd-journal-remote.service
416
-
417
- # reload systemd
418
- sudo systemctl daemon-reload
419
- ```
420
-
421
- This will make HTTP requests as priority.
422
-
423
-2. Reload the daemon configs
424
-
425
- ```sh
426
- # reload systemd
427
- sudo systemctl daemon-reload
428
- ```
429
-
430
-3. (OPTIONAL) If you want to change the port, edit the socket file of the `systemd-journal-remote`
431
-
432
- ```sh
433
- # copy the service file
434
- sudo systemctl edit systemd-journal-remote.socket
435
- ```
436
-
437
- and add the following lines into the instructed place, and choose your desired port; save and exit.
438
-
439
- ```sh
440
- [Socket]
441
- ListenStream=<DESIRED_PORT>
442
- ```
443
-
444
-
445
-4. Secure the endpoint from unauthorized access. That depends on your setup (e.g firewall setting, reverse proxies, etc)
446
-
447
-##### Centralization server with TLS and self-signed certificate
448
-
449
-Follow **all** the steps from
450
-the [Centralization server without TLS (use case; only in secure intranets](#centralization-server-without-tls--use-case-only-in-secure-intranets-)
451
-but omit step 2. Instead of step two, take the following steps:
452
-
453
-> 💡 You need to handcraft and use a self-signed certificate. A pretty straightforward way to do that is:
454
-
455
-1. Download OpenSSL
456
-
457
- ```sh
458
- # change this according to your distro
459
- sudo apt-get install openssl
460
- ```
461
-
462
-2. Create your own private certificate authority.
463
-
464
- ```sh
465
- mkdir self-signed-certificates && cd self-signed-certificates
466
-
467
- openssl req -newkey rsa:2048 -days 3650 -x509 -nodes \
468
- -out ca.pem -keyout ca.key -subj '/CN=My Certificate authority/'
469
-
470
- cat >ca.conf <<EOF
471
- [ ca ]
472
- default_ca = CA_default
473
- [ CA_default ]
474
- new_certs_dir = .
475
- certificate = ca.pem
476
- database = ./index
477
- private_key = ca.key
478
- serial = ./serial
479
- default_days = 3650
480
- default_md = default
481
- policy = policy_anything
482
- [ policy_anything ]
483
- countryName = optional
484
- stateOrProvinceName = optional
485
- localityName = optional
486
- organizationName = optional
487
- organizationalUnitName = optional
488
- commonName = supplied
489
- emailAddress = optional
490
- EOF
491
-
492
- touch index
493
- echo 0001 >serial
494
- ```
495
-
496
-3. Specify the Common Names for both the server and the clients (server who will push its journal logs).
497
-
498
- How each a client will reach the centralized the server? For instance if you want to reach them via public IP or DNS.
499
- There is a 1:1 correlation among the elements of `CLIENT_CNS`, `CLIENT_IPS` and `CLIENT_DNES`, if you want to omit
500
- a field, replace it with a placeholder string. You can also omit the DNSes but keep in mind to also omit the field
501
- `DNS:$SERVER_DNS` when you are signing the client certificates.
502
-
503
- ```sh
504
- SERVER_CN="myserver.example.com"
505
- SERVER_IP="Server_IP"
506
- SERVER_DNS="myserver.example.com"
507
-
508
- CLIENT_CNS=("client1.example.com" "client2.example.com")
509
- CLIENT_IPS=("client1_ip" "client2_ip")
510
- CLIENT_DNES=("client1.example.com" "client2.example.com")
511
- ```
512
-
513
-4. Create the self-signed certificates for the server and the clients
514
-
515
- ```sh
516
- openssl req -newkey rsa:2048 -nodes -out $SERVER_CN.csr -keyout $SERVER_CN.key -subj "/CN=$SERVER_CN/"
517
- echo "subjectAltName = IP:$SERVER_IP, DNS:$SERVER_DNS" > $SERVER_CN.ext
518
- openssl ca -batch -config ca.conf -notext -in $SERVER_CN.csr -out $SERVER_CN.pem -extfile $SERVER_CN.ext
519
-
520
- for i in "${!CLIENT_CNS[@]}"; do
521
- CLIENT_CN="${CLIENT_CNS[$i]}"
522
- CLIENT_IP="${CLIENT_IPS[$i]}"
523
- CLIENT_DNS="${CLIENT_DNES[$i]}"
524
- # Generate the client CSR
525
- openssl req -newkey rsa:2048 -nodes -out $CLIENT_CN.csr -keyout $CLIENT_CN.key -subj "/CN=$CLIENT_CN/"
526
- echo "subjectAltName = IP:$CLIENT_IP" > $CLIENT_CN.ext
527
-
528
- # Sign the client certificate using the CA configuration
529
- openssl ca -batch -config ca.conf -notext -in $CLIENT_CN.csr -out $CLIENT_CN.pem -extfile $CLIENT_CN.ext
530
- done
531
- ```
532
-
533
- Keep in mind we have already produced the client certificates, we will make use of them when we will configure the
534
- clients.
535
-
536
-5. Copy the key and the certificates into the `systemd-journal-remote`'s predefined places
537
-
538
- ```sh
539
- sudo mkdir /etc/ssl/private # make sure that you havent created this folder, and use it already, you may dont want to change it's permissions
540
- sudo chmod 755 /etc/ssl/private
541
- sudo mkdir /etc/ssl/ca/ # make sure that you havent created this folder, and use it already, you may dont want to change it's permissions
542
- sudo chmod 755 /etc/ssl/ca
543
- sudo cp "${SERVER_CN}".key /etc/ssl/private/journal-remote.key # This is not predefined but we need to clarify the key
544
- sudo cp "${SERVER_CN}".pem /etc/ssl/certs/journal-remote.pem
545
- sudo cp ca.pem /etc/ssl/ca/trusted.pem
546
- ```
547
-
548
-6. Adjust the permissions for the `systemd-journal-remote` to access them.
549
-
550
- ```sh
551
- sudo chgrp systemd-journal-remote /etc/ssl/private/journal-remote.key
552
- sudo chgrp systemd-journal-remote /etc/ssl/certs/journal-remote.pem
553
- sudo chgrp systemd-journal-remote /etc/ssl/ca/trusted.pem
554
-
555
-
556
- sudo chmod 0640 /etc/ssl/private/journal-remote.key
557
- sudo chmod 755 /etc/ssl/certs/journal-remote.pem
558
- sudo chmod 755 /etc/ssl/ca/trusted.pem
559
- ```
560
-
561
-7. Edit the `systemd-journal-remote.conf` to change the predefined key place and enable SSL.
562
-
563
- ```sh
564
- sudo nano /etc/systemd/journal-remote.conf
565
- ```
566
-
567
- You need to transform the corresponding section to something like this
568
- ```
569
- [Remote]
570
- Seal=false
571
- SplitMode=host
572
- ServerKeyFile=/etc/ssl/private/journal-remote.key
573
- ServerCertificateFile=/etc/ssl/certs/journal-remote.pem
574
- TrustedCertificateFile=/etc/ssl/ca/trusted.pem
575
- ```
576
-
577
-#### Configuring journal clients to push their logs to the server
578
-
579
-In this section we will configure the clients/hosts to push their journal logs into the centralization server. You will
580
-install `systemd-journal-remote`,
581
-configure `systemd-journal-upload` (with or without SSL), enable and start it.
582
-
583
-1. To install `systemd-journal-remote`, run:
584
-
585
- ```sh
586
- # change this according to your distro
587
- sudo apt-get install systemd-journal-remote
588
- ```
589
-
590
-
591
-2. **With SSL**: Copy to the client/hosts the self-signed certificates you created before (you created one per host)
592
-
593
- After this step, each host must have the following inside a directory (e.g. `home/user/incoming`),
594
- ```
595
- clientX.example.com.key
596
- clientX.example.com.pem
597
- ca.pem #common between the servers
598
- ```
599
-
600
-3. **With SSL**: _On each client/host;_ create a user and a group (with the same name) called `systemd-journal-upload`
601
-
602
- ```sh
603
- sudo adduser --system --home /run/systemd --no-create-home --disabled-login --group systemd-journal-upload
604
- ```
605
-
606
-4. **With SSL**: _On each client/host;_ Navigate under the directory you placed the certificates, copy them into the
607
- expected locations (by the `systemd-journal-upload` service)
608
-
609
- ```sh
610
- sudo mkdir /etc/ssl/private # make sure that you havent created this folder, and use it already, you may dont want to change it's permissions.
611
- sudo chmod 755 /etc/ssl/private
612
- sudo mkdir /etc/ssl/ca/ # make sure that you havent created this folder, and use it already, you may dont want to change it's permissions.
613
- sudo chmod 755 /etc/ssl/ca
614
-
615
- cd home/user/incoming #change it accordingly, this is the place where you copied your certificates.
616
- sudo cp clientX.example.com.key /etc/ssl/private/journal-upload.key
617
- sudo cp clientX.example.com.pem /etc/ssl/certs/journal-upload.pem
618
- sudo cp ca.pem /etc/ssl/ca/trusted.pem
619
- ```
620
-
621
-5. **With SSL**: _On each client/host;_ Adjust the permission so that the `systemd-journal-upload` service can access
622
- the files
623
-
624
- ```sh
625
- sudo chgrp systemd-journal-upload /etc/ssl/private/journal-upload.key
626
- sudo chgrp systemd-journal-upload /etc/ssl/certs/journal-upload.pem
627
- sudo chgrp systemd-journal-upload /etc/ssl/ca/trusted.pem
628
-
629
- sudo chmod 0640 /etc/ssl/private/journal-upload.key
630
- sudo chmod 755 /etc/ssl/certs/journal-upload.pem
631
- sudo chmod 755 /etc/ssl/ca/trusted.pem
632
- ```
633
-
634
-6. Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:
635
-
636
- ```
637
- [Upload]
638
- URL=http://centralization.server.ip:19532
639
- ```
640
-
641
- ⚠️ OR with SSL
642
-
643
- ```
644
- [Upload]
645
- URL=https://CENTRALIZED_SERVER_IP/DOMAIN :19532 #replace it accordingly
646
- ServerKeyFile=/etc/ssl/private/journal-upload.key
647
- ServerCertificateFile=/etc/ssl/certs/journal-upload.pem
648
- TrustedCertificateFile=/etc/ssl/ca/trusted.pem
649
- ```
650
-
651
-7. Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even
652
- if the server is temporarily not there, like this:
653
-
654
- ```sh
655
- sudo systemctl edit systemd-journal-upload
656
- ```
657
-
658
- At the top, add:
659
-
660
- ```
661
- [Service]
662
- Restart=always
663
- ```
664
-
665
-8. Enable and start `systemd-journal-upload`, like this:
666
-
667
- ```sh
668
- sudo systemctl enable systemd-journal-upload
669
- sudo systemctl start systemd-journal-upload
670
- ```
671
-
672
-Keep in mind that immediately after starting `systemd-journal-upload` on a server, a replication process starts pushing
673
-logs in the order they have been received. This means that depending on the size of the available logs, some time may be
674
-needed for Netdata to show the most recent logs of that server.
569
+`systemd-journal-remote` is now listening for incoming journals from remote hosts.
570
+
571
+Use [this script](https://gist.github.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7) to create a self-signed certificates authority and certificates for all your servers.
572
+
573
+```sh
574
+wget -O systemd-journal-self-signed-certs.sh "https://gist.githubusercontent.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7/raw/c346e61e0a66f45dc4095d254bd23917f0a01bd0/systemd-journal-self-signed-certs.sh"
575
+chmod 755 systemd-journal-self-signed-certs.sh
576
+```
577
+
578
+Edit the script and at its top, set your settings:
579
+
580
+```sh
581
+# set the server name
582
+SERVER="server-hostname"
583
+
584
+# Define all the names or IPs this server is reachable
585
+# journal clients can use any of them to connect to this server
586
+SERVER_ALIASES=("DNS:server-hostname1" "DNS:server-hostname2" "IP:1.2.3.4" "IP:10.1.1.1" "IP:172.16.1.1")
587
+
588
+# Define all the names of the clients who will be sending data to the server
589
+# These names are used by journal-remote to name the files in /var/log/journal/remote/
590
+CLIENTS=("vm1" "vm2" "vm3" "add_as_may_as_needed")
591
+```
592
+
593
+Then run the script:
594
+
595
+```sh
596
+sudo ./systemd-journal-self-signed-certs.sh
597
+```
598
+
599
+The script will create the directory `/etc/ssl/systemd-journal-remote` and in it you will find all the certificates needed.
600
+
601
+There will also be files named `runme-on-XXX.sh`. There will be 1 script for the server and 1 script for each of the clients. You can copy and paste (or `scp`) these scripts on your server and each of your clients and run them as root:
602
+
603
+```sh
604
+scp /etc/ssl/systemd-journal-remote/runme-on-XXX.sh XXX:/tmp/
605
+```
606
+
607
+Once the above is done, `ssh` to each server/client and do:
608
+
609
+```sh
610
+sudo bash /tmp/runme-on-XXX.sh
611
+```
612
+
613
+The scripts install the needed certificates, fix their file permissions to be accessible by systemd-journal-remote/upload, change `/etc/systemd/journal-remote.conf` (on the server) or `/etc/systemd/journal-upload.conf` on the clients and restart the relevant services.
614
+
615
+
616
+##### _passive_ client, with encryption and self-singed certificates
617
+
618
+On the clients, install `systemd-journal-remote`:
619
+
620
+```sh
621
+# change this according to your distro
622
+sudo apt-get install systemd-journal-remote
623
+```
624
+
625
+Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:
626
+
627
+```
628
+[Upload]
629
+URL=https://centralization.server.ip:19532
630
+```
631
+
632
+Make sure that `centralization.server.ip` is one of the `SERVER_ALIASES` when you created the certificates.
633
+
634
+Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:
635
+
636
+```sh
637
+sudo systemctl edit systemd-journal-upload
638
+```
639
+
640
+At the top, add:
641
+
642
+```
643
+[Service]
644
+Restart=always
645
+```
646
+
647
+Enable and start `systemd-journal-upload`, like this:
648
+
649
+```sh
650
+sudo systemctl enable systemd-journal-upload
651
+```
652
+
653
+Copy the relevant `runme-on-XXX.sh` script as described on server setup and run it:
654
+
655
+```sh
656
+sudo bash /tmp/runme-on-XXX.sh
657
+```
658
+
659
660
#### Limitations when using a logs centralization server
661