Update active_journal_centralization_guide_no_encryption.md
Costa Tsaousis committed
Oct 20, 2023 at 16:57 UTC
7be484bd20d7b1a9f03a0a6184e15c204d30f771
1 file changed
+44
-78
collectors/systemd-journal.plugin/active_journal_centralization_guide_no_encryption.md
+44
-78
@@ -1,23 +1,16 @@
1
-# Active journal centralization without encryption
1
+# Active journal source without encryption
2
3
-This page will guide you through creating an active journal centralization setup without the use of encryption.
3
+This page will guide you through creating an active journal source without the use of encryption.
4
5
-Once you centralize your infrastructure logs to a server, Netdata will automatically detect all the logs from all
6
-servers and organize them in sources.
7
-With the setup described in this document, journal files are identified by the hostnames of the clients you pull logs.
8
-
9
-An _active_ journal server fetch logs from clients, so in this setup we will:
10
-
11
-1. configure `systemd-journal-remote` on the server, to pull journal logs.
12
-2. configure `systemd-journal-gatewayd` on the clients, to serve their logs to the micro http server.
5
+Once you enable an active journal source on a server, `systemd-journal-gatewayd` will expose an REST API on TCP port 19531. This API can be used for querying the logs, exporting the logs, or monitoring new log entries, remotely.
6
7
> ⚠️ **IMPORTANT**<br/>
15
-> These instructions will copy your logs to a central server, without any encryption or authorization.<br/>
8
+> These instructions will expose your logs to the network, without any encryption or authorization.<br/>
9
> DO NOT USE THIS ON NON-TRUSTED NETWORKS.
10
18
-## Client configuration
11
+## Configuring an active journal source
12
20
-On the clients, install `systemd-journal-gateway`.
13
+On the server you want to expose their logs, install `systemd-journal-gateway`.
14
15
```bash
16
# change this according to your distro
@@ -44,91 +37,64 @@ Finally, enable it, so that it will start automatically upon receiving a connect
37
# enable systemd-journal-remote
38
sudo systemctl daemon-reload
39
sudo systemctl enable --now systemd-journal-gatewayd.socket
47
-sudo systemctl enable systemd-journal-gatewayd.service
48
-sudo systemctl start systemd-journal-gatewayd.service
40
```
41
51
-## Server configuration
52
-
53
-On the centralization server install `systemd-journal-remote`:
54
-
55
-```bash
56
-# change this according to your distro
57
-sudo apt-get install systemd-journal-remote
58
-```
42
+## Using the active journal source
43
60
-Start it once to make sure than the `systemd-journal-remote` created any necessary requirement to work as centralization
61
-server. To do that, you need to spin up a temporarily _passive_ server with http, then close it, if you won't use it
62
-also as a passive server.
44
+### Simple Logs Explorer
45
64
-```bash
65
-sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/
46
+`systemd-journal-gateway` provides a simple HTML5 application to browse the logs.
47
67
-# edit it to make sure it says:
68
-# --listen-http=-3
69
-# not:
70
-# --listen-https=-3
71
-sudo nano /etc/systemd/system/systemd-journal-remote.service
48
+To use it, open your web browser and navigate to:
49
73
-# reload systemd
74
-sudo systemctl daemon-reload
50
+```
51
+http://server.ip:19531/browse
52
```
53
77
-Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`
54
+A simple page like this will be presented:
55
79
-```bash
80
-# edit the socket file
81
-sudo systemctl edit systemd-journal-remote.socket
82
-```
56
+
57
84
-and add the following lines into the instructed place, and choose your desired port; save and exit.
58
+### Use it with `curl`
59
86
-```bash
87
-[Socket]
88
-ListenStream=<DESIRED_PORT>
89
-```
60
+`man systemd-journal-gatewayd` documents the supported API methods and provides examples to query the API using `curl` commands.
61
91
-Start and (stop it, if you won't use it also as _passive_).
62
+### Copying the logs to a central journals server
63
93
-```bash
94
-# enable systemd-journal-remote
95
-sudo systemctl start systemd-journal-remote.service
96
-sudo systemctl stop systemd-journal-remote.service
97
-```
64
+`systemd-journal-remote` has the ability to query instances of `systemd-journal-gatewayd` to fetch their logs, so that the central server fetches the logs, instead of waiting for the individual servers to push their logs to it.
65
99
-For each of your clients (endpoints that you want to fetch journal logs from) create a service that will use
100
-`systemd-journal-remote` will always fetch the logs.
66
+However, this kind of logs centralization has a key problem: **there is no guarantee that there will be no gaps in the logs replicated**. Theoretically, the REST API of `systemd-journal-gatewayd` supports querying past data, and `systemd-journal-remote` could keep track of the state of replication and automatically continue from the point it stopped last time. But it does not. So, currently the best logs centralization option is to use a **passive** centralization, where the clients push their logs to the server.
67
68
+Given these limitations, if you still want to configure an **active** journals centralization, this is what you need to do:
69
+
70
+On the centralization server install `systemd-journal-remote`:
71
72
```bash
104
-sudo nano /etc/systemd/system/systemd-journal-endpoint-X.service
73
+# change this according to your distro
74
+sudo apt-get install systemd-journal-remote
75
```
76
107
-Copy the service file above, replace the Description and `TARGET_HOST`, save and exit
77
+Then, copy `systemd-journal-remote.service` to configure it for querying the active source:
78
109
-```
110
-[Unit]
111
-Description=Fetching systemd journal logs from my endpoint X
79
+```bash
80
+# replace "clientX" with the name of the active client node
81
+sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/systemd-journal-remote-clientX.service
82
113
-[Service]
114
-ExecStart=/usr/lib/systemd/systemd-journal-remote --url http://<TARGET_HOST>:19531/entries?follow
115
-Type=simple
116
-Restart=always
117
-User=systemd-journal-remote
83
+# edit it to make sure it the ExecStart line is like this:
84
+# ExecStart=/usr/lib/systemd/systemd-journal-remote --url http://clientX:19531/entries?follow
85
+sudo nano /etc/systemd/system/systemd-journal-remote-clientX.service
86
119
-[Install]
120
-WantedBy=multi-user.target
87
+# reload systemd
88
+sudo systemctl daemon-reload
89
```
90
123
-Repeat the same for every host that you want to fetch journal logs.
124
-Reload the systemd daemon config, enable each service and start, like this:
125
-
91
```bash
127
-sudo systemctl daemon-reload
128
-sudo systemctl enable systemd-journal-endpoint-X
129
-sudo systemctl start systemd-journal-endpoint-X
92
+# enable systemd-journal-remote
93
+sudo systemctl enable --now systemd-journal-remote-clientX.service
94
```
95
96
+You can repeat this process to create as many `systemd-journal-remote` services, as the active source you have.
97
+
98
## Verify it works
99
100
To verify the central server is receiving logs, run this on the central server:
@@ -137,13 +103,13 @@ To verify the central server is receiving logs, run this on the central server:
103
sudo ls -l /var/log/journal/remote/
104
```
105
140
-You should see new files from the client's hostname.
106
+You should see new files from the client's hostname or IP.
107
142
-Also, any of the new service files (`systemctl status systemd-journal-endpoint-X`) should show something like this:
108
+Also, any of the new service files (`systemctl status systemd-journal-clientX`) should show something like this:
109
110
```bash
145
-● systemd-journal-client1.service - Fetching systemd journal logs from 192.168.2.146
146
- Loaded: loaded (/etc/systemd/system/systemd-journal-client1.service; enabled; preset: disabled)
111
+● systemd-journal-clientX.service - Fetching systemd journal logs from 192.168.2.146
112
+ Loaded: loaded (/etc/systemd/system/systemd-journal-clientX.service; enabled; preset: disabled)
113
Drop-In: /usr/lib/systemd/system/service.d
114
└─10-timeout-abort.conf
115
Active: active (running) since Wed 2023-10-18 07:35:52 EEST; 23min ago
@@ -151,10 +117,10 @@ Also, any of the new service files (`systemctl status systemd-journal-endpoint-X
117
Tasks: 2 (limit: 6928)
118
Memory: 7.7M
119
CPU: 518ms
154
- CGroup: /system.slice/systemd-journal-client1.service
120
+ CGroup: /system.slice/systemd-journal-clientX.service
121
├─77959 /usr/lib/systemd/systemd-journal-remote --url "http://192.168.2.146:19531/entries?follow"
122
└─77962 curl "-HAccept: application/vnd.fdo.journal" --silent --show-error "http://192.168.2.146:19531/entries?follow"
123
158
-Oct 18 07:35:52 systemd-journal-server systemd[1]: Started systemd-journal-client1.service - Fetching systemd journal logs from 192.168.2.146.
124
+Oct 18 07:35:52 systemd-journal-server systemd[1]: Started systemd-journal-clientX.service - Fetching systemd journal logs from 192.168.2.146.
125
Oct 18 07:35:52 systemd-journal-server systemd-journal-remote[77959]: Spawning curl http://192.168.2.146:19531/entries?follow...
160
-```
\ No newline at end of file
126
+```