Update passive_journal_centralization_guide_self_signed_certs.md
Costa Tsaousis committed
Oct 17, 2023 at 20:25 UTC
e2c901a546cb12bfc41f479acddf8326b6ac83fb
1 file changed
+118
-57
collectors/systemd-journal.plugin/passive_journal_centralization_guide_self_signed_certs.md
+118
-57
@@ -2,20 +2,72 @@
2
3
This page will guide you through creating a passive journal centralization setup using self-signed certificates for encryption.
4
5
-> A _passive_ journal server waits for clients to push their metrics to it.
5
+A _passive_ journal server waits for clients to push their metrics to it, so in this setup we will:
6
+
7
+1. configure a certificates authority and issue self-signed certificates for your servers.
8
+2. configure `systemd-journal-remote` on the server, to listen for incoming connections.
9
+3. configure `systemd-journal-upload` on the clients, to push their logs to the server.
10
+
11
+## Self-signed certificates
12
+
13
+Use [this script](https://gist.github.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7) to create a self-signed certificates authority and certificates for all your servers.
14
+
15
+```bash
16
+wget -O systemd-journal-self-signed-certs.sh "https://gist.githubusercontent.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7/raw/c346e61e0a66f45dc4095d254bd23917f0a01bd0/systemd-journal-self-signed-certs.sh"
17
+chmod 755 systemd-journal-self-signed-certs.sh
18
+```
19
+
20
+Edit the script and at its top, set your settings:
21
+
22
+```bash
23
+# The directory to save the generated certificates (and everything about this certificate authority).
24
+# This is only used on the node generating the certificates (usually on the journals server).
25
+DIR="/etc/ssl/systemd-journal-remote"
26
+
27
+# The journals centralization server name (the CN of the server certificate).
28
+SERVER="server-hostname"
29
+
30
+# All the DNS names or IPs this server is reachable at (the certificate will include them).
31
+# Journal clients can use any of them to connect to this server.
32
+# systemd-journal-upload validates its URL= hostname, against this list.
33
+SERVER_ALIASES=("DNS:server-hostname1" "DNS:server-hostname2" "IP:1.2.3.4" "IP:10.1.1.1" "IP:172.16.1.1")
34
+
35
+# All the names of the journal clients that will be sending logs to the server (the CNs of their certificates).
36
+# These names are used by systemd-journal-remote to name the journal files in /var/log/journal/remote/.
37
+# Also the remote hosts will be presented using these names on Netdata dashboards.
38
+CLIENTS=("vm1" "vm2" "vm3" "add_as_may_as_needed")
39
+```
40
+
41
+Then run the script:
42
+
43
+```bash
44
+sudo ./systemd-journal-self-signed-certs.sh
45
+```
46
+
47
+The script will create the directory `/etc/ssl/systemd-journal-remote` and in it you will find all the certificates needed.
48
+
49
+There will also be files named `runme-on-XXX.sh`. There will be 1 script for the server and 1 script for each of the clients.
50
+
51
+These `runme-on-XXX.sh` scripts install the needed certificates, fix their file permissions to be accessible by systemd-journal-remote/upload, change `/etc/systemd/journal-remote.conf` (on the server) or `/etc/systemd/journal-upload.conf` (on the clients) and restart the relevant services.
52
+
53
+You can copy and paste (or `scp`) these scripts on your server and each of your clients:
54
+
55
+```bash
56
+scp /etc/ssl/systemd-journal-remote/runme-on-XXX.sh XXX:/tmp/
57
+```
58
59
## Server configuration
60
61
On the centralization server install `systemd-journal-remote` and `openssl`:
62
11
-```sh
63
+```bash
64
# change this according to your distro
65
sudo apt-get install systemd-journal-remote openssl
66
```
67
68
Make sure the journal transfer protocol is `https`:
69
18
-```sh
70
+```bash
71
sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/
72
73
# edit it to make sure it says:
@@ -30,83 +82,39 @@ sudo systemctl daemon-reload
82
83
Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`
84
33
-```sh
85
+```bash
86
# edit the socket file
87
sudo systemctl edit systemd-journal-remote.socket
88
```
89
90
and add the following lines into the instructed place, and choose your desired port; save and exit.
91
40
-```sh
92
+```bash
93
[Socket]
94
ListenStream=<DESIRED_PORT>
95
```
96
97
Finally, enable it, so that it will start automatically upon receiving a connection:
98
47
-```sh
99
+```bash
100
# enable systemd-journal-remote
101
sudo systemctl enable --now systemd-journal-remote.socket
102
sudo systemctl enable systemd-journal-remote.service
103
```
104
53
-`systemd-journal-remote` is now listening for incoming journals from remote hosts.
54
-
55
-Use [this script](https://gist.github.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7) to create a self-signed certificates authority and certificates for all your servers.
56
-
57
-```sh
58
-wget -O systemd-journal-self-signed-certs.sh "https://gist.githubusercontent.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7/raw/c346e61e0a66f45dc4095d254bd23917f0a01bd0/systemd-journal-self-signed-certs.sh"
59
-chmod 755 systemd-journal-self-signed-certs.sh
60
-```
61
-
62
-Edit the script and at its top, set your settings:
63
-
64
-```sh
65
-# The directory to save the generated certificates (and everything about this certificate authority).
66
-# This is only used on the node generating the certificates (usually on the journals server).
67
-DIR="/etc/ssl/systemd-journal-remote"
68
-
69
-# The journals centralization server name (the CN of the server certificate).
70
-SERVER="server-hostname"
71
-
72
-# All the DNS names or IPs this server is reachable at (the certificate will include them).
73
-# Journal clients can use any of them to connect to this server.
74
-# systemd-journal-upload validates its URL= hostname, against this list.
75
-SERVER_ALIASES=("DNS:server-hostname1" "DNS:server-hostname2" "IP:1.2.3.4" "IP:10.1.1.1" "IP:172.16.1.1")
76
-
77
-# All the names of the journal clients that will be sending logs to the server (the CNs of their certificates).
78
-# These names are used by systemd-journal-remote to name the journal files in /var/log/journal/remote/.
79
-# Also the remote hosts will be presented using these names on Netdata dashboards.
80
-CLIENTS=("vm1" "vm2" "vm3" "add_as_may_as_needed")
81
-```
82
-
83
-Then run the script:
84
-
85
-```sh
86
-sudo ./systemd-journal-self-signed-certs.sh
87
-```
88
-
89
-The script will create the directory `/etc/ssl/systemd-journal-remote` and in it you will find all the certificates needed.
90
-
91
-There will also be files named `runme-on-XXX.sh`. There will be 1 script for the server and 1 script for each of the clients. You can copy and paste (or `scp`) these scripts on your server and each of your clients and run them as root:
92
-
93
-```sh
94
-scp /etc/ssl/systemd-journal-remote/runme-on-XXX.sh XXX:/tmp/
95
-```
96
-
97
-Once the above is done, `ssh` to each server/client and do:
105
+Assuming that you have already copied the `runme-on-XXX.sh` script on the server, run this:
106
99
-```sh
107
+```bash
108
sudo bash /tmp/runme-on-XXX.sh
109
```
110
103
-The scripts install the needed certificates, fix their file permissions to be accessible by systemd-journal-remote/upload, change `/etc/systemd/journal-remote.conf` (on the server) or `/etc/systemd/journal-upload.conf` on the clients and restart the relevant services.
111
+`systemd-journal-remote` is now listening for incoming journals from remote hosts.
112
113
## Client configuration
114
115
On the clients, install `systemd-journal-remote`:
116
109
-```sh
117
+```bash
118
# change this according to your distro
119
sudo apt-get install systemd-journal-remote
120
```
@@ -122,7 +130,7 @@ Make sure that `centralization.server.ip` is one of the `SERVER_ALIASES` when yo
130
131
Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:
132
125
-```sh
133
+```bash
134
sudo systemctl edit systemd-journal-upload
135
```
136
@@ -135,12 +143,65 @@ Restart=always
143
144
Enable and start `systemd-journal-upload`, like this:
145
138
-```sh
146
+```bash
147
sudo systemctl enable systemd-journal-upload
148
```
149
150
Copy the relevant `runme-on-XXX.sh` script as described on server setup and run it:
151
144
-```sh
152
+```bash
153
sudo bash /tmp/runme-on-XXX.sh
154
```
155
+
156
+The client should now be pushing logs to the central server.
157
+
158
+
159
+## Verify it works
160
+
161
+To verify the central server is receiving logs, run this on the central server:
162
+
163
+```bash
164
+sudo ls -l /var/log/journal/remote/
165
+```
166
+
167
+You should see new files from the client's IP.
168
+
169
+Also, `systemctl status systemd-journal-remote` should show something like this:
170
+
171
+```bash
172
+systemd-journal-remote.service - Journal Remote Sink Service
173
+ Loaded: loaded (/etc/systemd/system/systemd-journal-remote.service; indirect; preset: disabled)
174
+ Active: active (running) since Sun 2023-10-15 14:29:46 EEST; 2h 24min ago
175
+TriggeredBy: ● systemd-journal-remote.socket
176
+ Docs: man:systemd-journal-remote(8)
177
+ man:journal-remote.conf(5)
178
+ Main PID: 2118153 (systemd-journal)
179
+ Status: "Processing requests..."
180
+ Tasks: 1 (limit: 154152)
181
+ Memory: 2.2M
182
+ CPU: 71ms
183
+ CGroup: /system.slice/systemd-journal-remote.service
184
+ └─2118153 /usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/
185
+```
186
+
187
+Note the `status: "Processing requests..."` and the PID under `CGroup`.
188
+
189
+On the client `systemctl status systemd-journal-upload` should show something like this:
190
+
191
+```bash
192
+● systemd-journal-upload.service - Journal Remote Upload Service
193
+ Loaded: loaded (/lib/systemd/system/systemd-journal-upload.service; enabled; vendor preset: disabled)
194
+ Drop-In: /etc/systemd/system/systemd-journal-upload.service.d
195
+ └─override.conf
196
+ Active: active (running) since Sun 2023-10-15 10:39:04 UTC; 3h 17min ago
197
+ Docs: man:systemd-journal-upload(8)
198
+ Main PID: 4169 (systemd-journal)
199
+ Status: "Processing input..."
200
+ Tasks: 1 (limit: 13868)
201
+ Memory: 3.5M
202
+ CPU: 1.081s
203
+ CGroup: /system.slice/systemd-journal-upload.service
204
+ └─4169 /lib/systemd/systemd-journal-upload --save-state
205
+```
206
+
207
+Note the `Status: "Processing input..."` and the PID under `CGroup`.