41f4da811 · cryptotaxi247/netdata-1

@@ -4,9 +4,10 @@

4 [PLAY MODE](#play-mode) | [FULL TEXT SEARCH](#full-text-search) | [PERFORMANCE](#query-performance) |

5 [CONFIGURATION](#configuration-and-maintenance) | [FAQ](#faq)

6

7 -The `systemd` journal plugin by Netdata makes viewing, exploring and analyzing `systemd` journal logs simple and efficient.

7 +The `systemd` journal plugin by Netdata makes viewing, exploring and analyzing `systemd` journal logs simple and

8 +efficient.

9 It automatically discovers available journal sources, allows advanced filtering, offers interactive visual

9 -representations and supports exploring the logs of both individual servers and the logs on infrastructure wide

10 +representations and supports exploring the logs of both individual servers and the logs on infrastructure wide

11 journal centralization servers.

12

13 ![image](https://github.com/netdata/netdata/assets/2662304/691b7470-ec56-430c-8b81-0c9e49012679)

@@ -18,10 +19,12 @@ journal centralization servers.

19 - Supports `system`, `user`, `namespaces` and `remote` journals.

20 - Allows filtering on **any journal field** or **field value**, for any time-frame.

21 - Allows **full text search** (`grep`) on all journal fields, for any time-frame.

21 -- Provides a **histogram** for log entries over time, with a break down per field-value, for any field and any time-frame.

22 -- Works directly on journal files, without any other third party components.

22 +- Provides a **histogram** for log entries over time, with a break down per field-value, for any field and any

23 + time-frame.

24 +- Works directly on journal files, without any other third-party components.

25 - Supports coloring log entries, the same way `journalctl` does.

24 -- In PLAY mode provides the same experience as `journalctl -f`, showing new logs entries immediately after they are received.

26 +- In PLAY mode provides the same experience as `journalctl -f`, showing new log entries immediately after they are

27 + received.

28

29 ### Prerequisites

30

@@ -36,9 +39,16 @@ For more information check [this discussion](https://github.com/netdata/netdata/

39

40 The following are limitations related to the availability of the plugin:

41

39 -- This plugin is not available when Netdata is installed in a container. The problem is that `libsystemd` is not available in Alpine Linux (there is a `libsystemd`, but it is a dummy that returns failure on all calls). We plan to change this, by shipping Netdata containers based on Debian.

40 -- For the same reason (lack of `systemd` support for Alpine Linux), the plugin is not available on `static` builds of Netdata (which are based on `muslc`, not `glibc`).

41 -- On old systemd systems (like Centos 7), the plugin runs always in "full data query" mode, which makes it slower. The reason, is that systemd API is missing some important calls we need to use the field indexes of `systemd` journal. However, when running in this mode, the plugin offers also negative matches on the data (like filtering for all logs that do not have set some field), and this is the reason "full data query" mode is also offered as an option even on newer versions of `systemd`.

42 +- This plugin is not available when Netdata is installed in a container. The problem is that `libsystemd` is not

43 + available in Alpine Linux (there is a `libsystemd`, but it is a dummy that returns failure on all calls). We plan to

44 + change this, by shipping Netdata containers based on Debian.

45 +- For the same reason (lack of `systemd` support for Alpine Linux), the plugin is not available on `static` builds of

46 + Netdata (which are based on `muslc`, not `glibc`).

47 +- On old systemd systems (like Centos 7), the plugin runs always in "full data query" mode, which makes it slower. The

48 + reason, is that systemd API is missing some important calls we need to use the field indexes of `systemd` journal.

49 + However, when running in this mode, the plugin offers also negative matches on the data (like filtering for all logs

50 + that do not have set some field), and this is the reason "full data query" mode is also offered as an option even on

51 + newer versions of `systemd`.

52

53 To use the plugin, install one of our native distribution packages, or install it from source.

54

@@ -46,9 +56,14 @@ To use the plugin, install one of our native distribution packages, or install i

56

57 The following are limitations related to the features of `systemd` journal:

58

49 -- This plugin does not support binary field values. `systemd` journal has the ability to assign fields with binary data. This plugin assumes all fields contain text values (text in this context includes numbers).

50 -- This plugin does not support multiple values per field for any given log entry. `systemd` journal has the ability to accept the same field key, multiple times, with multiple values on a single log entry. This plugin will present the last value and ignore the others for this log entry.

51 -- This plugin will only read journal files located in `/var/log/journal` or `/run/log/journal`. `systemd-remote` has the ability to store journal files anywhere (user configured). If journal files are not located in `/var/log/journal` or `/run/log/journal` (and any of their subdirectories), the plugin will not find them.

59 +- This plugin does not support binary field values. `systemd` journal has the ability to assign fields with binary data.

60 + This plugin assumes all fields contain text values (text in this context includes numbers).

61 +- This plugin does not support multiple values per field for any given log entry. `systemd` journal has the ability to

62 + accept the same field key, multiple times, with multiple values on a single log entry. This plugin will present the

63 + last value and ignore the others for this log entry.

64 +- This plugin will only read journal files located in `/var/log/journal` or `/run/log/journal`. `systemd-remote` has the

65 + ability to store journal files anywhere (user configured). If journal files are not located in `/var/log/journal`

66 + or `/run/log/journal` (and any of their subdirectories), the plugin will not find them.

67

68 Other than the above, this plugin supports all features of `systemd` journals.

69

@@ -61,7 +76,8 @@ The plugin automatically detects the available journal sources, based on the jou

76

77 The plugin, by default, merges all journal sources together, to provide a unified view of all log messages available.

78

64 -> To improve query performance, we recommend selecting the relevant journal source, before doing more analysis on the logs.

79 +> To improve query performance, we recommend selecting the relevant journal source, before doing more analysis on the

80 +> logs.

81

82 ### `system` journals

83

@@ -101,14 +117,15 @@ This makes them a little more difficult to handle, from the administration persp

117

118 ### `remote` journals

119

104 -Remote journals are created by `systemd-journal-remote`. This `systemd` feature allows creating logs centralization points within

105 -your infrastructure, based exclusively on `systemd`.

120 +Remote journals are created by `systemd-journal-remote`. This `systemd` feature allows creating logs centralization

121 +points within your infrastructure, based exclusively on `systemd`.

122

123 Usually `remote` journals are named by the IP of the server sending these logs. The Netdata plugin automatically

124 extracts these IPs and performs a reverse DNS lookup to find their hostnames. When this is successful,

125 `remote` journals are named by the hostnames of the origin servers.

126

111 -For information about configuring a journals' centralization server, check [this FAQ item](#how-do-i-configure-a-journals-centralization-server).

127 +For information about configuring a journals' centralization server,

128 +check [this FAQ item](#how-do-i-configure-a-journals-centralization-server).

129

130 ## Journal Fields

131

@@ -119,7 +136,7 @@ while all fields and their values are indexed for fast querying.

136 This means that each application can log messages annotated with its own unique fields and values, and `systemd`

137 journals will automatically index all of them, without any configuration or manual action.

138

122 -For a description of the most frequent fields found in `systemd` journals, check `man systemd.journal-fields`.

139 +For a description of the most frequent fields found in `systemd` journals, check `man systemd.journal-fields`.

140

141 Fields found in the journal files are automatically added to the UI in multiple places to help you explore

142 and filter the data.

@@ -130,7 +147,8 @@ The plugin automatically enriches certain fields to make them more user-friendly

147 - `PRIORITY`: the numeric value is replaced with the human-readable name of each priority.

148 - `SYSLOG_FACILITY`: the encoded value is replaced with the human-readable name of each facility.

149 - `ERRNO`: the numeric value is annotated with the short name of each value.

133 -- `_UID` `_AUDIT_LOGINUID`, `_SYSTEMD_OWNER_UID`, `OBJECT_UID`, `OBJECT_SYSTEMD_OWNER_UID`, `OBJECT_AUDIT_LOGINUID`: the local user database is consulted to annotate them with usernames.

150 +- `_UID` `_AUDIT_LOGINUID`, `_SYSTEMD_OWNER_UID`, `OBJECT_UID`, `OBJECT_SYSTEMD_OWNER_UID`, `OBJECT_AUDIT_LOGINUID`:

151 + the local user database is consulted to annotate them with usernames.

152 - `_GID`, `OBJECT_GID`: the local group database is consulted to annotate them with group names.

153 - `_CAP_EFFECTIVE`: the encoded value is annotated with a human-readable list of the linux capabilities.

154 - `_SOURCE_REALTIME_TIMESTAMP`: the numeric value is annotated with human-readable datetime in UTC.

@@ -142,7 +160,7 @@ The values of all other fields are presented as found in the journals.

160 > For `remote` sources, the names presented may not reflect the actual user and group names on the origin server.

161 > The numeric value will still be visible though, as-is on the origin server.

162

145 -The annotations are not searchable with full text search. They are only added for the presentation of the fields.

163 +The annotations are not searchable with full-text search. They are only added for the presentation of the fields.

164

165 ### Journal fields as columns in the table

166

@@ -152,8 +170,8 @@ All journal fields available in the journal files are offered as columns on the

170

171 ### Journal fields as additional info to each log entry

172

155 -When you click a log line, the `info` sidebar will open on the right of the screen, to provide the full list of fields related to this

156 -log line. You can close this `info` sidebar, by selecting the filter icon at its top.

173 +When you click a log line, the `info` sidebar will open on the right of the screen, to provide the full list of fields

174 +related to this log line. You can close this `info` sidebar, by selecting the filter icon at its top.

175

176 ![image](https://github.com/netdata/netdata/assets/2662304/3207794c-a61b-444c-8ffe-6c07cbc90ae2)

177

@@ -166,7 +184,9 @@ of the query.

184 Internally the plugin has:

185

186 1. A white-list of fields, to be presented as filters.

169 -2. A black-list of fields, to prevent them from becoming filters. This list includes fields with a very high cardinality, like timestamps, unique message ids, etc. This is mainly for protecting the server's performance, to avoid building in memory indexes for the fields that almost each of their values is unique.

187 +2. A black-list of fields, to prevent them from becoming filters. This list includes fields with a very high

188 + cardinality, like timestamps, unique message ids, etc. This is mainly for protecting the server's performance,

189 + to avoid building in memory indexes for the fields that almost each of their values is unique.

190

191 Keep in mind that the values presented in the filters, and their sorting is affected by the "full data queries"

192 setting:

@@ -203,7 +223,8 @@ The histogram presented by the plugin is interactive:

223 The plugin supports PLAY mode, to continuously update the screen with new log entries found in the journal files.

224 Just hit the "play" button at the top of the Netdata dashboard screen.

225

206 -On centralized log servers, PLAY mode provides a unified view of all the new logs encountered across the entire infrastructure,

226 +On centralized log servers, PLAY mode provides a unified view of all the new logs encountered across the entire

227 +infrastructure,

228 from all hosts sending logs to the central logs server via `systemd-remote`.

229

230 ## Full-text search

@@ -212,7 +233,8 @@ The plugin supports searching for any text on all fields of the log entries.

233

234 Full text search is combined with the selected filters.

235

215 -The text box accepts asterisks `*` as wildcards. So, `a*b*c` means match anything that contains `a`, then `b` and then `c` with anything between them.

236 +The text box accepts asterisks `*` as wildcards. So, `a*b*c` means match anything that contains `a`, then `b` and

237 +then `c` with anything between them.

238

239 ## Query performance

240

@@ -226,15 +248,16 @@ On logs aggregation servers, the performance of the queries depend on the follow

248 1. The **number of files** involved in each query.

249

250 This is why we suggest to select a source when possible.

229 -

251 +

252 2. The **speed of the disks** hosting the journal files.

253

254 Journal files perform a lot of reading while querying, so the fastest the disks, the faster the query will finish.

233 -

255 +

256 3. The **memory available** for caching parts of the files.

257

236 - Increased memory will help the kernel cache the most frequently used parts of the journal files, avoiding disk I/O and speeding up queries.

237 -

258 + Increased memory will help the kernel cache the most frequently used parts of the journal files, avoiding disk I/O

259 + and speeding up queries.

260 +

261 4. The **number of filters** applied.

262

263 Queries are significantly faster when just a few filters are selected.

@@ -330,7 +353,7 @@ fields. But journald does exactly the opposite. Each log entry is unique and may

353 So, Loki and `systemd-journal` are good for different use cases.

354

355 `systemd-journal` already runs in your systems. You use it today. It is there inside all your systems

333 -collecting the system and applications logs. And for its use case, it has advantages over other

356 +collecting the system and app~~~~lications logs. And for its use case, it has advantages over other

357 centralization solutions. So, why not use it?

358

359 ### Is it worth to build a `systemd` logs centralization server?

@@ -343,9 +366,25 @@ by centralizing your system logs and the use of this Netdata plugin, is unparall

366 ### How do I configure a journals' centralization server?

367

368 A short summary to get journal server running can be found below.

369 +There are two strategies you can apply, when it comes down to a centralized server for systemd journal logs.

370 +

371 +1. _Active sources_, where the centralized server fetches the logs from each individual server

372 +2. _Passive sources_, where the centralized server accepts a log stream from an individual server.

373

374 For more options and reference to documentation, check `man systemd-journal-remote` and `man systemd-journal-upload`.

375

376 +We will focus on providing some instructions on setting up a _passive_ centralized server.

377 +

378 +⚠️ Two things to keep always in mind:

379 +

380 +1. `systemd-journal-remote` doesn't provide a mechanism to authorize each individual server to write its logs to

381 + the parent server. Especially in public-faced servers you need to make sure that the endpoints of this service

382 + are protected from "bad actors" (for instance; on the centralization server, allow traffic to the

383 + `systemd-journal-remote` specific port (`19532`) only from each individual server)

384 +2. Even with TLS enabled on the centralization server, we don't advise you to push systemd journal logs over the public

385 + network. Prefer cleaner approaches, for instance, create one centralization server per one specific subnet of your

386 + VPC.

387 +

388 #### Configuring a journals' centralization server

389

390 On the centralization server install `systemd-journal-remote`, and enable it with `systemctl`, like this:

@@ -361,73 +400,284 @@ sudo systemctl enable systemd-journal-remote.service

400

401 `systemd-journal-remote` is now listening for incoming journals from remote hosts, on port `19532`.

402 Please note that `systemd-journal-remote` supports using secure connections.

364 -To learn more run `man systemd-journal-remote`.

403

366 -To change the protocol of the journal transfer (HTTP/HTTPS) and the save location, do:

404 +##### Centralization server without TLS (use case; only for secure intranets)

405 +

406 +1. To change the protocol of the journal transfer (from HTTPS, which is the default to HTTP), edit the service file of

407 + the `systemd-journal-remote` service.

408 +

409 + ```sh

410 + sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/

411 +

412 + # edit it

413 + # --listen-http=-3 specifies the incoming journal for http.

414 + # If you want to use https, change it to --listen-https=-3.

415 + nano /etc/systemd/system/systemd-journal-remote.service

416 +

417 + # reload systemd

418 + sudo systemctl daemon-reload

419 + ```

420 +

421 + This will make HTTP requests as priority.

422 +

423 +2. Reload the daemon configs

424 +

425 + ```sh

426 + # reload systemd

427 + sudo systemctl daemon-reload

428 + ```

429 +

430 +3. (OPTIONAL) If you want to change the port, edit the socket file of the `systemd-journal-remote`

431 +

432 + ```sh

433 + # copy the service file

434 + sudo systemctl edit systemd-journal-remote.socket

435 + ```

436 +

437 + and add the following lines into the instructed place, and choose your desired port; save and exit.

438 +

439 + ```sh

440 + [Socket]

441 + ListenStream=<DESIRED_PORT>

442 + ```

443 +

444 +

445 +4. Secure the endpoint from unauthorized access. That depends on your setup (e.g firewall setting, reverse proxies, etc)

446 +

447 +##### Centralization server with TLS and self-signed certificate

448 +

449 +Follow **all** the steps from

450 +the [Centralization server without TLS (use case; only in secure intranets](#centralization-server-without-tls--use-case-only-in-secure-intranets-)

451 +but omit step 2. Instead of step two, take the following steps:

452 +

453 +> 💡 You need to handcraft and use a self-signed certificate. A pretty straightforward way to do that is:

454 +

455 +1. Download OpenSSL

456 +

457 + ```sh

458 + # change this according to your distro

459 + sudo apt-get install openssl

460 + ```

461 +

462 +2. Create your own private certificate authority.

463 +

464 + ```sh

465 + mkdir self-signed-certificates && cd self-signed-certificates

466 +

467 + openssl req -newkey rsa:2048 -days 3650 -x509 -nodes \

468 + -out ca.pem -keyout ca.key -subj '/CN=My Certificate authority/'

469 +

470 + cat >ca.conf <<EOF

471 + [ ca ]

472 + default_ca = CA_default

473 + [ CA_default ]

474 + new_certs_dir = .

475 + certificate = ca.pem

476 + database = ./index

477 + private_key = ca.key

478 + serial = ./serial

479 + default_days = 3650

480 + default_md = default

481 + policy = policy_anything

482 + [ policy_anything ]

483 + countryName = optional

484 + stateOrProvinceName = optional

485 + localityName = optional

486 + organizationName = optional

487 + organizationalUnitName = optional

488 + commonName = supplied

489 + emailAddress = optional

490 + EOF

491 +

492 + touch index

493 + echo 0001 >serial

494 + ```

495 +

496 +3. Specify the Common Names for both the server and the clients (server who will push its journal logs).

497 +

498 + How each a client will reach the centralized the server? For instance if you want to reach them via public IP or DNS.

499 + There is a 1:1 correlation among the elements of `CLIENT_CNS`, `CLIENT_IPS` and `CLIENT_DNES`, if you want to omit

500 + a field, replace it with a placeholder string. You can also omit the DNSes but keep in mind to also omit the field

501 + `DNS:$SERVER_DNS` when you are signing the client certificates.

502 +

503 + ```sh

504 + SERVER_CN="myserver.example.com"

505 + SERVER_IP="Server_IP"

506 + SERVER_DNS="myserver.example.com"

507 +

508 + CLIENT_CNS=("client1.example.com" "client2.example.com")

509 + CLIENT_IPS=("client1_ip" "client2_ip")

510 + CLIENT_DNES=("client1.example.com" "client2.example.com")

511 + ```

512 +

513 +4. Create the self-signed certificates for the server and the clients

514 +

515 + ```sh

516 + openssl req -newkey rsa:2048 -nodes -out $SERVER_CN.csr -keyout $SERVER_CN.key -subj "/CN=$SERVER_CN/"

517 + echo "subjectAltName = IP:$SERVER_IP, DNS:$SERVER_DNS" > $SERVER_CN.ext

518 + openssl ca -batch -config ca.conf -notext -in $SERVER_CN.csr -out $SERVER_CN.pem -extfile $SERVER_CN.ext

519 +

520 + for i in "${!CLIENT_CNS[@]}"; do

521 + CLIENT_CN="${CLIENT_CNS[$i]}"

522 + CLIENT_IP="${CLIENT_IPS[$i]}"

523 + CLIENT_DNS="${CLIENT_DNES[$i]}"

524 + # Generate the client CSR

525 + openssl req -newkey rsa:2048 -nodes -out $CLIENT_CN.csr -keyout $CLIENT_CN.key -subj "/CN=$CLIENT_CN/"

526 + echo "subjectAltName = IP:$CLIENT_IP" > $CLIENT_CN.ext

527 +

528 + # Sign the client certificate using the CA configuration

529 + openssl ca -batch -config ca.conf -notext -in $CLIENT_CN.csr -out $CLIENT_CN.pem -extfile $CLIENT_CN.ext

530 + done

531 + ```

532 +

533 + Keep in mind we have already produced the client certificates, we will make use of them when we will configure the

534 + clients.

535 +

536 +5. Copy the key and the certificates into the `systemd-journal-remote`'s predefined places

537 +

538 + ```sh

539 + sudo mkdir /etc/ssl/private # make sure that you havent created this folder, and use it already, you may dont want to change it's permissions

540 + sudo chmod 755 /etc/ssl/private

541 + sudo mkdir /etc/ssl/ca/ # make sure that you havent created this folder, and use it already, you may dont want to change it's permissions

542 + sudo chmod 755 /etc/ssl/ca

543 + sudo cp "${SERVER_CN}".key /etc/ssl/private/journal-remote.key # This is not predefined but we need to clarify the key

544 + sudo cp "${SERVER_CN}".pem /etc/ssl/certs/journal-remote.pem

545 + sudo cp ca.pem /etc/ssl/ca/trusted.pem

546 + ```

547 +

548 +6. Adjust the permissions for the `systemd-journal-remote` to access them.

549 +

550 + ```sh

551 + sudo chgrp systemd-journal-remote /etc/ssl/private/journal-remote.key

552 + sudo chgrp systemd-journal-remote /etc/ssl/certs/journal-remote.pem

553 + sudo chgrp systemd-journal-remote /etc/ssl/ca/trusted.pem

554 +

555 +

556 + sudo chmod 0640 /etc/ssl/private/journal-remote.key

557 + sudo chmod 755 /etc/ssl/certs/journal-remote.pem

558 + sudo chmod 755 /etc/ssl/ca/trusted.pem

559 + ```

560 +

561 +7. Edit the `systemd-journal-remote.conf` to change the predefined key place and enable SSL.

562 +

563 + ```sh

564 + sudo nano /etc/systemd/journal-remote.conf

565 + ```

566 +

567 + You need to transform the corresponding section to something like this

568 + ```

569 + [Remote]

570 + Seal=false

571 + SplitMode=host

572 + ServerKeyFile=/etc/ssl/private/journal-remote.key

573 + ServerCertificateFile=/etc/ssl/certs/journal-remote.pem

574 + TrustedCertificateFile=/etc/ssl/ca/trusted.pem

575 + ```

576

368 -```sh

369 -# copy the service file

370 -sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/

577 +#### Configuring journal clients to push their logs to the server

578

372 -# edit it

373 -# --listen-http=-3 specifies the incoming journal for http.

374 -# If you want to use https, change it to --listen-https=-3.

375 -nano /etc/systemd/system/systemd-journal-remote.service

579 +In this section we will configure the clients/hosts to push their journal logs into the centralization server. You will

580 +install `systemd-journal-remote`,

581 +configure `systemd-journal-upload` (with or without SSL), enable and start it.

582

377 -# reload systemd

378 -sudo systemctl daemon-reload

379 -```

583 +1. To install `systemd-journal-remote`, run:

584

381 -To change the port, copy `/lib/systemd/system/systemd-journal-remote.socket` to `/etc/systemd/system/` and edit it.

382 -Then do `sudo systemctrl daemon-reload`

585 + ```sh

586 + # change this according to your distro

587 + sudo apt-get install systemd-journal-remote

588 + ```

589

590

385 -#### Configuring journal clients to push their logs to the server

591 +2. **With SSL**: Copy to the client/hosts the self-signed certificates you created before (you created one per host)

592

387 -On the clients you want to centralize their logs, install `systemd-journal-remote`, configure `systemd-journal-upload`, enable it and start it with `systemctl`.

593 + After this step, each host must have the following inside a directory (e.g. `home/user/incoming`),

594 + ```

595 + clientX.example.com.key

596 + clientX.example.com.pem

597 + ca.pem #common between the servers

598 + ```

599

389 -To install it run:

600 +3. **With SSL**: _On each client/host;_ create a user and a group (with the same name) called `systemd-journal-upload`

601

391 -```sh

392 -# change this according to your distro

393 -sudo apt-get install systemd-journal-remote

394 -```

602 + ```sh

603 + sudo adduser --system --home /run/systemd --no-create-home --disabled-login --group systemd-journal-upload

604 + ```

605

396 -Then, edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like this:

606 +4. **With SSL**: _On each client/host;_ Navigate under the directory you placed the certificates, copy them into the

607 + expected locations (by the `systemd-journal-upload` service)

608

398 -```

399 -[Upload]

400 -URL=http://centralization.server.ip:19532

401 -```

609 + ```sh

610 + sudo mkdir /etc/ssl/private # make sure that you havent created this folder, and use it already, you may dont want to change it's permissions.

611 + sudo chmod 755 /etc/ssl/private

612 + sudo mkdir /etc/ssl/ca/ # make sure that you havent created this folder, and use it already, you may dont want to change it's permissions.

613 + sudo chmod 755 /etc/ssl/ca

614 +

615 + cd home/user/incoming #change it accordingly, this is the place where you copied your certificates.

616 + sudo cp clientX.example.com.key /etc/ssl/private/journal-upload.key

617 + sudo cp clientX.example.com.pem /etc/ssl/certs/journal-upload.pem

618 + sudo cp ca.pem /etc/ssl/ca/trusted.pem

619 + ```

620

403 -Remember to match the protocol (http/https) the server expects.

621 +5. **With SSL**: _On each client/host;_ Adjust the permission so that the `systemd-journal-upload` service can access

622 + the files

623

405 -Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:

624 + ```sh

625 + sudo chgrp systemd-journal-upload /etc/ssl/private/journal-upload.key

626 + sudo chgrp systemd-journal-upload /etc/ssl/certs/journal-upload.pem

627 + sudo chgrp systemd-journal-upload /etc/ssl/ca/trusted.pem

628 +

629 + sudo chmod 0640 /etc/ssl/private/journal-upload.key

630 + sudo chmod 755 /etc/ssl/certs/journal-upload.pem

631 + sudo chmod 755 /etc/ssl/ca/trusted.pem

632 + ```

633

407 -```sh

408 -sudo systemctl edit systemd-journal-upload

409 -```

634 +6. Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:

635

411 -At the top, add:

636 + ```

637 + [Upload]

638 + URL=http://centralization.server.ip:19532

639 + ```

640

413 -```

414 -[Service]

415 -Restart=always

416 -```

641 + ⚠️ OR with SSL

642

418 -Then, enable and start `systemd-journal-upload`, like this:

643 + ```

644 + [Upload]

645 + URL=https://CENTRALIZED_SERVER_IP/DOMAIN :19532 #replace it accordingly

646 + ServerKeyFile=/etc/ssl/private/journal-upload.key

647 + ServerCertificateFile=/etc/ssl/certs/journal-upload.pem

648 + TrustedCertificateFile=/etc/ssl/ca/trusted.pem

649 + ```

650

420 -```sh

421 -sudo systemctl enable systemd-journal-upload

422 -sudo systemctl start systemd-journal-upload

423 -```

651 +7. Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even

652 + if the server is temporarily not there, like this:

653 +

654 + ```sh

655 + sudo systemctl edit systemd-journal-upload

656 + ```

657 +

658 + At the top, add:

659 +

660 + ```

661 + [Service]

662 + Restart=always

663 + ```

664 +

665 +8. Enable and start `systemd-journal-upload`, like this:

666 +

667 + ```sh

668 + sudo systemctl enable systemd-journal-upload

669 + sudo systemctl start systemd-journal-upload

670 + ```

671

425 -Keep in mind that immediately after starting `systemd-journal-upload` on a server, a replication process starts pushing logs in the order they have been received. This means that depending on the size of the available logs, some time may be needed for Netdata to show the most recent logs of that server.

672 +Keep in mind that immediately after starting `systemd-journal-upload` on a server, a replication process starts pushing

673 +logs in the order they have been received. This means that depending on the size of the available logs, some time may be

674 +needed for Netdata to show the most recent logs of that server.

675

676 #### Limitations when using a logs centralization server

677

678 As of this writing `namespaces` support by `systemd` is limited:

679

680 - Docker containers cannot log to namespaces. Check [this issue](https://github.com/moby/moby/issues/41879).

432 -- `systemd-journal-upload` automatically uploads `system` and `user` journals, but not `namespaces` journals. For this you need to spawn a `systemd-journal-upload` per namespace.

681 +- `systemd-journal-upload` automatically uploads `system` and `user` journals, but not `namespaces` journals. For this

682 + you need to spawn a `systemd-journal-upload` per namespace.

683

Rework guide, add SSL with self-signed certs (#16153)