@cryptotaxi247 / netdata-1 / commits / 8faee578b

Changes to `systemd-journal` docs (#16225)

* changes to docs * tree to blob * small change

Fotis Voutsas committed Oct 17, 2023 at 17:08 UTC 8faee578b8d189cb11b399bc270882aa831aacc8

4 files changed +304 -294

collectors/systemd-journal.plugin/README.md

+12 -294

@@ -125,8 +125,8 @@ Usually `remote` journals are named by the IP of the server sending these logs.

125 extracts these IPs and performs a reverse DNS lookup to find their hostnames. When this is successful,

126 `remote` journals are named by the hostnames of the origin servers.

127

128 -For information about configuring a journals' centralization server,

129 -check [this FAQ item](#how-do-i-configure-a-journals-centralization-server).

128 +For information about configuring a journal centralization server,

129 +check [this FAQ item](#how-do-i-configure-a-journal-centralization-server).

130

131 ## Journal Fields

132

@@ -278,9 +278,9 @@ multiple journal files, over long time-frames.

278 During the development of this plugin, we submitted, to `systemd`, a number of patches to improve `journalctl`

279 performance by a factor of 14:

280

281 -- https://github.com/systemd/systemd/pull/29365

282 -- https://github.com/systemd/systemd/pull/29366

283 -- https://github.com/systemd/systemd/pull/29261

281 +- <https://github.com/systemd/systemd/pull/29365>

282 +- <https://github.com/systemd/systemd/pull/29366>

283 +- <https://github.com/systemd/systemd/pull/29261>

284

285 However, even after these patches are merged, `journalctl` will still be 2x slower than this Netdata plugin,

286 on multi-journal queries.

@@ -296,7 +296,7 @@ This Netdata plugin does not require any configuration or maintenance.

296

297 ## FAQ

298

299 -### Can I use this plugin on journals' centralization servers?

299 +### Can I use this plugin on journal centralization servers?

300

301 Yes. You can centralize your logs using `systemd-journal-remote`, and then install Netdata

302 on this logs centralization server to explore the logs of all your infrastructure.

@@ -304,7 +304,7 @@ on this logs centralization server to explore the logs of all your infrastructur

304 This plugin will automatically provide multi-node views of your logs and also give you the ability to combine the logs

305 of multiple servers, as you see fit.

306

307 -Check [configuring a logs centralization server](#configuring-a-journals-centralization-server).

307 +Check [configuring a logs centralization server](#how-do-i-configure-a-journal-centralization-server).

308

309 ### Can I use this plugin from a parent Netdata?

310

@@ -364,7 +364,7 @@ Yes. It is simple, fast and the software to do it is already in your systems.

364 For application and system logs, `systemd` journal is ideal and the visibility you can get

365 by centralizing your system logs and the use of this Netdata plugin, is unparalleled.

366

367 -### How do I configure a journals' centralization server?

367 +### How do I configure a journal centralization server?

368

369 A short summary to get journal server running can be found below.

370 There are two strategies you can apply, when it comes down to a centralized server for `systemd` journal logs.

@@ -374,294 +374,13 @@ There are two strategies you can apply, when it comes down to a centralized serv

374

375 For more options and reference to documentation, check `man systemd-journal-remote` and `man systemd-journal-upload`.

376

377 -#### _passive_ journals' centralization without encryption

377 +#### _passive_ journal centralization without encryption

378

379 -> ℹ️ _passive_ is a journal server that waits for clients to push their metrics to it.

379 +If you want to setup your own passive journal centralization setup without encryption, [check out guide on it](https://github.com/netdata/netdata/blob/master/collectors/systemd-journal.plugin/passive_journal_centralization_guide_no_encryption.md).

380

381 -> ⚠️ **IMPORTANT**

382 -> These instructions will copy your logs to a central server, without any encryption or authorization.

383 -> DO NOT USE THIS ON NON-TRUSTED NETWORKS.

384 -

385 -##### _passive_ server, without encryption

386 -

387 -On the centralization server install `systemd-journal-remote`:

388 -

389 -```sh

390 -# change this according to your distro

391 -sudo apt-get install systemd-journal-remote

392 -```

393 -

394 -Make sure the journal transfer protocol is `http`:

395 -

396 -```sh

397 -sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/

398 -

399 -# edit it to make sure it says:

400 -# --listen-http=-3

401 -# not:

402 -# --listen-https=-3

403 -sudo nano /etc/systemd/system/systemd-journal-remote.service

404 -

405 -# reload systemd

406 -sudo systemctl daemon-reload

407 -```

408 -

409 -Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`

410 -

411 -```sh

412 -# edit the socket file

413 -sudo systemctl edit systemd-journal-remote.socket

414 -```

415 -

416 -and add the following lines into the instructed place, and choose your desired port; save and exit.

417 -

418 -```sh

419 -[Socket]

420 -ListenStream=<DESIRED_PORT>

421 -```

422 -

423 -Finally, enable it, so that it will start automatically upon receiving a connection:

424 -

425 -```

426 -# enable systemd-journal-remote

427 -sudo systemctl enable --now systemd-journal-remote.socket

428 -sudo systemctl enable systemd-journal-remote.service

429 -```

430 -

431 -`systemd-journal-remote` is now listening for incoming journals from remote hosts.

432 -

433 -##### _passive_ client, without encryption

434 -

435 -On the clients, install `systemd-journal-remote`:

436 -

437 -```sh

438 -# change this according to your distro

439 -sudo apt-get install systemd-journal-remote

440 -```

441 -

442 -Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:

443 -

444 -```

445 -[Upload]

446 -URL=http://centralization.server.ip:19532

447 -```

448 -

449 -Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:

450 -

451 -```sh

452 -sudo systemctl edit systemd-journal-upload

453 -```

454 -

455 -At the top, add:

456 -

457 -```

458 -[Service]

459 -Restart=always

460 -```

461 -

462 -Enable and start `systemd-journal-upload`, like this:

463 -

464 -```sh

465 -sudo systemctl enable systemd-journal-upload

466 -sudo systemctl start systemd-journal-upload

467 -```

468 -

469 -##### verify it works

470 -

471 -To verify the central server is receiving logs, run this on the central server:

472 -

473 -```sh

474 -sudo ls -l /var/log/journal/remote/

475 -```

476 -

477 -You should see new files from the client's IP.

478 -

479 -Also, `systemctl status systemd-journal-remote` should show something like this:

480 -

481 -```

482 -systemd-journal-remote.service - Journal Remote Sink Service

483 - Loaded: loaded (/etc/systemd/system/systemd-journal-remote.service; indirect; preset: disabled)

484 - Active: active (running) since Sun 2023-10-15 14:29:46 EEST; 2h 24min ago

485 -TriggeredBy: ● systemd-journal-remote.socket

486 - Docs: man:systemd-journal-remote(8)

487 - man:journal-remote.conf(5)

488 - Main PID: 2118153 (systemd-journal)

489 - Status: "Processing requests..."

490 - Tasks: 1 (limit: 154152)

491 - Memory: 2.2M

492 - CPU: 71ms

493 - CGroup: /system.slice/systemd-journal-remote.service

494 - └─2118153 /usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/

495 -```

496 -

497 -Note the `status: "Processing requests..."` and the PID under `CGroup`.

498 -

499 -On the client `systemctl status systemd-journal-upload` should show something like this:

500 -

501 -```

502 -● systemd-journal-upload.service - Journal Remote Upload Service

503 - Loaded: loaded (/lib/systemd/system/systemd-journal-upload.service; enabled; vendor preset: disabled)

504 - Drop-In: /etc/systemd/system/systemd-journal-upload.service.d

505 - └─override.conf

506 - Active: active (running) since Sun 2023-10-15 10:39:04 UTC; 3h 17min ago

507 - Docs: man:systemd-journal-upload(8)

508 - Main PID: 4169 (systemd-journal)

509 - Status: "Processing input..."

510 - Tasks: 1 (limit: 13868)

511 - Memory: 3.5M

512 - CPU: 1.081s

513 - CGroup: /system.slice/systemd-journal-upload.service

514 - └─4169 /lib/systemd/systemd-journal-upload --save-state

515 -```

516 -

517 -Note the `Status: "Processing input..."` and the PID under `CGroup`.

518 -

519 -#### _passive_ journals' centralization with encryption using self-signed certificates

520 -

521 -> ℹ️ _passive_ is a journal server that waits for clients to push their metrics to it.

522 -

523 -##### _passive_ server, with encryption and self-singed certificates

524 -

525 -On the centralization server install `systemd-journal-remote` and `openssl`:

526 -

527 -```sh

528 -# change this according to your distro

529 -sudo apt-get install systemd-journal-remote openssl

530 -```

531 -

532 -Make sure the journal transfer protocol is `https`:

533 -

534 -```sh

535 -sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/

536 -

537 -# edit it to make sure it says:

538 -# --listen-https=-3

539 -# not:

540 -# --listen-http=-3

541 -sudo nano /etc/systemd/system/systemd-journal-remote.service

542 -

543 -# reload systemd

544 -sudo systemctl daemon-reload

545 -```

546 -

547 -Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`

548 -

549 -```sh

550 -# edit the socket file

551 -sudo systemctl edit systemd-journal-remote.socket

552 -```

553 -

554 -and add the following lines into the instructed place, and choose your desired port; save and exit.

555 -

556 -```sh

557 -[Socket]

558 -ListenStream=<DESIRED_PORT>

559 -```

560 -

561 -Finally, enable it, so that it will start automatically upon receiving a connection:

562 -

563 -```sh

564 -# enable systemd-journal-remote

565 -sudo systemctl enable --now systemd-journal-remote.socket

566 -sudo systemctl enable systemd-journal-remote.service

567 -```

568 -

569 -`systemd-journal-remote` is now listening for incoming journals from remote hosts.

570 -

571 -Use [this script](https://gist.github.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7) to create a self-signed certificates authority and certificates for all your servers.

572 -

573 -```sh

574 -wget -O systemd-journal-self-signed-certs.sh "https://gist.githubusercontent.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7/raw/c346e61e0a66f45dc4095d254bd23917f0a01bd0/systemd-journal-self-signed-certs.sh"

575 -chmod 755 systemd-journal-self-signed-certs.sh

576 -```

577 -

578 -Edit the script and at its top, set your settings:

579 -

580 -```sh

581 -# The directory to save the generated certificates (and everything about this certificate authority).

582 -# This is only used on the node generating the certificates (usually on the journals server).

583 -DIR="/etc/ssl/systemd-journal-remote"

584 -

585 -# The journals centralization server name (the CN of the server certificate).

586 -SERVER="server-hostname"

587 -

588 -# All the DNS names or IPs this server is reachable at (the certificate will include them).

589 -# Journal clients can use any of them to connect to this server.

590 -# systemd-journal-upload validates its URL= hostname, against this list.

591 -SERVER_ALIASES=("DNS:server-hostname1" "DNS:server-hostname2" "IP:1.2.3.4" "IP:10.1.1.1" "IP:172.16.1.1")

592 -

593 -# All the names of the journal clients that will be sending logs to the server (the CNs of their certificates).

594 -# These names are used by systemd-journal-remote to name the journal files in /var/log/journal/remote/.

595 -# Also the remote hosts will be presented using these names on Netdata dashboards.

596 -CLIENTS=("vm1" "vm2" "vm3" "add_as_may_as_needed")

597 -```

598 -

599 -Then run the script:

600 -

601 -```sh

602 -sudo ./systemd-journal-self-signed-certs.sh

603 -```

604 -

605 -The script will create the directory `/etc/ssl/systemd-journal-remote` and in it you will find all the certificates needed.

606 -

607 -There will also be files named `runme-on-XXX.sh`. There will be 1 script for the server and 1 script for each of the clients. You can copy and paste (or `scp`) these scripts on your server and each of your clients and run them as root:

608 -

609 -```sh

610 -scp /etc/ssl/systemd-journal-remote/runme-on-XXX.sh XXX:/tmp/

611 -```

612 -

613 -Once the above is done, `ssh` to each server/client and do:

614 -

615 -```sh

616 -sudo bash /tmp/runme-on-XXX.sh

617 -```

618 -

619 -The scripts install the needed certificates, fix their file permissions to be accessible by systemd-journal-remote/upload, change `/etc/systemd/journal-remote.conf` (on the server) or `/etc/systemd/journal-upload.conf` on the clients and restart the relevant services.

620 -

621 -

622 -##### _passive_ client, with encryption and self-singed certificates

623 -

624 -On the clients, install `systemd-journal-remote`:

625 -

626 -```sh

627 -# change this according to your distro

628 -sudo apt-get install systemd-journal-remote

629 -```

630 -

631 -Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:

632 -

633 -```

634 -[Upload]

635 -URL=https://centralization.server.ip:19532

636 -```

637 -

638 -Make sure that `centralization.server.ip` is one of the `SERVER_ALIASES` when you created the certificates.

639 -

640 -Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:

641 -

642 -```sh

643 -sudo systemctl edit systemd-journal-upload

644 -```

645 -

646 -At the top, add:

647 -

648 -```

649 -[Service]

650 -Restart=always

651 -```

652 -

653 -Enable and start `systemd-journal-upload`, like this:

654 -

655 -```sh

656 -sudo systemctl enable systemd-journal-upload

657 -```

658 -

659 -Copy the relevant `runme-on-XXX.sh` script as described on server setup and run it:

660 -

661 -```sh

662 -sudo bash /tmp/runme-on-XXX.sh

663 -```

381 +#### _passive_ journal centralization with encryption using self-signed certificates

382

383 +If you want to setup your own passive journal centralization setup using self-signed certificates for encryption, [check out guide on it](https://github.com/netdata/netdata/blob/master/collectors/systemd-journal.plugin/passive_journal_centralization_guide_self_signed_certs.md).

384

385 #### Limitations when using a logs centralization server

386

@@ -670,4 +389,3 @@ As of this writing `namespaces` support by `systemd` is limited:

389 - Docker containers cannot log to namespaces. Check [this issue](https://github.com/moby/moby/issues/41879).

390 - `systemd-journal-upload` automatically uploads `system` and `user` journals, but not `namespaces` journals. For this

391 you need to spawn a `systemd-journal-upload` per namespace.

673 -

collectors/systemd-journal.plugin/passive_journal_centralization_guide_no_encryption.md new

+143

@@ -0,0 +1,143 @@

1 +# Passive journal centralization without encryption

2 +

3 +This page will guide you through creating a passive journal centralization setup without the use of encryption.

4 +

5 +> A _passive_ journal server waits for clients to push their metrics to it.

6 +

7 +> ⚠️ **IMPORTANT**

8 +> These instructions will copy your logs to a central server, without any encryption or authorization.

9 +> DO NOT USE THIS ON NON-TRUSTED NETWORKS.

10 +

11 +## Server configuration

12 +

13 +On the centralization server install `systemd-journal-remote`:

14 +

15 +```sh

16 +# change this according to your distro

17 +sudo apt-get install systemd-journal-remote

18 +```

19 +

20 +Make sure the journal transfer protocol is `http`:

21 +

22 +```sh

23 +sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/

24 +

25 +# edit it to make sure it says:

26 +# --listen-http=-3

27 +# not:

28 +# --listen-https=-3

29 +sudo nano /etc/systemd/system/systemd-journal-remote.service

30 +

31 +# reload systemd

32 +sudo systemctl daemon-reload

33 +```

34 +

35 +Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`

36 +

37 +```sh

38 +# edit the socket file

39 +sudo systemctl edit systemd-journal-remote.socket

40 +```

41 +

42 +and add the following lines into the instructed place, and choose your desired port; save and exit.

43 +

44 +```sh

45 +[Socket]

46 +ListenStream=<DESIRED_PORT>

47 +```

48 +

49 +Finally, enable it, so that it will start automatically upon receiving a connection:

50 +

51 +```bash

52 +# enable systemd-journal-remote

53 +sudo systemctl enable --now systemd-journal-remote.socket

54 +sudo systemctl enable systemd-journal-remote.service

55 +```

56 +

57 +`systemd-journal-remote` is now listening for incoming journals from remote hosts.

58 +

59 +## Client configuration

60 +

61 +On the clients, install `systemd-journal-remote`:

62 +

63 +```sh

64 +# change this according to your distro

65 +sudo apt-get install systemd-journal-remote

66 +```

67 +

68 +Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:

69 +

70 +```conf

71 +[Upload]

72 +URL=http://centralization.server.ip:19532

73 +```

74 +

75 +Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:

76 +

77 +```sh

78 +sudo systemctl edit systemd-journal-upload

79 +```

80 +

81 +At the top, add:

82 +

83 +```conf

84 +[Service]

85 +Restart=always

86 +```

87 +

88 +Enable and start `systemd-journal-upload`, like this:

89 +

90 +```sh

91 +sudo systemctl enable systemd-journal-upload

92 +sudo systemctl start systemd-journal-upload

93 +```

94 +

95 +## Verify it works

96 +

97 +To verify the central server is receiving logs, run this on the central server:

98 +

99 +```sh

100 +sudo ls -l /var/log/journal/remote/

101 +```

102 +

103 +You should see new files from the client's IP.

104 +

105 +Also, `systemctl status systemd-journal-remote` should show something like this:

106 +

107 +```bash

108 +systemd-journal-remote.service - Journal Remote Sink Service

109 + Loaded: loaded (/etc/systemd/system/systemd-journal-remote.service; indirect; preset: disabled)

110 + Active: active (running) since Sun 2023-10-15 14:29:46 EEST; 2h 24min ago

111 +TriggeredBy: ● systemd-journal-remote.socket

112 + Docs: man:systemd-journal-remote(8)

113 + man:journal-remote.conf(5)

114 + Main PID: 2118153 (systemd-journal)

115 + Status: "Processing requests..."

116 + Tasks: 1 (limit: 154152)

117 + Memory: 2.2M

118 + CPU: 71ms

119 + CGroup: /system.slice/systemd-journal-remote.service

120 + └─2118153 /usr/lib/systemd/systemd-journal-remote --listen-http=-3 --output=/var/log/journal/remote/

121 +```

122 +

123 +Note the `status: "Processing requests..."` and the PID under `CGroup`.

124 +

125 +On the client `systemctl status systemd-journal-upload` should show something like this:

126 +

127 +```bash

128 +● systemd-journal-upload.service - Journal Remote Upload Service

129 + Loaded: loaded (/lib/systemd/system/systemd-journal-upload.service; enabled; vendor preset: disabled)

130 + Drop-In: /etc/systemd/system/systemd-journal-upload.service.d

131 + └─override.conf

132 + Active: active (running) since Sun 2023-10-15 10:39:04 UTC; 3h 17min ago

133 + Docs: man:systemd-journal-upload(8)

134 + Main PID: 4169 (systemd-journal)

135 + Status: "Processing input..."

136 + Tasks: 1 (limit: 13868)

137 + Memory: 3.5M

138 + CPU: 1.081s

139 + CGroup: /system.slice/systemd-journal-upload.service

140 + └─4169 /lib/systemd/systemd-journal-upload --save-state

141 +```

142 +

143 +Note the `Status: "Processing input..."` and the PID under `CGroup`.

collectors/systemd-journal.plugin/passive_journal_centralization_guide_self_signed_certs.md new

+146

@@ -0,0 +1,146 @@

1 +# Passive journal centralization with encryption using self-signed certificates

2 +

3 +This page will guide you through creating a passive journal centralization setup using self-signed certificates for encryption.

4 +

5 +> A _passive_ journal server waits for clients to push their metrics to it.

6 +

7 +## Server configuration

8 +

9 +On the centralization server install `systemd-journal-remote` and `openssl`:

10 +

11 +```sh

12 +# change this according to your distro

13 +sudo apt-get install systemd-journal-remote openssl

14 +```

15 +

16 +Make sure the journal transfer protocol is `https`:

17 +

18 +```sh

19 +sudo cp /lib/systemd/system/systemd-journal-remote.service /etc/systemd/system/

20 +

21 +# edit it to make sure it says:

22 +# --listen-https=-3

23 +# not:

24 +# --listen-http=-3

25 +sudo nano /etc/systemd/system/systemd-journal-remote.service

26 +

27 +# reload systemd

28 +sudo systemctl daemon-reload

29 +```

30 +

31 +Optionally, if you want to change the port (the default is `19532`), edit `systemd-journal-remote.socket`

32 +

33 +```sh

34 +# edit the socket file

35 +sudo systemctl edit systemd-journal-remote.socket

36 +```

37 +

38 +and add the following lines into the instructed place, and choose your desired port; save and exit.

39 +

40 +```sh

41 +[Socket]

42 +ListenStream=<DESIRED_PORT>

43 +```

44 +

45 +Finally, enable it, so that it will start automatically upon receiving a connection:

46 +

47 +```sh

48 +# enable systemd-journal-remote

49 +sudo systemctl enable --now systemd-journal-remote.socket

50 +sudo systemctl enable systemd-journal-remote.service

51 +```

52 +

53 +`systemd-journal-remote` is now listening for incoming journals from remote hosts.

54 +

55 +Use [this script](https://gist.github.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7) to create a self-signed certificates authority and certificates for all your servers.

56 +

57 +```sh

58 +wget -O systemd-journal-self-signed-certs.sh "https://gist.githubusercontent.com/ktsaou/d62b8a6501cf9a0da94f03cbbb71c5c7/raw/c346e61e0a66f45dc4095d254bd23917f0a01bd0/systemd-journal-self-signed-certs.sh"

59 +chmod 755 systemd-journal-self-signed-certs.sh

60 +```

61 +

62 +Edit the script and at its top, set your settings:

63 +

64 +```sh

65 +# The directory to save the generated certificates (and everything about this certificate authority).

66 +# This is only used on the node generating the certificates (usually on the journals server).

67 +DIR="/etc/ssl/systemd-journal-remote"

68 +

69 +# The journals centralization server name (the CN of the server certificate).

70 +SERVER="server-hostname"

71 +

72 +# All the DNS names or IPs this server is reachable at (the certificate will include them).

73 +# Journal clients can use any of them to connect to this server.

74 +# systemd-journal-upload validates its URL= hostname, against this list.

75 +SERVER_ALIASES=("DNS:server-hostname1" "DNS:server-hostname2" "IP:1.2.3.4" "IP:10.1.1.1" "IP:172.16.1.1")

76 +

77 +# All the names of the journal clients that will be sending logs to the server (the CNs of their certificates).

78 +# These names are used by systemd-journal-remote to name the journal files in /var/log/journal/remote/.

79 +# Also the remote hosts will be presented using these names on Netdata dashboards.

80 +CLIENTS=("vm1" "vm2" "vm3" "add_as_may_as_needed")

81 +```

82 +

83 +Then run the script:

84 +

85 +```sh

86 +sudo ./systemd-journal-self-signed-certs.sh

87 +```

88 +

89 +The script will create the directory `/etc/ssl/systemd-journal-remote` and in it you will find all the certificates needed.

90 +

91 +There will also be files named `runme-on-XXX.sh`. There will be 1 script for the server and 1 script for each of the clients. You can copy and paste (or `scp`) these scripts on your server and each of your clients and run them as root:

92 +

93 +```sh

94 +scp /etc/ssl/systemd-journal-remote/runme-on-XXX.sh XXX:/tmp/

95 +```

96 +

97 +Once the above is done, `ssh` to each server/client and do:

98 +

99 +```sh

100 +sudo bash /tmp/runme-on-XXX.sh

101 +```

102 +

103 +The scripts install the needed certificates, fix their file permissions to be accessible by systemd-journal-remote/upload, change `/etc/systemd/journal-remote.conf` (on the server) or `/etc/systemd/journal-upload.conf` on the clients and restart the relevant services.

104 +

105 +## Client configuration

106 +

107 +On the clients, install `systemd-journal-remote`:

108 +

109 +```sh

110 +# change this according to your distro

111 +sudo apt-get install systemd-journal-remote

112 +```

113 +

114 +Edit `/etc/systemd/journal-upload.conf` and set the IP address and the port of the server, like so:

115 +

116 +```conf

117 +[Upload]

118 +URL=https://centralization.server.ip:19532

119 +```

120 +

121 +Make sure that `centralization.server.ip` is one of the `SERVER_ALIASES` when you created the certificates.

122 +

123 +Edit `systemd-journal-upload`, and add `Restart=always` to make sure the client will keep trying to push logs, even if the server is temporarily not there, like this:

124 +

125 +```sh

126 +sudo systemctl edit systemd-journal-upload

127 +```

128 +

129 +At the top, add:

130 +

131 +```conf

132 +[Service]

133 +Restart=always

134 +```

135 +

136 +Enable and start `systemd-journal-upload`, like this:

137 +

138 +```sh

139 +sudo systemctl enable systemd-journal-upload

140 +```

141 +

142 +Copy the relevant `runme-on-XXX.sh` script as described on server setup and run it:

143 +

144 +```sh

145 +sudo bash /tmp/runme-on-XXX.sh

146 +```

docs/category-overview-pages/logs.md new

@@ -0,0 +1,3 @@

1 +# Logs

2 +

3 +This section talks about ways Netdata collects and visualizes logs, while also providing useful guides on log centralization setups that can be used with Netdata.