@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 156 lines 5.58 KB
Rendered Raw
1 # Centralizing systemd-journal Namespace Logs to a Central systemd-journal-remote Server
2
3 This guide explains how to forward `systemd-journald` logs from journal _namespaces_ to a remote server using `systemd-journal-upload`, particularly for distributions running systemd versions **prior to 254**, which lack native namespace support.
4
5 ## Current Limitations
6
7 While `systemd-journal-upload` can forward logs with the `--merge` option, it doesn't natively consolidate logs across different journal namespaces in older systemd versions. Each namespace stores logs in a separate directory and requires independent uploading.
8
9 Starting with systemd 254, the new `--namespace=NAMESPACE` option simplifies this process. Until this version becomes widely available, we need to run separate `systemd-journal-upload` instances for each namespace using the `--directory` option.
10
11 ## Prerequisites
12
13 This guide assumes you have:
14
15 - Configured `systemd-journal-upload` on your local machine
16 - Set up and enabled `systemd-journal-remote` on your central server
17
18 The solution will send namespace logs to the same destination as your regular system logs, **multiplexing** them on your central log server.
19
20 ## Solution
21
22 We'll create a dedicated `systemd-journal-upload@<namespace>` unit for each namespace. A helper script automatically locates the appropriate journal directory and launches `systemd-journal-upload` with the correct parameters.
23
24 **Advantages**:
25
26 - Minimal per-namespace configuration
27 - Proper state management using systemd's `StateDirectory=`
28 - Dynamic journal path detection
29
30 ### 1. Systemd Unit Template
31
32 Create this file at:` /etc/systemd/system/systemd-journal-upload@.service`
33
34 <details open><summary>systemd-journal-upload@.service</summary>
35
36 ```ini
37 [Unit]
38 Description=Journal Remote Upload Service for %I Namespace
39 Documentation=man:systemd-journal-upload(8) file:/usr/local/bin/start-journal-upload-namespace.sh
40 After=systemd-journald.service network-online.target
41 Wants=network-online.target
42
43 [Service]
44 User=systemd-journal-upload
45 SupplementaryGroups=systemd-journal systemd-journal-remote
46 PrivateTmp=yes
47 ProtectSystem=strict
48 ProtectHome=yes
49 StateDirectory=systemd/journal-upload.%i
50 StateDirectoryMode=0700
51 ExecStart=/usr/local/bin/start-journal-upload-namespace.sh %I
52 Restart=always
53 RestartSec=5
54
55 [Install]
56 WantedBy=multi-user.target
57 ```
58
59 </details>
60
61 Note that `%i` represents the unescaped instance name (e.g., netdata), and `StateDirectory=` ensures proper state file management per namespace.
62
63 ### 2. Helper Script
64
65 Create this file at: `/usr/local/bin/start-journal-upload-namespace.sh`
66
67 <details open><summary>start-journal-upload-namespace.sh</summary>
68
69 ```bash
70 #!/bin/bash
71 set -euo pipefail
72
73 if [[ $# -lt 1 || -z "$1" || "$1" == "--debug" ]]; then
74 echo "Usage: $0 <namespace_name> [--debug]" >&2
75 exit ${SYSTEMD_EXIT_CODE_CONFIG:-78}
76 fi
77 NAMESPACE_NAME="$1"
78 shift
79
80 DEBUG_MODE=0
81 [[ "${1:-}" == "--debug" ]] && DEBUG_MODE=1
82
83 JOURNAL_BASE_DIR="/var/log/journal"
84 NAMESPACE_SUFFIX=".$NAMESPACE_NAME"
85 UPLOADER_CMD="/lib/systemd/systemd-journal-upload"
86
87 mapfile -t dirs < <(find "$JOURNAL_BASE_DIR" -maxdepth 1 -mindepth 1 -type d -name "*${NAMESPACE_SUFFIX}")
88
89 if [[ ${#dirs[@]} -eq 0 ]]; then
90 echo "ERROR: No *${NAMESPACE_SUFFIX} journal directory found." >&2
91 exit ${SYSTEMD_EXIT_CODE_CONFIG:-78}
92 elif [[ ${#dirs[@]} -gt 1 ]]; then
93 echo "ERROR: Multiple *${NAMESPACE_SUFFIX} journal directories found:" >&2
94 printf " %s\n" "${dirs[@]}" >&2
95 exit ${SYSTEMD_EXIT_CODE_CONFIG:-78}
96 fi
97
98 NAMESPACE_JOURNAL_DIR="${dirs[0]}"
99
100 if [[ -z "${STATE_DIRECTORY:-}" ]]; then
101 echo "ERROR: \$STATE_DIRECTORY not set. Ensure 'StateDirectory=' is used in the unit." >&2
102 [[ $DEBUG_MODE -eq 0 ]] && exit ${SYSTEMD_EXIT_CODE_CONFIG:-78} || STATE_FILE_PATH="\$STATE_DIRECTORY/state (variable not set)"
103 else
104 STATE_FILE_PATH="${STATE_DIRECTORY}/state"
105 fi
106
107 cmd_args=("--directory=$NAMESPACE_JOURNAL_DIR")
108 [[ -n "${STATE_DIRECTORY:-}" ]] && cmd_args+=("--save-state=$STATE_FILE_PATH")
109
110 if [[ $DEBUG_MODE -eq 1 ]]; then
111 printf -v cmd_string "%q " "$UPLOADER_CMD" "${cmd_args[@]}"
112 echo "[DEBUG] Would execute: $cmd_string"
113 exit 0
114 else
115 [[ -z "${STATE_DIRECTORY:-}" ]] && exit 1
116 exec "$UPLOADER_CMD" "${cmd_args[@]}"
117 fi
118 ```
119
120 </details>
121
122 Remember to make the script executable:
123
124 ```bash
125 sudo chmod +x /usr/local/bin/start-journal-upload-namespace.sh
126 ```
127
128 ### 3. Enabling and Starting the Service
129
130 To enable and start the upload service for a namespace called `netdata`:
131
132 ```bash
133 sudo systemctl enable --now systemd-journal-upload@netdata.service
134 ```
135
136 ### 4. Remote Server Configuration
137
138 Ensure your central server is running `systemd-journal-remote` and is properly configured to receive logs from upload clients. All uploaded logs, including those from namespaces, will appear in the same journal unless you implement additional filtering.
139
140 ## Future Improvements
141
142 With systemd 254 and newer, you can use the built-in `--namespace=` option:
143
144 ```bash
145 systemd-journal-upload --namespace=*
146 ```
147
148 This single command uploads logs from all namespaces, including the default one, interleaved together. This approach greatly simplifies deployment by eliminating the need for multiple `systemd-journal-upload` instances.
149
150 Until systemd 254+ becomes widely adopted, the per-namespace approach described in this guide remains the most reliable method.
151
152 ## Additional Resources
153
154 - [man systemd-journal-upload](https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-upload.service.html)
155 - [man systemd-journald](https://www.freedesktop.org/software/systemd/man/latest/systemd-journald.service.html)
156 - [man systemd.exec](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html)