documentation and helpers for centralizing namespaced logs (#20217)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Costa Tsaousis committed
May 2, 2025 at 12:26 UTC
ff33522006a0d3fea91016fdd41e70dbec79a5db
1 file changed
+156
docs/observability-centralization-points/logs-centralization-points-with-systemd-journald/centralizing-journal-namespaces.md
new
+156
@@ -0,0 +1,156 @@
1
+# Centralizing systemd-journal Namespace Logs to a Central systemd-journal-remote Server
2
+
3
+This guide explains how to forward `systemd-journald` logs from journal _namespaces_ to a remote server using `systemd-journal-upload`, particularly for distributions running systemd versions **prior to 254**, which lack native namespace support.
4
+
5
+## Current Limitations
6
+
7
+While `systemd-journal-upload` can forward logs with the `--merge` option, it doesn't natively consolidate logs across different journal namespaces in older systemd versions. Each namespace stores logs in a separate directory and requires independent uploading.
8
+
9
+Starting with systemd 254, the new `--namespace=NAMESPACE` option simplifies this process. Until this version becomes widely available, we need to run separate `systemd-journal-upload` instances for each namespace using the `--directory` option.
10
+
11
+## Prerequisites
12
+
13
+This guide assumes you have:
14
+
15
+- Configured `systemd-journal-upload` on your local machine
16
+- Set up and enabled `systemd-journal-remote` on your central server
17
+
18
+The solution will send namespace logs to the same destination as your regular system logs, **multiplexing** them on your central log server.
19
+
20
+## Solution
21
+
22
+We'll create a dedicated `systemd-journal-upload@<namespace>` unit for each namespace. A helper script automatically locates the appropriate journal directory and launches `systemd-journal-upload` with the correct parameters.
23
+
24
+**Advantages**:
25
+
26
+- Minimal per-namespace configuration
27
+- Proper state management using systemd's `StateDirectory=`
28
+- Dynamic journal path detection
29
+
30
+### 1. Systemd Unit Template
31
+
32
+Create this file at:` /etc/systemd/system/systemd-journal-upload@.service`
33
+
34
+<details open><summary>systemd-journal-upload@.service</summary>
35
+
36
+```ini
37
+[Unit]
38
+Description=Journal Remote Upload Service for %I Namespace
39
+Documentation=man:systemd-journal-upload(8) file:/usr/local/bin/start-journal-upload-namespace.sh
40
+After=systemd-journald.service network-online.target
41
+Wants=network-online.target
42
+
43
+[Service]
44
+User=systemd-journal-upload
45
+SupplementaryGroups=systemd-journal systemd-journal-remote
46
+PrivateTmp=yes
47
+ProtectSystem=strict
48
+ProtectHome=yes
49
+StateDirectory=systemd/journal-upload.%i
50
+StateDirectoryMode=0700
51
+ExecStart=/usr/local/bin/start-journal-upload-namespace.sh %I
52
+Restart=always
53
+RestartSec=5
54
+
55
+[Install]
56
+WantedBy=multi-user.target
57
+```
58
+
59
+</details>
60
+
61
+Note that `%i` represents the unescaped instance name (e.g., netdata), and `StateDirectory=` ensures proper state file management per namespace.
62
+
63
+### 2. Helper Script
64
+
65
+Create this file at: `/usr/local/bin/start-journal-upload-namespace.sh`
66
+
67
+<details open><summary>start-journal-upload-namespace.sh</summary>
68
+
69
+```bash
70
+#!/bin/bash
71
+set -euo pipefail
72
+
73
+if [[ $# -lt 1 || -z "$1" || "$1" == "--debug" ]]; then
74
+ echo "Usage: $0 <namespace_name> [--debug]" >&2
75
+ exit ${SYSTEMD_EXIT_CODE_CONFIG:-78}
76
+fi
77
+NAMESPACE_NAME="$1"
78
+shift
79
+
80
+DEBUG_MODE=0
81
+[[ "${1:-}" == "--debug" ]] && DEBUG_MODE=1
82
+
83
+JOURNAL_BASE_DIR="/var/log/journal"
84
+NAMESPACE_SUFFIX=".$NAMESPACE_NAME"
85
+UPLOADER_CMD="/lib/systemd/systemd-journal-upload"
86
+
87
+mapfile -t dirs < <(find "$JOURNAL_BASE_DIR" -maxdepth 1 -mindepth 1 -type d -name "*${NAMESPACE_SUFFIX}")
88
+
89
+if [[ ${#dirs[@]} -eq 0 ]]; then
90
+ echo "ERROR: No *${NAMESPACE_SUFFIX} journal directory found." >&2
91
+ exit ${SYSTEMD_EXIT_CODE_CONFIG:-78}
92
+elif [[ ${#dirs[@]} -gt 1 ]]; then
93
+ echo "ERROR: Multiple *${NAMESPACE_SUFFIX} journal directories found:" >&2
94
+ printf " %s\n" "${dirs[@]}" >&2
95
+ exit ${SYSTEMD_EXIT_CODE_CONFIG:-78}
96
+fi
97
+
98
+NAMESPACE_JOURNAL_DIR="${dirs[0]}"
99
+
100
+if [[ -z "${STATE_DIRECTORY:-}" ]]; then
101
+ echo "ERROR: \$STATE_DIRECTORY not set. Ensure 'StateDirectory=' is used in the unit." >&2
102
+ [[ $DEBUG_MODE -eq 0 ]] && exit ${SYSTEMD_EXIT_CODE_CONFIG:-78} || STATE_FILE_PATH="\$STATE_DIRECTORY/state (variable not set)"
103
+else
104
+ STATE_FILE_PATH="${STATE_DIRECTORY}/state"
105
+fi
106
+
107
+cmd_args=("--directory=$NAMESPACE_JOURNAL_DIR")
108
+[[ -n "${STATE_DIRECTORY:-}" ]] && cmd_args+=("--save-state=$STATE_FILE_PATH")
109
+
110
+if [[ $DEBUG_MODE -eq 1 ]]; then
111
+ printf -v cmd_string "%q " "$UPLOADER_CMD" "${cmd_args[@]}"
112
+ echo "[DEBUG] Would execute: $cmd_string"
113
+ exit 0
114
+else
115
+ [[ -z "${STATE_DIRECTORY:-}" ]] && exit 1
116
+ exec "$UPLOADER_CMD" "${cmd_args[@]}"
117
+fi
118
+```
119
+
120
+</details>
121
+
122
+Remember to make the script executable:
123
+
124
+```bash
125
+sudo chmod +x /usr/local/bin/start-journal-upload-namespace.sh
126
+```
127
+
128
+### 3. Enabling and Starting the Service
129
+
130
+To enable and start the upload service for a namespace called `netdata`:
131
+
132
+```bash
133
+sudo systemctl enable --now systemd-journal-upload@netdata.service
134
+```
135
+
136
+### 4. Remote Server Configuration
137
+
138
+Ensure your central server is running `systemd-journal-remote` and is properly configured to receive logs from upload clients. All uploaded logs, including those from namespaces, will appear in the same journal unless you implement additional filtering.
139
+
140
+## Future Improvements
141
+
142
+With systemd 254 and newer, you can use the built-in `--namespace=` option:
143
+
144
+```bash
145
+systemd-journal-upload --namespace=*
146
+```
147
+
148
+This single command uploads logs from all namespaces, including the default one, interleaved together. This approach greatly simplifies deployment by eliminating the need for multiple `systemd-journal-upload` instances.
149
+
150
+Until systemd 254+ becomes widely adopted, the per-namespace approach described in this guide remains the most reliable method.
151
+
152
+## Additional Resources
153
+
154
+- [man systemd-journal-upload](https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-upload.service.html)
155
+- [man systemd-journald](https://www.freedesktop.org/software/systemd/man/latest/systemd-journald.service.html)
156
+- [man systemd.exec](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html)