| 1 | # Centralizing systemd-journal Namespace Logs to a Central systemd-journal-remote Server |
| 2 | |
| 3 | This guide explains how to forward `systemd-journald` logs from journal _namespaces_ to a remote server using `systemd-journal-upload`, particularly for distributions running systemd versions **prior to 254**, which lack native namespace support. |
| 4 | |
| 5 | ## Current Limitations |
| 6 | |
| 7 | While `systemd-journal-upload` can forward logs with the `--merge` option, it doesn't natively consolidate logs across different journal namespaces in older systemd versions. Each namespace stores logs in a separate directory and requires independent uploading. |
| 8 | |
| 9 | Starting with systemd 254, the new `--namespace=NAMESPACE` option simplifies this process. Until this version becomes widely available, we need to run separate `systemd-journal-upload` instances for each namespace using the `--directory` option. |
| 10 | |
| 11 | ## Prerequisites |
| 12 | |
| 13 | This guide assumes you have: |
| 14 | |
| 15 | - Configured `systemd-journal-upload` on your local machine |
| 16 | - Set up and enabled `systemd-journal-remote` on your central server |
| 17 | |
| 18 | The solution will send namespace logs to the same destination as your regular system logs, **multiplexing** them on your central log server. |
| 19 | |
| 20 | ## Solution |
| 21 | |
| 22 | We'll create a dedicated `systemd-journal-upload@<namespace>` unit for each namespace. A helper script automatically locates the appropriate journal directory and launches `systemd-journal-upload` with the correct parameters. |
| 23 | |
| 24 | **Advantages**: |
| 25 | |
| 26 | - Minimal per-namespace configuration |
| 27 | - Proper state management using systemd's `StateDirectory=` |
| 28 | - Dynamic journal path detection |
| 29 | |
| 30 | ### 1. Systemd Unit Template |
| 31 | |
| 32 | Create this file at:` /etc/systemd/system/systemd-journal-upload@.service` |
| 33 | |
| 34 | <details open><summary>systemd-journal-upload@.service</summary> |
| 35 | |
| 36 | ```ini |
| 37 | [Unit] |
| 38 | Description=Journal Remote Upload Service for %I Namespace |
| 39 | Documentation=man:systemd-journal-upload(8) file:/usr/local/bin/start-journal-upload-namespace.sh |
| 40 | After=systemd-journald.service network-online.target |
| 41 | Wants=network-online.target |
| 42 | |
| 43 | [Service] |
| 44 | User=systemd-journal-upload |
| 45 | SupplementaryGroups=systemd-journal systemd-journal-remote |
| 46 | PrivateTmp=yes |
| 47 | ProtectSystem=strict |
| 48 | ProtectHome=yes |
| 49 | StateDirectory=systemd/journal-upload.%i |
| 50 | StateDirectoryMode=0700 |
| 51 | ExecStart=/usr/local/bin/start-journal-upload-namespace.sh %I |
| 52 | Restart=always |
| 53 | RestartSec=5 |
| 54 | |
| 55 | [Install] |
| 56 | WantedBy=multi-user.target |
| 57 | ``` |
| 58 | |
| 59 | </details> |
| 60 | |
| 61 | Note that `%i` represents the unescaped instance name (e.g., netdata), and `StateDirectory=` ensures proper state file management per namespace. |
| 62 | |
| 63 | ### 2. Helper Script |
| 64 | |
| 65 | Create this file at: `/usr/local/bin/start-journal-upload-namespace.sh` |
| 66 | |
| 67 | <details open><summary>start-journal-upload-namespace.sh</summary> |
| 68 | |
| 69 | ```bash |
| 70 | #!/bin/bash |
| 71 | set -euo pipefail |
| 72 | |
| 73 | if [[ $# -lt 1 || -z "$1" || "$1" == "--debug" ]]; then |
| 74 | echo "Usage: $0 <namespace_name> [--debug]" >&2 |
| 75 | exit ${SYSTEMD_EXIT_CODE_CONFIG:-78} |
| 76 | fi |
| 77 | NAMESPACE_NAME="$1" |
| 78 | shift |
| 79 | |
| 80 | DEBUG_MODE=0 |
| 81 | [[ "${1:-}" == "--debug" ]] && DEBUG_MODE=1 |
| 82 | |
| 83 | JOURNAL_BASE_DIR="/var/log/journal" |
| 84 | NAMESPACE_SUFFIX=".$NAMESPACE_NAME" |
| 85 | UPLOADER_CMD="/lib/systemd/systemd-journal-upload" |
| 86 | |
| 87 | mapfile -t dirs < <(find "$JOURNAL_BASE_DIR" -maxdepth 1 -mindepth 1 -type d -name "*${NAMESPACE_SUFFIX}") |
| 88 | |
| 89 | if [[ ${#dirs[@]} -eq 0 ]]; then |
| 90 | echo "ERROR: No *${NAMESPACE_SUFFIX} journal directory found." >&2 |
| 91 | exit ${SYSTEMD_EXIT_CODE_CONFIG:-78} |
| 92 | elif [[ ${#dirs[@]} -gt 1 ]]; then |
| 93 | echo "ERROR: Multiple *${NAMESPACE_SUFFIX} journal directories found:" >&2 |
| 94 | printf " %s\n" "${dirs[@]}" >&2 |
| 95 | exit ${SYSTEMD_EXIT_CODE_CONFIG:-78} |
| 96 | fi |
| 97 | |
| 98 | NAMESPACE_JOURNAL_DIR="${dirs[0]}" |
| 99 | |
| 100 | if [[ -z "${STATE_DIRECTORY:-}" ]]; then |
| 101 | echo "ERROR: \$STATE_DIRECTORY not set. Ensure 'StateDirectory=' is used in the unit." >&2 |
| 102 | [[ $DEBUG_MODE -eq 0 ]] && exit ${SYSTEMD_EXIT_CODE_CONFIG:-78} || STATE_FILE_PATH="\$STATE_DIRECTORY/state (variable not set)" |
| 103 | else |
| 104 | STATE_FILE_PATH="${STATE_DIRECTORY}/state" |
| 105 | fi |
| 106 | |
| 107 | cmd_args=("--directory=$NAMESPACE_JOURNAL_DIR") |
| 108 | [[ -n "${STATE_DIRECTORY:-}" ]] && cmd_args+=("--save-state=$STATE_FILE_PATH") |
| 109 | |
| 110 | if [[ $DEBUG_MODE -eq 1 ]]; then |
| 111 | printf -v cmd_string "%q " "$UPLOADER_CMD" "${cmd_args[@]}" |
| 112 | echo "[DEBUG] Would execute: $cmd_string" |
| 113 | exit 0 |
| 114 | else |
| 115 | [[ -z "${STATE_DIRECTORY:-}" ]] && exit 1 |
| 116 | exec "$UPLOADER_CMD" "${cmd_args[@]}" |
| 117 | fi |
| 118 | ``` |
| 119 | |
| 120 | </details> |
| 121 | |
| 122 | Remember to make the script executable: |
| 123 | |
| 124 | ```bash |
| 125 | sudo chmod +x /usr/local/bin/start-journal-upload-namespace.sh |
| 126 | ``` |
| 127 | |
| 128 | ### 3. Enabling and Starting the Service |
| 129 | |
| 130 | To enable and start the upload service for a namespace called `netdata`: |
| 131 | |
| 132 | ```bash |
| 133 | sudo systemctl enable --now systemd-journal-upload@netdata.service |
| 134 | ``` |
| 135 | |
| 136 | ### 4. Remote Server Configuration |
| 137 | |
| 138 | Ensure your central server is running `systemd-journal-remote` and is properly configured to receive logs from upload clients. All uploaded logs, including those from namespaces, will appear in the same journal unless you implement additional filtering. |
| 139 | |
| 140 | ## Future Improvements |
| 141 | |
| 142 | With systemd 254 and newer, you can use the built-in `--namespace=` option: |
| 143 | |
| 144 | ```bash |
| 145 | systemd-journal-upload --namespace=* |
| 146 | ``` |
| 147 | |
| 148 | This single command uploads logs from all namespaces, including the default one, interleaved together. This approach greatly simplifies deployment by eliminating the need for multiple `systemd-journal-upload` instances. |
| 149 | |
| 150 | Until systemd 254+ becomes widely adopted, the per-namespace approach described in this guide remains the most reliable method. |
| 151 | |
| 152 | ## Additional Resources |
| 153 | |
| 154 | - [man systemd-journal-upload](https://www.freedesktop.org/software/systemd/man/latest/systemd-journal-upload.service.html) |
| 155 | - [man systemd-journald](https://www.freedesktop.org/software/systemd/man/latest/systemd-journald.service.html) |
| 156 | - [man systemd.exec](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html) |