Add an alert guide for reboot required (#19260)
Ralph Meijer committed
Dec 20, 2024 at 20:52 UTC
387a83d2d5c63d2c6f6b89a7f20cba39b91a9955
1 file changed
+119
src/health/guides/reboot_required/system_post_update_reboot_status.md
new
+119
@@ -0,0 +1,119 @@
1
+### Understand the alert
2
+
3
+*Added in version 2.1.0*
4
+
5
+This alert is triggered on Debian-based systems when a reboot is required to complete the update of (system) packages. The corresponding metric is Post-Update Reboot Status (`system.post_update_reboot_status`).
6
+
7
+### What does “reboot required” mean?
8
+
9
+When installing updated versions of packages, sometimes a reboot of the system is required for the update to take full effect. On Debian-based systems (including Ubuntu), this is marked by the existance of `/var/run/reboot-required`. If the file exists, it will contain the following text that is also shown shown when logging into the machine via the console or ssh:
10
+
11
+
12
+```*** System restart required ***```
13
+
14
+The reboot-required flag can be set in the following scenarios:
15
+
16
+- New kernel versions.
17
+- Updates to important libraries like glibc, OpenSSL.
18
+- Updates to long-running processes or their dependencies.
19
+- Security updates in general.
20
+
21
+> ⚠️ **Warning**
22
+>
23
+> If systems are never rebooted as standard practice, reboots become _untested_. Should an event occur that causes the system to restart unexpectly, it is then unknown if they will come back up properly!
24
+>
25
+> It is therefore *recommended* to plan for reboots, regularly and/or when the reboot-required flag is set and this alert is triggered.
26
+
27
+### Troubleshoot the alert
28
+
29
+The most appropriate action is to reboot the (virtual) machine at the earliest convenience. This ensures the operating system and all services run on the latest installed version of their components and dependencies.
30
+
31
+In some cases it may be sufficient to restart some services. This does not clear the flag, but may help to plan the appropriate time to schedule the reboot. The packages involved with setting the flag are listed in `/var/run/reboot-required.pkgs`. The flag and this list is managed by the `update-notifier-common` package.
32
+
33
+There are several tools to interpret the list of packages to futher understand which processes and services are involved. These all have a different perspective on what to do and may give an incomplete picture.
34
+
35
+#### checkrestart
36
+
37
+Part of the `debian-goodies` package. This tool scans the running processes and shows a report of its findings.
38
+
39
+#### needrestart
40
+
41
+In the `needrestart` package. This tool scans the running processes, kernel and microcode, and offers a UI for restarting services when appropriate. It also hooks into the packaging system to restart services automatically.
42
+
43
+You can run `needrestart` non-interactively, without performing any actions, with:
44
+
45
+
46
+```$ sudo DEBIAN_FRONTEND=noninteractive needrestart -n 2>&1 | cat```
47
+<details><summary>example output</summary>
48
+
49
+```
50
+
51
+Pending kernel upgrade!
52
+
53
+Running kernel version:
54
+ 5.15.0-67-generic
55
+
56
+Diagnostics:
57
+ The currently running kernel version is not the expected kernel version 5.15.0-125-generic.
58
+
59
+Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return]
60
+
61
+Failed to check for processor microcode upgrades.
62
+
63
+Services to be restarted:
64
+
65
+Service restarts being deferred:
66
+ systemctl restart ModemManager.service
67
+ systemctl restart NetworkManager.service
68
+ systemctl restart accounts-daemon.service
69
+ systemctl restart acpid.service
70
+ systemctl restart apache2.service
71
+ systemctl restart atd.service
72
+ systemctl restart avahi-daemon.service
73
+ systemctl restart colord.service
74
+ systemctl restart cron.service
75
+ /etc/needrestart/restart.d/dbus.service
76
+ systemctl restart gdm.service
77
+ systemctl restart gdm3.service
78
+ systemctl restart irqbalance.service
79
+ systemctl restart kerneloops.service
80
+ systemctl restart memcached.service
81
+ systemctl restart mongodb.service
82
+ systemctl restart networkd-dispatcher.service
83
+ systemctl restart polkit.service
84
+ systemctl restart postfix@-.service
85
+ systemctl restart prometheus-process-exporter.service
86
+ systemctl restart rsyslog.service
87
+ systemctl restart rtkit-daemon.service
88
+ systemctl restart smartmontools.service
89
+ systemctl restart ssh.service
90
+ systemctl restart switcheroo-control.service
91
+ systemctl restart systemd-journald.service
92
+ systemctl restart systemd-journald@netdata.service
93
+ systemctl restart systemd-logind.service
94
+ systemctl restart systemd-resolved.service
95
+ systemctl restart systemd-timesyncd.service
96
+ systemctl restart systemd-udevd.service
97
+ systemctl restart udisks2.service
98
+ systemctl restart unattended-upgrades.service
99
+ systemctl restart unifi.service
100
+ systemctl restart upower.service
101
+ systemctl restart uuidd.service
102
+ systemctl restart whoopsie.service
103
+ systemctl restart zfs-zed.service
104
+ systemctl restart zsysd.service
105
+
106
+No containers need to be restarted.
107
+
108
+User sessions running outdated binaries:
109
+ gdm @ user manager service: systemd[10233]
110
+ ralphm @ session #264890: ssh-agent[2400801]
111
+ ralphm @ session #269836: ssh-agent[1965630,2158281]
112
+ ralphm @ user manager service: systemd[2243900]
113
+```
114
+</details>
115
+
116
+
117
+### Useful resources
118
+
119
+1. [How to find out if my Ubuntu/Debian Linux server needs a reboot](https://www.cyberciti.biz/faq/how-to-find-out-if-my-ubuntudebian-linux-server-needs-a-reboot/)