@cryptotaxi247 / netdata-1 / commits / 3161e32dc

Add alarms for abnormally high load averages. (#4175)

* Add alarms for abnormally high load averages. This adds reasonably conservative alarms to send alarts on abnormally high load averages. Such a situation may be indicative of a DoS attack, runaway processes, or simply use of underpowered hardware. This intentionally does not compute averages, as doing so would be redundant (we are dealing with load _averages_ after all), which makes the lookup lines look a bit odd in comparison to most other alarms. The actual alarm calculation is as-follows: * Compute the baseline trigger threshold. This is either 2 or the maximum number of CPU's that were present in the system over the last minute, whichever is higher. This special-cases single-CPU systems to be a bit less aggressive,a s they are more often over-committed than systems with multiple cores. * For the 15 minute load average, if the maximum value over the last minute is greater than twice the trigger threshold, issue a warning. * For the 5 minute load average, if the maximum value over the last minute is greater than four times the ttrigger value, issue a warning. * For the 1 minute load average, if the maximum value over the last minute is greater than eight times the trigger value, issue a warning. * For all the load averages, if the value is greater than twice the warning requirement, issue a critical alert. * Down-hysteriesis is provided so that each alarm only resets wheen the value goes below 7/8 of the value for that alarm status. * Each alarm is evaluated once per minute. This behavior should be suitable for most server type systems and many workstations, but may be a bit overaggressive for certain types of system (build systems for example). * Fixed calculations of the base trigger value. Credit goes to @ktsaou for pointing out how the original implementation was incorrect. * Update alarms with correct OS information.

Austin S. Hemmelgarn committed Sep 12, 2018 at 15:20 UTC 3161e32dc24d42e8da38f5b59d3d4481e5e9565d
2 files changed +57
conf.d/Makefile.am
+1
@@ -112,6 +112,7 @@ dist_healthconfig_DATA = \
112 health.d/ipmi.conf \
113 health.d/isc_dhcpd.conf \
114 health.d/lighttpd.conf \
115 + health.d/load.conf \
116 health.d/mdstat.conf \
117 health.d/megacli.conf \
118 health.d/memcached.conf \
conf.d/health.d/load.conf new
+56
@@ -0,0 +1,56 @@
1 +
2 +# you can disable an alarm notification by setting the 'to' line to: silent
3 +
4 +# Calculate the base trigger point for the load average alarms.
5 +# This is the maximum number of CPU's in the system over the past 1
6 +# minute, with a special case for a single CPU of setting the trigger at 2.
7 +template: load_trigger
8 + on: system.cpu
9 + os: linux
10 + hosts: *
11 + calc: ($processors <= 2) ? ( 2 ) : ( $processors )
12 + units: cpus
13 + every: 1m
14 + info: trigger point for load average alarms
15 +
16 +# Send alarms if the load average is unusually high.
17 +# These intentionally _do not_ calculate the average over the sampled
18 +# time period because the values being checked already are averages.
19 +template: load_average_15
20 + on: system.load
21 + os: linux
22 + hosts: *
23 + lookup: max -1m unaligned of load15
24 + units: load
25 + every: 1m
26 + warn: $this > (($status >= $WARNING) ? (1.75 * $load_trigger) : (2 * $load_trigger))
27 + crit: $this > (($status == $CRITICAL) ? (3.5 * $load_trigger) : (4 * $load_trigger))
28 + delay: down 15m multiplier 1.5 max 1h
29 + info: fifteen-minute load average
30 + to: sysadmin
31 +
32 +template: load_average_5
33 + on: system.load
34 + os: linux
35 + hosts: *
36 + lookup: max -1m unaligned of load5
37 + units: load
38 + every: 1m
39 + warn: $this > (($status >= $WARNING) ? (3.5 * $load_trigger) : (4 * $load_trigger))
40 + crit: $this > (($status == $CRITICAL) ? (7 * $load_trigger) : (8 * $load_trigger))
41 + delay: down 15m multiplier 1.5 max 1h
42 + info: five-minute load average
43 + to: sysadmin
44 +
45 +template: load_average_1
46 + on: system.load
47 + os: linux
48 + hosts: *
49 + lookup: max -1m unaligned of load1
50 + units: load
51 + every: 1m
52 + warn: $this > (($status >= $WARNING) ? (7 * $load_trigger) : (8 * $load_trigger))
53 + crit: $this > (($status == $CRITICAL) ? (14 * $load_trigger) : (16 * $load_trigger))
54 + delay: down 15m multiplier 1.5 max 1h
55 + info: one-minute load average
56 + to: sysadmin