@cryptotaxi247 / netdata-1 / commits / 59d95ec88

Revert "Update alarms info" (#11633)

Ilya Mashchenko committed Oct 18, 2021 at 11:36 UTC 59d95ec88cd3663c5aaa7efd408fbbdca981a0b1
58 files changed +211 -599
health/health.d/apcupsd.conf
+2 -8
@@ -13,11 +13,7 @@ component: UPS
13 warn: $this > (($status >= $WARNING) ? (70) : (80))
14 crit: $this > (($status == $CRITICAL) ? (85) : (95))
15 delay: down 10m multiplier 1.5 max 1h
16 - info: Average UPS load over the last 10 minutes. \
17 - High UPS load. \
18 - It may result in either your UPS transferring to bypass or shutting down \
19 - as a self-protection and safety measure due to overload. \
20 - You should remove some attached equipment from the UPS.
16 + info: average UPS load over the last 10 minutes
17 to: sitemgr
18
19 # Discussion in https://github.com/netdata/netdata/pull/3928:
@@ -35,9 +31,7 @@ component: UPS
31 warn: $this < 100
32 crit: $this < (($status == $CRITICAL) ? (60) : (50))
33 delay: down 10m multiplier 1.5 max 1h
38 - info: Average UPS charge over the last minute. \
39 - The UPS is running on battery power. It will shut down if external power is not restored. \
40 - You should prepare any attached equipment for the shutdown.
34 + info: average UPS charge over the last minute
35 to: sitemgr
36
37 template: apcupsd_last_collected_secs
health/health.d/backend.conf
+3 -10
@@ -9,8 +9,7 @@ component: Exporting engine
9 every: 1m
10 warn: $this > 0
11 delay: down 5m multiplier 1.5 max 1h
12 - info: The backends subsystem is deprecated and will be removed soon. \
13 - Migrate your configuration to exporting.conf.
12 + info: the backends subsystem is deprecated and will be removed soon. Migrate your configuration to exporting.conf.
13 to: sysadmin
14
15 # make sure we are sending data to backend
@@ -26,10 +25,7 @@ component: Exporting engine
25 warn: $this > (($status >= $WARNING) ? ($update_every) : ( 5 * $update_every))
26 crit: $this > (($status == $CRITICAL) ? ($update_every) : (60 * $update_every))
27 delay: down 5m multiplier 1.5 max 1h
29 - info: Number of seconds since the last successful buffering of backend data. \
30 - The backend subsystem failed to buffer metrics for a while. Some metrics are lost while exporting. \
31 - It indicates that the backend destination is down or unreachable. \
32 - Short-term network availability issues might be fixed by increasing [buffer on failures] in netdata.conf.
28 + info: number of seconds since the last successful buffering of backend data
29 to: dba
30
31 alarm: backend_metrics_sent
@@ -42,8 +38,5 @@ component: Exporting engine
38 every: 10s
39 warn: $this != 100
40 delay: down 5m multiplier 1.5 max 1h
45 - info: Percentage of metrics sent to the backend server. \
46 - The backends subsystem failed to send all metrics. Some metrics are lost while exporting. \
47 - It indicates that the backend destination is down or unreachable. \
48 - Short-term network availability issues might be fixed by increasing [buffer on failures] in netdata.conf.
41 + info: percentage of metrics sent to the backend server
42 to: dba
health/health.d/bcache.conf
+5 -6
@@ -9,9 +9,9 @@ component: Disk
9 every: 1m
10 warn: $this > 0
11 delay: up 2m down 1h multiplier 1.5 max 2h
12 - info: Number of bcache read races in the last minute. \
13 - The bucket was reused and invalidated while reading from the cache. \
14 - When this occurs the data is reread from the backing device.
12 + info: number of times data was read from the cache, \
13 + the bucket was reused and invalidated in the last 10 minutes \
14 + (when this occurs the data is reread from the backing device)
15 to: sysadmin
16
17 template: bcache_cache_dirty
@@ -25,7 +25,6 @@ component: Disk
25 warn: $this > ( ($status >= $WARNING ) ? ( 70 ) : ( 90 ) )
26 crit: $this > ( ($status == $CRITICAL) ? ( 90 ) : ( 95 ) )
27 delay: up 1m down 1h multiplier 1.5 max 2h
28 - info: Percentage of cache space used for dirty data and metadata. \
29 - High block cache utilization by dirty data and metadata. \
30 - This usually means your SSD cache is too small.
28 + info: percentage of cache space used for dirty data and metadata \
29 + (this usually means your SSD cache is too small)
30 to: sysadmin
health/health.d/beanstalkd.conf
+3 -5
@@ -11,11 +11,9 @@ component: Beanstalk
11 warn: $this > 0
12 crit: $this > 10
13 delay: up 0 down 5m multiplier 1.2 max 1h
14 - info: Number of buried jobs across all tubes. \
15 - There are buried jobs. \
16 - It usually happens if something goes wrong while the consumer processes it. \
17 - The presence of buried jobs in a tube does not affect new jobs. \
18 - You need to manually kick the jobs, so they can be processed.
14 + info: number of buried jobs across all tubes. \
15 + You need to manually kick them so they can be processed. \
16 + Presence of buried jobs in a tube does not affect new jobs.
17 to: sysadmin
18
19 # get the number of buried jobs per queue
health/health.d/btrfs.conf
+4 -17
@@ -13,10 +13,7 @@ component: File system
13 warn: $this > (($status >= $WARNING) ? (90) : (95))
14 crit: $this > (($status == $CRITICAL) ? (95) : (98))
15 delay: up 1m down 15m multiplier 1.5 max 1h
16 - info: Percentage of allocated Btrfs physical disk space. \
17 - Most of the Btrfs physical disk space is allocated. \
18 - To fix it, first, try running Btrfs balance. \
19 - If that does not help, consider deleting snapshots or adding more physical space to the pool.
16 + info: percentage of allocated BTRFS physical disk space
17 to: sysadmin
18
19 template: btrfs_data
@@ -33,11 +30,7 @@ component: File system
30 warn: $this > (($status >= $WARNING) ? (90) : (95)) && $btrfs_allocated > 98
31 crit: $this > (($status == $CRITICAL) ? (95) : (98)) && $btrfs_allocated > 98
32 delay: up 1m down 15m multiplier 1.5 max 1h
36 - info: Percentage of used Btrfs data space. \
37 - High Btrfs data space utilization. \
38 - If there is enough unallocated memory, the data space will be automatically increased. \
39 - Otherwise, to fix, first try to run a balance. \
40 - If that does not help, you should add more physical space to the pool.
33 + info: utilization of BTRFS data space
34 to: sysadmin
35
36 template: btrfs_metadata
@@ -54,11 +47,7 @@ component: File system
47 warn: $this > (($status >= $WARNING) ? (90) : (95)) && $btrfs_allocated > 98
48 crit: $this > (($status == $CRITICAL) ? (95) : (98)) && $btrfs_allocated > 98
49 delay: up 1m down 15m multiplier 1.5 max 1h
57 - info: Percentage of used Btrfs metadata space. \
58 - High Btrfs metadata space utilization. \
59 - If there is enough unallocated memory, the metadata space will be automatically increased. \
60 - Otherwise, you may wish to run a balance on metadata only if you find you have very large amounts of \
61 - metadata space allocated, but unused.
50 + info: utilization of BTRFS metadata space
51 to: sysadmin
52
53 template: btrfs_system
@@ -75,7 +64,5 @@ component: File system
64 warn: $this > (($status >= $WARNING) ? (90) : (95)) && $btrfs_allocated > 98
65 crit: $this > (($status == $CRITICAL) ? (95) : (98)) && $btrfs_allocated > 98
66 delay: up 1m down 15m multiplier 1.5 max 1h
78 - info: Percentage of used Btrfs system space. \
79 - High Btrfs system space utilization. \
80 - If there is enough unallocated memory, the system space will be automatically increased.
67 + info: utilization of BTRFS system space
68 to: sysadmin
health/health.d/ceph.conf
+1 -3
@@ -11,7 +11,5 @@ component: Ceph
11 warn: $this > (($status >= $WARNING ) ? (85) : (90))
12 crit: $this > (($status == $CRITICAL) ? (90) : (98))
13 delay: down 5m multiplier 1.2 max 1h
14 - info: Percentage of used cluster disk space. \
15 - High disk space utilization. \
16 - To fix this, consider adding a node or removing unneeded data from the cluster.
14 + info: cluster disk space utilization
15 to: sysadmin
health/health.d/cgroups.conf
+2 -8
@@ -14,10 +14,7 @@ component: CPU
14 warn: $this > (($status >= $WARNING) ? (75) : (85))
15 crit: $this > (($status == $CRITICAL) ? (85) : (95))
16 delay: down 15m multiplier 1.5 max 1h
17 - info: Average CPU utilization over the last 10 minutes. \
18 - High cgroup CPU utilization. \
19 - The system will throttle the cgroup CPU usage when the usage is over the limit. \
20 - To fix, increase the cgroup CPU limit.
17 + info: average cgroup CPU utilization over the last 10 minutes
18 to: sysadmin
19
20 template: cgroup_ram_in_use
@@ -33,8 +30,5 @@ component: Memory
30 warn: $this > (($status >= $WARNING) ? (80) : (90))
31 crit: $this > (($status == $CRITICAL) ? (90) : (98))
32 delay: down 15m multiplier 1.5 max 1h
36 - info: Percentage of used memory. \
37 - High cgroup memory utilization. \
38 - OOM will kill some processes when the utilization reaches 100%. \
39 - To fix, increase the cgroup memory limit (if set).
33 + info: cgroup memory utilization
34 to: sysadmin
health/health.d/cockroachdb.conf
+5 -16
@@ -12,9 +12,7 @@ component: CockroachDB
12 warn: $this > (($status >= $WARNING) ? (80) : (85))
13 crit: $this > (($status == $CRITICAL) ? (85) : (95))
14 delay: down 15m multiplier 1.5 max 1h
15 - info: Percentage of used storage space. \
16 - High storage capacity utilization. \
17 - To fix, increase the space available for CockroachDB data.
15 + info: storage capacity utilization
16 to: dba
17
18 template: cockroachdb_used_usable_storage_capacity
@@ -28,9 +26,7 @@ component: CockroachDB
26 warn: $this > (($status >= $WARNING) ? (80) : (85))
27 crit: $this > (($status == $CRITICAL) ? (85) : (95))
28 delay: down 15m multiplier 1.5 max 1h
31 - info: Percentage of storage usable space. \
32 - High usable storage capacity utilization. \
33 - To fix, increase the space available for CockroachDB data.
29 + info: storage usable space utilization
30 to: dba
31
32 # Replication
@@ -45,10 +41,7 @@ component: CockroachDB
41 every: 10s
42 warn: $this > 0
43 delay: down 15m multiplier 1.5 max 1h
48 - info: Number of unavailable ranges. \
49 - There are ranges with fewer live replicas than needed for quorum. \
50 - If a majority of a range replicas are on nodes that are unavailable, \
51 - then the entire range is unavailable and will be unable to process queries.
44 + info: number of ranges with fewer live replicas than needed for quorum
45 to: dba
46
47 template: cockroachdb_underreplicated_ranges
@@ -61,9 +54,7 @@ component: CockroachDB
54 every: 10s
55 warn: $this > 0
56 delay: down 15m multiplier 1.5 max 1h
64 - info: Number of under-replicated ranges. \
65 - There are ranges with fewer live replicas than the replication target. \
66 - As soon as other nodes are available, they will replicate to them until they have reached their desired replication factor.
57 + info: number of ranges with fewer live replicas than the replication target
58 to: dba
59
60 # FD
@@ -78,7 +69,5 @@ component: CockroachDB
69 every: 10s
70 warn: $this > 80
71 delay: down 15m multiplier 1.5 max 1h
81 - info: Percentage of used file descriptors. \
82 - High file descriptors utilization (against softlimit). \
83 - To fix, adjust the file descriptors limit for the process or system-wide.
72 + info: open file descriptors utilization (against softlimit)
73 to: dba
health/health.d/cpu.conf
+4 -16
@@ -14,11 +14,7 @@ component: CPU
14 warn: $this > (($status >= $WARNING) ? (75) : (85))
15 crit: $this > (($status == $CRITICAL) ? (85) : (95))
16 delay: down 15m multiplier 1.5 max 1h
17 - info: Average CPU utilization over the last 10 minutes (excluding iowait, nice and steal). \
18 - High system CPU utilization. \
19 - A constantly high value might indicate CPU bottleneck, which can make the system run slower. \
20 - You can check the CPU PSI charts if there is CPU contention and \
21 - per-process CPU usage to find the top consumers.
17 + info: average CPU utilization over the last 10 minutes (excluding iowait, nice and steal)
18 to: sysadmin
19
20 template: 10min_cpu_iowait
@@ -34,9 +30,7 @@ component: CPU
30 warn: $this > (($status >= $WARNING) ? (20) : (40))
31 crit: $this > (($status == $CRITICAL) ? (40) : (50))
32 delay: down 15m multiplier 1.5 max 1h
37 - info: Average CPU iowait time over the last 10 minutes. \
38 - High system CPU iowait time. \
39 - A constantly high value indicates that IO is a bottleneck, which can make the system run slower.
33 + info: average CPU iowait time over the last 10 minutes
34 to: sysadmin
35
36 template: 20min_steal_cpu
@@ -52,10 +46,7 @@ component: CPU
46 warn: $this > (($status >= $WARNING) ? (5) : (10))
47 crit: $this > (($status == $CRITICAL) ? (20) : (30))
48 delay: down 1h multiplier 1.5 max 2h
55 - info: Average CPU steal time over the last 20 minutes. \
56 - High system CPU steal time. \
57 - A large amount of steal time indicates CPU contention on the host system, which can reduce guest performance. \
58 - To fix, increase the guest CPU priority or CPU quota, or run fewer guests on the host.
49 + info: average CPU steal time over the last 20 minutes
50 to: sysadmin
51
52 ## FreeBSD
@@ -72,8 +63,5 @@ component: CPU
63 warn: $this > (($status >= $WARNING) ? (75) : (85))
64 crit: $this > (($status == $CRITICAL) ? (85) : (95))
65 delay: down 15m multiplier 1.5 max 1h
75 - info: Average CPU utilization over the last 10 minutes (excluding nice). \
76 - High system CPU utilization. \
77 - A constantly high value might indicate CPU bottleneck, which can make the system run slower. \
78 - You can check per-process CPU usage to find the top consumers.
66 + info: average CPU utilization over the last 10 minutes (excluding nice)
67 to: sysadmin
health/health.d/dbengine.conf
+6 -10
@@ -13,8 +13,7 @@ component: DB engine
13 every: 10s
14 crit: $this > 0
15 delay: down 15m multiplier 1.5 max 1h
16 - info: Number of filesystem errors in the last 10 minutes. \
17 - Dbengine is experiencing filesystem errors (too many open files, wrong permissions, etc.).
16 + info: number of filesystem errors in the last 10 minutes (too many open files, wrong permissions, etc)
17 to: sysadmin
18
19 alarm: 10min_dbengine_global_io_errors
@@ -29,8 +28,7 @@ component: DB engine
28 every: 10s
29 crit: $this > 0
30 delay: down 1h multiplier 1.5 max 3h
32 - info: Number of IO errors in the last 10 minutes. \
33 - Dbengine is experiencing I/O errors (CRC errors, out of space, bad disk, etc.).
31 + info: number of IO errors in the last 10 minutes (CRC errors, out of space, bad disk, etc)
32 to: sysadmin
33
34 alarm: 10min_dbengine_global_flushing_warnings
@@ -45,9 +43,8 @@ component: DB engine
43 every: 10s
44 warn: $this > 0
45 delay: down 1h multiplier 1.5 max 3h
48 - info: Number of times when dbengine dirty pages were over 50% of the instance page cache in the last 10 minutes. \
49 - Metric data is at risk of not being stored in the database. \
50 - To remedy, reduce disk load or use faster disks.
46 + info: number of times when dbengine dirty pages were over 50% of the instance's page cache in the last 10 minutes. \
47 + Metric data are at risk of not being stored in the database. To remedy, reduce disk load or use faster disks.
48 to: sysadmin
49
50 alarm: 10min_dbengine_global_flushing_errors
@@ -62,7 +59,6 @@ component: DB engine
59 every: 10s
60 crit: $this != 0
61 delay: down 1h multiplier 1.5 max 3h
65 - info: Number of pages deleted due to failure to flush data to disk in the last 10 minutes. \
66 - Some metric data was dropped to unblock data collection. \
67 - To fix, reduce disk load or use faster disks.
62 + info: number of pages deleted due to failure to flush data to disk in the last 10 minutes. \
63 + Metric data were lost to unblock data collection. To fix, reduce disk load or use faster disks.
64 to: sysadmin
health/health.d/disks.conf
+3 -12
@@ -23,10 +23,7 @@ component: Disk
23 warn: $this > (($status >= $WARNING ) ? (80) : (90))
24 crit: $this > (($status == $CRITICAL) ? (90) : (98))
25 delay: up 1m down 15m multiplier 1.5 max 1h
26 - info: Percentage of used space by disk $family. \
27 - High disk space utilization. \
28 - You may experience slowdowns and crashes if the disk is full. \
29 - To fix, cleanup your disk or upgrade it.
26 + info: disk $family space utilization
27 to: sysadmin
28
29 template: disk_inode_usage
@@ -43,10 +40,7 @@ component: Disk
40 warn: $this > (($status >= $WARNING) ? (80) : (90))
41 crit: $this > (($status == $CRITICAL) ? (90) : (98))
42 delay: up 1m down 15m multiplier 1.5 max 1h
46 - info: Percentage of used inodes by disk $family. \
47 - High disk inode utilization. \
48 - The number of inodes indicates the number of files and folders you have. \
49 - To fix, clear cache files or delete unnecessary files and folders.
43 + info: disk $family inode utilization
44 to: sysadmin
45
46
@@ -153,10 +147,7 @@ component: Disk
147 every: 1m
148 warn: $this > 98 * (($status >= $WARNING) ? (0.7) : (1))
149 delay: down 15m multiplier 1.2 max 1h
156 - info: Average percentage of time $family disk was busy over the last 10 minutes. \
157 - High disk load. \
158 - The disk spent most of the time servicing read or write requests. \
159 - If the disk controller processes the operations in parallel, the alarm does not necessarily indicate a high load.
150 + info: average percentage of time $family disk was busy over the last 10 minutes
151 to: silent
152
153
health/health.d/dns_query.conf
+1 -2
@@ -11,6 +11,5 @@ component: DNS
11 every: 10s
12 warn: $this == nan
13 delay: up 20s down 5m multiplier 1.5 max 1h
14 - info: Average DNS query round trip time over the last 10 seconds. \
15 - Failed to query the DNS server.
14 + info: average DNS query round trip time over the last 10 seconds
15 to: sysadmin
health/health.d/dnsmasq_dhcp.conf
+1 -4
@@ -11,8 +11,5 @@ component: Dnsmasq
11 warn: $this > ( ($status >= $WARNING ) ? ( 80 ) : ( 90 ) )
12 crit: $this > ( ($status == $CRITICAL) ? ( 90 ) : ( 95 ) )
13 delay: down 5m
14 - info: Percentage of leased IP addresses. \
15 - High DHCP range utilization. \
16 - The number of DHCP addresses in use is close to the total number of provisioned DHCP addresses. \
17 - To fix, increase the number of IP addresses on a subnet.
14 + info: DHCP range utilization
15 to: sysadmin
health/health.d/dockerd.conf
+1 -4
@@ -7,8 +7,5 @@ component: Docker
7 every: 10s
8 lookup: average -10s
9 crit: $this > 0
10 - info: Average number of unhealthy docker containers over the last 10 seconds. \
11 - There are unhealthy docker containers. \
12 - Some containers are not running due to failed health checks. \
13 - To find unhealthy containers use [docker ps -a].
10 + info: average number of unhealthy docker containers over the last 10 seconds
11 to: sysadmin
health/health.d/entropy.conf
+1 -4
@@ -15,8 +15,5 @@ component: Cryptography
15 every: 5m
16 warn: $this < (($status >= $WARNING) ? (200) : (100))
17 delay: down 1h multiplier 1.5 max 2h
18 - info: Minimum number of bits in the random numbers pool in the last 5 minutes. \
19 - A low number of bits of entropy available. \
20 - It may have a negative impact on the security and performance of the system. \
21 - This can be fixed by installing the [haveged] or [rngd] daemon.
18 + info: minimum number of entries in the random numbers pool in the last 5 minutes
19 to: silent
health/health.d/exporting.conf
+2 -10
@@ -11,11 +11,7 @@ component: Exporting engine
11 warn: $this > (($status >= $WARNING) ? ($update_every) : ( 5 * $update_every))
12 crit: $this > (($status == $CRITICAL) ? ($update_every) : (60 * $update_every))
13 delay: down 5m multiplier 1.5 max 1h
14 - info: Number of seconds since the last successful buffering of exporting data. \
15 - Exporting engine failed to buffer metrics for a while. \
16 - Some metrics were lost while exporting. \
17 - It indicates that the exporting destination is down or unreachable. \
18 - Short-term network availability problems might be fixed by increasing [buffer on failures] in exporting.conf.
14 + info: number of seconds since the last successful buffering of exporting data
15 to: dba
16
17 template: exporting_metrics_sent
@@ -29,9 +25,5 @@ component: Exporting engine
25 every: 10s
26 warn: $this != 100
27 delay: down 5m multiplier 1.5 max 1h
32 - info: Percentage of metrics sent to the external database server. \
33 - Exporting engine failed to send all metrics. \
34 - Some metrics were lost while exporting. \
35 - It indicates that the exporting destination is down or unreachable. \
36 - Short-term network availability problems might be fixed by increasing [buffer on failures] in exporting.conf.
28 + info: percentage of metrics sent to the external database server
29 to: dba
health/health.d/fping.conf
+3 -9
@@ -25,9 +25,7 @@ component: Network
25 every: 10s
26 crit: $this == 0
27 delay: down 30m multiplier 1.5 max 2h
28 - info: Reachability status of the network host (0: unreachable, 1: reachable). \
29 - The network host is unreachable. \
30 - Most likely the host is down, or you are experiencing networking issues.
28 + info: reachability status of the network host (0: unreachable, 1: reachable)
29 to: sysadmin
30
31 template: fping_host_latency
@@ -44,9 +42,7 @@ component: Network
42 warn: $this > $green OR $max > $red
43 crit: $this > $red
44 delay: down 30m multiplier 1.5 max 2h
47 - info: Average latency to the network host over the last 10 seconds. \
48 - High latency to the network host. \
49 - Most likely you are experiencing networking issues or the host is overloaded.
45 + info: average latency to the network host over the last 10 seconds
46 to: sysadmin
47
48 template: fping_packet_loss
@@ -64,7 +60,5 @@ component: Network
60 warn: $this > $green
61 crit: $this > $red
62 delay: down 30m multiplier 1.5 max 2h
67 - info: Packet loss ratio to the network host over the last 10 minutes. \
68 - High packet loss to the network host. \
69 - Most likely you are experiencing networking issues or the host is overloaded.
63 + info: packet loss ratio to the network host over the last 10 minutes
64 to: sysadmin
health/health.d/gearman.conf
+1 -3
@@ -10,7 +10,5 @@ component: Gearman
10 warn: $this > 30000
11 crit: $this > 100000
12 delay: down 5m multiplier 1.5 max 1h
13 - info: Average number of queued jobs over the last 10 minutes. \
14 - A high number of queued jobs. \
15 - To fix, add more workers.
13 + info: average number of queued jobs over the last 10 minutes
14 to: sysadmin
health/health.d/haproxy.conf
+2 -4
@@ -7,8 +7,7 @@ component: HAProxy
7 every: 10s
8 lookup: average -10s
9 crit: $this > 0
10 - info: Average number of failed haproxy backend servers over the last 10 seconds. \
11 - Some haproxy backend servers are inaccessible or offline.
10 + info: average number of failed haproxy backend servers over the last 10 seconds
11 to: sysadmin
12
13 template: haproxy_backend_status
@@ -20,6 +19,5 @@ component: HAProxy
19 every: 10s
20 lookup: average -10s
21 crit: $this > 0
23 - info: Average number of failed haproxy backends over the last 10 seconds. \
24 - Some haproxy backends are offline.
22 + info: average number of failed haproxy backends over the last 10 seconds
23 to: sysadmin
health/health.d/hdfs.conf
+5 -22
@@ -12,9 +12,7 @@ component: HDFS
12 warn: $this > (($status >= $WARNING) ? (70) : (80))
13 crit: $this > (($status == $CRITICAL) ? (80) : (98))
14 delay: down 15m multiplier 1.5 max 1h
15 - info: Percentage of used space capacity across all datanodes. \
16 - High datanodes space capacity utilization. \
17 - To fix this, you can add additional disks/datanodes and rebalance the datanode/cluster.
15 + info: summary datanodes space capacity utilization
16 to: sysadmin
17
18
@@ -30,11 +28,7 @@ component: HDFS
28 every: 10s
29 warn: $this > 0
30 delay: down 15m multiplier 1.5 max 1h
33 - info: Number of missing blocks. \
34 - There are missing blocks on some datanodes. \
35 - It may indicate a problem with the underlying storage or filesystem of a datanode. \
36 - To fix, you can find what blocks are missing using [hdfs fsck] and bring the blocks back online. \
37 - If the file can not be restored, you should delete the file.
31 + info: number of missing blocks
32 to: sysadmin
33
34
@@ -48,11 +42,7 @@ component: HDFS
42 every: 10s
43 warn: $this > 0
44 delay: down 15m multiplier 1.5 max 1h
51 - info: Number of stale datanodes. \
52 - There are stale datanodes due to delayed heartbeat. \
53 - Some datanodes did not respond to heartbeats. \
54 - The stale datanodes are given the lowest priority for reads and writes. \
55 - If the datanode is not back online for a while, it will be considered dead.
45 + info: number of datanodes marked stale due to delayed heartbeat
46 to: sysadmin
47
48
@@ -66,9 +56,7 @@ component: HDFS
56 every: 10s
57 crit: $this > 0
58 delay: down 15m multiplier 1.5 max 1h
69 - info: Number of datanodes that are currently dead. \
70 - There are dead datanodes. \
71 - Any data that was registered to a dead datanode is not available to HDFS anymore.
59 + info: number of datanodes which are currently dead
60 to: sysadmin
61
62
@@ -84,10 +72,5 @@ component: HDFS
72 every: 10s
73 warn: $this > 0
74 delay: down 15m multiplier 1.5 max 1h
87 - info: Number of failed volumes. \
88 - There are failed volumes on some datanodes. \
89 - It may indicate a hardware failure or misconfiguration, e.g. duplicate mounts. \
90 - By default, a single volume failing on a datanode will cause the entire node to go offline. \
91 - The namenode must copy any under-replicated blocks that were lost on that node, \
92 - causing a burst in network traffic and potential performance degradation.
75 + info: number of failed volumes
76 to: sysadmin
health/health.d/ioping.conf
+1 -3
@@ -12,7 +12,5 @@ component: Disk
12 warn: $this > $green OR $max > $red
13 crit: $this > $red
14 delay: down 30m multiplier 1.5 max 2h
15 - info: Average I/O latency over the last 10 seconds. \
16 - High disk I/O latency. \
17 - It may indicate a high load of the disk or the disk is slow.
15 + info: average I/O latency over the last 10 seconds
16 to: sysadmin
health/health.d/ipc.conf
+2 -8
@@ -14,10 +14,7 @@ component: IPC
14 warn: $this > (($status >= $WARNING) ? (70) : (80))
15 crit: $this > (($status == $CRITICAL) ? (70) : (90))
16 delay: down 5m multiplier 1.5 max 1h
17 - info: Percentage of used IPC semaphores. \
18 - High IPC semaphore utilization. \
19 - A lack of available semaphores can affect application performance. \
20 - To fix, adjust semaphore limits on your system.
17 + info: IPC semaphore utilization
18 to: sysadmin
19
20 alarm: semaphore_arrays_used
@@ -33,8 +30,5 @@ component: IPC
30 warn: $this > (($status >= $WARNING) ? (70) : (80))
31 crit: $this > (($status == $CRITICAL) ? (70) : (90))
32 delay: down 5m multiplier 1.5 max 1h
36 - info: Percentage of used IPC semaphore arrays. \
37 - High IPC semaphore arrays utilization. \
38 - A lack of available semaphore arrays can affect application performance. \
39 - To fix, adjust semaphore limits on your system.
33 + info: IPC semaphore arrays utilization
34 to: sysadmin
health/health.d/ipfs.conf
+1 -2
@@ -10,6 +10,5 @@ component: IPFS
10 warn: $this > (($status >= $WARNING) ? (80) : (90))
11 crit: $this > (($status == $CRITICAL) ? (90) : (98))
12 delay: down 15m multiplier 1.5 max 1h
13 - info: Percentage of used IPFS datastore space. \
14 - High IPFS storage repository space utilization.
13 + info: IPFS datastore utilization
14 to: sysadmin
health/health.d/ipmi.conf
+2 -7
@@ -9,9 +9,7 @@ component: IPMI
9 warn: $this > 0
10 crit: $critical > 0
11 delay: up 5m down 15m multiplier 1.5 max 1h
12 - info: Number of IPMI sensors in the non-nominal state. \
13 - There are IPMI sensors in the warning or critical state. \
14 - You can find the system sensor type, state, and threshold using [ipmi-sensors].
12 + info: number of IPMI sensors in non-nominal state
13 to: sysadmin
14
15 alarm: ipmi_events
@@ -24,8 +22,5 @@ component: IPMI
22 every: 10s
23 warn: $this > 0
24 delay: up 5m down 15m multiplier 1.5 max 1h
27 - info: Number of events in the IPMI System Event Log (SEL). \
28 - IPMI System Event Log (SEL) is not empty. \
29 - It contains critical, warning, and informational events. \
30 - To view them use [ipmitool sel list].
25 + info: number of events in the IPMI System Event Log (SEL)
26 to: sysadmin
health/health.d/kubelet.conf
+10 -14
@@ -29,8 +29,7 @@ component: Kubelet
29 every: 10s
30 warn: $this > 0
31 delay: down 1m multiplier 1.5 max 2h
32 - info: Number of failed get authentication token requests. \
33 - There are failed get authentication token requests to Google Compute Engine (alternate token source).
32 + info: number of failed Token() requests to the alternate token source
33 to: sysadmin
34
35 # Docker and runtime operation errors
@@ -71,7 +70,7 @@ component: Kubelet
70 lookup: average -1m unaligned of kubelet_pleg_relist_latency_05
71 units: microseconds
72 every: 10s
74 - info: Average pod relisting duration over the last minute (quantile 0.5).
73 + info: average Pod Lifecycle Event Generator relisting latency over the last minute (quantile 0.5)
74
75 template: kubelet_10s_pleg_relist_latency_quantile_05
76 on: k8s_kubelet.kubelet_pleg_relist_latency_microseconds
@@ -85,9 +84,8 @@ component: Kubelet
84 warn: $this > (($status >= $WARNING)?(100):(200))
85 crit: $this > (($status >= $WARNING)?(200):(400))
86 delay: down 1m multiplier 1.5 max 2h
88 - info: Ratio of average pod relisting duration over the last 10 seconds, compared to the last minute (quantile 0.5). \
89 - Relisting time has increased significantly. \
90 - It may indicate some problems with the container runtime engine.
87 + info: ratio of average Pod Lifecycle Event Generator relisting latency over the last 10 seconds, \
88 + compared to the last minute (quantile 0.5)
89 to: sysadmin
90
91 # quantile 0.9
@@ -100,7 +98,7 @@ component: Kubelet
98 lookup: average -1m unaligned of kubelet_pleg_relist_latency_09
99 units: microseconds
100 every: 10s
103 - info: Average pod relisting duration over the last minute (quantile 0.9).
101 + info: average Pod Lifecycle Event Generator relisting latency over the last minute (quantile 0.9)
102
103 template: kubelet_10s_pleg_relist_latency_quantile_09
104 on: k8s_kubelet.kubelet_pleg_relist_latency_microseconds
@@ -114,9 +112,8 @@ component: Kubelet
112 warn: $this > (($status >= $WARNING)?(200):(400))
113 crit: $this > (($status >= $WARNING)?(400):(800))
114 delay: down 1m multiplier 1.5 max 2h
117 - info: Ratio of average pod relisting duration over the last 10 seconds, compared to the last minute (quantile 0.9). \
118 - Relisting time has increased significantly. \
119 - It may indicate some problems with the container runtime engine.
115 + info: ratio of average Pod Lifecycle Event Generator relisting latency over the last 10 seconds, \
116 + compared to the last minute (quantile 0.9)
117 to: sysadmin
118
119 # quantile 0.99
@@ -129,7 +126,7 @@ component: Kubelet
126 lookup: average -1m unaligned of kubelet_pleg_relist_latency_099
127 units: microseconds
128 every: 10s
132 - info: Average pod relisting duration over the last minute (quantile 0.99).
129 + info: average Pod Lifecycle Event Generator relisting latency over the last minute (quantile 0.99)
130
131 template: kubelet_10s_pleg_relist_latency_quantile_099
132 on: k8s_kubelet.kubelet_pleg_relist_latency_microseconds
@@ -143,7 +140,6 @@ component: Kubelet
140 warn: $this > (($status >= $WARNING)?(400):(800))
141 crit: $this > (($status >= $WARNING)?(800):(1200))
142 delay: down 1m multiplier 1.5 max 2h
146 - info: Ratio of average pod relisting duration over the last 10 seconds, compared to the last minute (quantile 0.99). \
147 - Relisting time has increased significantly. \
148 - It may indicate some problems with the container runtime engine.
143 + info: ratio of average Pod Lifecycle Event Generator relisting latency over the last 10 seconds, \
144 + compared to the last minute (quantile 0.99)
145 to: sysadmin
health/health.d/linux_power_supply.conf
+1 -4
@@ -11,8 +11,5 @@ component: Battery
11 warn: $this < 10
12 crit: $this < 5
13 delay: up 30s down 5m multiplier 1.2 max 1h
14 - info: Percentage of remaining power supply capacity. \
15 - The remaining power supply capacity is low. \
16 - The system can run out of power and switch off soon. \
17 - Prepare the system for the shutdown.
14 + info: percentage of remaining power supply capacity
15 to: sysadmin
health/health.d/load.conf
+3 -15
@@ -32,11 +32,7 @@ component: Load
32 every: 1m
33 warn: ($this * 100 / $load_cpu_number) > (($status >= $WARNING) ? 175 : 200)
34 delay: down 15m multiplier 1.5 max 1h
35 - info: System fifteen-minute load average. \
36 - High system fifteen-minute load average. \
37 - A constantly high value indicates that your system is overloaded. \
38 - It includes both CPU and I/O demand. \
39 - You might want to check per-process CPU/disk usage to find the top consumers.
35 + info: system fifteen-minute load average
36 to: sysadmin
37
38 alarm: load_average_5
@@ -51,11 +47,7 @@ component: Load
47 every: 1m
48 warn: ($this * 100 / $load_cpu_number) > (($status >= $WARNING) ? 350 : 400)
49 delay: down 15m multiplier 1.5 max 1h
54 - info: System five-minute load average. \
55 - High system five-minute load average. \
56 - A constantly high value indicates that your system is overloaded. \
57 - It includes both CPU and I/O demand. \
58 - You might want to check per-process CPU/disk usage to find the top consumers.
50 + info: system five-minute load average
51 to: sysadmin
52
53 alarm: load_average_1
@@ -70,9 +62,5 @@ component: Load
62 every: 1m
63 warn: ($this * 100 / $load_cpu_number) > (($status >= $WARNING) ? 700 : 800)
64 delay: down 15m multiplier 1.5 max 1h
73 - info: System one-minute load average. \
74 - High system one-minute load average. \
75 - A constantly high value indicates that your system is overloaded. \
76 - It includes both CPU and I/O demand. \
77 - You might want to check per-process CPU/disk usage to find the top consumers.
65 + info: system one-minute load average
66 to: sysadmin
health/health.d/mdstat.conf
+3 -7
@@ -20,10 +20,8 @@ component: RAID
20 every: 10s
21 calc: $down
22 crit: $this > 0
23 - info: Number of devices in the down state for the $family array. \
24 - The array is degraded. Some array devices are missing. \
25 - To fix, bring them back online or replace the faulty ones. \
26 - To find faulty devices use [mdadm --detail RAIDDEVICE].
23 + info: number of devices in the down state for the $family array. \
24 + Any number > 0 indicates that the array is degraded.
25 to: sysadmin
26
27 template: mdstat_mismatch_cnt
@@ -37,9 +35,7 @@ component: RAID
35 every: 60s
36 warn: $this > 1024
37 delay: up 30m
40 - info: Number of unsynchronized blocks for the $family array. \
41 - A high number of unsynchronized blocks for the $family array. \
42 - This might indicate that data on the array is corrupted.
38 + info: number of unsynchronized blocks for the $family array
39 to: sysadmin
40
41 template: mdstat_nonredundant_last_collected
health/health.d/megacli.conf
+5 -20
@@ -11,7 +11,7 @@ component: RAID
11 every: 10s
12 crit: $this > 0
13 delay: down 5m multiplier 2 max 10m
14 - info: Adapter is in the degraded state (0: false, 1: true).
14 + info: adapter is in the degraded state (0: false, 1: true)
15 to: sysadmin
16
17 ## Physical Disks
@@ -26,12 +26,7 @@ component: RAID
26 every: 10s
27 warn: $this > 0
28 delay: up 1m down 5m multiplier 2 max 10m
29 - info: Number of physical drive predictive failures. \
30 - There are physical drive predictive failures. \
31 - The failure prediction function for the hard disk drives determines the risk of a failure in advance \
32 - and issues a warning when the risk is high. \
33 - A hard disk can still operate normally but may fail in the near future. \
34 - You might want to consider replacing the disk.
29 + info: number of physical drive predictive failures
30 to: sysadmin
31
32 template: megacli_pd_media_errors
@@ -44,10 +39,7 @@ component: RAID
39 every: 10s
40 warn: $this > 0
41 delay: up 1m down 5m multiplier 2 max 10m
47 - info: Number of physical drive media errors. \
48 - There are physical drive media errors. \
49 - It may indicate that a bad sector was found on the media, there is a mechanical failure of the device, \
50 - the host device detected an invalid sequence, or the target device is missing.
42 + info: number of physical drive media errors
43 to: sysadmin
44
45 ## Battery Backup Units (BBU)
@@ -62,11 +54,7 @@ component: RAID
54 every: 10s
55 warn: $this <= (($status >= $WARNING) ? (85) : (80))
56 crit: $this <= (($status == $CRITICAL) ? (50) : (40))
65 - info: Average battery backup unit relative state of charge over the last 10 seconds. \
66 - The state of charge is low. \
67 - The relative state of charge is an indication of full charge capacity percentage in relation to the design capacity. \
68 - A constantly low value may indicate that the battery is worn out. \
69 - You might want to consider changing the battery.
57 + info: average battery backup unit (BBU) relative state of charge over the last 10 seconds
58 to: sysadmin
59
60 template: megacli_bbu_cycle_count
@@ -79,8 +67,5 @@ component: RAID
67 every: 10s
68 warn: $this >= 100
69 crit: $this >= 500
82 - info: Average battery backup unit charge cycles count over the last 10 seconds. \
83 - A high number of full recharge cycles. \
84 - It affects the battery capacity. \
85 - You might want to consider changing the battery.
70 + info: average battery backup unit (BBU) charge cycles count over the last 10 seconds
71 to: sysadmin
health/health.d/memcached.conf
+4 -9
@@ -12,10 +12,7 @@ component: Memcached
12 warn: $this > (($status >= $WARNING) ? (70) : (80))
13 crit: $this > (($status == $CRITICAL) ? (80) : (90))
14 delay: up 0 down 15m multiplier 1.5 max 1h
15 - info: Percentage of used memory. \
16 - High cache memory utilization. \
17 - If you are getting close to 100% you will probably start experiencing evictions. \
18 - Consider increasing the cache size.
15 + info: cache memory utilization
16 to: dba
17
18
@@ -30,7 +27,7 @@ component: Memcached
27 calc: ($this - $available) / (($now - $after) / 3600)
28 units: KB/hour
29 every: 1m
33 - info: Average rate the cache fills up (positive), or frees up (negative) space over the last hour.
30 + info: average rate the cache fills up (positive), or frees up (negative) space over the last hour
31
32
33 # find the hours remaining until memcached cache is full
@@ -46,8 +43,6 @@ component: Memcached
43 warn: $this > 0 and $this < (($status >= $WARNING) ? (48) : (8))
44 crit: $this > 0 and $this < (($status == $CRITICAL) ? (24) : (2))
45 delay: down 15m multiplier 1.5 max 1h
49 - info: Estimated time until the cache runs out of space \
50 - if the system continues to add data at the same rate as the past hour. \
51 - The cache will run out of space soon. \
52 - If you are getting close to 100% you will probably start experiencing evictions.
46 + info: estimated time the cache will run out of space \
47 + if the system continues to add data at the same rate as the past hour
48 to: dba
health/health.d/memory.conf
+3 -14
@@ -13,10 +13,7 @@ component: Memory
13 every: 1m
14 warn: $this > 0
15 delay: down 1h multiplier 1.5 max 1h
16 - info: Number of ECC correctable errors in the last 10 minutes. \
17 - There are ECC correctable errors. \
18 - This event does not immediately lead to problems, but it may indicate one of the DIMM modules is slowly failing. \
19 - Check contacts and, if this error occurs more than once, consider replacing the DIMM as a preventative measure.
16 + info: number of ECC correctable errors in the last 10 minutes
17 to: sysadmin
18
19 alarm: 1hour_ecc_memory_uncorrectable
@@ -31,11 +28,7 @@ component: Memory
28 every: 1m
29 crit: $this > 0
30 delay: down 1h multiplier 1.5 max 1h
34 - info: Number of ECC uncorrectable errors in the last 10 minutes. \
35 - There are ECC uncorrectable errors. This is a fatal issue. \
36 - While the error may be due to a failing DRAM chip, \
37 - it can also be caused by incorrect seating or improper contact between the socket and DIMM. \
38 - Check contacts and consider replacing the DIMM as a preventative measure.
31 + info: number of ECC uncorrectable errors in the last 10 minutes
32 to: sysadmin
33
34 alarm: 1hour_memory_hw_corrupted
@@ -50,9 +43,5 @@ component: Memory
43 every: 10s
44 warn: $this > 0
45 delay: down 1h multiplier 1.5 max 1h
53 - info: Amount of memory corrupted due to a hardware failure. \
54 - Memory is corrupted due to a hardware failure. \
55 - While the error may be due to a failing DRAM chip, \
56 - it can also be caused by incorrect seating or improper contact between the socket and DIMM. \
57 - Check contacts and consider replacing the DIMM as a preventative measure.
46 + info: amount of memory corrupted due to a hardware failure
47 to: sysadmin
health/health.d/mysql.conf
+11 -29
@@ -12,11 +12,7 @@ component: MySQL
12 warn: $this > (($status >= $WARNING) ? (5) : (10))
13 crit: $this > (($status == $CRITICAL) ? (10) : (20))
14 delay: down 5m multiplier 1.5 max 1h
15 - info: Number of slow queries in the last 10 seconds. \
16 - A high number of slow queries. \
17 - A query is slow if it has taken more than [long_query_time] seconds. \
18 - The value is measured in real-time, not CPU time. \
19 - If you are concerned with query execution time, check system load and consider optimizing the queries.
15 + info: number of slow queries in the last 10 seconds
16 to: dba
17
18
@@ -56,10 +52,7 @@ component: MySQL
52 warn: $this > (($status >= $WARNING) ? (10) : (25))
53 crit: $this > (($status == $CRITICAL) ? (25) : (50))
54 delay: down 30m multiplier 1.5 max 1h
59 - info: Ratio of waited table locks over the last 10 seconds. \
60 - High ratio of waited table locks. \
61 - If this is constantly high and you have performance problems, \
62 - you should first optimize your queries and then either split your table or tables or use replication.
55 + info: ratio of waited table locks over the last 10 seconds
56 to: dba
57
58
@@ -77,9 +70,7 @@ component: MySQL
70 warn: $this > (($status >= $WARNING) ? (60) : (70))
71 crit: $this > (($status == $CRITICAL) ? (80) : (90))
72 delay: down 15m multiplier 1.5 max 1h
80 - info: Percentage of used client connections. \
81 - High client connections utilization. \
82 - To fix, increase the number of client connections using [max_connections] MySQL system variable.
73 + info: client connections utilization
74 to: dba
75
76
@@ -96,10 +87,7 @@ component: MySQL
87 every: 10s
88 crit: $this == 0
89 delay: down 5m multiplier 1.5 max 1h
99 - info: Replication status (0: stopped, 1: working). \
100 - Replication is not working. \
101 - Either both or one of the I/O and SQL threads are not running. \
102 - Checking error.log may help to identify the problem.
90 + info: replication status (0: stopped, 1: working)
91 to: dba
92
93 template: mysql_replication_lag
@@ -113,10 +101,8 @@ component: MySQL
101 warn: $this > (($status >= $WARNING) ? (5) : (10))
102 crit: $this > (($status == $CRITICAL) ? (10) : (30))
103 delay: down 15m multiplier 1.5 max 1h
116 - info: The number of seconds that the replica is behind the master. \
117 - The replication SQL thread is far behind processing the source binary log. \
118 - A constantly high value (or an increasing one) indicates that the replica is unable \
119 - to handle events from the source in a timely fashion.
104 + info: difference between the timestamp of the latest transaction processed by the SQL thread and \
105 + the timestamp of the same transaction when it was processed on the master
106 to: dba
107
108
@@ -145,9 +131,7 @@ component: MySQL
131 warn: $this > $mysql_galera_cluster_size_max_2m
132 crit: $this < $mysql_galera_cluster_size_max_2m
133 delay: up 20s down 5m multiplier 1.5 max 1h
148 - info: Current Galera cluster size, compared to the maximum size in the last 2 minutes. \
149 - Galera cluster size has changed. \
150 - It may indicate a network connectivity problem, or maybe MySQL is down on one node.
134 + info: current galera cluster size, compared to the maximum size in the last 2 minutes
135 to: dba
136
137 # galera node state
@@ -162,9 +146,8 @@ component: MySQL
146 warn: $this == 2 OR $this == 3
147 crit: $this == 0 OR $this == 1 OR $this >= 5
148 delay: up 30s down 5m multiplier 1.5 max 1h
165 - info: Galera node state (0: Undefined, 1: Joining, 2: Donor/Desynced, 3: Joined, 4: Synced, 5: Inconsistent). \
166 - Galera node is not synced. \
167 - It may indicate that the node lost its connection to the Primary Component due to network partition.
149 + info: galera node state \
150 + (0: Undefined, 1: Joining, 2: Donor/Desynced, 3: Joined, 4: Synced, 5: Inconsistent)
151 to: dba
152
153
@@ -179,8 +162,7 @@ component: MySQL
162 every: 10s
163 crit: $mysql_galera_cluster_state != nan AND $this != 0
164 delay: up 30s down 5m multiplier 1.5 max 1h
182 - info: Galera node cluster component status \
165 + info: galera node cluster component status \
166 (-1: unknown, 0: primary/quorum present, 1: non-primary/quorum lost, 2: disconnected). \
184 - The node is a part of a nonoperational component. \
185 - It indicates that the node has lost a quorum or is unable to connect to other nodes/cluster.
167 + Any other value than primary indicates that the node is part of a nonoperational component.
168 to: dba
health/health.d/net.conf
+11 -33
@@ -31,9 +31,7 @@ component: Network
31 every: 10s
32 warn: $this > (($status >= $WARNING) ? (85) : (90))
33 delay: up 1m down 1m multiplier 1.5 max 1h
34 - info: Average inbound utilization for the network interface $family over the last minute. \
35 - High inbound utilization. \
36 - Network delays and packet drops are expected when you are getting closer to 100%.
34 + info: average inbound utilization for the network interface $family over the last minute
35 to: sysadmin
36
37 template: 1m_sent_traffic_overflow
@@ -50,9 +48,7 @@ component: Network
48 every: 10s
49 warn: $this > (($status >= $WARNING) ? (85) : (90))
50 delay: up 1m down 1m multiplier 1.5 max 1h
53 - info: Average outbound utilization for the network interface $family over the last minute. \
54 - High outbound utilization. \
55 - Network delays and packet drops are expected when you are getting closer to 100%.
51 + info: average outbound utilization for the network interface $family over the last minute
52 to: sysadmin
53
54 # -----------------------------------------------------------------------------
@@ -105,9 +101,7 @@ component: Network
101 every: 1m
102 warn: $this >= 2
103 delay: up 1m down 1h multiplier 1.5 max 2h
108 - info: Ratio of inbound dropped packets for the network interface $family over the last 10 minutes. \
109 - Network interface $family dropped many packets. \
110 - It means packets were received but not processed, e.g. due to lack of resources or unsupported protocol.
104 + info: ratio of inbound dropped packets for the network interface $family over the last 10 minutes
105 to: sysadmin
106
107 template: outbound_packets_dropped_ratio
@@ -124,9 +118,7 @@ component: Network
118 every: 1m
119 warn: $this >= 2
120 delay: up 1m down 1h multiplier 1.5 max 2h
127 - info: Ratio of outbound dropped packets for the network interface $family over the last 10 minutes. \
128 - Network interface $family dropped many packets. \
129 - It means packets were dropped on their way to transmission, e.g. due to lack of resources.
121 + info: ratio of outbound dropped packets for the network interface $family over the last 10 minutes
122 to: sysadmin
123
124 template: wifi_inbound_packets_dropped_ratio
@@ -143,9 +135,7 @@ component: Network
135 every: 1m
136 warn: $this >= 10
137 delay: up 1m down 1h multiplier 1.5 max 2h
146 - info: Ratio of inbound dropped packets for the network interface $family over the last 10 minutes. \
147 - Network interface $family dropped many packets. \
148 - It means packets were received but not processed, e.g. due to lack of resources or unsupported protocol.
138 + info: ratio of inbound dropped packets for the network interface $family over the last 10 minutes
139 to: sysadmin
140
141 template: wifi_outbound_packets_dropped_ratio
@@ -162,9 +152,7 @@ component: Network
152 every: 1m
153 warn: $this >= 10
154 delay: up 1m down 1h multiplier 1.5 max 2h
165 - info: Ratio of outbound dropped packets for the network interface $family over the last 10 minutes. \
166 - Network interface $family dropped many packets. \
167 - It means packets were dropped on their way to transmission, e.g. due to lack of resources.
155 + info: ratio of outbound dropped packets for the network interface $family over the last 10 minutes
156 to: sysadmin
157
158 # -----------------------------------------------------------------------------
@@ -183,9 +171,7 @@ component: Network
171 every: 1m
172 warn: $this >= 5
173 delay: down 1h multiplier 1.5 max 2h
186 - info: Number of inbound errors for the network interface $family in the last 10 minutes. \
187 - Network interface $family received many bad packets. \
188 - It includes length, CRC, frame errors, etc.
174 + info: number of inbound errors for the network interface $family in the last 10 minutes
175 to: sysadmin
176
177 template: interface_outbound_errors
@@ -201,9 +187,7 @@ component: Network
187 every: 1m
188 warn: $this >= 5
189 delay: down 1h multiplier 1.5 max 2h
204 - info: Number of outbound errors for the network interface $family in the last 10 minutes. \
205 - Network interface $family experienced many transmit problems. \
206 - It includes aborted, carrier, FIFO, heartbeat, window errors, etc.
190 + info: number of outbound errors for the network interface $family in the last 10 minutes
191 to: sysadmin
192
193 # -----------------------------------------------------------------------------
@@ -227,11 +211,7 @@ component: Network
211 every: 1m
212 warn: $this > 0
213 delay: down 1h multiplier 1.5 max 2h
230 - info: Number of FIFO errors for the network interface $family in the last 10 minutes. \
231 - Network interface $family experienced FIFO errors. \
232 - It indicates that the NIC is not able to handle the peak load of \
233 - incoming/outgoing packets with the current ring buffer size. \
234 - In case of a high packet drop rate, increasing the appropriate ring buffer can fix the issue.
214 + info: number of FIFO errors for the network interface $family in the last 10 minutes
215 to: sysadmin
216
217 # -----------------------------------------------------------------------------
@@ -271,8 +251,6 @@ component: Network
251 warn: $this > (($status >= $WARNING)?(200):(5000))
252 crit: $this > (($status == $CRITICAL)?(5000):(6000))
253 options: no-clear-notification
274 - info: Ratio of an average number of received packets for the network interface $family over the last 10 seconds, \
275 - compared to the rate over the last minute. \
276 - The number of received packets has increased significantly. \
277 - It may indicate a broadcast/multicast storm or DoS attack.
254 + info: ratio of average number of received packets for the network interface $family over the last 10 seconds, \
255 + compared to the rate over the last minute
256 to: sysadmin
health/health.d/netfilter.conf
+1 -4
@@ -15,8 +15,5 @@ component: Network
15 warn: $this > (($status >= $WARNING) ? (85) : (90))
16 crit: $this > (($status == $CRITICAL) ? (90) : (95))
17 delay: down 5m multiplier 1.5 max 1h
18 - info: Percentage of used netfilter tracked connections. \
19 - High netfilter connection tracking state table size utilization. \
20 - Network delays and packet drops are expected when you are getting closer to 100%. \
21 - To fix, increase the table size.
18 + info: netfilter connection tracker table size utilization
19 to: sysadmin
health/health.d/pihole.conf
+4 -11
@@ -12,8 +12,7 @@ component: Pi-hole
12 warn: $this > ( ($status >= $WARNING ) ? ( 45 ) : ( 55 ) )
13 crit: $this > ( ($status == $CRITICAL) ? ( 55 ) : ( 75 ) )
14 delay: up 2m down 5m
15 - info: Percentage of blocked DNS queries over the last 24 hours. \
16 - A high percentage of blocked DNS queries.
15 + info: percentage of blocked dns queries over the last 24 hour
16 to: sysadmin
17
18
@@ -30,9 +29,7 @@ component: Pi-hole
29 calc: $ago
30 warn: $this > 60 * 60 * 24 * 8
31 crit: $this > 60 * 60 * 24 * 8 * 2
33 - info: Blocklist file last update time. \
34 - The blocklist file has not been updated for a long time. \
35 - To rebuild the blocklist, run the [pihole -g] command.
32 + info: gravity.list (blocklist) file last update time
33 to: sysadmin
34
35 # Gravity file check (gravity.list).
@@ -47,9 +44,7 @@ component: Pi-hole
44 calc: $file_exists
45 crit: $this != 1
46 delay: up 2m down 5m
50 - info: Blocklist file state (0: exists, 1: not-exists). \
51 - The blocklist file does not exist. \
52 - To rebuild the blocklist, run the [pihole -g] command.
47 + info: gravity.list (blocklist) file existence state (0: exists, 1: not-exists)
48 to: sysadmin
49
50 # Pi-hole's ability to block unwanted domains.
@@ -65,7 +60,5 @@ component: Pi-hole
60 calc: $enabled
61 warn: $this != 1
62 delay: up 2m down 5m
68 - info: Unwanted domains blocking status (0: enabled, 1: disabled). \
69 - The ability of Pi-hole to block unwanted domains is disabled. \
70 - To fix, run the [pihole enable] command.
63 + info: unwanted domains blocking status (0: enabled, 1: disabled)
64 to: sysadmin
health/health.d/portcheck.conf
+2 -6
@@ -25,9 +25,7 @@ component: TCP endpoint
25 warn: $this >= 10 AND $this < 40
26 crit: $this >= 40
27 delay: down 5m multiplier 1.5 max 1h
28 - info: Average ratio of timeouts over the last 5 minutes. \
29 - Too many timeouts. The monitored endpoint is unreachable. \
30 - Most likely you are experiencing networking issues or the host/service is overloaded.
28 + info: average ratio of timeouts over the last 5 minutes
29 to: sysadmin
30
31 template: portcheck_connection_fails
@@ -42,7 +40,5 @@ component: TCP endpoint
40 warn: $this >= 10 AND $this < 40
41 crit: $this >= 40
42 delay: down 5m multiplier 1.5 max 1h
45 - info: Average ratio of failed connections over the last 5 minutes. \
46 - Too many failed connections. The monitored endpoint is unreachable. \
47 - Most likely the service is no longer running or access is denied by a firewall.
43 + info: average ratio of failed connections over the last 5 minutes
44 to: sysadmin
health/health.d/processes.conf
+1 -3
@@ -12,7 +12,5 @@ component: Processes
12 warn: $this > (($status >= $WARNING) ? (85) : (90))
13 crit: $this > (($status == $CRITICAL) ? (90) : (95))
14 delay: down 5m multiplier 1.5 max 1h
15 - info: Percentage of used PIDs. \
16 - High system process IDs (PID) space utilization. \
17 - If this value is 100% then the system can not start new processes.
15 + info: system process IDs (PID) space utilization
16 to: sysadmin
health/health.d/ram.conf
+5 -23
@@ -26,11 +26,7 @@ component: Memory
26 warn: $this > (($status >= $WARNING) ? (80) : (90))
27 crit: $this > (($status == $CRITICAL) ? (90) : (98))
28 delay: down 15m multiplier 1.5 max 1h
29 - info: Percentage of used RAM. \
30 - High RAM utilization. \
31 - It may affect the performance of applications. \
32 - If there is no swap space available, OOM Killer can start killing processes. \
33 - You might want to check per-process memory usage to find the top consumers.
29 + info: system memory utilization
30 to: sysadmin
31
32 alarm: ram_available
@@ -46,11 +42,7 @@ component: Memory
42 warn: $this < (($status >= $WARNING) ? (15) : (10))
43 crit: $this < (($status == $CRITICAL) ? (10) : ( 5))
44 delay: down 15m multiplier 1.5 max 1h
49 - info: Percentage of an estimated amount of RAM available for userspace processes, without causing swapping. \
50 - Low amount of available memory. \
51 - It may affect the performance of applications. \
52 - If there is no swap space available, OOM Killer can start killing processes. \
53 - You might want to check per-process memory usage to find the top consumers.
45 + info: percentage of estimated amount of RAM available for userspace processes, without causing swapping
46 to: sysadmin
47
48 alarm: oom_kill
@@ -63,9 +55,7 @@ component: Memory
55 warn: $this > 0
56 delay: down 10m
57 host labels: _is_k8s_node = false
66 - info: Number of out of memory kills in the last 30 minutes. \
67 - Some processes got killed by OOM Killer. \
68 - To fix, decrease memory usage, adjust memory limits for cgroups, or add more memory/swap to the system.
58 + info: number of out of memory kills in the last 30 minutes
59 to: sysadmin
60
61 ## FreeBSD
@@ -82,11 +72,7 @@ component: Memory
72 warn: $this > (($status >= $WARNING) ? (80) : (90))
73 crit: $this > (($status == $CRITICAL) ? (90) : (98))
74 delay: down 15m multiplier 1.5 max 1h
85 - info: Percentage of used RAM. \
86 - High RAM utilization. \
87 - It may affect the performance of applications. \
88 - If there is no swap space available, OOM Killer can start killing processes. \
89 - You might want to check per-process memory usage to find the top consumers.
75 + info: system memory utilization
76 to: sysadmin
77
78 alarm: ram_available
@@ -102,9 +88,5 @@ component: Memory
88 warn: $this < (($status >= $WARNING) ? (15) : (10))
89 crit: $this < (($status == $CRITICAL) ? (10) : ( 5))
90 delay: down 15m multiplier 1.5 max 1h
105 - info: Percentage of an estimated amount of RAM available for userspace processes without causing swapping. \
106 - Low amount of available memory. \
107 - It may affect the performance of applications. \
108 - If there is no swap space available, OOM Killer can start killing processes. \
109 - You might want to check per-process memory usage to find the top consumers.
91 + info: percentage of estimated amount of RAM available for userspace processes, without causing swapping
92 to: sysadmin
health/health.d/redis.conf
+2 -7
@@ -8,9 +8,7 @@ component: Redis
8 every: 10s
9 crit: $rdb_last_bgsave_status != 0
10 units: ok/failed
11 - info: Status of the last RDB save operation (0: ok, 1: error). \
12 - Redis failed to save the RDB snapshot on disk. \
13 - Check Redis logs for details about the error.
11 + info: status of the last RDB save operation (0: ok, 1: error)
12 delay: down 5m multiplier 1.5 max 1h
13 to: dba
14
@@ -24,9 +22,6 @@ component: Redis
22 warn: $rdb_bgsave_in_progress > 600
23 crit: $rdb_bgsave_in_progress > 1200
24 units: seconds
27 - info: Duration of the ongoing RDB save operation. \
28 - Saving RDB snapshot on disk is taking too long. \
29 - Possible reasons are that the dataset size is big or the lack of CPU resources. \
30 - It may result in Redis stopping serving clients for some millisecond or even for one second.
25 + info: duration of the on-going RDB save operation
26 delay: down 5m multiplier 1.5 max 1h
27 to: dba
health/health.d/retroshare.conf
+1 -2
@@ -12,6 +12,5 @@ component: Retroshare
12 warn: $this < (($status >= $WARNING) ? (120) : (100))
13 crit: $this < (($status == $CRITICAL) ? (10) : (1))
14 delay: up 0 down 15m multiplier 1.5 max 1h
15 - info: Number of DHT peers. \
16 - A low number of DHT peers.
15 + info: number of DHT peers
16 to: sysadmin
health/health.d/riakkv.conf
+8 -12
@@ -9,10 +9,7 @@ component: Riak KV
9 units: state machines
10 every: 10s
11 warn: $list_fsm_active > 0
12 - info: Number of currently running list keys finite state machines. \
13 - There are active list keys FSMs. \
14 - A key listing in Riak is a very expensive operation, and should not be used in production \
15 - as it will affect the performance of the cluster and not scale well.
12 + info: number of currently running list keys finite state machines
13 to: dba
14
15
@@ -41,9 +38,9 @@ component: Riak KV
38 every: 10s
39 warn: ($this > ($riakkv_1h_kv_get_mean_latency * 2) )
40 crit: ($this > ($riakkv_1h_kv_get_mean_latency * 3) )
44 - info: Average GET request processing time over the last 3 minutes, compared to the average over the last hour. \
45 - GET request processing time has increased significantly. \
46 - It indicates that the server is overloaded.
41 + info: average time between reception of client GET request and \
42 + subsequent response to the client over the last 3 minutes, \
43 + compared to the average over the last hour
44 delay: down 5m multiplier 1.5 max 1h
45 to: dba
46
@@ -71,9 +68,9 @@ component: Riak KV
68 every: 10s
69 warn: ($this > ($riakkv_1h_kv_put_mean_latency * 2) )
70 crit: ($this > ($riakkv_1h_kv_put_mean_latency * 3) )
74 - info: Average PUT request processing time over the last 3 minutes, compared to the average over the last hour. \
75 - PUT request processing time has increased significantly. \
76 - It indicates that the server is overloaded.
71 + info: average time between reception of client PUT request and \
72 + subsequent response to the client over the last 3 minutes, \
73 + compared to the average over the last hour
74 delay: down 5m multiplier 1.5 max 1h
75 to: dba
76
@@ -92,6 +89,5 @@ component: Riak KV
89 every: 10s
90 warn: $this > 10000
91 crit: $this > 100000
95 - info: Number of processes running in the Erlang VM. \
96 - The number of processes is high, you may see performance degradation due to scheduling overhead.
92 + info: number of processes running in the Erlang VM
93 to: dba
health/health.d/softnet.conf
+8 -15
@@ -15,11 +15,8 @@ component: Network
15 every: 10s
16 warn: $this > (($status >= $WARNING) ? (0) : (10))
17 delay: down 1h multiplier 1.5 max 2h
18 - info: Average number of dropped packets in the last minute due to exceeded netdev backlog queue. \
19 - Netdev backlog queue is full. \
20 - It results in packets received by the interface and not yet processed by the destined subsystem or \
21 - userland application being dropped. \
22 - To fix, increase the [net.core.netdev_max_backlog] limit.
18 + info: average number of dropped packets in the last minute \
19 + due to exceeded net.core.netdev_max_backlog
20 to: sysadmin
21
22 alarm: 1min_netdev_budget_ran_outs
@@ -34,11 +31,9 @@ component: Network
31 every: 10s
32 warn: $this > (($status >= $WARNING) ? (0) : (10))
33 delay: down 1h multiplier 1.5 max 2h
37 - info: Average number of times ksoftirq ran out of netdev_budget over the last minute. \
38 - SoftIRQ was unable to process all packets available before the CPU budget was exhausted. \
39 - It may result in packet drops and is typically caused by a high-bandwidth interface adding more packets \
40 - to the receive buffer than can be processed during NAPI polling. \
41 - To fix, increase the [net.core.netdev_budget] limit.
34 + info: average number of times ksoftirq ran out of sysctl net.core.netdev_budget or \
35 + net.core.netdev_budget_usecs with work remaining over the last minute \
36 + (this can be a cause for dropped packets)
37 to: silent
38
39 alarm: 10min_netisr_backlog_exceeded
@@ -53,9 +48,7 @@ component: Network
48 every: 10s
49 warn: $this > (($status >= $WARNING) ? (0) : (10))
50 delay: down 1h multiplier 1.5 max 2h
56 - info: Average number of dropped packets in the last minute due to exceeded netisr queue length. \
57 - Netisr queue is full. \
58 - It results in packets received by the interface and not yet processed by the destined subsystem or \
59 - userland application being dropped. \
60 - To fix, increase the [net.route.netisr_maxqlen] limit.
51 + info: average number of drops in the last minute \
52 + due to exceeded sysctl net.route.netisr_maxqlen \
53 + (this can be a cause for dropped packets)
54 to: sysadmin
health/health.d/swap.conf
+3 -8
@@ -15,8 +15,7 @@ component: Memory
15 every: 1m
16 warn: $this > (($status >= $WARNING) ? (20) : (30))
17 delay: down 15m multiplier 1.5 max 1h
18 - info: Percentage of the system RAM swapped in the last 30 minutes. \
19 - A lot of system RAM was swapped.
18 + info: percentage of the system RAM swapped in the last 30 minutes
19 to: sysadmin
20
21 alarm: used_swap
@@ -26,15 +25,11 @@ component: Memory
25 component: Memory
26 os: linux freebsd
27 hosts: *
29 - calc: ($used + $free) > 0 ? ($used * 100 / ($used + $free)) : 0
28 + calc: $used * 100 / ( $used + $free )
29 units: %
30 every: 10s
31 warn: $this > (($status >= $WARNING) ? (80) : (90))
32 crit: $this > (($status == $CRITICAL) ? (90) : (98))
33 delay: up 30s down 15m multiplier 1.5 max 1h
35 - info: Percentage of used swap. \
36 - High swap memory utilization. \
37 - It may be a sign that the system is experiencing memory pressure, which can affect the performance of your system. \
38 - If there is no RAM available, OOM Killer can start killing processes. \
39 - You might want to check per-process swap usage to find the top consumers.
34 + info: swap memory utilization
35 to: sysadmin
health/health.d/synchronization.conf
+2 -3
@@ -6,8 +6,7 @@
6 every: 1m
7 warn: $this > 6
8 delay: up 1m down 10m multiplier 1.5 max 1h
9 - info: Number of sync() system calls. \
10 - A high number of sync() system calls. \
11 - Every call is very expensive because it causes all pending modifications to filesystem metadata and \
9 + info: number of sync() system calls. \
10 + Every call causes all pending modifications to filesystem metadata and \
11 cached file data to be written to the underlying filesystems.
12 to: sysadmin
health/health.d/systemdunits.conf
+10 -40
@@ -12,10 +12,7 @@ component: Systemd units
12 every: 10s
13 warn: $this != nan AND $this == 5
14 delay: down 5m multiplier 1.5 max 1h
15 - info: One or more of the systemd service units are in the failed state. \
16 - It means that the service failed in some way \
17 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
18 - To see the details use [systemctl status SERVICENAME].
15 + info: one or more systemd service units are in the failed state
16 to: sysadmin
17
18 ## Socket units
@@ -29,10 +26,7 @@ component: Systemd units
26 every: 10s
27 warn: $this != nan AND $this == 5
28 delay: down 5m multiplier 1.5 max 1h
32 - info: One or more of the systemd socket units are in the failed state. \
33 - It means that the service failed in some way \
34 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
35 - To see the details use [systemctl status SERVICENAME].
29 + info: one or more systemd socket units are in the failed state
30 to: sysadmin
31
32 ## Target units
@@ -46,10 +40,7 @@ component: Systemd units
40 every: 10s
41 warn: $this != nan AND $this == 5
42 delay: down 5m multiplier 1.5 max 1h
49 - info: One or more of the systemd target units are in the failed state. \
50 - It means that the service failed in some way \
51 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
52 - To see the details use [systemctl status SERVICENAME].
43 + info: one or more systemd target units are in the failed state
44 to: sysadmin
45
46 ## Path units
@@ -63,10 +54,7 @@ component: Systemd units
54 every: 10s
55 warn: $this != nan AND $this == 5
56 delay: down 5m multiplier 1.5 max 1h
66 - info: One or more of the systemd path units are in the failed state. \
67 - It means that the service failed in some way \
68 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
69 - To see the details use [systemctl status SERVICENAME].
57 + info: one or more systemd path units are in the failed state
58 to: sysadmin
59
60 ## Device units
@@ -80,10 +68,7 @@ component: Systemd units
68 every: 10s
69 warn: $this != nan AND $this == 5
70 delay: down 5m multiplier 1.5 max 1h
83 - info: One or more of the systemd device units are in the failed state. \
84 - It means that the service failed in some way \
85 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
86 - To see the details use [systemctl status SERVICENAME].
71 + info: one or more the systemd device units are in the failed state
72 to: sysadmin
73
74 ## Mount units
@@ -97,10 +82,7 @@ component: Systemd units
82 every: 10s
83 warn: $this != nan AND $this == 5
84 delay: down 5m multiplier 1.5 max 1h
100 - info: One or more of the systemd mount units are in the failed state. \
101 - It means that the service failed in some way \
102 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
103 - To see the details use [systemctl status SERVICENAME].
85 + info: one or more the systemd mount units are in the failed state
86 to: sysadmin
87
88 ## Automount units
@@ -114,10 +96,7 @@ component: Systemd units
96 every: 10s
97 warn: $this != nan AND $this == 5
98 delay: down 5m multiplier 1.5 max 1h
117 - info: One or more of the systemd automount units are in the failed state. \
118 - It means that the service failed in some way \
119 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
120 - To see the details use [systemctl status SERVICENAME].
99 + info: one or more systemd automount units are in the failed state
100 to: sysadmin
101
102 ## Swap units
@@ -131,10 +110,7 @@ component: Systemd units
110 every: 10s
111 warn: $this != nan AND $this == 5
112 delay: down 5m multiplier 1.5 max 1h
134 - info: One or more of the systemd swap units are in the failed state. \
135 - It means that the service failed in some way \
136 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
137 - To see the details use [systemctl status SERVICENAME].
113 + info: one or more systemd swap units are in the failed state
114 to: sysadmin
115
116 ## Scope units
@@ -148,10 +124,7 @@ component: Systemd units
124 every: 10s
125 warn: $this != nan AND $this == 5
126 delay: down 5m multiplier 1.5 max 1h
151 - info: One or more of the systemd scope units are in the failed state. \
152 - It means that the service failed in some way \
153 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
154 - To see the details use [systemctl status SERVICENAME].
127 + info: one or more systemd scope units are in the failed state
128 to: sysadmin
129
130 ## Slice units
@@ -165,8 +138,5 @@ component: Systemd units
138 every: 10s
139 warn: $this != nan AND $this == 5
140 delay: down 5m multiplier 1.5 max 1h
168 - info: One or more of the systemd scope units are in the failed state. \
169 - It means that the service failed in some way \
170 - (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \
171 - To see the details use [systemctl status SERVICENAME].
141 + info: one or more systemd slice units are in the failed state
142 to: sysadmin
health/health.d/tcp_conn.conf
+1 -3
@@ -18,7 +18,5 @@ component: Network
18 warn: $this > (($status >= $WARNING ) ? ( 60 ) : ( 80 ))
19 crit: $this > (($status == $CRITICAL) ? ( 80 ) : ( 90 ))
20 delay: up 0 down 5m multiplier 1.5 max 1h
21 - info: Percentage of used IPv4 TCP connections. \
22 - High IPv4 TCP connections utilization. \
23 - If this value is 100% then the system is no longer able to establish new TCP connections.
21 + info: IPv4 TCP connections utilization
22 to: sysadmin
health/health.d/tcp_listen.conf
+5 -15
@@ -31,10 +31,7 @@ component: Network
31 warn: $this > 1
32 crit: $this > (($status == $CRITICAL) ? (1) : (5))
33 delay: up 0 down 5m multiplier 1.5 max 1h
34 - info: Average number of overflows in the TCP accept queue over the last minute. \
35 - TCP accept queue is full. \
36 - It may indicate SYN floods. \
37 - To increase the queue length, adjust the [net.ipv4.tcp_max_syn_backlog] limit.
34 + info: average number of overflows in the TCP accept queue over the last minute
35 to: sysadmin
36
37 # THIS IS TOO GENERIC
@@ -52,10 +49,7 @@ component: Network
49 warn: $this > 1
50 crit: $this > (($status == $CRITICAL) ? (1) : (5))
51 delay: up 0 down 5m multiplier 1.5 max 1h
55 - info: Average number of dropped packets in the TCP accept queue over the last minute. \
56 - The system is dropping incoming TCP connections. \
57 - It can indicate accept queue overflow, out of memory, security issues, no route to a destination, etc. \
58 - To fix overflows, increase the [net.ipv4.tcp_max_syn_backlog] limit.
52 + info: average number of dropped packets in the TCP accept queue over the last minute
53 to: sysadmin
54
55
@@ -80,10 +74,8 @@ component: Network
74 warn: $this > 1
75 crit: $this > (($status == $CRITICAL) ? (0) : (5))
76 delay: up 10 down 5m multiplier 1.5 max 1h
83 - info: Average number of SYN requests was dropped due to the full TCP SYN queue over the last minute. \
84 - TCP SYN queue is full. The system is dropping incoming TCP SYN requests. \
85 - It may indicate SYN floods. \
86 - If you can determine that the traffic is legitimate, consider enabling SYN cookies.
77 + info: average number of SYN requests was dropped due to the full TCP SYN queue over the last minute \
78 + (SYN cookies were not enabled)
79 to: sysadmin
80
81 alarm: 1m_tcp_syn_queue_cookies
@@ -99,8 +91,6 @@ component: Network
91 warn: $this > 1
92 crit: $this > (($status == $CRITICAL) ? (0) : (5))
93 delay: up 10 down 5m multiplier 1.5 max 1h
102 - info: Average number of sent SYN cookies due to the full TCP SYN queue over the last minute. \
103 - TCP SYN queue is full. \
104 - It may indicate SYN floods.
94 + info: average number of sent SYN cookies due to the full TCP SYN queue over the last minute
95 to: sysadmin
96
health/health.d/tcp_mem.conf
+1 -5
@@ -19,9 +19,5 @@ component: Network
19 warn: ${mem} > (($status >= $WARNING ) ? ( ${tcp_mem_pressure} * 0.8 ) : ( ${tcp_mem_pressure} ))
20 crit: ${mem} > (($status == $CRITICAL ) ? ( ${tcp_mem_pressure} ) : ( ${tcp_mem_high} * 0.9 ))
21 delay: up 0 down 5m multiplier 1.5 max 1h
22 - info: Percentage of used TCP memory. \
23 - High TCP memory utilization. \
24 - If the TCP protocol uses more memory than the limit, the system will throw an OOM error and \
25 - some applications become unresponsive. \
26 - To increase the limit, adjust [net.ipv4.tcp_rmem] and [net.ipv4.tcp_wmem].
22 + info: TCP memory utilization
23 to: sysadmin
health/health.d/tcp_orphans.conf
+1 -4
@@ -20,8 +20,5 @@ component: Network
20 warn: $this > (($status >= $WARNING ) ? ( 20 ) : ( 25 ))
21 crit: $this > (($status == $CRITICAL) ? ( 25 ) : ( 50 ))
22 delay: up 0 down 5m multiplier 1.5 max 1h
23 - info: Percentage of used orphan IPv4 TCP sockets. \
24 - High orphan IPv4 TCP sockets utilization. \
25 - When the limit is exceeded, orphaned connections (not attached to any user filehandle) are reset immediately. \
26 - To increase the limit, adjust [net.ipv4.tcp_max_orphans].
23 + info: orphan IPv4 TCP sockets utilization
24 to: sysadmin
health/health.d/tcp_resets.conf
+7 -7
@@ -29,9 +29,10 @@ component: Network
29 warn: $this > ((($1m_ipv4_tcp_resets_sent < 5)?(5):($1m_ipv4_tcp_resets_sent)) * (($status >= $WARNING) ? (1) : (20)))
30 delay: up 20s down 60m multiplier 1.2 max 2h
31 options: no-clear-notification
32 - info: Average number of sent TCP RESETS over the last 10 seconds. \
33 - A high number of sent TCP RESETS. \
34 - This can indicate a port scan or that a service running on the system has crashed.
32 + info: average number of sent TCP RESETS over the last 10 seconds. \
33 + This can indicate a port scan, \
34 + or that a service running on this host has crashed. \
35 + Netdata will not send a clear notification for this alarm.
36 to: sysadmin
37
38 # -----------------------------------------------------------------------------
@@ -62,8 +63,7 @@ component: Network
63 warn: $this > ((($1m_ipv4_tcp_resets_received < 5)?(5):($1m_ipv4_tcp_resets_received)) * (($status >= $WARNING) ? (1) : (10)))
64 delay: up 20s down 60m multiplier 1.2 max 2h
65 options: no-clear-notification
65 - info: Average number of received TCP RESETS over the last 10 seconds. \
66 - A high number of received TCP RESETS. \
67 - This can indicate that the system is trying to establish a connection \
68 - to a server port on which no process is listening.
66 + info: average number of received TCP RESETS over the last 10 seconds. \
67 + This can be an indication that a service this host needs has crashed. \
68 + Netdata will not send a clear notification for this alarm.
69 to: sysadmin
health/health.d/timex.conf
+1 -3
@@ -13,7 +13,5 @@ component: Clock
13 every: 10s
14 warn: $system.uptime.uptime > 17 * 60 AND $this == 0
15 delay: down 5m
16 - info: The system time is not synchronized to a reliable server. \
17 - It is strongly recommended having the clock in sync with NTP servers because otherwise, \
18 - it leads to unpredictable problems that are difficult to debug.
16 + info: the system time is not synchronized to a reliable server
17 to: silent
health/health.d/udp_errors.conf
+2 -8
@@ -15,10 +15,7 @@ component: Network
15 units: errors
16 every: 10s
17 warn: $this > (($status >= $WARNING) ? (0) : (10))
18 - info: Average number of UDP receive buffer errors over the last minute. \
19 - UDP receive buffer is full. \
20 - The system is dropping incoming UDP packets. \
21 - To increase the limit, adjust [net.ipv4.udp_rmem].
18 + info: average number of UDP receive buffer errors over the last minute
19 delay: up 1m down 60m multiplier 1.2 max 2h
20 to: sysadmin
21
@@ -36,9 +33,6 @@ component: Network
33 units: errors
34 every: 10s
35 warn: $this > (($status >= $WARNING) ? (0) : (10))
39 - info: Average number of UDP send buffer errors over the last minute. \
40 - UDP send buffer is full or no kernel memory available. \
41 - The system is dropping outgoing UDP packets. \
42 - To increase the limit, adjust [net.ipv4.udp_wmem].
36 + info: average number of UDP send buffer errors over the last minute
37 delay: up 1m down 60m multiplier 1.2 max 2h
38 to: sysadmin
health/health.d/unbound.conf
+2 -8
@@ -11,10 +11,7 @@ component: Unbound
11 every: 10s
12 warn: $this > 5
13 delay: up 10 down 5m multiplier 1.5 max 1h
14 - info: Number of overwritten queries. \
15 - Request queue is full. Unbound is overwriting old queued requests. \
16 - It can indicate a Denial of Service attack. \
17 - To increase the queue length, adjust [num-queries-per-thread].
14 + info: number of overwritten queries in the request-list
15 to: sysadmin
16
17 template: unbound_request_list_dropped
@@ -27,8 +24,5 @@ component: Unbound
24 every: 10s
25 warn: $this > 0
26 delay: up 10 down 5m multiplier 1.5 max 1h
30 - info: Number of dropped queries. \
31 - Request queue is full. Unbound is dropping new incoming requests. \
32 - It can indicate a Denial of Service attack. \
33 - To increase the queue length, adjust [num-queries-per-thread].
27 + info: number of dropped queries in the request-list
28 to: sysadmin
health/health.d/vcsa.conf
+16 -32
@@ -17,11 +17,8 @@ component: VMware vCenter
17 warn: ($this == 1) || ($this == 2)
18 crit: $this == 3
19 delay: down 1m multiplier 1.5 max 1h
20 - info: Overall system health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \
21 - Overall system status is unhealthy. \
22 - It indicates that one or more components might become overloaded soon (yellow), \
23 - or might be degraded (orange), or might be in an unusable status and the appliance might become unresponsive soon (red). \
24 - To view the details, check the VCSA Health Messages pane.
20 + info: overall system health status \
21 + (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)
22 to: sysadmin
23
24 # Components health:
@@ -42,10 +39,8 @@ component: VMware vCenter
39 warn: $this == 1
40 crit: ($this == 2) || ($this == 3)
41 delay: down 1m multiplier 1.5 max 1h
45 - info: Swap health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \
46 - The swap component is unhealthy. \
47 - It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \
48 - To view the details, check the VCSA Health Messages pane.
42 + info: swap health status \
43 + (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)
44 to: sysadmin
45
46 template: vcsa_storage_health
@@ -59,10 +54,8 @@ component: VMware vCenter
54 warn: $this == 1
55 crit: ($this == 2) || ($this == 3)
56 delay: down 1m multiplier 1.5 max 1h
62 - info: Storage health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \
63 - The storage component is unhealthy. \
64 - It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \
65 - To view the details, check the VCSA Health Messages pane.
57 + info: storage health status \
58 + (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)
59 to: sysadmin
60
61 template: vcsa_mem_health
@@ -76,10 +69,8 @@ component: VMware vCenter
69 warn: $this == 1
70 crit: ($this == 2) || ($this == 3)
71 delay: down 1m multiplier 1.5 max 1h
79 - info: Memory health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \
80 - The memory component is unhealthy. \
81 - It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \
82 - To view the details, check the VCSA Health Messages pane.
72 + info: memory health status \
73 + (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)
74 to: sysadmin
75
76 template: vcsa_load_health
@@ -93,10 +84,8 @@ component: VMware vCenter
84 warn: $this == 1
85 crit: ($this == 2) || ($this == 3)
86 delay: down 1m multiplier 1.5 max 1h
96 - info: Load health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \
97 - The load component is unhealthy. \
98 - It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \
99 - To view the details, check the VCSA Health Messages pane.
87 + info: load health status \
88 + (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)
89 to: sysadmin
90
91 template: vcsa_database_storage_health
@@ -110,10 +99,8 @@ component: VMware vCenter
99 warn: $this == 1
100 crit: ($this == 2) || ($this == 3)
101 delay: down 1m multiplier 1.5 max 1h
113 - info: Database storage health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \
114 - The database storage component is unhealthy. \
115 - It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \
116 - To view the details, check the VCSA Health Messages pane.
102 + info: database storage health status \
103 + (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)
104 to: sysadmin
105
106 template: vcsa_applmgmt_health
@@ -127,10 +114,8 @@ component: VMware vCenter
114 warn: $this == 1
115 crit: ($this == 2) || ($this == 3)
116 delay: down 1m multiplier 1.5 max 1h
130 - info: Applmgmt health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \
131 - The appliance management component is unhealthy. \
132 - It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \
133 - To view the details, check the VCSA Health Messages pane.
117 + info: applmgmt health status \
118 + (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)
119 to: sysadmin
120
121
@@ -151,7 +136,6 @@ component: VMware vCenter
136 warn: $this == 4
137 crit: $this == 3
138 delay: down 1m multiplier 1.5 max 1h
154 - info: Software updates availability status (-1: unknown, 0: green, 2: orange, 3: red, 4: grey). \
155 - Software updates might be available. \
156 - It indicates that security patches might be available (red) or an error retrieving information on software updates (grey).
139 + info: software updates availability status \
140 + (-1: unknown, 0: green, 2: orange, 3: red, 4: grey)
141 to: sysadmin
health/health.d/vernemq.conf
+5 -17
@@ -26,10 +26,7 @@ component: VerneMQ
26 every: 1m
27 warn: $this > (($status >= $WARNING) ? (0) : (5))
28 delay: up 2m down 5m multiplier 1.5 max 2h
29 - info: Number of dropped messages due to full queues in the last minute. \
30 - Message queues are full. VerneMQ is dropping messages. \
31 - This can indicate that consumers or VerneMQ are too slow, or publishers are too fast. \
32 - To increase the queue length, adjust [max_online_messages].
29 + info: number of dropped messaged due to full queues in the last minute
30 to: sysadmin
31
32 template: vernemq_queue_message_expired
@@ -42,8 +39,7 @@ component: VerneMQ
39 every: 1m
40 warn: $this > (($status >= $WARNING) ? (0) : (5))
41 delay: up 2m down 5m multiplier 1.5 max 2h
45 - info: Number of messages which expired before delivery in the last minute. \
46 - Too many messages were undelivered due to expiration.
42 + info: number of messages which expired before delivery in the last minute
43 to: sysadmin
44
45 template: vernemq_queue_message_unhandled
@@ -56,8 +52,7 @@ component: VerneMQ
52 every: 1m
53 warn: $this > (($status >= $WARNING) ? (0) : (5))
54 delay: up 2m down 5m multiplier 1.5 max 2h
59 - info: Number of unhandled messages in the last minute. \
60 - Too many messages were undelivered due to connections with clean session=true.
55 + info: number of unhandled messages (connections with clean session=true) in the last minute
56 to: sysadmin
57
58 # Erlang VM
@@ -88,11 +83,7 @@ component: VerneMQ
83 every: 1m
84 warn: $this > 0
85 delay: up 5m down 5m multiplier 1.5 max 1h
91 - info: Amount of traffic dropped during communication with the cluster nodes in the last minute. \
92 - The outgoing cluster buffer is full. \
93 - VerneMQ is experiencing problems with inter-node message delivery. \
94 - Most likely a remote node is down or unreachable. \
95 - To increase the buffer size, adjust [outgoing_clustering_buffer_size].
86 + info: amount of traffic dropped during communication with the cluster nodes in the last minute
87 to: sysadmin
88
89 template: vernemq_netsplits
@@ -105,10 +96,7 @@ component: VerneMQ
96 every: 10s
97 warn: $this > 0
98 delay: down 5m multiplier 1.5 max 2h
108 - info: Number of detected netsplits in the last minute. \
109 - Split-brain detected. \
110 - That is mostly the result of a failure of one or more network devices \
111 - resulting in a cluster where nodes can no longer reach each other.
99 + info: number of detected netsplits (split brain situation) in the last minute
100 to: sysadmin
101
102 # Unsuccessful CONNACK
health/health.d/whoisquery.conf
+1 -4
@@ -9,8 +9,5 @@ component: WHOIS
9 every: 60s
10 warn: $this < $days_until_expiration_warning*24*60*60
11 crit: $this < $days_until_expiration_critical*24*60*60
12 - info: Time until the domain name registration expires. \
13 - The domain name registration expires soon. \
14 - If you do not renew the domain it will be deactivated soon. \
15 - Consider scheduling the renewal.
12 + info: time until the domain name registration expires
13 to: webmaster
health/health.d/wmi.conf
+8 -16
@@ -14,8 +14,7 @@ component: CPU
14 warn: $this > (($status >= $WARNING) ? (75) : (85))
15 crit: $this > (($status == $CRITICAL) ? (85) : (95))
16 delay: down 15m multiplier 1.5 max 1h
17 - info: Average CPU utilization over the last 10 minutes. \
18 - High CPU utilization.
17 + info: average CPU utilization over the last 10 minutes
18 to: sysadmin
19
20
@@ -34,8 +33,7 @@ component: Memory
33 warn: $this > (($status >= $WARNING) ? (80) : (90))
34 crit: $this > (($status == $CRITICAL) ? (90) : (98))
35 delay: down 15m multiplier 1.5 max 1h
37 - info: Percentage of used RAM memory. \
38 - High RAM utilization.
36 + info: memory utilization
37 to: sysadmin
38
39 template: wmi_swap_in_use
@@ -51,8 +49,7 @@ component: Memory
49 warn: $this > (($status >= $WARNING) ? (80) : (90))
50 crit: $this > (($status == $CRITICAL) ? (90) : (98))
51 delay: down 15m multiplier 1.5 max 1h
54 - info: Percentage of used swap. \
55 - High swap memory utilization.
52 + info: swap memory utilization
53 to: sysadmin
54
55
@@ -71,8 +68,7 @@ component: Network
68 every: 1m
69 warn: $this >= 5
70 delay: down 1h multiplier 1.5 max 2h
74 - info: Number of inbound discarded packets for the network interface in the last 10 minutes. \
75 - The network interface discarded many inbound packets.
71 + info: number of inbound discarded packets for the network interface in the last 10 minutes
72 to: sysadmin
73
74 template: wmi_outbound_packets_discarded
@@ -88,8 +84,7 @@ component: Network
84 every: 1m
85 warn: $this >= 5
86 delay: down 1h multiplier 1.5 max 2h
91 - info: Number of outbound discarded packets for the network interface in the last 10 minutes. \
92 - The network interface discarded many outbound packets.
87 + info: number of outbound discarded packets for the network interface in the last 10 minutes
88 to: sysadmin
89
90 template: wmi_inbound_packets_errors
@@ -105,8 +100,7 @@ component: Network
100 every: 1m
101 warn: $this >= 5
102 delay: down 1h multiplier 1.5 max 2h
108 - info: Number of inbound errors for the network interface in the last 10 minutes. \
109 - The network interface received many bad packets.
103 + info: number of inbound errors for the network interface in the last 10 minutes
104 to: sysadmin
105
106 template: wmi_outbound_packets_errors
@@ -122,8 +116,7 @@ component: Network
116 every: 1m
117 warn: $this >= 5
118 delay: down 1h multiplier 1.5 max 2h
125 - info: Number of outbound errors for the network interface in the last 10 minutes. \
126 - The network interface experienced many transmit problems.
119 + info: number of outbound errors for the network interface in the last 10 minutes
120 to: sysadmin
121
122
@@ -142,6 +135,5 @@ component: Disk
135 warn: $this > (($status >= $WARNING) ? (80) : (90))
136 crit: $this > (($status == $CRITICAL) ? (90) : (98))
137 delay: down 15m multiplier 1.5 max 1h
145 - info: Percentage of used disk space. \
146 - High disk space utilization.
138 + info: disk space utilization
139 to: sysadmin
health/health.d/x509check.conf
+2 -5
@@ -9,9 +9,7 @@ component: x509 certificates
9 every: 60s
10 warn: $this < $days_until_expiration_warning*24*60*60
11 crit: $this < $days_until_expiration_critical*24*60*60
12 - info: Time until the X.509 certificate expires. \
13 - The X.509 certificate will expire soon. \
14 - Renew the certificate or create a new one.
12 + info: time until x509 certificate expires
13 to: webmaster
14
15 template: x509check_revocation_status
@@ -22,6 +20,5 @@ component: x509 certificates
20 calc: $revoked
21 every: 60s
22 crit: $this != nan AND $this != 0
25 - info: X.509 certificate revocation status (0: revoked, 1: valid). \
26 - The X.509 certificate has been revoked.
23 + info: x509 certificate revocation status (0: revoked, 1: valid)
24 to: webmaster