07946d9f0 · cryptotaxi247/netdata-1

health/health.d/apcupsd.conf

+8 -2

@@ -13,7 +13,11 @@ component: UPS

13 warn: $this > (($status >= $WARNING) ? (70) : (80))

14 crit: $this > (($status == $CRITICAL) ? (85) : (95))

15 delay: down 10m multiplier 1.5 max 1h

16 - info: average UPS load over the last 10 minutes

16 + info: Average UPS load over the last 10 minutes. \

17 + High UPS load. \

18 + It may result in either your UPS transferring to bypass or shutting down \

19 + as a self-protection and safety measure due to overload. \

20 + You should remove some attached equipment from the UPS.

21 to: sitemgr

22

23 # Discussion in https://github.com/netdata/netdata/pull/3928:

@@ -31,7 +35,9 @@ component: UPS

35 warn: $this < 100

36 crit: $this < (($status == $CRITICAL) ? (60) : (50))

37 delay: down 10m multiplier 1.5 max 1h

34 - info: average UPS charge over the last minute

38 + info: Average UPS charge over the last minute. \

39 + The UPS is running on battery power. It will shut down if external power is not restored. \

40 + You should prepare any attached equipment for the shutdown.

41 to: sitemgr

42

43 template: apcupsd_last_collected_secs

health/health.d/backend.conf

+10 -3

@@ -9,7 +9,8 @@ component: Exporting engine

9 every: 1m

10 warn: $this > 0

11 delay: down 5m multiplier 1.5 max 1h

12 - info: the backends subsystem is deprecated and will be removed soon. Migrate your configuration to exporting.conf.

12 + info: The backends subsystem is deprecated and will be removed soon. \

13 + Migrate your configuration to exporting.conf.

14 to: sysadmin

15

16 # make sure we are sending data to backend

@@ -25,7 +26,10 @@ component: Exporting engine

26 warn: $this > (($status >= $WARNING) ? ($update_every) : ( 5 * $update_every))

27 crit: $this > (($status == $CRITICAL) ? ($update_every) : (60 * $update_every))

28 delay: down 5m multiplier 1.5 max 1h

28 - info: number of seconds since the last successful buffering of backend data

29 + info: Number of seconds since the last successful buffering of backend data. \

30 + The backend subsystem failed to buffer metrics for a while. Some metrics are lost while exporting. \

31 + It indicates that the backend destination is down or unreachable. \

32 + Short-term network availability issues might be fixed by increasing [buffer on failures] in netdata.conf.

33 to: dba

34

35 alarm: backend_metrics_sent

@@ -38,5 +42,8 @@ component: Exporting engine

42 every: 10s

43 warn: $this != 100

44 delay: down 5m multiplier 1.5 max 1h

41 - info: percentage of metrics sent to the backend server

45 + info: Percentage of metrics sent to the backend server. \

46 + The backends subsystem failed to send all metrics. Some metrics are lost while exporting. \

47 + It indicates that the backend destination is down or unreachable. \

48 + Short-term network availability issues might be fixed by increasing [buffer on failures] in netdata.conf.

49 to: dba

health/health.d/bcache.conf

+6 -5

@@ -9,9 +9,9 @@ component: Disk

9 every: 1m

10 warn: $this > 0

11 delay: up 2m down 1h multiplier 1.5 max 2h

12 - info: number of times data was read from the cache, \

13 - the bucket was reused and invalidated in the last 10 minutes \

14 - (when this occurs the data is reread from the backing device)

12 + info: Number of bcache read races in the last minute. \

13 + The bucket was reused and invalidated while reading from the cache. \

14 + When this occurs the data is reread from the backing device.

15 to: sysadmin

16

17 template: bcache_cache_dirty

@@ -25,6 +25,7 @@ component: Disk

25 warn: $this > ( ($status >= $WARNING ) ? ( 70 ) : ( 90 ) )

26 crit: $this > ( ($status == $CRITICAL) ? ( 90 ) : ( 95 ) )

27 delay: up 1m down 1h multiplier 1.5 max 2h

28 - info: percentage of cache space used for dirty data and metadata \

29 - (this usually means your SSD cache is too small)

28 + info: Percentage of cache space used for dirty data and metadata. \

29 + High block cache utilization by dirty data and metadata. \

30 + This usually means your SSD cache is too small.

31 to: sysadmin

health/health.d/beanstalkd.conf

+5 -3

@@ -11,9 +11,11 @@ component: Beanstalk

11 warn: $this > 0

12 crit: $this > 10

13 delay: up 0 down 5m multiplier 1.2 max 1h

14 - info: number of buried jobs across all tubes. \

15 - You need to manually kick them so they can be processed. \

16 - Presence of buried jobs in a tube does not affect new jobs.

14 + info: Number of buried jobs across all tubes. \

15 + There are buried jobs. \

16 + It usually happens if something goes wrong while the consumer processes it. \

17 + The presence of buried jobs in a tube does not affect new jobs. \

18 + You need to manually kick the jobs, so they can be processed.

19 to: sysadmin

20

21 # get the number of buried jobs per queue

health/health.d/btrfs.conf

+17 -4

@@ -13,7 +13,10 @@ component: File system

13 warn: $this > (($status >= $WARNING) ? (90) : (95))

14 crit: $this > (($status == $CRITICAL) ? (95) : (98))

15 delay: up 1m down 15m multiplier 1.5 max 1h

16 - info: percentage of allocated BTRFS physical disk space

16 + info: Percentage of allocated Btrfs physical disk space. \

17 + Most of the Btrfs physical disk space is allocated. \

18 + To fix it, first, try running Btrfs balance. \

19 + If that does not help, consider deleting snapshots or adding more physical space to the pool.

20 to: sysadmin

21

22 template: btrfs_data

@@ -30,7 +33,11 @@ component: File system

33 warn: $this > (($status >= $WARNING) ? (90) : (95)) && $btrfs_allocated > 98

34 crit: $this > (($status == $CRITICAL) ? (95) : (98)) && $btrfs_allocated > 98

35 delay: up 1m down 15m multiplier 1.5 max 1h

33 - info: utilization of BTRFS data space

36 + info: Percentage of used Btrfs data space. \

37 + High Btrfs data space utilization. \

38 + If there is enough unallocated memory, the data space will be automatically increased. \

39 + Otherwise, to fix, first try to run a balance. \

40 + If that does not help, you should add more physical space to the pool.

41 to: sysadmin

42

43 template: btrfs_metadata

@@ -47,7 +54,11 @@ component: File system

54 warn: $this > (($status >= $WARNING) ? (90) : (95)) && $btrfs_allocated > 98

55 crit: $this > (($status == $CRITICAL) ? (95) : (98)) && $btrfs_allocated > 98

56 delay: up 1m down 15m multiplier 1.5 max 1h

50 - info: utilization of BTRFS metadata space

57 + info: Percentage of used Btrfs metadata space. \

58 + High Btrfs metadata space utilization. \

59 + If there is enough unallocated memory, the metadata space will be automatically increased. \

60 + Otherwise, you may wish to run a balance on metadata only if you find you have very large amounts of \

61 + metadata space allocated, but unused.

62 to: sysadmin

63

64 template: btrfs_system

@@ -64,5 +75,7 @@ component: File system

75 warn: $this > (($status >= $WARNING) ? (90) : (95)) && $btrfs_allocated > 98

76 crit: $this > (($status == $CRITICAL) ? (95) : (98)) && $btrfs_allocated > 98

77 delay: up 1m down 15m multiplier 1.5 max 1h

67 - info: utilization of BTRFS system space

78 + info: Percentage of used Btrfs system space. \

79 + High Btrfs system space utilization. \

80 + If there is enough unallocated memory, the system space will be automatically increased.

81 to: sysadmin

health/health.d/ceph.conf

+3 -1

@@ -11,5 +11,7 @@ component: Ceph

11 warn: $this > (($status >= $WARNING ) ? (85) : (90))

12 crit: $this > (($status == $CRITICAL) ? (90) : (98))

13 delay: down 5m multiplier 1.2 max 1h

14 - info: cluster disk space utilization

14 + info: Percentage of used cluster disk space. \

15 + High disk space utilization. \

16 + To fix this, consider adding a node or removing unneeded data from the cluster.

17 to: sysadmin

health/health.d/cgroups.conf

+8 -2

@@ -14,7 +14,10 @@ component: CPU

14 warn: $this > (($status >= $WARNING) ? (75) : (85))

15 crit: $this > (($status == $CRITICAL) ? (85) : (95))

16 delay: down 15m multiplier 1.5 max 1h

17 - info: average cgroup CPU utilization over the last 10 minutes

17 + info: Average CPU utilization over the last 10 minutes. \

18 + High cgroup CPU utilization. \

19 + The system will throttle the cgroup CPU usage when the usage is over the limit. \

20 + To fix, increase the cgroup CPU limit.

21 to: sysadmin

22

23 template: cgroup_ram_in_use

@@ -30,5 +33,8 @@ component: Memory

33 warn: $this > (($status >= $WARNING) ? (80) : (90))

34 crit: $this > (($status == $CRITICAL) ? (90) : (98))

35 delay: down 15m multiplier 1.5 max 1h

33 - info: cgroup memory utilization

36 + info: Percentage of used memory. \

37 + High cgroup memory utilization. \

38 + OOM will kill some processes when the utilization reaches 100%. \

39 + To fix, increase the cgroup memory limit (if set).

40 to: sysadmin

health/health.d/cockroachdb.conf

+16 -5

@@ -12,7 +12,9 @@ component: CockroachDB

12 warn: $this > (($status >= $WARNING) ? (80) : (85))

13 crit: $this > (($status == $CRITICAL) ? (85) : (95))

14 delay: down 15m multiplier 1.5 max 1h

15 - info: storage capacity utilization

15 + info: Percentage of used storage space. \

16 + High storage capacity utilization. \

17 + To fix, increase the space available for CockroachDB data.

18 to: dba

19

20 template: cockroachdb_used_usable_storage_capacity

@@ -26,7 +28,9 @@ component: CockroachDB

28 warn: $this > (($status >= $WARNING) ? (80) : (85))

29 crit: $this > (($status == $CRITICAL) ? (85) : (95))

30 delay: down 15m multiplier 1.5 max 1h

29 - info: storage usable space utilization

31 + info: Percentage of storage usable space. \

32 + High usable storage capacity utilization. \

33 + To fix, increase the space available for CockroachDB data.

34 to: dba

35

36 # Replication

@@ -41,7 +45,10 @@ component: CockroachDB

45 every: 10s

46 warn: $this > 0

47 delay: down 15m multiplier 1.5 max 1h

44 - info: number of ranges with fewer live replicas than needed for quorum

48 + info: Number of unavailable ranges. \

49 + There are ranges with fewer live replicas than needed for quorum. \

50 + If a majority of a range replicas are on nodes that are unavailable, \

51 + then the entire range is unavailable and will be unable to process queries.

52 to: dba

53

54 template: cockroachdb_underreplicated_ranges

@@ -54,7 +61,9 @@ component: CockroachDB

61 every: 10s

62 warn: $this > 0

63 delay: down 15m multiplier 1.5 max 1h

57 - info: number of ranges with fewer live replicas than the replication target

64 + info: Number of under-replicated ranges. \

65 + There are ranges with fewer live replicas than the replication target. \

66 + As soon as other nodes are available, they will replicate to them until they have reached their desired replication factor.

67 to: dba

68

69 # FD

@@ -69,5 +78,7 @@ component: CockroachDB

78 every: 10s

79 warn: $this > 80

80 delay: down 15m multiplier 1.5 max 1h

72 - info: open file descriptors utilization (against softlimit)

81 + info: Percentage of used file descriptors. \

82 + High file descriptors utilization (against softlimit). \

83 + To fix, adjust the file descriptors limit for the process or system-wide.

84 to: dba

health/health.d/cpu.conf

+16 -4

@@ -14,7 +14,11 @@ component: CPU

14 warn: $this > (($status >= $WARNING) ? (75) : (85))

15 crit: $this > (($status == $CRITICAL) ? (85) : (95))

16 delay: down 15m multiplier 1.5 max 1h

17 - info: average CPU utilization over the last 10 minutes (excluding iowait, nice and steal)

17 + info: Average CPU utilization over the last 10 minutes (excluding iowait, nice and steal). \

18 + High system CPU utilization. \

19 + A constantly high value might indicate CPU bottleneck, which can make the system run slower. \

20 + You can check the CPU PSI charts if there is CPU contention and \

21 + per-process CPU usage to find the top consumers.

22 to: sysadmin

23

24 template: 10min_cpu_iowait

@@ -30,7 +34,9 @@ component: CPU

34 warn: $this > (($status >= $WARNING) ? (20) : (40))

35 crit: $this > (($status == $CRITICAL) ? (40) : (50))

36 delay: down 15m multiplier 1.5 max 1h

33 - info: average CPU iowait time over the last 10 minutes

37 + info: Average CPU iowait time over the last 10 minutes. \

38 + High system CPU iowait time. \

39 + A constantly high value indicates that IO is a bottleneck, which can make the system run slower.

40 to: sysadmin

41

42 template: 20min_steal_cpu

@@ -46,7 +52,10 @@ component: CPU

52 warn: $this > (($status >= $WARNING) ? (5) : (10))

53 crit: $this > (($status == $CRITICAL) ? (20) : (30))

54 delay: down 1h multiplier 1.5 max 2h

49 - info: average CPU steal time over the last 20 minutes

55 + info: Average CPU steal time over the last 20 minutes. \

56 + High system CPU steal time. \

57 + A large amount of steal time indicates CPU contention on the host system, which can reduce guest performance. \

58 + To fix, increase the guest CPU priority or CPU quota, or run fewer guests on the host.

59 to: sysadmin

60

61 ## FreeBSD

@@ -63,5 +72,8 @@ component: CPU

72 warn: $this > (($status >= $WARNING) ? (75) : (85))

73 crit: $this > (($status == $CRITICAL) ? (85) : (95))

74 delay: down 15m multiplier 1.5 max 1h

66 - info: average CPU utilization over the last 10 minutes (excluding nice)

75 + info: Average CPU utilization over the last 10 minutes (excluding nice). \

76 + High system CPU utilization. \

77 + A constantly high value might indicate CPU bottleneck, which can make the system run slower. \

78 + You can check per-process CPU usage to find the top consumers.

79 to: sysadmin

health/health.d/dbengine.conf

+10 -6

@@ -13,7 +13,8 @@ component: DB engine

13 every: 10s

14 crit: $this > 0

15 delay: down 15m multiplier 1.5 max 1h

16 - info: number of filesystem errors in the last 10 minutes (too many open files, wrong permissions, etc)

16 + info: Number of filesystem errors in the last 10 minutes. \

17 + Dbengine is experiencing filesystem errors (too many open files, wrong permissions, etc.).

18 to: sysadmin

19

20 alarm: 10min_dbengine_global_io_errors

@@ -28,7 +29,8 @@ component: DB engine

29 every: 10s

30 crit: $this > 0

31 delay: down 1h multiplier 1.5 max 3h

31 - info: number of IO errors in the last 10 minutes (CRC errors, out of space, bad disk, etc)

32 + info: Number of IO errors in the last 10 minutes. \

33 + Dbengine is experiencing I/O errors (CRC errors, out of space, bad disk, etc.).

34 to: sysadmin

35

36 alarm: 10min_dbengine_global_flushing_warnings

@@ -43,8 +45,9 @@ component: DB engine

45 every: 10s

46 warn: $this > 0

47 delay: down 1h multiplier 1.5 max 3h

46 - info: number of times when dbengine dirty pages were over 50% of the instance's page cache in the last 10 minutes. \

47 - Metric data are at risk of not being stored in the database. To remedy, reduce disk load or use faster disks.

48 + info: Number of times when dbengine dirty pages were over 50% of the instance page cache in the last 10 minutes. \

49 + Metric data is at risk of not being stored in the database. \

50 + To remedy, reduce disk load or use faster disks.

51 to: sysadmin

52

53 alarm: 10min_dbengine_global_flushing_errors

@@ -59,6 +62,7 @@ component: DB engine

62 every: 10s

63 crit: $this != 0

64 delay: down 1h multiplier 1.5 max 3h

62 - info: number of pages deleted due to failure to flush data to disk in the last 10 minutes. \

63 - Metric data were lost to unblock data collection. To fix, reduce disk load or use faster disks.

65 + info: Number of pages deleted due to failure to flush data to disk in the last 10 minutes. \

66 + Some metric data was dropped to unblock data collection. \

67 + To fix, reduce disk load or use faster disks.

68 to: sysadmin

health/health.d/disks.conf

+12 -3

@@ -23,7 +23,10 @@ component: Disk

23 warn: $this > (($status >= $WARNING ) ? (80) : (90))

24 crit: $this > (($status == $CRITICAL) ? (90) : (98))

25 delay: up 1m down 15m multiplier 1.5 max 1h

26 - info: disk $family space utilization

26 + info: Percentage of used space by disk $family. \

27 + High disk space utilization. \

28 + You may experience slowdowns and crashes if the disk is full. \

29 + To fix, cleanup your disk or upgrade it.

30 to: sysadmin

31

32 template: disk_inode_usage

@@ -40,7 +43,10 @@ component: Disk

43 warn: $this > (($status >= $WARNING) ? (80) : (90))

44 crit: $this > (($status == $CRITICAL) ? (90) : (98))

45 delay: up 1m down 15m multiplier 1.5 max 1h

43 - info: disk $family inode utilization

46 + info: Percentage of used inodes by disk $family. \

47 + High disk inode utilization. \

48 + The number of inodes indicates the number of files and folders you have. \

49 + To fix, clear cache files or delete unnecessary files and folders.

50 to: sysadmin

51

52

@@ -147,7 +153,10 @@ component: Disk

153 every: 1m

154 warn: $this > 98 * (($status >= $WARNING) ? (0.7) : (1))

155 delay: down 15m multiplier 1.2 max 1h

150 - info: average percentage of time $family disk was busy over the last 10 minutes

156 + info: Average percentage of time $family disk was busy over the last 10 minutes. \

157 + High disk load. \

158 + The disk spent most of the time servicing read or write requests. \

159 + If the disk controller processes the operations in parallel, the alarm does not necessarily indicate a high load.

160 to: silent

161

162

health/health.d/dns_query.conf

+2 -1

@@ -11,5 +11,6 @@ component: DNS

11 every: 10s

12 warn: $this == nan

13 delay: up 20s down 5m multiplier 1.5 max 1h

14 - info: average DNS query round trip time over the last 10 seconds

14 + info: Average DNS query round trip time over the last 10 seconds. \

15 + Failed to query the DNS server.

16 to: sysadmin

health/health.d/dnsmasq_dhcp.conf

+4 -1

@@ -11,5 +11,8 @@ component: Dnsmasq

11 warn: $this > ( ($status >= $WARNING ) ? ( 80 ) : ( 90 ) )

12 crit: $this > ( ($status == $CRITICAL) ? ( 90 ) : ( 95 ) )

13 delay: down 5m

14 - info: DHCP range utilization

14 + info: Percentage of leased IP addresses. \

15 + High DHCP range utilization. \

16 + The number of DHCP addresses in use is close to the total number of provisioned DHCP addresses. \

17 + To fix, increase the number of IP addresses on a subnet.

18 to: sysadmin

health/health.d/dockerd.conf

+4 -1

@@ -7,5 +7,8 @@ component: Docker

7 every: 10s

8 lookup: average -10s

9 crit: $this > 0

10 - info: average number of unhealthy docker containers over the last 10 seconds

10 + info: Average number of unhealthy docker containers over the last 10 seconds. \

11 + There are unhealthy docker containers. \

12 + Some containers are not running due to failed health checks. \

13 + To find unhealthy containers use [docker ps -a].

14 to: sysadmin

health/health.d/entropy.conf

+4 -1

@@ -15,5 +15,8 @@ component: Cryptography

15 every: 5m

16 warn: $this < (($status >= $WARNING) ? (200) : (100))

17 delay: down 1h multiplier 1.5 max 2h

18 - info: minimum number of entries in the random numbers pool in the last 5 minutes

18 + info: Minimum number of bits in the random numbers pool in the last 5 minutes. \

19 + A low number of bits of entropy available. \

20 + It may have a negative impact on the security and performance of the system. \

21 + This can be fixed by installing the [haveged] or [rngd] daemon.

22 to: silent

health/health.d/exporting.conf

+10 -2

@@ -11,7 +11,11 @@ component: Exporting engine

11 warn: $this > (($status >= $WARNING) ? ($update_every) : ( 5 * $update_every))

12 crit: $this > (($status == $CRITICAL) ? ($update_every) : (60 * $update_every))

13 delay: down 5m multiplier 1.5 max 1h

14 - info: number of seconds since the last successful buffering of exporting data

14 + info: Number of seconds since the last successful buffering of exporting data. \

15 + Exporting engine failed to buffer metrics for a while. \

16 + Some metrics were lost while exporting. \

17 + It indicates that the exporting destination is down or unreachable. \

18 + Short-term network availability problems might be fixed by increasing [buffer on failures] in exporting.conf.

19 to: dba

20

21 template: exporting_metrics_sent

@@ -25,5 +29,9 @@ component: Exporting engine

29 every: 10s

30 warn: $this != 100

31 delay: down 5m multiplier 1.5 max 1h

28 - info: percentage of metrics sent to the external database server

32 + info: Percentage of metrics sent to the external database server. \

33 + Exporting engine failed to send all metrics. \

34 + Some metrics were lost while exporting. \

35 + It indicates that the exporting destination is down or unreachable. \

36 + Short-term network availability problems might be fixed by increasing [buffer on failures] in exporting.conf.

37 to: dba

health/health.d/fping.conf

+9 -3

@@ -25,7 +25,9 @@ component: Network

25 every: 10s

26 crit: $this == 0

27 delay: down 30m multiplier 1.5 max 2h

28 - info: reachability status of the network host (0: unreachable, 1: reachable)

28 + info: Reachability status of the network host (0: unreachable, 1: reachable). \

29 + The network host is unreachable. \

30 + Most likely the host is down, or you are experiencing networking issues.

31 to: sysadmin

32

33 template: fping_host_latency

@@ -42,7 +44,9 @@ component: Network

44 warn: $this > $green OR $max > $red

45 crit: $this > $red

46 delay: down 30m multiplier 1.5 max 2h

45 - info: average latency to the network host over the last 10 seconds

47 + info: Average latency to the network host over the last 10 seconds. \

48 + High latency to the network host. \

49 + Most likely you are experiencing networking issues or the host is overloaded.

50 to: sysadmin

51

52 template: fping_packet_loss

@@ -60,5 +64,7 @@ component: Network

64 warn: $this > $green

65 crit: $this > $red

66 delay: down 30m multiplier 1.5 max 2h

63 - info: packet loss ratio to the network host over the last 10 minutes

67 + info: Packet loss ratio to the network host over the last 10 minutes. \

68 + High packet loss to the network host. \

69 + Most likely you are experiencing networking issues or the host is overloaded.

70 to: sysadmin

health/health.d/gearman.conf

+3 -1

@@ -10,5 +10,7 @@ component: Gearman

10 warn: $this > 30000

11 crit: $this > 100000

12 delay: down 5m multiplier 1.5 max 1h

13 - info: average number of queued jobs over the last 10 minutes

13 + info: Average number of queued jobs over the last 10 minutes. \

14 + A high number of queued jobs. \

15 + To fix, add more workers.

16 to: sysadmin

health/health.d/haproxy.conf

+4 -2

@@ -7,7 +7,8 @@ component: HAProxy

7 every: 10s

8 lookup: average -10s

9 crit: $this > 0

10 - info: average number of failed haproxy backend servers over the last 10 seconds

10 + info: Average number of failed haproxy backend servers over the last 10 seconds. \

11 + Some haproxy backend servers are inaccessible or offline.

12 to: sysadmin

13

14 template: haproxy_backend_status

@@ -19,5 +20,6 @@ component: HAProxy

20 every: 10s

21 lookup: average -10s

22 crit: $this > 0

22 - info: average number of failed haproxy backends over the last 10 seconds

23 + info: Average number of failed haproxy backends over the last 10 seconds. \

24 + Some haproxy backends are offline.

25 to: sysadmin

health/health.d/hdfs.conf

+22 -5

@@ -12,7 +12,9 @@ component: HDFS

12 warn: $this > (($status >= $WARNING) ? (70) : (80))

13 crit: $this > (($status == $CRITICAL) ? (80) : (98))

14 delay: down 15m multiplier 1.5 max 1h

15 - info: summary datanodes space capacity utilization

15 + info: Percentage of used space capacity across all datanodes. \

16 + High datanodes space capacity utilization. \

17 + To fix this, you can add additional disks/datanodes and rebalance the datanode/cluster.

18 to: sysadmin

19

20

@@ -28,7 +30,11 @@ component: HDFS

30 every: 10s

31 warn: $this > 0

32 delay: down 15m multiplier 1.5 max 1h

31 - info: number of missing blocks

33 + info: Number of missing blocks. \

34 + There are missing blocks on some datanodes. \

35 + It may indicate a problem with the underlying storage or filesystem of a datanode. \

36 + To fix, you can find what blocks are missing using [hdfs fsck] and bring the blocks back online. \

37 + If the file can not be restored, you should delete the file.

38 to: sysadmin

39

40

@@ -42,7 +48,11 @@ component: HDFS

48 every: 10s

49 warn: $this > 0

50 delay: down 15m multiplier 1.5 max 1h

45 - info: number of datanodes marked stale due to delayed heartbeat

51 + info: Number of stale datanodes. \

52 + There are stale datanodes due to delayed heartbeat. \

53 + Some datanodes did not respond to heartbeats. \

54 + The stale datanodes are given the lowest priority for reads and writes. \

55 + If the datanode is not back online for a while, it will be considered dead.

56 to: sysadmin

57

58

@@ -56,7 +66,9 @@ component: HDFS

66 every: 10s

67 crit: $this > 0

68 delay: down 15m multiplier 1.5 max 1h

59 - info: number of datanodes which are currently dead

69 + info: Number of datanodes that are currently dead. \

70 + There are dead datanodes. \

71 + Any data that was registered to a dead datanode is not available to HDFS anymore.

72 to: sysadmin

73

74

@@ -72,5 +84,10 @@ component: HDFS

84 every: 10s

85 warn: $this > 0

86 delay: down 15m multiplier 1.5 max 1h

75 - info: number of failed volumes

87 + info: Number of failed volumes. \

88 + There are failed volumes on some datanodes. \

89 + It may indicate a hardware failure or misconfiguration, e.g. duplicate mounts. \

90 + By default, a single volume failing on a datanode will cause the entire node to go offline. \

91 + The namenode must copy any under-replicated blocks that were lost on that node, \

92 + causing a burst in network traffic and potential performance degradation.

93 to: sysadmin

health/health.d/ioping.conf

+3 -1

@@ -12,5 +12,7 @@ component: Disk

12 warn: $this > $green OR $max > $red

13 crit: $this > $red

14 delay: down 30m multiplier 1.5 max 2h

15 - info: average I/O latency over the last 10 seconds

15 + info: Average I/O latency over the last 10 seconds. \

16 + High disk I/O latency. \

17 + It may indicate a high load of the disk or the disk is slow.

18 to: sysadmin

health/health.d/ipc.conf

+8 -2

@@ -14,7 +14,10 @@ component: IPC

14 warn: $this > (($status >= $WARNING) ? (70) : (80))

15 crit: $this > (($status == $CRITICAL) ? (70) : (90))

16 delay: down 5m multiplier 1.5 max 1h

17 - info: IPC semaphore utilization

17 + info: Percentage of used IPC semaphores. \

18 + High IPC semaphore utilization. \

19 + A lack of available semaphores can affect application performance. \

20 + To fix, adjust semaphore limits on your system.

21 to: sysadmin

22

23 alarm: semaphore_arrays_used

@@ -30,5 +33,8 @@ component: IPC

33 warn: $this > (($status >= $WARNING) ? (70) : (80))

34 crit: $this > (($status == $CRITICAL) ? (70) : (90))

35 delay: down 5m multiplier 1.5 max 1h

33 - info: IPC semaphore arrays utilization

36 + info: Percentage of used IPC semaphore arrays. \

37 + High IPC semaphore arrays utilization. \

38 + A lack of available semaphore arrays can affect application performance. \

39 + To fix, adjust semaphore limits on your system.

40 to: sysadmin

health/health.d/ipfs.conf

+2 -1

@@ -10,5 +10,6 @@ component: IPFS

10 warn: $this > (($status >= $WARNING) ? (80) : (90))

11 crit: $this > (($status == $CRITICAL) ? (90) : (98))

12 delay: down 15m multiplier 1.5 max 1h

13 - info: IPFS datastore utilization

13 + info: Percentage of used IPFS datastore space. \

14 + High IPFS storage repository space utilization.

15 to: sysadmin

health/health.d/ipmi.conf

+7 -2

@@ -9,7 +9,9 @@ component: IPMI

9 warn: $this > 0

10 crit: $critical > 0

11 delay: up 5m down 15m multiplier 1.5 max 1h

12 - info: number of IPMI sensors in non-nominal state

12 + info: Number of IPMI sensors in the non-nominal state. \

13 + There are IPMI sensors in the warning or critical state. \

14 + You can find the system sensor type, state, and threshold using [ipmi-sensors].

15 to: sysadmin

16

17 alarm: ipmi_events

@@ -22,5 +24,8 @@ component: IPMI

24 every: 10s

25 warn: $this > 0

26 delay: up 5m down 15m multiplier 1.5 max 1h

25 - info: number of events in the IPMI System Event Log (SEL)

27 + info: Number of events in the IPMI System Event Log (SEL). \

28 + IPMI System Event Log (SEL) is not empty. \

29 + It contains critical, warning, and informational events. \

30 + To view them use [ipmitool sel list].

31 to: sysadmin

health/health.d/kubelet.conf

+14 -10

@@ -29,7 +29,8 @@ component: Kubelet

29 every: 10s

30 warn: $this > 0

31 delay: down 1m multiplier 1.5 max 2h

32 - info: number of failed Token() requests to the alternate token source

32 + info: Number of failed get authentication token requests. \

33 + There are failed get authentication token requests to Google Compute Engine (alternate token source).

34 to: sysadmin

35

36 # Docker and runtime operation errors

@@ -70,7 +71,7 @@ component: Kubelet

71 lookup: average -1m unaligned of kubelet_pleg_relist_latency_05

72 units: microseconds

73 every: 10s

73 - info: average Pod Lifecycle Event Generator relisting latency over the last minute (quantile 0.5)

74 + info: Average pod relisting duration over the last minute (quantile 0.5).

75

76 template: kubelet_10s_pleg_relist_latency_quantile_05

77 on: k8s_kubelet.kubelet_pleg_relist_latency_microseconds

@@ -84,8 +85,9 @@ component: Kubelet

85 warn: $this > (($status >= $WARNING)?(100):(200))

86 crit: $this > (($status >= $WARNING)?(200):(400))

87 delay: down 1m multiplier 1.5 max 2h

87 - info: ratio of average Pod Lifecycle Event Generator relisting latency over the last 10 seconds, \

88 - compared to the last minute (quantile 0.5)

88 + info: Ratio of average pod relisting duration over the last 10 seconds, compared to the last minute (quantile 0.5). \

89 + Relisting time has increased significantly. \

90 + It may indicate some problems with the container runtime engine.

91 to: sysadmin

92

93 # quantile 0.9

@@ -98,7 +100,7 @@ component: Kubelet

100 lookup: average -1m unaligned of kubelet_pleg_relist_latency_09

101 units: microseconds

102 every: 10s

101 - info: average Pod Lifecycle Event Generator relisting latency over the last minute (quantile 0.9)

103 + info: Average pod relisting duration over the last minute (quantile 0.9).

104

105 template: kubelet_10s_pleg_relist_latency_quantile_09

106 on: k8s_kubelet.kubelet_pleg_relist_latency_microseconds

@@ -112,8 +114,9 @@ component: Kubelet

114 warn: $this > (($status >= $WARNING)?(200):(400))

115 crit: $this > (($status >= $WARNING)?(400):(800))

116 delay: down 1m multiplier 1.5 max 2h

115 - info: ratio of average Pod Lifecycle Event Generator relisting latency over the last 10 seconds, \

116 - compared to the last minute (quantile 0.9)

117 + info: Ratio of average pod relisting duration over the last 10 seconds, compared to the last minute (quantile 0.9). \

118 + Relisting time has increased significantly. \

119 + It may indicate some problems with the container runtime engine.

120 to: sysadmin

121

122 # quantile 0.99

@@ -126,7 +129,7 @@ component: Kubelet

129 lookup: average -1m unaligned of kubelet_pleg_relist_latency_099

130 units: microseconds

131 every: 10s

129 - info: average Pod Lifecycle Event Generator relisting latency over the last minute (quantile 0.99)

132 + info: Average pod relisting duration over the last minute (quantile 0.99).

133

134 template: kubelet_10s_pleg_relist_latency_quantile_099

135 on: k8s_kubelet.kubelet_pleg_relist_latency_microseconds

@@ -140,6 +143,7 @@ component: Kubelet

143 warn: $this > (($status >= $WARNING)?(400):(800))

144 crit: $this > (($status >= $WARNING)?(800):(1200))

145 delay: down 1m multiplier 1.5 max 2h

143 - info: ratio of average Pod Lifecycle Event Generator relisting latency over the last 10 seconds, \

144 - compared to the last minute (quantile 0.99)

146 + info: Ratio of average pod relisting duration over the last 10 seconds, compared to the last minute (quantile 0.99). \

147 + Relisting time has increased significantly. \

148 + It may indicate some problems with the container runtime engine.

149 to: sysadmin

health/health.d/linux_power_supply.conf

+4 -1

@@ -11,5 +11,8 @@ component: Battery

11 warn: $this < 10

12 crit: $this < 5

13 delay: up 30s down 5m multiplier 1.2 max 1h

14 - info: percentage of remaining power supply capacity

14 + info: Percentage of remaining power supply capacity. \

15 + The remaining power supply capacity is low. \

16 + The system can run out of power and switch off soon. \

17 + Prepare the system for the shutdown.

18 to: sysadmin

health/health.d/load.conf

+15 -3

@@ -32,7 +32,11 @@ component: Load

32 every: 1m

33 warn: ($this * 100 / $load_cpu_number) > (($status >= $WARNING) ? 175 : 200)

34 delay: down 15m multiplier 1.5 max 1h

35 - info: system fifteen-minute load average

35 + info: System fifteen-minute load average. \

36 + High system fifteen-minute load average. \

37 + A constantly high value indicates that your system is overloaded. \

38 + It includes both CPU and I/O demand. \

39 + You might want to check per-process CPU/disk usage to find the top consumers.

40 to: sysadmin

41

42 alarm: load_average_5

@@ -47,7 +51,11 @@ component: Load

51 every: 1m

52 warn: ($this * 100 / $load_cpu_number) > (($status >= $WARNING) ? 350 : 400)

53 delay: down 15m multiplier 1.5 max 1h

50 - info: system five-minute load average

54 + info: System five-minute load average. \

55 + High system five-minute load average. \

56 + A constantly high value indicates that your system is overloaded. \

57 + It includes both CPU and I/O demand. \

58 + You might want to check per-process CPU/disk usage to find the top consumers.

59 to: sysadmin

60

61 alarm: load_average_1

@@ -62,5 +70,9 @@ component: Load

70 every: 1m

71 warn: ($this * 100 / $load_cpu_number) > (($status >= $WARNING) ? 700 : 800)

72 delay: down 15m multiplier 1.5 max 1h

65 - info: system one-minute load average

73 + info: System one-minute load average. \

74 + High system one-minute load average. \

75 + A constantly high value indicates that your system is overloaded. \

76 + It includes both CPU and I/O demand. \

77 + You might want to check per-process CPU/disk usage to find the top consumers.

78 to: sysadmin

health/health.d/mdstat.conf

+7 -3

@@ -20,8 +20,10 @@ component: RAID

20 every: 10s

21 calc: $down

22 crit: $this > 0

23 - info: number of devices in the down state for the $family array. \

24 - Any number > 0 indicates that the array is degraded.

23 + info: Number of devices in the down state for the $family array. \

24 + The array is degraded. Some array devices are missing. \

25 + To fix, bring them back online or replace the faulty ones. \

26 + To find faulty devices use [mdadm --detail RAIDDEVICE].

27 to: sysadmin

28

29 template: mdstat_mismatch_cnt

@@ -35,7 +37,9 @@ component: RAID

37 every: 60s

38 warn: $this > 1024

39 delay: up 30m

38 - info: number of unsynchronized blocks for the $family array

40 + info: Number of unsynchronized blocks for the $family array. \

41 + A high number of unsynchronized blocks for the $family array. \

42 + This might indicate that data on the array is corrupted.

43 to: sysadmin

44

45 template: mdstat_nonredundant_last_collected

health/health.d/megacli.conf

+20 -5

@@ -11,7 +11,7 @@ component: RAID

11 every: 10s

12 crit: $this > 0

13 delay: down 5m multiplier 2 max 10m

14 - info: adapter is in the degraded state (0: false, 1: true)

14 + info: Adapter is in the degraded state (0: false, 1: true).

15 to: sysadmin

16

17 ## Physical Disks

@@ -26,7 +26,12 @@ component: RAID

26 every: 10s

27 warn: $this > 0

28 delay: up 1m down 5m multiplier 2 max 10m

29 - info: number of physical drive predictive failures

29 + info: Number of physical drive predictive failures. \

30 + There are physical drive predictive failures. \

31 + The failure prediction function for the hard disk drives determines the risk of a failure in advance \

32 + and issues a warning when the risk is high. \

33 + A hard disk can still operate normally but may fail in the near future. \

34 + You might want to consider replacing the disk.

35 to: sysadmin

36

37 template: megacli_pd_media_errors

@@ -39,7 +44,10 @@ component: RAID

44 every: 10s

45 warn: $this > 0

46 delay: up 1m down 5m multiplier 2 max 10m

42 - info: number of physical drive media errors

47 + info: Number of physical drive media errors. \

48 + There are physical drive media errors. \

49 + It may indicate that a bad sector was found on the media, there is a mechanical failure of the device, \

50 + the host device detected an invalid sequence, or the target device is missing.

51 to: sysadmin

52

53 ## Battery Backup Units (BBU)

@@ -54,7 +62,11 @@ component: RAID

62 every: 10s

63 warn: $this <= (($status >= $WARNING) ? (85) : (80))

64 crit: $this <= (($status == $CRITICAL) ? (50) : (40))

57 - info: average battery backup unit (BBU) relative state of charge over the last 10 seconds

65 + info: Average battery backup unit relative state of charge over the last 10 seconds. \

66 + The state of charge is low. \

67 + The relative state of charge is an indication of full charge capacity percentage in relation to the design capacity. \

68 + A constantly low value may indicate that the battery is worn out. \

69 + You might want to consider changing the battery.

70 to: sysadmin

71

72 template: megacli_bbu_cycle_count

@@ -67,5 +79,8 @@ component: RAID

79 every: 10s

80 warn: $this >= 100

81 crit: $this >= 500

70 - info: average battery backup unit (BBU) charge cycles count over the last 10 seconds

82 + info: Average battery backup unit charge cycles count over the last 10 seconds. \

83 + A high number of full recharge cycles. \

84 + It affects the battery capacity. \

85 + You might want to consider changing the battery.

86 to: sysadmin

health/health.d/memcached.conf

+9 -4

@@ -12,7 +12,10 @@ component: Memcached

12 warn: $this > (($status >= $WARNING) ? (70) : (80))

13 crit: $this > (($status == $CRITICAL) ? (80) : (90))

14 delay: up 0 down 15m multiplier 1.5 max 1h

15 - info: cache memory utilization

15 + info: Percentage of used memory. \

16 + High cache memory utilization. \

17 + If you are getting close to 100% you will probably start experiencing evictions. \

18 + Consider increasing the cache size.

19 to: dba

20

21

@@ -27,7 +30,7 @@ component: Memcached

30 calc: ($this - $available) / (($now - $after) / 3600)

31 units: KB/hour

32 every: 1m

30 - info: average rate the cache fills up (positive), or frees up (negative) space over the last hour

33 + info: Average rate the cache fills up (positive), or frees up (negative) space over the last hour.

34

35

36 # find the hours remaining until memcached cache is full

@@ -43,6 +46,8 @@ component: Memcached

46 warn: $this > 0 and $this < (($status >= $WARNING) ? (48) : (8))

47 crit: $this > 0 and $this < (($status == $CRITICAL) ? (24) : (2))

48 delay: down 15m multiplier 1.5 max 1h

46 - info: estimated time the cache will run out of space \

47 - if the system continues to add data at the same rate as the past hour

49 + info: Estimated time until the cache runs out of space \

50 + if the system continues to add data at the same rate as the past hour. \

51 + The cache will run out of space soon. \

52 + If you are getting close to 100% you will probably start experiencing evictions.

53 to: dba

health/health.d/memory.conf

+14 -3

@@ -13,7 +13,10 @@ component: Memory

13 every: 1m

14 warn: $this > 0

15 delay: down 1h multiplier 1.5 max 1h

16 - info: number of ECC correctable errors in the last 10 minutes

16 + info: Number of ECC correctable errors in the last 10 minutes. \

17 + There are ECC correctable errors. \

18 + This event does not immediately lead to problems, but it may indicate one of the DIMM modules is slowly failing. \

19 + Check contacts and, if this error occurs more than once, consider replacing the DIMM as a preventative measure.

20 to: sysadmin

21

22 alarm: 1hour_ecc_memory_uncorrectable

@@ -28,7 +31,11 @@ component: Memory

31 every: 1m

32 crit: $this > 0

33 delay: down 1h multiplier 1.5 max 1h

31 - info: number of ECC uncorrectable errors in the last 10 minutes

34 + info: Number of ECC uncorrectable errors in the last 10 minutes. \

35 + There are ECC uncorrectable errors. This is a fatal issue. \

36 + While the error may be due to a failing DRAM chip, \

37 + it can also be caused by incorrect seating or improper contact between the socket and DIMM. \

38 + Check contacts and consider replacing the DIMM as a preventative measure.

39 to: sysadmin

40

41 alarm: 1hour_memory_hw_corrupted

@@ -43,5 +50,9 @@ component: Memory

50 every: 10s

51 warn: $this > 0

52 delay: down 1h multiplier 1.5 max 1h

46 - info: amount of memory corrupted due to a hardware failure

53 + info: Amount of memory corrupted due to a hardware failure. \

54 + Memory is corrupted due to a hardware failure. \

55 + While the error may be due to a failing DRAM chip, \

56 + it can also be caused by incorrect seating or improper contact between the socket and DIMM. \

57 + Check contacts and consider replacing the DIMM as a preventative measure.

58 to: sysadmin

health/health.d/mysql.conf

+29 -11

@@ -12,7 +12,11 @@ component: MySQL

12 warn: $this > (($status >= $WARNING) ? (5) : (10))

13 crit: $this > (($status == $CRITICAL) ? (10) : (20))

14 delay: down 5m multiplier 1.5 max 1h

15 - info: number of slow queries in the last 10 seconds

15 + info: Number of slow queries in the last 10 seconds. \

16 + A high number of slow queries. \

17 + A query is slow if it has taken more than [long_query_time] seconds. \

18 + The value is measured in real-time, not CPU time. \

19 + If you are concerned with query execution time, check system load and consider optimizing the queries.

20 to: dba

21

22

@@ -52,7 +56,10 @@ component: MySQL

56 warn: $this > (($status >= $WARNING) ? (10) : (25))

57 crit: $this > (($status == $CRITICAL) ? (25) : (50))

58 delay: down 30m multiplier 1.5 max 1h

55 - info: ratio of waited table locks over the last 10 seconds

59 + info: Ratio of waited table locks over the last 10 seconds. \

60 + High ratio of waited table locks. \

61 + If this is constantly high and you have performance problems, \

62 + you should first optimize your queries and then either split your table or tables or use replication.

63 to: dba

64

65

@@ -70,7 +77,9 @@ component: MySQL

77 warn: $this > (($status >= $WARNING) ? (60) : (70))

78 crit: $this > (($status == $CRITICAL) ? (80) : (90))

79 delay: down 15m multiplier 1.5 max 1h

73 - info: client connections utilization

80 + info: Percentage of used client connections. \

81 + High client connections utilization. \

82 + To fix, increase the number of client connections using [max_connections] MySQL system variable.

83 to: dba

84

85

@@ -87,7 +96,10 @@ component: MySQL

96 every: 10s

97 crit: $this == 0

98 delay: down 5m multiplier 1.5 max 1h

90 - info: replication status (0: stopped, 1: working)

99 + info: Replication status (0: stopped, 1: working). \

100 + Replication is not working. \

101 + Either both or one of the I/O and SQL threads are not running. \

102 + Checking error.log may help to identify the problem.

103 to: dba

104

105 template: mysql_replication_lag

@@ -101,8 +113,10 @@ component: MySQL

113 warn: $this > (($status >= $WARNING) ? (5) : (10))

114 crit: $this > (($status == $CRITICAL) ? (10) : (30))

115 delay: down 15m multiplier 1.5 max 1h

104 - info: difference between the timestamp of the latest transaction processed by the SQL thread and \

105 - the timestamp of the same transaction when it was processed on the master

116 + info: The number of seconds that the replica is behind the master. \

117 + The replication SQL thread is far behind processing the source binary log. \

118 + A constantly high value (or an increasing one) indicates that the replica is unable \

119 + to handle events from the source in a timely fashion.

120 to: dba

121

122

@@ -131,7 +145,9 @@ component: MySQL

145 warn: $this > $mysql_galera_cluster_size_max_2m

146 crit: $this < $mysql_galera_cluster_size_max_2m

147 delay: up 20s down 5m multiplier 1.5 max 1h

134 - info: current galera cluster size, compared to the maximum size in the last 2 minutes

148 + info: Current Galera cluster size, compared to the maximum size in the last 2 minutes. \

149 + Galera cluster size has changed. \

150 + It may indicate a network connectivity problem, or maybe MySQL is down on one node.

151 to: dba

152

153 # galera node state

@@ -146,8 +162,9 @@ component: MySQL

162 warn: $this == 2 OR $this == 3

163 crit: $this == 0 OR $this == 1 OR $this >= 5

164 delay: up 30s down 5m multiplier 1.5 max 1h

149 - info: galera node state \

150 - (0: Undefined, 1: Joining, 2: Donor/Desynced, 3: Joined, 4: Synced, 5: Inconsistent)

165 + info: Galera node state (0: Undefined, 1: Joining, 2: Donor/Desynced, 3: Joined, 4: Synced, 5: Inconsistent). \

166 + Galera node is not synced. \

167 + It may indicate that the node lost its connection to the Primary Component due to network partition.

168 to: dba

169

170

@@ -162,7 +179,8 @@ component: MySQL

179 every: 10s

180 crit: $mysql_galera_cluster_state != nan AND $this != 0

181 delay: up 30s down 5m multiplier 1.5 max 1h

165 - info: galera node cluster component status \

182 + info: Galera node cluster component status \

183 (-1: unknown, 0: primary/quorum present, 1: non-primary/quorum lost, 2: disconnected). \

167 - Any other value than primary indicates that the node is part of a nonoperational component.

184 + The node is a part of a nonoperational component. \

185 + It indicates that the node has lost a quorum or is unable to connect to other nodes/cluster.

186 to: dba

health/health.d/net.conf

+33 -11

@@ -31,7 +31,9 @@ component: Network

31 every: 10s

32 warn: $this > (($status >= $WARNING) ? (85) : (90))

33 delay: up 1m down 1m multiplier 1.5 max 1h

34 - info: average inbound utilization for the network interface $family over the last minute

34 + info: Average inbound utilization for the network interface $family over the last minute. \

35 + High inbound utilization. \

36 + Network delays and packet drops are expected when you are getting closer to 100%.

37 to: sysadmin

38

39 template: 1m_sent_traffic_overflow

@@ -48,7 +50,9 @@ component: Network

50 every: 10s

51 warn: $this > (($status >= $WARNING) ? (85) : (90))

52 delay: up 1m down 1m multiplier 1.5 max 1h

51 - info: average outbound utilization for the network interface $family over the last minute

53 + info: Average outbound utilization for the network interface $family over the last minute. \

54 + High outbound utilization. \

55 + Network delays and packet drops are expected when you are getting closer to 100%.

56 to: sysadmin

57

58 # -----------------------------------------------------------------------------

@@ -101,7 +105,9 @@ component: Network

105 every: 1m

106 warn: $this >= 2

107 delay: up 1m down 1h multiplier 1.5 max 2h

104 - info: ratio of inbound dropped packets for the network interface $family over the last 10 minutes

108 + info: Ratio of inbound dropped packets for the network interface $family over the last 10 minutes. \

109 + Network interface $family dropped many packets. \

110 + It means packets were received but not processed, e.g. due to lack of resources or unsupported protocol.

111 to: sysadmin

112

113 template: outbound_packets_dropped_ratio

@@ -118,7 +124,9 @@ component: Network

124 every: 1m

125 warn: $this >= 2

126 delay: up 1m down 1h multiplier 1.5 max 2h

121 - info: ratio of outbound dropped packets for the network interface $family over the last 10 minutes

127 + info: Ratio of outbound dropped packets for the network interface $family over the last 10 minutes. \

128 + Network interface $family dropped many packets. \

129 + It means packets were dropped on their way to transmission, e.g. due to lack of resources.

130 to: sysadmin

131

132 template: wifi_inbound_packets_dropped_ratio

@@ -135,7 +143,9 @@ component: Network

143 every: 1m

144 warn: $this >= 10

145 delay: up 1m down 1h multiplier 1.5 max 2h

138 - info: ratio of inbound dropped packets for the network interface $family over the last 10 minutes

146 + info: Ratio of inbound dropped packets for the network interface $family over the last 10 minutes. \

147 + Network interface $family dropped many packets. \

148 + It means packets were received but not processed, e.g. due to lack of resources or unsupported protocol.

149 to: sysadmin

150

151 template: wifi_outbound_packets_dropped_ratio

@@ -152,7 +162,9 @@ component: Network

162 every: 1m

163 warn: $this >= 10

164 delay: up 1m down 1h multiplier 1.5 max 2h

155 - info: ratio of outbound dropped packets for the network interface $family over the last 10 minutes

165 + info: Ratio of outbound dropped packets for the network interface $family over the last 10 minutes. \

166 + Network interface $family dropped many packets. \

167 + It means packets were dropped on their way to transmission, e.g. due to lack of resources.

168 to: sysadmin

169

170 # -----------------------------------------------------------------------------

@@ -171,7 +183,9 @@ component: Network

183 every: 1m

184 warn: $this >= 5

185 delay: down 1h multiplier 1.5 max 2h

174 - info: number of inbound errors for the network interface $family in the last 10 minutes

186 + info: Number of inbound errors for the network interface $family in the last 10 minutes. \

187 + Network interface $family received many bad packets. \

188 + It includes length, CRC, frame errors, etc.

189 to: sysadmin

190

191 template: interface_outbound_errors

@@ -187,7 +201,9 @@ component: Network

201 every: 1m

202 warn: $this >= 5

203 delay: down 1h multiplier 1.5 max 2h

190 - info: number of outbound errors for the network interface $family in the last 10 minutes

204 + info: Number of outbound errors for the network interface $family in the last 10 minutes. \

205 + Network interface $family experienced many transmit problems. \

206 + It includes aborted, carrier, FIFO, heartbeat, window errors, etc.

207 to: sysadmin

208

209 # -----------------------------------------------------------------------------

@@ -211,7 +227,11 @@ component: Network

227 every: 1m

228 warn: $this > 0

229 delay: down 1h multiplier 1.5 max 2h

214 - info: number of FIFO errors for the network interface $family in the last 10 minutes

230 + info: Number of FIFO errors for the network interface $family in the last 10 minutes. \

231 + Network interface $family experienced FIFO errors. \

232 + It indicates that the NIC is not able to handle the peak load of \

233 + incoming/outgoing packets with the current ring buffer size. \

234 + In case of a high packet drop rate, increasing the appropriate ring buffer can fix the issue.

235 to: sysadmin

236

237 # -----------------------------------------------------------------------------

@@ -251,6 +271,8 @@ component: Network

271 warn: $this > (($status >= $WARNING)?(200):(5000))

272 crit: $this > (($status == $CRITICAL)?(5000):(6000))

273 options: no-clear-notification

254 - info: ratio of average number of received packets for the network interface $family over the last 10 seconds, \

255 - compared to the rate over the last minute

274 + info: Ratio of an average number of received packets for the network interface $family over the last 10 seconds, \

275 + compared to the rate over the last minute. \

276 + The number of received packets has increased significantly. \

277 + It may indicate a broadcast/multicast storm or DoS attack.

278 to: sysadmin

health/health.d/netfilter.conf

+4 -1

@@ -15,5 +15,8 @@ component: Network

15 warn: $this > (($status >= $WARNING) ? (85) : (90))

16 crit: $this > (($status == $CRITICAL) ? (90) : (95))

17 delay: down 5m multiplier 1.5 max 1h

18 - info: netfilter connection tracker table size utilization

18 + info: Percentage of used netfilter tracked connections. \

19 + High netfilter connection tracking state table size utilization. \

20 + Network delays and packet drops are expected when you are getting closer to 100%. \

21 + To fix, increase the table size.

22 to: sysadmin

health/health.d/pihole.conf

+11 -4

@@ -12,7 +12,8 @@ component: Pi-hole

12 warn: $this > ( ($status >= $WARNING ) ? ( 45 ) : ( 55 ) )

13 crit: $this > ( ($status == $CRITICAL) ? ( 55 ) : ( 75 ) )

14 delay: up 2m down 5m

15 - info: percentage of blocked dns queries over the last 24 hour

15 + info: Percentage of blocked DNS queries over the last 24 hours. \

16 + A high percentage of blocked DNS queries.

17 to: sysadmin

18

19

@@ -29,7 +30,9 @@ component: Pi-hole

30 calc: $ago

31 warn: $this > 60 * 60 * 24 * 8

32 crit: $this > 60 * 60 * 24 * 8 * 2

32 - info: gravity.list (blocklist) file last update time

33 + info: Blocklist file last update time. \

34 + The blocklist file has not been updated for a long time. \

35 + To rebuild the blocklist, run the [pihole -g] command.

36 to: sysadmin

37

38 # Gravity file check (gravity.list).

@@ -44,7 +47,9 @@ component: Pi-hole

47 calc: $file_exists

48 crit: $this != 1

49 delay: up 2m down 5m

47 - info: gravity.list (blocklist) file existence state (0: exists, 1: not-exists)

50 + info: Blocklist file state (0: exists, 1: not-exists). \

51 + The blocklist file does not exist. \

52 + To rebuild the blocklist, run the [pihole -g] command.

53 to: sysadmin

54

55 # Pi-hole's ability to block unwanted domains.

@@ -60,5 +65,7 @@ component: Pi-hole

65 calc: $enabled

66 warn: $this != 1

67 delay: up 2m down 5m

63 - info: unwanted domains blocking status (0: enabled, 1: disabled)

68 + info: Unwanted domains blocking status (0: enabled, 1: disabled). \

69 + The ability of Pi-hole to block unwanted domains is disabled. \

70 + To fix, run the [pihole enable] command.

71 to: sysadmin

health/health.d/portcheck.conf

+6 -2

@@ -25,7 +25,9 @@ component: TCP endpoint

25 warn: $this >= 10 AND $this < 40

26 crit: $this >= 40

27 delay: down 5m multiplier 1.5 max 1h

28 - info: average ratio of timeouts over the last 5 minutes

28 + info: Average ratio of timeouts over the last 5 minutes. \

29 + Too many timeouts. The monitored endpoint is unreachable. \

30 + Most likely you are experiencing networking issues or the host/service is overloaded.

31 to: sysadmin

32

33 template: portcheck_connection_fails

@@ -40,5 +42,7 @@ component: TCP endpoint

42 warn: $this >= 10 AND $this < 40

43 crit: $this >= 40

44 delay: down 5m multiplier 1.5 max 1h

43 - info: average ratio of failed connections over the last 5 minutes

45 + info: Average ratio of failed connections over the last 5 minutes. \

46 + Too many failed connections. The monitored endpoint is unreachable. \

47 + Most likely the service is no longer running or access is denied by a firewall.

48 to: sysadmin

health/health.d/processes.conf

+3 -1

@@ -12,5 +12,7 @@ component: Processes

12 warn: $this > (($status >= $WARNING) ? (85) : (90))

13 crit: $this > (($status == $CRITICAL) ? (90) : (95))

14 delay: down 5m multiplier 1.5 max 1h

15 - info: system process IDs (PID) space utilization

15 + info: Percentage of used PIDs. \

16 + High system process IDs (PID) space utilization. \

17 + If this value is 100% then the system can not start new processes.

18 to: sysadmin

health/health.d/ram.conf

+23 -5

@@ -26,7 +26,11 @@ component: Memory

26 warn: $this > (($status >= $WARNING) ? (80) : (90))

27 crit: $this > (($status == $CRITICAL) ? (90) : (98))

28 delay: down 15m multiplier 1.5 max 1h

29 - info: system memory utilization

29 + info: Percentage of used RAM. \

30 + High RAM utilization. \

31 + It may affect the performance of applications. \

32 + If there is no swap space available, OOM Killer can start killing processes. \

33 + You might want to check per-process memory usage to find the top consumers.

34 to: sysadmin

35

36 alarm: ram_available

@@ -42,7 +46,11 @@ component: Memory

46 warn: $this < (($status >= $WARNING) ? (15) : (10))

47 crit: $this < (($status == $CRITICAL) ? (10) : ( 5))

48 delay: down 15m multiplier 1.5 max 1h

45 - info: percentage of estimated amount of RAM available for userspace processes, without causing swapping

49 + info: Percentage of an estimated amount of RAM available for userspace processes, without causing swapping. \

50 + Low amount of available memory. \

51 + It may affect the performance of applications. \

52 + If there is no swap space available, OOM Killer can start killing processes. \

53 + You might want to check per-process memory usage to find the top consumers.

54 to: sysadmin

55

56 alarm: oom_kill

@@ -55,7 +63,9 @@ component: Memory

63 warn: $this > 0

64 delay: down 10m

65 host labels: _is_k8s_node = false

58 - info: number of out of memory kills in the last 30 minutes

66 + info: Number of out of memory kills in the last 30 minutes. \

67 + Some processes got killed by OOM Killer. \

68 + To fix, decrease memory usage, adjust memory limits for cgroups, or add more memory/swap to the system.

69 to: sysadmin

70

71 ## FreeBSD

@@ -72,7 +82,11 @@ component: Memory

82 warn: $this > (($status >= $WARNING) ? (80) : (90))

83 crit: $this > (($status == $CRITICAL) ? (90) : (98))

84 delay: down 15m multiplier 1.5 max 1h

75 - info: system memory utilization

85 + info: Percentage of used RAM. \

86 + High RAM utilization. \

87 + It may affect the performance of applications. \

88 + If there is no swap space available, OOM Killer can start killing processes. \

89 + You might want to check per-process memory usage to find the top consumers.

90 to: sysadmin

91

92 alarm: ram_available

@@ -88,5 +102,9 @@ component: Memory

102 warn: $this < (($status >= $WARNING) ? (15) : (10))

103 crit: $this < (($status == $CRITICAL) ? (10) : ( 5))

104 delay: down 15m multiplier 1.5 max 1h

91 - info: percentage of estimated amount of RAM available for userspace processes, without causing swapping

105 + info: Percentage of an estimated amount of RAM available for userspace processes without causing swapping. \

106 + Low amount of available memory. \

107 + It may affect the performance of applications. \

108 + If there is no swap space available, OOM Killer can start killing processes. \

109 + You might want to check per-process memory usage to find the top consumers.

110 to: sysadmin

health/health.d/redis.conf

+7 -2

@@ -8,7 +8,9 @@ component: Redis

8 every: 10s

9 crit: $rdb_last_bgsave_status != 0

10 units: ok/failed

11 - info: status of the last RDB save operation (0: ok, 1: error)

11 + info: Status of the last RDB save operation (0: ok, 1: error). \

12 + Redis failed to save the RDB snapshot on disk. \

13 + Check Redis logs for details about the error.

14 delay: down 5m multiplier 1.5 max 1h

15 to: dba

16

@@ -22,6 +24,9 @@ component: Redis

24 warn: $rdb_bgsave_in_progress > 600

25 crit: $rdb_bgsave_in_progress > 1200

26 units: seconds

25 - info: duration of the on-going RDB save operation

27 + info: Duration of the ongoing RDB save operation. \

28 + Saving RDB snapshot on disk is taking too long. \

29 + Possible reasons are that the dataset size is big or the lack of CPU resources. \

30 + It may result in Redis stopping serving clients for some millisecond or even for one second.

31 delay: down 5m multiplier 1.5 max 1h

32 to: dba

health/health.d/retroshare.conf

+2 -1

@@ -12,5 +12,6 @@ component: Retroshare

12 warn: $this < (($status >= $WARNING) ? (120) : (100))

13 crit: $this < (($status == $CRITICAL) ? (10) : (1))

14 delay: up 0 down 15m multiplier 1.5 max 1h

15 - info: number of DHT peers

15 + info: Number of DHT peers. \

16 + A low number of DHT peers.

17 to: sysadmin

health/health.d/riakkv.conf

+12 -8

@@ -9,7 +9,10 @@ component: Riak KV

9 units: state machines

10 every: 10s

11 warn: $list_fsm_active > 0

12 - info: number of currently running list keys finite state machines

12 + info: Number of currently running list keys finite state machines. \

13 + There are active list keys FSMs. \

14 + A key listing in Riak is a very expensive operation, and should not be used in production \

15 + as it will affect the performance of the cluster and not scale well.

16 to: dba

17

18

@@ -38,9 +41,9 @@ component: Riak KV

41 every: 10s

42 warn: ($this > ($riakkv_1h_kv_get_mean_latency * 2) )

43 crit: ($this > ($riakkv_1h_kv_get_mean_latency * 3) )

41 - info: average time between reception of client GET request and \

42 - subsequent response to the client over the last 3 minutes, \

43 - compared to the average over the last hour

44 + info: Average GET request processing time over the last 3 minutes, compared to the average over the last hour. \

45 + GET request processing time has increased significantly. \

46 + It indicates that the server is overloaded.

47 delay: down 5m multiplier 1.5 max 1h

48 to: dba

49

@@ -68,9 +71,9 @@ component: Riak KV

71 every: 10s

72 warn: ($this > ($riakkv_1h_kv_put_mean_latency * 2) )

73 crit: ($this > ($riakkv_1h_kv_put_mean_latency * 3) )

71 - info: average time between reception of client PUT request and \

72 - subsequent response to the client over the last 3 minutes, \

73 - compared to the average over the last hour

74 + info: Average PUT request processing time over the last 3 minutes, compared to the average over the last hour. \

75 + PUT request processing time has increased significantly. \

76 + It indicates that the server is overloaded.

77 delay: down 5m multiplier 1.5 max 1h

78 to: dba

79

@@ -89,5 +92,6 @@ component: Riak KV

92 every: 10s

93 warn: $this > 10000

94 crit: $this > 100000

92 - info: number of processes running in the Erlang VM

95 + info: Number of processes running in the Erlang VM. \

96 + The number of processes is high, you may see performance degradation due to scheduling overhead.

97 to: dba

health/health.d/softnet.conf

+15 -8

@@ -15,8 +15,11 @@ component: Network

15 every: 10s

16 warn: $this > (($status >= $WARNING) ? (0) : (10))

17 delay: down 1h multiplier 1.5 max 2h

18 - info: average number of dropped packets in the last minute \

19 - due to exceeded net.core.netdev_max_backlog

18 + info: Average number of dropped packets in the last minute due to exceeded netdev backlog queue. \

19 + Netdev backlog queue is full. \

20 + It results in packets received by the interface and not yet processed by the destined subsystem or \

21 + userland application being dropped. \

22 + To fix, increase the [net.core.netdev_max_backlog] limit.

23 to: sysadmin

24

25 alarm: 1min_netdev_budget_ran_outs

@@ -31,9 +34,11 @@ component: Network

34 every: 10s

35 warn: $this > (($status >= $WARNING) ? (0) : (10))

36 delay: down 1h multiplier 1.5 max 2h

34 - info: average number of times ksoftirq ran out of sysctl net.core.netdev_budget or \

35 - net.core.netdev_budget_usecs with work remaining over the last minute \

36 - (this can be a cause for dropped packets)

37 + info: Average number of times ksoftirq ran out of netdev_budget over the last minute. \

38 + SoftIRQ was unable to process all packets available before the CPU budget was exhausted. \

39 + It may result in packet drops and is typically caused by a high-bandwidth interface adding more packets \

40 + to the receive buffer than can be processed during NAPI polling. \

41 + To fix, increase the [net.core.netdev_budget] limit.

42 to: silent

43

44 alarm: 10min_netisr_backlog_exceeded

@@ -48,7 +53,9 @@ component: Network

53 every: 10s

54 warn: $this > (($status >= $WARNING) ? (0) : (10))

55 delay: down 1h multiplier 1.5 max 2h

51 - info: average number of drops in the last minute \

52 - due to exceeded sysctl net.route.netisr_maxqlen \

53 - (this can be a cause for dropped packets)

56 + info: Average number of dropped packets in the last minute due to exceeded netisr queue length. \

57 + Netisr queue is full. \

58 + It results in packets received by the interface and not yet processed by the destined subsystem or \

59 + userland application being dropped. \

60 + To fix, increase the [net.route.netisr_maxqlen] limit.

61 to: sysadmin

health/health.d/swap.conf

+8 -3

@@ -15,7 +15,8 @@ component: Memory

15 every: 1m

16 warn: $this > (($status >= $WARNING) ? (20) : (30))

17 delay: down 15m multiplier 1.5 max 1h

18 - info: percentage of the system RAM swapped in the last 30 minutes

18 + info: Percentage of the system RAM swapped in the last 30 minutes. \

19 + A lot of system RAM was swapped.

20 to: sysadmin

21

22 alarm: used_swap

@@ -25,11 +26,15 @@ component: Memory

26 component: Memory

27 os: linux freebsd

28 hosts: *

28 - calc: $used * 100 / ( $used + $free )

29 + calc: ($used + $free) > 0 ? ($used * 100 / ($used + $free)) : 0

30 units: %

31 every: 10s

32 warn: $this > (($status >= $WARNING) ? (80) : (90))

33 crit: $this > (($status == $CRITICAL) ? (90) : (98))

34 delay: up 30s down 15m multiplier 1.5 max 1h

34 - info: swap memory utilization

35 + info: Percentage of used swap. \

36 + High swap memory utilization. \

37 + It may be a sign that the system is experiencing memory pressure, which can affect the performance of your system. \

38 + If there is no RAM available, OOM Killer can start killing processes. \

39 + You might want to check per-process swap usage to find the top consumers.

40 to: sysadmin

health/health.d/synchronization.conf

+3 -2

@@ -6,7 +6,8 @@

6 every: 1m

7 warn: $this > 6

8 delay: up 1m down 10m multiplier 1.5 max 1h

9 - info: number of sync() system calls. \

10 - Every call causes all pending modifications to filesystem metadata and \

9 + info: Number of sync() system calls. \

10 + A high number of sync() system calls. \

11 + Every call is very expensive because it causes all pending modifications to filesystem metadata and \

12 cached file data to be written to the underlying filesystems.

13 to: sysadmin

health/health.d/systemdunits.conf

+40 -10

@@ -12,7 +12,10 @@ component: Systemd units

12 every: 10s

13 warn: $this != nan AND $this == 5

14 delay: down 5m multiplier 1.5 max 1h

15 - info: one or more systemd service units are in the failed state

15 + info: One or more of the systemd service units are in the failed state. \

16 + It means that the service failed in some way \

17 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

18 + To see the details use [systemctl status SERVICENAME].

19 to: sysadmin

20

21 ## Socket units

@@ -26,7 +29,10 @@ component: Systemd units

29 every: 10s

30 warn: $this != nan AND $this == 5

31 delay: down 5m multiplier 1.5 max 1h

29 - info: one or more systemd socket units are in the failed state

32 + info: One or more of the systemd socket units are in the failed state. \

33 + It means that the service failed in some way \

34 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

35 + To see the details use [systemctl status SERVICENAME].

36 to: sysadmin

37

38 ## Target units

@@ -40,7 +46,10 @@ component: Systemd units

46 every: 10s

47 warn: $this != nan AND $this == 5

48 delay: down 5m multiplier 1.5 max 1h

43 - info: one or more systemd target units are in the failed state

49 + info: One or more of the systemd target units are in the failed state. \

50 + It means that the service failed in some way \

51 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

52 + To see the details use [systemctl status SERVICENAME].

53 to: sysadmin

54

55 ## Path units

@@ -54,7 +63,10 @@ component: Systemd units

63 every: 10s

64 warn: $this != nan AND $this == 5

65 delay: down 5m multiplier 1.5 max 1h

57 - info: one or more systemd path units are in the failed state

66 + info: One or more of the systemd path units are in the failed state. \

67 + It means that the service failed in some way \

68 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

69 + To see the details use [systemctl status SERVICENAME].

70 to: sysadmin

71

72 ## Device units

@@ -68,7 +80,10 @@ component: Systemd units

80 every: 10s

81 warn: $this != nan AND $this == 5

82 delay: down 5m multiplier 1.5 max 1h

71 - info: one or more the systemd device units are in the failed state

83 + info: One or more of the systemd device units are in the failed state. \

84 + It means that the service failed in some way \

85 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

86 + To see the details use [systemctl status SERVICENAME].

87 to: sysadmin

88

89 ## Mount units

@@ -82,7 +97,10 @@ component: Systemd units

97 every: 10s

98 warn: $this != nan AND $this == 5

99 delay: down 5m multiplier 1.5 max 1h

85 - info: one or more the systemd mount units are in the failed state

100 + info: One or more of the systemd mount units are in the failed state. \

101 + It means that the service failed in some way \

102 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

103 + To see the details use [systemctl status SERVICENAME].

104 to: sysadmin

105

106 ## Automount units

@@ -96,7 +114,10 @@ component: Systemd units

114 every: 10s

115 warn: $this != nan AND $this == 5

116 delay: down 5m multiplier 1.5 max 1h

99 - info: one or more systemd automount units are in the failed state

117 + info: One or more of the systemd automount units are in the failed state. \

118 + It means that the service failed in some way \

119 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

120 + To see the details use [systemctl status SERVICENAME].

121 to: sysadmin

122

123 ## Swap units

@@ -110,7 +131,10 @@ component: Systemd units

131 every: 10s

132 warn: $this != nan AND $this == 5

133 delay: down 5m multiplier 1.5 max 1h

113 - info: one or more systemd swap units are in the failed state

134 + info: One or more of the systemd swap units are in the failed state. \

135 + It means that the service failed in some way \

136 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

137 + To see the details use [systemctl status SERVICENAME].

138 to: sysadmin

139

140 ## Scope units

@@ -124,7 +148,10 @@ component: Systemd units

148 every: 10s

149 warn: $this != nan AND $this == 5

150 delay: down 5m multiplier 1.5 max 1h

127 - info: one or more systemd scope units are in the failed state

151 + info: One or more of the systemd scope units are in the failed state. \

152 + It means that the service failed in some way \

153 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

154 + To see the details use [systemctl status SERVICENAME].

155 to: sysadmin

156

157 ## Slice units

@@ -138,5 +165,8 @@ component: Systemd units

165 every: 10s

166 warn: $this != nan AND $this == 5

167 delay: down 5m multiplier 1.5 max 1h

141 - info: one or more systemd slice units are in the failed state

168 + info: One or more of the systemd scope units are in the failed state. \

169 + It means that the service failed in some way \

170 + (process returned error code on exit or crashed, an operation timed out, or after too many restarts). \

171 + To see the details use [systemctl status SERVICENAME].

172 to: sysadmin

health/health.d/tcp_conn.conf

+3 -1

@@ -18,5 +18,7 @@ component: Network

18 warn: $this > (($status >= $WARNING ) ? ( 60 ) : ( 80 ))

19 crit: $this > (($status == $CRITICAL) ? ( 80 ) : ( 90 ))

20 delay: up 0 down 5m multiplier 1.5 max 1h

21 - info: IPv4 TCP connections utilization

21 + info: Percentage of used IPv4 TCP connections. \

22 + High IPv4 TCP connections utilization. \

23 + If this value is 100% then the system is no longer able to establish new TCP connections.

24 to: sysadmin

health/health.d/tcp_listen.conf

+15 -5

@@ -31,7 +31,10 @@ component: Network

31 warn: $this > 1

32 crit: $this > (($status == $CRITICAL) ? (1) : (5))

33 delay: up 0 down 5m multiplier 1.5 max 1h

34 - info: average number of overflows in the TCP accept queue over the last minute

34 + info: Average number of overflows in the TCP accept queue over the last minute. \

35 + TCP accept queue is full. \

36 + It may indicate SYN floods. \

37 + To increase the queue length, adjust the [net.ipv4.tcp_max_syn_backlog] limit.

38 to: sysadmin

39

40 # THIS IS TOO GENERIC

@@ -49,7 +52,10 @@ component: Network

52 warn: $this > 1

53 crit: $this > (($status == $CRITICAL) ? (1) : (5))

54 delay: up 0 down 5m multiplier 1.5 max 1h

52 - info: average number of dropped packets in the TCP accept queue over the last minute

55 + info: Average number of dropped packets in the TCP accept queue over the last minute. \

56 + The system is dropping incoming TCP connections. \

57 + It can indicate accept queue overflow, out of memory, security issues, no route to a destination, etc. \

58 + To fix overflows, increase the [net.ipv4.tcp_max_syn_backlog] limit.

59 to: sysadmin

60

61

@@ -74,8 +80,10 @@ component: Network

80 warn: $this > 1

81 crit: $this > (($status == $CRITICAL) ? (0) : (5))

82 delay: up 10 down 5m multiplier 1.5 max 1h

77 - info: average number of SYN requests was dropped due to the full TCP SYN queue over the last minute \

78 - (SYN cookies were not enabled)

83 + info: Average number of SYN requests was dropped due to the full TCP SYN queue over the last minute. \

84 + TCP SYN queue is full. The system is dropping incoming TCP SYN requests. \

85 + It may indicate SYN floods. \

86 + If you can determine that the traffic is legitimate, consider enabling SYN cookies.

87 to: sysadmin

88

89 alarm: 1m_tcp_syn_queue_cookies

@@ -91,6 +99,8 @@ component: Network

99 warn: $this > 1

100 crit: $this > (($status == $CRITICAL) ? (0) : (5))

101 delay: up 10 down 5m multiplier 1.5 max 1h

94 - info: average number of sent SYN cookies due to the full TCP SYN queue over the last minute

102 + info: Average number of sent SYN cookies due to the full TCP SYN queue over the last minute. \

103 + TCP SYN queue is full. \

104 + It may indicate SYN floods.

105 to: sysadmin

106

health/health.d/tcp_mem.conf

+5 -1

@@ -19,5 +19,9 @@ component: Network

19 warn: ${mem} > (($status >= $WARNING ) ? ( ${tcp_mem_pressure} * 0.8 ) : ( ${tcp_mem_pressure} ))

20 crit: ${mem} > (($status == $CRITICAL ) ? ( ${tcp_mem_pressure} ) : ( ${tcp_mem_high} * 0.9 ))

21 delay: up 0 down 5m multiplier 1.5 max 1h

22 - info: TCP memory utilization

22 + info: Percentage of used TCP memory. \

23 + High TCP memory utilization. \

24 + If the TCP protocol uses more memory than the limit, the system will throw an OOM error and \

25 + some applications become unresponsive. \

26 + To increase the limit, adjust [net.ipv4.tcp_rmem] and [net.ipv4.tcp_wmem].

27 to: sysadmin

health/health.d/tcp_orphans.conf

+4 -1

@@ -20,5 +20,8 @@ component: Network

20 warn: $this > (($status >= $WARNING ) ? ( 20 ) : ( 25 ))

21 crit: $this > (($status == $CRITICAL) ? ( 25 ) : ( 50 ))

22 delay: up 0 down 5m multiplier 1.5 max 1h

23 - info: orphan IPv4 TCP sockets utilization

23 + info: Percentage of used orphan IPv4 TCP sockets. \

24 + High orphan IPv4 TCP sockets utilization. \

25 + When the limit is exceeded, orphaned connections (not attached to any user filehandle) are reset immediately. \

26 + To increase the limit, adjust [net.ipv4.tcp_max_orphans].

27 to: sysadmin

health/health.d/tcp_resets.conf

+7 -7

@@ -29,10 +29,9 @@ component: Network

29 warn: $this > ((($1m_ipv4_tcp_resets_sent < 5)?(5):($1m_ipv4_tcp_resets_sent)) * (($status >= $WARNING) ? (1) : (20)))

30 delay: up 20s down 60m multiplier 1.2 max 2h

31 options: no-clear-notification

32 - info: average number of sent TCP RESETS over the last 10 seconds. \

33 - This can indicate a port scan, \

34 - or that a service running on this host has crashed. \

35 - Netdata will not send a clear notification for this alarm.

32 + info: Average number of sent TCP RESETS over the last 10 seconds. \

33 + A high number of sent TCP RESETS. \

34 + This can indicate a port scan or that a service running on the system has crashed.

35 to: sysadmin

36

37 # -----------------------------------------------------------------------------

@@ -63,7 +62,8 @@ component: Network

62 warn: $this > ((($1m_ipv4_tcp_resets_received < 5)?(5):($1m_ipv4_tcp_resets_received)) * (($status >= $WARNING) ? (1) : (10)))

63 delay: up 20s down 60m multiplier 1.2 max 2h

64 options: no-clear-notification

66 - info: average number of received TCP RESETS over the last 10 seconds. \

67 - This can be an indication that a service this host needs has crashed. \

68 - Netdata will not send a clear notification for this alarm.

65 + info: Average number of received TCP RESETS over the last 10 seconds. \

66 + A high number of received TCP RESETS. \

67 + This can indicate that the system is trying to establish a connection \

68 + to a server port on which no process is listening.

69 to: sysadmin

health/health.d/timex.conf

+3 -1

@@ -13,5 +13,7 @@ component: Clock

13 every: 10s

14 warn: $system.uptime.uptime > 17 * 60 AND $this == 0

15 delay: down 5m

16 - info: the system time is not synchronized to a reliable server

16 + info: The system time is not synchronized to a reliable server. \

17 + It is strongly recommended having the clock in sync with NTP servers because otherwise, \

18 + it leads to unpredictable problems that are difficult to debug.

19 to: silent

health/health.d/udp_errors.conf

+8 -2

@@ -15,7 +15,10 @@ component: Network

15 units: errors

16 every: 10s

17 warn: $this > (($status >= $WARNING) ? (0) : (10))

18 - info: average number of UDP receive buffer errors over the last minute

18 + info: Average number of UDP receive buffer errors over the last minute. \

19 + UDP receive buffer is full. \

20 + The system is dropping incoming UDP packets. \

21 + To increase the limit, adjust [net.ipv4.udp_rmem].

22 delay: up 1m down 60m multiplier 1.2 max 2h

23 to: sysadmin

24

@@ -33,6 +36,9 @@ component: Network

36 units: errors

37 every: 10s

38 warn: $this > (($status >= $WARNING) ? (0) : (10))

36 - info: average number of UDP send buffer errors over the last minute

39 + info: Average number of UDP send buffer errors over the last minute. \

40 + UDP send buffer is full or no kernel memory available. \

41 + The system is dropping outgoing UDP packets. \

42 + To increase the limit, adjust [net.ipv4.udp_wmem].

43 delay: up 1m down 60m multiplier 1.2 max 2h

44 to: sysadmin

health/health.d/unbound.conf

+8 -2

@@ -11,7 +11,10 @@ component: Unbound

11 every: 10s

12 warn: $this > 5

13 delay: up 10 down 5m multiplier 1.5 max 1h

14 - info: number of overwritten queries in the request-list

14 + info: Number of overwritten queries. \

15 + Request queue is full. Unbound is overwriting old queued requests. \

16 + It can indicate a Denial of Service attack. \

17 + To increase the queue length, adjust [num-queries-per-thread].

18 to: sysadmin

19

20 template: unbound_request_list_dropped

@@ -24,5 +27,8 @@ component: Unbound

27 every: 10s

28 warn: $this > 0

29 delay: up 10 down 5m multiplier 1.5 max 1h

27 - info: number of dropped queries in the request-list

30 + info: Number of dropped queries. \

31 + Request queue is full. Unbound is dropping new incoming requests. \

32 + It can indicate a Denial of Service attack. \

33 + To increase the queue length, adjust [num-queries-per-thread].

34 to: sysadmin

health/health.d/vcsa.conf

+32 -16

@@ -17,8 +17,11 @@ component: VMware vCenter

17 warn: ($this == 1) || ($this == 2)

18 crit: $this == 3

19 delay: down 1m multiplier 1.5 max 1h

20 - info: overall system health status \

21 - (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)

20 + info: Overall system health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \

21 + Overall system status is unhealthy. \

22 + It indicates that one or more components might become overloaded soon (yellow), \

23 + or might be degraded (orange), or might be in an unusable status and the appliance might become unresponsive soon (red). \

24 + To view the details, check the VCSA Health Messages pane.

25 to: sysadmin

26

27 # Components health:

@@ -39,8 +42,10 @@ component: VMware vCenter

42 warn: $this == 1

43 crit: ($this == 2) || ($this == 3)

44 delay: down 1m multiplier 1.5 max 1h

42 - info: swap health status \

43 - (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)

45 + info: Swap health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \

46 + The swap component is unhealthy. \

47 + It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \

48 + To view the details, check the VCSA Health Messages pane.

49 to: sysadmin

50

51 template: vcsa_storage_health

@@ -54,8 +59,10 @@ component: VMware vCenter

59 warn: $this == 1

60 crit: ($this == 2) || ($this == 3)

61 delay: down 1m multiplier 1.5 max 1h

57 - info: storage health status \

58 - (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)

62 + info: Storage health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \

63 + The storage component is unhealthy. \

64 + It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \

65 + To view the details, check the VCSA Health Messages pane.

66 to: sysadmin

67

68 template: vcsa_mem_health

@@ -69,8 +76,10 @@ component: VMware vCenter

76 warn: $this == 1

77 crit: ($this == 2) || ($this == 3)

78 delay: down 1m multiplier 1.5 max 1h

72 - info: memory health status \

73 - (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)

79 + info: Memory health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \

80 + The memory component is unhealthy. \

81 + It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \

82 + To view the details, check the VCSA Health Messages pane.

83 to: sysadmin

84

85 template: vcsa_load_health

@@ -84,8 +93,10 @@ component: VMware vCenter

93 warn: $this == 1

94 crit: ($this == 2) || ($this == 3)

95 delay: down 1m multiplier 1.5 max 1h

87 - info: load health status \

88 - (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)

96 + info: Load health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \

97 + The load component is unhealthy. \

98 + It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \

99 + To view the details, check the VCSA Health Messages pane.

100 to: sysadmin

101

102 template: vcsa_database_storage_health

@@ -99,8 +110,10 @@ component: VMware vCenter

110 warn: $this == 1

111 crit: ($this == 2) || ($this == 3)

112 delay: down 1m multiplier 1.5 max 1h

102 - info: database storage health status \

103 - (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)

113 + info: Database storage health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \

114 + The database storage component is unhealthy. \

115 + It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \

116 + To view the details, check the VCSA Health Messages pane.

117 to: sysadmin

118

119 template: vcsa_applmgmt_health

@@ -114,8 +127,10 @@ component: VMware vCenter

127 warn: $this == 1

128 crit: ($this == 2) || ($this == 3)

129 delay: down 1m multiplier 1.5 max 1h

117 - info: applmgmt health status \

118 - (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey)

130 + info: Applmgmt health status (-1: unknown, 0: green, 1: yellow, 2: orange, 3: red, 4: grey). \

131 + The appliance management component is unhealthy. \

132 + It indicates that the component has minor (yellow) or severe (orange) problems, or will stop functioning soon (red). \

133 + To view the details, check the VCSA Health Messages pane.

134 to: sysadmin

135

136

@@ -136,6 +151,7 @@ component: VMware vCenter

151 warn: $this == 4

152 crit: $this == 3

153 delay: down 1m multiplier 1.5 max 1h

139 - info: software updates availability status \

140 - (-1: unknown, 0: green, 2: orange, 3: red, 4: grey)

154 + info: Software updates availability status (-1: unknown, 0: green, 2: orange, 3: red, 4: grey). \

155 + Software updates might be available. \

156 + It indicates that security patches might be available (red) or an error retrieving information on software updates (grey).

157 to: sysadmin

health/health.d/vernemq.conf

+17 -5

@@ -26,7 +26,10 @@ component: VerneMQ

26 every: 1m

27 warn: $this > (($status >= $WARNING) ? (0) : (5))

28 delay: up 2m down 5m multiplier 1.5 max 2h

29 - info: number of dropped messaged due to full queues in the last minute

29 + info: Number of dropped messages due to full queues in the last minute. \

30 + Message queues are full. VerneMQ is dropping messages. \

31 + This can indicate that consumers or VerneMQ are too slow, or publishers are too fast. \

32 + To increase the queue length, adjust [max_online_messages].

33 to: sysadmin

34

35 template: vernemq_queue_message_expired

@@ -39,7 +42,8 @@ component: VerneMQ

42 every: 1m

43 warn: $this > (($status >= $WARNING) ? (0) : (5))

44 delay: up 2m down 5m multiplier 1.5 max 2h

42 - info: number of messages which expired before delivery in the last minute

45 + info: Number of messages which expired before delivery in the last minute. \

46 + Too many messages were undelivered due to expiration.

47 to: sysadmin

48

49 template: vernemq_queue_message_unhandled

@@ -52,7 +56,8 @@ component: VerneMQ

56 every: 1m

57 warn: $this > (($status >= $WARNING) ? (0) : (5))

58 delay: up 2m down 5m multiplier 1.5 max 2h

55 - info: number of unhandled messages (connections with clean session=true) in the last minute

59 + info: Number of unhandled messages in the last minute. \

60 + Too many messages were undelivered due to connections with clean session=true.

61 to: sysadmin

62

63 # Erlang VM

@@ -83,7 +88,11 @@ component: VerneMQ

88 every: 1m

89 warn: $this > 0

90 delay: up 5m down 5m multiplier 1.5 max 1h

86 - info: amount of traffic dropped during communication with the cluster nodes in the last minute

91 + info: Amount of traffic dropped during communication with the cluster nodes in the last minute. \

92 + The outgoing cluster buffer is full. \

93 + VerneMQ is experiencing problems with inter-node message delivery. \

94 + Most likely a remote node is down or unreachable. \

95 + To increase the buffer size, adjust [outgoing_clustering_buffer_size].

96 to: sysadmin

97

98 template: vernemq_netsplits

@@ -96,7 +105,10 @@ component: VerneMQ

105 every: 10s

106 warn: $this > 0

107 delay: down 5m multiplier 1.5 max 2h

99 - info: number of detected netsplits (split brain situation) in the last minute

108 + info: Number of detected netsplits in the last minute. \

109 + Split-brain detected. \

110 + That is mostly the result of a failure of one or more network devices \

111 + resulting in a cluster where nodes can no longer reach each other.

112 to: sysadmin

113

114 # Unsuccessful CONNACK

health/health.d/whoisquery.conf

+4 -1

@@ -9,5 +9,8 @@ component: WHOIS

9 every: 60s

10 warn: $this < $days_until_expiration_warning*24*60*60

11 crit: $this < $days_until_expiration_critical*24*60*60

12 - info: time until the domain name registration expires

12 + info: Time until the domain name registration expires. \

13 + The domain name registration expires soon. \

14 + If you do not renew the domain it will be deactivated soon. \

15 + Consider scheduling the renewal.

16 to: webmaster

health/health.d/wmi.conf

+16 -8

@@ -14,7 +14,8 @@ component: CPU

14 warn: $this > (($status >= $WARNING) ? (75) : (85))

15 crit: $this > (($status == $CRITICAL) ? (85) : (95))

16 delay: down 15m multiplier 1.5 max 1h

17 - info: average CPU utilization over the last 10 minutes

17 + info: Average CPU utilization over the last 10 minutes. \

18 + High CPU utilization.

19 to: sysadmin

20

21

@@ -33,7 +34,8 @@ component: Memory

34 warn: $this > (($status >= $WARNING) ? (80) : (90))

35 crit: $this > (($status == $CRITICAL) ? (90) : (98))

36 delay: down 15m multiplier 1.5 max 1h

36 - info: memory utilization

37 + info: Percentage of used RAM memory. \

38 + High RAM utilization.

39 to: sysadmin

40

41 template: wmi_swap_in_use

@@ -49,7 +51,8 @@ component: Memory

51 warn: $this > (($status >= $WARNING) ? (80) : (90))

52 crit: $this > (($status == $CRITICAL) ? (90) : (98))

53 delay: down 15m multiplier 1.5 max 1h

52 - info: swap memory utilization

54 + info: Percentage of used swap. \

55 + High swap memory utilization.

56 to: sysadmin

57

58

@@ -68,7 +71,8 @@ component: Network

71 every: 1m

72 warn: $this >= 5

73 delay: down 1h multiplier 1.5 max 2h

71 - info: number of inbound discarded packets for the network interface in the last 10 minutes

74 + info: Number of inbound discarded packets for the network interface in the last 10 minutes. \

75 + The network interface discarded many inbound packets.

76 to: sysadmin

77

78 template: wmi_outbound_packets_discarded

@@ -84,7 +88,8 @@ component: Network

88 every: 1m

89 warn: $this >= 5

90 delay: down 1h multiplier 1.5 max 2h

87 - info: number of outbound discarded packets for the network interface in the last 10 minutes

91 + info: Number of outbound discarded packets for the network interface in the last 10 minutes. \

92 + The network interface discarded many outbound packets.

93 to: sysadmin

94

95 template: wmi_inbound_packets_errors

@@ -100,7 +105,8 @@ component: Network

105 every: 1m

106 warn: $this >= 5

107 delay: down 1h multiplier 1.5 max 2h

103 - info: number of inbound errors for the network interface in the last 10 minutes

108 + info: Number of inbound errors for the network interface in the last 10 minutes. \

109 + The network interface received many bad packets.

110 to: sysadmin

111

112 template: wmi_outbound_packets_errors

@@ -116,7 +122,8 @@ component: Network

122 every: 1m

123 warn: $this >= 5

124 delay: down 1h multiplier 1.5 max 2h

119 - info: number of outbound errors for the network interface in the last 10 minutes

125 + info: Number of outbound errors for the network interface in the last 10 minutes. \

126 + The network interface experienced many transmit problems.

127 to: sysadmin

128

129

@@ -135,5 +142,6 @@ component: Disk

142 warn: $this > (($status >= $WARNING) ? (80) : (90))

143 crit: $this > (($status == $CRITICAL) ? (90) : (98))

144 delay: down 15m multiplier 1.5 max 1h

138 - info: disk space utilization

145 + info: Percentage of used disk space. \

146 + High disk space utilization.

147 to: sysadmin

health/health.d/x509check.conf

+5 -2

@@ -9,7 +9,9 @@ component: x509 certificates

9 every: 60s

10 warn: $this < $days_until_expiration_warning*24*60*60

11 crit: $this < $days_until_expiration_critical*24*60*60

12 - info: time until x509 certificate expires

12 + info: Time until the X.509 certificate expires. \

13 + The X.509 certificate will expire soon. \

14 + Renew the certificate or create a new one.

15 to: webmaster

16

17 template: x509check_revocation_status

@@ -20,5 +22,6 @@ component: x509 certificates

22 calc: $revoked

23 every: 60s

24 crit: $this != nan AND $this != 0

23 - info: x509 certificate revocation status (0: revoked, 1: valid)

25 + info: X.509 certificate revocation status (0: revoked, 1: valid). \

26 + The X.509 certificate has been revoked.

27 to: webmaster

Update alarms info (#11481)