add z score alarm example (#11871)
* add z score alarm examples * give examples header titles
Andrew Maguire committed
Dec 9, 2021 at 20:08 UTC
c86f364032669fc3caa52a444583f9ff7e7810df
1 file changed
+32
-6
health/REFERENCE.md
+32
-6
@@ -679,7 +679,7 @@ Check the `health/health.d/` directory for all alarms shipped with Netdata.
679
680
Here are a few examples:
681
682
-### Example 1
682
+### Example 1 - check server alive
683
684
A simple check if an apache server is alive:
685
@@ -739,7 +739,7 @@ If these result in non-zero or true, they trigger the alarm.
739
So, the warning condition checks if we have not collected data from apache for 5
740
iterations and the critical condition checks for 10 iterations.
741
742
-### Example 2
742
+### Example 2 - disk space
743
744
Check if any of the disks is critically low on disk space:
745
@@ -760,7 +760,7 @@ So, the `calc` line finds the percentage of used space. `$this` resolves to this
760
This is a repeating alarm and if the alarm becomes CRITICAL it repeats the notifications every 10 seconds. It also
761
repeats notifications every 2 minutes if the alarm goes into WARNING mode.
762
763
-### Example 3
763
+### Example 3 - disk fill rate
764
765
Predict if any disk will run out of space in the near future.
766
@@ -803,7 +803,7 @@ Once this alarm triggers we will receive an email like this:
803
804
805
806
-### Example 4
806
+### Example 4 - dropped packets
807
808
Check if any network interface is dropping packets:
809
@@ -823,7 +823,7 @@ Note that the drops chart does not exist if a network interface has never droppe
823
When Netdata detects a dropped packet, it will add the chart and it will automatically attach this
824
alarm to it.
825
826
-### Example 5
826
+### Example 5 - CPU usage
827
828
Check if user or system dimension is using more than 50% of cpu:
829
@@ -842,7 +842,7 @@ The `lookup` line will calculate the average CPU usage from system and user in t
842
the foreach in the `lookup` line, Netdata will create two independent alarms called `dim_template_system`
843
and `dim_template_user` that will have all the other parameters shared among them.
844
845
-### Example 6
845
+### Example 6 - CPU usage
846
847
Check if all dimensions are using more than 50% of cpu:
848
@@ -860,6 +860,32 @@ lookup: average -3s percentage foreach *
860
The `lookup` line will calculate the average of CPU usage from system and user in the last 3 seconds. In this case
861
Netdata will create alarms for all dimensions of the chart.
862
863
+### Example 7 - Z-Score based alarm
864
+
865
+Derive a "[Z Score](https://en.wikipedia.org/wiki/Standard_score)" based alarm on `user` dimension of the `system.cpu` chart:
866
+
867
+```yaml
868
+ alarm: cpu_user_mean
869
+ on: system.cpu
870
+lookup: mean -60s of user
871
+ every: 10s
872
+
873
+ alarm: cpu_user_stddev
874
+ on: system.cpu
875
+lookup: stddev -60s of user
876
+ every: 10s
877
+
878
+ alarm: cpu_user_zscore
879
+ on: system.cpu
880
+lookup: mean -10s of user
881
+ calc: ($this - $cpu_user_mean) / $cpu_user_stddev
882
+ every: 10s
883
+ warn: $this < -2 or $this > 2
884
+ crit: $this < -3 or $this > 3
885
+```
886
+
887
+Since [`z = (x - mean) / stddev`](https://en.wikipedia.org/wiki/Standard_score) we create two input alarms, one for `mean` and one for `stddev` and then use them both as inputs in our final `cpu_user_zscore` alarm.
888
+
889
## Troubleshooting
890
891
You can compile Netdata with [debugging](/daemon/README.md#debugging) and then set in `netdata.conf`: