Add ml alerts examples (#13173)
* add ml alarm examples * Update Makefile.am * add hyperlinks and node level AR example
Andrew Maguire committed
Jun 21, 2022 at 10:19 UTC
73f803fbc8900d0b004a87369983d997b67c094d
3 files changed
+99
health/Makefile.am
+1
@@ -61,6 +61,7 @@ dist_healthconfig_DATA = \
61
health.d/megacli.conf \
62
health.d/memcached.conf \
63
health.d/memory.conf \
64
+ health.d/ml.conf \
65
health.d/mysql.conf \
66
health.d/net.conf \
67
health.d/netfilter.conf \
health/REFERENCE.md
+62
@@ -895,6 +895,68 @@ lookup: mean -10s of user
895
896
Since [`z = (x - mean) / stddev`](https://en.wikipedia.org/wiki/Standard_score) we create two input alarms, one for `mean` and one for `stddev` and then use them both as inputs in our final `cpu_user_zscore` alarm.
897
898
+### Example 8 - [Anomaly rate](https://learn.netdata.cloud/docs/agent/ml#anomaly-rate) based CPU dimensions alarm
899
+
900
+Warning if 5 minute rolling [anomaly rate](https://learn.netdata.cloud/docs/agent/ml#anomaly-rate) for any CPU dimension is above 5%, critical if it goes above 20%:
901
+
902
+```yaml
903
+template: ml_5min_cpu_dims
904
+ on: system.cpu
905
+ os: linux
906
+ hosts: *
907
+ lookup: average -5m anomaly-bit foreach *
908
+ calc: $this
909
+ units: %
910
+ every: 30s
911
+ warn: $this > (($status >= $WARNING) ? (5) : (20))
912
+ crit: $this > (($status == $CRITICAL) ? (20) : (100))
913
+ info: rolling 5min anomaly rate for each system.cpu dimension
914
+```
915
+
916
+The `lookup` line will calculate the average anomaly rate of each `system.cpu` dimension over the last 5 minues. In this case
917
+Netdata will create alarms for all dimensions of the chart.
918
+
919
+### Example 9 - [Anomaly rate](https://learn.netdata.cloud/docs/agent/ml#anomaly-rate) based CPU chart alarm
920
+
921
+Warning if 5 minute rolling [anomaly rate](https://learn.netdata.cloud/docs/agent/ml#anomaly-rate) averaged across all CPU dimensions is above 5%, critical if it goes above 20%:
922
+
923
+```yaml
924
+template: ml_5min_cpu_chart
925
+ on: system.cpu
926
+ os: linux
927
+ hosts: *
928
+ lookup: average -5m anomaly-bit of *
929
+ calc: $this
930
+ units: %
931
+ every: 30s
932
+ warn: $this > (($status >= $WARNING) ? (5) : (20))
933
+ crit: $this > (($status == $CRITICAL) ? (20) : (100))
934
+ info: rolling 5min anomaly rate for system.cpu chart
935
+```
936
+
937
+The `lookup` line will calculate the average anomaly rate across all `system.cpu` dimensions over the last 5 minues. In this case
938
+Netdata will create one alarm for the chart.
939
+
940
+### Example 10 - [Anomaly rate](https://learn.netdata.cloud/docs/agent/ml#anomaly-rate) based node level alarm
941
+
942
+Warning if 5 minute rolling [anomaly rate](https://learn.netdata.cloud/docs/agent/ml#anomaly-rate) averaged across all ML enabled dimensions is above 5%, critical if it goes above 20%:
943
+
944
+```yaml
945
+template: ml_5min_node
946
+ on: anomaly_detection.anomaly_rate
947
+ os: linux
948
+ hosts: *
949
+ lookup: average -5m of anomaly_rate
950
+ calc: $this
951
+ units: %
952
+ every: 30s
953
+ warn: $this > (($status >= $WARNING) ? (5) : (20))
954
+ crit: $this > (($status == $CRITICAL) ? (20) : (100))
955
+ info: rolling 5min anomaly rate for all ML enabled dims
956
+```
957
+
958
+The `lookup` line will use the `anomaly_rate` dimension of the `anomaly_detection.anomaly_rate` ML chart to calculate the average [node level anomaly rate](https://learn.netdata.cloud/docs/agent/ml#node-anomaly-rate) over the last 5 minues.
959
+
960
## Troubleshooting
961
962
You can compile Netdata with [debugging](/daemon/README.md#debugging) and then set in `netdata.conf`:
health/health.d/ml.conf
new
+36
@@ -0,0 +1,36 @@
1
+# below are some examples of using the `anomaly-bit` option to define alerts based on anomaly
2
+# rates as opposed to raw metric values. You can read more about the anomaly-bit and Netdata's
3
+# native anomaly detection here:
4
+# https://learn.netdata.cloud/docs/configure/machine-learning#anomaly-bit---100--anomalous-0--normal
5
+
6
+# examples below are commented, you would need to uncomment and adjust as desired to enable them.
7
+
8
+# alert per dimension example
9
+# if anomaly rate is between 5-20% then warning (pick your own threshold that works best via tial and error).
10
+# if anomaly rate is above 20% then critical (pick your own threshold that works best via tial and error).
11
+# template: ml_5min_cpu_dims
12
+# on: system.cpu
13
+# os: linux
14
+# hosts: *
15
+# lookup: average -5m anomaly-bit foreach *
16
+# calc: $this
17
+# units: %
18
+# every: 30s
19
+# warn: $this > (($status >= $WARNING) ? (5) : (20))
20
+# crit: $this > (($status == $CRITICAL) ? (20) : (100))
21
+# info: rolling 5min anomaly rate for each system.cpu dimension
22
+
23
+# alert per chart example
24
+# if anomaly rate is between 5-20% then warning (pick your own threshold that works best via tial and error).
25
+# if anomaly rate is above 20% then critical (pick your own threshold that works best via tial and error).
26
+# template: ml_5min_cpu_chart
27
+# on: system.cpu
28
+# os: linux
29
+# hosts: *
30
+# lookup: average -5m anomaly-bit of *
31
+# calc: $this
32
+# units: %
33
+# every: 30s
34
+# warn: $this > (($status >= $WARNING) ? (5) : (20))
35
+# crit: $this > (($status == $CRITICAL) ? (20) : (100))
36
+# info: rolling 5min anomaly rate for system.cpu chart
\ No newline at end of file