Add guide: Unsupervised anomaly detection for Raspberry Pi monitoring (#10713)
* Init guide * Tweaks * Finalize draft
Joel Hans committed
Mar 16, 2021 at 07:59 UTC
ae2bc09a34d939fdd7658a3a1ef3c4319daa3d24
1 file changed
+127
docs/guides/monitor/raspberry-pi-anomaly-detection.md
new
+127
@@ -0,0 +1,127 @@
1
+<!--
2
+title: "Unsupervised anomaly detection for Raspberry Pi monitoring"
3
+description: "Use a low-overhead machine learning algorithm and an open-source monitoring tool to detect anomalous metrics on a Raspberry Pi."
4
+image: /img/seo/guides/monitor/raspberry-pi-anomaly-detection.png
5
+author: "Andy Maguire"
6
+author_title: "Senior Machine Learning Engineer"
7
+author_img: "/img/authors/andy-maguire.jpg"
8
+custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/guides/monitor/raspberry-pi-anomaly-detection.md
9
+-->
10
+
11
+# Unsupervised anomaly detection for Raspberry Pi monitoring
12
+
13
+We love IoT and edge at Netdata, we also love machine learning. Even better if we can combine the two to ease the pain
14
+of monitoring increasingly complex systems.
15
+
16
+We recently explored what might be involved in enabling our Python-based [anomalies
17
+collector](/collectors/python.d.plugin/anomalies/README.md) on a Raspberry Pi. To our delight, it's actually quite
18
+straightforward!
19
+
20
+Read on to learn all the steps and enable unsupervised anomaly detection on your on Raspberry Pi(s).
21
+
22
+> Spoiler: It's just a couple of extra commands that will make you feel like a pro.
23
+
24
+## What you need to get started
25
+
26
+- A Raspberry Pi running Raspbian, which we'll call a _node_.
27
+- The [open-source Netdata Agent](https://github.com/netdata/netdata). If you don't have it installed on your node yet,
28
+ [get it now](/docs/get/README.md).
29
+
30
+## Install dependencies
31
+
32
+First make sure Netdata is using Python 3 when it runs Python-based data collectors.
33
+
34
+Next, open `netdata.conf` using [`edit-config`](/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files)
35
+from within the [Netdata config directory](/docs/configure/nodes.md#the-netdata-config-directory). Scroll down to the
36
+`[plugin:python.d]` section to pass in the `-ppython3` command option.
37
+
38
+```conf
39
+[plugin:python.d]
40
+ # update every = 1
41
+ command options = -ppython3
42
+```
43
+
44
+Next, install some of the underlying libraries used by the Python packages the collector depends upon.
45
+
46
+```bash
47
+sudo apt install llvm-9 libatlas3-base libgfortran5 libatlas-base-dev
48
+```
49
+
50
+Now you're ready to install the Python packages used by the collector itself. First, become the `netdata` user.
51
+
52
+```bash
53
+sudo su -s /bin/bash netdata
54
+```
55
+
56
+Then pass in the location to find `llvm` as an environment variable for `pip3`.
57
+
58
+```bash
59
+LLVM_CONFIG=llvm-config-9 pip3 install --user llvmlite numpy==1.20.1 netdata-pandas==0.0.32 numba==0.50.1 scikit-learn==0.23.2 pyod==0.8.3
60
+```
61
+
62
+## Enable the anomalies collector
63
+
64
+Now you're ready to enable the collector and [restart Netdata](/docs/configure/start-stop-restart.md).
65
+
66
+```bash
67
+sudo ./edit-config python.d.conf
68
+# set `anomalies: no` to `anomalies: yes`
69
+
70
+# restart netdata
71
+sudo systemctl restart netdata
72
+```
73
+
74
+And that should be it! Wait a minute or two, refresh your Netdata dashboard, you should see the default anomalies
75
+charts under the **Anomalies** section in the dashboard's menu.
76
+
77
+
79
+
80
+## Overhead on system
81
+
82
+Of course one of the most important considerations when trying to do anomaly detection at the edge (as opposed to in a
83
+centralized cloud somewhere) is the resource utilization impact of running a monitoring tool.
84
+
85
+With the default configuration, the anomalies collector uses about 6.5% of CPU at each run. During the retraining step,
86
+CPU utilization jumps to between 20-30% for a few seconds, but you can [configure
87
+retraining](/collectors/python.d.plugin/anomalies/README.md#configuration) to happen less often if you wish.
88
+
89
+
91
+
92
+In terms of the runtime of the collector, it was averaging around 250ms during each prediction step, jumping to about
93
+8-10 seconds during a retraining step. This jump equates only to a small gap in the anomaly charts for a few seconds.
94
+
95
+
97
+
98
+The last consideration then is the amount of RAM the collector needs to store both the models and some of the data
99
+during training. By default, the anomalies collector, along with all other running Python-based collectors, uses about
100
+100MB of system memory.
101
+
102
+
104
+
105
+## What's next?
106
+
107
+So, all in all, with a small little bit of extra set up and a small overhead on the Pi itself, the anomalies collector
108
+looks like a potentially useful addition to enable unsupervised anomaly detection on your Pi.
109
+
110
+See our two-part guide series for a more complete picture of configuring the anomalies collector, plus some best
111
+practices on using the charts it automatically generates:
112
+
113
+- [_Detect anomalies in systems and applications_](/docs/guides/monitor/anomaly-detection.md)
114
+- [_Monitor and visualize anomalies with Netdata_](/docs/guides/monitor/visualize-monitor-anomalies.md)
115
+
116
+If you're using your Raspberry Pi for other purposes, like blocking ads/trackers with Pi-hole, check out our companions
117
+Pi guide: [_Monitor Pi-hole (and a Raspberry Pi) with Netdata_](/docs/guides/monitor/pi-hole-raspberry-pi.md).
118
+
119
+Once you've had a chance to give unsupervised anomaly detection a go, share your use cases and let us know of any
120
+feedback on our [community forum](https://community.netdata.cloud/t/anomalies-collector-feedback-megathread/767).
121
+
122
+### Related reference documentation
123
+
124
+- [Netdata Agent · Get Netdata](/docs/get/README.md)
125
+- [Netdata Agent · Anomalies collector](/collectors/python.d.plugin/anomalies/README.md)
126
+
127
+[](<>)