add python changefinder collector (#10672)
* add python changefinder collector - adds python 'changefinder' based collector for online changepoint detection.
Andrew Maguire committed
Apr 28, 2021 at 14:47 UTC
6ec39d20ccb48f90b8e801dd91f51ab9141b941a
7 files changed
+498
collectors/python.d.plugin/Makefile.am
+1
@@ -48,6 +48,7 @@ include beanstalk/Makefile.inc
48
include bind_rndc/Makefile.inc
49
include boinc/Makefile.inc
50
include ceph/Makefile.inc
51
+include changefinder/Makefile.inc
52
include chrony/Makefile.inc
53
include couchdb/Makefile.inc
54
include dnsdist/Makefile.inc
collectors/python.d.plugin/changefinder/Makefile.inc
new
+13
@@ -0,0 +1,13 @@
1
+# SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+# THIS IS NOT A COMPLETE Makefile
4
+# IT IS INCLUDED BY ITS PARENT'S Makefile.am
5
+# IT IS REQUIRED TO REFERENCE ALL FILES RELATIVE TO THE PARENT
6
+
7
+# install these files
8
+dist_python_DATA += changefinder/changefinder.chart.py
9
+dist_pythonconfig_DATA += changefinder/changefinder.conf
10
+
11
+# do not install these files, but include them in the distribution
12
+dist_noinst_DATA += changefinder/README.md changefinder/Makefile.inc
13
+
collectors/python.d.plugin/changefinder/README.md
new
+218
@@ -0,0 +1,218 @@
1
+<!--
2
+title: "Online change point detection with Netdata"
3
+description: "Use ML-driven change point detection to narrow your focus and shorten root cause analysis."
4
+custom_edit_url: https://github.com/netdata/netdata/edit/master/collectors/python.d.plugin/changefinder/README.md
5
+-->
6
+
7
+# Online changepoint detection with Netdata
8
+
9
+This collector uses the Python [changefinder](https://github.com/shunsukeaihara/changefinder) library to
10
+perform [online](https://en.wikipedia.org/wiki/Online_machine_learning) [changepoint detection](https://en.wikipedia.org/wiki/Change_detection)
11
+on your Netdata charts and/or dimensions.
12
+
13
+Instead of this collector just _collecting_ data, it also does some computation on the data it collects to return a
14
+changepoint score for each chart or dimension you configure it to work on. This is
15
+an [online](https://en.wikipedia.org/wiki/Online_machine_learning) machine learning algorithim so there is no batch step
16
+to train the model, instead it evolves over time as more data arrives. That makes this particualr algorithim quite cheap
17
+to compute at each step of data collection (see the notes section below for more details) and it should scale fairly
18
+well to work on lots of charts or hosts (if running on a parent node for example).
19
+
20
+> As this is a somewhat unique collector and involves often subjective concepts like changepoints and anomalies, we would love to hear any feedback on it from the community. Please let us know on the [community forum](https://community.netdata.cloud/t/changefinder-collector-feedback/972) or drop us a note at [analytics-ml-team@netdata.cloud](mailto:analytics-ml-team@netdata.cloud) for any and all feedback, both positive and negative. This sort of feedback is priceless to help us make complex features more useful.
21
+
22
+## Charts
23
+
24
+Two charts are available:
25
+
26
+### ChangeFinder Scores (`changefinder.scores`)
27
+
28
+This chart shows the percentile of the score that is output from the ChangeFinder library (it is turned off by default
29
+but available with `show_scores: true`).
30
+
31
+A high observed score is more likley to be a valid changepoint worth exploring, even more so when multiple charts or
32
+dimensions have high changepoint scores at the same time or very close together.
33
+
34
+### ChangeFinder Flags (`changefinder.flags`)
35
+
36
+This chart shows `1` or `0` if the latest score has a percentile value that exceeds the `cf_threshold` threshold. By
37
+default, any scores that are in the 99th or above percentile will raise a flag on this chart.
38
+
39
+The raw changefinder score itself can be a little noisey and so limiting ourselves to just periods where it surpasses
40
+the 99th percentile can help manage the "[signal to noise ratio](https://en.wikipedia.org/wiki/Signal-to-noise_ratio)"
41
+better.
42
+
43
+The `cf_threshold` paramater might be one you want to play around with to tune things specifically for the workloads on
44
+your node and the specific charts you want to monitor. For example, maybe the 95th percentile might work better for you
45
+than the 99th percentile.
46
+
47
+Below is an example of the chart produced by this collector. The first 3/4 of the period looks normal in that we see a
48
+few individual changes being picked up somewhat randomly over time. But then at around 14:59 towards the end of the
49
+chart we see two periods with 'spikes' of multiple changes for a small period of time. This is the sort of pattern that
50
+might be a sign something on the system that has changed sufficiently enough to merit some investigation.
51
+
52
+
53
+
54
+## Requirements
55
+
56
+- This collector will only work with Python 3 and requires the packages below be installed.
57
+
58
+```bash
59
+# become netdata user
60
+sudo su -s /bin/bash netdata
61
+# install required packages for the netdata user
62
+pip3 install --user numpy==1.19.5 changefinder==0.03 scipy==1.5.4
63
+```
64
+
65
+**Note**: if you need to tell Netdata to use Python 3 then you can pass the below command in the python plugin section
66
+of your `netdata.conf` file.
67
+
68
+```yaml
69
+[ plugin:python.d ]
70
+ # update every = 1
71
+ command options = -ppython3
72
+```
73
+
74
+## Configuration
75
+
76
+Install the Python requirements above, enable the collector and restart Netdata.
77
+
78
+```bash
79
+cd /etc/netdata/
80
+sudo ./edit-config python.d.conf
81
+# Set `changefinder: no` to `changefinder: yes`
82
+sudo systemctl restart netdata
83
+```
84
+
85
+The configuration for the changefinder collector defines how it will behave on your system and might take some
86
+experimentation with over time to set it optimally for your node. Out of the box, the config comes with
87
+some [sane defaults](https://www.netdata.cloud/blog/redefining-monitoring-netdata/) to get you started that try to
88
+balance the flexibility and power of the ML models with the goal of being as cheap as possible in term of cost on the
89
+node resources.
90
+
91
+_**Note**: If you are unsure about any of the below configuration options then it's best to just ignore all this and
92
+leave the `changefinder.conf` file alone to begin with. Then you can return to it later if you would like to tune things
93
+a bit more once the collector is running for a while and you have a feeling for its performance on your node._
94
+
95
+Edit the `python.d/changefinder.conf` configuration file using `edit-config` from the your
96
+agent's [config directory](/docs/configure/nodes.md), which is usually at `/etc/netdata`.
97
+
98
+```bash
99
+cd /etc/netdata # Replace this path with your Netdata config directory, if different
100
+sudo ./edit-config python.d/changefinder.conf
101
+```
102
+
103
+The default configuration should look something like this. Here you can see each parameter (with sane defaults) and some
104
+information about each one and what it does.
105
+
106
+```yaml
107
+# ----------------------------------------------------------------------
108
+# JOBS (data collection sources)
109
+
110
+# Pull data from local Netdata node.
111
+local:
112
+
113
+ # A friendly name for this job.
114
+ name: 'local'
115
+
116
+ # What host to pull data from.
117
+ host: '127.0.0.1:19999'
118
+
119
+ # What charts to pull data for - A regex like 'system\..*|' or 'system\..*|apps.cpu|apps.mem' etc.
120
+ charts_regex: 'system\..*'
121
+
122
+ # Charts to exclude, useful if you would like to exclude some specific charts.
123
+ # Note: should be a ',' separated string like 'chart.name,chart.name'.
124
+ charts_to_exclude: ''
125
+
126
+ # Get ChangeFinder scores 'per_dim' or 'per_chart'.
127
+ mode: 'per_chart'
128
+
129
+ # Default parameters that can be passed to the changefinder library.
130
+ cf_r: 0.5
131
+ cf_order: 1
132
+ cf_smooth: 15
133
+
134
+ # The percentile above which scores will be flagged.
135
+ cf_threshold: 99
136
+
137
+ # The number of recent scores to use when calculating the percentile of the changefinder score.
138
+ n_score_samples: 14400
139
+
140
+ # Set to true if you also want to chart the percentile scores in addition to the flags.
141
+ # Mainly useful for debugging or if you want to dive deeper on how the scores are evolving over time.
142
+ show_scores: false
143
+```
144
+
145
+## Troubleshooting
146
+
147
+To see any relevant log messages you can use a command like below.
148
+
149
+```bash
150
+grep 'changefinder' /var/log/netdata/error.log
151
+```
152
+
153
+If you would like to log in as `netdata` user and run the collector in debug mode to see more detail.
154
+
155
+```bash
156
+# become netdata user
157
+sudo su -s /bin/bash netdata
158
+# run collector in debug using `nolock` option if netdata is already running the collector itself.
159
+/usr/libexec/netdata/plugins.d/python.d.plugin changefinder debug trace nolock
160
+```
161
+
162
+## Notes
163
+
164
+- It may take an hour or two (depending on your choice of `n_score_samples`) for the collector to 'settle' into it's
165
+ typical behaviour in terms of the trained models and scores you will see in the normal running of your node. Mainly
166
+ this is because it can take a while to build up a proper distribution of previous scores in over to convert the raw
167
+ score returned by the ChangeFinder algorithim into a percentile based on the most recent `n_score_samples` that have
168
+ already been produced. So when you first turn the collector on, it will have a lot of flags in the beginning and then
169
+ should 'settle down' once it has built up enough history. This is a typical characteristic of online machine learning
170
+ approaches which need some initial window of time before they can be useful.
171
+- As this collector does most of the work in Python itself, you may want to try it out first on a test or development
172
+ system to get a sense of its performance characteristics on a node similar to where you would like to use it.
173
+- On a development n1-standard-2 (2 vCPUs, 7.5 GB memory) vm running Ubuntu 18.04 LTS and not doing any work some of the
174
+ typical performance characteristics we saw from running this collector (with defaults) were:
175
+ - A runtime (`netdata.runtime_changefinder`) of ~30ms.
176
+ - Typically ~1% additional cpu usage.
177
+ - About ~85mb of ram (`apps.mem`) being continually used by the `python.d.plugin` under default configuration.
178
+
179
+## Useful links and further reading
180
+
181
+- [PyPi changefinder](https://pypi.org/project/changefinder/) reference page.
182
+- [GitHub repo](https://github.com/shunsukeaihara/changefinder) for the changefinder library.
183
+- Relevant academic papers:
184
+ - Yamanishi K, Takeuchi J. A unifying framework for detecting outliers and change points from nonstationary time
185
+ series data. 8th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD02. 2002:
186
+ 676. ([pdf](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.12.3469&rep=rep1&type=pdf))
187
+ - Kawahara Y, Sugiyama M. Sequential Change-Point Detection Based on Direct Density-Ratio Estimation. SIAM
188
+ International Conference on Data Mining. 2009:
189
+ 389–400. ([pdf](https://onlinelibrary.wiley.com/doi/epdf/10.1002/sam.10124))
190
+ - Liu S, Yamada M, Collier N, Sugiyama M. Change-point detection in time-series data by relative density-ratio
191
+ estimation. Neural Networks. Jul.2013 43:72–83. [PubMed: 23500502] ([pdf](https://arxiv.org/pdf/1203.0453.pdf))
192
+ - T. Iwata, K. Nakamura, Y. Tokusashi, and H. Matsutani, “Accelerating Online Change-Point Detection Algorithm using
193
+ 10 GbE FPGA NIC,” Proc. International European Conference on Parallel and Distributed Computing (Euro-Par’18)
194
+ Workshops, vol.11339, pp.506–517, Aug.
195
+ 2018 ([pdf](https://www.arc.ics.keio.ac.jp/~matutani/papers/iwata_heteropar2018.pdf))
196
+- The [ruptures](https://github.com/deepcharles/ruptures) python package is also a good place to learn more about
197
+ changepoint detection (mostly offline as opposed to online but deals with similar concepts).
198
+- A nice [blog post](https://techrando.com/2019/08/14/a-brief-introduction-to-change-point-detection-using-python/)
199
+ showing some of the other options and libraries for changepoint detection in Python.
200
+- [Bayesian changepoint detection](https://github.com/hildensia/bayesian_changepoint_detection) library - we may explore
201
+ implementing a collector for this or integrating this approach into this collector at a future date if there is
202
+ interest and it proves computationaly feasible.
203
+- You might also find the
204
+ Netdata [anomalies collector](https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin/anomalies)
205
+ interesting.
206
+- [Anomaly Detection](https://en.wikipedia.org/wiki/Anomaly_detection) wikipedia page.
207
+- [Anomaly Detection YouTube playlist](https://www.youtube.com/playlist?list=PL6Zhl9mK2r0KxA6rB87oi4kWzoqGd5vp0)
208
+ maintained by [andrewm4894](https://github.com/andrewm4894/) from Netdata.
209
+- [awesome-TS-anomaly-detection](https://github.com/rob-med/awesome-TS-anomaly-detection) Github list of useful tools,
210
+ libraries and resources.
211
+- [Mendeley public group](https://www.mendeley.com/community/interesting-anomaly-detection-papers/) with some
212
+ interesting anomaly detection papers we have been reading.
213
+- Good [blog post](https://www.anodot.com/blog/what-is-anomaly-detection/) from Anodot on time series anomaly detection.
214
+ Anodot also have some great whitepapers in this space too that some may find useful.
215
+- Novelty and outlier detection in
216
+ the [scikit-learn documentation](https://scikit-learn.org/stable/modules/outlier_detection.html).
217
+
218
+[]()
collectors/python.d.plugin/changefinder/changefinder.chart.py
new
+185
@@ -0,0 +1,185 @@
1
+# -*- coding: utf-8 -*-
2
+# Description: changefinder netdata python.d module
3
+# Author: andrewm4894
4
+# SPDX-License-Identifier: GPL-3.0-or-later
5
+
6
+from json import loads
7
+import re
8
+
9
+from bases.FrameworkServices.UrlService import UrlService
10
+
11
+import numpy as np
12
+import changefinder
13
+from scipy.stats import percentileofscore
14
+
15
+update_every = 5
16
+disabled_by_default = True
17
+
18
+ORDER = [
19
+ 'scores',
20
+ 'flags'
21
+]
22
+
23
+CHARTS = {
24
+ 'scores': {
25
+ 'options': [None, 'ChangeFinder', 'score', 'Scores', 'scores', 'line'],
26
+ 'lines': []
27
+ },
28
+ 'flags': {
29
+ 'options': [None, 'ChangeFinder', 'flag', 'Flags', 'flags', 'stacked'],
30
+ 'lines': []
31
+ }
32
+}
33
+
34
+DEFAULT_PROTOCOL = 'http'
35
+DEFAULT_HOST = '127.0.0.1:19999'
36
+DEFAULT_CHARTS_REGEX = 'system.*'
37
+DEFAULT_MODE = 'per_chart'
38
+DEFAULT_CF_R = 0.5
39
+DEFAULT_CF_ORDER = 1
40
+DEFAULT_CF_SMOOTH = 15
41
+DEFAULT_CF_DIFF = False
42
+DEFAULT_CF_THRESHOLD = 99
43
+DEFAULT_N_SCORE_SAMPLES = 14400
44
+DEFAULT_SHOW_SCORES = False
45
+
46
+
47
+class Service(UrlService):
48
+ def __init__(self, configuration=None, name=None):
49
+ UrlService.__init__(self, configuration=configuration, name=name)
50
+ self.order = ORDER
51
+ self.definitions = CHARTS
52
+ self.protocol = self.configuration.get('protocol', DEFAULT_PROTOCOL)
53
+ self.host = self.configuration.get('host', DEFAULT_HOST)
54
+ self.url = '{}://{}/api/v1/allmetrics?format=json'.format(self.protocol, self.host)
55
+ self.charts_regex = re.compile(self.configuration.get('charts_regex', DEFAULT_CHARTS_REGEX))
56
+ self.charts_to_exclude = self.configuration.get('charts_to_exclude', '').split(',')
57
+ self.mode = self.configuration.get('mode', DEFAULT_MODE)
58
+ self.n_score_samples = int(self.configuration.get('n_score_samples', DEFAULT_N_SCORE_SAMPLES))
59
+ self.show_scores = int(self.configuration.get('show_scores', DEFAULT_SHOW_SCORES))
60
+ self.cf_r = float(self.configuration.get('cf_r', DEFAULT_CF_R))
61
+ self.cf_order = int(self.configuration.get('cf_order', DEFAULT_CF_ORDER))
62
+ self.cf_smooth = int(self.configuration.get('cf_smooth', DEFAULT_CF_SMOOTH))
63
+ self.cf_diff = bool(self.configuration.get('cf_diff', DEFAULT_CF_DIFF))
64
+ self.cf_threshold = float(self.configuration.get('cf_threshold', DEFAULT_CF_THRESHOLD))
65
+ self.collected_dims = {'scores': set(), 'flags': set()}
66
+ self.models = {}
67
+ self.x_latest = {}
68
+ self.scores_latest = {}
69
+ self.scores_samples = {}
70
+
71
+ def get_score(self, x, model):
72
+ """Update the score for the model based on most recent data, flag if it's percentile passes self.cf_threshold.
73
+ """
74
+
75
+ # get score
76
+ if model not in self.models:
77
+ # initialise empty model if needed
78
+ self.models[model] = changefinder.ChangeFinder(r=self.cf_r, order=self.cf_order, smooth=self.cf_smooth)
79
+ # if the update for this step fails then just fallback to last known score
80
+ try:
81
+ score = self.models[model].update(x)
82
+ self.scores_latest[model] = score
83
+ except Exception as _:
84
+ score = self.scores_latest.get(model, 0)
85
+ score = 0 if np.isnan(score) else score
86
+
87
+ # update sample scores used to calculate percentiles
88
+ if model in self.scores_samples:
89
+ self.scores_samples[model].append(score)
90
+ else:
91
+ self.scores_samples[model] = [score]
92
+ self.scores_samples[model] = self.scores_samples[model][-self.n_score_samples:]
93
+
94
+ # convert score to percentile
95
+ score = percentileofscore(self.scores_samples[model], score)
96
+
97
+ # flag based on score percentile
98
+ flag = 1 if score >= self.cf_threshold else 0
99
+
100
+ return score, flag
101
+
102
+ def validate_charts(self, chart, data, algorithm='absolute', multiplier=1, divisor=1):
103
+ """If dimension not in chart then add it.
104
+ """
105
+ if not self.charts:
106
+ return
107
+
108
+ for dim in data:
109
+ if dim not in self.collected_dims[chart]:
110
+ self.collected_dims[chart].add(dim)
111
+ self.charts[chart].add_dimension([dim, dim, algorithm, multiplier, divisor])
112
+
113
+ for dim in list(self.collected_dims[chart]):
114
+ if dim not in data:
115
+ self.collected_dims[chart].remove(dim)
116
+ self.charts[chart].del_dimension(dim, hide=False)
117
+
118
+ def diff(self, x, model):
119
+ """Take difference of data.
120
+ """
121
+ x_diff = x - self.x_latest.get(model, 0)
122
+ self.x_latest[model] = x
123
+ x = x_diff
124
+ return x
125
+
126
+ def _get_data(self):
127
+
128
+ # pull data from self.url
129
+ raw_data = self._get_raw_data()
130
+ if raw_data is None:
131
+ return None
132
+
133
+ raw_data = loads(raw_data)
134
+
135
+ # filter to just the data for the charts specified
136
+ charts_in_scope = list(filter(self.charts_regex.match, raw_data.keys()))
137
+ charts_in_scope = [c for c in charts_in_scope if c not in self.charts_to_exclude]
138
+
139
+ data_score = {}
140
+ data_flag = {}
141
+
142
+ # process each chart
143
+ for chart in charts_in_scope:
144
+
145
+ if self.mode == 'per_chart':
146
+
147
+ # average dims on chart and run changefinder on that average
148
+ x = [raw_data[chart]['dimensions'][dim]['value'] for dim in raw_data[chart]['dimensions']]
149
+ x = [x for x in x if x is not None]
150
+
151
+ if len(x) > 0:
152
+
153
+ x = sum(x) / len(x)
154
+ x = self.diff(x, chart) if self.cf_diff else x
155
+
156
+ score, flag = self.get_score(x, chart)
157
+ if self.show_scores:
158
+ data_score['{}_score'.format(chart)] = score * 100
159
+ data_flag[chart] = flag
160
+
161
+ else:
162
+
163
+ # run changefinder on each individual dim
164
+ for dim in raw_data[chart]['dimensions']:
165
+
166
+ chart_dim = '{}|{}'.format(chart, dim)
167
+
168
+ x = raw_data[chart]['dimensions'][dim]['value']
169
+ x = x if x else 0
170
+ x = self.diff(x, chart_dim) if self.cf_diff else x
171
+
172
+ score, flag = self.get_score(x, chart_dim)
173
+ if self.show_scores:
174
+ data_score['{}_score'.format(chart_dim)] = score * 100
175
+ data_flag[chart_dim] = flag
176
+
177
+ self.validate_charts('flags', data_flag)
178
+
179
+ if self.show_scores & len(data_score) > 0:
180
+ data_score['average_score'] = sum(data_score.values()) / len(data_score)
181
+ self.validate_charts('scores', data_score, divisor=100)
182
+
183
+ data = {**data_score, **data_flag}
184
+
185
+ return data
collectors/python.d.plugin/changefinder/changefinder.conf
new
+74
@@ -0,0 +1,74 @@
1
+# netdata python.d.plugin configuration for example
2
+#
3
+# This file is in YaML format. Generally the format is:
4
+#
5
+# name: value
6
+#
7
+# There are 2 sections:
8
+# - global variables
9
+# - one or more JOBS
10
+#
11
+# JOBS allow you to collect values from multiple sources.
12
+# Each source will have its own set of charts.
13
+#
14
+# JOB parameters have to be indented (using spaces only, example below).
15
+
16
+# ----------------------------------------------------------------------
17
+# Global Variables
18
+# These variables set the defaults for all JOBs, however each JOB
19
+# may define its own, overriding the defaults.
20
+
21
+# update_every sets the default data collection frequency.
22
+# If unset, the python.d.plugin default is used.
23
+# update_every: 5
24
+
25
+# priority controls the order of charts at the netdata dashboard.
26
+# Lower numbers move the charts towards the top of the page.
27
+# If unset, the default for python.d.plugin is used.
28
+# priority: 60000
29
+
30
+# penalty indicates whether to apply penalty to update_every in case of failures.
31
+# Penalty will increase every 5 failed updates in a row. Maximum penalty is 10 minutes.
32
+# penalty: yes
33
+
34
+# autodetection_retry sets the job re-check interval in seconds.
35
+# The job is not deleted if check fails.
36
+# Attempts to start the job are made once every autodetection_retry.
37
+# This feature is disabled by default.
38
+# autodetection_retry: 0
39
+
40
+# ----------------------------------------------------------------------
41
+# JOBS (data collection sources)
42
+
43
+local:
44
+
45
+ # A friendly name for this job.
46
+ name: 'local'
47
+
48
+ # What host to pull data from.
49
+ host: '127.0.0.1:19999'
50
+
51
+ # What charts to pull data for - A regex like 'system\..*|' or 'system\..*|apps.cpu|apps.mem' etc.
52
+ charts_regex: 'system\..*'
53
+
54
+ # Charts to exclude, useful if you would like to exclude some specific charts.
55
+ # Note: should be a ',' separated string like 'chart.name,chart.name'.
56
+ charts_to_exclude: ''
57
+
58
+ # Get ChangeFinder scores 'per_dim' or 'per_chart'.
59
+ mode: 'per_chart'
60
+
61
+ # Default parameters that can be passed to the changefinder library.
62
+ cf_r: 0.5
63
+ cf_order: 1
64
+ cf_smooth: 15
65
+
66
+ # The percentile above which scores will be flagged.
67
+ cf_threshold: 99
68
+
69
+ # The number of recent scores to use when calculating the percentile of the changefinder score.
70
+ n_score_samples: 14400
71
+
72
+ # Set to true if you also want to chart the percentile scores in addition to the flags.
73
+ # Mainly useful for debugging or if you want to dive deeper on how the scores are evolving over time.
74
+ show_scores: false
collectors/python.d.plugin/python.d.conf
+1
@@ -38,6 +38,7 @@ apache_cache: no
38
# boinc: yes
39
# ceph: yes
40
chrony: no
41
+# changefinder: no
42
# couchdb: yes
43
# dns_query_time: yes
44
# dnsdist: yes
web/gui/dashboard_info.js
+6
@@ -607,6 +607,12 @@ netdataDashboard.menu = {
607
'A special <code>failed</code> state is available as well, which is very similar to <code>inactive</code> and is entered when the service failed in some way (process returned error code on exit, or crashed, an operation timed out, or after too many restarts). ' +
608
'For detailes, see <a href="https://www.freedesktop.org/software/systemd/man/systemd.html" target="_blank"> systemd(1)</a>.'
609
},
610
+
611
+ 'changefinder': {
612
+ title: 'ChangeFinder',
613
+ icon: '<i class="fas fa-flask"></i>',
614
+ info: 'Online changepoint detection using machine learning. More details <a href="https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/changefinder/README.md" target="_blank">here</a>.'
615
+ },
616
617
'zscores': {
618
title: 'Z-Scores',