remove python.d/anomalies (#18402)
Ilya Mashchenko committed
Aug 24, 2024 at 20:13 UTC
bb29dbf05d03705ea58c2a8f66327c2f8091ae10
7 files changed
-972
CMakeLists.txt
-2
@@ -2830,7 +2830,6 @@ install(FILES src/collectors/python.d.plugin/python.d.conf
2830
2831
install(FILES
2832
src/collectors/python.d.plugin/am2320/am2320.conf
2833
- src/collectors/python.d.plugin/anomalies/anomalies.conf
2833
src/collectors/python.d.plugin/ceph/ceph.conf
2834
src/collectors/python.d.plugin/go_expvar/go_expvar.conf
2835
src/collectors/python.d.plugin/haproxy/haproxy.conf
@@ -2848,7 +2847,6 @@ install(FILES
2847
2848
install(FILES
2849
src/collectors/python.d.plugin/am2320/am2320.chart.py
2851
- src/collectors/python.d.plugin/anomalies/anomalies.chart.py
2850
src/collectors/python.d.plugin/ceph/ceph.chart.py
2851
src/collectors/python.d.plugin/go_expvar/go_expvar.chart.py
2852
src/collectors/python.d.plugin/haproxy/haproxy.chart.py
src/collectors/python.d.plugin/anomalies/README.md
deleted
-248
@@ -1,248 +0,0 @@
1
-<!--
2
-title: "Anomaly detection with Netdata"
3
-description: "Use ML-driven anomaly detection to narrow your focus to only affected metrics and services/processes on your node to shorten root cause analysis."
4
-custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/python.d.plugin/anomalies/README.md"
5
-sidebar_url: "Anomalies"
6
-sidebar_label: "anomalies"
7
-learn_status: "Published"
8
-learn_rel_path: "Integrations/Monitor/Anything"
9
--->
10
-
11
-# Anomaly detection with Netdata
12
-
13
-**Note**: Check out the [Netdata Anomaly Advisor](/docs/dashboards-and-charts/anomaly-advisor-tab.md) for a more native anomaly detection experience within Netdata.
14
-
15
-This collector uses the Python [PyOD](https://pyod.readthedocs.io/en/latest/index.html) library to perform unsupervised [anomaly detection](https://en.wikipedia.org/wiki/Anomaly_detection) on your Netdata charts and/or dimensions.
16
-
17
-Instead of this collector just _collecting_ data, it also does some computation on the data it collects to return an anomaly probability and anomaly flag for each chart or custom model you define. This computation consists of a **train** function that runs every `train_n_secs` to train the ML models to learn what 'normal' typically looks like on your node. At each iteration there is also a **predict** function that uses the latest trained models and most recent metrics to produce an anomaly probability and anomaly flag for each chart or custom model you define.
18
-
19
-> As this is a somewhat unique collector and involves often subjective concepts like anomalies and anomaly probabilities, we would love to hear any feedback on it from the community. Please let us know on the [community forum](https://community.netdata.cloud/t/anomalies-collector-feedback-megathread/767) or drop us a note at [analytics-ml-team@netdata.cloud](mailto:analytics-ml-team@netdata.cloud) for any and all feedback, both positive and negative. This sort of feedback is priceless to help us make complex features more useful.
20
-
21
-## Charts
22
-
23
-Two charts are produced:
24
-
25
-- **Anomaly Probability** (`anomalies.probability`): This chart shows the probability that the latest observed data is anomalous based on the trained model for that chart (using the [`predict_proba()`](https://pyod.readthedocs.io/en/latest/api_cc.html#pyod.models.base.BaseDetector.predict_proba) method of the trained PyOD model).
26
-- **Anomaly** (`anomalies.anomaly`): This chart shows `1` or `0` predictions of if the latest observed data is considered anomalous or not based on the trained model (using the [`predict()`](https://pyod.readthedocs.io/en/latest/api_cc.html#pyod.models.base.BaseDetector.predict) method of the trained PyOD model).
27
-
28
-Below is an example of the charts produced by this collector and how they might look when things are 'normal' on the node. The anomaly probabilities tend to bounce randomly around a typically low probability range, one or two might randomly jump or drift outside of this range every now and then and show up as anomalies on the anomaly chart.
29
-
30
-
31
-
32
-If we then go onto the system and run a command like `stress-ng --all 2` to create some [stress](https://wiki.ubuntu.com/Kernel/Reference/stress-ng), we see some charts begin to have anomaly probabilities that jump outside the typical range. When the anomaly probabilities change enough, we will start seeing anomalies being flagged on the `anomalies.anomaly` chart. The idea is that these charts are the most anomalous right now so could be a good place to start your troubleshooting.
33
-
34
-
35
-
36
-Then, as the issue passes, the anomaly probabilities should settle back down into their 'normal' range again.
37
-
38
-
39
-
40
-## Requirements
41
-
42
-- This collector will only work with Python 3 and requires the packages below be installed.
43
-- Typically you will not need to do this, but, if needed, to ensure Python 3 is used you can add the below line to the `[plugin:python.d]` section of `netdata.conf`
44
-
45
-```conf
46
-[plugin:python.d]
47
- # update every = 1
48
- command options = -ppython3
49
-```
50
-
51
-Install the required python libraries.
52
-
53
-```bash
54
-# become netdata user
55
-sudo su -s /bin/bash netdata
56
-# install required packages for the netdata user
57
-pip3 install --user netdata-pandas==0.0.38 numba==0.50.1 scikit-learn==0.23.2 pyod==0.8.3
58
-```
59
-
60
-## Configuration
61
-
62
-Install the Python requirements above, enable the collector and restart Netdata.
63
-
64
-```bash
65
-cd /etc/netdata/
66
-sudo ./edit-config python.d.conf
67
-# Set `anomalies: no` to `anomalies: yes`
68
-sudo systemctl restart netdata
69
-```
70
-
71
-The configuration for the anomalies collector defines how it will behave on your system and might take some experimentation with over time to set it optimally for your node. Out of the box, the config comes with some [sane defaults](https://www.netdata.cloud/blog/redefining-monitoring-with-netdata/) to get you started that try to balance the flexibility and power of the ML models with the goal of being as cheap as possible in term of cost on the node resources.
72
-
73
-_**Note**: If you are unsure about any of the below configuration options then it's best to just ignore all this and leave the `anomalies.conf` file alone to begin with. Then you can return to it later if you would like to tune things a bit more once the collector is running for a while and you have a feeling for its performance on your node._
74
-
75
-Edit the `python.d/anomalies.conf` configuration file using `edit-config` from the your agent's [config
76
-directory](/docs/netdata-agent/configuration/README.md), which is usually at `/etc/netdata`.
77
-
78
-```bash
79
-cd /etc/netdata # Replace this path with your Netdata config directory, if different
80
-sudo ./edit-config python.d/anomalies.conf
81
-```
82
-
83
-The default configuration should look something like this. Here you can see each parameter (with sane defaults) and some information about each one and what it does.
84
-
85
-```conf
86
-# -
87
-# JOBS (data collection sources)
88
-
89
-# Pull data from local Netdata node.
90
-anomalies:
91
- name: 'Anomalies'
92
-
93
- # Host to pull data from.
94
- host: '127.0.0.1:19999'
95
-
96
- # Username and Password for Netdata if using basic auth.
97
- # username: '???'
98
- # password: '???'
99
-
100
- # Use http or https to pull data
101
- protocol: 'http'
102
-
103
- # SSL verify parameter for requests.get() calls
104
- tls_verify: true
105
-
106
- # What charts to pull data for - A regex like 'system\..*|' or 'system\..*|apps.cpu|apps.mem' etc.
107
- charts_regex: 'system\..*'
108
-
109
- # Charts to exclude, useful if you would like to exclude some specific charts.
110
- # Note: should be a ',' separated string like 'chart.name,chart.name'.
111
- charts_to_exclude: 'system.uptime,system.entropy'
112
-
113
- # What model to use - can be one of 'pca', 'hbos', 'iforest', 'cblof', 'loda', 'copod' or 'feature_bagging'.
114
- # More details here: https://pyod.readthedocs.io/en/latest/pyod.models.html.
115
- model: 'pca'
116
-
117
- # Max number of observations to train on, to help cap compute cost of training model if you set a very large train_n_secs.
118
- train_max_n: 100000
119
-
120
- # How often to re-train the model (assuming update_every=1 then train_every_n=1800 represents (re)training every 30 minutes).
121
- # Note: If you want to turn off re-training set train_every_n=0 and after initial training the models will not be retrained.
122
- train_every_n: 1800
123
-
124
- # The length of the window of data to train on (14400 = last 4 hours).
125
- train_n_secs: 14400
126
-
127
- # How many prediction steps after a train event to just use previous prediction value for.
128
- # Used to reduce possibility of the training step itself appearing as an anomaly on the charts.
129
- train_no_prediction_n: 10
130
-
131
- # If you would like to train the model for the first time on a specific window then you can define it using the below two variables.
132
- # Start of training data for initial model.
133
- # initial_train_data_after: 1604578857
134
-
135
- # End of training data for initial model.
136
- # initial_train_data_before: 1604593257
137
-
138
- # If you would like to ignore recent data in training then you can offset it by offset_n_secs.
139
- offset_n_secs: 0
140
-
141
- # How many lagged values of each dimension to include in the 'feature vector' each model is trained on.
142
- lags_n: 5
143
-
144
- # How much smoothing to apply to each dimension in the 'feature vector' each model is trained on.
145
- smooth_n: 3
146
-
147
- # How many differences to take in preprocessing your data.
148
- # More info on differencing here: https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average#Differencing
149
- # diffs_n=0 would mean training models on the raw values of each dimension.
150
- # diffs_n=1 means everything is done in terms of differences.
151
- diffs_n: 1
152
-
153
- # What is the typical proportion of anomalies in your data on average?
154
- # This parameter can control the sensitivity of your models to anomalies.
155
- # Some discussion here: https://github.com/yzhao062/pyod/issues/144
156
- contamination: 0.001
157
-
158
- # Set to true to include an "average_prob" dimension on anomalies probability chart which is
159
- # just the average of all anomaly probabilities at each time step
160
- include_average_prob: true
161
-
162
- # Define any custom models you would like to create anomaly probabilities for, some examples below to show how.
163
- # For example below example creates two custom models, one to run anomaly detection user and system cpu for our demo servers
164
- # and one on the cpu and mem apps metrics for the python.d.plugin.
165
- # custom_models:
166
- # - name: 'demos_cpu'
167
- # dimensions: 'london.my-netdata.io::system.cpu|user,london.my-netdata.io::system.cpu|system,newyork.my-netdata.io::system.cpu|user,newyork.my-netdata.io::system.cpu|system'
168
- # - name: 'apps_python_d_plugin'
169
- # dimensions: 'apps.cpu|python.d.plugin,apps.mem|python.d.plugin'
170
-
171
- # Set to true to normalize, using min-max standardization, features used for the custom models.
172
- # Useful if your custom models contain dimensions on very different scales an model you use does
173
- # not internally do its own normalization. Usually best to leave as false.
174
- # custom_models_normalize: false
175
-```
176
-
177
-## Custom models
178
-
179
-In the `anomalies.conf` file you can also define some "custom models" which you can use to group one or more metrics into a single model much like is done by default for the charts you specify. This is useful if you have a handful of metrics that exist in different charts but perhaps are related to the same underlying thing you would like to perform anomaly detection on, for example a specific app or user.
180
-
181
-To define a custom model you would include configuration like below in `anomalies.conf`. By default there should already be some commented out examples in there.
182
-
183
-`name` is a name you give your custom model, this is what will appear alongside any other specified charts in the `anomalies.probability` and `anomalies.anomaly` charts. `dimensions` is a string of metrics you want to include in your custom model. By default the [netdata-pandas](https://github.com/netdata/netdata-pandas) library used to pull the data from Netdata uses a "chart.a|dim.1" type of naming convention in the pandas columns it returns, hence the `dimensions` string should look like "chart.name|dimension.name,chart.name|dimension.name". The examples below hopefully make this clear.
184
-
185
-```yaml
186
-custom_models:
187
- # a model for anomaly detection on the netdata user in terms of cpu, mem, threads, processes and sockets.
188
- - name: 'user_netdata'
189
- dimensions: 'users.cpu|netdata,users.mem|netdata,users.threads|netdata,users.processes|netdata,users.sockets|netdata'
190
- # a model for anomaly detection on the netdata python.d.plugin app in terms of cpu, mem, threads, processes and sockets.
191
- - name: 'apps_python_d_plugin'
192
- dimensions: 'apps.cpu|python.d.plugin,apps.mem|python.d.plugin,apps.threads|python.d.plugin,apps.processes|python.d.plugin,apps.sockets|python.d.plugin'
193
-
194
-custom_models_normalize: false
195
-```
196
-
197
-## Troubleshooting
198
-
199
-To see any relevant log messages you can use a command like below.
200
-
201
-```bash
202
-`grep 'anomalies' /var/log/netdata/error.log`
203
-```
204
-
205
-If you would like to log in as `netdata` user and run the collector in debug mode to see more detail.
206
-
207
-```bash
208
-# become netdata user
209
-sudo su -s /bin/bash netdata
210
-# run collector in debug using `nolock` option if netdata is already running the collector itself.
211
-/usr/libexec/netdata/plugins.d/python.d.plugin anomalies debug trace nolock
212
-```
213
-
214
-## Deepdive tutorial
215
-
216
-If you would like to go deeper on what exactly the anomalies collector is doing under the hood then check out this [deepdive tutorial](https://github.com/netdata/community/blob/main/netdata-agent-api/netdata-pandas/anomalies_collector_deepdive.ipynb) in our community repo where you can play around with some data from our demo servers (or your own if its accessible to you) and work through the calculations step by step.
217
-
218
-(Note: as its a Jupyter Notebook it might render a little prettier on [nbviewer](https://nbviewer.jupyter.org/github/netdata/community/blob/main/netdata-agent-api/netdata-pandas/anomalies_collector_deepdive.ipynb))
219
-
220
-## Notes
221
-
222
-- Python 3 is required as the [`netdata-pandas`](https://github.com/netdata/netdata-pandas) package uses Python async libraries ([asks](https://pypi.org/project/asks/) and [trio](https://pypi.org/project/trio/)) to make asynchronous calls to the [Netdata REST API](/src/web/api/README.md) to get the required data for each chart.
223
-- Python 3 is also required for the underlying ML libraries of [numba](https://pypi.org/project/numba/), [scikit-learn](https://pypi.org/project/scikit-learn/), and [PyOD](https://pypi.org/project/pyod/).
224
-- It may take a few hours or so (depending on your choice of `train_secs_n`) for the collector to 'settle' into it's typical behaviour in terms of the trained models and probabilities you will see in the normal running of your node.
225
-- As this collector does most of the work in Python itself, with [PyOD](https://pyod.readthedocs.io/en/latest/) leveraging [numba](https://numba.pydata.org/) under the hood, you may want to try it out first on a test or development system to get a sense of its performance characteristics on a node similar to where you would like to use it.
226
-- `lags_n`, `smooth_n`, and `diffs_n` together define the preprocessing done to the raw data before models are trained and before each prediction. This essentially creates a [feature vector](https://en.wikipedia.org/wiki/Feature_(machine_learning)#:~:text=In%20pattern%20recognition%20and%20machine,features%20that%20represent%20some%20object.&text=Feature%20vectors%20are%20often%20combined,score%20for%20making%20a%20prediction.) for each chart model (or each custom model). The default settings for these parameters aim to create a rolling matrix of recent smoothed [differenced](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average#Differencing) values for each chart. The aim of the model then is to score how unusual this 'matrix' of features is for each chart based on what it has learned as 'normal' from the training data. So as opposed to just looking at the single most recent value of a dimension and considering how strange it is, this approach looks at a recent smoothed window of all dimensions for a chart (or dimensions in a custom model) and asks how unusual the data as a whole looks. This should be more flexible in capturing a wider range of [anomaly types](https://andrewm4894.com/2020/10/19/different-types-of-time-series-anomalies/) and be somewhat more robust to temporary 'spikes' in the data that tend to always be happening somewhere in your metrics but often are not the most important type of anomaly (this is all covered in a lot more detail in the [deepdive tutorial](https://nbviewer.jupyter.org/github/netdata/community/blob/main/netdata-agent-api/netdata-pandas/anomalies_collector_deepdive.ipynb)).
227
-- You can see how long model training is taking by looking in the logs for the collector `grep 'anomalies' /var/log/netdata/error.log | grep 'training'` and you should see lines like `2020-12-01 22:02:14: python.d INFO: anomalies[local] : training complete in 2.81 seconds (runs_counter=2700, model=pca, train_n_secs=14400, models=26, n_fit_success=26, n_fit_fails=0, after=1606845731, before=1606860131).`.
228
- - This also gives counts of the number of models, if any, that failed to fit and so had to default back to the DefaultModel (which is currently [HBOS](https://pyod.readthedocs.io/en/latest/_modules/pyod/models/hbos.html)).
229
- - `after` and `before` here refer to the start and end of the training data used to train the models.
230
-- On a development n1-standard-2 (2 vCPUs, 7.5 GB memory) vm running Ubuntu 18.04 LTS and not doing any work some of the typical performance characteristics we saw from running this collector (with defaults) were:
231
- - A runtime (`netdata.runtime_anomalies`) of ~80ms when doing scoring and ~3 seconds when training or retraining the models.
232
- - Typically ~3%-3.5% additional cpu usage from scoring, jumping to ~60% for a couple of seconds during model training.
233
- - About ~150mb of ram (`apps.mem`) being continually used by the `python.d.plugin`.
234
-- If you activate this collector on a fresh node, it might take a little while to build up enough data to calculate a realistic and useful model.
235
-- Some models like `iforest` can be comparatively expensive (on same n1-standard-2 system above ~2s runtime during predict, ~40s training time, ~50% cpu on both train and predict) so if you would like to use it you might be advised to set a relatively high `update_every` maybe 10, 15 or 30 in `anomalies.conf`.
236
-- Setting a higher `train_every_n` and `update_every` is an easy way to devote less resources on the node to anomaly detection. Specifying less charts and a lower `train_n_secs` will also help reduce resources at the expense of covering less charts and maybe a more noisy model if you set `train_n_secs` to be too small for how your node tends to behave.
237
-- If you would like to enable this on a Raspberry Pi, then check out [this guide](/docs/developer-and-contributor-corner/raspberry-pi-anomaly-detection.md) which will guide you through first installing LLVM.
238
-
239
-## Useful links and further reading
240
-
241
-- [PyOD documentation](https://pyod.readthedocs.io/en/latest/), [PyOD Github](https://github.com/yzhao062/pyod).
242
-- [Anomaly Detection](https://en.wikipedia.org/wiki/Anomaly_detection) wikipedia page.
243
-- [Anomaly Detection YouTube playlist](https://www.youtube.com/playlist?list=PL6Zhl9mK2r0KxA6rB87oi4kWzoqGd5vp0) maintained by [andrewm4894](https://github.com/andrewm4894/) from Netdata.
244
-- [awesome-TS-anomaly-detection](https://github.com/rob-med/awesome-TS-anomaly-detection) Github list of useful tools, libraries and resources.
245
-- [Mendeley public group](https://www.mendeley.com/community/interesting-anomaly-detection-papers/) with some interesting anomaly detection papers we have been reading.
246
-- Good [blog post](https://www.anodot.com/blog/what-is-anomaly-detection/) from Anodot on time series anomaly detection. Anodot also have some great whitepapers in this space too that some may find useful.
247
-- Novelty and outlier detection in the [scikit-learn documentation](https://scikit-learn.org/stable/modules/outlier_detection.html).
248
-
src/collectors/python.d.plugin/anomalies/anomalies.chart.py
deleted
-425
@@ -1,425 +0,0 @@
1
-# -*- coding: utf-8 -*-
2
-# Description: anomalies netdata python.d module
3
-# Author: andrewm4894
4
-# SPDX-License-Identifier: GPL-3.0-or-later
5
-
6
-import sys
7
-import time
8
-from datetime import datetime
9
-import re
10
-import warnings
11
-
12
-import requests
13
-import numpy as np
14
-import pandas as pd
15
-from netdata_pandas.data import get_data, get_allmetrics_async
16
-from pyod.models.hbos import HBOS
17
-from pyod.models.pca import PCA
18
-from pyod.models.loda import LODA
19
-from pyod.models.iforest import IForest
20
-from pyod.models.cblof import CBLOF
21
-from pyod.models.feature_bagging import FeatureBagging
22
-from pyod.models.copod import COPOD
23
-from sklearn.preprocessing import MinMaxScaler
24
-
25
-from bases.FrameworkServices.SimpleService import SimpleService
26
-
27
-# ignore some sklearn/numpy warnings that are ok
28
-warnings.filterwarnings('ignore', r'All-NaN slice encountered')
29
-warnings.filterwarnings('ignore', r'invalid value encountered in true_divide')
30
-warnings.filterwarnings('ignore', r'divide by zero encountered in true_divide')
31
-warnings.filterwarnings('ignore', r'invalid value encountered in subtract')
32
-
33
-disabled_by_default = True
34
-
35
-ORDER = ['probability', 'anomaly']
36
-
37
-CHARTS = {
38
- 'probability': {
39
- 'options': ['probability', 'Anomaly Probability', 'probability', 'anomalies', 'anomalies.probability', 'line'],
40
- 'lines': []
41
- },
42
- 'anomaly': {
43
- 'options': ['anomaly', 'Anomaly', 'count', 'anomalies', 'anomalies.anomaly', 'stacked'],
44
- 'lines': []
45
- },
46
-}
47
-
48
-
49
-class Service(SimpleService):
50
- def __init__(self, configuration=None, name=None):
51
- SimpleService.__init__(self, configuration=configuration, name=name)
52
- self.basic_init()
53
- self.charts_init()
54
- self.custom_models_init()
55
- self.data_init()
56
- self.model_params_init()
57
- self.models_init()
58
- self.collected_dims = {'probability': set(), 'anomaly': set()}
59
-
60
- def check(self):
61
- if not (sys.version_info[0] >= 3 and sys.version_info[1] >= 6):
62
- self.error("anomalies collector only works with Python>=3.6")
63
- if len(self.host_charts_dict[self.host]) > 0:
64
- _ = get_allmetrics_async(host_charts_dict=self.host_charts_dict, protocol=self.protocol, user=self.username, pwd=self.password)
65
- return True
66
-
67
- def basic_init(self):
68
- """Perform some basic initialization.
69
- """
70
- self.order = ORDER
71
- self.definitions = CHARTS
72
- self.protocol = self.configuration.get('protocol', 'http')
73
- self.host = self.configuration.get('host', '127.0.0.1:19999')
74
- self.username = self.configuration.get('username', None)
75
- self.password = self.configuration.get('password', None)
76
- self.tls_verify = self.configuration.get('tls_verify', True)
77
- self.fitted_at = {}
78
- self.df_allmetrics = pd.DataFrame()
79
- self.last_train_at = 0
80
- self.include_average_prob = bool(self.configuration.get('include_average_prob', True))
81
- self.reinitialize_at_every_step = bool(self.configuration.get('reinitialize_at_every_step', False))
82
-
83
- def charts_init(self):
84
- """Do some initialisation of charts in scope related variables.
85
- """
86
- self.charts_regex = re.compile(self.configuration.get('charts_regex','None'))
87
- self.charts_available = [c for c in list(requests.get(f'{self.protocol}://{self.host}/api/v1/charts', verify=self.tls_verify).json().get('charts', {}).keys())]
88
- self.charts_in_scope = list(filter(self.charts_regex.match, self.charts_available))
89
- self.charts_to_exclude = self.configuration.get('charts_to_exclude', '').split(',')
90
- if len(self.charts_to_exclude) > 0:
91
- self.charts_in_scope = [c for c in self.charts_in_scope if c not in self.charts_to_exclude]
92
-
93
- def custom_models_init(self):
94
- """Perform initialization steps related to custom models.
95
- """
96
- self.custom_models = self.configuration.get('custom_models', None)
97
- self.custom_models_normalize = bool(self.configuration.get('custom_models_normalize', False))
98
- if self.custom_models:
99
- self.custom_models_names = [model['name'] for model in self.custom_models]
100
- self.custom_models_dims = [i for s in [model['dimensions'].split(',') for model in self.custom_models] for i in s]
101
- self.custom_models_dims = [dim if '::' in dim else f'{self.host}::{dim}' for dim in self.custom_models_dims]
102
- self.custom_models_charts = list(set([dim.split('|')[0].split('::')[1] for dim in self.custom_models_dims]))
103
- self.custom_models_hosts = list(set([dim.split('::')[0] for dim in self.custom_models_dims]))
104
- self.custom_models_host_charts_dict = {}
105
- for host in self.custom_models_hosts:
106
- self.custom_models_host_charts_dict[host] = list(set([dim.split('::')[1].split('|')[0] for dim in self.custom_models_dims if dim.startswith(host)]))
107
- self.custom_models_dims_renamed = [f"{model['name']}|{dim}" for model in self.custom_models for dim in model['dimensions'].split(',')]
108
- self.models_in_scope = list(set([f'{self.host}::{c}' for c in self.charts_in_scope] + self.custom_models_names))
109
- self.charts_in_scope = list(set(self.charts_in_scope + self.custom_models_charts))
110
- self.host_charts_dict = {self.host: self.charts_in_scope}
111
- for host in self.custom_models_host_charts_dict:
112
- if host not in self.host_charts_dict:
113
- self.host_charts_dict[host] = self.custom_models_host_charts_dict[host]
114
- else:
115
- for chart in self.custom_models_host_charts_dict[host]:
116
- if chart not in self.host_charts_dict[host]:
117
- self.host_charts_dict[host].extend(chart)
118
- else:
119
- self.models_in_scope = [f'{self.host}::{c}' for c in self.charts_in_scope]
120
- self.host_charts_dict = {self.host: self.charts_in_scope}
121
- self.model_display_names = {model: model.split('::')[1] if '::' in model else model for model in self.models_in_scope}
122
- #self.info(f'self.host_charts_dict (len={len(self.host_charts_dict[self.host])}): {self.host_charts_dict}')
123
-
124
- def data_init(self):
125
- """Initialize some empty data objects.
126
- """
127
- self.data_probability_latest = {f'{m}_prob': 0 for m in self.charts_in_scope}
128
- self.data_anomaly_latest = {f'{m}_anomaly': 0 for m in self.charts_in_scope}
129
- self.data_latest = {**self.data_probability_latest, **self.data_anomaly_latest}
130
-
131
- def model_params_init(self):
132
- """Model parameters initialisation.
133
- """
134
- self.train_max_n = self.configuration.get('train_max_n', 100000)
135
- self.train_n_secs = self.configuration.get('train_n_secs', 14400)
136
- self.offset_n_secs = self.configuration.get('offset_n_secs', 0)
137
- self.train_every_n = self.configuration.get('train_every_n', 1800)
138
- self.train_no_prediction_n = self.configuration.get('train_no_prediction_n', 10)
139
- self.initial_train_data_after = self.configuration.get('initial_train_data_after', 0)
140
- self.initial_train_data_before = self.configuration.get('initial_train_data_before', 0)
141
- self.contamination = self.configuration.get('contamination', 0.001)
142
- self.lags_n = {model: self.configuration.get('lags_n', 5) for model in self.models_in_scope}
143
- self.smooth_n = {model: self.configuration.get('smooth_n', 5) for model in self.models_in_scope}
144
- self.diffs_n = {model: self.configuration.get('diffs_n', 5) for model in self.models_in_scope}
145
-
146
- def models_init(self):
147
- """Models initialisation.
148
- """
149
- self.model = self.configuration.get('model', 'pca')
150
- if self.model == 'pca':
151
- self.models = {model: PCA(contamination=self.contamination) for model in self.models_in_scope}
152
- elif self.model == 'loda':
153
- self.models = {model: LODA(contamination=self.contamination) for model in self.models_in_scope}
154
- elif self.model == 'iforest':
155
- self.models = {model: IForest(n_estimators=50, bootstrap=True, behaviour='new', contamination=self.contamination) for model in self.models_in_scope}
156
- elif self.model == 'cblof':
157
- self.models = {model: CBLOF(n_clusters=3, contamination=self.contamination) for model in self.models_in_scope}
158
- elif self.model == 'feature_bagging':
159
- self.models = {model: FeatureBagging(base_estimator=PCA(contamination=self.contamination), contamination=self.contamination) for model in self.models_in_scope}
160
- elif self.model == 'copod':
161
- self.models = {model: COPOD(contamination=self.contamination) for model in self.models_in_scope}
162
- elif self.model == 'hbos':
163
- self.models = {model: HBOS(contamination=self.contamination) for model in self.models_in_scope}
164
- else:
165
- self.models = {model: HBOS(contamination=self.contamination) for model in self.models_in_scope}
166
- self.custom_model_scalers = {model: MinMaxScaler() for model in self.models_in_scope}
167
-
168
- def model_init(self, model):
169
- """Model initialisation of a single model.
170
- """
171
- if self.model == 'pca':
172
- self.models[model] = PCA(contamination=self.contamination)
173
- elif self.model == 'loda':
174
- self.models[model] = LODA(contamination=self.contamination)
175
- elif self.model == 'iforest':
176
- self.models[model] = IForest(n_estimators=50, bootstrap=True, behaviour='new', contamination=self.contamination)
177
- elif self.model == 'cblof':
178
- self.models[model] = CBLOF(n_clusters=3, contamination=self.contamination)
179
- elif self.model == 'feature_bagging':
180
- self.models[model] = FeatureBagging(base_estimator=PCA(contamination=self.contamination), contamination=self.contamination)
181
- elif self.model == 'copod':
182
- self.models[model] = COPOD(contamination=self.contamination)
183
- elif self.model == 'hbos':
184
- self.models[model] = HBOS(contamination=self.contamination)
185
- else:
186
- self.models[model] = HBOS(contamination=self.contamination)
187
- self.custom_model_scalers[model] = MinMaxScaler()
188
-
189
- def reinitialize(self):
190
- """Reinitialize charts, models and data to a beginning state.
191
- """
192
- self.charts_init()
193
- self.custom_models_init()
194
- self.data_init()
195
- self.model_params_init()
196
- self.models_init()
197
-
198
- def save_data_latest(self, data, data_probability, data_anomaly):
199
- """Save the most recent data objects to be used if needed in the future.
200
- """
201
- self.data_latest = data
202
- self.data_probability_latest = data_probability
203
- self.data_anomaly_latest = data_anomaly
204
-
205
- def validate_charts(self, chart, data, algorithm='absolute', multiplier=1, divisor=1):
206
- """If dimension not in chart then add it.
207
- """
208
- for dim in data:
209
- if dim not in self.collected_dims[chart]:
210
- self.collected_dims[chart].add(dim)
211
- self.charts[chart].add_dimension([dim, dim, algorithm, multiplier, divisor])
212
-
213
- for dim in list(self.collected_dims[chart]):
214
- if dim not in data:
215
- self.collected_dims[chart].remove(dim)
216
- self.charts[chart].del_dimension(dim, hide=False)
217
-
218
- def add_custom_models_dims(self, df):
219
- """Given a df, select columns used by custom models, add custom model name as prefix, and append to df.
220
-
221
- :param df <pd.DataFrame>: dataframe to append new renamed columns to.
222
- :return: <pd.DataFrame> dataframe with additional columns added relating to the specified custom models.
223
- """
224
- df_custom = df[self.custom_models_dims].copy()
225
- df_custom.columns = self.custom_models_dims_renamed
226
- df = df.join(df_custom)
227
-
228
- return df
229
-
230
- def make_features(self, arr, train=False, model=None):
231
- """Take in numpy array and preprocess accordingly by taking diffs, smoothing and adding lags.
232
-
233
- :param arr <np.ndarray>: numpy array we want to make features from.
234
- :param train <bool>: True if making features for training, in which case need to fit_transform scaler and maybe sample train_max_n.
235
- :param model <str>: model to make features for.
236
- :return: <np.ndarray> transformed numpy array.
237
- """
238
-
239
- def lag(arr, n):
240
- res = np.empty_like(arr)
241
- res[:n] = np.nan
242
- res[n:] = arr[:-n]
243
-
244
- return res
245
-
246
- arr = np.nan_to_num(arr)
247
-
248
- diffs_n = self.diffs_n[model]
249
- smooth_n = self.smooth_n[model]
250
- lags_n = self.lags_n[model]
251
-
252
- if self.custom_models_normalize and model in self.custom_models_names:
253
- if train:
254
- arr = self.custom_model_scalers[model].fit_transform(arr)
255
- else:
256
- arr = self.custom_model_scalers[model].transform(arr)
257
-
258
- if diffs_n > 0:
259
- arr = np.diff(arr, diffs_n, axis=0)
260
- arr = arr[~np.isnan(arr).any(axis=1)]
261
-
262
- if smooth_n > 1:
263
- arr = np.cumsum(arr, axis=0, dtype=float)
264
- arr[smooth_n:] = arr[smooth_n:] - arr[:-smooth_n]
265
- arr = arr[smooth_n - 1:] / smooth_n
266
- arr = arr[~np.isnan(arr).any(axis=1)]
267
-
268
- if lags_n > 0:
269
- arr_orig = np.copy(arr)
270
- for lag_n in range(1, lags_n + 1):
271
- arr = np.concatenate((arr, lag(arr_orig, lag_n)), axis=1)
272
- arr = arr[~np.isnan(arr).any(axis=1)]
273
-
274
- if train:
275
- if len(arr) > self.train_max_n:
276
- arr = arr[np.random.randint(arr.shape[0], size=self.train_max_n), :]
277
-
278
- arr = np.nan_to_num(arr)
279
-
280
- return arr
281
-
282
- def train(self, models_to_train=None, train_data_after=0, train_data_before=0):
283
- """Pull required training data and train a model for each specified model.
284
-
285
- :param models_to_train <list>: list of models to train on.
286
- :param train_data_after <int>: integer timestamp for start of train data.
287
- :param train_data_before <int>: integer timestamp for end of train data.
288
- """
289
- now = datetime.now().timestamp()
290
- if train_data_after > 0 and train_data_before > 0:
291
- before = train_data_before
292
- after = train_data_after
293
- else:
294
- before = int(now) - self.offset_n_secs
295
- after = before - self.train_n_secs
296
-
297
- # get training data
298
- df_train = get_data(
299
- host_charts_dict=self.host_charts_dict, host_prefix=True, host_sep='::', after=after, before=before,
300
- sort_cols=True, numeric_only=True, protocol=self.protocol, float_size='float32', user=self.username, pwd=self.password,
301
- verify=self.tls_verify
302
- ).ffill()
303
- if self.custom_models:
304
- df_train = self.add_custom_models_dims(df_train)
305
-
306
- # train model
307
- self.try_fit(df_train, models_to_train=models_to_train)
308
- self.info(f'training complete in {round(time.time() - now, 2)} seconds (runs_counter={self.runs_counter}, model={self.model}, train_n_secs={self.train_n_secs}, models={len(self.fitted_at)}, n_fit_success={self.n_fit_success}, n_fit_fails={self.n_fit_fail}, after={after}, before={before}).')
309
- self.last_train_at = self.runs_counter
310
-
311
- def try_fit(self, df_train, models_to_train=None):
312
- """Try fit each model and try to fallback to a default model if fit fails for any reason.
313
-
314
- :param df_train <pd.DataFrame>: data to train on.
315
- :param models_to_train <list>: list of models to train.
316
- """
317
- if models_to_train is None:
318
- models_to_train = list(self.models.keys())
319
- self.n_fit_fail, self.n_fit_success = 0, 0
320
- for model in models_to_train:
321
- if model not in self.models:
322
- self.model_init(model)
323
- X_train = self.make_features(
324
- df_train[df_train.columns[df_train.columns.str.startswith(f'{model}|')]].values,
325
- train=True, model=model)
326
- try:
327
- self.models[model].fit(X_train)
328
- self.n_fit_success += 1
329
- except Exception as e:
330
- self.n_fit_fail += 1
331
- self.info(e)
332
- self.info(f'training failed for {model} at run_counter {self.runs_counter}, defaulting to hbos model.')
333
- self.models[model] = HBOS(contamination=self.contamination)
334
- self.models[model].fit(X_train)
335
- self.fitted_at[model] = self.runs_counter
336
-
337
- def predict(self):
338
- """Get latest data, make it into a feature vector, and get predictions for each available model.
339
-
340
- :return: (<dict>,<dict>) tuple of dictionaries, one for probability scores and the other for anomaly predictions.
341
- """
342
- # get recent data to predict on
343
- df_allmetrics = get_allmetrics_async(
344
- host_charts_dict=self.host_charts_dict, host_prefix=True, host_sep='::', wide=True, sort_cols=True,
345
- protocol=self.protocol, numeric_only=True, float_size='float32', user=self.username, pwd=self.password
346
- )
347
- if self.custom_models:
348
- df_allmetrics = self.add_custom_models_dims(df_allmetrics)
349
- self.df_allmetrics = self.df_allmetrics.append(df_allmetrics).ffill().tail((max(self.lags_n.values()) + max(self.smooth_n.values()) + max(self.diffs_n.values())) * 2)
350
-
351
- # get predictions
352
- data_probability, data_anomaly = self.try_predict()
353
-
354
- return data_probability, data_anomaly
355
-
356
- def try_predict(self):
357
- """Try make prediction and fall back to last known prediction if fails.
358
-
359
- :return: (<dict>,<dict>) tuple of dictionaries, one for probability scores and the other for anomaly predictions.
360
- """
361
- data_probability, data_anomaly = {}, {}
362
- for model in self.fitted_at.keys():
363
- model_display_name = self.model_display_names[model]
364
- try:
365
- X_model = np.nan_to_num(
366
- self.make_features(
367
- self.df_allmetrics[self.df_allmetrics.columns[self.df_allmetrics.columns.str.startswith(f'{model}|')]].values,
368
- model=model
369
- )[-1,:].reshape(1, -1)
370
- )
371
- data_probability[model_display_name + '_prob'] = np.nan_to_num(self.models[model].predict_proba(X_model)[-1][1]) * 10000
372
- data_anomaly[model_display_name + '_anomaly'] = self.models[model].predict(X_model)[-1]
373
- except Exception as _:
374
- #self.info(e)
375
- if model_display_name + '_prob' in self.data_latest:
376
- #self.info(f'prediction failed for {model} at run_counter {self.runs_counter}, using last prediction instead.')
377
- data_probability[model_display_name + '_prob'] = self.data_latest[model_display_name + '_prob']
378
- data_anomaly[model_display_name + '_anomaly'] = self.data_latest[model_display_name + '_anomaly']
379
- else:
380
- #self.info(f'prediction failed for {model} at run_counter {self.runs_counter}, skipping as no previous prediction.')
381
- continue
382
-
383
- return data_probability, data_anomaly
384
-
385
- def get_data(self):
386
-
387
- # initialize to what's available right now
388
- if self.reinitialize_at_every_step or len(self.host_charts_dict[self.host]) == 0:
389
- self.charts_init()
390
- self.custom_models_init()
391
- self.model_params_init()
392
-
393
- # if not all models have been trained then train those we need to
394
- if len(self.fitted_at) < len(self.models_in_scope):
395
- self.train(
396
- models_to_train=[m for m in self.models_in_scope if m not in self.fitted_at],
397
- train_data_after=self.initial_train_data_after,
398
- train_data_before=self.initial_train_data_before
399
- )
400
- # retrain all models as per schedule from config
401
- elif self.train_every_n > 0 and self.runs_counter % self.train_every_n == 0:
402
- self.reinitialize()
403
- self.train()
404
-
405
- # roll forward previous predictions around a training step to avoid the possibility of having the training itself trigger an anomaly
406
- if (self.runs_counter - self.last_train_at) <= self.train_no_prediction_n:
407
- data_probability = self.data_probability_latest
408
- data_anomaly = self.data_anomaly_latest
409
- else:
410
- data_probability, data_anomaly = self.predict()
411
- if self.include_average_prob:
412
- average_prob = np.mean(list(data_probability.values()))
413
- data_probability['average_prob'] = 0 if np.isnan(average_prob) else average_prob
414
-
415
- data = {**data_probability, **data_anomaly}
416
-
417
- self.validate_charts('probability', data_probability, divisor=100)
418
- self.validate_charts('anomaly', data_anomaly)
419
-
420
- self.save_data_latest(data, data_probability, data_anomaly)
421
-
422
- #self.info(f'len(data)={len(data)}')
423
- #self.info(f'data')
424
-
425
- return data
src/collectors/python.d.plugin/anomalies/anomalies.conf
deleted
-184
@@ -1,184 +0,0 @@
1
-# netdata python.d.plugin configuration for anomalies
2
-#
3
-# This file is in YaML format. Generally the format is:
4
-#
5
-# name: value
6
-#
7
-# There are 2 sections:
8
-# - global variables
9
-# - one or more JOBS
10
-#
11
-# JOBS allow you to collect values from multiple sources.
12
-# Each source will have its own set of charts.
13
-#
14
-# JOB parameters have to be indented (using spaces only, example below).
15
-
16
-# ----------------------------------------------------------------------
17
-# Global Variables
18
-# These variables set the defaults for all JOBs, however each JOB
19
-# may define its own, overriding the defaults.
20
-
21
-# update_every sets the default data collection frequency.
22
-# If unset, the python.d.plugin default is used.
23
-# update_every: 2
24
-
25
-# priority controls the order of charts at the netdata dashboard.
26
-# Lower numbers move the charts towards the top of the page.
27
-# If unset, the default for python.d.plugin is used.
28
-# priority: 60000
29
-
30
-# ----------------------------------------------------------------------
31
-# JOBS (data collection sources)
32
-
33
-# Pull data from local Netdata node.
34
-anomalies:
35
- name: 'Anomalies'
36
-
37
- # Host to pull data from.
38
- host: '127.0.0.1:19999'
39
-
40
- # Username and Password for Netdata if using basic auth.
41
- # username: '???'
42
- # password: '???'
43
-
44
- # Use http or https to pull data
45
- protocol: 'http'
46
-
47
- # SSL verify parameter for requests.get() calls
48
- tls_verify: true
49
-
50
- # What charts to pull data for - A regex like 'system\..*|' or 'system\..*|apps.cpu|apps.mem' etc.
51
- charts_regex: 'system\..*'
52
-
53
- # Charts to exclude, useful if you would like to exclude some specific charts.
54
- # Note: should be a ',' separated string like 'chart.name,chart.name'.
55
- charts_to_exclude: 'system.uptime,system.entropy'
56
-
57
- # What model to use - can be one of 'pca', 'hbos', 'iforest', 'cblof', 'loda', 'copod' or 'feature_bagging'.
58
- # More details here: https://pyod.readthedocs.io/en/latest/pyod.models.html.
59
- model: 'pca'
60
-
61
- # Max number of observations to train on, to help cap compute cost of training model if you set a very large train_n_secs.
62
- train_max_n: 100000
63
-
64
- # How often to re-train the model (assuming update_every=1 then train_every_n=1800 represents (re)training every 30 minutes).
65
- # Note: If you want to turn off re-training set train_every_n=0 and after initial training the models will not be retrained.
66
- train_every_n: 1800
67
-
68
- # The length of the window of data to train on (14400 = last 4 hours).
69
- train_n_secs: 14400
70
-
71
- # How many prediction steps after a train event to just use previous prediction value for.
72
- # Used to reduce possibility of the training step itself appearing as an anomaly on the charts.
73
- train_no_prediction_n: 10
74
-
75
- # If you would like to train the model for the first time on a specific window then you can define it using the below two variables.
76
- # Start of training data for initial model.
77
- # initial_train_data_after: 1604578857
78
-
79
- # End of training data for initial model.
80
- # initial_train_data_before: 1604593257
81
-
82
- # If you would like to ignore recent data in training then you can offset it by offset_n_secs.
83
- offset_n_secs: 0
84
-
85
- # How many lagged values of each dimension to include in the 'feature vector' each model is trained on.
86
- lags_n: 5
87
-
88
- # How much smoothing to apply to each dimension in the 'feature vector' each model is trained on.
89
- smooth_n: 3
90
-
91
- # How many differences to take in preprocessing your data.
92
- # More info on differencing here: https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average#Differencing
93
- # diffs_n=0 would mean training models on the raw values of each dimension.
94
- # diffs_n=1 means everything is done in terms of differences.
95
- diffs_n: 1
96
-
97
- # What is the typical proportion of anomalies in your data on average?
98
- # This parameter can control the sensitivity of your models to anomalies.
99
- # Some discussion here: https://github.com/yzhao062/pyod/issues/144
100
- contamination: 0.001
101
-
102
- # Set to true to include an "average_prob" dimension on anomalies probability chart which is
103
- # just the average of all anomaly probabilities at each time step
104
- include_average_prob: true
105
-
106
- # Define any custom models you would like to create anomaly probabilities for, some examples below to show how.
107
- # For example below example creates two custom models, one to run anomaly detection user and system cpu for our demo servers
108
- # and one on the cpu and mem apps metrics for the python.d.plugin.
109
- # custom_models:
110
- # - name: 'demos_cpu'
111
- # dimensions: 'london.my-netdata.io::system.cpu|user,london.my-netdata.io::system.cpu|system,newyork.my-netdata.io::system.cpu|user,newyork.my-netdata.io::system.cpu|system'
112
- # - name: 'apps_python_d_plugin'
113
- # dimensions: 'apps.cpu|python.d.plugin,apps.mem|python.d.plugin'
114
-
115
- # Set to true to normalize, using min-max standardization, features used for the custom models.
116
- # Useful if your custom models contain dimensions on very different scales an model you use does
117
- # not internally do its own normalization. Usually best to leave as false.
118
- # custom_models_normalize: false
119
-
120
-# Standalone Custom models example as an additional collector job.
121
-# custom:
122
-# name: 'custom'
123
-# host: '127.0.0.1:19999'
124
-# protocol: 'http'
125
-# charts_regex: 'None'
126
-# charts_to_exclude: 'None'
127
-# model: 'pca'
128
-# train_max_n: 100000
129
-# train_every_n: 1800
130
-# train_n_secs: 14400
131
-# offset_n_secs: 0
132
-# lags_n: 5
133
-# smooth_n: 3
134
-# diffs_n: 1
135
-# contamination: 0.001
136
-# custom_models:
137
-# - name: 'user_netdata'
138
-# dimensions: 'users.cpu|netdata,users.mem|netdata,users.threads|netdata,users.processes|netdata,users.sockets|netdata'
139
-# - name: 'apps_python_d_plugin'
140
-# dimensions: 'apps.cpu|python.d.plugin,apps.mem|python.d.plugin,apps.threads|python.d.plugin,apps.processes|python.d.plugin,apps.sockets|python.d.plugin'
141
-
142
-# Pull data from some demo nodes for cross node custom models.
143
-# demos:
144
-# name: 'demos'
145
-# host: '127.0.0.1:19999'
146
-# protocol: 'http'
147
-# charts_regex: 'None'
148
-# charts_to_exclude: 'None'
149
-# model: 'pca'
150
-# train_max_n: 100000
151
-# train_every_n: 1800
152
-# train_n_secs: 14400
153
-# offset_n_secs: 0
154
-# lags_n: 5
155
-# smooth_n: 3
156
-# diffs_n: 1
157
-# contamination: 0.001
158
-# custom_models:
159
-# - name: 'system.cpu'
160
-# dimensions: 'london.my-netdata.io::system.cpu|user,london.my-netdata.io::system.cpu|system,newyork.my-netdata.io::system.cpu|user,newyork.my-netdata.io::system.cpu|system'
161
-# - name: 'system.ip'
162
-# dimensions: 'london.my-netdata.io::system.ip|received,london.my-netdata.io::system.ip|sent,newyork.my-netdata.io::system.ip|received,newyork.my-netdata.io::system.ip|sent'
163
-# - name: 'system.net'
164
-# dimensions: 'london.my-netdata.io::system.net|received,london.my-netdata.io::system.net|sent,newyork.my-netdata.io::system.net|received,newyork.my-netdata.io::system.net|sent'
165
-# - name: 'system.io'
166
-# dimensions: 'london.my-netdata.io::system.io|in,london.my-netdata.io::system.io|out,newyork.my-netdata.io::system.io|in,newyork.my-netdata.io::system.io|out'
167
-
168
-# Example additional job if you want to also pull data from a child streaming to your
169
-# local parent or even a remote node so long as the Netdata REST API is accessible.
170
-# mychildnode1:
171
-# name: 'mychildnode1'
172
-# host: '127.0.0.1:19999/host/mychildnode1'
173
-# protocol: 'http'
174
-# charts_regex: 'system\..*'
175
-# charts_to_exclude: 'None'
176
-# model: 'pca'
177
-# train_max_n: 100000
178
-# train_every_n: 1800
179
-# train_n_secs: 14400
180
-# offset_n_secs: 0
181
-# lags_n: 5
182
-# smooth_n: 3
183
-# diffs_n: 1
184
-# contamination: 0.001
src/collectors/python.d.plugin/anomalies/metadata.yaml
deleted
-87
@@ -1,87 +0,0 @@
1
-# NOTE: this file is commented out as users are reccomended to use the
2
-# native anomaly detection capabilities on the agent instead.
3
-# meta:
4
-# plugin_name: python.d.plugin
5
-# module_name: anomalies
6
-# monitored_instance:
7
-# name: python.d anomalies
8
-# link: ""
9
-# categories: []
10
-# icon_filename: ""
11
-# related_resources:
12
-# integrations:
13
-# list: []
14
-# info_provided_to_referring_integrations:
15
-# description: ""
16
-# keywords: []
17
-# most_popular: false
18
-# overview:
19
-# data_collection:
20
-# metrics_description: ""
21
-# method_description: ""
22
-# supported_platforms:
23
-# include: []
24
-# exclude: []
25
-# multi_instance: true
26
-# additional_permissions:
27
-# description: ""
28
-# default_behavior:
29
-# auto_detection:
30
-# description: ""
31
-# limits:
32
-# description: ""
33
-# performance_impact:
34
-# description: ""
35
-# setup:
36
-# prerequisites:
37
-# list: []
38
-# configuration:
39
-# file:
40
-# name: ""
41
-# description: ""
42
-# options:
43
-# description: ""
44
-# folding:
45
-# title: ""
46
-# enabled: true
47
-# list: []
48
-# examples:
49
-# folding:
50
-# enabled: true
51
-# title: ""
52
-# list: []
53
-# troubleshooting:
54
-# problems:
55
-# list: []
56
-# alerts:
57
-# - name: anomalies_anomaly_probabilities
58
-# link: https://github.com/netdata/netdata/blob/master/src/health/health.d/anomalies.conf
59
-# metric: anomalies.probability
60
-# info: average anomaly probability over the last 2 minutes
61
-# - name: anomalies_anomaly_flags
62
-# link: https://github.com/netdata/netdata/blob/master/src/health/health.d/anomalies.conf
63
-# metric: anomalies.anomaly
64
-# info: number of anomalies in the last 2 minutes
65
-# metrics:
66
-# folding:
67
-# title: Metrics
68
-# enabled: false
69
-# description: ""
70
-# availability: []
71
-# scopes:
72
-# - name: global
73
-# description: ""
74
-# labels: []
75
-# metrics:
76
-# - name: anomalies.probability
77
-# description: Anomaly Probability
78
-# unit: "probability"
79
-# chart_type: line
80
-# dimensions:
81
-# - name: a dimension per probability
82
-# - name: anomalies.anomaly
83
-# description: Anomaly
84
-# unit: "count"
85
-# chart_type: stacked
86
-# dimensions:
87
-# - name: a dimension per anomaly
src/collectors/python.d.plugin/python.d.conf
-1
@@ -26,7 +26,6 @@ gc_run: yes
26
gc_interval: 300
27
28
# am2320: yes
29
-# anomalies: no
29
# ceph: yes
30
# this is just an example
31
go_expvar: no
src/health/health.d/anomalies.conf
deleted
-25
@@ -1,25 +0,0 @@
1
-## raise a warning alarm if an anomaly probability is consistently above 50%
2
-
3
-## "foreach" was removed, these alarms don't work anymore
4
-
5
-# template: anomalies_anomaly_probabilities
6
-# on: anomalies.probability
7
-# class: Errors
8
-# type: Netdata
9
-#component: ML
10
-# lookup: average -2m foreach *
11
-# every: 1m
12
-# warn: $this > 50
13
-# info: average anomaly probability over the last 2 minutes
14
-
15
-# raise a warning alarm if an anomaly flag is consistently firing
16
-
17
-# template: anomalies_anomaly_flags
18
-# on: anomalies.anomaly
19
-# class: Errors
20
-# type: Netdata
21
-#component: ML
22
-# lookup: sum -2m foreach *
23
-# every: 1m
24
-# warn: $this > 10
25
-# info: number of anomalies in the last 2 minutes