some collector metadata improvements (#16008)
* add sensors throubleshooting * add fail2ban debug mode * add changefinder metadata.yaml * fix formatting * fix yamlint * fix yamlint
Andrew Maguire committed
Sep 20, 2023 at 10:42 UTC
085c469c98e218084853e964369a04f8fc772625
3 files changed
+195
-34
collectors/python.d.plugin/changefinder/metadata.yaml
+151
-19
@@ -5,55 +5,187 @@ modules:
5
module_name: changefinder
6
monitored_instance:
7
name: python.d changefinder
8
- link: ''
8
+ link: ""
9
categories:
10
- data-collection.other
11
- icon_filename: ''
11
+ icon_filename: ""
12
related_resources:
13
integrations:
14
list: []
15
info_provided_to_referring_integrations:
16
- description: ''
17
- keywords: []
16
+ description: ""
17
+ keywords:
18
+ - change detection
19
+ - anomaly detection
20
+ - machine learning
21
+ - ml
22
most_popular: false
23
overview:
24
data_collection:
21
- metrics_description: ''
22
- method_description: ''
25
+ metrics_description: |
26
+ This collector uses the Python [changefinder](https://github.com/shunsukeaihara/changefinder) library to
27
+ perform [online](https://en.wikipedia.org/wiki/Online_machine_learning) [changepoint detection](https://en.wikipedia.org/wiki/Change_detection)
28
+ on your Netdata charts and/or dimensions.
29
+ method_description: >
30
+ Instead of this collector just _collecting_ data, it also does some computation on the data it collects to return a
31
+ changepoint score for each chart or dimension you configure it to work on. This is
32
+ an [online](https://en.wikipedia.org/wiki/Online_machine_learning) machine learning algorithm so there is no batch step
33
+ to train the model, instead it evolves over time as more data arrives. That makes this particular algorithm quite cheap
34
+ to compute at each step of data collection (see the notes section below for more details) and it should scale fairly
35
+ well to work on lots of charts or hosts (if running on a parent node for example).
36
+
37
+ ### Notes
38
+ - It may take an hour or two (depending on your choice of `n_score_samples`) for the collector to 'settle' into it's
39
+ typical behaviour in terms of the trained models and scores you will see in the normal running of your node. Mainly
40
+ this is because it can take a while to build up a proper distribution of previous scores in over to convert the raw
41
+ score returned by the ChangeFinder algorithm into a percentile based on the most recent `n_score_samples` that have
42
+ already been produced. So when you first turn the collector on, it will have a lot of flags in the beginning and then
43
+ should 'settle down' once it has built up enough history. This is a typical characteristic of online machine learning
44
+ approaches which need some initial window of time before they can be useful.
45
+ - As this collector does most of the work in Python itself, you may want to try it out first on a test or development
46
+ system to get a sense of its performance characteristics on a node similar to where you would like to use it.
47
+ - On a development n1-standard-2 (2 vCPUs, 7.5 GB memory) vm running Ubuntu 18.04 LTS and not doing any work some of the
48
+ typical performance characteristics we saw from running this collector (with defaults) were:
49
+ - A runtime (`netdata.runtime_changefinder`) of ~30ms.
50
+ - Typically ~1% additional cpu usage.
51
+ - About ~85mb of ram (`apps.mem`) being continually used by the `python.d.plugin` under default configuration.
52
supported_platforms:
53
include: []
54
exclude: []
55
multi_instance: true
56
additional_permissions:
28
- description: ''
57
+ description: ""
58
default_behavior:
59
auto_detection:
31
- description: ''
60
+ description: "By default this collector will work over all `system.*` charts."
61
limits:
33
- description: ''
62
+ description: ""
63
performance_impact:
35
- description: ''
64
+ description: ""
65
setup:
66
prerequisites:
38
- list: []
67
+ list:
68
+ - title: Python Requirements
69
+ description: |
70
+ This collector will only work with Python 3 and requires the packages below be installed.
71
+
72
+ ```bash
73
+ # become netdata user
74
+ sudo su -s /bin/bash netdata
75
+ # install required packages for the netdata user
76
+ pip3 install --user numpy==1.19.5 changefinder==0.03 scipy==1.5.4
77
+ ```
78
+
79
+ **Note**: if you need to tell Netdata to use Python 3 then you can pass the below command in the python plugin section
80
+ of your `netdata.conf` file.
81
+
82
+ ```yaml
83
+ [ plugin:python.d ]
84
+ # update every = 1
85
+ command options = -ppython3
86
+ ```
87
configuration:
88
file:
41
- name: ''
42
- description: ''
89
+ name: python.d/changefinder.conf
90
+ description: ""
91
options:
44
- description: ''
92
+ description: |
93
+ There are 2 sections:
94
+
95
+ * Global variables
96
+ * One or more JOBS that can define multiple different instances to monitor.
97
+
98
+ The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
99
+
100
+ Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
101
+
102
+ Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
103
folding:
46
- title: ''
104
+ title: "Config options"
105
enabled: true
48
- list: []
106
+ list:
107
+ - name: charts_regex
108
+ description: what charts to pull data for - A regex like `system\..*|` or `system\..*|apps.cpu|apps.mem` etc.
109
+ default_value: "system\\..*"
110
+ required: true
111
+ - name: charts_to_exclude
112
+ description: |
113
+ charts to exclude, useful if you would like to exclude some specific charts.
114
+ note: should be a ',' separated string like 'chart.name,chart.name'.
115
+ default_value: ""
116
+ required: false
117
+ - name: mode
118
+ description: get ChangeFinder scores 'per_dim' or 'per_chart'.
119
+ default_value: "per_chart"
120
+ required: true
121
+ - name: cf_r
122
+ description: default parameters that can be passed to the changefinder library.
123
+ default_value: 0.5
124
+ required: false
125
+ - name: cf_order
126
+ description: default parameters that can be passed to the changefinder library.
127
+ default_value: 1
128
+ required: false
129
+ - name: cf_smooth
130
+ description: default parameters that can be passed to the changefinder library.
131
+ default_value: 15
132
+ required: false
133
+ - name: cf_threshold
134
+ description: the percentile above which scores will be flagged.
135
+ default_value: 99
136
+ required: false
137
+ - name: n_score_samples
138
+ description: the number of recent scores to use when calculating the percentile of the changefinder score.
139
+ default_value: 14400
140
+ required: false
141
+ - name: show_scores
142
+ description: |
143
+ set to true if you also want to chart the percentile scores in addition to the flags. (mainly useful for debugging or if you want to dive deeper on how the scores are evolving over time)
144
+ default_value: false
145
+ required: false
146
examples:
147
folding:
148
enabled: true
52
- title: ''
53
- list: []
149
+ title: "Config"
150
+ list:
151
+ - name: Default
152
+ description: Default configuration.
153
+ folding:
154
+ enabled: false
155
+ config: |
156
+ local:
157
+ name: 'local'
158
+ host: '127.0.0.1:19999'
159
+ charts_regex: 'system\..*'
160
+ charts_to_exclude: ''
161
+ mode: 'per_chart'
162
+ cf_r: 0.5
163
+ cf_order: 1
164
+ cf_smooth: 15
165
+ cf_threshold: 99
166
+ n_score_samples: 14400
167
+ show_scores: false
168
troubleshooting:
169
problems:
56
- list: []
170
+ list:
171
+ - name: "Debug Mode"
172
+ description: |
173
+ If you would like to log in as `netdata` user and run the collector in debug mode to see more detail.
174
+
175
+ ```bash
176
+ # become netdata user
177
+ sudo su -s /bin/bash netdata
178
+ # run collector in debug using `nolock` option if netdata is already running the collector itself.
179
+ /usr/libexec/netdata/plugins.d/python.d.plugin changefinder debug trace nolock
180
+ ```
181
+ - name: "Log Messages"
182
+ description: |
183
+ To see any relevant log messages you can use a command like below.
184
+
185
+ ```bash
186
+ grep 'changefinder' /var/log/netdata/error.log
187
+ grep 'changefinder' /var/log/netdata/collector.log
188
+ ```
189
alerts: []
190
metrics:
191
folding:
collectors/python.d.plugin/fail2ban/metadata.yaml
+34
-14
@@ -35,29 +35,29 @@ modules:
35
The `fail2ban.log` file must be readable by the user `netdata`.
36
- change the file ownership and access permissions.
37
- update `/etc/logrotate.d/fail2ban`` to persist the changes after rotating the log file.
38
-
38
+
39
To change the file ownership and access permissions, execute the following:
40
-
40
+
41
```shell
42
sudo chown root:netdata /var/log/fail2ban.log
43
sudo chmod 640 /var/log/fail2ban.log
44
```
45
-
45
+
46
To persist the changes after rotating the log file, add `create 640 root netdata` to the `/etc/logrotate.d/fail2ban`:
47
-
47
+
48
```shell
49
/var/log/fail2ban.log {
50
-
50
+
51
weekly
52
rotate 4
53
compress
54
-
54
+
55
delaycompress
56
missingok
57
postrotate
58
fail2ban-client flushlogs 1>/dev/null
59
endscript
60
-
60
+
61
# If fail2ban runs as non-root it still needs to have write access
62
# to logfiles.
63
# create 640 fail2ban adm
@@ -67,7 +67,8 @@ modules:
67
default_behavior:
68
auto_detection:
69
description: |
70
- By default the collector will attempt to read log file at /var/log/fail2ban.log and conf file at /etc/fail2ban/jail.local. If conf file is not found default jail is ssh.
70
+ By default the collector will attempt to read log file at /var/log/fail2ban.log and conf file at /etc/fail2ban/jail.local.
71
+ If conf file is not found default jail is ssh.
72
limits:
73
description: ""
74
performance_impact:
@@ -77,19 +78,19 @@ modules:
78
list: []
79
configuration:
80
file:
80
- name: ""
81
+ name: python.d/fail2ban.conf
82
description: ""
83
options:
84
description: |
85
There are 2 sections:
85
-
86
+
87
* Global variables
88
* One or more JOBS that can define multiple different instances to monitor.
88
-
89
+
90
The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
90
-
91
+
92
Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
92
-
93
+
94
Every configuration JOB starts with a `job_name` value which will appear in the dashboard, unless a `name` parameter is specified.
95
folding:
96
title: Config options
@@ -146,7 +147,26 @@ modules:
147
conf_path: '/etc/fail2ban/jail.local'
148
troubleshooting:
149
problems:
149
- list: []
150
+ list:
151
+ - name: Debug Mode
152
+ description: |
153
+ To troubleshoot issues with the `fail2ban` module, run the `python.d.plugin` with the debug option enabled.
154
+ The output will give you the output of the data collection job or error messages on why the collector isn't working.
155
+
156
+ First, navigate to your plugins directory, usually they are located under `/usr/libexec/netdata/plugins.d/`. If that's
157
+ not the case on your system, open `netdata.conf` and look for the setting `plugins directory`. Once you're in the
158
+ plugin's directory, switch to the `netdata` user.
159
+
160
+ ```bash
161
+ cd /usr/libexec/netdata/plugins.d/
162
+ sudo su -s /bin/bash netdata
163
+ ```
164
+
165
+ Now you can manually run the `fail2ban` module in debug mode:
166
+
167
+ ```bash
168
+ ./python.d.plugin fail2ban debug trace
169
+ ```
170
alerts: []
171
metrics:
172
folding:
collectors/python.d.plugin/sensors/metadata.yaml
+10
-1
@@ -117,7 +117,16 @@ modules:
117
- humidity
118
troubleshooting:
119
problems:
120
- list: []
120
+ list:
121
+ - name: lm-sensors doesn't work on your device
122
+ description: |
123
+ When `lm-sensors` doesn't work on your device (e.g. for RPi temperatures),
124
+ use [the legacy bash collector](https://github.com/netdata/netdata/blob/master/collectors/charts.d.plugin/sensors/README.md)
125
+ - name: ACPI ring buffer errors are printed
126
+ description: |
127
+ There have been reports from users that on certain servers, ACPI ring buffer errors are printed by the kernel (`dmesg`)
128
+ when ACPI sensors are being accessed. We are tracking such cases in issue [#827](https://github.com/netdata/netdata/issues/827).
129
+ Please join this discussion for help.
130
alerts: []
131
metrics:
132
folding: