Anomaly Advisor documentation edits (#17518)
* anomaly advisor documentation edits * replace links * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Shyam Sreevalsan <shyam@netdata.cloud> * Apply suggestions from code review Co-authored-by: Shyam Sreevalsan <shyam@netdata.cloud> --------- Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud> Co-authored-by: Shyam Sreevalsan <shyam@netdata.cloud>
Fotis Voutsas committed
Apr 26, 2024 at 10:44 UTC
234f4a6811c5e609a5f39fa9bc2e54a62479bcdf
7 files changed
+35
-95
docs/cloud/alerts-notifications/notifications.md
+1
-1
@@ -11,7 +11,7 @@ Having this information centralized helps you:
11
* Easily [set up your alert notification process](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/manage-notification-methods.md):
12
methods to use and where to use them, filtering rules, etc.
13
* Quickly troubleshoot using [Metric Correlations](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/metric-correlations.md)
14
-or [Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md)
14
+or [Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/dashboard/anomaly-advisor-tab.md)
15
16
If a node is getting disconnected often or has many alerts, we protect you and your team from alert fatigue by sending
17
you a flood protection notification. Getting one of these notifications is a good signal of health or performance issues
docs/cloud/insights/anomaly-advisor.md
deleted
-87
@@ -1,87 +0,0 @@
1
-<!--
2
-title: "Anomaly Advisor"
3
-description: "Quickly find anomalous metrics anywhere in your infrastructure."
4
-custom_edit_url: "https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md"
5
-sidebar_label: "Anomaly Advisor"
6
-learn_status: "Published"
7
-learn_topic_type: "Tasks"
8
-learn_rel_path: "Operations"
9
--->
10
-
11
-# Anomaly Advisor
12
-
13
-import ReactPlayer from 'react-player'
14
-
15
-The Anomaly Advisor feature lets you quickly surface potentially anomalous metrics and charts related to a particular highlight window of
16
-interest.
17
-
18
-<ReactPlayer playing true controls true url='https://user-images.githubusercontent.com/24860547/165943403-1acb9759-7446-4704-8955-c566d04ad7ab.mp4' />
19
-
20
-## Getting Started
21
-
22
-If you are running a Netdata version higher than `v1.35.0-29-nightly` you will be able to use the Anomaly Advisor out of the box with zero configuration. If you are on an earlier Netdata version you will need to first enable ML on your nodes by following the steps below.
23
-
24
-To enable the Anomaly Advisor you must first enable ML on your nodes via a small config change in `netdata.conf`. Once the anomaly detection models have trained on the Agent (with default settings this takes a couple of hours until enough data has been seen to train the models) you will then be able to enable the Anomaly Advisor feature in Netdata Cloud.
25
-
26
-### Enable ML on Netdata Agent
27
-
28
-To enable ML on your Netdata Agent, you need to edit the `[ml]` section in your `netdata.conf` to look something like the following example.
29
-
30
-```bash
31
-[ml]
32
- enabled = yes
33
-```
34
-
35
-At a minimum you just need to set `enabled = yes` to enable ML with default params. More details about configuration can be found in the [Netdata Agent ML docs](https://github.com/netdata/netdata/blob/master/src/ml/README.md#configuration).
36
-
37
-When you have finished your configuration, restart Netdata with a command like `sudo systemctl restart netdata` for the config changes to take effect. You can find more info on restarting Netdata [here](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md#maintaining-a-netdata-agent-installation).
38
-
39
-After a brief delay, you should see the number of `trained` dimensions start to increase on the "dimensions" chart of the "Anomaly Detection" menu on the Overview page. By default the `minimum num samples to train = 3600` parameter means at least 1 hour of data is required to train initial models, but you could set this to `900` if you want to train initial models quicker but on less data. Over time, they will retrain on up to `maximum num samples to train = 14400` (4 hours by default), but you could increase this is you wanted to train on more data.
40
-
41
-
42
-
43
-Once this line flattens out all configured metrics should have models trained and predicting anomaly scores each second, ready to be used by the new "anomalies" tab of the Anomaly Advisor.
44
-
45
-## Using Anomaly Advisor
46
-
47
-To use the Anomaly Advisor, go to the "anomalies" tab. Once you highlight a particular timeframe of interest, a selection of the most anomalous dimensions will appear below.
48
-
49
-The aim here is to surface the most anomalous metrics in the space or room for the highlighted window to try and cut down on the amount of manual searching required to get to the root cause of your issues.
50
-
51
-
52
-
53
-The "Anomaly Rate" chart shows the percentage of anomalous metrics over time per node. For example, in the following image, 3.21% of the metrics on the "ml-demo-ml-disabled" node were considered anomalous. This elevated anomaly rate could be a sign of something worth investigating.
54
-
55
-**Note**: in this example the anomaly rates for this node are actually being calculated on the parent it streams to, you can run ml on the Agent itselt or on a parent the Agent stream to. Read more about the various configuration options in the [Agent docs](https://github.com/netdata/netdata/blob/master/src/ml/README.md).
56
-
57
-
58
-
59
-The "Count of Anomalous Metrics" chart (collapsed by default) shows raw counts of anomalous metrics per node so may often be similar to the anomaly rate chart, apart from where nodes may have different numbers of metrics.
60
-
61
-The "Anomaly Events Detected" chart (collapsed by default) shows if the anomaly rate per node was sufficiently elevated to trigger a node level anomaly. Anomaly events will appear slightly after the anomaly rate starts to increase in the timeline, this is because a significant number of metrics in the node need to be anomalous before an anomaly event is triggered.
62
-
63
-Once you have highlighted a window of interest, you should see an ordered list of anomaly rate sparklines in the "Anomalous metrics" section like below.
64
-
65
-
66
-
67
-You can expand any sparkline chart to see the underlying raw data to see how it relates to the corresponding anomaly rate.
68
-
69
-
70
-
71
-On the upper right hand side of the page you can select which nodes to filter on if you wish to do so. The ML training status of each node is also displayed.
72
-
73
-On the lower right hand side of the page an index of anomaly rates is displayed for the highlighted timeline of interest. The index is sorted from most anomalous metric (highest anomaly rate) to least (lowest anomaly rate). Clicking on an entry in the index will scroll the rest of the page to the corresponding anomaly rate sparkline for that metric.
74
-
75
-### Usage Tips
76
-
77
-- If you are interested in a subset of specific nodes then filtering to just those nodes before highlighting tends to give better results. This is because when you highlight a region, Netdata Cloud will ask the Agents for a ranking over all metrics so if you can filter this early to just the subset of nodes you are interested in, less 'averaging' will occur and so you might be a less noisy ranking.
78
-- Ideally try and highlight close to a spike or window of interest so that the resulting ranking can narrow in more easily on the timeline you are interested in.
79
-
80
-You can read more detail on how anomaly detection in the Netdata Agent works in our [Agent docs](https://github.com/netdata/netdata/blob/master/src/ml/README.md).
81
-
82
-🚧 **Note**: This functionality is still **under active development** and considered experimental. We dogfood it internally and among early adopters within the Netdata community to build the feature. If you would like to get involved and help us with feedback, you can reach us through any of the following channels:
83
-
84
-- Email us at analytics-ml-team@netdata.cloud
85
-- Comment on the [beta launch post](https://community.netdata.cloud/t/anomaly-advisor-beta-launch/2717) in the Netdata community
86
-- Join us in the [🤖-ml-powered-monitoring](https://discord.gg/4eRSEUpJnc) channel of the Netdata discord.
87
-- Or open a discussion in GitHub if that's more your thing
docs/dashboard/anomaly-advisor-tab.md
new
+27
@@ -0,0 +1,27 @@
1
+# Anomaly Advisor tab
2
+
3
+The Anomaly Advisor tab lets you focus on potentially anomalous metrics and charts related to a particular highlighted window of interest. In addition to this tab, each chart in the [Metrics tab](https://github.com/netdata/netdata/blob/master/docs/dashboard/metrics-tab-and-single-node-tabs.md) also has an [Anomaly Rate ribbon](https://github.com/netdata/netdata/blob/master/docs/dashboard/netdata-charts.md#anomaly-rate-ribbon).
4
+
5
+
6
+More details about configuration can be found in the [ML documentation](https://github.com/netdata/netdata/blob/master/src/ml/README.md#configuration).
7
+
8
+This tab uses our [Anomaly Rate ML feature](https://github.com/netdata/netdata/blob/master/src/ml/README.md#anomaly-rate---averageanomaly-bit) to score metrics in terms of anomalous behavior.
9
+
10
+- The "Anomaly Rate" chart shows the percentage of anomalous metrics over time per node.
11
+
12
+- The "Count of Anomalous Metrics" chart shows raw counts of anomalous metrics per node so may often be similar to the Anomaly Rate chart, apart from where nodes may have different numbers of metrics.
13
+
14
+- The "Anomaly Events Detected" chart shows whether the anomaly rate per node has increased enough to cause a node-level anomaly. Anomaly events will appear slightly after the anomaly rate starts to increase in the timeline, this is because a significant number of metrics in the node need to be anomalous before an anomaly event is triggered.
15
+
16
+Once you have highlighted a window of interest, you should see an ordered list of charts, with the Anomaly Rate being displayed as a purple ribbon in the chart.
17
+
18
+> **Tip**
19
+>
20
+> You can also use the [node filter](https://github.com/netdata/netdata/blob/master/docs/dashboard/node-filter.md) to select which nodes you want to include or exclude.
21
+
22
+The right side of the page displays an anomaly index for the highlighted timeline of interest. The index is sorted from most anomalous (highest level of anomaly) to least (lowest level of anomaly). Clicking on an entry in the index will get you to the corresponding chart for the anomalous metric.
23
+
24
+## Usage Tips
25
+
26
+- If you are interested in a subset of specific nodes then filtering to just those nodes before highlighting is recommended to get better results. When you highlight a timeframe, Netdata will ask the Agents for a ranking across all metrics, so if there is a subset of nodes there will be less "averaging" going on and you'll get a less noisy ranking.
27
+- Ideally try and highlight close to a spike or window of interest so that the resulting ranking can narrow-in more easily on the timeline you are interested in.
docs/glossary.md
+1
-1
@@ -23,7 +23,7 @@ Use the alphabatized list below to find the answer to your single-term questions
23
24
- [**Alarm Entity Type**](https://github.com/netdata/netdata/blob/master/src/health/REFERENCE.md#health-entity-reference): Entity types that are attached to specific charts and use the `alarm` label.
25
26
-- [**Anomaly Advisor**](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md): A Netdata feature that lets you quickly surface potentially anomalous metrics and charts related to a particular highlight window of interest.
26
+- [**Anomaly Advisor**](https://github.com/netdata/netdata/blob/master/docs/dashboard/anomaly-advisor-tab.md): A Netdata feature that lets you focus on potentially anomalous metrics and charts related to a particular highlight window of interest.
27
28
## B
29
docs/guides/monitor/anomaly-detection.md
+2
-2
@@ -35,7 +35,7 @@ This guide will explain how to get started using these ML based anomaly detectio
35
36
## Anomaly Advisor
37
38
-The [Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md) is the flagship anomaly detection feature within Netdata. In the "Anomalies" tab of Netdata you will see an overall "Anomaly Rate" chart that aggregates node level anomaly rate for all nodes in a space. The aim of this chart is to make it easy to quickly spot periods of time where the overall "[node anomaly rate](https://github.com/netdata/netdata/blob/master/src/ml/README.md#node-anomaly-rate)" is elevated in some unusual way and for what node or nodes this relates to.
38
+The [Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/dashboard/anomaly-advisor-tab.md) is the flagship anomaly detection feature within Netdata. In the "Anomalies" tab of Netdata you will see an overall "Anomaly Rate" chart that aggregates node level anomaly rate for all nodes in a space. The aim of this chart is to make it easy to quickly spot periods of time where the overall "[node anomaly rate](https://github.com/netdata/netdata/blob/master/src/ml/README.md#node-anomaly-rate)" is elevated in some unusual way and for what node or nodes this relates to.
39
40
41
@@ -67,7 +67,7 @@ You can see some example ML based alert configurations below:
67
Check out the resources below to learn more about how Netdata is approaching ML:
68
69
- [Agent ML documentation](https://github.com/netdata/netdata/blob/master/src/ml/README.md).
70
-- [Anomaly Advisor documentation](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md).
70
+- [Anomaly Advisor documentation](https://github.com/netdata/netdata/blob/master/docs/dashboard/anomaly-advisor-tab.md).
71
- [Metric Correlations documentation](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/metric-correlations.md).
72
- Anomaly Advisor [launch blog post](https://www.netdata.cloud/blog/introducing-anomaly-advisor-unsupervised-anomaly-detection-in-netdata/).
73
- Netdata Approach to ML [blog post](https://www.netdata.cloud/blog/our-approach-to-machine-learning/).
docs/quickstart/infrastructure.md
+3
-3
@@ -81,7 +81,7 @@ These tabs can be separated into "static", meaning they are by default presented
81
82
- The [Alerts tab](https://github.com/netdata/netdata/blob/master/docs/monitor/view-active-alerts.md) provides you with an overview for all the active alerts you receive for the nodes in this War Room, you can also see all the alerts that are configured to be triggered in any given moment.
83
84
-- The [Anomalies tab](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md) is dedicated to the Anomaly Advisor tool.
84
+- The [Anomalies tab](https://github.com/netdata/netdata/blob/master/docs/dashboard/anomaly-advisor-tab.md) is dedicated to the Anomaly Advisor tool.
85
86
- The [Functions tab](https://github.com/netdata/netdata/blob/master/docs/cloud/netdata-functions.md) gives you the ability to visualize functions that the Netdata Agent collectors are able to expose.
87
@@ -187,7 +187,7 @@ collect from across your infrastructure with Netdata.
187
<Box
188
title="Troubleshooting with Netdata Cloud">
189
<BoxListItemRegexLink to="[](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/metric-correlations.md)" title="Metric Correlations" />
190
- <BoxListItemRegexLink to="[](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md)" title="Anomaly Advisor" />
190
+ <BoxListItemRegexLink to="[](https://github.com/netdata/netdata/blob/master/docs/dashboard/anomaly-advisor-tab.md)" title="Anomaly Advisor" />
191
<BoxListItemRegexLink to="[](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/events-feed.md)" title="Events Feed" />
192
</Box>
193
<Box
@@ -215,7 +215,7 @@ collect from across your infrastructure with Netdata.
215
- [Alert notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.md)
216
- Troubleshooting with Netdata Cloud
217
- [Metric Correlations](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/metric-correlations.md)
218
- - [Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md)
218
+ - [Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/dashboard/anomaly-advisor-tab.md)
219
- [Events Feed](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/events-feed.md)
220
- Management and settings
221
- [Sign in with email, Google, or GitHub](https://github.com/netdata/netdata/blob/master/docs/cloud/manage/sign-in.md)
src/collectors/python.d.plugin/anomalies/README.md
+1
-1
@@ -10,7 +10,7 @@ learn_rel_path: "Integrations/Monitor/Anything"
10
11
# Anomaly detection with Netdata
12
13
-**Note**: Check out the [Netdata Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/cloud/insights/anomaly-advisor.md) for a more native anomaly detection experience within Netdata.
13
+**Note**: Check out the [Netdata Anomaly Advisor](https://github.com/netdata/netdata/blob/master/docs/dashboard/anomaly-advisor-tab.md) for a more native anomaly detection experience within Netdata.
14
15
This collector uses the Python [PyOD](https://pyod.readthedocs.io/en/latest/index.html) library to perform unsupervised [anomaly detection](https://en.wikipedia.org/wiki/Anomaly_detection) on your Netdata charts and/or dimensions.
16