@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 184 lines 10.3 KB
Rendered Raw
1 # Machine Learning and Anomaly Detection
2
3 ## Overview
4
5 You can leverage machine learning to detect patterns and anomalies in your large datasets, enabling you to identify issues early before they escalate.
6
7 Netdata offers **Anomaly Advisor**, a tool designed to improve your troubleshooting experience, reduce mean time to resolution, and prevent issues from escalating. You can access it through the [Netdata dashboard](/docs/dashboards-and-charts/README.md).
8
9 :::tip
10
11 To configure ML on your nodes, check the [ML configuration documentation](/src/ml/ml-configuration.md).
12
13 :::
14
15 ## Design Principles
16
17 When you use Netdata's machine learning models, you benefit from these key principles:
18
19 | Principle | Description |
20 |----------------------------|----------------------------------------------------------------------------------------------------------------------------|
21 | **Unsupervised Learning** | Models operate independently without requiring your input |
22 | **Real-time Performance** | While ML impacts CPU usage, you won't experience any compromise to Netdata's high-fidelity, real-time monitoring |
23 | **Seamless Integration** | ML-based insights are fully embedded into your existing Netdata infrastructure monitoring and troubleshooting workflow |
24 | **Assistance Over Alerts** | ML helps you investigate potential issues rather than triggering unnecessary alerts - no 3 AM wake-ups for minor anomalies |
25 | **Many Light Models** | Netdata uses many lightweight models instead of a few heavy ones, optimizing for resource usage while maintaining accuracy |
26 | **Scalable Architecture** | The system is designed to handle thousands of metrics simultaneously, scoring each one every second with minimal latency |
27
28 :::note
29
30 Netdata deliberately avoids using deep learning models, as they would introduce heavy dependencies and resource requirements that wouldn't align with Netdata's goal of running efficiently on any Linux system. Instead, the implementation uses the lightweight [dlib](https://github.com/davisking/dlib) library and spreads training costs over a wide window to minimize performance impact.
31
32 :::
33
34 ## Types of Anomalies You Can Detect
35
36 | Anomaly Type | Description | Business Impact |
37 |--------------------------|-------------------------------------------------------------------|------------------------------------------|
38 | **Point Anomalies** | Unusually high or low values compared to historical data | Early warning of service degradation |
39 | **Contextual Anomalies** | Sequences of values that deviate from expected patterns | Identification of unusual usage patterns |
40 | **Collective Anomalies** | Multivariate anomalies where a combination of metrics appears off | Detection of complex system issues |
41 | **Concept Drifts** | Gradual shifts leading to a new baseline | Recognition of evolving system behavior |
42 | **Change Points** | Sudden shifts resulting in a new normal state | Identification of system changes |
43
44 ## How Netdata ML Works
45
46 ```mermaid
47 flowchart TD
48 Raw("**Raw Metrics**<br/><br/>Last 4 Hours")
49 Preprocess("**Preprocess**<br/><br/>Feature Vectors")
50 Train("**Train k-means<br/><br/>k=2**")
51 Model("**Trained Model**")
52
53 M1("Model 1<br/><br/>**Recent Data**")
54 M2("Model 2<br/><br/>**Older Data**")
55 M3("Model 3<br/><br/>**Even Older Data**")
56 MN("Model N<br/><br/>**Up to 2 Days Old**")
57
58 NewData("**New Metrics**")
59 DistCalc("**Calculate**<br/><br/>Euclidean Distance<br/>to Cluster Centers")
60 Threshold("**Distance > 99th**<br/><br/>Percentile?")
61 FlagA("**Flag as Anomalous**<br/><br/>in This Model")
62 FlagN("**Flag as Normal**<br/><br/>in This Model")
63
64 AllResults("**Results from All Models**")
65 AllAgree("**All Models<br/><br/>Agree it's<br/><br/>Anomalous?**")
66 SetBit("**Set Anomaly Bit = 100**<br/><br/>True")
67 ClearBit("**Set Anomaly Bit = 0**<br/><br/>False")
68
69 Raw --> Preprocess
70 Preprocess --> Train
71 Train --> Model
72 Model --> M1
73 Model --> M2
74 Model --> M3
75 Model --> MN
76
77 M1 --> NewData
78 M2 --> NewData
79 M3 --> NewData
80 MN --> NewData
81
82 NewData --> DistCalc
83 DistCalc --> Threshold
84 Threshold -->|Yes| FlagA
85 Threshold -->|No| FlagN
86
87 FlagA --> AllResults
88 FlagN --> AllResults
89 AllResults --> AllAgree
90 AllAgree -->|Yes| SetBit
91 AllAgree -->|No| ClearBit
92
93 %% Style definitions
94 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
95 classDef process fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
96 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
97 classDef anomaly fill:#f44336,stroke:#000000,stroke-width:3px,color:#000000,font-size:16px
98
99 %% Apply styles
100 class Raw,Preprocess,NewData,AllResults neutral
101 class Train,Model,M1,M2,M3,MN,DistCalc,Threshold,AllAgree process
102 class FlagN,ClearBit complete
103 class FlagA,SetBit anomaly
104 ```
105
106 ### Training & Detection
107
108 When you enable ML, Netdata trains an unsupervised model for each of your metrics. By default, this model is a [k-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) algorithm (with k=2) trained on the last 4 hours of your data. Instead of just analyzing raw values, the model works with preprocessed feature vectors to improve your detection accuracy.
109
110 :::important
111
112 To reduce false positives in your environment, Netdata trains multiple models per time-series, covering over two days of data. **An anomaly is flagged only if all models agree on it, eliminating 99% of false positives**. This approach of requiring consensus across models trained on different time scales makes the system highly resistant to spurious anomalies while still being sensitive to real issues.
113
114 :::
115
116 The anomaly detection algorithm uses the [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) between recent metric patterns and the learned cluster centers. If this distance exceeds a threshold based on the 99th percentile of training data, that model considers the metric anomalous.
117
118 ### Anomaly Bit
119
120 Each trained model assigns an **anomaly score** at every time step based on how far your data deviates from learned clusters. If the score exceeds the 99th percentile of training data, the **anomaly bit** is set to `true` (100); otherwise, it remains `false` (0).
121
122 **Key benefits you'll experience:**
123
124 - No additional storage overhead since the anomaly bit is embedded in Netdata's floating point number format
125 - The query engine automatically computes anomaly rates without requiring extra queries
126
127 :::note
128
129 The anomaly bit is quite literally a bit in Netdata's [internal storage representation](https://github.com/netdata/netdata/blob/89f22f056ca2aae5d143da9a4e94fcab1f7ee1b8/libnetdata/storage_number/storage_number.c#L83). This ingenious design means that for every metric collected, Netdata can also track whether it's anomalous without increasing storage requirements.
130
131 :::
132
133 You can access the anomaly bits through Netdata's API by adding the `options=anomaly-bit` parameter to your query. For example:
134
135 ```
136 https://your-node/api/v1/data?chart=system.cpu&dimensions=user&after=-10&options=anomaly-bit
137 ```
138
139 This would return anomaly bits for the last 10 seconds of CPU user data, with values of either 0 (normal) or 100 (anomalous).
140
141 ### Anomaly Rate
142
143 You can see **Node Anomaly Rate (NAR)** and **Dimension Anomaly Rate (DAR)** calculated based on anomaly bits. Here's an example matrix:
144
145 | Time | d1 | d2 | d3 | d4 | d5 | **NAR** |
146 |---------|---------|---------|---------|---------|---------|-----------------------|
147 | t1 | 0 | 0 | 0 | 0 | 0 | **0%** |
148 | t2 | 0 | 0 | 0 | 0 | 100 | **20%** |
149 | t3 | 0 | 0 | 0 | 0 | 0 | **0%** |
150 | t4 | 0 | 100 | 0 | 0 | 0 | **20%** |
151 | t5 | 100 | 0 | 0 | 0 | 0 | **20%** |
152 | t6 | 0 | 100 | 100 | 0 | 100 | **60%** |
153 | t7 | 0 | 100 | 0 | 100 | 0 | **40%** |
154 | t8 | 0 | 0 | 0 | 0 | 100 | **20%** |
155 | t9 | 0 | 0 | 100 | 100 | 0 | **40%** |
156 | t10 | 0 | 0 | 0 | 0 | 0 | **0%** |
157 | **DAR** | **10%** | **30%** | **20%** | **20%** | **30%** | **_NAR_t1-10 = 22%_** |
158
159 - **DAR (Dimension Anomaly Rate):** Average anomalies for a specific metric over time
160 - **NAR (Node Anomaly Rate):** Average anomalies across all metrics at a given time
161 - **Overall anomaly rate:** Computed across your entire dataset for deeper insights
162
163 ### Node-Level Anomaly Detection
164
165 Netdata tracks the percentage of anomaly bits over time for you. When the **Node Anomaly Rate (NAR)** exceeds a set threshold and remains high for a period, a **node anomaly event** is triggered. These events are recorded in the `new_anomaly_event` dimension on the `anomaly_detection.anomaly_detection` chart.
166
167 ## Viewing Anomaly Data in Your Netdata Dashboard
168
169 Once you enable ML, you'll have access to an **Anomaly Detection** menu with key charts:
170
171 - **`anomaly_detection.dimensions`**: Number of dimensions flagged as anomalous
172 - **`anomaly_detection.anomaly_rate`**: Percentage of anomalous dimensions
173 - **`anomaly_detection.anomaly_detection`**: Flags (0 or 1) indicating when an anomaly event occurs
174
175 These insights help you quickly assess potential issues and take action before they escalate.
176
177 ## Summary
178
179 With Netdata ML, you get reliable, real-time anomaly detection with minimal false positives. By incorporating ML within your existing observability workflows, you can enhance troubleshooting and ensure proactive monitoring without unnecessary alerts.
180
181 For more information:
182
183 - [Anomaly Advisor](/docs/dashboards-and-charts/anomaly-advisor-tab.md)
184 - [ML Configuration Guide](/src/ml/ml-configuration.md)