@cryptotaxi247 / netdata
Code Commits Issues Pulls Sessions CI/CD
master
master
md 315 lines 19.1 KB
Rendered Raw
1 # Machine Learning Anomaly Detection
2
3 Netdata uses k-means clustering to detect anomalies for each collected metric automatically.
4
5 The system maintains 18 models per metric, each trained on 6-hour windows at 3-hour intervals, providing approximately 54 hours of rolling behavioral patterns. Anomaly detection occurs in real-time during data collection - a data point is flagged as anomalous only when all 18 models reach consensus, effectively eliminating noise while maintaining sensitivity to genuine issues.
6
7 Anomaly bits are stored alongside metric data in the time-series database, with the same retention period. The query engine calculates anomaly rates dynamically during data aggregation, exposing anomaly information on every chart without additional overhead.
8
9 A dedicated process correlates anomalies across all metrics within each node, generating real-time node-level anomaly charts. This correlation data feeds into Netdata's scoring engine - a specialized query system that can evaluate thousands of metrics simultaneously and return an ordered list ranked by anomaly severity, powering the Anomaly Advisor for rapid root cause analysis.
10
11 ## System Characteristics
12
13 | Aspect | Implementation | Benefit |
14 |-------------------------|---------------------------------------------------------------------------------------------|------------------------------------------------------|
15 | **Algorithm** | Unsupervised k-means clustering (k=2) via [dlib](https://github.com/davisking/dlib) | No manual training or labeled data required |
16 | **Model Architecture** | Rolling 18 models per metric, 3-hour staggered training | Eliminates 99% of false positives through consensus |
17 | **Processing Location** | Edge computation on each Netdata Agent | No cloud dependency, no data egress |
18 | **Resource Usage** | ~18KB RAM per metric, 2-4% of a single CPU for 10k metrics | Predictable linear scaling |
19 | **Configuration** | Zero-configuration with automatic adaptation | Works instantly on any metric type |
20 | **Detection Latency** | Real-time during data collection | Anomalies flagged within 1 second |
21 | **Historical Storage** | Anomaly bit embedded in metric storage | No additional storage overhead |
22 | **Query Performance** | On-the-fly anomaly rate calculation | No pre-aggregation needed |
23 | **Time-series Integrity** | Immutable anomaly history | No hindsight bias — shows what was detectable then |
24 | **Coverage** | Every metric, every dimension | No sampling, no blind spots |
25 | **Correlation Engine** | Real-time anomaly correlation across metrics | Powers Anomaly Advisor for root cause analysis |
26 | **Alert Philosophy** | Primarily an investigation aid; anomaly bits and anomaly rate can also drive health alerts | Reduces alert fatigue while enabling anomaly-based alerting |
27
28 :::note
29 Netdata avoids deep learning models to maintain lightweight operation on any Linux system. The entire ML system is designed to run efficiently without specialized hardware or dependencies.
30 :::
31
32 ## Types of Anomalies Detected
33
34 | Anomaly Type | Description | Business Impact |
35 |--------------------------|------------------------------------------------------------------|-------------------------------------------|
36 | **Point Anomalies** | Unusually high or low values compared to historical data | Early warning of service degradation |
37 | **Contextual Anomalies** | Sequences of values that deviate from expected patterns | Identification of unusual usage patterns |
38 | **Collective Anomalies** | Multivariate anomalies where a combination of metrics appears off | Detection of complex system issues |
39 | **Concept Drifts** | Gradual shifts leading to a new baseline | Recognition of evolving system behavior |
40 | **Change Points** | Sudden shifts resulting in a new normal state | Identification of system changes |
41
42 ## Technical Deep Dive: How Netdata ML Works
43
44 ```mermaid
45 flowchart TD
46 Raw("Raw Metrics<br/>Last 6 Hours")
47 Preprocess("Preprocess<br/>Feature Vectors")
48 Train("Train k-means<br/>k=2")
49 Model("Trained Model")
50
51 M1("Model 1<br/>Recent Data")
52 M2("Model 2<br/>Older Data")
53 M3("Model 3<br/>Even Older Data")
54 MN("Model N<br/>Up to 54 Hours Old")
55
56 NewData("New Metrics")
57 DistCalc("Calculate<br/>Euclidean Distance<br/>to Cluster Centers")
58 Threshold("Distance > 99th<br/>Percentile?")
59 FlagA("Flag as Anomalous<br/>in This Model")
60 FlagN("Flag as Normal<br/>in This Model")
61
62 AllResults("Results from All Models")
63 AllAgree("All Models<br/>Agree it's<br/>Anomalous?")
64 SetBit("Set Anomaly Bit = 1<br/>True")
65 ClearBit("Set Anomaly Bit = 0<br/>False")
66
67 Raw --> Preprocess
68 Preprocess --> Train
69 Train --> Model
70 Model --> M1
71 Model --> M2
72 Model --> M3
73 Model --> MN
74
75 M1 --> NewData
76 M2 --> NewData
77 M3 --> NewData
78 MN --> NewData
79
80 NewData --> DistCalc
81 DistCalc --> Threshold
82 Threshold -->|"Yes"| FlagA
83 Threshold -->|"No"| FlagN
84
85 FlagA --> AllResults
86 FlagN --> AllResults
87 AllResults --> AllAgree
88 AllAgree -->|"Yes"| SetBit
89 AllAgree -->|"No"| ClearBit
90
91 %% Style definitions
92 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
93 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
94 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
95 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
96
97 %% Apply styles
98 class Raw,NewData,AllResults alert
99 class Train,Model,M1,M2,M3,MN,DistCalc,Threshold,AllAgree neutral
100 class Preprocess,FlagN,ClearBit complete
101 class FlagA,SetBit database
102 ```
103
104 ### Training & Detection Process
105
106 When you enable ML, Netdata trains an unsupervised model for each of your metrics. By default, this model is a [k-means clustering](https://en.wikipedia.org/wiki/K-means_clustering) algorithm (with k=2) trained on the last 6 hours of your data. Instead of just analyzing raw values, the model works with preprocessed feature vectors to improve your detection accuracy.
107
108 :::important
109 To reduce false positives in your environment, Netdata trains multiple models per time-series, covering over two days of data. **An anomaly is flagged only if all models agree on it, eliminating 99% of false positives**. This approach of requiring consensus across models trained on different time scales makes the system highly resistant to spurious anomalies while still being sensitive to real issues.
110 :::
111
112 The anomaly detection algorithm uses the [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) between recent metric patterns and the learned cluster centers. If this distance exceeds a threshold based on the 99th percentile of training data, that model considers the metric anomalous.
113
114 ### The Anomaly Bit
115
116 Each trained model assigns an **anomaly score** at every time step based on how far your data deviates from learned clusters. If the score exceeds the 99th percentile of training data, the **anomaly bit** is set to `true` (100); otherwise, it remains `false` (0).
117
118 **Key benefits you'll experience:**
119
120 - No additional storage overhead since the anomaly bit is embedded in Netdata's floating point number format
121 - The query engine automatically computes anomaly rates without requiring extra queries
122
123 :::note
124 The anomaly bit is quite literally a bit in Netdata's [internal storage representation](https://github.com/netdata/netdata/blob/89f22f056ca2aae5d143da9a4e94fcab1f7ee1b8/libnetdata/storage_number/storage_number.c#L83). This ingenious design means that for every metric collected, Netdata can also track whether it's anomalous without increasing storage requirements.
125 :::
126
127 You can access the anomaly bits through Netdata's API by adding the `options=anomaly-bit` parameter to your query. For example:
128
129 ```
130 https://your-node/api/v3/data?chart=system.cpu&dimensions=user&after=-10&options=anomaly-bit
131 ```
132
133 This would return anomaly bits for the last 10 seconds of CPU user data, with values of either 0 (normal) or 100 (anomalous).
134
135 ### Anomaly Rate Calculations
136
137 You can see **Node Anomaly Rate (NAR)** and **Dimension Anomaly Rate (DAR)** calculated based on anomaly bits. Here's an example matrix:
138
139 | Time | d1 | d2 | d3 | d4 | d5 | **NAR** |
140 |---------|---------|---------|---------|---------|---------|-----------------------|
141 | t1 | 0 | 0 | 0 | 0 | 0 | **0%** |
142 | t2 | 0 | 0 | 0 | 0 | 100 | **20%** |
143 | t3 | 0 | 0 | 0 | 0 | 0 | **0%** |
144 | t4 | 0 | 100 | 0 | 0 | 0 | **20%** |
145 | t5 | 100 | 0 | 0 | 0 | 0 | **20%** |
146 | t6 | 0 | 100 | 100 | 0 | 100 | **60%** |
147 | t7 | 0 | 100 | 0 | 100 | 0 | **40%** |
148 | t8 | 0 | 0 | 0 | 0 | 100 | **20%** |
149 | t9 | 0 | 0 | 100 | 100 | 0 | **40%** |
150 | t10 | 0 | 0 | 0 | 0 | 0 | **0%** |
151 | **DAR** | **10%** | **30%** | **20%** | **20%** | **30%** | **_NAR_t1-10 = 22%_** |
152
153 - **DAR (Dimension Anomaly Rate):** Average anomalies for a specific metric over time
154 - **NAR (Node Anomaly Rate):** Average anomalies across all metrics at a given time
155 - **Overall anomaly rate:** Computed across your entire dataset for deeper insights
156
157 ### Node-Level Anomaly Detection
158
159 Netdata tracks the percentage of anomaly bits over time for you. When the **Node Anomaly Rate (NAR)** exceeds a set threshold and remains high for a period, a **node anomaly event** is triggered. These events are recorded in the `new_anomaly_event` dimension on the `anomaly_detection.anomaly_detection` chart.
160
161 ## Available Documentation
162
163 - **[ML Configuration](/src/ml/ml-configuration.md)** - Configuration and tuning guide
164 - **[Metric Correlations](/docs/metric-correlations.md)** - Finding related metrics during incidents
165
166 ## Viewing Anomaly Data in Your Netdata Dashboard
167
168 Once you enable ML, you'll have access to an **Anomaly Detection** menu with key charts:
169
170 - **`anomaly_detection.dimensions`**: Number of dimensions flagged as anomalous
171 - **`anomaly_detection.anomaly_rate`**: Percentage of anomalous dimensions
172 - **`anomaly_detection.anomaly_detection`**: Flags (0 or 1) indicating when an anomaly event occurs
173
174 These insights help you quickly assess potential issues and take action before they escalate.
175
176 ## Operational Details
177
178 ### Why 18 Models?
179
180 The number 18 balances three competing requirements:
181
182 1. **Incremental learning efficiency** - Training 48 hours of data every 3 hours would waste computational resources. Instead, each model trains on just 6 hours of data, with only one new model created every 3 hours.
183
184 2. **Adaptive memory duration** - When an anomaly occurs, the newest model will learn it as "normal" within 3 hours. The system gradually "forgets" this pattern as older models are replaced. With 18 models at 3-hour intervals, complete forgetting takes 54 hours (2.25 days).
185
186 3. **Consensus noise reduction** - Multiple models voting together eliminate random fluctuations. 18 models provide strong consensus without excessive memory use.
187
188 This creates a sliding window memory: recent anomalies become "normal" quickly (within 3 hours for the newest model), while the full consensus takes 54 hours to completely forget an anomalous pattern. This balance prevents both alert fatigue from repeated anomalies and blindness to recurring issues.
189
190 ### How Netdata Minimizes Training CPU Impact
191
192 ML typically doubles the agent's CPU usage - from ~2% to ~4% of a single core. This efficiency comes from several optimizations:
193
194 1. **Smart metric filtering** - Metrics with constant or fixed values are automatically excluded from training, eliminating wasted computation on unchanging data.
195
196 2. **Incremental training windows** - Each model trains on only 6 hours of data instead of the full 54-hour history, reducing computational requirements by ~90%.
197
198 3. **Even training distribution** - The agent dynamically throttles model training to spread the work evenly across each 3-hour window, preventing CPU spikes. With 10,000 metrics, this means training ~1 model per second instead of training 10,000 models in a burst.
199
200 4. **Distributed intelligence** - Child agents stream both trained models and anomaly bits to parent agents along with metric data. Parents receive pre-computed ML results, requiring zero additional ML computation for aggregated views.
201
202 This design ensures ML remains lightweight enough to run on production systems without impacting primary workloads.
203
204 **Dynamic prioritization**: ML automatically throttles or even pauses training during:
205
206 - Heavy query load - ensuring dashboards remain responsive
207 - Parent-child reconnections - prioritizing metric replication
208 - Any resource contention - backing off to protect core monitoring
209
210 Under these conditions, ML will completely stop training new models to ensure:
211
212 - User queries remain fast and responsive
213 - Metric streaming completes quickly after network interruptions
214 - Overall CPU and I/O consumption stays within bounds
215
216 This means ML is truly a background process - it uses spare cycles but immediately yields resources when needed for operational tasks.
217
218 ### Storage Impact
219
220 ML has **zero storage overhead** in the time-series database. The anomaly bit uses a previously unused bit in the existing sample storage format - no schema changes or storage expansion required.
221
222 The only storage impact comes from persisting trained models to disk for survival across restarts:
223
224 - Model files are small compared to the time-series data
225 - Negligible impact on overall storage requirements
226 - Models are retained only for active metrics
227
228 This means you can enable ML without provisioning additional storage capacity. Anomaly history is retained for the same period as your metrics, with no extra space required.
229
230 **Query performance impact: None**. The anomaly bit is loaded together with metric data in a single disk read - no additional I/O operations required. Querying metrics with anomaly data has the same disk I/O pattern as querying metrics without ML.
231
232 ### Cold Start Behavior
233
234 On a freshly installed agent, ML begins detecting anomalies within 10 minutes. However, early detection has important characteristics:
235
236 **Timeline:**
237
238 - **0-10 minutes**: Collecting initial data, no anomaly detection
239 - **10+ minutes**: First models trained, anomaly detection begins with high sensitivity
240 - **3 hours**: First model rotation, improved accuracy
241 - **54 hours**: Full model set established, optimal detection accuracy
242
243 **What to expect:**
244
245 - Initial hours show more anomalies due to limited training data
246 - False positive rate decreases as models accumulate more behavioral patterns
247 - Each 3-hour cycle improves detection quality
248 - After 2-3 days, the system reaches steady-state accuracy
249
250 **Operational tip**: During the first 48 hours after deployment, expect elevated anomaly rates. This is normal as the system learns your infrastructure's patterns. Use this period to observe ML behavior but avoid making critical decisions based solely on early anomaly detection.
251
252 ## Creating Anomaly-Based Health Alerts
253
254 You can create health alerts that trigger based on anomaly rates instead of raw metric values by using the `anomaly-bit` option in your alert's `lookup` line. Internally, anomaly bits mark samples as anomalous or normal, and the query/health pipeline exposes this as an anomaly-rate percentage in the 0-100 range. For aggregated or tiered data, returned values can be intermediate percentages rather than only 0 or 100, so averaging over a time window gives you the anomaly rate as a percentage.
255
256 ### Anomaly-rate alert
257
258 The following template triggers when the anomaly rate on `system.cpu` exceeds the defined thresholds:
259
260 ```text
261 template: ml_5min_cpu_chart
262 on: system.cpu
263 lookup: average -5m anomaly-bit of *
264 calc: $this
265 units: %
266 every: 30s
267 warn: $this > (($status >= $WARNING) ? (5) : (20))
268 crit: $this >= (($status == $CRITICAL) ? (20) : (100))
269 info: rolling 5min anomaly rate for system.cpu chart
270 ```
271
272 ### Pairing with actual values
273
274 An anomaly-rate alert tells you *something is unusual*, but not *what the actual numbers are*. To get the real values alongside the anomaly alert, create a companion alert on the raw metric:
275
276 ```text
277 alarm: cpu_usage_5min
278 on: system.cpu
279 lookup: average -5m of user,system
280 units: %
281 every: 30s
282 warn: $this > 80
283 crit: $this > 95
284 info: average user+system CPU utilization over the last 5 minutes
285 ```
286
287 When the anomaly alert fires, the companion alert can provide the concrete values in a separate alert — for example, "CPU anomaly rate 35%" alongside "CPU utilization 92%".
288
289 :::tip
290
291 Use `foreach` in a template to generate one alert instance per dimension (e.g., one per CPU state). Note that `foreach` and `of` serve different purposes: `of` selects which dimensions the `lookup` aggregates, while `foreach` creates separate alert instances for each matching dimension in a template. For the full alert syntax, see the [health configuration reference](/src/health/REFERENCE.md).
292
293 :::
294
295 ### Adding context to alert notifications
296
297 The `info` and `summary` fields support template variables that add contextual detail to notifications:
298
299 | Variable | Replaced With |
300 |----------------------|-----------------------------------|
301 | `${family}` | Family instance (for example, `eth0`) |
302 | `${label:LABEL_NAME}` | Chart label value |
303
304 :::important
305
306 `$this` is available only in `calc`, `warn`, and `crit` expressions — not in the `info` or `summary` fields.
307
308 :::
309
310 ### Investigating anomaly alerts
311
312 When an anomaly alert fires, use Netdata's built-in tools to investigate the root cause:
313
314 - **[Alert Troubleshooting](/docs/netdata-ai/troubleshooting/index.md)** — generate a one-click report from any fired alert, assessing its validity, uncovering correlated signals, and proposing a root-cause hypothesis
315 - **[Investigations](/docs/netdata-ai/investigations/index.md)** — ask open-ended questions about your infrastructure for deeper analysis beyond a single alert