@cryptotaxi247 / netdata
Code Commits Issues Pulls Sessions CI/CD
master
master
md 236 lines 12.5 KB
Rendered Raw
1 # ML Configuration
2
3 You can use Netdata's [Machine Learning](/src/ml/README.md) capabilities to detect anomalies in your infrastructure metrics. This feature is enabled by default if your [Database mode](/src/database/README.md) is set to `db = dbengine`.
4
5 ## Enabling or Disabling Machine Learning
6
7 To enable or disable Machine Learning capabilities on your node:
8
9 1. [Edit `netdata.conf`](/docs/netdata-agent/configuration/README.md#edit-configuration-files).
10 2. In the `[ml]` section:
11 - Set `enabled` to `yes` to enable ML.
12 - Set `enabled` to `no` to disable ML.
13 - Leave it at the default `auto` to enable ML only when [Database mode](/src/database/README.md) is set to `dbengine`.
14 3. [Restart Netdata](/docs/netdata-agent/start-stop-restart.md).
15
16 ## Technical Implementation
17
18 Netdata implements machine learning using the lightweight [dlib](https://github.com/davisking/dlib) C++ library. This implementation choice allows Netdata to:
19
20 ```mermaid
21 flowchart TD
22 Dlib("dlib C++ Library<br/>Implementation")
23
24 Efficiency("Run efficiently on<br/>any system without<br/>heavy dependencies")
25 Resources("Minimize resource usage<br/>while maintaining<br/>high accuracy")
26 Codebase("Operate within the<br/>constraints of agent's<br/>C/C++ codebase")
27
28 Dlib --> Efficiency
29 Dlib --> Resources
30 Dlib --> Codebase
31
32 %% Style definitions
33 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
34 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
35 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
36 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
37
38 %% Apply styles
39 class Dlib alert
40 class Efficiency,Resources,Codebase complete
41 ```
42
43 :::note
44
45 The ML implementation prioritizes minimal storage impact. Model objects are designed to be small, with each trained model typically requiring only a few kilobytes of memory.
46
47 :::
48
49 ## Available Configuration Parameters
50
51 Below is a list of all available configuration parameters and their default values:
52
53 ```bash
54 [ml]
55 # enabled = auto
56 # maximum num samples to train = 21600
57 # minimum num samples to train = 900
58 # train every = 3h
59 # number of models per dimension = 18
60 # dbengine anomaly rate every = 30
61 # num samples to diff = 1
62 # num samples to smooth = 3
63 # num samples to lag = 5
64 # random sampling ratio = 0.2
65 # maximum number of k-means iterations = 1000
66 # dimension anomaly score threshold = 0.99
67 # host anomaly rate threshold = 1.0
68 # anomaly detection grouping method = average
69 # anomaly detection grouping duration = 5m
70 # hosts to skip from training = !*
71 # charts to skip from training = netdata.*
72 # dimension anomaly rate suppression window = 15m
73 # dimension anomaly rate suppression threshold = 450
74 # delete models older than = 7d
75 ```
76
77 ## Multiple Models and False Positive Reduction
78
79 One of the **key strengths of Netdata's ML implementation is its use of multiple models** to virtually eliminate false positives.
80
81 When you configure the `number of models per dimension` parameter (default: 18), you're specifying how many trained models Netdata maintains for each metric.
82
83 :::note
84
85 The default setting of 18 models spanning approximately 54 hours (with models trained every 3 hours) ensures that anomalies are verified across different time frames before being flagged.
86
87 :::
88
89 ### How Multiple Models Work Together
90
91 When your system detects a potential anomaly:
92
93 1. **The new data point is evaluated against all trained models** (up to 18 by default)
94 2. **Each model independently determines if the data point is anomalous** based on its training period
95 3. Only if **ALL** models agree that the data point is anomalous will Netdata set the anomaly bit to 1 (anomalous)
96
97 ```mermaid
98 flowchart TD
99 NewData("New Metric<br/>Data Point")
100
101 Models("18 Models<br/>Evaluate Data")
102
103 AllAgree("All Models<br/>Agree?")
104
105 SetNormal("Set Anomaly Bit = 0<br/>Normal")
106 SetAnomalous("Set Anomaly Bit = 1<br/>Anomalous")
107
108 NewData --> Models
109 Models --> AllAgree
110 AllAgree -->|"Yes"| SetAnomalous
111 AllAgree -->|"No"| SetNormal
112
113 %% Style definitions
114 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
115 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
116 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
117 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
118
119 %% Apply styles
120 class NewData alert
121 class Models,AllAgree neutral
122 class SetNormal complete
123 class SetAnomalous database
124 ```
125
126 :::note
127
128 This consensus approach **reduces false positives by approximately 99%**, ensuring that you only see alerts for genuine anomalies that persist across different time scales.
129
130 :::
131
132 **Models trained on different time frames capture various patterns in your data**, making the system robust against temporary fluctuations.
133
134 ## Configuration Examples
135
136 If you want to **run ML on a parent instead of at the edge**, the examples below illustrate various configurations.
137
138 This example assumes three child nodes [streaming](/docs/observability-centralization-points/metrics-centralization-points/README.md) to one parent node. It shows different ways to configure ML:
139
140 - Running ML on the parent for some or all children.
141 - Running ML on the children themselves.
142 - A mixed approach.
143
144 ```mermaid
145 flowchart BT
146 C1("Netdata Child 0<br/>ML enabled")
147 C2("Netdata Child 1<br/>ML enabled")
148 C3("Netdata Child 2<br/>ML disabled")
149 P1("Netdata Parent<br/>ML enabled for itself<br/>and Child 1 & 2")
150
151 C1 --> P1
152 C2 --> P1
153 C3 --> P1
154
155 %% Style definitions
156 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
157 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
158 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
159 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
160
161 %% Apply styles
162 class C1,C2 complete
163 class C3 neutral
164 class P1 alert
165 ```
166
167 ```text
168 # Parent will run ML for itself and Child 1 & 2, but skip Child 0.
169 # Child 0 and Child 1 will run ML independently.
170 # Child 2 will rely on the parent for ML and will not run it itself.
171
172 # Parent configuration
173 [ml]
174 enabled = yes
175 hosts to skip from training = child-0-ml-enabled
176
177 # Child 0 configuration
178 [ml]
179 enabled = yes
180
181 # Child 1 configuration
182 [ml]
183 enabled = yes
184
185 # Child 2 configuration
186 [ml]
187 enabled = no
188 ```
189
190 ## Resource Considerations
191
192 When configuring machine learning, it's important to understand the resource implications:
193
194 :::note
195
196 Netdata's ML is designed to be lightweight, using approximately 1–2% of a single CPU core under default settings on a typical system.
197
198 :::
199
200 Several configuration options directly impact resource usage:
201
202 - **Increasing `number of models per dimension`** increases memory usage as more models are stored
203 - **Decreasing `train every`** increases CPU usage as models are trained more frequently
204 - **Increasing `maximum num samples to train`** increases memory usage during training but may improve accuracy
205 - **Adjusting `random sampling ratio`** allows you to control how much data is used during training (lower values reduce CPU usage)
206
207 For resource-constrained systems, consider these adjustments:
208
209 - Set `random sampling ratio = 0.1` to reduce training data by half from default
210 - Use `number of models per dimension = 6` to maintain multiple model consensus with reduced memory footprint
211 - Increase `train every = 6h` to reduce training frequency
212
213 ## Parameter Descriptions (Min/Max Values)
214
215 # ML Parameter Settings
216
217 | Category | Parameter | Range | Description |
218 |-----------------------------------|----------------------------------------|------------------|------------------------------------------------------------------------------------------------------------------------------------------|
219 | **General Settings** | `enabled` | `yes/no/auto` | Controls whether ML is enabled. `yes` to enable, `no` to disable, `auto` lets Netdata decide based on database mode. |
220 | | `maximum num samples to train` | `3600` - `86400` | Defines the maximum training period. Default `21600` trains on your last 6 hours of data. |
221 | | `minimum num samples to train` | `900` - `21600` | Minimum data needed to train a model. Training is skipped if less than `900` samples (15 minutes) are available. |
222 | | `train every` | `3h` - `6h` | How often models are retrained. Default `3h` means retraining every three hours. Training is staggered to distribute system load. |
223 | **Model Behavior** | `number of models per dimension` | `1` - `168` | Specifies how many trained models per dimension are used. Default `18` means models trained over the last ~54 hours are considered. |
224 | | `dbengine anomaly rate every` | `30` - `900` | How frequently Netdata aggregates anomaly bits into a single chart. |
225 | **Feature Processing** | `num samples to diff` | `0` - `1` | Determines whether ML operates on raw data (`0`) or differences (`1`). Using differences helps detect anomalies in cyclical patterns. |
226 | | `num samples to smooth` | `0` - `5` | Controls data smoothing. Default `3` averages the last three values to reduce noise. |
227 | | `num samples to lag` | `0` - `5` | How many past values are included in the feature vector. Default `5` helps detect patterns over time. |
228 | **Training Efficiency** | `random sampling ratio` | `0.2` - `1.0` | Fraction of data used for training. Default `0.2` means 20% of available data is used, reducing system load while maintaining accuracy. |
229 | | `maximum number of k-means iterations` | - | Limits iterations during k-means clustering (leave at default in most cases). |
230 | **Anomaly Detection Sensitivity** | `dimension anomaly score threshold` | `0.01` - `5.00` | Threshold for flagging an anomaly. Default `0.99` flags values in the top 1% of anomalies based on training data. |
231 | | `host anomaly rate threshold` | `0.1` - `10.0` | Percentage of dimensions that must be anomalous for host to be considered anomalous. Default `1.0` means more than 1% must be anomalous. |
232 | **Anomaly Detection Grouping** | `anomaly detection grouping method` | - | Method used to calculate node-level anomaly rate. |
233 | | `anomaly detection grouping duration` | `1m` - `15m` | Time window for calculating anomaly rates. Default `5m` calculates over a 5-minute rolling window. |
234 | **Skipping Hosts and Charts** | `hosts to skip from training` | - | Excludes specific child hosts from training. Default `!*` means no hosts are skipped. |
235 | | `charts to skip from training` | - | Excludes charts from anomaly detection. By default, Netdata-related charts are excluded. |
236 | **Model Retention** | `delete models older than` | `1d` - `7d` | How long old models are stored. Default `7d` removes unused models after seven days. |