extend ml default training from ~24 to ~48 hours (#15971)
double `number of models per dimension`
Andrew Maguire committed
Sep 18, 2023 at 13:43 UTC
717ba3e9b201e43e0b4064bae5bdc762b31ebf93
2 files changed
+4
-4
ml/Config.cc
+1
-1
@@ -29,7 +29,7 @@ void ml_config_load(ml_config_t *cfg) {
29
unsigned min_train_samples = config_get_number(config_section_ml, "minimum num samples to train", 1 * 900);
30
unsigned train_every = config_get_number(config_section_ml, "train every", 3 * 3600);
31
32
- unsigned num_models_to_use = config_get_number(config_section_ml, "number of models per dimension", 9);
32
+ unsigned num_models_to_use = config_get_number(config_section_ml, "number of models per dimension", 18);
33
unsigned delete_models_older_than = config_get_number(config_section_ml, "delete models older than", 60 * 60 * 24 * 7);
34
35
unsigned diff_n = config_get_number(config_section_ml, "num samples to diff", 1);
ml/README.md
+3
-3
@@ -130,7 +130,7 @@ Below is a list of all the available configuration params and their default valu
130
# maximum num samples to train = 21600
131
# minimum num samples to train = 900
132
# train every = 10800
133
- # number of models per dimension = 9
133
+ # number of models per dimension = 18
134
# dbengine anomaly rate every = 30
135
# num samples to diff = 1
136
# num samples to smooth = 3
@@ -190,7 +190,7 @@ This example assumes 3 child nodes [streaming](https://github.com/netdata/netdat
190
- `maximum num samples to train`: (`3600`/`86400`) This is the maximum amount of time you would like to train each model on. For example, the default of `21600` trains on the preceding 6 hours of data, assuming an `update every` of 1 second.
191
- `minimum num samples to train`: (`900`/`21600`) This is the minimum amount of data required to be able to train a model. For example, the default of `900` implies that once at least 15 minutes of data is available for training, a model is trained, otherwise it is skipped and checked again at the next training run.
192
- `train every`: (`1800`/`21600`) This is how often each model will be retrained. For example, the default of `10800` means that each model is retrained every 3 hours. Note: The training of all models is spread out across the `train every` period for efficiency, so in reality, it means that each model will be trained in a staggered manner within each `train every` period.
193
-- `number of models per dimension`: (`1`/`168`) This is the number of trained models that will be used for scoring. For example the default `number of models per dimension = 9` means that just the most recently trained 9 models for the dimension will be used to determine the corresponding anomaly bit. This means that under default settings of `maximum num samples to train = 21600`, `train every = 10800` and `number of models per dimension = 9`, netdata will store and use the last 9 trained models for each dimension when determining the anomaly bit. This means that for the latest feature vector in this configuration to be considered anomalous it would need to look anomalous across _all_ the models trained for that dimension in the last 9*(10800/3600) ~= 27 hours. As such, increasing `number of models per dimension` may reduce some false positives since it will result in more models (covering a wider time frame of training) being used during scoring.
193
+- `number of models per dimension`: (`1`/`168`) This is the number of trained models that will be used for scoring. For example the default `number of models per dimension = 18` means that the most recently trained 18 models for the dimension will be used to determine the corresponding anomaly bit. This means that under default settings of `maximum num samples to train = 21600`, `train every = 10800` and `number of models per dimension = 18`, netdata will store and use the last 18 trained models for each dimension when determining the anomaly bit. This means that for the latest feature vector in this configuration to be considered anomalous it would need to look anomalous across _all_ the models trained for that dimension in the last 18*(10800/3600) ~= 54 hours. As such, increasing `number of models per dimension` may reduce some false positives since it will result in more models (covering a wider time frame of training) being used during scoring.
194
- `dbengine anomaly rate every`: (`30`/`900`) This is how often netdata will aggregate all the anomaly bits into a single chart (`anomaly_detection.anomaly_rates`). The aggregation into a single chart allows enabling anomaly rate ranking over _all_ metrics with one API call as opposed to a call per chart.
195
- `num samples to diff`: (`0`/`1`) This is a `0` or `1` to determine if you want the model to operate on differences of the raw data or just the raw data. For example, the default of `1` means that we take differences of the raw values. Using differences is more general and works on dimensions that might naturally tend to have some trends or cycles in them that is normal behavior to which we don't want to be too sensitive.
196
- `num samples to smooth`: (`0`/`5`) This is a small integer that controls the amount of smoothing applied as part of the feature processing used by the model. For example, the default of `3` means that the rolling average of the last 3 values is used. Smoothing like this helps the model be a little more robust to spiky types of dimensions that naturally "jump" up or down as part of their normal behavior.
@@ -214,7 +214,7 @@ Once enabled, the "Anomaly Detection" menu and charts will be available on the d
214
In terms of anomaly detection, the most interesting charts would be the `anomaly_detection.dimensions` and `anomaly_detection.anomaly_rate` ones, which hold the `anomalous` and `anomaly_rate` dimensions that show the overall number of dimensions considered anomalous at any time and the corresponding anomaly rate.
215
216
- `anomaly_detection.dimensions`: Total count of dimensions considered anomalous or normal.
217
-- `anomaly_detection.dimensions`: Percentage of anomalous dimensions.
217
+- `anomaly_detection.anomaly_rate`: Percentage of anomalous dimensions.
218
- `anomaly_detection.anomaly_detection`: Flags (0 or 1) to show when an anomaly event has been triggered by the detector.
219
220
Below is an example of how these charts may look in the presence of an anomaly event.