docs: add anomaly rate alerting guidance and cross-references (#22283)
* docs: add anomaly rate alerting guidance and cross-references * docs: Improved the Anomaly Rate detection type description in the Alerts Configuration Manager doc: simplified the table entry, added an explanatory paragraph with percentage threshold details (0–100%), and converted the configuration guidance into a :::tip admonition with links to ML Anomaly Detection and Alert Configuration Reference (Examples 6–7), matching the plan guidance that was not fully implemented in the initial commit. * docs: Prettified all markdown tables in both PR-edited files. In the Alerts Configuration Manager doc, all 4 tables (Alert Detection Types, Metrics Lookup, Alert Conditions, Naming and Documentation) now have consistently aligned columns with separator dashes matching the widest cell width per column. In the ML Anomaly Detection doc, the System Characteristics table (3-column, 12-row) was also prettified — it was misaligned because the Alert Philosophy row had been lengthened in a prior commit without adjusting column widths. No table content was altered; only whitespace alignment changed. --------- Co-authored-by: nedi-app[bot] <nedi-app[bot]@users.noreply.github.com>
nedi-app[bot] committed
May 5, 2026 at 14:31 UTC
2d697d0db134bcbccd13ecb1d312137cbd489be9
2 files changed
+57
-39
docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md
+33
-25
@@ -39,14 +39,22 @@ flowchart LR
39
40
You can choose from different ways to trigger alerts based on your monitoring needs:
41
42
-| Type | Description |
43
-|---------------------|---------------------------------------------------------|
44
-| **Standard** | Fires when a metric crosses a set value |
45
-| **Metric Variance** | Fires based on variation in values over time |
46
-| **Anomaly Rate** | Fires when the anomaly rate exceeds a certain threshold |
42
+| Type | Description |
43
+|-----------------------|---------------------------------------------------------------|
44
+| **Standard** | Fires when a metric crosses a set value |
45
+| **Metric Variance** | Fires based on variation in values over time |
46
+| **Anomaly Rate** | Fires when the ML-calculated anomaly rate exceeds a threshold |
47
48
Choose the type that best suits the behavior you want to monitor.
49
50
+The **Anomaly Rate** detection type uses ML-generated anomaly bits to calculate what percentage of data points in a time window were flagged as anomalous. Thresholds are expressed as percentages (0–100% anomaly rate). You can configure it from any chart's alert bell icon by selecting Anomaly Rate as the detection type.
51
+
52
+:::tip
53
+
54
+For details on how anomalies are detected, see [ML Anomaly Detection](/docs/ml-ai/ml-anomaly-detection/ml-anomaly-detection.md). For the underlying alert configuration syntax with `anomaly-bit` lookups, see the [Alert Configuration Reference](/src/health/REFERENCE.md) (Examples 6 and 7).
55
+
56
+:::
57
+
58
## Metrics Lookup & Filters
59
60
Click **Show advanced** in the Alert Configuration Manager to access these options.
@@ -55,12 +63,12 @@ Click **Show advanced** in the Alert Configuration Manager to access these optio
63
64
You can define how Netdata should query the data before triggering an alert:
65
58
-| Parameter | Description |
59
-|--------------|--------------------------------------------------------------------|
60
-| `method` | How values are aggregated (`avg`, `min`, `max`, etc.) |
61
-| `duration` | Time window used for the check |
62
-| `dimensions` | Which metric dimensions to include |
63
-| `options` | Modify how values are interpreted (e.g., `percentage`, `absolute`) |
66
+| Parameter | Description |
67
+|----------------|----------------------------------------------------------------------|
68
+| `method` | How values are aggregated (`avg`, `min`, `max`, etc.) |
69
+| `duration` | Time window used for the check |
70
+| `dimensions` | Which metric dimensions to include |
71
+| `options` | Modify how values are interpreted (e.g., `percentage`, `absolute`) |
72
73
### Filtering Targets
74
@@ -86,23 +94,23 @@ Example:
94
95
You control how and when alerts are triggered, escalated, or resolved:
96
89
-| Setting | Purpose |
90
-|-----------------------------|-----------------------------------------------------------------|
91
-| **Thresholds** | Define values for `warning` and `critical` states |
92
-| **Recovery thresholds** | Set when the alert should downgrade or clear |
93
-| **Check interval** | How often the alert check runs (e.g., every 10 seconds) |
94
-| **Notification delay** | Delay before sending notifications for state changes |
95
-| **Repeat notifications** | How often to resend an alert if the issue persists (Agent only) |
96
-| **Notification recipients** | Define who gets alerted (Agent only) |
97
-| **Custom exec script** | Run a custom shell script when an alert triggers |
97
+| Setting | Purpose |
98
+|-------------------------------|-------------------------------------------------------------------|
99
+| **Thresholds** | Define values for `warning` and `critical` states |
100
+| **Recovery thresholds** | Set when the alert should downgrade or clear |
101
+| **Check interval** | How often the alert check runs (e.g., every 10 seconds) |
102
+| **Notification delay** | Delay before sending notifications for state changes |
103
+| **Repeat notifications** | How often to resend an alert if the issue persists (Agent only) |
104
+| **Notification recipients** | Define who gets alerted (Agent only) |
105
+| **Custom exec script** | Run a custom shell script when an alert triggers |
106
107
## Naming and Documentation
108
101
-| Field | Description |
102
-|-----------------|-----------------------------------------------------|
103
-| **Alert Name** | A unique name for the alert |
104
-| **Description** | What the alert does, in one or two sentences |
105
-| **Summary** | Optional: a short summary for display in dashboards |
109
+| Field | Description |
110
+|-------------------|-------------------------------------------------------|
111
+| **Alert Name** | A unique name for the alert |
112
+| **Description** | What the alert does, in one or two sentences |
113
+| **Summary** | Optional: a short summary for display in dashboards |
114
115
## Final Notes
116
docs/ml-ai/ml-anomaly-detection/ml-anomaly-detection.md
+24
-14
@@ -10,20 +10,20 @@ A dedicated process correlates anomalies across all metrics within each node, ge
10
11
## System Characteristics
12
13
-| Aspect | Implementation | Benefit |
14
-|--------|----------------|---------|
15
-| **Algorithm** | Unsupervised k-means clustering (k=2) via [dlib](https://github.com/davisking/dlib) | No manual training or labeled data required |
16
-| **Model Architecture** | rolling 18 models per metric, 3-hour staggered training | Eliminates 99% of false positives through consensus |
17
-| **Processing Location** | Edge computation on each Netdata agent | No cloud dependency, no data egress |
18
-| **Resource Usage** | ~18KB RAM per metric, 2-4% of a single CPU for 10k metrics | Predictable linear scaling |
19
-| **Configuration** | Zero-configuration with automatic adaptation | Works instantly on any metric type |
20
-| **Detection Latency** | Real-time during data collection | Anomalies flagged within 1 second |
21
-| **Historical Storage** | Anomaly bit embedded in metric storage | No additional storage overhead |
22
-| **Query Performance** | On-the-fly anomaly rate calculation | No pre-aggregation needed |
23
-| **Time-series Integrity** | Immutable anomaly history | No hindsight bias - shows what was detectable THEN |
24
-| **Coverage** | Every metric, every dimension | No sampling, no blind spots |
25
-| **Correlation Engine** | Real-time anomaly correlation across metrics | Powers Anomaly Advisor for root cause analysis |
26
-| **Alert Philosophy** | Investigation aid, not alert source | Reduces alert fatigue |
13
+| Aspect | Implementation | Benefit |
14
+|---------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------------------|
15
+| **Algorithm** | Unsupervised k-means clustering (k=2) via [dlib](https://github.com/davisking/dlib) | No manual training or labeled data required |
16
+| **Model Architecture** | rolling 18 models per metric, 3-hour staggered training | Eliminates 99% of false positives through consensus |
17
+| **Processing Location** | Edge computation on each Netdata agent | No cloud dependency, no data egress |
18
+| **Resource Usage** | ~18KB RAM per metric, 2-4% of a single CPU for 10k metrics | Predictable linear scaling |
19
+| **Configuration** | Zero-configuration with automatic adaptation | Works instantly on any metric type |
20
+| **Detection Latency** | Real-time during data collection | Anomalies flagged within 1 second |
21
+| **Historical Storage** | Anomaly bit embedded in metric storage | No additional storage overhead |
22
+| **Query Performance** | On-the-fly anomaly rate calculation | No pre-aggregation needed |
23
+| **Time-series Integrity** | Immutable anomaly history | No hindsight bias - shows what was detectable THEN |
24
+| **Coverage** | Every metric, every dimension | No sampling, no blind spots |
25
+| **Correlation Engine** | Real-time anomaly correlation across metrics | Powers Anomaly Advisor for root cause analysis |
26
+| **Alert Philosophy** | Primary: investigation aid; also supports Anomaly Rate alerts | Reduces alert fatigue while enabling proactive notification |
27
28
:::note
29
Netdata avoids deep learning models to maintain lightweight operation on any Linux system. The entire ML system is designed to run efficiently without specialized hardware or dependencies.
@@ -158,6 +158,15 @@ You can see **Node Anomaly Rate (NAR)** and **Dimension Anomaly Rate (DAR)** cal
158
159
Netdata tracks the percentage of anomaly bits over time for you. When the **Node Anomaly Rate (NAR)** exceeds a set threshold and remains high for a period, a **node anomaly event** is triggered. These events are recorded in the `new_anomaly_event` dimension on the `anomaly_detection.anomaly_detection` chart.
160
161
+## Alerting on Anomaly Rates
162
+
163
+Netdata supports creating alerts based on ML anomaly rates, so you can be notified when anomaly rates exceed thresholds. There are two patterns:
164
+
165
+- **Chart-level anomaly rate alerts**: Use `anomaly-bit` in the `lookup` line to calculate the anomaly rate for a specific chart's dimensions. For example, `lookup: average -5m anomaly-bit of *` computes the rolling 5-minute anomaly rate for a chart.
166
+- **Node-level anomaly rate alerts**: Use the `anomaly_detection.anomaly_rate` chart to monitor the overall anomaly rate across all ML-enabled dimensions on a node.
167
+
168
+You can create Anomaly Rate alerts through the [Alerts Configuration Manager](/docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md) UI wizard, or write them manually. The [Alert Configuration Reference](/src/health/REFERENCE.md) includes complete working examples for both patterns (Examples 6 and 7).
169
+
170
## Available Documentation
171
172
- **[ML Configuration](/src/ml/ml-configuration.md)** - Configuration and tuning guide
@@ -256,5 +265,6 @@ ML is enabled by default in recent Netdata versions. To use anomaly detection:
265
1. **View anomaly ribbons** - Purple overlays on all charts show anomaly rates
266
2. **Access Anomaly Advisor** - Click the Anomalies tab for guided troubleshooting
267
3. **Query historical anomalies** - Use the query engine to analyze past incidents
268
+4. **Create Anomaly Rate alerts** - Get notified when anomaly rates exceed thresholds
269
270
[Learn more about the Anomaly Advisor →](/docs/ml-ai/anomaly-advisor.md)