docs: add anomaly-bit health alert guide and reference entry (#22307)
* docs: add anomaly-bit health alert guide and reference entry Add a practical 'Creating Anomaly-Based Health Alerts' section to the ML anomaly detection doc, showing how to use anomaly-bit in lookup lines and pair anomaly alerts with actual metric value alerts for richer notification context. Document anomaly-bit as a lookup option in the health REFERENCE. * docs: Execution completed * docs: Execution completed * docs: Execution completed * docs: add anomaly-bit health alert guide and reference entry Add a practical 'Creating Anomaly-Based Health Alerts' section to the ML anomaly detection doc, showing how to use anomaly-bit in lookup lines and pair anomaly alerts with actual metric value alerts for richer notification context. Document anomaly-bit as a lookup option in the health REFERENCE. Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> * docs: Execution completed Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> * docs: Execution completed Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> * docs: Execution completed Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> * docs: rebase onto master and resolve doc merge conflicts Agent-Logs-Url: https://github.com/netdata/netdata/sessions/8fd826b3-f540-4449-89fa-afed9bda5a13 Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> * docs: fix copilot review findings in ml-anomaly-detection alerts section - Fix critical threshold `> (100)` → `>= (100)` so it can trigger at max anomaly rate - Fix "Alert Philosophy" table row to reflect that anomaly bits can also drive health alerts - Fix CPU companion alert info text to clarify only user+system dimensions are measured - Fix foreach/of tip: explain they serve different purposes (foreach = per-instance, of = dimension selection) Agent-Logs-Url: https://github.com/netdata/netdata/sessions/c23772e9-2849-4a7e-b089-de5e29c7aa6f Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> * fix: use space-separated dimensions in health lookup example Agent-Logs-Url: https://github.com/netdata/netdata/sessions/506bc4b4-f07b-49d3-8378-0962c142d89f Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> * docs: clarify anomaly alert examples Agent-Logs-Url: https://github.com/netdata/netdata/sessions/71e3d489-fa4f-4c7a-af60-a7a71316ee28 Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update creating-alerts-with-netdata-alerts-configuration-manager.md * Update ml-anomaly-detection.md --------- Co-authored-by: nedi-app[bot] <nedi-app[bot]@users.noreply.github.com> Co-authored-by: Ancairon <70198089+Ancairon@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Fotis Voutsas <fotis@netdata.cloud> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
nedi-app[bot] committed
May 22, 2026 at 09:46 UTC
ae981af12d4d105fe130c9babd0fbbd6a1e13541
5 files changed
+220
-178
docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md
+27
-33
@@ -39,21 +39,15 @@ flowchart LR
39
40
You can choose from different ways to trigger alerts based on your monitoring needs:
41
42
-| Type | Description |
43
-|-----------------------|---------------------------------------------------------------|
44
-| **Standard** | Fires when a metric crosses a set value |
45
-| **Metric Variance** | Fires based on variation in values over time |
46
-| **Anomaly Rate** | Fires when the ML-calculated anomaly rate exceeds a threshold |
42
+| Type | Description |
43
+|---------------------|---------------------------------------------------------------------|
44
+| **Standard** | Fires when a metric crosses a set value |
45
+| **Metric Variance** | Fires based on variation in values over time |
46
+| **Anomaly Rate** | Fires when Netdata's ML-derived anomaly rate exceeds a set threshold |
47
48
-Choose the type that best suits the behavior you want to monitor.
49
-
50
-The **Anomaly Rate** detection type uses ML-generated anomaly bits to calculate what percentage of data points in a time window were flagged as anomalous. Thresholds are expressed as percentages (0–100% anomaly rate). You can configure it from any chart's alert bell icon by selecting Anomaly Rate as the detection type.
51
-
52
-:::tip
48
+For **Anomaly Rate** alerts, the threshold is the ML anomaly rate on a `0`-`100` scale, where higher values indicate more anomalous behavior. See [ML-powered anomaly detection](/docs/ml-ai/ml-anomaly-detection/ml-anomaly-detection.md#creating-anomaly-based-health-alerts) and the [health reference](/src/health/REFERENCE.md) for more details on configuring anomaly-rate checks.
49
54
-For details on how anomalies are detected, see [ML Anomaly Detection](/docs/ml-ai/ml-anomaly-detection/ml-anomaly-detection.md). For the underlying alert configuration syntax with `anomaly-bit` lookups, see the [Alert Configuration Reference](/src/health/REFERENCE.md) (Examples 6 and 7).
55
-
56
-:::
50
+Choose the type that best suits the behavior you want to monitor.
51
52
## Metrics Lookup & Filters
53
@@ -63,12 +57,12 @@ Click **Show advanced** in the Alert Configuration Manager to access these optio
57
58
You can define how Netdata should query the data before triggering an alert:
59
66
-| Parameter | Description |
67
-|----------------|----------------------------------------------------------------------|
68
-| `method` | How values are aggregated (`avg`, `min`, `max`, etc.) |
69
-| `duration` | Time window used for the check |
70
-| `dimensions` | Which metric dimensions to include |
71
-| `options` | Modify how values are interpreted (e.g., `percentage`, `absolute`) |
60
+| Parameter | Description |
61
+|--------------|----------------------------------------------------------------------|
62
+| `method` | How values are aggregated (`avg`, `min`, `max`, etc.) |
63
+| `duration` | Time window used for the check |
64
+| `dimensions` | Which metric dimensions to include |
65
+| `options` | Modify how values are interpreted (e.g., `percentage`, `absolute`) |
66
67
### Filtering Targets
68
@@ -94,23 +88,23 @@ Example:
88
89
You control how and when alerts are triggered, escalated, or resolved:
90
97
-| Setting | Purpose |
98
-|-------------------------------|-------------------------------------------------------------------|
99
-| **Thresholds** | Define values for `warning` and `critical` states |
100
-| **Recovery thresholds** | Set when the alert should downgrade or clear |
101
-| **Check interval** | How often the alert check runs (e.g., every 10 seconds) |
102
-| **Notification delay** | Delay before sending notifications for state changes |
103
-| **Repeat notifications** | How often to resend an alert if the issue persists (Agent only) |
104
-| **Notification recipients** | Define who gets alerted (Agent only) |
105
-| **Custom exec script** | Run a custom shell script when an alert triggers |
91
+| Setting | Purpose |
92
+|------------------------------|------------------------------------------------------------------|
93
+| **Thresholds** | Define values for `warning` and `critical` states |
94
+| **Recovery thresholds** | Set when the alert should downgrade or clear |
95
+| **Check interval** | How often the alert check runs (e.g., every 10 seconds) |
96
+| **Notification delay** | Delay before sending notifications for state changes |
97
+| **Repeat notifications** | How often to resend an alert if the issue persists (Agent only) |
98
+| **Notification recipients** | Define who gets alerted (Agent only) |
99
+| **Custom exec script** | Run a custom shell script when an alert triggers |
100
101
## Naming and Documentation
102
109
-| Field | Description |
110
-|-------------------|-------------------------------------------------------|
111
-| **Alert Name** | A unique name for the alert |
112
-| **Description** | What the alert does, in one or two sentences |
113
-| **Summary** | Optional: a short summary for display in dashboards |
103
+| Field | Description |
104
+|-----------------|------------------------------------------------------|
105
+| **Alert Name** | A unique name for the alert |
106
+| **Description** | What the alert does, in one or two sentences |
107
+| **Summary** | Optional: a short summary for display in dashboards |
108
109
## Final Notes
110
docs/dashboards-and-charts/anomaly-advisor-tab.md
+9
-9
@@ -10,11 +10,11 @@ For configuration details, see the [ML documentation](/src/ml/README.md).
10
11
The Anomaly Advisor leverages Netdata’s machine learning to evaluate anomaly rates across your nodes. It provides three key visualizations:
12
13
-| Chart Name | Purpose | Why It Matters |
14
-|--------------------------------|-------------------------------------------------------------------|----------------------------------------------------------------|
15
-| **Anomaly Rate** | Shows the percentage of anomalous metrics over time per node. | Helps you quickly spot nodes behaving abnormally. |
16
-| **Count of Anomalous Metrics** | Displays raw counts of anomalous metrics per node. | Useful when nodes have different numbers of collected metrics. |
17
-| **Anomaly Events Detected** | Indicates when the anomaly rate has triggered a node-level event. | Focuses your attention on meaningful spikes, not just noise. |
13
+| Chart Name | Purpose | Why It Matters |
14
+|-------------------------------|--------------------------------------------------------------------|-------------------------------------------------------|
15
+| **Anomaly Rate** | Shows the percentage of anomalous metrics over time per node. | Helps you quickly spot nodes behaving abnormally. |
16
+| **Count of Anomalous Metrics**| Displays raw counts of anomalous metrics per node. | Useful when nodes have different numbers of metrics. |
17
+| **Anomaly Events Detected** | Indicates when the anomaly rate has triggered a node-level event. | Focuses your attention on meaningful spikes, not noise. |
18
19
:::note
20
@@ -38,10 +38,10 @@ Use the [node filter](/docs/dashboards-and-charts/node-filter.md) to focus on sp
38
39
## Usage Tips
40
41
-| Tip | Why It Matters |
42
-|-----------------------------------------------|------------------------------------------------------------------|
43
-| Filter to specific nodes before highlighting. | Reduces noise by limiting averaging across unrelated nodes. |
44
-| Highlight close to the anomaly spike. | Improves ranking accuracy by focusing on the relevant timeframe. |
41
+| Tip | Why It Matters |
42
+|------------------------------------------------|------------------------------------------------------------------|
43
+| Filter to specific nodes before highlighting. | Reduces noise by limiting averaging across unrelated nodes. |
44
+| Highlight close to the anomaly spike. | Improves ranking accuracy by focusing on the relevant timeframe. |
45
46
## Anomaly Advisor Diagram
47
docs/ml-ai/anomaly-advisor.md
+12
-12
@@ -12,18 +12,18 @@ Limitations: Works best for sudden changes and patterns not seen in the last 54
12
13
## System Characteristics
14
15
-| Aspect | Implementation | Operational Benefit |
16
-|--------|----------------|-------------------|
17
-| **Data Source** | Per-metric anomaly detection using k-means (k=2) with 18-model consensus | Comprehensive coverage, no blind spots |
18
-| **Correlation Engine** | Pre-computed Node Anomaly Rate (NAR) charts updated in real-time | Instant blast radius visualization |
19
-| **Query Engine** | Specialized scoring engine evaluating thousands of metrics simultaneously | Returns ranked list, not time-series data |
20
-| **Ranking Algorithm** | Anomaly severity scoring across selected time window | Root cause typically in top 30-50 results |
21
-| **Infrastructure View** | Dual charts: % anomalous and absolute count per node | Distinguishes small node spikes from large node issues |
22
-| **Time to Insight** | Highlight timeframe → ranked results in seconds |Minutes to root cause vs hours of hypothesis testing |
23
-| **Expertise Required** | No system-specific knowledge needed to identify anomalies | Minimal expertise to interpret results |
24
-| **Dependency Discovery** | Correlated anomalies reveal component relationships | Exposes hidden infrastructure dependencies |
25
-| **Best Use Cases** | Sudden changes, cascading failures, multi-node incidents | Excellent for "what just happened?" scenarios |
26
-| **Limitations** | Cannot detect stopped services, gradual degradation | Not a replacement for all monitoring |
15
+| Aspect | Implementation | Operational Benefit |
16
+|-------------------------|---------------------------------------------------------------------------------|----------------------------------------------------|
17
+| **Data Source** | Per-metric anomaly detection using k-means (k=2) with 18-model consensus | Comprehensive coverage, no blind spots |
18
+| **Correlation Engine** | Pre-computed Node Anomaly Rate (NAR) charts updated in real-time | Instant blast radius visualization |
19
+| **Query Engine** | Specialized scoring engine evaluating thousands of metrics simultaneously | Returns ranked list, not time-series data |
20
+| **Ranking Algorithm** | Anomaly severity scoring across selected time window | Root cause typically in top 30-50 results |
21
+| **Infrastructure View** | Dual charts: % anomalous and absolute count per node | Distinguishes small node spikes from large issues |
22
+| **Time to Insight** | Highlight timeframe → ranked results in seconds | Minutes to root cause vs hours of hypothesis testing |
23
+| **Expertise Required** | No system-specific knowledge needed to identify anomalies | Minimal expertise to interpret results |
24
+| **Dependency Discovery**| Correlated anomalies reveal component relationships | Exposes hidden infrastructure dependencies |
25
+| **Best Use Cases** | Sudden changes, cascading failures, multi-node incidents | Excellent for "what just happened?" scenarios |
26
+| **Limitations** | Cannot detect stopped services, gradual degradation | Not a replacement for all monitoring |
27
28
:::warning Limitations
29
docs/ml-ai/ml-anomaly-detection/ml-anomaly-detection.md
+82
-37
@@ -10,20 +10,20 @@ A dedicated process correlates anomalies across all metrics within each node, ge
10
11
## System Characteristics
12
13
-| Aspect | Implementation | Benefit |
14
-|---------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------------------|
15
-| **Algorithm** | Unsupervised k-means clustering (k=2) via [dlib](https://github.com/davisking/dlib) | No manual training or labeled data required |
16
-| **Model Architecture** | rolling 18 models per metric, 3-hour staggered training | Eliminates 99% of false positives through consensus |
17
-| **Processing Location** | Edge computation on each Netdata agent | No cloud dependency, no data egress |
18
-| **Resource Usage** | ~18KB RAM per metric, 2-4% of a single CPU for 10k metrics | Predictable linear scaling |
19
-| **Configuration** | Zero-configuration with automatic adaptation | Works instantly on any metric type |
20
-| **Detection Latency** | Real-time during data collection | Anomalies flagged within 1 second |
21
-| **Historical Storage** | Anomaly bit embedded in metric storage | No additional storage overhead |
22
-| **Query Performance** | On-the-fly anomaly rate calculation | No pre-aggregation needed |
23
-| **Time-series Integrity** | Immutable anomaly history | No hindsight bias - shows what was detectable THEN |
24
-| **Coverage** | Every metric, every dimension | No sampling, no blind spots |
25
-| **Correlation Engine** | Real-time anomaly correlation across metrics | Powers Anomaly Advisor for root cause analysis |
26
-| **Alert Philosophy** | Primary: investigation aid; also supports Anomaly Rate alerts | Reduces alert fatigue while enabling proactive notification |
13
+| Aspect | Implementation | Benefit |
14
+|-------------------------|---------------------------------------------------------------------------------------------|------------------------------------------------------|
15
+| **Algorithm** | Unsupervised k-means clustering (k=2) via [dlib](https://github.com/davisking/dlib) | No manual training or labeled data required |
16
+| **Model Architecture** | Rolling 18 models per metric, 3-hour staggered training | Eliminates 99% of false positives through consensus |
17
+| **Processing Location** | Edge computation on each Netdata Agent | No cloud dependency, no data egress |
18
+| **Resource Usage** | ~18KB RAM per metric, 2-4% of a single CPU for 10k metrics | Predictable linear scaling |
19
+| **Configuration** | Zero-configuration with automatic adaptation | Works instantly on any metric type |
20
+| **Detection Latency** | Real-time during data collection | Anomalies flagged within 1 second |
21
+| **Historical Storage** | Anomaly bit embedded in metric storage | No additional storage overhead |
22
+| **Query Performance** | On-the-fly anomaly rate calculation | No pre-aggregation needed |
23
+| **Time-series Integrity** | Immutable anomaly history | No hindsight bias — shows what was detectable then |
24
+| **Coverage** | Every metric, every dimension | No sampling, no blind spots |
25
+| **Correlation Engine** | Real-time anomaly correlation across metrics | Powers Anomaly Advisor for root cause analysis |
26
+| **Alert Philosophy** | Primarily an investigation aid; anomaly bits and anomaly rate can also drive health alerts | Reduces alert fatigue while enabling anomaly-based alerting |
27
28
:::note
29
Netdata avoids deep learning models to maintain lightweight operation on any Linux system. The entire ML system is designed to run efficiently without specialized hardware or dependencies.
@@ -31,13 +31,13 @@ Netdata avoids deep learning models to maintain lightweight operation on any Lin
31
32
## Types of Anomalies Detected
33
34
-| Anomaly Type | Description | Business Impact |
35
-|--------------------------|-------------------------------------------------------------------|------------------------------------------|
36
-| **Point Anomalies** | Unusually high or low values compared to historical data | Early warning of service degradation |
37
-| **Contextual Anomalies** | Sequences of values that deviate from expected patterns | Identification of unusual usage patterns |
38
-| **Collective Anomalies** | Multivariate anomalies where a combination of metrics appears off | Detection of complex system issues |
39
-| **Concept Drifts** | Gradual shifts leading to a new baseline | Recognition of evolving system behavior |
40
-| **Change Points** | Sudden shifts resulting in a new normal state | Identification of system changes |
34
+| Anomaly Type | Description | Business Impact |
35
+|--------------------------|------------------------------------------------------------------|-------------------------------------------|
36
+| **Point Anomalies** | Unusually high or low values compared to historical data | Early warning of service degradation |
37
+| **Contextual Anomalies** | Sequences of values that deviate from expected patterns | Identification of unusual usage patterns |
38
+| **Collective Anomalies** | Multivariate anomalies where a combination of metrics appears off | Detection of complex system issues |
39
+| **Concept Drifts** | Gradual shifts leading to a new baseline | Recognition of evolving system behavior |
40
+| **Change Points** | Sudden shifts resulting in a new normal state | Identification of system changes |
41
42
## Technical Deep Dive: How Netdata ML Works
43
@@ -158,15 +158,6 @@ You can see **Node Anomaly Rate (NAR)** and **Dimension Anomaly Rate (DAR)** cal
158
159
Netdata tracks the percentage of anomaly bits over time for you. When the **Node Anomaly Rate (NAR)** exceeds a set threshold and remains high for a period, a **node anomaly event** is triggered. These events are recorded in the `new_anomaly_event` dimension on the `anomaly_detection.anomaly_detection` chart.
160
161
-## Alerting on Anomaly Rates
162
-
163
-Netdata supports creating alerts based on ML anomaly rates, so you can be notified when anomaly rates exceed thresholds. There are two patterns:
164
-
165
-- **Chart-level anomaly rate alerts**: Use `anomaly-bit` in the `lookup` line to calculate the anomaly rate for a specific chart's dimensions. For example, `lookup: average -5m anomaly-bit of *` computes the rolling 5-minute anomaly rate for a chart.
166
-- **Node-level anomaly rate alerts**: Use the `anomaly_detection.anomaly_rate` chart to monitor the overall anomaly rate across all ML-enabled dimensions on a node.
167
-
168
-You can create Anomaly Rate alerts through the [Alerts Configuration Manager](/docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md) UI wizard, or write them manually. The [Alert Configuration Reference](/src/health/REFERENCE.md) includes complete working examples for both patterns (Examples 6 and 7).
169
-
161
## Available Documentation
162
163
- **[ML Configuration](/src/ml/ml-configuration.md)** - Configuration and tuning guide
@@ -258,13 +249,67 @@ On a freshly installed agent, ML begins detecting anomalies within 10 minutes. H
249
250
**Operational tip**: During the first 48 hours after deployment, expect elevated anomaly rates. This is normal as the system learns your infrastructure's patterns. Use this period to observe ML behavior but avoid making critical decisions based solely on early anomaly detection.
251
261
-## Getting Started
252
+## Creating Anomaly-Based Health Alerts
253
+
254
+You can create health alerts that trigger based on anomaly rates instead of raw metric values by using the `anomaly-bit` option in your alert's `lookup` line. Internally, anomaly bits mark samples as anomalous or normal, and the query/health pipeline exposes this as an anomaly-rate percentage in the 0-100 range. For aggregated or tiered data, returned values can be intermediate percentages rather than only 0 or 100, so averaging over a time window gives you the anomaly rate as a percentage.
255
+
256
+### Anomaly-rate alert
257
+
258
+The following template triggers when the anomaly rate on `system.cpu` exceeds the defined thresholds:
259
+
260
+```text
261
+ template: ml_5min_cpu_chart
262
+ on: system.cpu
263
+ lookup: average -5m anomaly-bit of *
264
+ calc: $this
265
+ units: %
266
+ every: 30s
267
+ warn: $this > (($status >= $WARNING) ? (5) : (20))
268
+ crit: $this >= (($status == $CRITICAL) ? (20) : (100))
269
+ info: rolling 5min anomaly rate for system.cpu chart
270
+```
271
+
272
+### Pairing with actual values
273
+
274
+An anomaly-rate alert tells you *something is unusual*, but not *what the actual numbers are*. To get the real values alongside the anomaly alert, create a companion alert on the raw metric:
275
+
276
+```text
277
+ alarm: cpu_usage_5min
278
+ on: system.cpu
279
+lookup: average -5m of user,system
280
+ units: %
281
+ every: 30s
282
+ warn: $this > 80
283
+ crit: $this > 95
284
+ info: average user+system CPU utilization over the last 5 minutes
285
+```
286
+
287
+When the anomaly alert fires, the companion alert can provide the concrete values in a separate alert — for example, "CPU anomaly rate 35%" alongside "CPU utilization 92%".
288
+
289
+:::tip
290
+
291
+Use `foreach` in a template to generate one alert instance per dimension (e.g., one per CPU state). Note that `foreach` and `of` serve different purposes: `of` selects which dimensions the `lookup` aggregates, while `foreach` creates separate alert instances for each matching dimension in a template. For the full alert syntax, see the [health configuration reference](/src/health/REFERENCE.md).
292
+
293
+:::
294
+
295
+### Adding context to alert notifications
296
+
297
+The `info` and `summary` fields support template variables that add contextual detail to notifications:
298
+
299
+| Variable | Replaced With |
300
+|----------------------|-----------------------------------|
301
+| `${family}` | Family instance (for example, `eth0`) |
302
+| `${label:LABEL_NAME}` | Chart label value |
303
+
304
+:::important
305
+
306
+`$this` is available only in `calc`, `warn`, and `crit` expressions — not in the `info` or `summary` fields.
307
+
308
+:::
309
263
-ML is enabled by default in recent Netdata versions. To use anomaly detection:
310
+### Investigating anomaly alerts
311
265
-1. **View anomaly ribbons** - Purple overlays on all charts show anomaly rates
266
-2. **Access Anomaly Advisor** - Click the Anomalies tab for guided troubleshooting
267
-3. **Query historical anomalies** - Use the query engine to analyze past incidents
268
-4. **Create Anomaly Rate alerts** - Get notified when anomaly rates exceed thresholds
312
+When an anomaly alert fires, use Netdata's built-in tools to investigate the root cause:
313
270
-[Learn more about the Anomaly Advisor →](/docs/ml-ai/anomaly-advisor.md)
314
+- **[Alert Troubleshooting](/docs/netdata-ai/troubleshooting/index.md)** — generate a one-click report from any fired alert, assessing its validity, uncovering correlated signals, and proposing a root-cause hypothesis
315
+- **[Investigations](/docs/netdata-ai/investigations/index.md)** — ask open-ended questions about your infrastructure for deeper analysis beyond a single alert
src/health/REFERENCE.md
+90
-87
@@ -82,7 +82,7 @@ crit: $this > (($status == $CRITICAL) ? (75) : (85))
82
### Task 2: Disable Unwanted Alerts
83
84
| Method | Use Case | Configuration File | How To |
85
-|-------------------------|-------------------------------------|--------------------|------------------------------------------|
85
+| ----------------------- | ----------------------------------- | ------------------ | ---------------------------------------- |
86
| Disable all alerts | Testing/maintenance | netdata.conf | Set `enabled = no` in `[health]` section |
87
| Disable specific alerts | Remove noisy alerts | netdata.conf | Set `enabled alarms = !alert_name *` |
88
| Silence notifications | Keep monitoring, stop notifications | Alert config file | Change `to: silent` |
@@ -138,7 +138,7 @@ sudo killall -USR2 netdata
138
**Configuration Locations:**
139
140
| Location | Purpose | Common Tasks | How to Edit |
141
-|-----------------------------------|------------------------------|----------------------------------------------------------------------------------------------------------|------------------------------------------|
141
+| --------------------------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------- |
142
| `netdata.conf` `[health]` section | Global health settings | • Disable all monitoring (`enabled = no`)<br />• Disable specific alerts<br />• Change check frequencies | Edit directly or use `edit-config` |
143
| `health.d/*.conf` files | Individual alert definitions | • Modify thresholds<br />• Change notification recipients<br />• Silence alerts (`to: silent`) | Use `edit-config health.d/filename.conf` |
144
@@ -223,7 +223,7 @@ This action requires that you [reload Netdata's health configuration](#how-to-re
223
**Use Case:** Scheduled maintenance or dynamic control
224
225
| Scenario | Solution | Method |
226
-|------------------------------------|--------------------------------------------|------------------------------------|
226
+| ---------------------------------- | ------------------------------------------ | ---------------------------------- |
227
| Disable alerts during backups | Use health management API | API calls without config changes |
228
| Suppress notifications temporarily | Keep checks running, silence notifications | API control of notification system |
229
@@ -273,7 +273,7 @@ lookup: average -1m percentage of used
273
**Step 3: Understand Each Component**
274
275
| Line | Purpose | This Example |
276
-|-------------|-------------------------------------------|-------------------------------------------------------|
276
+| ----------- | ----------------------------------------- | ----------------------------------------------------- |
277
| `alarm` | Entity name (alphanumeric, `.`, `_` only) | `ram_usage` |
278
| `on` | Chart to monitor | `system.ram` |
279
| `lookup` | How to process metrics | Average last 1 minute, percentage of `used` dimension |
@@ -311,7 +311,7 @@ Complete syntax reference for all alert configuration options. Use this section
311
### Entity Types Overview
312
313
| Type | Label | Purpose | Example Use Case |
314
-|---------------|-------------|----------------------------------|--------------------------------|
314
+| ------------- | ----------- | -------------------------------- | ------------------------------ |
315
| **Alerts** | `alarm:` | Attached to specific charts | Monitor specific server's CPU |
316
| **Templates** | `template:` | Apply to all charts of a context | Monitor all network interfaces |
317
@@ -350,7 +350,7 @@ For complete details on configuration loading order and precedence rules, see [A
350
### Complete Configuration Reference
351
352
| line | required | functionality |
353
-|-----------------------------------------------------|-----------------|--------------------------------------------------------------------------------------|
353
+| --------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------ |
354
| [`alarm`/`template`](#alert-line-alarm-or-template) | yes | Name of the alert/template |
355
| [`on`](#alert-line-on) | yes | The chart this alert should attach to |
356
| [`class`](#alert-line-class) | no | The general alert classification |
@@ -444,7 +444,7 @@ class: Latency
444
**Available Classes:**
445
446
| Class | Use Case |
447
-|-------------|--------------------------------|
447
+| ----------- | ------------------------------ |
448
| Errors | Error rate monitoring |
449
| Latency | Response time issues |
450
| Utilization | Resource usage monitoring |
@@ -469,7 +469,7 @@ type: Database
469
**Available Types:**
470
471
| Type | Description |
472
-|-----------------|------------------------------------------------------------------------------------------------|
472
+| --------------- | ---------------------------------------------------------------------------------------------- |
473
| Ad Filtering | Services related to Ad Filtering (like pi-hole) |
474
| Certificates | Certificate monitoring related |
475
| Cgroups | Alerts for CPU and memory usage of control groups |
@@ -529,34 +529,37 @@ lookup: METHOD(GROUPING OPTIONS) AFTER [at BEFORE] [every DURATION] [OPTIONS] [o
529
**Required Parameters:**
530
531
| Parameter | Description | Example |
532
-|-----------|--------------------------------------------------------------------|-------------------------|
532
+| --------- | ------------------------------------------------------------------ | ----------------------- |
533
| `METHOD` | [Grouping method](/src/web/api/queries/README.md#grouping-methods) | `average`, `min`, `max` |
534
| `AFTER` | How far back to look (negative number) | `-1m`, `-1h`, `-1d` |
535
536
**Optional Parameters:**
537
538
| Parameter | Purpose | Details |
539
-|--------------------|-----------------------------|----------------------------------------------------------------------------|
539
+| ------------------ | --------------------------- | -------------------------------------------------------------------------- |
540
| `GROUPING OPTIONS` | Conditional processing | `CONDITION VALUE` where condition is `!=`, `=`, `==`, `<=`, `<`, `>`, `>=` |
541
| `at BEFORE` | End of lookup timeframe | Default is 0 (now) |
542
| `every DURATION` | Update frequency | Supports `s`, `m`, `h`, `d` units |
543
| `OPTIONS` | Processing modifiers | See options table below |
544
-| `of DIMENSIONS` | Which dimensions to include | Space-separated list, supports patterns |
544
+| `of DIMENSIONS` | Which dimensions to include | Comma- or pipe-separated list, supports patterns; prefer `user,system` over `user, system` |
545
546
**Processing Options:**
547
548
-| Option | Effect |
549
-|---------------|---------------------------------------------------------|
550
-| `percentage` | Calculate percentage of selected dimensions over total |
551
-| `absolute` | Turn all sample values positive |
552
-| `min` | Return minimum of all dimensions after time-aggregation |
553
-| `max` | Return maximum of all dimensions after time-aggregation |
554
-| `average` | Return average of all dimensions after time-aggregation |
555
-| `sum` | Return sum of all dimensions (default) |
556
-| `min2max` | Return delta between min and max of dimensions |
557
-| `unaligned` | Prevent shifting query window to multiples of duration |
558
-| `match-ids` | Match dimensions by IDs (default) |
559
-| `match-names` | Match dimensions by names |
548
+| Option | Effect |
549
+| ------------- | -------------------------------------------------------------------------------------------------------- |
550
+| `percentage` | Calculate percentage of selected dimensions over total |
551
+| `absolute` | Turn all sample values positive |
552
+| `min` | Return minimum of all dimensions after time-aggregation |
553
+| `max` | Return maximum of all dimensions after time-aggregation |
554
+| `average` | Return average of all dimensions after time-aggregation |
555
+| `sum` | Return sum of all dimensions (default) |
556
+| `min2max` | Return delta between min and max of dimensions |
557
+| `unaligned` | Prevent shifting query window to multiples of duration |
558
+| `anomaly-bit` | Query anomaly-rate percentages (0-100) instead of raw values, enabling anomaly-rate-based alerting |
559
+| `match-ids` | Match dimensions by IDs (default) |
560
+| `match-names` | Match dimensions by names |
561
+
562
+When `anomaly-bit` is used, each data point returns the anomaly rate as a percentage from 0 to 100. At native resolution this is typically 0 (normal) or 100 (anomalous), while aggregated or lower-resolution data can yield intermediate values such as 12.5. For more details and practical examples, see the [ML anomaly detection documentation](/docs/ml-ai/ml-anomaly-detection/ml-anomaly-detection.md).
563
564
**Example:**
565
@@ -691,7 +694,7 @@ delay: [[[up U] [down D] multiplier M] max X]
694
**Parameters:**
695
696
| Parameter | Purpose | Default |
694
-|----------------|--------------------------------------------------------------|---------------|
697
+| -------------- | ------------------------------------------------------------ | ------------- |
698
| `up U` | Delay for status increases (CLEAR→WARNING, WARNING→CRITICAL) | 0 |
699
| `down D` | Delay for status decreases (CRITICAL→WARNING, WARNING→CLEAR) | 0 |
700
| `multiplier M` | Multiplies U and D when alert changes state during delay | 1.0 |
@@ -706,7 +709,7 @@ delay: up 10s down 15m multiplier 2 max 1h
709
Starting at `00:00:00` with CLEAR status:
710
711
| Time | New Status | Delay Applied | Notification At | Reason |
709
-|----------|------------|----------------|-----------------|-------------------------------------------|
712
+| -------- | ---------- | -------------- | --------------- | ----------------------------------------- |
713
| 00:00:01 | WARNING | `up 10s` | 00:00:11 | First state switch |
714
| 00:00:05 | CLEAR | `down 15m x2` | 00:30:05 | Alert changed during delay, so multiplied |
715
| 00:00:06 | WARNING | `up 10s x2 x2` | 00:00:26 | Multiplied twice |
@@ -722,7 +725,7 @@ repeat: [off] [warning DURATION] [critical DURATION]
725
**Options:**
726
727
| Option | Effect |
725
-|---------------------|----------------------------------------------------------|
728
+| ------------------- | -------------------------------------------------------- |
729
| `off` | Turns off repeating for this alert |
730
| `warning DURATION` | Repeat interval for WARNING state (use `0s` to disable) |
731
| `critical DURATION` | Repeat interval for CRITICAL state (use `0s` to disable) |
@@ -822,7 +825,7 @@ summary: Available Ram
825
**Variable Support:**
826
827
| Variable | Replaced With |
825
-|-----------------------|------------------------------|
828
+| --------------------- | ---------------------------- |
829
| `${family}` | Family instance (e.g., eth0) |
830
| `${label:LABEL_NAME}` | Chart label value |
831
@@ -851,7 +854,7 @@ info: Percentage of estimated amount of RAM available for userspace processes, w
854
**Variable Support:**
855
856
| Variable | Replaced With |
854
-|-----------------------|------------------------------|
857
+| --------------------- | ---------------------------- |
858
| `${family}` | Family instance (e.g., eth0) |
859
| `${label:LABEL_NAME}` | Chart label value |
860
@@ -892,10 +895,10 @@ How to write calculations and use variables in your alert definitions. Essential
895
**Supported Operators:**
896
897
| Type | Operators | Result |
895
-|------------|----------------------------------------|---------------------------|
898
+| ---------- | -------------------------------------- | ------------------------- |
899
| Arithmetic | `+`, `-`, `*`, `/` | Numeric values |
900
| Comparison | `<`, `==`, `<=`, `<>`, `!=`, `>`, `>=` | `1` (true) or `0` (false) |
898
-| Logical | `&&`, `||`,`!`,`AND`,`OR`,`NOT` | `1` (true) or `0` (false) |
901
+| Logical | `&&`, `||`,`!`,`AND`,`OR`,`NOT` | `1` (true) or `0` (false) |
902
903
**Special Functions:**
904
@@ -905,7 +908,7 @@ How to write calculations and use variables in your alert definitions. Essential
908
**Special Values:**
909
910
| Value | Purpose | Example Use |
908
-|-------|---------------------------------------|----------------|
911
+| ----- | ------------------------------------- | -------------- |
912
| `nan` | Not a number (database lookup failed) | `$this != nan` |
913
| `inf` | Infinite (division by zero) | `$this != inf` |
914
@@ -929,7 +932,7 @@ crit: $this > (($status == $CRITICAL) ? (85) : (95))
932
**How This Works:**
933
934
| Alert State | Triggers At | Clears At | Explanation |
932
-|-------------|-------------|-----------|-------------------------------------------------------------------|
935
+| ----------- | ----------- | --------- | ----------------------------------------------------------------- |
936
| Warning | 85% CPU | 75% CPU | Creates 10% buffer - CPU must drop below 75% to clear warning |
937
| Critical | 95% CPU | 85% CPU | Creates 10% buffer - CPU must drop below 85% to return to warning |
938
@@ -969,7 +972,7 @@ Although the `alarm_variables` link shows variables for a particular chart, the
972
**Special Chart Variables:**
973
974
| Variable | Contains |
972
-|------------------------|-----------------------------------------------|
975
+| ---------------------- | --------------------------------------------- |
976
| `$last_collected_t` | Unix timestamp of last data collection |
977
| `$collected_total_raw` | Sum of all dimensions (last collected values) |
978
| `$update_every` | Update frequency of the chart |
@@ -1044,7 +1047,7 @@ URL-encode the chart ID only in API query parameters. In alert expressions, use
1047
#### Special Variables
1048
1049
| Variable | Contains | Usage |
1047
-|-----------|------------------------|----------------------------------------|
1050
+| --------- | ---------------------- | -------------------------------------- |
1051
| `$this` | Current alert value | Result of `calc` line or current alert |
1052
| `$status` | Current alert status | Compare with status constants |
1053
| `$now` | Current unix timestamp | Time-based calculations |
@@ -1052,7 +1055,7 @@ URL-encode the chart ID only in API query parameters. In alert expressions, use
1055
**Alert Status Constants:**
1056
1057
| Constant | Numeric Value | Usage |
1055
-|------------------|---------------|--------------------------------|
1058
+| ---------------- | ------------- | ------------------------------ |
1059
| `$REMOVED` | -2 | Alert deleted (SIGUSR2 reload) |
1060
| `$UNINITIALIZED` | -1 | Alert not initialized |
1061
| `$UNDEFINED` | 0 | Calculation failed |
@@ -1122,7 +1125,7 @@ template: apache_last_collected_secs
1125
**How It Works:**
1126
1127
| Component | Purpose | This Example |
1125
-|------------|---------------------------------|------------------------------|
1128
+| ---------- | ------------------------------- | ---------------------------- |
1129
| `template` | Applies to all Apache servers | `apache_last_collected_secs` |
1130
| `on` | Chart context to monitor | `apache.requests` |
1131
| `calc` | Time since last data collection | `$now - $last_collected_t` |
@@ -1160,7 +1163,7 @@ template: disk_full_percent
1163
**How It Works:**
1164
1165
| Component | Purpose | This Example |
1163
-|-------------|----------------------------|----------------------------------------|
1166
+| ----------- | -------------------------- | -------------------------------------- |
1167
| `template` | Applies to all disks | `disk_full_percent` |
1168
| `on` | Chart context | `disk.space` |
1169
| `calc` | Calculate usage percentage | `$used * 100 / ($avail + $used)` |
@@ -1206,7 +1209,7 @@ template: disk_full_percent
1209
**How It Works:**
1210
1211
| Step | Purpose | Calculation |
1209
-|------|----------------------|----------------------------------------------------|
1212
+| ---- | -------------------- | -------------------------------------------------- |
1213
| 1 | Calculate fill rate | `(space_30min_ago - current_space) / 1800_seconds` |
1214
| 2 | Predict time to full | `current_available / fill_rate / 3600` |
1215
@@ -1237,7 +1240,7 @@ template: 30min_packet_drops
1240
**How It Works:**
1241
1242
| Component | Purpose | This Example |
1240
-|------------|-----------------------------------|-------------------------------|
1243
+| ---------- | --------------------------------- | ----------------------------- |
1244
| `template` | Applies to all network interfaces | `30min_packet_drops` |
1245
| `lookup` | Sum drops over 30 minutes | `sum -30m unaligned absolute` |
1246
| `crit` | Any drops trigger critical | `$this > 0` |
@@ -1281,7 +1284,7 @@ lookup: mean -10s of user
1284
**How It Works:**
1285
1286
| Alert | Purpose | Calculation |
1284
-|-------------------|-----------------------------|------------------------------------|
1287
+| ----------------- | --------------------------- | ---------------------------------- |
1288
| `cpu_user_mean` | Calculate average CPU usage | Mean over 60 seconds |
1289
| `cpu_user_stddev` | Calculate variability | Standard deviation over 60 seconds |
1290
| `cpu_user_zscore` | Detect anomalies | `(current - mean) / stddev` |
@@ -1311,17 +1314,17 @@ template: ml_5min_cpu_chart
1314
units: %
1315
every: 30s
1316
warn: $this > (($status >= $WARNING) ? (5) : (20))
1314
- crit: $this > (($status == $CRITICAL) ? (20) : (100))
1317
+ crit: $this >= (($status == $CRITICAL) ? (20) : (100))
1318
info: rolling 5min anomaly rate for system.cpu chart
1319
```
1320
1321
**How It Works:**
1322
1323
| Component | Purpose | This Example |
1321
-|---------------|--------------------------------------------|-------------------------------------|
1324
+| ------------- | ------------------------------------------ | ----------------------------------- |
1325
| `lookup` | Average anomaly rate across CPU dimensions | 5-minute rolling window |
1326
| Hysteresis | Prevent alert flapping | Warning: 20%→5%, Critical: 100%→20% |
1324
-| `anomaly-bit` | ML-generated anomaly indicators | 0 (normal) or 1 (anomalous) |
1327
+| `anomaly-bit` | ML-generated anomaly indicators | 0 (normal) or 100 (anomalous) |
1328
1329
<br/>
1330
</details>
@@ -1341,7 +1344,7 @@ template: ml_5min_node
1344
units: %
1345
every: 30s
1346
warn: $this > (($status >= $WARNING) ? (5) : (20))
1344
- crit: $this > (($status == $CRITICAL) ? (20) : (100))
1347
+ crit: $this >= (($status == $CRITICAL) ? (20) : (100))
1348
info: rolling 5min anomaly rate for all ML enabled dims
1349
```
1350
@@ -1372,7 +1375,7 @@ How to debug alert issues, understand why alerts aren't working, and get detaile
1375
You can find chart information in two places:
1376
1377
| Method | URL | Contains |
1375
-|---------------|-----------------------------------|-------------------|
1378
+| ------------- | --------------------------------- | ----------------- |
1379
| Configuration | `http://NODE:19999/netdata.conf` | All chart details |
1380
| API | `http://NODE:19999/api/v1/charts` | JSON chart data |
1381
@@ -1494,7 +1497,7 @@ flowchart TD
1497
**Possible Causes:**
1498
1499
| Problem | Check This | Solution |
1497
-|----------------------|---------------------------------|-------------------------------|
1500
+| -------------------- | ------------------------------- | ----------------------------- |
1501
| Wrong chart name | `on:` line matches actual chart | Use chart ID from dashboard |
1502
| Incorrect dimensions | Dimension names in `lookup` | Check available dimensions |
1503
| Missing data | Chart has recent data | Verify data collection |
@@ -1505,7 +1508,7 @@ flowchart TD
1508
**Possible Causes:**
1509
1510
| Problem | Check This | Solution |
1508
-|---------------------------|------------------------------------------|-------------------------|
1511
+| ------------------------- | ---------------------------------------- | ----------------------- |
1512
| Wrong threshold direction | `>` vs `<` in expressions | Review logic |
1513
| Units mismatch | Comparing percentages to absolute values | Check calculation units |
1514
| Variable name errors | `$this` vs `$chart.dimension` | Verify variable names |
@@ -1515,7 +1518,7 @@ flowchart TD
1518
**Possible Causes:**
1519
1520
| Problem | Solution |
1518
-|-----------------------|--------------------------------------------------------------|
1521
+| --------------------- | ------------------------------------------------------------ |
1522
| Values near threshold | Implement [hysteresis](#conditional-operator-for-hysteresis) |
1523
| Noisy data | Increase lookup time window |
1524
| Too frequent checks | Increase `every:` interval |
@@ -1590,7 +1593,7 @@ warn: $this > (($status >= $WARNING) ? (75) : (80))
1593
**Alert Impact on System:**
1594
1595
| Factor | Impact | Optimization |
1593
-|----------------------------|---------------------|------------------------------------|
1596
+| -------------------------- | ------------------- | ---------------------------------- |
1597
| Check frequency (`every:`) | CPU usage | Use appropriate intervals |
1598
| Lookup timeframe | Memory/CPU | Don't use excessively long periods |
1599
| Number of alerts | Overall performance | Disable unused alerts |
@@ -1599,7 +1602,7 @@ warn: $this > (($status >= $WARNING) ? (75) : (80))
1602
**Recommended Frequencies:**
1603
1604
| Alert Type | Suggested Frequency | Reason |
1602
-|-------------------------|---------------------|----------------------------------------|
1605
+| ----------------------- | ------------------- | -------------------------------------- |
1606
| Critical system metrics | 10-30s | Quick response needed |
1607
| Resource usage | 1-5m | Trends matter more than instant values |
1608
| Predictive alerts | 15m-1h | Based on longer-term patterns |
@@ -1638,44 +1641,44 @@ When seeking help, include:
1641
1642
The following variables are available in alert notification templates and custom notification scripts:
1643
1641
-| Variable name | Description |
1642
-|:---------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------|
1643
-| `${alarm}` | Like "name = value units" |
1644
-| `${status_message}` | Like "needs attention", "recovered", "is critical" |
1645
-| `${severity}` | Like "Escalated to CRITICAL", "Recovered from WARNING" |
1646
-| `${raised_for}` | Like "(alarm was raised for 10 minutes)" |
1647
-| `${host}` | The host generated this event |
1648
-| `${url_host}` | Same as `${host}` but URL encoded |
1649
-| `${unique_id}` | The unique id of this event |
1650
-| `${alarm_id}` | The unique id of the alarm that generated this event |
1651
-| `${event_id}` | The incremental id of the event, for this alarm id |
1652
-| `${when}` | The timestamp this event occurred |
1653
-| `${date}` | The date and time the event occurred (local timezone) |
1654
-| `${date_utc}` | The date and time the event occurred (UTC) |
1655
-| `${name}` | The name of the alarm, as given in netdata health.d entries |
1656
-| `${url_name}` | Same as `${name}` but URL encoded |
1657
-| `${chart}` | The name of the chart (type.id) |
1658
-| `${url_chart}` | Same as `${chart}` but URL encoded |
1659
-| `${status}` | The current status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
1660
-| `${old_status}` | The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
1661
-| `${value}` | The current value of the alarm |
1662
-| `${old_value}` | The previous value of the alarm |
1663
-| `${src}` | The line number and file the alarm has been configured |
1664
-| `${duration}` | The duration in seconds of the previous alarm state |
1665
-| `${duration_txt}` | Same as `${duration}` for humans |
1666
-| `${non_clear_duration}` | The total duration in seconds this is/was non-clear. For repeating alerts in WARNING or CRITICAL state, Netdata sends `${duration}` instead. |
1644
+| Variable name | Description |
1645
+| :-------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------- |
1646
+| `${alarm}` | Like "name = value units" |
1647
+| `${status_message}` | Like "needs attention", "recovered", "is critical" |
1648
+| `${severity}` | Like "Escalated to CRITICAL", "Recovered from WARNING" |
1649
+| `${raised_for}` | Like "(alarm was raised for 10 minutes)" |
1650
+| `${host}` | The host generated this event |
1651
+| `${url_host}` | Same as `${host}` but URL encoded |
1652
+| `${unique_id}` | The unique id of this event |
1653
+| `${alarm_id}` | The unique id of the alarm that generated this event |
1654
+| `${event_id}` | The incremental id of the event, for this alarm id |
1655
+| `${when}` | The timestamp this event occurred |
1656
+| `${date}` | The date and time the event occurred (local timezone) |
1657
+| `${date_utc}` | The date and time the event occurred (UTC) |
1658
+| `${name}` | The name of the alarm, as given in netdata health.d entries |
1659
+| `${url_name}` | Same as `${name}` but URL encoded |
1660
+| `${chart}` | The name of the chart (type.id) |
1661
+| `${url_chart}` | Same as `${chart}` but URL encoded |
1662
+| `${status}` | The current status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
1663
+| `${old_status}` | The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |
1664
+| `${value}` | The current value of the alarm |
1665
+| `${old_value}` | The previous value of the alarm |
1666
+| `${src}` | The line number and file the alarm has been configured |
1667
+| `${duration}` | The duration in seconds of the previous alarm state |
1668
+| `${duration_txt}` | Same as `${duration}` for humans |
1669
+| `${non_clear_duration}` | The total duration in seconds this is/was non-clear. For repeating alerts in WARNING or CRITICAL state, Netdata sends `${duration}` instead. |
1670
| `${non_clear_duration_txt}` | Same as `${non_clear_duration}` for humans |
1668
-| `${units}` | The units of the value |
1669
-| `${info}` | A short description of the alarm |
1670
-| `${value_string}` | Friendly value (with units) |
1671
-| `${old_value_string}` | Friendly old value (with units) |
1672
-| `${image}` | The URL of an image to represent the status of the alarm |
1673
-| `${color}` | A color in #AABBCC format for the alarm |
1674
-| `${goto_url}` | The URL the user can click to see the netdata dashboard |
1675
-| `${calc_expression}` | The expression evaluated to provide the value for the alarm |
1676
-| `${calc_param_values}` | The values of the variables in the evaluated expression |
1677
-| `${total_warnings}` | The total number of alarms in WARNING state on the host |
1678
-| `${total_critical}` | The total number of alarms in CRITICAL state on the host |
1671
+| `${units}` | The units of the value |
1672
+| `${info}` | A short description of the alarm |
1673
+| `${value_string}` | Friendly value (with units) |
1674
+| `${old_value_string}` | Friendly old value (with units) |
1675
+| `${image}` | The URL of an image to represent the status of the alarm |
1676
+| `${color}` | A color in #AABBCC format for the alarm |
1677
+| `${goto_url}` | The URL the user can click to see the netdata dashboard |
1678
+| `${calc_expression}` | The expression evaluated to provide the value for the alarm |
1679
+| `${calc_param_values}` | The values of the variables in the evaluated expression |
1680
+| `${total_warnings}` | The total number of alarms in WARNING state on the host |
1681
+| `${total_critical}` | The total number of alarms in CRITICAL state on the host |
1682
1683
## Related Pages
1684