Updated configuration reference (#20347)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Kanela committed
May 28, 2025 at 11:34 UTC
2a21c425233440017e17cff25e2337e2b0389b70
1 file changed
+1166
-600
src/health/REFERENCE.md
+1166
-600
@@ -1,49 +1,152 @@
1
-# Configure alerts
1
+# Configure Health Alerts
2
3
-Netdata's health watchdog is highly configurable, with support for dynamic thresholds, hysteresis, alert templates, and
4
-more. You can tweak any of the existing alerts based on your infrastructure's topology or specific monitoring needs, or
5
-create new entities.
3
+## Table of Contents
4
7
-You can use health alerts in conjunction with any of Netdata's [collectors](/src/collectors/README.md) (see
8
-the [supported collector list](/src/collectors/COLLECTORS.md)) to monitor the health of your systems, containers, and
9
-applications in real time.
5
+- [Quick Start Guide](#quick-start-guide) <!-- SECTION_TYPE: tutorial -->
6
+- [Common Tasks](#common-tasks) <!-- SECTION_TYPE: workflow -->
7
+- [How-To Guides](#how-to-guides) <!-- SECTION_TYPE: procedures -->
8
+- [Alert Configuration Reference](#alert-configuration-reference) <!-- SECTION_TYPE: reference -->
9
+- [Expressions and Variables](#expressions-and-variables) <!-- SECTION_TYPE: concepts -->
10
+- [Alert Examples](#alert-examples) <!-- SECTION_TYPE: examples -->
11
+- [Troubleshooting](#troubleshooting) <!-- SECTION_TYPE: problem_solving -->
12
11
-While you can see active alerts both on the local dashboard and Netdata Cloud, all health alerts are configured _per
12
-node_ via individual Netdata Agents. If you want to deploy a new alert across your
13
-[infrastructure](/docs/netdata-cloud/organize-your-infrastructure-invite-your-team.md), you must configure each node with the same health configuration
14
-files.
13
+## Quick Start Guide
14
16
-## Reload health configuration
15
+:::tip
16
18
-You don’t need to restart the Netdata Agent between changes to health configuration files, such as specific health entities. Instead, you can use `netdatacli` and the `reload-health` option to prevent gaps in metrics collection.
17
+**What You'll Learn**
18
+
19
+In 5 minutes, you'll know how to edit existing alerts, create new ones, and reload your configuration without downtime.
20
+
21
+:::
22
+
23
+### Get Started in 3 Steps
24
+
25
+**Step 1: Find Your Config Directory**
26
+
27
+Navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md)
28
+
29
+**Step 2: Edit an Alert**
30
+
31
+```bash
32
+sudo ./edit-config health.d/cpu.conf
33
+```
34
+
35
+**Step 3: Apply Changes**
36
37
```bash
38
sudo netdatacli reload-health
39
```
40
24
-If `netdatacli` doesn't work on your system, send a `SIGUSR2` signal to the daemon, which reloads health configuration without restarting the entire process.
41
+:::note
42
+
43
+**Key Concept**
44
+
45
+You can highly configure Netdata's health watchdog with support for dynamic thresholds, hysteresis, alert templates, and more. You can customize any existing alerts based on your infrastructure's topology or specific monitoring needs or create entirely new entities.
46
+
47
+You can use health alerts with any of Netdata's [collectors](/src/collectors/README.md) (see the [supported collector list](/src/collectors/COLLECTORS.md)) to monitor your systems, containers, and applications in real time.
48
+
49
+While you can view active alerts on both the local dashboard and Netdata Cloud, you configure all health alerts _per node_ via individual Netdata Agents. If you want to deploy a new alert across your [infrastructure](/docs/netdata-cloud/organize-your-infrastructure-invite-your-team.md), you must configure each node with the same health configuration files.
50
+
51
+:::
52
+
53
+**Next Steps:** Jump to [Common Tasks](#common-tasks) for specific workflows or continue reading for comprehensive guidance.
54
+
55
+## Common Tasks
56
+
57
+:::tip
58
+
59
+**What You'll Learn**
60
+
61
+Step-by-step workflows for the most frequent alert configuration tasks.
62
+
63
+:::
64
+
65
+### Task 1: Modify Alert Thresholds
66
+
67
+**Why This Matters:** Default thresholds may not fit your specific environment or requirements.
68
+
69
+**Quick Example:**
70
+
71
+```text
72
+# Change CPU warning from 85% to 75%
73
+warn: $this > (($status >= $WARNING) ? (60) : (75))
74
+crit: $this > (($status == $CRITICAL) ? (75) : (85))
75
+```
76
+
77
+**Step-by-Step:**
78
+
79
+1. Find the alert file: `sudo ./edit-config health.d/cpu.conf`
80
+2. Locate the alert (e.g., `10min_cpu_usage`)
81
+3. Modify `warn` and `crit` lines
82
+4. Save and reload: `sudo netdatacli reload-health`
83
+
84
+### Task 2: Disable Unwanted Alerts
85
+
86
+| Method | Use Case | Configuration File | How To |
87
+|-------------------------|-------------------------------------|--------------------|------------------------------------------|
88
+| Disable all alerts | Testing/maintenance | netdata.conf | Set `enabled = no` in `[health]` section |
89
+| Disable specific alerts | Remove noisy alerts | netdata.conf | Set `enabled alarms = !alert_name *` |
90
+| Silence notifications | Keep monitoring, stop notifications | Alert config file | Change `to: silent` |
91
+
92
+### Task 3: Create a Simple Alert
93
+
94
+**Real-World Example:** Monitor RAM usage above 80%
95
+
96
+```text
97
+alarm: ram_usage
98
+ on: system.ram
99
+lookup: average -1m percentage of used
100
+units: %
101
+every: 1m
102
+ warn: $this > 80
103
+ crit: $this > 90
104
+ info: RAM usage monitoring
105
+```
106
+
107
+**Next Steps:** See [How-To Guides](#how-to-guides) for detailed explanations.
108
+
109
+## How-To Guides
110
+
111
+:::tip
112
+
113
+**What You'll Learn**
114
+
115
+Detailed instructions for configuring, managing, and troubleshooting health alerts.
116
+
117
+:::
118
+
119
+### How to Reload Health Configuration
120
+
121
+**Why This Matters:** You don't need to restart your Netdata Agent when making changes, preventing gaps in monitoring.
122
+
123
+You don't need to restart your Netdata Agent when making changes to health configuration files, such as specific health entities. Instead, you can use `netdatacli` with the `reload-health` option to prevent gaps in metrics collection.
124
+
125
+```bash
126
+sudo netdatacli reload-health
127
+```
128
+
129
+**Alternative Method:**
130
+If `netdatacli` doesn't work on your system, you can send a `SIGUSR2` signal to the daemon, which reloads health configuration without restarting the entire process.
131
132
```bash
27
-killall -USR2 netdata
133
+sudo killall -USR2 netdata
134
```
135
30
-## Edit health configuration files
136
+### How to Edit Health Configuration Files
137
32
-You can configure the Agent's health watchdog service by editing files in two locations:
138
+**Configuration Locations:**
139
34
-- The `[health]` section in `netdata.conf`. By editing the daemon's behavior, you can disable health monitoring
35
- altogether, run health checks more or less often, and more. See
36
- [daemon configuration](/src/daemon/config/README.md#health-section-options) for a table of
37
- all the available settings, their default values, and what they control.
140
+**Configuration Locations:**
141
39
-- The individual `.conf` files in `health.d/`. These health entity files are organized by the type of metric they’re
40
- performing calculations on or their associated collector. You should edit these files using the `edit-config`
41
- script. For example: `sudo ./edit-config health.d/cpu.conf`.
142
+| Location | Purpose | Common Tasks | How to Edit |
143
+|-----------------------------------|------------------------------|----------------------------------------------------------------------------------------------------------|------------------------------------------|
144
+| `netdata.conf` `[health]` section | Global health settings | • Disable all monitoring (`enabled = no`)<br />• Disable specific alerts<br />• Change check frequencies | Edit directly or use `edit-config` |
145
+| `health.d/*.conf` files | Individual alert definitions | • Modify thresholds<br />• Change notification recipients<br />• Silence alerts (`to: silent`) | Use `edit-config health.d/filename.conf` |
146
43
-Navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md) and
44
-use `edit-config` to make changes to any of these files.
147
+Navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md) and use `edit-config` to make changes to any of these files.
148
46
-### Edit individual alerts
149
+**Edit Individual Alerts:**
150
151
For example, to edit the `cpu.conf` health configuration file, run:
152
@@ -51,10 +154,11 @@ For example, to edit the `cpu.conf` health configuration file, run:
154
sudo ./edit-config health.d/cpu.conf
155
```
156
54
-Each health configuration file contains one or more health _entities_, which always begin with `alarm:` or `template:`.
55
-For example, here is the first health entity in `health.d/cpu.conf`:
157
+**Understanding Alert Structure:**
158
+
159
+Each health configuration file contains one or more health [_entities_](#complete-configuration-reference), which always begin with `alarm:` or `template:`. Here's the first health entity in `health.d/cpu.conf`:
160
57
-```yaml
161
+```text
162
template: 10min_cpu_usage
163
on: system.cpu
164
class: Utilization
@@ -71,83 +175,93 @@ component: CPU
175
to: sysadmin
176
```
177
74
-To tune this alert to trigger warning and critical alerts at a lower CPU utilization, change the `warn` and `crit` lines
75
-to the values of your choosing. For example:
178
+To customize this alert to trigger warning and critical alerts at lower CPU utilization levels, you can change the `warn` and `crit` lines to values of your choosing. For example:
179
77
-```yaml
180
+```text
181
warn: $this > (($status >= $WARNING) ? (60) : (75))
182
crit: $this > (($status == $CRITICAL) ? (75) : (85))
183
```
184
82
-Save the file and [reload Netdata's health configuration](#reload-health-configuration) to apply your changes.
185
+Save the file and [reload Netdata's health configuration](#how-to-reload-health-configuration) to apply your changes.
186
+
187
+### How to Disable or Silence Alerts
188
+
189
+**Why This Matters:** Different situations require different approaches to managing alerts - permanent removal, temporary silencing, or selective filtering.
190
84
-## Disable or silence alerts
191
+You can disable alerts and notifications permanently via configuration changes, or temporarily via the [health management API](/src/web/api/health/README.md).
192
86
-Alerts and notifications can be disabled permanently via configuration changes, or temporarily, via the
87
-[health management API](/src/web/api/health/README.md). The
88
-available options are described below.
193
+#### Disable All Alerts
194
90
-### Disable all alerts
195
+**Use Case:** System maintenance or testing
196
92
-In the `netdata.conf` `[health]` section, set `enabled` to `no`, and restart the Agent.
197
+In the `netdata.conf` `[health]` section, set `enabled` to `no`, and restart your Agent.
198
94
-### Disable some alerts
199
+#### Disable Specific Alerts
200
96
-In the `netdata.conf` `[health]` section, set `enabled alarms` to a
97
-[simple pattern](https://github.com/netdata/netdata/edit/master/src/libnetdata/simple_pattern/README.md) that
98
-excludes one or more alerts. e.g. `enabled alarms = !oom_kill *` will load all alerts except `oom_kill`.
201
+**Use Case:** Remove known noisy or irrelevant alerts
202
100
-You can also [edit the file where the alert is defined](#edit-individual-alerts), comment out its definition,
101
-and [reload Netdata's health configuration](#reload-health-configuration).
203
+In the `netdata.conf` `[health]` section, use [pattern](/src/libnetdata/simple_pattern/README.md) exclusion with `enabled alarms = !oom_kill *` to load all alerts except `oom_kill`.
204
103
-### Silence an individual alert
205
+You can also [edit the file where the alert is defined](#how-to-edit-health-configuration-files), comment out its definition, and [reload Netdata's health configuration](#how-to-reload-health-configuration).
206
105
-You can stop receiving notification for an individual alert by [changing](#edit-individual-alerts) the `to:` line to `silent`.
207
+#### Silence Individual Alert Notifications
208
107
-```yaml
209
+**Use Case:** Keep monitoring active but stop notifications
210
+
211
+You can stop receiving notifications for an individual alert by [changing](#how-to-edit-health-configuration-files) the `to:` line to `silent` in the alert's configuration file.
212
+
213
+```text
214
to: silent
215
```
216
111
-This action requires that you [reload Netdata's health configuration](#reload-health-configuration).
217
+:::tip
218
+
219
+This action requires that you [reload Netdata's health configuration](#how-to-reload-health-configuration).
220
+
221
+:::
222
113
-### Temporarily disable alerts at runtime
223
+#### Temporary Runtime Control
224
115
-When you need to frequently disable all or some alerts from triggering during certain times (for instance,
116
-when running backups), you can use the
117
-[health management API](/src/web/api/health/README.md).
118
-The API allows you to issue commands to control the health engine's behavior without changing configuration,
119
-or restarting the Agent.
225
+**Use Case:** Scheduled maintenance or dynamic control
226
121
-### Temporarily silence notifications at runtime
227
+| Scenario | Solution | Method |
228
+|------------------------------------|--------------------------------------------|------------------------------------|
229
+| Disable alerts during backups | Use health management API | API calls without config changes |
230
+| Suppress notifications temporarily | Keep checks running, silence notifications | API control of notification system |
231
123
-If you want health checks to keep running and alerts to keep getting triggered, but notifications to be
124
-suppressed temporarily, you can use the
125
-[health management API](/src/web/api/health/README.md).
126
-The API allows you to issue commands to control the health engine's behavior without changing configuration,
127
-or restarting the Agent.
232
+:::tip
233
129
-## Write a new health entity
234
+You can use the [health management API](/src/web/api/health/README.md) to temporarily control alert behavior without changing configuration or restarting your Agent. The API allows you to:
235
131
-While tuning existing alerts may work in some cases, you may need to write entirely new health entities based on how
132
-your systems, containers, and applications work.
236
+- **Disable all or some alerts** from triggering during certain times (for instance, when running backups)
237
+- **Suppress notifications temporarily** while keeping health checks running and alerts triggering
238
134
-Read the [health entity reference](#health-entity-reference) for a full listing of the format,
135
-syntax, and functionality of health entities.
239
+:::
240
137
-To write a new health entity into a new file, navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md),
138
-then use `touch` to create a new file in the `health.d/` directory. Use `edit-config` to start editing the file.
241
+### How to Write a New Health Entity
242
140
-As an example, let's create a `ram-usage.conf` file.
243
+**Why This Matters:** While tuning existing alerts may work in some cases, you may need to write entirely new health entities based on how your systems, containers, and applications work.
244
+
245
+**Prerequisites:** Read the [Alert Configuration Reference](#alert-configuration-reference) for a complete listing of the format, syntax, and functionality of health entities.
246
+
247
+**Step-by-Step Process:**
248
+
249
+**Step 1: Create the Configuration File**
250
+
251
+Navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md), then use `touch` to create a new file in the `health.d/` directory. Use `edit-config` to start editing the file.
252
+
253
+As an example, let's create a `ram-usage.conf` file:
254
255
```bash
256
sudo touch health.d/ram-usage.conf
257
sudo ./edit-config health.d/ram-usage.conf
258
```
259
147
-For example, here is a health entity that triggers a warning alert when a node's RAM usage rises above 80%, and a
148
-critical alert above 90%:
260
+**Step 2: Write Your Alert**
261
+
262
+Here's a health entity that triggers a warning alert when your node's RAM usage rises above 80%, and a critical alert above 90%:
263
150
-```yaml
264
+```text
265
alarm: ram_usage
266
on: system.ram
267
lookup: average -1m percentage of used
@@ -158,146 +272,156 @@ lookup: average -1m percentage of used
272
info: The percentage of RAM being used by the system.
273
```
274
161
-Let's look into each of the lines to see how they create a working health entity.
275
+**Step 3: Understand Each Component**
276
+
277
+| Line | Purpose | This Example |
278
+|-------------|-------------------------------------------|-------------------------------------------------------|
279
+| `alarm` | Entity name (alphanumeric, `.`, `_` only) | `ram_usage` |
280
+| `on` | Chart to monitor | `system.ram` |
281
+| `lookup` | How to process metrics | Average last 1 minute, percentage of `used` dimension |
282
+| `units` | Display units | Percentages (`%`) |
283
+| `every` | Check frequency | Every 1 minute |
284
+| `warn/crit` | Trigger conditions | Warning > 80%, Critical > 90% |
285
+| `info` | Alert description | Appears in dashboard and notifications |
286
+
287
+:::note
288
163
-- `alarm`: The name for your new entity. The name needs to follow these requirements:
164
- - Any alphabet letter or number.
165
- - The symbols `.` and `_`.
166
- - Cannot be `chart name`, `dimension name`, `family name`, or `chart variable names`.
289
+**Understanding This Example**
290
168
-- `on`: Which chart the entity listens to.
291
+This health entity, named **ram_usage**, watches the **system.ram** chart. It looks up the last **1 minute** of metrics from the **used** dimension and calculates the **average** of all those metrics in a **percentage** format, using **% units**. The entity performs this lookup **every minute**.
292
170
-- `lookup`: Which metrics the alert monitors, the duration of time to monitor, and how to process the metrics into a
171
- usable format.
172
- - `average`: Calculate the average of all the metrics collected.
173
- - `-1m`: Use metrics from 1 minute ago until now to calculate that average.
174
- - `percentage`: Clarify that we're calculating a percentage of RAM usage.
175
- - `of used`: Specify which dimension (`used`) on the `system.ram` chart you want to monitor with this entity.
293
+If the average RAM usage percentage over the last 1 minute is **more than 80%**, the entity triggers a warning alert. If the usage is **more than 90%**, the entity triggers a critical alert.
294
177
-- `units`: Use percentages rather than absolute units.
295
+:::
296
179
-- `every`: How often to perform the `lookup` calculation to decide whether to trigger this alert.
297
+**Step 4: Activate Your Alert**
298
181
-- `warn`/`crit`: The value at which Netdata should trigger a warning or critical alert. This example uses simple
182
- syntax, but most pre-configured health entities use
183
- [hysteresis](#special-use-of-the-conditional-operator) to avoid superfluous notifications.
299
+When you finish writing this new health entity, [reload Netdata's health configuration](#how-to-reload-health-configuration) to see it live on your local dashboard or Netdata Cloud.
300
185
-- `info`: A description of the alert, which will appear in the dashboard and notifications.
301
+**Next Steps:** Explore [Alert Examples](#alert-examples) for more complex scenarios, or dive into the [Alert Configuration Reference](#alert-configuration-reference) for complete syntax details.
302
187
-In human-readable format:
303
+## Alert Configuration Reference
304
189
-> This health entity, named **ram_usage**, watches the **system.ram** chart. It looks up the last **1 minute** of
190
-> metrics from the **used** dimension and calculates the **average** of all those metrics in a **percentage** format,
191
-> using a **% unit**. The entity performs this lookup **every minute**.
192
->
193
-> If the average RAM usage percentage over the last 1 minute is **more than 80%**, the entity triggers a warning alert.
194
-> If the usage is **more than 90%**, the entity triggers a critical alert.
305
+:::tip
306
196
-When you finish writing this new health entity, [reload Netdata's health configuration](#reload-health-configuration) to
197
-see it live on the local dashboard or Netdata Cloud.
307
+**What You'll Learn**
308
199
-## Health entity reference
309
+Complete syntax reference for all alert configuration options. Use this section when you need specific technical details.
310
201
-The following reference contains information about the syntax and options of _health entities_, which Netdata attaches
202
-to charts to trigger alerts.
311
+:::
312
204
-### Entity types
313
+### Entity Types Overview
314
206
-There are two entity types: **alarms** and **templates**. They have the same format and feature set—the only difference
207
-is their label.
315
+| Type | Label | Purpose | Example Use Case |
316
+|---------------|-------------|----------------------------------|--------------------------------|
317
+| **Alerts** | `alarm:` | Attached to specific charts | Monitor specific server's CPU |
318
+| **Templates** | `template:` | Apply to all charts of a context | Monitor all network interfaces |
319
320
**Alerts** are attached to specific charts and use the `alarm` label.
321
211
-**Templates** define rules that apply to all charts of a specific context, and use the `template` label. Templates help
212
-you apply one entity to all disks, all network interfaces, all MySQL databases, and so on.
213
-
214
-Alerts have higher precedence and will override templates.
215
-If the `alert` and `template` entities have the same name and are attached to the same chart, Netdata will use `alarm`.
216
-
217
-### Entity format
218
-
219
-Netdata parses the following lines. Beneath the table is an in-depth explanation of each line's purpose and syntax.
220
-
221
-- The `alarm` or `template` line must be the first line of any entity.
222
-- The `on` line is **always required**.
223
-- The `every` line is **required** if not using `lookup`.
224
-- Each entity **must** have at least one of the following lines: `lookup`, `calc`, `warn`, or `crit`.
225
-- A few lines use space-separated lists to define how the entity behaves. You can use `*` as a wildcard or prefix with
226
- `!` for a negative match. Order is important, too! See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for
227
- more examples.
228
-- Lines terminated by a `\` are spliced together with the next line. The backslash is removed, and the following line is
229
- joined with the current one. No space is inserted, so you may split a line anywhere, even in the middle of a word.
230
- This comes in handy if your `info` line consists of several sentences.
231
-
232
-| line | required | functionality |
233
-|-----------------------------------------------------|-----------------|---------------------------------------------------------------------------------------|
234
-| [`alarm`/`template`](#alert-line-alarm-or-template) | yes | Name of the alert/template. |
235
-| [`on`](#alert-line-on) | yes | The chart this alert should attach to. |
236
-| [`class`](#alert-line-class) | no | The general alert classification. |
237
-| [`type`](#alert-line-type) | no | What area of the system the alert monitors. |
238
-| [`component`](#alert-line-component) | no | Specific component of the type of the alert. |
239
-| [`lookup`](#alert-line-lookup) | yes | The database lookup to find and process metrics for the chart specified through `on`. |
240
-| [`calc`](#alert-line-calc) | yes (see above) | A calculation to apply to the value found via `lookup` or another variable. |
241
-| [`every`](#alert-line-every) | no | The frequency of the alert. |
242
-| [`green`/`red`](#alert-lines-green-and-red) | no | Set the green and red thresholds of a chart. |
243
-| [`warn`/`crit`](#alert-lines-warn-and-crit) | yes (see above) | Expressions evaluating to true or false, and when true, will trigger the alert. |
244
-| [`to`](#alert-line-to) | no | A list of roles to send notifications to. |
245
-| [`exec`](#alert-line-exec) | no | The script to execute when the alert changes status. |
246
-| [`delay`](#alert-line-delay) | no | Optional hysteresis settings to prevent floods of notifications. |
247
-| [`repeat`](#alert-line-repeat) | no | The interval for sending notifications when an alert is in WARNING or CRITICAL mode. |
248
-| [`options`](#alert-line-options) | no | Add an option to not clear alerts. |
249
-| [`host labels`](#alert-line-host-labels) | no | Restrict an alert or template to a list of matching labels present on a host. |
250
-| [`chart labels`](#alert-line-chart-labels) | no | Restrict an alert or template to a list of matching labels present on a chart. |
251
-| [`summary`](#alert-line-summary) | no | A brief description of the alert. |
252
-| [`info`](#alert-line-info) | no | A longer text field that provides more information of this alert |
253
-
254
-The `alarm` or `template` line must be the first line of any entity.
255
-
256
-#### Alert line `alarm` or `template`
257
-
258
-This line starts an alert or template based on the [entity type](#entity-types) you're interested in creating.
259
-
260
-**Alert:**
261
-
262
-```yaml
322
+**Templates** define rules that apply to all charts of a specific context, and use the `template` label. Templates help you apply one entity to all disks, all network interfaces, all MySQL databases, and so on.
323
+
324
+:::note
325
+
326
+**Precedence**
327
+
328
+Alerts have higher precedence and will override templates. If you have `alarm` and `template` entities with the same name attached to the same chart, Netdata will use `alarm`.
329
+
330
+:::
331
+
332
+### Required vs Optional Configuration
333
+
334
+:::note
335
+
336
+**Configuration Requirements**
337
+
338
+- The `alarm` or `template` line must be the first line of any entity
339
+- The `on` line is **always required**
340
+- The `every` line is **required** if not using `lookup`
341
+- Each entity **must** have at least one of the following lines: `lookup`, `calc`, `warn`, or `crit`
342
+
343
+:::
344
+
345
+**Special Syntax Rules:**
346
+
347
+- A few lines use space-separated lists to define how the entity behaves. You can use `*` as a wildcard or prefix with `!` for a negative match. Order is important! See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for more examples
348
+- Lines terminated by a `\` are spliced together with the next line. The backslash is removed, and the following line is joined with the current one. No space is inserted, so you can split a line anywhere, even in the middle of a word. This is handy if your `info` line consists of several sentences
349
+
350
+### Complete Configuration Reference
351
+
352
+| line | required | functionality |
353
+|-----------------------------------------------------|-----------------|--------------------------------------------------------------------------------------|
354
+| [`alarm`/`template`](#alert-line-alarm-or-template) | yes | Name of the alert/template |
355
+| [`on`](#alert-line-on) | yes | The chart this alert should attach to |
356
+| [`class`](#alert-line-class) | no | The general alert classification |
357
+| [`type`](#alert-line-type) | no | What area of the system the alert monitors |
358
+| [`component`](#alert-line-component) | no | Specific component of the type of the alert |
359
+| [`lookup`](#alert-line-lookup) | yes | The database lookup to find and process metrics for the chart specified through `on` |
360
+| [`calc`](#alert-line-calc) | yes (see above) | A calculation to apply to the value found via `lookup` or another variable |
361
+| [`every`](#alert-line-every) | no | The frequency of the alert |
362
+| [`green`/`red`](#alert-lines-green-and-red) | no | Set the green and red thresholds of a chart |
363
+| [`warn`/`crit`](#alert-lines-warn-and-crit) | yes (see above) | Expressions evaluating to true or false, and when true, will trigger the alert |
364
+| [`to`](#alert-line-to) | no | A list of roles to send notifications to |
365
+| [`exec`](#alert-line-exec) | no | The script to execute when the alert changes status |
366
+| [`delay`](#alert-line-delay) | no | Optional hysteresis settings to prevent floods of notifications |
367
+| [`repeat`](#alert-line-repeat) | no | The interval for sending notifications when an alert is in WARNING or CRITICAL mode |
368
+| [`options`](#alert-line-options) | no | Add an option to not clear alerts |
369
+| [`host labels`](#alert-line-host-labels) | no | Restrict an alert or template to a list of matching labels present on a host |
370
+| [`chart labels`](#alert-line-chart-labels) | no | Restrict an alert or template to a list of matching labels present on a chart |
371
+| [`summary`](#alert-line-summary) | no | A brief description of the alert |
372
+| [`info`](#alert-line-info) | no | A longer text field that provides more information about this alert |
373
+
374
+### Configuration Line Details
375
+
376
+#### Alert Line `alarm` or `template`
377
+
378
+**Purpose:** This line starts an alert or template based on the [entity type](#entity-types-overview) you want to create.
379
+
380
+**Alert Syntax:**
381
+
382
+```text
383
alarm: NAME
384
```
385
266
-**Template:**
386
+**Template Syntax:**
387
268
-```yaml
388
+```text
389
template: NAME
390
```
391
272
-`NAME` can be any alpha character, with `.` (period) and `_` (underscore) as the only allowed symbols, but the names
273
-can’t be `chart name`, `dimension name`, `family name`, or `chart variables names`.
392
+**Naming Rules:**
393
+
394
+- `NAME` can be any alphanumeric character
395
+- Only `.` (period) and `_` (underscore) symbols allowed
396
+- Can’t be `chart name`, `dimension name`, `family name`, or `chart variable names`
397
275
-#### Alert line `on`
398
+#### Alert Line `on`
399
277
-This line defines the chart this alert should attach to.
400
+**Purpose:** This line defines the chart this alert should attach to.
401
279
-**Alerts:**
402
+**For Alerts:**
403
281
-```yaml
404
+```text
405
on: CHART
406
```
407
285
-The value `CHART` should be the unique ID or name of the chart you're interested in, as shown on the dashboard. In the
286
-image below, the unique ID is `system.cpu`.
408
+The value `CHART` should be the unique ID or name of the chart you're interested in, as shown on the dashboard. In the image below, the unique ID is `system.cpu`.
409
288
-
410
+
411
291
-**Template:**
412
+**For Templates:**
413
293
-```yaml
414
+```text
415
on: CONTEXT
416
```
417
418
The value `CONTEXT` should be the context you want this template to attach to.
419
299
-Need to find the context? Hover over the date on any given chart and look at the tooltip. In the image below, which
300
-shows a disk I/O chart, the tooltip reads: `proc:/proc/diskstats, disk.io`.
420
+:::tip
421
+
422
+**Finding the Context**
423
+
424
+Need to find the context? Hover over the date on any given chart and look at the tooltip. In the image below, which shows a disk I/O chart, the tooltip reads: `proc:/proc/diskstats, disk.io`.
425
426
427
@@ -305,632 +429,676 @@ You're interested in what comes after the comma: `disk.io`. That's the name of t
429
430
If you create a template using the `disk.io` context, it will apply an alert to every disk available on your system.
431
308
-#### Alert line `class`
432
+:::
433
310
-This indicates the type of error (or general problem area) that the alert or template applies to. For example, `Latency` can be used for alerts that trigger on latency issues on network interfaces, web servers, or database systems. Example:
434
+#### Alert Line `class`
435
312
-```yaml
436
+**Purpose:** This indicates the type of error (or general problem area) that the alert or template applies to.
437
+
438
+**Example Use:** `Latency` can be used for alerts that trigger on latency issues on network interfaces, web servers, or database systems.
439
+
440
+```text
441
class: Latency
442
```
443
316
-<details>
317
-<summary>Netdata's stock alerts use the following `class` attributes by default:</summary>
444
+**Available Classes:**
445
319
-| Class |
320
-|-------------|
321
-| Errors |
322
-| Latency |
323
-| Utilization |
324
-| Workload |
446
+| Class | Use Case |
447
+|-------------|--------------------------------|
448
+| Errors | Error rate monitoring |
449
+| Latency | Response time issues |
450
+| Utilization | Resource usage monitoring |
451
+| Workload | Load and throughput monitoring |
452
326
-</details>
453
+:::note
454
455
`class` will default to `Unknown` if the line is missing from the alert configuration.
456
330
-#### Alert line `type`
457
+:::
458
332
-Type can be used to indicate the broader area of the system that the alert applies to. For example, under the general `Database` type, you can group together alerts that operate on various database systems, like `MySQL`, `CockroachDB`, `CouchDB` etc. Example:
459
+#### Alert Line `type`
460
334
-```yaml
461
+**Purpose:** You can use `type` to indicate the broader area of the system that the alert applies to.
462
+
463
+**Example:** Under the general `Database` type, you can group together alerts that operate on various database systems, like `MySQL`, `CockroachDB`, `CouchDB`, etc.
464
+
465
+```text
466
type: Database
467
```
468
338
-<details>
339
-<summary>Netdata's stock alerts use the following `type` attributes by default, but feel free to adjust for your own requirements.</summary>
469
+**Available Types:**
470
471
| Type | Description |
472
|-----------------|------------------------------------------------------------------------------------------------|
473
| Ad Filtering | Services related to Ad Filtering (like pi-hole) |
344
-| Certificates | Certificates monitoring related |
345
-| Cgroups | Alerts for cpu and memory usage of control groups |
474
+| Certificates | Certificate monitoring related |
475
+| Cgroups | Alerts for CPU and memory usage of control groups |
476
| Computing | Alerts for shared computing applications (e.g. boinc) |
477
| Containers | Container related alerts (e.g. docker instances) |
478
| Database | Database systems (e.g. MySQL, PostgreSQL, etc) |
479
| Data Sharing | Used to group together alerts for data sharing applications |
350
-| DHCP | Alerts for dhcp related services |
351
-| DNS | Alerts for dns related services |
480
+| DHCP | Alerts for DHCP related services |
481
+| DNS | Alerts for DNS related services |
482
| Kubernetes | Alerts for kubernetes nodes monitoring |
483
| KV Storage | Key-Value pairs services alerts (e.g. memcached) |
484
| Linux | Services specific to Linux (e.g. systemd) |
485
| Messaging | Alerts for message passing services (e.g. vernemq) |
486
| Netdata | Internal Netdata components monitoring |
357
-| Other | When an alert doesn't fit in other types. |
487
+| Other | When an alert doesn't fit in other types |
488
| Power Supply | Alerts from power supply related services (e.g. apcupsd) |
489
| Search engine | Alerts for search services (e.g. elasticsearch) |
490
| Storage | Class for alerts dealing with storage services (storage devices typically live under `System`) |
361
-| System | General system alerts (e.g. cpu, network, etc.) |
491
+| System | General system alerts (e.g. CPU, network, etc.) |
492
| Virtual Machine | Virtual Machine software |
493
| Web Proxy | Web proxy software (e.g. squid) |
364
-| Web Server | Web server software (e.g. Apache, ngnix, etc.) |
365
-| Windows | Alerts for monitor of windows services |
494
+| Web Server | Web server software (e.g. Apache, nginx, etc.) |
495
+| Windows | Alerts for monitoring Windows services |
496
367
-</details>
497
+:::note
498
499
If an alert configuration is missing the `type` line, its value will default to `Unknown`.
500
371
-#### Alert line `component`
372
-
373
-Component can be used to narrow down what the previous `type` value specifies for each alert or template. Continuing from the previous example, `component` might include `MySQL`, `CockroachDB`, `MongoDB`, all under the same `Database` type. Example:
374
-
375
-```yaml
376
-component: MySQL
377
-```
501
+:::
502
379
-As with the `class` and `type` line, if `component` is missing from the configuration, its value will default to `Unknown`.
503
+#### Alert Line `component`
504
381
-#### Alert line `lookup`
505
+**Purpose:** You can use `component` to narrow down what the previous `type` value specifies for each alert or template.
506
383
-This line makes a database lookup to find a value. This result of this lookup is available as `$this`.
507
+**Example:** Continuing from the previous example, `component` might include `MySQL`, `CockroachDB`, `MongoDB`, all under the same `Database` type.
508
385
-The format is:
386
-
387
-```yaml
388
-lookup: METHOD(GROUPING OPTIONS) AFTER [at BEFORE] [every DURATION] [OPTIONS] [of DIMENSIONS]
509
+```text
510
+component: MySQL
511
```
512
391
-The full [database query API](/src/web/api/queries/README.md) is supported. In short:
392
-
393
-- `METHOD` is one of the available [grouping methods](/src/web/api/queries/README.md#grouping-methods) such as `average`, `min`, `max` etc.
394
- This is required.
513
+:::note
514
396
- - `GROUPING OPTIONS` are optional and can have the form `CONDITION VALUE`, where `CONDITION` is `!=`, `=`, `<=`, `<`, `>`, `>=` and `VALUE` is a number. The `CONDITION` and `VALUE` are required for `countif`, while `VALUE` is used by `percentile`, `trimmed_mean` and `trimmed_median`.
515
+As with the `class` and `type` lines, if `component` is missing from the configuration, its value will default to `Unknown`.
516
398
-- `AFTER` is a relative number of seconds, but it also accepts a single letter for changing
399
- the units, like `-1s` = 1 second in the past, `-1m` = 1 minute in the past, `-1h` = 1 hour
400
- in the past, `-1d` = 1 day in the past. You need a negative number (i.e., how far in the past
401
- to look for the value). **This is required**.
517
+:::
518
403
-- `at BEFORE` is by default 0 and is not required. Using this, you can define the end of the
404
- lookup. So data will be evaluated between `AFTER` and `BEFORE`.
519
+#### Alert Line `lookup`
520
406
-- `every DURATION` sets the updated frequency of the lookup (supports single letter units as
407
- above too).
521
+**Purpose:** This line makes a database lookup to find a value. The result of this lookup is available as `$this`.
522
409
-- `OPTIONS` is a space separated list of `percentage`, `absolute`, `min`, `max`, `average`, `sum`,
410
- `min2max`, `unaligned`, `match-ids`, `match-names`.
523
+**Full Syntax:**
524
412
- - `percentage` during time-aggregation, calculate the percentage of the selected dimensions over the total of all dimensions.
413
- - `absolute` during time-aggregation, turns all sample values positive before using them.
414
- - `min` after time-aggregation of each dimension, return the minimum of all dimensions.
415
- - `max` after time-aggregation of each dimension, return the maximum of all dimensions.
416
- - `average` after time-aggregation of each dimension, return the average of all dimensions.
417
- - `sum` after time-aggregation of each dimension, return the sum of all dimensions (this is the default).
418
- - `min2max` after time-aggregation of each dimension, return the delta between the min and the max of the dimensions.
419
- - `unaligned` prevents shifting the query window to multiples of the query duration.
420
- - `match-ids` matches the dimensions based on their IDs (the default is enabled, give `match-names` to disable).
421
- - `match-names` matches the dimension based on their names (the default is enabled, give `match-ids` to disable).
525
+```text
526
+lookup: METHOD(GROUPING OPTIONS) AFTER [at BEFORE] [every DURATION] [OPTIONS] [of DIMENSIONS]
527
+```
528
423
-- `of DIMENSIONS` is optional and has to be the last parameter. Dimensions have to be separated
424
- by `,` or `|`. The space characters found in dimensions will be kept as-is (a few dimensions
425
- have spaces in their names). This accepts Netdata simple patterns _(with `words` separated by
426
- `,` or `|` instead of spaces)_ and the `match-ids` and `match-names` options affect the searches
427
- for dimensions.
529
+**Required Parameters:**
530
+
531
+| Parameter | Description | Example |
532
+|-----------|--------------------------------------------------------------------|-------------------------|
533
+| `METHOD` | [Grouping method](/src/web/api/queries/README.md#grouping-methods) | `average`, `min`, `max` |
534
+| `AFTER` | How far back to look (negative number) | `-1m`, `-1h`, `-1d` |
535
+
536
+**Optional Parameters:**
537
+
538
+| Parameter | Purpose | Details |
539
+|--------------------|-----------------------------|----------------------------------------------------------------------|
540
+| `GROUPING OPTIONS` | Conditional processing | `CONDITION VALUE` where condition is `!=`, `=`, `<=`, `<`, `>`, `>=` |
541
+| `at BEFORE` | End of lookup timeframe | Default is 0 (now) |
542
+| `every DURATION` | Update frequency | Supports `s`, `m`, `h`, `d` units |
543
+| `OPTIONS` | Processing modifiers | See options table below |
544
+| `of DIMENSIONS` | Which dimensions to include | Space-separated list, supports patterns |
545
+
546
+**Processing Options:**
547
+
548
+| Option | Effect |
549
+|---------------|---------------------------------------------------------|
550
+| `percentage` | Calculate percentage of selected dimensions over total |
551
+| `absolute` | Turn all sample values positive |
552
+| `min` | Return minimum of all dimensions after time-aggregation |
553
+| `max` | Return maximum of all dimensions after time-aggregation |
554
+| `average` | Return average of all dimensions after time-aggregation |
555
+| `sum` | Return sum of all dimensions (default) |
556
+| `min2max` | Return delta between min and max of dimensions |
557
+| `unaligned` | Prevent shifting query window to multiples of duration |
558
+| `match-ids` | Match dimensions by IDs (default) |
559
+| `match-names` | Match dimensions by names |
560
+
561
+**Example:**
562
+
563
+```text
564
+lookup: average -10m unaligned of user,system,softirq,irq,guest
565
+```
566
429
-The result of the lookup will be available as `$this` and `$NAME` in expressions.
430
-The timestamps of the timeframe evaluated by the database lookup are available as variables
431
-`$after` and `$before` (both are unix timestamps).
567
+This looks back 10 minutes, calculates the average of the specified CPU dimensions, without aligning to time boundaries.
568
433
-#### Alert line `calc`
569
+The result of the lookup will be available as `$this` and `$NAME` in expressions. The timestamps of the timeframe evaluated by the database lookup are available as variables `$after` and `$before` (both are unix timestamps).
570
435
-A `calc` is designed to apply some calculation to the values or variables available to the entity. The result of the
436
-calculation will be made available at the `$this` variable, overwriting the value from your `lookup`, to use in warning
437
-and critical expressions.
571
+#### Alert Line `calc`
572
439
-When paired with `lookup`, `calc` will perform the calculation just after `lookup` has retrieved a value from Netdata's
440
-database.
573
+**Purpose:** You can design a `calc` to apply some calculation to the values or variables available to the entity.
574
442
-You can use `calc` without `lookup` if you are using [other available variables](#variables).
575
+**Key Points:**
576
444
-The `calc` line uses [expressions](#expressions) for its syntax.
577
+- The result becomes available as `$this` variable
578
+- Overwrites the value from your `lookup`
579
+- Can be used without `lookup` if using [other available variables](#variables-reference)
580
+- Uses [expressions](#expressions-overview) for syntax
581
446
-```yaml
582
+```text
583
calc: EXPRESSION
584
```
585
450
-#### Alert line `every`
586
+**When to Use:**
587
+
588
+- **With `lookup`:** Perform calculation after database retrieval
589
+- **Without `lookup`:** When using other available variables
590
+- **For complex logic:** Mathematical operations, conditions, transformations
591
452
-Sets the update frequency of this alert. This is the same to the `every DURATION` given
453
-in the `lookup` lines.
592
+#### Alert Line `every`
593
455
-Format:
594
+**Purpose:** Sets the update frequency of this alert.
595
457
-```yaml
596
+```text
597
every: DURATION
598
```
599
461
-`DURATION` accepts `s` for seconds, `m` is minutes, `h` for hours, `d` for days.
600
+**Supported Units:**
601
463
-#### Alert lines `green` and `red`
602
+- `s` for seconds
603
+- `m` for minutes
604
+- `h` for hours
605
+- `d` for days
606
465
-Set the green and red thresholds of a chart. Both are available as `$green` and `$red` in expressions. If multiple
466
-alerts define different thresholds, the ones defined by the first alert will be used. Eventually it will be visualized
467
-on the dashboard, so only one set of them is allowed If you need multiple sets of them in different alerts, use
468
-absolute numbers instead of `$red` and `$green`.
607
+**Example:** `every: 30s` checks the alert every 30 seconds.
608
470
-Format:
609
+#### Alert Lines `green` and `red`
610
472
-```yaml
611
+**Purpose:** Set the green and red thresholds of a chart for visualization.
612
+
613
+```text
614
green: NUMBER
615
red: NUMBER
616
```
617
477
-#### Alert lines `warn` and `crit`
618
+**Important Notes:**
619
+
620
+- Both values are available as `$green` and `$red` in expressions
621
+- If multiple alerts define different thresholds, the first alert's values are used
622
+- For multiple threshold sets, use absolute numbers instead of variables
623
479
-Define the expression that triggers either a warning or critical alert. These are optional and should evaluate to
480
-either true or false (or zero/non-zero).
624
+#### Alert Lines `warn` and `crit`
625
482
-The format uses Netdata's [expression syntax](#expressions).
626
+**Purpose:** Define the expressions that trigger warning or critical alerts.
627
484
-```yaml
628
+```text
629
warn: EXPRESSION
630
crit: EXPRESSION
631
```
632
489
-#### Alert line `to`
633
+**Key Points:**
634
+
635
+- Optional (but you need at least one)
636
+- Should evaluate to true/false (or zero/non-zero)
637
+- Uses Netdata's [expression syntax](#expressions-overview)
638
+- Can reference variables like `$this`, `$green`, `$red`
639
+
640
+**Examples:**
641
491
-This will be the first script parameter that will be executed when the alert changes its status. Its meaning is left up to
492
-the `exec` script.
642
+```text
643
+warn: $this > 80
644
+crit: $this > 95
645
+```
646
494
-The default `exec` script, `alarm-notify.sh`, uses this field as a space separated list of roles, which are then
495
-consulted to find the exact recipients per notification method.
647
+#### Alert Line `to`
648
497
-Format:
649
+**Purpose:** Specifies who receives notifications when the alert changes status.
650
499
-```yaml
651
+```text
652
to: ROLE1 ROLE2 ROLE3 ...
653
```
654
503
-#### Alert line `exec`
655
+**How It Works:**
656
+
657
+- First parameter passed to the `exec` script
658
+- Default script (`alarm-notify.sh`) treats this as a space-separated list of roles
659
+- Roles are consulted to find exact recipients per notification method
660
505
-Script to be executed when the alert status changes.
661
+#### Alert Line `exec`
662
507
-Format:
663
+**Purpose:** Script to execute when the alert status changes.
664
509
-```yaml
665
+```text
666
exec: SCRIPT
667
```
668
513
-The default `SCRIPT` is Netdata's `alarm-notify.sh`, which supports all the notifications methods Netdata supports,
514
-including custom hooks.
669
+**Default Behavior:**
670
516
-#### Alert line `delay`
671
+- Default script is Netdata's `alarm-notify.sh`
672
+- Supports all notification methods Netdata supports
673
+- Includes custom hooks
674
518
-This is used to provide optional hysteresis settings for the notifications, to defend against notification floods. These
519
-settings don’t affect the actual alert - only the time the `exec` script is executed.
675
+#### Alert Line `delay`
676
521
-Format:
522
-
523
-```yaml
524
-delay: [[[up U] [down D] multiplier M] max X]
525
-```
677
+**Purpose:** Provide optional hysteresis settings to prevent notification floods.
678
527
-- `up U` defines the delay to be applied to a notification for an alert that raised its status
528
- (i.e., CLEAR to WARNING, CLEAR to CRITICAL, WARNING to CRITICAL). For example, `up 10s`, the
529
- notification for this event will be sent 10 seconds after the actual event. This is used in
530
- the hope the alert will get back to its previous state within the duration given. The default `U`
531
- is zero.
679
+:::important
680
533
-- `down D` defines the delay to be applied to a notification for an alert that moves to lower
534
- state (i.e., CRITICAL to WARNING, CRITICAL to CLEAR, WARNING to CLEAR). For example, `down 1m`
535
- will delay the notification by 1 minute. This is used to prevent notifications for flapping
536
- alerts. The default `D` is zero.
681
+These settings don't affect the actual alert - only when the `exec` script is executed.
682
538
-- `multiplier M` multiplies `U` and `D` when an alert changes state, while a notification is
539
- delayed. The default multiplier is `1.0`.
683
+:::
684
541
-- `max X` defines the maximum absolute notification delay an alert may get. The default `X`
542
- is `max(U * M, D * M)` (i.e., the max duration of `U` or `D` multiplied once with `M`).
685
+**Full Syntax:**
686
544
- Example:
687
+```text
688
+delay: [[[up U] [down D] multiplier M] max X]
689
+```
690
546
- `delay: up 10s down 15m multiplier 2 max 1h`
691
+**Parameters:**
692
548
- The time is `00:00:00` and the status of the alert is CLEAR.
693
+| Parameter | Purpose | Default |
694
+|----------------|--------------------------------------------------------------|---------------|
695
+| `up U` | Delay for status increases (CLEAR→WARNING, WARNING→CRITICAL) | 0 |
696
+| `down D` | Delay for status decreases (CRITICAL→WARNING, WARNING→CLEAR) | 0 |
697
+| `multiplier M` | Multiplies U and D when alert changes state during delay | 1.0 |
698
+| `max X` | Maximum absolute notification delay | max(U×M, D×M) |
699
550
- | time of event | new status | delay | notification will be sent | why |
551
- |---------------|------------|---------------------|---------------------------|-------------------------------------------------------------------------------|
552
- | 00:00:01 | WARNING | `up 10s` | 00:00:11 | first state switch |
553
- | 00:00:05 | CLEAR | `down 15m x2` | 00:30:05 | the alert changes state while a notification is delayed, so it was multiplied |
554
- | 00:00:06 | WARNING | `up 10s x2 x2` | 00:00:26 | multiplied twice |
555
- | 00:00:07 | CLEAR | `down 15m x2 x2 x2` | 00:45:07 | multiplied 3 times. |
700
+**Example with Timeline:**
701
557
- So:
702
+```text
703
+delay: up 10s down 15m multiplier 2 max 1h
704
+```
705
559
- - `U` and `D` are multiplied by `M` every time the alert changes state (any state, not just
560
- their matching one) and a delay is in place.
561
- - All are reset to their defaults when the alert switches state without a delay in place.
706
+Starting at `00:00:00` with CLEAR status:
707
563
-#### Alert line `repeat`
708
+| Time | New Status | Delay Applied | Notification At | Reason |
709
+|----------|------------|----------------|-----------------|-------------------------------------------|
710
+| 00:00:01 | WARNING | `up 10s` | 00:00:11 | First state switch |
711
+| 00:00:05 | CLEAR | `down 15m x2` | 00:30:05 | Alert changed during delay, so multiplied |
712
+| 00:00:06 | WARNING | `up 10s x2 x2` | 00:00:26 | Multiplied twice |
713
565
-Defines the interval between repeating notifications for the alerts in CRITICAL or WARNING mode. This will override the
566
-default interval settings inherited from health settings in `netdata.conf`. The default settings for repeating
567
-notifications are `default repeat warning = DURATION` and `default repeat critical = DURATION` which can be found in
568
-health stock configuration, when one of these intervals is bigger than 0, Netdata will activate the repeat notification
569
-for `CRITICAL`, `CLEAR` and `WARNING` messages.
714
+#### Alert Line `repeat`
715
571
-Format:
716
+**Purpose:** Defines the interval between repeating notifications for alerts in CRITICAL or WARNING mode.
717
573
-```yaml
718
+```text
719
repeat: [off] [warning DURATION] [critical DURATION]
720
```
721
577
-- `off`: Turns off the repeating feature for the current alert. This is effective when the default repeat settings have
578
- been enabled in health configuration.
579
-- `warning DURATION`: Defines the interval when the alert is in WARNING state. Use `0s` to turn off the repeating
580
- notification for WARNING mode.
581
-- `critical DURATION`: Defines the interval when the alert is in CRITICAL state. Use `0s` to turn off the repeating
582
- notification for CRITICAL mode.
722
+**Options:**
723
584
-#### Alert line `options`
724
+| Option | Effect |
725
+|---------------------|----------------------------------------------------------|
726
+| `off` | Turns off repeating for this alert |
727
+| `warning DURATION` | Repeat interval for WARNING state (use `0s` to disable) |
728
+| `critical DURATION` | Repeat interval for CRITICAL state (use `0s` to disable) |
729
586
-The only possible value for the `options` line is
730
+**Why Use This:** Overrides default repeat settings from `netdata.conf` health configuration.
731
588
-```yaml
732
+#### Alert Line `options`
733
+
734
+**Purpose:** Special alert behavior options.
735
+
736
+```text
737
options: no-clear-notification
738
```
739
592
-For some alerts, we need to compare two time frames to detect anomalies. For example, `health.d/httpcheck.conf` has an
593
-alert template called `web_service_slow` that compares the average http call response time over the last 3 minutes,
594
-compared to the average over the last hour. It triggers a warning alert when the average of the last 3 minutes is twice
595
-the average of the last hour. In such cases, it is easy to trigger the alert, but challenging to tell when the alert is
596
-cleared. As time passes, the newest window moves into the older, so the average response time of the last hour will keep
597
-increasing. Eventually, the comparison will find the averages in the two time-frames close enough to clear the alert.
598
-However, the issue was not resolved; it's just a matter of the newer data "polluting" the old. For such alerts, it's a
599
-good idea to tell Netdata to not clear the notification, by using the `no-clear-notification` option.
740
+**Available Options:**
741
601
-#### Alert line `host labels`
742
+- `no-clear-notification` - Prevents clearing the alert notification
743
603
-Defines the list of labels present on a host. See our [host labels guide](/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts.md) for
604
-an explanation of host labels and how to implement them.
744
+**When to Use `no-clear-notification`:**
745
606
-For example, let's suppose that `netdata.conf` is configured with the following labels:
746
+- Alerts comparing two time frames (e.g., last 3 minutes vs last hour)
747
+- When newer data might "pollute" the baseline comparison
748
+- When clearing conditions are unreliable due to data characteristics
749
608
-```yaml
609
-[host labels]
610
- installed = 20191211
611
- room = server
612
-```
750
+**Example Use Case:** HTTP response time alert comparing recent average to historical average - as time passes, the recent slow responses become part of the historical data, making the alert appear "cleared" even though the underlying issue wasn't resolved.
751
+
752
+#### Alert Line `host labels`
753
+
754
+**Purpose:** Restricts alerts to hosts with matching labels.
755
614
-And more labels in `netdata.conf` for workstations:
756
+**Prerequisites:** See our [host labels guide](/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts.md) for setup instructions.
757
616
-```yaml
758
+**Example Configuration:**
759
+
760
+```text
761
[host labels]
618
- installed = 201705
619
- room = workstation
762
+ installed = 20191211
763
+ room = server
764
```
765
622
-By defining labels inside of `netdata.conf`, you can now apply labels to alerts. For example, you can add the following
623
-line to any alerts you'd like to apply to hosts that have the label `room = server`.
766
+**Usage in Alerts:**
767
625
-```yaml
768
+```text
769
host labels: room = server
770
```
771
629
-The `host labels` is a space-separated list that accepts simple patterns. For example, you can create an alert
630
-that will be applied to all hosts installed in the last decade with the following line:
772
+**Pattern Support:**
773
632
-```yaml
633
-host labels: installed = 201*
774
+```text
775
+host labels: installed = 201* # Matches all hosts installed in 2010s
776
```
777
636
-See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for more examples.
778
+**How It Works:**
779
638
-#### Alert line `chart labels`
780
+- Space-separated list
781
+- Accepts [simple patterns](/src/libnetdata/simple_pattern/README.md)
782
+- Alert only loads on matching hosts
783
640
-Similar to host labels, the `chart labels` key can be used to filter if an alert loads or not for a specific chart, based on
641
-whether these chart labels match or not.
784
+#### Alert Line `chart labels`
785
643
-The list of chart labels present on each chart can be obtained from <http://localhost:19999/api/v1/charts?all>
786
+**Purpose:** Filters alerts based on chart labels.
787
645
-For example, each `disk_space` chart defines a chart label called `mount_point` with each instance of this chart having
646
-a value there of which mount point it monitors.
788
+**How to Find Chart Labels:** Check `http://localhost:19999/api/v1/charts?all`
789
648
-If you have an e.g., external disk mounted on `/mnt/disk1` and you don't wish any related disk space alerts running for
649
-it (but you do for all other mount points), you can add the following to the alert's configuration:
790
+**Example Use Case:**
791
+Each `disk_space` chart has a `mount_point` label. To exclude external disk alerts:
792
651
-```yaml
793
+```text
794
chart labels: mount_point=!/mnt/disk1 *
795
```
796
655
-The `chart labels` is a space-separated list that accepts simple patterns. If you use multiple different chart labels,
656
-then the result is an AND between them. I.e. the following:
797
+**Multiple Label Logic:**
798
658
-```yaml
799
+```text
800
chart labels: mount_point=/mnt/disk1 device=sda
801
```
802
662
-Will create the alert if the `mount_point` is `/mnt/disk1` and the `device` is `sda`. Furthermore, if a chart label name
663
-is specified that doesn’t exist in the chart, the chart won't be matched.
803
+This requires BOTH conditions to be true (AND logic).
804
+
805
+**Important Notes:**
806
665
-See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for more examples.
807
+- Space-separated list with [simple patterns](/src/libnetdata/simple_pattern/README.md) support
808
+- If a specified label doesn't exist on the chart, the chart won't match
809
+- Multiple labels use AND logic
810
667
-#### Alert line `summary`
811
+#### Alert Line `summary`
812
669
-The summary field contains a brief title of the alert. It is used as the subject for the notifications, and in
670
-the dashboard list of alerts. An example for the `ram_available` alert is:
813
+**Purpose:** Brief title of the alert used in notifications and dashboard.
814
672
-```yaml
815
+```text
816
summary: Available Ram
817
```
818
676
-Summary fields can contain special variables in their text that will be replaced during run-time to provide more specific
677
-alert information. Current variables supported are:
819
+**Variable Support:**
820
679
-| variable | description |
680
-|---------------------|-------------------------------------------------------------------|
681
-| ${family} | Will be replaced by the family instance for the alert (e.g. eth0) |
682
-| ${label:LABEL_NAME} | The variable will be replaced with the value of the chart label |
821
+| Variable | Replaced With |
822
+|-----------------------|------------------------------|
823
+| `${family}` | Family instance (e.g., eth0) |
824
+| `${label:LABEL_NAME}` | Chart label value |
825
684
-For example, a summary field like the following:
826
+**Example with Variables:**
827
686
-```yaml
828
+```text
829
summary: 1 minute received traffic overflow for ${label:device}
830
```
831
690
-Will be rendered on the alert acting on interface `eth0` as:
832
+Renders as: `1 minute received traffic overflow for eth0`
833
692
-```yaml
693
-summary: 1 minute received traffic overflow for eth0
694
-```
834
+:::note
835
+
836
+Variable names are case-sensitive.
837
696
-> Please note that variable names are case-sensitive.
838
+:::
839
698
-#### Alert line `info`
840
+#### Alert Line `info`
841
700
-The info field can contain a small piece of text describing the alert or template. This will be rendered in
701
-notifications and UI elements whenever the specific alert is in focus. An example for the `ram_available` alert is:
842
+**Purpose:** Detailed description of the alert for notifications and UI elements.
843
703
-```yaml
844
+```text
845
info: Percentage of estimated amount of RAM available for userspace processes, without causing swapping
846
```
847
707
-Info fields can contain special variables in their text that will be replaced during run-time to provide more specific
708
-alert information. Current variables supported are:
848
+**Variable Support:**
849
+
850
+| Variable | Replaced With |
851
+|-----------------------|------------------------------|
852
+| `${family}` | Family instance (e.g., eth0) |
853
+| `${label:LABEL_NAME}` | Chart label value |
854
710
-| variable | description |
711
-|---------------------|-------------------------------------------------------------------|
712
-| ${family} | Will be replaced by the family instance for the alert (e.g. eth0) |
713
-| ${label:LABEL_NAME} | The variable will be replaced with the value of the chart label |
855
+**Examples with Variables:**
856
715
-For example, an info field like the following:
857
+**Family Variable:**
858
717
-```yaml
859
+```text
860
info: average inbound utilization for the network interface ${family} over the last minute
861
```
862
721
-Will be rendered on the alert acting on interface `eth0` as:
722
-
723
-```yaml
724
-info: average inbound utilization for the network interface eth0 over the last minute
725
-```
863
+Renders as: `average inbound utilization for the network interface eth0 over the last minute`
864
727
-An alert acting on a chart that has a chart label named e.g. `target`, with a value of `https://netdata.cloud/`,
728
-can be enriched as follows:
865
+**Label Variable:**
866
730
-```yaml
867
+```text
868
info: average ratio of HTTP responses with unexpected status over the last 5 minutes for the site ${label:target}
869
```
870
734
-Will become:
871
+Renders as: `average ratio of HTTP responses with unexpected status over the last 5 minutes for the site https://netdata.cloud/`
872
736
-```yaml
737
-info: average ratio of HTTP responses with unexpected status over the last 5 minutes for the site https://netdata.cloud/
738
-```
873
+**Next Steps:** Continue to [Expressions and Variables](#expressions-and-variables) to understand the calculation syntax, or jump to [Alert Examples](#alert-examples) for practical implementations.
874
+
875
+## Expressions and Variables
876
740
-> Please note that variable names are case-sensitive.
877
+:::tip
878
742
-## Expressions
879
+**What You'll Learn**
880
744
-Netdata has an internal infix expression parser under `libnetdata/eval`. This parses expressions and creates an internal
745
-structure that allows fast execution of them.
881
+How to write calculations and use variables in your alert definitions. Essential for creating custom logic and accessing chart data.
882
747
-These operators are supported `+`, `-`, `*`, `/`, `<`, `==`, `<=`, `<>`, `!=`, `>`, `>=`, `&&`, `||`, `!`, `AND`, `OR`, `NOT`.
748
-Boolean operators result in either `1` (true) or `0` (false).
883
+:::
884
750
-The conditional evaluation operator `?` is supported too. Using this operator, IF-THEN-ELSE conditional statements can be
751
-specified. The format is: `(condition) ? (true expression) : (false expression)`. So, Netdata will first evaluate the
752
-`condition` and based on the result will either evaluate `true expression` or `false expression`.
885
+### Expressions Overview
886
754
-Example: `($this > 0) ? ($avail * 2) : ($used / 2)`.
887
+**Why This Matters:** Netdata has an internal infix expression parser that allows you to create complex alert logic using mathematical operations, comparisons, and conditional statements.
888
756
-Nested such expressions are also supported (i.e. `true expression` and `false expression` can contain conditional
757
-evaluations).
889
+**Supported Operators:**
890
759
-Expressions also support the `abs()` function.
891
+| Type | Operators | Result |
892
+|------------|----------------------------------------|---------------------------|
893
+| Arithmetic | `+`, `-`, `*`, `/` | Numeric values |
894
+| Comparison | `<`, `==`, `<=`, `<>`, `!=`, `>`, `>=` | `1` (true) or `0` (false) |
895
+| Logical | `&&`, ` | |`, `!`, `AND`, `OR`, `NOT` | `1` (true) or `0` (false) |
896
761
-Expressions can have variables. Variables start with `$`. Check below for more information.
897
+**Special Functions:**
898
763
-There are two special values you can use:
899
+- `abs()` - Absolute value
900
+- `(condition) ? (true_expr) : (false_expr)` - Conditional operator
901
765
-- `nan`, for example `$this != nan` will check if the variable `this` is available. A variable can be `nan` if the
766
- database lookup failed. All calculations (i.e., addition, multiplication, etc.) with a `nan` result in a `nan`.
902
+**Special Values:**
903
768
-- `inf`, for example `$this != inf` will check if `this` is not infinite. A value or variable can be set to infinite
769
- if divided by zero. All calculations (i.e., addition, multiplication, etc.) with a `inf` result in a `inf`.
904
+| Value | Purpose | Example Use |
905
+|-------|---------------------------------------|----------------|
906
+| `nan` | Not a number (database lookup failed) | `$this != nan` |
907
+| `inf` | Infinite (division by zero) | `$this != inf` |
908
771
-### Special use of the conditional operator
909
+### Conditional Operator for Hysteresis
910
773
-The conditional operator (`? :`) can create "sticky" alert thresholds that prevent alert spam when values fluctuate around a threshold. This is called [hysteresis](https://en.wikipedia.org/wiki/Hysteresis)—where the threshold to trigger an alert is different from the threshold to clear it.
911
+**Why This Matters:** The conditional operator (`? :`) can create "sticky" alert thresholds that prevent alert spam when values fluctuate around a threshold. This is called [hysteresis](https://en.wikipedia.org/wiki/Hysteresis).
912
775
-**Example: CPU Usage Alert**
913
+**Basic Pattern:**
914
+
915
+```text
916
+warn: $this > (($status >= $WARNING) ? (lower_threshold) : (higher_threshold))
917
+```
918
777
-```yaml
919
+**Real Example - CPU Usage Alert:**
920
+
921
+```text
922
warn: $this > (($status >= $WARNING) ? (75) : (85))
923
crit: $this > (($status == $CRITICAL) ? (85) : (95))
924
```
925
782
-| Alert State | Triggers At | Clears At | Explanation |
783
-|-------------|-------------|-----------|---------------------------------------------------------------------------|
784
-| Warning | 85% CPU | 75% CPU | Creates a 10% buffer - CPU must drop below 75% to clear a warning state |
785
-| Critical | 95% CPU | 85% CPU | Creates a 10% buffer - CPU must drop below 85% to return to warning state |
926
+**How This Works:**
927
928
+| Alert State | Triggers At | Clears At | Explanation |
929
+|-------------|-------------|-----------|-------------------------------------------------------------------|
930
+| Warning | 85% CPU | 75% CPU | Creates 10% buffer - CPU must drop below 75% to clear warning |
931
+| Critical | 95% CPU | 85% CPU | Creates 10% buffer - CPU must drop below 85% to return to warning |
932
788
-If CPU usage fluctuates between 80–90%, you'll receive just one initial warning, rather than constant notifications.
933
+**Benefits:**
934
790
-This approach gives you the best of both worlds: quick alerting when issues arise, but protection against notification spam when values hover near a threshold.
935
+- **Quick alerting** when issues arise
936
+- **Protection against spam** when values hover near thresholds
937
+- **Single initial notification** instead of constant alerts during fluctuation
938
792
-## Variables
939
+**Example Scenario:** If CPU usage fluctuates between 80–90%, you'll receive just one initial warning, rather than constant notifications.
940
794
-You can find all the variables that can be used for a given chart, using
795
-`http://NODE:19999/api/v1/alarm_variables?chart=CHART_NAME`, replacing `NODE` with the IP address or hostname for your
796
-Agent dashboard. For example, [variables for the `system.cpu` chart of the
797
-registry](https://registry.my-netdata.io/api/v1/alarm_variables?chart=system.cpu).
941
+### Variables Reference
942
799
-<!-- > If you don't know how to find the CHART_NAME, you can read about it [here](/src/web/README.md#charts). -->
943
+**How to Find Available Variables:**
944
+You can find all variables for a given chart using: `http://NODE:19999/api/v1/alarm_variables?chart=CHART_NAME`
945
801
-Netdata supports three internal indexes for variables that will be used in health monitoring.
946
+**Example:** [Variables for the `system.cpu` chart](https://registry.my-netdata.io/api/v1/alarm_variables?chart=system.cpu)
947
803
-<details><summary>The variables below can be used in both chart alerts and context templates.</summary>
948
+:::note
949
805
-Although the `alarm_variables` link shows you variables for a particular chart, the same variables can also be used in
806
-templates for charts belonging to a given context. The reason is that all charts of a given
807
-context are essentially identical, with the only difference being the family that identifies a particular hardware or software instance.
950
+**Chart vs Template Variables**
951
809
-</details>
952
+Although the `alarm_variables` link shows variables for a particular chart, the same variables can also be used in templates for charts belonging to a given context. All charts of a given context are essentially identical, with the only difference being the family that identifies a particular hardware or software instance.
953
811
-- **chart local variables**. All the dimensions of the chart are exposed as local variables. The value of `$this` for
812
- the other configured alerts of the chart also appears, under the name of each configured alert.
954
+:::
955
814
- Charts also define a few special variables:
956
+### Variable Categories
957
816
- - `$last_collected_t` is the unix timestamp of the last data collection
817
- - `$collected_total_raw` is the sum of all the dimensions (their last collected values)
818
- - `$update_every` is the update frequency of the chart
819
- - `$green` and `$red` the threshold defined in alerts (these are per chart - the charts inherits them from the first alert that defined them)
958
+#### Chart Local Variables
959
821
- > Chart dimensions define their last calculated (i.e. interpolated) value, exactly as
822
- shown on the charts, but also a variable with their name and suffix `_raw` that resolves
823
- to the last collected value - as collected and another with suffix `_last_collected_t`
824
- that resolves to unix timestamp the dimension was last collected (there may be dimensions
825
- that fail to be collected while others continue normally).
960
+**What's Available:**
961
827
-- **host variables**. All the dimensions of all charts, including all alerts, in fullname.
828
- Fullname is `CHART.VARIABLE`, where `CHART` is either the chart id or the chart name (both
829
- are supported).
962
+- All chart dimensions as variables (e.g., `$user`, `$system` for CPU chart)
963
+- Values from other configured alerts on the same chart
964
+- Special chart variables (see table below)
965
831
-- **special variables** are:
966
+**Special Chart Variables:**
967
833
- - `$this`, which is resolved to the value of the current alert.
968
+| Variable | Contains |
969
+|------------------------|-----------------------------------------------|
970
+| `$last_collected_t` | Unix timestamp of last data collection |
971
+| `$collected_total_raw` | Sum of all dimensions (last collected values) |
972
+| `$update_every` | Update frequency of the chart |
973
+| `$green`, `$red` | Thresholds defined in alerts |
974
835
- - `$status`, which is resolved to the current status of the alert (the current = the last
836
- status, i.e., before the current database lookup and the evaluation of the `calc` line).
837
- This values can be compared with `$REMOVED`, `$UNINITIALIZED`, `$UNDEFINED`, `$CLEAR`,
838
- `$WARNING`, `$CRITICAL`. These values are incremental, e.g. `$status > $CLEAR` works as
839
- expected.
975
+**Dimension Value Types:**
976
841
- - `$now`, which is resolved to current unix timestamp.
977
+- **Default:** Last calculated (interpolated) value as shown on charts
978
+- **Raw suffix:** `$dimension_raw` - Last collected value
979
+- **Timestamp suffix:** `$dimension_last_collected_t` - Unix timestamp when dimension was last collected
980
843
-## Alert statuses
981
+#### Host Variables
982
845
-Alerts can have the following statuses:
983
+**What's Available:** All dimensions of all charts, including all alerts, in fullname format.
984
847
-- `REMOVED` - the alert has been deleted (this happens when a SIGUSR2 is sent to Netdata
848
- to reload health configuration)
985
+**Format:** `CHART.VARIABLE`
986
850
-- `UNINITIALIZED` - the alert is not initialized yet
987
+- `CHART` can be either chart ID or chart name
988
+- Both formats are supported
989
852
-- `UNDEFINED` - the alert failed to be calculated (i.e., the database lookup failed,
853
- a division by zero occurred, etc.)
990
+**Examples:**
991
855
-- `CLEAR` - the alert is not armed / raised (i.e., is OK)
992
+- `$system.cpu.user` - User CPU from system.cpu chart
993
+- `$disk.sda.reads` - Read operations from sda disk chart
994
857
-- `WARNING` - the warning expression resulted in true or non-zero
995
+#### Special Variables
996
859
-- `CRITICAL` - the critical expression resulted in true or non-zero
997
+| Variable | Contains | Usage |
998
+|-----------|------------------------|----------------------------------------|
999
+| `$this` | Current alert value | Result of `calc` line or current alert |
1000
+| `$status` | Current alert status | Compare with status constants |
1001
+| `$now` | Current unix timestamp | Time-based calculations |
1002
861
-The external script will be called for all status changes.
1003
+**Alert Status Constants:**
1004
863
-## Example alerts
1005
+| Constant | Numeric Value | Usage |
1006
+|------------------|---------------|--------------------------------|
1007
+| `$REMOVED` | -2 | Alert deleted (SIGUSR2 reload) |
1008
+| `$UNINITIALIZED` | -1 | Alert not initialized |
1009
+| `$UNDEFINED` | 0 | Calculation failed |
1010
+| `$CLEAR` | 1 | Alert OK/not triggered |
1011
+| `$WARNING` | 2 | Warning condition met |
1012
+| `$CRITICAL` | 3 | Critical condition met |
1013
865
-Check the `health/health.d/` directory for all alerts shipped with Netdata.
1014
+**Status Comparison Examples:**
1015
867
-Here are a few examples:
1016
+```text
1017
+# Check if alert is in warning or higher
1018
+warn: $status >= $WARNING
1019
869
-### Example 1 - check server alive
1020
+# Check if alert is specifically critical
1021
+crit: $status == $CRITICAL
1022
+```
1023
871
-A simple check if an apache server is alive:
1024
+:::note
1025
873
-```yaml
874
-template: apache_last_collected_secs
875
- on: apache.requests
876
- calc: $now - $last_collected_t
877
- every: 10s
878
- warn: $this > ( 5 * $update_every)
879
- crit: $this > (10 * $update_every)
880
-```
1026
+Status values increase with severity, so `$status > $CLEAR` will match both WARNING and CRITICAL states.
1027
882
-The above checks that Netdata is able to collect data from apache. In detail:
1028
+:::
1029
884
-```yaml
885
-template: apache_last_collected_secs
886
-```
1030
+### Alert Status Lifecycle
1031
888
-The above defines a **template** named `apache_last_collected_secs`.
889
-The name is important since `$apache_last_collected_secs` resolves to the `calc` line.
890
-So, try to give something descriptive.
1032
+**Status Flow:** `UNINITIALIZED` → `UNDEFINED`/`CLEAR` → `WARNING` → `CRITICAL`
1033
892
-```yaml
893
- on: apache.requests
894
-```
1034
+**When Status Changes:**
1035
896
-The above applies the **template** to all charts that have `context = apache.requests`
897
-(i.e., all your apache servers).
1036
+- **`REMOVED`** - Alert deleted during configuration reload
1037
+- **`UNINITIALIZED`** - Alert created but not yet calculated
1038
+- **`UNDEFINED`** - Database lookup failed, division by zero, etc.
1039
+- **`CLEAR`** - Alert conditions aren’t met (normal state)
1040
+- **`WARNING`** - Warning expression returned true/non-zero
1041
+- **`CRITICAL`** - Critical expression returned true/non-zero
1042
899
-```yaml
900
- calc: $now - $last_collected_t
901
-```
1043
+**Script Execution:** The external script (`exec` line) is called for ALL status changes.
1044
903
-- `$now` is a standard variable that resolves to the current timestamp.
1045
+**Next Steps:** Ready to see these concepts in action? Continue to [Alert Examples](#alert-examples) for practical implementations.
1046
905
-- `$last_collected_t` is the last data collection timestamp of the chart.
906
- So this calculation gives the number of seconds passed since the last data collection.
1047
+## Alert Examples
1048
908
-```yaml
909
- every: 10s
910
-```
1049
+:::tip
1050
+
1051
+**What You'll Learn**
1052
+
1053
+Real-world alert configurations that demonstrate different monitoring scenarios. Use these as templates for your own alerts.
1054
+
1055
+:::
1056
+
1057
+<details>
1058
+<summary><strong>Example 1: Server Alive Check</strong></summary><br/>
1059
+
1060
+**Scenario:** Monitor if the Apache server is collecting data properly.
1061
912
-The alert will be evaluated every 10 seconds.
1062
+**Why This Matters:** Detect when data collection stops, indicating potential server or network issues.
1063
914
-```yaml
1064
+```text
1065
+template: apache_last_collected_secs
1066
+ on: apache.requests
1067
+ calc: $now - $last_collected_t
1068
+ every: 10s
1069
warn: $this > ( 5 * $update_every)
1070
crit: $this > (10 * $update_every)
1071
```
1072
919
-If these results in non-zero or true, they trigger the alert.
1073
+**How It Works:**
1074
921
-- `$this` refers to the value of this alert (e.g., the result of the `calc` line).
922
- We could also use `$apache_last_collected_secs`.
1075
+| Component | Purpose | This Example |
1076
+|------------|---------------------------------|------------------------------|
1077
+| `template` | Applies to all Apache servers | `apache_last_collected_secs` |
1078
+| `on` | Chart context to monitor | `apache.requests` |
1079
+| `calc` | Time since last data collection | `$now - $last_collected_t` |
1080
+| `every` | Check frequency | Every 10 seconds |
1081
+| `warn` | Warning threshold | 5 missed collection cycles |
1082
+| `crit` | Critical threshold | 10 missed collection cycles |
1083
924
-`$update_every` is the update frequency of the chart, in seconds.
1084
+**Variables Used:**
1085
926
-So, the warning condition checks if we haven’t collected data from apache for five
927
-iterations and the critical condition checks for 10 iterations.
1086
+- `$now` - Current timestamp
1087
+- `$last_collected_t` - Last data collection timestamp
1088
+- `$update_every` - Chart update frequency
1089
+- `$this` - Result of calculation (seconds since last collection)
1090
+
1091
+<br/>
1092
+</details>
1093
+
1094
+<details>
1095
+<summary><strong>Example 2: Disk Space Monitoring</strong></summary><br/>
1096
929
-### Example 2 - disk space
1097
+**Scenario:** Alert when any disk is running low on space.
1098
931
-Check if any of the disks are critically low on disk space:
1099
+**Why This Matters:** Prevent system failures due to full disks.
1100
933
-```yaml
1101
+```text
1102
template: disk_full_percent
1103
on: disk.space
1104
calc: $used * 100 / ($avail + $used)
@@ -940,22 +1108,34 @@ template: disk_full_percent
1108
repeat: warning 120s critical 10s
1109
```
1110
943
-`$used` and `$avail` are the `used` and `avail` chart dimensions as shown on the dashboard.
1111
+**How It Works:**
1112
945
-So, the `calc` line finds the percentage of used space. `$this` resolves to this percentage.
1113
+| Component | Purpose | This Example |
1114
+|-------------|----------------------------|----------------------------------------|
1115
+| `template` | Applies to all disks | `disk_full_percent` |
1116
+| `on` | Chart context | `disk.space` |
1117
+| `calc` | Calculate usage percentage | `$used * 100 / ($avail + $used)` |
1118
+| `warn/crit` | Simple thresholds | 80% warning, 95% critical |
1119
+| `repeat` | Notification frequency | Every 2min (warning), 10sec (critical) |
1120
947
-This is a repeating alert, and if the alert becomes CRITICAL, it repeats the notifications every 10 seconds. It also
948
-repeats notifications every 2 minutes if the alert goes into WARNING mode.
1121
+**Variables Used:**
1122
950
-### Example 3 - disk fill rate
1123
+- `$used` - Used disk space dimension
1124
+- `$avail` - Available disk space dimension
1125
952
-Predict if any disk will run out of space soon.
1126
+<br/>
1127
+</details>
1128
+
1129
+<details>
1130
+<summary><strong>Example 3: Predictive Disk Full Alert</strong></summary><br/>
1131
+
1132
+**Scenario:** Predict when disks will run out of space based on the current fill rate.
1133
954
-We do this in 2 steps:
1134
+**Why This Matters:** Get warning before disk space becomes critical.
1135
956
-Calculate the disk fill rate:
1136
+**Step 1: Calculate Disk Fill Rate**
1137
958
-```yaml
1138
+```text
1139
template: disk_fill_rate
1140
on: disk.space
1141
lookup: max -1s at -30m unaligned of avail
@@ -963,17 +1143,9 @@ Calculate the disk fill rate:
1143
every: 15s
1144
```
1145
966
-In the `calc` line: `$this` is the result of the `lookup` line (i.e., the free space 30 minutes
967
-ago) and `$avail` is the current disk-free space. So the `calc` line will either have a positive
968
-number of GB/second if the disk is filling up, or a negative number of GB/second if the disk is
969
-freeing up space.
1146
+**Step 2: Predict Hours Until Full**
1147
971
-There is no `warn` or `crit` lines here. So, this template will just do the calculation and
972
-nothing more.
973
-
974
-Predict the hours after which the disk will run out of space:
975
-
976
-```yaml
1148
+```text
1149
template: disk_full_after_hours
1150
on: disk.space
1151
calc: $avail / $disk_fill_rate / 3600
@@ -982,19 +1154,30 @@ Predict the hours after which the disk will run out of space:
1154
crit: $this > 0 and $this < 24
1155
```
1156
985
-The `calc` line estimates the time in hours, we will run out of disk space. Of course, only
986
-positive values are interesting for this check, so the warning and critical conditions check
987
-for positive values and that we have enough free space for 48 and 24 hours respectively.
1157
+**How It Works:**
1158
+
1159
+| Step | Purpose | Calculation |
1160
+|------|----------------------|----------------------------------------------------|
1161
+| 1 | Calculate fill rate | `(space_30min_ago - current_space) / 1800_seconds` |
1162
+| 2 | Predict time to full | `current_available / fill_rate / 3600` |
1163
+
1164
+**Logic:**
1165
989
-Once this alert triggers, we will receive an email like this:
1166
+- Only positive predictions matter (disk filling up)
1167
+- Warning: Less than 48 hours of space remaining
1168
+- Critical: Less than 24 hours of space remaining
1169
991
-
1170
+<br/>
1171
+</details>
1172
+
1173
+<details>
1174
+<summary><strong>Example 4: Network Packet Drops</strong></summary><br/>
1175
993
-### Example 4 - dropped packets
1176
+**Scenario:** Alert on any network packet drops.
1177
995
-Check if any network interface is dropping packets:
1178
+**Why This Matters:** Packet drops indicate network issues that could affect performance.
1179
997
-```yaml
1180
+```text
1181
template: 30min_packet_drops
1182
on: net.drops
1183
lookup: sum -30m unaligned absolute
@@ -1002,19 +1185,31 @@ template: 30min_packet_drops
1185
crit: $this > 0
1186
```
1187
1005
-The `lookup` line will calculate the sum of the all dropped packets in the last 30 minutes.
1188
+**How It Works:**
1189
+
1190
+| Component | Purpose | This Example |
1191
+|------------|-----------------------------------|-------------------------------|
1192
+| `template` | Applies to all network interfaces | `30min_packet_drops` |
1193
+| `lookup` | Sum drops over 30 minutes | `sum -30m unaligned absolute` |
1194
+| `crit` | Any drops trigger critical | `$this > 0` |
1195
1007
-The `crit` line will issue a critical alert if even a single packet has been dropped.
1196
+**Key Points:**
1197
1009
-Note that the drops chart doesn’t exist if a network interface has never dropped a single packet.
1010
-When Netdata detects a dropped packet, it will add the chart, and it will automatically attach this
1011
-alert to it.
1198
+- The drops chart only exists when packets are dropped
1199
+- The alert automatically attaches when the first drop is detected
1200
+- Zero tolerance for packet loss
1201
+
1202
+<br/>
1203
+</details>
1204
+
1205
+<details>
1206
+<summary><strong>Example 5: Z-Score Based Alert</strong></summary><br/>
1207
1013
-### Example 5 - Z-Score based alert
1208
+**Scenario:** Detect CPU usage anomalies using statistical analysis.
1209
1015
-Derive a "[Z Score](https://en.wikipedia.org/wiki/Standard_score)" based alert on `user` dimension of the `system.cpu` chart:
1210
+**Why This Matters:** Identify unusual patterns that fixed thresholds might miss.
1211
1017
-```yaml
1212
+```text
1213
alarm: cpu_user_mean
1214
on: system.cpu
1215
lookup: mean -60s of user
@@ -1034,13 +1229,32 @@ lookup: mean -10s of user
1229
crit: $this < -3 or $this > 3
1230
```
1231
1037
-Since [`z = (x - mean) / stddev`](https://en.wikipedia.org/wiki/Standard_score) we create two input alerts, one for `mean` and one for `stddev` and then use them both as inputs in our final `cpu_user_zscore` alert.
1232
+**How It Works:**
1233
1039
-### Example 6 - [Anomaly rate](/src/ml/README.md#anomaly-rate) based CPU chart alert
1234
+| Alert | Purpose | Calculation |
1235
+|-------------------|-----------------------------|------------------------------------|
1236
+| `cpu_user_mean` | Calculate average CPU usage | Mean over 60 seconds |
1237
+| `cpu_user_stddev` | Calculate variability | Standard deviation over 60 seconds |
1238
+| `cpu_user_zscore` | Detect anomalies | `(current - mean) / stddev` |
1239
1041
-Warning if the 5-minute rolling [anomaly rate](/src/ml/README.md#anomaly-rate) averaged across all CPU dimensions is above 5%, critical if it goes above 20%:
1240
+**Z-Score Interpretation:**
1241
1043
-```yaml
1242
+- **±2**: Moderately unusual (warning)
1243
+- **±3**: Highly unusual (critical)
1244
+- **Negative**: Below normal
1245
+- **Positive**: Above normal
1246
+
1247
+<br/>
1248
+</details>
1249
+
1250
+<details>
1251
+<summary><strong>Example 6: Machine Learning Anomaly Detection</strong></summary><br/>
1252
+
1253
+**Scenario:** Use Netdata's built-in ML for chart-level anomaly detection.
1254
+
1255
+**Why This Matters:** Detect complex patterns across multiple metrics without manual threshold tuning.
1256
+
1257
+```text
1258
template: ml_5min_cpu_chart
1259
on: system.cpu
1260
lookup: average -5m anomaly-bit of *
@@ -1052,14 +1266,25 @@ template: ml_5min_cpu_chart
1266
info: rolling 5min anomaly rate for system.cpu chart
1267
```
1268
1055
-The `lookup` line will calculate the average anomaly rate across all `system.cpu` dimensions over the last 5 minutes. In this case,
1056
-Netdata will create one alert for the chart.
1269
+**How It Works:**
1270
+
1271
+| Component | Purpose | This Example |
1272
+|---------------|--------------------------------------------|-------------------------------------|
1273
+| `lookup` | Average anomaly rate across CPU dimensions | 5-minute rolling window |
1274
+| Hysteresis | Prevent alert flapping | Warning: 20%→5%, Critical: 100%→20% |
1275
+| `anomaly-bit` | ML-generated anomaly indicators | 0 (normal) or 1 (anomalous) |
1276
1058
-### Example 7 - [Anomaly rate](/src/ml/README.md#anomaly-rate) based node level alert
1277
+<br/>
1278
+</details>
1279
+
1280
+<details>
1281
+<summary><strong>Example 7: Node-Level ML Monitoring</strong></summary><br/>
1282
1060
-Warning if the 5-minute rolling [anomaly rate](/src/ml/README.md#anomaly-rate) averaged across all ML enabled dimensions is above 5%, critical if it goes above 20%:
1283
+**Scenario:** Monitor overall system health using ML across all metrics.
1284
1062
-```yaml
1285
+**Why This Matters:** Get a holistic view of system anomalies beyond individual charts.
1286
+
1287
+```text
1288
template: ml_5min_node
1289
on: anomaly_detection.anomaly_rate
1290
lookup: average -5m of anomaly_rate
@@ -1071,24 +1296,365 @@ template: ml_5min_node
1296
info: rolling 5min anomaly rate for all ML enabled dims
1297
```
1298
1074
-The `lookup` line will use the `anomaly_rate` dimension of the `anomaly_detection.anomaly_rate` ML chart to calculate the average [node level anomaly rate](/src/ml/README.md#anomaly-rate) over the last 5 minutes.
1299
+**Key Differences from Chart-Level:**
1300
+
1301
+- Uses `anomaly_detection.anomaly_rate` chart
1302
+- Monitors `anomaly_rate` dimension
1303
+- Covers all ML-enabled dimensions across the node
1304
+
1305
+<br/>
1306
+</details><br/>
1307
+
1308
+**Next Steps:** Having trouble with your alerts? Continue to [Troubleshooting](#troubleshooting) for debugging techniques.
1309
1310
## Troubleshooting
1311
1078
-You can compile Netdata with [debugging](/src/daemon/README.md#debugging) and then set in `netdata.conf`:
1312
+:::tip
1313
+
1314
+**What You'll Learn**
1315
+
1316
+How to debug alert issues, understand why alerts aren't working, and get detailed information about alert processing.
1317
+
1318
+:::
1319
+
1320
+### Find Chart and Context Information
1321
+
1322
+**Finding Chart Names:**
1323
+You can find chart information in two places:
1324
+
1325
+| Method | URL | Contains |
1326
+|---------------|-----------------------------------|-------------------|
1327
+| Configuration | `http://NODE:19999/netdata.conf` | All chart details |
1328
+| API | `http://NODE:19999/api/v1/charts` | JSON chart data |
1329
+
1330
+Replace `NODE` with your server's IP address or hostname.
1331
+
1332
+### Analyze Alert Expressions
1333
+
1334
+**Why This Matters:** Understand how Netdata interprets your expressions and what values are being calculated.
1335
+
1336
+**Check Alert Processing:**
1337
+Visit `http://NODE:19999/api/v1/alarms?all` to see:
1338
1080
-```yaml
1081
-[global]
1082
- debug flags = 0x0000000000800000
1339
+- Original expression as written in config
1340
+- Parsed expression with added parentheses showing evaluation flow
1341
+- Current alert status and values
1342
+- Available variables and their values
1343
+
1344
+**Expression Evaluation Flow:**
1345
+Netdata adds parentheses to show how it evaluates your expressions:
1346
+
1347
+**Your Expression:**
1348
+
1349
+```text
1350
+warn: $this > 80 and $status >= $WARNING
1351
+```
1352
+
1353
+**Netdata's Interpretation:**
1354
+
1355
+```text
1356
+warn: (($this > 80) and ($status >= $WARNING))
1357
+```
1358
+
1359
+### Troubleshooting Decision Trees
1360
+
1361
+:::note
1362
+
1363
+**How to Use These Decision Trees**
1364
+
1365
+Follow the flowcharts below to systematically diagnose and resolve alert issues. Each path leads to specific solutions with step-by-step instructions.
1366
+
1367
+:::
1368
+
1369
+#### Decision Tree: Alert Not Working
1370
+
1371
+```mermaid
1372
+flowchart TD
1373
+ A("Alert Not Working") --> B{"Alert appears<br/>in dashboard?"}
1374
+
1375
+ B -->|No| C("Configuration Issue")
1376
+ B -->|Yes| D{"Alert triggering<br/>at all?"}
1377
+
1378
+ C --> C1("Check syntax errors<br/>Reload configuration<br/>Review debug logs")
1379
+
1380
+ D -->|No| E{"Chart exists<br/>and has data?"}
1381
+ D -->|Yes| F("Threshold Issue")
1382
+
1383
+ E -->|No| E1("Verify chart name<br/>Check data collection<br/>Review collector status")
1384
+ E -->|Yes| G{"Variables<br/>resolving?"}
1385
+
1386
+ G -->|No| G1("Check variable names<br/>Verify dimensions<br/>Use alarm_variables API")
1387
+ G -->|Yes| H{"Expression<br/>logic correct?"}
1388
+
1389
+ H -->|No| H1("Review operators<br/>Check units<br/>Test with simple thresholds")
1390
+ H -->|Yes| I("Check timing<br/>and frequency")
1391
+
1392
+ F --> F1("Adjust warn/crit values<br/>Review hysteresis<br/>Check status conditions")
1393
+
1394
+ style A fill:#f44336,stroke:#d32f2f,stroke-width:3px,color:#fff
1395
+ style C fill:#ffeb3b,stroke:#fbc02d,stroke-width:2px,color:#000
1396
+ style F fill:#ffeb3b,stroke:#fbc02d,stroke-width:2px,color:#000
1397
+ style B fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1398
+ style D fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1399
+ style E fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1400
+ style G fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1401
+ style H fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1402
+ style E1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1403
+ style G1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1404
+ style H1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1405
+ style F1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1406
+ style C1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1407
+ style I fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1408
+
1409
+ linkStyle 0 stroke:#2196F3,stroke-width:3px
1410
+ linkStyle 1 stroke:#f44336,stroke-width:3px
1411
+ linkStyle 2 stroke:#2196F3,stroke-width:3px
1412
+ linkStyle 4 stroke:#f44336,stroke-width:3px
1413
+ linkStyle 5 stroke:#2196F3,stroke-width:3px
1414
+ linkStyle 7 stroke:#f44336,stroke-width:3px
1415
+ linkStyle 8 stroke:#2196F3,stroke-width:3px
1416
+ linkStyle 10 stroke:#f44336,stroke-width:3px
1417
+ linkStyle 11 stroke:#2196F3,stroke-width:3px
1418
```
1419
1085
-Then check your `/var/log/netdata/debug.log`. It will show you how it works. Important: this will generate a lot of
1086
-output in debug.log.
1420
+#### Decision Tree: Alert Always Triggering
1421
+
1422
+```mermaid
1423
+flowchart TD
1424
+ A("Alert Always Triggering") --> B{"Check current<br/>metric value"}
1425
+
1426
+ B --> C{"Value actually<br/>above threshold?"}
1427
+
1428
+ C -->|Yes| D("Legitimate Alert")
1429
+ C -->|No| E{"Expression<br/>logic error?"}
1430
+
1431
+ D --> D1("Adjust thresholds<br/>Review alert necessity<br/>Change frequency")
1432
+
1433
+ E -->|Yes| F{"Wrong operator<br/>direction?"}
1434
+ E -->|No| G{"Units<br/>mismatch?"}
1435
+
1436
+ F -->|Yes| F1("Change operators<br/>Fix direction<br/>Review logic")
1437
+ F -->|No| H{"Variable name<br/>incorrect?"}
1438
+
1439
+ G -->|Yes| G1("Check units<br/>Verify calculations<br/>Review methods")
1440
+ G -->|No| I{"Status condition<br/>issue?"}
1441
+
1442
+ H -->|Yes| H1("Verify variables<br/>Check dimensions<br/>Use API")
1443
+ H -->|No| J("Review expression<br/>Check syntax")
1444
+
1445
+ I -->|Yes| I1("Check status usage<br/>Review logic<br/>Simplify conditions")
1446
+ I -->|No| J
1447
+
1448
+ style A fill:#f44336,stroke:#d32f2f,stroke-width:3px,color:#fff
1449
+ style D fill:#ffeb3b,stroke:#fbc02d,stroke-width:2px,color:#000
1450
+ style B fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1451
+ style C fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1452
+ style E fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1453
+ style F fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1454
+ style G fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1455
+ style H fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1456
+ style I fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1457
+ style F1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1458
+ style G1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1459
+ style H1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1460
+ style I1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1461
+ style D1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1462
+ style J fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1463
+
1464
+ linkStyle 2 stroke:#2196F3,stroke-width:3px
1465
+ linkStyle 3 stroke:#f44336,stroke-width:3px
1466
+ linkStyle 5 stroke:#2196F3,stroke-width:3px
1467
+ linkStyle 6 stroke:#f44336,stroke-width:3px
1468
+ linkStyle 7 stroke:#2196F3,stroke-width:3px
1469
+ linkStyle 8 stroke:#f44336,stroke-width:3px
1470
+ linkStyle 10 stroke:#2196F3,stroke-width:3px
1471
+ linkStyle 11 stroke:#f44336,stroke-width:3px
1472
+ linkStyle 13 stroke:#2196F3,stroke-width:3px
1473
+ linkStyle 14 stroke:#f44336,stroke-width:3px
1474
+```
1475
+
1476
+#### Decision Tree: Alert Flapping
1477
+
1478
+```mermaid
1479
+flowchart TD
1480
+ A("Alert Flapping<br/>Too Many Notifications") --> B{"Values oscillating<br/>around threshold?"}
1481
+
1482
+ B -->|Yes| C("Implement Hysteresis")
1483
+ B -->|No| D{"Data source<br/>noisy/unstable?"}
1484
+
1485
+ C --> C1("Use conditional operator<br/>Set different thresholds<br/>Add hysteresis logic")
1486
+
1487
+ D -->|Yes| E("Smooth Data")
1488
+ D -->|No| F{"Check frequency<br/>too high?"}
1489
+
1490
+ E --> E1("Increase lookup time<br/>Use averages<br/>Reduce noise")
1491
+
1492
+ F -->|Yes| F1("Increase interval<br/>Match frequency<br/>Reduce checks")
1493
+ F -->|No| G{"Delay settings<br/>needed?"}
1494
+
1495
+ G -->|Yes| G1("Add delay line<br/>Set up/down delays<br/>Use multipliers")
1496
+ G -->|No| H("Review notification<br/>settings")
1497
+
1498
+ style A fill:#f44336,stroke:#d32f2f,stroke-width:3px,color:#fff
1499
+ style C fill:#ffeb3b,stroke:#fbc02d,stroke-width:2px,color:#000
1500
+ style E fill:#ffeb3b,stroke:#fbc02d,stroke-width:2px,color:#000
1501
+ style B fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1502
+ style D fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1503
+ style F fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1504
+ style G fill:#f9f9f9,stroke:#e0e0e0,stroke-width:2px,color:#000
1505
+ style C1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1506
+ style E1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1507
+ style F1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1508
+ style G1 fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1509
+ style H fill:#4caf50,stroke:#388e3c,stroke-width:2px,color:#fff
1510
+
1511
+ linkStyle 1 stroke:#2196F3,stroke-width:3px
1512
+ linkStyle 2 stroke:#f44336,stroke-width:3px
1513
+ linkStyle 4 stroke:#2196F3,stroke-width:3px
1514
+ linkStyle 5 stroke:#f44336,stroke-width:3px
1515
+ linkStyle 7 stroke:#2196F3,stroke-width:3px
1516
+ linkStyle 8 stroke:#f44336,stroke-width:3px
1517
+ linkStyle 9 stroke:#2196F3,stroke-width:3px
1518
+ linkStyle 10 stroke:#f44336,stroke-width:3px
1519
+```
1520
+
1521
+### Common Issues and Solutions
1522
+
1523
+#### Issue: Alert Not Triggering
1524
+
1525
+**Possible Causes:**
1526
+
1527
+| Problem | Check This | Solution |
1528
+|----------------------|---------------------------------|-------------------------------|
1529
+| Wrong chart name | `on:` line matches actual chart | Use chart ID from dashboard |
1530
+| Incorrect dimensions | Dimension names in `lookup` | Check available dimensions |
1531
+| Missing data | Chart has recent data | Verify data collection |
1532
+| Expression errors | Variables resolve correctly | Use `/api/v1/alarm_variables` |
1533
+
1534
+#### Issue: Alert Always Triggering
1535
+
1536
+**Possible Causes:**
1537
+
1538
+| Problem | Check This | Solution |
1539
+|---------------------------|------------------------------------------|-------------------------|
1540
+| Wrong threshold direction | `>` vs `<` in expressions | Review logic |
1541
+| Units mismatch | Comparing percentages to absolute values | Check calculation units |
1542
+| Variable name errors | `$this` vs `$chart.dimension` | Verify variable names |
1543
+
1544
+#### Issue: Alert Flapping
1545
+
1546
+**Possible Causes:**
1547
+
1548
+| Problem | Solution |
1549
+|-----------------------|--------------------------------------------------------------|
1550
+| Values near threshold | Implement [hysteresis](#conditional-operator-for-hysteresis) |
1551
+| Noisy data | Increase lookup time window |
1552
+| Too frequent checks | Increase `every:` interval |
1553
+
1554
+**Hysteresis Example:**
1555
+
1556
+```text
1557
+# Instead of simple threshold
1558
+warn: $this > 80
1559
+
1560
+# Use hysteresis
1561
+warn: $this > (($status >= $WARNING) ? (75) : (80))
1562
+```
1563
+
1564
+#### Issue: Variables Not Found
1565
+
1566
+**Debug Steps:**
1567
+
1568
+1. **Check Available Variables:**
1569
+ ```
1570
+ http://NODE:19999/api/v1/alarm_variables?chart=CHART_NAME
1571
+ ```
1572
+
1573
+2. **Verify Chart Context:**
1574
+ - For `alarm:` use chart name
1575
+ - For `template:` use chart context
1576
+
1577
+3. **Check Variable Syntax:**
1578
+ - Chart local: `$dimension_name`
1579
+ - Host variables: `$chart_name.dimension_name`
1580
+ - Special variables: `$this`, `$now`, `$status`
1581
+
1582
+### Testing Alert Changes
1583
+
1584
+**Safe Testing Process:**
1585
+
1586
+1. **Create Test File:**
1587
+ ```bash
1588
+ sudo touch health.d/test-alert.conf
1589
+ sudo ./edit-config health.d/test-alert.conf
1590
+ ```
1591
+
1592
+2. **Write Simple Alert:**
1593
+ ```text
1594
+ alarm: test_ram
1595
+ on: system.ram
1596
+ lookup: average -1m percentage of used
1597
+ every: 10s
1598
+ war: $this > 50 # Low threshold for testing
1599
+ info: Test alert - safe to ignore
1600
+ ```
1601
+
1602
+3. **Reload and Monitor:**
1603
+ ```bash
1604
+ sudo netdatacli reload-health
1605
+ # Watch dashboard for test alert appearance
1606
+ ```
1607
+
1608
+4. **Remove When Done:**
1609
+ ```bash
1610
+ sudo rm health.d/test-alert.conf
1611
+ sudo netdatacli reload-health
1612
+ ```
1613
+
1614
+### Performance Considerations
1615
+
1616
+**Alert Impact on System:**
1617
+
1618
+| Factor | Impact | Optimization |
1619
+|----------------------------|---------------------|------------------------------------|
1620
+| Check frequency (`every:`) | CPU usage | Use appropriate intervals |
1621
+| Lookup timeframe | Memory/CPU | Don't use excessively long periods |
1622
+| Number of alerts | Overall performance | Disable unused alerts |
1623
+| Complex expressions | CPU per check | Simplify where possible |
1624
+
1625
+**Recommended Frequencies:**
1626
+
1627
+| Alert Type | Suggested Frequency | Reason |
1628
+|-------------------------|---------------------|----------------------------------------|
1629
+| Critical system metrics | 10-30s | Quick response needed |
1630
+| Resource usage | 1-5m | Trends matter more than instant values |
1631
+| Predictive alerts | 15m-1h | Based on longer-term patterns |
1632
+
1633
+### Getting Help
1634
+
1635
+**Information to Provide:**
1636
+
1637
+When seeking help, include:
1638
+
1639
+1. **Alert Configuration:**
1640
+ ```text
1641
+ # Your complete alert definition
1642
+ ```
1643
+
1644
+2. **Chart Information:**
1645
+ ```
1646
+ http://your-server:19999/api/v1/alarm_variables?chart=chart_name
1647
+ ```
1648
+
1649
+3. **Current Status:**
1650
+ ```
1651
+ http://your-server:19999/api/v1/alarms?all
1652
+ ```
1653
+
1654
+**Community Resources:**
1655
1088
-You can find the context of charts by looking up the chart in either `http://NODE:19999/netdata.conf` or
1089
-`http://NODE:19999/api/v1/charts`, replacing `NODE` with the IP address or hostname for your Agent dashboard.
1656
+- [Netdata GitHub Issues](https://github.com/netdata/netdata/issues)
1657
+- [Netdata Community Forum](https://community.netdata.cloud)
1658
+- [Netdata Discord](https://discord.gg/mPZ6WZKKG2)
1659
1091
-You can find how Netdata interpreted the expressions by examining the alert at
1092
-`http://NODE:19999/api/v1/alarms?all`. For each expression, Netdata will return the expression as given in its
1093
-config file, and the same expression with additional parentheses added to indicate the evaluation flow of the
1094
-expression.
1660
+**Next Steps:** You now have comprehensive knowledge of Netdata health configuration. Start with the [Quick Start Guide](#quick-start-guide) for immediate needs or dive into [Common Tasks](#common-tasks) for specific workflows.