REFERENCE.md · cryptotaxi247/netdata-1

Code Commits Issues Pulls Sessions CI/CD

master

md 1,823 lines 79.1 KB

Rendered Raw

1	# Configure Health Alerts
2
3	:::tip
4
5	Don't want to write alert configs manually? [Alerts Automation](/docs/netdata-ai/alerts-automation/alerts-automation.md) uses AI to suggest alerts, generate the configuration, and test it against historical data — so you can validate thresholds before deployment without learning the syntax. You can also use the visual [Alerts Configuration Manager](/docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md).
6
7	:::
8
9	This page covers manual alert configuration — editing config files directly. For a comparison of all configuration methods, see [How alert configuration works in Netdata](/docs/netdata-ai/alerts-automation/alerts-automation.md#how-alert-configuration-works-in-netdata).
10
11	## Quick Start Guide
12
13	:::tip
14
15	What You'll Learn
16
17	In 5 minutes, you'll know how to edit existing alerts, create new ones, and reload your configuration without downtime.
18
19	:::
20
21	### Get Started in 3 Steps
22
23	Step 1: Find Your Config Directory
24
25	Navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md)
26
27	Step 2: Edit an Alert
28
29	```bash
30	sudo ./edit-config health.d/cpu.conf
31	```
32
33	Step 3: Apply Changes
34
35	```bash
36	sudo netdatacli reload-health
37	```
38
39	:::note
40
41	Key Concept
42
43	You can highly configure Netdata's health watchdog with support for dynamic thresholds, hysteresis, alert templates, and more. You can customize any existing alerts based on your infrastructure's topology or specific monitoring needs or create entirely new entities.
44
45	You can use health alerts with any of Netdata's [collectors](/src/collectors/README.md) (see the [supported collector list](/src/collectors/COLLECTORS.md)) to monitor your systems, containers, and applications in real time.
46
47	While you can view active alerts on both the local dashboard and Netdata Cloud, you configure all health alerts _per node_ via individual Netdata Agents. If you want to deploy a new alert across your [infrastructure](/docs/netdata-cloud/organize-your-infrastructure-invite-your-team.md), you must configure each node with the same health configuration files.
48
49	:::
50
51	Next Steps: Jump to [Common Tasks](#common-tasks) for specific workflows or continue reading for comprehensive guidance.
52
53	## Common Tasks
54
55	:::tip
56
57	What You'll Learn
58
59	Step-by-step workflows for the most frequent alert configuration tasks.
60
61	:::
62
63	### Task 1: Modify Alert Thresholds
64
65	Why This Matters: Default thresholds may not fit your specific environment or requirements.
66
67	Quick Example:
68
69	```text
70	# Change CPU warning from 85% to 75%
71	warn: $this > (($status >= $WARNING) ? (60) : (75))
72	crit: $this > (($status == $CRITICAL) ? (75) : (85))
73	```
74
75	Step-by-Step:
76
77	1. Find the alert file: `sudo ./edit-config health.d/cpu.conf`
78	2. Locate the alert (e.g., `10min_cpu_usage`)
79	3. Modify `warn` and `crit` lines
80	4. Save and reload: `sudo netdatacli reload-health`
81
82	### Task 2: Disable Unwanted Alerts
83
84	\| Method \| Use Case \| Configuration File \| How To \|
85	\| ----------------------- \| ----------------------------------- \| ------------------ \| ---------------------------------------- \|
86	\| Disable all alerts \| Testing/maintenance \| netdata.conf \| Set `enabled = no` in `[health]` section \|
87	\| Disable specific alerts \| Remove noisy alerts \| netdata.conf \| Set `enabled alarms = !alert_name *` \|
88	\| Silence notifications \| Keep monitoring, stop notifications \| Alert config file \| Change `to: silent` \|
89
90	### Task 3: Create a Simple Alert
91
92	Real-World Example: Monitor RAM usage above 80%
93
94	```text
95	alarm: ram_usage
96	on: system.ram
97	lookup: average -1m percentage of used
98	units: %
99	every: 1m
100	warn: $this > 80
101	crit: $this > 90
102	info: RAM usage monitoring
103	```
104
105	Next Steps: See [How-To Guides](#how-to-guides) for detailed explanations.
106
107	## How-To Guides
108
109	:::tip
110
111	What You'll Learn
112
113	Detailed instructions for configuring, managing, and troubleshooting health alerts.
114
115	:::
116
117	### How to Reload Health Configuration
118
119	Why This Matters: You don't need to restart your Netdata Agent when making changes, preventing gaps in monitoring.
120
121	You don't need to restart your Netdata Agent when making changes to health configuration files, such as specific health entities. Instead, you can use `netdatacli` with the `reload-health` option to prevent gaps in metrics collection.
122
123	```bash
124	sudo netdatacli reload-health
125	```
126
127	Alternative Method:
128	If `netdatacli` doesn't work on your system, you can send a `SIGUSR2` signal to the daemon, which reloads health configuration without restarting the entire process.
129
130	```bash
131	sudo killall -USR2 netdata
132	```
133
134	### How to Edit Health Configuration Files
135
136	Configuration Locations:
137
138	Configuration Locations:
139
140	\| Location \| Purpose \| Common Tasks \| How to Edit \|
141	\| --------------------------------- \| ---------------------------- \| -------------------------------------------------------------------------------------------------------- \| ---------------------------------------- \|
142	\| `netdata.conf` `[health]` section \| Global health settings \| • Disable all monitoring (`enabled = no`)<br />• Disable specific alerts<br />• Change check frequencies \| Edit directly or use `edit-config` \|
143	\| `health.d/*.conf` files \| Individual alert definitions \| • Modify thresholds<br />• Change notification recipients<br />• Silence alerts (`to: silent`) \| Use `edit-config health.d/filename.conf` \|
144
145	Navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md) and use `edit-config` to make changes to any of these files.
146
147	Edit Individual Alerts:
148
149	For example, to edit the `cpu.conf` health configuration file, run:
150
151	```bash
152	sudo ./edit-config health.d/cpu.conf
153	```
154
155	Understanding Alert Structure:
156
157	Each health configuration file contains one or more health [_entities_](#complete-configuration-reference), which always begin with `alarm:` or `template:`. Here's the first health entity in `health.d/cpu.conf`:
158
159	```text
160	template: 10min_cpu_usage
161	on: system.cpu
162	class: Utilization
163	type: System
164	component: CPU
165	lookup: average -10m unaligned of user,system,softirq,irq,guest
166	units: %
167	every: 1m
168	warn: $this > (($status >= $WARNING) ? (75) : (85))
169	crit: $this > (($status == $CRITICAL) ? (85) : (95))
170	delay: down 15m multiplier 1.5 max 1h
171	summary: CPU utilization
172	info: Average cpu utilization for the last 10 minutes (excluding iowait, nice and steal)
173	to: sysadmin
174	```
175
176	To customize this alert to trigger warning and critical alerts at lower CPU utilization levels, you can change the `warn` and `crit` lines to values of your choosing. For example:
177
178	```text
179	warn: $this > (($status >= $WARNING) ? (60) : (75))
180	crit: $this > (($status == $CRITICAL) ? (75) : (85))
181	```
182
183	Save the file and [reload Netdata's health configuration](#how-to-reload-health-configuration) to apply your changes.
184
185	### How to Disable or Silence Alerts
186
187	Why This Matters: Different situations require different approaches to managing alerts - permanent removal, temporary silencing, or selective filtering.
188
189	You can disable alerts and notifications permanently via configuration changes, or temporarily via the [health management API](/src/web/api/health/README.md).
190
191	#### Disable All Alerts
192
193	Use Case: System maintenance or testing
194
195	In the `netdata.conf` `[health]` section, set `enabled` to `no`, and restart your Agent.
196
197	#### Disable Specific Alerts
198
199	Use Case: Remove known noisy or irrelevant alerts
200
201	In the `netdata.conf` `[health]` section, use [pattern](/src/libnetdata/simple_pattern/README.md) exclusion with `enabled alarms = !oom_kill *` to load all alerts except `oom_kill`.
202
203	To exclude multiple specific alerts, list all exclusions before the wildcard and restart the Agent to apply the `netdata.conf` change:
204
205	```conf
206	[health]
207	enabled alarms = !oom_kill !disk_space_usage *
208	```
209
210	Restart your Netdata Agent after changing `netdata.conf` (`netdatacli reload-health` reloads health configuration files, but does not reload `netdata.conf`).
211
212	:::warning
213
214	Do not place `` between exclusions (`!alert1 !alert2 ` is incorrect — patterns are evaluated in order and the first match wins). When using exclusions, put all `!` patterns first, followed by a single trailing ``.
215
216	:::
217
218	You can also [edit the file where the alert is defined](#how-to-edit-health-configuration-files), comment out its definition, and [reload Netdata's health configuration](#how-to-reload-health-configuration).
219
220	#### Silence Individual Alert Notifications
221
222	Use Case: Keep monitoring active but stop notifications
223
224	You can stop receiving notifications for an individual alert by [changing](#how-to-edit-health-configuration-files) the `to:` line to `silent` in the alert's configuration file.
225
226	```text
227	to: silent
228	```
229
230	:::tip
231
232	This action requires that you [reload Netdata's health configuration](#how-to-reload-health-configuration).
233
234	:::
235
236	#### Temporary Runtime Control
237
238	Use Case: Scheduled maintenance or dynamic control
239
240	\| Scenario \| Solution \| Method \|
241	\| ---------------------------------- \| ------------------------------------------ \| ---------------------------------- \|
242	\| Disable alerts during backups \| Use health management API \| API calls without config changes \|
243	\| Suppress notifications temporarily \| Keep checks running, silence notifications \| API control of notification system \|
244
245	:::tip
246
247	You can use the [health management API](/src/web/api/health/README.md) to temporarily control alert behavior without changing configuration or restarting your Agent. The API allows you to:
248
249	- Disable all or some alerts from triggering during certain times (for instance, when running backups)
250	- Suppress notifications temporarily while keeping health checks running and alerts triggering
251
252	:::
253
254	### How to Write a New Health Entity
255
256	Why This Matters: While tuning existing alerts may work in some cases, you may need to write entirely new health entities based on how your systems, containers, and applications work.
257
258	Prerequisites: Read the [Alert Configuration Reference](#alert-configuration-reference) for a complete listing of the format, syntax, and functionality of health entities.
259
260	Step-by-Step Process:
261
262	Step 1: Create the Configuration File
263
264	Navigate to your [Netdata config directory](/docs/netdata-agent/configuration/README.md), then use `touch` to create a new file in the `health.d/` directory. Use `edit-config` to start editing the file.
265
266	As an example, let's create a `ram-usage.conf` file:
267
268	```bash
269	sudo touch health.d/ram-usage.conf
270	sudo ./edit-config health.d/ram-usage.conf
271	```
272
273	Step 2: Write Your Alert
274
275	Here's a health entity that triggers a warning alert when your node's RAM usage rises above 80%, and a critical alert above 90%:
276
277	```text
278	alarm: ram_usage
279	on: system.ram
280	lookup: average -1m percentage of used
281	units: %
282	every: 1m
283	warn: $this > 80
284	crit: $this > 90
285	info: The percentage of RAM being used by the system.
286	```
287
288	Step 3: Understand Each Component
289
290	\| Line \| Purpose \| This Example \|
291	\| ----------- \| ----------------------------------------- \| ----------------------------------------------------- \|
292	\| `alarm` \| Entity name (alphanumeric, `.`, `_` only) \| `ram_usage` \|
293	\| `on` \| Chart to monitor \| `system.ram` \|
294	\| `lookup` \| How to process metrics \| Average last 1 minute, percentage of `used` dimension \|
295	\| `units` \| Display units \| Percentages (`%`) \|
296	\| `every` \| Check frequency \| Every 1 minute \|
297	\| `warn/crit` \| Trigger conditions \| Warning > 80%, Critical > 90% \|
298	\| `info` \| Alert description \| Appears in dashboard and notifications \|
299
300	:::note
301
302	Understanding This Example
303
304	This health entity, named ram_usage, watches the system.ram chart. It looks up the last 1 minute of metrics from the used dimension and calculates the average of all those metrics in a percentage format, using % units. The entity performs this lookup every minute.
305
306	If the average RAM usage percentage over the last 1 minute is more than 80%, the entity triggers a warning alert. If the usage is more than 90%, the entity triggers a critical alert.
307
308	:::
309
310	Step 4: Activate Your Alert
311
312	When you finish writing this new health entity, [reload Netdata's health configuration](#how-to-reload-health-configuration) to see it live on your local dashboard or Netdata Cloud.
313
314	Next Steps: Explore [Alert Examples](#alert-examples) for more complex scenarios, or dive into the [Alert Configuration Reference](#alert-configuration-reference) for complete syntax details.
315
316	## Alert Configuration Reference
317
318	:::tip
319
320	What You'll Learn
321
322	Complete syntax reference for all alert configuration options. Use this section when you need specific technical details.
323
324	:::
325
326	### Entity Types Overview
327
328	\| Type \| Label \| Purpose \| Example Use Case \|
329	\| ------------- \| ----------- \| -------------------------------- \| ------------------------------ \|
330	\| Alerts \| `alarm:` \| Attached to specific charts \| Monitor specific server's CPU \|
331	\| Templates \| `template:` \| Apply to all charts of a context \| Monitor all network interfaces \|
332
333	Alerts are attached to specific charts and use the `alarm` label.
334
335	Templates define rules that apply to all charts of a specific context, and use the `template` label. Templates help you apply one entity to all disks, all network interfaces, all MySQL databases, and so on.
336
337	:::note
338
339	Precedence
340
341	Alarms are processed before templates. If you have `alarm` and `template` entities with the same name that both match the same chart, only the `alarm` will create an active alert for that chart.
342
343	For complete details on configuration loading order and precedence rules, see [Alert Configuration Ordering](/src/health/alert-configuration-ordering.md).
344
345	:::
346
347	### Required vs Optional Configuration
348
349	:::note
350
351	Configuration Requirements
352
353	- The `alarm` or `template` line must be the first line of any entity
354	- The `on` line is always required
355	- The `every` line is required if not using `lookup`
356	- Each entity must have at least one of the following lines: `lookup`, `calc`, `warn`, or `crit`
357
358	:::
359
360	Special Syntax Rules:
361
362	- A few lines use space-separated lists to define how the entity behaves. You can use `*` as a wildcard or prefix with `!` for a negative match. Order is important! See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for more examples
363	- Lines terminated by a `\` are spliced together with the next line. The backslash is removed, and the following line is joined with the current one. No space is inserted, so you can split a line anywhere, even in the middle of a word. This is handy if your `info` line consists of several sentences
364
365	### Complete Configuration Reference
366
367	\| line \| required \| functionality \|
368	\| --------------------------------------------------- \| --------------- \| ------------------------------------------------------------------------------------ \|
369	\| [`alarm`/`template`](#alert-line-alarm-or-template) \| yes \| Name of the alert/template \|
370	\| [`on`](#alert-line-on) \| yes \| The chart this alert should attach to \|
371	\| [`class`](#alert-line-class) \| no \| The general alert classification \|
372	\| [`type`](#alert-line-type) \| no \| What area of the system the alert monitors \|
373	\| [`component`](#alert-line-component) \| no \| Specific component of the type of the alert \|
374	\| [`lookup`](#alert-line-lookup) \| yes \| The database lookup to find and process metrics for the chart specified through `on` \|
375	\| [`calc`](#alert-line-calc) \| yes (see above) \| A calculation to apply to the value found via `lookup` or another variable \|
376	\| [`every`](#alert-line-every) \| no \| The frequency of the alert \|
377	\| [`green`/`red`](#alert-lines-green-and-red) \| no \| Set the green and red thresholds of a chart \|
378	\| [`warn`/`crit`](#alert-lines-warn-and-crit) \| yes (see above) \| Expressions evaluating to true or false, and when true, will trigger the alert \|
379	\| [`to`](#alert-line-to) \| no \| A list of roles to send notifications to \|
380	\| [`exec`](#alert-line-exec) \| no \| The script to execute when the alert changes status \|
381	\| [`delay`](#alert-line-delay) \| no \| Optional hysteresis settings to prevent floods of notifications \|
382	\| [`repeat`](#alert-line-repeat) \| no \| The interval for sending notifications when an alert is in WARNING or CRITICAL mode \|
383	\| [`options`](#alert-line-options) \| no \| Add an option to not clear alerts \|
384	\| [`host labels`](#alert-line-host-labels) \| no \| Restrict an alert or template to a list of matching labels present on a host \|
385	\| [`chart labels`](#alert-line-chart-labels) \| no \| Restrict an alert or template to a list of matching labels present on a chart \|
386	\| [`summary`](#alert-line-summary) \| no \| A brief description of the alert \|
387	\| [`info`](#alert-line-info) \| no \| A longer text field that provides more information about this alert \|
388
389	### Configuration Line Details
390
391	#### Alert Line `alarm` or `template`
392
393	Purpose: This line starts an alert or template based on the [entity type](#entity-types-overview) you want to create.
394
395	Alert Syntax:
396
397	```text
398	alarm: NAME
399	```
400
401	Template Syntax:
402
403	```text
404	template: NAME
405	```
406
407	Naming Rules:
408
409	- `NAME` can be any alphanumeric character
410	- Only `.` (period) and `_` (underscore) symbols allowed
411	- Can’t be `chart name`, `dimension name`, `family name`, or `chart variable names`
412
413	#### Alert Line `on`
414
415	Purpose: This line defines the chart this alert should attach to.
416
417	For Alerts:
418
419	```text
420	on: CHART
421	```
422
423	The value `CHART` should be the unique ID or name of the chart you're interested in, as shown on the dashboard. In the image below, the unique ID is `system.cpu`.
424
425	![Finding the unique ID of a chart](https://user-images.githubusercontent.com/1153921/67443082-43b16e80-f5b8-11e9-8d33-d6ee052c6678.png)
426
427	For Templates:
428
429	```text
430	on: CONTEXT
431	```
432
433	The value `CONTEXT` should be the context you want this template to attach to.
434
435	:::tip
436
437	Finding the Context
438
439	Need to find the context? Hover over the date on any given chart and look at the tooltip. In the image below, which shows a disk I/O chart, the tooltip reads: `proc:/proc/diskstats, disk.io`.
440
441	![Finding the context of a chart via the tooltip](https://user-images.githubusercontent.com/1153921/68882856-2b230880-06cd-11ea-923b-b28c4632d479.png)
442
443	You're interested in what comes after the comma: `disk.io`. That's the name of the chart's context.
444
445	If you create a template using the `disk.io` context, it will apply an alert to every disk available on your system.
446
447	:::
448
449	#### Alert Line `class`
450
451	Purpose: This indicates the type of error (or general problem area) that the alert or template applies to.
452
453	Example Use: `Latency` can be used for alerts that trigger on latency issues on network interfaces, web servers, or database systems.
454
455	```text
456	class: Latency
457	```
458
459	Available Classes:
460
461	\| Class \| Use Case \|
462	\| ----------- \| ------------------------------ \|
463	\| Errors \| Error rate monitoring \|
464	\| Latency \| Response time issues \|
465	\| Utilization \| Resource usage monitoring \|
466	\| Workload \| Load and throughput monitoring \|
467
468	:::note
469
470	`class` will default to `Unknown` if the line is missing from the alert configuration.
471
472	:::
473
474	#### Alert Line `type`
475
476	Purpose: You can use `type` to indicate the broader area of the system that the alert applies to.
477
478	Example: Under the general `Database` type, you can group together alerts that operate on various database systems, like `MySQL`, `CockroachDB`, `CouchDB`, etc.
479
480	```text
481	type: Database
482	```
483
484	Available Types:
485
486	\| Type \| Description \|
487	\| --------------- \| ---------------------------------------------------------------------------------------------- \|
488	\| Ad Filtering \| Services related to Ad Filtering (like pi-hole) \|
489	\| Certificates \| Certificate monitoring related \|
490	\| Cgroups \| Alerts for CPU and memory usage of control groups \|
491	\| Computing \| Alerts for shared computing applications (e.g. boinc) \|
492	\| Containers \| Container related alerts (e.g. docker instances) \|
493	\| Database \| Database systems (e.g. MySQL, PostgreSQL, etc) \|
494	\| Data Sharing \| Used to group together alerts for data sharing applications \|
495	\| DHCP \| Alerts for DHCP related services \|
496	\| DNS \| Alerts for DNS related services \|
497	\| Kubernetes \| Alerts for kubernetes nodes monitoring \|
498	\| KV Storage \| Key-Value pairs services alerts (e.g. memcached) \|
499	\| Linux \| Services specific to Linux (e.g. systemd) \|
500	\| Messaging \| Alerts for message passing services (e.g. vernemq) \|
501	\| Netdata \| Internal Netdata components monitoring \|
502	\| Other \| When an alert doesn't fit in other types \|
503	\| Power Supply \| Alerts from power supply related services (e.g. apcupsd) \|
504	\| Search engine \| Alerts for search services (e.g. elasticsearch) \|
505	\| Storage \| Class for alerts dealing with storage services (storage devices typically live under `System`) \|
506	\| System \| General system alerts (e.g. CPU, network, etc.) \|
507	\| Virtual Machine \| Virtual Machine software \|
508	\| Web Proxy \| Web proxy software (e.g. squid) \|
509	\| Web Server \| Web server software (e.g. Apache, nginx, etc.) \|
510	\| Windows \| Alerts for monitoring Windows services \|
511
512	:::note
513
514	If an alert configuration is missing the `type` line, its value will default to `Unknown`.
515
516	:::
517
518	#### Alert Line `component`
519
520	Purpose: You can use `component` to narrow down what the previous `type` value specifies for each alert or template.
521
522	Example: Continuing from the previous example, `component` might include `MySQL`, `CockroachDB`, `MongoDB`, all under the same `Database` type.
523
524	```text
525	component: MySQL
526	```
527
528	:::note
529
530	As with the `class` and `type` lines, if `component` is missing from the configuration, its value will default to `Unknown`.
531
532	:::
533
534	#### Alert Line `lookup`
535
536	Purpose: This line makes a database lookup to find a value. The result of this lookup is available as `$this`.
537
538	Full Syntax:
539
540	```text
541	lookup: METHOD(GROUPING OPTIONS) AFTER [at BEFORE] [every DURATION] [OPTIONS] [of DIMENSIONS]
542	```
543
544	Required Parameters:
545
546	\| Parameter \| Description \| Example \|
547	\| --------- \| ------------------------------------------------------------------ \| ----------------------- \|
548	\| `METHOD` \| [Grouping method](/src/web/api/queries/README.md#grouping-methods) \| `average`, `min`, `max` \|
549	\| `AFTER` \| How far back to look (negative number) \| `-1m`, `-1h`, `-1d` \|
550
551	Optional Parameters:
552
553	\| Parameter \| Purpose \| Details \|
554	\| ------------------ \| --------------------------- \| -------------------------------------------------------------------------- \|
555	\| `GROUPING OPTIONS` \| Conditional processing \| `CONDITION VALUE` where condition is `!=`, `=`, `==`, `<=`, `<`, `>`, `>=` \|
556	\| `at BEFORE` \| End of lookup timeframe \| Default is 0 (now) \|
557	\| `every DURATION` \| Update frequency \| Supports `s`, `m`, `h`, `d` units \|
558	\| `OPTIONS` \| Processing modifiers \| See options table below \|
559	\| `of DIMENSIONS` \| Which dimensions to include \| Comma- or pipe-separated list, supports patterns; prefer `user,system` over `user, system` \|
560
561	Processing Options:
562
563	\| Option \| Effect \|
564	\| ------------- \| -------------------------------------------------------------------------------------------------------- \|
565	\| `percentage` \| Calculate percentage of selected dimensions over total \|
566	\| `absolute` \| Turn all sample values positive \|
567	\| `min` \| Return minimum of all dimensions after time-aggregation \|
568	\| `max` \| Return maximum of all dimensions after time-aggregation \|
569	\| `average` \| Return average of all dimensions after time-aggregation \|
570	\| `sum` \| Return sum of all dimensions (default) \|
571	\| `min2max` \| Return delta between min and max of dimensions \|
572	\| `unaligned` \| Prevent shifting query window to multiples of duration \|
573	\| `anomaly-bit` \| Query anomaly-rate percentages (0-100) instead of raw values, enabling anomaly-rate-based alerting \|
574	\| `match-ids` \| Match dimensions by IDs (default) \|
575	\| `match-names` \| Match dimensions by names \|
576
577	When `anomaly-bit` is used, each data point returns the anomaly rate as a percentage from 0 to 100. At native resolution this is typically 0 (normal) or 100 (anomalous), while aggregated or lower-resolution data can yield intermediate values such as 12.5. For more details and practical examples, see the [ML anomaly detection documentation](/docs/ml-ai/ml-anomaly-detection/ml-anomaly-detection.md).
578
579	Example:
580
581	```text
582	lookup: average -10m unaligned of user,system,softirq,irq,guest
583	```
584
585	This looks back 10 minutes, calculates the average of the specified CPU dimensions, without aligning to time boundaries.
586
587	The result of the lookup will be available as `$this` and `$NAME` in expressions. The timestamps of the timeframe evaluated by the database lookup are available as variables `$after` and `$before` (both are unix timestamps).
588
589	#### Alert Line `calc`
590
591	Purpose: You can design a `calc` to apply some calculation to the values or variables available to the entity.
592
593	Key Points:
594
595	- The result becomes available as `$this` variable
596	- Overwrites the value from your `lookup`
597	- Can be used without `lookup` if using [other available variables](#variables-reference)
598	- Uses [expressions](#expressions-overview) for syntax
599
600	```text
601	calc: EXPRESSION
602	```
603
604	When to Use:
605
606	- With `lookup`: Perform calculation after database retrieval
607	- Without `lookup`: When using other available variables
608	- For complex logic: Mathematical operations, conditions, transformations
609
610	#### Alert Line `every`
611
612	Purpose: Sets the update frequency of this alert.
613
614	```text
615	every: DURATION
616	```
617
618	Supported Units:
619
620	- `s` for seconds
621	- `m` for minutes
622	- `h` for hours
623	- `d` for days
624
625	Example: `every: 30s` checks the alert every 30 seconds.
626
627	#### Alert Lines `green` and `red`
628
629	Purpose: Set the green and red thresholds of a chart for visualization.
630
631	```text
632	green: NUMBER
633	red: NUMBER
634	```
635
636	Important Notes:
637
638	- Both values are available as `$green` and `$red` in expressions
639	- If multiple alerts define different thresholds, the first alert's values are used
640	- For multiple threshold sets, use absolute numbers instead of variables
641
642	#### Alert Lines `warn` and `crit`
643
644	Purpose: Define the expressions that trigger warning or critical alerts.
645
646	```text
647	warn: EXPRESSION
648	crit: EXPRESSION
649	```
650
651	Key Points:
652
653	- Optional (but you need at least one)
654	- Should evaluate to true/false (or zero/non-zero)
655	- Uses Netdata's [expression syntax](#expressions-overview)
656	- Can reference variables like `$this`, `$green`, `$red`
657
658	Examples:
659
660	```text
661	warn: $this > 80
662	crit: $this > 95
663	```
664
665	#### Alert Line `to`
666
667	Purpose: Specifies who receives notifications when the alert changes status.
668
669	```text
670	to: ROLE1 ROLE2 ROLE3 ...
671	```
672
673	How It Works:
674
675	- First parameter passed to the `exec` script
676	- Default script (`alarm-notify.sh`) treats this as a space-separated list of roles
677	- Roles are consulted to find exact recipients per notification method
678
679	#### Alert Line `exec`
680
681	Purpose: Script to execute when the alert status changes.
682
683	```text
684	exec: SCRIPT
685	```
686
687	Default Behavior:
688
689	- Default script is Netdata's `alarm-notify.sh`
690	- Supports all notification methods Netdata supports
691	- Includes custom hooks
692
693	#### Alert Line `delay`
694
695	Purpose: Provide optional hysteresis settings to prevent notification floods.
696
697	:::important
698
699	These settings don't affect the actual alert - only when the `exec` script is executed.
700
701	:::
702
703	Full Syntax:
704
705	```text
706	delay: [[[up U] [down D] multiplier M] max X]
707	```
708
709	Parameters:
710
711	\| Parameter \| Purpose \| Default \|
712	\| -------------- \| ------------------------------------------------------------ \| ------------- \|
713	\| `up U` \| Delay for status increases (CLEAR→WARNING, WARNING→CRITICAL) \| 0 \|
714	\| `down D` \| Delay for status decreases (CRITICAL→WARNING, WARNING→CLEAR) \| 0 \|
715	\| `multiplier M` \| Multiplies U and D when alert changes state during delay \| 1.0 \|
716	\| `max X` \| Maximum absolute notification delay \| max(U×M, D×M) \|
717
718	Example with Timeline:
719
720	```text
721	delay: up 10s down 15m multiplier 2 max 1h
722	```
723
724	Starting at `00:00:00` with CLEAR status:
725
726	\| Time \| New Status \| Delay Applied \| Notification At \| Reason \|
727	\| -------- \| ---------- \| -------------- \| --------------- \| ----------------------------------------- \|
728	\| 00:00:01 \| WARNING \| `up 10s` \| 00:00:11 \| First state switch \|
729	\| 00:00:05 \| CLEAR \| `down 15m x2` \| 00:30:05 \| Alert changed during delay, so multiplied \|
730	\| 00:00:06 \| WARNING \| `up 10s x2 x2` \| 00:00:26 \| Multiplied twice \|
731
732	#### Alert Line `repeat`
733
734	Purpose: Defines the interval between repeating notifications for alerts in CRITICAL or WARNING mode.
735
736	```text
737	repeat: [off] [warning DURATION] [critical DURATION]
738	```
739
740	Options:
741
742	\| Option \| Effect \|
743	\| ------------------- \| -------------------------------------------------------- \|
744	\| `off` \| Turns off repeating for this alert \|
745	\| `warning DURATION` \| Repeat interval for WARNING state (use `0s` to disable) \|
746	\| `critical DURATION` \| Repeat interval for CRITICAL state (use `0s` to disable) \|
747
748	Why Use This: Overrides default repeat settings from `netdata.conf` health configuration.
749
750	#### Alert Line `options`
751
752	Purpose: Special alert behavior options.
753
754	```text
755	options: no-clear-notification
756	```
757
758	Available Options:
759
760	- `no-clear-notification` - Prevents clearing the alert notification
761
762	When to Use `no-clear-notification`:
763
764	- Alerts comparing two time frames (e.g., last 3 minutes vs last hour)
765	- When newer data might "pollute" the baseline comparison
766	- When clearing conditions are unreliable due to data characteristics
767
768	Example Use Case: HTTP response time alert comparing recent average to historical average - as time passes, the recent slow responses become part of the historical data, making the alert appear "cleared" even though the underlying issue wasn't resolved.
769
770	#### Alert Line `host labels`
771
772	Purpose: Restricts alerts to hosts with matching labels.
773
774	Prerequisites: See our [host labels guide](/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts.md) for setup instructions.
775
776	Example Configuration:
777
778	```text
779	[host labels]
780	installed = 20191211
781	room = server
782	```
783
784	Usage in Alerts:
785
786	```text
787	host labels: room = server
788	```
789
790	Pattern Support:
791
792	```text
793	host labels: installed = 201* # Matches all hosts installed in 2010s
794	```
795
796	How It Works:
797
798	- Space-separated list
799	- Accepts [simple patterns](/src/libnetdata/simple_pattern/README.md)
800	- Alert only loads on matching hosts
801
802	#### Alert Line `chart labels`
803
804	Purpose: Filters alerts based on chart labels.
805
806	How to Find Chart Labels: Check `http://localhost:19999/api/v1/charts?all`
807
808	Example Use Case:
809	Each `disk_space` chart has a `mount_point` label. To exclude external disk alerts:
810
811	```text
812	chart labels: mount_point=!/mnt/disk1 *
813	```
814
815	Multiple Label Logic:
816
817	```text
818	chart labels: mount_point=/mnt/disk1 device=sda
819	```
820
821	This requires BOTH conditions to be true (AND logic).
822
823	:::important
824
825	- Space-separated list with [simple patterns](/src/libnetdata/simple_pattern/README.md) support
826	- If a specified label doesn't exist on the chart, the chart won't match
827	- Multiple labels use AND logic
828	- Alerts based on `chart labels` require the underlying chart to exist. For example, a `disk.space` chart is only created when a mount point is present and collected. For example, if a CIFS mount fails to mount after a system reboot, no `disk.space` chart will exist for that mount point, and the alert will not activate
829
830	:::
831
832	#### Alert Line `summary`
833
834	Purpose: Brief title of the alert used in notifications and dashboard.
835
836	```text
837	summary: Available Ram
838	```
839
840	Variable Support:
841
842	\| Variable \| Replaced With \|
843	\| --------------------- \| ---------------------------- \|
844	\| `${family}` \| Family instance (e.g., eth0) \|
845	\| `${label:LABEL_NAME}` \| Chart label value \|
846
847	Example with Variables:
848
849	```text
850	summary: 1 minute received traffic overflow for ${label:device}
851	```
852
853	Renders as: `1 minute received traffic overflow for eth0`
854
855	:::note
856
857	Variable names are case-sensitive.
858
859	:::
860
861	#### Alert Line `info`
862
863	Purpose: Detailed description of the alert for notifications and UI elements.
864
865	```text
866	info: Percentage of estimated amount of RAM available for userspace processes, without causing swapping
867	```
868
869	Variable Support:
870
871	\| Variable \| Replaced With \|
872	\| --------------------- \| ---------------------------- \|
873	\| `${family}` \| Family instance (e.g., eth0) \|
874	\| `${label:LABEL_NAME}` \| Chart label value \|
875
876	Examples with Variables:
877
878	Family Variable:
879
880	```text
881	info: average inbound utilization for the network interface ${family} over the last minute
882	```
883
884	Renders as: `average inbound utilization for the network interface eth0 over the last minute`
885
886	Label Variable:
887
888	```text
889	info: average ratio of HTTP responses with unexpected status over the last 5 minutes for the site ${label:target}
890	```
891
892	Renders as: `average ratio of HTTP responses with unexpected status over the last 5 minutes for the site https://netdata.cloud/`
893
894	Next Steps: Continue to [Expressions and Variables](#expressions-and-variables) to understand the calculation syntax, or jump to [Alert Examples](#alert-examples) for practical implementations.
895
896	## Expressions and Variables
897
898	:::tip
899
900	What You'll Learn
901
902	How to write calculations and use variables in your alert definitions. Essential for creating custom logic and accessing chart data.
903
904	:::
905
906	### Expressions Overview
907
908	Why This Matters: Netdata has an internal infix expression parser that allows you to create complex alert logic using mathematical operations, comparisons, and conditional statements.
909
910	Supported Operators:
911
912	\| Type \| Operators \| Result \|
913	\| ---------- \| -------------------------------------- \| ------------------------- \|
914	\| Arithmetic \| `+`, `-`, `*`, `/` \| Numeric values \|
915	\| Comparison \| `<`, `==`, `<=`, `<>`, `!=`, `>`, `>=` \| `1` (true) or `0` (false) \|
916	\| Logical \| `&&`, `\|\|`,`!`,`AND`,`OR`,`NOT` \| `1` (true) or `0` (false) \|
917
918	Special Functions:
919
920	- `abs()` - Absolute value
921	- `(condition) ? (true_expr) : (false_expr)` - Conditional operator
922
923	Special Values:
924
925	\| Value \| Purpose \| Example Use \|
926	\| ----- \| ------------------------------------- \| -------------- \|
927	\| `nan` \| Not a number (database lookup failed) \| `$this != nan` \|
928	\| `inf` \| Infinite (division by zero) \| `$this != inf` \|
929
930	### Conditional Operator for Hysteresis
931
932	Why This Matters: The conditional operator (`? :`) can create "sticky" alert thresholds that prevent alert spam when values fluctuate around a threshold. This is called [hysteresis](https://en.wikipedia.org/wiki/Hysteresis).
933
934	Basic Pattern:
935
936	```text
937	warn: $this > (($status >= $WARNING) ? (lower_threshold) : (higher_threshold))
938	```
939
940	Real Example - CPU Usage Alert:
941
942	```text
943	warn: $this > (($status >= $WARNING) ? (75) : (85))
944	crit: $this > (($status == $CRITICAL) ? (85) : (95))
945	```
946
947	How This Works:
948
949	\| Alert State \| Triggers At \| Clears At \| Explanation \|
950	\| ----------- \| ----------- \| --------- \| ----------------------------------------------------------------- \|
951	\| Warning \| 85% CPU \| 75% CPU \| Creates 10% buffer - CPU must drop below 75% to clear warning \|
952	\| Critical \| 95% CPU \| 85% CPU \| Creates 10% buffer - CPU must drop below 85% to return to warning \|
953
954	Benefits:
955
956	- Quick alerting when issues arise
957	- Protection against spam when values hover near thresholds
958	- Single initial notification instead of constant alerts during fluctuation
959
960	Example Scenario: If CPU usage fluctuates between 80–90%, you'll receive just one initial warning, rather than constant notifications.
961
962	### Variables Reference
963
964	How to Find Available Variables:
965	You can find all variables for a given chart using: `http://NODE:19999/api/v1/alarm_variables?chart=CHART_NAME`
966
967	Example: [Variables for the `system.cpu` chart](https://registry.my-netdata.io/api/v1/alarm_variables?chart=system.cpu)
968
969	:::note
970
971	Chart vs Template Variables
972
973	Although the `alarm_variables` link shows variables for a particular chart, the same variables can also be used in templates for charts belonging to a given context. All charts of a given context are essentially identical, with the only difference being the family that identifies a particular hardware or software instance.
974
975	:::
976
977	### Variable Categories
978
979	#### Chart Local Variables
980
981	What's Available:
982
983	- All chart dimensions as variables (e.g., `$user`, `$system` for CPU chart)
984	- Values from other configured alerts on the same chart
985	- Special chart variables (see table below)
986
987	Special Chart Variables:
988
989	\| Variable \| Contains \|
990	\| ---------------------- \| --------------------------------------------- \|
991	\| `$last_collected_t` \| Unix timestamp of last data collection \|
992	\| `$collected_total_raw` \| Sum of all dimensions (last collected values) \|
993	\| `$update_every` \| Update frequency of the chart \|
994	\| `$green`, `$red` \| Thresholds defined in alerts \|
995
996	Dimension Value Types:
997
998	- Default: Last calculated (interpolated) value as shown on charts
999	- Raw suffix: `$dimension_raw` - Last collected value
1000	- Timestamp suffix: `$dimension_last_collected_t` - Unix timestamp when dimension was last collected
1001
1002	#### Host Variables
1003
1004	What's Available: All dimensions of all charts, including all alerts, in fullname format.
1005
1006	Format: `CHART.VARIABLE`
1007
1008	- `CHART` can be either chart ID or chart name
1009	- Both formats are supported
1010
1011	Examples:
1012
1013	- `$system.cpu.user` - User CPU from system.cpu chart
1014	- `$disk.sda.reads` - Read operations from sda disk chart
1015
1016	##### Cross-Chart Variable Examples from Stock Health Entities
1017
1018	Several stock health configurations use host variables to reference dimensions from other charts in their `calc`, `warn`, and `crit` expressions.
1019
1020	\| Health entity \| File \| Expression \| Cross-chart reference \|
1021	\|---------------\|------\|------------\|-----------------------\|
1022	\| `30min_ram_swapped_out` \| `health.d/swap.conf` \| `calc: $this / 1024 * 100 / ( $system.ram.used + $system.ram.cached + $system.ram.free )` \| `$system.ram.*` from within an alert on the `mem.swapio` chart \|
1023	\| `ram_available` \| `health.d/ram.conf` \| `calc: $avail * 100 / ($system.ram.used + $system.ram.cached + $system.ram.free + $system.ram.buffers)` \| `$system.ram.*` from within an alert on the `mem.available` chart \|
1024	\| `system_clock_sync_state` \| `health.d/timex.conf` \| `warn: $system.uptime.uptime > 17 * 60 AND $this == 0` \| `$system.uptime.uptime` from within an alert on the `system.clock_sync_state` chart \|
1025	\| `audit_backlog_utilization` \| `health.d/audit.conf` \| `warn: $this > 50 AND $audit.failure.panic == 1` \| `$audit.failure.panic` from within an alert on the `audit.backlog_utilization` chart \|
1026	\| `10s_ip_tcp_resets_sent` \| `health.d/tcp_resets.conf` \| `warn: $netdata.uptime.uptime > (1 * 60) AND ...` \| `$netdata.uptime.uptime` from within an alert on the `ip.tcphandshake` chart \|
1027	\| `streaming_never_connected` \| `health.d/streaming.conf` \| `warn: $netdata.uptime.uptime > 30 * 60 AND $this > 0` \| `$netdata.uptime.uptime` from within an alert on the `netdata.streaming_inbound` chart \|
1028
1029	##### Prometheus Collector Variables
1030
1031	For metrics collected by the go.d `prometheus` collector, each unique Prometheus label set usually produces a separate chart. The chart ID is built from the metric name followed by `-label=value` pairs for every label (e.g. `kubelet_volume_stats_used_bytes-persistentvolumeclaim=my-pvc`); characters in a label value that are not chart-ID-safe, such as `.`, are replaced with `_` in the chart ID, while the chart's label keeps the original value (so `addr="10.0.0.1"` yields `…-addr=10_0_0_1`). In the Netdata chart registry, the prefix comes from the go.d job `FullName`: it is `prometheus.<metric_name>-<label_set>` only when the job name is literally `prometheus`; otherwise it is `prometheus_<job_name>.<metric_name>-<label_set>` (for example, `prometheus_local.<metric_name>-<label_set>` or `prometheus_kubelet.<metric_name>-<label_set>`). Summary and histogram families also emit separate `_sum` and `_count` charts; the suffix is part of the metric name, so the IDs are `<metric_name>_sum-<label_set>` and `<metric_name>_count-<label_set>` (just `<metric_name>_sum` / `<metric_name>_count` when the series has no labels), while histogram buckets are dimensions of the base `<metric_name>` chart. Verify the exact chart ID you want to reference.
1032
1033	Because Prometheus chart IDs typically contain hyphens and `=` characters, use the `${...}` brace form to reference them in `calc`/`warn`/`crit` expressions — the unbraced `$var` form stops parsing at `-`. Apply the same rule for both the common `prometheus_<job_name>` prefix and the special-case plain `prometheus` prefix, including any `_sum` or `_count` chart variants.
1034
1035	Example — PVC volume usage alert using kubelet metrics from a named Prometheus job (`name: kubelet`):
1036
1037	```text
1038	alarm: kubelet_pvc_volume_usage
1039	on: prometheus_kubelet.kubelet_volume_stats_used_bytes-persistentvolumeclaim=my-pvc
1040	lookup: max -1m unaligned match-names of kubelet_volume_stats_used_bytes
1041	calc: $this * 100 / ${prometheus_kubelet.kubelet_volume_stats_capacity_bytes-persistentvolumeclaim=my-pvc.kubelet_volume_stats_capacity_bytes}
1042	warn: $this > 80
1043	crit: $this > 95
1044	units: %
1045	every: 1m
1046	info: PVC volume usage percentage
1047	to: sysadmin
1048	```
1049
1050	:::note
1051
1052	The exact chart ID and dimension names depend on your endpoint's label sets. Use the alarm variables API to discover the correct names:
1053
1054	```text
1055	http://NODE:19999/api/v1/alarm_variables?chart=prometheus_kubelet.kubelet_volume_stats_used_bytes-persistentvolumeclaim%3Dmy-pvc
1056	```
1057
1058	URL-encode the chart ID only in API query parameters. In alert expressions, use the chart ID as-is inside `${...}` braces.
1059
1060	:::
1061
1062	#### Special Variables
1063
1064	\| Variable \| Contains \| Usage \|
1065	\| --------- \| ---------------------- \| -------------------------------------- \|
1066	\| `$this` \| Current alert value \| Result of `calc` line or current alert \|
1067	\| `$status` \| Current alert status \| Compare with status constants \|
1068	\| `$now` \| Current unix timestamp \| Time-based calculations \|
1069
1070	Alert Status Constants:
1071
1072	\| Constant \| Numeric Value \| Usage \|
1073	\| ---------------- \| ------------- \| ------------------------------ \|
1074	\| `$REMOVED` \| -2 \| Alert deleted (SIGUSR2 reload) \|
1075	\| `$UNINITIALIZED` \| -1 \| Alert not initialized \|
1076	\| `$UNDEFINED` \| 0 \| Calculation failed \|
1077	\| `$CLEAR` \| 1 \| Alert OK/not triggered \|
1078	\| `$WARNING` \| 2 \| Warning condition met \|
1079	\| `$CRITICAL` \| 3 \| Critical condition met \|
1080
1081	Status Comparison Examples:
1082
1083	```text
1084	# Check if alert is in warning or higher
1085	warn: $status >= $WARNING
1086
1087	# Check if alert is specifically critical
1088	crit: $status == $CRITICAL
1089	```
1090
1091	:::note
1092
1093	Status values increase with severity, so `$status > $CLEAR` will match both WARNING and CRITICAL states.
1094
1095	:::
1096
1097	### Alert Status Lifecycle
1098
1099	Status Flow: `UNINITIALIZED` → `UNDEFINED`/`CLEAR` → `WARNING` → `CRITICAL`
1100
1101	When Status Changes:
1102
1103	- `REMOVED` - Alert deleted during configuration reload
1104	- `UNINITIALIZED` - Alert created but not yet calculated
1105	- `UNDEFINED` - Database lookup failed, division by zero, etc.
1106	- `CLEAR` - Alert conditions aren’t met (normal state)
1107	- `WARNING` - Warning expression returned true/non-zero
1108	- `CRITICAL` - Critical expression returned true/non-zero
1109
1110	Script Execution: The external script (`exec` line) is called for ALL status changes.
1111
1112	Next Steps: Ready to see these concepts in action? Continue to [Alert Examples](#alert-examples) for practical implementations.
1113
1114	## Alert Examples
1115
1116	:::tip
1117
1118	What You'll Learn
1119
1120	Real-world alert configurations that demonstrate different monitoring scenarios. Use these as templates for your own alerts.
1121
1122	:::
1123
1124	<details>
1125	<summary><strong>Example 1: Server Alive Check</strong></summary><br/>
1126
1127	Scenario: Monitor if the Apache server is collecting data properly.
1128
1129	Why This Matters: Detect when data collection stops, indicating potential server or network issues.
1130
1131	```text
1132	template: apache_last_collected_secs
1133	on: apache.requests
1134	calc: $now - $last_collected_t
1135	every: 10s
1136	warn: $this > ( 5 * $update_every)
1137	crit: $this > (10 * $update_every)
1138	```
1139
1140	How It Works:
1141
1142	\| Component \| Purpose \| This Example \|
1143	\| ---------- \| ------------------------------- \| ---------------------------- \|
1144	\| `template` \| Applies to all Apache servers \| `apache_last_collected_secs` \|
1145	\| `on` \| Chart context to monitor \| `apache.requests` \|
1146	\| `calc` \| Time since last data collection \| `$now - $last_collected_t` \|
1147	\| `every` \| Check frequency \| Every 10 seconds \|
1148	\| `warn` \| Warning threshold \| 5 missed collection cycles \|
1149	\| `crit` \| Critical threshold \| 10 missed collection cycles \|
1150
1151	Variables Used:
1152
1153	- `$now` - Current timestamp
1154	- `$last_collected_t` - Last data collection timestamp
1155	- `$update_every` - Chart update frequency
1156	- `$this` - Result of calculation (seconds since last collection)
1157
1158	<br/>
1159	</details>
1160
1161	<details>
1162	<summary><strong>Example 2: Disk Space Monitoring</strong></summary><br/>
1163
1164	Scenario: Alert when any disk is running low on space.
1165
1166	Why This Matters: Prevent system failures due to full disks.
1167
1168	```text
1169	template: disk_full_percent
1170	on: disk.space
1171	calc: $used * 100 / ($avail + $used)
1172	every: 1m
1173	warn: $this > 80
1174	crit: $this > 95
1175	repeat: warning 120s critical 10s
1176	```
1177
1178	How It Works:
1179
1180	\| Component \| Purpose \| This Example \|
1181	\| ----------- \| -------------------------- \| -------------------------------------- \|
1182	\| `template` \| Applies to all disks \| `disk_full_percent` \|
1183	\| `on` \| Chart context \| `disk.space` \|
1184	\| `calc` \| Calculate usage percentage \| `$used * 100 / ($avail + $used)` \|
1185	\| `warn/crit` \| Simple thresholds \| 80% warning, 95% critical \|
1186	\| `repeat` \| Notification frequency \| Every 2min (warning), 10sec (critical) \|
1187
1188	Variables Used:
1189
1190	- `$used` - Used disk space dimension
1191	- `$avail` - Available disk space dimension
1192
1193	<br/>
1194	</details>
1195
1196	<details>
1197	<summary><strong>Example 3: Predictive Disk Full Alert</strong></summary><br/>
1198
1199	Scenario: Predict when disks will run out of space based on the current fill rate.
1200
1201	Why This Matters: Get warning before disk space becomes critical.
1202
1203	Step 1: Calculate Disk Fill Rate
1204
1205	```text
1206	template: disk_fill_rate
1207	on: disk.space
1208	lookup: max -1s at -30m unaligned of avail
1209	calc: ($this - $avail) / (30 * 60)
1210	every: 15s
1211	```
1212
1213	Step 2: Predict Hours Until Full
1214
1215	```text
1216	template: disk_full_after_hours
1217	on: disk.space
1218	calc: $avail / $disk_fill_rate / 3600
1219	every: 10s
1220	warn: $this > 0 and $this < 48
1221	crit: $this > 0 and $this < 24
1222	```
1223
1224	How It Works:
1225
1226	\| Step \| Purpose \| Calculation \|
1227	\| ---- \| -------------------- \| -------------------------------------------------- \|
1228	\| 1 \| Calculate fill rate \| `(space_30min_ago - current_space) / 1800_seconds` \|
1229	\| 2 \| Predict time to full \| `current_available / fill_rate / 3600` \|
1230
1231	Logic:
1232
1233	- Only positive predictions matter (disk filling up)
1234	- Warning: Less than 48 hours of space remaining
1235	- Critical: Less than 24 hours of space remaining
1236
1237	<br/>
1238	</details>
1239
1240	<details>
1241	<summary><strong>Example 4: Network Packet Drops</strong></summary><br/>
1242
1243	Scenario: Alert on any network packet drops.
1244
1245	Why This Matters: Packet drops indicate network issues that could affect performance.
1246
1247	```text
1248	template: 30min_packet_drops
1249	on: net.drops
1250	lookup: sum -30m unaligned absolute
1251	every: 10s
1252	crit: $this > 0
1253	```
1254
1255	How It Works:
1256
1257	\| Component \| Purpose \| This Example \|
1258	\| ---------- \| --------------------------------- \| ----------------------------- \|
1259	\| `template` \| Applies to all network interfaces \| `30min_packet_drops` \|
1260	\| `lookup` \| Sum drops over 30 minutes \| `sum -30m unaligned absolute` \|
1261	\| `crit` \| Any drops trigger critical \| `$this > 0` \|
1262
1263	Key Points:
1264
1265	- The drops chart only exists when packets are dropped
1266	- The alert automatically attaches when the first drop is detected
1267	- Zero tolerance for packet loss
1268
1269	<br/>
1270	</details>
1271
1272	<details>
1273	<summary><strong>Example 5: Z-Score Based Alert</strong></summary><br/>
1274
1275	Scenario: Detect CPU usage anomalies using statistical analysis.
1276
1277	Why This Matters: Identify unusual patterns that fixed thresholds might miss.
1278
1279	```text
1280	alarm: cpu_user_mean
1281	on: system.cpu
1282	lookup: mean -60s of user
1283	every: 10s
1284
1285	alarm: cpu_user_stddev
1286	on: system.cpu
1287	lookup: stddev -60s of user
1288	every: 10s
1289
1290	alarm: cpu_user_zscore
1291	on: system.cpu
1292	lookup: mean -10s of user
1293	calc: ($this - $cpu_user_mean) / $cpu_user_stddev
1294	every: 10s
1295	warn: $this < -2 or $this > 2
1296	crit: $this < -3 or $this > 3
1297	```
1298
1299	How It Works:
1300
1301	\| Alert \| Purpose \| Calculation \|
1302	\| ----------------- \| --------------------------- \| ---------------------------------- \|
1303	\| `cpu_user_mean` \| Calculate average CPU usage \| Mean over 60 seconds \|
1304	\| `cpu_user_stddev` \| Calculate variability \| Standard deviation over 60 seconds \|
1305	\| `cpu_user_zscore` \| Detect anomalies \| `(current - mean) / stddev` \|
1306
1307	Z-Score Interpretation:
1308
1309	- ±2: Moderately unusual (warning)
1310	- ±3: Highly unusual (critical)
1311	- Negative: Below normal
1312	- Positive: Above normal
1313
1314	<br/>
1315	</details>
1316
1317	<details>
1318	<summary><strong>Example 6: Machine Learning Anomaly Detection</strong></summary><br/>
1319
1320	Scenario: Use Netdata's built-in ML for chart-level anomaly detection.
1321
1322	Why This Matters: Detect complex patterns across multiple metrics without manual threshold tuning.
1323
1324	```text
1325	template: ml_5min_cpu_chart
1326	on: system.cpu
1327	lookup: average -5m anomaly-bit of *
1328	calc: $this
1329	units: %
1330	every: 30s
1331	warn: $this > (($status >= $WARNING) ? (5) : (20))
1332	crit: $this >= (($status == $CRITICAL) ? (20) : (100))
1333	info: rolling 5min anomaly rate for system.cpu chart
1334	```
1335
1336	How It Works:
1337
1338	\| Component \| Purpose \| This Example \|
1339	\| ------------- \| ------------------------------------------ \| ----------------------------------- \|
1340	\| `lookup` \| Average anomaly rate across CPU dimensions \| 5-minute rolling window \|
1341	\| Hysteresis \| Prevent alert flapping \| Warning: 20%→5%, Critical: 100%→20% \|
1342	\| `anomaly-bit` \| ML-generated anomaly indicators \| 0 (normal) or 100 (anomalous) \|
1343
1344	<br/>
1345	</details>
1346
1347	<details>
1348	<summary><strong>Example 7: Node-Level ML Monitoring</strong></summary><br/>
1349
1350	Scenario: Monitor overall system health using ML across all metrics.
1351
1352	Why This Matters: Get a holistic view of system anomalies beyond individual charts.
1353
1354	```text
1355	template: ml_5min_node
1356	on: anomaly_detection.anomaly_rate
1357	lookup: average -5m of anomaly_rate
1358	calc: $this
1359	units: %
1360	every: 30s
1361	warn: $this > (($status >= $WARNING) ? (5) : (20))
1362	crit: $this >= (($status == $CRITICAL) ? (20) : (100))
1363	info: rolling 5min anomaly rate for all ML enabled dims
1364	```
1365
1366	Key Differences from Chart-Level:
1367
1368	- Uses `anomaly_detection.anomaly_rate` chart
1369	- Monitors `anomaly_rate` dimension
1370	- Covers all ML-enabled dimensions across the node
1371
1372	<br/>
1373	</details><br/>
1374
1375	<details>
1376	<summary><strong>Example 8: Boolean / Binary Metric Alerting</strong></summary><br/>
1377
1378	Scenario: Monitor a boolean 0/1 health-check gauge and choose the right aggregation method for your alerting intent.
1379
1380	Why This Matters: Boolean metrics require different aggregation strategies depending on whether you need to detect any single failure, confirm a sustained outage, or check the current state. Choosing the wrong method leads to missed alerts or alert noise.
1381
1382	Approach 1: Detect Any Failure Event (average)
1383
1384	```text
1385	alarm: service_failure_event
1386	on: my_service.health_status
1387	lookup: average -10s of health_status
1388	every: 10s
1389	warn: $this > 0
1390	info: any failure detected in the last 10 seconds
1391	to: sysadmin
1392	```
1393
1394	Use when the metric acts as a failure indicator — the value is 0 normally and 1 when a failure occurs. `average` over a short window naturally reflects any non-zero sample: if the metric was 1 at any point, the average will be greater than 0. This is the same pattern used by Netdata's Docker container health monitoring (`average -10s of unhealthy`, `warn: $this > 0`).
1395
1396	:::note
1397
1398	Do not use `sum` for boolean 0/1 gauges. While `sum -5m unaligned absolute` would technically detect failures (any non-zero sample makes the sum positive), `sum` produces a count of seconds in state 1 rather than an intuitive threshold. Use `sum` only for counter/cumulative metrics like packet drops or error totals — see [Example 4: Network Packet Drops](#example-4-network-packet-drops) for a correct `sum` use case.
1399
1400	:::
1401
1402	Approach 2: Detect Any Downtime (min) or Continuous Outage (max)
1403
1404	```text
1405	alarm: service_any_downtime
1406	on: my_service.health_status
1407	lookup: min -5m unaligned
1408	every: 10s
1409	crit: $this == 0
1410	info: metric dropped to 0 at some point in the last 5 minutes
1411	to: sysadmin
1412	```
1413
1414	Use when the metric is 1 = healthy and 0 = unhealthy. `min` returns the lowest value in the window — if the metric dropped to 0 at any point, the alert fires. This catches even brief outages.
1415
1416	For the stricter check of continuous outage (metric was never 1), use `max`:
1417
1418	```text
1419	alarm: service_continuous_outage
1420	on: my_service.health_status
1421	lookup: max -5m unaligned
1422	every: 10s
1423	crit: $this == 0
1424	info: service was down for the entire last 5 minutes
1425	to: sysadmin
1426	```
1427
1428	`max` returns the highest value in the window. If `max == 0`, the metric never reached 1 — the service was down the entire time.
1429
1430	Approach 3: Measure Failure Rate (average)
1431
1432	```text
1433	alarm: service_failure_rate
1434	on: my_service.health_status
1435	lookup: average -5m unaligned of health_status
1436	every: 1m
1437	warn: $this > 0.1
1438	crit: $this > 0.5
1439	info: failure rate exceeded threshold over the last 5 minutes
1440	to: sysadmin
1441	```
1442
1443	When the metric is 0 = healthy and 1 = failure, `average` over the window returns a value between 0.0 and 1.0 representing the fraction of time spent in failure. `warn: $this > 0.1` fires when the service was failing more than 10% of the time, and `crit: $this > 0.5` fires when failures exceeded half the window. This is useful for SLO-style alerting where occasional failures are acceptable.
1444
1445	:::note
1446	Note on `percentage`: The `percentage` option calculates each dimension's share of the chart total — it is designed for multi-dimension charts like `system.ram` (see [Task 3: Create a Simple Alert](#task-3-create-a-simple-alert): `lookup: average -1m percentage of used`). For a single-dimension boolean gauge, `percentage` always returns 100. Use plain `average` and compare against 0.0–1.0 thresholds instead.
1447	:::
1448
1449	Approach 4: Instant State Check (calc, no lookup)
1450
1451	```text
1452	alarm: service_current_state
1453	on: my_service.health_status
1454	calc: $health_status
1455	every: 10s
1456	crit: $this == 0
1457	info: service is currently down
1458	to: sysadmin
1459	delay: down 5m
1460	```
1461
1462	Use to check only the current value without time-window aggregation. The `calc: $health_status` references the chart dimension directly — no `lookup` needed. Note that `$status` is a built-in alert variable (the alert's own status code, −2 to 3) and must not be used here; use the dimension name instead (e.g. `$health_status` for a dimension named `health_status`). The `delay: down 5m` debounces recovery notifications, requiring the alert to stay clear for 5 minutes before sending recovery. This is the same pattern used in `health.d/timex.conf` for clock sync state monitoring (`calc: $state`).
1463
1464	Comparison: Which Method to Use
1465
1466	\| Intent \| Method \| Lookup / Calc \| Condition \| Fires When \|
1467	\| ------------------------------------------------------- \| ------ \| -------------------------- \| ------------ \| ----------------------------------------- \|
1468	\| Any failure event (metric is 0 normally, 1 on failure) \| `average` \| `average -10s of health_status` \| `$this > 0` \| Metric was non-zero at any point in the window \|
1469	\| Any downtime (metric is 1=healthy, 0=down) \| `min` \| `min -5m unaligned` \| `$this == 0` \| Metric hit 0 at any point in the window \|
1470	\| Continuous outage (metric is 1=healthy, 0=down) \| `max` \| `max -5m unaligned` \| `$this == 0` \| Metric was 0 for the entire window \|
1471	\| Failure rate over time \| `average` \| `average -5m unaligned of health_status` \| `$this > 0.N` \| Failure fraction exceeds threshold (0.0–1.0) \|
1472	\| Current state only \| `calc` \| `calc: $health_status` (no lookup) \| `$this == 0` \| Current value is 0 (debounce with delay) \|
1473
1474	For a full list of available lookup methods and processing options (`average`, `min`, `max`, `sum`, `percentage`, `absolute`, etc.), see the [Alert Line `lookup`](#alert-line-lookup) section.
1475
1476	Key Points:
1477
1478	- Boolean 0/1 metrics work with all standard lookup methods — the choice depends on your alerting intent
1479	- Use `average` over a short window for failure detection (`average -10s of <dimension>`, `warn: $this > 0`) — the same pattern Netdata uses in its own health configs (e.g., `health.d/docker.conf`)
1480	- Use `min` for "was it ever down?" and `max` for "was it continuously down?"
1481	- Use `average` for SLO-style failure-rate alerting (returns 0.0–1.0 fraction of time in failure state; compare against decimal thresholds)
1482	- Use `calc` without `lookup` for instant state checks, combined with `delay` for debouncing
1483	- Avoid `sum` on boolean gauges — it produces a count of seconds in state 1, not an intuitive threshold. Use `sum` only for counter/cumulative metrics (e.g., total packet drops in a time window)
1484
1485	Variables Used:
1486
1487	- `$this` — Result of the `lookup` or `calc` expression
1488	- `$health_status` — Dimension value from the chart (used in the `calc` approach; the variable name matches the dimension name, e.g. `health_status`)
1489
1490	<br/>
1491	</details><br/>
1492
1493	Next Steps: Having trouble with your alerts? Continue to [Troubleshooting](#troubleshooting) for debugging techniques.
1494
1495	## Troubleshooting
1496
1497	:::tip
1498
1499	What You'll Learn
1500
1501	How to debug alert issues, understand why alerts aren't working, and get detailed information about alert processing.
1502
1503	:::
1504
1505	### Find Chart and Context Information
1506
1507	Finding Chart Names:
1508	You can find chart information in two places:
1509
1510	\| Method \| URL \| Contains \|
1511	\| ------------- \| --------------------------------- \| ----------------- \|
1512	\| Configuration \| `http://NODE:19999/netdata.conf` \| All chart details \|
1513	\| API \| `http://NODE:19999/api/v1/charts` \| JSON chart data \|
1514
1515	Replace `NODE` with your server's IP address or hostname.
1516
1517	### Analyze Alert Expressions
1518
1519	Why This Matters: Understand how Netdata interprets your expressions and what values are being calculated.
1520
1521	Check Alert Processing:
1522	Visit `http://NODE:19999/api/v1/alarms?all` to see:
1523
1524	- Original expression as written in config
1525	- Parsed expression with added parentheses showing evaluation flow
1526	- Current alert status and values
1527	- Available variables and their values
1528
1529	Expression Evaluation Flow:
1530	Netdata adds parentheses to show how it evaluates your expressions:
1531
1532	Your Expression:
1533
1534	```text
1535	warn: $this > 80 and $status >= $WARNING
1536	```
1537
1538	Netdata's Interpretation:
1539
1540	```text
1541	warn: (($this > 80) and ($status >= $WARNING))
1542	```
1543
1544	### Troubleshooting Decision Trees
1545
1546	:::note
1547
1548	How to Use These Decision Trees
1549
1550	Follow the flowcharts below to systematically diagnose and resolve alert issues. Each path leads to specific solutions with step-by-step instructions.
1551
1552	:::
1553
1554	#### Decision Tree: Alert Not Working
1555
1556	```mermaid
1557	flowchart TD
1558	A("Alert Not Working") --> B("Check Configuration")
1559	A --> C("Check Data Source")
1560	A --> D("Check Thresholds")
1561
1562	B --> B1("Syntax errors<br/>Reload config<br/>Review logs")
1563	C --> C1("Chart exists<br/>Data collection<br/>Variable names")
1564	D --> D1("Adjust values<br/>Review logic<br/>Test expressions")
1565
1566	%% Style definitions
1567	classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1568	classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1569	classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1570	classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1571
1572	%% Apply styles
1573	class A alert
1574	class B,C,D database
1575	class B1,C1,D1 complete
1576	```
1577
1578	#### Decision Tree: Alert Always Triggering
1579
1580	```mermaid
1581	flowchart TD
1582	A("Alert Always Triggering") --> B("Check Values")
1583	A --> C("Check Logic")
1584	A --> D("Check Variables")
1585
1586	B --> B1("Current metrics<br/>Threshold comparison<br/>Units verification")
1587	C --> C1("Operator direction<br/>Expression syntax<br/>Status conditions")
1588	D --> D1("Variable names<br/>Dimension availability<br/>API verification")
1589
1590	%% Style definitions
1591	classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1592	classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1593	classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1594	classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1595
1596	%% Apply styles
1597	class A alert
1598	class B,C,D database
1599	class B1,C1,D1 complete
1600	```
1601
1602	#### Decision Tree: Alert Flapping
1603
1604	```mermaid
1605	flowchart TD
1606	A("Alert Flapping") --> B("Add Hysteresis")
1607	A --> C("Smooth Data")
1608	A --> D("Adjust Timing")
1609
1610	B --> B1("Conditional thresholds<br/>Different up/down values<br/>Status-based logic")
1611	C --> C1("Increase lookup time<br/>Use averages<br/>Reduce noise")
1612	D --> D1("Increase intervals<br/>Add delays<br/>Reduce frequency")
1613
1614	%% Style definitions
1615	classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1616	classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1617	classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1618	classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
1619
1620	%% Apply styles
1621	class A alert
1622	class B,C,D database
1623	class B1,C1,D1 complete
1624	```
1625
1626	### Common Issues and Solutions
1627
1628	#### Issue: Alert Not Triggering
1629
1630	Possible Causes:
1631
1632	\| Problem \| Check This \| Solution \|
1633	\| -------------------- \| ------------------------------- \| ----------------------------- \|
1634	\| Wrong chart name \| `on:` line matches actual chart \| Use chart ID from dashboard \|
1635	\| Incorrect dimensions \| Dimension names in `lookup` \| Check available dimensions \|
1636	\| Missing data \| Chart has recent data \| Verify data collection \|
1637	\| Expression errors \| Variables resolve correctly \| Use `/api/v1/alarm_variables` \|
1638
1639	#### Issue: Alert Always Triggering
1640
1641	Possible Causes:
1642
1643	\| Problem \| Check This \| Solution \|
1644	\| ------------------------- \| ---------------------------------------- \| ----------------------- \|
1645	\| Wrong threshold direction \| `>` vs `<` in expressions \| Review logic \|
1646	\| Units mismatch \| Comparing percentages to absolute values \| Check calculation units \|
1647	\| Variable name errors \| `$this` vs `$chart.dimension` \| Verify variable names \|
1648
1649	#### Issue: Alert Flapping
1650
1651	Possible Causes:
1652
1653	\| Problem \| Solution \|
1654	\| --------------------- \| ------------------------------------------------------------ \|
1655	\| Values near threshold \| Implement [hysteresis](#conditional-operator-for-hysteresis) \|
1656	\| Noisy data \| Increase lookup time window \|
1657	\| Too frequent checks \| Increase `every:` interval \|
1658
1659	Hysteresis Example:
1660
1661	```text
1662	# Instead of simple threshold
1663	warn: $this > 80
1664
1665	# Use hysteresis
1666	warn: $this > (($status >= $WARNING) ? (75) : (80))
1667	```
1668
1669	#### Issue: Variables Not Found
1670
1671	Debug Steps:
1672
1673	1. Check Available Variables:
1674
1675	```
1676	http://NODE:19999/api/v1/alarm_variables?chart=CHART_NAME
1677	```
1678
1679	2. Verify Chart Context:
1680	- For `alarm:` use chart name
1681	- For `template:` use chart context
1682
1683	3. Check Variable Syntax:
1684	- Chart local: `$dimension_name`
1685	- Host variables: `$chart_name.dimension_name`
1686	- Special variables: `$this`, `$now`, `$status`
1687
1688	### Testing Alert Changes
1689
1690	Safe Testing Process:
1691
1692	1. Create Test File:
1693
1694	```bash
1695	sudo touch health.d/test-alert.conf
1696	sudo ./edit-config health.d/test-alert.conf
1697	```
1698
1699	2. Write Simple Alert:
1700
1701	```text
1702	alarm: test_ram
1703	on: system.ram
1704	lookup: average -1m percentage of used
1705	every: 10s
1706	war: $this > 50 # Low threshold for testing
1707	info: Test alert - safe to ignore
1708	```
1709
1710	3. Reload and Monitor:
1711
1712	```bash
1713	sudo netdatacli reload-health
1714	# Watch dashboard for test alert appearance
1715	```
1716
1717	4. Remove When Done:
1718
1719	```bash
1720	sudo rm health.d/test-alert.conf
1721	sudo netdatacli reload-health
1722	```
1723
1724	### Performance Considerations
1725
1726	Alert Impact on System:
1727
1728	\| Factor \| Impact \| Optimization \|
1729	\| -------------------------- \| ------------------- \| ---------------------------------- \|
1730	\| Check frequency (`every:`) \| CPU usage \| Use appropriate intervals \|
1731	\| Lookup timeframe \| Memory/CPU \| Don't use excessively long periods \|
1732	\| Number of alerts \| Overall performance \| Disable unused alerts \|
1733	\| Complex expressions \| CPU per check \| Simplify where possible \|
1734
1735	Recommended Frequencies:
1736
1737	\| Alert Type \| Suggested Frequency \| Reason \|
1738	\| ----------------------- \| ------------------- \| -------------------------------------- \|
1739	\| Critical system metrics \| 10-30s \| Quick response needed \|
1740	\| Resource usage \| 1-5m \| Trends matter more than instant values \|
1741	\| Predictive alerts \| 15m-1h \| Based on longer-term patterns \|
1742
1743	### Getting Help
1744
1745	Information to Provide:
1746
1747	When seeking help, include:
1748
1749	1. Alert Configuration:
1750
1751	```text
1752	# Your complete alert definition
1753	```
1754
1755	2. Chart Information:
1756
1757	```
1758	http://your-server:19999/api/v1/alarm_variables?chart=chart_name
1759	```
1760
1761	3. Current Status:
1762
1763	```
1764	http://your-server:19999/api/v1/alarms?all
1765	```
1766
1767	Community Resources:
1768
1769	- [Netdata GitHub Issues](https://github.com/netdata/netdata/issues)
1770	- [Netdata Community Forum](https://community.netdata.cloud)
1771	- [Netdata Discord](https://discord.com/invite/2mEmfW735j)
1772
1773	### Alert Notification Variables
1774
1775	The following variables are available in alert notification templates and custom notification scripts:
1776
1777	\| Variable name \| Description \|
1778	\| :-------------------------: \| :------------------------------------------------------------------------------------------------------------------------------------------- \|
1779	\| `${alarm}` \| Like "name = value units" \|
1780	\| `${status_message}` \| Like "needs attention", "recovered", "is critical" \|
1781	\| `${severity}` \| Like "Escalated to CRITICAL", "Recovered from WARNING" \|
1782	\| `${raised_for}` \| Like "(alarm was raised for 10 minutes)" \|
1783	\| `${host}` \| The host generated this event \|
1784	\| `${url_host}` \| Same as `${host}` but URL encoded \|
1785	\| `${unique_id}` \| The unique id of this event \|
1786	\| `${alarm_id}` \| The unique id of the alarm that generated this event \|
1787	\| `${event_id}` \| The incremental id of the event, for this alarm id \|
1788	\| `${when}` \| The timestamp this event occurred \|
1789	\| `${date}` \| The date and time the event occurred (local timezone) \|
1790	\| `${date_utc}` \| The date and time the event occurred (UTC) \|
1791	\| `${name}` \| The name of the alarm, as given in netdata health.d entries \|
1792	\| `${url_name}` \| Same as `${name}` but URL encoded \|
1793	\| `${chart}` \| The name of the chart (type.id) \|
1794	\| `${url_chart}` \| Same as `${chart}` but URL encoded \|
1795	\| `${status}` \| The current status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL \|
1796	\| `${old_status}` \| The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL \|
1797	\| `${value}` \| The current value of the alarm \|
1798	\| `${old_value}` \| The previous value of the alarm \|
1799	\| `${src}` \| The line number and file the alarm has been configured \|
1800	\| `${duration}` \| The duration in seconds of the previous alarm state \|
1801	\| `${duration_txt}` \| Same as `${duration}` for humans \|
1802	\| `${non_clear_duration}` \| The total duration in seconds this is/was non-clear. For repeating alerts in WARNING or CRITICAL state, Netdata sends `${duration}` instead. \|
1803	\| `${non_clear_duration_txt}` \| Same as `${non_clear_duration}` for humans \|
1804	\| `${units}` \| The units of the value \|
1805	\| `${info}` \| A short description of the alarm \|
1806	\| `${value_string}` \| Friendly value (with units) \|
1807	\| `${old_value_string}` \| Friendly old value (with units) \|
1808	\| `${image}` \| The URL of an image to represent the status of the alarm \|
1809	\| `${color}` \| A color in #AABBCC format for the alarm \|
1810	\| `${goto_url}` \| The URL the user can click to see the netdata dashboard \|
1811	\| `${calc_expression}` \| The expression evaluated to provide the value for the alarm \|
1812	\| `${calc_param_values}` \| The values of the variables in the evaluated expression \|
1813	\| `${total_warnings}` \| The total number of alarms in WARNING state on the host \|
1814	\| `${total_critical}` \| The total number of alarms in CRITICAL state on the host \|
1815
1816	## Related Pages
1817
1818	- [Alerts Automation](/docs/netdata-ai/alerts-automation/alerts-automation.md) - Create and tune alerts using natural language with AI assistance (no manual configuration needed)
1819	- [Alerts Configuration Manager](/docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md) - Visual UI wizard for creating alerts
1820	- [ML Anomaly Detection](/docs/ml-ai/ml-anomaly-detection/ml-anomaly-detection.md) - Machine learning based anomaly detection for all metrics
1821	- [Alert Troubleshooting](/docs/troubleshooting/troubleshoot.md) - AI-powered alert investigation and root-cause analysis
1822
1823	Next Steps: You now have comprehensive knowledge of Netdata health configuration. Start with the [Quick Start Guide](#quick-start-guide) for immediate needs or dive into [Common Tasks](#common-tasks) for specific workflows.