improved alerting docs (#20891)
* improved alerting docs * Apply suggestion from @Ancairon * Apply suggestion from @Ancairon * Apply suggestion from @Ancairon * Fix links and typos in doc, also format * Update README.md removed duplicate turned passive voice to active --------- Co-authored-by: Fotis Voutsas <fotis@netdata.cloud> Co-authored-by: Kanela <kanela@netdata.cloud>
Costa Tsaousis committed
Aug 29, 2025 at 16:08 UTC
73990c55ad33714d75106d5a38d1c17ce0f22f9c
2 files changed
+440
-142
src/health/README.md
+439
-141
@@ -1,181 +1,479 @@
1
# Alerts and Notifications in Netdata
2
3
-Netdata provides two ways to send alert notifications. You can use either one or both at the same time.
3
+## Introduction to Alerting
4
5
-:::tip
5
+Netdata provides a **distributed, real-time health monitoring framework** that evaluates conditions against your metrics and executes actions on state transitions. You can configure notifications as one of these actions.
6
7
-Alerts are based on each node's health status. You can change thresholds, add new alerts, or silence-specific ones using Netdata's alerting system.
7
+Unlike traditional monitoring systems, Netdata evaluates alerts simultaneously at multiple levels - on the edge (Agents), at aggregation points (Parents), and deduplicates them in Netdata Cloud. This allows your teams to implement different alerting strategies at different infrastructure levels.
8
9
-[See how to configure alerts](/src/health/REFERENCE.md)
9
+### Understanding Alerts in Netdata
10
11
+Netdata alerts function as **component-level watchdogs**. You attach them to specific components/instances (network interfaces, database instances, web servers, containers, processes) where they evaluate metrics at configurable intervals.
12
+
13
+To simplify your configuration, you can define alert templates once and apply them to all matching components. The system matches instances by host labels, instance labels, and names, allowing you to define the same alert multiple times with different matching criteria.
14
+
15
+Each alert provides a name, value, unit, and status - making them easy to display in dashboards and send as meaningful notifications regardless of your infrastructure's complexity.
16
+
17
+### Where Your Alerts Run
18
+
19
+Your alerts evaluate at the edge. Every Netdata Agent and Parent runs alerts on the metrics it processes and stores (enabled by default, but you can disable alerting at any level). When you stream metrics to a Parent, the Parent evaluates its own alerts on those metrics independently of the child's alerts. Each Agent maintains its own alert configuration and evaluates alerts autonomously. Metric streaming doesn't propagate alert configurations or transitions to Parents.
20
+
21
+```
22
+┌─────────┐ Metrics ┌──────────┐ Metrics ┌──────────┐
23
+│ Child │ ───────────────> │ Parent 1 │ ───────────────> │ Parent 2 │
24
+│ Agent │ of child │ Agent │ of child + │ Agent │
25
+└────┬────┘ └────┬─────┘ Parent 1 └────┬─────┘
26
+ │ │ │
27
+ │ Evaluates alerts on │ Evaluates alerts on │ Evaluates alerts on
28
+ │ local metrics │ child + local metrics │ all streamed + local
29
+ │ │ │
30
+ ▼ ▼ ▼
31
+ Alerts Alerts Alerts
32
+```
33
+
34
+### Alert Actions and Notifications
35
+
36
+Your Netdata Agents treat notifications as **actions** triggered by alert status transitions. Agents can dispatch notifications or perform automation tasks like scaling services, restarting processes, or rotating logs. Actions are shell scripts or executable programs that receive all alert transition metadata from Netdata.
37
+
38
+When you claim Agents to Netdata Cloud, they send their alert configurations and transitions to Cloud, which deduplicates them (merging multiple transitions from different Agents for the same host). Netdata Cloud triggers notifications centrally through its integrations (Slack, Microsoft Teams, Amazon SNS, PagerDuty, OpsGenie).
39
+
40
+Netdata Cloud's intelligent deduplication works by:
41
+- **Consolidating multiple Agents** reporting the same alert
42
+- **Prioritizing highest severity**: CRITICAL > WARNING > CLEAR
43
+- **Creating unique keys**: Alert name + Instance + Node
44
+
45
+Your Agents and Netdata Cloud trigger actions independently using their own configurations and integrations.
46
+
47
+This design enables you to:
48
+1. **Maintain team independence**: Different teams run their own Parents with custom alerts
49
+2. **Implement edge intelligence**: Critical alerts trigger automations directly on nodes
50
+3. **Scale naturally**: Alert evaluation distributes with your infrastructure
51
+4. **Mix strategies**: Combine edge, regional, and central alerting
52
+
53
+### Quick Example
54
+
55
+```yaml
56
+Web Server (Child):
57
+ - Alert: system CPU > 80% triggers scale out
58
+ - Alert: process X memory > 90% restarts process X
59
+
60
+DevOps Parent:
61
+ - Alert: Response time > 500ms across all web servers
62
+ - Alert: Error rate > 1% for any service
63
+
64
+SRE Parent:
65
+ - Alert: Anomaly detection on traffic patterns
66
+ - Alert: Capacity planning thresholds
67
+
68
+Netdata Cloud:
69
+ - Receives all alert transitions
70
+ - Deduplicates overlapping alerts
71
+ - Shows CRITICAL if any instance reports CRITICAL
72
+ - Provides unified view for incident response
73
+```
74
+
75
+Each level operates independently while Netdata Cloud provides a coherent, deduplicated view of your entire infrastructure's health (when all agents connect directly to Cloud).
76
+
77
+## Managing Alert Configuration
78
+
79
+You configure Netdata alerts in 3 layers:
80
+
81
+1. **Stock Alerts**: Netdata provides hundreds of alert definitions in `/usr/lib/netdata/conf.d/health.d` to detect common issues. Don't edit these directly - updates will overwrite your changes.
82
+2. **Your Custom Alerts**: Create your own definitions in `/etc/netdata/health.d`.
83
+3. **Dynamic UI Configuration**: Use Netdata dashboards to edit, add, enable, or disable alerts on any node through the streaming transport.
84
+
85
+## Managing Notification Configuration
86
+
87
+You can configure notifications for any infrastructure node at 3 levels:
88
+
89
+| Level | What It Evaluates | Where Notifications Come From | Use Case | Documentation |
90
+|--------------------|-----------------------------|--------------------|-----------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
91
+| **Netdata Agent** | Local Metrics | Netdata Agent | Edge automation | [Agent integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications) |
92
+| **Netdata Parent** | Local and Children Metrics | Netdata Parent | Edge automation | [Agent integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications) |
93
+| **Netdata Cloud** | Receives Transitions | Netdata Cloud | Web-hooks, role/room based | [Cloud integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/centralized-cloud-notifications) |
94
+
95
+:::note
96
+When using Parents and Cloud with default settings, you may receive duplicate email notifications. Agents send emails by default when an MTA exists on their systems. Disable email notifications on Agents and Parents when using Cloud by setting `SEND_EMAIL="NO"` in `/etc/netdata/health_alarm_notify.conf` [using `edit-config`](/docs/netdata-agent/configuration/README.md).
97
:::
98
13
-## How Alert Notifications Work
99
+### Best Practices for Large Deployments
100
+
101
+#### Central Alerting Strategy
102
15
-| Method | Where Alerts Are Sent From | Customization | Highlights |
16
-|-------------------|----------------------------|---------------|-----------------------------------------------------------|
17
-| **Netdata Cloud** | Cloud UI | Medium | Centralized alerting using connected nodes' health status |
18
-| **Netdata Agent** | Local Netdata Agent | High | Node-level alerting with wide integration support |
103
+When you:
104
+- Don't need edge automation (no scripts reacting to alerts)
105
+- Use highly available Parents for all nodes
106
+- Use Netdata Cloud (at least for Parents)
107
20
-You can enable one or both methods depending on your needs.
108
+Follow these steps:
109
+1. Disable health monitoring on child nodes
110
+2. Share the same alert configuration across Parents (use git repo or CI/CD)
111
+3. Disable Parent notifications (`SEND_EMAIL="NO"` in `/etc/netdata/health_alarm_notify.conf`)
112
+4. Keep only Cloud notifications
113
22
-## Quick Start
114
+This emulates traditional monitoring tools where you configure alerts centrally and dispatch notifications centrally.
115
24
-Use this table to choose and set up your preferred alerting method:
116
+#### Edge Flexible Alerting Strategy
117
26
-| Option | Setup Location | Setup Effort | Best For |
27
-|-------------------|----------------------|--------------|-------------------------------|
28
-| **Netdata Cloud** | In the Cloud UI | Low | Teams managing multiple nodes |
29
-| **Netdata Agent** | On each Netdata node | Medium | Full control and flexibility |
118
+When you:
119
+- Need edge automation (scale out, restart processes)
120
+- Use Parents
121
+- Use Cloud for all nodes
122
+
123
+Follow these steps:
124
+- Disable stock alerts on children (`enable stock health configuration` to `no` in `/etc/netdata/netdata.conf` `[health]` section)
125
+- Configure only automation-required alerts on children
126
+- Keep stock alerts on Parents but disable notifications (`SEND_EMAIL="NO"`)
127
+- Keep only Cloud notifications
128
+
129
+This enables edge automation on children while maintaining central alerting control and deduplicated Cloud notifications.
130
131
## Set Up Alerts via Netdata Cloud
132
133
1. Connect your nodes to [Netdata Cloud](https://app.netdata.cloud/)
34
-2. In the UI, go to: `Space → Notifications`
35
-3. Choose an integration (e.g. Slack, Amazon SNS, Splunk)
36
-4. Set alert severity filters as needed
134
+2. Navigate to: `Space → Notifications`
135
+3. Choose your integration (Slack, Amazon SNS, Splunk)
136
+4. Configure alert severity filters
137
38
-[See all supported Cloud integrations](/docs/alerts-&-notifications/notifications/centralized-cloud-notifications)
138
+[View all Cloud integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/centralized-cloud-notifications)
139
140
## Set Up Alerts via Netdata Agent
141
42
-1. Open the notification config:
43
-
142
+1. Open notification config:
143
```bash
144
sudo ./edit-config health_alarm_notify.conf
145
```
146
48
-2. Enable your preferred method, for example, email:
49
-
147
+2. Enable your method (example: email):
148
```ini
149
SEND_EMAIL="YES"
150
DEFAULT_RECIPIENT_EMAIL="you@example.com"
151
```
152
55
-3. Ensure your system can send mail (via `sendmail`, SMTP relay, etc.)
56
-4. Restart the agent:
153
+3. Verify your system can send mail (sendmail, SMTP relay)
154
155
+4. Restart the agent:
156
```bash
157
sudo systemctl restart netdata
158
```
159
62
-[See all Agent-based integrations](/docs/alerts-&-notifications/notifications/agent-dispatched-notifications)
63
-
64
-## About the Agent's Health Monitoring
65
-
66
-The Netdata Agent continuously monitors system health and performance. It offers:
67
-
68
-- Hundreds of pre-configured alerts covering system, app, and service metrics
69
-- No setup is required to work out of the box
70
-- Dynamic customization you can fully control how, when, and what triggers an alert
71
-
72
-[See which collectors support alerts](/src/collectors/COLLECTORS.md)
73
-
74
-## Alert Philosophy and Design
75
-
76
-### The Anatomy of Intelligent Alerts
77
-
78
-| Component | Description |
79
-|---------------------|-------------------------------------------------------------------------------------------------------------------------|
80
-| **Metrics Data** | Collected every second (and at "event frequency" for eBPF metrics), giving you the most precise foundation for alerting |
81
-| **Filtering** | Alerts run against specific metrics series, with support for pattern matching, labels, hostnames, and operating systems |
82
-| **Frequency** | Configurable check intervals based on how quickly you need to know about issues |
83
-| **Templates** | Write once, apply everywhere with dimension templates and the ability to apply logic to multiple charts |
84
-| **Calculation** | Convert raw metrics into human-readable formats, like percentages, for easier understanding |
85
-| **Thresholds** | Pre-configured values that determine when metrics are outside acceptable ranges |
86
-| **Hysteresis** | Prevention of "flapping" alerts for metrics that hover around thresholds |
87
-| **Severity Levels** | CLEAR, WARNING, and CRITICAL states to indicate the urgency of action needed |
88
-
89
-### Preventing False Positives and Alert Fatigue
90
-
91
-| Feature | Benefit |
92
-|---------------------------|----------------------------------------------------------------------------|
93
-| **Intelligent Defaults** | Thresholds are carefully selected based on real-world experience |
94
-| **Dynamic Thresholds** | Some alerts adapt to your system's baseline performance |
95
-| **Hysteresis Protection** | Prevents floods of notifications when metrics fluctuate around a threshold |
96
-| **Notification Delays** | Configurable delays ensure transient issues don't trigger alerts |
97
-| **Severity Separation** | Different notification channels for different alert severities |
98
-| **Role-Based Routing** | Ensures alerts reach only the appropriate stakeholders |
99
-
100
-## Troubleshooting Alerts
101
-
102
-Netdata provides several tools to help you understand and resolve alert notifications:
103
-
104
-<details>
105
-<summary><strong>Netdata Assistant</strong></summary><br/>
106
-
107
-The [Netdata Assistant](https://learn.netdata.cloud/docs/machine-learning-and-anomaly-detection/ai-powered-troubleshooting-assistant) is an AI-powered feature that guides you through troubleshooting alerts. When an alert triggers, you can:
108
-
109
-1. Click on the alert in your Netdata Cloud dashboard
110
-2. Press the Assistant button to open a floating window
111
-3. Receive customized information and troubleshooting tips specific to that alert
112
-
113
-```mermaid
114
-%%{init: {'theme': 'default', 'themeVariables': {
115
- 'primaryColor': '#2b2b2b',
116
- 'primaryTextColor': '#fff',
117
- 'primaryBorderColor': '#7C0000',
118
- 'lineColor': '#F8B229',
119
- 'secondaryColor': '#006100',
120
- 'tertiaryColor': '#333',
121
- 'fontFamily': 'arial',
122
- 'fontSize': '16px'
123
-}}}%%
124
-flowchart TD
125
- A[Alert Detected] -->|Analyze Context| B(Alert Assistant)
126
- style A fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
127
- style B fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
128
-
129
- B --> C[Explanation of Alert]
130
- B --> D[Possible Causes]
131
- B --> E[Troubleshooting Steps]
132
- B --> F[Documentation Links]
133
-
134
- style C fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
135
- style D fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
136
- style E fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
137
- style F fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
138
-```
139
-
140
-The Assistant window follows you as you navigate through Netdata dashboards, making it easy to reference while investigating the issue.
141
-</details>
142
-
143
-<details>
144
-<summary><strong>Community Resources</strong></summary><br/>
145
-
146
-For more complex alert troubleshooting, Netdata maintains a dedicated [Alerts Troubleshooting space](https://community.netdata.cloud/c/alerts/28) in our community forum.
147
-
148
-```mermaid
149
-%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
150
-flowchart TD
151
- A((Community<br>Forum)) --- B[Ask Questions]
152
- A --- C[Share Solutions]
153
- A --- D[Suggest Improvements]
154
- A --- E[Learn from Others]
155
- A --- F[Find Alert Info]
156
-
157
- style A fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:20,ry:20
158
- style B fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
159
- style C fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
160
- style D fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
161
- style E fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
162
- style F fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
163
-```
164
-
165
-You can also get help through our [GitHub repository](https://github.com/netdata/netdata) or [Discord channel](https://discord.gg/kUk3nCmbtx) if you prefer those platforms. After resolving an issue, consider sharing your experience in the forum to help other users.
166
-</details>
167
-
168
-<details>
169
-<summary><strong>Customizing Alerts</strong></summary><br/>
170
-
171
-You can tune alerts to match your environment by adjusting thresholds, writing custom alert conditions, silencing alerts temporarily or permanently, and using statistical functions for smarter alerting.
172
-
173
-[Customize alerts](/src/health/REFERENCE.md)
174
-[Silence or disable alerts](/src/health/REFERENCE.md#disable-or-silence-alerts)
175
-</details>
160
+[View all Agent integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications)
161
+
162
+## Core Alerting Concepts
163
+
164
+Netdata supports two alert types:
165
+- **Alarms**: Attach to specific instances (specific network interface, database instance)
166
+- **Templates**: Apply to all matching instances (all network interfaces, all databases)
167
+
168
+### Alert Lifecycle and States
169
+
170
+Your alerts produce more than threshold checks. Each generates:
171
+- **A value**: Combines metrics or other alerts using time-series lookups and expressions
172
+- **A unit**: Makes alerts meaningful ("seconds", "%", "requests/s")
173
+- **A name**: Identifies the alert
174
+
175
+This enables sophisticated alerts like:
176
+- `out of disk space time: 450 seconds` - Predicts when disk fills based on current rate
177
+- `3xx redirects: 12.5 percent` - Calculates redirects as percentage of total
178
+- `response time vs yesterday: 150%` - Compares current to historical baseline
179
+
180
+### Alert States
181
+
182
+Your alerts exist in one of these states:
183
+
184
+| State | Description | Trigger |
185
+|-------------------|---------------------------------------------------|--------------------------------------------------------------------------|
186
+| **CLEAR** | Normal - conditions exist but not triggered | Warning and critical conditions evaluate to zero |
187
+| **WARNING** | Warning threshold exceeded | Warning condition evaluates to non-zero |
188
+| **CRITICAL** | Critical threshold exceeded | Critical condition evaluates to non-zero |
189
+| **UNDEFINED** | Cannot evaluate | No conditions defined, or value is NaN/Inf |
190
+| **UNINITIALIZED** | Never evaluated | Alert just created |
191
+| **REMOVED** | Alert deleted | Child disconnected, agent exit, or health reload |
192
+
193
+Alerts transition freely between states based on:
194
+- **Calculated value** (including NaN, Inf, or valid numbers)
195
+- **Warning/critical conditions** (evaluation results)
196
+- **External events** (disconnections, reloads, exits)
197
+
198
+Key behaviors:
199
+- Alerts jump directly from CLEAR to CRITICAL (no WARNING required)
200
+- WARNING and CRITICAL evaluate independently
201
+- Alerts return to appropriate state when data becomes available
202
+- CRITICAL takes precedence when both conditions are true
203
+
204
+### Alert Evaluation Process
205
+
206
+#### 1. Calculate Value
207
+
208
+Your alerts perform complex calculations:
209
+
210
+```
211
+ lookup calc warn,crit status
212
+ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────┐
213
+ │ Database │ │Expression│ │ Warning │ │ Execute │
214
+ │ Query │──────>│Processor │──────>│ Critical │ ───> │ Action on │
215
+ │(optional)│ $this │(optional)│ $this │ Checks │ │Transition │
216
+ └──────────┘ └──────────┘ └──────────┘ └───────────┘
217
+```
218
+
219
+Examples:
220
+```yaml
221
+# Simple threshold
222
+calc: $used
223
+# Result: $this = latest value of dimension 'used'
224
+
225
+# Time-series lookup
226
+lookup: average -1h of used
227
+# Result: $this = average of 'used' over last hour
228
+
229
+# Combined calculation
230
+lookup: average -1h of used
231
+calc: $this * 100 / $total
232
+# Result: $this = percentage of hourly average vs total
233
+
234
+# Baseline comparison
235
+lookup: average -1h of used
236
+calc: $this * 100 / $average_yesterday
237
+# Result: $this = percentage vs yesterday's average
238
+```
239
+
240
+#### 2. Evaluate Conditions
241
+
242
+After calculating, check conditions:
243
+
244
+```yaml
245
+# Simple conditions
246
+warn: $this > 80
247
+crit: $this > 90
248
+
249
+# Flapping prevention
250
+warn: ($status >= $WARNING) ? ($this > 50) : ($this > 80)
251
+crit: ($status == $CRITICAL) ? ($this > 70) : ($this > 90)
252
+
253
+# Complex conditions
254
+warn: $this > 80 AND $rate > 10
255
+crit: $this > 90 OR $failures > 5
256
+```
257
+
258
+#### 3. Determine State
259
+
260
+Each condition evaluates to:
261
+- NaN or Inf → UNDEFINED
262
+- Non-zero → RAISED
263
+- Zero → CLEAR
264
+
265
+Final status:
266
+- Critical RAISED → **CRITICAL** (priority)
267
+- Warning RAISED → **WARNING**
268
+- Either CLEAR → **CLEAR**
269
+- Both missing/UNDEFINED → **UNDEFINED**
270
+
271
+### Evaluation Timing
272
+
273
+Alert evaluation runs independently from data collection:
274
+
275
+```
276
+Data Collection Alert Evaluation
277
+ │ │
278
+ ▼ every 1s ▼ configurable interval
279
+ [Metrics] ──────────> [Alert Engine]
280
+ │
281
+ ▼
282
+ Query metrics,
283
+ Calculate values,
284
+ Check conditions
285
+```
286
+
287
+- **Default interval**: Query window duration (with lookup) or manual setting required
288
+- **Configurable**: Use `every` for custom intervals
289
+- **Constrained**: Cannot evaluate faster than data collection frequency
290
+
291
+### Anti-Flapping Mechanisms
292
+
293
+Netdata prevents alert flapping through:
294
+
295
+#### 1. Hysteresis
296
+```yaml
297
+warn: ($status < $WARNING) ? ($this > 80) : ($this > 50)
298
+```
299
+Triggers at 80, clears at 50, preventing flapping between 50-80.
300
+
301
+#### 2. Dynamic Delays
302
+Alerts transition immediately in dashboards but notifications use exponential backoff.
303
+
304
+#### 3. Duration Requirements
305
+```yaml
306
+lookup: average -10m of used
307
+warn: $this > 80
308
+```
309
+Requires 10 minutes of data before triggering.
310
+
311
+### Multi-Stage Alerts
312
+
313
+Create dependent alerts:
314
+
315
+```yaml
316
+# Stage 1: Baseline
317
+template: requests_average_yesterday
318
+ on: web_log.requests
319
+ lookup: average -1h at -1d
320
+ every: 10s
321
+
322
+# Stage 2: Current
323
+template: requests_average_now
324
+ on: web_log.requests
325
+ lookup: average -1h
326
+ every: 10s
327
+
328
+# Stage 3: Compare
329
+template: web_requests_vs_yesterday
330
+ on: web_log.requests
331
+ calc: $requests_average_now * 100 / $requests_average_yesterday
332
+ units: %
333
+ warn: $this > 150 || $this < 75
334
+ crit: $this > 200 || $this < 50
335
+```
336
+
337
+### Available Variables
338
+
339
+Variables resolve in order (first match wins):
340
+
341
+#### 1. Built-in Variables
342
+
343
+| Variable | Description | Value |
344
+|---------------------|---------------------------|--------------------|
345
+| `$this` | Current calculated value | Result from lookup/calc |
346
+| `$after` | Query start timestamp | Unix timestamp |
347
+| `$before` | Query end timestamp | Unix timestamp |
348
+| `$now` | Current time | Unix timestamp |
349
+| `$last_collected_t` | Last collection time | Unix timestamp |
350
+| `$update_every` | Collection frequency | Seconds |
351
+| `$status` | Current status code | -2 to 3 |
352
+| `$REMOVED` | Status constant | -2 |
353
+| `$UNINITIALIZED` | Status constant | -1 |
354
+| `$UNDEFINED` | Status constant | 0 |
355
+| `$CLEAR` | Status constant | 1 |
356
+| `$WARNING` | Status constant | 2 |
357
+| `$CRITICAL` | Status constant | 3 |
358
+
359
+#### 2. Dimension Values
360
+
361
+| Syntax | Description | Example |
362
+|------------------------------------|----------------------------------|------------------|
363
+| `$dimension_name` | Last normalized value | `$used` |
364
+| `$dimension_name_raw` | Last raw collected value | `$used_raw` |
365
+| `$dimension_name_last_collected_t` | Collection timestamp | `$used_last_collected_t` |
366
+
367
+```yaml
368
+template: disk_usage_percent
369
+ on: disk.space
370
+ calc: $used * 100 / ($used + $available)
371
+ units: %
372
+```
373
+
374
+#### 3. Chart Variables
375
+```yaml
376
+calc: $used > $threshold # If chart defines 'threshold'
377
+```
378
+
379
+#### 4. Host Variables
380
+```yaml
381
+warn: $connections > $max_connections * 0.8 # If host defines 'max_connections'
382
+```
383
+
384
+#### 5. Other Alerts
385
+```yaml
386
+# Alert 1
387
+template: cpu_baseline
388
+ calc: $system + $user
389
+
390
+# Alert 2
391
+template: cpu_check
392
+ calc: $system
393
+ warn: $this > $cpu_baseline * 1.5
394
+```
395
+
396
+#### 6. Cross-Context References
397
+```yaml
398
+template: disk_io_vs_iops
399
+ on: disk.io
400
+ calc: $reads / ${disk.iops.reads}
401
+ units: bytes per operation
402
+```
403
+
404
+### Variable Resolution and Label Scoring
405
+
406
+When alerts reference variables matching multiple instances, Netdata uses label similarity scoring:
407
+
408
+1. **Collect candidates** with matching names
409
+2. **Score by labels** - count common labels
410
+3. **Select best match** - highest label overlap
411
+
412
+Example: Alert on `disk.io` (labels: `device=sda`, `mount=/data`) references `${disk.iops.reads}`:
413
+- `disk.iops` for sda (labels match) → Score: 2
414
+- `disk.iops` for sdb (no match) → Score: 0
415
+Result: Uses sda's value
416
+
417
+### Missing Data Handling
418
+
419
+During lookups with missing data:
420
+- **All values NULL**: `$this` becomes `NaN`
421
+- **Some values exist**: Ignores NULL, continues calculation
422
+- **Dimension doesn't exist**: `$this` becomes `NaN`
423
+
424
+This handles intermittent collection, dynamic dimensions, and partial outages.
425
+
426
+### Evaluation Frequency
427
+
428
+Determine frequency by:
429
+
430
+1. **With lookup**: Defaults to window duration
431
+ ```yaml
432
+ lookup: average -5m # Evaluates every 5 minutes
433
+ ```
434
+
435
+2. **Without lookup**: Set explicitly
436
+ ```yaml
437
+ every: 10s
438
+ calc: $system + $user
439
+ ```
440
+
441
+3. **Custom interval**: Override default
442
+ ```yaml
443
+ lookup: average -1m
444
+ every: 10s # Check every 10s despite 1m window
445
+ ```
446
+
447
+Constraints:
448
+- Cannot exceed data collection frequency
449
+- High frequency impacts performance
450
+- Use larger intervals with `unaligned` for efficiency
451
+
452
+## Troubleshooting Your Alerts
453
+
454
+### Netdata Assistant
455
+
456
+The [Netdata Assistant](https://learn.netdata.cloud/docs/machine-learning-and-anomaly-detection/ai-powered-troubleshooting-assistant) provides AI-powered troubleshooting when alerts trigger:
457
+
458
+1. Click the alert in your dashboard
459
+2. Press the Assistant button
460
+3. Receive customized troubleshooting tips
461
+
462
+The Assistant window follows you through dashboards for easy reference while investigating.
463
+
464
+### Community Resources
465
+
466
+Visit our [Alerts Troubleshooting space](https://community.netdata.cloud/c/alerts/28) for complex issues. Get help through [GitHub](https://github.com/netdata/netdata) or [Discord](https://discord.gg/kUk3nCmbtx). Share your solutions to help others.
467
+
468
+### Customizing Alerts
469
+
470
+Tune alerts for your environment by adjusting thresholds, writing custom conditions, silencing alerts, and using statistical functions.
471
+
472
+- [Customize alerts](/src/health/REFERENCE.md)
473
+- [Silence or disable alerts](/src/health/REFERENCE.md#how-to-disable-or-silence-alerts)
474
475
## Related Documentation
476
179
-- [All alert notification methods](/docs/alerts-and-notifications/notifications/README.md)
477
+- [All notification methods](/docs/alerts-and-notifications/notifications/README.md)
478
- [Supported collectors](/src/collectors/COLLECTORS.md)
479
- [Full alert reference](/src/health/REFERENCE.md)
src/health/REFERENCE.md
+1
-1
@@ -892,7 +892,7 @@ How to write calculations and use variables in your alert definitions. Essential
892
|------------|----------------------------------------|---------------------------|
893
| Arithmetic | `+`, `-`, `*`, `/` | Numeric values |
894
| Comparison | `<`, `==`, `<=`, `<>`, `!=`, `>`, `>=` | `1` (true) or `0` (false) |
895
-| Logical | `&&`, ` | |`, `!`, `AND`, `OR`, `NOT` | `1` (true) or `0` (false) |
895
+| Logical | `&&`, `||`, `!`, `AND`, `OR`, `NOT` | `1` (true) or `0` (false) |
896
897
**Special Functions:**
898