| 1 | # Alerts and Notifications in Netdata |
| 2 | |
| 3 | ## Introduction to Alerting |
| 4 | |
| 5 | Netdata provides a **distributed, real-time health monitoring framework** that evaluates conditions against your metrics and executes actions on state transitions. You can configure notifications as one of these actions. |
| 6 | |
| 7 | Unlike traditional monitoring systems, Netdata evaluates alerts simultaneously at multiple levels - on the edge (Agents), at aggregation points (Parents), and deduplicates them in Netdata Cloud. This allows your teams to implement different alerting strategies at different infrastructure levels. |
| 8 | |
| 9 | ### Understanding Alerts in Netdata |
| 10 | |
| 11 | Netdata alerts function as **component-level watchdogs**. You attach them to specific components/instances (network interfaces, database instances, web servers, containers, processes) where they evaluate metrics at configurable intervals. |
| 12 | |
| 13 | To simplify your configuration, you can define alert templates once and apply them to all matching components. The system matches instances by host labels, instance labels, and names, allowing you to define the same alert multiple times with different matching criteria. |
| 14 | |
| 15 | Each alert provides a name, value, unit, and status - making them easy to display in dashboards and send as meaningful notifications regardless of your infrastructure's complexity. |
| 16 | |
| 17 | ### Where Your Alerts Run |
| 18 | |
| 19 | Your alerts evaluate at the edge. Every Netdata Agent and Parent runs alerts on the metrics it processes and stores (enabled by default, but you can disable alerting at any level). When you stream metrics to a Parent, the Parent evaluates its own alerts on those metrics independently of the child's alerts. Each Agent maintains its own alert configuration and evaluates alerts autonomously. Metric streaming doesn't propagate alert configurations or transitions to Parents. |
| 20 | |
| 21 | ``` |
| 22 | ┌─────────┐ Metrics ┌──────────┐ Metrics ┌──────────┐ |
| 23 | │ Child │ ───────────────> │ Parent 1 │ ───────────────> │ Parent 2 │ |
| 24 | │ Agent │ of child │ Agent │ of child + │ Agent │ |
| 25 | └────┬────┘ └────┬─────┘ Parent 1 └────┬─────┘ |
| 26 | │ │ │ |
| 27 | │ Evaluates alerts on │ Evaluates alerts on │ Evaluates alerts on |
| 28 | │ local metrics │ child + local metrics │ all streamed + local |
| 29 | │ │ │ |
| 30 | ▼ ▼ ▼ |
| 31 | Alerts Alerts Alerts |
| 32 | ``` |
| 33 | |
| 34 | ### Alert Actions and Notifications |
| 35 | |
| 36 | Your Netdata Agents treat notifications as **actions** triggered by alert status transitions. Agents can dispatch notifications or perform automation tasks like scaling services, restarting processes, or rotating logs. Actions are shell scripts or executable programs that receive all alert transition metadata from Netdata. |
| 37 | |
| 38 | When you claim Agents to Netdata Cloud, they send their alert configurations and transitions to Cloud, which deduplicates them (merging multiple transitions from different Agents for the same host). Netdata Cloud triggers notifications centrally through its integrations (Slack, Microsoft Teams, Amazon SNS, PagerDuty, OpsGenie). |
| 39 | |
| 40 | Netdata Cloud's intelligent deduplication works by: |
| 41 | - **Consolidating multiple Agents** reporting the same alert |
| 42 | - **Prioritizing highest severity**: CRITICAL > WARNING > CLEAR |
| 43 | - **Creating unique keys**: Alert name + Instance + Node |
| 44 | |
| 45 | Your Agents and Netdata Cloud trigger actions independently using their own configurations and integrations. |
| 46 | |
| 47 | This design enables you to: |
| 48 | 1. **Maintain team independence**: Different teams run their own Parents with custom alerts |
| 49 | 2. **Implement edge intelligence**: Critical alerts trigger automations directly on nodes |
| 50 | 3. **Scale naturally**: Alert evaluation distributes with your infrastructure |
| 51 | 4. **Mix strategies**: Combine edge, regional, and central alerting |
| 52 | |
| 53 | ### Quick Example |
| 54 | |
| 55 | ```yaml |
| 56 | Web Server (Child): |
| 57 | - Alert: system CPU > 80% triggers scale out |
| 58 | - Alert: process X memory > 90% restarts process X |
| 59 | |
| 60 | DevOps Parent: |
| 61 | - Alert: Response time > 500ms across all web servers |
| 62 | - Alert: Error rate > 1% for any service |
| 63 | |
| 64 | SRE Parent: |
| 65 | - Alert: Anomaly detection on traffic patterns |
| 66 | - Alert: Capacity planning thresholds |
| 67 | |
| 68 | Netdata Cloud: |
| 69 | - Receives all alert transitions |
| 70 | - Deduplicates overlapping alerts |
| 71 | - Shows CRITICAL if any instance reports CRITICAL |
| 72 | - Provides unified view for incident response |
| 73 | ``` |
| 74 | |
| 75 | Each level operates independently while Netdata Cloud provides a coherent, deduplicated view of your entire infrastructure's health (when all agents connect directly to Cloud). |
| 76 | |
| 77 | ## Managing Alert Configuration |
| 78 | |
| 79 | You configure Netdata alerts in 3 layers: |
| 80 | |
| 81 | 1. **Stock Alerts**: Netdata provides hundreds of alert definitions in `/usr/lib/netdata/conf.d/health.d` to detect common issues. Don't edit these directly - updates will overwrite your changes. |
| 82 | 2. **Your Custom Alerts**: Create your own definitions in `/etc/netdata/health.d`. |
| 83 | 3. **Dynamic UI Configuration**: Use Netdata dashboards to edit, add, enable, or disable alerts on any node through the streaming transport. |
| 84 | |
| 85 | ## Managing Notification Configuration |
| 86 | |
| 87 | You can configure notifications for any infrastructure node at 3 levels: |
| 88 | |
| 89 | | Level | What It Evaluates | Where Notifications Come From | Use Case | Documentation | |
| 90 | |--------------------|-----------------------------|--------------------|-----------------------------------|-----------------------------------------------------------------------------------------------------------------------------| |
| 91 | | **Netdata Agent** | Local Metrics | Netdata Agent | Edge automation | [Agent integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications) | |
| 92 | | **Netdata Parent** | Local and Children Metrics | Netdata Parent | Edge automation | [Agent integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications) | |
| 93 | | **Netdata Cloud** | Receives Transitions | Netdata Cloud | Web-hooks, role/room based | [Cloud integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/centralized-cloud-notifications) | |
| 94 | |
| 95 | :::note |
| 96 | When using Parents and Cloud with default settings, you may receive duplicate email notifications. Agents send emails by default when an MTA exists on their systems. Disable email notifications on Agents and Parents when using Cloud by setting `SEND_EMAIL="NO"` in `/etc/netdata/health_alarm_notify.conf` [using `edit-config`](/docs/netdata-agent/configuration/README.md). |
| 97 | ::: |
| 98 | |
| 99 | ### Best Practices for Large Deployments |
| 100 | |
| 101 | #### Central Alerting Strategy |
| 102 | |
| 103 | When you: |
| 104 | - Don't need edge automation (no scripts reacting to alerts) |
| 105 | - Use highly available Parents for all nodes |
| 106 | - Use Netdata Cloud (at least for Parents) |
| 107 | |
| 108 | Follow these steps: |
| 109 | 1. Disable health monitoring on child nodes |
| 110 | 2. Share the same alert configuration across Parents (use git repo or CI/CD) |
| 111 | 3. Disable Parent notifications (`SEND_EMAIL="NO"` in `/etc/netdata/health_alarm_notify.conf`) |
| 112 | 4. Keep only Cloud notifications |
| 113 | |
| 114 | This emulates traditional monitoring tools where you configure alerts centrally and dispatch notifications centrally. |
| 115 | |
| 116 | #### Edge Flexible Alerting Strategy |
| 117 | |
| 118 | When you: |
| 119 | - Need edge automation (scale out, restart processes) |
| 120 | - Use Parents |
| 121 | - Use Cloud for all nodes |
| 122 | |
| 123 | Follow these steps: |
| 124 | - Disable stock alerts on children (`enable stock health configuration` to `no` in `/etc/netdata/netdata.conf` `[health]` section) |
| 125 | - Configure only automation-required alerts on children |
| 126 | - Keep stock alerts on Parents but disable notifications (`SEND_EMAIL="NO"`) |
| 127 | - Keep only Cloud notifications |
| 128 | |
| 129 | This enables edge automation on children while maintaining central alerting control and deduplicated Cloud notifications. |
| 130 | |
| 131 | ## Set Up Alerts via Netdata Cloud |
| 132 | |
| 133 | 1. Connect your nodes to [Netdata Cloud](https://app.netdata.cloud/) |
| 134 | 2. Navigate to: `Space → Notifications` |
| 135 | 3. Choose your integration (Slack, Amazon SNS, Splunk) |
| 136 | 4. Configure alert severity filters |
| 137 | |
| 138 | [View all Cloud integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/centralized-cloud-notifications) |
| 139 | |
| 140 | ## Set Up Alerts via Netdata Agent |
| 141 | |
| 142 | 1. Open notification config: |
| 143 | ```bash |
| 144 | sudo ./edit-config health_alarm_notify.conf |
| 145 | ``` |
| 146 | |
| 147 | 2. Enable your method (example: email): |
| 148 | ```ini |
| 149 | SEND_EMAIL="YES" |
| 150 | DEFAULT_RECIPIENT_EMAIL="you@example.com" |
| 151 | ``` |
| 152 | |
| 153 | 3. Verify your system can send mail (sendmail, SMTP relay) |
| 154 | |
| 155 | 4. Restart the agent: |
| 156 | ```bash |
| 157 | sudo systemctl restart netdata |
| 158 | ``` |
| 159 | |
| 160 | [View all Agent integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications) |
| 161 | |
| 162 | ## Core Alerting Concepts |
| 163 | |
| 164 | Netdata supports two alert types: |
| 165 | - **Alarms**: Attach to specific instances (specific network interface, database instance) |
| 166 | - **Templates**: Apply to all matching instances (all network interfaces, all databases) |
| 167 | |
| 168 | ### Alert Lifecycle and States |
| 169 | |
| 170 | Your alerts produce more than threshold checks. Each generates: |
| 171 | - **A value**: Combines metrics or other alerts using time-series lookups and expressions |
| 172 | - **A unit**: Makes alerts meaningful ("seconds", "%", "requests/s") |
| 173 | - **A name**: Identifies the alert |
| 174 | |
| 175 | This enables sophisticated alerts like: |
| 176 | - `out of disk space time: 450 seconds` - Predicts when disk fills based on current rate |
| 177 | - `3xx redirects: 12.5 percent` - Calculates redirects as percentage of total |
| 178 | - `response time vs yesterday: 150%` - Compares current to historical baseline |
| 179 | |
| 180 | ### Alert States |
| 181 | |
| 182 | Your alerts exist in one of these states: |
| 183 | |
| 184 | | State | Description | Trigger | |
| 185 | |-------------------|---------------------------------------------------|--------------------------------------------------------------------------| |
| 186 | | **CLEAR** | Normal - conditions exist but not triggered | Warning and critical conditions evaluate to zero | |
| 187 | | **WARNING** | Warning threshold exceeded | Warning condition evaluates to non-zero | |
| 188 | | **CRITICAL** | Critical threshold exceeded | Critical condition evaluates to non-zero | |
| 189 | | **UNDEFINED** | Cannot evaluate | No conditions defined, or value is NaN/Inf | |
| 190 | | **UNINITIALIZED** | Never evaluated | Alert just created | |
| 191 | | **REMOVED** | Alert deleted | Child disconnected, agent exit, or health reload | |
| 192 | |
| 193 | Alerts transition freely between states based on: |
| 194 | - **Calculated value** (including NaN, Inf, or valid numbers) |
| 195 | - **Warning/critical conditions** (evaluation results) |
| 196 | - **External events** (disconnections, reloads, exits) |
| 197 | |
| 198 | Key behaviors: |
| 199 | - Alerts jump directly from CLEAR to CRITICAL (no WARNING required) |
| 200 | - WARNING and CRITICAL evaluate independently |
| 201 | - Alerts return to appropriate state when data becomes available |
| 202 | - CRITICAL takes precedence when both conditions are true |
| 203 | |
| 204 | ### Alert Evaluation Process |
| 205 | |
| 206 | #### 1. Calculate Value |
| 207 | |
| 208 | Your alerts perform complex calculations: |
| 209 | |
| 210 | ``` |
| 211 | lookup calc warn,crit status |
| 212 | ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────┐ |
| 213 | │ Database │ │Expression│ │ Warning │ │ Execute │ |
| 214 | │ Query │──────>│Processor │──────>│ Critical │ ───> │ Action on │ |
| 215 | │(optional)│ $this │(optional)│ $this │ Checks │ │Transition │ |
| 216 | └──────────┘ └──────────┘ └──────────┘ └───────────┘ |
| 217 | ``` |
| 218 | |
| 219 | Examples: |
| 220 | ```yaml |
| 221 | # Simple threshold |
| 222 | calc: $used |
| 223 | # Result: $this = latest value of dimension 'used' |
| 224 | |
| 225 | # Time-series lookup |
| 226 | lookup: average -1h of used |
| 227 | # Result: $this = average of 'used' over last hour |
| 228 | |
| 229 | # Combined calculation |
| 230 | lookup: average -1h of used |
| 231 | calc: $this * 100 / $total |
| 232 | # Result: $this = percentage of hourly average vs total |
| 233 | |
| 234 | # Baseline comparison |
| 235 | lookup: average -1h of used |
| 236 | calc: $this * 100 / $average_yesterday |
| 237 | # Result: $this = percentage vs yesterday's average |
| 238 | ``` |
| 239 | |
| 240 | #### 2. Evaluate Conditions |
| 241 | |
| 242 | After calculating, check conditions: |
| 243 | |
| 244 | ```yaml |
| 245 | # Simple conditions |
| 246 | warn: $this > 80 |
| 247 | crit: $this > 90 |
| 248 | |
| 249 | # Flapping prevention |
| 250 | warn: ($status >= $WARNING) ? ($this > 50) : ($this > 80) |
| 251 | crit: ($status == $CRITICAL) ? ($this > 70) : ($this > 90) |
| 252 | |
| 253 | # Complex conditions |
| 254 | warn: $this > 80 AND $rate > 10 |
| 255 | crit: $this > 90 OR $failures > 5 |
| 256 | ``` |
| 257 | |
| 258 | #### 3. Determine State |
| 259 | |
| 260 | Each condition evaluates to: |
| 261 | - NaN or Inf → UNDEFINED |
| 262 | - Non-zero → RAISED |
| 263 | - Zero → CLEAR |
| 264 | |
| 265 | Final status: |
| 266 | - Critical RAISED → **CRITICAL** (priority) |
| 267 | - Warning RAISED → **WARNING** |
| 268 | - Either CLEAR → **CLEAR** |
| 269 | - Both missing/UNDEFINED → **UNDEFINED** |
| 270 | |
| 271 | ### Evaluation Timing |
| 272 | |
| 273 | Alert evaluation runs independently from data collection: |
| 274 | |
| 275 | ``` |
| 276 | Data Collection Alert Evaluation |
| 277 | │ │ |
| 278 | ▼ every 1s ▼ configurable interval |
| 279 | [Metrics] ──────────> [Alert Engine] |
| 280 | │ |
| 281 | ▼ |
| 282 | Query metrics, |
| 283 | Calculate values, |
| 284 | Check conditions |
| 285 | ``` |
| 286 | |
| 287 | - **Default interval**: Query window duration (with lookup) or manual setting required |
| 288 | - **Configurable**: Use `every` for custom intervals |
| 289 | - **Constrained**: Cannot evaluate faster than data collection frequency |
| 290 | |
| 291 | ### Anti-Flapping Mechanisms |
| 292 | |
| 293 | Netdata prevents alert flapping through: |
| 294 | |
| 295 | #### 1. Hysteresis |
| 296 | ```yaml |
| 297 | warn: ($status < $WARNING) ? ($this > 80) : ($this > 50) |
| 298 | ``` |
| 299 | Triggers at 80, clears at 50, preventing flapping between 50-80. |
| 300 | |
| 301 | #### 2. Dynamic Delays |
| 302 | Alerts transition immediately in dashboards but notifications use exponential backoff. |
| 303 | |
| 304 | #### 3. Duration Requirements |
| 305 | ```yaml |
| 306 | lookup: average -10m of used |
| 307 | warn: $this > 80 |
| 308 | ``` |
| 309 | Requires 10 minutes of data before triggering. |
| 310 | |
| 311 | ### Multi-Stage Alerts |
| 312 | |
| 313 | Create dependent alerts: |
| 314 | |
| 315 | ```yaml |
| 316 | # Stage 1: Baseline |
| 317 | template: requests_average_yesterday |
| 318 | on: web_log.requests |
| 319 | lookup: average -1h at -1d |
| 320 | every: 10s |
| 321 | |
| 322 | # Stage 2: Current |
| 323 | template: requests_average_now |
| 324 | on: web_log.requests |
| 325 | lookup: average -1h |
| 326 | every: 10s |
| 327 | |
| 328 | # Stage 3: Compare |
| 329 | template: web_requests_vs_yesterday |
| 330 | on: web_log.requests |
| 331 | calc: $requests_average_now * 100 / $requests_average_yesterday |
| 332 | units: % |
| 333 | warn: $this > 150 || $this < 75 |
| 334 | crit: $this > 200 || $this < 50 |
| 335 | ``` |
| 336 | |
| 337 | ### Available Variables |
| 338 | |
| 339 | Variables resolve in order (first match wins): |
| 340 | |
| 341 | #### 1. Built-in Variables |
| 342 | |
| 343 | | Variable | Description | Value | |
| 344 | |---------------------|---------------------------|--------------------| |
| 345 | | `$this` | Current calculated value | Result from lookup/calc | |
| 346 | | `$after` | Query start timestamp | Unix timestamp | |
| 347 | | `$before` | Query end timestamp | Unix timestamp | |
| 348 | | `$now` | Current time | Unix timestamp | |
| 349 | | `$last_collected_t` | Last collection time | Unix timestamp | |
| 350 | | `$update_every` | Collection frequency | Seconds | |
| 351 | | `$status` | Current status code | -2 to 3 | |
| 352 | | `$REMOVED` | Status constant | -2 | |
| 353 | | `$UNINITIALIZED` | Status constant | -1 | |
| 354 | | `$UNDEFINED` | Status constant | 0 | |
| 355 | | `$CLEAR` | Status constant | 1 | |
| 356 | | `$WARNING` | Status constant | 2 | |
| 357 | | `$CRITICAL` | Status constant | 3 | |
| 358 | |
| 359 | #### 2. Dimension Values |
| 360 | |
| 361 | | Syntax | Description | Example | |
| 362 | |------------------------------------|----------------------------------|------------------| |
| 363 | | `$dimension_name` | Last normalized value | `$used` | |
| 364 | | `$dimension_name_raw` | Last raw collected value | `$used_raw` | |
| 365 | | `$dimension_name_last_collected_t` | Collection timestamp | `$used_last_collected_t` | |
| 366 | |
| 367 | ```yaml |
| 368 | template: disk_usage_percent |
| 369 | on: disk.space |
| 370 | calc: $used * 100 / ($used + $available) |
| 371 | units: % |
| 372 | ``` |
| 373 | |
| 374 | #### 3. Chart Variables |
| 375 | ```yaml |
| 376 | calc: $used > $threshold # If chart defines 'threshold' |
| 377 | ``` |
| 378 | |
| 379 | #### 4. Host Variables |
| 380 | ```yaml |
| 381 | warn: $connections > $max_connections * 0.8 # If host defines 'max_connections' |
| 382 | ``` |
| 383 | |
| 384 | #### 5. Other Alerts |
| 385 | ```yaml |
| 386 | # Alert 1 |
| 387 | template: cpu_baseline |
| 388 | calc: $system + $user |
| 389 | |
| 390 | # Alert 2 |
| 391 | template: cpu_check |
| 392 | calc: $system |
| 393 | warn: $this > $cpu_baseline * 1.5 |
| 394 | ``` |
| 395 | |
| 396 | #### 6. Cross-Context References |
| 397 | ```yaml |
| 398 | template: disk_io_vs_iops |
| 399 | on: disk.io |
| 400 | calc: $reads / ${disk.iops.reads} |
| 401 | units: bytes per operation |
| 402 | ``` |
| 403 | |
| 404 | ### Variable Resolution and Label Scoring |
| 405 | |
| 406 | When alerts reference variables matching multiple instances, Netdata uses label similarity scoring: |
| 407 | |
| 408 | 1. **Collect candidates** with matching names |
| 409 | 2. **Score by labels** - count common labels |
| 410 | 3. **Select best match** - highest label overlap |
| 411 | |
| 412 | Example: Alert on `disk.io` (labels: `device=sda`, `mount=/data`) references `${disk.iops.reads}`: |
| 413 | - `disk.iops` for sda (labels match) → Score: 2 |
| 414 | - `disk.iops` for sdb (no match) → Score: 0 |
| 415 | Result: Uses sda's value |
| 416 | |
| 417 | ### Missing Data Handling |
| 418 | |
| 419 | During lookups with missing data: |
| 420 | - **All values NULL**: `$this` becomes `NaN` |
| 421 | - **Some values exist**: Ignores NULL, continues calculation |
| 422 | - **Dimension doesn't exist**: `$this` becomes `NaN` |
| 423 | |
| 424 | This handles intermittent collection, dynamic dimensions, and partial outages. |
| 425 | |
| 426 | ### Evaluation Frequency |
| 427 | |
| 428 | Determine frequency by: |
| 429 | |
| 430 | 1. **With lookup**: Defaults to window duration |
| 431 | ```yaml |
| 432 | lookup: average -5m # Evaluates every 5 minutes |
| 433 | ``` |
| 434 | |
| 435 | 2. **Without lookup**: Set explicitly |
| 436 | ```yaml |
| 437 | every: 10s |
| 438 | calc: $system + $user |
| 439 | ``` |
| 440 | |
| 441 | 3. **Custom interval**: Override default |
| 442 | ```yaml |
| 443 | lookup: average -1m |
| 444 | every: 10s # Check every 10s despite 1m window |
| 445 | ``` |
| 446 | |
| 447 | Constraints: |
| 448 | - Cannot exceed data collection frequency |
| 449 | - High frequency impacts performance |
| 450 | - Use larger intervals with `unaligned` for efficiency |
| 451 | |
| 452 | ## Troubleshooting Your Alerts |
| 453 | |
| 454 | ### Netdata Assistant |
| 455 | |
| 456 | The [Netdata Assistant](https://learn.netdata.cloud/docs/machine-learning-and-anomaly-detection/ai-powered-troubleshooting-assistant) provides AI-powered troubleshooting when alerts trigger: |
| 457 | |
| 458 | 1. Click the alert in your dashboard |
| 459 | 2. Press the Assistant button |
| 460 | 3. Receive customized troubleshooting tips |
| 461 | |
| 462 | The Assistant window follows you through dashboards for easy reference while investigating. |
| 463 | |
| 464 | ### Community Resources |
| 465 | |
| 466 | Visit our [Alerts Troubleshooting space](https://community.netdata.cloud/c/alerts/28) for complex issues. Get help through [GitHub](https://github.com/netdata/netdata) or [Discord](https://discord.gg/kUk3nCmbtx). Share your solutions to help others. |
| 467 | |
| 468 | ### Customizing Alerts |
| 469 | |
| 470 | Tune alerts for your environment by adjusting thresholds, writing custom conditions, silencing alerts, and using statistical functions. |
| 471 | |
| 472 | - [Customize alerts](/src/health/REFERENCE.md) |
| 473 | - [Silence or disable alerts](/src/health/REFERENCE.md#how-to-disable-or-silence-alerts) |
| 474 | |
| 475 | ## Related Documentation |
| 476 | |
| 477 | - [All notification methods](/docs/alerts-and-notifications/notifications/README.md) |
| 478 | - [Supported collectors](/src/collectors/COLLECTORS.md) |
| 479 | - [Full alert reference](/src/health/REFERENCE.md) |