| 1 | # Alert Configuration Ordering |
| 2 | |
| 3 | This document explains how Netdata's alerting system is designed and how it determines which alert definition applies when multiple definitions could match the same data. |
| 4 | |
| 5 | ## The Problem: Alerts at Scale |
| 6 | |
| 7 | Netdata monitors infrastructure that can range from a single server to thousands of nodes, each with dozens of components: disks, network interfaces, databases, containers, and more. |
| 8 | |
| 9 | The challenge: How do you configure alerts that automatically apply to all Redis instances, all disks, or all network interfaces—without manually defining alerts for each one? And when one specific instance needs different thresholds, how do you override just that one predictably? |
| 10 | |
| 11 | ## The Solution: Contexts and Instances |
| 12 | |
| 13 | Netdata's alerting system is built around two concepts: |
| 14 | |
| 15 | | Concept | What it is | Example | |
| 16 | |---------|------------|---------| |
| 17 | | **Context** | Defines what the metrics ARE—their meaning and units | `disk.space` (disk space utilization in %) | |
| 18 | | **Instance** | An individual component being monitored | `/mnt/data`, `/home`, `/var` | |
| 19 | |
| 20 | A context groups all instances that share the same metric definition. For example, the `disk.space` context includes every mounted filesystem on the system. |
| 21 | |
| 22 | ## Templates vs Alarms |
| 23 | |
| 24 | Netdata provides two ways to define alerts: |
| 25 | |
| 26 | ### Templates: Match by Context |
| 27 | |
| 28 | A **template** applies to ALL instances of a context automatically. |
| 29 | |
| 30 | ```yaml |
| 31 | template: disk_space_usage |
| 32 | on: disk.space # matches the CONTEXT |
| 33 | lookup: max -1m percentage of avail |
| 34 | warn: $this < 20 |
| 35 | crit: $this < 10 |
| 36 | ``` |
| 37 | |
| 38 | This single definition creates alerts for every disk on every node—automatically. When a new disk is mounted, it gets this alert. No manual configuration needed. |
| 39 | |
| 40 | ### Alarms: Match by Instance |
| 41 | |
| 42 | An **alarm** applies to ONE specific instance. |
| 43 | |
| 44 | ```yaml |
| 45 | alarm: disk_space_usage |
| 46 | on: disk_space._mnt_data # matches a specific INSTANCE (chart ID) |
| 47 | lookup: max -1m percentage of avail |
| 48 | warn: $this < 5 |
| 49 | crit: $this < 2 |
| 50 | ``` |
| 51 | |
| 52 | Use alarms when a specific instance needs different treatment—like a data disk that's expected to run fuller than others. |
| 53 | |
| 54 | **Key difference**: The `on:` line specifies a **context** for templates, but a **chart ID** (specific instance) for alarms. |
| 55 | |
| 56 | ## Precedence: Same Name Required |
| 57 | |
| 58 | Multiple alerts with **different names** can coexist on the same instance. You can have `disk_space_usage`, `disk_io_latency`, and `disk_errors` all monitoring the same disk. |
| 59 | |
| 60 | The precedence rules only apply when alerts have the **same name**. In that case, only one alert with that name can exist per instance: |
| 61 | |
| 62 | | Priority | Type | What it matches | |
| 63 | |----------|------|-----------------| |
| 64 | | 1 (higher) | Alarm | Specific instance | |
| 65 | | 2 (lower) | Template | All instances of a context | |
| 66 | |
| 67 | **Example scenario:** |
| 68 | |
| 69 | 1. Stock template `disk_space_usage` on context `disk.space` → warn at 20% |
| 70 | 2. User alarm `disk_space_usage` on instance `disk_space._mnt_data` → warn at 5% |
| 71 | |
| 72 | Both have the **same name** (`disk_space_usage`), so precedence applies: |
| 73 | - `/mnt/data` gets the alarm's thresholds (warn at 5%) |
| 74 | - All other disks get the template's thresholds (warn at 20%) |
| 75 | |
| 76 | ## Configuration Loading Order |
| 77 | |
| 78 | Netdata loads alert configurations from two directories: |
| 79 | |
| 80 | 1. **User config** (loaded first): `/etc/netdata/health.d/` (default) |
| 81 | 2. **Stock config** (loaded second): `/usr/lib/netdata/conf.d/health.d/` (default) |
| 82 | |
| 83 | These paths can vary by installation. Check your `netdata.conf` `[directories]` section for exact paths. |
| 84 | |
| 85 | ### File Shadowing |
| 86 | |
| 87 | If a file with the **same name** exists in both directories, only the user file is loaded. The stock file is completely ignored. |
| 88 | |
| 89 | **Example:** |
| 90 | - Stock: `/usr/lib/netdata/conf.d/health.d/cpu.conf` |
| 91 | - User: `/etc/netdata/health.d/cpu.conf` |
| 92 | - Result: Only the user file is loaded |
| 93 | |
| 94 | This means if you copy a stock file to override it, you must include **all** alerts you want from that file, not just the ones you're modifying. |
| 95 | |
| 96 | ### Complete Precedence |
| 97 | |
| 98 | Combining type precedence with source precedence: |
| 99 | |
| 100 | | Priority | Type | Source | |
| 101 | |----------|------|--------| |
| 102 | | 1 (highest) | Alarm | User config | |
| 103 | | 2 | Alarm | Stock config | |
| 104 | | 3 | Template | User config | |
| 105 | | 4 (lowest) | Template | Stock config | |
| 106 | |
| 107 | ## First-Match-Wins (Same Name Only) |
| 108 | |
| 109 | Only **one** alert can exist per (instance, alert_name) pair. When multiple definitions **with the same name** could apply to an instance: |
| 110 | |
| 111 | 1. The first matching definition (by precedence) creates the alert |
| 112 | 2. Later definitions with the same name are skipped for that instance |
| 113 | |
| 114 | This is why overriding works: create an alert with the same name, and yours is processed first. |
| 115 | |
| 116 | ## Dynamic Configuration Exception |
| 117 | |
| 118 | Alerts created or modified through the Netdata UI or API behave differently: |
| 119 | |
| 120 | - **UI/API changes replace** any existing definition with the same name |
| 121 | - This is the only case where a definition overwrites another |
| 122 | |
| 123 | When you edit an alert through the dashboard, it completely replaces any file-based definition with that name. |
| 124 | |
| 125 | ## Summary |
| 126 | |
| 127 | | Goal | Use | |
| 128 | |------|-----| |
| 129 | | Alert on ALL instances of a type | Template matching a context | |
| 130 | | Alert on ONE specific instance | Alarm matching a chart ID | |
| 131 | | Override a stock alert globally | User template with same name | |
| 132 | | Override for just one instance | User alarm with same name | |
| 133 | |
| 134 | ## FAQ |
| 135 | |
| 136 | ### Can I have multiple alerts monitoring the same instance? |
| 137 | |
| 138 | Yes. Different alert names create independent alerts. You can have `disk_space_usage`, `disk_io_latency`, and `disk_write_errors` all monitoring the same disk simultaneously. |
| 139 | |
| 140 | The "only one alert per instance" rule applies only to alerts **with the same name**. |
| 141 | |
| 142 | ### What happens if I define the same alert name twice in my user config? |
| 143 | |
| 144 | The first one processed wins. Since file loading order within a directory is non-deterministic (depends on filesystem), keep all definitions for the same alert name in a single file to avoid surprises. |
| 145 | |
| 146 | ### How do I know which alert definition is currently active? |
| 147 | |
| 148 | Query the API: |
| 149 | ```bash |
| 150 | curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[] | select(.value.name == "alert_name") | .value' |
| 151 | ``` |
| 152 | |
| 153 | Key fields to check: |
| 154 | - `source`: which config file is active |
| 155 | - `lookup_*`: data query parameters |
| 156 | - `warn`, `crit`: threshold expressions |
| 157 | |
| 158 | Or check the Alerts tab in the dashboard—click on an alert to see its current configuration. |
| 159 | |
| 160 | ### Why isn't my override working? |
| 161 | |
| 162 | Common causes: |
| 163 | 1. **Name mismatch**: Alert names are case-sensitive |
| 164 | 2. **Not reloaded**: Run `sudo netdatacli reload-health` |
| 165 | 3. **File permissions**: Netdata must be able to read your config file |
| 166 | 4. **Syntax error**: Check logs with `journalctl --namespace netdata -g health` or `grep -i health /var/log/netdata/error.log` |
| 167 | |
| 168 | ### What's the difference between context and chart ID? |
| 169 | |
| 170 | - **Context** (`disk.space`): The metric type—shared by all instances |
| 171 | - **Chart ID** (`disk_space._mnt_data`): A specific instance |
| 172 | |
| 173 | Templates use contexts. Alarms use chart IDs. |
| 174 | |
| 175 | ### Do user configs completely replace stock configs? |
| 176 | |
| 177 | No. User configs are processed **before** stock configs, but both are loaded (unless file shadowing applies). Your alert with the same name wins because it's processed first, but you're not deleting the stock definition—just preempting it. |
| 178 | |
| 179 | ### What is file shadowing? |
| 180 | |
| 181 | If a file with the **same filename** exists in both user and stock directories, only the user file is loaded. The stock file is completely ignored. |
| 182 | |
| 183 | This is different from alert-level overriding. With shadowing, you must include ALL alerts you want from that file. |
| 184 | |
| 185 | ## Related Documentation |
| 186 | |
| 187 | - [Health Configuration Reference](/src/health/REFERENCE.md) |
| 188 | - [Overriding Stock Alerts](/src/health/overriding-stock-alerts.md) |