| 1 | # Overriding Stock Alerts |
| 2 | |
| 3 | This guide explains how to customize Netdata's stock alerts. User configurations survive upgrades, making this the recommended approach. |
| 4 | |
| 5 | ## Quick Reference |
| 6 | |
| 7 | | Goal | Method | |
| 8 | |------|--------| |
| 9 | | Change thresholds for ALL instances | Create a template with the same name | |
| 10 | | Change thresholds for ONE instance | Create an alarm with the same name | |
| 11 | | Disable an alert completely | Use `enabled alarms` in netdata.conf | |
| 12 | | Silence notifications only | Set `to: silent` | |
| 13 | |
| 14 | ## Understanding Overrides |
| 15 | |
| 16 | Netdata's alerting uses **templates** (match all instances of a context) and **alarms** (match one specific instance). Stock alerts are mostly templates—they apply to all disks, all CPUs, etc. |
| 17 | |
| 18 | To override, create an alert with the **same name**. User definitions are processed before stock definitions, so yours wins. |
| 19 | |
| 20 | See [Alert Configuration Ordering](/src/health/alert-configuration-ordering.md) for the full conceptual explanation. |
| 21 | |
| 22 | ## Where to Put Your Overrides |
| 23 | |
| 24 | **User config directory** (default): `/etc/netdata/health.d/` |
| 25 | |
| 26 | Files here survive upgrades. Stock files in `/usr/lib/netdata/conf.d/health.d/` are replaced during updates. |
| 27 | |
| 28 | Check your `netdata.conf` `[directories]` section for exact paths on your system. |
| 29 | |
| 30 | ## Method 1: Override All Instances (Template) |
| 31 | |
| 32 | Create a template with the same name to change thresholds for ALL instances. |
| 33 | |
| 34 | **Example: Raise CPU steal thresholds globally** |
| 35 | |
| 36 | Stock alert in `/usr/lib/netdata/conf.d/health.d/cpu.conf`: |
| 37 | ```yaml |
| 38 | template: 20min_steal_cpu |
| 39 | on: system.cpu |
| 40 | lookup: average -20m unaligned of steal |
| 41 | units: % |
| 42 | every: 5m |
| 43 | warn: $this > (($status >= $WARNING) ? (5) : (10)) |
| 44 | ``` |
| 45 | |
| 46 | Your override in `/etc/netdata/health.d/my-overrides.conf`: |
| 47 | ```yaml |
| 48 | template: 20min_steal_cpu |
| 49 | on: system.cpu |
| 50 | lookup: average -20m unaligned of steal |
| 51 | units: % |
| 52 | every: 5m |
| 53 | warn: $this > (($status >= $WARNING) ? (10) : (20)) |
| 54 | ``` |
| 55 | |
| 56 | **Why it works:** Same name + same context. Your template is processed first, creating the alert. The stock template is then skipped. |
| 57 | |
| 58 | > **Note:** Most stock alerts are templates. If a stock alert is an alarm (rare), you must override it with an alarm, not a template—alarms are always processed before templates. |
| 59 | |
| 60 | ## Method 2: Override One Instance (Alarm) |
| 61 | |
| 62 | Create an alarm to override thresholds for ONE specific instance while keeping stock thresholds for others. |
| 63 | |
| 64 | **Example: Different disk space threshold for `/mnt/data`** |
| 65 | |
| 66 | Stock template (applies to all disks): |
| 67 | ```yaml |
| 68 | template: disk_space_usage |
| 69 | on: disk.space |
| 70 | lookup: max -1m percentage of avail |
| 71 | warn: $this < 20 |
| 72 | crit: $this < 10 |
| 73 | ``` |
| 74 | |
| 75 | Your override in `/etc/netdata/health.d/my-overrides.conf`: |
| 76 | ```yaml |
| 77 | alarm: disk_space_usage |
| 78 | on: disk_space._mnt_data |
| 79 | lookup: max -1m percentage of avail |
| 80 | warn: $this < 5 |
| 81 | crit: $this < 2 |
| 82 | ``` |
| 83 | |
| 84 | **Why it works:** |
| 85 | - Both have the **same name** (`disk_space_usage`) |
| 86 | - Your **alarm** targets the specific chart ID `disk_space._mnt_data` |
| 87 | - Alarms are processed before templates (when names match) |
| 88 | - For `/mnt/data`: your alarm creates the alert, stock template is skipped |
| 89 | - For all other disks: stock template creates alerts normally |
| 90 | |
| 91 | **Key difference:** Templates use `on:` with a **context** (`disk.space`). Alarms use `on:` with a **chart ID** (`disk_space._mnt_data`). |
| 92 | |
| 93 | ### Finding Chart IDs |
| 94 | |
| 95 | To find the exact chart ID for an instance: |
| 96 | |
| 97 | ```bash |
| 98 | curl -s "http://localhost:19999/api/v1/charts" | grep -o '"id":"disk_space[^"]*"' |
| 99 | ``` |
| 100 | |
| 101 | Or check the chart title in the Netdata dashboard—the chart ID is shown in the URL when you click on a chart. |
| 102 | |
| 103 | ### Alternative: Using Chart Labels |
| 104 | |
| 105 | Instead of chart IDs, you can match by labels: |
| 106 | |
| 107 | ```yaml |
| 108 | template: disk_space_usage |
| 109 | on: disk.space |
| 110 | chart labels: mount_point=/mnt/data |
| 111 | lookup: max -1m percentage of avail |
| 112 | warn: $this < 5 |
| 113 | crit: $this < 2 |
| 114 | ``` |
| 115 | |
| 116 | Use labels when: |
| 117 | - You want to target multiple instances sharing a label |
| 118 | - Chart IDs are dynamic or unpredictable |
| 119 | |
| 120 | ## Method 3: Copy Entire Stock File |
| 121 | |
| 122 | If you want to modify many alerts in one stock file, copy it entirely: |
| 123 | |
| 124 | ```bash |
| 125 | cd /etc/netdata |
| 126 | sudo ./edit-config health.d/cpu.conf |
| 127 | ``` |
| 128 | |
| 129 | **Important:** When a file with the same name exists in both directories, Netdata loads **only** the user file. The stock file is completely ignored. |
| 130 | |
| 131 | This means your copy must include ALL alerts you want—not just the ones you're changing. |
| 132 | |
| 133 | ## Disabling Alerts |
| 134 | |
| 135 | ### Option A: Global Disable |
| 136 | |
| 137 | In `/etc/netdata/netdata.conf`: |
| 138 | |
| 139 | ```ini |
| 140 | [health] |
| 141 | enabled alarms = !20min_steal_cpu !disk_space_usage * |
| 142 | ``` |
| 143 | |
| 144 | This disables `20min_steal_cpu` and `disk_space_usage` while keeping all other alerts (`*`). |
| 145 | |
| 146 | ### Option B: Per-Alert Disable |
| 147 | |
| 148 | Create an override that never matches: |
| 149 | |
| 150 | ```yaml |
| 151 | template: 20min_steal_cpu |
| 152 | on: system.cpu |
| 153 | host labels: _hostname=!* |
| 154 | ``` |
| 155 | |
| 156 | The pattern `!*` is a special disable shortcut—the health config parser recognizes it and disables the alert. |
| 157 | |
| 158 | ### Option C: Silence Notifications Only |
| 159 | |
| 160 | Keep the alert monitoring but stop notifications: |
| 161 | |
| 162 | ```yaml |
| 163 | template: 20min_steal_cpu |
| 164 | on: system.cpu |
| 165 | lookup: average -20m unaligned of steal |
| 166 | units: % |
| 167 | every: 5m |
| 168 | warn: $this > (($status >= $WARNING) ? (5) : (10)) |
| 169 | to: silent |
| 170 | ``` |
| 171 | |
| 172 | The alert still appears in the dashboard but sends no notifications. |
| 173 | |
| 174 | ## Applying Changes |
| 175 | |
| 176 | Reload the health configuration: |
| 177 | |
| 178 | ```bash |
| 179 | sudo netdatacli reload-health |
| 180 | ``` |
| 181 | |
| 182 | If `netdatacli` isn't available, send `SIGUSR2` to the Netdata process. |
| 183 | |
| 184 | ### Verify Your Override |
| 185 | |
| 186 | Check via API: |
| 187 | ```bash |
| 188 | curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[] | select(.value.name == "20min_steal_cpu") | .value' |
| 189 | ``` |
| 190 | |
| 191 | Key fields to check: |
| 192 | - `source`: confirms which config file is active (user override vs stock) |
| 193 | - `lookup_*`: data query parameters |
| 194 | - `warn`, `crit`: threshold expressions |
| 195 | |
| 196 | Or navigate to the Alerts tab in the dashboard and verify thresholds match your override. |
| 197 | |
| 198 | ### Check for Errors |
| 199 | |
| 200 | If your override isn't working, check the logs: |
| 201 | |
| 202 | ```bash |
| 203 | # systemd journal (most Linux distributions): |
| 204 | journalctl --namespace netdata -g health --no-pager | tail -20 |
| 205 | |
| 206 | # Log files (if journal not available): |
| 207 | grep -i health /var/log/netdata/error.log | tail -20 |
| 208 | ``` |
| 209 | |
| 210 | Common issues: |
| 211 | - Syntax errors in configuration |
| 212 | - Alert name doesn't match exactly (case-sensitive) |
| 213 | - File permissions prevent Netdata from reading your config |
| 214 | |
| 215 | ## Troubleshooting |
| 216 | |
| 217 | ### Override Not Taking Effect |
| 218 | |
| 219 | 1. **Reload configuration**: `sudo netdatacli reload-health` |
| 220 | 2. **Check file permissions**: Netdata must be able to read your file |
| 221 | 3. **Verify exact name match**: Alert names are case-sensitive |
| 222 | 4. **Check for syntax errors**: Look in error.log |
| 223 | |
| 224 | ### Both Stock and Override Alerts Appear |
| 225 | |
| 226 | This happens when matching criteria don't overlap. For example: |
| 227 | - Your override has `host labels: production` |
| 228 | - Stock alert has no host labels restriction |
| 229 | |
| 230 | Both can create alerts on different hosts. Ensure your override matches at least the same scope as the stock alert. |
| 231 | |
| 232 | ### UI Edit Replaced My Config |
| 233 | |
| 234 | Editing an alert through the dashboard UI creates a dynamic configuration that **replaces** any file-based definition. To restore file-based behavior, remove the dynamic config through the UI or API. |
| 235 | |
| 236 | ## FAQ |
| 237 | |
| 238 | ### Do I need to copy all fields when overriding an alert? |
| 239 | |
| 240 | Yes. Your override is a complete alert definition, not a "patch" on the stock alert. Include all fields: `lookup`, `calc`, `warn`, `crit`, `units`, etc. |
| 241 | |
| 242 | If you omit a field, the alert uses its default value—not the stock alert's value. |
| 243 | |
| 244 | ### How do I override the same alert differently on different hosts? |
| 245 | |
| 246 | Use `host labels` to create host-specific overrides: |
| 247 | |
| 248 | ```yaml |
| 249 | # Production servers: stricter thresholds |
| 250 | template: cpu_usage |
| 251 | on: system.cpu |
| 252 | host labels: environment=production |
| 253 | warn: $this > 70 |
| 254 | |
| 255 | # Development servers: relaxed thresholds |
| 256 | template: cpu_usage |
| 257 | on: system.cpu |
| 258 | host labels: environment=development |
| 259 | warn: $this > 90 |
| 260 | ``` |
| 261 | |
| 262 | Both can coexist because they match different hosts. |
| 263 | |
| 264 | ### How do I find what stock alerts exist? |
| 265 | |
| 266 | List all stock alert files: |
| 267 | ```bash |
| 268 | ls /usr/lib/netdata/conf.d/health.d/ |
| 269 | ``` |
| 270 | |
| 271 | View a specific stock alert: |
| 272 | ```bash |
| 273 | cat /usr/lib/netdata/conf.d/health.d/cpu.conf |
| 274 | ``` |
| 275 | |
| 276 | Or use the API to list all alert names: |
| 277 | ```bash |
| 278 | curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[].value.name' | sort -u |
| 279 | ``` |
| 280 | |
| 281 | ### Can I add new alerts without affecting stock alerts? |
| 282 | |
| 283 | Yes. Create alerts with **different names** than stock alerts. They'll coexist independently. |
| 284 | |
| 285 | ```yaml |
| 286 | # This is a NEW alert, not an override |
| 287 | template: my_custom_disk_alert |
| 288 | on: disk.space |
| 289 | lookup: max -5m percentage of avail |
| 290 | warn: $this < 15 |
| 291 | ``` |
| 292 | |
| 293 | ### What happens to my overrides after a Netdata upgrade? |
| 294 | |
| 295 | User config files in `/etc/netdata/health.d/` are preserved. Stock files in `/usr/lib/netdata/conf.d/health.d/` are replaced. |
| 296 | |
| 297 | Your overrides continue working. However, if a stock alert is renamed or removed in a new version, your override may become orphaned (still works, but no longer overriding anything). |
| 298 | |
| 299 | ### How do I override for multiple specific instances? |
| 300 | |
| 301 | Option 1: Create multiple alarms (one per instance): |
| 302 | ```yaml |
| 303 | alarm: disk_space_usage |
| 304 | on: disk_space._mnt_data |
| 305 | warn: $this < 5 |
| 306 | |
| 307 | alarm: disk_space_usage |
| 308 | on: disk_space._mnt_backup |
| 309 | warn: $this < 5 |
| 310 | ``` |
| 311 | |
| 312 | Option 2: Use chart labels if instances share a label: |
| 313 | ```yaml |
| 314 | template: disk_space_usage |
| 315 | on: disk.space |
| 316 | chart labels: storage_tier=bulk |
| 317 | warn: $this < 5 |
| 318 | ``` |
| 319 | |
| 320 | ### Can I see what overrides are currently active? |
| 321 | |
| 322 | Check which config files Netdata loaded: |
| 323 | ```bash |
| 324 | # systemd journal: |
| 325 | journalctl --namespace netdata -g "health.*load\|health.*read" --no-pager |
| 326 | |
| 327 | # Log files: |
| 328 | grep -iE "health.*(load|read)" /var/log/netdata/error.log |
| 329 | ``` |
| 330 | |
| 331 | Compare your active alert config vs stock: |
| 332 | ```bash |
| 333 | # Your override |
| 334 | cat /etc/netdata/health.d/my-overrides.conf |
| 335 | |
| 336 | # Stock definition |
| 337 | cat /usr/lib/netdata/conf.d/health.d/disks.conf |
| 338 | ``` |
| 339 | |
| 340 | ### Why does editing an alert in the UI override my file-based config? |
| 341 | |
| 342 | UI edits create dynamic configurations that take precedence over all file-based configs. This is by design—it allows quick adjustments without SSH access. |
| 343 | |
| 344 | To restore file-based control, remove the dynamic config through the UI (reset to default) or via the API. |
| 345 | |
| 346 | ## Related Documentation |
| 347 | |
| 348 | - [Health Configuration Reference](/src/health/REFERENCE.md) |
| 349 | - [Alert Configuration Ordering](/src/health/alert-configuration-ordering.md) |