@cryptotaxi247 / netdata-1 / commits / ee75009c0

docs: clarify alert override ordering and add conceptual model (#21613)

* docs: clarify alert override ordering and paths * docs: restructure alert override docs for clarity - Lead with conceptual model (contexts vs instances, templates vs alarms) - Explain "why" before "how" - automatic alerting at scale - Remove implementation details (linked lists, RRDCALC internals) - Fix terminology: context vs chart ID throughout - Use real stock alert name (20min_steal_cpu) in examples - Add FAQ sections to both documents - Add log file fallback alongside journalctl commands - Note that alarm overrides require alarms (not templates)

Costa Tsaousis committed Jan 22, 2026 at 10:05 UTC ee75009c0319545c90ae3e845ec903c6a3528369
5 files changed +596 -1
TODO-ALERT-OVERRIDE-DOCS-FIXES.md new
+54
@@ -0,0 +1,54 @@
1 +# TODO: Alert Override Docs Fixes
2 +
3 +## TL;DR
4 +- Fix remaining doc accuracy issues: disk space chart ID example, disable‑trick explanation, and `edit-config` usage wording.
5 +- Ensure new docs are tracked and mapped under Alerts & Notifications in Learn.
6 +
7 +## Analysis (facts from code/docs)
8 +- Stock health config path defaults to `/usr/lib/netdata/conf.d/health.d`. Evidence: `system/edit-config:69-92`, `src/health/README.md:81`, `src/health/health.c:135-145`.
9 +- File shadowing exists: user `health.d` file/subdir with same name prevents stock file/subdir from loading. Evidence: `src/libnetdata/paths/paths.c:219-312`.
10 +- Alert application order: alarms before templates; only one alert per (chart,name) is created because RRDCALC key is `{alert,chart}` and conflicts are rejected. Evidence: `src/health/health_prototypes.c:591-616`, `src/health/rrdcalc.c:177-183`, `src/health/rrdcalc.c:349-352`, `src/health/rrdcalc.c:409-426`.
11 +- Disk space chart IDs are `disk_space.<sanitized_mount>`; context is `disk.space`. Evidence: `src/collectors/diskspace.plugin/plugin_diskspace.c:224-235`, `src/collectors/proc.plugin/proc_self_mountinfo.c:293-295`, `src/database/rrdset-index-id.c:444-445`.
12 +- The `!*` disable shortcut is handled by the health config parser (sets `ap->match.enabled = false`). Evidence: `src/health/health_config.c:506-516`.
13 +- Style guide discourages hardcoding `/etc/netdata/edit-config` in docs; recommends running `edit-config` from the config dir. Evidence: `docs/developer-and-contributor-corner/style-guide.md:303-305`.
14 +
15 +## Decisions (confirmed by Costa)
16 +1) **Paths must be stated by name and path**
17 + - Use the *config directory name* (e.g., “stock health config directory”) **and** the concrete path.
18 + - Also mention this can vary by package/prefix (e.g., `/opt/...`) and point to `edit-config` / `netdata.conf` for discovery.
19 +
20 +2) **Keep and document the `host labels: _hostname=!*` trick**
21 + - This pattern exists in stock configs today; document it clearly (with caution) rather than remove it.
22 +
23 +3) **Reload is reliable; do not imply otherwise**
24 + - Use `netdatacli reload-health` (and SIGUSR2 if needed) as the canonical method.
25 + - Avoid “restart required” language.
26 +
27 +4) **Docs are in repo; keep them and map them**
28 + - Add/track the new docs and keep the `REFERENCE.md` link.
29 + - Update `docs/.map/map.csv` so the new pages appear in Learn.
30 +
31 +5) **Learn path**
32 + - Place both new pages under `Alerts & Notifications` so they appear at `https://learn.netdata.cloud/docs/alerts-&-notifications/`.
33 + - Evidence: existing Alerts & Notifications entries are in `docs/.map/map.csv:159-169`.
34 +
35 +## Plan
36 +- Fix disk space chart ID example and update ID discovery snippet.
37 +- Clarify disable‑trick explanation (`!*` parser shortcut).
38 +- Align `edit-config` usage with style guide while keeping path names/paths.
39 +- Add new docs to git and keep map entries under Alerts & Notifications.
40 +- Re-run a focused doc scan to confirm no remaining contradictions.
41 +
42 +## Implied Decisions (if you approve recommendations)
43 +- Use flexible stock path wording with a default example.
44 +- Keep the disable trick and explain it as a parser shortcut.
45 +- Prefer reload-health; restart only as a last resort.
46 +- Add new docs to repo.
47 +
48 +## Testing Requirements
49 +- Not applicable (documentation-only).
50 +
51 +## Documentation Updates Required
52 +- `src/health/alert-configuration-ordering.md`
53 +- `src/health/overriding-stock-alerts.md`
54 +- `src/health/REFERENCE.md` (if link/path wording changes)
docs/.map/map.csv
+2
@@ -166,6 +166,8 @@ https://github.com/netdata/netdata/edit/master/docs/alerts-and-notifications/not
166 https://github.com/netdata/netdata/edit/master/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-alert-notification-silencing-rules.md,Manage alert notification silencing rules,Published,Alerts & Notifications/Notifications/Centralized Cloud Notifications,,Master Netdata alert management via notification silencing rules for efficient and focused monitoring.
167 cloud_notifications_integrations,,,,,
168 https://github.com/netdata/netdata/edit/master/src/health/REFERENCE.md,Alert Configuration Reference,Published,Alerts & Notifications,,
169 +https://github.com/netdata/netdata/edit/master/src/health/alert-configuration-ordering.md,Alert Configuration Ordering,Published,Alerts & Notifications,"alerts,health,ordering,precedence,templates,alarms",
170 +https://github.com/netdata/netdata/edit/master/src/health/overriding-stock-alerts.md,Overriding Stock Alerts,Published,Alerts & Notifications,"alerts,health,override,stock,templates,alarms",
171 https://github.com/netdata/netdata/edit/master/src/web/api/health/README.md,Health API Calls,Published,Alerts & Notifications,,
172 ,,,,,
173 https://github.com/netdata/netdata/edit/master/docs/category-overview-pages/machine-learning-and-assisted-troubleshooting.md,Netdata AI,Published,Netdata AI,,
src/health/REFERENCE.md
+3 -1
@@ -315,7 +315,9 @@ Complete syntax reference for all alert configuration options. Use this section
315
316 **Precedence**
317
318 -Alerts have higher precedence and will override templates. If you have `alarm` and `template` entities with the same name attached to the same chart, Netdata will use `alarm`.
318 +Alarms are processed before templates. If you have `alarm` and `template` entities with the same name that both match the same chart, only the `alarm` will create an active alert for that chart.
319 +
320 +For complete details on configuration loading order and precedence rules, see [Alert Configuration Ordering](alert-configuration-ordering.md).
321
322 :::
323
src/health/alert-configuration-ordering.md new
+188
@@ -0,0 +1,188 @@
1 +# Alert Configuration Ordering
2 +
3 +This document explains how Netdata's alerting system is designed and how it determines which alert definition applies when multiple definitions could match the same data.
4 +
5 +## The Problem: Alerts at Scale
6 +
7 +Netdata monitors infrastructure that can range from a single server to thousands of nodes, each with dozens of components: disks, network interfaces, databases, containers, and more.
8 +
9 +The challenge: How do you configure alerts that automatically apply to all Redis instances, all disks, or all network interfaces—without manually defining alerts for each one? And when one specific instance needs different thresholds, how do you override just that one predictably?
10 +
11 +## The Solution: Contexts and Instances
12 +
13 +Netdata's alerting system is built around two concepts:
14 +
15 +| Concept | What it is | Example |
16 +|---------|------------|---------|
17 +| **Context** | Defines what the metrics ARE—their meaning and units | `disk.space` (disk space utilization in %) |
18 +| **Instance** | An individual component being monitored | `/mnt/data`, `/home`, `/var` |
19 +
20 +A context groups all instances that share the same metric definition. For example, the `disk.space` context includes every mounted filesystem on the system.
21 +
22 +## Templates vs Alarms
23 +
24 +Netdata provides two ways to define alerts:
25 +
26 +### Templates: Match by Context
27 +
28 +A **template** applies to ALL instances of a context automatically.
29 +
30 +```yaml
31 +template: disk_space_usage
32 + on: disk.space # matches the CONTEXT
33 + lookup: max -1m percentage of avail
34 + warn: $this < 20
35 + crit: $this < 10
36 +```
37 +
38 +This single definition creates alerts for every disk on every node—automatically. When a new disk is mounted, it gets this alert. No manual configuration needed.
39 +
40 +### Alarms: Match by Instance
41 +
42 +An **alarm** applies to ONE specific instance.
43 +
44 +```yaml
45 +alarm: disk_space_usage
46 + on: disk_space._mnt_data # matches a specific INSTANCE (chart ID)
47 + lookup: max -1m percentage of avail
48 + warn: $this < 5
49 + crit: $this < 2
50 +```
51 +
52 +Use alarms when a specific instance needs different treatment—like a data disk that's expected to run fuller than others.
53 +
54 +**Key difference**: The `on:` line specifies a **context** for templates, but a **chart ID** (specific instance) for alarms.
55 +
56 +## Precedence: Same Name Required
57 +
58 +Multiple alerts with **different names** can coexist on the same instance. You can have `disk_space_usage`, `disk_io_latency`, and `disk_errors` all monitoring the same disk.
59 +
60 +The precedence rules only apply when alerts have the **same name**. In that case, only one alert with that name can exist per instance:
61 +
62 +| Priority | Type | What it matches |
63 +|----------|------|-----------------|
64 +| 1 (higher) | Alarm | Specific instance |
65 +| 2 (lower) | Template | All instances of a context |
66 +
67 +**Example scenario:**
68 +
69 +1. Stock template `disk_space_usage` on context `disk.space` → warn at 20%
70 +2. User alarm `disk_space_usage` on instance `disk_space._mnt_data` → warn at 5%
71 +
72 +Both have the **same name** (`disk_space_usage`), so precedence applies:
73 +- `/mnt/data` gets the alarm's thresholds (warn at 5%)
74 +- All other disks get the template's thresholds (warn at 20%)
75 +
76 +## Configuration Loading Order
77 +
78 +Netdata loads alert configurations from two directories:
79 +
80 +1. **User config** (loaded first): `/etc/netdata/health.d/` (default)
81 +2. **Stock config** (loaded second): `/usr/lib/netdata/conf.d/health.d/` (default)
82 +
83 +These paths can vary by installation. Check your `netdata.conf` `[directories]` section for exact paths.
84 +
85 +### File Shadowing
86 +
87 +If a file with the **same name** exists in both directories, only the user file is loaded. The stock file is completely ignored.
88 +
89 +**Example:**
90 +- Stock: `/usr/lib/netdata/conf.d/health.d/cpu.conf`
91 +- User: `/etc/netdata/health.d/cpu.conf`
92 +- Result: Only the user file is loaded
93 +
94 +This means if you copy a stock file to override it, you must include **all** alerts you want from that file, not just the ones you're modifying.
95 +
96 +### Complete Precedence
97 +
98 +Combining type precedence with source precedence:
99 +
100 +| Priority | Type | Source |
101 +|----------|------|--------|
102 +| 1 (highest) | Alarm | User config |
103 +| 2 | Alarm | Stock config |
104 +| 3 | Template | User config |
105 +| 4 (lowest) | Template | Stock config |
106 +
107 +## First-Match-Wins (Same Name Only)
108 +
109 +Only **one** alert can exist per (instance, alert_name) pair. When multiple definitions **with the same name** could apply to an instance:
110 +
111 +1. The first matching definition (by precedence) creates the alert
112 +2. Later definitions with the same name are skipped for that instance
113 +
114 +This is why overriding works: create an alert with the same name, and yours is processed first.
115 +
116 +## Dynamic Configuration Exception
117 +
118 +Alerts created or modified through the Netdata UI or API behave differently:
119 +
120 +- **UI/API changes replace** any existing definition with the same name
121 +- This is the only case where a definition overwrites another
122 +
123 +When you edit an alert through the dashboard, it completely replaces any file-based definition with that name.
124 +
125 +## Summary
126 +
127 +| Goal | Use |
128 +|------|-----|
129 +| Alert on ALL instances of a type | Template matching a context |
130 +| Alert on ONE specific instance | Alarm matching a chart ID |
131 +| Override a stock alert globally | User template with same name |
132 +| Override for just one instance | User alarm with same name |
133 +
134 +## FAQ
135 +
136 +### Can I have multiple alerts monitoring the same instance?
137 +
138 +Yes. Different alert names create independent alerts. You can have `disk_space_usage`, `disk_io_latency`, and `disk_write_errors` all monitoring the same disk simultaneously.
139 +
140 +The "only one alert per instance" rule applies only to alerts **with the same name**.
141 +
142 +### What happens if I define the same alert name twice in my user config?
143 +
144 +The first one processed wins. Since file loading order within a directory is non-deterministic (depends on filesystem), keep all definitions for the same alert name in a single file to avoid surprises.
145 +
146 +### How do I know which alert definition is currently active?
147 +
148 +Query the API:
149 +```bash
150 +curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[] | select(.value.name == "alert_name") | .value'
151 +```
152 +
153 +Key fields to check:
154 +- `source`: which config file is active
155 +- `lookup_*`: data query parameters
156 +- `warn`, `crit`: threshold expressions
157 +
158 +Or check the Alerts tab in the dashboard—click on an alert to see its current configuration.
159 +
160 +### Why isn't my override working?
161 +
162 +Common causes:
163 +1. **Name mismatch**: Alert names are case-sensitive
164 +2. **Not reloaded**: Run `sudo netdatacli reload-health`
165 +3. **File permissions**: Netdata must be able to read your config file
166 +4. **Syntax error**: Check logs with `journalctl --namespace netdata -g health` or `grep -i health /var/log/netdata/error.log`
167 +
168 +### What's the difference between context and chart ID?
169 +
170 +- **Context** (`disk.space`): The metric type—shared by all instances
171 +- **Chart ID** (`disk_space._mnt_data`): A specific instance
172 +
173 +Templates use contexts. Alarms use chart IDs.
174 +
175 +### Do user configs completely replace stock configs?
176 +
177 +No. User configs are processed **before** stock configs, but both are loaded (unless file shadowing applies). Your alert with the same name wins because it's processed first, but you're not deleting the stock definition—just preempting it.
178 +
179 +### What is file shadowing?
180 +
181 +If a file with the **same filename** exists in both user and stock directories, only the user file is loaded. The stock file is completely ignored.
182 +
183 +This is different from alert-level overriding. With shadowing, you must include ALL alerts you want from that file.
184 +
185 +## Related Documentation
186 +
187 +- [Health Configuration Reference](REFERENCE.md)
188 +- [Overriding Stock Alerts](overriding-stock-alerts.md)
src/health/overriding-stock-alerts.md new
+349
@@ -0,0 +1,349 @@
1 +# Overriding Stock Alerts
2 +
3 +This guide explains how to customize Netdata's stock alerts. User configurations survive upgrades, making this the recommended approach.
4 +
5 +## Quick Reference
6 +
7 +| Goal | Method |
8 +|------|--------|
9 +| Change thresholds for ALL instances | Create a template with the same name |
10 +| Change thresholds for ONE instance | Create an alarm with the same name |
11 +| Disable an alert completely | Use `enabled alarms` in netdata.conf |
12 +| Silence notifications only | Set `to: silent` |
13 +
14 +## Understanding Overrides
15 +
16 +Netdata's alerting uses **templates** (match all instances of a context) and **alarms** (match one specific instance). Stock alerts are mostly templates—they apply to all disks, all CPUs, etc.
17 +
18 +To override, create an alert with the **same name**. User definitions are processed before stock definitions, so yours wins.
19 +
20 +See [Alert Configuration Ordering](alert-configuration-ordering.md) for the full conceptual explanation.
21 +
22 +## Where to Put Your Overrides
23 +
24 +**User config directory** (default): `/etc/netdata/health.d/`
25 +
26 +Files here survive upgrades. Stock files in `/usr/lib/netdata/conf.d/health.d/` are replaced during updates.
27 +
28 +Check your `netdata.conf` `[directories]` section for exact paths on your system.
29 +
30 +## Method 1: Override All Instances (Template)
31 +
32 +Create a template with the same name to change thresholds for ALL instances.
33 +
34 +**Example: Raise CPU steal thresholds globally**
35 +
36 +Stock alert in `/usr/lib/netdata/conf.d/health.d/cpu.conf`:
37 +```yaml
38 +template: 20min_steal_cpu
39 + on: system.cpu
40 + lookup: average -20m unaligned of steal
41 + units: %
42 + every: 5m
43 + warn: $this > (($status >= $WARNING) ? (5) : (10))
44 +```
45 +
46 +Your override in `/etc/netdata/health.d/my-overrides.conf`:
47 +```yaml
48 +template: 20min_steal_cpu
49 + on: system.cpu
50 + lookup: average -20m unaligned of steal
51 + units: %
52 + every: 5m
53 + warn: $this > (($status >= $WARNING) ? (10) : (20))
54 +```
55 +
56 +**Why it works:** Same name + same context. Your template is processed first, creating the alert. The stock template is then skipped.
57 +
58 +> **Note:** Most stock alerts are templates. If a stock alert is an alarm (rare), you must override it with an alarm, not a template—alarms are always processed before templates.
59 +
60 +## Method 2: Override One Instance (Alarm)
61 +
62 +Create an alarm to override thresholds for ONE specific instance while keeping stock thresholds for others.
63 +
64 +**Example: Different disk space threshold for `/mnt/data`**
65 +
66 +Stock template (applies to all disks):
67 +```yaml
68 +template: disk_space_usage
69 + on: disk.space
70 + lookup: max -1m percentage of avail
71 + warn: $this < 20
72 + crit: $this < 10
73 +```
74 +
75 +Your override in `/etc/netdata/health.d/my-overrides.conf`:
76 +```yaml
77 +alarm: disk_space_usage
78 + on: disk_space._mnt_data
79 + lookup: max -1m percentage of avail
80 + warn: $this < 5
81 + crit: $this < 2
82 +```
83 +
84 +**Why it works:**
85 +- Both have the **same name** (`disk_space_usage`)
86 +- Your **alarm** targets the specific chart ID `disk_space._mnt_data`
87 +- Alarms are processed before templates (when names match)
88 +- For `/mnt/data`: your alarm creates the alert, stock template is skipped
89 +- For all other disks: stock template creates alerts normally
90 +
91 +**Key difference:** Templates use `on:` with a **context** (`disk.space`). Alarms use `on:` with a **chart ID** (`disk_space._mnt_data`).
92 +
93 +### Finding Chart IDs
94 +
95 +To find the exact chart ID for an instance:
96 +
97 +```bash
98 +curl -s "http://localhost:19999/api/v1/charts" | grep -o '"id":"disk_space[^"]*"'
99 +```
100 +
101 +Or check the chart title in the Netdata dashboard—the chart ID is shown in the URL when you click on a chart.
102 +
103 +### Alternative: Using Chart Labels
104 +
105 +Instead of chart IDs, you can match by labels:
106 +
107 +```yaml
108 +template: disk_space_usage
109 + on: disk.space
110 +chart labels: mount_point=/mnt/data
111 + lookup: max -1m percentage of avail
112 + warn: $this < 5
113 + crit: $this < 2
114 +```
115 +
116 +Use labels when:
117 +- You want to target multiple instances sharing a label
118 +- Chart IDs are dynamic or unpredictable
119 +
120 +## Method 3: Copy Entire Stock File
121 +
122 +If you want to modify many alerts in one stock file, copy it entirely:
123 +
124 +```bash
125 +cd /etc/netdata
126 +sudo ./edit-config health.d/cpu.conf
127 +```
128 +
129 +**Important:** When a file with the same name exists in both directories, Netdata loads **only** the user file. The stock file is completely ignored.
130 +
131 +This means your copy must include ALL alerts you want—not just the ones you're changing.
132 +
133 +## Disabling Alerts
134 +
135 +### Option A: Global Disable
136 +
137 +In `/etc/netdata/netdata.conf`:
138 +
139 +```ini
140 +[health]
141 + enabled alarms = !20min_steal_cpu !disk_space_usage *
142 +```
143 +
144 +This disables `20min_steal_cpu` and `disk_space_usage` while keeping all other alerts (`*`).
145 +
146 +### Option B: Per-Alert Disable
147 +
148 +Create an override that never matches:
149 +
150 +```yaml
151 +template: 20min_steal_cpu
152 + on: system.cpu
153 +host labels: _hostname=!*
154 +```
155 +
156 +The pattern `!*` is a special disable shortcut—the health config parser recognizes it and disables the alert.
157 +
158 +### Option C: Silence Notifications Only
159 +
160 +Keep the alert monitoring but stop notifications:
161 +
162 +```yaml
163 +template: 20min_steal_cpu
164 + on: system.cpu
165 + lookup: average -20m unaligned of steal
166 + units: %
167 + every: 5m
168 + warn: $this > (($status >= $WARNING) ? (5) : (10))
169 + to: silent
170 +```
171 +
172 +The alert still appears in the dashboard but sends no notifications.
173 +
174 +## Applying Changes
175 +
176 +Reload the health configuration:
177 +
178 +```bash
179 +sudo netdatacli reload-health
180 +```
181 +
182 +If `netdatacli` isn't available, send `SIGUSR2` to the Netdata process.
183 +
184 +### Verify Your Override
185 +
186 +Check via API:
187 +```bash
188 +curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[] | select(.value.name == "20min_steal_cpu") | .value'
189 +```
190 +
191 +Key fields to check:
192 +- `source`: confirms which config file is active (user override vs stock)
193 +- `lookup_*`: data query parameters
194 +- `warn`, `crit`: threshold expressions
195 +
196 +Or navigate to the Alerts tab in the dashboard and verify thresholds match your override.
197 +
198 +### Check for Errors
199 +
200 +If your override isn't working, check the logs:
201 +
202 +```bash
203 +# systemd journal (most Linux distributions):
204 +journalctl --namespace netdata -g health --no-pager | tail -20
205 +
206 +# Log files (if journal not available):
207 +grep -i health /var/log/netdata/error.log | tail -20
208 +```
209 +
210 +Common issues:
211 +- Syntax errors in configuration
212 +- Alert name doesn't match exactly (case-sensitive)
213 +- File permissions prevent Netdata from reading your config
214 +
215 +## Troubleshooting
216 +
217 +### Override Not Taking Effect
218 +
219 +1. **Reload configuration**: `sudo netdatacli reload-health`
220 +2. **Check file permissions**: Netdata must be able to read your file
221 +3. **Verify exact name match**: Alert names are case-sensitive
222 +4. **Check for syntax errors**: Look in error.log
223 +
224 +### Both Stock and Override Alerts Appear
225 +
226 +This happens when matching criteria don't overlap. For example:
227 +- Your override has `host labels: production`
228 +- Stock alert has no host labels restriction
229 +
230 +Both can create alerts on different hosts. Ensure your override matches at least the same scope as the stock alert.
231 +
232 +### UI Edit Replaced My Config
233 +
234 +Editing an alert through the dashboard UI creates a dynamic configuration that **replaces** any file-based definition. To restore file-based behavior, remove the dynamic config through the UI or API.
235 +
236 +## FAQ
237 +
238 +### Do I need to copy all fields when overriding an alert?
239 +
240 +Yes. Your override is a complete alert definition, not a "patch" on the stock alert. Include all fields: `lookup`, `calc`, `warn`, `crit`, `units`, etc.
241 +
242 +If you omit a field, the alert uses its default value—not the stock alert's value.
243 +
244 +### How do I override the same alert differently on different hosts?
245 +
246 +Use `host labels` to create host-specific overrides:
247 +
248 +```yaml
249 +# Production servers: stricter thresholds
250 +template: cpu_usage
251 + on: system.cpu
252 +host labels: environment=production
253 + warn: $this > 70
254 +
255 +# Development servers: relaxed thresholds
256 +template: cpu_usage
257 + on: system.cpu
258 +host labels: environment=development
259 + warn: $this > 90
260 +```
261 +
262 +Both can coexist because they match different hosts.
263 +
264 +### How do I find what stock alerts exist?
265 +
266 +List all stock alert files:
267 +```bash
268 +ls /usr/lib/netdata/conf.d/health.d/
269 +```
270 +
271 +View a specific stock alert:
272 +```bash
273 +cat /usr/lib/netdata/conf.d/health.d/cpu.conf
274 +```
275 +
276 +Or use the API to list all alert names:
277 +```bash
278 +curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[].value.name' | sort -u
279 +```
280 +
281 +### Can I add new alerts without affecting stock alerts?
282 +
283 +Yes. Create alerts with **different names** than stock alerts. They'll coexist independently.
284 +
285 +```yaml
286 +# This is a NEW alert, not an override
287 +template: my_custom_disk_alert
288 + on: disk.space
289 + lookup: max -5m percentage of avail
290 + warn: $this < 15
291 +```
292 +
293 +### What happens to my overrides after a Netdata upgrade?
294 +
295 +User config files in `/etc/netdata/health.d/` are preserved. Stock files in `/usr/lib/netdata/conf.d/health.d/` are replaced.
296 +
297 +Your overrides continue working. However, if a stock alert is renamed or removed in a new version, your override may become orphaned (still works, but no longer overriding anything).
298 +
299 +### How do I override for multiple specific instances?
300 +
301 +Option 1: Create multiple alarms (one per instance):
302 +```yaml
303 +alarm: disk_space_usage
304 + on: disk_space._mnt_data
305 + warn: $this < 5
306 +
307 +alarm: disk_space_usage
308 + on: disk_space._mnt_backup
309 + warn: $this < 5
310 +```
311 +
312 +Option 2: Use chart labels if instances share a label:
313 +```yaml
314 +template: disk_space_usage
315 + on: disk.space
316 +chart labels: storage_tier=bulk
317 + warn: $this < 5
318 +```
319 +
320 +### Can I see what overrides are currently active?
321 +
322 +Check which config files Netdata loaded:
323 +```bash
324 +# systemd journal:
325 +journalctl --namespace netdata -g "health.*load\|health.*read" --no-pager
326 +
327 +# Log files:
328 +grep -iE "health.*(load|read)" /var/log/netdata/error.log
329 +```
330 +
331 +Compare your active alert config vs stock:
332 +```bash
333 +# Your override
334 +cat /etc/netdata/health.d/my-overrides.conf
335 +
336 +# Stock definition
337 +cat /usr/lib/netdata/conf.d/health.d/disks.conf
338 +```
339 +
340 +### Why does editing an alert in the UI override my file-based config?
341 +
342 +UI edits create dynamic configurations that take precedence over all file-based configs. This is by design—it allows quick adjustments without SSH access.
343 +
344 +To restore file-based control, remove the dynamic config through the UI (reset to default) or via the API.
345 +
346 +## Related Documentation
347 +
348 +- [Health Configuration Reference](REFERENCE.md)
349 +- [Alert Configuration Ordering](alert-configuration-ordering.md)