Code Commits Issues Pulls Sessions CI/CD

master

md 10.2 KB

View source Raw

Overriding Stock Alerts

This guide explains how to customize Netdata's stock alerts. User configurations survive upgrades, making this the recommended approach.

Quick Reference

Goal	Method
Change thresholds for ALL instances	Create a template with the same name
Change thresholds for ONE instance	Create an alarm with the same name
Disable an alert completely	Use `enabled alarms` in netdata.conf
Silence notifications only	Set `to: silent`

Understanding Overrides

Netdata's alerting uses templates (match all instances of a context) and alarms (match one specific instance). Stock alerts are mostly templates—they apply to all disks, all CPUs, etc.

To override, create an alert with the same name. User definitions are processed before stock definitions, so yours wins.

See Alert Configuration Ordering for the full conceptual explanation.

Where to Put Your Overrides

User config directory (default): /etc/netdata/health.d/

Files here survive upgrades. Stock files in /usr/lib/netdata/conf.d/health.d/ are replaced during updates.

Check your netdata.conf [directories] section for exact paths on your system.

Method 1: Override All Instances (Template)

Create a template with the same name to change thresholds for ALL instances.

Example: Raise CPU steal thresholds globally

Stock alert in /usr/lib/netdata/conf.d/health.d/cpu.conf: yaml template: 20min_steal_cpu on: system.cpu lookup: average -20m unaligned of steal units: % every: 5m warn: $this > (($status >= $WARNING) ? (5) : (10))

Your override in /etc/netdata/health.d/my-overrides.conf: yaml template: 20min_steal_cpu on: system.cpu lookup: average -20m unaligned of steal units: % every: 5m warn: $this > (($status >= $WARNING) ? (10) : (20))

Why it works: Same name + same context. Your template is processed first, creating the alert. The stock template is then skipped.

Note: Most stock alerts are templates. If a stock alert is an alarm (rare), you must override it with an alarm, not a template—alarms are always processed before templates.

Method 2: Override One Instance (Alarm)

Create an alarm to override thresholds for ONE specific instance while keeping stock thresholds for others.

Example: Different disk space threshold for /mnt/data

Stock template (applies to all disks): yaml template: disk_space_usage on: disk.space lookup: max -1m percentage of avail warn: $this < 20 crit: $this < 10

Your override in /etc/netdata/health.d/my-overrides.conf: yaml alarm: disk_space_usage on: disk_space._mnt_data lookup: max -1m percentage of avail warn: $this < 5 crit: $this < 2

Why it works: - Both have the same name (disk_space_usage) - Your alarm targets the specific chart ID disk_space._mnt_data - Alarms are processed before templates (when names match) - For /mnt/data: your alarm creates the alert, stock template is skipped - For all other disks: stock template creates alerts normally

Key difference: Templates use on: with a context (disk.space). Alarms use on: with a chart ID (disk_space._mnt_data).

Finding Chart IDs

To find the exact chart ID for an instance:

curl -s "http://localhost:19999/api/v1/charts" | grep -o '"id":"disk_space[^"]*"'

Or check the chart title in the Netdata dashboard—the chart ID is shown in the URL when you click on a chart.

Alternative: Using Chart Labels

Instead of chart IDs, you can match by labels:

template: disk_space_usage
      on: disk.space
chart labels: mount_point=/mnt/data
   lookup: max -1m percentage of avail
     warn: $this < 5
     crit: $this < 2

Use labels when: - You want to target multiple instances sharing a label - Chart IDs are dynamic or unpredictable

Method 3: Copy Entire Stock File

If you want to modify many alerts in one stock file, copy it entirely:

cd /etc/netdata
sudo ./edit-config health.d/cpu.conf

Important: When a file with the same name exists in both directories, Netdata loads only the user file. The stock file is completely ignored.

This means your copy must include ALL alerts you want—not just the ones you're changing.

Disabling Alerts

Option A: Global Disable

In /etc/netdata/netdata.conf:

[health]
    enabled alarms = !20min_steal_cpu !disk_space_usage *

This disables 20min_steal_cpu and disk_space_usage while keeping all other alerts (*).

Option B: Per-Alert Disable

Create an override that never matches:

template: 20min_steal_cpu
      on: system.cpu
host labels: _hostname=!*

The pattern !* is a special disable shortcut—the health config parser recognizes it and disables the alert.

Option C: Silence Notifications Only

Keep the alert monitoring but stop notifications:

template: 20min_steal_cpu
      on: system.cpu
  lookup: average -20m unaligned of steal
   units: %
   every: 5m
    warn: $this > (($status >= $WARNING) ? (5) : (10))
      to: silent

The alert still appears in the dashboard but sends no notifications.

Applying Changes

Reload the health configuration:

sudo netdatacli reload-health

If netdatacli isn't available, send SIGUSR2 to the Netdata process.

Verify Your Override

Check via API: bash curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[] | select(.value.name == "20min_steal_cpu") | .value'

Key fields to check: - source: confirms which config file is active (user override vs stock) - lookup_*: data query parameters - warn, crit: threshold expressions

Or navigate to the Alerts tab in the dashboard and verify thresholds match your override.

Check for Errors

If your override isn't working, check the logs:

# systemd journal (most Linux distributions):
journalctl --namespace netdata -g health --no-pager | tail -20

# Log files (if journal not available):
grep -i health /var/log/netdata/error.log | tail -20

Common issues: - Syntax errors in configuration - Alert name doesn't match exactly (case-sensitive) - File permissions prevent Netdata from reading your config

Troubleshooting

Override Not Taking Effect

Reload configuration: sudo netdatacli reload-health
Check file permissions: Netdata must be able to read your file
Verify exact name match: Alert names are case-sensitive
Check for syntax errors: Look in error.log

Both Stock and Override Alerts Appear

This happens when matching criteria don't overlap. For example: - Your override has host labels: production - Stock alert has no host labels restriction

Both can create alerts on different hosts. Ensure your override matches at least the same scope as the stock alert.

UI Edit Replaced My Config

Editing an alert through the dashboard UI creates a dynamic configuration that replaces any file-based definition. To restore file-based behavior, remove the dynamic config through the UI or API.

FAQ

Do I need to copy all fields when overriding an alert?

Yes. Your override is a complete alert definition, not a "patch" on the stock alert. Include all fields: lookup, calc, warn, crit, units, etc.

If you omit a field, the alert uses its default value—not the stock alert's value.

How do I override the same alert differently on different hosts?

Use host labels to create host-specific overrides:

# Production servers: stricter thresholds
template: cpu_usage
      on: system.cpu
host labels: environment=production
     warn: $this > 70

# Development servers: relaxed thresholds
template: cpu_usage
      on: system.cpu
host labels: environment=development
     warn: $this > 90

Both can coexist because they match different hosts.

How do I find what stock alerts exist?

List all stock alert files: bash ls /usr/lib/netdata/conf.d/health.d/

View a specific stock alert: bash cat /usr/lib/netdata/conf.d/health.d/cpu.conf

Or use the API to list all alert names: bash curl -s "http://localhost:19999/api/v1/alarms?all" | jq '.alarms | to_entries[].value.name' | sort -u

Can I add new alerts without affecting stock alerts?

Yes. Create alerts with different names than stock alerts. They'll coexist independently.

# This is a NEW alert, not an override
template: my_custom_disk_alert
      on: disk.space
   lookup: max -5m percentage of avail
     warn: $this < 15

What happens to my overrides after a Netdata upgrade?

User config files in /etc/netdata/health.d/ are preserved. Stock files in /usr/lib/netdata/conf.d/health.d/ are replaced.

Your overrides continue working. However, if a stock alert is renamed or removed in a new version, your override may become orphaned (still works, but no longer overriding anything).

How do I override for multiple specific instances?

Option 1: Create multiple alarms (one per instance): ```yaml alarm: disk_space_usage on: disk_space._mnt_data warn: $this < 5

alarm: disk_space_usage on: disk_space._mnt_backup warn: $this < 5 ```

Option 2: Use chart labels if instances share a label: yaml template: disk_space_usage on: disk.space chart labels: storage_tier=bulk warn: $this < 5

Can I see what overrides are currently active?

Check which config files Netdata loaded: ```bash

systemd journal:

journalctl --namespace netdata -g "health.*load|health.*read" --no-pager

Log files:

grep -iE "health.*(load|read)" /var/log/netdata/error.log ```

Compare your active alert config vs stock: ```bash

Your override

cat /etc/netdata/health.d/my-overrides.conf

Stock definition

cat /usr/lib/netdata/conf.d/health.d/disks.conf ```

Why does editing an alert in the UI override my file-based config?

UI edits create dynamic configurations that take precedence over all file-based configs. This is by design—it allows quick adjustments without SSH access.

To restore file-based control, remove the dynamic config through the UI (reset to default) or via the API.

#Overriding Stock Alerts

#Quick Reference

#Understanding Overrides

#Where to Put Your Overrides

#Method 1: Override All Instances (Template)

#Method 2: Override One Instance (Alarm)

#Finding Chart IDs

#Alternative: Using Chart Labels

#Method 3: Copy Entire Stock File

#Disabling Alerts

#Option A: Global Disable

#Option B: Per-Alert Disable

#Option C: Silence Notifications Only

#Applying Changes

#Verify Your Override

#Check for Errors

#Troubleshooting

#Override Not Taking Effect

#Both Stock and Override Alerts Appear

#UI Edit Replaced My Config

#FAQ

#Do I need to copy all fields when overriding an alert?

#How do I override the same alert differently on different hosts?

#How do I find what stock alerts exist?

#Can I add new alerts without affecting stock alerts?

#What happens to my overrides after a Netdata upgrade?

#How do I override for multiple specific instances?

#Can I see what overrides are currently active?

#systemd journal:

#Log files:

#Your override

#Stock definition

#Why does editing an alert in the UI override my file-based config?

#Related Documentation