Alerts and Notifications (#20085)
* Alerts and Notifications * format * update "creating alerts" --------- Co-authored-by: ilyam8 <ilya@netdata.cloud>
kanelatechnical committed
Apr 8, 2025 at 16:29 UTC
caf7a18601807450dedc93755a0c18a0d855786b
4 files changed
+401
-189
docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md
+82
-29
@@ -1,44 +1,97 @@
1
# Creating Alerts with Netdata Alerts Configuration Manager
2
3
-The Netdata Alerts Configuration Manager enables subscribers to easily set up Alerts directly from the Netdata Dashboard. More details on subscriptions can be found [here](https://www.netdata.cloud/pricing/).
3
+The **Netdata Alerts Configuration Manager** lets you create and fine-tune alerts directly from the Netdata Cloud Dashboard.
4
+To use this feature, you’ll need an active Netdata subscription. → [View subscription plans](https://www.netdata.cloud/pricing/)
5
5
-## Using the Alerts Configuration Manager
6
+---
7
7
-1. Navigate to the **Metrics** tab and select the chart you want to configure for Alerts.
8
-2. Click the **Alert icon** in the top right corner of the chart.
9
-3. The Alert Configuration Manager will open, showing the default thresholds. Modify these thresholds as needed; the Alert definition on the right will update automatically.
10
-4. For additional settings, toggle **Show advanced**.
11
-5. After configuring the Alert, copy the generated Alert definition from the code box. Paste this into an existing or new custom health configuration file located at `<path to netdata install>/etc/netdata/health.d/` on a Parent Agent or a Standalone Child Agent. The guide to edit health configuration files is available [here](/src/health/REFERENCE.md#edit-health-configuration-files).
12
-6. To activate the new Alert, run the command `<path to netdata install>/usr/sbin/netdatacli reload-health`.
8
+## Creating Alerts: Quick Guide
9
14
-## Alerts Configuration Manager Sections
10
+1. Navigate to **Metrics** in your Netdata Cloud Dashboard and locate your target chart.
11
+2. Click the **alert icon** (🔔) in the chart's right corner, then select **Add alert** from the popup menu.
12
+3. Set your thresholds and watch the definition update in real-time.
13
+4. When finished configuring your alert, click the **Submit to nodes** button.
14
16
-### Alert Detection Method
15
+---
16
18
-An Alert is triggered whenever a metric crosses a threshold:
17
+## Alert Detection Types
18
20
-- **Standard Threshold**: Triggered when a metric crosses a predefined value.
21
-- **Metric Variance**: Triggered based on the variance of the metric.
22
-- **Anomaly Rate**: Triggered based on the anomaly rate of the metric.
19
+Netdata supports different ways of triggering alerts:
20
24
-### Metrics Lookup, Filtering, and Formula Section
21
+| Type | Description |
22
+|---------------------|---------------------------------------------------------|
23
+| **Standard** | Fires when a metric crosses a set value |
24
+| **Metric Variance** | Fires based on variation in values over time |
25
+| **Anomaly Rate** | Fires when the anomaly rate exceeds a certain threshold |
26
26
-You can read more about the different options in the [Alerts reference documentation](/src/health/REFERENCE.md).
27
+Choose the type that best suits the behavior you want to monitor.
28
28
-- **Metrics Lookup**: Adjust the database lookup parameters directly in the UI, including method (`avg`, `sum`, `min`, `max`, etc.), computation style, dimensions, duration, and options like `absolute` or `percentage`.
29
-- **Alert Filtering**: The **show advanced** checkbox allows filtering of Alert health checks for specific infrastructure components. Options include selecting hosts, nodes, instances, chart labels, and operating systems.
30
-- **Formula / Calculation**: The **show advanced** checkbox allows defining a formula for the metric value, which is then used to set Alert thresholds.
29
+---
30
32
-### Alerting Conditions
31
+## Metrics Lookup & Filters
32
34
-- **Thresholds**: Set thresholds for warning and critical Alert states, specifying whether the Alert should trigger above or below these thresholds. Advanced settings allow for custom formulas.
35
-- **Recovery Thresholds**: Set thresholds for downgrading the Alert from critical to warning or from warning to clear.
36
-- **Check Interval**: Define how frequently the health check should run.
37
-- **Delay Notifications**: Manage notification delays for Alert escalations or de-escalations.
38
-- **Agent-Specific Options**: Options exclusive to the Netdata Agent, like repeat notification frequencies and notification recipients.
39
-- **Custom Exec Script**: Define custom scripts to execute when an Alert triggers.
33
+Click **Show advanced** in the Alert Configuration Manager to access these options.
34
41
-### Alert Name, Description, and Summary Section
35
+### Metrics Lookup
36
43
-- **Alert Template Name**: Provide a unique name for the Alert.
44
-- **Alert Template Description**: Offer a brief explanation of what the Alert
37
+You can define how Netdata should query the data before triggering an alert:
38
+
39
+| Parameter | Description |
40
+|--------------|--------------------------------------------------------------------|
41
+| `method` | How values are aggregated (`avg`, `min`, `max`, etc.) |
42
+| `duration` | Time window used for the check |
43
+| `dimensions` | Which metric dimensions to include |
44
+| `options` | Modify how values are interpreted (e.g., `percentage`, `absolute`) |
45
+
46
+### Filtering Targets
47
+
48
+Limit the alert to specific infrastructure components:
49
+
50
+- Hosts
51
+- Nodes
52
+- Instances
53
+- Operating systems
54
+- Chart labels
55
+
56
+### Formulas
57
+
58
+Use a custom formula to manipulate values before comparing against thresholds.
59
+
60
+Example:
61
+
62
+```txt
63
+(metric1 - metric2) / 100
64
+```
65
+
66
+---
67
+
68
+## Defining Alert Conditions
69
+
70
+You can control how and when alerts are triggered, escalated, or resolved.
71
+
72
+| Setting | Purpose |
73
+|-----------------------------|-----------------------------------------------------------------|
74
+| **Thresholds** | Define values for `warning` and `critical` states |
75
+| **Recovery thresholds** | Set when the alert should downgrade or clear |
76
+| **Check interval** | How often the alert check runs (e.g., every 10 seconds) |
77
+| **Notification delay** | Delay before sending notifications for state changes |
78
+| **Repeat notifications** | How often to resend an alert if the issue persists (Agent only) |
79
+| **Notification recipients** | Define who gets alerted (Agent only) |
80
+| **Custom exec script** | Run a custom shell script when an alert triggers |
81
+
82
+---
83
+
84
+## Naming and Documentation
85
+
86
+| Field | Description |
87
+|-----------------|-----------------------------------------------------|
88
+| **Alert Name** | A unique name for the alert |
89
+| **Description** | What the alert does, in one or two sentences |
90
+| **Summary** | Optional: a short summary for display in dashboards |
91
+
92
+---
93
+
94
+## Final Notes
95
+
96
+- You can apply alert definitions to **Parent Agents** or **Standalone Child Agents**
97
+- If you need help writing custom alerts, check the [full alert reference](/src/health/REFERENCE.md)
docs/alerts-and-notifications/notifications/README.md
+65
-4
@@ -1,9 +1,70 @@
1
# Notifications
2
3
-This section includes the documentation of the integrations for both of Netdata's notification methods.
3
+Netdata supports two ways to send alert notifications: **from Netdata Cloud** or **from the Netdata Agent**.
4
+You can use either—or both—depending on how your infrastructure is set up.
5
5
-<!-- following links are virtual links to a generated page, should not lead somewhere upon click from GitHub -->
6
+:::tip Need alerts fast?
7
+Use Cloud for a centralized setup, or Agent for full control on each node.
8
+:::
9
7
-- Netdata Cloud provides centralized alert notifications, using the health status data already sent to Netdata Cloud from connected nodes to send alerts to configured integrations. [Supported integrations](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications) include Amazon SNS, Discord, Slack, Splunk, and others.
10
+---
11
9
-- The Netdata Agent offers a [wider range of notification options](/docs/alerts-and-notifications/notifications/agent-dispatched-notifications) directly from the Agent itself. You can choose from over a dozen services, including email, Slack, PagerDuty, Twilio, and others, for more granular control over notifications on each node.
12
+## Notification Methods
13
+
14
+### Netdata Cloud (Centralized)
15
+
16
+Netdata Cloud collects alert data from all connected nodes and sends notifications through your configured integrations.
17
+
18
+→ [See supported Cloud integrations](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications)
19
+
20
+**Popular integrations:**
21
+
22
+- Amazon SNS
23
+- Slack
24
+- Discord
25
+- Splunk
26
+- Microsoft Teams
27
+
28
+---
29
+
30
+### Netdata Agent (Local)
31
+
32
+The Agent sends alerts directly from the node, even if it's offline or not connected to the Cloud.
33
+
34
+→ [See supported Agent integrations](/docs/alerts-and-notifications/notifications/agent-dispatched-notifications)
35
+
36
+**Popular integrations:**
37
+
38
+- Email
39
+- Slack
40
+- PagerDuty
41
+- Twilio
42
+- Telegram
43
+- Opsgenie
44
+
45
+---
46
+
47
+## Which One Should I Use?
48
+
49
+Choose the option that fits your needs:
50
+
51
+| Use Case | Best Option |
52
+|---------------------------------------|---------------|
53
+| Manage multiple nodes centrally | Netdata Cloud |
54
+| Fewer configs, alerts from one place | Netdata Cloud |
55
+| Full control at node level | Netdata Agent |
56
+| No internet or external dependencies | Netdata Agent |
57
+| Fine-tuned control per system/service | Netdata Agent |
58
+| Want both simplicity and flexibility | Use **both** |
59
+
60
+---
61
+
62
+## Next Steps
63
+
64
+- 🔧 [Set up Cloud Notifications](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications)
65
+- ⚙️ [Set up Agent Notifications](/docs/alerts-and-notifications/notifications/agent-dispatched-notifications)
66
+
67
+:::info Want help with alert customization?
68
+You can tune thresholds, write custom conditions, and control who gets notified.
69
+[Learn more here →](/src/health/REFERENCE.md)
70
+:::
\ No newline at end of file
src/health/README.md
+92
-10
@@ -1,19 +1,101 @@
1
-# Alerts and notifications
1
+# Alerts and Notifications
2
3
-Netdata offers two ways to receive alert notifications on external integrations. These methods work independently, which means you can enable both at the same time to send alert notifications to any number of endpoints.
3
+Netdata provides two ways to send alert notifications. You can use either one—or both—at the same time.
4
5
-Both methods use a node's health alerts to generate the content of a notification.
5
+Alerts are based on each node's health status. You can change thresholds, add new alerts, or silence specific ones using Netdata's alerting system.
6
7
-Read our documentation on [configuring alerts](/src/health/REFERENCE.md) to change the pre-configured thresholds or to create tailored alerts for your infrastructure.
7
+→ [See how to configure alerts](/src/health/REFERENCE.md)
8
9
-<!-- virtual links below, should not lead anywhere outside of the rendered Learn doc -->
9
+---
10
11
-- Netdata Cloud provides centralized alert notifications, utilizing the health status data already sent to Netdata Cloud from connected nodes to send alerts to configured integrations. [Supported integrations](/docs/alerts-&-notifications/notifications/centralized-cloud-notifications) include Amazon SNS, Discord, Slack, Splunk, and others.
11
+## How Alert Notifications Work
12
13
-- The Netdata Agent offers a [wider range of notification options](/docs/alerts-&-notifications/notifications/agent-dispatched-notifications) directly from the Agent itself. You can choose from over a dozen services, including email, Slack, PagerDuty, Twilio, and others, for more granular control over notifications on each node.
13
+| Method | Where Alerts Are Sent From | Customization | Highlights |
14
+|-------------------|----------------------------|---------------|-----------------------------------------------------------|
15
+| **Netdata Cloud** | Cloud UI | Medium | Centralized alerting using connected nodes’ health status |
16
+| **Netdata Agent** | Local Netdata Agent | High | Node-level alerting with wide integration support |
17
15
-The Netdata Agent is a health watchdog for the health and performance of your systems, services, and applications. We've worked closely with our community of DevOps engineers, SREs, and developers to define hundreds of production-ready alerts that work without any configuration.
18
+You can enable one or both methods depending on your needs.
19
17
-The Agent's health monitoring system is also dynamic and fully customizable. You can write entirely new alerts, tune the pre-configured alerts for every app/service [the Agent collects metrics from](/src/collectors/COLLECTORS.md), or silence anything you're not interested in. You can even power complex lookups by running statistical algorithms against your metrics.
20
+---
21
19
-You can [use various alert notification methods](/docs/alerts-and-notifications/notifications/README.md), [customize alerts](/src/health/REFERENCE.md), and [disable/silence](/src/health/REFERENCE.md#disable-or-silence-alerts) alerts.
22
+## Quick Start
23
+
24
+Use this table to choose and set up your preferred alerting method:
25
+
26
+| Option | Setup Location | Setup Effort | Best For |
27
+|-------------------|----------------------|--------------|-------------------------------|
28
+| **Netdata Cloud** | In the Cloud UI | Low | Teams managing multiple nodes |
29
+| **Netdata Agent** | On each Netdata node | Medium | Full control and flexibility |
30
+
31
+---
32
+
33
+## Example 1: Set Up Alerts via Netdata Cloud
34
+
35
+1. Connect your nodes to [Netdata Cloud](https://app.netdata.cloud/).
36
+2. In the UI, go to:
37
+ `Space → Notifications`.
38
+3. Choose an integration (e.g. Slack, Amazon SNS, Splunk).
39
+4. Set alert severity filters as needed.
40
+
41
+→ [See all supported Cloud integrations](/docs/alerts-&-notifications/notifications/centralized-cloud-notifications)
42
+
43
+---
44
+
45
+## Example 2: Set Up Alerts via Netdata Agent
46
+
47
+1. Open the notification config:
48
+
49
+```bash
50
+sudo ./edit-config health_alarm_notify.conf
51
+```
52
+
53
+2. Enable your preferred method, for example email:
54
+
55
+```ini
56
+SEND_EMAIL="YES"
57
+DEFAULT_RECIPIENT_EMAIL="you@example.com"
58
+```
59
+
60
+3. Ensure your system can send mail (via `sendmail`, SMTP relay, etc.).
61
+4. Restart the agent:
62
+
63
+```bash
64
+sudo systemctl restart netdata
65
+```
66
+
67
+→ [See all Agent-based integrations](/docs/alerts-&-notifications/notifications/agent-dispatched-notifications)
68
+
69
+---
70
+
71
+## About the Agent's Health Monitoring
72
+
73
+The Netdata Agent continuously monitors system health and performance. It includes:
74
+
75
+- **Hundreds of pre-configured alerts** (covering system, app, and service metrics)
76
+- **No setup required** — works out of the box
77
+- **Dynamic customization** — you can fully control how, when, and what triggers an alert
78
+
79
+→ [See which collectors support alerts](/src/collectors/COLLECTORS.md)
80
+
81
+---
82
+
83
+## Customizing Alerts
84
+
85
+You can tune alerts to match your environment:
86
+
87
+- Adjust thresholds
88
+- Write custom alert conditions
89
+- Silence alerts temporarily or permanently
90
+- Use statistical functions for smarter alerting
91
+
92
+→ [Customize alerts](/src/health/REFERENCE.md)
93
+→ [Silence or disable alerts](/src/health/REFERENCE.md#disable-or-silence-alerts)
94
+
95
+---
96
+
97
+## Related Documentation
98
+
99
+- [All alert notification methods](/docs/alerts-and-notifications/notifications/README.md)
100
+- [Supported collectors](/src/collectors/COLLECTORS.md)
101
+- [Full alert reference](/src/health/REFERENCE.md)
\ No newline at end of file
src/health/notifications/README.md
+162
-146
@@ -1,207 +1,223 @@
1
-# Agent alert notifications
1
+# Agent Alert Notifications
2
3
-This is reference documentation for Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
3
+Netdata's Agent can send alert notifications directly from each node. It supports a wide range of services, multiple recipients, and role-based routing.
4
5
-The `script to execute on alarm` line in `netdata.conf` defines the external script that will be called once the alert is triggered.
5
+---
6
+
7
+## How It Works
8
+
9
+The Agent uses a notification script defined in `netdata.conf` under the `[health]` section:
10
+
11
+```ini
12
+script to execute on alarm = /usr/libexec/netdata/plugins.d/alarm-notify.sh
13
+```
14
15
The default script is `alarm-notify.sh`.
16
9
-> ### Info
10
->
11
-> This file mentions editing configuration files.
12
->
13
-> - To edit configuration files safely, we provide the [`edit config` script](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config)located in your [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.
14
-> Note that to run the script you need to be inside your Netdata config directory.
15
->
16
-> - Please also note that after most configuration changes, you will need to [restart the Agent](/docs/netdata-agent/start-stop-restart.md) for the changes to take effect.
17
->
18
-> It is recommended to use this way for configuring Netdata.
17
+This script handles:
18
+
19
+- Multiple recipients
20
+- Multiple notification methods
21
+- Role-based routing (e.g., `sysadmin`, `webmaster`, `dba`)
22
+
23
+---
24
+
25
+## Quick Setup
26
+
27
+:::tip Recommended
28
+Use the `edit-config` script to safely edit configuration files. It automatically creates the necessary files in the right place and opens them in your editor.
29
+→ [Learn how to use `edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config)
30
+:::
31
+
32
+1. Open the Agent’s health notification config:
33
+ ```bash
34
+ sudo ./edit-config health_alarm_notify.conf
35
+ ```
36
+
37
+2. Set up the required API keys or credentials for the service you want to use.
38
20
-You can change the default script globally by editing `netdata.conf` and changing the `script to execute on alarm` in the `[health]` section.
39
+3. Define recipients per **role** (see below).
40
22
-`alarm-notify.sh` is capable of sending notifications:
41
+4. Restart the Agent for changes to take effect:
42
+ ```bash
43
+ sudo systemctl restart netdata
44
+ ```
45
24
-- to multiple recipients
25
-- using multiple notification methods
26
-- filtering severity per recipient
46
+---
47
28
-It uses **roles**. For example `sysadmin`, `webmaster`, `dba`, etc.
48
+## Example: Alert with Role-Based Routing
49
30
-Each alert is assigned to one or more roles, using the `to` line of the alert configuration. For example, here is the alert configuration for `ram.conf` that defaults to the role `sysadmin`:
50
+Here’s an example alert assigned to the `sysadmin` role from the `ram.conf` file:
51
32
-```text
33
- alarm: ram_in_use
34
- on: system.ram
35
- class: Utilization
36
- type: System
52
+```ini
53
+alarm: ram_in_use
54
+ on: system.ram
55
+class: Utilization
56
+ type: System
57
component: Memory
38
- os: linux
39
- hosts: *
40
- calc: $used * 100 / ($used + $cached + $free + $buffers)
41
- units: %
42
- every: 10s
43
- warn: $this > (($status >= $WARNING) ? (80) : (90))
44
- crit: $this > (($status == $CRITICAL) ? (90) : (98))
45
- delay: down 15m multiplier 1.5 max 1h
46
- info: system memory utilization
47
- to: sysadmin
58
+ os: linux
59
+ hosts: *
60
+ calc: $used * 100 / ($used + $cached + $free + $buffers)
61
+ units: %
62
+ every: 10s
63
+ warn: $this > (($status >= $WARNING) ? (80) : (90))
64
+ crit: $this > (($status == $CRITICAL) ? (90) : (98))
65
+ delay: down 15m multiplier 1.5 max 1h
66
+ info: system memory utilization
67
+ to: sysadmin
68
```
69
50
-Then `alarm-notify.sh` uses its own configuration file `health_alarm_notify.conf`, which at the bottom of the file stores the recipients per role, for all notification methods.
70
+Then, in `health_alarm_notify.conf`, you assign recipients per notification method:
71
52
-Here is an example, of the `sysadmin`'s role recipients for the email notification.
53
-You can send the notification to multiple recipients by separating the emails with a space.
72
+```ini
73
+role_recipients_email[sysadmin]="admin1@example.com admin2@example.com"
74
+role_recipients_slack[sysadmin]="#alerts #infra"
75
+```
76
+
77
+---
78
55
-```text
79
+## Configuration Options
80
57
-###############################################################################
58
-# RECIPIENTS PER ROLE
81
+### Recipients Per Role
82
60
-# -----------------------------------------------------------------------------
61
-# generic system alerts
62
-# CPU, disks, network interfaces, entropy, etc
83
+Define who receives alerts and how:
84
64
-role_recipients_email[sysadmin]="someone@exaple.com someoneelse@example.com"
85
+```ini
86
+role_recipients_email[sysadmin]="team@example.com"
87
+role_recipients_telegram[webmaster]="123456789"
88
+role_recipients_slack[dba]="#database-alerts"
89
```
90
67
-Each role may have one or more destinations and one or more notification methods.
91
+Use spaces to separate multiple recipients.
92
69
-So, for example, the `sysadmin` role may send:
93
+To disable a notification method for a role, use:
94
71
-1. emails to admin1@example.com and admin2@example.com
72
-2. pushover.net notifications to USERTOKENS `A`, `B` and `C`.
73
-3. pushbullet.com push notifications to admin1@example.com and admin2@example.com
74
-4. messages to the `#alerts` and `#systems` channels of a Slack workspace.
75
-5. messages to Discord channels `#alerts` and `#systems`.
95
+```ini
96
+role_recipients_email[sysadmin]="disabled"
97
+```
98
77
-## Configuration
99
+If left empty, the default recipient for that method is used.
100
79
-You can edit `health_alarm_notify.conf` using the `edit-config` script to configure:
101
+---
102
81
-- **Settings** per notification method:
103
+### Alert Severity Filtering
104
83
- All notification methods, except email, require some configuration (i.e., API keys, tokens, destination rooms, channels, etc.). Please check this section's content to find the configuration guides for your notification option of choice
105
+You can limit certain recipients to only receive **critical** alerts:
106
85
-- **Recipients** per role per notification method
107
+```ini
108
+role_recipients_email[sysadmin]="user1@example.com user2@example.com|critical"
109
+```
110
87
- ```text
88
- role_recipients_email[sysadmin]="${DEFAULT_RECIPIENT_EMAIL}"
89
- role_recipients_pushover[sysadmin]="${DEFAULT_RECIPIENT_PUSHOVER}"
90
- role_recipients_pushbullet[sysadmin]="${DEFAULT_RECIPIENT_PUSHBULLET}"
91
- role_recipients_telegram[sysadmin]="${DEFAULT_RECIPIENT_TELEGRAM}"
92
- role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"
93
- ...
94
- ```
111
+This setup:
112
96
- Here you can change the `${DEFAULT_...}` values to the values of the recipients you want, separated by a space if you have multiple recipients.
113
+- Sends all alerts to `user1@example.com`
114
+- Sends only critical-related alerts to `user2@example.com`
115
98
-## Testing Alert Notifications
116
+Works for all supported methods: email, Slack, Telegram, Twilio, Discord, etc.
117
100
-You can run the following command by hand, to test alerts configuration:
118
+---
119
102
-```sh
103
-# become user netdata
104
-sudo su -s /bin/bash netdata
120
+### Proxy Settings
121
106
-# enable debugging info on the console
107
-export NETDATA_ALARM_NOTIFY_DEBUG=1
122
+To send notifications via a proxy, set these environment variables:
123
+
124
+```bash
125
+export http_proxy="http://10.0.0.1:3128/"
126
+export https_proxy="http://10.0.0.1:3128/"
127
+```
128
109
-# send test alerts to sysadmin
110
-/usr/libexec/netdata/plugins.d/alarm-notify.sh test
129
+---
130
112
-# send test alerts to any role
113
-/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
131
+### Notification Images
132
+
133
+By default, Netdata includes public image URLs in notifications (hosted by the global Registry).
134
+
135
+To use custom image paths:
136
+
137
+```ini
138
+images_base_url="http://my.public.netdata.server:19999"
139
```
140
116
-If you are [running your own Registry](/src/registry/CONFIGURATION.md#configure-a-custom-registry), add `export NETDATA_REGISTRY_URL=[YOUR_URL]` before calling `alarm-notify.sh`.
117
-
118
-> If you need to dig even deeper, you can trace the execution with `bash -x`. Note that in test mode, `alarm-notify.sh` calls itself with many more arguments. So first do:
119
->
120
->```sh
121
->bash -x /usr/libexec/netdata/plugins.d/alarm-notify.sh test
122
->```
123
->
124
-> And then look in the output for the alarm-notify.sh calls and run the one you want to trace with `bash -x`.
125
-
126
-## Global configuration options
127
-
128
-### Notification Filtering
129
-
130
-When you define recipients per role for notification methods, you can append `|critical` to limit the notifications that are sent.
131
-
132
-In the following examples, the first recipient receives all the alerts, while the second one receives only notifications for alerts that have at some point become critical.
133
-The second user may still receive warning and clear notifications, but only for the event that previously caused a critical alert.
134
-
135
-```text
136
- email : "user1@example.com user2@example.com|critical"
137
- pushover : "2987343...9437837 8756278...2362736|critical"
138
- telegram : "111827421 112746832|critical"
139
- slack : "alerts disasters|critical"
140
- alerta : "alerts disasters|critical"
141
- flock : "alerts disasters|critical"
142
- discord : "alerts disasters|critical"
143
- twilio : "+15555555555 +17777777777|critical"
144
- messagebird: "+15555555555 +17777777777|critical"
145
- kavenegar : "09155555555 09177777777|critical"
146
- pd : "<pd_service_key_1> <pd_service_key_2>|critical"
147
- irc : "<irc_channel_1> <irc_channel_2>|critical"
141
+---
142
+
143
+### Custom Date Format
144
+
145
+Change the timestamp format in notifications:
146
+
147
+```ini
148
+date_format="+%F %T%:z" # Example: RFC 3339
149
```
150
150
-If a per role recipient is set to an empty string, the default recipient of the given
151
-notification method (email, pushover, telegram, slack, alerta, etc.) will be used.
151
+Common formats:
152
153
-To disable a notification, use the recipient called: disabled
154
-This works for all notification methods (including the default recipients).
153
+| Format | String |
154
+|--------------------|-----------------------------|
155
+| ISO 8601 | `+%FT%T%z` |
156
+| RFC 5322 | `+%a, %d %b %Y %H:%M:%S %z` |
157
+| RFC 3339 | `+%F %T%:z` |
158
+| Local time | `+%x %X` |
159
+| ANSI C / asctime() | *(leave empty)* |
160
156
-### Proxy configuration
161
+→ See `man date` for more formatting options.
162
158
-If you need to send curl-based notifications (pushover, pushbullet, slack, alerta,
159
-flock, discord, telegram) via a proxy, you should set these variables to your proxy address:
163
+---
164
161
-```text
162
-export http_proxy="http://10.0.0.1:3128/"
163
-export https_proxy="http://10.0.0.1:3128/"
165
+### Hostname Format
166
+
167
+By default, Netdata uses the short hostname in notifications.
168
+
169
+To use the fully qualified domain name (FQDN), set:
170
+
171
+```ini
172
+use_fqdn=YES
173
```
174
166
-### Notification images
175
+If you’ve set a custom hostname in `netdata.conf`, that value takes priority.
176
168
-Images in notifications need to be downloaded from an Internet facing site.
177
+---
178
170
-To allow notification providers to fetch the icons/images, by default we set the URL of the global public Netdata Registry.
179
+## Testing Your Notification Setup
180
172
-If you have an Internet facing netdata (or you have copied the images/ folder
173
-of Netdata to your web server), set its URL here, to fetch the notification
174
-images from it.
181
+You can test alert notifications manually.
182
176
-```text
177
-images_base_url="http://my.public.netdata.server:19999"
183
+```bash
184
+# Switch to the Netdata user
185
+sudo su -s /bin/bash netdata
186
+
187
+# Enable debugging
188
+export NETDATA_ALARM_NOTIFY_DEBUG=1
189
+
190
+# Test default role (sysadmin)
191
+./plugins.d/alarm-notify.sh test
192
+
193
+# Test specific role
194
+./plugins.d/alarm-notify.sh test "webmaster"
195
```
196
180
-### Date handling
197
+:::info Using a custom Registry?
198
+If you’re running your own Netdata Registry, set:
199
182
-You can configure netdata alerts to send dates in any format you want via editing the `date_format` variable.
200
+```bash
201
+export NETDATA_REGISTRY_URL="https://your.registry.url"
202
+```
203
184
-This uses standard `date` command format strings. See `man date` for
185
-more info on what formats are supported.
204
+before testing.
205
+:::
206
187
-Note that this has to start with a '+'; otherwise it won't work.
207
+### Debugging with Trace
208
189
-- For ISO 8601 dates, use `+%FT%T%z`
190
-- For RFC 5322 dates, use `+%a, %d %b %Y %H:%M:%S %z`
191
-- For RFC 3339 dates, use `+%F %T%:z`
192
-- For RFC 1123 dates, use `+%a, %d %b %Y %H:%M:%S %Z`
193
-- For RFC 1036 dates, use `+%A, %d-%b-%y %H:%M:%S %Z`
194
-- For a reasonably local date and time (in that order), use `+%x %X`
195
-- For the old default behavior (compatible with ANSI C's `asctime()` function), leave the `date_format` field empty.
209
+To see full execution output:
210
197
-### Hostname handling
211
+```bash
212
+bash -x ./plugins.d/alarm-notify.sh test
213
+```
214
199
-By default, Netdata will use the simple hostname for the system (the hostname with everything after the first `.` removed) when displaying the hostname in alert notifications.
215
+Then look for the internal calls and re-run the one you want to trace in more detail.
216
201
-If you instead prefer to have Netdata use the host's fully qualified domain name, you can set `use_fdqn` to `YES`.
217
+---
218
203
-This setting does not account for child systems for which the system you are configuring is a parent.
219
+## Related Docs
220
205
-> ### Note
206
->
207
-> If the system's host name is overridden in `/etc/netdata.conf` with the `hostname` option, that name will be used unconditionally.
221
+- [How to configure alerts](/src/health/REFERENCE.md)
222
+- [Notification methods list](/docs/alerts-and-notifications/notifications/README.md#notification-methods)
223
+- [Netdata configuration basics](/docs/netdata-agent/configuration/README.md)
\ No newline at end of file