Update alerting and notification documentation Agent (#20329)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Kanela committed
May 22, 2025 at 18:15 UTC
ed4fe9cd517181707a1be8107dc452326846835e
4 files changed
+524
-126
docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md
+31
-21
@@ -1,22 +1,40 @@
1
# Creating Alerts with Netdata Alerts Configuration Manager
2
3
-The **Netdata Alerts Configuration Manager** lets you create and fine-tune alerts directly from the Netdata Cloud Dashboard.
4
-To use this feature, you’ll need an active Netdata subscription. → [View subscription plans](https://www.netdata.cloud/pricing/)
3
+The **Netdata Alerts Configuration Manager** lets you create and fine-tune alerts directly from the Netdata Cloud Dashboard.
4
6
----
5
+:::info
6
8
-## Creating Alerts: Quick Guide
7
+To use this feature, you'll need an active Netdata subscription. [View subscription plans](https://www.netdata.cloud/pricing/)
8
+
9
+:::
10
10
-1. Navigate to **Metrics** in your Netdata Cloud Dashboard and locate your target chart.
11
-2. Click the **alert icon** (🔔) in the chart's right corner, then select **Add alert** from the popup menu.
12
-3. Set your thresholds and watch the definition update in real-time.
13
-4. When finished configuring your alert, click the **Submit to nodes** button.
11
+## Creating Alerts: Quick Guide
12
15
----
13
+```mermaid
14
+%%{init: {'theme': 'default', 'themeVariables': {
15
+ 'primaryColor': '#2b2b2b',
16
+ 'primaryTextColor': '#fff',
17
+ 'primaryBorderColor': '#7C0000',
18
+ 'lineColor': '#F8B229',
19
+ 'secondaryColor': '#006100',
20
+ 'tertiaryColor': '#333',
21
+ 'fontFamily': 'arial',
22
+ 'fontSize': '16px'
23
+}}}%%
24
+flowchart LR
25
+ A[Navigate to Metrics] -->|Find Chart| B[Click Alert Icon]
26
+ B -->|Select Add Alert| C[Set Thresholds]
27
+ C -->|Configure Options| D[Submit to Nodes]
28
+
29
+ style A fill:#f9f9f9,stroke:#444,color:#000,stroke-width:1px,rx:10,ry:10
30
+ style B fill:#ffeb3b,stroke:#555,color:#000,stroke-width:1px,rx:10,ry:10
31
+ style C fill:#f44336,stroke:#333,color:#000,stroke-width:1px,rx:10,ry:10
32
+ style D fill:#4caf50,stroke:#333,color:#000,stroke-width:1px,rx:10,ry:10
33
+```
34
35
## Alert Detection Types
36
19
-Netdata supports different ways of triggering alerts:
37
+You can choose from different ways to trigger alerts based on your monitoring needs:
38
39
| Type | Description |
40
|---------------------|---------------------------------------------------------|
@@ -26,8 +44,6 @@ Netdata supports different ways of triggering alerts:
44
45
Choose the type that best suits the behavior you want to monitor.
46
29
----
30
-
47
## Metrics Lookup & Filters
48
49
Click **Show advanced** in the Alert Configuration Manager to access these options.
@@ -45,7 +61,7 @@ You can define how Netdata should query the data before triggering an alert:
61
62
### Filtering Targets
63
48
-Limit the alert to specific infrastructure components:
64
+You can limit your alert to specific infrastructure components:
65
66
- Hosts
67
- Nodes
@@ -55,7 +71,7 @@ Limit the alert to specific infrastructure components:
71
72
### Formulas
73
58
-Use a custom formula to manipulate values before comparing against thresholds.
74
+You can use a custom formula to manipulate values before comparing against thresholds.
75
76
Example:
77
@@ -63,11 +79,9 @@ Example:
79
(metric1 - metric2) / 100
80
```
81
66
----
67
-
82
## Defining Alert Conditions
83
70
-You can control how and when alerts are triggered, escalated, or resolved.
84
+You control how and when alerts are triggered, escalated, or resolved:
85
86
| Setting | Purpose |
87
|-----------------------------|-----------------------------------------------------------------|
@@ -79,8 +93,6 @@ You can control how and when alerts are triggered, escalated, or resolved.
93
| **Notification recipients** | Define who gets alerted (Agent only) |
94
| **Custom exec script** | Run a custom shell script when an alert triggers |
95
82
----
83
-
96
## Naming and Documentation
97
98
| Field | Description |
@@ -89,8 +101,6 @@ You can control how and when alerts are triggered, escalated, or resolved.
101
| **Description** | What the alert does, in one or two sentences |
102
| **Summary** | Optional: a short summary for display in dashboards |
103
92
----
93
-
104
## Final Notes
105
106
- You can apply alert definitions to **Parent Agents** or **Standalone Child Agents**
docs/alerts-and-notifications/notifications/README.md
+69
-18
@@ -1,13 +1,15 @@
1
# Notifications
2
3
-Netdata supports two ways to send alert notifications: **from Netdata Cloud** or **from the Netdata Agent**.
4
-You can use either—or both—depending on how your infrastructure is set up.
3
+Netdata supports two ways to send alert notifications: **from Netdata Cloud** or **from the Netdata Agent**.
4
6
-:::tip Need alerts fast?
5
+You can use either or both depending on how your infrastructure is set up.
6
+
7
+:::tip
8
+
9
+Need alerts *fast*?
10
Use Cloud for a centralized setup, or Agent for full control on each node.
8
-:::
11
10
----
12
+:::
13
14
## Notification Methods
15
@@ -15,7 +17,7 @@ Use Cloud for a centralized setup, or Agent for full control on each node.
17
18
Netdata Cloud collects alert data from all connected nodes and sends notifications through your configured integrations.
19
18
-→ [See supported Cloud integrations](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications)
20
+[See supported Cloud integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/centralized-cloud-notifications)
21
22
**Popular integrations:**
23
@@ -24,14 +26,13 @@ Netdata Cloud collects alert data from all connected nodes and sends notificatio
26
- Discord
27
- Splunk
28
- Microsoft Teams
27
-
28
----
29
+- Mattermost
30
31
### Netdata Agent (Local)
32
33
The Agent sends alerts directly from the node, even if it's offline or not connected to the Cloud.
34
34
-→ [See supported Agent integrations](/docs/alerts-and-notifications/notifications/agent-dispatched-notifications)
35
+[See supported Agent integrations](https://learn.netdata.cloud/docs/alerts-&-notifications/notifications/agent-dispatched-notifications)
36
37
**Popular integrations:**
38
@@ -42,7 +43,58 @@ The Agent sends alerts directly from the node, even if it's offline or not conne
43
- Telegram
44
- Opsgenie
45
45
----
46
+## Alert Intelligence and Severity Levels
47
+
48
+Netdata's alerts are designed to minimize false positives and prevent alarm fatigue.
49
+
50
+### Alert Severity Levels
51
+
52
+| Level | Description | Typical Action |
53
+|--------------|-------------------------------------------------------------------|-----------------------------------|
54
+| **CLEAR** | The metric has returned to normal range | No action needed |
55
+| **WARNING** | The metric shows concerning behavior that requires attention | Investigate during business hours |
56
+| **CRITICAL** | The metric indicates a serious problem requiring immediate action | Immediate response required |
57
+
58
+These severity levels help you prioritize your response and can be routed to different notification channels based on urgency.
59
+
60
+### Preventing Alert Fatigue
61
+
62
+| Feature | Benefit |
63
+|---------------------------|-----------------------------------------------------------------------|
64
+| **Intelligent Defaults** | Thresholds carefully selected based on real-world experience |
65
+| **Hysteresis Protection** | Prevents notification floods when metrics fluctuate around thresholds |
66
+| **Notification Delays** | Configurable delays ensure transient issues don't trigger alerts |
67
+| **Role-Based Routing** | Ensures alerts reach only the appropriate stakeholders |
68
+
69
+:::tip
70
+
71
+You can configure different notification channels for different severity levels. For example, you can send WARNING alerts to Slack and CRITICAL alerts to PagerDuty.
72
+
73
+:::
74
+
75
+## Troubleshooting Assistance
76
+
77
+When you receive an alert, Netdata provides tools to help you understand and resolve the issue:
78
+
79
+### Netdata Assistant
80
+
81
+The [Netdata Assistant](https://learn.netdata.cloud/docs/machine-learning-and-anomaly-detection/ai-powered-troubleshooting-assistant) is an AI-powered feature that guides you through troubleshooting alerts by providing:
82
+
83
+- Clear explanations of what the alert means
84
+- Assessment of potential causes
85
+- Recommended troubleshooting steps
86
+- Links to relevant documentation
87
+
88
+The Assistant window follows you as you navigate through dashboards, making troubleshooting faster and more efficient.
89
+
90
+### Community Resources
91
+
92
+For more complex issues, you can access the [Alerts Troubleshooting space](https://community.netdata.cloud/c/alerts/28) in our community forum, where you'll find:
93
+
94
+- Detailed information about all built-in alerts
95
+- Recommended troubleshooting actions from experts
96
+- A searchable history of previously solved issues
97
+- Ability to ask questions and share your own solutions
98
99
## Which One Should I Use?
100
@@ -57,14 +109,13 @@ Choose the option that fits your needs:
109
| Fine-tuned control per system/service | Netdata Agent |
110
| Want both simplicity and flexibility | Use **both** |
111
60
----
61
-
112
## Next Steps
113
64
-- 🔧 [Set up Cloud Notifications](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications)
65
-- ⚙️ [Set up Agent Notifications](/docs/alerts-and-notifications/notifications/agent-dispatched-notifications)
114
+- 🔧 [Set up Cloud Notifications](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md)
115
+- ⚙️ [Set up Agent Notifications](/src/health/notifications/README.md)
116
67
-:::info Want help with alert customization?
68
-You can tune thresholds, write custom conditions, and control who gets notified.
69
-[Learn more here →](/src/health/REFERENCE.md)
70
-:::
\ No newline at end of file
117
+:::info
118
+
119
+For additional alert customization options including threshold adjustments, custom conditions, and notification routing, check out our [alert configuration reference](https://learn.netdata.cloud/docs/alerts-&-notifications/alert-configuration-reference).
120
+
121
+:::
src/health/README.md
+134
-54
@@ -1,24 +1,24 @@
1
-# Alerts and Notifications
1
+# Alerts and Notifications in Netdata
2
3
-Netdata provides two ways to send alert notifications. You can use either one—or both—at the same time.
3
+Netdata provides two ways to send alert notifications. You can use either one or both at the same time.
4
5
-Alerts are based on each node's health status. You can change thresholds, add new alerts, or silence specific ones using Netdata's alerting system.
5
+:::tip
6
7
-→ [See how to configure alerts](/src/health/REFERENCE.md)
7
+Alerts are based on each node's health status. You can change thresholds, add new alerts, or silence-specific ones using Netdata's alerting system.
8
9
----
9
+[See how to configure alerts](/src/health/REFERENCE.md)
10
+
11
+:::
12
13
## How Alert Notifications Work
14
15
| Method | Where Alerts Are Sent From | Customization | Highlights |
16
|-------------------|----------------------------|---------------|-----------------------------------------------------------|
15
-| **Netdata Cloud** | Cloud UI | Medium | Centralized alerting using connected nodes’ health status |
17
+| **Netdata Cloud** | Cloud UI | Medium | Centralized alerting using connected nodes' health status |
18
| **Netdata Agent** | Local Netdata Agent | High | Node-level alerting with wide integration support |
19
20
You can enable one or both methods depending on your needs.
21
20
----
21
-
22
## Quick Start
23
24
Use this table to choose and set up your preferred alerting method:
@@ -28,74 +28,154 @@ Use this table to choose and set up your preferred alerting method:
28
| **Netdata Cloud** | In the Cloud UI | Low | Teams managing multiple nodes |
29
| **Netdata Agent** | On each Netdata node | Medium | Full control and flexibility |
30
31
----
32
-
33
-## Example 1: Set Up Alerts via Netdata Cloud
34
-
35
-1. Connect your nodes to [Netdata Cloud](https://app.netdata.cloud/).
36
-2. In the UI, go to:
37
- `Space → Notifications`.
38
-3. Choose an integration (e.g. Slack, Amazon SNS, Splunk).
39
-4. Set alert severity filters as needed.
31
+## Set Up Alerts via Netdata Cloud
32
41
-→ [See all supported Cloud integrations](/docs/alerts-&-notifications/notifications/centralized-cloud-notifications)
33
+1. Connect your nodes to [Netdata Cloud](https://app.netdata.cloud/)
34
+2. In the UI, go to: `Space → Notifications`
35
+3. Choose an integration (e.g. Slack, Amazon SNS, Splunk)
36
+4. Set alert severity filters as needed
37
43
----
38
+[See all supported Cloud integrations](/docs/alerts-&-notifications/notifications/centralized-cloud-notifications)
39
45
-## Example 2: Set Up Alerts via Netdata Agent
40
+## Set Up Alerts via Netdata Agent
41
42
1. Open the notification config:
43
49
-```bash
50
-sudo ./edit-config health_alarm_notify.conf
51
-```
44
+ ```bash
45
+ sudo ./edit-config health_alarm_notify.conf
46
+ ```
47
53
-2. Enable your preferred method, for example email:
48
+2. Enable your preferred method, for example, email:
49
55
-```ini
56
-SEND_EMAIL="YES"
57
-DEFAULT_RECIPIENT_EMAIL="you@example.com"
58
-```
50
+ ```ini
51
+ SEND_EMAIL="YES"
52
+ DEFAULT_RECIPIENT_EMAIL="you@example.com"
53
+ ```
54
60
-3. Ensure your system can send mail (via `sendmail`, SMTP relay, etc.).
55
+3. Ensure your system can send mail (via `sendmail`, SMTP relay, etc.)
56
4. Restart the agent:
57
63
-```bash
64
-sudo systemctl restart netdata
65
-```
58
+ ```bash
59
+ sudo systemctl restart netdata
60
+ ```
61
67
-→ [See all Agent-based integrations](/docs/alerts-&-notifications/notifications/agent-dispatched-notifications)
68
-
69
----
62
+[See all Agent-based integrations](/docs/alerts-&-notifications/notifications/agent-dispatched-notifications)
63
64
## About the Agent's Health Monitoring
65
73
-The Netdata Agent continuously monitors system health and performance. It includes:
74
-
75
-- **Hundreds of pre-configured alerts** (covering system, app, and service metrics)
76
-- **No setup required** — works out of the box
77
-- **Dynamic customization** — you can fully control how, when, and what triggers an alert
78
-
79
-→ [See which collectors support alerts](/src/collectors/COLLECTORS.md)
80
-
81
----
66
+The Netdata Agent continuously monitors system health and performance. It offers:
67
+
68
+- Hundreds of pre-configured alerts covering system, app, and service metrics
69
+- No setup is required to work out of the box
70
+- Dynamic customization you can fully control how, when, and what triggers an alert
71
+
72
+[See which collectors support alerts](/src/collectors/COLLECTORS.md)
73
+
74
+## Alert Philosophy and Design
75
+
76
+### The Anatomy of Intelligent Alerts
77
+
78
+| Component | Description |
79
+|---------------------|-------------------------------------------------------------------------------------------------------------------------|
80
+| **Metrics Data** | Collected every second (and at "event frequency" for eBPF metrics), giving you the most precise foundation for alerting |
81
+| **Filtering** | Alerts run against specific metrics series, with support for pattern matching, labels, hostnames, and operating systems |
82
+| **Frequency** | Configurable check intervals based on how quickly you need to know about issues |
83
+| **Templates** | Write once, apply everywhere with dimension templates and the ability to apply logic to multiple charts |
84
+| **Calculation** | Convert raw metrics into human-readable formats, like percentages, for easier understanding |
85
+| **Thresholds** | Pre-configured values that determine when metrics are outside acceptable ranges |
86
+| **Hysteresis** | Prevention of "flapping" alerts for metrics that hover around thresholds |
87
+| **Severity Levels** | CLEAR, WARNING, and CRITICAL states to indicate the urgency of action needed |
88
+
89
+### Preventing False Positives and Alert Fatigue
90
+
91
+| Feature | Benefit |
92
+|---------------------------|----------------------------------------------------------------------------|
93
+| **Intelligent Defaults** | Thresholds are carefully selected based on real-world experience |
94
+| **Dynamic Thresholds** | Some alerts adapt to your system's baseline performance |
95
+| **Hysteresis Protection** | Prevents floods of notifications when metrics fluctuate around a threshold |
96
+| **Notification Delays** | Configurable delays ensure transient issues don't trigger alerts |
97
+| **Severity Separation** | Different notification channels for different alert severities |
98
+| **Role-Based Routing** | Ensures alerts reach only the appropriate stakeholders |
99
+
100
+## Troubleshooting Alerts
101
+
102
+Netdata provides several tools to help you understand and resolve alert notifications:
103
+
104
+<details>
105
+<summary><strong>Netdata Assistant</strong></summary><br/>
106
+
107
+The [Netdata Assistant](https://learn.netdata.cloud/docs/machine-learning-and-anomaly-detection/ai-powered-troubleshooting-assistant) is an AI-powered feature that guides you through troubleshooting alerts. When an alert triggers, you can:
108
+
109
+1. Click on the alert in your Netdata Cloud dashboard
110
+2. Press the Assistant button to open a floating window
111
+3. Receive customized information and troubleshooting tips specific to that alert
112
+
113
+```mermaid
114
+%%{init: {'theme': 'default', 'themeVariables': {
115
+ 'primaryColor': '#2b2b2b',
116
+ 'primaryTextColor': '#fff',
117
+ 'primaryBorderColor': '#7C0000',
118
+ 'lineColor': '#F8B229',
119
+ 'secondaryColor': '#006100',
120
+ 'tertiaryColor': '#333',
121
+ 'fontFamily': 'arial',
122
+ 'fontSize': '16px'
123
+}}}%%
124
+flowchart TD
125
+ A[Alert Detected] -->|Analyze Context| B(Alert Assistant)
126
+ style A fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
127
+ style B fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
128
+
129
+ B --> C[Explanation of Alert]
130
+ B --> D[Possible Causes]
131
+ B --> E[Troubleshooting Steps]
132
+ B --> F[Documentation Links]
133
+
134
+ style C fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
135
+ style D fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
136
+ style E fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
137
+ style F fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
138
+```
139
83
-## Customizing Alerts
140
+The Assistant window follows you as you navigate through Netdata dashboards, making it easy to reference while investigating the issue.
141
+</details>
142
+
143
+<details>
144
+<summary><strong>Community Resources</strong></summary><br/>
145
+
146
+For more complex alert troubleshooting, Netdata maintains a dedicated [Alerts Troubleshooting space](https://community.netdata.cloud/c/alerts/28) in our community forum.
147
+
148
+```mermaid
149
+%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
150
+flowchart TD
151
+ A((Community<br>Forum)) --- B[Ask Questions]
152
+ A --- C[Share Solutions]
153
+ A --- D[Suggest Improvements]
154
+ A --- E[Learn from Others]
155
+ A --- F[Find Alert Info]
156
+
157
+ style A fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:20,ry:20
158
+ style B fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
159
+ style C fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
160
+ style D fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
161
+ style E fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
162
+ style F fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
163
+```
164
85
-You can tune alerts to match your environment:
165
+You can also get help through our [GitHub repository](https://github.com/netdata/netdata) or [Discord channel](https://discord.gg/kUk3nCmbtx) if you prefer those platforms. After resolving an issue, consider sharing your experience in the forum to help other users.
166
+</details>
167
87
-- Adjust thresholds
88
-- Write custom alert conditions
89
-- Silence alerts temporarily or permanently
90
-- Use statistical functions for smarter alerting
168
+<details>
169
+<summary><strong>Customizing Alerts</strong></summary><br/>
170
92
-→ [Customize alerts](/src/health/REFERENCE.md)
93
-→ [Silence or disable alerts](/src/health/REFERENCE.md#disable-or-silence-alerts)
171
+You can tune alerts to match your environment by adjusting thresholds, writing custom alert conditions, silencing alerts temporarily or permanently, and using statistical functions for smarter alerting.
172
95
----
173
+[Customize alerts](/src/health/REFERENCE.md)
174
+[Silence or disable alerts](/src/health/REFERENCE.md#disable-or-silence-alerts)
175
+</details>
176
177
## Related Documentation
178
179
- [All alert notification methods](/docs/alerts-and-notifications/notifications/README.md)
180
- [Supported collectors](/src/collectors/COLLECTORS.md)
101
-- [Full alert reference](/src/health/REFERENCE.md)
\ No newline at end of file
181
+- [Full alert reference](/src/health/REFERENCE.md)
src/health/notifications/README.md
+290
-33
@@ -2,8 +2,6 @@
2
3
Netdata's Agent can send alert notifications directly from each node. It supports a wide range of services, multiple recipients, and role-based routing.
4
5
----
6
-
5
## How It Works
6
7
The Agent uses a notification script defined in `netdata.conf` under the `[health]` section:
@@ -20,16 +18,95 @@ This script handles:
18
- Multiple notification methods
19
- Role-based routing (e.g., `sysadmin`, `webmaster`, `dba`)
20
23
----
21
+## Role-Based Routing Visualization
22
+
23
+```mermaid
24
+%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
25
+flowchart TD
26
+ Alert[High CPU Usage Alert] --> Check{Severity Level}
27
+ style Alert fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
28
+ style Check fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
29
+
30
+ Check -->|WARNING| WarningRouting{Role: SysAdmin}
31
+ Check -->|CRITICAL| CriticalRouting{Multiple Roles}
32
+
33
+ style WarningRouting fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
34
+ style CriticalRouting fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
35
+
36
+ WarningRouting --> SlackChannel[Slack]
37
+ WarningRouting --> EmailOps[Email]
38
+
39
+ CriticalRouting --> PagerDuty[PagerDuty]
40
+ CriticalRouting --> EmailManagers[Email]
41
+ CriticalRouting --> SlackUrgent[Slack]
42
+ CriticalRouting --> SMS[SMS]
43
+
44
+ style SlackChannel fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
45
+ style EmailOps fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
46
+ style PagerDuty fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
47
+ style EmailManagers fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
48
+ style SlackUrgent fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
49
+ style SMS fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
50
+```
51
+
52
+## Health Management API Workflow
53
+
54
+```mermaid
55
+%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
56
+flowchart TD
57
+ Start[Normal Operation] --> Maintenance{Maintenance<br/>Window?}
58
+ style Start fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
59
+ style Maintenance fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
60
+
61
+ Maintenance -->|No| NormalOps[Continue Normal Alerting]
62
+ Maintenance -->|Yes| ApiAction{Choose API Action}
63
+
64
+ style NormalOps fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
65
+ style ApiAction fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
66
+
67
+ ApiAction -->|Silence All| SilenceAll[SILENCE ALL]
68
+ ApiAction -->|Disable All| DisableAll[DISABLE ALL]
69
+ ApiAction -->|Selective Silence| SilenceSelect[SILENCE Specific]
70
+
71
+ SilenceAll --> AlertMode["Alerts: Yes<br/>Notifications: No"]
72
+ DisableAll --> CheckMode["Alerts: No<br/>Notifications: No"]
73
+ SilenceSelect --> SelectMode["Selected: Silenced<br/>Others: Normal"]
74
+
75
+ style SilenceAll fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
76
+ style DisableAll fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
77
+ style SilenceSelect fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
78
+
79
+ style AlertMode fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
80
+ style CheckMode fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
81
+ style SelectMode fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
82
+
83
+ AlertMode --> MaintenanceEnd{Maintenance<br/>Complete?}
84
+ CheckMode --> MaintenanceEnd
85
+ SelectMode --> MaintenanceEnd
86
+
87
+ style MaintenanceEnd fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
88
+
89
+ MaintenanceEnd -->|Yes| Reset[RESET]
90
+ MaintenanceEnd -->|No| Continue[Continue Maintenance]
91
+
92
+ style Reset fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
93
+ style Continue fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
94
+
95
+ Reset --> Restored[Normal Operations Restored]
96
+
97
+ style Restored fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
98
+```
99
100
## Quick Setup
101
27
-:::tip Recommended
102
+:::tip
103
+
104
Use the `edit-config` script to safely edit configuration files. It automatically creates the necessary files in the right place and opens them in your editor.
29
-→ [Learn how to use `edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config)
105
+[Learn how to use `edit-config`](/docs/netdata-agent/configuration/README.md#edit-a-configuration-file-using-edit-config)
106
+
107
:::
108
32
-1. Open the Agent’s health notification config:
109
+1. Open the Agent's health notification config:
110
```bash
111
sudo ./edit-config health_alarm_notify.conf
112
```
@@ -43,11 +120,9 @@ Use the `edit-config` script to safely edit configuration files. It automaticall
120
sudo systemctl restart netdata
121
```
122
46
----
47
-
123
## Example: Alert with Role-Based Routing
124
50
-Here’s an example alert assigned to the `sysadmin` role from the `ram.conf` file:
125
+Here's an example alert assigned to the `sysadmin` role from the `ram.conf` file:
126
127
```ini
128
alarm: ram_in_use
@@ -74,11 +149,130 @@ role_recipients_email[sysadmin]="admin1@example.com admin2@example.com"
149
role_recipients_slack[sysadmin]="#alerts #infra"
150
```
151
77
----
152
+## Advanced Role-Based Routing Examples
153
+
154
+<details>
155
+<summary><strong>DevOps Team Example</strong></summary><br/>
156
+
157
+```ini
158
+# Backend team receives database and application server alerts
159
+role_recipients_slack[backend]="#backend-team"
160
+role_recipients_pagerduty[backend]="PDK3Y5EXAMPLE"
161
+
162
+# Frontend team receives web server and CDN alerts
163
+role_recipients_slack[frontend]="#frontend-team"
164
+role_recipients_opsgenie[frontend]="key1example"
165
+
166
+# Security team receives all security-related alerts
167
+role_recipients_email[security]="security@example.com"
168
+role_recipients_slack[security]="#security-alerts"
169
+
170
+# SRE team receives critical infrastructure alerts 24/7
171
+role_recipients_slack[sre]="#sre-alerts"
172
+role_recipients_pagerduty[sre]="PDK3Y5SREXAMPLE"
173
+role_recipients_telegram[sre]="123456789"
174
+```
175
+
176
+</details>
177
+
178
+<details>
179
+<summary><strong>Time-Based Routing Example</strong></summary><br/>
180
+
181
+You can use external scripts to dynamically change recipients based on work hours, on-call schedules, etc.:
182
+
183
+```ini
184
+# Use a script to determine the current on-call engineer
185
+ONCALL_EMAIL=$(get_oncall_email.sh)
186
+role_recipients_email[oncall]="${ONCALL_EMAIL}"
187
+role_recipients_sms[oncall]="${ONCALL_PHONE}"
188
+
189
+# Standard business hours team gets non-critical alerts during work hours
190
+role_recipients_slack[business_hours]="#daytime-monitoring"
191
+```
192
+
193
+</details>
194
+
195
+## Health Management API
196
+
197
+Netdata provides a powerful Health Management API that lets you control alert behavior during maintenance windows, testing, or other planned activities.
198
+
199
+### API Authorization
200
+
201
+The API is protected by an authorization token stored in `/var/lib/netdata/netdata.api.key`:
202
+
203
+```bash
204
+# Get your token
205
+TOKEN=$(cat /var/lib/netdata/netdata.api.key)
206
+
207
+# Use the token in API calls
208
+curl "http://localhost:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: ${TOKEN}"
209
+```
210
+
211
+### Common API Commands
212
+
213
+<details>
214
+<summary><strong>Disable All Health Checks</strong></summary><br/>
215
+
216
+Completely stops evaluation of health checks during maintenance:
217
+
218
+```bash
219
+curl "http://localhost:19999/api/v1/manage/health?cmd=DISABLE ALL" -H "X-Auth-Token: ${TOKEN}"
220
+```
221
+
222
+</details>
223
+
224
+<details>
225
+<summary><strong>Silence All Notifications</strong></summary><br/>
226
+
227
+Continues to evaluate health checks but prevents notifications:
228
+
229
+```bash
230
+curl "http://localhost:19999/api/v1/manage/health?cmd=SILENCE ALL" -H "X-Auth-Token: ${TOKEN}"
231
+```
232
+
233
+</details>
234
+
235
+<details>
236
+<summary><strong>Disable Specific Alerts</strong></summary><br/>
237
+
238
+Target only certain alerts by name, chart, context, host, or family:
239
+
240
+```bash
241
+# Silence all disk space alerts
242
+curl "http://localhost:19999/api/v1/manage/health?cmd=SILENCE&context=disk_space" -H "X-Auth-Token: ${TOKEN}"
243
+
244
+# Disable CPU alerts for specific hosts
245
+curl "http://localhost:19999/api/v1/manage/health?cmd=DISABLE&context=cpu&hosts=prod-db-*" -H "X-Auth-Token: ${TOKEN}"
246
+```
247
+
248
+</details>
249
+
250
+<details>
251
+<summary><strong>View Current Silenced/Disabled Alerts</strong></summary><br/>
252
+
253
+Check what's currently silenced or disabled:
254
+
255
+```bash
256
+curl "http://localhost:19999/api/v1/manage/health?cmd=LIST" -H "X-Auth-Token: ${TOKEN}"
257
+```
258
+
259
+</details>
260
+
261
+<details>
262
+<summary><strong>Reset to Normal Operation</strong></summary><br/>
263
+
264
+Re-enable all health checks and notifications:
265
+
266
+```bash
267
+curl "http://localhost:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: ${TOKEN}"
268
+```
269
+
270
+</details>
271
272
## Configuration Options
273
81
-### Recipients Per Role
274
+<details>
275
+<summary><strong>Recipients Per Role</strong></summary><br/>
276
277
Define who receives alerts and how:
278
@@ -97,10 +291,10 @@ role_recipients_email[sysadmin]="disabled"
291
```
292
293
If left empty, the default recipient for that method is used.
294
+</details>
295
101
----
102
-
103
-### Alert Severity Filtering
296
+<details>
297
+<summary><strong>Alert Severity Filtering</strong></summary><br/>
298
299
You can limit certain recipients to only receive **critical** alerts:
300
@@ -114,10 +308,10 @@ This setup:
308
- Sends only critical-related alerts to `user2@example.com`
309
310
Works for all supported methods: email, Slack, Telegram, Twilio, Discord, etc.
311
+</details>
312
118
----
119
-
120
-### Proxy Settings
313
+<details>
314
+<summary><strong>Proxy Settings</strong></summary><br/>
315
316
To send notifications via a proxy, set these environment variables:
317
@@ -126,9 +320,10 @@ export http_proxy="http://10.0.0.1:3128/"
320
export https_proxy="http://10.0.0.1:3128/"
321
```
322
129
----
323
+</details>
324
131
-### Notification Images
325
+<details>
326
+<summary><strong>Notification Images</strong></summary><br/>
327
328
By default, Netdata includes public image URLs in notifications (hosted by the global Registry).
329
@@ -138,9 +333,10 @@ To use custom image paths:
333
images_base_url="http://my.public.netdata.server:19999"
334
```
335
141
----
336
+</details>
337
143
-### Custom Date Format
338
+<details>
339
+<summary><strong>Custom Date Format</strong></summary><br/>
340
341
Change the timestamp format in notifications:
342
@@ -158,11 +354,11 @@ Common formats:
354
| Local time | `+%x %X` |
355
| ANSI C / asctime() | *(leave empty)* |
356
161
-→ See `man date` for more formatting options.
357
+See `man date` for more formatting options.
358
+</details>
359
163
----
164
-
165
-### Hostname Format
360
+<details>
361
+<summary><strong>Hostname Format</strong></summary><br/>
362
363
By default, Netdata uses the short hostname in notifications.
364
@@ -172,9 +368,8 @@ To use the fully qualified domain name (FQDN), set:
368
use_fqdn=YES
369
```
370
175
-If you’ve set a custom hostname in `netdata.conf`, that value takes priority.
176
-
177
----
371
+If you've set a custom hostname in `netdata.conf`, that value takes priority.
372
+</details>
373
374
## Testing Your Notification Setup
375
@@ -194,19 +389,21 @@ export NETDATA_ALARM_NOTIFY_DEBUG=1
389
./plugins.d/alarm-notify.sh test "webmaster"
390
```
391
197
-:::info Using a custom Registry?
198
-If you’re running your own Netdata Registry, set:
392
+:::important
393
+
394
+If you're running your own Netdata Registry, set:
395
396
```bash
397
export NETDATA_REGISTRY_URL="https://your.registry.url"
398
```
399
400
before testing.
401
+
402
:::
403
404
### Debugging with Trace
405
209
-To see full execution output:
406
+To see the full execution output:
407
408
```bash
409
bash -x ./plugins.d/alarm-notify.sh test
@@ -214,10 +411,70 @@ bash -x ./plugins.d/alarm-notify.sh test
411
412
Then look for the internal calls and re-run the one you want to trace in more detail.
413
217
----
414
+## Troubleshooting Alert Notifications
415
+
416
+Here are solutions for common alert notification issues:
417
+
418
+### Email Notifications Not Working
419
+
420
+1. Verify your email configuration:
421
+ ```bash
422
+ grep -E "SEND_EMAIL|DEFAULT_RECIPIENT_EMAIL" /etc/netdata/health_alarm_notify.conf
423
+ ```
424
+
425
+2. Check if the system can send mail:
426
+ ```bash
427
+ echo "Test" | mail -s "Test Email" your@email.com
428
+ ```
429
+
430
+3. Look for errors in the Netdata log:
431
+ ```bash
432
+ tail -f /var/log/netdata/error.log | grep "alarm notify"
433
+ ```
434
+
435
+4. Test with debugging enabled:
436
+ ```bash
437
+ sudo su -s /bin/bash netdata
438
+ export NETDATA_ALARM_NOTIFY_DEBUG=1
439
+ ./plugins.d/alarm-notify.sh test
440
+ ```
441
+
442
+### Slack Notifications Failing
443
+
444
+1. Verify your webhook URL is correct:
445
+ ```bash
446
+ grep -E "SLACK_WEBHOOK_URL" /etc/netdata/health_alarm_notify.conf
447
+ ```
448
+
449
+2. Check for network connectivity to Slack:
450
+ ```bash
451
+ curl -X POST -H "Content-type: application/json" --data '{"text":"Test"}' YOUR_WEBHOOK_URL
452
+ ```
453
+
454
+3. Confirm channel names start with `#` in your configuration.
455
+
456
+### PagerDuty Integration Issues
457
+
458
+1. Verify your service key:
459
+ ```bash
460
+ grep -E "PAGERDUTY_SERVICE_KEY" /etc/netdata/health_alarm_notify.conf
461
+ ```
462
+
463
+2. Test the PagerDuty API directly:
464
+ ```bash
465
+ curl -H "Content-Type: application/json" -X POST -d '{"service_key":"YOUR_SERVICE_KEY","event_type":"trigger","description":"Test"}' https://events.pagerduty.com/generic/2010-04-15/create_event.json
466
+ ```
467
+
468
+### Notification Delays
469
+
470
+If notifications seem delayed:
471
+
472
+1. Check the `delay` parameter in your alarm configuration
473
+2. Verify your `health.d/*.conf` files for delay settings
474
+3. Check the `ALARM_NOTIFY_DELAY` setting in health_alarm_notify.conf
475
476
## Related Docs
477
478
- [How to configure alerts](/src/health/REFERENCE.md)
479
- [Notification methods list](/docs/alerts-and-notifications/notifications/README.md#notification-methods)
223
-- [Netdata configuration basics](/docs/netdata-agent/configuration/README.md)
\ No newline at end of file
480
+- [Netdata configuration basics](/docs/netdata-agent/configuration/README.md)