@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 463 lines 12.3 KB
Rendered Raw
1 # Agent Alert Notifications
2
3 Netdata's Agent can send alert notifications directly from each node. It supports a wide range of services, multiple recipients, and role-based routing.
4
5 ## How It Works
6
7 The Agent uses a notification script defined in `netdata.conf` under the `[health]` section:
8
9 ```ini
10 script to execute on alarm = /usr/libexec/netdata/plugins.d/alarm-notify.sh
11 ```
12
13 The default script is `alarm-notify.sh`.
14
15 This script handles:
16
17 - Multiple recipients
18 - Multiple notification methods
19 - Role-based routing (e.g., `sysadmin`, `webmaster`, `dba`)
20
21 ## Role-Based Routing Visualization
22
23 ```mermaid
24 flowchart TD
25 Alert("High CPU Usage Alert") --> Check("Severity Level")
26
27 Check -->|"WARNING"| WarningRouting("Role: SysAdmin")
28 Check -->|"CRITICAL"| CriticalRouting("Multiple Roles")
29
30 WarningRouting --> SlackChannel("Slack")
31 WarningRouting --> EmailOps("Email")
32
33 CriticalRouting --> PagerDuty("PagerDuty")
34 CriticalRouting --> EmailManagers("Email")
35 CriticalRouting --> SlackUrgent("Slack")
36 CriticalRouting --> SMS("SMS")
37
38 %% Style definitions
39 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
40 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
41 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
42 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
43
44 %% Apply styles
45 class Alert alert
46 class Check database
47 class WarningRouting,CriticalRouting complete
48 class SlackChannel,EmailOps,PagerDuty,EmailManagers,SlackUrgent,SMS neutral
49 ```
50
51 ## Health Management API Workflow
52
53 ```mermaid
54 flowchart TD
55 Start("Normal Operation") --> Maintenance("Maintenance Window?")
56
57 Maintenance -->|"No"| NormalOps("Continue Normal Alerting")
58 Maintenance -->|"Yes"| ApiAction("Choose Action")
59
60 ApiAction --> SilenceAll("SILENCE ALL")
61 ApiAction --> DisableAll("DISABLE ALL")
62 ApiAction --> SilenceSelect("SILENCE Specific")
63
64 SilenceAll --> Reset("RESET when done")
65 DisableAll --> Reset
66 SilenceSelect --> Reset
67
68 Reset --> Restored("Normal Operations Restored")
69
70 %% Style definitions
71 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
72 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
73 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
74 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px
75
76 %% Apply styles
77 class Start,Restored complete
78 class Maintenance database
79 class ApiAction alert
80 class NormalOps,SilenceAll,DisableAll,SilenceSelect,Reset neutral
81 ```
82
83 ## Quick Setup
84
85 :::tip
86
87 Use the `edit-config` script to safely edit configuration files. It automatically creates the necessary files in the right place and opens them in your editor.
88 [Learn how to use `edit-config`](/docs/netdata-agent/configuration/README.md#edit-configuration-files)
89
90 :::
91
92 1. Open the Agent's health notification config:
93 ```bash
94 sudo ./edit-config health_alarm_notify.conf
95 ```
96
97 2. Set up the required API keys or credentials for the service you want to use.
98
99 3. Define recipients per **role** (see below).
100
101 4. Restart the Agent for changes to take effect:
102 ```bash
103 sudo systemctl restart netdata
104 ```
105
106 ## Example: Alert with Role-Based Routing
107
108 Here's an example alert assigned to the `sysadmin` role from the `ram.conf` file:
109
110 ```ini
111 alarm: ram_in_use
112 on: system.ram
113 class: Utilization
114 type: System
115 component: Memory
116 os: linux
117 hosts: *
118 calc: $used * 100 / ($used + $cached + $free + $buffers)
119 units: %
120 every: 10s
121 warn: $this > (($status >= $WARNING) ? (80) : (90))
122 crit: $this > (($status == $CRITICAL) ? (90) : (98))
123 delay: down 15m multiplier 1.5 max 1h
124 info: system memory utilization
125 to: sysadmin
126 ```
127
128 Then, in `health_alarm_notify.conf`, you assign recipients per notification method:
129
130 ```ini
131 role_recipients_email[sysadmin]="admin1@example.com admin2@example.com"
132 role_recipients_slack[sysadmin]="#alerts #infra"
133 ```
134
135 ## Advanced Role-Based Routing Examples
136
137 <details>
138 <summary><strong>DevOps Team Example</strong></summary><br/>
139
140 ```ini
141 # Backend team receives database and application server alerts
142 role_recipients_slack[backend]="#backend-team"
143 role_recipients_pagerduty[backend]="PDK3Y5EXAMPLE"
144
145 # Frontend team receives web server and CDN alerts
146 role_recipients_slack[frontend]="#frontend-team"
147 role_recipients_opsgenie[frontend]="key1example"
148
149 # Security team receives all security-related alerts
150 role_recipients_email[security]="security@example.com"
151 role_recipients_slack[security]="#security-alerts"
152
153 # SRE team receives critical infrastructure alerts 24/7
154 role_recipients_slack[sre]="#sre-alerts"
155 role_recipients_pagerduty[sre]="PDK3Y5SREXAMPLE"
156 role_recipients_telegram[sre]="123456789"
157 ```
158
159 </details>
160
161 <details>
162 <summary><strong>Time-Based Routing Example</strong></summary><br/>
163
164 You can use external scripts to dynamically change recipients based on work hours, on-call schedules, etc.:
165
166 ```ini
167 # Use a script to determine the current on-call engineer
168 ONCALL_EMAIL=$(get_oncall_email.sh)
169 role_recipients_email[oncall]="${ONCALL_EMAIL}"
170 role_recipients_sms[oncall]="${ONCALL_PHONE}"
171
172 # Standard business hours team gets non-critical alerts during work hours
173 role_recipients_slack[business_hours]="#daytime-monitoring"
174 ```
175
176 </details>
177
178 ## Health Management API
179
180 Netdata provides a powerful Health Management API that lets you control alert behavior during maintenance windows, testing, or other planned activities.
181
182 ### API Authorization
183
184 The API is protected by an authorization token stored in `/var/lib/netdata/netdata.api.key`:
185
186 ```bash
187 # Get your token
188 TOKEN=$(cat /var/lib/netdata/netdata.api.key)
189
190 # Use the token in API calls
191 curl "http://localhost:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: ${TOKEN}"
192 ```
193
194 ### Common API Commands
195
196 <details>
197 <summary><strong>Disable All Health Checks</strong></summary><br/>
198
199 Completely stops evaluation of health checks during maintenance:
200
201 ```bash
202 curl "http://localhost:19999/api/v1/manage/health?cmd=DISABLE ALL" -H "X-Auth-Token: ${TOKEN}"
203 ```
204
205 </details>
206
207 <details>
208 <summary><strong>Silence All Notifications</strong></summary><br/>
209
210 Continues to evaluate health checks but prevents notifications:
211
212 ```bash
213 curl "http://localhost:19999/api/v1/manage/health?cmd=SILENCE ALL" -H "X-Auth-Token: ${TOKEN}"
214 ```
215
216 </details>
217
218 <details>
219 <summary><strong>Disable Specific Alerts</strong></summary><br/>
220
221 Target only certain alerts by name, chart, context, host, or family:
222
223 ```bash
224 # Silence all disk space alerts
225 curl "http://localhost:19999/api/v1/manage/health?cmd=SILENCE&context=disk_space" -H "X-Auth-Token: ${TOKEN}"
226
227 # Disable CPU alerts for specific hosts
228 curl "http://localhost:19999/api/v1/manage/health?cmd=DISABLE&context=cpu&hosts=prod-db-*" -H "X-Auth-Token: ${TOKEN}"
229 ```
230
231 </details>
232
233 <details>
234 <summary><strong>View Current Silenced/Disabled Alerts</strong></summary><br/>
235
236 Check what's currently silenced or disabled:
237
238 ```bash
239 curl "http://localhost:19999/api/v1/manage/health?cmd=LIST" -H "X-Auth-Token: ${TOKEN}"
240 ```
241
242 </details>
243
244 <details>
245 <summary><strong>Reset to Normal Operation</strong></summary><br/>
246
247 Re-enable all health checks and notifications:
248
249 ```bash
250 curl "http://localhost:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: ${TOKEN}"
251 ```
252
253 </details>
254
255 ## Configuration Options
256
257 <details>
258 <summary><strong>Recipients Per Role</strong></summary><br/>
259
260 Define who receives alerts and how:
261
262 ```ini
263 role_recipients_email[sysadmin]="team@example.com"
264 role_recipients_telegram[webmaster]="123456789"
265 role_recipients_slack[dba]="#database-alerts"
266 ```
267
268 Use spaces to separate multiple recipients.
269
270 To disable a notification method for a role, use:
271
272 ```ini
273 role_recipients_email[sysadmin]="disabled"
274 ```
275
276 If left empty, the default recipient for that method is used.
277 </details>
278
279 <details>
280 <summary><strong>Alert Severity Filtering</strong></summary><br/>
281
282 You can limit certain recipients to only receive **critical** alerts:
283
284 ```ini
285 role_recipients_email[sysadmin]="user1@example.com user2@example.com|critical"
286 ```
287
288 This setup:
289
290 - Sends all alerts to `user1@example.com`
291 - Sends only critical-related alerts to `user2@example.com`
292
293 Works for all supported methods: email, Slack, Telegram, Twilio, Discord, etc.
294 </details>
295
296 <details>
297 <summary><strong>Proxy Settings</strong></summary><br/>
298
299 To send notifications via a proxy, set these environment variables:
300
301 ```bash
302 export http_proxy="http://10.0.0.1:3128/"
303 export https_proxy="http://10.0.0.1:3128/"
304 ```
305
306 </details>
307
308 <details>
309 <summary><strong>Notification Images</strong></summary><br/>
310
311 By default, Netdata includes public image URLs in notifications (hosted by the global Registry).
312
313 To use custom image paths:
314
315 ```ini
316 images_base_url="http://my.public.netdata.server:19999"
317 ```
318
319 </details>
320
321 <details>
322 <summary><strong>Custom Date Format</strong></summary><br/>
323
324 Change the timestamp format in notifications:
325
326 ```ini
327 date_format="+%F %T%:z" # Example: RFC 3339
328 ```
329
330 Common formats:
331
332 | Format | String |
333 |--------------------|-----------------------------|
334 | ISO 8601 | `+%FT%T%z` |
335 | RFC 5322 | `+%a, %d %b %Y %H:%M:%S %z` |
336 | RFC 3339 | `+%F %T%:z` |
337 | Local time | `+%x %X` |
338 | ANSI C / asctime() | *(leave empty)* |
339
340 See `man date` for more formatting options.
341 </details>
342
343 <details>
344 <summary><strong>Hostname Format</strong></summary><br/>
345
346 By default, Netdata uses the short hostname in notifications.
347
348 To use the fully qualified domain name (FQDN), set:
349
350 ```ini
351 use_fqdn=YES
352 ```
353
354 If you've set a custom hostname in `netdata.conf`, that value takes priority.
355 </details>
356
357 ## Testing Your Notification Setup
358
359 You can test alert notifications manually.
360
361 ```bash
362 # Switch to the Netdata user
363 sudo su -s /bin/bash netdata
364
365 # Enable debugging
366 export NETDATA_ALARM_NOTIFY_DEBUG=1
367
368 # Test default role (sysadmin)
369 ./plugins.d/alarm-notify.sh test
370
371 # Test specific role
372 ./plugins.d/alarm-notify.sh test "webmaster"
373 ```
374
375 :::important
376
377 If you're running your own Netdata Registry, set:
378
379 ```bash
380 export NETDATA_REGISTRY_URL="https://your.registry.url"
381 ```
382
383 before testing.
384
385 :::
386
387 ### Debugging with Trace
388
389 To see the full execution output:
390
391 ```bash
392 bash -x ./plugins.d/alarm-notify.sh test
393 ```
394
395 Then look for the internal calls and re-run the one you want to trace in more detail.
396
397 ## Troubleshooting Alert Notifications
398
399 Here are solutions for common alert notification issues:
400
401 ### Email Notifications Not Working
402
403 1. Verify your email configuration:
404 ```bash
405 grep -E "SEND_EMAIL|DEFAULT_RECIPIENT_EMAIL" /etc/netdata/health_alarm_notify.conf
406 ```
407
408 2. Check if the system can send mail:
409 ```bash
410 echo "Test" | mail -s "Test Email" your@email.com
411 ```
412
413 3. Look for errors in the Netdata log:
414 ```bash
415 tail -f /var/log/netdata/error.log | grep "alarm notify"
416 ```
417
418 4. Test with debugging enabled:
419 ```bash
420 sudo su -s /bin/bash netdata
421 export NETDATA_ALARM_NOTIFY_DEBUG=1
422 ./plugins.d/alarm-notify.sh test
423 ```
424
425 ### Slack Notifications Failing
426
427 1. Verify your webhook URL is correct:
428 ```bash
429 grep -E "SLACK_WEBHOOK_URL" /etc/netdata/health_alarm_notify.conf
430 ```
431
432 2. Check for network connectivity to Slack:
433 ```bash
434 curl -X POST -H "Content-type: application/json" --data '{"text":"Test"}' YOUR_WEBHOOK_URL
435 ```
436
437 3. Confirm channel names start with `#` in your configuration.
438
439 ### PagerDuty Integration Issues
440
441 1. Verify your service key:
442 ```bash
443 grep -E "PAGERDUTY_SERVICE_KEY" /etc/netdata/health_alarm_notify.conf
444 ```
445
446 2. Test the PagerDuty API directly:
447 ```bash
448 curl -H "Content-Type: application/json" -X POST -d '{"service_key":"YOUR_SERVICE_KEY","event_type":"trigger","description":"Test"}' https://events.pagerduty.com/generic/2010-04-15/create_event.json
449 ```
450
451 ### Notification Delays
452
453 If notifications seem delayed:
454
455 1. Check the `delay` parameter in your alarm configuration
456 2. Verify your `health.d/*.conf` files for delay settings
457 3. Check the `ALARM_NOTIFY_DELAY` setting in health_alarm_notify.conf
458
459 ## Related Docs
460
461 - [How to configure alerts](/src/health/REFERENCE.md)
462 - [Notification methods list](/docs/alerts-and-notifications/notifications/README.md#notification-methods)
463 - [Netdata configuration basics](/docs/netdata-agent/configuration/README.md)