@cryptotaxi247 / netdata-1 / commits / 83b693987

Update Agent notification methods documentation (#14827)

* update health/notifications/README.md * Alerta notification method documentation update * Amazon SNS and some alerta changes * notification methods imporvements * alerta refinements * awssns refinements * custom alert refinements * discord refinements * email notifications documentation update * flock notifications documentation update * alerta edits * awssns edits * custom notification method edits * discord edits * email notification method edits * flock edits * IRC notifications update * Kavenegar notifications documentation update * matrix notifications documentation update * messagebird notifications documentation update * msteams notifications documentation update * wording change * twilio notifications documentation update * telegram notifications documentation update * syslog notifications update * smstools3 notifications documentation update * rocket.chat notifications documentation update * pushover notifications documentation update * pushbullet notifications documentation update * prowl notifications documentation update * pagerduty notifications documentation update * remove comments from example configuration * slight wording changes * more notification methods documentation updates * slack notification documentation update * add config options to the notifications Introduction page * crop image twilio * crop image slack * crop image pushover * crop images pushbullet * crop image messagebird * crop image kavenegar

Fotis Voutsas committed Apr 4, 2023 at 19:38 UTC 83b693987771ced3e9535a55a69403fa1242cc09

25 files changed +1567 -986

health/notifications/README.md

+163 -47

@@ -1,71 +1,107 @@

1 -<!--

2 -title: "Agent alert notifications"

3 -description: "Reference documentation for Netdata's alarm notification feature, which supports dozens of endpoints, user roles, and more."

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/README.md"

5 -sidebar_label: "Agent alert notifications"

6 -learn_status: "Published"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 --->

9 -

1 # Agent alert notifications

12 -The `exec` line in health configuration defines an external script that will be called once

13 -the alert is triggered. The default script is `alarm-notify.sh`.

3 +This is a reference documentation for Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +The `script to execute on alarm` line in `netdata.conf` defines the external script that will be called once the alert is triggered.

6 +

7 +The default script is `alarm-notify.sh`.

15 -You can change the default script globally by editing `/etc/netdata/netdata.conf`.

9 +> ### Info

10 +>

11 +> This file mentions editing configuration files.

12 +>

13 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

14 +> Note that to run the script you need to be inside your Netdata config directory.

15 +>

16 +> - Please also note that after most configuration changes you will need to [restart the Agent](https://github.com/netdata/netdata/blob/master/docs/configure/start-stop-restart.md) for the changes to take effect.

17 +>

18 +> It is recommended to use this way for configuring Netdata.

19 +

20 +You can change the default script globally by editing `netdata.conf` and changing the `script to execute on alarm` in the `[health]` section.

22 `alarm-notify.sh` is capable of sending notifications:

19 -- to multiple recipients

20 -- using multiple notification methods

21 -- filtering severity per recipient

24 +- to multiple recipients

25 +- using multiple notification methods

26 +- filtering severity per recipient

28 It uses **roles**. For example `sysadmin`, `webmaster`, `dba`, etc.

25 -Each alert is assigned to one or more roles, using the `to` line of the alert configuration. Then `alarm-notify.sh` uses

26 -its own configuration file `/etc/netdata/health_alarm_notify.conf`. To edit it on your system, run

27 -`/etc/netdata/edit-config health_alarm_notify.conf` and find the destination address of the notification for each

28 -method.

30 +Each alert is assigned to one or more roles, using the `to` line of the alert configuration. For example, here is the alert configuration for `ram.conf` that defaults to the role `sysadmin`:

31 +

32 +```conf

33 + alarm: ram_in_use

34 + on: system.ram

35 + class: Utilization

36 + type: System

37 +component: Memory

38 + os: linux

39 + hosts: *

40 + calc: $used * 100 / ($used + $cached + $free + $buffers)

41 + units: %

42 + every: 10s

43 + warn: $this > (($status >= $WARNING) ? (80) : (90))

44 + crit: $this > (($status == $CRITICAL) ? (90) : (98))

45 + delay: down 15m multiplier 1.5 max 1h

46 + info: system memory utilization

47 + to: sysadmin

48 +```

49 +

50 +Then `alarm-notify.sh` uses its own configuration file `health_alarm_notify.conf`, which at the bottom of the file stores the recipients per role, for all notification methods.

51 +

52 +Here is an example, of the `sysadmin`'s role recipients for the email notification.

53 +You can send the notification to multiple recipients by separating the emails with a space.

54 +

55 +```conf

56 +

57 +###############################################################################

58 +# RECIPIENTS PER ROLE

30 -Each role may have one or more destinations.

60 +# -----------------------------------------------------------------------------

61 +# generic system alarms

62 +# CPU, disks, network interfaces, entropy, etc

63 +

64 +role_recipients_email[sysadmin]="someone@exaple.com someoneelse@example.com"

65 +```

66 +

67 +Each role may have one or more destinations and one or more notification methods.

69 So, for example the `sysadmin` role may send:

34 -1. emails to admin1@example.com and admin2@example.com

35 -2. pushover.net notifications to USERTOKENS `A`, `B` and `C`.

36 -3. pushbullet.com push notifications to admin1@example.com and admin2@example.com

37 -4. messages to slack.com channel `#alarms` and `#systems`.

38 -5. messages to Discord channels `#alarms` and `#systems`.

71 +1. emails to admin1@example.com and admin2@example.com

72 +2. pushover.net notifications to USERTOKENS `A`, `B` and `C`.

73 +3. pushbullet.com push notifications to admin1@example.com and admin2@example.com

74 +4. messages to the `#alerts` and `#systems` channels of a Slack workspace.

75 +5. messages to Discord channels `#alerts` and `#systems`.

77 ## Configuration

42 -Edit `/etc/netdata/health_alarm_notify.conf` by running `/etc/netdata/edit-config health_alarm_notify.conf`:

79 +You can edit `health_alarm_notify.conf` using the `edit-config` script to configure:

44 -- settings per notification method:

81 +- **Settings** per notification method:

46 - all notification methods except email, require some configuration

47 - (i.e. API keys, tokens, destination rooms, channels, etc).

83 + All notification methods except email, require some configuration (i.e. API keys, tokens, destination rooms, channels, etc). Please check this section's content to find the configuration guides for your notification option of choice

49 -- **recipients** per **role** per **notification method**

85 +- **Recipients** per role per notification method

51 -```sh

52 -grep sysadmin /etc/netdata/health_alarm_notify.conf

53 -

54 -role_recipients_email[sysadmin]="${DEFAULT_RECIPIENT_EMAIL}"

55 -role_recipients_pushover[sysadmin]="${DEFAULT_RECIPIENT_PUSHOVER}"

56 -role_recipients_pushbullet[sysadmin]="${DEFAULT_RECIPIENT_PUSHBULLET}"

57 -role_recipients_telegram[sysadmin]="${DEFAULT_RECIPIENT_TELEGRAM}"

58 -role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"

59 -...

60 -```

87 + ```conf

88 + role_recipients_email[sysadmin]="${DEFAULT_RECIPIENT_EMAIL}"

89 + role_recipients_pushover[sysadmin]="${DEFAULT_RECIPIENT_PUSHOVER}"

90 + role_recipients_pushbullet[sysadmin]="${DEFAULT_RECIPIENT_PUSHBULLET}"

91 + role_recipients_telegram[sysadmin]="${DEFAULT_RECIPIENT_TELEGRAM}"

92 + role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"

93 + ...

94 + ```

95 +

96 + Here you can change the `${DEFAULT_...}` values to the values of the recipients you want, separated by a space if you have multiple recipients.

62 -## Testing Notifications

98 +## Testing Alert Notifications

100 You can run the following command by hand, to test alerts configuration:

101

102 ```sh

103 # become user netdata

68 -su -s /bin/bash netdata

104 +sudo su -s /bin/bash netdata

105

106 # enable debugging info on the console

107 export NETDATA_ALARM_NOTIFY_DEBUG=1

@@ -79,13 +115,93 @@ export NETDATA_ALARM_NOTIFY_DEBUG=1

115

116 If you are [running your own registry](https://github.com/netdata/netdata/blob/master/registry/README.md#run-your-own-registry), add `export NETDATA_REGISTRY_URL=[YOUR_URL]` before calling `alarm-notify.sh`.

117

82 -Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under `/usr/lib/netdata`). You can always find the location of the `alarm-notify.sh` script in `netdata.conf`.

118 +> If you need to dig even deeper, you can trace the execution with `bash -x`. Note that in test mode, `alarm-notify.sh` calls itself with many more arguments. So first do:

119 +>

120 +>```sh

121 +>bash -x /usr/libexec/netdata/plugins.d/alarm-notify.sh test

122 +>```

123 +>

124 +> And then look in the output for the alarm-notify.sh calls and run the one you want to trace with `bash -x`.

125 +

126 +## Global configuration options

127 +

128 +### Notification Filtering

129 +

130 +When you define recipients per role for notification methods, you can append `|critical` to limit the notifications that are sent.

131 +

132 +In the following examples, the first recipient receives all the alarms, while the second one receives only notifications for alarms that have at some point become critical.

133 +The second user may still receive warning and clear notifications, but only for the event that previously caused a critical alarm.

134 +

135 +```conf

136 + email : "user1@example.com user2@example.com|critical"

137 + pushover : "2987343...9437837 8756278...2362736|critical"

138 + telegram : "111827421 112746832|critical"

139 + slack : "alarms disasters|critical"

140 + alerta : "alarms disasters|critical"

141 + flock : "alarms disasters|critical"

142 + discord : "alarms disasters|critical"

143 + twilio : "+15555555555 +17777777777|critical"

144 + messagebird: "+15555555555 +17777777777|critical"

145 + kavenegar : "09155555555 09177777777|critical"

146 + pd : "<pd_service_key_1> <pd_service_key_2>|critical"

147 + irc : "<irc_channel_1> <irc_channel_2>|critical"

148 +```

149

84 -If you need to dig even deeper, you can trace the execution with `bash -x`. Note that in test mode, `alarm-notify.sh` calls itself with many more arguments. So first do

150 +If a per role recipient is set to an empty string, the default recipient of the given

151 +notification method (email, pushover, telegram, slack, alerta, etc) will be used.

152

86 -```sh

87 -bash -x /usr/libexec/netdata/plugins.d/alarm-notify.sh test

153 +To disable a notification, use the recipient called: disabled

154 +This works for all notification methods (including the default recipients).

155 +

156 +### Proxy configuration

157 +

158 +If you need to send curl based notifications (pushover, pushbullet, slack, alerta,

159 +flock, discord, telegram) via a proxy, you should set these variables to your proxy address:

160 +

161 +```conf

162 +export http_proxy="http://10.0.0.1:3128/"

163 +export https_proxy="http://10.0.0.1:3128/"

164 +```

165 +

166 +### Notification images

167 +

168 +Images in notifications need to be downloaded from an Internet facing site.

169 +

170 +To allow notification providers to fetch the icons/images, by default we set the URL of the global public netdata registry.

171 +

172 +If you have an Internet facing netdata (or you have copied the images/ folder

173 +of netdata to your web server), set its URL here, to fetch the notification

174 +images from it.

175 +

176 +```conf

177 +images_base_url="http://my.public.netdata.server:19999"

178 ```

179

90 - Then look in the output for the alarm-notify.sh calls and run the one you want to trace with `bash -x`.

180 +### Date handling

181 +

182 +You can configure netdata alerts to send dates in any format you want via editing the `date_format` variable.

183 +

184 +This uses standard `date` command format strings. See `man date` for

185 +more info on what formats are supported.

186 +

187 +Note that this has to start with a '+', otherwise it won't work.

188 +

189 +- For ISO 8601 dates, use `+%FT%T%z`

190 +- For RFC 5322 dates, use `+%a, %d %b %Y %H:%M:%S %z`

191 +- For RFC 3339 dates, use `+%F %T%:z`

192 +- For RFC 1123 dates, use `+%a, %d %b %Y %H:%M:%S %Z`

193 +- For RFC 1036 dates, use `+%A, %d-%b-%y %H:%M:%S %Z`

194 +- For a reasonably local date and time (in that order), use `+%x %X`

195 +- For the old default behavior (compatible with ANSI C's `asctime()` function), leave the `date_format` field empty.

196 +

197 +### Hostname handling

198 +

199 +By default, Netdata will use the simple hostname for the system (the hostname with everything after the first `.` removed) when displaying the hostname in alert notifications.

200 +

201 +If you instead prefer to have Netdata use the host's fully qualified domain name, you can set `use_fdqn` to `YES`.

202 +

203 +This setting does not account for child systems for which the system you are configuring is a parent.

204

205 +> ### Note

206 +>

207 +> If the system's host name is overridden in `/etc/netdata.conf` with the `hostname` option, that name will be used unconditionally.

health/notifications/alerta/README.md

+54 -64

@@ -1,86 +1,76 @@

1 -<!--

2 -title: "Alerta agent alert notifications"

3 -sidebar_label: "Alerta"

4 -description: "Send alarm notifications to Alerta to see the latest health status updates from multiple nodes in a single interface."

5 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/alerta/README.md"

6 -learn_status: "Published"

7 -learn_topic_type: "Tasks"

8 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

9 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

10 --->

1 +# Alerta Agent alert notifications

12 -# Alerta agent alert notifications

3 +Learn how to send notifications to Alerta using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

14 -The [Alerta](https://alerta.io) monitoring system is a tool used to

15 -consolidate and de-duplicate alerts from multiple sources for quick

16 -‘at-a-glance’ visualisation. With just one system you can monitor

17 -alerts from many other monitoring tools on a single screen.

5 +> ### Note

6 +>

7 +> This file assumes you have read the [Introduction to Agent alert notifications](https://github.com/netdata/netdata/blob/master/health/notifications/README.md), detailing how the Netdata Agent's alert notification method works.

19 -![Alerta dashboard](https://docs.alerta.io/_images/alerta-screen-shot-3.png "Alerta dashboard showing several alerts.")

9 +The [Alerta](https://alerta.io) monitoring system is a tool used to consolidate and de-duplicate alerts from multiple sources for quick ‘at-a-glance’ visualization.

10 +With just one system you can monitor alerts from many other monitoring tools on a single screen.

21 -Alerta's advantage is the main view, where you can see all active alarms with the most recent state. You can also view an alert history. You can send Netdata alerts to Alerta to see alerts coming from many Netdata hosts or also from a multi-host

22 -Netdata configuration.

12 +![Alerta dashboard showing several alerts](https://docs.alerta.io/_images/alerta-screen-shot-3.png)

24 -## Deploying Alerta

14 +Alerta's advantage is the main view, where you can see all active alert with the most recent state.

15 +You can also view an alert history.

26 -The recommended setup is using a dedicated server, VM or container. If you have other NGINX or Apache servers in your organization,

27 -it is recommended to proxy to this new server.

17 +You can send Netdata alerts to Alerta to see alerts coming from many Netdata hosts or also from a multi-host Netdata configuration.

29 -You can install Alerta in several ways:

30 -- **Docker**: Alerta provides a [Docker image](https://hub.docker.com/r/alerta/alerta-web/) to get you started quickly.

31 -- **Deployment on Ubuntu server**: Alerta's [getting started tutorial](https://docs.alerta.io/gettingstarted/tutorial-1-deploy-alerta.html) walks you through this process.

32 -- **Advanced deployment scenarios**: More ways to install and deploy Alerta are documented on the [Alerta docs](http://docs.alerta.io/en/latest/deployment.html).

19 +## Prerequisites

34 -## Sending alerts to Alerta

21 +You need:

36 -### Step 1. Create an API key (if authentication in Alerta is enabled)

23 +- an Alerta instance

24 +- an Alerta API key (if authentication in Alerta is enabled)

25 +- terminal access to the Agent you wish to configure

38 -You will need an API key to send messages from any source, if

39 -Alerta is configured to use authentication (recommended).

27 +## Configure Netdata to send alert notifications to Alerta

41 -Create a new API key in Alerta:

42 -1. Go to *Configuration* > *API Keys*

43 -2. Create a new API key called "netdata" with `write:alerts` permission.

29 +> ### Info

30 +>

31 +> This file mentions editing configuration files.

32 +>

33 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

34 +> Note that to run the script you need to be inside your Netdata config directory.

35 +>

36 +> It is recommended to use this way for configuring Netdata.

45 -### Step 2. Configure Netdata to send alerts to Alerta

46 -1. Edit the `health_alarm_notify.conf` by running:

47 -```sh

48 -/etc/netdata/edit-config health_alarm_notify.conf

49 -```

38 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

51 -2. Modify the file as below:

52 -```

53 -# enable/disable sending alerta notifications

54 -SEND_ALERTA="YES"

40 +1. Set `SEND_ALERTA` to `YES`.

41 +2. set `ALERTA_WEBHOOK_URL` to the API url you defined when you installed the Alerta server.

42 +3. Set `ALERTA_API_KEY` to your API key.

43 + You will need an API key to send messages from any source, if Alerta is configured to use authentication (recommended). To create a new API key:

44 + 1. Go to *Configuration* > *API Keys*.

45 + 2. Create a new API key called "netdata" with `write:alerts` permission.

46 +4. Set `DEFAULT_RECIPIENT_ALERTA` to the default recipient environment you want the alert notifications to be sent to.

47 + All roles will default to this variable if left unconfigured.

56 -# here set your alerta server API url

57 -# this is the API url you defined when installed Alerta server,

58 -# it is the same for all users. Do not include last slash.

59 -ALERTA_WEBHOOK_URL="http://yourserver/alerta/api"

49 +You can then have different recipient environments per **role**, by editing `DEFAULT_RECIPIENT_CUSTOM` with the environment name you want, in the following entries at the bottom of the same file:

61 -# Login with an administrative user to you Alerta server and create an API KEY

62 -# with write permissions.

63 -ALERTA_API_KEY="INSERT_YOUR_API_KEY_HERE"

64 -

65 -# you can define environments in /etc/alertad.conf option ALLOWED_ENVIRONMENTS

66 -# standard environments are Production and Development

67 -# if a role's recipients are not configured, a notification will be send to

68 -# this Environment (empty = do not send a notification for unconfigured roles):

69 -DEFAULT_RECIPIENT_ALERTA="Production"

51 +```conf

52 +role_recipients_alerta[sysadmin]="Systems"

53 +role_recipients_alerta[domainadmin]="Domains"

54 +role_recipients_alerta[dba]="Databases Systems"

55 +role_recipients_alerta[webmaster]="Marketing Development"

56 +role_recipients_alerta[proxyadmin]="Proxy"

57 +role_recipients_alerta[sitemgr]="Sites"

58 ```

72 -## Test alarms

60 +The values you provide should be defined as environments in `/etc/alertad.conf` option `ALLOWED_ENVIRONMENTS`.

74 -We can test alarms using the standard approach:

62 +An example working configuration would be:

76 -```sh

77 -/opt/netdata/netdata-plugins/plugins.d/alarm-notify.sh test

78 -```

79 -

80 -> **Note** This script will send 3 alarms.

81 -> Alerta will not show the alerts in the main page, because last alarm is "CLEAR".

82 -> To see the test alarms, you need to select "closed" alarms in the top-right lookup.

64 +```conf

65 +#------------------------------------------------------------------------------

66 +# alerta (alerta.io) global notification options

84 -For more information see the [Alerta documentation](https://docs.alerta.io)

68 +SEND_ALERTA="YES"

69 +ALERTA_WEBHOOK_URL="http://yourserver/alerta/api"

70 +ALERTA_API_KEY="INSERT_YOUR_API_KEY_HERE"

71 +DEFAULT_RECIPIENT_ALERTA="Production"

72 +```

74 +## Test the notification method

76 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/awssns/README.md

+117 -48

@@ -1,31 +1,29 @@

1 -<!--

2 -title: "Amazon SNS agent alert notifications"

3 -sidebar_label: "Amazon SNS"

4 -description: "hello"

5 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/awssns/README.md"

6 -learn_status: "Published"

7 -learn_topic_type: "Tasks"

8 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

9 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

10 --->

11 -

12 -# Amazon SNS agent alert notifications

13 -

14 -As part of its AWS suite, Amazon provides a notification broker service called 'Simple Notification Service' (SNS). Amazon SNS works similarly to Netdata's own notification system, allowing to dispatch a single notification to multiple subscribers of different types. While Amazon SNS supports sending differently formatted messages for different delivery methods, Netdata does not currently support this functionality.

15 -Among other things, SNS supports sending notifications to:

16 -

17 -- Email addresses.

18 -- Mobile Phones via SMS.

19 -- HTTP or HTTPS web hooks.

20 -- AWS Lambda functions.

21 -- AWS SQS queues.

22 -- Mobile applications via push notifications.

23 -

24 -For email notification support, we recommend using Netdata's email notifications, as it is has the following benefits:

1 +# Amazon SNS Agent alert notifications

2 +

3 +Learn how to send notifications through Amazon SNS using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

8 +

9 +As part of its AWS suite, Amazon provides a notification broker service called 'Simple Notification Service' (SNS). Amazon SNS works similarly to Netdata's own notification system, allowing to dispatch a single notification to multiple subscribers of different types. Among other things, SNS supports sending notifications to:

10 +

11 +- email addresses

12 +- mobile Phones via SMS

13 +- HTTP or HTTPS web hooks

14 +- AWS Lambda functions

15 +- AWS SQS queues

16 +- mobile applications via push notifications

17 +

18 +> ### Note

19 +>

20 +> While Amazon SNS supports sending differently formatted messages for different delivery methods, Netdata does not currently support this functionality.

21 +

22 +For email notification support, we recommend using Netdata's [email notifications](https://github.com/netdata/netdata/blob/master/health/notifications/email/README.md), as it is has the following benefits:

24 - In most cases, it requires less configuration.

25 - Netdata's emails are nicely pre-formatted and support features like threading, which requires a lot of manual effort in SNS.

28 -- It is less resource intensive and more cost-efficient than SNS.

26 +- It is less resource intensive and more cost-efficient than SNS.

28 Read on to learn how to set up Amazon SNS in Netdata.

@@ -33,26 +31,97 @@ Read on to learn how to set up Amazon SNS in Netdata.

32 Before you can enable SNS, you need:

36 -- The [Amazon Web Services CLI tools](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) (`awscli`).

37 -- An actual home directory for the user you run Netdata as, instead of just using `/` as a home directory. The setup depends on the distribution, but `/var/lib/netdata` is the recommended directory. If you are using Netdata as a dedicated user, the permissions will already be correct.

38 -- An Amazon SNS topic to send notifications to with one or more subscribers. The [Getting Started](https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html) section of the Amazon SNS documentation covers the basics of how to set this up. Make note of the **Topic ARN** when you create the topic.

39 -- While not mandatory, it is highly recommended to create a dedicated IAM user on your account for Netdata to send notifications. This user needs to have programmatic access, and should only allow access to SNS. For an additional layer of security, you can create one for each system or group of systems.

40 -

41 -## Enabling Amazon SNS

42 -

43 -To enable SNS:

44 -1. Run the following command as the user Netdata runs under:

45 - ```

46 - aws configure

47 - ```

48 -2. Enter the access key and secret key for accessing Amazon SNS. The system also prompts you to enter the default region and output format, but you can leave those blank because Netdata doesn't use them.

49 -

50 -3. Specify the desired topic ARN as a recipient, see [SNS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SetupSNS.html#set-up-sns-topic-cli).

51 -4. Optional: To change the notification format for SNS notifications, change the `AWSSNS_MESSAGE_FORMAT` variable in `health_alarm_notify.conf`.

52 -This variable supports all the same variables you can use in custom notifications.

53 -

54 - The default format looks like this:

55 - ```bash

56 - AWSSNS_MESSAGE_FORMAT="${status} on ${host} at ${date}: ${chart} ${value_string}"

57 - ```

58 -

34 +- The [Amazon Web Services CLI tools](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) (`awscli`).

35 +- An actual home directory for the user you run Netdata as, instead of just using `/` as a home directory.

36 + The setup depends on the distribution, but `/var/lib/netdata` is the recommended directory. If you are using Netdata as a dedicated user, the permissions will already be correct.

37 +- An Amazon SNS topic to send notifications to with one or more subscribers.

38 + The [Getting Started](https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html) section of the Amazon SNS documentation covers the basics of how to set this up. Make note of the **Topic ARN** when you create the topic.

39 +- While not mandatory, it is highly recommended to create a dedicated IAM user on your account for Netdata to send notifications.

40 + This user needs to have programmatic access, and should only allow access to SNS. For an additional layer of security, you can create one for each system or group of systems.

41 +- Terminal access to the Agent you wish to configure.

42 +

43 +## Configure Netdata to send alert notifications to Amazon SNS

44 +

45 +> ### Info

46 +>

47 +> This file mentions editing configuration files.

48 +>

49 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

50 +> Note that to run the script you need to be inside your Netdata config directory.

51 +>

52 +> It is recommended to use this way for configuring Netdata.

53 +

54 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

55 +

56 +1. Set `SEND_AWSNS` to `YES`.

57 +2. Set `AWSSNS_MESSAGE_FORMAT` to the string that you want the alert to be sent into.

58 +

59 + The supported variables are:

60 +

61 + | Variable name | Description |

62 + |:---------------------------:|:---------------------------------------------------------------------------------|

63 + | `${alarm}` | Like "name = value units" |

64 + | `${status_message}` | Like "needs attention", "recovered", "is critical" |

65 + | `${severity}` | Like "Escalated to CRITICAL", "Recovered from WARNING" |

66 + | `${raised_for}` | Like "(alarm was raised for 10 minutes)" |

67 + | `${host}` | The host generated this event |

68 + | `${url_host}` | Same as ${host} but URL encoded |

69 + | `${unique_id}` | The unique id of this event |

70 + | `${alarm_id}` | The unique id of the alarm that generated this event |

71 + | `${event_id}` | The incremental id of the event, for this alarm id |

72 + | `${when}` | The timestamp this event occurred |

73 + | `${name}` | The name of the alarm, as given in netdata health.d entries |

74 + | `${url_name}` | Same as ${name} but URL encoded |

75 + | `${chart}` | The name of the chart (type.id) |

76 + | `${url_chart}` | Same as ${chart} but URL encoded |

77 + | `${family}` | The family of the chart |

78 + | `${url_family}` | Same as ${family} but URL encoded |

79 + | `${status}` | The current status : REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |

80 + | `${old_status}` | The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |

81 + | `${value}` | The current value of the alarm |

82 + | `${old_value}` | The previous value of the alarm |

83 + | `${src}` | The line number and file the alarm has been configured |

84 + | `${duration}` | The duration in seconds of the previous alarm state |

85 + | `${duration_txt}` | Same as ${duration} for humans |

86 + | `${non_clear_duration}` | The total duration in seconds this is/was non-clear |

87 + | `${non_clear_duration_txt}` | Same as ${non_clear_duration} for humans |

88 + | `${units}` | The units of the value |

89 + | `${info}` | A short description of the alarm |

90 + | `${value_string}` | Friendly value (with units) |

91 + | `${old_value_string}` | Friendly old value (with units) |

92 + | `${image}` | The URL of an image to represent the status of the alarm |

93 + | `${color}` | A color in AABBCC format for the alarm |

94 + | `${goto_url}` | The URL the user can click to see the netdata dashboard |

95 + | `${calc_expression}` | The expression evaluated to provide the value for the alarm |

96 + | `${calc_param_values}` | The value of the variables in the evaluated expression |

97 + | `${total_warnings}` | The total number of alarms in WARNING state on the host |

98 + | `${total_critical}` | The total number of alarms in CRITICAL state on the host |

99 +

100 +3. Set `DEFAULT_RECIPIENT_AWSSNS` to the Topic ARN you noted down upon creating the Topic.

101 + All roles will default to this variable if left unconfigured.

102 +

103 +You can then have different recipient Topics per **role**, by editing `DEFAULT_RECIPIENT_AWSSNS` with the Topic ARN you want, in the following entries at the bottom of the same file:

104 +

105 +```conf

106 +role_recipients_awssns[sysadmin]="arn:aws:sns:us-east-2:123456789012:Systems"

107 +role_recipients_awssns[domainadmin]="arn:aws:sns:us-east-2:123456789012:Domains"

108 +role_recipients_awssns[dba]="arn:aws:sns:us-east-2:123456789012:Databases"

109 +role_recipients_awssns[webmaster]="arn:aws:sns:us-east-2:123456789012:Development"

110 +role_recipients_awssns[proxyadmin]="arn:aws:sns:us-east-2:123456789012:Proxy"

111 +role_recipients_awssns[sitemgr]="arn:aws:sns:us-east-2:123456789012:Sites"

112 +```

113 +

114 +An example working configuration would be:

115 +

116 +```conf

117 +#------------------------------------------------------------------------------

118 +# Amazon SNS notifications

119 +

120 +SEND_AWSSNS="YES"

121 +AWSSNS_MESSAGE_FORMAT="${status} on ${host} at ${date}: ${chart} ${value_string}"

122 +DEFAULT_RECIPIENT_AWSSNS="arn:aws:sns:us-east-2:123456789012:MyTopic"

123 +```

124 +

125 +## Test the notification method

126 +

127 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/custom/README.md

+133 -75

@@ -1,29 +1,128 @@

1 -<!--

2 -title: "Custom agent alert notifications"

3 -sidebar_label: "Custom endpoint"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/custom/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Custom agent alert notifications

12 -

13 -Netdata allows you to send custom notifications to any endpoint you choose.

14 -

15 -To configure custom notifications, you will need to customize `health_alarm_notify.conf`. Open the file for editing

16 -using [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) from the [Netdata config

17 -directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory), which is typically at `/etc/netdata`.

1 +# Custom Agent alert notifications

2 +

3 +Netdata Agent's alert notification feature allows you to send custom notifications to any endpoint you choose.

4 +

5 +> ### Note

6 +>

8 +

9 +## Prerequisites

10 +

11 +You need to have terminal access to the Agent you wish to configure.

12 +

13 +## Configure Netdata to send alert notifications to a custom endpoint

14 +

15 +> ### Info

16 +>

17 +> This file mentions editing configuration files.

18 +>

19 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

20 +> Note that to run the script you need to be inside your Netdata config directory.

21 +>

22 +> It is recommended to use this way for configuring Netdata.

23 +

24 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

25 +

26 +1. Set `SEND_CUSTOM` to `YES`.

27 +2. The `DEFAULT_RECIPIENT_CUSTOM`'s value is dependent on how you handle the `${to}` variable inside the `custom_sender()` function.

28 + All roles will default to this variable if left unconfigured.

29 +3. Edit the `custom_sender()` function.

30 + You can look at the other senders in `/usr/libexec/netdata/plugins.d/alarm-notify.sh` for examples of how to modify the function in this configuration file.

31 +

32 + The following is a sample `custom_sender()` function in `health_alarm_notify.conf`, to send an SMS via an imaginary HTTPS endpoint to the SMS gateway:

33 +

34 + ```sh

35 + custom_sender() {

36 + # example human readable SMS

37 + local msg="${host} ${status_message}: ${alarm} ${raised_for}"

38 +

39 + # limit it to 160 characters and encode it for use in a URL

40 + urlencode "${msg:0:160}" >/dev/null; msg="${REPLY}"

41 +

42 + # a space separated list of the recipients to send alarms to

43 + to="${1}"

44 +

45 + for phone in ${to}; do

46 + httpcode=$(docurl -X POST \

47 + --data-urlencode "From=XXX" \

48 + --data-urlencode "To=${phone}" \

49 + --data-urlencode "Body=${msg}" \

50 + -u "${accountsid}:${accounttoken}" \

51 + https://domain.website.com/)

52 +

53 + if [ "${httpcode}" = "200" ]; then

54 + info "sent custom notification ${msg} to ${phone}"

55 + sent=$((sent + 1))

56 + else

57 + error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."

58 + fi

59 + done

60 + }

61 + ```

62 +

63 + The supported variables that you can use for the function's `msg` variable are:

64 +

65 + | Variable name | Description |

66 + |:---------------------------:|:---------------------------------------------------------------------------------|

67 + | `${alarm}` | Like "name = value units" |

68 + | `${status_message}` | Like "needs attention", "recovered", "is critical" |

69 + | `${severity}` | Like "Escalated to CRITICAL", "Recovered from WARNING" |

70 + | `${raised_for}` | Like "(alarm was raised for 10 minutes)" |

71 + | `${host}` | The host generated this event |

72 + | `${url_host}` | Same as ${host} but URL encoded |

73 + | `${unique_id}` | The unique id of this event |

74 + | `${alarm_id}` | The unique id of the alarm that generated this event |

75 + | `${event_id}` | The incremental id of the event, for this alarm id |

76 + | `${when}` | The timestamp this event occurred |

77 + | `${name}` | The name of the alarm, as given in netdata health.d entries |

78 + | `${url_name}` | Same as ${name} but URL encoded |

79 + | `${chart}` | The name of the chart (type.id) |

80 + | `${url_chart}` | Same as ${chart} but URL encoded |

81 + | `${family}` | The family of the chart |

82 + | `${url_family}` | Same as ${family} but URL encoded |

83 + | `${status}` | The current status : REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |

84 + | `${old_status}` | The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL |

85 + | `${value}` | The current value of the alarm |

86 + | `${old_value}` | The previous value of the alarm |

87 + | `${src}` | The line number and file the alarm has been configured |

88 + | `${duration}` | The duration in seconds of the previous alarm state |

89 + | `${duration_txt}` | Same as ${duration} for humans |

90 + | `${non_clear_duration}` | The total duration in seconds this is/was non-clear |

91 + | `${non_clear_duration_txt}` | Same as ${non_clear_duration} for humans |

92 + | `${units}` | The units of the value |

93 + | `${info}` | A short description of the alarm |

94 + | `${value_string}` | Friendly value (with units) |

95 + | `${old_value_string}` | Friendly old value (with units) |

96 + | `${image}` | The URL of an image to represent the status of the alarm |

97 + | `${color}` | A color in AABBCC format for the alarm |

98 + | `${goto_url}` | The URL the user can click to see the netdata dashboard |

99 + | `${calc_expression}` | The expression evaluated to provide the value for the alarm |

100 + | `${calc_param_values}` | The value of the variables in the evaluated expression |

101 + | `${total_warnings}` | The total number of alarms in WARNING state on the host |

102 + | `${total_critical}` | The total number of alarms in CRITICAL state on the host |

103 +

104 +You can then have different `${to}` variables per **role**, by editing `DEFAULT_RECIPIENT_CUSTOM` with the variable you want, in the following entries at the bottom of the same file:

105 +

106 +```conf

107 +role_recipients_custom[sysadmin]="systems"

108 +role_recipients_custom[domainadmin]="domains"

109 +role_recipients_custom[dba]="databases systems"

110 +role_recipients_custom[webmaster]="marketing development"

111 +role_recipients_custom[proxyadmin]="proxy-admin"

112 +role_recipients_custom[sitemgr]="sites"

113 +```

114

19 -You can look at the other senders in `/usr/libexec/netdata/plugins.d/alarm-notify.sh` for examples of how to modify the `custom_sender()` function in `health_alarm_notify.conf`.

115 +An example working configuration would be:

116

21 -As with other notifications, you will also need to define the recipient list in `DEFAULT_RECIPIENT_CUSTOM` and/or the `role_recipients_custom` array.

117 +```conf

118 +#------------------------------------------------------------------------------

119 +# custom notifications

120

23 -The following is a sample `custom_sender` function in `health_alarm_notify.conf`, to send an SMS via an imaginary HTTPS endpoint to the SMS gateway:

121 +SEND_CUSTOM="YES"

122 +DEFAULT_RECIPIENT_CUSTOM=""

123

25 -```

26 - custom_sender() {

124 +# The custom_sender() is a custom function to do whatever you need to do

125 +custom_sender() {

126 # example human readable SMS

127 local msg="${host} ${status_message}: ${alarm} ${raised_for}"

128

@@ -35,63 +134,22 @@ The following is a sample `custom_sender` function in `health_alarm_notify.conf`

134

135 for phone in ${to}; do

136 httpcode=$(docurl -X POST \

38 - --data-urlencode "From=XXX" \

39 - --data-urlencode "To=${phone}" \

40 - --data-urlencode "Body=${msg}" \

41 - -u "${accountsid}:${accounttoken}" \

137 + --data-urlencode "From=XXX" \

138 + --data-urlencode "To=${phone}" \

139 + --data-urlencode "Body=${msg}" \

140 + -u "${accountsid}:${accounttoken}" \

141 https://domain.website.com/)

142

44 - if [ "${httpcode}" = "200" ]; then

45 - info "sent custom notification ${msg} to ${phone}"

46 - sent=$((sent + 1))

47 - else

48 - error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."

49 - fi

143 + if [ "${httpcode}" = "200" ]; then

144 + info "sent custom notification ${msg} to ${phone}"

145 + sent=$((sent + 1))

146 + else

147 + error "failed to send custom notification ${msg} to ${phone} with HTTP error code ${httpcode}."

148 + fi

149 done

150 }

151 ```

152

54 -Variables available to the custom_sender:

55 -

56 -- `${to_custom}` the list of recipients for the alarm

57 -- `${host}` the host generated this event

58 -- `${url_host}` same as `${host}` but URL encoded

59 -- `${unique_id}` the unique id of this event

60 -- `${alarm_id}` the unique id of the alarm that generated this event

61 -- `${event_id}` the incremental id of the event, for this alarm id

62 -- `${when}` the timestamp this event occurred

63 -- `${name}` the name of the alarm, as given in Netdata health.d entries

64 -- `${url_name}` same as `${name}` but URL encoded

65 -- `${chart}` the name of the chart (type.id)

66 -- `${url_chart}` same as `${chart}` but URL encoded

67 -- `${family}` the family of the chart

68 -- `${url_family}` same as `${family}` but URL encoded

69 -- `${status}` the current status : REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL

70 -- `${old_status}` the previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL

71 -- `${value}` the current value of the alarm

72 -- `${old_value}` the previous value of the alarm

73 -- `${src}` the line number and file the alarm has been configured

74 -- `${duration}` the duration in seconds of the previous alarm state

75 -- `${duration_txt}` same as `${duration}` for humans

76 -- `${non_clear_duration}` the total duration in seconds this is/was non-clear

77 -- `${non_clear_duration_txt}` same as `${non_clear_duration}` for humans

78 -- `${units}` the units of the value

79 -- `${info}` a short description of the alarm

80 -- `${value_string}` friendly value (with units)

81 -- `${old_value_string}` friendly old value (with units)

82 -- `${image}` the URL of an image to represent the status of the alarm

83 -- `${color}` a color in #AABBCC format for the alarm

84 -- `${goto_url}` the URL the user can click to see the Netdata dashboard

85 -- `${calc_expression}` the expression evaluated to provide the value for the alarm

86 -- `${calc_param_values}` the value of the variables in the evaluated expression

87 -- `${total_warnings}` the total number of alarms in WARNING state on the host

88 -- `${total_critical}` the total number of alarms in CRITICAL state on the host

89 -

90 -The following are more human friendly:

91 -

92 -- `${alarm}` like "name = value units"

93 -- `${status_message}` like "needs attention", "recovered", "is critical"

94 -- `${severity}` like "Escalated to CRITICAL", "Recovered from WARNING"

95 -- `${raised_for}` like "(alarm was raised for 10 minutes)"

96 -

153 +## Test the notification method

154

155 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/discord/README.md

+52 -34

@@ -1,53 +1,71 @@

1 -<!--

2 -title: "Discord agent alert notifications"

3 -sidebar_label: "Discord"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/discord/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Discord agent alert notifications

1 +# Discord Agent alert notifications

2 +

3 +Learn how to send notifications to Discord using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 This is what you will get:

11 ![image](https://cloud.githubusercontent.com/assets/7321975/22215935/b49ede7e-e162-11e6-98d0-ae8541e6b92e.png)

17 -You need:

13 +## Prerequisites

19 -1. The **incoming webhook URL** as given by Discord. Create a webhook by following the official [Discord documentation](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks). You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).

20 -2. One or more Discord channels to post the messages to.

15 +You will need:

22 -Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

17 +- The **incoming webhook URL** as given by Discord.

18 + Create a webhook by following the official [Discord documentation](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks). You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).

19 +- one or more Discord channels to post the messages to

20 +- terminal access to the Agent you wish to configure

24 -```

25 -###############################################################################

26 -# sending discord notifications

22 +## Configure Netdata to send alert notifications to Discord

28 -# note: multiple recipients can be given like this:

29 -# "CHANNEL1 CHANNEL2 ..."

24 +> ### Info

25 +>

26 +> This file mentions editing configuration files.

27 +>

28 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

29 +> Note that to run the script you need to be inside your Netdata config directory.

30 +>

31 +> It is recommended to use this way for configuring Netdata.

31 -# enable/disable sending discord notifications

32 -SEND_DISCORD="YES"

33 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

34 -# Create a webhook by following the official documentation -

35 -# https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks

36 -DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/XXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

35 +1. Set `SEND_DISCORD` to `YES`.

36 +2. Set `DISCORD_WEBHOOK_URL` to your webhook URL.

37 +3. Set `DEFAULT_RECIPIENT_DISCORD` to the channel you want the alert notifications to be sent to.

38 + You can define multiple channels like this: `alerts systems`.

39 + All roles will default to this variable if left unconfigured.

38 -# if a role's recipients are not configured, a notification will be send to

39 -# this discord channel (empty = do not send a notification for unconfigured

40 -# roles):

41 -DEFAULT_RECIPIENT_DISCORD="alarms"

42 -```

41 + > ### Note

42 + >

43 + > You don't have to include the hashtag "#" of the channel, just its name.

44 -You can define multiple channels like this: `alarms systems`.

45 -You can give different channels per **role** using these (at the same file):

45 +You can then have different channels per **role**, by editing `DEFAULT_RECIPIENT_DISCORD` with the channel you want, in the following entries at the bottom of the same file:

47 -```

47 +```conf

48 role_recipients_discord[sysadmin]="systems"

49 +role_recipients_discord[domainadmin]="domains"

50 role_recipients_discord[dba]="databases systems"

51 role_recipients_discord[webmaster]="marketing development"

52 +role_recipients_discord[proxyadmin]="proxy-admin"

53 +role_recipients_discord[sitemgr]="sites"

54 ```

53 -The keywords `systems`, `databases`, `marketing`, `development` are discord.com channels (they should already exist within your discord server).

56 +The values you provide should already exist as Discord channels in your server.

57 +

58 +An example of a working configuration would be:

59 +

60 +```conf

61 +#------------------------------------------------------------------------------

62 +# discord (discordapp.com) global notification options

63 +

64 +SEND_DISCORD="YES"

65 +DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/XXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

66 +DEFAULT_RECIPIENT_DISCORD="alerts"

67 +```

68 +

69 +## Test the notification method

70 +

71 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/dynatrace/README.md

+63 -36

@@ -1,39 +1,66 @@

1 -<!--

2 -title: "Dynatrace agent alert notifications"

3 -sidebar_label: "Dynatrace Events"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/dynatrace/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Dynatrace agent alert notifications

1 +# Dynatrace Agent alert notifications

2 +

3 +Learn how to send notifications to Dynatrace using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 Dynatrace allows you to receive notifications using their Events REST API.

15 -See [the Dynatrace documentation](https://www.dynatrace.com/support/help/extend-dynatrace/dynatrace-api/environment-api/events/post-event/) about POSTing an event in the Events API for more details.

16 -

17 -

18 -

19 -You need:

20 -

21 -1. Dynatrace Server. You can use the same on all your Netdata servers but make sure the server is network visible from your Netdata hosts.

22 -The Dynatrace server should be with protocol prefixed (`http://` or `https://`). For example: `https://monitor.example.com`

23 -This is a required parameter.

24 -2. API Token. Generate a secure access API token that enables access to your Dynatrace monitoring data via the REST-based API.

25 -Generate a Dynatrace API authentication token. On your Dynatrace server, go to **Settings** --> **Integration** --> **Dynatrace API** --> **Generate token**.

26 -See [Dynatrace API - Authentication](https://www.dynatrace.com/support/help/extend-dynatrace/dynatrace-api/basics/dynatrace-api-authentication/) for more details.

27 -This is a required parameter.

28 -3. API Space. This is the URL part of the page you have access in order to generate the API Token. For example, the URL

29 - for a generated API token might look like:

30 - `https://monitor.illumineit.com/e/2a93fe0e-4cd5-469a-9d0d-1a064235cfce/#settings/integration/apikeys;gf=all` In that

31 - case, my space is _2a93fe0e-4cd5-469a-9d0d-1a064235cfce_ This is a required parameter.

32 -4. Generate a Server Tag. On your Dynatrace Server, go to **Settings** --> **Tags** --> **Manually applied tags** and create the Tag.

33 -The Netdata alarm is sent as a Dynatrace Event to be correlated with all those hosts tagged with this Tag you have created.

34 -This is a required parameter.

35 -5. Specify the Dynatrace event. This can be one of `CUSTOM_INFO`, `CUSTOM_ANNOTATION`, `CUSTOM_CONFIGURATION`, and `CUSTOM_DEPLOYMENT`.

36 -The default value is `CUSTOM_INFO`.

37 -This is a required parameter.

38 -6. Specify the annotation type. This is the source of the Dynatrace event. Put whatever it fits you, for example,

39 -_Netdata Alarm_, which is also the default value.

11 +See [the Dynatrace documentation](https://www.dynatrace.com/support/help/dynatrace-api/environment-api/events-v2/post-event) about POSTing an event in the Events API for more details.

12 +

13 +## Prerequisites

14 +

15 +You will need:

16 +

17 +- A Dynatrace Server. You can use the same on all your Netdata servers but make sure the server is network visible from your Netdata hosts.

18 + The Dynatrace server should be with protocol prefixed (`http://` or `https://`), for example: `https://monitor.example.com`.

19 +- An API Token. Generate a secure access API token that enables access to your Dynatrace monitoring data via the REST-based API.

20 + See [Dynatrace API - Authentication](https://www.dynatrace.com/support/help/extend-dynatrace/dynatrace-api/basics/dynatrace-api-authentication/) for more details.

21 +- An API Space. This is the URL part of the page you have access in order to generate the API Token.

22 + For example, the URL for a generated API token might look like: `https://monitor.illumineit.com/e/2a93fe0e-4cd5-469a-9d0d-1a064235cfce/#settings/integration/apikeys;gf=all` In that case, the Space is `2a93fe0e-4cd5-469a-9d0d-1a064235cfce`.

23 +- A Server Tag. To generate one on your Dynatrace Server, go to **Settings** --> **Tags** --> **Manually applied tags** and create the Tag.

24 + The Netdata alarm is sent as a Dynatrace Event to be correlated with all those hosts tagged with this Tag you have created.

25 +- terminal access to the Agent you wish to configure

26 +

27 +## Configure Netdata to send alert notifications to Dynatrace

28 +

29 +> ### Info

30 +>

31 +> This file mentions editing configuration files.

32 +>

34 +> Note that to run the script you need to be inside your Netdata config directory.

35 +>

36 +> It is recommended to use this way for configuring Netdata.

37 +

38 +Edit `health_alarm_notify.conf`:

39 +

40 +1. Set `SEND_DYNATRACE` to `YES`.

41 +2. Set `DYNATRACE_SERVER` to the Dynatrace server with the protocol prefix, for example `https://monitor.example.com`.

42 +3. Set `DYNATRACE_TOKEN` to your Dynatrace API authentication token

43 +4. Set `DYNATRACE_SPACE` to the API Space, it is the URL part of the page you have access in order to generate the API Token. For example, the URL for a generated API token might look like: `https://monitor.illumineit.com/e/2a93fe0e-4cd5-469a-9d0d-1a064235cfce/#settings/integration/apikeys;gf=all` In that case, the Space is `2a93fe0e-4cd5-469a-9d0d-1a064235cfce`.

44 +5. Set `DYNATRACE_TAG_VALUE` to your Dynatrace Server Tag.

45 +6. `DYNATRACE_ANNOTATION_TYPE` can be left to its default value `Netdata Alarm`, but you can change it to better fit your needs.

46 +7. Set `DYNATRACE_EVENT` to the Dynatrace `eventType` you want, possible values are:

47 + `AVAILABILITY_EVENT`, `CUSTOM_ALERT`, `CUSTOM_ANNOTATION`, `CUSTOM_CONFIGURATION`, `CUSTOM_DEPLOYMENT`, `CUSTOM_INFO`, `ERROR_EVENT`, `MARKED_FOR_TERMINATION`, `PERFORMANCE_EVENT`, `RESOURCE_CONTENTION_EVENT`. You can read more [here](https://www.dynatrace.com/support/help/dynatrace-api/environment-api/events-v2/post-event#request-body-objects)

48 +

49 +An example of a working configuration would be:

50 +

51 +```conf

52 +#------------------------------------------------------------------------------

53 +# Dynatrace global notification options

54 +

55 +SEND_DYNATRACE="YES"

56 +DYNATRACE_SERVER="https://monitor.example.com"

57 +DYNATRACE_TOKEN="XXXXXXX"

58 +DYNATRACE_SPACE="2a93fe0e-4cd5-469a-9d0d-1a064235cfce"

59 +DYNATRACE_TAG_VALUE="SERVERTAG"

60 +DYNATRACE_ANNOTATION_TYPE="Netdata Alert"

61 +DYNATRACE_EVENT="AVAILABILITY_EVENT"

62 +```

63 +

64 +## Test the notification method

65 +

66 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/email/README.md

+82 -47

@@ -1,56 +1,83 @@

1 -<!--

2 -title: "Email agent alert notifications"

3 -sidebar_label: "Email"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/email/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': True, 'part_of_agent': True}"

9 --->

1 +# Email Agent alert notifications

11 -# Email agent alert notifications

3 +Learn how to send notifications via Email using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

13 -You need a working `sendmail` command for email alerts to work. Almost all MTAs provide a `sendmail` interface.

5 +> ### Note

6 +>

15 -Netdata sends all emails as user `netdata`, so make sure your `sendmail` works for local users.

9 +Email notifications look like this:

17 -If you are using our Docker images, or are running Netdata on a system that does not have a working `sendmail`

18 -command, see [the section below about using msmtp in place of sendmail](#using-msmtp-instead-of-sendmail).

11 +<img src="https://user-images.githubusercontent.com/1905463/133216974-a2ca0e4f-787b-4dce-b1b2-9996a8c5f718.png" alt="Email notification screenshot" width="50%"></img>

20 -email notifications look like this:

13 +## Prerequisites

22 -![image](https://user-images.githubusercontent.com/1905463/133216974-a2ca0e4f-787b-4dce-b1b2-9996a8c5f718.png)

15 +You will need:

24 -## Configuration

17 +- A working `sendmail` command for email alerts to work. Almost all MTAs provide a `sendmail` interface.

18 + Netdata sends all emails as user `netdata`, so make sure your `sendmail` works for local users.

26 -To edit `health_alarm_notify.conf` on your system run `/etc/netdata/edit-config health_alarm_notify.conf`.

20 + > ### Note

21 + >

22 + > If you are using our Docker images, or are running Netdata on a system that does not have a working `sendmail` command, see [the section below about using msmtp in place of sendmail](#using-msmtp-instead-of-sendmail).

23 +- terminal access to the Agent you wish to configure

28 -You can configure recipients in [`/etc/netdata/health_alarm_notify.conf`](https://github.com/netdata/netdata/blob/99d44b7d0c4e006b11318a28ba4a7e7d3f9b3bae/conf.d/health_alarm_notify.conf#L101).

25 +## Configure Netdata to send alerts via Email

30 -You can also configure per role recipients [in the same file, a few lines below](https://github.com/netdata/netdata/blob/99d44b7d0c4e006b11318a28ba4a7e7d3f9b3bae/conf.d/health_alarm_notify.conf#L313).

27 +> ### Info

28 +>

29 +> This file mentions editing configuration files.

30 +>

31 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

32 +> Note that to run the script you need to be inside your Netdata config directory.

33 +>

34 +> It is recommended to use this way for configuring Netdata.

32 -Changes to this file do not require a Netdata restart.

36 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

34 -You can test your configuration by issuing the commands:

38 +1. You can change `EMAIL_SENDER` to the email address sending the notifications, the default is the system user Netdata runs as, usually being `netdata`.

39 + Supported formats are:

36 -```sh

37 -# become user netdata

38 -sudo su -s /bin/bash netdata

41 + ```conf

42 + EMAIL_SENDER="user@domain"

43 + EMAIL_SENDER="User Name <user@domain>"

44 + EMAIL_SENDER="'User Name' <user@domain>"

45 + EMAIL_SENDER="\"User Name\" <user@domain>"

46 + ```

40 -# send a test alarm

41 -/usr/libexec/netdata/plugins.d/alarm-notify.sh test [ROLE]

48 +2. Set `SEND_EMAIL` to `YES`.

49 +3. Set `DEFAULT_RECIPIENT_EMAIL` to the email address you want the email to be sent by default.

50 + You can define multiple email addresses like this: `alarms@example.com systems@example.com`.

51 + All roles will default to this variable if left unconfigured.

52 +4. There are also other optional configuration entries that can be found in the same section of the file.

53 +

54 +You can then have different email addresses per **role**, by editing `DEFAULT_RECIPIENT_EMAIL` with the email address you want, in the following entries at the bottom of the same file:

55 +

56 +```conf

57 +role_recipients_email[sysadmin]="systems@example.com"

58 +role_recipients_email[domainadmin]="domains@example.com"

59 +role_recipients_email[dba]="databases@example.com systems@example.com"

60 +role_recipients_email[webmaster]="marketing@example.com development@example.com"

61 +role_recipients_email[proxyadmin]="proxy-admin@example.com"

62 +role_recipients_email[sitemgr]="sites@example.com"

63 ```

44 -Where `[ROLE]` is the role you want to test. The default (if you don't give a `[ROLE]`) is `sysadmin`.

65 +An example of a working configuration would be:

46 -Note that in versions before 1.16, the plugins.d directory may be installed in a different location in certain OSs (e.g. under `/usr/lib/netdata`).

47 -You can always find the location of the alarm-notify.sh script in `netdata.conf`.

67 +```conf

68 +#------------------------------------------------------------------------------

69 +# email global notification options

70 +

71 +EMAIL_SENDER="example@domain.com"

72 +SEND_EMAIL="YES"

73 +DEFAULT_RECIPIENT_EMAIL="recipient@example.com"

74 +```

49 -## Filtering

76 +### Filtering

78 Every notification email (both the plain text and the rich html versions) from the Netdata agent, contain a set of custom email headers that can be used for filtering using an email client. Example:

53 -```

80 +```conf

81 X-Netdata-Severity: warning

82 X-Netdata-Alert-Name: inbound_packets_dropped_ratio

83 X-Netdata-Chart: net_packets.enp2s0

@@ -60,7 +87,7 @@ X-Netdata-Host: winterland

87 X-Netdata-Role: sysadmin

88 ```

63 -## Using msmtp instead of sendmail

90 +### Using msmtp instead of sendmail

92 [msmtp](https://marlam.de/msmtp/) provides a simple alternative to a full-blown local mail server and `sendmail`

93 that will still allow you to send email notifications. It comes pre-installed in our Docker images, and is available

@@ -68,21 +95,29 @@ on most distributions in the system package repositories.

96 To use msmtp with Netdata for sending email alerts:

71 -1. If it’s not already installed, install msmtp. Most distributions have it in their package repositories with the

72 - package name `msmtp`.

98 +1. If it’s not already installed, install msmtp. Most distributions have it in their package repositories with the package name `msmtp`.

99 2. Modify the `sendmail` path in `health_alarm_notify.conf` to point to the location of `msmtp`:

74 -```

75 -# The full path to the sendmail command.

76 -# If empty, the system $PATH will be searched for it.

77 -# If not found, email notifications will be disabled (silently).

78 -sendmail="/usr/bin/msmtp"

79 -```

100 +

101 + ```conf

102 + # The full path to the sendmail command.

103 + # If empty, the system $PATH will be searched for it.

104 + # If not found, email notifications will be disabled (silently).

105 + sendmail="/usr/bin/msmtp"

106 + ```

107 +

108 3. Login as netdata:

81 -```sh

82 -(sudo) su -s /bin/bash netdata

83 -```

109 +

110 + ```sh

111 + (sudo) su -s /bin/bash netdata

112 + ```

113 +

114 4. Configure `~/.msmtprc` as shown [in the documentation](https://marlam.de/msmtp/documentation/).

115 5. Finally set the appropriate permissions on the `.msmtprc` file :

86 -```sh

87 -chmod 600 ~/.msmtprc

88 -```

116 +

117 + ```sh

118 + chmod 600 ~/.msmtprc

119 + ```

120 +

121 +## Test the notification method

122 +

123 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/flock/README.md

+49 -25

@@ -1,42 +1,66 @@

1 -<!--

2 -title: "Flock agent alert notifications"

3 -sidebar_label: "Flock"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/flock/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Flock agent alert notifications

1 +# Flock Agent alert notifications

2 +

3 +Learn how to send notifications to Flock using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 This is what you will get:

11 ![Flock](https://i.imgur.com/ok9bRzw.png)

17 -You need:

13 +## Prerequisites

14 +

15 +You will need:

16 +

17 +- The **incoming webhook URL** as given by flock.com

18 + You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).

19 + Read more about flock webhooks and how to get one [here](https://admin.flock.com/webhooks).

20 +- Terminal access to the Agent you wish to configure

21 +

22 +## Configure Netdata to send alert notifications to Flock

23 +

24 +> ### Info

25 +>

26 +> This file mentions editing configuration files.

27 +>

29 +> Note that to run the script you need to be inside your Netdata config directory.

30 +>

31 +> It is recommended to use this way for configuring Netdata.

19 -The **incoming webhook URL** as given by flock.com.

20 -You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).

33 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

22 -Get them here: <https://admin.flock.com/webhooks>

35 +1. Set `SEND_FLOCK` to `YES`.

36 +2. Set `FLOCK_WEBHOOK_URL` to your webhook URL.

37 +3. Set `DEFAULT_RECIPIENT_FLOCK` to the Flock channel you want the alert notifications to be sent to.

38 + All roles will default to this variable if left unconfigured.

24 -Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

40 +You can then have different channels per **role**, by editing `DEFAULT_RECIPIENT_FLOCK` with the channel you want, in the following entries at the bottom of the same file:

42 +```conf

43 +role_recipients_flock[sysadmin]="systems"

44 +role_recipients_flock[domainadmin]="domains"

45 +role_recipients_flock[dba]="databases systems"

46 +role_recipients_flock[webmaster]="marketing development"

47 +role_recipients_flock[proxyadmin]="proxy-admin"

48 +role_recipients_flock[sitemgr]="sites"

49 ```

27 -###############################################################################

28 -# sending flock notifications

30 -# enable/disable sending pushover notifications

31 -SEND_FLOCK="YES"

51 +The values you provide should already exist as Flock channels.

33 -# Login to flock.com and create an incoming webhook.

34 -# You need only one for all your Netdata servers.

35 -# Without it, Netdata cannot send flock notifications.

36 -FLOCK_WEBHOOK_URL="https://api.flock.com/hooks/sendMessage/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

53 +An example of a working configuration would be:

54 +

55 +```conf

56 +#------------------------------------------------------------------------------

57 +# flock (flock.com) global notification options

38 -# if a role recipient is not configured, no notification will be sent

59 +SEND_FLOCK="YES"

60 +FLOCK_WEBHOOK_URL="https://api.flock.com/hooks/sendMessage/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

61 DEFAULT_RECIPIENT_FLOCK="alarms"

62 ```

64 +## Test the notification method

health/notifications/gotify/README.md

+32 -49

@@ -1,66 +1,49 @@

1 -<!--

2 -title: "Gotify agent alert notifications"

3 -description: "Send alerts to your Gotify instance when an alert gets triggered in Netdata."

4 -sidebar_label: "Gotify"

5 -custom_edit_url: https://github.com/netdata/netdata/edit/master/health/notifications/gotify/README.md

6 -learn_status: "Published"

7 -learn_topic_type: "Tasks"

8 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

9 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

10 --->

11 -

1 # Gotify agent alert notifications

14 -[Gotify](https://gotify.net/) is a self-hosted push notification service created for sending and receiving messages in real time.

15 -

16 -## Configuring Gotify

3 +Learn how to send alerts to your Gotify instance using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

18 -### Prerequisites

5 +> ### Note

6 +>

20 -To use Gotify as your notification service, you need an application token.

21 -You can generate a new token in the Gotify Web UI.

9 +[Gotify](https://gotify.net/) is a self-hosted push notification service created for sending and receiving messages in real time.

23 -### Configuration

11 +This is what you will get:

25 -To set up Gotify in Netdata:

13 +<img src="https://user-images.githubusercontent.com/103264516/162509205-1e88e5d9-96b6-4f7f-9426-182776158128.png" alt="Example alarm notifications in Gotify" width="70%">

27 -1. Switch to your [config

28 -directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md) and edit the file `health_alarm_notify.conf` using the edit config script.

29 -

30 - ```bash

31 - ./edit-config health_alarm_notify.conf

32 - ```

15 +## Prerequisites

34 -2. Change the variable `GOTIFY_APP_TOKEN` to the application token you generated in the Gotify Web UI. Change

35 -`GOTIFY_APP_URL` to point to your Gotify instance.

17 +You will need:

37 - ```conf

38 - SEND_GOTIFY="YES"

19 +- An application token. You can generate a new token in the Gotify Web UI.

20 +- terminal access to the Agent you wish to configure

40 - # Application token

41 - # Gotify instance url

42 - GOTIFY_APP_TOKEN=XXXXXXXXXXXXXXX

43 - GOTIFY_APP_URL=https://push.example.de/

44 - ```

22 +## Configure Netdata to send alert notifications to Gotify

46 - Changes to `health_alarm_notify.conf` do not require a Netdata restart.

47 -

48 -3. Test your Gotify notifications configuration by running the following commands, replacing `ROLE` with your preferred role:

24 +> ### Info

25 +>

26 +> This file mentions editing configuration files.

27 +>

29 +> Note that to run the script you need to be inside your Netdata config directory.

30 +>

31 +> It is recommended to use this way for configuring Netdata.

50 - ```sh

51 - # become user netdata

52 - sudo su -s /bin/bash netdata

33 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

54 - # send a test alarm

55 - /usr/libexec/netdata/plugins.d/alarm-notify.sh test ROLE

56 - ```

35 +1. Set `SET_GOTIFY` to `YES`

36 +2. Set `GOTIFY_APP_TOKEN` to the app token you generated

37 +3. `GOTIFY_APP_URL` to point to your Gotify instance, for example `https://push.example.domain/`

58 - 🟢 If everything works, you'll see alarms in Gotify:

39 +An example of a working configuration would be:

60 - ![Example alarm notifications in Gotify](https://user-images.githubusercontent.com/103264516/162509205-1e88e5d9-96b6-4f7f-9426-182776158128.png)

41 +```conf

42 +SEND_GOTIFY="YES"

43 +GOTIFY_APP_TOKEN="XXXXXXXXXXXXXXX"

44 +GOTIFY_APP_URL="https://push.example.domain/"

45 +```

62 - 🔴 If sending the test notifications fails, check `/var/log/netdata/error.log` to find the relevant error message:

47 +## Test the notification method

64 - ```log

65 - 2020-09-03 23:07:00: alarm-notify.sh: ERROR: failed to send Gotify notification for: hades test.chart.test_alarm is CRITICAL, with HTTP error code 401.

66 - ```

49 +To test this alert refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/irc/README.md

+66 -61

@@ -1,83 +1,88 @@

1 -<!--

2 -title: "IRC agent alert notifications"

3 -sidebar_label: "IRC"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/irc/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# IRC agent alert notifications

1 +# IRC Agent alert notifications

2 +

3 +Learn how to send notifications to IRC using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 This is what you will get:

15 -IRCCloud web client:\

11 +IRCCloud web client:

12 ![image](https://user-images.githubusercontent.com/31221999/36793487-3735673e-1ca6-11e8-8880-d1d8b6cd3bc0.png)

18 -Irssi terminal client:

14 +Irssi terminal client:

15 ![image](https://user-images.githubusercontent.com/31221999/36793486-3713ada6-1ca6-11e8-8c12-70d956ad801e.png)

21 -You need:

22 -

23 -1. The `nc` utility. If you do not set the path, Netdata will search for it in your system `$PATH`.

24 -

25 -Set the path for `nc` in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

26 -

27 -```

28 -#------------------------------------------------------------------------------

29 -# external commands

30 -#

31 -# The full path of the nc command.

32 -# If empty, the system $PATH will be searched for it.

33 -# If not found, irc notifications will be silently disabled.

34 -nc="/usr/bin/nc"

17 +## Prerequisites

18 +

19 +You will need:

20 +

21 +- The `nc` utility.

22 + You can set the path to it, or Netdata will search for it in your system `$PATH`.

23 +- terminal access to the Agent you wish to configure

24 +

25 +## Configure Netdata to send alert notifications to IRC

26 +

27 +> ### Info

28 +>

29 +> This file mentions editing configuration files.

30 +>

32 +> Note that to run the script you need to be inside your Netdata config directory.

33 +>

34 +> It is recommended to use this way for configuring Netdata.

35 +

36 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

37 +

38 +1. Set the path for `nc`, otherwise Netdata will search for it in your system `$PATH`:

39 +

40 + ```conf

41 + #------------------------------------------------------------------------------

42 + # external commands

43 + #

44 + # The full path of the nc command.

45 + # If empty, the system $PATH will be searched for it.

46 + # If not found, irc notifications will be silently disabled.

47 + nc="/usr/bin/nc"

48 + ```

49 +

50 +2. Set `SEND_IRC` to `YES`

51 +3. Set `DEFAULT_RECIPIENT_IRC` to one or more channels to post the messages to.

52 + You can define multiple channels like this: `#alarms #systems`.

53 + All roles will default to this variable if left unconfigured.

54 +4. Set `IRC_NETWORK` to the IRC network which your preferred channels belong to.

55 +5. Set `IRC_PORT` to the IRC port to which a connection will occur.

56 +6. Set `IRC_NICKNAME` to the IRC nickname which is required to send the notification.

57 + It must not be an already registered name as the connection's `MODE` is defined as a `guest`.

58 +7. Set `IRC_REALNAME` to the IRC realname which is required in order to make he connection.

59 +

60 +You can then have different channels per **role**, by editing `DEFAULT_RECIPIENT_IRC` with the channel you want, in the following entries at the bottom of the same file:

61 +

62 +```conf

63 +role_recipients_irc[sysadmin]="#systems"

64 +role_recipients_irc[domainadmin]="#domains"

65 +role_recipients_irc[dba]="#databases #systems"

66 +role_recipients_irc[webmaster]="#marketing #development"

67 +role_recipients_irc[proxyadmin]="#proxy-admin"

68 +role_recipients_irc[sitemgr]="#sites"

69 ```

37 -2. Αn `IRC_NETWORK` to which your preferred channels belong to.

38 -3. One or more channels ( `DEFAULT_RECIPIENT_IRC` ) to post the messages to.

39 -4. An `IRC_NICKNAME` and an `IRC_REALNAME` to identify in IRC.

71 +The values you provide should be IRC channels which belong to the specified IRC network.

41 -Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

73 +An example of a working configuration would be:

43 -```

75 +```conf

76 #------------------------------------------------------------------------------

77 # irc notification options

78 #

47 -# irc notifications require only the nc utility to be installed.

48 -

49 -# multiple recipients can be given like this:

50 -# "<irc_channel_1> <irc_channel_2> ..."

51 -

52 -# enable/disable sending irc notifications

79 SEND_IRC="YES"

54 -

55 -# if a role's recipients are not configured, a notification will not be sent.

56 -# (empty = do not send a notification for unconfigured roles):

80 DEFAULT_RECIPIENT_IRC="#system-alarms"

58 -

59 -# The irc network to which the recipients belong. It must be the full network.

81 IRC_NETWORK="irc.freenode.net"

61 -

62 -# The irc nickname which is required to send the notification. It must not be

63 -# an already registered name as the connection's MODE is defined as a 'guest'.

82 IRC_NICKNAME="netdata-alarm-user"

65 -

66 -# The irc realname which is required in order to make the connection and is an

67 -# extra identifier.

83 IRC_REALNAME="netdata-user"

84 ```

71 -You can define multiple channels like this: `#system-alarms #networking-alarms`.\

72 -You can also filter the notifications like this: `#system-alarms|critical`.\

73 -You can give different channels per **role** using these (at the same file):

74 -

75 -```

76 -role_recipients_irc[sysadmin]="#user-alarms #networking-alarms #system-alarms"

77 -role_recipients_irc[dba]="#databases-alarms"

78 -role_recipients_irc[webmaster]="#networking-alarms"

79 -```

80 -

81 -The keywords `#user-alarms`, `#networking-alarms`, `#system-alarms`, `#databases-alarms` are irc channels which belong to the specified IRC network.

82 -

86 +## Test the notification method

88 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/kavenegar/README.md

+51 -35

@@ -1,51 +1,67 @@

1 -<!--

2 -title: "Kavenegar agent alert notifications"

3 -sidebar_label: "Kavenegar"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/kavenegar/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Kavenegar agent alert notifications

1 +# Kavenegar Agent alert notifications

2 +

3 +Learn how to send notifications to Kavenegar using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 [Kavenegar](https://kavenegar.com/) as service for software developers, based in Iran, provides send and receive SMS, calling voice by using its APIs.

15 -Will look like this on your Android device:

11 +This is what you will get:

12 +

13 +![image](https://user-images.githubusercontent.com/70198089/229841323-6c4b1956-dd91-423e-abaf-2799000f72a8.png)

17 -![image](https://cloud.githubusercontent.com/assets/17090999/20034652/620b6100-a39b-11e6-96af-4f83b8e830e2.png)

15 +## Prerequisites

17 You will need:

21 -1. Signup and Login to kavenegar.com

22 -2. Get your APIKEY and Sender from `http://panel.kavenegar.com/client/setting/account`

23 -3. Fill in KAVENEGAR_API_KEY="" KAVENEGAR_SENDER=""

24 -4. Add the recipient phone numbers to DEFAULT_RECIPIENT_KAVENEGAR=""

19 +- the `APIKEY` and Sender from <http://panel.kavenegar.com/client/setting/account>

20 +- terminal access to the Agent you wish to configure

21 +

22 +## Configure Netdata to send alert notifications to Kavenegar

23 +

24 +> ### Info

25 +>

26 +> This file mentions editing configuration files.

27 +>

29 +> Note that to run the script you need to be inside your Netdata config directory.

30 +>

31 +> It is recommended to use this way for configuring Netdata.

26 -Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

33 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

35 +1. Set `SEND_KAVENEGAR` to `YES`.

36 +2. Set `KAVENEGAR_API_KEY` to your `APIKEY`.

37 +3. Set `KAVENEGAR_SENDER` to the value of your Sender.

38 +4. Set `DEFAULT_RECIPIENT_KAVENEGAR` to the SMS recipient you want the alert notifications to be sent to.

39 + You can define multiple recipients like this: `09155555555 09177777777`.

40 + All roles will default to this variable if lest unconfigured.

41 +

42 +You can then have different SMS recipients per **role**, by editing `DEFAULT_RECIPIENT_KAVENEGAR` with the SMS recipients you want, in the following entries at the bottom of the same file:

43 +

44 +```conf

45 +role_recipients_kavenegar[sysadmin]="09100000000"

46 +role_recipients_kavenegar[domainadmin]="09111111111"

47 +role_recipients_kavenegar[dba]="0922222222"

48 +role_recipients_kavenegar[webmaster]="0933333333"

49 +role_recipients_kavenegar[proxyadmin]="0944444444"

50 +role_recipients_kavenegar[sitemgr]="0955555555"

51 ```

29 -###############################################################################

30 -# Kavenegar (kavenegar.com) SMS options

32 -# multiple recipients can be given like this:

33 -# "09155555555 09177777777"

53 +An example of a working configuration would be:

35 -# enable/disable sending kavenegar SMS

36 -SEND_KAVENEGAR="YES"

55 +```conf

56 +#------------------------------------------------------------------------------

57 +# Kavenegar (Kavenegar.com) SMS options

38 -# to get an access key, after selecting and purchasing your desired service

39 -# at http://kavenegar.com/pricing.html

40 -# login to your account, go to your dashboard and my account are

41 -# https://panel.kavenegar.com/Client/setting/account from API Key

42 -# copy your api key. You can generate new API Key too.

43 -# You can find and select kevenegar sender number from this place.

44 -

45 -# Without an API key, Netdata cannot send KAVENEGAR text messages.

46 -KAVENEGAR_API_KEY=""

47 -KAVENEGAR_SENDER=""

48 -DEFAULT_RECIPIENT_KAVENEGAR=""

59 +SEND_KAVENEGAR="YES"

60 +KAVENEGAR_API_KEY="XXXXXXXXXXXX"

61 +KAVENEGAR_SENDER="YYYYYYYY"

62 +DEFAULT_RECIPIENT_KAVENEGAR="0912345678"

63 ```

65 +## Test the notification method

67 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/matrix/README.md

+53 -40

@@ -1,61 +1,74 @@

1 -<!--

2 -title: "Matrix agent alert notifications"

3 -description: "Stay aware of warning or critical anomalies by sending health alarms to Matrix network rooms with Netdata's health monitoring watchdog."

4 -sidebar_label: "Matrix"

5 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/matrix/README.md"

6 -learn_status: "Published"

7 -learn_topic_type: "Tasks"

8 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

9 --->

1 +# Matrix Agent alert notifications

11 -# Matrix agent alert notifications

3 +Learn how to send notifications to Matrix network rooms using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

13 -Send notifications to [Matrix](https://matrix.org/) network rooms.

5 +> ### Note

6 +>

15 -The requirements for this notification method are:

9 +## Prerequisites

17 -1. The url of the homeserver (`https://homeserver:port`).

18 -2. Credentials for connecting to the homeserver, in the form of a valid access token for your account (or for a

19 - dedicated notification account). These tokens usually don't expire.

20 -3. The room ids that you want to sent the notification to.

11 +You will need:

22 -To obtain the access token, you can use the following `curl` command:

13 +- The url of the homeserver (`https://homeserver:port`).

14 +- Credentials for connecting to the homeserver, in the form of a valid access token for your account (or for a dedicated notification account). These tokens usually don't expire.

15 +- The room ids that you want to sent the notification to.

24 -```bash

25 -curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "https://homeserver:8448/_matrix/client/r0/login"

26 -```

17 +## Configure Netdata to send alert notifications to Matrix

18 +

19 +> ### Info

20 +>

21 +> This file mentions editing configuration files.

22 +>

23 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

24 +> Note that to run the script you need to be inside your Netdata config directory.

25 +>

26 +> It is recommended to use this way for configuring Netdata.

27 +

28 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

29 +

30 +1. Set `SEND_MATRIX` to `YES`.

31 +2. Set `MATRIX_HOMESERVER` to the URL of the Matrix homeserver.

32 +3. Set `MATRIX_ACCESSTOKEN` to the access token from your Matrix account.

33 + To obtain the access token, you can use the following `curl` command:

28 -The room ids are unique identifiers and can be obtained from the room settings in a Matrix client (e.g. Riot). Their

29 -format is `!uniqueid:homeserver`.

35 + ```bash

36 + curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "https://homeserver:8448/_matrix/client/r0/login"

37 + ```

31 -Multiple room ids can be defined by separating with a space character.

39 +4. Set `DEFAULT_RECIPIENT_MATRIX` to the rooms you want the alert notifications to be sent to.

40 + The format is `!roomid:homeservername`.

33 -Detailed information about the Matrix client API is available at the [official

34 -site](https://matrix.org/docs/guides/client-server.html).

42 + The room ids are unique identifiers and can be obtained from the room settings in a Matrix client (e.g. Riot).

36 -Your `health_alarm_notify.conf` should look like this:

44 + You can define multiple rooms like this: `!roomid1:homeservername !roomid2:homeservername`.

45 + All roles will default to this variable if left unconfigured.

46 +

47 +Detailed information about the Matrix client API is available at the [official site](https://matrix.org/docs/guides/client-server.html).

48 +

49 +You can then have different rooms per **role**, by editing `DEFAULT_RECIPIENT_MATRIX` with the `!roomid:homeservername` you want, in the following entries at the bottom of the same file:

51 ```conf

39 -###############################################################################

52 +role_recipients_matrix[sysadmin]="!roomid1:homeservername"

53 +role_recipients_matrix[domainadmin]="!roomid2:homeservername"

54 +role_recipients_matrix[dba]="!roomid3:homeservername"

55 +role_recipients_matrix[webmaster]="!roomid4:homeservername"

56 +role_recipients_matrix[proxyadmin]="!roomid5:homeservername"

57 +role_recipients_matrix[sitemgr]="!roomid6:homeservername"

58 +```

59 +

60 +An example of a working configuration would be:

61 +

62 +```conf

63 +#------------------------------------------------------------------------------

64 # Matrix notifications

41 -#

43 -# enable/disable Matrix notifications

66 SEND_MATRIX="YES"

45 -

46 -# The url of the Matrix homeserver

47 -# e.g https://matrix.org:8448

67 MATRIX_HOMESERVER="https://matrix.org:8448"

49 -

50 -# A access token from a valid Matrix account. Tokens usually don't expire,

51 -# can be controlled from a Matrix client.

52 -# See https://matrix.org/docs/guides/client-server.html

68 MATRIX_ACCESSTOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

54 -

55 -# Specify the default rooms to receive the notification if no rooms are provided

56 -# in a role's recipients.

57 -# The format is !roomid:homeservername

69 DEFAULT_RECIPIENT_MATRIX="!XXXXXXXXXXXX:matrix.org"

70 ```

72 +## Test the notification method

74 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/messagebird/README.md

+47 -31

@@ -1,49 +1,65 @@

1 -<!--

2 -title: "Messagebird agent alert notifications"

3 -sidebar_label: "Messagebird"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/messagebird/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 --->

1 +# MessageBird Agent alert notifications

10 -# Messagebird agent alert notifications

3 +Learn how to send notifications to MessageBird using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

12 -The messagebird notifications will look like this on your Android device:

5 +> ### Note

6 +>

14 -![image](https://cloud.githubusercontent.com/assets/17090999/20034652/620b6100-a39b-11e6-96af-4f83b8e830e2.png)

9 +This is what you will get:

10 +

11 +![image](https://user-images.githubusercontent.com/70198089/229841323-6c4b1956-dd91-423e-abaf-2799000f72a8.png)

12 +

13 +## Prerequisites

15 You will need:

18 -1. Signup and Login to messagebird.com

19 -2. Pick an SMS capable number after sign up to get some free credits

20 -3. Go to <https://www.messagebird.com/app/settings/developers/access>

21 -4. Create a new access key under 'API ACCESS (REST)' (you will want a live key)

22 -5. Fill in MESSAGEBIRD_ACCESS_KEY="XXXXXXXX" MESSAGEBIRD_NUMBER="+XXXXXXXXXXX"

23 -6. Add the recipient phone numbers to DEFAULT_RECIPIENT_MESSAGEBIRD="+XXXXXXXXXXX"

17 +- an access key under 'API ACCESS (REST)' (you will want a live key), you can read more [here](https://developers.messagebird.com/quickstarts/sms/test-credits-api-keys/)

18 +- terminal access to the Agent you wish to configure

19 +

20 +## Configure Netdata to send alert notifications to MessageBird

21 +

22 +> ### Info

23 +>

24 +> This file mentions editing configuration files.

25 +>

26 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

27 +> Note that to run the script you need to be inside your Netdata config directory.

28 +>

29 +> It is recommended to use this way for configuring Netdata.

30 +

31 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

32 +

33 +1. Set `SEND_MESSAGEBIRD` to `YES`.

34 +2. Set `MESSAGEBIRD_ACCESS_KEY` to your API access key.

35 +3. Set `MESSAGEBIRD_NUMBER` to the MessageBird number you want to use for the alert.

36 +4. Set `DEFAULT_RECIPIENT_MESSAGEBIRD` to the number you want the alert notification to be sent as an SMS.

37 + You can define multiple recipients like this: `+15555555555 +17777777777`.

38 + All roles will default to this variable if left unconfigured.

25 -Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

40 +You can then have different recipients per **role**, by editing `DEFAULT_RECIPIENT_MESSAGEBIRD` with the number you want, in the following entries at the bottom of the same file:

42 +```conf

43 +role_recipients_messagebird[sysadmin]="+15555555555"

44 +role_recipients_messagebird[domainadmin]="+15555555556"

45 +role_recipients_messagebird[dba]="+15555555557"

46 +role_recipients_messagebird[webmaster]="+15555555558"

47 +role_recipients_messagebird[proxyadmin]="+15555555559"

48 +role_recipients_messagebird[sitemgr]="+15555555550"

49 ```

50 +

51 +An example of a working configuration would be:

52 +

53 +```conf

54 #------------------------------------------------------------------------------

55 # Messagebird (messagebird.com) SMS options

31 -# multiple recipients can be given like this:

32 -# "+15555555555 +17777777777"

33 -

34 -# enable/disable sending messagebird SMS

57 SEND_MESSAGEBIRD="YES"

36 -

37 -# to get an access key, create a free account at https://www.messagebird.com

38 -# verify and activate the account (no CC info needed)

39 -# login to your account and enter your phonenumber to get some free credits

40 -# to get the API key, click on 'API' in the sidebar, then 'API Access (REST)'

41 -# click 'Add access key' and fill in data (you want a live key to send SMS)

42 -

43 -# Without an access key, Netdata cannot send Messagebird text messages.

58 MESSAGEBIRD_ACCESS_KEY="XXXXXXXX"

59 MESSAGEBIRD_NUMBER="XXXXXXX"

46 -DEFAULT_RECIPIENT_MESSAGEBIRD="XXXXXXX"

60 +DEFAULT_RECIPIENT_MESSAGEBIRD="+15555555555"

61 ```

63 +## Test the notification method

65 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/msteams/README.md

+48 -29

@@ -1,48 +1,67 @@

1 -<!--

2 -title: "Microsoft Teams agent alert notifications"

3 -sidebar_label: "Microsoft Teams"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/msteams/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Microsoft Teams agent alert notifications

1 +# Microsoft Teams Agent alert notifications

2 +

3 +Learn how to send notifications to Microsoft Teams using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 This is what you will get:

10 ![image](https://user-images.githubusercontent.com/1122372/92710359-0385e680-f358-11ea-8c52-f366a4fb57dd.png)

16 -You need:

12 +## Prerequisites

18 -1. The **incoming webhook URL** as given by Microsoft Teams. You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).

19 -2. One or more channels to post the messages to.

14 +You will need:

21 -In Microsoft Teams the channel name is encoded in the URI after `/IncomingWebhook/` (for clarity the marked with `[]` in the following example): `https://outlook.office.com/webhook/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX@XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/IncomingWebhook/[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX`

16 +- the **incoming webhook URL** as given by Microsoft Teams. You can use the same on all your Netdata servers (or you can have multiple if you like - your decision)

17 +- one or more channels to post the messages to

18 +- terminal access to the Agent you wish to configure

23 -You have to replace the encoded channel name by the placeholder `CHANNEL` in `MSTEAMS_WEBHOOK_URL`. The placeholder `CHANNEL` will be replaced by the actual encoded channel name before sending the notification. This makes it possible to publish to several channels in the same team.

20 +## Configure Netdata to send alert notifications to Microsoft Teams

25 -The encoded channel name must then be added to `DEFAULT_RECIPIENTS_MSTEAMS` or to one of the specific variables `role_recipients_msteams[]`. **At least one channel is mandatory for `DEFAULT_RECIPIENTS_MSTEAMS`.**

22 +> ### Info

23 +>

24 +> This file mentions editing configuration files.

25 +>

27 +> Note that to run the script you need to be inside your Netdata config directory.

28 +>

29 +> It is recommended to use this way for configuring Netdata.

27 -Set the webhook and the recipients in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

31 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

29 -```

30 -SEND_MSTEAMS="YES"

33 +1. Set `SEND_MSTEAMS` to `YES`.

34 +2. Set `MSTEAMS_WEBHOOK_URL` to the incoming webhook URL as given by Microsoft Teams.

35 +3. Set `DEFAULT_RECIPIENT_MSTEAMS` to the **encoded** Microsoft Teams channel name you want the alert notifications to be sent to.

36 + In Microsoft Teams the channel name is encoded in the URI after `/IncomingWebhook/`.

37 + You can define multiple channels like this: `CHANNEL1 CHANNEL2`.

38 + All roles will default to this variable if left unconfigured.

39 +4. You can also set the icons and colors for the different alerts in the same section of the file.

32 -MSTEAMS_WEBHOOK_URL="https://outlook.office.com/webhook/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX@XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/IncomingWebhook/CHANNEL/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"

41 +You can then have different channels per **role**, by editing `DEFAULT_RECIPIENT_MSTEAMS` with the channel you want, in the following entries at the bottom of the same file:

34 -DEFAULT_RECIPIENT_MSTEAMS="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

43 +```conf

44 +role_recipients_msteams[sysadmin]="CHANNEL1"

45 +role_recipients_msteams[domainadmin]="CHANNEL2"

46 +role_recipients_msteams[dba]="databases CHANNEL3"

47 +role_recipients_msteams[webmaster]="CHANNEL4"

48 +role_recipients_msteams[proxyadmin]="CHANNEL5"

49 +role_recipients_msteams[sitemgr]="CHANNEL6"

50 ```

37 -You can define multiple recipients by listing the encoded channel names like this: `XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY`.

38 -This example will send the alarm to the two channels specified by their encoded channel names.

52 +The values you provide should already exist as Microsoft Teams channels in the same Team.

40 -You can give different recipients per **role** using these (in the same file):

54 +An example of a working configuration would be:

42 -```

43 -role_recipients_msteams[sysadmin]="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

44 -role_recipients_msteams[dba]="YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"

45 -role_recipients_msteams[webmaster]="ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ"

56 +```conf

57 +#------------------------------------------------------------------------------

58 +# Microsoft Teams (office.com) global notification options

59 +

60 +SEND_MSTEAMS="YES"

61 +MSTEAMS_WEBHOOK_URL="https://outlook.office.com/webhook/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX@XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/IncomingWebhook/CHANNEL/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"

62 +DEFAULT_RECIPIENT_MSTEAMS="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

63 ```

65 +## Test the notification method

health/notifications/opsgenie/README.md

+38 -53

@@ -1,66 +1,51 @@

1 -<!--

2 -title: "Opsgenie agent alert notifications"

3 -description: "Send alerts to your Opsgenie incident response account any time an anomaly or performance issue strikes a node in your infrastructure."

4 -sidebar_label: "Opsgenie"

5 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/opsgenie/README.md"

6 -learn_status: "Published"

7 -learn_topic_type: "Tasks"

8 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

9 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

10 --->

11 -

12 -# Opsgenie agent alert notifications

13 -

14 -[Opsgenie](https://www.atlassian.com/software/opsgenie) is an alerting and incident response tool. It is designed to

15 -group and filter alarms, build custom routing rules for on-call teams, and correlate deployments and commits to

16 -incidents.

17 -

18 -The first step is to create a [Netdata integration](https://docs.opsgenie.com/docs/api-integration) in the

19 -[Opsgenie](https://www.atlassian.com/software/opsgenie) dashboard. After this, you need to edit

20 -`health_alarm_notify.conf` on your system, by running the following from

21 -your [config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md):

22 -

23 -```bash

24 -./edit-config health_alarm_notify.conf

25 -```

1 +# Opsgenie Agent alert notifications

27 -Change the variable `OPSGENIE_API_KEY` with the API key you got from Opsgenie. `OPSGENIE_API_URL` defaults to

28 -`https://api.opsgenie.com`, however there are region-specific API URLs such as `https://eu.api.opsgenie.com`, so set

29 -this if required.

3 +Learn how to send notifications to Opsgenie using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

31 -```conf

32 -SEND_OPSGENIE="YES"

5 +> ### Note

6 +>

34 -# Api key

35 -# Default Opsgenie API

36 -OPSGENIE_API_KEY="11111111-2222-3333-4444-555555555555"

37 -OPSGENIE_API_URL=""

38 -```

9 +[Opsgenie](https://www.atlassian.com/software/opsgenie) is an alerting and incident response tool.

10 +It is designed to group and filter alarms, build custom routing rules for on-call teams, and correlate deployments and commits to incidents.

40 -Changes to `health_alarm_notify.conf` do not require a Netdata restart. You can test your Opsgenie notifications

41 -configuration by issuing the commands, replacing `ROLE` with your preferred role:

12 +This is what you will get:

13 +![Example alarm notifications in

14 +Opsgenie](https://user-images.githubusercontent.com/49162938/92184518-f725f900-ee40-11ea-9afa-e7c639c72206.png)

43 -```sh

44 -# become user netdata

45 -sudo su -s /bin/bash netdata

16 +## Prerequisites

47 -# send a test alarm

48 -/usr/libexec/netdata/plugins.d/alarm-notify.sh test ROLE

49 -```

18 +You will need:

51 -If everything works, you'll see alarms in your Opsgenie platform:

20 +- An Opsgenie integration. You can create an [integration](https://docs.opsgenie.com/docs/api-integration) in the [Opsgenie](https://www.atlassian.com/software/opsgenie) dashboard.

53 -![Example alarm notifications in

54 -Opsgenie](https://user-images.githubusercontent.com/49162938/92184518-f725f900-ee40-11ea-9afa-e7c639c72206.png)

22 +- terminal access to the Agent you wish to configure

56 -If sending the test notifications fails, you can look in `/var/log/netdata/error.log` to find the relevant error

57 -message:

24 +## Configure Netdata to send alert notifications to your Opsgenie account

59 -```log

60 -2020-09-03 23:07:00: alarm-notify.sh: ERROR: failed to send opsgenie notification for: hades test.chart.test_alarm is CRITICAL, with HTTP error code 401.

61 -```

26 +> ### Info

27 +>

28 +> This file mentions editing configuration files.

29 +>

30 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

31 +> Note that to run the script you need to be inside your Netdata config directory.

32 +>

33 +> It is recommended to use this way for configuring Netdata.

34 +

35 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

36 +

37 +1. Set `SEND_OPSGENIE` to `YES`.

38 +2. Set `OPSGENIE_API_KEY` to the API key you got from Opsgenie.

39 +3. `OPSGENIE_API_URL` defaults to `https://api.opsgenie.com`, however there are region-specific API URLs such as `https://eu.api.opsgenie.com`, so set this if required.

40 +

41 +An example of a working configuration would be:

63 -You can find more details about the Opsgenie error codes in

64 -their [response docs](https://docs.opsgenie.com/docs/response).

43 +```conf

44 +SEND_OPSGENIE="YES"

45 +OPSGENIE_API_KEY="11111111-2222-3333-4444-555555555555"

46 +OPSGENIE_API_URL=""

47 +```

49 +## Test the notification method

51 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/pagerduty/README.md

+46 -44

@@ -1,67 +1,69 @@

1 -<!--

2 -title: "PagerDuty agent alert notifications"

3 -description: "Send alerts to your PagerDuty dashboard any time an anomaly or performance issue strikes a node in your infrastructure."

4 -sidebar_label: "PagerDuty"

5 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/pagerduty/README.md"

6 -learn_status: "Published"

7 -learn_topic_type: "Tasks"

8 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

9 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

10 --->

11 -

12 -# PagerDuty agent alert notifications

1 +# PagerDuty Agent alert notifications

2 +

3 +Learn how to send notifications to PagerDuty using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 [PagerDuty](https://www.pagerduty.com/company/) is an enterprise incident resolution service that integrates with ITOps

10 and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to

11 correlating them into incidents, PagerDuty streamlines the incident management process by reducing alert noise and

12 resolution times.

19 -## What you need to get started

14 +## Prerequisites

15 +

16 +You will need:

17 +

18 +- an installation of the [PagerDuty agent](https://www.pagerduty.com/docs/guides/agent-install-guide/) on the node running the Netdata Agent

19 +- a PagerDuty `Generic API` service using either the `Events API v2` or `Events API v1`

20 +- terminal access to the Agent you wish to configure

21 -- An installation of the open-source [Netdata](https://github.com/netdata/netdata/blob/master/packaging/installer/README.md) monitoring agent.

22 -- An installation of the [PagerDuty agent](https://www.pagerduty.com/docs/guides/agent-install-guide/) on the node

23 - running Netdata.

24 -- A PagerDuty `Generic API` service using either the `Events API v2` or `Events API v1`.

22 +## Configure Netdata to send alert notifications to PagerDuty

26 -## Setup

24 +> ### Info

25 +>

26 +> This file mentions editing configuration files.

27 +>

29 +> Note that to run the script you need to be inside your Netdata config directory.

30 +>

31 +> It is recommended to use this way for configuring Netdata.

28 -[Add a new service](https://support.pagerduty.com/docs/services-and-integrations#section-configuring-services-and-integrations)

33 +Firstly, [Add a new service](https://support.pagerduty.com/docs/services-and-integrations#section-configuring-services-and-integrations)

34 to PagerDuty. Click **Use our API directly** and select either `Events API v2` or `Events API v1`. Once you finish

35 creating the service, click on the **Integrations** tab to find your **Integration Key**.

32 -Navigate to the [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) and use

33 -[`edit-config`](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) to open

34 -`health_alarm_notify.conf`.

37 +Then, edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

36 -```bash

37 -cd /etc/netdata

38 -sudo ./edit-config health_alarm_notify.conf

39 -```

40 -

41 -Scroll down to the `# pagerduty.com notification options` section.

39 +1. Set `SEND_PD` to `YES`.

40 +2. Set `DEFAULT_RECIPIENT_PD` to the PagerDuty service key you want the alert notifications to be sent to.

41 + You can define multiple service keys like this: `pd_service_key_1 pd_service_key_2`.

42 + All roles will default to this variable if left unconfigured.

43 +3. If you chose `Events API v2` during service setup on PagerDuty, change `USE_PD_VERSION` to `2`.

43 -Ensure `SEND_PD` is set to `YES`, then copy your Integration Key into `DEFAULT_RECIPIENT_ID`. Change `USE_PD_VERSION` to

44 -`2` if you chose `Events API v2` during service setup on PagerDuty. Minus comments, the section should look like this:

45 +You can then have different PagerDuty service keys per **role**, by editing `DEFAULT_RECIPIENT_PD` with the service key you want, in the following entries at the bottom of the same file:

47 ```conf

47 -SEND_PD="YES"

48 -DEFAULT_RECIPIENT_PD="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

49 -USE_PD_VERSION="2"

48 +role_recipients_pd[sysadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa"

49 +role_recipients_pd[domainadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxb"

50 +role_recipients_pd[dba]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc"

51 +role_recipients_pd[webmaster]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxd"

52 +role_recipients_pd[proxyadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxe"

53 +role_recipients_pd[sitemgr]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxf"

54 ```

52 -## Testing

56 +An example of a working configuration would be:

54 -To test alert notifications to PagerDuty, run the following:

58 +```conf

59 +#------------------------------------------------------------------------------

60 +# pagerduty.com notification options

56 -```bash

57 -sudo su -s /bin/bash netdata

58 -/usr/libexec/netdata/plugins.d/alarm-notify.sh test

62 +SEND_PD="YES"

63 +DEFAULT_RECIPIENT_PD="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

64 +USE_PD_VERSION="2"

65 ```

61 -## Configuration

62 -

63 -Aside from the three values set in `health_alarm_notify.conf`, there is no further configuration required to send alert

64 -notifications to PagerDuty.

67 +## Test the notification method

66 -To configure individual alarms, read our [alert configuration](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md) doc or

67 -the [health entity reference](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md) doc.

69 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/prowl/README.md

+56 -18

@@ -1,21 +1,17 @@

1 -<!--

2 -title: "Prowl agent alert notifications"

3 -sidebar_label: "Prowl"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/prowl/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

1 +# Prowl Agent alert notifications

11 -# Prowl agent alert notifications

3 +Learn how to send notifications to Prowl using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

13 -[Prowl](https://www.prowlapp.com/) is a push notification service for iOS devices. Netdata

14 -supports delivering notifications to iOS devices through Prowl.

5 +> ### Note

6 +>

8 +

9 +[Prowl](https://www.prowlapp.com/) is a push notification service for iOS devices.

10 +Netdata supports delivering notifications to iOS devices through Prowl.

12 Because of how Netdata integrates with Prowl, there is a hard limit of

13 at most 1000 notifications per hour (starting from the first notification

18 -sent). Any alerts beyond the first thousand in an hour will be dropped.

14 +sent). Any alerts beyond the first thousand in an hour will be dropped.

16 Warning messages will be sent with the 'High' priority, critical messages

17 will be sent with the 'Emergency' priority, and all other messages will

@@ -23,10 +19,52 @@ be sent with the normal priority. Opening the notification's associated

19 URL will take you to the Netdata dashboard of the system that issued

20 the alert, directly to the chart that it triggered on.

26 -## configuration

22 +## Prerequisites

23 +

24 +You will need:

25 +

26 +- a Prowl API key, which can be requested through the Prowl website after registering

27 +- terminal access to the Agent you wish to configure

28 +

29 +## Configure Netdata to send alert notifications to Prowl

30 +

31 +> ### Info

32 +>

33 +> This file mentions editing configuration files.

34 +>

35 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

36 +> Note that to run the script you need to be inside your Netdata config directory.

37 +>

38 +> It is recommended to use this way for configuring Netdata.

39 +

40 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

41 +

42 +1. Set `SEND_PROWL` to `YES`.

43 +2. Set `DEFAULT_RECIPIENT_PROWL` to the Prowl API key you want the alert notifications to be sent to.

44 + You can define multiple API keys like this: `APIKEY1, APIKEY2`.

45 + All roles will default to this variable if left unconfigured.

46 +

47 +You can then have different API keys per **role**, by editing `DEFAULT_RECIPIENT_PROWL` with the API keys you want, in the following entries at the bottom of the same file:

48 +

49 +```conf

50 +role_recipients_prowl[sysadmin]="AAAAAAAA"

51 +role_recipients_prowl[domainadmin]="BBBBBBBBB"

52 +role_recipients_prowl[dba]="CCCCCCCCC"

53 +role_recipients_prowl[webmaster]="DDDDDDDDDD"

54 +role_recipients_prowl[proxyadmin]="EEEEEEEEEE"

55 +role_recipients_prowl[sitemgr]="FFFFFFFFFF"

56 +```

57 +

58 +An example of a working configuration would be:

59 +

60 +```conf

61 +#------------------------------------------------------------------------------

62 +# iOS Push Notifications

63 +

64 +SEND_PROWL="YES"

65 +DEFAULT_RECIPIENT_PROWL="XXXXXXXXXX"

66 +```

28 -To use this, you will need a Prowl API key, which can be requested through

29 -the Prowl website after registering.

68 +## Test the notification method

31 -Once you have an API key, simply specify that as a recipient for Prowl

32 -notifications.

70 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/pushbullet/README.md

+54 -37

@@ -1,55 +1,72 @@

1 -<!--

2 -title: "PushBullet agent alert notifications"

3 -sidebar_label: "PushBullet"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/pushbullet/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

1 +# Pushbullet Agent alert notifications

11 -# PushBullet agent alert notifications

3 +Learn how to send notifications to Pushbullet using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

13 -Will look like this on your browser:

14 -![image](https://cloud.githubusercontent.com/assets/4300670/19109636/278b1c0c-8aee-11e6-8a09-7fc94fdbfec8.png)

5 +> ### Note

6 +>

16 -And like this on your Android device:

9 +This is what it will look like this on your browser:

10 +![image](https://user-images.githubusercontent.com/70198089/229842827-e9c93e44-3c86-4ab6-9b44-d8b36a00b015.png)

18 -![image](https://cloud.githubusercontent.com/assets/4300670/19109635/278a1dde-8aee-11e6-9984-0bc87a13312d.png)

12 +And this is what it will look like on your Android device:

13 +

14 +![image](https://user-images.githubusercontent.com/70198089/229842936-ea7e8f92-a353-43ca-a993-b1cc08e8508b.png)

15 +

16 +## Prerequisites

18 You will need:

22 -1. Sign up and log in to [pushbullet.com](https://www.pushbullet.com/)

23 -2. Create a new access token in your [account settings](https://www.pushbullet.com/#settings/account).

24 -3. Fill in the `PUSHBULLET_ACCESS_TOKEN` with the newly generated access token.

25 -4. Add the recipient emails or channel tags (each channel tag must be prefixed with #, e.g. #channeltag) to `DEFAULT_RECIPIENT_PUSHBULLET`.

26 - > 🚨 The pushbullet notification service will send emails to the email recipient, regardless of if they have a pushbullet account.

20 +- a Pushbullet access token that can be created in your [account settings](https://www.pushbullet.com/#settings/account)

21 +- terminal access to the Agent you wish to configure

28 -To add notification channels, run `/etc/netdata/edit-config health_alarm_notify.conf`

23 +## Configure Netdata to send alert notifications to Pushbullet

30 -You can change the configuration like this:

25 +> ### Info

26 +>

27 +> This file mentions editing configuration files.

28 +>

29 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

30 +> Note that to run the script you need to be inside your Netdata config directory.

31 +>

32 +> It is recommended to use this way for configuring Netdata.

32 -```

33 -###############################################################################

34 -# pushbullet (pushbullet.com) push notification options

34 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

36 -# multiple recipients (a combination of email addresses or channel tags) can be given like this:

37 -# "user1@email.com user2@mail.com #channel1 #channel2"

36 +1. Set `Send_PUSHBULLET` to `YES`.

37 +2. Set `PUSHBULLET_ACCESS_TOKEN` to the token you generated.

38 +3. Set `DEFAULT_RECIPIENT_PUSHBULLET` to the email (e.g. `example@domain.com`) or the channel tag (e.g. `#channel`) you want the alert notifications to be sent to.

39 -# enable/disable sending pushbullet notifications

40 -SEND_PUSHBULLET="YES"

40 + > ### Note

41 + >

42 + > Please note that the Pushbullet notification service will send emails to the email recipient, regardless of if they have a Pushbullet account or not.

43 +

44 + You can define multiple entries like this: `user1@email.com user2@email.com`.

45 + All roles will default to this variable if left unconfigured.

46 +4. While optional, you can also set `PUSHBULLET_SOURCE_DEVICE` to the identifier of the sending device.

42 -# Signup and Login to pushbullet.com

43 -# To get your Access Token, go to https://www.pushbullet.com/#settings/account

44 -# And create a new access token

45 -# Then just set the recipients emails and/or channel tags (channel tags must be prefixed with #)

46 -# Please note that the if an email in the DEFAULT_RECIPIENT_PUSHBULLET does

47 -# not have a pushbullet account, the pushbullet service will send an email

48 -# to that address instead

48 +You can then have different recipients per **role**, by editing `DEFAULT_RECIPIENT_PUSHBULLET` with the recipients you want, in the following entries at the bottom of the same file:

50 -# Without an access token, Netdata cannot send pushbullet notifications.

51 -PUSHBULLET_ACCESS_TOKEN="o.Sometokenhere"

50 +```conf

51 +role_recipients_pushbullet[sysadmin]="user1@email.com"

52 +role_recipients_pushbullet[domainadmin]="user2@mail.com"

53 +role_recipients_pushbullet[dba]="#channel1"

54 +role_recipients_pushbullet[webmaster]="#channel2"

55 +role_recipients_pushbullet[proxyadmin]="user3@mail.com"

56 +role_recipients_pushbullet[sitemgr]="user4@mail.com"

57 +```

58 +

59 +An example of a working configuration would be:

60 +

61 +```conf

62 +#------------------------------------------------------------------------------

63 +# pushbullet (pushbullet.com) push notification options

64 +

65 +SEND_PUSHBULLET="YES"

66 +PUSHBULLET_ACCESS_TOKEN="XXXXXXXXX"

67 DEFAULT_RECIPIENT_PUSHBULLET="admin1@example.com admin3@somemail.com #examplechanneltag #anotherchanneltag"

68 ```

70 +## Test the notification method

72 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/pushover/README.md

+57 -18

@@ -1,28 +1,67 @@

1 -<!--

2 -title: "PushOver agent alert notifications"

3 -sidebar_label: "PushOver"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/pushover/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

1 +# Pushover Agent alert notifications

11 -# PushOver agent alert notifications

3 +Learn how to send notification to Pushover using Netdata's Agent alert notification

4 +feature, which supports dozens of endpoints, user roles, and more.

13 -pushover.net allows you to receive push notifications on your mobile phone. The service seems free for up to 7.500 messages per month.

6 +> ### Note

7 +>

8 +> This file assumes you have read the [Introduction to Agent alert notifications](https://github.com/netdata/netdata/blob/master/health/notifications/README.md), detailing how the Netdata Agent's alert notification method works.

15 -Netdata will send warning messages with priority `0` and critical messages with priority `1`. pushover.net allows you to select do-not-disturb hours. The way this is configured, critical notifications will ring and vibrate your phone, even during the do-not-disturb-hours. All other notifications will be delivered silently.

10 +This is what you will get:

17 -You need:

12 +![image](https://user-images.githubusercontent.com/70198089/229842244-4ac998bb-6158-4955-ac2d-766a9999cc98.png)

19 -1. APP TOKEN. You can use the same on all your Netdata servers.

20 -2. USER TOKEN for each user you are going to send notifications to. This is the actual recipient of the notification.

14 +Netdata will send warning messages with priority `0` and critical messages with priority `1`. Pushover allows you to select do-not-disturb hours. The way this is configured, critical notifications will ring and vibrate your phone, even during the do-not-disturb-hours. All other notifications will be delivered silently.

22 -The configuration is like above (slack messages).

16 +## Prerequisites

24 -pushover.net notifications look like this:

18 +You will need:

26 -![image](https://cloud.githubusercontent.com/assets/2662304/18407319/839c10c4-7715-11e6-92c0-12f8215128d3.png)

20 +- An Application token. You can use the same on all your Netdata servers.

21 +- A User token for each user you are going to send notifications to. This is the actual recipient of the notification.

22 +- terminal access to the Agent you wish to configure

24 +## Configure Netdata to send alert notifications to Pushover

26 +> ### Info

27 +>

28 +> This file mentions editing configuration files.

29 +>

31 +> Note that to run the script you need to be inside your Netdata config directory.

32 +>

33 +> It is recommended to use this way for configuring Netdata.

34 +

35 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

36 +

37 +1. Set `SEND_PUSHOVER` to `YES`.

38 +2. Set `PUSHOVER_APP_TOKEN` to your Pushover Application token.

39 +3. Set `DEFAULT_RECIPIENT_PUSHOVER` to the Pushover User token you want the alert notifications to be sent to.

40 + You can define multiple User tokens like this: `USERTOKEN1 USERTOKEN2`.

41 + All roles will default to this variable if left unconfigured.

42 +

43 +You can then have different User tokens per **role**, by editing `DEFAULT_RECIPIENT_PUSHOVER` with the token you want, in the following entries at the bottom of the same file:

44 +

45 +```conf

46 +role_recipients_pushover[sysadmin]="USERTOKEN1"

47 +role_recipients_pushover[domainadmin]="USERTOKEN2"

48 +role_recipients_pushover[dba]="USERTOKEN3 USERTOKEN4"

49 +role_recipients_pushover[webmaster]="USERTOKEN5"

50 +role_recipients_pushover[proxyadmin]="USERTOKEN6"

51 +role_recipients_pushover[sitemgr]="USERTOKEN7"

52 +```

53 +

54 +An example of a working configuration would be:

55 +

56 +```conf

57 +#------------------------------------------------------------------------------

58 +# pushover (pushover.net) global notification options

59 +

60 +SEND_PUSHOVER="YES"

61 +PUSHOVER_APP_TOKEN="XXXXXXXXX"

62 +DEFAULT_RECIPIENT_PUSHOVER="USERTOKEN"

63 +```

64 +

65 +## Test the notification method

66 +

health/notifications/rocketchat/README.md

+47 -38

@@ -1,57 +1,66 @@

1 -<!--

2 -title: "Rocket.Chat agent alert notifications"

3 -sidebar_label: "Rocket Chat"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/rocketchat/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Rocket.Chat agent alert notifications

1 +# Rocket.Chat Agent alert notifications

2 +

3 +Learn how to send notifications to Rocket.Chat using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 This is what you will get:

10 ![Netdata on RocketChat](https://i.imgur.com/Zu4t3j3.png)

15 -You need:

17 -1. The **incoming webhook URL** as given by RocketChat. You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).

18 -2. One or more channels to post the messages to.

12 +## Prerequisites

20 -Get them here: <https://rocket.chat/docs/administrator-guides/integrations/index.html#how-to-create-a-new-incoming-webhook>

14 +You will need:

22 -Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

16 +- The **incoming webhook URL** as given by RocketChat. You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).

17 +- one or more channels to post the messages to.

18 +- terminal access to the Agent you wish to configure

24 -```

25 -#------------------------------------------------------------------------------

26 -# rocketchat (rocket.chat) global notification options

20 +## Configure Netdata to send alert notifications to Rocket.Chat

28 -# multiple recipients can be given like this:

29 -# "CHANNEL1 CHANNEL2 ..."

22 +> ### Info

23 +>

24 +> This file mentions editing configuration files.

25 +>

27 +> Note that to run the script you need to be inside your Netdata config directory.

28 +>

29 +> It is recommended to use this way for configuring Netdata.

31 -# enable/disable sending rocketchat notifications

32 -SEND_ROCKETCHAT="YES"

31 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

34 -# Login to rocket.chat and create an incoming webhook. You need only one for all

35 -# your Netdata servers (or you can have one for each of your Netdata).

36 -# Without it, Netdata cannot send rocketchat notifications.

37 -ROCKETCHAT_WEBHOOK_URL="<your_incoming_webhook_url>"

33 +1. Set `SEND_ROCKETCHAT` to `YES`.

34 +2. Set `ROCKETCHAT_WEBHOOK_URL` to your webhook URL.

35 +3. Set `DEFAULT_RECIPIENT_ROCKETCHAT` to the channel you want the alert notifications to be sent to.

36 + You can define multiple channels like this: `alerts systems`.

37 + All roles will default to this variable if left unconfigured.

39 -# if a role's recipients are not configured, a notification will be send to

40 -# this rocketchat channel (empty = do not send a notification for unconfigured

41 -# roles).

42 -DEFAULT_RECIPIENT_ROCKETCHAT="monitoring_alarms"

43 -```

39 +You can then have different channels per **role**, by editing `DEFAULT_RECIPIENT_ROCKETCHAT` with the channel you want, in the following entries at the bottom of the same file:

45 -You can define multiple channels like this: `alarms systems`.

46 -You can give different channels per **role** using these (at the same file):

47 -

48 -```

41 +```conf

42 role_recipients_rocketchat[sysadmin]="systems"

43 +role_recipients_rocketchat[domainadmin]="domains"

44 role_recipients_rocketchat[dba]="databases systems"

45 role_recipients_rocketchat[webmaster]="marketing development"

46 +role_recipients_rocketchat[proxyadmin]="proxy_admin"

47 +role_recipients_rocketchat[sitemgr]="sites"

48 +```

49 +

50 +The values you provide should already exist as Rocket.Chat channels.

51 +

52 +An example of a working configuration would be:

53 +

54 +```conf

55 +#------------------------------------------------------------------------------

56 +# rocketchat (rocket.chat) global notification options

57 +

58 +SEND_ROCKETCHAT="YES"

59 +ROCKETCHAT_WEBHOOK_URL="<your_incoming_webhook_url>"

60 +DEFAULT_RECIPIENT_ROCKETCHAT="monitoring_alarms"

61 ```

54 -The keywords `systems`, `databases`, `marketing`, `development` are RocketChat channels (they should already exist).

55 -Both public and private channels can be used, even if they differ from the channel configured in your RocketChat incoming webhook.

63 +## Test the notification method

health/notifications/slack/README.md

+38 -39

@@ -1,55 +1,54 @@

1 -<!--

2 -title: "Slack agent alert notifications"

3 -sidebar_label: "Slack"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/slack/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Slack agent alert notifications

1 +# Slack Agent alert notifications

2 +

3 +Learn how to send notifications to a Slack workspace using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 This is what you will get:

14 -![image](https://cloud.githubusercontent.com/assets/2662304/18407116/bbd0fee6-7710-11e6-81cf-58c0defaee2b.png)

16 -You need:

11 +![image](https://user-images.githubusercontent.com/70198089/229841857-77ed2562-ee62-427b-803a-cef03d08238d.png)

18 -1. The **incoming webhook URL** as given by slack.com. You can use the same on all your Netdata servers (or you can have multiple if you like - your decision).

19 -2. One or more channels to post the messages to.

21 -To get a webhook that works on multiple channels, you will need to login to your slack.com workspace and create an incoming webhook using the [Incoming Webhooks App](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks).

22 -Do NOT use the instructions in <https://api.slack.com/incoming-webhooks#enable_webhooks>, as the particular webhooks work only for a single channel.

14 +## Prerequisites

24 -Set the webhook and the recipients in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

16 +You will need:

26 -```

27 -SEND_SLACK="YES"

18 +- a Slack app along with an incoming webhook, read Slack's guide on the topic [here](https://api.slack.com/messaging/webhooks)

19 +- one or more channels to post the messages to

20 +- terminal access to the Agent you wish to configure

29 -SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXX/XXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

22 +## Configure Netdata to send alert notifications to Slack

31 -# if a role's recipients are not configured, a notification will be send to:

32 -# - A slack channel (syntax: '#channel' or 'channel')

33 -# - A slack user (syntax: '@user')

34 -# - The channel or user defined in slack for the webhook (syntax: '#')

35 -# empty = do not send a notification for unconfigured roles

36 -DEFAULT_RECIPIENT_SLACK="alarms"

37 -```

24 +> ### Info

25 +>

26 +> This file mentions editing configuration files.

27 +>

29 +> Note that to run the script you need to be inside your Netdata config directory.

30 +>

31 +> It is recommended to use this way for configuring Netdata.

39 -You can define multiple recipients like this: `# #alarms systems @myuser`.

40 -This example will send the alarm to:

33 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

42 -- The recipient defined in slack for the webhook (not known to Netdata)

43 -- The channel 'alarms'

44 -- The channel 'systems'

45 -- The user @myuser

35 +1. Set `SEND_SLACK` to `YES`.

36 +2. Set `SLACK_WEBHOOK_URL` to your Slack app's webhook URL.

37 +3. Set `DEFAULT_RECIPIENT_SLACK` to the Slack channel your Slack app is set to send messages to.

38 + The syntax for channels is `#channel` or `channel`.

39 + All roles will default to this variable if left unconfigured.

47 -You can give different recipients per **role** using these (at the same file):

41 +An example of a working configuration would be:

49 -```

50 -role_recipients_slack[sysadmin]="systems"

51 -role_recipients_slack[dba]="databases systems"

52 -role_recipients_slack[webmaster]="marketing development"

43 +```conf

44 +#------------------------------------------------------------------------------

45 +# slack (slack.com) global notification options

46 +

47 +SEND_SLACK="YES"

48 +SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXX/XXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

49 +DEFAULT_RECIPIENT_SLACK="#alarms"

50 ```

52 +## Test the notification method

54 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/smstools3/README.md

+56 -32

@@ -1,49 +1,73 @@

1 -<!--

2 -title: "SMS Server Tools 3 agent alert notifications"

3 -sidebar_label: "SMS server"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/smstools3/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# SMS Server Tools 3 agent alert notifications

1 +# SMS Server Tools 3 Agent alert notifications

2 +

3 +Learn how to send notifications to `smstools3` using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 The [SMS Server Tools 3](http://smstools3.kekekasvi.com/) is a SMS Gateway software which can send and receive short messages through GSM modems and mobile phones.

15 -To have Netdata send notifications via SMS Server Tools 3, you'll first need to [install](http://smstools3.kekekasvi.com/index.php?p=compiling) and [configure](http://smstools3.kekekasvi.com/index.php?p=configure) smsd.

11 +## Prerequisites

17 -Ensure that the user `netdata` can execute `sendsms`. Any user executing `sendsms` needs to:

13 +You will need:

19 -- Have write permissions to `/tmp` and `/var/spool/sms/outgoing`

20 -- Be a member of group `smsd`

15 +- to [install](http://smstools3.kekekasvi.com/index.php?p=compiling) and [configure](http://smstools3.kekekasvi.com/index.php?p=configure) smsd

22 -To ensure that the steps above are successful, just `su netdata` and execute `sendsms phone message`.

17 +- To ensure that the user `netdata` can execute `sendsms`. Any user executing `sendsms` needs to:

18 + - have write permissions to `/tmp` and `/var/spool/sms/outgoing`

19 + - be a member of group `smsd`

24 -You then just need to configure the recipient phone numbers in `health_alarm_notify.conf`:

21 + To ensure that the steps above are successful, just `su netdata` and execute `sendsms phone message`.

22 +- terminal access to the Agent you wish to configure

26 -```sh

27 -#------------------------------------------------------------------------------

28 -# SMS Server Tools 3 (smstools3) global notification options

24 +## Configure Netdata to send alert notifications to SMS Server Tools 3

30 -# enable/disable sending SMS Server Tools 3 SMS notifications

31 -SEND_SMS="YES"

26 +> ### Info

27 +>

28 +> This file mentions editing configuration files.

29 +>

31 +> Note that to run the script you need to be inside your Netdata config directory.

32 +>

33 +> It is recommended to use this way for configuring Netdata.

34 +

35 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

33 -# if a role's recipients are not configured, a notification will be sent to

34 -# this SMS channel (empty = do not send a notification for unconfigured

35 -# roles). Multiple recipients can be given like this: "PHONE1 PHONE2 ..."

37 +1. Set the path for `sendsms`, otherwise Netdata will search for it in your system `$PATH`:

37 -DEFAULT_RECIPIENT_SMS=""

39 + ```conf

40 + # The full path of the sendsms command (smstools3).

41 + # If empty, the system $PATH will be searched for it.

42 + # If not found, SMS notifications will be silently disabled.

43 + sendsms="/usr/bin/sendsms"

44 + ```

45 +

46 +2. Set `SEND_SMS` to `YES`.

47 +3. Set `DEFAULT_RECIPIENT_SMS` to the phone number you want the alert notifications to be sent to.

48 + You can define multiple phone numbers like this: `PHONE1 PHONE2`.

49 + All roles will default to this variable if left unconfigured.

50 +

51 +You can then have different phone numbers per **role**, by editing `DEFAULT_RECIPIENT_IRC` with the phone number you want, in the following entries at the bottom of the same file:

52 +

53 +```conf

54 +role_recipients_sms[sysadmin]="PHONE1"

55 +role_recipients_sms[domainadmin]="PHONE2"

56 +role_recipients_sms[dba]="PHONE3"

57 +role_recipients_sms[webmaster]="PHONE4"

58 +role_recipients_sms[proxyadmin]="PHONE5"

59 +role_recipients_sms[sitemgr]="PHONE6"

60 ```

40 -Netdata uses the script `sendsms` that is installed by `smstools3` and just passes a phone number and a message to it. If `sendsms` is not in `$PATH`, you can pass its location in `health_alarm_notify.conf`:

62 +An example of a working configuration would be:

42 -```sh

43 -# The full path of the sendsms command (smstools3).

44 -# If empty, the system $PATH will be searched for it.

45 -# If not found, SMS notifications will be silently disabled.

46 -sendsms=""

64 +```conf

65 +#------------------------------------------------------------------------------

66 +# SMS Server Tools 3 (smstools3) global notification options

67 +SEND_SMS="YES"

68 +DEFAULT_RECIPIENT_SMS="1234567890"

69 ```

71 +## Test the notification method

73 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/syslog/README.md

+59 -21

@@ -1,39 +1,77 @@

1 -<!--

2 -title: "Syslog agent alert notifications"

3 -sidebar_label: "Syslog"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/syslog/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

1 +# Syslog Agent alert notifications

11 -# Syslog agent alert notifications

3 +Learn how to send notifications to Syslog using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

13 -You need a working `logger` command for this to work. This is the case on pretty much every Linux system in existence, and most BSD systems.

5 +> ### Note

6 +>

9 Logged messages will look like this:

17 -```

11 +```bash

12 netdata WARNING on hostname at Tue Apr 3 09:00:00 EDT 2018: disk_space._ out of disk space time = 5h

13 ```

21 -## configuration

15 +## Prerequisites

23 -System log targets are configured as recipients in [`/etc/netdata/health_alarm_notify.conf`](https://github.com/netdata/netdata/blob/36bedc044584dea791fd29455bdcd287c3306cb2/conf.d/health_alarm_notify.conf#L534) (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`).

17 +You will need:

25 -You can also configure per-role targets in the same file a bit further down.

19 +- A working `logger` command for this to work. This is the case on pretty much every Linux system in existence, and most BSD systems.

20 +- terminal access to the Agent you wish to configure

27 -Targets are defined as follows:

22 +## Configure Netdata to send alert notifications to Syslog

29 -```

30 -[[facility.level][@host[:port]]/]prefix

24 +> ### Info

25 +>

26 +> This file mentions editing configuration files.

27 +>

29 +> Note that to run the script you need to be inside your Netdata config directory.

30 +>

31 +> It is recommended to use this way for configuring Netdata.

32 +

33 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

34 +

35 +1. Set `SYSLOG_FACILITY` to the facility used for logging, by default this value is set to `local6`.

36 +2. Set `DEFAULT_RECIPIENT_SYSLOG` to the recipient you want the alert notifications to be sent to.

37 + Targets are defined as follows:

38 +

39 + ```conf

40 + [[facility.level][@host[:port]]/]prefix

41 + ```

42 +

43 + `prefix` defines what the log messages are prefixed with. By default, all lines are prefixed with 'netdata'.

44 +

45 + The `facility` and `level` are the standard syslog facility and level options, for more info on them see your local `logger` and `syslog` documentation. By default, Netdata will log to the `local6` facility, with a log level dependent on the type of message (`crit` for CRITICAL, `warning` for WARNING, and `info` for everything else).

46 +

47 + You can configure sending directly to remote log servers by specifying a host (and optionally a port). However, this has a somewhat high overhead, so it is much preferred to use your local syslog daemon to handle the forwarding of messages to remote systems (pretty much all of them allow at least simple forwarding, and most of the really popular ones support complex queueing and routing of messages to remote log servers).

48 +

49 + You can define multiple recipients like this: `daemon.notice@loghost:514/netdata daemon.notice@loghost2:514/netdata`.

50 + All roles will default to this variable if left unconfigured.

51 +3. Lastly, set `SEND_SYSLOG` to `YES`, make sure you have everything else configured _before_ turning this on.

52 +

53 +You can then have different recipients per **role**, by editing `DEFAULT_RECIPIENT_SYSLOG` with the recipient you want, in the following entries at the bottom of the same file:

54 +

55 +```conf

56 +role_recipients_syslog[sysadmin]="daemon.notice@loghost1:514/netdata"

57 +role_recipients_syslog[domainadmin]="daemon.notice@loghost2:514/netdata"

58 +role_recipients_syslog[dba]="daemon.notice@loghost3:514/netdata"

59 +role_recipients_syslog[webmaster]="daemon.notice@loghost4:514/netdata"

60 +role_recipients_syslog[proxyadmin]="daemon.notice@loghost5:514/netdata"

61 +role_recipients_syslog[sitemgr]="daemon.notice@loghost6:514/netdata"

62 ```

33 -`prefix` defines what the log messages are prefixed with. By default, all lines are prefixed with 'netdata'.

64 +An example of a working configuration would be:

35 -The `facility` and `level` are the standard syslog facility and level options, for more info on them see your local `logger` and `syslog` documentation. By default, Netdata will log to the `local6` facility, with a log level dependent on the type of message (`crit` for CRITICAL, `warning` for WARNING, and `info` for everything else).

66 +```conf

67 +#------------------------------------------------------------------------------

68 +# syslog notifications

37 -You can configure sending directly to remote log servers by specifying a host (and optionally a port). However, this has a somewhat high overhead, so it is much preferred to use your local syslog daemon to handle the forwarding of messages to remote systems (pretty much all of them allow at least simple forwarding, and most of the really popular ones support complex queueing and routing of messages to remote log servers).

70 +SEND_SYSLOG="YES"

71 +SYSLOG_FACILITY='local6'

72 +DEFAULT_RECIPIENT_SYSLOG="daemon.notice@loghost6:514/netdata"

73 +```

75 +## Test the notification method

77 +To test this alert notification method refer to the ["Testing Alert Notifications"](https://github.com/netdata/netdata/blob/master/health/notifications/README.md#testing-alert-notifications) section of the Agent alert notifications page.

health/notifications/telegram/README.md

+54 -33

@@ -1,50 +1,71 @@

1 -<!--

2 -title: "Telegram agent alert notifications"

3 -sidebar_label: "Telegram"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/telegram/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Telegram agent alert notifications

1 +# Telegram Agent alert notifications

2 +

3 +Learn how to send notifications to Telegram using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 [Telegram](https://telegram.org/) is a messaging app with a focus on speed and security, it’s super-fast, simple and free. You can use Telegram on all your devices at the same time — your messages sync seamlessly across any number of your phones, tablets or computers.

15 -With Telegram, you can send messages, photos, videos and files of any type (doc, zip, mp3, etc), as well as create groups for up to 100,000 people or channels for broadcasting to unlimited audiences. You can write to your phone contacts and find people by their usernames. As a result, Telegram is like SMS and email combined — and can take care of all your personal or business messaging needs.

11 +Telegram messages look like this:

12 +

13 +<img src="https://user-images.githubusercontent.com/1153921/66612223-f07dfb80-eb75-11e9-976f-5734ffd93ecd.png" width="50%"></img>

15 Netdata will send warning messages without vibration.

19 -You need to:

17 +## Prerequisites

18 +

19 +You will need:

20 +

21 +- A bot token. To get one, contact the [@BotFather](https://t.me/BotFather) bot and send the command `/newbot` and follow the instructions.

22 + Start a conversation with your bot or invite it into a group where you want it to send messages.

23 +- The chat ID for every chat you want to send messages to. Contact the [@myidbot](https://t.me/myidbot) bot and send the `/getid` command to get your personal chat ID or invite it into a group and use the `/getgroupid` command to get the group chat ID. Group IDs start with a hyphen, supergroup IDs start with `-100`.

21 -1. Get a bot token. To get one, contact the [@BotFather](https://t.me/BotFather) bot and send the command `/newbot`. Follow the instructions.

22 -2. Start a conversation with your bot or invite it into a group where you want it to send messages.

23 -3. Find the chat ID for every chat you want to send messages to. Contact the [@myidbot](https://t.me/myidbot) bot and send the `/getid` command to get your personal chat ID or invite it into a group and use the `/getgroupid` command to get the group chat ID. Group IDs start with a hyphen, supergroup IDs start with `-100`.

25 Alternatively, you can get the chat ID directly from the bot API. Send *your* bot a command in the chat you want to use, then check `https://api.telegram.org/bot{YourBotToken}/getUpdates`, eg. `https://api.telegram.org/bot111122223:7OpFlFFRzRBbrUUmIjj5HF9Ox2pYJZy5/getUpdates`

25 -4. Set the bot token and the chat ID of the recipient in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

26 -```

27 -SEND_TELEGRAM="YES"

28 -TELEGRAM_BOT_TOKEN="111122223:7OpFlFFRzRBbrUUmIjj5HF9Ox2pYJZy5"

29 -DEFAULT_RECIPIENT_TELEGRAM="-100233335555"

30 -```

26 +- terminal access to the Agent you wish to configure

32 -You can define multiple recipients like this: `"-100311112222 212341234|critical"`.

33 -This example will send:

28 +## Configure Netdata to send alert notifications to Telegram

35 -- All alerts to the group with ID -100311112222

36 -- Critical alerts to the user with ID 212341234

30 +> ### Info

31 +>

32 +> This file mentions editing configuration files.

33 +>

34 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

35 +> Note that to run the script you need to be inside your Netdata config directory.

36 +>

37 +> It is recommended to use this way for configuring Netdata.

38 -You can give different recipients per **role** using these (in the same file):

39 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

40 -```

41 -role_recipients_telegram[sysadmin]="212341234"

42 -role_recipients_telegram[dba]="-1004444333321"

43 -role_recipients_telegram[webmaster]="49999333322 -1009999222255"

41 +1. Set `SEND_TELEGRAM` to `YES`.

42 +2. Set `TELEGRAM_BOT_TOKEN` to your bot token.

43 +3. Set `DEFAULT_RECIPIENT_TELEGRAM` to the chat ID you want the alert notifications to be sent to.

44 + You can define multiple chat IDs like this: `49999333322 -1009999222255`.

45 + All roles will default to this variable if left unconfigured.

46 +

47 +You can then have different chats per **role**, by editing `DEFAULT_RECIPIENT_TELEGRAM` with the chat ID you want, in the following entries at the bottom of the same file:

48 +

49 +```conf

50 +role_recipients_telegram[sysadmin]="49999333324"

51 +role_recipients_telegram[domainadmin]="49999333389"

52 +role_recipients_telegram[dba]="-1009999222255"

53 +role_recipients_telegram[webmaster]="-1009999222255 49999333389"

54 +role_recipients_telegram[proxyadmin]="49999333344"

55 +role_recipients_telegram[sitemgr]="49999333876"

56 ```

46 -Telegram messages look like this:

58 +An example of a working configuration would be:

48 -![Netdata notifications via Telegram](https://user-images.githubusercontent.com/1153921/66612223-f07dfb80-eb75-11e9-976f-5734ffd93ecd.png)

60 +```conf

61 +#------------------------------------------------------------------------------

62 +# telegram (telegram.org) global notification options

63 +

64 +SEND_TELEGRAM="YES"

65 +TELEGRAM_BOT_TOKEN="111122223:7OpFlFFRzRBbrUUmIjj5HF9Ox2pYJZy5"

66 +DEFAULT_RECIPIENT_TELEGRAM="-100233335555"

67 +```

69 +## Test the notification method

health/notifications/twilio/README.md

+52 -32

@@ -1,52 +1,72 @@

1 -<!--

2 -title: "Twilio agent alert notifications"

3 -sidebar_label: "Twilio"

4 -custom_edit_url: "https://github.com/netdata/netdata/edit/master/health/notifications/twilio/README.md"

5 -learn_status: "Published"

6 -learn_topic_type: "Tasks"

7 -learn_rel_path: "Integrations/Notify/Agent alert notifications"

8 -learn_autogeneration_metadata: "{'part_of_cloud': False, 'part_of_agent': True}"

9 --->

10 -

11 -# Twilio agent alert notifications

1 +# Twilio Agent alert notifications

2 +

3 +Learn how to send notifications to Twilio using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

4 +

5 +> ### Note

6 +>

9 Will look like this on your Android device:

15 -![image](https://cloud.githubusercontent.com/assets/17090999/20034652/620b6100-a39b-11e6-96af-4f83b8e830e2.png)

11 +![image](https://user-images.githubusercontent.com/70198089/229841323-6c4b1956-dd91-423e-abaf-2799000f72a8.png)

12 +

13 +

14 +## Prerequisites

16 You will need:

19 -1. Signup and Login to twilio.com

20 -2. Pick an SMS capable number during sign up.

21 -3. Get your SID, and Token from <https://www.twilio.com/console>

22 -4. Fill in TWILIO_ACCOUNT_SID="XXXXXXXX" TWILIO_ACCOUNT_TOKEN="XXXXXXXXX" TWILIO_NUMBER="+XXXXXXXXXXX"

23 -5. Add the recipient phone numbers to DEFAULT_RECIPIENT_TWILIO="+XXXXXXXXXXX"

18 +- to get your SID, and Token from <https://www.twilio.com/console>

19 +- terminal access to the Agent you wish to configure

25 -!!PLEASE NOTE THAT IF YOUR ACCOUNT IS A TRIAL ACCOUNT YOU WILL ONLY BE ABLE TO SEND NOTIFICATIONS TO THE NUMBER YOU SIGNED UP WITH

21 +## Configure Netdata to send alert notifications to Twilio

27 -Set them in `/etc/netdata/health_alarm_notify.conf` (to edit it on your system run `/etc/netdata/edit-config health_alarm_notify.conf`), like this:

23 +> ### Info

24 +>

25 +> This file mentions editing configuration files.

26 +>

27 +> - To edit configuration files in a safe way, we provide the [`edit config` script](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#use-edit-config-to-edit-configuration-files) located in your [Netdata config directory](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md#the-netdata-config-directory) (typically is `/etc/netdata`) that creates the proper file and opens it in an editor automatically.

28 +> Note that to run the script you need to be inside your Netdata config directory.

29 +>

30 +> It is recommended to use this way for configuring Netdata.

29 -```

30 -###############################################################################

31 -# Twilio (twilio.com) SMS options

32 +Edit `health_alarm_notify.conf`, changes to this file do not require restarting Netdata:

33 -# multiple recipients can be given like this:

34 -# "+15555555555 +17777777777"

34 +1. Set `SEND_TWILIO` to `YES`.

35 +2. Set `TWILIO_ACCOUNT_SID` to your account SID.

36 +3. Set `TWILIO_ACCOUNT_TOKEN` to your account token.

37 +4. Set `TWILIO_NUMBER` to your account's number.

38 +5. Set `DEFAULT_RECIPIENT_TWILIO` to the number you want the alert notifications to be sent to.

39 + You can define multiple numbers like this: `+15555555555 +17777777777`.

40 + All roles will default to this variable if left unconfigured.

36 -# enable/disable sending twilio SMS

37 -SEND_TWILIO="YES"

42 + > ### Note

43 + >

44 + > Please not that if your account is a trial account you will only be able to send notifications to the number you signed up with.

39 -# Signup for free trial and select a SMS capable Twilio Number

40 -# To get your Account SID and Token, go to https://www.twilio.com/console

41 -# Place your sid, token and number below.

42 -# Then just set the recipients' phone numbers.

43 -# The trial account is only allowed to use the number specified when set up.

46 +You can then have different recipients per **role**, by editing `DEFAULT_RECIPIENT_TWILIO` with the recipient's number you want, in the following entries at the bottom of the same file:

45 -# Without an account sid and token, Netdata cannot send Twilio text messages.

48 +```conf

49 +role_recipients_twilio[sysadmin]="+15555555555"

50 +role_recipients_twilio[domainadmin]="+15555555556"

51 +role_recipients_twilio[dba]="+15555555557"

52 +role_recipients_twilio[webmaster]="+15555555558"

53 +role_recipients_twilio[proxyadmin]="+15555555559"

54 +role_recipients_twilio[sitemgr]="+15555555550"

55 +```

56 +

57 +An example of a working configuration would be:

58 +

59 +```conf

60 +#------------------------------------------------------------------------------

61 +# Twilio (twilio.com) SMS options

62 +

63 +SEND_TWILIO="YES"

64 TWILIO_ACCOUNT_SID="xxxxxxxxx"

65 TWILIO_ACCOUNT_TOKEN="xxxxxxxxxx"

66 TWILIO_NUMBER="xxxxxxxxxxx"

67 DEFAULT_RECIPIENT_TWILIO="+15555555555"

68 ```

70 +## Test the notification method