initial draft for the silencing docs (#15112)
* initial draft for the silencing docs * minor fixes upon local review
Hugo Valente committed
May 30, 2023 at 18:52 UTC
d3dc461f0ee504dc41cd7f0370690363aa7694ef
5 files changed
+109
-11
docs/cloud/alerts-notifications/manage-alert-notification-silencing-rules.md
new
+58
@@ -0,0 +1,58 @@
1
+# Manage alert notification silencing rules
2
+
3
+From the Cloud interface, you can manage your space's alert notification silencing rules settings as well as allow users to define their personal ones.
4
+
5
+## Prerequisites
6
+
7
+To manage **space's alert notification silencing rule settings**, you will need the following:
8
+
9
+- A Netdata Cloud account
10
+- Access to the space as an **administrator** or **manager** (**troubleshooters** can only view space rules)
11
+
12
+
13
+To manage your **personal alert notification silencing rule settings**, you will need the following:
14
+
15
+- A Netdata Cloud account
16
+- Access to the space with any roles except **billing**
17
+
18
+### Steps
19
+
20
+1. Click on the **Space settings** cog (located above your profile icon)
21
+1. Click on the **Alert & Notification** tab on the left hand-side
22
+1. Click on the **Notification Silencing Rules** tab
23
+1. You will be presented with a table of the configured alert notification silencing rules for:
24
+ * the space (if aren't an **observer**)
25
+ * yourself
26
+
27
+ You will be able to:
28
+ 1. **Add a new** alert notification silencing rule configuration.
29
+ - Choose if it applies to **All users** or **Myself** (All users is only available for **administrators** and **managers**)
30
+ - You need to provide a name for the configuration so you can easily refer to it
31
+ - Define criteria for Nodes: To which Rooms will this apply? What Nodes? Does it apply to host labels key-value pairs?
32
+ - Define criteria for Alerts: Which alert name is being targeted? What alert context? Will it apply to a specific alert role?
33
+ - Define when it will be applied:
34
+ - Immediately, from now till until it is turned off or until a specific duration (start and end date automatically set)
35
+ - Scheduled, you specify the start and end time for when the rule becomes active and then inactive (time is set according to your browser local timezone)
36
+ Note: You are only able to add a rule if your space is on a [paid plan](https://github.com/netdata/netdata/edit/master/docs/cloud/manage/plans.md).
37
+ 1. **Edit an existing** alert notification silencing rule configurations. You will be able to change:
38
+ - The name provided for it
39
+ - Who it applies to
40
+ - Selection criteria for Nodes and Alert
41
+ - When it will be applied
42
+ 1. **Enable/Disable** a given alert notification silencing rule configuration.
43
+ - Use the toggle to enable or disable
44
+ 1. **Delete an existing** alert notification silencing rule.
45
+ - Use the trash icon to delete your configuration
46
+
47
+## Silencing rules examples
48
+
49
+| Rule name | War Rooms | Nodes | Host Label | Alert name | Alert context | Alert role | Description |
50
+| :-- | :-- | :-- | :-- | :-- | :-- | :-- | :--|
51
+| Space silencing | All Rooms | * | * | * | * | * | This rule silences the entire space, targets all nodes and for all users. E.g. infrastructure wide maintenance window. |
52
+| DB Servers Rooms | PostgreSQL Servers | * | * | * | * | * | This rules silences the nodes in the room named PostgreSQL Servers, for example it doesn't silence the `All Nodes` room. E.g. My team with membership to this room doesn't want to receive notifications for these nodes. |
53
+| Node child1 | All Rooms | `child1` | * | * | * | * | This rule silences all alert state transitions for node `child1` on all rooms and for all users. E.g. node could be going under maintenance. |
54
+| Production nodes | All Rooms | * | `environment:production` | * | * | * | This rule silences all alert state transitions for nodes with the host label key-value pair `environment:production`. E.g. Maintenance window on nodes with specific host labels. |
55
+| Third party maintenance | All Rooms | * | * | `httpcheck_posthog_netdata_cloud.request_status` | * | * | This rule silences this specific alert since third party partner will be undergoing maintenance. |
56
+| Intended stress usage on CPU | All Rooms | * | * | * | `system.cpu` | * | This rule silences specific alerts across all nodes and their CPU cores. |
57
+| Silence role webmaster | All Rooms | * | * | * | * | `webmaster` | This rule silences all alerts configured with the role `webmaster`. |
58
+| Silence alert on node | All Rooms | `child1` | * | `httpcheck_posthog_netdata_cloud.request_status` | * | * | * | This rule silences the specific alert on the `child1` node. |
docs/cloud/alerts-notifications/manage-notification-methods.md
+3
-2
@@ -27,7 +27,8 @@ Notes:
27
### Steps
28
29
1. Click on the **Space settings** cog (located above your profile icon)
30
-1. Click on the **Notification** tab
30
+1. Click on the **Alerts & Notification** tab on the left hand-side
31
+1. Click on the **Notification Methods** tab
32
1. You will be presented with a table of the configured notification methods for the space. You will be able to:
33
1. **Add a new** notification method configuration.
34
- Choose the service from the list of the available ones, you'll may see a list of unavailable options if your plan doesn't allow some of them (you will see on the
@@ -42,7 +43,7 @@ Notes:
43
- Service specific inputs
44
1. **Enable/Disable** a given notification method configuration.
45
- Use the toggle to enable or disable the notification method configuration
45
- 1. **Delete an existing** notification method configuartion. Netdata provided ones can't be deleted, e.g. Email
46
+ 1. **Delete an existing** notification method configuration. Netdata provided ones can't be deleted, e.g. Email
47
- Use the trash icon to delete your configuration
48
49
## Manage user notification settings
docs/cloud/alerts-notifications/notifications.md
+34
-8
@@ -31,7 +31,7 @@ or add new alert that you see in Netdata Cloud, and receive via centralized aler
31
32
</Callout>
33
34
-### Alert notifications
34
+## Alert notifications
35
36
Netdata Cloud can send centralized alert notifications to your team whenever a node enters a warning, critical, or unreachable state. By enabling notifications,
37
you ensure no alert, on any node in your infrastructure, goes unnoticed by you or your team.
@@ -51,9 +51,9 @@ All users in a Space can personalize their notifications settings, for Personal
51
> ⚠️ Netdata Cloud supports different notification methods and their availability will depend on the plan you are at.
52
> For more details check [Service classification](#service-classification) or [netdata.cloud/pricing](https://www.netdata.cloud/pricing).
53
54
-#### Service level
54
+### Service level
55
56
-##### Personal
56
+#### Personal
57
58
The notifications methods classified as **Personal** are what we consider generic, meaning that these can't have specific rules for them set by the administrators.
59
@@ -63,7 +63,7 @@ manage what specific configurations they want for the Space / Room(s) and the de
63
64
One example of such a notification method is the E-mail.
65
66
-##### System
66
+#### System
67
68
For **System** notification methods, the destination of the channel will be a target that usually isn't specific to a single user, e.g. slack channel.
69
@@ -72,23 +72,49 @@ different targets depending on Rooms or Notification level settings.
72
73
Some examples of such notification methods are: Webhook, PagerDuty, Slack.
74
75
-#### Service classification
75
+### Service classification
76
77
-##### Community
77
+#### Community
78
79
Notification methods classified as Community can be used by everyone independent on the plan your space is at.
80
These are: Email and discord
81
82
-##### Pro
82
+#### Pro
83
84
Notification methods classified as Pro are only available for **Pro** and **Business** plans
85
These are: webhook
86
87
-##### Business
87
+#### Business
88
89
Notification methods classified as Business are only available for **Business** plans
90
These are: PagerDuty, Slack, Opsgenie
91
92
+## Silencing Alert notifications
93
+
94
+Netdata Cloud provides you a Silencing Rule engine which allows you to mute alert notifications. This muting action is specific to alert state transition notifications, it doesn't include node unreachable state transitions.
95
+
96
+The Silencing Rule engine is flexible and allows you to enter silence rules for the two main entities involved on alert notifications and can be set using different attributes. The main entities you can enter are **Nodes** and **Alerts** which can be used in combination or isolation to target specific needs - see some examples [here](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/manage-alert-notification-silencing-rules.md#silencing-rules-examples).
97
+
98
+### Scope definition for Nodes
99
+* **Space:** silencing the space, selecting `All Rooms`, silences all alert state transitions from any node claimed to the space.
100
+* **War Room:** silencing a specific room will silence all alert state transitions from any node in that room. Please note if the node belongs to
101
+another room which isn't silenced it can trigger alert notifications to the users with membership to that other room.
102
+* **Node:** silencing a specific node can be done for the entire space, selecting `All Rooms`, or for specific war room(s). The main difference is
103
+if the node should be silenced for the entire space or just for specific rooms (when specific rooms are selected only users with membership to that room won't receive notifications).
104
+
105
+### Scope definition for Alerts
106
+* **Alert name:** silencing a specific alert name silences all alert state transitions for that specific alert.
107
+* **Alert context:** silencing a specific alert context will silence all alert state transitions for alerts targeting that chart context, for more details check [alert configuration docs](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md#alarm-line-on).
108
+* **Alert role:** silencing a specific alert role will silence all the alert state transitions for alerts that are configured to be specific role recipients, for more details check [alert configuration docs](https://github.com/netdata/netdata/blob/master/health/REFERENCE.md#alarm-line-to).
109
+
110
+Beside the above two main entities there are another two important settings that you can define on a silencing rule:
111
+* Who does the rule affect? **All user** in the space or **Myself**
112
+* When does is to apply? **Immediately** or on a **Schedule** (when setting immediately you can set duration)
113
+
114
+For further help on setting alert notification silencing rules go to [Manage Alert Notification Silencing Rules](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/manage-alert-notification-silencing-rules.md).
115
+
116
+> ⚠️ This feature is only available for [Netdata paid plans](https://github.com/netdata/netdata/edit/master/docs/cloud/manage/plans.md).
117
+
118
## Flood protection
119
120
If a node has too many state changes like firing too many alerts or going from reachable to unreachable, Netdata Cloud
docs/cloud/manage/plans.md
+7
-1
@@ -101,7 +101,13 @@ The plan on your space will determine what type of notifications methods will be
101
* **Pro** - Email, Discord and webhook
102
* **Business** - Unlimited, this includes Slack, PagerDuty, Opsgenie etc.
103
104
-For mode details check the documentation under [Alert Notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.md).
104
+For mode details check the documentation under [Alert Notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.md#alert-notifications).
105
+
106
+##### Alert notification silencing rules
107
+
108
+The plan on your space will determine if you are able to add alert notification silencing rules since this feature will only be available for paid plans: **Pro** or **Business**.
109
+
110
+For mode details check the documentation under [Alert Notifications](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.md#silencing-alert-notifications).
111
112
### Related Topics
113
docs/cloud/manage/role-based-access.md
+7
@@ -84,6 +84,13 @@ In more detail, you can find on the following tables which functionalities are a
84
| Edit configuration | :heavy_check_mark: | - | - | - | - | - | Some exceptions apply depending on [service level](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/manage-notification-methods.md#available-actions-per-notification-methods-based-on-service-level) |
85
| Delete configuration | :heavy_check_mark: | - | - | - | - | - | |
86
| Edit personal level notification settings | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | [Manage user notification settings](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/manage-notification-methods.md#manage-user-notification-settings) |
87
+| See space alert notification silencing rules | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | - | - | - | |
88
+| Add new space alert notification silencing rule | :heavy_check_mark: | :heavy_check_mark: | - | - | - | - | |
89
+| Enable/Disable space alert notification silencing rule | :heavy_check_mark: | :heavy_check_mark: | - | - | - | - | |
90
+| Edit space alert notification silencing rule | :heavy_check_mark: | :heavy_check_mark: | - | - | - | - | |
91
+| Delete space alert notification silencing rule | :heavy_check_mark: | :heavy_check_mark: | - | - | - | - | |
92
+| See, add, edit or delete personal level alert notification silencing rule | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | - | - | |
93
+
94
95
Notes:
96
* Enable, Edit and Add actions over specific notification methods will only be allowed if your plan has access to those ([service classification](https://github.com/netdata/netdata/blob/master/docs/cloud/alerts-notifications/notifications.md#service-classification))