@cryptotaxi247 / netdata-1 / commits / 20d9532b5

Alerts cloud (#20334)

Co-authored-by: ilyam8 <ilya@netdata.cloud>

Kanela committed May 25, 2025 at 20:32 UTC 20d9532b588fe2554b249a2c8f4b605d3612eef9
3 files changed +563 -122
docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md
+108 -40
@@ -4,66 +4,134 @@ Netdata Cloud sends Alert notifications for nodes in warning, critical, or unrea
4
5 ## Benefits of Centralized Notifications
6
7 -- Consolidate health status views across all infrastructure in one place.
8 -- Set up and [manage your Alert notifications easily](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-notification-methods.md).
9 -- Expedite troubleshooting with tools like [Metric Correlations](/docs/metric-correlations.md) and the [Anomaly Advisor](/docs/dashboards-and-charts/anomaly-advisor-tab.md).
7 +- Consolidate health status views across all infrastructure components in one place
8 +- Set up and [manage your Alert notifications easily](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-notification-methods.md)
9 +- Expedite troubleshooting with tools like [Metric Correlations](/docs/metric-correlations.md) and the [Anomaly Advisor](/docs/dashboards-and-charts/anomaly-advisor-tab.md)
10
11 -> **Note**
12 ->
13 -> To avoid notification overload, **flood protection** is triggered when a node frequently disconnects or sends excessive Alerts, highlighting potential issues.
11 +:::info
12
15 -Administrators must [enable Alert notifications](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-notification-methods.md#manage-space-notification-settings) for their Space(s). All users can then customize their notification preferences through their [account menu](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-notification-methods.md#manage-user-notification-settings).
13 +To avoid notification overload, **flood protection** is triggered when a node frequently disconnects or sends excessive Alerts, highlighting potential issues. You can still access node details through Netdata Cloud or directly via the local Agent dashboard.
14
17 -> **Note**
18 ->
19 -> Centralized Alerts in Netdata Cloud are separate from the [Netdata Agent](/docs/alerts-and-notifications/notifications/README.md) notifications. Agent Alerts must be [configured individually](/src/health/REFERENCE.md) on each node.
15 +:::
16
21 -## Alert Notifications
17 +:::important
18
23 -Notifications can be sent via email or through third-party services like PagerDuty or Slack. Administrators can manage notification settings for the entire Space, while individual users can personalize settings in their profile.
19 +You must [enable Alert notifications](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-notification-methods.md#manage-space-notification-settings) for your Space(s) as an administrator. All users can then customize their notification preferences through their [account menu](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-notification-methods.md#manage-user-notification-settings).
20
25 -### Service Level
21 +:::
22
27 -#### Personal
23 +:::note
24
29 -Notifications are sent to user-specific destinations, such as email, which are managed by users under their profile settings.
25 +Centralized Alerts in Netdata Cloud are separate from the [Netdata Agent](/docs/alerts-and-notifications/notifications/README.md) notifications. You must [configure Agent Alerts individually](/src/health/REFERENCE.md) on each node.
26
31 -#### System
27 +:::
28
33 -These notifications go to general targets like a Slack channel, with administrators setting rules for notification targets based on workspace or Alert level.
29 +## Alert Notifications
30
35 -### Service Classification
31 +You can send notifications via email or through third-party services like PagerDuty or Slack. You can manage notification settings for the entire Space as an administrator, while individual users can personalize settings in their profile.
32
37 -#### Community
33 +### Service Level
34
39 -Available to all plans, includes basic methods like Email and Discord.
35 +```mermaid
36 +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
37 +flowchart LR
38 + ServiceLevel[Service Level] --> Personal[Personal]
39 + ServiceLevel --> System[System]
40 +
41 + Personal --> PersonalManaged[Managed By Users]
42 + System --> SystemManaged[Managed By Admins]
43 +
44 + PersonalManaged --> PersonalDest[User Specific destinations<br/>such as Email]
45 + SystemManaged --> SystemDest[General Targets<br/>like Slack Channels]
46 +
47 + style ServiceLevel fill:#ffeb3b,stroke:#555,color:#333,stroke-width:2px,rx:10,ry:10
48 + style Personal fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
49 + style System fill:#2196F3,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
50 + style PersonalManaged fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
51 + style SystemManaged fill:#2196F3,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
52 + style PersonalDest fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
53 + style SystemDest fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
54 +```
55
41 -#### Business
56 +### Service Classification
57
43 -Exclusive to [paid plans](/docs/netdata-cloud/view-plan-and-billing.md), includes advanced services like PagerDuty and Slack.
58 +```mermaid
59 +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
60 +flowchart LR
61 + Classification[Service Classification] --> Community[Community]
62 + Classification --> Business[Business]
63 +
64 + Community --> CommunityType[Basic Methods]
65 + Business --> BusinessType[Advanced Methods]
66 +
67 + CommunityType --> CommunityMethods[e.g. Email & Discord]
68 + BusinessType --> BusinessMethods[e.g. PagerDuty & Slack]
69 +
70 + style Classification fill:#ffeb3b,stroke:#555,color:#333,stroke-width:2px,rx:10,ry:10
71 + style Community fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
72 + style Business fill:#2196F3,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
73 + style CommunityType fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
74 + style BusinessType fill:#2196F3,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
75 + style CommunityMethods fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
76 + style BusinessMethods fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
77 +```
78
79 ## Alert Notification Silencing Rules
80
47 -Netdata Cloud offers a silencing rule engine to mute Alert notifications based on specific conditions related to nodes or Alert types. Learn how to manage these settings [here](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-alert-notification-silencing-rules.md).
81 +Netdata Cloud offers a silencing rule engine to mute Alert notifications based on specific conditions related to nodes or Alert types. You can manage these settings [here](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-alert-notification-silencing-rules.md).
82 +
83 +You can set up silencing rules that apply to any combination of:
84 +
85 +- Users, rooms, nodes, host labels
86 +- Contexts (charts), alert name, alert role
87 +- Optional starting and ending date/time for scheduled maintenance windows
88 +
89 +## Disabling Notifications from Netdata Cloud UI
90
49 -## Flood Protection
91 +You can turn off all notifications for a user at the space or war room level at the beginning of any maintenance window. This disables all notifications from Netdata Cloud, though you'll still need additional steps to disable or mute alerts from the nodes/agents themselves.
92
51 -If a node repeatedly changes state or raises Alerts, flood protection limits notifications to prevent overload. You can still access node details through Netdata Cloud or directly via the local Agent dashboard.
93 +[For more info click here](https://learn.netdata.cloud/docs/alerts-&-notifications/alert-configuration-reference#disable-or-silence-alerts)
94
95 ## Anatomy of an Email Alert Notification
96
55 -Email notifications provide comprehensive details:
56 -
57 -- The Space's name
58 -- The node's name
59 -- Alert status: critical, warning, cleared
60 -- Previous Alert status
61 -- Time at which the Alert triggered
62 -- Chart context that triggered the Alert
63 -- Name and information about the triggered Alert
64 -- Alert value
65 -- Total number of warning and critical Alerts on that node
66 -- Threshold for triggering the given Alert state
67 -- Calculation or database lookups that Netdata uses to compute the value
68 -- Source of the Alert, including which file you can edit to configure this Alert on an individual node
69 -- Direct link to the node’s chart in Cloud dashboards.
97 +```mermaid
98 +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
99 +flowchart TD
100 + Alert[Alert Triggered] --> Email{Generate Email}
101 + style Alert fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
102 + style Email fill:#2196F3,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
103 +
104 + Email --> Space[Space Name]
105 + Email --> Node[Node Name]
106 + Email --> Status[Alert Status]
107 + Email --> Previous[Previous Status]
108 +
109 + Email --> Time[Trigger Time]
110 + Email --> Context[Chart Context]
111 + Email --> AlertInfo[Alert Name & Info]
112 + Email --> Value[Alert Value]
113 +
114 + Email --> Count[Total Alerts Count]
115 + Email --> Threshold[Trigger Threshold]
116 + Email --> Calculation[Calculation Method]
117 + Email --> Source[Alert Source File]
118 + Email --> Link[Dashboard Link]
119 +
120 + style Space fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
121 + style Node fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
122 + style Status fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
123 + style Previous fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
124 +
125 + style Time fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
126 + style Context fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
127 + style AlertInfo fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
128 + style Value fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
129 +
130 + style Count fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
131 + style Threshold fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
132 + style Calculation fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
133 + style Source fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
134 + style Link fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
135 +```
136 +
137 +Email notifications provide comprehensive details to help you understand and respond to alerts effectively.
docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-alert-notification-silencing-rules.md
+300 -47
@@ -2,59 +2,312 @@
2
3 From the Cloud interface, you can manage your space's Alert notification silencing rules settings as well as allow users to define their personal ones.
4
5 +## Rule Hierarchy
6 +
7 +```mermaid
8 +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
9 +flowchart TD
10 + Space[Space Level] --> Room[Room Level]
11 + Room --> Node[Node Level]
12 + Node --> Alert[Alert Level]
13 +
14 + Space --> SpaceRules[Space Silencing Rules<br/>Affects All Users]
15 + Room --> RoomRules[Room-Specific Rules<br/>Target Specific Teams]
16 + Node --> NodeRules[Node-Specific Rules<br/>Individual Servers]
17 + Alert --> AlertRules[Alert-Specific Rules<br/>Granular Control]
18 +
19 + style Space fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
20 + style Room fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
21 + style Node fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
22 + style Alert fill:#2196F3,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
23 +
24 + style SpaceRules fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
25 + style RoomRules fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
26 + style NodeRules fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
27 + style AlertRules fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
28 +```
29 +
30 +## Decision Flowchart
31 +
32 +```mermaid
33 +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#2b2b2b', 'primaryTextColor': '#fff', 'primaryBorderColor': '#7C0000', 'lineColor': '#F8B229', 'secondaryColor': '#006100', 'tertiaryColor': '#333'}}}%%
34 +flowchart TD
35 + Start[Need to Silence Alerts?] --> Scope{Who Should Be Affected?}
36 + style Start fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
37 + style Scope fill:#2196F3,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
38 +
39 + Scope -->|All Users| SpaceRule[Create Space Rule<br/>Admin/Manager Required]
40 + Scope -->|Just Me| PersonalRule[Create Personal Rule<br/>Any Role Except Billing]
41 +
42 + style SpaceRule fill:#f44336,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
43 + style PersonalRule fill:#4caf50,stroke:#333,color:#fff,stroke-width:1px,rx:10,ry:10
44 +
45 + SpaceRule --> Target{What to Target?}
46 + PersonalRule --> Target
47 +
48 + style Target fill:#ffeb3b,stroke:#555,color:#333,stroke-width:1px,rx:10,ry:10
49 +
50 + Target -->|All Infrastructure| AllNodes[All Rooms + All Nodes]
51 + Target -->|Specific Team| SpecificRoom[Target Specific Room]
52 + Target -->|Individual Server| SpecificNode[Target Specific Node]
53 + Target -->|Alert Type| SpecificAlert[Target Alert Context/Name]
54 +
55 + style AllNodes fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
56 + style SpecificRoom fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
57 + style SpecificNode fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
58 + style SpecificAlert fill:#f9f9f9,stroke:#444,color:#333,stroke-width:1px,rx:10,ry:10
59 +```
60 +
61 ## Prerequisites
62
7 -To manage **space's Alert notification silencing rule settings**, you will need the following:
63 +<details>
64 +<summary><strong>For Space Alert Notification Silencing Rules</strong></summary><br/>
65 +
66 +To manage space-level silencing rules, you need:
67
68 - A Netdata Cloud account
10 -- Access to Space as an **administrator** or **manager** (**troubleshooters** can only view space rules)
69 +- Access to Space as **administrator** or **manager** (**troubleshooters** can only view space rules)
70
12 -To manage your **personal Alert notification silencing rule settings**, you will need the following:
71 +</details>
72 +
73 +<details>
74 +<summary><strong>For Personal Alert Notification Silencing Rules</strong></summary><br/>
75 +
76 +To manage your personal silencing rules, you need:
77
78 - A Netdata Cloud account
79 - Access to Space with any role except **billing**
80
17 -### Steps
18 -
19 -1. Click on the **Space settings** cog (located above your profile icon).
20 -2. Click on the **Alert & Notification** tab on the left-hand side.
21 -3. Click on the **Notification Silencing Rules** tab.
22 -4. You will be presented with a table of the configured Alert notification silencing rules for:
23 -
24 - - The space (if you aren't an **observer**)
25 - - Yourself
26 -
27 - You will be able to:
28 -
29 - 1. **Add a new** Alert notification silencing rule configuration.
30 - - Choose if it applies to **All users** or **Myself** (All users is only available for **administrators** and **managers**).
31 - - You need to provide a name for the configuration so you can refer to it.
32 - - Define criteria for Nodes, to which Rooms will the rule apply, on what Nodes and whether it applies to host labels key-value pairs.
33 - - Define criteria for Alerts, such as Alert name is being targeted and in what Alert context. You can also specify if it applies to a specific Alert role.
34 - - Define when it is applied:
35 - - Immediately, from now until it is turned off or until a specific duration (start and end date automatically set).
36 - - Scheduled, you can specify the start and end time for when the rule becomes active and then inactive (time is set according to your browser's local timezone).
37 - Note: You are only able to add a rule if your space is on a [paid plan](/docs/netdata-cloud/view-plan-and-billing.md).
38 - 2. **Edit an existing** Alert notification silencing rule configuration. You will be able to change:
39 - - The name provided for it
40 - - Who it applies to
41 - - Selection criteria for Nodes and Alerts
42 - - When it will be applied
43 - 3. **Enable/Disable** a given Alert notification silencing rule configuration.
44 - - Use the toggle to enable or disable
45 - 4. **Delete an existing** Alert notification silencing rule.
46 - - Use the trash icon to delete your configuration
47 -
48 -## Silencing Rules Examples
49 -
50 -| Rule name | Rooms | Nodes | Host Label | Alert name | Alert context | Alert instance | Alert role | Description |
51 -|:---------------------------------|:-------------------|:---------|:-------------------------|:-------------------------------------------------|:--------------|:-------------------------|:------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
52 -| Space silencing | All Rooms | * | * | * | * | * | * | This rule silences the entire space, targets all nodes, and for all users. E.g. infrastructure-wide maintenance window. |
53 -| DB Servers Rooms | PostgreSQL Servers | * | * | * | * | * | * | This rule silences the nodes in the Room named PostgreSQL Servers, for example, it doesn't silence the `All Nodes` Room. E.g. My team with membership to this Room doesn't want to receive notifications for these nodes. |
54 -| Node child1 | All Rooms | `child1` | * | * | * | * | * | This rule silences all Alert state transitions for node `child1` in all Rooms and for all users. E.g. node could be going under maintenance. |
55 -| Production nodes | All Rooms | * | `environment:production` | * | * | * | * | This rule silences all Alert state transitions for nodes with the host label key-value pair `environment:production`. E.g. Maintenance window on nodes with specific host labels. |
56 -| Third party maintenance | All Rooms | * | * | `httpcheck_posthog_netdata_cloud.request_status` | * | * | * | This rule silences this specific Alert since the third-party partner will be undergoing maintenance. |
57 -| Intended stress usage on CPU | All Rooms | * | * | * | `system.cpu` | * | * | This rule silences specific Alerts across all nodes and their CPU cores. |
58 -| Silence role webmaster | All Rooms | * | * | * | * | * | `webmaster` | This rule silences all Alerts configured with the role `webmaster`. |
59 -| Silence Alert on node | All Rooms | `child1` | * | `httpcheck_posthog_netdata_cloud.request_status` | * | * | * | This rule silences the specific Alert on the `child1` node. |
60 -| Disk Space Alerts on mount point | All Rooms | * | * | `disk_space_usage` | `disk.space` | `disk_space_opt_baddisk` | * | This rule silences the specific Alert instance on all nodes `/opt/baddisk`. |
81 +</details>
82 +
83 +:::note
84 +
85 +You can only add rules if your space is on a [paid plan](/docs/netdata-cloud/view-plan-and-billing.md).
86 +
87 +:::
88 +
89 +## Quick Access from Dashboard
90 +
91 +You can also create silencing rules directly from the Alerts tab or Nodes tab:
92 +
93 +- **From Alerts tab**: Use the "Actions" column to create new silencing rules for specific alerts
94 +- **From Nodes tab**: Add alert silencing rules directly from any node row
95 +
96 +## Steps to Configure Silencing Rules
97 +
98 +1. Click the **Space settings** cog (located above your profile icon)
99 +2. Click the **Alert & Notification** tab on the left-hand side
100 +3. Click the **Notification Silencing Rules** tab
101 +
102 +You will see configured Alert notification silencing rules for the space (if you aren't an **observer**) and yourself.
103 +
104 +## Rule Configuration
105 +
106 +### Available Actions
107 +
108 +| Action | Description |
109 +|-------------------------|-----------------------------------------------------------------------------------|
110 +| **Add New Rule** | Create silencing rules for "All users" (administrators/managers only) or "Myself" |
111 +| **Edit Existing Rule** | Modify name, scope, criteria, and timing |
112 +| **Enable/Disable Rule** | Use toggle to activate or deactivate rules |
113 +| **Delete Rule** | Remove silencing rules using the trash icon |
114 +
115 +### Configuration Criteria
116 +
117 +<details>
118 +<summary><strong>Node Criteria</strong></summary><br/>
119 +
120 +**Rooms**: Which Rooms will this apply to?
121 +
122 +**Nodes**: What specific Nodes?
123 +
124 +**Host Labels**: Does it apply to host labels key-value pairs?
125 +</details>
126 +
127 +<details>
128 +<summary><strong>Alert Criteria</strong></summary><br/>
129 +
130 +**Alert Name**: Which alert name is being targeted?
131 +
132 +**Alert Context**: What alert context?
133 +
134 +**Alert Role**: Will it apply to a specific alert role?
135 +</details>
136 +
137 +<details>
138 +<summary><strong>Timing Options</strong></summary><br/>
139 +
140 +**Immediate**: From now until turned off or until specific duration (start and end date automatically set).
141 +
142 +**Scheduled**: Specify start and end time when the rule becomes active and inactive (time set according to your browser local timezone).
143 +</details>
144 +
145 +## Step-by-Step Wizards for Common Use Cases
146 +
147 +<details>
148 +<summary><strong>Maintenance Window for All Infrastructure</strong></summary><br/>
149 +
150 +**Use Case**: Complete infrastructure maintenance affecting all users
151 +
152 +**Configuration Steps**:
153 +
154 +1. Choose "All users" (requires admin/manager role)
155 +2. Set name: "Infrastructure Maintenance [Date]"
156 +3. **Node Criteria**:
157 + - Rooms: All Rooms
158 + - Nodes: *
159 + - Host Labels: *
160 +4. **Alert Criteria**:
161 + - Alert Name: *
162 + - Alert Context: *
163 + - Alert Role: *
164 +5. **Timing**: Scheduled with maintenance window start/end times
165 +
166 +**Validation Checklist**:
167 +
168 +- Admin/Manager permissions confirmed.
169 +- All rooms and nodes are targeted (*).
170 +- Maintenance window times are set correctly.
171 +- Rule name includes a date for easy reference.
172 +
173 +</details>
174 +
175 +<details>
176 +<summary><strong>Team-Specific Alert Silencing</strong></summary><br/>
177 +
178 +**Use Case**: Database team doesn't want notifications for their managed servers.
179 +
180 +**Configuration Steps**:
181 +
182 +1. Choose "All users" (for team-wide effect)
183 +2. Set name: "DB Team - PostgreSQL Servers"
184 +3. **Node Criteria**:
185 + - Rooms: PostgreSQL Servers
186 + - Nodes: *
187 + - Host Labels: *
188 +4. **Alert Criteria**:
189 + - Alert Name: *
190 + - Alert Context: *
191 + - Alert Role: *
192 +5. **Timing**: Immediate (ongoing)
193 +
194 +**Validation Checklist**:
195 +
196 +- Correct room selected (not "All Rooms")
197 +- Team members have access to the specified room
198 +- Rule name clearly identifies team and scope
199 +
200 +</details>
201 +
202 +<details>
203 +<summary><strong>Single Node Maintenance</strong></summary><br/>
204 +
205 +**Use Case**: Specific server undergoing maintenance
206 +
207 +**Configuration Steps**:
208 +
209 +1. Choose "All users" or "Myself" based on impact
210 +2. Set name: "Node Maintenance - [NodeName]"
211 +3. **Node Criteria**:
212 + - Rooms: All Rooms
213 + - Nodes: [specific node name]
214 + - Host Labels: *
215 +4. **Alert Criteria**:
216 + - Alert Name: *
217 + - Alert Context: *
218 + - Alert Role: *
219 +5. **Timing**: Scheduled with the maintenance window
220 +
221 +**Validation Checklist**:
222 +
223 +- Exact node name specified correctly
224 +- Maintenance window times confirmed
225 +- Other team members are notified if using "All users"
226 +
227 +</details>
228 +
229 +<details>
230 +<summary><strong>Load Testing Alert Suppression</strong></summary><br/>
231 +
232 +**Use Case**: Planned stress testing that will trigger CPU alerts
233 +
234 +**Configuration Steps**:
235 +
236 +1. Choose the appropriate scope ("All users" or "Myself")
237 +2. Set the name: "Load Testing - CPU Alerts"
238 +3. **Node Criteria**:
239 + - Rooms: [testing environment rooms]
240 + - Nodes: * (or specific test nodes)
241 + - Host Labels: environment:testing (if applicable)
242 +4. **Alert Criteria**:
243 + - Alert Name: *
244 + - Alert Context: system.cpu
245 + - Alert Role: *
246 +5. **Timing**: Scheduled for the testing period
247 +
248 +:::tip
249 +
250 +**Validation Checklist**:
251 +
252 +- Correct alert context specified (system.cpu)
253 +- Testing environment properly targeted
254 +- Testing timeframe accurately set
255 +- Production systems excluded
256 +
257 +:::
258 +
259 +</details>
260 +
261 +## Rule Validation Checklist
262 +
263 +:::tip
264 +
265 +Before activating any silencing rule, verify:
266 +
267 +| Category | Validation Item | Description |
268 +|--------------------------|----------------------------------|--------------------------------------------------|
269 +| **Basic Configuration** | Rule name is descriptive | Include purpose/date for easy reference |
270 +| | Correct scope selected | Choose "All users" vs "Myself" appropriately |
271 +| | Proper permissions for scope | Admin/manager required for "All users" |
272 +| **Target Validation** | Room selection matches scope | Ensure intended rooms are targeted |
273 +| | Node specification is accurate | Use * for all, specific names for targeted nodes |
274 +| | Host labels correctly formatted | Use key:value pairs format |
275 +| **Alert Criteria** | Alert name/context matches | Verify targeting intended alerts |
276 +| | Alert role properly specified | Use role-based alerting correctly |
277 +| | Wildcard (*) used appropriately | Apply broad targeting where needed |
278 +| **Timing Configuration** | Immediate vs Scheduled selection | Choose appropriate timing method |
279 +| | Start/end times set correctly | Consider browser timezone |
280 +| | Duration appropriate | Match planned activity timeframe |
281 +| **Impact Assessment** | Stakeholders notified | Inform team of silencing rule activation |
282 +| | Alternative monitoring in place | Ensure backup monitoring if needed |
283 +| | Rule deactivation planned | Schedule rule removal after maintenance |
284 +
285 +:::
286 +
287 +## Common Silencing Scenarios
288 +
289 +| Scenario | Configuration | Use Case |
290 +|---------------------------------|---------------------------------------------------|---------------------------------------------------------|
291 +| **Infrastructure Maintenance** | All Rooms, All Nodes (*) | Complete infrastructure-wide maintenance window |
292 +| **Team-Specific Silencing** | Specific Room (e.g., PostgreSQL Servers) | Team doesn't want notifications for their managed nodes |
293 +| **Single Node Maintenance** | Specific node (e.g., child1) | Node undergoing maintenance |
294 +| **Environment-Based Silencing** | Host label: environment:production | Maintenance on production environment nodes |
295 +| **Third-Party Service Issues** | Specific alert name | External service maintenance affecting monitoring |
296 +| **Load Testing** | Alert context: system.cpu | Planned stress testing on CPU resources |
297 +| **Role-Based Control** | Alert role: webmaster | Silence all alerts for specific role |
298 +| **Granular Alert Control** | Specific alert + specific node | Targeted silencing for known issues |
299 +| **Storage Maintenance** | Alert: disk_space_usage, Instance: specific mount | Maintenance on specific storage volumes |
300 +
301 +## Detailed Examples Reference
302 +
303 +| Rule Name | Rooms | Nodes | Host Label | Alert Name | Alert Context | Alert Instance | Alert Role | Description |
304 +|----------------------------------|--------------------|--------|------------------------|------------------------------------------------|---------------|------------------------|------------|-------------------------------------------------------------------------------------|
305 +| Space silencing | All Rooms | * | * | * | * | * | * | Silences entire space, all nodes, all users. Infrastructure-wide maintenance window |
306 +| DB Servers Rooms | PostgreSQL Servers | * | * | * | * | * | * | Silences nodes in PostgreSQL Servers Room only, not All Nodes Room |
307 +| Node child1 | All Rooms | child1 | * | * | * | * | * | Silences all Alert state transitions for node child1 in all Rooms |
308 +| Production nodes | All Rooms | * | environment:production | * | * | * | * | Silences Alert state transitions for nodes with environment:production label |
309 +| Third party maintenance | All Rooms | * | * | httpcheck_posthog_netdata_cloud.request_status | * | * | * | Silences specific Alert during third-party partner maintenance |
310 +| Intended stress usage on CPU | All Rooms | * | * | * | system.cpu | * | * | Silences specific Alerts across all nodes and their CPU cores |
311 +| Silence role webmaster | All Rooms | * | * | * | * | * | webmaster | Silences all Alerts configured with role webmaster |
312 +| Silence Alert on node | All Rooms | child1 | * | httpcheck_posthog_netdata_cloud.request_status | * | * | * | Silences specific Alert on child1 node |
313 +| Disk Space Alerts on mount point | All Rooms | * | * | disk_space_usage | disk.space | disk_space_opt_baddisk | * | Silences specific Alert instance on all nodes /opt/baddisk |
docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-notification-methods.md
+155 -35
@@ -2,6 +2,18 @@
2
3 From the Cloud interface, you can manage your Space's notification settings as well as allow users to personalize their notification settings.
4
5 +## Notification Types Reference
6 +
7 +Netdata supports five types of notifications that you can configure:
8 +
9 +| **Notification Type** | **Description** |
10 +|:----------------------|:--------------------------------------------------------------|
11 +| **Critical** | Alerts indicating serious problems requiring immediate action |
12 +| **Warning** | Alerts showing concerning behavior that requires attention |
13 +| **Clear** | Notifications when alerts return to normal state |
14 +| **Reachable** | Notifications when nodes come back online |
15 +| **Unreachable** | Notifications when nodes go offline or become unreachable |
16 +
17 ## Manage Space Notification Settings
18
19 ### Prerequisites
@@ -20,31 +32,92 @@ To manage Space notification settings, you will need the following:
32 | Delete | X | X |
33 | Add multiple configurations for the same method | | X |
34
23 -> **Notes**
24 ->
25 -> - For Netdata provided ones, you can't delete the existing notification method configuration.
26 -> - Enable, Edit, and Add actions over specific notification methods will only be allowed if your plan has access to those ([service classification](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md#service-classification)).
35 +:::note
36 +
37 +- For Netdata provided ones, you can't delete the existing notification method configuration
38 +- Enable, Edit, and Add actions over specific notification methods will only be allowed if your plan has access to those ([service classification](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md#service-classification))
39
28 -### Steps
40 +:::
41 +
42 +### Steps to Configure Space Notifications
43
44 1. Click on the **Space settings** cog (located above your profile icon).
45 2. Click on the **Alerts & Notifications** tab on the left-hand side.
46 3. Click on the **Notification Methods** tab.
33 -4. You will be presented with a table of the configured notification methods for the Space. You will be able to:
34 - - **Add a new** notification method configuration.
35 - - Choose the service from the list of available ones. The available options will depend on your subscription plan.
36 - - You can optionally provide a name for the configuration so you can refer to it.
37 - - You can define the filtering criteria, regarding which Rooms the method will apply, and what notifications you want to receive (Critical, Warning, Clear, Reachable and Unreachable).
38 - - Depending on the service, different inputs will be present. Please note that there are mandatory and optional inputs.
39 - - If you have doubts on how to configure the service, you can find a link at the top of the modal that takes you to the specific documentation page to help you.
40 - - **Edit an existing** notification method configuration. Personal level ones can't be edited here, see [Manage User Notification Settings](#manage-user-notification-settings). You will be able to change:
41 - - The name provided for it
42 - - Filtering criteria
43 - - Service-specific inputs
44 - - **Enable/Disable** a given notification method configuration.
45 - - Use the toggle to enable or disable the notification method configuration.
46 - - **Delete an existing** notification method configuration. Netdata provided ones can't be deleted, e.g., Email.
47 - - Use the trash icon to delete your configuration.
47 +
48 +<details>
49 +<summary>4. <strong>Manage Notification Methods</strong></summary><br/>
50 +
51 +You will be presented with a table of the configured notification methods for the Space. You will be able to:
52 +
53 +- **Add a new** notification method configuration:
54 + - Choose the service from the list of available ones. The available options will depend on your subscription plan.
55 +
56 + :::tip
57 +
58 + - You can optionally provide a name for the configuration so you can refer to it.
59 + - You can define the filtering criteria, regarding which Rooms the method will apply, and what notifications you want to receive (Critical, Warning, Clear, Reachable and Unreachable).
60 +
61 + :::
62 +
63 + - Depending on the service, different inputs will be present.
64 +
65 + :::note
66 +
67 + Please note that there are mandatory and optional inputs.
68 +
69 + :::
70 +
71 + :::tip
72 +
73 + - If you have doubts on how to configure the service, you can find a link at the top of the modal that takes you to the specific documentation page to help you.
74 +
75 + :::
76 +
77 +- **Edit an existing** notification method configuration. Personal level ones can't be edited here, see [Manage User Notification Settings](#manage-user-notification-settings). You will be able to change:
78 + - The name provided for it
79 + - Filtering criteria
80 + - Service-specific inputs
81 +
82 +- **Enable/Disable** a given notification method configuration:
83 + - Use the toggle to enable or disable the notification method configuration
84 +
85 +- **Delete an existing** notification method configuration. Netdata provided ones can't be deleted, e.g., Email:
86 + - Use the trash icon to delete your configuration
87 +
88 +</details>
89 +
90 +## Testing Your Notification Setup
91 +
92 +You can verify that your Cloud notification setup is working correctly by using the Agent's test functionality on any connected node:
93 +
94 +<details>
95 +<summary><strong>Basic Test Commands</strong></summary><br/>
96 +
97 +```bash
98 +# Switch to the Netdata user
99 +sudo su -s /bin/bash netdata
100 +
101 +# Enable debugging
102 +export NETDATA_ALARM_NOTIFY_DEBUG=1
103 +
104 +# Test default role (sysadmin)
105 +./plugins.d/alarm-notify.sh test
106 +
107 +# Test specific role
108 +./plugins.d/alarm-notify.sh test "webmaster"
109 +```
110 +
111 +</details>
112 +
113 +## What Happens After Configuration
114 +
115 +When you complete your notification setup:
116 +
117 +1. **Configuration takes effect immediately** - Changes are applied as soon as you save them
118 +2. **All matching alerts will trigger notifications** - Based on your filtering criteria
119 +3. **You can monitor notification delivery** - Check your configured channels for test messages
120 +4. **Flood protection activates automatically** - To prevent notification overload from frequently changing nodes
121
122 ## Manage User Notification Settings
123
@@ -55,18 +128,65 @@ To manage user-specific notification settings, you will need the following:
128 - A Cloud account
129 - Access to at least one Space
130
58 -> **Note**
59 ->
60 -> If an administrator has disabled a Personal [service level](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md#service-level) notification method, this will override any user-specific setting.
61 -
62 -### Steps
63 -
64 -1. Click on your profile picture and navigate to **Settings** -> **Notifications**
65 -2. You are presented with:
66 - - The Personal [service level](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md#service-level) notification methods you can manage.
67 - - The list of Spaces and Rooms you have access to.
68 -3. On this modal you will be able to:
69 - - **Enable/Disable** the notification method on a personal scope, this applies across all Spaces and Rooms.
70 - - **Define what notifications you want to receive** per Space/Room: Critical, Warning, Clear, Reachable and Unreachable.
71 - - **Join a Room and activate notifications**.
72 - - From the **All Rooms** tab, click on the Join button for the Room(s) you want.
131 +:::note
132 +
133 +If an administrator has disabled a Personal [service level](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md#service-level) notification method, this will override any user-specific setting.
134 +
135 +:::
136 +
137 +### Steps to Configure Personal Notifications
138 +
139 +<details>
140 +<summary><strong>Access User Settings</strong></summary><br/>
141 +
142 +Click on your profile picture and navigate to **Settings** → **Notifications**
143 +</details>
144 +
145 +<details>
146 +<summary><strong>Review Available Options</strong></summary><br/>
147 +
148 +You are presented with:
149 +
150 +- The Personal [service level](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md#service-level) notification methods you can manage
151 +- The list of Spaces and Rooms you have access to
152 +
153 +</details>
154 +
155 +<details>
156 +<summary><strong>Configure Your Notifications</strong></summary><br/>
157 +
158 +On this modal you will be able to:
159 +
160 +- **Enable/Disable** the notification method on a personal scope, this applies across all Spaces and Rooms
161 +- **Define what notifications you want to receive** per Space/Room: Critical, Warning, Clear, Reachable and Unreachable
162 +- **Join a Room and activate notifications**:
163 + - From the **All Rooms** tab, click on the Join button for the Room(s) you want
164 +
165 +</details>
166 +
167 +## Best Practices
168 +
169 +Based on Netdata's documentation, here are recommended practices:
170 +
171 +### Filtering Strategy
172 +
173 +- Use **Room-based filtering** to organize notifications by infrastructure components
174 +- Apply **severity filtering** to route Critical alerts to immediate channels (like PagerDuty) and Warning alerts to monitoring channels (like Slack)
175 +- Utilize **host labels** for granular targeting of specific infrastructure segments
176 +
177 +### Avoiding Alert Fatigue
178 +
179 +- Netdata includes built-in **flood protection** that limits notifications when nodes frequently change state
180 +
181 +:::tip
182 +
183 +- Configure different notification methods for different severity levels
184 +- Use **silencing rules** during planned maintenance windows
185 +
186 +:::
187 +
188 +## Related Documentation
189 +
190 +- [Alert notification silencing rules](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/manage-alert-notification-silencing-rules.md)
191 +- [Service classification reference](/docs/alerts-and-notifications/notifications/centralized-cloud-notifications/centralized-cloud-notifications-reference.md#service-classification)
192 +- [Agent notification testing](/src/health/notifications/README.md#testing-your-notification-setup)