| 1 | # Manage Alert Notification Silencing Rules |
| 2 | |
| 3 | From the Cloud interface, you can manage your space's Alert notification silencing rules settings as well as allow users to define their personal ones. |
| 4 | |
| 5 | ## Rule Hierarchy |
| 6 | |
| 7 | ```mermaid |
| 8 | flowchart TD |
| 9 | Space("Space Level") --> Room("Room Level") |
| 10 | Room --> Node("Node Level") |
| 11 | Node --> Alert("Alert Level") |
| 12 | |
| 13 | Space --> SpaceRules("Space Silencing Rules<br/>Affects All Users") |
| 14 | Room --> RoomRules("Room-Specific Rules<br/>Target Specific Teams") |
| 15 | Node --> NodeRules("Node-Specific Rules<br/>Individual Servers") |
| 16 | Alert --> AlertRules("Alert-Specific Rules<br/>Granular Control") |
| 17 | |
| 18 | %% Style definitions |
| 19 | classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 20 | classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 21 | classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 22 | classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 23 | |
| 24 | %% Apply styles |
| 25 | class Space alert |
| 26 | class Room,Node complete |
| 27 | class Alert database |
| 28 | class SpaceRules,RoomRules,NodeRules,AlertRules neutral |
| 29 | ``` |
| 30 | |
| 31 | ## Decision Flowchart |
| 32 | |
| 33 | ```mermaid |
| 34 | flowchart TD |
| 35 | Start("Need to Silence Alerts?") --> Scope("Who Should Be Affected?") |
| 36 | |
| 37 | Scope -->|"All Users"| SpaceRule("Create Space Rule<br/>Admin/Manager Required") |
| 38 | Scope -->|"Just Me"| PersonalRule("Create Personal Rule<br/>Any Role Except Billing") |
| 39 | |
| 40 | SpaceRule --> Target("What to Target?") |
| 41 | PersonalRule --> Target |
| 42 | |
| 43 | Target -->|"All Infrastructure"| AllNodes("All Rooms + All Nodes") |
| 44 | Target -->|"Specific Team"| SpecificRoom("Target Specific Room") |
| 45 | Target -->|"Individual Server"| SpecificNode("Target Specific Node") |
| 46 | Target -->|"Alert Type"| SpecificAlert("Target Alert Context/Name") |
| 47 | |
| 48 | %% Style definitions |
| 49 | classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 50 | classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 51 | classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 52 | classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:18px |
| 53 | |
| 54 | %% Apply styles |
| 55 | class Start alert |
| 56 | class Scope,Target database |
| 57 | class SpaceRule complete |
| 58 | class PersonalRule complete |
| 59 | class AllNodes,SpecificRoom,SpecificNode,SpecificAlert neutral |
| 60 | ``` |
| 61 | |
| 62 | ## Prerequisites |
| 63 | |
| 64 | <details> |
| 65 | <summary><strong>For Space Alert Notification Silencing Rules</strong></summary><br/> |
| 66 | |
| 67 | To manage space-level silencing rules, you need: |
| 68 | |
| 69 | - A Netdata Cloud account |
| 70 | - Access to Space as **administrator** or **manager** (**troubleshooters** can only view space rules) |
| 71 | |
| 72 | </details> |
| 73 | |
| 74 | <details> |
| 75 | <summary><strong>For Personal Alert Notification Silencing Rules</strong></summary><br/> |
| 76 | |
| 77 | To manage your personal silencing rules, you need: |
| 78 | |
| 79 | - A Netdata Cloud account |
| 80 | - Access to Space with any role except **billing** |
| 81 | |
| 82 | </details> |
| 83 | |
| 84 | :::note |
| 85 | |
| 86 | You can only add rules if your space is on a [paid plan](/docs/netdata-cloud/view-plan-and-billing.md). |
| 87 | |
| 88 | ::: |
| 89 | |
| 90 | ## Quick Access from Dashboard |
| 91 | |
| 92 | You can also create silencing rules directly from the Alerts tab or Nodes tab: |
| 93 | |
| 94 | - **From Alerts tab**: Use the "Actions" column to create new silencing rules for specific alerts |
| 95 | - **From Nodes tab**: Add alert silencing rules directly from any node row |
| 96 | |
| 97 | ## Steps to Configure Silencing Rules |
| 98 | |
| 99 | 1. Click **Space Settings** (⚙️) on the left sidebar below the spaces list |
| 100 | 2. Click the **Alert & Notification** tab on the left-hand side |
| 101 | 3. Click the **Notification Silencing Rules** tab |
| 102 | |
| 103 | You will see configured Alert notification silencing rules for the space (if you aren't an **observer**) and yourself. |
| 104 | |
| 105 | ## Rule Configuration |
| 106 | |
| 107 | ### Available Actions |
| 108 | |
| 109 | | Action | Description | |
| 110 | |-------------------------|-----------------------------------------------------------------------------------| |
| 111 | | **Add New Rule** | Create silencing rules for "All users" (administrators/managers only) or "Myself" | |
| 112 | | **Edit Existing Rule** | Modify name, scope, criteria, and timing | |
| 113 | | **Enable/Disable Rule** | Use toggle to activate or deactivate rules | |
| 114 | | **Delete Rule** | Remove silencing rules using the trash icon | |
| 115 | |
| 116 | ### Configuration Criteria |
| 117 | |
| 118 | <details> |
| 119 | <summary><strong>Node Criteria</strong></summary><br/> |
| 120 | |
| 121 | **Rooms**: Which Rooms will this apply to? |
| 122 | |
| 123 | **Nodes**: What specific Nodes? |
| 124 | |
| 125 | **Host Labels**: Does it apply to host labels key-value pairs? |
| 126 | </details> |
| 127 | |
| 128 | <details> |
| 129 | <summary><strong>Alert Criteria</strong></summary><br/> |
| 130 | |
| 131 | **Alert Name**: Which alert name is being targeted? |
| 132 | |
| 133 | **Alert Context**: What alert context? |
| 134 | |
| 135 | **Alert Role**: Will it apply to a specific alert role? |
| 136 | </details> |
| 137 | |
| 138 | <details> |
| 139 | <summary><strong>Timing Options</strong></summary><br/> |
| 140 | |
| 141 | **Immediate**: From now until turned off or until specific duration (start and end date automatically set). |
| 142 | |
| 143 | **Scheduled**: Specify start and end time when the rule becomes active and inactive (time set according to your browser local timezone). |
| 144 | </details> |
| 145 | |
| 146 | ## Step-by-Step Wizards for Common Use Cases |
| 147 | |
| 148 | <details> |
| 149 | <summary><strong>Maintenance Window for All Infrastructure</strong></summary><br/> |
| 150 | |
| 151 | **Use Case**: Complete infrastructure maintenance affecting all users |
| 152 | |
| 153 | **Configuration Steps**: |
| 154 | |
| 155 | 1. Choose "All users" (requires admin/manager role) |
| 156 | 2. Set name: "Infrastructure Maintenance [Date]" |
| 157 | 3. **Node Criteria**: |
| 158 | - Rooms: All Rooms |
| 159 | - Nodes: * |
| 160 | - Host Labels: * |
| 161 | 4. **Alert Criteria**: |
| 162 | - Alert Name: * |
| 163 | - Alert Context: * |
| 164 | - Alert Role: * |
| 165 | 5. **Timing**: Scheduled with maintenance window start/end times |
| 166 | |
| 167 | **Validation Checklist**: |
| 168 | |
| 169 | - Admin/Manager permissions confirmed. |
| 170 | - All rooms and nodes are targeted (*). |
| 171 | - Maintenance window times are set correctly. |
| 172 | - Rule name includes a date for easy reference. |
| 173 | |
| 174 | </details> |
| 175 | |
| 176 | <details> |
| 177 | <summary><strong>Team-Specific Alert Silencing</strong></summary><br/> |
| 178 | |
| 179 | **Use Case**: Database team doesn't want notifications for their managed servers. |
| 180 | |
| 181 | **Configuration Steps**: |
| 182 | |
| 183 | 1. Choose "All users" (for team-wide effect) |
| 184 | 2. Set name: "DB Team - PostgreSQL Servers" |
| 185 | 3. **Node Criteria**: |
| 186 | - Rooms: PostgreSQL Servers |
| 187 | - Nodes: * |
| 188 | - Host Labels: * |
| 189 | 4. **Alert Criteria**: |
| 190 | - Alert Name: * |
| 191 | - Alert Context: * |
| 192 | - Alert Role: * |
| 193 | 5. **Timing**: Immediate (ongoing) |
| 194 | |
| 195 | **Validation Checklist**: |
| 196 | |
| 197 | - Correct room selected (not "All Rooms") |
| 198 | - Team members have access to the specified room |
| 199 | - Rule name clearly identifies team and scope |
| 200 | |
| 201 | </details> |
| 202 | |
| 203 | <details> |
| 204 | <summary><strong>Single Node Maintenance</strong></summary><br/> |
| 205 | |
| 206 | **Use Case**: Specific server undergoing maintenance |
| 207 | |
| 208 | **Configuration Steps**: |
| 209 | |
| 210 | 1. Choose "All users" or "Myself" based on impact |
| 211 | 2. Set name: "Node Maintenance - [NodeName]" |
| 212 | 3. **Node Criteria**: |
| 213 | - Rooms: All Rooms |
| 214 | - Nodes: [specific node name] |
| 215 | - Host Labels: * |
| 216 | 4. **Alert Criteria**: |
| 217 | - Alert Name: * |
| 218 | - Alert Context: * |
| 219 | - Alert Role: * |
| 220 | 5. **Timing**: Scheduled with the maintenance window |
| 221 | |
| 222 | **Validation Checklist**: |
| 223 | |
| 224 | - Exact node name specified correctly |
| 225 | - Maintenance window times confirmed |
| 226 | - Other team members are notified if using "All users" |
| 227 | |
| 228 | </details> |
| 229 | |
| 230 | <details> |
| 231 | <summary><strong>Load Testing Alert Suppression</strong></summary><br/> |
| 232 | |
| 233 | **Use Case**: Planned stress testing that will trigger CPU alerts |
| 234 | |
| 235 | **Configuration Steps**: |
| 236 | |
| 237 | 1. Choose the appropriate scope ("All users" or "Myself") |
| 238 | 2. Set the name: "Load Testing - CPU Alerts" |
| 239 | 3. **Node Criteria**: |
| 240 | - Rooms: [testing environment rooms] |
| 241 | - Nodes: * (or specific test nodes) |
| 242 | - Host Labels: environment:testing (if applicable) |
| 243 | 4. **Alert Criteria**: |
| 244 | - Alert Name: * |
| 245 | - Alert Context: system.cpu |
| 246 | - Alert Role: * |
| 247 | 5. **Timing**: Scheduled for the testing period |
| 248 | |
| 249 | :::tip |
| 250 | |
| 251 | **Validation Checklist**: |
| 252 | |
| 253 | - Correct alert context specified (system.cpu) |
| 254 | - Testing environment properly targeted |
| 255 | - Testing timeframe accurately set |
| 256 | - Production systems excluded |
| 257 | |
| 258 | ::: |
| 259 | |
| 260 | </details> |
| 261 | |
| 262 | ## Rule Validation Checklist |
| 263 | |
| 264 | :::tip |
| 265 | |
| 266 | Before activating any silencing rule, verify: |
| 267 | |
| 268 | | Category | Validation Item | Description | |
| 269 | |--------------------------|----------------------------------|--------------------------------------------------| |
| 270 | | **Basic Configuration** | Rule name is descriptive | Include purpose/date for easy reference | |
| 271 | | | Correct scope selected | Choose "All users" vs "Myself" appropriately | |
| 272 | | | Proper permissions for scope | Admin/manager required for "All users" | |
| 273 | | **Target Validation** | Room selection matches scope | Ensure intended rooms are targeted | |
| 274 | | | Node specification is accurate | Use * for all, specific names for targeted nodes | |
| 275 | | | Host labels correctly formatted | Use key:value pairs format | |
| 276 | | **Alert Criteria** | Alert name/context matches | Verify targeting intended alerts | |
| 277 | | | Alert role properly specified | Use role-based alerting correctly | |
| 278 | | | Wildcard (*) used appropriately | Apply broad targeting where needed | |
| 279 | | **Timing Configuration** | Immediate vs Scheduled selection | Choose appropriate timing method | |
| 280 | | | Start/end times set correctly | Consider browser timezone | |
| 281 | | | Duration appropriate | Match planned activity timeframe | |
| 282 | | **Impact Assessment** | Stakeholders notified | Inform team of silencing rule activation | |
| 283 | | | Alternative monitoring in place | Ensure backup monitoring if needed | |
| 284 | | | Rule deactivation planned | Schedule rule removal after maintenance | |
| 285 | |
| 286 | ::: |
| 287 | |
| 288 | ## Common Silencing Scenarios |
| 289 | |
| 290 | | Scenario | Configuration | Use Case | |
| 291 | |---------------------------------|---------------------------------------------------|---------------------------------------------------------| |
| 292 | | **Infrastructure Maintenance** | All Rooms, All Nodes (*) | Complete infrastructure-wide maintenance window | |
| 293 | | **Team-Specific Silencing** | Specific Room (e.g., PostgreSQL Servers) | Team doesn't want notifications for their managed nodes | |
| 294 | | **Single Node Maintenance** | Specific node (e.g., child1) | Node undergoing maintenance | |
| 295 | | **Environment-Based Silencing** | Host label: environment:production | Maintenance on production environment nodes | |
| 296 | | **Third-Party Service Issues** | Specific alert name | External service maintenance affecting monitoring | |
| 297 | | **Load Testing** | Alert context: system.cpu | Planned stress testing on CPU resources | |
| 298 | | **Role-Based Control** | Alert role: webmaster | Silence all alerts for specific role | |
| 299 | | **Granular Alert Control** | Specific alert + specific node | Targeted silencing for known issues | |
| 300 | | **Storage Maintenance** | Alert: disk_space_usage, Instance: specific mount | Maintenance on specific storage volumes | |
| 301 | |
| 302 | ## Detailed Examples Reference |
| 303 | |
| 304 | | Rule Name | Rooms | Nodes | Host Label | Alert Name | Alert Context | Alert Instance | Alert Role | Description | |
| 305 | |----------------------------------|--------------------|--------|------------------------|------------------------------------------------|---------------|------------------------|------------|-------------------------------------------------------------------------------------| |
| 306 | | Space silencing | All Rooms | * | * | * | * | * | * | Silences entire space, all nodes, all users. Infrastructure-wide maintenance window | |
| 307 | | DB Servers Rooms | PostgreSQL Servers | * | * | * | * | * | * | Silences nodes in PostgreSQL Servers Room only, not All Nodes Room | |
| 308 | | Node child1 | All Rooms | child1 | * | * | * | * | * | Silences all Alert state transitions for node child1 in all Rooms | |
| 309 | | Production nodes | All Rooms | * | environment:production | * | * | * | * | Silences Alert state transitions for nodes with environment:production label | |
| 310 | | Third party maintenance | All Rooms | * | * | httpcheck_posthog_netdata_cloud.request_status | * | * | * | Silences specific Alert during third-party partner maintenance | |
| 311 | | Intended stress usage on CPU | All Rooms | * | * | * | system.cpu | * | * | Silences specific Alerts across all nodes and their CPU cores | |
| 312 | | Silence role webmaster | All Rooms | * | * | * | * | * | webmaster | Silences all Alerts configured with role webmaster | |
| 313 | | Silence Alert on node | All Rooms | child1 | * | httpcheck_posthog_netdata_cloud.request_status | * | * | * | Silences specific Alert on child1 node | |
| 314 | | Disk Space Alerts on mount point | All Rooms | * | * | disk_space_usage | disk.space | disk_space_opt_baddisk | * | Silences specific Alert instance on all nodes /opt/baddisk | |