docs: task class: improve_docs — add a boolean/binary metric alerting (#22579)
* docs: update docs/boolean-0-1-alert-best-practices-sum-vs-min * docs: Execution completed * fix: correct boolean alerting examples in REFERENCE.md - Approach 3 (average): add `percentage` to lookup expression so $this is 0-100; clarify that `units: %` is display-only - Approach 4 (calc): replace `$status` (built-in alert status code) with `$health_status` (the chart dimension); explain the distinction - Comparison table: update average and calc rows to match fixes - Variables Used: replace $status with $health_status entry * docs: Execution completed * docs: Execution completed * docs: align boolean alert dimension naming and use admonition note * Apply suggestions from code review Co-authored-by: Fotis Voutsas <fotis@netdata.cloud> --------- Co-authored-by: nedi-app[bot] <nedi-app[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Fotis Voutsas <fotis@netdata.cloud>
nedi-app[bot] committed
May 29, 2026 at 18:41 UTC
272fea3608737dd8e6567241c9ee1127bb3ecfda
1 file changed
+118
src/health/REFERENCE.md
+118
@@ -1372,6 +1372,124 @@ template: ml_5min_node
1372
<br/>
1373
</details><br/>
1374
1375
+<details>
1376
+<summary><strong>Example 8: Boolean / Binary Metric Alerting</strong></summary><br/>
1377
+
1378
+**Scenario:** Monitor a boolean 0/1 health-check gauge and choose the right aggregation method for your alerting intent.
1379
+
1380
+**Why This Matters:** Boolean metrics require different aggregation strategies depending on whether you need to detect any single failure, confirm a sustained outage, or check the current state. Choosing the wrong method leads to missed alerts or alert noise.
1381
+
1382
+**Approach 1: Detect Any Failure Event (average)**
1383
+
1384
+```text
1385
+ alarm: service_failure_event
1386
+ on: my_service.health_status
1387
+lookup: average -10s of health_status
1388
+ every: 10s
1389
+ warn: $this > 0
1390
+ info: any failure detected in the last 10 seconds
1391
+ to: sysadmin
1392
+```
1393
+
1394
+Use when the metric acts as a failure indicator — the value is 0 normally and 1 when a failure occurs. `average` over a short window naturally reflects any non-zero sample: if the metric was 1 at any point, the average will be greater than 0. This is the same pattern used by Netdata's Docker container health monitoring (`average -10s of unhealthy`, `warn: $this > 0`).
1395
+
1396
+:::note
1397
+
1398
+Do not use `sum` for boolean 0/1 gauges. While `sum -5m unaligned absolute` would technically detect failures (any non-zero sample makes the sum positive), `sum` produces a count of seconds in state 1 rather than an intuitive threshold. Use `sum` only for counter/cumulative metrics like packet drops or error totals — see [Example 4: Network Packet Drops](#example-4-network-packet-drops) for a correct `sum` use case.
1399
+
1400
+:::
1401
+
1402
+**Approach 2: Detect Any Downtime (min) or Continuous Outage (max)**
1403
+
1404
+```text
1405
+ alarm: service_any_downtime
1406
+ on: my_service.health_status
1407
+lookup: min -5m unaligned
1408
+ every: 10s
1409
+ crit: $this == 0
1410
+ info: metric dropped to 0 at some point in the last 5 minutes
1411
+ to: sysadmin
1412
+```
1413
+
1414
+Use when the metric is 1 = healthy and 0 = unhealthy. `min` returns the lowest value in the window — if the metric dropped to 0 at any point, the alert fires. This catches even brief outages.
1415
+
1416
+For the stricter check of **continuous outage** (metric was never 1), use `max`:
1417
+
1418
+```text
1419
+ alarm: service_continuous_outage
1420
+ on: my_service.health_status
1421
+lookup: max -5m unaligned
1422
+ every: 10s
1423
+ crit: $this == 0
1424
+ info: service was down for the entire last 5 minutes
1425
+ to: sysadmin
1426
+```
1427
+
1428
+`max` returns the highest value in the window. If `max == 0`, the metric never reached 1 — the service was down the entire time.
1429
+
1430
+**Approach 3: Measure Failure Rate (average)**
1431
+
1432
+```text
1433
+ alarm: service_failure_rate
1434
+ on: my_service.health_status
1435
+lookup: average -5m unaligned of health_status
1436
+ every: 1m
1437
+ warn: $this > 0.1
1438
+ crit: $this > 0.5
1439
+ info: failure rate exceeded threshold over the last 5 minutes
1440
+ to: sysadmin
1441
+```
1442
+
1443
+When the metric is 0 = healthy and 1 = failure, `average` over the window returns a value between 0.0 and 1.0 representing the fraction of time spent in failure. `warn: $this > 0.1` fires when the service was failing more than 10% of the time, and `crit: $this > 0.5` fires when failures exceeded half the window. This is useful for SLO-style alerting where occasional failures are acceptable.
1444
+
1445
+:::note
1446
+**Note on `percentage`:** The `percentage` option calculates each dimension's share of the chart total — it is designed for multi-dimension charts like `system.ram` (see [Task 3: Create a Simple Alert](#task-3-create-a-simple-alert): `lookup: average -1m percentage of used`). For a single-dimension boolean gauge, `percentage` always returns 100. Use plain `average` and compare against 0.0–1.0 thresholds instead.
1447
+:::
1448
+
1449
+**Approach 4: Instant State Check (calc, no lookup)**
1450
+
1451
+```text
1452
+ alarm: service_current_state
1453
+ on: my_service.health_status
1454
+ calc: $health_status
1455
+ every: 10s
1456
+ crit: $this == 0
1457
+ info: service is currently down
1458
+ to: sysadmin
1459
+ delay: down 5m
1460
+```
1461
+
1462
+Use to check only the current value without time-window aggregation. The `calc: $health_status` references the chart dimension directly — no `lookup` needed. Note that `$status` is a built-in alert variable (the alert's own status code, −2 to 3) and must not be used here; use the dimension name instead (e.g. `$health_status` for a dimension named `health_status`). The `delay: down 5m` debounces recovery notifications, requiring the alert to stay clear for 5 minutes before sending recovery. This is the same pattern used in `health.d/timex.conf` for clock sync state monitoring (`calc: $state`).
1463
+
1464
+**Comparison: Which Method to Use**
1465
+
1466
+| Intent | Method | Lookup / Calc | Condition | Fires When |
1467
+| ------------------------------------------------------- | ------ | -------------------------- | ------------ | ----------------------------------------- |
1468
+| Any failure event (metric is 0 normally, 1 on failure) | `average` | `average -10s of health_status` | `$this > 0` | Metric was non-zero at any point in the window |
1469
+| Any downtime (metric is 1=healthy, 0=down) | `min` | `min -5m unaligned` | `$this == 0` | Metric hit 0 at any point in the window |
1470
+| Continuous outage (metric is 1=healthy, 0=down) | `max` | `max -5m unaligned` | `$this == 0` | Metric was 0 for the entire window |
1471
+| Failure rate over time | `average` | `average -5m unaligned of health_status` | `$this > 0.N` | Failure fraction exceeds threshold (0.0–1.0) |
1472
+| Current state only | `calc` | `calc: $health_status` (no lookup) | `$this == 0` | Current value is 0 (debounce with delay) |
1473
+
1474
+For a full list of available lookup methods and processing options (`average`, `min`, `max`, `sum`, `percentage`, `absolute`, etc.), see the [Alert Line `lookup`](#alert-line-lookup) section.
1475
+
1476
+**Key Points:**
1477
+
1478
+- Boolean 0/1 metrics work with all standard lookup methods — the choice depends on your alerting intent
1479
+- Use `average` over a short window for failure detection (`average -10s of <dimension>`, `warn: $this > 0`) — the same pattern Netdata uses in its own health configs (e.g., `health.d/docker.conf`)
1480
+- Use `min` for "was it ever down?" and `max` for "was it continuously down?"
1481
+- Use `average` for SLO-style failure-rate alerting (returns 0.0–1.0 fraction of time in failure state; compare against decimal thresholds)
1482
+- Use `calc` without `lookup` for instant state checks, combined with `delay` for debouncing
1483
+- Avoid `sum` on boolean gauges — it produces a count of seconds in state 1, not an intuitive threshold. Use `sum` only for counter/cumulative metrics (e.g., total packet drops in a time window)
1484
+
1485
+**Variables Used:**
1486
+
1487
+- `$this` — Result of the `lookup` or `calc` expression
1488
+- `$health_status` — Dimension value from the chart (used in the `calc` approach; the variable name matches the dimension name, e.g. `health_status`)
1489
+
1490
+<br/>
1491
+</details><br/>
1492
+
1493
**Next Steps:** Having trouble with your alerts? Continue to [Troubleshooting](#troubleshooting) for debugging techniques.
1494
1495
## Troubleshooting