ec915e762 · cryptotaxi247/netdata-1

@@ -15,7 +15,7 @@ files.

15

16 ## Reload health configuration

17

18 -You do not need to restart the Netdata Agent between changes to health configuration files, such as specific health entities. Instead, you can use `netdatacli` and the `reload-health` option to prevent gaps in metrics collection.

18 +You don’t need to restart the Netdata Agent between changes to health configuration files, such as specific health entities. Instead, you can use `netdatacli` and the `reload-health` option to prevent gaps in metrics collection.

19

20 ```bash

21 sudo netdatacli reload-health

@@ -36,7 +36,7 @@ You can configure the Agent's health watchdog service by editing files in two lo

36 [daemon configuration](/src/daemon/config/README.md#health-section-options) for a table of

37 all the available settings, their default values, and what they control.

38

39 -- The individual `.conf` files in `health.d/`. These health entity files are organized by the type of metric they are

39 +- The individual `.conf` files in `health.d/`. These health entity files are organized by the type of metric they’re

40 performing calculations on or their associated collector. You should edit these files using the `edit-config`

41 script. For example: `sudo ./edit-config health.d/cpu.conf`.

42

@@ -112,8 +112,8 @@ This action requires that you [reload Netdata's health configuration](#reload-he

112

113 ### Temporarily disable alerts at runtime

114

115 -When you need to frequently disable all or some alerts from triggering during certain times (for instance

116 -when running backups) you can use the

115 +When you need to frequently disable all or some alerts from triggering during certain times (for instance,

116 +when running backups), you can use the

117 [health management API](/src/web/api/health/README.md).

118 The API allows you to issue commands to control the health engine's behavior without changing configuration,

119 or restarting the Agent.

@@ -199,7 +199,7 @@ see it live on the local dashboard or Netdata Cloud.

199 ## Health entity reference

200

201 The following reference contains information about the syntax and options of _health entities_, which Netdata attaches

202 -to charts in order to trigger alerts.

202 +to charts to trigger alerts.

203

204 ### Entity types

205

@@ -225,7 +225,7 @@ Netdata parses the following lines. Beneath the table is an in-depth explanation

225 - A few lines use space-separated lists to define how the entity behaves. You can use `*` as a wildcard or prefix with

226 `!` for a negative match. Order is important, too! See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for

227 more examples.

228 -- Lines terminated by a `\` are spliced together with the next line. The backslash is removed and the following line is

228 +- Lines terminated by a `\` are spliced together with the next line. The backslash is removed, and the following line is

229 joined with the current one. No space is inserted, so you may split a line anywhere, even in the middle of a word.

230 This comes in handy if your `info` line consists of several sentences.

231

@@ -270,7 +270,7 @@ template: NAME

270 ```

271

272 `NAME` can be any alpha character, with `.` (period) and `_` (underscore) as the only allowed symbols, but the names

273 -cannot be `chart name`, `dimension name`, `family name`, or `chart variables names`.

273 +can’t be `chart name`, `dimension name`, `family name`, or `chart variables names`.

274

275 #### Alert line `on`

276

@@ -390,17 +390,17 @@ lookup: METHOD(GROUPING OPTIONS) AFTER [at BEFORE] [every DURATION] [OPTIONS] [o

390

391 The full [database query API](/src/web/api/queries/README.md) is supported. In short:

392

393 -- `METHOD` is one of the available [grouping methods](/src/web/api/queries/README.md#grouping-methods) such as `average`, `min`, `max` etc.

393 +- `METHOD` is one of the available [grouping methods](/src/web/api/queries/README.md#grouping-methods) such as `average`, `min`, `max` etc.

394 This is required.

395

396 - `GROUPING OPTIONS` are optional and can have the form `CONDITION VALUE`, where `CONDITION` is `!=`, `=`, `<=`, `<`, `>`, `>=` and `VALUE` is a number. The `CONDITION` and `VALUE` are required for `countif`, while `VALUE` is used by `percentile`, `trimmed_mean` and `trimmed_median`.

397

398 - `AFTER` is a relative number of seconds, but it also accepts a single letter for changing

399 the units, like `-1s` = 1 second in the past, `-1m` = 1 minute in the past, `-1h` = 1 hour

400 - in the past, `-1d` = 1 day in the past. You need a negative number (i.e. how far in the past

400 + in the past, `-1d` = 1 day in the past. You need a negative number (i.e., how far in the past

401 to look for the value). **This is required**.

402

403 -- `at BEFORE` is by default 0 and is not required. Using this you can define the end of the

403 +- `at BEFORE` is by default 0 and is not required. Using this, you can define the end of the

404 lookup. So data will be evaluated between `AFTER` and `BEFORE`.

405

406 - `every DURATION` sets the updated frequency of the lookup (supports single letter units as

@@ -427,7 +427,7 @@ The full [database query API](/src/web/api/queries/README.md) is supported. In s

427 for dimensions.

428

429 The result of the lookup will be available as `$this` and `$NAME` in expressions.

430 -The timestamps of the timeframe evaluated by the database lookup is available as variables

430 +The timestamps of the timeframe evaluated by the database lookup are available as variables

431 `$after` and `$before` (both are unix timestamps).

432

433 #### Alert line `calc`

@@ -476,10 +476,10 @@ red: NUMBER

476

477 #### Alert lines `warn` and `crit`

478

479 -Define the expression that triggers either a warning or critical alert. These are optional, and should evaluate to

479 +Define the expression that triggers either a warning or critical alert. These are optional and should evaluate to

480 either true or false (or zero/non-zero).

481

482 -The format uses Netdata's [expressions syntax](#expressions).

482 +The format uses Netdata's [expression syntax](#expressions).

483

484 ```yaml

485 warn: EXPRESSION

@@ -516,7 +516,7 @@ including custom hooks.

516 #### Alert line `delay`

517

518 This is used to provide optional hysteresis settings for the notifications, to defend against notification floods. These

519 -settings do not affect the actual alert - only the time the `exec` script is executed.

519 +settings don’t affect the actual alert - only the time the `exec` script is executed.

520

521 Format:

522

@@ -525,13 +525,13 @@ delay: [[[up U] [down D] multiplier M] max X]

525 ```

526

527 - `up U` defines the delay to be applied to a notification for an alert that raised its status

528 - (i.e. CLEAR to WARNING, CLEAR to CRITICAL, WARNING to CRITICAL). For example, `up 10s`, the

528 + (i.e., CLEAR to WARNING, CLEAR to CRITICAL, WARNING to CRITICAL). For example, `up 10s`, the

529 notification for this event will be sent 10 seconds after the actual event. This is used in

530 - hope the alert will get back to its previous state within the duration given. The default `U`

530 + the hope the alert will get back to its previous state within the duration given. The default `U`

531 is zero.

532

533 - `down D` defines the delay to be applied to a notification for an alert that moves to lower

534 - state (i.e. CRITICAL to WARNING, CRITICAL to CLEAR, WARNING to CLEAR). For example, `down 1m`

534 + state (i.e., CRITICAL to WARNING, CRITICAL to CLEAR, WARNING to CLEAR). For example, `down 1m`

535 will delay the notification by 1 minute. This is used to prevent notifications for flapping

536 alerts. The default `D` is zero.

537

@@ -539,7 +539,7 @@ delay: [[[up U] [down D] multiplier M] max X]

539 delayed. The default multiplier is `1.0`.

540

541 - `max X` defines the maximum absolute notification delay an alert may get. The default `X`

542 - is `max(U * M, D * M)` (i.e. the max duration of `U` or `D` multiplied once with `M`).

542 + is `max(U * M, D * M)` (i.e., the max duration of `U` or `D` multiplied once with `M`).

543

544 Example:

545

@@ -565,7 +565,7 @@ delay: [[[up U] [down D] multiplier M] max X]

565 Defines the interval between repeating notifications for the alerts in CRITICAL or WARNING mode. This will override the

566 default interval settings inherited from health settings in `netdata.conf`. The default settings for repeating

567 notifications are `default repeat warning = DURATION` and `default repeat critical = DURATION` which can be found in

568 -health stock configuration, when one of these interval is bigger than 0, Netdata will activate the repeat notification

568 +health stock configuration, when one of these intervals is bigger than 0, Netdata will activate the repeat notification

569 for `CRITICAL`, `CLEAR` and `WARNING` messages.

570

571 Format:

@@ -574,7 +574,7 @@ Format:

574 repeat: [off] [warning DURATION] [critical DURATION]

575 ```

576

577 -- `off`: Turns off the repeating feature for the current alert. This is effective when the default repeat settings has

577 +- `off`: Turns off the repeating feature for the current alert. This is effective when the default repeat settings have

578 been enabled in health configuration.

579 - `warning DURATION`: Defines the interval when the alert is in WARNING state. Use `0s` to turn off the repeating

580 notification for WARNING mode.

@@ -589,13 +589,13 @@ The only possible value for the `options` line is

589 options: no-clear-notification

590 ```

591

592 -For some alerts we need compare two time-frames, to detect anomalies. For example, `health.d/httpcheck.conf` has an

592 +For some alerts, we need to compare two time frames to detect anomalies. For example, `health.d/httpcheck.conf` has an

593 alert template called `web_service_slow` that compares the average http call response time over the last 3 minutes,

594 compared to the average over the last hour. It triggers a warning alert when the average of the last 3 minutes is twice

595 -the average of the last hour. In such cases, it is easy to trigger the alert, but difficult to tell when the alert is

595 +the average of the last hour. In such cases, it is easy to trigger the alert, but challenging to tell when the alert is

596 cleared. As time passes, the newest window moves into the older, so the average response time of the last hour will keep

597 increasing. Eventually, the comparison will find the averages in the two time-frames close enough to clear the alert.

598 -However, the issue was not resolved, it's just a matter of the newer data "polluting" the old. For such alerts, it's a

598 +However, the issue was not resolved; it's just a matter of the newer data "polluting" the old. For such alerts, it's a

599 good idea to tell Netdata to not clear the notification, by using the `no-clear-notification` option.

600

601 #### Alert line `host labels`

@@ -637,7 +637,7 @@ See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for mor

637

638 #### Alert line `chart labels`

639

640 -Similar to host labels, the `chart labels` key can be used to filter if an alert will load or not for a specific chart, based on

640 +Similar to host labels, the `chart labels` key can be used to filter if an alert loads or not for a specific chart, based on

641 whether these chart labels match or not.

642

643 The list of chart labels present on each chart can be obtained from <http://localhost:19999/api/v1/charts?all>

@@ -645,7 +645,7 @@ The list of chart labels present on each chart can be obtained from <http://loca

645 For example, each `disk_space` chart defines a chart label called `mount_point` with each instance of this chart having

646 a value there of which mount point it monitors.

647

648 -If you have an e.g. external disk mounted on `/mnt/disk1` and you don't wish any related disk space alerts running for

648 +If you have an e.g., external disk mounted on `/mnt/disk1` and you don't wish any related disk space alerts running for

649 it (but you do for all other mount points), you can add the following to the alert's configuration:

650

651 ```yaml

@@ -653,27 +653,27 @@ chart labels: mount_point=!/mnt/disk1 *

653 ```

654

655 The `chart labels` is a space-separated list that accepts simple patterns. If you use multiple different chart labels,

656 -then the result is an AND between them. i.e. the following:

656 +then the result is an AND between them. I.e. the following:

657

658 ```yaml

659 chart labels: mount_point=/mnt/disk1 device=sda

660 ```

661

662 Will create the alert if the `mount_point` is `/mnt/disk1` and the `device` is `sda`. Furthermore, if a chart label name

663 -is specified that does not exist in the chart, the chart won't be matched.

663 +is specified that doesn’t exist in the chart, the chart won't be matched.

664

665 See our [simple patterns docs](/src/libnetdata/simple_pattern/README.md) for more examples.

666

667 #### Alert line `summary`

668

669 The summary field contains a brief title of the alert. It is used as the subject for the notifications, and in

670 -dashboard list of alerts. An example for the `ram_available` alert is:

670 +the dashboard list of alerts. An example for the `ram_available` alert is:

671

672 ```yaml

673 summary: Available Ram

674 ```

675

676 -summary fields can contain special variables in their text that will be replaced during run-time to provide more specific

676 +Summary fields can contain special variables in their text that will be replaced during run-time to provide more specific

677 alert information. Current variables supported are:

678

679 | variable | description |

@@ -704,7 +704,7 @@ notifications and UI elements whenever the specific alert is in focus. An exampl

704 info: Percentage of estimated amount of RAM available for userspace processes, without causing swapping

705 ```

706

707 -info fields can contain special variables in their text that will be replaced during run-time to provide more specific

707 +Info fields can contain special variables in their text that will be replaced during run-time to provide more specific

708 alert information. Current variables supported are:

709

710 | variable | description |

@@ -747,7 +747,7 @@ structure that allows fast execution of them.

747 These operators are supported `+`, `-`, `*`, `/`, `<`, `==`, `<=`, `<>`, `!=`, `>`, `>=`, `&&`, `||`, `!`, `AND`, `OR`, `NOT`.

748 Boolean operators result in either `1` (true) or `0` (false).

749

750 -The conditional evaluation operator `?` is supported too. Using this operator IF-THEN-ELSE conditional statements can be

750 +The conditional evaluation operator `?` is supported too. Using this operator, IF-THEN-ELSE conditional statements can be

751 specified. The format is: `(condition) ? (true expression) : (false expression)`. So, Netdata will first evaluate the

752 `condition` and based on the result will either evaluate `true expression` or `false expression`.

753

@@ -763,43 +763,34 @@ Expressions can have variables. Variables start with `$`. Check below for more i

763 There are two special values you can use:

764

765 - `nan`, for example `$this != nan` will check if the variable `this` is available. A variable can be `nan` if the

766 - database lookup failed. All calculations (i.e. addition, multiplication, etc.) with a `nan` result in a `nan`.

766 + database lookup failed. All calculations (i.e., addition, multiplication, etc.) with a `nan` result in a `nan`.

767

768 - `inf`, for example `$this != inf` will check if `this` is not infinite. A value or variable can be set to infinite

769 - if divided by zero. All calculations (i.e. addition, multiplication, etc.) with a `inf` result in a `inf`.

769 + if divided by zero. All calculations (i.e., addition, multiplication, etc.) with a `inf` result in a `inf`.

770

771 ### Special use of the conditional operator

772

773 -A common (but not necessarily obvious) use of the conditional evaluation operator is to provide

774 -[hysteresis](https://en.wikipedia.org/wiki/Hysteresis) around the critical or warning thresholds. This usage helps to

775 -avoid bogus messages resulting from small variations in the value when it is varying regularly but staying close to the

776 -threshold value, without needing to delay sending messages at all.

773 +The conditional operator (`? :`) can create "sticky" alert thresholds that prevent alert spam when values fluctuate around a threshold. This is called [hysteresis](https://en.wikipedia.org/wiki/Hysteresis)—where the threshold to trigger an alert is different from the threshold to clear it.

774

778 -An example of such usage from the default CPU usage alerts bundled with Netdata is:

775 +**Example: CPU Usage Alert**

776

777 ```yaml

778 warn: $this > (($status >= $WARNING) ? (75) : (85))

779 crit: $this > (($status == $CRITICAL) ? (85) : (95))

780 ```

781

785 -The above say:

783 +|-------------|-------------|-----------|---------------------------------------------------------|

784 +| Warning | 85% CPU | 75% CPU | Once in warning state, requires a larger drop to clear |

785 +| Critical | 95% CPU | 85% CPU | Once in critical state, must drop back to warning level |

786

787 -- If the alert is currently a warning, then the threshold for being considered a warning is 75, otherwise it's 85.

787 +This creates a buffer zone that prevents alert flapping. For example:

788

789 -- If the alert is currently critical, then the threshold for being considered critical is 85, otherwise it's 95.

789 +- If CPU usage bounces between 80–90%, you'll get just one warning (when it first hits 85%)

790 +- The warning won't clear until the CPU drops well below the trigger point (75%)

791 +- The Same principle applies to critical alerts—they won't clear until the CPU drops significantly (85%)

792

791 -Which in turn, results in the following behavior:

792 -

793 -- While the value is rising, it will trigger a warning when it exceeds 85, and a critical alert when it exceeds 95.

794 -

795 -- While the value is falling, it will return to a warning state when it goes below 85, and a normal state when it goes

796 - below 75.

797 -

798 -- If the value is constantly varying between 80 and 90, then it will trigger a warning the first time it goes above

799 - 85, but will remain a warning until it goes below 75 (or goes above 85).

800 -

801 -- If the value is constantly varying between 90 and 100, then it will trigger a critical alert the first time it goes

802 - above 95, but will remain a critical alert goes below 85 (at which point it will return to being a warning).

793 +This approach gives you the best of both worlds: quick alerting when issues arise, but protection against notification spam when values hover near a threshold.

794

795 ## Variables

796

@@ -810,7 +801,7 @@ registry](https://registry.my-netdata.io/api/v1/alarm_variables?chart=system.cpu

801

802

803

813 -Netdata supports 3 internal indexes for variables that will be used in health monitoring.

804 +Netdata supports three internal indexes for variables that will be used in health monitoring.

805

806 <details><summary>The variables below can be used in both chart alerts and context templates.</summary>

807

@@ -845,7 +836,7 @@ context are essentially identical, with the only difference being the family tha

836 - `$this`, which is resolved to the value of the current alert.

837

838 - `$status`, which is resolved to the current status of the alert (the current = the last

848 - status, i.e. before the current database lookup and the evaluation of the `calc` line).

839 + status, i.e., before the current database lookup and the evaluation of the `calc` line).

840 This values can be compared with `$REMOVED`, `$UNINITIALIZED`, `$UNDEFINED`, `$CLEAR`,

841 `$WARNING`, `$CRITICAL`. These values are incremental, e.g. `$status > $CLEAR` works as

842 expected.

@@ -861,10 +852,10 @@ Alerts can have the following statuses:

852

853 - `UNINITIALIZED` - the alert is not initialized yet

854

864 -- `UNDEFINED` - the alert failed to be calculated (i.e. the database lookup failed,

855 +- `UNDEFINED` - the alert failed to be calculated (i.e., the database lookup failed,

856 a division by zero occurred, etc.)

857

867 -- `CLEAR` - the alert is not armed / raised (i.e. is OK)

858 +- `CLEAR` - the alert is not armed / raised (i.e., is OK)

859

860 - `WARNING` - the warning expression resulted in true or non-zero

861

@@ -906,7 +897,7 @@ So, try to give something descriptive.

897 ```

898

899 The above applies the **template** to all charts that have `context = apache.requests`

909 -(i.e. all your apache servers).

900 +(i.e., all your apache servers).

901

902 ```yaml

903 calc: $now - $last_collected_t

@@ -928,19 +919,19 @@ The alert will be evaluated every 10 seconds.

919 crit: $this > (10 * $update_every)

920 ```

921

931 -If these result in non-zero or true, they trigger the alert.

922 +If these results in non-zero or true, they trigger the alert.

923

933 -- `$this` refers to the value of this alert (e.g. the result of the `calc` line).

924 +- `$this` refers to the value of this alert (e.g., the result of the `calc` line).

925 We could also use `$apache_last_collected_secs`.

926

927 `$update_every` is the update frequency of the chart, in seconds.

928

938 -So, the warning condition checks if we have not collected data from apache for 5

929 +So, the warning condition checks if we haven’t collected data from apache for five

930 iterations and the critical condition checks for 10 iterations.

931

932 ### Example 2 - disk space

933

943 -Check if any of the disks is critically low on disk space:

934 +Check if any of the disks are critically low on disk space:

935

936 ```yaml

937 template: disk_full_percent

@@ -956,12 +947,12 @@ template: disk_full_percent

947

948 So, the `calc` line finds the percentage of used space. `$this` resolves to this percentage.

949

959 -This is a repeating alert and if the alert becomes CRITICAL it repeats the notifications every 10 seconds. It also

950 +This is a repeating alert, and if the alert becomes CRITICAL, it repeats the notifications every 10 seconds. It also

951 repeats notifications every 2 minutes if the alert goes into WARNING mode.

952

953 ### Example 3 - disk fill rate

954

964 -Predict if any disk will run out of space in the near future.

955 +Predict if any disk will run out of space soon.

956

957 We do this in 2 steps:

958

@@ -975,8 +966,8 @@ Calculate the disk fill rate:

966 every: 15s

967 ```

968

978 -In the `calc` line: `$this` is the result of the `lookup` line (i.e. the free space 30 minutes

979 -ago) and `$avail` is the current disk free space. So the `calc` line will either have a positive

969 +In the `calc` line: `$this` is the result of the `lookup` line (i.e., the free space 30 minutes

970 +ago) and `$avail` is the current disk-free space. So the `calc` line will either have a positive

971 number of GB/second if the disk is filling up, or a negative number of GB/second if the disk is

972 freeing up space.

973

@@ -998,7 +989,7 @@ The `calc` line estimates the time in hours, we will run out of disk space. Of c

989 positive values are interesting for this check, so the warning and critical conditions check

990 for positive values and that we have enough free space for 48 and 24 hours respectively.

991

1001 -Once this alert triggers we will receive an email like this:

992 +Once this alert triggers, we will receive an email like this:

993

994 ![image](https://cloud.githubusercontent.com/assets/2662304/17839993/87872b32-6802-11e6-8e08-b2e4afef93bb.png)

995

@@ -1018,7 +1009,7 @@ The `lookup` line will calculate the sum of the all dropped packets in the last

1009

1010 The `crit` line will issue a critical alert if even a single packet has been dropped.

1011

1021 -Note that the drops chart does not exist if a network interface has never dropped a single packet.

1012 +Note that the drops chart doesn’t exist if a network interface has never dropped a single packet.

1013 When Netdata detects a dropped packet, it will add the chart, and it will automatically attach this

1014 alert to it.

1015

@@ -1050,7 +1041,7 @@ Since [`z = (x - mean) / stddev`](https://en.wikipedia.org/wiki/Standard_score)

1041

1042 ### Example 6 - [Anomaly rate](/src/ml/README.md#anomaly-rate) based CPU chart alert

1043

1053 -Warning if 5 minute rolling [anomaly rate](/src/ml/README.md#anomaly-rate) averaged across all CPU dimensions is above 5%, critical if it goes above 20%:

1044 +Warning if the 5-minute rolling [anomaly rate](/src/ml/README.md#anomaly-rate) averaged across all CPU dimensions is above 5%, critical if it goes above 20%:

1045

1046 ```yaml

1047 template: ml_5min_cpu_chart

@@ -1064,12 +1055,12 @@ template: ml_5min_cpu_chart

1055 info: rolling 5min anomaly rate for system.cpu chart

1056 ```

1057

1067 -The `lookup` line will calculate the average anomaly rate across all `system.cpu` dimensions over the last 5 minutes. In this case

1058 +The `lookup` line will calculate the average anomaly rate across all `system.cpu` dimensions over the last 5 minutes. In this case,

1059 Netdata will create one alert for the chart.

1060

1061 ### Example 7 - [Anomaly rate](/src/ml/README.md#anomaly-rate) based node level alert

1062

1072 -Warning if 5 minute rolling [anomaly rate](/src/ml/README.md#anomaly-rate) averaged across all ML enabled dimensions is above 5%, critical if it goes above 20%:

1063 +Warning if the 5-minute rolling [anomaly rate](/src/ml/README.md#anomaly-rate) averaged across all ML enabled dimensions is above 5%, critical if it goes above 20%:

1064

1065 ```yaml

1066 template: ml_5min_node

docs(health): clarify "special user of the conf operator" (#19589)