Docs: Add daemon config to health section and standardize IP references (#8837)
* Fix Chris' bug and cleanup * Fixes for Thiago * Fixes for Thiago * Rephrase the health files bullets * Fix in quickstart for Thiago * Fix path * Fix broken link
Joel Hans committed
Aug 19, 2020 at 08:04 UTC
c177910666871afd86e8284c73c59a03370de6b5
5 files changed
+64
-76
collectors/REFERENCE.md
+1
-12
@@ -15,17 +15,6 @@ the internal plugins API.
15
To learn the basics of collecting metrics from other applications and services, see the [collector
16
quickstart](QUICKSTART.md).
17
18
-## What's in this reference guide
19
-
20
-- [Netdata's collector architecture](#netdatas-collector-architecture)
21
-- [Enable, configure, and disable modules](#enable-configure-and-disable-modules)
22
-- [Troubleshoot a collector](#troubleshoot-a-collector)
23
-- [Enable and disable plugins](#enable-and-disable-plugins)
24
-- [Internal plugins](#internal-plugins)
25
- - [Internal plugins API](#internal-plugins-api)
26
-- [External plugins](#external-plugins)
27
-- [Write a custom collector](#write-a-custom-collector)
28
-
18
## Netdata's collector architecture
19
20
Netdata has an intricate system for organizing and managing its collectors. **Collectors** are the processes/programs
@@ -48,7 +37,7 @@ There are three types of plugins:
37
independent processes. They communicate with the daemon via pipes.
38
- **Plugin orchestrators**, which are external plugins that instead support a number of **modules**. Modules are a
39
type of collector. We have a few plugin orchestrators available for those who want to develop their own collectors,
51
- but focus most of our efforts on the [Go plugin](/collectors/go.d.plugin/README.md).
40
+ but focus most of our efforts on the [Go plugin](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/).
41
42
## Enable, configure, and disable modules
43
health/QUICKSTART.md
+9
-12
@@ -1,24 +1,18 @@
1
<!--
2
---
3
title: "Health quickstart"
4
+date: 2020-04-27
5
custom_edit_url: https://github.com/netdata/netdata/edit/master/health/QUICKSTART.md
6
---
7
-->
8
9
# Health quickstart
10
10
-In this getting started guide, you'll learn the basics of editing health configuration files. With this knowledge, you
11
+In this quickstart guide, you'll learn the basics of editing health configuration files. With this knowledge, you
12
will be able to customize how and when Netdata triggers alarms based on the health and performance of your system or
13
infrastructure.
14
14
-To learn about more advanced health configurations, visit the [health reference guide](REFERENCE.md).
15
-
16
-## What's in this getting started guide
17
-
18
-- [Edit health configuration files](#edit-health-configuration-files)
19
-- [Reference Netdata's stock health configuration files](#reference-netdatas-stock-health-configuration-files)
20
-- [Write a new health entity](#write-a-new-health-entity)
21
-- [Reload health configuration](#reload-health-configuration)
15
+To learn about more advanced health configurations, visit the [health reference guide](/health/REFERENCE.md).
16
17
## Edit health configuration files
18
@@ -50,8 +44,9 @@ health configuration files Netdata ships with. Stock files can be useful as refe
44
file you should edit with `edit-config`.
45
46
By default, Netdata will put health configuration files in `/usr/lib/netdata/conf.d/health.d`. However, you can
53
-double-check the location of these files by navigating to `http://HOST:19999/netdata.conf` in your browser and looking
54
-for the `stock health configuration directory` option. The value here will show the correct path for your installation.
47
+double-check the location of these files by navigating to `http://NODE:19999/netdata.conf`, replacing `NODE` with the IP
48
+address or hostname for your Agent dashboard, looking for the `stock health configuration directory` option. The value
49
+here will show the correct path for your installation.
50
51
```conf
52
[health]
@@ -138,7 +133,9 @@ killall -USR2 netdata
133
134
## What's next?
135
141
-To learn about all of Netdata's health configuration options, view the [reference guide](/health/REFERENCE.md).
136
+To learn about all of Netdata's health configuration options, view the [reference guide](/health/REFERENCE.md) and
137
+[daemon configuration](/daemon/config/README.md#health-section-options) for additional options available in the
138
+`[health]` section of `netdata.conf`.
139
140
Or, get guided insights into specific health configurations with our [health guides](/health/README.md#guides).
141
health/README.md
+11
-9
@@ -1,21 +1,23 @@
1
<!--
2
---
3
title: "Health monitoring"
4
+date: 2020-04-24
5
custom_edit_url: https://github.com/netdata/netdata/edit/master/health/README.md
6
---
7
-->
8
9
# Health monitoring
10
10
-With Netdata, you can monitor the health and performance of your systems and applications. You start with hundreds of
11
-alarms that have been pre-configured by the Netdata community, but you can write new alarms, tune existing ones, or
12
-silence any that you're not interested in.
11
+The Netdata Agent is a health watchdog for the health and performance of your systems, services, and applications. We've
12
+worked closely with our community of DevOps engineers, SREs, and developers to define hundreds of production-ready
13
+alarms that work without any configuration.
14
14
-Netdata creates charts dynamically, and runs an independent thread to constantly evaluate them, which means Netdata
15
-operates like a health watchdog for your services and applications.
15
+The Agent's health monitoring system is also dynamic and fully customizable. You can write entirely new alarms, tune the
16
+community-configured alarms for every app/service [the Agent collects metrics from](/collectors/COLLECTORS.md), or
17
+silence anything you're not interested in. You can even power complex lookups by running statistical algorithms against
18
+your metrics.
19
17
-You can even run statistical algorithms against the metrics you've collected to power complex lookups. Your imagination,
18
-and the needs of your infrastructure, are your only limits.
20
+Ready to take the next steps with health monitoring?
21
22
[Quickstart](/health/QUICKSTART.md)
23
@@ -33,7 +35,7 @@ content.
35
36
## Related features
37
36
-**[Health notifications](/health/notifications/README.md)**: Get notified about Netdata's alarms via your favorite
37
-platform(s).
38
+**[Notifications](/health/notifications/README.md)**: Get notified about ongoing alarms from your Agents via your
39
+favorite platform(s), such as Slack, Discord, PagerDuty, email, and much more.
40
41
[](<>)
health/REFERENCE.md
+24
-25
@@ -1,8 +1,8 @@
1
<!--
2
---
3
title: "Health configuration reference"
4
-date: 2020-03-31
5
-custom_edit_url: https://github.com/netdata/netdata/edit/master/health/REFERENCE.md
4
+date: 2020-04-27
5
+custom_edit_url: <https://github.com/netdata/netdata/edit/master/health/REFERENCE.md>
6
---
7
-->
8
@@ -13,28 +13,26 @@ Welcome to the health configuration reference.
13
This guide contains information about editing health configuration files to tweak existing alarms or create new health
14
entities that are customized to the needs of your infrastructure.
15
16
-To learn the basics of locating and editing health configuration files, see the [health quickstart](QUICKSTART.md).
16
+To learn the basics of locating and editing health configuration files, see the [health
17
+quickstart](/health/QUICKSTART.md).
18
18
-## What's in this reference guide
19
+## Health configuration files
20
20
-- [Health entity reference](#health-entity-reference)
21
- - [Entity types](#entity-types)
22
- - [Entity format](#entity-format)
23
-- [Expressions](#expressions)
24
- - [Special use of the conditional operator](#special-use-of-the-conditional-operator)
25
-- [Variables](#variables)
26
-- [Alarm statuses](#alarm-statuses)
27
-- [Example alarms](#example-alarms)
28
-- [Troubleshooting](#troubleshooting)
29
-- [Disabling health checks or silencing notifications at runtime](#disabling-health-checks-or-silencing-notifications-at-runtime)
21
+You can configure the Agent's health watchdog service by editing files in two locations:
22
+
23
+- The `[health]` section in `netdata.conf`. By editing the daemon's behavior, you can disable health monitoring
24
+ altogether, run health checks more or less often, and more. See [daemon
25
+ configuration](/daemon/config/README.md#health-section-options) for a table of all the available settings, their
26
+ default values, and what they control.
27
+- The individual `.conf` files in `health.d/`. These health entitiy files are organized by the type of metric they are
28
+ performing calculations on or their associated collector. You should edit these files using the `edit-config`
29
+ script. For example: `sudo ./edit-config health.d/cpu.conf`.
30
31
## Health entity reference
32
33
The following reference contains information about the syntax and options of _health entities_, which Netdata attaches
34
to charts in order to trigger alarms.
35
36
-Entities are written into `.conf` files, inside of the `health.d/` directory, using YAML formatting.
37
-
36
### Entity types
37
38
There are two entity types: **alarms** and **templates**. They have the same format and feature set—the only difference
@@ -483,8 +481,9 @@ Which in turn, results in the following behavior:
481
## Variables
482
483
You can find all the variables that can be used for a given chart, using
486
-`http://your.netdata.ip:19999/api/v1/alarm_variables?chart=CHART_NAME` Example: [variables for the `system.cpu` chart of
487
-the registry](https://registry.my-netdata.io/api/v1/alarm_variables?chart=system.cpu).
484
+`http://NODE:19999/api/v1/alarm_variables?chart=CHART_NAME`, replacing `NODE` with the IP address or hostname for your
485
+Agent dashboard. For example, [variables for the `system.cpu` chart of the
486
+registry](https://registry.my-netdata.io/api/v1/alarm_variables?chart=system.cpu).
487
488
> If you don't know how to find the CHART_NAME, you can read about it [here](../web/README.md#charts).
489
@@ -746,21 +745,21 @@ Netdata will create alarms for all dimensions of the chart.
745
746
## Troubleshooting
747
749
-You can compile Netdata with [debugging](../daemon/README.md#debugging) and then set in `netdata.conf`:
748
+You can compile Netdata with [debugging](/daemon/README.md#debugging) and then set in `netdata.conf`:
749
750
```yaml
751
[global]
752
debug flags = 0x0000000000800000
753
```
754
756
-Then check your `/var/log/netdata/debug.log`. It will show you how it works.
757
-Important: this will generate a lot of output in debug.log.
755
+Then check your `/var/log/netdata/debug.log`. It will show you how it works. Important: this will generate a lot of
756
+output in debug.log.
757
759
-You can find the context of charts by looking up the chart in either
760
-`http://your.netdata:19999/netdata.conf` or `http://your.netdata:19999/api/v1/charts`.
758
+You can find the context of charts by looking up the chart in either `http://NODE:19999/netdata.conf` or
759
+`http://NODE:19999/api/v1/charts`, replacing `NODE` with the IP address or hostname for your Agent dashboard.
760
761
You can find how Netdata interpreted the expressions by examining the alarm at
763
-`http://your.netdata:19999/api/v1/alarms?all`. For each expression, Netdata will return the expression as given in its
762
+`http://NODE:19999/api/v1/alarms?all`. For each expression, Netdata will return the expression as given in its
763
config file, and the same expression with additional parentheses added to indicate the evaluation flow of the
764
expression.
765
@@ -769,6 +768,6 @@ expression.
768
It's currently not possible to schedule notifications from within the alarm template. For those scenarios where you need
769
to temporary disable notifications (for instance when running backups triggers a disk alert) you can disable or silence
770
notifications are runtime. The health checks can be controlled at runtime via the [health management
772
-api](../web/api/health/).
771
+api](/web/api/health/README.md).
772
773
[](<>)
web/api/health/README.md
+19
-18
@@ -1,6 +1,7 @@
1
<!--
2
---
3
title: "Health API Calls"
4
+date: 2020-04-27
5
custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/health/README.md
6
---
7
-->
@@ -11,17 +12,17 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/web/api/health/R
12
13
### Enabled Alarms
14
14
-NetData enables alarms on demand, i.e. when the chart they should be linked to starts collecting data. So, although many more alarms are configured, only the useful ones are enabled.
15
+Netdata enables alarms on demand, i.e. when the chart they should be linked to starts collecting data. So, although many
16
+more alarms are configured, only the useful ones are enabled.
17
16
-To get the list of all enabled alarms:
17
-
18
-`http://your.netdata.ip:19999/api/v1/alarms?all`
18
+To get the list of all enabled alarms, open your browser and navigate to `http://NODE:19999/api/v1/alarms?all`,
19
+replacing `NODE` with the IP address or hostname for your Agent dashboard.
20
21
### Raised Alarms
22
23
This API call will return the alarms currently in WARNING or CRITICAL state.
24
24
-`http://your.netdata.ip:19999/api/v1/alarms`
25
+`http://NODE:19999/api/v1/alarms`
26
27
### Event Log
28
@@ -35,11 +36,11 @@ The size of the alarm log is configured in `netdata.conf`. There are 2 settings:
36
37
The API call retrieves all entries of the alarm log:
38
38
-`http://your.netdata.ip:19999/api/v1/alarm_log`
39
+`http://NODE:19999/api/v1/alarm_log`
40
41
### Alarm Log Incremental Updates
42
42
-`http://your.netdata.ip:19999/api/v1/alarm_log?after=UNIQUEID`
43
+`http://NODE:19999/api/v1/alarm_log?after=UNIQUEID`
44
45
The above returns all the events in the alarm log that occurred after UNIQUEID (you poll it once without `after=`, remember the last UNIQUEID of the returned set, which you give back to get incrementally the next events).
46
@@ -47,7 +48,7 @@ The above returns all the events in the alarm log that occurred after UNIQUEID (
48
49
The following will return an SVG badge of the alarm named `NAME`, attached to the chart named `CHART`.
50
50
-`http://your.netdata.ip:19999/api/v1/badge.svg?alarm=NAME&chart=CHART`
51
+`http://NODE:19999/api/v1/badge.svg?alarm=NAME&chart=CHART`
52
53
## Health Management API
54
@@ -60,7 +61,7 @@ Specifically, the API allows you to:
61
- Silence alarm notifications. Alarm conditions will be evaluated, the alarms will appear in the log and the Netdata UI will show the alarms as active, but no notifications will be sent.
62
- Disable or Silence specific alarms that match selectors on alarm/template name, chart, context, host and family.
63
63
-The API is available by default, but it is protected by an `api authorization token` that is stored in the file you will see in the following entry of `http://localhost:19999/netdata.conf`:
64
+The API is available by default, but it is protected by an `api authorization token` that is stored in the file you will see in the following entry of `http://NODE:19999/netdata.conf`:
65
66
```
67
[registry]
@@ -70,7 +71,7 @@ The API is available by default, but it is protected by an `api authorization to
71
You can access the API via GET requests, by adding the bearer token to an `Authorization` http header, like this:
72
73
```
73
-curl "http://myserver/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: Mytoken"
74
+curl "http://NODE:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: Mytoken"
75
```
76
77
By default access to the health management API is only allowed from `localhost`. Accessing the API from anything else will return a 403 error with the message `You are not allowed to access this resource.`. You can change permissions by editing the `allow management from` variable in `netdata.conf` within the [web] section. See [web server access lists](/web/server/README.md#access-lists) for more information.
@@ -83,14 +84,14 @@ If you've configured and entered your token correclty, you should see the plain
84
If all you need is temporarily disable all health checks, then you issue the following before your maintenance period starts:
85
86
```sh
86
-curl "http://myserver/api/v1/manage/health?cmd=DISABLE ALL" -H "X-Auth-Token: Mytoken"
87
+curl "http://NODE:19999/api/v1/manage/health?cmd=DISABLE ALL" -H "X-Auth-Token: Mytoken"
88
```
89
90
The effect of disabling health checks is that the alarm criteria are not evaluated at all and nothing is written in the alarm log.
91
If you want the health checks to be running but to not receive any notifications during your maintenance period, you can instead use this:
92
93
```sh
93
-curl "http://myserver/api/v1/manage/health?cmd=SILENCE ALL" -H "X-Auth-Token: Mytoken"
94
+curl "http://NODE:19999/api/v1/manage/health?cmd=SILENCE ALL" -H "X-Auth-Token: Mytoken"
95
```
96
97
Alarms may then still be raised and logged in Netdata, so you'll be able to see them via the UI.
@@ -98,7 +99,7 @@ Alarms may then still be raised and logged in Netdata, so you'll be able to see
99
Regardless of the option you choose, at the end of your maintenance period you revert to the normal state via the RESET command.
100
101
```sh
101
- curl "http://myserver/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: Mytoken"
102
+ curl "http://NODE:19999/api/v1/manage/health?cmd=RESET" -H "X-Auth-Token: Mytoken"
103
```
104
105
### Disable or silence specific alarms
@@ -122,7 +123,7 @@ To clear all selectors and reset the mode to default, use the `RESET` command.
123
The following example silences notifications for all the alarms with context=load:
124
125
```
125
-curl "http://myserver/api/v1/manage/health?cmd=SILENCE&context=load" -H "X-Auth-Token: Mytoken"
126
+curl "http://NODE:19999/api/v1/manage/health?cmd=SILENCE&context=load" -H "X-Auth-Token: Mytoken"
127
```
128
129
#### Selection criteria
@@ -142,19 +143,19 @@ You can add any of the selection criteria you need on the request, to ensure tha
143
Example 1: Disable all health checks for context = `random`
144
145
```
145
-http://localhost/api/v1/manage/health?cmd=DISABLE&context=random
146
+http://NODE:19999/api/v1/manage/health?cmd=DISABLE&context=random
147
```
148
149
Example 2: Silence all alarms and templates with name starting with `out_of` on host `myhost`
150
151
```
151
-http://localhost/api/v1/manage/health?cmd=SILENCE&alarm=out_of*&hosts=myhost
152
+http://NODE:19999/api/v1/manage/health?cmd=SILENCE&alarm=out_of*&hosts=myhost
153
```
154
155
Example 2.2: Add one more selector, to also silence alarms for cpu1 and cpu2
156
157
```
157
-http://localhost/api/v1/manage/health?families=cpu1 cpu2
158
+http://NODE:19999/api/v1/manage/health?families=cpu1 cpu2
159
```
160
161
### List silencers
@@ -162,7 +163,7 @@ http://localhost/api/v1/manage/health?families=cpu1 cpu2
163
The command `LIST` was added in Netdata v1.16.0 and returns a JSON with the current status of the silencers.
164
165
```
165
- curl "http://myserver/api/v1/manage/health?cmd=LIST" -H "X-Auth-Token: Mytoken"
166
+ curl "http://NODE:19999/api/v1/manage/health?cmd=LIST" -H "X-Auth-Token: Mytoken"
167
```
168
169
As an example, the following response shows that we have two silencers configured, one for an alarm called `samplealarm` and one for alarms with context `random` on host `myhost`