Copy outdated alert guides to health/guides (#16352)
Fotis Voutsas committed
Nov 7, 2023 at 14:57 UTC
bda3efe11fe5d526f4f4915641669269a0f3ce22
175 files changed
+12315
health/guides/README.md
new
+151
@@ -0,0 +1,151 @@
1
+# Alert Guides
2
+
3
+> This directory contains outdated guides on our alerts, and should be revisited and updated.
4
+
5
+# Review process
6
+
7
+Each guide must pass one technical review and one phrasal/grammatical review. Technical reviews will be equally assigned to the senior staff - considering the team's priorities.
8
+
9
+The review should focus on:
10
+
11
+- Is the guide useful and on topic?
12
+- Is it accurate? Does it have any unclear technical instruction? Please note which part.
13
+- Is it easy to read? If not, please note which part was not.
14
+- Is it rich in information? If not, please note which part we should explain more.
15
+
16
+Any other suggestions are welcomed.
17
+
18
+# Threshold presentation
19
+
20
+- Some alerts might use the conditional operator to determine on which state the alarm is. Let's break down this block
21
+ of code:
22
+
23
+```sh
24
+warn: $this > (($status >= $WARNING) ? (75) : (85))
25
+crit: $this > (($status == $CRITICAL) ? (85) : (95))
26
+```
27
+
28
+In the above:
29
+
30
+If the alarm is currently a warning, then the threshold for being considered a warning is 75, otherwise it's 85.
31
+
32
+If the alarm is currently critical, then the threshold for being considered critical is 85, otherwise it's 95.
33
+
34
+Which in turn, results in the following behavior:
35
+
36
+While the value is rising, it will trigger a warning when it exceeds 85, and a critical alert when it exceeds 95.
37
+
38
+While the value is falling, it will return to a warning state when it goes below 85, and a normal state when it goes
39
+below 75.
40
+
41
+If the value is fluctuating between 80 and 90, then it will trigger a warning the first time it goes above 85
42
+and will remain a warning until it goes below 75 (or goes above 85).
43
+
44
+If the value is fluctuating between 90 and 100, then it will trigger a critical alert first time it goes
45
+above 95 and will remain a critical alert until it goes below 85 - at which point it will return to being a warning.
46
+
47
+- In our guides we should write a sentence similar to the example below:
48
+
49
+> - This alert is raised in a warning state when the percentage of used IPv4 TCP connections is greater than 80% and
50
+ > less than 90%.
51
+>- If the metric exceeds 90%, then the alert gets raised in critical state.
52
+
53
+Note: the thresholds can be customized, the above is just an example using the conditional operator.
54
+
55
+<br>
56
+
57
+**We will construct each post following the template below as closely as possible:**
58
+
59
+<details>
60
+<summary>Template</summary>
61
+
62
+# alarm_name
63
+
64
+## OS: <OS_Name>
65
+
66
+The initial topic for each alarm should provide as much of the following information as possible:
67
+
68
+- How this component works.
69
+
70
+- A description of what is being monitored.
71
+
72
+- A general description of what the specific alarm is about.
73
+
74
+- Negative effects of this abnormality.
75
+
76
+<details>
77
+<summary>More information about a highly technical detail</summary>
78
+We organize highly technical details into collapsible contents to keep the reader focused.
79
+</details>
80
+
81
+<details>
82
+<summary>References and sources</summary>
83
+
84
+1. [Descriptive sentece of the link1](https://community.netdata.cloud/)
85
+2. [Descriptive sentece of the link2](https://community.netdata.cloud/)
86
+
87
+General guidelines on the referenced resources
88
+
89
+1. Starting the guide with a quote, we use italic.
90
+
91
+_Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut
92
+labore et dolore magna aliqua._ <sup>[1](https://community.netdata.cloud/) </sup>
93
+
94
+2. Large quote with bullets
95
+
96
+You can see the info as somebody says on site | documented on the
97
+site <sup>[1](https://community.netdata.cloud/) </sup>
98
+
99
+- Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut
100
+ labore et dolore magna aliqua
101
+
102
+- Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
103
+ consequat.
104
+
105
+- Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
106
+ pariatur.
107
+
108
+- Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id
109
+ est laborum.
110
+
111
+3. Quotes that are just chunks that are 4-10 lines, use block quotes.
112
+
113
+> Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla convallis lobortis urna eu
114
+sollicitudin. Maecenas vel euismod lacus, vel pulvinar diam. Curabitur luctus metus vitae eros
115
+auctor, et dictum dolor commodo. Vivamus turpis ipsum, placerat et cursus sed, finibus sed velit.
116
+Pellentesque efficitur gravida nibh sit amet lacinia. Vestibulum blandit, enim eu sodales
117
+ullamcorper, dui velit viverra enim, sed fermentum ante orci a risus. Proin cursus, justo id tempor
118
+molestie, dolor ante tempor ante, id dignissim tortor lacus id purus. Nam diam nibh, gravida vitae
119
+sem at, venenatis suscipit ante. Duis quis nulla vel mauris facilisis laoreet et faucibus dui.
120
+Pellentesque quis venenatis est, vitae pharetra leo. <sup>[1](https://community.netdata.cloud/) </sup>
121
+
122
+4. For quotes shorter than 4 lines, use quotation marks.
123
+
124
+"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut
125
+labore et dolore magna aliqua" <sup>[1](https://community.netdata.cloud/) </sup>
126
+
127
+</details>
128
+
129
+### Troubleshooting section:
130
+
131
+<details>
132
+<summary>Check these actions regarding to this or that</summary>
133
+Any independent set of actions/suggestions is explained in a collapsible layout. Here we propose any action the user
134
+can take.
135
+</details>
136
+
137
+<details>
138
+<summary>A second set of actions.</summary>
139
+Here we propose any action the user can take.
140
+</details>
141
+
142
+## OS: <Second_OS_Name>
143
+
144
+If the alarm exists in different OS and the underlying mechanisms have major changes or different
145
+troubleshooting sections, we create a section for each OS.
146
+
147
+### Troubleshooting section:
148
+
149
+And more actions like above.
150
+
151
+</details>
\ No newline at end of file
health/guides/adaptec_raid/adaptec_raid_ld_status.md
new
+65
@@ -0,0 +1,65 @@
1
+# adaptec_raid_ld_status
2
+
3
+## OS: Any
4
+
5
+A RAID controller is a card or chip located between the operating system and a storage drive (usually
6
+a hard drive). This is an alert about the Adaptec raid controller. The Netdata Agent checks
7
+the logical device statuses which are managed by your raid controller.
8
+
9
+This alert is triggered in critical state when a logical device state value is in degraded or failed
10
+state. This can indicate that one or more disks in your RAID configuration failed. Below you can
11
+find how tolerant is each raid configuration in cases of disk failures.
12
+
13
+<details>
14
+<summary>Fault tolerance for the most popular raid configurations </summary>
15
+
16
+- _RAID 0_ provides no fault tolerance. Any drive failures will cause data loss, so do not use this
17
+ on a mission critical server.
18
+
19
+- _RAID 1_ configuration is best used for situations where capacity isn't a requirement but data
20
+ protection is. This set up mirrors two disks so you can have 1 drive fail and still be able to
21
+ recover your data.
22
+
23
+- _RAID 5_ can withstand a single drive failure with a tradeoff in performance.
24
+
25
+- _RAID 6_ can withstand two disk failures at one time.
26
+
27
+- _RAID 10_ can survive a single drive failure per array.
28
+
29
+</details>
30
+
31
+Your system manages your Adaptec raid controller via the ARCCONF command line tool. You can find
32
+more information about this utility from
33
+the [user's guide for the ARCCONF](https://download.adaptec.com/pdfs/user_guides/microsemi_cli_smarthba_smartraid_v3_00_23484_ug.pdf).
34
+
35
+### Troubleshooting section
36
+
37
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
38
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
39
+
40
+Your Adaptec RAID card will automatically start to rebuild a faulty hard drive when you
41
+replace it with a healthy one. Sometimes this operation may take some time or may not start
42
+automatically.
43
+
44
+
45
+<details>
46
+<summary>Manually change the status of your ld </summary>
47
+This action will trigger a rebuild on your RAID.
48
+
49
+1. Verify that a rebuild is not in process.
50
+
51
+ ```
52
+ root@netdata # arcconf GETSTATUS <Controller_num>
53
+ ```
54
+
55
+ 2. Check for idle/missing segments of logical devices.
56
+
57
+
58
+
59
+3. Manually change your ld status
60
+
61
+ ```
62
+ root@netdata # arcconf SETSTATE <Controller_num> LOGICALDRIVE <LD_num> OPTIMAL ADVANCED nocheck noprompt
63
+ ```
64
+
65
+</details>
health/guides/adaptec_raid/adaptec_raid_pd_state.md
new
+50
@@ -0,0 +1,50 @@
1
+# adaptec_raid_pd_state
2
+
3
+## OS: Any
4
+
5
+A RAID controller is a card or chip located between the operating system and a storage drive (usually a hard drive). This is an alert about the Adaptec raid controller. The Netdata Agent checks
6
+the physical device statuses which are managed by your raid controller.
7
+
8
+This alert is triggered in critical state when the physical device is offline. Below you can
9
+find how tolerant each RAID configuration is in cases of disk failures.
10
+
11
+<details>
12
+<summary>Fault tolerance for the most popular raid configurations </summary>
13
+
14
+- _RAID 0_ provides no fault tolerance. Any drive failures will cause data loss, so do not use this
15
+ on a mission critical server.
16
+
17
+- _RAID 1_ configuration is best used for situations where capacity isn't a requirement but data
18
+ protection is. This set up mirrors two disks so you can have 1 drive fail and still be able to
19
+ recover your data.
20
+
21
+- _RAID 5_ can withstand a single drive failure with a tradeoff in performance.
22
+
23
+- _RAID 6_ can withstand two disk failures at one time.
24
+
25
+- _RAID 10_ can survive a single drive failure per array.
26
+
27
+</details>
28
+
29
+Your system manages your Adaptec raid controller via the ARCCONF command line tool. You can find
30
+more information about this utility in
31
+the [user's guide for the ARCCONF](https://download.adaptec.com/pdfs/user_guides/microsemi_cli_smarthba_smartraid_v3_00_23484_ug.pdf).
32
+
33
+### Troubleshooting section
34
+
35
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
36
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
37
+
38
+
39
+<details>
40
+<summary>Verify a bad disk </summary>
41
+
42
+Check the smart report for the drives in your RAID controller:
43
+
44
+ ```
45
+ root@netdata # arcconf GETSMARTSTATS 1
46
+ ```
47
+
48
+If a disk is degraded, you should consider replacing it. Your Adaptec RAID card will
49
+ automatically start to rebuild a faulty hard drive when you replace it with a healthy one.
50
+
health/guides/apcupsd/apcupsd_10min_ups_load.md
new
+22
@@ -0,0 +1,22 @@
1
+# apcupsd_10min_ups_load
2
+
3
+**Power Supply | UPS**
4
+
5
+This is an alert about your American Power Conversion (APC) uninterruptible power supply (UPS) device.
6
+The Netdata Agent calculates the average UPS load over the last 10 minutes. Receiving
7
+this alert means that your UPS has a very high load. This issue may result
8
+in either your UPS transferring to bypass mode or shutting down as a safety
9
+measure due to overload. You should remove some attached equipment from the UPS.
10
+
11
+This alert is triggered in warning state when the average UPS load is between 70-80% and in critical
12
+state when it is between 85-95%.
13
+
14
+### Troubleshooting section:
15
+
16
+<details>
17
+<summary>Reduce the load on the UPS</summary>
18
+
19
+To avoid ungraceful shutdowns of your systems, consider reducing the load on this particular UPS.
20
+To achieve this, consider removing attached devices that are not mission critical.
21
+
22
+</details>
health/guides/apcupsd/apcupsd_last_collected_secs.md
new
+27
@@ -0,0 +1,27 @@
1
+# apcupsd_last_collected_secs
2
+
3
+**Power Supply | UPS device**
4
+
5
+This is an alert about your American Power Conversion (APC) uninterruptible power supply (UPS) device.
6
+The Netdata Agent monitors the number of seconds since
7
+the last successful data collection by querying the `apcaccess` tool. This alert indicates that no
8
+data collection has taken place for some time.
9
+
10
+### Troubleshooting section:
11
+
12
+<details>
13
+<summary>Check the APCU daemon </summary>
14
+
15
+1. Check the status of the APCU daemon
16
+ ```
17
+ root@netdata $ systemctl status apcupsd
18
+ ```
19
+
20
+2. Check for obvious and common errors.
21
+
22
+
23
+3. Restart the APCU daemon, if needed
24
+ ```
25
+ root@netdata $ systemctl restart apcupsd
26
+ ```
27
+</details>
health/guides/apcupsd/apcupsd_ups_charge.md
new
+21
@@ -0,0 +1,21 @@
1
+# apcupsd_ups_charge
2
+
3
+**Power Supply | UPS**
4
+
5
+This is an alert about your American Power Conversion (APC) uninterruptible power supply (UPS) device.
6
+The Netdata Agent calculates the average UPS charge over
7
+the last minute. The UPS is running on battery, and it will shut down if external power is not
8
+restored. You should prepare any attached equipment for shutdown.
9
+
10
+This alert is triggered in warning state when the average UPS charge is less than 100% and in
11
+critical state when it is less than 50%.
12
+
13
+### Troubleshooting section:
14
+
15
+<details>
16
+<summary>Prepare your machine for graceful shutdown</summary>
17
+
18
+If you can't restore the power supply to this UPC, you should prepare your machine for graceful
19
+shutdown.
20
+
21
+</details>
health/guides/bcache/bcache_cache_dirty.md
new
+38
@@ -0,0 +1,38 @@
1
+# bcache_cache_dirty
2
+
3
+## OS: Any
4
+
5
+This alarm presents the percentage of `bcache` cache space used for `dirty` data and `metadata`. If this alarm is
6
+raised, it means that your SSD cache is too small, and overpopulated with said data.
7
+
8
+You can view `bcache_cache_dirty` as the `bcache` analogous metric to `dirty memory`. `dirty memory` is memory that has
9
+been changed but has not yet been written out to disk. For example, you make a change to a file but do not save it. These
10
+temporary changes are stored in memory, waiting to be written to disk.
11
+So `dirty` data on `bcache` is data that is stored on the cache disk and waits to be written to the backing device (
12
+Normally your HDD).
13
+
14
+> `bcache` is a cache in the block layer of the Linux Kernel. **It allows fast storage devices**, as SSDs
15
+> (Solid State Drives), **to act as a cache for slower storage devices**, such as HDDs (Hard Disk Drives). As a result,
16
+> **hybrid volumes are made with performance improvements**. Generally, a cache device is divided up into `buckets`,
17
+> matching the physical disk's erase blocks.
18
+
19
+> `dirty` data is data in the cache that has not been written to the backing device (normally your HDD). So when the
20
+> system shuts down, the cache device and the backing device are not safe to be separated.
21
+> `metadata` in general, is data that provides information about other data.
22
+
23
+Links:
24
+[kernel.org](https://www.kernel.org/doc/html/latest/admin-guide/bcache.html#)
25
+[Wikipedia](https://en.wikipedia.org/wiki/Bcache)
26
+[Bcache](https://wiki.archlinux.org/title/bcache)
27
+[Bcache: Caching beyond just RAM](https://lwn.net/Articles/394672/)
28
+
29
+### Troubleshooting section
30
+
31
+<details>
32
+<summary>Upgrade your cache's capacity</summary>
33
+
34
+The alarm is raised when there is more than 70% *(for warning status)* of your cache populated by `dirty` data and
35
+`metadata`, it means that your current cache device doesn't have the capacity to support your workflow. Using a bigger
36
+capacity device as cache can solve the problem.
37
+
38
+</details>
\ No newline at end of file
health/guides/bcache/bcache_cache_errors.md
new
+23
@@ -0,0 +1,23 @@
1
+# bcache_cache_errors
2
+
3
+## OS: Any
4
+
5
+This alarm presents the number of `bcache` read races in the last minute. The `bucket` was reused and invalidated while
6
+reading from the cache. When this occurs, the data is reread from the backing device.
7
+
8
+> `bcache` is a cache in the block layer of the Linux Kernel. **It allows fast storage devices**, as SSDs
9
+> (Solid State Drives), **to act as a cache for slower storage devices**, such as HDDs (Hard Disk Drives). As a result,
10
+> **hybrid volumes are made with performance improvements**. Generally, a cache device is divided up into `buckets`,
11
+> matching the physical disk's erase blocks.
12
+
13
+There is a mechanism where `bcache` can keep the cache disk full (typically your SSD), and when it needs to write more
14
+data, it selects a `bucket`, invalidates it, and removes all pointers from it. **The alarm got triggered, because while
15
+there was a reading operation from the cache** *(meaning the data is stored inside a bucket)* **that bucket got
16
+invalidated so the read operation couldn't be completed.** Following up, the data is reread from the backing device
17
+(normally your HDD).
18
+
19
+Links:
20
+[kernel.org](https://www.kernel.org/doc/html/latest/admin-guide/bcache.html#)
21
+[Wikipedia](https://en.wikipedia.org/wiki/Bcache)
22
+[Bcache](https://wiki.archlinux.org/title/bcache)
23
+[Bcache: Caching beyond just RAM](https://lwn.net/Articles/394672/)
\ No newline at end of file
health/guides/boinc/boinc_active_tasks.md
new
+22
@@ -0,0 +1,22 @@
1
+# boinc_active_tasks
2
+
3
+**Computing | BOINC**
4
+
5
+The Berkeley Open Infrastructure for Network Computing (BOINC) is an open-source middleware system
6
+for volunteer computing and grid computing. The Netdata Agent monitors the average number of active
7
+tasks over the last 10 minutes.
8
+
9
+### Troubleshooting sections
10
+
11
+
12
+<details>
13
+
14
+<summary>Restart the BOINC client</summary>
15
+
16
+1. Restart BOINC client, in most of the linux distros
17
+
18
+ ```
19
+ root@netdata # /etc/init.d/boinc-client restart
20
+ ```
21
+
22
+</details>
health/guides/boinc/boinc_compute_errors.md
new
+23
@@ -0,0 +1,23 @@
1
+# boinc_compute_errors
2
+
3
+**Computing | BOINC**
4
+
5
+The Berkeley Open Infrastructure for Network Computing (BOINC) is an open-source middleware system
6
+for volunteer computing and grid computing. The Netdata Agent monitors the average number of compute
7
+errors over the last 10 minutes.
8
+
9
+<details>
10
+
11
+<summary>Restart the BOINC client</summary>
12
+
13
+1. Abort the running task
14
+
15
+2. Restart BOINC client, in most of the linux distros
16
+
17
+ ```
18
+ root@netdata # /etc/init.d/boinc-client restart
19
+ ```
20
+
21
+</details>
22
+
23
+
health/guides/boinc/boinc_total_tasks.md
new
+30
@@ -0,0 +1,30 @@
1
+# boinc_total_tasks
2
+
3
+**Computing | BOINC**
4
+
5
+The Berkeley Open Infrastructure for Network Computing (BOINC) is an open-source middleware system
6
+for volunteer computing and grid computing. The Netdata Agent monitors the average number of total
7
+tasks over the last 10 minutes.
8
+
9
+### Troubleshooting sections
10
+
11
+<details>
12
+
13
+<summary>Verify the project status</summary>
14
+
15
+Verify that the projects you contribute are not suspended. Check if the project has queued tasks to
16
+be done (https://boinc.berkeley.edu/projects.php)
17
+
18
+</details>
19
+
20
+<details>
21
+
22
+<summary>Restart the BOINC client</summary>
23
+
24
+1. In most of the linux distros
25
+
26
+ ```
27
+ root@netdata # /etc/init.d/boinc-client restart
28
+ ```
29
+
30
+</details>
health/guides/boinc/boinc_upload_errors.md
new
+19
@@ -0,0 +1,19 @@
1
+# boinc_upload_errors
2
+
3
+**Computing | BOINC**
4
+
5
+The Berkeley Open Infrastructure for Network Computing (BOINC) is an open-source middleware system
6
+for volunteer computing and grid computing. The Netdata Agent calculates the average number of
7
+failed uploads over the last 10 minutes.
8
+
9
+For some reason your node couldn't upload the files manifested.
10
+
11
+### Troubleshooting section
12
+
13
+<details>
14
+<summary>Check for new BOINC client's version</summary>
15
+
16
+You can consult the [boinc client downloads page](https://boinc.berkeley.edu/download_all.php) to
17
+download the _recommended_ version for your system
18
+
19
+</details>
\ No newline at end of file
health/guides/btrfs/btrfs_allocated.md
new
+211
@@ -0,0 +1,211 @@
1
+# btrfs_allocated
2
+
3
+## OS: Any
4
+
5
+*Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing
6
+on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots,
7
+checksums, and integral multi-device spanning in Linux file systems.* [[1]](https://wiki.archlinux.org/title/btrfs)
8
+
9
+Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical
10
+disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data),
11
+metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the
12
+filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This
13
+metric tracks space usage in the first allocation stage.
14
+The Netdata Agent monitors the percentage of allocated Btrfs physical disk space.
15
+
16
+<details>
17
+<summary>Subvolumes in Btrfs </summary>
18
+
19
+> A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one).
20
+Most other POSIX filesystems have a single mountable root. Btrfs has an independent mountable root for the volume (top
21
+level subvolume) and for each subvolume. A Btrfs volume can contain more than a single filetree; it can contain a forest
22
+of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace.
23
+>
24
+> A subvolume in Btrfs is not the same as a LVM logical volume or a ZFS subvolume. With LVM, a logical volume is a block
25
+device in its own right, which could, for example, contain any other filesystem or container like dm-crypt, MD RAID,
26
+etc. This is not the case with Btrfs.
27
+>
28
+> A Btrfs subvolume root directory differs from a directory in that each subvolume defines a distinct inode number
29
+> space (
30
+distinct inodes in different subvolumes can have the same inumber) and each inode under a subvolume has a distinct
31
+device number (as reported by stat(2)). Each subvolume root can be accessed as implicitly mounted via the volume (top
32
+level subvolume) root, if that is mounted, or it can be mounted in its own right. [[2]](https://btrfs.wiki.kernel.org)
33
+
34
+
35
+</details>
36
+
37
+<details>
38
+<summary>Snapshots in Btrfs</summary>
39
+
40
+> A snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs's COW
41
+> capabilities.
42
+>
43
+> Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new
44
+snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, use mv to rename the old
45
+subvolume to a temporary location, and then rename the snapshot to the original name. You can then remount the
46
+subvolume.
47
+>
48
+> At this point, the original subvolume may be deleted, if desired. Since a snapshot is a subvolume, snapshots of
49
+snapshots are also possible.
50
+>
51
+> Caution: Care must be taken when snapshots are created that are then visible to any user (e.g. when they're created
52
+> in a nested layout) as this may have security implications. Of course, the snapshot will have the same permissions
53
+> as the subvolume from which it was created at the time it was, but these permissions may be tightened later on, while
54
+> those of the snapshot wouldn't change, possibly allowing access to files that shouldn't be accessible anymore.
55
+> Similarly, especially on the system's "main" filesystem, the snapshot would contain any files (for example, setuid
56
+> programs) of the state when it was created. In the meantime however, security updates may have been rolled out on
57
+> the original subvolume, but when the snapshot is accessible (and for example the vulnerable setuid has been accessible
58
+> before) a user could still invoke it. [[2]](https://btrfs.wiki.kernel.org)
59
+
60
+</details>
61
+
62
+<details>
63
+<summary>Useful commands for btrfs tool</summary>
64
+
65
+You can see some commands from the man pages of Btrfs in [Ubuntu man pages (Bionic)](https://manpages.ubuntu.com/manpages/bionic/man8)
66
+
67
+- `btrfs subvolume` is used to create/delete/list/show btrfs subvolumes and snapshots. For example,
68
+ with `btrfs subvolume list <path>`, you can list the subvolumes present in the filesystem `<path>`.
69
+
70
+
71
+- `btrfs filesystem` is used to perform several whole filesystem level tasks, including all the regular filesystem
72
+ operations like resizing, space stats, label setting/getting, and defragmentation. For example
73
+ with `btrfs filesystem show [<path>|<uuid>|<device>|<label>]` you can see the Btrfs filesystem with some additional
74
+ info about devices and space allocation.
75
+
76
+
77
+- `btrfs balance` can balance (restripe) the allocated extents across all of the existing devices. The primary purpose
78
+ of the balance feature is to spread block groups across all devices so they match constraints defined by the
79
+ respective profiles. The balance operation is cancellable by the user. The on-disk state of the filesystem is always
80
+ consistent so an unexpected interruption (eg. system crash, reboot) does not corrupt the filesystem. The progress of
81
+ the balance operation is temporarily stored as an internal state and will be resumed upon mount, unless the mount
82
+ option skip_balance is specified. **Running balance without filters will take a lot of time as it basically rewrites
83
+ the entire filesystem and needs to update all block pointers**.
84
+
85
+
86
+- `btrfs device` command group is used to manage devices of the btrfs filesystems. For example, with `btrfs device add`
87
+ command, you can add new devices to a mounted filesystem.
88
+
89
+
90
+- `btrfs rescue` is used to try to recover a damaged btrfs filesystem.
91
+
92
+
93
+- `btrfs scrub <subcommand>` scrub command attempts to report and repair bad blocks on Btrfs file systems.
94
+ **Scrubbing is performed in the background by default**.
95
+
96
+</details>
97
+
98
+
99
+<details>
100
+<summary>References and sources:</summary>
101
+
102
+1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
103
+1. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
104
+
105
+
106
+</details>
107
+
108
+### Troubleshooting section:
109
+
110
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is
111
+not liable for any loss or corruption of any data, database, or software.
112
+
113
+<details>
114
+<summary>Add more physical space</summary>
115
+
116
+Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a
117
+new disk to a filesystem is:
118
+
119
+1. Determine which disk you want to add and in which path
120
+ ```
121
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
122
+ ```
123
+
124
+1. If you get an error that the drive is already mounted, you might have to unmount
125
+ ```
126
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
127
+ ```
128
+1. See the newly added disk
129
+ ```
130
+ root@netdata~ # btrfs filesystem show
131
+ Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
132
+ Total devices 2 FS bytes used 192.00KiB
133
+ devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
134
+ devid 2 size 10.01GiB used 0.00B path /dev/sdb
135
+ ```
136
+
137
+1. Balance the system to make use of the new drive.
138
+ ```
139
+ root@netdata~ # btrfs filesystem balance <path>
140
+ ```
141
+
142
+</details>
143
+
144
+<details>
145
+<summary>Delete snapshots</summary>
146
+
147
+You can identify and delete snapshots that you no longer need.
148
+
149
+1. Find the snapshots for a specific path.
150
+ ```
151
+ root@netdata~ # sudo btrfs subvolume list -s <path>
152
+ ```
153
+
154
+1. Delete a snapshot that you do not need any more.
155
+ ```
156
+ root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
157
+ ```
158
+
159
+</details>
160
+
161
+<details>
162
+<summary>Enable a compression mechanism</summary>
163
+
164
+> The `compress=alg` mount option into the `fstab` configuration file (or in the `mount` procedure) enables automatically
165
+considering every file for compression, where `alg` is either `zlib`, `lzo`, `zstd`, or `no` (for no compression). Using
166
+this option, Btrfs will check if compressing the first portion of the data shrinks it. If it does, the entire write to
167
+that file will be compressed. If it does not, none of it is compressed. With this option, if the first portion of the
168
+write does not shrink, no compression will be applied to the write even if the rest of the data would shrink
169
+tremendously. This is done to prevent making the disk wait to start writing until all of the data to be written is fully
170
+given to Btrfs and compressed. [[1]](https://wiki.archlinux.org/title/btrfs)
171
+
172
+1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd`
173
+ compression algorithm.
174
+
175
+ ```
176
+ root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
177
+ ```
178
+
179
+</details>
180
+
181
+<details>
182
+<summary>Enable a deduplication mechanism</summary>
183
+
184
+Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a
185
+file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks
186
+of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
187
+
188
+Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd
189
+party, and it is strongly suggested that you check their status before you decide to use them.
190
+
191
+- [duperemove](https://github.com/markfasheh/duperemove)
192
+- [bees](https://github.com/Zygo/bees)
193
+- [dduper](https://github.com/lakshmipathi/dduper)
194
+
195
+</details>
196
+
197
+<details>
198
+<summary>Perform a balance</summary>
199
+
200
+Especially in a Btrfs with multiple disks, there might be unevenly allocated data/metadata into the disks.
201
+
202
+```
203
+root@netdata # btrfs balance start -musage=10 -dusage=10 -susage=5 /mount/point
204
+```
205
+
206
+> This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this
207
+> example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends
208
+> with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter
209
+> until at least some chunks are relocated.
210
+
211
+</details>
health/guides/btrfs/btrfs_data.md
new
+213
@@ -0,0 +1,213 @@
1
+# btrfs_data
2
+
3
+## OS: Any
4
+
5
+*Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing
6
+on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots,
7
+checksums, and integral multi-device spanning in Linux file systems.* [[1]](https://wiki.archlinux.org/title/btrfs)
8
+
9
+Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical
10
+disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data),
11
+metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the
12
+filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This
13
+metric tracks space usage in the first allocation stage.
14
+The Netdata Agent monitors the percentage of used Btrfs data space.
15
+
16
+<details>
17
+<summary>Subvolumes in Btrfs </summary>
18
+
19
+> A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one).
20
+Most other POSIX filesystems have a single mountable root. Btrfs has an independent mountable root for the volume (top
21
+level subvolume) and for each subvolume. A Btrfs volume can contain more than a single filetree; it can contain a forest
22
+of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace.
23
+
24
+> A subvolume in Btrfs is not the same as a LVM logical volume or a ZFS subvolume. With LVM, a logical volume is a block
25
+device in its own right, which could, for example, contain any other filesystem or container like dm-crypt, MD RAID,
26
+etc. This is not the case with Btrfs.
27
+
28
+> A Btrfs subvolume root directory differs from a directory in that each subvolume defines a distinct inode number
29
+> space (
30
+distinct inodes in different subvolumes can have the same inumber) and each inode under a subvolume has a distinct
31
+device number (as reported by stat(2)). Each subvolume root can be accessed as implicitly mounted via the volume (top
32
+level subvolume) root, if that is mounted, or it can be mounted in its own right. [[2]](https://btrfs.wiki.kernel.org)
33
+
34
+
35
+</details>
36
+
37
+<details>
38
+<summary>Snapshots in Btrfs</summary>
39
+
40
+> A snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs's COW
41
+> capabilities.
42
+>
43
+> Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new
44
+snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, use mv to rename the old
45
+subvolume to a temporary location, and then rename the snapshot to the original name. You can then remount the
46
+subvolume.
47
+>
48
+> At this point, the original subvolume may be deleted, if desired. Since a snapshot is a subvolume, snapshots of
49
+snapshots are also possible.
50
+>
51
+> Caution: Care must be taken when snapshots are created that are then visible to any user (e.g. when they're created
52
+> in a nested layout) as this may have security implications. Of course, the snapshot will have the same permissions
53
+> as the subvolume from which it was created at the time it was, but these permissions may be tightened later on, while
54
+> those of the snapshot wouldn't change, possibly allowing access to files that shouldn't be accessible anymore.
55
+> Similarly, especially on the system's "main" filesystem, the snapshot would contain any files (for example, setuid
56
+> programs) of the state when it was created. In the meantime however, security updates may have been rolled out on
57
+> the original subvolume, but when the snapshot is accessible (and for example the vulnerable setuid has been accessible
58
+> before) a user could still invoke it. [[2]](https://btrfs.wiki.kernel.org)
59
+
60
+</details>
61
+
62
+<details>
63
+<summary>Useful commands for btrfs tool</summary>
64
+
65
+You can see some commands from the man pages of Btrfs in [Ubuntu man pages (Bionic)](https://manpages.ubuntu.com/manpages/bionic/man8)
66
+
67
+- `btrfs subvolume` is used to create/delete/list/show btrfs subvolumes and snapshots. For example,
68
+ with `btrfs subvolume list <path>`, you can list the subvolumes present in the filesystem `<path>`.
69
+
70
+
71
+- `btrfs filesystem` is used to perform several whole filesystem level tasks, including all the regular filesystem
72
+ operations like resizing, space stats, label setting/getting, and defragmentation. For example
73
+ with `btrfs filesystem show [<path>|<uuid>|<device>|<label>]` you can see the Btrfs filesystem with some additional
74
+ info about devices and space allocation.
75
+
76
+
77
+- `btrfs balance` can balance (restripe) the allocated extents across all of the existing devices. The primary purpose
78
+ of the balance feature is to spread block groups across all devices so they match constraints defined by the
79
+ respective profiles. The balance operation is cancellable by the user. The on-disk state of the filesystem is always
80
+ consistent so an unexpected interruption (eg. system crash, reboot) does not corrupt the filesystem. The progress of
81
+ the balance operation is temporarily stored as an internal state and will be resumed upon mount, unless the mount
82
+ option skip_balance is specified. **Running balance without filters will take a lot of time as it basically rewrites
83
+ the entire filesystem and needs to update all block pointers**.
84
+
85
+
86
+- `btrfs device` command group is used to manage devices of the btrfs filesystems. For example, with `btrfs device add`
87
+ command, you can add new devices to a mounted filesystem.
88
+
89
+
90
+- `btrfs rescue` is used to try to recover a damaged btrfs filesystem.
91
+
92
+
93
+- `btrfs scrub <subcommand>` scrub command attempts to report and repair bad blocks on Btrfs file systems.
94
+ **Scrubbing is performed in the background by default**.
95
+
96
+</details>
97
+
98
+
99
+<details>
100
+<summary>References and sources:</summary>
101
+
102
+1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
103
+1. [https://btrfs.wiki.kernel.org](https://btrfs.wiki.kernel.org)
104
+
105
+
106
+</details>
107
+
108
+### Troubleshooting section:
109
+
110
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is
111
+not liable for any loss or corruption of any data, database, or software.
112
+
113
+
114
+<details>
115
+<summary>Add more physical space</summary>
116
+
117
+Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a
118
+new disk to a filesystem is:
119
+
120
+1. Determine which disk you want to add and in which path
121
+ ```
122
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
123
+ ```
124
+
125
+1. If you get an error that the drive is already mounted, you might have to unmount
126
+ ```
127
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
128
+ ```
129
+1. See the newly added disk
130
+ ```
131
+ root@netdata~ # btrfs filesystem show
132
+ Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
133
+ Total devices 2 FS bytes used 192.00KiB
134
+ devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
135
+ devid 2 size 10.01GiB used 0.00B path /dev/sdb
136
+ ```
137
+
138
+1. Balance the system to make use of the new drive.
139
+ ```
140
+ root@netdata~ # btrfs filesystem balance <path>
141
+ ```
142
+
143
+</details>
144
+
145
+<details>
146
+<summary>Delete snapshots</summary>
147
+
148
+You can identify and delete snapshots that you no longer need.
149
+
150
+1. Find the snapshots for a specific path.
151
+ ```
152
+ root@netdata~ # sudo btrfs subvolume list -s <path>
153
+ ```
154
+
155
+1. Delete a snapshot that you do not need any more.
156
+ ```
157
+ root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
158
+ ```
159
+
160
+</details>
161
+
162
+<details>
163
+<summary>Enable a compression mechanism</summary>
164
+
165
+> The `compress=alg` mount option into the `fstab` configuration file (or in the `mount` procedure) enables
166
+automatically
167
+considering every file for compression, where `alg` is either `zlib`, `lzo`, `zstd`, or `no` (for no compression). Using
168
+this option, Btrfs will check if compressing the first portion of the data shrinks it. If it does, the entire write to
169
+that file will be compressed. If it does not, none of it is compressed. With this option, if the first portion of the
170
+write does not shrink, no compression will be applied to the write even if the rest of the data would shrink
171
+tremendously. This is done to prevent making the disk wait to start writing until all of the data to be written is fully
172
+given to Btrfs and compressed. [[1]](https://wiki.archlinux.org/title/btrfs)
173
+
174
+1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd`
175
+ compression algorithm.
176
+
177
+ ```
178
+ root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
179
+ ```
180
+
181
+</details>
182
+
183
+<details>
184
+<summary>Enable a deduplication mechanism</summary>
185
+
186
+Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a
187
+file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks
188
+of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
189
+
190
+Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd
191
+party, and it is strongly suggested that you check their status before you decide to use them.
192
+
193
+- [duperemove](https://github.com/markfasheh/duperemove)
194
+- [bees](https://github.com/Zygo/bees)
195
+- [dduper](https://github.com/lakshmipathi/dduper)
196
+
197
+</details>
198
+
199
+<details>
200
+<summary>Perform a balance</summary>
201
+
202
+Especially in a Btrfs with multiple disks, there might be unevenly allocated data/metadata into the disks.
203
+
204
+```
205
+root@netdata # btrfs balance start -musage=10 -dusage=50 -susage=5 /mount/point
206
+```
207
+
208
+> This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this
209
+> example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends
210
+> with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter
211
+> until at least some chunks are relocated.
212
+
213
+</details>
health/guides/btrfs/btrfs_metadata.md
new
+214
@@ -0,0 +1,214 @@
1
+# btrfs_metadata
2
+
3
+## OS: Any
4
+
5
+*Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing
6
+on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots,
7
+checksums, and integral multi-device spanning in Linux file systems.* [[1]](https://wiki.archlinux.org/title/btrfs)
8
+
9
+Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical
10
+disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data),
11
+metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the
12
+filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This
13
+metric tracks space usage in the first allocation stage.
14
+The Netdata Agent monitors the percentage of used Btrfs metadata space.
15
+
16
+<details>
17
+<summary>Subvolumes in Btrfs </summary>
18
+
19
+> A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one).
20
+Most other POSIX filesystems have a single mountable root. Btrfs has an independent mountable root for the volume (top
21
+level subvolume) and for each subvolume. A Btrfs volume can contain more than a single filetree; it can contain a forest
22
+of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace.
23
+>
24
+> A subvolume in Btrfs is not the same as a LVM logical volume or a ZFS subvolume. With LVM, a logical volume is a block
25
+device in its own right, which could, for example, contain any other filesystem or container like dm-crypt, MD RAID,
26
+etc. This is not the case with Btrfs.
27
+>
28
+> A Btrfs subvolume root directory differs from a directory in that each subvolume defines a distinct inode number
29
+> space (
30
+distinct inodes in different subvolumes can have the same inumber) and each inode under a subvolume has a distinct
31
+device number (as reported by stat(2)). Each subvolume root can be accessed as implicitly mounted via the volume (top
32
+level subvolume) root, if that is mounted, or it can be mounted in its own right. [[2]](https://btrfs.wiki.kernel.org)
33
+
34
+
35
+</details>
36
+
37
+<details>
38
+<summary>Snapshots in Btrfs</summary>
39
+
40
+> A snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs's COW
41
+> capabilities.
42
+>
43
+> Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new
44
+snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, use mv to rename the old
45
+subvolume to a temporary location, and then rename the snapshot to the original name. You can then remount the
46
+subvolume.
47
+>
48
+> At this point, the original subvolume may be deleted, if desired. Since a snapshot is a subvolume, snapshots of
49
+snapshots are also possible.
50
+>
51
+> Caution: Care must be taken when snapshots are created that are then visible to any user (e.g. when they're created
52
+> in a nested layout) as this may have security implications. Of course, the snapshot will have the same permissions
53
+> as the subvolume from which it was created at the time it was, but these permissions may be tightened later on, while
54
+> those of the snapshot wouldn't change, possibly allowing access to files that shouldn't be accessible anymore.
55
+> Similarly, especially on the system's "main" filesystem, the snapshot would contain any files (for example, setuid
56
+> programs) of the state when it was created. In the meantime however, security updates may have been rolled out on
57
+> the original subvolume, but when the snapshot is accessible (and for example the vulnerable setuid has been accessible
58
+> before) a user could still invoke it. [[2]](https://btrfs.wiki.kernel.org)
59
+
60
+</details>
61
+
62
+<details>
63
+<summary>Useful commands for btrfs tool</summary>
64
+
65
+You can see some commands from the man pages of Btrfs in [Ubuntu man pages (Bionic)](https://manpages.ubuntu.com/manpages/bionic/man8)
66
+
67
+- `btrfs subvolume` is used to create/delete/list/show btrfs subvolumes and snapshots. For example,
68
+ with `btrfs subvolume list <path>`, you can list the subvolumes present in the filesystem `<path>`.
69
+
70
+
71
+- `btrfs filesystem` is used to perform several whole filesystem level tasks, including all the regular filesystem
72
+ operations like resizing, space stats, label setting/getting, and defragmentation. For example
73
+ with `btrfs filesystem show [<path>|<uuid>|<device>|<label>]` you can see the Btrfs filesystem with some additional
74
+ info about devices and space allocation.
75
+
76
+
77
+- `btrfs balance` can balance (restripe) the allocated extents across all of the existing devices. The primary purpose
78
+ of the balance feature is to spread block groups across all devices so they match constraints defined by the
79
+ respective profiles. The balance operation is cancellable by the user. The on-disk state of the filesystem is always
80
+ consistent so an unexpected interruption (eg. system crash, reboot) does not corrupt the filesystem. The progress of
81
+ the balance operation is temporarily stored as an internal state and will be resumed upon mount, unless the mount
82
+ option skip_balance is specified. **Running balance without filters will take a lot of time as it basically rewrites
83
+ the entire filesystem and needs to update all block pointers**.
84
+
85
+
86
+- `btrfs device` command group is used to manage devices of the btrfs filesystems. For example, with `btrfs device add`
87
+ command, you can add new devices to a mounted filesystem.
88
+
89
+
90
+
91
+- `btrfs rescue` is used to try to recover a damaged btrfs filesystem.
92
+
93
+
94
+- `btrfs scrub <subcommand>` scrub command attempts to report and repair bad blocks on Btrfs file systems.
95
+ **Scrubbing is performed in the background by default**.
96
+
97
+</details>
98
+
99
+<details>
100
+<summary>References and sources:</summary>
101
+
102
+1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
103
+1. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
104
+
105
+
106
+</details>
107
+
108
+### Troubleshooting section:
109
+
110
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is
111
+not liable for any loss or corruption of any data, database, or software.
112
+
113
+
114
+<details>
115
+<summary>Add more physical space</summary>
116
+
117
+Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a
118
+new disk to a filesystem is:
119
+
120
+1. Determine which disk you want to add and in which path
121
+ ```
122
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
123
+ ```
124
+
125
+1. If you get an error that the drive is already mounted, you might have to unmount
126
+ ```
127
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
128
+ ```
129
+1. See the newly added disk
130
+ ```
131
+ root@netdata~ # btrfs filesystem show
132
+ Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
133
+ Total devices 2 FS bytes used 192.00KiB
134
+ devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
135
+ devid 2 size 10.01GiB used 0.00B path /dev/sdb
136
+ ```
137
+
138
+1. Balance the system to make use of the new drive.
139
+ ```
140
+ root@netdata~ # btrfs filesystem balance <path>
141
+ ```
142
+
143
+</details>
144
+
145
+<details>
146
+<summary>Delete snapshots</summary>
147
+
148
+You can identify and delete snapshots that you no longer need.
149
+
150
+1. Find the snapshots for a specific path.
151
+ ```
152
+ root@netdata~ # sudo btrfs subvolume list -s <path>
153
+ ```
154
+
155
+1. Delete a snapshot that you do not need any more.
156
+ ```
157
+ root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
158
+ ```
159
+
160
+</details>
161
+
162
+<details>
163
+<summary>Enable a compression mechanism</summary>
164
+
165
+> The `compress=alg` mount option into the `fstab` configuration file (or in the `mount` procedure) enables
166
+> automatically
167
+considering every file for compression, where `alg` is either `zlib`, `lzo`, `zstd`, or `no` (for no compression). Using
168
+this option, Btrfs will check if compressing the first portion of the data shrinks it. If it does, the entire write to
169
+that file will be compressed. If it does not, none of it is compressed. With this option, if the first portion of the
170
+write does not shrink, no compression will be applied to the write even if the rest of the data would shrink
171
+tremendously. This is done to prevent making the disk wait to start writing until all of the data to be written is fully
172
+given to Btrfs and compressed. [[1]](https://wiki.archlinux.org/title/btrfs)
173
+
174
+1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd`
175
+ compression algorithm.
176
+
177
+
178
+ ```
179
+ root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
180
+ ```
181
+
182
+</details>
183
+
184
+<details>
185
+<summary>Enable a deduplication mechanism</summary>
186
+
187
+Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a
188
+file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks
189
+of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
190
+
191
+Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd
192
+party, and it is strongly suggested that you check their status before you decide to use them.
193
+
194
+- [duperemove](https://github.com/markfasheh/duperemove)
195
+- [bees](https://github.com/Zygo/bees)
196
+- [dduper](https://github.com/lakshmipathi/dduper)
197
+
198
+</details>
199
+
200
+<details>
201
+<summary>Perform a balance</summary>
202
+
203
+Especially in a Btrfs with multiple disks, there might be unevenly allocated data/metadata into the disks.
204
+
205
+```
206
+root@netdata # btrfs balance start -musage=50 -dusage=10 -susage=5 /mount/point
207
+```
208
+
209
+> This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this
210
+> example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends
211
+> with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter
212
+> until at least some chunks are relocated.
213
+
214
+</details>
health/guides/btrfs/btrfs_system.md
new
+211
@@ -0,0 +1,211 @@
1
+# btrfs_system
2
+
3
+## OS: Any
4
+
5
+*Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing
6
+on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots,
7
+checksums, and integral multi-device spanning in Linux file systems.* [[1]](https://wiki.archlinux.org/title/btrfs)
8
+
9
+Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical
10
+disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data),
11
+metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the
12
+filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This
13
+metric tracks space usage in the first allocation stage.
14
+The Netdata Agent monitors the percentage of used Btrfs system space.
15
+
16
+<details>
17
+<summary>Subvolumes in Btrfs </summary>
18
+
19
+> A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one).
20
+Most other POSIX filesystems have a single mountable root. Btrfs has an independent mountable root for the volume (top
21
+level subvolume) and for each subvolume. A Btrfs volume can contain more than a single filetree; it can contain a forest
22
+of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace.
23
+>
24
+> A subvolume in Btrfs is not the same as a LVM logical volume or a ZFS subvolume. With LVM, a logical volume is a block
25
+device in its own right, which could, for example, contain any other filesystem or container like dm-crypt, MD RAID,
26
+etc. This is not the case with Btrfs.
27
+>
28
+> A Btrfs subvolume root directory differs from a directory in that each subvolume defines a distinct inode number space
29
+(distinct inodes in different subvolumes can have the same inumber) and each inode under a subvolume has a distinct
30
+device number (as reported by stat(2)). Each subvolume root can be accessed as implicitly mounted via the volume (top
31
+level subvolume) root, if that is mounted, or it can be mounted in its own right. [[2]](https://btrfs.wiki.kernel.org)
32
+
33
+
34
+</details>
35
+
36
+<details>
37
+<summary>Snapshots in Btrfs</summary>
38
+
39
+> A snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs's COW
40
+> capabilities.
41
+>
42
+> Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new
43
+snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, use mv to rename the old
44
+subvolume to a temporary location, and then rename the snapshot to the original name. You can then remount the
45
+subvolume.
46
+>
47
+> At this point, the original subvolume may be deleted, if desired. Since a snapshot is a subvolume, snapshots of
48
+snapshots are also possible.
49
+>
50
+> Caution: Care must be taken when snapshots are created that are then visible to any user (e.g. when they're created
51
+> in a nested layout) as this may have security implications. Of course, the snapshot will have the same permissions
52
+> as the subvolume from which it was created at the time it was, but these permissions may be tightened later on, while
53
+> those of the snapshot wouldn't change, possibly allowing access to files that shouldn't be accessible anymore.
54
+> Similarly, especially on the system's "main" filesystem, the snapshot would contain any files (for example, setuid
55
+> programs) of the state when it was created. In the meantime however, security updates may have been rolled out on
56
+> the original subvolume, but when the snapshot is accessible (and for example the vulnerable setuid has been accessible
57
+> before) a user could still invoke it. [[2]](https://btrfs.wiki.kernel.org)
58
+
59
+</details>
60
+
61
+<details>
62
+<summary>Useful commands for btrfs tool</summary>
63
+
64
+You can see some commands from the man pages of Btrfs in [Ubuntu man pages (Bionic)](https://manpages.ubuntu.com/manpages/bionic/man8)
65
+
66
+- `btrfs subvolume` is used to create/delete/list/show btrfs subvolumes and snapshots. For example,
67
+ with `btrfs subvolume list <path>`, you can list the subvolumes present in the filesystem `<path>`.
68
+
69
+
70
+- `btrfs filesystem` is used to perform several whole filesystem level tasks, including all the regular filesystem
71
+ operations like resizing, space stats, label setting/getting, and defragmentation. For example
72
+ with `btrfs filesystem show [<path>|<uuid>|<device>|<label>]` you can see the Btrfs filesystem with some additional
73
+ info about devices and space allocation.
74
+
75
+
76
+- `btrfs balance` can balance (restripe) the allocated extents across all of the existing devices. The primary purpose
77
+ of the balance feature is to spread block groups across all devices so they match constraints defined by the
78
+ respective profiles. The balance operation is cancellable by the user. The on-disk state of the filesystem is always
79
+ consistent so an unexpected interruption (eg. system crash, reboot) does not corrupt the filesystem. The progress of
80
+ the balance operation is temporarily stored as an internal state and will be resumed upon mount, unless the mount
81
+ option skip_balance is specified. **Running balance without filters will take a lot of time as it basically rewrites
82
+ the entire filesystem and needs to update all block pointers**.
83
+
84
+
85
+- `btrfs device` command group is used to manage devices of the btrfs filesystems. For example, with `btrfs device add`
86
+ command, you can add new devices to a mounted filesystem.
87
+
88
+
89
+- `btrfs rescue` is used to try to recover a damaged btrfs filesystem.
90
+
91
+
92
+- `btrfs scrub <subcommand>` scrub command attempts to report and repair bad blocks on Btrfs file systems.
93
+ **Scrubbing is performed in the background by default**.
94
+
95
+</details>
96
+
97
+
98
+<details>
99
+<summary>References and sources:</summary>
100
+
101
+1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
102
+1. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
103
+
104
+
105
+</details>
106
+
107
+### Troubleshooting section:
108
+
109
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is
110
+not liable for any loss or corruption of any data, database, or software.
111
+
112
+<details>
113
+<summary>Add more physical space</summary>
114
+
115
+Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a
116
+new disk to a filesystem is:
117
+
118
+1. Determine which disk you want to add and in which path
119
+ ```
120
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
121
+ ```
122
+
123
+1. If you get an error that the drive is already mounted, you might have to unmount
124
+ ```
125
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
126
+ ```
127
+1. See the newly added disk
128
+ ```
129
+ root@netdata~ # btrfs filesystem show
130
+ Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
131
+ Total devices 2 FS bytes used 192.00KiB
132
+ devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
133
+ devid 2 size 10.01GiB used 0.00B path /dev/sdb
134
+ ```
135
+
136
+1. Balance the system to make use of the new drive.
137
+ ```
138
+ root@netdata~ # btrfs filesystem balance <path>
139
+ ```
140
+
141
+</details>
142
+
143
+<details>
144
+<summary>Delete snapshots</summary>
145
+
146
+You can identify and delete snapshots that you no longer need.
147
+
148
+1. Find the snapshots for a specific path.
149
+ ```
150
+ root@netdata~ # sudo btrfs subvolume list -s <path>
151
+ ```
152
+
153
+1. Delete a snapshot that you do not need any more.
154
+ ```
155
+ root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
156
+ ```
157
+
158
+</details>
159
+
160
+<details>
161
+<summary>Enable a compression mechanism</summary>
162
+
163
+> The `compress=alg` mount option into the `fstab` configuration file (or in the `mount` procedure) enables
164
+automatically considering every file for compression, where `alg` is either `zlib`, `lzo`, `zstd`, or `no` (for no
165
+compression). Using this option, Btrfs will check if compressing the first portion of the data shrinks it. If it does,
166
+the entire write to
167
+that file will be compressed. If it does not, none of it is compressed. With this option, if the first portion of the
168
+write does not shrink, no compression will be applied to the write even if the rest of the data would shrink
169
+tremendously. This is done to prevent making the disk wait to start writing until all of the data to be written is fully
170
+-given to Btrfs and compressed. [[1]](https://wiki.archlinux.org/title/btrfs)
171
+
172
+1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd`
173
+ compression algorithm.
174
+
175
+ ```
176
+ root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
177
+ ```
178
+
179
+</details>
180
+
181
+<details>
182
+<summary>Enable a deduplication mechanism</summary>
183
+
184
+Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a
185
+file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks
186
+of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
187
+
188
+Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd
189
+party, and it is strongly suggested that you check their status before you decide to use them.
190
+
191
+- [duperemove](https://github.com/markfasheh/duperemove)
192
+- [bees](https://github.com/Zygo/bees)
193
+- [dduper](https://github.com/lakshmipathi/dduper)
194
+
195
+</details>
196
+
197
+<details>
198
+<summary>Perform a balance</summary>
199
+
200
+Especially in a Btrfs with multiple disks, data/metadata might be unevenly allocated into the disks.
201
+
202
+```
203
+root@netdata # btrfs balance start -musage=10 -dusage=10 -susage=50 /mount/point
204
+```
205
+
206
+> This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this
207
+> example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends
208
+> with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter
209
+> until at least some chunks are relocated.
210
+
211
+</details>
health/guides/ceph/ceph_cluster_space_usage.md
new
+34
@@ -0,0 +1,34 @@
1
+# ceph_cluster_space_usage
2
+
3
+**Storage | Ceph**
4
+
5
+Ceph is an open-source software-defined storage platform that implements object storage on a single
6
+distributed computer cluster and provides 3-in-1 interfaces for object-, block- and file-level
7
+storage. Ceph aims primarily for completely distributed operation without a single point of failure,
8
+scalability to the exabyte level, and to be freely available.
9
+
10
+The Netdata Agent calculates the percentage of used cluster disk space. Your cluster is in high disk
11
+space utilization.
12
+
13
+This alert is triggered in warning state when the percentage of used cluster disk space is between
14
+85-90% and in critical state when it is between 90-98%.
15
+
16
+### Troubleshooting section
17
+
18
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
19
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
20
+
21
+
22
+<details>
23
+<summary>Examine your cluster status </summary>
24
+
25
+1. In the master node, examine the disk space details of the cluster
26
+
27
+ ```
28
+ root@netdata # ceph df detail
29
+
30
+ ```
31
+
32
+2. Check for unused pools and delete them or and consider adding a node to your cluster.
33
+
34
+</details>
\ No newline at end of file
health/guides/cgroups/cgroup_10min_cpu_usage.md
new
+212
@@ -0,0 +1,212 @@
1
+# cgroup_10min_cpu_usage
2
+
3
+**Cgroups | CPU**
4
+
5
+_Control groups, usually referred to as Cgroups, are a Linux kernel feature which allow processes to
6
+be organized into hierarchical groups whose usage of various types of resources can then be limited
7
+and monitored. The Kernel's cgroup interface is provided through a pseudo-filesystem called
8
+cgroupfs (`/sys/fs/cgroup`). Grouping is implemented in the core Cgroup Kernel code, while resource
9
+tracking and limits are implemented in a set of per-resource-type subsystems (memory, CPU, and so
10
+on).<sup>[1](https://man7.org/linux/man-pages/man7/cgroups.7.html) </sup>_
11
+
12
+The Netdata Agent calculates the average CPU utilization over the last 10 minutes. This alert
13
+indicates that one of the group of you processes is in high CPU utilization. The system will
14
+throttle the group CPU usage when the usage is over the limit.
15
+
16
+This alert is raised in warning state when the average CPU utilization is between 75-80% of the
17
+limits and in critical when it is between 85-95%.
18
+
19
+<details>
20
+<summary>More about Cgroups</summary>
21
+
22
+Since Linux kernel 4.5 (March 2016) there are two implementations of Cgroups, v1 and v2.
23
+
24
+#### Cgroup controllers (in v1)
25
+
26
+In the vanilla kernel you will find the following controllers.
27
+
28
+- cpu: `CONFIG_CGROUP_SCHED`, Cgroups can be guaranteed a minimum number of "CPU shares" when a
29
+ system is busy.
30
+- cpuacct: `CONFIG_CGROUP_CPUACCT`, This controller provides accounting for CPU usage by groups of
31
+ processes.
32
+- cpuset: `CONFIG_CPUSETS`, This Cgroup can be used to bind the processes in a Cgroup to a specified
33
+ set of CPUs and NUMA nodes.
34
+- memory: `CONFIG_MEMCG`, The memory controller supports reporting and limiting of process memory,
35
+ kernel memory, and swap used by Cgroups.
36
+- devices: `CONFIG_CGROUP_DEVICE`, This supports controlling which processes may create (mknod)
37
+ devices as well as open them for reading or writing. The policies may be specified as allow-lists
38
+ and deny-lists. Hierarchy is enforced, so new rules must not violate existing rules for the target
39
+ or ancestor Cgroups.
40
+- freezer: `CONFIG_CGROUP_FREEZER`, The freezer Cgroup can suspend and restore (resume) all
41
+ processes in a Cgroup. Freezing a Cgroup /A also causes its children, for example, processes in
42
+ /A/B, to be frozen.
43
+- net_cls: `CONFIG_CGROUP_NET_CLASSID`, This places a classid, specified for the Cgroup, on network
44
+ packets created by a Cgroup. These classids can then be used in firewall rules, as well as used to
45
+ shape traffic using `tc`. This applies only to packets leaving the Cgroup, not to traffic arriving
46
+ at the Cgroup.
47
+- blkio: `CONFIG_BLK_CGROUP`, The blkio Cgroup controls and limits access to specified block devices
48
+ by applying IO control in the form of throttling and upper limits against leaf nodes and
49
+ intermediate nodes in the storage hierarchy. Two policies are available. The first is a
50
+ proportional- weight time-based division of disk implemented with CFQ. This is in effect for leaf
51
+ nodes using CFQ. The second is a throttling policy which specifies upper I/O rate limits on a
52
+ device.
53
+- perf_event: `CONFIG_CGROUP_PERF`, This controller allows perf monitoring of the set of processes
54
+ grouped in a Cgroup.
55
+- net_prio: `CONFIG_CGROUP_NET_PRIO`, This allows priorities to be specified, per network interface,
56
+ for Cgroups.
57
+- hugetlb: `CONFIG_CGROUP_HUGETLB`, This supports limiting the use of huge pages by Cgroups.
58
+- pids: `CONFIG_CGROUP_PIDS` , This controller permits limiting the number of process that may be
59
+ created in a Cgroup (
60
+ and its descendants).
61
+- rdma: `CONFIG_CGROUP_RDMA`, The RDMA controller permits limiting the use of RDMA/IB- specific
62
+ resources per Cgroup. _since Linux 4.11_
63
+
64
+Different variations of the Linux kernel can have more or less Cgroup controllers OR/AND enabled not
65
+all of them.
66
+
67
+#### Cgroups v1
68
+
69
+Under Cgroups v1, each controller may be mounted against a separate Cgroup filesystem that provides
70
+its own hierarchical organization of the processes on the system. It is also possible to co-mount
71
+multiple (or even all) Cgroups v1 controllers against the same Cgroup filesystem. That means that
72
+the mounted controllers manage the same hierarchical organization of processes.
73
+
74
+For each mounted hierarchy, the directory tree mirrors the control group hierarchy. Each control
75
+group is represented by a directory, with each of its child control Cgroups represented as a child
76
+directory. For instance, `/user/joe/1.session` represents control group 1.session, which is a child
77
+of Cgroup joe, which is a child of /user. Under each Cgroup directory is a set of files which can be
78
+read or written to, reflecting resource limits and a few general Cgroup properties.
79
+
80
+##### Hierarchy in Cgroups v1
81
+
82
+The Cgroups v1 is organized in a tree way:
83
+`/sys/fs/cgroup/{controller: cpu, cpuacct, cpupids, memory}/{cgroup_process: {process A, process B docker}/{rules: } `
84
+
85
+#### Cgroups v2
86
+
87
+In Cgroups v2, all mounted controllers reside in a single unified hierarchy. While (different)
88
+controllers may be simultaneously mounted under the v1 and v2 hierarchies, it is not possible to
89
+mount the same controller simultaneously under both the v1 and the v2 hierarchies.
90
+
91
+The new behaviors in Cgroups v2 are summarized here, and in some cases elaborated in the following
92
+subsections.
93
+
94
+1. Cgroups v2 provides a unified hierarchy against which all controllers are mounted.
95
+
96
+2. "Internal" processes are not permitted. Except of the root Cgroup, processes may reside only in
97
+ leaf nodes
98
+ (Cgroups that do not themselves contain child Cgroups). The details are somewhat more subtle than
99
+ this, and are described below.
100
+
101
+3. Active Cgroups must be specified via the files cgroup.controllers and cgroup.subtree_control.
102
+
103
+4. The tasks file has been removed. In addition, the cgroup.clone_children file that is employed by
104
+ the cpuset controller has been removed.
105
+
106
+5. An improved mechanism for notification of empty Cgroups is provided by the cgroup.events file.
107
+
108
+ For more changes, see the Documentation/admin-guide/cgroup-v2.rst file in the kernel source (or
109
+ Documentation/cgroup-v2.txt in Linux 4.17 and earlier).
110
+
111
+##### Hierarchy in Cgroups v2
112
+
113
+The Cgroups v2 is organized in a tree way:
114
+`/sys/fs/cgroup/{cgroup1: {/cgroup2, /cgroup3}, cgroup5: {/cgroup6: {/cgroup8} } } . . .`
115
+
116
+So every Cgroup contains over Cgroups, cgroup1 contains cgroup2 and cgroup3, cgroup5 contains
117
+cgroup6 which contains cgroup8 and so on so forth. So all the parent cgroups/processes shares their
118
+resource with their childs. Active Cgroups must be specified via the `cgroup.controllers` and
119
+`cgroup.subtree_control` files. Consult
120
+the [cgroup man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
121
+
122
+Cgroups v2 implements only a subset of the controllers available in Cgroups v1. **The two systems
123
+are implemented so that both v1 controllers and v2 controllers can be mounted on the same system**.
124
+Thus, for example, it is possible to use those controllers that are supported under version 2, while
125
+also using version 1 controllers where version 2 does not yet support those controllers. The only
126
+restriction here is that a controller can't be simultaneously employed in both a Cgroups v1
127
+hierarchy and in the Cgroups v2 hierarchy.
128
+
129
+#### Major between Cgroups v1 & v2:
130
+
131
+You might already have figure out some differences from the explanations above but let's note some
132
+of them.
133
+
134
+The biggest difference is that in Cgroups v2 a process can't join different groups for different
135
+controllers. For example in v1 a process could use the /sys/fs/cgroup/cpu/cgroupA and the
136
+/sys/fs/cgroup/memory/cgroupB controllers at the same time. In v2 a process joins only the
137
+/sys/fs/cgroups/cgroupC and is subject to all the controller of this Cgroup.
138
+
139
+Another difference is that, in Cgroups v1 when events like page cache writebacks or network packets
140
+reception occur, the resources allocated was not charged to the responsible Cgroup. For example,
141
+when your network interface receives packets, and the kernel is not aware of the destination Cgroup
142
+, the packets need to be processed first and re-routed to the corresponding subsystem or userland
143
+application. This operation was not included/calculated in the applications Cgroup.
144
+
145
+### More
146
+
147
+You can experiment with Cgroups, (safely, in a testing environment) create a new Cgroup and set limits
148
+in your system with tools like `cgcreate` and `cgset`from the `libcgroup-tools` package and then
149
+make your services run with those constrains. In the following link you can find a
150
+blog [in the linuxhint blog](https://linuxhint.com/limit_cpu_usage_process_linux/) which can guide
151
+you through it.
152
+
153
+</details>
154
+
155
+
156
+<details>
157
+<summary>References and sources:</summary>
158
+
159
+1. [cgroups(7) man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html)
160
+2. [linuxhint blog](https://linuxhint.com/limit_cpu_usage_process_linux/)
161
+3. [cgroups on Arch linux](https://wiki.archlinux.org/title/cgroups)
162
+
163
+</details>
164
+
165
+### Troubleshooting section
166
+
167
+When your service/app/container reaches it's Cgroup (cpu) hard limits (or on pressure the soft
168
+limits), it would be halted or/and start thrashing. You can imagine the case when a service halts in
169
+a lock, nightmare! If needed, you can extend those limits:
170
+
171
+<details>
172
+<summary>Native linux applications</summary>
173
+
174
+Control groups can be accessed with various tools, please consult this
175
+guide ([cgroups on Arch linux](https://wiki.archlinux.org/title/cgroups)) to do that
176
+
177
+For example if you would like to set a CPU share limit in your apache server:
178
+
179
+ ```
180
+ root@netdata # systemctl set-property --runtime httpd.service CPUShares=500
181
+ ```
182
+
183
+**Note**: To change it permanently, omit the `--runtime` flag
184
+
185
+</details>
186
+
187
+<details>
188
+<summary>Docker containers</summary>
189
+
190
+Follow
191
+the [Configure the default CFS scheduler](https://docs.docker.com/config/containers/resource_constraints/#configure-the-default-cfs-scheduler)
192
+section in the official docs to do that.
193
+
194
+</details>
195
+
196
+<details>
197
+<summary>Kubernetes </summary>
198
+
199
+You can apply Cgroups restrictions in any manifest that creates Pods. For example in a Deployment,
200
+in the PodSpecs configuration you can apply limits under `..spec.resources.limits.cpu: <value>`
201
+Consult
202
+this [resource management for pods and container](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
203
+guide in the official docs.
204
+
205
+You can also apply limits per namespaces or other policy, which is the best practice for handling
206
+Kubernetes resources. Consult
207
+this [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) guide in the
208
+official docs.
209
+
210
+
211
+</details>
212
+
health/guides/cgroups/cgroup_ram_in_use.md
new
+208
@@ -0,0 +1,208 @@
1
+# cgroup_ram_in_use
2
+
3
+**Cgroups | Memory**
4
+
5
+_Control groups, usually referred to as Cgroups, are a Linux kernel feature which allow processes to
6
+be organized into hierarchical groups whose usage of various types of resources can then be limited
7
+and monitored. The Kernel's cgroup interface is provided through a pseudo-filesystem called
8
+cgroupfs (`/sys/fs/cgroup`). Grouping is implemented in the core Cgroup kernel code, while resource tracking and limits
9
+are implemented in a set of per-resource-type subsystems (memory, CPU, and so on)
10
+.<sup>[1](https://man7.org/linux/man-pages/man7/cgroups.7.html) </sup>_
11
+
12
+The Netdata Agent calculates the percentage of used memory in a Cgroup compared to it's hard memory
13
+limit. This alert indicates high Cgroup memory utilization. When the utilization reaches it's 100%
14
+limit, the Out Of Memory (OOM) killer will be invoked and will try to evict the process that has the
15
+least favorable OOM_score in this particular Cgroup.
16
+
17
+This alert is triggered in warning state when the percentage of used memory is between 80-90% and in
18
+critical state between 90-98%.
19
+
20
+
21
+<details>
22
+<summary>More about Cgroups</summary>
23
+
24
+Since Linux kernel 4.5 (March 2016) there are two implementations of Cgroups, v1 and v2.
25
+
26
+#### Cgroup controllers (in v1)
27
+
28
+In the vanilla kernel you will find the following controllers.
29
+
30
+- cpu: `CONFIG_CGROUP_SCHED`, Cgroups can be guaranteed a minimum number of "CPU shares" when a
31
+ system is busy.
32
+- cpuacct: `CONFIG_CGROUP_CPUACCT`, This controller provides accounting for CPU usage by groups of
33
+ processes.
34
+- cpuset: `CONFIG_CPUSETS`, This Cgroup can be used to bind the processes in a Cgroup to a specified
35
+ set of CPUs and NUMA nodes.
36
+- memory: `CONFIG_MEMCG`, The memory controller supports reporting and limiting of process memory,
37
+ kernel memory, and swap used by Cgroups.
38
+- devices: `CONFIG_CGROUP_DEVICE`, This supports controlling which processes may create (mknod)
39
+ devices as well as open them for reading or writing. The policies may be specified as allow-lists
40
+ and deny-lists. Hierarchy is enforced, so new rules must not violate existing rules for the target
41
+ or ancestor Cgroups.
42
+- freezer: `CONFIG_CGROUP_FREEZER`, The freezer Cgroup can suspend and restore (resume) all
43
+ processes in a Cgroup. Freezing a Cgroup /A also causes its children, for example, processes in
44
+ /A/B, to be frozen.
45
+- net_cls: `CONFIG_CGROUP_NET_CLASSID`, This places a classid, specified for the Cgroup, on network
46
+ packets created by a Cgroup. These classids can then be used in firewall rules, as well as used to
47
+ shape traffic using `tc`. This applies only to packets leaving the Cgroup, not to traffic arriving
48
+ at the Cgroup.
49
+- blkio: `CONFIG_BLK_CGROUP`, The blkio Cgroup controls and limits access to specified block devices
50
+ by applying IO control in the form of throttling and upper limits against leaf nodes and
51
+ intermediate nodes in the storage hierarchy. Two policies are available. The first is a
52
+ proportional- weight time-based division of disk implemented with CFQ. This is in effect for leaf
53
+ nodes using CFQ. The second is a throttling policy which specifies upper I/O rate limits on a
54
+ device.
55
+- perf_event: `CONFIG_CGROUP_PERF`, This controller allows perf monitoring of the set of processes
56
+ grouped in a Cgroup.
57
+- net_prio: `CONFIG_CGROUP_NET_PRIO`, This allows priorities to be specified, per network interface,
58
+ for Cgroups.
59
+- hugetlb: `CONFIG_CGROUP_HUGETLB`, This supports limiting the use of huge pages by Cgroups.
60
+- pids: `CONFIG_CGROUP_PIDS` , This controller permits limiting the number of process that may be
61
+ created in a Cgroup (
62
+ and its descendants).
63
+- rdma: `CONFIG_CGROUP_RDMA`, The RDMA controller permits limiting the use of RDMA/IB- specific
64
+ resources per Cgroup. _since Linux 4.11_
65
+
66
+Different variations of the Linux kernel can have more or less Cgroup controllers OR/AND enabled not
67
+all of them.
68
+
69
+#### Cgroups v1
70
+
71
+Under Cgroups v1, each controller may be mounted against a separate Cgroup filesystem that provides
72
+its own hierarchical organization of the processes on the system. It is also possible to co-mount
73
+multiple (or even all) Cgroups v1 controllers against the same Cgroup filesystem. That means that
74
+the mounted controllers manage the same hierarchical organization of processes.
75
+
76
+For each mounted hierarchy, the directory tree mirrors the control group hierarchy. Each control
77
+group is represented by a directory, with each of its child control Cgroups represented as a child
78
+directory. For instance, `/user/joe/1.session` represents control group 1.session, which is a child
79
+of Cgroup joe, which is a child of /user. Under each Cgroup directory is a set of files which can be
80
+read or written to, reflecting resource limits and a few general Cgroup properties.
81
+
82
+##### Hierarchy in Cgroups v1
83
+
84
+The Cgroups v1 is organized in a tree way:
85
+`/sys/fs/cgroup/{controller: cpu, cpuacct, cpupids, memory}/{cgroup_process: {process A, process B docker}/{rules: } `
86
+
87
+#### Cgroups v2
88
+
89
+In Cgroups v2, all mounted controllers reside in a single unified hierarchy. While (different)
90
+controllers may be simultaneously mounted under the v1 and v2 hierarchies, it is not possible to
91
+mount the same controller simultaneously under both the v1 and the v2 hierarchies.
92
+
93
+The new behaviors in Cgroups v2 are summarized here, and in some cases elaborated in the following
94
+subsections.
95
+
96
+1. Cgroups v2 provides a unified hierarchy against which all controllers are mounted.
97
+
98
+2. "Internal" processes are not permitted. Except of the root Cgroup, processes may reside only in
99
+ leaf nodes
100
+ (Cgroups that do not themselves contain child Cgroups). The details are somewhat more subtle than
101
+ this, and are described below.
102
+
103
+3. Active Cgroups must be specified via the files cgroup.controllers and cgroup.subtree_control.
104
+
105
+4. The tasks file has been removed. In addition, the cgroup.clone_children file that is employed by
106
+ the cpuset controller has been removed.
107
+
108
+5. An improved mechanism for notification of empty Cgroups is provided by the cgroup.events file.
109
+
110
+ For more changes, see the Documentation/admin-guide/cgroup-v2.rst file in the kernel source (or
111
+ Documentation/cgroup-v2.txt in Linux 4.17 and earlier).
112
+
113
+##### Hierarchy in Cgroups v2
114
+
115
+The Cgroups v2 is organized in a tree way:
116
+`/sys/fs/cgroup/{cgroup1: {/cgroup2, /cgroup3}, cgroup5: {/cgroup6: {/cgroup8} } } . . .`
117
+
118
+So every Cgroup contains over Cgroups, cgroup1 contains cgroup2 and cgroup3, cgroup5 contains
119
+cgroup6 which contains cgroup8 and so on so forth. So all the parent cgroups/processes shares their
120
+resource with their childs. Active Cgroups must be specified via the `cgroup.controllers` and
121
+`cgroup.subtree_control` files. Consult
122
+the [cgroup man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
123
+
124
+Cgroups v2 implements only a subset of the controllers available in Cgroups v1. **The two systems
125
+are implemented so that both v1 controllers and v2 controllers can be mounted on the same system**.
126
+Thus, for example, it is possible to use those controllers that are supported under version 2, while
127
+also using version 1 controllers where version 2 does not yet support those controllers. The only
128
+restriction here is that a controller can't be simultaneously employed in both a Cgroups v1
129
+hierarchy and in the Cgroups v2 hierarchy.
130
+
131
+#### Major between Cgroups v1 & v2:
132
+
133
+You might already have figure out some differences from the explanations above but let's note some
134
+of them.
135
+
136
+The biggest difference is that in Cgroups v2 a process can't join different groups for different
137
+controllers. For example in v1 a process could use the /sys/fs/cgroup/cpu/cgroupA and the
138
+/sys/fs/cgroup/memory/cgroupB controllers at the same time. In v2 a process joins only the
139
+/sys/fs/cgroups/cgroupC and is subject to all the controller of this Cgroup.
140
+
141
+Another difference is that, in Cgroups v1 when events like page cache writebacks or network packets
142
+reception occur, the resources allocated was not charged to the responsible Cgroup. For example,
143
+when your network interface receives packets, and the kernel is not aware of the destination Cgroup
144
+, the packets need to be processed first and re-routed to the corresponding subsystem or userland
145
+application. This operation was not included/calculated in the applications Cgroup.
146
+
147
+### More
148
+
149
+You can experiment with Cgroups, (safely, in a testing environment) create a new Cgroup and set limits
150
+in your system with tools like `cgcreate` and `cgset`from the `libcgroup-tools` package and then
151
+make your services run with those constrains. In the following link you can find a
152
+blog [in the linuxhint blog](https://linuxhint.com/limit_cpu_usage_process_linux/) which can guide
153
+you through it.
154
+
155
+</details>
156
+
157
+<details>
158
+<summary>References and sources:</summary>
159
+
160
+1. [cgroups(7) man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html)
161
+2. [linuxhint blog](https://linuxhint.com/limit_cpu_usage_process_linux/)
162
+3. [cgroups on Arch linux](https://wiki.archlinux.org/title/cgroups)
163
+
164
+</details>
165
+
166
+### Troubleshooting section
167
+
168
+<details>
169
+<summary>Native linux applications</summary>
170
+
171
+Control groups can be accessed with various tools, please consult this
172
+guide ([cgroups on Arch linux](https://wiki.archlinux.org/title/cgroups)) to do that.
173
+
174
+For example if you would like to set a Memory limit in your apache server:
175
+
176
+ ```
177
+ root@netdata # systemctl set-property httpd.service MemoryLimit=500M
178
+ ```
179
+
180
+**Note**: To change it permanently, omit the `--runtime` flag
181
+
182
+</details>
183
+
184
+<details>
185
+<summary>Docker containers</summary>
186
+
187
+Follow
188
+the [limit a container's access to memory](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory)
189
+section in the official docs to do that.
190
+
191
+</details>
192
+
193
+<details>
194
+<summary>Kubernetes </summary>
195
+
196
+You can apply restriction in any manifest that create Pods. For example in a Deployment, in the
197
+PodSpecs configuration you can apply limits under `..spec.resources.limits.memory: <value>`
198
+Consult
199
+this [resource management for pods and container](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
200
+guide in the official docs.
201
+
202
+You can also apply limits per namespaces or other policy, which is the best practice for handling
203
+Kubernetes resources. Consult
204
+this [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) guide in the
205
+official docs.
206
+
207
+
208
+</details>
health/guides/cockroachdb/cockroachdb_open_file_descriptors_limit.md
new
+33
@@ -0,0 +1,33 @@
1
+# cockroachdb_open_file_descriptors_limit
2
+
3
+## Database | CockroachDB
4
+
5
+This alert presents the percentage of used file descriptors for CockroachDB.
6
+If you receive this, it means that there is high file descriptor utilization against the
7
+soft-limit.
8
+
9
+This alert is raised in a warning state when the metric exceeds 80%.
10
+
11
+> In Unix and Unix-like computer operating systems, a file descriptor (FD, less frequently
12
+> fildes) is a unique identifier (handle) for a file or other input/output resource, such as a
13
+> pipe or network socket.
14
+>
15
+> File descriptors typically have non-negative integer values, with negative values being
16
+> reserved to indicate "no value" or error conditions.<sup>[1](
17
+> https://en.wikipedia.org/wiki/File_descriptor) </sup>
18
+
19
+<details><summary>References and Sources</summary>
20
+
21
+1. [CockroachDB documentation](
22
+ https://www.cockroachlabs.com/docs/v21.2/recommended-production-settings#file-descriptors-limit)
23
+
24
+</details>
25
+
26
+### Troubleshooting Section
27
+
28
+<details><summary>Adjust the file descriptors limit for the process or system-wide</summary>
29
+
30
+Check out the [CockroachDB documentation](
31
+https://www.cockroachlabs.com/docs/v21.2/recommended-production-settings#file-descriptors-limit) for troubleshooting advice.
32
+
33
+</details>
health/guides/cockroachdb/cockroachdb_unavailable_ranges.md
new
+39
@@ -0,0 +1,39 @@
1
+# cockroachdb_unavailable_ranges
2
+
3
+## Database | CockroachDB
4
+
5
+This alert presents the number of unavailable ranges. If you receive this, it indicates that there
6
+are ranges with fewer live replicas than needed for quorum.
7
+
8
+This alert is raised in a warning state when unavailable ranges start to exist.
9
+
10
+<details><summary>What are unavailable ranges?</summary>
11
+
12
+> Unavailable ranges: If a majority of a range's replicas are on nodes that are unavailable,
13
+> then the entire range is unavailable and will be unable to process queries.
14
+>
15
+> CockroachDB uses consensus replication and requires a quorum of the replicas to
16
+> be available in order to allow both writes and reads to the range. The number of failures
17
+> that can be tolerated is equal to (Replication factor - 1)/2. Thus, CockroachDB requires (n-1)
18
+> /2 nodes to achieve quorum. For example, with 3x replication, one failure can be tolerated;
19
+> with 5x replication, two failures, and so on.<sup>[1](
20
+> https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges) </sup>
21
+
22
+</details>
23
+
24
+<details><summary>References and Sources</summary>
25
+
26
+1. [CockroachDB docs](
27
+ https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges)
28
+
29
+</details>
30
+
31
+### Troubleshooting Section
32
+
33
+<details><summary>Identify unavailable ranges</summary>
34
+
35
+Check out the [CockroachDB documentation](
36
+https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges) for troubleshooting advice.
37
+
38
+
39
+</details>
health/guides/cockroachdb/cockroachdb_underreplicated_ranges.md
new
+35
@@ -0,0 +1,35 @@
1
+# cockroachdb_underreplicated_ranges
2
+
3
+## Database | CockroachDB
4
+
5
+This alert presents the number of under-replicated ranges.
6
+
7
+This alert is raised in a warning state when under-replicated ranges start to exist.
8
+
9
+> Under-replicated ranges: When a cluster is first initialized, the few default starting ranges
10
+> will only have a single replica, but as soon as other nodes are available, they will
11
+> replicate to them until they've reached their desired replication factor. If a range does not
12
+> have enough replicas, the range is said to be "under-replicated".
13
+>
14
+> CockroachDB uses consensus replication and requires a quorum of the replicas to
15
+> be available in order to allow both writes and reads to the range. The number of failures
16
+> that can be tolerated is equal to (Replication factor - 1)/2. Thus, CockroachDB requires (n-1)
17
+> /2 nodes to achieve quorum. For example, with 3x replication, one failure can be tolerated;
18
+> with 5x replication, two failures, and so on.<sup>[1](
19
+> https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges) </sup>
20
+
21
+<details><summary>References and Sources</summary>
22
+
23
+1. [CockroachDB documentation](
24
+ https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges)
25
+
26
+</details>
27
+
28
+### Troubleshooting Section
29
+
30
+<details><summary>Identify under-replicated ranges</summary>
31
+
32
+Check out the [CockroachDB documentation](
33
+https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges)
34
+for troubleshooting advice.
35
+</details>
health/guides/cockroachdb/cockroachdb_used_storage_capacity.md
new
+39
@@ -0,0 +1,39 @@
1
+# cockroachdb_used_storage_capacity
2
+
3
+## Database | CockroachDB
4
+
5
+This alert presents the percentage of used storage space.
6
+If you receive this alert, then there is high storage capacity utilization from CockroachDB.
7
+
8
+- This alert is raised to warning when the metric exceeds 85%.
9
+- If the percentage of used storage space exceeds 95%, then the alert is raised to critical.
10
+
11
+Definition of "size" on CockroachDB:
12
+
13
+> The maximum size allocated to the node. When this size is reached, CockroachDB attempts to
14
+> rebalance data to other nodes with available capacity. When there's no capacity elsewhere,
15
+> this limit will be exceeded. Also, data may be written to the node faster than the cluster
16
+> can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB
17
+> will gradually rebalance data down to the store limit.<sup>[1](
18
+> https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store) </sup>
19
+
20
+
21
+<details><summary>References and Sources</summary>
22
+
23
+1. [CockroachDB Size](https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store)
24
+2. [CockroachDB Docs](https://www.cockroachlabs.com/docs/stable/ui-storage-dashboard.html)
25
+</details>
26
+
27
+### Troubleshooting Section
28
+
29
+<details><summary>Increase the space available for CockroachDB data</summary>
30
+
31
+If you had previously set a limit, then you can use the option `--store=path<YOUR PATH>,size=<SIZE>`
32
+to increase the amount of available space.
33
+Make sure to replace the "YOUR PATH" with the actual store path and "SIZE" with the new size you want to set CockroachDB to.
34
+
35
+> Note: If you haven't set a limit on the size, then the entire drive's size will be used. In this
36
+> case, you will see that the drive is full. Clearing some space or upgrading to a drive with a
37
+> larger capacity are potential solutions.
38
+
39
+</details>
health/guides/cockroachdb/cockroachdb_used_usable_storage_capacity.md
new
+34
@@ -0,0 +1,34 @@
1
+# cockroachdb_used_usable_storage_capacity
2
+
3
+## Database | CockroachDB
4
+
5
+This alert presents the percentage of used usable storage space allocated on CockroachDB.
6
+
7
+If you receive this alert, it means that the usable space for CockroachDB is being highly utilized.
8
+
9
+- This alert is raised to warning when the metric exceeds 85%.
10
+- If the percentage of used usable storage space exceeds 95%, then the alert is raised to critical.
11
+
12
+Definition of "size" on CockroachDB:
13
+
14
+> The maximum size allocated to the node. When this size is reached, CockroachDB attempts to
15
+> rebalance data to other nodes with available capacity. When there's no capacity elsewhere,
16
+> this limit will be exceeded. Also, data may be written to the node faster than the cluster
17
+> can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB
18
+> will gradually rebalance data down to the store limit.<sup>[1](
19
+> https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store) </sup>
20
+
21
+<details><summary>References and Sources</summary>
22
+
23
+1. [CockroachDB Size](https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store)
24
+2. [CockroachDB Docs](https://www.cockroachlabs.com/docs/stable/ui-storage-dashboard.html)
25
+
26
+</details>
27
+
28
+### Troubleshooting Section
29
+
30
+<details><summary>Increase the space available for CockroachDB data</summary>
31
+
32
+You can use the option `--store=path<YOUR PATH>,size=<SIZE>` Make sure to replace the "YOUR PATH"
33
+with the actual store path and "SIZE" with the new size you want to set CockroachDB to.
34
+</details>
health/guides/cpu/10min_cpu_iowait.md
new
+72
@@ -0,0 +1,72 @@
1
+# 10min_cpu_iowait
2
+
3
+## OS: Linux
4
+
5
+This alarm calculates the average time of `iowait` through 10 minute interval periods. `iowait` is the percentage of
6
+time where there has been at least one I/O request in progress while the CPU has been idle.
7
+
8
+We call I/O -on a process level- the use of the read and write services, such as reading data from a physical
9
+drive.
10
+
11
+It's important to note that during the time a process waits on I/O, the system can schedule other processes, but
12
+`iowait` is measured specifically while the CPU is idle.
13
+
14
+A common example of when this alert might be triggered would be when your CPU requests some data and the device
15
+responsible for it can't deliver it fast enough. As a result the CPU (in the next clock interrupt) is idle, so you
16
+encounter `iowait`. If this persists for some time and the average from the metrics we gather exceeds the value that is
17
+being checked in the `.conf` file, then the alert is raised because the CPU is being bottlenecked by your system’s
18
+disks. <sup> [1](https://blog.pregos.info/wp-content/uploads/2010/09/iowait.txt) [2](
19
+https://serverfault.com/questions/12679/can-anyone-explain-precisely-what-iowait-is) </sup>
20
+
21
+<details>
22
+<summary>References and Sources</summary>
23
+
24
+[[1] What exactly is "iowait"?](https://blog.pregos.info/wp-content/uploads/2010/09/iowait.txt)
25
+[[2] Serverfault](https://serverfault.com/questions/12679/can-anyone-explain-precisely-what-iowait-is)
26
+</details>
27
+
28
+### Troubleshooting Section
29
+
30
+<details>
31
+<summary>Check for main I/O related processes and hardware issues</summary>
32
+
33
+- Generally, this issue is caused by having slow hard drives that cannot keep up with the speed of your CPU. You can see
34
+ the percentage of `iowait` by going to your node on Netdata Cloud and clicking the `iowait` dimension under the Total
35
+ CPU Utilization chart.
36
+
37
+
38
+- You can use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds)
39
+
40
+```
41
+root@netdata~ # vmstat
42
+procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
43
+ r b swpd free buff cache si so bi bo in cs us sy id wa st
44
+ 8 0 1200384 168456 48840 1461540 4 14 65 51 334 196 3 1 95 0 0
45
+```
46
+
47
+The `procs` column, shows; \
48
+b: The number of processes blocked waiting for I/O to complete.
49
+
50
+After that, you can use `ps` and specifically `ps -eo s,user,cmd | grep ^[D]`.
51
+
52
+- The `grep` command will fetch the processes that their state code starts with `D` which means uninterruptible sleep
53
+ (usually IO).
54
+
55
+> It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
56
+processes you are closing and being certain that they are not necessary.
57
+
58
+- If you see that you don't have a lot of processes that you can terminate (or you need them for your workflow), then
59
+ you would have to upgrade your system’s drives; if you have an HDD, upgrading to an SSD or an NVME drive would make a
60
+ great impact on this metric.
61
+
62
+</details>
63
+
64
+<details>
65
+<summary>Check your database</summary>
66
+
67
+- As another example, in a database environment, you would want to optimize your operations. Check for potential inserts
68
+ on large data sets, keeping in mind that `write` operations take more time than `read`. You should also search for
69
+ complex requests, like large joins and queries over a big data set. These can introduce `iowait` and need to be
70
+ optimized.
71
+
72
+</details>
health/guides/cpu/10min_cpu_usage.md
new
+60
@@ -0,0 +1,60 @@
1
+# 10min_cpu_usage
2
+
3
+## OS: Linux & FreeBSD
4
+
5
+This alarm calculates an average on CPU utilization over a period of 10 minutes, **excluding** `iowait`, `nice`
6
+and `steal` values.
7
+
8
+> Note that on FreeBSD, the alert excludes only `nice`.
9
+
10
+The alarm is generally self-explanatory, but to clarify any confusion that might exist in the `iowait`, `nice`
11
+and `steal` metrics:
12
+
13
+`iowait` is the percentage of time the CPU waits on a disk for an I/O; it happens when the former is getting
14
+bottlenecked by the latter. At this point the CPU is being idle, waiting only on the I/O.
15
+
16
+`nice` value of a processor is the time it has spent on running low priority processes. Low priority processes are those
17
+with a 'nice' value greater than 0 (on UNIX-like systems, a higher ‘nice’ value indicates a lower priority).
18
+
19
+`steal`, in a virtual machine, is the percentage of time that particular virtual CPU has to wait for an available host
20
+CPU to run on. If this metric goes up, it means that your VM is not getting the processing power it needs.
21
+
22
+### Troubleshooting
23
+
24
+<details>
25
+<summary>Processes slowing down your CPU</summary>
26
+
27
+
28
+There are two primary cases in which this alarm is raised, and determining which applies to you requires
29
+understanding your own scenario.
30
+
31
+- Generally, if you have high CPU utilization alongside a high `nice` value, it means that the system is running through
32
+ all the low priority processes, and if some high priority process needs CPU time, it can get it at any time.
33
+- On the flipside, if you have high CPU utilization with low `nice` value, this means that the CPU is used on high
34
+ priority processes and new ones will not be able to take CPU time, and they will have to wait.
35
+
36
+The latter scenario is worth investigating if there is a process slowing down your CPU. We suggest you go to your node
37
+on Netdata Cloud and click the `nice` dimension under the `Total CPU Utilization` chart to see the value. You can then
38
+check per process CPU usage using `top`:
39
+
40
+<details>
41
+<summary>Linux</summary>
42
+
43
+```
44
+root@netdata~ # top -o +%CPU -i
45
+```
46
+</details>
47
+<details>
48
+<summary>FreeBSD</summary>
49
+
50
+```
51
+root@netdata~ # top -o cpu -I
52
+```
53
+</details>
54
+
55
+Here, you can see which processes are the main cpu consumers on the `CPU` column.
56
+
57
+
58
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
59
+processes you are closing and being certain that they are not necessary.
60
+</details>
health/guides/cpu/20min_steal_cpu.md
new
+25
@@ -0,0 +1,25 @@
1
+# 20min_steal_cpu
2
+
3
+## OS: Linux
4
+
5
+This alarm calculates average CPU `steal` time over the last 20 minutes
6
+
7
+`steal`, in a virtual machine, is the percentage of time that particular virtual CPU has to wait for an available host CPU
8
+to run on. If this metric goes up, it means that your VM is not getting the processing power it needs.
9
+
10
+### Troubleshooting section:
11
+
12
+<details>
13
+<summary>Check for CPU quota and host issues</summary>
14
+
15
+Generally, if `steal` is high, it could mean one of the following:
16
+
17
+- Another VM on the host system is hogging the CPU.
18
+- System services on the host system are monopolizing the CPU (for example, system updates).
19
+- The host CPUs are over-committed (you have more virtual CPUs assigned to VMs than the host system has physical CPUs) and too many VMs need CPU time simultanously.
20
+- The VM itself has a CPU quota that is too low.
21
+
22
+So in the end you can increase the CPU resources of that particular VM, and if the alert persists, move the guest to a
23
+different *physical* server.
24
+
25
+</details>
health/guides/disks/10min_disk_backlog.md
new
+16
@@ -0,0 +1,16 @@
1
+# 10min_disk_backlog
2
+
3
+## OS: Linux
4
+
5
+This alert presents the average backlog size of the disk raising this alarm over the last 10 minutes.
6
+
7
+- This alert is escalated to warning when the metric exceeds the size of 5000.
8
+
9
+<details>
10
+<summary>What is "disk backlog"?</summary>
11
+
12
+> Backlog is an indication of the duration of pending disk operations. On every I/O event the
13
+> system is multiplying the time spent doing I/O since the last update of this field with the
14
+> number of pending operations. While not accurate, this metric can provide an indication of the
15
+> expected completion time of the operations in progress.
16
+</details>
health/guides/disks/10min_disk_utilization.md
new
+57
@@ -0,0 +1,57 @@
1
+# 10min_disk_utilization
2
+
3
+## OS: Linux, FreeBSD
4
+
5
+This alert presents the average percentage of time the disk was busy over the
6
+last 10 minutes.
7
+If you receive this it indicates high disk load and that the disk spent most of the time servicing
8
+read or write requests.
9
+
10
+- This alert is triggered in a warning state when the metric exceeds 98%.
11
+
12
+This metric is the same as the %util column on the command `iostat -x`:
13
+
14
+> %util is the percentage of the time the drive was doing at least one thing.
15
+> Device saturation occurs when this value is close to 100% for devices serving requests serially.
16
+> But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number
17
+> does not reflect their performance limits.
18
+> As a measure of general IO busyness %util is fairly handy, but as an indication of how much the
19
+> system is doing compared to what it can do, it's terrible.<sup>[1](
20
+> https://brooker.co.za/blog/2014/07/04/iostat-pct.html) </sup>
21
+
22
+
23
+
24
+<details>
25
+<summary>References and Sources</summary>
26
+
27
+1. [Two traps in iostat: %util and svctm](https://brooker.co.za/blog/2014/07/04/iostat-pct.html)
28
+
29
+</details>
30
+
31
+### Troubleshooting Section
32
+
33
+#### Check per-process disk usage to find the top consumers
34
+
35
+> Note: If you got this alert for a device serving requests in parallel, you can ignore it.
36
+
37
+<details><summary>Use `iotop` on Linux</summary>
38
+
39
+ `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it,
40
+ then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
41
+ ```
42
+ root@netdata~ # sudo iotop
43
+ ```
44
+ Using this, you can see which processes are the main Disk I/O consumers on the `IO` column.
45
+
46
+</details>
47
+
48
+<details><summary>Use `top` on FreeBSD</summary>
49
+
50
+You can use `top`:
51
+ ```
52
+ root@netdata~ # top -m io -o total
53
+ ```
54
+ The `-m io` sets `top` to display I/O statistics, and the `-o total` indicates the results will be
55
+ ordered according to the field "Total".
56
+
57
+</details>
health/guides/disks/disk_inode_usage.md
new
+35
@@ -0,0 +1,35 @@
1
+# disk_inode_usage
2
+
3
+## OS: Linux, FreeBSD
4
+
5
+This alarm presents the percentage of used `inodes` storage of a particular disk.
6
+
7
+> The number of `inodes` indicates the number of files and folders you have. An `inode` is a data structure,
8
+> containing metadata about a file. All filenames are internally mapped to respective `inode` numbers, so if you have a
9
+> lot of files, it means there are a lot of `inodes`.
10
+
11
+If the alarm is raised, it means that your storage device is running out of `inode` space. Each disk has a particular
12
+**limitation on the amount of `inodes` it can store**, determined by its size.
13
+
14
+> Many modern filesystems use dynamically allocated `inodes` instead of a static table. These should not be presented
15
+> on the charts associated with this alarm, and should not ever trigger it. If such a filesystem ***does***
16
+> trigger this alarm, and it's constantly reporting max `inode` usage, it's probably a bug in the filesystem driver. \
17
+> Some such filesystems incorrectly report having max `inode` count when they should not because they have no max
18
+> limit, and in turn they trigger a false positive alarm.
19
+
20
+Links: \
21
+[Linux Inodes](https://www.javatpoint.com/linux-inodes) \
22
+[Understanding UNIX / Linux filesystem Inodes](
23
+https://www.cyberciti.biz/tips/understanding-unixlinux-filesystem-inodes.html)
24
+
25
+### Troubleshoot
26
+
27
+<details>
28
+<summary>Clear cache files or delete unnecessary files and folders</summary>
29
+
30
+- To reduce the amount of how many `inodes` you store currently, you can clear your cache, trash any unnecessary files
31
+ and folders in your system.
32
+
33
+> We strongly suggest that you practice a high degree of caution when cleaning up drives, and removing files, make sure
34
+> that you are certain that you delete only unnecessary files.
35
+</details>
\ No newline at end of file
health/guides/disks/disk_space_usage.md
new
+30
@@ -0,0 +1,30 @@
1
+# disk_space_usage
2
+
3
+## OS: Linux, FreeBSD
4
+
5
+This alarm presents the percentage of used space of a particular disk. If it is close to 100%, it means that your storage
6
+device is running out of space. If the particular disk raising the alarm is full, the system could experience slowdowns
7
+and even crashes.
8
+
9
+### Troubleshoot
10
+
11
+<details>
12
+<summary>Clean or upgrade the drive</summary>
13
+
14
+If your storage device is full and the alert is raised, there are two paths you can tend to:
15
+
16
+- Cleanup your drive, remove any unnecessary files (files on the trash directory, cache files etc.) to free up space. \
17
+Some areas that are safe to delete, are:
18
+ - Files under `/var/cache`
19
+ - Old logs in `/var/log`
20
+ - Old crash reports in `/var/crash` or `/var/dump`
21
+ - The `.cache` directory in user home directories
22
+
23
+
24
+- If your workflow requires all the space that is currently used, then you might want to look into upgrading the disk
25
+ that raised the alarm, because its capacity is small for your demands.
26
+
27
+> Netdata strongly suggests that you are careful when cleaning up drives, and removing files, make sure
28
+> that you are certain that you delete only unnecessary files.
29
+
30
+</details>
health/guides/dns_query/dns_query_time_query_time.md
new
+89
@@ -0,0 +1,89 @@
1
+# dns_query_time_query_time
2
+
3
+## DNS
4
+
5
+This alert presents the average DNS query round trip time (RTT) over the last 10 seconds.
6
+
7
+If the DNS query exceeds a time limit to complete its operation (aka if it times out), then the
8
+alert is raised into warning.
9
+
10
+<details><summary>What is Round Trip Time?</summary>
11
+
12
+> In networking, round-trip time (RTT), also known as round-trip delay time (RTD) is defined as
13
+> a metric that measures in milliseconds the amount of time it takes for a data packet to be
14
+> sent plus the amount of time it takes for acknowledgement of that signal to be received. This
15
+> time delay includes propagation times for the paths between the two communication endpoints.
16
+> <sup>[1](https://www.stormit.cloud/post/what-is-round-trip-time-rtt-meaning-calculation)
17
+
18
+</details>
19
+
20
+<details><summary>What is the main cause of DNS Latency?</summary>
21
+
22
+- Cache misses
23
+ Even if a resolver can provide very good cache hit latency, cache misses are unavoidable and are
24
+ very costly in terms of latency.
25
+
26
+ "Cache hits" is the terminology used for when a system asks a resolver about some data and the
27
+ resolver can provide it because he has it cached locally.
28
+
29
+ "Cache misses" occur when a system asks a resolver about some data, and he doesn't have it cached
30
+ locally. Then the resolver has to talk to other name servers, to see if they have the data
31
+ requested, which takes time and greatly increases latency.<sup>[2](
32
+ https://developers.google.com/speed/public-dns/docs/performance#introduction_causes_and_mitigations_of_dns_latency) </sup>
33
+
34
+
35
+- DNS Server Location
36
+ > The location of the DNS server you're accessing plays a huge role in your latency. The
37
+ > farther the server is to your place, the higher the latency gets. But this is not always
38
+ > the case as centralized DNS servers' latency isn't affected by the distance from the user.
39
+ > Transit links also vary from one server to another. Latency will be lower if the transit
40
+ > links are equipped with up-to-date technology.<sup>[3](
41
+ https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx) </sup>
42
+
43
+
44
+- Wireless networks
45
+ > Wireless networks have higher latency compared to wired networks. This happens because the
46
+ > transfer of data doesn't go through fixed lines. Instead, it goes through Wi-Fi routers or
47
+ > satellite dishes. These devices' efficiency also depends on the location where they're placed.<sup>[3](
48
+ https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx) </sup>
49
+
50
+
51
+- Malicious DNS Traffic
52
+ > Malicious DNS traffic can also cause high latency. It's because the DNS server will work
53
+ > double time in processing it. PRSD attacks are the most common type of malicious traffic.
54
+ > When this happens, it causes a lot of malware and botnet queries which cause high recursion
55
+ > rates. These consume and waste a lot of CPU cycles on the server.<sup>[3](
56
+ https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx) </sup>
57
+
58
+
59
+- Under-scaling of DNS Server
60
+ > Proper scaling of the DNS infrastructure is important. Because if it’s not scaled correctly,
61
+ > chances are is that it will use too much CPU power. When this happens, it will impact the
62
+ > latency and cause it to increase. The more the CPU is utilized, the higher your latency gets.<sup>[3](
63
+ https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx) </sup>
64
+
65
+</details>
66
+
67
+For further information, please refer to our *References and Sources* section.
68
+
69
+<details><summary>References and Sources</summary>
70
+
71
+1. [What is Round-Trip Time (RTT)?](
72
+ https://www.stormit.cloud/post/what-is-round-trip-time-rtt-meaning-calculation)
73
+2. [Causes and mitigations of DNS latency](
74
+ https://developers.google.com/speed/public-dns/docs/performance#introduction_causes_and_mitigations_of_dns_latency)
75
+3. [What is DNS Latency and Why Should You Care?](
76
+ https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx)
77
+4. [Configure your network settings to use Google Public DNS](https://developers.google.com/speed/public-dns/docs/using)
78
+
79
+</details>
80
+
81
+### Troubleshooting Section
82
+
83
+This alert can have multiple causes.
84
+
85
+As a first step, you can try changing your DNS server. Your current configuration might be using a
86
+slow server, or what your ISP provides might not be the best. You can find more information on
87
+[Google Developers](https://developers.google.com/speed/public-dns/docs/using) on how to configure
88
+your settings for your specific OS to use Google's Public DNS.
89
+
health/guides/docker/docker_container_unhealthy.md
new
+61
@@ -0,0 +1,61 @@
1
+# docker_containers_unhealthy
2
+
3
+**Containers | Docker**
4
+
5
+_Docker is an open source containerization platform. It enables developers to package applications
6
+into containers—standardized executable components combining application source code with the
7
+operating system (OS) libraries and dependencies required to run that code in any environment_
8
+
9
+Sometimes while our container is running, the application inside may have crashed. To foresee those
10
+events, container runtimes (CR) and orchestrators perform health checks to endpoints inside the
11
+functional units of the container. A container marked as unhealthy by the CR, is malfunctioning and
12
+should be stopped. Those health checks are defined by the creator of the container with the
13
+HEALTHCHECK
14
+instructions. <sup>[1](https://docs.docker.com/engine/reference/builder/#healthcheck) </sup>
15
+
16
+The Netdata Agent monitors the average number of unhealthy docker containers over the last 10
17
+seconds. This alert indicates that some containers are not running due to failed health checks.
18
+
19
+This alert is raised into warning when at least one container is unhealthy in your Docker engine.
20
+
21
+<details>
22
+<summary>References and sources</summary>
23
+
24
+1. [HEALTHCHECK instruction in Docker docs](https://docs.docker.com/engine/reference/builder/#healthcheck)
25
+
26
+</details>
27
+
28
+### Troubleshooting section
29
+
30
+<details>
31
+<summary>Inspect and restart the UNHEALTY container</summary>
32
+
33
+1. Check all the containers in the system.
34
+
35
+ ```
36
+ root@netdata # docker ps -a
37
+ ```
38
+
39
+2. Find the NAME of the container that is marked as UNHEALTHY.
40
+
41
+3. Check the logs of this container to get some insights into what's going wrong
42
+
43
+ ```
44
+ root@netdata # docker logs <UNHEALTHY_CONTAINER>
45
+ ```
46
+ In many cases, your app's logs may not appear in docker log collector. A simple workaround is
47
+ something like
48
+ this, [redirect your apps's logs into stderr](https://github.com/nginxinc/docker-nginx/blob/master/Dockerfile-debian.template#L90)
49
+ . Use this workaround purposefully. Another workaround is to redirect any log attempt to log
50
+ directly into the `/proc/self/fd/2`.
51
+
52
+
53
+4. Restart the container and see if this fixes the problem.
54
+
55
+ ```
56
+ root@netdata # docker logs <UNHEALTHY_CONTAINER>
57
+ ```
58
+
59
+5. If you receive this alert often, you may have to do further investigation on why this event occurs
60
+
61
+</details>
health/guides/entropy/lowest_entropy.md
new
+141
@@ -0,0 +1,141 @@
1
+# lowest_entropy
2
+
3
+## OS: Linux
4
+
5
+This alert presents the minimum amount of entropy in the kernel entropy pool in the last 5 minutes.
6
+
7
+Low entropy can lead to a reduction in the quality of random numbers produced by `/dev/random`
8
+and `/dev/urandom`.
9
+
10
+The Netdata Agent checks for the minimum entropy value in the last 5 minutes. The alert gets raised
11
+into warning if the value < 100, and cleared if the value > 200.
12
+
13
+For further information on how our alerts are calculated, please have a look at our [Documentation](
14
+https://learn.netdata.cloud/docs/agent/health/reference#expressions).
15
+
16
+
17
+<details>
18
+<summary>What is entropy and why do we need it?</summary>
19
+
20
+Entropy is similar to "randomness". A Linux system gathers "real" random numbers by keeping an eye
21
+on different events: network activity, hard drive rotation speeds, hardware random number
22
+generator (if available), key-clicks, and so on. It feeds those to the kernel entropy pool, which is
23
+used by `/dev/random`.<sup>[1](
24
+https://unixhealthcheck.com/blog?id=472) </sup>
25
+
26
+Encryption and cryptography applications require random numbers to operate. A function or an
27
+algorithm that produces numbers -*that seem to be random*- is very predictable, if you know what
28
+function is used.
29
+
30
+In real life, we use our surroundings and our thoughts to produce truly random numbers. A computer
31
+can't really do this by itself, so it gathers numbers from a lot of sources. For example, it can get
32
+the CO<sub>2</sub> levels in a room from a sensor on the system and use that as a random number.
33
+
34
+This way all the values are random and there is no pattern to be found among them.
35
+</details>
36
+
37
+For further information, please have a look at the _References and Sources_ section.
38
+
39
+<details>
40
+<summary>References and Sources</summary>
41
+
42
+1. [Entropy](https://unixhealthcheck.com/blog?id=472)
43
+2. [rng-tools](https://github.com/nhorman/rng-tools)
44
+3. [How to add more entropy to improve cryptographic randomness on Linux](
45
+ https://www.techrepublic.com/article/how-to-add-more-entropy-to-improve-cryptographic-randomness-on-linux/)
46
+4. [Haveged Installation - Archlinux Wiki](https://wiki.archlinux.org/title/Haveged#Installation)
47
+
48
+</details>
49
+
50
+### Troubleshooting Section
51
+
52
+The best tool to troubleshoot the lowest entropy alert is with `rng-tools`. If `rng-tools` are not
53
+available for your platform, or you run into trouble, you can use the tool `haveged` as an
54
+alternative.
55
+
56
+<details>
57
+<summary>Install and setup rng-tools</summary>
58
+
59
+`rng-tools` is a random number generator daemon.
60
+It monitors a set of entropy sources, and supplies entropy from them to the system kernel's
61
+/dev/random machinery.<sup>[2](https://github.com/nhorman/rng-tools) </sup>
62
+
63
+### Installation
64
+
65
+### Debian-based platforms
66
+
67
+```
68
+root@netdata~ # sudo apt-get update
69
+root@netdata~ # sudo apt-get install rng-tools
70
+```
71
+
72
+### RHEL/Fedora/CentOS machines
73
+
74
+1. Change to the root account;
75
+
76
+```
77
+root@netdata~ # su
78
+```
79
+
80
+2. And then install;
81
+
82
+```
83
+root@netdata~ # yum install rng-tools
84
+```
85
+
86
+### After the Installation
87
+
88
+You can run the service using the following command;
89
+
90
+```
91
+root@netdata~ # service rngd start
92
+```
93
+
94
+And also you can check the daemon status using the following command;
95
+
96
+```
97
+root@netdata~ # service rngd status
98
+```
99
+
100
+</details>
101
+
102
+
103
+
104
+<details><summary>Install Haveged</summary>
105
+
106
+Ideally, a system with high entropy demands should have a hardware device to generate random
107
+numbers. For example, a TPM is such a device. However, there are also several software-only options
108
+you may install, like `haveged` [(*read more*)](
109
+https://wiki.archlinux.org/title/Haveged#Installation).
110
+
111
+### Installation
112
+
113
+### Debian-based platforms
114
+
115
+1. To install `haveged`, run:
116
+
117
+ ```
118
+ root@netdata~ # sudo apt-get install haveged
119
+ ```
120
+
121
+2. Set `haveged` up to start at boot with the command `sudo update-rc.d haveged defaults`.<sup>[3](
122
+ https://www.techrepublic.com/article/how-to-add-more-entropy-to-improve-cryptographic-randomness-on-linux/) </sup>
123
+
124
+### RHEL/Fedora/CentOS machines
125
+
126
+1. Change to the root account:
127
+
128
+ ```
129
+ root@netdata~ # su
130
+ ```
131
+
132
+2. Install `haveged`:
133
+
134
+ ```
135
+ root@netdata~ # yum install haveged
136
+ ```
137
+
138
+3. Set `haveged` to start at boot with the command `chkconfig haveged on`.<sup>[3](
139
+ https://www.techrepublic.com/article/how-to-add-more-entropy-to-improve-cryptographic-randomness-on-linux/) </sup>
140
+
141
+</details>
health/guides/haproxy/haproxy_backend_server_status.md
new
+85
@@ -0,0 +1,85 @@
1
+# haproxy_backend_server_status
2
+
3
+**Web Proxy | HAProxy**
4
+
5
+HAProxy is a free, fast and reliable reverse-proxy offering high availability, load balancing,
6
+and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic
7
+web sites and powers a significant portion of the world's most visited ones. Over the years it has
8
+become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux
9
+distributions, and is often deployed by default in cloud platforms.
10
+
11
+The Netdata Agent monitors the average number of failed HAProxy backend servers over the last 10
12
+seconds. Receiving this alert (in critical state) means that one or more HAProxy backend servers are
13
+inaccessible or offline.
14
+
15
+_There are four essential sections to an HAProxy configuration file. They are global, defaults,
16
+frontend, and backend. These four sections define how the server as a whole performs, what your
17
+default settings are, and how client requests are received and routed to your backend
18
+servers._ <sup> [1](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/) </sup>
19
+
20
+<details>
21
+<summary>HA Proxy Backend Servers</summary>
22
+
23
+Backend servers are the cornerstone of the HA proxy architecture. HA proxy organizes multiple
24
+servers to `Backends` (a pool of servers) and implements different (defined by you) Layer 4 or Layer
25
+7 load balancing algorithms to assign the incoming requests to each individual server.
26
+
27
+> You can define a new server with the `server` setting or use the `default-server` configuration
28
+which is configured once. Its first argument is a name, followed by the IP address and port of the
29
+backend server. You can specify a domain name instead of an IP address. In that case, it will be
30
+resolved at startup or, if you add a `resolvers` argument, it will be updated during runtime. If the
31
+DNS entry contains an SRV record, the port and weight will be filled in from it too. If the port
32
+isn’t specified, then HAProxy will use the same port that the client connected on, which is useful
33
+for randomly used ports such as for active-mode FTP.
34
+>
35
+> Every `server` line should have a `maxconn` setting that limits the maximum number of concurrent
36
+requests that the server will be given. Even if it’s just a guess, having a value here puts you on
37
+the right foot for avoiding saturating your servers with requests and gives a baseline that can be
38
+adjusted later. <sup> [1](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/) </sup>
39
+
40
+</details>
41
+
42
+<details>
43
+<summary>References and sources</summary>
44
+
45
+1. [The Four Essential Sections of an HAProxy Configuration](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/)
46
+
47
+</details>
48
+
49
+### Troubleshooting section
50
+
51
+<details>
52
+<summary>Check the HA proxy's configuration file for errors</summary>
53
+
54
+Making changes in the configuration file may introduce errors. Make sure your always validate the
55
+correctness of the configuration file.
56
+
57
+1. In most Linux distros you can run the following check:
58
+
59
+```
60
+root@netadata # haproxy -c -f /etc/haproxy/haproxy.cfg
61
+```
62
+</details>
63
+
64
+<details>
65
+<summary>Check the HA proxy service for errors</summary>
66
+
67
+1. Use `journalctl` and inspect the log:
68
+
69
+```
70
+root@netdata # journalctl -u haproxy.service --reverse
71
+```
72
+</details>
73
+
74
+<details>
75
+<summary>Check the HA proxy's log</summary>
76
+
77
+1. By default HA proxy logs under `/var/log/haproxy.log`:
78
+
79
+```
80
+root@netdata # cat /var/log/haproxy.log | grep 'emerg\|alert\|crit\|err\|warning\|notice'
81
+```
82
+
83
+You can also search for log messages with `info` and `debug` tags.
84
+
85
+</details>
health/guides/haproxy/haproxy_backend_status.md
new
+80
@@ -0,0 +1,80 @@
1
+# haproxy_backend_status
2
+
3
+**Web Proxy | HAProxy**
4
+
5
+HAProxy is a free, very fast and reliable reverse-proxy offering high availability, load balancing,
6
+and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic
7
+web sites and powers a significant portion of the world's most visited ones. Over the years it has
8
+become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux
9
+distributions, and is often deployed by default in cloud platforms.
10
+
11
+The Netdata Agent monitors the average number of failed HAProxy backends over the last 10 seconds.
12
+Receiving this alert (in critical state) means that one or more HAProxy backend are inaccessible or
13
+offline.<sup> [1](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/) </sup>
14
+
15
+<details>
16
+<summary>HA Proxy Backends</summary>
17
+
18
+> A HA proxy `backend` is a set of servers that receives forwarded requests. Backends are defined in
19
+the backend section of the HAProxy configuration. In its most basic form, a backend can be defined by:
20
+>
21
+> - which load balance algorithm to use
22
+>
23
+> - a list of servers and ports
24
+>
25
+> A backend can contain one or many servers in it–generally speaking, adding more servers to your
26
+backend will increase your potential load capacity by spreading the load over multiple servers.
27
+Increase reliability is also achieved through this manner, in case some of your backend servers
28
+become unavailable. <sup>[2](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts) </sup>
29
+
30
+</details>
31
+
32
+<details>
33
+<summary>References and Sources</summary>
34
+
35
+1. [The Four Essential Sections of an HAProxy Configuration](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/)
36
+
37
+2. [HA proxy explained in DigitalOcean](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts)
38
+
39
+</details>
40
+
41
+### Troubleshooting Section
42
+
43
+<details>
44
+<summary>Check the HA proxy's configuration file for errors</summary>
45
+
46
+Making changes in the configuration file may introduce errors. Make sure your always validate the
47
+correctness of the configuration file.
48
+
49
+1. In most Linux distros you can run the following check:
50
+
51
+```
52
+root@netadata # haproxy -c -f /etc/haproxy/haproxy.cfg
53
+```
54
+
55
+</details>
56
+
57
+
58
+<details>
59
+<summary>Check the HA proxy service for errors</summary>
60
+
61
+1. Use `journalctl` and inspect the log:
62
+
63
+```
64
+root@netdata # journalctl -u haproxy.service --reverse
65
+```
66
+
67
+</details>
68
+
69
+<details>
70
+<summary>Check the HA proxy's log</summary>
71
+
72
+1. By default HA proxy logs under `/var/log/haproxy.log`:
73
+
74
+```
75
+root@netdata # cat /var/log/haproxy.log | grep 'emerg\|alert\|crit\|err\|warning\|notice'
76
+```
77
+
78
+You can also search for log messages with `info` and `debug` tags.
79
+
80
+</details>
health/guides/hdfs/hdfs_capacity_usage.md
new
+139
@@ -0,0 +1,139 @@
1
+# hdfs_capacity_usage
2
+
3
+**Storage | HDFS**
4
+
5
+_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
+written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
+lack of POSIX compliance, but it does provide shell commands and Java application programming
8
+interface (API) methods that are similar to other file
9
+systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
+
11
+The Netdata Agent calculates the percentage of used space capacity across all DataNodes. Receiving
12
+this alert indicates high DataNodes space capacity utilization.
13
+
14
+This alert is triggered into warning when the percentage of used space capacity across all DataNodes
15
+is between 70-80% and in critical when it is between 80-90%.
16
+
17
+<details>
18
+<summary>See more about Hadoop</summary>
19
+
20
+Wikipedia provides a great explanation of
21
+HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
22
+
23
+HDFS provides a software framework for distributed storage and processing of big data using the
24
+`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
25
+processing data. It achieves reliability by replicating the data across multiple hosts, and hence
26
+theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
27
+the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
28
+different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
29
+keep the replication of data high.
30
+
31
+HDFS has five services as follows:
32
+
33
+1. Name Node
34
+2. Secondary Name Node
35
+3. Job tracker
36
+4. Data Node
37
+5. Task Tracker
38
+
39
+Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
40
+communicate with each other and in the same way slave services can communicate with each other.
41
+NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
42
+each other.
43
+
44
+- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
45
+ track files, manage the file system and has the metadata of all the stored data within it. Some
46
+ information the NameNode keep track of are:
47
+
48
+ - details (metadata) of blocks
49
+ - in which DataNode each block lives, and its location
50
+ - replication metadata of each block
51
+
52
+ The NameNode is the gateway that a client uses to manage the HDFS cluster.
53
+
54
+- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
55
+ stores the actual data into HDFS which is responsible for the client to read and write. These are
56
+ slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
57
+ conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
58
+ for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
59
+ some other DataNode.
60
+
61
+- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
62
+ is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
63
+ NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
64
+ `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
65
+
66
+- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
67
+ tracker talks to the NameNode to know about the location of the data that will be used in
68
+ processing. The NameNode responds with the metadata of the required processing data.
69
+
70
+- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
71
+ Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
72
+ the file. The process of applying that code on the file is known as Mapper.
73
+
74
+Some more useful information/concepts about HDFS from the official
75
+website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
76
+
77
+- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
78
+ an application can create directories and store files inside these directories. The file system
79
+ namespace hierarchy is similar to most other existing file systems; one can create and remove
80
+ files, move a file from one directory to another, or rename a file. HDFS does not yet implement
81
+ user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
82
+ not preclude implementing these features.
83
+
84
+ The NameNode maintains the file system namespace. Any change to the file system namespace or its
85
+ properties is recorded by the NameNode. An application can specify the number of replicas of a
86
+ file that should be maintained by HDFS. The number of copies of a file is called the replication
87
+ factor of that file. This information is stored by the NameNode.
88
+
89
+- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
90
+ HDFS are those that deal with large data sets. These applications write their data only once but
91
+ they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
92
+ supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
93
+ Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
94
+ different DataNode.
95
+
96
+- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
97
+ might automatically move data from one DataNode to another if the free space on a DataNode falls
98
+ below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
99
+ might dynamically create additional replicas and rebalance other data in the cluster. These types
100
+ of data rebalancing schemes are not yet implemented.
101
+
102
+</details>
103
+
104
+<details>
105
+<summary>References and sources</summary>
106
+
107
+1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
108
+2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
109
+
110
+</details>
111
+
112
+### Troubleshooting section
113
+
114
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
115
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
116
+
117
+<details>
118
+<summary>Check your Disk Usage across the cluster </summary>
119
+
120
+1. Inspect the Disk Usage for each DataNode
121
+
122
+ ```
123
+ root@netdata # hadoop dfsadmin -report
124
+ ```
125
+
126
+ If all the DataNodes are in Disk pressure, you should consider adding more disk space. Otherwise,
127
+ you can perform a balance of data between the DataNodes.
128
+
129
+2. Perform a balance
130
+
131
+ ```
132
+ root@netdata # hdfs balancer –threshold 15
133
+ ```
134
+
135
+ This means that the balancer will balance data by moving blocks from over-utilized to
136
+ under-utilized nodes, until each DataNode’s disk usage differs by no more than plus or minus 15
137
+ percent.
138
+
139
+</details>
health/guides/hdfs/hdfs_dead_nodes.md
new
+138
@@ -0,0 +1,138 @@
1
+# hdfs_dead_nodes
2
+
3
+**Storage | HDFS**
4
+
5
+_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
+written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
+lack of POSIX compliance, but it does provide shell commands and Java application programming
8
+interface (API) methods that are similar to other file
9
+systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
+
11
+The Netdata Agent monitors the number of DataNodes that are currently dead. Receiving this alert
12
+indicates that there are dead DataNodes in your HDFS cluster. The NameNode characterizes a DataNode
13
+as dead if no heartbeat message is exchanged for approximately 10 minutes. Any data that was
14
+registered to a dead DataNode is not available to HDFS anymore.
15
+
16
+This alert is triggered into critical when the number of dead DataNodes is 1 or more.
17
+
18
+<details>
19
+<summary>See more about Hadoop</summary>
20
+
21
+Wikipedia provides a great explanation of
22
+HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
23
+
24
+HDFS provides a software framework for distributed storage and processing of big data using the
25
+`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
26
+processing data. It achieves reliability by replicating the data across multiple hosts, and hence
27
+theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
28
+the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
29
+different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
30
+keep the replication of data high.
31
+
32
+HDFS has five services as follows:
33
+
34
+1. Name Node
35
+2. Secondary Name Node
36
+3. Job tracker
37
+4. Data Node
38
+5. Task Tracker
39
+
40
+Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
41
+communicate with each other and in the same way slave services can communicate with each other.
42
+NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
43
+each other.
44
+
45
+- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
46
+ track files, manage the file system and has the metadata of all the stored data within it. Some
47
+ information the NameNode keep track of are:
48
+
49
+ - details (metadata) of blocks
50
+ - in which DataNode each block lives, and its location
51
+ - replication metadata of each block
52
+
53
+ The NameNode is the gateway that a client uses to manage the HDFS cluster.
54
+
55
+- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
56
+ stores the actual data into HDFS which is responsible for the client to read and write. These are
57
+ slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
58
+ conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
59
+ for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
60
+ some other DataNode.
61
+
62
+- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
63
+ is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
64
+ NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
65
+ `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
66
+
67
+- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
68
+ tracker talks to the NameNode to know about the location of the data that will be used in
69
+ processing. The NameNode responds with the metadata of the required processing data.
70
+
71
+- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
72
+ Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
73
+ the file. The process of applying that code on the file is known as Mapper.
74
+
75
+Some more useful information/concepts about HDFS from the official
76
+website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
77
+
78
+- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
79
+ an application can create directories and store files inside these directories. The file system
80
+ namespace hierarchy is similar to most other existing file systems; one can create and remove
81
+ files, move a file from one directory to another, or rename a file. HDFS does not yet implement
82
+ user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
83
+ not preclude implementing these features.
84
+
85
+ The NameNode maintains the file system namespace. Any change to the file system namespace or its
86
+ properties is recorded by the NameNode. An application can specify the number of replicas of a
87
+ file that should be maintained by HDFS. The number of copies of a file is called the replication
88
+ factor of that file. This information is stored by the NameNode.
89
+
90
+- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
91
+ HDFS are those that deal with large data sets. These applications write their data only once but
92
+ they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
93
+ supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
94
+ Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
95
+ different DataNode.
96
+
97
+- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
98
+ might automatically move data from one DataNode to another if the free space on a DataNode falls
99
+ below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
100
+ might dynamically create additional replicas and rebalance other data in the cluster. These types
101
+ of data rebalancing schemes are not yet implemented.
102
+
103
+</details>
104
+
105
+<details>
106
+<summary>References and sources</summary>
107
+
108
+1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
109
+2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
110
+
111
+</details>
112
+
113
+### Troubleshooting section
114
+
115
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
116
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
117
+
118
+<details>
119
+<summary>Fix corrupted or missing blocks</summary>
120
+
121
+1. Identify the dead node(s)
122
+
123
+ ```
124
+ root@netdata # hadoop dfsadmin -report
125
+ ```
126
+
127
+Inspect the output and check which DataNode is dead.
128
+
129
+2. Connect to the DataNode and check the log of the DataNode. You can also check for errors in the
130
+ system services.
131
+
132
+ ```
133
+ root@netdata # systemctl status hadoop
134
+ ```
135
+
136
+ Restart the service if needed.
137
+
138
+</details>
health/guides/hdfs/hdfs_missing_blocks.md
new
+156
@@ -0,0 +1,156 @@
1
+# hdfs_missing_blocks
2
+
3
+**Storage | HDFS**
4
+
5
+_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
+written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
+lack of POSIX compliance, but it does provide shell commands and Java application programming
8
+interface (API) methods that are similar to other file
9
+systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
+
11
+The Netdata Agent monitors the number of missing blocks in a HDFS. Receiving this alert into warning
12
+indicates that there is at least one missing block in one of the DataNodes. This issue could be
13
+caused by a problem with the underlying storage or filesystem of a DataNode.
14
+
15
+<details>
16
+<summary>See more about Hadoop</summary>
17
+
18
+Wikipedia provides a great explanation of
19
+HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
20
+
21
+HDFS provides a software framework for distributed storage and processing of big data using the
22
+`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
23
+processing data. It achieves reliability by replicating the data across multiple hosts, and hence
24
+theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
25
+the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
26
+different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
27
+keep the replication of data high.
28
+
29
+HDFS has five services as follows:
30
+
31
+1. Name Node
32
+2. Secondary Name Node
33
+3. Job tracker
34
+4. Data Node
35
+5. Task Tracker
36
+
37
+Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
38
+communicate with each other and in the same way slave services can communicate with each other.
39
+NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
40
+each other.
41
+
42
+- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
43
+ track files, manage the file system and has the metadata of all the stored data within it. Some
44
+ information the NameNode keep track of are:
45
+
46
+ - details (metadata) of blocks
47
+ - in which DataNode each block lives, and its location
48
+ - replication metadata of each block
49
+
50
+ The NameNode is the gateway that a client uses to manage the HDFS cluster.
51
+
52
+- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
53
+ stores the actual data into HDFS which is responsible for the client to read and write. These are
54
+ slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
55
+ conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
56
+ for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
57
+ some other DataNode.
58
+
59
+- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
60
+ is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
61
+ NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
62
+ `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
63
+
64
+- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
65
+ tracker talks to the NameNode to know about the location of the data that will be used in
66
+ processing. The NameNode responds with the metadata of the required processing data.
67
+
68
+- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
69
+ Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
70
+ the file. The process of applying that code on the file is known as Mapper.
71
+
72
+Some more useful information/concepts about HDFS from the official
73
+website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
74
+
75
+- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
76
+ an application can create directories and store files inside these directories. The file system
77
+ namespace hierarchy is similar to most other existing file systems; one can create and remove
78
+ files, move a file from one directory to another, or rename a file. HDFS does not yet implement
79
+ user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
80
+ not preclude implementing these features.
81
+
82
+ The NameNode maintains the file system namespace. Any change to the file system namespace or its
83
+ properties is recorded by the NameNode. An application can specify the number of replicas of a
84
+ file that should be maintained by HDFS. The number of copies of a file is called the replication
85
+ factor of that file. This information is stored by the NameNode.
86
+
87
+- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
88
+ HDFS are those that deal with large data sets. These applications write their data only once but
89
+ they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
90
+ supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
91
+ Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
92
+ different DataNode.
93
+
94
+- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
95
+ might automatically move data from one DataNode to another if the free space on a DataNode falls
96
+ below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
97
+ might dynamically create additional replicas and rebalance other data in the cluster. These types
98
+ of data rebalancing schemes are not yet implemented.
99
+
100
+</details>
101
+
102
+<details>
103
+<summary>References and sources</summary>
104
+
105
+1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
106
+2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
107
+3. [Man pages of fsck](https://linux.die.net/man/8/fsck)
108
+
109
+</details>
110
+
111
+### Troubleshooting section
112
+
113
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
114
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
115
+
116
+<details>
117
+<summary>Fix corrupted or missing blocks</summary>
118
+
119
+1. Identify which files are facing issues.
120
+
121
+ ```
122
+ root@netdata # hdfs fsck -list-corruptfileblocks
123
+ ```
124
+
125
+ Inspect the output and track the path(s) to the corrupted files
126
+
127
+
128
+2. Determine where the file's blocks might live. If the file is larger than your block size it
129
+ consists of multiple blocks.
130
+
131
+ ```
132
+ root@netdata # hdfs fsck <path_to_corrupted_file> -locations -blocks -files
133
+ ```
134
+
135
+ This command will print out locations for every "problematic" block.
136
+
137
+
138
+3. Search in the corresponding DataNodes and the NameNode's logs for the machine or machines on
139
+ which the blocks lived. Try looking for filesystem errors on those machines.
140
+ Use `fsck` <sup>[3](https://linux.die.net/man/8/fsck) </sup>.
141
+
142
+
143
+4. If there are files or blocks that you cannot fix, you must delete them so that the HDFS becomes
144
+ healthy again.
145
+
146
+ - for a specific file
147
+ ```
148
+ root@netdata # hdfs fs -rm <path_to_file_with_unrecovable_blocks>
149
+ ```
150
+ OR
151
+ - for all the "problematic" files
152
+ ```
153
+ hdfs fsck / -delete
154
+ ```
155
+
156
+</details>
health/guides/hdfs/hdfs_num_failed_volumes.md
new
+148
@@ -0,0 +1,148 @@
1
+# hdfs_num_failed_volumes
2
+
3
+**Storage | HDFS**
4
+
5
+_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
+written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
+lack of POSIX compliance, but it does provide shell commands and Java application programming
8
+interface (API) methods that are similar to other file
9
+systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
+The Netdata Agent monitors the number of failed volumes in your HDFS cluster. Receiving this alert
11
+into warning indicates that at least one volume has failed due to a hardware failure or
12
+misconfiguration, for example duplicate mount.
13
+
14
+In an HDFS cluster, a single volume failing on a DataNode will cause the entire node to go offline (
15
+to be more precise it depends on the `dfs.datanode.failed.volumes.tolerated` setting for your
16
+cluster). In case of a volume fail incident, the NameNode must copy any under-replicated blocks that
17
+were lost on that node, causing a burst in network traffic and potential performance degradation.
18
+
19
+
20
+<details>
21
+<summary>See more about Hadoop</summary>
22
+
23
+Wikipedia provides a great explanation of
24
+HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
25
+
26
+HDFS provides a software framework for distributed storage and processing of big data using the
27
+`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
28
+processing data. It achieves reliability by replicating the data across multiple hosts, and hence
29
+theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
30
+the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
31
+different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
32
+keep the replication of data high.
33
+
34
+HDFS has five services as follows:
35
+
36
+1. Name Node
37
+2. Secondary Name Node
38
+3. Job tracker
39
+4. Data Node
40
+5. Task Tracker
41
+
42
+Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
43
+communicate with each other and in the same way slave services can communicate with each other.
44
+NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
45
+each other.
46
+
47
+- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
48
+ track files, manage the file system and has the metadata of all the stored data within it. Some
49
+ information the NameNode keep track of are:
50
+
51
+ - details (metadata) of blocks
52
+ - in which DataNode each block lives, and its location
53
+ - replication metadata of each block
54
+
55
+ The NameNode is the gateway that a client uses to manage the HDFS cluster.
56
+
57
+- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
58
+ stores the actual data into HDFS which is responsible for the client to read and write. These are
59
+ slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
60
+ conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
61
+ for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
62
+ some other DataNode.
63
+
64
+- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
65
+ is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
66
+ NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
67
+ `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
68
+
69
+- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
70
+ tracker talks to the NameNode to know about the location of the data that will be used in
71
+ processing. The NameNode responds with the metadata of the required processing data.
72
+
73
+- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
74
+ Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
75
+ the file. The process of applying that code on the file is known as Mapper.
76
+
77
+Some more useful information/concepts about HDFS from the official
78
+website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
79
+
80
+- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
81
+ an application can create directories and store files inside these directories. The file system
82
+ namespace hierarchy is similar to most other existing file systems; one can create and remove
83
+ files, move a file from one directory to another, or rename a file. HDFS does not yet implement
84
+ user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
85
+ not preclude implementing these features.
86
+
87
+ The NameNode maintains the file system namespace. Any change to the file system namespace or its
88
+ properties is recorded by the NameNode. An application can specify the number of replicas of a
89
+ file that should be maintained by HDFS. The number of copies of a file is called the replication
90
+ factor of that file. This information is stored by the NameNode.
91
+
92
+- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
93
+ HDFS are those that deal with large data sets. These applications write their data only once but
94
+ they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
95
+ supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
96
+ Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
97
+ different DataNode.
98
+
99
+- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
100
+ might automatically move data from one DataNode to another if the free space on a DataNode falls
101
+ below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
102
+ might dynamically create additional replicas and rebalance other data in the cluster. These types
103
+ of data rebalancing schemes are not yet implemented.
104
+
105
+</details>
106
+
107
+<details>
108
+<summary>References and sources</summary>
109
+
110
+1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
111
+2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
112
+3. [HDFS 3.3.1 commands guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html)
113
+
114
+</details>
115
+
116
+### Troubleshooting section
117
+
118
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
119
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
120
+
121
+<details>
122
+<summary>Identify which DataNode has a failing volume and troubleshoot it </summary>
123
+
124
+
125
+> For Apache Hadoop 3.3.1 <sup>[3](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html) </sup>
126
+
127
+1. Identify which DataNodes are offline, these are the sources of your problems:
128
+
129
+ ```
130
+ root@netdata # dfsadmin -report
131
+ ```
132
+
133
+ Identify which node is not reported in the output of the above command. If all the nodes are
134
+ there, then you must run the next command for each DataNode.
135
+
136
+2. Review the volumes status:
137
+
138
+ ```
139
+ root@netdata # hdfs dfsadmin -getVolumeReport datanodehost:port
140
+ ```
141
+
142
+
143
+3. Connect to the node and inspect the `journalctl -xe`. If you have the Netdata Agent running in
144
+ the DataNodes as well, you should be able to identify the problem. You should be receiving alarms
145
+ about the disks and the mounts in this system.
146
+
147
+</details>
148
+
health/guides/hdfs/hdfs_stale_nodes.md
new
+138
@@ -0,0 +1,138 @@
1
+# hdfs_stale_nodes
2
+
3
+**Storage | HDFS**
4
+
5
+_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
+written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
+lack of POSIX compliance, but it does provide shell commands and Java application programming
8
+interface (API) methods that are similar to other file
9
+systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
+
11
+Receiving this alert into warning indicates that there is at least one stale DataNode due to missed
12
+heartbeats.
13
+
14
+A stale DataNode is one that has not been reachable for `dfs.namenode.stale.datanode.interval` (
15
+default is 30 seconds). Stale DataNodes are avoided, and marked as the last possible target for a
16
+read or write operation. By default, HDFS marks a node as dead if it is unreachable for 630 seconds.
17
+
18
+
19
+<details>
20
+<summary>See more about Hadoop</summary>
21
+
22
+Wikipedia provides a great explanation of
23
+HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
24
+
25
+HDFS provides a software framework for distributed storage and processing of big data using the
26
+`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
27
+processing data. It achieves reliability by replicating the data across multiple hosts, and hence
28
+theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
29
+the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
30
+different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
31
+keep the replication of data high.
32
+
33
+HDFS has five services as follows:
34
+
35
+1. Name Node
36
+2. Secondary Name Node
37
+3. Job tracker
38
+4. Data Node
39
+5. Task Tracker
40
+
41
+Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
42
+communicate with each other and in the same way slave services can communicate with each other.
43
+NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
44
+each other.
45
+
46
+- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
47
+ track files, manage the file system and has the metadata of all the stored data within it. Some
48
+ information the NameNode keep track of are:
49
+
50
+ - details (metadata) of blocks
51
+ - in which DataNode each block lives, and its location
52
+ - replication metadata of each block
53
+
54
+ The NameNode is the gateway that a client uses to manage the HDFS cluster.
55
+
56
+- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
57
+ stores the actual data into HDFS which is responsible for the client to read and write. These are
58
+ slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
59
+ conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
60
+ for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
61
+ some other DataNode.
62
+
63
+- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
64
+ is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
65
+ NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
66
+ `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
67
+
68
+- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
69
+ tracker talks to the NameNode to know about the location of the data that will be used in
70
+ processing. The NameNode responds with the metadata of the required processing data.
71
+
72
+- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
73
+ Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
74
+ the file. The process of applying that code on the file is known as Mapper.
75
+
76
+Some more useful information/concepts about HDFS from the official
77
+website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
78
+
79
+- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
80
+ an application can create directories and store files inside these directories. The file system
81
+ namespace hierarchy is similar to most other existing file systems; one can create and remove
82
+ files, move a file from one directory to another, or rename a file. HDFS does not yet implement
83
+ user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
84
+ not preclude implementing these features.
85
+
86
+ The NameNode maintains the file system namespace. Any change to the file system namespace or its
87
+ properties is recorded by the NameNode. An application can specify the number of replicas of a
88
+ file that should be maintained by HDFS. The number of copies of a file is called the replication
89
+ factor of that file. This information is stored by the NameNode.
90
+
91
+- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
92
+ HDFS are those that deal with large data sets. These applications write their data only once but
93
+ they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
94
+ supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
95
+ Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
96
+ different DataNode.
97
+
98
+- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
99
+ might automatically move data from one DataNode to another if the free space on a DataNode falls
100
+ below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
101
+ might dynamically create additional replicas and rebalance other data in the cluster. These types
102
+ of data rebalancing schemes are not yet implemented.
103
+
104
+</details>
105
+
106
+
107
+<details>
108
+<summary>References and sources</summary>
109
+
110
+1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
111
+2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
112
+
113
+</details>
114
+
115
+### Troubleshooting section
116
+
117
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
118
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
119
+
120
+<details>
121
+<summary>Fix corrupted or missing blocks</summary>
122
+
123
+1. Identify the stale node(s)
124
+
125
+ ```
126
+ root@netdata # hadoop dfsadmin -report
127
+ ```
128
+
129
+Inspect the output and check which DataNode is stale.
130
+
131
+2. Connect to the DataNode and check the log of the DataNode. You can also check for errors in the
132
+ system services.
133
+
134
+ ```
135
+ root@netdata # systemctl status hadoop
136
+ ```
137
+
138
+ Restart the service if needed.
health/guides/httpcheck/httpcheck_web_service_bad_content.md
new
+45
@@ -0,0 +1,45 @@
1
+# httpcheck_web_service_bad_content
2
+
3
+**Web Server | HTTP endpoint**
4
+
5
+The Netdata Agent monitors your HTTP endpoints. You can specify endpoints that the agent will monitor in
6
+Agent's Go module under `go.d/httpcheck.conf`. You can also specify the expected response pattern.
7
+This HTTP endpoint will send in the `response_match` option.
8
+<sup>[1](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck) </sup>
9
+If the endpoint's response does not match the `response_match` pattern, then the Agent marks the response
10
+as unexpected.
11
+
12
+The Netdata Agent calculates the average ratio of HTTP responses with unexpected content over the
13
+last 5 minutes.
14
+
15
+This alert is escalated to warning if the percentage of unexpected content is greater than 10%
16
+and then raised to critical if it is greater than 40%.
17
+
18
+<details>
19
+<summary>References and Sources</summary>
20
+
21
+1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
22
+
23
+</details>
24
+
25
+### Troubleshooting:
26
+
27
+<details>
28
+<summary>Check the actual response and the expected response.</summary>
29
+
30
+1. Try to implement a request with a verbose result:
31
+
32
+```
33
+root@netdata # curl -v <your_http_endpoint>:<port>/<path>
34
+```
35
+
36
+2. Compare it with the expected response.
37
+
38
+Check your configuration under `go.d/httpcheck.conf`:
39
+
40
+```
41
+cd /etc/netdata # Replace this path with your Netdata config directory
42
+sudo ./edit-config go.d/httpcheck.conf
43
+```
44
+
45
+</details>
health/guides/httpcheck/httpcheck_web_service_bad_status.md
new
+46
@@ -0,0 +1,46 @@
1
+# httpcheck_web_service_bad_status
2
+
3
+**Web Server | HTTP endpoint**
4
+
5
+The Netdata agent monitors your HTTP endpoints. You can specify endpoints the agent will monitor in
6
+Agent's Go module under `go.d/httpcheck.conf`. You can also specify the expected response statuses
7
+this HTTP endpoint must reply with, in the `status_accepted` option.
8
+<sup>[1](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck) </sup>
9
+If the endpoint responds with a response status that is not in the specified `status_accepted` codes, the Agent
10
+marks the response as "bad_status".
11
+
12
+The Netdata Agent calculates the average ratio of these unexpected (bad) HTTP status responses over
13
+the last 5 minutes.
14
+
15
+This alert is triggered in warning state when the ratio is greater than 10% and in critical state
16
+when it is greater than 40%.
17
+
18
+<details>
19
+<summary>References and sources</summary>
20
+
21
+1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
22
+
23
+</details>
24
+
25
+### Troubleshooting section:
26
+
27
+<details>
28
+<summary>Check the actual response status and the expected response statuses</summary>
29
+
30
+1. Try to implement a request with a verbose result:
31
+
32
+```
33
+root@netdata # curl -v <your_http_endpoint>:<port>/<path>
34
+```
35
+
36
+2. Compare it with the expected response
37
+
38
+Check your configuration under `go.d/httpcheck.conf` which are the `status_accepted` codes for this
39
+particular endpoint.
40
+
41
+```
42
+cd /etc/netdata # Replace this path with your Netdata config directory
43
+sudo ./edit-config go.d/httpcheck.conf
44
+```
45
+
46
+</details>
health/guides/httpcheck/httpcheck_web_service_slow.md
new
+32
@@ -0,0 +1,32 @@
1
+# httpcheck_web_service_slow
2
+
3
+**Web Server | HTTP Endpoints**
4
+
5
+The Netdata Agent monitors your HTTP endpoints. You can specify endpoints the Agent will monitor in
6
+the Agent's Go module under `go.d/httpcheck.conf`.
7
+<sup>[1](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck) </sup>
8
+The Agent calculates the average response time for every HTTP request made to the endpoint being monitored per hour.
9
+The Agent also calculates the average response time in a 3-min window.
10
+
11
+The Netdata Agent compares these two (average) values. If there is a significant increase in 3-min average, then it
12
+will trigger a warning alert when the response time 3-min average is at least twice as much as 1-hour average. The alert
13
+will escalate to critical when the response time 3-min average reaches three times the average amount per hour.
14
+
15
+<details>
16
+<summary>References and Sources</summary>
17
+
18
+1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
19
+
20
+</details>
21
+
22
+### Troubleshooting section:
23
+
24
+<details>
25
+<summary>General Approach</summary>
26
+
27
+To troubleshoot this issue, check for:
28
+
29
+- Network congestion in your system's network and/or in the remote endpoint's network.
30
+- If the endpoint is managed by you, then check the system load.
31
+
32
+</details>
health/guides/httpcheck/httpcheck_web_service_unreachable.md
new
+50
@@ -0,0 +1,50 @@
1
+# httpcheck_web_service_unreachable
2
+
3
+**Web Server | HTTP endpoint**
4
+
5
+The Netdata agent monitors your HTTP endpoints. You can specify endpoints the Agent will monitor in
6
+the Agent's Go module under `go.d/httpcheck.conf`.
7
+<sup>[1](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck) </sup>
8
+If your system fails to connect to your endpoint, or if the request to that endpoint times out, then
9
+the Agent will mark the requests and log them as "unreachable".
10
+
11
+The Netdata Agent calculates the ratio of these requests over the last 5 minutes.
12
+
13
+This alert is escalated to warning when the ratio is greater than 10% and then raised
14
+to critical when it is greater than 40%.
15
+
16
+<details>
17
+<summary>References and Sources</summary>
18
+
19
+1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
20
+
21
+</details>
22
+
23
+### Troubleshooting section:
24
+
25
+<details>
26
+<summary>General Approach</summary>
27
+
28
+To troubleshoot this error, check the following:
29
+
30
+- Verify that your system has access to the particular endpoint.
31
+
32
+ - Check for basic connectivity to known hosts.
33
+ - Make sure that requests and replies both to and from the endpoint are allowed in the firewall settings.
34
+ Ensure they're allowed on both your end as well as the endpoint's side.
35
+
36
+- Verify that your DNS can resolve endpoints.
37
+ - Check your current DNS (for example in linux you can use the host command):
38
+
39
+ ```
40
+ root@netdata # host -v <your_endpoint>
41
+ ```
42
+
43
+ - If the HTTP endpoint is suppose to be public facing endpoint, try an alternative DNS (for
44
+ example Cloudflare's DNS):
45
+
46
+ ```
47
+ root@netdata # host -v <your_endpoint> 1.1.1.1
48
+ ```
49
+
50
+</details>
health/guides/ioping/ioping_disk_latency.md
new
+36
@@ -0,0 +1,36 @@
1
+# ioping_disk_latency
2
+
3
+## OS: Any
4
+
5
+This alarm presents the average `I/O latency` over the last 10 seconds.
6
+
7
+> `I/O latency` is the time that is required to complete a single I/O operation on a block device.
8
+
9
+If this alarm is raised, it might indicate that your disk is under high load, or that the disk is slow.
10
+
11
+### Troubleshooting Section
12
+
13
+<details>
14
+<summary>Check related charts to find your case</summary>
15
+
16
+- First, you need to identify whether your disk is under high load or not.
17
+
18
+ <br>
19
+
20
+ 1. Go to your node on the Netdata Cloud, on the `Disks` section, and select the disk you want to
21
+ investigate.
22
+
23
+ <br>
24
+
25
+ 2. On the top of the page you can see the utilization of the disk, and you can also go to the `Disk I/O Bandwidth`
26
+ chart. There you can see the amount of transferred data `to` and `from` the particular disk.
27
+
28
+ <br>
29
+
30
+- If the utilization is low and there isn't a significant amount of data being transferred, your
31
+ drive's latency is slow, so you need to upgrade that drive. (If it is an HDD, then an SSD would be a significant upgrade
32
+ for latency issues.)
33
+
34
+
35
+- If there is a high load on the drive, it usually means that one or more processes are heavily utilizing the drive.
36
+</details>
\ No newline at end of file
health/guides/ipc/semaphore_arrays_used.md
new
+76
@@ -0,0 +1,76 @@
1
+# semaphores_arrays_used
2
+
3
+## OS: Linux
4
+
5
+This alert presents the percentage of allocated `System V IPC semaphore arrays (sets)`. \
6
+If you receive this alert, it means that your system is experiencing high `IPC semaphore arrays` utilization and a lack
7
+of available semaphore arrays can affect application performance.
8
+
9
+<details>
10
+<summary>What is an "IPC Semaphore array (or set)"</summary>
11
+
12
+"IPC" stands for "Interprocess Communication". IPC messages are a counterpart to UNIX pipes for IPC operations. \
13
+The fastest way to communicate through processes is with shared memory. semaphores, help synchronise shared memory
14
+access across processes.<sup> [1](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html) </sup>
15
+
16
+System V semaphores are allocated in groups called sets. \
17
+A "semaphore set" consists of a control structure and an array of individual semaphores.
18
+
19
+As illustrated by E. W. Dijkstra, semaphores can be better understood using his railroad model <sup> [2](
20
+https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html) </sup>.
21
+
22
+- Imagine a railroad, where only a single train at a time is allowed to pass. Responsible for the traffic is a
23
+ **semaphore**. Each train that wants to enter the single track must wait for the **semaphore** to be in a state that
24
+ allows access to the railroad. When a train enters the track, the **semaphore** changes the state to prevent all other
25
+ traffic in the track. When the train leaves the railroad, it must change the state of the **semaphore** to allow another
26
+ train to enter.
27
+
28
+
29
+- In the computer world, a **semaphore** is an integer and the train is a process (or a thread). For a process to proceed,
30
+ it has to wait for the semaphore's value to become 0. If it proceeds, it increments this value by 1. Upon finishing
31
+ the task, the process decrements the same value by 1.
32
+
33
+> **semaphores** let processes query or alter status information. They are often used to monitor and control the
34
+> availability of system resources such as shared memory segments.
35
+> <sup> [2](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html) </sup>
36
+
37
+
38
+
39
+</details>
40
+
41
+<br>
42
+
43
+<details>
44
+<summary>References and Sources</summary>
45
+
46
+[[1] Interprocess Communication](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html) \
47
+[[2] IPC:Semaphores](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html)
48
+
49
+</details>
50
+
51
+### Troubleshooting Section
52
+
53
+<details>
54
+ <summary>Adjust the semaphore limit on your system</summary>
55
+
56
+You can check current `semaphore arrays` limit on your machine, by running:
57
+
58
+```
59
+root@netdata~ # ipcs -ls
60
+```
61
+
62
+The output will be similar to this:
63
+
64
+```
65
+------ Semaphore Limits --------
66
+max number of arrays = 32000
67
+max semaphores per array = 32000
68
+max semaphores system wide = 1024000000
69
+max ops per semop call = 500
70
+semaphore max value = 32767
71
+```
72
+
73
+To adjust the limit of the max semaphores,you can go to the `/proc/sys/kernel/sem` file and adjust the fourth field
74
+accordingly.
75
+
76
+</details>
health/guides/ipc/semaphores_used.md
new
+72
@@ -0,0 +1,72 @@
1
+# semaphores_used
2
+
3
+## OS: Linux
4
+
5
+This alarm presents the percentage of allocated `System V IPC semaphores`. \
6
+If you receive this alarm, it means that your system is experiencing high `IPC semaphore` utilization and a lack of
7
+available `semaphores` can affect application performance.
8
+
9
+<details>
10
+<summary>What is an "IPC Semaphore"</summary>
11
+
12
+`IPC` stands for `Interprocess Communication`. IPC messages are a counterpart to UNIX pipes for IPC operations. \
13
+The
14
+fastest way to communicate through processes is with shared memory. `semaphores`, help synchronise shared memory access
15
+across processes.<sup> [1](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html) </sup>
16
+
17
+As illustrated by E. W. Dijkstra, semaphores can be better understood using his railroad model <sup> [2](
18
+https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html) </sup>.
19
+
20
+- Imagine a railroad, where only a single train at a time is allowed to pass. Responsible for the traffic is a
21
+ `semaphore`. Each train that wants to enter the single track must wait for the `semaphore` to be in a state that
22
+ allows access to the railroad. When a train enters the track, the `semaphore` changes the state to prevent all other
23
+ traffic in the track. When the train leaves the railroad, it must change the state of the `semaphore` to allow another
24
+ train to enter.
25
+
26
+
27
+- In the computer world, a `semaphore` is an integer and the train is a process (or a thread). For a process to proceed,
28
+ it has to wait for the semaphore's value to become 0. If it proceeds, it increments this value by 1. Upon finishing
29
+ the task, the process decrements the same value by 1.
30
+
31
+> `semaphores` let processes query or alter status information. They are often used to monitor and control the
32
+> availability of system resources such as shared memory segments.
33
+> <sup> [2](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html) </sup>
34
+
35
+</details>
36
+
37
+<br>
38
+
39
+<details>
40
+<summary>References</summary>
41
+
42
+[[1] Interprocess Communication](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html) \
43
+[[2] IPC:Semaphores](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html)
44
+
45
+</details>
46
+
47
+### Troubleshooting Section
48
+
49
+<details>
50
+ <summary>Adjust the semaphore limit on your system</summary>
51
+
52
+You can check current `semaphore` limit on your machine, by running:
53
+
54
+```
55
+root@netdata~ # ipcs -ls
56
+```
57
+
58
+The output will be similar to this:
59
+
60
+```
61
+------ Semaphore Limits --------
62
+max number of arrays = 32000
63
+max semaphores per array = 32000
64
+max semaphores system wide = 1024000000
65
+max ops per semop call = 500
66
+semaphore max value = 32767
67
+```
68
+
69
+To adjust the limit of the max semaphores,you can go to the `/proc/sys/kernel/sem` file and adjust the second field
70
+accordingly.
71
+
72
+</details>
\ No newline at end of file
health/guides/ipmi/ipmi_events.md
new
+37
@@ -0,0 +1,37 @@
1
+# ipmi_events
2
+
3
+## OS: Any
4
+
5
+This alert presents the number of events in the IPMI System Event Log (SEL).
6
+If this alert is received, then the Log contains critical, warning, and informational events.
7
+
8
+- This alert is raised in a warning state when the number of events in the IPMI SEL exceed 0
9
+ (in other words, when they exist).
10
+
11
+<br>
12
+
13
+<details>
14
+<summary>References and Sources</summary>
15
+
16
+1. ["ipmitool" manual page](
17
+ https://linux.die.net/man/1/ipmitool)
18
+
19
+</details>
20
+
21
+### Troubleshooting Section
22
+
23
+<details>
24
+<summary>Use "ipmitool"</summary>
25
+
26
+
27
+> ipmitool is a utility for managing and configuring devices that support
28
+> the Intelligent Platform Management Interface. [Github](https://github.com/ipmitool/ipmitool)
29
+
30
+You can view the System Event Log using ipmitool, by running the command:
31
+
32
+```
33
+root@netdata~ # ipmitool sel list
34
+```
35
+
36
+You can find more info and commands in the [manual page](https://linux.die.net/man/1/ipmitool)
37
+</details>
\ No newline at end of file
health/guides/ipmi/ipmi_sensor_state.md
new
+45
@@ -0,0 +1,45 @@
1
+# ipmi_sensors_states
2
+
3
+## OS: Any
4
+
5
+This alert presents the number of IPMI sensors in the non-nominal state. \
6
+If this alert is received, then there are IPMI sensors in the warning or critical state.
7
+
8
+- This alert is raised in a warning state when the amount of sensors in a warning state are greater
9
+ than 0.
10
+- If there are any sensors in a critical state, then the alert is also raised to critical.
11
+
12
+<br>
13
+
14
+<details>
15
+<summary>References and Sources</summary>
16
+
17
+1. ["ipmi-sensors" manual page](
18
+ https://www.gnu.org/software/freeipmi/manpages/man8/ipmi-sensors.8.html)
19
+
20
+</details>
21
+
22
+### Troubleshooting Section
23
+
24
+<details>
25
+<summary>Use "ipmi-sensors" tools</summary>
26
+
27
+ipmi-sensors is a free software used to display sensor information (the package name is "
28
+freeipmi-tools").
29
+
30
+Here are some useful commands:
31
+
32
+> -v, --verbose Output verbose sensor output.
33
+> This option will output additional information about sensors such as thresholds, ranges,
34
+> numbers, and event/reading type codes.
35
+
36
+> --output-sensor-state Output sensor state in output.
37
+> This will add an additional output reporting if a sensor is in a NOMINAL, WARNING, or CRITICAL
38
+> state. The sensor state is an interpreted value based on the current sensor event. The sensor
39
+> state interpretations are determined by the configuration file
40
+> /etc/freeipmi//freeipmi_interpret_sensor.conf.
41
+
42
+You can see more options in the [manual page](
43
+https://www.gnu.org/software/freeipmi/manpages/man8/ipmi-sensors.8.html).
44
+
45
+</details>
health/guides/kubelet/kubelet_10s_pleg_relist_latency_quantile_05.md
new
+77
@@ -0,0 +1,77 @@
1
+# kubelet_10s_pleg_relist_latency_quantile_05
2
+
3
+**Kubernetes | Kubelet**
4
+
5
+_The kubelet is the primary "node agent" that runs on each node. It makes sure that containers are
6
+running in a Pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms
7
+and ensures that the containers described in those PodSpecs are running and healthy and doesn't
8
+manage containers that were not created by Kubernetes._
9
+
10
+The PLEG (Pod Lifecycle Event Generator) module in Kubelet adjusts the container runtime state with
11
+each matched pod-level event and keeps the Pod's cache up to date. Big delays in the relist process
12
+of pods will eventually cause a "PLEG is not healthy" event which will make the node unavailable (
13
+NotReady).
14
+
15
+The Netadata Agent calculates the ratio of average Pod Lifecycle Event Generator relisting latency
16
+over the last 10 seconds, compared to the last minute (quantile 0.5). Receiving this alert means
17
+that the relisting time has increased significantly.
18
+
19
+> Different pods have different relisting latencies, more quantiles help you reduce the error rate in those metrics.
20
+
21
+
22
+<details>
23
+<summary>See more about the kubelet </summary>
24
+
25
+As we said before, the kubelet works in terms of a PodSpec. A PodSpec is a YAML or a JSON object
26
+that describes a pod. The PodSpec contains all information a kubelet needs to know to run the pod in
27
+the corresponding cluster node.
28
+
29
+Beside the PodSpecs provided from the Kubernetes APIserver, there are three ways to provide a
30
+kubelet with a container manifest:
31
+
32
+- File: Path passed as a flag on the command line. Files under this path will be monitored
33
+ periodically for updates. The monitoring period is 20s by default and is configurable via a flag.
34
+- HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This endpoint is checked
35
+ every 20 seconds (also configurable with a flag).
36
+- HTTP server: The kubelet can also listen for HTTP and respond to a simple API (underspec'd
37
+ currently) to submit a new manifest.
38
+
39
+See more about Kubelet in
40
+the [Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
41
+
42
+</details>
43
+
44
+
45
+<details>
46
+<summary>See more about PLEG and the relist process</summary>
47
+
48
+A kubelet keeps track of all the Pods that are about to run in the node. The node could have any
49
+kind of Container Runtime Interface (CRI) always compatible with Kubernetes. A Pod lifecycle event
50
+interprets the underlying container state change at the pod-level abstraction, making it
51
+container-runtime-agnostic. This abstraction shields a kubelet from the runtime specifics.
52
+
53
+In order to generate pod lifecycle events, PLEG needs to detect changes in container states. The
54
+PLEG module periodically relisting all containers (even then stopped ones) and compare then with
55
+their Kubelet's PodSpecs. The relist process takes longer when there are problems with the
56
+underlying CRI or overloading of a Node with too many pods.
57
+
58
+See more about the PLEG's mechanism in
59
+the [Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
60
+
61
+</details>
62
+
63
+<details>
64
+<summary>References and Sources</summary>
65
+
66
+1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
67
+2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
68
+
69
+</details>
70
+
71
+### Troubleshooting
72
+
73
+Most cloud providers address this issue by limiting the Pods that can run in particular nodes in
74
+their managed Kubernetes services. They often implement health checks into the underlying container
75
+runtime. However, you may encounter this issue in high-end nodes which can run hundreds of
76
+containers. If you have configured your cluster by yourself (let's say with `kubeadm`), you can
77
+update the value of max Pods.
health/guides/kubelet/kubelet_10s_pleg_relist_latency_quantile_09.md
new
+77
@@ -0,0 +1,77 @@
1
+# kubelet_10s_pleg_relist_latency_quantile_09
2
+
3
+**Kubernetes | Kubelet**
4
+
5
+_The kubelet is the primary "node agent" that runs on each node. It makes sure that containers are
6
+running in a Pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms
7
+and ensures that the containers described in those PodSpecs are running and healthy and doesn't
8
+manage containers that were not created by Kubernetes._
9
+
10
+The PLEG (Pod Lifecycle Event Generator) module in Kubelet adjusts the container runtime state with
11
+each matched pod-level event and keeps the Pod's cache up to date. Big delays in the relist process
12
+of pods will eventually cause a "PLEG is not healthy" event which will make the node unavailable (
13
+NotReady).
14
+
15
+The Netadata Agent calculates the ratio of average Pod Lifecycle Event Generator relisting latency
16
+over the last 10 seconds, compared to the last minute (quantile 0.9). Receiving this alert means
17
+that the relisting time has increased significantly.
18
+
19
+> Different pods have different relisting latencies, more quantiles help you reduce the error rate in those metrics.
20
+
21
+
22
+<details>
23
+<summary>See more about the kubelet </summary>
24
+
25
+As we said before, the kubelet works in terms of a PodSpec. A PodSpec is a YAML or a JSON object
26
+that describes a pod. The PodSpec contains all information a kubelet needs to know to run the pod in
27
+the corresponding cluster node.
28
+
29
+Beside the PodSpecs provided from the Kubernetes APIserver, there are three ways to provide a
30
+kubelet with a container manifest:
31
+
32
+- File: Path passed as a flag on the command line. Files under this path will be monitored
33
+ periodically for updates. The monitoring period is 20s by default and is configurable via a flag.
34
+- HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This endpoint is checked
35
+ every 20 seconds (also configurable with a flag).
36
+- HTTP server: The kubelet can also listen for HTTP and respond to a simple API (underspec'd
37
+ currently) to submit a new manifest.
38
+
39
+See more about Kubelet in
40
+the [Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
41
+
42
+</details>
43
+
44
+
45
+<details>
46
+<summary>See more about PLEG and the relist process</summary>
47
+
48
+A kubelet keeps track of all the Pods that are about to run in the node. The node could have any
49
+kind of Container Runtime Interface (CRI) always compatible with Kubernetes. A Pod lifecycle event
50
+interprets the underlying container state change at the pod-level abstraction, making it
51
+container-runtime-agnostic. This abstraction shields a kubelet from the runtime specifics.
52
+
53
+In order to generate pod lifecycle events, PLEG needs to detect changes in container states. The
54
+PLEG module periodically relisting all containers (even then stopped ones) and compare then with
55
+their Kubelet's PodSpecs. The relist process takes longer when there are problems with the
56
+underlying CRI or overloading of a Node with too many pods.
57
+
58
+See more about the PLEG's mechanism in
59
+the [Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
60
+
61
+</details>
62
+
63
+<details>
64
+<summary>References and Sources</summary>
65
+
66
+1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
67
+2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
68
+
69
+</details>
70
+
71
+### Troubleshooting
72
+
73
+Most cloud providers address this issue by limiting the Pods that can run in particular nodes in
74
+their managed Kubernetes services. They often implement health checks into the underlying container
75
+runtime. However, you may encounter this issue in high-end nodes which can run hundreds of
76
+containers. If you have configured your cluster by yourself (let's say with `kubeadm`), you can
77
+update the value of max Pods.
health/guides/kubelet/kubelet_10s_pleg_relist_latency_quantile_099.md
new
+77
@@ -0,0 +1,77 @@
1
+# kubelet_10s_pleg_relist_latency_quantile_099
2
+
3
+**Kubernetes | Kubelet**
4
+
5
+_The kubelet is the primary "node agent" that runs on each node. It makes sure that containers are
6
+running in a Pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms
7
+and ensures that the containers described in those PodSpecs are running and healthy and doesn't
8
+manage containers that were not created by Kubernetes._
9
+
10
+The PLEG (Pod Lifecycle Event Generator) module in Kubelet adjusts the container runtime state with
11
+each matched pod-level event and keeps the Pod's cache up to date. Big delays in the relist process
12
+of pods will eventually cause a "PLEG is not healthy" event which will make the node unavailable (
13
+NotReady).
14
+
15
+The Netadata Agent calculates the ratio of average Pod Lifecycle Event Generator relisting latency
16
+over the last 10 seconds, compared to the last minute (quantile 0.99). Receiving this alert means
17
+that the relisting time has increased significantly.
18
+
19
+> Different pods have different relisting latencies, more quantiles help you reduce the error rate in those metrics.
20
+
21
+
22
+<details>
23
+<summary>See more about the kubelet </summary>
24
+
25
+As we said before, the kubelet works in terms of a PodSpec. A PodSpec is a YAML or a JSON object
26
+that describes a pod. The PodSpec contains all information a kubelet needs to know to run the pod in
27
+the corresponding cluster node.
28
+
29
+Beside the PodSpecs provided from the Kubernetes APIserver, there are three ways to provide a
30
+kubelet with a container manifest:
31
+
32
+- File: Path passed as a flag on the command line. Files under this path will be monitored
33
+ periodically for updates. The monitoring period is 20s by default and is configurable via a flag.
34
+- HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This endpoint is checked
35
+ every 20 seconds (also configurable with a flag).
36
+- HTTP server: The kubelet can also listen for HTTP and respond to a simple API (underspec'd
37
+ currently) to submit a new manifest.
38
+
39
+See more about Kubelet in
40
+the [Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
41
+
42
+</details>
43
+
44
+
45
+<details>
46
+<summary>See more about PLEG and the relist process</summary>
47
+
48
+A kubelet keeps track of all the Pods that are about to run in the node. The node could have any
49
+kind of Container Runtime Interface (CRI) always compatible with Kubernetes. A Pod lifecycle event
50
+interprets the underlying container state change at the pod-level abstraction, making it
51
+container-runtime-agnostic. This abstraction shields a kubelet from the runtime specifics.
52
+
53
+In order to generate pod lifecycle events, PLEG needs to detect changes in container states. The
54
+PLEG module periodically relisting all containers (even then stopped ones) and compare then with
55
+their Kubelet's PodSpecs. The relist process takes longer when there are problems with the
56
+underlying CRI or overloading of a Node with too many pods.
57
+
58
+See more about the PLEG's mechanism in
59
+the [Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
60
+
61
+</details>
62
+
63
+<details>
64
+<summary>References and Sources</summary>
65
+
66
+1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
67
+2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
68
+
69
+</details>
70
+
71
+### Troubleshooting
72
+
73
+Most cloud providers address this issue by limiting the Pods that can run in particular nodes in
74
+their managed Kubernetes services. They often implement health checks into the underlying container
75
+runtime. However, you may encounter this issue in high-end nodes which can run hundreds of
76
+containers. If you have configured your cluster by yourself (let's say with `kubeadm`), you can
77
+update the value of max Pods.
health/guides/linux_power_supply/linux_power_supply_capacity.md
new
+18
@@ -0,0 +1,18 @@
1
+# linux_power_supply_capacity
2
+
3
+**Power Supply | Battery**
4
+
5
+The Netdata Agent monitors the percentage of remaining power supply capacity. This alert indicates
6
+that the remaining power supply capacity is low. The system can run out of power and switch off
7
+soon as well as prepare the system for the shutdown.
8
+
9
+This alert is triggered in warning state when the percentage of remaining power supply capacity is
10
+less than 10% and in critical state when it is less than 5%.
11
+
12
+### Troubleshooting section:
13
+
14
+Restore power, replace the battery, or even consider integrating a UPS to
15
+support your systems.
16
+
17
+If you receive this alert regularly you may also want to check the battery of the system. It's
18
+capacity may be degraded.
health/guides/load/load_average_1.md
new
+127
@@ -0,0 +1,127 @@
1
+# load_average_1
2
+
3
+## OS: Linux
4
+
5
+This alarm calculates the system `load average` (`CPU` and `I/O` demand) over the period of one minute.
6
+If you receive this alarm, it means that your system is `overloaded`.
7
+
8
+<details>
9
+<summary>What does "load average" mean</summary>
10
+
11
+The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those
12
+waiting to work** (CPU, disk, uninterruptible locks)
13
+<sup> [1](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) </sup>
14
+<sup> [2](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) </sup>
15
+. So simply stated: **it measures the number of threads that aren't idle.**
16
+
17
+</details>
18
+
19
+<details>
20
+<summary>What does "overloaded" mean</summary>
21
+
22
+The term `overloaded` can be better illustrated using an example as ***Andre Lewis*** says in ***Understanding Linux CPU
23
+Load - when should you be worried?***<sup> [3](https://scoutapm.com/blog/understanding-load-averages) </sup>, which you
24
+find in our links section.
25
+
26
+We are going to take a single core CPU system and think of its core count as bridge lanes.
27
+
28
+- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
29
+
30
+
31
+- If the load average is at 1, then the bridge is full, and it is utilized 100%.
32
+
33
+
34
+- If the load average gets to 2 *(remember we are on a single core machine)*, it means that there is one lane that is
35
+ passing the bridge and one **other** full lane that waits on the side. On this example, traffic and thus cars, are
36
+ processes.
37
+
38
+So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an
39
+analogous example there.
40
+
41
+</details>
42
+
43
+<details>
44
+<summary>How we calculate the alarm</summary>
45
+
46
+On Netdata, in the [load.conf](https://github.com/netdata/netdata/blob/master/health/health.d/load.conf) file, under the
47
+health.d directory, you can see how we calculate *when* the alarm should be raised.
48
+
49
+- First, there is `load_cpu_number` where it provides the `load_average` alarms with the core count of the machine.
50
+
51
+
52
+- In the line `warn: ($this * 100 / $load_cpu_number) > (($status >= $WARNING) ? 700 : 800)`, \
53
+ `($this * 100 / $load_cpu_number)` is the current system load average in %.
54
+
55
+
56
+- Last, we check if that value exceeds 700% or 800% (depending on the `$status` of the alarm).
57
+
58
+</details>
59
+
60
+<br>
61
+
62
+<details>
63
+<summary>References</summary>
64
+
65
+[[1] UNIX Load Average Part 1: How It Works](
66
+https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) \
67
+[[2] UNIX Load Average Part 2: Not Your Average Average](
68
+https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) \
69
+[[3] Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages) \
70
+[Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html) \
71
+[Understanding Linux Process States](
72
+https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
73
+</details>
74
+
75
+### Troubleshooting Section
76
+
77
+<details>
78
+ <summary>Determine if the problem is CPU or I/O bound</summary>
79
+
80
+First you need to check if you are running on a CPU load or an I/O load problem.
81
+
82
+- You can use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds)
83
+
84
+```
85
+root@netdata~ # vmstat
86
+procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
87
+ r b swpd free buff cache si so bi bo in cs us sy id wa st
88
+ 8 0 1200384 168456 48840 1461540 4 14 65 51 334 196 3 1 95 0 0
89
+```
90
+
91
+The `procs` column, shows; \
92
+r: The number of runnable processes (running or waiting for run time). \
93
+b: The number of processes blocked waiting for I/O to complete.
94
+
95
+After that, you can use the `ps` and specifically `ps -eo s,user,cmd | grep ^[RD]`.
96
+
97
+- The `grep` command will fetch the processes that their state code starts either with R (running or runnable (on run
98
+ queue)) or D(uninterruptible sleep (usually IO)).
99
+
100
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
101
+processes you are closing and being certain that they are not necessary.
102
+
103
+</details>
104
+
105
+<details>
106
+ <summary>Check per-process CPU/disk usage to find the top consumers</summary>
107
+
108
+1. Use `top`:
109
+
110
+ ```
111
+ root@netdata~ # top -o +%CPU -i
112
+ ```
113
+ Here, you can see which processes are the main cpu consumers on the `%CPU` column.
114
+
115
+
116
+2. Use `iotop`: \
117
+ `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it,
118
+ then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
119
+ ```
120
+ root@netdata~ # sudo iotop
121
+ ```
122
+ Using this, you can see which processes are the main Disk I/O consumers on the `IO` column.
123
+
124
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
125
+processes you are closing and being certain that they are not necessary.
126
+
127
+</details>
health/guides/load/load_average_15.md
new
+104
@@ -0,0 +1,104 @@
1
+# load_average_15
2
+
3
+## OS: Linux
4
+
5
+This alarm calculates the system `load average` (CPU and I/O demand) over the period of fifteen
6
+minutes.
7
+If you receive this alarm, it means that your system is "overloaded."
8
+
9
+The alert gets raised into warning if the metric is 2 times the expected value and cleared if
10
+the value is 1.75 times the expected value.
11
+
12
+For further information on how our alerts are calculated, please have a look at our [Documentation](
13
+https://learn.netdata.cloud/docs/agent/health/reference#expressions).
14
+
15
+
16
+<details>
17
+<summary>What does "load average" mean?</summary>
18
+
19
+The term `system load average` on a Linux machine, measures the **number of threads that are
20
+currently working and those waiting to work** (CPU, disk, uninterruptible locks)<sup> [1](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) </sup><sup> [2](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) </sup>. So simply stated: **System load average measures the number of threads that aren't idle.**
21
+
22
+</details>
23
+
24
+<details>
25
+<summary>What does "overloaded" mean?</summary>
26
+
27
+Andre Lewis explains the term "overloaded" by using an example in his Blog post "Understanding Linux CPU
28
+Load - when should you be worried?"<sup> [3](https://scoutapm.com/blog/understanding-load-averages) </sup>
29
+You can click on the footnote or find it in our links section.
30
+
31
+Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example:
32
+
33
+- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
34
+- If the load average is at 1, then the bridge is full, and it is utilized 100%.
35
+- If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge.
36
+
37
+So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an analogous example there.
38
+
39
+</details>
40
+
41
+<br>
42
+
43
+<details>
44
+<summary>References and Sources</summary>
45
+
46
+1. [UNIX Load Average Part 1: How It Works](
47
+ https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works)
48
+2. [UNIX Load Average Part 2: Not Your Average Average](
49
+ https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average)
50
+3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages)
51
+4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html)
52
+5. [Understanding Linux Process States](
53
+ https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
54
+</details>
55
+
56
+### Troubleshooting Section
57
+
58
+<details>
59
+ <summary>Determine if the problem is CPU or I/O bound</summary>
60
+
61
+First you need to check if you are running on a CPU load or an I/O load problem.
62
+
63
+1. To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds):
64
+
65
+```
66
+root@netdata~ # vmstat
67
+procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
68
+ r b swpd free buff cache si so bi bo in cs us sy id wa st
69
+ 8 0 1200384 168456 48840 1461540 4 14 65 51 334 196 3 1 95 0 0
70
+```
71
+
72
+The `procs` column, shows:
73
+r: The number of runnable processes (running or waiting for run time).
74
+b: The number of processes blocked waiting for I/O to complete.
75
+
76
+2. List your currently running processes using the `ps` command:
77
+
78
+The `grep` command will fetch the processes that their state code starts either with R (running or runnable (on run queue)) or D(uninterruptible sleep (usually IO)).
79
+
80
+3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
81
+
82
+</details>
83
+
84
+<details>
85
+ <summary>Check per-process CPU/disk usage to find the top consumers</summary>
86
+
87
+1. To see the processes that are the main CPU consumers, use the task manager program `top` like this:
88
+
89
+ ```
90
+ root@netdata~ # top -o +%CPU -i
91
+ ```
92
+
93
+
94
+2. Use `iotop`:
95
+ `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it,
96
+ then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
97
+ ```
98
+ root@netdata~ # sudo iotop
99
+ ```
100
+Note: If `iotop` is not installed on your machine, please refer to the [install instructions](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
101
+
102
+3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
103
+
104
+</details>
\ No newline at end of file
health/guides/load/load_average_5.md
new
+110
@@ -0,0 +1,110 @@
1
+# load_average_5
2
+
3
+## OS: Linux
4
+
5
+This alarm calculates the system `load average` (CPU and I/O demand) over the period of five minutes.
6
+If you receive this alarm, it means that your system is "overloaded."
7
+
8
+The alert gets raised into warning if the metric is 4 times the expected value and cleared if the value is 3.5 times the expected value.
9
+
10
+For further information on how our alerts are calculated, please have a look at our [Documentation](
11
+https://learn.netdata.cloud/docs/agent/health/reference#expressions).
12
+
13
+
14
+<details>
15
+<summary>What does "load average" mean?</summary>
16
+
17
+The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those
18
+waiting to work** (CPU, disk, uninterruptible locks)
19
+<sup> [1](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) </sup>
20
+<sup> [2](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) </sup>
21
+. So simply stated: **System load average measures the number of threads that aren't idle.**
22
+
23
+</details>
24
+
25
+<details>
26
+<summary>What does "overloaded" mean?</summary>
27
+
28
+Andre Lewis explains the term "overloaded" by using an example in his Blog post "Understanding Linux CPU
29
+Load - when should you be worried?"<sup> [3](https://scoutapm.com/blog/understanding-load-averages) </sup>
30
+You can click on the footnote or
31
+find it in our links section.
32
+
33
+Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example:
34
+
35
+- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
36
+- If the load average is at 1, then the bridge is full, and it is utilized 100%.
37
+- If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge.
38
+
39
+So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an
40
+analogous example there.
41
+
42
+</details>
43
+
44
+
45
+
46
+<br>
47
+
48
+<details>
49
+<summary>References and Sources</summary>
50
+
51
+1. [UNIX Load Average Part 1: How It Works](
52
+https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works)
53
+2. [UNIX Load Average Part 2: Not Your Average Average](
54
+https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average)
55
+3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages)
56
+4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html)
57
+5. [Understanding Linux Process States](
58
+https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
59
+</details>
60
+
61
+### Troubleshooting Section
62
+
63
+<details>
64
+ <summary>Determine if the problem is CPU or I/O bound</summary>
65
+
66
+First you need to check if you are running on a CPU load or an I/O load problem.
67
+
68
+1. To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds):
69
+
70
+```
71
+root@netdata~ # vmstat
72
+procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
73
+ r b swpd free buff cache si so bi bo in cs us sy id wa st
74
+ 8 0 1200384 168456 48840 1461540 4 14 65 51 334 196 3 1 95 0 0
75
+```
76
+
77
+The `procs` column, shows:
78
+r: The number of runnable processes (running or waiting for run time).
79
+b: The number of processes blocked waiting for I/O to complete.
80
+
81
+2. List your currently running processes using the `ps` command:
82
+
83
+The `grep` command will fetch the processes that their state code starts either with R (running or runnable (on run
84
+ queue)) or D(uninterruptible sleep (usually IO)).
85
+
86
+3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
87
+
88
+</details>
89
+
90
+<details>
91
+ <summary>Check per-process CPU/disk usage to find the top consumers</summary>
92
+
93
+1. To see the processes that are the main CPU consumers, use the task manager program `top` like this:
94
+
95
+ ```
96
+ root@netdata~ # top -o +%CPU -i
97
+ ```
98
+
99
+
100
+2. Use `iotop`:
101
+ `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it,
102
+ then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
103
+ ```
104
+ root@netdata~ # sudo iotop
105
+ ```
106
+Note: If `iotop` is not installed on your machine, please refer to the [install instructions](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
107
+
108
+3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
109
+
110
+</details>
health/guides/mdstat/mdstat_disks.md
new
+57
@@ -0,0 +1,57 @@
1
+# mdstat_disks
2
+
3
+## OS: Any
4
+
5
+This alert presents the number of devices in the down state for the respective RAID array raising
6
+it.
7
+If you receive this alert, then the array is degraded and some array devices are missing.
8
+
9
+- This alert is escalated to a warning when there are failed devices.
10
+
11
+<details>
12
+<summary>What is a "degraded array" event?</summary>
13
+
14
+> When a RAID array experiences the failure of one or more disks, it can enter degraded mode, a
15
+> fallback mode that generally allows the continued usage of the array, but either loses the
16
+> performance boosts of the RAID technique (such as a RAID-1 mirror across two disks when one of
17
+> them fails; performance will fall back to that of a normal, single drive) or experiences severe
18
+> performance penalties due to the necessity to reconstruct the damaged data from error correction
19
+> data.<sup>[1](https://en.wikipedia.org/wiki/Degraded_mode) </sup>
20
+
21
+</details>
22
+
23
+<br>
24
+
25
+<details>
26
+<summary>References and Sources</summary>
27
+
28
+1. [Degraded Mode](https://en.wikipedia.org/wiki/Degraded_mode)
29
+2. [Mdadm recover degraded array procedure](
30
+ https://www.thomas-krenn.com/en/wiki/Mdadm_recover_degraded_Array_procedure)
31
+3. [mdadm Manual page](https://linux.die.net/man/8/mdadm)
32
+4. [mdadm cheat sheet](https://www.ducea.com/2009/03/08/mdadm-cheat-sheet/)
33
+
34
+</details>
35
+
36
+
37
+### Troubleshooting Section
38
+
39
+<details>
40
+<summary>Examine for faulty or offline devices</summary>
41
+
42
+Having a degraded array means that one or more devices are faulty or missing.
43
+To fix this issue, check for faulty devices by running:
44
+
45
+```
46
+root@netdata~ # mdadm --detail <RAIDDEVICE>
47
+```
48
+
49
+Replace "RAIDDEVICE" with the name of your RAID device.
50
+
51
+To recover the array, replace the faulty devices or bring back any offline
52
+devices.
53
+For more information check: [Mdadm recover degraded array procedure](
54
+https://www.thomas-krenn.com/en/wiki/Mdadm_recover_degraded_Array_procedure)
55
+
56
+</details>
57
+
health/guides/mdstat/mdstat_mismatch_cnt.md
new
+28
@@ -0,0 +1,28 @@
1
+# mdstat_mismatch_cnt
2
+
3
+## OS: Any
4
+
5
+This alert presents the number of unsynchronized blocks for the RAID array in crisis.
6
+Receiving this alert indicates a high number of unsynchronized blocks for the RAID array.
7
+This might indicate that data on the array is corrupted.
8
+
9
+This alert is raised to warning when the metric exceeds 1024 unsynchronized blocks.
10
+
11
+<details><summary>References and Sources</summary>
12
+
13
+[Serverfault | Reasons for high mismatch_cnt on a RAID1/10 array](
14
+https://serverfault.com/questions/885565/what-are-raid-1-10-mismatch-cnt-0-causes-except-for-swap-file/885574#885574).
15
+
16
+</details>
17
+
18
+## Troubleshooting section
19
+
20
+There is no standard approach to troubleshooting this alert because the reasons can be various.
21
+
22
+For example, one of the reasons might be a swap on the array, which is relatively harmless.
23
+However, this alert can also be triggered by hardware issues which can lead to many
24
+problems and inconsistencies between the disks.
25
+
26
+You might find some troubleshooting ideas in this thread: [Reasons for high mismatch_cnt on a RAID1/10
27
+array](
28
+https://serverfault.com/questions/885565/what-are-raid-1-10-mismatch-cnt-0-causes-except-for-swap-file/885574#885574).
\ No newline at end of file
health/guides/megacli/megacli_adapter_state.md
new
+53
@@ -0,0 +1,53 @@
1
+# megacli_adapter_state
2
+
3
+## OS: Any
4
+
5
+A disk array controller is a device that manages the physical disk drives and presents them to the
6
+computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred
7
+to as RAID controller. It also often provides additional disk cache.
8
+
9
+The Netdata Agent checks the status of your MegaRAID controller by scraping the output of
10
+the `megacli -LDPDInfo -aAll` command. This alert indicates that the status of a virtual drive is in
11
+the degraded state (0: false, 1:true).
12
+
13
+#### States of a virtual drive:
14
+
15
+| State | Description |
16
+|:----------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
17
+| Optimal | The virtual drive operating condition is good. All configured drives are online. |
18
+| Degraded | The virtual drive operating condition is not optimal. One of the configured drives has failed or is offline. |
19
+| Partial Degraded | The operating condition in a RAID 6 virtual drive is not optimal. One of the configured drives has failed or is offline. RAID 6 can tolerate up to two drive failures. |
20
+| Failed | The virtual drive has failed. |
21
+| Offline | The virtual drive is not available to the RAID controller. |
22
+
23
+<details>
24
+<summary>References and source</summary>
25
+
26
+1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
27
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
28
+
29
+
30
+</details>
31
+
32
+### Troubleshooting section:
33
+
34
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
35
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
36
+
37
+<details>
38
+ <summary>General approach</summary>
39
+
40
+1. Gather more information about your virtual drives in all adapters
41
+
42
+ ```
43
+ root@netdata # megacli –LDInfo -Lall -aALL
44
+ ```
45
+
46
+2. Check which virtual drive is in degraded state and in which adapter
47
+
48
+3. Consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
49
+ 1. section `2.1.16` to check what is going wrong with your drives.
50
+ 2. section `7.18` to perform any action in drives. Focus on {`7.18.2`,`7.18.6`,`7.18.7`,`7.18.8`
51
+ ,`7.18.11`,`7.18.14`}
52
+
53
+</details>
\ No newline at end of file
health/guides/megacli/megacli_bbu_cycle_count.md
new
+46
@@ -0,0 +1,46 @@
1
+# megacli_bbu_cycle_count
2
+
3
+## OS: Any
4
+
5
+This is an alert about the battery backup unit in the MegaCLI controller. The Netdata Agent monitors
6
+the average battery backup unit charge cycles count over the last 10 seconds. This alert indicates
7
+that a high number of full recharge cycles have been elapsed in the unit's lifetime. This metrics
8
+may affect the battery relative capacity.
9
+
10
+This alert is triggered in warning state when the number of charge cycles is greater than 100 and in
11
+critical state when it is greater than 500.
12
+
13
+<details>
14
+<summary>References and source</summary>
15
+
16
+1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
17
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
18
+
19
+</details>
20
+
21
+### Troubleshooting section:
22
+
23
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
24
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
25
+
26
+<details>
27
+ <summary>General approach</summary>
28
+
29
+1. Gather more information about your battery units in all of your adapters
30
+
31
+ ```
32
+ root@netdata # megacli -AdpBbuCmd -GetBbuStatus -aALL
33
+ ```
34
+
35
+2. Perform a battery check in the battery which had low relative charge. **Before perform any
36
+ action, consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
37
+ section {`7.14`}**
38
+
39
+ ```
40
+ root@netdata # megacli -AdpBbuCmd -BbuLearn -aX // X is the adaptor's number
41
+ ```
42
+
43
+3. Replace the battery in question if needed.
44
+
45
+</details>
46
+
health/guides/megacli/megacli_bbu_relative_charge.md
new
+50
@@ -0,0 +1,50 @@
1
+# megacli_bbu_relative_charge
2
+
3
+## OS: Any
4
+
5
+A disk array controller is a device that manages the physical disk drives and presents them to the
6
+computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred
7
+to as RAID controller. It also often provides additional disk cache.
8
+
9
+The Netdata Agent calculates the average battery backup unit relative state of charge over the last
10
+10 seconds. This alert indicates that the state of charge is low. The relative state of charge is an
11
+indication of full charge capacity percentage in relation to the design capacity. A constantly low
12
+value may indicate that the battery is worn out. You might want to consider changing the battery.
13
+
14
+This alert is raised into warning when the relative state of charge of a battery is below 80% and in
15
+critical when it is below 50%.
16
+
17
+
18
+<details>
19
+<summary>References and source</summary>
20
+
21
+1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
22
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
23
+
24
+</details>
25
+
26
+### Troubleshooting section:
27
+
28
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
29
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
30
+
31
+<details>
32
+ <summary>General approach</summary>
33
+
34
+1. Gather more information about your battery units in all of your adapters
35
+
36
+ ```
37
+ root@netdata # megacli -AdpBbuCmd -GetBbuStatus -aALL
38
+ ```
39
+
40
+2. Perform a battery check in the battery which had low relative charge. **Before perform any
41
+ action, consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
42
+ section {`7.14`}**
43
+
44
+ ```
45
+ root@netdata # megacli -AdpBbuCmd -BbuLearn -aX // X is the adaptor's number
46
+ ```
47
+
48
+3. Replace the battery in question if needed.
49
+
50
+</details>
health/guides/megacli/megacli_pd_media_errors.md
new
+64
@@ -0,0 +1,64 @@
1
+# megacli_pd_media_errors
2
+
3
+## OS: Any
4
+
5
+A disk array controller is a device that manages the physical disk drives and presents them to the
6
+computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred
7
+to as RAID controller. It also often provides additional disk cache.
8
+
9
+A media error is an event where a storage disk was unable to perform the requested I/O operation
10
+because of problems accessing the stored data.
11
+
12
+This is an alert about the physical disks attached to the MegaCLI controller. The Netdata Agent
13
+monitors the number of physical drive media errors. This alert indicates that a bad sector was found
14
+on the drive during a patrol check or from a rebuild operation on a specific disk by the raid
15
+adapter.
16
+
17
+This alert is raised into warning if any media error occur. This doesn't mean that there is an
18
+imminent disk failure, but you should keep an eye on this particular disk
19
+
20
+<details>
21
+<summary> More about media errors </summary>
22
+
23
+Media errors are more common on read transactions but might occur on writes as well. A media error
24
+on a `write` may occur when the disk has problems locating the position to write the data. On reads,
25
+in addition to these positioning faults, the disk may experience problems retrieving the data. When
26
+a disk writes data, it writes other information as well, such as to record the position, note CRC or
27
+checksum to confirm data write integrity.
28
+
29
+</details>
30
+
31
+<details>
32
+<summary>References and source</summary>
33
+
34
+1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
35
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
36
+
37
+</details>
38
+
39
+### Troubleshooting section:
40
+
41
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
42
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
43
+
44
+<details>
45
+ <summary>General approach</summary>
46
+
47
+1. Gather more information about your virtual drives in all adapters
48
+
49
+ ```
50
+ root@netdata # megacli –LDInfo -Lall -aALL
51
+ ```
52
+
53
+2. Check which virtual drive is reporting media errors and in which adapter
54
+
55
+3. Check the Bad block table for the virtual drive in question
56
+
57
+ ```
58
+ root@netdata # megacli –GetBbtEntries -LX -aY // X: virtual drive , Y the adapter
59
+ ```
60
+
61
+4. Consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
62
+ section `7.17.11` to recheck these block entries. **This operation removes any data stored on the
63
+ physical drives. Back up the good data on the drives before making any changes to the
64
+ configuration**
\ No newline at end of file
health/guides/megacli/megacli_pd_predictive_failures.md
new
+47
@@ -0,0 +1,47 @@
1
+# megacli_pd_predictive_failures
2
+
3
+## OS: Any
4
+
5
+A disk array controller is a device that manages the physical disk drives and presents them to the
6
+computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred
7
+to as RAID controller. It also often provides additional disk cache.
8
+
9
+A predictive drive failure (self-monitoring analysis and reporting
10
+technology [S.M.A.R.T.](https://en.wikipedia.org/wiki/S.M.A.R.T.#:~:text=(Self%2DMonitoring%2C%20Analysis%20and,SSDs)%2C%20and%20eMMC%20drives)
11
+error).
12
+
13
+This is an alert about the physical disks attached to the MegaCLI controller. The Netdata Agent
14
+calculates the number of physical drive predictive failures. The failure prediction function for the
15
+hard disk drives determines the risk of a failure in advance and issues a warning when the risk is
16
+high. A hard disk can still operate normally but may fail in the near future. You might want to
17
+consider replacing the disk.
18
+
19
+<details>
20
+<summary>References and source</summary>
21
+
22
+1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
23
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
24
+
25
+</details>
26
+
27
+### Troubleshooting section:
28
+
29
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
30
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
31
+
32
+<details>
33
+ <summary>General approach</summary>
34
+
35
+1. Gather more information about your virtual drives in all adapters
36
+
37
+ ```
38
+ root@netdata # megacli –LDInfo -Lall -aALL
39
+ ```
40
+
41
+2. Check which virtual drive is reporting media errors and in which adapter
42
+
43
+
44
+3. Consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
45
+ 1. section `2.1.16` to check what is going wrong with your drives.
46
+ 2. section `7.18` to perform any action in drives. Focus on {`7.18.2`,`7.18.6`,`7.18.7`,`7.18.8`
47
+ ,`7.18.11`,`7.18.14`}
health/guides/memory/1hour_memory_hw_corrupted.md
new
+40
@@ -0,0 +1,40 @@
1
+# 1hour_memory_hw_corrupted
2
+
3
+## OS: Linux
4
+
5
+The Linux kernel keeps track of the system memory state. You can find the actual values it tracks in
6
+the man pages <sup>[1](https://man7.org/linux/man-pages/man5/proc.5.html) </sup> under
7
+the `/proc/meminfo` subsection. One of the values that the kernel reports is the `HardwareCorrupted`
8
+, which is the amount of memory, in kibibytes (1024 bytes), with physical memory corruption
9
+problems, identified by the hardware and set aside by the kernel so it does not get used.
10
+
11
+The Netdata Agent monitors this value. This alert indicates that the memory is corrupted due to a
12
+hardware failure. While primarily the error may be due to a failing RAM chip, it can also be caused
13
+by incorrect seating or improper contact between the socket and memory module.
14
+
15
+<details>
16
+<summary>References and Sources</summary>
17
+
18
+1. [man pages /proc](https://man7.org/linux/man-pages/man5/proc.5.html)
19
+
20
+1. [memtester homepage](https://pyropus.ca/software/memtester/)
21
+
22
+</details>
23
+
24
+### Troubleshooting section:
25
+
26
+<details>
27
+<summary>Verify a bad memory module</summary>
28
+
29
+Most of the times, uncorrectable errors will make your system and reboot/shutdown in a state of
30
+panic. If not, that means that your tolerance level is high enough to not make the system go into
31
+panic. You must identify the defective module immediately.
32
+
33
+1. `memtester` is a userspace utility for testing the memory subsystem for faults. It's portable and
34
+ should compile and work on any 32 or 64-bit Unix-like system. For hardware developers, memtester
35
+ can be told to test memory starting at a particular physical address (memtester v4.1.0+).
36
+ <sup>[2](https://pyropus.ca/software/memtester/)
37
+
38
+You may also receive this error as a result of incorrect seating or improper contact between the
39
+socket and RAM module. Check on both before consider replacing the RAM module.
40
+</details>
health/guides/memory/ecc_memory_mc_correctable.md
new
+49
@@ -0,0 +1,49 @@
1
+# ecc_memory_mc_correctable
2
+
3
+## OS: Linux
4
+
5
+*Error correction code memory (ECC memory) is a type of computer data storage that uses an error
6
+correction code (ECC) to detect and correct n-bit data corruption which occurs in memory. ECC
7
+memory is used in most computers where data corruption cannot be tolerated under any circumstances,
8
+like industrial control applications, critical databases, and infrastructural memory
9
+caches.* <sup>[1](https://en.wikipedia.org/wiki/ECC_memory) </sup>
10
+
11
+"Correctable errors are generally single-bit errors that the system or the built-in ECC mechanism
12
+can correct. These errors do not cause system downtime of data
13
+corruption." <sup>[2](https://www.atpinc.com/blog/ecc-dimm-memory-ram-errors-types-chipkill) </sup>
14
+
15
+Netdata agent monitors the number of ECC correctable errors in the last 10 minutes.
16
+
17
+<details>
18
+<summary>References and sources:</summary>
19
+
20
+1. [ECC memory on wikipedia](https://en.wikipedia.org/wiki/ECC_memory)
21
+
22
+1. [RAM types and ECC technologies](https://www.atpinc.com/blog/ecc-dimm-memory-ram-errors-types-chipkill)
23
+
24
+1. [memtester homepage](https://pyropus.ca/software/memtester/)
25
+
26
+</details>
27
+
28
+### Troubleshooting section:
29
+
30
+<details>
31
+<summary>Verify a bad memory module</summary>
32
+
33
+Correctable errors do not necessarily indicate hardware failures, but should generally still be investigated.
34
+
35
+1. `memtester` is a userspace utility for testing the memory subsystem for faults. It's portable and
36
+ should compile and work on any 32 or 64-bit Unix-like system. For hardware developers, memtester
37
+ can be told to test memory starting at a particular physical address (memtester v4.1.0+).
38
+ <sup>[3](https://pyropus.ca/software/memtester/)
39
+
40
+You can also get this kind of errors by incorrect seating or improper contact between the socket and
41
+RAM module. Check on both before consider replacing the RAM module.
42
+</details>
43
+
44
+<details>
45
+<summary>Check for BIOS updates</summary>
46
+
47
+You should check for critical BIOS updates on your hardware's vendor support page.
48
+
49
+</details>
health/guides/memory/ecc_memory_mc_uncorrectable.md
new
+93
@@ -0,0 +1,93 @@
1
+# ecc_memory_mc_uncorrectable
2
+
3
+## OS: Linux
4
+
5
+Error correction code memory (ECC memory) is a type of computer data storage that uses an error
6
+correction code (ECC) to detect and correct n-bit data corruption which occurs in memory. Error
7
+correction codes protect against undetected memory data corruption, and is used in computers where
8
+such corruption is unacceptable, for example in some scientific and financial computing
9
+applications, or in database and file
10
+servers. <sup>[1](https://en.wikipedia.org/wiki/ECC_memory) </sup>
11
+
12
+The Netdata Agent monitors the number of ECC uncorrectable errors in the last 10 minutes.
13
+
14
+
15
+<details>
16
+<summary>See more on uncorrectable errors.</summary>
17
+
18
+There are two main categories of Uncorrectable Errors (UE) as documented in the
19
+kernel.org <sup>[2](https://www.kernel.org/doc/Documentation/admin-guide/ras.rst) </sup>
20
+
21
+1. Fatal Error, when a UE error happens on a critical component of the system (for example, a piece
22
+ of the Kernel got corrupted by a UE). The only reliable way to avoid data corruption is to hang
23
+ or reboot the machine.
24
+
25
+1. Non-fatal Error, when a UE error happens on an unused component, like an unused memory bank. The
26
+ system may still run, eventually replacing the affected hardware by a hot spare, if available.
27
+
28
+</details>
29
+
30
+
31
+<details>
32
+<summary>See more on machine checks</summary>
33
+
34
+> Machine checks report internal hardware error conditions detected by the CPU. Uncorrected errors
35
+typically cause a machine check (often with panic), corrected ones cause a machine check log entry.
36
+>
37
+> The behavior your machine will have when UE occurs depends on the tolerance level settings. The
38
+tolerance level configures how hard the kernel tries to recover even at some risk of deadlock.
39
+Higher tolerant values trade potentially better uptime with the risk of a crash or even corruption (
40
+for tolerant >= 3). The Default is 1.
41
+>
42
+> - 0: always panic on uncorrected errors, log corrected errors
43
+>
44
+> - 1: panic or SIGBUS on uncorrected errors, log corrected errors
45
+>
46
+> - 2: SIGBUS or log uncorrected errors, log corrected errors
47
+>
48
+> - 3: never panic or SIGBUS, log all errors (for testing
49
+> only)
50
+>
51
+> Also, when an error happens on a userspace process, it is also possible to kill such process and
52
+> let userspace restart it. <sup>[3](https://www.kernel.org/doc/html/v5.15-rc6/x86/x86_64/machinecheck.html) </sup>
53
+
54
+
55
+</details>
56
+
57
+<details>
58
+<summary>References and sources:</summary>
59
+
60
+1. [ECC memory on wikipedia](https://en.wikipedia.org/wiki/ECC_memory)
61
+
62
+1. [Reliability, Availability and Serviceability concepts](https://www.kernel.org/doc/Documentation/admin-guide/ras.rst)
63
+
64
+1. [Machine checks](https://www.kernel.org/doc/html/v5.2/x86/x86_64/machinecheck.html)
65
+
66
+1. [memtester homepage](https://pyropus.ca/software/memtester/)
67
+
68
+</details>
69
+
70
+### Troubleshooting section:
71
+
72
+<details>
73
+<summary>Verify a bad memory module</summary>
74
+
75
+Most of the times, uncorrectable errors will make your system and reboot/shutdown in a state of panic. If
76
+not, that means that your tolerance level is high enough to not make the system go into panic. You
77
+must identify the defective module immediately.
78
+
79
+1. `memtester` is a userspace utility for testing the memory subsystem for faults. It's portable and
80
+ should compile and work on any 32 or 64-bit Unix-like system. For hardware developers, memtester
81
+ can be told to test memory starting at a particular physical address (memtester v4.1.0+).
82
+ <sup>[2](https://pyropus.ca/software/memtester/)
83
+
84
+You may also receive this error as a result of incorrect seating or improper contact between the socket and
85
+RAM module. Check on both before consider replacing the RAM module.
86
+</details>
87
+
88
+<details>
89
+<summary>Check for BIOS updates</summary>
90
+
91
+You should check for critical BIOS updates on your hardware's vendor support page.
92
+
93
+</details>
health/guides/mysql/mysql_10s_slow_queries.md
new
+36
@@ -0,0 +1,36 @@
1
+# mysql_10s_slow_queries
2
+
3
+# Database | MySQL
4
+
5
+This alert presents the number of slow queries in the last 10 seconds. If you receive this, it
6
+indicates a high number of slow queries.
7
+
8
+The metric is raised in a warning state when the value is larger than 10. If the number of slow
9
+queries in the last 10 seconds exceeds 20, then the alert is raised in critical state.
10
+
11
+Queries are defined as "slow", if they have taken more than `long_query_time` seconds, a predefined
12
+variable. Also, the value is measured in real time, not CPU time.
13
+
14
+<details><summary>References and Sources</summary>
15
+
16
+1. [SQL Query Optimisation](https://opensource.com/article/17/5/speed-your-mysql-queries-300-times)
17
+</details>
18
+
19
+### Troubleshooting Section
20
+
21
+<details><summary>Determine which queries are the problem and try to optimise
22
+them</summary>
23
+
24
+To identify the slow queries, you can enable the slow-query log of MySQL:
25
+
26
+1. Locate the `my.cnf` file
27
+2. Enable the slow-query log by setting the `slow_query_log variable` to `On`.
28
+3. Enter a path where the log files should be stored in the `slow_query_log_file` variable.
29
+
30
+After you know which queries are the ones taking longer than preferred, you can use the `EXPLAIN`
31
+keyword to overview how many rows are accessed, what operations are being done etc.
32
+
33
+After you've found the cause for the slow queries, you can start optimizing your queries. Consider to use an index and think about how you can change the way you `JOIN` tables. Both of these methods aid to reduce the amount of data that is being accessed without it really being needed.
34
+
35
+Read more about [speeding up your mySQL queries](https://opensource.com/article/17/5/speed-your-mysql-queries-300-times).
36
+</details>
health/guides/mysql/mysql_connections.md
new
+36
@@ -0,0 +1,36 @@
1
+# mysql_connections
2
+
3
+## Database | MySQL, MariaDB
4
+
5
+This alert presents the percentage of used client connections.
6
+Receiving this alert means that there is a high client connection utilization
7
+compared to the limit.
8
+
9
+This alert is raised to warning when the percentage exceeds 70%.
10
+If the metric exceeds 90%, then the alert is escalated to critical.
11
+
12
+<details><summary>References and Sources</summary>
13
+
14
+1. [MySQL max connections](https://ubiq.co/database-blog/how-to-increase-max-connections-in-mysql/)
15
+
16
+</details>
17
+
18
+### Troubleshooting Section
19
+
20
+<details><summary>Increase the Connection Limit</summary>
21
+
22
+To increase the connection limit, log into MySQL form the terminal and use the following code:
23
+`show variables like "max_connections";`
24
+to see the current limit.
25
+
26
+Using:
27
+`set global max_connections = "LIMIT";`
28
+Where "LIMIT" is the new limit you will choose, you can alter the limit without restarting the
29
+server.
30
+
31
+To increase the limit permanently, locate the `my.cnf` file (typically under `/etc` but depends on
32
+installation) and append `max_connections = 200` under the `mysqld` section.
33
+
34
+You can read more in our References and Sources section.
35
+
36
+</details>
health/guides/mysql/mysql_galera_cluster_size.md
new
+31
@@ -0,0 +1,31 @@
1
+# mysql_galera_cluster_size
2
+
3
+## Database | MySQL, MariaDB
4
+
5
+This alert presents the current Galera cluster size, compared to the maximum size in the last 2
6
+minutes.
7
+
8
+If you receive this alert, then it may indicate a network connectivity problem or
9
+that MySQL is down on one node.
10
+
11
+This alert is raised into warning if the current Galera cluster size is larger than the maximum
12
+size in the last 2 minutes.
13
+
14
+If the current Galera cluster size is less than the maximum size in the last sixty seconds, then the
15
+alert is escalated into critical.
16
+
17
+<details><summary>References and Sources</summary>
18
+
19
+1. [Galera Cluster Training Library](
20
+ https://galeracluster.com/library/training/tutorials/galera-monitoring.html)
21
+
22
+</details>
23
+
24
+### Troubleshooting Section
25
+
26
+<details><summary>Check Node Status</summary>
27
+
28
+Refer to the [Galera Cluster training library](https://galeracluster.com/library/training/tutorials/galera-monitoring.html)
29
+for documentation on cluster health monitoring.
30
+
31
+</details>
health/guides/mysql/mysql_galera_cluster_state.md
new
+27
@@ -0,0 +1,27 @@
1
+# mysql_galera_cluster_state
2
+
3
+## Database | MySQL, MariaDB
4
+
5
+This alert presents the state of a node in the Galera cluster. If you receive this, it could be an
6
+indication that the node lost its connection to the Primary Component due to network partition.
7
+
8
+The alert gets raised into warning if the metric has one of the values:
9
+
10
+| Code | Description | Alert Status |
11
+|:----:|:--------------------------------------------------------------------------------|:------------:|
12
+| `0` | Undefined - indicates a starting node that is not part of the Primary Component | Critical |
13
+| `1` | Joining (requesting/receiving State Transfer) - node is joining the cluster | Critical |
14
+| `2` | Donor/Desynced - node is the donor to the node joining the cluster | Warning |
15
+| `3` | Joined - node has joined the cluster | Warning |
16
+| `>5` | - | Critical |
17
+
18
+For further information, please have a look at the *References and Sources* section.
19
+
20
+<details><summary>References and Sources</summary>
21
+
22
+1. [Galera Cluster Glossary](https://galeracluster.com/library/documentation/glossary.html)
23
+2. [Wsrep status index](
24
+ https://www.percona.com/doc/percona-xtradb-cluster/5.5/wsrep-status-index.html)
25
+3. [Galera Cluster Notification command](
26
+ https://galeracluster.com/library/documentation/notification-cmd.html)
27
+</details>
health/guides/mysql/mysql_galera_cluster_status.md
new
+47
@@ -0,0 +1,47 @@
1
+# mysql_galera_cluster_status
2
+
3
+## Database | MySQL, MariaDB
4
+
5
+This alert presents the status of the Galera node cluster component. If you receive this, it is an
6
+indication the cluster has been split into several components due to network failure.
7
+
8
+<details><summary>What is Quorum</summary>
9
+
10
+> A majority (> 50%) of nodes. In the event of a network partition, only the cluster partition
11
+> that retains a quorum (if any) will remain Primary by default.<sup>[1](https://galeracluster.com/library/documentation/glossary.html#:~:text=A%20majority%20(%3E%2050%25)%20of%20nodes.%20In%20the%20event%20of%20a%20network%20partition%2C%20only%20the%20cluster%20partition%20that%20retains%20a%20quorum%20(if%20any)%20will%20remain%20Primary%20by%20default.) </sup>
12
+</details>
13
+
14
+<details><summary>What is a Primary Component?</summary>
15
+
16
+> In addition to single-node failures, the cluster may be split into several components due to
17
+> network failure. In such a situation, only one of the components can continue to modify the
18
+> database state to avoid history divergence. This component is called the Primary Component (PC)
19
+> .<sup>[1](https://galeracluster.com/library/documentation/glossary.html#:~:text=from%20the%20database.-,Primary%20Component,For%20more%20information%20on%20the%20Primary%20Component%2C%20see%20Quorum%20Components.,-Quorum) </sup>
20
+</details>
21
+
22
+<details><summary>What is a Non-primary state Component?</summary>
23
+
24
+> The clusters without the quorum enter the non-primary state and begin attempt to connect with the
25
+> Primary Component.<sup>[2](https://galeracluster.com/library/documentation/weighted-quorum.html#:~:text=while%20those%20without%20quorum%20enter%20the%20non%2Dprimary%20state%20and%20begin%20attempt%20to%20connect%20with%20the%20Primary%20Component.) </sup>
26
+</details>
27
+
28
+The codes of the Galera node cluster component status can be:
29
+
30
+| Code | Description | Alert Status |
31
+|:----:|:-------------------------|:------------:|
32
+| `-1` | Unknown. | Critical |
33
+| `0` | Primary | Clear |
34
+| `1` | Non-primary/quorum lost | Critical |
35
+| `2` | Disconnected | Critical |
36
+
37
+For further information on Primary and non-Primary components please have a look at the
38
+*References and Sources* section.
39
+
40
+<details><summary>References and Sources</summary>
41
+
42
+1. [Galera CLuster Glossary](
43
+ https://galeracluster.com/library/documentation/glossary.html)
44
+2. [Galera Cluster Documentation](
45
+ https://galeracluster.com/library/documentation/weighted-quorum.html)
46
+
47
+</details>
\ No newline at end of file
health/guides/mysql/mysql_replication.md
new
+79
@@ -0,0 +1,79 @@
1
+# mysql_replication
2
+
3
+## Database | MySQL, MariaDB
4
+
5
+This alert monitors the replication status of the MySQL server.
6
+If you receive this, either both or one of the I/O and SQL threads are not running.
7
+
8
+This alert is raised into critical if replication has stopped.
9
+
10
+
11
+> In MySQL, replication involves the source database writing down every change made to the data
12
+> held within one or more databases in a special file known as the binary log. Once the replica
13
+> instance has been initialized, it creates two threaded processes. The first, called the IO
14
+> thread, connects to the source MySQL instance and reads the binary log events line by line,
15
+> and then copies them over to a local file on the replica’s server called the relay log. The
16
+> second thread, called the SQL thread, reads events from the relay log and then applies them
17
+> to the replica instance as fast as possible.
18
+>
19
+> Recent versions of MySQL support two methods for replicating data. The difference between these
20
+> replication methods has to do with how replicas track which database events from the source
21
+> they’ve already processed.<sup>[1](
22
+> https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql) </sup>
23
+
24
+For further information, please have a look at the _References and Sources_ section.
25
+
26
+
27
+<details><summary>References and Sources</summary>
28
+
29
+1. [Replication in MySQL](
30
+ https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql)
31
+2. [MySQL documentation](
32
+ https://dev.mysql.com/doc/refman/5.7/en/replication-administration-status.html)
33
+3. [Section 8.14.6, “Replication Replica I/O
34
+ Thread States”](https://dev.mysql.com/doc/refman/5.7/en/replica-io-thread-states.html)
35
+4. [Section 8.14.7, “Replication Replica SQL Thread
36
+ States”](https://dev.mysql.com/doc/refman/5.7/en/replica-sql-thread-states.html)
37
+</details>
38
+
39
+### Troubleshooting Section
40
+
41
+<details><summary>Check which thread is not running</summary>
42
+
43
+From the MySQL command line you can run:
44
+
45
+- For MySQL and MariaDB before v10.2.0:
46
+
47
+ ```
48
+ SHOW SLAVE STATUS\G
49
+ ```
50
+- For MariaDB v10.2.0+:
51
+
52
+ ```
53
+ SHOW ALL SLAVES STATUS\G
54
+ ```
55
+
56
+This will show you three important rows among other info:
57
+
58
+> - Slave_IO_State:
59
+ The current status of the replica. See [Section 8.14.6, “Replication Replica I/O
60
+ Thread States”](https://dev.mysql.com/doc/refman/5.7/en/replica-io-thread-states.html), and
61
+ [Section 8.14.7, “Replication Replica SQL Thread
62
+ States”](https://dev.mysql.com/doc/refman/5.7/en/replica-sql-thread-states.html), for more
63
+ information.
64
+>
65
+>
66
+> - Slave_IO_Running:
67
+ Whether the I/O thread for reading the source's binary log is running.
68
+ Normally, you want this to be **Yes** unless you have not yet started replication or have
69
+ explicitly stopped it with STOP SLAVE.
70
+>
71
+>
72
+> - Slave_SQL_Running:
73
+ Whether the SQL thread for executing events in the relay log is running. As
74
+ with the I/O thread, this should normally be **Yes**.<sup> [2](
75
+ https://dev.mysql.com/doc/refman/5.7/en/replication-administration-status.html) </sup>
76
+
77
+For more info you can refer to the [MySQL documentation](
78
+https://dev.mysql.com/doc/refman/5.7/en/replication-administration-status.html).
79
+</details>
health/guides/mysql/mysql_replication_lag.md
new
+60
@@ -0,0 +1,60 @@
1
+# mysql_replication_lag
2
+
3
+## Database | MySQL, MariaDB
4
+
5
+This alert presents the number of seconds that the replica is behind the master.
6
+Receiving this means that the replication SQL thread is far behind processing the source binary log.
7
+A constantly high value (or an increasing one) indicates that the replica is unable to handle events
8
+from the source in a timely fashion.
9
+
10
+This alert is raised into warning when the metric exceeds 10 seconds.
11
+If the number of seconds that the replica is behind the master exceeds 30 seconds then the alert is
12
+raised into critical.
13
+
14
+
15
+> In MySQL, replication involves the source database writing down every change made to the data
16
+> held within one or more databases in a special file known as the binary log. Once the replica
17
+> instance has been initialized, it creates two threaded processes. The first, called the IO
18
+> thread, connects to the source MySQL instance and reads the binary log events line by line,
19
+> and then copies them over to a local file on the replica’s server called the relay log. The
20
+> second thread, called the SQL thread, reads events from the relay log and then applies them
21
+> to the replica instance as fast as possible.
22
+>
23
+> Recent versions of MySQL support two methods for replicating data. The difference between these
24
+> replication methods has to do with how replicas track which database events from the source
25
+> they’ve already processed.<sup>[1](
26
+> https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql) </sup>
27
+
28
+For further information, please have a look at the _References and Sources_ section.
29
+
30
+<details><summary>References and Sources</summary>
31
+
32
+1. [Replication in MySQL](
33
+ https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql)
34
+2. [MySQL Replication Slave Lag](
35
+ https://www.percona.com/blog/2014/05/02/how-to-identify-and-cure-mysql-replication-slave-lag/)
36
+3. [log_slow_slave_statements](
37
+ https://dev.mysql.com/doc/refman/8.0/en/replication-options-replica.html#sysvar_log_slow_slave_statements)
38
+4. [long_query_time](
39
+ https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_long_query_time)
40
+5. [log_slow_verbosity](
41
+ https://www.percona.com/doc/percona-server/5.1/diagnostics/slow_extended.html?id=percona-server:features:slow_extended_51&redirect=2#log_slow_verbosity)
42
+
43
+</details>
44
+
45
+### Troubleshooting Section
46
+
47
+<details><summary>Query optimization and "log_slow_slave_statements"</summary>
48
+
49
+To minimize slave `SQL_THREAD` lag, focus on query optimization. The following logs will help you identify the problem:
50
+1. Enable [log_slow_slave_statements](
51
+https://dev.mysql.com/doc/refman/8.0/en/replication-options-replica.html#sysvar_log_slow_slave_statements)
52
+to see queries executed by slave that take more than [long_query_time](
53
+https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_long_query_time).
54
+2. To get more information about query performance, set the configuration option [log_slow_verbosity](
55
+https://www.percona.com/doc/percona-server/5.1/diagnostics/slow_extended.html?id=percona-server:features:slow_extended_51&redirect=2#log_slow_verbosity) to `full`.
56
+
57
+You can also read the Percona blog for a nice write-up about[MySQL replication slave lag](
58
+https://www.percona.com/blog/2014/05/02/how-to-identify-and-cure-mysql-replication-slave-lag/).
59
+
60
+</details>
health/guides/net/10min_fifo_errors.md
new
+71
@@ -0,0 +1,71 @@
1
+# 10min_fifo_errors
2
+
3
+## OS: Linux
4
+
5
+Between the IP stack and the Network Interface Controller (NIC) lies the driver queue. This queue is typically
6
+implemented as a FIFO ring buffer into the memory space allocated by the driver. The NIC receive frames and place them into
7
+memory as skb_buff data structures (SocKet Buffer). We can have queues (ingress queues) and transmitted (egress queues)
8
+but these queues do not contain any actual packet data. Each queue has a pointer to the devices associated with it, and
9
+to the skb_buff data structures that store the ingress/egress packets. The number of frames this queue can handle is
10
+limited. Queues fill up when an interface receives packets faster than kernel can process them.
11
+
12
+The Netdata agent monitors the number of FIFO errors (number of times an overflow occurs in the ring buffer) for a specific
13
+network interface in the last 10 minutes. This alarm is triggered when the NIC is not able to handle the peak load of
14
+incoming/outgoing packets with the current ring buffer size.
15
+
16
+Not all NICs support FIFO queue operations.
17
+
18
+<details>
19
+<summary>See more on SKB</summary>
20
+The SocKet Buffer (SKB), is the most fundamental data structure in the Linux networking code. Every packet sent or
21
+received is handled using this data structure. This is a large struct containing all the control information required
22
+for the packet (datagram, cell, etc).
23
+
24
+The struct sk_buff has the following fields to point to the specific network layer headers:
25
+
26
+- transport_header (previously called h) – This field points to layer 4, the transport layer (and can include tcp header or udp header or
27
+ icmp header, and more)
28
+
29
+- network_header (previously called nh) – This field points to layer 3, the network layer (and can include ip header or ipv6 header or arp
30
+ header).
31
+
32
+- mac_header (previously called mac) – This field points to layer 2, the link layer.
33
+
34
+- skb_network_header(skb), skb_transport_header(skb) and skb_mac_header(skb) - These return pointer to the header.
35
+
36
+</details>
37
+
38
+### Troubleshooting section:
39
+
40
+<details>
41
+<summary>Update the ring buffer size</summary>
42
+
43
+1. To view the maximum RX ring buffer size:
44
+
45
+ ```
46
+ root@netdata ~ # ethtool -g enp1s0
47
+ Ring parameters for enp1s0:
48
+ Pre-set maximums:
49
+ RX: 4080
50
+ RX Mini: 0
51
+ RX Jumbo: 16320
52
+ TX: 255
53
+ Current hardware settings:
54
+ RX: 255
55
+ RX Mini: 0
56
+ RX Jumbo: 0
57
+ TX: 255
58
+ ```
59
+
60
+2. If the values in the Pre-set maximums section are higher than in the Current hardware settings section, increase RX (
61
+ or TX) ring buffer:
62
+
63
+ ```
64
+ root@netdata ~ # enp1s0 rx 4080
65
+ ```
66
+
67
+3. Verify the change to make sure that you no longer receive the alarm when running the same workload. To
68
+ make this permanently, you must consult your distribution guides.
69
+
70
+</details>
71
+
health/guides/net/10s_received_packets_storm.md
new
+80
@@ -0,0 +1,80 @@
1
+# 10s_received_packets_storm
2
+
3
+## OS: Linux, FreeBSD
4
+
5
+A packet storm is defined as an unusually high amount of traffic on a specific interface. In a sliding window of one minute
6
+Netdata agent monitors for significant increases of packets (ratio of an average number of received packets) in the last
7
+10 seconds. If this system is expected to have spikes, you can cautiously ignore this alarm, but should continue to monitor as this
8
+alarm may also indicate a broadcast/multicast storm or DoS attack.
9
+
10
+<details>
11
+<summary>See more on broadcast storms.</summary>
12
+
13
+A broadcast storm is the accumulation of broadcast and multicast traffic on a computer network. Extreme amounts of
14
+broadcast traffic constitute a "broadcast storm". It can consume sufficient network resources so as to render the
15
+network unable to transport normal traffic. Most commonly the cause is a switching loop in the Ethernet wiring topology.
16
+As broadcasts and multicasts are forwarded by switches out of every port, the switch or switches will repeatedly
17
+rebroadcast broadcast messages and flood the network. Since the Layer 2 header does not support a time to live (TTL)
18
+value, if a frame is sent into a looped topology, it can loop forever.
19
+
20
+</details>
21
+
22
+
23
+<details>
24
+<summary>See more on DoS attacks.</summary>
25
+
26
+A Denial-of-Service (DoS) attack is an attack meant to shut down a machine or network, making it inaccessible to its
27
+intended users. DoS attacks accomplish this by flooding the target with traffic, or sending it information that triggers
28
+a crash. We can categorize the attacks into two types.
29
+
30
+- Infrastructure Layer Attacks:
31
+
32
+ Attacks at Layer 3 and 4 of the OSI model are typically categorized as Infrastructure layer attacks. The most common
33
+ type of DDoS attack include vectors, like synchronized (SYN) floods, and other reflection attacks, like User Datagram
34
+ Packet (UDP) floods. These attacks are usually large in volume and aim to overload the capacity of the network or the
35
+ application servers. Fortunately, these are also the type of attacks that have clear signatures and are easier to
36
+ detect.
37
+
38
+- Application Layer Attacks:
39
+
40
+ Attacks at Layer 6 and 7 of the OSI model, are often categorized as Application layer attacks. While these attacks are
41
+ less common, they also tend to be more complex. These attacks are typically small in volume compared to the
42
+ Infrastructure layer attacks, but tend to focus on particular expensive parts of the application, thereby making it
43
+ unavailable for real users. Common examples of this type of attack include a flood of HTTP requests to a login page, or an expensive search API, or
44
+ even Wordpress XML-RPC floods.
45
+
46
+</details>
47
+
48
+
49
+### Troubleshooting section:
50
+
51
+</details>
52
+
53
+<details>
54
+<summary>Counter measures on DoS and DDoS attacks</summary>
55
+
56
+- Use a service like Cloudflare. Cloudflare DDoS protection secures websites, applications, and entire networks while
57
+ ensuring the performance of legitimate traffic is not compromised.
58
+
59
+- Limit broadcasting. Often attacks will send requests to every device on the network, amplifying the attack. Limiting
60
+ or turning off broadcast forwarding where possible can disrupt attacks. Users can also disable echo and chargen
61
+ services where possible.
62
+
63
+</details>
64
+
65
+
66
+</details>
67
+
68
+<details>
69
+
70
+<summary>Counter measures on broadcast storms</summary>
71
+
72
+- Switching loops are largely addressed through link aggregation, shortest path bridging, or spanning tree protocol. In
73
+ Metro Ethernet rings, it is prevented using the Ethernet Ring Protection Switching (ERPS) or Ethernet Automatic
74
+ Protection System (EAPS) protocols.
75
+
76
+- You can filter broadcasts by Layer 3 equipment, most typically routers or even switches that employ advanced filtering.
77
+
78
+- Routers and firewalls can be configured to detect and prevent maliciously inducted broadcast storms
79
+
80
+</details>
\ No newline at end of file
health/guides/net/1m_received_traffic_overflow.md
new
+48
@@ -0,0 +1,48 @@
1
+# 1m_received_traffic_overflow
2
+
3
+## OS: Linux
4
+
5
+Network interfaces are categorized primarily on the bandwidth they can operate (1 Gbps, 10 Gbps, etc). High network
6
+utilization occurs when the volume of data on a network link approaches the capacity of the link. Netdata agent
7
+calculates the average outbound utilization for a specific network interface over the last minute. High outbound
8
+utilization increases latency and packet loss because packet bursts are buffered
9
+
10
+This alarm may indicate either network congestion or malicious activity.
11
+
12
+### Troubleshooting section
13
+
14
+<details>
15
+ <summary>Prioritize important traffic</summary>
16
+
17
+Quality of service (QoS) is the use of routing prioritization to control traffic and ensure the performance of
18
+critical applications. QoS works best when low-priority traffic exists that can be dropped when congestion occurs. The
19
+higher-priority traffic must fit within the bandwidth limitations of the link or path. The following are two open source
20
+solutions to apply QoS policies to your network interfaces.
21
+
22
+- `FireQOS`:
23
+
24
+ FireQOS is a traffic shaping helper. It has a very simple shell scripting language to express traffic shaping.
25
+
26
+ [See more on FireQOS](https://firehol.org/tutorial/fireqos-new-user/)
27
+
28
+- `tcconfig`:
29
+
30
+ Tcconfig is a command wrapper that makes it easy to set up traffic control of network
31
+ bandwidth/latency/packet-loss/packet-corruption/etc.
32
+
33
+ [See more on tcconfig](https://tcconfig.readthedocs.io/en/latest/index.html)
34
+
35
+</details>
36
+
37
+
38
+<details>
39
+ <summary>Add bandwidth</summary>
40
+
41
+- For **Cloud infrastructures**, adding bandwidth might be easy. It depends on your cloud infrastracture and your cloud
42
+ provider. Some of them either offer you the service to upgrade machines to a higher bandwidth rate or upgrade you
43
+ machine to a more powerful one with higher bandwidth rate.
44
+
45
+- For **Bare-metal** machines, you will need either a hardware upgrade or the addition of a network card using link
46
+ aggregation to combine multiple network connections in parallel (e.g LACP).
47
+
48
+</details>
health/guides/net/1m_sent_traffic_overflow.md
new
+47
@@ -0,0 +1,47 @@
1
+# 1m_sent_traffic_overflow
2
+
3
+## OS: Linux
4
+
5
+Network interfaces are categorized primarily on the bandwidth rate at which they can operate (1 Gbps, 10 Gbps, etc). High network
6
+utilization occurs when the volume of data on a network link approaches the capacity of the link. Netdata agent
7
+calculates the average outbound utilization for a specific network interface over the last minute. High outbound
8
+utilization increases latency and packet loss because packet bursts are buffered.
9
+
10
+This alarm may indicate either a network congestion or malicious activity.
11
+
12
+### Troubleshooting section
13
+
14
+<details>
15
+ <summary>Prioritize important traffic</summary>
16
+
17
+Quality of service (QoS) is the use of mechanisms or technologies to control traffic and ensure the performance of
18
+critical applications. QoS works best when low-priority traffic exists that can be dropped when congestion occurs. The
19
+higher-priority traffic must fit within the bandwidth limitations of the link or path. The following are two open source
20
+solutions to apply QoS policies to your network interfaces.
21
+
22
+- `FireQOS`:
23
+
24
+ FireQOS is a traffic shaping helper. It has a very simple shell scripting language to express traffic shaping.
25
+
26
+ [See more on FireQOS](https://firehol.org/tutorial/fireqos-new-user/)
27
+
28
+- `tcconfig`:
29
+
30
+ Tcconfig is a command wrapper that makes it easy to set up traffic control of network
31
+ bandwidth, latency, packet-loss, packet-corruption, etc.
32
+
33
+ [See more on tcconfig](https://tcconfig.readthedocs.io/en/latest/index.html)
34
+
35
+</details>
36
+
37
+<details>
38
+ <summary>Add bandwidth</summary>
39
+
40
+- For **Cloud infrastructures**, adding bandwidth might be easy. It depends on your cloud infrastracture and your cloud
41
+ provider. Some of them either offer you the service to upgrade machines to a higher bandwidth rate or upgrade you
42
+ machine to a more powerful one with higher bandwidth rate.
43
+
44
+- For **Bare-metal** machines you will need either a hardware upgrade or the addition of a network card using link
45
+ aggregation to combine multiple network connections in parallel (e.g LACP).
46
+
47
+</details>
health/guides/net/inbound_packets_dropped_ratio.md
new
+83
@@ -0,0 +1,83 @@
1
+# inbound_packets_dropped_ratio
2
+
3
+## OS: Linux
4
+
5
+Packet drops indicate that your system received some packets but could not process them. A sizeable amount of packet
6
+drops can consume significant amount of resources in your system. Some reasons that packets drops occurred in your
7
+system could be:
8
+
9
+- Your system receives packets with bad VLAN tags.
10
+- The packets you are receiving are using a protocol that is unknown to your system.
11
+- You receive IPv6 packets, but your system is not configured for IPv6.
12
+
13
+All these packets consume resources until being dropped (and for a short period after). For example, your NIC stores them
14
+in a ring-buffer until they are forwarded to the destined subsystem or userland application for further process.
15
+
16
+Netdata agent calculates the ratio of inbound dropped packets for your wired network interface over the last 10 minutes.
17
+
18
+### Troubleshooting section:
19
+
20
+The best way to resolve these kinds of problems is to be extremely knowledgeable about your network topologies. These
21
+are some actions you might consider check:
22
+
23
+<details>
24
+ <summary>Identify VLANs in your interface</summary>
25
+
26
+There are cases in which traffic is routed to your host due to the existence of multiple VLAN in your network.
27
+
28
+1. Identify VLAN tagged packet in your interface.
29
+
30
+```
31
+root@netdata # tcpdump -i <your_interface> -nn -e vlan
32
+```
33
+
34
+2. Monitor the output of the `tcpdump`, identify VLANs which may exist. If no output is displayed, your interface
35
+ probably uses traditional ethernet frames.
36
+
37
+3. Depending on your network topology, you may consider removing unnecessary VLANs from the switch trunk port
38
+ toward your host.
39
+
40
+</details>
41
+
42
+<details>
43
+<summary>Update the ring buffer size on your interface</summary>
44
+
45
+1. To view the maximum RX ring buffer size:
46
+
47
+ ```
48
+ root@netdata ~ # ethtool -g enp1s0
49
+ Ring parameters for enp1s0:
50
+ Pre-set maximums:
51
+ RX: 4080
52
+ RX Mini: 0
53
+ RX Jumbo: 16320
54
+ TX: 255
55
+ Current hardware settings:
56
+ RX: 255
57
+ RX Mini: 0
58
+ RX Jumbo: 0
59
+ TX: 255
60
+ ```
61
+
62
+2. If the values in the Pre-set maximums section are higher than in the current hardware settings section, increase RX
63
+ ring buffer:
64
+
65
+ ```
66
+ root@netdata ~ # enp1s0 rx 4080
67
+ ```
68
+
69
+3. Verify the change to make sure that you no longer receive the alarm when running the same workload. To make this
70
+ permanently, you must consult your distribution guides.
71
+
72
+</details>
73
+
74
+<details>
75
+<summary>Inspect the packets your network interface receives</summary>
76
+
77
+Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and
78
+communications protocol development.
79
+
80
+[See more about Wireshark here](https://www.wireshark.org/)
81
+
82
+</details>
83
+
health/guides/net/interface_inbound_errors.md
new
+120
@@ -0,0 +1,120 @@
1
+# interface_inbound_errors
2
+
3
+## OS: FreeBSD
4
+
5
+When a packet is received by your system, it can be processed in one of four ways:
6
+
7
+- It can be passed as input to a higher-level protocol.
8
+
9
+- It can encounter an error which is reported back to the source.
10
+
11
+- It can be dropped due to an error.
12
+
13
+- It can be forwarded to the next hop on its path to its destination.
14
+
15
+There are mechanisms to identify packets with errors and verify the integrity of the packet such as the Cyclic
16
+Redundancy Check (CRC), the Frame check sequence (FCS), the header checksum (IPv4), and length checks. The Netdata agent
17
+monitors the number of inbound errors for a specific network interface in the last 10 minutes.
18
+
19
+
20
+<details>
21
+<summary>The life of a packet</summary>
22
+
23
+The following list from "Design and Implementation of the FreeBSD Operating System, The, 2nd Edition" (McKusick,
24
+Neville-Neil and Watson) [[1]](https://www.pearson.com/us/higher-education/program/Mc-Kusick-Design-and-Implementation-of-the-Free-BSD-Operating-System-The-2nd-Edition/PGM224032.html)
25
+provides a brief description of every action taken by your system for every packet it receives:
26
+
27
+1. Verifies that the packet is at least as long as an IPv4 or IPv6 header and ensures that the header is contiguous.
28
+
29
+2. For IPv4, checksums the header of the packet, and discards the packet if there is an error.
30
+
31
+3. Verifies that the packet is at least as long as the header indicates, and drops the packet if it is not.
32
+
33
+4. Does any filtering or security functions (ipfw, IPSec).
34
+
35
+5. Processes any options associated with the header.
36
+
37
+6. Checks whether the packet is for this host. If it is, continues processing the packet. If it is not, and if the
38
+ system is acting as a router, your system will try to forward the packet. Otherwise, the packet is dropped.
39
+
40
+7. If the packet has been fragmented, keeps it until all its fragments are received and reassembled, If the reassemble
41
+ process takes a significant amount of time, the system drops it.
42
+
43
+8. Passes the packet to the input routine of the next-higher-level protocol.
44
+
45
+</details>
46
+
47
+<details>
48
+<summary>See more on CRC</summary>
49
+
50
+> A cyclic redundancy check (CRC) is an error-detecting code commonly used in digital networks and storage devices to
51
+> detect accidental changes to raw data. Blocks of data entering these systems get a short check value attached, based
52
+> on the remainder of a polynomial division of their contents. On retrieval, the calculation is repeated and, in the
53
+> event the check values do not match, corrective action can be taken against data
54
+> corruption. [[2]](https://en.wikipedia.org/wiki/Cyclic_redundancy_check)
55
+
56
+</details>
57
+
58
+<details>
59
+<summary>See more on FCS</summary>
60
+
61
+> A frame check sequence (FCS) is an error-detecting code added to a frame in a communication protocol. All frames and
62
+> the bits, bytes, and fields contained within them, are susceptible to errors from a variety of sources. The FCS field
63
+> contains a number that is calculated by the source node based on the data in the frame. This number is added to the
64
+> end of a frame that is sent. When the destination node receives the frame the FCS number is recalculated and compared
65
+> with the FCS number included in the frame. If the two numbers are different, an error is assumed and the frame
66
+> is discarded. [[3]](https://en.wikipedia.org/wiki/Frame_check_sequence)
67
+
68
+</details>
69
+
70
+<details>
71
+<summary>See more on header checksum</summary>
72
+
73
+> The IPv4 header checksum is a checksum used in version 4 of the Internet Protocol (IPv4) to detect corruption in the
74
+> header of IPv4 packets. It is carried in the IP packet header and represents the 16-bit result of summation of the
75
+> header words. [[4]](https://en.wikipedia.org/wiki/IPv4_header_checksum)
76
+
77
+</details>
78
+
79
+<details>
80
+<summary>References and sources</summary>
81
+
82
+1. [Book: Design and Implementation of the FreeBSD Operating System (2nd-Edition)](https://www.pearson.com/us/higher-education/program/Mc-Kusick-Design-and-Implementation-of-the-Free-BSD-Operating-System-The-2nd-Edition/PGM224032.html)
83
+
84
+1. [Cyclic redundancy check protocol](https://en.wikipedia.org/wiki/Cyclic_redundancy_check)
85
+
86
+1. [Frame check sequence protocol](https://en.wikipedia.org/wiki/Frame_check_sequence)
87
+
88
+1. [IPv4 header checksum protocol](https://en.wikipedia.org/wiki/IPv4_header_checksum)
89
+
90
+
91
+</details>
92
+
93
+### Troubleshooting section:
94
+
95
+<details>
96
+<summary>General approach</summary>
97
+
98
+In any case, a good starting point is to get more information about the nature of your errors.
99
+
100
+- Netdata dashboard provides an overview of these errors. You can see more in the `errors` chart under the IPv4 (or
101
+ IPv6) section.
102
+
103
+- `netstat` (network statistics) is a command-line network utility that displays network connections for Transmission
104
+ Control Protocol, routing tables and network protocol statistics for any interface in your system.
105
+
106
+ ```
107
+ root@netdata~ # netstat -sI <your_interface>
108
+ ```
109
+
110
+</details>
111
+
112
+<details>
113
+<summary>Troubleshoot hardware errors in the link of the interface</summary>
114
+
115
+You must identify which part of your topology causes these errors. Some actions you can take.
116
+
117
+- Remove and re-install the optical fibers and optical modules and check whether the fiber connectors are damaged or
118
+ contaminated. For ethernet interfaces check for damaged cables and/or for damages in the interfaces themselves.
119
+
120
+
\ No newline at end of file
health/guides/net/interface_outbound_errors.md
new
+75
@@ -0,0 +1,75 @@
1
+# interface_outbound_errors
2
+
3
+## OS: FreeBSD
4
+
5
+When we want to investigate the outbound traffic, the journey of a network packet starts at the application layer. Data
6
+are written (commonly) to a socket by a user program. The programmer may (raw sockets) or may not (datagram and stream
7
+sockets) have the possibility of absolute control over the data which is being sent through the network. The kernel will
8
+take the data which is written in a socket queue and allocate the necessary socket buffers. The kernel will try to
9
+forward the packets to their destination encapsulating the routing metadata (headers, checksums, fragmentation
10
+information) for each packet through a network interface. The Netdata agent monitors the number of outbound errors for a
11
+specific network interface in the last 10 minutes. Some of the errors that may occur in this process include:
12
+
13
+- Errors due to aborted connections
14
+
15
+- Carrier sense errors
16
+
17
+- FIFO errors
18
+
19
+- Heartbeat errors
20
+
21
+- Window errors
22
+
23
+<details>
24
+ <summary>See more on Carrier Sense Errors</summary>
25
+
26
+Carrier Sense Errors occur when an interface attempts to transmit a frame, but no carrier is detected. In that case if
27
+the frame cannot be transmitted, it is discarded.
28
+
29
+</details>
30
+
31
+
32
+<details>
33
+ <summary>See more about heartbeat </summary>
34
+
35
+> A heartbeat protocol is generally used to negotiate and monitor the availability of a resource, such as a floating IP
36
+> address, and the procedure involves sending network packets to all the nodes in the culture to verify its
37
+> reachability. Typically when a heartbeat starts on a machine, it will perform an election process with other machines
38
+> on the heartbeat network to determine which machine, if any, owns the resource. On heartbeat networks of more than two
39
+> machines, it is important to take into account partitioning, where two halves of the network could be functioning but
40
+> not able to communicate with each other. In a situation such as this, it is important that the resource is only owned
41
+> by one machine, not one machine in each partition.
42
+>
43
+> As a heartbeat is intended to be used to indicate the health of a machine, it is important that the heartbeat protocol
44
+> and the transport that it runs on are as reliable as possible. Causing a failover because of a false alarm may
45
+> depending on the resource, be highly undesirable. It is also important to react quickly to an actual failure, further
46
+> signifig the reliability of the heartbeat messages. For this reason, it is often desirable to have a heartbeat running
47
+> over more than one transport; for instance, an Ethernet segment using UDP/IP, and a serial
48
+> link. <sup> [1](https://en.wikipedia.org/wiki/Heartbeat_(computing)</sup>
49
+
50
+</details>
51
+
52
+
53
+<details>
54
+<summary>References and sources:</summary>
55
+
56
+1. [Heartbeat definition on Wikipedia](https://en.wikipedia.org/wiki/Heartbeat_(computing))
57
+
58
+</details>
59
+
60
+### Troubleshooting section:
61
+
62
+<details>
63
+<summary>General approach</summary>
64
+
65
+In any case, a good starting point is to get more information about the nature of your errors.
66
+
67
+- `netstat` (network statistics) is a command-line network utility that displays, network connections for Transmission
68
+ Control Protocol, routing tables and network protocol statistics for any interface in your system.
69
+
70
+ ```
71
+ root@netdata~ # netstat -sI <your_interface>
72
+ ```
73
+
74
+</details>
75
+
health/guides/net/outbound_packets_dropped_ratio.md
new
+41
@@ -0,0 +1,41 @@
1
+# outbound_packets_dropped_ratio
2
+
3
+## OS: Linux
4
+
5
+When we want to investigate the outbound traffic, the journey of a network packet starts at the
6
+application layer. Data are written (commonly) to a socket by a user program. The programmer may (
7
+raw sockets) or may not (datagram and stream sockets) have the possibility of absolute control over
8
+the data which is being sent through the network. The kernel will take the data which is written in
9
+a socket queue and allocate the necessary socket buffers. The kernel will try to forward the packets
10
+to their destination encapsulating the routing metadata (headers, checksums, fragmentation
11
+information) for each packet through a network interface.
12
+
13
+The Netdata Agent calculates the ratio of outbound dropped packets for a specific network interface
14
+over the last 10 minutes. Receiving this alarm means that packets were dropped on their way to
15
+transmission.
16
+
17
+This alert is triggered in warning state when the ratio of outbound dropped packets for a specific
18
+network interface over the last 10 minutes is more than 2%.
19
+
20
+The main reasons of outbound packet drops are:
21
+
22
+1. Link congestion
23
+1. Overburdened devices
24
+1. Defective hardware
25
+1. Faulty network configuration
26
+1. Restricted access from firewall rules
27
+
28
+### Troubleshooting section:
29
+
30
+The best way to resolve these kind of problems is to be extremely knowledgeable about your network
31
+topologies.
32
+
33
+<details>
34
+<summary>Inspect the packets your network interface sends</summary>
35
+
36
+Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting,
37
+analysis, software and communications protocol development.
38
+
39
+[See more about Wireshark here](https://www.wireshark.org/)
40
+
41
+</details>
health/guides/netfilter/netfilter_conntrack_full.md
new
+66
@@ -0,0 +1,66 @@
1
+# netfilter_conntrack_full
2
+
3
+## OS: Linux
4
+
5
+This alert presents the percentage of used netfilter tracked connections. \
6
+If you receive this alert, there is high utilization of the netfilter "connection tracking state"
7
+table size.
8
+
9
+Network delays and packet drops are expected when you are getting closer to 100%.
10
+
11
+- This alert gets raised to warning when the percentage exceeds 90%.
12
+- If the metric exceeds 95%, then the alert gets raised to a critical state.
13
+
14
+<details>
15
+<summary>What is the "netfilter" framework</summary>
16
+
17
+> Netfilter is a framework provided by the Linux kernel that allows various networking-related
18
+> operations to be implemented in the form of customized handlers. Netfilter offers various
19
+> functions and operations for packet filtering, network address translation, and port
20
+> translation, which provide the functionality required for directing packets through a network
21
+> and prohibiting packets from reaching sensitive locations within a network.<sup>[1](
22
+> https://en.wikipedia.org/wiki/Netfilter) </sup>
23
+
24
+</details>
25
+
26
+<br>
27
+
28
+<details>
29
+<summary>References and Sources</summary>
30
+
31
+1. [Netfilter](https://en.wikipedia.org/wiki/Netfilter)
32
+2. [Full Conntrack Table](
33
+ https://morganwu277.github.io/2018/05/26/Solve-production-issue-of-nf-conntrack-table-full-dropping-packet/)
34
+
35
+</details>
36
+
37
+### Troubleshooting Section
38
+
39
+<details>
40
+<summary>Table Size Limits</summary>
41
+
42
+You can see the table size by running:
43
+
44
+```
45
+root@netdata~ # cat /proc/sys/net/netfilter/nf_conntrack_count
46
+```
47
+
48
+You can see the table size limit by running:
49
+
50
+```
51
+root@netdata~ # cat /proc/sys/net/netfilter/nf_conntrack_max
52
+```
53
+
54
+> Recommended Size: CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 / (ARCH / 32).
55
+> Eg, If we have 8GB RAM in a x86_64 OS, we would use 8*1024^3/16384/2=262144.<sup> [2](
56
+> https://morganwu277.github.io/2018/05/26/Solve-production-issue-of-nf-conntrack-table-full-dropping-packet/)
57
+> </sup>
58
+
59
+You can modify the table size limit by running:
60
+
61
+```
62
+root@netdata~ # sysctl -w net.netfilter.nf_conntrack_max=<YOUR DESIRED LIMIT HERE>
63
+root@netdata~ # echo "net.netfilter.nf_conntrack_max=<YOUR DESIRED LIMIT HERE>" >> /etc/sysctl.conf
64
+```
65
+
66
+</details>
health/guides/pihole/pihole_blocklist_gravity_file.md
new
+30
@@ -0,0 +1,30 @@
1
+# pihole_blocklist_gravity_file
2
+
3
+## Ad Filtering | Pi-hole
4
+
5
+This alert indicates the existence of the blocklist file. If you receive this, it means that the
6
+gravity.list file (blocklist) is non-existent.
7
+
8
+- The alert is raised in a critical state when the metric gets the value of 1.
9
+
10
+> The gravity table consists of the domains that have been processed by Pi-hole's gravity
11
+> (pihole -g) command. The domains in this list are the collection of domains sourced from the
12
+> configured sources (see the adlist table).<sup>[1](
13
+> https://docs.pi-hole.net/database/gravity/#gravity-table-gravity) </sup>
14
+<details><summary>References and Sources</summary>
15
+
16
+1. [Pi-hole Docs](https://docs.pi-hole.net/database/gravity/#gravity-table-gravity)
17
+
18
+</details>
19
+
20
+### Troubleshooting Section
21
+
22
+<details>
23
+<summary>Rebuild the blocklist</summary>
24
+To rebuild the gravity.list (blocklist), run the command:
25
+
26
+```
27
+root@netdata~ # pihole -g
28
+```
29
+
30
+</details>
\ No newline at end of file
health/guides/pihole/pihole_blocklist_last_update.md
new
+31
@@ -0,0 +1,31 @@
1
+# pihole_blocklist_last_update
2
+
3
+## Ad Filtering | Pi-hole
4
+
5
+This alert presents how much time has passed from the time the blocklist file (Gravity) was
6
+updated in seconds.
7
+Receiving this means that the blocklist file has not been updated for a long time.
8
+
9
+- This alert is raised to warning when the time in seconds exceeds 8 days.
10
+- If the metric exceeds 16 days, then the alert is raised to critical.
11
+
12
+> The gravity table consists of the domains that have been processed by Pi-hole's gravity
13
+> (pihole -g) command. The domains in this list are the collection of domains sourced from the
14
+> configured sources (see the adlist table).<sup>[1](
15
+> https://docs.pi-hole.net/database/gravity/#gravity-table-gravity) </sup>
16
+
17
+<details><summary>References and Sources</summary>
18
+
19
+1. [Pi-hole Docs](https://docs.pi-hole.net/database/gravity/#gravity-table-gravity)
20
+</details>
21
+
22
+### Troubleshooting Section
23
+
24
+<details>
25
+<summary>Rebuild the blocklist</summary>
26
+To rebuild the blocklist, run the command:
27
+
28
+```
29
+root@netdata~ # pihole -g
30
+```
31
+</details>
health/guides/pihole/pihole_status.md
new
+21
@@ -0,0 +1,21 @@
1
+# pihole_status
2
+
3
+## Ad Filtering | Pi-hole
4
+
5
+This alert monitors if Pi-hole's ability of blocking unwanted domains is active.
6
+
7
+- It is triggered in a warning state if pi-hole is disabled.
8
+
9
+
10
+### Troubleshooting Section
11
+
12
+<details>
13
+<summary>Rebuild the blocklist</summary>
14
+To fix, run the command:
15
+
16
+```
17
+root@netdata~ # pihole enable
18
+```
19
+
20
+*This feature should be enabled. The whole point of Pi-hole!*
21
+</details>
\ No newline at end of file
health/guides/ping/ping_host_latency.md
new
+50
@@ -0,0 +1,50 @@
1
+# fping_host_latency
2
+
3
+**Other | Network**
4
+
5
+`fping` is a command line tool to send ICMP (Internet Control Message Protocol) echo requests to
6
+network hosts, similar to ping, but performing much better when pinging multiple hosts. The Netdata
7
+Agent utilizes `fping` to monitor latency, packet loss, uptime and reachability of any number of
8
+network endpoints.
9
+
10
+For the `fping_host_latency` alert, the Netdata Agent monitors the average latency to the network
11
+host over the last 10 seconds. Receiving this alert indicates high latency to the network host. It is
12
+likely you are experiencing networking issues or the host is overloaded.
13
+
14
+### Troubleshooting section
15
+
16
+<details>
17
+ <summary>Customize the ICMP requests for each endpoint</summary>
18
+
19
+Different endpoints could be in different networks. For example, a server in your intra network
20
+would require less time to be accessed than your cloud infrastructures in terms of latency. You
21
+should always consider not to use a global approach for checking every endpoint of yours. You can
22
+find more information about how to configure every endpoint separately in
23
+the [fping.plugin alarm guide](https://learn.netdata.cloud/docs/agent/collectors/fping.plugin/#additional-tips).
24
+
25
+</details>
26
+
27
+<details>
28
+ <summary>Prioritize traffic on your endpoints</summary>
29
+
30
+Quality of service (QoS) is the use of mechanisms or technologies to control traffic and ensure the
31
+performance of critical applications. QoS works best when low-priority traffic exists that can be
32
+dropped when congestion occurs. The higher-priority traffic must fit within the bandwidth
33
+limitations of the link or path. The following are two open source solutions to apply QoS policies
34
+to your network interfaces.
35
+
36
+- `FireQOS`:
37
+
38
+ FireQOS is a traffic shaping helper. It has a very simple shell scripting language to express
39
+ traffic shaping.
40
+
41
+ [See more on FireQOS](https://firehol.org/tutorial/fireqos-new-user/)
42
+
43
+- `tcconfig`:
44
+
45
+ Tcconfig is a command wrapper that makes it easy to set up traffic control of network bandwidth,
46
+ latency, packet-loss, packet-corruption, etc.
47
+
48
+ [See more on tcconfig](https://tcconfig.readthedocs.io/en/latest/index.html)
49
+
50
+</details>
health/guides/ping/ping_host_reachable.md
new
+85
@@ -0,0 +1,85 @@
1
+# fping_host_reachable
2
+
3
+**Other | Network**
4
+
5
+`fping` is a command line tool to send ICMP (Internet Control Message Protocol) echo requests to
6
+network hosts, similar to ping, but performing much better when pinging multiple hosts. The Netdata
7
+Agent utilizes `fping` to monitor latency, packet loss, uptime and reachability of any number of
8
+network endpoints.
9
+
10
+The `fping_host_reachable` alert in the Netdata Agent checks the reachability of a network
11
+host (0: unreachable, 1: reachable). Receiving a critical alert indicates that your
12
+endpoints are unreachable. It is likely that the host is down or your system is experiencing
13
+networking issues.
14
+
15
+### Troubleshooting
16
+
17
+<details>
18
+<summary>Check network connectivity</summary>
19
+
20
+Verify that your system has access to the particular endpoint. Check for basic connectivity to known
21
+hosts from both your host and the endpoint.
22
+
23
+</details>
24
+
25
+<details>
26
+<summary>DNS settings</summary>
27
+
28
+If you are using DNS resolution to check your endpoint, you should always consider check your DNS
29
+settings. To troubleshoot this issue, verify that your DNS can resolve your endpoints.
30
+
31
+1. Check your current DNS (for example in linux you can use the host command):
32
+
33
+ ```
34
+ root@netdata # host -v <your_endpoint>
35
+ ```
36
+
37
+2. If the HTTP endpoint is supposed to be public facing endpoint, try an alternative DNS (for
38
+ example Cloudflare's DNS):
39
+
40
+ ```
41
+ root@netdata # host -v <your_endpoint> 1.1.1.1
42
+ ```
43
+</details>
44
+
45
+<details>
46
+<summary>Verify access restrictions in the remote host</summary>
47
+
48
+If the remote host is a Linux-based machine and you have access to it, you can check the followings.
49
+
50
+**Check the ICMP settings**
51
+
52
+In most linux distributions you can restrict the ICMP echo operations.
53
+
54
+ 1. Check your current setting.
55
+
56
+ ```
57
+ root@netdata # systemctl net.ipv4.icmp_echo_ignore_all
58
+ ```
59
+
60
+
61
+ If this value is set to 1 your system ignore incoming ICMP echo requests.
62
+
63
+ 2. To change this, bump this `net.ipv4.icmp_echo_ignore_all=0` entry under `/etc/sysctl.conf`.
64
+
65
+ 3. Reload the sysctl settings.
66
+
67
+ ```
68
+ root@netdata # sysctl -p
69
+ ```
70
+
71
+ **Check your firewall rules**
72
+
73
+ Depending on what firewall you use, the commands might differ from what's shown below. For example, if you are using
74
+ IP tables you can check for restriction rules upon `icmp`.
75
+
76
+ ```
77
+ root@netdata # iptables -L | grep ICMP
78
+ ```
79
+
80
+ For futher investigation or changes in your firewall settings we **strongly** advise you to consult
81
+ your firewall's documentation and guidelines.
82
+
83
+
84
+</details>
85
+
health/guides/ping/ping_packet_loss.md
new
+40
@@ -0,0 +1,40 @@
1
+# fping_packet_loss
2
+
3
+## OS: Any
4
+
5
+`fping` is a command line tool to send ICMP (Internet Control Message Protocol) echo requests to
6
+network hosts, similar to ping, but performing much better when pinging multiple hosts. The Netdata
7
+Agent utilizes `fping` to monitor latency, packet loss, uptime and reachability of any number of
8
+network endpoints.
9
+
10
+For `fping_packet_loss`, the Netdata Agent calculates the packet loss ratio to a network host
11
+over the last 10 minutes. Receiving this alert indicates high packet loss towards a network host.
12
+This could be caused by link congestion, link node faults, high server load, or incorrect system
13
+settings.
14
+
15
+### Troubleshooting
16
+
17
+<details>
18
+ <summary>Prioritize important traffic on your endpoint (linux based endpoints)</summary>
19
+
20
+Quality of service (QoS) is the use of mechanisms or technologies to control traffic and ensure the
21
+performance of critical applications. QoS works best when low-priority traffic exists that can be
22
+dropped when congestion occurs. The higher-priority traffic must fit within the bandwidth
23
+limitations of the link or path. The following are two open source solutions to apply QoS policies
24
+to your network interfaces.
25
+
26
+- `FireQOS`:
27
+
28
+ FireQOS is a traffic shaping helper. It has a very simple shell scripting language to express
29
+ traffic shaping.
30
+
31
+ [See more on FireQOS](https://firehol.org/tutorial/fireqos-new-user/)
32
+
33
+- `tcconfig`:
34
+
35
+ Tcconfig is a command wrapper that makes it easy to set up traffic control of network bandwidth,
36
+ latency, packet-loss, packet-corruption, etc.
37
+
38
+ [See more on tcconfig](https://tcconfig.readthedocs.io/en/latest/index.html)
39
+
40
+</details>
health/guides/portcheck/portcheck_connection_fails.md
new
+31
@@ -0,0 +1,31 @@
1
+# portcheck_connection_fails
2
+
3
+**Other | TCP endpoint**
4
+
5
+The Netdata Agent calculates the average ratio of failed connections over the last 5 minutes. This
6
+alert indicates that too many connections failed. Receiving this alert means that your endpoint is
7
+unreachable due to:
8
+
9
+1. The service is no longer running or not working properly.
10
+2. Access to this port is denied by a firewall.
11
+3. Port forwarding rile is incorrectly configured
12
+4. The IP of the node you want to access is set to a private IP address
13
+
14
+This alert is triggered in warning state when the ratio of failed connections is between 10-40% and
15
+in critical state when it is greater than 40%.
16
+
17
+### Troubleshooting section
18
+
19
+<details>
20
+<summary>Check the firewall rules in the remote</summary>
21
+
22
+Check the INPUT chain rules, verify that you have allowed access from the host (Agent configured in it)
23
+to the remote node.
24
+
25
+**IPtables**
26
+
27
+ ```
28
+ root@netdata # iptables -L INPUT
29
+ ```
30
+
31
+</details>
\ No newline at end of file
health/guides/portcheck/portcheck_connection_timeouts.md
new
+42
@@ -0,0 +1,42 @@
1
+# portcheck_connection_timeouts
2
+
3
+**Other | TCP endpoint**
4
+
5
+TCP provides a “virtual connection” between two nodes. When there is no activity, “keep alive”
6
+packet are exchanged on a regular interval. Should the “keep alive” not arrive after specified
7
+amount of time, the “connection times out” because there was not traffic during the timeout
8
+interval.
9
+
10
+The Netdata Agent calculates the average ratio of timeouts over the last 5 minutes. Receiving this
11
+alert means that the monitored endpoint is either unreachable or most likely you are experiencing
12
+networking issues or, the remote host/service is overloaded.
13
+
14
+This alert is triggered in warning state when the ratio of timeouts is between 10-40% and in
15
+critical state when it is greater than 40%.
16
+
17
+### Troubleshooting section
18
+
19
+<details>
20
+
21
+ <summary>General approach</summary>
22
+
23
+You should try to use Wireshark to inspect the network packets in the remote
24
+
25
+Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting,
26
+analysis, software and communications protocol development.
27
+
28
+[See more about Wireshark here](https://www.wireshark.org/)
29
+
30
+Since you might won't be able to probe your traffic with wireshark in your host machine, You can
31
+export it in a dump file and analyze it in a second iteration.
32
+
33
+1. Try to export the traffic in your remote with `tcpdump`.
34
+
35
+ ```
36
+ root@netdata # tcpdump -i any 'port <PORT_YOU_MONITOR>' -s 65535 -w output.pcap
37
+ ```
38
+
39
+You must stop the capture after a certain observation period (60s up to 5 minutes). This command
40
+will create a dump file which can be interpreted by Wireshark that contains all the traffic from any
41
+interface for a specific port.
42
+
health/guides/processes/active_processes.md
new
+138
@@ -0,0 +1,138 @@
1
+# active_processes
2
+
3
+## OS: Any
4
+
5
+This alert presents the percentage of used PIDs. If this alert gets raised it means that your system is experiencing
6
+high system process IDs (PID) space utilization. \
7
+If this value is 100% then the system can not start new processes.
8
+
9
+Note also that **zombie** processes could be responsible for high used PID percentage.
10
+
11
+<details>
12
+<summary>What are "PIDs"</summary>
13
+
14
+> A "PID" (i.e., process identification number) is an identification number that is automatically assigned to
15
+> each `process` when it is created on a Unix-like operating system.
16
+> A `process` is an executing (i.e., running) instance of a program. Each process is guaranteed a unique **PID**,
17
+> which is always a non-negative integer.<sup> [1](http://www.linfo.org/pid.html) </sup>
18
+
19
+As you can understand, a system has a finite number of PIDs, as it can run a limited number of processes at any given
20
+time.
21
+</details>
22
+
23
+<details>
24
+<summary>Zombie Processes</summary>
25
+
26
+> A **zombie** process is one that never received a signal from the parent process that created it, a child process is
27
+> one that has its origin in a higher level process known as the parent process that is in charge of sending the
28
+> signals to the child processes generated by it to indicate that their life span has ended. So, if the parent
29
+> process isn't programmed properly it can spawn **zombie** processes.<sup> [2](
30
+> https://blog.desdelinux.net/en/procesos-zombies/) </sup>
31
+
32
+So if the system has **a lot of zombie** processes, it **can't repurpose the PIDs used** on those processes.
33
+</details>
34
+
35
+> On Linux, the **max PID** is runtime configurable and by using `sysctl /kernel/pid_max` you can see that max value.
36
+
37
+> Furthermore, large numbers of processes are not optimal for reasons other than PID exhaustion. Each process on the
38
+> system has some fixed overhead in the form of some memory usage (ir reaches up to a few hundred bytes on Linux) and
39
+> scheduling overhead. Even if you are not exhausting available PIDs, you generally do not want any more processes
40
+> running than you absolutely need, *though this only truly matters in an HPC environment. (HPC stands for
41
+> "high-performance computing")*
42
+
43
+<br>
44
+
45
+<details>
46
+<summary>References and Sources</summary>
47
+
48
+1. [PID Definition](http://www.linfo.org/pid.html)
49
+2. [Zombie Processes](https://blog.desdelinux.net/en/procesos-zombies/)
50
+</details>
51
+
52
+### Troubleshooting Section
53
+
54
+<details>
55
+<summary>Terminate processes you don't need</summary>
56
+
57
+<details>
58
+<summary>Linux</summary>
59
+
60
+Use the `ps` command;
61
+
62
+```
63
+root@netdata~ # ps
64
+```
65
+
66
+it will display all the active `processes` in your system alongside with their `PIDs`.
67
+
68
+From there you can determine which processes you don't need and use the `kill` command to terminate them;
69
+
70
+```
71
+root@netdata~ # kill <the pid number you want to terminate>
72
+```
73
+
74
+</details>
75
+
76
+> It would be helpful to close any unneeded processes, but Netdata strongly suggests knowing exactly what
77
+> processes you are closing and being certain that they are not necessary.
78
+</details>
79
+
80
+<details>
81
+<summary>Check for zombie processes</summary>
82
+
83
+As we have seen, a system can report high percentage of used PIDs due to zombie processes (*among other things*). \
84
+You can't simply kill zombie processes. So to clean your system up from these processes, you should locate the parent of
85
+that process. Killing the parent would result in the process getting inherited by `pid 1`
86
+which will wait on it and eventually clear it from the process table.
87
+
88
+- Use `ps`;
89
+
90
+<details><summary>Freebsd and MacOS</summary>
91
+
92
+```
93
+root@netdata~ # ps -al | grep Z
94
+```
95
+
96
+</details>
97
+
98
+<details>
99
+<summary>Linux</summary>
100
+
101
+```
102
+root@netdata~ # ps -elf | grep Z
103
+```
104
+
105
+</details>
106
+
107
+This command will show you the processes currently at a zombie state.
108
+
109
+```
110
+F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
111
+...
112
+```
113
+
114
+`S` stands for `state`, on this column the **zombie** processes have a `Z`. \
115
+`PID` is the PID of the process \
116
+`PPID` is the PID of the parent of that process \
117
+`CMD` Should give you more info about the process and help you determine if it is needed or not
118
+
119
+After you have decided that you want to clean up a zombie process, you can;
120
+
121
+- `kill -s SIGCHLD <pid>` where `<pid>` is the `PPID` of the zombie process.
122
+
123
+This will send a signal to the parent process telling it to clean up its zombie processes. If the parent isn't
124
+programmed properly, it might be ignoring the `SIGCHILD` signal we would be sending. In this case, we would need to
125
+`kill` the parent, so the zombie can get inherited by `pid 1` which will wait on it and eventually clear it from the
126
+process table. \
127
+To do so:
128
+
129
+```
130
+root@netdata~ # kill <PID>
131
+```
132
+
133
+Where `<PID>` is the `PPID` *(parent process PID)* of the zombie process.
134
+
135
+> Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
136
+
137
+
138
+</details>
health/guides/ram/oom_kill.md
new
+128
@@ -0,0 +1,128 @@
1
+# oom_kill
2
+
3
+## OS: Linux
4
+
5
+The OOM Killer (Out of Memory Killer) is a process that the Linux kernel uses when the system is
6
+critically low on memory or a process reached its memory limits. As the name suggests, it has the
7
+duty to review all running processes and kill one or more of them in order to free up memory and
8
+keep the system running.<sup>[1](https://neo4j.com/developer/kb/linux-out-of-memory-killer/) </sup>
9
+
10
+Linux Kernel 4.19 introduced cgroup awareness of OOM killer implementation which adds an ability to
11
+kill a cgroup as a single unit and to guarantee the integrity of the workload. In a nutshell,
12
+cgroups allow the limitation of memory, disk I/O, and network usage for a group of processes.
13
+Furthermore, cgroups may set usage quotas, and prioritize a process group to receive more CPU time
14
+or memory than other groups. You can see more about cgroups in
15
+the [cgroup man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html)
16
+
17
+The Netdata Agent monitors the number of Out Of Memory (OOM) kills in the last 30 minutes. Receiving
18
+this alert indicates that some processes got killed by OOM Killer.
19
+
20
+<details>
21
+<summary>References and Sources</summary>
22
+
23
+1. [Linux Out of Memory Killer](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)
24
+2. [Memory Resource Controller in linux kernel](https://docs.kernel.org/admin-guide/cgroup-v1/memory.html?highlight=oom)
25
+3. [OOM killer blogspot](https://www.psce.com/en/blog/2012/05/31/mysql-oom-killer-and-everything-related/)
26
+
27
+</details>
28
+
29
+### Troubleshooting Section
30
+
31
+<details>
32
+<summary>Troubleshoot issues in the OOM killer</summary>
33
+
34
+The OOM Killer uses a heuristic system to choose a processes for termination. It is based on a score
35
+associated with each running application, which is calculated by `oom_badness()` call inside Linux
36
+kernel <sup>[3](https://www.psce.com/en/blog/2012/05/31/mysql-oom-killer-and-everything-related/) </sup>
37
+
38
+1. To identify which process/apps was killed from the OOM killer, inspect the logs:
39
+
40
+```
41
+root@netdata~ # dmesg -T | egrep -i 'killed process'
42
+```
43
+The system response looks similar to this:
44
+```
45
+Jan 7 07:12:33 mysql-server-01 kernel: Out of Memory: Killed process 3154 (mysqld).
46
+```
47
+
48
+2. To see the current `oom_score` (the priority in which OOM killer will act upon your processes) run the following script.
49
+The script prints all running processes (by pid and name) with likelihood to be killed by the OOM killer (second column).
50
+The greater the `oom_score` (second column) the more propably to be killed by OOM killer.
51
+
52
+```
53
+root@netdata~ # while read -r pid comm; do
54
+ printf '%d\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$comm";
55
+done < <(ps -e -o pid= -o comm=) | sort -k 2n
56
+```
57
+
58
+3. Adjust the `oom_score` to protect processes using the `choom` util from
59
+the `util-linux` [package v2.33-rc1+](https://github.com/util-linux/util-linux/commit/8fa223daba1963c34cc828075ce6773ff01fafe3)
60
+
61
+```
62
+root@netdata~ # choom -p PID -n number
63
+```
64
+
65
+> Note: Setting an adjust score value of +500, for example, is roughly equivalent to allowing
66
+> the remainder of tasks sharing the same system, cpuset, mempolicy, or memory controller resources to
67
+> use at least 50% more memory. A value of -500, on the other hand, would be roughly equivalent to
68
+> discounting 50% of the task’s allowed memory from being considered as scoring against the task.
69
+
70
+4. Once the settings work to your case, make the change permanent. In the unit file of your service, under the [Service] section, add the following value: `OOMScoreAdjust=<PREFFERRED_VALUE>`
71
+
72
+</details>
73
+
74
+<details>
75
+<summary>Check the per-process RAM usage to find the top consumers</summary>
76
+
77
+1. To see which processes are the main RAM consumers, use `top utility`. The `%MEM` column displays RAM consumption in percent.
78
+
79
+```
80
+root@netdata~ # top -b -o +%MEM | head -n 22
81
+```
82
+
83
+2. Close any of the main consumer processes. Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
84
+</details>
85
+
86
+
87
+<details>
88
+<summary>Add a temporary swap file</summary>
89
+
90
+Keep in mind this requires creating a swap file in one of the disks. Performance of your system may
91
+be affected.
92
+
93
+1. Decide where your swapfile will live. It is strongly advised to allocate the swap file under in
94
+ the root directory. A swap file is like an extension of your RAM and it should be protected, far
95
+ from normal user accessible directories. Run the following command:
96
+
97
+ ```
98
+ root@netdata # dd if=/dev/zero of=<path_in_root> bs=1024 count=<size_in_bytes>
99
+ ```
100
+
101
+2. Grant root only access to the swap file:
102
+
103
+ ```
104
+ root@netdata # chmod 600 <path_to_the_swap_file_you_created>
105
+ ```
106
+
107
+3. Make it a Linux swap area:
108
+
109
+ ```
110
+ root@netdata # mkswap <path_to_the_swap_file_you_created>
111
+ ```
112
+
113
+4. Enable the swap with the following command:
114
+
115
+ ```
116
+ root@netdata # swapon <path_to_the_swap_file_you_created>
117
+ ```
118
+
119
+5. If you plan to use it a regular basis, you should update the `/etc/fstab` config. The entry you
120
+ will add would look like:
121
+
122
+ ```
123
+ /swap_file swap sw 0 0
124
+ ```
125
+
126
+ For more information see the fstab manpage: `man fstab`.
127
+
128
+</details>
health/guides/ram/ram_available.md
new
+60
@@ -0,0 +1,60 @@
1
+# ram_available
2
+
3
+## OS: Linux, FreeBSD
4
+
5
+This alarm shows the percentage of an estimated amount of RAM that is available for use in userspace processes without causing
6
+swapping. If this alarm gets raised it means that your system has low amount of available RAM memory, and it may affect the
7
+performance of running applications.
8
+
9
+- If there is no `swap` space available, the OOM Killer can start killing processes.
10
+
11
+- When a system runs out of RAM memory, it can store its inactive content in another storage's partition (e.g. your
12
+main drive). The borrowed space is called `swap` or "swap space".
13
+
14
+- The OOM Killer (Out of Memory Killer) is a process that the Linux Kernel uses when the system is critically low on
15
+RAM. As the name suggests, it has the duty to review all running processes and kill one or more of them in order
16
+to free up RAM memory and keep the system running.<sup>[1](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)</sup>
17
+
18
+<br>
19
+
20
+<details>
21
+<summary>References and Sources</summary>
22
+
23
+[[1] Linux Out of Memory Killer](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)
24
+</details>
25
+
26
+### Troubleshooting section:
27
+
28
+<details>
29
+<summary>Check per-process RAM usage to find the top consumers</summary>
30
+
31
+<details>
32
+<summary>Linux</summary>
33
+
34
+Use `top`:
35
+
36
+```
37
+root@netdata~ # top -b -o +%MEM | head -n 22
38
+```
39
+
40
+here, you can see which processes are the main RAM consumers on the `%MEM` column (it is calculated in percentage).
41
+
42
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
43
+processes you are closing and being certain that they are not necessary.
44
+</details>
45
+
46
+<details>
47
+<summary>FreeBSD</summary>
48
+
49
+Use `top`:
50
+
51
+```
52
+root@netdata~ # top -b -o res | head -n 22
53
+```
54
+
55
+Here, you can see which processes are the main RAM consumers on the `RES` column (calculated in percentage).
56
+
57
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
58
+processes you are closing and being certain that they are not necessary.
59
+</details>
60
+</details>
health/guides/ram/ram_in_use.md
new
+51
@@ -0,0 +1,51 @@
1
+# ram_in_use
2
+
3
+## OS: Linux, FreeBSD
4
+
5
+This alert shows the percentage of used RAM. If you receive this alert, there is high RAM utilization on the node. Running
6
+low on RAM memory, means that the performance of running applications might be affected.
7
+
8
+If there is no `swap` space available, the OOM Killer can start killing processes.
9
+
10
+> When a system runs out of RAM, it can store it's inactive content in persistent storage (e.g. your
11
+> main drive). The borrowed space is called `swap` or "swap space".
12
+
13
+> The OOM Killer (Out of Memory Killer) is a process that the Linux Kernel uses when the system is critically low on
14
+> RAM. As the name suggests, it has the duty to review all running processes and kill one or more of them in order
15
+> to free up RAM memory and keep the system running.
16
+
17
+### Troubleshooting section:
18
+
19
+<details>
20
+<summary>Check per-process RAM usage to find the top consumers</summary>
21
+
22
+<details>
23
+<summary>Linux</summary>
24
+
25
+Use `top`:
26
+
27
+```
28
+root@netdata~ # top -b -o +%MEM | head -n 22
29
+```
30
+
31
+Here, you can see which processes are the main RAM consumers on the `%MEM` column (it is calculated in percentage).
32
+
33
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
34
+processes your are closing and being certain that they are not necessary.
35
+</details>
36
+
37
+<details>
38
+<summary>FreeBSD</summary>
39
+
40
+Use `top`:
41
+
42
+```
43
+root@netdata~ # top -b -o res | head -n 22
44
+```
45
+
46
+Here, you can see which processes are the main RAM consumers on the `RES` column (it is calculated in percentage).
47
+
48
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
49
+processes your are closing and being certain that they are not necessary.
50
+</details>
51
+</details>
health/guides/softnet/10min_netisr_backlog_exceeded.md
new
+54
@@ -0,0 +1,54 @@
1
+# 10min_netisr_backlog_exceeded
2
+
3
+## OS: FreeBSD
4
+
5
+The netisr_maxqlen is a queue within the network kernel dispatch service of FreeBSD kernel which keeps packets
6
+received by interfaces and not yet processed by destined subsystems or userland applications. The system drops new packets
7
+when the queue is full. There may be several netisr packet queues in the system and raising netisr_maxqlen
8
+allows all of them to grow. The default netisr_maxqlen value should be 256 in most of the FreeBSD versions.
9
+ However this may not be enough in some cases, such as:
10
+
11
+- Multiple interfaces operating at 1Gbps, or even a single interface at 10Gbps.
12
+
13
+- Lower powered systems process very large amounts of network traffic.
14
+
15
+Netdata agent monitors the average number of dropped packets in the last minute due to exceeded netisr queue length.
16
+
17
+### Troubleshooting section:
18
+
19
+ <details>
20
+ <summary>Increase the netisr_maxqlen value.</summary>
21
+
22
+1. Check your current value.
23
+
24
+ ```
25
+ root@netdata~ # sysctl net.route.netisr_maxqlen
26
+ net.route.netisr_maxqlen: 256
27
+ ```
28
+
29
+2. Try to increase it by a factor of 4.
30
+
31
+ ```
32
+ root@netdata~ # sysctl -w net.route.netisr_maxqlen=1024
33
+ ```
34
+
35
+3. Verify the change and test with the same workload that triggered the alarm originally.
36
+
37
+ ```
38
+ root@netdata~ # sysctl net.route.netisr_maxqlen
39
+ net.route.netisr_maxqlen: 1024
40
+ ```
41
+
42
+4. If this change works for your system, you could make it permanently.
43
+
44
+ Bump this `net.route.netisr_maxqlen=1024` entry under `/etc/sysctl.conf`
45
+
46
+
47
+5. Reload the sysctl settings.
48
+
49
+ ```
50
+ root@netdata~ # /etc/rc.d/sysctl reload
51
+ ```
52
+</details>
53
+
54
+
health/guides/softnet/1min_netdev_backlog_exceeded.md
new
+53
@@ -0,0 +1,53 @@
1
+# 1min_netdev_backlog_exceeded
2
+
3
+## OS: Linux
4
+
5
+The linux kernel contains queues where packets are stored after reception from a network interface controller before
6
+being processed by the next protocol stack. There is one netdev backlog queue per CPU core. netdev_max_backlog defines
7
+the maximum number of packets that can enter the queue. Queues fill up when an interface receives packets faster than
8
+kernel can process them. The default netdev_max_backlog value should be 1000. However this may not be enough in cases
9
+such as:
10
+
11
+- Multiple interfaces operating at 1Gbps, or even a single interface at 10Gbps.
12
+
13
+- Lower powered systems process very large amounts of network traffic.
14
+
15
+Netdata agent monitors the average number of dropped packets in the last minute due to exceeding the netdev backlog
16
+queue.
17
+
18
+### Troubleshooting section:
19
+
20
+ <details>
21
+ <summary>Increase the netdev_max_backlog value</summary>
22
+
23
+1. Check your current value:
24
+
25
+ ```
26
+ root@netdata~ # sysctl net.core.netdev_max_backlog
27
+ net.core.netdev_max_backlog = 1000
28
+ ```
29
+
30
+2. Try to increase it by a factor of 2.
31
+
32
+ ```
33
+ root@netdata~ # sysctl -w net.core.netdev_max_backlog=2000
34
+ ```
35
+
36
+3. Verify the change and test with the same workload that triggered the alarm originally.
37
+
38
+ ```
39
+ root@netdata~ # sysctl net.core.netdev_max_backlog
40
+ net.core.netdev_max_backlog = 2000
41
+ ```
42
+
43
+4. If this change works for your system, you could make it permanently.
44
+
45
+ Bump this `net.core.netdev_max_backlog=2000` entry under `/etc/sysctl.conf`.
46
+
47
+5. Reload the sysctl settings.
48
+
49
+ ```
50
+ root@netdata~ # sysctl -p
51
+ ```
52
+
53
+</details>
health/guides/softnet/1min_netdev_budget_ran_outs.md
new
+70
@@ -0,0 +1,70 @@
1
+# 1min_netdev_budget_ran_outs
2
+
3
+## OS: Linux
4
+
5
+Your system communicates with the devices attached to it through interrupt requests. In a nutshell, when an interrupt
6
+occurs, the operating system stops what it was doing and starts addressing that interrupt.
7
+
8
+Network interfaces can receive thousands of packets per second. To avoid burying the system with thousands of interrupts,
9
+the Linux kernel uses the NAPI polling framework. In this way, we can replace hundreds of hardware interrupts with one poll
10
+by managing them with a few Soft Interrupt ReQuests (Soft IRQs). Ksoftirqd is a per-CPU kernel thread responsible for
11
+handling those unserved Soft Interrupt ReQuests (Soft IRQs). The Netdata agent inspects the average number of times
12
+Ksoftirqd ran out of netdev_budget or CPU time when there was still work to be done. This abnormality may cause packet
13
+overflow on the intermediate buffers and, as a result, drop packet on your network interfaces.
14
+
15
+The default value of the netdev_budget is 300. However, this may not be enough in some cases, such as:
16
+
17
+- Multiple interfaces operating at 1Gbps, or even a single interface at 10Gbps.
18
+
19
+- Lower powered systems processing very large amounts of network traffic.
20
+
21
+<details>
22
+<Summary>See more on the NAPI polling mechanism. </summary>
23
+
24
+The design of NAPI allows the network driver to go into a polling mode, buffering the packets it receives into a
25
+ring-buffer, and raises a soft interrupt to start a NAPI polling cycle instead of being hard-interrupted for
26
+every packet. Linux kernel through NAPI will poll data from the buffer until the netdev_budget_usecs times out or
27
+the number of packets reaches the netdev_budget limit.
28
+
29
+- netdev_budget_usecs variable defines the maximum number of microseconds in one NAPI polling cycle.
30
+- netdev_budget variable defines the maximum number of packets taken from all interfaces in one polling cycle.
31
+
32
+</details>
33
+
34
+### Troubleshooting section:
35
+
36
+ <details>
37
+ <summary>Increase the netdev_budget value.</summary>
38
+
39
+1. Check your current value.
40
+
41
+ ```
42
+ root@netdata~ $ sysctl net.core.netdev_budget
43
+ net.core.netdev_budget = 300
44
+ ```
45
+
46
+2. Try to increase it gradually with increments of 100.
47
+
48
+ ```
49
+ root@netdata~ $ sysctl -w net.core.netdev_budget=400
50
+ ```
51
+
52
+3. Verify the change and test it with the same workload that triggered the alarm originally. If the problem still exists, try to
53
+ increment it again.
54
+
55
+ ```
56
+ root@netdata~ $ sysctl net.core.netdev_budget
57
+ net.core.netdev_budget = 400
58
+ ```
59
+
60
+4. If this change works for your system, you could make it permanently.
61
+
62
+ Bump this `net.core.netdev_budget=<desired_value>` entry under `/etc/sysctl.conf`
63
+
64
+
65
+5. Reload the sysctl settings.
66
+
67
+ ```
68
+ root@netdata~ $ sysctl -p
69
+ ```
70
+</details>
health/guides/swap/30min_ram_swapped_out.md
new
+60
@@ -0,0 +1,60 @@
1
+# 30min_ram_swapped_out
2
+
3
+If the system needs more memory resources than your available RAM, inactive pages in memory can be
4
+moved into the swap space (or swap file). The swap space (or swap file) is located on hard drives,
5
+which have a slower access time than physical memory.
6
+
7
+The Netdata Agent calculates the percentage of the system RAM swapped in the last 30 minutes.
8
+
9
+This alert is triggered in warning state if the percentage of the system RAM swapped in is more than
10
+20%.
11
+
12
+
13
+## OS: Linux
14
+
15
+### Troubleshooting section:
16
+
17
+You can find the most resource greedy processes in your system, but if you receive this alert many
18
+times you must consider upgrade your system's RAM.
19
+
20
+<details>
21
+<summary>Find the processes that consume the most RAM </summary>
22
+
23
+1. Use `top` to see the top RAM consumers
24
+ ```
25
+ root@netdata~ # top -b -o +%MEM | head -n 22
26
+ ```
27
+
28
+Here, you can see which processes are the main RAM consumers on the `%MEM` column (it is calculated
29
+in percentage). It would be wise to close/kill any of the main consumer processes that you do not
30
+need to avoid thrashing.
31
+
32
+Netdata strongly suggests knowing exactly what processes you are closing and being certain that they
33
+are not necessary.
34
+</details>
35
+
36
+## OS: FreeBSD
37
+
38
+### Troubleshooting section:
39
+
40
+You can find the most resource greedy processes in your system, but if you receive this alert many
41
+times you must consider upgrade your system's RAM.
42
+
43
+<details>
44
+<summary>Find the processes that consume the most RAM </summary>
45
+
46
+1. Use `top` to see the top RAM consumers
47
+ ```
48
+ root@netdata~ # top -b -o res | head -n 22
49
+ ```
50
+
51
+Here, you can see which processes are the main RAM consumers on the `RES` column (calculated in
52
+percentage). It would be wise to close/kill any of the main consumer processes that you do not need
53
+to avoid thrashing, though Netdata strongly suggests knowing exactly what processes you are closing
54
+and being certain that they are not necessary.
55
+</details>
56
+
57
+
58
+
59
+
60
+
health/guides/swap/used_swap.md
new
+51
@@ -0,0 +1,51 @@
1
+# used_swap
2
+
3
+If the system needs more memory resources than your available RAM, inactive pages in memory can be
4
+moved into the swap space (or swap file). The Swap space (or swap file) is located on hard drives,
5
+which have a slower access time than physical memory.
6
+
7
+The Netdata Agent calculates the percentage of the used swap. This alert indicates high swap memory
8
+utilization. It may be a sign that the system has experienced memory pressure, which can affect the
9
+performance of your system. If there is no RAM and swap available, OOM Killer can start killing
10
+processes.
11
+
12
+This alert is triggered in warning state when the percentage of used swap is between 80-90% and in
13
+critical state when it is between 90-98%.
14
+
15
+## OS: Linux
16
+
17
+### Troubleshooting section:
18
+
19
+<details>
20
+<summary>Find the processes that consume the most RAM </summary>
21
+
22
+1. Use `top` to see the top RAM consumers
23
+ ```
24
+ root@netdata~ # top -b -o +%MEM | head -n 22
25
+ ```
26
+
27
+Here, you can see which processes are the main RAM consumers on the `%MEM` column (it is calculated
28
+in percentage). It would be wise to close/kill any of the main consumer processes that you do not
29
+need to avoid thrashing.
30
+
31
+Netdata strongly suggests knowing exactly what processes you are closing and being certain that they
32
+are not necessary.
33
+</details>
34
+
35
+## OS: FreeBSD
36
+
37
+### Troubleshooting section:
38
+
39
+<details>
40
+<summary>Find the processes that consume the most RAM </summary>
41
+
42
+1. Use `top` to see the top RAM consumers
43
+ ```
44
+ root@netdata~ # top -b -o res | head -n 22
45
+ ```
46
+
47
+Here, you can see which processes are the main RAM consumers on the `RES` column (calculated in
48
+percentage). It would be wise to close/kill any of the main consumer processes that you do not need
49
+to avoid thrashing, though Netdata strongly suggests knowing exactly what processes you are closing
50
+and being certain that they are not necessary.
51
+</details>
health/guides/synchronization/sync_freq.md
new
+48
@@ -0,0 +1,48 @@
1
+# sync_freq
2
+
3
+## OS: Any
4
+
5
+By default, the Linux kernel writes data to disk asynchronously. Writes are buffered (cached) in
6
+memory, and written to the storage device at the optimal time.
7
+
8
+Whenever you issue a write or send syscall or write to file-backed mappings or similar things, the
9
+kernel is not forced to flush that data straight to persistent storage, the underlying network stack,
10
+or any other subsystem. This buffering is implemented by the kernel for performance reasons.
11
+
12
+The `sync()` system call writes any data buffered in memory out to disk. This can include (but is not limited to)
13
+modified superblocks, modified inodes, and delayed reads and writes.
14
+
15
+The Netdata Agent monitors the number of sync() system calls. Receiving this alert indicates a high
16
+number of sync() system calls. Every call is very expensive because it causes all pending
17
+modifications to filesystem metadata and cached file data to be written to the underlying
18
+filesystems.
19
+
20
+This alert is triggered in warning state when the number of sync() system calls is greater than 6.
21
+
22
+<details>
23
+ <summary>References and source </summary>
24
+
25
+ 1. [sync man pages](https://man7.org/linux/man-pages/man2/sync.2.html)
26
+</details>
27
+
28
+### Troubleshooting section
29
+
30
+The `sync()` is expected to occur when the system is about to become unstable, or a storage device to become
31
+suddenly unavailable, and you want to ensure all data is written to disk. If you receive this alert often, you
32
+should gather more information on why this event is happening.
33
+
34
+<details>
35
+ <summary>Use bpftrace to identify which process is causing these sync events</summary>
36
+
37
+ `bpftrace` is a high-level tracing language for Linux enhanced Berkeley Packet Filter (eBPF) available in recent
38
+ Linux kernels (4.x). bpftrace uses LLVM as a backend to compile scripts to BPF-bytecode and makes use of BCC
39
+ for interacting with the Linux BPF system, as well as existing Linux tracing capabilities such as kernel dynamic
40
+ tracing (kprobes), user-level dynamic tracing (uprobes), and tracepoints.
41
+
42
+ One of the builtin tools in the `bpftrace` is the [syncsnoop](https://github.com/iovisor/bpftrace/blob/master/tools/syncsnoop_example.txt)
43
+ which tracing the `sync` events
44
+
45
+</details>
46
+
47
+
48
+</details>
health/guides/systemdunits/systemd_automount_units_state.md
new
+114
@@ -0,0 +1,114 @@
1
+# systemd_automount_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the `systemd.automount` units. The `systemd_automount_units_state` alert
9
+indicates that one or more of the `systemd.automount` units have failed.
10
+A systemd automount unit "failed" when the service process returned error code on exit, or crashed, an
11
+operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
+
13
+<details>
14
+<summary>Read More About systemd</summary>
15
+
16
+Here is some useful information about systemd from
17
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
+
19
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
+daemons
23
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
+Poettering described systemd not as one program, but rather a large software suite that includes 69
25
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
+its control. systemd also integrates many other services that are common on Linux systems by
28
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
+logging, hostnames and locales.
30
+
31
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
+parent of a process when the original parent terminates. Therefore, the first process is
36
+particularly well suited for the purpose of monitoring daemons.
37
+
38
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
+be preserved in a snapshot for future recall.
42
+
43
+Systemd's core components include the following:
44
+
45
+- `systemd` is a system and service manager for Linux operating systems.
46
+
47
+- `systemctl` is a command to introspect and control the state of the systemd system and service
48
+ manager. Not to be confused with sysctl.
49
+
50
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
+ other state and tracing information from the system and service manager.
52
+
53
+</details>
54
+
55
+
56
+<details>
57
+<summary>See More On systemd - `.automount` Units</summary>
58
+
59
+A unit configuration file whose name ends in `.automount` encodes information about a file system
60
+automount point controlled and supervised by `systemd`. Automount units must be named after the
61
+automount directories they control. For instance, the automount point `/home/lennart` must be
62
+configured in a unit file `home-lennart.automount`. For details about the escaping logic used to
63
+convert a file system path to a unit name see `systemd.unit(5)`. Note that automount units cannot be
64
+templated, nor is it possible to add multiple names to an automount unit by creating additional
65
+symlinks to its unit file.
66
+
67
+For each automount unit file a matching mount unit file (see systemd.mount(5) for details) must
68
+exist which is activated when the automount path is accessed. For instance, if an automount unit
69
+`home-lennart.automount` is active and the user accesses `/home/lennart` the mount unit
70
+`home-lennart.mount` will be
71
+activated. <sup> [2](https://www.freedesktop.org/software/systemd/man/systemd.automount.html) </sup>
72
+
73
+</details>
74
+
75
+<details>
76
+<summary>References and Source</summary>
77
+
78
+1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
79
+2. [man page for systemd.automount](https://www.freedesktop.org/software/systemd/man/systemd.automount.html)
80
+
81
+</details>
82
+
83
+### Troubleshooting Section:
84
+
85
+<details>
86
+<summary>General Approach</summary>
87
+
88
+If an automount has failed, then you should always try to collect more information to diagnose the cause of
89
+the failure.
90
+
91
+1. Identify which automount fails. Open the Netdata dashboard, find the current active alarms under
92
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
93
+ into its chart.
94
+ (`systemdunits_automount_units.automount_unit_state`). In this chart, identify which automount
95
+ units are in state with value 5
96
+
97
+2. Gather more information about the failing automount. We advise you to run the following commands
98
+ in two different terminals.
99
+
100
+ ```
101
+ root@netdata~ # journalctl -u <automount_name>.automount -f
102
+ root@netdata~ # journalctl -u <automount_name>.mount -f
103
+ ```
104
+
105
+3. In your main terminal, try mount the automount manually.
106
+
107
+ ```
108
+ root@netdata~ # mount -v <automount_name>
109
+ ```
110
+
111
+ This command will try to mount your automount unit in verbose mode.
112
+4. Check the output messages from both terminals for abnormalities.
113
+
114
+</details>
health/guides/systemdunits/systemd_device_units_state.md
new
+94
@@ -0,0 +1,94 @@
1
+# systemd_device_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the `systemd.device` units. The `systemd_device_units_state` alert
9
+indicates that one or more of the `systemd.device` units have failed.
10
+A systemd device unit "failed" when the service process returned error code on exit, or crashed, an
11
+operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
+
13
+<details>
14
+<summary>Read More About systemd</summary>
15
+
16
+Here is some useful information about systemd from
17
+Wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
+
19
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
+daemons
23
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
+Poettering described systemd not as one program, but rather a large software suite that includes 69
25
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
+its control. systemd also integrates many other services that are common on Linux systems by
28
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
+logging, hostnames and locales.
30
+
31
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
+parent of a process when the original parent terminates. Therefore, the first process is
36
+particularly well suited for the purpose of monitoring daemons.
37
+
38
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
+be preserved in a snapshot for future recall.
42
+
43
+Systemd's core components include the following:
44
+
45
+- `systemd` is a system and service manager for Linux operating systems.
46
+
47
+- `systemctl` is a command to introspect and control the state of the systemd system and service
48
+ manager. Not to be confused with sysctl.
49
+
50
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
+ other state and tracing information from the system and service manager.
52
+
53
+</details>
54
+
55
+<details>
56
+<summary>More On systemd-`.device` Units</summary>
57
+
58
+A unit configuration file whose name ends in `.device` encodes information about a device unit as
59
+exposed in the `sysfs/udev(7)` device tree. `udev` provides a dynamic device directory containing
60
+only the files for actually present devices. It creates or removes device node files in the `/dev`
61
+directory, or it renames network interfaces.
62
+
63
+This unit type has no specific options. Device units are named after the `/sys` and `/dev` paths
64
+they control. For example, the device `/dev/sda5` is exposed in systemd as `dev-sda5.device`.
65
+</details>
66
+
67
+<details>
68
+<summary>References and Source</summary>
69
+
70
+1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
71
+2. [man page for systemd.device](https://www.freedesktop.org/software/systemd/man/systemd.device.html)
72
+
73
+</details>
74
+
75
+### Troubleshooting Section:
76
+
77
+<details>
78
+<summary>General Approach</summary>
79
+
80
+If a device has failed, then you should always try to gather more information about the cause of the failure.
81
+
82
+1. Identify which device units fail. Open the Netdata dashboard, find the current active alarms under
83
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
84
+ into its chart
85
+ (`systemdunits_device_units.device_unit_state`). In this chart, identify which device
86
+ units are in state with value 5.
87
+
88
+2. Check the status of the device
89
+
90
+ ```
91
+ root@netdata~ # systemctl status <device_name>.device
92
+ ```
93
+
94
+</details>
health/guides/systemdunits/systemd_mount_units_state.md
new
+137
@@ -0,0 +1,137 @@
1
+# systemd_mount_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the `systemd.mount` units state. The `systemd_mount_units_state` alert
9
+indicates that one or more of the `systemd.mount` units are in the failed state.
10
+A systemd mount unit "failed" when the service process returned error code on exit, or crashed, an
11
+operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
+
13
+<details>
14
+<summary>Read more about systemd</summary>
15
+
16
+Here is some useful information about systemd from
17
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
+
19
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
+daemons
23
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
+Poettering described systemd not as one program, but rather a large software suite that includes 69
25
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
+its control. systemd also integrates many other services that are common on Linux systems by
28
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
+logging, hostnames and locales.
30
+
31
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
+parent of a process when the original parent terminates. Therefore, the first process is
36
+particularly well suited for the purpose of monitoring daemons.
37
+
38
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
+be preserved in a snapshot for future recall.
42
+
43
+Systemd's core components include the following:
44
+
45
+- `systemd` is a system and service manager for Linux operating systems.
46
+
47
+- `systemctl` is a command to introspect and control the state of the systemd system and service
48
+ manager. Not to be confused with sysctl.
49
+
50
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
+ other state and tracing information from the system and service manager.
52
+
53
+</details>
54
+
55
+
56
+
57
+<details>
58
+<summary>See more on systemd-mount units</summary>
59
+
60
+A unit configuration file whose name ends in `.mount` encodes information about a file system mount
61
+point controlled and supervised by `systemd`. Additional options are listed in _systemd.exec(5)_,
62
+which define the execution environment the _mount(8)_ program is executed in, and in _systemd.kill(
63
+5)_, which define the way the processes are terminated, and in
64
+_systemd.resource-control(5)_, which configure resource control settings for the processes of the
65
+service.
66
+
67
+The options `User=` and `Group=` are not useful for mount units. systemd passes two parameters to
68
+mount(8) the values of `What=` and `Where=`. When invoked in this way, _mount(8)_ does not read any
69
+options from `/etc/fstab`, and must be run as UID 0.
70
+
71
+Mount units must be named after the mount point directories they control. For instance, the mount
72
+point `/home/lennart` must be configured in a unit file `home-lennart.mount`. For details about the
73
+escaping logic used to convert a file system path to a unit name, see _systemd.unit(5)_. Note that
74
+mount units cannot be templated, nor is possible to add multiple names to a mount unit by creating
75
+additional symlinks to it.
76
+
77
+Mount units may either be configured via unit files, or via `/etc/fstab` (see `man fstab` for
78
+details). Mounts listed in /etc/fstab will be converted into native units dynamically at boot and
79
+when the configuration of the system manager is reloaded. In general, configuring mount points
80
+through `/etc/fstab` is the preferred approach. See _systemd-fstab-generator(8)_ for details about
81
+the
82
+conversion. <sup> [2](https://www.freedesktop.org/software/systemd/man/systemd.mount.html) </sup>
83
+
84
+</details>
85
+
86
+
87
+<details>
88
+<summary>References and source</summary>
89
+
90
+1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
91
+2. [man page for systemd.mount](https://www.freedesktop.org/software/systemd/man/systemd.mount.html)
92
+
93
+</details>
94
+
95
+### Troubleshooting section:
96
+
97
+<details>
98
+<summary>General approach</summary>
99
+
100
+When a mount is in failed state, you should always try to gather more information about it.
101
+
102
+1. Identify which mount units fail. Open the Netdata dashboard, find the current active alarms under
103
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
104
+ into its chart.
105
+ (`systemdunits_mount_units.mount_unit_state`). In this chart, identify which mount
106
+ units are in state with value 5
107
+
108
+2. Gather more information about the failing mount. We advise you to run the following command
109
+ in a second terminal.
110
+
111
+ ```
112
+ root@netdata~ # journalctl -u <mount_name>.mount -f
113
+ ```
114
+ This command will monitor the journalctl log messages for your mount unit.
115
+3. In your main terminal, try mount the mount manually.
116
+
117
+ ```
118
+ root@netdata~ # mount -v <mount_name>
119
+ ```
120
+
121
+ This command will try to mount your mount unit in verbose mode.
122
+4. Check the output messages from both terminals for abnormalities.
123
+
124
+</details>
125
+
126
+
127
+<details>
128
+<summary>Verify the fstab configuration</summary>
129
+
130
+1. Open a terminal and run the following command
131
+
132
+ ```
133
+ root@netdata~ # sudo findmnt --verify --verbose
134
+ ```
135
+
136
+This command will check mount table content (default: `/etc/fstab`) in verbose mode
137
+</details>
health/guides/systemdunits/systemd_path_units_state.md
new
+134
@@ -0,0 +1,134 @@
1
+# systemd_path_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the systemd path units state. This alert indicates that one or more of
9
+the systemd path units are in the failed state.
10
+A systemd path unit "failed" when the service process returned error code on exit, or crashed, an
11
+operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
+
13
+<details>
14
+<summary>Read more about systemd</summary>
15
+
16
+Here is some useful information about systemd from
17
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
+
19
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
+daemons
23
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
+Poettering described systemd not as one program, but rather a large software suite that includes 69
25
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
+its control. systemd also integrates many other services that are common on Linux systems by
28
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
+logging, hostnames and locales.
30
+
31
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
+parent of a process when the original parent terminates. Therefore, the first process is
36
+particularly well suited for the purpose of monitoring daemons.
37
+
38
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
+be preserved in a snapshot for future recall.
42
+
43
+Systemd's core components include the following:
44
+
45
+- `systemd` is a system and service manager for Linux operating systems.
46
+
47
+- `systemctl` is a command to introspect and control the state of the systemd system and service
48
+ manager. Not to be confused with sysctl.
49
+
50
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
+ other state and tracing information from the system and service manager.
52
+
53
+</details>
54
+
55
+
56
+<details>
57
+<summary>See more on systemd.path units</summary>
58
+
59
+> A unit configuration file whose name ends in ".path" encodes information about a path monitored by system. With path units, you can monitor files and directories for certain events. If a specified event occurs, a service unit is executed, and it usually carries the same name as the path unit.
60
+>
61
+> In the [Path] section, `PathChanged=` specifies the absolute path to the file to be monitored, while
62
+`Unit=` indicates which service unit to execute if the file changes. <sup>[2](https://www.redhat.com/sysadmin/introduction-path-units) </sup>
63
+
64
+Path units are very useful to monitor files for changes with systemd.
65
+</details>
66
+
67
+
68
+<details>
69
+<summary>References and source</summary>
70
+
71
+1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
72
+2. [A Brief introduction to path units by Jörg Kastning](https://www.redhat.com/sysadmin/introduction-path-units)
73
+
74
+</details>
75
+
76
+### Troubleshooting section:
77
+
78
+<details>
79
+<summary>General approach</summary>
80
+
81
+When a path is in failed state, you should always try to gather more information about it.
82
+
83
+1. Identify which path units fail. Open the Netdata dashboard, find the current active alarms under
84
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
85
+ into its chart.
86
+ (`systemdunits_path_units.path_unit_state`). In this chart, identify which path units are in
87
+ state with value 5
88
+
89
+2. Gather more information about the failing path unit and the service that manages it (in most of
90
+ the cases they will have the same name). We advise you to run the following commands in two
91
+ different terminals.
92
+
93
+ ```
94
+ root@netdata~ # journalctl -u <service_name>.service -f
95
+ root@netdata~ # journalctl -u <path_name>.socket -f
96
+ ```
97
+ These commands will monitor the journalctl log messages for your path/service unit.
98
+3. In a new terminal, try to restart the service.
99
+
100
+ ```
101
+ root@netdata~ # systemctl restart <service_name>.service
102
+ ```
103
+
104
+4. Check the log messages from the commands of step 2.
105
+
106
+</details>
107
+
108
+<details>
109
+<summary>Run the service of the path unit in debug mode</summary>
110
+
111
+1. Identify which path units fail. Open the Netdata dashboard, find the current active alarms under
112
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
113
+ into its chart.
114
+ (`systemdunits_path_units.path_unit_state`). In this chart, identify which path units are in
115
+ state with value 5
116
+
117
+2. Stop the service that manages this path (in most of the cases the service will have the same name
118
+ with the path)
119
+
120
+ ```
121
+ root@netdata~ # systemctl stop <service_name>.service
122
+ ```
123
+
124
+3. Try to start it with the `SYSTEMD_LOG_LEVEL=debug` env variable. Let's assume in our case we want
125
+ to debug the `systemd-networkd` service
126
+
127
+ ```
128
+ root@netdata~ # SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
129
+
130
+ ```
131
+
132
+4. Check the log messages.
133
+
134
+</details>
health/guides/systemdunits/systemd_scope_units_state.md
new
+109
@@ -0,0 +1,109 @@
1
+# systemd_scope_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the systemd scope units state. The `systemd_scope_units_state` alert
9
+indicates that one or more of the systemd-scope units are in the failed state.
10
+A systemd scope unit "failed" when the service process returned error code on exit, or crashed, an
11
+operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
+
13
+<details>
14
+<summary>Read more about systemd</summary>
15
+
16
+Here is some useful information about systemd from
17
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
+
19
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
+entire software bundle around it, which, in addition to the systemd `init` daemon, includes the
22
+daemons
23
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
+Poettering described systemd not as one program, but rather a large software suite that includes 69
25
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
+its control. Systemd also integrates many other services that are common on Linux systems by
28
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
+logging, hostnames and locales.
30
+
31
+Like the `init daemon`, Systemd is a daemon that manages other daemons, which, including `systemd`
32
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
+process tree. The first process (`PID 1`) has a special role on Unix systems, as it replaces the
35
+parent of a process when the original parent terminates. Therefore, the first process is
36
+particularly well suited for the purpose of monitoring daemons.
37
+
38
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
+the traditional startup sequence approach. For inter-process communication (IPC), systemd makes Unix
40
+domain sockets and D-Bus available to the running daemons. The state of `systemd` itself can also be
41
+preserved in a snapshot for future recall.
42
+
43
+systemd's core components include the following:
44
+
45
+- `systemd` is a system and service manager for Linux operating systems.
46
+
47
+- `systemctl` is a command to introspect and control the state of the systemd system and service
48
+ manager. Not to be confused with sysctl.
49
+
50
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
+ other state and tracing information from the system and service manager.
52
+
53
+</details>
54
+
55
+
56
+<details>
57
+<summary>See more on systemd-scope</summary>
58
+The following text originates from the systemd.scope man page.<sup>[2](https://www.freedesktop.org/software/systemd/man/systemd.scope.html) </sup>
59
+
60
+Scope units are not configured via unit configuration files, but are only created programmatically
61
+using the bus interfaces of systemd. They are named similar to filenames. A unit whose name ends
62
+in `.scope` refers to a scope unit. Scopes units manage a set of system processes. Unlike service
63
+units, scope units manage externally created processes, and do not fork off processes on its own.
64
+
65
+The main purpose of scope units is grouping worker processes of a system service for organization
66
+and for managing resources.
67
+
68
+Unlike service units, scope units have no "main" process: all processes in the scope are equivalent.
69
+The lifecycle of the scope unit is thus not bound to the lifetime of one specific process, but to
70
+the existence of at least one process in the scope. This also means that the exit statuses of these
71
+processes are not relevant for the scope unit failure state. Scope units may still enter a failure
72
+state, for example due to resource exhaustion or stop timeouts being reached, but not due to
73
+programs inside of them terminating uncleanly. Since processes managed as scope units generally
74
+remain children of the original process that forked them off, it is also the job of that process to
75
+collect their exit statuses and act on them as
76
+needed.
77
+
78
+</details>
79
+
80
+<details>
81
+<summary>References and source</summary>
82
+
83
+1. [systemd on Wikipedia](https://en.wikipedia.org/wiki/Systemd)
84
+2. [Man page for systemd.scope](https://www.freedesktop.org/software/systemd/man/systemd.scope.html)
85
+
86
+</details>
87
+
88
+### Troubleshooting section:
89
+
90
+<details>
91
+<summary>General approach</summary>
92
+
93
+When a scope is in a failed state, you should always try to gather more information about it.
94
+
95
+1. Identify which scope units fail. Open the Netdata dashboard, find the current active alarms under
96
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
97
+ into its chart.
98
+ (`systemdunits_scope_units.scope_unit_state`). In this chart, identify which slice units are in
99
+ state with value 5
100
+
101
+2. Gather more information about the failing scope unit
102
+
103
+ ```
104
+ root@netdata~ # systemctl status <scope_name>.scope
105
+ ```
106
+
107
+3. Check the log messages from the command of step 2.
108
+
109
+</details>
health/guides/systemdunits/systemd_service_units_state.md
new
+138
@@ -0,0 +1,138 @@
1
+# systemd_service_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the systemd service units. The `systemd_service_units_state` alert
9
+indicates that one or more of the systemd service units are in the `failed` state. One of the
10
+following reasons can cause this alert:
11
+
12
+- The process of the service returns an error code on exit.
13
+- The process of the service crashed.
14
+- An operation timed out occurred.
15
+- The service failed after too many restarts.
16
+
17
+<details>
18
+<summary>Read more about systemd</summary>
19
+
20
+Here is some useful information about systemd from
21
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
22
+
23
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
24
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
25
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
26
+daemons
27
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
28
+Poettering described systemd not as one program, but rather a large software suite that includes 69
29
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
30
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
31
+its control. systemd also integrates many other services that are common on Linux systems by
32
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
33
+logging, hostnames and locales.
34
+
35
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
36
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
37
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
38
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
39
+parent of a process when the original parent terminates. Therefore, the first process is
40
+particularly well suited for the purpose of monitoring daemons.
41
+
42
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
43
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
44
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
45
+be preserved in a snapshot for future recall.
46
+
47
+Systemd's core components include the following:
48
+
49
+- `systemd` is a system and service manager for Linux operating systems.
50
+
51
+- `systemctl` is a command to introspect and control the state of the systemd system and service
52
+ manager. Not to be confused with sysctl.
53
+
54
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
55
+ other state and tracing information from the system and service manager.
56
+
57
+</details>
58
+
59
+
60
+<details>
61
+<summary>See more on systemd-services</summary>
62
+
63
+A unit configuration file whose name ends in `.service` encodes information about a process
64
+controlled and supervised by systemd. To `view`, `start`, `stop`, `restart`, `enable`, or `disable`
65
+system services, use the `systemctl` command line interface. It is common that services are ordered
66
+to start after some specified service that depends on (try the
67
+command `systemctl list-dependencies --before|after <service_name>.service`)
68
+
69
+See more in the man pages, `man systemd.service`
70
+
71
+</details>
72
+
73
+<details>
74
+<summary>References and source</summary>
75
+
76
+1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
77
+
78
+</details>
79
+
80
+### Troubleshooting section:
81
+
82
+<details>
83
+<summary>General approach</summary>
84
+
85
+When a service is in failed state, you should always try to gather more information about it.
86
+
87
+1. Identify which service units fail. Open the Netdata dashboard, find the current active alarms under
88
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
89
+ into its chart.
90
+ (`systemdunits_service_units.service_unit_state`). In this chart, identify which service
91
+ units are in state with value 5
92
+
93
+2. Gather more information about the failing service. We advise you to run the following command in
94
+ a second terminal.
95
+
96
+ ```
97
+ root@netdata~ # journalctl -u <service_name>.service -f
98
+ ```
99
+ This command will monitor the journalctl log messages for your service.
100
+
101
+3. In a new terminal, try to restart the service.
102
+
103
+ ```
104
+ root@netdata~ # systemctl restart <service_name>.service
105
+ ```
106
+ This command will restart your service.
107
+
108
+4. Check the log messages from the command of step 2.
109
+
110
+</details>
111
+
112
+<details>
113
+<summary>Run the service in debug mode</summary>
114
+
115
+1. Identify which service units fail. Open the Netdata dashboard, find the current active alarms under
116
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
117
+ into its chart.
118
+ (`systemdunits_service_units.service_unit_state`). In this chart, identify which service
119
+ units are in state with value 5
120
+
121
+
122
+2. Stop the service
123
+
124
+ ```
125
+ root@netdata~ # systemctl stop <service_name>.service
126
+ ```
127
+
128
+3. Try to start it with the `SYSTEMD_LOG_LEVEL=debug` env variable. Let's assume in our case we want
129
+ to debug the `systemd-networkd` service.
130
+
131
+ ```
132
+ root@netdata~ # SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
133
+
134
+ ```
135
+ This command will start your service in debug mode.
136
+4. Check the log messages.
137
+
138
+</details>
health/guides/systemdunits/systemd_slice_units_state.md
new
+114
@@ -0,0 +1,114 @@
1
+# systemd_slice_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the systemd slice units state. The `systemd_slice_units_state` alert
9
+indicates that one or more of the systemd slice units are in the failed state.
10
+A systemd slice unit "failed" when the service process returned error code on exit, or crashed, an
11
+operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
+
13
+<details>
14
+<summary>Read more about systemd</summary>
15
+
16
+Here is some useful information about systemd from
17
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
+
19
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
+daemons
23
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
+Poettering described systemd not as one program, but rather a large software suite that includes 69
25
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
+its control. systemd also integrates many other services that are common on Linux systems by
28
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
+logging, hostnames and locales.
30
+
31
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
+parent of a process when the original parent terminates. Therefore, the first process is
36
+particularly well suited for the purpose of monitoring daemons.
37
+
38
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
+be preserved in a snapshot for future recall.
42
+
43
+Systemd's core components include the following:
44
+
45
+- `systemd` is a system and service manager for Linux operating systems.
46
+
47
+- `systemctl` is a command to introspect and control the state of the systemd system and service
48
+ manager. Not to be confused with sysctl.
49
+
50
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
+ other state and tracing information from the system and service manager.
52
+
53
+</details>
54
+
55
+
56
+<details>
57
+<summary>See more on systemd-slice</summary>
58
+
59
+The following text originates from the systemd.slice man page.<sup>[2](https://www.freedesktop.org/software/systemd/man/systemd.slice.html) </sup>
60
+
61
+A unit configuration file whose name ends in ".slice" encodes information about a slice unit. A
62
+slice unit is a concept for hierarchically managing resources of a group of processes. This
63
+management is performed by creating a node in the Linux Control Group (cgroup) tree. Units that
64
+manage processes (primarily scope and service units) may be assigned to a specific slice. For each
65
+slice, certain resource limits may be set that apply to all processes of all units contained in that
66
+slice. Slices are organized hierarchically in a tree. The name of the slice encodes the location in
67
+the tree. The name consists of a dash-separated series of names, which describes the path to the
68
+slice from the root slice. The root slice is named -.slice. For example, foo-bar.slice is a slice
69
+that is located within foo.slice, which in turn is located in the root slice -.slice.
70
+
71
+Note that slice units cannot be templated, nor is possible to add multiple names to a slice unit by
72
+creating additional symlinks to its unit file.
73
+
74
+By default, service and scope units are placed in `system.slice`, virtual machines and containers
75
+registered with `systemd-machined` are found in `machine.slice`, and user sessions handled by
76
+`systemd-logind` in `user.slice`.
77
+
78
+The slice specific configuration options are configured in the `[Slice]` section. Currently, only
79
+generic resource control settings as described in systemd.resource-control(5) are allowed.
80
+
81
+</details>
82
+
83
+
84
+<details>
85
+
86
+<summary>References and source</summary>
87
+
88
+1. [systemd on Wikipedia](https://en.wikipedia.org/wiki/Systemd)
89
+2. [Man page for systemd.slice](https://www.freedesktop.org/software/systemd/man/systemd.slice.html)
90
+
91
+</details>
92
+
93
+### Troubleshooting section:
94
+
95
+<details>
96
+<summary>General approach</summary>
97
+
98
+When a slice is in a failed state, you should always try to gather more information about it.
99
+
100
+1. Identify which slice units fail. Open the Netdata dashboard, find the current active alarms under
101
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
102
+ into its chart.
103
+ (`systemdunits_slice_units.slice_unit_state`). In this chart, identify which slice
104
+ units are in state with value 5.
105
+
106
+2. Gather more information about the failing slice unit
107
+
108
+ ```
109
+ root@netdata~ # systemctl status <slice_name>.slice
110
+ ```
111
+
112
+3. Check the log messages from the command of step 2.
113
+
114
+</details>
health/guides/systemdunits/systemd_socket_units_state.md
new
+137
@@ -0,0 +1,137 @@
1
+# systemd_socket_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the systemd socket units state. Receiving this alerts indicates that one
9
+or more of the systemd socket units are in the failed state. In most of the cases this is correlated
10
+with the service, which manages the socket.
11
+A systemd socket unit "failed" when the service process returned error code on exit, or crashed, an
12
+operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
13
+
14
+<details>
15
+<summary>Read more about systemd</summary>
16
+
17
+Here is some useful information about systemd from
18
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
19
+
20
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
21
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
22
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
23
+daemons
24
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
25
+Poettering described systemd not as one program, but rather a large software suite that includes 69
26
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
27
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
28
+its control. systemd also integrates many other services that are common on Linux systems by
29
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
30
+logging, hostnames and locales.
31
+
32
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
33
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
34
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
35
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
36
+parent of a process when the original parent terminates. Therefore, the first process is
37
+particularly well suited for the purpose of monitoring daemons.
38
+
39
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
40
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
41
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
42
+be preserved in a snapshot for future recall.
43
+
44
+Systemd's core components include the following:
45
+
46
+- `systemd` is a system and service manager for Linux operating systems.
47
+
48
+- `systemctl` is a command to introspect and control the state of the systemd system and service
49
+ manager. Not to be confused with sysctl.
50
+
51
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
52
+ other state and tracing information from the system and service manager.
53
+
54
+</details>
55
+
56
+
57
+<details>
58
+<summary>See more on systemd.socket units</summary>
59
+
60
+> A unit configuration file whose name ends in `.socket` encodes information about an IPC or network socket or a file system FIFO controlled and supervised by systemd, for socket-based activation. For each socket unit, a matching service unit must exist, describing the service to start on incoming traffic on the socket. The name of the .service unit is by default the same as the name of the .socket unit.
61
+>
62
+> Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd's native socket passing interface (see sd_listen_fds(3) for details about the precise protocol used and the order in which the file descriptors are passed) or via traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).
63
+>
64
+> All network sockets allocated through .socket units are allocated in the host's network namespace
65
+(see network_namespaces(7)). This does not mean however that the service activated by a configured socket unit has to be part of the host's network namespace as well. It is supported and even good practice to run services in their own network namespace (for example through PrivateNetwork=, see systemd.exec(5)), receiving only the sockets configured through socket-activation from the host's namespace. In such a set-up communication within the host's network namespace is only permitted through the activation sockets passed in while all sockets allocated from the service code itself will be associated with the service's own namespace, and thus possibly subject to a a much more restrictive configuration. <sup>[2](https://manpages.debian.org/testing/systemd/systemd.socket.5.en.html) </sup>
66
+
67
+</details>
68
+
69
+
70
+<details>
71
+<summary>References and source</summary>
72
+
73
+1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
74
+2. [systemd.socket on debian.org](https://manpages.debian.org/testing/systemd/systemd.socket.5.en.html)
75
+
76
+</details>
77
+
78
+### Troubleshooting section:
79
+
80
+<details>
81
+<summary>General approach</summary>
82
+
83
+When a socket is in failed state, you should always try to gather more information about it.
84
+
85
+1. Identify which socket units fail. Open the Netdata dashboard, find the current active alarms under
86
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
87
+ into its chart.
88
+ `systemdunits_service_units.socket_unit_state` chart. Check which sockets are in state with value 5.
89
+
90
+
91
+2. Gather more information about the failing socket and the service that manages it (in most of the
92
+ cases they will have the same name). We advise you to run the following commands in two different
93
+ terminals.
94
+
95
+ ```
96
+ root@netdata~ # journalctl -u <service_name>.service -f
97
+ root@netdata~ # journalctl -u <socket_name>.socket -f
98
+ ```
99
+
100
+ These commands will monitor the journalctl log messages for your socket/service unit.
101
+3. In a new terminal, try to restart the service.
102
+
103
+ ```
104
+ root@netdata~ # systemctl restart <service_name>.service
105
+ ```
106
+
107
+4. Check the log messages from the command of step 2.
108
+
109
+</details>
110
+
111
+<details>
112
+<summary>Run the service of the socket in debug mode</summary>
113
+
114
+1. Identify which socket units fail. Open the Netdata dashboard, find the current active alarms under
115
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
116
+ into its chart.
117
+ `systemdunits_service_units.socket_unit_state` chart. Check which sockets are in state with value 5.
118
+
119
+2. Stop the service that manages this socket (in most of the cases the service will have the same
120
+ name with the socket)
121
+
122
+ ```
123
+ root@netdata~ # systemctl stop <service_name>.service
124
+ ```
125
+
126
+3. Try to start it with the `SYSTEMD_LOG_LEVEL=debug` env variable. Let's assume in our case we want
127
+ to debug the `systemd-networkd` service
128
+
129
+ ```
130
+ root@netdata~ # SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
131
+
132
+ ```
133
+
134
+ This command will start your service in debug mode.
135
+4. Check the log messages.
136
+
137
+</details>
health/guides/systemdunits/systemd_swap_units_state.md
new
+116
@@ -0,0 +1,116 @@
1
+# systemd_swap_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the systemd swap units state. The `systemd_swap_units_state` alert
9
+indicates that one or more of the systemd swap units are in the failed state.
10
+A systemd swap unit "failed" when the service process returned error code on exit, or crashed, an
11
+operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
+
13
+<details>
14
+<summary>Read more about systemd</summary>
15
+
16
+Here is some useful information about systemd from
17
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
+
19
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
+daemons
23
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
+Poettering described systemd not as one program, but rather a large software suite that includes 69
25
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
+its control. systemd also integrates many other services that are common on Linux systems by
28
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
+logging, hostnames and locales.
30
+
31
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
+parent of a process when the original parent terminates. Therefore, the first process is
36
+particularly well suited for the purpose of monitoring daemons.
37
+
38
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
+be preserved in a snapshot for future recall.
42
+
43
+Systemd's core components include the following:
44
+
45
+- `systemd` is a system and service manager for Linux operating systems.
46
+
47
+- `systemctl` is a command to introspect and control the state of the systemd system and service
48
+ manager. Not to be confused with sysctl.
49
+
50
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
+ other state and tracing information from the system and service manager.
52
+
53
+</details>
54
+
55
+<details>
56
+<summary>See more on systemd-swap</summary>
57
+The following text originates from the systemd.swap man page.<sup>[2](https://www.freedesktop.org/software/systemd/man/systemd.swap.html) </sup>
58
+
59
+A unit configuration file whose name ends in `.swap` encodes information about a swap device or
60
+file for memory paging controlled and supervised by systemd. Swap units must be named after the
61
+devices or files they control. For instance, the swap device `/dev/sda5` must be configured in a
62
+unit file`dev-sda5.swap`. Note that swap units cannot be templated, nor is possible to add multiple
63
+names to a swap unit by creating additional symlinks to it.
64
+Swap units may either be configured via unit files, or via `/etc/fstab` (see `man fstab(5)` for
65
+details). Swaps listed in `/etc/fstab` will be converted into native units dynamically at boot and
66
+when the configuration of the system manager is reloaded. See `man systemd-fstab-generator` for
67
+details about the conversion.
68
+If a swap device or file is configured in both `/etc/fstab` and a unit file, the configuration in
69
+the latter takes precedence.
70
+When reading `/etc/fstab`, a few special options are understood by systemd which influence how
71
+dependencies are created for swap units. With `noauto`, the swap unit will not be added as a
72
+dependency for `swap.target`. This means that it will not be activated automatically during boot,
73
+unless it is pulled in by some other unit. The auto option has the opposite meaning and is the
74
+default. With `nofail`, the swap unit will be only wanted, not required by `swap.target`. This means
75
+that the boot will continue even if this swap device is not activated
76
+successfully.
77
+
78
+</details>
79
+
80
+
81
+<details>
82
+<summary>References and source</summary>
83
+
84
+1. [Systemd on Wikipedia](https://en.wikipedia.org/wiki/Systemd)
85
+2. [Man page for systemd.swap](https://www.freedesktop.org/software/systemd/man/systemd.swap.html)
86
+
87
+</details>
88
+
89
+### Troubleshooting section:
90
+
91
+<details>
92
+<summary>General approach</summary>
93
+
94
+Check the log messages for failing reasons:
95
+
96
+ ```
97
+ root@netdata # journalctl -xe | grep -A 5 -B 5 swap
98
+ ```
99
+
100
+</details>
101
+
102
+
103
+<details>
104
+<summary>Check your fstab for errors</summary>
105
+
106
+Open the fstab config file and verify the syntax of the fstab entries with `TYPE=swap` are
107
+ correct.
108
+
109
+ ```
110
+ root@netdata # vim /etc/fstab
111
+ ```
112
+
113
+ Consult the [man pages of fstab](https://www.man7.org/linux/man-pages/man5/fstab.5.html) for
114
+ misconfigurations.
115
+
116
+</details>
health/guides/systemdunits/systemd_target_units_state.md
new
+103
@@ -0,0 +1,103 @@
1
+# systemd_target_units_state
2
+
3
+**Linux | Systemd units**
4
+
5
+_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
+manager that runs as PID 1 and starts the rest of the system._
7
+
8
+The Netdata Agent monitors the systemd target units state. Receiving this alert indicates that one
9
+or more of the systemd target units are in the failed state. A systemd target unit "failed" when
10
+the service process returned error code on exit, or crashed, an operation timed out, or after
11
+too many restarts. The cause of a failed states is stored in a log.
12
+
13
+<details>
14
+<summary>Read more about systemd</summary>
15
+
16
+Here is some useful information about systemd from
17
+wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
+
19
+Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
+and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
+entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
+daemons
23
+`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
+Poettering described systemd not as one program, but rather a large software suite that includes 69
25
+individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
+runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
+its control. systemd also integrates many other services that are common on Linux systems by
28
+handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
+logging, hostnames and locales.
30
+
31
+Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
+itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
+daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
+process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
+parent of a process when the original parent terminates. Therefore, the first process is
36
+particularly well suited for the purpose of monitoring daemons.
37
+
38
+Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
+the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
+Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
+be preserved in a snapshot for future recall.
42
+
43
+Systemd's core components include the following:
44
+
45
+- `systemd` is a system and service manager for Linux operating systems.
46
+
47
+- `systemctl` is a command to introspect and control the state of the systemd system and service
48
+ manager. Not to be confused with sysctl.
49
+
50
+- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
+ other state and tracing information from the system and service manager.
52
+
53
+</details>
54
+
55
+<details>
56
+<summary>See more on systemd.target units</summary>
57
+
58
+> Target units file ends with the `.target` file extension and their only purpose is to group together
59
+other systemd units through a chain of dependencies. For example, the graphical.target unit, which
60
+is used to start a graphical session, starts system services such as the GNOME Display Manager (
61
+gdm.service) or Accounts Service (accounts-daemon .service) and also activates the multi-user.target
62
+unit. Similarly, the multi-user.target unit starts other essential system services such as
63
+NetworkManager (NetworkManager.service) or D-Bus (dbus.service) and activates another target unit
64
+named
65
+basic.target. <sup> [2](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/working-with-systemd-targets_configuring-basic-system-settings) </sup>
66
+>
67
+> Among other things, target units are a more flexible replacement for SysV runlevels in the classic
68
+SysV init system. For compatibility reasons special target units such as runlevel3.target exist
69
+which are used by the SysV runlevel compatibility code in systemd.
70
+
71
+</details>
72
+
73
+
74
+<details>
75
+<summary>References and source</summary>
76
+
77
+1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
78
+2. [systemd.target explained on Redhat's documentantion](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/working-with-systemd-targets_configuring-basic-system-settings)
79
+
80
+</details>
81
+
82
+### Troubleshooting section:
83
+
84
+<details>
85
+<summary>General approach</summary>
86
+
87
+When a target is in a failed state, you should always try to gather more information about it.
88
+
89
+1. Identify which target units fail. Open the Netdata dashboard, find the current active alarms under
90
+ the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
91
+ into its chart.
92
+ (`systemdunits_target_units.target_unit_state`). In this chart, identify which target
93
+ units are in state with value 5
94
+
95
+2. Gather more information about the failing target unit
96
+
97
+ ```
98
+ root@netdata~ # systemctl status <target_name>.target
99
+ ```
100
+
101
+3. Check the log messages from the command of step 2.
102
+
103
+</details>
health/guides/tcp_conn/tcp_connections.md
new
+31
@@ -0,0 +1,31 @@
1
+# tcp_connections
2
+
3
+## OS: Linux
4
+
5
+This alert presents the percentage of used IPv4 TCP connections. If you receive it, it is an
6
+indication of high IPv4 TCP connections utilization.
7
+
8
+If this value is 100% then the system is no longer able to establish new TCP connections.
9
+
10
+<details>
11
+
12
+<summary>TCP Connections Alarm Settings</summary>
13
+
14
+Inside the [tcp_conn.conf](
15
+https://github.com/netdata/netdata/blob/master/health/health.d/tcp_conn.conf), on the `calc:` line,
16
+there is this block of code:
17
+`(${tcp_max_connections} > 0) ? ( ${connections} * 100 / ${tcp_max_connections} ) : 0`
18
+
19
+- That line of code will calculate the value of `$this` in the following lines.
20
+- Essentially, if the max connections are not dynamic, and there is a limit, then we calculate the
21
+ percentage of used IPv4 TCP connections. Otherwise, we have a dynamic threshold *(
22
+ so `$ {tcp_max_connections}` may be nan or -1)*, which in this case the alert and `$this` will
23
+ always be zero.
24
+
25
+</details>
26
+
27
+<br>
28
+
29
+- This alert is raised to a state of warning when the percentage of used IPv4 TCP connections is
30
+greater than 80% and less than 90%.
31
+- If the percentage of used IPv4 TCP connections exceeds 90%, then the alert gets raised to critical.
health/guides/tcp_listen/1m_tcp_accept_queue_drops.md
new
+52
@@ -0,0 +1,52 @@
1
+# 1m_tcp_accept_queue_drops
2
+
3
+## OS: Linux
4
+
5
+This alert presents the average number of dropped packets in the TCP accept queue over the last
6
+sixty seconds. If it is raised, then the system is dropping incoming TCP connections. This could also be
7
+an indication of accepted queue overflow, low memory, security issues, no route to a destination,
8
+etc.
9
+- This alert gets raised to warning when the value is greater than 1 and less than 5.
10
+- If the number of queue drops over the last minute exceeds 5, then the alert gets raised to critical.
11
+
12
+
13
+<details>
14
+<summary>TCP Accept Queue Drops</summary>
15
+
16
+The accept queue holds fully established TCP connections waiting to be handled by the listening
17
+application. It overflows when the server application fails to accept new connections at the rate
18
+they are coming in.
19
+
20
+</details>
21
+
22
+<details>
23
+ <summary>References and sources</summary>
24
+
25
+1. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
26
+2. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
27
+
28
+</details>
29
+
30
+### Troubleshooting Section
31
+
32
+<details>
33
+<summary>Check for queue overflows</summary>
34
+
35
+If you receive this alert, then you can cross-check its results with the
36
+`1m_tcp_accept_queue_overflows` alert. If that alert is also in a warning or critical state,
37
+then the system is experiencing accept queue overflowing. To fix that you can do the following:
38
+
39
+1. Open the /etc/sysctl.conf file and look for the entry " net.ipv4.tcp_max_syn_backlog".
40
+ > The `tcp_max_syn_backlog` is the maximal number of remembered connection requests
41
+ > (SYN_RECV), which have not received an acknowledgment from connecting client. <sup> [1](
42
+ > https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt) </sup>
43
+2. If the entry does not exist, then append the following default entry to the
44
+ file; `net.ipv4.tcp_max_syn_backlog=1280`. Otherwise, adjust the limit to suit your needs.
45
+3. Save your changes and run;
46
+ ```
47
+ root@netdata~ #sysctl -p
48
+ ```
49
+ to apply the changes.
50
+
51
+> Note: Netdata strongly suggests knowing exactly what values you need before making system changes.
52
+</details>
health/guides/tcp_listen/1m_tcp_accept_queue_overflows.md
new
+54
@@ -0,0 +1,54 @@
1
+# 1m_tcp_accept_queue_overflows
2
+
3
+## OS: Linux
4
+
5
+This alert presents the average number of overflows in the TCP accept queue over the last minute.
6
+
7
+- This alert gets raised in a warning state when the value is greater than 1 and less than 5.
8
+- If the overflow average exceeds 5 in the last minute, then the alert gets raised in the critical
9
+ state.
10
+
11
+<details>
12
+<summary>What is the Accept queue</summary>
13
+
14
+The accept queue holds fully established TCP connections waiting to be handled by the listening
15
+application. It overflows when the server application fails to accept new connections at the rate
16
+they are coming in.
17
+
18
+</details>
19
+
20
+This alert might also indicate a SYN flood;
21
+> A SYN flood is a form of denial-of-service attack in which an attacker rapidly initiates a
22
+> connection to a server without finalizing the connection. The server has to spend resources
23
+> waiting for half-opened connections, which can consume enough resources to make the system
24
+> unresponsive to legitimate traffic. <sup> [1](https://en.wikipedia.org/wiki/SYN_flood) </sup>
25
+
26
+
27
+<details>
28
+ <summary>References and sources</summary>
29
+
30
+1. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
31
+2. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
32
+3. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
33
+
34
+</details>
35
+
36
+### Troubleshooting Section
37
+
38
+<details>
39
+<summary>Increase the queue length</summary>
40
+
41
+1. Open the /etc/sysctl.conf file and look for the entry " net.ipv4.tcp_max_syn_backlog".
42
+ > The `tcp_max_syn_backlog` is the maximal number of remembered connection requests
43
+ > (SYN_RECV), which have not received an acknowledgment from connecting client. <sup> [2](
44
+ > https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt) </sup>
45
+2. If the entry does not exist, you can append the following default entry to the
46
+ file; `net.ipv4. tcp_max_syn_backlog=1280`. Otherwise, adjust the limit to suit your needs.
47
+3. Save your changes and run;
48
+ ```
49
+ root@netdata~ #sysctl -p
50
+ ```
51
+ to apply the changes.
52
+
53
+Netdata strongly suggests knowing exactly what values you need before making system changes.
54
+</details>
\ No newline at end of file
health/guides/tcp_listen/1m_tcp_syn_queue_cookies.md
new
+87
@@ -0,0 +1,87 @@
1
+# 1m_tcp_syn_queue_cookies
2
+
3
+## OS: Linux
4
+
5
+This alert presents the average number of sent SYN cookies due to the full TCP SYN queue over the
6
+sixty seconds. Receiving this means that the incoming traffic is excessive. SYN queue cookies are
7
+used to resist any potential SYN flood attacks.
8
+
9
+This alert is raised to warning when the average exceeds 1 and will enter critical when the
10
+value exceeds an average of 5 sent SYN cookies in sixty seconds.
11
+
12
+<details>
13
+<summary>SYN Queue Cookies</summary>
14
+
15
+> The SYN Queue stores inbound SYN packets (specifically: struct inet_request_sock). It is
16
+> responsible for sending out SYN+ACK packets and retrying them on timeout. \
17
+> After transmitting the SYN+ACK, the SYN Queue waits for an ACK packet from the client - the last
18
+> packet in the three-way-handshake. All received ACK packets must first be matched against the
19
+> fully established connection table, and only then against data in the relevant SYN Queue. On
20
+> SYN Queue match, the kernel removes the item from the SYN Queue, successfully creates a full
21
+> connection (specifically: struct inet_sock), and adds it to the Accept Queue.<sup> [1](
22
+> https://blog.cloudflare.com/syn-packet-handling-in-the-wild/) </sup>
23
+</details>
24
+
25
+<br>
26
+
27
+<details>
28
+<summary>This alert likely indicates a SYN flood.</summary>
29
+
30
+> A SYN flood is a form of denial-of-service attack in which an attacker rapidly initiates a
31
+> connection to a server without finalizing the connection. The server has to spend resources
32
+> waiting for half-opened connections, which can consume enough resources to make the system
33
+> unresponsive to legitimate traffic. <sup> [2](https://en.wikipedia.org/wiki/SYN_flood) </sup>
34
+</details>
35
+
36
+<br>
37
+
38
+<details>
39
+<summary>SYN Cookies</summary>
40
+
41
+> SYN cookies are a technique used to resist IP address spoofing attacks. The technique's primary
42
+> inventor, Daniel J. Bernstein, defines SYN cookies as "particular choices of initial TCP sequence
43
+> numbers by TCP servers." In particular, the use of SYN cookies allows a server to avoid dropping
44
+> connections when the SYN queue fills up. Instead of storing additional connections, a SYN queue
45
+> entry is encoded into the sequence number sent in the SYN+ACK response. If the server then
46
+> receives a subsequent ACK response from the client with the incremented sequence number, then the
47
+> server is able to reconstruct the SYN queue entry using information encoded in the TCP
48
+> sequence number and proceed as usual with the connection. <sup> [3](
49
+> https://en.wikipedia.org/wiki/SYN_cookies) </sup>
50
+</details>
51
+
52
+<br>
53
+
54
+<details>
55
+ <summary>References and sources</summary>
56
+
57
+1. [SYN packet handling](https://blog.cloudflare.com/syn-packet-handling-in-the-wild/)
58
+2. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
59
+3. [SYN Cookies](https://en.wikipedia.org/wiki/SYN_cookies)
60
+4. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
61
+5. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
62
+
63
+</details>
64
+
65
+### Troubleshooting Section
66
+
67
+<details>
68
+<summary>If the traffic is legitimate, then increase the limit of the SYN queue.</summary>
69
+
70
+If you can determine that the traffic is legitimate, consider expanding the limit of the SYN
71
+queue through configuration; \
72
+*(If the traffic is not legitimate, then this is not safe! You will
73
+expose more resources to an attacker if the traffic is not legitimate.)*
74
+
75
+1. Open the /etc/sysctl.conf file and look for the entry "net.core.somaxconn". This value will
76
+ affect both SYN and accept queue limits on newer Linux systems.
77
+2. Set the value accordingly (By default it is set to 128) `net.core.somaxconn=128` (if the value
78
+ doesn't exist, append it to the file)
79
+3. Save your changes and run;
80
+ ```
81
+ root@netdata~ #sysctl -p
82
+ ```
83
+ to apply the changes.
84
+
85
+> Note: Netdata strongly suggests knowing exactly what you are configuring before making system
86
+> changes.
87
+</details>
health/guides/tcp_listen/1m_tcp_syn_queue_drops.md
new
+82
@@ -0,0 +1,82 @@
1
+# 1m_tcp_syn_queue_drops
2
+
3
+## OS: Linux
4
+
5
+This alert presents the average number of SYN requests that were dropped due to the TCP SYN queue
6
+being full over the last sixty seconds. Receiving this means that the TCP SYN queue is full and that the
7
+system is dropping incoming TCP SYN requests.
8
+
9
+The alert is raised to warning when this average exceeds 1 per minute. The alert is raised to critical
10
+when the value exceeds an average of 5 dropped SYN packets per sixty seconds.
11
+
12
+<details>
13
+<summary>What is the SYN queue?</summary>
14
+
15
+> The SYN Queue stores inbound SYN packets (specifically: struct inet_request_sock). It's
16
+> responsible for sending out SYN+ACK packets and retrying them on timeout. \
17
+> After transmitting the SYN+ACK, the SYN Queue waits for an ACK packet from the client - the last
18
+> packet in the three-way-handshake. All received ACK packets must first be matched against the
19
+> fully established connection table, and only then against data in the relevant SYN Queue. On
20
+> SYN Queue match, the kernel removes the item from the SYN Queue, happily creates a fully fledged
21
+> connection (specifically: struct inet_sock), and adds it to the Accept Queue.<sup> [1](
22
+> https://blog.cloudflare.com/syn-packet-handling-in-the-wild/) </sup>
23
+</details>
24
+
25
+<br>
26
+
27
+<details>
28
+<summary>This alert has a high probability of indicating a SYN flood.</summary>
29
+
30
+> A SYN flood is a form of denial-of-service attack in which an attacker rapidly initiates a
31
+> connection to a server without finalizing the connection. The server has to spend resources
32
+> waiting for half-opened connections, which can consume enough resources to make the system
33
+> unresponsive to legitimate traffic. <sup> [2](https://en.wikipedia.org/wiki/SYN_flood) </sup>
34
+
35
+</details>
36
+
37
+<br>
38
+
39
+<details>
40
+<summary>SYN Cookies</summary>
41
+
42
+> SYN cookies are a technique used to resist IP address spoofing attacks. The technique's primary
43
+> inventor, Daniel J. Bernstein, defines SYN cookies as "particular choices of initial TCP sequence
44
+> numbers by TCP servers." In particular, the use of SYN cookies allows a server to avoid dropping
45
+> connections when the SYN queue fills up. Instead of storing additional connections, a SYN queue
46
+> entry is encoded into the sequence number sent in the SYN+ACK response. If the server then
47
+> receives a subsequent ACK response from the client with the incremented sequence number, the
48
+> server is able to reconstruct the SYN queue entry using information encoded in the TCP
49
+> sequence number and proceed as usual with the connection. <sup> [3](
50
+> https://en.wikipedia.org/wiki/SYN_cookies) </sup>
51
+</details>
52
+<br>
53
+
54
+<details>
55
+ <summary>References and sources</summary>
56
+
57
+1. [SYN packet handling](https://blog.cloudflare.com/syn-packet-handling-in-the-wild/)
58
+2. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
59
+3. [SYN Cookies](https://en.wikipedia.org/wiki/SYN_cookies)
60
+4. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
61
+5. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
62
+
63
+</details>
64
+
65
+### Troubleshooting Section
66
+
67
+<details>
68
+<summary>Enable SYN Cookies</summary>
69
+
70
+If you can determine that the traffic is legitimate, consider enabling SYN cookies. To enable SYN cookies:
71
+
72
+1. Open the /etc/sysctl.conf file and look for the entry "net.ipv4.tcp_syncookies".
73
+2. Set the value to 1. `net.ipv4.tcp_syncookies=1` (if the value doesn't exist, append it to the
74
+ file)
75
+3. Save your changes and run;
76
+ ```
77
+ root@netdata~ #sysctl -p
78
+ ```
79
+ to apply the changes.
80
+
81
+Netdata strongly suggests knowing exactly what you are configuring before making system changes.
82
+</details>
health/guides/tcp_mem/tcp_memory.md
new
+111
@@ -0,0 +1,111 @@
1
+# tcp_memory
2
+
3
+## OS: Linux
4
+
5
+The Netdata Agent calculates the percentage of used TCP memory. Receiving this alert indicates that
6
+the TCP memory utilization uses more memory than the limit.
7
+
8
+By default, the Linux network stack is not configured for high speed large file transfer across WAN
9
+links. This is done to save memory resources. System performs out of memory checks, if the memory
10
+used by the TCP protocol is higher than the third value(max) of the `net.ipv4.tcp_mem`, it throws
11
+OOM error. This will make some applications to become unresponsive.
12
+
13
+
14
+
15
+<details>
16
+<summary>See more about TCP buffers </summary>
17
+
18
+There are 3 main parameters to configure regarding the TCP buffers.
19
+
20
+> - `tcp_mem` is a vector of 3 integers: `"low, pressure, high"`. These bounds, measured in units of
21
+ the system page size, are used by TCP to track its memory usage. The defaults are calculated at
22
+ boot time from the amount of available memory. (TCP can only use low memory for this, which is
23
+ limited to around 900 megabytes on 32-bit systems. 64-bit systems do not suffer this limitation.)
24
+
25
+> - low TCP doesn't regulate its memory allocation when the number of pages it has allocated
26
+ globally is below this number.
27
+> - pressure When the amount of memory allocated by TCP exceeds this number of pages, TCP
28
+ moderates its memory consumption. This memory pressure state is exited once the number of
29
+ pages allocated falls below the low mark.
30
+> - high The maximum number of pages, globally, that TCP will allocate. This value overrides any
31
+ other limits imposed by the kernel. <sup>[1](https://man7.org/linux/man-pages/man7/tcp.7.html) </sup>
32
+
33
+The min/pressure/max TCP buffer space are automatically set in `/proc/sys/net/ipv4/tcp_mem` during
34
+the boot time based on available RAM size.
35
+
36
+> - `net.ipv4.tcp_rmem` contains three values that represent `"minimum default maximum_size` of the
37
+ TCP socket receive buffer.
38
+
39
+> - The minimum represents the smallest receive buffer size guaranteed, even under memory
40
+ pressure. The minimum value defaults to 1 page or 4096 bytes.
41
+
42
+> - The default value represents the initial size of a TCP sockets receive buffer. This value
43
+ supersedes net.core.rmem_default used by other protocols. The default value for this setting
44
+ is 87380 bytes. It also sets the tcp_adv_win_scale and initializes the TCP window size to
45
+ 65535 bytes.
46
+
47
+> - The maximum represents the largest receive buffer size automatically selected for TCP sockets.
48
+ This value does not override net.core.rmem_max. The default value for this setting is
49
+ somewhere between 87380 bytes and 6M bytes based on the amount of memory in the system.
50
+>
51
+>
52
+> The recommendation is to use the maximum value of 16M bytes or higher (kernel level dependent)
53
+ especially for 10 Gigabit adapters.
54
+>
55
+>
56
+> - `net.ipv4.tcp_wmem` parameter also consists of 3 values `"minimum default maximum"`.
57
+> - The minimum represents the smallest receive buffer size a newly created socket is entitled to
58
+ as part of its creation. The minimum value defaults to 1 page or 4096 bytes.
59
+> - The default value represents the initial size of a TCP sockets receive buffer. This value
60
+ supersedes net.core.rmem_default used by other protocols. It is typically set lower than
61
+ net.core.wmem_default. The default value for this setting is 16K bytes.
62
+> - The maximum represents the largest receive buffer size for auto-tuned send buffers for TCP
63
+ sockets. This value does not override net.core.rmem_max. The default value for this setting is
64
+ somewhere between 64K bytes and 4M bytes based on the amount of memory available in the
65
+ system.
66
+>
67
+>
68
+>The recommendation is to use the maximum value of 16M bytes or higher (kernel level dependent)
69
+especially for 10 Gigabit adapters. <sup>[2](https://www.ibm.com/docs/en/linux-on-systems?topic=tuning-tcpip-ipv4-setting) </sup>
70
+
71
+</details>
72
+
73
+<details>
74
+<summary>References and sources</summary>
75
+
76
+1. [man pages of tcp](https://man7.org/linux/man-pages/man7/tcp.7.html)
77
+1. [Adjustments for IPv4 settings from IBM](https://www.ibm.com/docs/en/linux-on-systems?topic=tuning-tcpip-ipv4-settings)
78
+</details>
79
+
80
+### Troubleshooting section:
81
+
82
+<details>
83
+<summary>Increase the TCP memory </summary>
84
+
85
+Increasing the TCP memory available in the Linux network stack may resolve this issue.
86
+
87
+1. Try to increase the `tcp_mem` bounds
88
+
89
+ ```
90
+ root@netdata # sysctl -w net.ipv4.tcp_mem="819200 1091174 1638400"
91
+ ```
92
+
93
+1. Verify the change and test it with the same workload that triggered the alarm originally.
94
+ If the problem still exists, you can always consider increase it more.
95
+
96
+ ```
97
+ root@netdata~ # sysctl net.ipv4.tcp_mem
98
+ net.ipv4.tcp_mem=819200 1091174 1638400
99
+ ```
100
+
101
+1. If this change works for your system, you could make it permanently. Bump these entries
102
+ under `/etc/sysctl.conf`
103
+
104
+
105
+1. Reload the sysctl settings.
106
+
107
+ ```
108
+ root@netdata~ # sysctl -p
109
+ ```
110
+
111
+
health/guides/tcp_orphans/tcp_orphans.md
new
+82
@@ -0,0 +1,82 @@
1
+# tcp_orphans
2
+
3
+## OS: Linux
4
+
5
+This alert presents the percentage of used orphan IPV4 TCP sockets. If it is raised, it indicates
6
+that your system is experiencing high IPv4 TCP sockets utilization.
7
+When the system exceeds the limit, orphaned connections (connections not attached to any user filehandle) are
8
+reset immediately.
9
+
10
+This alert is triggered in warning state when the percentage of used orphan IPv4 TCP sockets is
11
+above 25% and in critical state when that value exceeds 50%.
12
+
13
+
14
+<Details>
15
+<summary>What is a network socket</summary>
16
+
17
+> A network socket is a software structure within a network node of a computer network that
18
+> serves as an endpoint for sending and receiving data across the network. The structure and
19
+> properties of a socket are defined by an application programming interface (API) for the
20
+> networking architecture. Sockets are created only during the lifetime of a process of an
21
+> application running in the node.
22
+
23
+> Because of the standardization of the TCP/IP protocols in the development of the Internet, the
24
+> term network socket is most commonly used in the context of the Internet protocol suite, and
25
+> is therefore often also referred to as Internet socket.
26
+
27
+> In this context, a socket is externally identified to other hosts by its socket address,
28
+> which is the triad of transport protocol, IP address, and port number.
29
+> <sup>[1](https://en.wikipedia.org/wiki/Network_socket) </sup>
30
+
31
+</Details>
32
+
33
+<br>
34
+
35
+<details>
36
+<summary>What is a "filehandle" or "socket descriptor"</summary>
37
+
38
+> The application programming interface (API) for the network protocol stack creates a handle
39
+> for each socket created by an application, commonly referred to as a socket descriptor. In
40
+> Unix-like operating systems, this descriptor is a type of file descriptor. It is stored by
41
+> the application process for use with every read and write operation on the communication channel.
42
+> <sup>[1](https://en.wikipedia.org/wiki/Network_socket) </sup>
43
+
44
+</details>
45
+
46
+<br>
47
+
48
+> An orphan socket is a socket that isn't associated with a file descriptor, usually after the
49
+> close() call and there is no longer a file descriptor that reference it, but the socket still
50
+> exists in memory, until TCP is done with it.<sup> [2](
51
+> http://www.linux-admins.net/2013/01/troubleshooting-out-of-socket-memory.html) </sup>
52
+
53
+<br>
54
+
55
+<details>
56
+<summary> References and Sources </summary>
57
+
58
+1. [Network_sockets](https://en.wikipedia.org/wiki/Network_socket)
59
+2. [Linux-admins.com](http://www.linux-admins.net/2013/01/troubleshooting-out-of-socket-memory.html)
60
+</sup>
61
+
62
+</details>
63
+
64
+### Troubleshooting Section
65
+
66
+<details>
67
+<summary>Increase the orphan socket limit</summary>
68
+
69
+To counteract this behavior, you can increase the limit in the
70
+file: `/proc/sys/net/ipv4/tcp_max_orphans`. Simply run:
71
+
72
+```
73
+root@netdata~ # echo {DESIRED_AMOUNT} > /proc/sys/net/ipv4/tcp_max_orphans
74
+```
75
+
76
+The kernel may penalize orphans by 2x or even 4x (hence the small warning and critical thresholds).
77
+You may need to watch for orphaned sockets during peak hours and consider multiplying that number by
78
+3 or 4. That should give you a good starting point.
79
+
80
+> Note: Netdata strongly suggests knowing exactly what you are configuring before making system
81
+> changes.
82
+</details>
health/guides/tcp_resets/10s_ipv4_tcp_resets_received.md
new
+142
@@ -0,0 +1,142 @@
1
+# 10s_ipv4_tcp_resets_received
2
+
3
+## OS: Linux, FreeBSD
4
+
5
+TCP reset is an abrupt closure of the session. It causes the resources allocated to the connection
6
+to be immediately released and all other information about the connection to be erased.
7
+
8
+The Netdata Agent monitors the average number of received TCP RESETS over the last 10 seconds. This
9
+can indicate that the system is trying to establish a connection to a server port on which no
10
+process is listening. This can also indicate a SYN reset attack.
11
+
12
+<details>
13
+ <summary>See more about TCP Resets </summary>
14
+
15
+TCP uses a three-way handshake to establish a reliable connection. The connection is full duplex,
16
+and both sides synchronize (SYN) and acknowledge (ACK) each other. The exchange of these four flags
17
+is performed in three steps: SYN, SYN-ACK, and ACK
18
+
19
+When an unexpected TCP packet arrives at a host, that host usually responds by sending a reset
20
+packet back on the same connection. A reset packet is one with no payload and with the RST bit set
21
+in the TCP header flags. There are a few circumstances in which a TCP packet might not be expected.
22
+The most common cases are:
23
+
24
+ 1. A TCP packet received in a non-existed TCP PORT
25
+
26
+1. An aborting connection
27
+
28
+1. Half opened connections
29
+
30
+1. Time wait assassination
31
+
32
+1. Listening endpoint Queue is Full
33
+
34
+1. A TCP Buffer Overflow
35
+
36
+Basically, A TCP Reset usually occurs when a system receives data which doesn't agree with its view
37
+of the connection.
38
+
39
+</details>
40
+
41
+
42
+<details>
43
+
44
+ <summary>References and source:</summary>
45
+
46
+1. [TCP reset explanation](https://www.pico.net/kb/what-is-a-tcp-reset-rst/)
47
+1. [TCP 3-way handshake on wikipedia](https://en.wikipedia.org/wiki/Handshaking)
48
+
49
+
50
+</details>
51
+
52
+### Troubleshooting section:
53
+
54
+<details>
55
+
56
+ <summary>General approach</summary>
57
+
58
+Try using Wireshark to inspect the network packets.
59
+
60
+Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting,
61
+analysis, software and communications protocol development.
62
+
63
+[See more about Wireshark here](https://www.wireshark.org/)
64
+
65
+Since you might won't be able to probe your traffic with wireshark in your host machine, You can
66
+export it in a dump file and analyze it in a second iteration.
67
+
68
+1. Try to export the traffic in your host with `tcpdump`.
69
+
70
+ ```
71
+ root@netdata # tcpdump -i any 'tcp[tcpflags] & (tcp-rst) == (tcp-rst)' -s 65535 -w output.pcap
72
+ ```
73
+
74
+You must stop the capture after a certain observation period (60s up to 5 minutes). This command
75
+will create a dump file which can be interpreted by Wireshark that contains all the TCP packets with
76
+RST flag set.
77
+
78
+2. Copy this file in your workstation and examine it with Wireshark.
79
+
80
+</details>
81
+
82
+
83
+<details>
84
+
85
+ <summary>Counter measure on malicious TCP resets</summary>
86
+
87
+SYN cookie is a technique used to resist IP address spoofing attacks. In particular, the use of SYN
88
+cookies allows a server to avoid dropping connections when the SYN queue fills up.
89
+
90
+ <details>
91
+
92
+ <summary>Enable SYN cookies in Linux</summary>
93
+
94
+ 1. Check if your system has the SYN cookies service enabled
95
+
96
+ ```
97
+ root@netdata # cat /proc/sys/net/ipv4/tcp_syncookies
98
+ ```
99
+ If the value is 1, then the service is enabled, if not proceed to step 2.
100
+
101
+
102
+ 2. Bump this `net.ipv4.tcp_syncookies=1` value under `/etc/sysctl.conf`
103
+
104
+
105
+ 3. Apply the configuration
106
+
107
+ ```
108
+ root@netdata # sysctl -p to apply the configuration.
109
+ ```
110
+
111
+ </details>
112
+
113
+ <details>
114
+
115
+ <summary>Enable SYN cookies in FreeBSD</summary>
116
+
117
+ 1. Check if your system has the SYN cookies service enabled
118
+
119
+ ```
120
+ root@netdata # sysctl net.inet.tcp.syncookies_only
121
+ ```
122
+ If the value is 1, then the service is enabled, if not proceed to step 2.
123
+
124
+
125
+ 2. Bump this `net.inet.tcp.syncookies_only=1` value under `/etc/sysctl.conf`
126
+
127
+
128
+ 3. Apply the configuration
129
+
130
+ ```
131
+ root@netdata~ # /etc/rc.d/sysctl reload
132
+ ```
133
+
134
+
135
+ </details>
136
+
137
+
138
+The use of SYN cookies does not break any protocol specifications, and therefore should be
139
+compatible with all TCP implementations. There are, however, a few caveats that take effect when SYN
140
+cookies are in use.
141
+</details>
142
+
health/guides/tcp_resets/10s_ipv4_tcp_resets_sent.md
new
+103
@@ -0,0 +1,103 @@
1
+# 10s_ipv4_tcp_resets_sent
2
+
3
+## OS: Linux
4
+
5
+TCP reset is an abrupt closure of the session. It causes the resources allocated to the connection
6
+to be immediately released and all other information about the connection is erased.
7
+
8
+The Netdata Agent monitors the average number of sent TCP RESETS over the last 10 seconds. This can
9
+indicate a port scan or that a service running on the system has crashed. Additionally, it's a
10
+result of a high number of sent TCP RESETS. Furthermore, it can also indicate a SYN reset attack.
11
+
12
+
13
+<details>
14
+ <summary>See more about TCP Resets </summary>
15
+
16
+TCP uses a three-way handshake to establish a reliable connection. The connection is full duplex,
17
+and both sides synchronize (SYN) and acknowledge (ACK) each other. The exchange of these four flags
18
+is performed in three steps: SYN, SYN-ACK, and ACK
19
+
20
+When an unexpected TCP packet arrives at a host, that host usually responds by sending a reset
21
+packet back on the same connection. A reset packet is one with no payload and with the RST bit set
22
+in the TCP header flags. There are a few circumstances in which a TCP packet might not be expected.
23
+The most common cases are:
24
+
25
+1. A TCP packet received on a port that is not open.
26
+
27
+2. An aborting connection
28
+
29
+3. Half opened connections
30
+
31
+4. Time wait assassination
32
+
33
+5. Listening endpoint Queue is Full
34
+
35
+6. A TCP Buffer Overflow
36
+
37
+Basically, A TCP Reset usually occurs when a system receives data which doesn't agree with its view
38
+of the connection.
39
+
40
+When your system cannot establish a connection it will retry by default `net.ipv4.tcp_syn_retries`
41
+times.
42
+</details>
43
+
44
+
45
+<details>
46
+
47
+ <summary>References and sources</summary>
48
+
49
+1. [TCP reset explanation](https://www.pico.net/kb/what-is-a-tcp-reset-rst/)
50
+2. [TCP 3-way handshake on wikipedia](https://en.wikipedia.org/wiki/Handshaking)
51
+
52
+</details>
53
+
54
+### Troubleshooting section:
55
+
56
+<details>
57
+
58
+ <summary>General approach</summary>
59
+
60
+Try using Wireshark to inspect the network packets.
61
+
62
+Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting,
63
+analysis, software and communications protocol development.
64
+
65
+[See more about Wireshark here](https://www.wireshark.org/)
66
+
67
+Since you might won't be able to probe your traffic with wireshark in your host machine, You can
68
+export it in a dump file and analyze it in a second iteration.
69
+
70
+1. Try to export the traffic in your host with `tcpdump`.
71
+
72
+ ```
73
+ root@netdata # tcpdump -i any 'tcp[tcpflags] & (tcp-rst) == (tcp-rst)' -s 65535 -w output.pcap
74
+ ```
75
+
76
+You must stop the capture after a certain observation period (60s up to 5 minutes). This command
77
+will create a dump file which can be interpreted by Wireshark that contains all the TCP packets with
78
+RST flag set.
79
+
80
+2. Copy this file in your workstation and examine it with Wireshark.
81
+
82
+</details>
83
+
84
+<details>
85
+
86
+ <summary>Identify which application sends TCP resets</summary>
87
+
88
+1. Inspect the packet flow with a packet sniffer like Wireshark. You can consult the _General
89
+ approach_ troubleshooting action in the current guide.
90
+
91
+
92
+2. Check the instances of `RST` events of the TCP protocol. Wireshark also displays the ports on
93
+ which the two systems tried to establish the TCP connection, (XXXXXX -> XXXXXX).
94
+
95
+
96
+3. To check which application is using this port, run the following code:
97
+
98
+ ```
99
+ root@netdata # lsof -i:XXXXXX -P -n
100
+ ```
101
+
102
+</details>
103
+
health/guides/timex/system_clock_sync_state.md
new
+21
@@ -0,0 +1,21 @@
1
+# system_clock_sync_state
2
+
3
+## OS: Linux
4
+
5
+The Netdata Agent checks if your system is in sync with a Network Time Protocol (NTP) server. This
6
+alert indicates that the system time is not synchronized to a reliable server. It is strongly
7
+recommended having the clock in sync with NTP servers, because, otherwise, it leads to unpredictable
8
+problems that are difficult to debug especially in matters of security.
9
+
10
+Here you can find a great article on
11
+[best practices for NTP servers](https://bluecatnetworks.com/blog/seven-best-practices-to-keep-your-ntp-resilient/).
12
+
13
+# Troubleshooting section:
14
+
15
+<details>
16
+<summary> General approach </summary>
17
+
18
+Different linux distros utilize different NTP tools. You can always install `ntp`. If your clock is
19
+out of sync, you should first check for issues in your network connectivity.
20
+</details>
21
+
health/guides/udp_errors/1m_ipv4_udp_receive_buffer_errors.md
new
+155
@@ -0,0 +1,155 @@
1
+# 1m_ipv4_udp_receive_buffer_errors
2
+
3
+*In computer networking, the User Datagram Protocol (UDP) is one of the core members of the Internet
4
+protocol suite.*
5
+
6
+In both Linux and FreeBSD variants, the kernel allocates buffers to serve the UDP protocol operations.
7
+Packets after reception from a network interface are forwarded to these buffers to be processed by
8
+the UDP protocol stack in a system's socket.
9
+
10
+The Netdata Agent monitors the average number of UDP receive buffer errors over the last minute.
11
+Receiving this alert means that your system is dropping incoming UDP packets. This may indicate that
12
+the UDP receive buffer queue is full.
13
+
14
+This alert is triggered in warning state when the number of UDP receive buffer errors over the last
15
+minute is more than 10.
16
+
17
+<details>
18
+<summary>See more on UDP protocol</summary>
19
+
20
+> UDP uses a simple connectionless communication model with a minimum of protocol mechanisms. UDP
21
+provides checksums for data integrity, and port numbers for addressing different functions at the
22
+source and destination of the datagram. It has no handshaking dialogues, and thus exposes the user's
23
+program to any unreliability of the underlying network. There is no guarantee of delivery, ordering,
24
+or duplicate protection.<sup>[1](https://en.wikipedia.org/wiki/User_Datagram_Protocol) </sup> If no
25
+firewall exists any host can send udp packets to any port, which your server doesn't listen.
26
+
27
+</details>
28
+
29
+<details>
30
+<summary>References and sources</summary>
31
+
32
+1. [UDP definition on wikipedia](https://en.wikipedia.org/wiki/User_Datagram_Protocol)
33
+2. [Man page of UDP protocol](https://man7.org/linux/man-pages/man7/udp.7.html)
34
+3. [Redhat networking tuning guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings)
35
+4. [UDP on freebsd (blog)](https://awasihba.wordpress.com/2008/10/13/udp-on-freebsd/)
36
+
37
+
38
+</details>
39
+
40
+## OS: Linux
41
+
42
+Receive buffer sizes for UDP are controlled by 4
43
+variables.<sup> [2](https://man7.org/linux/man-pages/man7/udp.7.html) </sup>
44
+
45
+- `net.core.rmem_default`, the default setting of the socket receive buffer in bytes.
46
+
47
+- `net.core.rmem_max`, the maximum receive socket buffer size in bytes.. Each socket gets
48
+ `rmem_default` receive buffer size by default, and can request up to `rmem_max` with `setsockopt`
49
+ option `SO_RCVBUF`.
50
+
51
+- `net.ipv4.udp_mem`, this is a vector of three integers (min, pressure, max) governing the number
52
+ of pages allowed for queueing by all UDP sockets.
53
+ - min: Below this number of pages, UDP is not bothered about its memory appetite. When the
54
+ amount of memory allocated by UDP exceeds this number, UDP starts to moderate memory usage.
55
+ - pressure: This value was introduced to follow the format of tcp_mem (see tcp(7)).
56
+ - max: Defaults values for these three items are calculated at boot time from the amount of
57
+ available memory.
58
+
59
+- `net.ipv4.udp_rmem_min`, the minimal size (in bytes) of receive buffer used by UDP sockets in
60
+ moderation. Each UDP socket is able to use the size for receiving data, even if total pages of UDP
61
+ sockets exceed udp_mem pressure.
62
+
63
+In general, issues with buffers that allocated dynamically are correlated with the kernel
64
+memory, you must always be aware of memory pressure events. This can cause buffer errors.
65
+
66
+### Troubleshooting section:
67
+
68
+ <details>
69
+ <summary>Increase the net.core.rmem_default and net.core.rmem_max values</summary>
70
+
71
+1. Try to increase them, RedHat suggests the value of 262144
72
+ bytes <sup> [3](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings) </sup>
73
+
74
+ ```
75
+ sysctl -w net.core.rmem_default=262144
76
+ sysctl -w net.core.rmem_max=262144
77
+ ```
78
+
79
+1. Verify the change and test with the same workload that triggered the alarm originally.
80
+
81
+ ```
82
+ root@netdata~ # sysctl net.core.rmem_default net.core.rmem_max
83
+ net.core.rmem_default=262144
84
+ net.core.rmem_max=262144
85
+ ```
86
+
87
+1. If this change works for your system, you could make it permanently.
88
+
89
+ Bump these `net.core.rmem_default=262144` & `net.core.rmem_max=262144` entries under
90
+ `/etc/sysctl.conf`.
91
+
92
+1. Reload the sysctl settings.
93
+
94
+ ```
95
+ root@netdata~ # sysctl -p
96
+ ```
97
+
98
+</details>
99
+
100
+## OS: FreeBSD
101
+
102
+Buffer space for any UDP connection on freebsd is affected by following parameters, as mentioned by
103
+Awasihba in his personal blog. [4](https://awasihba.wordpress.com/2008/10/13/udp-on-freebsd/)
104
+
105
+- `net.inet.udp.recvspace`, when you open any UDP socket this parameter decides default receiving
106
+ buffer space for userland data for that socket. You can override that size with help of
107
+ `setsockopt` in your code.
108
+
109
+- `kern.ipc.maxsockbuf`, the buffer space for socket, is determined by this parameter. So if
110
+ you try to open socket with large send and receive buffer, and you get error like "no buffer space
111
+ available" then you should consider tweaking `kern.ipc.maxsockbuf`. Sometimes you see frequent UDP
112
+ drops while dealing with large number of tiny UDP packets. Even if your `recvspace` buffer is not
113
+ filled up completely, still you will drop the packets. After digging around for a while we
114
+ figured out that it was happening because we were hitting another hard limit of`sockbuf->sb_mbmax`,
115
+ it specifies maximum number of `mbufs` allocated for each socket. You can increase that limit by
116
+ increasing `kern.ipc.maxsockbuf`. You need to restart related services to apply this parameter.
117
+
118
+- `kern.ipc.nmbcluster`, this parameter governs the total amount memory you have to allocate for all
119
+ the open sockets on your system. This value defines how many numbers of mbuf cluster should be
120
+ allocated. Usually each cluster is of 2k size. For example , if you are planning to open 1000
121
+ sockets with each having 8k sending and 8k size receiving buffer each socket will need 16k of
122
+ memory and in total you will need 16M (16k x 1000 ) of memory to handle all 1000 connections.
123
+
124
+In general, issues with buffers that allocated dynamically are correlated with the kernel
125
+memory, you must always be aware of memory pressure events. This can cause buffer errors.
126
+
127
+### Troubleshooting section:
128
+
129
+ <details>
130
+ <summary>Increase the kern.ipc.maxsockbuf value</summary>
131
+
132
+1. Try to set this value to at least 16MB for 10GE overall
133
+
134
+ ```
135
+ root@netdata~ # sysctl -w kern.ipc.maxsockbuf=16777216
136
+ ```
137
+
138
+1. Verify the change and test with the same workload that triggered the alarm originally.
139
+
140
+ ```
141
+ root@netdata~ # sysctl kern.ipc.maxsockbuf
142
+ kern.ipc.maxsockbuf=16777216
143
+ ```
144
+
145
+1. If this change works for your system, you could make it permanently.
146
+
147
+ Bump this `kern.ipc.maxsockbuf=16777216` entry under `/etc/sysctl.conf`.
148
+
149
+1. Reload the sysctl settings.
150
+
151
+ ```
152
+ root@netdata~ # /etc/rc.d/sysctl reload
153
+ ```
154
+
155
+</details>
health/guides/udp_errors/1m_ipv4_udp_send_buffer_errors.md
new
+96
@@ -0,0 +1,96 @@
1
+# 1m_ipv4_udp_send_buffer_errors
2
+
3
+## OS: Linux
4
+
5
+*In computer networking, the User Datagram Protocol (UDP) is one of the core members of the Internet
6
+protocol suite.*
7
+
8
+The linux kernel allocates buffers to serve the UDP protocol operations. Data is written into
9
+sockets that utilize UDP to send data to an another system/subsystem.
10
+
11
+The Netdata Agent monitors the average number of UDP send buffer errors over the last minute. This
12
+alert indicates that the UDP send buffer is full or no kernel memory available. Receiving this alert
13
+means that your system is dropping outgoing UDP packets
14
+
15
+This alert is triggered in warning state when the number of UDP send buffer errors over the last
16
+minute is more than 10.
17
+
18
+<details>
19
+<summary>See more on UDP protocol</summary>
20
+
21
+> UDP uses a simple connectionless communication model with a minimum of protocol mechanisms. UDP
22
+provides checksums for data integrity, and port numbers for addressing different functions at the
23
+source and destination of the datagram. It has no handshaking dialogues, and thus exposes the user's
24
+program to any unreliability of the underlying network. There is no guarantee of delivery, ordering,
25
+or duplicate protection.<sup>[1](https://en.wikipedia.org/wiki/User_Datagram_Protocol) </sup> If no
26
+firewall exists any host can send udp packets to any port, which your server doesn't listen.
27
+
28
+</details>
29
+
30
+
31
+Send buffer sizes for UDP are controlled by 4
32
+variables.<sup> [2](https://man7.org/linux/man-pages/man7/udp.7.html) </sup>
33
+
34
+- `net.core.wmem_default`, the default setting of the socket send buffer in bytes.
35
+
36
+- `net.core.wmem_max`, default and max socket send buffer size in bytes. Each socket gets
37
+ `wmem_default` send buffer size by default, and can request up to `wmem_max` with `setsockopt`
38
+ option `SO_SNDBUF`.
39
+
40
+- `net.ipv4.udp_mem`, this is a vector of three integers (min, pressure, max) governing the number
41
+ of pages allowed for queueing by all UDP sockets.
42
+ - min: Below this number of pages, UDP is not bothered about its memory appetite. When the
43
+ amount of memory allocated by UDP exceeds this number, UDP starts to moderate memory usage.
44
+ - pressure: This value was introduced to follow the format of tcp_mem (see tcp(7)).
45
+ - max: Defaults values for these three items are calculated at boot time from the amount of
46
+ available memory.
47
+
48
+- `net.ipv4.udp_wmem`, the minimal size (in bytes) of send buffer used by UDP sockets in moderation.
49
+ Each UDP socket is able to use the size for sending data, even if total pages of UDP sockets
50
+ exceed `udp_mem` pressure.
51
+
52
+In general, issues with buffers that allocated dynamically are correlated with the kernel memory,
53
+you must always be aware of memory pressure events. This can cause buffer errors.
54
+
55
+<details>
56
+<summary>References and sources</summary>
57
+
58
+1. [UDP definition on wikipedia](https://en.wikipedia.org/wiki/User_Datagram_Protocol)
59
+2. [Man page of UDP protocol](https://man7.org/linux/man-pages/man7/udp.7.html)
60
+3. [Redhat networking tuning guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings)
61
+
62
+</details>
63
+
64
+### Troubleshooting section:
65
+
66
+ <details>
67
+ <summary>Increase the net.core.wmem_default and net.core.wmem_max values</summary>
68
+
69
+1. Try to increase them, RedHat suggests the value of 262144
70
+ bytes <sup> [3](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings) </sup>
71
+
72
+ ```
73
+ sysctl -w net.core.wmem_default=262144
74
+ sysctl -w net.core.wmem_max=262144
75
+ ```
76
+
77
+1. Verify the change and test with the same workload that triggered the alarm originally.
78
+
79
+ ```
80
+ root@netdata~ # sysctl net.core.wmem_default net.core.wmem_max
81
+ net.core.wmem_default=262144
82
+ net.core.wmem_max=262144
83
+ ```
84
+
85
+1. If this change works for your system, you could make it permanently.
86
+
87
+ Bump these `net.core.wmem_default=262144` & `net.core.wmem_max=262144` entries under
88
+ `/etc/sysctl.conf`.
89
+
90
+1. Reload the sysctl settings.
91
+
92
+ ```
93
+ root@netdata~ # sysctl -p
94
+ ```
95
+
96
+</details>
health/guides/upsd/upsd_10min_ups_load.md
new
+26
@@ -0,0 +1,26 @@
1
+# nut_10min_ups_load
2
+
3
+**Power Supply | UPS**
4
+
5
+Network UPS Tools (NUT) is a suite of software component designed to monitor power devices, such as
6
+uninterruptible power supplies, power distribution units, solar controllers and servers power supply
7
+units.
8
+
9
+The Netdata Agent monitors the average UPS load over the last 10 minutes.
10
+
11
+<details>
12
+<summary>References and Sources</summary>
13
+
14
+1. [NUT user manual]https://networkupstools.org/docs/user-manual.chunked/index.html
15
+
16
+</details>
17
+
18
+### Troubleshooting section:
19
+
20
+<details>
21
+<summary>Reduce the load on the UPS</summary>
22
+
23
+To avoid ungraceful shutdowns of your systems, consider to reduce the load on this particular UPS.
24
+To achieve this, consider removing attached devices that are not mission critical.
25
+
26
+</details>
health/guides/upsd/upsd_last_collected_secs.md
new
+54
@@ -0,0 +1,54 @@
1
+# nut_last_collected_secs
2
+
3
+**Power Supply | UPS**
4
+
5
+Network UPS Tools (NUT) is a suite of software component designed to monitor power devices, such as
6
+uninterruptible power supplies, power distribution units, solar controllers and servers power supply
7
+units.
8
+
9
+The Netdata Agent monitors the number of seconds since the last successful data collection
10
+
11
+<details>
12
+<summary>References and Sources</summary>
13
+
14
+1. [NUT user manual]https://networkupstools.org/docs/user-manual.chunked/index.html
15
+
16
+</details>
17
+
18
+### Troubleshooting section:
19
+
20
+<details>
21
+<summary>Check the upsd server </summary>
22
+
23
+1. Check the status of the upsd daemon
24
+ ```
25
+ root@netdata $ systemctl status upsd
26
+ ```
27
+
28
+2. Check for obvious and common errors.
29
+
30
+
31
+3. Restart the daemon if needed
32
+ ```
33
+ root@netdata $ systemctl restart apcupsd
34
+ ```
35
+
36
+</details>
37
+
38
+<details>
39
+<summary>Diagnose a bad driver</summary>
40
+
41
+`upsd` expects the drivers to either update their status regularly or at least answer periodic
42
+queries, called pings. If a driver doesn’t answer, `upsd` will declare it "stale" and no more
43
+information will be provided to the clients.
44
+
45
+If upsd complains about staleness when you start it, then either your driver or configuration files
46
+are probably broken. Be sure that the driver is actually running, and that the UPS definition in
47
+[ups.conf(5)](https://networkupstools.org/docs/man/ups.conf.html) is correct. Also make sure that
48
+you start your driver(s) before starting upsd.
49
+
50
+Data can also be marked stale if the driver can no longer communicate with the UPS. In this case,
51
+the driver should also provide diagnostic information in the syslog. If this happens, check the
52
+serial or USB cabling, or inspect the network path in the case of a SNMP UPS.
53
+</details>
54
+
health/guides/upsd/upsd_ups_battery_charge.md
new
+26
@@ -0,0 +1,26 @@
1
+# nut_ups_charge
2
+
3
+**Power Supply | UPS**
4
+
5
+Network UPS Tools (NUT) is a suite of software component designed to monitor power devices, such as
6
+uninterruptible power supplies, power distribution units, solar controllers and servers power supply
7
+units.
8
+
9
+The Netdata Agent monitors the average UPS charge over the last minute.
10
+
11
+<details>
12
+<summary>References and Sources</summary>
13
+
14
+1. [NUT user manual]https://networkupstools.org/docs/user-manual.chunked/index.html
15
+
16
+</details>
17
+
18
+### Troubleshooting section:
19
+
20
+<details>
21
+<summary>Prepare your machine for graceful shutdown</summary>
22
+
23
+If you can't restore the power supply to this UPC, you should prepare your machine for graceful
24
+shutdown.
25
+
26
+</details>
health/guides/vcsa/vcsa_applmgmt_health.md
new
+29
@@ -0,0 +1,29 @@
1
+# vcsa_applmgmt_health
2
+
3
+## Virtual Machine | VMware vCenter
4
+
5
+
6
+This alert presents the appliance management component health status.
7
+The values for every component's health can be:
8
+
9
+| Code | Color | Description | Alert Status |
10
+|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
11
+| `-1` | no color | Unknown. | Clear |
12
+| `0` |  | The component is healthy. | Clear |
13
+| `1` |  | The component is healthy but may have some problems. | Warning |
14
+| `2` |  | The component is degraded, and may have serious problems. | Critical |
15
+| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
16
+| `4` |  | No health data is available. | Clear |
17
+
18
+For more information, please see the [VMware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html).
19
+
20
+### Troubleshooting Section
21
+
22
+To find out why the alert was raised, follow the steps in
23
+the [VMware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html).
24
+
25
+<details><summary>References and Sources</summary>
26
+
27
+[VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
28
+
29
+</details>
health/guides/vcsa/vcsa_database_storage_health.md
new
+30
@@ -0,0 +1,30 @@
1
+# vcsa_database_storage_health
2
+
3
+## Virtual Machine | VMware vCenter
4
+
5
+
6
+This alert presents the database storage component health status.
7
+The values for every component's health can be:
8
+
9
+| Code | Color | Description | Alert Status |
10
+|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
11
+| `-1` | no color | Unknown. | Clear |
12
+| `0` |  | The component is healthy. | Clear |
13
+| `1` |  | The component is healthy but may have some problems. | Warning |
14
+| `2` |  | The component is degraded, and may have serious problems. | Critical |
15
+| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
16
+| `4` |  | No health data is available. | Clear |
17
+
18
+For further information, please have a look at the *References and Sources* section.
19
+
20
+<details><summary>References and Sources</summary>
21
+
22
+1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
23
+
24
+</details>
25
+
26
+### Troubleshooting Section
27
+
28
+To find out why the alert was raised, follow the steps in
29
+the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
30
+.
health/guides/vcsa/vcsa_load_health.md
new
+29
@@ -0,0 +1,29 @@
1
+# vcsa_load_health
2
+
3
+## Virtual Machine | VMware vCenter
4
+
5
+This alert presents the load health status.
6
+The values for every component's health can be:
7
+
8
+| Code | Color | Description | Alert Status |
9
+|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
10
+| `-1` | no color | Unknown. | Clear |
11
+| `0` |  | The component is healthy. | Clear |
12
+| `1` |  | The component is healthy but may have some problems. | Warning |
13
+| `2` |  | The component is degraded, and may have serious problems. | Critical |
14
+| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
15
+| `4` |  | No health data is available. | Clear |
16
+
17
+For further information, please have a look at the *References and Sources* section.
18
+
19
+<details><summary>References and Sources</summary>
20
+
21
+1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
22
+
23
+</details>
24
+
25
+### Troubleshooting Section
26
+
27
+To find out why the alert was raised, follow the steps in
28
+the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
29
+.
\ No newline at end of file
health/guides/vcsa/vcsa_mem_health.md
new
+29
@@ -0,0 +1,29 @@
1
+# vcsa_mem_health
2
+
3
+## Virtual Machine | VMware vCenter
4
+
5
+This alert presents the memory health status.
6
+The values for every component's health can be:
7
+
8
+| Code | Color | Description | Alert Status |
9
+|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
10
+| `-1` | no color | Unknown. | Clear |
11
+| `0` |  | The component is healthy. | Clear |
12
+| `1` |  | The component is healthy but may have some problems. | Warning |
13
+| `2` |  | The component is degraded, and may have serious problems. | Critical |
14
+| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
15
+| `4` |  | No health data is available. | Clear |
16
+
17
+For further information, please have a look at the *References and Sources* section.
18
+
19
+<details><summary>References and Sources</summary>
20
+
21
+1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
22
+
23
+</details>
24
+
25
+### Troubleshooting Section
26
+
27
+To find out why the alert was raised, follow the steps in
28
+the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
29
+.
\ No newline at end of file
health/guides/vcsa/vcsa_software_updates_health.md
new
+31
@@ -0,0 +1,31 @@
1
+# vcsa_software_updates_health
2
+
3
+## Virtual Machine | VMware vCenter
4
+
5
+This alert presents the software updates availability status.
6
+The values can be:
7
+
8
+| Code | Color | Description | Alert Status |
9
+|:----:|:---------------------------------------------------------------:|:-----------------------------------------------------|:------------:|
10
+| `-1` | no color | Unknown. | Clear |
11
+| `0` |  | no updates available. | Clear |
12
+| `2` |  | non-security updates are available. | Clear |
13
+| `3` |  | security updates are available. | Critical |
14
+| `4` |  | an error retrieving information on software updates. | Warning |
15
+
16
+For further information, please have a look at the *References and Sources* section.
17
+
18
+<details><summary>References and Sources</summary>
19
+
20
+1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
21
+
22
+</details>
23
+
24
+
25
+### Troubleshooting Section
26
+
27
+If the alert was raised into critical, proceed by installing the security updates that are
28
+available. If the alert was raised into warning, consider viewing the details in the Health Messages
29
+pane.
30
+
31
+You can also find more details in the [VMware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html).
health/guides/vcsa/vcsa_storage_health.md
new
+29
@@ -0,0 +1,29 @@
1
+# vcsa_storage_health
2
+
3
+## Virtual Machine | VMware vCenter
4
+
5
+This alert presents the storage health status.
6
+The values for every component's health can be:
7
+
8
+| Code | Color | Description | Alert Status |
9
+|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:-------------:|
10
+| `-1` | no color | Unknown. | Clear |
11
+| `0` |  | The component is healthy. | Clear |
12
+| `1` |  | The component is healthy but may have some problems. | Warning |
13
+| `2` |  | The component is degraded, and may have serious problems. | Critical |
14
+| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
15
+| `4` |  | No health data is available. | Clear |
16
+
17
+For further information, please have a look at the *References and Sources* section.
18
+
19
+<details><summary>References and Sources</summary>
20
+
21
+1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
22
+
23
+</details>
24
+
25
+### Troubleshooting Section
26
+
27
+To find out why the alert was raised, follow the steps in
28
+the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
29
+.
health/guides/vcsa/vcsa_swap_health.md
new
+29
@@ -0,0 +1,29 @@
1
+# vcsa_swap_health
2
+
3
+## Virtual Machine | VMware vCenter
4
+
5
+This alert presents the swap health status.
6
+The values for every component's health can be:
7
+
8
+
9
+| Code | Color | Description | Alert Status |
10
+|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
11
+| `-1` | no color | Unknown. | Clear |
12
+| `0` |  | The component is healthy. | Clear |
13
+| `1` |  | The component is healthy but may have some problems. | Warning |
14
+| `2` |  | The component is degraded, and may have serious problems. | Critical |
15
+| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
16
+| `4` |  | No health data is available. | Clear |
17
+
18
+For further information, please have a look at the *References and Sources* section.
19
+
20
+<details><summary>References and Sources</summary>
21
+
22
+1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
23
+
24
+</details>
25
+
26
+
27
+### Troubleshooting Section
28
+
29
+To find out why the alert was raised, follow the steps in the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html).
health/guides/vcsa/vcsa_system_health.md
new
+34
@@ -0,0 +1,34 @@
1
+# vcsa_system_health
2
+
3
+## Virtual Machine | VMware vCenter
4
+
5
+This alert presents the overall system health status.
6
+It can take the values:
7
+
8
+- -1: unknown (no color)
9
+- 0: all components are healthy. (green)
10
+- 1: one or more components might become overloaded soon. (yellow)
11
+- 2: one or more components in the appliance might be degraded. (orange)
12
+- 3: one or more components might be in an unusable status and the appliance might become unresponsive soon. (red)
13
+- 4: no health data is available. (grey)
14
+
15
+
16
+If you receive this alert, it means that the overall system status is unhealthy. One or more
17
+components might become overloaded soon (yellow), or might be degraded (orange), or
18
+might be in an unusable status and the appliance might become unresponsive soon (red).
19
+
20
+This alert is raised into warning if the status has a code of 1 or 2.
21
+If the metric reaches a value of 3, the alert is raised into critical.
22
+
23
+For further information, please have a look at the *References and Sources* section.
24
+
25
+<details><summary>References and Sources</summary>
26
+
27
+1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
28
+
29
+</details>
30
+
31
+
32
+### Troubleshooting Section
33
+
34
+To troubleshoot the issue, you need to log into vCenter Server Management Interface and follow the information in the [vmware documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html).
health/guides/vernemq/vernemq_average_scheduler_utilization.md
new
+43
@@ -0,0 +1,43 @@
1
+# vernemq_average_scheduler_utilization
2
+
3
+**Messaging | VerneMQ**
4
+
5
+VerneMQ is implemented in Erlang and therefore runs on top of the BEAM runtime environment (roughly equivalent to
6
+the JRE for Java applications).
7
+For performance reasons, BEAM utilizes it’s own intenral scheduler that operates largely independently of the operating
8
+system’s process scheduling.
9
+
10
+The Netdata Agent calculates the average VerneMQ's scheduler utilization over the last 10 minutes.
11
+This alert indicates high scheduler utilization.
12
+
13
+This alert is raised into warning when the scheduler's utilization is between 75-85% and in critical
14
+when it is between 85-95%.
15
+
16
+### Troubleshooting section:
17
+
18
+<details>
19
+<summary>Check for CPU throttling issues </summary>
20
+
21
+If you are receiving this alert often, it means that your node is running at maximum CPU utilization.
22
+You should consider upgrading your system (instance in your cloud) to provide more or faster CPUs.
23
+
24
+**Important**:
25
+
26
+By default, the VerneMQ broker deploys its Erlang VM architecture into 4 cores. If you already
27
+run VerneMQ in a multicore machine (for example, an 8-core machine) you should consider changing
28
+the `vmq_bcrypt.nif_pool_size` parameter:
29
+
30
+1. In the `vernemq.conf`, update the `vmq_bcrypt.nif_pool_size` parameter to `auto`. The value `auto`
31
+detect all cores (n) and set the value to n-1.
32
+
33
+
34
+2. Restart the VerneMQ service.
35
+
36
+ ```
37
+ root@netdata # systemctl restart vernemq.service
38
+ ```
39
+
40
+3. Open the Netdata dashboard, and locate the `scheduler_utilization` chart. See if VerneMQ utilizes
41
+ the preferred number of cores.
42
+
43
+</details>
\ No newline at end of file
health/guides/vernemq/vernemq_cluster_dropped.md
new
+51
@@ -0,0 +1,51 @@
1
+# vernemq_cluster_dropped
2
+
3
+**Messaging | VerneMQ**
4
+
5
+VerneMQ is a MQTT publish/subscribe message broker which implements the OASIS industry standard MQTT
6
+protocol.
7
+
8
+The Netdata agent calculates the amount of traffic dropped during communication with the cluster
9
+nodes in the last minute. This alert indicates that the outgoing cluster buffer is full.
10
+
11
+Receiving this alert most likely means that a remote node is down or unreachable, but it could also
12
+indicate that the VerneMQ is experiencing problems with inter-node message delivery. The
13
+non-dispatched messages are queued in this buffer.
14
+
15
+### Troubleshooting section:
16
+
17
+<details>
18
+<summary>Increase the cluster buffer </summary>
19
+
20
+To make your cluster more tolerant to disconnections of nodes, you can increase the size of the
21
+`outgoing_clustering_buffer_size` buffer.
22
+
23
+1. Edit the VerneMQ configuration file. By default it is located under `/etc/vernemq` folder.
24
+
25
+ ```
26
+ root@netdata # vim /etc/vernemq/vernemq.conf
27
+ ```
28
+
29
+2. Append the `outgoing_clustering_buffer_size` value, the default value is 10000 bytes. Try to
30
+ increase it to 15000
31
+
32
+ ```
33
+ # vim /etc/vernemq/vernemq.conf
34
+ . . .
35
+ outgoing_clustering_buffer_size = 15000
36
+ . . .
37
+ ```
38
+
39
+3. Restart the VerneMQ service
40
+
41
+ ```
42
+ root@netdata # systemctl restart vernemq.service
43
+ ```
44
+
45
+4. Test with the same workload that triggered the alarm originally. If this alert still occurs, try
46
+ to double this value and re-test.
47
+
48
+5. In case the problem still exists, you must check for issues in the nodes that are unavailable.
49
+
50
+</details>
51
+
health/guides/vernemq/vernemq_mqtt_connack_sent_reason_unsuccessful.md
new
+50
@@ -0,0 +1,50 @@
1
+# vernemq_mqtt_connack_sent_reason_unsuccessful
2
+
3
+**Messaging | VerneMQ**
4
+
5
+In the MQTT protocol, the CONNACK packet is the packet sent by the server in response to a CONNECT
6
+attempt from a client. The first packet sent from the server to the client must be a CONNACK packet.
7
+If the client does not receive a CONNACK packet from the server within a reasonable amount of time,
8
+the client should close the Network Connection. The "reasonable" amount of time depends on the type
9
+of application and the communications infrastructure.
10
+
11
+For various scenarios, there are specific CONNACK responses for both MQTT v3 and v5 . You can find
12
+the detailed response codes and descriptions for each protocol in the official documentation of
13
+MQTT,
14
+for [v3 (subsection 3.2.2.3 Connect Return code)](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718035)
15
+and
16
+for [v5 (subsection 3.2.2.2 Connect Reason Code)](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901074)
17
+
18
+MQTT v5 supports a wider variety of negative acknowledgements (unsuccessful CONNACK packets ), which
19
+makes it a lot easier for both the client and the VerneMQ admin to understand what's happening.
20
+
21
+The Netdata Agent monitors the number of sent unsuccessful v3/v5 CONNACK packets over the last
22
+minute. This alert is raised into warning when your VerneMQ server sends more than 5 unsuccessful
23
+packets in the last minute.
24
+
25
+<details>
26
+<summary>References and Sources</summary>
27
+
28
+1. [MQTT v3 docs, CONNACK description](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718033)
29
+2. [MQTT v5 docs, CONNACK description](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901074)
30
+
31
+</details>
32
+
33
+### Troubleshooting Section
34
+
35
+<details>
36
+<summary>General approach</summary>
37
+
38
+Open the alerts Dashboard, and locate the chart of this alert (`mqtt_connack_sent_reason`). Inspect
39
+which CONNACK packets (by reason) triggered this alert. As soon as you inspect the reason (by
40
+consulting the subsections: _connect reason code_ which we mentioned above for your protocol ), you
41
+will have to examine
42
+your [logs](https://docs.vernemq.com/configuring-vernemq/logging#console-logging) to check which
43
+client(s) are raising these issues. These kinds of issues appear in the warning log level, so you
44
+may have to set your log level appropriately.
45
+
46
+```
47
+root@netdata # cat /var/log/vernemq/console.log | grep "due to <keywords: error> "
48
+```
49
+
50
+</details>
health/guides/vernemq/vernemq_mqtt_disconnect_received_reason_not_normal.md
new
+48
@@ -0,0 +1,48 @@
1
+# vernemq_mqtt_disconnect_received_reason_not_normal
2
+
3
+**Messaging | VerneMQ**
4
+
5
+_The DISCONNECT packet is the final MQTT Control Packet sent from the Client or the Server. It
6
+indicates the reason why the Network Connection is being closed. The Client or Server may send a
7
+DISCONNECT packet before closing the Network Connection. If the Network Connection is closed without
8
+the Client first sending a DISCONNECT packet with Reason Code 0x00 (Normal disconnection) and the
9
+Connection has a Will Message, the Will Message is
10
+published. <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205) </sup>_
11
+
12
+The Netadata agent monitors the number of received not normal v5 DISCONNECT packets over the last
13
+minute. This alert is raised into warning when your VerneMQ server receive more than 5 DISCONNECT
14
+packets over the last minute.
15
+
16
+For various scenarios, there are specific DISCONNECT responses for MQTT protocol v5. You can find
17
+the detailed response codes which were sent by a client and their descriptions in the official
18
+documentation of MQTT in
19
+the [MQTT v5 docs, subsection 3.14.2.1: Disconnect reason code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205)
20
+
21
+<details>
22
+<summary>References and sources</summary>
23
+
24
+1. [MQTT v5 docs DISCONNECT notification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205)
25
+
26
+</details>
27
+
28
+### Troubleshooting Section
29
+
30
+<details>
31
+<summary>General approach</summary>
32
+
33
+Open the alerts Dashboard and locate the chart of this alert (`mqtt_disconnect_received_reason`).
34
+Inspect which DISCONNECT packets (by reason) triggered this alert. You can clarify why your server
35
+received those responses from a client by consulting the subsection _Disconnect reason
36
+code <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205) </sup>_
37
+which we mentioned above.
38
+
39
+For example: your server may receive some DISCONNECT packets with the reason: "Disconnect with Will
40
+Message." This is not abnormal except in the case in which the network connection is closed
41
+abruptly. This may indicate problems in the connectivity with your clients.
42
+
43
+</details>
44
+
45
+
46
+
47
+
48
+
health/guides/vernemq/vernemq_mqtt_disconnect_sent_reason_not_normal.md
new
+43
@@ -0,0 +1,43 @@
1
+# vernemq_mqtt_disconnect_sent_reason_not_normal
2
+
3
+**Messaging | VerneMQ**
4
+
5
+_The DISCONNECT packet is the final MQTT Control Packet sent from the Client or the Server. It
6
+indicates the reason why the Network Connection is being closed. The Client or Server may send a
7
+DISCONNECT packet before closing the Network
8
+Connection. <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205) </sup>_
9
+
10
+The Netdata Agent monitors the number of sent _not normal_ v5 DISCONNECT packets over the last
11
+minute. This alert is raised into warning when your VerneMQ server sends more than 5 DISCONNECT
12
+packets over the last minute.
13
+
14
+For various scenarios, there are specific DISCONNECT responses for MQTT protocol v5. You can find
15
+the detailed response codes which were sent by the Server and their descriptions in the official
16
+documentation of MQTT in
17
+the [MQTT v5 docs, subsection 3.14.2.1: Disconnect reason code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205)
18
+
19
+<details>
20
+<summary>References and sources</summary>
21
+
22
+1. [MQTT v5 docs DISCONNECT notification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205)
23
+
24
+</details>
25
+
26
+### Troubleshooting Section
27
+
28
+<details>
29
+<summary>General approach</summary>
30
+
31
+Open the alerts Dashboard and locate the chart of this alert (`mqtt_disconnect_sent_reason`).
32
+Inspect which DISCONNECT packets (by reason) triggered this alert. Inspect the reason why your
33
+server sent those responses by consulting the subsection _Disconnect reason
34
+code <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205) </sup>_
35
+mentioned above.
36
+
37
+For example, your server may respond to a client with `QoS not supported`. In that case, the client must
38
+change the QoS settings.
39
+
40
+</details>
41
+
42
+
43
+
health/guides/vernemq/vernemq_mqtt_puback_received_reason_unsuccessful.md
new
+46
@@ -0,0 +1,46 @@
1
+# vernemq_mqtt_puback_received_reason_unsuccessful
2
+
3
+**Messaging | VerneMQ**
4
+
5
+A PUBACK packet is the response to a PUBLISH packet with QoS 1. The Netdata Agent monitors the
6
+number of received unsuccessful v5 PUBACK packets in the last minute. For various scenarios, there
7
+are specific PUBACK responses for MQTT protocol v5. You can find the detailed response codes which
8
+were sent by a client or a server and their descriptions in the official documentation of MQTT in
9
+the [MQTT v5 docs, PUBACK Reason Code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)
10
+. The Client or Server sending the PUBACK packet must always use one of the PUBACK Reason Codes.
11
+
12
+
13
+
14
+<details>
15
+<summary>See more about QoS 1 </summary>
16
+
17
+The Quality of Service (QoS) level is an agreement between the sender of a message and the receiver
18
+of a message that defines the guarantee of delivery for a specific message. In QoS 1, a client will
19
+receive a confirmation message from the broker upon receipt. If the expected confirmation is not
20
+received within a certain time frame, the client has to retry the message. A message received by a
21
+client must be acknowledged on time as well, otherwise the broker will re-deliver the
22
+message. <sup>[1](https://vernemq.com/intro/mqtt-primer/quality_of_service.html) </sup>
23
+
24
+</details>
25
+
26
+<details>
27
+<summary>References and sources</summary>
28
+
29
+1. [Quality of service explained, VerneMQ docs](https://vernemq.com/intro/mqtt-primer/quality_of_service.html)
30
+2. [MQTT v5 docs, PUBACK description](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
31
+
32
+</details>
33
+
34
+### Troubleshooting Section
35
+
36
+<details>
37
+<summary>General approach</summary>
38
+
39
+Open the alerts Dashboard and locate the chart of this alert (`mqtt_puback_received_reason`).
40
+Identify which PUBACK packets (by reason) triggered this alert. Inspect the reason why a client sent
41
+those responses by consulting the subsection _PUBACK Reason
42
+Code <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)_ </sup>
43
+mentioned above.
44
+
45
+</details>
46
+
health/guides/vernemq/vernemq_mqtt_puback_sent_reason_unsuccessful.md
new
+43
@@ -0,0 +1,43 @@
1
+# vernemq_mqtt_puback_sent_reason_unsuccessful
2
+
3
+**Messaging | VerneMQ**
4
+
5
+A PUBACK packet is the response to a PUBLISH packet with QoS 1. The Netdata Agent monitors the
6
+number of sent unsuccessful v5 PUBACK packets in the last minute. For various scenarios, there are
7
+specific PUBACK responses for MQTT protocol v5. You can find the detailed response codes which were
8
+sent by a client or a server and their descriptions in the official documentation of MQTT in
9
+the [MQTT v5 docs, PUBACK Reason Code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)
10
+. The Client or Server sending the PUBACK packet must always use one of the PUBACK Reason Codes.
11
+
12
+<details>
13
+<summary>See more about QoS 1 </summary>
14
+
15
+The Quality of Service (QoS) level is an agreement between the sender of a message and the receiver
16
+of a message that defines the guarantee of delivery for a specific message. In QoS 1, a client will
17
+receive a confirmation message from the broker upon receipt. If the expected confirmation is not
18
+received within a certain time frame, the client has to retry the message. A message received by a
19
+client must be acknowledged on time as well, otherwise the broker will re-deliver the
20
+message. <sup>[1](https://vernemq.com/intro/mqtt-primer/quality_of_service.html) </sup>
21
+
22
+</details>
23
+
24
+<details>
25
+<summary>References and sources</summary>
26
+
27
+1. [Quality of service explained, VerneMQ docs](https://vernemq.com/intro/mqtt-primer/quality_of_service.html)
28
+2. [MQTT v5 docs, PUBACK description](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
29
+
30
+</details>
31
+
32
+### Troubleshooting Section
33
+
34
+<details>
35
+<summary>General approach</summary>
36
+
37
+Open the alerts Dashboard and locate the chart of this alert (`mqtt_puback_sent_reason`). Identify
38
+which PUBACK packets (by reason) triggered this alert. Inspect the reason why you server sent those
39
+responses by consulting the subsection _PUBACK Reason
40
+Code <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)_ </sup>
41
+mentioned above.
42
+
43
+</details>
health/guides/vernemq/vernemq_mqtt_puback_unexpected.md
new
+51
@@ -0,0 +1,51 @@
1
+# vernemq_mqtt_puback_unexpected
2
+
3
+**Messaging | VerneMQ**
4
+
5
+A PUBACK packet is the response to a PUBLISH packet with QoS 1. The Netdata Agent monitors the
6
+number of received unexpected v3/v5 PUBACK packets in the last minute.
7
+
8
+MQTT v5 protocol provides detailed PUBACK reasons codes as opposed to MQTT v3. You can find the
9
+detailed response codes which were sent by a client or a server and their descriptions in the
10
+official documentation of MQTT in
11
+the [MQTT v5 docs, PUBACK Reason Code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)
12
+. The Client or Server sending the PUBACK packet must always use one of the PUBACK Reason Codes.
13
+
14
+
15
+<details>
16
+<summary>See more about QoS 1 </summary>
17
+
18
+The Quality of Service (QoS) level is an agreement between the sender of a message and the receiver
19
+of a message that defines the guarantee of delivery for a specific message. In QoS 1, a client will
20
+receive a confirmation message from the broker upon receipt. If the expected confirmation is not
21
+received within a certain time frame, the client has to retry the message. A message received by a
22
+client must be acknowledged on time as well, otherwise the broker will re-deliver the
23
+message. <sup>[1](https://vernemq.com/intro/mqtt-primer/quality_of_service.html) </sup>
24
+
25
+</details>
26
+
27
+
28
+<details>
29
+<summary>References and sources</summary>
30
+
31
+1. [Quality of service explained, VerneMQ docs](https://vernemq.com/intro/mqtt-primer/quality_of_service.html)
32
+2. [MQTT v5 docs, PUBACK description](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
33
+3. [MQTT v3 docs, PUBACK description](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718043)
34
+
35
+</details>
36
+
37
+### Troubleshooting Section
38
+
39
+<details>
40
+<summary>General approach</summary>
41
+
42
+This alert monitors the PUBACK packets for both v3 and v5 MQTT protocol. In case you didn't receive
43
+any other alerts (`vernemq_mqtt_puback_received_reason_unsuccessful`
44
+, `vernemq_mqtt_puback_sent_unsuccessful`) (in which you can consult their troubleshooting
45
+sections), that means that the unexpected PUBACK packets was came(sent) from(to) clients which are
46
+using the MQTT v3 protocol. In that case you can inspect your MQTT server access log for further
47
+investigation.
48
+
49
+
50
+</details>
51
+
health/guides/vernemq/vernemq_mqtt_pubcomp_received_reason_unsuccessful.md
new
+76
@@ -0,0 +1,76 @@
1
+# vernemq_mqtt_pubcomp_received_reason_unsuccessful
2
+
3
+**Messaging | VerneMQ**
4
+
5
+The PUBCOMP packet is the response to a PUBREL packet. It is the fourth and final packet of the QoS
6
+2 protocol exchange.
7
+
8
+The Netdata Agent monitors the number of received unsuccessful v5 PUBCOMP packets in the last
9
+minute.
10
+
11
+For various scenarios, there are specific DISCONNECT responses for MQTT protocol v5. You can find
12
+Message. This is not something abnormal.
13
+
14
+
15
+<details>
16
+<summary>MQTT basic concepts and more</summary>
17
+
18
+Basic concepts in every MQTT
19
+architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
20
+
21
+- _Broker_ - The broker is the server that distributes the information to the interested clients
22
+ connected to the server.
23
+- _Client_ - The device that connects to broker to send or receive information.
24
+- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
25
+- _Publish_ - Clients that send information to the broker to distribute to interested clients based
26
+ on the topic name.
27
+- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
28
+ subscribes to a topic, any message published to the broker is distributed to the subscribers of
29
+ that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
30
+ topic.
31
+- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
32
+ integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
33
+ transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
34
+ QoS 0.
35
+
36
+ - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
37
+ delivery. This is often referred to as fire and forget.
38
+ - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
39
+ received, known otherwise as acknowledged delivery.
40
+ - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
41
+ ensure only one copy of the message is received, known as assured delivery.
42
+
43
+- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
44
+ communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
45
+ of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
46
+ WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
47
+ documentation in
48
+ the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
49
+ section
50
+
51
+</details>
52
+
53
+<details>
54
+<summary>References and sources</summary>
55
+
56
+1. [Introduction to MQTT](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
57
+2. [MQTT v5 docs, PUBCOMP reason codes](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901154)
58
+
59
+</details>
60
+
61
+### Troubleshooting Section
62
+
63
+<details>
64
+<summary>General approach</summary>
65
+
66
+Open the alerts Dashboard and locate the chart of this alert (`mqtt_pubcomp_received_reason`). Inspect
67
+which PUBCOMP packets (by reason) triggered this alert. Inspect the reason why your server received
68
+those responses by consulting the subsection _OPERATION REASON
69
+CODE_ <sup>[2](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901154) </sup>
70
+which we mentioned above.
71
+
72
+For example, your server may respond to a client `QoS: packet identifier not found`, This is not an
73
+error during recovery, but at other times indicates a mismatch between the Session State on the
74
+Client and Server.
75
+
76
+</details>
health/guides/vernemq/vernemq_mqtt_pubcomp_sent_reason_unsuccessful.md
new
+67
@@ -0,0 +1,67 @@
1
+# vernemq_mqtt_pubcomp_sent_reason_unsuccessful
2
+
3
+**Messaging | VerneMQ**
4
+
5
+The PUBCOMP packet is the response to a PUBREL packet. It is the fourth and final packet of the QoS
6
+2 protocol exchange. The Netdata Agent monitors the number of sent unsuccessful v5 PUBCOMP packets
7
+in the last minute.
8
+
9
+<details>
10
+<summary>MQTT basic concepts and more</summary>
11
+
12
+Basic concepts in every MQTT
13
+architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
14
+
15
+- _Broker_ - The broker is the server that distributes the information to the interested clients
16
+ connected to the server.
17
+- _Client_ - The device that connects to broker to send or receive information.
18
+- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
19
+- _Publish_ - Clients that send information to the broker to distribute to interested clients based
20
+ on the topic name.
21
+- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
22
+ subscribes to a topic, any message published to the broker is distributed to the subscribers of
23
+ that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
24
+ topic.
25
+- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
26
+ integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
27
+ transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
28
+ QoS 0.
29
+
30
+ - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
31
+ delivery. This is often referred to as fire and forget.
32
+ - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
33
+ received, known otherwise as acknowledged delivery.
34
+ - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
35
+ ensure only one copy of the message is received, known as assured delivery.
36
+
37
+- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
38
+ communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
39
+ of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
40
+ WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
41
+ documentation in
42
+ the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
43
+ section
44
+
45
+</details>
46
+
47
+<details>
48
+<summary>References and sources</summary>
49
+
50
+1. [Introduction to MQTT](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
51
+2. [MQTT v5 docs, PUBCOMP reason codes](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901154)
52
+
53
+</details>
54
+
55
+### Troubleshooting Section
56
+
57
+<details>
58
+<summary>General approach</summary>
59
+
60
+Open the alerts Dashboard and locate the chart of this alert (`mqtt_pubcomp_sent_reason`). Inspect
61
+which PUBCOMP packets (by reason) triggered this alert. Inspect the reason why your server sent
62
+those responses by consulting the subsection _OPERATION REASON
63
+CODE_ <sup>[2](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901154) </sup>
64
+which we mentioned above.
65
+
66
+</details>
67
+
health/guides/vernemq/vernemq_mqtt_pubcomp_unexpected.md
new
+70
@@ -0,0 +1,70 @@
1
+# vernemq_mqtt_pubcomp_unexpected
2
+
3
+**Messaging | VerneMQ**
4
+
5
+The PUBCOMP packet is the response to a PUBREL packet. It is the fourth and final packet of the QoS
6
+2 protocol exchange. The Netdata Agent monitors the number of received unexpected v3/v5 PUBCOMP
7
+packets in the last minute.
8
+
9
+<details>
10
+<summary>MQTT basic concepts and more</summary>
11
+
12
+Basic concepts in every MQTT
13
+architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
14
+
15
+- _Broker_ - The broker is the server that distributes the information to the interested clients
16
+ connected to the server.
17
+- _Client_ - The device that connects to broker to send or receive information.
18
+- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
19
+- _Publish_ - Clients that send information to the broker to distribute to interested clients based
20
+ on the topic name.
21
+- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
22
+ subscribes to a topic, any message published to the broker is distributed to the subscribers of
23
+ that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
24
+ topic.
25
+- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
26
+ integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
27
+ transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
28
+ QoS 0.
29
+
30
+ - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
31
+ delivery. This is often referred to as fire and forget.
32
+ - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
33
+ received, known otherwise as acknowledged delivery.
34
+ - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
35
+ ensure only one copy of the message is received, known as assured delivery.
36
+
37
+- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
38
+ communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
39
+ of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
40
+ WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
41
+ documentation in
42
+ the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
43
+ section
44
+
45
+</details>
46
+
47
+<details>
48
+<summary>References and sources</summary>
49
+
50
+1. [Introduction to MQTT](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
51
+2. [MQTT v5 docs, PUBCOMP packets](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901151)
52
+3. [MQTT v3 docs, PUBCOMP packets](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718058)
53
+
54
+</details>
55
+
56
+### Troubleshooting Section
57
+
58
+<details>
59
+<summary>General approach</summary>
60
+
61
+This alert monitors the PUBCOMP packets for both v3 and v5 MQTT protocol. In case you didn't receive
62
+any other alerts (`vernemq_mqtt_pubcomp_received_reason_unsuccessful`
63
+, `vernemq_mqtt_pubcomp_sent_unsuccessful`) (in which you can consult their troubleshooting
64
+sections), that means that the unexpected PUBCOMP packets was received(sent) from(to) clients which
65
+are using the MQTT v3 protocol. In that case you can inspect your MQTT server access log for further
66
+investigation.
67
+
68
+
69
+</details>
70
+
health/guides/vernemq/vernemq_mqtt_publish_auth_errors.md
new
+72
@@ -0,0 +1,72 @@
1
+# vernemq_mqtt_publish_auth_errors
2
+
3
+**Messaging | VerneMQ**
4
+
5
+A PUBLISH Control Packet is sent from a Client to a Server or from Server to a Client to transport
6
+an Application Message. The Netdata Agent monitors the number of unauthorized v3/v5 PUBLISH attempts
7
+in the last minute.
8
+
9
+<details>
10
+<summary>MQTT basic concepts and more</summary>
11
+
12
+Basic concepts in every MQTT
13
+architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
14
+
15
+- _Broker_ - The broker is the server that distributes the information to the interested clients
16
+ connected to the server.
17
+- _Client_ - The device that connects to broker to send or receive information.
18
+- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
19
+- _Publish_ - Clients that send information to the broker to distribute to interested clients based
20
+ on the topic name.
21
+- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
22
+ subscribes to a topic, any message published to the broker is distributed to the subscribers of
23
+ that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
24
+ topic.
25
+- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
26
+ integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
27
+ transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
28
+ QoS 0.
29
+
30
+ - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
31
+ delivery. This is often refered to as fire and forget.
32
+ - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
33
+ received, known otherwise as acknowledged delivery.
34
+ - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
35
+ ensure only one copy of the message is received, known as assured delivery.
36
+
37
+- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
38
+ communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
39
+ of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
40
+ WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
41
+ documentation in
42
+ the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
43
+ section
44
+
45
+</details>
46
+
47
+<details>
48
+<summary>References and sources</summary>
49
+
50
+1. [MQTT basic concepts](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
51
+2. [MQTT v5 docs PUBLISH packets](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
52
+
53
+</details>
54
+
55
+### Troubleshooting Section
56
+
57
+<details>
58
+<summary>General approach </summary>
59
+
60
+You may encounter authorization errors either from a misconfigured MQTT client or from a server side
61
+error (for example a misconfigured web hook.)
62
+
63
+In any case you should inspect
64
+your [logs](https://docs.vernemq.com/configuring-vernemq/logging#console-logging) to check with which
65
+client(s) these issues are raised. These kinds of issues appear in the warning log level,
66
+so you may have to set your log level appropriately.
67
+
68
+```
69
+root@netdata # cat /var/log/vernemq/console.log | grep "auth_on_publish"
70
+```
71
+
72
+</details>
\ No newline at end of file
health/guides/vernemq/vernemq_mqtt_publish_errors.md
new
+72
@@ -0,0 +1,72 @@
1
+# vernemq_mqtt_publish_errors
2
+
3
+**Messaging | VerneMQ**
4
+
5
+A PUBLISH Control Packet is sent from a Client to a Server or from Server to a Client to transport
6
+an Application Message. The Netdata Agent monitors the number of failed v3/v5 PUBLISH operations in
7
+the last minute.
8
+
9
+<details>
10
+<summary>MQTT basic concepts and more</summary>
11
+
12
+Basic concepts in every MQTT
13
+architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
14
+
15
+- _Broker_ - The broker is the server that distributes the information to the interested clients
16
+ connected to the server.
17
+- _Client_ - The device that connects to broker to send or receive information.
18
+- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
19
+- _Publish_ - Clients that send information to the broker to distribute to interested clients based
20
+ on the topic name.
21
+- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
22
+ subscribes to a topic, any message published to the broker is distributed to the subscribers of
23
+ that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
24
+ topic.
25
+- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
26
+ integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
27
+ transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
28
+ QoS 0.
29
+
30
+ - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
31
+ delivery. This is often refered to as fire and forget.
32
+ - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
33
+ received, known otherwise as acknowledged delivery.
34
+ - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
35
+ ensure only one copy of the message is received, known as assured delivery.
36
+
37
+- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
38
+ communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
39
+ of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
40
+ WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
41
+ documentation in
42
+ the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
43
+ section
44
+
45
+</details>
46
+
47
+<details>
48
+<summary>References and sources</summary>
49
+
50
+1. [MQTT basic concepts](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
51
+2. [MQTT v5 docs PUBLISH packets](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
52
+3. [Dealing with netsplits](https://docs.vernemq.com/vernemq-clustering/netsplits)
53
+
54
+</details>
55
+
56
+### Troubleshooting Section
57
+
58
+These kinds of errors can appear in cases of a network partition in your cluster (aka netsplit)
59
+
60
+<details>
61
+<summary>Check connectivity between nodes</summary>
62
+
63
+You must ensure that the connectivity between your cluster nodes is valid. As soon as the partition
64
+is healed, and connectivity reestablished, the VerneMQ nodes replicate the latest changes made to
65
+the subscription data. This includes all the changes 'accidentally' made during the Window of
66
+Uncertainty. Using Dotted Version Vectors VerneMQ ensures that convergence regarding subscription
67
+data and retained messages is eventually reached.
68
+
69
+</details>
70
+
71
+
72
+
health/guides/vernemq/vernemq_mqtt_subscribe_auth_error.md
new
+80
@@ -0,0 +1,80 @@
1
+# vernemq_mqtt_subscribe_auth_error
2
+
3
+**Messaging | VerneMQ**
4
+
5
+The SUBSCRIBE packet is sent from the client to the server to create one or more subscriptions. Each
6
+subscription registers a client’s interest in one or more topics. The server sends PUBLISH packets
7
+to the client in order to forward Application Messages that were published to topics that match
8
+these Subscriptions. The SUBSCRIBE packet also specifies (for each Subscription) the maximum QoS
9
+with which the server can send Application Messages to the client. The Netdata Agent monitors the
10
+number of unauthorized v3/v5 SUBSCRIBE attempts in the last minute.
11
+
12
+<details>
13
+<summary>MQTT basic concepts and more</summary>
14
+
15
+Basic concepts in every MQTT
16
+architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
17
+
18
+- _Broker_ - The broker is the server that distributes the information to the interested clients
19
+ connected to the server.
20
+- _Client_ - The device that connects to broker to send or receive information.
21
+- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
22
+- _Publish_ - Clients that send information to the broker to distribute to interested clients based
23
+ on the topic name.
24
+- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
25
+ subscribes to a topic, any message published to the broker is distributed to the subscribers of
26
+ that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
27
+ topic.
28
+- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
29
+ integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
30
+ transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
31
+ QoS 0.
32
+
33
+ - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
34
+ delivery. This is often refered to as fire and forget.
35
+ - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
36
+ received, known otherwise as acknowledged delivery.
37
+ - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
38
+ ensure only one copy of the message is received, known as assured delivery.
39
+
40
+- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
41
+ communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
42
+ of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
43
+ WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
44
+ documentation in
45
+ the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
46
+ section
47
+
48
+</details>
49
+
50
+<details>
51
+<summary>References and sources</summary>
52
+
53
+1. [MQTT basic concepts](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
54
+2. [MQTT v5 docs SUBSCRIBE packet](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901161)
55
+3. [auth using VerneMQ files](https://docs.vernemq.com/configuring-vernemq/file-auth)
56
+4. [auth using database](https://docs.vernemq.com/configuring-vernemq/db-auth)
57
+
58
+</details>
59
+
60
+### Troubleshooting Section
61
+
62
+<details>
63
+<summary>General approach</summary>
64
+
65
+You may encounter authentication errors either when an unauthorized client wants to subscribe to a
66
+topic or when for some reason a client's credentials are outdated.
67
+
68
+You can check which client is trying to make an unauthorized request by inspecting your VerneMQ
69
+logs.
70
+
71
+```
72
+root@netdata # cat /var/log/vernemq/console.log | grep "auth_on_subscribe"
73
+```
74
+
75
+In any case we advise you to consult the official docs to short any issue with authentication (for
76
+authentication methods using auth
77
+files <sup>[3](https://docs.vernemq.com/configuring-vernemq/file-auth) </sup> or auth via a
78
+database<sup> [4](https://docs.vernemq.com/configuring-vernemq/file-auth) </sup> )
79
+</details>
80
+
health/guides/vernemq/vernemq_mqtt_subscribe_error.md
new
+73
@@ -0,0 +1,73 @@
1
+# vernemq_mqtt_subscribe_error
2
+
3
+**Messaging | VerneMQ**
4
+
5
+The SUBSCRIBE packet is sent from the client to the server to create one or more subscriptions. Each
6
+subscription registers a Client’s interest in one or more Topics. The server sends PUBLISH packets
7
+to the Client in order to forward Application Messages that were published to Topics that match
8
+these subscriptions. The SUBSCRIBE packet also specifies (for each subscription) the maximum QoS
9
+with which the server can send Application Messages to the Client. The Netdata Agent monitors the
10
+number of failed v3/v5 SUBSCRIBE operations in the last minute.
11
+
12
+<details>
13
+<summary>MQTT basic concepts and more</summary>
14
+
15
+Basic concepts in every MQTT
16
+architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>
17
+
18
+- _Broker_ - The broker is the server that distributes the information to the interested clients
19
+ connected to the server.
20
+- _Client_ - The device that connects to broker to send or receive information.
21
+- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
22
+- _Publish_ - Clients that send information to the broker to distribute to interested clients based
23
+ on the topic name.
24
+- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
25
+ subscribes to a topic, any message published to the broker is distributed to the subscribers of
26
+ that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
27
+ topic.
28
+- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
29
+ integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
30
+ transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
31
+ QoS 0.
32
+
33
+ - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
34
+ delivery. This is often refered to as fire and forget.
35
+ - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
36
+ received, known otherwise as acknowledged delivery.
37
+ - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
38
+ ensure only one copy of the message is received, known as assured delivery.
39
+
40
+- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
41
+ communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
42
+ of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
43
+ WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
44
+ documentation in
45
+ the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
46
+ section
47
+
48
+</details>
49
+
50
+<details>
51
+<summary>References and sources</summary>
52
+
53
+1. [MQTT basic concepts](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
54
+2. [MQTT v5 docs SUBSCRIBE packet](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901161)
55
+
56
+
57
+</details>
58
+
59
+### Troubleshooting Section
60
+
61
+These kinds of errors can appear in cases of a network partition in your cluster (aka netsplit)
62
+
63
+<details>
64
+<summary>Check connectivity between nodes</summary>
65
+
66
+You must ensure that the connectivity between your cluster nodes is valid. As soon as the partition
67
+is healed, and connectivity reestablished, the VerneMQ nodes replicate the latest changes made to
68
+the subscription data. This includes all the changes 'accidentally' made during the Window of
69
+Uncertainty. Using Dotted Version Vectors VerneMQ ensures that convergence regarding subscription
70
+data and retained messages is eventually reached.
71
+
72
+</details>
73
+
health/guides/vernemq/vernemq_mqtt_unsubscribe_error.md
new
+42
@@ -0,0 +1,42 @@
1
+# vernemq_mqtt_unsubscribe_error
2
+
3
+**Messaging | VerneMQ**
4
+
5
+Publish/Subscribe is a messaging pattern that aims to decouple the sending (Publisher) and
6
+receiving (Subscriber) party. A real world example could be a sport mobile app that shows you
7
+up-to-date information of a particular football game you're interested in. In this case you are the
8
+subscriber, as you express interest in this specific game. On the other side sits the publisher,
9
+which is an online reporter that feeds a system with the actual match data. This system, which is
10
+often referred as the message broker brings the two parties together by sending the new data to all
11
+interested subscribers.
12
+<sup>[1](https://vernemq.com/intro/mqtt-primer/publish-subscribe.html) <sup>
13
+
14
+An UNSUBSCRIBE packet is sent by the Client to the Server, to unsubscribe from topics. The Netdata
15
+Agent monitors the number of failed v3/v5 UNSUBSCRIBE operations in the last minute.
16
+
17
+MQTT v5 protocol provides detailed UNSUBSCRIBE reasons codes as opposed to MQTT v3. You can find the
18
+detailed response codes which was sent by a client or a server and their descriptions in the
19
+official documentation of MQTT in
20
+the [MQTT v5 docs, UNSUBSCRIBE Reason Code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901194)
21
+
22
+
23
+<details>
24
+<summary>References and sources</summary>
25
+
26
+1. [Pub/Sub process explained on VerneMQ site](https://vernemq.com/intro/mqtt-primer/publish-subscribe.html)
27
+2. [MQTT v3 docs UNSUBSCRIBE request](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718072)
28
+3. [MQTT v5 docs UNSUBSCRIBE request](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901179)
29
+
30
+</details>
31
+
32
+### Troubleshooting Section
33
+
34
+<details>
35
+<summary>General approach</summary>
36
+
37
+Open the alerts Dashboard and locate the chart of this alert (`mqtt_unsubscribe_error`). Inspect the
38
+error log of your VerneMQ cluster in the timestamp of this alert.
39
+
40
+</details>
41
+
42
+
health/guides/vernemq/vernemq_netsplits.md
new
+25
@@ -0,0 +1,25 @@
1
+# vernemq_netsplits
2
+
3
+**Messaging | VerneMQ**
4
+
5
+A netsplit (also known as split-brain) is mostly the result of a failure of one or more network devices resulting in a cluster
6
+where nodes can no longer reach each other.
7
+
8
+The VerneMQ documentation explains how VerneMQ [deals with netsplits](https://docs.vernemq.com/v/master/vernemq-clustering/netsplits).
9
+
10
+The Netdata Agent monitors the number of detected netsplits within the last minute. This alert indicates a split-brain situation.
11
+
12
+### Troubleshooting section
13
+
14
+<details>
15
+<summary>Check connectivity between nodes</summary>
16
+
17
+You must ensure that the connectivity between your cluster nodes is valid. As soon as the partition
18
+is healed, and connectivity reestablished, the VerneMQ nodes replicate the latest changes made to
19
+the subscription data. This includes all the changes 'accidentally' made during the window of
20
+uncertainty. VerneMQ uses dotted version vectors to ensure that convergence regarding subscription
21
+data and retained messages is eventually reached.
22
+
23
+</details>
24
+
25
+
health/guides/vsphere/vsphere_host_mem_usage.md
new
+26
@@ -0,0 +1,26 @@
1
+# vsphere_host_mem_usage
2
+
3
+## Virtual Machine | Memory
4
+
5
+This alert presents the host's memory utilization.
6
+
7
+It is raised into warning if it exceeds 90%.
8
+If the metric exceeds 98%, then the alert is raised into critical.
9
+
10
+For more information on virtual machine memory utilization,
11
+please see the [VMware Documentation](
12
+ https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-115861E6-810A-43BB-8CDB-EE99CF8F3250.html)
13
+ .
14
+
15
+### Troubleshooting
16
+
17
+To find out why the alert was raised, follow the steps in the [VMware Documentation](
18
+https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-115861E6-810A-43BB-8CDB-EE99CF8F3250.html)
19
+.
20
+
21
+<details><summary>References and Sources</summary>
22
+
23
+1. [VMware Documentation](
24
+ https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-115861E6-810A-43BB-8CDB-EE99CF8F3250.html)
25
+
26
+</details>
health/guides/vsphere/vsphere_inbound_packets_errors_ratio.md
new
+48
@@ -0,0 +1,48 @@
1
+# vsphere_inbound_packets_errors_ratio
2
+
3
+## Virtual Machine | Network
4
+
5
+This alert presents the ratio of inbound packet errors for the network interface over the last 10
6
+minutes.
7
+
8
+The percentage of dropped packets is calculated over the last 10 minutes. To raise the alert, the minimum number of packets must be at least 10k within the last 10 minutes; otherwise the alert is never raised.
9
+
10
+If the value is >= 2% the alarm gets raised into the warning state.
11
+
12
+<details><summary>What are Packet Errors?</summary>
13
+
14
+A packet error means there’s something wrong with the packet. There are two types of packet
15
+errors that usually occur:
16
+- Transmission errors, where a packet is damaged on its way to its destination – like a fragile Amazon order that gets dinged up en route.
17
+- Format errors, where a packet’s format isn’t what the receiving device was expecting (or wanting). Think ordering a Coca-Cola in a restaurant and getting a Pepsi instead.
18
+
19
+Packets can easily become damaged on their way through a network. Common reasons for damaged packages are if a device is connected to Ethernet through a:
20
+- Bad cable
21
+- Bad port
22
+- Broken fiber cable
23
+- Dirty fiber connector
24
+
25
+Access points are also susceptible to packet errors. Offices often have multiple sources of
26
+high radio frequency interference thanks to Bluetooth devices, unmanaged access points,
27
+microwaves, and more. So packets traveling wirelessly are easily damaged.
28
+
29
+If a packet error occurs, TCP (Transmission Control Protocol) will resend the same information
30
+repeatedly, in hopes the data will eventually reach the destination without any problems.
31
+UDP (User Datagram Protocol) will keep trucking forward even when packets fail to reach
32
+their destination.<sup>[1](https://www.auvik.com/franklyit/blog/packet-errors-packet-discards-packet-loss/) </sup>
33
+</details>
34
+
35
+For further information, please have a look at the *References and Sources* section.
36
+
37
+<details><summary>References and Sources</summary>
38
+
39
+1. [Packet Errors](https://www.auvik.com/franklyit/blog/packet-errors-packet-discards-packet-loss/)
40
+
41
+2. [VMware Documentation](
42
+ https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-6DB73F20-C99A-43D4-9EE0-3277974EF8BF.html)
43
+</details>
44
+
45
+### Troubleshooting Section
46
+
47
+To find out why the alert was raised, follow the steps in the [VMware Documentation](
48
+https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-6DB73F20-C99A-43D4-9EE0-3277974EF8BF.html).
health/guides/vsphere/vsphere_vm_mem_usage.md
new
+23
@@ -0,0 +1,23 @@
1
+# vsphere_vm_mem_usage
2
+
3
+## Virtual Machine | Memory
4
+
5
+This alert presents the virtual machine's memory utilization.
6
+
7
+It is raised into warning if it exceeds 90%.
8
+If the metric exceeds 98%, then the alert is raised into critical.
9
+
10
+For further information, please have a look at the *References and Sources* section.
11
+
12
+<details><summary>References and Sources</summary>
13
+
14
+[VMware Documentation](
15
+ https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-81E25CBB-16D9-416B-AD6F-5A96D7CD0A2D.html)
16
+
17
+</details>
18
+
19
+### Troubleshooting section
20
+
21
+To find out why the alert was raised, follow the steps in the [VMware Documentation](
22
+https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-81E25CBB-16D9-416B-AD6F-5A96D7CD0A2D.html)
23
+.
health/guides/web_log/1m_bad_requests.md
new
+161
@@ -0,0 +1,161 @@
1
+# 1m_bad_requests
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+The 4xx class of status code is intended for cases in which the client seems to have erred. Except
9
+when responding to a HEAD request, the server should include an entity containing an explanation of
10
+the error situation, and whether it is a temporary or permanent condition. These status codes are
11
+applicable to any request
12
+method. <sup>[1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4) </sup>
13
+
14
+The Netdata Agent calculates the ratio of client error HTTP requests over the last minute. This
15
+metric does not include the 401 errors.
16
+
17
+<details>
18
+ <summary>Client error responses (4XX) </summary>
19
+
20
+The client error codes below contain the descriptions as provided by Mozilla. <sup>
21
+[2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
22
+</sup>
23
+
24
+- _400 Bad Request_: The server could not understand the request due to invalid syntax.
25
+
26
+- _401 Unauthorized_: Although the HTTP standard specifies "unauthorized", semantically this
27
+ response means "unauthenticated". That is, the client must authenticate itself to get the
28
+ requested response.
29
+
30
+- _402 Payment Required_: This response code is reserved for future use. The initial aim for
31
+ creating this code was using it for digital payment systems, however this status code is used very
32
+ rarely and no standard convention exists.
33
+
34
+- _403 Forbidden_: The client does not have access rights to the content; that is, it is
35
+ unauthorized, so the server is refusing to give the requested resource. Unlike 401 Unauthorized,
36
+ the client's identity is known to the server.
37
+
38
+- _404 Not Found_: The server can not find the requested resource. In the browser, this means the
39
+ URL is not recognized. In an API, this can also mean that the endpoint is valid but the resource
40
+ itself does not exist. Servers may also send this response instead of 403 Forbidden to hide the
41
+ existence of a resource from an unauthorized client. This response code is probably the most well
42
+ known due to its frequent occurrence on the web.
43
+
44
+- _405 Method Not Allowed_: The request method is known by the server but is not supported by the
45
+ target resource. For example, an API may not allow calling DELETE to remove a resource.
46
+
47
+- _406 Not Acceptable_: This response is sent when the web server, after performing server-driven
48
+ content negotiation, doesn't find any content that conforms to the criteria given by the user
49
+ agent.
50
+
51
+- _407 Proxy Authentication Required_: This is similar to 401 Unauthorized but authentication is
52
+ needed to be done by a proxy.
53
+
54
+- _408 Request Timeout_: This response is sent on an idle connection by some servers, even without
55
+ any previous request by the client. It means that the server would like to shut down this unused
56
+ connection. This response is used much more since some browsers, like Chrome, Firefox 27+, or IE9,
57
+ use HTTP pre-connection mechanisms to speed up surfing. Also note that some servers merely shut
58
+ down the connection without sending this message.
59
+
60
+- _409 Conflict_: This response is sent when a request conflicts with the current state of the
61
+ server.
62
+
63
+- _410 Gone_: This response is sent when the requested content has been permanently deleted from
64
+ server, with no forwarding address. Clients are expected to remove their caches and links to the
65
+ resource. The HTTP specification intends this status code to be used for "limited-time,
66
+ promotional services". APIs should not feel compelled to indicate resources that have been deleted
67
+ with this status code.
68
+
69
+- _411 Length Required_: Server rejected the request because the Content-Length header field is not
70
+ defined and the server requires it.
71
+
72
+- _412 Precondition Failed_: The client has indicated preconditions in its headers which the server
73
+ does not meet.
74
+
75
+- _413 Payload Too Large_: Request entity is larger than limits defined by server. The server might
76
+ close the connection or return an Retry-After header field.
77
+
78
+- _414 URI Too Long_: The URI requested by the client is longer than the server is willing to
79
+ interpret.
80
+
81
+- _415 Unsupported Media Type_: The media format of the requested data is not supported by the
82
+ server, so the server is rejecting the request.
83
+
84
+- _416 Range Not Satisfiable_: The range specified by the Range header field in the request cannot
85
+ be fulfilled. It's possible that the range is outside the size of the target URI's data.
86
+
87
+- _417 Expectation Failed_: This response code means the expectation indicated by the Expect request
88
+ header field cannot be met by the server.
89
+
90
+- _418 I'm a teapot_: The server refuses the attempt to brew coffee with a teapot.
91
+
92
+- _421 Misdirected Request_: The request was directed at a server that is not able to produce a
93
+ response. This can be sent by a server that is not configured to produce responses for the
94
+ combination of scheme and authority that are included in the request URI.
95
+
96
+- _422 Unprocessable Entity (WebDAV)_:
97
+ The request was well-formed but was unable to be followed due to semantic errors.
98
+
99
+- _423 Locked (WebDAV)_:
100
+ The resource that is being accessed is locked.
101
+
102
+- _424 Failed Dependency (WebDAV)_:
103
+ The request failed due to failure of a previous request.
104
+
105
+- _425 Too Early_: Indicates that the server is unwilling to risk processing a request that might be
106
+ replayed.
107
+
108
+- _426 Upgrade Required_: The server refuses to perform the request using the current protocol but
109
+ might be willing to do so after the client upgrades to a different protocol. The server sends an
110
+ Upgrade header in a 426 response to indicate the required protocol(s).
111
+
112
+- _428 Precondition Required_: The origin server requires the request to be conditional. This
113
+ response is intended to prevent the 'lost update' problem, where a client GETs a resource's state,
114
+ modifies it and PUTs it back to the server, when meanwhile a third party has modified the state on
115
+ the server, leading to a conflict.
116
+
117
+- _429 Too Many Requests_: The user has sent too many requests in a given amount of time ("rate
118
+ limiting").
119
+
120
+- _431 Request Header Fields Too Large_: The server is unwilling to process the request because its
121
+ header fields are too large. The request may be resubmitted after reducing the size of the request
122
+ header fields.
123
+
124
+- _451 Unavailable For Legal Reasons_: The user agent requested a resource that cannot legally be
125
+ provided, such as a web page censored by a government.
126
+
127
+</details>
128
+
129
+<details>
130
+ <summary>References and sources</summary>
131
+
132
+1. [https://datatracker.ietf.org/doc/html/rfc2616#section-10.4](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4)
133
+2. [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
134
+
135
+</details>
136
+
137
+### Troubleshooting section:
138
+
139
+<details>
140
+<summary>General approach</summary>
141
+
142
+
143
+To identify the HTTP response code your web server sends back:
144
+
145
+1. Open the Netdata dashboard.
146
+2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps
147
+track of exactly what error codes your web server sends out.
148
+
149
+You should also check server logs for more details about how the server is handling the requests.
150
+For example, web servers such as Apache or Nginx produce two files called access.log and error.log
151
+(by default under `/var/log/{nginx, apache2}/{access.log, error.log}`)
152
+
153
+</details>
154
+
155
+<details>
156
+<summary>Troubleshoot 404 codes in the server side </summary>
157
+
158
+The 404 requests indicate outdated links on your website or in other websites that redirect to your website.
159
+To check for dead links on your on website, use a `broken link checker` software periodically.
160
+
161
+</details>
health/guides/web_log/1m_internal_errors.md
new
+99
@@ -0,0 +1,99 @@
1
+# 1m_internal_errors
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+Response status codes beginning with the digit "5" indicate cases in which the server is aware that
9
+it has erred or is incapable of performing the request. Except when responding to a HEAD request,
10
+the server should include an entity containing an explanation of the error situation, and whether it
11
+is a temporary or permanent condition. User agents should display any included entity to the user.
12
+These response codes are applicable to any request
13
+method.<sup>[1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5) </sup>
14
+
15
+The Netdata Agent calculates the ratio of server error HTTP requests over the last minute.
16
+
17
+<details>
18
+ <summary>Server error responses (5XX)</summary>
19
+
20
+The error codes below contain the descriptions as provided by
21
+Mozilla. <sup>[2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses) </sup>
22
+
23
+- _500 Internal Server Error_: The server has encountered a situation it does not know how to
24
+ handle.
25
+
26
+- _501 Not Implemented_: The request method is not supported by the server and cannot be handled.
27
+ The only methods that servers are required to support (and therefore that must not return this
28
+ code)
29
+ are GET and HEAD.
30
+
31
+- _502 Bad Gateway_: This error response means that the server, while working as a gateway to get a
32
+ response needed to handle the request, got an invalid response.
33
+
34
+- _503 Service Unavailable_: The server is not ready to handle the request. Common causes are a
35
+ server that is down for maintenance or that is overloaded. Note that together with this response,
36
+ a user-friendly page explaining the problem should be sent. This response should be used for
37
+ temporary conditions and the Retry-After HTTP header should, if possible, contain the estimated
38
+ time before the recovery of the service. The webmaster must also take care about the
39
+ caching-related headers that are sent along with this response, as these temporary condition
40
+ responses should usually not be cached.
41
+
42
+- _504 Gateway Timeout_: This error response is given when the server is acting as a gateway and
43
+ cannot get a response in time.
44
+
45
+- _505 HTTP Version Not Supported_: The HTTP version used in the request is not supported by the
46
+ server.
47
+
48
+- _506 Variant Also Negotiates_: The server has an internal configuration error: the chosen variant
49
+ resource is configured to engage in transparent content negotiation itself, and is therefore not a
50
+ proper end point in the negotiation process.
51
+
52
+- _507 Insufficient Storage (WebDAV)_:
53
+ The method could not be performed on the resource because the server is unable to store the
54
+ representation needed to successfully complete the request.
55
+
56
+- _508 Loop Detected (WebDAV)_:
57
+ The server detected an infinite loop while processing the request.
58
+
59
+- _510 Not Extended_: Further extensions to the request are required for the server to fulfill it.
60
+
61
+- _511 Network Authentication_: Required Indicates that the client needs to authenticate to gain
62
+ network access.
63
+
64
+Source: [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
65
+
66
+</details>
67
+
68
+<details>
69
+ <summary>References and sources</summary>
70
+
71
+1. [Server errors on Datatracker](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5)
72
+
73
+2. [HTTP server errors on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
74
+
75
+</details>
76
+
77
+### Troubleshooting section:
78
+
79
+<details>
80
+<summary>General approach</summary>
81
+
82
+To identify the HTTP response code your web server sends back:
83
+
84
+1. Open the Netdata dashboard.
85
+2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps
86
+track of exactly what error codes your web server sends out.
87
+
88
+You should also check the server error logs. For example, web servers such as Apache or Nginx
89
+produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
90
+
91
+</details>
92
+
93
+<details>
94
+<summary>Troubleshoot 500 error code </summary>
95
+
96
+One of the things that can cause HTTP 500 response errors is a misconfiguration in the `.htaccess`
97
+file of your web server.
98
+
99
+</details>
health/guides/web_log/1m_redirects.md
new
+86
@@ -0,0 +1,86 @@
1
+# 1m_redirects
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+The 3XX class of status code indicates that further action needs to be taken by the user agent in
9
+order to fulfill the request. The action required may be carried out by the user agent without
10
+interaction with the user if and only if the method used in the second request is GET or HEAD. A
11
+client SHOULD detect infinite redirection loops, since such loops generate network traffic for each
12
+redirection.
13
+<sup> [1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3) </sup>
14
+
15
+The Netdata Agent calculates the ratio of redirection HTTP requests over the last minute. This
16
+metric does not include the "304 Not modified" message.
17
+
18
+<details>
19
+ <summary>Redirection messages (3XX)</summary>
20
+
21
+The redirect messages below contain the descriptions as provided by
22
+Mozilla.<sup> [2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages) </sup>
23
+
24
+- _300 Multiple Choice_: The request has more than one possible response. The user agent or user
25
+ should choose one of them. (There is no standardized way of choosing one of the responses, but
26
+ HTML links to the possibilities are recommended so the user can pick.)
27
+
28
+- _301 Moved Permanently_: The URL of the requested resource has been changed permanently. The new
29
+ URL is given in the response.
30
+
31
+- _302 Found_: This response code means that the URI of requested resource has been changed
32
+ temporarily. Further changes in the URI might be made in the future. Therefore, this same URI
33
+ should be used by the client in future requests.
34
+
35
+- _303 See Other_: The server sent this response to direct the client to get the requested resource
36
+ at another URI with a GET request.
37
+
38
+- _304 Not Modified_: This is used for caching purposes. It tells the client that the response has
39
+ not been modified, so the client can continue to use the same cached version of the response.
40
+
41
+- _305 Use Proxy_: Defined in a previous version of the HTTP specification to indicate that a
42
+ requested response must be accessed by a proxy. It has been deprecated due to security concerns
43
+ regarding in-band configuration of a proxy.
44
+
45
+- _306 unused_: This response code is no longer used; it is just reserved. It was used in a previous
46
+ version of the HTTP/1.1 specification.
47
+
48
+- _307 Temporary Redirect_: The server sends this response to direct the client to get the requested
49
+ resource at another URI with same method that was used in the prior request. This has the same
50
+ semantics as the 302 Found HTTP response code, with the exception that the user agent must not
51
+ change the HTTP method used: if a POST was used in the first request, a POST must be used in the
52
+ second request.
53
+
54
+- _308 Permanent Redirect_: This means that the resource is now permanently located at another URI,
55
+ specified by the Location: HTTP Response header. This has the same semantics as the 301 Moved
56
+ Permanently HTTP response code, with the exception that the user agent must not change the HTTP
57
+ method used: if a POST was used in the first request, a POST must be used in the second request.
58
+
59
+</details>
60
+
61
+
62
+<details>
63
+ <summary>References and sources</summary>
64
+
65
+1. [3XX codes in the HTTP protocol](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3)
66
+
67
+2. [HTTP redirection messages on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages)
68
+
69
+</details>
70
+
71
+### Troubleshooting section:
72
+
73
+<details>
74
+<summary>General approach</summary>
75
+
76
+You can identify exactly what HTTP response code your web server send back to your clients by opening the Netdata
77
+dashboard and inspecting the `detailed_response_codes` chart for your web server. This chart keeps
78
+track of exactly what error codes your web server sends out.
79
+
80
+You should also check the server error logs. For example, web servers such as Apache or Nginx
81
+produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
82
+
83
+</details>
84
+
85
+
86
+
health/guides/web_log/1m_successful.md
new
+117
@@ -0,0 +1,117 @@
1
+# 1m_successful
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+The Netdata Agent calculates the ratio of successful HTTP requests over the last minute. These
9
+requests consist of 1xx, 2xx, 304, 401 response codes. You receive this alert in warning when the
10
+percentage of successful requests is less than 85% and in critical when it is below 75%. This alert
11
+can indicate:
12
+
13
+- A malfunction in the services of your web server
14
+- Malicious activity towards your website
15
+- Broken links towards your servers.
16
+
17
+In most cases, the Agent will send you another alert indicating high incidences
18
+of "abnormal" HTTP requests code, for example you could also receive the `1m_bad_requests` alert.
19
+
20
+
21
+<details>
22
+ <summary>See more about the response codes this alert track </summary>
23
+
24
+The response codes below contain the descriptions as provided by
25
+Mozilla. <sup> [1](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) </sup>
26
+
27
+**Information responses (1XX)**
28
+
29
+- _100 Continue_:This interim response indicates that the client should continue the request or ignore
30
+ the response if the request is already finished.
31
+
32
+- _101 Switching Protocol_: This code is sent in response to an Upgrade request header from the client
33
+ and indicates the protocol the server is switching to.
34
+
35
+- _102 Processing (WebDAV)_:
36
+ This code indicates that the server has received and is processing the request, but no response is
37
+ available yet.
38
+
39
+- _103 Early Hints_: This status code is primarily intended to be used with the link header, letting
40
+ the user agent start preloading resources while the server prepares a response.
41
+
42
+**Successful responses (2XX)**
43
+
44
+- _200 OK_: The request succeeded. The result meaning of "success" depends on the HTTP method:
45
+
46
+ * GET: The resource has been fetched and transmitted in the message body.
47
+ * HEAD: The representation headers are included in the response without any message body.
48
+ * PUT or POST: The resource describing the result of the action is transmitted in the message
49
+ body.
50
+ * TRACE: The message body contains the request message as received by the server.
51
+
52
+- _201 Created_: The request succeeded, and a new resource created as a result. This is typically the
53
+ response sent after POST requests, or some PUT requests.
54
+
55
+- _202 Accepted_: The request has been received but not yet acted upon. It is noncommittal, since there
56
+ is no way in HTTP to later send an asynchronous response indicating the outcome of the request. It
57
+ is intended for cases where another process or server handles the request, or for batch
58
+ processing.
59
+
60
+- _203 Non-Authoritative Information_: This response code means the returned metadata is not exactly
61
+ the same as is available from the origin server, but is collected from a local or a third-party
62
+ copy. This is mostly used for mirrors or backups of another resource. Except for that specific
63
+ case, the 200 OK response is preferred to this status.
64
+
65
+- _204 No Content_: There is no content to send for this request, but the headers may be useful. The
66
+ user agent may update its cached headers for this resource with the new ones.
67
+
68
+- _205 Reset Content_: Tells the user agent to reset the document which sent this request.
69
+
70
+- _206 Partial Content_: This response code is used when the range header is sent from the client to
71
+ request only part of a resource.
72
+
73
+- _207 Multi-Status (WebDAV)_:
74
+ Conveys information about multiple resources, for situations where multiple status codes might be
75
+ appropriate.
76
+
77
+- _208 Already Reported (WebDAV)_:
78
+ Used inside a <dav:propstat> response element to avoid repeatedly enumerating the internal members
79
+ of multiple bindings to the same collection.
80
+
81
+- _226 IM Used (HTTP Delta encoding)_:
82
+ The server has fulfilled a GET request for the resource, and the response is a representation of
83
+ the result of one or more instance-manipulations applied to the current instance.
84
+
85
+**Redirection messages (3XX)**
86
+
87
+- _304 Not Modified _: This is used for caching purposes. It tells the client that the response has not
88
+ been modified, so the client can continue to use the same cached version of the response.
89
+
90
+**Client error responses (4XX)**
91
+
92
+- _401 Unauthorized_: Although the HTTP standard specifies "unauthorized", semantically this response
93
+ means "
94
+ unauthenticated". That is, the client must authenticate itself to get the requested response.
95
+
96
+</details>
97
+
98
+<details>
99
+ <summary>References and Sources </summary>
100
+
101
+1. [HTTP status codes on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
102
+
103
+</details>
104
+
105
+### Troubleshooting section:
106
+
107
+There are a number of reasons triggering this alert. All of them could eventually cause bad user
108
+experience with your web services.
109
+
110
+<details>
111
+<summary>General approach</summary>
112
+
113
+Identify exactly what HTTP response code your web server sent back to your clients.
114
+Open the Netdata dashboard and inspect the `detailed_response_codes` chart for your web server. This chart keeps
115
+track of exactly what error codes your web server sends out.
116
+
117
+</details>
health/guides/web_log/1m_unmatched.md
new
+43
@@ -0,0 +1,43 @@
1
+# 1m_unmatched
2
+
3
+**Web Server | Web log**
4
+
5
+In a webserver, all activity should be monitored. By default, most of the webservers log
6
+activity in an `access.log` file. The access log is a list of all requests for
7
+individual files that people or bots have requested from a website. Log File strings include notes
8
+about their requests for the HTML files and their embedded graphic images, along with any other
9
+associated files that are transmitted.
10
+
11
+The Netdata Agent calculates the percentage of unparsed log lines over the last minute. These are
12
+entries in the log file that didn't match in any of the common pattern operations of
13
+the webserver (1XX, 2XX, etc). This can indicate an abnormal activity on your web server, or that your server is
14
+performing operations that you cannot monitor with the Agent.
15
+
16
+Web servers like NGINX and Apache2 gives you the ability to modify the log patterns for each request.
17
+If you have done that, you will also need to adjust the Netdata Agent to parse those patterns.
18
+
19
+### Troubleshooting section:
20
+
21
+<details>
22
+<summary>Create a custom log format job</summary>
23
+
24
+This alert is triggered by the `python.d.plugin`. You must create a new job in the `web_log`
25
+collector for your Agent.
26
+
27
+1. See how you can [configure this collector](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/web_log#configuration)
28
+
29
+
30
+2. Follow the job template specified in
31
+the [default web_log.conf file](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/web_log/web_log.conf#L53-L86)
32
+, focus on the
33
+lines [83:85](https://github.com/netdata/netdata/blob/e6d9fbc4a53f1d35363e9b342231bb11627bafbd/collectors/python.d.plugin/web_log/web_log.conf#L83-L85)
34
+where you can see how you define a `custom_log_format`.
35
+
36
+
37
+3. Restart the Netdata Agent
38
+ ```
39
+ root@netdata # systemctl restart netdata
40
+ ```
41
+</details>
42
+
43
+
health/guides/web_log/5m_requests_ratio.md
new
+24
@@ -0,0 +1,24 @@
1
+# 5m_requests_ratio
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+The Netdata Agent calculates the ratio of successful HTTP requests over the last 5 minutes, compared
9
+with the previous 5 minutes. These requests consist of 1xx, 2xx, 304, 401 response codes.
10
+
11
+You receive this alert in warning when the percentage of requests is increased more than 25% in the
12
+last 5 minutes compared with the previous 5 minutes and in critical when it increased more than 50%.
13
+A clear notification for this alert **will not be sent** when the ratio becomes normal again.
14
+
15
+### Troubleshooting section:
16
+
17
+This alert is not always a bad thing. It means that there is a slight increase in the
18
+requests towards to your Web server. You should just keep an eye on this metrics. If you receive
19
+this alert regularly you should consider take action in advance to avoid server overload.
20
+
21
+You can find an interesting article on actions you can take
22
+to [manage sudden traffic](https://www.nginx.com/blog/how-to-manage-sudden-traffic-surges-server-overload/)
23
+on a web server (this article is produced by the NGINX associates, but nearly same principles
24
+applied in any case)
health/guides/web_log/web_log_1m_bad_requests.md
new
+161
@@ -0,0 +1,161 @@
1
+# web_log_1m_bad_requests
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+The 4xx class of status code is intended for cases in which the client seems to have erred. Except
9
+when responding to a HEAD request, the server should include an entity containing an explanation of
10
+the error situation, and whether it is a temporary or permanent condition. These status codes are
11
+applicable to any request
12
+method. <sup>[1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4) </sup>
13
+
14
+The Netdata Agent calculates the ratio of client error HTTP requests over the last minute. This
15
+metric does not include the 401 errors.
16
+
17
+<details>
18
+ <summary>Client error responses (4XX) </summary>
19
+
20
+The client error codes below contain the descriptions as provided by Mozilla. <sup>
21
+[2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
22
+</sup>
23
+
24
+- _400 Bad Request_: The server could not understand the request due to invalid syntax.
25
+
26
+- _401 Unauthorized_: Although the HTTP standard specifies "unauthorized", semantically this
27
+ response means "unauthenticated". That is, the client must authenticate itself to get the
28
+ requested response.
29
+
30
+- _402 Payment Required_: This response code is reserved for future use. The initial aim for
31
+ creating this code was using it for digital payment systems, however this status code is used very
32
+ rarely and no standard convention exists.
33
+
34
+- _403 Forbidden_: The client does not have access rights to the content; that is, it is
35
+ unauthorized, so the server is refusing to give the requested resource. Unlike 401 Unauthorized,
36
+ the client's identity is known to the server.
37
+
38
+- _404 Not Found_: The server can not find the requested resource. In the browser, this means the
39
+ URL is not recognized. In an API, this can also mean that the endpoint is valid but the resource
40
+ itself does not exist. Servers may also send this response instead of 403 Forbidden to hide the
41
+ existence of a resource from an unauthorized client. This response code is probably the most well
42
+ known due to its frequent occurrence on the web.
43
+
44
+- _405 Method Not Allowed_: The request method is known by the server but is not supported by the
45
+ target resource. For example, an API may not allow calling DELETE to remove a resource.
46
+
47
+- _406 Not Acceptable_: This response is sent when the web server, after performing server-driven
48
+ content negotiation, doesn't find any content that conforms to the criteria given by the user
49
+ agent.
50
+
51
+- _407 Proxy Authentication Required_: This is similar to 401 Unauthorized but authentication is
52
+ needed to be done by a proxy.
53
+
54
+- _408 Request Timeout_: This response is sent on an idle connection by some servers, even without
55
+ any previous request by the client. It means that the server would like to shut down this unused
56
+ connection. This response is used much more since some browsers, like Chrome, Firefox 27+, or IE9,
57
+ use HTTP pre-connection mechanisms to speed up surfing. Also note that some servers merely shut
58
+ down the connection without sending this message.
59
+
60
+- _409 Conflict_: This response is sent when a request conflicts with the current state of the
61
+ server.
62
+
63
+- _410 Gone_: This response is sent when the requested content has been permanently deleted from
64
+ server, with no forwarding address. Clients are expected to remove their caches and links to the
65
+ resource. The HTTP specification intends this status code to be used for "limited-time,
66
+ promotional services". APIs should not feel compelled to indicate resources that have been deleted
67
+ with this status code.
68
+
69
+- _411 Length Required_: Server rejected the request because the Content-Length header field is not
70
+ defined and the server requires it.
71
+
72
+- _412 Precondition Failed_: The client has indicated preconditions in its headers which the server
73
+ does not meet.
74
+
75
+- _413 Payload Too Large_: Request entity is larger than limits defined by server. The server might
76
+ close the connection or return an Retry-After header field.
77
+
78
+- _414 URI Too Long_: The URI requested by the client is longer than the server is willing to
79
+ interpret.
80
+
81
+- _415 Unsupported Media Type_: The media format of the requested data is not supported by the
82
+ server, so the server is rejecting the request.
83
+
84
+- _416 Range Not Satisfiable_: The range specified by the Range header field in the request cannot
85
+ be fulfilled. It's possible that the range is outside the size of the target URI's data.
86
+
87
+- _417 Expectation Failed_: This response code means the expectation indicated by the Expect request
88
+ header field cannot be met by the server.
89
+
90
+- _418 I'm a teapot_: The server refuses the attempt to brew coffee with a teapot.
91
+
92
+- _421 Misdirected Request_: The request was directed at a server that is not able to produce a
93
+ response. This can be sent by a server that is not configured to produce responses for the
94
+ combination of scheme and authority that are included in the request URI.
95
+
96
+- _422 Unprocessable Entity (WebDAV)_:
97
+ The request was well-formed but was unable to be followed due to semantic errors.
98
+
99
+- _423 Locked (WebDAV)_:
100
+ The resource that is being accessed is locked.
101
+
102
+- _424 Failed Dependency (WebDAV)_:
103
+ The request failed due to failure of a previous request.
104
+
105
+- _425 Too Early_: Indicates that the server is unwilling to risk processing a request that might be
106
+ replayed.
107
+
108
+- _426 Upgrade Required_: The server refuses to perform the request using the current protocol but
109
+ might be willing to do so after the client upgrades to a different protocol. The server sends an
110
+ Upgrade header in a 426 response to indicate the required protocol(s).
111
+
112
+- _428 Precondition Required_: The origin server requires the request to be conditional. This
113
+ response is intended to prevent the 'lost update' problem, where a client GETs a resource's state,
114
+ modifies it and PUTs it back to the server, when meanwhile a third party has modified the state on
115
+ the server, leading to a conflict.
116
+
117
+- _429 Too Many Requests_: The user has sent too many requests in a given amount of time ("rate
118
+ limiting").
119
+
120
+- _431 Request Header Fields Too Large_: The server is unwilling to process the request because its
121
+ header fields are too large. The request may be resubmitted after reducing the size of the request
122
+ header fields.
123
+
124
+- _451 Unavailable For Legal Reasons_: The user agent requested a resource that cannot legally be
125
+ provided, such as a web page censored by a government.
126
+
127
+</details>
128
+
129
+<details>
130
+ <summary>References and sources</summary>
131
+
132
+1. [https://datatracker.ietf.org/doc/html/rfc2616#section-10.4](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4)
133
+2. [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
134
+
135
+</details>
136
+
137
+### Troubleshooting section:
138
+
139
+<details>
140
+<summary>General approach</summary>
141
+
142
+
143
+To identify the HTTP response code your web server sends back:
144
+
145
+1. Open the Netdata dashboard.
146
+2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps
147
+track of exactly what error codes your web server sends out.
148
+
149
+You should also check server logs for more details about how the server is handling the requests.
150
+For example, web servers such as Apache or Nginx produce two files called access.log and error.log
151
+(by default under `/var/log/{nginx, apache2}/{access.log, error.log}`)
152
+
153
+</details>
154
+
155
+<details>
156
+<summary>Troubleshoot 404 codes in the server side </summary>
157
+
158
+The 404 requests indicate outdated links on your website or in other websites that redirect to your
159
+website. To check for dead links on your on website, use a `broken link checker` software periodically.
160
+
161
+</details>
health/guides/web_log/web_log_1m_internal_errors.md
new
+99
@@ -0,0 +1,99 @@
1
+# web_log_1m_internal_errors
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+Response status codes beginning with the digit "5" indicate cases in which the server is aware that
9
+it has erred or is incapable of performing the request. Except when responding to a HEAD request,
10
+the server should include an entity containing an explanation of the error situation, and whether it
11
+is a temporary or permanent condition. User agents should display any included entity to the user.
12
+These response codes are applicable to any request
13
+method.<sup>[1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5) </sup>
14
+
15
+The Netdata Agent calculates the ratio of server error HTTP requests over the last minute.
16
+
17
+<details>
18
+ <summary>Server error responses (5XX)</summary>
19
+
20
+The error codes below contain the descriptions as provided by
21
+Mozilla. <sup>[2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses) </sup>
22
+
23
+- _500 Internal Server Error_: The server has encountered a situation it does not know how to
24
+ handle.
25
+
26
+- _501 Not Implemented_: The request method is not supported by the server and cannot be handled.
27
+ The only methods that servers are required to support (and therefore that must not return this
28
+ code)
29
+ are GET and HEAD.
30
+
31
+- _502 Bad Gateway_: This error response means that the server, while working as a gateway to get a
32
+ response needed to handle the request, got an invalid response.
33
+
34
+- _503 Service Unavailable_: The server is not ready to handle the request. Common causes are a
35
+ server that is down for maintenance or that is overloaded. Note that together with this response,
36
+ a user-friendly page explaining the problem should be sent. This response should be used for
37
+ temporary conditions and the Retry-After HTTP header should, if possible, contain the estimated
38
+ time before the recovery of the service. The webmaster must also take care about the
39
+ caching-related headers that are sent along with this response, as these temporary condition
40
+ responses should usually not be cached.
41
+
42
+- _504 Gateway Timeout_: This error response is given when the server is acting as a gateway and
43
+ cannot get a response in time.
44
+
45
+- _505 HTTP Version Not Supported_: The HTTP version used in the request is not supported by the
46
+ server.
47
+
48
+- _506 Variant Also Negotiates_: The server has an internal configuration error: the chosen variant
49
+ resource is configured to engage in transparent content negotiation itself, and is therefore not a
50
+ proper end point in the negotiation process.
51
+
52
+- _507 Insufficient Storage (WebDAV)_:
53
+ The method could not be performed on the resource because the server is unable to store the
54
+ representation needed to successfully complete the request.
55
+
56
+- _508 Loop Detected (WebDAV)_:
57
+ The server detected an infinite loop while processing the request.
58
+
59
+- _510 Not Extended_: Further extensions to the request are required for the server to fulfill it.
60
+
61
+- _511 Network Authentication_: Required Indicates that the client needs to authenticate to gain
62
+ network access.
63
+
64
+Source: [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
65
+
66
+</details>
67
+
68
+<details>
69
+ <summary>References and sources</summary>
70
+
71
+1. [Server errors on Datatracker](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5)
72
+
73
+2. [HTTP server errors on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
74
+
75
+</details>
76
+
77
+### Troubleshooting section:
78
+
79
+<details>
80
+<summary>General approach</summary>
81
+
82
+To identify the HTTP response code your web server sends back:
83
+
84
+1. Open the Netdata dashboard.
85
+2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps
86
+track of exactly what error codes your web server sends out.
87
+
88
+You should also check the server error logs. For example, web servers such as Apache or Nginx
89
+produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
90
+
91
+</details>
92
+
93
+<details>
94
+<summary>Troubleshoot 500 error code </summary>
95
+
96
+One of the things that can cause HTTP 500 response errors is a misconfiguration in the `.htaccess`
97
+file of your web server.
98
+
99
+</details>
health/guides/web_log/web_log_1m_redirects.md
new
+86
@@ -0,0 +1,86 @@
1
+# web_log_1m_redirects
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+The 3XX class of status code indicates that further action needs to be taken by the user agent in
9
+order to fulfill the request. The action required may be carried out by the user agent without
10
+interaction with the user if and only if the method used in the second request is GET or HEAD. A
11
+client SHOULD detect infinite redirection loops, since such loops generate network traffic for each
12
+redirection. <sup> [1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3) </sup>
13
+
14
+The Netdata Agent calculates the ratio of redirection HTTP requests over the last minute. This
15
+metric does not include the
16
+"304 Not modified" message.
17
+
18
+<details>
19
+ <summary>Redirection messages (3XX)</summary>
20
+
21
+The redirect messages below contain the descriptions as provided by
22
+Mozilla.<sup> [2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages) </sup>
23
+
24
+- _300 Multiple Choice_: The request has more than one possible response. The user agent or user
25
+ should choose one of them. (There is no standardized way of choosing one of the responses, but
26
+ HTML links to the possibilities are recommended so the user can pick.)
27
+
28
+- _301 Moved Permanently_: The URL of the requested resource has been changed permanently. The new
29
+ URL is given in the response.
30
+
31
+- _302 Found_: This response code means that the URI of requested resource has been changed
32
+ temporarily. Further changes in the URI might be made in the future. Therefore, this same URI
33
+ should be used by the client in future requests.
34
+
35
+- _303 See Other_: The server sent this response to direct the client to get the requested resource
36
+ at another URI with a GET request.
37
+
38
+- _304 Not Modified_: This is used for caching purposes. It tells the client that the response has
39
+ not been modified, so the client can continue to use the same cached version of the response.
40
+
41
+- _305 Use Proxy_: Defined in a previous version of the HTTP specification to indicate that a
42
+ requested response must be accessed by a proxy. It has been deprecated due to security concerns
43
+ regarding in-band configuration of a proxy.
44
+
45
+- _306 unused_: This response code is no longer used; it is just reserved. It was used in a previous
46
+ version of the HTTP/1.1 specification.
47
+
48
+- _307 Temporary Redirect_: The server sends this response to direct the client to get the requested
49
+ resource at another URI with same method that was used in the prior request. This has the same
50
+ semantics as the 302 Found HTTP response code, with the exception that the user agent must not
51
+ change the HTTP method used: if a POST was used in the first request, a POST must be used in the
52
+ second request.
53
+
54
+- _308 Permanent Redirect_: This means that the resource is now permanently located at another URI,
55
+ specified by the Location: HTTP Response header. This has the same semantics as the 301 Moved
56
+ Permanently HTTP response code, with the exception that the user agent must not change the HTTP
57
+ method used: if a POST was used in the first request, a POST must be used in the second request.
58
+
59
+</details>
60
+
61
+
62
+<details>
63
+ <summary>References and sources</summary>
64
+
65
+1. [3XX codes in the HTTP protocol](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3)
66
+
67
+2. [HTTP redirection messages on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages)
68
+
69
+</details>
70
+
71
+### Troubleshooting section:
72
+
73
+<details>
74
+<summary>General approach</summary>
75
+
76
+You can identify exactly what HTTP response code your web server send back to your clients, by opening the Netdata
77
+dashboard and inspecting the `detailed_response_codes` chart for your web server. This chart keeps
78
+track of exactly what error codes your web server sends out.
79
+
80
+You should also check the server error logs. For example, web servers such as Apache or Nginx
81
+produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
82
+
83
+</details>
84
+
85
+
86
+
health/guides/web_log/web_log_1m_successful.md
new
+118
@@ -0,0 +1,118 @@
1
+# web_log_1m_successful
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+The Netdata Agent calculates the ratio of successful HTTP requests over the last minute. These
9
+requests consist of 1xx, 2xx, 304, 401 response codes. You receive this alert in warning when the
10
+percentage of successful requests is less than 85% and in critical when it is below 75%. This alert
11
+can indicate:
12
+
13
+- A malfunction in the services of your web server
14
+- Malicious activity towards your website
15
+- Broken links towards your servers.
16
+
17
+In most cases, the Agent will send you another alert indicating high incidences
18
+of "abnormal" HTTP requests code, for example you could also receive the `web_log_1m_bad_requests` alert.
19
+
20
+
21
+<details>
22
+ <summary>See more about the response codes this alert track </summary>
23
+
24
+The response codes below contain the descriptions as provided by
25
+Mozilla. <sup> [1](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) </sup>
26
+
27
+**Information responses (1XX)**
28
+
29
+- _100 Continue_:This interim response indicates that the client should continue the request or ignore
30
+ the response if the request is already finished.
31
+
32
+- _101 Switching Protocol_: This code is sent in response to an Upgrade request header from the client
33
+ and indicates the protocol the server is switching to.
34
+
35
+- _102 Processing (WebDAV)_:
36
+ This code indicates that the server has received and is processing the request, but no response is
37
+ available yet.
38
+
39
+- _103 Early Hints_: This status code is primarily intended to be used with the link header, letting
40
+ the user agent start preloading resources while the server prepares a response.
41
+
42
+**Successful responses (2XX)**
43
+
44
+- _200 OK_: The request succeeded. The result meaning of "success" depends on the HTTP method:
45
+
46
+ * GET: The resource has been fetched and transmitted in the message body.
47
+ * HEAD: The representation headers are included in the response without any message body.
48
+ * PUT or POST: The resource describing the result of the action is transmitted in the message
49
+ body.
50
+ * TRACE: The message body contains the request message as received by the server.
51
+
52
+- _201 Created_: The request succeeded, and a new resource created as a result. This is typically the
53
+ response sent after POST requests, or some PUT requests.
54
+
55
+- _202 Accepted_: The request has been received but not yet acted upon. It is noncommittal, since there
56
+ is no way in HTTP to later send an asynchronous response indicating the outcome of the request. It
57
+ is intended for cases where another process or server handles the request, or for batch
58
+ processing.
59
+
60
+- _203 Non-Authoritative Information_: This response code means the returned metadata is not exactly
61
+ the same as is available from the origin server, but is collected from a local or a third-party
62
+ copy. This is mostly used for mirrors or backups of another resource. Except for that specific
63
+ case, the 200 OK response is preferred to this status.
64
+
65
+- _204 No Content_: There is no content to send for this request, but the headers may be useful. The
66
+ user agent may update its cached headers for this resource with the new ones.
67
+
68
+- _205 Reset Content_: Tells the user agent to reset the document which sent this request.
69
+
70
+- _206 Partial Content_: This response code is used when the range header is sent from the client to
71
+ request only part of a resource.
72
+
73
+- _207 Multi-Status (WebDAV)_:
74
+ Conveys information about multiple resources, for situations where multiple status codes might be
75
+ appropriate.
76
+
77
+- _208 Already Reported (WebDAV)_:
78
+ Used inside a <dav:propstat> response element to avoid repeatedly enumerating the internal members
79
+ of multiple bindings to the same collection.
80
+
81
+- _226 IM Used (HTTP Delta encoding)_:
82
+ The server has fulfilled a GET request for the resource, and the response is a representation of
83
+ the result of one or more instance-manipulations applied to the current instance.
84
+
85
+**Redirection messages (3XX)**
86
+
87
+- _304 Not Modified _: This is used for caching purposes. It tells the client that the response has not
88
+ been modified, so the client can continue to use the same cached version of the response.
89
+
90
+**Client error responses (4XX)**
91
+
92
+- _401 Unauthorized_: Although the HTTP standard specifies "unauthorized", semantically this response
93
+ means "
94
+ unauthenticated". That is, the client must authenticate itself to get the requested response.
95
+
96
+</details>
97
+
98
+<details>
99
+ <summary>References and Sources </summary>
100
+
101
+1. [HTTP status codes on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
102
+
103
+</details>
104
+
105
+### Troubleshooting section:
106
+
107
+There are a number of reasons triggering this alert. All of them could eventually cause bad user
108
+experience with your web services.
109
+
110
+<details>
111
+<summary>General approach</summary>
112
+
113
+Identify exactly what HTTP response code your web server sent back to your clients.
114
+Open the Netdata dashboard and inspect the `detailed_response_codes` chart for your web server. This chart keeps
115
+track of exactly what error codes your web server sends out.
116
+
117
+</details>
118
+
health/guides/web_log/web_log_1m_unmatched.md
new
+43
@@ -0,0 +1,43 @@
1
+# web_log_1m_unmatched
2
+
3
+**Web Server | Web log**
4
+
5
+In a webserver, all activity should be monitored. By default, most of the webservers log
6
+activity in an `access.log` file. The access log is a list of all requests for
7
+individual files that people or bots have requested from a website. Log File strings include notes
8
+about their requests for the HTML files and their embedded graphic images, along with any other
9
+associated files that are transmitted.
10
+
11
+The Netdata Agent calculates the percentage of unparsed log lines over the last minute. These are
12
+entries in the log file that didn't match in any of the common pattern operations (1XX, 2XX, etc) of
13
+the webserver. This can indicate an abnormal activity on your web server, or that your server is
14
+performing operations that you cannot monitor with the Agent.
15
+
16
+Web servers like NGINX and Apache2 give you the ability to modify the log patterns for each request.
17
+If you have done that, you also need to adjust the Netdata Agent to parse those patterns.
18
+
19
+### Troubleshooting section:
20
+
21
+<details>
22
+<summary>Create a custom log format job</summary>
23
+
24
+This alert is triggered by the `python.d.plugin`. You must create a new job in the `web_log`
25
+collector for your Agent.
26
+
27
+1. See how you can [configure this collector](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/web_log#configuration)
28
+
29
+
30
+2. Follow the job template specified in
31
+the [default web_log.conf file](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/web_log/web_log.conf#L53-L86)
32
+, focus on the
33
+lines [83:85](https://github.com/netdata/netdata/blob/e6d9fbc4a53f1d35363e9b342231bb11627bafbd/collectors/python.d.plugin/web_log/web_log.conf#L83-L85)
34
+where you can see how you define a `custom_log_format`.
35
+
36
+
37
+3. Restart the Netdata Agent
38
+ ```
39
+ root@netdata # systemctl restart netdata
40
+ ```
41
+</details>
42
+
43
+
health/guides/web_log/web_log_5m_requests_ratio.md
new
+24
@@ -0,0 +1,24 @@
1
+# web_log_5m_requests_ratio
2
+
3
+**Web Server | Web log**
4
+
5
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
+or not.
7
+
8
+The Netdata Agent calculates the ratio of successful HTTP requests over the last 5 minutes, compared
9
+with the previous 5 minutes. These requests consist of 1xx, 2xx, 304, 401 response codes.
10
+
11
+There is a warning alert when the percentage of requests is increased more than 25% in the
12
+last 5 minutes compared with the previous 5 minutes and in critical when it increased more than 50%.
13
+A clear notification for this alert **will not be sent** when the ratio becomes normal again.
14
+
15
+### Troubleshooting section:
16
+
17
+This alert is not always a bad thing. It means that there is a slight increase in the
18
+requests towards to your Web server. You should just keep an eye on this metrics. If you receive
19
+this alert regularly you should consider take action in advance to avoid server overload.
20
+
21
+You can find an interesting article on actions you can take
22
+to [manage sudden traffic](https://www.nginx.com/blog/how-to-manage-sudden-traffic-surges-server-overload/)
23
+on a web server (this article is produced by the NGINX associates, but nearly same principles
24
+applied in any case)
health/guides/web_log/web_log_web_slow.md
new
+54
@@ -0,0 +1,54 @@
1
+# web_log_web_slow
2
+
3
+**Web Server | Web log**
4
+
5
+When a client sends a request to your web server, there are many independent phases for this request
6
+that can introduce delay. Some of them may be:
7
+
8
+- DNS lookup
9
+- Establish a TCP connection
10
+- Perform a TLS handshake
11
+- The server to process the request
12
+- Transfer the actual content
13
+
14
+The Netdata Agent calculates the average HTTP response time over the last minute for your web server
15
+(NGINX, Apache). You receive this alert when your web server's average response time has increased.
16
+The alert is raised in warning when the average HTTP response time is twice as much. When the average HTTP response time is four times as much, you receive a critical alert.
17
+
18
+### Troubleshooting section
19
+
20
+The causes of a slow request response may vary. Some options you can explore are:
21
+
22
+<details>
23
+<summary>Your web server utilization is high </summary>
24
+
25
+This problem could be addressed on many levels:
26
+- Check if your host machine can handle the traffic: Check the CPU, memory and traffic utilization.
27
+ If this is not an issue:
28
+- Consider raising the resource limitations for your web server (for example add more worker processes).
29
+ Please consult your web server docs. If this also doesn't resolve the issue:
30
+- Set up an architecture with multiple web servers and load balancer to handle the traffic for your site.
31
+
32
+</details>
33
+
34
+<details>
35
+<summary>Optimize Databases </summary>
36
+
37
+The response speed is dependent on database optimization. As you first set up a website, the
38
+database responds quickly to queries. As time passes, the database accumulates information. The
39
+compilation results in massive amounts of stored data and might slow down response times.
40
+
41
+If you manage your database with MySQL, this blogpost proposes ways to [tune MySQL operations](https://www.cloudways.com/blog/mysql-performance-tuning/).
42
+</details>
43
+
44
+<details>
45
+<summary>Configure Caching </summary>
46
+
47
+Caching ensures fast delivery to visitors. Without caching, a browser requests assets from the
48
+server each time a page loads instead of accessing them from a local or intermediary cache.
49
+
50
+To enable caching on your server, refer to the respective documentation:
51
+- [NGINX caching guide](https://www.nginx.com/blog/nginx-caching-guide/)
52
+- [Apache caching guide](https://httpd.apache.org/docs/2.4/caching.html)
53
+
54
+</details>
health/guides/web_log/web_slow.md
new
+54
@@ -0,0 +1,54 @@
1
+# web_slow
2
+
3
+**Web Server | Web log**
4
+
5
+When a client sends a request to your web server, there are many independent phases for this request
6
+that can introduce delay. Some of them may be:
7
+
8
+- DNS lookup
9
+- Establish a TCP connection
10
+- Perform a TLS handshake
11
+- The server to process the request
12
+- Transfer the actual content
13
+
14
+The Netdata Agent calculates the average HTTP response time over the last minute for your web server
15
+(NGINX, Apache). You receive this alert when your web server's average response time has increased.
16
+The alert is raised in warning when the average HTTP response time is twice as much. When the average HTTP response time is four times as much, you receive a critical alert.
17
+
18
+### Troubleshooting section
19
+
20
+The causes of a slow request response may vary. Some options you can explore are:
21
+
22
+<details>
23
+<summary>Your web server utilization is high </summary>
24
+
25
+This problem could be addressed on many levels:
26
+- Check if your host machine can handle the traffic: Check the CPU, memory and traffic utilization.
27
+ If this is not an issue:
28
+- Consider raising the resource limitations for your web server (for example add more worker processes).
29
+ Please consult your web server docs. If this also doesn't resolve the issue:
30
+- Set up an architecture with multiple web servers and load balancer to handle the traffic for your site.
31
+
32
+</details>
33
+
34
+<details>
35
+<summary>Optimize Databases </summary>
36
+
37
+The response speed is dependent on database optimization. As you first set up a website, the
38
+database responds quickly to queries. As time passes, the database accumulates information. The
39
+compilation results in massive amounts of stored data and might slow down response times.
40
+
41
+If you manage your database with MySQL, this blogpost proposes ways to [tune MySQL operations](https://www.cloudways.com/blog/mysql-performance-tuning/).
42
+</details>
43
+
44
+<details>
45
+<summary>Configure Caching </summary>
46
+
47
+Caching ensures fast delivery to visitors. Without caching, a browser requests assets from the
48
+server each time a page loads instead of accessing them from a local or intermediary cache.
49
+
50
+To enable caching on your server, refer to the respective documentation:
51
+- [NGINX caching guide](https://www.nginx.com/blog/nginx-caching-guide/)
52
+- [Apache caching guide](https://httpd.apache.org/docs/2.4/caching.html)
53
+
54
+</details>
health/guides/whoisquery/whoisquery_days_until_expiration.md
new
+19
@@ -0,0 +1,19 @@
1
+# whoisquery_days_until_expiration
2
+
3
+**Other | WHOIS**
4
+
5
+The Netdata Agent checks the time until the domain name registration expires. This alert indicates
6
+that the domain name registration expires soon. If you do not renew the domain it will be
7
+deactivated soon. Consider scheduling the renewal.
8
+
9
+This alert is raised into warning when your domain is about to expire in less than 90 days and in
10
+critical when it will expire in less than 30 days.
11
+
12
+### Troubleshooting section
13
+
14
+If you still want this domain you should renew it. If you need help renewing your domain name, you
15
+should immediately contact the registrar (or reseller) that provided your domain name registration
16
+services for possible renewal options available to you.
17
+
18
+To determine your current registrar of record, you may conduct a WHOIS search by visiting
19
+whois.icann.org. Registrar information can be found on the ICANN-Accredited Registrar list.
\ No newline at end of file
health/guides/windows/windows_10min_cpu_usage.md
new
+32
@@ -0,0 +1,32 @@
1
+# windows_10min_cpu_usage
2
+
3
+## Windows | CPU
4
+
5
+This alarm calculates the average of CPU utilization over a period of 10 minutes.
6
+
7
+It is raised into warning if the value exceeds 85%.
8
+If the average exceeds 95%, then the alert gets raised into critical.
9
+
10
+### Troubleshooting Section
11
+
12
+<details>
13
+<summary>Processes slowing down your CPU</summary>
14
+
15
+In Windows, you can open up the Task Manager from the menu or by pressing
16
+`ctrl`+`shift`+`esc`.
17
+
18
+- Under the processes tab, you can see a list of the processes currently running on the machine.
19
+ - To get a better picture of the main consumers, order them by their total CPU usage by
20
+ clicking the CPU column. That will sort the top main processes utilizing your CPU.
21
+
22
+
23
+- To get a more detailed look, click the "Performance" tab (next to the processes tab) and
24
+ then click on the bottom of the window "Open Resource Monitor".
25
+ - That will open up a window with a more detailed view on the processes.
26
+ - On the "Processes" table, look for the column "Average CPU". Clicking this will order the
27
+ processes again by CPU utilization.
28
+
29
+> It would be helpful to close any of the main consumer processes, but Netdata strongly suggests
30
+> knowing exactly what processes you are closing and being certain that they are not necessary to
31
+> your workflow or system.
32
+</details>
health/guides/x509check/x509check_days_until_expiration.md
new
+111
@@ -0,0 +1,111 @@
1
+# x509check_days_until_expiration
2
+
3
+**Certificates | x509 certificates**
4
+
5
+_An X.509 certificate is a digital certificate based on the widely accepted International
6
+Telecommunications Union (ITU) X.509 standard, which defines the format of public key
7
+infrastructure (PKI) certificates. They are used to manage identity and security in internet
8
+communications and computer networking. They are unobtrusive and ubiquitous, and we encounter them
9
+every day when using websites, mobile apps, online documents, and connected
10
+devices._ <sup>[1](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.) </sup>
11
+
12
+The Netdata Agent monitors the time until an X.509 certificate expires. This alert indicates that,
13
+the X.509 certificate will expire soon. Check more about
14
+the [x509 certificate monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/x509check).
15
+
16
+By default, this alert is triggered in warning state when your certificate has less than 14 days to expire and
17
+in critical state when it has less than 7 days to expire, but these levels are configurable.
18
+
19
+A certification authority (CA) is an entity that issues digital certificates. A digital certificate
20
+certifies the ownership of a public key by the named subject of the certificate. This allows
21
+others (relying parties) to rely upon signatures or on assertions made about the private key that
22
+corresponds to the certified public key. A CA acts as a trusted third party—trusted both by the
23
+subject (owner) of the certificate and by the party relying upon the certificate. The format of
24
+these certificates is specified by the X.509 or EMV standard.
25
+
26
+<details>
27
+<summary>Where and why we need X.509 certificates</summary>
28
+
29
+The following provides a comprehensive explanation from the sectigo's website: <sup> [1](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.) </sup>
30
+
31
+
32
+Common Applications of X.509 Public Key Infrastructure Many internet protocols rely on X.509, and
33
+there are many applications of the PKI technology that are used every day, including Web server
34
+security, digital signatures and document signing, and digital identities.
35
+
36
+- **Web Server Security with TLS/SSL Certificates:**
37
+ PKI is the basis for the secure sockets layer (SSL)
38
+ and transport layer security (TLS) protocols that are the foundation of HTTPS secure browser
39
+ connections. Without SSL certificates or TLS to establish secure connections, cybercriminals could
40
+ exploit the Internet or other IP networks using a variety of attack vectors, such as
41
+ man-in-the-middle attacks, to intercept messages and access their contents.
42
+
43
+- **Digital Signatures and Document Signing:**
44
+ In addition to being used to secure messages, PKI-based certificates can be used for digital
45
+ signatures and document signing. Digital signatures are a specific type of electronic signature
46
+ that leverages PKI to authenticate the identity of the signer and the integrity of the signature
47
+ and the document. Digital signatures cannot be altered or duplicated in any way, as the signature
48
+ is created by generating a hash, which is encrypted using a sender's private key. This
49
+ cryptographic verification mathematically binds the signature to the original message to ensure
50
+ that the sender is authenticated and the message itself has not been altered.
51
+
52
+- **Code Signing:**
53
+ Code Signing enables application developers to add a layer of assurance by digitally signing
54
+ applications, drivers, and software programs so that end users can verify that a third party has
55
+ not altered or compromised the code they receive. To verify the code is safe and trusted, these
56
+ digital certificates include the software developer's signature, the company name, and
57
+ timestamping.
58
+
59
+- **Email Certificates:**
60
+ S/MIME certificates validate email senders and encrypt email contents to protect against
61
+ increasingly sophisticated social engineering and spear phishing attacks. By encrypting/decrypting
62
+ email messages and attachments and by validating identity, S/MIME email certificates assure users
63
+ that emails are authentic and unmodified.
64
+
65
+- **SSH Keys:**
66
+ SSH keys are a form of X.509 certificate that provides a secure access credential used in the
67
+ Secure Shell (SSH) protocol. As the SSH protocol is widely used for communication in cloud
68
+ services, network environments, file transfer tools, and configuration management tools, most
69
+ organizations use SSH keys to authenticate identity and protect those services from unintended use
70
+ or malicious attacks. SSH keys not only improve security, but also enable the automation of
71
+ connected processes, single sign-on (SSO), and identity and access management at the scale that
72
+ today's businesses require.
73
+
74
+- **Digital Identities:**
75
+ X.509 digital certificates also provide effective digital identity authentication. As data and
76
+ applications expand beyond traditional networks to mobile devices, public clouds, private clouds,
77
+ and Internet of Things devices, securing identities becomes more important than ever. And digital
78
+ identities don't have to be restricted to devices; they can also be used to authenticate people,
79
+ data, or applications. Digital identity certificates based on this standard enable organizations
80
+ to improve security by replacing passwords, which attackers have become increasingly adept at
81
+ stealing.
82
+
83
+</details>
84
+
85
+<details>
86
+<summary>Popular CAs </summary>
87
+
88
+1. https://letsencrypt.org/
89
+2. https://securitycloud.symantec.com/cc/landing
90
+3. https://www.geotrust.com/
91
+4. https://sectigo.com/
92
+5. https://www.digicert.com/
93
+
94
+</details>
95
+
96
+<details>
97
+<summary>References and source </summary>
98
+
99
+1. [X.509 explained](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.)
100
+
101
+</details>
102
+
103
+
104
+### Troubleshooting section
105
+
106
+Anyone can issue an X.509 certificate, and a X.509 certificate may or may not have an expiration date.
107
+In most cases the certificates which are issued by a CA have a validity period. In order to
108
+persist your certificate's validity, you must either renew (or re-key) it. If your certificate is
109
+issued by a CA, you must manage it from your CA.
110
+
111
+
health/guides/x509check/x509check_revocation_status.md
new
+120
@@ -0,0 +1,120 @@
1
+# x509check_revocation_status
2
+
3
+**Certificates | x509 certificates**
4
+
5
+_An X.509 certificate is a digital certificate based on the widely accepted International
6
+Telecommunications Union (ITU) X.509 standard, which defines the format of public key
7
+infrastructure (PKI) certificates. They are used to manage identity and security in internet
8
+communications and computer networking. They are unobtrusive and ubiquitous, and we encounter them
9
+every day when using websites, mobile apps, online documents, and connected
10
+devices._ <sup>[1](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.) </sup>
11
+
12
+The certificate is also a confirmation or validation by an trusted Certificate Authority (CA) that the public
13
+key contained in the certificate belongs to the person, organization, server or other entity noted in the
14
+certificate.
15
+
16
+For many reasons, primary, security reasons, we may want to revoke the validity of an X.509
17
+certificate. In a nutshell, an X.509 certificate should be revoked when:
18
+
19
+- Encryption keys of the certificate have been compromised.
20
+- Errors occur within an issued certificate.
21
+- We want to change the usage of the certificate.
22
+- Certificate owner is no longer deemed trusted.
23
+
24
+The Netdata Agent checks the X.509 certificate revocation status (0: revoked, 1: valid). This alert
25
+indicates that the X.509 certificate has been revoked. Check more about
26
+the [x509 certificate monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/x509check)
27
+
28
+This alert is triggered in critical state when the X.509 certificate is available and not valid.
29
+
30
+<details>
31
+<summary>Where and why we need X.509 certificates</summary>
32
+
33
+The following provides a comprehensive explanation from the sectigo's
34
+website <sup> [1](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.) </sup>
35
+
36
+Common Applications of X.509 Public Key Infrastructure Many internet protocols rely on X.509, and
37
+there are many applications of the PKI technology that are used every day, including Web server
38
+security, digital signatures and document signing, and digital identities.
39
+
40
+- **Web Server Security with TLS/SSL Certificates:**
41
+ PKI is the basis for the secure sockets layer (SSL)
42
+ and transport layer security (TLS) protocols that are the foundation of HTTPS secure browser
43
+ connections. Without SSL certificates or TLS to establish secure connections, cybercriminals could
44
+ exploit the Internet or other IP networks using a variety of attack vectors, such as
45
+ man-in-the-middle attacks, to intercept messages and access their contents.
46
+
47
+- **Digital Signatures and Document Signing:**
48
+ In addition to being used to secure messages, PKI-based certificates can be used for digital
49
+ signatures and document signing. Digital signatures are a specific type of electronic signature
50
+ that leverages PKI to authenticate the identity of the signer and the integrity of the signature
51
+ and the document. Digital signatures cannot be altered or duplicated in any way, as the signature
52
+ is created by generating a hash, which is encrypted using a sender's private key. This
53
+ cryptographic verification mathematically binds the signature to the original message to ensure
54
+ that the sender is authenticated and the message itself has not been altered.
55
+
56
+- **Code Signing:**
57
+ Code Signing enables application developers to add a layer of assurance by digitally signing
58
+ applications, drivers, and software programs so that end users can verify that a third party has
59
+ not altered or compromised the code they receive. To verify the code is safe and trusted, these
60
+ digital certificates include the software developer's signature, the company name, and
61
+ timestamping.
62
+
63
+- **Email Certificates:**
64
+ S/MIME certificates validate email senders and encrypt email contents to protect against
65
+ increasingly sophisticated social engineering and spear phishing attacks. By encrypting/decrypting
66
+ email messages and attachments and by validating identity, S/MIME email certificates assure users
67
+ that emails are authentic and unmodified.
68
+
69
+- **SSH Keys:**
70
+ SSH keys are a form of X.509 certificate that provides a secure access credential used in the
71
+ Secure Shell (SSH) protocol. As the SSH protocol is widely used for communication in cloud
72
+ services, network environments, file transfer tools, and configuration management tools, most
73
+ organizations use SSH keys to authenticate identity and protect those services from unintended use
74
+ or malicious attacks. SSH keys not only improve security, but also enable the automation of
75
+ connected processes, single sign-on (SSO), and identity and access management at the scale that
76
+ today's businesses require.
77
+
78
+- **Digital Identities:**
79
+ X.509 digital certificates also provide effective digital identity authentication. As data and
80
+ applications expand beyond traditional networks to mobile devices, public clouds, private clouds,
81
+ and Internet of Things devices, securing identities becomes more important than ever. And digital
82
+ identities don't have to be restricted to devices; they can also be used to authenticate people,
83
+ data, or applications. Digital identity certificates based on this standard enable organizations
84
+ to improve security by replacing passwords, which attackers have become increasingly adept at
85
+ stealing.
86
+
87
+</details>
88
+
89
+<details>
90
+<summary>See more about the Certificate Authorities</summary>
91
+
92
+A certification authority (CA) is an entity that issues digital certificates. A digital certificate
93
+certifies the ownership of a public key by the named subject of the certificate. This allows
94
+others (relying parties) to rely upon signatures or on assertions made about the private key that
95
+corresponds to the certified public key. A CA acts as a trusted third party—trusted both by the
96
+subject (owner) of the certificate and by the party relying upon the certificate. The format of
97
+these certificates is specified by the X.509 or EMV standard.
98
+
99
+**Popular CAs**
100
+
101
+ 1. https://letsencrypt.org/
102
+ 2. https://securitycloud.symantec.com/cc/landing
103
+ 3. https://www.geotrust.com/
104
+ 4. https://sectigo.com/
105
+ 5. https://www.digicert.com/
106
+
107
+</details>
108
+
109
+<details>
110
+<summary>References and source </summary>
111
+
112
+1. [X.509 explained](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.)
113
+
114
+</details>
115
+
116
+
117
+### Troubleshooting section
118
+
119
+A revocation of a certificate is irreversible. That means that this certificate is no longer
120
+useful. You must stop using it in any way.
health/guides/zfs/zfs_memory_throttle.md
new
+75
@@ -0,0 +1,75 @@
1
+# zfs_memory_throttle
2
+
3
+## OS: Linux | FreeBSD
4
+
5
+This alert presents the number of times ZFS had to limit the Adaptive Replacement Cache (ARC) growth in the last 10 minutes.
6
+
7
+The alert is raised to a warning state when the metric starts counting (when it is greater than 0).
8
+
9
+<details>
10
+<summary>Linux: What is the ARC?</summary>
11
+
12
+The ARC stores the most recently used, and most frequently used data within RAM.
13
+Having a large ARC can take up a lot of RAM, but it will decrease as other
14
+applications need it. ARC can be set to customized optimal settings for your system.
15
+
16
+ARC uses varying portions of the most recently used and the most often used data by
17
+allocating more space to one or the other whenever a cold hit occurs. A cold hit occurs when
18
+some data is requested that was previously cached, but has already been pushed out to allow
19
+the ARC to store new data. ZFS keeps track of what data was stored in the cache after it is
20
+removed in order to enable the recognition of cold hits. As new data comes in, data that
21
+hasn't been used in a while, or that has not been used as much as the new data, will be pushed
22
+out.
23
+
24
+The more RAM your system has the better, as it will just give you enhanced read performance. There
25
+will be physical and cost limitations to adding more ARC due to motherboard RAM slots and budget
26
+constraints.<sup>[1](https://www.45drives.com/community/articles/zfs-caching/) </sup>
27
+
28
+</details>
29
+
30
+<br>
31
+
32
+<details>
33
+<summary>FreeBSD: What is the ARC?</summary>
34
+
35
+The ARC functions by storing the most recently used, and most frequently used data within RAM.
36
+Having a large ARC can take up a lot of RAM, but it will give it up as other applications need
37
+it and can be set to whatever you think is optimal for your system.
38
+
39
+**Single Copy ARC**
40
+OpenZFS caches disk blocks in-memory in the adaptive replacement cache (ARC). Originally when
41
+the same disk block was accessed from different clones it was cached multiple times (one for
42
+each clone accessing the block) in case a clone planned to modify the block. OpenZFS caches
43
+at most one copy of every block unless a clone is actually modifying the block.
44
+
45
+**ARC Shouldn't Cache Freed Blocks**
46
+Originally cached blocks in the ARC remained cached until they were evicted due to memory
47
+pressure, even if the underlying disk block was freed. In some workloads these freed blocks
48
+were so frequently accessed before they were freed that the ARC continued to cache them while
49
+evicting blocks which had not been freed yet. Since freed blocks could never be accessed
50
+again continuing to cache them was unnecessary. In OpenZFS ARC blocks are evicted immediately
51
+when their underlying data blocks are freed.<sup>[2](https://openzfs.org/w/index.php?title=Features&mobileaction=toggle_view_mobile#Single_Copy_ARC)
52
+</sup>
53
+
54
+</details>
55
+
56
+<br>
57
+
58
+<details>
59
+<summary>References and Sources</summary>
60
+
61
+1. [Linux: ZFS Caching](https://www.45drives.com/community/articles/zfs-caching/)
62
+2. [FreeBSD: OpenZFS documentation](https://openzfs.org/w/index.php?title=Features&mobileaction=toggle_view_mobile#Single_Copy_ARC)
63
+ </details>
64
+
65
+### Troubleshooting Section
66
+
67
+<details>
68
+<summary>Linux | FreeBSD: Increase your RAM capacity and effectively increase ARC size</summary>
69
+
70
+ZFS will throttle the ARC growth as the system needs more RAM for other tasks.
71
+If you are experiencing a lot of throttling, then you should consider increasing your RAM capacity.
72
+If the ARC size needs to be limited, then the read performance of the system will drop and cold hits
73
+are more likely to happen.
74
+
75
+</details>
health/guides/zfs/zfs_pool_state_crit.md
new
+79
@@ -0,0 +1,79 @@
1
+# zfs_pool_state_crit
2
+
3
+## OS: Any
4
+
5
+ZFS is a local file system and logical volume manager created by Sun Microsystems Inc. to direct and
6
+control the placement, storage, and retrieval of data in enterprise-class computing systems. ZFS is
7
+scalable, suitable for high storage capacities, and includes extensive protection against data corruption.
8
+
9
+The Netdata Agent monitors the state of the ZFS pool. Receiving this alert means that the ZFS pool
10
+is faulted or unavailable.
11
+
12
+<details>
13
+<summary>ZFS pool health status</summary>
14
+
15
+The ZFS pool health status as described in the Oracle's
16
+website <sup>[1](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html) </sup>
17
+
18
+ZFS provides an integrated method of examining pool and device health. The health of a pool is
19
+determined from the state of all its devices. This state information is displayed by using the `zpool
20
+status` command. In addition, potential pool and device failures are reported by fmd, displayed on
21
+the system console, and logged in the /var/adm/messages file.
22
+
23
+Each device can fall into one of the following states:
24
+
25
+- ONLINE, the device or virtual device is in normal working order. Although some transient errors
26
+ might still occur, the device is otherwise in working order.
27
+
28
+- DEGRADED, the virtual device has experienced a failure but can still function. This state is most
29
+ common when a mirror or RAID-Z device has lost one or more constituent devices. The fault
30
+ tolerance of the pool might be compromised, as a subsequent fault in another device might be
31
+ unrecoverable.
32
+
33
+- FAULTED, the device or virtual device is completely inaccessible. This status typically indicates
34
+ total failure of the device, such that ZFS is incapable of sending data to it or receiving data
35
+ from it. If a top-level virtual device is in this state, then the pool is completely inaccessible.
36
+
37
+- OFFLINE, the device has been explicitly taken offline by the administrator.
38
+
39
+- UNAVAIL, the device or virtual device cannot be opened. In some cases, pools with UNAVAIL devices
40
+ appear in DEGRADED mode. If a top-level virtual device is UNAVAIL, then nothing in the pool can be
41
+ accessed.
42
+
43
+- REMOVED, the device was physically removed while the system was running. Device removal detection
44
+ is hardware-dependent and might not be supported on all platforms.
45
+
46
+The health of a pool is determined from the health of all its top-level virtual devices. If all
47
+virtual devices are ONLINE, then the pool is also ONLINE. If any one of the virtual devices is
48
+DEGRADED or UNAVAIL, then the pool is also DEGRADED. If a top-level virtual device is FAULTED or
49
+OFFLINE, then the pool is also FAULTED. A pool in the FAULTED state is completely inaccessible. No
50
+data can be recovered until the necessary devices are attached or repaired. A pool in the DEGRADED
51
+state continues to run, but you might not achieve the same level of data redundancy or data
52
+throughput than if the pool were online. <sup>[1](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html) </sup>
53
+
54
+</details>
55
+
56
+<details>
57
+<summary>References and source</summary>
58
+
59
+1. [Determining the Health Status of ZFS Storage Pools](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html)
60
+1. [Chapter 11, Oracle Solaris ZFS Troubleshooting and Pool Recovery](https://docs.oracle.com/cd/E53394_01/html/E54801/gavwg.html)
61
+1. [ZFS on FreeBSD documentation](https://docs.freebsd.org/en/books/handbook/zfs/)
62
+1. [OpenZFS documentation](https://openzfs.github.io/openzfs-docs/)
63
+
64
+</details>
65
+
66
+### Troubleshooting section
67
+
68
+<details>
69
+<summary>Migrate ZFS Storage Pools</summary>
70
+
71
+If the state of the ZFS pool is UNAVAIL, then you should consider migrating your ZFS pool. To do so follow the
72
+workflow at the [official troubleshooting section in Oracle's website](https://docs.oracle.com/cd/E53394_01/html/E54801/gbchy.html#scrolltoc).
73
+
74
+In this workflow, there are no notable changes in the commands for FreeBSD.
75
+<sup>[3](https://docs.freebsd.org/en/books/handbook/zfs/) </sup> or ZFS on
76
+linux <sup>[4](https://openzfs.github.io/openzfs-docs/) </sup>, for completeness you can refer to
77
+individual guides.
78
+
79
+</details>
health/guides/zfs/zfs_pool_state_warn.md
new
+113
@@ -0,0 +1,113 @@
1
+# zfs_pool_state_warn
2
+
3
+## OS: Any
4
+
5
+ZFS is a local file system and logical volume manager created by Sun Microsystems Inc. to direct and
6
+control the placement, storage, and retrieval of data in enterprise-class computing systems. ZFS is
7
+scalable, suitable for high storage capacities, and includes extensive protection against data corruption.
8
+
9
+The Netdata Agent monitors the state of the ZFS pool. Receiving this alert means that the ZFS pool
10
+is degraded.
11
+
12
+<details>
13
+<summary>See more on ZFS pool health status</summary>
14
+
15
+The ZFS pool health status as described in the Oracle's
16
+website <sup>[1](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html) </sup>
17
+
18
+ZFS provides an integrated method of examining pool and device health. The health of a pool is
19
+determined from the state of all its devices. This state information is displayed by using the zpool
20
+status command. In addition, potential pool and device failures are reported by fmd, displayed on
21
+the system console, and logged in the /var/adm/messages file.
22
+
23
+Each device can fall into one of the following states:
24
+
25
+- ONLINE, the device or virtual device is in normal working order. Although some transient errors
26
+ might still occur, the device is otherwise in working order.
27
+
28
+- DEGRADED, the virtual device has experienced a failure but can still function. This state is most
29
+ common when a mirror or RAID-Z device has lost one or more constituent devices. The fault
30
+ tolerance of the pool might be compromised, as a subsequent fault in another device might be
31
+ unrecoverable.
32
+
33
+- FAULTED, the device or virtual device is completely inaccessible. This status typically indicates
34
+ total failure of the device, such that ZFS is incapable of sending data to it or receiving data
35
+ from it. If a top-level virtual device is in this state, then the pool is completely inaccessible.
36
+
37
+- OFFLINE, the device has been explicitly taken offline by the administrator.
38
+
39
+- UNAVAIL, the device or virtual device cannot be opened. In some cases, pools with UNAVAIL devices
40
+ appear in DEGRADED mode. If a top-level virtual device is UNAVAIL, then nothing in the pool can be
41
+ accessed.
42
+
43
+- REMOVED, the device was physically removed while the system was running. Device removal detection
44
+ is hardware-dependent and might not be supported on all platforms.
45
+
46
+The health of a pool is determined from the health of all its top-level virtual devices. If all
47
+virtual devices are ONLINE, then the pool is also ONLINE. If any one of the virtual devices is
48
+DEGRADED or UNAVAIL, then the pool is also DEGRADED. If a top-level virtual device is FAULTED or
49
+OFFLINE, then the pool is also FAULTED. A pool in the FAULTED state is completely inaccessible. No
50
+data can be recovered until the necessary devices are attached or repaired. A pool in the DEGRADED
51
+state continues to run, but you might not achieve the same level of data redundancy or data
52
+throughput than if the pool were online. <sup>[1](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html) </sup>
53
+
54
+</details>
55
+
56
+
57
+<details>
58
+<summary>References and source</summary>
59
+
60
+1. [Determining the Health Status of ZFS Storage Pools](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html)
61
+1. [Chapter 11, Oracle Solaris ZFS Troubleshooting and Pool Recovery](https://docs.oracle.com/cd/E53394_01/html/E54801/gavwg.html)
62
+1. [AlchemyCS blogspot](https://alchemycs.com/2019/05/how-to-force-zfs-to-replace-a-failed-drive-in-place/)
63
+
64
+</details>
65
+
66
+### Troubleshooting section
67
+
68
+<details>
69
+<summary>Replace a failed drive</summary>
70
+
71
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
72
+steps. Netdata is not liable for any loss or corruption of any data, database, or software.
73
+
74
+Based on a nice guide from the Alchemycs blogspot.
75
+
76
+1. Check your zpool status
77
+
78
+```
79
+root@netdata # zpool status my_pool
80
+ pool: my_pool
81
+ state: DEGRADED
82
+status: One or more devices could not be used because the label is missing or
83
+ invalid. Sufficient replicas exist for the pool to continue
84
+ functioning in a degraded state.
85
+action: Replace the device using 'zpool replace'.
86
+ see: http://zfsonlinux.org/msg/ZFS-8000-4J
87
+ scan: scrub repaired 0B in 0h0m with 0 errors on Sun May 12 00:24:51 2019
88
+config:
89
+
90
+ NAME STATE READ WRITE CKSUM
91
+ my_pool DEGRADED 0 0 0
92
+ mirror-0 DEGRADED 0 0 0
93
+ 52009894889112747750 UNAVAIL 0 0 0 was /dev/sdm5
94
+ sdb5 ONLINE 0 0 0errors: No known data errors
95
+```
96
+
97
+1. Find the UUIDs (for GPT) of the faulty dev (the UNAVAIL) and the new disk you want to add.
98
+ Use `blkid` for Linux or `geom` utility for FreeBSD
99
+
100
+
101
+1. Offline the UNAVAIL drive
102
+
103
+```
104
+root@netdata # zpool offline my_pool /dev/disk/by-uuid/{UUID_BAD_DRIVE}
105
+```
106
+
107
+1. Replace it in place
108
+
109
+```
110
+root@netdata # zpool replace -f my_pool /dev/disk/by-uuid/{UUID_OLD_DRIVE} /dev/disk/by-uuid/{UUID_NEW_DRIVE}
111
+```
112
+
113
+</details>