Import alert guides from Netdata Assistant (#16355)
Ralph Meijer committed
Nov 8, 2023 at 08:47 UTC
e74127c634d53ac05aed7566a99a472b8b77e5fa
479 files changed
+12248
-12217
health/guides/10min_cpu_iowait.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alarm calculates the average time of `iowait` through 10 minute interval periods. `iowait` is the percentage of time where there has been at least one I/O request in progress while the CPU has been idle.
4
+
5
+I/O -at a process level- is the use of the read and write services, such as reading data from a physical drive.
6
+
7
+It's important to note that during the time a process waits on I/O, the system can schedule other processes, but `iowait` is measured specifically while the CPU is idle.
8
+
9
+A common example of when this alert might be triggered would be when your CPU requests some data and the device responsible for it can't deliver it fast enough. As a result the CPU (in the next clock interrupt) is idle, so you
10
+encounter `iowait`. If this persists for some time and the average from the metrics we gather exceeds the value that is being checked in the `.conf` file, then the alert is raised because the CPU is being bottlenecked by your system’s disks.
11
+
12
+### Troubleshooting Section
13
+
14
+- Check for main I/O related processes and hardware issues
15
+
16
+Generally, this issue is caused by having slow hard drives that cannot keep up with the speed of your CPU. You can see the percentage of `iowait` by going to your node on Netdata Cloud and clicking the `iowait` dimension under the Total CPU Utilization chart.
17
+
18
+- You can use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds)
19
+
20
+The `procs` column, shows the number of processes blocked waiting for I/O to complete.
21
+
22
+After that, you can use `ps` and specifically `ps -eo s,user,cmd | grep ^[D]` to fetch the processes that their state code starts with `D` which means uninterruptible sleep (usually IO).
23
+
24
+- It could be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
25
+
26
+- If you see that you don't have a lot of processes that you can terminate (or you need them for your workflow), then you would have to upgrade your system’s drives; if you have an HDD, upgrading to an SSD or an NVME drive would make a great impact on this metric.
27
+
28
+### Are you operating a database?
29
+
30
+In a database environment, you would want to optimize your operations. Check for potential inserts on large data sets, keeping in mind that `write` operations take more time than `read`. You should also search for
31
+ complex requests, like large joins and queries over a big data set. These can introduce `iowait` and need to be optimized.
32
+
33
+### Useful resources
34
+
35
+- [What exactly is "iowait"?](https://serverfault.com/questions/12679/can-anyone-explain-precisely-what-iowait-is)
36
+
health/guides/10min_cpu_usage.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alarm calculates an average on CPU utilization over a period of 10 minutes, **excluding** `iowait`, `nice` and `steal` values.
4
+
5
+*Note that on FreeBSD, the alert excludes only `nice`.
6
+
7
+`iowait` is the percentage of time the CPU waits on a disk for an I/O; it happens when the former is getting bottlenecked by the latter. At this point the CPU is being idle, waiting only on the I/O.
8
+
9
+`nice` value of a processor is the time it has spent on running low priority processes. Low priority processes are those with a 'nice' value greater than 0 (on UNIX-like systems, a higher ‘nice’ value indicates a lower priority).
10
+
11
+`steal`, in a virtual machine, is the percentage of time that particular virtual CPU has to wait for an available host CPU to run on. If this metric goes up, it means that your VM is not getting the processing power it needs.
12
+
13
+### Troubleshooting Section
14
+
15
+- Processes slowing down your CPU
16
+
17
+There are two primary cases in which this alarm is raised, and determining which applies to you requires understanding your own scenario.
18
+
19
+1. High CPU utilization with high `nice` value means that the system is running through all the low priority processes, and if some high priority process needs CPU time, it can get it at any time.
20
+2. High CPU utilization with low `nice` value means that the CPU is used on high priority processes and new ones will not be able to take CPU time, and they will have to wait.
21
+
22
+The latter scenario is worth investigating if there is a process slowing down your CPU. We suggest you go to your node on Netdata Cloud and click the `nice` dimension under the `Total CPU Utilization` chart to see the value. You can then check per process CPU usage using `top`:
23
+
24
+If you're using Linux:
25
+```
26
+root@netdata~ # top -o +%CPU -i
27
+```
28
+
29
+And for FreeBSD:
30
+```
31
+root@netdata~ # top -o cpu -I
32
+```
33
+
34
+Here, you can see which processes are the main cpu consumers on the `CPU` column.
35
+
36
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
37
+
health/guides/10min_dbengine_global_flushing_errors.md
new
+13
@@ -0,0 +1,13 @@
1
+### Understand the alert
2
+
3
+The Database Engine works like a traditional database. It dedicates a certain amount of RAM to data caching and indexing, while the rest of the data resides compressed on disk. Unlike other memory modes, the amount of historical metrics stored is based on the amount of disk space you allocate and the effective compression ratio, not a fixed number of metrics collected.
4
+
5
+By using both RAM and disk space, the database engine allows for long-term storage of per-second metrics inside of the Netdata Agent itself.
6
+
7
+Netdata monitors the number of pages deleted due to failure to flush data to disk in the last 10 minutes. In this situation some metric data was dropped to unblock data collection. To remedy this issue, reduce disk load or use
8
+faster disks. This alert is triggered in critical state when the number deleted pages is greater than 0.
9
+
10
+### Useful resources
11
+
12
+[Read more about Netdata DB engine](https://learn.netdata.cloud/docs/agent/database/engine)
13
+
health/guides/10min_dbengine_global_flushing_warnings.md
new
+15
@@ -0,0 +1,15 @@
1
+### Understand the alert
2
+
3
+The Database Engine works like a traditional database. It dedicates a certain amount of RAM to data caching and indexing, while the rest of the data resides compressed on disk. Unlike other memory modes, the amount of historical metrics stored is based on the amount of disk space you allocate and the effective compression ratio, not a fixed number
4
+of metrics collected.
5
+
6
+By using both RAM and disk space, the database engine allows for long-term storage of per-second metrics inside of the Netdata Agent itself.
7
+
8
+Netdata monitors the number of times when `dbengine` dirty pages were over 50% of the instance page cache in the last 10 minutes. In this situation, the metric data are at risk of not being stored in the database. To remedy this issue, reduce disk load or use faster disks.
9
+
10
+This alert is triggered in warn state when the number of `dbengine` dirty pages which were over 50% of the instance is greater than 0.
11
+
12
+### Useful resources
13
+
14
+[Read more about Netdata DB engine](https://learn.netdata.cloud/docs/agent/database/engine)
15
+
health/guides/10min_dbengine_global_fs_errors.md
new
+14
@@ -0,0 +1,14 @@
1
+### Understand the alert
2
+
3
+The Database Engine works like a traditional database. It dedicates a certain amount of RAM to data caching and indexing, while the rest of the data resides compressed on disk. Unlike other memory modes, the amount of historical metrics stored is based on the amount of disk space you allocate and the effective compression ratio, not a fixed number of metrics collected.
4
+
5
+By using both RAM and disk space, the database engine allows for long-term storage of per-second metrics inside of the Netdata agent itself.
6
+
7
+Netdata monitors the number of filesystem errors in the last 10 minutes. The Dbengine is experiencing filesystem errors (too many open files, wrong permissions, etc.)
8
+
9
+This alert is triggered in warning state when the number of filesystem errors is greater than 0.
10
+
11
+### Useful resources
12
+
13
+[Read more about Netdata DB engine](https://learn.netdata.cloud/docs/agent/database/engine)
14
+
health/guides/10min_dbengine_global_io_errors.md
new
+14
@@ -0,0 +1,14 @@
1
+### Understand the alert
2
+
3
+The Database Engine works like a traditional database. It dedicates a certain amount of RAM to data caching and indexing, while the rest of the data resides compressed on disk. Unlike other memory modes, the amount of historical metrics stored is based on the amount of disk space you allocate and the effective compression ratio, not a fixed number of metrics collected.
4
+
5
+By using both RAM and disk space, the database engine allows for long-term storage of per-second metrics inside of the Netdata Agent itself.
6
+
7
+The Netdata Agent monitors the number of IO errors in the last 10 minutes. The dbengine is experiencing I/O errors (CRC errors, out of space, bad disk, etc.).
8
+
9
+This alert is triggered in critical state when the number of IO errors is greater that 0.
10
+
11
+### Useful resources
12
+
13
+[Read more about Netdata DB engine](https://learn.netdata.cloud/docs/agent/database/engine)
14
+
health/guides/10min_disk_backlog.md
new
+10
@@ -0,0 +1,10 @@
1
+### Understand the alert
2
+
3
+This alert presents the average backlog size of the disk raising this alarm over the last 10 minutes.
4
+
5
+This alert is escalated to warning when the metric exceeds the size of 5000.
6
+
7
+### What is "disk backlog"?
8
+
9
+Backlog is an indication of the duration of pending disk operations. On every I/O event the system is multiplying the time spent doing I/O since the last update of this field with the number of pending operations. While not accurate, this metric can provide an indication of the expected completion time of the operations in progress.
10
+
health/guides/10min_disk_utilization.md
new
+28
@@ -0,0 +1,28 @@
1
+### Understand the alert
2
+
3
+This alert presents the average percentage of time the disk was busy over the last 10 minutes. If you receive this it indicates high disk load and that the disk spent most of the time servicing
4
+read or write requests.
5
+
6
+This alert is triggered in a warning state when the metric exceeds 98%.
7
+
8
+This metric is the same as the %util column on the command `iostat -x`.
9
+
10
+### Troubleshoot the alert
11
+
12
+- Check per-process disk usage to find the top consumers (If you got this alert for a device serving requests in parallel, you can ignore it)
13
+
14
+On Linux use `iotop` to see which processes are the main Disk I/O consumers on the `IO` column.
15
+ ```
16
+ sudo iotop
17
+ ```
18
+ Using this, you can see which processes are the main Disk I/O consumers on the `IO` column.
19
+
20
+On FreeBSD use `top`
21
+ ```
22
+ top -m io -o total
23
+ ```
24
+### Useful resources
25
+
26
+1. [Two traps in iostat: %util and svctm](https://brooker.co.za/blog/2014/07/04/iostat-pct.html)
27
+
28
+2. `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it, then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
health/guides/10min_fifo_errors.md
new
+42
@@ -0,0 +1,42 @@
1
+### Understand the alert
2
+
3
+Between the IP stack and the Network Interface Controller (NIC) lies the driver queue. This queue is typically implemented as a FIFO ring buffer into the memory space allocated by the driver. The NIC receive frames and place them into memory as skb_buff data structures (SocKet Buffer). We can have queues (ingress queues) and transmitted (egress queues) but these queues do not contain any actual packet data. Each queue has a pointer to the devices associated with it, and to the skb_buff data structures that store the ingress/egress packets. The number of frames this queue can handle is limited. Queues fill up when an interface receives packets faster than kernel can process them.
4
+
5
+Netdata monitors the number of FIFO errors (number of times an overflow occurs in the ring buffer) for a specific network interface in the last 10 minutes. This alarm is triggered when the NIC is not able to handle the peak load of incoming/outgoing packets with the current ring buffer size.
6
+
7
+Not all NICs support FIFO queue operations.
8
+
9
+### More about SKB
10
+
11
+The SocKet Buffer (SKB), is the most fundamental data structure in the Linux networking code. Every packet sent or received is handled using this data structure. This is a large struct containing all the control information required for the packet (datagram, cell, etc).
12
+
13
+The struct sk_buff has the following fields to point to the specific network layer headers:
14
+
15
+- transport_header (previously called h) – This field points to layer 4, the transport layer (and can include tcp header or udp header or
16
+ icmp header, and more)
17
+
18
+- network_header (previously called nh) – This field points to layer 3, the network layer (and can include ip header or ipv6 header or arp
19
+ header).
20
+
21
+- mac_header (previously called mac) – This field points to layer 2, the link layer.
22
+
23
+- skb_network_header(skb), skb_transport_header(skb) and skb_mac_header(skb) - These return pointer to the header.
24
+
25
+### Troubleshoot the alert
26
+
27
+- Update the ring buffer size
28
+
29
+1. To view the maximum RX ring buffer size:
30
+
31
+ ```
32
+ ethtool -g enp1s0
33
+ ```
34
+
35
+2. If the values in the Pre-set maximums section are higher than in the Current hardware settings section, increase RX (or TX) ring buffer:
36
+
37
+ ```
38
+ enp1s0 rx 4080
39
+ ```
40
+
41
+3. Verify the change to make sure that you no longer receive the alarm when running the same workload. To make this permanently, you must consult your distribution guides.
42
+
health/guides/10min_netisr_backlog_exceeded.md
new
+56
@@ -0,0 +1,56 @@
1
+### Understand the alert
2
+
3
+The `10min_netisr_backlog_exceeded` alert occurs when the `netisr_maxqlen` queue within FreeBSD's network kernel dispatch service reaches its maximum capacity. This queue stores packets received by interfaces and waiting to be processed by the destined subsystems or userland applications. When the queue is full, the system drops new packets. This alert indicates that the average number of dropped packets in the last minute has exceeded the netisr queue length.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Increase the netisr_maxqlen value**
8
+
9
+ a. Check the current value:
10
+
11
+ ```
12
+ root@netdata~ # sysctl net.route.netisr_maxqlen
13
+ net.route.netisr_maxqlen: 256
14
+ ```
15
+
16
+ b. Increase the value by a factor of 4:
17
+
18
+ ```
19
+ root@netdata~ # sysctl -w net.route.netisr_maxqlen=1024
20
+ ```
21
+
22
+ c. Verify the change and test with the same workload that triggered the alarm originally:
23
+
24
+ ```
25
+ root@netdata~ # sysctl net.route.netisr_maxqlen
26
+ net.route.netisr_maxqlen: 1024
27
+ ```
28
+
29
+ d. If the change works for your system, make it permanent by adding this entry, `net.route.netisr_maxqlen=1024`, to `/etc/sysctl.conf`.
30
+
31
+ e. Reload the sysctl settings:
32
+
33
+ ```
34
+ root@netdata~ # /etc/rc.d/sysctl reload
35
+ ```
36
+
37
+2. **Monitor the system**
38
+
39
+ After increasing the `netisr_maxqlen` value, continue to monitor your system's dropped packet statistics using tools like `netstat` to determine if the queue backlog situation has improved. If you are still experiencing high packet drop rates, you may need to further increase the `netisr_maxqlen` value, or explore other optimizations for your networking stack.
40
+
41
+3. **Check hardware and system resources**
42
+
43
+ In some cases, overloaded or underpowered hardware may cause issues with packet processing. Ensure that your hardware (network cards, switches, routers, etc.) is performing optimally, and that your system has enough CPU and RAM resources to handle the traffic load.
44
+
45
+4. **Network traffic analysis**
46
+
47
+ Analyze your network traffic using tools like `tcpdump`, `iftop`, or `iptraf` to identify specific traffic patterns or types causing the backlog issue. This analysis can help you optimize your network infrastructure or take actions to reduce unnecessary traffic.
48
+
49
+5. **Update FreeBSD version**
50
+
51
+ Ensure that your FreeBSD system is up to date, as newer kernel versions may include performance improvements and optimizations for packet processing. Updating to a newer version might help resolve netisr backlog issues.
52
+
53
+### Useful resources
54
+
55
+1. [FreeBSD Performance Tuning](https://calomel.org/freebsd_network_tuning.html)
56
+2. [FreeBSD Handbook: Tuning Kernel Limits](https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kernel-limits.html)
health/guides/10min_qos_packet_drops.md
new
+38
@@ -0,0 +1,38 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when there are `packet drops` within the last 10 minutes in your system's `Quality of Service` (`QoS`). If you receive this alert, it means your system's `network performance` may be suffering due to dropped packets.
4
+
5
+### What does packet drops mean?
6
+
7
+Packet drops refer to situations where one or more packets of data traveling across a computer network fail to reach their destination, often caused by network congestion or faulty hardware. Dropped packets can result in poor QoS, including degraded voice and video quality, or even data loss in severe cases.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check the network utilization, packet loss, and latency
12
+
13
+ You can use the `netdata` dashboard to check the network utilization, packet loss, and latency. This will help you identify if there is any congestion or excessive usage in your network that could be causing the packet drops.
14
+
15
+- Examine the system logs
16
+
17
+ Inspect your system logs to identify any potential hardware issues or network-related errors that could be causing the packet drops. You can use tools like `dmesg`, `journalctl`, or check the `/var/log` directory for log files.
18
+
19
+- Check for faulty hardware or misconfigurations
20
+
21
+ Inspect your network devices, such as routers, switches, and network interfaces, for any signs of faulty hardware or misconfigurations that could be causing dropped packets.
22
+
23
+- Optimize your network configuration
24
+
25
+ Review your network configuration for any settings that could be causing dropped packets, such as improper buffer sizes, incorrect QoS settings, or misconfigured packet handling mechanisms.
26
+
27
+- Update network device drivers or firmware
28
+
29
+ Ensure that you are using the latest drivers and firmware for your network devices. Outdated or buggy drivers can sometimes cause packet drops.
30
+
31
+- Monitor the network continuously
32
+
33
+ Regularly monitor the performance of your network to identify and address any issues that may be causing packet drops. You can use tools like `tc`, `ip`, `ifconfig`, and others for this purpose.
34
+
35
+### Useful resources
36
+
37
+1. [Netdata - Real-Time Performance Monitoring](https://www.netdata.cloud/)
38
+2. [Linux Advanced Routing & Traffic Control](https://lartc.org/)
health/guides/10s_ipv4_tcp_resets_received.md
new
+67
@@ -0,0 +1,67 @@
1
+### Understand the alert
2
+
3
+TCP reset is an abrupt closure of the session. It causes the resources allocated to the connection to be immediately released and all other information about the connection is erased.
4
+
5
+The Netdata Agent monitors the average number of sent TCP RESETS over the last 10 seconds. This can indicate a port scan or that a service running on the system has crashed. Additionally, it's a result of a high number of sent TCP RESETS. Furthermore, it can also indicate a SYN reset attack.
6
+
7
+### More about TCP Resets
8
+
9
+TCP uses a three-way handshake to establish a reliable connection. The connection is full duplex, and both sides synchronize (SYN) and acknowledge (ACK) each other. The exchange of these four flags
10
+is performed in three steps: SYN, SYN-ACK, and ACK.
11
+
12
+When an unexpected TCP packet arrives at a host, that host usually responds by sending a reset packet back on the same connection. A reset packet is one with no payload and with the RST bit set in the TCP header flags. There are a few circumstances in which a TCP packet might not be expected. The most common cases are:
13
+
14
+1. A TCP packet received on a port that is not open.
15
+2. An aborting connection
16
+3. Half opened connections
17
+4. Time wait assassination
18
+5. Listening endpoint Queue is Full
19
+6. A TCP Buffer Overflow
20
+
21
+Basically, A TCP Reset usually occurs when a system receives data which doesn't agree with its view of the connection.
22
+
23
+### Troubleshoot the alert
24
+
25
+- Use tcpdump to capture the traffic and use Wireshark to inspect the network packets. You must stop the capture after a certain observation period (60s up to 5 minutes). This command will create a dump file which can be interpreted by Wireshark that contains all the TCP packets with RST flag set.
26
+ ```
27
+ tcpdump -i any 'tcp[tcpflags] & (tcp-rst) == (tcp-rst)' -s 65535 -w output.pcap
28
+ ```
29
+
30
+- Counter measure on malicious TCP resets
31
+
32
+SYN cookie is a technique used to resist IP address spoofing attacks. In particular, the use of SYN cookies allows a server to avoid dropping connections when the SYN queue fills up.
33
+
34
+Enable SYN cookies in Linux:
35
+
36
+ 1. Check if your system has the SYN cookies service enabled. If the value is 1, then the service is enabled, if not proceed to step 2.
37
+ ```
38
+ cat /proc/sys/net/ipv4/tcp_syncookies
39
+ ```
40
+
41
+ 2. Bump this `net.ipv4.tcp_syncookies=1` value under `/etc/sysctl.conf`
42
+
43
+ 3. Apply the configuration
44
+ ```
45
+ sysctl -p
46
+ ```
47
+
48
+Enable SYN cookies in FreeBSD:
49
+
50
+ 1. Check if your system has the SYN cookies service enabled. If the value is 1, then the service is enabled, if not proceed to step 2.
51
+ ```
52
+ sysctl net.inet.tcp.syncookies_only
53
+ ```
54
+
55
+ 2. Bump this `net.inet.tcp.syncookies_only=1` value under `/etc/sysctl.conf`
56
+
57
+ 3. Apply the configuration
58
+ ```
59
+ /etc/rc.d/sysctl reload
60
+ ```
61
+
62
+The use of SYN cookies does not break any protocol specifications, and therefore should be compatible with all TCP implementations. There are, however, a few caveats that take effect when SYN cookies are in use.
63
+
64
+### Useful resources
65
+
66
+1. [TCP reset explanation](https://www.pico.net/kb/what-is-a-tcp-reset-rst/)
67
+2. [TCP 3-way handshake on wikipedia](https://en.wikipedia.org/wiki/Handshaking)
health/guides/10s_ipv4_tcp_resets_sent.md
new
+43
@@ -0,0 +1,43 @@
1
+### Understand the alert
2
+
3
+TCP reset is an abrupt closure of the session. It causes the resources allocated to the connection to be immediately released and all other information about the connection is erased.
4
+
5
+The Netdata Agent monitors the average number of sent TCP RESETS over the last 10 seconds. This can indicate a port scan or that a service running on the system has crashed. Additionally, it's a result of a high number of sent TCP RESETS. Furthermore, it can also indicate a SYN reset attack.
6
+
7
+### More about TCP Resets
8
+
9
+TCP uses a three-way handshake to establish a reliable connection. The connection is full duplex, and both sides synchronize (SYN) and acknowledge (ACK) each other. The exchange of these four flags
10
+is performed in three steps: SYN, SYN-ACK, and ACK.
11
+
12
+When an unexpected TCP packet arrives at a host, that host usually responds by sending a reset packet back on the same connection. A reset packet is one with no payload and with the RST bit set in the TCP header flags. There are a few circumstances in which a TCP packet might not be expected. The most common cases are:
13
+
14
+1. A TCP packet received on a port that is not open.
15
+2. An aborting connection
16
+3. Half opened connections
17
+4. Time wait assassination
18
+5. Listening endpoint Queue is Full
19
+6. A TCP Buffer Overflow
20
+
21
+Basically, A TCP Reset usually occurs when a system receives data which doesn't agree with its view of the connection.
22
+
23
+When your system cannot establish a connection it will retry by default `net.ipv4.tcp_syn_retries` times.
24
+
25
+### Troubleshoot the alert
26
+
27
+- Use tcpdump to capture the traffic and use Wireshark to inspect the network packets. You must stop the capture after a certain observation period (60s up to 5 minutes). This command will create a dump file which can be interpreted by Wireshark that contains all the TCP packets with RST flag set.
28
+ ```
29
+ tcpdump -i any 'tcp[tcpflags] & (tcp-rst) == (tcp-rst)' -s 65535 -w output.pcap
30
+ ```
31
+
32
+- Identify which application sends TCP resets
33
+
34
+1. Check the instances of `RST` events of the TCP protocol. Wireshark also displays the ports on which the two systems tried to establish the TCP connection, (XXXXXX -> XXXXXX).
35
+2. To check which application is using this port, run the following code:
36
+ ```
37
+ lsof -i:XXXXXX -P -n
38
+ ```
39
+### Useful resources
40
+
41
+1. [TCP reset explanation](https://www.pico.net/kb/what-is-a-tcp-reset-rst/)
42
+2. [TCP 3-way handshake on wikipedia](https://en.wikipedia.org/wiki/Handshaking)
43
+3. [Read more about Wireshark here](https://www.wireshark.org/)
\ No newline at end of file
health/guides/10s_received_packets_storm.md
new
+23
@@ -0,0 +1,23 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when there is a significant increase in the number of received packets within a 10-second interval. It indicates a potential packet storm, which may cause network congestion, dropped packets, and reduced performance.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check network utilization**: Monitor network utilization on the affected interface to identify potential bottlenecks, high bandwidth usage, or network saturation.
8
+
9
+2. **Identify the source**: Determine the source of the increased packet rate. This may be caused by a misconfigured application, a faulty network device, or a Denial of Service (DoS) attack.
10
+
11
+3. **Inspect network devices**: Check network devices such as routers, switches, and firewalls for potential issues, misconfigurations, or firmware updates that may resolve the problem.
12
+
13
+4. **Verify application behavior**: Ensure that the applications running on your network are behaving as expected and not generating excessive traffic.
14
+
15
+5. **Implement rate limiting**: If the packet storm is caused by a specific application or service, consider implementing rate limiting to control the number of packets being sent.
16
+
17
+6. **Monitor network security**: Check for signs of a DoS attack or other security threats, and take appropriate action to mitigate the risk.
18
+
19
+### Useful resources
20
+
21
+1. [Wireshark User's Guide](https://www.wireshark.org/docs/wsug_html_chunked/)
22
+2. [Tcpdump Manual Page](https://www.tcpdump.org/manpages/tcpdump.1.html)
23
+3. [Iperf - Network Bandwidth Measurement Tool](https://iperf.fr/)
health/guides/1hour_ecc_memory_correctable.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alert, `1hour_ecc_memory_correctable`, monitors the number of Error Correcting Code (ECC) correctable errors that occur within an hour. If you receive this alert, it means that there are ECC correctable errors in your system's memory. While it does not pose an immediate threat, it may indicate that a memory module is slowly deteriorating.
4
+
5
+### ECC Memory
6
+
7
+ECC memory is a type of computer data storage that can detect and correct the most common kinds of internal data corruption. It is used in systems that require high reliability and stability, such as servers or mission-critical applications.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Inspect the memory modules
12
+
13
+ If the alert is triggered, start by physically checking the memory modules in the system. Ensure that the contacts are clean, and all modules are firmly seated in their respective slots.
14
+
15
+2. Perform a memory test
16
+
17
+ Run a thorough memory test using a tool like Memtest86+. This will help identify if any memory chips have problems that can cause the ECC errors.
18
+
19
+ ```
20
+ sudo apt-get install memtester
21
+ sudo memtester 1024M 5
22
+ ```
23
+
24
+ Replace `1024M` with the amount of memory you'd like to test (in MB) and `5` with the number of loops for the test.
25
+
26
+3. Monitor the errors
27
+
28
+ Monitor the frequency of ECC correctable errors. Keep a record of when they occur and if there are any patterns or trends. If errors continue to occur, move to step 4.
29
+
30
+4. Replace faulty memory modules
31
+
32
+ If ECC correctable errors persist, identify the memory modules with the highest error rates and consider replacing them as a preventive measure. This will help maintain the reliability and stability of your system.
33
+
34
+### Useful resources
35
+
36
+1. [Memtest86+ - Advanced Memory Diagnostic Tool](https://www.memtest.org/)
37
+2. [How to Diagnose, Check, and Test for Bad Memory](https://www.computerhope.com/issues/ch001089.htm)
health/guides/1hour_ecc_memory_uncorrectable.md
new
+27
@@ -0,0 +1,27 @@
1
+### Understand the alert
2
+
3
+This alert, `1hour_ecc_memory_uncorrectable`, indicates that there are ECC (Error-Correcting Code) uncorrectable errors detected in your system's memory within the last hour. ECC errors are caused by issues in the system's RAM (Random Access Memory). These uncorrectable errors are severe and may lead to system crashes or data corruption.
4
+
5
+### What are ECC errors?
6
+
7
+ECC memory is designed to detect and, in some cases, correct data corruption in the memory, preventing system crashes and providing overall system stability. ECC errors fall into two categories:
8
+
9
+1. **Correctable Errors**: These are errors that the ECC memory can detect and correct, preventing system crashes and ensuring data integrity.
10
+2. **Uncorrectable Errors**: These are more severe errors that the ECC memory cannot correct, often requiring faulty memory modules to be replaced to prevent system crashes and data corruption.
11
+
12
+### Troubleshoot the alert
13
+
14
+- **Inspect the memory modules**: Power off the system and check the memory modules for any signs of damage or poor contact with the socket. Ensure that the memory modules are seated firmly and there is proper contact.
15
+
16
+- **Run memory diagnostics**: Run memory diagnostic tools, like [Memtest86+](https://www.memtest.org/) to identify any memory errors and verify the memory's health. If errors are detected, it's an indication that the memory modules need to be replaced.
17
+
18
+- **Replace faulty memory modules**: If uncorrectable errors continue occurring or if diagnostics identify faulty memory modules, consider replacing them. Before doing so, check if the memory modules are still covered under warranty.
19
+
20
+- **Check system logs**: Review system logs, such as Event Viewer on Windows or `/var/log` on Linux systems, for any related messages or errors that may help to diagnose the issue further.
21
+
22
+- **Update firmware**: Ensure your system's firmware and BIOS are up-to-date. Manufacturers often release stability and performance improvements that can potentially resolve or mitigate ECC errors.
23
+
24
+
25
+### Useful resources
26
+
27
+1. [How to Check Memory Problems in Linux](https://www.cyberciti.biz/faq/linux-check-memory-usage/)
health/guides/1hour_memory_hw_corrupted.md
new
+19
@@ -0,0 +1,19 @@
1
+
2
+### Understand the alert
3
+The Linux kernel keeps track of the system memory state. You can find the actual values it tracks in the [man pages](https://man7.org/linux/man-pages/man5/proc.5.html) under the `/proc/meminfo` subsection. One of the values that the kernel reports is the `HardwareCorrupted` , which is the amount of memory, in kibibytes (1024 bytes), with physical memory corruption problems, identified by the hardware and set aside by the kernel so it does not get used.
4
+
5
+The Netdata Agent monitors this value. This alert indicates that the memory is corrupted due to a hardware failure. While primarily the error may be due to a failing RAM chip, it can also be caused by incorrect seating or improper contact between the socket and memory module.
6
+
7
+### Troubleshoot the alert
8
+
9
+Most of the time uncorrectable errors will make your system reboot/shutdown in a state of panic. If not, that means that your tolerance level is high enough to not make the system go into panic. You must identify the defective module immediately.
10
+
11
+`memtester` is a userspace utility for testing the memory subsystem for faults.
12
+
13
+You may also receive this error as a result of incorrect seating or improper contact between the socket and RAM module. Check both before consider replacing the RAM module.
14
+
15
+### Useful resources
16
+
17
+1. [man pages /proc](https://man7.org/linux/man-pages/man5/proc.5.html)
18
+2. [memtester homepage](https://pyropus.ca/software/memtester/)
19
+
health/guides/1m_bad_requests.md
new
+21
@@ -0,0 +1,21 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the ratio of client error HTTP requests (4xx class status codes, excluding 401) within the last minute is higher than normal. Client errors indicate that the issue is on the client's side, such as incorrect requests or invalid URLs.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Analyze response codes**: Identify the specific HTTP response codes your web server is sending to clients. Use the Netdata dashboard and inspect the `detailed_response_codes` chart for your web server to track the error codes being sent.
8
+
9
+2. **Check server logs**: Review the web server logs (e.g., access.log and error.log) to identify any issues, patterns, or errors causing the increase in client errors. These logs can typically be found under `/var/log/{nginx, apache2}/{access.log, error.log}`.
10
+
11
+3. **Verify application behavior**: Check the behavior of applications running on your web server to ensure they are not generating incorrect URLs or causing issues with client requests.
12
+
13
+4. **Identify broken links**: If there is a high number of 404 errors, use a broken link checker tool to identify and fix any dead links on your website or other websites that redirect to your website.
14
+
15
+5. **Monitor server performance**: Keep an eye on the web server's performance metrics to ensure that changes in client errors do not negatively impact server performance or resource usage.
16
+
17
+### Useful resources
18
+
19
+1. [RFC 2616 - HTTP/1.1 Status Code Definitions](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4)
20
+2. [Mozilla - HTTP Status Codes - Client Error Responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
21
+3. [Broken Link Checker Tools](https://www.google.com/search?q=broken+link+checker)
health/guides/1m_internal_errors.md
new
+24
@@ -0,0 +1,24 @@
1
+### Understand the alert
2
+
3
+This alert indicates that there has been an increase in the number of HTTP 5XX server errors in the last minute. These errors typically indicate a problem with the server's ability to process requests, such as misconfigurations, overloaded resources, or other server-side issues.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Inspect server logs**: Check the server error logs for any error messages, warnings, or unusual patterns. For Apache and Nginx, the error logs are usually found under `/var/log/{apache2, nginx}/error.log`. Analyze the logs to identify potential issues with the server, such as misconfigurations or resource limitations.
8
+
9
+2. **Check .htaccess file**: If you're using Apache, examine the `.htaccess` file for any misconfigurations or incorrect settings. Ensure that the directives in the file are valid and properly formatted. If necessary, temporarily disable the `.htaccess` file to see if it resolves the issue.
10
+
11
+3. **Review server resources**: Monitor the server's CPU, RAM, and disk usage to determine if the server is experiencing resource limitations. High resource usage can lead to server errors, as the server may be unable to handle incoming requests. Consider upgrading your server resources or optimizing the server for better performance.
12
+
13
+4. **Examine server software**: Check for any issues with the server software, such as outdated versions, security vulnerabilities, or software bugs. Update your server software to the latest version and apply any necessary patches to resolve potential issues.
14
+
15
+5. **Monitor third-party services**: If your server relies on third-party services or APIs, verify that these services are functioning correctly. Server errors may occur if your server is unable to communicate with these services or if they are experiencing downtime.
16
+
17
+6. **Test server functionality**: Use tools such as `curl` or web browser developer tools to send HTTP requests to your server and examine the responses. This can help you identify specific issues with the server, such as incorrect response headers or missing resources.
18
+
19
+### Useful resources
20
+
21
+1. [Apache HTTP Server Documentation](https://httpd.apache.org/docs/)
22
+2. [Nginx Documentation](https://nginx.org/en/docs/)
23
+3. [Mozilla Developer Network - HTTP Status Codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
24
+
health/guides/1m_ipv4_tcp_resets_received.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert, `1m_ipv4_tcp_resets_received`, calculates the average number of TCP RESETS received (`AttemptFails`) over the last minute on your system. If you receive this alert, it means that there is an increase in the number of TCP RESETS, which might indicate a problem with your networked applications or servers.
4
+
5
+### What does TCP RESET mean?
6
+
7
+`TCP RESET` is a signal that is sent from one connection end to the other when an ongoing connection is immediately terminated without an orderly close. This usually happens when a networked application encounters an issue, such as an incorrect connection request, invalid data packet, or a closed port.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the top consumers of TCP RESETS:
12
+
13
+ You can use the `ss` utility to list the TCP sockets and their states:
14
+
15
+ ```
16
+ sudo ss -tan
17
+ ```
18
+
19
+ Look for the `State` column to see which sockets have a `CLOSE-WAIT`, `FIN-WAIT`, `TIME-WAIT`, or `LAST-ACK` status. These states usually have a high number of TCP RESETS.
20
+
21
+2. Check the logs of the concerned applications:
22
+
23
+ If you have identified the problematic applications or servers, inspect their logs for any error messages, warnings, or unusual activity related to network connection issues.
24
+
25
+3. Inspect the system logs:
26
+
27
+ Check the system logs, such as `/var/log/syslog` on Linux or `/var/log/system.log` on FreeBSD, for any network-related issues. This could help you find possible reasons for the increased number of TCP RESETS.
28
+
29
+4. Monitor and diagnose network issues:
30
+
31
+ Use tools like `tcpdump`, `wireshark`, or `iftop` to capture packets and observe network traffic. This can help you identify patterns that may be causing the increased number of TCP RESETS.
32
+
33
+5. Check for resource constraints:
34
+
35
+ Ensure that your system's resources, such as CPU, memory, and disk space, are not under heavy load or reaching their limits. High resource usage could cause networked applications to behave unexpectedly, resulting in an increased number of TCP RESETS.
36
+
37
+### Useful resources
38
+
39
+1. [ss Utility - Investigate Network Connections & Sockets](https://www.binarytides.com/linux-ss-command/)
40
+2. [Wireshark - A Network Protocol Analyzer](https://www.wireshark.org/)
41
+3. [Monitoring Network Traffic with iftop](https://www.tecmint.com/iftop-linux-network-bandwidth-monitoring-tool/)
health/guides/1m_ipv4_tcp_resets_sent.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average number of TCP resets (`OutRsts`) sent by the host over the last minute. If you receive this alert, it means that your system is experiencing an unusually high rate of TCP resets, which might signal connection issues or potential attacks.
4
+
5
+### What is a TCP reset?
6
+
7
+A TCP reset (or RST packet) is a signal used in the Transmission Control Protocol (TCP) to abruptly close an active connection between two devices. It can be sent by either the client or server to inform the other party that they should consider the connection terminated.
8
+
9
+### Why are high numbers of TCP resets a concern?
10
+
11
+When there's a high rate of TCP resets sent by a host, it generally indicates problems in communication with other devices or services. This could be due to network latency, misconfigured firewalls, or aggressive timeouts causing connections to break. In some cases, it could also signal a potential Denial of Service (DoS) attack, where an attacker sends multiple resets to disrupt a service or network.
12
+
13
+### Troubleshoot the alert
14
+
15
+- Check the network performance
16
+
17
+ Investigate if there are any network latency issues or congestion in your system. You can use tools like `ping`, `traceroute`, or `mtr` to check the network quality and connectivity to other hosts.
18
+
19
+- Analyze packet captures for communication issues
20
+
21
+ Use a packet capture tool like `tcpdump` or `Wireshark` to capture and analyze network traffic during the period of high resets. Look for patterns or specific connections that are frequently terminated with a reset. This could help pinpoint misconfigured services, firewalls, or devices causing the issue.
22
+
23
+- Check firewall settings
24
+
25
+ Ensure that your firewall settings are properly configured to allow necessary connections and not aggressively closing them. Look for rules related to connection timeouts, max connections, and SYN flood protection to see if they might be causing the resets.
26
+
27
+- Review system logs for errors
28
+
29
+ Check system and application logs for any error messages or events that correlate to the time of the alert. This might give you more information about the cause of the issue.
30
+
31
+- Monitor for potential attacks
32
+
33
+ If the above steps don't help determine the cause, consider monitoring your network and system for potential DoS attacks. Implement security measures such as rate-limiting and access control to protect your services and network from malicious traffic.
34
+
35
+### Useful resources
36
+
37
+1. [TCP Connection Resets and How to Troubleshoot Them](https://blog.wireshark.org/tcp/connection/resets/troubleshoot/)
health/guides/1m_ipv4_udp_receive_buffer_errors.md
new
+65
@@ -0,0 +1,65 @@
1
+### Understand the alert
2
+
3
+In both Linux and FreeBSD variants, the kernel allocates buffers to serve the UDP protocol operations. Packets after reception from a network interface are forwarded to these buffers to be processed by the UDP protocol stack in a system's socket.
4
+
5
+The Netdata Agent monitors the average number of UDP receive buffer errors over the last minute. Receiving this alert means that your system is dropping incoming UDP packets. This may indicate that the UDP receive buffer queue is full. This alert is triggered in warning state when the number of UDP receive buffer errors over the last minute is more than 10.
6
+
7
+In general, issues with buffers that allocated dynamically are correlated with the kernel memory, you must always be aware of memory pressure events. This can cause buffer errors.
8
+
9
+### Troubleshoot the alert (Linux)
10
+
11
+- Increase the net.core.rmem_default and net.core.rmem_max values
12
+
13
+1. Try to increase them, RedHat suggests the value of 262144 bytes
14
+ ```
15
+ sysctl -w net.core.rmem_default=262144
16
+ sysctl -w net.core.rmem_max=262144
17
+ ```
18
+
19
+2. Verify the change and test with the same workload that triggered the alarm originally.
20
+ ```
21
+ sysctl net.core.rmem_default net.core.rmem_max
22
+ net.core.rmem_default=262144
23
+ net.core.rmem_max=262144
24
+ ```
25
+
26
+3. If this change works for your system, you could make it permanently.
27
+
28
+ Bump these `net.core.rmem_default=262144` & `net.core.rmem_max=262144` entries under `/etc/sysctl.conf`.
29
+
30
+4. Reload the sysctl settings.
31
+
32
+ ```
33
+ sysctl -p
34
+ ```
35
+
36
+### Troubleshoot the alert (FreeBSD)
37
+
38
+- Increase the kern.ipc.maxsockbuf value
39
+
40
+1. Try to set this value to at least 16MB for 10GE overall
41
+ ```
42
+ sysctl -w kern.ipc.maxsockbuf=16777216
43
+ ```
44
+
45
+2. Verify the change and test with the same workload that triggered the alarm originally.
46
+ ```
47
+ sysctl kern.ipc.maxsockbuf
48
+ kern.ipc.maxsockbuf=16777216
49
+ ```
50
+
51
+3. If this change works for your system, you could make it permanently.
52
+
53
+ Bump this `kern.ipc.maxsockbuf=16777216` entry under `/etc/sysctl.conf`.
54
+
55
+4. Reload the sysctl settings.
56
+ ```
57
+ /etc/rc.d/sysctl reload
58
+ ```
59
+
60
+### Useful resources
61
+
62
+1. [UDP definition on wikipedia](https://en.wikipedia.org/wiki/User_Datagram_Protocol)
63
+2. [Man page of UDP protocol](https://man7.org/linux/man-pages/man7/udp.7.html)
64
+3. [Redhat networking tuning guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings)
65
+4. [UDP on freebsd (blog)](https://awasihba.wordpress.com/2008/10/13/udp-on-freebsd/)
health/guides/1m_ipv4_udp_send_buffer_errors.md
new
+43
@@ -0,0 +1,43 @@
1
+### Understand the alert
2
+
3
+The linux kernel allocates buffers to serve the UDP protocol operations. Data is written into sockets that utilize UDP to send data to an another system/subsystem.
4
+
5
+The Netdata Agent monitors the average number of UDP send buffer errors over the last minute. This alert indicates that the UDP send buffer is full or no kernel memory available. Receiving this alert
6
+means that your system is dropping outgoing UDP packets. This alert is triggered in warning state when the number of UDP send buffer errors over the last minute is more than 10.
7
+
8
+In general, issues with buffers that allocated dynamically are correlated with the kernel memory, you must always be aware of memory pressure events. This can cause buffer errors.
9
+
10
+### Troubleshooting section:
11
+
12
+- Increase the net.core.wmem_default and net.core.wmem_max values
13
+
14
+1. Try to increase them, RedHat suggests the value of 262144 bytes
15
+
16
+ ```
17
+ sysctl -w net.core.wmem_default=262144
18
+ sysctl -w net.core.wmem_max=262144
19
+ ```
20
+
21
+2. Verify the change and test with the same workload that triggered the alarm originally.
22
+
23
+ ```
24
+ sysctl net.core.wmem_default net.core.wmem_max
25
+ net.core.wmem_default=262144
26
+ net.core.wmem_max=262144
27
+ ```
28
+
29
+3. If this change works for your system, you could make it permanently.
30
+
31
+ Bump these `net.core.wmem_default=262144` & `net.core.wmem_max=262144` entries under `/etc/sysctl.conf`.
32
+
33
+4. Reload the sysctl settings.
34
+
35
+ ```
36
+ sysctl -p
37
+ ```
38
+
39
+### Useful resources
40
+
41
+1. [UDP definition on wikipedia](https://en.wikipedia.org/wiki/User_Datagram_Protocol)
42
+2. [Man page of UDP protocol](https://man7.org/linux/man-pages/man7/udp.7.html)
43
+3. [Redhat networking tuning guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings)
health/guides/1m_received_packets_rate.md
new
+45
@@ -0,0 +1,45 @@
1
+### Understand the alert
2
+
3
+1m_received_packets_rate alert indicates the average number of packets received by the network interface on your system over the last minute. If you receive this alert, it signifies higher than usual network traffic incoming.
4
+
5
+### What do received packets mean?
6
+
7
+A received packet is a unit of data that is transmitted through the network interface to your system. Higher received packets rate means an increase in incoming network traffic to your system. It could be due to legitimate usage or could signal a potential issue such as a network misconfiguration, an attack, or a system malfunction.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Analyze the network throughput: Use the `nload` or `iftop` command to check the incoming traffic on your system's network interfaces. These commands display the current network traffic and will help you monitor the incoming data.
12
+
13
+ ```
14
+ sudo nload <network_interface> // or
15
+ sudo iftop -i <network_interface>
16
+ ```
17
+
18
+ Replace `<network_interface>` with your network interface (e.g., eth0).
19
+
20
+2. Check for specific processes consuming unusually high network bandwidth: Use the `netstat` command combined with `grep` to filter the results and find processes with high network traffic.
21
+
22
+ ```
23
+ sudo netstat -tunap | grep <network_interface>
24
+ ```
25
+
26
+ Replace `<network_interface>` with your network interface (e.g., eth0).
27
+
28
+3. Identify host-consuming bandwidth: After identifying the processes consuming a high network, you can trace back their respective hosts. Use the `tcpdump` command to capture live network traffic and analyze it for specific IP addresses causing the high packets rate.
29
+
30
+ ```
31
+ sudo tcpdump -n -i <network_interface> -c 100
32
+ ```
33
+
34
+ Replace `<network_interface>` with your network interface (e.g., eth0).
35
+
36
+4. Mitigate the issue: Depending on the root cause, apply appropriate remedial actions. This may include:
37
+ - Adjusting application/service configuration to reduce network traffic
38
+ - Updating firewall rules to block undesired sources/IPs
39
+ - Ensuring network devices are appropriately configured
40
+ - Addressing system overload issues that hamper network performance
41
+
42
+### Useful resources
43
+
44
+1. [nload - Monitor Linux Network Traffic and Bandwidth Usage in Real Time](https://www.tecmint.com/nload-monitor-linux-network-traffic-bandwidth-usage/)
45
+2. [An Introduction to the ss Command](http://www.binarytides.com/linux-ss-command/)
health/guides/1m_received_traffic_overflow.md
new
+24
@@ -0,0 +1,24 @@
1
+### Understand the alert
2
+
3
+Network interfaces are categorized primarily on the bandwidth they can operate (1 Gbps, 10 Gbps, etc). High network utilization occurs when the volume of data on a network link approaches the capacity of the link. Netdata agent
4
+calculates the average outbound utilization for a specific network interface over the last minute. High outbound utilization increases latency and packet loss because packet bursts are buffered
5
+
6
+This alarm may indicate either network congestion or malicious activity.
7
+
8
+### Troubleshoot the alert
9
+
10
+- Prioritize important traffic
11
+
12
+Quality of service (QoS) is the use of routing prioritization to control traffic and ensure the performance of critical applications. QoS works best when low-priority traffic exists that can be dropped when congestion occurs. The higher-priority traffic must fit within the bandwidth limitations of the link or path.
13
+
14
+- Add more bandwidth
15
+
16
+ - For **Cloud infrastructures**, adding bandwidth might be easy. It depends on your cloud infrastracture and your cloud provider. Some of them either offer you the service to upgrade machines to a higher bandwidth rate or upgrade you machine to a more powerful one with higher bandwidth rate.
17
+
18
+ - For **Bare-metal** machines, you will need either a hardware upgrade or the addition of a network card using link aggregation to combine multiple network connections in parallel (e.g LACP).
19
+
20
+### Useful resources
21
+
22
+- [FireQOS](https://firehol.org/tutorial/fireqos-new-user/) is a traffic shaping helper. It has a very simple shell scripting language to express traffic shaping.
23
+
24
+- [`tcconfig`](https://tcconfig.readthedocs.io/en/latest/index.html) is a command wrapper that makes it easy to set up traffic control of network bandwidth/latency/packet-loss/packet-corruption/etc.
\ No newline at end of file
health/guides/1m_sent_traffic_overflow.md
new
+23
@@ -0,0 +1,23 @@
1
+### Understand the alert
2
+
3
+Network interfaces are categorized primarily on the bandwidth rate at which they can operate (1 Gbps, 10 Gbps, etc). High network utilization occurs when the volume of data on a network link approaches the capacity of the link. Netdata agent calculates the average outbound utilization for a specific network interface over the last minute. High outbound utilization increases latency and packet loss because packet bursts are buffered.
4
+
5
+This alarm may indicate either a network congestion or malicious activity.
6
+
7
+### Troubleshoot the alert
8
+
9
+- Prioritize important traffic
10
+
11
+Quality of service (QoS) is the use of mechanisms or technologies to control traffic and ensure the performance of critical applications. QoS works best when low-priority traffic exists that can be dropped when congestion occurs. The higher-priority traffic must fit within the bandwidth limitations of the link or path.
12
+
13
+- Add more bandwidth
14
+
15
+ - For **Cloud infrastructures**, adding bandwidth might be easy. It depends on your cloud infrastracture and your cloud provider. Some of them either offer you the service to upgrade machines to a higher bandwidth rate or upgrade you machine to a more powerful one with higher bandwidth rate.
16
+
17
+ - For **Bare-metal** machines, you will need either a hardware upgrade or the addition of a network card using link aggregation to combine multiple network connections in parallel (e.g LACP).
18
+
19
+### Useful resources
20
+
21
+- [FireQOS](https://firehol.org/tutorial/fireqos-new-user/) is a traffic shaping helper. It has a very simple shell scripting language to express traffic shaping.
22
+
23
+- [`tcconfig`](https://tcconfig.readthedocs.io/en/latest/index.html) is a command wrapper that makes it easy to set up traffic control of network bandwidth/latency/packet-loss/packet-corruption/etc.
\ No newline at end of file
health/guides/1m_successful.md
new
+23
@@ -0,0 +1,23 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the percentage of successful HTTP requests (1xx, 2xx, 304, 401 response codes) within the last minute falls below a certain threshold. A warning state occurs when the success rate is below 85%, and a critical state occurs when it falls below 75%. This alert can indicate a malfunction in your web server's services, malicious activity towards your website, or broken links.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Analyze response codes**: Identify the specific HTTP response codes your web server is sending to clients. Use the Netdata dashboard and inspect the `detailed_response_codes` chart for your web server to track the error codes being sent.
8
+
9
+2. **Check server logs**: Review the web server logs to identify any issues, patterns, or errors causing the decrease in successful requests. Investigate any unusual or unexpected response codes.
10
+
11
+3. **Inspect application logs**: Check the logs of applications running on your web server for any errors or issues that might be affecting the success rate of HTTP requests.
12
+
13
+4. **Verify server resources**: Ensure your server has adequate resources (CPU, RAM, disk space) to handle the workload, as resource limitations can impact the success rate of HTTP requests.
14
+
15
+5. **Review server configuration**: Check your web server's configuration for any misconfigurations, incorrect permissions, or improper settings that may be causing the issue.
16
+
17
+6. **Monitor security**: Look for signs of malicious activity, such as a high number of requests from a specific IP address or a sudden spike in requests. Implement security measures, such as rate limiting, IP blocking, or Web Application Firewalls (WAF), if necessary.
18
+
19
+### Useful resources
20
+
21
+1. [HTTP status codes on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
22
+2. [Apache HTTP Server Documentation](https://httpd.apache.org/docs/)
23
+3. [Nginx Documentation](https://nginx.org/en/docs/)
health/guides/1m_tcp_accept_queue_drops.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert presents the average number of dropped packets in the TCP accept queue over the last sixty seconds. If it is raised, then the system is dropping incoming TCP connections. This could also be an indication of accepted queue overflow, low memory, security issues, no route to a destination, etc.
4
+- This alert gets raised to warning when the value is greater than 1 and less than 5.
5
+- If the number of queue drops over the last minute exceeds 5, then the alert gets raised to critical.
6
+
7
+### TCP Accept Queue Drops
8
+
9
+The accept queue holds fully established TCP connections waiting to be handled by the listening application. It overflows when the server application fails to accept new connections at the rate they are coming in.
10
+
11
+### Troubleshooting Section
12
+
13
+- Check for queue overflows.
14
+
15
+If you receive this alert, then you can cross-check its results with the `1m_tcp_accept_queue_overflows` alert. If that alert is also in a warning or critical state, then the system is experiencing accept queue overflowing. To fix that you can do the following:
16
+
17
+1. Open the /etc/sysctl.conf file and look for the entry " net.ipv4.tcp_max_syn_backlog".
18
+ The `tcp_max_syn_backlog` is the maximal number of remembered connection requests (SYN_RECV), which have not received an acknowledgment from connecting client.
19
+2. If the entry does not exist, then append the following default entry to the file; `net.ipv4.tcp_max_syn_backlog=1280`. Otherwise, adjust the limit to suit your needs.
20
+3. Save your changes and run:
21
+ ```
22
+ sysctl -p
23
+ ```
24
+
25
+Note: Netdata strongly suggests knowing exactly what values you need before making system changes.
26
+
27
+### Useful resources
28
+
29
+1. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
30
+2. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
health/guides/1m_tcp_accept_queue_overflows.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+This alert presents the average number of overflows in the TCP accept queue over the last minute.
4
+
5
+- This alert gets raised in a warning state when the value is greater than 1 and less than 5.
6
+- If the overflow average exceeds 5 in the last minute, then the alert gets raised in the critical state.
7
+
8
+### What is the Accept queue?
9
+
10
+The accept queue holds fully established TCP connections waiting to be handled by the listening application. It overflows when the server application fails to accept new connections at the rate they are coming in.
11
+
12
+### This alert might also indicate a SYN flood.
13
+
14
+A SYN flood is a form of denial-of-service attack in which an attacker rapidly initiates a connection to a server without finalizing the connection. The server has to spend resources waiting for half-opened connections, which can consume enough resources to make the system unresponsive to legitimate traffic.
15
+
16
+### Troubleshooting Section
17
+
18
+Increase the queue length
19
+
20
+1. Open the /etc/sysctl.conf file and look for the entry " net.ipv4.tcp_max_syn_backlog".
21
+ The `tcp_max_syn_backlog` is the maximal number of remembered connection requests (SYN_RECV), which have not received an acknowledgment from connecting client.
22
+2. If the entry does not exist, you can append the following default entry to the file; `net.ipv4. tcp_max_syn_backlog=1280`. Otherwise, adjust the limit to suit your needs.
23
+3. Save your changes and run;
24
+ ```
25
+ sysctl -p
26
+ ```
27
+
28
+Note: Netdata strongly suggests knowing exactly what values you need before making system changes.
29
+
30
+### Useful resources
31
+
32
+1. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
33
+2. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
34
+3. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
35
+
health/guides/1m_tcp_syn_queue_cookies.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+This alert presents the average number of sent SYN cookies due to the full TCP SYN queue over the sixty seconds. Receiving this means that the incoming traffic is excessive. SYN queue cookies are used to resist any potential SYN flood attacks.
4
+
5
+This alert is raised to warning when the average exceeds 1 and will enter critical when the value exceeds an average of 5 sent SYN cookies in sixty seconds.
6
+
7
+###What are SYN Queue Cookies?
8
+
9
+The SYN Queue stores inbound SYN packets (specifically: struct inet_request_sock). It is responsible for sending out SYN+ACK packets and retrying them on timeout. After transmitting the SYN+ACK, the SYN Queue waits for an ACK packet from the client - the last packet in the three-way-handshake. All received ACK packets must first be matched against the fully established connection table, and only then against data in the relevant SYN Queue. On SYN Queue match, the kernel removes the item from the SYN Queue, successfully creates a full connection (specifically: struct inet_sock), and adds it to the Accept Queue.
10
+
11
+### SYN flood
12
+
13
+This alert likely indicates a SYN flood.
14
+
15
+A SYN flood is a form of denial-of-service attack in which an attacker rapidly initiates a connection to a server without finalizing the connection. The server has to spend resources waiting for half-opened connections, which can consume enough resources to make the system unresponsive to legitimate traffic.
16
+
17
+### Troubleshoot the alert
18
+
19
+If the traffic is legitimate, then increase the limit of the SYN queue.
20
+
21
+If you can determine that the traffic is legitimate, consider expanding the limit of the SYN queue through configuration;
22
+
23
+*(If the traffic is not legitimate, then this is not safe! You will expose more resources to an attacker if the traffic is not legitimate.)*
24
+
25
+1. Open the /etc/sysctl.conf file and look for the entry "net.core.somaxconn". This value will affect both SYN and accept queue limits on newer Linux systems.
26
+2. Set the value accordingly (By default it is set to 128) `net.core.somaxconn=128` (if the value doesn't exist, append it to the file)
27
+3. Save your changes and run this command to apply the changes.
28
+ ```
29
+ sysctl -p
30
+ ```
31
+Note: Netdata strongly suggests knowing exactly what values you need before making system changes.
32
+
33
+### Useful resources
34
+
35
+1. [SYN packet handling](https://blog.cloudflare.com/syn-packet-handling-in-the-wild/)
36
+2. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
37
+3. [SYN Cookies](https://en.wikipedia.org/wiki/SYN_cookies)
38
+4. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
39
+5. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
health/guides/1m_tcp_syn_queue_drops.md
new
+22
@@ -0,0 +1,22 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the average number of SYN requests dropped due to the TCP SYN queue being full has exceeded a specific threshold in the last minute. A high number of dropped SYN requests may indicate a SYN flood attack, causing the system to become unresponsive to legitimate traffic.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Monitor incoming traffic**: Analyze the incoming network traffic to determine if there is a sudden surge in SYN requests, which might indicate a SYN flood attack. Use tools like `tcpdump`, `iftop`, or `nload` to monitor network traffic.
8
+
9
+2. **Check system resources**: Inspect the system's CPU and memory usage to ensure there are enough resources available to handle incoming connections. High resource usage might lead to dropped SYN requests.
10
+
11
+3. **Enable SYN cookies**: If the traffic is legitimate, consider enabling SYN cookies to help mitigate the impact of a SYN flood attack, as described in the provided guide above.
12
+
13
+4. **Adjust SYN queue settings**: Increase the SYN queue size by adjusting the `net.core.somaxconn` and `net.ipv4.tcp_max_syn_backlog` sysctl parameters. Make sure to set these values according to your system's capacity and traffic requirements.
14
+
15
+5. **Implement traffic filtering**: Use traffic filtering techniques such as rate limiting, IP blocking, or firewall rules to mitigate the impact of SYN flood attacks.
16
+
17
+### Useful resources
18
+
19
+1. [SYN packet handling](https://blog.cloudflare.com/syn-packet-handling-in-the-wild/)
20
+2. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
21
+3. [SYN Cookies](https://en.wikipedia.org/wiki/SYN_cookies)
22
+4. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
health/guides/1min_netdev_backlog_exceeded.md
renamed
+5
-14
@@ -1,24 +1,16 @@
1
-# 1min_netdev_backlog_exceeded
1
+### Understand the alert
2
3
-## OS: Linux
4
-
5
-The linux kernel contains queues where packets are stored after reception from a network interface controller before
6
-being processed by the next protocol stack. There is one netdev backlog queue per CPU core. netdev_max_backlog defines
7
-the maximum number of packets that can enter the queue. Queues fill up when an interface receives packets faster than
8
-kernel can process them. The default netdev_max_backlog value should be 1000. However this may not be enough in cases
9
-such as:
3
+The linux kernel contains queues where packets are stored after reception from a network interface controller before being processed by the next protocol stack. There is one netdev backlog queue per CPU core. netdev_max_backlog defines the maximum number of packets that can enter the queue. Queues fill up when an interface receives packets faster than kernel can process them. The default netdev_max_backlog value should be 1000. However this may not be enough in cases such as:
4
5
- Multiple interfaces operating at 1Gbps, or even a single interface at 10Gbps.
6
7
- Lower powered systems process very large amounts of network traffic.
8
15
-Netdata agent monitors the average number of dropped packets in the last minute due to exceeding the netdev backlog
16
-queue.
9
+Netdata monitors the average number of dropped packets in the last minute due to exceeding the netdev backlog queue.
10
18
-### Troubleshooting section:
11
+### Troubleshoot the alert
12
20
- <details>
21
- <summary>Increase the netdev_max_backlog value</summary>
13
+- Increase the netdev_max_backlog value
14
15
1. Check your current value:
16
@@ -50,4 +42,3 @@ queue.
42
root@netdata~ # sysctl -p
43
```
44
53
-</details>
health/guides/1min_netdev_budget_ran_outs.md
renamed
+9
-24
@@ -1,16 +1,8 @@
1
-# 1min_netdev_budget_ran_outs
1
+### Understand the alert
2
3
-## OS: Linux
3
+Your system communicates with the devices attached to it through interrupt requests. In a nutshell, when an interrupt occurs, the operating system stops what it was doing and starts addressing that interrupt.
4
5
-Your system communicates with the devices attached to it through interrupt requests. In a nutshell, when an interrupt
6
-occurs, the operating system stops what it was doing and starts addressing that interrupt.
7
-
8
-Network interfaces can receive thousands of packets per second. To avoid burying the system with thousands of interrupts,
9
-the Linux kernel uses the NAPI polling framework. In this way, we can replace hundreds of hardware interrupts with one poll
10
-by managing them with a few Soft Interrupt ReQuests (Soft IRQs). Ksoftirqd is a per-CPU kernel thread responsible for
11
-handling those unserved Soft Interrupt ReQuests (Soft IRQs). The Netdata agent inspects the average number of times
12
-Ksoftirqd ran out of netdev_budget or CPU time when there was still work to be done. This abnormality may cause packet
13
-overflow on the intermediate buffers and, as a result, drop packet on your network interfaces.
5
+Network interfaces can receive thousands of packets per second. To avoid burying the system with thousands of interrupts, the Linux kernel uses the NAPI polling framework. In this way, we can replace hundreds of hardware interrupts with one poll by managing them with a few Soft Interrupt ReQuests (Soft IRQs). Ksoftirqd is a per-CPU kernel thread responsible for handling those unserved Soft Interrupt ReQuests (Soft IRQs). The Netdata agent inspects the average number of times Ksoftirqd ran out of netdev_budget or CPU time when there was still work to be done. This abnormality may cause packet overflow on the intermediate buffers and, as a result, drop packet on your network interfaces.
6
7
The default value of the netdev_budget is 300. However, this may not be enough in some cases, such as:
8
@@ -18,23 +10,17 @@ The default value of the netdev_budget is 300. However, this may not be enough
10
11
- Lower powered systems processing very large amounts of network traffic.
12
21
-<details>
22
-<Summary>See more on the NAPI polling mechanism. </summary>
13
+### NAPI polling mechanism.
14
24
-The design of NAPI allows the network driver to go into a polling mode, buffering the packets it receives into a
25
-ring-buffer, and raises a soft interrupt to start a NAPI polling cycle instead of being hard-interrupted for
26
-every packet. Linux kernel through NAPI will poll data from the buffer until the netdev_budget_usecs times out or
27
-the number of packets reaches the netdev_budget limit.
15
+The design of NAPI allows the network driver to go into a polling mode, buffering the packets it receives into a ring-buffer, and raises a soft interrupt to start a NAPI polling cycle instead of being hard-interrupted for
16
+every packet. Linux kernel through NAPI will poll data from the buffer until the netdev_budget_usecs times out or the number of packets reaches the netdev_budget limit.
17
18
- netdev_budget_usecs variable defines the maximum number of microseconds in one NAPI polling cycle.
19
- netdev_budget variable defines the maximum number of packets taken from all interfaces in one polling cycle.
20
32
-</details>
33
-
34
-### Troubleshooting section:
21
+### Troubleshoot the alert
22
36
- <details>
37
- <summary>Increase the netdev_budget value.</summary>
23
+- Increase the netdev_budget value.
24
25
1. Check your current value.
26
@@ -66,5 +52,4 @@ the number of packets reaches the netdev_budget limit.
52
53
```
54
root@netdata~ $ sysctl -p
69
- ```
70
-</details>
55
+ ```
\ No newline at end of file
health/guides/20min_steal_cpu.md
renamed
+5
-12
@@ -1,16 +1,12 @@
1
-# 20min_steal_cpu
2
-
3
-## OS: Linux
1
+### Understand the alert
2
3
This alarm calculates average CPU `steal` time over the last 20 minutes
4
7
-`steal`, in a virtual machine, is the percentage of time that particular virtual CPU has to wait for an available host CPU
8
-to run on. If this metric goes up, it means that your VM is not getting the processing power it needs.
5
+`steal`, in a virtual machine, is the percentage of time that particular virtual CPU has to wait for an available host CPU to run on. If this metric goes up, it means that your VM is not getting the processing power it needs.
6
10
-### Troubleshooting section:
7
+### Troubleshoot the alert
8
12
-<details>
13
-<summary>Check for CPU quota and host issues</summary>
9
+Check for CPU quota and host issues.
10
11
Generally, if `steal` is high, it could mean one of the following:
12
@@ -19,7 +15,4 @@ Generally, if `steal` is high, it could mean one of the following:
15
- The host CPUs are over-committed (you have more virtual CPUs assigned to VMs than the host system has physical CPUs) and too many VMs need CPU time simultanously.
16
- The VM itself has a CPU quota that is too low.
17
22
-So in the end you can increase the CPU resources of that particular VM, and if the alert persists, move the guest to a
23
-different *physical* server.
24
-
25
-</details>
18
+So in the end you can increase the CPU resources of that particular VM, and if the alert persists, move the guest to a different *physical* server.
health/guides/30min_ram_swapped_out.md
new
+26
@@ -0,0 +1,26 @@
1
+### Understand the alert
2
+
3
+If the system needs more memory resources than your available RAM, inactive pages in memory can be moved into the swap space (or swap file). The swap space (or swap file) is located on hard drives,
4
+which have a slower access time than physical memory.
5
+
6
+The Netdata Agent calculates the percentage of the system RAM swapped in the last 30 minutes. This alert is triggered in warning state if the percentage of the system RAM swapped in is more than 20%.
7
+
8
+### Troubleshoot the alert
9
+
10
+You can find the most resource greedy processes in your system, but if you receive this alert many times you must consider upgrading your system's RAM.
11
+
12
+- Find the processes that consume the most RAM
13
+
14
+Linux:
15
+```
16
+top -b -o +%MEM | head -n 22
17
+```
18
+
19
+FreeBSD:
20
+```
21
+top -b -o res | head -n 22
22
+```
23
+
24
+Here, you can see which processes are the main RAM consumers. Consider killing any of the main consumer processes that you do not need to avoid thrashing.
25
+
26
+Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
health/guides/README.md
deleted
-151
@@ -1,151 +0,0 @@
1
-# Alert Guides
2
-
3
-> This directory contains outdated guides on our alerts, and should be revisited and updated.
4
-
5
-# Review process
6
-
7
-Each guide must pass one technical review and one phrasal/grammatical review. Technical reviews will be equally assigned to the senior staff - considering the team's priorities.
8
-
9
-The review should focus on:
10
-
11
-- Is the guide useful and on topic?
12
-- Is it accurate? Does it have any unclear technical instruction? Please note which part.
13
-- Is it easy to read? If not, please note which part was not.
14
-- Is it rich in information? If not, please note which part we should explain more.
15
-
16
-Any other suggestions are welcomed.
17
-
18
-# Threshold presentation
19
-
20
-- Some alerts might use the conditional operator to determine on which state the alarm is. Let's break down this block
21
- of code:
22
-
23
-```sh
24
-warn: $this > (($status >= $WARNING) ? (75) : (85))
25
-crit: $this > (($status == $CRITICAL) ? (85) : (95))
26
-```
27
-
28
-In the above:
29
-
30
-If the alarm is currently a warning, then the threshold for being considered a warning is 75, otherwise it's 85.
31
-
32
-If the alarm is currently critical, then the threshold for being considered critical is 85, otherwise it's 95.
33
-
34
-Which in turn, results in the following behavior:
35
-
36
-While the value is rising, it will trigger a warning when it exceeds 85, and a critical alert when it exceeds 95.
37
-
38
-While the value is falling, it will return to a warning state when it goes below 85, and a normal state when it goes
39
-below 75.
40
-
41
-If the value is fluctuating between 80 and 90, then it will trigger a warning the first time it goes above 85
42
-and will remain a warning until it goes below 75 (or goes above 85).
43
-
44
-If the value is fluctuating between 90 and 100, then it will trigger a critical alert first time it goes
45
-above 95 and will remain a critical alert until it goes below 85 - at which point it will return to being a warning.
46
-
47
-- In our guides we should write a sentence similar to the example below:
48
-
49
-> - This alert is raised in a warning state when the percentage of used IPv4 TCP connections is greater than 80% and
50
- > less than 90%.
51
->- If the metric exceeds 90%, then the alert gets raised in critical state.
52
-
53
-Note: the thresholds can be customized, the above is just an example using the conditional operator.
54
-
55
-<br>
56
-
57
-**We will construct each post following the template below as closely as possible:**
58
-
59
-<details>
60
-<summary>Template</summary>
61
-
62
-# alarm_name
63
-
64
-## OS: <OS_Name>
65
-
66
-The initial topic for each alarm should provide as much of the following information as possible:
67
-
68
-- How this component works.
69
-
70
-- A description of what is being monitored.
71
-
72
-- A general description of what the specific alarm is about.
73
-
74
-- Negative effects of this abnormality.
75
-
76
-<details>
77
-<summary>More information about a highly technical detail</summary>
78
-We organize highly technical details into collapsible contents to keep the reader focused.
79
-</details>
80
-
81
-<details>
82
-<summary>References and sources</summary>
83
-
84
-1. [Descriptive sentece of the link1](https://community.netdata.cloud/)
85
-2. [Descriptive sentece of the link2](https://community.netdata.cloud/)
86
-
87
-General guidelines on the referenced resources
88
-
89
-1. Starting the guide with a quote, we use italic.
90
-
91
-_Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut
92
-labore et dolore magna aliqua._ <sup>[1](https://community.netdata.cloud/) </sup>
93
-
94
-2. Large quote with bullets
95
-
96
-You can see the info as somebody says on site | documented on the
97
-site <sup>[1](https://community.netdata.cloud/) </sup>
98
-
99
-- Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut
100
- labore et dolore magna aliqua
101
-
102
-- Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
103
- consequat.
104
-
105
-- Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
106
- pariatur.
107
-
108
-- Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id
109
- est laborum.
110
-
111
-3. Quotes that are just chunks that are 4-10 lines, use block quotes.
112
-
113
-> Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla convallis lobortis urna eu
114
-sollicitudin. Maecenas vel euismod lacus, vel pulvinar diam. Curabitur luctus metus vitae eros
115
-auctor, et dictum dolor commodo. Vivamus turpis ipsum, placerat et cursus sed, finibus sed velit.
116
-Pellentesque efficitur gravida nibh sit amet lacinia. Vestibulum blandit, enim eu sodales
117
-ullamcorper, dui velit viverra enim, sed fermentum ante orci a risus. Proin cursus, justo id tempor
118
-molestie, dolor ante tempor ante, id dignissim tortor lacus id purus. Nam diam nibh, gravida vitae
119
-sem at, venenatis suscipit ante. Duis quis nulla vel mauris facilisis laoreet et faucibus dui.
120
-Pellentesque quis venenatis est, vitae pharetra leo. <sup>[1](https://community.netdata.cloud/) </sup>
121
-
122
-4. For quotes shorter than 4 lines, use quotation marks.
123
-
124
-"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut
125
-labore et dolore magna aliqua" <sup>[1](https://community.netdata.cloud/) </sup>
126
-
127
-</details>
128
-
129
-### Troubleshooting section:
130
-
131
-<details>
132
-<summary>Check these actions regarding to this or that</summary>
133
-Any independent set of actions/suggestions is explained in a collapsible layout. Here we propose any action the user
134
-can take.
135
-</details>
136
-
137
-<details>
138
-<summary>A second set of actions.</summary>
139
-Here we propose any action the user can take.
140
-</details>
141
-
142
-## OS: <Second_OS_Name>
143
-
144
-If the alarm exists in different OS and the underlying mechanisms have major changes or different
145
-troubleshooting sections, we create a section for each OS.
146
-
147
-### Troubleshooting section:
148
-
149
-And more actions like above.
150
-
151
-</details>
\ No newline at end of file
health/guides/active_processes.md
new
+16
@@ -0,0 +1,16 @@
1
+### Understand the alert
2
+
3
+This alert indicates that your system's Process ID (PID) space utilization is at high levels, meaning that there is a limited number of PIDs available for new processes. A warning state occurs when the percentage of used PIDs is between 85-90%, and a critical state occurs when it is between 90-95%. If the value reaches 100%, no new processes can be started.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Identify high PID usage**: Use the `top` or `htop` command to identify processes with high PID usage. These processes may be causing the high PID space utilization.
8
+
9
+2. **Check for zombie processes**: Zombie processes are processes that have completed execution but still occupy a PID, leading to high PID space utilization. Use the `ps axo stat,ppid,pid,comm | grep -w defunct` command to identify zombie processes. If you find any, investigate their parent processes and, if necessary, restart or terminate them to release the occupied PIDs.
10
+
11
+3. **Monitor PID usage**: Continuously monitor your system's PID usage to understand normal behavior and identify potential issues before they become critical. You can use tools like Netdata for real-time monitoring.
12
+
13
+4. **Adjust PID limits**: If your system consistently experiences high PID space utilization, consider increasing the maximum number of PIDs allowed. On Linux systems, you can adjust the `kernel.pid_max` sysctl parameter. Make sure to set this value according to your system's capacity and workload requirements.
14
+
15
+5. **Optimize system performance**: Evaluate your system's workload and identify any specific processes or applications that are causing high PID usage. Optimize or limit these processes if necessary. Additionally, review your system's resource allocation and ensure there is sufficient capacity for process execution.
16
+
health/guides/adaptec_raid/adaptec_raid_ld_status.md
deleted
-65
@@ -1,65 +0,0 @@
1
-# adaptec_raid_ld_status
2
-
3
-## OS: Any
4
-
5
-A RAID controller is a card or chip located between the operating system and a storage drive (usually
6
-a hard drive). This is an alert about the Adaptec raid controller. The Netdata Agent checks
7
-the logical device statuses which are managed by your raid controller.
8
-
9
-This alert is triggered in critical state when a logical device state value is in degraded or failed
10
-state. This can indicate that one or more disks in your RAID configuration failed. Below you can
11
-find how tolerant is each raid configuration in cases of disk failures.
12
-
13
-<details>
14
-<summary>Fault tolerance for the most popular raid configurations </summary>
15
-
16
-- _RAID 0_ provides no fault tolerance. Any drive failures will cause data loss, so do not use this
17
- on a mission critical server.
18
-
19
-- _RAID 1_ configuration is best used for situations where capacity isn't a requirement but data
20
- protection is. This set up mirrors two disks so you can have 1 drive fail and still be able to
21
- recover your data.
22
-
23
-- _RAID 5_ can withstand a single drive failure with a tradeoff in performance.
24
-
25
-- _RAID 6_ can withstand two disk failures at one time.
26
-
27
-- _RAID 10_ can survive a single drive failure per array.
28
-
29
-</details>
30
-
31
-Your system manages your Adaptec raid controller via the ARCCONF command line tool. You can find
32
-more information about this utility from
33
-the [user's guide for the ARCCONF](https://download.adaptec.com/pdfs/user_guides/microsemi_cli_smarthba_smartraid_v3_00_23484_ug.pdf).
34
-
35
-### Troubleshooting section
36
-
37
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
38
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
39
-
40
-Your Adaptec RAID card will automatically start to rebuild a faulty hard drive when you
41
-replace it with a healthy one. Sometimes this operation may take some time or may not start
42
-automatically.
43
-
44
-
45
-<details>
46
-<summary>Manually change the status of your ld </summary>
47
-This action will trigger a rebuild on your RAID.
48
-
49
-1. Verify that a rebuild is not in process.
50
-
51
- ```
52
- root@netdata # arcconf GETSTATUS <Controller_num>
53
- ```
54
-
55
- 2. Check for idle/missing segments of logical devices.
56
-
57
-
58
-
59
-3. Manually change your ld status
60
-
61
- ```
62
- root@netdata # arcconf SETSTATE <Controller_num> LOGICALDRIVE <LD_num> OPTIMAL ADVANCED nocheck noprompt
63
- ```
64
-
65
-</details>
health/guides/adaptec_raid/adaptec_raid_pd_state.md
deleted
-50
@@ -1,50 +0,0 @@
1
-# adaptec_raid_pd_state
2
-
3
-## OS: Any
4
-
5
-A RAID controller is a card or chip located between the operating system and a storage drive (usually a hard drive). This is an alert about the Adaptec raid controller. The Netdata Agent checks
6
-the physical device statuses which are managed by your raid controller.
7
-
8
-This alert is triggered in critical state when the physical device is offline. Below you can
9
-find how tolerant each RAID configuration is in cases of disk failures.
10
-
11
-<details>
12
-<summary>Fault tolerance for the most popular raid configurations </summary>
13
-
14
-- _RAID 0_ provides no fault tolerance. Any drive failures will cause data loss, so do not use this
15
- on a mission critical server.
16
-
17
-- _RAID 1_ configuration is best used for situations where capacity isn't a requirement but data
18
- protection is. This set up mirrors two disks so you can have 1 drive fail and still be able to
19
- recover your data.
20
-
21
-- _RAID 5_ can withstand a single drive failure with a tradeoff in performance.
22
-
23
-- _RAID 6_ can withstand two disk failures at one time.
24
-
25
-- _RAID 10_ can survive a single drive failure per array.
26
-
27
-</details>
28
-
29
-Your system manages your Adaptec raid controller via the ARCCONF command line tool. You can find
30
-more information about this utility in
31
-the [user's guide for the ARCCONF](https://download.adaptec.com/pdfs/user_guides/microsemi_cli_smarthba_smartraid_v3_00_23484_ug.pdf).
32
-
33
-### Troubleshooting section
34
-
35
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
36
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
37
-
38
-
39
-<details>
40
-<summary>Verify a bad disk </summary>
41
-
42
-Check the smart report for the drives in your RAID controller:
43
-
44
- ```
45
- root@netdata # arcconf GETSMARTSTATS 1
46
- ```
47
-
48
-If a disk is degraded, you should consider replacing it. Your Adaptec RAID card will
49
- automatically start to rebuild a faulty hard drive when you replace it with a healthy one.
50
-
health/guides/adaptec_raid_ld_status.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alert is related to the Adaptec RAID controller, which manages the logical device statuses on your RAID configuration. When this alert is triggered in a critical state, it means that a logical device state value is in a degraded or failed state, indicating that one or more disks in your RAID configuration have failed.
4
+
5
+### Troubleshoot the alert
6
+
7
+Data is priceless. Before taking any action, ensure to have necessary backups in place. Netdata is not liable for any loss or corruption of any data, database, or software.
8
+
9
+Your Adaptec RAID card will automatically start rebuilding a faulty hard drive when you replace it with a healthy one. Sometimes this operation may take some time or may not start automatically.
10
+
11
+#### 1. Verify that a rebuild is not in process
12
+
13
+Check if the rebuild process is already running:
14
+
15
+```
16
+root@netdata # arcconf GETSTATUS <Controller_num>
17
+```
18
+
19
+Replace `<Controller_num>` with the number of your RAID controller.
20
+
21
+#### 2. Check for idle/missing segments of logical devices
22
+
23
+Examine the output of the previous command to find any segments that are idle or missing.
24
+
25
+#### 3. Manually change your ld status
26
+
27
+If the rebuild process hasn't started automatically, change the logical device (ld) status manually. This action will trigger a rebuild on your RAID:
28
+
29
+```
30
+root@netdata # arcconf SETSTATE <Controller_num> LOGICALDRIVE <LD_num> OPTIMAL ADVANCED nocheck noprompt
31
+```
32
+
33
+Replace `<Controller_num>` with the number of your RAID controller and `<LD_num>` with the number of the logical device.
34
+
35
+### Useful resources
36
+
37
+1. [Microsemi Adaptec ARCCONF User's Guide](https://download.adaptec.com/pdfs/user_guides/microsemi_cli_smarthba_smartraid_v3_00_23484_ug.pdf)
health/guides/adaptec_raid_pd_state.md
new
+66
@@ -0,0 +1,66 @@
1
+### Understand the Alert
2
+
3
+A RAID controller is a card or chip located between the operating system and a storage drive (usually a hard drive). This is an alert about the Adaptec raid controller. The Netdata Agent checks the physical device statuses which are managed by your raid controller.
4
+
5
+This alert is triggered in critical state when the physical device is offline.
6
+
7
+### Troubleshoot the Alert
8
+
9
+- Check the Offline Disk
10
+
11
+Use the `arcconf` CLI tool to identify which drive or drives are offline:
12
+
13
+```
14
+root@netdata # arcconf GETCONFIG 1 AL
15
+```
16
+
17
+This command will display the configuration of all the managed Adaptec RAID controllers in your system. Check the "DEVICE #" and "DEVICE_DEFINITION" fields for details about the offline devices.
18
+
19
+- Examine RAID Array Health
20
+
21
+Check the array health status to better understand the overall array's stability and functionality:
22
+
23
+```
24
+root@netdata # arcconf GETSTATUS 1
25
+```
26
+
27
+This will provide an overview of your RAID controller's health status, including the operational mode, failure state, and rebuild progress (if applicable).
28
+
29
+- Replace the Offline Disk
30
+
31
+Before replacing an offline disk, ensure that you have a current backup of your data. Follow these steps to replace the drive:
32
+
33
+1. Power off your system.
34
+2. Remove the offline drive.
35
+3. Insert the new drive.
36
+4. Power on your system.
37
+
38
+After the drive replacement, the Adaptec RAID card should automatically start rebuilding the faulty disk drive using the new disk. You can check the progress of the rebuild process with the `arcconf` command:
39
+
40
+```
41
+root@netdata # arcconf GETSTATUS 1
42
+```
43
+
44
+- Monitor Rebuild Progress
45
+
46
+It's essential to monitor the RAID array's rebuild process to ensure it completes successfully. Use the `arcconf` command to verify the rebuild status:
47
+
48
+```
49
+root@netdata # arcconf GETSTATUS 1
50
+```
51
+
52
+This command will display the progress and status of the rebuild process. Keep an eye on it until it's completed.
53
+
54
+- Verify RAID Array Health
55
+
56
+After the rebuild is complete, use the `arcconf` command again to verify the health status of the RAID array:
57
+
58
+```
59
+root@netdata # arcconf GETSTATUS 1
60
+```
61
+
62
+Make sure that the RAID array's status is "Optimal" or "Ready" and that the replaced disk drive is now online.
63
+
64
+### Useful Resources
65
+
66
+1. [Adaptec Command Line Interface User’s Guide](https://download.adaptec.com/pdfs/user_guides/microsemi_cli_smarthba_smartraid_v3_00_23484_ug.pdf)
health/guides/anomalies_anomaly_flags.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert, `anomalies_anomaly_flags`, is triggered when the Netdata Agent detects more than 10 anomalies in the past 2 minutes. Anomalies are events or observations that are significantly different from the majority of the data, raising suspicions about potential issues.
4
+
5
+### What does an anomaly mean?
6
+
7
+An anomaly is an unusual pattern, behavior, or event in your system's operations. These occurrences are typically unexpected and can be either positive or negative. In the context of this alert, the anomalies are most likely related to performance issues, such as a sudden spike in CPU usage, disk I/O, or network activity.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the source of the anomalies:
12
+
13
+ To understand the cause of these anomalies, you should examine the various charts in Netdata dashboard for potential performance issues. Look for sudden spikes, drops, or other irregular patterns in CPU usage, memory usage, disk I/O, and network activity.
14
+
15
+2. Check for any application or system errors:
16
+
17
+ Review system and application log files to detect any errors or warnings that may be related to the anomalies. Be sure to check logs of your applications, services, and databases for any error messages or unusual behavior.
18
+
19
+3. Monitor resource usage:
20
+
21
+ You can use the Anomalies tab in Netdata to dive deeper into what could be triggering anomalies in your infrastructure.
22
+
23
+4. Adjust thresholds or address the underlying issue:
24
+
25
+ If the anomalies are due to normal variations in your system's operation or expected spikes in resource usage, consider adjusting the threshold for this alert to avoid false positives. If the anomalies indicate an actual problem or point to a misconfiguration, take appropriate action to address the root cause.
26
+
27
+5. Observe the results:
28
+
29
+ After implementing changes or adjustments, continue monitoring the system using Netdata and other tools to ensure the anomalies are resolved and do not persist.
30
+
health/guides/anomalies_anomaly_probabilities.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert, `anomalies_anomaly_probabilities`, is generated by the Netdata agent when the average anomaly probability over the last 2 minutes is 50. An anomaly probability is a value calculated by the machine learning (ML) component in Netdata, aiming to detect unusual events or behavior in system metrics.
4
+
5
+### What is anomaly probability?
6
+
7
+Anomaly probability is a percentage calculated by the Netdata's ML feature that represents the probability of an observed metric value being considered an anomaly. A higher anomaly probability indicates a higher chance that the system behavior is deviating from its historical patterns or expected behavior.
8
+
9
+### What does an average anomaly probability of 50 mean?
10
+
11
+An average anomaly probability of 50 indicates that there might be some unusual events, metrics, or behavior in your monitored system. This might not necessarily indicate an issue, but rather, it raises suspicious deviations in the system metrics that are worth investigating.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Investigate the unusual events or behavior
16
+
17
+ The first step is to identify the metric(s) or series of metric values that are causing the alert. Look for changes in the monitored metrics or a combination of metrics that deviate significantly from their historical patterns.
18
+
19
+2. Check system performance and resource usage
20
+
21
+ Use the overview and anomalies tab to explore the metrics that could be contributing to anomalies.
22
+
23
+3. Inspect system logs
24
+
25
+ System logs can provide valuable information about unusual events or behaviors. Check system logs using tools like `journalctl`, `dmesg`, or `tail` for any error messages, warnings, or critical events that might be related to the anomaly.
26
+
27
+4. Review the alert settings
28
+
29
+ In some cases, the alert may be caused by overly strict or sensitive settings, leading to the triggering of false positives. Review the settings and consider adjusting the anomaly probability threshold, if necessary.
30
+
health/guides/apcupsd/apcupsd_10min_ups_load.md
deleted
-22
@@ -1,22 +0,0 @@
1
-# apcupsd_10min_ups_load
2
-
3
-**Power Supply | UPS**
4
-
5
-This is an alert about your American Power Conversion (APC) uninterruptible power supply (UPS) device.
6
-The Netdata Agent calculates the average UPS load over the last 10 minutes. Receiving
7
-this alert means that your UPS has a very high load. This issue may result
8
-in either your UPS transferring to bypass mode or shutting down as a safety
9
-measure due to overload. You should remove some attached equipment from the UPS.
10
-
11
-This alert is triggered in warning state when the average UPS load is between 70-80% and in critical
12
-state when it is between 85-95%.
13
-
14
-### Troubleshooting section:
15
-
16
-<details>
17
-<summary>Reduce the load on the UPS</summary>
18
-
19
-To avoid ungraceful shutdowns of your systems, consider reducing the load on this particular UPS.
20
-To achieve this, consider removing attached devices that are not mission critical.
21
-
22
-</details>
health/guides/apcupsd/apcupsd_last_collected_secs.md
deleted
-27
@@ -1,27 +0,0 @@
1
-# apcupsd_last_collected_secs
2
-
3
-**Power Supply | UPS device**
4
-
5
-This is an alert about your American Power Conversion (APC) uninterruptible power supply (UPS) device.
6
-The Netdata Agent monitors the number of seconds since
7
-the last successful data collection by querying the `apcaccess` tool. This alert indicates that no
8
-data collection has taken place for some time.
9
-
10
-### Troubleshooting section:
11
-
12
-<details>
13
-<summary>Check the APCU daemon </summary>
14
-
15
-1. Check the status of the APCU daemon
16
- ```
17
- root@netdata $ systemctl status apcupsd
18
- ```
19
-
20
-2. Check for obvious and common errors.
21
-
22
-
23
-3. Restart the APCU daemon, if needed
24
- ```
25
- root@netdata $ systemctl restart apcupsd
26
- ```
27
-</details>
health/guides/apcupsd/apcupsd_ups_charge.md
deleted
-21
@@ -1,21 +0,0 @@
1
-# apcupsd_ups_charge
2
-
3
-**Power Supply | UPS**
4
-
5
-This is an alert about your American Power Conversion (APC) uninterruptible power supply (UPS) device.
6
-The Netdata Agent calculates the average UPS charge over
7
-the last minute. The UPS is running on battery, and it will shut down if external power is not
8
-restored. You should prepare any attached equipment for shutdown.
9
-
10
-This alert is triggered in warning state when the average UPS charge is less than 100% and in
11
-critical state when it is less than 50%.
12
-
13
-### Troubleshooting section:
14
-
15
-<details>
16
-<summary>Prepare your machine for graceful shutdown</summary>
17
-
18
-If you can't restore the power supply to this UPC, you should prepare your machine for graceful
19
-shutdown.
20
-
21
-</details>
health/guides/apcupsd_10min_ups_load.md
new
+22
@@ -0,0 +1,22 @@
1
+### Understand the alert
2
+
3
+This alert is related to your APC uninterruptible power supply (UPS) device. If you receive this alert, it means that your UPS is experiencing high load, which could result in it entering bypass mode or shutdown to protect the device. The alert is triggered in a warning state when the average UPS load is between 70-80% and in a critical state when it is between 85-95%.
4
+
5
+### Troubleshoot the alert
6
+
7
+Follow these steps to address the high load on your UPS device:
8
+
9
+1. **Identify devices connected to the UPS**: Make a list of all the devices connected to the UPS. This list could include computers, servers, routers, and other essential equipment.
10
+
11
+2. **Assess the importance of each device**: Prioritize the devices connected to the UPS based on their importance to your network infrastructure. Determine which devices are mission-critical and which ones can be temporarily disconnected without causing significant disruptions.
12
+
13
+3. **Disconnect non-critical devices**: Once you have assessed the importance of the connected devices, disconnect any non-critical devices to reduce the load on the UPS. This action will help ensure that the mission-critical devices continue to receive power during a utility failure.
14
+
15
+4. **Consider additional UPS capacity**: If you frequently receive this alert or are unable to disconnect enough devices to reduce the load on the UPS, consider adding additional UPS capacity to your infrastructure. This additional capacity could come in the form of additional UPS units or a larger UPS with a higher output capacity.
16
+
17
+5. **Monitor the UPS load**: After disconnecting non-critical devices or adding additional UPS capacity, continue monitoring the UPS load using the Netdata Agent to ensure the load stays within acceptable limits. If the alert persists, you may need to reevaluate your infrastructure and device connections.
18
+
19
+### Useful resources
20
+
21
+1. [APC UPS Management](https://www.schneider-electric.com/en/product-category/870_IDSof_0145_NET/!ut/p/z1/hZBNbsIwDMD3ejK_Sh4xWb1tgwEfkFLCVKrUYKngigoXrWtJ_gSCk_bm0RfbT707TIAx8WuuDIwdwmK28Q2YY3Agq3XkKAGwpTEgZUPAHD7HxAqcAkgxV7OuBHSkrBSV7eGzvdN1jQZSYhNnhP7YvfFGttb8j7LlPvTXSuC7V-q1DXce8XtWjZmfrniT7ufcTtT8AKaWHzA!!/dz/d5/L2dBISEvZ0FBIS9nQSEh/)
22
+2. [Understanding the Different Types of UPS Systems](https://www.apc.com/us/en/faqs/FA157448/)
health/guides/apcupsd_last_collected_secs.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+This alert is related to your American Power Conversion (APC) uninterruptible power supply (UPS) device. The Netdata Agent monitors the number of seconds since the last successful data collection by querying the `apcaccess` tool. If you receive this alert, it means that no data collection has taken place for some time, which might indicate a problem with the APC UPS device or connection.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Verify the `apcaccess` tool is installed and functioning properly
8
+ ```
9
+ $ apcaccess status
10
+ ```
11
+ This command should provide you with a status display of the UPS. If the command is not found, you may need to install the `apcaccess` tool.
12
+
13
+2. Check the APC UPS daemon
14
+
15
+ a. Check the status of the APC UPS daemon
16
+ ```
17
+ $ systemctl status apcupsd
18
+ ```
19
+
20
+ b. Check for obvious and common errors, such as wrong device path, incorrect permissions, or configuration issues in `/etc/apcupsd/apcupsd.conf`.
21
+
22
+ c. If needed, restart the APC UPS daemon
23
+ ```
24
+ $ systemctl restart apcupsd
25
+ ```
26
+
27
+3. Inspect system logs
28
+
29
+ Check the system logs for any error messages related to APC UPS or `apcupsd`, which might give more insights into the issue.
30
+
31
+4. Verify UPS Connection
32
+
33
+ Ensure that the UPS device is properly connected to your server, both physically (USB/Serial) and in the configuration file (`/etc/apcupsd/apcupsd.conf`).
34
+
35
+5. Update Netdata configuration
36
+
37
+ If the issue is still not resolved, you can try updating the Netdata configuration file for the `apcupsd_last_collected_secs` collector.
38
+
39
+6. Check your UPS device
40
+
41
+ If all previous steps have been completed and the issue persists, your UPS device might be faulty. Consider contacting the manufacturer for support or replace the device with a known-good unit.
42
+
43
+### Useful resources
44
+
45
+1. [Netdata - APC UPS monitoring](https://learn.netdata.cloud/docs/data-collection/ups/apc-ups)
46
+2. [`apcupsd` - Power management and control software for APC UPS](https://github.com/apcupsd/apcupsd)
health/guides/apcupsd_ups_charge.md
new
+45
@@ -0,0 +1,45 @@
1
+### Understand the alert
2
+
3
+This alert is related to the charge level of your American Power Conversion (APC) uninterruptible power supply (UPS) device. When the UPS charge level drops below a certain threshold, you receive an alert indicating that the system is running on battery and may shut down if external power is not restored soon.
4
+
5
+- Warning state: UPS charge < 100%
6
+- Critical state: UPS charge < 50%
7
+
8
+The main purpose of a UPS is to provide a temporary power source to connected devices in case of a power outage. As the battery charge decreases, you need to either restore the power supply or prepare the equipment for a graceful shutdown.
9
+
10
+### Troubleshoot the alert
11
+
12
+1. Check the UPS charge level and status
13
+
14
+ To investigate the current status and charge level of the UPS, you can use the `apcaccess` command which provides information about the APC UPS device.
15
+
16
+ ```
17
+ apcaccess
18
+ ```
19
+
20
+ Look for the `STATUS` and `BCHARGE` fields in the output.
21
+
22
+2. Restore the power supply (if possible)
23
+
24
+ If the power outage is temporary or local (e.g. due to a tripped circuit breaker), try to restore the power supply to the UPS by fixing the issue or connecting the UPS to a different power source.
25
+
26
+3. Prepare for a graceful shutdown
27
+
28
+ If you cannot restore power to the UPS, or if the battery charge is critically low, you should immediately prepare your machine and any connected devices for a graceful shutdown. This will help to avoid data loss or system corruption due to an abrupt shutdown.
29
+
30
+ For Linux systems, you can execute the following command to perform a graceful shutdown:
31
+
32
+ ```
33
+ sudo shutdown -h +1 "UPS battery is low. The system will shut down in 1 minute."
34
+ ```
35
+
36
+ For Windows systems, open a command prompt with admin privileges and execute the following command to perform a graceful shutdown:
37
+
38
+ ```
39
+ shutdown /s /t 60 /c "UPS battery is low. The system will shut down in 1 minute."
40
+ ```
41
+
42
+4. Monitor UPS and system logs
43
+
44
+ Keep an eye on UPS and system logs to detect any issues with the power supply or UPS device. This can help you stay informed about the system's status and troubleshoot any potential problems.
45
+
health/guides/bcache/bcache_cache_dirty.md
deleted
-38
@@ -1,38 +0,0 @@
1
-# bcache_cache_dirty
2
-
3
-## OS: Any
4
-
5
-This alarm presents the percentage of `bcache` cache space used for `dirty` data and `metadata`. If this alarm is
6
-raised, it means that your SSD cache is too small, and overpopulated with said data.
7
-
8
-You can view `bcache_cache_dirty` as the `bcache` analogous metric to `dirty memory`. `dirty memory` is memory that has
9
-been changed but has not yet been written out to disk. For example, you make a change to a file but do not save it. These
10
-temporary changes are stored in memory, waiting to be written to disk.
11
-So `dirty` data on `bcache` is data that is stored on the cache disk and waits to be written to the backing device (
12
-Normally your HDD).
13
-
14
-> `bcache` is a cache in the block layer of the Linux Kernel. **It allows fast storage devices**, as SSDs
15
-> (Solid State Drives), **to act as a cache for slower storage devices**, such as HDDs (Hard Disk Drives). As a result,
16
-> **hybrid volumes are made with performance improvements**. Generally, a cache device is divided up into `buckets`,
17
-> matching the physical disk's erase blocks.
18
-
19
-> `dirty` data is data in the cache that has not been written to the backing device (normally your HDD). So when the
20
-> system shuts down, the cache device and the backing device are not safe to be separated.
21
-> `metadata` in general, is data that provides information about other data.
22
-
23
-Links:
24
-[kernel.org](https://www.kernel.org/doc/html/latest/admin-guide/bcache.html#)
25
-[Wikipedia](https://en.wikipedia.org/wiki/Bcache)
26
-[Bcache](https://wiki.archlinux.org/title/bcache)
27
-[Bcache: Caching beyond just RAM](https://lwn.net/Articles/394672/)
28
-
29
-### Troubleshooting section
30
-
31
-<details>
32
-<summary>Upgrade your cache's capacity</summary>
33
-
34
-The alarm is raised when there is more than 70% *(for warning status)* of your cache populated by `dirty` data and
35
-`metadata`, it means that your current cache device doesn't have the capacity to support your workflow. Using a bigger
36
-capacity device as cache can solve the problem.
37
-
38
-</details>
\ No newline at end of file
health/guides/bcache/bcache_cache_errors.md
deleted
-23
@@ -1,23 +0,0 @@
1
-# bcache_cache_errors
2
-
3
-## OS: Any
4
-
5
-This alarm presents the number of `bcache` read races in the last minute. The `bucket` was reused and invalidated while
6
-reading from the cache. When this occurs, the data is reread from the backing device.
7
-
8
-> `bcache` is a cache in the block layer of the Linux Kernel. **It allows fast storage devices**, as SSDs
9
-> (Solid State Drives), **to act as a cache for slower storage devices**, such as HDDs (Hard Disk Drives). As a result,
10
-> **hybrid volumes are made with performance improvements**. Generally, a cache device is divided up into `buckets`,
11
-> matching the physical disk's erase blocks.
12
-
13
-There is a mechanism where `bcache` can keep the cache disk full (typically your SSD), and when it needs to write more
14
-data, it selects a `bucket`, invalidates it, and removes all pointers from it. **The alarm got triggered, because while
15
-there was a reading operation from the cache** *(meaning the data is stored inside a bucket)* **that bucket got
16
-invalidated so the read operation couldn't be completed.** Following up, the data is reread from the backing device
17
-(normally your HDD).
18
-
19
-Links:
20
-[kernel.org](https://www.kernel.org/doc/html/latest/admin-guide/bcache.html#)
21
-[Wikipedia](https://en.wikipedia.org/wiki/Bcache)
22
-[Bcache](https://wiki.archlinux.org/title/bcache)
23
-[Bcache: Caching beyond just RAM](https://lwn.net/Articles/394672/)
\ No newline at end of file
health/guides/bcache_cache_dirty.md
new
+74
@@ -0,0 +1,74 @@
1
+### Understand the Alert
2
+
3
+`bcache` is a cache in the block layer of the Linux Kernel. **It allows fast storage devices**, as SSDs (Solid State Drives), **to act as a cache for slower storage devices**, such as HDDs (Hard Disk Drives). As a result, **hybrid volumes are made with performance improvements**. Generally, a cache device is divided up into `buckets`, matching the physical disk's erase blocks.
4
+
5
+This alert indicates that your SSD cache is too small, and overpopulated with data.
6
+
7
+You can view `bcache_cache_dirty` as the `bcache` analogous metric to `dirty memory`. `dirty memory` is memory that has been changed but has not yet been written out to disk. For example, you make a change to a file but do not save it. These temporary changes are stored in memory, waiting to be written to disk. So `dirty` data on `bcache` is data that is stored on the cache disk and waits to be written to the backing device (Normally your HDD).
8
+
9
+`dirty` data is data in the cache that has not been written to the backing device (normally your HDD). So when the system shuts down, the cache device and the backing device are not safe to be separated.
10
+`metadata` in general, is data that provides information about other data.
11
+
12
+### Troubleshoot the Alert
13
+
14
+- Upgrade your cache's capacity
15
+
16
+This alert is raised when there is more than 70% *(for warning status)* of your cache populated by `dirty` data and `metadata`, it means that your current cache device doesn't have the capacity to support your workflow. Using a bigger
17
+capacity device as cache can solve the problem.
18
+
19
+- Monitor cache usage regularly
20
+
21
+Keep an eye on the cache usage regularly to understand the pattern of how your cache gets filled up with dirty data and metadata. This can help you better manage the cache and take proactive measures before facing a performance bottleneck.
22
+
23
+ To monitor cache usage, use `cat` command on the cache device's sysfs directory like this:
24
+
25
+ ```
26
+ cat /sys/fs/bcache/<CACHE_DEV_UUID>/cache0/bcache/stats_five_minute/cache_hit_ratio
27
+ ```
28
+
29
+ Replace `<CACHE_DEV_UUID>` with your cache device's UUID.
30
+
31
+- Periodically write dirty data to the backing device
32
+
33
+If the cache becomes frequently filled with dirty data, you can try periodically writing dirty data to the backing device to create more space in the cache. This can especially help if your caching device isn't frequently reaching its full capacity.
34
+
35
+ To perform this, you can use the `cron` job scheduler to run a command that flushes dirty data to the HDD periodically. Add the following line to your crontab:
36
+
37
+ ```
38
+ */5 * * * * echo writeback > /sys/fs/bcache/<CACHE_DEV_UUID>/cache0/bcache/writeback_rate_debug
39
+ ```
40
+
41
+ Replace `<CACHE_DEV_UUID>` with your cache device's UUID. This configuration will flush the dirty data to the backing device every 5 minutes.
42
+
43
+- Check for I/O bottlenecks
44
+
45
+If you experience performance issues with bcache, it's essential to identify the cause, which could be I/O bottlenecks. Look for any I/O errors or an overloaded I/O subsystem that may be affecting your cache device's performance.
46
+
47
+ To check I/O statistics, you can use tools like `iotop`, `iostat` or `vmstat`:
48
+
49
+ ```bash
50
+ iotop
51
+ iostat -x -d -z -t 5 5 # run 5 times with a 5-second interval between each report
52
+ vmstat -d
53
+ ```
54
+
55
+ Analyze the output and look for any signs of a bottleneck, such as excessive disk utilization, slow transfer speeds, or high I/O wait times.
56
+
57
+- Optimize cache configuration
58
+
59
+Review your current cache configuration and make sure it's optimized for your system's workload. In some cases, adjusting cache settings could help improve the hit ratio and reduce the amount of dirty data.
60
+
61
+ To view the bcache settings:
62
+
63
+ ```
64
+ cat /sys/fs/bcache/<CACHE_DEV_UUID>/cache0/bcache/*
65
+ ```
66
+
67
+ Replace `<CACHE_DEV_UUID>` with your cache device's UUID.
68
+
69
+ You can also make changes to the cache settings by echoing the new values to the corresponding sysfs files. Please refer to the [Cache Settings section in the Bcache documentation](https://www.kernel.org/doc/Documentation/bcache.txt) for more details.
70
+
71
+### Useful resources
72
+
73
+1. [Bcache documentation](https://www.kernel.org/doc/Documentation/bcache.txt)
74
+2. [Arch Linux Wiki: Bcache](https://wiki.archlinux.org/title/bcache)
health/guides/bcache_cache_errors.md
new
+66
@@ -0,0 +1,66 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of read races in the last minute on a `bcache` system has increased. A read race occurs when a `bucket` is reused and invalidated while it's being read from the cache. In this situation, the data is reread from the slower backing device.
4
+
5
+### What is bcache?
6
+
7
+`bcache` is a cache within the block layer of the Linux kernel. It enables fast storage devices, such as SSDs (Solid State Drives), to act as a cache for slower storage devices like HDDs (Hard Disk Drives). This creates hybrid volumes with improved performance. A cache device is usually divided into `buckets` that match the physical disk's erase blocks.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Verify the current `bcache` cache errors:
12
+
13
+ ```
14
+ grep bcache_cache_errors /sys/fs/bcache/*/stats_total/*
15
+ ```
16
+
17
+ This command will show the total number of cache errors for all `bcache` devices.
18
+
19
+2. Identify the affected backing device:
20
+
21
+ You can determine the affected backing device by checking the `/sys/fs/bcache` directory. Look for the symbolic link that points to the problematic device.
22
+
23
+ ```
24
+ ls -l /sys/fs/bcache
25
+ ```
26
+
27
+ This command will show the list of devices with corresponding names.
28
+
29
+3. Monitor the cache device's performance:
30
+
31
+ Use `iostat` to check the cache device's I/O performance.
32
+
33
+ ```
34
+ iostat -x -h -p /dev/YOUR_CACHE_DEVICE
35
+ ```
36
+
37
+ Note that you should replace `YOUR_CACHE_DEVICE` with the actual cache device name.
38
+
39
+4. Check the utilization of the cache and backing devices:
40
+
41
+ Use the following commands to check the utilization percentage of the cache and backing devices:
42
+
43
+ ```
44
+ # for the cache device (/dev/YOUR_CACHE_DEVICE)
45
+ cat /sys/block/YOUR_CACHE_DEVICE/bcache/utilization
46
+
47
+ # for the backing device (/dev/YOUR_BACKING_DEVICE)
48
+ cat /sys/block/YOUR_BACKING_DEVICE/bcache/utilization
49
+ ```
50
+
51
+ Replace `YOUR_CACHE_DEVICE` and `YOUR_BACKING_DEVICE` with the respective device names.
52
+
53
+5. Optimize the cache:
54
+
55
+ - If the cache utilization is high, consider increasing the cache size or adding more cache devices.
56
+ - If the cache device is heavily utilized, consider upgrading it to a faster SSD.
57
+ - In case the read races persist, consider using a [priority caching strategy](https://www.kernel.org/doc/html/latest/admin-guide/bcache.html#priority-caching).
58
+
59
+ You may also need to review your system's overall I/O load and adjust your caching strategy accordingly.
60
+
61
+### Useful resources
62
+
63
+1. [Bcache: Caching beyond just RAM](https://lwn.net/Articles/394672/)
64
+2. [Kernel Documentation - Bcache](https://www.kernel.org/doc/html/latest/admin-guide/bcache.html)
65
+3. [Arch Linux Wiki - Bcache](https://wiki.archlinux.org/title/bcache)
66
+4. [Wikipedia - Bcache](https://en.wikipedia.org/wiki/Bcache)
health/guides/beanstalk_number_of_tubes.md
new
+40
@@ -0,0 +1,40 @@
1
+### Understand the alert
2
+
3
+This alert monitors the current number of tubes on a Beanstalk server. If the number of tubes drops below 5, you will receive a warning. Tubes are used as queues for jobs in Beanstalk, and having a low number of tubes may indicate an issue with service configuration or job processing.
4
+
5
+### What are tubes in Beanstalk?
6
+
7
+Beanstalk is a simple, fast work queue service that allows you to distribute tasks among different workers. In Beanstalk, *tubes* are essentially queues for jobs. Each tube stores jobs with specific priorities, Time-to-run (TTR) values, and other relevant data. Workers can reserve jobs from specific tubes, process the jobs, and delete them when finished.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check Beanstalk server status.
12
+
13
+ Use the following command to display the current Beanstalk server status:
14
+
15
+ ```
16
+ beanstalkctl stats
17
+ ```
18
+
19
+ Look for the current number of tubes (`current-tubes`). If it is too low (below 5), proceed to the next step.
20
+
21
+2. Identify recently deleted tubes.
22
+
23
+ Determine if any tubes have been deleted recently. Check your application logs, Beanstalk daemon logs, or discuss with your development team to find out if any tube deletion is intentional.
24
+
25
+3. Check for misconfigurations or code issues.
26
+
27
+ Inspect your Beanstalk server configuration and verify that the expected tubes are correctly defined. Additionally, review the application code and deployment scripts to ensure that tubes are being created and used as intended.
28
+
29
+4. Check worker status and processing.
30
+
31
+ Verify that your worker processes are running and processing jobs from the tubes correctly. If there are issues with worker processes, it may lead to unused or unprocessed tubes.
32
+
33
+5. Create missing tubes if necessary.
34
+
35
+ If you've identified that some tubes are missing and need to be created, add the required tubes using your application code or Beanstalk configuration.
36
+
37
+### Useful resources
38
+
39
+1. [Beanstalk Introduction](https://beanstalkd.github.io/)
40
+2. [Beanstalk Protocol Documentation](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt)
health/guides/beanstalk_server_buried_jobs.md
new
+32
@@ -0,0 +1,32 @@
1
+### Understand the alert
2
+
3
+This alert is related to the `Beanstalk` message queue system and is triggered when there are buried jobs in the queue across all tubes. A buried job is one that has encountered an issue during processing by the consumer, so it remains in the queue waiting for manual action. This alert is raised in a warning state if there are more than 0 buried jobs and in a critical state if there are more than 10.
4
+
5
+### What are buried jobs?
6
+
7
+Buried jobs are tasks that have faced an error or issue during processing by the consumer, and as a result, have been `buried`. This means these jobs remain in the queue, awaiting manual intervention for them to be processed again. The presence of buried jobs does not affect the processing of new jobs in the queue.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the buried jobs: Use the `beanstalk-tool` to inspect the Beanstalk server and list the buried jobs in the tubes. If you don't have `beanstalk-tool`, install it using pip:
12
+
13
+ ```
14
+ pip install beanstalkc
15
+ beanstalk-tool <beanstalk server host>:<beanstalk server port> stats_tube <tube_name>
16
+ ```
17
+
18
+2. Examine the buried jobs: To investigate the cause of the buried jobs, find related logs, either from the Beanstalk server or from the consumer application. Analyzing the logs can lead to the root cause of the problem.
19
+
20
+3. Fix the issue: Once you identify the cause, resolve the issue in either the consumer application, or if necessary, in the Beanstalk server configuration.
21
+
22
+4. Kick the buried jobs: After resolving the issue, you need to manually kick the buried jobs back into the queue for processing. Use the following command with `beanstalk-tool`:
23
+
24
+ ```
25
+ beanstalk-tool <beanstalk server host>:<beanstalk server port> kick <number_of_jobs_to_kick> --tube=<tube_name>
26
+ ```
27
+
28
+5. Monitor the queue: After kicking the buried jobs, monitor the queue and ensure that the jobs are processed without encountering more errors.
29
+
30
+### Useful resources
31
+
32
+1. [Beanstalk Documentation](https://beanstalkd.github.io/)
health/guides/beanstalk_tube_buried_jobs.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of buried jobs in each beanstalkd tube. If you receive this alert, it means that there are jobs that have been buried, and you need to investigate the cause. The warning threshold is set at more than zero buried jobs, and the critical threshold is set at more than ten buried jobs.
4
+
5
+### What are buried jobs?
6
+
7
+In Beanstalkd, buried jobs are jobs that were moved to the buried state deliberately or jobs that have failed repeatedly. They're kept in a separate queue and will not be processed by workers until they're explicitly handled or deleted.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the Beanstalkd logs for any errors or pertinent information related to the buried jobs. You can find the logs in the `/var/log/beanstalkd.log` file (the default log file location) or any other custom location defined for Beanstalkd.
12
+
13
+2. Use the `beanstalk-console` or a similar tool to inspect the buried jobs to determine their causes. You can download `beanstalk-console` [here](https://github.com/ptrofimov/beanstalk_console).
14
+
15
+3. Review the applications or workers that are interacting with the affected tubes to find any possible issues or bugs.
16
+
17
+4. If the buried jobs are blocking the processing of other jobs, consider moving them to another tube with higher priority or increase the number of workers processing the tube.
18
+
19
+5. If the buried jobs are safe to delete or requeue, do so to clear the count and alleviate the alert. You can use the following commands to kick or delete jobs using the `beanstalk-cli`:
20
+ ```
21
+ beanstalk-cli kick-job [<job-id>]
22
+ beanstalk-cli delete-job [<job-id>]
23
+ ```
24
+
25
+6. If none of the steps above help mitigate the issue, consider contacting the sysadmin or developers of the application using Beanstalkd.
26
+
27
+### Useful resources
28
+
29
+1. [Beanstalkd Protocol](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt)
30
+2. [Beanstalk_console - a web-based beanstalk queue server console](https://github.com/ptrofimov/beanstalk_console)
health/guides/bind_rndc_stats_file_size.md
new
+55
@@ -0,0 +1,55 @@
1
+### Understand the alert
2
+
3
+This alert is related to the `BIND` DNS server and its statistics file. If you receive this alert, it means that the file size has crossed a predetermined threshold (warning state at 512 MB and critical state at 1024 MB). This can negatively impact the performance of your DNS server.
4
+
5
+### What is BIND?
6
+
7
+BIND (Berkeley Internet Name Domain) is an open-source DNS server that provides DNS services on Linux servers. It is widely used to implement DNS services on the internet.
8
+
9
+### What is the BIND statistics file?
10
+
11
+BIND keeps track of various metrics and statistics in a `*.stats` file, which is defined by the `statistics-file` option in the BIND configuration. This file can grow in size over time as it accumulates more data.
12
+
13
+### Troubleshoot the alert
14
+
15
+If you receive this alert, it's time to take action to reduce the size of the BIND statistics file. You can do this by:
16
+
17
+1. Review and determine which statistics in the file are necessary:
18
+
19
+ - You can check the content of the statistics file to identify the metrics collected.
20
+ - Consult the documentation or support forums of the relevant network services to determine which statistics are important for your use case.
21
+
22
+2. Update your BIND configuration to reduce the collection of unnecessary statistics:
23
+
24
+ - Edit your BIND configuration file (usually `/etc/bind/named.conf` or `/etc/named.conf`) to remove or comment out the unnecessary statistics options.
25
+ - If you're not sure which options to remove, search the BIND documentation or seek assistance from administrators who have experience with BIND configuration.
26
+
27
+3. Restart the BIND service to apply your changes:
28
+
29
+ ```
30
+ sudo systemctl restart bind9
31
+ ```
32
+
33
+ (replace `bind9` with the service name of your BIND installation if different)
34
+
35
+4. Manually delete or rotate the large statistics file:
36
+
37
+ - To delete the file, use this command:
38
+
39
+ ```
40
+ sudo rm /path/to/your/stats/file
41
+ ```
42
+
43
+ - To rotate the file, you can use the `logrotate` utility:
44
+
45
+ ```
46
+ sudo logrotate --force /etc/logrotate.d/bind
47
+ ```
48
+
49
+ (update the configuration file path if your set up uses a different location)
50
+
51
+After completing these steps, monitor the size of the BIND statistics file to ensure it doesn't grow beyond your desired threshold.
52
+
53
+### Useful resources
54
+
55
+1. [BIND Documentation](https://bind9.readthedocs.io/en/latest/)
health/guides/boinc/boinc_active_tasks.md
deleted
-22
@@ -1,22 +0,0 @@
1
-# boinc_active_tasks
2
-
3
-**Computing | BOINC**
4
-
5
-The Berkeley Open Infrastructure for Network Computing (BOINC) is an open-source middleware system
6
-for volunteer computing and grid computing. The Netdata Agent monitors the average number of active
7
-tasks over the last 10 minutes.
8
-
9
-### Troubleshooting sections
10
-
11
-
12
-<details>
13
-
14
-<summary>Restart the BOINC client</summary>
15
-
16
-1. Restart BOINC client, in most of the linux distros
17
-
18
- ```
19
- root@netdata # /etc/init.d/boinc-client restart
20
- ```
21
-
22
-</details>
health/guides/boinc/boinc_compute_errors.md
deleted
-23
@@ -1,23 +0,0 @@
1
-# boinc_compute_errors
2
-
3
-**Computing | BOINC**
4
-
5
-The Berkeley Open Infrastructure for Network Computing (BOINC) is an open-source middleware system
6
-for volunteer computing and grid computing. The Netdata Agent monitors the average number of compute
7
-errors over the last 10 minutes.
8
-
9
-<details>
10
-
11
-<summary>Restart the BOINC client</summary>
12
-
13
-1. Abort the running task
14
-
15
-2. Restart BOINC client, in most of the linux distros
16
-
17
- ```
18
- root@netdata # /etc/init.d/boinc-client restart
19
- ```
20
-
21
-</details>
22
-
23
-
health/guides/boinc/boinc_total_tasks.md
deleted
-30
@@ -1,30 +0,0 @@
1
-# boinc_total_tasks
2
-
3
-**Computing | BOINC**
4
-
5
-The Berkeley Open Infrastructure for Network Computing (BOINC) is an open-source middleware system
6
-for volunteer computing and grid computing. The Netdata Agent monitors the average number of total
7
-tasks over the last 10 minutes.
8
-
9
-### Troubleshooting sections
10
-
11
-<details>
12
-
13
-<summary>Verify the project status</summary>
14
-
15
-Verify that the projects you contribute are not suspended. Check if the project has queued tasks to
16
-be done (https://boinc.berkeley.edu/projects.php)
17
-
18
-</details>
19
-
20
-<details>
21
-
22
-<summary>Restart the BOINC client</summary>
23
-
24
-1. In most of the linux distros
25
-
26
- ```
27
- root@netdata # /etc/init.d/boinc-client restart
28
- ```
29
-
30
-</details>
health/guides/boinc/boinc_upload_errors.md
deleted
-19
@@ -1,19 +0,0 @@
1
-# boinc_upload_errors
2
-
3
-**Computing | BOINC**
4
-
5
-The Berkeley Open Infrastructure for Network Computing (BOINC) is an open-source middleware system
6
-for volunteer computing and grid computing. The Netdata Agent calculates the average number of
7
-failed uploads over the last 10 minutes.
8
-
9
-For some reason your node couldn't upload the files manifested.
10
-
11
-### Troubleshooting section
12
-
13
-<details>
14
-<summary>Check for new BOINC client's version</summary>
15
-
16
-You can consult the [boinc client downloads page](https://boinc.berkeley.edu/download_all.php) to
17
-download the _recommended_ version for your system
18
-
19
-</details>
\ No newline at end of file
health/guides/boinc_active_tasks.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+This alert monitors the BOINC (Berkeley Open Infrastructure for Network Computing) client's average number of active tasks over the last 10 minutes. If you receive this alert, it means that there might be an issue with your BOINC tasks or client.
4
+
5
+### Troubleshoot the alert
6
+
7
+- Check the BOINC client logs
8
+
9
+1. Locate the BOINC client log file, usually in `/var/lib/boinc-client/`.
10
+2. Inspect the log file for any issues or error messages related to task execution, connection, or client behavior.
11
+
12
+- Check the status of the BOINC client
13
+
14
+1. To check the status, run the following command:
15
+
16
+ ```
17
+ sudo /etc/init.d/boinc-client status
18
+ ```
19
+
20
+2. If the client is not running, start it using:
21
+
22
+ ```
23
+ sudo /etc/init.d/boinc-client start
24
+ ```
25
+
26
+- Restart the BOINC client
27
+
28
+1. Restart the BOINC client, in most of the Linux distros:
29
+
30
+ ```
31
+ sudo /etc/init.d/boinc-client restart
32
+ ```
33
+
34
+- Ensure your system has adequate resources
35
+
36
+Monitoring and managing your computer resources (CPU, memory, disk space) can help ensure smooth operation of the BOINC client and its tasks. If your system is low on resources, consider freeing up space or upgrading your hardware.
37
+
38
+- Update the BOINC client
39
+
40
+Make sure your BOINC client is up-to-date by checking the official BOINC website (https://boinc.berkeley.edu/download.php) for the latest version.
41
+
42
+### Useful resources
43
+
44
+1. [BOINC User Manual](https://boinc.berkeley.edu/wiki/User_manual)
health/guides/boinc_compute_errors.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+The `boinc_compute_errors` alert indicates that your system has experienced an increase in the average number of compute errors over the last 10 minutes when running BOINC tasks. It is important to identify the cause of these errors and take appropriate action to minimize the impact on your system.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the BOINC client logs
8
+ BOINC client logs can provide useful information about compute errors. The logs can usually be found in the `/var/lib/boinc-client/` directory. Look for any error messages or information that could indicate the cause of the issues.
9
+
10
+2. Verify the system requirements
11
+ Ensure that your system meets the minimum requirements to run the BOINC tasks. This includes checking the CPU, RAM, disk space, and any other device-specific requirements. If your system does not meet the requirements, you may need to upgrade your hardware or reduce the number of tasks you are running simultaneously.
12
+
13
+3. Check for software and hardware compatibility
14
+ Some BOINC tasks may have specific hardware or software requirements, such as GPU support or compatibility with certain operating systems. Check the BOINC project documentation for any specific requirements you may be missing.
15
+
16
+4. Update the BOINC client software
17
+ Make sure your BOINC client software is up-to-date, as outdated versions can cause errors or unexpected behavior. You can check for updates and download the latest version from the [official BOINC website](https://boinc.berkeley.edu/download.php).
18
+
19
+5. Restart the BOINC client
20
+ If the issue persists, try restarting the BOINC client following the steps provided in the alert:
21
+
22
+ - Abort the running task
23
+ - Restart the BOINC client:
24
+ ```
25
+ root@netdata # /etc/init.d/boinc-client restart
26
+ ```
27
+
28
+6. Seek assistance from the BOINC community
29
+ If you continue to experience issues after following these troubleshooting steps, consider seeking assistance from the BOINC community through forums or mailing lists.
30
+
31
+### Useful resources
32
+
33
+1. [BOINC hardware and software requirements](https://boinc.berkeley.edu/wiki/System_requirements)
health/guides/boinc_total_tasks.md
new
+32
@@ -0,0 +1,32 @@
1
+### Understand the alert
2
+
3
+This alert monitors the average number of total tasks for the BOINC system over the last 10 minutes. If you receive this alert, it means that there is a deviation in the number of total tasks for your BOINC system, which may indicate an issue with the projects, the client, or even the tasks themselves.
4
+
5
+### Troubleshoot the alert
6
+
7
+#### Verify the project status
8
+
9
+1. Verify that the projects you contribute to are not suspended. You can check if the project has queued tasks to be done on the [BOINC projects page](https://boinc.berkeley.edu/projects.php).
10
+
11
+2. Access your BOINC Manager, go to the _Projects_ tab, and check if the projects you contribute to are in the correct state (Active or Running). If a project is suspended, you can select it and click _Resume_ to reactivate it.
12
+
13
+#### Investigate task issues
14
+
15
+1. Access your BOINC Manager and go to the _Tasks_ tab to check the status of the current tasks. Look for any _Failed_, _Error_, or _Postponed_ tasks.
16
+
17
+2. If there are failed tasks, try to reset them by selecting the task, right-clicking on it, and choosing _Update_ or _Reset_. Be aware that resetting a task will discard any progress made on it.
18
+
19
+#### Restart the BOINC client
20
+
21
+1. For most Linux distributions:
22
+
23
+ ```
24
+ sudo /etc/init.d/boinc-client restart
25
+ ```
26
+
27
+2. For other operating systems or custom installations, refer to the BOINC's documentation for restarting the client: https://boinc.berkeley.edu/wiki/Stop_or_restart_BOINC
28
+
29
+#### Check system resources
30
+
31
+BOINC tasks may fail or slow down if there is not enough system resources (CPU, RAM, or Disk Space) available. Monitor your system performance using tools like `top`, `free`, and `df`, and make adjustments if necessary to ensure that BOINC has enough resources to complete tasks.
32
+
health/guides/boinc_upload_errors.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert indicates that your BOINC node is experiencing an increase in the average number of failed uploads over the last 10 minutes. Failed uploads can affect the overall efficiency of your BOINC setup and may result in lost work and wasted computational resources.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check for a new BOINC client's version
8
+
9
+ Verify if there's a new version of the BOINC client available for your system on the [BOINC client downloads page](https://boinc.berkeley.edu/download_all.php). If there's a new version available, download and install the _recommended_ version.
10
+
11
+2. Verify BOINC project configuration
12
+
13
+ Ensure that your BOINC client is properly configured to work on the projects you're participating in. Double-check your account login credentials, project URLs, and other settings in the BOINC client. Refer to the [BOINC User Manual](https://boinc.berkeley.edu/wiki/User_manual) for more information on configuring your BOINC client.
14
+
15
+3. Check network connectivity
16
+
17
+ Verify that your system has a stable network connection and there are no firewall or proxy issues that might be blocking your BOINC client from uploading files. If necessary, adjust your firewall settings or connect to a different network.
18
+
19
+4. Inspect BOINC client logs
20
+
21
+ Consult the BOINC client logs to gain insight into the upload errors. The logs can be found in the client's data directory. Refer to the [BOINC log file documentation](https://boinc.berkeley.edu/wiki/Log_Files) for more information on how to read and analyze the logs.
22
+
23
+5. Contact project support
24
+
25
+ If you're still experiencing upload issues after following the steps above, consider reaching out to the support forums or mailing lists of the relevant BOINC project. The project's support team might be able to offer assistance or guidance in resolving your issue.
26
+
27
+### Useful resources
28
+
29
+1. [BOINC User Manual](https://boinc.berkeley.edu/wiki/User_manual)
30
+2. [BOINC Downloads](https://boinc.berkeley.edu/download_all.php)
health/guides/btrfs/btrfs_allocated.md
deleted
-211
@@ -1,211 +0,0 @@
1
-# btrfs_allocated
2
-
3
-## OS: Any
4
-
5
-*Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing
6
-on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots,
7
-checksums, and integral multi-device spanning in Linux file systems.* [[1]](https://wiki.archlinux.org/title/btrfs)
8
-
9
-Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical
10
-disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data),
11
-metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the
12
-filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This
13
-metric tracks space usage in the first allocation stage.
14
-The Netdata Agent monitors the percentage of allocated Btrfs physical disk space.
15
-
16
-<details>
17
-<summary>Subvolumes in Btrfs </summary>
18
-
19
-> A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one).
20
-Most other POSIX filesystems have a single mountable root. Btrfs has an independent mountable root for the volume (top
21
-level subvolume) and for each subvolume. A Btrfs volume can contain more than a single filetree; it can contain a forest
22
-of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace.
23
->
24
-> A subvolume in Btrfs is not the same as a LVM logical volume or a ZFS subvolume. With LVM, a logical volume is a block
25
-device in its own right, which could, for example, contain any other filesystem or container like dm-crypt, MD RAID,
26
-etc. This is not the case with Btrfs.
27
->
28
-> A Btrfs subvolume root directory differs from a directory in that each subvolume defines a distinct inode number
29
-> space (
30
-distinct inodes in different subvolumes can have the same inumber) and each inode under a subvolume has a distinct
31
-device number (as reported by stat(2)). Each subvolume root can be accessed as implicitly mounted via the volume (top
32
-level subvolume) root, if that is mounted, or it can be mounted in its own right. [[2]](https://btrfs.wiki.kernel.org)
33
-
34
-
35
-</details>
36
-
37
-<details>
38
-<summary>Snapshots in Btrfs</summary>
39
-
40
-> A snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs's COW
41
-> capabilities.
42
->
43
-> Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new
44
-snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, use mv to rename the old
45
-subvolume to a temporary location, and then rename the snapshot to the original name. You can then remount the
46
-subvolume.
47
->
48
-> At this point, the original subvolume may be deleted, if desired. Since a snapshot is a subvolume, snapshots of
49
-snapshots are also possible.
50
->
51
-> Caution: Care must be taken when snapshots are created that are then visible to any user (e.g. when they're created
52
-> in a nested layout) as this may have security implications. Of course, the snapshot will have the same permissions
53
-> as the subvolume from which it was created at the time it was, but these permissions may be tightened later on, while
54
-> those of the snapshot wouldn't change, possibly allowing access to files that shouldn't be accessible anymore.
55
-> Similarly, especially on the system's "main" filesystem, the snapshot would contain any files (for example, setuid
56
-> programs) of the state when it was created. In the meantime however, security updates may have been rolled out on
57
-> the original subvolume, but when the snapshot is accessible (and for example the vulnerable setuid has been accessible
58
-> before) a user could still invoke it. [[2]](https://btrfs.wiki.kernel.org)
59
-
60
-</details>
61
-
62
-<details>
63
-<summary>Useful commands for btrfs tool</summary>
64
-
65
-You can see some commands from the man pages of Btrfs in [Ubuntu man pages (Bionic)](https://manpages.ubuntu.com/manpages/bionic/man8)
66
-
67
-- `btrfs subvolume` is used to create/delete/list/show btrfs subvolumes and snapshots. For example,
68
- with `btrfs subvolume list <path>`, you can list the subvolumes present in the filesystem `<path>`.
69
-
70
-
71
-- `btrfs filesystem` is used to perform several whole filesystem level tasks, including all the regular filesystem
72
- operations like resizing, space stats, label setting/getting, and defragmentation. For example
73
- with `btrfs filesystem show [<path>|<uuid>|<device>|<label>]` you can see the Btrfs filesystem with some additional
74
- info about devices and space allocation.
75
-
76
-
77
-- `btrfs balance` can balance (restripe) the allocated extents across all of the existing devices. The primary purpose
78
- of the balance feature is to spread block groups across all devices so they match constraints defined by the
79
- respective profiles. The balance operation is cancellable by the user. The on-disk state of the filesystem is always
80
- consistent so an unexpected interruption (eg. system crash, reboot) does not corrupt the filesystem. The progress of
81
- the balance operation is temporarily stored as an internal state and will be resumed upon mount, unless the mount
82
- option skip_balance is specified. **Running balance without filters will take a lot of time as it basically rewrites
83
- the entire filesystem and needs to update all block pointers**.
84
-
85
-
86
-- `btrfs device` command group is used to manage devices of the btrfs filesystems. For example, with `btrfs device add`
87
- command, you can add new devices to a mounted filesystem.
88
-
89
-
90
-- `btrfs rescue` is used to try to recover a damaged btrfs filesystem.
91
-
92
-
93
-- `btrfs scrub <subcommand>` scrub command attempts to report and repair bad blocks on Btrfs file systems.
94
- **Scrubbing is performed in the background by default**.
95
-
96
-</details>
97
-
98
-
99
-<details>
100
-<summary>References and sources:</summary>
101
-
102
-1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
103
-1. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
104
-
105
-
106
-</details>
107
-
108
-### Troubleshooting section:
109
-
110
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is
111
-not liable for any loss or corruption of any data, database, or software.
112
-
113
-<details>
114
-<summary>Add more physical space</summary>
115
-
116
-Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a
117
-new disk to a filesystem is:
118
-
119
-1. Determine which disk you want to add and in which path
120
- ```
121
- root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
122
- ```
123
-
124
-1. If you get an error that the drive is already mounted, you might have to unmount
125
- ```
126
- root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
127
- ```
128
-1. See the newly added disk
129
- ```
130
- root@netdata~ # btrfs filesystem show
131
- Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
132
- Total devices 2 FS bytes used 192.00KiB
133
- devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
134
- devid 2 size 10.01GiB used 0.00B path /dev/sdb
135
- ```
136
-
137
-1. Balance the system to make use of the new drive.
138
- ```
139
- root@netdata~ # btrfs filesystem balance <path>
140
- ```
141
-
142
-</details>
143
-
144
-<details>
145
-<summary>Delete snapshots</summary>
146
-
147
-You can identify and delete snapshots that you no longer need.
148
-
149
-1. Find the snapshots for a specific path.
150
- ```
151
- root@netdata~ # sudo btrfs subvolume list -s <path>
152
- ```
153
-
154
-1. Delete a snapshot that you do not need any more.
155
- ```
156
- root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
157
- ```
158
-
159
-</details>
160
-
161
-<details>
162
-<summary>Enable a compression mechanism</summary>
163
-
164
-> The `compress=alg` mount option into the `fstab` configuration file (or in the `mount` procedure) enables automatically
165
-considering every file for compression, where `alg` is either `zlib`, `lzo`, `zstd`, or `no` (for no compression). Using
166
-this option, Btrfs will check if compressing the first portion of the data shrinks it. If it does, the entire write to
167
-that file will be compressed. If it does not, none of it is compressed. With this option, if the first portion of the
168
-write does not shrink, no compression will be applied to the write even if the rest of the data would shrink
169
-tremendously. This is done to prevent making the disk wait to start writing until all of the data to be written is fully
170
-given to Btrfs and compressed. [[1]](https://wiki.archlinux.org/title/btrfs)
171
-
172
-1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd`
173
- compression algorithm.
174
-
175
- ```
176
- root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
177
- ```
178
-
179
-</details>
180
-
181
-<details>
182
-<summary>Enable a deduplication mechanism</summary>
183
-
184
-Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a
185
-file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks
186
-of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
187
-
188
-Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd
189
-party, and it is strongly suggested that you check their status before you decide to use them.
190
-
191
-- [duperemove](https://github.com/markfasheh/duperemove)
192
-- [bees](https://github.com/Zygo/bees)
193
-- [dduper](https://github.com/lakshmipathi/dduper)
194
-
195
-</details>
196
-
197
-<details>
198
-<summary>Perform a balance</summary>
199
-
200
-Especially in a Btrfs with multiple disks, there might be unevenly allocated data/metadata into the disks.
201
-
202
-```
203
-root@netdata # btrfs balance start -musage=10 -dusage=10 -susage=5 /mount/point
204
-```
205
-
206
-> This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this
207
-> example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends
208
-> with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter
209
-> until at least some chunks are relocated.
210
-
211
-</details>
health/guides/btrfs/btrfs_data.md
deleted
-213
@@ -1,213 +0,0 @@
1
-# btrfs_data
2
-
3
-## OS: Any
4
-
5
-*Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing
6
-on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots,
7
-checksums, and integral multi-device spanning in Linux file systems.* [[1]](https://wiki.archlinux.org/title/btrfs)
8
-
9
-Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical
10
-disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data),
11
-metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the
12
-filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This
13
-metric tracks space usage in the first allocation stage.
14
-The Netdata Agent monitors the percentage of used Btrfs data space.
15
-
16
-<details>
17
-<summary>Subvolumes in Btrfs </summary>
18
-
19
-> A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one).
20
-Most other POSIX filesystems have a single mountable root. Btrfs has an independent mountable root for the volume (top
21
-level subvolume) and for each subvolume. A Btrfs volume can contain more than a single filetree; it can contain a forest
22
-of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace.
23
-
24
-> A subvolume in Btrfs is not the same as a LVM logical volume or a ZFS subvolume. With LVM, a logical volume is a block
25
-device in its own right, which could, for example, contain any other filesystem or container like dm-crypt, MD RAID,
26
-etc. This is not the case with Btrfs.
27
-
28
-> A Btrfs subvolume root directory differs from a directory in that each subvolume defines a distinct inode number
29
-> space (
30
-distinct inodes in different subvolumes can have the same inumber) and each inode under a subvolume has a distinct
31
-device number (as reported by stat(2)). Each subvolume root can be accessed as implicitly mounted via the volume (top
32
-level subvolume) root, if that is mounted, or it can be mounted in its own right. [[2]](https://btrfs.wiki.kernel.org)
33
-
34
-
35
-</details>
36
-
37
-<details>
38
-<summary>Snapshots in Btrfs</summary>
39
-
40
-> A snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs's COW
41
-> capabilities.
42
->
43
-> Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new
44
-snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, use mv to rename the old
45
-subvolume to a temporary location, and then rename the snapshot to the original name. You can then remount the
46
-subvolume.
47
->
48
-> At this point, the original subvolume may be deleted, if desired. Since a snapshot is a subvolume, snapshots of
49
-snapshots are also possible.
50
->
51
-> Caution: Care must be taken when snapshots are created that are then visible to any user (e.g. when they're created
52
-> in a nested layout) as this may have security implications. Of course, the snapshot will have the same permissions
53
-> as the subvolume from which it was created at the time it was, but these permissions may be tightened later on, while
54
-> those of the snapshot wouldn't change, possibly allowing access to files that shouldn't be accessible anymore.
55
-> Similarly, especially on the system's "main" filesystem, the snapshot would contain any files (for example, setuid
56
-> programs) of the state when it was created. In the meantime however, security updates may have been rolled out on
57
-> the original subvolume, but when the snapshot is accessible (and for example the vulnerable setuid has been accessible
58
-> before) a user could still invoke it. [[2]](https://btrfs.wiki.kernel.org)
59
-
60
-</details>
61
-
62
-<details>
63
-<summary>Useful commands for btrfs tool</summary>
64
-
65
-You can see some commands from the man pages of Btrfs in [Ubuntu man pages (Bionic)](https://manpages.ubuntu.com/manpages/bionic/man8)
66
-
67
-- `btrfs subvolume` is used to create/delete/list/show btrfs subvolumes and snapshots. For example,
68
- with `btrfs subvolume list <path>`, you can list the subvolumes present in the filesystem `<path>`.
69
-
70
-
71
-- `btrfs filesystem` is used to perform several whole filesystem level tasks, including all the regular filesystem
72
- operations like resizing, space stats, label setting/getting, and defragmentation. For example
73
- with `btrfs filesystem show [<path>|<uuid>|<device>|<label>]` you can see the Btrfs filesystem with some additional
74
- info about devices and space allocation.
75
-
76
-
77
-- `btrfs balance` can balance (restripe) the allocated extents across all of the existing devices. The primary purpose
78
- of the balance feature is to spread block groups across all devices so they match constraints defined by the
79
- respective profiles. The balance operation is cancellable by the user. The on-disk state of the filesystem is always
80
- consistent so an unexpected interruption (eg. system crash, reboot) does not corrupt the filesystem. The progress of
81
- the balance operation is temporarily stored as an internal state and will be resumed upon mount, unless the mount
82
- option skip_balance is specified. **Running balance without filters will take a lot of time as it basically rewrites
83
- the entire filesystem and needs to update all block pointers**.
84
-
85
-
86
-- `btrfs device` command group is used to manage devices of the btrfs filesystems. For example, with `btrfs device add`
87
- command, you can add new devices to a mounted filesystem.
88
-
89
-
90
-- `btrfs rescue` is used to try to recover a damaged btrfs filesystem.
91
-
92
-
93
-- `btrfs scrub <subcommand>` scrub command attempts to report and repair bad blocks on Btrfs file systems.
94
- **Scrubbing is performed in the background by default**.
95
-
96
-</details>
97
-
98
-
99
-<details>
100
-<summary>References and sources:</summary>
101
-
102
-1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
103
-1. [https://btrfs.wiki.kernel.org](https://btrfs.wiki.kernel.org)
104
-
105
-
106
-</details>
107
-
108
-### Troubleshooting section:
109
-
110
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is
111
-not liable for any loss or corruption of any data, database, or software.
112
-
113
-
114
-<details>
115
-<summary>Add more physical space</summary>
116
-
117
-Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a
118
-new disk to a filesystem is:
119
-
120
-1. Determine which disk you want to add and in which path
121
- ```
122
- root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
123
- ```
124
-
125
-1. If you get an error that the drive is already mounted, you might have to unmount
126
- ```
127
- root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
128
- ```
129
-1. See the newly added disk
130
- ```
131
- root@netdata~ # btrfs filesystem show
132
- Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
133
- Total devices 2 FS bytes used 192.00KiB
134
- devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
135
- devid 2 size 10.01GiB used 0.00B path /dev/sdb
136
- ```
137
-
138
-1. Balance the system to make use of the new drive.
139
- ```
140
- root@netdata~ # btrfs filesystem balance <path>
141
- ```
142
-
143
-</details>
144
-
145
-<details>
146
-<summary>Delete snapshots</summary>
147
-
148
-You can identify and delete snapshots that you no longer need.
149
-
150
-1. Find the snapshots for a specific path.
151
- ```
152
- root@netdata~ # sudo btrfs subvolume list -s <path>
153
- ```
154
-
155
-1. Delete a snapshot that you do not need any more.
156
- ```
157
- root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
158
- ```
159
-
160
-</details>
161
-
162
-<details>
163
-<summary>Enable a compression mechanism</summary>
164
-
165
-> The `compress=alg` mount option into the `fstab` configuration file (or in the `mount` procedure) enables
166
-automatically
167
-considering every file for compression, where `alg` is either `zlib`, `lzo`, `zstd`, or `no` (for no compression). Using
168
-this option, Btrfs will check if compressing the first portion of the data shrinks it. If it does, the entire write to
169
-that file will be compressed. If it does not, none of it is compressed. With this option, if the first portion of the
170
-write does not shrink, no compression will be applied to the write even if the rest of the data would shrink
171
-tremendously. This is done to prevent making the disk wait to start writing until all of the data to be written is fully
172
-given to Btrfs and compressed. [[1]](https://wiki.archlinux.org/title/btrfs)
173
-
174
-1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd`
175
- compression algorithm.
176
-
177
- ```
178
- root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
179
- ```
180
-
181
-</details>
182
-
183
-<details>
184
-<summary>Enable a deduplication mechanism</summary>
185
-
186
-Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a
187
-file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks
188
-of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
189
-
190
-Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd
191
-party, and it is strongly suggested that you check their status before you decide to use them.
192
-
193
-- [duperemove](https://github.com/markfasheh/duperemove)
194
-- [bees](https://github.com/Zygo/bees)
195
-- [dduper](https://github.com/lakshmipathi/dduper)
196
-
197
-</details>
198
-
199
-<details>
200
-<summary>Perform a balance</summary>
201
-
202
-Especially in a Btrfs with multiple disks, there might be unevenly allocated data/metadata into the disks.
203
-
204
-```
205
-root@netdata # btrfs balance start -musage=10 -dusage=50 -susage=5 /mount/point
206
-```
207
-
208
-> This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this
209
-> example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends
210
-> with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter
211
-> until at least some chunks are relocated.
212
-
213
-</details>
health/guides/btrfs/btrfs_metadata.md
deleted
-214
@@ -1,214 +0,0 @@
1
-# btrfs_metadata
2
-
3
-## OS: Any
4
-
5
-*Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing
6
-on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots,
7
-checksums, and integral multi-device spanning in Linux file systems.* [[1]](https://wiki.archlinux.org/title/btrfs)
8
-
9
-Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical
10
-disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data),
11
-metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the
12
-filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This
13
-metric tracks space usage in the first allocation stage.
14
-The Netdata Agent monitors the percentage of used Btrfs metadata space.
15
-
16
-<details>
17
-<summary>Subvolumes in Btrfs </summary>
18
-
19
-> A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one).
20
-Most other POSIX filesystems have a single mountable root. Btrfs has an independent mountable root for the volume (top
21
-level subvolume) and for each subvolume. A Btrfs volume can contain more than a single filetree; it can contain a forest
22
-of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace.
23
->
24
-> A subvolume in Btrfs is not the same as a LVM logical volume or a ZFS subvolume. With LVM, a logical volume is a block
25
-device in its own right, which could, for example, contain any other filesystem or container like dm-crypt, MD RAID,
26
-etc. This is not the case with Btrfs.
27
->
28
-> A Btrfs subvolume root directory differs from a directory in that each subvolume defines a distinct inode number
29
-> space (
30
-distinct inodes in different subvolumes can have the same inumber) and each inode under a subvolume has a distinct
31
-device number (as reported by stat(2)). Each subvolume root can be accessed as implicitly mounted via the volume (top
32
-level subvolume) root, if that is mounted, or it can be mounted in its own right. [[2]](https://btrfs.wiki.kernel.org)
33
-
34
-
35
-</details>
36
-
37
-<details>
38
-<summary>Snapshots in Btrfs</summary>
39
-
40
-> A snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs's COW
41
-> capabilities.
42
->
43
-> Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new
44
-snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, use mv to rename the old
45
-subvolume to a temporary location, and then rename the snapshot to the original name. You can then remount the
46
-subvolume.
47
->
48
-> At this point, the original subvolume may be deleted, if desired. Since a snapshot is a subvolume, snapshots of
49
-snapshots are also possible.
50
->
51
-> Caution: Care must be taken when snapshots are created that are then visible to any user (e.g. when they're created
52
-> in a nested layout) as this may have security implications. Of course, the snapshot will have the same permissions
53
-> as the subvolume from which it was created at the time it was, but these permissions may be tightened later on, while
54
-> those of the snapshot wouldn't change, possibly allowing access to files that shouldn't be accessible anymore.
55
-> Similarly, especially on the system's "main" filesystem, the snapshot would contain any files (for example, setuid
56
-> programs) of the state when it was created. In the meantime however, security updates may have been rolled out on
57
-> the original subvolume, but when the snapshot is accessible (and for example the vulnerable setuid has been accessible
58
-> before) a user could still invoke it. [[2]](https://btrfs.wiki.kernel.org)
59
-
60
-</details>
61
-
62
-<details>
63
-<summary>Useful commands for btrfs tool</summary>
64
-
65
-You can see some commands from the man pages of Btrfs in [Ubuntu man pages (Bionic)](https://manpages.ubuntu.com/manpages/bionic/man8)
66
-
67
-- `btrfs subvolume` is used to create/delete/list/show btrfs subvolumes and snapshots. For example,
68
- with `btrfs subvolume list <path>`, you can list the subvolumes present in the filesystem `<path>`.
69
-
70
-
71
-- `btrfs filesystem` is used to perform several whole filesystem level tasks, including all the regular filesystem
72
- operations like resizing, space stats, label setting/getting, and defragmentation. For example
73
- with `btrfs filesystem show [<path>|<uuid>|<device>|<label>]` you can see the Btrfs filesystem with some additional
74
- info about devices and space allocation.
75
-
76
-
77
-- `btrfs balance` can balance (restripe) the allocated extents across all of the existing devices. The primary purpose
78
- of the balance feature is to spread block groups across all devices so they match constraints defined by the
79
- respective profiles. The balance operation is cancellable by the user. The on-disk state of the filesystem is always
80
- consistent so an unexpected interruption (eg. system crash, reboot) does not corrupt the filesystem. The progress of
81
- the balance operation is temporarily stored as an internal state and will be resumed upon mount, unless the mount
82
- option skip_balance is specified. **Running balance without filters will take a lot of time as it basically rewrites
83
- the entire filesystem and needs to update all block pointers**.
84
-
85
-
86
-- `btrfs device` command group is used to manage devices of the btrfs filesystems. For example, with `btrfs device add`
87
- command, you can add new devices to a mounted filesystem.
88
-
89
-
90
-
91
-- `btrfs rescue` is used to try to recover a damaged btrfs filesystem.
92
-
93
-
94
-- `btrfs scrub <subcommand>` scrub command attempts to report and repair bad blocks on Btrfs file systems.
95
- **Scrubbing is performed in the background by default**.
96
-
97
-</details>
98
-
99
-<details>
100
-<summary>References and sources:</summary>
101
-
102
-1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
103
-1. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
104
-
105
-
106
-</details>
107
-
108
-### Troubleshooting section:
109
-
110
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is
111
-not liable for any loss or corruption of any data, database, or software.
112
-
113
-
114
-<details>
115
-<summary>Add more physical space</summary>
116
-
117
-Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a
118
-new disk to a filesystem is:
119
-
120
-1. Determine which disk you want to add and in which path
121
- ```
122
- root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
123
- ```
124
-
125
-1. If you get an error that the drive is already mounted, you might have to unmount
126
- ```
127
- root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
128
- ```
129
-1. See the newly added disk
130
- ```
131
- root@netdata~ # btrfs filesystem show
132
- Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
133
- Total devices 2 FS bytes used 192.00KiB
134
- devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
135
- devid 2 size 10.01GiB used 0.00B path /dev/sdb
136
- ```
137
-
138
-1. Balance the system to make use of the new drive.
139
- ```
140
- root@netdata~ # btrfs filesystem balance <path>
141
- ```
142
-
143
-</details>
144
-
145
-<details>
146
-<summary>Delete snapshots</summary>
147
-
148
-You can identify and delete snapshots that you no longer need.
149
-
150
-1. Find the snapshots for a specific path.
151
- ```
152
- root@netdata~ # sudo btrfs subvolume list -s <path>
153
- ```
154
-
155
-1. Delete a snapshot that you do not need any more.
156
- ```
157
- root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
158
- ```
159
-
160
-</details>
161
-
162
-<details>
163
-<summary>Enable a compression mechanism</summary>
164
-
165
-> The `compress=alg` mount option into the `fstab` configuration file (or in the `mount` procedure) enables
166
-> automatically
167
-considering every file for compression, where `alg` is either `zlib`, `lzo`, `zstd`, or `no` (for no compression). Using
168
-this option, Btrfs will check if compressing the first portion of the data shrinks it. If it does, the entire write to
169
-that file will be compressed. If it does not, none of it is compressed. With this option, if the first portion of the
170
-write does not shrink, no compression will be applied to the write even if the rest of the data would shrink
171
-tremendously. This is done to prevent making the disk wait to start writing until all of the data to be written is fully
172
-given to Btrfs and compressed. [[1]](https://wiki.archlinux.org/title/btrfs)
173
-
174
-1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd`
175
- compression algorithm.
176
-
177
-
178
- ```
179
- root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
180
- ```
181
-
182
-</details>
183
-
184
-<details>
185
-<summary>Enable a deduplication mechanism</summary>
186
-
187
-Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a
188
-file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks
189
-of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
190
-
191
-Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd
192
-party, and it is strongly suggested that you check their status before you decide to use them.
193
-
194
-- [duperemove](https://github.com/markfasheh/duperemove)
195
-- [bees](https://github.com/Zygo/bees)
196
-- [dduper](https://github.com/lakshmipathi/dduper)
197
-
198
-</details>
199
-
200
-<details>
201
-<summary>Perform a balance</summary>
202
-
203
-Especially in a Btrfs with multiple disks, there might be unevenly allocated data/metadata into the disks.
204
-
205
-```
206
-root@netdata # btrfs balance start -musage=50 -dusage=10 -susage=5 /mount/point
207
-```
208
-
209
-> This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this
210
-> example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends
211
-> with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter
212
-> until at least some chunks are relocated.
213
-
214
-</details>
health/guides/btrfs/btrfs_system.md
deleted
-211
@@ -1,211 +0,0 @@
1
-# btrfs_system
2
-
3
-## OS: Any
4
-
5
-*Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing
6
-on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots,
7
-checksums, and integral multi-device spanning in Linux file systems.* [[1]](https://wiki.archlinux.org/title/btrfs)
8
-
9
-Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical
10
-disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data),
11
-metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the
12
-filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This
13
-metric tracks space usage in the first allocation stage.
14
-The Netdata Agent monitors the percentage of used Btrfs system space.
15
-
16
-<details>
17
-<summary>Subvolumes in Btrfs </summary>
18
-
19
-> A Btrfs subvolume is an independently mountable POSIX filetree and not a block device (and cannot be treated as one).
20
-Most other POSIX filesystems have a single mountable root. Btrfs has an independent mountable root for the volume (top
21
-level subvolume) and for each subvolume. A Btrfs volume can contain more than a single filetree; it can contain a forest
22
-of filetrees. A Btrfs subvolume can be thought of as a POSIX file namespace.
23
->
24
-> A subvolume in Btrfs is not the same as a LVM logical volume or a ZFS subvolume. With LVM, a logical volume is a block
25
-device in its own right, which could, for example, contain any other filesystem or container like dm-crypt, MD RAID,
26
-etc. This is not the case with Btrfs.
27
->
28
-> A Btrfs subvolume root directory differs from a directory in that each subvolume defines a distinct inode number space
29
-(distinct inodes in different subvolumes can have the same inumber) and each inode under a subvolume has a distinct
30
-device number (as reported by stat(2)). Each subvolume root can be accessed as implicitly mounted via the volume (top
31
-level subvolume) root, if that is mounted, or it can be mounted in its own right. [[2]](https://btrfs.wiki.kernel.org)
32
-
33
-
34
-</details>
35
-
36
-<details>
37
-<summary>Snapshots in Btrfs</summary>
38
-
39
-> A snapshot is a subvolume that shares its data (and metadata) with some other subvolume, using Btrfs's COW
40
-> capabilities.
41
->
42
-> Once a [writable] snapshot is made, there is no difference in status between the original subvolume, and the new
43
-snapshot subvolume. To roll back to a snapshot, unmount the modified original subvolume, use mv to rename the old
44
-subvolume to a temporary location, and then rename the snapshot to the original name. You can then remount the
45
-subvolume.
46
->
47
-> At this point, the original subvolume may be deleted, if desired. Since a snapshot is a subvolume, snapshots of
48
-snapshots are also possible.
49
->
50
-> Caution: Care must be taken when snapshots are created that are then visible to any user (e.g. when they're created
51
-> in a nested layout) as this may have security implications. Of course, the snapshot will have the same permissions
52
-> as the subvolume from which it was created at the time it was, but these permissions may be tightened later on, while
53
-> those of the snapshot wouldn't change, possibly allowing access to files that shouldn't be accessible anymore.
54
-> Similarly, especially on the system's "main" filesystem, the snapshot would contain any files (for example, setuid
55
-> programs) of the state when it was created. In the meantime however, security updates may have been rolled out on
56
-> the original subvolume, but when the snapshot is accessible (and for example the vulnerable setuid has been accessible
57
-> before) a user could still invoke it. [[2]](https://btrfs.wiki.kernel.org)
58
-
59
-</details>
60
-
61
-<details>
62
-<summary>Useful commands for btrfs tool</summary>
63
-
64
-You can see some commands from the man pages of Btrfs in [Ubuntu man pages (Bionic)](https://manpages.ubuntu.com/manpages/bionic/man8)
65
-
66
-- `btrfs subvolume` is used to create/delete/list/show btrfs subvolumes and snapshots. For example,
67
- with `btrfs subvolume list <path>`, you can list the subvolumes present in the filesystem `<path>`.
68
-
69
-
70
-- `btrfs filesystem` is used to perform several whole filesystem level tasks, including all the regular filesystem
71
- operations like resizing, space stats, label setting/getting, and defragmentation. For example
72
- with `btrfs filesystem show [<path>|<uuid>|<device>|<label>]` you can see the Btrfs filesystem with some additional
73
- info about devices and space allocation.
74
-
75
-
76
-- `btrfs balance` can balance (restripe) the allocated extents across all of the existing devices. The primary purpose
77
- of the balance feature is to spread block groups across all devices so they match constraints defined by the
78
- respective profiles. The balance operation is cancellable by the user. The on-disk state of the filesystem is always
79
- consistent so an unexpected interruption (eg. system crash, reboot) does not corrupt the filesystem. The progress of
80
- the balance operation is temporarily stored as an internal state and will be resumed upon mount, unless the mount
81
- option skip_balance is specified. **Running balance without filters will take a lot of time as it basically rewrites
82
- the entire filesystem and needs to update all block pointers**.
83
-
84
-
85
-- `btrfs device` command group is used to manage devices of the btrfs filesystems. For example, with `btrfs device add`
86
- command, you can add new devices to a mounted filesystem.
87
-
88
-
89
-- `btrfs rescue` is used to try to recover a damaged btrfs filesystem.
90
-
91
-
92
-- `btrfs scrub <subcommand>` scrub command attempts to report and repair bad blocks on Btrfs file systems.
93
- **Scrubbing is performed in the background by default**.
94
-
95
-</details>
96
-
97
-
98
-<details>
99
-<summary>References and sources:</summary>
100
-
101
-1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
102
-1. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
103
-
104
-
105
-</details>
106
-
107
-### Troubleshooting section:
108
-
109
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is
110
-not liable for any loss or corruption of any data, database, or software.
111
-
112
-<details>
113
-<summary>Add more physical space</summary>
114
-
115
-Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a
116
-new disk to a filesystem is:
117
-
118
-1. Determine which disk you want to add and in which path
119
- ```
120
- root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
121
- ```
122
-
123
-1. If you get an error that the drive is already mounted, you might have to unmount
124
- ```
125
- root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
126
- ```
127
-1. See the newly added disk
128
- ```
129
- root@netdata~ # btrfs filesystem show
130
- Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
131
- Total devices 2 FS bytes used 192.00KiB
132
- devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
133
- devid 2 size 10.01GiB used 0.00B path /dev/sdb
134
- ```
135
-
136
-1. Balance the system to make use of the new drive.
137
- ```
138
- root@netdata~ # btrfs filesystem balance <path>
139
- ```
140
-
141
-</details>
142
-
143
-<details>
144
-<summary>Delete snapshots</summary>
145
-
146
-You can identify and delete snapshots that you no longer need.
147
-
148
-1. Find the snapshots for a specific path.
149
- ```
150
- root@netdata~ # sudo btrfs subvolume list -s <path>
151
- ```
152
-
153
-1. Delete a snapshot that you do not need any more.
154
- ```
155
- root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
156
- ```
157
-
158
-</details>
159
-
160
-<details>
161
-<summary>Enable a compression mechanism</summary>
162
-
163
-> The `compress=alg` mount option into the `fstab` configuration file (or in the `mount` procedure) enables
164
-automatically considering every file for compression, where `alg` is either `zlib`, `lzo`, `zstd`, or `no` (for no
165
-compression). Using this option, Btrfs will check if compressing the first portion of the data shrinks it. If it does,
166
-the entire write to
167
-that file will be compressed. If it does not, none of it is compressed. With this option, if the first portion of the
168
-write does not shrink, no compression will be applied to the write even if the rest of the data would shrink
169
-tremendously. This is done to prevent making the disk wait to start writing until all of the data to be written is fully
170
--given to Btrfs and compressed. [[1]](https://wiki.archlinux.org/title/btrfs)
171
-
172
-1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd`
173
- compression algorithm.
174
-
175
- ```
176
- root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
177
- ```
178
-
179
-</details>
180
-
181
-<details>
182
-<summary>Enable a deduplication mechanism</summary>
183
-
184
-Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a
185
-file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks
186
-of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
187
-
188
-Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd
189
-party, and it is strongly suggested that you check their status before you decide to use them.
190
-
191
-- [duperemove](https://github.com/markfasheh/duperemove)
192
-- [bees](https://github.com/Zygo/bees)
193
-- [dduper](https://github.com/lakshmipathi/dduper)
194
-
195
-</details>
196
-
197
-<details>
198
-<summary>Perform a balance</summary>
199
-
200
-Especially in a Btrfs with multiple disks, data/metadata might be unevenly allocated into the disks.
201
-
202
-```
203
-root@netdata # btrfs balance start -musage=10 -dusage=10 -susage=50 /mount/point
204
-```
205
-
206
-> This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this
207
-> example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends
208
-> with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter
209
-> until at least some chunks are relocated.
210
-
211
-</details>
health/guides/btrfs_allocated.md
new
+75
@@ -0,0 +1,75 @@
1
+### Understand the alert
2
+
3
+Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing on fault tolerance, repair and easy administration. Btrfs is intended to address the lack of pooling, snapshots, checksums, and integral multi-device spanning in Linux file systems.
4
+
5
+Unlike most filesystems, Btrfs allocates disk space in two distinct stages. The first stage allocates chunks of physical disk space for usage by a particular type of filesystem blocks, either data blocks (which store actual file data), metadata blocks (which store inodes and other file metadata), and system blocks (which store metadata about the filesystem itself). The second stage then allocates actual blocks within those chunks for usage by the filesystem. This metric tracks space usage in the first allocation stage.
6
+
7
+The Netdata Agent monitors the percentage of allocated Btrfs physical disk space.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Add more physical space
12
+
13
+Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. An easy way to add a new disk to a filesystem is:
14
+
15
+1. Determine which disk you want to add and in which path
16
+ ```
17
+ btrfs device add -f /dev/<new_disk> <path>
18
+ ```
19
+
20
+2. If you get an error that the drive is already mounted, you might have to unmount
21
+ ```
22
+ btrfs device add -f /dev/<new_disk> <path>
23
+ ```
24
+3. See the newly added disk
25
+ ```
26
+ btrfs filesystem show
27
+ ```
28
+4. Balance the system to make use of the new drive.
29
+ ```
30
+ btrfs filesystem balance <path>
31
+ ```
32
+
33
+- Delete snapshots
34
+
35
+You can identify and delete snapshots that you no longer need.
36
+
37
+1. Find the snapshots for a specific path.
38
+ ```
39
+ sudo btrfs subvolume list -s <path>
40
+ ```
41
+
42
+2. Delete a snapshot that you do not need any more.
43
+ ```
44
+ btrfs subvolume delete <path>/@some_dir-snapshot-test
45
+ ```
46
+
47
+- Enable a compression mechanism
48
+
49
+1. Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd` compression algorithm.
50
+
51
+ ```
52
+ btrfs filesystem defragment -r -v -czstd /mount/point
53
+ ```
54
+
55
+- Enable a deduplication mechanism
56
+
57
+Using copy-on-write, Btrfs is able to copy files or whole subvolumes without actually copying the data. However, when a file is altered, a new proper copy is created. Deduplication takes this a step further, by actively identifying blocks of data which share common sequences and combining them into an extent with the same copy-on-write semantics.
58
+
59
+Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd party, and it is strongly suggested that you check their status before you decide to use them.
60
+
61
+- Perform a balance
62
+
63
+Especially in a Btrfs with multiple disks, there might be unevenly allocated data/metadata into the disks.
64
+```
65
+btrfs balance start -musage=10 -dusage=10 -susage=5 /mount/point
66
+```
67
+This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter until at least some chunks are relocated.
68
+
69
+### Useful resources
70
+
71
+1. [The Btrfs filesystem on Arch linux website](https://wiki.archlinux.org/title/btrfs)
72
+2. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
73
+3. [duperemove](https://github.com/markfasheh/duperemove)
74
+4. [bees](https://github.com/Zygo/bees)
75
+5. [dduper](https://github.com/lakshmipathi/dduper)
health/guides/btrfs_data.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the percentage of used Btrfs data space exceeds the configured threshold. Btrfs (B-tree file system) is a modern copy-on-write (CoW) filesystem for Linux which focuses on fault tolerance, repair, and easy administration. This filesystem also provides advanced features like snapshots, checksums, and multi-device spanning.
4
+
5
+### What does high Btrfs data usage mean?
6
+
7
+High Btrfs data usage indicates that a significant amount of the allocated space for data blocks in the filesystem is being used. This could be a result of many factors, such as large files, numerous smaller files, or multiple snapshots.
8
+
9
+### Troubleshoot the alert
10
+
11
+Before you attempt any troubleshooting, make sure you have backed up your data to prevent potential data loss or corruption.
12
+
13
+1. **Add more physical space**: You can add a new disk to the filesystem, depending on your infrastructure and disk RAID configuration. Remember to unmount the drive if it's already mounted, then use the `btrfs device add` command to add the new disk and balance the system.
14
+
15
+2. **Delete snapshots**: Review the snapshots in your Btrfs filesystem and delete any unnecessary snapshots. Use the `btrfs subvolume list` command to find snapshots and `btrfs subvolume delete` to remove them.
16
+
17
+3. **Enable compression**: By enabling compression, you can save disk space without deleting files or snapshots. Add the `compress=alg` mount option in your `fstab` configuration file or during the mount procedure, where `alg` is the compression algorithm you want to use (e.g., `zlib`, `lzo`, `zstd`). You can apply compression to existing files using the `btrfs filesystem defragment` command.
18
+
19
+4. **Enable deduplication**: Implement deduplication to identify and merge blocks of data with common sequences using copy-on-write semantics. You can use third-party tools dedicated to Btrfs deduplication, such as duperemove, bees, and dduper. However, research their stability and reliability before employing them.
20
+
21
+5. **Perform a balance**: If the data and metadata are unevenly allocated among disks, especially in Btrfs filesystems with multiple disks, you can perform a balance operation to reallocate space between data and metadata. Use the `btrfs balance` command with appropriate usage parameters to start the balance process.
22
+
23
+### Useful resources
24
+
25
+1. [Btrfs Wiki](https://btrfs.wiki.kernel.org)
26
+2. [The Btrfs filesystem on the Arch Linux website](https://wiki.archlinux.org/title/btrfs)
27
+3. [Ubuntu man pages for Btrfs commands](https://manpages.ubuntu.com/manpages/bionic/man8)
28
+4. [duperemove](https://github.com/markfasheh/duperemove)
29
+5. [bees](https://github.com/Zygo/bees)
30
+6. [dduper](https://github.com/lakshmipathi/dduper)
\ No newline at end of file
health/guides/btrfs_device_corruption_errors.md
new
+57
@@ -0,0 +1,57 @@
1
+### Understand the alert
2
+
3
+This alert monitors the `corruption_errs` metric in the `btrfs.device_errors` chart. If you receive this alert, it means that your system's BTRFS file system has encountered one or more corruption errors in the past 10 minutes. These errors indicate data inconsistencies on the file system that could lead to data loss or other issues.
4
+
5
+### What are BTRFS corruption errors?
6
+
7
+BTRFS (B-Tree File System) is a modern, fault-tolerant, and highly scalable file system used in several Linux distributions. Corruption errors in a BTRFS file system refer to inconsistencies in the data structures that the file system relies on to store and manage data. Such inconsistencies can stem from software bugs, hardware failures, or other causes.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check for system messages:
12
+
13
+ Review your system's kernel message log (`dmesg` output) for any BTRFS-related errors or warnings. These messages can provide insights into the cause of the corruption and help you diagnose the issue.
14
+
15
+ ```
16
+ dmesg | grep -i btrfs
17
+ ```
18
+
19
+2. Run a file system check:
20
+
21
+ Use the `btrfs scrub` command to scan the file system for inconsistencies and attempt to automatically repair them. Note that this command may take a long time to complete, depending on the size of your BTRFS file system.
22
+
23
+ ```
24
+ sudo btrfs scrub start /path/to/btrfs/mountpoint
25
+ ```
26
+
27
+ After the scrub finishes, check the status with:
28
+
29
+ ```
30
+ sudo btrfs scrub status /path/to/btrfs/mountpoint
31
+ ```
32
+
33
+3. Assess your storage hardware
34
+
35
+ In some cases, BTRFS corruption errors may be caused by failing storage devices, such as a disk drive nearing the end of its lifetime. Check the S.M.A.R.T. status of your disks using the `smartctl` tool to identify potential hardware issues.
36
+
37
+ ```
38
+ sudo smartctl -a /dev/sdX
39
+ ```
40
+
41
+ Replace `/dev/sdX` with the actual device path of your disk.
42
+
43
+4. Update your system
44
+
45
+ Ensuring that your system has the latest kernel, BTRFS tools package, and other relevant updates can help prevent software-related corruption errors.
46
+
47
+ For example, on Ubuntu or Debian-based systems, you can update with:
48
+
49
+ ```
50
+ sudo apt-get update
51
+ sudo apt-get upgrade
52
+ ```
53
+
54
+5. Backup essential data
55
+
56
+ As file system corruption might result in data loss, ensure that you have proper backups of any critical data stored on your BTRFS file system. Regularly back up your data to an external or secondary storage device.
57
+
health/guides/btrfs_device_flush_errors.md
new
+54
@@ -0,0 +1,54 @@
1
+### Understand the alert
2
+
3
+This alert indicates that `BTRFS` flush errors have been detected on your file system. If you receive this alert, it means that your system has encountered problems while flushing data from memory to disk, which may result in data corruption or data loss.
4
+
5
+### What is BTRFS?
6
+
7
+`BTRFS` (B-Tree File System) is a modern, copy-on-write (CoW) file system for Linux designed to address various weaknesses in traditional file systems. It provides advanced features like data pooling, snapshots, and checksums that enhance fault tolerance.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Verify the alert
12
+
13
+Check the `Netdata` dashboard or query the monitoring API to confirm that the alert is genuine and not a false positive.
14
+
15
+2. Review and analyze syslog
16
+
17
+Check your system's `/var/log/syslog` or `/var/log/messages`, looking for `BTRFS`-related errors. These messages will provide essential information about the cause of the flush errors.
18
+
19
+3. Confirm BTRFS status
20
+
21
+Run the following command to display the state of the BTRFS file system and ensure it is mounted and healthy:
22
+
23
+```
24
+sudo btrfs filesystem show
25
+```
26
+
27
+4. Check disk space
28
+
29
+Ensure your system has sufficient disk space allocated to the BTRFS file system. A full or nearly full disk might cause flush errors. You can use the `df -h` command to examine the available disk space.
30
+
31
+5. Check system I/O usage
32
+
33
+Use the `iotop` command to inspect disk I/O usage for any abnormally high activity, which could be related to the flush errors.
34
+
35
+```
36
+sudo iotop
37
+```
38
+
39
+6. Upgrade or rollback BTRFS version
40
+
41
+Verify that you are using a stable version of the BTRFS utilities and kernel module. If not, consider upgrading or rolling back to a more stable version.
42
+
43
+7. Inspect hardware health
44
+
45
+Inspect your disks and RAM for possible hardware problems, as these can cause flush errors. SMART data can help assess disk health (`smartctl -a /dev/sdX`), and `memtest86+` can be used to scrutinize RAM.
46
+
47
+8. Create backups
48
+
49
+Take backups of your critical BTRFS data immediately to avoid potential data loss due to flush errors.
50
+
51
+### Useful resources
52
+
53
+1. [BTRFS official website](https://btrfs.wiki.kernel.org/index.php/Main_Page)
54
+2. [BTRFS utilities on GitHub](https://github.com/kdave/btrfs-progs)
health/guides/btrfs_device_generation_errors.md
new
+52
@@ -0,0 +1,52 @@
1
+### Understand the alert
2
+
3
+This alert is about `BTRFS generation errors`. When you receive this alert, it means that your BTRFS file system has encountered errors during its operation.
4
+
5
+### What are BTRFS generation errors?
6
+
7
+BTRFS is a modern copy-on-write (CoW) filesystem, which is developed to address various weaknesses in traditional Linux file systems. It features snapshotting, checksumming, and performs background scrubbing to find and repair errors.
8
+
9
+A `BTRFS generation error` occurs when the file system encounters issues while updating the data and metadata associated with a snapshot or subvolume. This could be due to software bugs, hardware issues, or data corruption.
10
+
11
+### Troubleshoot the alert
12
+
13
+1. Verify the issue: Check your system logs for any BTRFS-related errors to further understand the problem. This can be done using the `dmesg` command:
14
+
15
+ ```
16
+ sudo dmesg | grep BTRFS
17
+ ```
18
+
19
+2. Check the BTRFS filesystem status: Use the `btrfs filesystem` command to get information about your BTRFS filesystem, including the UUID, total size, used size, and device information:
20
+
21
+ ```
22
+ sudo btrfs filesystem show
23
+ ```
24
+
25
+3. Perform a BTRFS scrub: Scrubbing is a process that scans the entire filesystem, verifies the data and metadata, and attempts to repair any detected errors. Run the following command to start a scrub operation:
26
+
27
+ ```
28
+ sudo btrfs scrub start -Bd /path/to/btrfs/mountpoint
29
+ ```
30
+
31
+ The `-B` flag will run the scrub in the background, and the `-d` flag will provide detailed information about the operation.
32
+
33
+4. Monitor scrub progress: You can monitor the scrub progress using the `btrfs scrub status` command:
34
+
35
+ ```
36
+ sudo btrfs scrub status /path/to/btrfs/mountpoint
37
+ ```
38
+
39
+5. Analyze scrub results: The scrub operation will provide information about the total data scrubbed, the number of errors found, and the number of errors fixed. This information can help you determine the extent of the issue and any further action required.
40
+
41
+6. Address BTRFS issues: Depending on the nature of the errors, you may need to take further action, such as updating the BTRFS tools, updating your Linux kernel, or even replacing faulty hardware to resolve the errors.
42
+
43
+7. Set up a regular scrub schedule: You can schedule regular scrubs to keep your BTRFS filesystem healthy. This can be done using `cron`. For example, you can add the following line to `/etc/crontab` to run a scrub on the 1st of each month:
44
+
45
+ ```
46
+ 0 0 1 * * root btrfs scrub start -B /path/to/btrfs/mountpoint
47
+ ```
48
+
49
+### Useful resources
50
+
51
+1. [BTRFS Wiki Homepage](https://btrfs.wiki.kernel.org/index.php/Main_Page)
52
+2. [Btrfs Documentation](https://www.kernel.org/doc/Documentation/filesystems/btrfs.txt)
health/guides/btrfs_device_read_errors.md
new
+50
@@ -0,0 +1,50 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of BTRFS read errors on a device. If you receive this alert, it means that your system has encountered at least one BTRFS read error in the last 10 minutes.
4
+
5
+### What are BTRFS read errors?
6
+
7
+BTRFS (B-Tree File System) is a modern file system designed for Linux. BTRFS read errors are instances where the file system fails to read data from a device. This can occur due to various reasons like hardware failure, file system corruption, or disk problems.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check system logs for BTRFS errors
12
+
13
+ Review the output from the following command to identify any BTRFS errors:
14
+ ```
15
+ sudo journalctl -k | grep -i BTRFS
16
+ ```
17
+
18
+2. Identify the affected BTRFS device and partition
19
+
20
+ List all BTRFS devices with their respective information by running the following command:
21
+ ```
22
+ sudo btrfs filesystem show
23
+ ```
24
+
25
+3. Perform a BTRFS filesystem check
26
+
27
+ To check the integrity of the BTRFS file system, run the following command, replacing `<device>` with the affected device path:
28
+ ```
29
+ sudo btrfs check --readonly <device>
30
+ ```
31
+ Note: Be careful when using the `--repair` option, as it may cause data loss. It is recommended to take a backup before attempting a repair.
32
+
33
+4. Verify the disk health
34
+
35
+ Check the disk health using SMART tools to determine if there are any hardware issues. This can be done by first installing `smartmontools` if not already installed:
36
+ ```
37
+ sudo apt install smartmontools
38
+ ```
39
+ Then running a disk health check on the affected device:
40
+ ```
41
+ sudo smartctl -a <device>
42
+ ```
43
+
44
+5. Analyze the read error patterns
45
+
46
+ If the read errors are happening consistently or increasing, consider replacing the affected device with a new one or adding redundancy to the system by using RAID or BTRFS built-in features.
47
+
48
+### Useful resources
49
+
50
+1. [smartmontools documentation](https://www.smartmontools.org/)
health/guides/btrfs_device_write_errors.md
new
+42
@@ -0,0 +1,42 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when BTRFS (B-tree file system) encounters write errors on your system. BTRFS is a modern copy-on-write (COW) filesystem designed to address various weaknesses in traditional Linux file systems. If you receive this alert, it means that there have been issues with writing data to the file system.
4
+
5
+### What are BTRFS write errors?
6
+
7
+BTRFS write errors can occur when there are problems with the underlying storage devices, such as bad disks or data corruption. These errors may result in data loss or the inability to write new data to the file system. It is important to address these errors to prevent potential data loss and maintain the integrity of your file system.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check the BTRFS system status
12
+
13
+Execute the following command to get the current status of your BTRFS system:
14
+```
15
+sudo btrfs device stats [Mount point]
16
+```
17
+Replace `[Mount point]` with the actual mount point of your BTRFS file system.
18
+
19
+- Examine system logs for potential issues
20
+
21
+Check the system logs for any signs of issues with the BTRFS file system or underlying storage devices:
22
+```
23
+sudo journalctl -u btrfs
24
+```
25
+
26
+- Check the health of the storage devices
27
+
28
+Use the `smartctl` tool to assess the health of your storage devices. For example, to check the device `/dev/sda`, use the following command:
29
+```
30
+sudo smartctl -a /dev/sda
31
+```
32
+
33
+- Repair the BTRFS file system
34
+
35
+If there are issues with the file system, run the following command to repair it:
36
+```
37
+sudo btrfs check --repair [Mount point]
38
+```
39
+Replace `[Mount point]` with the actual mount point of your BTRFS file system.
40
+
41
+**WARNING:** The `--repair` option should be used with caution, as it may result in data loss under certain circumstances. It is recommended to back up your data before attempting to repair the file system.
42
+
health/guides/btrfs_metadata.md
new
+70
@@ -0,0 +1,70 @@
1
+### Understand the alert
2
+
3
+The `btrfs_metadata` alert calculates the percentage of used Btrfs metadata space for a Btrfs filesystem. If you receive this alert, it indicates that your Btrfs filesystem's metadata space is being utilized at a high rate.
4
+
5
+### Troubleshoot the alert
6
+
7
+**Warning: Data is valuable. Before performing any actions, make sure to take necessary backup steps. Netdata is not responsible for any loss or corruption of data, database, or software.**
8
+
9
+1. **Add more physical space**
10
+
11
+ - Determine which disk you want to add and in which path:
12
+ ```
13
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
14
+ ```
15
+
16
+ - If you get an error that the drive is already mounted, you might have to unmount:
17
+ ```
18
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
19
+ ```
20
+
21
+ - Check the newly added disk:
22
+ ```
23
+ root@netdata~ # btrfs filesystem show
24
+ ```
25
+
26
+ - Balance the system to make use of the new drive:
27
+ ```
28
+ root@netdata~ # btrfs filesystem balance <path>
29
+ ```
30
+
31
+2. **Delete snapshots**
32
+
33
+ - List the snapshots for a specific path:
34
+ ```
35
+ root@netdata~ # sudo btrfs subvolume list -s <path>
36
+ ```
37
+
38
+ - Delete an unnecessary snapshot:
39
+ ```
40
+ root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
41
+ ```
42
+
43
+3. **Enable a compression mechanism**
44
+
45
+ Apply compression to existing files by modifying the `fstab` configuration file (or during the `mount` procedure) with the `compress=alg` option. Replace `alg` with `zlib`, `lzo`, `zstd`, or `no` (for no compression). For example, to re-compress the `/mount/point` path with `zstd` compression:
46
+
47
+ ```
48
+ root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
49
+ ```
50
+
51
+4. **Enable a deduplication mechanism**
52
+
53
+ Deduplication tools like duperemove, bees, and dduper can help identify blocks of data sharing common sequences and combine extents via copy-on-write semantics. Ensure you check the status of these 3rd party tools before using them.
54
+
55
+ - [duperemove](https://github.com/markfasheh/duperemove)
56
+ - [bees](https://github.com/Zygo/bees)
57
+ - [dduper](https://github.com/lakshmipathi/dduper)
58
+
59
+5. **Perform a balance**
60
+
61
+ Balance data/metadata/system-data in empty or near-empty chunks for Btrfs filesystems with multiple disks, allowing space to be reassigned:
62
+
63
+ ```
64
+ root@netdata # btrfs balance start -musage=50 -dusage=10 -susage=5 /mount/point
65
+ ```
66
+
67
+### Useful resources
68
+
69
+1. [The Btrfs filesystem on Arch Linux website](https://wiki.archlinux.org/title/btrfs)
70
+2. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
\ No newline at end of file
health/guides/btrfs_system.md
new
+75
@@ -0,0 +1,75 @@
1
+### Understand the alert
2
+
3
+The `btrfs_system` alert monitors the percentage of used Btrfs system space. If you receive this alert, it means that your Btrfs system space usage has reached a critical level and could potentially cause issues on your system.
4
+
5
+### Troubleshoot the alert
6
+
7
+**Important**: Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is not liable for any loss or corruption of any data, database, or software.
8
+
9
+1. Add more physical space
10
+
11
+ Adding a new disk always depends on your infrastructure, disk RAID configuration, encryption, etc. To add a new disk to a filesystem:
12
+
13
+ - Determine which disk you want to add and in which path:
14
+ ```
15
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
16
+ ```
17
+ - If you get an error that the drive is already mounted, you might have to unmount:
18
+ ```
19
+ root@netdata~ # btrfs device add -f /dev/<new_disk> <path>
20
+ ```
21
+ - See the newly added disk:
22
+ ```
23
+ root@netdata~ # btrfs filesystem show
24
+ Label: none uuid: d6b9d7bc-5978-2677-ac2e-0e68204b2c7b
25
+ Total devices 2 FS bytes used 192.00KiB
26
+ devid 1 size 10.01GiB used 536.00MiB path /dev/sda1
27
+ devid 2 size 10.01GiB used 0.00B path /dev/sdb
28
+ ```
29
+ - Balance the system to make use of the new drive:
30
+ ```
31
+ root@netdata~ # btrfs filesystem balance <path>
32
+ ```
33
+
34
+2. Delete snapshots
35
+
36
+ You can identify and delete snapshots that you no longer need.
37
+
38
+ - Find the snapshots for a specific path:
39
+ ```
40
+ root@netdata~ # sudo btrfs subvolume list -s <path>
41
+ ```
42
+ - Delete a snapshot that you do not need any more:
43
+ ```
44
+ root@netdata~ # btrfs subvolume delete <path>/@some_dir-snapshot-test
45
+ ```
46
+
47
+3. Enable a compression mechanism
48
+
49
+ - Apply compression to existing files. This command will re-compress the `mount/point` path, with the `zstd` compression algorithm:
50
+ ```
51
+ root@netdata # btrfs filesystem defragment -r -v -czstd /mount/point
52
+ ```
53
+
54
+4. Enable a deduplication mechanism
55
+
56
+ Tools dedicated to deduplicate a Btrfs formatted partition include duperemove, bees, and dduper. These projects are 3rd party, and it is strongly suggested that you check their status before you decide to use them.
57
+
58
+ - [duperemove](https://github.com/markfasheh/duperemove)
59
+ - [bees](https://github.com/Zygo/bees)
60
+ - [dduper](https://github.com/lakshmipathi/dduper)
61
+
62
+5. Perform a balance
63
+
64
+ Especially in a Btrfs with multiple disks, data/metadata might be unevenly allocated into the disks.
65
+
66
+ ```
67
+ root@netdata # btrfs balance start -musage=10 -dusage=10 -susage=50 /mount/point
68
+ ```
69
+
70
+ > This command will attempt to relocate data/metdata/system-data in empty or near-empty chunks (at most X% used, in this example), allowing the space to be reclaimed and reassigned between data and metadata. If the balance command ends with "Done, had to relocate 0 out of XX chunks", then you need to increase the "dusage/musage" percentage parameter until at least some chunks are relocated.
71
+
72
+### Useful resources
73
+
74
+1. [The Btrfs filesystem on Arch Linux website](https://wiki.archlinux.org/title/btrfs)
75
+2. [The Btrfs filesystem on kernel.org website](https://btrfs.wiki.kernel.org)
\ No newline at end of file
health/guides/ceph/ceph_cluster_space_usage.md
deleted
-34
@@ -1,34 +0,0 @@
1
-# ceph_cluster_space_usage
2
-
3
-**Storage | Ceph**
4
-
5
-Ceph is an open-source software-defined storage platform that implements object storage on a single
6
-distributed computer cluster and provides 3-in-1 interfaces for object-, block- and file-level
7
-storage. Ceph aims primarily for completely distributed operation without a single point of failure,
8
-scalability to the exabyte level, and to be freely available.
9
-
10
-The Netdata Agent calculates the percentage of used cluster disk space. Your cluster is in high disk
11
-space utilization.
12
-
13
-This alert is triggered in warning state when the percentage of used cluster disk space is between
14
-85-90% and in critical state when it is between 90-98%.
15
-
16
-### Troubleshooting section
17
-
18
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
19
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
20
-
21
-
22
-<details>
23
-<summary>Examine your cluster status </summary>
24
-
25
-1. In the master node, examine the disk space details of the cluster
26
-
27
- ```
28
- root@netdata # ceph df detail
29
-
30
- ```
31
-
32
-2. Check for unused pools and delete them or and consider adding a node to your cluster.
33
-
34
-</details>
\ No newline at end of file
health/guides/ceph_cluster_space_usage.md
new
+53
@@ -0,0 +1,53 @@
1
+### Understand the alert
2
+
3
+The `ceph_cluster_space_usage` alert is triggered when the percentage of used disk space in your Ceph cluster reaches a high level. Ceph is a distributed storage system designed to provide excellent performance, reliability, and scalability. If the usage surpasses certain thresholds (warning: 85-90%, critical: 90-98%), this indicates high disk space utilization, which may affect the performance and reliability of your Ceph cluster.
4
+
5
+### Troubleshoot the alert
6
+
7
+Perform the following actions:
8
+
9
+1. Check the Ceph cluster status
10
+
11
+ Run the following command to see the overall health of the Ceph cluster:
12
+
13
+ ```
14
+ ceph status
15
+ ```
16
+
17
+ Pay attention to the `HEALTH` status and the `cluster` section, which provides information about the used and total disk space.
18
+
19
+2. Review the storage utilization for each pool
20
+
21
+ Run the following command to review the storage usage for each pool in the Ceph cluster:
22
+
23
+ ```
24
+ ceph df
25
+ ```
26
+
27
+ Identify the pools with high utilization and consider moving or removing data from these pools.
28
+
29
+3. Investigate high storage usage clients or applications
30
+
31
+ Check the clients or applications that interact with the Ceph cluster and the associated file systems. You can use monitoring tools, disk usage analysis programs, or log analysis tools to identify any unusual patterns, such as excessive file creation, large file uploads, or high I/O operations.
32
+
33
+4. Add more storage or nodes to the cluster
34
+
35
+ If the cluster is reaching its full capacity due to normal usage, consider adding more storage or nodes to the Ceph cluster. This can help prevent the cluster from becoming overloaded and maintain its performance and reliability.
36
+
37
+ You can use the following commands to add more storage or nodes to the Ceph cluster:
38
+
39
+ ```
40
+ ceph osd create
41
+ ceph osd add
42
+ ```
43
+
44
+5. Optimize data replication and placement
45
+
46
+ The high disk usage might be a result of non-optimal data replication and distribution across the cluster. Review the Ceph replication and placement settings, and update the CRUSH map if needed to ensure better distribution of data.
47
+
48
+### Useful resources
49
+
50
+1. [Ceph Storage Cluster](https://docs.ceph.com/en/latest/architecture/#storage-cluster)
51
+2. [Ceph Troubleshooting Guide](https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/troubleshooting_guide/index)
52
+3. [Managing Ceph Placement Groups](https://docs.ceph.com/en/latest/rados/operations/placement-groups/)
53
+4. [Ceph: Adding and Removing OSDs](https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/)
\ No newline at end of file
health/guides/cgroup_10min_cpu_usage.md
new
+5
@@ -0,0 +1,5 @@
1
+### Understand the alert
2
+
3
+The Netdata Agent calculates the average CPU utilization over the last 10 minutes. This alert indicates that your system is in high cgroup CPU utilization. The system will throttle the group CPU usage when the usage is over the limit. To fix this issue, try to increase the cgroup CPU limit.
4
+
5
+This alert is triggered in warning state when the average CPU utilization is between 75-80% and in critical state when it is between 85-95%.
\ No newline at end of file
health/guides/cgroup_10s_received_packets_storm.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alert checks the ratio of the average number of received packets for a network interface over the last 10 seconds, compared to the rate over the last minute. If the rate of received packets increases significantly over a short period, it may indicate a packet storm, which can impact network performance and connectivity.
4
+
5
+### What is a packet storm?
6
+
7
+A packet storm is a sudden increase in network traffic due to a large number of packets being sent simultaneously. This can cause network congestion, packet loss, and increased latency, leading to a degradation of network performance and potential loss of connectivity.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the affected interface and examine its traffic patterns:
12
+
13
+Use `iftop` or a similar monitoring tool to view real-time network traffic on the affected interface.
14
+
15
+```
16
+sudo iftop -i <interface_name>
17
+```
18
+
19
+Replace `<interface_name>` with the name of the network interface experiencing the packet storm (e.g., eth0).
20
+
21
+2. Check for possible packet flood sources:
22
+
23
+Examine logs, firewall rules, and traffic patterns for evidence of a Denial of Service (DoS) attack or a misconfigured application. Use tools like `tcpdump` or `wireshark` to capture network packets and analyze traffic.
24
+
25
+3. Limit or block unwanted traffic:
26
+
27
+Apply traffic shaping or Quality of Service (QoS) policies, firewall rules, or Intrusion Prevention System (IPS) to limit or block the sources of unwanted traffic.
28
+
29
+4. Monitor network performance:
30
+
31
+Continuously monitor network performance to ensure the issue is resolved and prevent future packet storms. Use monitoring tools like Netdata to keep track of network performance metrics.
32
+
33
+### Useful resources
34
+
35
+1. [iftop: Linux Network Bandwidth Monitoring Tool](https://www.tecmint.com/iftop-linux-network-bandwidth-monitoring-tool/)
36
+2. [tcpdump: A powerful command-line packet analyzer](https://www.tcpdump.org/)
37
+3. [Wireshark: A network protocol analyzer for UNIX and Windows](https://www.wireshark.org/)
health/guides/cgroup_1m_received_packets_rate.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average number of packets received by the network interface `${label:device}` over the period of one minute. If you receive this alert, it means that the rate of received packets has significantly increased, which could indicate a potential network bottleneck or an increased network workload.
4
+
5
+### What does received packets rate mean?
6
+
7
+`Received packets rate` represents the speed at which packets are arriving at the network interface of your machine. A packet is a single unit of data transmitted over the network. A high rate of received packets indicates that your network is under significant workload as it processes incoming data.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Monitor your network traffic
12
+
13
+ Use the `iftop` command to get a real-time report of bandwidth usage on your network interfaces:
14
+ ```
15
+ sudo iftop -i ${label:device}
16
+ ```
17
+ If you don't have `iftop` installed, install it using your package manager.
18
+
19
+2. Identify the top consumers of network bandwidth
20
+
21
+ Inspect the output of `iftop` to identify if any IP addresses or hosts are using an unusual amount of bandwidth. This can help you pinpoint any sudden surges in network traffic caused by specific services or applications.
22
+
23
+3. Check for possible network congestion
24
+
25
+ Determine if the high received packets rate is caused by network congestion. Network congestion occurs when the volume of data being transmitted exceeds the available capacity of the network. You can use `ping` or `traceroute` commands to check for latency and packet loss.
26
+
27
+4. Examine your application logs
28
+
29
+ Investigate your application logs to identify any unusual activity or network spikes. This can provide valuable information about potential issues, such as a sudden increase in incoming client connections, improperly optimized application configurations, or the presence of malicious traffic.
30
+
31
+5. Optimize your network configuration
32
+
33
+ Review your networking configurations to ensure they are optimized for the current workload. Check for any misconfigurations or resource limitations that might be causing the high received packets rate. You might consider increasing the maximum number of open file descriptors, changing your network driver settings, or adjusting your network buffer sizes.
34
+
35
+### Useful resources
36
+
37
+1. [Iftop Guide – Monitor Network Bandwidth](https://www.tecmint.com/iftop-linux-network-bandwidth-monitoring-tool/)
health/guides/cgroup_ram_in_use.md
new
+5
@@ -0,0 +1,5 @@
1
+### Understand the alert
2
+
3
+The Netdata Agent calculates the percentage of used memory. This alert indicates high cgroup memory utilization. Out Of Memory (OOM) killer will kill some processes when the utilization reaches 100%. To fix this issue, try to increase the cgroup memory limit (if set).
4
+
5
+This alert is triggered in warning state when the percentage of used memory is between 80-90% and in critical state between 90-98%.
health/guides/cgroups/cgroup_10min_cpu_usage.md
deleted
-212
@@ -1,212 +0,0 @@
1
-# cgroup_10min_cpu_usage
2
-
3
-**Cgroups | CPU**
4
-
5
-_Control groups, usually referred to as Cgroups, are a Linux kernel feature which allow processes to
6
-be organized into hierarchical groups whose usage of various types of resources can then be limited
7
-and monitored. The Kernel's cgroup interface is provided through a pseudo-filesystem called
8
-cgroupfs (`/sys/fs/cgroup`). Grouping is implemented in the core Cgroup Kernel code, while resource
9
-tracking and limits are implemented in a set of per-resource-type subsystems (memory, CPU, and so
10
-on).<sup>[1](https://man7.org/linux/man-pages/man7/cgroups.7.html) </sup>_
11
-
12
-The Netdata Agent calculates the average CPU utilization over the last 10 minutes. This alert
13
-indicates that one of the group of you processes is in high CPU utilization. The system will
14
-throttle the group CPU usage when the usage is over the limit.
15
-
16
-This alert is raised in warning state when the average CPU utilization is between 75-80% of the
17
-limits and in critical when it is between 85-95%.
18
-
19
-<details>
20
-<summary>More about Cgroups</summary>
21
-
22
-Since Linux kernel 4.5 (March 2016) there are two implementations of Cgroups, v1 and v2.
23
-
24
-#### Cgroup controllers (in v1)
25
-
26
-In the vanilla kernel you will find the following controllers.
27
-
28
-- cpu: `CONFIG_CGROUP_SCHED`, Cgroups can be guaranteed a minimum number of "CPU shares" when a
29
- system is busy.
30
-- cpuacct: `CONFIG_CGROUP_CPUACCT`, This controller provides accounting for CPU usage by groups of
31
- processes.
32
-- cpuset: `CONFIG_CPUSETS`, This Cgroup can be used to bind the processes in a Cgroup to a specified
33
- set of CPUs and NUMA nodes.
34
-- memory: `CONFIG_MEMCG`, The memory controller supports reporting and limiting of process memory,
35
- kernel memory, and swap used by Cgroups.
36
-- devices: `CONFIG_CGROUP_DEVICE`, This supports controlling which processes may create (mknod)
37
- devices as well as open them for reading or writing. The policies may be specified as allow-lists
38
- and deny-lists. Hierarchy is enforced, so new rules must not violate existing rules for the target
39
- or ancestor Cgroups.
40
-- freezer: `CONFIG_CGROUP_FREEZER`, The freezer Cgroup can suspend and restore (resume) all
41
- processes in a Cgroup. Freezing a Cgroup /A also causes its children, for example, processes in
42
- /A/B, to be frozen.
43
-- net_cls: `CONFIG_CGROUP_NET_CLASSID`, This places a classid, specified for the Cgroup, on network
44
- packets created by a Cgroup. These classids can then be used in firewall rules, as well as used to
45
- shape traffic using `tc`. This applies only to packets leaving the Cgroup, not to traffic arriving
46
- at the Cgroup.
47
-- blkio: `CONFIG_BLK_CGROUP`, The blkio Cgroup controls and limits access to specified block devices
48
- by applying IO control in the form of throttling and upper limits against leaf nodes and
49
- intermediate nodes in the storage hierarchy. Two policies are available. The first is a
50
- proportional- weight time-based division of disk implemented with CFQ. This is in effect for leaf
51
- nodes using CFQ. The second is a throttling policy which specifies upper I/O rate limits on a
52
- device.
53
-- perf_event: `CONFIG_CGROUP_PERF`, This controller allows perf monitoring of the set of processes
54
- grouped in a Cgroup.
55
-- net_prio: `CONFIG_CGROUP_NET_PRIO`, This allows priorities to be specified, per network interface,
56
- for Cgroups.
57
-- hugetlb: `CONFIG_CGROUP_HUGETLB`, This supports limiting the use of huge pages by Cgroups.
58
-- pids: `CONFIG_CGROUP_PIDS` , This controller permits limiting the number of process that may be
59
- created in a Cgroup (
60
- and its descendants).
61
-- rdma: `CONFIG_CGROUP_RDMA`, The RDMA controller permits limiting the use of RDMA/IB- specific
62
- resources per Cgroup. _since Linux 4.11_
63
-
64
-Different variations of the Linux kernel can have more or less Cgroup controllers OR/AND enabled not
65
-all of them.
66
-
67
-#### Cgroups v1
68
-
69
-Under Cgroups v1, each controller may be mounted against a separate Cgroup filesystem that provides
70
-its own hierarchical organization of the processes on the system. It is also possible to co-mount
71
-multiple (or even all) Cgroups v1 controllers against the same Cgroup filesystem. That means that
72
-the mounted controllers manage the same hierarchical organization of processes.
73
-
74
-For each mounted hierarchy, the directory tree mirrors the control group hierarchy. Each control
75
-group is represented by a directory, with each of its child control Cgroups represented as a child
76
-directory. For instance, `/user/joe/1.session` represents control group 1.session, which is a child
77
-of Cgroup joe, which is a child of /user. Under each Cgroup directory is a set of files which can be
78
-read or written to, reflecting resource limits and a few general Cgroup properties.
79
-
80
-##### Hierarchy in Cgroups v1
81
-
82
-The Cgroups v1 is organized in a tree way:
83
-`/sys/fs/cgroup/{controller: cpu, cpuacct, cpupids, memory}/{cgroup_process: {process A, process B docker}/{rules: } `
84
-
85
-#### Cgroups v2
86
-
87
-In Cgroups v2, all mounted controllers reside in a single unified hierarchy. While (different)
88
-controllers may be simultaneously mounted under the v1 and v2 hierarchies, it is not possible to
89
-mount the same controller simultaneously under both the v1 and the v2 hierarchies.
90
-
91
-The new behaviors in Cgroups v2 are summarized here, and in some cases elaborated in the following
92
-subsections.
93
-
94
-1. Cgroups v2 provides a unified hierarchy against which all controllers are mounted.
95
-
96
-2. "Internal" processes are not permitted. Except of the root Cgroup, processes may reside only in
97
- leaf nodes
98
- (Cgroups that do not themselves contain child Cgroups). The details are somewhat more subtle than
99
- this, and are described below.
100
-
101
-3. Active Cgroups must be specified via the files cgroup.controllers and cgroup.subtree_control.
102
-
103
-4. The tasks file has been removed. In addition, the cgroup.clone_children file that is employed by
104
- the cpuset controller has been removed.
105
-
106
-5. An improved mechanism for notification of empty Cgroups is provided by the cgroup.events file.
107
-
108
- For more changes, see the Documentation/admin-guide/cgroup-v2.rst file in the kernel source (or
109
- Documentation/cgroup-v2.txt in Linux 4.17 and earlier).
110
-
111
-##### Hierarchy in Cgroups v2
112
-
113
-The Cgroups v2 is organized in a tree way:
114
-`/sys/fs/cgroup/{cgroup1: {/cgroup2, /cgroup3}, cgroup5: {/cgroup6: {/cgroup8} } } . . .`
115
-
116
-So every Cgroup contains over Cgroups, cgroup1 contains cgroup2 and cgroup3, cgroup5 contains
117
-cgroup6 which contains cgroup8 and so on so forth. So all the parent cgroups/processes shares their
118
-resource with their childs. Active Cgroups must be specified via the `cgroup.controllers` and
119
-`cgroup.subtree_control` files. Consult
120
-the [cgroup man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
121
-
122
-Cgroups v2 implements only a subset of the controllers available in Cgroups v1. **The two systems
123
-are implemented so that both v1 controllers and v2 controllers can be mounted on the same system**.
124
-Thus, for example, it is possible to use those controllers that are supported under version 2, while
125
-also using version 1 controllers where version 2 does not yet support those controllers. The only
126
-restriction here is that a controller can't be simultaneously employed in both a Cgroups v1
127
-hierarchy and in the Cgroups v2 hierarchy.
128
-
129
-#### Major between Cgroups v1 & v2:
130
-
131
-You might already have figure out some differences from the explanations above but let's note some
132
-of them.
133
-
134
-The biggest difference is that in Cgroups v2 a process can't join different groups for different
135
-controllers. For example in v1 a process could use the /sys/fs/cgroup/cpu/cgroupA and the
136
-/sys/fs/cgroup/memory/cgroupB controllers at the same time. In v2 a process joins only the
137
-/sys/fs/cgroups/cgroupC and is subject to all the controller of this Cgroup.
138
-
139
-Another difference is that, in Cgroups v1 when events like page cache writebacks or network packets
140
-reception occur, the resources allocated was not charged to the responsible Cgroup. For example,
141
-when your network interface receives packets, and the kernel is not aware of the destination Cgroup
142
-, the packets need to be processed first and re-routed to the corresponding subsystem or userland
143
-application. This operation was not included/calculated in the applications Cgroup.
144
-
145
-### More
146
-
147
-You can experiment with Cgroups, (safely, in a testing environment) create a new Cgroup and set limits
148
-in your system with tools like `cgcreate` and `cgset`from the `libcgroup-tools` package and then
149
-make your services run with those constrains. In the following link you can find a
150
-blog [in the linuxhint blog](https://linuxhint.com/limit_cpu_usage_process_linux/) which can guide
151
-you through it.
152
-
153
-</details>
154
-
155
-
156
-<details>
157
-<summary>References and sources:</summary>
158
-
159
-1. [cgroups(7) man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html)
160
-2. [linuxhint blog](https://linuxhint.com/limit_cpu_usage_process_linux/)
161
-3. [cgroups on Arch linux](https://wiki.archlinux.org/title/cgroups)
162
-
163
-</details>
164
-
165
-### Troubleshooting section
166
-
167
-When your service/app/container reaches it's Cgroup (cpu) hard limits (or on pressure the soft
168
-limits), it would be halted or/and start thrashing. You can imagine the case when a service halts in
169
-a lock, nightmare! If needed, you can extend those limits:
170
-
171
-<details>
172
-<summary>Native linux applications</summary>
173
-
174
-Control groups can be accessed with various tools, please consult this
175
-guide ([cgroups on Arch linux](https://wiki.archlinux.org/title/cgroups)) to do that
176
-
177
-For example if you would like to set a CPU share limit in your apache server:
178
-
179
- ```
180
- root@netdata # systemctl set-property --runtime httpd.service CPUShares=500
181
- ```
182
-
183
-**Note**: To change it permanently, omit the `--runtime` flag
184
-
185
-</details>
186
-
187
-<details>
188
-<summary>Docker containers</summary>
189
-
190
-Follow
191
-the [Configure the default CFS scheduler](https://docs.docker.com/config/containers/resource_constraints/#configure-the-default-cfs-scheduler)
192
-section in the official docs to do that.
193
-
194
-</details>
195
-
196
-<details>
197
-<summary>Kubernetes </summary>
198
-
199
-You can apply Cgroups restrictions in any manifest that creates Pods. For example in a Deployment,
200
-in the PodSpecs configuration you can apply limits under `..spec.resources.limits.cpu: <value>`
201
-Consult
202
-this [resource management for pods and container](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
203
-guide in the official docs.
204
-
205
-You can also apply limits per namespaces or other policy, which is the best practice for handling
206
-Kubernetes resources. Consult
207
-this [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) guide in the
208
-official docs.
209
-
210
-
211
-</details>
212
-
health/guides/cgroups/cgroup_ram_in_use.md
deleted
-208
@@ -1,208 +0,0 @@
1
-# cgroup_ram_in_use
2
-
3
-**Cgroups | Memory**
4
-
5
-_Control groups, usually referred to as Cgroups, are a Linux kernel feature which allow processes to
6
-be organized into hierarchical groups whose usage of various types of resources can then be limited
7
-and monitored. The Kernel's cgroup interface is provided through a pseudo-filesystem called
8
-cgroupfs (`/sys/fs/cgroup`). Grouping is implemented in the core Cgroup kernel code, while resource tracking and limits
9
-are implemented in a set of per-resource-type subsystems (memory, CPU, and so on)
10
-.<sup>[1](https://man7.org/linux/man-pages/man7/cgroups.7.html) </sup>_
11
-
12
-The Netdata Agent calculates the percentage of used memory in a Cgroup compared to it's hard memory
13
-limit. This alert indicates high Cgroup memory utilization. When the utilization reaches it's 100%
14
-limit, the Out Of Memory (OOM) killer will be invoked and will try to evict the process that has the
15
-least favorable OOM_score in this particular Cgroup.
16
-
17
-This alert is triggered in warning state when the percentage of used memory is between 80-90% and in
18
-critical state between 90-98%.
19
-
20
-
21
-<details>
22
-<summary>More about Cgroups</summary>
23
-
24
-Since Linux kernel 4.5 (March 2016) there are two implementations of Cgroups, v1 and v2.
25
-
26
-#### Cgroup controllers (in v1)
27
-
28
-In the vanilla kernel you will find the following controllers.
29
-
30
-- cpu: `CONFIG_CGROUP_SCHED`, Cgroups can be guaranteed a minimum number of "CPU shares" when a
31
- system is busy.
32
-- cpuacct: `CONFIG_CGROUP_CPUACCT`, This controller provides accounting for CPU usage by groups of
33
- processes.
34
-- cpuset: `CONFIG_CPUSETS`, This Cgroup can be used to bind the processes in a Cgroup to a specified
35
- set of CPUs and NUMA nodes.
36
-- memory: `CONFIG_MEMCG`, The memory controller supports reporting and limiting of process memory,
37
- kernel memory, and swap used by Cgroups.
38
-- devices: `CONFIG_CGROUP_DEVICE`, This supports controlling which processes may create (mknod)
39
- devices as well as open them for reading or writing. The policies may be specified as allow-lists
40
- and deny-lists. Hierarchy is enforced, so new rules must not violate existing rules for the target
41
- or ancestor Cgroups.
42
-- freezer: `CONFIG_CGROUP_FREEZER`, The freezer Cgroup can suspend and restore (resume) all
43
- processes in a Cgroup. Freezing a Cgroup /A also causes its children, for example, processes in
44
- /A/B, to be frozen.
45
-- net_cls: `CONFIG_CGROUP_NET_CLASSID`, This places a classid, specified for the Cgroup, on network
46
- packets created by a Cgroup. These classids can then be used in firewall rules, as well as used to
47
- shape traffic using `tc`. This applies only to packets leaving the Cgroup, not to traffic arriving
48
- at the Cgroup.
49
-- blkio: `CONFIG_BLK_CGROUP`, The blkio Cgroup controls and limits access to specified block devices
50
- by applying IO control in the form of throttling and upper limits against leaf nodes and
51
- intermediate nodes in the storage hierarchy. Two policies are available. The first is a
52
- proportional- weight time-based division of disk implemented with CFQ. This is in effect for leaf
53
- nodes using CFQ. The second is a throttling policy which specifies upper I/O rate limits on a
54
- device.
55
-- perf_event: `CONFIG_CGROUP_PERF`, This controller allows perf monitoring of the set of processes
56
- grouped in a Cgroup.
57
-- net_prio: `CONFIG_CGROUP_NET_PRIO`, This allows priorities to be specified, per network interface,
58
- for Cgroups.
59
-- hugetlb: `CONFIG_CGROUP_HUGETLB`, This supports limiting the use of huge pages by Cgroups.
60
-- pids: `CONFIG_CGROUP_PIDS` , This controller permits limiting the number of process that may be
61
- created in a Cgroup (
62
- and its descendants).
63
-- rdma: `CONFIG_CGROUP_RDMA`, The RDMA controller permits limiting the use of RDMA/IB- specific
64
- resources per Cgroup. _since Linux 4.11_
65
-
66
-Different variations of the Linux kernel can have more or less Cgroup controllers OR/AND enabled not
67
-all of them.
68
-
69
-#### Cgroups v1
70
-
71
-Under Cgroups v1, each controller may be mounted against a separate Cgroup filesystem that provides
72
-its own hierarchical organization of the processes on the system. It is also possible to co-mount
73
-multiple (or even all) Cgroups v1 controllers against the same Cgroup filesystem. That means that
74
-the mounted controllers manage the same hierarchical organization of processes.
75
-
76
-For each mounted hierarchy, the directory tree mirrors the control group hierarchy. Each control
77
-group is represented by a directory, with each of its child control Cgroups represented as a child
78
-directory. For instance, `/user/joe/1.session` represents control group 1.session, which is a child
79
-of Cgroup joe, which is a child of /user. Under each Cgroup directory is a set of files which can be
80
-read or written to, reflecting resource limits and a few general Cgroup properties.
81
-
82
-##### Hierarchy in Cgroups v1
83
-
84
-The Cgroups v1 is organized in a tree way:
85
-`/sys/fs/cgroup/{controller: cpu, cpuacct, cpupids, memory}/{cgroup_process: {process A, process B docker}/{rules: } `
86
-
87
-#### Cgroups v2
88
-
89
-In Cgroups v2, all mounted controllers reside in a single unified hierarchy. While (different)
90
-controllers may be simultaneously mounted under the v1 and v2 hierarchies, it is not possible to
91
-mount the same controller simultaneously under both the v1 and the v2 hierarchies.
92
-
93
-The new behaviors in Cgroups v2 are summarized here, and in some cases elaborated in the following
94
-subsections.
95
-
96
-1. Cgroups v2 provides a unified hierarchy against which all controllers are mounted.
97
-
98
-2. "Internal" processes are not permitted. Except of the root Cgroup, processes may reside only in
99
- leaf nodes
100
- (Cgroups that do not themselves contain child Cgroups). The details are somewhat more subtle than
101
- this, and are described below.
102
-
103
-3. Active Cgroups must be specified via the files cgroup.controllers and cgroup.subtree_control.
104
-
105
-4. The tasks file has been removed. In addition, the cgroup.clone_children file that is employed by
106
- the cpuset controller has been removed.
107
-
108
-5. An improved mechanism for notification of empty Cgroups is provided by the cgroup.events file.
109
-
110
- For more changes, see the Documentation/admin-guide/cgroup-v2.rst file in the kernel source (or
111
- Documentation/cgroup-v2.txt in Linux 4.17 and earlier).
112
-
113
-##### Hierarchy in Cgroups v2
114
-
115
-The Cgroups v2 is organized in a tree way:
116
-`/sys/fs/cgroup/{cgroup1: {/cgroup2, /cgroup3}, cgroup5: {/cgroup6: {/cgroup8} } } . . .`
117
-
118
-So every Cgroup contains over Cgroups, cgroup1 contains cgroup2 and cgroup3, cgroup5 contains
119
-cgroup6 which contains cgroup8 and so on so forth. So all the parent cgroups/processes shares their
120
-resource with their childs. Active Cgroups must be specified via the `cgroup.controllers` and
121
-`cgroup.subtree_control` files. Consult
122
-the [cgroup man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
123
-
124
-Cgroups v2 implements only a subset of the controllers available in Cgroups v1. **The two systems
125
-are implemented so that both v1 controllers and v2 controllers can be mounted on the same system**.
126
-Thus, for example, it is possible to use those controllers that are supported under version 2, while
127
-also using version 1 controllers where version 2 does not yet support those controllers. The only
128
-restriction here is that a controller can't be simultaneously employed in both a Cgroups v1
129
-hierarchy and in the Cgroups v2 hierarchy.
130
-
131
-#### Major between Cgroups v1 & v2:
132
-
133
-You might already have figure out some differences from the explanations above but let's note some
134
-of them.
135
-
136
-The biggest difference is that in Cgroups v2 a process can't join different groups for different
137
-controllers. For example in v1 a process could use the /sys/fs/cgroup/cpu/cgroupA and the
138
-/sys/fs/cgroup/memory/cgroupB controllers at the same time. In v2 a process joins only the
139
-/sys/fs/cgroups/cgroupC and is subject to all the controller of this Cgroup.
140
-
141
-Another difference is that, in Cgroups v1 when events like page cache writebacks or network packets
142
-reception occur, the resources allocated was not charged to the responsible Cgroup. For example,
143
-when your network interface receives packets, and the kernel is not aware of the destination Cgroup
144
-, the packets need to be processed first and re-routed to the corresponding subsystem or userland
145
-application. This operation was not included/calculated in the applications Cgroup.
146
-
147
-### More
148
-
149
-You can experiment with Cgroups, (safely, in a testing environment) create a new Cgroup and set limits
150
-in your system with tools like `cgcreate` and `cgset`from the `libcgroup-tools` package and then
151
-make your services run with those constrains. In the following link you can find a
152
-blog [in the linuxhint blog](https://linuxhint.com/limit_cpu_usage_process_linux/) which can guide
153
-you through it.
154
-
155
-</details>
156
-
157
-<details>
158
-<summary>References and sources:</summary>
159
-
160
-1. [cgroups(7) man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html)
161
-2. [linuxhint blog](https://linuxhint.com/limit_cpu_usage_process_linux/)
162
-3. [cgroups on Arch linux](https://wiki.archlinux.org/title/cgroups)
163
-
164
-</details>
165
-
166
-### Troubleshooting section
167
-
168
-<details>
169
-<summary>Native linux applications</summary>
170
-
171
-Control groups can be accessed with various tools, please consult this
172
-guide ([cgroups on Arch linux](https://wiki.archlinux.org/title/cgroups)) to do that.
173
-
174
-For example if you would like to set a Memory limit in your apache server:
175
-
176
- ```
177
- root@netdata # systemctl set-property httpd.service MemoryLimit=500M
178
- ```
179
-
180
-**Note**: To change it permanently, omit the `--runtime` flag
181
-
182
-</details>
183
-
184
-<details>
185
-<summary>Docker containers</summary>
186
-
187
-Follow
188
-the [limit a container's access to memory](https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory)
189
-section in the official docs to do that.
190
-
191
-</details>
192
-
193
-<details>
194
-<summary>Kubernetes </summary>
195
-
196
-You can apply restriction in any manifest that create Pods. For example in a Deployment, in the
197
-PodSpecs configuration you can apply limits under `..spec.resources.limits.memory: <value>`
198
-Consult
199
-this [resource management for pods and container](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
200
-guide in the official docs.
201
-
202
-You can also apply limits per namespaces or other policy, which is the best practice for handling
203
-Kubernetes resources. Consult
204
-this [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) guide in the
205
-official docs.
206
-
207
-
208
-</details>
health/guides/cockroachdb/cockroachdb_open_file_descriptors_limit.md
deleted
-33
@@ -1,33 +0,0 @@
1
-# cockroachdb_open_file_descriptors_limit
2
-
3
-## Database | CockroachDB
4
-
5
-This alert presents the percentage of used file descriptors for CockroachDB.
6
-If you receive this, it means that there is high file descriptor utilization against the
7
-soft-limit.
8
-
9
-This alert is raised in a warning state when the metric exceeds 80%.
10
-
11
-> In Unix and Unix-like computer operating systems, a file descriptor (FD, less frequently
12
-> fildes) is a unique identifier (handle) for a file or other input/output resource, such as a
13
-> pipe or network socket.
14
->
15
-> File descriptors typically have non-negative integer values, with negative values being
16
-> reserved to indicate "no value" or error conditions.<sup>[1](
17
-> https://en.wikipedia.org/wiki/File_descriptor) </sup>
18
-
19
-<details><summary>References and Sources</summary>
20
-
21
-1. [CockroachDB documentation](
22
- https://www.cockroachlabs.com/docs/v21.2/recommended-production-settings#file-descriptors-limit)
23
-
24
-</details>
25
-
26
-### Troubleshooting Section
27
-
28
-<details><summary>Adjust the file descriptors limit for the process or system-wide</summary>
29
-
30
-Check out the [CockroachDB documentation](
31
-https://www.cockroachlabs.com/docs/v21.2/recommended-production-settings#file-descriptors-limit) for troubleshooting advice.
32
-
33
-</details>
health/guides/cockroachdb/cockroachdb_unavailable_ranges.md
deleted
-39
@@ -1,39 +0,0 @@
1
-# cockroachdb_unavailable_ranges
2
-
3
-## Database | CockroachDB
4
-
5
-This alert presents the number of unavailable ranges. If you receive this, it indicates that there
6
-are ranges with fewer live replicas than needed for quorum.
7
-
8
-This alert is raised in a warning state when unavailable ranges start to exist.
9
-
10
-<details><summary>What are unavailable ranges?</summary>
11
-
12
-> Unavailable ranges: If a majority of a range's replicas are on nodes that are unavailable,
13
-> then the entire range is unavailable and will be unable to process queries.
14
->
15
-> CockroachDB uses consensus replication and requires a quorum of the replicas to
16
-> be available in order to allow both writes and reads to the range. The number of failures
17
-> that can be tolerated is equal to (Replication factor - 1)/2. Thus, CockroachDB requires (n-1)
18
-> /2 nodes to achieve quorum. For example, with 3x replication, one failure can be tolerated;
19
-> with 5x replication, two failures, and so on.<sup>[1](
20
-> https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges) </sup>
21
-
22
-</details>
23
-
24
-<details><summary>References and Sources</summary>
25
-
26
-1. [CockroachDB docs](
27
- https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges)
28
-
29
-</details>
30
-
31
-### Troubleshooting Section
32
-
33
-<details><summary>Identify unavailable ranges</summary>
34
-
35
-Check out the [CockroachDB documentation](
36
-https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges) for troubleshooting advice.
37
-
38
-
39
-</details>
health/guides/cockroachdb/cockroachdb_underreplicated_ranges.md
deleted
-35
@@ -1,35 +0,0 @@
1
-# cockroachdb_underreplicated_ranges
2
-
3
-## Database | CockroachDB
4
-
5
-This alert presents the number of under-replicated ranges.
6
-
7
-This alert is raised in a warning state when under-replicated ranges start to exist.
8
-
9
-> Under-replicated ranges: When a cluster is first initialized, the few default starting ranges
10
-> will only have a single replica, but as soon as other nodes are available, they will
11
-> replicate to them until they've reached their desired replication factor. If a range does not
12
-> have enough replicas, the range is said to be "under-replicated".
13
->
14
-> CockroachDB uses consensus replication and requires a quorum of the replicas to
15
-> be available in order to allow both writes and reads to the range. The number of failures
16
-> that can be tolerated is equal to (Replication factor - 1)/2. Thus, CockroachDB requires (n-1)
17
-> /2 nodes to achieve quorum. For example, with 3x replication, one failure can be tolerated;
18
-> with 5x replication, two failures, and so on.<sup>[1](
19
-> https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges) </sup>
20
-
21
-<details><summary>References and Sources</summary>
22
-
23
-1. [CockroachDB documentation](
24
- https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges)
25
-
26
-</details>
27
-
28
-### Troubleshooting Section
29
-
30
-<details><summary>Identify under-replicated ranges</summary>
31
-
32
-Check out the [CockroachDB documentation](
33
-https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges)
34
-for troubleshooting advice.
35
-</details>
health/guides/cockroachdb/cockroachdb_used_storage_capacity.md
deleted
-39
@@ -1,39 +0,0 @@
1
-# cockroachdb_used_storage_capacity
2
-
3
-## Database | CockroachDB
4
-
5
-This alert presents the percentage of used storage space.
6
-If you receive this alert, then there is high storage capacity utilization from CockroachDB.
7
-
8
-- This alert is raised to warning when the metric exceeds 85%.
9
-- If the percentage of used storage space exceeds 95%, then the alert is raised to critical.
10
-
11
-Definition of "size" on CockroachDB:
12
-
13
-> The maximum size allocated to the node. When this size is reached, CockroachDB attempts to
14
-> rebalance data to other nodes with available capacity. When there's no capacity elsewhere,
15
-> this limit will be exceeded. Also, data may be written to the node faster than the cluster
16
-> can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB
17
-> will gradually rebalance data down to the store limit.<sup>[1](
18
-> https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store) </sup>
19
-
20
-
21
-<details><summary>References and Sources</summary>
22
-
23
-1. [CockroachDB Size](https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store)
24
-2. [CockroachDB Docs](https://www.cockroachlabs.com/docs/stable/ui-storage-dashboard.html)
25
-</details>
26
-
27
-### Troubleshooting Section
28
-
29
-<details><summary>Increase the space available for CockroachDB data</summary>
30
-
31
-If you had previously set a limit, then you can use the option `--store=path<YOUR PATH>,size=<SIZE>`
32
-to increase the amount of available space.
33
-Make sure to replace the "YOUR PATH" with the actual store path and "SIZE" with the new size you want to set CockroachDB to.
34
-
35
-> Note: If you haven't set a limit on the size, then the entire drive's size will be used. In this
36
-> case, you will see that the drive is full. Clearing some space or upgrading to a drive with a
37
-> larger capacity are potential solutions.
38
-
39
-</details>
health/guides/cockroachdb/cockroachdb_used_usable_storage_capacity.md
deleted
-34
@@ -1,34 +0,0 @@
1
-# cockroachdb_used_usable_storage_capacity
2
-
3
-## Database | CockroachDB
4
-
5
-This alert presents the percentage of used usable storage space allocated on CockroachDB.
6
-
7
-If you receive this alert, it means that the usable space for CockroachDB is being highly utilized.
8
-
9
-- This alert is raised to warning when the metric exceeds 85%.
10
-- If the percentage of used usable storage space exceeds 95%, then the alert is raised to critical.
11
-
12
-Definition of "size" on CockroachDB:
13
-
14
-> The maximum size allocated to the node. When this size is reached, CockroachDB attempts to
15
-> rebalance data to other nodes with available capacity. When there's no capacity elsewhere,
16
-> this limit will be exceeded. Also, data may be written to the node faster than the cluster
17
-> can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB
18
-> will gradually rebalance data down to the store limit.<sup>[1](
19
-> https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store) </sup>
20
-
21
-<details><summary>References and Sources</summary>
22
-
23
-1. [CockroachDB Size](https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store)
24
-2. [CockroachDB Docs](https://www.cockroachlabs.com/docs/stable/ui-storage-dashboard.html)
25
-
26
-</details>
27
-
28
-### Troubleshooting Section
29
-
30
-<details><summary>Increase the space available for CockroachDB data</summary>
31
-
32
-You can use the option `--store=path<YOUR PATH>,size=<SIZE>` Make sure to replace the "YOUR PATH"
33
-with the actual store path and "SIZE" with the new size you want to set CockroachDB to.
34
-</details>
health/guides/cockroachdb_open_file_descriptors_limit.md
new
+57
@@ -0,0 +1,57 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the usage of file descriptors in your CockroachDB is reaching a high percentage against the soft-limit. High file descriptor utilization can cause issues, such as failures to open new files or establish network connections.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the current file descriptor limit and usage for CockroachDB:
8
+
9
+ Use the `lsof` command to display information about all open file descriptors associated with the process running CockroachDB:
10
+
11
+ ```
12
+ lsof -p <PID>
13
+ ```
14
+
15
+ Replace `<PID>` with the process ID of CockroachDB.
16
+
17
+ To display only the total number of open file descriptors, you can use this command:
18
+
19
+ ```
20
+ lsof -p <PID> | wc -l
21
+ ```
22
+
23
+2. Monitor file descriptor usage:
24
+
25
+ Regularly monitoring file descriptor usage can help you identify patterns and trends, making it easier to determine if adjustments are needed. You can use tools like `lsof` or `sar` to monitor file descriptor usage on your system.
26
+
27
+3. Adjust the file descriptors limit for the process:
28
+
29
+ You can raise the soft-limit for the CockroachDB process by modifying the `ulimit` configuration:
30
+
31
+ ```
32
+ ulimit -n <new_limit>
33
+ ```
34
+
35
+ Replace `<new_limit>` with the desired value, which must be less than or equal to the system-wide hard limit.
36
+
37
+ Note that changes made using `ulimit` only apply to the current shell session. To make the changes persistent, you should add the `ulimit` command to the CockroachDB service startup script or modify the system-wide limits in `/etc/security/limits.conf`.
38
+
39
+4. Adjust the system-wide file descriptors limit:
40
+
41
+ If necessary, you can also adjust the system-wide limits for file descriptors in `/etc/security/limits.conf`. Edit this file as a root user, and add or modify the following lines:
42
+
43
+ ```
44
+ * soft nofile <new_soft_limit>
45
+ * hard nofile <new_hard_limit>
46
+ ```
47
+
48
+ Replace `<new_soft_limit>` and `<new_hard_limit>` with the desired values. You must restart the system or CockroachDB for the changes to take effect.
49
+
50
+5. Optimize CockroachDB configuration:
51
+
52
+ Review the CockroachDB configuration and ensure that it's optimized for your workload. If appropriate, adjust settings such as cache size, query optimization, and memory usage to reduce the number of file descriptors needed.
53
+
54
+### Useful resources
55
+
56
+1. [CockroachDB recommended production settings](https://www.cockroachlabs.com/docs/v21.2/recommended-production-settings#file-descriptors-limit)
57
+2. [Increasing file descriptor limits on Linux](https://www.tecmint.com/increase-set-open-file-limits-in-linux/)
health/guides/cockroachdb_unavailable_ranges.md
new
+51
@@ -0,0 +1,51 @@
1
+### Understand the alert
2
+
3
+This alert indicates that there are unavailable ranges in your CockroachDB cluster. Unavailable ranges occur when a majority of a range's replicas are on nodes that are unavailable. This can cause the entire range to be unable to process queries.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check for dead or unavailable nodes
8
+
9
+ Use the `./cockroach node status` command to list the status of all nodes in your cluster. Look for nodes that are marked as dead or unavailable and try to bring them back online.
10
+
11
+ ```
12
+ ./cockroach node status --certs-dir=<your_cert_directory>
13
+ ```
14
+
15
+2. Inspect the logs
16
+
17
+ CockroachDB logs can provide valuable information about issues that may be affecting your cluster. Check the logs for errors or warnings related to unavailable ranges using `grep`:
18
+
19
+ ```
20
+ grep -i 'unavailable range' /path/to/cockroachdb/logs
21
+ ```
22
+
23
+3. Check replication factor
24
+
25
+ Make sure your cluster's replication factor is set to an appropriate value. A higher replication factor can help tolerate node failures and prevent unavailable ranges. You can check the replication factor by running the following SQL query:
26
+
27
+ ```
28
+ SHOW CLUSTER SETTING kv.range_replicas;
29
+ ```
30
+
31
+ To set the replication factor, run the following SQL command:
32
+
33
+ ```
34
+ SET CLUSTER SETTING kv.range_replicas=<desired_replication_factor>;
35
+ ```
36
+
37
+4. Investigate and resolve network issues
38
+
39
+ Network issues can cause nodes to become unavailable and lead to unavailable ranges. Check the status of your network and any firewalls, load balancers, or other network components that may be affecting connectivity between nodes.
40
+
41
+5. Monitor and manage hardware resources
42
+
43
+ Insufficient hardware resources, such as CPU, memory, or disk space, can cause nodes to become unavailable. Monitor your nodes' resource usage and ensure that they have adequate resources to handle the workload.
44
+
45
+6. Consider rebalancing the cluster
46
+
47
+ Rebalancing the cluster can help distribute the load more evenly across nodes and reduce the number of unavailable ranges. See the [CockroachDB documentation](https://www.cockroachlabs.com/docs/stable/training/manual-rebalancing.html) for more information on manual rebalancing.
48
+
49
+### Useful resources
50
+
51
+1. [CockroachDB troubleshooting guide](https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges)
health/guides/cockroachdb_underreplicated_ranges.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert is related to CockroachDB, a scalable and distributed SQL database. When you receive this alert, it means that there are under-replicated ranges in your database cluster. Under-replicated ranges can impact the availability and fault tolerance of your database, leading to potential data loss or unavailability in case of node failures.
4
+
5
+### What are under-replicated ranges?
6
+
7
+In a CockroachDB cluster, data is split into small chunks called ranges. These ranges are then replicated across multiple nodes to ensure fault tolerance and high availability. The desired replication factor determines the number of replicas for each range.
8
+
9
+When a range has fewer replicas than the desired replication factor, it is considered as "under-replicated". This situation can occur if nodes are unavailable or if the cluster is in the process of recovering from failures.
10
+
11
+### Troubleshoot the alert
12
+
13
+1. Access the CockroachDB Admin UI
14
+
15
+ Access the Admin UI by navigating to the URL `http://<any-node-ip>:8080` on any of your cluster nodes.
16
+
17
+2. Check the 'Replication Status' in the dashboard
18
+
19
+ In the Admin UI, check the 'Under-replicated Ranges' metric on the main 'Dashboard' or 'Metrics' page.
20
+
21
+3. Inspect the logs of your CockroachDB nodes
22
+
23
+ Look for any error messages or issues that could be causing under-replication. For example, you may see errors related to node failures or network issues.
24
+
25
+4. Check cluster health and capacity
26
+
27
+ Make sure that all nodes in the cluster are running and healthy. You can do this by running the command `cockroach node status`. Consider adding more nodes or increasing the capacity if your nodes are overworked.
28
+
29
+5. Verify replication factor configuration
30
+
31
+ Check your cluster's replication factor configuration to ensure it is set to an appropriate value. The default replication factor is 3, which can tolerate one failure. You can view and change it using the [`zone configurations`](https://www.cockroachlabs.com/docs/stable/configure-replication-zones.html).
32
+
33
+6. Consider decommissioning problematic nodes
34
+
35
+ If specific nodes are causing under-replication, consider decommissioning them to allow the cluster to automatically rebalance the ranges. Follow the [decommissioning guide](https://www.cockroachlabs.com/docs/stable/remove-nodes.html) in the CockroachDB documentation.
36
+
37
+### Useful resources
38
+
39
+1. [CockroachDB: Troubleshoot Under-replicated and Unavailable Ranges](https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#db-console-shows-under-replicated-unavailable-ranges)
40
+2. [CockroachDB: Configuring Replication Zones](https://www.cockroachlabs.com/docs/stable/configure-replication-zones.html)
41
+3. [CockroachDB: Decommission a Node](https://www.cockroachlabs.com/docs/stable/remove-nodes.html)
\ No newline at end of file
health/guides/cockroachdb_used_storage_capacity.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the Alert
2
+
3
+This alert indicates high storage capacity utilization in CockroachDB.
4
+
5
+### Definition of "size" on CockroachDB:
6
+
7
+The maximum size allocated to the node. When this size is reached, CockroachDB attempts to rebalance data to other nodes with available capacity. When there's no capacity elsewhere, this limit will be exceeded. Also, data may be written to the node faster than the cluster can rebalance it away; in this case, as long as capacity is available elsewhere, CockroachDB will gradually rebalance data down to the store limit.
8
+
9
+### Troubleshoot the Alert
10
+
11
+- Increase the space available for CockroachDB data
12
+
13
+If you had previously set a limit, then you can use the option `--store=path<YOUR PATH>,size=<SIZE>` to increase the amount of available space. Make sure to replace the "YOUR PATH" with the actual store path and "SIZE" with the new size you want to set CockroachDB to.
14
+
15
+Note: If you haven't set a limit on the size, then the entire drive's size will be used. In this case, you will see that the drive is full. Clearing some space or upgrading to a drive with a larger capacity are potential solutions.
16
+
17
+- Inspect the disk usage by tables and indexes
18
+
19
+CockroachDB provides the `experimental_disk_usage` builtin SQL function that allows you to check the disk usage by tables and indexes within a given database. This can help you identify the main storage consumers in your cluster.
20
+
21
+To run this command, first connect to your CockroachDB instance with `cockroach sql`, then execute the following query:
22
+
23
+```sql
24
+SELECT * FROM [SHOW experimental_disk_usage('<database_name>')];
25
+```
26
+
27
+Make sure to replace `<database_name>` with the actual name of the database you want to inspect. This will return a list of tables and indexes with their respective disk usage.
28
+
29
+- Rebalance the cluster data to other nodes with available capacity
30
+
31
+CockroachDB automatically rebalances data across nodes by default. If the data rebalancing is not happening fast enough, you can try to speed up this process by [adjusting `zone configurations`](https://www.cockroachlabs.com/docs/stable/configure-replication-zones.html) or by [increasing the default rebalancing rate](https://www.cockroachlabs.com/docs/stable/cluster-settings.html#kv_range_replication_rate_bytes_per_second).
32
+
33
+- Purge old, unnecessary data
34
+
35
+Inspect your data and consider purging old or unnecessary data from the database. Be cautious while performing this operation and double-check the data you intend to remove.
36
+
37
+- Archive old data
38
+
39
+If the data cannot be purged, consider archiving it in a more compact format or moving it to a separate database or storage system to reduce the storage usage on the affected CockroachDB node.
40
+
41
+
42
+## Useful resources
43
+
44
+1. [CockroachDB Size](https://www.cockroachlabs.com/docs/v21.2/cockroach-start#store)
45
+2. [CockroachDB Docs](https://www.cockroachlabs.com/docs/stable/ui-storage-dashboard.html)
46
+
health/guides/cockroachdb_used_usable_storage_capacity.md
new
+63
@@ -0,0 +1,63 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the usable storage space allocated for your CockroachDB is being highly utilized. If the percentage of used space exceeds 85%, the alert raises a warning, and if it exceeds 95%, the alert becomes critical. High storage utilization can lead to performance issues and potential data loss if not properly managed.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the current storage utilization
8
+
9
+To understand the current utilization, you can use SQL commands to query the `crdb_internal.kv_store_status` table.
10
+
11
+```sql
12
+SELECT node_id, store_id, capacity, used, available
13
+FROM crdb_internal.kv_store_status;
14
+```
15
+
16
+This query will provide information about the available and used storage capacity of each node in your CockroachDB cluster.
17
+
18
+2. Identify tables and databases with high storage usage
19
+
20
+Use the following command to list the top databases in terms of storage usage:
21
+
22
+```sql
23
+SELECT database_name, sum(data_size_int) as total_size
24
+FROM crdb_internal.tables
25
+WHERE database_name != 'crdb_internal'
26
+GROUP BY database_name
27
+ORDER BY total_size DESC
28
+LIMIT 10;
29
+```
30
+
31
+Additionally, you can list the top tables in terms of storage usage:
32
+
33
+```sql
34
+SELECT database_name, table_name, data_size
35
+FROM crdb_internal.tables
36
+WHERE database_name != 'crdb_internal'
37
+ORDER BY data_size_int DESC
38
+LIMIT 10;
39
+```
40
+
41
+3. Optimize storage usage
42
+
43
+Based on your findings from steps 1 and 2, consider the following actions:
44
+
45
+- Delete unneeded data from tables with high storage usage.
46
+- Apply data compression to reduce the overall storage consumption.
47
+- Archive old data or move it to external storage.
48
+
49
+4. Add more storage to the nodes
50
+
51
+If necessary, increase the storage allocated to your CockroachDB cluster by adding more space to each node.
52
+
53
+- To increase the usable storage capacity, modify the `--store` flag when restarting your CockroachDB nodes. Set the new size by replacing `<YOUR_PATH>` with the actual store path and `<SIZE>` with the desired new size:
54
+
55
+ ```
56
+ --store=path=<YOUR_PATH>,size=<SIZE>
57
+ ```
58
+
59
+5. Add more nodes to the cluster
60
+
61
+If increasing the storage capacity of your existing nodes isn't enough, consider adding more nodes to your CockroachDB cluster. By adding more nodes, you can distribute storage more evenly and prevent single points of failure due to storage limitations.
62
+
63
+Refer to the [CockroachDB documentation](https://www.cockroachlabs.com/docs/stable/start-a-node.html) on how to add a new node to a cluster.
\ No newline at end of file
health/guides/consul_autopilot_health_status.md
new
+53
@@ -0,0 +1,53 @@
1
+### Understand the alert
2
+
3
+This alert checks the health status of the Consul cluster regarding its autopilot functionality. If you receive this alert, it means that the Consul datacenter is experiencing issues, and its health status has been reported as `unhealthy` by the Consul server.
4
+
5
+### What is Consul autopilot?
6
+
7
+Consul's autopilot feature provides automatic management and stabilization features for Consul server clusters, ensuring that the clusters remain in a healthy state. These features include server health monitoring, automatic dead server reaping, and stable server introduction.
8
+
9
+### What does unhealthy mean?
10
+
11
+An unhealthy Consul cluster could experience issues regarding its operations, services, leader elections, and cluster consistency. In this alert scenario, the cluster health functionality is not working correctly, and it could lead to stability and performance problems.
12
+
13
+### Troubleshoot the alert
14
+
15
+Here are some steps to troubleshoot the consul_autopilot_health_status alert:
16
+
17
+1. Check the logs of the Consul server to identify any error messages or warning signs. The logs will often provide insights into the underlying problems.
18
+
19
+ ```
20
+ journalctl -u consul
21
+ ```
22
+
23
+2. Inspect the Consul health status using the Consul CLI or API:
24
+
25
+ ```
26
+ consul operator autopilot get-config
27
+ ```
28
+
29
+ Using the Consul HTTP API:
30
+ ```
31
+ curl http://<consul_server>:8500/v1/operator/autopilot/health
32
+ ```
33
+
34
+3. Verify the configuration of Consul servers, check the `retry_join` and addresses of the Consul servers in the configuration file:
35
+
36
+ ```
37
+ cat /etc/consul.d/consul.hcl | grep retry_join
38
+ ```
39
+
40
+4. Ensure that there is a sufficient number of Consul servers and that they are healthy. The `consul members` command will show the status of cluster members:
41
+
42
+ ```
43
+ consul members
44
+ ```
45
+
46
+5. Check the network connectivity between Consul servers by running network diagnostics like ping and traceroute.
47
+
48
+6. Review Consul documentation to gain a deeper understanding of the autopilot health issues and potential configuration problems.
49
+
50
+
51
+### Useful resources
52
+
53
+- [Consul CLI reference](https://www.consul.io/docs/commands)
health/guides/consul_autopilot_server_health_status.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+The `consul_autopilot_server_health_status` alert triggers when a Consul server in your service mesh is marked `unhealthy`. This can affect the overall stability and performance of the service mesh. Regular monitoring and addressing unhealthy servers are crucial in maintaining a smooth functioning environment.
4
+
5
+### What is Consul?
6
+
7
+`Consul` is a service mesh solution that provides a full-featured control plane with service discovery, configuration, and segmentation functionalities. It is used to connect, secure, and configure services across any runtime platform and public or private cloud.
8
+
9
+### Troubleshoot the alert
10
+
11
+Follow the steps below to identify and resolve the issue of an unhealthy Consul server:
12
+
13
+1. Check Consul server logs
14
+
15
+ Inspect the logs of the unhealthy server to identify the root cause of the issue. You can find logs typically in `/var/log/consul` or use `journalctl` with Consul:
16
+
17
+ ```
18
+ journalctl -u consul
19
+ ```
20
+
21
+2. Verify connectivity
22
+
23
+ Ensure that the unhealthy server can communicate with other servers in the datacenter. Check for any misconfigurations or network issues.
24
+
25
+3. Review server resources
26
+
27
+ Monitor the resource usage of the unhealthy server (CPU, memory, disk I/O, network). High resource usage can impact the server's health status. Use tools like `top`, `htop`, `iotop`, or `nload` to monitor the resources.
28
+
29
+4. Restart the Consul server
30
+
31
+ If the issue persists and you cannot identify the root cause, try restarting the Consul server:
32
+
33
+ ```
34
+ sudo systemctl restart consul
35
+ ```
36
+
37
+5. Refer to Consul's documentation
38
+
39
+ Consult the official [Consul troubleshooting documentation](https://developer.hashicorp.com/consul/tutorials/datacenter-operations/troubleshooting) for further assistance.
40
+
41
+6. Inspect the Consul UI
42
+
43
+ Check the Consul UI for the server health status and any additional information related to the unhealthy server. You can find the Consul UI at `http://<consul-server-ip>:8500/ui/`.
44
+
45
+### Useful resources
46
+
47
+1. [Consul Documentation](https://www.consul.io/docs)
48
+2. [Running Consul as a Systemd Service](https://learn.hashicorp.com/tutorials/consul/deployment-guide#systemd-service)
health/guides/consul_client_rpc_requests_exceeded.md
new
+38
@@ -0,0 +1,38 @@
1
+### Understand the alert
2
+
3
+This alert triggers when the rate of rate-limited RPC (Remote Procedure Call) requests made by a Consul server within the specified datacenter has exceeded a certain threshold. If you receive this alert, it means that your Consul server is experiencing an increased number of rate-limited RPC requests, which may affect its performance and availability.
4
+
5
+### What is Consul?
6
+
7
+Consul is a service mesh solution used for service discovery, configuration, and segmentation. It provides a distributed platform to build robust, scalable, and secured services while simplifying network infrastructure.
8
+
9
+### What are RPC requests?
10
+
11
+Remote Procedure Call (RPC) is a protocol that allows a computer to execute a procedure on another computer across a network. In the context of Consul, RPC requests are used for communication between Consul servers and clients.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the Consul server logs for any relevant error messages or warnings. These logs can provide valuable information on the cause of the increased RPC requests.
16
+
17
+ ```
18
+ journalctl -u consul
19
+ ```
20
+
21
+2. Monitor the Consul server's resource usage, such as CPU and memory utilization, to ensure that it is not running out of resources. High resource usage may cause an increase in rate-limited RPC requests.
22
+
23
+ ```
24
+ top -o +%CPU
25
+ ```
26
+
27
+3. Analyze the Consul client's usage patterns and identify any misconfigured services or clients contributing to the increased RPC requests. Identify any services that may be sending a high number of requests per second or are not appropriately rate-limited.
28
+
29
+4. Review the Consul rate-limiting configurations to ensure that they are set appropriately based on the expected workload. Adjust the rate limits if necessary to better accommodate the workload.
30
+
31
+5. If the issue persists, consider scaling up the Consul server resources or deploying more Consul servers to handle increased traffic and prevent performance issues.
32
+
33
+### Useful resources
34
+
35
+1. [Consul Official Documentation](https://www.consul.io/docs/)
36
+2. [Consul Rate Limiting Guide](https://developer.hashicorp.com/consul/docs/agent/limits)
37
+3. [Understanding Remote Procedure Calls (RPC)](https://www.smashingmagazine.com/2016/09/understanding-rest-and-rpc-for-http-apis/)
38
+4. [Troubleshooting Consul](https://developer.hashicorp.com/consul/tutorials/datacenter-operations/troubleshooting)
health/guides/consul_client_rpc_requests_failed.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of failed RPC (Remote Procedure Call) requests made by the Consul server in a datacenter surpasses a specific threshold. Consul is a service mesh solution and is responsible for discovering, configuring, and segmenting services in distributed systems.
4
+
5
+### What are RPC requests?
6
+
7
+Remote Procedure Call (RPC) is a protocol that allows one computer to execute remote procedures (subroutines) on another computer. In the context of Consul, clients make RPC requests to servers to obtain information about the service configurations or to execute actions.
8
+
9
+### What does it mean when RPC requests fail?
10
+
11
+When Consul's client RPC requests fail, it means that there is an issue in the communication between the Consul client and the server. It could be due to various reasons like network issues, incorrect configurations, high server load, or even software bugs.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Verify the connectivity between Consul clients and servers.
16
+
17
+ Check the network connections between the Consul client and the server. Ensure that the required ports are open and the network is functioning correctly. You can use tools like `ping`, `traceroute`, and `telnet` to verify connectivity.
18
+
19
+2. Check Consul server logs.
20
+
21
+ Analyze the Consul server's logs to look for any error messages or unusual patterns related to RPC requests. Server logs can be found in the default Consul log directory, usually `/var/log/consul`.
22
+
23
+3. Review Consul client and server configurations.
24
+
25
+ Ensure that Consul client and server configurations are correct and in accordance with the best practices. You can find more information about Consul's configuration recommendations [here](https://learn.hashicorp.com/tutorials/consul/reference-architecture?in=consul/production-deploy).
26
+
27
+4. Monitor server load and resources.
28
+
29
+ High server load or resource constraints can cause RPC request failures. Monitor your Consul servers' CPU, memory, and disk usage. If you find any resource bottlenecks, consider adjusting the server's resource allocation or scaling your Consul servers horizontally.
30
+
31
+5. Update Consul to the latest version.
32
+
33
+ Software bugs can lead to RPC request failures. Ensure that your Consul clients and servers are running the latest version of Consul. Check the [Consul releases page](https://github.com/hashicorp/consul/releases) for the latest version.
34
+
35
+### Useful resources
36
+
37
+1. [Consul official documentation](https://www.consul.io/docs)
38
+2. [Consul Reference Architecture](https://learn.hashicorp.com/tutorials/consul/reference-architecture?in=consul/production-deploy)
39
+3. [Troubleshooting Consul guide](https://developer.hashicorp.com/consul/tutorials/datacenter-operations/troubleshooting)
health/guides/consul_gc_pause_time.md
new
+23
@@ -0,0 +1,23 @@
1
+### Understand the alert
2
+
3
+This alert calculates the time spent in stop-the-world garbage collection (GC) pauses on a Consul server node within a one-minute interval. Consul is a distributed service mesh software providing service discovery, configuration, and segmentation functionality. If you receive this alert, it means that the Consul server is experiencing an increased amount of time in GC pauses, which may lead to performance degradation of your service mesh.
4
+
5
+### What are garbage collection pauses?
6
+
7
+Garbage collection (GC) in Consul is a mechanism to clean up unused memory resources and improve the overall system performance. During a GC pause, all running processes in Consul server are stopped to allow the garbage collection process to complete. If the duration of GC pauses is too high, it indicates that the Consul server might be under memory pressure, which can affect the overall performance of the system.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. **Check the Consul server logs**: Examine the Consul server's logs for any errors or warnings related to memory pressure, increased heap usage, or GC pauses. You can typically find the logs in `/var/log/consul`.
12
+
13
+2. **Monitor Consul server metrics**: Check the Consul server's memory usage, heap usage and GC pause metrics using or Netdata. This can help you identify the cause of increased GC pause time.
14
+
15
+3. **Optimize Consul server configuration**: Ensure that your Consul server is properly configured based on your system resources and workload. Review and adjust the [Consul server configuration parameters](https://www.consul.io/docs/agent/options) as needed.
16
+
17
+4. **Reduce memory pressure**: If you have identified memory pressure as the root cause, consider adding more memory resources to your Consul server or adjusting the Consul server's memory limits.
18
+
19
+5. **Update Consul server**: Make sure that your Consul server is running the latest version, which can include optimizations and performance improvements.
20
+
21
+### Useful resources
22
+
23
+- [Consul Server Configuration Parameters](https://www.consul.io/docs/agent/options)
health/guides/consul_license_expiration_time.md
new
+50
@@ -0,0 +1,50 @@
1
+### Understand the alert
2
+
3
+This alert checks the Consul Enterprise license expiration time. It triggers a warning if the license expiration time is less than 14 days, and critical if it's less than 7 days.
4
+
5
+_consul.license_expiration_time_: Monitors the remaining time in seconds until the Consul Enterprise license expires.
6
+
7
+### What is Consul?
8
+
9
+Consul is a service mesh solution that enables organizations to discover services and safely process network traffic across dynamic, distributed environments.
10
+
11
+### Troubleshoot the alert
12
+
13
+1. Check the current license expiration time
14
+
15
+ You can check the remaining license expiration time for your Consul Enterprise instance using the Consul API:
16
+
17
+ ```
18
+ curl http://localhost:8500/v1/operator/license
19
+ ```
20
+
21
+ Look for the `ExpirationTime` field in the returned JSON output.
22
+
23
+2. Renew the license
24
+
25
+ If your license is about to expire, you will need to acquire a new license. Contact [HashiCorp Support](https://support.hashicorp.com/) to obtain and renew the license key.
26
+
27
+3. Apply the new license
28
+
29
+ You can apply the new license key either by restarting Consul with the new key specified via the `CONSUL_LICENSE` environment variable or the `license_path` configuration option, or by updating the license through the Consul API:
30
+
31
+ ```
32
+ curl -X PUT -d @new_license.json http://localhost:8500/v1/operator/license
33
+ ```
34
+
35
+ Replace `new_license.json` with the path to a file containing the new license key in JSON format.
36
+
37
+4. Verify the new license expiration time
38
+
39
+ After applying the new license, you can check the new license expiration time using the Consul API again:
40
+
41
+ ```
42
+ curl http://localhost:8500/v1/operator/license
43
+ ```
44
+
45
+ Ensure that the `ExpirationTime` field shows the new expiration time.
46
+
47
+### Useful resources
48
+
49
+1. [Consul License Documentation](https://www.consul.io/docs/enterprise/license)
50
+2. [HashiCorp Support](https://support.hashicorp.com/)
health/guides/consul_node_health_check_status.md
new
+34
@@ -0,0 +1,34 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a Consul node health check status indicates a failure. Consul is a service mesh solution for service discovery and configuration. If you receive this alert, it means that the health check for a specific service on a node within the Consul cluster has failed.
4
+
5
+### What does the health check status mean?
6
+
7
+Consul performs health checks to ensure the services registered within the cluster are functioning as expected. The health check status represents the result of these checks, with a non-zero value indicating a failed health check. A failed health check can potentially cause downtime or degraded performance for the affected service.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the alert details: The alert information provided should include the `check_name`, `node_name`, and `datacenter` affected. Note these details as they will be useful in further troubleshooting.
12
+
13
+2. Verify the health check status in Consul: To confirm the health check failure, access the Consul UI or use the Consul command-line tool to query the health status of the affected service and node:
14
+
15
+ ```
16
+ consul members
17
+ ```
18
+
19
+ ```
20
+ consul monitor
21
+ ```
22
+
23
+3. Investigate the failed service: Once you confirm the health check failure, start investigating the specific service affected. Check logs, resource usage, configuration files, and other relevant information to identify the root cause of the failure.
24
+
25
+4. Fix the issue: Based on your investigation, apply the necessary fixes to the service or its configuration. This may include restarting the service, adjusting resource allocation, or fixing any configuration errors.
26
+
27
+5. Verify service health: After applying the required fixes, verify the health status of the service once again through the Consul UI or command-line tool. If the service health check status has returned to normal (zero value), the issue has been resolved.
28
+
29
+6. Monitor for any recurrence: Keep an eye on the service, node, and overall Consul cluster health to ensure the issue does not reappear and to catch any other potential problems.
30
+
31
+### Useful resources
32
+
33
+1. [Consul documentation](https://www.consul.io/docs/)
34
+2. [Service and Node Health](https://www.consul.io/api-docs/health)
health/guides/consul_raft_leader_last_contact_time.md
new
+40
@@ -0,0 +1,40 @@
1
+### Understand the alert
2
+
3
+This alert monitors the time since the Consul Raft leader server was last able to contact its follower nodes. If the time since the last contact exceeds the warning or critical thresholds, the alert will be triggered. High values indicate a potential issue with the Consul Raft leader's connection to its follower nodes.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check Consul logs
8
+
9
+Inspect the logs of the Consul leader server and follower nodes for any errors or relevant information. You can find the logs in `/var/log/consul` by default.
10
+
11
+2. Verify Consul agent health
12
+
13
+Ensure that the Consul agents running on the leader and follower nodes are healthy. Use the following command to check the overall health:
14
+
15
+ ```
16
+ consul members
17
+ ```
18
+
19
+3. Review networking connectivity
20
+
21
+Check the network connectivity between the leader and follower nodes. Verify the nodes are reachable, and there are no firewalls or security groups blocking the necessary ports. Consul uses these ports by default:
22
+
23
+ - Server RPC (8300)
24
+ - Serf LAN (8301)
25
+ - Serf WAN (8302)
26
+ - HTTP API (8500)
27
+ - DNS Interface (8600)
28
+
29
+4. Monitor Consul server's resource usage
30
+
31
+Ensure that the Consul server isn't facing any resource constraints, such as high CPU, memory, or disk usage. Use system monitoring tools like `top`, `vmstat`, or `iotop` to observe resource usage and address bottlenecks.
32
+
33
+5. Verify the Consul server configuration
34
+
35
+Examine the Consul server's configuration file (usually located at `/etc/consul/consul.hcl`) and ensure that there are no errors, inconsistencies, or misconfigurations with server addresses, datacenter names, or communication settings.
36
+
37
+### Useful resources
38
+
39
+1. [Consul Docs: Troubleshooting](https://developer.hashicorp.com/consul/tutorials/datacenter-operations/troubleshooting)
40
+2. [Consul Docs: Agent Configuration](https://www.consul.io/docs/agent/options)
health/guides/consul_raft_leadership_transitions.md
new
+54
@@ -0,0 +1,54 @@
1
+### Understand the alert
2
+
3
+This alert triggers when there is a `leadership transition` in the `Consul` service mesh. If you receive this alert, it means that server `${label:node_name}` in datacenter `${label:datacenter}` has become the new leader.
4
+
5
+### What does consul_raft_leadership_transitions mean?
6
+
7
+Consul is a service mesh solution that provides service discovery, configuration, and segmentation functionality. It uses the Raft consensus algorithm to maintain a consistent data state across the cluster. A leadership transition occurs when the current leader node loses its leadership status and a different node takes over.
8
+
9
+### What causes leadership transitions?
10
+
11
+Leadership transitions in Consul can be caused by various reasons, such as:
12
+
13
+1. Network communication issues between the nodes.
14
+2. High resource utilization on the leader node, causing it to miss heartbeat messages.
15
+3. Nodes crashing or being intentionally shut down.
16
+4. A forced leadership transition triggered by an operator.
17
+
18
+Frequent leadership transitions may lead to service disruptions, increased latency, and reduced availability. Therefore, it's essential to identify and resolve the root cause promptly.
19
+
20
+### Troubleshoot the alert
21
+
22
+1. Check the Consul logs for indications of network issues or node failures:
23
+
24
+ ```
25
+ journalctl -u consul.service
26
+ ```
27
+ Alternatively, you can check the Consul log file, which is usually located at `/var/log/consul/consul.log`.
28
+
29
+2. Inspect the health and status of the Consul cluster using the `consul members` command:
30
+
31
+ ```
32
+ consul members
33
+ ```
34
+ This command lists all cluster members and their roles, including the new leader node.
35
+
36
+3. Determine if there's high resource usage on the affected nodes by monitoring CPU, memory, and disk usage:
37
+
38
+ ```
39
+ top
40
+ ```
41
+
42
+4. Examine network connectivity between nodes using tools like `ping`, `traceroute`, or `mtr`.
43
+
44
+5. If the transitions are forced by operators, review the changes made and their impact on the cluster.
45
+
46
+6. Consider increasing the heartbeat timeout configuration to allow the leader more time to respond, especially if high resource usage is causing frequent leadership transitions.
47
+
48
+7. Review Consul's documentation on [consensus and leadership](https://developer.hashicorp.com/consul/docs/architecture/consensus) and [operation and maintenance](https://developer.hashicorp.com/consul/docs/guides) to gain insights into best practices and ways to mitigate leadership transitions.
49
+
50
+### Useful resources
51
+
52
+1. [Consul: Service Mesh Overview](https://www.consul.io/docs/intro)
53
+2. [Consul: Understanding Consensus and Leadership](https://developer.hashicorp.com/consul/docs/architecture/consensus)
54
+3. [Consul: Installation, Configuration, and Maintenance](https://developer.hashicorp.com/consul/docs/guides)
health/guides/consul_raft_thread_fsm_saturation.md
new
+42
@@ -0,0 +1,42 @@
1
+### Understand the alert
2
+
3
+This alert monitors the `consul_raft_thread_fsm_saturation` metric, which represents the saturation of the `FSM Raft` goroutine in Consul, a service mesh. If you receive this alert, it indicates that the Raft goroutine on a specific Consul server is becoming saturated.
4
+
5
+### What is Consul?
6
+
7
+Consul is a distributed service mesh that provides a full-featured control plane with service discovery, configuration, and segmentation functionalities. It enables organizations to build and operate large-scale, dynamic, and resilient systems. The Raft FSM goroutine is responsible for executing finite state machine (FSM) operations on the Consul servers.
8
+
9
+### What does FSM Raft goroutine saturation mean?
10
+
11
+Saturation of the FSM Raft goroutine means that it is spending more time executing operations, which may cause delays in Consul's ability to process requests and manage the overall service mesh. High saturation levels can lead to performance issues, increased latency, or even downtime for your Consul deployment.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Identify the Consul server and datacenter with the high Raft goroutine saturation:
16
+
17
+ The alert has labels `label:node_name` and `label:datacenter`, indicating the affected Consul server and its respective datacenter.
18
+
19
+2. Examine Consul server logs:
20
+
21
+ Check the logs of the affected Consul server for any error messages or indications of high resource usage. This can provide valuable information on the cause of the saturation.
22
+
23
+3. Monitor Consul cluster performance:
24
+
25
+ Use Consul's built-in monitoring tools to keep an eye on your Consul cluster's health and performance. For instance, you may monitor Raft metrics via the Consul `/v1/agent/metrics` API endpoint.
26
+
27
+4. Scale your Consul infrastructure:
28
+
29
+ If the increased saturation is due to high demand, scaling your Consul infrastructure by adding more servers or increasing the resources available to existing servers can help mitigate the issue.
30
+
31
+5. Review and optimize Consul configuration:
32
+
33
+ Review your Consul configuration and make any necessary optimizations to ensure the best performance. For instance, you could adjust the [Raft read and write timeouts](https://www.consul.io/docs/agent/options).
34
+
35
+6. Investigate and resolve any underlying issues causing the saturation:
36
+
37
+ Look for any factors contributing to the increased load on the FSM Raft goroutine and address those issues. This may involve reviewing application workloads, network latency, or hardware limitations.
38
+
39
+### Useful resources
40
+
41
+1. [Consul Telemetry](https://www.consul.io/docs/agent/telemetry)
42
+2. [Consul Configuration - Raft](https://www.consul.io/docs/agent/options#raft)
health/guides/consul_raft_thread_main_saturation.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert triggers when the main Raft goroutine's saturation percentage reaches a certain threshold. If you receive this alert, it means that your Consul server is experiencing high utilization of the main Raft goroutine.
4
+
5
+### What is Consul?
6
+
7
+Consul is a service discovery, configuration, and orchestration solution developed by HashiCorp. It is used in microservice architectures and distributed systems to make services aware and discoverable by other services. Raft is a consensus-based algorithm used for maintaining the state of the Consul servers.
8
+
9
+### What is the main Raft goroutine?
10
+
11
+The main Raft goroutine is responsible for carrying out consensus-related tasks in the Consul server. It ensures the consistency and reliability of the server's state. High saturation of this goroutine can lead to performance issues in the Consul server cluster.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Verify the current status of the Consul server.
16
+ Check the health status and logs of the Consul server using the following command:
17
+ ```
18
+ consul monitor
19
+ ```
20
+
21
+2. Monitor Raft metrics.
22
+ Use the Consul telemetry feature to collect and analyze Raft performance metrics. Consult the [Consul official documentation](https://www.consul.io/docs/agent/telemetry) on setting up telemetry.
23
+
24
+3. Review the server's resources.
25
+ Confirm whether the server hosting the Consul service has enough resources (CPU, memory, and disk space) to handle the current load. Upgrade the server resources or adjust the Consul configurations accordingly.
26
+
27
+4. Inspect the Consul server's log files.
28
+ Analyze the log files to identify any errors or issues that could be affecting the performance of the main Raft goroutine.
29
+
30
+5. Monitor network latency between Consul servers.
31
+ High network latency can affect the performance of the Raft algorithm. Use monitoring tools like `ping` or `traceroute` to measure the latency between the Consul servers.
32
+
33
+6. Check for disruptions in the Consul cluster.
34
+ Investigate possible disruptions caused by external factors, such as server failures, network partitioning or misconfigurations in the cluster.
35
+
36
+### Useful resources
37
+
38
+1. [Consul: Service Mesh for Microservices Networking](https://www.consul.io/)
39
+2. [Consul Documentation](https://www.consul.io/docs)
40
+3. [Consul Telemetry](https://www.consul.io/docs/agent/telemetry)
41
+4. [Understanding Raft Consensus Algorithm](https://raft.github.io/)
health/guides/consul_service_health_check_status.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the `health check status` of a service in a `Consul` service mesh changes to a `warning` or `critical` state. It occurs when a service health check for a specific service `${label:service_name}` fails on a server `${label:node_name}` in a datacenter `${label:datacenter}`.
4
+
5
+### What is Consul?
6
+
7
+`Consul` is a service mesh solution developed by HashiCorp that can be used to connect and secure services across dynamic, distributed infrastructure. It maintains a registry of service instances, performs health checks, and offers a flexible and high-performance service discovery mechanism.
8
+
9
+### What is a service health check?
10
+
11
+A service health check is a way to determine whether a particular service in a distributed system is running correctly, reachable, and responsive. It is an essential component of service discovery and can be used to assess the overall health of a distributed system.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the health status of the service that triggered the alert in the Consul UI.
16
+
17
+ Access the Consul UI and navigate to the affected service's details page. Look for the health status information and the specific health check that caused the alert.
18
+
19
+2. Inspect the logs of the service that failed the health check.
20
+
21
+ Access the logs of the affected service and look for any error messages or events that might have caused the health check to fail. Depending on the service, this might be application logs, system logs, or container logs (if the service is running in a container).
22
+
23
+3. Identify and fix the issue causing the health check failure.
24
+
25
+ Based on the information from the logs and your knowledge of the system, address the issue that's causing the health check to fail. This might involve fixing a bug in the service, resolving a connection issue, or making a configuration change.
26
+
27
+4. Verify that the health check status has returned to a healthy state.
28
+
29
+ After addressing the issue, monitor the service in the Consul UI and confirm that its health check status has returned to a healthy state. If the issue persists, continue investigating and resolving any underlying causes until the health check is successful.
30
+
31
+### Useful resources
32
+
33
+1. [Consul Introduction](https://www.consul.io/intro)
34
+2. [Consul Health Check Documentation](https://www.consul.io/docs/discovery/checks)
35
+3. [HashiCorp Learn: Consul Service Monitoring](https://learn.hashicorp.com/tutorials/consul/service-monitoring-and-alerting?in=consul/developer-discovery)
\ No newline at end of file
health/guides/cpu/10min_cpu_iowait.md
deleted
-72
@@ -1,72 +0,0 @@
1
-# 10min_cpu_iowait
2
-
3
-## OS: Linux
4
-
5
-This alarm calculates the average time of `iowait` through 10 minute interval periods. `iowait` is the percentage of
6
-time where there has been at least one I/O request in progress while the CPU has been idle.
7
-
8
-We call I/O -on a process level- the use of the read and write services, such as reading data from a physical
9
-drive.
10
-
11
-It's important to note that during the time a process waits on I/O, the system can schedule other processes, but
12
-`iowait` is measured specifically while the CPU is idle.
13
-
14
-A common example of when this alert might be triggered would be when your CPU requests some data and the device
15
-responsible for it can't deliver it fast enough. As a result the CPU (in the next clock interrupt) is idle, so you
16
-encounter `iowait`. If this persists for some time and the average from the metrics we gather exceeds the value that is
17
-being checked in the `.conf` file, then the alert is raised because the CPU is being bottlenecked by your system’s
18
-disks. <sup> [1](https://blog.pregos.info/wp-content/uploads/2010/09/iowait.txt) [2](
19
-https://serverfault.com/questions/12679/can-anyone-explain-precisely-what-iowait-is) </sup>
20
-
21
-<details>
22
-<summary>References and Sources</summary>
23
-
24
-[[1] What exactly is "iowait"?](https://blog.pregos.info/wp-content/uploads/2010/09/iowait.txt)
25
-[[2] Serverfault](https://serverfault.com/questions/12679/can-anyone-explain-precisely-what-iowait-is)
26
-</details>
27
-
28
-### Troubleshooting Section
29
-
30
-<details>
31
-<summary>Check for main I/O related processes and hardware issues</summary>
32
-
33
-- Generally, this issue is caused by having slow hard drives that cannot keep up with the speed of your CPU. You can see
34
- the percentage of `iowait` by going to your node on Netdata Cloud and clicking the `iowait` dimension under the Total
35
- CPU Utilization chart.
36
-
37
-
38
-- You can use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds)
39
-
40
-```
41
-root@netdata~ # vmstat
42
-procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
43
- r b swpd free buff cache si so bi bo in cs us sy id wa st
44
- 8 0 1200384 168456 48840 1461540 4 14 65 51 334 196 3 1 95 0 0
45
-```
46
-
47
-The `procs` column, shows; \
48
-b: The number of processes blocked waiting for I/O to complete.
49
-
50
-After that, you can use `ps` and specifically `ps -eo s,user,cmd | grep ^[D]`.
51
-
52
-- The `grep` command will fetch the processes that their state code starts with `D` which means uninterruptible sleep
53
- (usually IO).
54
-
55
-> It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
56
-processes you are closing and being certain that they are not necessary.
57
-
58
-- If you see that you don't have a lot of processes that you can terminate (or you need them for your workflow), then
59
- you would have to upgrade your system’s drives; if you have an HDD, upgrading to an SSD or an NVME drive would make a
60
- great impact on this metric.
61
-
62
-</details>
63
-
64
-<details>
65
-<summary>Check your database</summary>
66
-
67
-- As another example, in a database environment, you would want to optimize your operations. Check for potential inserts
68
- on large data sets, keeping in mind that `write` operations take more time than `read`. You should also search for
69
- complex requests, like large joins and queries over a big data set. These can introduce `iowait` and need to be
70
- optimized.
71
-
72
-</details>
health/guides/cpu/10min_cpu_usage.md
deleted
-60
@@ -1,60 +0,0 @@
1
-# 10min_cpu_usage
2
-
3
-## OS: Linux & FreeBSD
4
-
5
-This alarm calculates an average on CPU utilization over a period of 10 minutes, **excluding** `iowait`, `nice`
6
-and `steal` values.
7
-
8
-> Note that on FreeBSD, the alert excludes only `nice`.
9
-
10
-The alarm is generally self-explanatory, but to clarify any confusion that might exist in the `iowait`, `nice`
11
-and `steal` metrics:
12
-
13
-`iowait` is the percentage of time the CPU waits on a disk for an I/O; it happens when the former is getting
14
-bottlenecked by the latter. At this point the CPU is being idle, waiting only on the I/O.
15
-
16
-`nice` value of a processor is the time it has spent on running low priority processes. Low priority processes are those
17
-with a 'nice' value greater than 0 (on UNIX-like systems, a higher ‘nice’ value indicates a lower priority).
18
-
19
-`steal`, in a virtual machine, is the percentage of time that particular virtual CPU has to wait for an available host
20
-CPU to run on. If this metric goes up, it means that your VM is not getting the processing power it needs.
21
-
22
-### Troubleshooting
23
-
24
-<details>
25
-<summary>Processes slowing down your CPU</summary>
26
-
27
-
28
-There are two primary cases in which this alarm is raised, and determining which applies to you requires
29
-understanding your own scenario.
30
-
31
-- Generally, if you have high CPU utilization alongside a high `nice` value, it means that the system is running through
32
- all the low priority processes, and if some high priority process needs CPU time, it can get it at any time.
33
-- On the flipside, if you have high CPU utilization with low `nice` value, this means that the CPU is used on high
34
- priority processes and new ones will not be able to take CPU time, and they will have to wait.
35
-
36
-The latter scenario is worth investigating if there is a process slowing down your CPU. We suggest you go to your node
37
-on Netdata Cloud and click the `nice` dimension under the `Total CPU Utilization` chart to see the value. You can then
38
-check per process CPU usage using `top`:
39
-
40
-<details>
41
-<summary>Linux</summary>
42
-
43
-```
44
-root@netdata~ # top -o +%CPU -i
45
-```
46
-</details>
47
-<details>
48
-<summary>FreeBSD</summary>
49
-
50
-```
51
-root@netdata~ # top -o cpu -I
52
-```
53
-</details>
54
-
55
-Here, you can see which processes are the main cpu consumers on the `CPU` column.
56
-
57
-
58
-It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
59
-processes you are closing and being certain that they are not necessary.
60
-</details>
health/guides/disk_inode_usage.md
new
+23
@@ -0,0 +1,23 @@
1
+### Understand the alert
2
+
3
+This alarm presents the percentage of used `inodes` storage of a particular disk.
4
+
5
+The number of `inodes` indicates the number of files and folders you have. An `inode` is a data structure, containing metadata about a file. All filenames are internally mapped to respective `inode` numbers, so if you have a
6
+lot of files, it means there are a lot of `inodes`.
7
+
8
+If the alarm is raised, it means that your storage device is running out of `inode` space. Each disk has a particular **limitation on the amount of `inodes` it can store**, determined by its size.
9
+
10
+Many modern filesystems use dynamically allocated `inodes` instead of a static table. These should not be presented on the charts associated with this alarm, and should not ever trigger it. If such a filesystem **does** trigger this alarm, and it's constantly reporting max `inode` usage, it's probably a bug in the filesystem driver. Some such filesystems incorrectly report having max `inode` count when they should not because they have no max limit, and in turn they trigger a false positive alarm.
11
+
12
+### Troubleshoot the alert
13
+
14
+Clear cache files or delete unnecessary files and folders
15
+
16
+- To reduce the amount of how many `inodes` you store currently, you can clear your cache, trash any unnecessary files and folders in your system.
17
+
18
+We strongly suggest that you practice a high degree of caution when cleaning up drives, and removing files, make sure that you are certain that you delete only unnecessary files.
19
+
20
+### Useful resources
21
+
22
+[Linux Inodes](https://www.javatpoint.com/linux-inodes)
23
+[Understanding UNIX / Linux filesystem Inodes](https://www.cyberciti.biz/tips/understanding-unixlinux-filesystem-inodes.html)
\ No newline at end of file
health/guides/disk_space_usage.md
new
+19
@@ -0,0 +1,19 @@
1
+### Understand the alert
2
+
3
+This alarm presents the percentage of used space of a particular disk. If it is close to 100%, it means that your storage device is running out of space. If the particular disk raising the alarm is full, the system could experience slowdowns and even crashes.
4
+
5
+### Troubleshoot the alert
6
+
7
+Clean or upgrade the drive.
8
+
9
+If your storage device is full and the alert is raised, there are two paths you can tend to:
10
+
11
+- Cleanup your drive, remove any unnecessary files (files on the trash directory, cache files etc.) to free up space. Some areas that are safe to delete, are:
12
+ - Files under `/var/cache`
13
+ - Old logs in `/var/log`
14
+ - Old crash reports in `/var/crash` or `/var/dump`
15
+ - The `.cache` directory in user home directories
16
+
17
+- If your workflow requires all the space that is currently used, then you might want to look into upgrading the disk that raised the alarm, because its capacity is small for your demands.
18
+
19
+Netdata strongly suggests that you are careful when cleaning up drives, and removing files, make sure that you are certain that you delete only unnecessary files.
\ No newline at end of file
health/guides/disks/10min_disk_backlog.md
deleted
-16
@@ -1,16 +0,0 @@
1
-# 10min_disk_backlog
2
-
3
-## OS: Linux
4
-
5
-This alert presents the average backlog size of the disk raising this alarm over the last 10 minutes.
6
-
7
-- This alert is escalated to warning when the metric exceeds the size of 5000.
8
-
9
-<details>
10
-<summary>What is "disk backlog"?</summary>
11
-
12
-> Backlog is an indication of the duration of pending disk operations. On every I/O event the
13
-> system is multiplying the time spent doing I/O since the last update of this field with the
14
-> number of pending operations. While not accurate, this metric can provide an indication of the
15
-> expected completion time of the operations in progress.
16
-</details>
health/guides/disks/10min_disk_utilization.md
deleted
-57
@@ -1,57 +0,0 @@
1
-# 10min_disk_utilization
2
-
3
-## OS: Linux, FreeBSD
4
-
5
-This alert presents the average percentage of time the disk was busy over the
6
-last 10 minutes.
7
-If you receive this it indicates high disk load and that the disk spent most of the time servicing
8
-read or write requests.
9
-
10
-- This alert is triggered in a warning state when the metric exceeds 98%.
11
-
12
-This metric is the same as the %util column on the command `iostat -x`:
13
-
14
-> %util is the percentage of the time the drive was doing at least one thing.
15
-> Device saturation occurs when this value is close to 100% for devices serving requests serially.
16
-> But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number
17
-> does not reflect their performance limits.
18
-> As a measure of general IO busyness %util is fairly handy, but as an indication of how much the
19
-> system is doing compared to what it can do, it's terrible.<sup>[1](
20
-> https://brooker.co.za/blog/2014/07/04/iostat-pct.html) </sup>
21
-
22
-
23
-
24
-<details>
25
-<summary>References and Sources</summary>
26
-
27
-1. [Two traps in iostat: %util and svctm](https://brooker.co.za/blog/2014/07/04/iostat-pct.html)
28
-
29
-</details>
30
-
31
-### Troubleshooting Section
32
-
33
-#### Check per-process disk usage to find the top consumers
34
-
35
-> Note: If you got this alert for a device serving requests in parallel, you can ignore it.
36
-
37
-<details><summary>Use `iotop` on Linux</summary>
38
-
39
- `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it,
40
- then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
41
- ```
42
- root@netdata~ # sudo iotop
43
- ```
44
- Using this, you can see which processes are the main Disk I/O consumers on the `IO` column.
45
-
46
-</details>
47
-
48
-<details><summary>Use `top` on FreeBSD</summary>
49
-
50
-You can use `top`:
51
- ```
52
- root@netdata~ # top -m io -o total
53
- ```
54
- The `-m io` sets `top` to display I/O statistics, and the `-o total` indicates the results will be
55
- ordered according to the field "Total".
56
-
57
-</details>
health/guides/disks/disk_inode_usage.md
deleted
-35
@@ -1,35 +0,0 @@
1
-# disk_inode_usage
2
-
3
-## OS: Linux, FreeBSD
4
-
5
-This alarm presents the percentage of used `inodes` storage of a particular disk.
6
-
7
-> The number of `inodes` indicates the number of files and folders you have. An `inode` is a data structure,
8
-> containing metadata about a file. All filenames are internally mapped to respective `inode` numbers, so if you have a
9
-> lot of files, it means there are a lot of `inodes`.
10
-
11
-If the alarm is raised, it means that your storage device is running out of `inode` space. Each disk has a particular
12
-**limitation on the amount of `inodes` it can store**, determined by its size.
13
-
14
-> Many modern filesystems use dynamically allocated `inodes` instead of a static table. These should not be presented
15
-> on the charts associated with this alarm, and should not ever trigger it. If such a filesystem ***does***
16
-> trigger this alarm, and it's constantly reporting max `inode` usage, it's probably a bug in the filesystem driver. \
17
-> Some such filesystems incorrectly report having max `inode` count when they should not because they have no max
18
-> limit, and in turn they trigger a false positive alarm.
19
-
20
-Links: \
21
-[Linux Inodes](https://www.javatpoint.com/linux-inodes) \
22
-[Understanding UNIX / Linux filesystem Inodes](
23
-https://www.cyberciti.biz/tips/understanding-unixlinux-filesystem-inodes.html)
24
-
25
-### Troubleshoot
26
-
27
-<details>
28
-<summary>Clear cache files or delete unnecessary files and folders</summary>
29
-
30
-- To reduce the amount of how many `inodes` you store currently, you can clear your cache, trash any unnecessary files
31
- and folders in your system.
32
-
33
-> We strongly suggest that you practice a high degree of caution when cleaning up drives, and removing files, make sure
34
-> that you are certain that you delete only unnecessary files.
35
-</details>
\ No newline at end of file
health/guides/disks/disk_space_usage.md
deleted
-30
@@ -1,30 +0,0 @@
1
-# disk_space_usage
2
-
3
-## OS: Linux, FreeBSD
4
-
5
-This alarm presents the percentage of used space of a particular disk. If it is close to 100%, it means that your storage
6
-device is running out of space. If the particular disk raising the alarm is full, the system could experience slowdowns
7
-and even crashes.
8
-
9
-### Troubleshoot
10
-
11
-<details>
12
-<summary>Clean or upgrade the drive</summary>
13
-
14
-If your storage device is full and the alert is raised, there are two paths you can tend to:
15
-
16
-- Cleanup your drive, remove any unnecessary files (files on the trash directory, cache files etc.) to free up space. \
17
-Some areas that are safe to delete, are:
18
- - Files under `/var/cache`
19
- - Old logs in `/var/log`
20
- - Old crash reports in `/var/crash` or `/var/dump`
21
- - The `.cache` directory in user home directories
22
-
23
-
24
-- If your workflow requires all the space that is currently used, then you might want to look into upgrading the disk
25
- that raised the alarm, because its capacity is small for your demands.
26
-
27
-> Netdata strongly suggests that you are careful when cleaning up drives, and removing files, make sure
28
-> that you are certain that you delete only unnecessary files.
29
-
30
-</details>
health/guides/dns_query/dns_query_time_query_time.md
deleted
-89
@@ -1,89 +0,0 @@
1
-# dns_query_time_query_time
2
-
3
-## DNS
4
-
5
-This alert presents the average DNS query round trip time (RTT) over the last 10 seconds.
6
-
7
-If the DNS query exceeds a time limit to complete its operation (aka if it times out), then the
8
-alert is raised into warning.
9
-
10
-<details><summary>What is Round Trip Time?</summary>
11
-
12
-> In networking, round-trip time (RTT), also known as round-trip delay time (RTD) is defined as
13
-> a metric that measures in milliseconds the amount of time it takes for a data packet to be
14
-> sent plus the amount of time it takes for acknowledgement of that signal to be received. This
15
-> time delay includes propagation times for the paths between the two communication endpoints.
16
-> <sup>[1](https://www.stormit.cloud/post/what-is-round-trip-time-rtt-meaning-calculation)
17
-
18
-</details>
19
-
20
-<details><summary>What is the main cause of DNS Latency?</summary>
21
-
22
-- Cache misses
23
- Even if a resolver can provide very good cache hit latency, cache misses are unavoidable and are
24
- very costly in terms of latency.
25
-
26
- "Cache hits" is the terminology used for when a system asks a resolver about some data and the
27
- resolver can provide it because he has it cached locally.
28
-
29
- "Cache misses" occur when a system asks a resolver about some data, and he doesn't have it cached
30
- locally. Then the resolver has to talk to other name servers, to see if they have the data
31
- requested, which takes time and greatly increases latency.<sup>[2](
32
- https://developers.google.com/speed/public-dns/docs/performance#introduction_causes_and_mitigations_of_dns_latency) </sup>
33
-
34
-
35
-- DNS Server Location
36
- > The location of the DNS server you're accessing plays a huge role in your latency. The
37
- > farther the server is to your place, the higher the latency gets. But this is not always
38
- > the case as centralized DNS servers' latency isn't affected by the distance from the user.
39
- > Transit links also vary from one server to another. Latency will be lower if the transit
40
- > links are equipped with up-to-date technology.<sup>[3](
41
- https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx) </sup>
42
-
43
-
44
-- Wireless networks
45
- > Wireless networks have higher latency compared to wired networks. This happens because the
46
- > transfer of data doesn't go through fixed lines. Instead, it goes through Wi-Fi routers or
47
- > satellite dishes. These devices' efficiency also depends on the location where they're placed.<sup>[3](
48
- https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx) </sup>
49
-
50
-
51
-- Malicious DNS Traffic
52
- > Malicious DNS traffic can also cause high latency. It's because the DNS server will work
53
- > double time in processing it. PRSD attacks are the most common type of malicious traffic.
54
- > When this happens, it causes a lot of malware and botnet queries which cause high recursion
55
- > rates. These consume and waste a lot of CPU cycles on the server.<sup>[3](
56
- https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx) </sup>
57
-
58
-
59
-- Under-scaling of DNS Server
60
- > Proper scaling of the DNS infrastructure is important. Because if it’s not scaled correctly,
61
- > chances are is that it will use too much CPU power. When this happens, it will impact the
62
- > latency and cause it to increase. The more the CPU is utilized, the higher your latency gets.<sup>[3](
63
- https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx) </sup>
64
-
65
-</details>
66
-
67
-For further information, please refer to our *References and Sources* section.
68
-
69
-<details><summary>References and Sources</summary>
70
-
71
-1. [What is Round-Trip Time (RTT)?](
72
- https://www.stormit.cloud/post/what-is-round-trip-time-rtt-meaning-calculation)
73
-2. [Causes and mitigations of DNS latency](
74
- https://developers.google.com/speed/public-dns/docs/performance#introduction_causes_and_mitigations_of_dns_latency)
75
-3. [What is DNS Latency and Why Should You Care?](
76
- https://www.smartdnsproxy.com/news/smart-dns-proxy/what-is-dns-latency-and-why-should-you-care-158.aspx)
77
-4. [Configure your network settings to use Google Public DNS](https://developers.google.com/speed/public-dns/docs/using)
78
-
79
-</details>
80
-
81
-### Troubleshooting Section
82
-
83
-This alert can have multiple causes.
84
-
85
-As a first step, you can try changing your DNS server. Your current configuration might be using a
86
-slow server, or what your ISP provides might not be the best. You can find more information on
87
-[Google Developers](https://developers.google.com/speed/public-dns/docs/using) on how to configure
88
-your settings for your specific OS to use Google's Public DNS.
89
-
health/guides/dns_query_query_status.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the success rate of DNS requests of a specific type to a specified server starts to fail. The alert checks the DNS `query_status` and warns if the success rate is not `1`, indicating unsuccessful DNS queries.
4
+
5
+### What is a DNS query?
6
+
7
+A DNS query is a request for information from a client machine to a DNS server, typically to resolve domain names (such as www.example.com) to IP addresses. A successful query will return the matching IP address, while an unsuccessful query may result from various issues, such as DNS server problems or network connectivity issues.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the DNS server status and logs
12
+
13
+ Verify if the DNS server (mentioned in the alert `${label:server}`) is up and running. Inspect the server logs for any error messages or suspicious activity.
14
+
15
+2. Examine network connectivity
16
+
17
+ Make sure that your system can communicate with the specified DNS server. Use standard network troubleshooting tools, such as `traceroute`, to identify possible network issues between the client machine and the DNS server.
18
+
19
+3. Inspect the DNS query type
20
+
21
+ This alert is specific to the DNS request type `${label:record_type}`. Check if this particular type of request is causing the issue, or if the problem is widespread across all DNS queries. Understanding the scope of the issue can help narrow down the possible causes.
22
+
23
+4. Analyze local DNS resolver configuration
24
+
25
+ Examine your system's `/etc/resolv.conf` file and make sure that the specified DNS server is configured correctly. Review any recent changes in the resolver configuration.
26
+
27
+5. Monitor success rate improvements
28
+
29
+ After resolving the issue, keep an eye on the alert to ensure that the success rate returns to `1`, indicating successful DNS requests.
30
+
31
+### Useful resources
32
+
33
+1. [DNS Query Types](https://www.cloudflare.com/learning/dns/dns-records/)
health/guides/dnsmasq_dhcp_dhcp_range_utilization.md
new
+50
@@ -0,0 +1,50 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the number of leased IP addresses in your DHCP range, managed by dnsmasq, is close to the total number of provisioned DHCP addresses. The alert will be triggered in a warning state when the percentage of leased IP addresses is between 80-90% and in a critical state when it is between 90-95%.
4
+
5
+### What is DHCP?
6
+
7
+Dynamic Host Configuration Protocol (DHCP) is a network management protocol that dynamically assigns IP addresses and other configuration information to devices connected to the network. It helps network administrators to manage the IP address allocation process efficiently.
8
+
9
+### What is dnsmasq?
10
+
11
+`dnsmasq` is a lightweight, easy to configure DNS forwarder, DHCP server, and TFTP server. It is designed to provide DNS and optionally, DHCP, services to a small-scale network. Dnsmasq can serve the names of local machines which are not in the global DNS.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the current DHCP lease utilization
16
+
17
+To see the current percentage of DHCP leases in use, run the following command:
18
+
19
+```
20
+cat /var/lib/misc/dnsmasq.leases | wc -l
21
+```
22
+
23
+2. Verify the configured DHCP range
24
+
25
+Check the `/etc/dnsmasq.conf` file to ensure that the DHCP range is configured correctly:
26
+
27
+```
28
+grep -i "dhcp-range" /etc/dnsmasq.conf
29
+```
30
+
31
+Make sure that the range provides enough IP addresses for the number of devices in your network.
32
+
33
+3. Increase the DHCP range
34
+
35
+If required, increase the number of available IP addresses within the DHCP range by modifying the `/etc/dnsmasq.conf` file, expanding the range and/or decreasing the lease time.
36
+
37
+After modifying the configuration, restart the dnsmasq service to apply the changes:
38
+
39
+```
40
+sudo systemctl restart dnsmasq
41
+```
42
+
43
+4. Monitor the DHCP lease utilization
44
+
45
+Keep monitoring the DHCP lease utilization to ensure that the new range and lease settings are sufficient for your network's needs.
46
+
47
+### Useful resources
48
+
49
+1. [The Dnsmasq Homepage](http://www.thekelleys.org.uk/dnsmasq/doc.html)
50
+2. [Ubuntu Community Help Wiki: Dnsmasq](https://help.ubuntu.com/community/Dnsmasq)
health/guides/docker/docker_container_unhealthy.md
deleted
-61
@@ -1,61 +0,0 @@
1
-# docker_containers_unhealthy
2
-
3
-**Containers | Docker**
4
-
5
-_Docker is an open source containerization platform. It enables developers to package applications
6
-into containers—standardized executable components combining application source code with the
7
-operating system (OS) libraries and dependencies required to run that code in any environment_
8
-
9
-Sometimes while our container is running, the application inside may have crashed. To foresee those
10
-events, container runtimes (CR) and orchestrators perform health checks to endpoints inside the
11
-functional units of the container. A container marked as unhealthy by the CR, is malfunctioning and
12
-should be stopped. Those health checks are defined by the creator of the container with the
13
-HEALTHCHECK
14
-instructions. <sup>[1](https://docs.docker.com/engine/reference/builder/#healthcheck) </sup>
15
-
16
-The Netdata Agent monitors the average number of unhealthy docker containers over the last 10
17
-seconds. This alert indicates that some containers are not running due to failed health checks.
18
-
19
-This alert is raised into warning when at least one container is unhealthy in your Docker engine.
20
-
21
-<details>
22
-<summary>References and sources</summary>
23
-
24
-1. [HEALTHCHECK instruction in Docker docs](https://docs.docker.com/engine/reference/builder/#healthcheck)
25
-
26
-</details>
27
-
28
-### Troubleshooting section
29
-
30
-<details>
31
-<summary>Inspect and restart the UNHEALTY container</summary>
32
-
33
-1. Check all the containers in the system.
34
-
35
- ```
36
- root@netdata # docker ps -a
37
- ```
38
-
39
-2. Find the NAME of the container that is marked as UNHEALTHY.
40
-
41
-3. Check the logs of this container to get some insights into what's going wrong
42
-
43
- ```
44
- root@netdata # docker logs <UNHEALTHY_CONTAINER>
45
- ```
46
- In many cases, your app's logs may not appear in docker log collector. A simple workaround is
47
- something like
48
- this, [redirect your apps's logs into stderr](https://github.com/nginxinc/docker-nginx/blob/master/Dockerfile-debian.template#L90)
49
- . Use this workaround purposefully. Another workaround is to redirect any log attempt to log
50
- directly into the `/proc/self/fd/2`.
51
-
52
-
53
-4. Restart the container and see if this fixes the problem.
54
-
55
- ```
56
- root@netdata # docker logs <UNHEALTHY_CONTAINER>
57
- ```
58
-
59
-5. If you receive this alert often, you may have to do further investigation on why this event occurs
60
-
61
-</details>
health/guides/docker_container_unhealthy.md
new
+49
@@ -0,0 +1,49 @@
1
+### Understand the alert
2
+
3
+This alert, `docker_container_unhealthy`, is triggered when the health status of a Docker container is marked as unhealthy. If you receive this alert, it means that one of your Docker containers is not functioning properly, which can affect the services or applications running inside the container.
4
+
5
+### What does container health status mean?
6
+
7
+The container health status is a Docker feature that allows you to define custom health checks to verify the proper functioning of your containers. If a container has a health check defined, Docker will execute it at regular intervals to monitor the container's health. If the health check fails a specific number of times in a row, Docker will mark the container as unhealthy, and this alert will be triggered.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the affected container:
12
+
13
+ Find the container name in the alert's info field: `${label:container_name} docker container health status is unhealthy`. Use this container name in the following steps.
14
+
15
+2. Check the logs of the affected container:
16
+
17
+ Use the `docker logs` command to view the logs of the unhealthy container. This may provide information on what caused the container to become unhealthy.
18
+
19
+ ```
20
+ docker logs <container_name>
21
+ ```
22
+
23
+3. Inspect the container's health check configuration:
24
+
25
+ Use the `docker inspect` command to view the health check settings for the affected container. Look for any misconfigurations that could lead to the container being marked as unhealthy.
26
+
27
+ ```
28
+ docker inspect <container_name> --format='{{json .Config.Healthcheck}}'
29
+ ```
30
+
31
+4. Check the container's health status history:
32
+
33
+ Use the `docker inspect` command again to review the health check history for the affected container.
34
+
35
+ ```
36
+ docker inspect <container_name> --format='{{json .State.Health}}'
37
+ ```
38
+
39
+5. Investigate and fix container issues:
40
+
41
+ Based on the information gathered from the previous steps, investigate and fix any issues with the container's service, configuration, or resources. You might need to restart the container or reconfigure its health check settings.
42
+
43
+ ```
44
+ docker restart <container_name>
45
+ ```
46
+
47
+### Useful resources
48
+
49
+1. [Docker's HEALTHCHECK instruction](https://stackoverflow.com/questions/38546755/how-to-use-dockers-healthcheck-instruction)
health/guides/elasticsearch_cluster_health_status_red.md
new
+55
@@ -0,0 +1,55 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the Elasticsearch cluster health status turns `RED`. If you receive this alert, it means that there is a problem that needs immediate attention, such as data loss or one or more primary and replica shards are not allocated to the cluster.
4
+
5
+### Elasticsearch Cluster Health Status
6
+
7
+Elasticsearch cluster health status provides an indication of the cluster's overall health, based on the state of its shards. The status can be `green`, `yellow`, or `red`:
8
+
9
+- `Green`: All primary and replica shards are allocated.
10
+- `Yellow`: All primary shards are allocated, but some replica shards are not.
11
+- `Red`: One or more primary shards are not allocated, leading to data loss.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the Elasticsearch cluster health using the `_cat` API:
16
+
17
+```
18
+curl -XGET 'http://localhost:9200/_cat/health?v'
19
+```
20
+
21
+Examine the output to understand the current health status, the number of nodes and shards, and any unassigned shards.
22
+
23
+2. To get more details on the unassigned shards, use the `_cat/shards` API:
24
+
25
+```
26
+curl -XGET 'http://localhost:9200/_cat/shards?v'
27
+```
28
+
29
+Look for shards with the status `UNASSIGNED`.
30
+
31
+3. Identify the root cause of the issue, such as:
32
+
33
+ - A node has left the cluster or failed, causing the primary shard to become unassigned.
34
+ - Insufficient disk space is available, preventing shards from being allocated.
35
+ - Cluster settings or shard allocation settings are misconfigured.
36
+
37
+4. Take appropriate action based on the root cause:
38
+
39
+ - Ensure all Elasticsearch nodes are running and connected to the cluster.
40
+ - Add more nodes or increase disk space as needed.
41
+ - Review and correct cluster and shard allocation settings.
42
+
43
+5. Monitor the health status as the cluster recovers:
44
+
45
+```
46
+curl -XGET 'http://localhost:9200/_cat/health?v'
47
+```
48
+
49
+If the health status turns `YELLOW` or `GREEN`, the cluster is no longer in the `RED` state.
50
+
51
+### Useful resources
52
+
53
+1. [Elasticsearch Cluster Health](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)
54
+2. [Fixing Elasticsearch Cluster Health Status "RED"](https://www.elastic.co/guide/en/elasticsearch/guide/current/_cluster_health.html)
55
+3. [Elasticsearch Shard Allocation](https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html)
\ No newline at end of file
health/guides/elasticsearch_cluster_health_status_yellow.md
new
+57
@@ -0,0 +1,57 @@
1
+### Understand the alert
2
+
3
+The `elasticsearch_cluster_health_status_yellow` alert triggers when the Elasticsearch cluster's health status is `yellow` for longer than 10 minutes. This may indicate potential issues in the cluster, like unassigned or missing replicas. The alert class is `Errors`, and the type is `SearchEngine`.
4
+
5
+### What does the health status mean?
6
+
7
+In Elasticsearch, cluster health status can be one of three colors:
8
+
9
+- Green: All primary shards and replicas are active and properly assigned to each index.
10
+- Yellow: All primary shards are active, but one or more replicas are unassigned or missing.
11
+- Red: One or more primary shards are unassigned or missing.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the Elasticsearch cluster health.
16
+
17
+You can check the health of the Elasticsearch cluster using the `/_cluster/health` API endpoint:
18
+
19
+```
20
+curl -XGET 'http://localhost:9200/_cluster/health?pretty'
21
+```
22
+
23
+2. Identify the unassigned or missing replicas.
24
+
25
+You can check for any unassigned or missing shards using the `/_cat/shards` API endpoint:
26
+
27
+```
28
+curl -XGET 'http://localhost:9200/_cat/shards?v&h=index,shard,prirep,state'
29
+```
30
+
31
+3. Check Elasticsearch logs for any errors or warnings:
32
+
33
+```
34
+sudo journalctl --unit elasticsearch
35
+```
36
+
37
+4. Check disk space on all Elasticsearch nodes. Insufficient disk space may lead to unassigned or missing replicas:
38
+
39
+```
40
+df -h
41
+```
42
+
43
+5. Ensure Elasticsearch is properly configured.
44
+
45
+Check the `elasticsearch.yml` configuration file on all nodes for any misconfigurations or errors:
46
+
47
+```
48
+sudo nano /etc/elasticsearch/elasticsearch.yml
49
+```
50
+
51
+6. Review the Elasticsearch documentation on [Cluster-Level Shard Allocation and Routing Settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html) to understand how to properly assign and balance shards.
52
+
53
+### Useful resources
54
+
55
+1. [Elasticsearch Cluster Health](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)
56
+2. [Elasticsearch Shards](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html)
57
+3. [Allocation Awareness in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html)
\ No newline at end of file
health/guides/elasticsearch_node_index_health_red.md
new
+49
@@ -0,0 +1,49 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the health status of an Elasticsearch node index turns `red`. If you receive this alert, it means that at least one primary shard and its replicas are not allocated to any node, and the data in the index is potentially at risk.
4
+
5
+### What does a red index health status mean?
6
+
7
+In Elasticsearch, the index health status can be green, yellow, or red:
8
+
9
+- Green: All primary and replica shards are allocated and active.
10
+- Yellow: All primary shards are active, but not all replicas are allocated due to the lack of available nodes.
11
+- Red: At least one primary shard and its replicas are not allocated, which means the cluster can't serve all the incoming data, and data loss is possible.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the cluster health
16
+
17
+ Use the Elasticsearch `_cluster/health` endpoint to check the health status of your cluster:
18
+ ```
19
+ curl -X GET "localhost:9200/_cluster/health?pretty"
20
+ ```
21
+
22
+2. Identify the unassigned shards
23
+
24
+ Use the Elasticsearch `_cat/shards` endpoint to view the status of all shards in your cluster:
25
+ ```
26
+ curl -X GET "localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason&pretty"
27
+ ```
28
+
29
+3. Check Elasticsearch logs
30
+
31
+ Examine the Elasticsearch logs for any error messages or alerts related to shard allocation. The log file is usually located at `/var/log/elasticsearch/`.
32
+
33
+4. Resolve shard allocation issues
34
+
35
+ Depending on the cause of the unassigned shards, you may need to perform actions such as:
36
+
37
+ - Add more nodes to the cluster to distribute the load evenly.
38
+ - Reallocate shards manually using the Elasticsearch `_cluster/reroute` API.
39
+ - Adjust shard allocation settings in the Elasticsearch `elasticsearch.yml` configuration file.
40
+
41
+5. Recheck the cluster health
42
+
43
+ After addressing the issues found in the previous steps, use the `_cluster/health` endpoint again to check if the health status of the affected index has improved.
44
+
45
+### Useful resources
46
+
47
+1. [Elasticsearch: Cluster Health](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html)
48
+2. [Elasticsearch: Shards and Replicas](https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html#shards-and-replicas)
49
+3. [Elasticsearch: Shard Allocation and Cluster-Level Settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html)
\ No newline at end of file
health/guides/elasticsearch_node_indices_search_time_fetch.md
new
+49
@@ -0,0 +1,49 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the Elasticsearch node's average `search_time_fetch` exceeds the warning or critical thresholds over a 10-minute window. The `search_time_fetch` measures the time spent fetching data from shards during search operations. If you receive this alert, it means your Elasticsearch search performance is degraded, and fetches are running slowly.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the Elasticsearch cluster health
8
+
9
+Run the following command to check the health of your Elasticsearch cluster:
10
+
11
+```
12
+curl -XGET 'http://localhost:9200/_cluster/health?pretty'
13
+```
14
+
15
+Look for the `status` field in the output, which indicates the overall health of the cluster:
16
+
17
+- green: All primary and replica shards are active and allocated.
18
+- yellow: All primary shards are active, but not all replica shards are active.
19
+- red: Some primary shards are not active.
20
+
21
+2. Identify slow search queries
22
+
23
+Run the following command to gather information on slow search queries:
24
+
25
+```
26
+curl -XGET 'http://localhost:9200/_nodes/stats/indices/search?pretty'
27
+```
28
+
29
+Look for the `query`, `fetch`, and `take` fields in the output, which indicate the time taken by different parts of the search operation.
30
+
31
+3. Check Elasticsearch node resources
32
+
33
+Ensure the Elasticsearch node has sufficient resources (CPU, memory, disk space, and disk I/O). Use system monitoring tools like `top`, `htop`, `vmstat`, and `iostat` to analyze the resource usage on the Elasticsearch node.
34
+
35
+4. Optimize search queries
36
+
37
+If slow search queries are identified in Step 2, consider optimizing them for better performance. Some techniques for optimizing Elasticsearch search performance include using filters, limiting result set size, and disabling expensive operations like sorting and faceting when not needed.
38
+
39
+5. Review Elasticsearch configuration
40
+
41
+Check your Elasticsearch configuration to ensure it is optimized for search performance. Verify settings such as index refresh intervals, query caches, and field data caches. Consult the Elasticsearch documentation for best practices on configuration settings.
42
+
43
+6. Consider horizontal scaling
44
+
45
+If your Elasticsearch node is experiencing high search loads regularly, consider adding more nodes to distribute the load evenly across the cluster.
46
+
47
+### Useful resources
48
+
49
+1. [Elasticsearch Performance Tuning](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html)
health/guides/elasticsearch_node_indices_search_time_query.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the average search time for Elasticsearch queries has been higher than the defined warning thresholds. If you receive this alert, it means that your search performance is degraded, and queries are running slower than usual.
4
+
5
+### What does search performance mean?
6
+
7
+Search performance in Elasticsearch refers to how quickly and efficiently search queries are executed, and the respective results are returned. Good search performance is essential for providing fast and relevant results in applications and services relying on Elasticsearch for their search capabilities.
8
+
9
+### What causes degraded search performance?
10
+
11
+Several factors can cause search performance degradation, including:
12
+
13
+- High system load, causing CPU, memory or disk I/O bottlenecks
14
+- Poorly optimized search queries
15
+- High query rate, resulting in a large number of concurrent queries
16
+- Insufficient hardware or resources allocated to Elasticsearch
17
+
18
+### Troubleshoot the alert
19
+
20
+1. Check the Elasticsearch logs for any error messages or warnings:
21
+
22
+ ```
23
+ cat /var/log/elasticsearch/elasticsearch.log
24
+ ```
25
+
26
+2. Monitor the system resources (CPU, memory, and disk I/O) using tools like `top`, `vmstat`, and `iotop`. Determine if there are any resource bottlenecks affecting the search performance.
27
+
28
+3. Analyze and optimize the slow search queries by using the Elasticsearch [Slow Log](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html).
29
+
30
+4. Evaluate the cluster health status by running the following Elasticsearch API command:
31
+
32
+ ```
33
+ curl -XGET 'http://localhost:9200/_cluster/health?pretty'
34
+ ```
35
+
36
+ Check for any issues that may be impacting the search performance.
37
+
38
+5. Assess the number of concurrent queries and, if possible, reduce the query rate or distribute the load among additional Elasticsearch nodes.
39
+
40
+6. If the issue persists, consider scaling up your Elasticsearch deployment or allocating additional resources to the affected nodes to improve their performance.
41
+
42
+### Useful resources
43
+
44
+1. [Tune for Search Speed - Elasticsearch Guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html)
health/guides/entropy/lowest_entropy.md
deleted
-141
@@ -1,141 +0,0 @@
1
-# lowest_entropy
2
-
3
-## OS: Linux
4
-
5
-This alert presents the minimum amount of entropy in the kernel entropy pool in the last 5 minutes.
6
-
7
-Low entropy can lead to a reduction in the quality of random numbers produced by `/dev/random`
8
-and `/dev/urandom`.
9
-
10
-The Netdata Agent checks for the minimum entropy value in the last 5 minutes. The alert gets raised
11
-into warning if the value < 100, and cleared if the value > 200.
12
-
13
-For further information on how our alerts are calculated, please have a look at our [Documentation](
14
-https://learn.netdata.cloud/docs/agent/health/reference#expressions).
15
-
16
-
17
-<details>
18
-<summary>What is entropy and why do we need it?</summary>
19
-
20
-Entropy is similar to "randomness". A Linux system gathers "real" random numbers by keeping an eye
21
-on different events: network activity, hard drive rotation speeds, hardware random number
22
-generator (if available), key-clicks, and so on. It feeds those to the kernel entropy pool, which is
23
-used by `/dev/random`.<sup>[1](
24
-https://unixhealthcheck.com/blog?id=472) </sup>
25
-
26
-Encryption and cryptography applications require random numbers to operate. A function or an
27
-algorithm that produces numbers -*that seem to be random*- is very predictable, if you know what
28
-function is used.
29
-
30
-In real life, we use our surroundings and our thoughts to produce truly random numbers. A computer
31
-can't really do this by itself, so it gathers numbers from a lot of sources. For example, it can get
32
-the CO<sub>2</sub> levels in a room from a sensor on the system and use that as a random number.
33
-
34
-This way all the values are random and there is no pattern to be found among them.
35
-</details>
36
-
37
-For further information, please have a look at the _References and Sources_ section.
38
-
39
-<details>
40
-<summary>References and Sources</summary>
41
-
42
-1. [Entropy](https://unixhealthcheck.com/blog?id=472)
43
-2. [rng-tools](https://github.com/nhorman/rng-tools)
44
-3. [How to add more entropy to improve cryptographic randomness on Linux](
45
- https://www.techrepublic.com/article/how-to-add-more-entropy-to-improve-cryptographic-randomness-on-linux/)
46
-4. [Haveged Installation - Archlinux Wiki](https://wiki.archlinux.org/title/Haveged#Installation)
47
-
48
-</details>
49
-
50
-### Troubleshooting Section
51
-
52
-The best tool to troubleshoot the lowest entropy alert is with `rng-tools`. If `rng-tools` are not
53
-available for your platform, or you run into trouble, you can use the tool `haveged` as an
54
-alternative.
55
-
56
-<details>
57
-<summary>Install and setup rng-tools</summary>
58
-
59
-`rng-tools` is a random number generator daemon.
60
-It monitors a set of entropy sources, and supplies entropy from them to the system kernel's
61
-/dev/random machinery.<sup>[2](https://github.com/nhorman/rng-tools) </sup>
62
-
63
-### Installation
64
-
65
-### Debian-based platforms
66
-
67
-```
68
-root@netdata~ # sudo apt-get update
69
-root@netdata~ # sudo apt-get install rng-tools
70
-```
71
-
72
-### RHEL/Fedora/CentOS machines
73
-
74
-1. Change to the root account;
75
-
76
-```
77
-root@netdata~ # su
78
-```
79
-
80
-2. And then install;
81
-
82
-```
83
-root@netdata~ # yum install rng-tools
84
-```
85
-
86
-### After the Installation
87
-
88
-You can run the service using the following command;
89
-
90
-```
91
-root@netdata~ # service rngd start
92
-```
93
-
94
-And also you can check the daemon status using the following command;
95
-
96
-```
97
-root@netdata~ # service rngd status
98
-```
99
-
100
-</details>
101
-
102
-
103
-
104
-<details><summary>Install Haveged</summary>
105
-
106
-Ideally, a system with high entropy demands should have a hardware device to generate random
107
-numbers. For example, a TPM is such a device. However, there are also several software-only options
108
-you may install, like `haveged` [(*read more*)](
109
-https://wiki.archlinux.org/title/Haveged#Installation).
110
-
111
-### Installation
112
-
113
-### Debian-based platforms
114
-
115
-1. To install `haveged`, run:
116
-
117
- ```
118
- root@netdata~ # sudo apt-get install haveged
119
- ```
120
-
121
-2. Set `haveged` up to start at boot with the command `sudo update-rc.d haveged defaults`.<sup>[3](
122
- https://www.techrepublic.com/article/how-to-add-more-entropy-to-improve-cryptographic-randomness-on-linux/) </sup>
123
-
124
-### RHEL/Fedora/CentOS machines
125
-
126
-1. Change to the root account:
127
-
128
- ```
129
- root@netdata~ # su
130
- ```
131
-
132
-2. Install `haveged`:
133
-
134
- ```
135
- root@netdata~ # yum install haveged
136
- ```
137
-
138
-3. Set `haveged` to start at boot with the command `chkconfig haveged on`.<sup>[3](
139
- https://www.techrepublic.com/article/how-to-add-more-entropy-to-improve-cryptographic-randomness-on-linux/) </sup>
140
-
141
-</details>
health/guides/exporting_last_buffering.md
new
+29
@@ -0,0 +1,29 @@
1
+### Understand the alert
2
+
3
+This alert is related to the Netdata Exporting engine, which calculates the number of seconds since the last successful buffering of exporting data. If you receive this alert, it means the exporting engine failed to buffer metrics for a while, and some metrics were lost during exporting. There might be issues with the exporting destination being down or unreachable.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the exporting destination status and accessibility: If the exporting destination (e.g. a remote server or database) is down or unreachable, your priority should be to fix the connection issue or bring the destination back up.
8
+
9
+2. Investigate short-term network availability problems: Short-term network connectivity issues might cause temporary errors in the exporting process. You may want to check and monitor your network to confirm this is the case and fix any issues.
10
+
11
+3. Increase the `buffer on failures` value in `exporting.conf`: You can try to prevent short-term problems from causing alert issues by increasing the `buffer on failures` value in the `exporting.conf` file. To do this, edit the configuration file, find the parameter `buffer on failures`, and increase its value.
12
+
13
+ ```
14
+ [exporting:global]
15
+ buffer on failures = new_value
16
+ ```
17
+ Replace `new_value` with the desired number that matches your system requirements.
18
+
19
+4. Restart the Netdata Agent: After modifying the `exporting.conf` file, don't forget to restart the Netdata Agent for changes to take effect. Use the following command to restart the agent:
20
+
21
+ ```
22
+ sudo systemctl restart netdata
23
+ ```
24
+
25
+5. Monitor the `exporting_last_buffering` alert: After applying the changes, keep monitoring the `exporting_last_buffering` alert to check if the issue is resolved. If the alert continues, further investigate possible issues with the exporting engine or destination.
26
+
27
+### Useful resources
28
+
29
+1. [Netdata Exporting Engine](https://learn.netdata.cloud/docs/exporting-data-to-other-systems/exporting-reference)
health/guides/exporting_metrics_sent.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+The `exporting_metrics_sent` alert is triggered when the Netdata Agent fails to send all metrics to the configured external database server. This could be due to the exporting destination being down, unreachable, or short-term network availability problems.
4
+
5
+### Troubleshoot the alert
6
+
7
+To troubleshoot this alert, follow these steps:
8
+
9
+1. Verify the exporting destination status:
10
+
11
+ - Make sure the external database server is up and running.
12
+ - Check if there are any issues with the server, such as high CPU usage, low memory, or a full disk.
13
+
14
+2. Check the network connection between the Netdata Agent and the external database server:
15
+
16
+ - Use tools like `ping` or `traceroute` to test the connection.
17
+ - Check for any firewall rules that may be blocking the connection.
18
+
19
+3. Increase the `buffer on failures` in `exporting.conf`:
20
+
21
+ - Open the `exporting.conf` file, which is typically located at `/etc/netdata/exporting.conf`.
22
+
23
+ - Increase the value of the `buffer on failures` setting to allow for more metrics to be stored when network/connectivity issues occur. For example, if the current setting is `10000`, try increasing it to `20000` or higher, depending on your server's available memory.
24
+
25
+ ```
26
+ [exporting:global]
27
+ buffer on failures = 20000
28
+ ```
29
+
30
+ - Save and exit the file.
31
+
32
+ - Restart the Netdata Agent to apply the changes.
33
+
34
+4. Review the Netdata Agent logs:
35
+
36
+ - Check for any error messages or warnings related to the exporting engine in the Netdata Agent logs (`/var/log/netdata/error.log`).
37
+
38
+ - Use the information from the logs to troubleshoot any issues you find.
39
+
40
+5. Ensure your configuration settings are correct:
41
+
42
+ - Double-check your exporting configuration settings (located in `/etc/netdata/exporting.conf`) to ensure they match the requirements of your external database server.
43
+
44
+### Useful resources
45
+
46
+1. [Netdata Exporting Engine documentation](https://learn.netdata.cloud/docs/exporting-data-to-other-systems/exporting-reference)
health/guides/fping_host_latency.md
new
+22
@@ -0,0 +1,22 @@
1
+### Understand the alert
2
+
3
+`fping` is a command line tool to send ICMP (Internet Control Message Protocol) echo requests to network hosts, similar to ping, but performing much better when pinging multiple hosts. The Netdata
4
+Agent utilizes `fping` to monitor latency, packet loss, uptime and reachability of any number of network endpoints.
5
+
6
+For the `fping_host_latency` alert, the Netdata Agent monitors the average latency to the network host over the last 10 seconds. Receiving this alert indicates high latency to the network host. It is likely you are experiencing networking issues or the host is overloaded.
7
+
8
+### Troubleshoot the alert
9
+
10
+- Customize the ICMP requests for each endpoint
11
+
12
+Different endpoints could be in different networks. For example, a server in your intra network would require less time to be accessed than your cloud infrastructures in terms of latency. You should always consider not to use a global approach for checking every endpoint of yours. You can find more information about how to configure every endpoint separately in the [fping.plugin alarm guide](https://learn.netdata.cloud/docs/agent/collectors/fping.plugin/#additional-tips).
13
+
14
+- Prioritize traffic on your endpoints
15
+
16
+Quality of service (QoS) is the use of mechanisms or technologies to control traffic and ensure the performance of critical applications. QoS works best when low-priority traffic exists that can be dropped when congestion occurs. The higher-priority traffic must fit within the bandwidth limitations of the link or path. The following are two open source solutions to apply QoS policies to your network interfaces.
17
+
18
+### Useful resources
19
+
20
+- [FireQOS]((https://firehol.org/tutorial/fireqos-new-user/) is a traffic shaping helper. It has a very simple shell scripting language to express traffic shaping.
21
+
22
+- [`tcconfig`](https://tcconfig.readthedocs.io/en/latest/index.html) is a command wrapper that makes it easy to set up traffic control of network bandwidth, latency, packet-loss, packet-corruption, etc.
\ No newline at end of file
health/guides/fping_host_reachable.md
new
+55
@@ -0,0 +1,55 @@
1
+### Understand the alert
2
+
3
+`fping` is a command line tool to send ICMP (Internet Control Message Protocol) echo requests to network hosts, similar to ping, but performing much better when pinging multiple hosts. The Netdata
4
+Agent utilizes `fping` to monitor latency, packet loss, uptime and reachability of any number of network endpoints.
5
+
6
+The `fping_host_reachable` alert in the Netdata Agent checks the reachability of a network host (0: unreachable, 1: reachable). Receiving a critical alert indicates that your endpoints are unreachable. It is likely that the host is down or your system is experiencing networking issues.
7
+
8
+### Troubleshoot the alert
9
+
10
+- Check network connectivity
11
+
12
+Verify that your system has access to the particular endpoint. Check for basic connectivity to known hosts from both your host and the endpoint.
13
+
14
+- DNS settings
15
+
16
+If you are using DNS resolution to check your endpoint, you should always consider check your DNS settings. To troubleshoot this issue, verify that your DNS can resolve your endpoints.
17
+
18
+1. Check your current DNS (for example in linux you can use the host command):
19
+
20
+ ```
21
+ host -v <your_endpoint>
22
+ ```
23
+
24
+2. If the HTTP endpoint is supposed to be public facing endpoint, try an alternative DNS (for example Cloudflare's DNS):
25
+
26
+ ```
27
+ host -v <your_endpoint> 1.1.1.1
28
+ ```
29
+- Verify access restrictions in the remote host</summary>
30
+
31
+If the remote host is a Linux-based machine and you have access to it, you can check the followings.
32
+
33
+**Check the ICMP settings**
34
+
35
+In most linux distributions you can restrict the ICMP echo operations.
36
+
37
+ 1. Check your current setting. If this value is set to 1 your system ignore incoming ICMP echo requests.
38
+ ```
39
+ systemctl net.ipv4.icmp_echo_ignore_all
40
+ ```
41
+ 2. To change this, bump this `net.ipv4.icmp_echo_ignore_all=0` entry under `/etc/sysctl.conf`.
42
+
43
+ 3. Reload the sysctl settings.
44
+ ```
45
+ sysctl -p
46
+ ```
47
+
48
+**Check your firewall rules**
49
+
50
+Depending on what firewall you use, the commands might differ from what's shown below. For example, if you are using IP tables you can check for restriction rules upon `icmp`.
51
+ ```
52
+ iptables -L | grep ICMP
53
+ ```
54
+
55
+For futher investigation or changes in your firewall settings we **strongly** advise you to consult your firewall's documentation and guidelines.
\ No newline at end of file
health/guides/gearman_workers_queued.md
new
+43
@@ -0,0 +1,43 @@
1
+### Understand the alert
2
+
3
+This alert is related to the Gearman application framework. If you receive this alert, it means that the average number of queued jobs in the last 10 minutes is significantly high, indicating that more workers may be needed to maintain an efficient workflow.
4
+
5
+### What is Gearman?
6
+
7
+Gearman is an open-source, distributed job scheduling framework that allows applications to distribute processing tasks among multiple worker machines. It is useful to parallelize tasks and manage workloads between different systems.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the status of Gearman with the following command:
12
+
13
+ ```
14
+ gearadmin --status
15
+ ```
16
+
17
+2. Analyze the output and identify queues with a high number of jobs:
18
+
19
+ Example output:
20
+
21
+ ```
22
+ queue1 50000 10 0
23
+ queue2 65000 20 0
24
+ ```
25
+
26
+ In this example, `queue1` and `queue2` have a high number of queued jobs (50,000 and 65,000), with 10 and 20 workers working on them respectively.
27
+
28
+3. Increase the number of workers:
29
+
30
+ To increase the number of workers, you may need to start additional worker instances or adjust the configurable number of workers in your Gearman deployment. For instance, if you use a script to start workers, you can update this script and start more instances.
31
+
32
+4. Monitor the Gearman metrics:
33
+
34
+ Continue to monitor the metrics for some time to ensure that the additional workers are effectively reducing the number of queued jobs.
35
+
36
+5. If necessary, further optimize the Gearman deployment:
37
+
38
+ If the problem persists, you may need to analyze the queues in further detail, such as looking into possible bottlenecks, inefficient operations, or other performance-related factors.
39
+
40
+### Useful resources
41
+
42
+1. [Monitoring Gearman with Netdata](https://www.netdata.cloud/gearman-monitoring/)
43
+2. [Gearman Documentation](http://gearman.org/documentation/)
health/guides/geth_chainhead_diff_between_header_block.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+The `geth_chainhead_diff_between_header_block` alert is generated by the Netdata Agent when monitoring an Ethereum node using the Geth client. The alert is triggered when there is a significant difference between the chain head header and the actual block number. This indicates that your Ethereum node is out of sync with the rest of the network, and you may experience issues when trying to interact with the blockchain.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the sync status of your Ethereum node**: To determine the current sync status of your Ethereum node, execute the following command in the Geth console:
8
+
9
+ ```
10
+ eth.syncing
11
+ ```
12
+
13
+ If your node is syncing, you'll see information about the syncing progress. If it's not syncing, the command will return `false`.
14
+
15
+2. **Restart Geth**: If your node is not syncing, try restarting Geth with the appropriate command for your operating system. For example:
16
+
17
+ ```
18
+ sudo systemctl restart geth
19
+ ```
20
+
21
+ If Geth isn't running as a system service, you may need to restart it manually. Ensure you're using the correct command-line flags and options.
22
+
23
+3. **Check Geth logs**: Inspect the Geth logs for any issues or errors that may provide insight into the problem. You can find the logs in the default log directory or use the `--log-dir` flag to specify a custom log directory. To view the logs in real-time, run:
24
+
25
+ ```
26
+ tail -f /path/to/your/log/directory/geth.log
27
+ ```
28
+
29
+4. **Ensure your system time is accurate**: An incorrect system time can cause syncing issues. Make sure your system clock is accurate and synchronized with an NTP server:
30
+
31
+ ```
32
+ sudo ntpdate -s time.nist.gov
33
+ ```
34
+
35
+5. **Upgrade Geth**: Ensure you are running the latest version of Geth. Upgrading to the latest version can resolve issues and improve synchronization. You can find the latest release on the [Geth GitHub repository](https://github.com/ethereum/go-ethereum/releases).
36
+
37
+6. **Check your network connection**: Verify that your node has a stable and reliable network connection. If you're on a shared network, consider increasing the bandwidth or moving the node to a dedicated network.
38
+
39
+### Useful resources
40
+
41
+1. [Geth Documentation](https://geth.ethereum.org/docs/)
42
+2. [Ethereum Stack Exchange](https://ethereum.stackexchange.com/)
43
+3. [Netdata Ethereum Monitoring Guide](https://blog.netdata.cloud/how-to-monitor-the-geth-node-in-under-5-minutes/)
44
+4. [Geth GitHub Repository](https://github.com/ethereum/go-ethereum)
health/guides/go.d_job_last_collected_secs.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+The Netdata Agent also monitors itself, so this is an alert about the Netdata go.d plugin. The Netdata Agent keeps track of the number of seconds since the last successful data collection for each data collection job.
4
+This alert indicates that a particular job has failed to collect metrics for several consecutive attempts.
5
+
6
+You can see all the modules that are orchestrated by the go.d.plugin in our [go.d.plugin documentation page](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin#available-modules)
7
+
8
+### Troubleshoot the alert
9
+
10
+- Check the Netdata logs
11
+
12
+You need to identify why the Agent cannot collect metrics for a specific job. Inspect the Agent logs for this specific job.
13
+
14
+Host machine:
15
+
16
+ ```
17
+ tail -f /var/log/netdata/error.log | grep <job> OR <module_name>
18
+ ```
19
+
20
+Docker:
21
+
22
+ ```
23
+ docker logs <netdata_container> 2>&1 | grep <job> OR <module_name>
24
+ ```
25
+
26
+Kubernetes:
27
+ 1. Find the pod name of the node which produced the alert.
28
+
29
+ ```
30
+ kubectl -n <namespace> get pod -o wide -l app=netdata | grep <node_name>
31
+ ```
32
+ 2. Inspect it's logs
33
+
34
+ ```
35
+ kubectl logs -n <namespace> <pod_name> | grep <job> OR <module_name>
36
+ ```
37
+
\ No newline at end of file
health/guides/haproxy/haproxy_backend_server_status.md
deleted
-85
@@ -1,85 +0,0 @@
1
-# haproxy_backend_server_status
2
-
3
-**Web Proxy | HAProxy**
4
-
5
-HAProxy is a free, fast and reliable reverse-proxy offering high availability, load balancing,
6
-and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic
7
-web sites and powers a significant portion of the world's most visited ones. Over the years it has
8
-become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux
9
-distributions, and is often deployed by default in cloud platforms.
10
-
11
-The Netdata Agent monitors the average number of failed HAProxy backend servers over the last 10
12
-seconds. Receiving this alert (in critical state) means that one or more HAProxy backend servers are
13
-inaccessible or offline.
14
-
15
-_There are four essential sections to an HAProxy configuration file. They are global, defaults,
16
-frontend, and backend. These four sections define how the server as a whole performs, what your
17
-default settings are, and how client requests are received and routed to your backend
18
-servers._ <sup> [1](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/) </sup>
19
-
20
-<details>
21
-<summary>HA Proxy Backend Servers</summary>
22
-
23
-Backend servers are the cornerstone of the HA proxy architecture. HA proxy organizes multiple
24
-servers to `Backends` (a pool of servers) and implements different (defined by you) Layer 4 or Layer
25
-7 load balancing algorithms to assign the incoming requests to each individual server.
26
-
27
-> You can define a new server with the `server` setting or use the `default-server` configuration
28
-which is configured once. Its first argument is a name, followed by the IP address and port of the
29
-backend server. You can specify a domain name instead of an IP address. In that case, it will be
30
-resolved at startup or, if you add a `resolvers` argument, it will be updated during runtime. If the
31
-DNS entry contains an SRV record, the port and weight will be filled in from it too. If the port
32
-isn’t specified, then HAProxy will use the same port that the client connected on, which is useful
33
-for randomly used ports such as for active-mode FTP.
34
->
35
-> Every `server` line should have a `maxconn` setting that limits the maximum number of concurrent
36
-requests that the server will be given. Even if it’s just a guess, having a value here puts you on
37
-the right foot for avoiding saturating your servers with requests and gives a baseline that can be
38
-adjusted later. <sup> [1](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/) </sup>
39
-
40
-</details>
41
-
42
-<details>
43
-<summary>References and sources</summary>
44
-
45
-1. [The Four Essential Sections of an HAProxy Configuration](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/)
46
-
47
-</details>
48
-
49
-### Troubleshooting section
50
-
51
-<details>
52
-<summary>Check the HA proxy's configuration file for errors</summary>
53
-
54
-Making changes in the configuration file may introduce errors. Make sure your always validate the
55
-correctness of the configuration file.
56
-
57
-1. In most Linux distros you can run the following check:
58
-
59
-```
60
-root@netadata # haproxy -c -f /etc/haproxy/haproxy.cfg
61
-```
62
-</details>
63
-
64
-<details>
65
-<summary>Check the HA proxy service for errors</summary>
66
-
67
-1. Use `journalctl` and inspect the log:
68
-
69
-```
70
-root@netdata # journalctl -u haproxy.service --reverse
71
-```
72
-</details>
73
-
74
-<details>
75
-<summary>Check the HA proxy's log</summary>
76
-
77
-1. By default HA proxy logs under `/var/log/haproxy.log`:
78
-
79
-```
80
-root@netdata # cat /var/log/haproxy.log | grep 'emerg\|alert\|crit\|err\|warning\|notice'
81
-```
82
-
83
-You can also search for log messages with `info` and `debug` tags.
84
-
85
-</details>
health/guides/haproxy/haproxy_backend_status.md
deleted
-80
@@ -1,80 +0,0 @@
1
-# haproxy_backend_status
2
-
3
-**Web Proxy | HAProxy**
4
-
5
-HAProxy is a free, very fast and reliable reverse-proxy offering high availability, load balancing,
6
-and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic
7
-web sites and powers a significant portion of the world's most visited ones. Over the years it has
8
-become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux
9
-distributions, and is often deployed by default in cloud platforms.
10
-
11
-The Netdata Agent monitors the average number of failed HAProxy backends over the last 10 seconds.
12
-Receiving this alert (in critical state) means that one or more HAProxy backend are inaccessible or
13
-offline.<sup> [1](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/) </sup>
14
-
15
-<details>
16
-<summary>HA Proxy Backends</summary>
17
-
18
-> A HA proxy `backend` is a set of servers that receives forwarded requests. Backends are defined in
19
-the backend section of the HAProxy configuration. In its most basic form, a backend can be defined by:
20
->
21
-> - which load balance algorithm to use
22
->
23
-> - a list of servers and ports
24
->
25
-> A backend can contain one or many servers in it–generally speaking, adding more servers to your
26
-backend will increase your potential load capacity by spreading the load over multiple servers.
27
-Increase reliability is also achieved through this manner, in case some of your backend servers
28
-become unavailable. <sup>[2](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts) </sup>
29
-
30
-</details>
31
-
32
-<details>
33
-<summary>References and Sources</summary>
34
-
35
-1. [The Four Essential Sections of an HAProxy Configuration](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/)
36
-
37
-2. [HA proxy explained in DigitalOcean](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts)
38
-
39
-</details>
40
-
41
-### Troubleshooting Section
42
-
43
-<details>
44
-<summary>Check the HA proxy's configuration file for errors</summary>
45
-
46
-Making changes in the configuration file may introduce errors. Make sure your always validate the
47
-correctness of the configuration file.
48
-
49
-1. In most Linux distros you can run the following check:
50
-
51
-```
52
-root@netadata # haproxy -c -f /etc/haproxy/haproxy.cfg
53
-```
54
-
55
-</details>
56
-
57
-
58
-<details>
59
-<summary>Check the HA proxy service for errors</summary>
60
-
61
-1. Use `journalctl` and inspect the log:
62
-
63
-```
64
-root@netdata # journalctl -u haproxy.service --reverse
65
-```
66
-
67
-</details>
68
-
69
-<details>
70
-<summary>Check the HA proxy's log</summary>
71
-
72
-1. By default HA proxy logs under `/var/log/haproxy.log`:
73
-
74
-```
75
-root@netdata # cat /var/log/haproxy.log | grep 'emerg\|alert\|crit\|err\|warning\|notice'
76
-```
77
-
78
-You can also search for log messages with `info` and `debug` tags.
79
-
80
-</details>
health/guides/haproxy_backend_server_status.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+The `haproxy_backend_server_status` alert is triggered when one or more backend servers that are managed by HAProxy are inaccessible or offline. HAProxy is a reverse-proxy that provides high availability, load balancing, and proxying for TCP and HTTP-based applications. If you receive this alert, it means that there may be a problem with your backend server(s), and incoming requests could face delays or not be processed correctly.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the HAProxy backend server status**
8
+
9
+ You can check the status of each individual backend server by accessing the HAProxy Statistics Report. By default, this report can be accessed on the HAProxy server using the URL:
10
+
11
+ ```
12
+ http://<Your-HAProxy-Server-IP>:9000/haproxy_stats
13
+ ```
14
+
15
+ Replace `<Your-HAProxy-Server-IP>` with the IP address of your HAProxy server. If you have configured a different port for the statistics report, use that instead of `9000`.
16
+
17
+ In the report, look for any backend server(s) with a `DOWN` status.
18
+
19
+2. **Investigate the problematic backend server(s)**
20
+
21
+ For each of the backend servers that are in a `DOWN` status, check the availability and health of the server. Make sure that the server is running, and check its resources (CPU, memory, disk space, network) to identify any potential issues.
22
+
23
+3. **Validate the HAProxy configuration**
24
+
25
+ As mentioned in the provided guide, it is essential to validate the correctness of the HAProxy configuration file. If you haven't already, follow the steps in the guide to check for any configuration errors or warnings.
26
+
27
+4. **Check for recent changes**
28
+
29
+ If the backend servers were previously working correctly, inquire about any recent changes to the infrastructure, such as software updates or configuration changes.
30
+
31
+5. **Restart the HAProxy service**
32
+
33
+ If the backend server(s) seem to be healthy, but the alert still persists, try restarting the HAProxy service:
34
+
35
+ ```
36
+ sudo systemctl restart haproxy
37
+ ```
38
+
39
+6. **Monitor the alert and backend server status**
40
+
41
+ After applying any changes or restarting the HAProxy service, monitor the alert and the backend server status in the HAProxy Statistics Report to see if the issue has been resolved.
42
+
43
+### Useful resources
44
+
45
+1. [HAProxy Configuration Manual](https://cbonte.github.io/haproxy-dconv/2.0/configuration.html)
46
+2. [HAProxy Log Customization](https://www.haproxy.com/blog/introduction-to-haproxy-logging/)
health/guides/haproxy_backend_status.md
new
+49
@@ -0,0 +1,49 @@
1
+### Understand the alert
2
+
3
+This alert monitors the average number of failed HAProxy backends over the last 10 seconds. If you receive this alert in a critical state, it means that one or more HAProxy backends are inaccessible or offline.
4
+
5
+HAProxy is a reverse-proxy that provides high availability, load balancing, and proxying for TCP and HTTP-based applications. A backend in HAProxy is a set of servers that receive forwarded requests and are defined in the backend section of the configuration.
6
+
7
+### Troubleshoot the alert
8
+
9
+- Check the HAProxy configuration file for errors
10
+
11
+ Making changes in the configuration file may introduce errors. Always validate the correctness of the configuration file. In most Linux distros, you can run the following check:
12
+
13
+ ```
14
+ haproxy -c -f /etc/haproxy/haproxy.cfg
15
+ ```
16
+
17
+- Check the HAProxy service for errors
18
+
19
+ 1. Use `journalctl` and inspect the log:
20
+
21
+ ```
22
+ journalctl -u haproxy.service --reverse
23
+ ```
24
+
25
+- Check the HAProxy log
26
+
27
+ 1. By default, HAProxy logs under `/var/log/haproxy.log`:
28
+
29
+ ```
30
+ cat /var/log/haproxy.log | grep 'emerg\|alert\|crit\|err\|warning\|notice'
31
+ ```
32
+
33
+ You can also search for log messages with `info` and `debug` tags.
34
+
35
+- Investigate the backend servers
36
+
37
+ 1. Verify that the backend servers are online and accepting connections.
38
+ 2. Check the backend server logs for any errors or issues.
39
+ 3. Ensure that firewall rules or security groups are not blocking traffic from HAProxy to the backend servers.
40
+
41
+- Review the HAProxy load balancing algorithm and configuration
42
+
43
+ 1. Analyze the load balancing algorithm used in the configuration to ensure it is suitable for your setup.
44
+ 2. Check for any misconfigurations, such as incorrect server addresses, ports, or weights.
45
+
46
+### Useful resources
47
+
48
+1. [The Four Essential Sections of an HAProxy Configuration](https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/)
49
+2. [HAProxy Explained in DigitalOcean](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts)
\ No newline at end of file
health/guides/hdfs/hdfs_capacity_usage.md
deleted
-139
@@ -1,139 +0,0 @@
1
-# hdfs_capacity_usage
2
-
3
-**Storage | HDFS**
4
-
5
-_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
-written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
-lack of POSIX compliance, but it does provide shell commands and Java application programming
8
-interface (API) methods that are similar to other file
9
-systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
-
11
-The Netdata Agent calculates the percentage of used space capacity across all DataNodes. Receiving
12
-this alert indicates high DataNodes space capacity utilization.
13
-
14
-This alert is triggered into warning when the percentage of used space capacity across all DataNodes
15
-is between 70-80% and in critical when it is between 80-90%.
16
-
17
-<details>
18
-<summary>See more about Hadoop</summary>
19
-
20
-Wikipedia provides a great explanation of
21
-HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
22
-
23
-HDFS provides a software framework for distributed storage and processing of big data using the
24
-`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
25
-processing data. It achieves reliability by replicating the data across multiple hosts, and hence
26
-theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
27
-the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
28
-different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
29
-keep the replication of data high.
30
-
31
-HDFS has five services as follows:
32
-
33
-1. Name Node
34
-2. Secondary Name Node
35
-3. Job tracker
36
-4. Data Node
37
-5. Task Tracker
38
-
39
-Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
40
-communicate with each other and in the same way slave services can communicate with each other.
41
-NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
42
-each other.
43
-
44
-- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
45
- track files, manage the file system and has the metadata of all the stored data within it. Some
46
- information the NameNode keep track of are:
47
-
48
- - details (metadata) of blocks
49
- - in which DataNode each block lives, and its location
50
- - replication metadata of each block
51
-
52
- The NameNode is the gateway that a client uses to manage the HDFS cluster.
53
-
54
-- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
55
- stores the actual data into HDFS which is responsible for the client to read and write. These are
56
- slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
57
- conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
58
- for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
59
- some other DataNode.
60
-
61
-- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
62
- is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
63
- NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
64
- `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
65
-
66
-- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
67
- tracker talks to the NameNode to know about the location of the data that will be used in
68
- processing. The NameNode responds with the metadata of the required processing data.
69
-
70
-- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
71
- Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
72
- the file. The process of applying that code on the file is known as Mapper.
73
-
74
-Some more useful information/concepts about HDFS from the official
75
-website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
76
-
77
-- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
78
- an application can create directories and store files inside these directories. The file system
79
- namespace hierarchy is similar to most other existing file systems; one can create and remove
80
- files, move a file from one directory to another, or rename a file. HDFS does not yet implement
81
- user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
82
- not preclude implementing these features.
83
-
84
- The NameNode maintains the file system namespace. Any change to the file system namespace or its
85
- properties is recorded by the NameNode. An application can specify the number of replicas of a
86
- file that should be maintained by HDFS. The number of copies of a file is called the replication
87
- factor of that file. This information is stored by the NameNode.
88
-
89
-- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
90
- HDFS are those that deal with large data sets. These applications write their data only once but
91
- they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
92
- supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
93
- Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
94
- different DataNode.
95
-
96
-- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
97
- might automatically move data from one DataNode to another if the free space on a DataNode falls
98
- below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
99
- might dynamically create additional replicas and rebalance other data in the cluster. These types
100
- of data rebalancing schemes are not yet implemented.
101
-
102
-</details>
103
-
104
-<details>
105
-<summary>References and sources</summary>
106
-
107
-1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
108
-2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
109
-
110
-</details>
111
-
112
-### Troubleshooting section
113
-
114
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
115
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
116
-
117
-<details>
118
-<summary>Check your Disk Usage across the cluster </summary>
119
-
120
-1. Inspect the Disk Usage for each DataNode
121
-
122
- ```
123
- root@netdata # hadoop dfsadmin -report
124
- ```
125
-
126
- If all the DataNodes are in Disk pressure, you should consider adding more disk space. Otherwise,
127
- you can perform a balance of data between the DataNodes.
128
-
129
-2. Perform a balance
130
-
131
- ```
132
- root@netdata # hdfs balancer –threshold 15
133
- ```
134
-
135
- This means that the balancer will balance data by moving blocks from over-utilized to
136
- under-utilized nodes, until each DataNode’s disk usage differs by no more than plus or minus 15
137
- percent.
138
-
139
-</details>
health/guides/hdfs/hdfs_dead_nodes.md
deleted
-138
@@ -1,138 +0,0 @@
1
-# hdfs_dead_nodes
2
-
3
-**Storage | HDFS**
4
-
5
-_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
-written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
-lack of POSIX compliance, but it does provide shell commands and Java application programming
8
-interface (API) methods that are similar to other file
9
-systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
-
11
-The Netdata Agent monitors the number of DataNodes that are currently dead. Receiving this alert
12
-indicates that there are dead DataNodes in your HDFS cluster. The NameNode characterizes a DataNode
13
-as dead if no heartbeat message is exchanged for approximately 10 minutes. Any data that was
14
-registered to a dead DataNode is not available to HDFS anymore.
15
-
16
-This alert is triggered into critical when the number of dead DataNodes is 1 or more.
17
-
18
-<details>
19
-<summary>See more about Hadoop</summary>
20
-
21
-Wikipedia provides a great explanation of
22
-HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
23
-
24
-HDFS provides a software framework for distributed storage and processing of big data using the
25
-`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
26
-processing data. It achieves reliability by replicating the data across multiple hosts, and hence
27
-theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
28
-the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
29
-different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
30
-keep the replication of data high.
31
-
32
-HDFS has five services as follows:
33
-
34
-1. Name Node
35
-2. Secondary Name Node
36
-3. Job tracker
37
-4. Data Node
38
-5. Task Tracker
39
-
40
-Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
41
-communicate with each other and in the same way slave services can communicate with each other.
42
-NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
43
-each other.
44
-
45
-- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
46
- track files, manage the file system and has the metadata of all the stored data within it. Some
47
- information the NameNode keep track of are:
48
-
49
- - details (metadata) of blocks
50
- - in which DataNode each block lives, and its location
51
- - replication metadata of each block
52
-
53
- The NameNode is the gateway that a client uses to manage the HDFS cluster.
54
-
55
-- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
56
- stores the actual data into HDFS which is responsible for the client to read and write. These are
57
- slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
58
- conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
59
- for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
60
- some other DataNode.
61
-
62
-- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
63
- is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
64
- NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
65
- `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
66
-
67
-- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
68
- tracker talks to the NameNode to know about the location of the data that will be used in
69
- processing. The NameNode responds with the metadata of the required processing data.
70
-
71
-- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
72
- Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
73
- the file. The process of applying that code on the file is known as Mapper.
74
-
75
-Some more useful information/concepts about HDFS from the official
76
-website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
77
-
78
-- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
79
- an application can create directories and store files inside these directories. The file system
80
- namespace hierarchy is similar to most other existing file systems; one can create and remove
81
- files, move a file from one directory to another, or rename a file. HDFS does not yet implement
82
- user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
83
- not preclude implementing these features.
84
-
85
- The NameNode maintains the file system namespace. Any change to the file system namespace or its
86
- properties is recorded by the NameNode. An application can specify the number of replicas of a
87
- file that should be maintained by HDFS. The number of copies of a file is called the replication
88
- factor of that file. This information is stored by the NameNode.
89
-
90
-- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
91
- HDFS are those that deal with large data sets. These applications write their data only once but
92
- they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
93
- supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
94
- Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
95
- different DataNode.
96
-
97
-- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
98
- might automatically move data from one DataNode to another if the free space on a DataNode falls
99
- below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
100
- might dynamically create additional replicas and rebalance other data in the cluster. These types
101
- of data rebalancing schemes are not yet implemented.
102
-
103
-</details>
104
-
105
-<details>
106
-<summary>References and sources</summary>
107
-
108
-1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
109
-2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
110
-
111
-</details>
112
-
113
-### Troubleshooting section
114
-
115
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
116
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
117
-
118
-<details>
119
-<summary>Fix corrupted or missing blocks</summary>
120
-
121
-1. Identify the dead node(s)
122
-
123
- ```
124
- root@netdata # hadoop dfsadmin -report
125
- ```
126
-
127
-Inspect the output and check which DataNode is dead.
128
-
129
-2. Connect to the DataNode and check the log of the DataNode. You can also check for errors in the
130
- system services.
131
-
132
- ```
133
- root@netdata # systemctl status hadoop
134
- ```
135
-
136
- Restart the service if needed.
137
-
138
-</details>
health/guides/hdfs/hdfs_missing_blocks.md
deleted
-156
@@ -1,156 +0,0 @@
1
-# hdfs_missing_blocks
2
-
3
-**Storage | HDFS**
4
-
5
-_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
-written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
-lack of POSIX compliance, but it does provide shell commands and Java application programming
8
-interface (API) methods that are similar to other file
9
-systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
-
11
-The Netdata Agent monitors the number of missing blocks in a HDFS. Receiving this alert into warning
12
-indicates that there is at least one missing block in one of the DataNodes. This issue could be
13
-caused by a problem with the underlying storage or filesystem of a DataNode.
14
-
15
-<details>
16
-<summary>See more about Hadoop</summary>
17
-
18
-Wikipedia provides a great explanation of
19
-HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
20
-
21
-HDFS provides a software framework for distributed storage and processing of big data using the
22
-`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
23
-processing data. It achieves reliability by replicating the data across multiple hosts, and hence
24
-theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
25
-the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
26
-different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
27
-keep the replication of data high.
28
-
29
-HDFS has five services as follows:
30
-
31
-1. Name Node
32
-2. Secondary Name Node
33
-3. Job tracker
34
-4. Data Node
35
-5. Task Tracker
36
-
37
-Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
38
-communicate with each other and in the same way slave services can communicate with each other.
39
-NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
40
-each other.
41
-
42
-- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
43
- track files, manage the file system and has the metadata of all the stored data within it. Some
44
- information the NameNode keep track of are:
45
-
46
- - details (metadata) of blocks
47
- - in which DataNode each block lives, and its location
48
- - replication metadata of each block
49
-
50
- The NameNode is the gateway that a client uses to manage the HDFS cluster.
51
-
52
-- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
53
- stores the actual data into HDFS which is responsible for the client to read and write. These are
54
- slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
55
- conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
56
- for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
57
- some other DataNode.
58
-
59
-- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
60
- is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
61
- NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
62
- `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
63
-
64
-- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
65
- tracker talks to the NameNode to know about the location of the data that will be used in
66
- processing. The NameNode responds with the metadata of the required processing data.
67
-
68
-- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
69
- Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
70
- the file. The process of applying that code on the file is known as Mapper.
71
-
72
-Some more useful information/concepts about HDFS from the official
73
-website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
74
-
75
-- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
76
- an application can create directories and store files inside these directories. The file system
77
- namespace hierarchy is similar to most other existing file systems; one can create and remove
78
- files, move a file from one directory to another, or rename a file. HDFS does not yet implement
79
- user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
80
- not preclude implementing these features.
81
-
82
- The NameNode maintains the file system namespace. Any change to the file system namespace or its
83
- properties is recorded by the NameNode. An application can specify the number of replicas of a
84
- file that should be maintained by HDFS. The number of copies of a file is called the replication
85
- factor of that file. This information is stored by the NameNode.
86
-
87
-- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
88
- HDFS are those that deal with large data sets. These applications write their data only once but
89
- they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
90
- supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
91
- Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
92
- different DataNode.
93
-
94
-- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
95
- might automatically move data from one DataNode to another if the free space on a DataNode falls
96
- below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
97
- might dynamically create additional replicas and rebalance other data in the cluster. These types
98
- of data rebalancing schemes are not yet implemented.
99
-
100
-</details>
101
-
102
-<details>
103
-<summary>References and sources</summary>
104
-
105
-1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
106
-2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
107
-3. [Man pages of fsck](https://linux.die.net/man/8/fsck)
108
-
109
-</details>
110
-
111
-### Troubleshooting section
112
-
113
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
114
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
115
-
116
-<details>
117
-<summary>Fix corrupted or missing blocks</summary>
118
-
119
-1. Identify which files are facing issues.
120
-
121
- ```
122
- root@netdata # hdfs fsck -list-corruptfileblocks
123
- ```
124
-
125
- Inspect the output and track the path(s) to the corrupted files
126
-
127
-
128
-2. Determine where the file's blocks might live. If the file is larger than your block size it
129
- consists of multiple blocks.
130
-
131
- ```
132
- root@netdata # hdfs fsck <path_to_corrupted_file> -locations -blocks -files
133
- ```
134
-
135
- This command will print out locations for every "problematic" block.
136
-
137
-
138
-3. Search in the corresponding DataNodes and the NameNode's logs for the machine or machines on
139
- which the blocks lived. Try looking for filesystem errors on those machines.
140
- Use `fsck` <sup>[3](https://linux.die.net/man/8/fsck) </sup>.
141
-
142
-
143
-4. If there are files or blocks that you cannot fix, you must delete them so that the HDFS becomes
144
- healthy again.
145
-
146
- - for a specific file
147
- ```
148
- root@netdata # hdfs fs -rm <path_to_file_with_unrecovable_blocks>
149
- ```
150
- OR
151
- - for all the "problematic" files
152
- ```
153
- hdfs fsck / -delete
154
- ```
155
-
156
-</details>
health/guides/hdfs/hdfs_num_failed_volumes.md
deleted
-148
@@ -1,148 +0,0 @@
1
-# hdfs_num_failed_volumes
2
-
3
-**Storage | HDFS**
4
-
5
-_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
-written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
-lack of POSIX compliance, but it does provide shell commands and Java application programming
8
-interface (API) methods that are similar to other file
9
-systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
-The Netdata Agent monitors the number of failed volumes in your HDFS cluster. Receiving this alert
11
-into warning indicates that at least one volume has failed due to a hardware failure or
12
-misconfiguration, for example duplicate mount.
13
-
14
-In an HDFS cluster, a single volume failing on a DataNode will cause the entire node to go offline (
15
-to be more precise it depends on the `dfs.datanode.failed.volumes.tolerated` setting for your
16
-cluster). In case of a volume fail incident, the NameNode must copy any under-replicated blocks that
17
-were lost on that node, causing a burst in network traffic and potential performance degradation.
18
-
19
-
20
-<details>
21
-<summary>See more about Hadoop</summary>
22
-
23
-Wikipedia provides a great explanation of
24
-HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
25
-
26
-HDFS provides a software framework for distributed storage and processing of big data using the
27
-`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
28
-processing data. It achieves reliability by replicating the data across multiple hosts, and hence
29
-theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
30
-the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
31
-different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
32
-keep the replication of data high.
33
-
34
-HDFS has five services as follows:
35
-
36
-1. Name Node
37
-2. Secondary Name Node
38
-3. Job tracker
39
-4. Data Node
40
-5. Task Tracker
41
-
42
-Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
43
-communicate with each other and in the same way slave services can communicate with each other.
44
-NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
45
-each other.
46
-
47
-- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
48
- track files, manage the file system and has the metadata of all the stored data within it. Some
49
- information the NameNode keep track of are:
50
-
51
- - details (metadata) of blocks
52
- - in which DataNode each block lives, and its location
53
- - replication metadata of each block
54
-
55
- The NameNode is the gateway that a client uses to manage the HDFS cluster.
56
-
57
-- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
58
- stores the actual data into HDFS which is responsible for the client to read and write. These are
59
- slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
60
- conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
61
- for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
62
- some other DataNode.
63
-
64
-- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
65
- is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
66
- NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
67
- `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
68
-
69
-- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
70
- tracker talks to the NameNode to know about the location of the data that will be used in
71
- processing. The NameNode responds with the metadata of the required processing data.
72
-
73
-- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
74
- Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
75
- the file. The process of applying that code on the file is known as Mapper.
76
-
77
-Some more useful information/concepts about HDFS from the official
78
-website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
79
-
80
-- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
81
- an application can create directories and store files inside these directories. The file system
82
- namespace hierarchy is similar to most other existing file systems; one can create and remove
83
- files, move a file from one directory to another, or rename a file. HDFS does not yet implement
84
- user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
85
- not preclude implementing these features.
86
-
87
- The NameNode maintains the file system namespace. Any change to the file system namespace or its
88
- properties is recorded by the NameNode. An application can specify the number of replicas of a
89
- file that should be maintained by HDFS. The number of copies of a file is called the replication
90
- factor of that file. This information is stored by the NameNode.
91
-
92
-- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
93
- HDFS are those that deal with large data sets. These applications write their data only once but
94
- they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
95
- supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
96
- Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
97
- different DataNode.
98
-
99
-- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
100
- might automatically move data from one DataNode to another if the free space on a DataNode falls
101
- below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
102
- might dynamically create additional replicas and rebalance other data in the cluster. These types
103
- of data rebalancing schemes are not yet implemented.
104
-
105
-</details>
106
-
107
-<details>
108
-<summary>References and sources</summary>
109
-
110
-1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
111
-2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
112
-3. [HDFS 3.3.1 commands guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html)
113
-
114
-</details>
115
-
116
-### Troubleshooting section
117
-
118
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
119
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
120
-
121
-<details>
122
-<summary>Identify which DataNode has a failing volume and troubleshoot it </summary>
123
-
124
-
125
-> For Apache Hadoop 3.3.1 <sup>[3](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html) </sup>
126
-
127
-1. Identify which DataNodes are offline, these are the sources of your problems:
128
-
129
- ```
130
- root@netdata # dfsadmin -report
131
- ```
132
-
133
- Identify which node is not reported in the output of the above command. If all the nodes are
134
- there, then you must run the next command for each DataNode.
135
-
136
-2. Review the volumes status:
137
-
138
- ```
139
- root@netdata # hdfs dfsadmin -getVolumeReport datanodehost:port
140
- ```
141
-
142
-
143
-3. Connect to the node and inspect the `journalctl -xe`. If you have the Netdata Agent running in
144
- the DataNodes as well, you should be able to identify the problem. You should be receiving alarms
145
- about the disks and the mounts in this system.
146
-
147
-</details>
148
-
health/guides/hdfs/hdfs_stale_nodes.md
deleted
-138
@@ -1,138 +0,0 @@
1
-# hdfs_stale_nodes
2
-
3
-**Storage | HDFS**
4
-
5
-_The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file system
6
-written in Java for the Hadoop framework. Some consider it to instead be a data store due to its
7
-lack of POSIX compliance, but it does provide shell commands and Java application programming
8
-interface (API) methods that are similar to other file
9
-systems._<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>
10
-
11
-Receiving this alert into warning indicates that there is at least one stale DataNode due to missed
12
-heartbeats.
13
-
14
-A stale DataNode is one that has not been reachable for `dfs.namenode.stale.datanode.interval` (
15
-default is 30 seconds). Stale DataNodes are avoided, and marked as the last possible target for a
16
-read or write operation. By default, HDFS marks a node as dead if it is unreachable for 630 seconds.
17
-
18
-
19
-<details>
20
-<summary>See more about Hadoop</summary>
21
-
22
-Wikipedia provides a great explanation of
23
-HDFS<sup>[1](https://en.wikipedia.org/wiki/Apache_Hadoop) </sup>. Here are the main takeaways:
24
-
25
-HDFS provides a software framework for distributed storage and processing of big data using the
26
-`MapReduce` programming model. HDFS is used for storing the data and `MapReduce` is used for
27
-processing data. It achieves reliability by replicating the data across multiple hosts, and hence
28
-theoretically does not require redundant array of independent disks (RAID) storage on hosts. With
29
-the default replication value, 3, data is stored on three nodes, two on the same rack, and one on a
30
-different rack. DataNodes can talk to each other to rebalance data, to move copies around, and to
31
-keep the replication of data high.
32
-
33
-HDFS has five services as follows:
34
-
35
-1. Name Node
36
-2. Secondary Name Node
37
-3. Job tracker
38
-4. Data Node
39
-5. Task Tracker
40
-
41
-Top three are master Services/Daemons/Nodes and bottom two are slave Services. Master Services can
42
-communicate with each other and in the same way slave services can communicate with each other.
43
-NameNode is a master node and DataNode(s) is its corresponding slave(s) node(s) and can talk with
44
-each other.
45
-
46
-- NameNode: HDFS consists of only one NameNode that is called the master node. The master node can
47
- track files, manage the file system and has the metadata of all the stored data within it. Some
48
- information the NameNode keep track of are:
49
-
50
- - details (metadata) of blocks
51
- - in which DataNode each block lives, and its location
52
- - replication metadata of each block
53
-
54
- The NameNode is the gateway that a client uses to manage the HDFS cluster.
55
-
56
-- DataNode: A DataNode stores data in it as blocks. This is also known as the Slave node and it
57
- stores the actual data into HDFS which is responsible for the client to read and write. These are
58
- slave daemons. Every DataNode sends a Heartbeat message to the NameNode every 3 seconds and
59
- conveys that it is alive. In this way when NameNode does not receive a heartbeat from a DataNode
60
- for 2 minutes, it will take that DataNode as dead and starts the process of block replications on
61
- some other DataNode.
62
-
63
-- Secondary NameNode: This is only to take care of the checkpoints of the file system metadata which
64
- is in the NameNode. This is also known as the checkpoint node. It is the helper node for the
65
- NameNode. The secondary NameNode instructs the NameNode to create and send an `fsimage` and
66
- `editlog` file. The secondary NameNode create a compacted `fsimage` file using these inputs.
67
-
68
-- Job Tracker: Job Tracker receives the requests for `MapReduce` execution from the client. Job
69
- tracker talks to the NameNode to know about the location of the data that will be used in
70
- processing. The NameNode responds with the metadata of the required processing data.
71
-
72
-- Task Tracker: It is the slave node for the Job Tracker and, it will take the task from the Job
73
- Tracker. It also receives code from the Job Tracker. Task Tracker will take the code and apply on
74
- the file. The process of applying that code on the file is known as Mapper.
75
-
76
-Some more useful information/concepts about HDFS from the official
77
-website <sup>[2](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) </sup> :
78
-
79
-- The File System Namespace: HDFS supports a traditional hierarchical file organization. A user or
80
- an application can create directories and store files inside these directories. The file system
81
- namespace hierarchy is similar to most other existing file systems; one can create and remove
82
- files, move a file from one directory to another, or rename a file. HDFS does not yet implement
83
- user quotas. HDFS does not support hard links or soft links. However, the HDFS architecture does
84
- not preclude implementing these features.
85
-
86
- The NameNode maintains the file system namespace. Any change to the file system namespace or its
87
- properties is recorded by the NameNode. An application can specify the number of replicas of a
88
- file that should be maintained by HDFS. The number of copies of a file is called the replication
89
- factor of that file. This information is stored by the NameNode.
90
-
91
-- Data Blocks: HDFS is designed to support very large files. Applications that are compatible with
92
- HDFS are those that deal with large data sets. These applications write their data only once but
93
- they read it one or more times and require these reads to be satisfied at streaming speeds. HDFS
94
- supports write-once-read-many semantics on files. A typical block size used by HDFS is 64 MB.
95
- Thus, an HDFS file is chopped up into 64 MB chunks, and if possible, each chunk will reside on a
96
- different DataNode.
97
-
98
-- Cluster Rebalancing: The HDFS architecture is compatible with data rebalancing schemes. A scheme
99
- might automatically move data from one DataNode to another if the free space on a DataNode falls
100
- below a certain threshold. In the event of a sudden high demand for a particular file, a scheme
101
- might dynamically create additional replicas and rebalance other data in the cluster. These types
102
- of data rebalancing schemes are not yet implemented.
103
-
104
-</details>
105
-
106
-
107
-<details>
108
-<summary>References and sources</summary>
109
-
110
-1. [Apache Hadoop on wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
111
-2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
112
-
113
-</details>
114
-
115
-### Troubleshooting section
116
-
117
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
118
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
119
-
120
-<details>
121
-<summary>Fix corrupted or missing blocks</summary>
122
-
123
-1. Identify the stale node(s)
124
-
125
- ```
126
- root@netdata # hadoop dfsadmin -report
127
- ```
128
-
129
-Inspect the output and check which DataNode is stale.
130
-
131
-2. Connect to the DataNode and check the log of the DataNode. You can also check for errors in the
132
- system services.
133
-
134
- ```
135
- root@netdata # systemctl status hadoop
136
- ```
137
-
138
- Restart the service if needed.
health/guides/hdfs_capacity_usage.md
new
+42
@@ -0,0 +1,42 @@
1
+### Understand the alert
2
+
3
+This alert calculates the percentage of used space capacity across all DataNodes in the Hadoop Distributed File System (HDFS). If you receive this alert, it means that your HDFS DataNodes space capacity utilization is high.
4
+
5
+The alert is triggered into warning when the percentage of used space capacity across all DataNodes is between 70-80% and in critical when it is between 80-90%.
6
+
7
+### Troubleshoot the alert
8
+
9
+Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is not liable for any loss or corruption of any data, database, or software.
10
+
11
+#### Check your Disk Usage across the cluster
12
+
13
+1. Inspect the Disk Usage for each DataNode:
14
+
15
+ ```
16
+ root@netdata # hadoop dfsadmin -report
17
+ ```
18
+
19
+ If all the DataNodes are in Disk pressure, you should consider adding more disk space. Otherwise, you can perform a balance of data between the DataNodes.
20
+
21
+2. Perform a balance:
22
+
23
+ ```
24
+ root@netdata # hdfs balancer –threshold 15
25
+ ```
26
+
27
+ This means that the balancer will balance data by moving blocks from over-utilized to under-utilized nodes, until each DataNode’s disk usage differs by no more than plus or minus 15 percent.
28
+
29
+#### Investigate high disk usage
30
+
31
+1. Review your Hadoop applications, jobs, and scripts that write data to HDFS. Identify the ones with excessive disk usage or logging.
32
+
33
+2. Optimize or refactor these applications, jobs, or scripts to reduce their disk usage.
34
+
35
+3. Delete any unnecessary or temporary files from HDFS, if safe to do so.
36
+
37
+4. Consider data compression or deduplication strategies, if applicable, to reduce storage usage in HDFS.
38
+
39
+### Useful resources
40
+
41
+1. [Apache Hadoop on Wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
42
+2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
\ No newline at end of file
health/guides/hdfs_dead_nodes.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the Alert
2
+
3
+The Netdata Agent monitors the number of DataNodes that are currently dead. Receiving this alert indicates that there are dead DataNodes in your HDFS cluster. The NameNode characterizes a DataNode as dead if no heartbeat message is exchanged for approximately 10 minutes. Any data that was registered to a dead DataNode is not available to HDFS anymore.
4
+
5
+This alert is triggered into critical when the number of dead DataNodes is 1 or more.
6
+
7
+### Troubleshoot the Alert
8
+
9
+1. Fix corrupted or missing blocks.
10
+
11
+ ```
12
+ root@netdata # hadoop dfsadmin -report
13
+ ```
14
+
15
+ Inspect the output and check which DataNode is dead.
16
+
17
+2. Connect to the DataNode and check the log of the DataNode. You can also check for errors in the system services.
18
+
19
+ ```
20
+ root@netdata # systemctl status hadoop
21
+ ```
22
+
23
+ Restart the service if needed.
24
+
25
+
26
+3. Verify that the network connectivity between NameNode and DataNodes is functional. You can use tools like `ping` and `traceroute` to confirm the connectivity.
27
+
28
+4. Check the logs of the dead DataNode(s) for any issues. Log location may vary depending on your installation, but you can typically find them in the `/var/log/hadoop-hdfs/` directory. Analyze the logs to identify any errors or issues that may have caused the DataNode to become dead.
29
+
30
+ ```
31
+ root@netdata # tail -f /var/log/hadoop-hdfs/hadoop-hdfs-datanode-*.log
32
+ ```
33
+
34
+5. If the DataNode service is not running or has crashed, attempt to restart it.
35
+
36
+ ```
37
+ root@netdata # systemctl restart hadoop
38
+ ```
39
+
40
+### Useful resources
41
+
42
+1. [Hadoop Commands Guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html)
43
+
44
+Remember that troubleshooting and resolving issues, especially on a production environment, requires a good understanding of the system and its architecture. Proceed with caution and always ensure data backup and environmental safety before performing any action.
health/guides/hdfs_missing_blocks.md
new
+47
@@ -0,0 +1,47 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of missing blocks in a Hadoop Distributed File System (HDFS). If you receive this alert, it means that there is at least one missing block in one of the DataNodes. This issue could be caused by a problem with the underlying storage or filesystem of a DataNode.
4
+
5
+### Troubleshooting the alert
6
+
7
+#### Fix corrupted or missing blocks
8
+
9
+Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is not liable for any loss or corruption of any data, database, or software.
10
+
11
+1. Identify which files are facing issues.
12
+
13
+```sh
14
+root@netdata # hdfs fsck -list-corruptfileblocks
15
+```
16
+
17
+Inspect the output and track the path(s) to the corrupted files.
18
+
19
+2. Determine where the file's blocks might live. If the file is larger than your block size, it consists of multiple blocks.
20
+
21
+```sh
22
+root@netdata # hdfs fsck <path_to_corrupted_file> -locations -blocks -files
23
+```
24
+
25
+This command will print out locations for every "problematic" block.
26
+
27
+3. Search in the corresponding DataNode and the NameNode's logs for the machine or machines on which the blocks lived. Try looking for filesystem errors on those machines. Use `fsck`.
28
+
29
+4. If there are files or blocks that you cannot fix, you must delete them so that the HDFS becomes healthy again.
30
+
31
+- For a specific file:
32
+
33
+```sh
34
+root@netdata # hdfs fs -rm <path_to_file_with_unrecoverable_blocks>
35
+```
36
+
37
+- For all the "problematic" files:
38
+
39
+```sh
40
+hdfs fsck / -delete
41
+```
42
+
43
+### Useful resources
44
+
45
+1. [Apache Hadoop on Wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
46
+2. [HDFS Architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
47
+3. [Man Pages of fsck](https://linux.die.net/man/8/fsck)
\ No newline at end of file
health/guides/hdfs_num_failed_volumes.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of failed volumes in your Hadoop Distributed File System (HDFS) cluster increases. A failed volume may be due to hardware failure or misconfiguration, such as duplicate mounts. When a single volume fails on a DataNode, the entire node may go offline depending on the `dfs.datanode.failed.volumes.tolerated` setting for your cluster. This can lead to increased network traffic and potential performance degradation as the NameNode needs to copy any under-replicated blocks lost on that node.
4
+
5
+### Troubleshoot the alert
6
+
7
+#### 1. Identify which DataNode has a failing volume
8
+
9
+Use the `dfsadmin -report` command to identify the DataNodes that are offline:
10
+
11
+```bash
12
+root@netdata # dfsadmin -report
13
+```
14
+
15
+Find any nodes that are not reported in the output of the command. If all nodes are listed, you'll need to run the next command for each DataNode.
16
+
17
+#### 2. Review the volumes status
18
+
19
+Use the `hdfs dfsadmin -getVolumeReport` command, specifying the DataNode hostname and port:
20
+
21
+```bash
22
+root@netdata # hdfs dfsadmin -getVolumeReport datanodehost:port
23
+```
24
+
25
+#### 3. Inspect the DataNode logs
26
+
27
+Connect to the affected DataNode and check its logs using `journalctl -xe`. If you have the Netdata Agent running on the DataNodes, you should be able to identify the problem. You may also receive alerts about the disks and mounts on this system.
28
+
29
+#### 4. Take necessary actions
30
+
31
+Based on the information gathered in the previous steps, take appropriate actions to resolve the issue. This may include:
32
+
33
+- Repairing or replacing faulty hardware.
34
+- Fixing misconfigurations such as duplicate mounts.
35
+- Ensuring that the HDFS processes are running on the affected DataNode.
36
+- Ensuring that the affected DataNode is properly communicating with the NameNode.
37
+
38
+**Note**: When working with HDFS, it's essential to have proper backups of your data. Netdata is not responsible for any loss or corruption of data, database, or software.
39
+
40
+### Useful resources
41
+
42
+1. [Apache Hadoop on Wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
43
+2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
44
+3. [HDFS 3.3.1 commands guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html)
health/guides/hdfs_stale_nodes.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+The `hdfs_stale_nodes` alert is triggered when there is at least one stale DataNode in the Hadoop Distributed File System (HDFS) due to missed heartbeats. A stale DataNode is one that has not been reachable for `dfs.namenode.stale.datanode.interval` (default is 30 seconds). Stale DataNodes are avoided and marked as the last possible target for a read or write operation.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify the stale node(s)
8
+
9
+ Run the following command to generate a report on the state of the HDFS cluster:
10
+
11
+ ```
12
+ hadoop dfsadmin -report
13
+ ```
14
+
15
+ Inspect the output and look for any stale DataNodes.
16
+
17
+2. Check the DataNode logs and system services status
18
+
19
+ Connect to the identified stale DataNode and check the log of the DataNode for any issues. Also, check the status of the system services.
20
+
21
+ ```
22
+ systemctl status hadoop
23
+ ```
24
+
25
+ If required, restart the HDFS service:
26
+
27
+ ```
28
+ systemctl restart hadoop
29
+ ```
30
+
31
+3. Monitor the HDFS cluster
32
+
33
+ After resolving issues identified in the logs or restarting the service, continue to monitor the HDFS cluster to ensure the problem is resolved. Re-run the `hadoop dfsadmin -report` command to check if the stale DataNode status has been cleared.
34
+
35
+4. Ensure redundant data storage
36
+
37
+ To protect against data loss or unavailability, HDFS stores data in multiple nodes, providing fault tolerance. Make sure that the replication factor for your HDFS cluster is set correctly, typically with a factor of 3, so that data is stored on three different nodes. A higher replication factor will increase data redundancy and reliability.
38
+
39
+5. Review HDFS cluster configuration
40
+
41
+ Examine the HDFS cluster's configuration settings to ensure that they are appropriate for your specific use case and hardware setup. Identifying performance bottlenecks, such as slow or unreliable network connections, can help avoid stale DataNodes in the future.
42
+
43
+### Useful resources
44
+
45
+1. [Apache Hadoop on Wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
46
+2. [HDFS Architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html)
\ No newline at end of file
health/guides/httpcheck/httpcheck_web_service_bad_content.md
deleted
-45
@@ -1,45 +0,0 @@
1
-# httpcheck_web_service_bad_content
2
-
3
-**Web Server | HTTP endpoint**
4
-
5
-The Netdata Agent monitors your HTTP endpoints. You can specify endpoints that the agent will monitor in
6
-Agent's Go module under `go.d/httpcheck.conf`. You can also specify the expected response pattern.
7
-This HTTP endpoint will send in the `response_match` option.
8
-<sup>[1](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck) </sup>
9
-If the endpoint's response does not match the `response_match` pattern, then the Agent marks the response
10
-as unexpected.
11
-
12
-The Netdata Agent calculates the average ratio of HTTP responses with unexpected content over the
13
-last 5 minutes.
14
-
15
-This alert is escalated to warning if the percentage of unexpected content is greater than 10%
16
-and then raised to critical if it is greater than 40%.
17
-
18
-<details>
19
-<summary>References and Sources</summary>
20
-
21
-1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
22
-
23
-</details>
24
-
25
-### Troubleshooting:
26
-
27
-<details>
28
-<summary>Check the actual response and the expected response.</summary>
29
-
30
-1. Try to implement a request with a verbose result:
31
-
32
-```
33
-root@netdata # curl -v <your_http_endpoint>:<port>/<path>
34
-```
35
-
36
-2. Compare it with the expected response.
37
-
38
-Check your configuration under `go.d/httpcheck.conf`:
39
-
40
-```
41
-cd /etc/netdata # Replace this path with your Netdata config directory
42
-sudo ./edit-config go.d/httpcheck.conf
43
-```
44
-
45
-</details>
health/guides/httpcheck/httpcheck_web_service_bad_status.md
deleted
-46
@@ -1,46 +0,0 @@
1
-# httpcheck_web_service_bad_status
2
-
3
-**Web Server | HTTP endpoint**
4
-
5
-The Netdata agent monitors your HTTP endpoints. You can specify endpoints the agent will monitor in
6
-Agent's Go module under `go.d/httpcheck.conf`. You can also specify the expected response statuses
7
-this HTTP endpoint must reply with, in the `status_accepted` option.
8
-<sup>[1](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck) </sup>
9
-If the endpoint responds with a response status that is not in the specified `status_accepted` codes, the Agent
10
-marks the response as "bad_status".
11
-
12
-The Netdata Agent calculates the average ratio of these unexpected (bad) HTTP status responses over
13
-the last 5 minutes.
14
-
15
-This alert is triggered in warning state when the ratio is greater than 10% and in critical state
16
-when it is greater than 40%.
17
-
18
-<details>
19
-<summary>References and sources</summary>
20
-
21
-1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
22
-
23
-</details>
24
-
25
-### Troubleshooting section:
26
-
27
-<details>
28
-<summary>Check the actual response status and the expected response statuses</summary>
29
-
30
-1. Try to implement a request with a verbose result:
31
-
32
-```
33
-root@netdata # curl -v <your_http_endpoint>:<port>/<path>
34
-```
35
-
36
-2. Compare it with the expected response
37
-
38
-Check your configuration under `go.d/httpcheck.conf` which are the `status_accepted` codes for this
39
-particular endpoint.
40
-
41
-```
42
-cd /etc/netdata # Replace this path with your Netdata config directory
43
-sudo ./edit-config go.d/httpcheck.conf
44
-```
45
-
46
-</details>
health/guides/httpcheck/httpcheck_web_service_slow.md
deleted
-32
@@ -1,32 +0,0 @@
1
-# httpcheck_web_service_slow
2
-
3
-**Web Server | HTTP Endpoints**
4
-
5
-The Netdata Agent monitors your HTTP endpoints. You can specify endpoints the Agent will monitor in
6
-the Agent's Go module under `go.d/httpcheck.conf`.
7
-<sup>[1](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck) </sup>
8
-The Agent calculates the average response time for every HTTP request made to the endpoint being monitored per hour.
9
-The Agent also calculates the average response time in a 3-min window.
10
-
11
-The Netdata Agent compares these two (average) values. If there is a significant increase in 3-min average, then it
12
-will trigger a warning alert when the response time 3-min average is at least twice as much as 1-hour average. The alert
13
-will escalate to critical when the response time 3-min average reaches three times the average amount per hour.
14
-
15
-<details>
16
-<summary>References and Sources</summary>
17
-
18
-1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
19
-
20
-</details>
21
-
22
-### Troubleshooting section:
23
-
24
-<details>
25
-<summary>General Approach</summary>
26
-
27
-To troubleshoot this issue, check for:
28
-
29
-- Network congestion in your system's network and/or in the remote endpoint's network.
30
-- If the endpoint is managed by you, then check the system load.
31
-
32
-</details>
health/guides/httpcheck/httpcheck_web_service_unreachable.md
deleted
-50
@@ -1,50 +0,0 @@
1
-# httpcheck_web_service_unreachable
2
-
3
-**Web Server | HTTP endpoint**
4
-
5
-The Netdata agent monitors your HTTP endpoints. You can specify endpoints the Agent will monitor in
6
-the Agent's Go module under `go.d/httpcheck.conf`.
7
-<sup>[1](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck) </sup>
8
-If your system fails to connect to your endpoint, or if the request to that endpoint times out, then
9
-the Agent will mark the requests and log them as "unreachable".
10
-
11
-The Netdata Agent calculates the ratio of these requests over the last 5 minutes.
12
-
13
-This alert is escalated to warning when the ratio is greater than 10% and then raised
14
-to critical when it is greater than 40%.
15
-
16
-<details>
17
-<summary>References and Sources</summary>
18
-
19
-1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
20
-
21
-</details>
22
-
23
-### Troubleshooting section:
24
-
25
-<details>
26
-<summary>General Approach</summary>
27
-
28
-To troubleshoot this error, check the following:
29
-
30
-- Verify that your system has access to the particular endpoint.
31
-
32
- - Check for basic connectivity to known hosts.
33
- - Make sure that requests and replies both to and from the endpoint are allowed in the firewall settings.
34
- Ensure they're allowed on both your end as well as the endpoint's side.
35
-
36
-- Verify that your DNS can resolve endpoints.
37
- - Check your current DNS (for example in linux you can use the host command):
38
-
39
- ```
40
- root@netdata # host -v <your_endpoint>
41
- ```
42
-
43
- - If the HTTP endpoint is suppose to be public facing endpoint, try an alternative DNS (for
44
- example Cloudflare's DNS):
45
-
46
- ```
47
- root@netdata # host -v <your_endpoint> 1.1.1.1
48
- ```
49
-
50
-</details>
health/guides/httpcheck_web_service_bad_content.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+The Netdata Agent monitors your HTTP endpoints. You can specify endpoints that the agent will monitor in Agent's Go module under `go.d/httpcheck.conf`. You can also specify the expected response pattern. This HTTP endpoint will send in the `response_match` option. If the endpoint's response does not match the `response_match` pattern, then the Agent marks the response as unexpected.
4
+
5
+The Netdata Agent calculates the average ratio of HTTP responses with unexpected content over the last 5 minutes.
6
+
7
+This alert is escalated to warning if the percentage of unexpected content is greater than 10% and then raised to critical if it is greater than 40%.
8
+
9
+### Troubleshoot the alert
10
+
11
+Check the actual response and the expected response.
12
+
13
+1. Try to implement a request with a verbose result:
14
+
15
+```
16
+curl -v <your_http_endpoint>:<port>/<path>
17
+```
18
+
19
+2. Compare it with the expected response.
20
+
21
+Check your configuration under `go.d/httpcheck.conf`:
22
+
23
+```
24
+cd /etc/netdata # Replace this path with your Netdata config directory
25
+sudo ./edit-config go.d/httpcheck.conf
26
+```
27
+
28
+### Useful resources
29
+
30
+1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
\ No newline at end of file
health/guides/httpcheck_web_service_bad_status.md
new
+21
@@ -0,0 +1,21 @@
1
+### Understand the alert
2
+
3
+The `httpcheck_web_service_bad_status` alert is generated by the Netdata Agent when monitoring the status of an HTTP web service using the `httpcheck` collector. This alert is triggered when the HTTP web service returns a non-successful status code (anything other than 2xx or 3xx), indicating that there is an issue with the web service, preventing it from responding to requests as expected.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Verify the target URL**: Ensure that the target URL configured in the `httpcheck` collector is correct and accessible. Check for any typos or incorrect domain names.
8
+
9
+2. **Check the actual response status and the expected response status**: Try to implement a request with a verbose result:
10
+
11
+```
12
+root@netdata # curl -v <your_http_endpoint>:<port>/<path>
13
+```
14
+
15
+3. **Verify server resources**: Ensure that your server has enough resources (CPU, RAM, disk space) to handle the current workload. High resource utilization can lead to web service issues. You can use Netdata's dashboard to monitor the server resources in real-time.
16
+
17
+4. **Check server configuration**: Review the configuration files of the web service for any misconfigurations or settings that may be causing the issue. For example, incorrect permissions, wrong file paths, or improper configurations can lead to bad status codes.
18
+
19
+### Useful resources
20
+
21
+1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
health/guides/httpcheck_web_service_no_connection.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+This alert monitors the percentage of failed HTTP requests to a specific URL in the last 5 minutes. If you receive this alert, it means that your web service experienced connection issues.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Verify HTTP service status
8
+
9
+Check if the web service is running and accepting requests. If the service is down, restart it and monitor the situation.
10
+
11
+2. Review server logs
12
+
13
+Examine the logs of the web server hosting the HTTP service. Look for any errors or warning messages that may provide more information about the cause of the connection issues.
14
+
15
+3. Check network connectivity
16
+
17
+If the server hosting the HTTP service is experiencing connectivity issues, it can lead to failed requests. Ensure that the server has stable network connectivity.
18
+
19
+4. Monitor server resources
20
+
21
+Inspect the server's resource usage to check if it is running out of resources, such as CPU, memory, or disk space. If the server is running low on resources, it can cause the HTTP service to malfunction. In this case, free up resources or upgrade the server.
22
+
23
+5. Review client connections
24
+
25
+It is also possible that the clients are having connectivity issues. Make sure that the clients are in a good network condition and can connect to the server without any issues.
26
+
27
+6. Test the HTTP service
28
+
29
+Perform HTTP requests to the service manually or using monitoring tools to measure response times and verify if the issue persists.
30
+
31
+### Useful resources
32
+
33
+1. [Apache Log Files](https://httpd.apache.org/docs/2.4/logs.html)
34
+2. [NGINX Log Files](https://docs.nginx.com/nginx/admin-guide/monitoring/logging/)
35
+3. [HTTP status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
health/guides/httpcheck_web_service_slow.md
new
+18
@@ -0,0 +1,18 @@
1
+### Understand the alert
2
+
3
+The Netdata Agent monitors your HTTP endpoints. You can specify endpoints the Agent will monitor in the Agent's Go module under `go.d/httpcheck.conf`.
4
+The Agent calculates the average response time for every HTTP request made to the endpoint being monitored per hour. The Agent also calculates the average response time in a 3-min window.
5
+
6
+The Netdata Agent compares these two (average) values. If there is a significant increase in 3-min average, then it will trigger a warning alert when the response time 3-min average is at least twice as much as 1-hour average. The alert will escalate to critical when the response time 3-min average reaches three times the average amount per hour.
7
+
8
+### Troubleshoot the alert
9
+
10
+To troubleshoot this issue, check for:
11
+
12
+- Network congestion in your system's network and/or in the remote endpoint's network.
13
+- If the endpoint is managed by you, then check the system load.
14
+
15
+### Useful resources
16
+
17
+1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
18
+
health/guides/httpcheck_web_service_timeouts.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the percentage of timed-out HTTP requests to a specific URL goes above a certain threshold in the last 5 minutes. The alert levels are determined by the following percentage thresholds:
4
+
5
+- Warning: 10% to 40%
6
+- Critical: 40% or higher
7
+
8
+The alert is designed to notify you about potential issues with the accessed HTTP endpoint.
9
+
10
+### What does HTTP request timeout mean?
11
+
12
+An HTTP request timeout occurs when a client (such as a web browser) sends a request to a webserver but does not receive a response within the specified time period. This can lead to a poor user experience, as the user may be unable to access the requested content or services.
13
+
14
+### Troubleshoot the alert
15
+
16
+- Verify the issue
17
+
18
+Check the HTTP endpoint to see if it is responsive and reachable. You can use tools like `curl` or online services like [https://www.isitdownrightnow.com/](https://www.isitdownrightnow.com/) to check the availability of the website or service.
19
+
20
+- Analyze server logs
21
+
22
+Examine the server logs for any error messages or unusual patterns of behavior that may indicate a root cause for the timeout issue. For web servers such as Apache or Nginx, look for log files located in the `/var/log` directory.
23
+
24
+- Check resource usage
25
+
26
+High resource usage, such as CPU, memory, or disk I/O, can cause HTTP request timeouts. Use tools like `top`, `vmstat`, or `iotop` to identify resource-intensive processes. Address any performance bottlenecks by resizing the server, optimizing performance, or distributing the load across multiple servers.
27
+
28
+- Review server configurations
29
+
30
+Make sure your web server configurations are optimized for performance. For instance:
31
+
32
+ 1. Ensure that the `KeepAlive` feature is enabled and properly configured.
33
+ 2. Make sure that your server's timeout settings are appropriate for the type of traffic and workload it experiences.
34
+ 3. Confirm that your server is correctly configured for the number of concurrent connections it handles.
35
+
36
+- Verify network configurations
37
+
38
+Examine the network configurations for potential issues that can lead to HTTP request timeouts. Check for misconfigured firewalls or faulty load balancers that may be interfering with traffic to the HTTP endpoint.
39
+
health/guides/httpcheck_web_service_unreachable.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+The Netdata agent monitors your HTTP endpoints. You can specify endpoints the Agent will monitor in the Agent's Go module under `go.d/httpcheck.conf`.
4
+
5
+If your system fails to connect to your endpoint, or if the request to that endpoint times out, then the Agent will mark the requests and log them as "unreachable".
6
+
7
+The Netdata Agent calculates the ratio of these requests over the last 5 minutes. This alert is escalated to warning when the ratio is greater than 10% and then raised to critical when it is greater than 40%.
8
+
9
+### Troubleshoot the alert
10
+
11
+To troubleshoot this error, check the following:
12
+
13
+- Verify that your system has access to the particular endpoint.
14
+
15
+ - Check for basic connectivity to known hosts.
16
+ - Make sure that requests and replies both to and from the endpoint are allowed in the firewall settings. Ensure they're allowed on both your end as well as the endpoint's side.
17
+
18
+- Verify that your DNS can resolve endpoints.
19
+ - Check your current DNS (for example in linux you can use the host command):
20
+
21
+ ```
22
+ host -v <your_endpoint>
23
+ ```
24
+
25
+ - If the HTTP endpoint is suppose to be public facing endpoint, try an alternative DNS (for example Cloudflare's DNS):
26
+
27
+ ```
28
+ host -v <your_endpoint> 1.1.1.1
29
+ ```
30
+
31
+### Useful resources
32
+
33
+1. [HTTP endpoint monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/httpcheck)
\ No newline at end of file
health/guides/httpcheck_web_service_up.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+The `httpcheck_web_service_up` alert monitors the liveness status of an HTTP endpoint by checking its response over the past minute. If the success percentage is below 75%, this alert will trigger, indicating that the web service may be experiencing issues.
4
+
5
+### What does an HTTP endpoint liveness status mean?
6
+
7
+An HTTP endpoint is like a door where clients make requests to access web services or APIs. The liveness status reveals whether the service is available and responding to client requests. Ideally, this success percentage should be near 100%, indicating that the endpoint is consistently accessible.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check logs for any errors or warnings related to the web server or application.
12
+
13
+ Depending on your web server or application, look for log files that may provide insights into the causes of the issues. Some common log locations are:
14
+
15
+ - Apache: `/var/log/apache2/`
16
+ - Nginx: `/var/log/nginx/`
17
+ - Node.js: Check your application-specific log location.
18
+
19
+2. Examine server resources such as CPU, memory, and disk usage.
20
+
21
+ High resource usage can cause web services to become slow or unresponsive. Use system monitoring tools like `top`, `htop`, or `free` to check the resource usage.
22
+
23
+3. Test the HTTP endpoint manually.
24
+
25
+ You can use tools like `curl`, `wget`, or `httpie` to send requests to the HTTP endpoint and inspect the responses. Examine the response codes, headers, and contents to spot any problems.
26
+
27
+ Example using `curl`:
28
+
29
+ ```
30
+ curl -I http://example.com/some/endpoint
31
+ ```
32
+
33
+4. Check for network issues between the monitoring agent and the HTTP endpoint.
34
+
35
+ Use tools like `ping`, `traceroute`, or `mtr` to check for network latency or packet loss between the monitoring agent and the HTTP endpoint.
36
+
37
+5. Review the web server or application configuration.
38
+
39
+ Ensure the web server and application configurations are correct and not causing issues. Look for misconfigurations, incorrect settings, or other issues that may affect the liveness of the HTTP endpoint.
40
+
41
+### Useful resources
42
+
43
+1. [Monitoring Linux Performance with vmstat and iostat](https://www.tecmint.com/linux-performance-monitoring-with-vmstat-and-iostat-commands/)
44
+2. [16 Useful Bandwidth Monitoring Tools to Analyze Network Usage in Linux](https://www.tecmint.com/linux-network-bandwidth-monitoring-tools/)
health/guides/inbound_packets_dropped.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of inbound dropped packets for a network interface exceeds a specified threshold during the last 10 minutes. A dropped packet means that the network device could not process the packet, hence it was discarded.
4
+
5
+### What are the common causes of dropped packets?
6
+
7
+1. Network Congestion: When the network traffic is too high, the buffer may overflow before the device can process the packets, causing some packets to be dropped.
8
+2. Link Layer Errors: Packets can be dropped due to errors in the link layer causing frames to be corrupted.
9
+3. Insufficient Resources: The network interface may fail to process incoming packets due to a lack of memory or CPU resources.
10
+
11
+### Troubleshoot the alert
12
+
13
+1. Check the overall system resources
14
+
15
+ Run the `vmstat` command to get a report about your system statistics.
16
+
17
+ ```
18
+ vmstat 1
19
+ ```
20
+
21
+ Check if the CPU or memory usage is high. If either is near full utilization, consider upgrading system resources or managing the load more efficiently.
22
+
23
+2. Check network interface statistics
24
+
25
+ Run the `ifconfig` command to get more information on the network interface.
26
+
27
+ ```
28
+ ifconfig <INTERFACE>
29
+ ```
30
+
31
+ Look for the `RX dropped` field to confirm the number of dropped packets.
32
+
33
+3. Monitor network traffic
34
+
35
+ Use `iftop` or `nload` to monitor the network traffic in real time. If you don't have these tools, install them:
36
+
37
+ ```
38
+ sudo apt install iftop nload
39
+ ```
40
+
41
+ ```
42
+ iftop -i <INTERFACE>
43
+ nload <INTERFACE>
44
+ ```
45
+
46
+ Identify if there is unusually high traffic on the network interface.
47
+
48
+4. Check logs for any related errors
49
+
50
+ Check the system logs for any errors related to the network interface or driver:
51
+
52
+ ```
53
+ sudo dmesg | grep -i "eth0"
54
+ sudo journalctl -u networking.service
55
+ ```
56
+
57
+ If you find any errors, you can research the specific problem and apply the necessary fixes.
58
+
health/guides/inbound_packets_dropped_ratio.md
new
+52
@@ -0,0 +1,52 @@
1
+### Understand the alert
2
+
3
+Packet drops indicate that your system received some packets but could not process them. A sizeable amount of packet drops can consume significant amount of resources in your system. Some reasons that packets drops occurred in your system could be:
4
+
5
+- Your system receives packets with bad VLAN tags.
6
+- The packets you are receiving are using a protocol that is unknown to your system.
7
+- You receive IPv6 packets, but your system is not configured for IPv6.
8
+
9
+All these packets consume resources until being dropped (and for a short period after). For example, your NIC stores them in a ring-buffer until they are forwarded to the destined subsystem or userland application for further process.
10
+
11
+Netdata calculates the ratio of inbound dropped packets for your wired network interface over the last 10 minutes.
12
+
13
+### Identify VLANs in your interface
14
+
15
+There are cases in which traffic is routed to your host due to the existence of multiple VLAN in your network.
16
+
17
+1. Identify VLAN tagged packet in your interface.
18
+
19
+```
20
+tcpdump -i <your_interface> -nn -e vlan
21
+```
22
+
23
+2. Monitor the output of the `tcpdump`, identify VLANs which may exist. If no output is displayed, your interface probably uses traditional ethernet frames.
24
+
25
+3. Depending on your network topology, you may consider removing unnecessary VLANs from the switch trunk port toward your host.
26
+
27
+### Update the ring buffer size on your interface
28
+
29
+1. To view the maximum RX ring buffer size:
30
+
31
+ ```
32
+ ethtool -g enp1s0
33
+ ```
34
+
35
+2. If the values in the Pre-set maximums section are higher than in the current hardware settings section, increase RX
36
+ ring buffer:
37
+
38
+ ```
39
+ enp1s0 rx 4080
40
+ ```
41
+
42
+3. Verify the change to make sure that you no longer receive the alarm when running the same workload. To make this
43
+ permanently, you must consult your distribution guides.
44
+
45
+
46
+### Inspect the packets your network interface receives
47
+
48
+Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development.
49
+
50
+### Useful resources
51
+
52
+[Read more about Wireshark here](https://www.wireshark.org/)
\ No newline at end of file
health/guides/interface_inbound_errors.md
new
+36
@@ -0,0 +1,36 @@
1
+- Troubleshoot errors related to network congestion
2
+
3
+Network congestion can cause packets to be dropped, leading to interface inbound errors. To determine if congestion is the issue, you can monitor the network for any signs of excessive workload or high utilization rates.
4
+
5
+1. Use `ifconfig` to check the network interface utilization:
6
+ ```
7
+ ifconfig <your_interface>
8
+ ```
9
+
10
+2. Check the network switch/router logs for any indication of high utilization, errors or warnings.
11
+
12
+3. Use monitoring tools like `iftop`, `nload`, or `iptraf` to monitor network traffic and identify any bottle-necks or usage spikes.
13
+
14
+If you find that congestion is causing the inbound errors, consider ways to alleviate the issue including upgrading your network infrastructure or load balancing the traffic.
15
+
16
+- Troubleshoot errors caused by faulty network equipment
17
+
18
+Faulty network devices, such as switches and routers, can introduce errors in packets. To identify the cause, you should review the logs and statistics of any network devices in the path of the communication between the sender and this system.
19
+
20
+1. Check the logs of the network equipment for any indications of errors, problems or unusual behavior.
21
+
22
+2. Review the error counters and statistics of the network equipment to identify any trends or issues.
23
+
24
+3. Consider replacing or upgrading faulty equipment if it is found to be responsible for inbound errors.
25
+
26
+- Troubleshoot errors caused by software or configuration issues
27
+
28
+Incorrect configurations or software issues can also contribute to interface inbound errors. Some steps to troubleshoot these potential causes are:
29
+
30
+1. Review the system logs for any errors or warnings related to the network subsystem.
31
+
32
+2. Ensure that the network interface is configured correctly, and proper drivers are installed and up-to-date.
33
+
34
+3. Examine the system's firewall and security settings to verify that there are no inappropriate blockings or restrictions that may be causing the errors.
35
+
36
+In conclusion, by following these troubleshooting steps, you should be able to identify and resolve the cause of interface inbound errors on your FreeBSD system. Remember to monitor the situation regularly and address any new issues that may arise to ensure a stable and efficient networking environment.
\ No newline at end of file
health/guides/interface_outbound_errors.md
new
+42
@@ -0,0 +1,42 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when there is a high number of outbound errors on a specific network interface in the last 10 minutes on a FreeBSD system. When you receive this alert, it means that the network interface is facing transmission-related issues, such as aborted, carrier, FIFO, heartbeat, or window errors.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify the network interface with the problem
8
+ Use `ifconfig` to get a list of all network interfaces and their error count:
9
+ ```
10
+ ifconfig -a
11
+ ```
12
+ Check the "Oerrs" (Outbound errors) field for each interface to find the one with the issue.
13
+
14
+2. Check the interface speed and duplex settings
15
+ The speed and duplex settings may mismatch between the network interface and the network equipment (like switches and routers) that it is connected to. Use `ifconfig` or `ethtool` to check these settings.
16
+
17
+ With `ifconfig`:
18
+ ```
19
+ ifconfig <interface_name>
20
+ ```
21
+
22
+ If required, adjust the speed and duplex settings using `ifconfig`:
23
+ ```
24
+ ifconfig <interface_name> media <media_type>
25
+ ```
26
+ `<media_type>` can be one of the following: 10baseT/UTP, 100baseTX, 1000baseTX, etc., and can include half-duplex or full-duplex.
27
+ Example:
28
+ ```
29
+ ifconfig em0 media 1000baseTX mediaopt full-duplex
30
+ ```
31
+ Ensure both the network interface and the connected device use the same settings.
32
+
33
+3. Check network cables and devices
34
+ Check the physical connections of the network cable to both the network interface and the network equipment it connects to. Replace the network cable if necessary. Additionally, verify if the issue is related to the connected network equipment (switches and routers).
35
+
36
+4. Analyze network traffic
37
+ Use tools like `tcpdump` or `Wireshark` to analyze the network traffic on the affected interface. This can give you insights into the root cause of the errors and help in troubleshooting device or network-related issues.
38
+
39
+### Useful resources
40
+
41
+1. [FreeBSD ifconfig man page](https://www.freebsd.org/cgi/man.cgi?ifconfig(8))
42
+2. [FreeBSD Handbook - Configuring the Network](https://www.freebsd.org/doc/handbook/config-network-setup.html)
health/guides/interface_speed.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+This alert indicates the current speed of the network interface `${label:device}`. If you receive this alert, it means that there is a significant change or reduction in the speed of your network interface.
4
+
5
+### What does interface speed mean?
6
+
7
+Interface speed refers to the maximum throughput an interface (network card or adapter) can support in terms of transmitting and receiving data. It is measured in Megabits per second (Mbit/s) and determines the performance of a network connection.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check the network interface speed.
12
+
13
+To see the interface speed and other information about the network interface, run the following command in the terminal:
14
+
15
+```
16
+ethtool ${label:device}
17
+```
18
+
19
+Replace `${label:device}` with your network interface name, e.g., `eth0` or `enp2s0`.
20
+
21
+- Confirm if there is a network congestion issue.
22
+
23
+High network traffic or congestion might cause reduced interface speed. Use the `iftop` utility to monitor the traffic on the network interface. If you don't have `iftop` installed, then [install it](https://www.binarytides.com/linux-commands-monitor-network/).
24
+
25
+Run the following command in the terminal:
26
+
27
+```
28
+sudo iftop -i ${label:device}
29
+```
30
+
31
+Replace `${label:device}` with your network interface name.
32
+
33
+- Verify cable connections and quality.
34
+
35
+Physical cable issues might cause reduced speed in the network interface. Check the connections and quality of the cables connecting your system to the network devices such as routers, switches, or hubs.
36
+
37
+- Update network drivers.
38
+
39
+Outdated network drivers can also lead to reduced speed in the network interface. Update the network drivers to the latest version to avoid any compatibility issues or performance degradations.
40
+
41
+- Check for EMI (Electromagnetic Interference).
42
+
43
+Network cables and devices located near power cables or electronic devices producing electromagnetic fields might experience reduced network interface speed. Make sure that your network cables and devices are not in proximity to potential sources of EMI.
44
+
health/guides/ioping/ioping_disk_latency.md
deleted
-36
@@ -1,36 +0,0 @@
1
-# ioping_disk_latency
2
-
3
-## OS: Any
4
-
5
-This alarm presents the average `I/O latency` over the last 10 seconds.
6
-
7
-> `I/O latency` is the time that is required to complete a single I/O operation on a block device.
8
-
9
-If this alarm is raised, it might indicate that your disk is under high load, or that the disk is slow.
10
-
11
-### Troubleshooting Section
12
-
13
-<details>
14
-<summary>Check related charts to find your case</summary>
15
-
16
-- First, you need to identify whether your disk is under high load or not.
17
-
18
- <br>
19
-
20
- 1. Go to your node on the Netdata Cloud, on the `Disks` section, and select the disk you want to
21
- investigate.
22
-
23
- <br>
24
-
25
- 2. On the top of the page you can see the utilization of the disk, and you can also go to the `Disk I/O Bandwidth`
26
- chart. There you can see the amount of transferred data `to` and `from` the particular disk.
27
-
28
- <br>
29
-
30
-- If the utilization is low and there isn't a significant amount of data being transferred, your
31
- drive's latency is slow, so you need to upgrade that drive. (If it is an HDD, then an SSD would be a significant upgrade
32
- for latency issues.)
33
-
34
-
35
-- If there is a high load on the drive, it usually means that one or more processes are heavily utilizing the drive.
36
-</details>
\ No newline at end of file
health/guides/ioping_disk_latency.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+This alert presents the average `I/O latency` over the last 10 seconds. `I/O latency` is the time that is required to complete a single I/O operation on a block device.
4
+
5
+This alert might indicate that your disk is under high load, or that the disk is slow.
6
+
7
+### Troubleshoot the alert
8
+
9
+1. Check per-process I/O usage:
10
+
11
+ Use `iotop` to see the processes that are the main I/O consumers:
12
+
13
+ ```
14
+ sudo iotop
15
+ ```
16
+
17
+ If you don't have `iotop` installed, then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
18
+
19
+2. Analyze the running processes:
20
+
21
+ Investigate the top I/O consumers and determine if these processes are expected to consume that much I/O, or if there might be an issue with these processes.
22
+
23
+3. Minimize the load by closing any unnecessary main consumer processes:
24
+
25
+ If you find that any unnecessary or unexpected processes are heavily utilizing your disk, try stopping or closing those processes to reduce the load on the disk. Always double-check if the process you want to close is necessary.
26
+
27
+4. Verify your disk health:
28
+
29
+ Make sure your disk is not facing any hardware issues or failures. For this, you can use the `smartmontools` package, which contains the `smartctl` utility. If it's not installed, you can [install it](https://www.smartmontools.org/wiki/Download).
30
+
31
+ To check the disk health, run:
32
+
33
+ ```
34
+ sudo smartctl -a /dev/sdX
35
+ ```
36
+
37
+ Replace `/dev/sdX` with the correct disk device identifier (for example, `/dev/sda`).
38
+
39
+5. Consider upgrading your disk:
40
+
41
+ If your disk consistently experiences high latency and you have already addressed any performance issues with the running processes, consider upgrading your disk to a faster drive (e.g., replace an HDD with an SSD).
42
+
43
+### Useful resources
44
+
45
+1. [iotop - Monitor Linux Disk I/O Activity](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
46
+2. [smartmontools - SMART monitoring tools](https://www.smartmontools.org/)
health/guides/ipc/semaphore_arrays_used.md
deleted
-76
@@ -1,76 +0,0 @@
1
-# semaphores_arrays_used
2
-
3
-## OS: Linux
4
-
5
-This alert presents the percentage of allocated `System V IPC semaphore arrays (sets)`. \
6
-If you receive this alert, it means that your system is experiencing high `IPC semaphore arrays` utilization and a lack
7
-of available semaphore arrays can affect application performance.
8
-
9
-<details>
10
-<summary>What is an "IPC Semaphore array (or set)"</summary>
11
-
12
-"IPC" stands for "Interprocess Communication". IPC messages are a counterpart to UNIX pipes for IPC operations. \
13
-The fastest way to communicate through processes is with shared memory. semaphores, help synchronise shared memory
14
-access across processes.<sup> [1](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html) </sup>
15
-
16
-System V semaphores are allocated in groups called sets. \
17
-A "semaphore set" consists of a control structure and an array of individual semaphores.
18
-
19
-As illustrated by E. W. Dijkstra, semaphores can be better understood using his railroad model <sup> [2](
20
-https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html) </sup>.
21
-
22
-- Imagine a railroad, where only a single train at a time is allowed to pass. Responsible for the traffic is a
23
- **semaphore**. Each train that wants to enter the single track must wait for the **semaphore** to be in a state that
24
- allows access to the railroad. When a train enters the track, the **semaphore** changes the state to prevent all other
25
- traffic in the track. When the train leaves the railroad, it must change the state of the **semaphore** to allow another
26
- train to enter.
27
-
28
-
29
-- In the computer world, a **semaphore** is an integer and the train is a process (or a thread). For a process to proceed,
30
- it has to wait for the semaphore's value to become 0. If it proceeds, it increments this value by 1. Upon finishing
31
- the task, the process decrements the same value by 1.
32
-
33
-> **semaphores** let processes query or alter status information. They are often used to monitor and control the
34
-> availability of system resources such as shared memory segments.
35
-> <sup> [2](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html) </sup>
36
-
37
-
38
-
39
-</details>
40
-
41
-<br>
42
-
43
-<details>
44
-<summary>References and Sources</summary>
45
-
46
-[[1] Interprocess Communication](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html) \
47
-[[2] IPC:Semaphores](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html)
48
-
49
-</details>
50
-
51
-### Troubleshooting Section
52
-
53
-<details>
54
- <summary>Adjust the semaphore limit on your system</summary>
55
-
56
-You can check current `semaphore arrays` limit on your machine, by running:
57
-
58
-```
59
-root@netdata~ # ipcs -ls
60
-```
61
-
62
-The output will be similar to this:
63
-
64
-```
65
------- Semaphore Limits --------
66
-max number of arrays = 32000
67
-max semaphores per array = 32000
68
-max semaphores system wide = 1024000000
69
-max ops per semop call = 500
70
-semaphore max value = 32767
71
-```
72
-
73
-To adjust the limit of the max semaphores,you can go to the `/proc/sys/kernel/sem` file and adjust the fourth field
74
-accordingly.
75
-
76
-</details>
health/guides/ipc/semaphores_used.md
deleted
-72
@@ -1,72 +0,0 @@
1
-# semaphores_used
2
-
3
-## OS: Linux
4
-
5
-This alarm presents the percentage of allocated `System V IPC semaphores`. \
6
-If you receive this alarm, it means that your system is experiencing high `IPC semaphore` utilization and a lack of
7
-available `semaphores` can affect application performance.
8
-
9
-<details>
10
-<summary>What is an "IPC Semaphore"</summary>
11
-
12
-`IPC` stands for `Interprocess Communication`. IPC messages are a counterpart to UNIX pipes for IPC operations. \
13
-The
14
-fastest way to communicate through processes is with shared memory. `semaphores`, help synchronise shared memory access
15
-across processes.<sup> [1](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html) </sup>
16
-
17
-As illustrated by E. W. Dijkstra, semaphores can be better understood using his railroad model <sup> [2](
18
-https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html) </sup>.
19
-
20
-- Imagine a railroad, where only a single train at a time is allowed to pass. Responsible for the traffic is a
21
- `semaphore`. Each train that wants to enter the single track must wait for the `semaphore` to be in a state that
22
- allows access to the railroad. When a train enters the track, the `semaphore` changes the state to prevent all other
23
- traffic in the track. When the train leaves the railroad, it must change the state of the `semaphore` to allow another
24
- train to enter.
25
-
26
-
27
-- In the computer world, a `semaphore` is an integer and the train is a process (or a thread). For a process to proceed,
28
- it has to wait for the semaphore's value to become 0. If it proceeds, it increments this value by 1. Upon finishing
29
- the task, the process decrements the same value by 1.
30
-
31
-> `semaphores` let processes query or alter status information. They are often used to monitor and control the
32
-> availability of system resources such as shared memory segments.
33
-> <sup> [2](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html) </sup>
34
-
35
-</details>
36
-
37
-<br>
38
-
39
-<details>
40
-<summary>References</summary>
41
-
42
-[[1] Interprocess Communication](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html) \
43
-[[2] IPC:Semaphores](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html)
44
-
45
-</details>
46
-
47
-### Troubleshooting Section
48
-
49
-<details>
50
- <summary>Adjust the semaphore limit on your system</summary>
51
-
52
-You can check current `semaphore` limit on your machine, by running:
53
-
54
-```
55
-root@netdata~ # ipcs -ls
56
-```
57
-
58
-The output will be similar to this:
59
-
60
-```
61
------- Semaphore Limits --------
62
-max number of arrays = 32000
63
-max semaphores per array = 32000
64
-max semaphores system wide = 1024000000
65
-max ops per semop call = 500
66
-semaphore max value = 32767
67
-```
68
-
69
-To adjust the limit of the max semaphores,you can go to the `/proc/sys/kernel/sem` file and adjust the second field
70
-accordingly.
71
-
72
-</details>
\ No newline at end of file
health/guides/ipfs_datastore_usage.md
new
+53
@@ -0,0 +1,53 @@
1
+### Understand the alert
2
+
3
+This alert is related to the InterPlanetary File System (IPFS) distributed file system. It calculates the percentage of used IPFS datastore space. When you receive this alert, it means that your IPFS storage repository space is highly utilized.
4
+
5
+### What does high datastore usage mean?
6
+
7
+High datastore usage means your IPFS storage is close to its capacity. This can affect the system's performance and stability. It is essential to keep an eye on IPFS storage usage to ensure smooth functioning and avoid running out of storage.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check IPFS datastore usage
12
+
13
+ To check the current IPFS datastore storage utilization, use the `ipfs repo stat` command:
14
+
15
+ ```
16
+ ipfs repo stat
17
+ ```
18
+
19
+2. Identify large files and folders within the datastore
20
+
21
+ To find the largest files and folders within your IPFS datastore, use the following command:
22
+
23
+ ```
24
+ ipfs pin ls --type=recursive | xargs -n1 -I {} echo -n "{} " && ipfs object stat {} | head -n1 | awk '{print $2}'
25
+ ```
26
+
27
+3. Clean up IPFS datastore
28
+
29
+ You can clean up and remove files that are no longer needed from your datastore using `ipfs pin rm` and `ipfs repo gc` commands. Be cautious while removing data to avoid losing any essential files.
30
+
31
+ For example:
32
+
33
+ ```
34
+ ipfs pin rm <CID>
35
+ ipfs repo gc
36
+ ```
37
+
38
+4. Consider increasing the size of your datastore
39
+
40
+ If your datastore is continuously getting filled, you might need to increase its capacity to ensure smooth operation. This can be done by adjusting the `Datastore.StorageMax` configuration setting in the `config` file, which is typically located in the `.ipfs` folder.
41
+
42
+ ```
43
+ ipfs config Datastore.StorageMax <new size>
44
+ ```
45
+
46
+5. Monitor datastore usage over time
47
+
48
+ Regularly monitor your IPFS datastore usage using `ipfs repo stat` command to stay informed about its storage utilization and plan for any necessary adjustments.
49
+
50
+### Useful resources
51
+
52
+1. [IPFS Documentation](https://docs.ipfs.io/)
53
+2. [IPFS resize datastore](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#datastorestoragemax)
health/guides/ipmi/ipmi_events.md
deleted
-37
@@ -1,37 +0,0 @@
1
-# ipmi_events
2
-
3
-## OS: Any
4
-
5
-This alert presents the number of events in the IPMI System Event Log (SEL).
6
-If this alert is received, then the Log contains critical, warning, and informational events.
7
-
8
-- This alert is raised in a warning state when the number of events in the IPMI SEL exceed 0
9
- (in other words, when they exist).
10
-
11
-<br>
12
-
13
-<details>
14
-<summary>References and Sources</summary>
15
-
16
-1. ["ipmitool" manual page](
17
- https://linux.die.net/man/1/ipmitool)
18
-
19
-</details>
20
-
21
-### Troubleshooting Section
22
-
23
-<details>
24
-<summary>Use "ipmitool"</summary>
25
-
26
-
27
-> ipmitool is a utility for managing and configuring devices that support
28
-> the Intelligent Platform Management Interface. [Github](https://github.com/ipmitool/ipmitool)
29
-
30
-You can view the System Event Log using ipmitool, by running the command:
31
-
32
-```
33
-root@netdata~ # ipmitool sel list
34
-```
35
-
36
-You can find more info and commands in the [manual page](https://linux.die.net/man/1/ipmitool)
37
-</details>
\ No newline at end of file
health/guides/ipmi/ipmi_sensor_state.md
deleted
-45
@@ -1,45 +0,0 @@
1
-# ipmi_sensors_states
2
-
3
-## OS: Any
4
-
5
-This alert presents the number of IPMI sensors in the non-nominal state. \
6
-If this alert is received, then there are IPMI sensors in the warning or critical state.
7
-
8
-- This alert is raised in a warning state when the amount of sensors in a warning state are greater
9
- than 0.
10
-- If there are any sensors in a critical state, then the alert is also raised to critical.
11
-
12
-<br>
13
-
14
-<details>
15
-<summary>References and Sources</summary>
16
-
17
-1. ["ipmi-sensors" manual page](
18
- https://www.gnu.org/software/freeipmi/manpages/man8/ipmi-sensors.8.html)
19
-
20
-</details>
21
-
22
-### Troubleshooting Section
23
-
24
-<details>
25
-<summary>Use "ipmi-sensors" tools</summary>
26
-
27
-ipmi-sensors is a free software used to display sensor information (the package name is "
28
-freeipmi-tools").
29
-
30
-Here are some useful commands:
31
-
32
-> -v, --verbose Output verbose sensor output.
33
-> This option will output additional information about sensors such as thresholds, ranges,
34
-> numbers, and event/reading type codes.
35
-
36
-> --output-sensor-state Output sensor state in output.
37
-> This will add an additional output reporting if a sensor is in a NOMINAL, WARNING, or CRITICAL
38
-> state. The sensor state is an interpreted value based on the current sensor event. The sensor
39
-> state interpretations are determined by the configuration file
40
-> /etc/freeipmi//freeipmi_interpret_sensor.conf.
41
-
42
-You can see more options in the [manual page](
43
-https://www.gnu.org/software/freeipmi/manpages/man8/ipmi-sensors.8.html).
44
-
45
-</details>
health/guides/ipmi_events.md
new
+38
@@ -0,0 +1,38 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when there are events recorded in the IPMI System Event Log (SEL). These events can range from critical, warning, and informational events. The alert enters a warning state when the number of events in the IPMI SEL exceeds 0, meaning there are recorded events that may require your attention.
4
+
5
+### What is IPMI SEL?
6
+
7
+The Intelligent Platform Management Interface (IPMI) System Event Log (SEL) is a log that records events related to hardware components and firmware on a server. These events can provide insight into potential issues with the server's hardware or firmware, which could impact the server's overall performance or stability.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. **Use `ipmitool` to view the IPMI SEL events:**
12
+
13
+ You can view the System Event Log using the `ipmitool` command. If you don't have `ipmitool` installed, you might need to install it first. Once `ipmitool` is installed, use the following command to list the SEL events:
14
+
15
+ ```
16
+ ipmitool sel list
17
+ ```
18
+
19
+ This command will display the recorded events with their respective timestamp, event ID, and a brief description.
20
+
21
+2. **Identify and resolve issues:**
22
+
23
+ Analyze the events listed to identify any critical or warning events that may require immediate attention. You may need to refer to your server's hardware documentation or firmware updates to resolve the issue.
24
+
25
+3. **Clear the IPMI SEL events (optional):**
26
+
27
+ If you have resolved the issues or if the events listed are no longer relevant, you can clear the IPMI SEL events using the following command:
28
+
29
+ ```
30
+ ipmitool sel clear
31
+ ```
32
+
33
+ Note: Clearing the SEL events may cause you to lose important historical information related to your hardware components and firmware. Be cautious when using this command, and ensure that you have resolved any critical issues before clearing the event log.
34
+
35
+### Useful resources
36
+
37
+1. [IPMITOOL GitHub Repository](https://github.com/ipmitool/ipmitool)
38
+2. [IPMITOOL Manual Page](https://linux.die.net/man/1/ipmitool)
health/guides/ipmi_sensors_states.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert is related to the IPMI (Intelligent Platform Management Interface) sensors in your system. IPMI is a hardware management interface used for monitoring server health and collecting information on various hardware components. The alert is triggered when any of the IPMI sensors detect conditions that are outside the normal operating range, and are in a warning or critical state.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check IPMI sensor status:
8
+
9
+ To check the status of IPMI sensors, you can use the `ipmi-sensors` command with appropriate flags. For instance:
10
+
11
+ ```
12
+ sudo ipmi-sensors --output-sensor-state
13
+ ```
14
+
15
+ This command will provide you with detailed information on the current state of each sensor, allowing you to determine which ones are in a warning or critical state.
16
+
17
+2. Analyze sensor data:
18
+
19
+ Based on the output obtained in the previous step, identify the sensors that are causing the alert. Take note of their current values and thresholds.
20
+
21
+ To obtain more detailed information, you can also use the `-v` (verbose) flag with the command:
22
+
23
+ ```
24
+ sudo ipmi-sensors -v --output-sensor-state
25
+ ```
26
+
27
+3. Investigate the cause of the issue:
28
+
29
+ Once you have identified the sensors in a non-nominal state, start investigating the root cause of the issue. This may involve checking the hardware components, system logs, or contacting your hardware vendor for additional support.
30
+
31
+4. Resolve the issue:
32
+
33
+ Based on your investigation, take the necessary steps to resolve the issue. This may include replacing faulty hardware, addressing configuration errors, or applying firmware updates.
34
+
35
+5. Verify resolution:
36
+
37
+ After addressing the issue, use the `ipmi-sensors` command to check the status of the affected sensors. Ensure that they have returned to the nominal state, and no additional warning or critical conditions are being reported.
38
+
39
+### Useful resources
40
+
41
+1. ["ipmi-sensors" manual page](https://www.gnu.org/software/freeipmi/manpages/man8/ipmi-sensors.8.html)
health/guides/k8s_cgroup_10min_cpu_usage.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average `cgroup CPU utilization` over the past 10 minutes in a Kubernetes cluster. If you receive this alert at the warning or critical levels, it means that your cgroup is heavily utilizing the available CPU resources.
4
+
5
+### What does cgroup CPU utilization mean?
6
+
7
+In Kubernetes, `cgroups` are a Linux kernel feature that helps to limit and isolate the resource usage (CPU, memory, disk I/O, etc.) of a collection of processes. The `cgroup CPU utilization` measures the percentage of available CPU resources consumed by the processes within a cgroup.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Identify the over-utilizing cgroup
12
+
13
+Check the alert message for the specific cgroup that is causing high CPU utilization.
14
+
15
+- Determine the processes utilizing the most CPU resources in the cgroup
16
+
17
+To find the processes within the cgroup with high CPU usage, you can use `systemd-cgtop` on the Kubernetes nodes:
18
+
19
+```
20
+systemd-cgtop -m -1 -p -n10
21
+```
22
+
23
+- Analyze the Kubernetes resource usage
24
+
25
+Use `kubectl top` to get an overview of the resource usage in your Kubernetes cluster:
26
+
27
+```
28
+kubectl top nodes
29
+kubectl top pods
30
+```
31
+
32
+- Investigate the Kubernetes events and logs
33
+
34
+Examine the events and logs of the Kubernetes cluster and the specific resources that are causing the high CPU utilization.
35
+
36
+```
37
+kubectl get events --sort-by='.metadata.creationTimestamp'
38
+kubectl logs <pod-name> -n <namespace> --timestamps -f
39
+```
40
+
41
+- Optimize the resource usage of the cluster
42
+
43
+You may need to scale your cluster by adding more resources, adjusting the resource limits, or optimizing the application code to minimize CPU usage.
44
+
45
+### Useful resources
46
+
47
+1. [Overview of a Pod](https://kubernetes.io/docs/concepts/workloads/pods/)
48
+2. [Assign CPU Resources to Containers and Pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/)
health/guides/k8s_cgroup_10s_received_packets_storm.md
new
+59
@@ -0,0 +1,59 @@
1
+### Understand the alert
2
+
3
+This alert indicates a potential `received packets storm` in your Kubernetes (k8s) cluster's network on a cgroup (control group) network interface. A received packets storm occurs when the average number of received packets in the last 10 seconds significantly exceeds the rate over the last minute.
4
+
5
+### What is a cgroup?
6
+
7
+A `cgroup` (control group) is a Linux kernel feature to limit, account, and isolate resource usage (CPU, memory, disk I/O, etc.) for a process or a group of processes. In Kubernetes, cgroups are used to manage resources for each container within a pod.
8
+
9
+### What is a received packets storm?
10
+
11
+A received packets storm occurs when the average number of received packets on a network interface becomes significantly higher than the recent background rate. This can cause network congestion, increased latency, or even denial of service, affecting the performance of services running on the Kubernetes cluster.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Inspect overall network activity on the affected node(s):
16
+
17
+ Use the `iftop` command to monitor network activity on the host in real-time:
18
+
19
+ ```
20
+ sudo iftop
21
+ ```
22
+
23
+ If you don't have `iftop` installed, install it before running the command.
24
+
25
+2. Identify the container(s) responsible for the high packet rate:
26
+
27
+ To list the running container(s) and their associated cgroups, run the following command:
28
+
29
+ ```
30
+ sudo kubectl get pods --all-namespaces -o jsonpath='{range.items[*]}{.metadata.namespace}:{.metadata.name}{"\t"}{.status.containerStatuses[].containerID}{"\t"}{"cgroup id: "}{"\n"}{end}'
31
+ ```
32
+
33
+ Now, check the network interface statistics for each container:
34
+
35
+ ```
36
+ cat /sys/fs/cgroup/net_cls,net_prio/net_cls.classid
37
+ ```
38
+
39
+3. Investigate the cause:
40
+
41
+ - Inspect the logs of the affected container(s) for any errors or unusual activity:
42
+
43
+ ```
44
+ sudo kubectl logs -f <pod-name> -c <container-name> -n <namespace>
45
+ ```
46
+
47
+ - Check if there are any misconfigurations or if network rate limits are not set correctly in the Kubernetes Deployment, StatefulSet, or DaemonSet manifest.
48
+
49
+4. Mitigate the issue:
50
+
51
+ - If unnecessary traffic is causing the packet storm, consider implementing network throttling or limiting the rate at which packets are generated or received by the container(s).
52
+
53
+ - If the issue is caused by a bug or misconfiguration, fix the problem and redeploy the affected component(s).
54
+
55
+### Useful resources
56
+
57
+1. [Kubernetes Cgroups documentation](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/#debugging-the-kernel-cgroups-and-kubernetes-primitives)
58
+2. [Monitoring and Visualizing Network Bandwidth on Linux](https://www.tecmint.com/linux-network-bandwidth-monitoring-tools/)
59
+3. [Networking in Kubernetes](https://kubernetes.io/docs/concepts/cluster-administration/networking/)
health/guides/k8s_cgroup_1m_received_packets_rate.md
new
+49
@@ -0,0 +1,49 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average number of packets received by a specific network interface (denoted as `${label:device}` in the alert) on a Kubernetes cluster node over the last minute. If you receive this alert, it indicates that there is a significant amount of network traffic received by the node.
4
+
5
+### What does high received packets rate mean?
6
+
7
+A high received packets rate means that the network interface on the Kubernetes cluster node is processing a large number of incoming network packets. This can be due to increased legitimate traffic to the services running on the cluster or may indicate a potential network issue or Distributed Denial of Service (DDoS) attack.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Verify the current network traffic on the Kubernetes node:
12
+
13
+ You can use the `nethogs` tool to analyze the network traffic on the Kubernetes node. If the tool is not installed, you can install it with:
14
+
15
+ ```
16
+ sudo apt install nethogs # Ubuntu/Debian
17
+ sudo yum install nethogs # CentOS/RHEL
18
+ ```
19
+
20
+ Run `nethogs` to check the network traffic:
21
+
22
+ ```
23
+ sudo nethogs
24
+ ```
25
+
26
+2. Check the services running on the Kubernetes cluster:
27
+
28
+ Use the command `kubectl get pods --all-namespaces` to list all the pods running on the cluster. Inspect the output and identify any services that might be consuming a high amount of network traffic.
29
+
30
+3. Inspect logs for any anomalies:
31
+
32
+ Check the application and Kubernetes logs for any unusual activity, errors, or repeated access attempts that may indicate a network issue or potential attack.
33
+
34
+4. Close unnecessary processes or services:
35
+
36
+ Based on your analysis, if you find any unnecessary processes or services consuming a high amount of network traffic, consider terminating or scaling them down.
37
+
38
+5. Check for DDoS attacks:
39
+
40
+ If you suspect a DDoS attack, consider implementing traffic filtering, rate limiting, or using a DDoS protection service to mitigate the attack.
41
+
42
+6. Monitor network traffic:
43
+
44
+ Continue monitoring the network traffic on the Kubernetes node to ensure that the received packets rate returns to normal levels.
45
+
46
+### Useful resources
47
+
48
+1. [Kubernetes Networking](https://kubernetes.io/docs/concepts/cluster-administration/networking/)
49
+2. [How to Monitor and Identify Issues with Kubernetes Networking](https://www.stackrox.com/post/2017/03/how-to-monitor-and-identify-issues-with-kubernetes-networking/)
health/guides/k8s_cgroup_ram_in_use.md
new
+42
@@ -0,0 +1,42 @@
1
+### Understand the alert
2
+
3
+This alert monitors the `RAM usage` in a Kubernetes cluster by calculating the ratio of the memory used by a cgroup to its memory limit. If the memory usage exceeds certain thresholds, the alert triggers and indicates that the system's memory resources are under pressure.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check overall RAM usage in the cluster
8
+
9
+ Use the `kubectl top nodes` command to check the overall memory usage on the cluster nodes:
10
+ ```
11
+ kubectl top nodes
12
+ ```
13
+
14
+2. Identify Pods with high memory usage
15
+
16
+ Use the `kubectl top pods --all-namespaces` command to identify Pods consuming a high amount of memory:
17
+ ```
18
+ kubectl top pods --all-namespaces
19
+ ```
20
+
21
+3. Inspect logs for errors or misconfigurations
22
+
23
+ Check the logs of Pods consuming high memory for any issues or misconfigurations:
24
+ ```
25
+ kubectl logs -n <namespace> <pod_name>
26
+ ```
27
+
28
+4. Inspect container resource limits
29
+
30
+ Review the resource limits defined in the Pod's yaml file, particularly the `limits` and `requests` sections. If you're not setting limits on Pods, then consider setting appropriate limits to prevent running out of resources.
31
+
32
+5. Scale or optimize applications
33
+
34
+ If high memory usage is expected and justified, consider scaling the application by adding replicas or increasing the allocated resources.
35
+
36
+ If the memory usage is not justified, optimizing the application code or configurations may help reduce memory usage.
37
+
38
+### Useful resources
39
+
40
+1. [Kubernetes best practices: Organizing with Namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/)
41
+2. [Managing Resources for Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
42
+3. [Configure Default Memory Requests and Limits](https://kubernetes.io/docs/tasks/administer-cluster/memory-default-namespace/)
\ No newline at end of file
health/guides/kubelet/kubelet_10s_pleg_relist_latency_quantile_05.md
deleted
-77
@@ -1,77 +0,0 @@
1
-# kubelet_10s_pleg_relist_latency_quantile_05
2
-
3
-**Kubernetes | Kubelet**
4
-
5
-_The kubelet is the primary "node agent" that runs on each node. It makes sure that containers are
6
-running in a Pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms
7
-and ensures that the containers described in those PodSpecs are running and healthy and doesn't
8
-manage containers that were not created by Kubernetes._
9
-
10
-The PLEG (Pod Lifecycle Event Generator) module in Kubelet adjusts the container runtime state with
11
-each matched pod-level event and keeps the Pod's cache up to date. Big delays in the relist process
12
-of pods will eventually cause a "PLEG is not healthy" event which will make the node unavailable (
13
-NotReady).
14
-
15
-The Netadata Agent calculates the ratio of average Pod Lifecycle Event Generator relisting latency
16
-over the last 10 seconds, compared to the last minute (quantile 0.5). Receiving this alert means
17
-that the relisting time has increased significantly.
18
-
19
-> Different pods have different relisting latencies, more quantiles help you reduce the error rate in those metrics.
20
-
21
-
22
-<details>
23
-<summary>See more about the kubelet </summary>
24
-
25
-As we said before, the kubelet works in terms of a PodSpec. A PodSpec is a YAML or a JSON object
26
-that describes a pod. The PodSpec contains all information a kubelet needs to know to run the pod in
27
-the corresponding cluster node.
28
-
29
-Beside the PodSpecs provided from the Kubernetes APIserver, there are three ways to provide a
30
-kubelet with a container manifest:
31
-
32
-- File: Path passed as a flag on the command line. Files under this path will be monitored
33
- periodically for updates. The monitoring period is 20s by default and is configurable via a flag.
34
-- HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This endpoint is checked
35
- every 20 seconds (also configurable with a flag).
36
-- HTTP server: The kubelet can also listen for HTTP and respond to a simple API (underspec'd
37
- currently) to submit a new manifest.
38
-
39
-See more about Kubelet in
40
-the [Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
41
-
42
-</details>
43
-
44
-
45
-<details>
46
-<summary>See more about PLEG and the relist process</summary>
47
-
48
-A kubelet keeps track of all the Pods that are about to run in the node. The node could have any
49
-kind of Container Runtime Interface (CRI) always compatible with Kubernetes. A Pod lifecycle event
50
-interprets the underlying container state change at the pod-level abstraction, making it
51
-container-runtime-agnostic. This abstraction shields a kubelet from the runtime specifics.
52
-
53
-In order to generate pod lifecycle events, PLEG needs to detect changes in container states. The
54
-PLEG module periodically relisting all containers (even then stopped ones) and compare then with
55
-their Kubelet's PodSpecs. The relist process takes longer when there are problems with the
56
-underlying CRI or overloading of a Node with too many pods.
57
-
58
-See more about the PLEG's mechanism in
59
-the [Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
60
-
61
-</details>
62
-
63
-<details>
64
-<summary>References and Sources</summary>
65
-
66
-1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
67
-2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
68
-
69
-</details>
70
-
71
-### Troubleshooting
72
-
73
-Most cloud providers address this issue by limiting the Pods that can run in particular nodes in
74
-their managed Kubernetes services. They often implement health checks into the underlying container
75
-runtime. However, you may encounter this issue in high-end nodes which can run hundreds of
76
-containers. If you have configured your cluster by yourself (let's say with `kubeadm`), you can
77
-update the value of max Pods.
health/guides/kubelet/kubelet_10s_pleg_relist_latency_quantile_09.md
deleted
-77
@@ -1,77 +0,0 @@
1
-# kubelet_10s_pleg_relist_latency_quantile_09
2
-
3
-**Kubernetes | Kubelet**
4
-
5
-_The kubelet is the primary "node agent" that runs on each node. It makes sure that containers are
6
-running in a Pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms
7
-and ensures that the containers described in those PodSpecs are running and healthy and doesn't
8
-manage containers that were not created by Kubernetes._
9
-
10
-The PLEG (Pod Lifecycle Event Generator) module in Kubelet adjusts the container runtime state with
11
-each matched pod-level event and keeps the Pod's cache up to date. Big delays in the relist process
12
-of pods will eventually cause a "PLEG is not healthy" event which will make the node unavailable (
13
-NotReady).
14
-
15
-The Netadata Agent calculates the ratio of average Pod Lifecycle Event Generator relisting latency
16
-over the last 10 seconds, compared to the last minute (quantile 0.9). Receiving this alert means
17
-that the relisting time has increased significantly.
18
-
19
-> Different pods have different relisting latencies, more quantiles help you reduce the error rate in those metrics.
20
-
21
-
22
-<details>
23
-<summary>See more about the kubelet </summary>
24
-
25
-As we said before, the kubelet works in terms of a PodSpec. A PodSpec is a YAML or a JSON object
26
-that describes a pod. The PodSpec contains all information a kubelet needs to know to run the pod in
27
-the corresponding cluster node.
28
-
29
-Beside the PodSpecs provided from the Kubernetes APIserver, there are three ways to provide a
30
-kubelet with a container manifest:
31
-
32
-- File: Path passed as a flag on the command line. Files under this path will be monitored
33
- periodically for updates. The monitoring period is 20s by default and is configurable via a flag.
34
-- HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This endpoint is checked
35
- every 20 seconds (also configurable with a flag).
36
-- HTTP server: The kubelet can also listen for HTTP and respond to a simple API (underspec'd
37
- currently) to submit a new manifest.
38
-
39
-See more about Kubelet in
40
-the [Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
41
-
42
-</details>
43
-
44
-
45
-<details>
46
-<summary>See more about PLEG and the relist process</summary>
47
-
48
-A kubelet keeps track of all the Pods that are about to run in the node. The node could have any
49
-kind of Container Runtime Interface (CRI) always compatible with Kubernetes. A Pod lifecycle event
50
-interprets the underlying container state change at the pod-level abstraction, making it
51
-container-runtime-agnostic. This abstraction shields a kubelet from the runtime specifics.
52
-
53
-In order to generate pod lifecycle events, PLEG needs to detect changes in container states. The
54
-PLEG module periodically relisting all containers (even then stopped ones) and compare then with
55
-their Kubelet's PodSpecs. The relist process takes longer when there are problems with the
56
-underlying CRI or overloading of a Node with too many pods.
57
-
58
-See more about the PLEG's mechanism in
59
-the [Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
60
-
61
-</details>
62
-
63
-<details>
64
-<summary>References and Sources</summary>
65
-
66
-1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
67
-2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
68
-
69
-</details>
70
-
71
-### Troubleshooting
72
-
73
-Most cloud providers address this issue by limiting the Pods that can run in particular nodes in
74
-their managed Kubernetes services. They often implement health checks into the underlying container
75
-runtime. However, you may encounter this issue in high-end nodes which can run hundreds of
76
-containers. If you have configured your cluster by yourself (let's say with `kubeadm`), you can
77
-update the value of max Pods.
health/guides/kubelet/kubelet_10s_pleg_relist_latency_quantile_099.md
deleted
-77
@@ -1,77 +0,0 @@
1
-# kubelet_10s_pleg_relist_latency_quantile_099
2
-
3
-**Kubernetes | Kubelet**
4
-
5
-_The kubelet is the primary "node agent" that runs on each node. It makes sure that containers are
6
-running in a Pod. The kubelet takes a set of PodSpecs that are provided through various mechanisms
7
-and ensures that the containers described in those PodSpecs are running and healthy and doesn't
8
-manage containers that were not created by Kubernetes._
9
-
10
-The PLEG (Pod Lifecycle Event Generator) module in Kubelet adjusts the container runtime state with
11
-each matched pod-level event and keeps the Pod's cache up to date. Big delays in the relist process
12
-of pods will eventually cause a "PLEG is not healthy" event which will make the node unavailable (
13
-NotReady).
14
-
15
-The Netadata Agent calculates the ratio of average Pod Lifecycle Event Generator relisting latency
16
-over the last 10 seconds, compared to the last minute (quantile 0.99). Receiving this alert means
17
-that the relisting time has increased significantly.
18
-
19
-> Different pods have different relisting latencies, more quantiles help you reduce the error rate in those metrics.
20
-
21
-
22
-<details>
23
-<summary>See more about the kubelet </summary>
24
-
25
-As we said before, the kubelet works in terms of a PodSpec. A PodSpec is a YAML or a JSON object
26
-that describes a pod. The PodSpec contains all information a kubelet needs to know to run the pod in
27
-the corresponding cluster node.
28
-
29
-Beside the PodSpecs provided from the Kubernetes APIserver, there are three ways to provide a
30
-kubelet with a container manifest:
31
-
32
-- File: Path passed as a flag on the command line. Files under this path will be monitored
33
- periodically for updates. The monitoring period is 20s by default and is configurable via a flag.
34
-- HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This endpoint is checked
35
- every 20 seconds (also configurable with a flag).
36
-- HTTP server: The kubelet can also listen for HTTP and respond to a simple API (underspec'd
37
- currently) to submit a new manifest.
38
-
39
-See more about Kubelet in
40
-the [Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
41
-
42
-</details>
43
-
44
-
45
-<details>
46
-<summary>See more about PLEG and the relist process</summary>
47
-
48
-A kubelet keeps track of all the Pods that are about to run in the node. The node could have any
49
-kind of Container Runtime Interface (CRI) always compatible with Kubernetes. A Pod lifecycle event
50
-interprets the underlying container state change at the pod-level abstraction, making it
51
-container-runtime-agnostic. This abstraction shields a kubelet from the runtime specifics.
52
-
53
-In order to generate pod lifecycle events, PLEG needs to detect changes in container states. The
54
-PLEG module periodically relisting all containers (even then stopped ones) and compare then with
55
-their Kubelet's PodSpecs. The relist process takes longer when there are problems with the
56
-underlying CRI or overloading of a Node with too many pods.
57
-
58
-See more about the PLEG's mechanism in
59
-the [Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
60
-
61
-</details>
62
-
63
-<details>
64
-<summary>References and Sources</summary>
65
-
66
-1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
67
-2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
68
-
69
-</details>
70
-
71
-### Troubleshooting
72
-
73
-Most cloud providers address this issue by limiting the Pods that can run in particular nodes in
74
-their managed Kubernetes services. They often implement health checks into the underlying container
75
-runtime. However, you may encounter this issue in high-end nodes which can run hundreds of
76
-containers. If you have configured your cluster by yourself (let's say with `kubeadm`), you can
77
-update the value of max Pods.
health/guides/kubelet_10s_pleg_relist_latency_quantile_05.md
new
+35
@@ -0,0 +1,35 @@
1
+### Troubleshoot the alert
2
+
3
+1. Check Kubelet logs
4
+ To diagnose issues with the PLEG relist process, look at the Kubelet logs. The following command can be used to fetch the logs from the affected node:
5
+
6
+ ```
7
+ kubectl logs -n kube-system <node_name>
8
+ ```
9
+
10
+ Look for any error messages related to PLEG or container runtime.
11
+
12
+2. Check container runtime status
13
+ Monitor the health status and performance of the container runtime (e.g. Docker, containerd) by running the appropriate commands like `docker ps`, `docker info` or `ctr version` and `ctr info`. Check container runtime logs for any issues as well.
14
+
15
+3. Inspect node resources
16
+ Verify if the node is overloaded or under excessive pressure by checking the CPU, memory, disk, and network resources. Use tools like `top`, `vmstat`, `df`, and `iostat`. You can also use the Kubernetes `kubectl top node` command to view resource utilization on your nodes.
17
+
18
+4. Limit maximum Pods per node
19
+ To avoid overloading nodes in your cluster, consider limiting the maximum number of Pods that can run on a single node. You can follow these steps to update the max Pods value:
20
+
21
+ - Edit the Kubelet configuration file (usually located at `/etc/kubernetes/kubelet.conf` or `/var/lib/kubelet/config.yaml`) on the affected node.
22
+ - Change the value of the `maxPods` parameter to a more appropriate number. The default value is 110.
23
+ - Restart the Kubelet service with `systemctl restart kubelet` or `service kubelet restart`.
24
+ - Check the Kubelet logs to ensure the new value is effective.
25
+
26
+5. Check Pod eviction thresholds
27
+ Review the Pod eviction thresholds defined in the Kubelet configuration, which might cause Pods to be evicted due to resource pressure. Adjust the threshold values if needed.
28
+
29
+6. Investigate Pods causing high relisting latency
30
+ Analyze the Pods running on the affected node and identify any Pods that might be causing high PLEG relist latency. These could be Pods with a large number of containers or high resource usage. Consider optimizing or removing these Pods if they are not essential to your workload.
31
+
32
+### Useful resources
33
+
34
+1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
35
+2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes/)
health/guides/kubelet_10s_pleg_relist_latency_quantile_09.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the average relisting latency of the Pod Lifecycle Event Generator (PLEG) in Kubelet over the last 10 seconds compared to the last minute (quantile 0.9) has increased significantly. This can cause the node to become unavailable (NotReady) due to a "PLEG is not healthy" event.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check for high node resource usage
8
+
9
+ First, ensure that the node does not have an overly high number of Pods. High resource usage could increase the PLEG relist latency, leading to poor Kubelet performance. You can check the current number of running Pods on a node using the following command:
10
+
11
+ ```
12
+ kubectl get pods --all-namespaces -o wide | grep <node-name>
13
+ ```
14
+
15
+2. Check Kubelet logs for errors
16
+
17
+ Inspect the Kubelet logs for any errors that might be causing the increased PLEG relist latency. You can check the Kubelet logs using the following command:
18
+
19
+ ```
20
+ sudo journalctl -u kubelet
21
+ ```
22
+
23
+ Look for any errors associated with PLEG or the container runtime, such as Docker or containerd.
24
+
25
+3. Check container runtime health
26
+
27
+ If you find any issues in the Kubelet logs related to the container runtime, investigate the health of the container runtime, such as Docker or containerd, and its logs to identify any issues:
28
+
29
+ - For Docker, you can check its health using:
30
+
31
+ ```
32
+ sudo docker info
33
+ sudo journalctl -u docker
34
+ ```
35
+
36
+ - For containerd, you can check its health using:
37
+
38
+ ```
39
+ sudo ctr version
40
+ sudo journalctl -u containerd
41
+ ```
42
+
43
+4. Adjust the maximum number of Pods per node
44
+
45
+ If you have configured your cluster manually (e.g., with `kubeadm`), you can update the value of max Pods in the Kubelet configuration file. The default file location is `/var/lib/kubelet/config.yaml`. Change the `maxPods` value according to your requirements and restart the Kubelet service:
46
+
47
+ ```
48
+ sudo systemctl restart kubelet
49
+ ```
50
+
51
+5. Monitor the PLEG relist latency
52
+
53
+ After making any necessary changes, continue monitoring the PLEG relist latency to ensure the issue has been resolved.
54
+
55
+### Useful resources
56
+
57
+1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
58
+2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
\ No newline at end of file
health/guides/kubelet_10s_pleg_relist_latency_quantile_099.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+This alert is related to the Kubernetes Kubelet, which is the primary node agent responsible for ensuring containers run in a Pod. The alert specifically relates to the Pod Lifecycle Event Generator (PLEG) module, which is responsible for adjusting the container runtime state and maintaining the Pod's cache. When there is a significant increase in the relisting time for PLEG, you'll receive a `kubelet_10s_pleg_relist_latency_quantile_099` alert.
4
+
5
+### Troubleshoot the alert
6
+
7
+Follow the steps below to troubleshoot this alert:
8
+
9
+1. Check the container runtime health status
10
+
11
+ If you are using Docker as the container runtime, run the following command:
12
+
13
+ ```
14
+ sudo docker info
15
+ ```
16
+
17
+ Check for any reported errors or issues.
18
+
19
+ If you are using a different container runtime like containerd or CRI-O, refer to the respective documentation for health check commands.
20
+
21
+2. Check Kubelet logs for any errors.
22
+
23
+ You can do this by running the following command:
24
+
25
+ ```
26
+ sudo journalctl -u kubelet -n 1000
27
+ ```
28
+
29
+ Look for any relevant error messages or warnings in the output.
30
+
31
+3. Validate that the node is not overloaded with too many Pods.
32
+
33
+ Run the following commands:
34
+
35
+ ```
36
+ kubectl get nodes
37
+ kubectl describe node <node_name>
38
+ ```
39
+
40
+ Adjust the max number of Pods per node if needed, by editing the Kubelet configuration file `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf`, adding the `--max-pods=<NUMBER>` flag, and restarting Kubelet:
41
+
42
+ ```
43
+ sudo systemctl daemon-reload
44
+ sudo systemctl restart kubelet
45
+ ```
46
+
47
+4. Check for issues related to the underlying storage or network.
48
+
49
+ Inspect the Node's storage and ensure there are no I/O limitations or bottlenecks causing the increased latency. Also, check for network-related issues that could affect the communication between the Kubelet and the container runtime.
50
+
51
+5. Verify the performance and health of the Kubernetes API server.
52
+
53
+ High workload on the API server could affect the Kubelet's ability to communicate and process Pod updates. Check the API server logs and metrics to find any performance bottlenecks or errors.
54
+
55
+### Useful resources
56
+
57
+1. [Kubelet CLI in Kubernetes official docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)
58
+2. [PLEG mechanism explained in Redhat's blogspot](https://developers.redhat.com/blog/2019/11/13/pod-lifecycle-event-generator-understanding-the-pleg-is-not-healthy-issue-in-kubernetes#)
\ No newline at end of file
health/guides/kubelet_1m_pleg_relist_latency_quantile_05.md
new
+59
@@ -0,0 +1,59 @@
1
+### Understand the alert
2
+
3
+This alert is related to Kubernetes and is triggered when the average `Pod Lifecycle Event Generator (PLEG)` relisting latency over the last minute is higher than the expected threshold (quantile 0.5). If you receive this alert, it means that the kubelet is experiencing some latency issues, which may affect the scheduling and management of your Kubernetes Pods.
4
+
5
+### What is PLEG?
6
+
7
+The Pod Lifecycle Event Generator (PLEG) is a component within the kubelet responsible for keeping track of changes (events) to the Pod and updating the kubelet's internal status. This ensures that the kubelet can successfully manage and schedule Pods on the Kubernetes node.
8
+
9
+### What does relisting latency mean?
10
+
11
+Relisting latency refers to the time taken by the PLEG to detect, process, and update the kubelet about the events or changes in a Pod's lifecycle. High relisting latency can lead to delays in the kubelet reacting to these changes, which can affect the overall functioning of the Kubernetes cluster.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the kubelet logs for any errors or warnings related to PLEG:
16
+
17
+ ```
18
+ sudo journalctl -u kubelet
19
+ ```
20
+
21
+ Look for any logs related to PLEG delays, issues, or timeouts.
22
+
23
+2. Restart the kubelet if necessary:
24
+
25
+ ```
26
+ sudo systemctl restart kubelet
27
+ ```
28
+
29
+ Sometimes, restarting the kubelet can resolve sporadic latency issues.
30
+
31
+3. Monitor the Kubernetes node's resource usage (CPU, Memory, Disk) using `kubectl top nodes`:
32
+
33
+ ```
34
+ kubectl top nodes
35
+ ```
36
+
37
+ If the node's resource usage is too high, consider scaling your cluster or optimizing workloads.
38
+
39
+4. Check the overall health of your Kubernetes cluster:
40
+
41
+ ```
42
+ kubectl get nodes
43
+ kubectl get pods --all-namespaces
44
+ ```
45
+
46
+ These commands will help you identify any issues with other nodes or Pods in your cluster.
47
+
48
+5. Investigate the specific Pods experiencing latency in PLEG:
49
+
50
+ ```
51
+ kubectl describe pod <pod_name> -n <namespace>
52
+ ```
53
+
54
+ Look for any signs of the Pod being stuck in a pending state, startup issues, or container crashes.
55
+
56
+### Useful resources
57
+
58
+1. [Kubernetes Kubelet - PLEG](https://kubernetes.io/docs/concepts/overview/components/#kubelet)
59
+2. [Kubernetes Troubleshooting](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/)
health/guides/kubelet_1m_pleg_relist_latency_quantile_09.md
new
+45
@@ -0,0 +1,45 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average Pod Lifecycle Event Generator (PLEG) relisting latency over the period of one minute, using the 0.9 quantile. This alert is related to Kubelet, a critical component in the Kubernetes cluster that ensures the correct running of containers inside pods. If you receive this alert, it means that the relisting latency has increased in your Kubernetes cluster, possibly affecting the performance of your workloads.
4
+
5
+### What does PLEG relisting latency mean?
6
+
7
+In Kubernetes, PLEG is responsible for keeping track of container lifecycle events, such as container start, stop, or pause. It periodically relists these events and updates the Kubernetes Pod status, ensuring the scheduler and other components know the correct state of the containers. An increased relisting latency could lead to slower updates on Pod status and overall degraded performance.
8
+
9
+### What does 0.9 quantile mean?
10
+
11
+The 0.9 quantile represents the value below which 90% of the latencies are. An alert based on the 0.9 quantile suggests that 90% of relisting latencies are below the specified threshold, meaning that the remaining 10% are experiencing increased latency, which could lead to issues in your cluster.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check Kubelet logs for errors or warnings related to PLEG:
16
+
17
+ Access the logs of the Kubelet component running on the affected node:
18
+
19
+ ```
20
+ sudo journalctl -u kubelet
21
+ ```
22
+
23
+2. Monitor the overall performance of your Kubernetes cluster:
24
+
25
+ Use `kubectl top nodes` to check the resource usage of your nodes and identify any bottlenecks, such as high CPU or memory consumption.
26
+
27
+3. Check the status of Pods:
28
+
29
+ Use `kubectl get pods --all-namespaces` to check the status of all Pods in your cluster. Look for Pods in an abnormal state (e.g., Pending, CrashLoopBackOff, or Terminating), which could be related to high PLEG relisting latency.
30
+
31
+4. Analyze Pod logs for issues:
32
+
33
+ Investigate the logs of the affected Pods to understand any issues with the container lifecycle events:
34
+
35
+ ```
36
+ kubectl logs <pod-name> -n <namespace>
37
+ ```
38
+
39
+5. Review the Kubelet configuration:
40
+
41
+ Ensure that your Kubelet configuration is set up correctly to handle your workloads. If necessary, adjust the settings to improve PLEG relisting performance.
42
+
43
+### Useful resources
44
+
45
+1. [Kubernetes Troubleshooting Guide](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/)
health/guides/kubelet_1m_pleg_relist_latency_quantile_099.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average Pod Lifecycle Event Generator (PLEG) relisting latency over the last minute with a quantile of 0.99 in microseconds. If you receive this alert, it means that the Kubelet's PLEG latency is high, which can slow down your Kubernetes cluster.
4
+
5
+### What does PLEG latency mean?
6
+
7
+Pod Lifecycle Event Generator (PLEG) is a component of the Kubelet that watches for container events on the system and generates events for a pod's lifecycle. High PLEG latency indicates a delay in processing these events, which can cause delays in pod startup, termination, and updates.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the overall Kubelet performance and system load:
12
+
13
+ a. Run `kubectl get nodes` to check the status of the nodes in your cluster.
14
+ b. Investigate the node with high PLEG latency using `kubectl describe node <NODE_NAME>` to view detailed information about resource usage and events.
15
+ c. Use monitoring tools like `top`, `htop`, or `vmstat` to check for high CPU, memory, or disk usage on the node.
16
+
17
+2. Look for problematic pods or containers:
18
+
19
+ a. Run `kubectl get pods --all-namespaces` to check the status of all pods across namespaces.
20
+ b. Use `kubectl logs <POD_NAME> -n <NAMESPACE>` to check the logs of the pods in the namespace.
21
+ c. Investigate pods with high restart counts, crash loops, or other abnormal statuses.
22
+
23
+3. Verify Kubelet configurations and logs:
24
+
25
+ a. Check the Kubelet configuration on the node. Look for any misconfigurations or settings that could cause high latency.
26
+ b. Check Kubelet logs using `journalctl -u kubelet` for more information about PLEG events and errors.
27
+
28
+4. Consider evaluating your workloads and scaling your cluster:
29
+
30
+ a. If you have multiple nodes experiencing high PLEG latency or if the overall load on your nodes is consistently high, you might need to scale your cluster.
31
+ b. Evaluate your workloads and adjust resource requests and limits to make the best use of your available resources.
32
+
33
+### Useful resources
34
+
35
+1. [Understanding the Kubernetes Kubelet](https://kubernetes.io/docs/concepts/overview/components/#kubelet)
36
+2. [Troubleshooting Kubernetes Clusters](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/)
health/guides/kubelet_node_config_error.md
new
+56
@@ -0,0 +1,56 @@
1
+### Understand the alert
2
+
3
+This alert, `kubelet_node_config_error`, is related to the Kubernetes Kubelet component. If you receive this alert, it means that there is a configuration-related error in one of the nodes in your Kubernetes cluster.
4
+
5
+### What is Kubernetes Kubelet?
6
+
7
+Kubernetes Kubelet is an agent that runs on each node in a Kubernetes cluster. It ensures that containers are running in a pod and manages the lifecycle of those containers.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the node with the configuration error
12
+
13
+ The alert should provide information about the node experiencing the issue. You can also use the `kubectl get nodes` command to list all nodes in your cluster and their statuses:
14
+
15
+ ```
16
+ kubectl get nodes
17
+ ```
18
+
19
+2. Check the Kubelet logs on the affected node
20
+
21
+ The logs for Kubelet can be found on each node of your cluster. Login to the affected node and check its logs using either `journalctl` or the log files in `/var/log/`.
22
+
23
+ ```
24
+ journalctl -u kubelet
25
+ ```
26
+ or
27
+ ```
28
+ sudo cat /var/log/kubelet.log
29
+ ```
30
+
31
+ Look for any error messages related to the configuration issue or other problems.
32
+
33
+3. Review and update the node configuration
34
+
35
+ Based on the error messages you found in the logs, review the Kubelet configuration on the affected node. You might need to update the `kubelet-config.yaml` file or other related files specific to your setup.
36
+
37
+ If any changes are made, don't forget to restart the Kubelet service on the affected node:
38
+
39
+ ```
40
+ sudo systemctl restart kubelet
41
+ ```
42
+
43
+4. Check the health of the cluster
44
+
45
+ After the configuration issue is resolved, make sure to check the health of your cluster using `kubectl`:
46
+
47
+ ```
48
+ kubectl get nodes
49
+ ```
50
+
51
+ Ensure that all nodes are in a `Ready` state and no errors are reported for the affected node.
52
+
53
+### Useful resources
54
+
55
+1. [Kubernetes Documentation: Kubelet](https://kubernetes.io/docs/concepts/overview/components/#kubelet)
56
+2. [Kubernetes Troubleshooting Guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/)
\ No newline at end of file
health/guides/kubelet_operations_error.md
new
+61
@@ -0,0 +1,61 @@
1
+### Understand the alert
2
+
3
+This alert indicates that there is an increase in the number of Docker or runtime operation errors in your Kubernetes cluster's kubelet. A high number of errors can affect the overall stability and performance of your cluster.
4
+
5
+### What are Docker or runtime operation errors?
6
+
7
+Docker or runtime operation errors are errors that occur while the kubelet is managing container-related operations. These errors can be related to creating, starting, stopping, or deleting containers in your Kubernetes cluster.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check kubelet logs:
12
+
13
+ You need to inspect the kubelet logs of the affected nodes to find more information about the reported errors. SSH into the affected node and use the following command to stream the kubelet logs:
14
+
15
+ ```
16
+ journalctl -u kubelet -f
17
+ ```
18
+
19
+ Look for any error messages or patterns that could indicate a problem.
20
+
21
+2. Inspect containers' logs:
22
+
23
+ If an error is related to a specific container, you can inspect the logs of that container using the following command:
24
+
25
+ ```
26
+ kubectl logs <container_name> -n <namespace>
27
+ ```
28
+
29
+ Replace `<container_name>` and `<namespace>` with the appropriate values.
30
+
31
+3. Check Docker or runtime logs:
32
+
33
+ On the affected node, check Docker or container runtime logs for any issues:
34
+
35
+ - For Docker, use: `journalctl -u docker`
36
+ - For containerd, use: `journalctl -u containerd`
37
+ - For CRI-O, use: `journalctl -u crio`
38
+
39
+4. Examine Kubernetes events:
40
+
41
+ Run the following command to see recent events in your cluster:
42
+
43
+ ```
44
+ kubectl get events
45
+ ```
46
+
47
+ Look for any error messages or patterns that could indicate a kubelet or container-related problem.
48
+
49
+5. Verify resource allocation:
50
+
51
+ Ensure that the node has enough resources available (such as CPU, memory, and disk space) for the containers running on it. You can use commands like `kubectl describe node <node_name>` or monitor your cluster resources using Netdata.
52
+
53
+6. Investigate other issues:
54
+
55
+ If the above steps didn't reveal the cause of the errors, investigate other potential causes, such as network issues, filesystem corruption, hardware problems, or misconfigurations.
56
+
57
+### Useful resources
58
+
59
+1. [Kubernetes Debugging and Troubleshooting](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/)
60
+2. [Troubleshoot the Kubelet](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application-introspection/)
61
+3. [Access Clusters Using the Kubernetes API](https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/)
\ No newline at end of file
health/guides/kubelet_token_requests.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+This alert is related to Kubernetes Kubelet token requests. It monitors the number of failed `Token()` requests to an alternate token source. If you receive this alert, it means that your system is experiencing an increased rate of token request failures.
4
+
5
+### What does a token request in Kubernetes mean?
6
+
7
+In Kubernetes, tokens are used for authentication purposes when making requests to the API server. The Kubelet uses tokens to authenticate itself when it needs to access cluster information or manage resources on the API server.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Investigate the reason behind the failed token requests
12
+
13
+1. Check the Kubelet logs for any error messages or warnings related to the token requests. You can use the following command to view the logs:
14
+
15
+ ```
16
+ journalctl -u kubelet
17
+ ```
18
+
19
+ Look for any entries related to `Token()` request failures or authentication issues.
20
+
21
+2. Verify the alternate token source configuration
22
+
23
+ Review the Kubelet configuration file, usually located at `/etc/kubernetes/kubelet/config.yaml`. Check the `authentication` and `authorization` sections to ensure all the required settings have been correctly configured.
24
+
25
+ Make sure that the specified alternate token source is available and working correctly.
26
+
27
+3. Check the API server logs
28
+
29
+ Inspect the logs of the API server to identify any issues that may prevent the Kubelet from successfully requesting tokens. Use the following command to view the logs:
30
+
31
+ ```
32
+ kubectl logs -n kube-system kube-apiserver-<YOUR_NODE_NAME>
33
+ ```
34
+
35
+ Look for any entries related to authentication, especially if they are connected to the alternate token source.
36
+
37
+4. Monitor kubelet_token_requests metric
38
+
39
+ Keep an eye on the `kubelet_token_requests` metric using the Netdata dashboard or a monitoring system of your choice. If the number of failed requests continues to increase, this might indicate an underlying issue that requires further investigation.
40
+
41
+### Useful resources
42
+
43
+1. [Understanding Kubernetes authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/)
44
+2. [Kubelet configuration reference](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/)
health/guides/linux_power_supply/linux_power_supply_capacity.md
deleted
-18
@@ -1,18 +0,0 @@
1
-# linux_power_supply_capacity
2
-
3
-**Power Supply | Battery**
4
-
5
-The Netdata Agent monitors the percentage of remaining power supply capacity. This alert indicates
6
-that the remaining power supply capacity is low. The system can run out of power and switch off
7
-soon as well as prepare the system for the shutdown.
8
-
9
-This alert is triggered in warning state when the percentage of remaining power supply capacity is
10
-less than 10% and in critical state when it is less than 5%.
11
-
12
-### Troubleshooting section:
13
-
14
-Restore power, replace the battery, or even consider integrating a UPS to
15
-support your systems.
16
-
17
-If you receive this alert regularly you may also want to check the battery of the system. It's
18
-capacity may be degraded.
health/guides/linux_power_supply_capacity.md
new
+18
@@ -0,0 +1,18 @@
1
+### Understand the alert
2
+
3
+The `linux_power_supply_capacity` alert is triggered when the remaining power supply capacity of a Linux system is low. A warning state occurs when the capacity falls below 10%, and a critical state occurs when it falls below 5%. This alert indicates that the system may run out of power and shut down soon.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Restore power**: Connect the system to a power source to recharge the battery and prevent an unexpected shutdown.
8
+
9
+2. **Check battery health**: Inspect the health of the system's battery. If the capacity is consistently low or degrading, consider replacing the battery.
10
+
11
+3. **Consider a UPS**: If your system experiences frequent power interruptions, you may want to integrate an uninterruptible power supply (UPS) to provide temporary power and prevent system shutdowns.
12
+
13
+4. **Monitor power supply metrics**: Keep an eye on power supply metrics, such as remaining capacity and charge/discharge rate, to ensure the system is functioning optimally.
14
+
15
+### Useful resources
16
+
17
+1. [Battery Health Monitoring on Linux](https://wiki.archlinux.org/title/Laptop#Battery)
18
+2. [Monitoring Power Supply on Linux](https://askubuntu.com/questions/69556/how-to-check-battery-status-using-terminal)
health/guides/load/load_average_1.md
deleted
-127
@@ -1,127 +0,0 @@
1
-# load_average_1
2
-
3
-## OS: Linux
4
-
5
-This alarm calculates the system `load average` (`CPU` and `I/O` demand) over the period of one minute.
6
-If you receive this alarm, it means that your system is `overloaded`.
7
-
8
-<details>
9
-<summary>What does "load average" mean</summary>
10
-
11
-The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those
12
-waiting to work** (CPU, disk, uninterruptible locks)
13
-<sup> [1](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) </sup>
14
-<sup> [2](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) </sup>
15
-. So simply stated: **it measures the number of threads that aren't idle.**
16
-
17
-</details>
18
-
19
-<details>
20
-<summary>What does "overloaded" mean</summary>
21
-
22
-The term `overloaded` can be better illustrated using an example as ***Andre Lewis*** says in ***Understanding Linux CPU
23
-Load - when should you be worried?***<sup> [3](https://scoutapm.com/blog/understanding-load-averages) </sup>, which you
24
-find in our links section.
25
-
26
-We are going to take a single core CPU system and think of its core count as bridge lanes.
27
-
28
-- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
29
-
30
-
31
-- If the load average is at 1, then the bridge is full, and it is utilized 100%.
32
-
33
-
34
-- If the load average gets to 2 *(remember we are on a single core machine)*, it means that there is one lane that is
35
- passing the bridge and one **other** full lane that waits on the side. On this example, traffic and thus cars, are
36
- processes.
37
-
38
-So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an
39
-analogous example there.
40
-
41
-</details>
42
-
43
-<details>
44
-<summary>How we calculate the alarm</summary>
45
-
46
-On Netdata, in the [load.conf](https://github.com/netdata/netdata/blob/master/health/health.d/load.conf) file, under the
47
-health.d directory, you can see how we calculate *when* the alarm should be raised.
48
-
49
-- First, there is `load_cpu_number` where it provides the `load_average` alarms with the core count of the machine.
50
-
51
-
52
-- In the line `warn: ($this * 100 / $load_cpu_number) > (($status >= $WARNING) ? 700 : 800)`, \
53
- `($this * 100 / $load_cpu_number)` is the current system load average in %.
54
-
55
-
56
-- Last, we check if that value exceeds 700% or 800% (depending on the `$status` of the alarm).
57
-
58
-</details>
59
-
60
-<br>
61
-
62
-<details>
63
-<summary>References</summary>
64
-
65
-[[1] UNIX Load Average Part 1: How It Works](
66
-https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) \
67
-[[2] UNIX Load Average Part 2: Not Your Average Average](
68
-https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) \
69
-[[3] Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages) \
70
-[Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html) \
71
-[Understanding Linux Process States](
72
-https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
73
-</details>
74
-
75
-### Troubleshooting Section
76
-
77
-<details>
78
- <summary>Determine if the problem is CPU or I/O bound</summary>
79
-
80
-First you need to check if you are running on a CPU load or an I/O load problem.
81
-
82
-- You can use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds)
83
-
84
-```
85
-root@netdata~ # vmstat
86
-procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
87
- r b swpd free buff cache si so bi bo in cs us sy id wa st
88
- 8 0 1200384 168456 48840 1461540 4 14 65 51 334 196 3 1 95 0 0
89
-```
90
-
91
-The `procs` column, shows; \
92
-r: The number of runnable processes (running or waiting for run time). \
93
-b: The number of processes blocked waiting for I/O to complete.
94
-
95
-After that, you can use the `ps` and specifically `ps -eo s,user,cmd | grep ^[RD]`.
96
-
97
-- The `grep` command will fetch the processes that their state code starts either with R (running or runnable (on run
98
- queue)) or D(uninterruptible sleep (usually IO)).
99
-
100
-It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
101
-processes you are closing and being certain that they are not necessary.
102
-
103
-</details>
104
-
105
-<details>
106
- <summary>Check per-process CPU/disk usage to find the top consumers</summary>
107
-
108
-1. Use `top`:
109
-
110
- ```
111
- root@netdata~ # top -o +%CPU -i
112
- ```
113
- Here, you can see which processes are the main cpu consumers on the `%CPU` column.
114
-
115
-
116
-2. Use `iotop`: \
117
- `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it,
118
- then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
119
- ```
120
- root@netdata~ # sudo iotop
121
- ```
122
- Using this, you can see which processes are the main Disk I/O consumers on the `IO` column.
123
-
124
-It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
125
-processes you are closing and being certain that they are not necessary.
126
-
127
-</details>
health/guides/load/load_average_15.md
deleted
-104
@@ -1,104 +0,0 @@
1
-# load_average_15
2
-
3
-## OS: Linux
4
-
5
-This alarm calculates the system `load average` (CPU and I/O demand) over the period of fifteen
6
-minutes.
7
-If you receive this alarm, it means that your system is "overloaded."
8
-
9
-The alert gets raised into warning if the metric is 2 times the expected value and cleared if
10
-the value is 1.75 times the expected value.
11
-
12
-For further information on how our alerts are calculated, please have a look at our [Documentation](
13
-https://learn.netdata.cloud/docs/agent/health/reference#expressions).
14
-
15
-
16
-<details>
17
-<summary>What does "load average" mean?</summary>
18
-
19
-The term `system load average` on a Linux machine, measures the **number of threads that are
20
-currently working and those waiting to work** (CPU, disk, uninterruptible locks)<sup> [1](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) </sup><sup> [2](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) </sup>. So simply stated: **System load average measures the number of threads that aren't idle.**
21
-
22
-</details>
23
-
24
-<details>
25
-<summary>What does "overloaded" mean?</summary>
26
-
27
-Andre Lewis explains the term "overloaded" by using an example in his Blog post "Understanding Linux CPU
28
-Load - when should you be worried?"<sup> [3](https://scoutapm.com/blog/understanding-load-averages) </sup>
29
-You can click on the footnote or find it in our links section.
30
-
31
-Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example:
32
-
33
-- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
34
-- If the load average is at 1, then the bridge is full, and it is utilized 100%.
35
-- If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge.
36
-
37
-So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an analogous example there.
38
-
39
-</details>
40
-
41
-<br>
42
-
43
-<details>
44
-<summary>References and Sources</summary>
45
-
46
-1. [UNIX Load Average Part 1: How It Works](
47
- https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works)
48
-2. [UNIX Load Average Part 2: Not Your Average Average](
49
- https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average)
50
-3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages)
51
-4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html)
52
-5. [Understanding Linux Process States](
53
- https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
54
-</details>
55
-
56
-### Troubleshooting Section
57
-
58
-<details>
59
- <summary>Determine if the problem is CPU or I/O bound</summary>
60
-
61
-First you need to check if you are running on a CPU load or an I/O load problem.
62
-
63
-1. To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds):
64
-
65
-```
66
-root@netdata~ # vmstat
67
-procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
68
- r b swpd free buff cache si so bi bo in cs us sy id wa st
69
- 8 0 1200384 168456 48840 1461540 4 14 65 51 334 196 3 1 95 0 0
70
-```
71
-
72
-The `procs` column, shows:
73
-r: The number of runnable processes (running or waiting for run time).
74
-b: The number of processes blocked waiting for I/O to complete.
75
-
76
-2. List your currently running processes using the `ps` command:
77
-
78
-The `grep` command will fetch the processes that their state code starts either with R (running or runnable (on run queue)) or D(uninterruptible sleep (usually IO)).
79
-
80
-3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
81
-
82
-</details>
83
-
84
-<details>
85
- <summary>Check per-process CPU/disk usage to find the top consumers</summary>
86
-
87
-1. To see the processes that are the main CPU consumers, use the task manager program `top` like this:
88
-
89
- ```
90
- root@netdata~ # top -o +%CPU -i
91
- ```
92
-
93
-
94
-2. Use `iotop`:
95
- `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it,
96
- then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
97
- ```
98
- root@netdata~ # sudo iotop
99
- ```
100
-Note: If `iotop` is not installed on your machine, please refer to the [install instructions](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
101
-
102
-3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
103
-
104
-</details>
\ No newline at end of file
health/guides/load/load_average_5.md
deleted
-110
@@ -1,110 +0,0 @@
1
-# load_average_5
2
-
3
-## OS: Linux
4
-
5
-This alarm calculates the system `load average` (CPU and I/O demand) over the period of five minutes.
6
-If you receive this alarm, it means that your system is "overloaded."
7
-
8
-The alert gets raised into warning if the metric is 4 times the expected value and cleared if the value is 3.5 times the expected value.
9
-
10
-For further information on how our alerts are calculated, please have a look at our [Documentation](
11
-https://learn.netdata.cloud/docs/agent/health/reference#expressions).
12
-
13
-
14
-<details>
15
-<summary>What does "load average" mean?</summary>
16
-
17
-The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those
18
-waiting to work** (CPU, disk, uninterruptible locks)
19
-<sup> [1](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works) </sup>
20
-<sup> [2](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average) </sup>
21
-. So simply stated: **System load average measures the number of threads that aren't idle.**
22
-
23
-</details>
24
-
25
-<details>
26
-<summary>What does "overloaded" mean?</summary>
27
-
28
-Andre Lewis explains the term "overloaded" by using an example in his Blog post "Understanding Linux CPU
29
-Load - when should you be worried?"<sup> [3](https://scoutapm.com/blog/understanding-load-averages) </sup>
30
-You can click on the footnote or
31
-find it in our links section.
32
-
33
-Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example:
34
-
35
-- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
36
-- If the load average is at 1, then the bridge is full, and it is utilized 100%.
37
-- If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge.
38
-
39
-So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an
40
-analogous example there.
41
-
42
-</details>
43
-
44
-
45
-
46
-<br>
47
-
48
-<details>
49
-<summary>References and Sources</summary>
50
-
51
-1. [UNIX Load Average Part 1: How It Works](
52
-https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works)
53
-2. [UNIX Load Average Part 2: Not Your Average Average](
54
-https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average)
55
-3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages)
56
-4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html)
57
-5. [Understanding Linux Process States](
58
-https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
59
-</details>
60
-
61
-### Troubleshooting Section
62
-
63
-<details>
64
- <summary>Determine if the problem is CPU or I/O bound</summary>
65
-
66
-First you need to check if you are running on a CPU load or an I/O load problem.
67
-
68
-1. To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds):
69
-
70
-```
71
-root@netdata~ # vmstat
72
-procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
73
- r b swpd free buff cache si so bi bo in cs us sy id wa st
74
- 8 0 1200384 168456 48840 1461540 4 14 65 51 334 196 3 1 95 0 0
75
-```
76
-
77
-The `procs` column, shows:
78
-r: The number of runnable processes (running or waiting for run time).
79
-b: The number of processes blocked waiting for I/O to complete.
80
-
81
-2. List your currently running processes using the `ps` command:
82
-
83
-The `grep` command will fetch the processes that their state code starts either with R (running or runnable (on run
84
- queue)) or D(uninterruptible sleep (usually IO)).
85
-
86
-3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
87
-
88
-</details>
89
-
90
-<details>
91
- <summary>Check per-process CPU/disk usage to find the top consumers</summary>
92
-
93
-1. To see the processes that are the main CPU consumers, use the task manager program `top` like this:
94
-
95
- ```
96
- root@netdata~ # top -o +%CPU -i
97
- ```
98
-
99
-
100
-2. Use `iotop`:
101
- `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it,
102
- then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
103
- ```
104
- root@netdata~ # sudo iotop
105
- ```
106
-Note: If `iotop` is not installed on your machine, please refer to the [install instructions](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
107
-
108
-3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
109
-
110
-</details>
health/guides/load_average_1.md
new
+51
@@ -0,0 +1,51 @@
1
+### Understand the alert
2
+
3
+This alarm calculates the system `load average` (`CPU` and `I/O` demand) over the period of one minute. If you receive this alarm, it means that your system is `overloaded`.
4
+
5
+### What does "load average" mean?
6
+
7
+The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those waiting to work** (CPU, disk, uninterruptible locks). So simply stated: **System load average measures the number of threads that aren't idle.**
8
+
9
+### What does "overloaded" mean?
10
+
11
+Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example:
12
+
13
+- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
14
+- If the load average is at 1, then the bridge is full, and it is utilized 100%.
15
+- If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge.
16
+
17
+So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an analogous example there.
18
+
19
+### Troubleshoot the alert
20
+
21
+- Determine if the problem is CPU load or I/O load
22
+
23
+To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds):
24
+
25
+The `procs` column, shows:
26
+r: The number of runnable processes (running or waiting for run time).
27
+b: The number of processes blocked waiting for I/O to complete.
28
+
29
+- Check per-process CPU/disk usage to find the top consumers
30
+
31
+1. To see the processes that are the main CPU consumers, use the task manager program `top` like this:
32
+
33
+ ```
34
+ top -o +%CPU -i
35
+ ```
36
+
37
+2. Use `iotop`:
38
+ `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it, then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
39
+ ```
40
+ sudo iotop
41
+ ```
42
+
43
+3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
44
+
45
+### Useful resources
46
+
47
+1. [UNIX Load Average Part 1: How It Works](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works)
48
+2. [UNIX Load Average Part 2: Not Your Average Average](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average)
49
+3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages)
50
+4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html)
51
+5. [Understanding Linux Process States](https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
health/guides/load_average_15.md
new
+55
@@ -0,0 +1,55 @@
1
+### Understand the alert
2
+
3
+This alarm calculates the system `load average` (CPU and I/O demand) over the period of fifteen minutes. If you receive this alarm, it means that your system is "overloaded."
4
+
5
+The alert gets raised into warning if the metric is 2 times the expected value and cleared if the value is 1.75 times the expected value.
6
+
7
+For further information on how our alerts are calculated, please have a look at our [Documentation](https://learn.netdata.cloud/docs/agent/health/reference#expressions).
8
+
9
+### What does "load average" mean?
10
+
11
+The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those waiting to work** (CPU, disk, uninterruptible locks). So simply stated: **System load average measures the number of threads that aren't idle.**
12
+
13
+### What does "overloaded" mean?
14
+
15
+Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example:
16
+
17
+- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
18
+- If the load average is at 1, then the bridge is full, and it is utilized 100%.
19
+- If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge.
20
+
21
+So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an analogous example there.
22
+
23
+### Troubleshoot the alert
24
+
25
+- Determine if the problem is CPU load or I/O load
26
+
27
+To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds):
28
+
29
+The `procs` column, shows:
30
+r: The number of runnable processes (running or waiting for run time).
31
+b: The number of processes blocked waiting for I/O to complete.
32
+
33
+- Check per-process CPU/disk usage to find the top consumers
34
+
35
+1. To see the processes that are the main CPU consumers, use the task manager program `top` like this:
36
+
37
+ ```
38
+ top -o +%CPU -i
39
+ ```
40
+
41
+2. Use `iotop`:
42
+ `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it, then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
43
+ ```
44
+ sudo iotop
45
+ ```
46
+
47
+3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
48
+
49
+### Useful resources
50
+
51
+1. [UNIX Load Average Part 1: How It Works](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works)
52
+2. [UNIX Load Average Part 2: Not Your Average Average](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average)
53
+3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages)
54
+4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html)
55
+5. [Understanding Linux Process States](https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
health/guides/load_average_5.md
new
+66
@@ -0,0 +1,66 @@
1
+### Understand the alert
2
+
3
+This alarm calculates the system `load average` (CPU and I/O demand) over the period of five minutes. If you receive this alarm, it means that your system is "overloaded."
4
+
5
+The alert gets raised into warning if the metric is 4 times the expected value and cleared if the value is 3.5 times the expected value.
6
+
7
+For further information on how our alerts are calculated, please have a look at our [Documentation](https://learn.netdata.cloud/docs/agent/health/reference#expressions).
8
+
9
+
10
+### What does "load average" mean?
11
+
12
+The term `system load average` on a Linux machine, measures the **number of threads that are currently working and those waiting to work** (CPU, disk, uninterruptible locks). So simply stated: **System load average measures the number of threads that aren't idle.**
13
+
14
+### What does "overloaded" mean?
15
+
16
+Let's look at a single core CPU system and think of its core count as car lanes on a bridge. A car represents a process in this example:
17
+
18
+- On a 0.5 load average, the traffic on the bridge is fine, it is at 50% of its capacity.
19
+- If the load average is at 1, then the bridge is full, and it is utilized 100%.
20
+- If the load average gets to 2 (remember we are on a single core machine), it means that there is one car lane that is passing the bridge. However, there is **another** full car lane that waits to pass the bridge.
21
+
22
+So this is how you can imagine CPU load, but keep in mind that `load average` counts also I/O demand, so there is an analogous example there.
23
+
24
+### Useful resources
25
+
26
+1. [UNIX Load Average Part 1: How It Works](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works)
27
+2. [UNIX Load Average Part 2: Not Your Average Average](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average)
28
+3. [Understanding Linux CPU Load](https://scoutapm.com/blog/understanding-load-averages)
29
+4. [Linux Load Averages: Solving the Mystery](https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html)
30
+5. [Understanding Linux Process States](https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
31
+
32
+
33
+### Troubleshoot the alert
34
+
35
+- Determine if the problem is CPU or I/O bound
36
+
37
+First you need to check if you are running on a CPU load or an I/O load problem.
38
+
39
+1. To get a report about your system statistics, use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds):
40
+
41
+The `procs` column, shows:
42
+r: The number of runnable processes (running or waiting for run time).
43
+b: The number of processes blocked waiting for I/O to complete.
44
+
45
+2. List your currently running processes using the `ps` command:
46
+
47
+The `grep` command will fetch the processes that their state code starts either with R (running or runnable (on run queue)) or D(uninterruptible sleep (usually IO)).
48
+
49
+3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
50
+
51
+- Check per-process CPU/disk usage to find the top consumers
52
+
53
+1. To see the processes that are the main CPU consumers, use the task manager program `top` like this:
54
+
55
+ ```
56
+ top -o +%CPU -i
57
+ ```
58
+
59
+2. Use `iotop`:
60
+ `iotop` is a useful tool, similar to `top`, used to monitor Disk I/O usage, if you don't have it, then [install it](https://www.tecmint.com/iotop-monitor-linux-disk-io-activity-per-process/)
61
+ ```
62
+ sudo iotop
63
+ ```
64
+
65
+3. Minimize the load by closing any unnecessary main consumer processes. We strongly advise you to double-check if the process you want to close is necessary.
66
+
health/guides/load_cpu_number.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+This alert, `load_cpu_number`, calculates the base trigger point for load average alarms, which helps identify when the system is overloaded. The alert checks the maximum number of CPUs in the system over the past 1 minute. If there is only one CPU, the trigger is set at 2.
4
+
5
+### What does load average mean?
6
+
7
+The term `system load average` on a Linux machine measures the number of threads that are currently working and those waiting to work (CPU, disk, uninterruptible locks). In simpler terms, the load average measures the number of threads that aren't idle.
8
+
9
+### What does overloaded mean?
10
+
11
+An overloaded system is when the demand on the system's resources (CPUs, disks, etc.) is higher than its capacity to handle tasks. This can lead to increased wait times, slower processing, and in worst cases, system crashes.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Determine the current load average on the system:
16
+
17
+ Use the `uptime` command in the terminal to see the current load average:
18
+ ```
19
+ uptime
20
+ ```
21
+
22
+2. Identify if the problem is CPU load or I/O load:
23
+
24
+ Use `vmstat` (or `vmstat 1`, to set a delay between updates in seconds) to get a report on system statistics:
25
+
26
+ The `procs` column shows:
27
+ r: The number of runnable processes (running or waiting for run time).
28
+ b: The number of processes blocked waiting for I/O to complete.
29
+
30
+3. Check per-process CPU/disk usage to find the top consumers:
31
+
32
+ a. Use `top` to see the processes that are the main CPU consumers:
33
+ ```
34
+ top -o +%CPU -i
35
+ ```
36
+
37
+ b. Use `iotop` to monitor Disk I/O usage (install it if not available):
38
+ ```
39
+ sudo iotop
40
+ ```
41
+
42
+4. Minimize the load by closing any unnecessary main consumer processes. Double-check if the process you want to close is necessary.
43
+
44
+### Useful resources
45
+
46
+1. [Unix Load Average Part 1: How It Works](https://www.helpsystems.com/resources/guides/unix-load-average-part-1-how-it-works)
47
+2. [Unix Load Average Part 2: Not Your Average Average](https://www.helpsystems.com/resources/guides/unix-load-average-part-2-not-your-average-average)
48
+3. [Understanding Linux Process States](https://access.redhat.com/sites/default/files/attachments/processstates_20120831.pdf)
\ No newline at end of file
health/guides/lowest_entropy.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert presents the minimum amount of entropy in the kernel entropy pool in the last 5 minutes. Low entropy can lead to a reduction in the quality of random numbers produced by `/dev/random` and `/dev/urandom`.
4
+
5
+The Netdata Agent checks for the minimum entropy value in the last 5 minutes. The alert gets raised into warning if the value < 100, and cleared if the value > 200.
6
+
7
+For further information on how our alerts are calculated, please have a look at our [Documentation](https://learn.netdata.cloud/docs/agent/health/reference#expressions).
8
+
9
+### What is entropy and why do we need it?
10
+
11
+Entropy is similar to "randomness". A Linux system gathers "real" random numbers by keeping an eye on different events: network activity, hard drive rotation speeds, hardware random number generator (if available), key-clicks, and so on. It feeds those to the kernel entropy pool, which is used by `/dev/random`.
12
+
13
+Encryption and cryptography applications require random numbers to operate. A function or an algorithm that produces numbers -*that seem to be random*- is very predictable, if you know what function is used.
14
+
15
+In real life, we use our surroundings and our thoughts to produce truly random numbers. A computer can't really do this by itself, so it gathers numbers from a lot of sources. For example, it can get the CO2 levels in a room from a sensor on the system and use that as a random number.
16
+
17
+This way all the values are random and there is no pattern to be found among them.
18
+
19
+### Troubleshoot the alert
20
+
21
+The best tool to troubleshoot the lowest entropy alert is with `rng-tools`.
22
+
23
+If `rng-tools` are not available for your platform, or you run into trouble, you can use the tool `haveged` as an alternative.
24
+
25
+### Useful resources
26
+
27
+1. [Entropy](https://unixhealthcheck.com/blog?id=472)
28
+2. [rng-tools](https://github.com/nhorman/rng-tools)
29
+3. [How to add more entropy to improve cryptographic randomness on Linux](https://www.techrepublic.com/article/how-to-add-more-entropy-to-improve-cryptographic-randomness-on-linux/)
30
+4. [Haveged Installation - Archlinux Wiki](https://wiki.archlinux.org/title/Haveged#Installation)
health/guides/mdstat/mdstat_disks.md
deleted
-57
@@ -1,57 +0,0 @@
1
-# mdstat_disks
2
-
3
-## OS: Any
4
-
5
-This alert presents the number of devices in the down state for the respective RAID array raising
6
-it.
7
-If you receive this alert, then the array is degraded and some array devices are missing.
8
-
9
-- This alert is escalated to a warning when there are failed devices.
10
-
11
-<details>
12
-<summary>What is a "degraded array" event?</summary>
13
-
14
-> When a RAID array experiences the failure of one or more disks, it can enter degraded mode, a
15
-> fallback mode that generally allows the continued usage of the array, but either loses the
16
-> performance boosts of the RAID technique (such as a RAID-1 mirror across two disks when one of
17
-> them fails; performance will fall back to that of a normal, single drive) or experiences severe
18
-> performance penalties due to the necessity to reconstruct the damaged data from error correction
19
-> data.<sup>[1](https://en.wikipedia.org/wiki/Degraded_mode) </sup>
20
-
21
-</details>
22
-
23
-<br>
24
-
25
-<details>
26
-<summary>References and Sources</summary>
27
-
28
-1. [Degraded Mode](https://en.wikipedia.org/wiki/Degraded_mode)
29
-2. [Mdadm recover degraded array procedure](
30
- https://www.thomas-krenn.com/en/wiki/Mdadm_recover_degraded_Array_procedure)
31
-3. [mdadm Manual page](https://linux.die.net/man/8/mdadm)
32
-4. [mdadm cheat sheet](https://www.ducea.com/2009/03/08/mdadm-cheat-sheet/)
33
-
34
-</details>
35
-
36
-
37
-### Troubleshooting Section
38
-
39
-<details>
40
-<summary>Examine for faulty or offline devices</summary>
41
-
42
-Having a degraded array means that one or more devices are faulty or missing.
43
-To fix this issue, check for faulty devices by running:
44
-
45
-```
46
-root@netdata~ # mdadm --detail <RAIDDEVICE>
47
-```
48
-
49
-Replace "RAIDDEVICE" with the name of your RAID device.
50
-
51
-To recover the array, replace the faulty devices or bring back any offline
52
-devices.
53
-For more information check: [Mdadm recover degraded array procedure](
54
-https://www.thomas-krenn.com/en/wiki/Mdadm_recover_degraded_Array_procedure)
55
-
56
-</details>
57
-
health/guides/mdstat/mdstat_mismatch_cnt.md
deleted
-28
@@ -1,28 +0,0 @@
1
-# mdstat_mismatch_cnt
2
-
3
-## OS: Any
4
-
5
-This alert presents the number of unsynchronized blocks for the RAID array in crisis.
6
-Receiving this alert indicates a high number of unsynchronized blocks for the RAID array.
7
-This might indicate that data on the array is corrupted.
8
-
9
-This alert is raised to warning when the metric exceeds 1024 unsynchronized blocks.
10
-
11
-<details><summary>References and Sources</summary>
12
-
13
-[Serverfault | Reasons for high mismatch_cnt on a RAID1/10 array](
14
-https://serverfault.com/questions/885565/what-are-raid-1-10-mismatch-cnt-0-causes-except-for-swap-file/885574#885574).
15
-
16
-</details>
17
-
18
-## Troubleshooting section
19
-
20
-There is no standard approach to troubleshooting this alert because the reasons can be various.
21
-
22
-For example, one of the reasons might be a swap on the array, which is relatively harmless.
23
-However, this alert can also be triggered by hardware issues which can lead to many
24
-problems and inconsistencies between the disks.
25
-
26
-You might find some troubleshooting ideas in this thread: [Reasons for high mismatch_cnt on a RAID1/10
27
-array](
28
-https://serverfault.com/questions/885565/what-are-raid-1-10-mismatch-cnt-0-causes-except-for-swap-file/885574#885574).
\ No newline at end of file
health/guides/mdstat_disks.md
new
+26
@@ -0,0 +1,26 @@
1
+### Understand the alert
2
+
3
+This alert presents the number of devices in the down state for the respective RAID array raising it. If you receive this alert, then the array is degraded and some array devices are missing.
4
+
5
+### What is a "degraded array" event?
6
+
7
+When a RAID array experiences the failure of one or more disks, it can enter degraded mode, a fallback mode that generally allows the continued usage of the array, but either loses the performance boosts of the RAID technique (such as a RAID-1 mirror across two disks when one of them fails; performance will fall back to that of a normal, single drive) or experiences severe performance penalties due to the necessity to reconstruct the damaged data from error correction data.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Examine for faulty or offline devices
12
+
13
+Having a degraded array means that one or more devices are faulty or missing. To fix this issue, check for faulty devices by running:
14
+```
15
+mdadm --detail <RAIDDEVICE>
16
+```
17
+Replace "RAIDDEVICE" with the name of your RAID device.
18
+
19
+To recover the array, replace the faulty devices or bring back any offline devices.
20
+
21
+### Useful resources
22
+
23
+1. [Degraded Mode](https://en.wikipedia.org/wiki/Degraded_mode)
24
+2. [Mdadm recover degraded array procedure](https://www.thomas-krenn.com/en/wiki/Mdadm_recover_degraded_Array_procedure)
25
+3. [mdadm Manual page](https://linux.die.net/man/8/mdadm)
26
+4. [mdadm cheat sheet](https://www.ducea.com/2009/03/08/mdadm-cheat-sheet/)
\ No newline at end of file
health/guides/mdstat_last_collected.md
new
+49
@@ -0,0 +1,49 @@
1
+### Understand the alert
2
+
3
+The `mdstat_last_collected` alert is generated when there is a delay or absence of data collection from the Multiple Device (md) driver for an extended period of time. This can be a sign of an issue with the RAID array or the system itself.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the status of the RAID array
8
+
9
+ The status of the RAID array can be checked using the following command:
10
+
11
+ ```
12
+ cat /proc/mdstat
13
+ ```
14
+
15
+ This will display the RAID array's current status, including any errors, degraded state, or rebuilding progress.
16
+
17
+2. Ensure the Netdata Agent is running
18
+
19
+ Verify that the Netdata Agent is running and collecting data from the system using the following command:
20
+
21
+ ```
22
+ sudo systemctl status netdata
23
+ ```
24
+
25
+ If the Netdata Agent is not running, start it using:
26
+
27
+ ```
28
+ sudo systemctl start netdata
29
+ ```
30
+
31
+3. Check if the `mdstat` plugin is enabled in `/etc/netdata/netdata.conf`
32
+
33
+ Ensure that the plugin responsible for collecting data from the md driver is enabled. Look for the following lines in `/etc/netdata/netdata.conf`:
34
+
35
+ ```
36
+ [plugin:proc:/proc/mdstat]
37
+ dedicated lines for md devices = no (auto)
38
+ ```
39
+
40
+ Make sure that the option is set as shown above.
41
+
42
+4. Check for any hardware issues or faulty disks
43
+
44
+ If the RAID array status shows errors or a degraded state, investigate the disks and the RAID controller for any hardware issues or failures. If needed, replace the faulty disk and rebuild the array.
45
+
46
+5. Monitor the RAID array and system status
47
+
48
+ Keep an eye on the RAID array's status and overall system health. If the issue persists or worsens, consider scheduling downtime for further diagnostics and maintenance.
49
+
health/guides/mdstat_mismatch_cnt.md
new
+15
@@ -0,0 +1,15 @@
1
+### Understand the alert
2
+
3
+This alert presents the number of unsynchronized blocks for the RAID array in crisis. Receiving this alert indicates a high number of unsynchronized blocks for the RAID array. This might indicate that data on the array is corrupted.
4
+
5
+This alert is raised to warning when the metric exceeds 1024 unsynchronized blocks.
6
+
7
+### Troubleshoot the alert
8
+
9
+There is no standard approach to troubleshooting this alert because the reasons can be various.
10
+
11
+For example, one of the reasons might be a swap on the array, which is relatively harmless. However, this alert can also be triggered by hardware issues which can lead to many problems and inconsistencies between the disks.
12
+
13
+### Useful resources
14
+
15
+[Serverfault | Reasons for high mismatch_cnt on a RAID1/10 array](https://serverfault.com/questions/885565/what-are-raid-1-10-mismatch-cnt-0-causes-except-for-swap-file/885574#885574)
health/guides/mdstat_nonredundant_last_collected.md
new
+55
@@ -0,0 +1,55 @@
1
+### Understand the alert
2
+
3
+This alert, `mdstat_nonredundant_last_collected`, is triggered when the Netdata Agent fails to collect data from the Multiple Device (md) driver for a certain period. The md driver is used to manage software RAID arrays in Linux.
4
+
5
+### What is the md driver?
6
+
7
+The md (multiple device) driver is responsible for managing software RAID arrays on Linux systems. It provides a way to combine multiple physical disks into a single logical disk, increasing capacity and providing redundancy, depending on the RAID level. Monitoring the status of these devices is crucial to ensure data integrity and redundancy.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the status of the md driver:
12
+
13
+ To inspect the status of the RAID arrays managed by the md driver, use the `cat` command:
14
+
15
+ ```
16
+ cat /proc/mdstat
17
+ ```
18
+
19
+ This will display the status and configuration of all active RAID arrays. Look for any abnormal status, such as failed or degraded disks, and replace or fix them as needed.
20
+
21
+2. Verify the Netdata configuration:
22
+
23
+ Ensure that the Netdata Agent is properly configured to collect data from the md driver. Open the `netdata.conf` configuration file found in `/etc/netdata/` or `/opt/netdata/etc/netdata/`, and look for the `[plugin:proc:/proc/mdstat]` section.
24
+
25
+ Make sure that the `enabled` option is set to `yes`:
26
+
27
+ ```
28
+ [plugin:proc:/proc/mdstat]
29
+ # enabled = yes
30
+ ```
31
+
32
+ If you make any changes to the configuration, restart the Netdata Agent for the changes to take effect:
33
+
34
+ ```
35
+ sudo systemctl restart netdata
36
+ ```
37
+
38
+3. Check the md driver data collection:
39
+
40
+ After verifying the Netdata configuration, check if data collection is successful. On the Netdata dashboard, go to the "Disks" section, and look for "mdX" (where "X" is a number) in the list of available disks. If you can see the charts for your RAID array(s), it means data collection is working correctly.
41
+
42
+4. Investigate system logs:
43
+
44
+ If the issue persists, check the system logs for any errors or messages related to the md driver or Netdata Agent. You can use `journalctl` for this purpose:
45
+
46
+ ```
47
+ journalctl -u netdata
48
+ ```
49
+
50
+ Look for any error messages or warnings that could indicate the cause of the problem.
51
+
52
+### Useful resources
53
+
54
+1. [Linux RAID: A Quick Guide](https://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html)
55
+2. [Netdata Agent Configuration Guide](https://learn.netdata.cloud/docs/agent/daemon/config)
health/guides/megacli/megacli_adapter_state.md
deleted
-53
@@ -1,53 +0,0 @@
1
-# megacli_adapter_state
2
-
3
-## OS: Any
4
-
5
-A disk array controller is a device that manages the physical disk drives and presents them to the
6
-computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred
7
-to as RAID controller. It also often provides additional disk cache.
8
-
9
-The Netdata Agent checks the status of your MegaRAID controller by scraping the output of
10
-the `megacli -LDPDInfo -aAll` command. This alert indicates that the status of a virtual drive is in
11
-the degraded state (0: false, 1:true).
12
-
13
-#### States of a virtual drive:
14
-
15
-| State | Description |
16
-|:----------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
17
-| Optimal | The virtual drive operating condition is good. All configured drives are online. |
18
-| Degraded | The virtual drive operating condition is not optimal. One of the configured drives has failed or is offline. |
19
-| Partial Degraded | The operating condition in a RAID 6 virtual drive is not optimal. One of the configured drives has failed or is offline. RAID 6 can tolerate up to two drive failures. |
20
-| Failed | The virtual drive has failed. |
21
-| Offline | The virtual drive is not available to the RAID controller. |
22
-
23
-<details>
24
-<summary>References and source</summary>
25
-
26
-1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
27
-2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
28
-
29
-
30
-</details>
31
-
32
-### Troubleshooting section:
33
-
34
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
35
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
36
-
37
-<details>
38
- <summary>General approach</summary>
39
-
40
-1. Gather more information about your virtual drives in all adapters
41
-
42
- ```
43
- root@netdata # megacli –LDInfo -Lall -aALL
44
- ```
45
-
46
-2. Check which virtual drive is in degraded state and in which adapter
47
-
48
-3. Consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
49
- 1. section `2.1.16` to check what is going wrong with your drives.
50
- 2. section `7.18` to perform any action in drives. Focus on {`7.18.2`,`7.18.6`,`7.18.7`,`7.18.8`
51
- ,`7.18.11`,`7.18.14`}
52
-
53
-</details>
\ No newline at end of file
health/guides/megacli/megacli_bbu_cycle_count.md
deleted
-46
@@ -1,46 +0,0 @@
1
-# megacli_bbu_cycle_count
2
-
3
-## OS: Any
4
-
5
-This is an alert about the battery backup unit in the MegaCLI controller. The Netdata Agent monitors
6
-the average battery backup unit charge cycles count over the last 10 seconds. This alert indicates
7
-that a high number of full recharge cycles have been elapsed in the unit's lifetime. This metrics
8
-may affect the battery relative capacity.
9
-
10
-This alert is triggered in warning state when the number of charge cycles is greater than 100 and in
11
-critical state when it is greater than 500.
12
-
13
-<details>
14
-<summary>References and source</summary>
15
-
16
-1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
17
-2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
18
-
19
-</details>
20
-
21
-### Troubleshooting section:
22
-
23
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
24
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
25
-
26
-<details>
27
- <summary>General approach</summary>
28
-
29
-1. Gather more information about your battery units in all of your adapters
30
-
31
- ```
32
- root@netdata # megacli -AdpBbuCmd -GetBbuStatus -aALL
33
- ```
34
-
35
-2. Perform a battery check in the battery which had low relative charge. **Before perform any
36
- action, consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
37
- section {`7.14`}**
38
-
39
- ```
40
- root@netdata # megacli -AdpBbuCmd -BbuLearn -aX // X is the adaptor's number
41
- ```
42
-
43
-3. Replace the battery in question if needed.
44
-
45
-</details>
46
-
health/guides/megacli/megacli_bbu_relative_charge.md
deleted
-50
@@ -1,50 +0,0 @@
1
-# megacli_bbu_relative_charge
2
-
3
-## OS: Any
4
-
5
-A disk array controller is a device that manages the physical disk drives and presents them to the
6
-computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred
7
-to as RAID controller. It also often provides additional disk cache.
8
-
9
-The Netdata Agent calculates the average battery backup unit relative state of charge over the last
10
-10 seconds. This alert indicates that the state of charge is low. The relative state of charge is an
11
-indication of full charge capacity percentage in relation to the design capacity. A constantly low
12
-value may indicate that the battery is worn out. You might want to consider changing the battery.
13
-
14
-This alert is raised into warning when the relative state of charge of a battery is below 80% and in
15
-critical when it is below 50%.
16
-
17
-
18
-<details>
19
-<summary>References and source</summary>
20
-
21
-1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
22
-2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
23
-
24
-</details>
25
-
26
-### Troubleshooting section:
27
-
28
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
29
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
30
-
31
-<details>
32
- <summary>General approach</summary>
33
-
34
-1. Gather more information about your battery units in all of your adapters
35
-
36
- ```
37
- root@netdata # megacli -AdpBbuCmd -GetBbuStatus -aALL
38
- ```
39
-
40
-2. Perform a battery check in the battery which had low relative charge. **Before perform any
41
- action, consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
42
- section {`7.14`}**
43
-
44
- ```
45
- root@netdata # megacli -AdpBbuCmd -BbuLearn -aX // X is the adaptor's number
46
- ```
47
-
48
-3. Replace the battery in question if needed.
49
-
50
-</details>
health/guides/megacli/megacli_pd_media_errors.md
deleted
-64
@@ -1,64 +0,0 @@
1
-# megacli_pd_media_errors
2
-
3
-## OS: Any
4
-
5
-A disk array controller is a device that manages the physical disk drives and presents them to the
6
-computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred
7
-to as RAID controller. It also often provides additional disk cache.
8
-
9
-A media error is an event where a storage disk was unable to perform the requested I/O operation
10
-because of problems accessing the stored data.
11
-
12
-This is an alert about the physical disks attached to the MegaCLI controller. The Netdata Agent
13
-monitors the number of physical drive media errors. This alert indicates that a bad sector was found
14
-on the drive during a patrol check or from a rebuild operation on a specific disk by the raid
15
-adapter.
16
-
17
-This alert is raised into warning if any media error occur. This doesn't mean that there is an
18
-imminent disk failure, but you should keep an eye on this particular disk
19
-
20
-<details>
21
-<summary> More about media errors </summary>
22
-
23
-Media errors are more common on read transactions but might occur on writes as well. A media error
24
-on a `write` may occur when the disk has problems locating the position to write the data. On reads,
25
-in addition to these positioning faults, the disk may experience problems retrieving the data. When
26
-a disk writes data, it writes other information as well, such as to record the position, note CRC or
27
-checksum to confirm data write integrity.
28
-
29
-</details>
30
-
31
-<details>
32
-<summary>References and source</summary>
33
-
34
-1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
35
-2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
36
-
37
-</details>
38
-
39
-### Troubleshooting section:
40
-
41
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
42
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
43
-
44
-<details>
45
- <summary>General approach</summary>
46
-
47
-1. Gather more information about your virtual drives in all adapters
48
-
49
- ```
50
- root@netdata # megacli –LDInfo -Lall -aALL
51
- ```
52
-
53
-2. Check which virtual drive is reporting media errors and in which adapter
54
-
55
-3. Check the Bad block table for the virtual drive in question
56
-
57
- ```
58
- root@netdata # megacli –GetBbtEntries -LX -aY // X: virtual drive , Y the adapter
59
- ```
60
-
61
-4. Consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
62
- section `7.17.11` to recheck these block entries. **This operation removes any data stored on the
63
- physical drives. Back up the good data on the drives before making any changes to the
64
- configuration**
\ No newline at end of file
health/guides/megacli/megacli_pd_predictive_failures.md
deleted
-47
@@ -1,47 +0,0 @@
1
-# megacli_pd_predictive_failures
2
-
3
-## OS: Any
4
-
5
-A disk array controller is a device that manages the physical disk drives and presents them to the
6
-computer as logical units. It almost always implements hardware RAID, thus it is sometimes referred
7
-to as RAID controller. It also often provides additional disk cache.
8
-
9
-A predictive drive failure (self-monitoring analysis and reporting
10
-technology [S.M.A.R.T.](https://en.wikipedia.org/wiki/S.M.A.R.T.#:~:text=(Self%2DMonitoring%2C%20Analysis%20and,SSDs)%2C%20and%20eMMC%20drives)
11
-error).
12
-
13
-This is an alert about the physical disks attached to the MegaCLI controller. The Netdata Agent
14
-calculates the number of physical drive predictive failures. The failure prediction function for the
15
-hard disk drives determines the risk of a failure in advance and issues a warning when the risk is
16
-high. A hard disk can still operate normally but may fail in the near future. You might want to
17
-consider replacing the disk.
18
-
19
-<details>
20
-<summary>References and source</summary>
21
-
22
-1. [MegaRAID SAS Software User Guide \[pdf download\]](https://docs.broadcom.com/docs/12353236)
23
-2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
24
-
25
-</details>
26
-
27
-### Troubleshooting section:
28
-
29
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
30
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
31
-
32
-<details>
33
- <summary>General approach</summary>
34
-
35
-1. Gather more information about your virtual drives in all adapters
36
-
37
- ```
38
- root@netdata # megacli –LDInfo -Lall -aALL
39
- ```
40
-
41
-2. Check which virtual drive is reporting media errors and in which adapter
42
-
43
-
44
-3. Consult the manual's <sup>[1](https://docs.broadcom.com/docs/12353236) </sup>
45
- 1. section `2.1.16` to check what is going wrong with your drives.
46
- 2. section `7.18` to perform any action in drives. Focus on {`7.18.2`,`7.18.6`,`7.18.7`,`7.18.8`
47
- ,`7.18.11`,`7.18.14`}
health/guides/megacli_adapter_state.md
new
+29
@@ -0,0 +1,29 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the status of a virtual drive on your MegaRAID controller is in a degraded state. A degraded state means that the virtual drive's operating condition is not optimal, and one of the configured drives has failed or is offline.
4
+
5
+### Troubleshoot the alert
6
+
7
+#### General approach
8
+
9
+1. Gather more information about your virtual drives in all adapters:
10
+
11
+```
12
+root@netdata # megacli –LDInfo -Lall -aALL
13
+```
14
+
15
+2. Check which virtual drive is in a degraded state and in which adapter.
16
+
17
+3. Consult the MegaRAID SAS Software User Guide [1]:
18
+
19
+ 1. Section `2.1.16` to check what is going wrong with your drives.
20
+ 2. Section `7.18` to perform any action on drives. Focus on sections `7.18.2`, `7.18.6`, `7.18.7`, `7.18.8`, `7.18.11`, and `7.18.14`.
21
+
22
+### Warning
23
+
24
+Data is priceless. Before performing any action, make sure that you have taken any necessary backup steps. Netdata is not liable for any loss or corruption of any data, database, or software.
25
+
26
+### Useful resources
27
+
28
+1. [MegaRAID SAS Software User Guide [PDF download]](https://docs.broadcom.com/docs/12353236)
29
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
\ No newline at end of file
health/guides/megacli_bbu_cycle_count.md
new
+28
@@ -0,0 +1,28 @@
1
+### Understand the alert
2
+
3
+The `megacli_bbu_cycle_count` alert is related to the battery backup unit (BBU) of your MegaCLI controller. This alert is triggered when the average number of full recharge cycles during the BBU's lifetime exceeds a predefined threshold. High numbers of charge cycles can affect the battery's relative capacity.
4
+
5
+A warning state is triggered when the number of charge cycles is greater than 100, and a critical state is triggered when the number of charge cycles is greater than 500.
6
+
7
+### Troubleshoot the alert
8
+
9
+**Caution:** Before performing any troubleshooting steps, ensure that you have taken the necessary backup measures to protect your data. Netdata is not liable for any data loss or corruption.
10
+
11
+1. Gather information about the battery units for all of your adapters:
12
+
13
+ ```
14
+ megacli -AdpBbuCmd -GetBbuStatus -aALL
15
+ ```
16
+
17
+2. Perform a battery check on the BBU with a low relative charge. Before taking any action, consult the manual's[section 7.14](https://docs.broadcom.com/docs/12353236):
18
+
19
+ ```
20
+ megacli -AdpBbuCmd -BbuLearn -aX // X is the adapter's number
21
+ ```
22
+
23
+3. If necessary, replace the battery in question.
24
+
25
+### Useful resources
26
+
27
+1. [MegaRAID SAS Software User Guide (PDF download)](https://docs.broadcom.com/docs/12353236)
28
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
\ No newline at end of file
health/guides/megacli_bbu_relative_charge.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alert is related to the disk array controller's battery backup unit (BBU) relative state of charge. If you receive this alert, it means that the battery backup unit's charge is low, which may affect your RAID controller's performance or lead to data loss in case of a power failure.
4
+
5
+### What does low BBU relative charge mean?
6
+
7
+A low BBU relative charge indicates that the state of charge is low compared to its design capacity. The relative state of charge is a percentage indication of the full charge capacity compared to its designed capacity. If the relative charge is constantly low, it may suggest that the battery is worn out and needs replacement.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Gather information about your battery units for all controllers:
12
+
13
+ ```
14
+ sudo megacli -AdpBbuCmd -GetBbuStatus -aALL
15
+ ```
16
+
17
+ This command will provide you with detailed information about the BBU status for each controller.
18
+
19
+2. Perform a manual battery calibration (learning cycle) on the battery with a low relative charge:
20
+
21
+ ```
22
+ sudo megacli -AdpBbuCmd -BbuLearn -aX
23
+ ```
24
+
25
+ Replace `X` with the controller's number. Please consult the [MegaRAID SAS Software User Guide](https://docs.broadcom.com/docs/12353236), section 7.14, before performing this action.
26
+
27
+ A learning cycle discharges and recharges the battery, which can help recalibrate the battery and improve its relative state of charge. However, it may temporarily disable the write cache during this process.
28
+
29
+3. Monitor the BBU relative charge after the learning cycle. If the relative charge remains low, consider replacing the battery in question. Consult your hardware vendor's documentation for guidance on replacing the BBU.
30
+
31
+### Useful resources
32
+
33
+1. [MegaRAID SAS Software User Guide [pdf download]](https://docs.broadcom.com/docs/12353236)
34
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
35
+
36
+**Note**: Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is not liable for any loss or corruption of any data, database, or software.
\ No newline at end of file
health/guides/megacli_pd_media_errors.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+The `megacli_pd_media_errors` alert is triggered when there are media errors on the physical disks attached to the MegaCLI controller. A media error is an event where a storage disk was unable to perform the requested I/O operation due to problems accessing the stored data. This alert indicates that a bad sector was found on the drive during a patrol check or from a rebuild operation on a specific disk by the RAID adapter. Although this does not mean imminent disk failure, it is a warning, and you should monitor the affected disk.
4
+
5
+### Troubleshoot the alert
6
+
7
+**Data is priceless. Before you perform any action, make sure that you have taken any necessary backup steps. Netdata is not liable for any loss or corruption of any data, database, or software.**
8
+
9
+1. Gather more information about your virtual drives on all adapters:
10
+
11
+ ```
12
+ megacli –LDInfo -Lall -aALL
13
+ ```
14
+
15
+2. Check which virtual drive is reporting media errors and in which adapter.
16
+
17
+3. Check the Bad block table for the virtual drive in question:
18
+
19
+ ```
20
+ megacli –GetBbtEntries -LX -aY // X: virtual drive, Y: the adapter
21
+ ```
22
+
23
+4. Consult the MegaRAID SAS Software User Guide's section 7.17.11[^1] to recheck these block entries. **This operation removes any data stored on the physical drives. Back up the good data on the drives before making any changes to the configuration.**
24
+
25
+### Useful resources
26
+
27
+1. [MegaRAID SAS Software User Guide [PDF download]](https://docs.broadcom.com/docs/12353236)
28
+2. [MegaCLI command cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
29
+
30
+[^1]: https://docs.broadcom.com/docs/12353236
\ No newline at end of file
health/guides/megacli_pd_predictive_failures.md
new
+29
@@ -0,0 +1,29 @@
1
+### Understand the alert
2
+
3
+This alert indicates that one or more physical disks attached to the MegaCLI controller are experiencing predictive failures. A predictive failure is a warning that a hard disk may fail in the near future, even if it's still working normally. The failure prediction relies on the self-monitoring and analysis technology (S.M.A.R.T.) built into the disk drive.
4
+
5
+### Troubleshoot the alert
6
+
7
+**Make sure you have taken necessary backup steps before performing any action. Netdata is not liable for any loss or corruption of data, databases, or software.**
8
+
9
+1. Identify the problematic drives:
10
+
11
+ Use the following command to gather information about your virtual drives in all adapters:
12
+
13
+ ```
14
+ megacli –LDInfo -Lall -aALL
15
+ ```
16
+
17
+2. Determine the virtual drive and adapter reporting media errors.
18
+
19
+3. Consult the MegaRAID SAS Software User Guide [1]:
20
+
21
+ 1. Refer to Section 2.1.16 to check for issues with your drives.
22
+ 2. Refer to Section 7.18 to perform any appropriate actions on drives. Focus on Sections 7.18.2, 7.18.6, 7.18.7, 7.18.8, 7.18.11, and 7.18.14.
23
+
24
+4. Consider replacing the problematic disk(s) to prevent imminent failures and potential data loss.
25
+
26
+### Useful resources
27
+
28
+1. [MegaRAID SAS Software User Guide (PDF download)](https://docs.broadcom.com/docs/12353236)
29
+2. [MegaCLI commands cheatsheet](https://www.broadcom.com/support/knowledgebase/1211161496959/megacli-commands)
\ No newline at end of file
health/guides/memcached_cache_fill_rate.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert, `memcached_cache_fill_rate`, measures the average rate at which the Memcached cache fills up (positive value) or frees up (negative value) space over the last hour. The units are in `KB/hour`. If you receive this alert, it means that your Memcached cache is either filling up or freeing up space at a noticeable rate.
4
+
5
+### What is Memcached?
6
+
7
+Memcached is a high-performance, distributed memory object caching system used to speed up web applications by temporarily storing frequently-used data in RAM. It reduces the load on the database and improves performance by minimizing the need for repeated costly database queries.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the current cache usage:
12
+
13
+You can view the current cache usage using the following command, where `IP` and `PORT` are the Memcached server's IP address and port number:
14
+
15
+```
16
+echo "stats" | nc IP PORT
17
+```
18
+
19
+Look for the `bytes` and `limit_maxbytes` fields in the output to see the current cache usage and the maximum cache size allowed, respectively.
20
+
21
+2. Identify heavy cache users:
22
+
23
+Find out which applications or services are generating a significant number of requests to Memcached. You may be able to optimize them to reduce cache usage. You can check Memcached logs for more details about requests and operations.
24
+
25
+3. Optimize cache storage:
26
+
27
+If the cache is filling up too quickly, consider optimizing your cache storage policies. For example, you can adjust the expiration times of stored items, prioritize essential data, or use a more efficient caching strategy.
28
+
29
+4. Increase the cache size:
30
+
31
+If needed, you can increase the cache size to accommodate a higher fill rate. To do this, stop the Memcached service and restart it with the `-m` option, specifying the desired memory size in megabytes:
32
+
33
+```
34
+memcached -d -u memcached -m NEW_SIZE -l IP -p PORT
35
+```
36
+
37
+Replace `NEW_SIZE` with the desired cache size in MB.
38
+
39
+### Useful resources
40
+
41
+1. [Memcached Official Site](https://memcached.org/)
health/guides/memcached_cache_memory_usage.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+This alert indicates the percentage of used cached memory in your Memcached instance. High cache memory utilization can lead to evictions and performance degradation. The warning state is triggered when the cache memory utilization is between 70-80%, and the critical state is triggered when it's between 80-90%.
4
+
5
+### What does cache memory utilization mean?
6
+
7
+Cache memory utilization refers to the percentage of memory used by Memcached for caching data. A high cache memory utilization indicates that your Memcached instance is close to its maximum capacity, and it may start evicting data to accommodate new entries, which can negatively impact performance.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. **Monitor cache usage and evictions**: Use the following command to display the current cache usage and evictions metrics:
12
+
13
+ ```
14
+ echo "stats" | nc localhost 11211
15
+ ```
16
+ Look for the `bytes` and `evictions` metrics in the output. High evictions indicate that your cache size is insufficient for the current workload, and you may need to increase it.
17
+
18
+2. **Increase cache size**: To increase the cache size, edit the Memcached configuration file (usually `/etc/memcached.conf`) and update the value of the `-m` option. For example, to set the cache size to 2048 megabytes, update the configuration as follows:
19
+
20
+ ```
21
+ -m 2048
22
+ ```
23
+ Save the file and restart the Memcached service for the changes to take effect.
24
+
25
+ ```
26
+ sudo systemctl restart memcached
27
+ ```
28
+
29
+3. **Optimize your caching strategy**: Review your caching strategy to ensure that you are only caching necessary data and using appropriate expiration times. Making updates that reduce the amount of cached data can help prevent high cache memory usage.
30
+
31
+4. **Consider cache sharding or partitioning**: If increasing the cache size or optimizing your caching strategy doesn't resolve the issue, you may need to consider cache sharding or partitioning. This approach involves using multiple Memcached instances, dividing the data across them, which can help distribute the load and reduce cache memory usage.
32
+
33
+### Useful resources
34
+
35
+1. [Memcached Official Documentation](https://memcached.org/)
health/guides/memcached_out_of_cache_space_time.md
new
+19
@@ -0,0 +1,19 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the Memcached cache is running out of space and will likely become full soon, based on the data addition rate over the past hour. If the cache reaches 100% capacity, evictions may occur, resulting in a loss of cached data and decreased performance.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Monitor cache usage**: Use the `stats` command in Memcached to check the current cache usage and the number of evictions. This will help you understand the severity of the issue and whether evictions are already happening.
8
+
9
+2. **Evaluate cache settings**: Review your Memcached configuration file (`/etc/memcached.conf` or `/etc/sysconfig/memcached`) and check the cache size setting (`-m` parameter). Ensure that the cache size is set appropriately based on your system's available memory and workload requirements.
10
+
11
+3. **Increase cache size**: If the cache is consistently running out of space, consider increasing the cache size by adjusting the `-m` parameter in the Memcached configuration file. Be cautious not to allocate too much memory, as this can cause other system processes to suffer.
12
+
13
+4. **Optimize cache usage**: Analyze the cache usage patterns of your applications and optimize their caching strategies. This may involve adjusting the cache TTL (time-to-live) settings, using different cache eviction policies, or implementing a more efficient caching mechanism.
14
+
15
+5. **Monitor application performance**: Check the performance of your applications that use Memcached to identify any issues or bottlenecks. If performance is degrading due to cache evictions, consider optimizing the applications or increasing cache capacity.
16
+
17
+### Useful resources
18
+
19
+1. [Memcached Configuration Options](https://github.com/memcached/memcached/wiki/ConfiguringServer)
health/guides/memory/1hour_memory_hw_corrupted.md
deleted
-40
@@ -1,40 +0,0 @@
1
-# 1hour_memory_hw_corrupted
2
-
3
-## OS: Linux
4
-
5
-The Linux kernel keeps track of the system memory state. You can find the actual values it tracks in
6
-the man pages <sup>[1](https://man7.org/linux/man-pages/man5/proc.5.html) </sup> under
7
-the `/proc/meminfo` subsection. One of the values that the kernel reports is the `HardwareCorrupted`
8
-, which is the amount of memory, in kibibytes (1024 bytes), with physical memory corruption
9
-problems, identified by the hardware and set aside by the kernel so it does not get used.
10
-
11
-The Netdata Agent monitors this value. This alert indicates that the memory is corrupted due to a
12
-hardware failure. While primarily the error may be due to a failing RAM chip, it can also be caused
13
-by incorrect seating or improper contact between the socket and memory module.
14
-
15
-<details>
16
-<summary>References and Sources</summary>
17
-
18
-1. [man pages /proc](https://man7.org/linux/man-pages/man5/proc.5.html)
19
-
20
-1. [memtester homepage](https://pyropus.ca/software/memtester/)
21
-
22
-</details>
23
-
24
-### Troubleshooting section:
25
-
26
-<details>
27
-<summary>Verify a bad memory module</summary>
28
-
29
-Most of the times, uncorrectable errors will make your system and reboot/shutdown in a state of
30
-panic. If not, that means that your tolerance level is high enough to not make the system go into
31
-panic. You must identify the defective module immediately.
32
-
33
-1. `memtester` is a userspace utility for testing the memory subsystem for faults. It's portable and
34
- should compile and work on any 32 or 64-bit Unix-like system. For hardware developers, memtester
35
- can be told to test memory starting at a particular physical address (memtester v4.1.0+).
36
- <sup>[2](https://pyropus.ca/software/memtester/)
37
-
38
-You may also receive this error as a result of incorrect seating or improper contact between the
39
-socket and RAM module. Check on both before consider replacing the RAM module.
40
-</details>
health/guides/memory/ecc_memory_mc_correctable.md
deleted
-49
@@ -1,49 +0,0 @@
1
-# ecc_memory_mc_correctable
2
-
3
-## OS: Linux
4
-
5
-*Error correction code memory (ECC memory) is a type of computer data storage that uses an error
6
-correction code (ECC) to detect and correct n-bit data corruption which occurs in memory. ECC
7
-memory is used in most computers where data corruption cannot be tolerated under any circumstances,
8
-like industrial control applications, critical databases, and infrastructural memory
9
-caches.* <sup>[1](https://en.wikipedia.org/wiki/ECC_memory) </sup>
10
-
11
-"Correctable errors are generally single-bit errors that the system or the built-in ECC mechanism
12
-can correct. These errors do not cause system downtime of data
13
-corruption." <sup>[2](https://www.atpinc.com/blog/ecc-dimm-memory-ram-errors-types-chipkill) </sup>
14
-
15
-Netdata agent monitors the number of ECC correctable errors in the last 10 minutes.
16
-
17
-<details>
18
-<summary>References and sources:</summary>
19
-
20
-1. [ECC memory on wikipedia](https://en.wikipedia.org/wiki/ECC_memory)
21
-
22
-1. [RAM types and ECC technologies](https://www.atpinc.com/blog/ecc-dimm-memory-ram-errors-types-chipkill)
23
-
24
-1. [memtester homepage](https://pyropus.ca/software/memtester/)
25
-
26
-</details>
27
-
28
-### Troubleshooting section:
29
-
30
-<details>
31
-<summary>Verify a bad memory module</summary>
32
-
33
-Correctable errors do not necessarily indicate hardware failures, but should generally still be investigated.
34
-
35
-1. `memtester` is a userspace utility for testing the memory subsystem for faults. It's portable and
36
- should compile and work on any 32 or 64-bit Unix-like system. For hardware developers, memtester
37
- can be told to test memory starting at a particular physical address (memtester v4.1.0+).
38
- <sup>[3](https://pyropus.ca/software/memtester/)
39
-
40
-You can also get this kind of errors by incorrect seating or improper contact between the socket and
41
-RAM module. Check on both before consider replacing the RAM module.
42
-</details>
43
-
44
-<details>
45
-<summary>Check for BIOS updates</summary>
46
-
47
-You should check for critical BIOS updates on your hardware's vendor support page.
48
-
49
-</details>
health/guides/memory/ecc_memory_mc_uncorrectable.md
deleted
-93
@@ -1,93 +0,0 @@
1
-# ecc_memory_mc_uncorrectable
2
-
3
-## OS: Linux
4
-
5
-Error correction code memory (ECC memory) is a type of computer data storage that uses an error
6
-correction code (ECC) to detect and correct n-bit data corruption which occurs in memory. Error
7
-correction codes protect against undetected memory data corruption, and is used in computers where
8
-such corruption is unacceptable, for example in some scientific and financial computing
9
-applications, or in database and file
10
-servers. <sup>[1](https://en.wikipedia.org/wiki/ECC_memory) </sup>
11
-
12
-The Netdata Agent monitors the number of ECC uncorrectable errors in the last 10 minutes.
13
-
14
-
15
-<details>
16
-<summary>See more on uncorrectable errors.</summary>
17
-
18
-There are two main categories of Uncorrectable Errors (UE) as documented in the
19
-kernel.org <sup>[2](https://www.kernel.org/doc/Documentation/admin-guide/ras.rst) </sup>
20
-
21
-1. Fatal Error, when a UE error happens on a critical component of the system (for example, a piece
22
- of the Kernel got corrupted by a UE). The only reliable way to avoid data corruption is to hang
23
- or reboot the machine.
24
-
25
-1. Non-fatal Error, when a UE error happens on an unused component, like an unused memory bank. The
26
- system may still run, eventually replacing the affected hardware by a hot spare, if available.
27
-
28
-</details>
29
-
30
-
31
-<details>
32
-<summary>See more on machine checks</summary>
33
-
34
-> Machine checks report internal hardware error conditions detected by the CPU. Uncorrected errors
35
-typically cause a machine check (often with panic), corrected ones cause a machine check log entry.
36
->
37
-> The behavior your machine will have when UE occurs depends on the tolerance level settings. The
38
-tolerance level configures how hard the kernel tries to recover even at some risk of deadlock.
39
-Higher tolerant values trade potentially better uptime with the risk of a crash or even corruption (
40
-for tolerant >= 3). The Default is 1.
41
->
42
-> - 0: always panic on uncorrected errors, log corrected errors
43
->
44
-> - 1: panic or SIGBUS on uncorrected errors, log corrected errors
45
->
46
-> - 2: SIGBUS or log uncorrected errors, log corrected errors
47
->
48
-> - 3: never panic or SIGBUS, log all errors (for testing
49
-> only)
50
->
51
-> Also, when an error happens on a userspace process, it is also possible to kill such process and
52
-> let userspace restart it. <sup>[3](https://www.kernel.org/doc/html/v5.15-rc6/x86/x86_64/machinecheck.html) </sup>
53
-
54
-
55
-</details>
56
-
57
-<details>
58
-<summary>References and sources:</summary>
59
-
60
-1. [ECC memory on wikipedia](https://en.wikipedia.org/wiki/ECC_memory)
61
-
62
-1. [Reliability, Availability and Serviceability concepts](https://www.kernel.org/doc/Documentation/admin-guide/ras.rst)
63
-
64
-1. [Machine checks](https://www.kernel.org/doc/html/v5.2/x86/x86_64/machinecheck.html)
65
-
66
-1. [memtester homepage](https://pyropus.ca/software/memtester/)
67
-
68
-</details>
69
-
70
-### Troubleshooting section:
71
-
72
-<details>
73
-<summary>Verify a bad memory module</summary>
74
-
75
-Most of the times, uncorrectable errors will make your system and reboot/shutdown in a state of panic. If
76
-not, that means that your tolerance level is high enough to not make the system go into panic. You
77
-must identify the defective module immediately.
78
-
79
-1. `memtester` is a userspace utility for testing the memory subsystem for faults. It's portable and
80
- should compile and work on any 32 or 64-bit Unix-like system. For hardware developers, memtester
81
- can be told to test memory starting at a particular physical address (memtester v4.1.0+).
82
- <sup>[2](https://pyropus.ca/software/memtester/)
83
-
84
-You may also receive this error as a result of incorrect seating or improper contact between the socket and
85
-RAM module. Check on both before consider replacing the RAM module.
86
-</details>
87
-
88
-<details>
89
-<summary>Check for BIOS updates</summary>
90
-
91
-You should check for critical BIOS updates on your hardware's vendor support page.
92
-
93
-</details>
health/guides/ml_1min_node_ar.md
new
+26
@@ -0,0 +1,26 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the [node anomaly rate](https://learn.netdata.cloud/docs/ml-and-troubleshooting/machine-learning-ml-powered-anomaly-detection#node-anomaly-rate) exceeds the threshold defined in the [alert configuration](https://github.com/netdata/netdata/blob/master/health/health.d/ml.conf) over the most recent 1 minute window evaluated.
4
+
5
+For example, with the default of `warn: $this > 1`, this means that 1% or more of the metrics collected on the node have across the most recent 1 minute window been flagged as [anomalous](https://learn.netdata.cloud/docs/ml-and-troubleshooting/machine-learning-ml-powered-anomaly-detection) by Netdata.
6
+
7
+### Troubleshoot the alert
8
+
9
+This alert is a signal that some significant percentage of metrics within your infrastructure have been flagged as anomalous accoring to the ML based anomaly detection models the Netdata agent continually trains and re-trains for each metric. This tells us something somewhere might look strange in some way. THe next step is to try drill in and see what metrics are actually driving this.
10
+
11
+1. **Filter for the node or nodes relevant**: First we need to reduce as much noise as possible by filtering for just those nodes that have the elevated node anomaly rate. Look at the `anomaly_detection.anomaly_rate` chart and group by `node` to see which nodes have an elevated anomaly rate. Filter for just those nodes since this will reduce any noise as much as possible.
12
+
13
+2. **Highlight the area of interest**: Highlight the timeframne of interest where you see an elevated anomaly rate.
14
+
15
+3. **Check the anomalies tab**: Check the [Anomaly Advisor](https://learn.netdata.cloud/docs/ml-and-troubleshooting/anomaly-advisor) ("Anomalies" tab) to see an ordered list of what metrics were most anomalous in the highlighted window.
16
+
17
+4. **Press the AR% button on Overview**: You can also press the "[AR%](https://blog.netdata.cloud/anomaly-rates-in-the-menu/)" button on the Overview or single node dashboard to see what parts of the menu have the highest chart anomaly rates. Pressing the AR% button should add some "pills" to each menu item and if you hover over it you will see that chart within each menu section that was most anomalous during the highlighted timeframe.
18
+
19
+5. **Use Metric Correlations**: Use [metric correlations](https://learn.netdata.cloud/docs/ml-and-troubleshooting/metric-correlations) to see what metrics may have changed most significantly comparing before to the highlighted timeframe.
20
+
21
+### Useful resources
22
+
23
+1. [Machine learning (ML) powered anomaly detection](https://learn.netdata.cloud/docs/ml-and-troubleshooting/machine-learning-ml-powered-anomaly-detection)
24
+2. [Anomaly Advisor](https://learn.netdata.cloud/docs/ml-and-troubleshooting/anomaly-advisor)
25
+3. [Metric Correlations](https://learn.netdata.cloud/docs/ml-and-troubleshooting/metric-correlations)
26
+4. [Anomaly Rates in the Menu!](https://blog.netdata.cloud/anomaly-rates-in-the-menu/)
health/guides/mysql/mysql_10s_slow_queries.md
deleted
-36
@@ -1,36 +0,0 @@
1
-# mysql_10s_slow_queries
2
-
3
-# Database | MySQL
4
-
5
-This alert presents the number of slow queries in the last 10 seconds. If you receive this, it
6
-indicates a high number of slow queries.
7
-
8
-The metric is raised in a warning state when the value is larger than 10. If the number of slow
9
-queries in the last 10 seconds exceeds 20, then the alert is raised in critical state.
10
-
11
-Queries are defined as "slow", if they have taken more than `long_query_time` seconds, a predefined
12
-variable. Also, the value is measured in real time, not CPU time.
13
-
14
-<details><summary>References and Sources</summary>
15
-
16
-1. [SQL Query Optimisation](https://opensource.com/article/17/5/speed-your-mysql-queries-300-times)
17
-</details>
18
-
19
-### Troubleshooting Section
20
-
21
-<details><summary>Determine which queries are the problem and try to optimise
22
-them</summary>
23
-
24
-To identify the slow queries, you can enable the slow-query log of MySQL:
25
-
26
-1. Locate the `my.cnf` file
27
-2. Enable the slow-query log by setting the `slow_query_log variable` to `On`.
28
-3. Enter a path where the log files should be stored in the `slow_query_log_file` variable.
29
-
30
-After you know which queries are the ones taking longer than preferred, you can use the `EXPLAIN`
31
-keyword to overview how many rows are accessed, what operations are being done etc.
32
-
33
-After you've found the cause for the slow queries, you can start optimizing your queries. Consider to use an index and think about how you can change the way you `JOIN` tables. Both of these methods aid to reduce the amount of data that is being accessed without it really being needed.
34
-
35
-Read more about [speeding up your mySQL queries](https://opensource.com/article/17/5/speed-your-mysql-queries-300-times).
36
-</details>
health/guides/mysql/mysql_connections.md
deleted
-36
@@ -1,36 +0,0 @@
1
-# mysql_connections
2
-
3
-## Database | MySQL, MariaDB
4
-
5
-This alert presents the percentage of used client connections.
6
-Receiving this alert means that there is a high client connection utilization
7
-compared to the limit.
8
-
9
-This alert is raised to warning when the percentage exceeds 70%.
10
-If the metric exceeds 90%, then the alert is escalated to critical.
11
-
12
-<details><summary>References and Sources</summary>
13
-
14
-1. [MySQL max connections](https://ubiq.co/database-blog/how-to-increase-max-connections-in-mysql/)
15
-
16
-</details>
17
-
18
-### Troubleshooting Section
19
-
20
-<details><summary>Increase the Connection Limit</summary>
21
-
22
-To increase the connection limit, log into MySQL form the terminal and use the following code:
23
-`show variables like "max_connections";`
24
-to see the current limit.
25
-
26
-Using:
27
-`set global max_connections = "LIMIT";`
28
-Where "LIMIT" is the new limit you will choose, you can alter the limit without restarting the
29
-server.
30
-
31
-To increase the limit permanently, locate the `my.cnf` file (typically under `/etc` but depends on
32
-installation) and append `max_connections = 200` under the `mysqld` section.
33
-
34
-You can read more in our References and Sources section.
35
-
36
-</details>
health/guides/mysql/mysql_galera_cluster_size.md
deleted
-31
@@ -1,31 +0,0 @@
1
-# mysql_galera_cluster_size
2
-
3
-## Database | MySQL, MariaDB
4
-
5
-This alert presents the current Galera cluster size, compared to the maximum size in the last 2
6
-minutes.
7
-
8
-If you receive this alert, then it may indicate a network connectivity problem or
9
-that MySQL is down on one node.
10
-
11
-This alert is raised into warning if the current Galera cluster size is larger than the maximum
12
-size in the last 2 minutes.
13
-
14
-If the current Galera cluster size is less than the maximum size in the last sixty seconds, then the
15
-alert is escalated into critical.
16
-
17
-<details><summary>References and Sources</summary>
18
-
19
-1. [Galera Cluster Training Library](
20
- https://galeracluster.com/library/training/tutorials/galera-monitoring.html)
21
-
22
-</details>
23
-
24
-### Troubleshooting Section
25
-
26
-<details><summary>Check Node Status</summary>
27
-
28
-Refer to the [Galera Cluster training library](https://galeracluster.com/library/training/tutorials/galera-monitoring.html)
29
-for documentation on cluster health monitoring.
30
-
31
-</details>
health/guides/mysql/mysql_galera_cluster_state.md
deleted
-27
@@ -1,27 +0,0 @@
1
-# mysql_galera_cluster_state
2
-
3
-## Database | MySQL, MariaDB
4
-
5
-This alert presents the state of a node in the Galera cluster. If you receive this, it could be an
6
-indication that the node lost its connection to the Primary Component due to network partition.
7
-
8
-The alert gets raised into warning if the metric has one of the values:
9
-
10
-| Code | Description | Alert Status |
11
-|:----:|:--------------------------------------------------------------------------------|:------------:|
12
-| `0` | Undefined - indicates a starting node that is not part of the Primary Component | Critical |
13
-| `1` | Joining (requesting/receiving State Transfer) - node is joining the cluster | Critical |
14
-| `2` | Donor/Desynced - node is the donor to the node joining the cluster | Warning |
15
-| `3` | Joined - node has joined the cluster | Warning |
16
-| `>5` | - | Critical |
17
-
18
-For further information, please have a look at the *References and Sources* section.
19
-
20
-<details><summary>References and Sources</summary>
21
-
22
-1. [Galera Cluster Glossary](https://galeracluster.com/library/documentation/glossary.html)
23
-2. [Wsrep status index](
24
- https://www.percona.com/doc/percona-xtradb-cluster/5.5/wsrep-status-index.html)
25
-3. [Galera Cluster Notification command](
26
- https://galeracluster.com/library/documentation/notification-cmd.html)
27
-</details>
health/guides/mysql/mysql_galera_cluster_status.md
deleted
-47
@@ -1,47 +0,0 @@
1
-# mysql_galera_cluster_status
2
-
3
-## Database | MySQL, MariaDB
4
-
5
-This alert presents the status of the Galera node cluster component. If you receive this, it is an
6
-indication the cluster has been split into several components due to network failure.
7
-
8
-<details><summary>What is Quorum</summary>
9
-
10
-> A majority (> 50%) of nodes. In the event of a network partition, only the cluster partition
11
-> that retains a quorum (if any) will remain Primary by default.<sup>[1](https://galeracluster.com/library/documentation/glossary.html#:~:text=A%20majority%20(%3E%2050%25)%20of%20nodes.%20In%20the%20event%20of%20a%20network%20partition%2C%20only%20the%20cluster%20partition%20that%20retains%20a%20quorum%20(if%20any)%20will%20remain%20Primary%20by%20default.) </sup>
12
-</details>
13
-
14
-<details><summary>What is a Primary Component?</summary>
15
-
16
-> In addition to single-node failures, the cluster may be split into several components due to
17
-> network failure. In such a situation, only one of the components can continue to modify the
18
-> database state to avoid history divergence. This component is called the Primary Component (PC)
19
-> .<sup>[1](https://galeracluster.com/library/documentation/glossary.html#:~:text=from%20the%20database.-,Primary%20Component,For%20more%20information%20on%20the%20Primary%20Component%2C%20see%20Quorum%20Components.,-Quorum) </sup>
20
-</details>
21
-
22
-<details><summary>What is a Non-primary state Component?</summary>
23
-
24
-> The clusters without the quorum enter the non-primary state and begin attempt to connect with the
25
-> Primary Component.<sup>[2](https://galeracluster.com/library/documentation/weighted-quorum.html#:~:text=while%20those%20without%20quorum%20enter%20the%20non%2Dprimary%20state%20and%20begin%20attempt%20to%20connect%20with%20the%20Primary%20Component.) </sup>
26
-</details>
27
-
28
-The codes of the Galera node cluster component status can be:
29
-
30
-| Code | Description | Alert Status |
31
-|:----:|:-------------------------|:------------:|
32
-| `-1` | Unknown. | Critical |
33
-| `0` | Primary | Clear |
34
-| `1` | Non-primary/quorum lost | Critical |
35
-| `2` | Disconnected | Critical |
36
-
37
-For further information on Primary and non-Primary components please have a look at the
38
-*References and Sources* section.
39
-
40
-<details><summary>References and Sources</summary>
41
-
42
-1. [Galera CLuster Glossary](
43
- https://galeracluster.com/library/documentation/glossary.html)
44
-2. [Galera Cluster Documentation](
45
- https://galeracluster.com/library/documentation/weighted-quorum.html)
46
-
47
-</details>
\ No newline at end of file
health/guides/mysql/mysql_replication.md
deleted
-79
@@ -1,79 +0,0 @@
1
-# mysql_replication
2
-
3
-## Database | MySQL, MariaDB
4
-
5
-This alert monitors the replication status of the MySQL server.
6
-If you receive this, either both or one of the I/O and SQL threads are not running.
7
-
8
-This alert is raised into critical if replication has stopped.
9
-
10
-
11
-> In MySQL, replication involves the source database writing down every change made to the data
12
-> held within one or more databases in a special file known as the binary log. Once the replica
13
-> instance has been initialized, it creates two threaded processes. The first, called the IO
14
-> thread, connects to the source MySQL instance and reads the binary log events line by line,
15
-> and then copies them over to a local file on the replica’s server called the relay log. The
16
-> second thread, called the SQL thread, reads events from the relay log and then applies them
17
-> to the replica instance as fast as possible.
18
->
19
-> Recent versions of MySQL support two methods for replicating data. The difference between these
20
-> replication methods has to do with how replicas track which database events from the source
21
-> they’ve already processed.<sup>[1](
22
-> https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql) </sup>
23
-
24
-For further information, please have a look at the _References and Sources_ section.
25
-
26
-
27
-<details><summary>References and Sources</summary>
28
-
29
-1. [Replication in MySQL](
30
- https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql)
31
-2. [MySQL documentation](
32
- https://dev.mysql.com/doc/refman/5.7/en/replication-administration-status.html)
33
-3. [Section 8.14.6, “Replication Replica I/O
34
- Thread States”](https://dev.mysql.com/doc/refman/5.7/en/replica-io-thread-states.html)
35
-4. [Section 8.14.7, “Replication Replica SQL Thread
36
- States”](https://dev.mysql.com/doc/refman/5.7/en/replica-sql-thread-states.html)
37
-</details>
38
-
39
-### Troubleshooting Section
40
-
41
-<details><summary>Check which thread is not running</summary>
42
-
43
-From the MySQL command line you can run:
44
-
45
-- For MySQL and MariaDB before v10.2.0:
46
-
47
- ```
48
- SHOW SLAVE STATUS\G
49
- ```
50
-- For MariaDB v10.2.0+:
51
-
52
- ```
53
- SHOW ALL SLAVES STATUS\G
54
- ```
55
-
56
-This will show you three important rows among other info:
57
-
58
-> - Slave_IO_State:
59
- The current status of the replica. See [Section 8.14.6, “Replication Replica I/O
60
- Thread States”](https://dev.mysql.com/doc/refman/5.7/en/replica-io-thread-states.html), and
61
- [Section 8.14.7, “Replication Replica SQL Thread
62
- States”](https://dev.mysql.com/doc/refman/5.7/en/replica-sql-thread-states.html), for more
63
- information.
64
->
65
->
66
-> - Slave_IO_Running:
67
- Whether the I/O thread for reading the source's binary log is running.
68
- Normally, you want this to be **Yes** unless you have not yet started replication or have
69
- explicitly stopped it with STOP SLAVE.
70
->
71
->
72
-> - Slave_SQL_Running:
73
- Whether the SQL thread for executing events in the relay log is running. As
74
- with the I/O thread, this should normally be **Yes**.<sup> [2](
75
- https://dev.mysql.com/doc/refman/5.7/en/replication-administration-status.html) </sup>
76
-
77
-For more info you can refer to the [MySQL documentation](
78
-https://dev.mysql.com/doc/refman/5.7/en/replication-administration-status.html).
79
-</details>
health/guides/mysql/mysql_replication_lag.md
deleted
-60
@@ -1,60 +0,0 @@
1
-# mysql_replication_lag
2
-
3
-## Database | MySQL, MariaDB
4
-
5
-This alert presents the number of seconds that the replica is behind the master.
6
-Receiving this means that the replication SQL thread is far behind processing the source binary log.
7
-A constantly high value (or an increasing one) indicates that the replica is unable to handle events
8
-from the source in a timely fashion.
9
-
10
-This alert is raised into warning when the metric exceeds 10 seconds.
11
-If the number of seconds that the replica is behind the master exceeds 30 seconds then the alert is
12
-raised into critical.
13
-
14
-
15
-> In MySQL, replication involves the source database writing down every change made to the data
16
-> held within one or more databases in a special file known as the binary log. Once the replica
17
-> instance has been initialized, it creates two threaded processes. The first, called the IO
18
-> thread, connects to the source MySQL instance and reads the binary log events line by line,
19
-> and then copies them over to a local file on the replica’s server called the relay log. The
20
-> second thread, called the SQL thread, reads events from the relay log and then applies them
21
-> to the replica instance as fast as possible.
22
->
23
-> Recent versions of MySQL support two methods for replicating data. The difference between these
24
-> replication methods has to do with how replicas track which database events from the source
25
-> they’ve already processed.<sup>[1](
26
-> https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql) </sup>
27
-
28
-For further information, please have a look at the _References and Sources_ section.
29
-
30
-<details><summary>References and Sources</summary>
31
-
32
-1. [Replication in MySQL](
33
- https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql)
34
-2. [MySQL Replication Slave Lag](
35
- https://www.percona.com/blog/2014/05/02/how-to-identify-and-cure-mysql-replication-slave-lag/)
36
-3. [log_slow_slave_statements](
37
- https://dev.mysql.com/doc/refman/8.0/en/replication-options-replica.html#sysvar_log_slow_slave_statements)
38
-4. [long_query_time](
39
- https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_long_query_time)
40
-5. [log_slow_verbosity](
41
- https://www.percona.com/doc/percona-server/5.1/diagnostics/slow_extended.html?id=percona-server:features:slow_extended_51&redirect=2#log_slow_verbosity)
42
-
43
-</details>
44
-
45
-### Troubleshooting Section
46
-
47
-<details><summary>Query optimization and "log_slow_slave_statements"</summary>
48
-
49
-To minimize slave `SQL_THREAD` lag, focus on query optimization. The following logs will help you identify the problem:
50
-1. Enable [log_slow_slave_statements](
51
-https://dev.mysql.com/doc/refman/8.0/en/replication-options-replica.html#sysvar_log_slow_slave_statements)
52
-to see queries executed by slave that take more than [long_query_time](
53
-https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_long_query_time).
54
-2. To get more information about query performance, set the configuration option [log_slow_verbosity](
55
-https://www.percona.com/doc/percona-server/5.1/diagnostics/slow_extended.html?id=percona-server:features:slow_extended_51&redirect=2#log_slow_verbosity) to `full`.
56
-
57
-You can also read the Percona blog for a nice write-up about[MySQL replication slave lag](
58
-https://www.percona.com/blog/2014/05/02/how-to-identify-and-cure-mysql-replication-slave-lag/).
59
-
60
-</details>
health/guides/mysql_10s_slow_queries.md
new
+25
@@ -0,0 +1,25 @@
1
+### Understand the alert
2
+
3
+This alert presents the number of slow queries in the last 10 seconds. If you receive this, it indicates a high number of slow queries.
4
+
5
+The metric is raised in a warning state when the value is larger than 10. If the number of slow queries in the last 10 seconds exceeds 20, then the alert is raised in critical state.
6
+
7
+Queries are defined as "slow", if they have taken more than `long_query_time` seconds, a predefined variable. Also, the value is measured in real time, not CPU time.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Determine which queries are the problem and try to optimise them
12
+
13
+To identify the slow queries, you can enable the slow-query log of MySQL:
14
+
15
+1. Locate the `my.cnf` file
16
+2. Enable the slow-query log by setting the `slow_query_log variable` to `On`.
17
+3. Enter a path where the log files should be stored in the `slow_query_log_file` variable.
18
+
19
+After you know which queries are the ones taking longer than preferred, you can use the `EXPLAIN` keyword to overview how many rows are accessed, what operations are being done etc.
20
+
21
+After you've found the cause for the slow queries, you can start optimizing your queries. Consider to use an index and think about how you can change the way you `JOIN` tables. Both of these methods aid to reduce the amount of data that is being accessed without it really being needed.
22
+
23
+### Useful resources
24
+[SQL Query Optimisation](https://opensource.com/article/17/5/speed-your-mysql-queries-300-times)
25
+
health/guides/mysql_10s_table_locks_immediate.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of table immediate locks in MySQL increases within the last 10 seconds. Table locks are used to control concurrent access to tables, and immediate locks are granted when the requested lock is available.
4
+
5
+### What are table immediate locks?
6
+
7
+In MySQL, table immediate locks are a mechanism for managing concurrent access to tables. When a table lock is requested and is available, an immediate lock is granted, allowing the process to continue execution. This ensures that multiple processes can't modify the data simultaneously, which could cause data inconsistencies.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the queries causing the table locks:
12
+
13
+ You can use the following command to display the process list in MySQL, which will include information about the locks:
14
+
15
+ ```
16
+ SHOW FULL PROCESSLIST;
17
+ ```
18
+
19
+2. Analyze the queries:
20
+
21
+ Check the queries causing the table locks to determine if they are necessary, can be optimized, or should be terminated. To terminate a specific query, use the `KILL QUERY` command followed by the connection ID:
22
+
23
+ ```
24
+ KILL QUERY connection_id;
25
+ ```
26
+
27
+3. Check table lock status:
28
+
29
+ To get more information about the lock status, you can use the following command to display the lock status of all tables:
30
+
31
+ ```
32
+ SHOW OPEN TABLES WHERE in_use > 0;
33
+ ```
34
+
35
+4. Optimize database queries and configurations:
36
+
37
+ Improve query performance by optimizing the queries and indexing the tables. Additionally, check your MySQL configuration and adjust it if necessary to minimize the number of locks required.
38
+
39
+5. Monitor the lock situation:
40
+
41
+ Keep monitoring the lock situation with the `SHOW FULL PROCESSLIST` command to see if the problem persists. If the issue is not resolved, consider increasing the MySQL lock timeout or seek assistance from a database administrator or the MySQL community.
42
+
43
+### Useful resources
44
+
45
+1. [MySQL Table Locking](https://dev.mysql.com/doc/refman/8.0/en/table-locking.html)
46
+2. [MySQL Lock Information](https://dev.mysql.com/doc/refman/8.0/en/innodb-locking.html)
health/guides/mysql_10s_table_locks_waited.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when there's a high number of `table locks waited` in the last 10 seconds for a MySQL database. Table locks prevent multiple processes from writing to a table at the same time, ensuring the integrity of the data. However, too many table locks waiting can indicate a performance issue, as it could mean that some queries are causing deadlocks or taking too long to complete.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify queries causing locks
8
+
9
+ Use the following MySQL command to view the currently running queries and identify the ones causing the table locks:
10
+
11
+ ```
12
+ SHOW FULL PROCESSLIST;
13
+ ```
14
+
15
+2. Examine locked tables
16
+
17
+ Use the following command to find more information about the locked tables:
18
+
19
+ ```
20
+ SHOW OPEN TABLES WHERE In_use > 0;
21
+ ```
22
+
23
+3. Optimize query performance
24
+
25
+ Analyze the queries causing the table locks and optimize them to improve performance. This may include creating or modifying indexes, optimizing the SQL query structure, or adjusting the MySQL server configuration settings.
26
+
27
+4. Consider using InnoDB
28
+
29
+ If your MySQL database is using MyISAM storage engine, consider switching to InnoDB storage engine to take advantage of row-level locking and reduce the number of table locks.
30
+
31
+5. Monitor MySQL performance
32
+
33
+ Keep an eye on MySQL performance metrics such as table locks, query response times, and overall database performance to prevent future issues. Tools like the Netdata Agent can help in monitoring MySQL performance.
34
+
35
+### Useful resources
36
+
37
+1. [InnoDB Locking and Transaction Model](https://dev.mysql.com/doc/refman/8.0/en/innodb-locking-transaction-model.html)
health/guides/mysql_10s_waited_locks_ratio.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alert indicates a high ratio of waited table locks in your MySQL database over the last 10 seconds. If you receive this alert, it means that there might be performance issues due to contention for table locks.
4
+
5
+### What are table locks?
6
+
7
+Table locks are a method used by MySQL to ensure data consistency and prevent multiple clients from modifying the same data at the same time. When a client attempts to modify data, it must first acquire a lock on the table. If the lock is not available, the client must wait until the lock is released by another client.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify problematic queries:
12
+
13
+ Use the following command to display the queries that are causing table locks in your MySQL database:
14
+
15
+ ```
16
+ SHOW FULL PROCESSLIST;
17
+ ```
18
+
19
+ Look for queries with a state of `'Locked'` or `'Waiting for table lock'` and note down their details.
20
+
21
+2. Optimize your queries:
22
+
23
+ Analyze the problematic queries identified in the previous step and try to optimize them. You can use `EXPLAIN` or other similar tools to get insights into the performance of the queries.
24
+
25
+3. Consider splitting your table(s):
26
+
27
+ If the problem persists after optimizing the queries, consider splitting the large tables into smaller ones. This can help to reduce contention for table locks and improve performance.
28
+
29
+4. Use replication:
30
+
31
+ Another solution to this issue is the implementation of MySQL replication, which can reduce contention for table locks by allowing read queries to be executed on replica servers rather than the primary server.
32
+
33
+### Useful resources
34
+
35
+1. [Documentation: Table Locking Issues](https://dev.mysql.com/doc/refman/5.7/en/table-locking.html)
36
+2. [MySQL Replication](https://dev.mysql.com/doc/refman/8.0/en/replication.html)
health/guides/mysql_connections.md
new
+74
@@ -0,0 +1,74 @@
1
+### Understand the alert
2
+
3
+The `mysql_connections` alert indicates the percentage of used client connections compared to the maximum configured connections. When you receive this alert, it means your MySQL or MariaDB server is reaching its connection limit, which could lead to performance issues or failed connections for clients.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the current connection usage**
8
+
9
+ Use the following command to see the current used and total connections:
10
+
11
+ ```
12
+ mysql -u root -p -e "SHOW STATUS LIKE 'max_used_connections'; SHOW VARIABLES LIKE 'max_connections';"
13
+ ```
14
+
15
+ This will display the maximum number of connections used since the server was started and the maximum allowed number of connections (`max_connections`).
16
+
17
+2. **Monitor connections over time**
18
+
19
+ You can monitor the connection usage over time using the following command:
20
+
21
+ ```
22
+ watch -n 1 "mysql -u root -p -e 'SHOW STATUS LIKE \"Threads_connected\";'"
23
+ ```
24
+
25
+ This will update the number of currently connected threads every second.
26
+
27
+3. **Identify connection-consuming processes**
28
+
29
+ If connection usage is high, check which processes or clients are using connections:
30
+
31
+ ```
32
+ mysql -u root -p -e "SHOW PROCESSLIST;"
33
+ ```
34
+
35
+ This gives you an overview of the currently connected clients, their states, and queries being executed.
36
+
37
+4. **Optimize client connections**
38
+
39
+ Analyze the processes using connections and ensure they close their connections properly when done, utilize connection pooling, and reduce the number of connections where possible.
40
+
41
+5. **Increase the connection limit (if necessary)**
42
+
43
+ If you need to increase the `max_connections` value, follow these steps:
44
+
45
+ - Log into MySQL from the terminal as shown in the troubleshooting section:
46
+
47
+ ```
48
+ mysql -u root -p
49
+ ```
50
+
51
+ - Check the current limit:
52
+
53
+ ```
54
+ show variables like "max_connections";
55
+ ```
56
+
57
+ - Set a new limit temporarily:
58
+
59
+ ```
60
+ set global max_connections = "LIMIT";
61
+ ```
62
+
63
+ Replace "LIMIT" with the desired new limit.
64
+
65
+ - To set the limit permanently, locate the `my.cnf` file (typically under `/etc`, but it may vary depending on your installation) and append `max_connections = LIMIT` under the `[mysqld]` section.
66
+
67
+ Replace "LIMIT" with the desired new limit, then restart the MySQL/MariaDB service.
68
+
69
+### Useful resources
70
+
71
+1. [How to Increase Max Connections in MySQL](https://ubiq.co/database-blog/how-to-increase-max-connections-in-mysql/)
72
+2. [MySQL 5.7 Reference Manual: SHOW STATUS Syntax](https://dev.mysql.com/doc/refman/5.7/en/show-status.html)
73
+3. [MySQL 5.7 Reference Manual: SHOW PROCESSLIST Syntax](https://dev.mysql.com/doc/refman/5.7/en/show-processlist.html)
74
+4. [MySQL 5.7 Reference Manual: mysqld – The MySQL Server](https://dev.mysql.com/doc/refman/5.7/en/mysqld.html)
health/guides/mysql_galera_cluster_size.md
new
+50
@@ -0,0 +1,50 @@
1
+### Understand the alert
2
+
3
+This alert monitors the Galera cluster size and checks if there is a discrepancy between the current cluster size and the maximum size in the last 2 minutes. A warning is raised if the current size is larger, and a critical alert is raised if the current size is smaller than the maximum size in the last minute.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the network connectivity:
8
+
9
+ Galera Cluster relies on persistent network connections. Review your system logs for any connectivity issues or network errors. If you find such issues, work with your network administrator to resolve them.
10
+
11
+2. Check the status of MySQL nodes:
12
+
13
+ You can use the following query to examine the status of all nodes in the Galera cluster:
14
+
15
+ ```
16
+ SHOW STATUS LIKE 'wsrep_cluster_%';
17
+ ```
18
+
19
+ Look for the `wsrep_cluster_size` and `wsrep_cluster_status` values, and analyze if there are any inconsistencies or issues.
20
+
21
+3. Review Galera logs:
22
+
23
+ Inspect the logs of the Galera cluster for any errors, warnings or issues. The log files are usually located in `/var/log/mysql` or `/var/lib/mysql` directories.
24
+
25
+4. Check node synchronization:
26
+
27
+ - Ensure that all nodes are synced by checking the `wsrep_local_state_comment` status variable. A value of 'Synced' indicates that the node is in sync with the cluster.
28
+
29
+ ```
30
+ SHOW STATUS LIKE 'wsrep_local_state_comment';
31
+ ```
32
+
33
+ - If any node is not synced, check its logs to find the cause of the issue and resolve it.
34
+
35
+5. Restart nodes if necessary:
36
+
37
+ If you find that a node is not working properly, you can try to restart the MySQL service on the affected node:
38
+
39
+ ```
40
+ sudo systemctl restart mysql
41
+ ```
42
+
43
+ Keep in mind that restarting a node can cause temporary downtime for applications connecting to that specific node.
44
+
45
+6. If the issue persists, consider contacting the Galera Cluster support team for assistance or consult the [Galera Cluster documentation](https://galeracluster.com/library/documentation/) for further guidance.
46
+
47
+### Useful resources
48
+
49
+1. [Galera Cluster Monitoring](https://galeracluster.com/library/training/tutorials/galera-monitoring.html)
50
+2. [Galera Cluster Documentation](https://galeracluster.com/library/documentation/)
health/guides/mysql_galera_cluster_size_max_2m.md
new
+40
@@ -0,0 +1,40 @@
1
+### Understand the alert
2
+
3
+This alert calculates the maximum size of the MySQL Galera cluster over a 2-minute period, starting from one minute ago. If you receive this alert, it means that there has been a significant change in the cluster size, which might affect the database's performance, stability, and data consistency.
4
+
5
+### What is MySQL Galera Cluster?
6
+
7
+MySQL Galera Cluster is a synchronous multi-master cluster for MySQL, built on the Galera replication plugin. It provides high-availability and improved performance for MySQL databases by synchronizing data across multiple nodes.
8
+
9
+### What does the cluster size mean?
10
+
11
+The cluster size refers to the number of nodes participating in a MySQL Galera Cluster. An optimal cluster size ensures that the database can handle more significant workloads, handle node failures, and perform automatic failovers.
12
+
13
+### Troubleshoot the alert
14
+
15
+- Determine the current cluster size
16
+
17
+ 1. Connect to any node in the cluster and run the following SQL query:
18
+
19
+ ```
20
+ SHOW STATUS LIKE 'wsrep_cluster_size';
21
+ ```
22
+
23
+ 2. The query will display the current number of nodes in the cluster.
24
+
25
+- Identify the cause of the cluster size change
26
+
27
+ 1. Check the MySQL and Galera logs on all nodes to identify any issues, such as network connectivity issues, node crashes, or hardware problems.
28
+
29
+ 2. Review the logs for events such as joining or leaving of the cluster nodes. Look for patterns that could lead to instability (e.g., frequent node join & leave events).
30
+
31
+- Resolve the issue
32
+
33
+ 1. Fix any identified problems causing the cluster size change. This may involve monitoring and resolving any network issues, restarting failed nodes, or replacing faulty hardware.
34
+
35
+ 2. If necessary, plan and execute a controlled reconfiguration of the Galera cluster to maintain the optimal cluster size.
36
+
37
+### Useful resources
38
+
39
+1. [Galera Cluster Documentation](https://galeracluster.com/library/documentation/)
40
+2. [Monitoring Galera Cluster for MySQL or MariaDB](https://severalnines.com/database-blog/monitoring-galera-cluster-mysql-or-mariadb)
\ No newline at end of file
health/guides/mysql_galera_cluster_state_crit.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+The `mysql_galera_cluster_state_crit` alert is triggered when the Galera node state is either `Undefined`, `Joining`, or `Error`. This indicates that there is an issue with a Galera node in your MySQL Galera Cluster.
4
+
5
+### What is a MySQL Galera Cluster?
6
+
7
+MySQL Galera Cluster is a synchronous, multi-master database cluster that provides high availability, no data loss, and scalability for your MySQL databases. It uses Galera replication library and MySQL server to achieve these goals.
8
+
9
+### Troubleshoot the alert
10
+
11
+To troubleshoot the MySQL Galera Cluster State Critical alert, follow these steps:
12
+
13
+1. Inspect the MariaDB error log
14
+
15
+ Check the MariaDB error log for any relevant error messages that can help identify the issue.
16
+
17
+ ```
18
+ sudo tail -f /var/log/mysql/error.log
19
+ ```
20
+
21
+2. Check the Galera node's status
22
+
23
+ Connect to the problematic MySQL node and check the Galera node status by running the following query:
24
+
25
+ ```
26
+ SHOW STATUS LIKE 'wsrep_%';
27
+ ```
28
+
29
+ Take note of the value of `wsrep_local_state` and `wsrep_local_state_comment`.
30
+
31
+3. Diagnose the issue
32
+
33
+ - If `wsrep_local_state` is 0 (`Undefined`), it means the node is not part of any cluster.
34
+ - If `wsrep_local_state` is 1 (`Joining`), it means the node is trying to connect or reconnect to the cluster.
35
+ - If `wsrep_local_state` is 5 (`Error`), it means the node has encountered a consistency error.
36
+
37
+4. Resolve the issue
38
+
39
+ - For an `Undefined` state, check and fix the wsrep configuration settings and restart the node.
40
+ - For a `Joining` state, ensure that the node can communicate with the other nodes in the cluster and make sure that the cluster's state is healthy. Then, retry joining the node to the cluster.
41
+ - For an `Error` state, the node may need to be resynchronized with the cluster. Restart the mysqld process on the affected node, or you may need to perform a full state transfer to recover.
42
+
43
+5. Monitor the cluster
44
+
45
+ After resolving the issue, monitor the cluster to ensure that all nodes are healthy and remain in-sync.
46
+
health/guides/mysql_galera_cluster_state_warn.md
new
+43
@@ -0,0 +1,43 @@
1
+### Understand the alert
2
+
3
+This alert checks the state of a Galera node in a MySQL Galera cluster. If you receive this alert, it means that the node is either in the **Donor/Desynced** state or the **Joined** state, which can indicate potential issues within the cluster.
4
+
5
+### What does Donor/Desynced and Joined state mean?
6
+
7
+1. **Donor/Desynced**: When a node is in the Donor/Desynced state, it is providing a State Snapshot Transfer (SST) to another node in the cluster. During this time, the node is not synchronized with the rest of the cluster and cannot process any write or commit requests.
8
+
9
+2. **Joined**: In the Joined state, a node has completed the initial SST and is now catching up with any missing transactions through an Incremental State Transfer (IST).
10
+
11
+### Troubleshoot the alert
12
+
13
+1. Check the Galera cluster status with the following command:
14
+
15
+ ```
16
+ SHOW STATUS LIKE 'wsrep_%';
17
+ ```
18
+
19
+2. Verify if any node is in the Donor/Desynced or Joined state:
20
+
21
+ ```
22
+ SELECT VARIABLE_NAME, VARIABLE_VALUE FROM information_schema.GLOBAL_STATUS WHERE VARIABLE_NAME IN ('wsrep_local_state_comment', 'wsrep_cluster_status', 'wsrep_ready');
23
+ ```
24
+
25
+3. Identify the cause of the node state change. Some possible reasons are:
26
+
27
+ - A new node has joined the cluster and requires an SST.
28
+ - A node has been restarted, and it is rejoining the cluster.
29
+ - A node experienced a temporary network issue and is now resynchronizing with the cluster.
30
+
31
+4. Monitor the progress of the resynchronization process using the `SHOW STATUS` command, as provided above, and wait for the node to reach the *Synced* state.
32
+
33
+5. If the node remains in the Donor/Desynced or Joined state for an extended period, investigate further to determine the cause of the issue:
34
+
35
+ - Inspect the MySQL error log for any relevant messages.
36
+ - Check for network issues or connectivity problems between the nodes.
37
+ - Verify the cluster configuration and ensure all nodes have a consistent configuration.
38
+
39
+6. Contact your DBA for assistance if the issue persists, as they may need to perform additional investigation and troubleshooting.
40
+
41
+### Useful resources
42
+
43
+1. [Galera Cluster's Documentation](https://galeracluster.com/library/documentation/)
health/guides/mysql_galera_cluster_status.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+This alert indicates the current status of the Galera node cluster component in your MySQL or MariaDB database. Receiving this alert means that there is a potential issue with the cluster, such as a network partition that has caused the cluster to split into multiple components.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the status of the Galera cluster**
8
+
9
+ First, you need to determine the current status of the cluster to understand the severity of the issue. Check the value of the alert. Refer to the table in the given alert description to see which state your cluster is in.
10
+
11
+2. **Verify cluster connectivity**
12
+
13
+ If your cluster is in a non-primary state or disconnected, you should verify if all the nodes in your cluster can communicate with each other. You can use tools like `ping`, `traceroute`, or `mtr` to test connectivity between the cluster nodes. If there is a network issue, get in touch with your network administrator to resolve it.
14
+
15
+3. **Examine node logs**
16
+
17
+ Check the logs on each node for any indication of issues or error messages that can help identify the root cause of the problem. The logs are usually located in the `/var/log/mysqld.log` file or in the `/var/log/mysql/error.log` file. Look for lines that contain "ERROR" or "WARNING" as a starting point.
18
+
19
+4. **Inspect Galera cluster settings**
20
+
21
+ Analyze your Galera cluster configuration file (`/etc/my.cnf` or `/etc/mysql/my.cnf`) to make sure you have the correct settings, including the initial `wsrep_cluster_address` value, which defines the initial list of nodes in the cluster. If you find any misconfiguration, correct it and restart your database service.
22
+
23
+5. **Force a new primary component**
24
+
25
+ If you have a split-brain scenario, where multiple parts of the cluster are claiming to be the primary component, you need to force a new primary component. To do this, you can use the `SET GLOBAL wsrep_provider_options='pc.bootstrap=YES';` statement on one of the nodes that has the most up-to-date data. This action will force that node to act as the new primary component.
26
+
27
+### Prevention
28
+
29
+To minimize the risks of cluster issues, ensure the following:
30
+
31
+1. Use reliable and redundant network connections between nodes.
32
+2. Configure Galera cluster settings correctly.
33
+3. Regularly monitor the cluster status and review logs.
34
+4. Use the latest stable version of the Galera cluster software.
35
+
36
+### Useful resources
37
+
38
+1. [MariaDB Galera Cluster Documentation](
39
+ https://mariadb.com/kb/en/getting-started-with-mariadb-galera-cluster/)
health/guides/mysql_replication.md
new
+64
@@ -0,0 +1,64 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the replication status of a MySQL server is indicating a problem or failure. Replication is important for redundancy, data backup, or load balancing. Issues with replication threads can lead to data inconsistencies or potential loss of data.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify the failing thread:
8
+
9
+ As mentioned above, use the appropriate command for your MySQL or MariaDB version to check the status of replication threads and determine which of them (I/O or SQL) is not running.
10
+
11
+ For MySQL and MariaDB before v10.2.0, use:
12
+
13
+ ```
14
+ SHOW SLAVE STATUS\G
15
+ ```
16
+
17
+ For MariaDB v10.2.0+, use:
18
+
19
+ ```
20
+ SHOW ALL SLAVES STATUS\G
21
+ ```
22
+
23
+2. Inspect the MySQL error log:
24
+
25
+ The MySQL error log can provide valuable information about the possible cause of the replication issues. Check the log for any replication-related error messages:
26
+
27
+ ```
28
+ tail -f /path/to/mysql/error.log
29
+ ```
30
+
31
+ Replace `/path/to/mysql/error.log` with the correct path to the MySQL error log file.
32
+
33
+3. Check the source MySQL server:
34
+
35
+ Replication issues can also originate from the source MySQL server. Make sure that the source server is properly configured and running, and that the binary logs are being written and flushed correctly.
36
+
37
+ Refer to the [MySQL documentation](https://dev.mysql.com/doc/refman/5.7/en/replication-howto.html) for more information on configuring replication.
38
+
39
+4. Restart the replication threads:
40
+
41
+ After identifying and resolving any issues found in the previous steps, you can try restarting the replication threads:
42
+
43
+ ```
44
+ STOP SLAVE;
45
+ START SLAVE;
46
+ ```
47
+
48
+ For MariaDB v10.2.0+ with multi-source replication, you may need to specify the connection name:
49
+
50
+ ```
51
+ STOP ALL SLAVES;
52
+ START ALL SLAVES;
53
+ ```
54
+
55
+5. Verify the replication status:
56
+
57
+ After restarting the replication threads, use the appropriate command from step 1 to verify that the threads are running, and that the replication is working as expected.
58
+
59
+### Useful resources
60
+
61
+1. [How To Set Up Replication in MySQL](https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql)
62
+2. [MySQL Replication Administration and Status](https://dev.mysql.com/doc/refman/5.7/en/replication-administration-status.html)
63
+3. [Replication Replica I/O Thread States](https://dev.mysql.com/doc/refman/5.7/en/replica-io-thread-states.html)
64
+4. [Replication Replica SQL Thread States](https://dev.mysql.com/doc/refman/5.7/en/replica-sql-thread-states.html)
\ No newline at end of file
health/guides/mysql_replication_lag.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert presents the number of seconds that the replica is behind the master. Receiving this means that the replication SQL thread is far behind processing the source binary log. A constantly high value (or an increasing one) indicates that the replica is unable to handle events from the source in a timely fashion.
4
+
5
+This alert is raised into warning when the metric exceeds 10 seconds. If the number of seconds that the replica is behind the master exceeds 30 seconds then the alert is raised into critical.
6
+
7
+
8
+### Troubleshoot the alert
9
+
10
+- Query optimization and "log_slow_slave_statements"
11
+
12
+To minimize slave `SQL_THREAD` lag, focus on query optimization. The following logs will help you identify the problem:
13
+1. Enable [log_slow_slave_statements](https://dev.mysql.com/doc/refman/8.0/en/replication-options-replica.html#sysvar_log_slow_slave_statements) to see queries executed by slave that take more than [long_query_time](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_long_query_time).
14
+2. To get more information about query performance, set the configuration option [log_slow_verbosity](https://www.percona.com/doc/percona-server/5.1/diagnostics/slow_extended.html?id=percona-server:features:slow_extended_51&redirect=2#log_slow_verbosity) to `full`.
15
+
16
+You can also read the Percona blog for a nice write-up about[MySQL replication slave lag](https://www.percona.com/blog/2014/05/02/how-to-identify-and-cure-mysql-replication-slave-lag/).
17
+
18
+### Useful resources
19
+
20
+1. [Replication in MySQL](
21
+ https://www.digitalocean.com/community/tutorials/how-to-set-up-replication-in-mysql)
22
+2. [MySQL Replication Slave Lag](
23
+ https://www.percona.com/blog/2014/05/02/how-to-identify-and-cure-mysql-replication-slave-lag/)
24
+3. [log_slow_slave_statements](
25
+ https://dev.mysql.com/doc/refman/8.0/en/replication-options-replica.html#sysvar_log_slow_slave_statements)
26
+4. [long_query_time](
27
+ https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_long_query_time)
28
+5. [log_slow_verbosity](
29
+ https://www.percona.com/doc/percona-server/5.1/diagnostics/slow_extended.html?id=percona-server:features:slow_extended_51&redirect=2#log_slow_verbosity)
30
+
health/guides/net/10min_fifo_errors.md
deleted
-71
@@ -1,71 +0,0 @@
1
-# 10min_fifo_errors
2
-
3
-## OS: Linux
4
-
5
-Between the IP stack and the Network Interface Controller (NIC) lies the driver queue. This queue is typically
6
-implemented as a FIFO ring buffer into the memory space allocated by the driver. The NIC receive frames and place them into
7
-memory as skb_buff data structures (SocKet Buffer). We can have queues (ingress queues) and transmitted (egress queues)
8
-but these queues do not contain any actual packet data. Each queue has a pointer to the devices associated with it, and
9
-to the skb_buff data structures that store the ingress/egress packets. The number of frames this queue can handle is
10
-limited. Queues fill up when an interface receives packets faster than kernel can process them.
11
-
12
-The Netdata agent monitors the number of FIFO errors (number of times an overflow occurs in the ring buffer) for a specific
13
-network interface in the last 10 minutes. This alarm is triggered when the NIC is not able to handle the peak load of
14
-incoming/outgoing packets with the current ring buffer size.
15
-
16
-Not all NICs support FIFO queue operations.
17
-
18
-<details>
19
-<summary>See more on SKB</summary>
20
-The SocKet Buffer (SKB), is the most fundamental data structure in the Linux networking code. Every packet sent or
21
-received is handled using this data structure. This is a large struct containing all the control information required
22
-for the packet (datagram, cell, etc).
23
-
24
-The struct sk_buff has the following fields to point to the specific network layer headers:
25
-
26
-- transport_header (previously called h) – This field points to layer 4, the transport layer (and can include tcp header or udp header or
27
- icmp header, and more)
28
-
29
-- network_header (previously called nh) – This field points to layer 3, the network layer (and can include ip header or ipv6 header or arp
30
- header).
31
-
32
-- mac_header (previously called mac) – This field points to layer 2, the link layer.
33
-
34
-- skb_network_header(skb), skb_transport_header(skb) and skb_mac_header(skb) - These return pointer to the header.
35
-
36
-</details>
37
-
38
-### Troubleshooting section:
39
-
40
-<details>
41
-<summary>Update the ring buffer size</summary>
42
-
43
-1. To view the maximum RX ring buffer size:
44
-
45
- ```
46
- root@netdata ~ # ethtool -g enp1s0
47
- Ring parameters for enp1s0:
48
- Pre-set maximums:
49
- RX: 4080
50
- RX Mini: 0
51
- RX Jumbo: 16320
52
- TX: 255
53
- Current hardware settings:
54
- RX: 255
55
- RX Mini: 0
56
- RX Jumbo: 0
57
- TX: 255
58
- ```
59
-
60
-2. If the values in the Pre-set maximums section are higher than in the Current hardware settings section, increase RX (
61
- or TX) ring buffer:
62
-
63
- ```
64
- root@netdata ~ # enp1s0 rx 4080
65
- ```
66
-
67
-3. Verify the change to make sure that you no longer receive the alarm when running the same workload. To
68
- make this permanently, you must consult your distribution guides.
69
-
70
-</details>
71
-
health/guides/net/10s_received_packets_storm.md
deleted
-80
@@ -1,80 +0,0 @@
1
-# 10s_received_packets_storm
2
-
3
-## OS: Linux, FreeBSD
4
-
5
-A packet storm is defined as an unusually high amount of traffic on a specific interface. In a sliding window of one minute
6
-Netdata agent monitors for significant increases of packets (ratio of an average number of received packets) in the last
7
-10 seconds. If this system is expected to have spikes, you can cautiously ignore this alarm, but should continue to monitor as this
8
-alarm may also indicate a broadcast/multicast storm or DoS attack.
9
-
10
-<details>
11
-<summary>See more on broadcast storms.</summary>
12
-
13
-A broadcast storm is the accumulation of broadcast and multicast traffic on a computer network. Extreme amounts of
14
-broadcast traffic constitute a "broadcast storm". It can consume sufficient network resources so as to render the
15
-network unable to transport normal traffic. Most commonly the cause is a switching loop in the Ethernet wiring topology.
16
-As broadcasts and multicasts are forwarded by switches out of every port, the switch or switches will repeatedly
17
-rebroadcast broadcast messages and flood the network. Since the Layer 2 header does not support a time to live (TTL)
18
-value, if a frame is sent into a looped topology, it can loop forever.
19
-
20
-</details>
21
-
22
-
23
-<details>
24
-<summary>See more on DoS attacks.</summary>
25
-
26
-A Denial-of-Service (DoS) attack is an attack meant to shut down a machine or network, making it inaccessible to its
27
-intended users. DoS attacks accomplish this by flooding the target with traffic, or sending it information that triggers
28
-a crash. We can categorize the attacks into two types.
29
-
30
-- Infrastructure Layer Attacks:
31
-
32
- Attacks at Layer 3 and 4 of the OSI model are typically categorized as Infrastructure layer attacks. The most common
33
- type of DDoS attack include vectors, like synchronized (SYN) floods, and other reflection attacks, like User Datagram
34
- Packet (UDP) floods. These attacks are usually large in volume and aim to overload the capacity of the network or the
35
- application servers. Fortunately, these are also the type of attacks that have clear signatures and are easier to
36
- detect.
37
-
38
-- Application Layer Attacks:
39
-
40
- Attacks at Layer 6 and 7 of the OSI model, are often categorized as Application layer attacks. While these attacks are
41
- less common, they also tend to be more complex. These attacks are typically small in volume compared to the
42
- Infrastructure layer attacks, but tend to focus on particular expensive parts of the application, thereby making it
43
- unavailable for real users. Common examples of this type of attack include a flood of HTTP requests to a login page, or an expensive search API, or
44
- even Wordpress XML-RPC floods.
45
-
46
-</details>
47
-
48
-
49
-### Troubleshooting section:
50
-
51
-</details>
52
-
53
-<details>
54
-<summary>Counter measures on DoS and DDoS attacks</summary>
55
-
56
-- Use a service like Cloudflare. Cloudflare DDoS protection secures websites, applications, and entire networks while
57
- ensuring the performance of legitimate traffic is not compromised.
58
-
59
-- Limit broadcasting. Often attacks will send requests to every device on the network, amplifying the attack. Limiting
60
- or turning off broadcast forwarding where possible can disrupt attacks. Users can also disable echo and chargen
61
- services where possible.
62
-
63
-</details>
64
-
65
-
66
-</details>
67
-
68
-<details>
69
-
70
-<summary>Counter measures on broadcast storms</summary>
71
-
72
-- Switching loops are largely addressed through link aggregation, shortest path bridging, or spanning tree protocol. In
73
- Metro Ethernet rings, it is prevented using the Ethernet Ring Protection Switching (ERPS) or Ethernet Automatic
74
- Protection System (EAPS) protocols.
75
-
76
-- You can filter broadcasts by Layer 3 equipment, most typically routers or even switches that employ advanced filtering.
77
-
78
-- Routers and firewalls can be configured to detect and prevent maliciously inducted broadcast storms
79
-
80
-</details>
\ No newline at end of file
health/guides/net/1m_received_traffic_overflow.md
deleted
-48
@@ -1,48 +0,0 @@
1
-# 1m_received_traffic_overflow
2
-
3
-## OS: Linux
4
-
5
-Network interfaces are categorized primarily on the bandwidth they can operate (1 Gbps, 10 Gbps, etc). High network
6
-utilization occurs when the volume of data on a network link approaches the capacity of the link. Netdata agent
7
-calculates the average outbound utilization for a specific network interface over the last minute. High outbound
8
-utilization increases latency and packet loss because packet bursts are buffered
9
-
10
-This alarm may indicate either network congestion or malicious activity.
11
-
12
-### Troubleshooting section
13
-
14
-<details>
15
- <summary>Prioritize important traffic</summary>
16
-
17
-Quality of service (QoS) is the use of routing prioritization to control traffic and ensure the performance of
18
-critical applications. QoS works best when low-priority traffic exists that can be dropped when congestion occurs. The
19
-higher-priority traffic must fit within the bandwidth limitations of the link or path. The following are two open source
20
-solutions to apply QoS policies to your network interfaces.
21
-
22
-- `FireQOS`:
23
-
24
- FireQOS is a traffic shaping helper. It has a very simple shell scripting language to express traffic shaping.
25
-
26
- [See more on FireQOS](https://firehol.org/tutorial/fireqos-new-user/)
27
-
28
-- `tcconfig`:
29
-
30
- Tcconfig is a command wrapper that makes it easy to set up traffic control of network
31
- bandwidth/latency/packet-loss/packet-corruption/etc.
32
-
33
- [See more on tcconfig](https://tcconfig.readthedocs.io/en/latest/index.html)
34
-
35
-</details>
36
-
37
-
38
-<details>
39
- <summary>Add bandwidth</summary>
40
-
41
-- For **Cloud infrastructures**, adding bandwidth might be easy. It depends on your cloud infrastracture and your cloud
42
- provider. Some of them either offer you the service to upgrade machines to a higher bandwidth rate or upgrade you
43
- machine to a more powerful one with higher bandwidth rate.
44
-
45
-- For **Bare-metal** machines, you will need either a hardware upgrade or the addition of a network card using link
46
- aggregation to combine multiple network connections in parallel (e.g LACP).
47
-
48
-</details>
health/guides/net/1m_sent_traffic_overflow.md
deleted
-47
@@ -1,47 +0,0 @@
1
-# 1m_sent_traffic_overflow
2
-
3
-## OS: Linux
4
-
5
-Network interfaces are categorized primarily on the bandwidth rate at which they can operate (1 Gbps, 10 Gbps, etc). High network
6
-utilization occurs when the volume of data on a network link approaches the capacity of the link. Netdata agent
7
-calculates the average outbound utilization for a specific network interface over the last minute. High outbound
8
-utilization increases latency and packet loss because packet bursts are buffered.
9
-
10
-This alarm may indicate either a network congestion or malicious activity.
11
-
12
-### Troubleshooting section
13
-
14
-<details>
15
- <summary>Prioritize important traffic</summary>
16
-
17
-Quality of service (QoS) is the use of mechanisms or technologies to control traffic and ensure the performance of
18
-critical applications. QoS works best when low-priority traffic exists that can be dropped when congestion occurs. The
19
-higher-priority traffic must fit within the bandwidth limitations of the link or path. The following are two open source
20
-solutions to apply QoS policies to your network interfaces.
21
-
22
-- `FireQOS`:
23
-
24
- FireQOS is a traffic shaping helper. It has a very simple shell scripting language to express traffic shaping.
25
-
26
- [See more on FireQOS](https://firehol.org/tutorial/fireqos-new-user/)
27
-
28
-- `tcconfig`:
29
-
30
- Tcconfig is a command wrapper that makes it easy to set up traffic control of network
31
- bandwidth, latency, packet-loss, packet-corruption, etc.
32
-
33
- [See more on tcconfig](https://tcconfig.readthedocs.io/en/latest/index.html)
34
-
35
-</details>
36
-
37
-<details>
38
- <summary>Add bandwidth</summary>
39
-
40
-- For **Cloud infrastructures**, adding bandwidth might be easy. It depends on your cloud infrastracture and your cloud
41
- provider. Some of them either offer you the service to upgrade machines to a higher bandwidth rate or upgrade you
42
- machine to a more powerful one with higher bandwidth rate.
43
-
44
-- For **Bare-metal** machines you will need either a hardware upgrade or the addition of a network card using link
45
- aggregation to combine multiple network connections in parallel (e.g LACP).
46
-
47
-</details>
health/guides/net/inbound_packets_dropped_ratio.md
deleted
-83
@@ -1,83 +0,0 @@
1
-# inbound_packets_dropped_ratio
2
-
3
-## OS: Linux
4
-
5
-Packet drops indicate that your system received some packets but could not process them. A sizeable amount of packet
6
-drops can consume significant amount of resources in your system. Some reasons that packets drops occurred in your
7
-system could be:
8
-
9
-- Your system receives packets with bad VLAN tags.
10
-- The packets you are receiving are using a protocol that is unknown to your system.
11
-- You receive IPv6 packets, but your system is not configured for IPv6.
12
-
13
-All these packets consume resources until being dropped (and for a short period after). For example, your NIC stores them
14
-in a ring-buffer until they are forwarded to the destined subsystem or userland application for further process.
15
-
16
-Netdata agent calculates the ratio of inbound dropped packets for your wired network interface over the last 10 minutes.
17
-
18
-### Troubleshooting section:
19
-
20
-The best way to resolve these kinds of problems is to be extremely knowledgeable about your network topologies. These
21
-are some actions you might consider check:
22
-
23
-<details>
24
- <summary>Identify VLANs in your interface</summary>
25
-
26
-There are cases in which traffic is routed to your host due to the existence of multiple VLAN in your network.
27
-
28
-1. Identify VLAN tagged packet in your interface.
29
-
30
-```
31
-root@netdata # tcpdump -i <your_interface> -nn -e vlan
32
-```
33
-
34
-2. Monitor the output of the `tcpdump`, identify VLANs which may exist. If no output is displayed, your interface
35
- probably uses traditional ethernet frames.
36
-
37
-3. Depending on your network topology, you may consider removing unnecessary VLANs from the switch trunk port
38
- toward your host.
39
-
40
-</details>
41
-
42
-<details>
43
-<summary>Update the ring buffer size on your interface</summary>
44
-
45
-1. To view the maximum RX ring buffer size:
46
-
47
- ```
48
- root@netdata ~ # ethtool -g enp1s0
49
- Ring parameters for enp1s0:
50
- Pre-set maximums:
51
- RX: 4080
52
- RX Mini: 0
53
- RX Jumbo: 16320
54
- TX: 255
55
- Current hardware settings:
56
- RX: 255
57
- RX Mini: 0
58
- RX Jumbo: 0
59
- TX: 255
60
- ```
61
-
62
-2. If the values in the Pre-set maximums section are higher than in the current hardware settings section, increase RX
63
- ring buffer:
64
-
65
- ```
66
- root@netdata ~ # enp1s0 rx 4080
67
- ```
68
-
69
-3. Verify the change to make sure that you no longer receive the alarm when running the same workload. To make this
70
- permanently, you must consult your distribution guides.
71
-
72
-</details>
73
-
74
-<details>
75
-<summary>Inspect the packets your network interface receives</summary>
76
-
77
-Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and
78
-communications protocol development.
79
-
80
-[See more about Wireshark here](https://www.wireshark.org/)
81
-
82
-</details>
83
-
health/guides/net/interface_inbound_errors.md
deleted
-120
@@ -1,120 +0,0 @@
1
-# interface_inbound_errors
2
-
3
-## OS: FreeBSD
4
-
5
-When a packet is received by your system, it can be processed in one of four ways:
6
-
7
-- It can be passed as input to a higher-level protocol.
8
-
9
-- It can encounter an error which is reported back to the source.
10
-
11
-- It can be dropped due to an error.
12
-
13
-- It can be forwarded to the next hop on its path to its destination.
14
-
15
-There are mechanisms to identify packets with errors and verify the integrity of the packet such as the Cyclic
16
-Redundancy Check (CRC), the Frame check sequence (FCS), the header checksum (IPv4), and length checks. The Netdata agent
17
-monitors the number of inbound errors for a specific network interface in the last 10 minutes.
18
-
19
-
20
-<details>
21
-<summary>The life of a packet</summary>
22
-
23
-The following list from "Design and Implementation of the FreeBSD Operating System, The, 2nd Edition" (McKusick,
24
-Neville-Neil and Watson) [[1]](https://www.pearson.com/us/higher-education/program/Mc-Kusick-Design-and-Implementation-of-the-Free-BSD-Operating-System-The-2nd-Edition/PGM224032.html)
25
-provides a brief description of every action taken by your system for every packet it receives:
26
-
27
-1. Verifies that the packet is at least as long as an IPv4 or IPv6 header and ensures that the header is contiguous.
28
-
29
-2. For IPv4, checksums the header of the packet, and discards the packet if there is an error.
30
-
31
-3. Verifies that the packet is at least as long as the header indicates, and drops the packet if it is not.
32
-
33
-4. Does any filtering or security functions (ipfw, IPSec).
34
-
35
-5. Processes any options associated with the header.
36
-
37
-6. Checks whether the packet is for this host. If it is, continues processing the packet. If it is not, and if the
38
- system is acting as a router, your system will try to forward the packet. Otherwise, the packet is dropped.
39
-
40
-7. If the packet has been fragmented, keeps it until all its fragments are received and reassembled, If the reassemble
41
- process takes a significant amount of time, the system drops it.
42
-
43
-8. Passes the packet to the input routine of the next-higher-level protocol.
44
-
45
-</details>
46
-
47
-<details>
48
-<summary>See more on CRC</summary>
49
-
50
-> A cyclic redundancy check (CRC) is an error-detecting code commonly used in digital networks and storage devices to
51
-> detect accidental changes to raw data. Blocks of data entering these systems get a short check value attached, based
52
-> on the remainder of a polynomial division of their contents. On retrieval, the calculation is repeated and, in the
53
-> event the check values do not match, corrective action can be taken against data
54
-> corruption. [[2]](https://en.wikipedia.org/wiki/Cyclic_redundancy_check)
55
-
56
-</details>
57
-
58
-<details>
59
-<summary>See more on FCS</summary>
60
-
61
-> A frame check sequence (FCS) is an error-detecting code added to a frame in a communication protocol. All frames and
62
-> the bits, bytes, and fields contained within them, are susceptible to errors from a variety of sources. The FCS field
63
-> contains a number that is calculated by the source node based on the data in the frame. This number is added to the
64
-> end of a frame that is sent. When the destination node receives the frame the FCS number is recalculated and compared
65
-> with the FCS number included in the frame. If the two numbers are different, an error is assumed and the frame
66
-> is discarded. [[3]](https://en.wikipedia.org/wiki/Frame_check_sequence)
67
-
68
-</details>
69
-
70
-<details>
71
-<summary>See more on header checksum</summary>
72
-
73
-> The IPv4 header checksum is a checksum used in version 4 of the Internet Protocol (IPv4) to detect corruption in the
74
-> header of IPv4 packets. It is carried in the IP packet header and represents the 16-bit result of summation of the
75
-> header words. [[4]](https://en.wikipedia.org/wiki/IPv4_header_checksum)
76
-
77
-</details>
78
-
79
-<details>
80
-<summary>References and sources</summary>
81
-
82
-1. [Book: Design and Implementation of the FreeBSD Operating System (2nd-Edition)](https://www.pearson.com/us/higher-education/program/Mc-Kusick-Design-and-Implementation-of-the-Free-BSD-Operating-System-The-2nd-Edition/PGM224032.html)
83
-
84
-1. [Cyclic redundancy check protocol](https://en.wikipedia.org/wiki/Cyclic_redundancy_check)
85
-
86
-1. [Frame check sequence protocol](https://en.wikipedia.org/wiki/Frame_check_sequence)
87
-
88
-1. [IPv4 header checksum protocol](https://en.wikipedia.org/wiki/IPv4_header_checksum)
89
-
90
-
91
-</details>
92
-
93
-### Troubleshooting section:
94
-
95
-<details>
96
-<summary>General approach</summary>
97
-
98
-In any case, a good starting point is to get more information about the nature of your errors.
99
-
100
-- Netdata dashboard provides an overview of these errors. You can see more in the `errors` chart under the IPv4 (or
101
- IPv6) section.
102
-
103
-- `netstat` (network statistics) is a command-line network utility that displays network connections for Transmission
104
- Control Protocol, routing tables and network protocol statistics for any interface in your system.
105
-
106
- ```
107
- root@netdata~ # netstat -sI <your_interface>
108
- ```
109
-
110
-</details>
111
-
112
-<details>
113
-<summary>Troubleshoot hardware errors in the link of the interface</summary>
114
-
115
-You must identify which part of your topology causes these errors. Some actions you can take.
116
-
117
-- Remove and re-install the optical fibers and optical modules and check whether the fiber connectors are damaged or
118
- contaminated. For ethernet interfaces check for damaged cables and/or for damages in the interfaces themselves.
119
-
120
-
\ No newline at end of file
health/guides/net/interface_outbound_errors.md
deleted
-75
@@ -1,75 +0,0 @@
1
-# interface_outbound_errors
2
-
3
-## OS: FreeBSD
4
-
5
-When we want to investigate the outbound traffic, the journey of a network packet starts at the application layer. Data
6
-are written (commonly) to a socket by a user program. The programmer may (raw sockets) or may not (datagram and stream
7
-sockets) have the possibility of absolute control over the data which is being sent through the network. The kernel will
8
-take the data which is written in a socket queue and allocate the necessary socket buffers. The kernel will try to
9
-forward the packets to their destination encapsulating the routing metadata (headers, checksums, fragmentation
10
-information) for each packet through a network interface. The Netdata agent monitors the number of outbound errors for a
11
-specific network interface in the last 10 minutes. Some of the errors that may occur in this process include:
12
-
13
-- Errors due to aborted connections
14
-
15
-- Carrier sense errors
16
-
17
-- FIFO errors
18
-
19
-- Heartbeat errors
20
-
21
-- Window errors
22
-
23
-<details>
24
- <summary>See more on Carrier Sense Errors</summary>
25
-
26
-Carrier Sense Errors occur when an interface attempts to transmit a frame, but no carrier is detected. In that case if
27
-the frame cannot be transmitted, it is discarded.
28
-
29
-</details>
30
-
31
-
32
-<details>
33
- <summary>See more about heartbeat </summary>
34
-
35
-> A heartbeat protocol is generally used to negotiate and monitor the availability of a resource, such as a floating IP
36
-> address, and the procedure involves sending network packets to all the nodes in the culture to verify its
37
-> reachability. Typically when a heartbeat starts on a machine, it will perform an election process with other machines
38
-> on the heartbeat network to determine which machine, if any, owns the resource. On heartbeat networks of more than two
39
-> machines, it is important to take into account partitioning, where two halves of the network could be functioning but
40
-> not able to communicate with each other. In a situation such as this, it is important that the resource is only owned
41
-> by one machine, not one machine in each partition.
42
->
43
-> As a heartbeat is intended to be used to indicate the health of a machine, it is important that the heartbeat protocol
44
-> and the transport that it runs on are as reliable as possible. Causing a failover because of a false alarm may
45
-> depending on the resource, be highly undesirable. It is also important to react quickly to an actual failure, further
46
-> signifig the reliability of the heartbeat messages. For this reason, it is often desirable to have a heartbeat running
47
-> over more than one transport; for instance, an Ethernet segment using UDP/IP, and a serial
48
-> link. <sup> [1](https://en.wikipedia.org/wiki/Heartbeat_(computing)</sup>
49
-
50
-</details>
51
-
52
-
53
-<details>
54
-<summary>References and sources:</summary>
55
-
56
-1. [Heartbeat definition on Wikipedia](https://en.wikipedia.org/wiki/Heartbeat_(computing))
57
-
58
-</details>
59
-
60
-### Troubleshooting section:
61
-
62
-<details>
63
-<summary>General approach</summary>
64
-
65
-In any case, a good starting point is to get more information about the nature of your errors.
66
-
67
-- `netstat` (network statistics) is a command-line network utility that displays, network connections for Transmission
68
- Control Protocol, routing tables and network protocol statistics for any interface in your system.
69
-
70
- ```
71
- root@netdata~ # netstat -sI <your_interface>
72
- ```
73
-
74
-</details>
75
-
health/guides/net/outbound_packets_dropped_ratio.md
deleted
-41
@@ -1,41 +0,0 @@
1
-# outbound_packets_dropped_ratio
2
-
3
-## OS: Linux
4
-
5
-When we want to investigate the outbound traffic, the journey of a network packet starts at the
6
-application layer. Data are written (commonly) to a socket by a user program. The programmer may (
7
-raw sockets) or may not (datagram and stream sockets) have the possibility of absolute control over
8
-the data which is being sent through the network. The kernel will take the data which is written in
9
-a socket queue and allocate the necessary socket buffers. The kernel will try to forward the packets
10
-to their destination encapsulating the routing metadata (headers, checksums, fragmentation
11
-information) for each packet through a network interface.
12
-
13
-The Netdata Agent calculates the ratio of outbound dropped packets for a specific network interface
14
-over the last 10 minutes. Receiving this alarm means that packets were dropped on their way to
15
-transmission.
16
-
17
-This alert is triggered in warning state when the ratio of outbound dropped packets for a specific
18
-network interface over the last 10 minutes is more than 2%.
19
-
20
-The main reasons of outbound packet drops are:
21
-
22
-1. Link congestion
23
-1. Overburdened devices
24
-1. Defective hardware
25
-1. Faulty network configuration
26
-1. Restricted access from firewall rules
27
-
28
-### Troubleshooting section:
29
-
30
-The best way to resolve these kind of problems is to be extremely knowledgeable about your network
31
-topologies.
32
-
33
-<details>
34
-<summary>Inspect the packets your network interface sends</summary>
35
-
36
-Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting,
37
-analysis, software and communications protocol development.
38
-
39
-[See more about Wireshark here](https://www.wireshark.org/)
40
-
41
-</details>
health/guides/netfilter/netfilter_conntrack_full.md
deleted
-66
@@ -1,66 +0,0 @@
1
-# netfilter_conntrack_full
2
-
3
-## OS: Linux
4
-
5
-This alert presents the percentage of used netfilter tracked connections. \
6
-If you receive this alert, there is high utilization of the netfilter "connection tracking state"
7
-table size.
8
-
9
-Network delays and packet drops are expected when you are getting closer to 100%.
10
-
11
-- This alert gets raised to warning when the percentage exceeds 90%.
12
-- If the metric exceeds 95%, then the alert gets raised to a critical state.
13
-
14
-<details>
15
-<summary>What is the "netfilter" framework</summary>
16
-
17
-> Netfilter is a framework provided by the Linux kernel that allows various networking-related
18
-> operations to be implemented in the form of customized handlers. Netfilter offers various
19
-> functions and operations for packet filtering, network address translation, and port
20
-> translation, which provide the functionality required for directing packets through a network
21
-> and prohibiting packets from reaching sensitive locations within a network.<sup>[1](
22
-> https://en.wikipedia.org/wiki/Netfilter) </sup>
23
-
24
-</details>
25
-
26
-<br>
27
-
28
-<details>
29
-<summary>References and Sources</summary>
30
-
31
-1. [Netfilter](https://en.wikipedia.org/wiki/Netfilter)
32
-2. [Full Conntrack Table](
33
- https://morganwu277.github.io/2018/05/26/Solve-production-issue-of-nf-conntrack-table-full-dropping-packet/)
34
-
35
-</details>
36
-
37
-### Troubleshooting Section
38
-
39
-<details>
40
-<summary>Table Size Limits</summary>
41
-
42
-You can see the table size by running:
43
-
44
-```
45
-root@netdata~ # cat /proc/sys/net/netfilter/nf_conntrack_count
46
-```
47
-
48
-You can see the table size limit by running:
49
-
50
-```
51
-root@netdata~ # cat /proc/sys/net/netfilter/nf_conntrack_max
52
-```
53
-
54
-> Recommended Size: CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 / (ARCH / 32).
55
-> Eg, If we have 8GB RAM in a x86_64 OS, we would use 8*1024^3/16384/2=262144.<sup> [2](
56
-> https://morganwu277.github.io/2018/05/26/Solve-production-issue-of-nf-conntrack-table-full-dropping-packet/)
57
-> </sup>
58
-
59
-You can modify the table size limit by running:
60
-
61
-```
62
-root@netdata~ # sysctl -w net.netfilter.nf_conntrack_max=<YOUR DESIRED LIMIT HERE>
63
-root@netdata~ # echo "net.netfilter.nf_conntrack_max=<YOUR DESIRED LIMIT HERE>" >> /etc/sysctl.conf
64
-```
65
-
66
-</details>
health/guides/netfilter_conntrack_full.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+This alert presents the percentage of used netfilter tracked connections. If you receive this alert, there is high utilization of the netfilter "connection tracking state" table size.
4
+
5
+Network delays and packet drops are expected when you are getting closer to 100%.
6
+
7
+- This alert gets raised to warning when the percentage exceeds 90%.
8
+- If the metric exceeds 95%, then the alert gets raised to a critical state.
9
+
10
+### What is the "netfilter" framework?
11
+
12
+Netfilter is a framework provided by the Linux kernel that allows various networking-related operations to be implemented in the form of customized handlers. Netfilter offers various functions and operations for packet filtering, network address translation, and port translation, which provide the functionality required for directing packets through a network and prohibiting packets from reaching sensitive locations within a network.
13
+
14
+### Troubleshoot the alert
15
+
16
+- Table Size Limits
17
+
18
+You can see the table size by running:
19
+
20
+```
21
+cat /proc/sys/net/netfilter/nf_conntrack_count
22
+```
23
+
24
+You can see the table size limit by running:
25
+
26
+```
27
+cat /proc/sys/net/netfilter/nf_conntrack_max
28
+```
29
+
30
+Recommended Size: CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 / (ARCH / 32).
31
+Eg, If we have 8GB RAM in a x86_64 OS, we would use 8*1024^3/16384/2=262144.
32
+
33
+You can modify the table size limit by running:
34
+
35
+```
36
+sysctl -w net.netfilter.nf_conntrack_max=<YOUR DESIRED LIMIT HERE>
37
+echo "net.netfilter.nf_conntrack_max=<YOUR DESIRED LIMIT HERE>" >> /etc/sysctl.conf
38
+```
39
+
40
+</details>
41
+
42
+
43
+### Useful resources
44
+
45
+1. [Netfilter](https://en.wikipedia.org/wiki/Netfilter)
46
+2. [Full Conntrack Table](https://morganwu277.github.io/2018/05/26/Solve-production-issue-of-nf-conntrack-table-full-dropping-packet/)
health/guides/nut_10min_ups_load.md
new
+38
@@ -0,0 +1,38 @@
1
+### Understand the alert
2
+
3
+This alert is based on the `nut_10min_ups_load` metric, which measures the average UPS load over the last 10 minutes. If you receive this alert, it means that the load on your UPS is higher than expected, which may lead to an unstable power supply and ungraceful system shutdowns.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Verify the UPS load status
8
+
9
+ Check the current load on the UPS using the `upsc` command with your UPS identifier:
10
+ ```
11
+ upsc <your_ups_identifier>
12
+ ```
13
+ Look for the `ups.load` metric in the command output to identify the current load percentage.
14
+
15
+2. Analyze the connected devices
16
+
17
+ Make an inventory of all devices connected to the UPS, including servers, networking devices, and other equipment. Determine if all devices are essential or if some can be moved to another power source or disconnected entirely.
18
+
19
+3. Balance the load between multiple UPS units (if available)
20
+
21
+ If you have more than one UPS, consider distributing the connected devices across multiple units to balance the load and ensure that each UPS isn't overloaded.
22
+
23
+4. Upgrade or replace the UPS
24
+
25
+ If necessary, consider upgrading your UPS to a higher capacity model to handle the increased load or replacing the current unit if it's malfunctioning or unable to provide the required power.
26
+
27
+5. Monitor power usage trends
28
+
29
+ Regularly review your power usage patterns and system logs, and take action to prevent load spikes that could trigger the `nut_10min_ups_load` alert.
30
+
31
+6. Optimize device power consumption
32
+
33
+ Implement power-saving strategies for connected devices, such as enabling power-saving modes, reducing CPU usage, or using power-efficient networking equipment.
34
+
35
+### Useful resources
36
+
37
+1. [NUT user manual](https://networkupstools.org/docs/user-manual.chunked/index.html)
38
+2. [Five steps to reduce UPS energy consumption](https://sp.ts.fujitsu.com/dmsp/Publications/public/wp-reduce-ups-energy-consumption-ww-en.pdf)
health/guides/nut_last_collected_secs.md
new
+34
@@ -0,0 +1,34 @@
1
+### Understand the alert
2
+
3
+This alert is related to the Network UPS Tools (NUT) which monitors power devices, such as uninterruptible power supplies, power distribution units, solar controllers, and server power supply units. If you receive this alert, it means that there is an issue with the data collection process and needs troubleshooting to ensure the monitoring process works correctly.
4
+
5
+### Troubleshoot the alert
6
+
7
+#### Check the upsd server
8
+
9
+1. Check the status of the upsd daemon:
10
+
11
+ ```
12
+ $ systemctl status upsd
13
+ ```
14
+
15
+2. Check for obvious and common errors in the log or output. If any errors are found, resolve them accordingly.
16
+
17
+3. Restart the daemon if needed:
18
+
19
+ ```
20
+ $ systemctl restart upsd
21
+ ```
22
+
23
+#### Diagnose a bad driver
24
+
25
+1. `upsd` expects the drivers to either update their status regularly or at least answer periodic queries, called pings. If a driver doesn't answer, `upsd` will declare it "stale" and no more information will be provided to the clients.
26
+
27
+2. If upsd complains about staleness when you start it, then either your driver or configuration files are probably broken. Be sure that the driver is actually running, and that the UPS definition in [ups.conf(5)](https://networkupstools.org/docs/man/ups.conf.html) is correct. Also, make sure that you start your driver(s) before starting upsd.
28
+
29
+3. Data can also be marked stale if the driver can no longer communicate with the UPS. In this case, the driver should also provide diagnostic information in the syslog. If this happens, check the serial or USB cabling, or inspect the network path in the case of a SNMP UPS.
30
+
31
+### Useful resources
32
+
33
+1. [NUT User Manual](https://networkupstools.org/docs/user-manual.chunked/index.html)
34
+2. [ups.conf(5)](https://networkupstools.org/docs/man/ups.conf.html)
\ No newline at end of file
health/guides/nut_ups_charge.md
new
+38
@@ -0,0 +1,38 @@
1
+### Understand the alert
2
+
3
+The `nut_ups_charge` alert indicates that the average UPS charge over the last minute has dropped below a predefined threshold. This might be due to a power outage, a UPS malfunction, or a sudden surge in power demands that the UPS can't handle.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check UPS status and connections
8
+
9
+Inspect the UPS physical connections, including power cables, communication cables, and any other devices connected to it. Ensure that everything is plugged in correctly and firmly.
10
+
11
+2. Check UPS logs and error messages
12
+
13
+Review the UPS logs for any error messages or events that might have occurred around the time the alert was triggered. This information could help you pinpoint the cause of the issue. You can find the logs in the Network UPS Tools (NUT) software.
14
+
15
+3. Monitor UPS charge level
16
+
17
+Keep an eye on the UPS charge level to determine if it's increasing or decreasing. This information can help you understand the overall health of your UPS.
18
+
19
+4. Test UPS batteries
20
+
21
+Test the UPS batteries to ensure that they are functioning correctly and have enough charge to power your devices during a power outage. Replace any faulty batteries or upgrade to higher-capacity batteries if needed.
22
+
23
+5. Check the UPS load
24
+
25
+Review the devices connected to the UPS and calculate their total power consumption. Ensure that the UPS is not overloaded and is capable of supporting the power demands of your devices.
26
+
27
+6. Restore the power supply
28
+
29
+If the UPS charge level remains low, try restoring the power supply to your UPS. This could involve switching to a different power source, fixing any faulty connections, or resolving issues with your local power grid.
30
+
31
+7. Prepare for a graceful shutdown
32
+
33
+If you can't restore the power supply to this UPS or if the problem persists,prepare your machine for a graceful shutdown to minimize the risk of data loss or hardware damage.
34
+
35
+### Useful resources
36
+
37
+1. [NUT User Manual](https://networkupstools.org/docs/user-manual.chunked/index.html)
38
+2. [UPS troubleshooting guide](https://www.apc.com/us/en/faqs/FA158852/)
health/guides/nvme_device_critical_warnings_state.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when an `NVMe device` experiences `critical warnings`. The alert is focusing on your `NVMe` (Non-Volatile Memory Express) SSD storage device which is designed for high-performance and low-latency storage.
4
+
5
+### What does critical warnings mean?
6
+
7
+A critical warning state indicates that the NVMe device has experienced an event, error, or condition which could negatively impact performance, data integrity or device longevity. This could result from a variety of reasons such as high temperature, hardware failures, internal errors, or device reaching end of life.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the affected NVMe device(s):
12
+
13
+This alert provides information in the `info` field about the affected device. It should look like: "NVMe device ${label:device} has critical warnings", where `${label:device}` will be replaced with the actual device name.
14
+
15
+2. Check device SMART information:
16
+
17
+`SMART` (Self-Monitoring, Analysis, and Reporting Technology) provides detailed information about the current health and performance of your NVMe device. To check SMART information for the affected NVMe device, use `smartctl` command:
18
+
19
+ ```
20
+ sudo smartctl -a /dev/nvme0n1
21
+ ```
22
+
23
+ Replace `/dev/nvme0n1` with the actual device name identified in step 1.
24
+
25
+3. Evaluate the SMART information for critical issues:
26
+
27
+Review the output of the `smartctl` command to identify the critical warnings or any other concerning attributes. You might see high temperature, high uncorrectable error counts, or high percent of used endurance. These values might help you diagnose the issue with your NVMe device.
28
+
29
+4. Take appropriate action based on SMART data:
30
+
31
+- If the temperature of the device is high, ensure proper cooling and airflow in the system.
32
+- If the device is reaching its end of life, plan for a replacement or backup.
33
+- If the device has high uncorrectable error counts, consider backing up critical data and contact the manufacturer for support, as this could indicate a possible hardware failure.
34
+
35
+Make sure to replace, stop the usage of, or seek support for the problematic NVMe device(s) depending on the analysis.
36
+
health/guides/oom_kill.md
new
+89
@@ -0,0 +1,89 @@
1
+### Understand the alert
2
+
3
+The OOM Killer (Out of Memory Killer) is a process that the Linux kernel uses when the system is critically low on memory or a process reached its memory limits. As the name suggests, it has the duty to review all running processes and kill one or more of them in order to free up memory and keep the system running.
4
+
5
+Linux Kernel 4.19 introduced cgroup awareness of OOM killer implementation which adds an ability to kill a cgroup as a single unit and to guarantee the integrity of the workload. In a nutshell, cgroups allow the limitation of memory, disk I/O, and network usage for a group of processes. Furthermore, cgroups may set usage quotas, and prioritize a process group to receive more CPU time or memory than other groups. You can see more about cgroups in
6
+the [cgroup man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html)
7
+
8
+The Netdata Agent monitors the number of Out Of Memory (OOM) kills in the last 30 minutes. Receiving this alert indicates that some processes got killed by OOM Killer.
9
+
10
+### Troubleshoot the alert
11
+
12
+- Troubleshoot issues in the OOM killer
13
+
14
+The OOM Killer uses a heuristic system to choose a processes for termination. It is based on a score associated with each running application, which is calculated by `oom_badness()` call inside Linux kernel
15
+
16
+1. To identify which process/apps was killed from the OOM killer, inspect the logs:
17
+
18
+```
19
+dmesg -T | egrep -i 'killed process'
20
+```
21
+The system response looks similar to this:
22
+```
23
+Jan 7 07:12:33 mysql-server-01 kernel: Out of Memory: Killed process 3154 (mysqld).
24
+```
25
+
26
+2. To see the current `oom_score` (the priority in which OOM killer will act upon your processes) run the following script.
27
+The script prints all running processes (by pid and name) with likelihood to be killed by the OOM killer (second column).
28
+The greater the `oom_score` (second column) the more propably to be killed by OOM killer.
29
+
30
+```
31
+while read -r pid comm; do
32
+ printf '%d\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$comm";
33
+done < <(ps -e -o pid= -o comm=) | sort -k 2n
34
+```
35
+
36
+3. Adjust the `oom_score` to protect processes using the `choom` util from
37
+the `util-linux` [package v2.33-rc1+](https://github.com/util-linux/util-linux/commit/8fa223daba1963c34cc828075ce6773ff01fafe3)
38
+
39
+```
40
+choom -p PID -n number
41
+```
42
+
43
+4. Once the settings work to your case, make the change permanent. In the unit file of your service, under the [Service] section, add the following value: `OOMScoreAdjust=<PREFFERRED_VALUE>`
44
+
45
+- Add a temporary swap file</summary>
46
+
47
+Keep in mind this requires creating a swap file in one of the disks. Performance of your system may be affected.
48
+
49
+1. Decide where your swapfile will live. It is strongly advised to allocate the swap file under in
50
+ the root directory. A swap file is like an extension of your RAM and it should be protected, far
51
+ from normal user accessible directories. Run the following command:
52
+
53
+ ```
54
+ dd if=/dev/zero of=<path_in_root> bs=1024 count=<size_in_bytes>
55
+ ```
56
+
57
+2. Grant root only access to the swap file:
58
+
59
+ ```
60
+ chmod 600 <path_to_the_swap_file_you_created>
61
+ ```
62
+
63
+3. Make it a Linux swap area:
64
+
65
+ ```
66
+ mkswap <path_to_the_swap_file_you_created>
67
+ ```
68
+
69
+4. Enable the swap with the following command:
70
+
71
+ ```
72
+ swapon <path_to_the_swap_file_you_created>
73
+ ```
74
+
75
+5. If you plan to use it a regular basis, you should update the `/etc/fstab` config. The entry you
76
+ will add would look like:
77
+
78
+ ```
79
+ /swap_file swap sw 0 0
80
+ ```
81
+
82
+ For more information see the fstab manpage: `man fstab`.
83
+
84
+
85
+### Useful resources
86
+
87
+1. [Linux Out of Memory Killer](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)
88
+2. [Memory Resource Controller in linux kernel](https://docs.kernel.org/admin-guide/cgroup-v1/memory.html?highlight=oom)
89
+3. [OOM killer blogspot](https://www.psce.com/en/blog/2012/05/31/mysql-oom-killer-and-everything-related/)
health/guides/outbound_packets_dropped.md
new
+57
@@ -0,0 +1,57 @@
1
+### Understand the alert
2
+
3
+This alert tracks the number of dropped outbound packets on a specific network interface (`${label:device}`) within the last 10 minutes. If you receive this alert, it means that your system has experienced dropped outbound packets in the monitored network interface, which might indicate network congestion or other issues affecting network performance.
4
+
5
+### What are dropped packets?
6
+
7
+Dropped packets refer to network packets that are discarded or lost within a computer network during transmission. In general, this can be caused by various factors, such as network congestion, faulty hardware, misconfigured devices, or packet errors.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the affected network interface:
12
+
13
+Check the alert message for the `${label:device}` placeholder. It indicates the network interface experiencing the dropped outbound packets.
14
+
15
+2. Verify network congestion or excessive traffic:
16
+
17
+Excessive traffic or network congestion can lead to dropped packets. To check network traffic, use the `nload` tool. If it isn't installed, you can follow the instructions given [here](https://www.howtoforge.com/tutorial/install-nload-on-linux/).
18
+
19
+```bash
20
+nload ${label:device}
21
+```
22
+
23
+This will display the current network bandwidth usage on the specified interface. Look for unusually high or fluctuating usage patterns, which could indicate congestion or excessive traffic.
24
+
25
+3. Verify hardware issues:
26
+
27
+Check the network interface and related hardware components (such as the network card, cables, and switches) for visible damage, loose connections, or other issues. Replace any defective components as needed.
28
+
29
+4. Check network interface configuration:
30
+
31
+Review your network interface configuration to ensure that it is correctly set up. To do this, you can use the `ip` or `ifconfig` command. For example:
32
+
33
+```bash
34
+ip addr show ${label:device}
35
+```
36
+
37
+or
38
+
39
+```bash
40
+ifconfig ${label:device}
41
+```
42
+
43
+Verify that the IP address, subnet mask, and other network settings match your network configuration.
44
+
45
+5. Check system logs for networking errors:
46
+
47
+Review your system logs to identify any networking error messages that might provide more information on the cause of the dropped packets.
48
+
49
+```bash
50
+grep -i "error" /var/log/syslog | grep "${label:device}"
51
+```
52
+
53
+6. Monitor your network for packet errors using tools like `tcpdump` or `wireshark`.
54
+
55
+### Useful resources
56
+
57
+1. [How to monitor network bandwidth and traffic in Linux](https://www.binarytides.com/linux-commands-monitor-network/)
health/guides/outbound_packets_dropped_ratio.md
new
+27
@@ -0,0 +1,27 @@
1
+### Understand the alert
2
+
3
+When we want to investigate the outbound traffic, the journey of a network packet starts at the application layer.
4
+
5
+Data are written (commonly) to a socket by a user program. The programmer may (raw sockets) or may not (datagram and stream sockets) have the possibility of absolute control over the data which is being sent through the network. The kernel will take the data which is written in a socket queue and allocate the necessary socket buffers. The kernel will try to forward the packets to their destination encapsulating the routing metadata (headers, checksums, fragmentation information) for each packet through a network interface.
6
+
7
+The Netdata Agent calculates the ratio of outbound dropped packets for a specific network interface over the last 10 minutes. Receiving this alarm means that packets were dropped on their way to transmission.
8
+
9
+This alert is triggered in warning state when the ratio of outbound dropped packets for a specific network interface over the last 10 minutes is more than 2%.
10
+
11
+The main reasons of outbound packet drops are:
12
+
13
+1. Link congestion
14
+2. Overburdened devices
15
+3. Defective hardware
16
+4. Faulty network configuration
17
+5. Restricted access from firewall rules
18
+
19
+### Troubleshoot the alert:
20
+
21
+Inspect the packets your network interface sends using Wireshark.
22
+
23
+Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development.
24
+
25
+### Useful resources
26
+
27
+[Read more about Wireshark here](https://www.wireshark.org/)
\ No newline at end of file
health/guides/pihole/pihole_blocklist_gravity_file.md
deleted
-30
@@ -1,30 +0,0 @@
1
-# pihole_blocklist_gravity_file
2
-
3
-## Ad Filtering | Pi-hole
4
-
5
-This alert indicates the existence of the blocklist file. If you receive this, it means that the
6
-gravity.list file (blocklist) is non-existent.
7
-
8
-- The alert is raised in a critical state when the metric gets the value of 1.
9
-
10
-> The gravity table consists of the domains that have been processed by Pi-hole's gravity
11
-> (pihole -g) command. The domains in this list are the collection of domains sourced from the
12
-> configured sources (see the adlist table).<sup>[1](
13
-> https://docs.pi-hole.net/database/gravity/#gravity-table-gravity) </sup>
14
-<details><summary>References and Sources</summary>
15
-
16
-1. [Pi-hole Docs](https://docs.pi-hole.net/database/gravity/#gravity-table-gravity)
17
-
18
-</details>
19
-
20
-### Troubleshooting Section
21
-
22
-<details>
23
-<summary>Rebuild the blocklist</summary>
24
-To rebuild the gravity.list (blocklist), run the command:
25
-
26
-```
27
-root@netdata~ # pihole -g
28
-```
29
-
30
-</details>
\ No newline at end of file
health/guides/pihole/pihole_blocklist_last_update.md
deleted
-31
@@ -1,31 +0,0 @@
1
-# pihole_blocklist_last_update
2
-
3
-## Ad Filtering | Pi-hole
4
-
5
-This alert presents how much time has passed from the time the blocklist file (Gravity) was
6
-updated in seconds.
7
-Receiving this means that the blocklist file has not been updated for a long time.
8
-
9
-- This alert is raised to warning when the time in seconds exceeds 8 days.
10
-- If the metric exceeds 16 days, then the alert is raised to critical.
11
-
12
-> The gravity table consists of the domains that have been processed by Pi-hole's gravity
13
-> (pihole -g) command. The domains in this list are the collection of domains sourced from the
14
-> configured sources (see the adlist table).<sup>[1](
15
-> https://docs.pi-hole.net/database/gravity/#gravity-table-gravity) </sup>
16
-
17
-<details><summary>References and Sources</summary>
18
-
19
-1. [Pi-hole Docs](https://docs.pi-hole.net/database/gravity/#gravity-table-gravity)
20
-</details>
21
-
22
-### Troubleshooting Section
23
-
24
-<details>
25
-<summary>Rebuild the blocklist</summary>
26
-To rebuild the blocklist, run the command:
27
-
28
-```
29
-root@netdata~ # pihole -g
30
-```
31
-</details>
health/guides/pihole/pihole_status.md
deleted
-21
@@ -1,21 +0,0 @@
1
-# pihole_status
2
-
3
-## Ad Filtering | Pi-hole
4
-
5
-This alert monitors if Pi-hole's ability of blocking unwanted domains is active.
6
-
7
-- It is triggered in a warning state if pi-hole is disabled.
8
-
9
-
10
-### Troubleshooting Section
11
-
12
-<details>
13
-<summary>Rebuild the blocklist</summary>
14
-To fix, run the command:
15
-
16
-```
17
-root@netdata~ # pihole enable
18
-```
19
-
20
-*This feature should be enabled. The whole point of Pi-hole!*
21
-</details>
\ No newline at end of file
health/guides/pihole_blocklist_last_update.md
new
+55
@@ -0,0 +1,55 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the Pi-hole blocklist (Gravity) file hasn't been updated for an extended period of time. The blocklist file contains domains that have been processed by Pi-hole to filter ads and malicious content. An outdated blocklist may leave your system more vulnerable to unwanted content and threats.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the current blocklist update status**
8
+
9
+ To see how long it has been since the last update, you can use the following command:
10
+
11
+ ```
12
+ root@netdata~ # pihole -q -adlist
13
+ ```
14
+
15
+ This will display the timestamp of the last update.
16
+
17
+2. **Rebuild the blocklist**
18
+
19
+ If the alert indicates that your blocklist file is outdated, it's essential to update it by running:
20
+
21
+ ```
22
+ root@netdata~ # pihole -g
23
+ ```
24
+
25
+ This command will download the necessary files and rebuild the blocklist.
26
+
27
+3. **Check for errors during the update**
28
+
29
+ If you encounter any issues during the update, check the `/var/log/pihole.log` file for errors. You can also check the `/var/log/pihole-FTL.log` file for more detailed information on the update process.
30
+
31
+4. **Verify the blocklist update interval**
32
+
33
+ To ensure that your blocklist file is updated regularly, make sure you configure a regular update interval. You can do this by editing the `cron` job for Pi-hole:
34
+
35
+ ```
36
+ root@netdata~ # crontab -e
37
+ ```
38
+
39
+ This will open an editor. Look for the line containing the `pihole -g` command and adjust the schedule accordingly. For example, to update the blocklist daily, add the following line:
40
+
41
+ ```
42
+ 0 0 * * * /usr/local/bin/pihole -g
43
+ ```
44
+
45
+ Save the file and exit the editor to apply the changes.
46
+
47
+5. **Monitor the blocklist update status**
48
+
49
+ After performing the necessary troubleshooting steps, keep an eye on the `pihole_blocklist_last_update` alert to ensure that your blocklist file is updated as expected.
50
+
51
+### Useful resources
52
+
53
+1. [Pi-hole Blocklists](https://docs.pi-hole.net/database/gravity/)
54
+2. [Rebuilding the Blocklist](https://docs.pi-hole.net/ftldns/blockingmode/)
55
+3. [Pi-hole Documentation](https://docs.pi-hole.net/)
\ No newline at end of file
health/guides/pihole_status.md
new
+54
@@ -0,0 +1,54 @@
1
+### Understand the alert
2
+
3
+This alert monitors if Pi-hole's ability of blocking unwanted domains is active. If you receive this alert, it means that your Pi-hole's ad filtering is currently disabled.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the status of Pi-hole
8
+
9
+To check the current status of Pi-hole, run the following command:
10
+```
11
+pihole status
12
+```
13
+This command will show if Pi-hole is active or disabled.
14
+
15
+2. Re-enable Pi-hole
16
+
17
+If Pi-hole is disabled as per the status, you can re-enable it by running the following command:
18
+
19
+```
20
+pihole enable
21
+```
22
+
23
+3. Confirm Pi-hole is enabled
24
+
25
+After running the previous command, run `pihole status` again to confirm that Pi-hole is now enabled and blocking unwanted domains.
26
+
27
+4. Check for errors or warnings
28
+
29
+If Pi-hole is still not enabled, take a look at the logs for any errors or warnings:
30
+
31
+```
32
+cat /var/log/pihole.log | grep -i error
33
+cat /var/log/pihole.log | grep -i warning
34
+```
35
+
36
+5. Rebuild the blocklist
37
+
38
+If you still face issues, you can try rebuilding the blocklist by running:
39
+
40
+```
41
+pihole -g
42
+```
43
+
44
+6. Update Pi-hole
45
+
46
+If the problem persists, consider updating Pi-hole to the latest version:
47
+
48
+```
49
+pihole -up
50
+```
51
+
52
+### Useful resources
53
+
54
+1. [Pi-hole Official Documentation](https://docs.pi-hole.net/)
health/guides/ping/ping_host_latency.md
deleted
-50
@@ -1,50 +0,0 @@
1
-# fping_host_latency
2
-
3
-**Other | Network**
4
-
5
-`fping` is a command line tool to send ICMP (Internet Control Message Protocol) echo requests to
6
-network hosts, similar to ping, but performing much better when pinging multiple hosts. The Netdata
7
-Agent utilizes `fping` to monitor latency, packet loss, uptime and reachability of any number of
8
-network endpoints.
9
-
10
-For the `fping_host_latency` alert, the Netdata Agent monitors the average latency to the network
11
-host over the last 10 seconds. Receiving this alert indicates high latency to the network host. It is
12
-likely you are experiencing networking issues or the host is overloaded.
13
-
14
-### Troubleshooting section
15
-
16
-<details>
17
- <summary>Customize the ICMP requests for each endpoint</summary>
18
-
19
-Different endpoints could be in different networks. For example, a server in your intra network
20
-would require less time to be accessed than your cloud infrastructures in terms of latency. You
21
-should always consider not to use a global approach for checking every endpoint of yours. You can
22
-find more information about how to configure every endpoint separately in
23
-the [fping.plugin alarm guide](https://learn.netdata.cloud/docs/agent/collectors/fping.plugin/#additional-tips).
24
-
25
-</details>
26
-
27
-<details>
28
- <summary>Prioritize traffic on your endpoints</summary>
29
-
30
-Quality of service (QoS) is the use of mechanisms or technologies to control traffic and ensure the
31
-performance of critical applications. QoS works best when low-priority traffic exists that can be
32
-dropped when congestion occurs. The higher-priority traffic must fit within the bandwidth
33
-limitations of the link or path. The following are two open source solutions to apply QoS policies
34
-to your network interfaces.
35
-
36
-- `FireQOS`:
37
-
38
- FireQOS is a traffic shaping helper. It has a very simple shell scripting language to express
39
- traffic shaping.
40
-
41
- [See more on FireQOS](https://firehol.org/tutorial/fireqos-new-user/)
42
-
43
-- `tcconfig`:
44
-
45
- Tcconfig is a command wrapper that makes it easy to set up traffic control of network bandwidth,
46
- latency, packet-loss, packet-corruption, etc.
47
-
48
- [See more on tcconfig](https://tcconfig.readthedocs.io/en/latest/index.html)
49
-
50
-</details>
health/guides/ping/ping_host_reachable.md
deleted
-85
@@ -1,85 +0,0 @@
1
-# fping_host_reachable
2
-
3
-**Other | Network**
4
-
5
-`fping` is a command line tool to send ICMP (Internet Control Message Protocol) echo requests to
6
-network hosts, similar to ping, but performing much better when pinging multiple hosts. The Netdata
7
-Agent utilizes `fping` to monitor latency, packet loss, uptime and reachability of any number of
8
-network endpoints.
9
-
10
-The `fping_host_reachable` alert in the Netdata Agent checks the reachability of a network
11
-host (0: unreachable, 1: reachable). Receiving a critical alert indicates that your
12
-endpoints are unreachable. It is likely that the host is down or your system is experiencing
13
-networking issues.
14
-
15
-### Troubleshooting
16
-
17
-<details>
18
-<summary>Check network connectivity</summary>
19
-
20
-Verify that your system has access to the particular endpoint. Check for basic connectivity to known
21
-hosts from both your host and the endpoint.
22
-
23
-</details>
24
-
25
-<details>
26
-<summary>DNS settings</summary>
27
-
28
-If you are using DNS resolution to check your endpoint, you should always consider check your DNS
29
-settings. To troubleshoot this issue, verify that your DNS can resolve your endpoints.
30
-
31
-1. Check your current DNS (for example in linux you can use the host command):
32
-
33
- ```
34
- root@netdata # host -v <your_endpoint>
35
- ```
36
-
37
-2. If the HTTP endpoint is supposed to be public facing endpoint, try an alternative DNS (for
38
- example Cloudflare's DNS):
39
-
40
- ```
41
- root@netdata # host -v <your_endpoint> 1.1.1.1
42
- ```
43
-</details>
44
-
45
-<details>
46
-<summary>Verify access restrictions in the remote host</summary>
47
-
48
-If the remote host is a Linux-based machine and you have access to it, you can check the followings.
49
-
50
-**Check the ICMP settings**
51
-
52
-In most linux distributions you can restrict the ICMP echo operations.
53
-
54
- 1. Check your current setting.
55
-
56
- ```
57
- root@netdata # systemctl net.ipv4.icmp_echo_ignore_all
58
- ```
59
-
60
-
61
- If this value is set to 1 your system ignore incoming ICMP echo requests.
62
-
63
- 2. To change this, bump this `net.ipv4.icmp_echo_ignore_all=0` entry under `/etc/sysctl.conf`.
64
-
65
- 3. Reload the sysctl settings.
66
-
67
- ```
68
- root@netdata # sysctl -p
69
- ```
70
-
71
- **Check your firewall rules**
72
-
73
- Depending on what firewall you use, the commands might differ from what's shown below. For example, if you are using
74
- IP tables you can check for restriction rules upon `icmp`.
75
-
76
- ```
77
- root@netdata # iptables -L | grep ICMP
78
- ```
79
-
80
- For futher investigation or changes in your firewall settings we **strongly** advise you to consult
81
- your firewall's documentation and guidelines.
82
-
83
-
84
-</details>
85
-
health/guides/ping/ping_packet_loss.md
deleted
-40
@@ -1,40 +0,0 @@
1
-# fping_packet_loss
2
-
3
-## OS: Any
4
-
5
-`fping` is a command line tool to send ICMP (Internet Control Message Protocol) echo requests to
6
-network hosts, similar to ping, but performing much better when pinging multiple hosts. The Netdata
7
-Agent utilizes `fping` to monitor latency, packet loss, uptime and reachability of any number of
8
-network endpoints.
9
-
10
-For `fping_packet_loss`, the Netdata Agent calculates the packet loss ratio to a network host
11
-over the last 10 minutes. Receiving this alert indicates high packet loss towards a network host.
12
-This could be caused by link congestion, link node faults, high server load, or incorrect system
13
-settings.
14
-
15
-### Troubleshooting
16
-
17
-<details>
18
- <summary>Prioritize important traffic on your endpoint (linux based endpoints)</summary>
19
-
20
-Quality of service (QoS) is the use of mechanisms or technologies to control traffic and ensure the
21
-performance of critical applications. QoS works best when low-priority traffic exists that can be
22
-dropped when congestion occurs. The higher-priority traffic must fit within the bandwidth
23
-limitations of the link or path. The following are two open source solutions to apply QoS policies
24
-to your network interfaces.
25
-
26
-- `FireQOS`:
27
-
28
- FireQOS is a traffic shaping helper. It has a very simple shell scripting language to express
29
- traffic shaping.
30
-
31
- [See more on FireQOS](https://firehol.org/tutorial/fireqos-new-user/)
32
-
33
-- `tcconfig`:
34
-
35
- Tcconfig is a command wrapper that makes it easy to set up traffic control of network bandwidth,
36
- latency, packet-loss, packet-corruption, etc.
37
-
38
- [See more on tcconfig](https://tcconfig.readthedocs.io/en/latest/index.html)
39
-
40
-</details>
health/guides/ping_host_latency.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average latency (`ping round-trip time`) to a network host (${label:host}) over the last 10 seconds. If you receive this alert, it means there might be issues with your network connectivity or host responsiveness.
4
+
5
+### What does latency mean?
6
+
7
+Latency is the time it takes for a packet of data to travel from the sender to the receiver, and back from the receiver to the sender. In this case, we're measuring the latency using the `ping` command, which sends an ICMP echo request to the host and then waits for the ICMP echo reply.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Double-check the network connection:
12
+
13
+ Verify the network connectivity between your system and the target host. Check if the host is accessible via other tools such as `traceroute` or `mtr`.
14
+
15
+ ```
16
+ traceroute ${label:host}
17
+ mtr ${label:host}
18
+ ```
19
+
20
+2. Check for packet loss:
21
+
22
+ Packet loss can make latency appear higher than it actually is. Use the `ping` command to check for packet loss:
23
+
24
+ ```
25
+ ping -c 10 ${label:host}
26
+ ```
27
+
28
+ Look for the percentage of packet loss in the output.
29
+
30
+3. Investigate the host:
31
+
32
+ If no packet loss is detected and the network connection is stable, the problem might be related to the host itself. Check the host for overloaded resources, such as high CPU usage, disk I/O, or network traffic.
33
+
34
+4. Check DNS resolution:
35
+
36
+ If the alert's `${label:host}` is a domain name, make sure that DNS resolution is working properly:
37
+
38
+ ```
39
+ nslookup ${label:host}
40
+ ```
41
+
42
+5. Verify firewall and routing:
43
+
44
+ Check if any firewall rules or routing policies might be affecting the network traffic between your system and the target host.
45
+
46
+### Useful resources
47
+
48
+1. [Using Ping and Traceroute to troubleshoot network connectivity](https://support.cloudflare.com/hc/en-us/articles/200169336-Using-Ping-and-Traceroute-to-troubleshoot-network-connectivity)
health/guides/ping_host_reachable.md
new
+27
@@ -0,0 +1,27 @@
1
+### Understand the alert
2
+
3
+This `ping_host_reachable` alert checks the network reachability status of a specific host. When you receive this alert, it means that the host is either `up` (reachable) or `down` (unreachable).
4
+
5
+### What is network reachability?
6
+
7
+Network reachability refers to the ability of a particular host to communicate with other devices or systems within a network. In this alert, the reachability is monitored using the `ping` command, which sends packets to the host and checks for the response. The alert evaluates the packet loss percentage over a 30-second period.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Verify if the alert is accurate: Check if there are transient network issues or if there is a problem with the particular host. You can run the `ping` command manually to see if the packet loss percentage is consistent over time.
12
+
13
+ ```
14
+ ping -c 10 <host IP or domain>
15
+ ```
16
+
17
+2. Check the network connectivity: Ensure there are no issues with the local network or the physical connections (switches, routers, etc.). Look for potential network bottlenecks, high traffic, and hardware failures that can affect reachability.
18
+
19
+3. Check the host's health: If the host is reachable, log in to the system and examine its performance, stability, and resource usage. Look for indicators of high system load, resource constraints, or unresponsive processes.
20
+
21
+4. Examine network security policies and firewalls: Network reachability can be affected by misconfigured firewalls or security policies. Ensure there are no restrictions blocking the communication between the monitoring system and the host.
22
+
23
+5. Analyze logs for any relevant information: Check system logs (e.g., `/var/log/syslog`) and application logs on both the monitoring system and the target host. Look for error messages, timeouts, or connectivity problems.
24
+
25
+### Useful resources
26
+
27
+1. [Understanding High Packet Loss in Networking](https://www.fiberplex.com/blog/understanding-high-packet-loss-in-networking)
health/guides/ping_packet_loss.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+This alert calculates the `ping packet loss` percentage to the network host over the last 10 minutes. If you receive this alert, it means that your network is experiencing increased packet loss.
4
+
5
+### What does ping packet loss mean?
6
+
7
+Ping is a command used to test the reachability of a host on a network. It measures the round-trip-time (RTT) for packets sent from the source host to the destination host. Packet loss occurs when these packets are not successfully delivered to their destination.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check for network congestion:
12
+
13
+ Excessive network traffic can cause packet loss. Use tools like `iftop`, `nload`, or `bmon` to monitor your network bandwidth usage and identify possible congestion sources.
14
+
15
+2. Inspect the network hardware:
16
+
17
+ Faulty network hardware like routers, switches, and cables can lead to packet loss. Examine the physical network hardware for possible issues and ensure that all devices are functioning properly.
18
+
19
+3. Test the connection to the destination host:
20
+
21
+ Use the `ping` command to test the connection to the destination host:
22
+
23
+ ```
24
+ ping <destination_host>
25
+ ```
26
+
27
+ If you experience consistent packet loss, it may indicate an issue with the destination host or the network path leading to it.
28
+
29
+4. Check the destination host:
30
+
31
+ If the destination host is under heavy load or experiencing issues, it may cause packet loss. Check the host's resources, such as CPU usage, memory usage, and disk space, and resolve any issues if necessary.
32
+
33
+5. Investigate possible packet loss causes:
34
+
35
+ Some factors that can cause packet loss include network congestion, poor network equipment performance, corrupt data packets, or interference from other devices. Analyze your network traffic and pinpoint the cause of the packet loss.
36
+
37
+6. Rectify any identified issues:
38
+
39
+ Once you've identified the cause of the packet loss, take appropriate measures to resolve it. This may involve updating network hardware, optimizing network traffic, or fixing issues with the destination host.
40
+
41
+### Useful resources
42
+
43
+1. [How to Troubleshoot Packet Loss](https://www.lifewire.com/how-to-troubleshoot-packet-loss-on-your-network-4685249)
44
+2. [Diagnosing Network Issues with MTR](https://www.linode.com/community/questions/17967/diagnosing-network-issues-with-mtr)
health/guides/portcheck/portcheck_connection_fails.md
deleted
-31
@@ -1,31 +0,0 @@
1
-# portcheck_connection_fails
2
-
3
-**Other | TCP endpoint**
4
-
5
-The Netdata Agent calculates the average ratio of failed connections over the last 5 minutes. This
6
-alert indicates that too many connections failed. Receiving this alert means that your endpoint is
7
-unreachable due to:
8
-
9
-1. The service is no longer running or not working properly.
10
-2. Access to this port is denied by a firewall.
11
-3. Port forwarding rile is incorrectly configured
12
-4. The IP of the node you want to access is set to a private IP address
13
-
14
-This alert is triggered in warning state when the ratio of failed connections is between 10-40% and
15
-in critical state when it is greater than 40%.
16
-
17
-### Troubleshooting section
18
-
19
-<details>
20
-<summary>Check the firewall rules in the remote</summary>
21
-
22
-Check the INPUT chain rules, verify that you have allowed access from the host (Agent configured in it)
23
-to the remote node.
24
-
25
-**IPtables**
26
-
27
- ```
28
- root@netdata # iptables -L INPUT
29
- ```
30
-
31
-</details>
\ No newline at end of file
health/guides/portcheck/portcheck_connection_timeouts.md
deleted
-42
@@ -1,42 +0,0 @@
1
-# portcheck_connection_timeouts
2
-
3
-**Other | TCP endpoint**
4
-
5
-TCP provides a “virtual connection” between two nodes. When there is no activity, “keep alive”
6
-packet are exchanged on a regular interval. Should the “keep alive” not arrive after specified
7
-amount of time, the “connection times out” because there was not traffic during the timeout
8
-interval.
9
-
10
-The Netdata Agent calculates the average ratio of timeouts over the last 5 minutes. Receiving this
11
-alert means that the monitored endpoint is either unreachable or most likely you are experiencing
12
-networking issues or, the remote host/service is overloaded.
13
-
14
-This alert is triggered in warning state when the ratio of timeouts is between 10-40% and in
15
-critical state when it is greater than 40%.
16
-
17
-### Troubleshooting section
18
-
19
-<details>
20
-
21
- <summary>General approach</summary>
22
-
23
-You should try to use Wireshark to inspect the network packets in the remote
24
-
25
-Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting,
26
-analysis, software and communications protocol development.
27
-
28
-[See more about Wireshark here](https://www.wireshark.org/)
29
-
30
-Since you might won't be able to probe your traffic with wireshark in your host machine, You can
31
-export it in a dump file and analyze it in a second iteration.
32
-
33
-1. Try to export the traffic in your remote with `tcpdump`.
34
-
35
- ```
36
- root@netdata # tcpdump -i any 'port <PORT_YOU_MONITOR>' -s 65535 -w output.pcap
37
- ```
38
-
39
-You must stop the capture after a certain observation period (60s up to 5 minutes). This command
40
-will create a dump file which can be interpreted by Wireshark that contains all the traffic from any
41
-interface for a specific port.
42
-
health/guides/portcheck_connection_fails.md
new
+32
@@ -0,0 +1,32 @@
1
+### Understand the alert
2
+
3
+This alert indicates that too many connections are failing to a specific TCP endpoint in the last 5 minutes. It suggests that the monitored service on that endpoint is most likely down, unreachable, or access is being denied by firewall/security rules.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the service
8
+ Investigate if the service at the endpoint (specific IP and port) is running as expected. Inspect service logs for issues, error messages, or indications of a shutdown event.
9
+
10
+2. Test the endpoint
11
+ Try to establish a connection to the flagged endpoint using tools like `telnet`, `curl`, or `nc`. These tools provide real-time feedback that can help identify problems with the endpoint:
12
+
13
+ Example using `telnet`:
14
+ ```
15
+ telnet IP_ADDRESS PORT_NUMBER
16
+ ```
17
+
18
+3. Examine firewall and security group rules
19
+ Verify if there are any recent changes or newly added firewall/security group rules that might be causing the connectivity issues. Look for any rules that could be blocking the monitored port specifically or the IP range.
20
+
21
+4. Inspect network connectivity
22
+ Check the network connectivity between the Netdata Agent and the monitored endpoint. Ensure there are no intermittent network failures or high latency affecting the communication between the two.
23
+
24
+5. Examine the alert configuration
25
+ Validate the alert configuration in the `netdata.conf` file to confirm that the alert thresholds and monitored percentage of failed connections are set appropriately.
26
+
27
+6. Check resource utilization
28
+ High resource utilization might affect the availability of the monitored endpoint. Check if the system hosting the service has enough resources available (CPU, memory, and storage) to serve incoming requests.
29
+
30
+### Useful resources
31
+
32
+1. [How to use netcat (nc) command: Examples for network testing/debugging](https://www.nixcraft.com/t/how-to-use-netcat-nc-command-examples-for-network-testing-debugging/3332)
health/guides/portcheck_connection_timeouts.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+The `portcheck_connection_timeouts` alert calculates the average ratio of connection timeouts when trying to connect to a TCP endpoint over the last 5 minutes. If you receive this alert, it means that the monitored TCP endpoint is unreachable, potentially due to networking issues or an overloaded host/service.
4
+
5
+This alert triggers a warning state when the ratio of timeouts is between 10-40% and a critical state if the ratio is greater than 40%.
6
+
7
+### Troubleshoot the alert
8
+
9
+1. Check the network connectivity
10
+ - Use the `ping` command to check network connectivity between your system and the monitored TCP endpoint.
11
+ ```
12
+ ping <tcp_endpoint_ip>
13
+ ```
14
+ If the connectivity is intermittent or not established, it indicates network issues. Reach out to your network administrator for assistance.
15
+
16
+2. Check the status of the monitored TCP service
17
+ - Identify the service running on the monitored TCP endpoint by checking the port number.
18
+ - Use the `netstat` command to check the service status:
19
+
20
+ ```
21
+ netstat -tnlp | grep <port_number>
22
+ ```
23
+ If the service is not running or unresponsive, restart the service or investigate further into the application logs for any issues.
24
+
25
+3. Verify the load on the TCP endpoint host
26
+ - Connect to the host and analyze its resource consumption (CPU, memory, disk I/O, and network bandwidth) with tools like `top`, `vmstat`, `iostat`, and `iftop`.
27
+ - Identify resource-consuming processes or applications and apply corrective measures (kill/restart the process, allocate more resources, etc.).
28
+
29
+4. Examine the firewall rules and security groups
30
+ - Ensure that there are no blocking rules or security groups for your incoming connections to the TCP endpoint.
31
+ - If required, update the rules or create new allow rules for the required ports and IP addresses.
32
+
33
+5. Check the Netdata configuration
34
+ - Review the Netdata configuration file `/etc/netdata/netdata.conf` to ensure the `portcheck` plugin settings are correctly configured for monitoring the TCP endpoint.
35
+ - If necessary, update and restart the Netdata agent.
36
+
37
+### Useful resources
38
+
39
+1. [Netstat Command in Linux](https://www.tecmint.com/20-netstat-commands-for-linux-network-management/)
40
+2. [Iostat Command Usage and Examples](https://www.thomas-krenn.com/en/wiki/Iostat_command_usage_and_examples)
41
+3. [Iftop Guide](https://www.tecmint.com/iftop-linux-network-bandwidth-monitoring-tool/)
health/guides/portcheck_service_reachable.md
new
+32
@@ -0,0 +1,32 @@
1
+### Understand the alert
2
+
3
+This alert checks if a particular TCP service on a specified host and port is reachable. If the average percentage of successful checks within the last minute is below 75%, it triggers an alert indicating the TCP service is not functioning properly.
4
+
5
+### Troubleshoot the alert
6
+
7
+- Verify if the problem is network-related or service-related
8
+
9
+ 1. Check if the host and port are correct and the service is configured to listen on that specific port.
10
+
11
+ 2. Use `ping` or `traceroute` to diagnose the connectivity issues between your machine and the host.
12
+
13
+ 3. Use `telnet` or `nc` to check if the specific port on the host is reachable. For example, `telnet example.com port_number` or `nc example.com port_number`.
14
+
15
+ 4. Check the network configuration, firewall settings, and routing rules on both the local machine and the target host.
16
+
17
+- Check if the TCP service is running and functioning properly
18
+
19
+ 1. Check the service logs for any errors or issues that may prevent it from working correctly.
20
+
21
+ 2. Restart the service and monitor its behavior.
22
+
23
+ 3. Investigate if there are any recent changes in the service configuration or updates that may cause the issue.
24
+
25
+ 4. Monitor system resources such as CPU, memory, and disk usage to ensure they are not causing any performance bottlenecks.
26
+
27
+- Optimize the service configuration
28
+
29
+ 1. Review the service's performance-related configurations and fine-tune them, if necessary.
30
+
31
+ 2. Check if there are any optimizations or best practices that can be applied to boost the service performance and reliability.
32
+
health/guides/postgres_acquired_locks_utilization.md
new
+43
@@ -0,0 +1,43 @@
1
+### Understand the alert
2
+
3
+This alert monitors the average `acquired locks utilization` over the last minute in PostgreSQL databases. If you receive this alert, it means that the acquired locks utilization for your system is near or above the warning threshold (15% or 20%).
4
+
5
+### What are acquired locks?
6
+
7
+In PostgreSQL, a lock is a mechanism used to control access to shared resources, such as database tables or rows. When multiple users or tasks are working with the database, locks help coordinate their activities and prevent conflicts.
8
+
9
+Acquired locks utilization refers to the percentage of locks currently in use in the system, compared to the total number of locks available.
10
+
11
+### Troubleshoot the alert
12
+
13
+1. Identify the most lock-intensive queries:
14
+
15
+ You can use the following SQL query to get the list of most lock-intensive queries running on your PostgreSQL server:
16
+
17
+ ```
18
+ SELECT pid, locktype, mode, granted, client_addr, query_start, now() - query_start AS duration, query
19
+ FROM pg_locks l
20
+ JOIN pg_stat_activity a ON l.pid = a.pid
21
+ WHERE query != '<IDLE>'
22
+ ORDER BY duration DESC;
23
+ ```
24
+
25
+2. Analyze the problematic queries and look for ways to optimize them, such as:
26
+
27
+ a. Adding missing indexes for faster query execution.
28
+ b. Updating and optimizing query plans.
29
+ c. Adjusting lock types or lock levels, if possible.
30
+
31
+3. Check the overall health and performance of your PostgreSQL server:
32
+
33
+ a. Monitor the CPU, memory, and disk usage.
34
+ b. Consider configuring the autovacuum settings to maintain your database's health.
35
+
36
+4. Monitor database server logs for any errors or issues.
37
+
38
+5. If the problem persists, consider adjusting the warning threshold (`warn` option), or even increasing the available locks in the PostgreSQL configuration (`max_locks_per_transaction`).
39
+
40
+### Useful resources
41
+
42
+1. [PostgreSQL Locks Monitoring](https://www.postgresql.org/docs/current/monitoring-locks.html)
43
+2. [PostgreSQL Server Activity statistics](https://www.postgresql.org/docs/current/monitoring-stats.html)
health/guides/postgres_db_cache_io_ratio.md
new
+51
@@ -0,0 +1,51 @@
1
+### Understand the alert
2
+
3
+The `postgres_db_cache_io_ratio` alert is related to PostgreSQL databases and measures the `cache hit ratio` in the last minute. If you receive this alert, it means that your database server cache is not as efficient as it should be, and your system is frequently reading data from disk instead of cache, causing possible slow performance and higher I/O workload.
4
+
5
+### What does cache hit ratio mean?
6
+
7
+Cache hit ratio is an indicator of how frequently the data required for a query is found in the cache instead of reading it directly from disk. Higher cache hit ratios mean increased query performance and less disk I/O, which can greatly impact your database performance.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Determine if the cache hit ratio issue is affecting your overall database performance using `htop`:
12
+
13
+ ```
14
+ htop
15
+ ```
16
+
17
+ Check the `Load average` gauge, if it's in the safe zone (green), the cache hit ratio issue might not be affecting overall performance. If it's in the yellow or red zone, further troubleshooting is necessary.
18
+
19
+2. Check per-database cache hit ratio:
20
+
21
+ Run the following query to see cache hit ratios for each database:
22
+ ```
23
+ SELECT dbname, (block_cache_hit_kb / (block_cache_miss_read_kb + block_cache_hit_kb)) * 100 AS cache_hit_ratio
24
+ FROM (SELECT datname as dbname,
25
+ sum(blks_read * 8.0 / 1024) as block_cache_miss_read_kb,
26
+ sum(blks_hit * 8.0 / 1024) as block_cache_hit_kb
27
+ FROM pg_stat_database
28
+ GROUP BY datname) T;
29
+ ```
30
+
31
+ Analyze the results to determine which databases have a low cache hit ratio.
32
+
33
+3. Analyze PostgreSQL cache settings:
34
+
35
+ Check the cache settings in the `postgresql.conf` file. You may need to increase the `shared_buffers` parameter to allocate more memory for caching purposes, if there is available memory on the host.
36
+
37
+ For example, set increased shared_buffers value:
38
+ ```
39
+ shared_buffers = 2GB # Change the value according to your host's available memory.
40
+ ```
41
+
42
+ Restart the PostgreSQL service to apply the changes:
43
+ ```
44
+ sudo systemctl restart postgresql
45
+ ```
46
+
47
+ Monitor the cache hit ratio to determine if the changes improved performance. It might take some time for the changes to take effect, so be patient and monitor the cache hit ratio and overall system health over time.
48
+
49
+### Useful resources
50
+
51
+1. [Tuning Your PostgreSQL Server](https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server)
health/guides/postgres_db_deadlocks_rate.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+This alert calculates the number of deadlocks in your PostgreSQL database in the last minute. If you receive this alert, it means that the number of deadlocks has surpassed the warning threshold (10 deadlocks per minute by default).
4
+
5
+### What are deadlocks?
6
+
7
+In a PostgreSQL database, a deadlock occurs when two or more transactions are waiting for one another to release a lock, causing a cyclical dependency. As a result, none of these transactions can proceed, and the database server may be unable to process other requests.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Identify deadlock occurrences and problematic queries
12
+
13
+1. Check the PostgreSQL log for deadlock occurrence messages. You can typically find these logs in `/var/log/postgresql/` or `/pg_log/`.
14
+
15
+ Look for messages like: `DETAIL: Process 12345 waits for ShareLock on transaction 67890; blocked by process 98765.`
16
+
17
+2. To find the problematic queries, examine the log entries before the deadlock messages. Most often, these entries will contain the SQL queries that led to the deadlocks.
18
+
19
+- Analyze and optimize the problematic queries
20
+
21
+1. Analyze the execution plans of the problematic queries using the `EXPLAIN` command. This can help you identify which parts of the query are causing the deadlock.
22
+
23
+2. Optimize the queries by rewriting them or by adding appropriate indices to speed up the processing time.
24
+
25
+- Avoid long-running transactions
26
+
27
+1. Long-running transactions increase the chances of deadlocks. Monitor your database for long-running transactions and try to minimize their occurrence.
28
+
29
+2. Set sensible lock timeouts to avoid transactions waiting indefinitely for a lock.
30
+
31
+- Review your application logic
32
+
33
+1. Inspect your application code for any circular dependencies that could lead to deadlocks.
34
+
35
+2. Use advisory locks when possible to minimize lock contention in the database.
36
+
37
+### Useful resources
38
+
39
+1. [PostgreSQL: Deadlocks](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-DEADLOCKS)
health/guides/postgres_db_transactions_rollback_ratio.md
new
+55
@@ -0,0 +1,55 @@
1
+### Understand the alert
2
+
3
+This alert calculates the `PostgreSQL database transactions rollback ratio` for the last five minutes. If you receive this alert, it means that the percentage of `aborted transactions` in the specified PostgreSQL database is higher than the defined threshold.
4
+
5
+### What does transactions rollback ratio mean?
6
+
7
+In a PostgreSQL database, the transactions rollback ratio represents the proportion of aborted transactions (those that roll back) in relation to the total number of transactions processed. A high rollback ratio may indicate issues with the application logic, database performance or excessive `deadlocks` causing transactions to be aborted frequently.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the PostgreSQL logs for any error messages or unusual activities related to transactions that might help identify the cause of the high rollback ratio.
12
+
13
+ ```
14
+ vi /var/log/postgresql/postgresql.log
15
+ ```
16
+
17
+ Replace `/var/log/postgresql/postgresql.log` with the appropriate path to your PostgreSQL log file.
18
+
19
+2. Investigate recent database changes or application code modifications that might have led to the increased rollback ratio.
20
+
21
+3. Examine the PostgreSQL database table and index statistics to identify potential performance bottlenecks.
22
+
23
+ ```
24
+ SELECT relname, seq_scan, idx_scan, n_tup_ins, n_tup_upd, n_tup_del, n_tup_hot_upd, n_live_tup, n_dead_tup, last_vacuum, last_analyze
25
+ FROM pg_stat_all_tables
26
+ WHERE schemaname = 'your_schema_name';
27
+ ```
28
+
29
+ Replace `your_schema_name` with the appropriate schema name.
30
+
31
+4. Identify the most frequent queries that cause transaction rollbacks using pg_stat_statements view:
32
+
33
+ ```
34
+ SELECT substring(query, 1, 50) as short_query, calls, total_time, rows, 100.0 * shared_blks_hit/nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
35
+ FROM pg_stat_statements
36
+ WHERE calls > 50
37
+ ORDER BY (total_time / calls) DESC;
38
+ ```
39
+
40
+5. Investigate database locks and deadlocks using pg_locks:
41
+
42
+ ```
43
+ SELECT database, relation::regclass, mode, transactionid AS tid, virtualtransaction AS vtid, pid, granted
44
+ FROM pg_catalog.pg_locks;
45
+ ```
46
+
47
+6. Make necessary changes in the application logic or database configuration to resolve the issues causing a high rollback ratio. Consult a PostgreSQL expert, if needed.
48
+
49
+### Useful resources
50
+
51
+1. [Monitoring PostgreSQL - rollback ratio](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-STATS-VIEWS)
52
+2. [PostgreSQL: Database Indexes](https://www.postgresql.org/docs/current/indexes.html)
53
+3. [PostgreSQL: Deadlocks](https://www.postgresql.org/docs/current/explicit-locking.html#LOCK-BUILT-IN-DEADLOCK-AVOIDANCE)
54
+4. [PostgreSQL: Log files](https://www.postgresql.org/docs/current/runtime-config-logging.html)
55
+5. [PostgreSQL: pg_stat_statements module](https://www.postgresql.org/docs/current/pgstatstatements.html)
\ No newline at end of file
health/guides/postgres_index_bloat_size_perc.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+This alert monitors index bloat in a PostgreSQL database table. If you receive this alert, it indicates that the index is bloated and is taking up more disk space than necessary, which can lead to performance issues.
4
+
5
+### What does index bloat mean?
6
+
7
+In PostgreSQL, when a row is updated or deleted, the old row data remains in the index while the new data is added. Over time, this causes the index to grow in size (bloat), leading to increased disk usage and degraded query performance. This alert measures the bloat size percentage for each index in the specified database and table.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the bloated index in your PostgreSQL database, as mentioned in the alert's info field (e.g. `db [database] table [table] index [index]`).
12
+
13
+2. Rebuild the bloated index:
14
+
15
+ Use the `REINDEX` command to rebuild the bloated index. This will free up the space occupied by the old row data and help optimize query performance.
16
+
17
+ ```
18
+ REINDEX INDEX [index_name];
19
+ ```
20
+
21
+ **Note:** `REINDEX` might lock the table for the time it takes to rebuild the index, so plan to run this command during maintenance periods or during low database usage periods.
22
+
23
+3. Monitor the index bloat size after rebuilding:
24
+
25
+ After rebuilding the index, continue monitoring the index bloat size and performance to ensure the issue has been resolved.
26
+
27
+ You can use tools like [pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) (a built-in PostgreSQL extension) and pg_stat_indexes (user-defined database views that collect index-related statistics) to keep an eye on your database's performance and catch any bloat issues before they negatively impact your PostgreSQL setup.
28
+
29
+### Useful resources
30
+
31
+1. [PostgreSQL documentation: REINDEX](https://www.postgresql.org/docs/current/sql-reindex.html)
32
+2. [PostgreSQL documentation: pg_stat_statements](https://www.postgresql.org/docs/current/pgstatstatements.html)
33
+3. [PostgreSQL documentation: Routine Vacuuming](https://www.postgresql.org/docs/current/routine-vacuuming.html)
health/guides/postgres_table_bloat_size_perc.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+The `postgres_table_bloat_size_perc` alert measures the bloat size percentage in a PostgreSQL database table. If you receive this alert, it means that the bloat size in a particular table in your PostgreSQL database has crossed the warning or critical threshold.
4
+
5
+### What is bloat size?
6
+
7
+In PostgreSQL, bloat size refers to the wasted storage space caused by dead rows and unused space that accumulates in database tables over time. It is a result of frequent database operations (inserts, updates, and deletes), impacting database performance and storage footprint.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Investigate the bloat size and impacted table
12
+
13
+To get a detailed report on bloated tables in your PostgreSQL database, use the [`pgstattuple`](https://www.postgresql.org/docs/current/pgstattuple.html) extension. First, install the extension if it isn't already installed:
14
+
15
+ ```
16
+ CREATE EXTENSION pgstattuple;
17
+ ```
18
+
19
+Then, run the following query to find the bloated tables:
20
+
21
+ ```sql
22
+ SELECT
23
+ schemaname, tablename,
24
+ pg_size_pretty(bloat_size) AS bloat_size,
25
+ round(bloat_ratio::numeric, 2) AS bloat_ratio
26
+ FROM (
27
+ SELECT
28
+ schemaname, tablename,
29
+ bloat_size, table_size, (bloat_size / table_size) * 100 as bloat_ratio
30
+ FROM pgstattuple.schema_bloat
31
+ ) sub_query
32
+ WHERE bloat_ratio > 10
33
+ ORDER BY bloat_ratio DESC;
34
+ ```
35
+
36
+- Reclaim storage space
37
+
38
+Reducing the bloat size in PostgreSQL tables involves reclaiming wasted storage space. Here are two approaches:
39
+
40
+ 1. **VACUUM**: The `VACUUM` command helps clean up dead rows and compact the space used by the table. Use the following command to clean up the impacted table:
41
+
42
+ ```
43
+ VACUUM VERBOSE ANALYZE <schema_name>.<table_name>;
44
+ ```
45
+
46
+ 2. **REINDEX**: If the issue persists after using `VACUUM`, consider REINDEXing the table. This command rebuilds the table's indexes, which can improve query performance and reduce bloat. It can be more intrusive than `VACUUM`, be sure you understand its implications before running:
47
+
48
+ ```
49
+ REINDEX TABLE <schema_name>.<table_name>;
50
+ ```
51
+
52
+- Monitor the bloat size
53
+
54
+Continue monitoring the bloat size in your PostgreSQL tables by regularly checking the `postgres_table_bloat_size_perc` alert on Netdata.
55
+
56
+### Useful resources
57
+
58
+1. [How to monitor and fix Database bloats in PostgreSQL?](https://blog.netdata.cloud/postgresql-database-bloat/)
health/guides/postgres_table_cache_io_ratio.md
new
+32
@@ -0,0 +1,32 @@
1
+### Understand the alert
2
+
3
+This alert monitors the PostgreSQL table cache hit ratio, which is the percentage of database read requests that can be served from the cache without requiring I/O operations. If you receive this alert, it means your PostgreSQL table cache hit ratio is too low, indicating performance issues with the database.
4
+
5
+### What does PostgreSQL table cache hit ratio mean?
6
+
7
+The PostgreSQL table cache hit ratio is an important metric for analyzing the performance of a database. A high cache hit ratio means that most read requests are being served from the cache, reducing the need for disk I/O operations and improving overall database performance. On the other hand, a low cache hit ratio indicates that more I/O operations are required, which can lead to performance degradation.
8
+
9
+### Troubleshoot the alert
10
+
11
+To address the low cache hit ratio issue, follow these steps:
12
+
13
+1. Analyze database performance:
14
+
15
+Analyze the database performance to identify potential bottlenecks and areas for optimization. You can use PostgreSQL performance monitoring tools such as `pg_top`, `pg_stat_statements`, and `pg_stat_user_tables` to gather information about query execution, table access patterns, and other performance metrics.
16
+
17
+2. Optimize queries:
18
+
19
+Review and optimize complex or long-running SQL queries that may be causing performance issues. Utilize PostgreSQL features like `EXPLAIN` and `EXPLAIN ANALYZE` to analyze query execution plans and identify optimization opportunities. Indexing and query optimization can reduce I/O requirements and improve cache hit ratios.
20
+
21
+3. Increase shared_buffers:
22
+
23
+If you have a dedicated database server with sufficient memory, you can consider increasing the `shared_buffers` in your PostgreSQL configuration. This increases the amount of memory available to the PostgreSQL cache and can help improve cache hit ratios. Before making changes to the configuration, ensure that you analyze the existing memory usage patterns and leave enough free memory for other system processes and caching demands.
24
+
25
+4. Monitor cache hit ratios:
26
+
27
+Keep monitoring cache hit ratios after making changes to your configuration or optimization efforts. Depending on the results, you may need to adjust further settings, indexes, or queries to optimize database performance.
28
+
29
+### Useful resources
30
+
31
+1. [Tuning Your PostgreSQL Server](https://www.postgresql.org/docs/current/runtime-config-resource.html)
32
+2. [Performance Monitoring and Tuning in PostgreSQL](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/postgres#monitoring)
health/guides/postgres_table_index_cache_io_ratio.md
new
+45
@@ -0,0 +1,45 @@
1
+### Understand the alert
2
+
3
+This alert monitors the PostgreSQL table index cache hit ratio, specifically the average index cache hit ratio over the last minute, for a specific database and table. If you receive this alert, it means that your table index caching is not efficient and might result in slow database performance.
4
+
5
+### What does cache hit ratio mean?
6
+
7
+Cache hit ratio is the percentage of cache accesses to an existing item in the cache, compared to cache accesses to a non-existing item. A higher cache hit ratio means that your database entries are found in the cache more often, reducing the need to access the disk and consequently speeding up the execution times for database operations.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check cache configuration settings
12
+
13
+- `shared_buffers`: This parameter sets the amount of shared memory used for the buffer pool, which is the most common caching mechanism. You can check its current value by running the following query:
14
+
15
+ ```
16
+ SHOW shared_buffers;
17
+ ```
18
+
19
+- `effective_cache_size`: This parameter is used by the PostgreSQL query planner to estimate how much of the buffer pool data will be cached in the operating system's page cache. To check its current value, run:
20
+
21
+ ```
22
+ SHOW effective_cache_size;
23
+ ```
24
+
25
+2. Analyze the query workload
26
+
27
+- Queries using inefficient indexes or not using indexes properly might contribute to a higher cache miss ratio. To find the most expensive queries, you can run:
28
+
29
+ ```
30
+ SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;
31
+ ```
32
+
33
+- Check if your database is using proper indexes. You can create a missing index based on your query plan or modify existing indexes to cover more cases.
34
+
35
+3. Increase cache size
36
+
37
+- If the cache settings are low and disk I/O is high, you might need to increase the cache size. Remember that increasing the cache size may also impact system memory usage, so monitor the changes and adjust the settings accordingly.
38
+
39
+4. Optimize storage performance
40
+
41
+- Verify that the underlying storage system performs well by monitoring disk latency and throughput rates. If required, consider upgrading the disk subsystem or using faster disks.
42
+
43
+### Useful resources
44
+
45
+1. [PostgreSQL Performance Tuning Guide](https://www.cybertec-postgresql.com/en/postgresql-performance-tuning/)
health/guides/postgres_table_last_autoanalyze_time.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the time elapsed since a PostgreSQL table was last analyzed by the AutoVacuum daemon exceeds one week. AutoVacuum is responsible for recovering storage, optimizing the database, and updating statistics used by the PostgreSQL query planner. If you receive this alert, it indicates that one or more of your PostgreSQL tables have not been analyzed recently which may impact performance.
4
+
5
+### What is PostgreSQL table autoanalyze?
6
+
7
+In PostgreSQL, table autoanalyze is a process carried out by the AutoVacuum daemon. This process analyzes the table contents and gathers statistics for the query planner to help it make better decisions about optimizing your queries. Regular autoanalyze is crucial for maintaining good performance in your PostgreSQL database.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the current AutoVacuum settings: To verify if AutoVacuum is enabled and configured correctly in your PostgreSQL database, run the following SQL command:
12
+
13
+ ```sql
14
+ SHOW autovacuum;
15
+ ```
16
+
17
+ If it returns `on`, AutoVacuum is enabled. Otherwise, enable AutoVacuum by modifying the `postgresql.conf` file, and set `autovacuum = on`. Then, restart the PostgreSQL service.
18
+
19
+2. Analyze the table manually: If AutoVacuum is enabled but the table has not been analyzed recently, you can manually analyze the table by running the following SQL command:
20
+
21
+ ```sql
22
+ ANALYZE [VERBOSE] [schema_name.]table_name;
23
+ ```
24
+
25
+ Replace `[schema_name.]table_name` with the appropriate schema and table name. The optional `VERBOSE` keyword provides detailed information about the analyze process.
26
+
27
+3. Investigate any errors during autoanalyze: If AutoVacuum is enabled and running but you still receive this alert, check the PostgreSQL log files for any errors or issues related to the AutoVacuum process. Address any issues discovered in the logs.
28
+
29
+4. Monitor AutoVacuum activity: To get an overview of AutoVacuum activity, you can monitor the `pg_stat_progress_vacuum` view. Run the following SQL command to inspect the view:
30
+
31
+ ```sql
32
+ SELECT * FROM pg_stat_progress_vacuum;
33
+ ```
34
+
35
+ Analyze the results to determine if there are any inefficiencies or issues with the AutoVacuum settings.
36
+
37
+### Useful resources
38
+
39
+1. [PostgreSQL: AutoVacuum](https://www.postgresql.org/docs/current/routine-vacuuming.html)
40
+2. [PostgreSQL: Analyzing a Table](https://www.postgresql.org/docs/current/sql-analyze.html)
41
+3. [PostgreSQL: Monitoring AutoVacuum Progress](https://www.postgresql.org/docs/current/progress-reporting.html#VACUUM-PART)
\ No newline at end of file
health/guides/postgres_table_last_autovacuum_time.md
new
+50
@@ -0,0 +1,50 @@
1
+### Understand the alert
2
+
3
+This alert is related to the PostgreSQL database and checks the time since the last autovacuum operation occurred on a specific table. If you receive this alert, it means that the table has not been vacuumed by the autovacuum daemon for more than a week (7 days).
4
+
5
+### What is autovacuum in PostgreSQL?
6
+
7
+Autovacuum is a feature in PostgreSQL that automates the maintenance of the database by reclaiming storage, optimizing the performance of the database, and updating statistics. It operates on individual tables and performs the following tasks:
8
+
9
+1. Reclaims storage occupied by dead rows and updates the Free Space Map.
10
+2. Optimizes the performance by updating statistics and executing the `ANALYZE` command.
11
+3. Removes dead rows and updates the visibility map in order to reduce the need for vacuuming.
12
+
13
+### Troubleshoot the alert
14
+
15
+- Check the autovacuum status
16
+
17
+To check if the autovacuum daemon is running for the PostgreSQL instance, run the following SQL command:
18
+
19
+ ```
20
+ SHOW autovacuum;
21
+ ```
22
+
23
+If the result is "off", then the autovacuum is disabled for the PostgreSQL instance. You can enable it by modifying the `postgresql.conf` configuration file and setting `autovacuum = on`.
24
+
25
+- Verify table-specific autovacuum settings
26
+
27
+Sometimes, autovacuum settings might be altered for individual tables. To check the autovacuum settings for the specific table mentioned in the alert, run the following SQL command:
28
+
29
+ ```
30
+ SELECT relname, reloptions FROM pg_class JOIN pg_namespace ON pg_namespace.oid = pg_class.relnamespace WHERE relname = '<table_name>' AND nspname = '<schema_name>';
31
+ ```
32
+
33
+Look for any custom `autovacuum_*` settings in the `reloptions` column and adjust them accordingly to allow the autovacuum daemon to run on the table.
34
+
35
+- Monitor the PostgreSQL logs
36
+
37
+Inspect the PostgreSQL logs for any error messages or unusual behavior related to autovacuum. The log file location depends on your PostgreSQL installation and configuration.
38
+
39
+- Manually vacuum the table
40
+
41
+If the autovacuum daemon has not run for a long time on the table, you can manually vacuum the table to reclaim storage and update statistics. To perform a manual vacuum, run the following SQL command:
42
+
43
+ ```
44
+ VACUUM (VERBOSE, ANALYZE) <schema_name>.<table_name>;
45
+ ```
46
+
47
+### Useful resources
48
+
49
+1. [PostgreSQL: Autovacuum](https://www.postgresql.org/docs/current/runtime-config-autovacuum.html)
50
+2. [PostgreSQL: Routine Vacuuming](https://www.postgresql.org/docs/current/routine-vacuuming.html)
health/guides/postgres_table_toast_cache_io_ratio.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+This alert monitors the TOAST hit ratio (i.e., cached I/O efficiency) of a specific table in a PostgreSQL database. If the hit ratio is low, it indicates that the database is performing more disk I/O operations than needed for the table, which may cause performance issues.
4
+
5
+### What is TOAST?
6
+
7
+TOAST (The Oversized-Attribute Storage Technique) is a mechanism in PostgreSQL to efficiently store large data items. It allows you to store large values (such as text or binary data) in a separate table, improving the overall performance of the database.
8
+
9
+### What does the hit ratio mean?
10
+
11
+The hit ratio is the percentage of cache hits (successful reads from the cache) compared to total cache requests (hits + misses). A high hit ratio indicates that the data frequently needed is stored in the cache, resulting in fewer disk I/O operations and better performance.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Verify if the alert is accurate by checking the TOAST hit ratio in the affected PostgreSQL system. You can use the following query to retrieve the hit ratio of a specific table:
16
+
17
+ ```sql
18
+ SELECT CASE
19
+ WHEN blks_hit + blks_read = 0 THEN 0
20
+ ELSE 100 * blks_hit / (blks_hit + blks_read)
21
+ END as cache_hit_ratio
22
+ FROM pg_statio_user_tables
23
+ WHERE schemaname = 'your_schema' AND relname = 'your_table';
24
+ ```
25
+
26
+ Replace `your_schema` and `your_table` with the appropriate values.
27
+
28
+2. Examine the table's indexes, and consider creating new indexes to improve query performance. Be cautious when creating indexes, as too many can negatively impact performance.
29
+
30
+3. Analyze the table's read and write patterns to determine if you need to adjust the cache settings, such as increasing the `shared_buffers` configuration value.
31
+
32
+4. Inspect the application's queries to see if any can be optimized to improve performance. For example, use EXPLAIN ANALYZE to determine if the queries are using indexes effectively.
33
+
34
+5. Monitor overall PostgreSQL performance with tools like pg_stat_statements or pg_stat_activity to identify potential bottlenecks and areas for improvement.
35
+
36
+### Useful resources
37
+
38
+1. [PostgreSQL TOAST Overview](https://www.postgresql.org/docs/current/storage-toast.html)
39
+2. [Tuning Your PostgreSQL Server](https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server)
health/guides/postgres_table_toast_index_cache_io_ratio.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert monitors the `PostgreSQL` TOAST index cache hit ratio for a specific table in a database. A low hit ratio indicates a potential performance issue, as it means that a high number of cache misses are occurring. If you receive this alert, it suggests that your system is experiencing higher cache miss rates, which may lead to increased I/O load and reduced query performance.
4
+
5
+### What is TOAST?
6
+
7
+TOAST (The Oversized-Attribute Storage Technique) is a technique used by PostgreSQL to handle large data values. It allows PostgreSQL to store large records more efficiently by compressing and storing them separately from the main table. The TOAST index cache helps PostgreSQL efficiently access large data values, and a high cache hit ratio is desired for better performance.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check the current cache hit ratio
12
+
13
+ Run the following query in the PostgreSQL prompt to see the current hit ratio:
14
+
15
+ ```
16
+ SELECT schemaname, relname, toastidx_scan, toastidx_fetch, 100 * (1 - (toastidx_fetch / toastidx_scan)) as hit_ratio
17
+ FROM pg_stat_all_tables
18
+ WHERE toastidx_scan > 0 and relname='${label:table}' and schemaname='${label:database}';
19
+ ```
20
+
21
+- Investigate the workload on the database
22
+
23
+ Inspect the queries running on the database to determine if any specific queries are causing excessive cache misses. Use [`pg_stat_statements`](https://www.postgresql.org/docs/current/pgstatstatements.html) module to gather information on query performance.
24
+
25
+- Increase `work_mem` configuration value
26
+
27
+ If the issue persists, consider increasing the `work_mem` value in the PostgreSQL configuration file (`postgresql.conf`). This parameter determines the amount of memory PostgreSQL can use for internal sort operations and hash tables, which may help reduce cache misses.
28
+
29
+ Remember to restart the PostgreSQL server after making changes to the configuration file for the changes to take effect.
30
+
31
+- Optimize table structure
32
+
33
+ Assess if the table design can be optimized to reduce the number of large data values or if additional indexes can be created to improve cache hit ratio.
34
+
35
+- Monitor the effect of increased cache miss ratios
36
+
37
+ Keep an eye on overall database performance metrics, such as query execution times and I/O load, to determine the impact of increased cache miss ratios on database performance.
38
+
39
+### Useful resources
40
+
41
+1. [PostgreSQL: The TOAST Technique](https://www.postgresql.org/docs/current/storage-toast.html)
health/guides/postgres_total_connection_utilization.md
new
+45
@@ -0,0 +1,45 @@
1
+### Understand the alert
2
+
3
+This alert monitors the total `connection utilization` of a PostgreSQL database. If you receive this alert, it means that your `PostgreSQL` database is experiencing a high demand for connections. This can lead to performance degradation and, in extreme cases, could potentially prevent new connections from being established.
4
+
5
+### What does connection utilization mean?
6
+
7
+`Connection utilization` refers to the percentage of `database connections` currently in use compared to the maximum number of connections allowed by the PostgreSQL server. A high connection utilization implies that the server is handling a large number of concurrent connections, and its resources may be strained, leading to decreased performance.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the current connections to the PostgreSQL database:
12
+
13
+ You can use the following SQL query to check the number of active connections for each database:
14
+
15
+ ```
16
+ SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;
17
+ ```
18
+
19
+ or use the following command to check the total connections to all databases:
20
+
21
+ ```
22
+ SELECT count(*) FROM pg_stat_activity;
23
+ ```
24
+
25
+2. Identify the source of increased connections:
26
+
27
+ To find out which user or application is responsible for the high connection count, you can use the following SQL query:
28
+
29
+ ```
30
+ SELECT usename, application_name, count(*) FROM pg_stat_activity GROUP BY usename, application_name;
31
+ ```
32
+
33
+ This query shows the number of connections per user and application, which can help you identify the source of the increased connection demand.
34
+
35
+3. Optimize connection pooling:
36
+
37
+ If you are using an application server, such as `pgBouncer`, that supports connection pooling, consider adjusting the connection pool settings to better manage the available connections. This can help mitigate high connection utilization.
38
+
39
+4. Increase the maximum connections limit:
40
+
41
+ If your server has the necessary resources, you may consider increasing the maximum number of connections allowed by the PostgreSQL server. To do this, modify the `max_connections` configuration parameter in the `postgresql.conf` file and then restart the PostgreSQL service.
42
+
43
+### Useful resources
44
+
45
+1. [PostgreSQL: max_connections](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-MAX-CONNECTIONS)
health/guides/postgres_txid_exhaustion_perc.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+This alert monitors the percentage of transaction ID (TXID) exhaustion in a PostgreSQL database, specifically the rate at which the system is approaching a `TXID wraparound`. If the alert is triggered, it means that your PostgreSQL database is more than 90% towards exhausting its available transaction IDs, and you should take action to prevent transaction ID wraparound.
4
+
5
+### What is TXID wraparound?
6
+
7
+In PostgreSQL, transaction IDs are 32-bit integers, and a new one is assigned to each new transaction. Once the system has used all possible 32-bit integers for transaction IDs, it wraps back around to the beginning, reusing previous transaction IDs. This wraparound can lead to data loss or database unavailability if transactions' tuple visibility information becomes muddled.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the number of remaining transactions before wraparound. Connect to your PostgreSQL database, and run the following SQL query:
12
+
13
+ ```sql
14
+ SELECT datname, age(datfrozenxid) as age, current_limit FROM pg_database JOIN (SELECT setting AS current_limit FROM pg_settings WHERE name = 'autovacuum_vacuum_scale_factor') AS t1 ORDER BY age DESC;
15
+ ```
16
+
17
+2. Vacuum the database to prevent transaction ID wraparound. Run the following command:
18
+
19
+ ```
20
+ vacuumdb --all --freeze
21
+ ```
22
+
23
+ The command `vacuumdb` reclaims storage, optimizes the database for better performance, and prevents transaction ID wraparound.
24
+
25
+3. Configure Autovacuum settings for long-term prevention. Adjust `autovacuum_vacuum_scale_factor`, `autovacuum_analyze_scale_factor`, `vacuum_cost_limit`, and `maintenance_work_mem` in the PostgreSQL configuration file `postgresql.conf`. Then, restart the PostgreSQL service for the changes to take effect.
26
+
27
+ ```
28
+ service postgresql restart
29
+ ```
30
+
31
+### Useful resources
32
+
33
+1. [Preventing Transaction ID Wraparound Failures](https://www.postgresql.org/docs/current/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND)
health/guides/processes/active_processes.md
deleted
-138
@@ -1,138 +0,0 @@
1
-# active_processes
2
-
3
-## OS: Any
4
-
5
-This alert presents the percentage of used PIDs. If this alert gets raised it means that your system is experiencing
6
-high system process IDs (PID) space utilization. \
7
-If this value is 100% then the system can not start new processes.
8
-
9
-Note also that **zombie** processes could be responsible for high used PID percentage.
10
-
11
-<details>
12
-<summary>What are "PIDs"</summary>
13
-
14
-> A "PID" (i.e., process identification number) is an identification number that is automatically assigned to
15
-> each `process` when it is created on a Unix-like operating system.
16
-> A `process` is an executing (i.e., running) instance of a program. Each process is guaranteed a unique **PID**,
17
-> which is always a non-negative integer.<sup> [1](http://www.linfo.org/pid.html) </sup>
18
-
19
-As you can understand, a system has a finite number of PIDs, as it can run a limited number of processes at any given
20
-time.
21
-</details>
22
-
23
-<details>
24
-<summary>Zombie Processes</summary>
25
-
26
-> A **zombie** process is one that never received a signal from the parent process that created it, a child process is
27
-> one that has its origin in a higher level process known as the parent process that is in charge of sending the
28
-> signals to the child processes generated by it to indicate that their life span has ended. So, if the parent
29
-> process isn't programmed properly it can spawn **zombie** processes.<sup> [2](
30
-> https://blog.desdelinux.net/en/procesos-zombies/) </sup>
31
-
32
-So if the system has **a lot of zombie** processes, it **can't repurpose the PIDs used** on those processes.
33
-</details>
34
-
35
-> On Linux, the **max PID** is runtime configurable and by using `sysctl /kernel/pid_max` you can see that max value.
36
-
37
-> Furthermore, large numbers of processes are not optimal for reasons other than PID exhaustion. Each process on the
38
-> system has some fixed overhead in the form of some memory usage (ir reaches up to a few hundred bytes on Linux) and
39
-> scheduling overhead. Even if you are not exhausting available PIDs, you generally do not want any more processes
40
-> running than you absolutely need, *though this only truly matters in an HPC environment. (HPC stands for
41
-> "high-performance computing")*
42
-
43
-<br>
44
-
45
-<details>
46
-<summary>References and Sources</summary>
47
-
48
-1. [PID Definition](http://www.linfo.org/pid.html)
49
-2. [Zombie Processes](https://blog.desdelinux.net/en/procesos-zombies/)
50
-</details>
51
-
52
-### Troubleshooting Section
53
-
54
-<details>
55
-<summary>Terminate processes you don't need</summary>
56
-
57
-<details>
58
-<summary>Linux</summary>
59
-
60
-Use the `ps` command;
61
-
62
-```
63
-root@netdata~ # ps
64
-```
65
-
66
-it will display all the active `processes` in your system alongside with their `PIDs`.
67
-
68
-From there you can determine which processes you don't need and use the `kill` command to terminate them;
69
-
70
-```
71
-root@netdata~ # kill <the pid number you want to terminate>
72
-```
73
-
74
-</details>
75
-
76
-> It would be helpful to close any unneeded processes, but Netdata strongly suggests knowing exactly what
77
-> processes you are closing and being certain that they are not necessary.
78
-</details>
79
-
80
-<details>
81
-<summary>Check for zombie processes</summary>
82
-
83
-As we have seen, a system can report high percentage of used PIDs due to zombie processes (*among other things*). \
84
-You can't simply kill zombie processes. So to clean your system up from these processes, you should locate the parent of
85
-that process. Killing the parent would result in the process getting inherited by `pid 1`
86
-which will wait on it and eventually clear it from the process table.
87
-
88
-- Use `ps`;
89
-
90
-<details><summary>Freebsd and MacOS</summary>
91
-
92
-```
93
-root@netdata~ # ps -al | grep Z
94
-```
95
-
96
-</details>
97
-
98
-<details>
99
-<summary>Linux</summary>
100
-
101
-```
102
-root@netdata~ # ps -elf | grep Z
103
-```
104
-
105
-</details>
106
-
107
-This command will show you the processes currently at a zombie state.
108
-
109
-```
110
-F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
111
-...
112
-```
113
-
114
-`S` stands for `state`, on this column the **zombie** processes have a `Z`. \
115
-`PID` is the PID of the process \
116
-`PPID` is the PID of the parent of that process \
117
-`CMD` Should give you more info about the process and help you determine if it is needed or not
118
-
119
-After you have decided that you want to clean up a zombie process, you can;
120
-
121
-- `kill -s SIGCHLD <pid>` where `<pid>` is the `PPID` of the zombie process.
122
-
123
-This will send a signal to the parent process telling it to clean up its zombie processes. If the parent isn't
124
-programmed properly, it might be ignoring the `SIGCHILD` signal we would be sending. In this case, we would need to
125
-`kill` the parent, so the zombie can get inherited by `pid 1` which will wait on it and eventually clear it from the
126
-process table. \
127
-To do so:
128
-
129
-```
130
-root@netdata~ # kill <PID>
131
-```
132
-
133
-Where `<PID>` is the `PPID` *(parent process PID)* of the zombie process.
134
-
135
-> Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
136
-
137
-
138
-</details>
health/guides/python.d_job_last_collected_secs.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+The Netdata Agent also monitors itself, so this is an alert about the Netdata Python plugin. The Netdata Agent monitors the number of seconds since the last successful data collection for each of python plugin modules. This alert indicates that a specific module cannot reach the component it monitors to collect metrics from it.
4
+
5
+You can see all the modules that are orchestrated by the python.d.plugin in [our GitHub repo](https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin)
6
+
7
+### Troubleshoot the alert
8
+
9
+- Check the Netdata logs
10
+
11
+You need to identify why the Agent cannot collect metrics for a specific job. Inspect the Agent logs for this specific job.
12
+
13
+Host machine:
14
+
15
+ ```
16
+ tail -f /var/log/netdata/error.log | grep <job> OR <module_name>
17
+ ```
18
+
19
+Docker:
20
+
21
+ ```
22
+ docker logs <netdata_container> 2>&1 | grep <job> OR <module_name>
23
+ ```
24
+
25
+Kubernetes:
26
+ 1. Find the pod name of the node which produced the alert.
27
+
28
+ ```
29
+ kubectl -n <namespace> get pod -o wide -l app=netdata | grep <node_name>
30
+ ```
31
+ 2. Inspect it's logs
32
+
33
+ ```
34
+ kubectl logs -n <namespace> <pod_name> | grep <job> OR <module_name>
35
+ ```
health/guides/ram/oom_kill.md
deleted
-128
@@ -1,128 +0,0 @@
1
-# oom_kill
2
-
3
-## OS: Linux
4
-
5
-The OOM Killer (Out of Memory Killer) is a process that the Linux kernel uses when the system is
6
-critically low on memory or a process reached its memory limits. As the name suggests, it has the
7
-duty to review all running processes and kill one or more of them in order to free up memory and
8
-keep the system running.<sup>[1](https://neo4j.com/developer/kb/linux-out-of-memory-killer/) </sup>
9
-
10
-Linux Kernel 4.19 introduced cgroup awareness of OOM killer implementation which adds an ability to
11
-kill a cgroup as a single unit and to guarantee the integrity of the workload. In a nutshell,
12
-cgroups allow the limitation of memory, disk I/O, and network usage for a group of processes.
13
-Furthermore, cgroups may set usage quotas, and prioritize a process group to receive more CPU time
14
-or memory than other groups. You can see more about cgroups in
15
-the [cgroup man pages](https://man7.org/linux/man-pages/man7/cgroups.7.html)
16
-
17
-The Netdata Agent monitors the number of Out Of Memory (OOM) kills in the last 30 minutes. Receiving
18
-this alert indicates that some processes got killed by OOM Killer.
19
-
20
-<details>
21
-<summary>References and Sources</summary>
22
-
23
-1. [Linux Out of Memory Killer](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)
24
-2. [Memory Resource Controller in linux kernel](https://docs.kernel.org/admin-guide/cgroup-v1/memory.html?highlight=oom)
25
-3. [OOM killer blogspot](https://www.psce.com/en/blog/2012/05/31/mysql-oom-killer-and-everything-related/)
26
-
27
-</details>
28
-
29
-### Troubleshooting Section
30
-
31
-<details>
32
-<summary>Troubleshoot issues in the OOM killer</summary>
33
-
34
-The OOM Killer uses a heuristic system to choose a processes for termination. It is based on a score
35
-associated with each running application, which is calculated by `oom_badness()` call inside Linux
36
-kernel <sup>[3](https://www.psce.com/en/blog/2012/05/31/mysql-oom-killer-and-everything-related/) </sup>
37
-
38
-1. To identify which process/apps was killed from the OOM killer, inspect the logs:
39
-
40
-```
41
-root@netdata~ # dmesg -T | egrep -i 'killed process'
42
-```
43
-The system response looks similar to this:
44
-```
45
-Jan 7 07:12:33 mysql-server-01 kernel: Out of Memory: Killed process 3154 (mysqld).
46
-```
47
-
48
-2. To see the current `oom_score` (the priority in which OOM killer will act upon your processes) run the following script.
49
-The script prints all running processes (by pid and name) with likelihood to be killed by the OOM killer (second column).
50
-The greater the `oom_score` (second column) the more propably to be killed by OOM killer.
51
-
52
-```
53
-root@netdata~ # while read -r pid comm; do
54
- printf '%d\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$comm";
55
-done < <(ps -e -o pid= -o comm=) | sort -k 2n
56
-```
57
-
58
-3. Adjust the `oom_score` to protect processes using the `choom` util from
59
-the `util-linux` [package v2.33-rc1+](https://github.com/util-linux/util-linux/commit/8fa223daba1963c34cc828075ce6773ff01fafe3)
60
-
61
-```
62
-root@netdata~ # choom -p PID -n number
63
-```
64
-
65
-> Note: Setting an adjust score value of +500, for example, is roughly equivalent to allowing
66
-> the remainder of tasks sharing the same system, cpuset, mempolicy, or memory controller resources to
67
-> use at least 50% more memory. A value of -500, on the other hand, would be roughly equivalent to
68
-> discounting 50% of the task’s allowed memory from being considered as scoring against the task.
69
-
70
-4. Once the settings work to your case, make the change permanent. In the unit file of your service, under the [Service] section, add the following value: `OOMScoreAdjust=<PREFFERRED_VALUE>`
71
-
72
-</details>
73
-
74
-<details>
75
-<summary>Check the per-process RAM usage to find the top consumers</summary>
76
-
77
-1. To see which processes are the main RAM consumers, use `top utility`. The `%MEM` column displays RAM consumption in percent.
78
-
79
-```
80
-root@netdata~ # top -b -o +%MEM | head -n 22
81
-```
82
-
83
-2. Close any of the main consumer processes. Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
84
-</details>
85
-
86
-
87
-<details>
88
-<summary>Add a temporary swap file</summary>
89
-
90
-Keep in mind this requires creating a swap file in one of the disks. Performance of your system may
91
-be affected.
92
-
93
-1. Decide where your swapfile will live. It is strongly advised to allocate the swap file under in
94
- the root directory. A swap file is like an extension of your RAM and it should be protected, far
95
- from normal user accessible directories. Run the following command:
96
-
97
- ```
98
- root@netdata # dd if=/dev/zero of=<path_in_root> bs=1024 count=<size_in_bytes>
99
- ```
100
-
101
-2. Grant root only access to the swap file:
102
-
103
- ```
104
- root@netdata # chmod 600 <path_to_the_swap_file_you_created>
105
- ```
106
-
107
-3. Make it a Linux swap area:
108
-
109
- ```
110
- root@netdata # mkswap <path_to_the_swap_file_you_created>
111
- ```
112
-
113
-4. Enable the swap with the following command:
114
-
115
- ```
116
- root@netdata # swapon <path_to_the_swap_file_you_created>
117
- ```
118
-
119
-5. If you plan to use it a regular basis, you should update the `/etc/fstab` config. The entry you
120
- will add would look like:
121
-
122
- ```
123
- /swap_file swap sw 0 0
124
- ```
125
-
126
- For more information see the fstab manpage: `man fstab`.
127
-
128
-</details>
health/guides/ram/ram_available.md
deleted
-60
@@ -1,60 +0,0 @@
1
-# ram_available
2
-
3
-## OS: Linux, FreeBSD
4
-
5
-This alarm shows the percentage of an estimated amount of RAM that is available for use in userspace processes without causing
6
-swapping. If this alarm gets raised it means that your system has low amount of available RAM memory, and it may affect the
7
-performance of running applications.
8
-
9
-- If there is no `swap` space available, the OOM Killer can start killing processes.
10
-
11
-- When a system runs out of RAM memory, it can store its inactive content in another storage's partition (e.g. your
12
-main drive). The borrowed space is called `swap` or "swap space".
13
-
14
-- The OOM Killer (Out of Memory Killer) is a process that the Linux Kernel uses when the system is critically low on
15
-RAM. As the name suggests, it has the duty to review all running processes and kill one or more of them in order
16
-to free up RAM memory and keep the system running.<sup>[1](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)</sup>
17
-
18
-<br>
19
-
20
-<details>
21
-<summary>References and Sources</summary>
22
-
23
-[[1] Linux Out of Memory Killer](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)
24
-</details>
25
-
26
-### Troubleshooting section:
27
-
28
-<details>
29
-<summary>Check per-process RAM usage to find the top consumers</summary>
30
-
31
-<details>
32
-<summary>Linux</summary>
33
-
34
-Use `top`:
35
-
36
-```
37
-root@netdata~ # top -b -o +%MEM | head -n 22
38
-```
39
-
40
-here, you can see which processes are the main RAM consumers on the `%MEM` column (it is calculated in percentage).
41
-
42
-It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
43
-processes you are closing and being certain that they are not necessary.
44
-</details>
45
-
46
-<details>
47
-<summary>FreeBSD</summary>
48
-
49
-Use `top`:
50
-
51
-```
52
-root@netdata~ # top -b -o res | head -n 22
53
-```
54
-
55
-Here, you can see which processes are the main RAM consumers on the `RES` column (calculated in percentage).
56
-
57
-It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
58
-processes you are closing and being certain that they are not necessary.
59
-</details>
60
-</details>
health/guides/ram/ram_in_use.md
deleted
-51
@@ -1,51 +0,0 @@
1
-# ram_in_use
2
-
3
-## OS: Linux, FreeBSD
4
-
5
-This alert shows the percentage of used RAM. If you receive this alert, there is high RAM utilization on the node. Running
6
-low on RAM memory, means that the performance of running applications might be affected.
7
-
8
-If there is no `swap` space available, the OOM Killer can start killing processes.
9
-
10
-> When a system runs out of RAM, it can store it's inactive content in persistent storage (e.g. your
11
-> main drive). The borrowed space is called `swap` or "swap space".
12
-
13
-> The OOM Killer (Out of Memory Killer) is a process that the Linux Kernel uses when the system is critically low on
14
-> RAM. As the name suggests, it has the duty to review all running processes and kill one or more of them in order
15
-> to free up RAM memory and keep the system running.
16
-
17
-### Troubleshooting section:
18
-
19
-<details>
20
-<summary>Check per-process RAM usage to find the top consumers</summary>
21
-
22
-<details>
23
-<summary>Linux</summary>
24
-
25
-Use `top`:
26
-
27
-```
28
-root@netdata~ # top -b -o +%MEM | head -n 22
29
-```
30
-
31
-Here, you can see which processes are the main RAM consumers on the `%MEM` column (it is calculated in percentage).
32
-
33
-It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
34
-processes your are closing and being certain that they are not necessary.
35
-</details>
36
-
37
-<details>
38
-<summary>FreeBSD</summary>
39
-
40
-Use `top`:
41
-
42
-```
43
-root@netdata~ # top -b -o res | head -n 22
44
-```
45
-
46
-Here, you can see which processes are the main RAM consumers on the `RES` column (it is calculated in percentage).
47
-
48
-It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what
49
-processes your are closing and being certain that they are not necessary.
50
-</details>
51
-</details>
health/guides/ram_available.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alarm shows the percentage of an estimated amount of RAM that is available for use in userspace processes without causing swapping. If this alarm gets raised it means that your system has low amount of available RAM memory, and it may affect the performance of running applications.
4
+
5
+- If there is no `swap` space available, the OOM Killer can start killing processes.
6
+
7
+- When a system runs out of RAM memory, it can store its inactive content in another storage's partition (e.g. your
8
+main drive). The borrowed space is called `swap` or "swap space".
9
+
10
+- The OOM Killer (Out of Memory Killer) is a process that the Linux Kernel uses when the system is critically low on
11
+RAM. As the name suggests, it has the duty to review all running processes and kill one or more of them in order
12
+to free up RAM memory and keep the system running.<sup>[1](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)</sup>
13
+
14
+### Troubleshoot the alert
15
+
16
+- Check per-process RAM usage to find the top consumers
17
+
18
+Linux:
19
+```
20
+top -b -o +%MEM | head -n 22
21
+```
22
+FreeBSD:
23
+```
24
+top -b -o res | head -n 22
25
+```
26
+
27
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
28
+
29
+### Useful resources
30
+[Linux Out of Memory Killer](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)
health/guides/ram_in_use.md
new
+28
@@ -0,0 +1,28 @@
1
+### Understand the alert
2
+
3
+This alert shows the percentage of used RAM. If you receive this alert, there is high RAM utilization on the node. Running low on RAM memory, means that the performance of running applications might be affected.
4
+
5
+If there is no `swap` space available, the OOM Killer can start killing processes.
6
+
7
+When a system runs out of RAM, it can store it's inactive content in persistent storage (e.g. your main drive). The borrowed space is called `swap` or "swap space".
8
+
9
+The OOM Killer (Out of Memory Killer) is a process that the Linux Kernel uses when the system is critically low on RAM. As the name suggests, it has the duty to review all running processes and kill one or more of them in order
10
+to free up RAM memory and keep the system running.
11
+
12
+### Troubleshoot the alert
13
+
14
+- Check per-process RAM usage to find the top consumers
15
+
16
+Linux:
17
+```
18
+top -b -o +%MEM | head -n 22
19
+```
20
+FreeBSD:
21
+```
22
+top -b -o res | head -n 22
23
+```
24
+
25
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
26
+
27
+### Useful resources
28
+[Linux Out of Memory Killer](https://neo4j.com/developer/kb/linux-out-of-memory-killer/)
health/guides/redis_bgsave_broken.md
new
+23
@@ -0,0 +1,23 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the Redis server fails to save the RDB snapshot to disk. This can indicate issues with the disk, the Redis server itself, or other factors affecting the save operation.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check Redis logs**: Inspect the Redis logs to identify any error messages or issues related to the failed RDB save operation. You can typically find the logs in `/var/log/redis/redis-server.log`.
8
+
9
+2. **Verify disk space**: Ensure that your server has enough disk space available for the RDB snapshot. Insufficient disk space can cause the save operation to fail.
10
+
11
+3. **Check disk health**: Use disk health monitoring tools like `smartctl` to inspect the health of the disk where the RDB snapshot is being saved.
12
+
13
+4. **Review Redis configuration**: Check your Redis server's configuration file (`redis.conf`) for any misconfigurations or settings that may be causing the issue. Ensure that the `dir` and `dbfilename` options are correctly set.
14
+
15
+5. **Monitor server resources**: Monitor your server's resources, such as CPU and RAM usage, to ensure that they are not causing issues with the save operation.
16
+
17
+6. **Restart Redis**: If the issue persists, consider restarting the Redis server to clear any temporary issues or stuck processes.
18
+
19
+### Useful resources
20
+
21
+1. [Redis Configuration Documentation](https://redis.io/topics/config)
22
+2. [Redis Persistence Documentation](https://redis.io/topics/persistence)
23
+3. [Redis Troubleshooting Guide](https://redis.io/topics/problems)
health/guides/redis_bgsave_slow.md
new
+54
@@ -0,0 +1,54 @@
1
+### Understand the alert
2
+
3
+This alert, `redis_bgsave_slow`, indicates that the duration of the ongoing Redis RDB save operation is taking too long. This can be due to a large dataset size or a lack of CPU resources. As a result, Redis might stop serving clients for a few milliseconds, or even up to a second.
4
+
5
+### What is the Redis RDB save operation?
6
+
7
+Redis RDB (Redis Database) is a point-in-time snapshot of the dataset. It's a binary file that represents the dataset at the time of saving. The RDB save operation is the process of writing the dataset to disk, which occurs in the background.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the CPU usage
12
+
13
+Use the `top` command to see if the CPU usage is unusually high.
14
+
15
+```bash
16
+top
17
+```
18
+
19
+If the CPU usage is high, identify the processes that are consuming the most CPU resources and determine if they are necessary. Minimize the load by closing unnecessary processes.
20
+
21
+2. Analyze the dataset size
22
+
23
+Check the size of your Redis dataset using the `INFO` command:
24
+
25
+```bash
26
+redis-cli INFO | grep "used_memory_human"
27
+```
28
+
29
+If the dataset size is large, consider optimizing your data structure or implementing data management strategies, such as data expiration or partitioning.
30
+
31
+3. Monitor the Redis RDB save operation
32
+
33
+Use the following command to obtain the Redis statistics:
34
+
35
+```bash
36
+redis-cli INFO | grep "rdb_last_bgsave_time_sec"
37
+```
38
+
39
+Review the duration of the RDB save operation (rdb_last_bgsave_time_sec). If the save operation takes an unusually long time or fails frequently, consider optimizing your Redis configuration or improving your hardware resources like CPU and disk I/O.
40
+
41
+4. Change the save operation frequency
42
+
43
+To limit the frequency of RDB save operations, adjust the `save` configuration directive in your Redis configuration file (redis.conf). For example, to save the dataset only after 300 seconds (5 minutes) and at least 10000 changes:
44
+
45
+```
46
+save 300 10000
47
+```
48
+
49
+After modifying the configuration, restart the Redis service for the changes to take effect.
50
+
51
+### Useful resources
52
+
53
+1. [Redis Persistence](https://redis.io/topics/persistence)
54
+2. [Redis configuration](https://redis.io/topics/config)
health/guides/redis_connections_rejected.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+The `redis_connections_rejected` alert is triggered when the number of connections rejected by Redis due to the `maxclients` limit being reached in the last minute is greater than 0. This means that Redis is no longer able to accept new connections as it has reached its maximum allowed clients.
4
+
5
+### What does maxclients limit mean?
6
+
7
+The `maxclients` limit in Redis is the maximum number of clients that can be connected to the Redis instance at the same time. When the Redis server reaches its `maxclients` limit, any new connection attempts will be rejected.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the current number of connections in Redis:
12
+
13
+ Use the `redis-cli` command-line tool to check the current number of clients connected to the Redis server:
14
+
15
+ ```
16
+ redis-cli client list | wc -l
17
+ ```
18
+
19
+2. Check Redis configuration file for the maxclients setting:
20
+
21
+ The `maxclients` value can be found in the Redis configuration file, usually called `redis.conf`. Open the file and search for `maxclients` to find the current limit.
22
+
23
+ ```
24
+ grep 'maxclients' /etc/redis/redis.conf
25
+ ```
26
+
27
+3. Increase the maxclients limit.
28
+
29
+ If necessary, increase the `maxclients` limit in the Redis configuration file (`redis.conf`), and then restart the Redis service to apply the changes:
30
+
31
+ ```
32
+ sudo systemctl restart redis
33
+ ```
34
+
35
+ _**Note**: Keep in mind that increasing the `maxclients` limit might cause increased memory consumption._
36
+
37
+4. Inspect client connections.
38
+
39
+ Determine if the connections are legitimate and needed for your application's requirements, or if some clients are connecting unnecessarily. Optimize your application or services as needed to reduce the number of unwanted connections.
40
+
41
+5. Monitor connection usage.
42
+
43
+ Keep an eye on connection usage over time to better understand the trends and patterns in your system, and adjust the `maxclients` configuration accordingly.
44
+
45
+### Useful resources
46
+
47
+1. [Redis Clients documentation](https://redis.io/topics/clients)
48
+2. [Redis configuration documentation](https://redis.io/topics/config)
health/guides/redis_master_link_down.md
new
+50
@@ -0,0 +1,50 @@
1
+### Understand the alert
2
+
3
+The `redis_master_link_down` alert is triggered when there is a disconnection between a Redis master and its slave for more than 10 seconds. This alert indicates a potential problem with the replication process and can impact the data consistency across multiple instances.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the Redis logs
8
+
9
+ Examine the Redis logs for any errors or issues regarding the disconnection between the master and slave instances. By default, Redis log files are located at `/var/log/redis/redis.log`. Look for messages related to replication, network errors or timeouts.
10
+
11
+ ```
12
+ grep -i "replication" /var/log/redis/redis.log
13
+ grep -i "timeout" /var/log/redis/redis.log
14
+ ```
15
+
16
+2. Check the Redis replication status
17
+
18
+ Connect to the Redis master using the `redis-cli` tool, and execute the `INFO` command to get the detailed information about the master instance:
19
+
20
+ ```
21
+ redis-cli
22
+ INFO REPLICATION
23
+ ```
24
+
25
+ Also, check the replication status on the slave instance. If you have access to the IP address and port of the slave, connect to it and run the same `INFO` command.
26
+
27
+3. Verify the network connection between the master and slave instances
28
+
29
+ Test the network connectivity using `ping` and `telnet` or `nc` commands, ensuring that the connection between the master and slave instances is stable and there are no issues with firewalls or network policies.
30
+
31
+ ```
32
+ ping <slave_ip_address>
33
+ telnet <slave_ip_address> <redis_port>
34
+ ```
35
+
36
+4. Restart the Redis instances (if needed)
37
+
38
+ If Redis instances are experiencing issues or are unable to reconnect, consider restarting them. Be cautious as restarting instances might result in data loss or consistency issues.
39
+
40
+ ```
41
+ sudo systemctl restart redis
42
+ ```
43
+
44
+5. Monitor the situation
45
+
46
+ After addressing the potential issues, keep an eye on the Redis instances to ensure that the problem doesn't reoccur.
47
+
48
+### Useful resources
49
+
50
+1. [Redis Replication Documentation](https://redis.io/topics/replication)
health/guides/retroshare_dht_working.md
new
+32
@@ -0,0 +1,32 @@
1
+### Understand the alert
2
+
3
+The `retroshare_dht_working` alert is related to the Retroshare service, which is a secure communication and file sharing platform. Retroshare uses a Distributed Hash Table (DHT) to manage the network of connected users.
4
+
5
+If you receive this alert, it means that the number of DHT peers for your Retroshare service is low. This can lead to slow communication and file sharing, impacting the performance of the service.
6
+
7
+### Troubleshoot the alert
8
+
9
+1. Check the Retroshare service status
10
+
11
+Make sure that the Retroshare service is running and has an active connection to the internet. You can verify this by checking the service logs or by accessing the Retroshare interface.
12
+
13
+2. Inspect the network configuration
14
+
15
+Verify that your Retroshare service can connect to the required ports for DHT (UDP) to function correctly. Also, ensure the ports are open in any firewall or security software.
16
+
17
+3. Increase the number of bootstrap nodes
18
+
19
+Retroshare requires a list of bootstrap nodes for the initial connection to the DHT network. If the current bootstrap nodes are not sufficient or unresponsive, try adding more bootstrap nodes to the list.
20
+
21
+4. Update your Retroshare software
22
+
23
+Older versions of the Retroshare service may not connect correctly and might have outdated DHT peers list. Ensure your Retroshare service is up-to-date and working with the latest version.
24
+
25
+5. Check the Retroshare community
26
+
27
+If you continue to experience issues with the DHT peer count, visit the Retroshare community forums or support channels to see if other users have encountered similar issues and whether any solutions are suggested.
28
+
29
+### Useful resources
30
+
31
+1. [Retroshare Official Website](https://retroshare.cc/)
32
+2. [Retroshare GitHub Repository](https://github.com/RetroShare/RetroShare)
health/guides/riakkv_1h_kv_get_mean_latency.md
new
+52
@@ -0,0 +1,52 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average time between the reception of client `GET` requests and their subsequent responses in a `Riak KV` cluster over the last hour. If you receive this alert, it means that the average `GET` request latency in your Riak database has increased.
4
+
5
+### What does mean latency mean?
6
+
7
+Mean latency measures the average time taken between the start of a request and its completion, indicating the efficiency of the Riak system in processing `GET` requests. High mean latency implies slower processing times, which can negatively impact your application's performance.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check the system resources
12
+
13
+1. High latency might be related to resource bottlenecks on your Riak nodes. Check CPU, memory, and disk usage using `top` or `htop` tools.
14
+ ```
15
+ top
16
+ ```
17
+ or
18
+ ```
19
+ htop
20
+ ```
21
+
22
+2. If you find any resource constraint, consider scaling your Riak cluster or optimize resource usage by tuning the application configurations.
23
+
24
+- Investigate network issues
25
+
26
+1. Networking problems between the Riak nodes or the client and the nodes could cause increased latency. Check for network performance issues using `ping` or `traceroute`.
27
+
28
+ ```
29
+ ping node_ip_address
30
+ ```
31
+ or
32
+ ```
33
+ traceroute node_ip_address
34
+ ```
35
+
36
+2. Investigate any anomalies or network congestion and address them accordingly.
37
+
38
+- Analyze Riak KV configurations
39
+
40
+1. Check Riak configuration settings, like read/write parameters and anti-entropy settings, for any misconfigurations.
41
+
42
+2. Re-evaluate and optimize settings for performance based on your application requirements.
43
+
44
+- Monitor application performance
45
+
46
+1. Analyze your application's request patterns and workload. High request rates or large amounts of data being fetched can cause increased latency.
47
+
48
+2. Optimize your application workload to reduce latency and distribute requests uniformly across the Riak nodes.
49
+
50
+### Useful resources
51
+
52
+1. [Riak KV documentation](https://riak.com/posts/technical/official-riak-kv-documentation-2.2/)
health/guides/riakkv_1h_kv_put_mean_latency.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+The `riakkv_1h_kv_put_mean_latency` alert calculates the average time (in milliseconds) between the reception of client `PUT` requests and the subsequent responses to the clients over the last hour in a Riak KV database. If you receive this alert, it means that your Riak KV database is experiencing higher than normal latency in processing `PUT` requests.
4
+
5
+### What is Riak KV?
6
+
7
+Riak KV is a distributed NoSQL key-value data store designed to provide high availability, fault tolerance, operational simplicity, and scalability. The primary access method is through `PUT`, `GET`, `DELETE`, and `LIST` operations on keys.
8
+
9
+### What does `PUT` latency mean?
10
+
11
+`PUT` latency refers to the time it takes for the system to process a `PUT` request - from the moment the server receives the request until it sends a response back to the client. High `PUT` latency can impact the performance and responsiveness of applications relying on the Riak KV database.
12
+
13
+### Troubleshoot the alert
14
+
15
+- Check the Riak KV cluster health
16
+
17
+ Use the `riak-admin cluster status` command to get an overview of the Riak KV cluster's health. Make sure there are no unreachable or down nodes in the cluster.
18
+
19
+- Verify the Riak KV node performance
20
+
21
+ Use the `riak-admin status` command to display various statistics of the Riak KV nodes. Pay attention to the `node_put_fsm_time_mean` and `node_put_fsm_time_95` metrics, as they are related to `PUT` latency.
22
+
23
+- Inspect network conditions
24
+
25
+ Use networking tools (e.g., `ping`, `traceroute`, `mtr`, `iftop`) to check for potential network latency issues between clients and the Riak KV servers.
26
+
27
+- Evaluate the workload
28
+
29
+ If the client application is heavily write-intensive, consider optimizing it to reduce the number of write operations or increase the capacity of the Riak KV cluster to handle the load.
30
+
31
+- Review Riak KV logs
32
+
33
+ Examine the Riak KV logs (`/var/log/riak/riak_kv.log` by default) for any error messages or unusual patterns that might be related to the increased `PUT` latency.
34
+
35
+### Useful resources
36
+
37
+1. [Riak KV Official Documentation](https://riak.com/docs/)
health/guides/riakkv_kv_get_slow.md
new
+22
@@ -0,0 +1,22 @@
1
+### Understand the alert
2
+
3
+The `riakkv_kv_get_slow` alert is related to Riak KV, a distributed NoSQL key-value data store. This alert is triggered when the average processing time for GET requests significantly increases in the last 3 minutes compared to the average time over the last hour. If you receive this alert, it means that your Riak KV server is overloaded.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check Riak KV server load**: Investigate the current load on your Riak KV server. High CPU, memory, or disk usage can contribute to slow GET request processing times. Use monitoring tools like `top`, `htop`, `vmstat`, or `iotop` to identify any processes consuming excessive resources.
8
+
9
+2. **Analyze Riak KV logs**: Inspect the Riak KV logs for any error messages or warnings that could help identify the cause of the slow GET request processing times. The logs are typically located at `/var/log/riak` or `/var/log/riak_kv`. Look for messages related to timeouts, failures, or high latencies.
10
+
11
+3. **Monitor Riak KV metrics**: Check Riak KV metrics, such as read or write latencies, vnode operations, and disk usage, to identify possible bottlenecks contributing to the slow GET request processing times. Use tools like `riak-admin` or the Riak HTTP API to access these metrics.
12
+
13
+4. **Optimize query performance**: Analyze your application's Riak KV queries to identify any inefficient GET requests that could be contributing to slow processing times. Consider implementing caching mechanisms or adjusting Riak KV settings to improve query performance.
14
+
15
+5. **Evaluate hardware resources**: Ensure that your hardware resources are sufficient to handle the current load on your Riak KV server. If your server has insufficient resources, consider upgrading your hardware or adding additional nodes to your Riak KV cluster.
16
+
17
+### Useful resources
18
+
19
+1. [Riak KV documentation](https://riak.com/documentation/)
20
+2. [Monitoring Riak KV with Netdata](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/riakkv/)
21
+3. [Riak Control: Monitoring and Administration Interface](https://docs.riak.com/riak/kv/2.2.3/configuring/reference/riak-vars/#riak-control)
22
+4. [Riak KV Monitoring and Metrics](https://docs.riak.com/riak/kv/2.2.3/using/performance/monitoring/index.html)
health/guides/riakkv_kv_put_slow.md
new
+43
@@ -0,0 +1,43 @@
1
+### Understand the alert
2
+
3
+The `riakkv_kv_put_slow` alert is triggered when the average processing time for PUT requests in Riak KV database increases significantly in comparison to the last hour's average, suggesting that the server is overloaded.
4
+
5
+### What does server overloaded mean?
6
+
7
+An overloaded server means that the server is unable to handle the incoming requests efficiently, leading to increased processing times and degraded performance. Sometimes, it might result in request timeouts or even crashes.
8
+
9
+### Troubleshoot the alert
10
+
11
+To troubleshoot this alert, follow the below steps:
12
+
13
+1. **Check current Riak KV performance**
14
+
15
+ Use `riak-admin` tool's `status` command to check the current performance of the Riak KV node:
16
+
17
+ ```
18
+ riak-admin status
19
+ ```
20
+
21
+ Look for the following key performance indicators (KPIs) for PUT requests:
22
+ - riak_kv.put_fsm.time.95 (95th percentile processing time for PUT requests)
23
+ - riak_kv.put_fsm.time.99 (99th percentile processing time for PUT requests)
24
+ - riak_kv.put_fsm.time.100 (Maximum processing time for PUT requests)
25
+
26
+ If any of these values are significantly higher than their historical values, it may indicate an issue with the node's performance.
27
+
28
+2. **Identify high-load operations**
29
+
30
+ Examine the application logs or Riak KV logs for recent activity such as high volume of PUT requests, bulk updates or deletions, or other intensive database operations that could potentially cause the slowdown.
31
+
32
+3. **Investigate other system performance indicators**
33
+
34
+ Check the server's CPU, memory, and disk I/O usage to identify any resource constraints that could be affecting the performance of the Riak KV node.
35
+
36
+4. **Review Riak KV configuration**
37
+
38
+ Analyze the Riak KV configuration settings to ensure that they are optimized for your specific use case. Improperly configured settings can lead to performance issues.
39
+
40
+5. **Consider scaling the Riak KV cluster**
41
+
42
+ If the current Riak KV cluster is not able to handle the increasing workload, consider adding new nodes to the cluster to distribute the load and improve performance.
43
+
health/guides/riakkv_list_keys_active.md
new
+31
@@ -0,0 +1,31 @@
1
+### Understand the alert
2
+
3
+This alert indicates that currently there are active `list keys` operations in Finite State Machines (FSM) on your Riak KV database. Running `list keys` in Riak is a resource-intensive operation and can significantly affect the performance of the cluster, and it is not recommended for production use.
4
+
5
+### What are list keys operations in Riak?
6
+
7
+`List keys` operations in Riak involve iterating through all keys in a bucket to return a list of keys. The reason this is expensive in terms of resources is that Riak needs to traverse the entire dataset to generate a list of keys. As the dataset grows, the operation consumes more resources and takes longer to process the list, which can lead to reduced performance and scalability.
8
+
9
+### Troubleshoot the alert
10
+
11
+To address the `riakkv_list_keys_active` alert, follow these steps:
12
+
13
+1. Identify the processes and applications running `list keys` operations:
14
+
15
+ Monitor your application logs and identify the processes or applications that are using these operations. You may need to enable additional logging to capture information related to `list keys`.
16
+
17
+2. Evaluate the necessity of `list keys` operations:
18
+
19
+ Work with your development team and determine if there's a specific reason these operations are being used. If they are not necessary, consider replacing them with other, more efficient data retrieval techniques.
20
+
21
+3. Optimize data retrieval:
22
+
23
+ If it is necessary to retrieve keys in your application, consider using an alternative strategy such as Secondary Indexes (2i) or implementing a custom solution tailored to your specific use case.
24
+
25
+4. Monitor the system:
26
+
27
+ After making changes to your application, continue monitoring the active list key FSMs using Netdata to ensure that the number of active list keys operations is reduced.
28
+
29
+### Useful resources
30
+
31
+1. [Riak KV Operations](https://docs.riak.com/riak/kv/latest/developing/usage/operations/index.html)
health/guides/riakkv_vm_high_process_count.md
new
+31
@@ -0,0 +1,31 @@
1
+### Understand the alert
2
+
3
+The `riakkv_vm_high_process_count` alert is related to the Riak KV database. It warns you when the number of processes running in the Erlang VM is high. High process counts can result in performance degradation due to scheduling overhead.
4
+
5
+This alert is triggered in the warning state when the number of processes is greater than 10,000 and in the critical state when it is greater than 100,000.
6
+
7
+### Troubleshoot the alert
8
+
9
+1. Check the current number of processes in the Erlang VM. You can use the following command to see the active processes:
10
+
11
+ ```
12
+ riak-admin status | grep vnode_management_procs
13
+ ```
14
+
15
+2. Check the Riak KV logs (/var/log/riak) to see if there are any error messages or stack traces. This can help you identify issues and potential bottlenecks in your system.
16
+
17
+3. Check the CPU, memory, and disk space usage on the system hosting the Riak KV database. High usage in any of these areas can also contribute to performance issues and the high process count. Use commands like `top`, `free`, and `df` to monitor these resources.
18
+
19
+4. Review your Riak KV configuration settings. You may need to adjust the `+P` and `+S` flags, which control the maximum number of processes and scheduler threads (respectively) that the Erlang runtime system can create. These settings can be found in the `vm.args` file.
20
+
21
+ ```
22
+ vim /etc/riak/vm.args
23
+ ```
24
+
25
+5. If needed, optimize the Riak KV database by adjusting the configuration settings or by adding more resources to your system, such as RAM or CPU cores.
26
+
27
+6. Ensure that your application is not creating an excessive number of processes. You may need to examine your code and see if there are any ways to reduce the Riak KV process count.
28
+
29
+### Useful resources
30
+
31
+1. [Riak KV Documentation](http://docs.basho.com/riak/kv/2.2.3/)
health/guides/scaleio_sdc_mdm_connection_state.md
new
+43
@@ -0,0 +1,43 @@
1
+### Understand the alert
2
+
3
+The `scaleio_sdc_mdm_connection_state` alert indicates that your ScaleIO Data Client (SDC) is disconnected from the ScaleIO MetaData Manager (MDM). This disconnection can lead to potential performance issues or data unavailability in your storage infrastructure.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the connectivity between SDC and MDM nodes.
8
+
9
+Verify that the SDC and MDM nodes are reachable by performing a `ping` or using `traceroute` from the SDC node to the MDM node and vice versa. Network connectivity issues such as high latency or packet loss may cause the disconnection between SDC and MDM.
10
+
11
+2. Examine log files.
12
+
13
+Review the SDC and MDM log files to identify any error messages or warnings that can indicate the reason for the disconnection. Common log file locations are:
14
+
15
+ - SDC logs: `/opt/emc/scaleio/sdc/logs/sdc.log`
16
+ - MDM logs: `/opt/emc/scaleio/mdm/logs/mdm.log`
17
+
18
+3. Check the status of ScaleIO services.
19
+
20
+Verify that the ScaleIO services are running on both the SDC and MDM nodes. You can check the service status with the following commands:
21
+
22
+ - SDC service status: `sudo systemctl status scaleio-sdc`
23
+ - MDM service status: `sudo systemctl status scaleio-mdm`
24
+
25
+If any of the services are not running, start them and check the connection state again.
26
+
27
+4. Reconnect SDC to MDM.
28
+
29
+If the issue still persists after verifying the network connectivity and services' statuses, try to reconnect the SDC to MDM manually. Use the following command on the SDC node:
30
+
31
+ ```
32
+ sudo scli --reconnect_sdc --mdm_ip <MDM_IP_ADDRESS>
33
+ ```
34
+
35
+Replace `<MDM_IP_ADDRESS>` with the IP address of your MDM node.
36
+
37
+5. Contact support.
38
+
39
+If the disconnection issue persists after trying the above steps, consider contacting technical support for assistance.
40
+
41
+### Useful resources
42
+
43
+1. [ScaleIO Troubleshooting](https://www.dell.com/support/home/en-us/product-support/product/scaleio)
health/guides/scaleio_storage_pool_capacity_utilization.md
new
+34
@@ -0,0 +1,34 @@
1
+### Understand the alert
2
+
3
+The `scaleio_storage_pool_capacity_utilization` alert is related to storage capacity in ScaleIO, a software-defined storage solution. If you receive this alert, it means that the storage pool capacity utilization is high, potentially leading to performance issues or running out of space.
4
+
5
+### What does high storage pool capacity utilization mean?
6
+
7
+High storage pool capacity utilization means that the allocated storage space in the ScaleIO storage pool is being used at a high percentage. Warning and critical alerts are triggered at 80-90% and 90-98% utilization, respectively. When the storage pool capacity utilization is high, it may impact the performance of the system and may prevent new data from being stored, as available space is limited.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. **Verify the storage pool capacity utilization**
12
+
13
+ Check the Netdata dashboard or use Netdata API to verify the storage pool capacity utilization. Take note of the storage pools with high utilization.
14
+
15
+2. **Investigate storage usage**
16
+
17
+ Inspect the storage usage in your environment, and determine which data or applications are consuming the most space. You can use tools like `du`, `df`, and `ncdu` to analyze disk usage.
18
+
19
+3. **Delete or move unnecessary files**
20
+
21
+ If you found any unnecessary files or backup copies occupying large amounts of space, consider deleting them or moving them to different storage devices to free up space in the storage pool.
22
+
23
+4. **Optimize storage provisioning**
24
+
25
+ Evaluate the storage provisioning for your applications, and ensure that appropriate storage space is allocated based on the actual needs. Adjust storage allocations if needed.
26
+
27
+5. **Consider expanding the storage pool**
28
+
29
+ If the high storage pool capacity utilization is expected based on your application and data storage needs, consider expanding the storage pool by adding new devices or increasing the allocated storage space on the existing devices in the pool.
30
+
31
+6. **Monitor storage pool capacity utilization trends**
32
+
33
+ Keep track of the storage pool capacity utilization trends and be proactive in addressing potential storage capacity issues in the future.
34
+
health/guides/semaphore_arrays_used.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+This alarm monitors the percentage of used `System V IPC semaphore arrays (sets)`. If you receive this alert, it means that your system has a high utilization of `IPC semaphore arrays`, which can affect application performance.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the current usage of semaphore arrays
8
+
9
+ Use the `ipcs -u` command to display a summary of the current usage of semaphore arrays on your system. Look for the "allocated semaphores" section, which indicates the number of semaphore arrays being used.
10
+
11
+ ```
12
+ ipcs -u
13
+ ```
14
+
15
+2. Identify processes using semaphore arrays
16
+
17
+ Use the `ipcs -s` command to list all active semaphore arrays and their associated process IDs (PIDs). This information can help you identify which processes are using semaphore arrays.
18
+
19
+ ```
20
+ ipcs -s
21
+ ```
22
+
23
+3. Investigate and optimize processes using semaphore arrays
24
+
25
+ Based on the information from the previous step, investigate the processes that are using semaphore arrays. If any of these processes can be optimized or terminated to free up semaphore arrays, do so carefully after ensuring that they are not critical to your system.
26
+
27
+4. Adjust the semaphore limit on your system
28
+
29
+ If the semaphore array usage is still high after optimizing processes, you may need to increase the semaphore limit on your system. As mentioned earlier, you can adjust the limit in the `/proc/sys/kernel/sem` file.
30
+
31
+ ```
32
+ vi /proc/sys/kernel/sem
33
+ ```
34
+
35
+ Edit the fourth field to increase the max semaphores limit. Save the file and exit. To apply the changes, run:
36
+
37
+ ```
38
+ sysctl -p
39
+ ```
40
+
41
+ Please note that increasing the limit might consume more system resources. Monitor your system closely to ensure that it remains stable after making these changes.
42
+
43
+### Useful resources
44
+
45
+1. [Interprocess Communication](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html)
46
+2. [IPC: Semaphores](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html)
health/guides/semaphores_used.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+This alert monitors the percentage of allocated `System V IPC semaphores`. If you receive this alert, it means that your system is experiencing high IPC semaphore utilization, and a lack of available semaphores can affect application performance.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify processes using IPC semaphores
8
+
9
+ You can use the `ipcs` command to display information about allocated semaphores. Run the following command to display a list of active semaphores:
10
+
11
+ ```
12
+ ipcs -s
13
+ ```
14
+
15
+ The output will show the key, ID, owner's UID, permissions, and other related information for each semaphore.
16
+
17
+2. Analyze process usage of IPC semaphores
18
+
19
+ You can use `ps` or `top` commands to analyze which processes are using the IPC semaphores. This can help you identify if any process is causing high semaphore usage.
20
+
21
+ ```
22
+ ps -eo pid,cmd | grep [process-name]
23
+ ```
24
+
25
+ Replace `[process-name]` with the name of the process you suspect is related to the semaphore usage.
26
+
27
+3. Adjust semaphore limits if necessary
28
+
29
+ If you determine that the high semaphore usage is a result of an inadequately configured limit, you can update the limits using the following steps:
30
+
31
+ - Check the current semaphore limits as mentioned earlier, using the `ipcs -ls` command.
32
+ - To increase the limit to a more appropriate value, edit the `/proc/sys/kernel/sem` file. The second field in the file represents the maximum number of semaphores that can be allocated per array.
33
+
34
+ ```
35
+ echo "32000 64000 1024000000 500" > /proc/sys/kernel/sem
36
+ ```
37
+
38
+ This command doubles the number of semaphores per array. Make sure to adjust the value according to your system requirements.
39
+
40
+4. Monitor semaphore usage after changes
41
+
42
+ After making the necessary changes, continue to monitor semaphore usage to ensure that the changes were effective in resolving the issue. If the issue persists, further investigation may be required to identify the root cause.
43
+
44
+### Useful resources
45
+
46
+1. [Interprocess Communication](https://docs.oracle.com/cd/E19455-01/806-4750/6jdqdfltn/index.html)
47
+2. [IPC: Semaphores](https://users.cs.cf.ac.uk/Dave.Marshall/C/node26.html)
48
+3. [Linux Kernel Documentation - IPC Semaphores](https://www.kernel.org/doc/Documentation/ipc/semaphore.txt)
\ No newline at end of file
health/guides/softnet/10min_netisr_backlog_exceeded.md
deleted
-54
@@ -1,54 +0,0 @@
1
-# 10min_netisr_backlog_exceeded
2
-
3
-## OS: FreeBSD
4
-
5
-The netisr_maxqlen is a queue within the network kernel dispatch service of FreeBSD kernel which keeps packets
6
-received by interfaces and not yet processed by destined subsystems or userland applications. The system drops new packets
7
-when the queue is full. There may be several netisr packet queues in the system and raising netisr_maxqlen
8
-allows all of them to grow. The default netisr_maxqlen value should be 256 in most of the FreeBSD versions.
9
- However this may not be enough in some cases, such as:
10
-
11
-- Multiple interfaces operating at 1Gbps, or even a single interface at 10Gbps.
12
-
13
-- Lower powered systems process very large amounts of network traffic.
14
-
15
-Netdata agent monitors the average number of dropped packets in the last minute due to exceeded netisr queue length.
16
-
17
-### Troubleshooting section:
18
-
19
- <details>
20
- <summary>Increase the netisr_maxqlen value.</summary>
21
-
22
-1. Check your current value.
23
-
24
- ```
25
- root@netdata~ # sysctl net.route.netisr_maxqlen
26
- net.route.netisr_maxqlen: 256
27
- ```
28
-
29
-2. Try to increase it by a factor of 4.
30
-
31
- ```
32
- root@netdata~ # sysctl -w net.route.netisr_maxqlen=1024
33
- ```
34
-
35
-3. Verify the change and test with the same workload that triggered the alarm originally.
36
-
37
- ```
38
- root@netdata~ # sysctl net.route.netisr_maxqlen
39
- net.route.netisr_maxqlen: 1024
40
- ```
41
-
42
-4. If this change works for your system, you could make it permanently.
43
-
44
- Bump this `net.route.netisr_maxqlen=1024` entry under `/etc/sysctl.conf`
45
-
46
-
47
-5. Reload the sysctl settings.
48
-
49
- ```
50
- root@netdata~ # /etc/rc.d/sysctl reload
51
- ```
52
-</details>
53
-
54
-
health/guides/swap/30min_ram_swapped_out.md
deleted
-60
@@ -1,60 +0,0 @@
1
-# 30min_ram_swapped_out
2
-
3
-If the system needs more memory resources than your available RAM, inactive pages in memory can be
4
-moved into the swap space (or swap file). The swap space (or swap file) is located on hard drives,
5
-which have a slower access time than physical memory.
6
-
7
-The Netdata Agent calculates the percentage of the system RAM swapped in the last 30 minutes.
8
-
9
-This alert is triggered in warning state if the percentage of the system RAM swapped in is more than
10
-20%.
11
-
12
-
13
-## OS: Linux
14
-
15
-### Troubleshooting section:
16
-
17
-You can find the most resource greedy processes in your system, but if you receive this alert many
18
-times you must consider upgrade your system's RAM.
19
-
20
-<details>
21
-<summary>Find the processes that consume the most RAM </summary>
22
-
23
-1. Use `top` to see the top RAM consumers
24
- ```
25
- root@netdata~ # top -b -o +%MEM | head -n 22
26
- ```
27
-
28
-Here, you can see which processes are the main RAM consumers on the `%MEM` column (it is calculated
29
-in percentage). It would be wise to close/kill any of the main consumer processes that you do not
30
-need to avoid thrashing.
31
-
32
-Netdata strongly suggests knowing exactly what processes you are closing and being certain that they
33
-are not necessary.
34
-</details>
35
-
36
-## OS: FreeBSD
37
-
38
-### Troubleshooting section:
39
-
40
-You can find the most resource greedy processes in your system, but if you receive this alert many
41
-times you must consider upgrade your system's RAM.
42
-
43
-<details>
44
-<summary>Find the processes that consume the most RAM </summary>
45
-
46
-1. Use `top` to see the top RAM consumers
47
- ```
48
- root@netdata~ # top -b -o res | head -n 22
49
- ```
50
-
51
-Here, you can see which processes are the main RAM consumers on the `RES` column (calculated in
52
-percentage). It would be wise to close/kill any of the main consumer processes that you do not need
53
-to avoid thrashing, though Netdata strongly suggests knowing exactly what processes you are closing
54
-and being certain that they are not necessary.
55
-</details>
56
-
57
-
58
-
59
-
60
-
health/guides/swap/used_swap.md
deleted
-51
@@ -1,51 +0,0 @@
1
-# used_swap
2
-
3
-If the system needs more memory resources than your available RAM, inactive pages in memory can be
4
-moved into the swap space (or swap file). The Swap space (or swap file) is located on hard drives,
5
-which have a slower access time than physical memory.
6
-
7
-The Netdata Agent calculates the percentage of the used swap. This alert indicates high swap memory
8
-utilization. It may be a sign that the system has experienced memory pressure, which can affect the
9
-performance of your system. If there is no RAM and swap available, OOM Killer can start killing
10
-processes.
11
-
12
-This alert is triggered in warning state when the percentage of used swap is between 80-90% and in
13
-critical state when it is between 90-98%.
14
-
15
-## OS: Linux
16
-
17
-### Troubleshooting section:
18
-
19
-<details>
20
-<summary>Find the processes that consume the most RAM </summary>
21
-
22
-1. Use `top` to see the top RAM consumers
23
- ```
24
- root@netdata~ # top -b -o +%MEM | head -n 22
25
- ```
26
-
27
-Here, you can see which processes are the main RAM consumers on the `%MEM` column (it is calculated
28
-in percentage). It would be wise to close/kill any of the main consumer processes that you do not
29
-need to avoid thrashing.
30
-
31
-Netdata strongly suggests knowing exactly what processes you are closing and being certain that they
32
-are not necessary.
33
-</details>
34
-
35
-## OS: FreeBSD
36
-
37
-### Troubleshooting section:
38
-
39
-<details>
40
-<summary>Find the processes that consume the most RAM </summary>
41
-
42
-1. Use `top` to see the top RAM consumers
43
- ```
44
- root@netdata~ # top -b -o res | head -n 22
45
- ```
46
-
47
-Here, you can see which processes are the main RAM consumers on the `RES` column (calculated in
48
-percentage). It would be wise to close/kill any of the main consumer processes that you do not need
49
-to avoid thrashing, though Netdata strongly suggests knowing exactly what processes you are closing
50
-and being certain that they are not necessary.
51
-</details>
health/guides/sync_freq.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of `sync()` system calls is greater than 6. The `sync()` system call writes any data buffered in memory out to disk, including modified superblocks, modified inodes, and delayed reads and writes. A higher number of `sync()` calls indicates that the system is often trying to flush buffered data to disk, which can cause performance issues.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify the process causing sync events
8
+
9
+ Use `bpftrace` to identify which processes are causing the sync events. Make sure you have `bpftrace` installed on your system; if not, follow the instructions here: [Installing bpftrace](https://github.com/iovisor/bpftrace/blob/master/INSTALL.md)
10
+
11
+ Run the `syncsnoop.bt` script from the `bpftrace` tools:
12
+
13
+ ```
14
+ sudo bpftrace /path/to/syncsnoop.bt
15
+ ```
16
+
17
+ This script will trace sync events and display the process ID (PID), process name, and the stack trace.
18
+
19
+2. Analyze the output
20
+
21
+ Focus on processes with a high number of sync events, and investigate whether you can optimize these processes or reduce their impact on the system.
22
+
23
+ - Check if these processes are essential to system functionality.
24
+ - Look for potential bugs or misconfigurations that may trigger undue `sync()` calls.
25
+ - Consider modifying the process itself to reduce disk I/O or change how it handles write operations.
26
+
27
+3. Monitor your system's I/O performance
28
+
29
+ Keep an eye on overall I/O performance using tools like `iostat`, `iotop`, or `vmstat`.
30
+
31
+ For example, you can use `iostat` to monitor disk I/O:
32
+
33
+ ```
34
+ iostat -xz 1
35
+ ```
36
+
37
+ This command displays extended disk I/O statistics with a 1-second sampling interval.
38
+
39
+ Check for high `await` values, which indicate the average time taken for I/O requests to be completed. Look for high `%util` values, representing the percentage of time the device was busy servicing requests.
40
+
41
+### Useful resources
42
+
43
+1. [sync man pages](https://man7.org/linux/man-pages/man2/sync.2.html)
44
+2. [bpftrace GitHub repository](https://github.com/iovisor/bpftrace)
45
+3. [syncsnoop example](https://github.com/iovisor/bpftrace/blob/master/tools/syncsnoop_example.txt)
46
+4. [iostat man pages](https://man7.org/linux/man-pages/man1/iostat.1.html)
\ No newline at end of file
health/guides/synchronization/sync_freq.md
deleted
-48
@@ -1,48 +0,0 @@
1
-# sync_freq
2
-
3
-## OS: Any
4
-
5
-By default, the Linux kernel writes data to disk asynchronously. Writes are buffered (cached) in
6
-memory, and written to the storage device at the optimal time.
7
-
8
-Whenever you issue a write or send syscall or write to file-backed mappings or similar things, the
9
-kernel is not forced to flush that data straight to persistent storage, the underlying network stack,
10
-or any other subsystem. This buffering is implemented by the kernel for performance reasons.
11
-
12
-The `sync()` system call writes any data buffered in memory out to disk. This can include (but is not limited to)
13
-modified superblocks, modified inodes, and delayed reads and writes.
14
-
15
-The Netdata Agent monitors the number of sync() system calls. Receiving this alert indicates a high
16
-number of sync() system calls. Every call is very expensive because it causes all pending
17
-modifications to filesystem metadata and cached file data to be written to the underlying
18
-filesystems.
19
-
20
-This alert is triggered in warning state when the number of sync() system calls is greater than 6.
21
-
22
-<details>
23
- <summary>References and source </summary>
24
-
25
- 1. [sync man pages](https://man7.org/linux/man-pages/man2/sync.2.html)
26
-</details>
27
-
28
-### Troubleshooting section
29
-
30
-The `sync()` is expected to occur when the system is about to become unstable, or a storage device to become
31
-suddenly unavailable, and you want to ensure all data is written to disk. If you receive this alert often, you
32
-should gather more information on why this event is happening.
33
-
34
-<details>
35
- <summary>Use bpftrace to identify which process is causing these sync events</summary>
36
-
37
- `bpftrace` is a high-level tracing language for Linux enhanced Berkeley Packet Filter (eBPF) available in recent
38
- Linux kernels (4.x). bpftrace uses LLVM as a backend to compile scripts to BPF-bytecode and makes use of BCC
39
- for interacting with the Linux BPF system, as well as existing Linux tracing capabilities such as kernel dynamic
40
- tracing (kprobes), user-level dynamic tracing (uprobes), and tracepoints.
41
-
42
- One of the builtin tools in the `bpftrace` is the [syncsnoop](https://github.com/iovisor/bpftrace/blob/master/tools/syncsnoop_example.txt)
43
- which tracing the `sync` events
44
-
45
-</details>
46
-
47
-
48
-</details>
health/guides/system_clock_sync_state.md
new
+11
@@ -0,0 +1,11 @@
1
+### Understand the alert
2
+
3
+The Netdata Agent checks if your system is in sync with a Network Time Protocol (NTP) server. This alert indicates that the system time is not synchronized to a reliable server. It is strongly recommended having the clock in sync with NTP servers, because, otherwise, it leads to unpredictable problems that are difficult to debug especially in matters of security.
4
+
5
+### Troubleshoot the alert
6
+
7
+Different linux distros utilize different NTP tools. You can always install `ntp`. If your clock is out of sync, you should first check for issues in your network connectivity.
8
+
9
+### Useful resources
10
+
11
+[Best practices for NTP servers](https://bluecatnetworks.com/blog/seven-best-practices-to-keep-your-ntp-resilient/).
\ No newline at end of file
health/guides/systemd_automount_unit_failed_state.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a `systemd` automount unit enters the `failed` state. It means that a mounted filesystem has failed or experienced an error and thus is not available for use.
4
+
5
+### What is an automount unit?
6
+
7
+An automount unit is a type of `systemd` unit that handles automounting filesystems. It defines when, where, and how a filesystem should be automatically mounted on the system. Automount units use the `.automount` file extension and are typically located in the `/etc/systemd/system` directory.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the failed automount unit(s)
12
+
13
+To list all `systemd` automount units and their states, run the following command:
14
+
15
+```
16
+systemctl list-units --all --type=automount
17
+```
18
+
19
+Look for the unit(s) with a `failed` state.
20
+
21
+2. Check the automount unit file
22
+
23
+Examine the failed unit's configuration file in `/etc/systemd/system/` or `/lib/systemd/system/` (depending on your system). If there is an error in the configuration, fix it and reload the `systemd` configuration.
24
+
25
+```
26
+sudo systemctl daemon-reload
27
+```
28
+
29
+3. Check the journal for errors
30
+
31
+Use the `journalctl` command to check for any system logs related to the failed automount unit:
32
+
33
+```
34
+sudo journalctl -u [UnitName].automount
35
+```
36
+
37
+Replace `[UnitName]` with the name of the failed automount unit. Analyze the logs to identify the root cause of the failure.
38
+
39
+4. Attempt to restart the automount unit
40
+
41
+After identifying and addressing the cause of the failure, try to restart the automount unit:
42
+
43
+```
44
+sudo systemctl restart [UnitName].automount
45
+```
46
+
47
+Check the unit's status:
48
+
49
+```
50
+systemctl status [UnitName].automount
51
+```
52
+
53
+If it's in the `active` state, the issue has been resolved.
54
+
55
+### Useful resources
56
+
57
+1. [Arch Linux Wiki: systemd automount](https://wiki.archlinux.org/title/Fstab#systemd_automount)
58
+2. [systemd automount unit file example](https://www.freedesktop.org/software/systemd/man/systemd.automount.html#Examples)
health/guides/systemd_device_unit_failed_state.md
new
+65
@@ -0,0 +1,65 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a `systemd device unit` enters a `failed state`. If you receive this alert, it means that a device managed by `systemd` on your Linux system has encountered an issue and is currently in a non-operational state.
4
+
5
+### What is a systemd device unit?
6
+
7
+`Systemd` is a system and service manager for Linux operating systems. A `device unit` in `systemd` is a unit that encapsulates a device in the system's device tree (e.g., `/sys` directory). The device units are used to automatically discover and manage devices present on the system.
8
+
9
+### What does a failed state mean?
10
+
11
+A `failed state` implies that the device has encountered an issue and is currently non-operational. The problem could be related to hardware, driver, or configuration issues.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Identify the failed device unit:
16
+
17
+ Check the `systemd` status for failed units using the following command:
18
+
19
+ ```
20
+ systemctl --failed --type=device
21
+ ```
22
+
23
+ This will show you the list of device units that are currently in a failed state.
24
+
25
+2. Check logs for errors:
26
+
27
+ Use the `journalctl` command to check the logs for any error messages related to the failed device unit. For instance, if the failed unit is `example.device`, you can execute:
28
+
29
+ ```
30
+ journalctl -xe -u example.device
31
+ ```
32
+
33
+ This will show you the logs with any error messages that will help you identify the root cause of the failure.
34
+
35
+3. Fix the issue:
36
+
37
+ Depending on the results from the previous steps, you might need to:
38
+
39
+ - Check the hardware connections and make sure they are properly connected.
40
+ - Update or reinstall the device driver.
41
+ - Check and correct device configurations if needed.
42
+
43
+4. Restart the device unit:
44
+
45
+ Once the issue has been fixed, restart the device unit using `systemctl`:
46
+
47
+ ```
48
+ sudo systemctl restart example.device
49
+ ```
50
+
51
+ Replace `example.device` with the specific device unit name.
52
+
53
+5. Validate the fix:
54
+
55
+ Check if the device unit is now operational by executing the following command:
56
+
57
+ ```
58
+ systemctl status example.device
59
+ ```
60
+
61
+ This should show you that the device unit is now active and running properly.
62
+
63
+### Useful resources
64
+
65
+1. [Systemd Device Units](https://www.freedesktop.org/software/systemd/man/systemd.device.html)
health/guides/systemd_mount_unit_failed_state.md
new
+54
@@ -0,0 +1,54 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a `systemd` mount unit enters a `failed state`. If you receive this alert, it means that your system has encountered an issue with mounting a filesystem or a mount point.
4
+
5
+### What is a systemd mount unit?
6
+
7
+`systemd` is the init system used in most Linux distributions to manage services, processes, and system startup. A mount unit is a configuration file that describes how a filesystem or mount point should be mounted and managed by `systemd`.
8
+
9
+### What does a failed state mean?
10
+
11
+A `failed state` indicates that there was an issue with mounting the filesystem, or the mount point failed to function as expected. This can be caused by multiple factors, such as incorrect configuration, missing dependencies, or hardware issues.
12
+
13
+### Troubleshoot the alert
14
+
15
+- Identify the failed mount unit
16
+
17
+ Check the status of your `systemd` mount units by running:
18
+ ```
19
+ systemctl list-units --type=mount
20
+ ```
21
+ Look for units with a `failed` state.
22
+
23
+- Check the journal logs
24
+
25
+ To gain more insight into the issue, check the `systemd` journal logs for the failed mount unit:
26
+ ```
27
+ journalctl -u [unit-name]
28
+ ```
29
+ Replace `[unit-name]` with the actual name of the failed mount unit.
30
+
31
+- Verify the mount unit configuration
32
+
33
+ Review the mount unit configuration file located at `/etc/systemd/system/[unit-name].mount`. Ensure that options such as the filesystem type, device, and mount point are correct.
34
+
35
+- Check system logs for hardware or filesystem issues
36
+
37
+ Review the system logs (e.g., `/var/log/syslog` or `/var/log/messages`) for any hardware or filesystem related errors. Ensure that the device and mount point are properly connected and accessible.
38
+
39
+- Restart the mount unit
40
+
41
+ If you have made any changes to the configuration or resolved a hardware issue, attempt to restart the mount unit by running:
42
+ ```
43
+ systemctl restart [unit-name].mount
44
+ ```
45
+
46
+- Seek technical support
47
+
48
+ If the issue persists, consider reaching out to support, as there might be an underlying issue that needs to be addressed.
49
+
50
+### Useful resources
51
+
52
+1. [systemd.mount - Mount unit configuration](https://www.freedesktop.org/software/systemd/man/systemd.mount.html)
53
+2. [systemctl - Control the systemd system and service manager](https://www.freedesktop.org/software/systemd/man/systemctl.html)
54
+3. [journalctl - Query the systemd journal](https://www.freedesktop.org/software/systemd/man/journalctl.html)
\ No newline at end of file
health/guides/systemd_path_unit_failed_state.md
new
+61
@@ -0,0 +1,61 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a `systemd path unit` enters a `failed state`. Service units in a failed state indicate an issue with the service's startup, runtime, or shutdown, which can result in the service being marked as failed.
4
+
5
+### What is a systemd path unit?
6
+
7
+`systemd` is an init system and system manager that manages services and their dependencies on Linux systems. A `path unit` is a type of unit configuration file that runs a service in response to the existence or modification of files and directories. These units are used to monitor files and directories and trigger actions based on changes to them.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the failed systemd path unit
12
+
13
+First, you need to identify which path unit is experiencing issues. To list all failed units:
14
+
15
+ ```
16
+ systemctl --state=failed
17
+ ```
18
+
19
+Take note of the units indicated as 'path' in the output.
20
+
21
+2. Inspect the path unit status
22
+
23
+To get more details about the specific failed path unit, run:
24
+
25
+ ```
26
+ systemctl status <failed-path-unit>
27
+ ```
28
+
29
+Replace `<failed-path-unit>` with the name of the failed path unit you identified previously.
30
+
31
+3. Review logs for the failed path unit
32
+
33
+To view the logs for the failed path unit, use the `journalctl` command:
34
+
35
+ ```
36
+ journalctl -u <failed-path-unit>
37
+ ```
38
+
39
+Again, replace `<failed-path-unit>` with the name of the failed path unit. Review the logs to identify possible reasons for the failure.
40
+
41
+4. Reload the unit configuration (if necessary)
42
+
43
+If you discovered an issue in the unit configuration file and resolved it, reload the configuration by running:
44
+
45
+ ```
46
+ sudo systemctl daemon-reload
47
+ ```
48
+
49
+5. Restart the failed path unit
50
+
51
+Once you have identified and resolved the issue causing the failed state, try to restart the path unit:
52
+
53
+ ```
54
+ sudo systemctl restart <failed-path-unit>
55
+ ```
56
+
57
+Replace `<failed-path-unit>` with the name of the failed path unit. Then, monitor the path unit status to ensure it is running without issues.
58
+
59
+### Useful resources
60
+
61
+1. [Introduction to Systemd Units and Unit Files](https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files)
health/guides/systemd_scope_unit_failed_state.md
new
+57
@@ -0,0 +1,57 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a systemd scope unit enters a failed state. If you receive this alert, it means that one of your systemd scope units is not working properly and requires attention.
4
+
5
+### What is a systemd scope unit?
6
+
7
+Systemd is the system and service manager on modern Linux systems. It is responsible for managing and controlling system processes, services, and units. A scope unit is a type of systemd unit that groups several processes together in a single unit. It is used to organize and manage resources of a group of processes.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the systemd scope unit in the failed state
12
+
13
+To list all the systemd scope units on the system, run the following command:
14
+
15
+```
16
+systemctl list-units --type=scope
17
+```
18
+
19
+Look for the units with a 'failed' state.
20
+
21
+2. Check the status of the systemd scope unit
22
+
23
+To get more information about the failed systemd scope unit, use the `systemctl status` command followed by the unit name:
24
+
25
+```
26
+systemctl status UNIT_NAME
27
+```
28
+
29
+This command will display the unit status, any error messages, and the last few lines of the unit logs.
30
+
31
+3. Consult the logs for further details
32
+
33
+To get additional information about the unit's failure, you can use the `journalctl` command for the specific unit:
34
+
35
+```
36
+journalctl -u UNIT_NAME
37
+```
38
+
39
+This command will display the logs of the systemd scope unit, allowing you to identify any issues or error messages.
40
+
41
+4. Restart the systemd scope unit
42
+
43
+If the issue appears to be temporary, try restarting the unit using the following command:
44
+
45
+```
46
+systemctl restart UNIT_NAME
47
+```
48
+
49
+This will attempt to stop the failed unit and start it again.
50
+
51
+5. Debug and fix the issue
52
+
53
+If the systemd scope unit keeps failing, refer to the documentation and logs to debug the issue and apply the necessary fixes. You might need to update the unit's configuration, fix application issues, or address system resource limitations.
54
+
55
+### Useful resources
56
+
57
+1. [Systemd - Understanding and Managing System Startup](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/chap-Managing_Services_with_systemd)
health/guides/systemd_service_unit_failed_state.md
new
+66
@@ -0,0 +1,66 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a `systemd service unit` enters the `failed state`. If you receive this alert, it means that a critical service on your system has stopped working, and it requires immediate attention.
4
+
5
+### What is a systemd service unit?
6
+
7
+A `systemd service unit` is a simply stated, a service configuration file that describes how a specific service should be controlled and managed on a Linux system. It includes information about service dependencies, the order in which it should start, and more. Systemd is responsible for managing these services and making sure they are functioning as intended.
8
+
9
+### What does the failed state mean?
10
+
11
+When a `systemd service unit` enters the `failed state`, it indicates that the service has encountered a fault, such as an incorrect configuration file, crashing, or failing to start due to other dependencies. When this occurs, the service is rendered non-functional, and you should troubleshoot the issue to restore normal functionality.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Identify the failed service unit
16
+
17
+ Use the following command to list all failed service units:
18
+
19
+ ```
20
+ systemctl --state=failed
21
+ ```
22
+
23
+ Take note of the failed service unit name as you will use it in the next steps.
24
+
25
+2. Check the service unit status
26
+
27
+ Use the following command to investigate the status and any error messages:
28
+
29
+ ```
30
+ systemctl status <failed_service_unit>
31
+ ```
32
+
33
+ Replace `<failed_service_unit>` with the name of the failed service unit you identified earlier.
34
+
35
+3. Examine the logs for the failed service
36
+
37
+ Use the following command to inspect the logs for any clues:
38
+
39
+ ```
40
+ journalctl -u <failed_service_unit> --since "1 hour ago"
41
+ ```
42
+
43
+ Adjust the `--since` parameter to view logs from a specific timeframe.
44
+
45
+4. Resolve the issue
46
+
47
+ Based on the information gathered from the status and logs, try to resolve the issue causing the failure. This can involve updating configuration files, installing missing dependencies, or addressing issues with other services that the failed service unit depends on.
48
+
49
+5. Restart the service
50
+
51
+ Once the issue has been addressed, restart the service to restore functionality:
52
+
53
+ ```
54
+ systemctl start <failed_service_unit>
55
+ ```
56
+
57
+ Verify that the service has started successfully:
58
+
59
+ ```
60
+ systemctl status <failed_service_unit>
61
+ ```
62
+
63
+### Useful resources
64
+
65
+1. [Systemd: Managing Services (ArchWiki)](https://wiki.archlinux.org/title/Systemd#Managing_services)
66
+2. [Troubleshooting Systemd Services (Digital Ocean)](https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units)
health/guides/systemd_slice_unit_failed_state.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a `systemd slice unit` enters a `failed state`. Systemd slice units are a way to organize and manage system processes in a hierarchical manner. If you receive this alert, it means that there is an issue with a specific slice unit, which can be crucial for system stability and performance.
4
+
5
+### What does the failed state mean?
6
+
7
+A `failed state` in the context of systemd units means that the unit has encountered a problem and is not functioning properly. This could be caused by a variety of reasons, such as misconfiguration, dependency issues, or unhandled errors in the underlying service.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Identify the problematic systemd slice unit.
12
+
13
+ Run the following command to list all systemd units and their states:
14
+
15
+ ```bash
16
+ systemctl --all
17
+ ```
18
+
19
+ Look for the units with the `failed` state in the output, and take note of the affected unit(s).
20
+
21
+- Investigate the specific issue with the failed unit.
22
+
23
+ Use the `systemctl status` command followed by the unit name to get more information about the problem:
24
+
25
+ ```bash
26
+ systemctl status <unit-name>
27
+ ```
28
+
29
+ The output will provide more details on the issue and may include error messages or log entries that can help identify the root cause.
30
+
31
+- Check the unit logs for additional clues.
32
+
33
+ The `journalctl` command can be used to view the logs related to a specific unit by specifying the `-u` flag followed by the unit name:
34
+
35
+ ```bash
36
+ journalctl -u <unit-name>
37
+ ```
38
+
39
+ Analyze the log entries for any reported errors or warnings that could be related to the failure.
40
+
41
+- Address the root cause of the issue.
42
+
43
+ Based on the information gathered, take the necessary steps to resolve the issue with the failed unit. This may involve reconfiguring the unit, adjusting dependencies, or fixing the underlying service.
44
+
45
+- Restart the unit and verify its status.
46
+
47
+ Once the issue has been resolved, restart the systemd unit using the `systemctl restart` command:
48
+
49
+ ```bash
50
+ systemctl restart <unit-name>
51
+ ```
52
+
53
+ Afterwards, check the unit's status to confirm that it is no longer in a failed state and is functioning properly:
54
+
55
+ ```bash
56
+ systemctl status <unit-name>
57
+ ```
58
+
health/guides/systemd_socket_unit_failed_state.md
new
+65
@@ -0,0 +1,65 @@
1
+### Understand the alert
2
+
3
+The `systemd_socket_unit_failed_state` alert is triggered when a `systemd` socket unit on your Linux server enters a failed state. This could indicate issues with the services that depend on these socket units, impacting their functionality or performance.
4
+
5
+### What is a systemd socket unit?
6
+
7
+`systemd` is the system and service manager for modern Linux systems. It initializes and manages the services on the system, ensuring a smooth boot process and operation.
8
+
9
+A socket unit is a special kind of `systemd` unit that encapsulates local and remote IPC (Inter-process communication) sockets. They are defined by .socket files and are used to start and manage services automatically when incoming traffic is received on socket addresses managed by the socket unit.
10
+
11
+### Troubleshoot the alert
12
+
13
+1. Identify the failed socket unit(s):
14
+
15
+To list all the socket units with their current state, run:
16
+
17
+```
18
+systemctl --state=failed --type=socket
19
+```
20
+
21
+This command will display the socket units in a failed state.
22
+
23
+2. Check the status of the failed socket unit:
24
+
25
+To view the detailed status of a particular failed socket unit, use:
26
+
27
+```
28
+systemctl status your_socket_unit.socket
29
+```
30
+
31
+Replace `your_socket_unit` with the name of the failed socket unit you're investigating. This will provide more information about the socket unit and possible error messages.
32
+
33
+3. Examine the logs:
34
+
35
+Check the logs for any errors or issues related to the failed socket unit:
36
+
37
+```
38
+journalctl -u your_socket_unit.socket
39
+```
40
+
41
+Replace `your_socket_unit` with the name of the failed socket unit you're investigating. This will display relevant logs for the socket unit.
42
+
43
+4. Restart the failed socket unit:
44
+
45
+Once the issue is identified and resolved, you can attempt to restart the failed socket unit:
46
+
47
+```
48
+systemctl restart your_socket_unit.socket
49
+```
50
+
51
+Replace `your_socket_unit` with the name of the failed socket unit you're investigating. This will attempt to restart the socket unit and put it into an active state.
52
+
53
+5. Monitor the socket unit:
54
+
55
+After restarting the socket unit, monitor its status to ensure it stays active and operational:
56
+
57
+```
58
+systemctl status your_socket_unit.socket
59
+```
60
+
61
+Replace `your_socket_unit` with the name of the failed socket unit you're investigating. Verify that the socket unit remains in an active state.
62
+
63
+### Useful resources
64
+
65
+1. [Sockets in Systemd Linux Operating System](https://www.freedesktop.org/software/systemd/man/systemd.socket.html)
health/guides/systemd_swap_unit_failed_state.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+This alert monitors the state of your `systemd` swap units and is triggered when a swap unit is in the `failed` state. If you receive this alert, it means that you have an issue with one or more of your swap units managed by `systemd`.
4
+
5
+### What is a swap unit?
6
+
7
+A swap unit in Linux is a dedicated partition or a file on the filesystem (called a swap file) used for expanding system memory. When the physical memory (RAM) gets full, the Linux system swaps some of the least used memory pages to this swap space, allowing more applications to run without the need for extra physical memory.
8
+
9
+### What does the failed state mean?
10
+
11
+If a `systemd` swap unit is in the `failed` state, it means that there was an issue initializing or activating the swap space. This might be due to configuration issues, disk space limitations, or filesystem errors.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the status of the swap units:
16
+
17
+ To list the swap units and their states, run the following command:
18
+
19
+ ```
20
+ systemctl list-units --type=swap
21
+ ```
22
+
23
+ Look for the failed swap units and note their names.
24
+
25
+2. Investigate the failed swap units:
26
+
27
+ For each failed swap unit, check its status and any relevant messages by running:
28
+
29
+ ```
30
+ systemctl status <swap_unit_name>
31
+ ```
32
+
33
+ Replace `<swap_unit_name>` with the name of the failed swap unit.
34
+
35
+3. Check system logs:
36
+
37
+ Examine the system logs for any errors or information related to the failed swap units with:
38
+
39
+ ```
40
+ journalctl -xeu <swap_unit_name>
41
+ ```
42
+
43
+4. Identify the issue and take corrective actions:
44
+
45
+ Based on the information from the previous steps, you may need to:
46
+
47
+ - Adjust swap unit configurations
48
+ - Increase disk space or allocate a larger swap partition
49
+ - Resolve disk or filesystem issues
50
+ - Restart the swap units
51
+
52
+5. Verify that the swap units are working:
53
+
54
+ After resolving the issue, ensure the swap units are active and running by repeating step 1.
55
+
56
+### Useful resources
57
+
58
+1. [systemd.swap — Swap unit configuration](https://www.freedesktop.org/software/systemd/man/systemd.swap.html)
health/guides/systemd_target_unit_failed_state.md
new
+52
@@ -0,0 +1,52 @@
1
+### Understand the alert
2
+
3
+The `systemd_target_unit_failed_state` alert is triggered when a `systemd` target unit goes into a failed state. Systemd is the system and service manager for Linux, and target units are groups of systemd units that are organized for a specific purpose. If this alert is triggered, it means there is an issue with one of your systemd target units.
4
+
5
+### What does failed state mean?
6
+
7
+A systemd target unit in the failed state means that one or more units/tasks of that target, whether it's a service, or any other kind of systemd unit, have encountered an issue and cannot continue running.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. First, you need to identify which systemd target unit is causing the alert. You can list all the failed units by running:
12
+
13
+ ```
14
+ systemctl --failed --all
15
+ ```
16
+
17
+2. Once you have identified the problematic target unit, check its status for more information about the issue. Replace `<target_unit>` with the actual target unit name:
18
+
19
+ ```
20
+ systemctl status <target_unit>
21
+ ```
22
+
23
+3. Look at the logs of the failed target unit to collect more details on the issue:
24
+
25
+ ```
26
+ journalctl -u <target_unit>
27
+ ```
28
+
29
+4. Based on the information gathered in steps 2 and 3, troubleshoot and fix the problem(s) in your target unit. This may involve:
30
+ - Editing the unit file
31
+ - Checking the services and processes that compose the target
32
+ - Looking into configuration files and directories.
33
+
34
+5. Reload the systemctl daemon to apply any changes you made, then restart the target unit:
35
+
36
+ ```
37
+ sudo systemctl daemon-reload
38
+ sudo systemctl restart <target_unit>
39
+ ```
40
+
41
+6. Verify that the target unit has been successfully restarted:
42
+
43
+ ```
44
+ systemctl is-active <target_unit>
45
+ ```
46
+
47
+7. Continue monitoring the target unit to ensure that it remains stable and does not return to a failed state.
48
+
49
+### Useful resources
50
+
51
+1. [systemd man pages (targets)](https://www.freedesktop.org/software/systemd/man/systemd.target.html)
52
+2. [systemd Targets - ArchWiki](https://wiki.archlinux.org/title/Systemd#Targets)
health/guides/systemdunits/systemd_automount_units_state.md
deleted
-114
@@ -1,114 +0,0 @@
1
-# systemd_automount_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the `systemd.automount` units. The `systemd_automount_units_state` alert
9
-indicates that one or more of the `systemd.automount` units have failed.
10
-A systemd automount unit "failed" when the service process returned error code on exit, or crashed, an
11
-operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
-
13
-<details>
14
-<summary>Read More About systemd</summary>
15
-
16
-Here is some useful information about systemd from
17
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
-
19
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
-daemons
23
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
-Poettering described systemd not as one program, but rather a large software suite that includes 69
25
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
-its control. systemd also integrates many other services that are common on Linux systems by
28
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
-logging, hostnames and locales.
30
-
31
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
-parent of a process when the original parent terminates. Therefore, the first process is
36
-particularly well suited for the purpose of monitoring daemons.
37
-
38
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
-be preserved in a snapshot for future recall.
42
-
43
-Systemd's core components include the following:
44
-
45
-- `systemd` is a system and service manager for Linux operating systems.
46
-
47
-- `systemctl` is a command to introspect and control the state of the systemd system and service
48
- manager. Not to be confused with sysctl.
49
-
50
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
- other state and tracing information from the system and service manager.
52
-
53
-</details>
54
-
55
-
56
-<details>
57
-<summary>See More On systemd - `.automount` Units</summary>
58
-
59
-A unit configuration file whose name ends in `.automount` encodes information about a file system
60
-automount point controlled and supervised by `systemd`. Automount units must be named after the
61
-automount directories they control. For instance, the automount point `/home/lennart` must be
62
-configured in a unit file `home-lennart.automount`. For details about the escaping logic used to
63
-convert a file system path to a unit name see `systemd.unit(5)`. Note that automount units cannot be
64
-templated, nor is it possible to add multiple names to an automount unit by creating additional
65
-symlinks to its unit file.
66
-
67
-For each automount unit file a matching mount unit file (see systemd.mount(5) for details) must
68
-exist which is activated when the automount path is accessed. For instance, if an automount unit
69
-`home-lennart.automount` is active and the user accesses `/home/lennart` the mount unit
70
-`home-lennart.mount` will be
71
-activated. <sup> [2](https://www.freedesktop.org/software/systemd/man/systemd.automount.html) </sup>
72
-
73
-</details>
74
-
75
-<details>
76
-<summary>References and Source</summary>
77
-
78
-1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
79
-2. [man page for systemd.automount](https://www.freedesktop.org/software/systemd/man/systemd.automount.html)
80
-
81
-</details>
82
-
83
-### Troubleshooting Section:
84
-
85
-<details>
86
-<summary>General Approach</summary>
87
-
88
-If an automount has failed, then you should always try to collect more information to diagnose the cause of
89
-the failure.
90
-
91
-1. Identify which automount fails. Open the Netdata dashboard, find the current active alarms under
92
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
93
- into its chart.
94
- (`systemdunits_automount_units.automount_unit_state`). In this chart, identify which automount
95
- units are in state with value 5
96
-
97
-2. Gather more information about the failing automount. We advise you to run the following commands
98
- in two different terminals.
99
-
100
- ```
101
- root@netdata~ # journalctl -u <automount_name>.automount -f
102
- root@netdata~ # journalctl -u <automount_name>.mount -f
103
- ```
104
-
105
-3. In your main terminal, try mount the automount manually.
106
-
107
- ```
108
- root@netdata~ # mount -v <automount_name>
109
- ```
110
-
111
- This command will try to mount your automount unit in verbose mode.
112
-4. Check the output messages from both terminals for abnormalities.
113
-
114
-</details>
health/guides/systemdunits/systemd_device_units_state.md
deleted
-94
@@ -1,94 +0,0 @@
1
-# systemd_device_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the `systemd.device` units. The `systemd_device_units_state` alert
9
-indicates that one or more of the `systemd.device` units have failed.
10
-A systemd device unit "failed" when the service process returned error code on exit, or crashed, an
11
-operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
-
13
-<details>
14
-<summary>Read More About systemd</summary>
15
-
16
-Here is some useful information about systemd from
17
-Wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
-
19
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
-daemons
23
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
-Poettering described systemd not as one program, but rather a large software suite that includes 69
25
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
-its control. systemd also integrates many other services that are common on Linux systems by
28
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
-logging, hostnames and locales.
30
-
31
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
-parent of a process when the original parent terminates. Therefore, the first process is
36
-particularly well suited for the purpose of monitoring daemons.
37
-
38
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
-be preserved in a snapshot for future recall.
42
-
43
-Systemd's core components include the following:
44
-
45
-- `systemd` is a system and service manager for Linux operating systems.
46
-
47
-- `systemctl` is a command to introspect and control the state of the systemd system and service
48
- manager. Not to be confused with sysctl.
49
-
50
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
- other state and tracing information from the system and service manager.
52
-
53
-</details>
54
-
55
-<details>
56
-<summary>More On systemd-`.device` Units</summary>
57
-
58
-A unit configuration file whose name ends in `.device` encodes information about a device unit as
59
-exposed in the `sysfs/udev(7)` device tree. `udev` provides a dynamic device directory containing
60
-only the files for actually present devices. It creates or removes device node files in the `/dev`
61
-directory, or it renames network interfaces.
62
-
63
-This unit type has no specific options. Device units are named after the `/sys` and `/dev` paths
64
-they control. For example, the device `/dev/sda5` is exposed in systemd as `dev-sda5.device`.
65
-</details>
66
-
67
-<details>
68
-<summary>References and Source</summary>
69
-
70
-1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
71
-2. [man page for systemd.device](https://www.freedesktop.org/software/systemd/man/systemd.device.html)
72
-
73
-</details>
74
-
75
-### Troubleshooting Section:
76
-
77
-<details>
78
-<summary>General Approach</summary>
79
-
80
-If a device has failed, then you should always try to gather more information about the cause of the failure.
81
-
82
-1. Identify which device units fail. Open the Netdata dashboard, find the current active alarms under
83
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
84
- into its chart
85
- (`systemdunits_device_units.device_unit_state`). In this chart, identify which device
86
- units are in state with value 5.
87
-
88
-2. Check the status of the device
89
-
90
- ```
91
- root@netdata~ # systemctl status <device_name>.device
92
- ```
93
-
94
-</details>
health/guides/systemdunits/systemd_mount_units_state.md
deleted
-137
@@ -1,137 +0,0 @@
1
-# systemd_mount_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the `systemd.mount` units state. The `systemd_mount_units_state` alert
9
-indicates that one or more of the `systemd.mount` units are in the failed state.
10
-A systemd mount unit "failed" when the service process returned error code on exit, or crashed, an
11
-operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
-
13
-<details>
14
-<summary>Read more about systemd</summary>
15
-
16
-Here is some useful information about systemd from
17
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
-
19
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
-daemons
23
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
-Poettering described systemd not as one program, but rather a large software suite that includes 69
25
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
-its control. systemd also integrates many other services that are common on Linux systems by
28
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
-logging, hostnames and locales.
30
-
31
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
-parent of a process when the original parent terminates. Therefore, the first process is
36
-particularly well suited for the purpose of monitoring daemons.
37
-
38
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
-be preserved in a snapshot for future recall.
42
-
43
-Systemd's core components include the following:
44
-
45
-- `systemd` is a system and service manager for Linux operating systems.
46
-
47
-- `systemctl` is a command to introspect and control the state of the systemd system and service
48
- manager. Not to be confused with sysctl.
49
-
50
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
- other state and tracing information from the system and service manager.
52
-
53
-</details>
54
-
55
-
56
-
57
-<details>
58
-<summary>See more on systemd-mount units</summary>
59
-
60
-A unit configuration file whose name ends in `.mount` encodes information about a file system mount
61
-point controlled and supervised by `systemd`. Additional options are listed in _systemd.exec(5)_,
62
-which define the execution environment the _mount(8)_ program is executed in, and in _systemd.kill(
63
-5)_, which define the way the processes are terminated, and in
64
-_systemd.resource-control(5)_, which configure resource control settings for the processes of the
65
-service.
66
-
67
-The options `User=` and `Group=` are not useful for mount units. systemd passes two parameters to
68
-mount(8) the values of `What=` and `Where=`. When invoked in this way, _mount(8)_ does not read any
69
-options from `/etc/fstab`, and must be run as UID 0.
70
-
71
-Mount units must be named after the mount point directories they control. For instance, the mount
72
-point `/home/lennart` must be configured in a unit file `home-lennart.mount`. For details about the
73
-escaping logic used to convert a file system path to a unit name, see _systemd.unit(5)_. Note that
74
-mount units cannot be templated, nor is possible to add multiple names to a mount unit by creating
75
-additional symlinks to it.
76
-
77
-Mount units may either be configured via unit files, or via `/etc/fstab` (see `man fstab` for
78
-details). Mounts listed in /etc/fstab will be converted into native units dynamically at boot and
79
-when the configuration of the system manager is reloaded. In general, configuring mount points
80
-through `/etc/fstab` is the preferred approach. See _systemd-fstab-generator(8)_ for details about
81
-the
82
-conversion. <sup> [2](https://www.freedesktop.org/software/systemd/man/systemd.mount.html) </sup>
83
-
84
-</details>
85
-
86
-
87
-<details>
88
-<summary>References and source</summary>
89
-
90
-1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
91
-2. [man page for systemd.mount](https://www.freedesktop.org/software/systemd/man/systemd.mount.html)
92
-
93
-</details>
94
-
95
-### Troubleshooting section:
96
-
97
-<details>
98
-<summary>General approach</summary>
99
-
100
-When a mount is in failed state, you should always try to gather more information about it.
101
-
102
-1. Identify which mount units fail. Open the Netdata dashboard, find the current active alarms under
103
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
104
- into its chart.
105
- (`systemdunits_mount_units.mount_unit_state`). In this chart, identify which mount
106
- units are in state with value 5
107
-
108
-2. Gather more information about the failing mount. We advise you to run the following command
109
- in a second terminal.
110
-
111
- ```
112
- root@netdata~ # journalctl -u <mount_name>.mount -f
113
- ```
114
- This command will monitor the journalctl log messages for your mount unit.
115
-3. In your main terminal, try mount the mount manually.
116
-
117
- ```
118
- root@netdata~ # mount -v <mount_name>
119
- ```
120
-
121
- This command will try to mount your mount unit in verbose mode.
122
-4. Check the output messages from both terminals for abnormalities.
123
-
124
-</details>
125
-
126
-
127
-<details>
128
-<summary>Verify the fstab configuration</summary>
129
-
130
-1. Open a terminal and run the following command
131
-
132
- ```
133
- root@netdata~ # sudo findmnt --verify --verbose
134
- ```
135
-
136
-This command will check mount table content (default: `/etc/fstab`) in verbose mode
137
-</details>
health/guides/systemdunits/systemd_path_units_state.md
deleted
-134
@@ -1,134 +0,0 @@
1
-# systemd_path_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the systemd path units state. This alert indicates that one or more of
9
-the systemd path units are in the failed state.
10
-A systemd path unit "failed" when the service process returned error code on exit, or crashed, an
11
-operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
-
13
-<details>
14
-<summary>Read more about systemd</summary>
15
-
16
-Here is some useful information about systemd from
17
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
-
19
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
-daemons
23
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
-Poettering described systemd not as one program, but rather a large software suite that includes 69
25
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
-its control. systemd also integrates many other services that are common on Linux systems by
28
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
-logging, hostnames and locales.
30
-
31
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
-parent of a process when the original parent terminates. Therefore, the first process is
36
-particularly well suited for the purpose of monitoring daemons.
37
-
38
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
-be preserved in a snapshot for future recall.
42
-
43
-Systemd's core components include the following:
44
-
45
-- `systemd` is a system and service manager for Linux operating systems.
46
-
47
-- `systemctl` is a command to introspect and control the state of the systemd system and service
48
- manager. Not to be confused with sysctl.
49
-
50
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
- other state and tracing information from the system and service manager.
52
-
53
-</details>
54
-
55
-
56
-<details>
57
-<summary>See more on systemd.path units</summary>
58
-
59
-> A unit configuration file whose name ends in ".path" encodes information about a path monitored by system. With path units, you can monitor files and directories for certain events. If a specified event occurs, a service unit is executed, and it usually carries the same name as the path unit.
60
->
61
-> In the [Path] section, `PathChanged=` specifies the absolute path to the file to be monitored, while
62
-`Unit=` indicates which service unit to execute if the file changes. <sup>[2](https://www.redhat.com/sysadmin/introduction-path-units) </sup>
63
-
64
-Path units are very useful to monitor files for changes with systemd.
65
-</details>
66
-
67
-
68
-<details>
69
-<summary>References and source</summary>
70
-
71
-1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
72
-2. [A Brief introduction to path units by Jörg Kastning](https://www.redhat.com/sysadmin/introduction-path-units)
73
-
74
-</details>
75
-
76
-### Troubleshooting section:
77
-
78
-<details>
79
-<summary>General approach</summary>
80
-
81
-When a path is in failed state, you should always try to gather more information about it.
82
-
83
-1. Identify which path units fail. Open the Netdata dashboard, find the current active alarms under
84
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
85
- into its chart.
86
- (`systemdunits_path_units.path_unit_state`). In this chart, identify which path units are in
87
- state with value 5
88
-
89
-2. Gather more information about the failing path unit and the service that manages it (in most of
90
- the cases they will have the same name). We advise you to run the following commands in two
91
- different terminals.
92
-
93
- ```
94
- root@netdata~ # journalctl -u <service_name>.service -f
95
- root@netdata~ # journalctl -u <path_name>.socket -f
96
- ```
97
- These commands will monitor the journalctl log messages for your path/service unit.
98
-3. In a new terminal, try to restart the service.
99
-
100
- ```
101
- root@netdata~ # systemctl restart <service_name>.service
102
- ```
103
-
104
-4. Check the log messages from the commands of step 2.
105
-
106
-</details>
107
-
108
-<details>
109
-<summary>Run the service of the path unit in debug mode</summary>
110
-
111
-1. Identify which path units fail. Open the Netdata dashboard, find the current active alarms under
112
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
113
- into its chart.
114
- (`systemdunits_path_units.path_unit_state`). In this chart, identify which path units are in
115
- state with value 5
116
-
117
-2. Stop the service that manages this path (in most of the cases the service will have the same name
118
- with the path)
119
-
120
- ```
121
- root@netdata~ # systemctl stop <service_name>.service
122
- ```
123
-
124
-3. Try to start it with the `SYSTEMD_LOG_LEVEL=debug` env variable. Let's assume in our case we want
125
- to debug the `systemd-networkd` service
126
-
127
- ```
128
- root@netdata~ # SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
129
-
130
- ```
131
-
132
-4. Check the log messages.
133
-
134
-</details>
health/guides/systemdunits/systemd_scope_units_state.md
deleted
-109
@@ -1,109 +0,0 @@
1
-# systemd_scope_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the systemd scope units state. The `systemd_scope_units_state` alert
9
-indicates that one or more of the systemd-scope units are in the failed state.
10
-A systemd scope unit "failed" when the service process returned error code on exit, or crashed, an
11
-operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
-
13
-<details>
14
-<summary>Read more about systemd</summary>
15
-
16
-Here is some useful information about systemd from
17
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
-
19
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
-entire software bundle around it, which, in addition to the systemd `init` daemon, includes the
22
-daemons
23
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
-Poettering described systemd not as one program, but rather a large software suite that includes 69
25
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
-its control. Systemd also integrates many other services that are common on Linux systems by
28
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
-logging, hostnames and locales.
30
-
31
-Like the `init daemon`, Systemd is a daemon that manages other daemons, which, including `systemd`
32
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
-process tree. The first process (`PID 1`) has a special role on Unix systems, as it replaces the
35
-parent of a process when the original parent terminates. Therefore, the first process is
36
-particularly well suited for the purpose of monitoring daemons.
37
-
38
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
-the traditional startup sequence approach. For inter-process communication (IPC), systemd makes Unix
40
-domain sockets and D-Bus available to the running daemons. The state of `systemd` itself can also be
41
-preserved in a snapshot for future recall.
42
-
43
-systemd's core components include the following:
44
-
45
-- `systemd` is a system and service manager for Linux operating systems.
46
-
47
-- `systemctl` is a command to introspect and control the state of the systemd system and service
48
- manager. Not to be confused with sysctl.
49
-
50
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
- other state and tracing information from the system and service manager.
52
-
53
-</details>
54
-
55
-
56
-<details>
57
-<summary>See more on systemd-scope</summary>
58
-The following text originates from the systemd.scope man page.<sup>[2](https://www.freedesktop.org/software/systemd/man/systemd.scope.html) </sup>
59
-
60
-Scope units are not configured via unit configuration files, but are only created programmatically
61
-using the bus interfaces of systemd. They are named similar to filenames. A unit whose name ends
62
-in `.scope` refers to a scope unit. Scopes units manage a set of system processes. Unlike service
63
-units, scope units manage externally created processes, and do not fork off processes on its own.
64
-
65
-The main purpose of scope units is grouping worker processes of a system service for organization
66
-and for managing resources.
67
-
68
-Unlike service units, scope units have no "main" process: all processes in the scope are equivalent.
69
-The lifecycle of the scope unit is thus not bound to the lifetime of one specific process, but to
70
-the existence of at least one process in the scope. This also means that the exit statuses of these
71
-processes are not relevant for the scope unit failure state. Scope units may still enter a failure
72
-state, for example due to resource exhaustion or stop timeouts being reached, but not due to
73
-programs inside of them terminating uncleanly. Since processes managed as scope units generally
74
-remain children of the original process that forked them off, it is also the job of that process to
75
-collect their exit statuses and act on them as
76
-needed.
77
-
78
-</details>
79
-
80
-<details>
81
-<summary>References and source</summary>
82
-
83
-1. [systemd on Wikipedia](https://en.wikipedia.org/wiki/Systemd)
84
-2. [Man page for systemd.scope](https://www.freedesktop.org/software/systemd/man/systemd.scope.html)
85
-
86
-</details>
87
-
88
-### Troubleshooting section:
89
-
90
-<details>
91
-<summary>General approach</summary>
92
-
93
-When a scope is in a failed state, you should always try to gather more information about it.
94
-
95
-1. Identify which scope units fail. Open the Netdata dashboard, find the current active alarms under
96
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
97
- into its chart.
98
- (`systemdunits_scope_units.scope_unit_state`). In this chart, identify which slice units are in
99
- state with value 5
100
-
101
-2. Gather more information about the failing scope unit
102
-
103
- ```
104
- root@netdata~ # systemctl status <scope_name>.scope
105
- ```
106
-
107
-3. Check the log messages from the command of step 2.
108
-
109
-</details>
health/guides/systemdunits/systemd_service_units_state.md
deleted
-138
@@ -1,138 +0,0 @@
1
-# systemd_service_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the systemd service units. The `systemd_service_units_state` alert
9
-indicates that one or more of the systemd service units are in the `failed` state. One of the
10
-following reasons can cause this alert:
11
-
12
-- The process of the service returns an error code on exit.
13
-- The process of the service crashed.
14
-- An operation timed out occurred.
15
-- The service failed after too many restarts.
16
-
17
-<details>
18
-<summary>Read more about systemd</summary>
19
-
20
-Here is some useful information about systemd from
21
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
22
-
23
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
24
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
25
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
26
-daemons
27
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
28
-Poettering described systemd not as one program, but rather a large software suite that includes 69
29
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
30
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
31
-its control. systemd also integrates many other services that are common on Linux systems by
32
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
33
-logging, hostnames and locales.
34
-
35
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
36
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
37
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
38
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
39
-parent of a process when the original parent terminates. Therefore, the first process is
40
-particularly well suited for the purpose of monitoring daemons.
41
-
42
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
43
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
44
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
45
-be preserved in a snapshot for future recall.
46
-
47
-Systemd's core components include the following:
48
-
49
-- `systemd` is a system and service manager for Linux operating systems.
50
-
51
-- `systemctl` is a command to introspect and control the state of the systemd system and service
52
- manager. Not to be confused with sysctl.
53
-
54
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
55
- other state and tracing information from the system and service manager.
56
-
57
-</details>
58
-
59
-
60
-<details>
61
-<summary>See more on systemd-services</summary>
62
-
63
-A unit configuration file whose name ends in `.service` encodes information about a process
64
-controlled and supervised by systemd. To `view`, `start`, `stop`, `restart`, `enable`, or `disable`
65
-system services, use the `systemctl` command line interface. It is common that services are ordered
66
-to start after some specified service that depends on (try the
67
-command `systemctl list-dependencies --before|after <service_name>.service`)
68
-
69
-See more in the man pages, `man systemd.service`
70
-
71
-</details>
72
-
73
-<details>
74
-<summary>References and source</summary>
75
-
76
-1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
77
-
78
-</details>
79
-
80
-### Troubleshooting section:
81
-
82
-<details>
83
-<summary>General approach</summary>
84
-
85
-When a service is in failed state, you should always try to gather more information about it.
86
-
87
-1. Identify which service units fail. Open the Netdata dashboard, find the current active alarms under
88
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
89
- into its chart.
90
- (`systemdunits_service_units.service_unit_state`). In this chart, identify which service
91
- units are in state with value 5
92
-
93
-2. Gather more information about the failing service. We advise you to run the following command in
94
- a second terminal.
95
-
96
- ```
97
- root@netdata~ # journalctl -u <service_name>.service -f
98
- ```
99
- This command will monitor the journalctl log messages for your service.
100
-
101
-3. In a new terminal, try to restart the service.
102
-
103
- ```
104
- root@netdata~ # systemctl restart <service_name>.service
105
- ```
106
- This command will restart your service.
107
-
108
-4. Check the log messages from the command of step 2.
109
-
110
-</details>
111
-
112
-<details>
113
-<summary>Run the service in debug mode</summary>
114
-
115
-1. Identify which service units fail. Open the Netdata dashboard, find the current active alarms under
116
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
117
- into its chart.
118
- (`systemdunits_service_units.service_unit_state`). In this chart, identify which service
119
- units are in state with value 5
120
-
121
-
122
-2. Stop the service
123
-
124
- ```
125
- root@netdata~ # systemctl stop <service_name>.service
126
- ```
127
-
128
-3. Try to start it with the `SYSTEMD_LOG_LEVEL=debug` env variable. Let's assume in our case we want
129
- to debug the `systemd-networkd` service.
130
-
131
- ```
132
- root@netdata~ # SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
133
-
134
- ```
135
- This command will start your service in debug mode.
136
-4. Check the log messages.
137
-
138
-</details>
health/guides/systemdunits/systemd_slice_units_state.md
deleted
-114
@@ -1,114 +0,0 @@
1
-# systemd_slice_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the systemd slice units state. The `systemd_slice_units_state` alert
9
-indicates that one or more of the systemd slice units are in the failed state.
10
-A systemd slice unit "failed" when the service process returned error code on exit, or crashed, an
11
-operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
-
13
-<details>
14
-<summary>Read more about systemd</summary>
15
-
16
-Here is some useful information about systemd from
17
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
-
19
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
-daemons
23
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
-Poettering described systemd not as one program, but rather a large software suite that includes 69
25
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
-its control. systemd also integrates many other services that are common on Linux systems by
28
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
-logging, hostnames and locales.
30
-
31
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
-parent of a process when the original parent terminates. Therefore, the first process is
36
-particularly well suited for the purpose of monitoring daemons.
37
-
38
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
-be preserved in a snapshot for future recall.
42
-
43
-Systemd's core components include the following:
44
-
45
-- `systemd` is a system and service manager for Linux operating systems.
46
-
47
-- `systemctl` is a command to introspect and control the state of the systemd system and service
48
- manager. Not to be confused with sysctl.
49
-
50
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
- other state and tracing information from the system and service manager.
52
-
53
-</details>
54
-
55
-
56
-<details>
57
-<summary>See more on systemd-slice</summary>
58
-
59
-The following text originates from the systemd.slice man page.<sup>[2](https://www.freedesktop.org/software/systemd/man/systemd.slice.html) </sup>
60
-
61
-A unit configuration file whose name ends in ".slice" encodes information about a slice unit. A
62
-slice unit is a concept for hierarchically managing resources of a group of processes. This
63
-management is performed by creating a node in the Linux Control Group (cgroup) tree. Units that
64
-manage processes (primarily scope and service units) may be assigned to a specific slice. For each
65
-slice, certain resource limits may be set that apply to all processes of all units contained in that
66
-slice. Slices are organized hierarchically in a tree. The name of the slice encodes the location in
67
-the tree. The name consists of a dash-separated series of names, which describes the path to the
68
-slice from the root slice. The root slice is named -.slice. For example, foo-bar.slice is a slice
69
-that is located within foo.slice, which in turn is located in the root slice -.slice.
70
-
71
-Note that slice units cannot be templated, nor is possible to add multiple names to a slice unit by
72
-creating additional symlinks to its unit file.
73
-
74
-By default, service and scope units are placed in `system.slice`, virtual machines and containers
75
-registered with `systemd-machined` are found in `machine.slice`, and user sessions handled by
76
-`systemd-logind` in `user.slice`.
77
-
78
-The slice specific configuration options are configured in the `[Slice]` section. Currently, only
79
-generic resource control settings as described in systemd.resource-control(5) are allowed.
80
-
81
-</details>
82
-
83
-
84
-<details>
85
-
86
-<summary>References and source</summary>
87
-
88
-1. [systemd on Wikipedia](https://en.wikipedia.org/wiki/Systemd)
89
-2. [Man page for systemd.slice](https://www.freedesktop.org/software/systemd/man/systemd.slice.html)
90
-
91
-</details>
92
-
93
-### Troubleshooting section:
94
-
95
-<details>
96
-<summary>General approach</summary>
97
-
98
-When a slice is in a failed state, you should always try to gather more information about it.
99
-
100
-1. Identify which slice units fail. Open the Netdata dashboard, find the current active alarms under
101
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
102
- into its chart.
103
- (`systemdunits_slice_units.slice_unit_state`). In this chart, identify which slice
104
- units are in state with value 5.
105
-
106
-2. Gather more information about the failing slice unit
107
-
108
- ```
109
- root@netdata~ # systemctl status <slice_name>.slice
110
- ```
111
-
112
-3. Check the log messages from the command of step 2.
113
-
114
-</details>
health/guides/systemdunits/systemd_socket_units_state.md
deleted
-137
@@ -1,137 +0,0 @@
1
-# systemd_socket_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the systemd socket units state. Receiving this alerts indicates that one
9
-or more of the systemd socket units are in the failed state. In most of the cases this is correlated
10
-with the service, which manages the socket.
11
-A systemd socket unit "failed" when the service process returned error code on exit, or crashed, an
12
-operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
13
-
14
-<details>
15
-<summary>Read more about systemd</summary>
16
-
17
-Here is some useful information about systemd from
18
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
19
-
20
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
21
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
22
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
23
-daemons
24
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
25
-Poettering described systemd not as one program, but rather a large software suite that includes 69
26
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
27
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
28
-its control. systemd also integrates many other services that are common on Linux systems by
29
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
30
-logging, hostnames and locales.
31
-
32
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
33
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
34
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
35
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
36
-parent of a process when the original parent terminates. Therefore, the first process is
37
-particularly well suited for the purpose of monitoring daemons.
38
-
39
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
40
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
41
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
42
-be preserved in a snapshot for future recall.
43
-
44
-Systemd's core components include the following:
45
-
46
-- `systemd` is a system and service manager for Linux operating systems.
47
-
48
-- `systemctl` is a command to introspect and control the state of the systemd system and service
49
- manager. Not to be confused with sysctl.
50
-
51
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
52
- other state and tracing information from the system and service manager.
53
-
54
-</details>
55
-
56
-
57
-<details>
58
-<summary>See more on systemd.socket units</summary>
59
-
60
-> A unit configuration file whose name ends in `.socket` encodes information about an IPC or network socket or a file system FIFO controlled and supervised by systemd, for socket-based activation. For each socket unit, a matching service unit must exist, describing the service to start on incoming traffic on the socket. The name of the .service unit is by default the same as the name of the .socket unit.
61
->
62
-> Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd's native socket passing interface (see sd_listen_fds(3) for details about the precise protocol used and the order in which the file descriptors are passed) or via traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).
63
->
64
-> All network sockets allocated through .socket units are allocated in the host's network namespace
65
-(see network_namespaces(7)). This does not mean however that the service activated by a configured socket unit has to be part of the host's network namespace as well. It is supported and even good practice to run services in their own network namespace (for example through PrivateNetwork=, see systemd.exec(5)), receiving only the sockets configured through socket-activation from the host's namespace. In such a set-up communication within the host's network namespace is only permitted through the activation sockets passed in while all sockets allocated from the service code itself will be associated with the service's own namespace, and thus possibly subject to a a much more restrictive configuration. <sup>[2](https://manpages.debian.org/testing/systemd/systemd.socket.5.en.html) </sup>
66
-
67
-</details>
68
-
69
-
70
-<details>
71
-<summary>References and source</summary>
72
-
73
-1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
74
-2. [systemd.socket on debian.org](https://manpages.debian.org/testing/systemd/systemd.socket.5.en.html)
75
-
76
-</details>
77
-
78
-### Troubleshooting section:
79
-
80
-<details>
81
-<summary>General approach</summary>
82
-
83
-When a socket is in failed state, you should always try to gather more information about it.
84
-
85
-1. Identify which socket units fail. Open the Netdata dashboard, find the current active alarms under
86
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
87
- into its chart.
88
- `systemdunits_service_units.socket_unit_state` chart. Check which sockets are in state with value 5.
89
-
90
-
91
-2. Gather more information about the failing socket and the service that manages it (in most of the
92
- cases they will have the same name). We advise you to run the following commands in two different
93
- terminals.
94
-
95
- ```
96
- root@netdata~ # journalctl -u <service_name>.service -f
97
- root@netdata~ # journalctl -u <socket_name>.socket -f
98
- ```
99
-
100
- These commands will monitor the journalctl log messages for your socket/service unit.
101
-3. In a new terminal, try to restart the service.
102
-
103
- ```
104
- root@netdata~ # systemctl restart <service_name>.service
105
- ```
106
-
107
-4. Check the log messages from the command of step 2.
108
-
109
-</details>
110
-
111
-<details>
112
-<summary>Run the service of the socket in debug mode</summary>
113
-
114
-1. Identify which socket units fail. Open the Netdata dashboard, find the current active alarms under
115
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
116
- into its chart.
117
- `systemdunits_service_units.socket_unit_state` chart. Check which sockets are in state with value 5.
118
-
119
-2. Stop the service that manages this socket (in most of the cases the service will have the same
120
- name with the socket)
121
-
122
- ```
123
- root@netdata~ # systemctl stop <service_name>.service
124
- ```
125
-
126
-3. Try to start it with the `SYSTEMD_LOG_LEVEL=debug` env variable. Let's assume in our case we want
127
- to debug the `systemd-networkd` service
128
-
129
- ```
130
- root@netdata~ # SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
131
-
132
- ```
133
-
134
- This command will start your service in debug mode.
135
-4. Check the log messages.
136
-
137
-</details>
health/guides/systemdunits/systemd_swap_units_state.md
deleted
-116
@@ -1,116 +0,0 @@
1
-# systemd_swap_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the systemd swap units state. The `systemd_swap_units_state` alert
9
-indicates that one or more of the systemd swap units are in the failed state.
10
-A systemd swap unit "failed" when the service process returned error code on exit, or crashed, an
11
-operation timed out, or after too many restarts. The cause of a failed states is stored in a log.
12
-
13
-<details>
14
-<summary>Read more about systemd</summary>
15
-
16
-Here is some useful information about systemd from
17
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
-
19
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
-daemons
23
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
-Poettering described systemd not as one program, but rather a large software suite that includes 69
25
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
-its control. systemd also integrates many other services that are common on Linux systems by
28
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
-logging, hostnames and locales.
30
-
31
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
-parent of a process when the original parent terminates. Therefore, the first process is
36
-particularly well suited for the purpose of monitoring daemons.
37
-
38
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
-be preserved in a snapshot for future recall.
42
-
43
-Systemd's core components include the following:
44
-
45
-- `systemd` is a system and service manager for Linux operating systems.
46
-
47
-- `systemctl` is a command to introspect and control the state of the systemd system and service
48
- manager. Not to be confused with sysctl.
49
-
50
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
- other state and tracing information from the system and service manager.
52
-
53
-</details>
54
-
55
-<details>
56
-<summary>See more on systemd-swap</summary>
57
-The following text originates from the systemd.swap man page.<sup>[2](https://www.freedesktop.org/software/systemd/man/systemd.swap.html) </sup>
58
-
59
-A unit configuration file whose name ends in `.swap` encodes information about a swap device or
60
-file for memory paging controlled and supervised by systemd. Swap units must be named after the
61
-devices or files they control. For instance, the swap device `/dev/sda5` must be configured in a
62
-unit file`dev-sda5.swap`. Note that swap units cannot be templated, nor is possible to add multiple
63
-names to a swap unit by creating additional symlinks to it.
64
-Swap units may either be configured via unit files, or via `/etc/fstab` (see `man fstab(5)` for
65
-details). Swaps listed in `/etc/fstab` will be converted into native units dynamically at boot and
66
-when the configuration of the system manager is reloaded. See `man systemd-fstab-generator` for
67
-details about the conversion.
68
-If a swap device or file is configured in both `/etc/fstab` and a unit file, the configuration in
69
-the latter takes precedence.
70
-When reading `/etc/fstab`, a few special options are understood by systemd which influence how
71
-dependencies are created for swap units. With `noauto`, the swap unit will not be added as a
72
-dependency for `swap.target`. This means that it will not be activated automatically during boot,
73
-unless it is pulled in by some other unit. The auto option has the opposite meaning and is the
74
-default. With `nofail`, the swap unit will be only wanted, not required by `swap.target`. This means
75
-that the boot will continue even if this swap device is not activated
76
-successfully.
77
-
78
-</details>
79
-
80
-
81
-<details>
82
-<summary>References and source</summary>
83
-
84
-1. [Systemd on Wikipedia](https://en.wikipedia.org/wiki/Systemd)
85
-2. [Man page for systemd.swap](https://www.freedesktop.org/software/systemd/man/systemd.swap.html)
86
-
87
-</details>
88
-
89
-### Troubleshooting section:
90
-
91
-<details>
92
-<summary>General approach</summary>
93
-
94
-Check the log messages for failing reasons:
95
-
96
- ```
97
- root@netdata # journalctl -xe | grep -A 5 -B 5 swap
98
- ```
99
-
100
-</details>
101
-
102
-
103
-<details>
104
-<summary>Check your fstab for errors</summary>
105
-
106
-Open the fstab config file and verify the syntax of the fstab entries with `TYPE=swap` are
107
- correct.
108
-
109
- ```
110
- root@netdata # vim /etc/fstab
111
- ```
112
-
113
- Consult the [man pages of fstab](https://www.man7.org/linux/man-pages/man5/fstab.5.html) for
114
- misconfigurations.
115
-
116
-</details>
health/guides/systemdunits/systemd_target_units_state.md
deleted
-103
@@ -1,103 +0,0 @@
1
-# systemd_target_units_state
2
-
3
-**Linux | Systemd units**
4
-
5
-_Systemd is a suite of basic building blocks for a Linux system. It provides a system and service
6
-manager that runs as PID 1 and starts the rest of the system._
7
-
8
-The Netdata Agent monitors the systemd target units state. Receiving this alert indicates that one
9
-or more of the systemd target units are in the failed state. A systemd target unit "failed" when
10
-the service process returned error code on exit, or crashed, an operation timed out, or after
11
-too many restarts. The cause of a failed states is stored in a log.
12
-
13
-<details>
14
-<summary>Read more about systemd</summary>
15
-
16
-Here is some useful information about systemd from
17
-wikipedia <sup>[1](https://en.wikipedia.org/wiki/Systemd) </sup>
18
-
19
-Systemd includes features like on-demand starting of daemons, snapshot support, process tracking,
20
-and Inhibitor Locks. Systemd is not just the name of the `init` daemon, but also refers to the
21
-entire software bundle around it, which, in addition to the `systemd` `init` daemon, includes the
22
-daemons
23
-`journald`, `logind` and `networkd`, and many other low-level components. In January 2013,
24
-Poettering described systemd not as one program, but rather a large software suite that includes 69
25
-individual binaries. As an integrated software suite, systemd replaces the startup sequences and
26
-runlevels controlled by the traditional `init` daemon, along with the shell scripts executed under
27
-its control. systemd also integrates many other services that are common on Linux systems by
28
-handling user logins, the system console, device hotplugging, scheduled execution (replacing cron),
29
-logging, hostnames and locales.
30
-
31
-Like the `init` daemon, `systemd` is a daemon that manages other daemons, which, including `systemd`
32
-itself, are background processes. `systemd` is the first daemon to start during booting and the last
33
-daemon to terminate during shutdown. The `systemd` daemon serves as the root of the user space's
34
-process tree. The first process (`PID1`) has a special role on Unix systems, as it replaces the
35
-parent of a process when the original parent terminates. Therefore, the first process is
36
-particularly well suited for the purpose of monitoring daemons.
37
-
38
-Systemd executes elements of its startup sequence in parallel, which is theoretically faster than
39
-the traditional startup sequence approach. For inter-process communication (IPC), `systemd` makes
40
-Unix domain sockets and D-Bus available to the running daemons. The state of systemd itself can also
41
-be preserved in a snapshot for future recall.
42
-
43
-Systemd's core components include the following:
44
-
45
-- `systemd` is a system and service manager for Linux operating systems.
46
-
47
-- `systemctl` is a command to introspect and control the state of the systemd system and service
48
- manager. Not to be confused with sysctl.
49
-
50
-- `systemd-analyze` may be used to determine system boot-up performance statistics and retrieve
51
- other state and tracing information from the system and service manager.
52
-
53
-</details>
54
-
55
-<details>
56
-<summary>See more on systemd.target units</summary>
57
-
58
-> Target units file ends with the `.target` file extension and their only purpose is to group together
59
-other systemd units through a chain of dependencies. For example, the graphical.target unit, which
60
-is used to start a graphical session, starts system services such as the GNOME Display Manager (
61
-gdm.service) or Accounts Service (accounts-daemon .service) and also activates the multi-user.target
62
-unit. Similarly, the multi-user.target unit starts other essential system services such as
63
-NetworkManager (NetworkManager.service) or D-Bus (dbus.service) and activates another target unit
64
-named
65
-basic.target. <sup> [2](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/working-with-systemd-targets_configuring-basic-system-settings) </sup>
66
->
67
-> Among other things, target units are a more flexible replacement for SysV runlevels in the classic
68
-SysV init system. For compatibility reasons special target units such as runlevel3.target exist
69
-which are used by the SysV runlevel compatibility code in systemd.
70
-
71
-</details>
72
-
73
-
74
-<details>
75
-<summary>References and source</summary>
76
-
77
-1. [systemd on wikipedia](https://en.wikipedia.org/wiki/Systemd)
78
-2. [systemd.target explained on Redhat's documentantion](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/working-with-systemd-targets_configuring-basic-system-settings)
79
-
80
-</details>
81
-
82
-### Troubleshooting section:
83
-
84
-<details>
85
-<summary>General approach</summary>
86
-
87
-When a target is in a failed state, you should always try to gather more information about it.
88
-
89
-1. Identify which target units fail. Open the Netdata dashboard, find the current active alarms under
90
- the [active alarms](https://learn.netdata.cloud/docs/monitor/view-active-alarms) tab and look
91
- into its chart.
92
- (`systemdunits_target_units.target_unit_state`). In this chart, identify which target
93
- units are in state with value 5
94
-
95
-2. Gather more information about the failing target unit
96
-
97
- ```
98
- root@netdata~ # systemctl status <target_name>.target
99
- ```
100
-
101
-3. Check the log messages from the command of step 2.
102
-
103
-</details>
health/guides/tcp_conn/tcp_connections.md
deleted
-31
@@ -1,31 +0,0 @@
1
-# tcp_connections
2
-
3
-## OS: Linux
4
-
5
-This alert presents the percentage of used IPv4 TCP connections. If you receive it, it is an
6
-indication of high IPv4 TCP connections utilization.
7
-
8
-If this value is 100% then the system is no longer able to establish new TCP connections.
9
-
10
-<details>
11
-
12
-<summary>TCP Connections Alarm Settings</summary>
13
-
14
-Inside the [tcp_conn.conf](
15
-https://github.com/netdata/netdata/blob/master/health/health.d/tcp_conn.conf), on the `calc:` line,
16
-there is this block of code:
17
-`(${tcp_max_connections} > 0) ? ( ${connections} * 100 / ${tcp_max_connections} ) : 0`
18
-
19
-- That line of code will calculate the value of `$this` in the following lines.
20
-- Essentially, if the max connections are not dynamic, and there is a limit, then we calculate the
21
- percentage of used IPv4 TCP connections. Otherwise, we have a dynamic threshold *(
22
- so `$ {tcp_max_connections}` may be nan or -1)*, which in this case the alert and `$this` will
23
- always be zero.
24
-
25
-</details>
26
-
27
-<br>
28
-
29
-- This alert is raised to a state of warning when the percentage of used IPv4 TCP connections is
30
-greater than 80% and less than 90%.
31
-- If the percentage of used IPv4 TCP connections exceeds 90%, then the alert gets raised to critical.
health/guides/tcp_connections.md
new
+51
@@ -0,0 +1,51 @@
1
+### Understand the alert
2
+
3
+This alert is related to the percentage of used IPv4 TCP connections. If you receive this alert, it means that your system has high TCP connections utilization, and you might be approaching the limit of maximum connections.
4
+
5
+### What does high IPv4 TCP connections utilization mean?
6
+
7
+When the number of IPv4 TCP connections gets too high, the system's ability to establish new connections decreases. This is because there are limitations due to resources such as memory or system settings. High utilization could lead to connection-related issues or service interruptions.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check current TCP connections:
12
+
13
+ To see the current number of TCP connections, you can use the `ss` or `netstat` command:
14
+
15
+ ```
16
+ ss -t | grep ESTAB | wc -l
17
+ ```
18
+
19
+ or
20
+
21
+ ```
22
+ netstat -ant | grep ESTABLISHED | wc -l
23
+ ```
24
+
25
+2. Identify connections with high usage:
26
+
27
+ To list the connections with their state (e.g., ESTABLISHED, LISTEN), use the following command:
28
+
29
+ ```
30
+ ss -tan
31
+ ```
32
+
33
+ Look for connections with a high number of ESTABLISHED connections, as these may be contributing to the high utilization.
34
+
35
+3. Inspect running processes to identify potential culprits:
36
+
37
+ You can use the `lsof` command to list all open files and the processes that are using them:
38
+
39
+ ```
40
+ sudo lsof -iTCP
41
+ ```
42
+
43
+ Look for processes with a high number of open files, as these are likely responsible for the increased TCP connections utilization.
44
+
45
+4. Take action:
46
+
47
+ Once you have identified the processes contributing to high TCP connections utilization, you can take appropriate action. This may involve optimizing the application, adjusting system settings, or optimizing hardware resources.
48
+
49
+### Useful resources
50
+
51
+1. [Linux lsof command tutorial](https://www.howtoforge.com/linux-lsof-command/)
health/guides/tcp_listen/1m_tcp_accept_queue_drops.md
deleted
-52
@@ -1,52 +0,0 @@
1
-# 1m_tcp_accept_queue_drops
2
-
3
-## OS: Linux
4
-
5
-This alert presents the average number of dropped packets in the TCP accept queue over the last
6
-sixty seconds. If it is raised, then the system is dropping incoming TCP connections. This could also be
7
-an indication of accepted queue overflow, low memory, security issues, no route to a destination,
8
-etc.
9
-- This alert gets raised to warning when the value is greater than 1 and less than 5.
10
-- If the number of queue drops over the last minute exceeds 5, then the alert gets raised to critical.
11
-
12
-
13
-<details>
14
-<summary>TCP Accept Queue Drops</summary>
15
-
16
-The accept queue holds fully established TCP connections waiting to be handled by the listening
17
-application. It overflows when the server application fails to accept new connections at the rate
18
-they are coming in.
19
-
20
-</details>
21
-
22
-<details>
23
- <summary>References and sources</summary>
24
-
25
-1. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
26
-2. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
27
-
28
-</details>
29
-
30
-### Troubleshooting Section
31
-
32
-<details>
33
-<summary>Check for queue overflows</summary>
34
-
35
-If you receive this alert, then you can cross-check its results with the
36
-`1m_tcp_accept_queue_overflows` alert. If that alert is also in a warning or critical state,
37
-then the system is experiencing accept queue overflowing. To fix that you can do the following:
38
-
39
-1. Open the /etc/sysctl.conf file and look for the entry " net.ipv4.tcp_max_syn_backlog".
40
- > The `tcp_max_syn_backlog` is the maximal number of remembered connection requests
41
- > (SYN_RECV), which have not received an acknowledgment from connecting client. <sup> [1](
42
- > https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt) </sup>
43
-2. If the entry does not exist, then append the following default entry to the
44
- file; `net.ipv4.tcp_max_syn_backlog=1280`. Otherwise, adjust the limit to suit your needs.
45
-3. Save your changes and run;
46
- ```
47
- root@netdata~ #sysctl -p
48
- ```
49
- to apply the changes.
50
-
51
-> Note: Netdata strongly suggests knowing exactly what values you need before making system changes.
52
-</details>
health/guides/tcp_listen/1m_tcp_accept_queue_overflows.md
deleted
-54
@@ -1,54 +0,0 @@
1
-# 1m_tcp_accept_queue_overflows
2
-
3
-## OS: Linux
4
-
5
-This alert presents the average number of overflows in the TCP accept queue over the last minute.
6
-
7
-- This alert gets raised in a warning state when the value is greater than 1 and less than 5.
8
-- If the overflow average exceeds 5 in the last minute, then the alert gets raised in the critical
9
- state.
10
-
11
-<details>
12
-<summary>What is the Accept queue</summary>
13
-
14
-The accept queue holds fully established TCP connections waiting to be handled by the listening
15
-application. It overflows when the server application fails to accept new connections at the rate
16
-they are coming in.
17
-
18
-</details>
19
-
20
-This alert might also indicate a SYN flood;
21
-> A SYN flood is a form of denial-of-service attack in which an attacker rapidly initiates a
22
-> connection to a server without finalizing the connection. The server has to spend resources
23
-> waiting for half-opened connections, which can consume enough resources to make the system
24
-> unresponsive to legitimate traffic. <sup> [1](https://en.wikipedia.org/wiki/SYN_flood) </sup>
25
-
26
-
27
-<details>
28
- <summary>References and sources</summary>
29
-
30
-1. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
31
-2. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
32
-3. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
33
-
34
-</details>
35
-
36
-### Troubleshooting Section
37
-
38
-<details>
39
-<summary>Increase the queue length</summary>
40
-
41
-1. Open the /etc/sysctl.conf file and look for the entry " net.ipv4.tcp_max_syn_backlog".
42
- > The `tcp_max_syn_backlog` is the maximal number of remembered connection requests
43
- > (SYN_RECV), which have not received an acknowledgment from connecting client. <sup> [2](
44
- > https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt) </sup>
45
-2. If the entry does not exist, you can append the following default entry to the
46
- file; `net.ipv4. tcp_max_syn_backlog=1280`. Otherwise, adjust the limit to suit your needs.
47
-3. Save your changes and run;
48
- ```
49
- root@netdata~ #sysctl -p
50
- ```
51
- to apply the changes.
52
-
53
-Netdata strongly suggests knowing exactly what values you need before making system changes.
54
-</details>
\ No newline at end of file
health/guides/tcp_listen/1m_tcp_syn_queue_cookies.md
deleted
-87
@@ -1,87 +0,0 @@
1
-# 1m_tcp_syn_queue_cookies
2
-
3
-## OS: Linux
4
-
5
-This alert presents the average number of sent SYN cookies due to the full TCP SYN queue over the
6
-sixty seconds. Receiving this means that the incoming traffic is excessive. SYN queue cookies are
7
-used to resist any potential SYN flood attacks.
8
-
9
-This alert is raised to warning when the average exceeds 1 and will enter critical when the
10
-value exceeds an average of 5 sent SYN cookies in sixty seconds.
11
-
12
-<details>
13
-<summary>SYN Queue Cookies</summary>
14
-
15
-> The SYN Queue stores inbound SYN packets (specifically: struct inet_request_sock). It is
16
-> responsible for sending out SYN+ACK packets and retrying them on timeout. \
17
-> After transmitting the SYN+ACK, the SYN Queue waits for an ACK packet from the client - the last
18
-> packet in the three-way-handshake. All received ACK packets must first be matched against the
19
-> fully established connection table, and only then against data in the relevant SYN Queue. On
20
-> SYN Queue match, the kernel removes the item from the SYN Queue, successfully creates a full
21
-> connection (specifically: struct inet_sock), and adds it to the Accept Queue.<sup> [1](
22
-> https://blog.cloudflare.com/syn-packet-handling-in-the-wild/) </sup>
23
-</details>
24
-
25
-<br>
26
-
27
-<details>
28
-<summary>This alert likely indicates a SYN flood.</summary>
29
-
30
-> A SYN flood is a form of denial-of-service attack in which an attacker rapidly initiates a
31
-> connection to a server without finalizing the connection. The server has to spend resources
32
-> waiting for half-opened connections, which can consume enough resources to make the system
33
-> unresponsive to legitimate traffic. <sup> [2](https://en.wikipedia.org/wiki/SYN_flood) </sup>
34
-</details>
35
-
36
-<br>
37
-
38
-<details>
39
-<summary>SYN Cookies</summary>
40
-
41
-> SYN cookies are a technique used to resist IP address spoofing attacks. The technique's primary
42
-> inventor, Daniel J. Bernstein, defines SYN cookies as "particular choices of initial TCP sequence
43
-> numbers by TCP servers." In particular, the use of SYN cookies allows a server to avoid dropping
44
-> connections when the SYN queue fills up. Instead of storing additional connections, a SYN queue
45
-> entry is encoded into the sequence number sent in the SYN+ACK response. If the server then
46
-> receives a subsequent ACK response from the client with the incremented sequence number, then the
47
-> server is able to reconstruct the SYN queue entry using information encoded in the TCP
48
-> sequence number and proceed as usual with the connection. <sup> [3](
49
-> https://en.wikipedia.org/wiki/SYN_cookies) </sup>
50
-</details>
51
-
52
-<br>
53
-
54
-<details>
55
- <summary>References and sources</summary>
56
-
57
-1. [SYN packet handling](https://blog.cloudflare.com/syn-packet-handling-in-the-wild/)
58
-2. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
59
-3. [SYN Cookies](https://en.wikipedia.org/wiki/SYN_cookies)
60
-4. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
61
-5. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
62
-
63
-</details>
64
-
65
-### Troubleshooting Section
66
-
67
-<details>
68
-<summary>If the traffic is legitimate, then increase the limit of the SYN queue.</summary>
69
-
70
-If you can determine that the traffic is legitimate, consider expanding the limit of the SYN
71
-queue through configuration; \
72
-*(If the traffic is not legitimate, then this is not safe! You will
73
-expose more resources to an attacker if the traffic is not legitimate.)*
74
-
75
-1. Open the /etc/sysctl.conf file and look for the entry "net.core.somaxconn". This value will
76
- affect both SYN and accept queue limits on newer Linux systems.
77
-2. Set the value accordingly (By default it is set to 128) `net.core.somaxconn=128` (if the value
78
- doesn't exist, append it to the file)
79
-3. Save your changes and run;
80
- ```
81
- root@netdata~ #sysctl -p
82
- ```
83
- to apply the changes.
84
-
85
-> Note: Netdata strongly suggests knowing exactly what you are configuring before making system
86
-> changes.
87
-</details>
health/guides/tcp_listen/1m_tcp_syn_queue_drops.md
deleted
-82
@@ -1,82 +0,0 @@
1
-# 1m_tcp_syn_queue_drops
2
-
3
-## OS: Linux
4
-
5
-This alert presents the average number of SYN requests that were dropped due to the TCP SYN queue
6
-being full over the last sixty seconds. Receiving this means that the TCP SYN queue is full and that the
7
-system is dropping incoming TCP SYN requests.
8
-
9
-The alert is raised to warning when this average exceeds 1 per minute. The alert is raised to critical
10
-when the value exceeds an average of 5 dropped SYN packets per sixty seconds.
11
-
12
-<details>
13
-<summary>What is the SYN queue?</summary>
14
-
15
-> The SYN Queue stores inbound SYN packets (specifically: struct inet_request_sock). It's
16
-> responsible for sending out SYN+ACK packets and retrying them on timeout. \
17
-> After transmitting the SYN+ACK, the SYN Queue waits for an ACK packet from the client - the last
18
-> packet in the three-way-handshake. All received ACK packets must first be matched against the
19
-> fully established connection table, and only then against data in the relevant SYN Queue. On
20
-> SYN Queue match, the kernel removes the item from the SYN Queue, happily creates a fully fledged
21
-> connection (specifically: struct inet_sock), and adds it to the Accept Queue.<sup> [1](
22
-> https://blog.cloudflare.com/syn-packet-handling-in-the-wild/) </sup>
23
-</details>
24
-
25
-<br>
26
-
27
-<details>
28
-<summary>This alert has a high probability of indicating a SYN flood.</summary>
29
-
30
-> A SYN flood is a form of denial-of-service attack in which an attacker rapidly initiates a
31
-> connection to a server without finalizing the connection. The server has to spend resources
32
-> waiting for half-opened connections, which can consume enough resources to make the system
33
-> unresponsive to legitimate traffic. <sup> [2](https://en.wikipedia.org/wiki/SYN_flood) </sup>
34
-
35
-</details>
36
-
37
-<br>
38
-
39
-<details>
40
-<summary>SYN Cookies</summary>
41
-
42
-> SYN cookies are a technique used to resist IP address spoofing attacks. The technique's primary
43
-> inventor, Daniel J. Bernstein, defines SYN cookies as "particular choices of initial TCP sequence
44
-> numbers by TCP servers." In particular, the use of SYN cookies allows a server to avoid dropping
45
-> connections when the SYN queue fills up. Instead of storing additional connections, a SYN queue
46
-> entry is encoded into the sequence number sent in the SYN+ACK response. If the server then
47
-> receives a subsequent ACK response from the client with the incremented sequence number, the
48
-> server is able to reconstruct the SYN queue entry using information encoded in the TCP
49
-> sequence number and proceed as usual with the connection. <sup> [3](
50
-> https://en.wikipedia.org/wiki/SYN_cookies) </sup>
51
-</details>
52
-<br>
53
-
54
-<details>
55
- <summary>References and sources</summary>
56
-
57
-1. [SYN packet handling](https://blog.cloudflare.com/syn-packet-handling-in-the-wild/)
58
-2. [SYN Floods](https://en.wikipedia.org/wiki/SYN_flood)
59
-3. [SYN Cookies](https://en.wikipedia.org/wiki/SYN_cookies)
60
-4. [ip-sysctl.txt](https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
61
-5. [Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)
62
-
63
-</details>
64
-
65
-### Troubleshooting Section
66
-
67
-<details>
68
-<summary>Enable SYN Cookies</summary>
69
-
70
-If you can determine that the traffic is legitimate, consider enabling SYN cookies. To enable SYN cookies:
71
-
72
-1. Open the /etc/sysctl.conf file and look for the entry "net.ipv4.tcp_syncookies".
73
-2. Set the value to 1. `net.ipv4.tcp_syncookies=1` (if the value doesn't exist, append it to the
74
- file)
75
-3. Save your changes and run;
76
- ```
77
- root@netdata~ #sysctl -p
78
- ```
79
- to apply the changes.
80
-
81
-Netdata strongly suggests knowing exactly what you are configuring before making system changes.
82
-</details>
health/guides/tcp_mem/tcp_memory.md
deleted
-111
@@ -1,111 +0,0 @@
1
-# tcp_memory
2
-
3
-## OS: Linux
4
-
5
-The Netdata Agent calculates the percentage of used TCP memory. Receiving this alert indicates that
6
-the TCP memory utilization uses more memory than the limit.
7
-
8
-By default, the Linux network stack is not configured for high speed large file transfer across WAN
9
-links. This is done to save memory resources. System performs out of memory checks, if the memory
10
-used by the TCP protocol is higher than the third value(max) of the `net.ipv4.tcp_mem`, it throws
11
-OOM error. This will make some applications to become unresponsive.
12
-
13
-
14
-
15
-<details>
16
-<summary>See more about TCP buffers </summary>
17
-
18
-There are 3 main parameters to configure regarding the TCP buffers.
19
-
20
-> - `tcp_mem` is a vector of 3 integers: `"low, pressure, high"`. These bounds, measured in units of
21
- the system page size, are used by TCP to track its memory usage. The defaults are calculated at
22
- boot time from the amount of available memory. (TCP can only use low memory for this, which is
23
- limited to around 900 megabytes on 32-bit systems. 64-bit systems do not suffer this limitation.)
24
-
25
-> - low TCP doesn't regulate its memory allocation when the number of pages it has allocated
26
- globally is below this number.
27
-> - pressure When the amount of memory allocated by TCP exceeds this number of pages, TCP
28
- moderates its memory consumption. This memory pressure state is exited once the number of
29
- pages allocated falls below the low mark.
30
-> - high The maximum number of pages, globally, that TCP will allocate. This value overrides any
31
- other limits imposed by the kernel. <sup>[1](https://man7.org/linux/man-pages/man7/tcp.7.html) </sup>
32
-
33
-The min/pressure/max TCP buffer space are automatically set in `/proc/sys/net/ipv4/tcp_mem` during
34
-the boot time based on available RAM size.
35
-
36
-> - `net.ipv4.tcp_rmem` contains three values that represent `"minimum default maximum_size` of the
37
- TCP socket receive buffer.
38
-
39
-> - The minimum represents the smallest receive buffer size guaranteed, even under memory
40
- pressure. The minimum value defaults to 1 page or 4096 bytes.
41
-
42
-> - The default value represents the initial size of a TCP sockets receive buffer. This value
43
- supersedes net.core.rmem_default used by other protocols. The default value for this setting
44
- is 87380 bytes. It also sets the tcp_adv_win_scale and initializes the TCP window size to
45
- 65535 bytes.
46
-
47
-> - The maximum represents the largest receive buffer size automatically selected for TCP sockets.
48
- This value does not override net.core.rmem_max. The default value for this setting is
49
- somewhere between 87380 bytes and 6M bytes based on the amount of memory in the system.
50
->
51
->
52
-> The recommendation is to use the maximum value of 16M bytes or higher (kernel level dependent)
53
- especially for 10 Gigabit adapters.
54
->
55
->
56
-> - `net.ipv4.tcp_wmem` parameter also consists of 3 values `"minimum default maximum"`.
57
-> - The minimum represents the smallest receive buffer size a newly created socket is entitled to
58
- as part of its creation. The minimum value defaults to 1 page or 4096 bytes.
59
-> - The default value represents the initial size of a TCP sockets receive buffer. This value
60
- supersedes net.core.rmem_default used by other protocols. It is typically set lower than
61
- net.core.wmem_default. The default value for this setting is 16K bytes.
62
-> - The maximum represents the largest receive buffer size for auto-tuned send buffers for TCP
63
- sockets. This value does not override net.core.rmem_max. The default value for this setting is
64
- somewhere between 64K bytes and 4M bytes based on the amount of memory available in the
65
- system.
66
->
67
->
68
->The recommendation is to use the maximum value of 16M bytes or higher (kernel level dependent)
69
-especially for 10 Gigabit adapters. <sup>[2](https://www.ibm.com/docs/en/linux-on-systems?topic=tuning-tcpip-ipv4-setting) </sup>
70
-
71
-</details>
72
-
73
-<details>
74
-<summary>References and sources</summary>
75
-
76
-1. [man pages of tcp](https://man7.org/linux/man-pages/man7/tcp.7.html)
77
-1. [Adjustments for IPv4 settings from IBM](https://www.ibm.com/docs/en/linux-on-systems?topic=tuning-tcpip-ipv4-settings)
78
-</details>
79
-
80
-### Troubleshooting section:
81
-
82
-<details>
83
-<summary>Increase the TCP memory </summary>
84
-
85
-Increasing the TCP memory available in the Linux network stack may resolve this issue.
86
-
87
-1. Try to increase the `tcp_mem` bounds
88
-
89
- ```
90
- root@netdata # sysctl -w net.ipv4.tcp_mem="819200 1091174 1638400"
91
- ```
92
-
93
-1. Verify the change and test it with the same workload that triggered the alarm originally.
94
- If the problem still exists, you can always consider increase it more.
95
-
96
- ```
97
- root@netdata~ # sysctl net.ipv4.tcp_mem
98
- net.ipv4.tcp_mem=819200 1091174 1638400
99
- ```
100
-
101
-1. If this change works for your system, you could make it permanently. Bump these entries
102
- under `/etc/sysctl.conf`
103
-
104
-
105
-1. Reload the sysctl settings.
106
-
107
- ```
108
- root@netdata~ # sysctl -p
109
- ```
110
-
111
-
health/guides/tcp_memory.md
new
+50
@@ -0,0 +1,50 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the TCP memory usage on your system is higher than the allowed limit. High TCP memory utilization can cause applications to become unresponsive and result in poor system performance.
4
+
5
+### Troubleshoot the alert
6
+
7
+To resolve the TCP memory alert, you can follow these steps:
8
+
9
+1. Verify the current TCP memory usage:
10
+
11
+ Check the current values of TCP memory buffers by running the following command:
12
+
13
+ ```
14
+ cat /proc/sys/net/ipv4/tcp_mem
15
+ ```
16
+
17
+ The output consists of three values: low, pressure (memory pressure), and high (memory limit).
18
+
19
+2. Monitor system performance:
20
+
21
+ Use the `vmstat` command to monitor the system's performance and understand the memory consumption in detail:
22
+
23
+ ```
24
+ vmstat 5
25
+ ```
26
+
27
+ This will display the system's statistics every 5 seconds. Pay attention to the `si` and `so` columns, which represent swap-ins and swap-outs. High values in these columns may indicate memory pressure on the system.
28
+
29
+3. Identify high memory-consuming processes:
30
+
31
+ Use the `top` command to identify processes that consume the most memory:
32
+
33
+ ```
34
+ top -o %MEM
35
+ ```
36
+
37
+ Look for processes with high memory usage and determine if they are necessary for your system. If they are not, consider stopping or killing these processes to free up memory.
38
+
39
+4. Increase the TCP memory:
40
+
41
+ Follow the steps mentioned in the provided guide to increase the TCP memory. This includes:
42
+
43
+ - Increase the `tcp_mem` bounds using the `sysctl` command.
44
+ - Verify the change and test it with the same workload that triggered the alarm originally.
45
+ - If the change works, make it permanent by adding the new values to `/etc/sysctl.conf`.
46
+ - Reload the sysctl settings with `sysctl -p`.
47
+
48
+### Useful resources
49
+
50
+1. [man pages of tcp](https://man7.org/linux/man-pages/man7/tcp.7.html)
health/guides/tcp_orphans.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+This alert indicates that your system is experiencing high IPv4 TCP socket utilization, specifically orphaned sockets. Orphaned connections are those not attached to any user file handle. When these connections exceed the limit, they are reset immediately. The warning state is triggered when the percentage of used orphan IPv4 TCP sockets exceeds 25%, and the critical state is triggered when the value exceeds 50%.
4
+
5
+### Troubleshoot the alert
6
+
7
+- Check the current orphan socket usage
8
+
9
+To check the number of orphan sockets in your system, run the following command:
10
+
11
+ ```
12
+ cat /proc/sys/net/ipv4/tcp_max_orphans
13
+ ```
14
+
15
+- Identify the processes causing high orphan socket usage
16
+
17
+To identify the processes causing high orphan socket usage, you can use the `ss` command:
18
+
19
+ ```
20
+ sudo ss -tan state time-wait state close-wait
21
+ ```
22
+
23
+ Look for connections with a large number of orphan sockets and investigate the related processes.
24
+
25
+- Increase the orphan socket limit
26
+
27
+If you need to increase the orphan socket limit to accommodate legitimate connections, you can update the value in the `/proc/sys/net/ipv4/tcp_max_orphans` file. Replace `{DESIRED_AMOUNT}` with the new limit:
28
+
29
+ ```
30
+ echo {DESIRED_AMOUNT} > /proc/sys/net/ipv4/tcp_max_orphans
31
+ ```
32
+
33
+ Consider the kernel's penalty factor for orphan sockets (usually 2x or 4x) when determining the appropriate limit.
34
+
35
+ **Note**: Be cautious when making system changes and ensure you understand the implications of updating these settings.
36
+
37
+- Review and optimize application behavior
38
+
39
+Investigate the applications generating a high number of orphan sockets and consider optimizing their behavior. This may involve updating application settings or code to better manage network connections.
40
+
41
+- Monitor your system
42
+
43
+Keep an eye on your system's orphan socket usage, particularly during peak hours. Adjust the limit as needed to accommodate legitimate connections.
44
+
45
+### Useful resources
46
+
47
+1. [Network Sockets](https://en.wikipedia.org/wiki/Network_socket)
48
+2. [Linux-admins.com - Troubleshooting Out of Socket Memory](http://www.linux-admins.net/2013/01/troubleshooting-out-of-socket-memory.html)
\ No newline at end of file
health/guides/tcp_orphans/tcp_orphans.md
deleted
-82
@@ -1,82 +0,0 @@
1
-# tcp_orphans
2
-
3
-## OS: Linux
4
-
5
-This alert presents the percentage of used orphan IPV4 TCP sockets. If it is raised, it indicates
6
-that your system is experiencing high IPv4 TCP sockets utilization.
7
-When the system exceeds the limit, orphaned connections (connections not attached to any user filehandle) are
8
-reset immediately.
9
-
10
-This alert is triggered in warning state when the percentage of used orphan IPv4 TCP sockets is
11
-above 25% and in critical state when that value exceeds 50%.
12
-
13
-
14
-<Details>
15
-<summary>What is a network socket</summary>
16
-
17
-> A network socket is a software structure within a network node of a computer network that
18
-> serves as an endpoint for sending and receiving data across the network. The structure and
19
-> properties of a socket are defined by an application programming interface (API) for the
20
-> networking architecture. Sockets are created only during the lifetime of a process of an
21
-> application running in the node.
22
-
23
-> Because of the standardization of the TCP/IP protocols in the development of the Internet, the
24
-> term network socket is most commonly used in the context of the Internet protocol suite, and
25
-> is therefore often also referred to as Internet socket.
26
-
27
-> In this context, a socket is externally identified to other hosts by its socket address,
28
-> which is the triad of transport protocol, IP address, and port number.
29
-> <sup>[1](https://en.wikipedia.org/wiki/Network_socket) </sup>
30
-
31
-</Details>
32
-
33
-<br>
34
-
35
-<details>
36
-<summary>What is a "filehandle" or "socket descriptor"</summary>
37
-
38
-> The application programming interface (API) for the network protocol stack creates a handle
39
-> for each socket created by an application, commonly referred to as a socket descriptor. In
40
-> Unix-like operating systems, this descriptor is a type of file descriptor. It is stored by
41
-> the application process for use with every read and write operation on the communication channel.
42
-> <sup>[1](https://en.wikipedia.org/wiki/Network_socket) </sup>
43
-
44
-</details>
45
-
46
-<br>
47
-
48
-> An orphan socket is a socket that isn't associated with a file descriptor, usually after the
49
-> close() call and there is no longer a file descriptor that reference it, but the socket still
50
-> exists in memory, until TCP is done with it.<sup> [2](
51
-> http://www.linux-admins.net/2013/01/troubleshooting-out-of-socket-memory.html) </sup>
52
-
53
-<br>
54
-
55
-<details>
56
-<summary> References and Sources </summary>
57
-
58
-1. [Network_sockets](https://en.wikipedia.org/wiki/Network_socket)
59
-2. [Linux-admins.com](http://www.linux-admins.net/2013/01/troubleshooting-out-of-socket-memory.html)
60
-</sup>
61
-
62
-</details>
63
-
64
-### Troubleshooting Section
65
-
66
-<details>
67
-<summary>Increase the orphan socket limit</summary>
68
-
69
-To counteract this behavior, you can increase the limit in the
70
-file: `/proc/sys/net/ipv4/tcp_max_orphans`. Simply run:
71
-
72
-```
73
-root@netdata~ # echo {DESIRED_AMOUNT} > /proc/sys/net/ipv4/tcp_max_orphans
74
-```
75
-
76
-The kernel may penalize orphans by 2x or even 4x (hence the small warning and critical thresholds).
77
-You may need to watch for orphaned sockets during peak hours and consider multiplying that number by
78
-3 or 4. That should give you a good starting point.
79
-
80
-> Note: Netdata strongly suggests knowing exactly what you are configuring before making system
81
-> changes.
82
-</details>
health/guides/tcp_resets/10s_ipv4_tcp_resets_received.md
deleted
-142
@@ -1,142 +0,0 @@
1
-# 10s_ipv4_tcp_resets_received
2
-
3
-## OS: Linux, FreeBSD
4
-
5
-TCP reset is an abrupt closure of the session. It causes the resources allocated to the connection
6
-to be immediately released and all other information about the connection to be erased.
7
-
8
-The Netdata Agent monitors the average number of received TCP RESETS over the last 10 seconds. This
9
-can indicate that the system is trying to establish a connection to a server port on which no
10
-process is listening. This can also indicate a SYN reset attack.
11
-
12
-<details>
13
- <summary>See more about TCP Resets </summary>
14
-
15
-TCP uses a three-way handshake to establish a reliable connection. The connection is full duplex,
16
-and both sides synchronize (SYN) and acknowledge (ACK) each other. The exchange of these four flags
17
-is performed in three steps: SYN, SYN-ACK, and ACK
18
-
19
-When an unexpected TCP packet arrives at a host, that host usually responds by sending a reset
20
-packet back on the same connection. A reset packet is one with no payload and with the RST bit set
21
-in the TCP header flags. There are a few circumstances in which a TCP packet might not be expected.
22
-The most common cases are:
23
-
24
- 1. A TCP packet received in a non-existed TCP PORT
25
-
26
-1. An aborting connection
27
-
28
-1. Half opened connections
29
-
30
-1. Time wait assassination
31
-
32
-1. Listening endpoint Queue is Full
33
-
34
-1. A TCP Buffer Overflow
35
-
36
-Basically, A TCP Reset usually occurs when a system receives data which doesn't agree with its view
37
-of the connection.
38
-
39
-</details>
40
-
41
-
42
-<details>
43
-
44
- <summary>References and source:</summary>
45
-
46
-1. [TCP reset explanation](https://www.pico.net/kb/what-is-a-tcp-reset-rst/)
47
-1. [TCP 3-way handshake on wikipedia](https://en.wikipedia.org/wiki/Handshaking)
48
-
49
-
50
-</details>
51
-
52
-### Troubleshooting section:
53
-
54
-<details>
55
-
56
- <summary>General approach</summary>
57
-
58
-Try using Wireshark to inspect the network packets.
59
-
60
-Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting,
61
-analysis, software and communications protocol development.
62
-
63
-[See more about Wireshark here](https://www.wireshark.org/)
64
-
65
-Since you might won't be able to probe your traffic with wireshark in your host machine, You can
66
-export it in a dump file and analyze it in a second iteration.
67
-
68
-1. Try to export the traffic in your host with `tcpdump`.
69
-
70
- ```
71
- root@netdata # tcpdump -i any 'tcp[tcpflags] & (tcp-rst) == (tcp-rst)' -s 65535 -w output.pcap
72
- ```
73
-
74
-You must stop the capture after a certain observation period (60s up to 5 minutes). This command
75
-will create a dump file which can be interpreted by Wireshark that contains all the TCP packets with
76
-RST flag set.
77
-
78
-2. Copy this file in your workstation and examine it with Wireshark.
79
-
80
-</details>
81
-
82
-
83
-<details>
84
-
85
- <summary>Counter measure on malicious TCP resets</summary>
86
-
87
-SYN cookie is a technique used to resist IP address spoofing attacks. In particular, the use of SYN
88
-cookies allows a server to avoid dropping connections when the SYN queue fills up.
89
-
90
- <details>
91
-
92
- <summary>Enable SYN cookies in Linux</summary>
93
-
94
- 1. Check if your system has the SYN cookies service enabled
95
-
96
- ```
97
- root@netdata # cat /proc/sys/net/ipv4/tcp_syncookies
98
- ```
99
- If the value is 1, then the service is enabled, if not proceed to step 2.
100
-
101
-
102
- 2. Bump this `net.ipv4.tcp_syncookies=1` value under `/etc/sysctl.conf`
103
-
104
-
105
- 3. Apply the configuration
106
-
107
- ```
108
- root@netdata # sysctl -p to apply the configuration.
109
- ```
110
-
111
- </details>
112
-
113
- <details>
114
-
115
- <summary>Enable SYN cookies in FreeBSD</summary>
116
-
117
- 1. Check if your system has the SYN cookies service enabled
118
-
119
- ```
120
- root@netdata # sysctl net.inet.tcp.syncookies_only
121
- ```
122
- If the value is 1, then the service is enabled, if not proceed to step 2.
123
-
124
-
125
- 2. Bump this `net.inet.tcp.syncookies_only=1` value under `/etc/sysctl.conf`
126
-
127
-
128
- 3. Apply the configuration
129
-
130
- ```
131
- root@netdata~ # /etc/rc.d/sysctl reload
132
- ```
133
-
134
-
135
- </details>
136
-
137
-
138
-The use of SYN cookies does not break any protocol specifications, and therefore should be
139
-compatible with all TCP implementations. There are, however, a few caveats that take effect when SYN
140
-cookies are in use.
141
-</details>
142
-
health/guides/tcp_resets/10s_ipv4_tcp_resets_sent.md
deleted
-103
@@ -1,103 +0,0 @@
1
-# 10s_ipv4_tcp_resets_sent
2
-
3
-## OS: Linux
4
-
5
-TCP reset is an abrupt closure of the session. It causes the resources allocated to the connection
6
-to be immediately released and all other information about the connection is erased.
7
-
8
-The Netdata Agent monitors the average number of sent TCP RESETS over the last 10 seconds. This can
9
-indicate a port scan or that a service running on the system has crashed. Additionally, it's a
10
-result of a high number of sent TCP RESETS. Furthermore, it can also indicate a SYN reset attack.
11
-
12
-
13
-<details>
14
- <summary>See more about TCP Resets </summary>
15
-
16
-TCP uses a three-way handshake to establish a reliable connection. The connection is full duplex,
17
-and both sides synchronize (SYN) and acknowledge (ACK) each other. The exchange of these four flags
18
-is performed in three steps: SYN, SYN-ACK, and ACK
19
-
20
-When an unexpected TCP packet arrives at a host, that host usually responds by sending a reset
21
-packet back on the same connection. A reset packet is one with no payload and with the RST bit set
22
-in the TCP header flags. There are a few circumstances in which a TCP packet might not be expected.
23
-The most common cases are:
24
-
25
-1. A TCP packet received on a port that is not open.
26
-
27
-2. An aborting connection
28
-
29
-3. Half opened connections
30
-
31
-4. Time wait assassination
32
-
33
-5. Listening endpoint Queue is Full
34
-
35
-6. A TCP Buffer Overflow
36
-
37
-Basically, A TCP Reset usually occurs when a system receives data which doesn't agree with its view
38
-of the connection.
39
-
40
-When your system cannot establish a connection it will retry by default `net.ipv4.tcp_syn_retries`
41
-times.
42
-</details>
43
-
44
-
45
-<details>
46
-
47
- <summary>References and sources</summary>
48
-
49
-1. [TCP reset explanation](https://www.pico.net/kb/what-is-a-tcp-reset-rst/)
50
-2. [TCP 3-way handshake on wikipedia](https://en.wikipedia.org/wiki/Handshaking)
51
-
52
-</details>
53
-
54
-### Troubleshooting section:
55
-
56
-<details>
57
-
58
- <summary>General approach</summary>
59
-
60
-Try using Wireshark to inspect the network packets.
61
-
62
-Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting,
63
-analysis, software and communications protocol development.
64
-
65
-[See more about Wireshark here](https://www.wireshark.org/)
66
-
67
-Since you might won't be able to probe your traffic with wireshark in your host machine, You can
68
-export it in a dump file and analyze it in a second iteration.
69
-
70
-1. Try to export the traffic in your host with `tcpdump`.
71
-
72
- ```
73
- root@netdata # tcpdump -i any 'tcp[tcpflags] & (tcp-rst) == (tcp-rst)' -s 65535 -w output.pcap
74
- ```
75
-
76
-You must stop the capture after a certain observation period (60s up to 5 minutes). This command
77
-will create a dump file which can be interpreted by Wireshark that contains all the TCP packets with
78
-RST flag set.
79
-
80
-2. Copy this file in your workstation and examine it with Wireshark.
81
-
82
-</details>
83
-
84
-<details>
85
-
86
- <summary>Identify which application sends TCP resets</summary>
87
-
88
-1. Inspect the packet flow with a packet sniffer like Wireshark. You can consult the _General
89
- approach_ troubleshooting action in the current guide.
90
-
91
-
92
-2. Check the instances of `RST` events of the TCP protocol. Wireshark also displays the ports on
93
- which the two systems tried to establish the TCP connection, (XXXXXX -> XXXXXX).
94
-
95
-
96
-3. To check which application is using this port, run the following code:
97
-
98
- ```
99
- root@netdata # lsof -i:XXXXXX -P -n
100
- ```
101
-
102
-</details>
103
-
health/guides/timex/system_clock_sync_state.md
deleted
-21
@@ -1,21 +0,0 @@
1
-# system_clock_sync_state
2
-
3
-## OS: Linux
4
-
5
-The Netdata Agent checks if your system is in sync with a Network Time Protocol (NTP) server. This
6
-alert indicates that the system time is not synchronized to a reliable server. It is strongly
7
-recommended having the clock in sync with NTP servers, because, otherwise, it leads to unpredictable
8
-problems that are difficult to debug especially in matters of security.
9
-
10
-Here you can find a great article on
11
-[best practices for NTP servers](https://bluecatnetworks.com/blog/seven-best-practices-to-keep-your-ntp-resilient/).
12
-
13
-# Troubleshooting section:
14
-
15
-<details>
16
-<summary> General approach </summary>
17
-
18
-Different linux distros utilize different NTP tools. You can always install `ntp`. If your clock is
19
-out of sync, you should first check for issues in your network connectivity.
20
-</details>
21
-
health/guides/udp_errors/1m_ipv4_udp_receive_buffer_errors.md
deleted
-155
@@ -1,155 +0,0 @@
1
-# 1m_ipv4_udp_receive_buffer_errors
2
-
3
-*In computer networking, the User Datagram Protocol (UDP) is one of the core members of the Internet
4
-protocol suite.*
5
-
6
-In both Linux and FreeBSD variants, the kernel allocates buffers to serve the UDP protocol operations.
7
-Packets after reception from a network interface are forwarded to these buffers to be processed by
8
-the UDP protocol stack in a system's socket.
9
-
10
-The Netdata Agent monitors the average number of UDP receive buffer errors over the last minute.
11
-Receiving this alert means that your system is dropping incoming UDP packets. This may indicate that
12
-the UDP receive buffer queue is full.
13
-
14
-This alert is triggered in warning state when the number of UDP receive buffer errors over the last
15
-minute is more than 10.
16
-
17
-<details>
18
-<summary>See more on UDP protocol</summary>
19
-
20
-> UDP uses a simple connectionless communication model with a minimum of protocol mechanisms. UDP
21
-provides checksums for data integrity, and port numbers for addressing different functions at the
22
-source and destination of the datagram. It has no handshaking dialogues, and thus exposes the user's
23
-program to any unreliability of the underlying network. There is no guarantee of delivery, ordering,
24
-or duplicate protection.<sup>[1](https://en.wikipedia.org/wiki/User_Datagram_Protocol) </sup> If no
25
-firewall exists any host can send udp packets to any port, which your server doesn't listen.
26
-
27
-</details>
28
-
29
-<details>
30
-<summary>References and sources</summary>
31
-
32
-1. [UDP definition on wikipedia](https://en.wikipedia.org/wiki/User_Datagram_Protocol)
33
-2. [Man page of UDP protocol](https://man7.org/linux/man-pages/man7/udp.7.html)
34
-3. [Redhat networking tuning guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings)
35
-4. [UDP on freebsd (blog)](https://awasihba.wordpress.com/2008/10/13/udp-on-freebsd/)
36
-
37
-
38
-</details>
39
-
40
-## OS: Linux
41
-
42
-Receive buffer sizes for UDP are controlled by 4
43
-variables.<sup> [2](https://man7.org/linux/man-pages/man7/udp.7.html) </sup>
44
-
45
-- `net.core.rmem_default`, the default setting of the socket receive buffer in bytes.
46
-
47
-- `net.core.rmem_max`, the maximum receive socket buffer size in bytes.. Each socket gets
48
- `rmem_default` receive buffer size by default, and can request up to `rmem_max` with `setsockopt`
49
- option `SO_RCVBUF`.
50
-
51
-- `net.ipv4.udp_mem`, this is a vector of three integers (min, pressure, max) governing the number
52
- of pages allowed for queueing by all UDP sockets.
53
- - min: Below this number of pages, UDP is not bothered about its memory appetite. When the
54
- amount of memory allocated by UDP exceeds this number, UDP starts to moderate memory usage.
55
- - pressure: This value was introduced to follow the format of tcp_mem (see tcp(7)).
56
- - max: Defaults values for these three items are calculated at boot time from the amount of
57
- available memory.
58
-
59
-- `net.ipv4.udp_rmem_min`, the minimal size (in bytes) of receive buffer used by UDP sockets in
60
- moderation. Each UDP socket is able to use the size for receiving data, even if total pages of UDP
61
- sockets exceed udp_mem pressure.
62
-
63
-In general, issues with buffers that allocated dynamically are correlated with the kernel
64
-memory, you must always be aware of memory pressure events. This can cause buffer errors.
65
-
66
-### Troubleshooting section:
67
-
68
- <details>
69
- <summary>Increase the net.core.rmem_default and net.core.rmem_max values</summary>
70
-
71
-1. Try to increase them, RedHat suggests the value of 262144
72
- bytes <sup> [3](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings) </sup>
73
-
74
- ```
75
- sysctl -w net.core.rmem_default=262144
76
- sysctl -w net.core.rmem_max=262144
77
- ```
78
-
79
-1. Verify the change and test with the same workload that triggered the alarm originally.
80
-
81
- ```
82
- root@netdata~ # sysctl net.core.rmem_default net.core.rmem_max
83
- net.core.rmem_default=262144
84
- net.core.rmem_max=262144
85
- ```
86
-
87
-1. If this change works for your system, you could make it permanently.
88
-
89
- Bump these `net.core.rmem_default=262144` & `net.core.rmem_max=262144` entries under
90
- `/etc/sysctl.conf`.
91
-
92
-1. Reload the sysctl settings.
93
-
94
- ```
95
- root@netdata~ # sysctl -p
96
- ```
97
-
98
-</details>
99
-
100
-## OS: FreeBSD
101
-
102
-Buffer space for any UDP connection on freebsd is affected by following parameters, as mentioned by
103
-Awasihba in his personal blog. [4](https://awasihba.wordpress.com/2008/10/13/udp-on-freebsd/)
104
-
105
-- `net.inet.udp.recvspace`, when you open any UDP socket this parameter decides default receiving
106
- buffer space for userland data for that socket. You can override that size with help of
107
- `setsockopt` in your code.
108
-
109
-- `kern.ipc.maxsockbuf`, the buffer space for socket, is determined by this parameter. So if
110
- you try to open socket with large send and receive buffer, and you get error like "no buffer space
111
- available" then you should consider tweaking `kern.ipc.maxsockbuf`. Sometimes you see frequent UDP
112
- drops while dealing with large number of tiny UDP packets. Even if your `recvspace` buffer is not
113
- filled up completely, still you will drop the packets. After digging around for a while we
114
- figured out that it was happening because we were hitting another hard limit of`sockbuf->sb_mbmax`,
115
- it specifies maximum number of `mbufs` allocated for each socket. You can increase that limit by
116
- increasing `kern.ipc.maxsockbuf`. You need to restart related services to apply this parameter.
117
-
118
-- `kern.ipc.nmbcluster`, this parameter governs the total amount memory you have to allocate for all
119
- the open sockets on your system. This value defines how many numbers of mbuf cluster should be
120
- allocated. Usually each cluster is of 2k size. For example , if you are planning to open 1000
121
- sockets with each having 8k sending and 8k size receiving buffer each socket will need 16k of
122
- memory and in total you will need 16M (16k x 1000 ) of memory to handle all 1000 connections.
123
-
124
-In general, issues with buffers that allocated dynamically are correlated with the kernel
125
-memory, you must always be aware of memory pressure events. This can cause buffer errors.
126
-
127
-### Troubleshooting section:
128
-
129
- <details>
130
- <summary>Increase the kern.ipc.maxsockbuf value</summary>
131
-
132
-1. Try to set this value to at least 16MB for 10GE overall
133
-
134
- ```
135
- root@netdata~ # sysctl -w kern.ipc.maxsockbuf=16777216
136
- ```
137
-
138
-1. Verify the change and test with the same workload that triggered the alarm originally.
139
-
140
- ```
141
- root@netdata~ # sysctl kern.ipc.maxsockbuf
142
- kern.ipc.maxsockbuf=16777216
143
- ```
144
-
145
-1. If this change works for your system, you could make it permanently.
146
-
147
- Bump this `kern.ipc.maxsockbuf=16777216` entry under `/etc/sysctl.conf`.
148
-
149
-1. Reload the sysctl settings.
150
-
151
- ```
152
- root@netdata~ # /etc/rc.d/sysctl reload
153
- ```
154
-
155
-</details>
health/guides/udp_errors/1m_ipv4_udp_send_buffer_errors.md
deleted
-96
@@ -1,96 +0,0 @@
1
-# 1m_ipv4_udp_send_buffer_errors
2
-
3
-## OS: Linux
4
-
5
-*In computer networking, the User Datagram Protocol (UDP) is one of the core members of the Internet
6
-protocol suite.*
7
-
8
-The linux kernel allocates buffers to serve the UDP protocol operations. Data is written into
9
-sockets that utilize UDP to send data to an another system/subsystem.
10
-
11
-The Netdata Agent monitors the average number of UDP send buffer errors over the last minute. This
12
-alert indicates that the UDP send buffer is full or no kernel memory available. Receiving this alert
13
-means that your system is dropping outgoing UDP packets
14
-
15
-This alert is triggered in warning state when the number of UDP send buffer errors over the last
16
-minute is more than 10.
17
-
18
-<details>
19
-<summary>See more on UDP protocol</summary>
20
-
21
-> UDP uses a simple connectionless communication model with a minimum of protocol mechanisms. UDP
22
-provides checksums for data integrity, and port numbers for addressing different functions at the
23
-source and destination of the datagram. It has no handshaking dialogues, and thus exposes the user's
24
-program to any unreliability of the underlying network. There is no guarantee of delivery, ordering,
25
-or duplicate protection.<sup>[1](https://en.wikipedia.org/wiki/User_Datagram_Protocol) </sup> If no
26
-firewall exists any host can send udp packets to any port, which your server doesn't listen.
27
-
28
-</details>
29
-
30
-
31
-Send buffer sizes for UDP are controlled by 4
32
-variables.<sup> [2](https://man7.org/linux/man-pages/man7/udp.7.html) </sup>
33
-
34
-- `net.core.wmem_default`, the default setting of the socket send buffer in bytes.
35
-
36
-- `net.core.wmem_max`, default and max socket send buffer size in bytes. Each socket gets
37
- `wmem_default` send buffer size by default, and can request up to `wmem_max` with `setsockopt`
38
- option `SO_SNDBUF`.
39
-
40
-- `net.ipv4.udp_mem`, this is a vector of three integers (min, pressure, max) governing the number
41
- of pages allowed for queueing by all UDP sockets.
42
- - min: Below this number of pages, UDP is not bothered about its memory appetite. When the
43
- amount of memory allocated by UDP exceeds this number, UDP starts to moderate memory usage.
44
- - pressure: This value was introduced to follow the format of tcp_mem (see tcp(7)).
45
- - max: Defaults values for these three items are calculated at boot time from the amount of
46
- available memory.
47
-
48
-- `net.ipv4.udp_wmem`, the minimal size (in bytes) of send buffer used by UDP sockets in moderation.
49
- Each UDP socket is able to use the size for sending data, even if total pages of UDP sockets
50
- exceed `udp_mem` pressure.
51
-
52
-In general, issues with buffers that allocated dynamically are correlated with the kernel memory,
53
-you must always be aware of memory pressure events. This can cause buffer errors.
54
-
55
-<details>
56
-<summary>References and sources</summary>
57
-
58
-1. [UDP definition on wikipedia](https://en.wikipedia.org/wiki/User_Datagram_Protocol)
59
-2. [Man page of UDP protocol](https://man7.org/linux/man-pages/man7/udp.7.html)
60
-3. [Redhat networking tuning guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings)
61
-
62
-</details>
63
-
64
-### Troubleshooting section:
65
-
66
- <details>
67
- <summary>Increase the net.core.wmem_default and net.core.wmem_max values</summary>
68
-
69
-1. Try to increase them, RedHat suggests the value of 262144
70
- bytes <sup> [3](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-changing_network_kernel_settings) </sup>
71
-
72
- ```
73
- sysctl -w net.core.wmem_default=262144
74
- sysctl -w net.core.wmem_max=262144
75
- ```
76
-
77
-1. Verify the change and test with the same workload that triggered the alarm originally.
78
-
79
- ```
80
- root@netdata~ # sysctl net.core.wmem_default net.core.wmem_max
81
- net.core.wmem_default=262144
82
- net.core.wmem_max=262144
83
- ```
84
-
85
-1. If this change works for your system, you could make it permanently.
86
-
87
- Bump these `net.core.wmem_default=262144` & `net.core.wmem_max=262144` entries under
88
- `/etc/sysctl.conf`.
89
-
90
-1. Reload the sysctl settings.
91
-
92
- ```
93
- root@netdata~ # sysctl -p
94
- ```
95
-
96
-</details>
health/guides/unbound_request_list_dropped.md
new
+38
@@ -0,0 +1,38 @@
1
+### Understand the alert
2
+
3
+The `unbound_request_list_dropped` alert indicates that the Unbound DNS resolver is dropping new incoming requests because its request queue is full. This situation may be caused by a high volume of DNS queries, possibly from a Denial of Service (DoS) attack or poor server optimization.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the request queue length**: Inspect the Unbound configuration file (usually located at `/etc/unbound/unbound.conf`) and check the `num-queries-per-thread` setting. If the value is too low for your system, you may encounter issues with dropped requests.
8
+
9
+2. **Increase the queue length**: If necessary, increase the `num-queries-per-thread` value in the Unbound configuration file. For example, if the current value is 1024, you can try setting it to a higher value, such as 2048 or 4096. Save the changes and restart the Unbound service:
10
+
11
+ ```
12
+ sudo systemctl restart unbound
13
+ ```
14
+
15
+3. **Monitor dropped requests**: Use the `unbound-control` command to monitor the number of dropped requests in real-time:
16
+
17
+ ```
18
+ sudo unbound-control stats_noreset | grep num.requestlist.dropped
19
+ ```
20
+
21
+ If you see the dropped requests decreasing, your changes to the `num-queries-per-thread` value may have resolved the issue.
22
+
23
+4. **Inspect server logs**: Check the Unbound log file (usually located at `/var/log/unbound.log`) for any suspicious activity or error messages that may indicate the cause of the increased DNS queries.
24
+
25
+5. **Check for potential DoS attacks**: Use tools like `iftop`, `nload`, or `nethogs` to monitor network traffic and identify any potential DoS attacks or unusual traffic patterns.
26
+
27
+ If you believe your server is experiencing a DoS attack:
28
+
29
+ - Investigate the source IP addresses of the high-volume traffic
30
+ - Block malicious traffic using firewall tools like `iptables` or `ufw`
31
+ - Contact your hosting provider, ISP, or network administrator for assistance
32
+
33
+6. **Optimize Unbound**: Review the [official Unbound documentation](https://nlnetlabs.nl/documentation/unbound/) and tune the settings in the Unbound configuration file to ensure optimal performance for your specific environment.
34
+
35
+### Useful resources
36
+
37
+1. [Unbound Official Documentation](https://nlnetlabs.nl/documentation/unbound/)
38
+2. [How to set up a DNS Resolver with Unbound](https://calomel.org/unbound_dns.html)
health/guides/unbound_request_list_overwritten.md
new
+50
@@ -0,0 +1,50 @@
1
+### Understand the alert
2
+
3
+The `unbound_request_list_overwritten` alert is triggered when Unbound, a popular DNS resolver, overwrites old queued requests because its request queue is full. This alert can indicate a Denial of Service (DoS) attack or network saturation.
4
+
5
+### What does request list overwritten mean?
6
+
7
+When the request queue is full, Unbound starts overwriting the oldest requests in the queue with newer incoming requests. This is done to handle increasing load, but it may also lead to dropped or lost queries.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check the Unbound log file for any unusual events or error messages. The default log file location is `/var/log/unbound.log`. You may find more information about the cause of the request queue overload, such as a high number of incoming queries or sudden spikes in traffic.
12
+
13
+- Monitor Unbound's real-time statistics using the `unbound-control` command, which allows you to view various metrics related to the performance of the Unbound server:
14
+
15
+ ```
16
+ sudo unbound-control stats_noreset
17
+ ```
18
+
19
+ Look for the `num.query.list` and `num.query.list.overwritten` values to determine how many queries are in the request queue and how many of them are being overwritten.
20
+
21
+- Analyze the incoming DNS queries to check for suspicious patterns, such as high query rates from specific clients or repeated queries for the same domain. You can use tools like `tcpdump` to capture and inspect DNS traffic:
22
+
23
+ ```
24
+ sudo tcpdump -i any -nn -s0 -w dns_traffic.pcap 'port 53'
25
+ ```
26
+
27
+ You can then analyze the captured data using packet analyzers like Wireshark or tshark.
28
+
29
+- Increase the request queue length by adjusting the `num-queries-per-thread` value in the Unbound configuration file (`/etc/unbound/unbound.conf`), which determines the maximum number of queries that can be queued per thread before overwriting begins. Increasing this value may help to accommodate higher incoming query loads:
30
+
31
+ ```
32
+ server:
33
+ num-queries-per-thread: 4096
34
+ ```
35
+
36
+ Remember to restart the Unbound service for the changes to take effect (`sudo systemctl restart unbound`).
37
+
38
+- Consider implementing rate limiting to prevent a single client from overloading the server. Unbound supports rate limiting using the `ratelimit` configuration option:
39
+
40
+ ```
41
+ server:
42
+ ratelimit: 1000
43
+ ```
44
+
45
+ This example sets a limit of 1000 queries per second, but you should tune it according to your environment.
46
+
47
+### Useful resources
48
+
49
+1. [Unbound Configuration Guide](https://nlnetlabs.nl/documentation/unbound/unbound.conf/)
50
+2. [Unbound Rate Limiting](https://calomel.org/unbound_dns.html#ratelimit)
health/guides/upsd/upsd_10min_ups_load.md
deleted
-26
@@ -1,26 +0,0 @@
1
-# nut_10min_ups_load
2
-
3
-**Power Supply | UPS**
4
-
5
-Network UPS Tools (NUT) is a suite of software component designed to monitor power devices, such as
6
-uninterruptible power supplies, power distribution units, solar controllers and servers power supply
7
-units.
8
-
9
-The Netdata Agent monitors the average UPS load over the last 10 minutes.
10
-
11
-<details>
12
-<summary>References and Sources</summary>
13
-
14
-1. [NUT user manual]https://networkupstools.org/docs/user-manual.chunked/index.html
15
-
16
-</details>
17
-
18
-### Troubleshooting section:
19
-
20
-<details>
21
-<summary>Reduce the load on the UPS</summary>
22
-
23
-To avoid ungraceful shutdowns of your systems, consider to reduce the load on this particular UPS.
24
-To achieve this, consider removing attached devices that are not mission critical.
25
-
26
-</details>
health/guides/upsd/upsd_last_collected_secs.md
deleted
-54
@@ -1,54 +0,0 @@
1
-# nut_last_collected_secs
2
-
3
-**Power Supply | UPS**
4
-
5
-Network UPS Tools (NUT) is a suite of software component designed to monitor power devices, such as
6
-uninterruptible power supplies, power distribution units, solar controllers and servers power supply
7
-units.
8
-
9
-The Netdata Agent monitors the number of seconds since the last successful data collection
10
-
11
-<details>
12
-<summary>References and Sources</summary>
13
-
14
-1. [NUT user manual]https://networkupstools.org/docs/user-manual.chunked/index.html
15
-
16
-</details>
17
-
18
-### Troubleshooting section:
19
-
20
-<details>
21
-<summary>Check the upsd server </summary>
22
-
23
-1. Check the status of the upsd daemon
24
- ```
25
- root@netdata $ systemctl status upsd
26
- ```
27
-
28
-2. Check for obvious and common errors.
29
-
30
-
31
-3. Restart the daemon if needed
32
- ```
33
- root@netdata $ systemctl restart apcupsd
34
- ```
35
-
36
-</details>
37
-
38
-<details>
39
-<summary>Diagnose a bad driver</summary>
40
-
41
-`upsd` expects the drivers to either update their status regularly or at least answer periodic
42
-queries, called pings. If a driver doesn’t answer, `upsd` will declare it "stale" and no more
43
-information will be provided to the clients.
44
-
45
-If upsd complains about staleness when you start it, then either your driver or configuration files
46
-are probably broken. Be sure that the driver is actually running, and that the UPS definition in
47
-[ups.conf(5)](https://networkupstools.org/docs/man/ups.conf.html) is correct. Also make sure that
48
-you start your driver(s) before starting upsd.
49
-
50
-Data can also be marked stale if the driver can no longer communicate with the UPS. In this case,
51
-the driver should also provide diagnostic information in the syslog. If this happens, check the
52
-serial or USB cabling, or inspect the network path in the case of a SNMP UPS.
53
-</details>
54
-
health/guides/upsd/upsd_ups_battery_charge.md
deleted
-26
@@ -1,26 +0,0 @@
1
-# nut_ups_charge
2
-
3
-**Power Supply | UPS**
4
-
5
-Network UPS Tools (NUT) is a suite of software component designed to monitor power devices, such as
6
-uninterruptible power supplies, power distribution units, solar controllers and servers power supply
7
-units.
8
-
9
-The Netdata Agent monitors the average UPS charge over the last minute.
10
-
11
-<details>
12
-<summary>References and Sources</summary>
13
-
14
-1. [NUT user manual]https://networkupstools.org/docs/user-manual.chunked/index.html
15
-
16
-</details>
17
-
18
-### Troubleshooting section:
19
-
20
-<details>
21
-<summary>Prepare your machine for graceful shutdown</summary>
22
-
23
-If you can't restore the power supply to this UPC, you should prepare your machine for graceful
24
-shutdown.
25
-
26
-</details>
health/guides/used_swap.md
new
+24
@@ -0,0 +1,24 @@
1
+### Understand the alert
2
+
3
+If the system needs more memory resources than your available RAM, inactive pages in memory can be moved into the swap space (or swap file). The Swap space (or swap file) is located on hard drives, which have a slower access time than physical memory.
4
+
5
+The Netdata Agent calculates the percentage of the used swap. This alert indicates high swap memory utilization. It may be a sign that the system has experienced memory pressure, which can affect the
6
+performance of your system. If there is no RAM and swap available, OOM Killer can start killing processes.
7
+
8
+This alert is triggered in warning state when the percentage of used swap is between 80-90% and in critical state when it is between 90-98%.
9
+
10
+### Troubleshoot the alert
11
+
12
+- Check per-process RAM usage to find the top consumers
13
+
14
+Linux:
15
+```
16
+top -b -o +%MEM | head -n 22
17
+```
18
+FreeBSD:
19
+```
20
+top -b -o res | head -n 22
21
+```
22
+
23
+It would be helpful to close any of the main consumer processes, but Netdata strongly suggests knowing exactly what processes you are closing and being certain that they are not necessary.
24
+
health/guides/vcsa/vcsa_applmgmt_health.md
deleted
-29
@@ -1,29 +0,0 @@
1
-# vcsa_applmgmt_health
2
-
3
-## Virtual Machine | VMware vCenter
4
-
5
-
6
-This alert presents the appliance management component health status.
7
-The values for every component's health can be:
8
-
9
-| Code | Color | Description | Alert Status |
10
-|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
11
-| `-1` | no color | Unknown. | Clear |
12
-| `0` |  | The component is healthy. | Clear |
13
-| `1` |  | The component is healthy but may have some problems. | Warning |
14
-| `2` |  | The component is degraded, and may have serious problems. | Critical |
15
-| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
16
-| `4` |  | No health data is available. | Clear |
17
-
18
-For more information, please see the [VMware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html).
19
-
20
-### Troubleshooting Section
21
-
22
-To find out why the alert was raised, follow the steps in
23
-the [VMware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html).
24
-
25
-<details><summary>References and Sources</summary>
26
-
27
-[VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
28
-
29
-</details>
health/guides/vcsa/vcsa_database_storage_health.md
deleted
-30
@@ -1,30 +0,0 @@
1
-# vcsa_database_storage_health
2
-
3
-## Virtual Machine | VMware vCenter
4
-
5
-
6
-This alert presents the database storage component health status.
7
-The values for every component's health can be:
8
-
9
-| Code | Color | Description | Alert Status |
10
-|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
11
-| `-1` | no color | Unknown. | Clear |
12
-| `0` |  | The component is healthy. | Clear |
13
-| `1` |  | The component is healthy but may have some problems. | Warning |
14
-| `2` |  | The component is degraded, and may have serious problems. | Critical |
15
-| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
16
-| `4` |  | No health data is available. | Clear |
17
-
18
-For further information, please have a look at the *References and Sources* section.
19
-
20
-<details><summary>References and Sources</summary>
21
-
22
-1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
23
-
24
-</details>
25
-
26
-### Troubleshooting Section
27
-
28
-To find out why the alert was raised, follow the steps in
29
-the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
30
-.
health/guides/vcsa/vcsa_load_health.md
deleted
-29
@@ -1,29 +0,0 @@
1
-# vcsa_load_health
2
-
3
-## Virtual Machine | VMware vCenter
4
-
5
-This alert presents the load health status.
6
-The values for every component's health can be:
7
-
8
-| Code | Color | Description | Alert Status |
9
-|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
10
-| `-1` | no color | Unknown. | Clear |
11
-| `0` |  | The component is healthy. | Clear |
12
-| `1` |  | The component is healthy but may have some problems. | Warning |
13
-| `2` |  | The component is degraded, and may have serious problems. | Critical |
14
-| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
15
-| `4` |  | No health data is available. | Clear |
16
-
17
-For further information, please have a look at the *References and Sources* section.
18
-
19
-<details><summary>References and Sources</summary>
20
-
21
-1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
22
-
23
-</details>
24
-
25
-### Troubleshooting Section
26
-
27
-To find out why the alert was raised, follow the steps in
28
-the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
29
-.
\ No newline at end of file
health/guides/vcsa/vcsa_mem_health.md
deleted
-29
@@ -1,29 +0,0 @@
1
-# vcsa_mem_health
2
-
3
-## Virtual Machine | VMware vCenter
4
-
5
-This alert presents the memory health status.
6
-The values for every component's health can be:
7
-
8
-| Code | Color | Description | Alert Status |
9
-|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
10
-| `-1` | no color | Unknown. | Clear |
11
-| `0` |  | The component is healthy. | Clear |
12
-| `1` |  | The component is healthy but may have some problems. | Warning |
13
-| `2` |  | The component is degraded, and may have serious problems. | Critical |
14
-| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
15
-| `4` |  | No health data is available. | Clear |
16
-
17
-For further information, please have a look at the *References and Sources* section.
18
-
19
-<details><summary>References and Sources</summary>
20
-
21
-1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
22
-
23
-</details>
24
-
25
-### Troubleshooting Section
26
-
27
-To find out why the alert was raised, follow the steps in
28
-the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
29
-.
\ No newline at end of file
health/guides/vcsa/vcsa_software_updates_health.md
deleted
-31
@@ -1,31 +0,0 @@
1
-# vcsa_software_updates_health
2
-
3
-## Virtual Machine | VMware vCenter
4
-
5
-This alert presents the software updates availability status.
6
-The values can be:
7
-
8
-| Code | Color | Description | Alert Status |
9
-|:----:|:---------------------------------------------------------------:|:-----------------------------------------------------|:------------:|
10
-| `-1` | no color | Unknown. | Clear |
11
-| `0` |  | no updates available. | Clear |
12
-| `2` |  | non-security updates are available. | Clear |
13
-| `3` |  | security updates are available. | Critical |
14
-| `4` |  | an error retrieving information on software updates. | Warning |
15
-
16
-For further information, please have a look at the *References and Sources* section.
17
-
18
-<details><summary>References and Sources</summary>
19
-
20
-1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
21
-
22
-</details>
23
-
24
-
25
-### Troubleshooting Section
26
-
27
-If the alert was raised into critical, proceed by installing the security updates that are
28
-available. If the alert was raised into warning, consider viewing the details in the Health Messages
29
-pane.
30
-
31
-You can also find more details in the [VMware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html).
health/guides/vcsa/vcsa_storage_health.md
deleted
-29
@@ -1,29 +0,0 @@
1
-# vcsa_storage_health
2
-
3
-## Virtual Machine | VMware vCenter
4
-
5
-This alert presents the storage health status.
6
-The values for every component's health can be:
7
-
8
-| Code | Color | Description | Alert Status |
9
-|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:-------------:|
10
-| `-1` | no color | Unknown. | Clear |
11
-| `0` |  | The component is healthy. | Clear |
12
-| `1` |  | The component is healthy but may have some problems. | Warning |
13
-| `2` |  | The component is degraded, and may have serious problems. | Critical |
14
-| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
15
-| `4` |  | No health data is available. | Clear |
16
-
17
-For further information, please have a look at the *References and Sources* section.
18
-
19
-<details><summary>References and Sources</summary>
20
-
21
-1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
22
-
23
-</details>
24
-
25
-### Troubleshooting Section
26
-
27
-To find out why the alert was raised, follow the steps in
28
-the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
29
-.
health/guides/vcsa/vcsa_swap_health.md
deleted
-29
@@ -1,29 +0,0 @@
1
-# vcsa_swap_health
2
-
3
-## Virtual Machine | VMware vCenter
4
-
5
-This alert presents the swap health status.
6
-The values for every component's health can be:
7
-
8
-
9
-| Code | Color | Description | Alert Status |
10
-|:----:|:---------------------------------------------------------------:|:------------------------------------------------------------|:------------:|
11
-| `-1` | no color | Unknown. | Clear |
12
-| `0` |  | The component is healthy. | Clear |
13
-| `1` |  | The component is healthy but may have some problems. | Warning |
14
-| `2` |  | The component is degraded, and may have serious problems. | Critical |
15
-| `3` |  | The component is unavailable or will stop functioning soon. | Critical |
16
-| `4` |  | No health data is available. | Clear |
17
-
18
-For further information, please have a look at the *References and Sources* section.
19
-
20
-<details><summary>References and Sources</summary>
21
-
22
-1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
23
-
24
-</details>
25
-
26
-
27
-### Troubleshooting Section
28
-
29
-To find out why the alert was raised, follow the steps in the [vmware vCenter Server documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html).
health/guides/vcsa/vcsa_system_health.md
deleted
-34
@@ -1,34 +0,0 @@
1
-# vcsa_system_health
2
-
3
-## Virtual Machine | VMware vCenter
4
-
5
-This alert presents the overall system health status.
6
-It can take the values:
7
-
8
-- -1: unknown (no color)
9
-- 0: all components are healthy. (green)
10
-- 1: one or more components might become overloaded soon. (yellow)
11
-- 2: one or more components in the appliance might be degraded. (orange)
12
-- 3: one or more components might be in an unusable status and the appliance might become unresponsive soon. (red)
13
-- 4: no health data is available. (grey)
14
-
15
-
16
-If you receive this alert, it means that the overall system status is unhealthy. One or more
17
-components might become overloaded soon (yellow), or might be degraded (orange), or
18
-might be in an unusable status and the appliance might become unresponsive soon (red).
19
-
20
-This alert is raised into warning if the status has a code of 1 or 2.
21
-If the metric reaches a value of 3, the alert is raised into critical.
22
-
23
-For further information, please have a look at the *References and Sources* section.
24
-
25
-<details><summary>References and Sources</summary>
26
-
27
-1. [VMware Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
28
-
29
-</details>
30
-
31
-
32
-### Troubleshooting Section
33
-
34
-To troubleshoot the issue, you need to log into vCenter Server Management Interface and follow the information in the [vmware documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html).
health/guides/vcsa_applmgmt_health.md
new
+40
@@ -0,0 +1,40 @@
1
+### Understand the alert
2
+
3
+The `vcsa_applmgmt_health` alert is related to the health of VMware vCenter Server Appliance (VCSA) components. This alert is triggered when the health of one or more components is in a degraded or critical state, meaning that your VMware vCenter Server Appliance may be experiencing issues.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Access the vSphere Client for the affected vCenter Server Appliance
8
+
9
+ Log in to the vSphere Client to check detailed health information and manage your VCSA.
10
+
11
+2. Check the health status of VCSA components
12
+
13
+ In the vSphere Client, navigate to `Administration` > `System Configuration` > `Services` and `Nodes` tab. The component health status will be shown in the `Health` column.
14
+
15
+3. Inspect the affected component(s)
16
+
17
+ If any components show a status other than "green" (healthy), click on the component to view more details and understand the issue.
18
+
19
+4. Check logs related to the affected component(s)
20
+
21
+ Access the vCenter Server Appliance Management Interface (VAMI) by navigating to `https://<appliance-IP-address-or-FQDN>:5480` and logging in with the administrator account.
22
+
23
+ In the VAMI, click on the `Monitoring` tab > `Logs`. Download and inspect the logs to identify the root cause of the issue.
24
+
25
+5. Take appropriate actions
26
+
27
+ Depending on the nature of the issue identified, perform the necessary actions or modifications to resolve it. Consult the VMware documentation for recommended solutions for specific component health issues.
28
+
29
+6. Monitor the component health
30
+
31
+ After performing appropriate actions, continue to monitor the VCSA component health in the vSphere Client to ensure they return to a healthy status.
32
+
33
+7. Contact VMware support
34
+
35
+ If you are unable to resolve the issue, contact VMware support for further assistance.
36
+
37
+### Useful resources
38
+
39
+1. [VMware vCenter Server 7.0 Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
40
+2. [VMware Support](https://www.vmware.com/support.html)
health/guides/vcsa_database_storage_health.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+The `vcsa_database_storage_health` alert monitors the health of database storage components in a VMware vCenter Server Appliance (vCSA). When this alert is triggered, it indicates that one or more components have a health status of Warning, Critical or Unknown.
4
+
5
+### What do the different health statuses mean?
6
+
7
+- Unknown (`-1`): The system is unable to determine the component's health status.
8
+- Healthy (`0`): The component is functioning correctly and has no known issues.
9
+- Warning (`1`): The component is currently operating but may be experiencing minor problems.
10
+- Critical (`2`): The component is degraded and might have significant issues affecting functionality.
11
+- Critical (`3`): The component is unavailable or expected to stop functioning soon, requiring immediate attention.
12
+- No health data (`4`): There is no health data available for the component.
13
+
14
+### Troubleshoot the alert
15
+
16
+1. **Identify the affected components**: To begin troubleshooting the alert, you need to identify which components are experiencing health issues. You can check the vCenter Server Appliance Management Interface (VAMI) to review the health status of all components.
17
+
18
+ - Access the VAMI by navigating to `https://<appliance-IP>/ui` in your web browser.
19
+ - Log in with your vCenter credentials.
20
+ - Click on the `Health` tab in the left-hand menu to view the health status of all components.
21
+
22
+2. **Investigate the issues**: Once you have identified the affected components, review the alarms and events in vCenter to determine the root cause of the problems. Pay close attention to any recent changes or updates that may have impacted system functionality.
23
+
24
+3. **Review the vCenter Server logs**: If necessary, examine the logs in vCenter Server to gather more information about any possible issues. The logs can be accessed via SSH, the VAMI, or using the Log Browser in the vSphere Web Client.
25
+
26
+4. **Take corrective actions**: Based on your findings from the previous steps, address the issues affecting the health status of the components.
27
+
28
+ - In the case of insufficient storage, increasing the storage capacity or deleting unnecessary files might resolve the problem.
29
+ - If the issues are caused by hardware failures, consider replacing or repairing the affected hardware components.
30
+ - For software-related issues, ensure that all components are up-to-date and properly configured.
31
+
32
+5. **Monitor the component health**: After taking corrective actions, continue to monitor the health statuses of the affected components through the VAMI to ensure that the issues have been successfully resolved.
33
+
health/guides/vcsa_load_health.md
new
+18
@@ -0,0 +1,18 @@
1
+### Understand the alert
2
+
3
+The `vcsa_load_health` alert indicates the current health status of the VMware vCenter Server Appliance (VCSA) system components. The color-coded health indicators help quickly understand the overall state of the system.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Log in to the vCenter Server Appliance Management Interface (VAMI):** Open a web browser and navigate to `https://vcsa_address:5480`, where `vcsa_address` is the IP address or domain name of the VCSA. Log in with the appropriate credentials (by default, the `root` user).
8
+
9
+2. **Inspect the health status of VCSA components:** Once logged in, go to the `Summary` tab, which displays the health status of various components, such as Database, Management, and Networking. You can hover over the component's health icon to get more information about its status.
10
+
11
+3. **Check for specific component warnings or critical issues:** If any component has a warning or critical health status, click on the `Monitor` tab and then on the component in question to get more details about the specific problem.
12
+
13
+4. **Review log files:** For further investigation, review the log files associated with the affected VCSA component. The log files can be accessed on the VAMI interface under the `Logs` tab.
14
+
15
+5. **Resolve the issue:** Based on the information gathered from the VAMI interface and log files, take appropriate action to resolve the issue or contact VMware support for assistance.
16
+
17
+6. **Monitor VCSA Health:** After resolving the issue, monitor the health status of the VCSA components on the `Summary` tab in VAMI to ensure that the health indicators return to a normal state.
18
+
health/guides/vcsa_mem_health.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+The `vcsa_mem_health` alert indicates the memory health status of a virtual machine within the VMware vCenter. If you receive this alert, it means that the system's memory health could be compromised, and might lead to degraded performance, serious problems, or stop functioning.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the vCenter Server Appliance health**:
8
+ - Log in to the vSphere Client and select the vCenter Server instance.
9
+ - Navigate to the Monitor tab > Health section.
10
+ - Check the Memory Health status, and take note of any concerning warnings or critical issues.
11
+
12
+2. **Analyze the memory usage**:
13
+ - Log in to the vSphere Client and select the virtual machine.
14
+ - Navigate to the Monitor tab > Performance section > Memory.
15
+ - Evaluate the memory usage trends and look for any unusual spikes or prolonged high memory usage.
16
+
17
+3. **Identify processes consuming high memory**:
18
+ - Log in to the affected virtual machine.
19
+ - Use the appropriate task manager or command, depending on the OS, to list processes and their memory usage.
20
+ - Terminate any unnecessary processes that are consuming high memory, but ensure that the process is not critical to system operation.
21
+
22
+4. **Optimize the virtual machine's memory allocation**:
23
+ - If the virtual machine consistently experiences high memory usage, consider increasing the allocated memory or optimizing applications running on the virtual machine to consume less memory.
24
+
25
+5. **Update VMware tools**:
26
+ - Ensuring that the VMware tools are up to date can help in better memory management and improve overall system health.
27
+
28
+6. **Check hardware issues**:
29
+ - If the problem persists, check hardware components such as memory sticks, processors, and data stores for any faults that could be causing the problem.
30
+
31
+7. **Contact VMware Support**:
32
+ - If you can't resolve the `vcsa_mem_health` alert or are unable to identify the root cause, contact VMware Support for further assistance.
33
+
34
+### Useful resources
35
+
36
+1. [VMware vCenter Server Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
health/guides/vcsa_software_updates_health.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+The `vcsa_software_updates_health` alert monitors the software updates availability status for a VMware vCenter Server Appliance (VCSA). The alert can have different statuses depending on the software updates state, with critical indicating that security updates are available.
4
+
5
+### Troubleshoot the alert
6
+
7
+Follow these troubleshooting steps according to the alert status:
8
+
9
+1. **Critical (security updates available):**
10
+
11
+ - Access the vCenter Server Appliance Management Interface (VAMI) by browsing to `https://<vcsa-address>:5480`.
12
+ - Log in with the appropriate user credentials (typically `root` user).
13
+ - Click on the `Update` menu item.
14
+ - Review the available patches and updates, especially those related to security.
15
+ - Click `Stage and Install` to download and install the security updates.
16
+ - Monitor the progress of the update installation and, if needed, address any issues that might occur during the process.
17
+
18
+2. **Warning (error retrieving information on software updates):**
19
+
20
+ - Access the vCenter Server Appliance Management Interface (VAMI) by browsing to `https://<vcsa-address>:5480`.
21
+ - Log in with the appropriate user credentials (typically `root` user).
22
+ - Click on the `Update` menu item.
23
+ - Check for any error messages in the `Update` section.
24
+ - Ensure that the VCSA has access to the internet and can reach the VMware update repositories.
25
+ - Verify that there are no issues with the system time or SSL certificates.
26
+ - If the issue persists, consider searching for relevant information in the VMware Knowledge Base or contacting VMware Support.
27
+
28
+3. **Clear (no updates available, non-security updates available, or unknown status):**
29
+
30
+ - No immediate action is required. However, it's a good practice to periodically check for updates to ensure the VMware vCenter Server Appliance remains up-to-date and secure.
31
+
32
+### Useful resources
33
+
34
+1. [VMware vCenter Server Appliance Management](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
35
+2. [VMware Knowledge Base](https://kb.vmware.com/)
health/guides/vcsa_storage_health.md
new
+28
@@ -0,0 +1,28 @@
1
+### Understand the alert
2
+
3
+The `vcsa_storage_health` alert indicates the health status of the storage components in your VMware vCenter Server Appliance (vCSA). It notifies you when the storage components are experiencing issues or are at risk of failure.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify the affected component(s): Check the alert details and note the component(s) with the corresponding health codes to determine their status.
8
+
9
+2. Access the vCenter Server Appliance Management Interface (VAMI): Open a supported browser and enter the URL: `https://<appliance-IP-address-or-FQDN>:5480`. Log in with the administrator or root credentials.
10
+
11
+3. Navigate to the Storage tab: In the VAMI, click on the 'Monitor' tab and then click on 'Storage.'
12
+
13
+4. Analyze the storage health: Review the reported storage health status for each component, match the health status with the information in the alert, and identify any issues.
14
+
15
+5. Remediate the issue: Depending on the identified problem, take the necessary actions to resolve the issue. Examples include:
16
+
17
+ - Check for any hardware faults and replace faulty components.
18
+ - Investigate possible disk space issues and free up space or increase the storage capacity.
19
+ - Verify that the storage subsystem is properly configured, and no misconfigurations are causing the issue.
20
+ - Look for software issues, such as failed updates, and resolve them or rollback changes.
21
+ - Consult VMware support if further assistance is needed.
22
+
23
+6. Verify resolution: After resolving the issue, verify that the storage health status has improved by checking the current status in the VAMI Storage tab.
24
+
25
+### Useful resources
26
+
27
+1. [VMware vCenter Server Appliance Management Interface](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
28
+2. [VMware vSphere Documentation](https://docs.vmware.com/en/VMware-vSphere/index.html)
health/guides/vcsa_swap_health.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+The vcsa_swap_health alert presents the swap health status of the VMware vCenter virtual machine. It is an indicator of the overall health of memory swapping on the vCenter virtual machine.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. First, identify the health status of the alert by checking the color and its corresponding description in the table above.
8
+
9
+2. Log in to the VMware vSphere Web Client:
10
+ - Navigate to `https://<vCenter-IP-address-or-domain-name>:<port>/vsphere-client`, where `<vCenter-IP-address-or-domain-name>` is your vCenter Server system IP or domain name, and `<port>` is the port number over which to access the vSphere Web Client.
11
+ - Enter the username and password, and click Login.
12
+
13
+3. Navigate to the vCenter virtual machine, and select the Monitor tab.
14
+
15
+4. Verify the swap file size by selecting the `Performance` tab, and choosing `Advanced` view.
16
+
17
+5. Monitor the swap usage on the virtual machine:
18
+ - On the `Performance` tab, look for high swap usage (`200 MB` or above). If necessary, consider increasing the swap file size.
19
+ - On the `Summary` tab, check for any warning or error messages related to the swap file or its usage.
20
+
21
+6. Check if there are any leading processes consuming an unreasonable amount of memory:
22
+ - If running a Linux-based virtual machine, use command-line utilities like `free`, `top`, `vmstat`, or `htop`. Look out for processes with high `%MEM` or `RES` values.
23
+ - If running a Windows-based virtual machine, use Task Manager or Performance Monitor to check for memory usage.
24
+
25
+7. Optimize the virtual machine memory settings:
26
+ - Verify if the virtual machine has sufficient memory allocation.
27
+ - Check the virtual machine's memory reservation and limit settings.
28
+ - Consider enabling memory ballooning for a better utilization of available memory.
29
+
30
+8. If the swap health status does not improve or you are unsure how to proceed, consult VMware documentation or contact VMware support for further assistance.
31
+
32
+### Useful resources
33
+
34
+1. [Configuring VMware vCenter 7.0](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-ACEC0944-EFA7-482B-84DF-6A084C0868B3.html)
35
+2. [Virtual Machine Memory Management Concepts](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/perf-vsphere-memory_management.pdf)
health/guides/vcsa_system_health.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+The `vcsa_system_health` alert indicates the overall health status of your VMware vCenter Server Appliance (vCSA). If you receive this alert, it means that one or more components in the appliance are in a degraded or unhealthy state that could lead to reduced performance or even appliance unresponsiveness.
4
+
5
+### Troubleshoot the alert
6
+
7
+Perform the following steps to identify and resolve the issue:
8
+
9
+1. Log in to the vCenter Server Appliance Management Interface (VAMI).
10
+
11
+ You can access the VAMI by navigating to `https://<your_vcenter_address>:5480` in a web browser. Log in with the appropriate credentials.
12
+
13
+2. Check the System Health status.
14
+
15
+ In the VAMI, click on the `Monitor` tab, and then click on `Health`. This will provide you with an overview of the different components and their individual health status.
16
+
17
+3. Analyze the affected components.
18
+
19
+ Identify the components that are displaying warning (yellow), degraded (orange), or critical (red) health status. These components may be causing the overall `vcsa_system_health` alert.
20
+
21
+4. Investigate the problematic components.
22
+
23
+ Click on each affected component to find more information about the issue. This may include error messages, suggested actions, and links to relevant documentation.
24
+
25
+5. Resolve the issues.
26
+
27
+ Follow the recommended actions or consult the VMware documentation to resolve the issues with the affected components.
28
+
29
+6. Verify the system health.
30
+
31
+ Once the issues have been resolved, refresh the Health page in the VAMI to ensure that all components now display a healthy (green) status. The `vcsa_system_health` alert should clear automatically.
32
+
33
+### Useful resources
34
+
35
+1. [VMware vSphere 7.0 vCenter Appliance Management](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-52AF3379-8D78-437F-96EF-25D1A1100BEE.html)
health/guides/vernemq/vernemq_average_scheduler_utilization.md
deleted
-43
@@ -1,43 +0,0 @@
1
-# vernemq_average_scheduler_utilization
2
-
3
-**Messaging | VerneMQ**
4
-
5
-VerneMQ is implemented in Erlang and therefore runs on top of the BEAM runtime environment (roughly equivalent to
6
-the JRE for Java applications).
7
-For performance reasons, BEAM utilizes it’s own intenral scheduler that operates largely independently of the operating
8
-system’s process scheduling.
9
-
10
-The Netdata Agent calculates the average VerneMQ's scheduler utilization over the last 10 minutes.
11
-This alert indicates high scheduler utilization.
12
-
13
-This alert is raised into warning when the scheduler's utilization is between 75-85% and in critical
14
-when it is between 85-95%.
15
-
16
-### Troubleshooting section:
17
-
18
-<details>
19
-<summary>Check for CPU throttling issues </summary>
20
-
21
-If you are receiving this alert often, it means that your node is running at maximum CPU utilization.
22
-You should consider upgrading your system (instance in your cloud) to provide more or faster CPUs.
23
-
24
-**Important**:
25
-
26
-By default, the VerneMQ broker deploys its Erlang VM architecture into 4 cores. If you already
27
-run VerneMQ in a multicore machine (for example, an 8-core machine) you should consider changing
28
-the `vmq_bcrypt.nif_pool_size` parameter:
29
-
30
-1. In the `vernemq.conf`, update the `vmq_bcrypt.nif_pool_size` parameter to `auto`. The value `auto`
31
-detect all cores (n) and set the value to n-1.
32
-
33
-
34
-2. Restart the VerneMQ service.
35
-
36
- ```
37
- root@netdata # systemctl restart vernemq.service
38
- ```
39
-
40
-3. Open the Netdata dashboard, and locate the `scheduler_utilization` chart. See if VerneMQ utilizes
41
- the preferred number of cores.
42
-
43
-</details>
\ No newline at end of file
health/guides/vernemq/vernemq_cluster_dropped.md
deleted
-51
@@ -1,51 +0,0 @@
1
-# vernemq_cluster_dropped
2
-
3
-**Messaging | VerneMQ**
4
-
5
-VerneMQ is a MQTT publish/subscribe message broker which implements the OASIS industry standard MQTT
6
-protocol.
7
-
8
-The Netdata agent calculates the amount of traffic dropped during communication with the cluster
9
-nodes in the last minute. This alert indicates that the outgoing cluster buffer is full.
10
-
11
-Receiving this alert most likely means that a remote node is down or unreachable, but it could also
12
-indicate that the VerneMQ is experiencing problems with inter-node message delivery. The
13
-non-dispatched messages are queued in this buffer.
14
-
15
-### Troubleshooting section:
16
-
17
-<details>
18
-<summary>Increase the cluster buffer </summary>
19
-
20
-To make your cluster more tolerant to disconnections of nodes, you can increase the size of the
21
-`outgoing_clustering_buffer_size` buffer.
22
-
23
-1. Edit the VerneMQ configuration file. By default it is located under `/etc/vernemq` folder.
24
-
25
- ```
26
- root@netdata # vim /etc/vernemq/vernemq.conf
27
- ```
28
-
29
-2. Append the `outgoing_clustering_buffer_size` value, the default value is 10000 bytes. Try to
30
- increase it to 15000
31
-
32
- ```
33
- # vim /etc/vernemq/vernemq.conf
34
- . . .
35
- outgoing_clustering_buffer_size = 15000
36
- . . .
37
- ```
38
-
39
-3. Restart the VerneMQ service
40
-
41
- ```
42
- root@netdata # systemctl restart vernemq.service
43
- ```
44
-
45
-4. Test with the same workload that triggered the alarm originally. If this alert still occurs, try
46
- to double this value and re-test.
47
-
48
-5. In case the problem still exists, you must check for issues in the nodes that are unavailable.
49
-
50
-</details>
51
-
health/guides/vernemq/vernemq_mqtt_connack_sent_reason_unsuccessful.md
deleted
-50
@@ -1,50 +0,0 @@
1
-# vernemq_mqtt_connack_sent_reason_unsuccessful
2
-
3
-**Messaging | VerneMQ**
4
-
5
-In the MQTT protocol, the CONNACK packet is the packet sent by the server in response to a CONNECT
6
-attempt from a client. The first packet sent from the server to the client must be a CONNACK packet.
7
-If the client does not receive a CONNACK packet from the server within a reasonable amount of time,
8
-the client should close the Network Connection. The "reasonable" amount of time depends on the type
9
-of application and the communications infrastructure.
10
-
11
-For various scenarios, there are specific CONNACK responses for both MQTT v3 and v5 . You can find
12
-the detailed response codes and descriptions for each protocol in the official documentation of
13
-MQTT,
14
-for [v3 (subsection 3.2.2.3 Connect Return code)](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718035)
15
-and
16
-for [v5 (subsection 3.2.2.2 Connect Reason Code)](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901074)
17
-
18
-MQTT v5 supports a wider variety of negative acknowledgements (unsuccessful CONNACK packets ), which
19
-makes it a lot easier for both the client and the VerneMQ admin to understand what's happening.
20
-
21
-The Netdata Agent monitors the number of sent unsuccessful v3/v5 CONNACK packets over the last
22
-minute. This alert is raised into warning when your VerneMQ server sends more than 5 unsuccessful
23
-packets in the last minute.
24
-
25
-<details>
26
-<summary>References and Sources</summary>
27
-
28
-1. [MQTT v3 docs, CONNACK description](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718033)
29
-2. [MQTT v5 docs, CONNACK description](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901074)
30
-
31
-</details>
32
-
33
-### Troubleshooting Section
34
-
35
-<details>
36
-<summary>General approach</summary>
37
-
38
-Open the alerts Dashboard, and locate the chart of this alert (`mqtt_connack_sent_reason`). Inspect
39
-which CONNACK packets (by reason) triggered this alert. As soon as you inspect the reason (by
40
-consulting the subsections: _connect reason code_ which we mentioned above for your protocol ), you
41
-will have to examine
42
-your [logs](https://docs.vernemq.com/configuring-vernemq/logging#console-logging) to check which
43
-client(s) are raising these issues. These kinds of issues appear in the warning log level, so you
44
-may have to set your log level appropriately.
45
-
46
-```
47
-root@netdata # cat /var/log/vernemq/console.log | grep "due to <keywords: error> "
48
-```
49
-
50
-</details>
health/guides/vernemq/vernemq_mqtt_disconnect_received_reason_not_normal.md
deleted
-48
@@ -1,48 +0,0 @@
1
-# vernemq_mqtt_disconnect_received_reason_not_normal
2
-
3
-**Messaging | VerneMQ**
4
-
5
-_The DISCONNECT packet is the final MQTT Control Packet sent from the Client or the Server. It
6
-indicates the reason why the Network Connection is being closed. The Client or Server may send a
7
-DISCONNECT packet before closing the Network Connection. If the Network Connection is closed without
8
-the Client first sending a DISCONNECT packet with Reason Code 0x00 (Normal disconnection) and the
9
-Connection has a Will Message, the Will Message is
10
-published. <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205) </sup>_
11
-
12
-The Netadata agent monitors the number of received not normal v5 DISCONNECT packets over the last
13
-minute. This alert is raised into warning when your VerneMQ server receive more than 5 DISCONNECT
14
-packets over the last minute.
15
-
16
-For various scenarios, there are specific DISCONNECT responses for MQTT protocol v5. You can find
17
-the detailed response codes which were sent by a client and their descriptions in the official
18
-documentation of MQTT in
19
-the [MQTT v5 docs, subsection 3.14.2.1: Disconnect reason code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205)
20
-
21
-<details>
22
-<summary>References and sources</summary>
23
-
24
-1. [MQTT v5 docs DISCONNECT notification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205)
25
-
26
-</details>
27
-
28
-### Troubleshooting Section
29
-
30
-<details>
31
-<summary>General approach</summary>
32
-
33
-Open the alerts Dashboard and locate the chart of this alert (`mqtt_disconnect_received_reason`).
34
-Inspect which DISCONNECT packets (by reason) triggered this alert. You can clarify why your server
35
-received those responses from a client by consulting the subsection _Disconnect reason
36
-code <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205) </sup>_
37
-which we mentioned above.
38
-
39
-For example: your server may receive some DISCONNECT packets with the reason: "Disconnect with Will
40
-Message." This is not abnormal except in the case in which the network connection is closed
41
-abruptly. This may indicate problems in the connectivity with your clients.
42
-
43
-</details>
44
-
45
-
46
-
47
-
48
-
health/guides/vernemq/vernemq_mqtt_disconnect_sent_reason_not_normal.md
deleted
-43
@@ -1,43 +0,0 @@
1
-# vernemq_mqtt_disconnect_sent_reason_not_normal
2
-
3
-**Messaging | VerneMQ**
4
-
5
-_The DISCONNECT packet is the final MQTT Control Packet sent from the Client or the Server. It
6
-indicates the reason why the Network Connection is being closed. The Client or Server may send a
7
-DISCONNECT packet before closing the Network
8
-Connection. <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205) </sup>_
9
-
10
-The Netdata Agent monitors the number of sent _not normal_ v5 DISCONNECT packets over the last
11
-minute. This alert is raised into warning when your VerneMQ server sends more than 5 DISCONNECT
12
-packets over the last minute.
13
-
14
-For various scenarios, there are specific DISCONNECT responses for MQTT protocol v5. You can find
15
-the detailed response codes which were sent by the Server and their descriptions in the official
16
-documentation of MQTT in
17
-the [MQTT v5 docs, subsection 3.14.2.1: Disconnect reason code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205)
18
-
19
-<details>
20
-<summary>References and sources</summary>
21
-
22
-1. [MQTT v5 docs DISCONNECT notification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205)
23
-
24
-</details>
25
-
26
-### Troubleshooting Section
27
-
28
-<details>
29
-<summary>General approach</summary>
30
-
31
-Open the alerts Dashboard and locate the chart of this alert (`mqtt_disconnect_sent_reason`).
32
-Inspect which DISCONNECT packets (by reason) triggered this alert. Inspect the reason why your
33
-server sent those responses by consulting the subsection _Disconnect reason
34
-code <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901205) </sup>_
35
-mentioned above.
36
-
37
-For example, your server may respond to a client with `QoS not supported`. In that case, the client must
38
-change the QoS settings.
39
-
40
-</details>
41
-
42
-
43
-
health/guides/vernemq/vernemq_mqtt_puback_received_reason_unsuccessful.md
deleted
-46
@@ -1,46 +0,0 @@
1
-# vernemq_mqtt_puback_received_reason_unsuccessful
2
-
3
-**Messaging | VerneMQ**
4
-
5
-A PUBACK packet is the response to a PUBLISH packet with QoS 1. The Netdata Agent monitors the
6
-number of received unsuccessful v5 PUBACK packets in the last minute. For various scenarios, there
7
-are specific PUBACK responses for MQTT protocol v5. You can find the detailed response codes which
8
-were sent by a client or a server and their descriptions in the official documentation of MQTT in
9
-the [MQTT v5 docs, PUBACK Reason Code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)
10
-. The Client or Server sending the PUBACK packet must always use one of the PUBACK Reason Codes.
11
-
12
-
13
-
14
-<details>
15
-<summary>See more about QoS 1 </summary>
16
-
17
-The Quality of Service (QoS) level is an agreement between the sender of a message and the receiver
18
-of a message that defines the guarantee of delivery for a specific message. In QoS 1, a client will
19
-receive a confirmation message from the broker upon receipt. If the expected confirmation is not
20
-received within a certain time frame, the client has to retry the message. A message received by a
21
-client must be acknowledged on time as well, otherwise the broker will re-deliver the
22
-message. <sup>[1](https://vernemq.com/intro/mqtt-primer/quality_of_service.html) </sup>
23
-
24
-</details>
25
-
26
-<details>
27
-<summary>References and sources</summary>
28
-
29
-1. [Quality of service explained, VerneMQ docs](https://vernemq.com/intro/mqtt-primer/quality_of_service.html)
30
-2. [MQTT v5 docs, PUBACK description](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
31
-
32
-</details>
33
-
34
-### Troubleshooting Section
35
-
36
-<details>
37
-<summary>General approach</summary>
38
-
39
-Open the alerts Dashboard and locate the chart of this alert (`mqtt_puback_received_reason`).
40
-Identify which PUBACK packets (by reason) triggered this alert. Inspect the reason why a client sent
41
-those responses by consulting the subsection _PUBACK Reason
42
-Code <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)_ </sup>
43
-mentioned above.
44
-
45
-</details>
46
-
health/guides/vernemq/vernemq_mqtt_puback_sent_reason_unsuccessful.md
deleted
-43
@@ -1,43 +0,0 @@
1
-# vernemq_mqtt_puback_sent_reason_unsuccessful
2
-
3
-**Messaging | VerneMQ**
4
-
5
-A PUBACK packet is the response to a PUBLISH packet with QoS 1. The Netdata Agent monitors the
6
-number of sent unsuccessful v5 PUBACK packets in the last minute. For various scenarios, there are
7
-specific PUBACK responses for MQTT protocol v5. You can find the detailed response codes which were
8
-sent by a client or a server and their descriptions in the official documentation of MQTT in
9
-the [MQTT v5 docs, PUBACK Reason Code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)
10
-. The Client or Server sending the PUBACK packet must always use one of the PUBACK Reason Codes.
11
-
12
-<details>
13
-<summary>See more about QoS 1 </summary>
14
-
15
-The Quality of Service (QoS) level is an agreement between the sender of a message and the receiver
16
-of a message that defines the guarantee of delivery for a specific message. In QoS 1, a client will
17
-receive a confirmation message from the broker upon receipt. If the expected confirmation is not
18
-received within a certain time frame, the client has to retry the message. A message received by a
19
-client must be acknowledged on time as well, otherwise the broker will re-deliver the
20
-message. <sup>[1](https://vernemq.com/intro/mqtt-primer/quality_of_service.html) </sup>
21
-
22
-</details>
23
-
24
-<details>
25
-<summary>References and sources</summary>
26
-
27
-1. [Quality of service explained, VerneMQ docs](https://vernemq.com/intro/mqtt-primer/quality_of_service.html)
28
-2. [MQTT v5 docs, PUBACK description](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
29
-
30
-</details>
31
-
32
-### Troubleshooting Section
33
-
34
-<details>
35
-<summary>General approach</summary>
36
-
37
-Open the alerts Dashboard and locate the chart of this alert (`mqtt_puback_sent_reason`). Identify
38
-which PUBACK packets (by reason) triggered this alert. Inspect the reason why you server sent those
39
-responses by consulting the subsection _PUBACK Reason
40
-Code <sup>[1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)_ </sup>
41
-mentioned above.
42
-
43
-</details>
health/guides/vernemq/vernemq_mqtt_puback_unexpected.md
deleted
-51
@@ -1,51 +0,0 @@
1
-# vernemq_mqtt_puback_unexpected
2
-
3
-**Messaging | VerneMQ**
4
-
5
-A PUBACK packet is the response to a PUBLISH packet with QoS 1. The Netdata Agent monitors the
6
-number of received unexpected v3/v5 PUBACK packets in the last minute.
7
-
8
-MQTT v5 protocol provides detailed PUBACK reasons codes as opposed to MQTT v3. You can find the
9
-detailed response codes which were sent by a client or a server and their descriptions in the
10
-official documentation of MQTT in
11
-the [MQTT v5 docs, PUBACK Reason Code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901124)
12
-. The Client or Server sending the PUBACK packet must always use one of the PUBACK Reason Codes.
13
-
14
-
15
-<details>
16
-<summary>See more about QoS 1 </summary>
17
-
18
-The Quality of Service (QoS) level is an agreement between the sender of a message and the receiver
19
-of a message that defines the guarantee of delivery for a specific message. In QoS 1, a client will
20
-receive a confirmation message from the broker upon receipt. If the expected confirmation is not
21
-received within a certain time frame, the client has to retry the message. A message received by a
22
-client must be acknowledged on time as well, otherwise the broker will re-deliver the
23
-message. <sup>[1](https://vernemq.com/intro/mqtt-primer/quality_of_service.html) </sup>
24
-
25
-</details>
26
-
27
-
28
-<details>
29
-<summary>References and sources</summary>
30
-
31
-1. [Quality of service explained, VerneMQ docs](https://vernemq.com/intro/mqtt-primer/quality_of_service.html)
32
-2. [MQTT v5 docs, PUBACK description](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
33
-3. [MQTT v3 docs, PUBACK description](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718043)
34
-
35
-</details>
36
-
37
-### Troubleshooting Section
38
-
39
-<details>
40
-<summary>General approach</summary>
41
-
42
-This alert monitors the PUBACK packets for both v3 and v5 MQTT protocol. In case you didn't receive
43
-any other alerts (`vernemq_mqtt_puback_received_reason_unsuccessful`
44
-, `vernemq_mqtt_puback_sent_unsuccessful`) (in which you can consult their troubleshooting
45
-sections), that means that the unexpected PUBACK packets was came(sent) from(to) clients which are
46
-using the MQTT v3 protocol. In that case you can inspect your MQTT server access log for further
47
-investigation.
48
-
49
-
50
-</details>
51
-
health/guides/vernemq/vernemq_mqtt_pubcomp_received_reason_unsuccessful.md
deleted
-76
@@ -1,76 +0,0 @@
1
-# vernemq_mqtt_pubcomp_received_reason_unsuccessful
2
-
3
-**Messaging | VerneMQ**
4
-
5
-The PUBCOMP packet is the response to a PUBREL packet. It is the fourth and final packet of the QoS
6
-2 protocol exchange.
7
-
8
-The Netdata Agent monitors the number of received unsuccessful v5 PUBCOMP packets in the last
9
-minute.
10
-
11
-For various scenarios, there are specific DISCONNECT responses for MQTT protocol v5. You can find
12
-Message. This is not something abnormal.
13
-
14
-
15
-<details>
16
-<summary>MQTT basic concepts and more</summary>
17
-
18
-Basic concepts in every MQTT
19
-architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
20
-
21
-- _Broker_ - The broker is the server that distributes the information to the interested clients
22
- connected to the server.
23
-- _Client_ - The device that connects to broker to send or receive information.
24
-- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
25
-- _Publish_ - Clients that send information to the broker to distribute to interested clients based
26
- on the topic name.
27
-- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
28
- subscribes to a topic, any message published to the broker is distributed to the subscribers of
29
- that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
30
- topic.
31
-- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
32
- integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
33
- transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
34
- QoS 0.
35
-
36
- - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
37
- delivery. This is often referred to as fire and forget.
38
- - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
39
- received, known otherwise as acknowledged delivery.
40
- - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
41
- ensure only one copy of the message is received, known as assured delivery.
42
-
43
-- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
44
- communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
45
- of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
46
- WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
47
- documentation in
48
- the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
49
- section
50
-
51
-</details>
52
-
53
-<details>
54
-<summary>References and sources</summary>
55
-
56
-1. [Introduction to MQTT](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
57
-2. [MQTT v5 docs, PUBCOMP reason codes](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901154)
58
-
59
-</details>
60
-
61
-### Troubleshooting Section
62
-
63
-<details>
64
-<summary>General approach</summary>
65
-
66
-Open the alerts Dashboard and locate the chart of this alert (`mqtt_pubcomp_received_reason`). Inspect
67
-which PUBCOMP packets (by reason) triggered this alert. Inspect the reason why your server received
68
-those responses by consulting the subsection _OPERATION REASON
69
-CODE_ <sup>[2](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901154) </sup>
70
-which we mentioned above.
71
-
72
-For example, your server may respond to a client `QoS: packet identifier not found`, This is not an
73
-error during recovery, but at other times indicates a mismatch between the Session State on the
74
-Client and Server.
75
-
76
-</details>
health/guides/vernemq/vernemq_mqtt_pubcomp_sent_reason_unsuccessful.md
deleted
-67
@@ -1,67 +0,0 @@
1
-# vernemq_mqtt_pubcomp_sent_reason_unsuccessful
2
-
3
-**Messaging | VerneMQ**
4
-
5
-The PUBCOMP packet is the response to a PUBREL packet. It is the fourth and final packet of the QoS
6
-2 protocol exchange. The Netdata Agent monitors the number of sent unsuccessful v5 PUBCOMP packets
7
-in the last minute.
8
-
9
-<details>
10
-<summary>MQTT basic concepts and more</summary>
11
-
12
-Basic concepts in every MQTT
13
-architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
14
-
15
-- _Broker_ - The broker is the server that distributes the information to the interested clients
16
- connected to the server.
17
-- _Client_ - The device that connects to broker to send or receive information.
18
-- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
19
-- _Publish_ - Clients that send information to the broker to distribute to interested clients based
20
- on the topic name.
21
-- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
22
- subscribes to a topic, any message published to the broker is distributed to the subscribers of
23
- that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
24
- topic.
25
-- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
26
- integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
27
- transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
28
- QoS 0.
29
-
30
- - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
31
- delivery. This is often referred to as fire and forget.
32
- - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
33
- received, known otherwise as acknowledged delivery.
34
- - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
35
- ensure only one copy of the message is received, known as assured delivery.
36
-
37
-- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
38
- communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
39
- of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
40
- WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
41
- documentation in
42
- the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
43
- section
44
-
45
-</details>
46
-
47
-<details>
48
-<summary>References and sources</summary>
49
-
50
-1. [Introduction to MQTT](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
51
-2. [MQTT v5 docs, PUBCOMP reason codes](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901154)
52
-
53
-</details>
54
-
55
-### Troubleshooting Section
56
-
57
-<details>
58
-<summary>General approach</summary>
59
-
60
-Open the alerts Dashboard and locate the chart of this alert (`mqtt_pubcomp_sent_reason`). Inspect
61
-which PUBCOMP packets (by reason) triggered this alert. Inspect the reason why your server sent
62
-those responses by consulting the subsection _OPERATION REASON
63
-CODE_ <sup>[2](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901154) </sup>
64
-which we mentioned above.
65
-
66
-</details>
67
-
health/guides/vernemq/vernemq_mqtt_pubcomp_unexpected.md
deleted
-70
@@ -1,70 +0,0 @@
1
-# vernemq_mqtt_pubcomp_unexpected
2
-
3
-**Messaging | VerneMQ**
4
-
5
-The PUBCOMP packet is the response to a PUBREL packet. It is the fourth and final packet of the QoS
6
-2 protocol exchange. The Netdata Agent monitors the number of received unexpected v3/v5 PUBCOMP
7
-packets in the last minute.
8
-
9
-<details>
10
-<summary>MQTT basic concepts and more</summary>
11
-
12
-Basic concepts in every MQTT
13
-architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
14
-
15
-- _Broker_ - The broker is the server that distributes the information to the interested clients
16
- connected to the server.
17
-- _Client_ - The device that connects to broker to send or receive information.
18
-- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
19
-- _Publish_ - Clients that send information to the broker to distribute to interested clients based
20
- on the topic name.
21
-- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
22
- subscribes to a topic, any message published to the broker is distributed to the subscribers of
23
- that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
24
- topic.
25
-- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
26
- integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
27
- transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
28
- QoS 0.
29
-
30
- - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
31
- delivery. This is often referred to as fire and forget.
32
- - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
33
- received, known otherwise as acknowledged delivery.
34
- - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
35
- ensure only one copy of the message is received, known as assured delivery.
36
-
37
-- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
38
- communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
39
- of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
40
- WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
41
- documentation in
42
- the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
43
- section
44
-
45
-</details>
46
-
47
-<details>
48
-<summary>References and sources</summary>
49
-
50
-1. [Introduction to MQTT](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
51
-2. [MQTT v5 docs, PUBCOMP packets](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901151)
52
-3. [MQTT v3 docs, PUBCOMP packets](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718058)
53
-
54
-</details>
55
-
56
-### Troubleshooting Section
57
-
58
-<details>
59
-<summary>General approach</summary>
60
-
61
-This alert monitors the PUBCOMP packets for both v3 and v5 MQTT protocol. In case you didn't receive
62
-any other alerts (`vernemq_mqtt_pubcomp_received_reason_unsuccessful`
63
-, `vernemq_mqtt_pubcomp_sent_unsuccessful`) (in which you can consult their troubleshooting
64
-sections), that means that the unexpected PUBCOMP packets was received(sent) from(to) clients which
65
-are using the MQTT v3 protocol. In that case you can inspect your MQTT server access log for further
66
-investigation.
67
-
68
-
69
-</details>
70
-
health/guides/vernemq/vernemq_mqtt_publish_auth_errors.md
deleted
-72
@@ -1,72 +0,0 @@
1
-# vernemq_mqtt_publish_auth_errors
2
-
3
-**Messaging | VerneMQ**
4
-
5
-A PUBLISH Control Packet is sent from a Client to a Server or from Server to a Client to transport
6
-an Application Message. The Netdata Agent monitors the number of unauthorized v3/v5 PUBLISH attempts
7
-in the last minute.
8
-
9
-<details>
10
-<summary>MQTT basic concepts and more</summary>
11
-
12
-Basic concepts in every MQTT
13
-architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
14
-
15
-- _Broker_ - The broker is the server that distributes the information to the interested clients
16
- connected to the server.
17
-- _Client_ - The device that connects to broker to send or receive information.
18
-- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
19
-- _Publish_ - Clients that send information to the broker to distribute to interested clients based
20
- on the topic name.
21
-- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
22
- subscribes to a topic, any message published to the broker is distributed to the subscribers of
23
- that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
24
- topic.
25
-- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
26
- integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
27
- transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
28
- QoS 0.
29
-
30
- - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
31
- delivery. This is often refered to as fire and forget.
32
- - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
33
- received, known otherwise as acknowledged delivery.
34
- - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
35
- ensure only one copy of the message is received, known as assured delivery.
36
-
37
-- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
38
- communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
39
- of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
40
- WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
41
- documentation in
42
- the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
43
- section
44
-
45
-</details>
46
-
47
-<details>
48
-<summary>References and sources</summary>
49
-
50
-1. [MQTT basic concepts](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
51
-2. [MQTT v5 docs PUBLISH packets](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
52
-
53
-</details>
54
-
55
-### Troubleshooting Section
56
-
57
-<details>
58
-<summary>General approach </summary>
59
-
60
-You may encounter authorization errors either from a misconfigured MQTT client or from a server side
61
-error (for example a misconfigured web hook.)
62
-
63
-In any case you should inspect
64
-your [logs](https://docs.vernemq.com/configuring-vernemq/logging#console-logging) to check with which
65
-client(s) these issues are raised. These kinds of issues appear in the warning log level,
66
-so you may have to set your log level appropriately.
67
-
68
-```
69
-root@netdata # cat /var/log/vernemq/console.log | grep "auth_on_publish"
70
-```
71
-
72
-</details>
\ No newline at end of file
health/guides/vernemq/vernemq_mqtt_publish_errors.md
deleted
-72
@@ -1,72 +0,0 @@
1
-# vernemq_mqtt_publish_errors
2
-
3
-**Messaging | VerneMQ**
4
-
5
-A PUBLISH Control Packet is sent from a Client to a Server or from Server to a Client to transport
6
-an Application Message. The Netdata Agent monitors the number of failed v3/v5 PUBLISH operations in
7
-the last minute.
8
-
9
-<details>
10
-<summary>MQTT basic concepts and more</summary>
11
-
12
-Basic concepts in every MQTT
13
-architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
14
-
15
-- _Broker_ - The broker is the server that distributes the information to the interested clients
16
- connected to the server.
17
-- _Client_ - The device that connects to broker to send or receive information.
18
-- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
19
-- _Publish_ - Clients that send information to the broker to distribute to interested clients based
20
- on the topic name.
21
-- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
22
- subscribes to a topic, any message published to the broker is distributed to the subscribers of
23
- that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
24
- topic.
25
-- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
26
- integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
27
- transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
28
- QoS 0.
29
-
30
- - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
31
- delivery. This is often refered to as fire and forget.
32
- - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
33
- received, known otherwise as acknowledged delivery.
34
- - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
35
- ensure only one copy of the message is received, known as assured delivery.
36
-
37
-- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
38
- communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
39
- of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
40
- WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
41
- documentation in
42
- the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
43
- section
44
-
45
-</details>
46
-
47
-<details>
48
-<summary>References and sources</summary>
49
-
50
-1. [MQTT basic concepts](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
51
-2. [MQTT v5 docs PUBLISH packets](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901100)
52
-3. [Dealing with netsplits](https://docs.vernemq.com/vernemq-clustering/netsplits)
53
-
54
-</details>
55
-
56
-### Troubleshooting Section
57
-
58
-These kinds of errors can appear in cases of a network partition in your cluster (aka netsplit)
59
-
60
-<details>
61
-<summary>Check connectivity between nodes</summary>
62
-
63
-You must ensure that the connectivity between your cluster nodes is valid. As soon as the partition
64
-is healed, and connectivity reestablished, the VerneMQ nodes replicate the latest changes made to
65
-the subscription data. This includes all the changes 'accidentally' made during the Window of
66
-Uncertainty. Using Dotted Version Vectors VerneMQ ensures that convergence regarding subscription
67
-data and retained messages is eventually reached.
68
-
69
-</details>
70
-
71
-
72
-
health/guides/vernemq/vernemq_mqtt_subscribe_auth_error.md
deleted
-80
@@ -1,80 +0,0 @@
1
-# vernemq_mqtt_subscribe_auth_error
2
-
3
-**Messaging | VerneMQ**
4
-
5
-The SUBSCRIBE packet is sent from the client to the server to create one or more subscriptions. Each
6
-subscription registers a client’s interest in one or more topics. The server sends PUBLISH packets
7
-to the client in order to forward Application Messages that were published to topics that match
8
-these Subscriptions. The SUBSCRIBE packet also specifies (for each Subscription) the maximum QoS
9
-with which the server can send Application Messages to the client. The Netdata Agent monitors the
10
-number of unauthorized v3/v5 SUBSCRIBE attempts in the last minute.
11
-
12
-<details>
13
-<summary>MQTT basic concepts and more</summary>
14
-
15
-Basic concepts in every MQTT
16
-architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>:
17
-
18
-- _Broker_ - The broker is the server that distributes the information to the interested clients
19
- connected to the server.
20
-- _Client_ - The device that connects to broker to send or receive information.
21
-- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
22
-- _Publish_ - Clients that send information to the broker to distribute to interested clients based
23
- on the topic name.
24
-- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
25
- subscribes to a topic, any message published to the broker is distributed to the subscribers of
26
- that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
27
- topic.
28
-- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
29
- integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
30
- transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
31
- QoS 0.
32
-
33
- - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
34
- delivery. This is often refered to as fire and forget.
35
- - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
36
- received, known otherwise as acknowledged delivery.
37
- - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
38
- ensure only one copy of the message is received, known as assured delivery.
39
-
40
-- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
41
- communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
42
- of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
43
- WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
44
- documentation in
45
- the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
46
- section
47
-
48
-</details>
49
-
50
-<details>
51
-<summary>References and sources</summary>
52
-
53
-1. [MQTT basic concepts](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
54
-2. [MQTT v5 docs SUBSCRIBE packet](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901161)
55
-3. [auth using VerneMQ files](https://docs.vernemq.com/configuring-vernemq/file-auth)
56
-4. [auth using database](https://docs.vernemq.com/configuring-vernemq/db-auth)
57
-
58
-</details>
59
-
60
-### Troubleshooting Section
61
-
62
-<details>
63
-<summary>General approach</summary>
64
-
65
-You may encounter authentication errors either when an unauthorized client wants to subscribe to a
66
-topic or when for some reason a client's credentials are outdated.
67
-
68
-You can check which client is trying to make an unauthorized request by inspecting your VerneMQ
69
-logs.
70
-
71
-```
72
-root@netdata # cat /var/log/vernemq/console.log | grep "auth_on_subscribe"
73
-```
74
-
75
-In any case we advise you to consult the official docs to short any issue with authentication (for
76
-authentication methods using auth
77
-files <sup>[3](https://docs.vernemq.com/configuring-vernemq/file-auth) </sup> or auth via a
78
-database<sup> [4](https://docs.vernemq.com/configuring-vernemq/file-auth) </sup> )
79
-</details>
80
-
health/guides/vernemq/vernemq_mqtt_subscribe_error.md
deleted
-73
@@ -1,73 +0,0 @@
1
-# vernemq_mqtt_subscribe_error
2
-
3
-**Messaging | VerneMQ**
4
-
5
-The SUBSCRIBE packet is sent from the client to the server to create one or more subscriptions. Each
6
-subscription registers a Client’s interest in one or more Topics. The server sends PUBLISH packets
7
-to the Client in order to forward Application Messages that were published to Topics that match
8
-these subscriptions. The SUBSCRIBE packet also specifies (for each subscription) the maximum QoS
9
-with which the server can send Application Messages to the Client. The Netdata Agent monitors the
10
-number of failed v3/v5 SUBSCRIBE operations in the last minute.
11
-
12
-<details>
13
-<summary>MQTT basic concepts and more</summary>
14
-
15
-Basic concepts in every MQTT
16
-architecture <sup>[1](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all) </sup>
17
-
18
-- _Broker_ - The broker is the server that distributes the information to the interested clients
19
- connected to the server.
20
-- _Client_ - The device that connects to broker to send or receive information.
21
-- _Topic_ - The name that the message is about. Clients publish, subscribe, or do both to a topic.
22
-- _Publish_ - Clients that send information to the broker to distribute to interested clients based
23
- on the topic name.
24
-- _Subscribe_ - Clients tell the broker which topic(s) they're interested in. When a client
25
- subscribes to a topic, any message published to the broker is distributed to the subscribers of
26
- that topic. Clients can also unsubscribe to stop receiving messages from the broker about that
27
- topic.
28
-- _QoS_ - Quality of Service. Each connection can specify a quality of service to the broker with an
29
- integer value ranging from 0-2. The QoS does not affect the handling of the TCP data
30
- transmissions, only between the MQTT clients. Note: In the examples later on, we'll only be using
31
- QoS 0.
32
-
33
- - _QoS 0_ specifies at most once, or once and only once without requiring an acknowledgment of
34
- delivery. This is often refered to as fire and forget.
35
- - _QoS 1_ specifies at least once. The message is sent multiple times until an acknowledgment is
36
- received, known otherwise as acknowledged delivery.
37
- - _QoS 2_ specifies exactly once. The sender and receiver clients use a two level handshake to
38
- ensure only one copy of the message is received, known as assured delivery.
39
-
40
-- _VerneMQ WebSockets_ - WebSocket is a computer communications protocol, providing full-duplex
41
- communication channels over a single TCP connection. VerneMQ supports the WebSocket protocol out
42
- of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a
43
- WebSocket listener or Secure WebSocket listener in the `vernemq.conf`. See more in the official
44
- documentation in
45
- the [how to configure WebSocket](https://docs.vernemq.com/configuring-vernemq/websockets)
46
- section
47
-
48
-</details>
49
-
50
-<details>
51
-<summary>References and sources</summary>
52
-
53
-1. [MQTT basic concepts](https://learn.sparkfun.com/tutorials/introduction-to-mqtt/all)
54
-2. [MQTT v5 docs SUBSCRIBE packet](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901161)
55
-
56
-
57
-</details>
58
-
59
-### Troubleshooting Section
60
-
61
-These kinds of errors can appear in cases of a network partition in your cluster (aka netsplit)
62
-
63
-<details>
64
-<summary>Check connectivity between nodes</summary>
65
-
66
-You must ensure that the connectivity between your cluster nodes is valid. As soon as the partition
67
-is healed, and connectivity reestablished, the VerneMQ nodes replicate the latest changes made to
68
-the subscription data. This includes all the changes 'accidentally' made during the Window of
69
-Uncertainty. Using Dotted Version Vectors VerneMQ ensures that convergence regarding subscription
70
-data and retained messages is eventually reached.
71
-
72
-</details>
73
-
health/guides/vernemq/vernemq_mqtt_unsubscribe_error.md
deleted
-42
@@ -1,42 +0,0 @@
1
-# vernemq_mqtt_unsubscribe_error
2
-
3
-**Messaging | VerneMQ**
4
-
5
-Publish/Subscribe is a messaging pattern that aims to decouple the sending (Publisher) and
6
-receiving (Subscriber) party. A real world example could be a sport mobile app that shows you
7
-up-to-date information of a particular football game you're interested in. In this case you are the
8
-subscriber, as you express interest in this specific game. On the other side sits the publisher,
9
-which is an online reporter that feeds a system with the actual match data. This system, which is
10
-often referred as the message broker brings the two parties together by sending the new data to all
11
-interested subscribers.
12
-<sup>[1](https://vernemq.com/intro/mqtt-primer/publish-subscribe.html) <sup>
13
-
14
-An UNSUBSCRIBE packet is sent by the Client to the Server, to unsubscribe from topics. The Netdata
15
-Agent monitors the number of failed v3/v5 UNSUBSCRIBE operations in the last minute.
16
-
17
-MQTT v5 protocol provides detailed UNSUBSCRIBE reasons codes as opposed to MQTT v3. You can find the
18
-detailed response codes which was sent by a client or a server and their descriptions in the
19
-official documentation of MQTT in
20
-the [MQTT v5 docs, UNSUBSCRIBE Reason Code](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901194)
21
-
22
-
23
-<details>
24
-<summary>References and sources</summary>
25
-
26
-1. [Pub/Sub process explained on VerneMQ site](https://vernemq.com/intro/mqtt-primer/publish-subscribe.html)
27
-2. [MQTT v3 docs UNSUBSCRIBE request](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718072)
28
-3. [MQTT v5 docs UNSUBSCRIBE request](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901179)
29
-
30
-</details>
31
-
32
-### Troubleshooting Section
33
-
34
-<details>
35
-<summary>General approach</summary>
36
-
37
-Open the alerts Dashboard and locate the chart of this alert (`mqtt_unsubscribe_error`). Inspect the
38
-error log of your VerneMQ cluster in the timestamp of this alert.
39
-
40
-</details>
41
-
42
-
health/guides/vernemq/vernemq_netsplits.md
deleted
-25
@@ -1,25 +0,0 @@
1
-# vernemq_netsplits
2
-
3
-**Messaging | VerneMQ**
4
-
5
-A netsplit (also known as split-brain) is mostly the result of a failure of one or more network devices resulting in a cluster
6
-where nodes can no longer reach each other.
7
-
8
-The VerneMQ documentation explains how VerneMQ [deals with netsplits](https://docs.vernemq.com/v/master/vernemq-clustering/netsplits).
9
-
10
-The Netdata Agent monitors the number of detected netsplits within the last minute. This alert indicates a split-brain situation.
11
-
12
-### Troubleshooting section
13
-
14
-<details>
15
-<summary>Check connectivity between nodes</summary>
16
-
17
-You must ensure that the connectivity between your cluster nodes is valid. As soon as the partition
18
-is healed, and connectivity reestablished, the VerneMQ nodes replicate the latest changes made to
19
-the subscription data. This includes all the changes 'accidentally' made during the window of
20
-uncertainty. VerneMQ uses dotted version vectors to ensure that convergence regarding subscription
21
-data and retained messages is eventually reached.
22
-
23
-</details>
24
-
25
-
health/guides/vernemq_average_scheduler_utilization.md
new
+66
@@ -0,0 +1,66 @@
1
+### Understand the alert
2
+
3
+This alert is related to VerneMQ, which is an MQTT broker. The Netdata Agent calculates the average VerneMQ's scheduler utilization over the last 10 minutes. If you receive this alert, it means your VerneMQ scheduler's utilization is high, which may indicate performance issues or resource constraints.
4
+
5
+### What does scheduler utilization mean?
6
+
7
+VerneMQ uses schedulers to manage its tasks and processes. In this context, scheduler utilization represents the degree to which the VerneMQ schedulers are being used. High scheduler utilization may cause delays in processing tasks, leading to performance degradation and possibly affecting the proper functioning of the MQTT broker.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Verify the VerneMQ scheduler utilization
12
+
13
+1. To check the scheduler utilization, you can use the `vmq-admin` command like this:
14
+
15
+ ```
16
+ vmq-admin metrics show | grep scheduler
17
+ ```
18
+
19
+ This command will display the scheduler utilization percentage.
20
+
21
+- Analyze the VerneMQ MQTT traffic
22
+
23
+1. To analyze the MQTT traffic, use the `vmq-admin` `session` and `client` subcommands. These can give you insights into the current subscription and client status:
24
+
25
+ ```
26
+ vmq-admin session show
27
+ vmq-admin client show
28
+ ```
29
+
30
+ This can help you identify if there is any abnormal activity or an increase in the number of clients or subscriptions that may be affecting the scheduler's performance.
31
+
32
+- Evaluate VerneMQ system resources
33
+
34
+1. Assess CPU and memory usage of the VerneMQ process using the `top` or `htop` commands:
35
+
36
+ ```
37
+ top -p $(pgrep -f vernemq)
38
+ ```
39
+
40
+ This will show you the CPU and memory usage for the VerneMQ process. If the process is consuming too many resources, it might be affecting the scheduler's utilization.
41
+
42
+2. Evaluate the system's available resources (CPU, memory, and I/O) using commands like `vmstat`, `free`, and `iostat`.
43
+
44
+ ```
45
+ vmstat
46
+ free
47
+ iostat
48
+ ```
49
+
50
+ These commands can help you understand if your system's resources are nearing their limits or if there are any bottlenecks affecting the overall performance.
51
+
52
+3. Check the VerneMQ logs for any errors or warnings. The default location for VerneMQ logs is `/var/log/vernemq`. Look for messages that may indicate issues affecting the scheduler's performance.
53
+
54
+- Optimize VerneMQ performance or adjust resources
55
+
56
+1. If the MQTT traffic is high or has increased recently, consider scaling up your VerneMQ instance by adding more resources (CPU or memory) or by distributing the load across multiple nodes.
57
+
58
+2. If your system resources are limited, consider optimizing your VerneMQ configuration to improve performance. Some example options include adjusting the `max_online_messages`, `max_inflight_messages`, or `queue_deliver_mode`.
59
+
60
+3. If the alert persists even after evaluating and making changes to the above steps, consult the VerneMQ documentation or community for further assistance.
61
+
62
+### Useful resources
63
+
64
+1. [VerneMQ Documentation](https://vernemq.com/docs/)
65
+2. [VerneMQAdministration Guide](https://vernemq.com/docs/administration/)
66
+3. [VerneMQ Configuration Guide](https://vernemq.com/docs/configuration/)
\ No newline at end of file
health/guides/vernemq_cluster_dropped.md
new
+49
@@ -0,0 +1,49 @@
1
+### Understand the alert
2
+
3
+This alert indicates that VerneMQ, an MQTT broker, is experiencing issues with inter-node message delivery within a clustered environment. The Netdata agent calculates the amount of traffic dropped during communication with cluster nodes in the last minute. If you receive this alert, it means that the outgoing cluster buffer is full and some messages cannot be delivered.
4
+
5
+### What does dropped messages mean?
6
+
7
+Dropped messages occur when the outgoing cluster buffer becomes full, and VerneMQ cannot deliver messages between its nodes. This can happen due to a remote node being down or unreachable, causing the buffer to fill up and preventing efficient message delivery.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the connectivity and status of cluster nodes
12
+
13
+ Verify that all cluster nodes are up, running and reachable. Use `vmq-admin cluster show` to get an overview of the cluster nodes and their connectivity status.
14
+
15
+ ```
16
+ vmq-admin cluster show
17
+ ```
18
+
19
+2. Investigate logs for any errors or warnings
20
+
21
+ Inspect the logs of the VerneMQ node(s) for any errors or warning messages. This can provide insight into any potential problems related to the cluster or network.
22
+
23
+ ```
24
+ sudo journalctl -u vernemq
25
+ ```
26
+
27
+3. Increase the buffer size
28
+
29
+ If the issue persists, consider increasing the buffer size. Adjust the `outgoing_clustering_buffer_size` value in the `vernemq.conf` file.
30
+
31
+ ```
32
+ outgoing_clustering_buffer_size = <new_buffer_size>
33
+ ```
34
+
35
+ Replace `<new_buffer_size>` with a larger value, for example, doubling the current buffer size. After updating the configuration, restart the VerneMQ service to apply the changes.
36
+
37
+ ```
38
+ sudo systemctl restart vernemq
39
+ ```
40
+
41
+4. Monitor the dropped messages
42
+
43
+ Continue to monitor the dropped messages using Netdata, and check if the issue is resolved after increasing the buffer size.
44
+
45
+### Useful resources
46
+
47
+1. [VerneMQ Documentation - Clustering](https://vernemq.com/docs/clustering/)
48
+2. [VerneMQ Logging and Monitoring](https://docs.vernemq.com/monitoring-vernemq/logging)
49
+3. [Managing VerneMQ Configuration](https://docs.vernemq.com/configuration/)
\ No newline at end of file
health/guides/vernemq_mqtt_connack_sent_reason_unsuccessful.md
new
+20
@@ -0,0 +1,20 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when there is a significant increase in the number of unsuccessful v3/v5 CONNACK packets sent by the VerneMQ broker within the last minute. A higher-than-normal rate of unsuccessful CONNACKs indicates that clients are experiencing difficulties establishing a connection with the MQTT broker.
4
+
5
+### What is a CONNACK packet?
6
+
7
+A CONNACK packet is an acknowledgment packet sent by the MQTT broker to a client in response to a CONNECT command. The CONNACK packet informs the client if the connection has been accepted or rejected, which is indicated by the return code. An unsuccessful CONNACK packet indicates a rejected connection.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. **Check VerneMQ logs**: Inspect the VerneMQ logs for error messages or reasons why the connections are being rejected. By default, these logs are located at `/var/log/vernemq/console.log` and `/var/log/vernemq/error.log`. Look for entries with "CONNACK" and discern the cause of the unsuccessful connections.
12
+
13
+2. **Diagnose client configuration issues**: Analyze the rejected connection attempts' client configurations, such as incorrect credentials, unsupported protocol versions, or security settings. Debug the client-side applications, fix the configurations, and try reconnecting to the MQTT broker.
14
+
15
+3. **Evaluate broker capacity**: Check the system resources and settings of the VerneMQ broker. An overloaded broker or insufficient system resources, such as CPU and memory, can cause connection rejections. Optimize the VerneMQ configuration, upgrade the broker's hardware, or distribute the load between multiple brokers to resolve the issue.
16
+
17
+4. **Assess network issues**: Verify the network topology, firewalls, and router settings to ensure clients can reach the MQTT broker. Network latency or misconfigurations can lead to unsuccessful CONNACKs. Use monitoring tools such as `ping`, `traceroute`, or `netstat` to diagnose network issues and assess connectivity between clients and the broker.
18
+
19
+5. **Verify security settings and permissions**: Check the VerneMQ broker's security settings, including access control lists (ACL), user permissions, and authentication/authorization settings. Restricted access or incorrect permissions can lead to connection rejections. Update the security settings accordingly and test the connection again.
20
+
health/guides/vernemq_mqtt_disconnect_received_reason_not_normal.md
new
+40
@@ -0,0 +1,40 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of not normal v5 DISCONNECT packets received by VerneMQ in the last minute is above a certain threshold. This indicates that there is an issue with MQTT clients connecting to your VerneMQ MQTT broker that requires attention.
4
+
5
+### What does not normal mean?
6
+
7
+In the context of this alert, "not normal" refers to v5 DISCONNECT packets that were received with a reason code other than "normal disconnection", as specified in the MQTT v5 protocol. Normal disconnection refers to clients disconnecting gracefully without any issues.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Inspect VerneMQ logs
12
+
13
+ Check the VerneMQ logs for any relevant information about the MQTT clients that are experiencing not normal disconnects. This can provide important context to identify the root cause of the issue.
14
+
15
+ ```
16
+ sudo journalctl -u vernemq
17
+ ```
18
+
19
+2. Check the MQTT clients
20
+
21
+ Investigate the MQTT clients that are experiencing not normal disconnects. This may involve inspecting client logs or usage patterns, as well as verifying that the clients are using the correct MQTT version (v5) and have the appropriate configurations.
22
+
23
+3. Monitor VerneMQ metrics
24
+
25
+ Use the VerneMQ metrics to monitor the broker's performance and identify any sudden spikes in abnormal disconnects or other relevant metrics.
26
+
27
+ To view the VerneMQ metrics, access the VerneMQ admin interface, usually available at `http://<your_vernemq_address>:8888/metrics`.
28
+
29
+4. Review network conditions
30
+
31
+ Verify that there are no networking issues between the MQTT clients and the VerneMQ MQTT broker, as these issues could cause MQTT clients to disconnect unexpectedly.
32
+
33
+5. Review VerneMQ configuration
34
+
35
+ Review your VerneMQ configuration to ensure it is correctly set up to handle the expected MQTT client load and usage patterns.
36
+
37
+### Useful resources
38
+
39
+1. [VerneMQ documentation](https://vernemq.com/docs/)
40
+2. [MQTT v5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html)
health/guides/vernemq_mqtt_disconnect_sent_reason_not_normal.md
new
+45
@@ -0,0 +1,45 @@
1
+### Understand the alert
2
+
3
+This alert indicates that VerneMQ, a high-performance, distributed MQTT message broker, is sending an abnormal number of v5 DISCONNECT packets in the last minute. This may signify an issue in the MQTT messaging system and impact the functioning of IoT devices or other MQTT clients connected to VerneMQ.
4
+
5
+### What does an abnormal v5 DISCONNECT packet mean?
6
+
7
+In MQTT v5, the DISCONNECT packet is sent by a client or server to indicate the end of a session. A "not normal" DISCONNECT packet, generally refers to a DISCONNECT packet sent with a reason code other than "Normal Disconnection" (0x00). These reason codes might include:
8
+
9
+- Protocol errors
10
+- Invalid DISCONNECT payloads
11
+- Authorization or authentication violations
12
+- Exceeded keep-alive timers
13
+- Server/connection errors
14
+- User-triggered disconnects
15
+
16
+A high number of not normal DISCONNECT packets, might indicate an issue in your MQTT infrastructure, misconfigured clients, or security breaches.
17
+
18
+### Troubleshoot the alert
19
+
20
+1. **Inspect VerneMQ logs**: VerneMQ logs can provide detailed information about connections, disconnections, and possible issues. Check the VerneMQ logs for errors and information about unusual disconnects.
21
+
22
+ ```
23
+ cat /var/log/vernemq/console.log
24
+ cat /var/log/vernemq/error.log
25
+ ```
26
+
27
+2. **Monitor VerneMQ status**: Use the `vmq-admin` command-line tool to monitor VerneMQ and view its runtime status. Check the number of connected clients, subscriptions, and sessions.
28
+
29
+ ```
30
+ sudo vmq-admin cluster show
31
+ sudo vmq-admin session show
32
+ sudo vmq-admin listener show
33
+ ```
34
+
35
+3. **Check clients and configurations**: Review client configurations for potential errors, like incorrect authentication credentials, misconfigured keep-alive timers, or invalid packet formats. If possible, isolate problematic clients and test their behavior.
36
+
37
+4. **Consider resource limitations**: If your VerneMQ instance is reaching resource limitations (CPU, memory, network), it might automatically terminate some connections to maintain performance. Monitor system resources using the `top` command or tools like Netdata.
38
+
39
+5. **Evaluate security**: If the issue persists, consider checking the security of your MQTT infrastructure. Investigate possible cyber threats, such as a DDoS attack or unauthorized clients attempting to connect.
40
+
41
+### Useful resources
42
+
43
+1. [VerneMQ Documentation](https://docs.vernemq.com/)
44
+2. [MQTT v5 Specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html)
45
+3. [Debugging MQTT Connections](https://www.hivemq.com/blog/mqtt-essentials-part-9-last-will-and-testament/)
\ No newline at end of file
health/guides/vernemq_mqtt_puback_received_reason_unsuccessful.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+This alert tracks the number of `unsuccessful v5 PUBACK packets` received by the VerneMQ broker within the last minute. If you receive this alert, there might be an issue with your MQTT clients or the packets they send to the VerneMQ broker.
4
+
5
+### What are v5 PUBACK packets?
6
+
7
+In MQTT v5, the `PUBACK` packet is sent by the server or subscriber client to acknowledge the receipt of a `PUBLISH` packet. In the MQTT v5 protocol, the `PUBACK` packet can contain a reason code indicating whether the message was successfully processed or if there was an error.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the VerneMQ logs: Analyze the logs to check for any errors or issues related to the MQTT clients or the incoming messages. VerneMQ's logs are usually located at `/var/log/vernemq/` directory, or you can check the log location in the VerneMQ configuration files.
12
+
13
+ ```
14
+ less /var/log/vernemq/console.log
15
+ less /var/log/vernemq/error.log
16
+ ```
17
+
18
+2. Verify MQTT clients' configurations: Review your MQTT clients' settings to ensure that they are configured correctly, especially the protocol version, QoS levels, and any MQTT v5 specific settings. Make any necessary adjustments and restart the clients.
19
+
20
+3. Monitor VerneMQ performance: Use the VerneMQ `vmq-admin` tool to monitor the broker's performance, check connections, subscriptions, and session information. This can help you identify potential issues affecting the processing of incoming messages.
21
+
22
+ ```
23
+ vmq-admin metrics show
24
+ vmq-admin session list
25
+ vmq-admin listener show
26
+ ```
27
+
28
+4. Check the `PUBLISH` messages: Inspect the contents of `PUBLISH` messages being sent by the MQTT clients to ensure they are correctly formatted and adhere to the MQTT v5 protocol specifications. If necessary, correct any issues and send test messages to confirm the problem is resolved.
29
+
30
+### Useful resources
31
+
32
+1. [VerneMQ documentation](https://vernemq.com/docs/)
33
+2. [MQTT v5.0 Specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html)
health/guides/vernemq_mqtt_puback_sent_reason_unsuccessful.md
new
+32
@@ -0,0 +1,32 @@
1
+### Understand the alert
2
+
3
+This alert is related to VerneMQ, an MQTT message broker. If you receive this alert, it means that an increasing number of unsuccessful v5 PUBACK packets have been sent in the last minute.
4
+
5
+### What does "unsuccessful v5 PUBACK" mean?
6
+
7
+In the MQTT protocol, when a client sends a Publish message with a Quality of Service (QoS) level 1, the message broker sends a PUBACK packet to acknowledge receipt of the message. However, MQTT v5 has added a reason code field in the PUBACK packet, allowing brokers to report any issues or errors that occurred during message delivery. An "unsuccessful v5 PUBACK" refers to a PUBACK packet that reports a delivery problem or issue.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check VerneMQ logs for possible errors or warnings: VerneMQ logs can provide valuable insights into the broker's runtime behavior, including connection issues or problems with authentication/authorization. Look for errors or warnings in the logs that could indicate the cause of the unsuccessful PUBACK packets.
12
+
13
+ ```
14
+ sudo journalctl -u vernemq
15
+ ```
16
+
17
+2. Verify client connections: Connection issues can be a possible cause of unsuccessful PUBACK packets. Use the `vmq-admin session show` command to view the client connections, and check for any abnormal behavior (e.g., frequent disconnects and reconnects).
18
+
19
+ ```
20
+ sudo vmq-admin session show
21
+ ```
22
+
23
+3. Check MQTT client logs: Review the logs from the devices that connect to your VerneMQ broker instance to verify if they encounter any issues or errors when sending messages.
24
+
25
+4. Monitor the broker's resources usage: High system load or insufficient resources may affect VerneMQ's performance and prevent it from processing PUBACK packets as expected. Use monitoring tools like `top` and `iotop` to observe CPU and I/O usage, and assess whether the broker has enough resources to handle the MQTT traffic.
26
+
27
+5. Update VerneMQ configuration: Double-check your VerneMQ settings for any misconfiguration related to QoS, message storage, or security policies that could prevent PUBACK packets from being sent or processed successfully.
28
+
29
+### Useful resources
30
+
31
+1. [VerneMQ Documentation](https://vernemq.com/docs/)
32
+2. [MQTT Version 5 Features](https://www.hivemq.com/blog/mqtt-5-foundational-changes-in-the-protocol/)
health/guides/vernemq_mqtt_puback_unexpected.md
new
+34
@@ -0,0 +1,34 @@
1
+### Understand the alert
2
+
3
+This alert is related to VerneMQ, a high-performance MQTT broker. It monitors the number of unexpected v3/v5 PUBACK packets received in the last minute. If you receive this alert, it means that there are more PUBACK packets received than expected, which could indicate an issue with your MQTT broker or your MQTT client application(s).
4
+
5
+### What are PUBACK packets?
6
+
7
+In MQTT (Message Queuing Telemetry Transport) protocol, PUBACK packets are acknowledgement packets sent by the MQTT broker to confirm the receipt of a PUBLISH message with QoS (Quality of Service) level 1. The MQTT client will wait for this acknowledgment packet before it can continue with the next transaction.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check VerneMQ logs for any unusual events, errors, or issues that could be related to the PUBACK packets. The VerneMQ logs can be found in `/var/log/vernemq` by default, or any custom location defined in the configuration file.
12
+
13
+ ```
14
+ sudo tail -f /var/log/vernemq/console.log
15
+ ```
16
+
17
+2. Investigate your MQTT client application(s) to ensure they are handling the PUBLISH messages correctly and not causing duplicate or unexpected PUBACK packets. You can use an MQTT client library that supports QoS level 1 to eliminate the possibility of custom code not following the MQTT protocol properly.
18
+
19
+3. Monitor your MQTT broker and client application(s) for any network connectivity issues that could cause unexpected PUBACK packets. You can use tools like `ping` and `traceroute` to check the network connectivity between the MQTT broker and client application(s).
20
+
21
+4. Analyze the load and performance of your MQTT broker using the various metrics provided by VerneMQ. You can access the VerneMQ status and metrics using the `vmq-admin` command:
22
+
23
+ ```
24
+ sudo vmq-admin metrics show
25
+ ```
26
+
27
+ Look for any unusual spikes or bottlenecks that could cause unexpected PUBACK packets in the output.
28
+
29
+5. If none of the above steps resolve the issue, consider reaching out to the VerneMQ community or opening a GitHub issue to seek further assistance.
30
+
31
+### Useful resources
32
+
33
+1. [VerneMQ Documentation](https://vernemq.com/docs/)
34
+2. [Understanding MQTT QoS Levels](https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels/)
health/guides/vernemq_mqtt_pubcomp_received_reason_unsuccessful.md
new
+26
@@ -0,0 +1,26 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the VerneMQ broker has received an increased number of unsuccessful MQTT v5 PUBCOMP (Publish Complete) packets in the last minute. The PUBCOMP packet is the fourth and final packet in the QoS 2 publish flow. It means that there are issues in the MQTT message delivery process at Quality of Service (QoS) level 2, which could lead to message loss or duplicated messages.
4
+
5
+### What does an unsuccessful PUBCOMP mean?
6
+
7
+An unsuccessful PUBCOMP occurs when the recipient of a PUBLISH message (subscriber) acknowledges reception but encounters a problem while processing the message. The PUBCOMP packet contains a Reason Code, indicating the outcome of processing the PUBLISH message. In a successful case, the code would be 0x00 (Success); otherwise, it would be one of the following: 0x80 (Unspecified Error), 0x83 (Implementation Specific Error), 0x87 (Not Authorized), 0xD0 (Packet Identifier in Use), or 0xD2 (Packet Identifier Not Found).
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the VerneMQ error logs: VerneMQ logs can provide valuable information on encountered errors or any misconfiguration that leads to unsuccessful PUBCOMP messages. Generally, their location is `/var/log/vernemq/console.log`, `/var/log/vernemq/error.log`, and `/var/log/vernemq/crash.log`.
12
+
13
+2. Review MQTT clients' logs: Inspect the logs of the MQTT clients that are publishing or subscribing to the messages on the VerneMQ broker. This may help you identify specific clients causing the problem or any pattern associated with unsuccessful PUBCOMP messages.
14
+
15
+3. Verify the Quality of Service (QoS) level: Check if the QoS level for PUBCOMP packets is set to 2, as required. If necessary, adjust the settings for the MQTT clients to match the expected QoS level.
16
+
17
+4. Investigate authorization and access control: If the Reason Code is related to authorization (0x87), verify that the MQTT clients involved have the correct permissions to publish and subscribe to the topics in question. Make sure that the VerneMQ Access Control List (ACL) or external authentication mechanisms are correctly configured.
18
+
19
+5. Monitor network connectivity: Unsuccessful PUBCOMP messages could be due to network issues between the MQTT clients and the VerneMQ broker. Monitor and analyze network latency or packet loss between clients and the VerneMQ server to identify any potential issues.
20
+
21
+### Useful resources
22
+
23
+1. [VerneMQ Documentation](https://vernemq.com/docs/)
24
+2. [MQTT v5 Specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html)
25
+3. [Troubleshooting VerneMQ](https://vernemq.com/docs/guide/introduction/troubleshooting/)
26
+4. [VerneMQ ACL Configuration](https://vernemq.com/docs/configuration/acl.html)
\ No newline at end of file
health/guides/vernemq_mqtt_pubcomp_sent_reason_unsuccessful.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the number of unsuccessful v5 PUBCOMP (Publish Complete) packets sent within the last minute has increased. VerneMQ is an MQTT broker, which plays a crucial role in managing and processing the message flow between MQTT clients. If you receive this alert, it implies that there are issues in the message flow, which might affect the communication between MQTT clients and the broker.
4
+
5
+### What does PUBCOMP mean?
6
+
7
+In MQTT protocol, PUBCOMP is the fourth and final packet in the Quality of Service (QoS) 2 protocol exchange. The flow consists of PUBLISH, PUBREC (Publish Received), PUBREL (Publish Release), and PUBCOMP packets. PUBCOMP is sent by the receiver (MQTT client or broker) to confirm that it has received and processed the PUBREL packet. Unsuccessful PUBCOMP packets indicate that the receiver was not able to process the message properly.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check VerneMQ logs for errors or warnings
12
+
13
+ VerneMQ logs can provide valuable information about issues with the message flow. Locate the log file (usually at `/var/log/vernemq/console.log`) and inspect it for any error messages or warnings related to the PUBCOMP packet or its predecessors (PUBLISH, PUBREC, PUBREL) in the QoS 2 flow.
14
+
15
+- Identify problematic MQTT clients
16
+
17
+ Analyze the logs to identify the MQTT clients that are frequently involved in unsuccessful PUBCOMP packets exchange. These clients might have connection or configuration issues that lead to unsuccessful PUBCOMP packets.
18
+
19
+- Validate MQTT clients configurations
20
+
21
+ Ensure that the MQTT clients involved in unsuccessful PUBCOMP packets have valid configurations and that they are compatible with the broker (VerneMQ). Check parameters such as QoS level, protocol version, authentication, etc.
22
+
23
+- Monitor VerneMQ metrics
24
+
25
+ Use Netdata or other monitoring tools to observe VerneMQ metrics and identify unusual patterns in the broker's performance. Increased load on the broker, high memory or CPU usage, slow response times, or network hiccups might contribute to unsuccessful PUBCOMP packets.
26
+
27
+- Ensure proper MQTT payload size
28
+
29
+ Unsuccessful PUBCOMP packets can be caused by oversized payload or incorrect Message ID. Verify that the payload size respects the Maximum Transmission Unit (MTU) and that the Message ID follows the MQTT protocol specifications.
30
+
31
+### Useful resources
32
+
33
+1. [VerneMQ - Troubleshooting](https://vernemq.com/docs/troubleshooting/)
34
+2. [MQTT Protocol Specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html)
35
+3. [VerneMQ - Monitoring](https://vernemq.com/docs/monitoring/)
\ No newline at end of file
health/guides/vernemq_mqtt_pubcomp_unexpected.md
new
+29
@@ -0,0 +1,29 @@
1
+### Understand the alert
2
+
3
+This alert is related to VerneMQ, a high-performance MQTT message broker. It monitors the number of unexpected PUBCOMP (publish complete) packets received in the last minute. If you receive this alert, it means there's an issue with the MQTT message flow between clients and the broker, which might lead to data inconsistencies.
4
+
5
+### What are PUBCOMP packets?
6
+
7
+In MQTT, the PUBCOMP packet is used when QoS (Quality of Service) 2 is applied. It's the fourth and final packet in the four-packet flow to ensure that messages are delivered exactly once. An unexpected PUBCOMP packet means that the client or the broker received a PUBCOMP packet that it didn't expect in the message flow, which can cause issues in processing the message correctly.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Inspect the VerneMQ logs: Check the VerneMQ logs for any error messages or unusual activity that could indicate a problem with the message flow. By default, VerneMQ logs are located in `/var/log/vernemq/`, but this might be different for your system.
12
+
13
+ ```
14
+ sudo tail -f /var/log/vernemq/console.log
15
+ sudo tail -f /var/log/vernemq/error.log
16
+ ```
17
+
18
+2. Identify problematic clients: Inspect the MQTT client logs to identify which clients are causing the unexpected PUBCOMP packets. Some MQTT client libraries provide logging features, while others might require debugging or setting a higher log level.
19
+
20
+3. Check QoS settings: Ensure that the clients and the MQTT broker have the same QoS settings to avoid inconsistencies in the four-packet flow.
21
+
22
+4. Monitor the VerneMQ metrics: Use Netdata or other monitoring tools to keep an eye on MQTT message flows and observe any anomalies that require further investigation.
23
+
24
+5. Update client libraries and VerneMQ: Ensure that all MQTT client libraries and the VerneMQ server are up-to-date to avoid any incompatibilities or bugs that could lead to unexpected behavior.
25
+
26
+### Useful resources
27
+
28
+1. [VerneMQ Documentation](https://vernemq.com/documentation/)
29
+2. [MQTT Specification - MQTT Control Packets](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901046)
health/guides/vernemq_mqtt_publish_auth_errors.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the Netdata Agent detects a spike in unauthorized MQTT v3/v5 `PUBLISH` attempts in the last minute on your VerneMQ broker. If you receive this alert, it means that there might be clients attempting to publish messages without the proper authentication, which could indicate a misconfiguration or potential security risk.
4
+
5
+### What are MQTT and VerneMQ?
6
+
7
+MQTT (Message Queuing Telemetry Transport) is a lightweight, publish-subscribe protocol designed for low-bandwidth, high-latency, or unreliable networks. VerneMQ is a high-performance, distributed MQTT broker that supports a wide range of industry standards and can handle millions of clients.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Verify the clients' credentials
12
+
13
+ To check if the clients are using the correct credentials while connecting and publishing to the VerneMQ broker, inspect their log files or debug messages to find authentication-related issues.
14
+
15
+2. Review VerneMQ broker configuration
16
+
17
+ Ensure that the VerneMQ configuration allows for proper authentication of clients. Verify that the correct authentication plugins and settings are enabled. The configuration file is usually located at `/etc/vernemq/vernemq.conf`. For more information on VerneMQ config, please refer to [VerneMQ documentation](https://vernemq.com/docs/configuration/index.html).
18
+
19
+3. Analyze VerneMQ logs
20
+
21
+ Inspect the VerneMQ logs to identify unauthorized attempts and assess any potential risks. The logs typically reside in the `/var/log/vernemq` directory, and you can tail the logs using the following command:
22
+
23
+ ```
24
+ tail -f /var/log/vernemq/console.log
25
+ ```
26
+
27
+4. Configure firewall rules
28
+
29
+ If you find unauthorized or suspicious IP addresses attempting to connect to your VerneMQ broker, consider blocking those addresses using firewall rules to prevent unauthorized access.
30
+
31
+### Useful resources
32
+
33
+1. [VerneMQ documentation](https://vernemq.com/docs/index.html)
34
+2. [Getting started with MQTT](https://mqtt.org/getting-started/)
35
+3. [MQTT Security Fundamentals](https://www.hivemq.com/mqtt-security-fundamentals/)
36
+4. [VerneMQ configuration options](https://vernemq.com/docs/configuration/)
\ No newline at end of file
health/guides/vernemq_mqtt_publish_errors.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of failed v3/v5 PUBLISH operations in the last minute for VerneMQ, an MQTT broker. If you receive this alert, it means that there is an issue with the MQTT message publishing process in your VerneMQ broker.
4
+
5
+### What is MQTT?
6
+
7
+MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol designed for constrained devices and low-bandwidth, high latency, or unreliable networks. It is based on the publish-subscribe model, where clients (devices or applications) can subscribe and publish messages to topics.
8
+
9
+### What is VerneMQ?
10
+
11
+VerneMQ is a high-performance, distributed MQTT message broker. It is designed to handle thousands of concurrent clients while providing low latency and high throughput.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the VerneMQ log files for any error messages or warnings related to the MQTT PUBLISH operation failures. The log files are usually located in the `/var/log/vernemq` directory.
16
+
17
+ ```
18
+ sudo tail -f /var/log/vernemq/vernemq.log
19
+ ```
20
+
21
+2. Check VerneMQ metrics to identify any bottlenecks in the system's performance. You can do this by using the `vmq-admin` tool, which comes with VerneMQ. Run the following command to get an overview of the broker's performance:
22
+
23
+ ```
24
+ sudo vmq-admin metrics show
25
+ ```
26
+
27
+ Pay attention to the metrics related to PUBLISH operation failures, such as `mqtt.publish.error_code.*`.
28
+
29
+3. Assess the performance of connected clients. Use the `vmq-admin` tool to list client connections along with details like the client's state and the number of published messages:
30
+
31
+ ```
32
+ sudo vmq-admin session show --client_id --is_online --is_authenticated --session_publish_errors
33
+ ```
34
+
35
+ Investigate the clients with `session_publish_errors` to find out if there's an issue with specific clients.
36
+
37
+4. Review your MQTT topic configuration, such as the retained flag, QoS levels, and the permissions for publishing to ensure your setup aligns with the intended behavior.
38
+
39
+5. If the issue persists or requires further investigation, consider examining the network conditions, such as latency or connection issues, which might hinder the MQTT PUBLISH operation's efficiency.
40
+
41
+### Useful resources
42
+
43
+1. [VerneMQ documentation](https://vernemq.com/docs/)
44
+2. [An introduction to MQTT](https://www.hivemq.com/mqtt-essentials/)
health/guides/vernemq_mqtt_pubrec_invalid_error.md
new
+34
@@ -0,0 +1,34 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the Netdata Agent monitors an unexpected increase in the number of VerneMQ v3 MQTT `PUBREC` packets received during the last minute. VerneMQ is an MQTT broker that is essential for message distribution in IoT applications. MQTT v3 is one of the protocol versions used by the MQTT brokers.
4
+
5
+### What does an invalid PUBREC packet mean?
6
+
7
+`PUBREC` is a control packet in the MQTT protocol that acknowledges receipt of a `PUBLISH` packet. This packet is used during Quality of Service (QoS) level 2 message delivery, ensuring that the message is received exactly once. An invalid `PUBREC` packet means that VerneMQ has received a `PUBREC` packet that contains incorrect, unexpected, or duplicate data.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check VerneMQ logs
12
+
13
+ Investigate the VerneMQ logs to see if there are any error messages or warnings related to the processing of `PUBREC` packets. The logs can be found in `/var/log/vernemq/console.log` or `/usr/local/var/log/vernemq/console.log`. Look for any entries with specific error messages mentioning `PUBREC`.
14
+
15
+- Check MQTT Clients
16
+
17
+ Monitor the MQTT clients that are connected to the VerneMQ broker to identify which clients are sending invalid `PUBREC` packets. Check the logs or monitoring systems of those clients to understand the root cause of the problem. They might be experiencing issues or bugs causing them to send incorrect `PUBREC` packets.
18
+
19
+- Check the MQTT topics
20
+
21
+ Monitor the MQTT topics with high levels of QoS 2 message delivery and determine if a specific topic is causing the spike in invalid `PUBREC` packets.
22
+
23
+- Upgrade or fix MQTT Clients
24
+
25
+ If the issue arises from specific client implementations, consider upgrading the MQTT client libraries, fixing any configuration issues or reporting the bug to the appropriate development teams.
26
+
27
+- Review VerneMQ configuration
28
+
29
+ Verify that the VerneMQ broker configuration is set up correctly and that MQTT v3 protocol is enabled. If necessary, adjust the configuration to better handle the volume of QoS 2 messages being processed.
30
+
31
+### Useful resources
32
+
33
+1. [VerneMQ documentation](https://vernemq.com/docs/index.html)
34
+2. [MQTT v3.1.1 specification](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html)
health/guides/vernemq_mqtt_pubrec_received_reason_unsuccessful.md
new
+26
@@ -0,0 +1,26 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the number of received unsuccessful v5 `PUBREC` packets in the last minute is higher than expected. VerneMQ is an open-source MQTT broker. MQTT is a lightweight messaging protocol for small sensors and mobile devices optimized for high-latency or unreliable networks. `PUBREC` is an MQTT packet that is part of the quality of service 2 (QoS 2) message flow for MQTT publish/subscribe model. An unsuccessful `PUBREC` could mean that there are issues with the MQTT messages being processed by the MQTT broker.
4
+
5
+### What does PUBREC mean?
6
+
7
+`PUBREC` stands for "Publish Received." In MQTT, it is part of the QoS 2 message flow to ensure end-to-end delivery of a message between clients (publishers) and subscribers connected to an MQTT broker. When a client sends a `PUBLISH` message with QoS 2, the broker acknowledges the receipt with a `PUBREC` message.
8
+
9
+### Troubleshoot the alert
10
+
11
+To address this alert and identify the root cause, follow these steps:
12
+
13
+1. **Check the VerneMQ log files**: Inspect the VerneMQ log files to find any issues or errors related to the processing of MQTT messages. Look for messages related to `PUBREC` or QoS 2 issues. The logs are typically located at `/var/log/vernemq/console.log`or `/var/log/vernemq/error.log`.
14
+
15
+2. **Monitor the VerneMQ metrics**: Check VerneMQ metrics using tools like `vmq-admin` to get insights into the broker's performance and message statistics. The command `vmq-admin metrics show` provides various metrics, including the number of received `PUBREC` and the number of unsuccessful `PUBREC` messages.
16
+
17
+3. **Verify the publisher's configuration**: Check the configuration of the MQTT clients (publishers) that are sending the QoS 2 messages to ensure a proper message flow. It's crucial to confirm that the clients are using the correct version of MQTT and adhere to the limitations set by MQTT v5, like the packet size or the maximum topic aliases used.
18
+
19
+4. **Identify unsupported features**: Some MQTT brokers may not support all MQTT v5 features. Verify that the publisher's MQTT library supports MQTT v5 features in use, such as user properties or message expiration interval, and that it is compatible with VerneMQ.
20
+
21
+5. **Analyze network conditions**: Unreliable network conditions or high traffic load may cause unsuccessful MQTT messages. Evaluate the network and identify any issues causing packet loss or latency. Often, improving the network conditions, migrating the broker/server to a stronger network, or adjusting the user's connection settings can help with such issues.
22
+
23
+### Useful resources
24
+
25
+1. [VerneMQ Documentation](https://vernemq.com/docs/)
26
+2. [MQTT v5 Specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/cs02/mqtt-v5.0-cs02.html)
health/guides/vernemq_mqtt_pubrec_sent_reason_unsuccessful.md
new
+30
@@ -0,0 +1,30 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of sent unsuccessful v5 PUBREC packets in the last minute in the VerneMQ MQTT broker. If you receive this alert, it means that there is an issue with successfully acknowledging receipt of PUBLISH packets in the MQTT system.
4
+
5
+### What does PUBREC mean?
6
+
7
+In the MQTT protocol, when a client sends a PUBLISH message with Quality of Service (QoS) level 2, it expects an acknowledgment from the server in the form of a PUBREC (Publish Received) message. This confirms the successful receipt of the PUBLISH message by the server. If a PUBREC message is marked as unsuccessful, it indicates a problem with the message acknowledgment process.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check VerneMQ log files for any errors or warnings related to unsuccessful PUBREC messages. VerneMQ logs can be found in `/var/log/vernemq` (by default) or the directory specified in your configuration file.
12
+
13
+ ```
14
+ sudo tail -f /var/log/vernemq/console.log
15
+ sudo tail -f /var/log/vernemq/error.log
16
+ ```
17
+
18
+2. Verify if any clients are having issues with the MQTT connection, such as intermittent network problems or misconfigured settings. Check the client logs for any issues and take appropriate action.
19
+
20
+3. Review the MQTT QoS settings for the clients in the system. If possible, consider lowering the QoS level to 1 or 0, which uses less resources and bandwidth. QoS level 2 might not be necessary for some use cases.
21
+
22
+4. Inspect the VerneMQ system and environment for resource bottlenecks or other performance issues. Use tools like `top`, `htop`, `vmstat`, or `iotop` to monitor system resources and identify any potential problems.
23
+
24
+5. If the issue persists, consider seeking support from the VerneMQ community or the software vendor for further assistance.
25
+
26
+### Useful resources
27
+
28
+1. [VerneMQ Documentation](https://vernemq.com/documentation.html)
29
+2. [MQTT Essentials – All Core MQTT Concepts explained](https://www.hivemq.com/mqtt-essentials/)
30
+3. [Understanding QoS Levels in MQTT](https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels/)
\ No newline at end of file
health/guides/vernemq_mqtt_pubrel_received_reason_unsuccessful.md
new
+43
@@ -0,0 +1,43 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of received `unsuccessful v5 PUBREL` packets in the last minute in the VerneMQ MQTT broker. If you receive this alert, it means that there were unsuccessful PUBREL attempts in VerneMQ, which might indicate an issue during the message delivery process.
4
+
5
+### What are MQTT and PUBREL?
6
+
7
+MQTT (Message Queuing Telemetry Transport) is a lightweight, low-code and low-latency messaging protocol that works with a subscription-based system. It utilizes a broker, like VerneMQ, to facilitate communication.
8
+
9
+A `PUBREL` packet is the third one in a QoS-2 (Quality of Service level 2) message flow. QoS-2 is the highest available level in MQTT and strives to provide once-and-only-once message delivery to subscribers. The `PUBREL` packet is sent by the publisher to acknowledge its receipt of a `PUBREC` packet and signal that it is OK to release the message.
10
+
11
+An unsuccessful `PUBREL` packet indicates that the message release process encountered issues and may not have been completed as expected.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check the VerneMQ broker logs for any unusual messages:
16
+
17
+ ```
18
+ sudo journalctl -u vernemq
19
+ ```
20
+
21
+ Look for errors or warnings that might be related to the unsuccessful `PUBREL` packets.
22
+
23
+2. Examine the configuration files of VerneMQ:
24
+
25
+ ```
26
+ cat /etc/vernemq/vernemq.conf
27
+ ```
28
+
29
+ Check if there are any misconfigurations or unsupported features that could cause issues with QoS-2 message flow. Refer to the [VerneMQ Documentation](https://docs.vernemq.com/configuration/introduction) for correct configurations.
30
+
31
+3. Analyze the clients' logs, which can be publishers or subscribers, for any errors or issues related to MQTT connections and QoS levels. Make sure the clients are using the correct QoS levels and are following the MQTT protocol.
32
+
33
+4. Monitor VerneMQ's RAM, CPU, and file descriptor usage to determine if the broker's performance is degraded. Resolve any performance bottlenecks or resource constraints to prevent further unsuccessful `PUBREL` packets.
34
+
35
+5. For in-depth analysis, enable VerneMQ's debug logs by setting `log.console.level` to `debug` in its configuration file and restarting the service. Be cautious, as this might generate large amounts of log data.
36
+
37
+6. If the issue persists, consider reaching out to the VerneMQ support channels, such as their [GitHub](https://github.com/vernemq/vernemq) repository.
38
+
39
+### Useful resources
40
+
41
+1. [VerneMQ Documentation](https://docs.vernemq.com/)
42
+2. [MQTT Essentials](https://www.hivemq.com/mqtt-essentials/)
43
+3. [Understanding MQTT QoS Levels - Part 1](https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels/)
health/guides/vernemq_mqtt_pubrel_sent_reason_unsuccessful.md
new
+49
@@ -0,0 +1,49 @@
1
+### Understand the alert
2
+
3
+This alert is related to VerneMQ, a high-performance MQTT broker. It monitors the number of unsuccessful v5 `PUBREL` packets sent in the last minute. If you receive this alert, it means that there was an issue with sending `PUBREL` packets in your VerneMQ instance.
4
+
5
+### What does PUBREL mean?
6
+
7
+`PUBREL` is a type of MQTT control packet that indicates the release of an application message from the server to the client. It is the third message in the QoS 2 (Quality of Service level 2) protocol exchange, where QoS 2 ensures that a message is delivered exactly once. An unsuccessful v5 `PUBREL` packet means that there was an error during the packet processing, and the message wasn't delivered to the client as expected.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the VerneMQ logs:
12
+
13
+ VerneMQ logs can give you valuable information about possible errors that might have occurred during the processing of `PUBREL` packets. Look for any error messages or traces related to the `PUBREL` packets in the logs.
14
+
15
+ ```
16
+ sudo journalctl -u vernemq -f
17
+ ```
18
+
19
+ Alternatively, if you're using a custom log location:
20
+
21
+ ```
22
+ tail -f /path/to/custom/log
23
+ ```
24
+
25
+2. Check the MQTT client-side logs:
26
+
27
+ Check the logs of the MQTT client that might have caused the unsuccessful `PUBREL` packets. Look for any connection issues, error messages, or traces related to the MQTT protocol exchanges.
28
+
29
+3. Ensure proper configuration for VerneMQ:
30
+
31
+ Verify that the VerneMQ configuration settings related to QoS 2 protocol timeouts and retries are correctly set. Check the VerneMQ [documentation](https://docs.vernemq.com/configuration) for guidance on the proper configuration.
32
+
33
+ ```
34
+ cat /etc/vernemq/vernemq.conf
35
+ ```
36
+
37
+4. Monitor VerneMQ metrics:
38
+
39
+ Use Netdata to monitor VerneMQ metrics to analyze the MQTT server's performance and resource usage. This can help you identify possible issues with the server.
40
+
41
+5. Address network or service issues:
42
+
43
+ If the above steps don't resolve the alert, look for possible network or service-related issues that might be causing the unsuccessful `PUBREL` packets. This could require additional investigation based on your specific infrastructure and environment.
44
+
45
+### Useful resources
46
+
47
+1. [VerneMQ - Official Documentation](https://docs.vernemq.com/)
48
+2. [MQTT Essentials: Quality of Service 2 (QoS 2)](https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels/)
49
+3. [Netdata - VerneMQ monitoring](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/vernemq)
\ No newline at end of file
health/guides/vernemq_mqtt_subscribe_auth_error.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alert indicates that there have been unauthorized MQTT (Message Queuing Telemetry Transport) v3/v5 SUBSCRIBE attempts in the last minute. This could mean that there are clients trying to subscribe to topics without proper authentication or authorization in your VerneMQ broker.
4
+
5
+### What does unauthorized subscribe mean?
6
+
7
+In the MQTT protocol, clients can subscribe to topics to receive messages published by other clients to the broker. An unauthorized subscribe occurs when a client tries to subscribe to a topic but does not have the required permissions or has not provided valid credentials.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the VerneMQ logs for unauthorized subscribe attempts:
12
+
13
+ The first step in troubleshooting this issue is to check the VerneMQ logs to identify the source of the unauthorized attempts. Look for log messages related to authentication or authorization errors in the log files (`/var/log/vernemq/console.log` or `/var/log/vernemq/error.log`).
14
+
15
+ Example log message:
16
+ ```
17
+ date time [warning] <client_id>@<client_IP> MQTT SUBSCRIBE authorization failure for user "<username>", topic "<topic_name>"
18
+ ```
19
+
20
+2. Verify client authentication and authorization configuration:
21
+
22
+ Check the client configurations to ensure they have the correct credentials (username and password) and are authorized to subscribe to the intended topics. Remember that topic permissions are case-sensitive and might have wildcards. Update the client configurations if necessary and restart the MQTT clients.
23
+
24
+3. Review the VerneMQ broker configurations:
25
+
26
+ Verify the authentication and authorization plugins or settings in the VerneMQ broker (`/etc/vernemq/vernemq.conf` or `/etc/vernemq/vmq.acl` for access control). Make sure the settings are correctly configured to allow the clients to subscribe to the intended topics. Update the configurations if necessary and restart the VerneMQ broker.
27
+
28
+4. Monitor the unauthorized subscribe attempts using the Netdata dashboard or configuration file:
29
+
30
+ Continue monitoring the unauthorized subscribe attempts using the Netdata dashboard or by configuring the alert thresholds in the Netdata configuration file. This will help you track the issue and ensure that the problem has been resolved.
31
+
32
+### Useful resources
33
+
34
+1. [VerneMQ documentation](https://vernemq.com/docs/)
35
+2. [MQTT v3.1.1 specification](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html)
36
+3. [MQTT v5.0 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html)
37
+4. [Understanding MQTT topic permissions and wildcards](http://www.steves-internet-guide.com/understanding-mqtt-topics/)
\ No newline at end of file
health/guides/vernemq_mqtt_subscribe_error.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+This alert is related to `VerneMQ`, the open-source, distributed MQTT message broker. If you receive this alert, it means that the number of failed v3/v5 `SUBSCRIBE` operations has increased in the last minute.
4
+
5
+### What do v3 and v5 SUBSCRIBE operations mean?
6
+
7
+MQTT v3 and v5 are different versions of the MQTT protocol, used for the Internet of Things (IoT) devices and their communication. The `SUBSCRIBE` operation allows a client (device) to subscribe to a specific topic and receive messages published under that topic.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check the VerneMQ logs
12
+
13
+1. Identify the location of the VerneMQ logs. The default location is `/var/log/vernemq`. If you have changed the default location, you can find it in the `vernemq.conf` file by looking for `log.console.file` and `log.error.file`.
14
+
15
+ ```
16
+ grep log.console.file /etc/vernemq/vernemq.conf
17
+ grep log.error.file /etc/vernemq/vernemq.conf
18
+ ```
19
+
20
+2. Analyze the logs for any errors or issues related to the `SUBSCRIBE` operation:
21
+
22
+ ```
23
+ tail -f /path/to/vernemq/logs
24
+ ```
25
+
26
+- Check the system resources
27
+
28
+1. Check the available resources (RAM and CPU) on your system:
29
+
30
+ ```
31
+ top
32
+ ```
33
+
34
+2. If you find that the system resources are low, consider adding more resources or stopping unnecessary processes/applications.
35
+
36
+- Check the client-side logs
37
+
38
+1. Most MQTT clients (e.g., Mosquitto, Paho, MQTT.js) provide their logs to help you identify any issues related to the `SUBSCRIBE` operation.
39
+
40
+2. Analyze the client logs for errors in connecting, subscribing, or receiving messages from the MQTT broker.
41
+
42
+- Analyze the topics and subscriptions
43
+
44
+1. Verify if there are any invalid, restricted, or forbidden topics in your MQTT broker.
45
+
46
+2. Check the ACLs (Access Control Lists) and client authentication settings in your VerneMQ `vernemq.conf` file.
47
+
48
+ ```
49
+ grep -E '^(allow_anonymous|vmq_acl.acl_file|vmq_passwd.password_file)' /etc/vernemq/vernemq.conf
50
+ ```
51
+
52
+3. Ensure the `ACLs` and authentication configuration are correct and allow the clients to subscribe to the required topics.
53
+
54
+### Useful resources
55
+
56
+1. [VerneMQ Administration](https://vernemq.com/docs/administration/)
57
+2. [VerneMQ Configuration](https://vernemq.com/docs/configuration/)
58
+3. [VerneMQ Logging](https://vernemq.com/docs/guide/internals.html#logging)
\ No newline at end of file
health/guides/vernemq_mqtt_unsubscribe_error.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of failed v3/v5 `UNSUBSCRIBE` operations in VerneMQ in the last minute. If you receive this alert, it means that there is a significant number of failed `UNSUBSCRIBE` operations, which may impact the MQTT messaging on your system.
4
+
5
+### What is VerneMQ?
6
+
7
+VerneMQ is a high-performance, distributed MQTT message broker. It provides scalable and reliable communication for Internet of Things (IoT) systems and applications.
8
+
9
+### What is an MQTT UNSUBSCRIBE operation?
10
+
11
+An `UNSUBSCRIBE` operation in MQTT protocol is a request sent by a client to the server to remove one or more topics from the subscription list. It allows clients to stop receiving messages for particular topics.
12
+
13
+### Troubleshoot the alert
14
+
15
+1. Check VerneMQ logs for any error messages or indications of issues with the `UNSUBSCRIBE` operation:
16
+
17
+ ```
18
+ sudo journalctl -u vernemq
19
+ ```
20
+
21
+ Alternatively, you may find the logs in `/var/log/vernemq/` directory, if using the default configuration:
22
+
23
+ ```
24
+ cat /var/log/vernemq/console.log
25
+ cat /var/log/vernemq/error.log
26
+ ```
27
+
28
+2. Review the VerneMQ configuration to ensure it is properly set up. The default configuration file is located at `/etc/vernemq/vernemq.conf`. Make sure that the settings are correct, especially those related to the MQTT protocol version and the supported QoS levels.
29
+
30
+3. Monitor the VerneMQ metrics using the `vmq-admin metrics show` command. This will provide you with an overview of the broker's performance and help you identify any abnormal metrics that could be related to the failed `UNSUBSCRIBE` operations:
31
+
32
+ ```
33
+ sudo vmq-admin metrics show
34
+ ```
35
+
36
+ Pay attention to the `mqtt.unsubscribe_error` metric, which indicates the number of failed `UNSUBSCRIBE` operations.
37
+
38
+4. Check the MQTT clients that are sending the `UNSUBSCRIBE` requests. It is possible that the client itself is misconfigured or has some faulty logic in its communication with the MQTT broker. Review the client's logs and configuration to identify any issues.
39
+
health/guides/vernemq_netsplits.md
new
+44
@@ -0,0 +1,44 @@
1
+### Understand the alert
2
+
3
+This alert indicates that your VerneMQ cluster has experienced a netsplit (split-brain) situation within the last minute. This can lead to inconsistencies in the cluster, and you need to troubleshoot the problem to maintain proper cluster operation.
4
+
5
+### What is a netsplit?
6
+
7
+In distributed systems, a netsplit occurs when a cluster of nodes loses connectivity to one or more nodes due to a network failure, leaving the cluster to operate in a degraded state. In the context of VerneMQ, a netsplit can lead to inconsistencies in the subscription data and retained messages.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Confirm the alert issue
12
+
13
+ Review the VerneMQ logs to check for any signs of network partitioning or netsplits.
14
+
15
+- Check connectivity between nodes
16
+
17
+ Ensure that the network connectivity between your cluster nodes is restored. You can use tools like `ping` and `traceroute` to verify network connectivity.
18
+
19
+- Inspect node status
20
+
21
+ Use the `vmq-admin cluster show` command to inspect the current status of the nodes in the VerneMQ cluster, and check for any disconnected nodes:
22
+
23
+ ```
24
+ vmq-admin cluster show
25
+ ```
26
+
27
+- Reestablish connections and heal partitions
28
+
29
+ If a node is disconnected, reconnect it using the `vmq-admin cluster join` command:
30
+
31
+ ```
32
+ vmq-admin cluster join discovery-node=IP_ADDRESS_OF_ANOTHER_NODE
33
+ ```
34
+
35
+ As soon as the partition is healed, and connectivity is reestablished, the VerneMQ nodes will replicate the latest changes made to the subscription data.
36
+
37
+- Ensure node connectivity remains active
38
+
39
+ Monitor the cluster and network to maintain consistent connectivity between the nodes. Set up monitoring tools and consider using an auto-healing or auto-scaling framework to help maintain node connectivity.
40
+
41
+### Useful resources
42
+
43
+1. [VerneMQ Clustering Guide: Netsplits](https://docs.vernemq.com/v/master/vernemq-clustering/netsplits)
44
+2. [VerneMQ Documentation](https://docs.vernemq.com/)
health/guides/vernemq_queue_message_drop.md
new
+53
@@ -0,0 +1,53 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of dropped messages in VerneMQ due to full message queues within the last minute. If you receive this alert, it means that message queues are full and VerneMQ is dropping messages. This can be a result of slow consumers, slow VerneMQ performance, or fast publishers.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the message queue length and performance metrics of VerneMQ
8
+
9
+ Monitor the current message queue length for each topic by using the command:
10
+
11
+ ```
12
+ vmq-admin metrics show | grep queue | sort | uniq -c
13
+ ```
14
+
15
+ You can also monitor VerneMQ performance metrics like CPU utilization, memory usage, and network I/O by using the `top` command:
16
+
17
+ ```
18
+ top
19
+ ```
20
+
21
+2. Identify slow consumers, slow VerneMQ, or fast publishers
22
+
23
+ Analyze the message flow and performance data to determine if the issue is caused by slow consumers, slow VerneMQ performance, or fast publishers.
24
+
25
+ - Slow Consumers: If you identify slow consumers, consider optimizing their processing capabilities or scaling them to handle more load.
26
+ - Slow VerneMQ: If VerneMQ itself is slow, consider optimizing its configuration, increasing resources, or scaling the nodes in the cluster.
27
+ - Fast Publishers: If fast publishers are causing the issue, consider rate-limiting them or breaking their input into smaller chunks.
28
+
29
+3. Increase the queue length or adjust max_online_messages
30
+
31
+ If increasing the capacity of your infrastructure is not a viable solution, consider increasing the queue length or adjusting the `max_online_messages` value in VerneMQ. This can help mitigate the issue of dropped messages due to full queues.
32
+
33
+ Update the VerneMQ configuration file (`vernemq.conf`) to set the desired `max_online_messages` value:
34
+
35
+ ```
36
+ max_online_messages=<your_desired_value>
37
+ ```
38
+
39
+ Then, restart VerneMQ to apply the changes:
40
+
41
+ ```
42
+ sudo service vernemq restart
43
+ ```
44
+
45
+4. Monitor the situation
46
+
47
+ Continue to monitor the message queue length and VerneMQ performance metrics after making changes, to ensure that the issue is resolved or mitigated.
48
+
49
+### Useful resources
50
+
51
+1. [VerneMQ Documentation](https://vernemq.com/docs/)
52
+2. [Understanding and Monitoring VerneMQ Metrics](https://docs.vernemq.com/monitoring/introduction)
53
+3. [VerneMQ Configuration Guide](https://docs.vernemq.com/configuration/introduction)
\ No newline at end of file
health/guides/vernemq_queue_message_expired.md
new
+53
@@ -0,0 +1,53 @@
1
+### Understand the alert
2
+
3
+This alert is related to VerneMQ, a scalable and open-source MQTT broker. The `vernemq_queue_message_expired` alert indicates that there is a high number of expired messages that could not be delivered in the last minute.
4
+
5
+### What does message expiration mean?
6
+
7
+In MQTT, messages are kept in queues until they are delivered to their respective subscribers. Sometimes, messages might have a specific lifespan given by the Time to Live (TTL) attribute, and if they are not delivered within this time, they expire.
8
+
9
+Expired messages are removed from the queue and are not delivered to subscribers. This usually means that clients are unable to process the incoming messages fast enough, putting the VerneMQ system under stress.
10
+
11
+### Troubleshoot the alert
12
+
13
+1. **Check VerneMQ status**: Use the `vernemq` command along with the `vmq-admin` tool to monitor the status of your VerneMQ broker:
14
+
15
+ ```
16
+ sudo vmq-admin cluster show
17
+ ```
18
+
19
+ Analyze the output to make sure that the cluster is up and running without issues.
20
+
21
+2. **Check the message rate and throughput**: You can use the `vmq-admin metrics show` command to display key metrics related to your VerneMQ cluster:
22
+
23
+ ```
24
+ sudo vmq-admin metrics show
25
+ ```
26
+
27
+ Analyze the output and identify any sudden increase in the message rate or unusual rate of message expiration.
28
+
29
+3. **Identify slow or malfunctioning clients**: VerneMQ provides a command to list all clients connected to the cluster. You can use the following command to identify slow or malfunctioning clients:
30
+
31
+ ```
32
+ sudo vmq-admin session show
33
+ ```
34
+
35
+ Check the output for clients who have a high amount of queue delay, low queued messages, or are not receiving messages properly.
36
+
37
+4. **Optimize client connections**: Increasing the message TTL or decreasing the message rate can help decrease the number of expired messages. Adjust the client settings accordingly, ensuring they match the application requirements.
38
+
39
+5. **Ensure proper resource allocation**: Check whether the VerneMQ broker has enough resources by monitoring CPU, memory, and disk usage using tools like `top`1, `vmstat`, or `iotop`.
40
+
41
+6. **Check VerneMQ logs**: VerneMQ logs can provide valuable insight into the underlying issue. Check the logs for any relevant error messages or warnings:
42
+
43
+ ```
44
+ sudo tail -f /var/log/vernemq/console.log
45
+ sudo tail -f /var/log/vernemq/error.log
46
+ ```
47
+
48
+7. **Monitor Netdata charts**: Monitor Netdata's VerneMQ dashboard to gain more insight into the behavior of your MQTT broker over time. Look for spikes in the number of expired messages, slow message delivery, or increasing message queues.
49
+
50
+### Useful resources
51
+
52
+1. [VerneMQ Documentation](https://vernemq.com/docs/)
53
+2. [How to Monitor VerneMQ MQTT broker with Netdata](https://learn.netdata.cloud/guides/monitor/vernemq.html)
health/guides/vernemq_queue_message_unhandled.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert is raised when the number of unhandled messages in the last minute, monitored by the Netdata Agent, is too high. It indicates that many messages were not delivered due to connections with `clean_session=true` in a VerneMQ messaging system.
4
+
5
+### What does clean_session=true mean?
6
+
7
+In MQTT, `clean_session=true` means that the client doesn't want to store any session state on the broker for the duration of its connection. When the session is terminated, all subscriptions and messages are deleted. The broker won't store any messages or send any missed messages once the client reconnects.
8
+
9
+### What are VerneMQ unhandled messages?
10
+
11
+Unhandled messages are messages that cannot be delivered to subscribers due to connection issues, protocol limitations, or session configurations. These messages are often related to clients' settings for `clean_session=true`, which means they don't store any session state on the broker.
12
+
13
+### Troubleshoot the alert
14
+
15
+- Identify clients causing unhandled messages
16
+
17
+ One way to find the clients causing unhandled messages is by analyzing the VerneMQ log files. Look for warning or error messages related to undelivered messages or clean sessions. The log files are typically located in `/var/log/vernemq/`.
18
+
19
+- Check clients' clean_session settings
20
+
21
+ Review your MQTT clients' configurations to verify if they have `clean_session=true`. Consider changing the setting to `clean_session=false` if you want the broker to store session state and send missed messages upon reconnection.
22
+
23
+- Monitor VerneMQ statistics
24
+
25
+ Use the following command to see an overview of the VerneMQ statistics:
26
+
27
+ ```
28
+ vmq-admin metrics show
29
+ ```
30
+
31
+ Look for metrics related to dropped or unhandled messages, such as `gauge.queue_message_unhandled`.
32
+
33
+- Examine your system resources
34
+
35
+ High unhandled message rates can also be a result of insufficient system resources. Check your system resources (CPU, memory, disk usage) and consider upgrading if necessary.
36
+
37
+### Useful resources
38
+
39
+1. [VerneMQ - An MQTT Broker](https://vernemq.com/)
40
+2. [VerneMQ Documentation: Monitoring & Metrics](https://docs.vernemq.com/monitoring/)
41
+3. [Understanding MQTT Clean Sessions, Queuing, Retained Messages and QoS](https://www.hivemq.com/blog/mqtt-essentials-part-7-persistent-session-queuing-messages/)
\ No newline at end of file
health/guides/vernemq_socket_errors.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+This alert is related to the VerneMQ MQTT broker, and it triggers when there is a high number of socket errors in the last minute. Socket errors can occur due to various reasons, such as network connectivity issues or resource contention on the system running the VerneMQ broker.
4
+
5
+### What are socket errors?
6
+
7
+Socket errors are issues related to network communication between the VerneMQ broker and its clients. They usually occur when there are problems establishing or maintaining a stable network connection between the server and clients. Examples of socket errors include connection timeouts, connection resets, unreachable hosts, and other network-related problems.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check the VerneMQ logs for more information:
12
+
13
+ VerneMQ logs can give you a better understanding of the cause of the socket errors. You can find the logs at `/var/log/vernemq/console.log` or `/var/log/vernemq/error.log`. Look for any errors or warning messages that might be related to the socket errors.
14
+
15
+2. Monitor the system's resources:
16
+
17
+ Use the `top`, `vmstat`, `iostat`, or `netstat` commands to monitor your system's resource usage, such as CPU, RAM, disk I/O, and network activity. Check if there are any resource bottlenecks or excessive usage that might be causing the socket errors.
18
+
19
+3. Check network connectivity:
20
+
21
+ Verify that there are no issues with the network connectivity between the VerneMQ broker and its clients. Use tools such as `ping`, `traceroute`, or `mtr` to check the connectivity and latency of the network.
22
+
23
+4. Make sure the VerneMQ broker is running:
24
+
25
+ Ensure that the VerneMQ broker process is running and listening for connections. You can use the `ps` command to check if the `vernemq` process is running, and the `netstat` command to verify that it's listening on the expected ports.
26
+
27
+5. Inspect client configurations and logs:
28
+
29
+ It's possible that the root cause of the socket errors is related to the MQTT clients. Check their configurations and logs for any signs of issues or misconfigurations that could be causing socket errors when connecting to the VerneMQ broker.
30
+
31
+### Useful resources
32
+
33
+1. [VerneMQ Documentation](https://vernemq.com/docs/)
health/guides/vsphere/vsphere_host_mem_usage.md
deleted
-26
@@ -1,26 +0,0 @@
1
-# vsphere_host_mem_usage
2
-
3
-## Virtual Machine | Memory
4
-
5
-This alert presents the host's memory utilization.
6
-
7
-It is raised into warning if it exceeds 90%.
8
-If the metric exceeds 98%, then the alert is raised into critical.
9
-
10
-For more information on virtual machine memory utilization,
11
-please see the [VMware Documentation](
12
- https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-115861E6-810A-43BB-8CDB-EE99CF8F3250.html)
13
- .
14
-
15
-### Troubleshooting
16
-
17
-To find out why the alert was raised, follow the steps in the [VMware Documentation](
18
-https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-115861E6-810A-43BB-8CDB-EE99CF8F3250.html)
19
-.
20
-
21
-<details><summary>References and Sources</summary>
22
-
23
-1. [VMware Documentation](
24
- https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-115861E6-810A-43BB-8CDB-EE99CF8F3250.html)
25
-
26
-</details>
health/guides/vsphere/vsphere_inbound_packets_errors_ratio.md
deleted
-48
@@ -1,48 +0,0 @@
1
-# vsphere_inbound_packets_errors_ratio
2
-
3
-## Virtual Machine | Network
4
-
5
-This alert presents the ratio of inbound packet errors for the network interface over the last 10
6
-minutes.
7
-
8
-The percentage of dropped packets is calculated over the last 10 minutes. To raise the alert, the minimum number of packets must be at least 10k within the last 10 minutes; otherwise the alert is never raised.
9
-
10
-If the value is >= 2% the alarm gets raised into the warning state.
11
-
12
-<details><summary>What are Packet Errors?</summary>
13
-
14
-A packet error means there’s something wrong with the packet. There are two types of packet
15
-errors that usually occur:
16
-- Transmission errors, where a packet is damaged on its way to its destination – like a fragile Amazon order that gets dinged up en route.
17
-- Format errors, where a packet’s format isn’t what the receiving device was expecting (or wanting). Think ordering a Coca-Cola in a restaurant and getting a Pepsi instead.
18
-
19
-Packets can easily become damaged on their way through a network. Common reasons for damaged packages are if a device is connected to Ethernet through a:
20
-- Bad cable
21
-- Bad port
22
-- Broken fiber cable
23
-- Dirty fiber connector
24
-
25
-Access points are also susceptible to packet errors. Offices often have multiple sources of
26
-high radio frequency interference thanks to Bluetooth devices, unmanaged access points,
27
-microwaves, and more. So packets traveling wirelessly are easily damaged.
28
-
29
-If a packet error occurs, TCP (Transmission Control Protocol) will resend the same information
30
-repeatedly, in hopes the data will eventually reach the destination without any problems.
31
-UDP (User Datagram Protocol) will keep trucking forward even when packets fail to reach
32
-their destination.<sup>[1](https://www.auvik.com/franklyit/blog/packet-errors-packet-discards-packet-loss/) </sup>
33
-</details>
34
-
35
-For further information, please have a look at the *References and Sources* section.
36
-
37
-<details><summary>References and Sources</summary>
38
-
39
-1. [Packet Errors](https://www.auvik.com/franklyit/blog/packet-errors-packet-discards-packet-loss/)
40
-
41
-2. [VMware Documentation](
42
- https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-6DB73F20-C99A-43D4-9EE0-3277974EF8BF.html)
43
-</details>
44
-
45
-### Troubleshooting Section
46
-
47
-To find out why the alert was raised, follow the steps in the [VMware Documentation](
48
-https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-6DB73F20-C99A-43D4-9EE0-3277974EF8BF.html).
health/guides/vsphere/vsphere_vm_mem_usage.md
deleted
-23
@@ -1,23 +0,0 @@
1
-# vsphere_vm_mem_usage
2
-
3
-## Virtual Machine | Memory
4
-
5
-This alert presents the virtual machine's memory utilization.
6
-
7
-It is raised into warning if it exceeds 90%.
8
-If the metric exceeds 98%, then the alert is raised into critical.
9
-
10
-For further information, please have a look at the *References and Sources* section.
11
-
12
-<details><summary>References and Sources</summary>
13
-
14
-[VMware Documentation](
15
- https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-81E25CBB-16D9-416B-AD6F-5A96D7CD0A2D.html)
16
-
17
-</details>
18
-
19
-### Troubleshooting section
20
-
21
-To find out why the alert was raised, follow the steps in the [VMware Documentation](
22
-https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-81E25CBB-16D9-416B-AD6F-5A96D7CD0A2D.html)
23
-.
health/guides/vsphere_cpu_usage.md
new
+29
@@ -0,0 +1,29 @@
1
+### Understand the alert
2
+
3
+The `vsphere_cpu_usage` alert monitors the average CPU utilization of virtual machines in the vSphere platform. The alert is triggered in a warning state when the CPU utilization is between 75-85% and in a critical state when it is between 85-95%.
4
+
5
+### What does high CPU usage mean?
6
+
7
+High CPU usage indicates that the virtual machine's CPU resources are being heavily utilized. This can lead to performance issues, slow response times, and decreased stability.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Confirm the high CPU usage by logging into the vSphere management console and checking the CPU performance metrics for the affected virtual machine(s).
12
+
13
+2. Identify the cause of high CPU usage:
14
+
15
+ - Check the virtual machine's running processes to identify any resource-intensive applications or services. You can use the `top` command on Linux-based virtual machines or Task Manager on Windows-based virtual machines.
16
+ - Inspect application logs and system logs for any signs of issues, errors, or crashes that could be contributing to high CPU usage.
17
+ - Verify if the virtual machine has adequate CPU resources allocated. If the virtual machine is consistently using a high percentage of its allocated CPU resources, consider increasing the allocated CPU resources.
18
+
19
+3. Remediate the issue:
20
+
21
+ - If an application or service is responsible for the high CPU usage, try restarting it or addressing the specific issue causing the problem.
22
+ - If the virtual machine is consistently using a high percentage of its allocated CPU resources, consider increasing the allocated CPU resources or optimizing the virtual machine's performance through application and OS tuning.
23
+ - Monitor the CPU usage after making changes to ensure that the issue has been resolved.
24
+
25
+### Useful resources
26
+
27
+1. [vSphere Monitoring and Performance Guide](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-0C94837C-8CA4-4A4E-9694-FE9828979A77.html)
28
+2. [Identifying and Troubleshooting CPU Performance Issues in VMware](https://kb.vmware.com/s/article/2090599)
29
+3. [Optimizing Performance on Hyper-V and VMware Virtual Machines](https://info.raindanceit.com/blog/optimizing-performance-hyper-v-vmware)
\ No newline at end of file
health/guides/vsphere_host_mem_usage.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+The `vsphere_host_mem_usage` alert is triggered when the memory utilization of a vSphere host reaches critical levels. This alert is raised to a warning level when the utilization exceeds 90% and becomes critical when it exceeds 98%. High memory utilization can lead to performance issues on the virtual machines running on the host.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Log in to the vSphere client:
8
+
9
+ Access the vSphere client to get an overview of your host's memory utilization and to identify which virtual machines are consuming the most memory.
10
+
11
+2. Identify high memory-consuming virtual machines:
12
+
13
+ In the vSphere client, go to the "Hosts and Clusters" view and select the affected host. In the "Virtual Machines" tab, you can now see the memory usage of each virtual machine running on the host. Identify any virtual machines that are consuming a high amount of memory.
14
+
15
+3. Analyze the memory usage in the virtual machines:
16
+
17
+ Connect to the high memory-consuming virtual machines and use their respective task managers (e.g., "top" command in Linux or Task Manager in Windows) to identify the applications and processes that are causing the high memory usage.
18
+
19
+4. Take action:
20
+
21
+ - If an application or process is consuming an excessive amount of memory and is not required, consider stopping it.
22
+ - Alternatively, if the application or process is essential, you may need to allocate more memory to the virtual machine or consider moving the workload to a different host with more available resources.
23
+ - Ensure the virtual machine's memory is optimally configured, as over-allocating memory may cause contention.
24
+
25
+5. Monitor the situation:
26
+
27
+ Keep an eye on the memory utilization of the host and the virtual machines after making changes. If memory utilization remains high, consider analyzing other virtual machines or adding more memory to the host.
28
+
29
+### Useful resources
30
+
31
+1. [Understanding Memory Utilization in VMware vSphere Host](https://www.altaro.com/vmware/memory-utilization-vmware-esxi/)
32
+2. [vSphere Monitoring and Performance Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-115861E6-810A-43BB-8CDB-EE99CF8F3250.html)
33
+3. [Optimizing Memory Performance in VMware vSphere](https://blogs.vmware.com/performance/2021/04/optimizing-memory-performance-in-vmware-vsphere.html)
\ No newline at end of file
health/guides/vsphere_inbound_packets_dropped.md
new
+27
@@ -0,0 +1,27 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a significant number of inbound dropped packets are detected on the network interface of a Virtual Machine (VM) over the last 10 minutes. It indicates a potential issue with the VM's network connectivity or performance.
4
+
5
+### What does inbound packets dropped mean?
6
+
7
+Inbound dropped packets refer to packets that are received by a network interface but discarded before they are processed by the VM. This can occur for various reasons, such as network congestion, errors in packet content, or insufficient resources to handle the incoming data.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. **Check for network congestion**: High network usage can lead to packet drops when the network is saturated, or bandwidth is insufficient to handle the incoming traffic. Monitor the overall network usage in your environment to identify if this is the cause.
12
+
13
+2. **Inspect network errors**: Errors in packet content, such as checksum errors or framing errors, can result in dropped packets. Examine logs at the hypervisor and VM level for any indication of network errors.
14
+
15
+3. **Check resource usage within the VM**: Inspect CPU, memory, and disk usage within the VM. High resource utilization can lead to degraded network performance and dropped packets.
16
+
17
+4. **Verify VM network configuration**: Ensure that the VM's network configuration, such as its IP address, subnet mask, and default gateway, are correctly set. Misconfigured network settings can cause network issues, including higher rates of dropped packets.
18
+
19
+5. **Check for faulty network hardware**: Damaged or malfunctioning network hardware, such as network interface cards (NICs) or cables, can result in dropped packets. Check the hardware components involved in the VM's network connection and replace any faulty components.
20
+
21
+6. **Evaluate hypervisor performance and configuration**: The performance of the hypervisor hosting the VM can also impact network performance. Ensure the hypervisor has adequate resources and is configured correctly for optimal VM network performance.
22
+
23
+### Useful resources
24
+
25
+1. [vSphere Networking Guide](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-32DA33D2-7B68-471B-AF7F-0AE5456070EC.html)
26
+2. [vSphere Troubleshooting Guide](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.troubleshooting.doc/GUID-12989131-47E7-4005-B940-5BA5F5C089CF.html)
27
+3. [VM Network Troubleshooting Best Practices](https://www.vmwareblog.org/troubleshooting-vm-network-performance-part-1/)
\ No newline at end of file
health/guides/vsphere_inbound_packets_dropped_ratio.md
new
+31
@@ -0,0 +1,31 @@
1
+### Understand the alert
2
+
3
+This alert, `vsphere_inbound_packets_dropped_ratio`, is triggered when there is a high ratio of dropped inbound packets for the network interface in a vSphere (VMware) environment for a virtual machine. If you receive this alert, it means that the network interface is experiencing packet loss on inbound traffic over the last 10 minutes, which can result in poor network performance and degraded application functionality.
4
+
5
+### What does a high ratio of dropped inbound packets mean?
6
+
7
+A high ratio of dropped inbound packets means that a significant percentage of the incoming network packets are not being processed by the virtual machine. This can be caused by various reasons, such as network congestion, faulty hardware, incorrect network configuration, or overwhelmed virtual machine resources. A high packet loss in a network can significantly degrade its performance and affect the proper functioning of applications relying on the network.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Verify the packet loss rate
12
+ - Monitor the inbound dropped packets ratio using the Netdata dashboard or any other network monitoring tool you have available. Identify trends or patterns in the packet loss and try to correlate them with any specific events or changes in the infrastructure.
13
+
14
+2. Check the network congestion
15
+ - Examine your network traffic to determine if network congestion or high network utilization is causing the dropped inbound packets. If congestion is the issue, identify and resolve the bottleneck, such as by increasing bandwidth or optimizing the network configuration.
16
+
17
+3. Assess virtual machine resources
18
+ - Review the virtual machine's CPU usage, memory usage, and disk I/O. If the resources seem to be strained, consider allocating more resources or optimizing the virtual machine for better performance.
19
+
20
+4. Inspect the network hardware
21
+ - Check the physical network hardware, such as switches, routers, and network interface cards (NICs), for any failures or connectivity issues. Replace any faulty hardware if necessary.
22
+
23
+5. Validate network configuration
24
+ - Ensure that the network configuration on the virtual machine and vSphere host is correct and properly optimized for your specific environment.
25
+
26
+6. Monitor the vSphere environment
27
+ - Review the vSphere environment and look for any issues with the host, datastore, or other virtual machines that may be contributing to the high ratio of dropped inbound packets.
28
+
29
+7. Consult VMware documentation and support
30
+ - If the issue persists, refer to VMware's official documentation and knowledge base articles for further assistance, or contact VMware support for guidance.
31
+
health/guides/vsphere_inbound_packets_errors.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+The `vsphere_inbound_packets_errors` alert is generated when there are inbound network errors in a VMware vSphere virtual machine. It calculates the number of inbound errors for the network interface in the last 10 minutes. If you receive this alert, it indicates that your virtual machine's network is experiencing errors, which could lead to issues with network performance, reliability, or availability.
4
+
5
+### Causes of network errors
6
+
7
+There are several reasons for network errors, including:
8
+
9
+1. Faulty hardware: physical problems with network adapters, cables, or switch ports.
10
+2. Configuration issues: incorrect network settings or driver issues.
11
+3. Network congestion: heavy traffic leading to packet loss or delays.
12
+4. Corrupted packets: data transmission errors caused by software bugs or electro-magnetic interference.
13
+
14
+### Troubleshoot the alert
15
+
16
+Follow these steps to troubleshoot the `vsphere_inbound_packets_errors` alert:
17
+
18
+1. Log in to the vSphere client and select the affected virtual machine.
19
+
20
+2. Check the VM's network settings:
21
+ - Verify that the network adapter is connected.
22
+ - Check if the network adapter's driver is up-to-date.
23
+
24
+3. Review network performance:
25
+ - Examine the virtual machine's performance charts to identify high network utilization or packet loss.
26
+ - Use network monitoring tools, like `ping`, `traceroute`, and `mtr`, to check the network connectivity and latency.
27
+
28
+4. Inspect the physical network:
29
+ - Look for damaged cables or disconnected switch ports.
30
+ - Ensure that the network equipment, like switches and routers, is operating correctly and is up-to-date.
31
+
32
+5. Analyze system logs:
33
+ - Check the virtual machine's logs for any network-related errors or warnings.
34
+ - Investigate the vSphere host logs for issues involving network hardware or configurations.
35
+
36
+6. If errors persist, consult VMware support or documentation for further guidance.
37
+
38
+### Useful resources
39
+
40
+1. [vSphere Networking Documentation](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-2B11DBB8-CB3C-4AFF-8885-EFEA0FC562F4.html)
41
+2. [Troubleshooting VMware Network Issues](https://kb.vmware.com/s/article/1004109)
health/guides/vsphere_inbound_packets_errors_ratio.md
new
+33
@@ -0,0 +1,33 @@
1
+### Understand the alert
2
+
3
+The `vsphere_inbound_packets_errors_ratio` alert presents the ratio of inbound packet errors for the network interface of a virtual machine (VM) in VMware vSphere. If the ratio is equal to or greater than 2% and there are at least 10k packets within a 10 minute period, the alert switches to the warning state.
4
+
5
+### What are packet errors?
6
+
7
+Packet errors occur when there's an issue with the packet during transmission. Common reasons include:
8
+
9
+1. Transmission errors, where a packet is damaged on its way to its destination.
10
+2. Format errors, where the packet's format doesn't match what the receiving device was expecting.
11
+
12
+Damaged packets can occur due to bad cables, bad ports, broken fiber cables, dirty fiber connectors, or high radio frequency interference.
13
+
14
+### Troubleshoot the alert
15
+
16
+1. Identify the affected virtual machine and its corresponding network interface by checking the alert details.
17
+
18
+2. Inspect the network hardware by checking for any visible damage or loose connections related to the affected network interface. This may include Ethernet cables, fiber cables, and connectors. Replace or repair any damaged components.
19
+
20
+3. Check for radio frequency interference from nearby devices, such as Bluetooth devices or microwaves. If interference is suspected, move or disable the interfering devices, or consider using shielded cables for network connections.
21
+
22
+4. Monitor vSphere network performance and error metrics by using VMware vSphere's monitoring tools or other third-party monitoring software, such as Netdata. This can help pinpoint which network devices, interfaces, or protocols are causing packet errors.
23
+
24
+5. Verify that network devices and virtual machines are configured correctly to ensure optimal network performance. This may include checking Quality of Service (QoS) settings, VLAN configurations, or network resource allocation.
25
+
26
+6. Update VMware vSphere to the latest version, as well as the network drivers and firmware of the physical host, to ensure compatibility and bug fixes are applied.
27
+
28
+7. If the issue persists, consider reaching out to VMware support for further assistance.
29
+
30
+### Useful resources
31
+
32
+1. [Packet Errors, Packet Discards & Packet Loss](https://www.auvik.com/franklyit/blog/packet-errors-packet-discards-packet-loss/)
33
+2. [VMware vSphere Networking Guide](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-6DB73F20-C99A-43D4-9EE0-3277974EF8BF.html)
\ No newline at end of file
health/guides/vsphere_outbound_packets_dropped.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+This alert is generated when the number of outbound `packets dropped` on a network interface of a `vSphere Virtual Machine` exceeds a specified threshold in the last 10 minutes. Packet drops are an indication of network congestion or misconfiguration, and can cause degraded performance and application slowdowns.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify the Virtual Machine (VM) and network interface experiencing the issue:
8
+
9
+ Use the details in the alert to find the Virtual Machine and network interface that triggered the alert. Note the name and location of the VM and the associated network interface.
10
+
11
+2. Check for network congestion or misconfiguration:
12
+
13
+ Possible reasons for dropped packets can include network congestion, faulty network hardware, or VM configuration issues. Common ways to check for these problems are:
14
+
15
+ - Check the performance charts in the vSphere Client for the affected VM, specifically the `Network` section, to visualize the network usage, dropped packets, and other relevant metrics.
16
+
17
+ - Verify the VM's network adapter settings are correct, such as its speed, duplex settings, and MTU size.
18
+
19
+ - Check the VM's host machine and its physical network connections for issues, like overutilization or faulty hardware.
20
+
21
+ - Review any network traffic shaping policies on the vSphere side, such as rate-limiters or Quality of Service (QoS) configurations.
22
+
23
+ - Examine the VM's guest OS network settings for configuration issues, such as incorrect IP addresses, subnet masks, or gateway settings.
24
+
25
+3. Diagnose application or protocol issues:
26
+
27
+ If the network settings and hardware appear to be functioning correctly, the dropped packets could be a result of specific application or protocol issues. Inspect the network traffic to see if it's associated with certain applications. In the VM's guest OS, use tools like `tcpdump`, `wireshark`, or `iftop` to capture network packets and check for problematic patterns, or review application logs for any network issues.
28
+
29
+4. Address the problem and monitor the situation:
30
+
31
+ Once you've identified and addressed the underlying cause of the dropped packets, continue monitoring the VM's network performance to verify that the issue has been resolved. If the alert persists or the problem comes back, consider escalating the issue to the network engineering team or VMware support for further assistance.
32
+
33
+### Useful resources
34
+
35
+1. [VMware Knowledge Base - Diagnosing Network Performance Issues](https://kb.vmware.com/s/article/1004089)
health/guides/vsphere_outbound_packets_dropped_ratio.md
new
+37
@@ -0,0 +1,37 @@
1
+### Understand the alert
2
+
3
+This alert calculates the ratio of `outbound dropped packets` for a network interface on a VMware vSphere Virtual Machine over the last 10 minutes. If you receive this alert, it means your Virtual Machine may be experiencing network performance issues due to dropped packets.
4
+
5
+### What does outbound dropped packets mean?
6
+
7
+Outbound dropped packets are network packets that are discarded by a network interface when they are supposed to be transmitted (sent) from the Virtual Machine to the destination. This can be caused by several factors, such as network congestion, insufficient buffer resources, or malfunctioning hardware.
8
+
9
+### What can cause a high ratio of outbound dropped packets?
10
+
11
+There are several possible reasons for a high ratio of outbound dropped packets, including:
12
+
13
+1. Network congestion: High traffic may cause your network interface to drop packets if it cannot process all the outbound packets fast enough.
14
+2. Insufficient buffer resources: The network interface requires buffer memory to store and process outbound packets. If not enough buffer memory is available, packets may be dropped.
15
+3. Malfunctioning hardware: Issues with network hardware, such as the network adapter, could result in dropped packets.
16
+
17
+### Troubleshoot the alert
18
+
19
+- Check for network congestion
20
+ 1. Monitor your network traffic using monitoring tools such as `vSphere Client`, `vRealize Network Insight`, or other third-party tools.
21
+ 2. Identify whether there is an increase in traffic that could be causing congestion.
22
+ 3. Resolve any issues related to the cause of the increased traffic to relieve the congestion.
23
+
24
+- Inspect buffer resources
25
+ 1. Use `vSphere Client` to check your Virtual Machine's network interface settings for correct buffer allocation.
26
+ 2. Increase buffer allocation if required or tune the buffer settings to ensure better resource usage.
27
+
28
+- Verify network hardware
29
+ 1. Check the status of the network adapter using the `vSphere Client` or the VMware vSphere Command-Line Interface (vSphere CLI). Look for any signs of errors or issues.
30
+ 2. Verify that the network adapter driver is up-to-date and compatible with your vSphere environment.
31
+ 3. Consider troubleshooting or replacing the network adapter if hardware issues are suspected.
32
+
33
+### Useful resources
34
+
35
+1. [vSphere Monitoring and Performance Documentation (VMware Documentation)](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-4D4F408E-F28E-4D34-A769-EEE9D9EB02AD.html)
36
+2. [vSphere Administration Guide](https://docs.vmware.com/en/VMware-vSphere/index.html)
37
+3. [vRealize Network Insight](https://www.vmware.com/products/vrealize-network-insight.html)
\ No newline at end of file
health/guides/vsphere_outbound_packets_errors.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+The `vsphere_outbound_packets_errors` alert is triggered when there is a high number of outbound network errors on a virtual machine's network interface in the last 10 minutes. This alert is related to the vSphere environment and indicates a possible issue with the virtual machine's network configuration or the underlying virtual network infrastructure.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Identify the virtual machine with the issue
8
+
9
+ The alert should show you the name or identifier of the virtual machine(s) facing the high number of outbound packet errors.
10
+
11
+2. Check the network interface configuration
12
+
13
+ Verify the virtual machine's network interface configuration within vSphere. Please ensure the configuration matches the expected settings and is correctly connected to the right virtual network.
14
+
15
+3. Monitor virtual network infrastructure
16
+
17
+ Inspect the virtual switches (vSwitches), port groups, and distributed switches in the vSphere environment. Look for misconfigurations, high packet loss rates, or other issues that may cause these errors.
18
+
19
+4. Check physical network infrastructure
20
+
21
+ Investigate if there are any problems with the physical network components, such as NICs (Network Interface Cards), switches, or cables. As issues at the physical layer could also result in network packet errors.
22
+
23
+5. Examine virtual machine logs
24
+
25
+ Review the virtual machine's logs for any network-related errors or warnings. This might give you more information about the root cause of the problem.
26
+
27
+6. Update network drivers and tools
28
+
29
+ Ensure that the latest version of network drivers and VMware tools are installed on the virtual machine. Outdated or incorrect drivers can result in packet errors.
30
+
31
+7. Contact support
32
+
33
+ If you cannot resolve the issue after completing the above steps, contact your vSphere support team for further assistance.
34
+
35
+### Useful resources
36
+
37
+1. [vSphere Networking Guide](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-7CB8DB92-468E-404E-BC56-EC3241BFC2C6.html)
38
+2. [VMware Network Troubleshooting](https://kb.vmware.com/s/article/1004099)
39
+3. [Troubleshooting VMware Network Performance](https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/virtual_network_performance-white-paper.pdf)
\ No newline at end of file
health/guides/vsphere_outbound_packets_errors_ratio.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the ratio of outbound errors for the network interface of a virtual machine in vSphere is greater than 1 over the last 10 minutes. Network outbound errors can include dropped, discarded, or errored packets that couldn't be transmitted by the network interface.
4
+
5
+### What are outbound packet errors?
6
+
7
+Outbound packet errors occur when a network interface is unable to transmit packets due to issues like network congestion, hardware problems, or misconfigurations. A high number of outbound packet errors can indicate problems in the network and affect the performance of the virtual machine, resulting in poor application responsiveness and reduced bandwidth.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Verify the virtual machine's network configuration.
12
+ - Check virtual machine settings in vSphere to ensure the correct network adapters are assigned and configured properly.
13
+ - Check the virtual machine's guest operating system network configuration for possible errors or misconfigurations.
14
+
15
+2. Monitor vSphere network performance counters.
16
+ - Review the network performance counters in vSphere to identify issues or bottlenecks that might be causing the outbound packet errors.
17
+
18
+3. Check the physical network.
19
+ - Verify the physical network connections to the virtual machine, including cabling, switches, and routers.
20
+ - Inspect the network hardware to ensure proper functioning and identify faulty hardware.
21
+
22
+4. Evaluate network congestion.
23
+ - High network traffic can cause congestion, leading to increased outbound packet errors. Evaluate the network's current usage and identify potential bottlenecks.
24
+
25
+5. Review vSphere network policies.
26
+ - Check the network policies applied to the virtual machine, such as rate limiting or other traffic shaping policies, that may be causing the increased rate of outbound packet errors.
27
+
28
+6. Examine applications and services.
29
+ - Review the applications and services running on the virtual machine to determine if any of them are generating excessive or abnormal network traffic, resulting in outbound packet errors.
30
+
31
+### Useful resources
32
+
33
+1. [VMware: Troubleshooting Network Performance](https://www.vmware.com/support/ws5/doc/ws_performance_network.html)
34
+2. [vSphere Networking Guide](https://docs.vmware.com/en/VMware-vSphere/7.0/vsphere-esxi-vcenter-server-70-networking-guide.pdf)
35
+3. [VMware: Monitoring Network Performance Using vSphere Web Client](https://kb.vmware.com/s/article/1004099)
health/guides/vsphere_vm_mem_usage.md
new
+20
@@ -0,0 +1,20 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when a virtual machine's memory usage in a vSphere environment is significantly higher than normal, indicating potential performance issues or insufficient memory allocation.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check memory usage**: Monitor the virtual machine's memory usage to identify potential bottlenecks or high memory consumption by applications.
8
+
9
+2. **Inspect applications**: Review applications running on the virtual machine to identify those consuming excessive memory, and determine if it's expected behavior or a memory leak.
10
+
11
+3. **Adjust memory allocation**: If the virtual machine consistently has high memory usage, consider increasing the allocated memory to improve performance.
12
+
13
+4. **Optimize applications**: Identify opportunities to optimize applications running on the virtual machine to reduce memory consumption.
14
+
15
+5. **Monitor performance**: Keep an eye on the virtual machine's performance metrics to ensure that changes in memory usage or allocation do not negatively impact performance.
16
+
17
+### Useful resources
18
+
19
+1. [VMware vSphere Documentation](https://docs.vmware.com/en/VMware-vSphere/index.html)
20
+2. [vSphere Performance Monitoring and Analysis](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-81E25CBB-16D9-416B-AD6F-5A96D7CD0A2D.html)
health/guides/web_log/1m_bad_requests.md
deleted
-161
@@ -1,161 +0,0 @@
1
-# 1m_bad_requests
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-The 4xx class of status code is intended for cases in which the client seems to have erred. Except
9
-when responding to a HEAD request, the server should include an entity containing an explanation of
10
-the error situation, and whether it is a temporary or permanent condition. These status codes are
11
-applicable to any request
12
-method. <sup>[1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4) </sup>
13
-
14
-The Netdata Agent calculates the ratio of client error HTTP requests over the last minute. This
15
-metric does not include the 401 errors.
16
-
17
-<details>
18
- <summary>Client error responses (4XX) </summary>
19
-
20
-The client error codes below contain the descriptions as provided by Mozilla. <sup>
21
-[2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
22
-</sup>
23
-
24
-- _400 Bad Request_: The server could not understand the request due to invalid syntax.
25
-
26
-- _401 Unauthorized_: Although the HTTP standard specifies "unauthorized", semantically this
27
- response means "unauthenticated". That is, the client must authenticate itself to get the
28
- requested response.
29
-
30
-- _402 Payment Required_: This response code is reserved for future use. The initial aim for
31
- creating this code was using it for digital payment systems, however this status code is used very
32
- rarely and no standard convention exists.
33
-
34
-- _403 Forbidden_: The client does not have access rights to the content; that is, it is
35
- unauthorized, so the server is refusing to give the requested resource. Unlike 401 Unauthorized,
36
- the client's identity is known to the server.
37
-
38
-- _404 Not Found_: The server can not find the requested resource. In the browser, this means the
39
- URL is not recognized. In an API, this can also mean that the endpoint is valid but the resource
40
- itself does not exist. Servers may also send this response instead of 403 Forbidden to hide the
41
- existence of a resource from an unauthorized client. This response code is probably the most well
42
- known due to its frequent occurrence on the web.
43
-
44
-- _405 Method Not Allowed_: The request method is known by the server but is not supported by the
45
- target resource. For example, an API may not allow calling DELETE to remove a resource.
46
-
47
-- _406 Not Acceptable_: This response is sent when the web server, after performing server-driven
48
- content negotiation, doesn't find any content that conforms to the criteria given by the user
49
- agent.
50
-
51
-- _407 Proxy Authentication Required_: This is similar to 401 Unauthorized but authentication is
52
- needed to be done by a proxy.
53
-
54
-- _408 Request Timeout_: This response is sent on an idle connection by some servers, even without
55
- any previous request by the client. It means that the server would like to shut down this unused
56
- connection. This response is used much more since some browsers, like Chrome, Firefox 27+, or IE9,
57
- use HTTP pre-connection mechanisms to speed up surfing. Also note that some servers merely shut
58
- down the connection without sending this message.
59
-
60
-- _409 Conflict_: This response is sent when a request conflicts with the current state of the
61
- server.
62
-
63
-- _410 Gone_: This response is sent when the requested content has been permanently deleted from
64
- server, with no forwarding address. Clients are expected to remove their caches and links to the
65
- resource. The HTTP specification intends this status code to be used for "limited-time,
66
- promotional services". APIs should not feel compelled to indicate resources that have been deleted
67
- with this status code.
68
-
69
-- _411 Length Required_: Server rejected the request because the Content-Length header field is not
70
- defined and the server requires it.
71
-
72
-- _412 Precondition Failed_: The client has indicated preconditions in its headers which the server
73
- does not meet.
74
-
75
-- _413 Payload Too Large_: Request entity is larger than limits defined by server. The server might
76
- close the connection or return an Retry-After header field.
77
-
78
-- _414 URI Too Long_: The URI requested by the client is longer than the server is willing to
79
- interpret.
80
-
81
-- _415 Unsupported Media Type_: The media format of the requested data is not supported by the
82
- server, so the server is rejecting the request.
83
-
84
-- _416 Range Not Satisfiable_: The range specified by the Range header field in the request cannot
85
- be fulfilled. It's possible that the range is outside the size of the target URI's data.
86
-
87
-- _417 Expectation Failed_: This response code means the expectation indicated by the Expect request
88
- header field cannot be met by the server.
89
-
90
-- _418 I'm a teapot_: The server refuses the attempt to brew coffee with a teapot.
91
-
92
-- _421 Misdirected Request_: The request was directed at a server that is not able to produce a
93
- response. This can be sent by a server that is not configured to produce responses for the
94
- combination of scheme and authority that are included in the request URI.
95
-
96
-- _422 Unprocessable Entity (WebDAV)_:
97
- The request was well-formed but was unable to be followed due to semantic errors.
98
-
99
-- _423 Locked (WebDAV)_:
100
- The resource that is being accessed is locked.
101
-
102
-- _424 Failed Dependency (WebDAV)_:
103
- The request failed due to failure of a previous request.
104
-
105
-- _425 Too Early_: Indicates that the server is unwilling to risk processing a request that might be
106
- replayed.
107
-
108
-- _426 Upgrade Required_: The server refuses to perform the request using the current protocol but
109
- might be willing to do so after the client upgrades to a different protocol. The server sends an
110
- Upgrade header in a 426 response to indicate the required protocol(s).
111
-
112
-- _428 Precondition Required_: The origin server requires the request to be conditional. This
113
- response is intended to prevent the 'lost update' problem, where a client GETs a resource's state,
114
- modifies it and PUTs it back to the server, when meanwhile a third party has modified the state on
115
- the server, leading to a conflict.
116
-
117
-- _429 Too Many Requests_: The user has sent too many requests in a given amount of time ("rate
118
- limiting").
119
-
120
-- _431 Request Header Fields Too Large_: The server is unwilling to process the request because its
121
- header fields are too large. The request may be resubmitted after reducing the size of the request
122
- header fields.
123
-
124
-- _451 Unavailable For Legal Reasons_: The user agent requested a resource that cannot legally be
125
- provided, such as a web page censored by a government.
126
-
127
-</details>
128
-
129
-<details>
130
- <summary>References and sources</summary>
131
-
132
-1. [https://datatracker.ietf.org/doc/html/rfc2616#section-10.4](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4)
133
-2. [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
134
-
135
-</details>
136
-
137
-### Troubleshooting section:
138
-
139
-<details>
140
-<summary>General approach</summary>
141
-
142
-
143
-To identify the HTTP response code your web server sends back:
144
-
145
-1. Open the Netdata dashboard.
146
-2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps
147
-track of exactly what error codes your web server sends out.
148
-
149
-You should also check server logs for more details about how the server is handling the requests.
150
-For example, web servers such as Apache or Nginx produce two files called access.log and error.log
151
-(by default under `/var/log/{nginx, apache2}/{access.log, error.log}`)
152
-
153
-</details>
154
-
155
-<details>
156
-<summary>Troubleshoot 404 codes in the server side </summary>
157
-
158
-The 404 requests indicate outdated links on your website or in other websites that redirect to your website.
159
-To check for dead links on your on website, use a `broken link checker` software periodically.
160
-
161
-</details>
health/guides/web_log/1m_internal_errors.md
deleted
-99
@@ -1,99 +0,0 @@
1
-# 1m_internal_errors
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-Response status codes beginning with the digit "5" indicate cases in which the server is aware that
9
-it has erred or is incapable of performing the request. Except when responding to a HEAD request,
10
-the server should include an entity containing an explanation of the error situation, and whether it
11
-is a temporary or permanent condition. User agents should display any included entity to the user.
12
-These response codes are applicable to any request
13
-method.<sup>[1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5) </sup>
14
-
15
-The Netdata Agent calculates the ratio of server error HTTP requests over the last minute.
16
-
17
-<details>
18
- <summary>Server error responses (5XX)</summary>
19
-
20
-The error codes below contain the descriptions as provided by
21
-Mozilla. <sup>[2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses) </sup>
22
-
23
-- _500 Internal Server Error_: The server has encountered a situation it does not know how to
24
- handle.
25
-
26
-- _501 Not Implemented_: The request method is not supported by the server and cannot be handled.
27
- The only methods that servers are required to support (and therefore that must not return this
28
- code)
29
- are GET and HEAD.
30
-
31
-- _502 Bad Gateway_: This error response means that the server, while working as a gateway to get a
32
- response needed to handle the request, got an invalid response.
33
-
34
-- _503 Service Unavailable_: The server is not ready to handle the request. Common causes are a
35
- server that is down for maintenance or that is overloaded. Note that together with this response,
36
- a user-friendly page explaining the problem should be sent. This response should be used for
37
- temporary conditions and the Retry-After HTTP header should, if possible, contain the estimated
38
- time before the recovery of the service. The webmaster must also take care about the
39
- caching-related headers that are sent along with this response, as these temporary condition
40
- responses should usually not be cached.
41
-
42
-- _504 Gateway Timeout_: This error response is given when the server is acting as a gateway and
43
- cannot get a response in time.
44
-
45
-- _505 HTTP Version Not Supported_: The HTTP version used in the request is not supported by the
46
- server.
47
-
48
-- _506 Variant Also Negotiates_: The server has an internal configuration error: the chosen variant
49
- resource is configured to engage in transparent content negotiation itself, and is therefore not a
50
- proper end point in the negotiation process.
51
-
52
-- _507 Insufficient Storage (WebDAV)_:
53
- The method could not be performed on the resource because the server is unable to store the
54
- representation needed to successfully complete the request.
55
-
56
-- _508 Loop Detected (WebDAV)_:
57
- The server detected an infinite loop while processing the request.
58
-
59
-- _510 Not Extended_: Further extensions to the request are required for the server to fulfill it.
60
-
61
-- _511 Network Authentication_: Required Indicates that the client needs to authenticate to gain
62
- network access.
63
-
64
-Source: [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
65
-
66
-</details>
67
-
68
-<details>
69
- <summary>References and sources</summary>
70
-
71
-1. [Server errors on Datatracker](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5)
72
-
73
-2. [HTTP server errors on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
74
-
75
-</details>
76
-
77
-### Troubleshooting section:
78
-
79
-<details>
80
-<summary>General approach</summary>
81
-
82
-To identify the HTTP response code your web server sends back:
83
-
84
-1. Open the Netdata dashboard.
85
-2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps
86
-track of exactly what error codes your web server sends out.
87
-
88
-You should also check the server error logs. For example, web servers such as Apache or Nginx
89
-produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
90
-
91
-</details>
92
-
93
-<details>
94
-<summary>Troubleshoot 500 error code </summary>
95
-
96
-One of the things that can cause HTTP 500 response errors is a misconfiguration in the `.htaccess`
97
-file of your web server.
98
-
99
-</details>
health/guides/web_log/1m_redirects.md
deleted
-86
@@ -1,86 +0,0 @@
1
-# 1m_redirects
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-The 3XX class of status code indicates that further action needs to be taken by the user agent in
9
-order to fulfill the request. The action required may be carried out by the user agent without
10
-interaction with the user if and only if the method used in the second request is GET or HEAD. A
11
-client SHOULD detect infinite redirection loops, since such loops generate network traffic for each
12
-redirection.
13
-<sup> [1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3) </sup>
14
-
15
-The Netdata Agent calculates the ratio of redirection HTTP requests over the last minute. This
16
-metric does not include the "304 Not modified" message.
17
-
18
-<details>
19
- <summary>Redirection messages (3XX)</summary>
20
-
21
-The redirect messages below contain the descriptions as provided by
22
-Mozilla.<sup> [2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages) </sup>
23
-
24
-- _300 Multiple Choice_: The request has more than one possible response. The user agent or user
25
- should choose one of them. (There is no standardized way of choosing one of the responses, but
26
- HTML links to the possibilities are recommended so the user can pick.)
27
-
28
-- _301 Moved Permanently_: The URL of the requested resource has been changed permanently. The new
29
- URL is given in the response.
30
-
31
-- _302 Found_: This response code means that the URI of requested resource has been changed
32
- temporarily. Further changes in the URI might be made in the future. Therefore, this same URI
33
- should be used by the client in future requests.
34
-
35
-- _303 See Other_: The server sent this response to direct the client to get the requested resource
36
- at another URI with a GET request.
37
-
38
-- _304 Not Modified_: This is used for caching purposes. It tells the client that the response has
39
- not been modified, so the client can continue to use the same cached version of the response.
40
-
41
-- _305 Use Proxy_: Defined in a previous version of the HTTP specification to indicate that a
42
- requested response must be accessed by a proxy. It has been deprecated due to security concerns
43
- regarding in-band configuration of a proxy.
44
-
45
-- _306 unused_: This response code is no longer used; it is just reserved. It was used in a previous
46
- version of the HTTP/1.1 specification.
47
-
48
-- _307 Temporary Redirect_: The server sends this response to direct the client to get the requested
49
- resource at another URI with same method that was used in the prior request. This has the same
50
- semantics as the 302 Found HTTP response code, with the exception that the user agent must not
51
- change the HTTP method used: if a POST was used in the first request, a POST must be used in the
52
- second request.
53
-
54
-- _308 Permanent Redirect_: This means that the resource is now permanently located at another URI,
55
- specified by the Location: HTTP Response header. This has the same semantics as the 301 Moved
56
- Permanently HTTP response code, with the exception that the user agent must not change the HTTP
57
- method used: if a POST was used in the first request, a POST must be used in the second request.
58
-
59
-</details>
60
-
61
-
62
-<details>
63
- <summary>References and sources</summary>
64
-
65
-1. [3XX codes in the HTTP protocol](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3)
66
-
67
-2. [HTTP redirection messages on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages)
68
-
69
-</details>
70
-
71
-### Troubleshooting section:
72
-
73
-<details>
74
-<summary>General approach</summary>
75
-
76
-You can identify exactly what HTTP response code your web server send back to your clients by opening the Netdata
77
-dashboard and inspecting the `detailed_response_codes` chart for your web server. This chart keeps
78
-track of exactly what error codes your web server sends out.
79
-
80
-You should also check the server error logs. For example, web servers such as Apache or Nginx
81
-produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
82
-
83
-</details>
84
-
85
-
86
-
health/guides/web_log/1m_successful.md
deleted
-117
@@ -1,117 +0,0 @@
1
-# 1m_successful
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-The Netdata Agent calculates the ratio of successful HTTP requests over the last minute. These
9
-requests consist of 1xx, 2xx, 304, 401 response codes. You receive this alert in warning when the
10
-percentage of successful requests is less than 85% and in critical when it is below 75%. This alert
11
-can indicate:
12
-
13
-- A malfunction in the services of your web server
14
-- Malicious activity towards your website
15
-- Broken links towards your servers.
16
-
17
-In most cases, the Agent will send you another alert indicating high incidences
18
-of "abnormal" HTTP requests code, for example you could also receive the `1m_bad_requests` alert.
19
-
20
-
21
-<details>
22
- <summary>See more about the response codes this alert track </summary>
23
-
24
-The response codes below contain the descriptions as provided by
25
-Mozilla. <sup> [1](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) </sup>
26
-
27
-**Information responses (1XX)**
28
-
29
-- _100 Continue_:This interim response indicates that the client should continue the request or ignore
30
- the response if the request is already finished.
31
-
32
-- _101 Switching Protocol_: This code is sent in response to an Upgrade request header from the client
33
- and indicates the protocol the server is switching to.
34
-
35
-- _102 Processing (WebDAV)_:
36
- This code indicates that the server has received and is processing the request, but no response is
37
- available yet.
38
-
39
-- _103 Early Hints_: This status code is primarily intended to be used with the link header, letting
40
- the user agent start preloading resources while the server prepares a response.
41
-
42
-**Successful responses (2XX)**
43
-
44
-- _200 OK_: The request succeeded. The result meaning of "success" depends on the HTTP method:
45
-
46
- * GET: The resource has been fetched and transmitted in the message body.
47
- * HEAD: The representation headers are included in the response without any message body.
48
- * PUT or POST: The resource describing the result of the action is transmitted in the message
49
- body.
50
- * TRACE: The message body contains the request message as received by the server.
51
-
52
-- _201 Created_: The request succeeded, and a new resource created as a result. This is typically the
53
- response sent after POST requests, or some PUT requests.
54
-
55
-- _202 Accepted_: The request has been received but not yet acted upon. It is noncommittal, since there
56
- is no way in HTTP to later send an asynchronous response indicating the outcome of the request. It
57
- is intended for cases where another process or server handles the request, or for batch
58
- processing.
59
-
60
-- _203 Non-Authoritative Information_: This response code means the returned metadata is not exactly
61
- the same as is available from the origin server, but is collected from a local or a third-party
62
- copy. This is mostly used for mirrors or backups of another resource. Except for that specific
63
- case, the 200 OK response is preferred to this status.
64
-
65
-- _204 No Content_: There is no content to send for this request, but the headers may be useful. The
66
- user agent may update its cached headers for this resource with the new ones.
67
-
68
-- _205 Reset Content_: Tells the user agent to reset the document which sent this request.
69
-
70
-- _206 Partial Content_: This response code is used when the range header is sent from the client to
71
- request only part of a resource.
72
-
73
-- _207 Multi-Status (WebDAV)_:
74
- Conveys information about multiple resources, for situations where multiple status codes might be
75
- appropriate.
76
-
77
-- _208 Already Reported (WebDAV)_:
78
- Used inside a <dav:propstat> response element to avoid repeatedly enumerating the internal members
79
- of multiple bindings to the same collection.
80
-
81
-- _226 IM Used (HTTP Delta encoding)_:
82
- The server has fulfilled a GET request for the resource, and the response is a representation of
83
- the result of one or more instance-manipulations applied to the current instance.
84
-
85
-**Redirection messages (3XX)**
86
-
87
-- _304 Not Modified _: This is used for caching purposes. It tells the client that the response has not
88
- been modified, so the client can continue to use the same cached version of the response.
89
-
90
-**Client error responses (4XX)**
91
-
92
-- _401 Unauthorized_: Although the HTTP standard specifies "unauthorized", semantically this response
93
- means "
94
- unauthenticated". That is, the client must authenticate itself to get the requested response.
95
-
96
-</details>
97
-
98
-<details>
99
- <summary>References and Sources </summary>
100
-
101
-1. [HTTP status codes on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
102
-
103
-</details>
104
-
105
-### Troubleshooting section:
106
-
107
-There are a number of reasons triggering this alert. All of them could eventually cause bad user
108
-experience with your web services.
109
-
110
-<details>
111
-<summary>General approach</summary>
112
-
113
-Identify exactly what HTTP response code your web server sent back to your clients.
114
-Open the Netdata dashboard and inspect the `detailed_response_codes` chart for your web server. This chart keeps
115
-track of exactly what error codes your web server sends out.
116
-
117
-</details>
health/guides/web_log/1m_unmatched.md
deleted
-43
@@ -1,43 +0,0 @@
1
-# 1m_unmatched
2
-
3
-**Web Server | Web log**
4
-
5
-In a webserver, all activity should be monitored. By default, most of the webservers log
6
-activity in an `access.log` file. The access log is a list of all requests for
7
-individual files that people or bots have requested from a website. Log File strings include notes
8
-about their requests for the HTML files and their embedded graphic images, along with any other
9
-associated files that are transmitted.
10
-
11
-The Netdata Agent calculates the percentage of unparsed log lines over the last minute. These are
12
-entries in the log file that didn't match in any of the common pattern operations of
13
-the webserver (1XX, 2XX, etc). This can indicate an abnormal activity on your web server, or that your server is
14
-performing operations that you cannot monitor with the Agent.
15
-
16
-Web servers like NGINX and Apache2 gives you the ability to modify the log patterns for each request.
17
-If you have done that, you will also need to adjust the Netdata Agent to parse those patterns.
18
-
19
-### Troubleshooting section:
20
-
21
-<details>
22
-<summary>Create a custom log format job</summary>
23
-
24
-This alert is triggered by the `python.d.plugin`. You must create a new job in the `web_log`
25
-collector for your Agent.
26
-
27
-1. See how you can [configure this collector](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/web_log#configuration)
28
-
29
-
30
-2. Follow the job template specified in
31
-the [default web_log.conf file](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/web_log/web_log.conf#L53-L86)
32
-, focus on the
33
-lines [83:85](https://github.com/netdata/netdata/blob/e6d9fbc4a53f1d35363e9b342231bb11627bafbd/collectors/python.d.plugin/web_log/web_log.conf#L83-L85)
34
-where you can see how you define a `custom_log_format`.
35
-
36
-
37
-3. Restart the Netdata Agent
38
- ```
39
- root@netdata # systemctl restart netdata
40
- ```
41
-</details>
42
-
43
-
health/guides/web_log/5m_requests_ratio.md
deleted
-24
@@ -1,24 +0,0 @@
1
-# 5m_requests_ratio
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-The Netdata Agent calculates the ratio of successful HTTP requests over the last 5 minutes, compared
9
-with the previous 5 minutes. These requests consist of 1xx, 2xx, 304, 401 response codes.
10
-
11
-You receive this alert in warning when the percentage of requests is increased more than 25% in the
12
-last 5 minutes compared with the previous 5 minutes and in critical when it increased more than 50%.
13
-A clear notification for this alert **will not be sent** when the ratio becomes normal again.
14
-
15
-### Troubleshooting section:
16
-
17
-This alert is not always a bad thing. It means that there is a slight increase in the
18
-requests towards to your Web server. You should just keep an eye on this metrics. If you receive
19
-this alert regularly you should consider take action in advance to avoid server overload.
20
-
21
-You can find an interesting article on actions you can take
22
-to [manage sudden traffic](https://www.nginx.com/blog/how-to-manage-sudden-traffic-surges-server-overload/)
23
-on a web server (this article is produced by the NGINX associates, but nearly same principles
24
-applied in any case)
health/guides/web_log/web_log_1m_bad_requests.md
deleted
-161
@@ -1,161 +0,0 @@
1
-# web_log_1m_bad_requests
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-The 4xx class of status code is intended for cases in which the client seems to have erred. Except
9
-when responding to a HEAD request, the server should include an entity containing an explanation of
10
-the error situation, and whether it is a temporary or permanent condition. These status codes are
11
-applicable to any request
12
-method. <sup>[1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4) </sup>
13
-
14
-The Netdata Agent calculates the ratio of client error HTTP requests over the last minute. This
15
-metric does not include the 401 errors.
16
-
17
-<details>
18
- <summary>Client error responses (4XX) </summary>
19
-
20
-The client error codes below contain the descriptions as provided by Mozilla. <sup>
21
-[2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
22
-</sup>
23
-
24
-- _400 Bad Request_: The server could not understand the request due to invalid syntax.
25
-
26
-- _401 Unauthorized_: Although the HTTP standard specifies "unauthorized", semantically this
27
- response means "unauthenticated". That is, the client must authenticate itself to get the
28
- requested response.
29
-
30
-- _402 Payment Required_: This response code is reserved for future use. The initial aim for
31
- creating this code was using it for digital payment systems, however this status code is used very
32
- rarely and no standard convention exists.
33
-
34
-- _403 Forbidden_: The client does not have access rights to the content; that is, it is
35
- unauthorized, so the server is refusing to give the requested resource. Unlike 401 Unauthorized,
36
- the client's identity is known to the server.
37
-
38
-- _404 Not Found_: The server can not find the requested resource. In the browser, this means the
39
- URL is not recognized. In an API, this can also mean that the endpoint is valid but the resource
40
- itself does not exist. Servers may also send this response instead of 403 Forbidden to hide the
41
- existence of a resource from an unauthorized client. This response code is probably the most well
42
- known due to its frequent occurrence on the web.
43
-
44
-- _405 Method Not Allowed_: The request method is known by the server but is not supported by the
45
- target resource. For example, an API may not allow calling DELETE to remove a resource.
46
-
47
-- _406 Not Acceptable_: This response is sent when the web server, after performing server-driven
48
- content negotiation, doesn't find any content that conforms to the criteria given by the user
49
- agent.
50
-
51
-- _407 Proxy Authentication Required_: This is similar to 401 Unauthorized but authentication is
52
- needed to be done by a proxy.
53
-
54
-- _408 Request Timeout_: This response is sent on an idle connection by some servers, even without
55
- any previous request by the client. It means that the server would like to shut down this unused
56
- connection. This response is used much more since some browsers, like Chrome, Firefox 27+, or IE9,
57
- use HTTP pre-connection mechanisms to speed up surfing. Also note that some servers merely shut
58
- down the connection without sending this message.
59
-
60
-- _409 Conflict_: This response is sent when a request conflicts with the current state of the
61
- server.
62
-
63
-- _410 Gone_: This response is sent when the requested content has been permanently deleted from
64
- server, with no forwarding address. Clients are expected to remove their caches and links to the
65
- resource. The HTTP specification intends this status code to be used for "limited-time,
66
- promotional services". APIs should not feel compelled to indicate resources that have been deleted
67
- with this status code.
68
-
69
-- _411 Length Required_: Server rejected the request because the Content-Length header field is not
70
- defined and the server requires it.
71
-
72
-- _412 Precondition Failed_: The client has indicated preconditions in its headers which the server
73
- does not meet.
74
-
75
-- _413 Payload Too Large_: Request entity is larger than limits defined by server. The server might
76
- close the connection or return an Retry-After header field.
77
-
78
-- _414 URI Too Long_: The URI requested by the client is longer than the server is willing to
79
- interpret.
80
-
81
-- _415 Unsupported Media Type_: The media format of the requested data is not supported by the
82
- server, so the server is rejecting the request.
83
-
84
-- _416 Range Not Satisfiable_: The range specified by the Range header field in the request cannot
85
- be fulfilled. It's possible that the range is outside the size of the target URI's data.
86
-
87
-- _417 Expectation Failed_: This response code means the expectation indicated by the Expect request
88
- header field cannot be met by the server.
89
-
90
-- _418 I'm a teapot_: The server refuses the attempt to brew coffee with a teapot.
91
-
92
-- _421 Misdirected Request_: The request was directed at a server that is not able to produce a
93
- response. This can be sent by a server that is not configured to produce responses for the
94
- combination of scheme and authority that are included in the request URI.
95
-
96
-- _422 Unprocessable Entity (WebDAV)_:
97
- The request was well-formed but was unable to be followed due to semantic errors.
98
-
99
-- _423 Locked (WebDAV)_:
100
- The resource that is being accessed is locked.
101
-
102
-- _424 Failed Dependency (WebDAV)_:
103
- The request failed due to failure of a previous request.
104
-
105
-- _425 Too Early_: Indicates that the server is unwilling to risk processing a request that might be
106
- replayed.
107
-
108
-- _426 Upgrade Required_: The server refuses to perform the request using the current protocol but
109
- might be willing to do so after the client upgrades to a different protocol. The server sends an
110
- Upgrade header in a 426 response to indicate the required protocol(s).
111
-
112
-- _428 Precondition Required_: The origin server requires the request to be conditional. This
113
- response is intended to prevent the 'lost update' problem, where a client GETs a resource's state,
114
- modifies it and PUTs it back to the server, when meanwhile a third party has modified the state on
115
- the server, leading to a conflict.
116
-
117
-- _429 Too Many Requests_: The user has sent too many requests in a given amount of time ("rate
118
- limiting").
119
-
120
-- _431 Request Header Fields Too Large_: The server is unwilling to process the request because its
121
- header fields are too large. The request may be resubmitted after reducing the size of the request
122
- header fields.
123
-
124
-- _451 Unavailable For Legal Reasons_: The user agent requested a resource that cannot legally be
125
- provided, such as a web page censored by a government.
126
-
127
-</details>
128
-
129
-<details>
130
- <summary>References and sources</summary>
131
-
132
-1. [https://datatracker.ietf.org/doc/html/rfc2616#section-10.4](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4)
133
-2. [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
134
-
135
-</details>
136
-
137
-### Troubleshooting section:
138
-
139
-<details>
140
-<summary>General approach</summary>
141
-
142
-
143
-To identify the HTTP response code your web server sends back:
144
-
145
-1. Open the Netdata dashboard.
146
-2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps
147
-track of exactly what error codes your web server sends out.
148
-
149
-You should also check server logs for more details about how the server is handling the requests.
150
-For example, web servers such as Apache or Nginx produce two files called access.log and error.log
151
-(by default under `/var/log/{nginx, apache2}/{access.log, error.log}`)
152
-
153
-</details>
154
-
155
-<details>
156
-<summary>Troubleshoot 404 codes in the server side </summary>
157
-
158
-The 404 requests indicate outdated links on your website or in other websites that redirect to your
159
-website. To check for dead links on your on website, use a `broken link checker` software periodically.
160
-
161
-</details>
health/guides/web_log/web_log_1m_internal_errors.md
deleted
-99
@@ -1,99 +0,0 @@
1
-# web_log_1m_internal_errors
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-Response status codes beginning with the digit "5" indicate cases in which the server is aware that
9
-it has erred or is incapable of performing the request. Except when responding to a HEAD request,
10
-the server should include an entity containing an explanation of the error situation, and whether it
11
-is a temporary or permanent condition. User agents should display any included entity to the user.
12
-These response codes are applicable to any request
13
-method.<sup>[1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5) </sup>
14
-
15
-The Netdata Agent calculates the ratio of server error HTTP requests over the last minute.
16
-
17
-<details>
18
- <summary>Server error responses (5XX)</summary>
19
-
20
-The error codes below contain the descriptions as provided by
21
-Mozilla. <sup>[2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses) </sup>
22
-
23
-- _500 Internal Server Error_: The server has encountered a situation it does not know how to
24
- handle.
25
-
26
-- _501 Not Implemented_: The request method is not supported by the server and cannot be handled.
27
- The only methods that servers are required to support (and therefore that must not return this
28
- code)
29
- are GET and HEAD.
30
-
31
-- _502 Bad Gateway_: This error response means that the server, while working as a gateway to get a
32
- response needed to handle the request, got an invalid response.
33
-
34
-- _503 Service Unavailable_: The server is not ready to handle the request. Common causes are a
35
- server that is down for maintenance or that is overloaded. Note that together with this response,
36
- a user-friendly page explaining the problem should be sent. This response should be used for
37
- temporary conditions and the Retry-After HTTP header should, if possible, contain the estimated
38
- time before the recovery of the service. The webmaster must also take care about the
39
- caching-related headers that are sent along with this response, as these temporary condition
40
- responses should usually not be cached.
41
-
42
-- _504 Gateway Timeout_: This error response is given when the server is acting as a gateway and
43
- cannot get a response in time.
44
-
45
-- _505 HTTP Version Not Supported_: The HTTP version used in the request is not supported by the
46
- server.
47
-
48
-- _506 Variant Also Negotiates_: The server has an internal configuration error: the chosen variant
49
- resource is configured to engage in transparent content negotiation itself, and is therefore not a
50
- proper end point in the negotiation process.
51
-
52
-- _507 Insufficient Storage (WebDAV)_:
53
- The method could not be performed on the resource because the server is unable to store the
54
- representation needed to successfully complete the request.
55
-
56
-- _508 Loop Detected (WebDAV)_:
57
- The server detected an infinite loop while processing the request.
58
-
59
-- _510 Not Extended_: Further extensions to the request are required for the server to fulfill it.
60
-
61
-- _511 Network Authentication_: Required Indicates that the client needs to authenticate to gain
62
- network access.
63
-
64
-Source: [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
65
-
66
-</details>
67
-
68
-<details>
69
- <summary>References and sources</summary>
70
-
71
-1. [Server errors on Datatracker](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5)
72
-
73
-2. [HTTP server errors on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
74
-
75
-</details>
76
-
77
-### Troubleshooting section:
78
-
79
-<details>
80
-<summary>General approach</summary>
81
-
82
-To identify the HTTP response code your web server sends back:
83
-
84
-1. Open the Netdata dashboard.
85
-2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps
86
-track of exactly what error codes your web server sends out.
87
-
88
-You should also check the server error logs. For example, web servers such as Apache or Nginx
89
-produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
90
-
91
-</details>
92
-
93
-<details>
94
-<summary>Troubleshoot 500 error code </summary>
95
-
96
-One of the things that can cause HTTP 500 response errors is a misconfiguration in the `.htaccess`
97
-file of your web server.
98
-
99
-</details>
health/guides/web_log/web_log_1m_redirects.md
deleted
-86
@@ -1,86 +0,0 @@
1
-# web_log_1m_redirects
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-The 3XX class of status code indicates that further action needs to be taken by the user agent in
9
-order to fulfill the request. The action required may be carried out by the user agent without
10
-interaction with the user if and only if the method used in the second request is GET or HEAD. A
11
-client SHOULD detect infinite redirection loops, since such loops generate network traffic for each
12
-redirection. <sup> [1](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3) </sup>
13
-
14
-The Netdata Agent calculates the ratio of redirection HTTP requests over the last minute. This
15
-metric does not include the
16
-"304 Not modified" message.
17
-
18
-<details>
19
- <summary>Redirection messages (3XX)</summary>
20
-
21
-The redirect messages below contain the descriptions as provided by
22
-Mozilla.<sup> [2](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages) </sup>
23
-
24
-- _300 Multiple Choice_: The request has more than one possible response. The user agent or user
25
- should choose one of them. (There is no standardized way of choosing one of the responses, but
26
- HTML links to the possibilities are recommended so the user can pick.)
27
-
28
-- _301 Moved Permanently_: The URL of the requested resource has been changed permanently. The new
29
- URL is given in the response.
30
-
31
-- _302 Found_: This response code means that the URI of requested resource has been changed
32
- temporarily. Further changes in the URI might be made in the future. Therefore, this same URI
33
- should be used by the client in future requests.
34
-
35
-- _303 See Other_: The server sent this response to direct the client to get the requested resource
36
- at another URI with a GET request.
37
-
38
-- _304 Not Modified_: This is used for caching purposes. It tells the client that the response has
39
- not been modified, so the client can continue to use the same cached version of the response.
40
-
41
-- _305 Use Proxy_: Defined in a previous version of the HTTP specification to indicate that a
42
- requested response must be accessed by a proxy. It has been deprecated due to security concerns
43
- regarding in-band configuration of a proxy.
44
-
45
-- _306 unused_: This response code is no longer used; it is just reserved. It was used in a previous
46
- version of the HTTP/1.1 specification.
47
-
48
-- _307 Temporary Redirect_: The server sends this response to direct the client to get the requested
49
- resource at another URI with same method that was used in the prior request. This has the same
50
- semantics as the 302 Found HTTP response code, with the exception that the user agent must not
51
- change the HTTP method used: if a POST was used in the first request, a POST must be used in the
52
- second request.
53
-
54
-- _308 Permanent Redirect_: This means that the resource is now permanently located at another URI,
55
- specified by the Location: HTTP Response header. This has the same semantics as the 301 Moved
56
- Permanently HTTP response code, with the exception that the user agent must not change the HTTP
57
- method used: if a POST was used in the first request, a POST must be used in the second request.
58
-
59
-</details>
60
-
61
-
62
-<details>
63
- <summary>References and sources</summary>
64
-
65
-1. [3XX codes in the HTTP protocol](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3)
66
-
67
-2. [HTTP redirection messages on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages)
68
-
69
-</details>
70
-
71
-### Troubleshooting section:
72
-
73
-<details>
74
-<summary>General approach</summary>
75
-
76
-You can identify exactly what HTTP response code your web server send back to your clients, by opening the Netdata
77
-dashboard and inspecting the `detailed_response_codes` chart for your web server. This chart keeps
78
-track of exactly what error codes your web server sends out.
79
-
80
-You should also check the server error logs. For example, web servers such as Apache or Nginx
81
-produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
82
-
83
-</details>
84
-
85
-
86
-
health/guides/web_log/web_log_1m_successful.md
deleted
-118
@@ -1,118 +0,0 @@
1
-# web_log_1m_successful
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-The Netdata Agent calculates the ratio of successful HTTP requests over the last minute. These
9
-requests consist of 1xx, 2xx, 304, 401 response codes. You receive this alert in warning when the
10
-percentage of successful requests is less than 85% and in critical when it is below 75%. This alert
11
-can indicate:
12
-
13
-- A malfunction in the services of your web server
14
-- Malicious activity towards your website
15
-- Broken links towards your servers.
16
-
17
-In most cases, the Agent will send you another alert indicating high incidences
18
-of "abnormal" HTTP requests code, for example you could also receive the `web_log_1m_bad_requests` alert.
19
-
20
-
21
-<details>
22
- <summary>See more about the response codes this alert track </summary>
23
-
24
-The response codes below contain the descriptions as provided by
25
-Mozilla. <sup> [1](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) </sup>
26
-
27
-**Information responses (1XX)**
28
-
29
-- _100 Continue_:This interim response indicates that the client should continue the request or ignore
30
- the response if the request is already finished.
31
-
32
-- _101 Switching Protocol_: This code is sent in response to an Upgrade request header from the client
33
- and indicates the protocol the server is switching to.
34
-
35
-- _102 Processing (WebDAV)_:
36
- This code indicates that the server has received and is processing the request, but no response is
37
- available yet.
38
-
39
-- _103 Early Hints_: This status code is primarily intended to be used with the link header, letting
40
- the user agent start preloading resources while the server prepares a response.
41
-
42
-**Successful responses (2XX)**
43
-
44
-- _200 OK_: The request succeeded. The result meaning of "success" depends on the HTTP method:
45
-
46
- * GET: The resource has been fetched and transmitted in the message body.
47
- * HEAD: The representation headers are included in the response without any message body.
48
- * PUT or POST: The resource describing the result of the action is transmitted in the message
49
- body.
50
- * TRACE: The message body contains the request message as received by the server.
51
-
52
-- _201 Created_: The request succeeded, and a new resource created as a result. This is typically the
53
- response sent after POST requests, or some PUT requests.
54
-
55
-- _202 Accepted_: The request has been received but not yet acted upon. It is noncommittal, since there
56
- is no way in HTTP to later send an asynchronous response indicating the outcome of the request. It
57
- is intended for cases where another process or server handles the request, or for batch
58
- processing.
59
-
60
-- _203 Non-Authoritative Information_: This response code means the returned metadata is not exactly
61
- the same as is available from the origin server, but is collected from a local or a third-party
62
- copy. This is mostly used for mirrors or backups of another resource. Except for that specific
63
- case, the 200 OK response is preferred to this status.
64
-
65
-- _204 No Content_: There is no content to send for this request, but the headers may be useful. The
66
- user agent may update its cached headers for this resource with the new ones.
67
-
68
-- _205 Reset Content_: Tells the user agent to reset the document which sent this request.
69
-
70
-- _206 Partial Content_: This response code is used when the range header is sent from the client to
71
- request only part of a resource.
72
-
73
-- _207 Multi-Status (WebDAV)_:
74
- Conveys information about multiple resources, for situations where multiple status codes might be
75
- appropriate.
76
-
77
-- _208 Already Reported (WebDAV)_:
78
- Used inside a <dav:propstat> response element to avoid repeatedly enumerating the internal members
79
- of multiple bindings to the same collection.
80
-
81
-- _226 IM Used (HTTP Delta encoding)_:
82
- The server has fulfilled a GET request for the resource, and the response is a representation of
83
- the result of one or more instance-manipulations applied to the current instance.
84
-
85
-**Redirection messages (3XX)**
86
-
87
-- _304 Not Modified _: This is used for caching purposes. It tells the client that the response has not
88
- been modified, so the client can continue to use the same cached version of the response.
89
-
90
-**Client error responses (4XX)**
91
-
92
-- _401 Unauthorized_: Although the HTTP standard specifies "unauthorized", semantically this response
93
- means "
94
- unauthenticated". That is, the client must authenticate itself to get the requested response.
95
-
96
-</details>
97
-
98
-<details>
99
- <summary>References and Sources </summary>
100
-
101
-1. [HTTP status codes on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
102
-
103
-</details>
104
-
105
-### Troubleshooting section:
106
-
107
-There are a number of reasons triggering this alert. All of them could eventually cause bad user
108
-experience with your web services.
109
-
110
-<details>
111
-<summary>General approach</summary>
112
-
113
-Identify exactly what HTTP response code your web server sent back to your clients.
114
-Open the Netdata dashboard and inspect the `detailed_response_codes` chart for your web server. This chart keeps
115
-track of exactly what error codes your web server sends out.
116
-
117
-</details>
118
-
health/guides/web_log/web_log_1m_unmatched.md
deleted
-43
@@ -1,43 +0,0 @@
1
-# web_log_1m_unmatched
2
-
3
-**Web Server | Web log**
4
-
5
-In a webserver, all activity should be monitored. By default, most of the webservers log
6
-activity in an `access.log` file. The access log is a list of all requests for
7
-individual files that people or bots have requested from a website. Log File strings include notes
8
-about their requests for the HTML files and their embedded graphic images, along with any other
9
-associated files that are transmitted.
10
-
11
-The Netdata Agent calculates the percentage of unparsed log lines over the last minute. These are
12
-entries in the log file that didn't match in any of the common pattern operations (1XX, 2XX, etc) of
13
-the webserver. This can indicate an abnormal activity on your web server, or that your server is
14
-performing operations that you cannot monitor with the Agent.
15
-
16
-Web servers like NGINX and Apache2 give you the ability to modify the log patterns for each request.
17
-If you have done that, you also need to adjust the Netdata Agent to parse those patterns.
18
-
19
-### Troubleshooting section:
20
-
21
-<details>
22
-<summary>Create a custom log format job</summary>
23
-
24
-This alert is triggered by the `python.d.plugin`. You must create a new job in the `web_log`
25
-collector for your Agent.
26
-
27
-1. See how you can [configure this collector](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/web_log#configuration)
28
-
29
-
30
-2. Follow the job template specified in
31
-the [default web_log.conf file](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/web_log/web_log.conf#L53-L86)
32
-, focus on the
33
-lines [83:85](https://github.com/netdata/netdata/blob/e6d9fbc4a53f1d35363e9b342231bb11627bafbd/collectors/python.d.plugin/web_log/web_log.conf#L83-L85)
34
-where you can see how you define a `custom_log_format`.
35
-
36
-
37
-3. Restart the Netdata Agent
38
- ```
39
- root@netdata # systemctl restart netdata
40
- ```
41
-</details>
42
-
43
-
health/guides/web_log/web_log_5m_requests_ratio.md
deleted
-24
@@ -1,24 +0,0 @@
1
-# web_log_5m_requests_ratio
2
-
3
-**Web Server | Web log**
4
-
5
-HTTP response status codes indicate whether a specific HTTP request has been successfully completed
6
-or not.
7
-
8
-The Netdata Agent calculates the ratio of successful HTTP requests over the last 5 minutes, compared
9
-with the previous 5 minutes. These requests consist of 1xx, 2xx, 304, 401 response codes.
10
-
11
-There is a warning alert when the percentage of requests is increased more than 25% in the
12
-last 5 minutes compared with the previous 5 minutes and in critical when it increased more than 50%.
13
-A clear notification for this alert **will not be sent** when the ratio becomes normal again.
14
-
15
-### Troubleshooting section:
16
-
17
-This alert is not always a bad thing. It means that there is a slight increase in the
18
-requests towards to your Web server. You should just keep an eye on this metrics. If you receive
19
-this alert regularly you should consider take action in advance to avoid server overload.
20
-
21
-You can find an interesting article on actions you can take
22
-to [manage sudden traffic](https://www.nginx.com/blog/how-to-manage-sudden-traffic-surges-server-overload/)
23
-on a web server (this article is produced by the NGINX associates, but nearly same principles
24
-applied in any case)
health/guides/web_log/web_log_web_slow.md
deleted
-54
@@ -1,54 +0,0 @@
1
-# web_log_web_slow
2
-
3
-**Web Server | Web log**
4
-
5
-When a client sends a request to your web server, there are many independent phases for this request
6
-that can introduce delay. Some of them may be:
7
-
8
-- DNS lookup
9
-- Establish a TCP connection
10
-- Perform a TLS handshake
11
-- The server to process the request
12
-- Transfer the actual content
13
-
14
-The Netdata Agent calculates the average HTTP response time over the last minute for your web server
15
-(NGINX, Apache). You receive this alert when your web server's average response time has increased.
16
-The alert is raised in warning when the average HTTP response time is twice as much. When the average HTTP response time is four times as much, you receive a critical alert.
17
-
18
-### Troubleshooting section
19
-
20
-The causes of a slow request response may vary. Some options you can explore are:
21
-
22
-<details>
23
-<summary>Your web server utilization is high </summary>
24
-
25
-This problem could be addressed on many levels:
26
-- Check if your host machine can handle the traffic: Check the CPU, memory and traffic utilization.
27
- If this is not an issue:
28
-- Consider raising the resource limitations for your web server (for example add more worker processes).
29
- Please consult your web server docs. If this also doesn't resolve the issue:
30
-- Set up an architecture with multiple web servers and load balancer to handle the traffic for your site.
31
-
32
-</details>
33
-
34
-<details>
35
-<summary>Optimize Databases </summary>
36
-
37
-The response speed is dependent on database optimization. As you first set up a website, the
38
-database responds quickly to queries. As time passes, the database accumulates information. The
39
-compilation results in massive amounts of stored data and might slow down response times.
40
-
41
-If you manage your database with MySQL, this blogpost proposes ways to [tune MySQL operations](https://www.cloudways.com/blog/mysql-performance-tuning/).
42
-</details>
43
-
44
-<details>
45
-<summary>Configure Caching </summary>
46
-
47
-Caching ensures fast delivery to visitors. Without caching, a browser requests assets from the
48
-server each time a page loads instead of accessing them from a local or intermediary cache.
49
-
50
-To enable caching on your server, refer to the respective documentation:
51
-- [NGINX caching guide](https://www.nginx.com/blog/nginx-caching-guide/)
52
-- [Apache caching guide](https://httpd.apache.org/docs/2.4/caching.html)
53
-
54
-</details>
health/guides/web_log/web_slow.md
deleted
-54
@@ -1,54 +0,0 @@
1
-# web_slow
2
-
3
-**Web Server | Web log**
4
-
5
-When a client sends a request to your web server, there are many independent phases for this request
6
-that can introduce delay. Some of them may be:
7
-
8
-- DNS lookup
9
-- Establish a TCP connection
10
-- Perform a TLS handshake
11
-- The server to process the request
12
-- Transfer the actual content
13
-
14
-The Netdata Agent calculates the average HTTP response time over the last minute for your web server
15
-(NGINX, Apache). You receive this alert when your web server's average response time has increased.
16
-The alert is raised in warning when the average HTTP response time is twice as much. When the average HTTP response time is four times as much, you receive a critical alert.
17
-
18
-### Troubleshooting section
19
-
20
-The causes of a slow request response may vary. Some options you can explore are:
21
-
22
-<details>
23
-<summary>Your web server utilization is high </summary>
24
-
25
-This problem could be addressed on many levels:
26
-- Check if your host machine can handle the traffic: Check the CPU, memory and traffic utilization.
27
- If this is not an issue:
28
-- Consider raising the resource limitations for your web server (for example add more worker processes).
29
- Please consult your web server docs. If this also doesn't resolve the issue:
30
-- Set up an architecture with multiple web servers and load balancer to handle the traffic for your site.
31
-
32
-</details>
33
-
34
-<details>
35
-<summary>Optimize Databases </summary>
36
-
37
-The response speed is dependent on database optimization. As you first set up a website, the
38
-database responds quickly to queries. As time passes, the database accumulates information. The
39
-compilation results in massive amounts of stored data and might slow down response times.
40
-
41
-If you manage your database with MySQL, this blogpost proposes ways to [tune MySQL operations](https://www.cloudways.com/blog/mysql-performance-tuning/).
42
-</details>
43
-
44
-<details>
45
-<summary>Configure Caching </summary>
46
-
47
-Caching ensures fast delivery to visitors. Without caching, a browser requests assets from the
48
-server each time a page loads instead of accessing them from a local or intermediary cache.
49
-
50
-To enable caching on your server, refer to the respective documentation:
51
-- [NGINX caching guide](https://www.nginx.com/blog/nginx-caching-guide/)
52
-- [Apache caching guide](https://httpd.apache.org/docs/2.4/caching.html)
53
-
54
-</details>
health/guides/web_log_10m_response_time.md
new
+42
@@ -0,0 +1,42 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average `HTTP response time` of your web server over the last 10 minutes. If you receive this alert, it means that the `latency` of your web server has increased, and might be affecting the user experience.
4
+
5
+### What does HTTP response time mean?
6
+
7
+`HTTP response time` is a measure of the time it takes for your web server to process a request and deliver the corresponding response to the client. A high response time can lead to slow loading pages, indicating that your server is struggling to handle the requests or there are issues with the network.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. **Check the server load**: A high server load can cause increased latency. Check the server load using tools like `top`, `htop`, or `glances`. If server load is high, consider optimizing your server, offloading some services to a separate server, or scaling up your infrastructure.
12
+
13
+ ```
14
+ top
15
+ ```
16
+
17
+2. **Analyze the web server logs**: Look for patterns or specific requests that may be causing the increased latency. This can be achieved by parsing logs and correlating the response time with requests. For example, for Apache logs:
18
+
19
+ ```
20
+ sudo cat /var/log/apache2/access.log | awk '{print $NF " " $0}' | sort -nr | head -n 10
21
+ ```
22
+
23
+ For Nginx logs:
24
+
25
+ ```
26
+ sudo cat /var/log/nginx/access.log | awk '{print $NF " " $0}' | sort -nr | head -n 10
27
+ ```
28
+
29
+3. **Network issues**: Check if there are any issues with the network connecting your server to the clients, such as high latency, packet loss or a high number of dropped packets. You can use the `traceroute` command to diagnose any network-related issues.
30
+
31
+ ```
32
+ traceroute example.com
33
+ ```
34
+
35
+4. **Review your server's configuration**: Check your web server's configuration for any issues, misconfigurations, or suboptimal settings that may be causing the high response time.
36
+
37
+5. **Monitoring and profiling**: Use application monitoring tools like New Relic, AppDynamics, or Dynatrace to get detailed insights about the response time and locate any bottlenecks or problematic requests.
38
+
39
+### Useful resources
40
+
41
+1. [How to Optimize Nginx Performance](https://calomel.org/nginx.html)
42
+2. [Apache Performance Tuning](https://httpd.apache.org/docs/2.4/misc/perf-tuning.html)
health/guides/web_log_1m_bad_requests.md
new
+27
@@ -0,0 +1,27 @@
1
+### Understand the alert
2
+
3
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed or not.
4
+
5
+The 4xx class of status code is intended for cases in which the client seems to have erred. Except when responding to a HEAD request, the server should include an entity containing an explanation of
6
+the error situation, and whether it is a temporary or permanent condition. These status codes are applicable to any request method.
7
+
8
+The Netdata Agent calculates the ratio of client error HTTP requests over the last minute. This metric does not include the 401 errors.
9
+
10
+
11
+### Troubleshoot the alert
12
+
13
+To identify the HTTP response code your web server sends back:
14
+
15
+1. Open the Netdata dashboard.
16
+2. Inspect the `detailed_response_codes` chart for your web server. This chart keeps track of exactly what error codes your web server sends out.
17
+
18
+You should also check server logs for more details about how the server is handling the requests. For example, web servers such as Apache or Nginx produce two files called access.log and error.log (by default under `/var/log/{nginx, apache2}/{access.log, error.log}`)
19
+
20
+3. Troubleshoot 404 codes on the server side
21
+
22
+The 404 requests indicate outdated links on your website or in other websites that redirect to your website. To check for dead links on your on website, use a `broken link checker` software periodically.
23
+
24
+### Useful resources
25
+
26
+1. [https://datatracker.ietf.org/doc/html/rfc2616#section-10.4](https://datatracker.ietf.org/doc/html/rfc2616#section-10.4)
27
+2. [https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#client_error_responses)
\ No newline at end of file
health/guides/web_log_1m_internal_errors.md
new
+31
@@ -0,0 +1,31 @@
1
+### Understand the alert
2
+
3
+This alert is generated by the Netdata Agent when monitoring web server logs. This alert is triggered when the web server has experienced an unusually high number of internal errors (HTTP status codes 5xx) within the last minute. Internal errors indicate that there is an issue with the server or the application running on it, which is causing the server to fail in processing client requests.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the web server logs**: Inspect the web server logs to identify the specific internal errors and any patterns that might be causing the issue. Depending on the web server you are using (e.g., Apache, Nginx, etc.), the log files will be located in different directories. You can usually find the logs in the following locations:
8
+
9
+ - Apache: `/var/log/apache2/` (Debian/Ubuntu) or `/var/log/httpd/` (RHEL/CentOS)
10
+ - Nginx: `/var/log/nginx/`
11
+
12
+ To view the logs in real-time, you can use the `tail` command:
13
+
14
+ ```
15
+ tail -f /path/to/your/log/directory/access.log
16
+ ```
17
+
18
+2. **Analyze the application logs**: If you have an application running on the web server (e.g., PHP, Node.js, Python), check the application logs for any errors or issues that might be causing the internal errors.
19
+
20
+3. **Verify server resources**: Ensure that your server has enough resources (CPU, RAM, disk space) to handle the current workload. High resource utilization can lead to internal errors. You can use Netdata's dashboard to monitor the server resources in real-time.
21
+
22
+4. **Check server configuration**: Review the web server's configuration files for any misconfigurations or settings that may be causing the issue. For example, incorrect permissions, wrong file paths, or improper configurations can lead to internal errors.
23
+
24
+5. **Inspect application code**: Review your application code to identify any bugs, memory leaks, or issues that could be causing the internal errors. If you recently deployed new code or made changes, consider rolling back to a previous version to see if the issue persists.
25
+
26
+6. **Monitor web server metrics**: Keep an eye on the web server's metrics, such as response times and request rates, to identify any performance bottlenecks or potential issues that may be causing the internal errors.
27
+
28
+### Useful resources
29
+
30
+1. [Server errors on Datatracker](https://datatracker.ietf.org/doc/html/rfc2616#section-10.5)
31
+2. [HTTP server errors on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#server_error_responses)
health/guides/web_log_1m_redirects.md
new
+22
@@ -0,0 +1,22 @@
1
+### Understand the alert
2
+
3
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed or not.
4
+
5
+The 3XX class of status code indicates that further action needs to be taken by the user agent in order to fulfill the request. The action required may be carried out by the user agent without interaction with the user if and only if the method used in the second request is GET or HEAD. A client SHOULD detect infinite redirection loops, since such loops generate network traffic for each redirection.
6
+
7
+The Netdata Agent calculates the ratio of redirection HTTP requests over the last minute. This metric does not include the "304 Not modified" message.
8
+
9
+### Troubleshoot the alert
10
+
11
+You can identify exactly what HTTP response code your web server send back to your clients, by opening the Netdata dashboard and inspecting the `detailed_response_codes` chart for your web server. This chart keeps
12
+track of exactly what error codes your web server sends out.
13
+
14
+You should also check the server error logs. For example, web servers such as Apache or Nginx produce and error logs, by default under `/var/log/{nginx, apache2}/{access.log, error.log}`
15
+
16
+### Useful resources
17
+
18
+1. [3XX codes in the HTTP protocol](https://datatracker.ietf.org/doc/html/rfc2616#section-10.3)
19
+
20
+2. [HTTP redirection messages on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#redirection_messages)
21
+
22
+
health/guides/web_log_1m_requests.md
new
+31
@@ -0,0 +1,31 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of HTTP requests received by your web server in the last minute. If you receive this alert, it means that there is an increase in the workload on your web server.
4
+
5
+### What does the number of HTTP requests mean?
6
+
7
+HTTP requests are messages sent by clients (like web browsers) to the server to request various resources, such as web pages, images, scripts, and more. An increase in the number of HTTP requests means that there are more clients accessing your web server, which can result in increased resource usage, decreased response times, or potential overloading.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Determine if the increase in requests is legitimate or malicious:
12
+
13
+ - Review traffic logs to see if the increase in requests is coming from legitimate users or search engine bots, or if it is potentially malicious traffic resulting from bots, crawlers, or DDoS attacks.
14
+
15
+2. Analyze server logs for anomalies or abnormal request patterns:
16
+
17
+ - Look for sudden spikes, repeating requests, or any other suspicious patterns in the server logs. You may use tools like `grep`, `awk`, or web server-specific log analyzers to help with this.
18
+
19
+3. Check server resources and response times:
20
+
21
+ - Monitor your server's CPU, memory, and disk usage to see if the increased requests are causing resource strains or degradations in server performance.
22
+ - Use tools like `top`, `htop`, `vmstat`, or monitoring applications for your specific web server software (e.g., `apachetop` for Apache) to help identify the source of the problem.
23
+
24
+4. Optimize web server performance:
25
+
26
+ - If you find that the increase in requests is legitimate, consider optimizing the web server by enabling caching, improving database query performance, or upgrading hardware and server resources to handle the increased demand.
27
+
28
+5. Implement security measures:
29
+
30
+ - If you have determined that the increase in requests is coming from malicious sources, consider implementing security measures such as rate-limiting, IP blocking, or configuring a Web Application Firewall (WAF).
31
+
health/guides/web_log_1m_successful.md
new
+23
@@ -0,0 +1,23 @@
1
+### Understand the alert
2
+
3
+HTTP response status codes indicate whether a specific HTTP request has been successfully completed or not.
4
+
5
+The Netdata Agent calculates the ratio of successful HTTP requests over the last minute. These requests consist of 1xx, 2xx, 304, 401 response codes. You receive this alert in warning when the percentage of successful requests is less than 85% and in critical when it is below 75%. This alert can indicate:
6
+
7
+- A malfunction in the services of your web server
8
+- Malicious activity towards your website
9
+- Broken links towards your servers.
10
+
11
+In most cases, Netdata will send you another alert indicating high incidences of "abnormal" HTTP requests code, for example you could also receive the `web_log_1m_bad_requests` alert.
12
+
13
+### Troubleshoot the alert
14
+
15
+There are a number of reasons triggering this alert. All of them could eventually cause bad user experience with your web services.
16
+
17
+Identify exactly what HTTP response code your web server sent back to your clients.
18
+
19
+Open the Netdata dashboard and inspect the `detailed_response_codes` chart for your web server. This chart keeps track of exactly what error codes your web server sends out.
20
+
21
+### Useful resources
22
+
23
+1. [HTTP status codes on Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)
\ No newline at end of file
health/guides/web_log_1m_total_requests.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alert calculates the total number of HTTP requests received by the web server in the last minute. If you receive this alert, it means that your web server is experiencing an increase in workload, which might affect its performance or availability.
4
+
5
+### What does an increase in workload mean?
6
+
7
+An increase in workload means that your web server is handling more traffic than usual, or there might be an unexpected spike in the number of HTTP requests received. This might be because of a variety of reasons, like marketing campaigns, product promotions, or even a sudden surge in user demand.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Analyze web traffic logs
12
+
13
+ To understand the reason behind the increased workload, the first step is to analyze the web server traffic logs. Look for any patterns, specific time intervals, or specific user agents that are contributing to the high number of requests.
14
+
15
+2. Check the web server performance
16
+
17
+ Monitoring web server performance metrics like CPU usage, memory usage, and disk space can provide insight into the resource utilization. Use tools like `top`, `vmstat`, `iostat`, and `free` for this assessment.
18
+
19
+3. Monitor response times
20
+
21
+ Checking the response time statistics, like average response time and peak response time, can help to understand if the server is struggling to serve the high number of requests. Tools like `apachetop` or `logstash` can be used to track this information.
22
+
23
+4. Evaluate server scaling options
24
+
25
+ If none of the previous steps help to identify or resolve the issue, it might be time to consider scaling options. If the server is unable to handle the increased workload, vertically or horizontally scaling the system can help.
26
+
27
+5. Investigate application-level issues
28
+
29
+ Application-level issues might also be the reason for high web server traffic. Profiling the web application, checking for slow database queries, or inefficient scripts can help to identify and resolve performance issues.
30
+
31
+### Useful resources
32
+
33
+1. [Analyzing Web server logs with ApacheTop](https://www.howtoforge.com/how-to-analyze-apache-web-server-logs-apachetop)
34
+2. [Logstash Guide: Analyzing Logs](https://www.elastic.co/guide/en/logstash/current/logstash-intro.html)
35
+3. [Web Application Performance Monitoring with New Relic](https://newrelic.com/platform/web-application-monitoring)
36
+4. [Vertically or Horizontally Scaling Your Web Server](https://www.digitalocean.com/community/tutorials/5-common-server-setups-for-your-web-application)
\ No newline at end of file
health/guides/web_log_1m_unmatched.md
new
+25
@@ -0,0 +1,25 @@
1
+### Understand the alert
2
+
3
+In a webserver, all activity should be monitored. By default, most of the webservers log activity in an `access.log` file. The access log is a list of all requests for individual files that people or bots have requested from a website. Log File strings include notes about their requests for the HTML files and their embedded graphic images, along with any other associated files that are transmitted.
4
+
5
+The Netdata Agent calculates the percentage of unparsed log lines over the last minute. These are entries in the log file that didn't match in any of the common pattern operations (1XX, 2XX, etc) of the webserver. This can indicate an abnormal activity on your web server, or that your server is performing operations that you cannot monitor with the Agent.
6
+
7
+Web servers like NGINX and Apache2 give you the ability to modify the log patterns for each request. If you have done that, you also need to adjust the Netdata Agent to parse those patterns.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Create a custom log format job
12
+
13
+You must create a new job in the `web_log` collector for your Agent.
14
+
15
+1. See how you can [configure this collector](https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/web_log#configuration)
16
+
17
+2. Follow the job template specified in the [default web_log.conf file](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/web_log/web_log.conf#L53-L86), focus on the lines [83:85](https://github.com/netdata/netdata/blob/e6d9fbc4a53f1d35363e9b342231bb11627bafbd/collectors/python.d.plugin/web_log/web_log.conf#L83-L85) where you can see how you define a `custom_log_format`.
18
+
19
+3. Restart the Netdata Agent
20
+ ```
21
+ systemctl restart netdata
22
+ ```
23
+
24
+
25
+
health/guides/web_log_5m_requests_ratio.md
new
+34
@@ -0,0 +1,34 @@
1
+### Understand the alert
2
+
3
+The `web_log_5m_requests_ratio` alert indicates that there is a significant increase in the number of successful HTTP requests to your web server in the last 5 minutes compared to the previous 5 minutes. This alert is important for monitoring sudden traffic surges, which can potentially overload your server.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. Check the source of the increased traffic
8
+ Use web server logs to determine the source of the increased traffic. Identify if the requests are coming from a specific IP address, group of IP addresses, or even bots.
9
+
10
+ For example, for Nginx, you can check the log files at `/var/log/nginx/access.log`. For Apache, the logs can be found at `/var/log/apache2/access.log`.
11
+
12
+2. Analyze the requests
13
+ Look at the type of requests (GET, POST, etc.) and the requested resources (URLs). This analysis can help you understand if the increase in traffic is legitimate or if it's due to an issue like a DDoS attack or a web crawler.
14
+
15
+3. Monitor server performance
16
+ Use monitoring tools like `top`, `iotop`, or Netdata itself to check your server's performance metrics. Keep an eye on CPU, RAM, and disk usage to ensure that the server is not getting overloaded.
17
+
18
+4. Optimize server resources and configuration
19
+ If you find that the traffic increase is legitimate and your server is struggling to handle the load, consider optimizing your server resources and configuration. Techniques include:
20
+
21
+ - Increasing server resources (CPU, RAM, disk)
22
+ - Using a caching mechanism
23
+ - Load balancing and scaling out your infrastructure
24
+ - User connection rate limiting and request throttling
25
+
26
+5. Mitigate potential attacks
27
+ If the analysis reveals that the increase in traffic is due to a DDoS attack, implement mitigation strategies like firewalls, IP blocking, or using a web application firewall (WAF). Ensure that you have a robust security system in place to protect your server from such attacks.
28
+
29
+### Useful resources
30
+
31
+1. [How to Manage Sudden Traffic Surges and Server Overload](https://www.nginx.com/blog/how-to-manage-sudden-traffic-surges-server-overload/)
32
+2. [Attacks on Network Infrastructure](https://www.cloudflare.com/learning/ddos/ddos-attacks/)
33
+3. [Using Nginx to Rate Limit IP Addresses](https://calomel.org/nginx.html)
34
+4. [Setting up a Super Fast Apache Server with Cache](https://hostadvice.com/how-to/how-to-configure-apache-web-server-cache-on-ubuntu/)
\ No newline at end of file
health/guides/web_log_5m_successful.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alert monitors the average number of successful HTTP requests per second, over the last 5 minutes (`web_log.type_requests`). If you receive this alert, it means that there has been a significant change in the number of successful HTTP requests to your web server.
4
+
5
+### What does successful HTTP request mean?
6
+
7
+A successful HTTP request is one that receives a response with an HTTP status code in the range of `200-299`. In other words, these requests have been processed correctly by the web server and returned the expected results to the client.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check your web server logs
12
+
13
+ Inspect your web server logs for any abnormal activity or issues that might have led to increased or decreased successful HTTP requests. Depending on your web server (e.g., Apache, Nginx), the location of the logs will vary.
14
+
15
+2. Analyze the type of requests
16
+
17
+ Check the logs for request types (e.g., GET, POST, PUT, DELETE) and their corresponding distribution during the time of the alert. This might help you identify a pattern or source of the issue.
18
+
19
+3. Monitor web server resources
20
+
21
+ Use monitoring tools like `top`, `htop`, or `glances` to check the resource usage of your web server during the alert period. High resource usage may indicate that your server is struggling to handle the load, causing an abnormal number of successful HTTP requests.
22
+
23
+4. Verify client connections
24
+
25
+ Investigate the IP addresses and user agents that are making a significant number of requests during the alert period. If there's a spike in requests from a single or a few IPs, it could be a sign of a coordinated attack, excessive crawling, or other unexpected behavior.
26
+
27
+5. Check your web application
28
+
29
+ Make sure that your web application is functioning well and generating the expected response for clients, which can impact successful HTTP requests.
30
+
31
+### Useful resources
32
+
33
+1. [Apache Log Files](https://httpd.apache.org/docs/current/logs.html)
34
+2. [Nginx Log Files](https://nginx.org/en/docs/ngx_core_module.html#error_log)
35
+3. [Introduction to Identifying Security Vulnerabilities in Web Servers](https://www.acunetix.com/blog/articles/introduction-identifying-security-vulnerabilities-web-servers)
36
+4. [Web Application Performance Analysis and Monitoring](https://www.site24x7.com/learning/web-application-performance.html)
\ No newline at end of file
health/guides/web_log_5m_successful_old.md
new
+29
@@ -0,0 +1,29 @@
1
+### Understand the alert
2
+
3
+This alert, `web_log_5m_successful_old`, calculates the average number of successful HTTP requests per second for the 5 minutes starting 10 minutes ago. If you receive this alert, it means that there might be a significant change in the number of requests your web server is serving.
4
+
5
+### What does the alert mean?
6
+
7
+The alert is useful for understanding the workload on your web server based on historical request data. It helps to ensure that the web server is functioning as expected and can handle the current number of users without negatively impacting their experience.
8
+
9
+### Troubleshoot the alert
10
+
11
+To troubleshoot this alert, follow these steps:
12
+
13
+1. **Check the current number of successful HTTP requests** to compare with the historical data of the alert. You can use Netdata's web dashboard to see the current requests rate in real-time. If the number of requests has increased significantly, it might indicate a potential issue.
14
+
15
+2. **Identify any potential issues or errors on your web server.** Check the server's error logs for any signs of abnormal behavior or error messages. This can help you determine if there are any underlying issues causing the increase in requests.
16
+
17
+3. **Analyze the user traffic** to understand the cause of the increase in successful requests. This could be caused by a sudden spike in website visitors, a DDoS attack, or the introduction of new and popular content on your website. You can use tools like Google Analytics or server access logs to get detailed information about user traffic.
18
+
19
+4. **Review server resources and performance** to ensure the web server has adequate resources to handle the request load. If the number of requests is higher than usual, check the server's CPU usage, memory usage, and network bandwidth to ensure optimal performance.
20
+
21
+5. **Evaluate server configuration** to check for any misconfigurations, outdated software, or resource limitations that may impact the handling of requests. Update or adjust configurations as necessary to improve the web server's performance.
22
+
23
+6. **Monitor and take necessary actions** based on your findings. If the increase in successful requests is a result of legitimate traffic, ensure that your web server can handle the extra load. If the traffic is malicious or the result of an attack, consider implementing security measures like rate-limiting or blocking IPs.
24
+
25
+### Useful resources
26
+
27
+1. [Monitoring Web Server Performance with Netdata](https://www.netdata.cloud/webserver-monitoring/)
28
+2. [How to Analyze Access Logs](https://www.scalyr.com/blog/analyze-access-logs/)
29
+3. [Optimizing Web Server Performance](https://www.keycdn.com/blog/web-server-performance)
health/guides/web_log_web_slow.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+The `web_log_web_slow` alert is triggered when the average HTTP response time of your web server (NGINX, Apache) has increased over the last minute. It indicates that your web server's performance might be affected, resulting in slow response times for client requests.
4
+
5
+### Troubleshoot the alert
6
+
7
+There are several factors that can cause slow web server performance. To troubleshoot the `web_log_web_slow` alert, examine the following areas:
8
+
9
+1. **Monitor web server utilization:**
10
+
11
+ Use monitoring tools like `top`, `htop`, or `glances` to check the CPU, memory, and traffic utilization of your web server. If you find high resource usage, consider taking action to address the issue:
12
+ - Increase your server's resources (CPU, memory) or move to a more powerful machine.
13
+ - Adjust the web server configuration to use more worker processes or threads.
14
+ - Implement load balancing across multiple web servers to distribute the traffic load.
15
+
16
+2. **Optimize databases:**
17
+
18
+ Slow database performance can directly impact web server response times. Monitor and optimize your database to improve response speeds:
19
+ - Check for slow or inefficient queries and optimize them.
20
+ - Regularly clean and optimize your database by removing outdated or unnecessary data, and by using tools like `mysqlcheck` or `pg_dump`.
21
+ - Enable database caching for faster results on recurring queries.
22
+
23
+3. **Configure caching:**
24
+
25
+ Implement browser or server-side caching to reduce the load on your web server and speed up content delivery:
26
+ - Enable browser caching using proper cache-control headers in your server configuration.
27
+ - Implement server-side caching with tools like Varnish or use full-page caching in your web server (NGINX FastCGI cache, Apache mod_cache).
28
+
29
+4. **Examine web server logs:**
30
+
31
+ Analyze your web server logs to identify specific requests or resources that may be causing slow responses. Tools like `goaccess` or `awstats` can help you analyze web server logs and identify issues:
32
+ - Check for slow request URIs or resources and optimize them.
33
+ - Identify slow third-party services, such as CDNs, external APIs, or database connections, and troubleshoot these connections as needed.
34
+
35
+5. **Optimize web server configuration:**
36
+
37
+ Review your web server's configuration settings to ensure optimal performance:
38
+ - Ensure that your web server is using the latest stable version for performance improvements and security updates.
39
+ - Disable unnecessary modules or features to reduce resource usage.
40
+ - Review and optimize settings related to timeouts, buffer sizes, and compression for better performance.
41
+
42
+### Useful resources
43
+
44
+1. [Optimizing NGINX for Performance](https://easyengine.io/tutorials/nginx/performance/)
45
+2. [Apache Performance Tuning](https://httpd.apache.org/docs/2.4/misc/perf-tuning.html)
46
+3. [Top 10 MySQL Performance Tuning Tips](https://www.databasejournal.com/features/mysql/top-10-mysql-performance-tuning-tips.html)
47
+4. [10 Tips for Optimal PostgreSQL Performance](https://www.digitalocean.com/community/tutorials/10-tips-for-optimizing-postgresql-performance-on-a-digitalocean-droplet)
48
+5. [A Beginner's Guide to HTTP Cache Headers](https://www.keycdn.com/blog/http-cache-headers)
\ No newline at end of file
health/guides/whoisquery/whoisquery_days_until_expiration.md
deleted
-19
@@ -1,19 +0,0 @@
1
-# whoisquery_days_until_expiration
2
-
3
-**Other | WHOIS**
4
-
5
-The Netdata Agent checks the time until the domain name registration expires. This alert indicates
6
-that the domain name registration expires soon. If you do not renew the domain it will be
7
-deactivated soon. Consider scheduling the renewal.
8
-
9
-This alert is raised into warning when your domain is about to expire in less than 90 days and in
10
-critical when it will expire in less than 30 days.
11
-
12
-### Troubleshooting section
13
-
14
-If you still want this domain you should renew it. If you need help renewing your domain name, you
15
-should immediately contact the registrar (or reseller) that provided your domain name registration
16
-services for possible renewal options available to you.
17
-
18
-To determine your current registrar of record, you may conduct a WHOIS search by visiting
19
-whois.icann.org. Registrar information can be found on the ICANN-Accredited Registrar list.
\ No newline at end of file
health/guides/whoisquery_days_until_expiration.md
new
+26
@@ -0,0 +1,26 @@
1
+### Understand the alert
2
+
3
+This alert indicates that your domain name registration is close to its expiration date. It will trigger a warning when there are less than 90 days remaining and a critical alert when there are less than 30 days remaining. If you do not renew the domain registration, it will be deactivated, and you might lose ownership of the domain.
4
+
5
+### What does domain name expiration mean?
6
+
7
+Domain names are registered for a limited period, usually between 1 and 10 years. When the registration period comes to an end, you need to renew it if you want to continue using the domain. If you fail to renew the domain, it will become inactive, and after a certain period (known as the grace period), it will be available for anyone else to register. This might result in losing access to your website and email services associated with the domain.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Verify domain expiration date: To verify the current expiration date of your domain, you can conduct a WHOIS search by visiting whois.icann.org. This will provide you with details about your domain, including the registrar information, registration date, and expiration date.
12
+
13
+2. Contact your domain registrar: If you want to renew your domain, contact your domain registrar (or reseller) as soon as possible to discuss the renewal options available to you. You can find the contact information for your domain registrar in the WHOIS search results.
14
+
15
+3. Renew your domain: Once you have contacted your domain registrar, follow their instructions to renew your domain. It's a good idea to renew it for a longer period to avoid the risk of domain expiration in the near future.
16
+
17
+4. Review domain auto-renewal settings: Many registrars offer an auto-renewal option, which can help you avoid domain expiration by automatically renewing the domain before it expires. Make sure to review your account settings and enable auto-renewal if you wish to take advantage of this feature.
18
+
19
+5. Monitor the domain renewal process: After renewing your domain, verify that the updated expiration date is reflected in the WHOIS search results. Keep an eye on this to ensure that the renewal process has been completed successfully.
20
+
21
+### Useful resources
22
+
23
+1. [ICANN WHOIS Search](https://whois.icann.org)
24
+2. [ICANN-Accredited Registrar List](https://www.icann.org/registrar-reports/accredited-list.html)
25
+3. [How to Renew Your Domain Name Registration](https://www.icann.org/resources/pages/renewal-registrant-2016-02-04-en)
26
+4. [What to Do If Your Domain Name Expires](https://www.icann.org/resources/pages/existing-registrant-2016-06-01-en)
\ No newline at end of file
health/guides/wifi_inbound_packets_dropped_ratio.md
new
+52
@@ -0,0 +1,52 @@
1
+### Understand the alert
2
+
3
+This alert calculates the ratio of inbound dropped packets for a specific network interface over the last 10 minutes. If you receive this alert, it means that your WiFi network interface dropped a significant number of packets, which could be due to lack of resources or unsupported protocol.
4
+
5
+### What does "inbound dropped packets" mean?
6
+
7
+In the context of networking, "inbound dropped packets" means that packets were received by the network interface but were not processed. This can happen due to various reasons, including:
8
+
9
+1. Insufficient resources (e.g., CPU, memory) to handle the packet.
10
+2. Unsupported protocol.
11
+3. Network congestion, leading to packets being dropped.
12
+4. Hardware or configuration issues.
13
+
14
+### Troubleshoot the alert
15
+
16
+- Check the system resource utilization
17
+
18
+Using the `top` command, check the resource utilization (CPU, memory, and I/O) in your system. High resource usage might indicate that your system is struggling to process the incoming packets.
19
+
20
+```
21
+top
22
+```
23
+
24
+- Inspect network configuration and hardware
25
+
26
+1. Check if there are any hardware issues or misconfigurations in your WiFi adapter or network interface. Refer to your hardware's documentation or manufacturer's support for troubleshooting steps.
27
+
28
+2. Make sure your network device drivers are up-to-date.
29
+
30
+- Monitor network traffic
31
+
32
+Use the `iftop` command to monitor network traffic on your interface. High network traffic can cause congestion, leading to dropped packets. If you don't have it installed, follow the [installation instructions](https://www.tecmint.com/iftop-linux-network-bandwidth-monitoring-tool/).
33
+
34
+```
35
+sudo iftop -i <interface_name>
36
+```
37
+
38
+- Investigate network protocols
39
+
40
+Inbound dropped packets may be caused by unsupported network protocols. Use the `tcpdump` command to examine network traffic for any abnormalities or unknown protocols.
41
+
42
+```
43
+sudo tcpdump -i <interface_name>
44
+```
45
+
46
+### Useful resources
47
+
48
+1. [Top 20 Netstat Command Examples in Linux](https://www.tecmint.com/20-netstat-commands-for-linux-network-management/)
49
+2. [iftop command in Linux to monitor network traffic](https://www.tecmint.com/iftop-linux-network-bandwidth-monitoring-tool/)
50
+3. [An Overview of Packet Sniffing using Tcpdump](https://www.ubuntupit.com/tcpdump-useful-unix-packet-sniffer-command/)
51
+
52
+Remember to replace `<interface_name>` with the actual name of the WiFi network interface causing the alert.
\ No newline at end of file
health/guides/wifi_outbound_packets_dropped_ratio.md
new
+54
@@ -0,0 +1,54 @@
1
+### Understand the alert
2
+
3
+The `wifi_outbound_packets_dropped_ratio` alert indicates that a significant number of packets were dropped on the way to transmission over the last 10 minutes. This could be due to a lack of resources or other issues with the network interface.
4
+
5
+### What does dropped packets mean?
6
+
7
+Dropped packets refer to data packets that are discarded by a network interface instead of being transmitted through the network. This can occur for various reasons such as hardware failures, lack of resources (e.g., memory, processing power), or network congestion.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check interface statistics
12
+
13
+Use the `ifconfig` command to view information about your network interfaces, including their packet drop rates. Look for the dropped packets count in the TX (transmit) section.
14
+
15
+```bash
16
+ifconfig <interface_name>
17
+```
18
+
19
+Replace `<interface_name>` with the name of the network interface you are investigating, such as `wlan0` for a wireless interface.
20
+
21
+2. Check system logs
22
+
23
+System logs can provide valuable information about any potential issues. Check the logs for any errors or warnings related to the network interface or driver.
24
+
25
+For example, use `dmesg` command to display kernel messages:
26
+
27
+```bash
28
+dmesg | grep -i "<interface_name>"
29
+```
30
+
31
+Replace `<interface_name>` with the name of the network interface you are investigating.
32
+
33
+3. Check for hardware issues
34
+
35
+Inspect the network interface for any signs of hardware failure or malfunction. This may include damaged cables, loose connections, or issues with other networking equipment (e.g. switches, routers).
36
+
37
+4. Monitor network congestion
38
+
39
+High packet drop rates can be caused by network congestion. Monitor network usage and performance using tools such as `iftop`, `nload`, or `vnstat`. Identify and address any traffic bottlenecks or excessive usage.
40
+
41
+5. Update network drivers
42
+
43
+Outdated or faulty network drivers may cause packet drop issues. Check for driver updates and install any available updates following the manufacturer's instructions.
44
+
45
+6. Optimize network settings
46
+
47
+You can adjust network settings, like buffers or queues, to mitigate dropped packets. Consult your operating system or network device documentation for specific recommendations on adjusting these settings.
48
+
49
+### Useful resources
50
+
51
+1. [ifconfig command in Linux](https://www.geeksforgeeks.org/ifconfig-command-in-linux-with-examples/)
52
+2. [How to monitor network usage with iftop](https://www.binarytides.com/monitor-network-usage-with-iftop/)
53
+3. [nload – Monitor Network Traffic and Bandwidth Usage in Real Time](https://www.tecmint.com/nload-monitor-linux-network-traffic-bandwidth-usage/)
54
+4. [VNstat – A Network Traffic Monitor](https://www.tecmint.com/vnstat-monitor-network-traffic-in-linux/)
\ No newline at end of file
health/guides/windows/windows_10min_cpu_usage.md
deleted
-32
@@ -1,32 +0,0 @@
1
-# windows_10min_cpu_usage
2
-
3
-## Windows | CPU
4
-
5
-This alarm calculates the average of CPU utilization over a period of 10 minutes.
6
-
7
-It is raised into warning if the value exceeds 85%.
8
-If the average exceeds 95%, then the alert gets raised into critical.
9
-
10
-### Troubleshooting Section
11
-
12
-<details>
13
-<summary>Processes slowing down your CPU</summary>
14
-
15
-In Windows, you can open up the Task Manager from the menu or by pressing
16
-`ctrl`+`shift`+`esc`.
17
-
18
-- Under the processes tab, you can see a list of the processes currently running on the machine.
19
- - To get a better picture of the main consumers, order them by their total CPU usage by
20
- clicking the CPU column. That will sort the top main processes utilizing your CPU.
21
-
22
-
23
-- To get a more detailed look, click the "Performance" tab (next to the processes tab) and
24
- then click on the bottom of the window "Open Resource Monitor".
25
- - That will open up a window with a more detailed view on the processes.
26
- - On the "Processes" table, look for the column "Average CPU". Clicking this will order the
27
- processes again by CPU utilization.
28
-
29
-> It would be helpful to close any of the main consumer processes, but Netdata strongly suggests
30
-> knowing exactly what processes you are closing and being certain that they are not necessary to
31
-> your workflow or system.
32
-</details>
health/guides/windows_10min_cpu_usage.md
new
+36
@@ -0,0 +1,36 @@
1
+### Understand the alert
2
+
3
+This alert calculates the average total `CPU utilization` on a Windows system over the last 10 minutes. If you receive this warning or critical alert, it means that your system is experiencing high CPU usage, which could lead to performance issues.
4
+
5
+### What does CPU utilization mean?
6
+
7
+`CPU utilization` is the percentage of time the CPU spends executing tasks, as opposed to being idle. A high CPU utilization means that the CPU is working on a large number of tasks and may not have enough processing power to handle additional tasks efficiently. This can result in slow response times and overall system performance issues.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify high CPU usage processes:
12
+
13
+ Open Task Manager by pressing `Ctrl + Shift + Esc` on your keyboard, or right-click on the Taskbar and select "Task Manager." Click the "Processes" tab, and sort by the "CPU" column to identify the processes consuming the most CPU resources.
14
+
15
+2. Analyze process details:
16
+
17
+ Right-click on the process with high CPU usage and select "Properties" or "Go to details" to learn more about the process, its location, and its purpose.
18
+
19
+3. Determine if the process is essential:
20
+
21
+ Research the process in question to ensure that it is safe to terminate. Some processes are integral to the system, and terminating them may cause instability or crashes.
22
+
23
+4. Terminate or optimize the problematic process:
24
+
25
+ If the process is not essential, you can right-click on it and select "End task" to stop it. If the process is necessary, consider optimizing its performance or updating the software responsible for the process. In some cases, restarting the system may help resolve temporary high CPU usage issues.
26
+
27
+5. Monitor CPU usage after taking action:
28
+
29
+ Continue monitoring CPU usage to ensure that the issue has been resolved. If the problem persists, further investigation may be required, such as examining system logs or using performance analysis tools like Windows Performance Monitor.
30
+
31
+### Useful resources
32
+
33
+1. [How to Monitor CPU Usage on Windows](https://www.tomsguide.com/how-to/how-to-monitor-cpu-usage-on-windows)
34
+2. [Windows Task Manager: A Troubleshooting Guide](https://www.howtogeek.com/66622/stupid-geek-tricks-6-ways-to-open-windows-task-manager/)
35
+3. [How to Use the Performance Monitor on Windows](https://www.digitalcitizen.life/how-use-performance-monitor-windows/)
36
+4. [Understanding Process Explorer](https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer)
\ No newline at end of file
health/guides/windows_disk_in_use.md
new
+34
@@ -0,0 +1,34 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the disk space utilization on a Windows system surpasses the defined thresholds. If you receive this alert, it means your system's disk usage is high, and you might need to free up space.
4
+
5
+### Why is disk space utilization important?
6
+
7
+Disk space utilization is crucial for the stable and efficient operation of your system. As the disk fills up, system processes may slow down or fail due to insufficient storage space. Moreover, new applications and updates may require additional storage, which can cause issues if not enough disk space is available.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check disk usage in detail
12
+
13
+ To check the disk usage on your Windows system, you can use `Disk Management` tool by searching for it in the Start menu, or by right-clicking on Computer in the File Explorer and selecting "Manage."
14
+
15
+2. Analyze disk usage by folders and files
16
+
17
+ Use a disk space analyzer tool like [TreeSize](https://www.jam-software.com/treesize_free) or [WinDirStat](https://windirstat.net/) to find the largest files and folders on your system. These tools will help you identify areas where you can free up space.
18
+
19
+3. Clean up unnecessary files
20
+
21
+ - Empty the recycle bin on your Windows system.
22
+ - In the File Explorer, right-click on the system drive (usually C:), and select "Properties." Navigate to the "General" tab and click on "Disk Cleanup" to free up space by removing temporary files, system files and other items that can be safely deleted.
23
+ - Uninstall unused applications using the Programs and Features setting in the Control Panel.
24
+ - Move larger files such as media or documents to an external storage device or cloud storage service.
25
+
26
+4. Monitor disk usage
27
+
28
+ Keep an eye on the disk usage to prevent it from surpassing the threshold again in the future. Pay attention to system and software updates that may require additional storage, as well as the growth of log files or temporary files generated by your computer's operation.
29
+
30
+### Useful resources
31
+
32
+1. [Windows 10 Tips & Tricks: Analyze Disk Space & Free Space - YouTube](https://www.youtube.com/watch?v=NolLC9tBP_Y)
33
+2. [5 Free Tools to Visualize Disk Space Usage on Windows](https://www.hongkiat.com/blog/visualize-hard-disk-usage-free-tools-for-windows/)
34
+3. [10 Ways to Free Up Hard Drive Space on Windows](https://www.howtogeek.com/125923/7-ways-to-free-up-hard-disk-space-on-windows/)
\ No newline at end of file
health/guides/windows_inbound_packets_discarded.md
new
+39
@@ -0,0 +1,39 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of inbound discarded packets for a network interface on a Windows system exceeds the threshold (5 packets) within the last 10 minutes. If you receive this alert, it means that your network interface may have an issue that is causing packets to be discarded.
4
+
5
+### What does inbound discarded packets mean?
6
+
7
+Inbound discarded packets refer to network packets that are received by the network interface but are not processed by the system. Packets may be discarded for various reasons such as network congestion, packet corruption, or reaching the system's capacity limits.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the problematic network interface
12
+
13
+To find out which network interface is causing the problem, log in to the Windows system and open **Performance Monitor**. Go to the **Windows → Networking → Network Interface** section in the left pane and check the **Packets Received Discarded** counter to identify the offending interface.
14
+
15
+2. Check network interface hardware
16
+
17
+Verify that the network interface is working correctly and hasn't malfunctioned. Inspect the cables and ensure that they are connected properly. If possible, try a different network interface.
18
+
19
+3. Check network congestion and bandwidth usage
20
+
21
+High network congestion and bandwidth usage can cause packets to be discarded. Monitor your network's usage and check for any unusual patterns or excessive bandwidth usage. Consider using a network monitoring tool to gather more in-depth information about your network.
22
+
23
+4. Inspect system logs
24
+
25
+Check system logs for errors or warnings related to the network interface. The Windows Event Viewer can be a valuable resource for identifying issues related to the network interface.
26
+
27
+5. Update network adapter drivers
28
+
29
+Outdated or incompatible drivers can cause network issues, including inbound discarded packets. Ensure that your network adapter drivers are up-to-date and provided by a reliable source.
30
+
31
+6. Investigate packet corruption
32
+
33
+Packet corruption can be caused by faulty hardware, software issues, or even cyber-attacks. Ensure that your system is adequately protected, and investigate any possible software-related issues that may lead to packet corruption.
34
+
35
+### Useful resources
36
+
37
+1. [Windows Performance Monitor](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/perfmon)
38
+2. [Windows Event Viewer](https://docs.microsoft.com/en-us/windows/win32/eventlog/event-log-reference)
39
+3. [How to troubleshoot networking problems on the Windows platform](https://support.microsoft.com/en-us/help/10267)
\ No newline at end of file
health/guides/windows_inbound_packets_errors.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert informs you about the number of `inbound errors` on the network interface of your Windows machine within the last 10 minutes. If you receive this alert, it indicates that there might be issues with your network connection or hardware.
4
+
5
+### What are inbound errors?
6
+
7
+Inbound errors refer to problems that occur when packets are coming into the network interface of your machine from external sources. These errors might occur due to various reasons such as packet loss during transmission, hardware problems in the network interface card (NIC), or incorrect network configurations.
8
+
9
+### Troubleshoot the alert
10
+
11
+To troubleshoot this alert, you can perform the following steps:
12
+
13
+1. Check the network connection
14
+
15
+ Ensure that the network connection is stable and the cables (if any) are properly connected. If you're using a wireless connection, verify that the signal strength is good and that there are no known Wi-Fi issues in your area.
16
+
17
+2. Verify network configurations
18
+
19
+ Go through your network configurations and ensure that they are properly set. Some common issues include incorrect IP addresses, subnet masks or gateways. Open the Network Connections window (press Windows key + R, type `ncpa.cpl` and click OK), then right-click your network adapter, select `Properties`, and recheck your configurations.
20
+
21
+3. Inspect the hardware
22
+
23
+ Check if the NIC experiences any physical issues or if it gets overheated. If you suspect a hardware problem, consider replacing the NIC or connecting to a different network interface to isolate the issue.
24
+
25
+4. Monitor the network for any anomalies
26
+
27
+ You can use native Windows tools like `Performance Monitor` or `Resource Monitor` to keep an eye on network performance and packet errors. Open the respective tools by searching in the Start Menu.
28
+
29
+5. Review Event Viewer logs
30
+
31
+ Look for any network-related errors logged in the `Event Viewer`. Press Windows key + X, select Event Viewer, and navigate to `Windows Logs` > `System`. Filter the logs by choosing the `Network Profile` event source and review the error messages.
32
+
33
+6. Update NIC drivers
34
+
35
+ Sometimes, outdated or faulty NIC drivers might cause inbound packet errors. Ensure that you've installed the latest drivers for your NIC. Visit the manufacturer's website to download and install the most recent drivers compatible with your Windows operating system.
36
+
37
+### Useful resources
38
+
39
+1. [How to use Network Monitor in Windows](https://docs.microsoft.com/en-us/windows/client-management/troubleshoot-tcpip-network-monitor)
40
+2. [Network Troubleshooting Guide for Windows](https://techcommunity.microsoft.com/t5/networking-blog/network-troubleshooting-guide-for-windows/ba-p/428114)
41
+3. [How to Troubleshoot Network Connections with Ping and Tracert](https://www.windowscentral.com/how-troubleshoot-network-connection-ping-and-traceroute)
\ No newline at end of file
health/guides/windows_outbound_packets_discarded.md
new
+48
@@ -0,0 +1,48 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the number of outbound discarded packets for a network interface on a Windows system reaches or exceeds 5 in the last 10 minutes. Discarded packets indicate network problems or misconfigurations and can lead to decreased performance, slow connections and communication errors.
4
+
5
+### What are outbound discarded packets?
6
+
7
+Outbound discarded packets are network packets that were not sent successfully from a Windows host to the intended destination. This might be due to various reasons such as buffer overflows, device driver errors, or network congestion. Discarded packets may result in retransmissions, which could cause increased latencies and reduced network throughput.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Check network performance statistics
12
+
13
+Use the built-in `netstat` command to display network statistics:
14
+```
15
+netstat -s
16
+```
17
+
18
+Look for errors or high discard rates, which may indicate network problems.
19
+
20
+2. Monitor network interface performance
21
+
22
+Use the `Performance Monitor` tool in Windows to monitor the network interface for issues. Look for counters related to discarded packets, such as `Packets Outbound Errors`, `Packets Received Errors`, and `Packets Sent/sec`.
23
+
24
+3. Identify if there are specific applications with high discard rates
25
+
26
+Use the `Resource Monitor` tool in Windows to check which applications are consuming the most network resources and identify if any specific application is causing high discard rates.
27
+
28
+4. Check for errors, warnings, or unusual events in the Windows Event Viewer
29
+
30
+Open the `Event Viewer` in Windows and browse through the System and Application logs for any network-related events. Look for errors or warnings that could be related to network configurations, device driver problems, or application-specific issues.
31
+
32
+5. Update or reinstall network drivers
33
+
34
+Outdated or corrupt network drivers can cause discarded packets. Ensure your network drivers are up to date and, if necessary, reinstall the drivers.
35
+
36
+6. Check network components and configurations
37
+
38
+Inspect network cables, switches, and routers for any physical damage or malfunction. Check the network settings on the Windows host to ensure they are correctly configured, including DNS, gateway, and subnet mask.
39
+
40
+7. Network congestion
41
+
42
+If your network is congested, it can cause an increase in discarded packets. Consider upgrading network equipment or implementing quality of service (QoS) policies to prioritize and manage network traffic more effectively.
43
+
44
+### Useful resources
45
+
46
+1. [Using Performance Monitor to monitor network performance](https://techcommunity.microsoft.com/t5/ask-the-performance-team/using-perfmon-to-monitor-your-servers-network-performance/ba-p/373944)
47
+2. [Monitoring Network Performance with Resource Monitor](https://www.online-tech-tips.com/computer-tips/monitoring-network-performance-with-resource-monitor/)
48
+3. [Event Viewer in Windows](https://www.dummies.com/computers/operating-systems/windows-10/how-to-use-event-viewer-in-windows-10/)
\ No newline at end of file
health/guides/windows_outbound_packets_errors.md
new
+46
@@ -0,0 +1,46 @@
1
+### Understand the alert
2
+
3
+This alert monitors the number of `outbound errors` on the network interface of a Windows system over the last 10 minutes. If you receive this alert, it means that there are `5 or more errors` in outbound packets during that period.
4
+
5
+### What are outbound errors?
6
+
7
+`Outbound errors` refer to problems that occur during the transmission of packets from the network interface of your system. These errors can be due to various reasons, such as faulty hardware, incorrect configuration, or network congestion.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Identify the network interface(s) with high outbound errors
12
+
13
+Use the `netstat -e` command to display network statistics for each interface on your system:
14
+
15
+```
16
+netstat -e
17
+```
18
+
19
+This will show you the interfaces with errors, along with a count of errors.
20
+
21
+2. Check for faulty hardware or cables
22
+
23
+Visually inspect the network interface and cables for any signs of damage or disconnection. If the hardware appears to be faulty, replace it as necessary.
24
+
25
+3. Review network configuration settings
26
+
27
+Ensure that the network configuration on your system is correct, including the IP address, subnet mask, gateway, and DNS settings. If the configuration is incorrect, update it accordingly.
28
+
29
+4. Monitor network traffic
30
+
31
+Use network monitoring tools such as `Wireshark` or `tcpdump` to capture traffic on the affected interface. Analyze the captured traffic to identify any issues or patterns that may be causing the errors.
32
+
33
+5. Check for network congestion
34
+
35
+If the errors are due to network congestion, identify the sources of high traffic and implement measures to reduce congestion, such as traffic shaping, prioritizing, or rate limiting.
36
+
37
+6. Update network drivers and firmware
38
+
39
+Ensure that your network interface card (NIC) drivers and firmware are up-to-date. Check the manufacturer's website for updates and apply them as necessary.
40
+
41
+### Useful resources
42
+
43
+1. [Netstat Command Usage on Windows](https://www.computerhope.com/issues/ch001/stat.htm)
44
+2. [Wireshark - A Network Protocol Analyzer](https://www.wireshark.org/)
45
+3. [Tcpdump - A Packet Analyzer](https://www.tcpdump.org/)
46
+4. [Network Performance Monitoring and Diagnostics Guide](https://docs.microsoft.com/en-us/windows-server/networking/technologies/npmd/npmd)
\ No newline at end of file
health/guides/windows_ram_in_use.md
new
+38
@@ -0,0 +1,38 @@
1
+### Understand the alert
2
+
3
+The `windows_ram_in_use` alert is triggered when memory utilization on a Windows system reaches the specified warning or critical thresholds. If you receive this alert, it means that your Windows system is running low on available memory.
4
+
5
+### What does memory utilization mean?
6
+
7
+Memory utilization refers to the percentage of a system's RAM that is currently being used by applications, processes, and the operating system. High memory utilization can lead to performance issues and may cause applications to crash or become unresponsive.
8
+
9
+### Troubleshoot the alert
10
+
11
+- Check current memory usage on the system
12
+
13
+1. Press `Ctrl + Shift + Esc` to open Task Manager.
14
+2. Click on the `Performance` tab.
15
+3. View the `Memory` section to see the total memory usage and available memory.
16
+
17
+- Identify high memory usage processes
18
+
19
+1. In Task Manager, click on the `Processes` tab.
20
+2. Click on the `Memory` column to sort processes by memory usage.
21
+3. Identify processes that are using a high percentage of memory.
22
+
23
+- Optimize memory usage
24
+
25
+1. Close unnecessary applications and processes to free up memory.
26
+2. Investigate if running processes have a known memory leak issue.
27
+3. Consider upgrading the system's RAM if memory usage is consistently high.
28
+
29
+- Monitor memory usage over time
30
+
31
+1. Use Windows Performance Monitor to create a Data Collector Set that collects memory usage metrics.
32
+2. Analyze the collected data to identify trends and potential issues.
33
+
34
+### Useful resources
35
+
36
+1. [How to use Task Manager to monitor Windows PC's performance](https://support.microsoft.com/en-us/windows/how-to-use-task-manager-to-monitor-windows-pc-s-performance-171100cb-5e7d-aaba-29abfedfb06f)
37
+2. [How to use Performance Monitor on Windows 10](https://www.windowscentral.com/how-use-performance-monitor-windows-10)
38
+3. [How to fix high memory usage in Windows](https://pureinfotech.com/reduce-ram-memory-usage-windows/)
\ No newline at end of file
health/guides/windows_swap_in_use.md
new
+41
@@ -0,0 +1,41 @@
1
+### Understand the alert
2
+
3
+This alert monitors the swap memory utilization on a Windows system. If you receive this alert, it means that your system's swap memory usage is nearing or has exceeded the defined thresholds (`warning` at 80-90% and `critical` at 90-98%).
4
+
5
+### What is swap memory?
6
+
7
+Swap memory is a virtual memory management technique where a portion of the disk space is used as an extension of the physical memory (RAM). When the system runs low on RAM, it moves inactive data from RAM to swap memory to free up space for active processes. While swap memory can help prevent the system from running out of memory, keep in mind that accessing data from swap memory is slower than from RAM.
8
+
9
+### Troubleshoot the alert
10
+
11
+1. Determine the system's memory and swap usage.
12
+
13
+ Use the Windows Task Manager to monitor the overall system performance:
14
+
15
+ ```
16
+ Ctrl+Shift+Esc
17
+ ```
18
+
19
+ Navigate to the Performance tab to see the used and available memory, as well as swap usage.
20
+
21
+2. Check per-process memory usage to find the top consumers.
22
+
23
+ In the Task Manager, navigate to the Processes tab. Sort the processes by memory usage to identify the processes consuming the most memory.
24
+
25
+3. Optimize or close the high memory-consuming processes.
26
+
27
+ Analyze the processes and determine whether they are essential. Terminate or optimize non-critical processes that consume a significant amount of memory. Ensure to double-check before closing any process to avoid unintentionally closing necessary processes.
28
+
29
+4. Increase the system's memory or adjust swap file settings.
30
+
31
+ If your system consistently runs low on memory, consider upgrading the hardware to add more RAM or adjusting the swap memory settings to allocate more disk space.
32
+
33
+5. Prevent memory leaks.
34
+
35
+ Memory leaks occur when an application uses memory but fails to release it when no longer needed, causing gradual memory depletion. Ensure that all software running on your system, particularly custom or in-house applications, is well-designed and tested for memory leaks.
36
+
37
+### Useful resources
38
+
39
+1. [How to Manage Virtual Memory (Pagefile) in Windows 10](https://www.techbout.com/manage-virtual-memory-pagefile-windows-10-29638/)
40
+2. [Troubleshooting Windows Performance Issues Using the Resource Monitor](https://docs.microsoft.com/en-us/archive/blogs/askcore/troubleshooting-windows-performance-issues-using-the-resource-monitor)
41
+3. [Windows Performance Monitor](https://docs.microsoft.com/en-us/windows-server/administration/windows-server-2008-help/troubleshoot/windows-rel-performance-monitor)
\ No newline at end of file
health/guides/x509check/x509check_days_until_expiration.md
deleted
-111
@@ -1,111 +0,0 @@
1
-# x509check_days_until_expiration
2
-
3
-**Certificates | x509 certificates**
4
-
5
-_An X.509 certificate is a digital certificate based on the widely accepted International
6
-Telecommunications Union (ITU) X.509 standard, which defines the format of public key
7
-infrastructure (PKI) certificates. They are used to manage identity and security in internet
8
-communications and computer networking. They are unobtrusive and ubiquitous, and we encounter them
9
-every day when using websites, mobile apps, online documents, and connected
10
-devices._ <sup>[1](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.) </sup>
11
-
12
-The Netdata Agent monitors the time until an X.509 certificate expires. This alert indicates that,
13
-the X.509 certificate will expire soon. Check more about
14
-the [x509 certificate monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/x509check).
15
-
16
-By default, this alert is triggered in warning state when your certificate has less than 14 days to expire and
17
-in critical state when it has less than 7 days to expire, but these levels are configurable.
18
-
19
-A certification authority (CA) is an entity that issues digital certificates. A digital certificate
20
-certifies the ownership of a public key by the named subject of the certificate. This allows
21
-others (relying parties) to rely upon signatures or on assertions made about the private key that
22
-corresponds to the certified public key. A CA acts as a trusted third party—trusted both by the
23
-subject (owner) of the certificate and by the party relying upon the certificate. The format of
24
-these certificates is specified by the X.509 or EMV standard.
25
-
26
-<details>
27
-<summary>Where and why we need X.509 certificates</summary>
28
-
29
-The following provides a comprehensive explanation from the sectigo's website: <sup> [1](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.) </sup>
30
-
31
-
32
-Common Applications of X.509 Public Key Infrastructure Many internet protocols rely on X.509, and
33
-there are many applications of the PKI technology that are used every day, including Web server
34
-security, digital signatures and document signing, and digital identities.
35
-
36
-- **Web Server Security with TLS/SSL Certificates:**
37
- PKI is the basis for the secure sockets layer (SSL)
38
- and transport layer security (TLS) protocols that are the foundation of HTTPS secure browser
39
- connections. Without SSL certificates or TLS to establish secure connections, cybercriminals could
40
- exploit the Internet or other IP networks using a variety of attack vectors, such as
41
- man-in-the-middle attacks, to intercept messages and access their contents.
42
-
43
-- **Digital Signatures and Document Signing:**
44
- In addition to being used to secure messages, PKI-based certificates can be used for digital
45
- signatures and document signing. Digital signatures are a specific type of electronic signature
46
- that leverages PKI to authenticate the identity of the signer and the integrity of the signature
47
- and the document. Digital signatures cannot be altered or duplicated in any way, as the signature
48
- is created by generating a hash, which is encrypted using a sender's private key. This
49
- cryptographic verification mathematically binds the signature to the original message to ensure
50
- that the sender is authenticated and the message itself has not been altered.
51
-
52
-- **Code Signing:**
53
- Code Signing enables application developers to add a layer of assurance by digitally signing
54
- applications, drivers, and software programs so that end users can verify that a third party has
55
- not altered or compromised the code they receive. To verify the code is safe and trusted, these
56
- digital certificates include the software developer's signature, the company name, and
57
- timestamping.
58
-
59
-- **Email Certificates:**
60
- S/MIME certificates validate email senders and encrypt email contents to protect against
61
- increasingly sophisticated social engineering and spear phishing attacks. By encrypting/decrypting
62
- email messages and attachments and by validating identity, S/MIME email certificates assure users
63
- that emails are authentic and unmodified.
64
-
65
-- **SSH Keys:**
66
- SSH keys are a form of X.509 certificate that provides a secure access credential used in the
67
- Secure Shell (SSH) protocol. As the SSH protocol is widely used for communication in cloud
68
- services, network environments, file transfer tools, and configuration management tools, most
69
- organizations use SSH keys to authenticate identity and protect those services from unintended use
70
- or malicious attacks. SSH keys not only improve security, but also enable the automation of
71
- connected processes, single sign-on (SSO), and identity and access management at the scale that
72
- today's businesses require.
73
-
74
-- **Digital Identities:**
75
- X.509 digital certificates also provide effective digital identity authentication. As data and
76
- applications expand beyond traditional networks to mobile devices, public clouds, private clouds,
77
- and Internet of Things devices, securing identities becomes more important than ever. And digital
78
- identities don't have to be restricted to devices; they can also be used to authenticate people,
79
- data, or applications. Digital identity certificates based on this standard enable organizations
80
- to improve security by replacing passwords, which attackers have become increasingly adept at
81
- stealing.
82
-
83
-</details>
84
-
85
-<details>
86
-<summary>Popular CAs </summary>
87
-
88
-1. https://letsencrypt.org/
89
-2. https://securitycloud.symantec.com/cc/landing
90
-3. https://www.geotrust.com/
91
-4. https://sectigo.com/
92
-5. https://www.digicert.com/
93
-
94
-</details>
95
-
96
-<details>
97
-<summary>References and source </summary>
98
-
99
-1. [X.509 explained](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.)
100
-
101
-</details>
102
-
103
-
104
-### Troubleshooting section
105
-
106
-Anyone can issue an X.509 certificate, and a X.509 certificate may or may not have an expiration date.
107
-In most cases the certificates which are issued by a CA have a validity period. In order to
108
-persist your certificate's validity, you must either renew (or re-key) it. If your certificate is
109
-issued by a CA, you must manage it from your CA.
110
-
111
-
health/guides/x509check/x509check_revocation_status.md
deleted
-120
@@ -1,120 +0,0 @@
1
-# x509check_revocation_status
2
-
3
-**Certificates | x509 certificates**
4
-
5
-_An X.509 certificate is a digital certificate based on the widely accepted International
6
-Telecommunications Union (ITU) X.509 standard, which defines the format of public key
7
-infrastructure (PKI) certificates. They are used to manage identity and security in internet
8
-communications and computer networking. They are unobtrusive and ubiquitous, and we encounter them
9
-every day when using websites, mobile apps, online documents, and connected
10
-devices._ <sup>[1](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.) </sup>
11
-
12
-The certificate is also a confirmation or validation by an trusted Certificate Authority (CA) that the public
13
-key contained in the certificate belongs to the person, organization, server or other entity noted in the
14
-certificate.
15
-
16
-For many reasons, primary, security reasons, we may want to revoke the validity of an X.509
17
-certificate. In a nutshell, an X.509 certificate should be revoked when:
18
-
19
-- Encryption keys of the certificate have been compromised.
20
-- Errors occur within an issued certificate.
21
-- We want to change the usage of the certificate.
22
-- Certificate owner is no longer deemed trusted.
23
-
24
-The Netdata Agent checks the X.509 certificate revocation status (0: revoked, 1: valid). This alert
25
-indicates that the X.509 certificate has been revoked. Check more about
26
-the [x509 certificate monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/x509check)
27
-
28
-This alert is triggered in critical state when the X.509 certificate is available and not valid.
29
-
30
-<details>
31
-<summary>Where and why we need X.509 certificates</summary>
32
-
33
-The following provides a comprehensive explanation from the sectigo's
34
-website <sup> [1](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.) </sup>
35
-
36
-Common Applications of X.509 Public Key Infrastructure Many internet protocols rely on X.509, and
37
-there are many applications of the PKI technology that are used every day, including Web server
38
-security, digital signatures and document signing, and digital identities.
39
-
40
-- **Web Server Security with TLS/SSL Certificates:**
41
- PKI is the basis for the secure sockets layer (SSL)
42
- and transport layer security (TLS) protocols that are the foundation of HTTPS secure browser
43
- connections. Without SSL certificates or TLS to establish secure connections, cybercriminals could
44
- exploit the Internet or other IP networks using a variety of attack vectors, such as
45
- man-in-the-middle attacks, to intercept messages and access their contents.
46
-
47
-- **Digital Signatures and Document Signing:**
48
- In addition to being used to secure messages, PKI-based certificates can be used for digital
49
- signatures and document signing. Digital signatures are a specific type of electronic signature
50
- that leverages PKI to authenticate the identity of the signer and the integrity of the signature
51
- and the document. Digital signatures cannot be altered or duplicated in any way, as the signature
52
- is created by generating a hash, which is encrypted using a sender's private key. This
53
- cryptographic verification mathematically binds the signature to the original message to ensure
54
- that the sender is authenticated and the message itself has not been altered.
55
-
56
-- **Code Signing:**
57
- Code Signing enables application developers to add a layer of assurance by digitally signing
58
- applications, drivers, and software programs so that end users can verify that a third party has
59
- not altered or compromised the code they receive. To verify the code is safe and trusted, these
60
- digital certificates include the software developer's signature, the company name, and
61
- timestamping.
62
-
63
-- **Email Certificates:**
64
- S/MIME certificates validate email senders and encrypt email contents to protect against
65
- increasingly sophisticated social engineering and spear phishing attacks. By encrypting/decrypting
66
- email messages and attachments and by validating identity, S/MIME email certificates assure users
67
- that emails are authentic and unmodified.
68
-
69
-- **SSH Keys:**
70
- SSH keys are a form of X.509 certificate that provides a secure access credential used in the
71
- Secure Shell (SSH) protocol. As the SSH protocol is widely used for communication in cloud
72
- services, network environments, file transfer tools, and configuration management tools, most
73
- organizations use SSH keys to authenticate identity and protect those services from unintended use
74
- or malicious attacks. SSH keys not only improve security, but also enable the automation of
75
- connected processes, single sign-on (SSO), and identity and access management at the scale that
76
- today's businesses require.
77
-
78
-- **Digital Identities:**
79
- X.509 digital certificates also provide effective digital identity authentication. As data and
80
- applications expand beyond traditional networks to mobile devices, public clouds, private clouds,
81
- and Internet of Things devices, securing identities becomes more important than ever. And digital
82
- identities don't have to be restricted to devices; they can also be used to authenticate people,
83
- data, or applications. Digital identity certificates based on this standard enable organizations
84
- to improve security by replacing passwords, which attackers have become increasingly adept at
85
- stealing.
86
-
87
-</details>
88
-
89
-<details>
90
-<summary>See more about the Certificate Authorities</summary>
91
-
92
-A certification authority (CA) is an entity that issues digital certificates. A digital certificate
93
-certifies the ownership of a public key by the named subject of the certificate. This allows
94
-others (relying parties) to rely upon signatures or on assertions made about the private key that
95
-corresponds to the certified public key. A CA acts as a trusted third party—trusted both by the
96
-subject (owner) of the certificate and by the party relying upon the certificate. The format of
97
-these certificates is specified by the X.509 or EMV standard.
98
-
99
-**Popular CAs**
100
-
101
- 1. https://letsencrypt.org/
102
- 2. https://securitycloud.symantec.com/cc/landing
103
- 3. https://www.geotrust.com/
104
- 4. https://sectigo.com/
105
- 5. https://www.digicert.com/
106
-
107
-</details>
108
-
109
-<details>
110
-<summary>References and source </summary>
111
-
112
-1. [X.509 explained](https://sectigo.com/resource-library/what-is-x509-certificate#:~:text=Share%20this-,An%20X.,internet%20communications%20and%20computer%20networking.)
113
-
114
-</details>
115
-
116
-
117
-### Troubleshooting section
118
-
119
-A revocation of a certificate is irreversible. That means that this certificate is no longer
120
-useful. You must stop using it in any way.
health/guides/x509check_days_until_expiration.md
new
+45
@@ -0,0 +1,45 @@
1
+### Understand the alert
2
+
3
+This alert indicates that your X.509 certificate will expire soon. By default, it is triggered in a warning state when your certificate has less than 14 days to expire and in a critical state when it has less than 7 days to expire. However, these levels are configurable.
4
+
5
+An X.509 certificate is a digital certificate used to manage identity and security in internet communications and computer networking. If your certificate expires, your system may encounter security and authentication issues which can disrupt your services.
6
+
7
+### Troubleshoot the alert
8
+
9
+**Step 1: Check the certificate's expiration details**
10
+
11
+To check the details of your X.509 certificate, including its expiration date, run the following command:
12
+
13
+```
14
+openssl x509 -in path/to/your/certificate.crt -text -noout
15
+```
16
+
17
+Replace `path/to/your/certificate.crt` with the path to your X.509 certificate file.
18
+
19
+**Step 2: Renew or re-key the certificate**
20
+
21
+If your X.509 certificate is issued by a Certification Authority (CA), you need to renew or re-key the certificate before it expires. The process for renewing or re-keying your certificate depends on your CA. Refer to your CA's documentation or help resources for guidance.
22
+
23
+Examples of popular CAs include:
24
+
25
+1. [Let's Encrypt](https://letsencrypt.org/)
26
+2. [Symantec](https://securitycloud.symantec.com/cc/landing)
27
+3. [GeoTrust](https://www.geotrust.com/)
28
+4. [Sectigo](https://sectigo.com/)
29
+5. [DigiCert](https://www.digicert.com/)
30
+
31
+**Step 3: Update your system with the new certificate**
32
+
33
+After renewing or re-keying your certificate, you need to update your system with the new certificate file. The process for updating your system depends on the services and platforms you are using. Refer to their documentation for guidance on how to update your certificate.
34
+
35
+**Step 4: Verify the new certificate**
36
+
37
+Ensure that your system is running with the updated certificate by checking its details again, as described in Step 1.
38
+
39
+If there are still issues or the alert persists, double-check your certificate management process and consult your CA's documentation for any additional help or support.
40
+
41
+### Useful resources
42
+
43
+1. [Sectigo: What is an X.509 certificate?](https://sectigo.com/resource-library/what-is-x509-certificate)
44
+2. [Netdata: x509 certificate monitoring](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/x509check)
45
+3. [OpenSSL: X.509 Certificate Commands](https://www.openssl.org/docs/man1.1.1/man1/x509.html)
\ No newline at end of file
health/guides/x509check_revocation_status.md
new
+35
@@ -0,0 +1,35 @@
1
+### Understand the alert
2
+
3
+This alert indicates that the X.509 certificate has been revoked, meaning that it is no longer valid or trusted. The certificate can be revoked for various reasons, such as key compromise, errors within the certificate, change of usage, or the certificate owner no longer being deemed trustworthy.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Identify the affected certificate**: The alert should provide information about the affected X.509 certificate. Take note of the certificate's details, such as the domain name, subject, issuer, and serial number.
8
+
9
+2. **Verify the revocation status**: You can use the `openssl` command to verify the revocation status of the affected certificate. Use the following command to check the certificate against the Certificate Revocation List (CRL) provided by the CA:
10
+
11
+ ```
12
+ openssl verify -crl_check -CAfile CA_certificate.pem -CRLfile CRL.pem certificate.pem
13
+ ```
14
+
15
+ Replace `CA_certificate.pem`, `CRL.pem`, and `certificate.pem` with the appropriate file names of the CA certificate, CRL file, and the target X.509 certificate.
16
+
17
+ Alternatively, you can use online tools such as [SSL Shopper's SSL Checker](https://www.sslshopper.com/ssl-checker.html) to verify the revocation status. Be sure to input the domain and port associated with the revoked certificate.
18
+
19
+3. **Remove or replace the revoked certificate**: If you have confirmed that the certificate is indeed revoked, you should stop using it immediately. Remove the revoked certificate from your server or application, and replace it with a valid one.
20
+
21
+ - If the certificate was issued by a commercial CA, you can request a new certificate from the CA. The CA might provide you with a free replacement or require you to purchase a new one.
22
+ - If the certificate was issued by [Let's Encrypt](https://letsencrypt.org/), you can renew the certificate using [Certbot](https://certbot.eff.org/) or another ACME client.
23
+ - If the certificate was self-signed, you can create a new self-signed certificate using the `openssl` command or another certificate management tool.
24
+
25
+4. **Update server or application configuration**: After obtaining a new certificate, update your server or application configuration to use the new certificate. Make sure to restart the server or application for the changes to take effect.
26
+
27
+5. **Monitor the new certificate**: Keep an eye on the new certificate's status using the X.509 monitoring tools provided by Netdata. Regularly check for any new alerts or changes in the certificate's status.
28
+
29
+### Useful resources
30
+
31
+1. [X.509 Certificate Monitoring with Netdata](https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/x509check)
32
+2. [How to use OpenSSL to verify a certificate against a CRL](https://raymii.org/s/tutorials/OpenSSL_command_line_Root_and_Intermediate_CA_including_OCSP_CRL_Signed_Certs.html)
33
+3. [SSL Shopper's SSL Checker](https://www.sslshopper.com/ssl-checker.html)
34
+4. [Renewing certificates with Certbot](https://certbot.eff.org/docs/using.html#renewing-certificates)
35
+5. [Creating a Self-Signed SSL Certificate](https://www.akadia.com/services/ssh_test_certificate.html)
\ No newline at end of file
health/guides/zfs/zfs_memory_throttle.md
deleted
-75
@@ -1,75 +0,0 @@
1
-# zfs_memory_throttle
2
-
3
-## OS: Linux | FreeBSD
4
-
5
-This alert presents the number of times ZFS had to limit the Adaptive Replacement Cache (ARC) growth in the last 10 minutes.
6
-
7
-The alert is raised to a warning state when the metric starts counting (when it is greater than 0).
8
-
9
-<details>
10
-<summary>Linux: What is the ARC?</summary>
11
-
12
-The ARC stores the most recently used, and most frequently used data within RAM.
13
-Having a large ARC can take up a lot of RAM, but it will decrease as other
14
-applications need it. ARC can be set to customized optimal settings for your system.
15
-
16
-ARC uses varying portions of the most recently used and the most often used data by
17
-allocating more space to one or the other whenever a cold hit occurs. A cold hit occurs when
18
-some data is requested that was previously cached, but has already been pushed out to allow
19
-the ARC to store new data. ZFS keeps track of what data was stored in the cache after it is
20
-removed in order to enable the recognition of cold hits. As new data comes in, data that
21
-hasn't been used in a while, or that has not been used as much as the new data, will be pushed
22
-out.
23
-
24
-The more RAM your system has the better, as it will just give you enhanced read performance. There
25
-will be physical and cost limitations to adding more ARC due to motherboard RAM slots and budget
26
-constraints.<sup>[1](https://www.45drives.com/community/articles/zfs-caching/) </sup>
27
-
28
-</details>
29
-
30
-<br>
31
-
32
-<details>
33
-<summary>FreeBSD: What is the ARC?</summary>
34
-
35
-The ARC functions by storing the most recently used, and most frequently used data within RAM.
36
-Having a large ARC can take up a lot of RAM, but it will give it up as other applications need
37
-it and can be set to whatever you think is optimal for your system.
38
-
39
-**Single Copy ARC**
40
-OpenZFS caches disk blocks in-memory in the adaptive replacement cache (ARC). Originally when
41
-the same disk block was accessed from different clones it was cached multiple times (one for
42
-each clone accessing the block) in case a clone planned to modify the block. OpenZFS caches
43
-at most one copy of every block unless a clone is actually modifying the block.
44
-
45
-**ARC Shouldn't Cache Freed Blocks**
46
-Originally cached blocks in the ARC remained cached until they were evicted due to memory
47
-pressure, even if the underlying disk block was freed. In some workloads these freed blocks
48
-were so frequently accessed before they were freed that the ARC continued to cache them while
49
-evicting blocks which had not been freed yet. Since freed blocks could never be accessed
50
-again continuing to cache them was unnecessary. In OpenZFS ARC blocks are evicted immediately
51
-when their underlying data blocks are freed.<sup>[2](https://openzfs.org/w/index.php?title=Features&mobileaction=toggle_view_mobile#Single_Copy_ARC)
52
-</sup>
53
-
54
-</details>
55
-
56
-<br>
57
-
58
-<details>
59
-<summary>References and Sources</summary>
60
-
61
-1. [Linux: ZFS Caching](https://www.45drives.com/community/articles/zfs-caching/)
62
-2. [FreeBSD: OpenZFS documentation](https://openzfs.org/w/index.php?title=Features&mobileaction=toggle_view_mobile#Single_Copy_ARC)
63
- </details>
64
-
65
-### Troubleshooting Section
66
-
67
-<details>
68
-<summary>Linux | FreeBSD: Increase your RAM capacity and effectively increase ARC size</summary>
69
-
70
-ZFS will throttle the ARC growth as the system needs more RAM for other tasks.
71
-If you are experiencing a lot of throttling, then you should consider increasing your RAM capacity.
72
-If the ARC size needs to be limited, then the read performance of the system will drop and cold hits
73
-are more likely to happen.
74
-
75
-</details>
health/guides/zfs/zfs_pool_state_crit.md
deleted
-79
@@ -1,79 +0,0 @@
1
-# zfs_pool_state_crit
2
-
3
-## OS: Any
4
-
5
-ZFS is a local file system and logical volume manager created by Sun Microsystems Inc. to direct and
6
-control the placement, storage, and retrieval of data in enterprise-class computing systems. ZFS is
7
-scalable, suitable for high storage capacities, and includes extensive protection against data corruption.
8
-
9
-The Netdata Agent monitors the state of the ZFS pool. Receiving this alert means that the ZFS pool
10
-is faulted or unavailable.
11
-
12
-<details>
13
-<summary>ZFS pool health status</summary>
14
-
15
-The ZFS pool health status as described in the Oracle's
16
-website <sup>[1](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html) </sup>
17
-
18
-ZFS provides an integrated method of examining pool and device health. The health of a pool is
19
-determined from the state of all its devices. This state information is displayed by using the `zpool
20
-status` command. In addition, potential pool and device failures are reported by fmd, displayed on
21
-the system console, and logged in the /var/adm/messages file.
22
-
23
-Each device can fall into one of the following states:
24
-
25
-- ONLINE, the device or virtual device is in normal working order. Although some transient errors
26
- might still occur, the device is otherwise in working order.
27
-
28
-- DEGRADED, the virtual device has experienced a failure but can still function. This state is most
29
- common when a mirror or RAID-Z device has lost one or more constituent devices. The fault
30
- tolerance of the pool might be compromised, as a subsequent fault in another device might be
31
- unrecoverable.
32
-
33
-- FAULTED, the device or virtual device is completely inaccessible. This status typically indicates
34
- total failure of the device, such that ZFS is incapable of sending data to it or receiving data
35
- from it. If a top-level virtual device is in this state, then the pool is completely inaccessible.
36
-
37
-- OFFLINE, the device has been explicitly taken offline by the administrator.
38
-
39
-- UNAVAIL, the device or virtual device cannot be opened. In some cases, pools with UNAVAIL devices
40
- appear in DEGRADED mode. If a top-level virtual device is UNAVAIL, then nothing in the pool can be
41
- accessed.
42
-
43
-- REMOVED, the device was physically removed while the system was running. Device removal detection
44
- is hardware-dependent and might not be supported on all platforms.
45
-
46
-The health of a pool is determined from the health of all its top-level virtual devices. If all
47
-virtual devices are ONLINE, then the pool is also ONLINE. If any one of the virtual devices is
48
-DEGRADED or UNAVAIL, then the pool is also DEGRADED. If a top-level virtual device is FAULTED or
49
-OFFLINE, then the pool is also FAULTED. A pool in the FAULTED state is completely inaccessible. No
50
-data can be recovered until the necessary devices are attached or repaired. A pool in the DEGRADED
51
-state continues to run, but you might not achieve the same level of data redundancy or data
52
-throughput than if the pool were online. <sup>[1](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html) </sup>
53
-
54
-</details>
55
-
56
-<details>
57
-<summary>References and source</summary>
58
-
59
-1. [Determining the Health Status of ZFS Storage Pools](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html)
60
-1. [Chapter 11, Oracle Solaris ZFS Troubleshooting and Pool Recovery](https://docs.oracle.com/cd/E53394_01/html/E54801/gavwg.html)
61
-1. [ZFS on FreeBSD documentation](https://docs.freebsd.org/en/books/handbook/zfs/)
62
-1. [OpenZFS documentation](https://openzfs.github.io/openzfs-docs/)
63
-
64
-</details>
65
-
66
-### Troubleshooting section
67
-
68
-<details>
69
-<summary>Migrate ZFS Storage Pools</summary>
70
-
71
-If the state of the ZFS pool is UNAVAIL, then you should consider migrating your ZFS pool. To do so follow the
72
-workflow at the [official troubleshooting section in Oracle's website](https://docs.oracle.com/cd/E53394_01/html/E54801/gbchy.html#scrolltoc).
73
-
74
-In this workflow, there are no notable changes in the commands for FreeBSD.
75
-<sup>[3](https://docs.freebsd.org/en/books/handbook/zfs/) </sup> or ZFS on
76
-linux <sup>[4](https://openzfs.github.io/openzfs-docs/) </sup>, for completeness you can refer to
77
-individual guides.
78
-
79
-</details>
health/guides/zfs/zfs_pool_state_warn.md
deleted
-113
@@ -1,113 +0,0 @@
1
-# zfs_pool_state_warn
2
-
3
-## OS: Any
4
-
5
-ZFS is a local file system and logical volume manager created by Sun Microsystems Inc. to direct and
6
-control the placement, storage, and retrieval of data in enterprise-class computing systems. ZFS is
7
-scalable, suitable for high storage capacities, and includes extensive protection against data corruption.
8
-
9
-The Netdata Agent monitors the state of the ZFS pool. Receiving this alert means that the ZFS pool
10
-is degraded.
11
-
12
-<details>
13
-<summary>See more on ZFS pool health status</summary>
14
-
15
-The ZFS pool health status as described in the Oracle's
16
-website <sup>[1](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html) </sup>
17
-
18
-ZFS provides an integrated method of examining pool and device health. The health of a pool is
19
-determined from the state of all its devices. This state information is displayed by using the zpool
20
-status command. In addition, potential pool and device failures are reported by fmd, displayed on
21
-the system console, and logged in the /var/adm/messages file.
22
-
23
-Each device can fall into one of the following states:
24
-
25
-- ONLINE, the device or virtual device is in normal working order. Although some transient errors
26
- might still occur, the device is otherwise in working order.
27
-
28
-- DEGRADED, the virtual device has experienced a failure but can still function. This state is most
29
- common when a mirror or RAID-Z device has lost one or more constituent devices. The fault
30
- tolerance of the pool might be compromised, as a subsequent fault in another device might be
31
- unrecoverable.
32
-
33
-- FAULTED, the device or virtual device is completely inaccessible. This status typically indicates
34
- total failure of the device, such that ZFS is incapable of sending data to it or receiving data
35
- from it. If a top-level virtual device is in this state, then the pool is completely inaccessible.
36
-
37
-- OFFLINE, the device has been explicitly taken offline by the administrator.
38
-
39
-- UNAVAIL, the device or virtual device cannot be opened. In some cases, pools with UNAVAIL devices
40
- appear in DEGRADED mode. If a top-level virtual device is UNAVAIL, then nothing in the pool can be
41
- accessed.
42
-
43
-- REMOVED, the device was physically removed while the system was running. Device removal detection
44
- is hardware-dependent and might not be supported on all platforms.
45
-
46
-The health of a pool is determined from the health of all its top-level virtual devices. If all
47
-virtual devices are ONLINE, then the pool is also ONLINE. If any one of the virtual devices is
48
-DEGRADED or UNAVAIL, then the pool is also DEGRADED. If a top-level virtual device is FAULTED or
49
-OFFLINE, then the pool is also FAULTED. A pool in the FAULTED state is completely inaccessible. No
50
-data can be recovered until the necessary devices are attached or repaired. A pool in the DEGRADED
51
-state continues to run, but you might not achieve the same level of data redundancy or data
52
-throughput than if the pool were online. <sup>[1](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html) </sup>
53
-
54
-</details>
55
-
56
-
57
-<details>
58
-<summary>References and source</summary>
59
-
60
-1. [Determining the Health Status of ZFS Storage Pools](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html)
61
-1. [Chapter 11, Oracle Solaris ZFS Troubleshooting and Pool Recovery](https://docs.oracle.com/cd/E53394_01/html/E54801/gavwg.html)
62
-1. [AlchemyCS blogspot](https://alchemycs.com/2019/05/how-to-force-zfs-to-replace-a-failed-drive-in-place/)
63
-
64
-</details>
65
-
66
-### Troubleshooting section
67
-
68
-<details>
69
-<summary>Replace a failed drive</summary>
70
-
71
-Data is priceless. Before you perform any action, make sure that you have taken any necessary backup
72
-steps. Netdata is not liable for any loss or corruption of any data, database, or software.
73
-
74
-Based on a nice guide from the Alchemycs blogspot.
75
-
76
-1. Check your zpool status
77
-
78
-```
79
-root@netdata # zpool status my_pool
80
- pool: my_pool
81
- state: DEGRADED
82
-status: One or more devices could not be used because the label is missing or
83
- invalid. Sufficient replicas exist for the pool to continue
84
- functioning in a degraded state.
85
-action: Replace the device using 'zpool replace'.
86
- see: http://zfsonlinux.org/msg/ZFS-8000-4J
87
- scan: scrub repaired 0B in 0h0m with 0 errors on Sun May 12 00:24:51 2019
88
-config:
89
-
90
- NAME STATE READ WRITE CKSUM
91
- my_pool DEGRADED 0 0 0
92
- mirror-0 DEGRADED 0 0 0
93
- 52009894889112747750 UNAVAIL 0 0 0 was /dev/sdm5
94
- sdb5 ONLINE 0 0 0errors: No known data errors
95
-```
96
-
97
-1. Find the UUIDs (for GPT) of the faulty dev (the UNAVAIL) and the new disk you want to add.
98
- Use `blkid` for Linux or `geom` utility for FreeBSD
99
-
100
-
101
-1. Offline the UNAVAIL drive
102
-
103
-```
104
-root@netdata # zpool offline my_pool /dev/disk/by-uuid/{UUID_BAD_DRIVE}
105
-```
106
-
107
-1. Replace it in place
108
-
109
-```
110
-root@netdata # zpool replace -f my_pool /dev/disk/by-uuid/{UUID_OLD_DRIVE} /dev/disk/by-uuid/{UUID_NEW_DRIVE}
111
-```
112
-
113
-</details>
health/guides/zfs_memory_throttle.md
new
+21
@@ -0,0 +1,21 @@
1
+### Understand the alert
2
+
3
+This alert indicates the number of times ZFS had to limit the Adaptive Replacement Cache (ARC) growth in the last 10 minutes. ARC stores the most recently used and most frequently used data in RAM, helping to improve read performance. When ARC growth is throttled, it can impact read performance due to a higher chance of cold hits.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Monitor RAM usage**: Check your system's RAM usage to determine if there is sufficient memory available for ARC. If other processes are consuming a large amount of RAM, ARC growth may be throttled to free up resources.
8
+
9
+2. **Increase RAM capacity**: If you consistently experience ARC throttling, consider increasing your RAM capacity. This will allow for a larger ARC size, improving read performance and reducing the likelihood of cold hits.
10
+
11
+3. **Adjust ARC size**: If you are using ZFS on Linux, you can adjust the ARC size by modifying the `zfs_arc_min` and `zfs_arc_max` parameters in the `/etc/modprobe.d/zfs.conf` file. On FreeBSD, you can adjust the `vfs.zfs.arc_max` sysctl parameter. Make sure to set these values according to your system's RAM capacity and workload requirements.
12
+
13
+4. **Evaluate workload**: Analyze your system's workload to identify if there are any specific processes or applications that are causing high memory usage, leading to ARC throttling. Optimize or limit these processes if necessary.
14
+
15
+
16
+### Useful resources
17
+
18
+1. [Linux: ZFS Caching](https://www.45drives.com/community/articles/zfs-caching/)
19
+2. [FreeBSD: OpenZFS documentation](https://openzfs.org/w/index.php?title=Features&mobileaction=toggle_view_mobile#Single_Copy_ARC)
20
+3. [ZFS on Linux Performance Tuning Guide](https://github.com/zfsonlinux/zfs/wiki/Performance-Tuning)
21
+4. [FreeBSD ZFS Tuning Guide](https://wiki.freebsd.org/ZFSTuningGuide)
health/guides/zfs_pool_state_crit.md
new
+58
@@ -0,0 +1,58 @@
1
+### Understand the alert
2
+
3
+The `zfs_pool_state_crit` alert indicates that your ZFS pool is faulted or unavailable, which can cause access and data loss problems. It is important to identify the current state of the pool and take corrective actions to remedy the situation.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check the current ZFS pool state**
8
+
9
+ Run the `zpool status` command to view the status of all ZFS pools:
10
+
11
+ ```
12
+ zpool status
13
+ ```
14
+
15
+ This will display the pool state, device states, and any errors that occurred. Take note of any devices that are in DEGRADED, FAULTED, UNAVAIL, or OFFLINE states.
16
+
17
+2. **Assess the problematic devices**
18
+
19
+ Check for any hardware issues or file system errors on the affected devices. For example, if a device is FAULTED due to a hardware failure, replace the device. If a device is UNAVAIL or OFFLINE, check the connectivity and make sure it's properly accessible.
20
+
21
+3. **Repair the pool**
22
+
23
+ Depending on the root cause of the problem, you may need to take different actions:
24
+
25
+ - Repair file system errors using the `zpool scrub` command. This will initiate a scrub, which attempts to fix any errors in the pool.
26
+
27
+ ```
28
+ zpool scrub [pool_name]
29
+ ```
30
+
31
+ - Replace a failed device using the `zpool replace` command. For example, if you have a new device `/dev/sdb` that will replace `/dev/sda`, run the following command:
32
+
33
+ ```
34
+ zpool replace [pool_name] /dev/sda /dev/sdb
35
+ ```
36
+
37
+ - Bring an OFFLINE device back ONLINE using the `zpool online` command:
38
+
39
+ ```
40
+ zpool online [pool_name] [device]
41
+ ```
42
+
43
+ Note: Make sure to replace `[pool_name]` and `[device]` with the appropriate values for your system.
44
+
45
+4. **Verify the pool state**
46
+
47
+ After taking the necessary corrective actions, run the `zpool status` command again to verify that the pool state has improved.
48
+
49
+5. **Monitor pool health**
50
+
51
+ Continuously monitor the health of your ZFS pools to avoid future issues. Consider setting up periodic scrubs and reviewing system logs to catch any hardware or file system errors.
52
+
53
+### Useful resources
54
+
55
+1. [Determining the Health Status of ZFS Storage Pools](https://docs.oracle.com/cd/E19253-01/819-5461/gamno/index.html)
56
+2. [Chapter 11, Oracle Solaris ZFS Troubleshooting and Pool Recovery](https://docs.oracle.com/cd/E53394_01/html/E54801/gavwg.html)
57
+3. [ZFS on FreeBSD documentation](https://docs.freebsd.org/en/books/handbook/zfs/)
58
+4. [OpenZFS documentation](https://openzfs.github.io/openzfs-docs/)
\ No newline at end of file
health/guides/zfs_pool_state_warn.md
new
+20
@@ -0,0 +1,20 @@
1
+### Understand the alert
2
+
3
+This alert is triggered when the state of a ZFS pool changes to a warning state, indicating potential issues with the pool, such as disk errors, corruption, or degraded performance.
4
+
5
+### Troubleshoot the alert
6
+
7
+1. **Check pool status**: Use the `zpool status` command to check the status of the ZFS pool and identify any issues or errors.
8
+
9
+2. **Review disk health**: Inspect the health of the disks in the ZFS pool using `smartctl` or other disk health monitoring tools.
10
+
11
+3. **Replace faulty disks**: If a disk in the ZFS pool is faulty, replace it with a new one and perform a resilvering operation using `zpool replace`.
12
+
13
+4. **Scrub the pool**: Run a manual scrub operation on the ZFS pool with `zpool scrub` to verify data integrity and repair any detected issues.
14
+
15
+5. **Monitor pool health**: Keep an eye on the ZFS pool's health and performance metrics to ensure that issues are resolved and do not recur.
16
+
17
+### Useful resources
18
+
19
+1. [ZFS on Linux Documentation](https://openzfs.github.io/openzfs-docs/)
20
+2. [FreeBSD Handbook - ZFS](https://www.freebsd.org/doc/handbook/zfs.html)