| 1 | ### Understand the alert |
| 2 | |
| 3 | This alert is triggered when the number of failed volumes in your Hadoop Distributed File System (HDFS) cluster increases. A failed volume may be due to hardware failure or misconfiguration, such as duplicate mounts. When a single volume fails on a DataNode, the entire node may go offline depending on the `dfs.datanode.failed.volumes.tolerated` setting for your cluster. This can lead to increased network traffic and potential performance degradation as the NameNode needs to copy any under-replicated blocks lost on that node. |
| 4 | |
| 5 | ### Troubleshoot the alert |
| 6 | |
| 7 | #### 1. Identify which DataNode has a failing volume |
| 8 | |
| 9 | Use the `dfsadmin -report` command to identify the DataNodes that are offline: |
| 10 | |
| 11 | ```bash |
| 12 | root@netdata # dfsadmin -report |
| 13 | ``` |
| 14 | |
| 15 | Find any nodes that are not reported in the output of the command. If all nodes are listed, you'll need to run the next command for each DataNode. |
| 16 | |
| 17 | #### 2. Review the volumes status |
| 18 | |
| 19 | Use the `hdfs dfsadmin -getVolumeReport` command, specifying the DataNode hostname and port: |
| 20 | |
| 21 | ```bash |
| 22 | root@netdata # hdfs dfsadmin -getVolumeReport datanodehost:port |
| 23 | ``` |
| 24 | |
| 25 | #### 3. Inspect the DataNode logs |
| 26 | |
| 27 | Connect to the affected DataNode and check its logs using `journalctl -xe`. If you have the Netdata Agent running on the DataNodes, you should be able to identify the problem. You may also receive alerts about the disks and mounts on this system. |
| 28 | |
| 29 | #### 4. Take necessary actions |
| 30 | |
| 31 | Based on the information gathered in the previous steps, take appropriate actions to resolve the issue. This may include: |
| 32 | |
| 33 | - Repairing or replacing faulty hardware. |
| 34 | - Fixing misconfigurations such as duplicate mounts. |
| 35 | - Ensuring that the HDFS processes are running on the affected DataNode. |
| 36 | - Ensuring that the affected DataNode is properly communicating with the NameNode. |
| 37 | |
| 38 | **Note**: When working with HDFS, it's essential to have proper backups of your data. Netdata is not responsible for any loss or corruption of data, database, or software. |
| 39 | |
| 40 | ### Useful resources |
| 41 | |
| 42 | 1. [Apache Hadoop on Wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop) |
| 43 | 2. [HDFS architecture](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) |
| 44 | 3. [HDFS 3.3.1 commands guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html) |