| 1 | # Monitor a Hadoop cluster with Netdata |
| 2 | |
| 3 | Hadoop is an [Apache project](https://hadoop.apache.org/) is a framework for processing large sets of data across a |
| 4 | distributed cluster of systems. |
| 5 | |
| 6 | And while Hadoop is designed to be a highly available and fault-tolerant service, those who operate a Hadoop cluster |
| 7 | will want to monitor the health and performance of their [Hadoop Distributed File System |
| 8 | (HDFS)](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) and [Zookeeper](https://zookeeper.apache.org/) |
| 9 | implementations. |
| 10 | |
| 11 | Netdata comes with built-in and pre-configured support for monitoring both HDFS and Zookeeper. |
| 12 | |
| 13 | This guide assumes you have a Hadoop cluster, with HDFS and Zookeeper, running already. If you don't, please follow |
| 14 | the [official Hadoop |
| 15 | instructions](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html) or an |
| 16 | alternative, like the guide available from |
| 17 | [DigitalOcean](https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-in-stand-alone-mode-on-ubuntu-18-04). |
| 18 | |
| 19 | For more specifics on the collection modules used in this guide, read the respective pages in our documentation: |
| 20 | |
| 21 | - [HDFS](/src/go/plugin/go.d/collector/hdfs/README.md) |
| 22 | - [Zookeeper](/src/go/plugin/go.d/collector/zookeeper/README.md) |
| 23 | |
| 24 | ## Set up your HDFS and Zookeeper installations |
| 25 | |
| 26 | As with all data sources, Netdata can auto-detect HDFS and Zookeeper nodes if you installed them using the standard |
| 27 | installation procedure. |
| 28 | |
| 29 | For Netdata to collect HDFS metrics, it needs to be able to access the node's `/jmx` endpoint. You can test whether an |
| 30 | JMX endpoint is accessible by using `curl HDFS-IP:PORT/jmx`. For a NameNode, you should see output similar to the |
| 31 | following: |
| 32 | |
| 33 | ```json |
| 34 | { |
| 35 | "beans" : [ { |
| 36 | "name" : "Hadoop:service=NameNode,name=JvmMetrics", |
| 37 | "modelerType" : "JvmMetrics", |
| 38 | "MemNonHeapUsedM" : 65.67851, |
| 39 | "MemNonHeapCommittedM" : 67.3125, |
| 40 | "MemNonHeapMaxM" : -1.0, |
| 41 | "MemHeapUsedM" : 154.46341, |
| 42 | "MemHeapCommittedM" : 215.0, |
| 43 | "MemHeapMaxM" : 843.0, |
| 44 | "MemMaxM" : 843.0, |
| 45 | "GcCount" : 15, |
| 46 | "GcTimeMillis" : 305, |
| 47 | "GcNumWarnThresholdExceeded" : 0, |
| 48 | "GcNumInfoThresholdExceeded" : 0, |
| 49 | "GcTotalExtraSleepTime" : 92, |
| 50 | "ThreadsNew" : 0, |
| 51 | "ThreadsRunnable" : 6, |
| 52 | "ThreadsBlocked" : 0, |
| 53 | "ThreadsWaiting" : 7, |
| 54 | "ThreadsTimedWaiting" : 34, |
| 55 | "ThreadsTerminated" : 0, |
| 56 | "LogFatal" : 0, |
| 57 | "LogError" : 0, |
| 58 | "LogWarn" : 2, |
| 59 | "LogInfo" : 348 |
| 60 | }, |
| 61 | { ... } |
| 62 | ] |
| 63 | } |
| 64 | ``` |
| 65 | |
| 66 | The JSON result for a DataNode's `/jmx` endpoint is slightly different: |
| 67 | |
| 68 | ```json |
| 69 | { |
| 70 | "beans" : [ { |
| 71 | "name" : "Hadoop:service=DataNode,name=DataNodeActivity-dev-slave-01.dev.local-9866", |
| 72 | "modelerType" : "DataNodeActivity-dev-slave-01.dev.local-9866", |
| 73 | "tag.SessionId" : null, |
| 74 | "tag.Context" : "dfs", |
| 75 | "tag.Hostname" : "dev-slave-01.dev.local", |
| 76 | "BytesWritten" : 500960407, |
| 77 | "TotalWriteTime" : 463, |
| 78 | "BytesRead" : 80689178, |
| 79 | "TotalReadTime" : 41203, |
| 80 | "BlocksWritten" : 16, |
| 81 | "BlocksRead" : 16, |
| 82 | "BlocksReplicated" : 4, |
| 83 | ... |
| 84 | }, |
| 85 | { ... } |
| 86 | ] |
| 87 | } |
| 88 | ``` |
| 89 | |
| 90 | If Netdata can't access the `/jmx` endpoint for either a NameNode or DataNode, it will not be able to auto-detect and |
| 91 | collect metrics from your HDFS implementation. |
| 92 | |
| 93 | Zookeeper auto-detection relies on an accessible client port and an allow-listed `mntr` command. For more details on |
| 94 | `mntr`, see Zookeeper's documentation on [cluster |
| 95 | options](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_clusterOptions) and [Zookeeper |
| 96 | commands](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_zkCommands). |
| 97 | |
| 98 | ## Configure the HDFS and Zookeeper modules |
| 99 | |
| 100 | To configure Netdata's HDFS module, navigate to your Netdata directory (typically at `/etc/netdata/`) and use |
| 101 | `edit-config` to initialize and edit your HDFS configuration file. |
| 102 | |
| 103 | ```bash |
| 104 | cd /etc/netdata/ |
| 105 | sudo ./edit-config go.d/hdfs.conf |
| 106 | ``` |
| 107 | |
| 108 | At the bottom of the file, you will see two example jobs, both of which are commented out: |
| 109 | |
| 110 | ```yaml |
| 111 | # [ JOBS ] |
| 112 | #jobs: |
| 113 | # - name: namenode |
| 114 | # url: http://127.0.0.1:9870/jmx |
| 115 | # |
| 116 | # - name: datanode |
| 117 | # url: http://127.0.0.1:9864/jmx |
| 118 | ``` |
| 119 | |
| 120 | Uncomment these lines and edit the `url` value(s) according to your setup. Now's the time to add any other configuration |
| 121 | details, which you can find inside the `hdfs.conf` file itself. Most production implementations will require TLS |
| 122 | certificates. |
| 123 | |
| 124 | The result for a simple HDFS setup, running entirely on `localhost` and without certificate authentication, might look |
| 125 | like this: |
| 126 | |
| 127 | ```yaml |
| 128 | # [ JOBS ] |
| 129 | jobs: |
| 130 | - name: namenode |
| 131 | url: http://127.0.0.1:9870/jmx |
| 132 | |
| 133 | - name: datanode |
| 134 | url: http://127.0.0.1:9864/jmx |
| 135 | ``` |
| 136 | |
| 137 | At this point, Netdata should be configured to collect metrics from your HDFS servers. Let's move on to Zookeeper. |
| 138 | |
| 139 | Next, use `edit-config` again to initialize/edit your `zookeeper.conf` file. |
| 140 | |
| 141 | ```bash |
| 142 | cd /etc/netdata/ |
| 143 | sudo ./edit-config go.d/zookeeper.conf |
| 144 | ``` |
| 145 | |
| 146 | As with the `hdfs.conf` file, head to the bottom, uncomment the example jobs, and tweak the `address` values according |
| 147 | to your setup. Again, you may need to add additional configuration options, like TLS certificates. |
| 148 | |
| 149 | ```yaml |
| 150 | jobs: |
| 151 | - name : local |
| 152 | address : 127.0.0.1:2181 |
| 153 | |
| 154 | - name : remote |
| 155 | address : 203.0.113.10:2182 |
| 156 | ``` |
| 157 | |
| 158 | Finally, [restart Netdata](/docs/netdata-agent/start-stop-restart.md). |
| 159 | |
| 160 | ```sh |
| 161 | sudo systemctl restart netdata |
| 162 | ``` |
| 163 | |
| 164 | Upon restart, Netdata should recognize your HDFS/Zookeeper servers, enable the HDFS and Zookeeper modules, and begin |
| 165 | showing real-time metrics for both in your Netdata dashboard. 🎉 |
| 166 | |
| 167 | ## Configuring HDFS and Zookeeper alerts |
| 168 | |
| 169 | The Netdata community helped us create sane defaults for alerts related to both HDFS and Zookeeper. You may want to |
| 170 | investigate these to ensure they work well with your Hadoop implementation. |
| 171 | |
| 172 | - [HDFS alerts](https://raw.githubusercontent.com/netdata/netdata/master/src/health/health.d/hdfs.conf) |
| 173 | |
| 174 | You can also access/edit these files directly with `edit-config`: |
| 175 | |
| 176 | ```bash |
| 177 | sudo /etc/netdata/edit-config health.d/hdfs.conf |
| 178 | sudo /etc/netdata/edit-config health.d/zookeeper.conf |
| 179 | ``` |
| 180 | |
| 181 | For more information about editing the defaults or writing new alert entities, see our [health monitoring documentation](/src/health/README.md). |