| 1 | # Netdata, Prometheus, Grafana stack |
| 2 | |
| 3 | ## Intro |
| 4 | |
| 5 | In this article I will walk you through the basics of getting Netdata, Prometheus and Grafana all working together and |
| 6 | monitoring your application servers. This article will be using docker on your local workstation. We will be working |
| 7 | with docker in an ad-hoc way, launching containers that run `/bin/bash` and attaching a TTY to them. I use docker here |
| 8 | in a purely academic fashion and do not condone running Netdata in a container. I pick this method so individuals |
| 9 | without cloud accounts or access to VMs can try this out and for it's speed of deployment. |
| 10 | |
| 11 | ## Why Netdata, Prometheus, and Grafana |
| 12 | |
| 13 | Some time ago I was introduced to Netdata by a coworker. We were attempting to troubleshoot python code which seemed to |
| 14 | be bottlenecked. I was instantly impressed by the amount of metrics Netdata exposes to you. I quickly added Netdata to |
| 15 | my set of go-to tools when troubleshooting systems performance. |
| 16 | |
| 17 | Some time ago, even later, I was introduced to Prometheus. Prometheus is a monitoring application which flips the normal |
| 18 | architecture around and polls rest endpoints for its metrics. This architectural change greatly simplifies and decreases |
| 19 | the time necessary to begin monitoring your applications. Compared to current monitoring solutions the time spent on |
| 20 | designing the infrastructure is greatly reduced. Running a single Prometheus server per application becomes feasible |
| 21 | with the help of Grafana. |
| 22 | |
| 23 | Grafana has been the go to graphing tool for… some time now. It's awesome, anyone that has used it knows it's awesome. |
| 24 | We can point Grafana at Prometheus and use Prometheus as a data source. This allows a pretty simple overall monitoring |
| 25 | architecture: Install Netdata on your application servers, point Prometheus at Netdata, and then point Grafana at |
| 26 | Prometheus. |
| 27 | |
| 28 | I'm omitting an important ingredient in this stack in order to keep this tutorial simple and that is service discovery. |
| 29 | My personal preference is to use Consul. Prometheus can plug into consul and automatically begin to scrape new hosts |
| 30 | that register a Netdata client with Consul. |
| 31 | |
| 32 | At the end of this tutorial you will understand how each technology fits together to create a modern monitoring stack. |
| 33 | This stack will offer you visibility into your application and systems performance. |
| 34 | |
| 35 | ## Getting Started - Netdata |
| 36 | |
| 37 | To begin let's create our container which we will install Netdata on. We need to run a container, forward the necessary |
| 38 | port that Netdata listens on, and attach a tty so we can interact with the bash shell on the container. But before we do |
| 39 | this we want name resolution between the two containers to work. In order to accomplish this we will create a |
| 40 | user-defined network and attach both containers to this network. The first command we should run is: |
| 41 | |
| 42 | ```sh |
| 43 | docker network create --driver bridge netdata-tutorial |
| 44 | ``` |
| 45 | |
| 46 | With this user-defined network created we can now launch our container we will install Netdata on and point it to this |
| 47 | network. |
| 48 | |
| 49 | ```sh |
| 50 | docker run -it --name netdata --hostname netdata --network=netdata-tutorial -p 19999:19999 centos:latest '/bin/bash' |
| 51 | ``` |
| 52 | |
| 53 | This command creates an interactive tty session (`-it`), gives the container both a name in relation to the docker |
| 54 | daemon and a hostname (this is so you know what container is which when working in the shells and docker maps hostname |
| 55 | resolution to this container), forwards the local port 19999 to the container's port 19999 (`-p 19999:19999`), sets the |
| 56 | command to run (`/bin/bash`) and then chooses the base container images (`centos:latest`). After running this you should |
| 57 | be sitting inside the shell of the container. |
| 58 | |
| 59 | After we have entered the shell we can install Netdata. This process could not be easier. If you take a look at [this |
| 60 | link](/packaging/installer/README.md), the Netdata devs give us several one-liners to install Netdata. I have not had |
| 61 | any issues with these one liners and their bootstrapping scripts so far (If you guys run into anything do share). Run |
| 62 | the following command in your container. |
| 63 | |
| 64 | <!-- candidate for reuse --> |
| 65 | ```sh |
| 66 | wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh && sh /tmp/netdata-kickstart.sh --dont-wait |
| 67 | ``` |
| 68 | |
| 69 | After the install completes you should be able to hit the Netdata dashboard at <http://localhost:19999/> (replace |
| 70 | localhost if you're doing this on a VM or have the docker container hosted on a machine not on your local system). If |
| 71 | this is your first time using Netdata I suggest you take a look around. The amount of time I've spent digging through |
| 72 | `/proc` and calculating my own metrics has been greatly reduced by this tool. Take it all in. |
| 73 | |
| 74 | Next I want to draw your attention to a particular endpoint. Navigate to |
| 75 | <http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes> In your browser. This is the endpoint which |
| 76 | publishes all the metrics in a format which Prometheus understands. Let's take a look at one of these metrics. |
| 77 | `netdata_disk_space_GiB_average{chart="disk_space._run",dimension="avail",family="/run",mount_point="/run",filesystem="tmpfs",mount_root="/"} 0.0298195 1684951093000` |
| 78 | This metric is representing several things which I will go in more details in the section on Prometheus. For now understand |
| 79 | that this metric: `netdata_disk_space_GiB_average` has several labels: (`chart`, `family`, `dimension`, `mountt_point`, `filesystem`, `mount_root`). |
| 80 | This corresponds with disk space you see on the Netdata dashboard. |
| 81 | |
| 82 |  |
| 83 | |
| 84 | This CHART is called `system.cpu`, The FAMILY is `cpu`, and the DIMENSION we are observing is `system`. You can begin to |
| 85 | draw links between the charts in Netdata to the Prometheus metrics format in this manner. |
| 86 | |
| 87 | ## Prometheus |
| 88 | |
| 89 | We will be installing Prometheus in a container for purpose of demonstration. While Prometheus does have an official |
| 90 | container I would like to walk through the install process and setup on a fresh container. This will allow anyone |
| 91 | reading to migrate this tutorial to a VM or Server of any sort. |
| 92 | |
| 93 | Let's start another container in the same fashion as we did the Netdata container. |
| 94 | |
| 95 | ```sh |
| 96 | docker run -it --name prometheus --hostname prometheus \ |
| 97 | --network=netdata-tutorial -p 9090:9090 centos:latest '/bin/bash' |
| 98 | ``` |
| 99 | |
| 100 | This should drop you into a shell once again. Once there quickly install your favorite editor as we will be editing |
| 101 | files later in this tutorial. |
| 102 | |
| 103 | ```sh |
| 104 | yum install vim -y |
| 105 | ``` |
| 106 | |
| 107 | You will also need `wget` and `curl` to download files and `sudo` if you are not root. |
| 108 | |
| 109 | ```sh |
| 110 | yum install curl sudo wget -y |
| 111 | ``` |
| 112 | |
| 113 | Prometheus provides a tarball of their latest stable versions [here](https://prometheus.io/download/). |
| 114 | |
| 115 | Let's download the latest version and install into your container. |
| 116 | |
| 117 | ```sh |
| 118 | cd /tmp && curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \ |
| 119 | | grep "browser_download_url.*linux-amd64.tar.gz" \ |
| 120 | | cut -d '"' -f 4 \ |
| 121 | | wget -qi - |
| 122 | |
| 123 | mkdir /opt/prometheus |
| 124 | |
| 125 | sudo tar -xvf /tmp/prometheus-*linux-amd64.tar.gz -C /opt/prometheus --strip=1 |
| 126 | ``` |
| 127 | |
| 128 | This should get Prometheus installed into the container. Let's test that we can run Prometheus and connect to it's web |
| 129 | interface. |
| 130 | |
| 131 | ```sh |
| 132 | /opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml |
| 133 | ``` |
| 134 | |
| 135 | Now attempt to go to <http://localhost:9090/>. You should be presented with the Prometheus homepage. This is a good |
| 136 | point to talk about Prometheus's data model which can be viewed here: <https://prometheus.io/docs/concepts/data_model/> |
| 137 | As explained we have two key elements in Prometheus metrics. We have the _metric_ and its _labels_. Labels allow for |
| 138 | granularity between metrics. Let's use our previous example to further explain. |
| 139 | |
| 140 | ```text |
| 141 | netdata_disk_space_GiB_average{chart="disk_space._run",dimension="avail",family="/run",mount_point="/run",filesystem="tmpfs",mount_root="/"} 0.0298195 1684951093000 |
| 142 | ``` |
| 143 | |
| 144 | Here our metric is `netdata_disk_space_GiB_average` and our common labels are `chart`, `family`, and `dimension`. The |
| 145 | last two values constitute the actual metric value for the metric type (gauge, counter, etc…). We also have specific |
| 146 | label for this chart named `mount_point`,`filesystem`, and `mount_root`. We can begin graphing system metrics with this information, |
| 147 | but first we need to hook up Prometheus to poll Netdata stats. |
| 148 | |
| 149 | Let's move our attention to Prometheus's configuration. Prometheus gets it config from the file located (in our example) |
| 150 | at `/opt/prometheus/prometheus.yml`. I won't spend an extensive amount of time going over the configuration values |
| 151 | documented here: <https://prometheus.io/docs/operating/configuration/>. We will be adding a new job under the |
| 152 | `scrape_configs`. Let's make the `scrape_configs` section look like this (we can use the DNS name Netdata due to the |
| 153 | custom user-defined network we created in docker beforehand). |
| 154 | |
| 155 | ```yaml |
| 156 | scrape_configs: |
| 157 | # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. |
| 158 | - job_name: 'prometheus' |
| 159 | |
| 160 | # metrics_path defaults to '/metrics' |
| 161 | # scheme defaults to 'http'. |
| 162 | |
| 163 | static_configs: |
| 164 | - targets: ['localhost:9090'] |
| 165 | |
| 166 | - job_name: 'netdata' |
| 167 | |
| 168 | metrics_path: /api/v1/allmetrics |
| 169 | params: |
| 170 | format: [ prometheus ] |
| 171 | |
| 172 | static_configs: |
| 173 | - targets: ['netdata:19999'] |
| 174 | ``` |
| 175 | |
| 176 | Let's start Prometheus once again by running `/opt/prometheus/prometheus`. If we now navigate to Prometheus at |
| 177 | <http://localhost:9090/targets> we should see our target being successfully scraped. If we now go back to the |
| 178 | Prometheus's homepage and begin to type `netdata\_` Prometheus should auto complete metrics it is now scraping. |
| 179 | |
| 180 |  |
| 181 | |
| 182 | Let's now start exploring how we can graph some metrics. Back in our Netdata container lets get the CPU spinning with a |
| 183 | pointless busy loop. On the shell do the following: |
| 184 | |
| 185 | ```sh |
| 186 | [root@netdata /]# while true; do echo "HOT HOT HOT CPU"; done |
| 187 | ``` |
| 188 | |
| 189 | Our Netdata cpu graph should be showing some activity. Let's represent this in Prometheus. In order to do this let's |
| 190 | keep our metrics page open for reference: <http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes>. We are |
| 191 | setting out to graph the data in the CPU chart so let's search for `system.cpu` in the metrics page above. We come |
| 192 | across a section of metrics with the first comments `# COMMENT homogeneous chart "system.cpu", context "system.cpu", |
| 193 | family "cpu", units "percentage"` followed by the metrics. This is a good start now let us drill down to the specific |
| 194 | metric we would like to graph. |
| 195 | |
| 196 | ```text |
| 197 | # COMMENT |
| 198 | netdata_system_cpu_percentage_average: dimension "system", value is percentage, gauge, dt 1501275951 to 1501275951 inclusive |
| 199 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 0.0000000 1501275951000 |
| 200 | ``` |
| 201 | |
| 202 | Here we learn that the metric name we care about is `netdata_system_cpu_percentage_average` so throw this into |
| 203 | Prometheus and see what we get. We should see something similar to this (I shut off my busy loop) |
| 204 | |
| 205 |  |
| 206 | |
| 207 | This is a good step toward what we want. Also make note that Prometheus will tag on an `instance` label for us which |
| 208 | corresponds to our statically defined job in the configuration file. This allows us to tailor our queries to specific |
| 209 | instances. Now we need to isolate the dimension we want in our query. To do this let us refine the query slightly. Let's |
| 210 | query the dimension also. Place this into our query text box. |
| 211 | `netdata_system_cpu_percentage_average{dimension="system"}` We now wind up with the following graph. |
| 212 | |
| 213 |  |
| 214 | |
| 215 | Awesome, this is exactly what we wanted. If you haven't caught on yet we can emulate entire charts from Netdata by using |
| 216 | the `chart` dimension. If you'd like you can combine the `chart` and `instance` dimension to create per-instance charts. |
| 217 | Let's give this a try: `netdata_system_cpu_percentage_average{chart="system.cpu", instance="netdata:19999"}` |
| 218 | |
| 219 | This is the basics of using Prometheus to query Netdata. I'd advise everyone at this point to read [this |
| 220 | page](/src/exporting/prometheus/README.md#using-netdata-with-prometheus). The key point here is that Netdata can export metrics from |
| 221 | its internal DB or can send metrics _as-collected_ by specifying the `source=as-collected` URL parameter like so. |
| 222 | <http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes&types=yes&source=as-collected> If you choose to use |
| 223 | this method you will need to use Prometheus's set of functions here: <https://prometheus.io/docs/querying/functions/> to |
| 224 | obtain useful metrics as you are now dealing with raw counters from the system. For example you will have to use the |
| 225 | `irate()` function over a counter to get that metric's rate per second. If your graphing needs are met by using the |
| 226 | metrics returned by Netdata's internal database (not specifying any source= URL parameter) then use that. If you find |
| 227 | limitations then consider re-writing your queries using the raw data and using Prometheus functions to get the desired |
| 228 | chart. |
| 229 | |
| 230 | ## Grafana |
| 231 | |
| 232 | Finally we make it to grafana. This is the easiest part in my opinion. This time we will actually run the official |
| 233 | grafana docker container as all configuration we need to do is done via the GUI. Let's run the following command: |
| 234 | |
| 235 | ```sh |
| 236 | docker run -i -p 3000:3000 --network=netdata-tutorial grafana/grafana |
| 237 | ``` |
| 238 | |
| 239 | This will get grafana running at <http://localhost:3000/>. Let's go there and |
| 240 | login using the credentials Admin:Admin. |
| 241 | |
| 242 | The first thing we want to do is click "Add data source". Let's make it look like the following screenshot |
| 243 | |
| 244 |  |
| 245 | |
| 246 | With this completed let's graph! Create a new Dashboard by clicking on the top left Grafana Icon and create a new graph |
| 247 | in that dashboard. Fill in the query like we did above and save. |
| 248 | |
| 249 |  |
| 250 | |
| 251 | ## Conclusion |
| 252 | |
| 253 | There you have it, a complete systems monitoring stack which is very easy to deploy. From here I would begin to |
| 254 | understand how Prometheus and a service discovery mechanism such as Consul can play together nicely. My current prod |
| 255 | deployments automatically register Netdata services into Consul and Prometheus automatically begins to scrape them. Once |
| 256 | achieved you do not have to think about the monitoring system until Prometheus cannot keep up with your scale. Once this |
| 257 | happens there are options presented in the Prometheus documentation for solving this. Hope this was helpful, happy |
| 258 | monitoring. |