@cryptotaxi247 / netdata-1 / commits / 398fdc44a

Prometheus version update (#6652)

* fix prometheus version download link * fix prometheus version and few lint warnings * remove specific version of prometheus * remove prometheus versions and some minor fixes

Promise Akpan committed Aug 14, 2019 at 16:04 UTC 398fdc44a66d21fee7e1b13a4cd63a21d6fdfd7e

2 files changed +62 -59

backends/WALKTHROUGH.md

+35 -35

@@ -47,14 +47,14 @@ before we do this we want name resolution between the two containers to work.

47 In order to accomplish this we will create a user-defined network and attach

48 both containers to this network. The first command we should run is:

50 -```

50 +```sh

51 docker network create --driver bridge netdata-tutorial

52 ```

54 With this user-defined network created we can now launch our container we will

55 install Netdata on and point it to this network.

57 -```

57 +```sh

58 docker run -it --name netdata --hostname netdata --network=netdata-tutorial -p 19999:19999 centos:latest '/bin/bash'

59 ```

@@ -72,19 +72,19 @@ several one-liners to install Netdata. I have not had any issues with these one

72 liners and their bootstrapping scripts so far (If you guys run into anything do

73 share). Run the following command in your container.

75 -```

75 +```sh

76 bash <(curl -Ss https://my-netdata.io/kickstart.sh) --dont-wait

77 ```

79 After the install completes you should be able to hit the Netdata dashboard at

80 -http://localhost:19999/ (replace localhost if you’re doing this on a VM or have

80 +<http://localhost:19999/> (replace localhost if you’re doing this on a VM or have

81 the docker container hosted on a machine not on your local system). If this is

82 your first time using Netdata I suggest you take a look around. The amount of

83 time I’ve spent digging through /proc and calculating my own metrics has been

84 greatly reduced by this tool. Take it all in.

86 Next I want to draw your attention to a particular endpoint. Navigate to

87 -http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes In your

87 +<http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes> In your

88 browser. This is the endpoint which publishes all the metrics in a format which

89 Prometheus understands. Let’s take a look at one of these metrics.

90 `netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"}

@@ -107,43 +107,43 @@ the install process and setup on a fresh container. This will allow anyone

107 reading to migrate this tutorial to a VM or Server of any sort.

108

109 Let’s start another container in the same fashion as we did the Netdata

110 -container. `docker run -it --name prometheus --hostname prometheus

111 ---network=netdata-tutorial -p 9090:9090 centos:latest '/bin/bash'` This should

112 -drop you into a shell once again. Once there quickly install your favorite

113 -editor as we will be editing files later in this tutorial. `yum install vim -y`

110 +container.

111 +

112 +```sh

113 +docker run -it --name prometheus --hostname prometheus

114 +--network=netdata-tutorial -p 9090:9090 centos:latest '/bin/bash'

115 +```

116

115 -Prometheus provides a tarball of their latest stable versions here:

116 -https://prometheus.io/download/. Let’s download the latest version and install

117 -into your container.

117 +This should drop you into a shell once again. Once there quickly install your favorite editor as we will be editing files later in this tutorial.

118

119 +```sh

120 +yum install vim -y

121 ```

120 -curl -L 'https://github.com/prometheus/prometheus/releases/download/v1.7.1/prometheus-1.7.1.linux-amd64.tar.gz' -o /tmp/prometheus.tar.gz

122 +

123 +Prometheus provides a tarball of their latest stable versions [here](https://prometheus.io/download/).

124 +

125 +Let’s download the latest version and install into your container.

126 +

127 +```sh

128 +cd /tmp && curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \

129 +| grep "browser_download_url.*linux-amd64.tar.gz" \

130 +| cut -d '"' -f 4 \

131 +| wget -qi -

132

133 mkdir /opt/prometheus

134

124 -tar -xf /tmp/prometheus.tar.gz -C /opt/prometheus/ --strip-components 1

135 +sudo tar -xvf /tmp/prometheus-*linux-amd64.tar.gz -C /opt/prometheus --strip=1

136 ```

137

127 -This should get prometheus installed into the container. Let’s test that we can run

128 -prometheus and connect to it’s web interface. This will look similar to what

129 -follows:

138 +This should get prometheus installed into the container. Let’s test that we can run prometheus and connect to it’s web interface.

139

131 -```

132 -[root@prometheus prometheus]# /opt/prometheus/prometheus

133 -INFO[0000] Starting prometheus (version=1.7.1, branch=master, revision=3afb3fffa3a29c3de865e1172fb740442e9d0133)

134 - source="main.go:88"

135 -INFO[0000] Build context (go=go1.8.3, user=root@0aa1b7fc430d, date=20170612-11:44:05) source="main.go:89"

136 -INFO[0000] Host details (Linux 4.9.36-moby #1 SMP Wed Jul 12 15:29:07 UTC 2017 x86_64 prometheus (none)) source="main.go:90"

137 -INFO[0000] Loading configuration file prometheus.yml source="main.go:252"

138 -INFO[0000] Loading series map and head chunks... source="storage.go:428"

139 -INFO[0000] 0 series loaded. source="storage.go:439"

140 -INFO[0000] Starting target manager... source="targetmanager.go:63"

141 -INFO[0000] Listening on :9090 source="web.go:259"

140 +```sh

141 +/opt/prometheus/prometheus

142 ```

143

144 -Now attempt to go to http://localhost:9090/. You should be presented with the

144 +Now attempt to go to <http://localhost:9090/>. You should be presented with the

145 prometheus homepage. This is a good point to talk about Prometheus’s data model

146 -which can be viewed here: https://prometheus.io/docs/concepts/data_model/ As

146 +which can be viewed here: <https://prometheus.io/docs/concepts/data_model/> As

147 explained we have two key elements in Prometheus metrics. We have the ‘metric’

148 and its ‘labels’. Labels allow for granularity between metrics. Let’s use our

149 previous example to further explain.

@@ -162,7 +162,7 @@ Let’s move our attention to Prometheus’s configuration. Prometheus gets it

162 config from the file located (in our example) at

163 `/opt/prometheus/prometheus.yml`. I won’t spend an extensive amount of time

164 going over the configuration values documented here:

165 -https://prometheus.io/docs/operating/configuration/. We will be adding a new

165 +<https://prometheus.io/docs/operating/configuration/>. We will be adding a new

166 “job” under the “scrape_configs”. Let’s make the “scrape_configs” section look

167 like this (we can use the dns name Netdata due to the custom user-defined

168 network we created in docker beforehand).

@@ -189,7 +189,7 @@ scrape_configs:

189 ```

190

191 Let’s start prometheus once again by running `/opt/prometheus/prometheus`. If we

192 -now navigate to prometheus at ‘http://localhost:9090/targets’ we should see our

192 +now navigate to prometheus at ‘<http://localhost:9090/targets>’ we should see our

193 target being successfully scraped. If we now go back to the Prometheus’s

194 homepage and begin to type ‘netdata_’ Prometheus should auto complete metrics

195 it is now scraping.

@@ -206,7 +206,7 @@ the following:

206

207 Our NetData cpu graph should be showing some activity. Let’s represent this in

208 Prometheus. In order to do this let’s keep our metrics page open for reference:

209 -http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes We are

209 +<http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes> We are

210 setting out to graph the data in the CPU chart so let’s search for “system.cpu”

211 in the metrics page above. We come across a section of metrics with the first

212 comments `# COMMENT homogeneous chart "system.cpu", context "system.cpu", family

@@ -249,7 +249,7 @@ can send metrics “as-collected” by specifying the ‘source=as-collected’

249 parameter like so.

250 http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes&types=yes&source=as-collected

251 If you choose to use this method you will need to use Prometheus's set of

252 -functions here: https://prometheus.io/docs/querying/functions/ to obtain useful

252 +functions here: <https://prometheus.io/docs/querying/functions/> to obtain useful

253 metrics as you are now dealing with raw counters from the system. For example

254 you will have to use the `irate()` function over a counter to get that metric's

255 rate per second. If your graphing needs are met by using the metrics returned by

@@ -266,7 +266,7 @@ we need to do is done via the GUI. Let’s run the following command:

266 docker run -i -p 3000:3000 --network=netdata-tutorial grafana/grafana

267 ```

268

269 -This will get grafana running at ‘http://localhost:3000/’ Let’s go there and

269 +This will get grafana running at ‘<http://localhost:3000/>’ Let’s go there and

270 login using the credentials Admin:Admin.

271

272 The first thing we want to do is click ‘Add data source’. Let’s make it look

backends/prometheus/README.md

+27 -24

@@ -4,7 +4,6 @@

5 Prometheus is a distributed monitoring system which offers a very simple setup along with a robust data model. Recently Netdata added support for Prometheus. I'm going to quickly show you how to install both Netdata and prometheus on the same server. We can then use grafana pointed at Prometheus to obtain long term metrics Netdata offers. I'm assuming we are starting at a fresh ubuntu shell (whether you'd like to follow along in a VM or a cloud instance is up to you).

7 -

7 ## Installing Netdata and prometheus

9 ### Installing Netdata

@@ -12,7 +11,7 @@ Prometheus is a distributed monitoring system which offers a very simple setup a

11 There are number of ways to install Netdata according to [Installation](../../packaging/installer/#installation)

12 The suggested way of installing the latest Netdata and keep it upgrade automatically. Using one line installation:

15 -```

14 +```sh

15 bash <(curl -Ss https://my-netdata.io/kickstart.sh)

16 ```

@@ -22,7 +21,7 @@ At this point we should have Netdata listening on port 19999. Attempt to take yo

21 http://your.netdata.ip:19999

22 ```

25 -*(replace `your.netdata.ip` with the IP or hostname of the server running Netdata)*

24 +_(replace `your.netdata.ip` with the IP or hostname of the server running Netdata)_

26 ### Installing Prometheus

@@ -31,7 +30,10 @@ In order to install prometheus we are going to introduce our own systemd startup

30 #### Download Prometheus

32 ```sh

34 -wget -O /tmp/prometheus-2.3.2.linux-amd64.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz

33 +cd /tmp && curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \

34 +| grep "browser_download_url.*linux-amd64.tar.gz" \

35 +| cut -d '"' -f 4 \

36 +| wget -qi -

37 ```

39 #### Create prometheus system user

@@ -50,7 +52,7 @@ sudo chown prometheus:prometheus /opt/prometheus

52 #### Untar prometheus directory

54 ```sh

53 -sudo tar -xvf /tmp/prometheus-2.3.2.linux-amd64.tar.gz -C /opt/prometheus --strip=1

55 +sudo tar -xvf /tmp/prometheus-*linux-amd64.tar.gz -C /opt/prometheus --strip=1

56 ```

58 #### Install prometheus.yml

@@ -111,8 +113,8 @@ scrape_configs:

113

114 #### Install nodes.yml

115

114 -The following is completely optional, it will enable Prometheus to generate alerts from some NetData sources. Tweak the values to your own needs. We will use the following `nodes.yml` file below. Save it at `/opt/prometheus/nodes.yml`, and add a *- "nodes.yml"* entry under the *rule_files:* section in the example prometheus.yml file above.

115 -```

116 +The following is completely optional, it will enable Prometheus to generate alerts from some NetData sources. Tweak the values to your own needs. We will use the following `nodes.yml` file below. Save it at `/opt/prometheus/nodes.yml`, and add a _- "nodes.yml"_ entry under the _rule_files:_ section in the example prometheus.yml file above.

117 +```yaml

118 groups:

119 - name: nodes

120

@@ -173,7 +175,7 @@ WantedBy=multi-user.target

175 ```

176 ##### Start Prometheus

177

176 -```

178 +```sh

179 sudo systemctl start prometheus

180 sudo systemctl enable prometheus

181 ```

@@ -192,21 +194,21 @@ Before explaining the changes, we have to understand the key differences between

194

195 ### understanding Netdata metrics

196

195 -##### charts

197 +#### charts

198

199 Each chart in Netdata has several properties (common to all its metrics):

200

199 -- `chart_id` - uniquely identifies a chart.

201 +- `chart_id` - uniquely identifies a chart.

202

201 -- `chart_name` - a more human friendly name for `chart_id`, also unique.

203 +- `chart_name` - a more human friendly name for `chart_id`, also unique.

204

203 -- `context` - this is the template of the chart. All disk I/O charts have the same context, all mysql requests charts have the same context, etc. This is used for alarm templates to match all the charts they should be attached to.

205 +- `context` - this is the template of the chart. All disk I/O charts have the same context, all mysql requests charts have the same context, etc. This is used for alarm templates to match all the charts they should be attached to.

206

205 -- `family` groups a set of charts together. It is used as the submenu of the dashboard.

207 +- `family` groups a set of charts together. It is used as the submenu of the dashboard.

208

207 -- `units` is the units for all the metrics attached to the chart.

209 +- `units` is the units for all the metrics attached to the chart.

210

209 -##### dimensions

211 +#### dimensions

212

213 Then each Netdata chart contains metrics called `dimensions`. All the dimensions of a chart have the same units of measurement, and are contextually in the same category (ie. the metrics for disk bandwidth are `read` and `write` and they are both in the same chart).

214

@@ -214,7 +216,7 @@ Then each Netdata chart contains metrics called `dimensions`. All the dimensions

216

217 Netdata can send metrics to prometheus from 3 data sources:

218

217 -- `as collected` or `raw` - this data source sends the metrics to prometheus as they are collected. No conversion is done by Netdata. The latest value for each metric is just given to prometheus. This is the most preferred method by prometheus, but it is also the harder to work with. To work with this data source, you will need to understand how to get meaningful values out of them.

219 +- `as collected` or `raw` - this data source sends the metrics to prometheus as they are collected. No conversion is done by Netdata. The latest value for each metric is just given to prometheus. This is the most preferred method by prometheus, but it is also the harder to work with. To work with this data source, you will need to understand how to get meaningful values out of them.

220

221 The format of the metrics is: `CONTEXT{chart="CHART",family="FAMILY",dimension="DIMENSION"}`.

222

@@ -222,13 +224,13 @@ Netdata can send metrics to prometheus from 3 data sources:

224

225 Unlike prometheus, Netdata allows each dimension of a chart to have a different algorithm and conversion constants (`multiplier` and `divisor`). In this case, that the dimensions of a charts are heterogeneous, Netdata will use this format: `CONTEXT_DIMENSION{chart="CHART",family="FAMILY"}`

226

225 -- `average` - this data source uses the Netdata database to send the metrics to prometheus as they are presented on the Netdata dashboard. So, all the metrics are sent as gauges, at the units they are presented in the Netdata dashboard charts. This is the easiest to work with.

227 +- `average` - this data source uses the Netdata database to send the metrics to prometheus as they are presented on the Netdata dashboard. So, all the metrics are sent as gauges, at the units they are presented in the Netdata dashboard charts. This is the easiest to work with.

228

229 The format of the metrics is: `CONTEXT_UNITS_average{chart="CHART",family="FAMILY",dimension="DIMENSION"}`.

230

231 When this source is used, Netdata keeps track of the last access time for each prometheus server fetching the metrics. This last access time is used at the subsequent queries of the same prometheus server to identify the time-frame the `average` will be calculated. So, no matter how frequently prometheus scrapes Netdata, it will get all the database data. To identify each prometheus server, Netdata uses by default the IP of the client fetching the metrics. If there are multiple prometheus servers fetching data from the same Netdata, using the same IP, each prometheus server can append `server=NAME` to the URL. Netdata will use this `NAME` to uniquely identify the prometheus server.

232

231 -- `sum` or `volume`, is like `average` but instead of averaging the values, it sums them.

233 +- `sum` or `volume`, is like `average` but instead of averaging the values, it sums them.

234

235 The format of the metrics is: `CONTEXT_UNITS_sum{chart="CHART",family="FAMILY",dimension="DIMENSION"}`.

236 All the other operations are the same with `average`.

@@ -241,7 +243,7 @@ Fetch with your web browser this URL:

243

244 `http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes`

245

244 -*(replace `your.netdata.ip` with the ip or hostname of your Netdata server)*

246 +_(replace `your.netdata.ip` with the ip or hostname of your Netdata server)_

247

248 Netdata will respond with all the metrics it sends to prometheus.

249

@@ -272,7 +274,8 @@ netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension=

274 # COMMENT netdata_system_cpu_percentage_average: dimension "idle", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive

275 netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="idle"} 92.3630770 1500066662000

276 ```

275 -*(Netdata response for `system.cpu` with source=`average`)*

277 +

278 +_(Netdata response for `system.cpu` with source=`average`)_

279

280 In `average` or `sum` data sources, all values are normalized and are reported to prometheus as gauges. Now, use the 'expression' text form in prometheus. Begin to type the metrics we are looking for: `netdata_system_cpu`. You should see that the text form begins to auto-fill as prometheus knows about this metric.

281

@@ -302,7 +305,7 @@ netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="iowait"} 233

305 netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="idle"} 918470 1500066716438

306 ```

307

305 -*(Netdata response for `system.cpu` with source=`as-collected`)*

308 +_(Netdata response for `system.cpu` with source=`as-collected`)_

309

310 For more information check prometheus documentation.

311

@@ -310,7 +313,7 @@ For more information check prometheus documentation.

313

314 The `format=prometheus` parameter only exports the host's Netdata metrics. If you are using the master/slave functionality of Netdata this ignores any upstream hosts - so you should consider using the below in your **prometheus.yml**:

315

313 -```

316 +```yaml

317 metrics_path: '/api/v1/allmetrics'

318 params:

319 format: [prometheus_all_hosts]

@@ -348,8 +351,8 @@ The default is controlled in `netdata.conf`:

351

352 You can overwrite it from prometheus, by appending to the URL:

353

351 -* `&names=no` to get IDs (the old behaviour)

352 -* `&names=yes` to get names

354 +- `&names=no` to get IDs (the old behaviour)

355 +- `&names=yes` to get names

356

357 ### Filtering metrics sent to prometheus

358