Update the exporting documentation (#9066)
Vladimir Kobal committed
May 18, 2020 at 19:03 UTC
06ffc8f9f5c3f71aa39b4cef516773bca736859d
9 files changed
+744
-37
exporting/Makefile.am
+6
@@ -14,6 +14,12 @@ SUBDIRS = \
14
mongodb \
15
$(NULL)
16
17
+dist_libconfig_DATA = \
18
+ exporting.conf \
19
+ $(NULL)
20
+
21
dist_noinst_DATA = \
22
README.md \
23
+ TIMESCALE.md \
24
+ WALKTHROUGH.md \
25
$(NULL)
exporting/README.md
+260
-30
@@ -4,48 +4,278 @@ description: "With the exporting engine, you can archive your Netdata metrics to
4
custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/README.md
5
-->
6
7
-# Exporting metrics to external databases (experimental)
7
+# Exporting metrics to external databases
8
9
-The exporting engine is an update for the former [backends](/backends/README.md). It's still work in progress. It has a
10
-modular structure and supports metric exporting via multiple exporting connector instances at the same time. You can
11
-have different update intervals and filters configured for every exporting connector instance. The exporting engine has
12
-its own configuration file `exporting.conf`. Configuration is almost similar to [backends](/backends/README.md#configuration).
13
-The only difference is that the type of a connector should be specified in a section name before a colon and a name after
14
-the colon. At the moment only four types of connectors are supported: `graphite`, `json`, `opentsdb`, `opentsdb:http`.
9
+The exporting engine is an update for the former [backends](/backends/README.md) which is deprecated and will be deleted
10
+soon. It has a modular structure and supports metric exporting via multiple exporting connector instances at the same
11
+time. You can have different update intervals and filters configured for every exporting connector instance. The
12
+exporting engine has its own configuration file `exporting.conf`. Configuration is almost similar to
13
+[backends](/backends/README.md#configuration). The most important difference is that type of a connector should be
14
+specified in a section name before a colon and an instance name after the colon. Also, you can't use `host tags`
15
+anymore. Set your labels using the [`[host labels]`](/docs/tutorials/using-host-labels.md) section in `netdata.conf`.
16
+
17
+# Metrics long term archiving
18
+
19
+Netdata supports external databases and services for archiving the metrics, or providing long term dashboards, using
20
+Grafana or other tools, like this:
21
+
22
+
23
+
24
+Since Netdata collects thousands of metrics per server per second, which would easily congest any database server when
25
+several Netdata servers are sending data to it, Netdata allows sending metrics at a lower frequency, by resampling them.
26
+
27
+So, although Netdata collects metrics every second, it can send to the external database servers averages or sums every
28
+X seconds (though, it can send them per second if you need it to).
29
+
30
+## features
31
+
32
+1. Supported databases and services
33
+
34
+ - **graphite** (`plaintext interface`, used by **Graphite**, **InfluxDB**, **KairosDB**, **Blueflood**,
35
+ **ElasticSearch** via logstash tcp input and the graphite codec, etc)
36
+
37
+ metrics are sent to the database server as `prefix.hostname.chart.dimension`. `prefix` is configured below,
38
+ `hostname` is the hostname of the machine (can also be configured).
39
+
40
+ - **opentsdb** (`telnet or HTTP interfaces`, used by **OpenTSDB**, **InfluxDB**, **KairosDB**, etc)
41
+
42
+ metrics are sent to OpenTSDB as `prefix.chart.dimension` with tag `host=hostname`.
43
+
44
+ - **json** document DBs
45
+
46
+ metrics are sent to a document DB, `JSON` formatted.
47
+
48
+ - **prometheus** is described at [prometheus page](/exporting/prometheus/README.md) since it pulls data from
49
+ Netdata.
50
+
51
+ - **prometheus remote write** (a binary snappy-compressed protocol buffer encoding over HTTP used by
52
+ **Elasticsearch**, **Gnocchi**, **Graphite**, **InfluxDB**, **Kafka**, **OpenTSDB**, **PostgreSQL/TimescaleDB**,
53
+ **Splunk**, **VictoriaMetrics**, and a lot of other [storage
54
+ providers](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage))
55
+
56
+ metrics are labeled in the format, which is used by Netdata for the [plaintext prometheus
57
+ protocol](/exporting/prometheus/README.md). Notes on using the remote write connector are
58
+ [here](/exporting/prometheus/remote_write/README.md).
59
+
60
+ - **TimescaleDB** via [community-built connector](/exporting/TIMESCALE.md) that takes JSON streams from a Netdata
61
+ client and writes them to a TimescaleDB table.
62
+
63
+ - **AWS Kinesis Data Streams**
64
+
65
+ metrics are sent to the service in `JSON` format.
66
+
67
+ - **Google Cloud Pub/Sub Service**
68
+
69
+ metrics are sent to the service in `JSON` format.
70
+
71
+ - **MongoDB**
72
+
73
+ metrics are sent to the database in `JSON` format.
74
+
75
+2. Netdata can filter metrics (at the chart level), to send only a subset of the collected metrics.
76
+
77
+3. Netdata supports three modes of operation for all exporting connectors:
78
+
79
+ - `as-collected` sends to external databases the metrics as they are collected, in the units they are collected.
80
+ So, counters are sent as counters and gauges are sent as gauges, much like all data collectors do. For example,
81
+ to calculate CPU utilization in this format, you need to know how to convert kernel ticks to percentage.
82
+
83
+ - `average` sends to external databases normalized metrics from the Netdata database. In this mode, all metrics
84
+ are sent as gauges, in the units Netdata uses. This abstracts data collection and simplifies visualization, but
85
+ you will not be able to copy and paste queries from other sources to convert units. For example, CPU utilization
86
+ percentage is calculated by Netdata, so Netdata will convert ticks to percentage and send the average percentage
87
+ to the external database.
88
+
89
+ - `sum` or `volume`: the sum of the interpolated values shown on the Netdata graphs is sent to the external
90
+ database. So, if Netdata is configured to send data to the database every 10 seconds, the sum of the 10 values
91
+ shown on the Netdata charts will be used.
92
+
93
+ Time-series databases suggest to collect the raw values (`as-collected`). If you plan to invest on building your
94
+ monitoring around a time-series database and you already know (or you will invest in learning) how to convert units
95
+ and normalize the metrics in Grafana or other visualization tools, we suggest to use `as-collected`.
96
+
97
+ If, on the other hand, you just need long term archiving of Netdata metrics and you plan to mainly work with
98
+ Netdata, we suggest to use `average`. It decouples visualization from data collection, so it will generally be a lot
99
+ simpler. Furthermore, if you use `average`, the charts shown in the external service will match exactly what you
100
+ see in Netdata, which is not necessarily true for the other modes of operation.
101
+
102
+4. This code is smart enough, not to slow down Netdata, independently of the speed of the external database server. You
103
+ should keep in mind though that many exporting connector instances can consume a lot of CPU resources if they run
104
+ their batches at the same time. You can set different update intervals for every exporting connector instance, but
105
+ even in that case they can occasionally synchronize their batches for a moment.
106
+
107
+## configuration
108
+
109
+In `/etc/netdata/exporting.conf` you should have something like this:
110
16
-An example configuration:
111
```conf
112
[exporting:global]
19
-enabled = yes
113
+ enabled = yes
114
+ send configured labels = no
115
+ send automatic labels = no
116
+ update every = 10
117
+
118
+[prometheus:exporter]
119
+ send charts matching = system.processes
120
21
-[graphite:my_instance1]
22
-enabled = yes
23
-destination = localhost:2003
24
-data source = sum
25
-update every = 5
26
-send charts matching = system.load
121
+[graphite:my_instance_1]
122
+ enabled = yes
123
+ destination = localhost:2003
124
+ data source = average
125
+ prefix = Netdata
126
+ hostname = my-name
127
+ update every = 10
128
+ buffer on failures = 10
129
+ timeout ms = 20000
130
+ send charts matching = *
131
+ send hosts matching = localhost *
132
+ send names instead of ids = yes
133
134
[json:my_instance2]
29
-enabled = yes
30
-destination = localhost:5448
31
-data source = as collected
32
-update every = 2
33
-send charts matching = system.active_processes
135
+ enabled = yes
136
+ destination = localhost:5448
137
+ data source = as collected
138
+ update every = 2
139
+ send charts matching = system.active_processes
140
141
[opentsdb:my_instance3]
36
-enabled = yes
37
-destination = localhost:4242
38
-data source = sum
39
-update every = 10
40
-send charts matching = system.cpu
142
+ enabled = yes
143
+ destination = localhost:4242
144
+ data source = sum
145
+ update every = 10
146
+ send charts matching = system.cpu
147
148
[opentsdb:http:my_instance4]
43
-enabled = yes
44
-destination = localhost:4243
45
-data source = average
46
-update every = 3
47
-send charts matching = system.active_processes
149
+ enabled = yes
150
+ destination = localhost:4243
151
+ data source = average
152
+ update every = 3
153
+ send charts matching = system.active_processes
154
+```
155
+
156
+Sections:
157
+- `[exporting:global]` is a section where you can set your defaults for all exporting connectors
158
+- `[prometheus:exporter]` defines settings for Prometheus exporter API queries (e.g.:
159
+ `http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes&source=as-collected`).
160
+- `[<type>:<name>]` keeps settings for a particular exporting connector instance, where:
161
+ - `type` selects the exporting connector type: graphite | opentsdb:telnet | opentsdb:http | opentsdb:https |
162
+ prometheus_remote_write | json | kinesis | pubsub | mongodb
163
+ - `name` can be arbitrary instance name you chose.
164
+
165
+Options:
166
+- `enabled = yes | no`, enables or disables an exporting connector instance
167
+
168
+- `destination = host1 host2 host3 ...`, accepts **a space separated list** of hostnames, IPs (IPv4 and IPv6) and
169
+ ports to connect to. Netdata will use the **first available** to send the metrics.
170
171
+ The format of each item in this list, is: `[PROTOCOL:]IP[:PORT]`.
172
+
173
+ `PROTOCOL` can be `udp` or `tcp`. `tcp` is the default and only supported by the current exporting engine.
174
+
175
+ `IP` can be `XX.XX.XX.XX` (IPv4), or `[XX:XX...XX:XX]` (IPv6). For IPv6 you can to enclose the IP in `[]` to
176
+ separate it from the port.
177
+
178
+ `PORT` can be a number of a service name. If omitted, the default port for the exporting connector will be used
179
+ (graphite = 2003, opentsdb = 4242).
180
+
181
+ Example IPv4:
182
+
183
+```conf
184
+ destination = 10.11.14.2:4242 10.11.14.3:4242 10.11.14.4:4242
185
+```
186
+
187
+ Example IPv6 and IPv4 together:
188
+
189
+```conf
190
+ destination = [ffff:...:0001]:2003 10.11.12.1:2003
191
```
192
193
+ When multiple servers are defined, Netdata will try the next one when the previous one fails.
194
+
195
+ Netdata also ships `nc-exporting.sh`, a script that can be used as a fallback exporting connector to save the
196
+ metrics to disk and push them to the time-series database when it becomes available again. It can also be used to
197
+ monitor / trace / debug the metrics Netdata generates.
198
+
199
+ For the Kinesis exporting connector `destination` should be set to an AWS region (for example, `us-east-1`).
200
+
201
+ For the MongoDB exporting connector `destination` should be set to a
202
+ [MongoDB URI](https://docs.mongodb.com/manual/reference/connection-string/).
203
+
204
+ For the Pub/Sub exporting connector `destination` can be set to a specific service endpoint.
205
+
206
+- `data source = as collected`, or `data source = average`, or `data source = sum`, selects the kind of data that will
207
+ be sent to the external database.
208
+
209
+- `hostname = my-name`, is the hostname to be used for sending data to the external database server. By default this
210
+ is `[global].hostname`.
211
+
212
+- `prefix = Netdata`, is the prefix to add to all metrics.
213
+
214
+- `update every = 10`, is the number of seconds between sending data to the external datanase. Netdata will add some
215
+ randomness to this number, to prevent stressing the external server when many Netdata servers send data to the same
216
+ database. This randomness does not affect the quality of the data, only the time they are sent.
217
+
218
+- `buffer on failures = 10`, is the number of iterations (each iteration is `update every` seconds) to buffer data,
219
+ when the external database server is not available. If the server fails to receive the data after that many
220
+ failures, data loss on the connector instance is expected (Netdata will also log it).
221
+
222
+- `timeout ms = 20000`, is the timeout in milliseconds to wait for the external database server to process the data.
223
+ By default this is `2 * update_every * 1000`.
224
+
225
+- `send hosts matching = localhost *` includes one or more space separated patterns, using `*` as wildcard (any number
226
+ of times within each pattern). The patterns are checked against the hostname (the localhost is always checked as
227
+ `localhost`), allowing us to filter which hosts will be sent to the external database when this Netdata is a central
228
+ Netdata aggregating multiple hosts. A pattern starting with `!` gives a negative match. So to match all hosts named
229
+ `*db*` except hosts containing `*slave*`, use `!*slave* *db*` (so, the order is important: the first pattern
230
+ matching the hostname will be used - positive or negative).
231
+
232
+- `send charts matching = *` includes one or more space separated patterns, using `*` as wildcard (any number of times
233
+ within each pattern). The patterns are checked against both chart id and chart name. A pattern starting with `!`
234
+ gives a negative match. So to match all charts named `apps.*` except charts ending in `*reads`, use `!*reads
235
+ apps.*` (so, the order is important: the first pattern matching the chart id or the chart name will be used -
236
+ positive or negative).
237
+
238
+- `send names instead of ids = yes | no` controls the metric names Netdata should send to the external database.
239
+ Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system
240
+ and names are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several
241
+ cases they are different: disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc.
242
+
243
+- `send configured labels = yes | no` controls if labels defined in the `[host labels]` section in `netdata.conf`
244
+ should be sent to the external database
245
+
246
+- `send automatic labels = yes | no` controls if automatially created labels, like `_os_name` or `_architecture`
247
+ should be sent to the external database
248
+
249
+> Starting from Netdata v1.20 the host tags (defined in the `[backend]` section of `netdata.conf`) are parsed in
250
+> accordance with a configured backend type and stored as host labels so that they can be reused in API responses and
251
+> exporting connectors. The parsing is supported for graphite, json, opentsdb, and prometheus (default) backend types.
252
+> You can check how the host tags were parsed using the /api/v1/info API call. But, keep in mind that backends subsystem
253
+> is deprecated and will be deleted soon. Please move your existing tags to the `[host labels]` section.
254
+
255
+## monitoring operation
256
+
257
+Netdata provides 5 charts:
258
+
259
+1. **Buffered metrics**, the number of metrics Netdata added to the buffer for dispatching them to the
260
+ external database server.
261
+
262
+2. **Exporting data size**, the amount of data (in KB) Netdata added the buffer.
263
+
264
+3. **Exporting operations**, the number of operations performed by Netdata.
265
+
266
+4. **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending the
267
+ metrics to the external database server.
268
+
269
+
270
+
271
+## alarms
272
+
273
+Netdata adds 3 alarms:
274
+
275
+1. `exporting_last_buffering`, number of seconds since the last successful buffering of exported data
276
+2. `exporting_metrics_sent`, percentage of metrics sent to the external database server
277
+3. `exporting_metrics_lost`, number of metrics lost due to repeating failures to contact the external database server
278
+
279
+
280
+
281
[](<>)
exporting/TIMESCALE.md
new
+69
@@ -0,0 +1,69 @@
1
+<!--
2
+title: "Writing metrics to TimescaleDB"
3
+description: "Send Netdata metrics to TimescaleDB for long-term archiving and further analysis."
4
+custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/TIMESCALE.md
5
+sidebar_label: Writing metrics to TimescaleDB
6
+-->
7
+
8
+# Writing metrics to TimescaleDB
9
+
10
+Thanks to Netdata's community of developers and system administrators, and Mahlon Smith
11
+([GitHub](https://github.com/mahlonsmith)/[Website](http://www.martini.nu/)) in particular, Netdata now supports
12
+archiving metrics directly to TimescaleDB.
13
+
14
+What's TimescaleDB? Here's how their team defines the project on their [GitHub page](https://github.com/timescale/timescaledb):
15
+
16
+> TimescaleDB is an open-source database designed to make SQL scalable for time-series data. It is engineered up from
17
+> PostgreSQL, providing automatic partitioning across time and space (partitioning key), as well as full SQL support.
18
+
19
+## Quickstart
20
+
21
+To get started archiving metrics to TimescaleDB right away, check out Mahlon's [`netdata-timescale-relay`
22
+repository](https://github.com/mahlonsmith/netdata-timescale-relay) on GitHub. Please be aware that backends subsystem
23
+is deprecated and Netdata configuration should be moved to the new `exporting conf` configuration file. Use
24
+```conf
25
+[json:my_instance]
26
+```
27
+in `exporting.conf` instead of
28
+```conf
29
+[backend]
30
+ type = json
31
+```
32
+in `netdata.conf`.
33
+
34
+This small program takes JSON streams from a Netdata client and writes them to a PostgreSQL (aka TimescaleDB) table.
35
+You'll run this program in parallel with Netdata, and after a short [configuration
36
+process](https://github.com/mahlonsmith/netdata-timescale-relay#configuration), your metrics should start populating
37
+TimescaleDB.
38
+
39
+Finally, another member of Netdata's community has built a project that quickly launches Netdata, TimescaleDB, and
40
+Grafana in easy-to-manage Docker containers. Rune Juhl Jacobsen's
41
+[project](https://github.com/runejuhl/grafana-timescaledb) uses a `Makefile` to create everything, which makes it
42
+perferct for testing and experimentation.
43
+
44
+## Netdata↔TimescaleDB in action
45
+
46
+Aside from creating incredible contributions to Netdata, Mahlon works at [LAIKA](https://www.laika.com/), an
47
+Oregon-based animation studio that's helped create acclaimed films like _Coraline_ and _Kubo and the Two Strings_.
48
+
49
+As part of his work to maintain the company's infrastructure of render farms, workstations, and virtual machines, he's
50
+using Netdata, `netdata-timescale-relay`, and TimescaleDB to store Netdata metrics alongside other data from other
51
+sources.
52
+
53
+> LAIKA is a long-time PostgreSQL user and added TimescaleDB to their infrastructure in 2018 to help manage and store
54
+> their IT metrics and time-series data. So far, the tool has been in production at LAIKA for over a year and helps them
55
+> with their use case of time-based logging, where they record over 8 million metrics an hour for netdata content alone.
56
+
57
+By archiving Netdata metrics to a database like TimescaleDB, LAIKA can consolidate metrics data from distributed
58
+machines efficiently. Mahlon can then correlate Netdata metrics with other sources directly in TimescaleDB.
59
+
60
+And, because LAIKA will soon be storing years worth of Netdata metrics data in TimescaleDB, they can analyze long-term
61
+metrics as their films move from concept to final cut.
62
+
63
+Read the full blog post from LAIKA at the [TimescaleDB
64
+blog](https://blog.timescale.com/blog/writing-it-metrics-from-netdata-to-timescaledb/amp/).
65
+
66
+Thank you to Mahlon, Rune, TimescaleDB, and the members of the Netdata community that requested and then built this
67
+exporting connection between Netdata and TimescaleDB!
68
+
69
+[](<>)
exporting/WALKTHROUGH.md
new
+259
@@ -0,0 +1,259 @@
1
+<!--
2
+title: "Exporting to Netdata, Prometheus, Grafana stack"
3
+description: "Using Netdata in conjunction with Prometheus and Grafana."
4
+custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/WALKTHROUGH.md
5
+sidebar_label: Netdata, Prometheus, Grafana stack
6
+-->
7
+
8
+# Netdata, Prometheus, Grafana stack
9
+
10
+## Intro
11
+
12
+In this article I will walk you through the basics of getting Netdata, Prometheus and Grafana all working together and
13
+monitoring your application servers. This article will be using docker on your local workstation. We will be working
14
+with docker in an ad-hoc way, launching containers that run `/bin/bash` and attaching a TTY to them. I use docker here
15
+in a purely academic fashion and do not condone running Netdata in a container. I pick this method so individuals
16
+without cloud accounts or access to VMs can try this out and for it's speed of deployment.
17
+
18
+## Why Netdata, Prometheus, and Grafana
19
+
20
+Some time ago I was introduced to Netdata by a coworker. We were attempting to troubleshoot python code which seemed to
21
+be bottlenecked. I was instantly impressed by the amount of metrics Netdata exposes to you. I quickly added Netdata to
22
+my set of go-to tools when troubleshooting systems performance.
23
+
24
+Some time ago, even later, I was introduced to Prometheus. Prometheus is a monitoring application which flips the normal
25
+architecture around and polls rest endpoints for its metrics. This architectural change greatly simplifies and decreases
26
+the time necessary to begin monitoring your applications. Compared to current monitoring solutions the time spent on
27
+designing the infrastructure is greatly reduced. Running a single Prometheus server per application becomes feasible
28
+with the help of Grafana.
29
+
30
+Grafana has been the go to graphing tool for… some time now. It's awesome, anyone that has used it knows it's awesome.
31
+We can point Grafana at Prometheus and use Prometheus as a data source. This allows a pretty simple overall monitoring
32
+architecture: Install Netdata on your application servers, point Prometheus at Netdata, and then point Grafana at
33
+Prometheus.
34
+
35
+I'm omitting an important ingredient in this stack in order to keep this tutorial simple and that is service discovery.
36
+My personal preference is to use Consul. Prometheus can plug into consul and automatically begin to scrape new hosts
37
+that register a Netdata client with Consul.
38
+
39
+At the end of this tutorial you will understand how each technology fits together to create a modern monitoring stack.
40
+This stack will offer you visibility into your application and systems performance.
41
+
42
+## Getting Started - Netdata
43
+
44
+To begin let's create our container which we will install Netdata on. We need to run a container, forward the necessary
45
+port that Netdata listens on, and attach a tty so we can interact with the bash shell on the container. But before we do
46
+this we want name resolution between the two containers to work. In order to accomplish this we will create a
47
+user-defined network and attach both containers to this network. The first command we should run is:
48
+
49
+```sh
50
+docker network create --driver bridge netdata-tutorial
51
+```
52
+
53
+With this user-defined network created we can now launch our container we will install Netdata on and point it to this
54
+network.
55
+
56
+```sh
57
+docker run -it --name netdata --hostname netdata --network=netdata-tutorial -p 19999:19999 centos:latest '/bin/bash'
58
+```
59
+
60
+This command creates an interactive tty session (`-it`), gives the container both a name in relation to the docker
61
+daemon and a hostname (this is so you know what container is which when working in the shells and docker maps hostname
62
+resolution to this container), forwards the local port 19999 to the container's port 19999 (`-p 19999:19999`), sets the
63
+command to run (`/bin/bash`) and then chooses the base container images (`centos:latest`). After running this you should
64
+be sitting inside the shell of the container.
65
+
66
+After we have entered the shell we can install Netdata. This process could not be easier. If you take a look at [this
67
+link](/packaging/installer/README.md), the Netdata devs give us several one-liners to install Netdata. I have not had
68
+any issues with these one liners and their bootstrapping scripts so far (If you guys run into anything do share). Run
69
+the following command in your container.
70
+
71
+```sh
72
+bash <(curl -Ss https://my-netdata.io/kickstart.sh) --dont-wait
73
+```
74
+
75
+After the install completes you should be able to hit the Netdata dashboard at <http://localhost:19999/> (replace
76
+localhost if you're doing this on a VM or have the docker container hosted on a machine not on your local system). If
77
+this is your first time using Netdata I suggest you take a look around. The amount of time I've spent digging through
78
+`/proc` and calculating my own metrics has been greatly reduced by this tool. Take it all in.
79
+
80
+Next I want to draw your attention to a particular endpoint. Navigate to
81
+<http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes> In your browser. This is the endpoint which
82
+publishes all the metrics in a format which Prometheus understands. Let's take a look at one of these metrics.
83
+`netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 0.0831255 1501271696000` This
84
+metric is representing several things which I will go in more details in the section on Prometheus. For now understand
85
+that this metric: `netdata_system_cpu_percentage_average` has several labels: (`chart`, `family`, `dimension`). This
86
+corresponds with the first cpu chart you see on the Netdata dashboard.
87
+
88
+
89
+
90
+This CHART is called `system.cpu`, The FAMILY is `cpu`, and the DIMENSION we are observing is `system`. You can begin to
91
+draw links between the charts in Netdata to the Prometheus metrics format in this manner.
92
+
93
+## Prometheus
94
+
95
+We will be installing Prometheus in a container for purpose of demonstration. While Prometheus does have an official
96
+container I would like to walk through the install process and setup on a fresh container. This will allow anyone
97
+reading to migrate this tutorial to a VM or Server of any sort.
98
+
99
+Let's start another container in the same fashion as we did the Netdata container.
100
+
101
+```sh
102
+docker run -it --name prometheus --hostname prometheus
103
+--network=netdata-tutorial -p 9090:9090 centos:latest '/bin/bash'
104
+```
105
+
106
+This should drop you into a shell once again. Once there quickly install your favorite editor as we will be editing
107
+files later in this tutorial.
108
+
109
+```sh
110
+yum install vim -y
111
+```
112
+
113
+Prometheus provides a tarball of their latest stable versions [here](https://prometheus.io/download/).
114
+
115
+Let's download the latest version and install into your container.
116
+
117
+```sh
118
+cd /tmp && curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \
119
+| grep "browser_download_url.*linux-amd64.tar.gz" \
120
+| cut -d '"' -f 4 \
121
+| wget -qi -
122
+
123
+mkdir /opt/prometheus
124
+
125
+sudo tar -xvf /tmp/prometheus-*linux-amd64.tar.gz -C /opt/prometheus --strip=1
126
+```
127
+
128
+This should get Prometheus installed into the container. Let's test that we can run Prometheus and connect to it's web
129
+interface.
130
+
131
+```sh
132
+/opt/prometheus/prometheus
133
+```
134
+
135
+Now attempt to go to <http://localhost:9090/>. You should be presented with the Prometheus homepage. This is a good
136
+point to talk about Prometheus's data model which can be viewed here: <https://prometheus.io/docs/concepts/data_model/>
137
+As explained we have two key elements in Prometheus metrics. We have the _metric_ and its _labels_. Labels allow for
138
+granularity between metrics. Let's use our previous example to further explain.
139
+
140
+```conf
141
+netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 0.0831255 1501271696000
142
+```
143
+
144
+Here our metric is `netdata_system_cpu_percentage_average` and our labels are `chart`, `family`, and `dimension`. The
145
+last two values constitute the actual metric value for the metric type (gauge, counter, etc…). We can begin graphing
146
+system metrics with this information, but first we need to hook up Prometheus to poll Netdata stats.
147
+
148
+Let's move our attention to Prometheus's configuration. Prometheus gets it config from the file located (in our example)
149
+at `/opt/prometheus/prometheus.yml`. I won't spend an extensive amount of time going over the configuration values
150
+documented here: <https://prometheus.io/docs/operating/configuration/>. We will be adding a new job under the
151
+`scrape_configs`. Let's make the `scrape_configs` section look like this (we can use the DNS name Netdata due to the
152
+custom user-defined network we created in docker beforehand).
153
+
154
+```yaml
155
+scrape_configs:
156
+ # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
157
+ - job_name: 'prometheus'
158
+
159
+ # metrics_path defaults to '/metrics'
160
+ # scheme defaults to 'http'.
161
+
162
+ static_configs:
163
+ - targets: ['localhost:9090']
164
+
165
+ - job_name: 'netdata'
166
+
167
+ metrics_path: /api/v1/allmetrics
168
+ params:
169
+ format: [ prometheus ]
170
+
171
+ static_configs:
172
+ - targets: ['netdata:19999']
173
+```
174
+
175
+Let's start Prometheus once again by running `/opt/prometheus/prometheus`. If we now navigate to Prometheus at
176
+<http://localhost:9090/targets> we should see our target being successfully scraped. If we now go back to the
177
+Prometheus's homepage and begin to type `netdata\_` Prometheus should auto complete metrics it is now scraping.
178
+
179
+
180
+
181
+Let's now start exploring how we can graph some metrics. Back in our NetData container lets get the CPU spinning with a
182
+pointless busy loop. On the shell do the following:
183
+
184
+```sh
185
+[root@netdata /]# while true; do echo "HOT HOT HOT CPU"; done
186
+```
187
+
188
+Our NetData cpu graph should be showing some activity. Let's represent this in Prometheus. In order to do this let's
189
+keep our metrics page open for reference: <http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes>. We are
190
+setting out to graph the data in the CPU chart so let's search for `system.cpu` in the metrics page above. We come
191
+across a section of metrics with the first comments `# COMMENT homogeneous chart "system.cpu", context "system.cpu",
192
+family "cpu", units "percentage"` followed by the metrics. This is a good start now let us drill down to the specific
193
+metric we would like to graph.
194
+
195
+```conf
196
+# COMMENT
197
+netdata_system_cpu_percentage_average: dimension "system", value is percentage, gauge, dt 1501275951 to 1501275951 inclusive
198
+netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 0.0000000 1501275951000
199
+```
200
+
201
+Here we learn that the metric name we care about is `netdata_system_cpu_percentage_average` so throw this into
202
+Prometheus and see what we get. We should see something similar to this (I shut off my busy loop)
203
+
204
+
205
+
206
+This is a good step toward what we want. Also make note that Prometheus will tag on an `instance` label for us which
207
+corresponds to our statically defined job in the configuration file. This allows us to tailor our queries to specific
208
+instances. Now we need to isolate the dimension we want in our query. To do this let us refine the query slightly. Let's
209
+query the dimension also. Place this into our query text box.
210
+`netdata_system_cpu_percentage_average{dimension="system"}` We now wind up with the following graph.
211
+
212
+
213
+
214
+Awesome, this is exactly what we wanted. If you haven't caught on yet we can emulate entire charts from NetData by using
215
+the `chart` dimension. If you'd like you can combine the `chart` and `instance` dimension to create per-instance charts.
216
+Let's give this a try: `netdata_system_cpu_percentage_average{chart="system.cpu", instance="netdata:19999"}`
217
+
218
+This is the basics of using Prometheus to query NetData. I'd advise everyone at this point to read [this
219
+page](/exporting/prometheus/#using-netdata-with-prometheus). The key point here is that NetData can export metrics from
220
+its internal DB or can send metrics _as-collected_ by specifying the `source=as-collected` URL parameter like so.
221
+<http://localhost:19999/api/v1/allmetrics?format=prometheus&help=yes&types=yes&source=as-collected> If you choose to use
222
+this method you will need to use Prometheus's set of functions here: <https://prometheus.io/docs/querying/functions/> to
223
+obtain useful metrics as you are now dealing with raw counters from the system. For example you will have to use the
224
+`irate()` function over a counter to get that metric's rate per second. If your graphing needs are met by using the
225
+metrics returned by NetData's internal database (not specifying any source= URL parameter) then use that. If you find
226
+limitations then consider re-writing your queries using the raw data and using Prometheus functions to get the desired
227
+chart.
228
+
229
+## Grafana
230
+
231
+Finally we make it to grafana. This is the easiest part in my opinion. This time we will actually run the official
232
+grafana docker container as all configuration we need to do is done via the GUI. Let's run the following command:
233
+
234
+```sh
235
+docker run -i -p 3000:3000 --network=netdata-tutorial grafana/grafana
236
+```
237
+
238
+This will get grafana running at <http://localhost:3000/>. Let's go there and
239
+login using the credentials Admin:Admin.
240
+
241
+The first thing we want to do is click "Add data source". Let's make it look like the following screenshot
242
+
243
+
244
+
245
+With this completed let's graph! Create a new Dashboard by clicking on the top left Grafana Icon and create a new graph
246
+in that dashboard. Fill in the query like we did above and save.
247
+
248
+
249
+
250
+## Conclusion
251
+
252
+There you have it, a complete systems monitoring stack which is very easy to deploy. From here I would begin to
253
+understand how Prometheus and a service discovery mechanism such as Consul can play together nicely. My current prod
254
+deployments automatically register Netdata services into Consul and Prometheus automatically begins to scrape them. Once
255
+achieved you do not have to think about the monitoring system until Prometheus cannot keep up with your scale. Once this
256
+happens there are options presented in the Prometheus documentation for solving this. Hope this was helpful, happy
257
+monitoring.
258
+
259
+[](<>)
exporting/exporting.conf
new
+88
@@ -0,0 +1,88 @@
1
+[exporting:global]
2
+ enabled = no
3
+ # send configured labels = yes
4
+ # send automatic labels = no
5
+ # update every = 10
6
+
7
+[prometheus:exporter]
8
+ # send names instead of ids = yes
9
+ # send configured labels = yes
10
+ # send automatic labels = no
11
+ # send charts matching = *
12
+ # send hosts matching = localhost *
13
+ # prefix = netdata
14
+
15
+# An example configuration for graphite, json, opentsdb exporting connectors
16
+# [graphite:my_graphite_instance]
17
+ # enabled = no
18
+ # destination = localhost
19
+ # data source = average
20
+ # prefix = netdata
21
+ # hostname = my_hostname
22
+ # update every = 10
23
+ # buffer on failures = 10
24
+ # timeout ms = 20000
25
+ # send names instead of ids = yes
26
+ # send charts matching = *
27
+ # send hosts matching = localhost *
28
+
29
+# [prometheus_remote_write:my_prometheus_remote_write_instance]
30
+ # enabled = no
31
+ # destination = localhost
32
+ # remote write URL path = /receive
33
+ # data source = average
34
+ # prefix = netdata
35
+ # hostname = my_hostname
36
+ # update every = 10
37
+ # buffer on failures = 10
38
+ # timeout ms = 20000
39
+ # send names instead of ids = yes
40
+ # send charts matching = *
41
+ # send hosts matching = localhost *
42
+
43
+# [kinesis:my_kinesis_instance]
44
+ # enabled = no
45
+ # destination = us-east-1
46
+ # stream name = netdata
47
+ # aws_access_key_id = my_access_key_id
48
+ # aws_secret_access_key = my_aws_secret_access_key
49
+ # data source = average
50
+ # prefix = netdata
51
+ # hostname = my_hostname
52
+ # update every = 10
53
+ # buffer on failures = 10
54
+ # timeout ms = 20000
55
+ # send names instead of ids = yes
56
+ # send charts matching = *
57
+ # send hosts matching = localhost *
58
+
59
+# [pubsub:my_pubsub_instance]
60
+ # enabled = no
61
+ # destination = pubsub.googleapis.com
62
+ # credentials file = /etc/netdata/pubsub_credentials.json
63
+ # project id = my_project
64
+ # topic id = my_topic
65
+ # data source = average
66
+ # prefix = netdata
67
+ # hostname = my_hostname
68
+ # update every = 10
69
+ # buffer on failures = 10
70
+ # timeout ms = 20000
71
+ # send names instead of ids = yes
72
+ # send charts matching = *
73
+ # send hosts matching = localhost *
74
+
75
+# [mongodb:my_mongodb_instance]
76
+ # enabled = no
77
+ # destination = localhost
78
+ # database = my_database
79
+ # collection = my_collection
80
+ # data source = average
81
+ # prefix = netdata
82
+ # hostname = my_hostname
83
+ # update every = 10
84
+ # buffer on failures = 10
85
+ # timeout ms = 20000
86
+ # send names instead of ids = yes
87
+ # send charts matching = *
88
+ # send hosts matching = localhost *
exporting/opentsdb/Makefile.am
+4
@@ -2,3 +2,7 @@
2
3
AUTOMAKE_OPTIONS = subdir-objects
4
MAINTAINERCLEANFILES = $(srcdir)/Makefile.in
5
+
6
+dist_noinst_DATA = \
7
+ README.md \
8
+ $(NULL)
exporting/opentsdb/README.md
new
+40
@@ -0,0 +1,40 @@
1
+<!--
2
+title: "Export metrics to OpenTSDB with HTTP"
3
+description: "Archive your Agent's metrics to a OpenTSDB database for long-term storage and further analysis."
4
+custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/opentsdb/README.md
5
+sidebar_label: OpenTSDB with HTTP
6
+-->
7
+
8
+# Export metrics to OpenTSDB with HTTP
9
+
10
+Netdata can easily communicate with OpenTSDB using HTTP API. To enable this channel, run `./edit-config exporting.conf`
11
+in the Netdata configuration directory and set the following options:
12
+
13
+```conf
14
+[opentsdb:http:my_instance]
15
+ enabled = yes
16
+ destination = localhost:4242
17
+```
18
+
19
+In this example, OpenTSDB is running with its default port, which is `4242`. If you run OpenTSDB on a different port,
20
+change the `destination = localhost:4242` line accordingly.
21
+
22
+## HTTPS
23
+
24
+As of [v1.16.0](https://github.com/netdata/netdata/releases/tag/v1.16.0), Netdata can send metrics to OpenTSDB using
25
+TLS/SSL. Unfortunately, OpenTDSB does not support encrypted connections, so you will have to configure a reverse proxy
26
+to enable HTTPS communication between Netdata and OpenTSBD. You can set up a reverse proxy with
27
+[Nginx](/docs/Running-behind-nginx.md).
28
+
29
+After your proxy is configured, make the following changes to `exporting.conf`:
30
+
31
+```conf
32
+[opentsdb:https:my_instance]
33
+ enabled = yes
34
+ destination = localhost:8082
35
+```
36
+
37
+In this example, we used the port `8082` for our reverse proxy. If your reverse proxy listens on a different port,
38
+change the `destination = localhost:8082` line accordingly.
39
+
40
+[](<>)
exporting/prometheus/README.md
+12
-5
@@ -1,3 +1,10 @@
1
+<!--
2
+title: "Export metrics to Prometheus"
3
+description: "Export Netdata metrics to Prometheus for archiving and further analysis."
4
+custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/prometheus/README.md
5
+sidebar_label: Using Netdata with Prometheus
6
+-->
7
+
8
# Using Netdata with Prometheus
9
10
> IMPORTANT: the format Netdata sends metrics to Prometheus has changed since Netdata v1.7. The new Prometheus exporting
@@ -13,7 +20,7 @@ are starting at a fresh ubuntu shell (whether you'd like to follow along in a VM
20
21
### Installing Netdata
22
16
-There are number of ways to install Netdata according to [Installation](../../packaging/installer/). The suggested way
23
+There are number of ways to install Netdata according to [Installation](/packaging/installer/README.md). The suggested way
24
of installing the latest Netdata and keep it upgrade automatically. Using one line installation:
25
26
```sh
@@ -390,10 +397,10 @@ names are human friendly labels (also unique).
397
Most charts and metrics have the same ID and name, but in several cases they are different: disks with device-mapper,
398
interrupts, QoS classes, statsd synthetic charts, etc.
399
393
-The default is controlled in `netdata.conf`:
400
+The default is controlled in `exporting.conf`:
401
402
```conf
396
-[backend]
403
+[prometheus:exporter]
404
send names instead of ids = yes | no
405
```
406
@@ -407,7 +414,7 @@ You can overwrite it from Prometheus, by appending to the URL:
414
Netdata can filter the metrics it sends to Prometheus with this setting:
415
416
```conf
410
-[backend]
417
+[prometheus:exporter]
418
send charts matching = *
419
```
420
@@ -422,7 +429,7 @@ is used.
429
Netdata sends all metrics prefixed with `netdata_`. You can change this in `netdata.conf`, like this:
430
431
```conf
425
-[backend]
432
+[prometheus:exporter]
433
prefix = netdata
434
```
435
exporting/pubsub/README.md
+6
-2
@@ -11,8 +11,12 @@ sidebar_label: Google Cloud Pub/Sub Service
11
12
To use the Pub/Sub service for metric collecting and processing, you should first
13
[install](https://github.com/googleapis/cpp-cmakefiles) Google Cloud Platform C++ Proto Libraries.
14
-Pub/Sub support is also dependent on the dependencies of those libraries, like `protobuf` and `grpc`. Next, Netdata
15
-should be re-installed from the source. The installer will detect that the required libraries are now available.
14
+Pub/Sub support is also dependent on the dependencies of those libraries, like `protobuf`, `protoc`, and `grpc`. Next,
15
+Netdata should be re-installed from the source. The installer will detect that the required libraries are now available.
16
+
17
+> Some distributions don't have `.cmake` files in packages. To build the C++ Proto Libraries on such distributions we
18
+> advise you to delete `protobuf`, `protoc`, and `grpc` related packages and
19
+> [install](https://github.com/grpc/grpc/blob/master/BUILDING.md) `grpc` with its dependencies from source.
20
21
## Configuration
22