Add new exporting "home base" document (#9246)
* Init new export doc * Touch of cleanup * A few additional fixes * Fix broken links * Fix for Thiago * Fixes for Vlad
Joel Hans committed
Jun 2, 2020 at 11:08 UTC
f3d762437a01d5f7d2d5829b9120a20ccb69ff97
4 files changed
+133
-24
backends/README.md
+3
@@ -7,6 +7,9 @@ custom_edit_url: https://github.com/netdata/netdata/edit/master/backends/README.
7
8
# Metrics long term archiving
9
10
+> ⚠️ The backends system is now deprecated in favor of the [exporting engine](/exporting/README.md). Please see the
11
+> [migration guide](/docs/export/) for details on how to get started with exporting.
12
+
13
Netdata supports backends for archiving the metrics, or providing long term dashboards, using Grafana or other tools,
14
like this:
15
docs/export/README.md
new
+101
@@ -0,0 +1,101 @@
1
+<!--
2
+title: Export metrics
3
+description: "Archive your Netdata metrics to multiple external time series databases for long-term storage or further analysis."
4
+custom_edit_url: https://github.com/netdata/netdata/edit/master/docs/export/README.md
5
+-->
6
+
7
+# Export metrics
8
+
9
+One of Netdata's pillars is interoperability with other monitoring and visualization solutions. To this end, you can use
10
+the Agent's [exporting engine](/exporting/README.md) to send metrics to multiple external databases/services in
11
+parallel. Once you connect Netdata metrics to other solutions, you can apply machine learning analysis or correlation
12
+with other tools, such as application tracing.
13
+
14
+The exporting engine supports a number of connectors, including AWS Kinesis Data Streams, Graphite, JSON, MongoDB,
15
+OpenTSDB, Prometheus remote write, and more, via exporting **connectors**. These connectors help you seamlessly send
16
+Netdata metrics to more than 20 different endpoints, including every [service that
17
+supports](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) Prometheus remote write. See
18
+the [exporting reference guide](/exporting/README.md) for the full list.
19
+
20
+## Exporting quickstart
21
+
22
+Let's cover the process of enabling an exporting connector, using the Graphite connector as an example. These steps can
23
+be applied to other connectors as well.
24
+
25
+> If you are migrating from the deprecated backends system, this quickstart will also help you update your configuration
26
+> to the new format. For the most part, the configurations are identical, but there are two exceptions. First,
27
+> `exporting.conf` uses a new `[<type>:<name>]` format for defining connector instances. Second, the `host tags` setting
28
+> is deprecated. Instead, use [host labels](/docs/tutorials/using-host-labels.md) to tag exported metrics.
29
+
30
+Open the `exporting.conf` file with `edit-config`.
31
+
32
+```bash
33
+cd /etc/netdata # Replace this path with your Netdata config directory
34
+sudo ./edit-config exporting.conf
35
+```
36
+
37
+### Enable the exporting engine
38
+
39
+Enable the exporting engine by setting `enabled` to `yes`:
40
+
41
+```conf
42
+[exporting:global]
43
+ enabled = yes
44
+```
45
+
46
+### Change how often the exporting engine sends metrics
47
+
48
+By default, the exporting engine only sends metrics to external databases every 10 seconds to avoid congesting the
49
+destination with thousands of per-second metrics.
50
+
51
+You can change this frequency for all connectors based on how you use exported metrics or the resources you can allocate
52
+to long-term storage. Use the `update every` setting to change the frequency in seconds.
53
+
54
+```conf
55
+[exporting:global]
56
+ update every = 10
57
+```
58
+
59
+### Enable a connector (Graphite)
60
+
61
+To enable the Graphite connector, find the `[graphite:my_graphite_instance]` example section in `exporting.conf`. You
62
+can use this (or the respective example for the connector you want to use) as a framework for your configration.
63
+
64
+`[graphite:my_graphite_instance]` is an example of the new `[<type>:<name>]` format for defining connector instances.
65
+
66
+Uncomment the section itself and replace `my_graphite_instance` with a name of your choice. Then set `enabled` to `yes`
67
+and uncomment the line.
68
+
69
+```conf
70
+[graphite:my_graphite_instance]
71
+ enabled = yes
72
+ # destination = localhost:2003
73
+ # data source = average
74
+ # prefix = netdata
75
+ # hostname = my_hostname
76
+ # update every = 10
77
+ # buffer on failures = 10
78
+ # timeout ms = 20000
79
+ # send names instead of ids = yes
80
+ # send charts matching = *
81
+ # send hosts matching = localhost *
82
+```
83
+
84
+Next, edit and uncomment any other lines necessary to connect the exporting engine to your endpoint. If migrating from
85
+backends, port your settings over and uncomment any lines you change. You must edit the `destination` setting in most
86
+situations.
87
+
88
+For details on all the configuration options, see the [exporting reference](/exporting/README.md#configuration).
89
+
90
+Restart your Agent to begin exporting to the destination of your choice. Because the Agent exports metrics as they're
91
+collected, you should start seeing data in your external database after only a few seconds.
92
+
93
+## Exporting reference and related features
94
+
95
+- [Exporting reference guide](/exporting/README.md)
96
+- [Backends (deprecated)](/backends/README.md)
97
+- [Use host labels to organize systems, metrics, and alarms](/docs/tutorials/using-host-labels.md)
98
+- [Database engine](/database/engine/README.md)
99
+- [Change how long Netdata stores metrics (long-term storage)](/docs/tutorials/longer-metrics-storage.md)
100
+
101
+[](<>)
exporting/README.md
+25
-20
@@ -1,25 +1,29 @@
1
<!--
2
-title: "Export metrics to external databases"
2
+title: "Exporting engine reference"
3
description: "With the exporting engine, you can archive your Netdata metrics to multiple external databases for long-term storage or further analysis."
4
+sidebar_label: Reference guide
5
custom_edit_url: https://github.com/netdata/netdata/edit/master/exporting/README.md
6
-->
7
7
-# Exporting metrics to external databases
8
+# Exporting engine reference
9
9
-The exporting engine is an update for the former [backends](/backends/README.md) which is deprecated and will be deleted
10
-soon. It has a modular structure and supports metric exporting via multiple exporting connector instances at the same
11
-time. You can have different update intervals and filters configured for every exporting connector instance. The
12
-exporting engine has its own configuration file `exporting.conf`. Configuration is almost similar to
13
-[backends](/backends/README.md#configuration). The most important difference is that type of a connector should be
14
-specified in a section name before a colon and an instance name after the colon. Also, you can't use `host tags`
15
-anymore. Set your labels using the [`[host labels]`](/docs/tutorials/using-host-labels.md) section in `netdata.conf`.
10
+Welcome to the exporting engine reference guide.
11
+
12
+This guide contains comprehensive information about enabling, configuring, and monitoring Netdata's exporting engine,
13
+which allows you to send metrics to more than 20 external time series databases.
14
17
-# Metrics long term archiving
15
+To learn the basics of locating and editing health configuration files, read up on [how to export
16
+metrics](/docs/export/README.md), and follow the [exporting
17
+quickstart](/docs/export/README.md#exporting-quickstart).
18
19
-Netdata supports external databases and services for archiving the metrics, or providing long term dashboards, using
20
-Grafana or other tools, like this:
19
+The exporting engine is an update for the former [backends](/backends/README.md), which is deprecated and will be
20
+deleted soon. It has a modular structure and supports metric exporting via multiple exporting connector instances at the
21
+same time. You can have different update intervals and filters configured for every exporting connector instance.
22
22
-
23
+The exporting engine has its own configuration file `exporting.conf`. Configuration is almost similar to
24
+[backends](/backends/README.md#configuration). The most important difference is that type of a connector should be
25
+specified in a section name before a colon and an instance name after the colon. Also, you can't use `host tags`
26
+anymore. Set your labels using the [`[host labels]`](/docs/tutorials/using-host-labels.md) section in `netdata.conf`.
27
28
Since Netdata collects thousands of metrics per server per second, which would easily congest any database server when
29
several Netdata servers are sending data to it, Netdata allows sending metrics at a lower frequency, by resampling them.
@@ -27,7 +31,7 @@ several Netdata servers are sending data to it, Netdata allows sending metrics a
31
So, although Netdata collects metrics every second, it can send to the external database servers averages or sums every
32
X seconds (though, it can send them per second if you need it to).
33
30
-## features
34
+## Features
35
36
1. Supported databases and services
37
@@ -104,7 +108,7 @@ X seconds (though, it can send them per second if you need it to).
108
their batches at the same time. You can set different update intervals for every exporting connector instance, but
109
even in that case they can occasionally synchronize their batches for a moment.
110
107
-## configuration
111
+## Configuration
112
113
In `/etc/netdata/exporting.conf` you should have something like this:
114
@@ -252,9 +256,10 @@ Options:
256
> You can check how the host tags were parsed using the /api/v1/info API call. But, keep in mind that backends subsystem
257
> is deprecated and will be deleted soon. Please move your existing tags to the `[host labels]` section.
258
255
-## monitoring operation
259
+## Exporting engine monitoring
260
257
-Netdata provides 5 charts:
261
+Netdata creates five charts in the dashboard, under the **Netdata Monitoring** section, to help you monitor the health
262
+and performance of the exporting engine itself:
263
264
1. **Buffered metrics**, the number of metrics Netdata added to the buffer for dispatching them to the
265
external database server.
@@ -263,12 +268,12 @@ Netdata provides 5 charts:
268
269
3. **Exporting operations**, the number of operations performed by Netdata.
270
266
-4. **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending the
267
- metrics to the external database server.
271
+4. **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending
272
+ the metrics to the external database server.
273
274
275
271
-## alarms
276
+## Exporting engine alarms
277
278
Netdata adds 3 alarms:
279
exporting/mongodb/README.md
+4
-4
@@ -7,8 +7,8 @@ sidebar_label: MongoDB
7
8
# Export metrics to MongoDB
9
10
-You can use the MongoDB connector and the experimental [exporting engine](../README.md) to archive your agent's metrics
11
-to a MongoDB database for long-term storage, further analysis, or correlation with data from other sources.
10
+You can use the MongoDB connector for the [exporting engine](/exporting/README.md) to archive your agent's metrics to a
11
+MongoDB database for long-term storage, further analysis, or correlation with data from other sources.
12
13
## Prerequisites
14
@@ -18,8 +18,8 @@ from the source, which detects that the required library is now available.
18
19
## Configuration
20
21
-To enable data exporting to a MongoDB database, run `./edit-config exporting.conf`
22
-in the Netdata configuration directory and set the following options:
21
+To enable data exporting to a MongoDB database, run `./edit-config exporting.conf` in the Netdata configuration
22
+directory and set the following options:
23
24
```conf
25
[mongodb:my_instance]