Improve exporting documentation clarity and structure (#20890)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Kanela committed
Aug 26, 2025 at 20:33 UTC
ad3a74ac9c820ada26ea57921da4a24b408a3c1c
4 files changed
+355
-380
docs/exporting-metrics/README.md
+61
-55
@@ -1,55 +1,61 @@
1
-# Export metrics to external time-series databases
2
-
3
-Netdata natively supports long term retention of metrics. Its tiered database design typically provides significantly longer retention (months to years) and faster queries (typically 20+ times faster), compared to other common time-series databases.
4
-
5
-For integration with other observability tools, Netdata provides a number of exporters allow you to copy metrics to third party time-series databases for additional analysis or integration with other tools.
6
-
7
-Exporters enable connections to [more than thirty](#supported-databases) supported databases, including InfluxDB, Prometheus, Graphite, ElasticSearch, and much more.
8
-
9
-The exporting engine is able to downsample Netdata's per-second metrics at a user-configurable interval (e.g per minute), and can export metrics to multiple time-series databases simultaneously.
10
-
11
-Based on your needs and resources you allocated to your external time-series database, you can configure the interval
12
-that metrics are exported or export only certain charts with filtering. You can also choose whether metrics are exported
13
-as-collected, a normalized average, or the sum/volume of metrics values over the configured interval.
14
-
15
-## Supported databases
16
-
17
-Netdata supports exporting metrics to the following databases through several
18
-[connectors](/src/exporting/README.md#features). Once you find the connector that works for your database, open its
19
-documentation and the [enabling a connector](/docs/exporting-metrics/enable-an-exporting-connector.md) doc for details on enabling it.
20
-
21
-- **AppOptics**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
22
-- **AWS Kinesis**: [AWS Kinesis Data Streams](/src/exporting/aws_kinesis/README.md).
23
-- **Azure Data Explorer**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
24
-- **Azure Event Hubs**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
25
-- **Blueflood**: [Graphite](/src/exporting/graphite/README.md).
26
-- **Chronix**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
27
-- **Cortex**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
28
-- **CrateDB**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
29
-- **ElasticSearch**: [Graphite](/src/exporting/graphite/README.md), [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
30
-- **Gnocchi**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
31
-- **Google BigQuery**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
32
-- **Google Cloud Pub/Sub**: [Google Cloud Pub/Sub Service](/src/exporting/pubsub/README.md).
33
-- **Graphite**: [Graphite](/src/exporting/graphite/README.md), [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
34
-- **InfluxDB**: [Graphite](/src/exporting/graphite/README.md), [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
35
-- **IRONdb**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
36
-- **JSON**: [JSON document databases](/src/exporting/json/README.md).
37
-- **Kafka**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
38
-- **KairosDB**: [Graphite](/src/exporting/graphite/README.md), [OpenTSDB](/src/exporting/opentsdb/README.md).
39
-- **M3DB**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
40
-- **MetricFire**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
41
-- **MongoDB**: [MongoDB](/src/exporting/mongodb/README.md).
42
-- **New Relic**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
43
-- **OpenTSDB**: [OpenTSDB](/src/exporting/opentsdb/README.md), [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
44
-- **PostgreSQL**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) via [PostgreSQL Prometheus Adapter](https://github.com/CrunchyData/postgresql-prometheus-adapter).
45
-- **Prometheus**: [Prometheus scraper](/src/exporting/prometheus/README.md).
46
-- **TimescaleDB**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md), [netdata-timescale-relay](/src/exporting/TIMESCALE.md).
47
-- **QuasarDB**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
48
-- **SignalFx**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
49
-- **Splunk**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
50
-- **TiKV**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
51
-- **Thanos**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
52
-- **VictoriaMetrics**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
53
-- **Wavefront**: [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md).
54
-
55
-Can't find your preferred external time-series database? Ask our [community](https://community.netdata.cloud/) for solutions, or file an [issue on GitHub](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml).
1
+# Export Metrics to External Time-Series Databases
2
+
3
+Netdata natively provides long-term metrics retention through its tiered database design. This architecture delivers significantly longer retention (months to years) and faster queries (typically 20+ times faster) compared to other common time-series databases.
4
+
5
+For integration with other observability tools, Netdata includes exporters that copy metrics to third-party time-series databases for additional analysis or integration with other tools.
6
+
7
+## Exporting Capabilities
8
+
9
+The exporting engine provides these key features:
10
+
11
+- **Multi-database support**: Export to [more than thirty](#supported-databases) databases including InfluxDB, Prometheus, Graphite, ElasticSearch, and more
12
+- **Downsampling**: Configure export intervals from Netdata's per-second metrics (e.g., per minute)
13
+- **Simultaneous exports**: Send metrics to multiple time-series databases at once
14
+- **Flexible data processing**: Export metrics as-collected, normalized averages, or sum/volume over configured intervals
15
+- **Selective exporting**: Filter specific charts based on your needs and allocated resources
16
+
17
+## Supported Databases
18
+
19
+Netdata exports metrics to the following databases through various [connectors](/src/exporting/README.md#supported-connectors). Each connector includes documentation with [enabling instructions](/docs/exporting-metrics/enable-an-exporting-connector.md).
20
+
21
+| Database | Supported Connectors |
22
+|:------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
23
+| **AppOptics** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
24
+| **AWS Kinesis** | [AWS Kinesis Data Streams](/src/exporting/aws_kinesis/README.md) |
25
+| **Azure Data Explorer** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
26
+| **Azure Event Hubs** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
27
+| **Blueflood** | [Graphite](/src/exporting/graphite/README.md) |
28
+| **Chronix** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
29
+| **Cortex** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
30
+| **CrateDB** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
31
+| **ElasticSearch** | [Graphite](/src/exporting/graphite/README.md), [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
32
+| **Gnocchi** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
33
+| **Google BigQuery** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
34
+| **Google Cloud Pub/Sub** | [Google Cloud Pub/Sub Service](/src/exporting/pubsub/README.md) |
35
+| **Graphite** | [Graphite](/src/exporting/graphite/README.md), [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
36
+| **InfluxDB** | [Graphite](/src/exporting/graphite/README.md), [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
37
+| **IRONdb** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
38
+| **JSON** | [JSON document databases](/src/exporting/json/README.md) |
39
+| **Kafka** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
40
+| **KairosDB** | [Graphite](/src/exporting/graphite/README.md), [OpenTSDB](/src/exporting/opentsdb/README.md) |
41
+| **M3DB** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
42
+| **MetricFire** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
43
+| **MongoDB** | [MongoDB](/src/exporting/mongodb/README.md) |
44
+| **New Relic** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
45
+| **OpenTSDB** | [OpenTSDB](/src/exporting/opentsdb/README.md), [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
46
+| **PostgreSQL** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) via [PostgreSQL Prometheus Adapter](https://github.com/CrunchyData/postgresql-prometheus-adapter) |
47
+| **Prometheus** | [Prometheus scraper](/src/exporting/prometheus/README.md) |
48
+| **TimescaleDB** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md), [netdata-timescale-relay](/src/exporting/TIMESCALE.md) |
49
+| **QuasarDB** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
50
+| **SignalFx** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
51
+| **Splunk** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
52
+| **TiKV** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
53
+| **Thanos** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
54
+| **VictoriaMetrics** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
55
+| **Wavefront** | [Prometheus remote write](/src/exporting/prometheus/remote_write/README.md) |
56
+
57
+:::tip
58
+
59
+**Can't find your preferred external time-series database?** Ask our [community](https://community.netdata.cloud/) for solutions, or file an [issue on GitHub](https://github.com/netdata/netdata/issues/new?assignees=&labels=bug%2Cneeds+triage&template=BUG_REPORT.yml).
60
+
61
+:::
docs/exporting-metrics/enable-an-exporting-connector.md
+37
-18
@@ -1,34 +1,32 @@
1
-# Enable an exporting connector
1
+# Enable an Exporting Connector
2
3
-Now that you found the right connector for your [external time-series
4
-database](/docs/exporting-metrics/README.md#supported-databases), you can now enable the exporting engine and the
5
-connector itself. We'll walk through the process of enabling the exporting engine itself, followed by two examples using
6
-the OpenTSDB and Graphite connectors.
3
+After selecting the right connector for your [external time-series database](/docs/exporting-metrics/README.md#supported-databases), you can enable the exporting engine and configure your connector. This guide walks through enabling the exporting engine itself, followed by two examples using the OpenTSDB and Graphite connectors.
4
8
-> **Note**
9
->
10
-> When you enable the exporting engine and a connector, the Netdata Agent exports metrics _beginning from the time you
11
-> restart its process_, not the entire
12
-> [database of long-term metrics](/src/database/README.md).
5
+:::info
6
14
-Once you understand how to enable a connector, you can apply that knowledge to any other connector.
7
+When you enable the exporting engine and a connector, Netdata exports metrics **starting from the Agent restart time**, not the entire [historical database](/src/database/README.md).
8
+
9
+:::
10
16
-## Enable the exporting engine
11
+Once you understand how to enable a connector, you can apply that knowledge to any other connector.
12
18
-Use `edit-config` from your [Netdata config directory](/docs/netdata-agent/configuration/README.md#the-netdata-config-directory) to edit `exporting.conf`.
13
+## Enable the Exporting Engine
14
20
-Enable the exporting engine itself by setting `enabled` to `yes`:
15
+Edit `exporting.conf` using `edit-config` from your [Netdata config directory](/docs/netdata-agent/configuration/README.md#edit-configuration-files):
16
17
```text
18
[exporting:global]
19
enabled = yes
20
```
21
27
-Save the file but keep it open, as you will edit it again to enable specific connectors.
22
+Save the file but keep it open—you will edit it again to enable specific connectors.
23
+
24
+## Examples
25
29
-## Example: Enable the OpenTSDB connector
26
+<details>
27
+<summary><strong>Enable the OpenTSDB Connector</strong></summary>
28
31
-Use the following configuration as a starting point. Copy and paste it into `exporting.conf`.
29
+Use the following configuration as a starting point. Copy and paste it into `exporting.conf`:
30
31
```text
32
[opentsdb:http:my_opentsdb_http_instance]
@@ -40,4 +38,25 @@ Replace `my_opentsdb_http_instance` with an instance name of your choice, and ch
38
39
[Restart your Agent](/docs/netdata-agent/start-stop-restart.md) to initiate exporting to your OpenTSDB database. The Netdata Agent continuously exports metrics collected from the moment it starts. You can expect to see data appear in your OpenTSDB database within seconds of restarting the Agent.
40
43
-Any further configuration is optional, based on your needs and the configuration of your OpenTSDB database. See the [OpenTSDB connector doc](/src/exporting/opentsdb/README.md) and [exporting engine reference](/src/exporting/README.md#configuration) for details.
41
+Any further configuration is optional, based on your needs and the configuration of your OpenTSDB database. See the [OpenTSDB connector doc](/src/exporting/opentsdb/README.md) and [exporting engine reference](/src/exporting/README.md#configuration-structure) for details.
42
+
43
+</details>
44
+
45
+<details>
46
+<summary><strong>Enable the Graphite Connector</strong></summary>
47
+
48
+Use the following configuration as a starting point. Copy and paste it into `exporting.conf`:
49
+
50
+```text
51
+[graphite:netdata]
52
+ enabled = yes
53
+ destination = localhost:2003
54
+```
55
+
56
+Replace `netdata` with an instance name of your choice, and change the `destination` setting to the IP address or hostname of your Graphite database.
57
+
58
+[Restart your Agent](/docs/netdata-agent/start-stop-restart.md) to initiate exporting to your Graphite database. The Netdata Agent continuously exports metrics collected from the moment it starts. You can expect to see data appear in your Graphite database within seconds of restarting the Agent.
59
+
60
+Any further configuration is optional, based on your needs and the configuration of your Graphite database. See the [Graphite connector doc](/src/exporting/graphite/README.md) and [exporting engine reference](/src/exporting/README.md#configuration-structure) for details.
61
+
62
+</details>
src/exporting/README.md
+138
-175
@@ -1,110 +1,60 @@
1
-# Exporting reference
1
+# Exporting Reference
2
3
-Welcome to the exporting engine reference guide. This guide contains comprehensive information about enabling,
4
-configuring, and monitoring Netdata's exporting engine, which allows you to send metrics to external time-series
5
-databases.
3
+This reference guide provides comprehensive information about enabling, configuring, and monitoring Netdata's exporting engine for sending metrics to external time-series databases.
4
7
-For a quick introduction to the exporting engine's features, read our doc on [exporting metrics to time-series
8
-databases](/docs/exporting-metrics/README.md), or jump in to [enabling a connector](/docs/exporting-metrics/enable-an-exporting-connector.md).
5
+For a quick introduction, read our [exporting metrics overview](/docs/exporting-metrics/README.md) or start with [enabling a connector](/docs/exporting-metrics/enable-an-exporting-connector.md).
6
10
-The exporting engine has a modular structure and supports metric exporting via multiple exporting connector instances at
11
-the same time. You can have different update intervals and filters configured for every exporting connector instance.
7
+## Core Capabilities
8
13
-When you enable the exporting engine and a connector, the Netdata Agent exports metrics _beginning from the time you
14
-restart its process_, not the entire [database of long-term metrics](/src/database/README.md).
9
+The exporting engine features a modular structure that supports:
10
16
-Since Netdata collects thousands of metrics per server per second, which would easily congest any database server when
17
-several Netdata servers are sending data to it, Netdata allows sending metrics at a lower frequency, by resampling them.
11
+- Multiple connector instances running simultaneously
12
+- Different update intervals per connector
13
+- Custom filters per connector instance
14
+- Metric resampling to reduce database congestion
15
19
-So, although Netdata collects metrics every second, it can send to the external database servers averages or sums every
20
-X seconds (though, it can send them per second if you need it to).
16
+:::info
17
22
-## Features
18
+When you enable the exporting engine, Netdata exports metrics **starting from the restart time**, not the entire [historical database](/src/database/README.md).
19
24
-### Integration
20
+:::
21
26
-The exporting engine uses a number of connectors to send Netdata metrics to external time-series databases. See our
27
-[list of supported databases](/docs/exporting-metrics/README.md#supported-databases) for information on which
28
-connector to enable and configure for your database of choice.
22
+## Operation Modes
23
30
-- [**AWS Kinesis Data Streams**](/src/exporting/aws_kinesis/README.md): Metrics are sent to the service in `JSON`
31
- format.
32
-- [**Google Cloud Pub/Sub Service**](/src/exporting/pubsub/README.md): Metrics are sent to the service in `JSON`
33
- format.
34
-- [**Graphite**](/src/exporting/graphite/README.md): A plaintext interface. Metrics are sent to the database server as
35
- `prefix.hostname.chart.dimension`. `prefix` is configured below, `hostname` is the hostname of the machine (can
36
- also be configured). Learn more in our guide to [export and visualize Netdata metrics in
37
- Graphite](/src/exporting/graphite/README.md).
38
-- [**JSON** document databases](/src/exporting/json/README.md)
39
-- [**OpenTSDB**](/src/exporting/opentsdb/README.md): Use a plaintext or HTTP interfaces. Metrics are sent to
40
- OpenTSDB as `prefix.chart.dimension` with tag `host=hostname`.
41
-- [**MongoDB**](/src/exporting/mongodb/README.md): Metrics are sent to the database in `JSON` format.
42
-- [**Prometheus**](/src/exporting/prometheus/README.md): Use an existing Prometheus installation to scrape metrics
43
- from node using the Netdata API.
44
-- [**Prometheus remote write**](/src/exporting/prometheus/remote_write/README.md). A binary snappy-compressed protocol
45
- buffer encoding over HTTP. Supports many [storage
46
- providers](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage).
47
-- [**TimescaleDB**](/src/exporting/TIMESCALE.md): Use a community-built connector that takes JSON streams from a
48
- Netdata client and writes them to a TimescaleDB table.
24
+Netdata provides three data export modes:
25
50
-### Chart filtering
26
+| Mode | Description | Data Format | Use Case |
27
+|:----------------:|:-----------------------------------------|:-----------------------------------------------|:--------------------------------------------------------|
28
+| **as-collected** | Raw metrics in original units | Counters remain counters, gauges remain gauges | Time-series database experts who need raw data |
29
+| **average** | Normalized metrics from Netdata database | All metrics sent as gauges in Netdata units | Simplified visualization with Netdata-centric workflows |
30
+| **sum/volume** | Sum of interpolated values | Aggregated values over the export interval | Long-term trend analysis |
31
52
-Netdata can filter metrics, to send only a subset of the collected metrics. You can use the
53
-configuration file
32
+:::tip
33
55
-```text
56
-[prometheus:exporter]
57
- send charts matching = system.*
58
-```
59
-
60
-or the URL parameter `filter` in the `allmetrics` API call.
61
-
62
-```text
63
-http://localhost:19999/api/v1/allmetrics?format=shell&filter=system.*
64
-```
65
-
66
-### Operation modes
67
-
68
-Netdata supports three modes of operation for all exporting connectors:
69
-
70
-- `as-collected` sends to external databases the metrics as they are collected, in the units they are collected.
71
- So, counters are sent as counters and gauges are sent as gauges, much like all data collectors do. For example,
72
- to calculate CPU utilization in this format, you need to know how to convert kernel ticks to percentage.
73
-
74
-- `average` sends to external databases normalized metrics from the Netdata database. In this mode, all metrics
75
- are sent as gauges, in the units Netdata uses. This abstracts data collection and simplifies visualization, but
76
- you will not be able to copy and paste queries from other sources to convert units. For example, CPU utilization
77
- percentage is calculated by Netdata, so Netdata will convert ticks to percentage and send the average percentage
78
- to the external database.
34
+**Choosing the Right Mode**:
35
80
-- `sum` or `volume`: the sum of the interpolated values shown on the Netdata graphs is sent to the external
81
- database. So, if Netdata is configured to send data to the database every 10 seconds, the sum of the 10 values
82
- shown on the Netdata charts will be used.
36
+- Use `as-collected` if you're building monitoring around a time-series database and know how to convert units
37
+- Use `average` for simpler long-term archiving that matches Netdata's visualization exactly
38
84
-Time-series databases suggest to collect the raw values (`as-collected`). If you plan to invest on building your
85
-monitoring around a time-series database and you already know (or you will invest in learning) how to convert units
86
-and normalize the metrics in Grafana or other visualization tools, we suggest to use `as-collected`.
39
+:::
40
88
-If, on the other hand, you just need long term archiving of Netdata metrics and you plan to mainly work with
89
-Netdata, we suggest to use `average`. It decouples visualization from data collection, so it will generally be a lot
90
-simpler. Furthermore, if you use `average`, the charts shown in the external service will match exactly what you
91
-see in Netdata, which is not necessarily true for the other modes of operation.
41
+## Supported Connectors
42
93
-### Independent operation
43
+| Connector | Protocol/Format | Metric Format |
44
+|:----------------------------------------------------------------------------|:---------------------------|:-----------------------------------|
45
+| [AWS Kinesis](/src/exporting/aws_kinesis/README.md) | JSON | Stream-based |
46
+| [Google Pub/Sub](/src/exporting/pubsub/README.md) | JSON | Message-based |
47
+| [Graphite](/src/exporting/graphite/README.md) | Plaintext | `prefix.hostname.chart.dimension` |
48
+| [JSON Databases](/src/exporting/json/README.md) | JSON | Document-based |
49
+| [OpenTSDB](/src/exporting/opentsdb/README.md) | Plaintext/HTTP | `prefix.chart.dimension` with tags |
50
+| [MongoDB](/src/exporting/mongodb/README.md) | JSON | Document-based |
51
+| [Prometheus](/src/exporting/prometheus/README.md) | HTTP scraping | Prometheus exposition format |
52
+| [Prometheus Remote Write](/src/exporting/prometheus/remote_write/README.md) | Snappy-compressed protobuf | Binary over HTTP |
53
+| [TimescaleDB](/src/exporting/TIMESCALE.md) | JSON streams | Time-series tables |
54
95
-This code is smart enough, not to slow down Netdata, independently of the speed of the external database server.
55
+## Configuration Structure
56
97
-> ❗ You should keep in mind though that many exporting connector instances can consume a lot of CPU resources if they
98
-> run their batches at the same time. You can set different update intervals for every exporting connector instance,
99
-> but even in that case they can occasionally synchronize their batches for a moment.
100
-
101
-## Configuration
102
-
103
-Here are the configuration blocks for every supported connector. Your current `exporting.conf` file may look a little
104
-different.
105
-
106
-You can configure each connector individually using the available [options](#options). The
107
-`[graphite:my_graphite_instance]` block contains examples of some of these additional options in action.
57
+Your `exporting.conf` file contains these configuration blocks:
58
59
```text
60
[exporting:global]
@@ -180,136 +130,149 @@ You can configure each connector individually using the available [options](#opt
130
destination = localhost:8082
131
```
132
183
-### Sections
133
+### Configuration Sections
134
185
-- `[exporting:global]` is a section where you can set your defaults for all exporting connectors
186
-- `[prometheus:exporter]` defines settings for Prometheus exporter API queries (e.g.:
187
- `http://NODE:19999/api/v1/allmetrics?format=prometheus&help=yes&source=as-collected`).
188
-- `[<type>:<name>]` keeps settings for a particular exporting connector instance, where:
189
-- `type` selects the exporting connector type: graphite | opentsdb:telnet | opentsdb:http |
190
- prometheus_remote_write | json | kinesis | pubsub | mongodb. For graphite, opentsdb,
191
- json, and prometheus_remote_write connectors you can also use `:http` or `:https` modifiers
192
- (e.g.: `opentsdb:https`).
193
-- `name` can be arbitrary instance name you chose.
135
+| Section | Purpose |
136
+|:------------------------|:--------------------------------------------|
137
+| `[exporting:global]` | Default settings for all connectors |
138
+| `[prometheus:exporter]` | Prometheus API endpoint settings |
139
+| `[<type>:<name>]` | Individual connector instance configuration |
140
195
-### Options
141
+### Connector Types
142
197
-Configure individual connectors and override any global settings with the following options.
143
+Available connector types with optional modifiers:
144
199
-- `enabled = yes | no`, enables or disables an exporting connector instance
145
+- `graphite` | `graphite:http` | `graphite:https`
146
+- `opentsdb:telnet` | `opentsdb:http` | `opentsdb:https`
147
+- `prometheus_remote_write` | `prometheus_remote_write:http` | `prometheus_remote_write:https`
148
+- `json` | `json:http` | `json:https`
149
+- `kinesis` | `pubsub` | `mongodb`
150
201
-- `destination = host1 host2 host3 ...`, accepts **a space separated list** of hostnames, IPs (IPv4 and IPv6) and
202
- ports to connect to. Netdata will use the **first available** to send the metrics.
151
+## Configuration Options
152
204
- The format of each item in this list, is: `[PROTOCOL:]IP[:PORT]`.
153
+### Basic Settings
154
206
- `PROTOCOL` can be `udp` or `tcp`. `tcp` is the default and only supported by the current exporting engine.
155
+| Option | Values | Description |
156
+|:---------------|:-------------------------|:--------------------------------------------------------------|
157
+| `enabled` | yes/no | Activates the connector instance |
158
+| `data source` | as-collected/average/sum | Selects data export mode |
159
+| `hostname` | string | Hostname for external database (default: `[global].hostname`) |
160
+| `prefix` | string | Prefix added to all metrics |
161
+| `update every` | seconds | Export interval with automatic randomization |
162
208
- `IP` can be `XX.XX.XX.XX` (IPv4), or `[XX:XX...XX:XX]` (IPv6). For IPv6 you can to enclose the IP in `[]` to
209
- separate it from the port.
163
+### Connection Settings
164
211
- `PORT` can be a number of a service name. If omitted, the default port for the exporting connector will be used
212
- (graphite = 2003, opentsdb = 4242).
165
+| Option | Format | Description |
166
+|:---------------------|:---------------------|:--------------------------------------------------------|
167
+| `destination` | space-separated list | Target servers in `[PROTOCOL:]IP[:PORT]` format |
168
+| `buffer on failures` | iterations | Buffer size when database unavailable |
169
+| `timeout ms` | milliseconds | Processing timeout (default: `2 * update_every * 1000`) |
170
214
- Example IPv4:
171
+#### Destination Examples
172
+
173
+IPv4 configuration:
174
175
```text
217
- destination = 10.11.14.2:4242 10.11.14.3:4242 10.11.14.4:4242
176
+destination = 10.11.14.2:4242 10.11.14.3:4242 10.11.14.4:4242
177
```
178
220
- Example IPv6 and IPv4 together:
179
+IPv6 and IPv4 combined:
180
181
```text
223
- destination = [ffff:...:0001]:2003 10.11.12.1:2003
182
+destination = [ffff:...:0001]:2003 10.11.12.1:2003
183
```
184
226
- When multiple servers are defined, Netdata will try the next one when the previous one fails.
185
+Special destinations:
186
+
187
+- **Kinesis**: AWS region (e.g., `us-east-1`)
188
+- **MongoDB**: [MongoDB URI](https://docs.mongodb.com/manual/reference/connection-string/)
189
+- **Pub/Sub**: Service endpoint
190
+
191
+### Filtering Options
192
+
193
+| Option | Pattern Format | Description |
194
+|:-----------------------|:-------------------------|:--------------------------------------------------|
195
+| `send hosts matching` | space-separated patterns | Filter hosts using `*` wildcard, `!` for negation |
196
+| `send charts matching` | space-separated patterns | Filter charts by ID/name, `!` for negation |
197
+
198
+:::important
199
+
200
+Pattern matching follows first-match logic. Order matters when using negative patterns (`!`).
201
+
202
+Example: `!*child* *db*` matches all `*db*` hosts except those containing `*child*`.
203
228
- Netdata also ships `nc-exporting.sh`, a script that can be used as a fallback exporting connector to save the
229
- metrics to disk and push them to the time-series database when it becomes available again. It can also be used to
230
- monitor / trace / debug the metrics Netdata generates.
204
+:::
205
232
- For the Kinesis exporting connector `destination` should be set to an AWS region (for example, `us-east-1`).
206
+### Label Settings
207
234
- For the MongoDB exporting connector `destination` should be set to a
235
- [MongoDB URI](https://docs.mongodb.com/manual/reference/connection-string/).
208
+| Option | Values | Description |
209
+|:----------------------------|:-------|:------------------------------------------------------------|
210
+| `send names instead of ids` | yes/no | Use human-friendly names vs system IDs |
211
+| `send configured labels` | yes/no | Include `[host labels]` from `netdata.conf` |
212
+| `send automatic labels` | yes/no | Include auto-generated labels (`_os_name`, `_architecture`) |
213
237
- For the Pub/Sub exporting connector `destination` can be set to a specific service endpoint.
214
+## Chart Filtering
215
239
-- `data source = as collected`, or `data source = average`, or `data source = sum`, selects the kind of data that will
240
- be sent to the external database.
216
+Filter metrics through two methods:
217
242
-- `hostname = my-name`, is the hostname to be used for sending data to the external database server. By default this
243
- is `[global].hostname`.
218
+1. **Configuration file**:
219
245
-- `prefix = Netdata`, is the prefix to add to all metrics.
220
+ ```text
221
+ [prometheus:exporter]
222
+ send charts matching = system.*
223
+ ```
224
247
-- `update every = 10`, is the number of seconds between sending data to the external database. Netdata will add some
248
- randomness to this number, to prevent stressing the external server when many Netdata servers send data to the same
249
- database. This randomness does not affect the quality of the data, only the time they are sent.
225
+2. **URL parameter**:
226
251
-- `buffer on failures = 10`, is the number of iterations (each iteration is `update every` seconds) to buffer data,
252
- when the external database server is not available. If the server fails to receive the data after that many
253
- failures, data loss on the connector instance is expected (Netdata will also log it).
227
+ ```text
228
+ http://localhost:19999/api/v1/allmetrics?format=shell&filter=system.*
229
+ ```
230
255
-- `timeout ms = 20000`, is the timeout in milliseconds to wait for the external database server to process the data.
256
- By default this is `2 * update_every * 1000`.
231
+## HTTPS Support
232
258
-- `send hosts matching = localhost *` includes one or more space separated patterns, using `*` as wildcard (any number
259
- of times within each pattern). The patterns are checked against the hostname (the localhost is always checked as
260
- `localhost`), allowing us to filter which hosts will be sent to the external database when this Netdata is a central
261
- Netdata aggregating multiple hosts. A pattern starting with `!` gives a negative match. So to match all hosts named
262
- `*db*` except hosts containing `*child*`, use `!*child* *db*` (so, the order is important: the first
263
- pattern matching the hostname will be used - positive or negative).
233
+For databases without native TLS/SSL support, configure a reverse proxy:
234
265
-- `send charts matching = *` includes one or more space separated patterns, using `*` as wildcard (any number of times
266
- within each pattern). The patterns are checked against both chart id and chart name. A pattern starting with `!`
267
- gives a negative match. So to match all charts named `apps.*` except charts ending in `*reads`, use `!*reads
268
- apps.*` (so, the order is important: the first pattern matching the chart id or the chart name will be used -
269
- positive or negative). There is also a URL parameter `filter` that can be used while querying `allmetrics`. The URL
270
- parameter has a higher priority than the configuration option.
235
+- [Nginx reverse proxy setup](/docs/netdata-agent/configuration/running-the-netdata-agent-behind-a-reverse-proxy/Running-behind-nginx.md)
236
272
-- `send names instead of ids = yes | no` controls the metric names Netdata should send to the external database.
273
- Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system
274
- and names are human friendly labels (also unique). Most charts and metrics have the same ID and name, but in several
275
- cases they are different: disks with device-mapper, interrupts, QoS classes, statsd synthetic charts, etc.
237
+## Performance Considerations
238
277
-- `send configured labels = yes | no` controls if host labels defined in the `[host labels]` section in `netdata.conf`
278
- should be sent to the external database
239
+The exporting engine operates independently to avoid slowing down Netdata. However:
240
280
-- `send automatic labels = yes | no` controls if automatically created labels, like `_os_name` or `_architecture`
281
- should be sent to the external database
241
+:::warning
242
283
-## HTTPS
243
+Multiple connector instances running batches simultaneously can consume significant CPU resources. Configure different update intervals to prevent synchronization.
244
285
-Netdata can send metrics to external databases using the TLS/SSL protocol. Unfortunately, some of
286
-them does not support encrypted connections, so you will have to configure a reverse proxy to enable
287
-HTTPS communication between Netdata and an external database. You can set up a reverse proxy with
288
-[Nginx](/docs/netdata-agent/configuration/running-the-netdata-agent-behind-a-reverse-proxy/Running-behind-nginx.md).
245
+:::
246
290
-## Exporting engine monitoring
247
+## Monitoring the Exporting Engine
248
292
-Netdata creates five charts in the dashboard, under the **Netdata Monitoring** section, to help you monitor the health
293
-and performance of the exporting engine itself:
249
+Netdata provides five monitoring charts under **Netdata Monitoring**:
250
295
-1. **Buffered metrics**, the number of metrics Netdata added to the buffer for dispatching them to the
296
- external database server.
251
+| Chart | Monitors |
252
+|:-------------------------------|:-------------------------------------------|
253
+| **Buffered metrics** | Number of metrics added to dispatch buffer |
254
+| **Exporting data size** | Data volume (KB) added to buffer |
255
+| **Exporting operations** | Operation count performed |
256
+| **Exporting thread CPU usage** | CPU resources consumed by exporting thread |
257
298
-2. **Exporting data size**, the amount of data (in KB) Netdata added the buffer.
258
+
259
300
-3. **Exporting operations**, the number of operations performed by Netdata.
260
+## Built-in Alerts
261
302
-4. **Exporting thread CPU usage**, the CPU resources consumed by the Netdata thread, that is responsible for sending
303
- the metrics to the external database server.
262
+The exporting engine includes three automatic alerts:
263
305
-
264
+| Alert | Monitors |
265
+|:---------------------------|:----------------------------------------|
266
+| `exporting_last_buffering` | Seconds since last successful buffering |
267
+| `exporting_metrics_sent` | Percentage of successfully sent metrics |
268
+| `exporting_metrics_lost` | Metrics lost due to repeated failures |
269
307
-## Exporting engine alerts
270
+
271
309
-Netdata adds 3 alerts:
272
+## Fallback Script
273
311
-1. `exporting_last_buffering`, number of seconds since the last successful buffering of exported data
312
-2. `exporting_metrics_sent`, percentage of metrics sent to the external database server
313
-3. `exporting_metrics_lost`, number of metrics lost due to repeating failures to contact the external database server
274
+Netdata includes `nc-exporting.sh` for:
275
315
-
276
+- Saving metrics to disk during database outages
277
+- Pushing cached metrics when database recovers
278
+- Monitoring/tracing/debugging metric generation
src/exporting/prometheus/README.md
+119
-132
@@ -1,99 +1,95 @@
1
# Using Netdata with Prometheus
2
3
-Netdata supports exporting metrics to Prometheus in two ways:
3
+Netdata exports metrics to Prometheus through two methods:
4
5
- - You can [configure Prometheus to scrape Netdata metrics](#configure-prometheus-to-scrape-netdata-metrics).
5
+- **[Configure Prometheus to scrape Netdata metrics](#configure-prometheus-to-scrape-netdata-metrics)** - Pull metrics from Netdata
6
+- **[Configure Netdata to push metrics to Prometheus](/src/exporting/prometheus/remote_write/README.md)** - Push using remote write API
7
7
- - You can [configure Netdata to push metrics to Prometheus](/src/exporting/prometheus/remote_write/README.md)
8
- , using the Prometheus remote write API.
8
+## Netdata Support for Prometheus
9
10
-## Netdata support for Prometheus
10
+Before configuring either method, understand how Netdata structures its exported metrics and available capabilities. These concepts apply to both scraping and remote write methods.
11
12
-Regardless of the methodology, you first need to understand how Netdata structures the metrics it exports to Prometheus
13
-and the capabilities it provides. The examples provided in this document assume that you will be using Netdata as
14
-a metrics endpoint, but the concepts apply as well to the remote write API method.
15
-
16
-### Understanding Netdata metrics
12
+### Understanding Netdata Metrics
13
14
#### Charts
15
20
-Each chart in Netdata has several properties (common to all its metrics):
16
+Each Netdata chart has several properties common to all its metrics:
17
22
-- `chart_id` - uniquely identifies a chart.
18
+| Property | Description |
19
+|:-------------|:-----------------------------------------------------------------------------------------------------------------------------------------------|
20
+| `chart_id` | Uniquely identifies a chart |
21
+| `chart_name` | Human-friendly name for `chart_id`, also unique |
22
+| `context` | Chart template - all disk I/O charts share the same context, all MySQL request charts share the same context. Used for alert template matching |
23
+| `family` | Groups charts together as dashboard submenus |
24
+| `units` | Units for all metrics in the chart |
25
24
-- `chart_name` - a more human friendly name for `chart_id`, also unique.
26
+#### Dimensions
27
26
-- `context` - this is the template of the chart. All disk I/O charts have the same context, all mysql requests charts
27
- have the same context, etc. This is used for alert templates to match all the charts they should be attached to.
28
+Each Netdata chart contains metrics called `dimensions`. All dimensions in a chart:
29
29
-- `family` groups a set of charts together. It is used as the submenu of the dashboard.
30
+- Share the same units of measurement
31
+- Belong to the same contextual category (e.g., disk bandwidth contains `read` and `write` dimensions)
32
31
-- `units` is the units for all the metrics attached to the chart.
33
+### Netdata Data Source
34
33
-#### Dimensions
35
+Netdata sends metrics to Prometheus from 3 data sources:
36
35
-Then each Netdata chart contains metrics called `dimensions`. All the dimensions of a chart have the same units of
36
-measurement, and are contextually in the same category (ie. the metrics for disk bandwidth are `read` and `write` and
37
-they are both in the same chart).
37
+#### 1. As-Collected (Raw)
38
39
-### Netdata data source
39
+Sends metrics exactly as collected without conversion. Prometheus prefers this method, but it requires understanding how to extract meaningful values.
40
41
-Netdata can send metrics to Prometheus from 3 data sources:
41
+**Metric formats:**
42
43
-- `as collected` or `raw` - this data source sends the metrics to Prometheus as they are collected. No conversion is
44
- done by Netdata. The latest value for each metric is just given to Prometheus. This is the most preferred method by
45
- Prometheus, but it is also the harder to work with. To work with this data source, you will need to understand how
46
- to get meaningful values out of them.
43
+- Standard: `CONTEXT{chart="CHART",family="FAMILY",dimension="DIMENSION"}`
44
+- Counters: `CONTEXT_total{chart="CHART",family="FAMILY",dimension="DIMENSION"}`
45
+- Heterogeneous dimensions: `CONTEXT_DIMENSION{chart="CHART",family="FAMILY"}`
46
48
- The format of the metrics is: `CONTEXT{chart="CHART",family="FAMILY",dimension="DIMENSION"}`.
47
+:::info
48
50
- If the metric is a counter (`incremental` in Netdata lingo), `_total` is appended the context.
49
+Unlike Prometheus, Netdata allows each dimension to have different algorithms and conversion constants (`multiplier` and `divisor`). When dimensions are heterogeneous, Netdata uses the `CONTEXT_DIMENSION` format.
50
52
- Unlike Prometheus, Netdata allows each dimension of a chart to have a different algorithm and conversion constants
53
- (`multiplier` and `divisor`). In this case, that the dimensions of a charts are heterogeneous, Netdata will use this
54
- format: `CONTEXT_DIMENSION{chart="CHART",family="FAMILY"}`
51
+:::
52
56
-- `average` - this data source uses the Netdata database to send the metrics to Prometheus as they are presented on
57
- the Netdata dashboard. So, all the metrics are sent as gauges, at the units they are presented in the Netdata
58
- dashboard charts. This is the easiest to work with.
53
+#### 2. Average
54
60
- The format of the metrics is: `CONTEXT_UNITS_average{chart="CHART",family="FAMILY",dimension="DIMENSION"}`.
55
+Sends metrics as they appear on the dashboard. All metrics become gauges in their dashboard units. This is the easiest to work with.
56
62
- When this source is used, Netdata keeps track of the last access time for each Prometheus server fetching the
63
- metrics. This last access time is used at the subsequent queries of the same Prometheus server to identify the
64
- time-frame the `average` will be calculated.
57
+**Format:** `CONTEXT_UNITS_average{chart="CHART",family="FAMILY",dimension="DIMENSION"}`
58
66
- So, no matter how frequently Prometheus scrapes Netdata, it will get all the database data.
67
- To identify each Prometheus server, Netdata uses by default the IP of the client fetching the metrics.
59
+Netdata tracks each Prometheus server's last access time to calculate averages for the time-frame between queries. This ensures no data loss regardless of scrape frequency. By default, Netdata identifies servers by client IP. For multiple servers using the same IP, append `server=NAME` to the URL for unique identification.
60
69
- If there are multiple Prometheus servers fetching data from the same Netdata, using the same IP, each Prometheus
70
- server can append `server=NAME` to the URL. Netdata will use this `NAME` to uniquely identify the Prometheus server.
61
+#### 3. Sum (Volume)
62
72
-- `sum` or `volume`, is like `average` but instead of averaging the values, it sums them.
63
+Like `average` but sums values instead of averaging them.
64
74
- The format of the metrics is: `CONTEXT_UNITS_sum{chart="CHART",family="FAMILY",dimension="DIMENSION"}`. All the
75
- other operations are the same with `average`.
65
+**Format:** `CONTEXT_UNITS_sum{chart="CHART",family="FAMILY",dimension="DIMENSION"}`
66
77
- To change the data source to `sum` or `as-collected` you need to provide the `source` parameter in the request URL.
78
- e.g.: `http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes&source=as-collected`
67
+To change the data source, add the `source` parameter to the URL:
68
80
- Keep in mind that early versions of Netdata were sending the metrics as: `CHART_DIMENSION{}`.
69
+```
70
+http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&source=as-collected
71
+```
72
82
-### Querying Metrics
73
+:::info
74
84
-Fetch with your web browser this URL:
75
+Early Netdata versions sent metrics as `CHART_DIMENSION{}`.
76
86
-`http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes`
77
+:::
78
88
-_(replace `your.netdata.ip` with the ip or hostname of your Netdata server)_
79
+### Querying Metrics
80
+
81
+Test the metrics endpoint in your browser:
82
+
83
+```
84
+http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes
85
+```
86
90
-Netdata will respond with all the metrics it sends to Prometheus.
87
+Replace `your.netdata.ip` with your Netdata server's IP or hostname.
88
92
-If you search that page for `"system.cpu"` you will find all the metrics Netdata is exporting to Prometheus for this
93
-chart. `system.cpu` is the chart name on the Netdata dashboard (on the Netdata dashboard all charts have a text heading
94
-such as : `Total CPU utilization (system.cpu)`. What we are interested here in the chart name: `system.cpu`).
89
+Netdata responds with all metrics it sends to Prometheus. Search for `"system.cpu"` to find all CPU metrics (the chart name from the dashboard heading "Total CPU utilization (system.cpu)").
90
96
-Searching for `"system.cpu"` reveals:
91
+<details>
92
+<summary><strong>Example: system.cpu with average source</strong></summary>
93
94
```sh
95
# COMMENT homogeneous chart "system.cpu", context "system.cpu", family "cpu", units "percentage"
@@ -119,13 +115,12 @@ netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension=
115
netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="idle"} 92.3630770 1500066662000
116
```
117
122
-_(Netdata response for `system.cpu` with source=`average`)_
118
+</details>
119
124
-In `average` or `sum` data sources, all values are normalized and are reported to Prometheus as gauges. Now, use the
125
-'expression' text form in Prometheus. Begin to type the metrics we are looking for: `netdata_system_cpu`. You should see
126
-that the text form begins to auto-fill as Prometheus knows about this metric.
120
+In `average` or `sum` sources, all values are normalized and reported as gauges. Type `netdata_system_cpu` in the Prometheus expression field - it auto-completes as Prometheus recognizes the metric.
121
128
-If the data source was `as collected`, the response would be:
122
+<details>
123
+<summary><strong>Example: system.cpu with as-collected source</strong></summary>
124
125
```sh
126
# COMMENT homogeneous chart "system.cpu", context "system.cpu", family "cpu", units "percentage"
@@ -151,121 +146,112 @@ netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="iowait"} 233
146
netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="idle"} 918470 1500066716438
147
```
148
154
-_(Netdata response for `system.cpu` with source=`as-collected`)_
149
+</details>
150
156
-For more information check Prometheus documentation.
151
+For more information, check Prometheus documentation.
152
158
-### Streaming data from upstream hosts
153
+### Streaming Data from Upstream Hosts
154
160
-The `format=prometheus` parameter only exports the host's Netdata metrics. If you are using the parent-child
161
-functionality of Netdata this ignores any upstream hosts - so you should consider using the below in your
162
-**prometheus.yml**:
155
+The `format=prometheus` parameter only exports the local host's metrics. For parent-child Netdata setups, use this configuration in **prometheus.yml**:
156
157
```yaml
165
- metrics_path: '/api/v1/allmetrics'
166
- params:
167
- format: [ prometheus_all_hosts ]
168
- honor_labels: true
158
+metrics_path: '/api/v1/allmetrics'
159
+params:
160
+ format: [prometheus_all_hosts]
161
+honor_labels: true
162
```
163
171
-This will report all upstream host data, and `honor_labels` will make Prometheus take note of the instance names
172
-provided.
164
+This reports all upstream host data with proper instance names.
165
166
### Timestamps
167
176
-To pass the metrics through Prometheus pushgateway, Netdata supports the option `×tamps=no` to send the metrics
177
-without timestamps.
168
+To pass metrics through Prometheus pushgateway, use `×tamps=no` to send metrics without timestamps.
169
179
-## Netdata host variables
170
+## Netdata Host Variables
171
181
-Netdata collects various system configuration metrics, like the max number of TCP sockets supported, the max number of
182
-files allowed system-wide, various IPC sizes, etc. These metrics are not exposed to Prometheus by default.
172
+Netdata collects system configuration metrics (max TCP sockets, system-wide file limits, IPC sizes, etc.) not exposed to Prometheus by default.
173
184
-To expose them, append `variables=yes` to the Netdata URL.
174
+To expose them, append `variables=yes` to the URL.
175
176
### TYPE and HELP
177
188
-To save bandwidth, and because Prometheus does not use them anyway, `# TYPE` and `# HELP` lines are suppressed. If
189
-wanted they can be re-enabled via `types=yes` and `help=yes`, e.g.
190
-`/api/v1/allmetrics?format=prometheus&types=yes&help=yes`
178
+`# TYPE` and `# HELP` lines are suppressed by default to save bandwidth (Prometheus doesn't use them). Re-enable with:
179
+
180
+```
181
+/api/v1/allmetrics?format=prometheus&types=yes&help=yes
182
+```
183
192
-Note that if enabled, the `# TYPE` and `# HELP` lines are repeated for every occurrence of a metric, which goes against
193
-the Prometheus
194
-documentation's [specification for these lines](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#comments-help-text-and-type-information).
184
+:::warning
185
196
-### Names and IDs
186
+When enabled, `# TYPE` and `# HELP` lines repeat for every metric occurrence, violating [Prometheus specifications](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#comments-help-text-and-type-information).
187
198
-Netdata supports names and IDs for charts and dimensions. Usually IDs are unique identifiers as read by the system and
199
-names are human friendly labels (also unique).
188
+:::
189
201
-Most charts and metrics have the same ID and name, but in several cases they are different: disks with device-mapper,
202
-interrupts, QoS classes, statsd synthetic charts, etc.
190
+### Names and IDs
191
+
192
+Netdata supports both names and IDs for charts and dimensions. IDs are unique system identifiers; names are human-friendly labels (also unique). Most charts have identical ID and name, but some differ (device-mapper disks, interrupts, QoS classes, statsd synthetic charts).
193
204
-The default is controlled in `exporting.conf`:
194
+Control the default in `exporting.conf`:
195
196
```text
197
[prometheus:exporter]
198
send names instead of ids = yes | no
199
```
200
211
-You can overwrite it from Prometheus, by appending to the URL:
201
+Override via URL:
202
213
-- `&names=no` to get IDs (the old behaviour)
214
-- `&names=yes` to get names
203
+- `&names=no` for IDs (old behavior)
204
+- `&names=yes` for names
205
216
-### Filtering metrics sent to Prometheus
206
+### Filtering Metrics Sent to Prometheus
207
218
-Netdata can filter the metrics it sends to Prometheus with this setting:
208
+Filter metrics with this setting:
209
210
```text
211
[prometheus:exporter]
212
send charts matching = *
213
```
214
225
-This settings accepts a space separated list
226
-of [simple patterns](/src/libnetdata/simple_pattern/README.md) to match the
227
-**charts** to be sent to Prometheus. Each pattern can use `*` as wildcard, any number of times (e.g `*a*b*c*` is valid).
228
-Patterns starting with `!` give a negative match (e.g `!*.bad users.* groups.*` will send all the users and groups
229
-except `bad` user and `bad` group). The order is important: the first match (positive or negative) left to right, is
230
-used.
215
+This accepts space-separated [simple patterns](/src/libnetdata/simple_pattern/README.md) to match charts. Pattern rules:
216
+
217
+- `*` as wildcard (e.g., `*a*b*c*` is valid)
218
+- `!` prefix for negative match
219
+- First match (positive or negative) wins
220
+- Example: `!*.bad users.* groups.*` sends all users and groups except `bad` ones
221
232
-### Changing the prefix of Netdata metrics
222
+### Changing the Prefix of Netdata Metrics
223
234
-Netdata sends all metrics prefixed with `netdata_`. You can change this in `netdata.conf`, like this:
224
+Netdata prefixes all metrics with `netdata_`. Change in `netdata.conf`:
225
226
```text
227
[prometheus:exporter]
228
prefix = netdata
229
```
230
241
-It can also be changed from the URL, by appending `&prefix=netdata`.
231
+Or append `&prefix=netdata` to the URL.
232
233
### Metric Units
234
245
-The default source `average` adds the unit of measurement to the name of each metric (e.g. `_KiB_persec`). To hide the
246
-units and get the same metric names as with the other sources, append to the URL `&hideunits=yes`.
235
+| Source | Unit Behavior | Control |
236
+|:--------------------|:------------------------------------------|:------------------------------------|
237
+| `average` (default) | Adds units to names (e.g., `_KiB_persec`) | `&hideunits=yes` to hide |
238
+| `as-collected` | No units in names | N/A |
239
+| All sources | v1.12+ standardized units | `&oldunits=yes` for pre-v1.12 names |
240
248
-The units were standardized in v1.12, with the effect of changing the metric names. To get the metric names as they were
249
-before v1.12, append to the URL `&oldunits=yes`
241
+### Accuracy of Average and Sum Data Sources
242
251
-### Accuracy of `average` and `sum` data sources
243
+When using `average` or `sum` sources, Netdata remembers each client's last access time to calculate values for the period since last access. This prevents data loss regardless of scrape frequency.
244
253
-When the data source is set to `average` or `sum`, Netdata remembers the last access of each client accessing Prometheus
254
-metrics and uses this last access time to respond with the `average` or `sum` of all the entries in the database since
255
-that. This means that Prometheus servers are not losing data when they access Netdata with data source = `average` or
256
-`sum`.
245
+Server identification:
246
+| Scenario | Method |
247
+|:---------|:-------|
248
+| Direct connection | Client IP (default) |
249
+| Behind proxy or NAT | Append `&server=NAME` to URL |
250
+| Multiple servers, same IP | Each uses unique `&server=NAME` |
251
258
-To uniquely identify each Prometheus server, Netdata uses the IP of the client accessing the metrics. If however the IP
259
-is not good enough for identifying a single Prometheus server (e.g. when Prometheus servers are accessing Netdata
260
-through a web proxy, or when multiple Prometheus servers are NATed to a single IP), each Prometheus may append
261
-`&server=NAME` to the URL. This `NAME` is used by Netdata to uniquely identify each Prometheus server and keep track of
262
-its last access time.
252
+## Configure Prometheus to Scrape Netdata Metrics
253
264
-## Configure Prometheus to scrape Netdata metrics
265
-
266
-The following `prometheus.yml` file will scrape all netdata metrics "as collected".
267
-
268
-Make sure to replace `your.netdata.ip` with the IP or hostname of the host running Netdata.
254
+The following `prometheus.yml` scrapes all Netdata metrics "as collected":
255
256
```yaml
257
# my global config
@@ -294,7 +280,7 @@ scrape_configs:
280
# scheme defaults to 'http'.
281
282
static_configs:
297
- - targets: [ '0.0.0.0:9090' ]
283
+ - targets: ['0.0.0.0:9090']
284
285
- job_name: 'netdata-scrape'
286
@@ -302,7 +288,7 @@ scrape_configs:
288
params:
289
# format: prometheus | prometheus_all_hosts
290
# You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP
305
- format: [ prometheus ]
291
+ format: [prometheus]
292
#
293
# sources: as-collected | raw | average | sum | volume
294
# default is: average
@@ -314,13 +300,14 @@ scrape_configs:
300
honor_labels: true
301
302
static_configs:
317
- - targets: [ '{your.netdata.ip}:19999' ]
303
+ - targets: ['{your.netdata.ip}:19999']
304
```
305
320
-### Prometheus alerts for Netdata metrics
306
+Replace `{your.netdata.ip}` with your Netdata host's IP or hostname.
307
+
308
+### Prometheus Alerts for Netdata Metrics
309
322
-The following is an example of a `nodes.yml` file that will allow Prometheus to generate alerts from some Netdata sources.
323
-Save it at `/opt/prometheus/nodes.yml`, and add a _- "nodes.yml"_ entry under the _rule_files:_ section in the example prometheus.yml file above.
310
+Example `nodes.yml` file for generating alerts from Netdata metrics. Save at `/opt/prometheus/nodes.yml` and add `- "nodes.yml"` under `rule_files:` in prometheus.yml:
311
312
```yaml
313
groups: