| 1 | # Using Netdata with Prometheus |
| 2 | |
| 3 | Netdata exports metrics to Prometheus through two methods: |
| 4 | |
| 5 | - **[Configure Prometheus to scrape Netdata metrics](#configure-prometheus-to-scrape-netdata-metrics)** - Pull metrics from Netdata |
| 6 | - **[Configure Netdata to push metrics to Prometheus](/src/exporting/prometheus/remote_write/README.md)** - Push using remote write API |
| 7 | |
| 8 | ## Netdata Support for Prometheus |
| 9 | |
| 10 | Before configuring either method, understand how Netdata structures its exported metrics and available capabilities. These concepts apply to both scraping and remote write methods. |
| 11 | |
| 12 | ### Understanding Netdata Metrics |
| 13 | |
| 14 | #### Charts |
| 15 | |
| 16 | Each Netdata chart has several properties common to all its metrics: |
| 17 | |
| 18 | | Property | Description | |
| 19 | |:-------------|:-----------------------------------------------------------------------------------------------------------------------------------------------| |
| 20 | | `chart_id` | Uniquely identifies a chart | |
| 21 | | `chart_name` | Human-friendly name for `chart_id`, also unique | |
| 22 | | `context` | Chart template - all disk I/O charts share the same context, all MySQL request charts share the same context. Used for alert template matching | |
| 23 | | `family` | Groups charts together as dashboard submenus | |
| 24 | | `units` | Units for all metrics in the chart | |
| 25 | |
| 26 | #### Dimensions |
| 27 | |
| 28 | Each Netdata chart contains metrics called `dimensions`. All dimensions in a chart: |
| 29 | |
| 30 | - Share the same units of measurement |
| 31 | - Belong to the same contextual category (e.g., disk bandwidth contains `read` and `write` dimensions) |
| 32 | |
| 33 | #### Negative Values in Exported Metrics |
| 34 | |
| 35 | Some Netdata dimensions use a `-1` multiplier by design so that read and write appear as opposing values on the dashboard. This multiplier is applied in the RRD layer during collection and is reflected in all Prometheus data sources (`as-collected`, `average`, and `sum`), causing those dimensions to export as negative values. You can identify affected charts from the dashboard — they display read and write as stacked opposing areas. |
| 36 | |
| 37 | :::note |
| 38 | |
| 39 | To get absolute values in PromQL, use `abs()`. For example: `abs(netdata_cgroup_io_KiB_persec_average{dimension="write"})`. |
| 40 | |
| 41 | ::: |
| 42 | |
| 43 | ### Netdata Data Source |
| 44 | |
| 45 | Netdata sends metrics to Prometheus from 3 data sources: |
| 46 | |
| 47 | #### 1. As-Collected (Raw) |
| 48 | |
| 49 | Sends metrics exactly as collected without conversion. Prometheus prefers this method, but it requires understanding how to extract meaningful values. |
| 50 | |
| 51 | **Metric formats:** |
| 52 | |
| 53 | - Standard: `CONTEXT{chart="CHART",family="FAMILY",dimension="DIMENSION"}` |
| 54 | - Counters: `CONTEXT_total{chart="CHART",family="FAMILY",dimension="DIMENSION"}` |
| 55 | - Heterogeneous dimensions: `CONTEXT_DIMENSION{chart="CHART",family="FAMILY"}` |
| 56 | |
| 57 | :::info |
| 58 | |
| 59 | Unlike Prometheus, Netdata allows each dimension to have different algorithms and conversion constants (`multiplier` and `divisor`). When dimensions are heterogeneous, Netdata uses the `CONTEXT_DIMENSION` format. |
| 60 | |
| 61 | ::: |
| 62 | |
| 63 | #### 2. Average |
| 64 | |
| 65 | Sends metrics as they appear on the dashboard. All metrics become gauges in their dashboard units. This is the easiest to work with. |
| 66 | |
| 67 | **Format:** `CONTEXT_UNITS_average{chart="CHART",family="FAMILY",dimension="DIMENSION"}` |
| 68 | |
| 69 | Netdata tracks each Prometheus server's last access time to calculate averages for the time-frame between queries. This ensures no data loss regardless of scrape frequency. By default, Netdata identifies servers by client IP. For multiple servers using the same IP, append `server=NAME` to the URL for unique identification. |
| 70 | |
| 71 | #### 3. Sum (Volume) |
| 72 | |
| 73 | Like `average` but sums values instead of averaging them. |
| 74 | |
| 75 | **Format:** `CONTEXT_UNITS_sum{chart="CHART",family="FAMILY",dimension="DIMENSION"}` |
| 76 | |
| 77 | To change the data source, add the `source` parameter to the URL: |
| 78 | |
| 79 | ``` |
| 80 | http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&source=as-collected |
| 81 | ``` |
| 82 | |
| 83 | :::info |
| 84 | |
| 85 | Early Netdata versions sent metrics as `CHART_DIMENSION{}`. |
| 86 | |
| 87 | ::: |
| 88 | |
| 89 | ### Querying Metrics |
| 90 | |
| 91 | Test the metrics endpoint in your browser: |
| 92 | |
| 93 | ``` |
| 94 | http://your.netdata.ip:19999/api/v1/allmetrics?format=prometheus&help=yes |
| 95 | ``` |
| 96 | |
| 97 | Replace `your.netdata.ip` with your Netdata server's IP or hostname. |
| 98 | |
| 99 | Netdata responds with all metrics it sends to Prometheus. Search for `"system.cpu"` to find all CPU metrics (the chart name from the dashboard heading "Total CPU utilization (system.cpu)"). |
| 100 | |
| 101 | <details> |
| 102 | <summary><strong>Example: system.cpu with average source</strong></summary> |
| 103 | |
| 104 | ```sh |
| 105 | # COMMENT homogeneous chart "system.cpu", context "system.cpu", family "cpu", units "percentage" |
| 106 | # COMMENT netdata_system_cpu_percentage_average: dimension "guest_nice", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 107 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="guest_nice"} 0.0000000 1500066662000 |
| 108 | # COMMENT netdata_system_cpu_percentage_average: dimension "guest", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 109 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="guest"} 1.7837326 1500066662000 |
| 110 | # COMMENT netdata_system_cpu_percentage_average: dimension "steal", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 111 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="steal"} 0.0000000 1500066662000 |
| 112 | # COMMENT netdata_system_cpu_percentage_average: dimension "softirq", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 113 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="softirq"} 0.5275442 1500066662000 |
| 114 | # COMMENT netdata_system_cpu_percentage_average: dimension "irq", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 115 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="irq"} 0.2260836 1500066662000 |
| 116 | # COMMENT netdata_system_cpu_percentage_average: dimension "user", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 117 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="user"} 2.3362762 1500066662000 |
| 118 | # COMMENT netdata_system_cpu_percentage_average: dimension "system", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 119 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="system"} 1.7961062 1500066662000 |
| 120 | # COMMENT netdata_system_cpu_percentage_average: dimension "nice", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 121 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="nice"} 0.0000000 1500066662000 |
| 122 | # COMMENT netdata_system_cpu_percentage_average: dimension "iowait", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 123 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="iowait"} 0.9671802 1500066662000 |
| 124 | # COMMENT netdata_system_cpu_percentage_average: dimension "idle", value is percentage, gauge, dt 1500066653 to 1500066662 inclusive |
| 125 | netdata_system_cpu_percentage_average{chart="system.cpu",family="cpu",dimension="idle"} 92.3630770 1500066662000 |
| 126 | ``` |
| 127 | |
| 128 | </details> |
| 129 | |
| 130 | In `average` or `sum` sources, all values are normalized and reported as gauges. Type `netdata_system_cpu` in the Prometheus expression field - it auto-completes as Prometheus recognizes the metric. |
| 131 | |
| 132 | <details> |
| 133 | <summary><strong>Example: system.cpu with as-collected source</strong></summary> |
| 134 | |
| 135 | ```sh |
| 136 | # COMMENT homogeneous chart "system.cpu", context "system.cpu", family "cpu", units "percentage" |
| 137 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "guest_nice", value * 1 / 1 delta gives percentage (counter) |
| 138 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="guest_nice"} 0 1500066716438 |
| 139 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "guest", value * 1 / 1 delta gives percentage (counter) |
| 140 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="guest"} 63945 1500066716438 |
| 141 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "steal", value * 1 / 1 delta gives percentage (counter) |
| 142 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="steal"} 0 1500066716438 |
| 143 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "softirq", value * 1 / 1 delta gives percentage (counter) |
| 144 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="softirq"} 8295 1500066716438 |
| 145 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "irq", value * 1 / 1 delta gives percentage (counter) |
| 146 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="irq"} 4079 1500066716438 |
| 147 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "user", value * 1 / 1 delta gives percentage (counter) |
| 148 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="user"} 116488 1500066716438 |
| 149 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "system", value * 1 / 1 delta gives percentage (counter) |
| 150 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="system"} 35084 1500066716438 |
| 151 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "nice", value * 1 / 1 delta gives percentage (counter) |
| 152 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="nice"} 505 1500066716438 |
| 153 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "iowait", value * 1 / 1 delta gives percentage (counter) |
| 154 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="iowait"} 23314 1500066716438 |
| 155 | # COMMENT netdata_system_cpu_total: chart "system.cpu", context "system.cpu", family "cpu", dimension "idle", value * 1 / 1 delta gives percentage (counter) |
| 156 | netdata_system_cpu_total{chart="system.cpu",family="cpu",dimension="idle"} 918470 1500066716438 |
| 157 | ``` |
| 158 | |
| 159 | </details> |
| 160 | |
| 161 | For more information, check Prometheus documentation. |
| 162 | |
| 163 | ### Streaming Data from Upstream Hosts |
| 164 | |
| 165 | The `format=prometheus` parameter only exports the local host's metrics. For parent-child Netdata setups, use this configuration in **prometheus.yml**: |
| 166 | |
| 167 | ```yaml |
| 168 | metrics_path: '/api/v1/allmetrics' |
| 169 | params: |
| 170 | format: [prometheus_all_hosts] |
| 171 | honor_labels: true |
| 172 | ``` |
| 173 | |
| 174 | This reports all upstream host data with proper instance names. |
| 175 | |
| 176 | ### Timestamps |
| 177 | |
| 178 | To pass metrics through Prometheus pushgateway, use `×tamps=no` to send metrics without timestamps. |
| 179 | |
| 180 | ## Netdata Host Variables |
| 181 | |
| 182 | Netdata collects system configuration metrics (max TCP sockets, system-wide file limits, IPC sizes, etc.) not exposed to Prometheus by default. |
| 183 | |
| 184 | To expose them, append `variables=yes` to the URL. |
| 185 | |
| 186 | ### TYPE and HELP |
| 187 | |
| 188 | `# TYPE` and `# HELP` lines are suppressed by default to save bandwidth (Prometheus doesn't use them). Re-enable with: |
| 189 | |
| 190 | ``` |
| 191 | /api/v1/allmetrics?format=prometheus&types=yes&help=yes |
| 192 | ``` |
| 193 | |
| 194 | :::warning |
| 195 | |
| 196 | When enabled, `# TYPE` and `# HELP` lines repeat for every metric occurrence, violating [Prometheus specifications](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#comments-help-text-and-type-information). |
| 197 | |
| 198 | ::: |
| 199 | |
| 200 | ### Names and IDs |
| 201 | |
| 202 | Netdata supports both names and IDs for charts and dimensions. IDs are unique system identifiers; names are human-friendly labels (also unique). Most charts have identical ID and name, but some differ (device-mapper disks, interrupts, QoS classes, statsd synthetic charts). |
| 203 | |
| 204 | Control the default in `exporting.conf`: |
| 205 | |
| 206 | ```text |
| 207 | [prometheus:exporter] |
| 208 | send names instead of ids = yes | no |
| 209 | ``` |
| 210 | |
| 211 | Override via URL: |
| 212 | |
| 213 | - `&names=no` for IDs (old behavior) |
| 214 | - `&names=yes` for names |
| 215 | |
| 216 | ### Filtering Metrics Sent to Prometheus |
| 217 | |
| 218 | Filter metrics with this setting: |
| 219 | |
| 220 | ```text |
| 221 | [prometheus:exporter] |
| 222 | send charts matching = * |
| 223 | ``` |
| 224 | |
| 225 | This accepts space-separated [simple patterns](/src/libnetdata/simple_pattern/README.md) to match charts. Pattern rules: |
| 226 | |
| 227 | - `*` as wildcard (e.g., `*a*b*c*` is valid) |
| 228 | - `!` prefix for negative match |
| 229 | - First match (positive or negative) wins |
| 230 | - Example: `!*.bad users.* groups.*` sends all users and groups except `bad` ones |
| 231 | |
| 232 | ### Changing the Prefix of Netdata Metrics |
| 233 | |
| 234 | Netdata prefixes all metrics with `netdata_`. Change in `netdata.conf`: |
| 235 | |
| 236 | ```text |
| 237 | [prometheus:exporter] |
| 238 | prefix = netdata |
| 239 | ``` |
| 240 | |
| 241 | Or append `&prefix=netdata` to the URL. |
| 242 | |
| 243 | ### Metric Units |
| 244 | |
| 245 | | Source | Unit Behavior | Control | |
| 246 | |:--------------------|:------------------------------------------|:------------------------------------| |
| 247 | | `average` (default) | Adds units to names (e.g., `_KiB_persec`) | `&hideunits=yes` to hide | |
| 248 | | `as-collected` | No units in names | N/A | |
| 249 | | All sources | v1.12+ standardized units | `&oldunits=yes` for pre-v1.12 names | |
| 250 | |
| 251 | ### Accuracy of Average and Sum Data Sources |
| 252 | |
| 253 | When using `average` or `sum` sources, Netdata remembers each client's last access time to calculate values for the period since last access. This prevents data loss regardless of scrape frequency. |
| 254 | |
| 255 | Server identification: |
| 256 | | Scenario | Method | |
| 257 | |:---------|:-------| |
| 258 | | Direct connection | Client IP (default) | |
| 259 | | Behind proxy or NAT | Append `&server=NAME` to URL | |
| 260 | | Multiple servers, same IP | Each uses unique `&server=NAME` | |
| 261 | |
| 262 | ### Host Labels |
| 263 | |
| 264 | Netdata supports custom host labels that are exported to Prometheus. Configure labels in `/etc/netdata/netdata.conf`: |
| 265 | |
| 266 | ```ini |
| 267 | [host labels] |
| 268 | environment = production |
| 269 | region = us-east-1 |
| 270 | datacenter = dc1 |
| 271 | ``` |
| 272 | |
| 273 | After defining them, the labels will appear in the `netdata_info` metric, for example: |
| 274 | |
| 275 | ```text |
| 276 | netdata_info{ |
| 277 | instance="some-server", |
| 278 | application="netdata", |
| 279 | version="v2.7.2", |
| 280 | datacenter="dc1", |
| 281 | region="us-east-1", |
| 282 | environment="production", |
| 283 | _is_ephemeral="false" |
| 284 | } 1 1761148307085 |
| 285 | ``` |
| 286 | |
| 287 | ## Configure Prometheus to Scrape Netdata Metrics |
| 288 | |
| 289 | The following `prometheus.yml` scrapes all Netdata metrics "as collected": |
| 290 | |
| 291 | ```yaml |
| 292 | # my global config |
| 293 | global: |
| 294 | scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default is every 1 minute. |
| 295 | evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute. |
| 296 | # scrape_timeout is set to the global default (10s). |
| 297 | |
| 298 | # Attach these labels to any time series or alerts when communicating with |
| 299 | # external systems (federation, remote storage, Alertmanager). |
| 300 | external_labels: |
| 301 | monitor: 'codelab-monitor' |
| 302 | |
| 303 | # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. |
| 304 | rule_files: |
| 305 | # - "first.rules" |
| 306 | # - "second.rules" |
| 307 | |
| 308 | # A scrape configuration containing exactly one endpoint to scrape: |
| 309 | # Here it's Prometheus itself. |
| 310 | scrape_configs: |
| 311 | # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. |
| 312 | - job_name: 'prometheus' |
| 313 | |
| 314 | # metrics_path defaults to '/metrics' |
| 315 | # scheme defaults to 'http'. |
| 316 | |
| 317 | static_configs: |
| 318 | - targets: ['0.0.0.0:9090'] |
| 319 | |
| 320 | - job_name: 'netdata-scrape' |
| 321 | |
| 322 | metrics_path: '/api/v1/allmetrics' |
| 323 | params: |
| 324 | # format: prometheus | prometheus_all_hosts |
| 325 | # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP |
| 326 | format: [prometheus] |
| 327 | # |
| 328 | # sources: as-collected | raw | average | sum | volume |
| 329 | # default is: average |
| 330 | #source: [as-collected] |
| 331 | # |
| 332 | # server name for this prometheus - the default is the client IP |
| 333 | # for Netdata to uniquely identify it |
| 334 | #server: ['prometheus1'] |
| 335 | honor_labels: true |
| 336 | |
| 337 | static_configs: |
| 338 | - targets: ['{your.netdata.ip}:19999'] |
| 339 | ``` |
| 340 | |
| 341 | Replace `{your.netdata.ip}` with your Netdata host's IP or hostname. |
| 342 | |
| 343 | ### Prometheus Alerts for Netdata Metrics |
| 344 | |
| 345 | Example `nodes.yml` file for generating alerts from Netdata metrics. Save at `/opt/prometheus/nodes.yml` and add `- "nodes.yml"` under `rule_files:` in prometheus.yml: |
| 346 | |
| 347 | ```yaml |
| 348 | groups: |
| 349 | - name: nodes |
| 350 | |
| 351 | rules: |
| 352 | - alert: node_high_cpu_usage_70 |
| 353 | expr: sum(sum_over_time(netdata_system_cpu_percentage_average{dimension=~"(user|system|softirq|irq|guest)"}[10m])) by (job) / sum(count_over_time(netdata_system_cpu_percentage_average{dimension="idle"}[10m])) by (job) > 70 |
| 354 | for: 1m |
| 355 | annotations: |
| 356 | description: '{{ $labels.job }} on ''{{ $labels.job }}'' CPU usage is at {{ humanize $value }}%.' |
| 357 | summary: CPU alert for container node '{{ $labels.job }}' |
| 358 | |
| 359 | - alert: node_high_memory_usage_70 |
| 360 | expr: 100 / sum(netdata_system_ram_MB_average) by (job) |
| 361 | * sum(netdata_system_ram_MB_average{dimension=~"free|cached"}) by (job) < 30 |
| 362 | for: 1m |
| 363 | annotations: |
| 364 | description: '{{ $labels.job }} memory usage is {{ humanize $value}}%.' |
| 365 | summary: Memory alert for container node '{{ $labels.job }}' |
| 366 | |
| 367 | - alert: node_low_root_filesystem_space_20 |
| 368 | expr: 100 / sum(netdata_disk_space_GB_average{family="/"}) by (job) |
| 369 | * sum(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}) by (job) < 20 |
| 370 | for: 1m |
| 371 | annotations: |
| 372 | description: '{{ $labels.job }} root filesystem space is {{ humanize $value}}%.' |
| 373 | summary: Root filesystem alert for container node '{{ $labels.job }}' |
| 374 | |
| 375 | - alert: node_root_filesystem_fill_rate_6h |
| 376 | expr: predict_linear(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}[1h], 6 * 3600) < 0 |
| 377 | for: 1h |
| 378 | labels: |
| 379 | severity: critical |
| 380 | annotations: |
| 381 | description: Container node {{ $labels.job }} root filesystem is going to fill up in 6h. |
| 382 | summary: Disk fill alert for Swarm node '{{ $labels.job }}' |
| 383 | ``` |